{"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _terms(self):\n res = []\n for sign, terms in self.terms.items():\n for ID, lon in terms.items():\n res.append(self.T(ID, sign))\n return res", "response": "Returns a list with the objects as terms."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a sorted list with all the primary directions. Arcs and significators are returned.", "response": "def getList(self, aspList):\n \"\"\" Returns a sorted list with all\n primary directions. \n \n \"\"\"\n # Significators\n objects = self._elements(self.SIG_OBJECTS, self.N, [0])\n houses = self._elements(self.SIG_HOUSES, self.N, [0])\n angles = self._elements(self.SIG_ANGLES, self.N, [0])\n significators = objects + houses + angles\n \n # Promissors\n objects = self._elements(self.SIG_OBJECTS, self.N, aspList)\n terms = self._terms()\n antiscias = self._elements(self.SIG_OBJECTS, self.A, [0])\n cantiscias = self._elements(self.SIG_OBJECTS, self.C, [0])\n promissors = objects + terms + antiscias + cantiscias\n\n # Compute all\n res = []\n for prom in promissors:\n for sig in significators:\n if (prom['id'] == sig['id']):\n continue\n arcs = self._arc(prom, sig)\n for (x,y) in [('arcm', 'M'), ('arcz', 'Z')]:\n arc = arcs[x]\n if 0 < arc < self.MAX_ARC:\n res.append([\n arcs[x],\n prom['id'],\n sig['id'],\n y,\n ])\n\n return sorted(res)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the directions within the arcmin and arcmax arcs.", "response": "def view(self, arcmin, arcmax):\n \"\"\" Returns the directions within the\n min and max arcs.\n\n \"\"\"\n res = []\n for direction in self.table:\n if arcmin < direction[0] < arcmax:\n res.append(direction)\n return res"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning all directions to a significator.", "response": "def bySignificator(self, ID):\n \"\"\" Returns all directions to a significator. \"\"\"\n res = []\n for direction in self.table:\n if ID in direction[2]:\n res.append(direction)\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef byPromissor(self, ID):\n res = []\n for direction in self.table:\n if ID in direction[1]:\n res.append(direction)\n return res", "response": "Returns all directions to a promissor."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a deep copy of this chart.", "response": "def copy(self):\n \"\"\" Returns a deep copy of this chart. \"\"\"\n chart = Chart.__new__(Chart)\n chart.date = self.date\n chart.pos = self.pos\n chart.hsys = self.hsys\n chart.objects = self.objects.copy()\n chart.houses = self.houses.copy()\n chart.angles = self.angles.copy()\n return chart"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, ID):\n if ID.startswith('House'):\n return self.getHouse(ID)\n elif ID in const.LIST_ANGLES:\n return self.getAngle(ID)\n else:\n return self.getObject(ID)", "response": "Returns an object house or angle \n from the chart."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getFixedStars(self):\n IDs = const.LIST_FIXED_STARS\n return ephem.getFixedStarList(IDs, self.date)", "response": "Returns a list with all fixed stars."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef isHouse1Asc(self):\n house1 = self.getHouse(const.HOUSE1)\n asc = self.getAngle(const.ASC)\n dist = angle.closestdistance(house1.lon, asc.lon)\n return abs(dist) < 0.0003", "response": "Returns true if House1 is the same as Asc."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning true if House10 is the same as the MC.", "response": "def isHouse10MC(self):\n \"\"\" Returns true if House10 is the same as the MC. \"\"\"\n house10 = self.getHouse(const.HOUSE10)\n mc = self.getAngle(const.MC)\n dist = angle.closestdistance(house10.lon, mc.lon)\n return abs(dist) < 0.0003"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning true if this chart is diurnal.", "response": "def isDiurnal(self):\n \"\"\" Returns true if this chart is diurnal. \"\"\"\n sun = self.getObject(const.SUN)\n mc = self.getAngle(const.MC)\n \n # Get ecliptical positions and check if the\n # sun is above the horizon.\n lat = self.pos.lat\n sunRA, sunDecl = utils.eqCoords(sun.lon, sun.lat)\n mcRA, mcDecl = utils.eqCoords(mc.lon, 0)\n return utils.isAboveHorizon(sunRA, sunDecl, mcRA, lat)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef getMoonPhase(self):\n sun = self.getObject(const.SUN)\n moon = self.getObject(const.MOON)\n dist = angle.distance(sun.lon, moon.lon)\n if dist < 90:\n return const.MOON_FIRST_QUARTER\n elif dist < 180:\n return const.MOON_SECOND_QUARTER\n elif dist < 270:\n return const.MOON_THIRD_QUARTER\n else:\n return const.MOON_LAST_QUARTER", "response": "Returns the phase of the moon."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef solarReturn(self, year):\n sun = self.getObject(const.SUN)\n date = Datetime('{0}/01/01'.format(year),\n '00:00',\n self.date.utcoffset)\n srDate = ephem.nextSolarReturn(date, sun.lon)\n return Chart(srDate, self.pos, hsys=self.hsys)", "response": "Returns this chart s solar return for a \n given year."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the longitude of an object.", "response": "def objLon(ID, chart):\n \"\"\" Returns the longitude of an object. \"\"\"\n if ID.startswith('$R'):\n # Return Ruler\n ID = ID[2:]\n obj = chart.get(ID)\n rulerID = essential.ruler(obj.sign)\n ruler = chart.getObject(rulerID)\n return ruler.lon\n elif ID.startswith('Pars'):\n # Return an arabic part\n return partLon(ID, chart)\n else:\n # Return an object\n obj = chart.get(ID)\n return obj.lon"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the longitude of an arabic part.", "response": "def partLon(ID, chart):\n \"\"\" Returns the longitude of an arabic part. \"\"\"\n # Get diurnal or nocturnal formula\n abc = FORMULAS[ID][0] if chart.isDiurnal() else FORMULAS[ID][1]\n a = objLon(abc[0], chart)\n b = objLon(abc[1], chart)\n c = objLon(abc[2], chart)\n return c + b - a"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getPart(ID, chart):\n obj = GenericObject()\n obj.id = ID\n obj.type = const.OBJ_ARABIC_PART\n obj.relocate(partLon(ID, chart))\n return obj", "response": "Returns an Arabic Part."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an object from the Ephemeris.", "response": "def sweObject(obj, jd):\n \"\"\" Returns an object from the Ephemeris. \"\"\"\n sweObj = SWE_OBJECTS[obj]\n sweList = swisseph.calc_ut(jd, sweObj)\n return {\n 'id': obj,\n 'lon': sweList[0],\n 'lat': sweList[1],\n 'lonspeed': sweList[3],\n 'latspeed': sweList[4]\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sweObjectLon(obj, jd):\n sweObj = SWE_OBJECTS[obj]\n sweList = swisseph.calc_ut(jd, sweObj)\n return sweList[0]", "response": "Returns the longitude of an object in the given JD."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the julian date of the next transit of the object.", "response": "def sweNextTransit(obj, jd, lat, lon, flag):\n \"\"\" Returns the julian date of the next transit of\n an object. The flag should be 'RISE' or 'SET'. \n \n \"\"\"\n sweObj = SWE_OBJECTS[obj]\n flag = swisseph.CALC_RISE if flag == 'RISE' else swisseph.CALC_SET\n trans = swisseph.rise_trans(jd, sweObj, lon, lat, 0, 0, 0, flag)\n return trans[1][0]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of lists of houses and angles.", "response": "def sweHouses(jd, lat, lon, hsys):\n \"\"\" Returns lists of houses and angles. \"\"\"\n hsys = SWE_HOUSESYS[hsys]\n hlist, ascmc = swisseph.houses(jd, lat, lon, hsys)\n # Add first house to the end of 'hlist' so that we\n # can compute house sizes with an iterator \n hlist += (hlist[0],)\n houses = [\n {\n 'id': const.LIST_HOUSES[i],\n 'lon': hlist[i], \n 'size': angle.distance(hlist[i], hlist[i+1])\n } for i in range(12)\n ]\n angles = [\n {'id': const.ASC, 'lon': ascmc[0]}, \n {'id': const.MC, 'lon': ascmc[1]},\n {'id': const.DESC, 'lon': angle.norm(ascmc[0] + 180)},\n {'id': const.IC, 'lon': angle.norm(ascmc[1] + 180)}\n ]\n return (houses, angles)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn lists with house and angle longitudes.", "response": "def sweHousesLon(jd, lat, lon, hsys):\n \"\"\" Returns lists with house and angle longitudes. \"\"\"\n hsys = SWE_HOUSESYS[hsys]\n hlist, ascmc = swisseph.houses(jd, lat, lon, hsys)\n angles = [\n ascmc[0],\n ascmc[1],\n angle.norm(ascmc[0] + 180), \n angle.norm(ascmc[1] + 180)\n ]\n return (hlist, angles)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a fixed star from the Ephemeris.", "response": "def sweFixedStar(star, jd):\n \"\"\" Returns a fixed star from the Ephemeris. \"\"\"\n sweList = swisseph.fixstar_ut(star, jd)\n mag = swisseph.fixstar_mag(star)\n return {\n 'id': star, \n 'mag': mag,\n 'lon': sweList[0],\n 'lat': sweList[1]\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the jd details of the previous or next global solar eclipse.", "response": "def solarEclipseGlobal(jd, backward):\n \"\"\" Returns the jd details of previous or next global solar eclipse. \"\"\"\n\n sweList = swisseph.sol_eclipse_when_glob(jd, backward=backward)\n return {\n 'maximum': sweList[1][0],\n 'begin': sweList[1][2],\n 'end': sweList[1][3],\n 'totality_begin': sweList[1][4],\n 'totality_end': sweList[1][5],\n 'center_line_begin': sweList[1][6],\n 'center_line_end': sweList[1][7],\n }"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lunarEclipseGlobal(jd, backward):\n\n sweList = swisseph.lun_eclipse_when(jd, backward=backward)\n return {\n 'maximum': sweList[1][0],\n 'partial_begin': sweList[1][2],\n 'partial_end': sweList[1][3],\n 'totality_begin': sweList[1][4],\n 'totality_end': sweList[1][5],\n 'penumbral_begin': sweList[1][6],\n 'penumbral_end': sweList[1][7],\n }", "response": "Returns the jd details of the previous or next global lunar eclipse."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert date to Julian Day Number.", "response": "def dateJDN(year, month, day, calendar):\n \"\"\" Converts date to Julian Day Number. \"\"\"\n a = (14 - month) // 12\n y = year + 4800 - a\n m = month + 12*a - 3\n if calendar == GREGORIAN:\n return day + (153*m + 2)//5 + 365*y + y//4 - y//100 + y//400 - 32045\n else:\n return day + (153*m + 2)//5 + 365*y + y//4 - 32083"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts Julian Day Number to Gregorian date.", "response": "def jdnDate(jdn):\n \"\"\" Converts Julian Day Number to Gregorian date. \"\"\"\n a = jdn + 32044\n b = (4*a + 3) // 146097\n c = a - (146097*b) // 4\n d = (4*c + 3) // 1461\n e = c - (1461*d) // 4\n m = (5*e + 2) // 153\n day = e + 1 - (153*m + 2) // 5\n month = m + 3 - 12*(m//10)\n year = 100*b + d - 4800 + m//10\n return [year, month, day]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef toList(self):\n date = self.date()\n sign = '+' if date[0] >= 0 else '-'\n date[0] = abs(date[0])\n return list(sign) + date", "response": "Returns date as signed list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a string representation of the current time entry.", "response": "def toString(self):\n \"\"\" Returns date as string. \"\"\"\n slist = self.toList()\n sign = '' if slist[0] == '+' else '-'\n string = '/'.join(['%02d' % v for v in slist[1:]])\n return sign + string"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getUTC(self, utcoffset):\n newTime = (self.value - utcoffset.value) % 24\n return Time(newTime)", "response": "Returns a new Time object set to UTC given \n an offset Time object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn time as list [ hh mm ss ]", "response": "def time(self):\n \"\"\" Returns time as list [hh,mm,ss]. \"\"\"\n slist = self.toList()\n if slist[0] == '-':\n slist[1] *= -1\n # We must do a trick if we want to \n # make negative zeros explicit\n if slist[1] == -0:\n slist[1] = -0.0\n return slist[1:]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn time as signed list.", "response": "def toList(self):\n \"\"\" Returns time as signed list. \"\"\"\n slist = angle.toList(self.value)\n # Keep hours in 0..23\n slist[1] = slist[1] % 24\n return slist"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn time as string.", "response": "def toString(self):\n \"\"\" Returns time as string. \"\"\"\n slist = self.toList()\n string = angle.slistStr(slist)\n return string if slist[0] == '-' else string[1:]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nbuild a Datetime object given a jd and utc offset.", "response": "def fromJD(jd, utcoffset):\n \"\"\" Builds a Datetime object given a jd and utc offset. \"\"\"\n if not isinstance(utcoffset, Time):\n utcoffset = Time(utcoffset)\n localJD = jd + utcoffset.value / 24.0\n date = Date(round(localJD))\n time = Time((localJD + 0.5 - date.jdn) * 24)\n return Datetime(date, time, utcoffset)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn this Datetime localized for UTC.", "response": "def getUTC(self):\n \"\"\" Returns this Datetime localized for UTC. \"\"\"\n timeUTC = self.time.getUTC(self.utcoffset)\n dateUTC = Date(round(self.jd))\n return Datetime(dateUTC, timeUTC)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns an object for a specific date and location.", "response": "def getObject(ID, jd, lat, lon):\n \"\"\" Returns an object for a specific date and \n location.\n \n \"\"\"\n if ID == const.SOUTH_NODE:\n obj = swe.sweObject(const.NORTH_NODE, jd)\n obj.update({\n 'id': const.SOUTH_NODE,\n 'lon': angle.norm(obj['lon'] + 180)\n })\n elif ID == const.PARS_FORTUNA:\n pflon = tools.pfLon(jd, lat, lon)\n obj = {\n 'id': ID,\n 'lon': pflon,\n 'lat': 0,\n 'lonspeed': 0,\n 'latspeed': 0\n }\n elif ID == const.SYZYGY:\n szjd = tools.syzygyJD(jd)\n obj = swe.sweObject(const.MOON, szjd)\n obj['id'] = const.SYZYGY\n else:\n obj = swe.sweObject(ID, jd)\n \n _signInfo(obj)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getHouses(jd, lat, lon, hsys):\n houses, angles = swe.sweHouses(jd, lat, lon, hsys)\n for house in houses:\n _signInfo(house)\n for angle in angles:\n _signInfo(angle)\n return (houses, angles)", "response": "Returns a list of lists of houses and angles."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a fixed star.", "response": "def getFixedStar(ID, jd):\n \"\"\" Returns a fixed star. \"\"\"\n star = swe.sweFixedStar(ID, jd)\n _signInfo(star)\n return star"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the JD of the next sunrise.", "response": "def nextSunrise(jd, lat, lon):\n \"\"\" Returns the JD of the next sunrise. \"\"\"\n return swe.sweNextTransit(const.SUN, jd, lat, lon, 'RISE')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the JD of the next sunset", "response": "def nextSunset(jd, lat, lon):\n \"\"\" Returns the JD of the next sunset. \"\"\"\n return swe.sweNextTransit(const.SUN, jd, lat, lon, 'SET')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds the sign id and longitude to an object.", "response": "def _signInfo(obj):\n \"\"\" Appends the sign id and longitude to an object. \"\"\"\n lon = obj['lon']\n obj.update({\n 'sign': const.LIST_SIGNS[int(lon / 30)],\n 'signlon': lon % 30\n })"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pfLon(jd, lat, lon):\n sun = swe.sweObjectLon(const.SUN, jd)\n moon = swe.sweObjectLon(const.MOON, jd)\n asc = swe.sweHousesLon(jd, lat, lon,\n const.HOUSES_DEFAULT)[1][0]\n \n if isDiurnal(jd, lat, lon):\n return angle.norm(asc + moon - sun)\n else:\n return angle.norm(asc + sun - moon)", "response": "Returns the ecliptic longitude of Pars Fortuna."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns true if the sun is above the horizon of a given date and location.", "response": "def isDiurnal(jd, lat, lon):\n \"\"\" Returns true if the sun is above the horizon\n of a given date and location. \n \n \"\"\"\n sun = swe.sweObject(const.SUN, jd)\n mc = swe.sweHousesLon(jd, lat, lon, \n const.HOUSES_DEFAULT)[1][1]\n ra, decl = utils.eqCoords(sun['lon'], sun['lat'])\n mcRA, _ = utils.eqCoords(mc, 0.0)\n return utils.isAboveHorizon(ra, decl, mcRA, lat)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind the latest new or full moon and returns the julian date of that event.", "response": "def syzygyJD(jd):\n \"\"\" Finds the latest new or full moon and\n returns the julian date of that event. \n \n \"\"\"\n sun = swe.sweObjectLon(const.SUN, jd)\n moon = swe.sweObjectLon(const.MOON, jd)\n dist = angle.distance(sun, moon)\n \n # Offset represents the Syzygy type. \n # Zero is conjunction and 180 is opposition.\n offset = 180 if (dist >= 180) else 0\n while abs(dist) > MAX_ERROR:\n jd = jd - dist / 13.1833 # Moon mean daily motion\n sun = swe.sweObjectLon(const.SUN, jd)\n moon = swe.sweObjectLon(const.MOON, jd)\n dist = angle.closestdistance(sun - offset, moon)\n return jd"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds the julian date before or after jd when the sun is at longitude lon.", "response": "def solarReturnJD(jd, lon, forward=True):\n \"\"\" Finds the julian date before or after \n 'jd' when the sun is at longitude 'lon'. \n It searches forward by default.\n \n \"\"\"\n sun = swe.sweObjectLon(const.SUN, jd)\n if forward:\n dist = angle.distance(sun, lon)\n else:\n dist = -angle.distance(lon, sun)\n \n while abs(dist) > MAX_ERROR:\n jd = jd + dist / 0.9833 # Sun mean motion\n sun = swe.sweObjectLon(const.SUN, jd)\n dist = angle.closestdistance(sun, lon)\n return jd"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef nextStationJD(ID, jd):\n speed = swe.sweObject(ID, jd)['lonspeed']\n for i in range(2000):\n nextjd = jd + i / 2\n nextspeed = swe.sweObject(ID, nextjd)['lonspeed']\n if speed * nextspeed <= 0:\n return nextjd\n return None", "response": "Finds the aproximate julian date of the next station of a planet. Returns None if no next station is found."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef clean_caches(path):\n\n for dirname, subdirlist, filelist in os.walk(path):\n\n for f in filelist:\n if f.endswith('pyc'):\n try:\n os.remove(os.path.join(dirname, f))\n except FileNotFoundError:\n pass\n\n if dirname.endswith('__pycache__'):\n shutil.rmtree(dirname)", "response": "Removes all python cache files recursively on a path."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean_py_files(path):\n\n for dirname, subdirlist, filelist in os.walk(path):\n\n for f in filelist:\n if f.endswith('py'):\n os.remove(os.path.join(dirname, f))", "response": "Removes all. py files in the path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef toFloat(value):\n if isinstance(value, str):\n # Find lat/lon char in string and insert angle sign\n value = value.upper()\n for char in ['N', 'S', 'E', 'W']:\n if char in value:\n value = SIGN[char] + value.replace(char, ':')\n break\n return angle.toFloat(value)", "response": "Converts angle representation to float. \n Accepts strings such as 12W30 : 00."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a float to a string.", "response": "def toString(value, mode):\n \"\"\" Converts angle float to string. \n Mode refers to LAT/LON.\n \n \"\"\"\n string = angle.toString(value)\n sign = string[0]\n separator = CHAR[mode][sign]\n string = string.replace(':', separator, 1)\n return string[1:]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns lat / lon as strings.", "response": "def strings(self):\n \"\"\" Return lat/lon as strings. \"\"\"\n return [\n toString(self.lat, LAT),\n toString(self.lon, LON)\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list with the orb and angular distances from obj1 to obj2 considering a list of possible aspects.", "response": "def _orbList(obj1, obj2, aspList):\n \"\"\" Returns a list with the orb and angular\n distances from obj1 to obj2, considering a\n list of possible aspects. \n \n \"\"\"\n sep = angle.closestdistance(obj1.lon, obj2.lon)\n absSep = abs(sep)\n return [\n {\n 'type': asp,\n 'orb': abs(absSep - asp),\n 'separation': sep,\n } for asp in aspList\n ]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _aspectDict(obj1, obj2, aspList):\n # Ignore aspects from same and Syzygy\n if obj1 == obj2 or obj1.id == const.SYZYGY:\n return None\n \n orbs = _orbList(obj1, obj2, aspList)\n for aspDict in orbs:\n asp = aspDict['type']\n orb = aspDict['orb'] \n \n # Check if aspect is within orb\n if asp in const.MAJOR_ASPECTS:\n # Ignore major aspects out of orb\n if obj1.orb() < orb and obj2.orb() < orb:\n continue\n else:\n # Ignore minor aspects out of max orb\n if MAX_MINOR_ASP_ORB < orb:\n continue\n \n # Only conjunctions for Pars Fortuna and Nodes\n if obj1.id in [const.PARS_FORTUNA, \n const.NORTH_NODE, \n const.SOUTH_NODE] and \\\n asp != const.CONJUNCTION:\n continue\n \n # We have a valid aspect within orb\n return aspDict\n\n return None", "response": "Returns the properties of the aspect of \n obj1 to obj2 considering a list of possible aspects."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the properties of an aspect between obj1 and obj2 given by aspDict.", "response": "def _aspectProperties(obj1, obj2, aspDict):\n \"\"\" Returns the properties of an aspect between\n obj1 and obj2, given by 'aspDict'. \n \n This function assumes obj1 to be the active object, \n i.e., the one responsible for starting the aspect.\n \n \"\"\"\n orb = aspDict['orb']\n asp = aspDict['type']\n sep = aspDict['separation']\n \n # Properties\n prop1 = {\n 'id': obj1.id,\n 'inOrb': False,\n 'movement': const.NO_MOVEMENT \n }\n prop2 = {\n 'id': obj2.id,\n 'inOrb': False,\n 'movement': const.NO_MOVEMENT \n }\n prop = {\n 'type': asp,\n 'orb': orb,\n 'direction': -1,\n 'condition': -1,\n 'active': prop1,\n 'passive': prop2 \n }\n \n if asp == const.NO_ASPECT:\n return prop\n \n # Aspect within orb\n prop1['inOrb'] = orb <= obj1.orb()\n prop2['inOrb'] = orb <= obj2.orb()\n \n # Direction\n prop['direction'] = const.DEXTER if sep <= 0 else const.SINISTER\n \n # Sign conditions\n # Note: if obj1 is before obj2, orbDir will be less than zero\n orbDir = sep-asp if sep >= 0 else sep+asp\n offset = obj1.signlon + orbDir\n if 0 <= offset < 30:\n prop['condition'] = const.ASSOCIATE\n else:\n prop['condition'] = const.DISSOCIATE \n \n # Movement of the individual objects\n if abs(orbDir) < MAX_EXACT_ORB:\n prop1['movement'] = prop2['movement'] = const.EXACT\n else:\n # Active object applies to Passive if it is before \n # and direct, or after the Passive and Rx..\n prop1['movement'] = const.SEPARATIVE\n if (orbDir > 0 and obj1.isDirect()) or \\\n (orbDir < 0 and obj1.isRetrograde()):\n prop1['movement'] = const.APPLICATIVE\n elif obj1.isStationary():\n prop1['movement'] = const.STATIONARY\n \n # The Passive applies or separates from the Active \n # if it has a different direction..\n # Note: Non-planets have zero speed\n prop2['movement'] = const.NO_MOVEMENT\n obj2speed = obj2.lonspeed if obj2.isPlanet() else 0.0\n sameDir = obj1.lonspeed * obj2speed >= 0\n if not sameDir:\n prop2['movement'] = prop1['movement']\n \n return prop"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _getActivePassive(obj1, obj2):\n speed1 = abs(obj1.lonspeed) if obj1.isPlanet() else -1.0\n speed2 = abs(obj2.lonspeed) if obj2.isPlanet() else -1.0\n if speed1 > speed2:\n return {\n 'active': obj1,\n 'passive': obj2\n }\n else:\n return {\n 'active': obj2,\n 'passive': obj1\n }", "response": "Returns which is the active and the passive objects."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the aspect type between objects considering a list of possible aspect types.", "response": "def aspectType(obj1, obj2, aspList):\n \"\"\" Returns the aspect type between objects considering\n a list of possible aspect types.\n \n \"\"\"\n ap = _getActivePassive(obj1, obj2)\n aspDict = _aspectDict(ap['active'], ap['passive'], aspList)\n return aspDict['type'] if aspDict else const.NO_ASPECT"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef hasAspect(obj1, obj2, aspList):\n aspType = aspectType(obj1, obj2, aspList)\n return aspType != const.NO_ASPECT", "response": "Returns True if there is an aspect between objects \n considering a list of possible aspect types."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning True if obj1 aspects obj2 within its orb considering a list of possible aspect types.", "response": "def isAspecting(obj1, obj2, aspList):\n \"\"\" Returns if obj1 aspects obj2 within its orb,\n considering a list of possible aspect types. \n \n \"\"\"\n aspDict = _aspectDict(obj1, obj2, aspList)\n if aspDict:\n return aspDict['orb'] < obj1.orb()\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getAspect(obj1, obj2, aspList):\n ap = _getActivePassive(obj1, obj2)\n aspDict = _aspectDict(ap['active'], ap['passive'], aspList)\n if not aspDict:\n aspDict = {\n 'type': const.NO_ASPECT,\n 'orb': 0,\n 'separation': 0,\n } \n aspProp = _aspectProperties(ap['active'], ap['passive'], aspDict)\n return Aspect(aspProp)", "response": "Returns an Aspect object for the aspect between two objects considering a list of possible aspect types."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the movement of this aspect.", "response": "def movement(self):\n \"\"\" Returns the movement of this aspect. \n The movement is the one of the active object, except\n if the active is separating but within less than 1 \n degree.\n \n \"\"\"\n mov = self.active.movement\n if self.orb < 1 and mov == const.SEPARATIVE:\n mov = const.EXACT\n return mov"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getRole(self, ID):\n if self.active.id == ID:\n return {\n 'role': 'active',\n 'inOrb': self.active.inOrb,\n 'movement': self.active.movement\n }\n elif self.passive.id == ID:\n return {\n 'role': 'passive',\n 'inOrb': self.passive.inOrb,\n 'movement': self.passive.movement\n }\n return None", "response": "Returns the role of an object in this aspect."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the default faces for the given variant", "response": "def setFaces(variant):\n \"\"\"\n Sets the default faces variant\n\n \"\"\"\n global FACES\n if variant == CHALDEAN_FACES:\n FACES = tables.CHALDEAN_FACES\n else:\n FACES = tables.TRIPLICITY_FACES"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef setTerms(variant):\n global TERMS\n if variant == EGYPTIAN_TERMS:\n TERMS = tables.EGYPTIAN_TERMS\n elif variant == TETRABIBLOS_TERMS:\n TERMS = tables.TETRABIBLOS_TERMS\n elif variant == LILLY_TERMS:\n TERMS = tables.LILLY_TERMS", "response": "Sets the default terms of the Dignities\n table."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the term for a sign and longitude.", "response": "def term(sign, lon):\n \"\"\" Returns the term for a sign and longitude. \"\"\"\n terms = TERMS[sign]\n for (ID, a, b) in terms:\n if (a <= lon < b):\n return ID\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the face for a sign and longitude.", "response": "def face(sign, lon):\n \"\"\" Returns the face for a sign and longitude. \"\"\"\n faces = FACES[sign]\n if lon < 10:\n return faces[0]\n elif lon < 20:\n return faces[1]\n else:\n return faces[2]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getInfo(sign, lon):\n return {\n 'ruler': ruler(sign),\n 'exalt': exalt(sign),\n 'dayTrip': dayTrip(sign),\n 'nightTrip': nightTrip(sign),\n 'partTrip': partTrip(sign),\n 'term': term(sign, lon),\n 'face': face(sign, lon),\n 'exile': exile(sign),\n 'fall': fall(sign)\n }", "response": "Returns the complete essential dignities\n for a sign and longitude."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if an object is peregrine on a sign and longitude.", "response": "def isPeregrine(ID, sign, lon):\n \"\"\" Returns if an object is peregrine\n on a sign and longitude.\n\n \"\"\"\n info = getInfo(sign, lon)\n for dign, objID in info.items():\n if dign not in ['exile', 'fall'] and ID == objID:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the score of an object on a sign and longitude.", "response": "def score(ID, sign, lon):\n \"\"\" Returns the score of an object on\n a sign and longitude.\n\n \"\"\"\n info = getInfo(sign, lon)\n dignities = [dign for (dign, objID) in info.items() if objID == ID]\n return sum([SCORES[dign] for dign in dignities])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the almutem for a given sign and longitude.", "response": "def almutem(sign, lon):\n \"\"\" Returns the almutem for a given\n sign and longitude.\n\n \"\"\"\n planets = const.LIST_SEVEN_PLANETS\n res = [None, 0]\n for ID in planets:\n sc = score(ID, sign, lon)\n if sc > res[1]:\n res = [ID, sc]\n return res[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the dignities belonging to this object.", "response": "def getDignities(self):\n \"\"\" Returns the dignities belonging to this object. \"\"\"\n info = self.getInfo()\n dignities = [dign for (dign, objID) in info.items()\n if objID == self.obj.id]\n return dignities"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef isPeregrine(self):\n return isPeregrine(self.obj.id,\n self.obj.sign,\n self.obj.signlon)", "response": "Returns if this object is peregrine."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the solar return of a Chart after a specific date.", "response": "def nextSolarReturn(chart, date):\n \"\"\" Returns the solar return of a Chart\n after a specific date.\n \n \"\"\"\n sun = chart.getObject(const.SUN)\n srDate = ephem.nextSolarReturn(date, sun.lon)\n return _computeChart(chart, srDate)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef hourTable(date, pos):\n \n lastSunrise = ephem.lastSunrise(date, pos)\n middleSunset = ephem.nextSunset(lastSunrise, pos)\n nextSunrise = ephem.nextSunrise(date, pos)\n table = []\n \n # Create diurnal hour sequence\n length = (middleSunset.jd - lastSunrise.jd) / 12.0\n for i in range(12):\n start = lastSunrise.jd + i * length\n end = start + length\n ruler = nthRuler(i, lastSunrise.date.dayofweek())\n table.append([start, end, ruler])\n \n # Create nocturnal hour sequence\n length = (nextSunrise.jd - middleSunset.jd) / 12.0\n for i in range(12):\n start = middleSunset.jd + i * length\n end = start + length\n ruler = nthRuler(i + 12, lastSunrise.date.dayofweek())\n table.append([start, end, ruler])\n \n return table", "response": "Creates the planetary hour table for a date \n and position."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getHourTable(date, pos):\n table = hourTable(date, pos)\n return HourTable(table, date)", "response": "Returns an HourTable object for the given date and position."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef index(self, date):\n for (i, (start, end, ruler)) in enumerate(self.table):\n if start <= date.jd <= end:\n return i\n return None", "response": "Returns the index of a date in the table."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef indexInfo(self, index):\n entry = self.table[index]\n info = {\n # Default is diurnal\n 'mode': 'Day',\n 'ruler': self.dayRuler(),\n 'dayRuler': self.dayRuler(),\n 'nightRuler': self.nightRuler(),\n 'hourRuler': entry[2],\n 'hourNumber': index + 1,\n 'tableIndex': index,\n 'start': Datetime.fromJD(entry[0], self.date.utcoffset),\n 'end': Datetime.fromJD(entry[1], self.date.utcoffset)\n }\n if index >= 12:\n # Set information as nocturnal\n info.update({\n 'mode': 'Night',\n 'ruler': info['nightRuler'],\n 'hourNumber': index + 1 - 12\n })\n return info", "response": "Returns information about a specific \n planetary time. \n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing a profection chart for a given date.", "response": "def compute(chart, date, fixedObjects=False):\n \"\"\" Returns a profection chart for a given\n date. Receives argument 'fixedObjects' to\n fix chart objects in their natal locations.\n \n \"\"\"\n \n sun = chart.getObject(const.SUN)\n prevSr = ephem.prevSolarReturn(date, sun.lon)\n nextSr = ephem.nextSolarReturn(date, sun.lon)\n \n # In one year, rotate chart 30\u00ba\n rotation = 30 * (date.jd - prevSr.jd) / (nextSr.jd - prevSr.jd)\n \n # Include 30\u00ba for each previous year\n age = math.floor((date.jd - chart.date.jd) / 365.25)\n rotation = 30 * age + rotation\n \n # Create a copy of the chart and rotate content\n pChart = chart.copy()\n for obj in pChart.objects:\n if not fixedObjects:\n obj.relocate(obj.lon + rotation)\n for house in pChart.houses:\n house.relocate(house.lon + rotation)\n for angle in pChart.angles:\n angle.relocate(angle.lon + rotation)\n \n return pChart"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmerges two list of objects removing repetitions.", "response": "def _merge(listA, listB):\n \"\"\" Merges two list of objects removing\n repetitions. \n \n \"\"\"\n listA = [x.id for x in listA]\n listB = [x.id for x in listB]\n listA.extend(listB)\n set_ = set(listA)\n return list(set_)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing the behavior of the current object.", "response": "def compute(chart):\n \"\"\" Computes the behavior. \"\"\"\n \n factors = []\n \n # Planets in House1 or Conjunct Asc\n house1 = chart.getHouse(const.HOUSE1)\n planetsHouse1 = chart.objects.getObjectsInHouse(house1)\n asc = chart.getAngle(const.ASC)\n planetsConjAsc = chart.objects.getObjectsAspecting(asc, [0])\n \n _set = _merge(planetsHouse1, planetsConjAsc)\n factors.append(['Planets in House1 or Conj Asc', _set])\n \n # Planets conjunct Moon or Mercury\n moon = chart.get(const.MOON)\n mercury = chart.get(const.MERCURY)\n planetsConjMoon = chart.objects.getObjectsAspecting(moon, [0])\n planetsConjMercury = chart.objects.getObjectsAspecting(mercury, [0])\n \n _set = _merge(planetsConjMoon, planetsConjMercury)\n factors.append(['Planets Conj Moon or Mercury', _set])\n \n # Asc ruler if aspected by disposer\n ascRulerID = essential.ruler(asc.sign)\n ascRuler = chart.getObject(ascRulerID)\n disposerID = essential.ruler(ascRuler.sign)\n disposer = chart.getObject(disposerID)\n \n _set = []\n if aspects.isAspecting(disposer, ascRuler, const.MAJOR_ASPECTS):\n _set = [ascRuler.id]\n factors.append(['Asc Ruler if aspected by its disposer', _set]);\n \n # Planets aspecting Moon or Mercury\n aspMoon = chart.objects.getObjectsAspecting(moon, [60,90,120,180])\n aspMercury = chart.objects.getObjectsAspecting(mercury, [60,90,120,180])\n \n _set = _merge(aspMoon, aspMercury)\n factors.append(['Planets Asp Moon or Mercury', _set])\n \n return factors"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef termLons(TERMS):\n res = []\n for i, sign in enumerate(SIGN_LIST):\n termList = TERMS[sign]\n res.extend([\n ID,\n sign,\n start + 30 * i,\n ] for (ID, start, end) in termList)\n return res", "response": "Returns a list with the absolute longitude \n of all terms."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the Almutem table.", "response": "def compute(chart):\n \"\"\" Computes the Almutem table. \"\"\"\n almutems = {}\n \n # Hylegic points\n hylegic = [\n chart.getObject(const.SUN),\n chart.getObject(const.MOON),\n chart.getAngle(const.ASC),\n chart.getObject(const.PARS_FORTUNA),\n chart.getObject(const.SYZYGY)\n ]\n for hyleg in hylegic:\n row = newRow()\n digInfo = essential.getInfo(hyleg.sign, hyleg.signlon)\n \n # Add the scores of each planet where hyleg has dignities\n for dignity in DIGNITY_LIST:\n objID = digInfo[dignity]\n if objID:\n score = essential.SCORES[dignity]\n row[objID]['string'] += '+%s' % score\n row[objID]['score'] += score\n \n almutems[hyleg.id] = row\n \n # House positions\n row = newRow()\n for objID in OBJECT_LIST:\n obj = chart.getObject(objID)\n house = chart.houses.getObjectHouse(obj)\n score = HOUSE_SCORES[house.id]\n row[objID]['string'] = '+%s' % score\n row[objID]['score'] = score\n almutems['Houses'] = row\n \n # Planetary time\n row = newRow()\n table = planetarytime.getHourTable(chart.date, chart.pos)\n ruler = table.currRuler()\n hourRuler = table.hourRuler()\n row[ruler] = {\n 'string': '+7',\n 'score': 7\n }\n row[hourRuler] = {\n 'string': '+6',\n 'score': 6\n }\n almutems['Rulers'] = row;\n \n # Compute scores\n scores = newRow()\n for _property, _list in almutems.items():\n for objID, values in _list.items():\n scores[objID]['string'] += values['string']\n scores[objID]['score'] += values['score']\n almutems['Score'] = scores\n \n return almutems"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the details for one or more resources by ID.", "response": "def get(self,\n resource_id=None,\n resource_action=None,\n resource_cls=None,\n single_resource=False):\n \"\"\" Gets the details for one or more resources by ID\n\n Args:\n cls - gophish.models.Model - The resource class\n resource_id - str - The endpoint (URL path) for the resource\n resource_action - str - An action to perform on the resource\n resource_cls - cls - A class to use for parsing, if different than\n the base resource\n single_resource - bool - An override to tell Gophish that even\n though we aren't requesting a single resource, we expect a\n single response object\n\n Returns:\n One or more instances of cls parsed from the returned JSON\n \"\"\"\n\n endpoint = self.endpoint\n\n if not resource_cls:\n resource_cls = self._cls\n\n if resource_id:\n endpoint = self._build_url(endpoint, resource_id)\n\n if resource_action:\n endpoint = self._build_url(endpoint, resource_action)\n\n response = self.api.execute(\"GET\", endpoint)\n if not response.ok:\n raise Error.parse(response.json())\n\n if resource_id or single_resource:\n return resource_cls.parse(response.json())\n\n return [resource_cls.parse(resource) for resource in response.json()]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef post(self, resource):\n response = self.api.execute(\n \"POST\", self.endpoint, json=(resource.as_dict()))\n\n if not response.ok:\n raise Error.parse(response.json())\n\n return self._cls.parse(response.json())", "response": "Creates a new instance of the resource."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nedit an existing resource instance", "response": "def put(self, resource):\n \"\"\" Edits an existing resource\n\n Args:\n resource - gophish.models.Model - The resource instance\n \"\"\"\n\n endpoint = self.endpoint\n\n if resource.id:\n endpoint = self._build_url(endpoint, resource.id)\n\n response = self.api.execute(\"PUT\", endpoint, json=resource.as_dict())\n\n if not response.ok:\n raise Error.parse(response.json())\n\n return self._cls.parse(response.json())"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef delete(self, resource_id):\n\n endpoint = '{}/{}'.format(self.endpoint, resource_id)\n\n response = self.api.execute(\"DELETE\", endpoint)\n\n if not response.ok:\n raise Error.parse(response.json())\n\n return self._cls.parse(response.json())", "response": "Deletes an existing resource in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef as_dict(self):\n result = {}\n for key in self._valid_properties:\n val = getattr(self, key)\n if isinstance(val, datetime):\n val = val.isoformat()\n # Parse custom classes\n elif val and not Model._is_builtin(val):\n val = val.as_dict()\n # Parse lists of objects\n elif isinstance(val, list):\n # We only want to call as_dict in the case where the item\n # isn't a builtin type.\n for i in range(len(val)):\n if Model._is_builtin(val[i]):\n continue\n val[i] = val[i].as_dict()\n # If it's a boolean, add it regardless of the value\n elif isinstance(val, bool):\n result[key] = val\n\n # Add it if it's not None\n if val:\n result[key] = val\n return result", "response": "Returns a dict representation of the resource."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef execute(self, method, path, **kwargs):\n\n url = \"{}{}\".format(self.host, path)\n kwargs.update(self._client_kwargs)\n response = requests.request(\n method,\n url,\n headers={\"Authorization\": \"Bearer {}\".format(self.api_key)},\n **kwargs)\n return response", "response": "Executes a request to a given endpoint returning the response"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef complete(self, campaign_id):\n\n return super(API, self).get(\n resource_id=campaign_id, resource_action='complete')", "response": "Complete an existing campaign"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef summary(self, campaign_id=None):\n resource_cls = CampaignSummary\n single_resource = False\n\n if not campaign_id:\n resource_cls = CampaignSummaries\n single_resource = True\n\n return super(API, self).get(\n resource_id=campaign_id,\n resource_action='summary',\n resource_cls=resource_cls,\n single_resource=single_resource)", "response": "Returns the summary of the current user s infrastructure."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef results(self, campaign_id):\n return super(API, self).get(\n resource_id=campaign_id,\n resource_action='results',\n resource_cls=CampaignResults)", "response": "Returns just the results for a given campaign"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_path(self, file_path):\n if not file_path:\n self.read_data = self.memory_read\n self.write_data = self.memory_write\n elif not is_valid(file_path):\n self.write_data(file_path, {})\n\n self.path = file_path", "response": "Set the path of the database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete(self, key):\n obj = self._get_content()\n obj.pop(key, None)\n\n self.write_data(self.path, obj)", "response": "Removes the specified key from the database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef data(self, **kwargs):\n\n key = kwargs.pop('key', None)\n value = kwargs.pop('value', None)\n dictionary = kwargs.pop('dictionary', None)\n\n # Fail if a key and a dictionary or a value and a dictionary are given\n if (key is not None and dictionary is not None) or \\\n (value is not None and dictionary is not None):\n raise ValueError\n\n # If only a key was provided return the corresponding value\n if key is not None and value is None:\n return self._get_content(key)\n\n # if a key and a value are passed in\n if key is not None and value is not None:\n self._set_content(key, value)\n\n if dictionary is not None:\n for key in dictionary.keys():\n value = dictionary[key]\n self._set_content(key, value)\n\n return self._get_content()", "response": "Get the content of the entry in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef filter(self, filter_arguments):\n results = self._get_content()\n\n # Filter based on a dictionary of search parameters\n if isinstance(filter_arguments, dict):\n for item, content in iteritems(self._get_content()):\n for key, value in iteritems(filter_arguments):\n keys = key.split('.')\n value = filter_arguments[key]\n\n if not self._contains_value({item: content}, keys, value):\n del results[item]\n\n # Filter based on an input string that should match database key\n if isinstance(filter_arguments, str):\n if filter_arguments in results:\n return [{filter_arguments: results[filter_arguments]}]\n else:\n return []\n\n return results", "response": "Returns a list of objects based on a dictionary of filter parameters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndropping the database by deleting the JSON file.", "response": "def drop(self):\n \"\"\"\n Remove the database by deleting the JSON file.\n \"\"\"\n import os\n\n if self.path:\n if os.path.exists(self.path):\n os.remove(self.path)\n else:\n # Clear the in-memory data if there is no file path\n self._data = {}"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read_data(file_path):\n\n if not is_valid(file_path):\n write_data(file_path, {})\n\n db = open_file_for_reading(file_path)\n content = db.read()\n\n obj = decode(content)\n\n db.close()\n\n return obj", "response": "Reads a file and returns a json encoded representation of the file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef write_data(path, obj):\n with open_file_for_writing(path) as db:\n db.write(encode(obj))\n\n return obj", "response": "Writes a dictionary to a file and returns the updated file content."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking to see if a file exists or is empty.", "response": "def is_valid(file_path):\n \"\"\"\n Check to see if a file exists or is empty.\n \"\"\"\n from os import path, stat\n\n can_open = False\n\n try:\n with open(file_path) as fp:\n can_open = True\n except IOError:\n return False\n\n is_file = path.isfile(file_path)\n\n return path.exists(file_path) and is_file and stat(file_path).st_size > 0"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _refine(node_coords, cells_nodes, edge_nodes, cells_edges):\n num_nodes = len(node_coords)\n num_new_nodes = len(edge_nodes)\n\n # new_nodes = numpy.empty(num_new_nodes, dtype=numpy.dtype((float, 2)))\n node_coords.resize(num_nodes + num_new_nodes, 3, refcheck=False)\n # Set starting index for new nodes.\n new_node_gid = num_nodes\n\n # After the refinement step, all previous edge-node associations will be\n # obsolete, so record *all* the new edges.\n num_edges = len(edge_nodes)\n num_cells = len(cells_nodes)\n assert num_cells == len(cells_edges)\n num_new_edges = 2 * num_edges + 3 * num_cells\n new_edges_nodes = numpy.empty(num_new_edges, dtype=numpy.dtype((int, 2)))\n\n new_edge_gid = 0\n\n # After the refinement step, all previous cell-node associations will be\n # obsolete, so record *all* the new cells.\n num_new_cells = 4 * num_cells\n new_cells_nodes = numpy.empty(num_new_cells, dtype=numpy.dtype((int, 3)))\n new_cells_edges = numpy.empty(num_new_cells, dtype=numpy.dtype((int, 3)))\n new_cell_gid = 0\n\n is_edge_divided = numpy.zeros(num_edges, dtype=bool)\n edge_midpoint_gids = numpy.empty(num_edges, dtype=int)\n edge_newedges_gids = numpy.empty(num_edges, dtype=numpy.dtype((int, 2)))\n\n # Loop over all elements.\n for cell_id, cell in enumerate(zip(cells_edges, cells_nodes)):\n cell_edges, cell_nodes = cell\n # Divide edges.\n local_edge_midpoint_gids = numpy.empty(3, dtype=int)\n local_edge_newedges = numpy.empty(3, dtype=numpy.dtype((int, 2)))\n local_neighbor_midpoints = [[], [], []]\n local_neighbor_newedges = [[], [], []]\n for k, edge_gid in enumerate(cell_edges):\n edgenodes_gids = edge_nodes[edge_gid]\n if is_edge_divided[edge_gid]:\n # Edge is already divided. Just keep records for the cell\n # creation.\n local_edge_midpoint_gids[k] = edge_midpoint_gids[edge_gid]\n local_edge_newedges[k] = edge_newedges_gids[edge_gid]\n else:\n # Create new node at the edge midpoint.\n node_coords[new_node_gid] = 0.5 * (\n node_coords[edgenodes_gids[0]] + node_coords[edgenodes_gids[1]]\n )\n local_edge_midpoint_gids[k] = new_node_gid\n new_node_gid += 1\n edge_midpoint_gids[edge_gid] = local_edge_midpoint_gids[k]\n\n # Divide edge into two.\n new_edges_nodes[new_edge_gid] = numpy.array(\n [edgenodes_gids[0], local_edge_midpoint_gids[k]]\n )\n new_edge_gid += 1\n new_edges_nodes[new_edge_gid] = numpy.array(\n [local_edge_midpoint_gids[k], edgenodes_gids[1]]\n )\n new_edge_gid += 1\n\n local_edge_newedges[k] = [new_edge_gid - 2, new_edge_gid - 1]\n edge_newedges_gids[edge_gid] = local_edge_newedges[k]\n # Do the household.\n is_edge_divided[edge_gid] = True\n # Keep a record of the new neighbors of the old nodes.\n # Get local node IDs.\n edgenodes_lids = [\n numpy.nonzero(cell_nodes == edgenodes_gids[0])[0][0],\n numpy.nonzero(cell_nodes == edgenodes_gids[1])[0][0],\n ]\n local_neighbor_midpoints[edgenodes_lids[0]].append(\n local_edge_midpoint_gids[k]\n )\n local_neighbor_midpoints[edgenodes_lids[1]].append(\n local_edge_midpoint_gids[k]\n )\n local_neighbor_newedges[edgenodes_lids[0]].append(local_edge_newedges[k][0])\n local_neighbor_newedges[edgenodes_lids[1]].append(local_edge_newedges[k][1])\n\n new_edge_opposite_of_local_node = numpy.empty(3, dtype=int)\n # New edges: Connect the three midpoints.\n for k in range(3):\n new_edges_nodes[new_edge_gid] = local_neighbor_midpoints[k]\n new_edge_opposite_of_local_node[k] = new_edge_gid\n new_edge_gid += 1\n\n # Create new elements.\n # Center cell:\n new_cells_nodes[new_cell_gid] = local_edge_midpoint_gids\n new_cells_edges[new_cell_gid] = new_edge_opposite_of_local_node\n new_cell_gid += 1\n # The three corner elements:\n for k in range(3):\n new_cells_nodes[new_cell_gid] = numpy.array(\n [\n cells_nodes[cell_id][k],\n local_neighbor_midpoints[k][0],\n local_neighbor_midpoints[k][1],\n ]\n )\n new_cells_edges[new_cell_gid] = numpy.array(\n [\n new_edge_opposite_of_local_node[k],\n local_neighbor_newedges[k][0],\n local_neighbor_newedges[k][1],\n ]\n )\n new_cell_gid += 1\n\n return node_coords, new_cells_nodes, new_edges_nodes, new_cells_edges", "response": "Refines a mesh by inserting nodes at all edge midpoints\n and make four triangular elements where there was one."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_edges(cells_nodes):\n # Create the idx_hierarchy (nodes->edges->cells), i.e., the value of\n # `self.idx_hierarchy[0, 2, 27]` is the index of the node of cell 27, edge\n # 2, node 0. The shape of `self.idx_hierarchy` is `(2, 3, n)`, where `n` is\n # the number of cells. Make sure that the k-th edge is opposite of the k-th\n # point in the triangle.\n local_idx = numpy.array([[1, 2], [2, 0], [0, 1]]).T\n # Map idx back to the nodes. This is useful if quantities which are in\n # idx shape need to be added up into nodes (e.g., equation system rhs).\n nds = cells_nodes.T\n idx_hierarchy = nds[local_idx]\n\n s = idx_hierarchy.shape\n a = numpy.sort(idx_hierarchy.reshape(s[0], s[1] * s[2]).T)\n\n b = numpy.ascontiguousarray(a).view(\n numpy.dtype((numpy.void, a.dtype.itemsize * a.shape[1]))\n )\n _, idx, inv, cts = numpy.unique(\n b, return_index=True, return_inverse=True, return_counts=True\n )\n\n # No edge has more than 2 cells. This assertion fails, for example, if\n # cells are listed twice.\n assert all(cts < 3)\n\n edge_nodes = a[idx]\n cells_edges = inv.reshape(3, -1).T\n\n return edge_nodes, cells_edges", "response": "Create edge - node and edge - cell relations. Adapted from voropy.\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plot2d(points, cells, mesh_color=\"k\", show_axes=False):\n import matplotlib.pyplot as plt\n from matplotlib.collections import LineCollection\n\n fig = plt.figure()\n ax = fig.gca()\n plt.axis(\"equal\")\n if not show_axes:\n ax.set_axis_off()\n\n xmin = numpy.amin(points[:, 0])\n xmax = numpy.amax(points[:, 0])\n ymin = numpy.amin(points[:, 1])\n ymax = numpy.amax(points[:, 1])\n\n width = xmax - xmin\n xmin -= 0.1 * width\n xmax += 0.1 * width\n\n height = ymax - ymin\n ymin -= 0.1 * height\n ymax += 0.1 * height\n\n ax.set_xlim(xmin, xmax)\n ax.set_ylim(ymin, ymax)\n\n edge_nodes, _ = create_edges(cells)\n\n # Get edges, cut off z-component.\n e = points[edge_nodes][:, :, :2]\n line_segments = LineCollection(e, color=mesh_color)\n ax.add_collection(line_segments)\n return fig", "response": "Plot a 2D mesh using matplotlib."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nperforms a job by a member in the pool and return the result.", "response": "def put(self, job, result):\n \"Perform a job by a member in the pool and return the result.\"\n self.job.put(job)\n r = result.get()\n return r"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nperform a contract on a number of jobs and block until a result is retrieved for each job.", "response": "def contract(self, jobs, result):\n \"\"\"\n Perform a contract on a number of jobs and block until a result is\n retrieved for each job.\n \"\"\"\n for j in jobs:\n WorkerPool.put(self, j)\n\n r = []\n for i in xrange(len(jobs)):\n r.append(result.get())\n\n return r"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cube(\n xmin=0.0, xmax=1.0, ymin=0.0, ymax=1.0, zmin=0.0, zmax=1.0, nx=11, ny=11, nz=11\n):\n \"\"\"Canonical tetrahedrization of the cube.\n Input:\n Edge lenghts of the cube\n Number of nodes along the edges.\n \"\"\"\n # Generate suitable ranges for parametrization\n x_range = numpy.linspace(xmin, xmax, nx)\n y_range = numpy.linspace(ymin, ymax, ny)\n z_range = numpy.linspace(zmin, zmax, nz)\n\n # Create the vertices.\n x, y, z = numpy.meshgrid(x_range, y_range, z_range, indexing=\"ij\")\n # Alternative with slightly different order:\n # ```\n # nodes = numpy.stack([x, y, z]).T.reshape(-1, 3)\n # ```\n nodes = numpy.array([x, y, z]).T.reshape(-1, 3)\n\n # Create the elements (cells).\n # There is 1 way to split a cube into 5 tetrahedra,\n # and 12 ways to split it into 6 tetrahedra.\n # See\n # \n # Also interesting: .\n\n a0 = numpy.add.outer(numpy.array(range(nx - 1)), nx * numpy.array(range(ny - 1)))\n a = numpy.add.outer(a0, nx * ny * numpy.array(range(nz - 1)))\n\n # The general scheme here is:\n # * Initialize everything with `a`, equivalent to\n # [i + nx * j + nx*ny * k].\n # * Add the \"even\" elements.\n # * Switch the element styles for every other element to make sure the\n # edges match at the faces of the cubes.\n # The last step requires adapting the original pattern at\n # [1::2, 0::2, 0::2, :]\n # [0::2, 1::2, 0::2, :]\n # [0::2, 0::2, 1::2, :]\n # [1::2, 1::2, 1::2, :]\n #\n\n # Tetrahedron 0:\n # [\n # i + nx*j + nx*ny * k,\n # i + nx*(j+1) + nx*ny * k,\n # i+1 + nx*j + nx*ny * k,\n # i + nx*j + nx*ny * (k+1)\n # ]\n # TODO get\n # ```\n # elems0 = numpy.stack([a, a + nx, a + 1, a + nx*ny]).T\n # ```\n # back.\n elems0 = numpy.concatenate(\n [a[..., None], a[..., None] + nx, a[..., None] + 1, a[..., None] + nx * ny],\n axis=3,\n )\n\n # Every other element cube:\n # [\n # i+1 + nx * j + nx*ny * k,\n # i+1 + nx * (j+1) + nx*ny * k,\n # i + nx * j + nx*ny * k,\n # i+1 + nx * j + nx*ny * (k+1)\n # ]\n elems0[1::2, 0::2, 0::2, 0] += 1\n elems0[0::2, 1::2, 0::2, 0] += 1\n elems0[0::2, 0::2, 1::2, 0] += 1\n elems0[1::2, 1::2, 1::2, 0] += 1\n\n elems0[1::2, 0::2, 0::2, 1] += 1\n elems0[0::2, 1::2, 0::2, 1] += 1\n elems0[0::2, 0::2, 1::2, 1] += 1\n elems0[1::2, 1::2, 1::2, 1] += 1\n\n elems0[1::2, 0::2, 0::2, 2] -= 1\n elems0[0::2, 1::2, 0::2, 2] -= 1\n elems0[0::2, 0::2, 1::2, 2] -= 1\n elems0[1::2, 1::2, 1::2, 2] -= 1\n\n elems0[1::2, 0::2, 0::2, 3] += 1\n elems0[0::2, 1::2, 0::2, 3] += 1\n elems0[0::2, 0::2, 1::2, 3] += 1\n elems0[1::2, 1::2, 1::2, 3] += 1\n\n # Tetrahedron 1:\n # [\n # i + nx*(j+1) + nx*ny * k,\n # i+1 + nx*(j+1) + nx*ny * k,\n # i+1 + nx*j + nx*ny * k,\n # i+1 + nx*(j+1) + nx*ny * (k+1)\n # ]\n # elems1 = numpy.stack([a + nx, a + 1 + nx, a + 1, a + 1 + nx + nx*ny]).T\n elems1 = numpy.concatenate(\n [\n a[..., None] + nx,\n a[..., None] + 1 + nx,\n a[..., None] + 1,\n a[..., None] + 1 + nx + nx * ny,\n ],\n axis=3,\n )\n\n # Every other element cube:\n # [\n # i+1 + nx * (j+1) + nx*ny * k,\n # i + nx * (j+1) + nx*ny * k,\n # i + nx * j + nx*ny * k,\n # i + nx * (j+1) + nx*ny * (k+1)\n # ]\n elems1[1::2, 0::2, 0::2, 0] += 1\n elems1[0::2, 1::2, 0::2, 0] += 1\n elems1[0::2, 0::2, 1::2, 0] += 1\n elems1[1::2, 1::2, 1::2, 0] += 1\n\n elems1[1::2, 0::2, 0::2, 1] -= 1\n elems1[0::2, 1::2, 0::2, 1] -= 1\n elems1[0::2, 0::2, 1::2, 1] -= 1\n elems1[1::2, 1::2, 1::2, 1] -= 1\n\n elems1[1::2, 0::2, 0::2, 2] -= 1\n elems1[0::2, 1::2, 0::2, 2] -= 1\n elems1[0::2, 0::2, 1::2, 2] -= 1\n elems1[1::2, 1::2, 1::2, 2] -= 1\n\n elems1[1::2, 0::2, 0::2, 3] -= 1\n elems1[0::2, 1::2, 0::2, 3] -= 1\n elems1[0::2, 0::2, 1::2, 3] -= 1\n elems1[1::2, 1::2, 1::2, 3] -= 1\n\n # Tetrahedron 2:\n # [\n # i + nx*(j+1) + nx*ny * k,\n # i+1 + nx*j + nx*ny * k,\n # i + nx*j + nx*ny * (k+1),\n # i+1 + nx*(j+1) + nx*ny * (k+1)\n # ]\n # elems2 = numpy.stack([a + nx, a + 1, a + nx*ny, a + 1 + nx + nx*ny]).T\n elems2 = numpy.concatenate(\n [\n a[..., None] + nx,\n a[..., None] + 1,\n a[..., None] + nx * ny,\n a[..., None] + 1 + nx + nx * ny,\n ],\n axis=3,\n )\n\n # Every other element cube:\n # [\n # i+1 + nx * (j+1) + nx*ny * k,\n # i + nx * j + nx*ny * k,\n # i+1 + nx * j + nx*ny * (k+1),\n # i + nx * (j+1) + nx*ny * (k+1)\n # ]\n elems2[1::2, 0::2, 0::2, 0] += 1\n elems2[0::2, 1::2, 0::2, 0] += 1\n elems2[0::2, 0::2, 1::2, 0] += 1\n elems2[1::2, 1::2, 1::2, 0] += 1\n\n elems2[1::2, 0::2, 0::2, 1] -= 1\n elems2[0::2, 1::2, 0::2, 1] -= 1\n elems2[0::2, 0::2, 1::2, 1] -= 1\n elems2[1::2, 1::2, 1::2, 1] -= 1\n\n elems2[1::2, 0::2, 0::2, 2] += 1\n elems2[0::2, 1::2, 0::2, 2] += 1\n elems2[0::2, 0::2, 1::2, 2] += 1\n elems2[1::2, 1::2, 1::2, 2] += 1\n\n elems2[1::2, 0::2, 0::2, 3] -= 1\n elems2[0::2, 1::2, 0::2, 3] -= 1\n elems2[0::2, 0::2, 1::2, 3] -= 1\n elems2[1::2, 1::2, 1::2, 3] -= 1\n\n # Tetrahedron 3:\n # [\n # i + nx * (j+1) + nx*ny * k,\n # i + nx * j + nx*ny * (k+1),\n # i + nx * (j+1) + nx*ny * (k+1),\n # i+1 + nx * (j+1) + nx*ny * (k+1)\n # ]\n # elems3 = numpy.stack([\n # a + nx,\n # a + nx*ny,\n # a + nx + nx*ny,\n # a + 1 + nx + nx*ny\n # ]).T\n elems3 = numpy.concatenate(\n [\n a[..., None] + nx,\n a[..., None] + nx * ny,\n a[..., None] + nx + nx * ny,\n a[..., None] + 1 + nx + nx * ny,\n ],\n axis=3,\n )\n\n # Every other element cube:\n # [\n # i+1 + nx * (j+1) + nx*ny * k,\n # i+1 + nx * j + nx*ny * (k+1),\n # i+1 + nx * (j+1) + nx*ny * (k+1),\n # i + nx * (j+1) + nx*ny * (k+1)\n # ]\n elems3[1::2, 0::2, 0::2, 0] += 1\n elems3[0::2, 1::2, 0::2, 0] += 1\n elems3[0::2, 0::2, 1::2, 0] += 1\n elems3[1::2, 1::2, 1::2, 0] += 1\n\n elems3[1::2, 0::2, 0::2, 1] += 1\n elems3[0::2, 1::2, 0::2, 1] += 1\n elems3[0::2, 0::2, 1::2, 1] += 1\n elems3[1::2, 1::2, 1::2, 1] += 1\n\n elems3[1::2, 0::2, 0::2, 2] += 1\n elems3[0::2, 1::2, 0::2, 2] += 1\n elems3[0::2, 0::2, 1::2, 2] += 1\n elems3[1::2, 1::2, 1::2, 2] += 1\n\n elems3[1::2, 0::2, 0::2, 3] -= 1\n elems3[0::2, 1::2, 0::2, 3] -= 1\n elems3[0::2, 0::2, 1::2, 3] -= 1\n elems3[1::2, 1::2, 1::2, 3] -= 1\n\n # Tetrahedron 4:\n # [\n # i+1 + nx * j + nx*ny * k,\n # i + nx * j + nx*ny * (k+1),\n # i+1 + nx * (j+1) + nx*ny * (k+1),\n # i+1 + nx * j + nx*ny * (k+1)\n # ]\n # elems4 = numpy.stack([\n # a + 1,\n # a + nx*ny,\n # a + 1 + nx + nx*ny,\n # a + 1 + nx*ny\n # ]).T\n elems4 = numpy.concatenate(\n [\n a[..., None] + 1,\n a[..., None] + nx * ny,\n a[..., None] + 1 + nx + nx * ny,\n a[..., None] + 1 + nx * ny,\n ],\n axis=3,\n )\n\n # Every other element cube:\n # [\n # i + nx * j + nx*ny * k,\n # i+1 + nx * j + nx*ny * (k+1),\n # i + nx * (j+1) + nx*ny * (k+1),\n # i + nx * j + nx*ny * (k+1)\n # ]\n elems4[1::2, 0::2, 0::2, 0] -= 1\n elems4[0::2, 1::2, 0::2, 0] -= 1\n elems4[0::2, 0::2, 1::2, 0] -= 1\n elems4[1::2, 1::2, 1::2, 0] -= 1\n\n elems4[1::2, 0::2, 0::2, 1] += 1\n elems4[0::2, 1::2, 0::2, 1] += 1\n elems4[0::2, 0::2, 1::2, 1] += 1\n elems4[1::2, 1::2, 1::2, 1] += 1\n\n elems4[1::2, 0::2, 0::2, 2] -= 1\n elems4[0::2, 1::2, 0::2, 2] -= 1\n elems4[0::2, 0::2, 1::2, 2] -= 1\n elems4[1::2, 1::2, 1::2, 2] -= 1\n\n elems4[1::2, 0::2, 0::2, 3] -= 1\n elems4[0::2, 1::2, 0::2, 3] -= 1\n elems4[0::2, 0::2, 1::2, 3] -= 1\n elems4[1::2, 1::2, 1::2, 3] -= 1\n\n elems = numpy.vstack(\n [\n elems0.reshape(-1, 4),\n elems1.reshape(-1, 4),\n elems2.reshape(-1, 4),\n elems3.reshape(-1, 4),\n elems4.reshape(-1, 4),\n ]\n )\n\n return nodes, elems", "response": "Canonical tetrahedrization of the cube."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd another worker to the pool.", "response": "def grow(self):\n \"Add another worker to the pool.\"\n t = self.worker_factory(self)\n t.start()\n self._size += 1"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef shrink(self):\n \"Get rid of one worker from the pool. Raises IndexError if empty.\"\n if self._size <= 0:\n raise IndexError(\"pool is already empty\")\n self._size -= 1\n self.put(SuicideJob())", "response": "Get rid of one worker from the pool. Raises IndexError if empty."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nperforming a map operation distributed among the workers. Will \" \"block until done.", "response": "def map(self, fn, *seq):\n \"Perform a map operation distributed among the workers. Will \"\n \"block until done.\"\n results = Queue()\n args = zip(*seq)\n for seq in args:\n j = SimpleJob(results, fn, seq)\n self.put(j)\n\n # Aggregate results\n r = []\n for i in range(len(list(args))):\n r.append(results.get())\n\n return r"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a M\u00f6bius strip as a simplistic triangular mesh.", "response": "def moebius(\n num_twists=1, # How many twists are there in the 'paper'?\n nl=60, # Number of nodes along the length of the strip\n nw=11, # Number of nodes along the width of the strip (>= 2)\n mode=\"classical\",\n):\n \"\"\"Creates a simplistic triangular mesh on a slightly M\u00f6bius strip. The\n M\u00f6bius strip here deviates slightly from the ordinary geometry in that it\n is constructed in such a way that the two halves can be exchanged as to\n allow better comparison with the pseudo-M\u00f6bius geometry.\n\n The mode is either `'classical'` or `'smooth'`. The first is the classical\n M\u00f6bius band parametrization, the latter a smoothed variant matching\n `'pseudo'`.\n \"\"\"\n # The width of the strip\n width = 1.0\n scale = 10.0\n\n # radius of the strip when flattened out\n r = 1.0\n\n # seam displacement\n alpha0 = 0.0 # pi / 2\n\n # How flat the strip will be.\n # Positive values result in left-turning M\u00f6bius strips, negative in\n # right-turning ones.\n # Also influences the width of the strip.\n flatness = 1.0\n\n # Generate suitable ranges for parametrization\n u_range = numpy.linspace(0.0, 2 * numpy.pi, num=nl, endpoint=False)\n v_range = numpy.linspace(-0.5 * width, 0.5 * width, num=nw)\n\n # Create the vertices. This is based on the parameterization\n # of the M\u00f6bius strip as given in\n # \n sin_u = numpy.sin(u_range)\n cos_u = numpy.cos(u_range)\n alpha = num_twists * 0.5 * u_range + alpha0\n sin_alpha = numpy.sin(alpha)\n cos_alpha = numpy.cos(alpha)\n\n if mode == \"classical\":\n a = cos_alpha\n b = sin_alpha\n reverse_seam = num_twists % 2 == 1\n elif mode == \"smooth\":\n # The fundamental difference with the ordinary M\u00f6bius band here are the\n # squares.\n # It is also possible to to abs() the respective sines and cosines, but\n # this results in a non-smooth manifold.\n a = numpy.copysign(cos_alpha ** 2, cos_alpha)\n b = numpy.copysign(sin_alpha ** 2, sin_alpha)\n reverse_seam = num_twists % 2 == 1\n else:\n assert mode == \"pseudo\"\n a = cos_alpha ** 2\n b = sin_alpha ** 2\n reverse_seam = False\n\n nodes = (\n scale\n * numpy.array(\n [\n numpy.outer(a * cos_u, v_range) + r * cos_u[:, numpy.newaxis],\n numpy.outer(a * sin_u, v_range) + r * sin_u[:, numpy.newaxis],\n numpy.outer(b, v_range) * flatness,\n ]\n )\n .reshape(3, -1)\n .T\n )\n\n elems = _create_elements(nl, nw, reverse_seam)\n return nodes, elems"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run(self):\n \"Get jobs from the queue and perform them as they arrive.\"\n while 1:\n # Sleep until there is a job to perform.\n job = self.jobs.get()\n\n # Yawn. Time to get some work done.\n try:\n job.run()\n self.jobs.task_done()\n except TerminationNotice:\n self.jobs.task_done()\n break", "response": "Get jobs from the queue and perform them as they arrive."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndisplaying a single object in a web browser pointing to geojson. io with the specified content.", "response": "def display(contents, domain=DEFAULT_DOMAIN, force_gist=False):\n \"\"\"\n Open a web browser pointing to geojson.io with the specified content.\n\n If the content is large, an anonymous gist will be created on github and\n the URL will instruct geojson.io to download the gist data and then\n display. If the content is small, this step is not needed as the data can\n be included in the URL\n\n Parameters\n ----------\n content - (see make_geojson)\n domain - string, default http://geojson.io\n force_gist - bool, default False\n Create an anonymous gist on Github regardless of the size of the\n contents\n\n \"\"\"\n url = make_url(contents, domain, force_gist)\n webbrowser.open(url)\n return url"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nembed geojson. io in an iframe in Jupyter notebook.", "response": "def embed(contents='', width='100%', height=512, *args, **kwargs):\n \"\"\"\n Embed geojson.io in an iframe in Jupyter/IPython notebook.\n\n Parameters\n ----------\n contents - see make_url()\n width - string, default '100%' - width of the iframe\n height - string / int, default 512 - height of the iframe\n kwargs - additional arguments are passed to `make_url()`\n\n \"\"\"\n from IPython.display import HTML\n\n url = make_url(contents, *args, **kwargs)\n html = ''.format(\n url=url, width=width, height=height)\n return HTML(html)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_url(contents, domain=DEFAULT_DOMAIN, force_gist=False,\n size_for_gist=MAX_URL_LEN):\n \"\"\"\n Returns the URL to open given the domain and contents.\n\n If the file contents are large, an anonymous gist will be created.\n\n Parameters\n ----------\n contents\n * string - assumed to be GeoJSON\n * an object that implements __geo_interface__\n A FeatureCollection will be constructed with one feature,\n the object.\n * a sequence of objects that each implement __geo_interface__\n A FeatureCollection will be constructed with the objects\n as the features\n domain - string, default http://geojson.io\n force_gist - force gist creation regardless of file size.\n\n For more information about __geo_interface__ see:\n https://gist.github.com/sgillies/2217756\n\n If the contents are large, then a gist will be created.\n\n \"\"\"\n contents = make_geojson(contents)\n if len(contents) <= size_for_gist and not force_gist:\n url = data_url(contents, domain)\n else:\n gist = _make_gist(contents)\n url = gist_url(gist.id, domain)\n\n return url", "response": "Returns the URL to open given the domain and contents."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a GeoJSON string from a variety of inputs.", "response": "def make_geojson(contents):\n \"\"\"\n Return a GeoJSON string from a variety of inputs.\n See the documentation for make_url for the possible contents\n input.\n\n Returns\n -------\n GeoJSON string\n\n \"\"\"\n if isinstance(contents, six.string_types):\n return contents\n\n if hasattr(contents, '__geo_interface__'):\n features = [_geo_to_feature(contents)]\n else:\n try:\n feature_iter = iter(contents)\n except TypeError:\n raise ValueError('Unknown type for input')\n\n features = []\n for i, f in enumerate(feature_iter):\n if not hasattr(f, '__geo_interface__'):\n raise ValueError('Unknown type at index {0}'.format(i))\n features.append(_geo_to_feature(f))\n\n data = {'type': 'FeatureCollection', 'features': features}\n return json.dumps(data)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the URL for embedding the GeoJSON data in the URL hash", "response": "def data_url(contents, domain=DEFAULT_DOMAIN):\n \"\"\"\n Return the URL for embedding the GeoJSON data in the URL hash\n\n Parameters\n ----------\n contents - string of GeoJSON\n domain - string, default http://geojson.io\n\n \"\"\"\n url = (domain + '#data=data:application/json,' +\n urllib.parse.quote(contents))\n return url"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _make_gist(contents, description='', filename='data.geojson'):\n ghapi = github3.GitHub()\n files = {filename: {'content': contents}}\n gist = ghapi.create_gist(description, files)\n\n return gist", "response": "Create and return an anonymous gist with a single file and specified contents"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef clenshaw(a, alpha, beta, t):\n n = len(alpha)\n assert len(beta) == n\n assert len(a) == n + 1\n\n try:\n b = numpy.empty((n + 1,) + t.shape)\n except AttributeError: # 'float' object has no attribute 'shape'\n b = numpy.empty(n + 1)\n\n # b[0] is unused, can be any value\n # TODO shift the array\n b[0] = 1.0\n\n b[n] = a[n]\n b[n - 1] = a[n - 1] + (t - alpha[n - 1]) * b[n]\n for k in range(n - 2, 0, -1):\n b[k] = a[k] + (t - alpha[k]) * b[k + 1] - beta[k + 1] * b[k + 2]\n\n phi0 = 1.0\n phi1 = t - alpha[0]\n\n return phi0 * a[0] + phi1 * b[1] - beta[1] * phi0 * b[2]", "response": "Clenshaw s algorithm for evaluating the kth orthogonal polynomial defined by the\n recurrence coefficients alpha and beta."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tree(X, n, alpha=0, symbolic=False):\n args = recurrence_coefficients(n, alpha=alpha, symbolic=symbolic)\n return line_tree(X, *args)", "response": "Recurrence coefficients for generalized Laguerre polynomials. Set\n alpha=0 (default) to get classical Laguerre."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the recurrence coefficients for generalized Laguerre polynomials.", "response": "def recurrence_coefficients(n, alpha, standardization=\"normal\", symbolic=False):\n \"\"\"Recurrence coefficients for generalized Laguerre polynomials.\n\n vals_k = vals_{k-1} * (t*a_k - b_k) - vals{k-2} * c_k\n \"\"\"\n S = sympy.S if symbolic else lambda x: x\n sqrt = sympy.sqrt if symbolic else numpy.sqrt\n gamma = sympy.gamma if symbolic else scipy.special.gamma\n\n if standardization == \"monic\":\n p0 = 1\n a = n * [1]\n b = [2 * k + 1 + alpha for k in range(n)]\n c = [k * (k + alpha) for k in range(n)]\n c[0] = gamma(alpha + 1)\n elif standardization == \"classical\":\n p0 = 1\n a = [-S(1) / (k + 1) for k in range(n)]\n b = [-S(2 * k + 1 + alpha) / (k + 1) for k in range(n)]\n c = [S(k + alpha) / (k + 1) for k in range(n)]\n c[0] = numpy.nan\n else:\n assert (\n standardization == \"normal\"\n ), \"Unknown Laguerre standardization '{}'.\".format(\n standardization\n )\n p0 = 1 / sqrt(gamma(alpha + 1))\n a = [-1 / sqrt((k + 1) * (k + 1 + alpha)) for k in range(n)]\n b = [-(2 * k + 1 + alpha) / sqrt((k + 1) * (k + 1 + alpha)) for k in range(n)]\n c = [sqrt(k * S(k + alpha) / ((k + 1) * (k + 1 + alpha))) for k in range(n)]\n c[0] = numpy.nan\n\n return p0, numpy.array(a), numpy.array(b), numpy.array(c)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef jacobi(n, alpha, beta, standardization, symbolic=False):\n gamma = sympy.gamma if symbolic else lambda x: scipy.special.gamma(float(x))\n\n def rational(x, y):\n # \n return (\n sympy.Rational(x, y)\n if all([isinstance(val, int) for val in [x, y]])\n else x / y\n )\n\n frac = rational if symbolic else lambda x, y: x / y\n sqrt = sympy.sqrt if symbolic else numpy.sqrt\n\n int_1 = (\n 2 ** (alpha + beta + 1)\n * gamma(alpha + 1)\n * gamma(beta + 1)\n / gamma(alpha + beta + 2)\n )\n\n if standardization == \"monic\":\n p0 = 1\n\n a = numpy.ones(n, dtype=int)\n\n # work around bug \n if isinstance(alpha, numpy.int64):\n alpha = int(alpha)\n if isinstance(beta, numpy.int64):\n beta = int(beta)\n\n b = [\n frac(beta - alpha, alpha + beta + 2)\n if N == 0\n else frac(\n beta ** 2 - alpha ** 2,\n (2 * N + alpha + beta) * (2 * N + alpha + beta + 2),\n )\n for N in range(n)\n ]\n\n # c[0] is not used in the actual recurrence, but is often defined\n # as the integral of the weight function of the domain, i.e.,\n # ```\n # int_{-1}^{+1} (1-x)^a * (1+x)^b dx =\n # 2^(a+b+1) * Gamma(a+1) * Gamma(b+1) / Gamma(a+b+2).\n # ```\n # Note also that we have the treat the case N==1 separately to avoid\n # division by 0 for alpha=beta=-1/2.\n c = [\n int_1\n if N == 0\n else frac(\n 4 * (1 + alpha) * (1 + beta),\n (2 + alpha + beta) ** 2 * (3 + alpha + beta),\n )\n if N == 1\n else frac(\n 4 * (N + alpha) * (N + beta) * N * (N + alpha + beta),\n (2 * N + alpha + beta) ** 2\n * (2 * N + alpha + beta + 1)\n * (2 * N + alpha + beta - 1),\n )\n for N in range(n)\n ]\n\n elif standardization == \"p(1)=(n+alpha over n)\" or (\n alpha == 0 and standardization == \"p(1)=1\"\n ):\n p0 = 1\n\n # work around bug \n if isinstance(alpha, numpy.int64):\n alpha = int(alpha)\n if isinstance(beta, numpy.int64):\n beta = int(beta)\n\n # Treat N==0 separately to avoid division by 0 for alpha=beta=-1/2.\n a = [\n frac(alpha + beta + 2, 2)\n if N == 0\n else frac(\n (2 * N + alpha + beta + 1) * (2 * N + alpha + beta + 2),\n 2 * (N + 1) * (N + alpha + beta + 1),\n )\n for N in range(n)\n ]\n\n b = [\n frac(beta - alpha, 2)\n if N == 0\n else frac(\n (beta ** 2 - alpha ** 2) * (2 * N + alpha + beta + 1),\n 2 * (N + 1) * (N + alpha + beta + 1) * (2 * N + alpha + beta),\n )\n for N in range(n)\n ]\n\n c = [\n int_1\n if N == 0\n else frac(\n (N + alpha) * (N + beta) * (2 * N + alpha + beta + 2),\n (N + 1) * (N + alpha + beta + 1) * (2 * N + alpha + beta),\n )\n for N in range(n)\n ]\n\n else:\n assert standardization == \"normal\", \"Unknown standardization '{}'.\".format(\n standardization\n )\n\n p0 = sqrt(1 / int_1)\n\n # Treat N==0 separately to avoid division by 0 for alpha=beta=-1/2.\n a = [\n frac(alpha + beta + 2, 2)\n * sqrt(frac(alpha + beta + 3, (alpha + 1) * (beta + 1)))\n if N == 0\n else frac(2 * N + alpha + beta + 2, 2)\n * sqrt(\n frac(\n (2 * N + alpha + beta + 1) * (2 * N + alpha + beta + 3),\n (N + 1) * (N + alpha + 1) * (N + beta + 1) * (N + alpha + beta + 1),\n )\n )\n for N in range(n)\n ]\n\n b = [\n (\n frac(beta - alpha, 2)\n if N == 0\n else frac(beta ** 2 - alpha ** 2, 2 * (2 * N + alpha + beta))\n )\n * sqrt(\n frac(\n (2 * N + alpha + beta + 3) * (2 * N + alpha + beta + 1),\n (N + 1) * (N + alpha + 1) * (N + beta + 1) * (N + alpha + beta + 1),\n )\n )\n for N in range(n)\n ]\n\n c = [\n int_1\n if N == 0\n else frac(4 + alpha + beta, 2 + alpha + beta)\n * sqrt(\n frac(\n (1 + alpha) * (1 + beta) * (5 + alpha + beta),\n 2 * (2 + alpha) * (2 + beta) * (2 + alpha + beta),\n )\n )\n if N == 1\n else frac(2 * N + alpha + beta + 2, 2 * N + alpha + beta)\n * sqrt(\n frac(\n N\n * (N + alpha)\n * (N + beta)\n * (N + alpha + beta)\n * (2 * N + alpha + beta + 3),\n (N + 1)\n * (N + alpha + 1)\n * (N + beta + 1)\n * (N + alpha + beta + 1)\n * (2 * N + alpha + beta - 1),\n )\n )\n for N in range(n)\n ]\n\n return p0, numpy.array(a), numpy.array(b), numpy.array(c)", "response": "Generate the recurrence coefficients a_k b_k c_k for the Jacobi polynomials which are orthogonal on [ 1 - 1 )."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot(f, lcar=1.0e-1):\n import matplotlib\n import matplotlib.pyplot as plt\n import pygmsh\n\n geom = pygmsh.built_in.Geometry()\n geom.add_circle([0.0, 0.0, 0.0], 1.0, lcar, num_sections=4, compound=True)\n points, cells, _, _, _ = pygmsh.generate_mesh(geom, verbose=True)\n\n x = points[:, 0]\n y = points[:, 1]\n triang = matplotlib.tri.Triangulation(x, y, cells[\"triangle\"])\n\n plt.tripcolor(triang, f(points.T), shading=\"flat\")\n plt.colorbar()\n\n # Choose a diverging colormap such that the zeros are clearly\n # distinguishable.\n plt.set_cmap(\"coolwarm\")\n # Make sure the color map limits are symmetric around 0.\n clim = plt.gci().get_clim()\n mx = max(abs(clim[0]), abs(clim[1]))\n plt.clim(-mx, mx)\n\n # circle outline\n circle = plt.Circle((0, 0), 1.0, edgecolor=\"k\", fill=False)\n plt.gca().add_artist(circle)\n\n plt.gca().set_aspect(\"equal\")\n plt.axis(\"off\")\n return", "response": "Plot function over a disk."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nevaluate the entire tree of orthogonal polynomials for the n - cube X.", "response": "def tree(X, n, symbolic=False):\n \"\"\"Evaluates the entire tree of orthogonal polynomials for the n-cube\n\n The computation is organized such that tree returns a list of arrays, L={0,\n ..., dim}, where each level corresponds to the polynomial degree L.\n Further, each level is organized like a discrete (dim-1)-dimensional\n simplex. Let's demonstrate this for 3D:\n\n L = 1:\n (0, 0, 0)\n\n L = 2:\n (1, 0, 0)\n (0, 1, 0) (0, 0, 1)\n\n L = 3:\n (2, 0, 0)\n (1, 1, 0) (1, 0, 1)\n (0, 2, 0) (0, 1, 1) (0, 0, 2)\n\n The main insight here that makes computation for n dimensions easy is that\n the next level is composed by:\n\n * Taking the whole previous level and adding +1 to the first entry.\n * Taking the last row of the previous level and adding +1 to the second\n entry.\n * Taking the last entry of the last row of the previous and adding +1 to\n the third entry.\n\n In the same manner this can be repeated for `dim` dimensions.\n \"\"\"\n p0, a, b, c = legendre(n + 1, \"normal\", symbolic=symbolic)\n\n dim = X.shape[0]\n\n p0n = p0 ** dim\n out = []\n\n level = numpy.array([numpy.ones(X.shape[1:], dtype=int) * p0n])\n out.append(level)\n\n # TODO use a simpler binom implementation\n for L in range(n):\n level = []\n for i in range(dim - 1):\n m1 = int(scipy.special.binom(L + dim - i - 1, dim - i - 1))\n if L > 0:\n m2 = int(scipy.special.binom(L + dim - i - 2, dim - i - 1))\n r = 0\n for k in range(L + 1):\n m = int(scipy.special.binom(k + dim - i - 2, dim - i - 2))\n val = out[L][-m1:][r : r + m] * (a[L - k] * X[i] - b[L - k])\n if L - k > 0:\n val -= out[L - 1][-m2:][r : r + m] * c[L - k]\n r += m\n level.append(val)\n\n # treat the last one separately\n val = out[L][-1] * (a[L] * X[-1] - b[L])\n if L > 0:\n val -= out[L - 1][-1] * c[L]\n level.append([val])\n\n out.append(numpy.concatenate(level))\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef line_evaluate(t, p0, a, b, c):\n vals1 = numpy.zeros_like(t, dtype=int)\n # The order is important here; see\n # .\n vals2 = numpy.ones_like(t) * p0\n\n for a_k, b_k, c_k in zip(a, b, c):\n vals0, vals1 = vals1, vals2\n vals2 = vals1 * (t * a_k - b_k) - vals0 * c_k\n return vals2", "response": "Evaluate the orthogonal polynomial defined by its recurrence coefficients\n a b and c at the point s."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef plot(corners, f, n=100):\n import matplotlib.tri\n import matplotlib.pyplot as plt\n\n # discretization points\n def partition(boxes, balls):\n # \n def rec(boxes, balls, parent=tuple()):\n if boxes > 1:\n for i in range(balls + 1):\n for x in rec(boxes - 1, i, parent + (balls - i,)):\n yield x\n else:\n yield parent + (balls,)\n\n return list(rec(boxes, balls))\n\n bary = numpy.array(partition(3, n)).T / n\n X = numpy.sum([numpy.outer(bary[k], corners[:, k]) for k in range(3)], axis=0).T\n\n # plot the points\n # plt.plot(X[0], X[1], 'xk')\n\n x = numpy.array(X[0])\n y = numpy.array(X[1])\n z = numpy.array(f(bary), dtype=float)\n\n triang = matplotlib.tri.Triangulation(x, y)\n plt.tripcolor(triang, z, shading=\"flat\")\n plt.colorbar()\n\n # Choose a diverging colormap such that the zeros are clearly\n # distinguishable.\n plt.set_cmap(\"coolwarm\")\n # Make sure the color map limits are symmetric around 0.\n clim = plt.gci().get_clim()\n mx = max(abs(clim[0]), abs(clim[1]))\n plt.clim(-mx, mx)\n\n # triangle outlines\n X = numpy.column_stack([corners, corners[:, 0]])\n plt.plot(X[0], X[1], \"-k\")\n\n plt.gca().set_aspect(\"equal\")\n plt.axis(\"off\")\n return", "response": "Plot function over a triangle."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _iget(key, lookup_dict):\n for k, v in lookup_dict.items():\n if k.lower() == key.lower():\n return v\n return None", "response": "Case - insensitive search for key within keys of lookup_dict."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getlang_by_name(name):\n direct_match = _iget(name, _LANGUAGE_NAME_LOOKUP)\n if direct_match:\n return direct_match\n else:\n simple_name = name.split(',')[0] # take part before comma\n simple_name = simple_name.split('(')[0].strip() # and before any bracket\n return _LANGUAGE_NAME_LOOKUP.get(simple_name, None)", "response": "Try to lookup a Language object by name e. g. English in internal language list."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getlang_by_native_name(native_name):\n direct_match = _iget(native_name, _LANGUAGE_NATIVE_NAME_LOOKUP)\n if direct_match:\n return direct_match\n else:\n simple_native_name = native_name.split(',')[0] # take part before comma\n simple_native_name = simple_native_name.split('(')[0].strip() # and before any bracket\n return _LANGUAGE_NATIVE_NAME_LOOKUP.get(simple_native_name, None)", "response": "Try to lookup a Language object by native name e. g. English in internal language list."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getlang_by_alpha2(code):\n # Handle special cases for language codes returned by YouTube API\n if code == 'iw': # handle old Hebrew code 'iw' and return modern code 'he'\n return getlang('he')\n elif 'zh-Hans' in code:\n return getlang('zh-CN') # use code `zh-CN` for all simplified Chinese\n elif 'zh-Hant' in code or re.match('zh(.*)?-HK', code):\n return getlang('zh-TW') # use code `zh-TW` for all traditional Chinese\n\n # extract prefix only if specified with subcode: e.g. zh-Hans --> zh\n first_part = code.split('-')[0]\n\n # See if pycountry can find this language\n try:\n pyc_lang = pycountry.languages.get(alpha_2=first_part)\n if pyc_lang:\n if hasattr(pyc_lang, 'inverted_name'):\n lang_name = pyc_lang.inverted_name\n else:\n lang_name = pyc_lang.name\n return getlang_by_name(lang_name)\n else:\n return None\n except KeyError:\n return None", "response": "Returns a Language object for the language code code."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write(filename, f):\n import meshio\n import meshzoo\n\n points, cells = meshzoo.iso_sphere(5)\n # get spherical coordinates from points\n polar = numpy.arccos(points[:, 2])\n azimuthal = numpy.arctan2(points[:, 1], points[:, 0])\n vals = f(polar, azimuthal)\n meshio.write(filename, points, {\"triangle\": cells}, point_data={\"f\": vals})\n return", "response": "Write a function f defined in terms of spherical coordinates to a file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nevaluates all spherical harmonics of degree at most n at angles polar and azimuthal.", "response": "def tree_sph(polar, azimuthal, n, standardization, symbolic=False):\n \"\"\"Evaluate all spherical harmonics of degree at most `n` at angles `polar`,\n `azimuthal`.\n \"\"\"\n cos = numpy.vectorize(sympy.cos) if symbolic else numpy.cos\n\n # Conventions from\n # .\n config = {\n \"acoustic\": (\"complex spherical\", False),\n \"quantum mechanic\": (\"complex spherical\", True),\n \"geodetic\": (\"complex spherical 1\", False),\n \"schmidt\": (\"schmidt\", False),\n }\n\n standard, cs_phase = config[standardization]\n\n return tree_alp(\n cos(polar),\n n,\n phi=azimuthal,\n standardization=standard,\n with_condon_shortley_phase=cs_phase,\n symbolic=symbolic,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tree_alp(\n x, n, standardization, phi=None, with_condon_shortley_phase=True, symbolic=False\n):\n \"\"\"Evaluates the entire tree of associated Legendre polynomials up to depth\n n.\n There are many recurrence relations that can be used to construct the\n associated Legendre polynomials. However, only few are numerically stable.\n Many implementations (including this one) use the classical Legendre\n recurrence relation with increasing L.\n\n Useful references are\n\n Taweetham Limpanuparb, Josh Milthorpe,\n Associated Legendre Polynomials and Spherical Harmonics Computation for\n Chemistry Applications,\n Proceedings of The 40th Congress on Science and Technology of Thailand;\n 2014 Dec 2-4, Khon Kaen, Thailand. P. 233-241.\n \n\n and\n\n Schneider et al.,\n A new Fortran 90 program to compute regular and irregular associated\n Legendre functions,\n Computer Physics Communications,\n Volume 181, Issue 12, December 2010, Pages 2091-2097,\n .\n\n The return value is a list of arrays, where `out[k]` hosts the `2*k+1`\n values of the `k`th level of the tree\n\n (0, 0)\n (-1, 1) (0, 1) (1, 1)\n (-2, 2) (-1, 2) (0, 2) (1, 2) (2, 2)\n ... ... ... ... ...\n \"\"\"\n # assert numpy.all(numpy.abs(x) <= 1.0)\n\n d = {\n \"natural\": (_Natural, [x, symbolic]),\n \"spherical\": (_Spherical, [x, symbolic]),\n \"complex spherical\": (_ComplexSpherical, [x, phi, symbolic, False]),\n \"complex spherical 1\": (_ComplexSpherical, [x, phi, symbolic, True]),\n \"normal\": (_Normal, [x, symbolic]),\n \"schmidt\": (_Schmidt, [x, phi, symbolic]),\n }\n fun, args = d[standardization]\n c = fun(*args)\n\n if with_condon_shortley_phase:\n\n def z1_factor_CSP(L):\n return -1 * c.z1_factor(L)\n\n else:\n z1_factor_CSP = c.z1_factor\n\n # Here comes the actual loop.\n e = numpy.ones_like(x, dtype=int)\n out = [[e * c.p0]]\n for L in range(1, n + 1):\n out.append(\n numpy.concatenate(\n [\n [out[L - 1][0] * c.z0_factor(L)],\n out[L - 1] * numpy.multiply.outer(c.C0(L), x),\n [out[L - 1][-1] * z1_factor_CSP(L)],\n ]\n )\n )\n\n if L > 1:\n out[-1][2:-2] -= numpy.multiply.outer(c.C1(L), e) * out[L - 2]\n\n return out", "response": "Evaluates the entire tree of associated Legendre polynomials up to depth n."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tree(X, n, symbolic=False):\n frac = sympy.Rational if symbolic else lambda x, y: x / y\n sqrt = sympy.sqrt if symbolic else numpy.sqrt\n pi = sympy.pi if symbolic else numpy.pi\n\n mu = frac(1, 2)\n\n p0 = 1 / sqrt(pi)\n\n def alpha(n):\n return numpy.array(\n [\n 2\n * sqrt(\n frac(\n (n + mu + frac(1, 2)) * (n + mu - frac(1, 2)),\n (n - k) * (n + k + 2 * mu),\n )\n )\n for k in range(n)\n ]\n )\n\n def beta(n):\n return 2 * sqrt(\n frac((n + mu - 1) * (n + mu + frac(1, 2)), (n + 2 * mu - 1) * n)\n )\n\n def gamma(n):\n return numpy.array(\n [\n sqrt(\n frac(\n (n - 1 - k) * (n + mu + frac(1, 2)) * (n + k + 2 * mu - 1),\n (n - k) * (n + mu - frac(3, 2)) * (n + k + 2 * mu),\n )\n )\n for k in range(n - 1)\n ]\n )\n\n def delta(n):\n return sqrt(\n frac(\n (n - 1)\n * (n + 2 * mu - 2)\n * (n + mu - frac(1, 2))\n * (n + mu + frac(1, 2)),\n n * (n + 2 * mu - 1) * (n + mu - 1) * (n + mu - 2),\n )\n )\n\n out = [numpy.array([0 * X[0] + p0])]\n\n one_min_x2 = 1 - X[0] ** 2\n\n for L in range(1, n + 1):\n out.append(\n numpy.concatenate(\n [\n out[L - 1] * numpy.multiply.outer(alpha(L), X[0]),\n [out[L - 1][L - 1] * beta(L) * X[1]],\n ]\n )\n )\n\n if L > 1:\n out[-1][: L - 1] -= (out[L - 2].T * gamma(L)).T\n out[-1][-1] -= out[L - 2][L - 2] * delta(L) * one_min_x2\n\n return out", "response": "Evaluates the entire tree of orthogonal polynomials on the unit disk."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nevaluate the entire tree of orthogonal triangle polynomials.", "response": "def tree(bary, n, standardization, symbolic=False):\n \"\"\"Evaluates the entire tree of orthogonal triangle polynomials.\n\n The return value is a list of arrays, where `out[k]` hosts the `2*k+1`\n values of the `k`th level of the tree\n\n (0, 0)\n (0, 1) (1, 1)\n (0, 2) (1, 2) (2, 2)\n ... ... ...\n\n For reference, see\n\n Abedallah Rababah,\n Recurrence Relations for Orthogonal Polynomials on Triangular Domains,\n Mathematics 2016, 4(2), 25,\n .\n \"\"\"\n S = numpy.vectorize(sympy.S) if symbolic else lambda x: x\n sqrt = numpy.vectorize(sympy.sqrt) if symbolic else numpy.sqrt\n\n if standardization == \"1\":\n p0 = 1\n\n def alpha(n):\n r = numpy.arange(n)\n return S(n * (2 * n + 1)) / ((n - r) * (n + r + 1))\n\n def beta(n):\n r = numpy.arange(n)\n return S(n * (2 * r + 1) ** 2) / ((n - r) * (n + r + 1) * (2 * n - 1))\n\n def gamma(n):\n r = numpy.arange(n - 1)\n return S((n - r - 1) * (n + r) * (2 * n + 1)) / (\n (n - r) * (n + r + 1) * (2 * n - 1)\n )\n\n def delta(n):\n return S(2 * n - 1) / n\n\n def epsilon(n):\n return S(n - 1) / n\n\n else:\n # The coefficients here are based on the insight that\n #\n # int_T P_{n, r}^2 =\n # int_0^1 L_r^2(t) dt * int_0^1 q_{n,r}(w)^2 (1-w)^(r+s+1) dw.\n #\n # For reference, see\n # page 219 (and the reference to Gould, 1972) in\n #\n # Farouki, Goodman, Sauer,\n # Construction of orthogonal bases for polynomials in Bernstein form\n # on triangular and simplex domains,\n # Computer Aided Geometric Design 20 (2003) 209\u2013230.\n #\n # The Legendre integral is 1/(2*r+1), and one gets\n #\n # int_T P_{n, r}^2 = 1 / (2*r+1) / (2*n+2)\n # sum_{i=0}^{n-r} sum_{j=0}^{n-r}\n # (-1)**(i+j) * binom(n+r+1, i) * binom(n-r, i)\n # * binom(n+r+1, j) * binom(n-r, j)\n # / binom(2*n+1, i+j)\n #\n # Astonishingly, the double sum is always 1, hence\n #\n # int_T P_{n, r}^2 = 1 / (2*r+1) / (2*n+2).\n #\n assert standardization == \"normal\"\n p0 = sqrt(2)\n\n def alpha(n):\n r = numpy.arange(n)\n return sqrt((n + 1) * n) * (S(2 * n + 1) / ((n - r) * (n + r + 1)))\n\n def beta(n):\n r = numpy.arange(n)\n return (\n sqrt((n + 1) * n)\n * S((2 * r + 1) ** 2)\n / ((n - r) * (n + r + 1) * (2 * n - 1))\n )\n\n def gamma(n):\n r = numpy.arange(n - 1)\n return sqrt(S(n + 1) / (n - 1)) * (\n S((n - r - 1) * (n + r) * (2 * n + 1))\n / ((n - r) * (n + r + 1) * (2 * n - 1))\n )\n\n def delta(n):\n return sqrt(S((2 * n + 1) * (n + 1) * (2 * n - 1)) / n ** 3)\n\n def epsilon(n):\n return sqrt(S((2 * n + 1) * (n + 1) * (n - 1)) / ((2 * n - 3) * n ** 2))\n\n u, v, w = bary\n\n out = [numpy.array([numpy.zeros_like(u) + p0])]\n\n for L in range(1, n + 1):\n out.append(\n numpy.concatenate(\n [\n out[L - 1]\n * (numpy.multiply.outer(alpha(L), 1 - 2 * w).T - beta(L)).T,\n [delta(L) * out[L - 1][L - 1] * (u - v)],\n ]\n )\n )\n\n if L > 1:\n out[-1][: L - 1] -= (out[L - 2].T * gamma(L)).T\n out[-1][-1] -= epsilon(L) * out[L - 2][L - 2] * (u + v) ** 2\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_if_this_file_exist(filename):\n #get the absolute path\n filename = os.path.abspath(filename)\n\n #Boolean\n this_file_exist = os.path.exists(filename)\n a_directory = os.path.isdir(filename)\n\n result = this_file_exist and not a_directory\n if result == False:\n raise ValueError('The filename given was either non existent or was a directory')\n else:\n return result", "response": "Check if the given filename actually exists and if it s a directory and if it s not a file"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nhandles the command line call keyword arguments return 0 if error or a string for the command line output", "response": "def command_line(cmd):\n \"\"\"Handle the command line call\n\n keyword arguments:\n cmd = a list\n\n return\n 0 if error\n or a string for the command line output\n \"\"\"\n try:\n s = subprocess.Popen(cmd, stdout=subprocess.PIPE)\n s = s.stdout.read()\n\n return s.strip()\n\n except subprocess.CalledProcessError:\n return 0"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the exif of a file", "response": "def information(filename):\n \"\"\"Returns the file exif\"\"\"\n check_if_this_file_exist(filename)\n filename = os.path.abspath(filename)\n result = get_json(filename)\n result = result[0]\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a json value of the exif", "response": "def get_json(filename):\n \"\"\" Return a json value of the exif\n\n Get a filename and return a JSON object\n\n Arguments:\n filename {string} -- your filename\n\n Returns:\n [JSON] -- Return a JSON object\n \"\"\"\n check_if_this_file_exist(filename)\n\n #Process this function\n filename = os.path.abspath(filename)\n s = command_line(['exiftool', '-G', '-j', '-sort', filename])\n if s:\n #convert bytes to string\n s = s.decode('utf-8').rstrip('\\r\\n')\n return json.loads(s)\n else:\n return s"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_csv(filename):\n check_if_this_file_exist(filename)\n\n #Process this function\n filename = os.path.abspath(filename)\n s = command_line(['exiftool', '-G', '-csv', '-sort', filename])\n if s:\n #convert bytes to string\n s = s.decode('utf-8')\n return s\n else:\n return 0", "response": "Returns a csv representation of the exif\n get a filename and returns a unicode string with a csv format"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef print_a_header(message=\"-\"):\n print(\"-\".center(60,'-'))\n print(message.center(60,'-'))\n print(\"-\".center(60,'-'))\n print()", "response": "This function will output a message in a header"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef check_if_exiftool_is_already_installed():\n result = 1;\n command = [\"exiftool\", \"-ver\"]\n\n with open(os.devnull, \"w\") as fnull:\n result = subprocess.call(\n command,\n stdout = fnull,\n stderr = fnull\n )\n #Exiftool is not installed\n if result != 0:\n print_a_header('Exiftool needs to be installed on your system')\n print_a_header('Visit http://www.sno.phy.queensu.ca/~phil/exiftool/')\n return False\n else:\n return True", "response": "Checks if Exiftool is installed on your system and returns True if it is otherwise False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef render(self, context=None, clean=False):\n # Load template if it is not loaded yet.\n if not self.template:\n self.load_template(self.template_name)\n # The signature of the `render()` method was changed in Django 1.7.\n # https://docs.djangoproject.com/en/1.8/ref/templates/upgrading/#get-template-and-select-template\n if hasattr(self.template, 'template'):\n context = (context or self.context).copy()\n else:\n context = Context(context or self.context)\n # Add tag strings to the context.\n context.update(self.extra_context)\n result = self.template.render(context)\n # Don't overwrite default value with empty one.\n subject = self._get_block(result, 'subject')\n if subject:\n self.subject = self._get_block(result, 'subject')\n body = self._get_block(result, 'body')\n is_html_body = False\n # The html block is optional, and it also may be set manually.\n html = self._get_block(result, 'html')\n if html:\n if not body:\n # This is an html message without plain text part.\n body = html\n is_html_body = True\n else:\n # Add alternative content.\n self.attach_alternative(html, 'text/html')\n # Don't overwrite default value with empty one.\n if body:\n self.body = body\n if is_html_body:\n self.content_subtype = 'html'\n self._is_rendered = True\n if clean:\n self.clean()", "response": "Render the message with the provided context."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend the message to the related object.", "response": "def send(self, *args, **kwargs):\n \"\"\"\n Send email message, render if it is not rendered yet.\n\n Note\n ----\n Any extra arguments are passed to\n :class:`EmailMultiAlternatives.send() `.\n\n Keyword Arguments\n -----------------\n clean : bool\n If ``True``, remove any template specific properties from the\n message object. Default is ``False``.\n \"\"\"\n clean = kwargs.pop('clean', False)\n if not self._is_rendered:\n self.render()\n if clean:\n self.clean()\n return super(EmailMessage, self).send(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_mail(template_name, context, from_email, recipient_list,\n fail_silently=False, auth_user=None, auth_password=None,\n connection=None, **kwargs):\n \"\"\"\n Easy wrapper for sending a single email message to a recipient list using\n django template system.\n\n It works almost the same way as the standard\n :func:`send_mail()` function.\n\n .. |main_difference| replace:: The main\n difference is that two first arguments ``subject`` and ``body`` are\n replaced with ``template_name`` and ``context``. However you still can\n pass subject or body as keyword arguments to provide static content if\n needed.\n\n |main_difference|\n\n The ``template_name``, ``context``, ``from_email`` and ``recipient_list``\n parameters are required.\n\n\n Note\n ----\n |args_note|\n\n Arguments\n ---------\n template_name : str\n |template_name|\n context : dict\n |context|\n from_email : str\n |from_email|\n recipient_list : list\n |recipient_list|\n\n Keyword Arguments\n -----------------\n fail_silently : bool\n If it's False, send_mail will raise an :exc:`smtplib.SMTPException`.\n See the :mod:`smtplib` docs for a list of possible exceptions, all of\n which are subclasses of :exc:`smtplib.SMTPException`.\n auth_user | str\n The optional username to use to authenticate to the SMTP server. If\n this isn't provided, Django will use the value of the\n :django:setting:`EMAIL_HOST_USER` setting.\n auth_password | str\n The optional password to use to authenticate to the SMTP server. If\n this isn't provided, Django will use the value of the\n :django:setting:`EMAIL_HOST_PASSWORD` setting.\n connection : EmailBackend\n The optional email backend to use to send the mail. If unspecified,\n an instance of the default backend will be used. See the documentation\n on :ref:`Email backends` for more details.\n subject : str\n |subject|\n body : str\n |body|\n render : bool\n |render|\n\n Returns\n -------\n int\n The number of successfully delivered messages (which can be 0 or 1\n since it can only send one message).\n\n See Also\n --------\n :func:`django.core.mail.send_mail`\n Documentation for the standard ``send_mail()`` function.\n \"\"\"\n\n connection = connection or mail.get_connection(username=auth_user,\n password=auth_password,\n fail_silently=fail_silently)\n clean = kwargs.pop('clean', True)\n return EmailMessage(\n template_name, context, from_email, recipient_list,\n connection=connection, **kwargs).send(clean=clean)", "response": "This function is a convenience function for sending an email message to a list of recipients."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the mean Silhouette Coefficient of all samples.", "response": "def silhouette_score(X, labels, metric='euclidean', sample_size=None,\n random_state=None, **kwds):\n \"\"\"Compute the mean Silhouette Coefficient of all samples.\n\n The Silhouette Coefficient is calculated using the mean intra-cluster\n distance (``a``) and the mean nearest-cluster distance (``b``) for each\n sample. The Silhouette Coefficient for a sample is ``(b - a) / max(a,\n b)``. To clarify, ``b`` is the distance between a sample and the nearest\n cluster that the sample is not a part of.\n Note that Silhouette Coefficient is only defined if number of labels\n is 2 <= n_labels <= n_samples - 1.\n\n This function returns the mean Silhouette Coefficient over all samples.\n To obtain the values for each sample, use :func:`silhouette_samples`.\n\n The best value is 1 and the worst value is -1. Values near 0 indicate\n overlapping clusters. Negative values generally indicate that a sample has\n been assigned to the wrong cluster, as a different cluster is more similar.\n\n Read more in the :ref:`User Guide `.\n\n Parameters\n ----------\n X : array [n_samples_a, n_samples_a] if metric == \"precomputed\", or, \\\n [n_samples_a, n_features] otherwise\n Array of pairwise distances between samples, or a feature array.\n\n labels : array, shape = [n_samples]\n Predicted labels for each sample.\n\n metric : string, or callable\n The metric to use when calculating distance between instances in a\n feature array. If metric is a string, it must be one of the options\n allowed by :func:`metrics.pairwise.pairwise_distances\n `. If X is the distance\n array itself, use ``metric=\"precomputed\"``.\n\n sample_size : int or None\n The size of the sample to use when computing the Silhouette Coefficient\n on a random subset of the data.\n If ``sample_size is None``, no sampling is used.\n\n random_state : int, RandomState instance or None, optional (default=None)\n The generator used to randomly select a subset of samples. If int,\n random_state is the seed used by the random number generator; If\n RandomState instance, random_state is the random number generator; If\n None, the random number generator is the RandomState instance used by\n `np.random`. Used when ``sample_size is not None``.\n\n **kwds : optional keyword parameters\n Any further parameters are passed directly to the distance function.\n If using a scipy.spatial.distance metric, the parameters are still\n metric dependent. See the scipy docs for usage examples.\n\n Returns\n -------\n silhouette : float\n Mean Silhouette Coefficient for all samples.\n\n References\n ----------\n\n .. [1] `Peter J. Rousseeuw (1987). \"Silhouettes: a Graphical Aid to the\n Interpretation and Validation of Cluster Analysis\". Computational\n and Applied Mathematics 20: 53-65.\n `_\n\n .. [2] `Wikipedia entry on the Silhouette Coefficient\n `_\n\n \"\"\"\n if sample_size is not None:\n X, labels = check_X_y(X, labels, accept_sparse=['csc', 'csr'])\n random_state = check_random_state(random_state)\n indices = random_state.permutation(X.shape[0])[:sample_size]\n if metric == \"precomputed\":\n X, labels = X[indices].T[indices].T, labels[indices]\n else:\n X, labels = X[indices], labels[indices]\n return np.mean(silhouette_samples(X, labels, metric=metric, **kwds))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the Silhouette Coefficient for each sample in X.", "response": "def silhouette_samples(X, labels, metric='euclidean', **kwds):\n \"\"\"Compute the Silhouette Coefficient for each sample.\n\n The Silhouette Coefficient is a measure of how well samples are clustered\n with samples that are similar to themselves. Clustering models with a high\n Silhouette Coefficient are said to be dense, where samples in the same\n cluster are similar to each other, and well separated, where samples in\n different clusters are not very similar to each other.\n\n The Silhouette Coefficient is calculated using the mean intra-cluster\n distance (``a``) and the mean nearest-cluster distance (``b``) for each\n sample. The Silhouette Coefficient for a sample is ``(b - a) / max(a,\n b)``.\n Note that Silhouette Coefficient is only defined if number of labels\n is 2 <= n_labels <= n_samples - 1.\n\n This function returns the Silhouette Coefficient for each sample.\n\n The best value is 1 and the worst value is -1. Values near 0 indicate\n overlapping clusters.\n\n Read more in the :ref:`User Guide `.\n\n Parameters\n ----------\n X : array [n_samples_a, n_samples_a] if metric == \"precomputed\", or, \\\n [n_samples_a, n_features] otherwise\n Array of pairwise distances between samples, or a feature array.\n\n labels : array, shape = [n_samples]\n label values for each sample\n\n metric : string, or callable\n The metric to use when calculating distance between instances in a\n feature array. If metric is a string, it must be one of the options\n allowed by :func:`sklearn.metrics.pairwise.pairwise_distances`. If X is\n the distance array itself, use \"precomputed\" as the metric.\n\n **kwds : optional keyword parameters\n Any further parameters are passed directly to the distance function.\n If using a ``scipy.spatial.distance`` metric, the parameters are still\n metric dependent. See the scipy docs for usage examples.\n\n Returns\n -------\n silhouette : array, shape = [n_samples]\n Silhouette Coefficient for each samples.\n\n References\n ----------\n\n .. [1] `Peter J. Rousseeuw (1987). \"Silhouettes: a Graphical Aid to the\n Interpretation and Validation of Cluster Analysis\". Computational\n and Applied Mathematics 20: 53-65.\n `_\n\n .. [2] `Wikipedia entry on the Silhouette Coefficient\n `_\n\n \"\"\"\n X, labels = check_X_y(X, labels, accept_sparse=['csc', 'csr'])\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n check_number_of_labels(len(le.classes_), X.shape[0])\n\n distances = pairwise_distances(X, metric=metric, **kwds)\n unique_labels = le.classes_\n n_samples_per_label = np.bincount(labels, minlength=len(unique_labels))\n\n # For sample i, store the mean distance of the cluster to which\n # it belongs in intra_clust_dists[i]\n intra_clust_dists = np.zeros(distances.shape[0], dtype=distances.dtype)\n\n # For sample i, store the mean distance of the second closest\n # cluster in inter_clust_dists[i]\n inter_clust_dists = np.inf + intra_clust_dists\n\n for curr_label in range(len(unique_labels)):\n\n # Find inter_clust_dist for all samples belonging to the same\n # label.\n mask = labels == curr_label\n current_distances = distances[mask]\n\n # Leave out current sample.\n n_samples_curr_lab = n_samples_per_label[curr_label] - 1\n if n_samples_curr_lab != 0:\n intra_clust_dists[mask] = np.sum(\n current_distances[:, mask], axis=1) / n_samples_curr_lab\n\n # Now iterate over all other labels, finding the mean\n # cluster distance that is closest to every sample.\n for other_label in range(len(unique_labels)):\n if other_label != curr_label:\n other_mask = labels == other_label\n other_distances = np.mean(\n current_distances[:, other_mask], axis=1)\n inter_clust_dists[mask] = np.minimum(\n inter_clust_dists[mask], other_distances)\n\n sil_samples = inter_clust_dists - intra_clust_dists\n sil_samples /= np.maximum(intra_clust_dists, inter_clust_dists)\n # score 0 for clusters of size 1, according to the paper\n sil_samples[n_samples_per_label.take(labels) == 1] = 0\n return sil_samples"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calinski_harabaz_score(X, labels):\n X, labels = check_X_y(X, labels)\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n\n n_samples, _ = X.shape\n n_labels = len(le.classes_)\n\n check_number_of_labels(n_labels, n_samples)\n\n extra_disp, intra_disp = 0., 0.\n mean = np.mean(X, axis=0)\n for k in range(n_labels):\n cluster_k = X[labels == k]\n mean_k = np.mean(cluster_k, axis=0)\n extra_disp += len(cluster_k) * np.sum((mean_k - mean) ** 2)\n intra_disp += np.sum((cluster_k - mean_k) ** 2)\n\n return (1. if intra_disp == 0. else\n extra_disp * (n_samples - n_labels) /\n (intra_disp * (n_labels - 1.)))", "response": "Compute the Calinski and Harabaz score."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling zeroes in a single scale.", "response": "def handle_zeros_in_scale(scale, copy=True):\n ''' Makes sure that whenever scale is zero, we handle it correctly.\n This happens in most scalers when we have constant features.\n Adapted from sklearn.preprocessing.data'''\n\n # if we are fitting on 1D arrays, scale might be a scalar\n if np.isscalar(scale):\n if scale == .0:\n scale = 1.\n return scale\n elif isinstance(scale, np.ndarray):\n if copy:\n # New array to avoid side-effects\n scale = scale.copy()\n scale[scale == 0.0] = 1.0\n return scale"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomputing joint probabilities p_ij from distances.", "response": "def _joint_probabilities(distances, desired_perplexity, verbose):\n \"\"\"Compute joint probabilities p_ij from distances.\n\n Parameters\n ----------\n distances : array, shape (n_samples * (n_samples-1) / 2,)\n Distances of samples are stored as condensed matrices, i.e.\n we omit the diagonal and duplicate entries and store everything\n in a one-dimensional array.\n\n desired_perplexity : float\n Desired perplexity of the joint probability distributions.\n\n verbose : int\n Verbosity level.\n\n Returns\n -------\n P : array, shape (n_samples * (n_samples-1) / 2,)\n Condensed joint probability matrix.\n \"\"\"\n # Compute conditional probabilities such that they approximately match\n # the desired perplexity\n distances = distances.astype(np.float32, copy=False)\n conditional_P = _utils._binary_search_perplexity(\n distances, None, desired_perplexity, verbose)\n P = conditional_P + conditional_P.T\n sum_P = np.maximum(np.sum(P), MACHINE_EPSILON)\n P = np.maximum(squareform(P) / sum_P, MACHINE_EPSILON)\n return P"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _joint_probabilities_nn(distances, neighbors, desired_perplexity, verbose):\n t0 = time()\n # Compute conditional probabilities such that they approximately match\n # the desired perplexity\n n_samples, k = neighbors.shape\n distances = distances.astype(np.float32, copy=False)\n neighbors = neighbors.astype(np.int64, copy=False)\n conditional_P = _utils._binary_search_perplexity(\n distances, neighbors, desired_perplexity, verbose)\n assert np.all(np.isfinite(conditional_P)), \\\n \"All probabilities should be finite\"\n\n # Symmetrize the joint probability distribution using sparse operations\n P = csr_matrix((conditional_P.ravel(), neighbors.ravel(),\n range(0, n_samples * k + 1, k)),\n shape=(n_samples, n_samples))\n P = P + P.T\n\n # Normalize the joint probability distribution\n sum_P = np.maximum(P.sum(), MACHINE_EPSILON)\n P /= sum_P\n\n assert np.all(np.abs(P.data) <= 1.0)\n if verbose >= 2:\n duration = time() - t0\n print(\"[t-SNE] Computed conditional probabilities in {:.3f}s\"\n .format(duration))\n return P", "response": "Compute joint probabilities p_ij from distances using just nearest - neighbors."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _kl_divergence(params, P, degrees_of_freedom, n_samples, n_components,\n skip_num_points=0):\n \"\"\"t-SNE objective function: gradient of the KL divergence\n of p_ijs and q_ijs and the absolute error.\n\n Parameters\n ----------\n params : array, shape (n_params,)\n Unraveled embedding.\n\n P : array, shape (n_samples * (n_samples-1) / 2,)\n Condensed joint probability matrix.\n\n degrees_of_freedom : float\n Degrees of freedom of the Student's-t distribution.\n\n n_samples : int\n Number of samples.\n\n n_components : int\n Dimension of the embedded space.\n\n skip_num_points : int (optional, default:0)\n This does not compute the gradient for points with indices below\n `skip_num_points`. This is useful when computing transforms of new\n data where you'd like to keep the old data fixed.\n\n Returns\n -------\n kl_divergence : float\n Kullback-Leibler divergence of p_ij and q_ij.\n\n grad : array, shape (n_params,)\n Unraveled gradient of the Kullback-Leibler divergence with respect to\n the embedding.\n \"\"\"\n X_embedded = params.reshape(n_samples, n_components)\n\n # Q is a heavy-tailed distribution: Student's t-distribution\n dist = pdist(X_embedded, \"sqeuclidean\")\n dist += 1.\n dist /= degrees_of_freedom\n dist **= (degrees_of_freedom + 1.0) / -2.0\n Q = np.maximum(dist / (2.0 * np.sum(dist)), MACHINE_EPSILON)\n\n # Optimization trick below: np.dot(x, y) is faster than\n # np.sum(x * y) because it calls BLAS\n\n # Objective: C (Kullback-Leibler divergence of P and Q)\n kl_divergence = 2.0 * np.dot(P, np.log(np.maximum(P, MACHINE_EPSILON) / Q))\n\n # Gradient: dC/dY\n # pdist always returns double precision distances. Thus we need to take\n grad = np.ndarray((n_samples, n_components), dtype=params.dtype)\n PQd = squareform((P - Q) * dist)\n for i in range(skip_num_points, n_samples):\n grad[i] = np.dot(np.ravel(PQd[i], order='K'),\n X_embedded[i] - X_embedded)\n grad = grad.ravel()\n c = 2.0 * (degrees_of_freedom + 1.0) / degrees_of_freedom\n grad *= c\n\n return kl_divergence, grad", "response": "t - SNE objective function for Kullback - Leibler divergence of p_ij and q_ij."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _gradient_descent(objective, p0, it, n_iter,\n n_iter_check=1, n_iter_without_progress=300,\n momentum=0.8, learning_rate=200.0, min_gain=0.01,\n min_grad_norm=1e-7, verbose=0, args=None, kwargs=None):\n \"\"\"Batch gradient descent with momentum and individual gains.\n\n Parameters\n ----------\n objective : function or callable\n Should return a tuple of cost and gradient for a given parameter\n vector. When expensive to compute, the cost can optionally\n be None and can be computed every n_iter_check steps using\n the objective_error function.\n\n p0 : array-like, shape (n_params,)\n Initial parameter vector.\n\n it : int\n Current number of iterations (this function will be called more than\n once during the optimization).\n\n n_iter : int\n Maximum number of gradient descent iterations.\n\n n_iter_check : int\n Number of iterations before evaluating the global error. If the error\n is sufficiently low, we abort the optimization.\n\n n_iter_without_progress : int, optional (default: 300)\n Maximum number of iterations without progress before we abort the\n optimization.\n\n momentum : float, within (0.0, 1.0), optional (default: 0.8)\n The momentum generates a weight for previous gradients that decays\n exponentially.\n\n learning_rate : float, optional (default: 200.0)\n The learning rate for t-SNE is usually in the range [10.0, 1000.0]. If\n the learning rate is too high, the data may look like a 'ball' with any\n point approximately equidistant from its nearest neighbours. If the\n learning rate is too low, most points may look compressed in a dense\n cloud with few outliers.\n\n min_gain : float, optional (default: 0.01)\n Minimum individual gain for each parameter.\n\n min_grad_norm : float, optional (default: 1e-7)\n If the gradient norm is below this threshold, the optimization will\n be aborted.\n\n verbose : int, optional (default: 0)\n Verbosity level.\n\n args : sequence\n Arguments to pass to objective function.\n\n kwargs : dict\n Keyword arguments to pass to objective function.\n\n Returns\n -------\n p : array, shape (n_params,)\n Optimum parameters.\n\n error : float\n Optimum.\n\n i : int\n Last iteration.\n \"\"\"\n if args is None:\n args = []\n if kwargs is None:\n kwargs = {}\n\n p = p0.copy().ravel()\n update = np.zeros_like(p)\n gains = np.ones_like(p)\n error = np.finfo(np.float).max\n best_error = np.finfo(np.float).max\n best_iter = i = it\n\n tic = time()\n for i in range(it, n_iter):\n error, grad = objective(p, *args, **kwargs)\n grad_norm = linalg.norm(grad)\n\n inc = update * grad < 0.0\n dec = np.invert(inc)\n gains[inc] += 0.2\n gains[dec] *= 0.8\n np.clip(gains, min_gain, np.inf, out=gains)\n grad *= gains\n update = momentum * update - learning_rate * grad\n p += update\n\n if (i + 1) % n_iter_check == 0:\n toc = time()\n duration = toc - tic\n tic = toc\n\n if verbose >= 2:\n print(\"[t-SNE] Iteration %d: error = %.7f,\"\n \" gradient norm = %.7f\"\n \" (%s iterations in %0.3fs)\"\n % (i + 1, error, grad_norm, n_iter_check, duration))\n\n if error < best_error:\n best_error = error\n best_iter = i\n elif i - best_iter > n_iter_without_progress:\n if verbose >= 2:\n print(\"[t-SNE] Iteration %d: did not make any progress \"\n \"during the last %d episodes. Finished.\"\n % (i + 1, n_iter_without_progress))\n break\n if grad_norm <= min_grad_norm:\n if verbose >= 2:\n print(\"[t-SNE] Iteration %d: gradient norm %f. Finished.\"\n % (i + 1, grad_norm))\n break\n\n return p, error, i", "response": "Batch gradient descent for a given parameter vector."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the trustworthiness of the local structure.", "response": "def trustworthiness(X, X_embedded, n_neighbors=5, precomputed=False):\n \"\"\"Expresses to what extent the local structure is retained.\n\n The trustworthiness is within [0, 1]. It is defined as\n\n .. math::\n\n T(k) = 1 - \\frac{2}{nk (2n - 3k - 1)} \\sum^n_{i=1}\n \\sum_{j \\in U^{(k)}_i} (r(i, j) - k)\n\n where :math:`r(i, j)` is the rank of the embedded datapoint j\n according to the pairwise distances between the embedded datapoints,\n :math:`U^{(k)}_i` is the set of points that are in the k nearest\n neighbors in the embedded space but not in the original space.\n\n * \"Neighborhood Preservation in Nonlinear Projection Methods: An\n Experimental Study\"\n J. Venna, S. Kaski\n * \"Learning a Parametric Embedding by Preserving Local Structure\"\n L.J.P. van der Maaten\n\n Parameters\n ----------\n X : array, shape (n_samples, n_features) or (n_samples, n_samples)\n If the metric is 'precomputed' X must be a square distance\n matrix. Otherwise it contains a sample per row.\n\n X_embedded : array, shape (n_samples, n_components)\n Embedding of the training data in low-dimensional space.\n\n n_neighbors : int, optional (default: 5)\n Number of neighbors k that will be considered.\n\n precomputed : bool, optional (default: False)\n Set this flag if X is a precomputed square distance matrix.\n\n Returns\n -------\n trustworthiness : float\n Trustworthiness of the low-dimensional embedding.\n \"\"\"\n if precomputed:\n dist_X = X\n else:\n dist_X = pairwise_distances(X, squared=True)\n dist_X_embedded = pairwise_distances(X_embedded, squared=True)\n ind_X = np.argsort(dist_X, axis=1)\n ind_X_embedded = np.argsort(dist_X_embedded, axis=1)[:, 1:n_neighbors + 1]\n\n n_samples = X.shape[0]\n t = 0.0\n ranks = np.zeros(n_neighbors)\n for i in range(n_samples):\n for j in range(n_neighbors):\n ranks[j] = np.where(ind_X[i] == ind_X_embedded[i, j])[0][0]\n ranks -= n_neighbors\n t += np.sum(ranks[ranks > 0])\n t = 1.0 - t * (2.0 / (n_samples * n_neighbors *\n (2.0 * n_samples - 3.0 * n_neighbors - 1.0)))\n return t"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _fit(self, X, skip_num_points=0):\n if self.method not in ['barnes_hut', 'exact']:\n raise ValueError(\"'method' must be 'barnes_hut' or 'exact'\")\n if self.angle < 0.0 or self.angle > 1.0:\n raise ValueError(\"'angle' must be between 0.0 - 1.0\")\n if self.metric == \"precomputed\":\n if isinstance(self.init, string_types) and self.init == 'pca':\n raise ValueError(\"The parameter init=\\\"pca\\\" cannot be \"\n \"used with metric=\\\"precomputed\\\".\")\n if X.shape[0] != X.shape[1]:\n raise ValueError(\"X should be a square distance matrix\")\n if np.any(X < 0):\n raise ValueError(\"All distances should be positive, the \"\n \"precomputed distances given as X is not \"\n \"correct\")\n if self.method == 'barnes_hut' and sp.issparse(X):\n raise TypeError('A sparse matrix was passed, but dense '\n 'data is required for method=\"barnes_hut\". Use '\n 'X.toarray() to convert to a dense numpy array if '\n 'the array is small enough for it to fit in '\n 'memory. Otherwise consider dimensionality '\n 'reduction techniques (e.g. TruncatedSVD)')\n else:\n X = check_array(X, accept_sparse=['csr', 'csc', 'coo'],\n dtype=[np.float32, np.float64])\n if self.method == 'barnes_hut' and self.n_components > 3:\n raise ValueError(\"'n_components' should be inferior to 4 for the \"\n \"barnes_hut algorithm as it relies on \"\n \"quad-tree or oct-tree.\")\n random_state = check_random_state(self.random_state)\n\n if self.early_exaggeration < 1.0:\n raise ValueError(\"early_exaggeration must be at least 1, but is {}\"\n .format(self.early_exaggeration))\n\n if self.n_iter < 250:\n raise ValueError(\"n_iter should be at least 250\")\n\n n_samples = X.shape[0]\n\n neighbors_nn = None\n if self.method == \"exact\":\n # Retrieve the distance matrix, either using the precomputed one or\n # computing it.\n if self.metric == \"precomputed\":\n distances = X\n else:\n if self.verbose:\n print(\"[t-SNE] Computing pairwise distances...\")\n\n if self.metric == \"euclidean\":\n distances = pairwise_distances(X, metric=self.metric,\n squared=True)\n else:\n distances = pairwise_distances(X, metric=self.metric)\n\n if np.any(distances < 0):\n raise ValueError(\"All distances should be positive, the \"\n \"metric given is not correct\")\n\n # compute the joint probability distribution for the input space\n P = _joint_probabilities(distances, self.perplexity, self.verbose)\n assert np.all(np.isfinite(P)), \"All probabilities should be finite\"\n assert np.all(P >= 0), \"All probabilities should be non-negative\"\n assert np.all(P <= 1), (\"All probabilities should be less \"\n \"or then equal to one\")\n\n else:\n # Cpmpute the number of nearest neighbors to find.\n # LvdM uses 3 * perplexity as the number of neighbors.\n # In the event that we have very small # of points\n # set the neighbors to n - 1.\n k = min(n_samples - 1, int(3. * self.perplexity + 1))\n\n if self.verbose:\n print(\"[t-SNE] Computing {} nearest neighbors...\".format(k))\n\n # Find the nearest neighbors for every point\n neighbors_method = 'ball_tree'\n if (self.metric == 'precomputed'):\n neighbors_method = 'brute'\n knn = AnnoyIndex(X.shape[1], metric='euclidean')\n t0 = time()\n for i in range(n_samples):\n knn.add_item(i, X[i, :])\n knn.build(50)\n duration = time() - t0\n if self.verbose:\n print(\"[t-SNE] Indexed {} samples in {:.3f}s...\".format(\n n_samples, duration))\n\n t0 = time()\n neighbors_nn = np.zeros((n_samples, k), dtype=int)\n distances_nn = np.zeros((n_samples, k))\n for i in range(n_samples):\n (neighbors_nn[i, :], distances_nn[i, :]) = knn.get_nns_by_vector(\n X[i, :], k, include_distances=True\n )\n duration = time() - t0\n if self.verbose:\n print(\"[t-SNE] Computed neighbors for {} samples in {:.3f}s...\"\n .format(n_samples, duration))\n\n # Free the memory used by the ball_tree\n del knn\n\n if self.metric == \"euclidean\":\n # knn return the euclidean distance but we need it squared\n # to be consistent with the 'exact' method. Note that the\n # the method was derived using the euclidean method as in the\n # input space. Not sure of the implication of using a different\n # metric.\n distances_nn **= 2\n\n # compute the joint probability distribution for the input space\n P = _joint_probabilities_nn(distances_nn, neighbors_nn,\n self.perplexity, self.verbose)\n\n if isinstance(self.init, np.ndarray):\n X_embedded = self.init\n elif self.init == 'pca':\n pca = PCA(n_components=self.n_components, svd_solver='randomized',\n random_state=random_state)\n X_embedded = pca.fit_transform(X).astype(np.float32, copy=False)\n elif self.init == 'random':\n # The embedding is initialized with iid samples from Gaussians with\n # standard deviation 1e-4.\n X_embedded = 1e-4 * random_state.randn(\n n_samples, self.n_components).astype(np.float32)\n else:\n raise ValueError(\"'init' must be 'pca', 'random', or \"\n \"a numpy array\")\n\n # Degrees of freedom of the Student's t-distribution. The suggestion\n # degrees_of_freedom = n_components - 1 comes from\n # \"Learning a Parametric Embedding by Preserving Local Structure\"\n # Laurens van der Maaten, 2009.\n degrees_of_freedom = max(self.n_components - 1.0, 1)\n\n return self._tsne(P, degrees_of_freedom, n_samples, random_state,\n X_embedded=X_embedded,\n neighbors=neighbors_nn,\n skip_num_points=skip_num_points)", "response": "Fit the model using X as training data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun t - SNE.", "response": "def _tsne(self, P, degrees_of_freedom, n_samples, random_state, X_embedded,\n neighbors=None, skip_num_points=0):\n \"\"\"Runs t-SNE.\"\"\"\n # t-SNE minimizes the Kullback-Leiber divergence of the Gaussians P\n # and the Student's t-distributions Q. The optimization algorithm that\n # we use is batch gradient descent with two stages:\n # * initial optimization with early exaggeration and momentum at 0.5\n # * final optimization with momentum at 0.8\n params = X_embedded.ravel()\n\n opt_args = {\n \"it\": 0,\n \"n_iter_check\": self._N_ITER_CHECK,\n \"min_grad_norm\": self.min_grad_norm,\n \"learning_rate\": self.learning_rate,\n \"verbose\": self.verbose,\n \"kwargs\": dict(skip_num_points=skip_num_points),\n \"args\": [P, degrees_of_freedom, n_samples, self.n_components],\n \"n_iter_without_progress\": self._EXPLORATION_N_ITER,\n \"n_iter\": self._EXPLORATION_N_ITER,\n \"momentum\": 0.5,\n }\n if self.method == 'barnes_hut':\n obj_func = _kl_divergence_bh\n opt_args['kwargs']['angle'] = self.angle\n # Repeat verbose argument for _kl_divergence_bh\n opt_args['kwargs']['verbose'] = self.verbose\n else:\n obj_func = _kl_divergence\n\n # Learning schedule (part 1): do 250 iteration with lower momentum but\n # higher learning rate controlled via the early exageration parameter\n P *= self.early_exaggeration\n params, kl_divergence, it = _gradient_descent(obj_func, params,\n **opt_args)\n if self.verbose:\n print(\"[t-SNE] KL divergence after %d iterations with early \"\n \"exaggeration: %f\" % (it + 1, kl_divergence))\n\n # Learning schedule (part 2): disable early exaggeration and finish\n # optimization with a higher momentum at 0.8\n P /= self.early_exaggeration\n remaining = self.n_iter - self._EXPLORATION_N_ITER\n if it < self._EXPLORATION_N_ITER or remaining > 0:\n opt_args['n_iter'] = self.n_iter\n opt_args['it'] = it + 1\n opt_args['momentum'] = 0.8\n opt_args['n_iter_without_progress'] = self.n_iter_without_progress\n params, kl_divergence, it = _gradient_descent(obj_func, params,\n **opt_args)\n\n # Save the final number of iterations\n self.n_iter_ = it\n\n if self.verbose:\n print(\"[t-SNE] Error after %d iterations: %f\"\n % (it + 1, kl_divergence))\n\n X_embedded = params.reshape(n_samples, self.n_components)\n self.kl_divergence_ = kl_divergence\n\n return X_embedded"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fit_transform(self, X, y=None):\n embedding = self._fit(X)\n self.embedding_ = embedding\n return self.embedding_", "response": "Fit X into an embedded space and return that transformed\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef correct(datasets_full, genes_list, return_dimred=False,\n batch_size=BATCH_SIZE, verbose=VERBOSE, ds_names=None,\n dimred=DIMRED, approx=APPROX, sigma=SIGMA, alpha=ALPHA, knn=KNN,\n return_dense=False, hvg=None, union=False,\n geosketch=False, geosketch_max=20000):\n \"\"\"Integrate and batch correct a list of data sets.\n\n Parameters\n ----------\n datasets_full : `list` of `scipy.sparse.csr_matrix` or of `numpy.ndarray`\n Data sets to integrate and correct.\n genes_list: `list` of `list` of `string`\n List of genes for each data set.\n return_dimred: `bool`, optional (default: `False`)\n In addition to returning batch corrected matrices, also returns\n integrated low-dimesional embeddings.\n batch_size: `int`, optional (default: `5000`)\n The batch size used in the alignment vector computation. Useful when\n correcting very large (>100k samples) data sets. Set to large value\n that runs within available memory.\n verbose: `bool` or `int`, optional (default: 2)\n When `True` or not equal to 0, prints logging output.\n ds_names: `list` of `string`, optional\n When `verbose=True`, reports data set names in logging output.\n dimred: `int`, optional (default: 100)\n Dimensionality of integrated embedding.\n approx: `bool`, optional (default: `True`)\n Use approximate nearest neighbors, greatly speeds up matching runtime.\n sigma: `float`, optional (default: 15)\n Correction smoothing parameter on Gaussian kernel.\n alpha: `float`, optional (default: 0.10)\n Alignment score minimum cutoff.\n knn: `int`, optional (default: 20)\n Number of nearest neighbors to use for matching.\n return_dense: `bool`, optional (default: `False`)\n Return `numpy.ndarray` matrices instead of `scipy.sparse.csr_matrix`.\n hvg: `int`, optional (default: None)\n Use this number of top highly variable genes based on dispersion.\n\n Returns\n -------\n corrected, genes\n By default (`return_dimred=False`), returns a two-tuple containing a\n list of `scipy.sparse.csr_matrix` each with batch corrected values,\n and a single list of genes containing the intersection of inputted\n genes.\n\n integrated, corrected, genes\n When `return_dimred=False`, returns a three-tuple containing a list\n of `numpy.ndarray` with integrated low dimensional embeddings, a list\n of `scipy.sparse.csr_matrix` each with batch corrected values, and a\n a single list of genes containing the intersection of inputted genes.\n \"\"\"\n datasets_full = check_datasets(datasets_full)\n \n datasets, genes = merge_datasets(datasets_full, genes_list,\n ds_names=ds_names, union=union)\n datasets_dimred, genes = process_data(datasets, genes, hvg=hvg,\n dimred=dimred)\n \n datasets_dimred = assemble(\n datasets_dimred, # Assemble in low dimensional space.\n expr_datasets=datasets, # Modified in place.\n verbose=verbose, knn=knn, sigma=sigma, approx=approx,\n alpha=alpha, ds_names=ds_names, batch_size=batch_size,\n geosketch=geosketch, geosketch_max=geosketch_max,\n )\n\n if return_dense:\n datasets = [ ds.toarray() for ds in datasets ]\n\n if return_dimred:\n return datasets_dimred, datasets, genes\n\n return datasets, genes", "response": "Integrate and batch correct a list of data sets."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef integrate(datasets_full, genes_list, batch_size=BATCH_SIZE,\n verbose=VERBOSE, ds_names=None, dimred=DIMRED, approx=APPROX,\n sigma=SIGMA, alpha=ALPHA, knn=KNN, geosketch=False,\n geosketch_max=20000, n_iter=1, union=False, hvg=None):\n \"\"\"Integrate a list of data sets.\n\n Parameters\n ----------\n datasets_full : `list` of `scipy.sparse.csr_matrix` or of `numpy.ndarray`\n Data sets to integrate and correct.\n genes_list: `list` of `list` of `string`\n List of genes for each data set.\n batch_size: `int`, optional (default: `5000`)\n The batch size used in the alignment vector computation. Useful when\n correcting very large (>100k samples) data sets. Set to large value\n that runs within available memory.\n verbose: `bool` or `int`, optional (default: 2)\n When `True` or not equal to 0, prints logging output.\n ds_names: `list` of `string`, optional\n When `verbose=True`, reports data set names in logging output.\n dimred: `int`, optional (default: 100)\n Dimensionality of integrated embedding.\n approx: `bool`, optional (default: `True`)\n Use approximate nearest neighbors, greatly speeds up matching runtime.\n sigma: `float`, optional (default: 15)\n Correction smoothing parameter on Gaussian kernel.\n alpha: `float`, optional (default: 0.10)\n Alignment score minimum cutoff.\n knn: `int`, optional (default: 20)\n Number of nearest neighbors to use for matching.\n hvg: `int`, optional (default: None)\n Use this number of top highly variable genes based on dispersion.\n\n Returns\n -------\n integrated, genes\n Returns a two-tuple containing a list of `numpy.ndarray` with\n integrated low dimensional embeddings and a single list of genes\n containing the intersection of inputted genes.\n \"\"\"\n datasets_full = check_datasets(datasets_full)\n\n datasets, genes = merge_datasets(datasets_full, genes_list,\n ds_names=ds_names, union=union)\n datasets_dimred, genes = process_data(datasets, genes, hvg=hvg,\n dimred=dimred)\n\n for _ in range(n_iter):\n datasets_dimred = assemble(\n datasets_dimred, # Assemble in low dimensional space.\n verbose=verbose, knn=knn, sigma=sigma, approx=approx,\n alpha=alpha, ds_names=ds_names, batch_size=batch_size,\n geosketch=geosketch, geosketch_max=geosketch_max,\n )\n\n return datasets_dimred, genes", "response": "Integrate a list of data sets and correct a list of genes."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbatches correct a list of scanpy. api. AnnData.", "response": "def correct_scanpy(adatas, **kwargs):\n \"\"\"Batch correct a list of `scanpy.api.AnnData`.\n\n Parameters\n ----------\n adatas : `list` of `scanpy.api.AnnData`\n Data sets to integrate and/or correct.\n kwargs : `dict`\n See documentation for the `correct()` method for a full list of\n parameters to use for batch correction.\n\n Returns\n -------\n corrected\n By default (`return_dimred=False`), returns a list of\n `scanpy.api.AnnData` with batch corrected values in the `.X` field.\n\n corrected, integrated\n When `return_dimred=False`, returns a two-tuple containing a list of\n `np.ndarray` with integrated low-dimensional embeddings and a list\n of `scanpy.api.AnnData` with batch corrected values in the `.X`\n field.\n \"\"\"\n if 'return_dimred' in kwargs and kwargs['return_dimred']:\n datasets_dimred, datasets, genes = correct(\n [adata.X for adata in adatas],\n [adata.var_names.values for adata in adatas],\n **kwargs\n )\n else:\n datasets, genes = correct(\n [adata.X for adata in adatas],\n [adata.var_names.values for adata in adatas],\n **kwargs\n )\n\n new_adatas = []\n for i, adata in enumerate(adatas):\n adata.X = datasets[i]\n new_adatas.append(adata)\n\n if 'return_dimred' in kwargs and kwargs['return_dimred']:\n return datasets_dimred, new_adatas\n else:\n return new_adatas"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nintegrate a list of scanpy. api. AnnData objects.", "response": "def integrate_scanpy(adatas, **kwargs):\n \"\"\"Integrate a list of `scanpy.api.AnnData`.\n\n Parameters\n ----------\n adatas : `list` of `scanpy.api.AnnData`\n Data sets to integrate.\n kwargs : `dict`\n See documentation for the `integrate()` method for a full list of\n parameters to use for batch correction.\n\n Returns\n -------\n integrated\n Returns a list of `np.ndarray` with integrated low-dimensional\n embeddings.\n \"\"\"\n datasets_dimred, genes = integrate(\n [adata.X for adata in adatas],\n [adata.var_names.values for adata in adatas],\n **kwargs\n )\n\n return datasets_dimred"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef augknt(knots, order):\n if isinstance(knots, np.ndarray) and knots.ndim > 1:\n raise ValueError(\"knots must be a list or a rank-1 array\")\n knots = list(knots) # ensure Python list\n\n # One copy of knots[0] and knots[-1] will come from \"knots\" itself,\n # so we only need to prepend/append \"order\" copies.\n #\n return np.array( [knots[0]] * order + knots + [knots[-1]] * order )", "response": "Augment a knot vector with a new order."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the running average of k successive elements of t. Return the averaged array.", "response": "def aveknt(t, k):\n \"\"\"Compute the running average of `k` successive elements of `t`. Return the averaged array.\n\nParameters:\n t:\n Python list or rank-1 array\n k:\n int, >= 2, how many successive elements to average\n\nReturns:\n rank-1 array, averaged data. If k > len(t), returns a zero-length array.\n\nCaveat:\n This is slightly different from MATLAB's aveknt, which returns the running average\n of `k`-1 successive elements of ``t[1:-1]`` (and the empty vector if ``len(t) - 2 < k - 1``).\n\n\"\"\"\n t = np.atleast_1d(t)\n if t.ndim > 1:\n raise ValueError(\"t must be a list or a rank-1 array\")\n\n n = t.shape[0]\n u = max(0, n - (k-1)) # number of elements in the output array\n out = np.empty( (u,), dtype=t.dtype )\n\n for j in range(u):\n out[j] = sum( t[j:(j+k)] ) / k\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef aptknt(tau, order):\n tau = np.atleast_1d(tau)\n k = order + 1\n\n if tau.ndim > 1:\n raise ValueError(\"tau must be a list or a rank-1 array\")\n\n # emulate MATLAB behavior for the \"k\" parameter\n #\n # See\n # https://se.mathworks.com/help/curvefit/aptknt.html\n #\n if len(tau) < k:\n k = len(tau)\n\n if not (tau == sorted(tau)).all():\n raise ValueError(\"tau must be nondecreasing\")\n\n # last processed element needs to be:\n # i + k - 1 = len(tau)- 1\n # => i + k = len(tau)\n # => i = len(tau) - k\n #\n u = len(tau) - k\n for i in range(u):\n if tau[i+k-1] == tau[i]:\n raise ValueError(\"k-fold (or higher) repeated sites not allowed, but tau[i+k-1] == tau[i] for i = %d, k = %d\" % (i,k))\n\n # form the output sequence\n #\n prefix = [ tau[0] ] * k\n suffix = [ tau[-1] ] * k\n\n # https://se.mathworks.com/help/curvefit/aveknt.html\n # MATLAB's aveknt():\n # - averages successive k-1 entries, but ours averages k\n # - seems to ignore the endpoints\n #\n tmp = aveknt(tau[1:-1], k-1)\n middle = tmp.tolist()\n return np.array( prefix + middle + suffix, dtype=tmp.dtype )", "response": "Create an acceptable knot vector."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncounting multiplicities of elements in a sorted list or rank - 1 array.", "response": "def knt2mlt(t):\n \"\"\"Count multiplicities of elements in a sorted list or rank-1 array.\n\nMinimal emulation of MATLAB's ``knt2mlt``.\n\nParameters:\n t:\n Python list or rank-1 array. Must be sorted!\n\nReturns:\n out\n rank-1 array such that\n out[k] = #{ t[i] == t[k] for i < k }\n\nExample:\n If ``t = [1, 1, 2, 3, 3, 3]``, then ``out = [0, 1, 0, 0, 1, 2]``.\n\nCaveat:\n Requires input to be already sorted (this is not checked).\n\"\"\"\n t = np.atleast_1d(t)\n if t.ndim > 1:\n raise ValueError(\"t must be a list or a rank-1 array\")\n\n out = []\n e = None\n for k in range(t.shape[0]):\n if t[k] != e:\n e = t[k]\n count = 0\n else:\n count += 1\n out.append(count)\n\n return np.array( out )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef spcol(knots, order, tau):\n m = knt2mlt(tau)\n B = bspline.Bspline(knots, order)\n\n dummy = B(0.)\n nbasis = len(dummy) # perform dummy evaluation to get number of basis functions\n\n A = np.empty( (tau.shape[0], nbasis), dtype=dummy.dtype )\n for i,item in enumerate(zip(tau,m)):\n taui,mi = item\n f = B.diff(order=mi)\n A[i,:] = f(taui)\n\n return A", "response": "Return collocation matrix. spcol is the minimum emulation of MATLAB s pcol."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __basis0(self, xi):\n return np.where(np.all([self.knot_vector[:-1] <= xi,\n xi < self.knot_vector[1:]],axis=0), 1.0, 0.0)", "response": "Order zero basis ( for internal use."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef d(self, xi):\n return self.__basis(xi, self.p, compute_derivatives=True)", "response": "Convenience function to compute first derivative of basis functions. Memoized for speed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplot basis functions over full range of knots. Convenience function. Requires matplotlib.", "response": "def plot(self):\n \"\"\"Plot basis functions over full range of knots.\n\n Convenience function. Requires matplotlib.\n \"\"\"\n\n try:\n import matplotlib.pyplot as plt\n except ImportError:\n from sys import stderr\n print(\"ERROR: matplotlib.pyplot not found, matplotlib must be installed to use this function\", file=stderr)\n raise\n\n x_min = np.min(self.knot_vector)\n x_max = np.max(self.knot_vector)\n\n x = np.linspace(x_min, x_max, num=1000)\n\n N = np.array([self(i) for i in x]).T\n\n for n in N:\n plt.plot(x,n)\n\n return plt.show()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndifferentiate a B - spline once and return the resulting coefficients and Bspline objects.", "response": "def __diff_internal(self):\n \"\"\"Differentiate a B-spline once, and return the resulting coefficients and Bspline objects.\n\nThis preserves the Bspline object nature of the data, enabling recursive implementation\nof higher-order differentiation (see `diff`).\n\nThe value of the first derivative of `B` at a point `x` can be obtained as::\n\n def diff1(B, x):\n terms = B.__diff_internal()\n return sum( ci*Bi(x) for ci,Bi in terms )\n\nReturns:\n tuple of tuples, where each item is (coefficient, Bspline object).\n\nSee:\n `diff`: differentiation of any order >= 0\n\"\"\"\n assert self.p > 0, \"order of Bspline must be > 0\" # we already handle the other case in diff()\n\n # https://www.cs.mtu.edu/~shene/COURSES/cs3621/NOTES/spline/B-spline/bspline-derv.html\n #\n t = self.knot_vector\n p = self.p\n Bi = Bspline( t[:-1], p-1 )\n Bip1 = Bspline( t[1:], p-1 )\n\n numer1 = +p\n numer2 = -p\n denom1 = t[p:-1] - t[:-(p+1)]\n denom2 = t[(p+1):] - t[1:-p]\n\n with np.errstate(divide='ignore', invalid='ignore'):\n ci = np.where(denom1 != 0., (numer1 / denom1), 0.)\n cip1 = np.where(denom2 != 0., (numer2 / denom2), 0.)\n\n return ( (ci,Bi), (cip1,Bip1) )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef diff(self, order=1):\n order = int(order)\n if order < 0:\n raise ValueError(\"order must be >= 0, got %d\" % (order))\n\n if order == 0:\n return self.__call__\n\n if order > self.p: # identically zero, but force the same output format as in the general case\n dummy = self.__call__(0.) # get number of basis functions and output dtype\n nbasis = dummy.shape[0]\n return lambda x: np.zeros( (nbasis,), dtype=dummy.dtype ) # accept but ignore input x\n\n # At each differentiation, each term maps into two new terms.\n # The number of terms in the result will be 2**order.\n #\n # This will cause an exponential explosion in the number of terms for high derivative orders,\n # but for the first few orders (practical usage; >3 is rarely needed) the approach works.\n #\n terms = [ (1.,self) ]\n for k in range(order):\n tmp = []\n for Ci,Bi in terms:\n tmp.extend( (Ci*cn, Bn) for cn,Bn in Bi.__diff_internal() ) # NOTE: also propagate Ci\n terms = tmp\n\n # perform final summation at call time\n return lambda x: sum( ci*Bi(x) for ci,Bi in terms )", "response": "Returns a function that evaluates the order - th derivative of a B - spline."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef collmat(self, tau, deriv_order=0):\n # get number of basis functions and output dtype\n dummy = self.__call__(0.)\n nbasis = dummy.shape[0]\n\n tau = np.atleast_1d(tau)\n if tau.ndim > 1:\n raise ValueError(\"tau must be a list or a rank-1 array\")\n\n A = np.empty( (tau.shape[0], nbasis), dtype=dummy.dtype )\n f = self.diff(order=deriv_order)\n for i,taui in enumerate(tau):\n A[i,:] = f(taui)\n\n return np.squeeze(A)", "response": "Compute collocation matrix for the given set of functions."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef build_interfaces_by_method(interfaces):\n def traverse(block, parts):\n try:\n method = block['method'].lower()\n except KeyError:\n for k, v in compat.iteritems(block):\n traverse(v, parts + [k])\n else:\n path = '.'.join(parts)\n try:\n methods[method]\n except KeyError:\n methods[method] = {}\n methods[method][path] = block\n methods = {}\n for key, val in compat.iteritems(interfaces):\n traverse(val, [key])\n return methods", "response": "Build a dictionary of methods that are used by the method."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_normalized_request_string(method, url, nonce, params, ext='', body_hash=None):\n urlparts = urlparse.urlparse(url)\n if urlparts.query:\n norm_url = '%s?%s' % (urlparts.path, urlparts.query)\n elif params:\n norm_url = '%s?%s' % (urlparts.path, get_normalized_params(params))\n else:\n norm_url = urlparts.path\n\n if not body_hash:\n body_hash = get_body_hash(params)\n\n port = urlparts.port\n if not port:\n assert urlparts.scheme in ('http', 'https')\n\n if urlparts.scheme == 'http':\n port = 80\n elif urlparts.scheme == 'https':\n port = 443\n\n output = [nonce, method.upper(), norm_url, urlparts.hostname, port, body_hash, ext, '']\n\n return '\\n'.join(map(str, output))", "response": "Returns a normalized request string as described in iN OAuth 2 MAC spec."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the body hash of the given params.", "response": "def get_body_hash(params):\n \"\"\"\n Returns BASE64 ( HASH (text) ) as described in OAuth2 MAC spec.\n\n http://tools.ietf.org/html/draft-ietf-oauth-v2-http-mac-00#section-3.2\n \"\"\"\n norm_params = get_normalized_params(params)\n\n return binascii.b2a_base64(hashlib.sha1(norm_params).digest())[:-1]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_mac_signature(api_secret, norm_request_string):\n hashed = hmac.new(str(api_secret), norm_request_string, hashlib.sha1)\n return binascii.b2a_base64(hashed.digest())[:-1]", "response": "Returns HMAC - SHA1 signature"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef open_file(self, access_mode=\"r\"):\n try:\n with open(self, access_mode, encoding='utf-8') as file:\n return file.read()\n\n except IOError:\n print(self + \" File not found.\")\n sys.exit(0)", "response": "open file and return contents."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _forward_backward(self, itraj):\n # get parameters\n A = self._hmm.transition_matrix\n pi = self._hmm.initial_distribution\n obs = self._observations[itraj]\n T = len(obs)\n # compute output probability matrix\n # t1 = time.time()\n self._hmm.output_model.p_obs(obs, out=self._pobs)\n # t2 = time.time()\n # self._fbtimings[0] += t2-t1\n # forward variables\n logprob = hidden.forward(A, self._pobs, pi, T=T, alpha_out=self._alpha)[0]\n # t3 = time.time()\n # self._fbtimings[1] += t3-t2\n # backward variables\n hidden.backward(A, self._pobs, T=T, beta_out=self._beta)\n # t4 = time.time()\n # self._fbtimings[2] += t4-t3\n # gamma\n hidden.state_probabilities(self._alpha, self._beta, T=T, gamma_out=self._gammas[itraj])\n # t5 = time.time()\n # self._fbtimings[3] += t5-t4\n # count matrix\n hidden.transition_counts(self._alpha, self._beta, A, self._pobs, T=T, out=self._Cs[itraj])\n # t6 = time.time()\n # self._fbtimings[4] += t6-t5\n # return results\n return logprob", "response": "This method computes the forward and backward steps for the internal state of the HMM."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the HMM model given the hidden state assignment and count matrices.", "response": "def _update_model(self, gammas, count_matrices, maxiter=10000000):\n \"\"\"\n Maximization step: Updates the HMM model given the hidden state assignment and count matrices\n\n Parameters\n ----------\n gamma : [ ndarray(T,N, dtype=float) ]\n list of state probabilities for each trajectory\n count_matrix : [ ndarray(N,N, dtype=float) ]\n list of the Baum-Welch transition count matrices for each hidden\n state trajectory\n maxiter : int\n maximum number of iterations of the transition matrix estimation if\n an iterative method is used.\n\n \"\"\"\n gamma0_sum = self._init_counts(gammas)\n C = self._transition_counts(count_matrices)\n logger().info(\"Initial count = \\n\"+str(gamma0_sum))\n logger().info(\"Count matrix = \\n\"+str(C))\n\n # compute new transition matrix\n from bhmm.estimators._tmatrix_disconnected import estimate_P, stationary_distribution\n T = estimate_P(C, reversible=self._hmm.is_reversible, fixed_statdist=self._fixed_stationary_distribution,\n maxiter=maxiter, maxerr=1e-12, mincount_connectivity=1e-16)\n # print 'P:\\n', T\n # estimate stationary or init distribution\n if self._stationary:\n if self._fixed_stationary_distribution is None:\n pi = stationary_distribution(T, C=C, mincount_connectivity=1e-16)\n else:\n pi = self._fixed_stationary_distribution\n else:\n if self._fixed_initial_distribution is None:\n pi = gamma0_sum / np.sum(gamma0_sum)\n else:\n pi = self._fixed_initial_distribution\n # print 'pi: ', pi, ' stationary = ', self._hmm.is_stationary\n\n # update model\n self._hmm.update(pi, T)\n\n logger().info(\"T: \\n\"+str(T))\n logger().info(\"pi: \\n\"+str(pi))\n\n # update output model\n self._hmm.output_model.estimate(self._observations, gammas)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compute_viterbi_paths(self):\n # get parameters\n K = len(self._observations)\n A = self._hmm.transition_matrix\n pi = self._hmm.initial_distribution\n\n # compute viterbi path for each trajectory\n paths = np.empty(K, dtype=object)\n for itraj in range(K):\n obs = self._observations[itraj]\n # compute output probability matrix\n pobs = self._hmm.output_model.p_obs(obs)\n # hidden path\n paths[itraj] = hidden.viterbi(A, pobs, pi)\n\n # done\n return paths", "response": "Computes the viterbi paths for the current trajectory and returns a numpy array of the paths."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfit the maximum likelihood estimation of the HMM using Baum - Welch algorithm.", "response": "def fit(self):\n \"\"\"\n Maximum-likelihood estimation of the HMM using the Baum-Welch algorithm\n\n Returns\n -------\n model : HMM\n The maximum likelihood HMM model.\n\n \"\"\"\n logger().info(\"=================================================================\")\n logger().info(\"Running Baum-Welch:\")\n logger().info(\" input observations: \"+str(self.nobservations)+\" of lengths \"+str(self.observation_lengths))\n logger().info(\" initial HMM guess:\"+str(self._hmm))\n\n initial_time = time.time()\n\n it = 0\n self._likelihoods = np.zeros(self.maxit)\n loglik = 0.0\n # flag if connectivity has changed (e.g. state lost) - in that case the likelihood\n # is discontinuous and can't be used as a convergence criterion in that iteration.\n tmatrix_nonzeros = self.hmm.transition_matrix.nonzero()\n converged = False\n\n while not converged and it < self.maxit:\n # self._fbtimings = np.zeros(5)\n t1 = time.time()\n loglik = 0.0\n for k in range(self._nobs):\n loglik += self._forward_backward(k)\n assert np.isfinite(loglik), it\n t2 = time.time()\n\n # convergence check\n if it > 0:\n dL = loglik - self._likelihoods[it-1]\n # print 'dL ', dL, 'iter_P ', maxiter_P\n if dL < self._accuracy:\n # print \"CONVERGED! Likelihood change = \",(loglik - self.likelihoods[it-1])\n converged = True\n\n # update model\n self._update_model(self._gammas, self._Cs, maxiter=self._maxit_P)\n t3 = time.time()\n\n # connectivity change check\n tmatrix_nonzeros_new = self.hmm.transition_matrix.nonzero()\n if not np.array_equal(tmatrix_nonzeros, tmatrix_nonzeros_new):\n converged = False # unset converged\n tmatrix_nonzeros = tmatrix_nonzeros_new\n\n # print 't_fb: ', str(1000.0*(t2-t1)), 't_up: ', str(1000.0*(t3-t2)), 'L = ', loglik, 'dL = ', (loglik - self._likelihoods[it-1])\n # print ' fb timings (ms): pobs', (1000.0*self._fbtimings).astype(int)\n\n logger().info(str(it) + \" ll = \" + str(loglik))\n # print self.model.output_model\n # print \"---------------------\"\n\n # end of iteration\n self._likelihoods[it] = loglik\n it += 1\n\n # final update with high precision\n # self._update_model(self._gammas, self._Cs, maxiter=10000000)\n\n # truncate likelihood history\n self._likelihoods = self._likelihoods[:it]\n # set final likelihood\n self._hmm.likelihood = loglik\n # set final count matrix\n self.count_matrix = self._transition_counts(self._Cs)\n self.initial_count = self._init_counts(self._gammas)\n\n final_time = time.time()\n elapsed_time = final_time - initial_time\n\n logger().info(\"maximum likelihood HMM:\"+str(self._hmm))\n logger().info(\"Elapsed time for Baum-Welch solution: %.3f s\" % elapsed_time)\n logger().info(\"Computing Viterbi path:\")\n\n initial_time = time.time()\n\n # Compute hidden state trajectories using the Viterbi algorithm.\n self._hmm.hidden_state_trajectories = self.compute_viterbi_paths()\n\n final_time = time.time()\n elapsed_time = final_time - initial_time\n\n logger().info(\"Elapsed time for Viterbi path computation: %.3f s\" % elapsed_time)\n logger().info(\"=================================================================\")\n\n return self._hmm"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sample_gaussian(mean, covar, covariance_type='diag', n_samples=1,\n random_state=None):\n \"\"\"Generate random samples from a Gaussian distribution.\n\n Parameters\n ----------\n mean : array_like, shape (n_features,)\n Mean of the distribution.\n\n covar : array_like, optional\n Covariance of the distribution. The shape depends on `covariance_type`:\n scalar if 'spherical',\n (n_features) if 'diag',\n (n_features, n_features) if 'tied', or 'full'\n\n covariance_type : string, optional\n Type of the covariance parameters. Must be one of\n 'spherical', 'tied', 'diag', 'full'. Defaults to 'diag'.\n\n n_samples : int, optional\n Number of samples to generate. Defaults to 1.\n\n Returns\n -------\n X : array, shape (n_features, n_samples)\n Randomly generated sample\n \"\"\"\n rng = check_random_state(random_state)\n n_dim = len(mean)\n rand = rng.randn(n_dim, n_samples)\n if n_samples == 1:\n rand.shape = (n_dim,)\n\n if covariance_type == 'spherical':\n rand *= np.sqrt(covar)\n elif covariance_type == 'diag':\n rand = np.dot(np.diag(np.sqrt(covar)), rand)\n else:\n s, U = linalg.eigh(covar)\n s.clip(0, out=s) # get rid of tiny negatives\n np.sqrt(s, out=s)\n U *= s\n rand = np.dot(U, rand)\n\n return (rand.T + mean).T", "response": "Generate random samples from a Gaussian distribution."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nperform the covariance M step for diagonal cases", "response": "def _covar_mstep_diag(gmm, X, responsibilities, weighted_X_sum, norm,\n min_covar):\n \"\"\"Performing the covariance M step for diagonal cases\"\"\"\n avg_X2 = np.dot(responsibilities.T, X * X) * norm\n avg_means2 = gmm.means_ ** 2\n avg_X_means = gmm.means_ * weighted_X_sum * norm\n return avg_X2 - 2 * avg_X_means + avg_means2 + min_covar"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nperforms the covariance M step for spherical cases", "response": "def _covar_mstep_spherical(*args):\n \"\"\"Performing the covariance M step for spherical cases\"\"\"\n cv = _covar_mstep_diag(*args)\n return np.tile(cv.mean(axis=1)[:, np.newaxis], (1, cv.shape[1]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nperforming the covariance M step for full cases.", "response": "def _covar_mstep_full(gmm, X, responsibilities, weighted_X_sum, norm,\n min_covar):\n \"\"\"Performing the covariance M step for full cases\"\"\"\n # Eq. 12 from K. Murphy, \"Fitting a Conditional Linear Gaussian\n # Distribution\"\n n_features = X.shape[1]\n cv = np.empty((gmm.n_components, n_features, n_features))\n for c in range(gmm.n_components):\n post = responsibilities[:, c]\n mu = gmm.means_[c]\n diff = X - mu\n with np.errstate(under='ignore'):\n # Underflow Errors in doing post * X.T are not important\n avg_cv = np.dot(post * diff.T, diff) / (post.sum() + 10 * EPS)\n cv[c] = avg_cv + min_covar * np.eye(n_features)\n return cv"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the covariance parameters for each mixture component.", "response": "def _get_covars(self):\n \"\"\"Covariance parameters for each mixture component.\n The shape depends on `cvtype`::\n\n (`n_states`, 'n_features') if 'spherical',\n (`n_features`, `n_features`) if 'tied',\n (`n_states`, `n_features`) if 'diag',\n (`n_states`, `n_features`, `n_features`) if 'full'\n \"\"\"\n if self.covariance_type == 'full':\n return self.covars_\n elif self.covariance_type == 'diag':\n return [np.diag(cov) for cov in self.covars_]\n elif self.covariance_type == 'tied':\n return [self.covars_] * self.n_components\n elif self.covariance_type == 'spherical':\n return [np.diag(cov) for cov in self.covars_]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nproviding values for covariance", "response": "def _set_covars(self, covars):\n \"\"\"Provide values for covariance\"\"\"\n covars = np.asarray(covars)\n _validate_covars(covars, self.covariance_type, self.n_components)\n self.covars_ = covars"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the log probability of each data point in X and return the posterior distribution of each mixture component of each element of X.", "response": "def score_samples(self, X):\n \"\"\"Return the per-sample likelihood of the data under the model.\n\n Compute the log probability of X under the model and\n return the posterior distribution (responsibilities) of each\n mixture component for each element of X.\n\n Parameters\n ----------\n X: array_like, shape (n_samples, n_features)\n List of n_features-dimensional data points. Each row\n corresponds to a single data point.\n\n Returns\n -------\n logprob : array_like, shape (n_samples,)\n Log probabilities of each data point in X.\n\n responsibilities : array_like, shape (n_samples, n_components)\n Posterior probabilities of each mixture component for each\n observation\n \"\"\"\n check_is_fitted(self, 'means_')\n\n X = check_array(X)\n if X.ndim == 1:\n X = X[:, np.newaxis]\n if X.size == 0:\n return np.array([]), np.empty((0, self.n_components))\n if X.shape[1] != self.means_.shape[1]:\n raise ValueError('The shape of X is not compatible with self')\n\n lpr = (log_multivariate_normal_density(X, self.means_, self.covars_,\n self.covariance_type)\n + np.log(self.weights_))\n logprob = logsumexp(lpr, axis=1)\n responsibilities = np.exp(lpr - logprob[:, np.newaxis])\n return logprob, responsibilities"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef score(self, X, y=None):\n logprob, _ = self.score_samples(X)\n return logprob", "response": "Compute the log probability under the model."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\npredicts label for data.", "response": "def predict(self, X):\n \"\"\"Predict label for data.\n\n Parameters\n ----------\n X : array-like, shape = [n_samples, n_features]\n\n Returns\n -------\n C : array, shape = (n_samples,)\n \"\"\"\n logprob, responsibilities = self.score_samples(X)\n return responsibilities.argmax(axis=1)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates random samples from the model.", "response": "def sample(self, n_samples=1, random_state=None):\n \"\"\"Generate random samples from the model.\n\n Parameters\n ----------\n n_samples : int, optional\n Number of samples to generate. Defaults to 1.\n\n Returns\n -------\n X : array_like, shape (n_samples, n_features)\n List of samples\n \"\"\"\n check_is_fitted(self, 'means_')\n\n if random_state is None:\n random_state = self.random_state\n random_state = check_random_state(random_state)\n weight_cdf = np.cumsum(self.weights_)\n\n X = np.empty((n_samples, self.means_.shape[1]))\n rand = random_state.rand(n_samples)\n # decide which component to use for each sample\n comps = weight_cdf.searchsorted(rand)\n # for each component, generate all needed samples\n for comp in range(self.n_components):\n # occurrences of current component in X\n comp_in_X = (comp == comps)\n # number of those occurrences\n num_comp_in_X = comp_in_X.sum()\n if num_comp_in_X > 0:\n if self.covariance_type == 'tied':\n cv = self.covars_\n elif self.covariance_type == 'spherical':\n cv = self.covars_[comp][0]\n else:\n cv = self.covars_[comp]\n X[comp_in_X] = sample_gaussian(\n self.means_[comp], cv, self.covariance_type,\n num_comp_in_X, random_state=random_state).T\n return X"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nestimate model parameters with the expectation-maximization algorithm. A initialization step is performed before entering the em algorithm. If you want to avoid this step, set the keyword argument init_params to the empty string '' when creating the GMM object. Likewise, if you would like just to do an initialization, set n_iter=0. Parameters ---------- X : array_like, shape (n, n_features) List of n_features-dimensional data points. Each row corresponds to a single data point.", "response": "def fit(self, X, y=None):\n \"\"\"Estimate model parameters with the expectation-maximization\n algorithm.\n\n A initialization step is performed before entering the em\n algorithm. If you want to avoid this step, set the keyword\n argument init_params to the empty string '' when creating the\n GMM object. Likewise, if you would like just to do an\n initialization, set n_iter=0.\n\n Parameters\n ----------\n X : array_like, shape (n, n_features)\n List of n_features-dimensional data points. Each row\n corresponds to a single data point.\n \"\"\"\n # initialization step\n X = check_array(X, dtype=np.float64)\n if X.shape[0] < self.n_components:\n raise ValueError(\n 'GMM estimation with %s components, but got only %s samples' %\n (self.n_components, X.shape[0]))\n\n max_log_prob = -np.infty\n\n for _ in range(self.n_init):\n if 'm' in self.init_params or not hasattr(self, 'means_'):\n if np.issubdtype(X.dtype, np.float32):\n from bhmm._external.clustering.kmeans_clustering_32 import init_centers\n elif np.issubdtype(X.dtype, np.float64):\n from bhmm._external.clustering.kmeans_clustering_64 import init_centers\n else:\n raise ValueError(\"Could not handle dtype %s for clustering!\" % X.dtype)\n centers = init_centers(X, 'euclidean', self.n_components)\n self.means_ = centers\n\n if 'w' in self.init_params or not hasattr(self, 'weights_'):\n self.weights_ = np.tile(1.0 / self.n_components,\n self.n_components)\n\n if 'c' in self.init_params or not hasattr(self, 'covars_'):\n cv = np.cov(X.T) + self.min_covar * np.eye(X.shape[1])\n if not cv.shape:\n cv.shape = (1, 1)\n self.covars_ = \\\n distribute_covar_matrix_to_match_covariance_type(\n cv, self.covariance_type, self.n_components)\n\n # EM algorithms\n current_log_likelihood = None\n # reset self.converged_ to False\n self.converged_ = False\n\n # this line should be removed when 'thresh' is removed in v0.18\n tol = (self.tol if self.thresh is None\n else self.thresh / float(X.shape[0]))\n\n for i in range(self.n_iter):\n prev_log_likelihood = current_log_likelihood\n # Expectation step\n log_likelihoods, responsibilities = self.score_samples(X)\n current_log_likelihood = log_likelihoods.mean()\n\n # Check for convergence.\n # (should compare to self.tol when dreprecated 'thresh' is\n # removed in v0.18)\n if prev_log_likelihood is not None:\n change = abs(current_log_likelihood - prev_log_likelihood)\n if change < tol:\n self.converged_ = True\n break\n\n # Maximization step\n self._do_mstep(X, responsibilities, self.params,\n self.min_covar)\n\n # if the results are better, keep it\n if self.n_iter:\n if current_log_likelihood > max_log_prob:\n max_log_prob = current_log_likelihood\n best_params = {'weights': self.weights_,\n 'means': self.means_,\n 'covars': self.covars_}\n # check the existence of an init param that was not subject to\n # likelihood computation issue.\n if np.isneginf(max_log_prob) and self.n_iter:\n raise RuntimeError(\n \"EM algorithm was never able to compute a valid likelihood \" +\n \"given initial parameters. Try different init parameters \" +\n \"(or increasing n_init) or check for degenerate data.\")\n # self.n_iter == 0 occurs when using GMM within HMM\n if self.n_iter:\n self.covars_ = best_params['covars']\n self.means_ = best_params['means']\n self.weights_ = best_params['weights']\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nperform the Mstep of the EM algorithm and return the class weihgts.", "response": "def _do_mstep(self, X, responsibilities, params, min_covar=0):\n \"\"\" Perform the Mstep of the EM algorithm and return the class weihgts.\n \"\"\"\n weights = responsibilities.sum(axis=0)\n weighted_X_sum = np.dot(responsibilities.T, X)\n inverse_weights = 1.0 / (weights[:, np.newaxis] + 10 * EPS)\n\n if 'w' in params:\n self.weights_ = (weights / (weights.sum() + 10 * EPS) + EPS)\n if 'm' in params:\n self.means_ = weighted_X_sum * inverse_weights\n if 'c' in params:\n covar_mstep_func = _covar_mstep_funcs[self.covariance_type]\n self.covars_ = covar_mstep_func(\n self, X, responsibilities, weighted_X_sum, inverse_weights,\n min_covar)\n return weights"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _n_parameters(self):\n ndim = self.means_.shape[1]\n if self.covariance_type == 'full':\n cov_params = self.n_components * ndim * (ndim + 1) / 2.\n elif self.covariance_type == 'diag':\n cov_params = self.n_components * ndim\n elif self.covariance_type == 'tied':\n cov_params = ndim * (ndim + 1) / 2.\n elif self.covariance_type == 'spherical':\n cov_params = self.n_components\n mean_params = ndim * self.n_components\n return int(cov_params + mean_params + self.n_components - 1)", "response": "Return the number of free parameters in the model."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_param_names(cls):\n # fetch the constructor or the original constructor before\n # deprecation wrapping if any\n init = getattr(cls.__init__, 'deprecated_original', cls.__init__)\n if init is object.__init__:\n # No explicit constructor to introspect\n return []\n\n # introspect the constructor arguments to find the model parameters\n # to represent\n args, varargs, kw, default = inspect.getargspec(init)\n if varargs is not None:\n raise RuntimeError(\"scikit-learn estimators should always \"\n \"specify their parameters in the signature\"\n \" of their __init__ (no varargs).\"\n \" %s doesn't follow this convention.\"\n % (cls, ))\n # Remove 'self'\n # XXX: This is going to fail if the init is a staticmethod, but\n # who would do this?\n args.pop(0)\n args.sort()\n return args", "response": "Get the parameter names for the estimator"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the parameters of this object.", "response": "def set_params(self, **params):\n \"\"\"Set the parameters of this estimator.\n\n The method works on simple estimators as well as on nested objects\n (such as pipelines). The former have parameters of the form\n ``__`` so that it's possible to update each\n component of a nested object.\n\n Returns\n -------\n self\n \"\"\"\n if not params:\n # Simple optimisation to gain speed (inspect is slow)\n return self\n valid_params = self.get_params(deep=True)\n for key, value in six.iteritems(params):\n split = key.split('__', 1)\n if len(split) > 1:\n # nested objects case\n name, sub_name = split\n if not name in valid_params:\n raise ValueError('Invalid parameter %s for estimator %s' %\n (name, self))\n sub_object = valid_params[name]\n sub_object.set_params(**{sub_name: value})\n else:\n # simple objects case\n if not key in valid_params:\n raise ValueError('Invalid parameter %s ' 'for estimator %s'\n % (key, self.__class__.__name__))\n setattr(self, key, value)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef init_model_gaussian1d(observations, nstates, reversible=True):\n ntrajectories = len(observations)\n\n # Concatenate all observations.\n collected_observations = np.array([], dtype=config.dtype)\n for o_t in observations:\n collected_observations = np.append(collected_observations, o_t)\n\n # Fit a Gaussian mixture model to obtain emission distributions and state stationary probabilities.\n from bhmm._external.sklearn import mixture\n gmm = mixture.GMM(n_components=nstates)\n gmm.fit(collected_observations[:,None])\n from bhmm import GaussianOutputModel\n output_model = GaussianOutputModel(nstates, means=gmm.means_[:,0], sigmas=np.sqrt(gmm.covars_[:,0]))\n\n logger().info(\"Gaussian output model:\\n\"+str(output_model))\n\n # Extract stationary distributions.\n Pi = np.zeros([nstates], np.float64)\n Pi[:] = gmm.weights_[:]\n\n logger().info(\"GMM weights: %s\" % str(gmm.weights_))\n\n # Compute fractional state memberships.\n Nij = np.zeros([nstates, nstates], np.float64)\n for o_t in observations:\n # length of trajectory\n T = o_t.shape[0]\n # output probability\n pobs = output_model.p_obs(o_t)\n # normalize\n pobs /= pobs.sum(axis=1)[:,None]\n # Accumulate fractional transition counts from this trajectory.\n for t in range(T-1):\n Nij[:,:] = Nij[:,:] + np.outer(pobs[t,:], pobs[t+1,:])\n\n logger().info(\"Nij\\n\"+str(Nij))\n\n # Compute transition matrix maximum likelihood estimate.\n import msmtools.estimation as msmest\n import msmtools.analysis as msmana\n Tij = msmest.transition_matrix(Nij, reversible=reversible)\n pi = msmana.stationary_distribution(Tij)\n\n # Update model.\n model = HMM(pi, Tij, output_model)\n\n return model", "response": "Generate an initial model with 1D - Gaussian output densities for a set of observations."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _p_o(self, o):\n if self.__impl__ == self.__IMPL_C__:\n return gc.p_o(o, self.means, self.sigmas, out=None, dtype=type(o))\n elif self.__impl__ == self.__IMPL_PYTHON__:\n if np.any(self.sigmas < np.finfo(self.sigmas.dtype).eps):\n raise RuntimeError('at least one sigma is too small to continue.')\n C = 1.0 / (np.sqrt(2.0 * np.pi) * self.sigmas)\n Pobs = C * np.exp(-0.5 * ((o-self.means)/self.sigmas)**2)\n return Pobs\n else:\n raise RuntimeError('Implementation '+str(self.__impl__)+' not available')", "response": "Returns the output probability density of a single observation from all hidden states."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef p_obs(self, obs, out=None):\n if self.__impl__ == self.__IMPL_C__:\n res = gc.p_obs(obs, self.means, self.sigmas, out=out, dtype=config.dtype)\n return self._handle_outliers(res)\n elif self.__impl__ == self.__IMPL_PYTHON__:\n T = len(obs)\n if out is None:\n res = np.zeros((T, self.nstates), dtype=config.dtype)\n else:\n res = out\n for t in range(T):\n res[t, :] = self._p_o(obs[t])\n return self._handle_outliers(res)\n else:\n raise RuntimeError('Implementation '+str(self.__impl__)+' not available')", "response": "Returns the output probabilities for an observation model and synthetic observation trajectory."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfit the output model given the observations and weights.", "response": "def estimate(self, observations, weights):\n \"\"\"\n Fits the output model given the observations and weights\n\n Parameters\n ----------\n observations : [ ndarray(T_k,) ] with K elements\n A list of K observation trajectories, each having length T_k and d dimensions\n weights : [ ndarray(T_k,nstates) ] with K elements\n A list of K weight matrices, each having length T_k\n weights[k][t,n] is the weight assignment from observations[k][t] to state index n\n\n Examples\n --------\n\n Generate an observation model and samples from each state.\n\n >>> ntrajectories = 3\n >>> nobs = 1000\n >>> output_model = GaussianOutputModel(nstates=3, means=[-1, 0, +1], sigmas=[0.5, 1, 2])\n >>> observations = [ np.random.randn(nobs) for _ in range(ntrajectories) ] # random observations\n >>> weights = [ np.random.dirichlet([2, 3, 4], size=nobs) for _ in range(ntrajectories) ] # random weights\n\n Update the observation model parameters my a maximum-likelihood fit.\n\n >>> output_model.estimate(observations, weights)\n\n \"\"\"\n # sizes\n N = self.nstates\n K = len(observations)\n\n # fit means\n self._means = np.zeros(N)\n w_sum = np.zeros(N)\n for k in range(K):\n # update nominator\n for i in range(N):\n self.means[i] += np.dot(weights[k][:, i], observations[k])\n # update denominator\n w_sum += np.sum(weights[k], axis=0)\n # normalize\n self._means /= w_sum\n\n # fit variances\n self._sigmas = np.zeros(N)\n w_sum = np.zeros(N)\n for k in range(K):\n # update nominator\n for i in range(N):\n Y = (observations[k] - self.means[i])**2\n self.sigmas[i] += np.dot(weights[k][:, i], Y)\n # update denominator\n w_sum += np.sum(weights[k], axis=0)\n # normalize\n self._sigmas /= w_sum\n self._sigmas = np.sqrt(self.sigmas)\n if np.any(self._sigmas < np.finfo(self._sigmas.dtype).eps):\n raise RuntimeError('at least one sigma is too small to continue.')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sample(self, observations, prior=None):\n for state_index in range(self.nstates):\n # Update state emission distribution parameters.\n\n observations_in_state = observations[state_index]\n # Determine number of samples in this state.\n nsamples_in_state = len(observations_in_state)\n\n # Skip update if no observations.\n if nsamples_in_state == 0:\n logger().warn('Warning: State %d has no observations.' % state_index)\n if nsamples_in_state > 0: # Sample new mu.\n self.means[state_index] = np.random.randn()*self.sigmas[state_index]/np.sqrt(nsamples_in_state) + np.mean(observations_in_state)\n if nsamples_in_state > 1: # Sample new sigma\n # This scheme uses the improper Jeffreys prior on sigma^2, P(mu, sigma^2) \\propto 1/sigma\n chisquared = np.random.chisquare(nsamples_in_state-1)\n sigmahat2 = np.mean((observations_in_state - self.means[state_index])**2)\n self.sigmas[state_index] = np.sqrt(sigmahat2) / np.sqrt(chisquared / nsamples_in_state)\n\n return", "response": "Sample a new set of distribution parameters given a sample of observations from the given state."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a synthetic observation data from a given state.", "response": "def generate_observation_from_state(self, state_index):\n \"\"\"\n Generate a single synthetic observation data from a given state.\n\n Parameters\n ----------\n state_index : int\n Index of the state from which observations are to be generated.\n\n Returns\n -------\n observation : float\n A single observation from the given state.\n\n Examples\n --------\n\n Generate an observation model.\n\n >>> output_model = GaussianOutputModel(nstates=2, means=[0, 1], sigmas=[1, 2])\n\n Generate sample from a state.\n\n >>> observation = output_model.generate_observation_from_state(0)\n\n \"\"\"\n observation = self.sigmas[state_index] * np.random.randn() + self.means[state_index]\n return observation"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates synthetic observation data from a given state.", "response": "def generate_observations_from_state(self, state_index, nobs):\n \"\"\"\n Generate synthetic observation data from a given state.\n\n Parameters\n ----------\n state_index : int\n Index of the state from which observations are to be generated.\n nobs : int\n The number of observations to generate.\n\n Returns\n -------\n observations : numpy.array of shape(nobs,)\n A sample of `nobs` observations from the specified state.\n\n Examples\n --------\n\n Generate an observation model.\n\n >>> output_model = GaussianOutputModel(nstates=2, means=[0, 1], sigmas=[1, 2])\n\n Generate samples from each state.\n\n >>> observations = [ output_model.generate_observations_from_state(state_index, nobs=100) for state_index in range(output_model.nstates) ]\n\n \"\"\"\n observations = self.sigmas[state_index] * np.random.randn(nobs) + self.means[state_index]\n return observations"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate synthetic observation data from a given state sequence.", "response": "def generate_observation_trajectory(self, s_t):\n \"\"\"\n Generate synthetic observation data from a given state sequence.\n\n Parameters\n ----------\n s_t : numpy.array with shape (T,) of int type\n s_t[t] is the hidden state sampled at time t\n\n Returns\n -------\n o_t : numpy.array with shape (T,) of type dtype\n o_t[t] is the observation associated with state s_t[t]\n\n Examples\n --------\n\n Generate an observation model and synthetic state trajectory.\n\n >>> nobs = 1000\n >>> output_model = GaussianOutputModel(nstates=3, means=[-1, 0, +1], sigmas=[0.5, 1, 2])\n >>> s_t = np.random.randint(0, output_model.nstates, size=[nobs])\n\n Generate a synthetic trajectory\n\n >>> o_t = output_model.generate_observation_trajectory(s_t)\n\n \"\"\"\n\n # Determine number of samples to generate.\n T = s_t.shape[0]\n\n o_t = np.zeros([T], dtype=config.dtype)\n for t in range(T):\n s = s_t[t]\n o_t[t] = self.sigmas[s] * np.random.randn() + self.means[s]\n return o_t"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef initial_distribution_samples(self):\n res = np.empty((self.nsamples, self.nstates), dtype=config.dtype)\n for i in range(self.nsamples):\n res[i, :] = self._sampled_hmms[i].stationary_distribution\n return res", "response": "r Returns the samples of the initial distribution"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef lifetimes_samples(self):\n res = np.empty((self.nsamples, self.nstates), dtype=config.dtype)\n for i in range(self.nsamples):\n res[i, :] = self._sampled_hmms[i].lifetimes\n return res", "response": "r Returns a numpy array of the lifetimes of the timescales"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the implementation of this module.", "response": "def set_implementation(self, impl):\n \"\"\"\n Sets the implementation of this module\n\n Parameters\n ----------\n impl : str\n One of [\"python\", \"c\"]\n\n \"\"\"\n if impl.lower() == 'python':\n self.__impl__ = self.__IMPL_PYTHON__\n elif impl.lower() == 'c':\n self.__impl__ = self.__IMPL_C__\n else:\n import warnings\n warnings.warn('Implementation '+impl+' is not known. Using the fallback python implementation.')\n self.__impl__ = self.__IMPL_PYTHON__"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef log_p_obs(self, obs, out=None, dtype=np.float32):\n if out is None:\n return np.log(self.p_obs(obs))\n else:\n self.p_obs(obs, out=out, dtype=dtype)\n np.log(out, out=out)\n return out", "response": "Returns the element - wise logarithm of the output probabilities for an entire trajectory and all hidden states."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _handle_outliers(self, p_o):\n if self.ignore_outliers:\n outliers = np.where(p_o.sum(axis=1)==0)[0]\n if outliers.size > 0:\n p_o[outliers, :] = 1.0\n self.found_outliers = True\n return p_o", "response": "Sets observation probabilities of outliers to uniform if ignore_outliers is set."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the forward term of the given sequence A and B.", "response": "def forward(A, pobs, pi, T=None, alpha_out=None, dtype=np.float32):\n \"\"\"Compute P( obs | A, B, pi ) and all forward coefficients.\n\n Parameters\n ----------\n A : ndarray((N,N), dtype = float)\n transition matrix of the hidden states\n pobs : ndarray((T,N), dtype = float)\n pobs[t,i] is the observation probability for observation at time t given hidden state i\n pi : ndarray((N), dtype = float)\n initial distribution of hidden states\n T : int, optional, default = None\n trajectory length. If not given, T = pobs.shape[0] will be used.\n alpha_out : ndarray((T,N), dtype = float), optional, default = None\n container for the alpha result variables. If None, a new container will be created.\n dtype : type, optional, default = np.float32\n data type of the result.\n\n Returns\n -------\n logprob : float\n The probability to observe the sequence `ob` with the model given\n by `A`, `B` and `pi`.\n alpha : ndarray((T,N), dtype = float), optional, default = None\n alpha[t,i] is the ith forward coefficient of time t. These can be\n used in many different algorithms related to HMMs.\n\n \"\"\"\n # set T\n if T is None:\n T = pobs.shape[0] # if not set, use the length of pobs as trajectory length\n elif T > pobs.shape[0]:\n raise ValueError('T must be at most the length of pobs.')\n # set N\n N = A.shape[0]\n # initialize output if necessary\n if alpha_out is None:\n alpha_out = np.zeros((T, N), dtype=dtype)\n elif T > alpha_out.shape[0]:\n raise ValueError('alpha_out must at least have length T in order to fit trajectory.')\n # log-likelihood\n logprob = 0.0\n\n # initial values\n # alpha_i(0) = pi_i * B_i,ob[0]\n np.multiply(pi, pobs[0, :], out=alpha_out[0])\n # scaling factor\n scale = np.sum(alpha_out[0, :])\n # scale\n alpha_out[0, :] /= scale\n logprob += np.log(scale)\n\n # induction\n for t in range(T-1):\n # alpha_j(t+1) = sum_i alpha_i(t) * A_i,j * B_j,ob(t+1)\n np.multiply(np.dot(alpha_out[t, :], A), pobs[t+1, :], out=alpha_out[t+1])\n # scaling factor\n scale = np.sum(alpha_out[t+1, :])\n # scale\n alpha_out[t+1, :] /= scale\n # update logprob\n logprob += np.log(scale)\n\n return logprob, alpha_out"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes all backward coefficients of the hidden states A and pobs.", "response": "def backward(A, pobs, T=None, beta_out=None, dtype=np.float32):\n \"\"\"Compute all backward coefficients. With scaling!\n\n Parameters\n ----------\n A : ndarray((N,N), dtype = float)\n transition matrix of the hidden states\n pobs : ndarray((T,N), dtype = float)\n pobs[t,i] is the observation probability for observation at time t given hidden state i\n beta_out : ndarray((T,N), dtype = float), optional, default = None\n containter for the beta result variables. If None, a new container will be created.\n dtype : type, optional, default = np.float32\n data type of the result.\n\n Returns\n -------\n beta : ndarray((T,N), dtype = float), optional, default = None\n beta[t,i] is the ith backward coefficient of time t. These can be\n used in many different algorithms related to HMMs.\n\n \"\"\"\n # set T\n if T is None:\n T = pobs.shape[0] # if not set, use the length of pobs as trajectory length\n elif T > pobs.shape[0]:\n raise ValueError('T must be at most the length of pobs.')\n # set N\n N = A.shape[0]\n # initialize output if necessary\n if beta_out is None:\n beta_out = np.zeros((T, N), dtype=dtype)\n elif T > beta_out.shape[0]:\n raise ValueError('beta_out must at least have length T in order to fit trajectory.')\n\n # initialization\n beta_out[T-1, :] = 1.0\n # scaling factor\n scale = np.sum(beta_out[T-1, :])\n # scale\n beta_out[T-1, :] /= scale\n\n # induction\n for t in range(T-2, -1, -1):\n # beta_i(t) = sum_j A_i,j * beta_j(t+1) * B_j,ob(t+1)\n np.dot(A, beta_out[t+1, :] * pobs[t+1, :], out=beta_out[t, :])\n # scaling factor\n scale = np.sum(beta_out[t, :])\n # scale\n beta_out[t, :] /= scale\n return beta_out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transition_counts(alpha, beta, A, pobs, T=None, out=None, dtype=np.float32):\n # set T\n if T is None:\n T = pobs.shape[0] # if not set, use the length of pobs as trajectory length\n elif T > pobs.shape[0]:\n raise ValueError('T must be at most the length of pobs.')\n # set N\n N = len(A)\n # output\n if out is None:\n out = np.zeros((N, N), dtype=dtype, order='C')\n else:\n out[:] = 0.0\n # compute transition counts\n xi = np.zeros((N, N), dtype=dtype, order='C')\n for t in range(T-1):\n # xi_i,j(t) = alpha_i(t) * A_i,j * B_j,ob(t+1) * beta_j(t+1)\n np.dot(alpha[t, :][:, None] * A, np.diag(pobs[t+1, :] * beta[t+1, :]), out=xi)\n # normalize to 1 for each time step\n xi /= np.sum(xi)\n # add to counts\n np.add(out, xi, out)\n # return\n return out", "response": "Sum for all t the probability to transition from state i to state j."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nestimating the hidden pathway of maximum likelihood using the Viterbi algorithm.", "response": "def viterbi(A, pobs, pi, dtype=np.float32):\n \"\"\" Estimate the hidden pathway of maximum likelihood using the Viterbi algorithm.\n\n Parameters\n ----------\n A : ndarray((N,N), dtype = float)\n transition matrix of the hidden states\n pobs : ndarray((T,N), dtype = float)\n pobs[t,i] is the observation probability for observation at time t given hidden state i\n pi : ndarray((N), dtype = float)\n initial distribution of hidden states\n\n Returns\n -------\n q : numpy.array shape (T)\n maximum likelihood hidden path\n\n \"\"\"\n T, N = pobs.shape[0], pobs.shape[1]\n # temporary viterbi state\n psi = np.zeros((T, N), dtype=int)\n # initialize\n v = pi * pobs[0, :]\n # rescale\n v /= v.sum()\n psi[0] = 0.0\n # iterate\n for t in range(1, T):\n vA = np.dot(np.diag(v), A)\n # propagate v\n v = pobs[t, :] * np.max(vA, axis=0)\n # rescale\n v /= v.sum()\n psi[t] = np.argmax(vA, axis=0)\n # iterate\n q = np.zeros(T, dtype=int)\n q[T-1] = np.argmax(v)\n for t in range(T-2, -1, -1):\n q[t] = psi[t+1, q[t+1]]\n # done\n return q"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsamples from the path of the hidden state.", "response": "def sample_path(alpha, A, pobs, T=None, dtype=np.float32):\n \"\"\"\n alpha : ndarray((T,N), dtype = float), optional, default = None\n alpha[t,i] is the ith forward coefficient of time t.\n beta : ndarray((T,N), dtype = float), optional, default = None\n beta[t,i] is the ith forward coefficient of time t.\n A : ndarray((N,N), dtype = float)\n transition matrix of the hidden states\n pobs : ndarray((T,N), dtype = float)\n pobs[t,i] is the observation probability for observation at time t given hidden state i\n \"\"\"\n N = pobs.shape[1]\n # set T\n if T is None:\n T = pobs.shape[0] # if not set, use the length of pobs as trajectory length\n elif T > pobs.shape[0] or T > alpha.shape[0]:\n raise ValueError('T must be at most the length of pobs and alpha.')\n\n # initialize path\n S = np.zeros(T, dtype=int)\n\n # Sample final state.\n psel = alpha[T-1, :]\n psel /= psel.sum() # make sure it's normalized\n # Draw from this distribution.\n S[T-1] = np.random.choice(range(N), size=1, p=psel)\n\n # Work backwards from T-2 to 0.\n for t in range(T-2, -1, -1):\n # Compute P(s_t = i | s_{t+1}..s_T).\n psel = alpha[t, :] * A[:, S[t+1]]\n psel /= psel.sum() # make sure it's normalized\n # Draw from this distribution.\n S[t] = np.random.choice(range(N), size=1, p=psel)\n\n return S"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef coarse_grain_transition_matrix(P, M):\n # coarse-grain matrix: Pc = (M' M)^-1 M' P M\n W = np.linalg.inv(np.dot(M.T, M))\n A = np.dot(np.dot(M.T, P), M)\n P_coarse = np.dot(W, A)\n\n # this coarse-graining can lead to negative elements. Setting them to zero here.\n P_coarse = np.maximum(P_coarse, 0)\n # and renormalize\n P_coarse /= P_coarse.sum(axis=1)[:, None]\n\n return P_coarse", "response": "Coarse grain transition matrix P using memberships M"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nregularizes the hidden initial distribution and transition matrix P.", "response": "def regularize_hidden(p0, P, reversible=True, stationary=False, C=None, eps=None):\n \"\"\" Regularizes the hidden initial distribution and transition matrix.\n\n Makes sure that the hidden initial distribution and transition matrix have\n nonzero probabilities by setting them to eps and then renormalizing.\n Avoids zeros that would cause estimation algorithms to crash or get stuck\n in suboptimal states.\n\n Parameters\n ----------\n p0 : ndarray(n)\n Initial hidden distribution of the HMM\n P : ndarray(n, n)\n Hidden transition matrix\n reversible : bool\n HMM is reversible. Will make sure it is still reversible after modification.\n stationary : bool\n p0 is the stationary distribution of P. In this case, will not regularize\n p0 separately. If stationary=False, the regularization will be applied to p0.\n C : ndarray(n, n)\n Hidden count matrix. Only needed for stationary=True and P disconnected.\n epsilon : float or None\n minimum value of the resulting transition matrix. Default: evaluates\n to 0.01 / n. The coarse-graining equation can lead to negative elements\n and thus epsilon should be set to at least 0. Positive settings of epsilon\n are similar to a prior and enforce minimum positive values for all\n transition probabilities.\n\n Return\n ------\n p0 : ndarray(n)\n regularized initial distribution\n P : ndarray(n, n)\n regularized transition matrix\n\n \"\"\"\n # input\n n = P.shape[0]\n if eps is None: # default output probability, in order to avoid zero columns\n eps = 0.01 / n\n\n # REGULARIZE P\n P = np.maximum(P, eps)\n # and renormalize\n P /= P.sum(axis=1)[:, None]\n # ensure reversibility\n if reversible:\n P = _tmatrix_disconnected.enforce_reversible_on_closed(P)\n\n # REGULARIZE p0\n if stationary:\n _tmatrix_disconnected.stationary_distribution(P, C=C)\n else:\n p0 = np.maximum(p0, eps)\n p0 /= p0.sum()\n\n return p0, P"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nregularizes the output probabilities of the given set of observed states.", "response": "def regularize_pobs(B, nonempty=None, separate=None, eps=None):\n \"\"\" Regularizes the output probabilities.\n\n Makes sure that the output probability distributions has\n nonzero probabilities by setting them to eps and then renormalizing.\n Avoids zeros that would cause estimation algorithms to crash or get stuck\n in suboptimal states.\n\n Parameters\n ----------\n B : ndarray(n, m)\n HMM output probabilities\n nonempty : None or iterable of int\n Nonempty set. Only regularize on this subset.\n separate : None or iterable of int\n Force the given set of observed states to stay in a separate hidden state.\n The remaining nstates-1 states will be assigned by a metastable decomposition.\n reversible : bool\n HMM is reversible. Will make sure it is still reversible after modification.\n\n Returns\n -------\n B : ndarray(n, m)\n Regularized output probabilities\n\n \"\"\"\n # input\n B = B.copy() # modify copy\n n, m = B.shape # number of hidden / observable states\n if eps is None: # default output probability, in order to avoid zero columns\n eps = 0.01 / m\n # observable sets\n if nonempty is None:\n nonempty = np.arange(m)\n\n if separate is None:\n B[:, nonempty] = np.maximum(B[:, nonempty], eps)\n else:\n nonempty_nonseparate = np.array(list(set(nonempty) - set(separate)), dtype=int)\n nonempty_separate = np.array(list(set(nonempty).intersection(set(separate))), dtype=int)\n B[:n-1, nonempty_nonseparate] = np.maximum(B[:n-1, nonempty_nonseparate], eps)\n B[n-1, nonempty_separate] = np.maximum(B[n-1, nonempty_separate], eps)\n\n # renormalize and return copy\n B /= B.sum(axis=1)[:, None]\n return B"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef init_discrete_hmm_spectral(C_full, nstates, reversible=True, stationary=True, active_set=None, P=None,\n eps_A=None, eps_B=None, separate=None):\n \"\"\"Initializes discrete HMM using spectral clustering of observation counts\n\n Initializes HMM as described in [1]_. First estimates a Markov state model\n on the given observations, then uses PCCA+ to coarse-grain the transition\n matrix [2]_ which initializes the HMM transition matrix. The HMM output\n probabilities are given by Bayesian inversion from the PCCA+ memberships [1]_.\n\n The regularization parameters eps_A and eps_B are used\n to guarantee that the hidden transition matrix and output probability matrix\n have no zeros. HMM estimation algorithms such as the EM algorithm and the\n Bayesian sampling algorithm cannot recover from zero entries, i.e. once they\n are zero, they will stay zero.\n\n Parameters\n ----------\n C_full : ndarray(N, N)\n Transition count matrix on the full observable state space\n nstates : int\n The number of hidden states.\n reversible : bool\n Estimate reversible HMM transition matrix.\n stationary : bool\n p0 is the stationary distribution of P. In this case, will not\n active_set : ndarray(n, dtype=int) or None\n Index area. Will estimate kinetics only on the given subset of C\n P : ndarray(n, n)\n Transition matrix estimated from C (with option reversible). Use this\n option if P has already been estimated to avoid estimating it twice.\n eps_A : float or None\n Minimum transition probability. Default: 0.01 / nstates\n eps_B : float or None\n Minimum output probability. Default: 0.01 / nfull\n separate : None or iterable of int\n Force the given set of observed states to stay in a separate hidden state.\n The remaining nstates-1 states will be assigned by a metastable decomposition.\n\n Returns\n -------\n p0 : ndarray(n)\n Hidden state initial distribution\n A : ndarray(n, n)\n Hidden state transition matrix\n B : ndarray(n, N)\n Hidden-to-observable state output probabilities\n\n Raises\n ------\n ValueError\n If the given active set is illegal.\n NotImplementedError\n If the number of hidden states exceeds the number of observed states.\n\n Examples\n --------\n Generate initial model for a discrete output model.\n\n >>> import numpy as np\n >>> C = np.array([[0.5, 0.5, 0.0], [0.4, 0.5, 0.1], [0.0, 0.1, 0.9]])\n >>> initial_model = init_discrete_hmm_spectral(C, 2)\n\n References\n ----------\n .. [1] F. Noe, H. Wu, J.-H. Prinz and N. Plattner: Projected and hidden\n Markov models for calculating kinetics and metastable states of\n complex molecules. J. Chem. Phys. 139, 184114 (2013)\n .. [2] S. Kube and M. Weber: A coarse graining method for the identification\n of transition rates between molecular conformations.\n J. Chem. Phys. 126, 024103 (2007)\n\n \"\"\"\n # MICROSTATE COUNT MATRIX\n nfull = C_full.shape[0]\n\n # INPUTS\n if eps_A is None: # default transition probability, in order to avoid zero columns\n eps_A = 0.01 / nstates\n if eps_B is None: # default output probability, in order to avoid zero columns\n eps_B = 0.01 / nfull\n # Manage sets\n symsum = C_full.sum(axis=0) + C_full.sum(axis=1)\n nonempty = np.where(symsum > 0)[0]\n if active_set is None:\n active_set = nonempty\n else:\n if np.any(symsum[active_set] == 0):\n raise ValueError('Given active set has empty states') # don't tolerate empty states\n if P is not None:\n if np.shape(P)[0] != active_set.size: # needs to fit to active\n raise ValueError('Given initial transition matrix P has shape ' + str(np.shape(P))\n + 'while active set has size ' + str(active_set.size))\n # when using separate states, only keep the nonempty ones (the others don't matter)\n if separate is None:\n active_nonseparate = active_set.copy()\n nmeta = nstates\n else:\n if np.max(separate) >= nfull:\n raise ValueError('Separate set has indexes that do not exist in full state space: '\n + str(np.max(separate)))\n active_nonseparate = np.array(list(set(active_set) - set(separate)))\n nmeta = nstates - 1\n # check if we can proceed\n if active_nonseparate.size < nmeta:\n raise NotImplementedError('Trying to initialize ' + str(nmeta) + '-state HMM from smaller '\n + str(active_nonseparate.size) + '-state MSM.')\n\n # MICROSTATE TRANSITION MATRIX (MSM).\n C_active = C_full[np.ix_(active_set, active_set)]\n if P is None: # This matrix may be disconnected and have transient states\n P_active = _tmatrix_disconnected.estimate_P(C_active, reversible=reversible, maxiter=10000) # short iteration\n else:\n P_active = P\n\n # MICROSTATE EQUILIBRIUM DISTRIBUTION\n pi_active = _tmatrix_disconnected.stationary_distribution(P_active, C=C_active)\n pi_full = np.zeros(nfull)\n pi_full[active_set] = pi_active\n\n # NONSEPARATE TRANSITION MATRIX FOR PCCA+\n C_active_nonseparate = C_full[np.ix_(active_nonseparate, active_nonseparate)]\n if reversible and separate is None: # in this case we already have a reversible estimate with the right size\n P_active_nonseparate = P_active\n else: # not yet reversible. re-estimate\n P_active_nonseparate = _tmatrix_disconnected.estimate_P(C_active_nonseparate, reversible=True)\n\n # COARSE-GRAINING WITH PCCA+\n if active_nonseparate.size > nmeta:\n from msmtools.analysis.dense.pcca import PCCA\n pcca_obj = PCCA(P_active_nonseparate, nmeta)\n M_active_nonseparate = pcca_obj.memberships # memberships\n B_active_nonseparate = pcca_obj.output_probabilities # output probabilities\n else: # equal size\n M_active_nonseparate = np.eye(nmeta)\n B_active_nonseparate = np.eye(nmeta)\n\n # ADD SEPARATE STATE IF NEEDED\n if separate is None:\n M_active = M_active_nonseparate\n else:\n M_full = np.zeros((nfull, nstates))\n M_full[active_nonseparate, :nmeta] = M_active_nonseparate\n M_full[separate, -1] = 1\n M_active = M_full[active_set]\n\n # COARSE-GRAINED TRANSITION MATRIX\n P_hmm = coarse_grain_transition_matrix(P_active, M_active)\n if reversible:\n P_hmm = _tmatrix_disconnected.enforce_reversible_on_closed(P_hmm)\n C_hmm = M_active.T.dot(C_active).dot(M_active)\n pi_hmm = _tmatrix_disconnected.stationary_distribution(P_hmm, C=C_hmm) # need C_hmm in case if A is disconnected\n\n # COARSE-GRAINED OUTPUT DISTRIBUTION\n B_hmm = np.zeros((nstates, nfull))\n B_hmm[:nmeta, active_nonseparate] = B_active_nonseparate\n if separate is not None: # add separate states\n B_hmm[-1, separate] = pi_full[separate]\n\n # REGULARIZE SOLUTION\n pi_hmm, P_hmm = regularize_hidden(pi_hmm, P_hmm, reversible=reversible, stationary=stationary, C=C_hmm, eps=eps_A)\n B_hmm = regularize_pobs(B_hmm, nonempty=nonempty, separate=separate, eps=eps_B)\n\n # print 'cg pi: ', pi_hmm\n # print 'cg A:\\n ', P_hmm\n # print 'cg B:\\n ', B_hmm\n\n logger().info('Initial model: ')\n logger().info('initial distribution = \\n'+str(pi_hmm))\n logger().info('transition matrix = \\n'+str(P_hmm))\n logger().info('output matrix = \\n'+str(B_hmm.T))\n\n return pi_hmm, P_hmm, B_hmm", "response": "Initializes discrete HMM using spectral clustering of observation counts C_full."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef init_discrete_hmm_ml(C_full, nstates, reversible=True, stationary=True, active_set=None, P=None,\n eps_A=None, eps_B=None, separate=None):\n \"\"\"Initializes discrete HMM using maximum likelihood of observation counts\"\"\"\n raise NotImplementedError('ML-initialization not yet implemented')", "response": "Initializes discrete HMM using maximum likelihood of observation counts"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_stationary(self):\n # for disconnected matrices, the stationary distribution depends on the estimator, so we can't compute\n # it directly. Therefore we test whether the initial distribution is stationary.\n return np.allclose(np.dot(self._Pi, self._Tij), self._Pi)", "response": "r Returns True if the MSM is stationary."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute the stationary distribution of hidden states if possible.", "response": "def stationary_distribution(self):\n r\"\"\" Compute stationary distribution of hidden states if possible.\n\n Raises\n ------\n ValueError if the HMM is not stationary\n\n \"\"\"\n assert _tmatrix_disconnected.is_connected(self._Tij, strong=False), \\\n 'No unique stationary distribution because transition matrix is not connected'\n import msmtools.analysis as msmana\n return msmana.stationary_distribution(self._Tij)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the relaxation timescales of the hidden transition matrix.", "response": "def timescales(self):\n r\"\"\" Relaxation timescales of the hidden transition matrix\n\n Returns\n -------\n ts : ndarray(m)\n relaxation timescales in units of the input trajectory time step,\n defined by :math:`-tau / ln | \\lambda_i |, i = 2,...,nstates`, where\n :math:`\\lambda_i` are the hidden transition matrix eigenvalues.\n\n \"\"\"\n from msmtools.analysis.dense.decomposition import timescales_from_eigenvalues as _timescales\n\n self._ensure_spectral_decomposition()\n ts = _timescales(self._eigenvalues, tau=self._lag)\n return ts[1:]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the lifetimes of the hidden transition matrix.", "response": "def lifetimes(self):\n r\"\"\" Lifetimes of states of the hidden transition matrix\n\n Returns\n -------\n l : ndarray(nstates)\n state lifetimes in units of the input trajectory time step,\n defined by :math:`-tau / ln | p_{ii} |, i = 1,...,nstates`, where\n :math:`p_{ii}` are the diagonal entries of the hidden transition matrix.\n\n \"\"\"\n return -self._lag / np.log(np.diag(self.transition_matrix))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sub_hmm(self, states):\n # restrict initial distribution\n pi_sub = self._Pi[states]\n pi_sub /= pi_sub.sum()\n\n # restrict transition matrix\n P_sub = self._Tij[states, :][:, states]\n # checks if this selection is possible\n assert np.all(P_sub.sum(axis=1) > 0), \\\n 'Illegal sub_hmm request: transition matrix cannot be normalized on ' + str(states)\n P_sub /= P_sub.sum(axis=1)[:, None]\n\n # restrict output model\n out_sub = self.output_model.sub_output_model(states)\n\n return HMM(pi_sub, P_sub, out_sub, lag=self.lag)", "response": "r Returns the HMM restricted to the selected subset of states"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef count_matrix(self):\n # TODO: does this belong here or to the BHMM sampler, or in a subclass containing HMM with data?\n \"\"\"Compute the transition count matrix from hidden state trajectory.\n\n Returns\n -------\n C : numpy.array with shape (nstates,nstates)\n C[i,j] is the number of transitions observed from state i to state j\n\n Raises\n ------\n RuntimeError\n A RuntimeError is raised if the HMM model does not yet have a hidden state trajectory associated with it.\n\n Examples\n --------\n\n \"\"\"\n if self.hidden_state_trajectories is None:\n raise RuntimeError('HMM model does not have a hidden state trajectory.')\n\n C = msmest.count_matrix(self.hidden_state_trajectories, 1, nstates=self._nstates)\n return C.toarray()", "response": "Compute the transition count matrix from the hidden state trajectory."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the counts at the first time step", "response": "def count_init(self):\n \"\"\"Compute the counts at the first time step\n\n Returns\n -------\n n : ndarray(nstates)\n n[i] is the number of trajectories starting in state i\n\n \"\"\"\n if self.hidden_state_trajectories is None:\n raise RuntimeError('HMM model does not have a hidden state trajectory.')\n\n n = [traj[0] for traj in self.hidden_state_trajectories]\n return np.bincount(n, minlength=self.nstates)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef collect_observations_in_state(self, observations, state_index):\n # TODO: this would work well in a subclass with data\n \"\"\"Collect a vector of all observations belonging to a specified hidden state.\n\n Parameters\n ----------\n observations : list of numpy.array\n List of observed trajectories.\n state_index : int\n The index of the hidden state for which corresponding observations are to be retrieved.\n dtype : numpy.dtype, optional, default=numpy.float64\n The numpy dtype to use to store the collected observations.\n\n Returns\n -------\n collected_observations : numpy.array with shape (nsamples,)\n The collected vector of observations belonging to the specified hidden state.\n\n Raises\n ------\n RuntimeError\n A RuntimeError is raised if the HMM model does not yet have a hidden state trajectory associated with it.\n\n \"\"\"\n if not self.hidden_state_trajectories:\n raise RuntimeError('HMM model does not have a hidden state trajectory.')\n\n dtype = observations[0].dtype\n collected_observations = np.array([], dtype=dtype)\n for (s_t, o_t) in zip(self.hidden_state_trajectories, observations):\n indices = np.where(s_t == state_index)[0]\n collected_observations = np.append(collected_observations, o_t[indices])\n\n return collected_observations", "response": "Collect a vector of all observations belonging to a specified hidden state."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate a synthetic state trajectory.", "response": "def generate_synthetic_state_trajectory(self, nsteps, initial_Pi=None, start=None, stop=None, dtype=np.int32):\n \"\"\"Generate a synthetic state trajectory.\n\n Parameters\n ----------\n nsteps : int\n Number of steps in the synthetic state trajectory to be generated.\n initial_Pi : np.array of shape (nstates,), optional, default=None\n The initial probability distribution, if samples are not to be taken from the intrinsic\n initial distribution.\n start : int\n starting state. Exclusive with initial_Pi\n stop : int\n stopping state. Trajectory will terminate when reaching the stopping state before length number of steps.\n dtype : numpy.dtype, optional, default=numpy.int32\n The numpy dtype to use to store the synthetic trajectory.\n\n Returns\n -------\n states : np.array of shape (nstates,) of dtype=np.int32\n The trajectory of hidden states, with each element in range(0,nstates).\n\n Examples\n --------\n\n Generate a synthetic state trajectory of a specified length.\n\n >>> from bhmm import testsystems\n >>> model = testsystems.dalton_model()\n >>> states = model.generate_synthetic_state_trajectory(nsteps=100)\n\n \"\"\"\n # consistency check\n if initial_Pi is not None and start is not None:\n raise ValueError('Arguments initial_Pi and start are exclusive. Only set one of them.')\n\n # Generate first state sample.\n if start is None:\n if initial_Pi is not None:\n start = np.random.choice(range(self._nstates), size=1, p=initial_Pi)\n else:\n start = np.random.choice(range(self._nstates), size=1, p=self._Pi)\n\n # Generate and return trajectory\n from msmtools import generation as msmgen\n traj = msmgen.generate_traj(self.transition_matrix, nsteps, start=start, stop=stop, dt=1)\n return traj.astype(dtype)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_synthetic_observation_trajectory(self, length, initial_Pi=None):\n # First, generate synthetic state trajetory.\n s_t = self.generate_synthetic_state_trajectory(length, initial_Pi=initial_Pi)\n\n # Next, generate observations from these states.\n o_t = self.output_model.generate_observation_trajectory(s_t)\n\n return [o_t, s_t]", "response": "Generate a synthetic state trajectory for an equilibrium realization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a number of synthetic realization of observables from this model.", "response": "def generate_synthetic_observation_trajectories(self, ntrajectories, length, initial_Pi=None):\n \"\"\"Generate a number of synthetic realization of observables from this model.\n\n Parameters\n ----------\n ntrajectories : int\n The number of trajectories to be generated.\n length : int\n Length of synthetic state trajectory to be generated.\n initial_Pi : np.array of shape (nstates,), optional, default=None\n The initial probability distribution, if samples are not to be taken from equilibrium.\n\n Returns\n -------\n O : list of np.array of shape (nstates,) of dtype=np.float32\n The trajectories of observations\n S : list of np.array of shape (nstates,) of dtype=np.int32\n The trajectories of hidden states\n\n Examples\n --------\n\n Generate a number of synthetic trajectories.\n\n >>> from bhmm import testsystems\n >>> model = testsystems.dalton_model()\n >>> O, S = model.generate_synthetic_observation_trajectories(ntrajectories=10, length=100)\n\n Use an initial nonequilibrium distribution.\n\n >>> from bhmm import testsystems\n >>> model = testsystems.dalton_model(nstates=3)\n >>> O, S = model.generate_synthetic_observation_trajectories(ntrajectories=10, length=100, initial_Pi=np.array([1,0,0]))\n\n \"\"\"\n O = list() # observations\n S = list() # state trajectories\n for trajectory_index in range(ntrajectories):\n o_t, s_t = self.generate_synthetic_observation_trajectory(length=length, initial_Pi=initial_Pi)\n O.append(o_t)\n S.append(s_t)\n\n return O, S"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting notebook to python script", "response": "def nb_to_python(nb_path):\n \"\"\"convert notebook to python script\"\"\"\n exporter = python.PythonExporter()\n output, resources = exporter.from_filename(nb_path)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting notebook to html", "response": "def nb_to_html(nb_path):\n \"\"\"convert notebook to html\"\"\"\n exporter = html.HTMLExporter(template_file='full')\n output, resources = exporter.from_filename(nb_path)\n header = output.split('', 1)[1].split('',1)[0]\n body = output.split('', 1)[1].split('',1)[0]\n\n # http://imgur.com/eR9bMRH\n header = header.replace('']\n lines.append(header)\n lines.append(body)\n lines.append('')\n return '\\n'.join(lines)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the output probabilities for an entire trajectory and all hidden states at time point t.", "response": "def p_obs(self, obs, out=None):\n \"\"\"\n Returns the output probabilities for an entire trajectory and all hidden states\n\n Parameters\n ----------\n obs : ndarray((T), dtype=int)\n a discrete trajectory of length T\n\n Return\n ------\n p_o : ndarray (T,N)\n the probability of generating the symbol at time point t from any of the N hidden states\n\n \"\"\"\n if out is None:\n out = self._output_probabilities[:, obs].T\n # out /= np.sum(out, axis=1)[:,None]\n return self._handle_outliers(out)\n else:\n if obs.shape[0] == out.shape[0]:\n np.copyto(out, self._output_probabilities[:, obs].T)\n elif obs.shape[0] < out.shape[0]:\n out[:obs.shape[0], :] = self._output_probabilities[:, obs].T\n else:\n raise ValueError('output array out is too small: '+str(out.shape[0])+' < '+str(obs.shape[0]))\n # out /= np.sum(out, axis=1)[:,None]\n return self._handle_outliers(out)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sample(self, observations_by_state):\n from numpy.random import dirichlet\n N, M = self._output_probabilities.shape # nstates, nsymbols\n for i, obs_by_state in enumerate(observations_by_state):\n # count symbols found in data\n count = np.bincount(obs_by_state, minlength=M).astype(float)\n # sample dirichlet distribution\n count += self.prior[i]\n positive = count > 0\n # if counts at all: can't sample, so leave output probabilities as they are.\n self._output_probabilities[i, positive] = dirichlet(count[positive])", "response": "Sample a new set of distribution parameters given a sample of observations from the given state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a single synthetic observation data from a given state.", "response": "def generate_observation_from_state(self, state_index):\n \"\"\"\n Generate a single synthetic observation data from a given state.\n\n Parameters\n ----------\n state_index : int\n Index of the state from which observations are to be generated.\n\n Returns\n -------\n observation : float\n A single observation from the given state.\n\n Examples\n --------\n\n Generate an observation model.\n\n >>> output_model = DiscreteOutputModel(np.array([[0.5,0.5],[0.1,0.9]]))\n\n Generate sample from each state.\n\n >>> observation = output_model.generate_observation_from_state(0)\n\n \"\"\"\n # generate random generator (note that this is inefficient - better use one of the next functions\n import scipy.stats\n gen = scipy.stats.rv_discrete(values=(range(len(self._output_probabilities[state_index])), \n self._output_probabilities[state_index]))\n gen.rvs(size=1)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_observations_from_state(self, state_index, nobs):\n import scipy.stats\n gen = scipy.stats.rv_discrete(values=(range(self._nsymbols), self._output_probabilities[state_index]))\n gen.rvs(size=nobs)", "response": "Generate synthetic observation data from a given state."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_observation_trajectory(self, s_t, dtype=None):\n if dtype is None:\n dtype = np.int32\n\n # Determine number of samples to generate.\n T = s_t.shape[0]\n nsymbols = self._output_probabilities.shape[1]\n\n if (s_t.max() >= self.nstates) or (s_t.min() < 0):\n msg = ''\n msg += 's_t = %s\\n' % s_t\n msg += 's_t.min() = %d, s_t.max() = %d\\n' % (s_t.min(), s_t.max())\n msg += 's_t.argmax = %d\\n' % s_t.argmax()\n msg += 'self.nstates = %d\\n' % self.nstates\n msg += 's_t is out of bounds.\\n'\n raise Exception(msg)\n\n # generate random generators\n # import scipy.stats\n # gens = [scipy.stats.rv_discrete(values=(range(len(self.B[state_index])), self.B[state_index]))\n # for state_index in range(self.B.shape[0])]\n # o_t = np.zeros([T], dtype=dtype)\n # for t in range(T):\n # s = s_t[t]\n # o_t[t] = gens[s].rvs(size=1)\n # return o_t\n\n o_t = np.zeros([T], dtype=dtype)\n for t in range(T):\n s = s_t[t]\n o_t[t] = np.random.choice(nsymbols, p=self._output_probabilities[s, :])\n\n return o_t", "response": "Generate synthetic observation data from a given state sequence."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_implementation(impl):\n global __impl__\n if impl.lower() == 'python':\n __impl__ = __IMPL_PYTHON__\n elif impl.lower() == 'c':\n __impl__ = __IMPL_C__\n else:\n import warnings\n warnings.warn('Implementation '+impl+' is not known. Using the fallback python implementation.')\n __impl__ = __IMPL_PYTHON__", "response": "Sets the implementation of this module\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes P ( obs | A B pi ) and all forward coefficients.", "response": "def forward(A, pobs, pi, T=None, alpha_out=None):\n \"\"\"Compute P( obs | A, B, pi ) and all forward coefficients.\n\n Parameters\n ----------\n A : ndarray((N,N), dtype = float)\n transition matrix of the hidden states\n pobs : ndarray((T,N), dtype = float)\n pobs[t,i] is the observation probability for observation at time t given hidden state i\n pi : ndarray((N), dtype = float)\n initial distribution of hidden states\n T : int, optional, default = None\n trajectory length. If not given, T = pobs.shape[0] will be used.\n alpha_out : ndarray((T,N), dtype = float), optional, default = None\n containter for the alpha result variables. If None, a new container will be created.\n\n Returns\n -------\n logprob : float\n The probability to observe the sequence `ob` with the model given\n by `A`, `B` and `pi`.\n alpha : ndarray((T,N), dtype = float), optional, default = None\n alpha[t,i] is the ith forward coefficient of time t. These can be\n used in many different algorithms related to HMMs.\n\n \"\"\"\n if __impl__ == __IMPL_PYTHON__:\n return ip.forward(A, pobs, pi, T=T, alpha_out=alpha_out, dtype=config.dtype)\n elif __impl__ == __IMPL_C__:\n return ic.forward(A, pobs, pi, T=T, alpha_out=alpha_out, dtype=config.dtype)\n else:\n raise RuntimeError('Nonexisting implementation selected: '+str(__impl__))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef backward(A, pobs, T=None, beta_out=None):\n if __impl__ == __IMPL_PYTHON__:\n return ip.backward(A, pobs, T=T, beta_out=beta_out, dtype=config.dtype)\n elif __impl__ == __IMPL_C__:\n return ic.backward(A, pobs, T=T, beta_out=beta_out, dtype=config.dtype)\n else:\n raise RuntimeError('Nonexisting implementation selected: '+str(__impl__))", "response": "Compute all backward coefficients of the hidden states A."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef state_probabilities(alpha, beta, T=None, gamma_out=None):\n # get summation helper - we use matrix multiplication with 1's because it's faster than the np.sum function (yes!)\n global ones_size\n if ones_size != alpha.shape[1]:\n global ones\n ones = np.ones(alpha.shape[1])[:, None]\n ones_size = alpha.shape[1]\n #\n if alpha.shape[0] != beta.shape[0]:\n raise ValueError('Inconsistent sizes of alpha and beta.')\n # determine T to use\n if T is None:\n if gamma_out is None:\n T = alpha.shape[0]\n else:\n T = gamma_out.shape[0]\n # compute\n if gamma_out is None:\n gamma_out = alpha * beta\n if T < gamma_out.shape[0]:\n gamma_out = gamma_out[:T]\n else:\n if gamma_out.shape[0] < alpha.shape[0]:\n np.multiply(alpha[:T], beta[:T], gamma_out)\n else:\n np.multiply(alpha, beta, gamma_out)\n # normalize\n np.divide(gamma_out, np.dot(gamma_out, ones), out=gamma_out)\n # done\n return gamma_out", "response": "Calculates the probability matrix for being in state i at time t."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsums the probabilities of being in state i to time t", "response": "def state_counts(gamma, T, out=None):\n \"\"\" Sum the probabilities of being in state i to time t\n\n Parameters\n ----------\n gamma : ndarray((T,N), dtype = float), optional, default = None\n gamma[t,i] is the probabilty at time t to be in state i !\n T : int\n number of time steps\n\n Returns\n -------\n count : numpy.array shape (N)\n count[i] is the summed probabilty to be in state i !\n\n See Also\n --------\n state_probabilities : to calculate `gamma`\n\n \"\"\"\n return np.sum(gamma[0:T], axis=0, out=out)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transition_counts(alpha, beta, A, pobs, T=None, out=None):\n if __impl__ == __IMPL_PYTHON__:\n return ip.transition_counts(alpha, beta, A, pobs, T=T, out=out, dtype=config.dtype)\n elif __impl__ == __IMPL_C__:\n return ic.transition_counts(alpha, beta, A, pobs, T=T, out=out, dtype=config.dtype)\n else:\n raise RuntimeError('Nonexisting implementation selected: '+str(__impl__))", "response": "Sum for all t the probability to transition from state i to state j."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef viterbi(A, pobs, pi):\n if __impl__ == __IMPL_PYTHON__:\n return ip.viterbi(A, pobs, pi, dtype=config.dtype)\n elif __impl__ == __IMPL_C__:\n return ic.viterbi(A, pobs, pi, dtype=config.dtype)\n else:\n raise RuntimeError('Nonexisting implementation selected: '+str(__impl__))", "response": "Estimate the hidden pathway of maximum likelihood using the Viterbi algorithm."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sample_path(alpha, A, pobs, T=None):\n if __impl__ == __IMPL_PYTHON__:\n return ip.sample_path(alpha, A, pobs, T=T, dtype=config.dtype)\n elif __impl__ == __IMPL_C__:\n return ic.sample_path(alpha, A, pobs, T=T, dtype=config.dtype)\n else:\n raise RuntimeError('Nonexisting implementation selected: '+str(__impl__))", "response": "Sample the hidden pathway S from the conditional distribution P."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef logger(name='BHMM', pattern='%(asctime)s %(levelname)s %(name)s: %(message)s',\n date_format='%H:%M:%S', handler=logging.StreamHandler(sys.stdout)):\n \"\"\"\n Retrieves the logger instance associated to the given name.\n\n :param name: The name of the logger instance.\n :type name: str\n :param pattern: The associated pattern.\n :type pattern: str\n :param date_format: The date format to be used in the pattern.\n :type date_format: str\n :param handler: The logging handler, by default console output.\n :type handler: FileHandler or StreamHandler or NullHandler\n\n :return: The logger.\n :rtype: Logger\n \"\"\"\n _logger = logging.getLogger(name)\n _logger.setLevel(config.log_level())\n if not _logger.handlers:\n formatter = logging.Formatter(pattern, date_format)\n handler.setFormatter(formatter)\n handler.setLevel(config.log_level())\n _logger.addHandler(handler)\n _logger.propagate = False\n return _logger", "response": "Creates a logger instance associated with the given name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sample(self, nsamples, nburn=0, nthin=1, save_hidden_state_trajectory=False,\n call_back=None):\n \"\"\"Sample from the BHMM posterior.\n\n Parameters\n ----------\n nsamples : int\n The number of samples to generate.\n nburn : int, optional, default=0\n The number of samples to discard to burn-in, following which `nsamples` will be generated.\n nthin : int, optional, default=1\n The number of Gibbs sampling updates used to generate each returned sample.\n save_hidden_state_trajectory : bool, optional, default=False\n If True, the hidden state trajectory for each sample will be saved as well.\n call_back : function, optional, default=None\n a call back function with no arguments, which if given is being called\n after each computed sample. This is useful for implementing progress bars.\n\n Returns\n -------\n models : list of bhmm.HMM\n The sampled HMM models from the Bayesian posterior.\n\n Examples\n --------\n\n >>> from bhmm import testsystems\n >>> [model, observations, states, sampled_model] = testsystems.generate_random_bhmm(ntrajectories=5, length=1000)\n >>> nburn = 5 # run the sampler a bit before recording samples\n >>> nsamples = 10 # generate 10 samples\n >>> nthin = 2 # discard one sample in between each recorded sample\n >>> samples = sampled_model.sample(nsamples, nburn=nburn, nthin=nthin)\n\n \"\"\"\n\n # Run burn-in.\n for iteration in range(nburn):\n logger().info(\"Burn-in %8d / %8d\" % (iteration, nburn))\n self._update()\n\n # Collect data.\n models = list()\n for iteration in range(nsamples):\n logger().info(\"Iteration %8d / %8d\" % (iteration, nsamples))\n # Run a number of Gibbs sampling updates to generate each sample.\n for thin in range(nthin):\n self._update()\n # Save a copy of the current model.\n model_copy = copy.deepcopy(self.model)\n # print \"Sampled: \\n\",repr(model_copy)\n if not save_hidden_state_trajectory:\n model_copy.hidden_state_trajectory = None\n models.append(model_copy)\n if call_back is not None:\n call_back()\n\n # Return the list of models saved.\n return models", "response": "Sample from the BHMM posterior."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _update(self):\n initial_time = time.time()\n\n self._updateHiddenStateTrajectories()\n self._updateEmissionProbabilities()\n self._updateTransitionMatrix()\n\n final_time = time.time()\n elapsed_time = final_time - initial_time\n logger().info(\"BHMM update iteration took %.3f s\" % elapsed_time)", "response": "Update the current model using one round of Gibbs sampling."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the hidden state trajectories attribute of the current state model.", "response": "def _updateHiddenStateTrajectories(self):\n \"\"\"Sample a new set of state trajectories from the conditional distribution P(S | T, E, O)\n\n \"\"\"\n self.model.hidden_state_trajectories = list()\n for trajectory_index in range(self.nobs):\n hidden_state_trajectory = self._sampleHiddenStateTrajectory(self.observations[trajectory_index])\n self.model.hidden_state_trajectories.append(hidden_state_trajectory)\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsamples a hidden state trajectory from the conditional distribution P s | T E o.", "response": "def _sampleHiddenStateTrajectory(self, obs, dtype=np.int32):\n \"\"\"Sample a hidden state trajectory from the conditional distribution P(s | T, E, o)\n\n Parameters\n ----------\n o_t : numpy.array with dimensions (T,)\n observation[n] is the nth observation\n dtype : numpy.dtype, optional, default=numpy.int32\n The dtype to to use for returned state trajectory.\n\n Returns\n -------\n s_t : numpy.array with dimensions (T,) of type `dtype`\n Hidden state trajectory, with s_t[t] the hidden state corresponding to observation o_t[t]\n\n Examples\n --------\n >>> import bhmm\n >>> [model, observations, states, sampled_model] = bhmm.testsystems.generate_random_bhmm(ntrajectories=5, length=1000)\n >>> o_t = observations[0]\n >>> s_t = sampled_model._sampleHiddenStateTrajectory(o_t)\n\n \"\"\"\n\n # Determine observation trajectory length\n T = obs.shape[0]\n\n # Convenience access.\n A = self.model.transition_matrix\n pi = self.model.initial_distribution\n\n # compute output probability matrix\n self.model.output_model.p_obs(obs, out=self.pobs)\n # compute forward variables\n logprob = hidden.forward(A, self.pobs, pi, T=T, alpha_out=self.alpha)[0]\n # sample path\n S = hidden.sample_path(self.alpha, A, self.pobs, T=T)\n\n return S"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the emission probabilites of the current state.", "response": "def _updateEmissionProbabilities(self):\n \"\"\"Sample a new set of emission probabilites from the conditional distribution P(E | S, O)\n\n \"\"\"\n observations_by_state = [self.model.collect_observations_in_state(self.observations, state)\n for state in range(self.model.nstates)]\n self.model.output_model.sample(observations_by_state)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates the hidden - state transition matrix and the initial distribution.", "response": "def _updateTransitionMatrix(self):\n \"\"\"\n Updates the hidden-state transition matrix and the initial distribution\n\n \"\"\"\n # TRANSITION MATRIX\n C = self.model.count_matrix() + self.prior_C # posterior count matrix\n\n # check if we work with these options\n if self.reversible and not _tmatrix_disconnected.is_connected(C, strong=True):\n raise NotImplementedError('Encountered disconnected count matrix with sampling option reversible:\\n '\n + str(C) + '\\nUse prior to ensure connectivity or use reversible=False.')\n # ensure consistent sparsity pattern (P0 might have additional zeros because of underflows)\n # TODO: these steps work around a bug in msmtools. Should be fixed there\n P0 = msmest.transition_matrix(C, reversible=self.reversible, maxiter=10000, warn_not_converged=False)\n zeros = np.where(P0 + P0.T == 0)\n C[zeros] = 0\n # run sampler\n Tij = msmest.sample_tmatrix(C, nsample=1, nsteps=self.transition_matrix_sampling_steps,\n reversible=self.reversible)\n\n # INITIAL DISTRIBUTION\n if self.stationary: # p0 is consistent with P\n p0 = _tmatrix_disconnected.stationary_distribution(Tij, C=C)\n else:\n n0 = self.model.count_init().astype(float)\n first_timestep_counts_with_prior = n0 + self.prior_n0\n positive = first_timestep_counts_with_prior > 0\n p0 = np.zeros_like(n0)\n p0[positive] = np.random.dirichlet(first_timestep_counts_with_prior[positive]) # sample p0 from posterior\n\n # update HMM with new sample\n self.model.update(p0, Tij)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes using an MLHMM and generate a model", "response": "def _generateInitialModel(self, output_model_type):\n \"\"\"Initialize using an MLHMM.\n\n \"\"\"\n logger().info(\"Generating initial model for BHMM using MLHMM...\")\n from bhmm.estimators.maximum_likelihood import MaximumLikelihoodEstimator\n mlhmm = MaximumLikelihoodEstimator(self.observations, self.nstates, reversible=self.reversible,\n output=output_model_type)\n model = mlhmm.fit()\n return model"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the connected sets of the given count matrix.", "response": "def connected_sets(C, mincount_connectivity=0, strong=True):\n \"\"\" Computes the connected sets of C.\n\n C : count matrix\n mincount_connectivity : float\n Minimum count which counts as a connection.\n strong : boolean\n True: Seek strongly connected sets. False: Seek weakly connected sets.\n\n \"\"\"\n import msmtools.estimation as msmest\n Cconn = C.copy()\n Cconn[np.where(C <= mincount_connectivity)] = 0\n # treat each connected set separately\n S = msmest.connected_sets(Cconn, directed=strong)\n return S"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef closed_sets(C, mincount_connectivity=0):\n n = np.shape(C)[0]\n S = connected_sets(C, mincount_connectivity=mincount_connectivity, strong=True)\n closed = []\n for s in S:\n mask = np.zeros(n, dtype=bool)\n mask[s] = True\n if C[np.ix_(mask, ~mask)].sum() == 0: # closed set, take it\n closed.append(s)\n return closed", "response": "Computes strongly connected closed sets of a single site."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the set of states that have at least one incoming or outgoing count.", "response": "def nonempty_set(C, mincount_connectivity=0):\n \"\"\" Returns the set of states that have at least one incoming or outgoing count \"\"\"\n # truncate to states with at least one observed incoming or outgoing count.\n if mincount_connectivity > 0:\n C = C.copy()\n C[np.where(C < mincount_connectivity)] = 0\n return np.where(C.sum(axis=0) + C.sum(axis=1) > 0)[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nestimate full transition matrix for general connectivity structure.", "response": "def estimate_P(C, reversible=True, fixed_statdist=None, maxiter=1000000, maxerr=1e-8, mincount_connectivity=0):\n \"\"\" Estimates full transition matrix for general connectivity structure\n\n Parameters\n ----------\n C : ndarray\n count matrix\n reversible : bool\n estimate reversible?\n fixed_statdist : ndarray or None\n estimate with given stationary distribution\n maxiter : int\n Maximum number of reversible iterations.\n maxerr : float\n Stopping criterion for reversible iteration: Will stop when infinity\n norm of difference vector of two subsequent equilibrium distributions\n is below maxerr.\n mincount_connectivity : float\n Minimum count which counts as a connection.\n\n \"\"\"\n import msmtools.estimation as msmest\n n = np.shape(C)[0]\n # output matrix. Set initially to Identity matrix in order to handle empty states\n P = np.eye(n, dtype=np.float64)\n # decide if we need to proceed by weakly or strongly connected sets\n if reversible and fixed_statdist is None: # reversible to unknown eq. dist. - use strongly connected sets.\n S = connected_sets(C, mincount_connectivity=mincount_connectivity, strong=True)\n for s in S:\n mask = np.zeros(n, dtype=bool)\n mask[s] = True\n if C[np.ix_(mask, ~mask)].sum() > np.finfo(C.dtype).eps: # outgoing transitions - use partial rev algo.\n transition_matrix_partial_rev(C, P, mask, maxiter=maxiter, maxerr=maxerr)\n else: # closed set - use standard estimator\n I = np.ix_(mask, mask)\n if s.size > 1: # leave diagonal 1 if single closed state.\n P[I] = msmest.transition_matrix(C[I], reversible=True, warn_not_converged=False,\n maxiter=maxiter, maxerr=maxerr)\n else: # nonreversible or given equilibrium distribution - weakly connected sets\n S = connected_sets(C, mincount_connectivity=mincount_connectivity, strong=False)\n for s in S:\n I = np.ix_(s, s)\n if not reversible:\n Csub = C[I]\n # any zero rows? must set Cii = 1 to avoid dividing by zero\n zero_rows = np.where(Csub.sum(axis=1) == 0)[0]\n Csub[zero_rows, zero_rows] = 1.0\n P[I] = msmest.transition_matrix(Csub, reversible=False)\n elif reversible and fixed_statdist is not None:\n P[I] = msmest.transition_matrix(C[I], reversible=True, fixed_statdist=fixed_statdist,\n maxiter=maxiter, maxerr=maxerr)\n else: # unknown case\n raise NotImplementedError('Transition estimation for the case reversible=' + str(reversible) +\n ' fixed_statdist=' + str(fixed_statdist is not None) + ' not implemented.')\n # done\n return P"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transition_matrix_partial_rev(C, P, S, maxiter=1000000, maxerr=1e-8):\n # test input\n assert np.array_equal(C.shape, P.shape)\n # constants\n A = C[S][:, S]\n B = C[S][:, ~S]\n ATA = A + A.T\n countsums = C[S].sum(axis=1)\n # initialize\n X = 0.5 * ATA\n Y = C[S][:, ~S]\n # normalize X, Y\n totalsum = X.sum() + Y.sum()\n X /= totalsum\n Y /= totalsum\n # rowsums\n rowsums = X.sum(axis=1) + Y.sum(axis=1)\n err = 1.0\n it = 0\n while err > maxerr and it < maxiter:\n # update\n d = countsums / rowsums\n X = ATA / (d[:, None] + d)\n Y = B / d[:, None]\n # normalize X, Y\n totalsum = X.sum() + Y.sum()\n X /= totalsum\n Y /= totalsum\n # update sums\n rowsums_new = X.sum(axis=1) + Y.sum(axis=1)\n # compute error\n err = np.max(np.abs(rowsums_new - rowsums))\n # update\n rowsums = rowsums_new\n it += 1\n # write to P\n P[np.ix_(S, S)] = X\n P[np.ix_(S, ~S)] = Y\n P[S] /= P[S].sum(axis=1)[:, None]", "response": "This function computes the partial reversible transition matrix."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef enforce_reversible_on_closed(P):\n import msmtools.analysis as msmana\n n = np.shape(P)[0]\n Prev = P.copy()\n # treat each weakly connected set separately\n sets = closed_sets(P)\n for s in sets:\n I = np.ix_(s, s)\n # compute stationary probability\n pi_s = msmana.stationary_distribution(P[I])\n # symmetrize\n X_s = pi_s[:, None] * P[I]\n X_s = 0.5 * (X_s + X_s.T)\n # normalize\n Prev[I] = X_s / X_s.sum(axis=1)[:, None]\n return Prev", "response": "Enforces transition matrix P to be reversible on its closed sets."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning if P is reversible on its weakly connected sets.", "response": "def is_reversible(P):\n \"\"\" Returns if P is reversible on its weakly connected sets \"\"\"\n import msmtools.analysis as msmana\n # treat each weakly connected set separately\n sets = connected_sets(P, strong=False)\n for s in sets:\n Ps = P[s, :][:, s]\n if not msmana.is_transition_matrix(Ps):\n return False # isn't even a transition matrix!\n pi = msmana.stationary_distribution(Ps)\n X = pi[:, None] * Ps\n if not np.allclose(X, X.T):\n return False\n # survived.\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _guess_output_type(observations):\n from bhmm.util import types as _types\n\n o1 = _np.array(observations[0])\n\n # CASE: vector of int? Then we want a discrete HMM\n if _types.is_int_vector(o1):\n return 'discrete'\n\n # CASE: not int type, but everything is an integral number. Then we also go for discrete\n if _np.allclose(o1, _np.round(o1)):\n isintegral = True\n for i in range(1, len(observations)):\n if not _np.allclose(observations[i], _np.round(observations[i])):\n isintegral = False\n break\n if isintegral:\n return 'discrete'\n\n # CASE: vector of double? Then we want a gaussian\n if _types.is_float_vector(o1):\n return 'gaussian'\n\n # None of the above? Then we currently do not support this format!\n raise TypeError('Observations is neither sequences of integers nor 1D-sequences of floats. The current version'\n 'does not support your input.')", "response": "Suggests a HMM model type based on the observation data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lag_observations(observations, lag, stride=1):\n obsnew = []\n for obs in observations:\n for shift in range(0, lag, stride):\n obs_lagged = (obs[shift:][::lag])\n if len(obs_lagged) > 1:\n obsnew.append(obs_lagged)\n return obsnew", "response": "r Returns a list of observations that are subsampled at lag but shifted by 1."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize a 1D - Gaussian HMM with the given initial distribution pi P and means and sigmas. Returns a 1D - Gaussian HMM containing the current state of the current state.", "response": "def gaussian_hmm(pi, P, means, sigmas):\n \"\"\" Initializes a 1D-Gaussian HMM\n\n Parameters\n ----------\n pi : ndarray(nstates, )\n Initial distribution.\n P : ndarray(nstates,nstates)\n Hidden transition matrix\n means : ndarray(nstates, )\n Means of Gaussian output distributions\n sigmas : ndarray(nstates, )\n Standard deviations of Gaussian output distributions\n stationary : bool, optional, default=True\n If True: initial distribution is equal to stationary distribution of transition matrix\n reversible : bool, optional, default=True\n If True: transition matrix will fulfill detailed balance constraints.\n\n \"\"\"\n from bhmm.hmm.gaussian_hmm import GaussianHMM\n from bhmm.output_models.gaussian import GaussianOutputModel\n # count states\n nstates = _np.array(P).shape[0]\n # initialize output model\n output_model = GaussianOutputModel(nstates, means, sigmas)\n # initialize general HMM\n from bhmm.hmm.generic_hmm import HMM as _HMM\n ghmm = _HMM(pi, P, output_model)\n # turn it into a Gaussian HMM\n ghmm = GaussianHMM(ghmm)\n return ghmm"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef discrete_hmm(pi, P, pout):\n from bhmm.hmm.discrete_hmm import DiscreteHMM\n from bhmm.output_models.discrete import DiscreteOutputModel\n # initialize output model\n output_model = DiscreteOutputModel(pout)\n # initialize general HMM\n from bhmm.hmm.generic_hmm import HMM as _HMM\n dhmm = _HMM(pi, P, output_model)\n # turn it into a Gaussian HMM\n dhmm = DiscreteHMM(dhmm)\n return dhmm", "response": "Initializes a discrete HMM with the given initial distribution P and pout."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef init_hmm(observations, nstates, lag=1, output=None, reversible=True):\n # select output model type\n if output is None:\n output = _guess_output_type(observations)\n\n if output == 'discrete':\n return init_discrete_hmm(observations, nstates, lag=lag, reversible=reversible)\n elif output == 'gaussian':\n return init_gaussian_hmm(observations, nstates, lag=lag, reversible=reversible)\n else:\n raise NotImplementedError('output model type '+str(output)+' not yet implemented.')", "response": "Use a heuristic scheme to generate an initial model for a set of states."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init_gaussian_hmm(observations, nstates, lag=1, reversible=True):\n from bhmm.init import gaussian\n if lag > 1:\n observations = lag_observations(observations, lag)\n hmm0 = gaussian.init_model_gaussian1d(observations, nstates, reversible=reversible)\n hmm0._lag = lag\n return hmm0", "response": "Generate a new initial model for a gaussian model."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef init_discrete_hmm(observations, nstates, lag=1, reversible=True, stationary=True, regularize=True,\n method='connect-spectral', separate=None):\n \"\"\"Use a heuristic scheme to generate an initial model.\n\n Parameters\n ----------\n observations : list of ndarray((T_i))\n list of arrays of length T_i with observation data\n nstates : int\n The number of states.\n lag : int\n Lag time at which the observations should be counted.\n reversible : bool\n Estimate reversible HMM transition matrix.\n stationary : bool\n p0 is the stationary distribution of P. Currently only reversible=True is implemented\n regularize : bool\n Regularize HMM probabilities to avoid 0's.\n method : str\n * 'lcs-spectral' : Does spectral clustering on the largest connected set\n of observed states.\n * 'connect-spectral' : Uses a weak regularization to connect the weakly\n connected sets and then initializes HMM using spectral clustering on\n the nonempty set.\n * 'spectral' : Uses spectral clustering on the nonempty subsets. Separated\n observed states will end up in separate hidden states. This option is\n only recommended for small observation spaces. Use connect-spectral for\n large observation spaces.\n separate : None or iterable of int\n Force the given set of observed states to stay in a separate hidden state.\n The remaining nstates-1 states will be assigned by a metastable decomposition.\n\n Examples\n --------\n\n Generate initial model for a discrete output model.\n\n >>> import bhmm\n >>> [model, observations, states] = bhmm.testsystems.generate_synthetic_observations(output='discrete')\n >>> initial_model = init_discrete_hmm(observations, model.nstates)\n\n \"\"\"\n import msmtools.estimation as msmest\n from bhmm.init.discrete import init_discrete_hmm_spectral\n C = msmest.count_matrix(observations, lag).toarray()\n # regularization\n if regularize:\n eps_A = None\n eps_B = None\n else:\n eps_A = 0\n eps_B = 0\n if not stationary:\n raise NotImplementedError('Discrete-HMM initialization with stationary=False is not yet implemented.')\n\n if method=='lcs-spectral':\n lcs = msmest.largest_connected_set(C)\n p0, P, B = init_discrete_hmm_spectral(C, nstates, reversible=reversible, stationary=stationary,\n active_set=lcs, separate=separate, eps_A=eps_A, eps_B=eps_B)\n elif method=='connect-spectral':\n # make sure we're strongly connected\n C += msmest.prior_neighbor(C, 0.001)\n nonempty = _np.where(C.sum(axis=0) + C.sum(axis=1) > 0)[0]\n C[nonempty, nonempty] = _np.maximum(C[nonempty, nonempty], 0.001)\n p0, P, B = init_discrete_hmm_spectral(C, nstates, reversible=reversible, stationary=stationary,\n active_set=nonempty, separate=separate, eps_A=eps_A, eps_B=eps_B)\n elif method=='spectral':\n p0, P, B = init_discrete_hmm_spectral(C, nstates, reversible=reversible, stationary=stationary,\n active_set=None, separate=separate, eps_A=eps_A, eps_B=eps_B)\n else:\n raise NotImplementedError('Unknown discrete-HMM initialization method ' + str(method))\n\n hmm0 = discrete_hmm(p0, P, B)\n hmm0._lag = lag\n return hmm0", "response": "Generate a discrete HMM for a given set of observations."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef estimate_hmm(observations, nstates, lag=1, initial_model=None, output=None,\n reversible=True, stationary=False, p=None, accuracy=1e-3, maxit=1000, maxit_P=100000,\n mincount_connectivity=1e-2):\n r\"\"\" Estimate maximum-likelihood HMM\n\n Generic maximum-likelihood estimation of HMMs\n\n Parameters\n ----------\n observations : list of numpy arrays representing temporal data\n `observations[i]` is a 1d numpy array corresponding to the observed trajectory index `i`\n nstates : int\n The number of states in the model.\n lag : int\n the lag time at which observations should be read\n initial_model : HMM, optional, default=None\n If specified, the given initial model will be used to initialize the BHMM.\n Otherwise, a heuristic scheme is used to generate an initial guess.\n output : str, optional, default=None\n Output model type from [None, 'gaussian', 'discrete']. If None, will automatically select an output\n model type based on the format of observations.\n reversible : bool, optional, default=True\n If True, a prior that enforces reversible transition matrices (detailed balance) is used;\n otherwise, a standard non-reversible prior is used.\n stationary : bool, optional, default=False\n If True, the initial distribution of hidden states is self-consistently computed as the stationary\n distribution of the transition matrix. If False, it will be estimated from the starting states.\n Only set this to true if you're sure that the observation trajectories are initiated from a global\n equilibrium distribution.\n p : ndarray (nstates), optional, default=None\n Initial or fixed stationary distribution. If given and stationary=True, transition matrices will be\n estimated with the constraint that they have p as their stationary distribution. If given and\n stationary=False, p is the fixed initial distribution of hidden states.\n accuracy : float\n convergence threshold for EM iteration. When two the likelihood does not increase by more than accuracy, the\n iteration is stopped successfully.\n maxit : int\n stopping criterion for EM iteration. When so many iterations are performed without reaching the requested\n accuracy, the iteration is stopped without convergence (a warning is given)\n\n Return\n ------\n hmm : :class:`HMM `\n\n \"\"\"\n # select output model type\n if output is None:\n output = _guess_output_type(observations)\n\n if lag > 1:\n observations = lag_observations(observations, lag)\n\n # construct estimator\n from bhmm.estimators.maximum_likelihood import MaximumLikelihoodEstimator as _MaximumLikelihoodEstimator\n est = _MaximumLikelihoodEstimator(observations, nstates, initial_model=initial_model, output=output,\n reversible=reversible, stationary=stationary, p=p, accuracy=accuracy,\n maxit=maxit, maxit_P=maxit_P)\n # run\n est.fit()\n # set lag time\n est.hmm._lag = lag\n # return model\n # TODO: package into specific class (DiscreteHMM, GaussianHMM)\n return est.hmm", "response": "r Estimates the maximum - likelihood of a given set of observations."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bayesian_hmm(observations, estimated_hmm, nsample=100, reversible=True, stationary=False,\n p0_prior='mixed', transition_matrix_prior='mixed', store_hidden=False, call_back=None):\n r\"\"\" Bayesian HMM based on sampling the posterior\n\n Generic maximum-likelihood estimation of HMMs\n\n Parameters\n ----------\n observations : list of numpy arrays representing temporal data\n `observations[i]` is a 1d numpy array corresponding to the observed trajectory index `i`\n estimated_hmm : HMM\n HMM estimated from estimate_hmm or initialize_hmm\n reversible : bool, optional, default=True\n If True, a prior that enforces reversible transition matrices (detailed balance) is used;\n otherwise, a standard non-reversible prior is used.\n stationary : bool, optional, default=False\n If True, the stationary distribution of the transition matrix will be used as initial distribution.\n Only use True if you are confident that the observation trajectories are started from a global\n equilibrium. If False, the initial distribution will be estimated as usual from the first step\n of the hidden trajectories.\n nsample : int, optional, default=100\n number of Gibbs sampling steps\n p0_prior : None, str, float or ndarray(n)\n Prior for the initial distribution of the HMM. Will only be active\n if stationary=False (stationary=True means that p0 is identical to\n the stationary distribution of the transition matrix).\n Currently implements different versions of the Dirichlet prior that\n is conjugate to the Dirichlet distribution of p0. p0 is sampled from:\n .. math:\n p0 \\sim \\prod_i (p0)_i^{a_i + n_i - 1}\n where :math:`n_i` are the number of times a hidden trajectory was in\n state :math:`i` at time step 0 and :math:`a_i` is the prior count.\n Following options are available:\n | 'mixed' (default), :math:`a_i = p_{0,init}`, where :math:`p_{0,init}`\n is the initial distribution of initial_model.\n | 'uniform', :math:`a_i = 1`\n | ndarray(n) or float,\n the given array will be used as A.\n | None, :math:`a_i = 0`. This option ensures coincidence between\n sample mean an MLE. Will sooner or later lead to sampling problems,\n because as soon as zero trajectories are drawn from a given state,\n the sampler cannot recover and that state will never serve as a starting\n state subsequently. Only recommended in the large data regime and\n when the probability to sample zero trajectories from any state\n is negligible.\n transition_matrix_prior : str or ndarray(n, n)\n Prior for the HMM transition matrix.\n Currently implements Dirichlet priors if reversible=False and reversible\n transition matrix priors as described in [1]_ if reversible=True. For the\n nonreversible case the posterior of transition matrix :math:`P` is:\n .. math:\n P \\sim \\prod_{i,j} p_{ij}^{b_{ij} + c_{ij} - 1}\n where :math:`c_{ij}` are the number of transitions found for hidden\n trajectories and :math:`b_{ij}` are prior counts.\n | 'mixed' (default), :math:`b_{ij} = p_{ij,init}`, where :math:`p_{ij,init}`\n is the transition matrix of initial_model. That means one prior\n count will be used per row.\n | 'uniform', :math:`b_{ij} = 1`\n | ndarray(n, n) or broadcastable,\n the given array will be used as B.\n | None, :math:`b_ij = 0`. This option ensures coincidence between\n sample mean an MLE. Will sooner or later lead to sampling problems,\n because as soon as a transition :math:`ij` will not occur in a\n sample, the sampler cannot recover and that transition will never\n be sampled again. This option is not recommended unless you have\n a small HMM and a lot of data.\n store_hidden : bool, optional, default=False\n store hidden trajectories in sampled HMMs\n call_back : function, optional, default=None\n a call back function with no arguments, which if given is being called\n after each computed sample. This is useful for implementing progress bars.\n\n Return\n ------\n hmm : :class:`SampledHMM `\n\n References\n ----------\n .. [1] Trendelkamp-Schroer, B., H. Wu, F. Paul and F. Noe:\n Estimation and uncertainty of reversible Markov models.\n J. Chem. Phys. 143, 174101 (2015).\n\n \"\"\"\n # construct estimator\n from bhmm.estimators.bayesian_sampling import BayesianHMMSampler as _BHMM\n sampler = _BHMM(observations, estimated_hmm.nstates, initial_model=estimated_hmm,\n reversible=reversible, stationary=stationary, transition_matrix_sampling_steps=1000,\n p0_prior=p0_prior, transition_matrix_prior=transition_matrix_prior,\n output=estimated_hmm.output_model.model_type)\n\n # Sample models.\n sampled_hmms = sampler.sample(nsamples=nsample, save_hidden_state_trajectory=store_hidden,\n call_back=call_back)\n # return model\n from bhmm.hmm.generic_sampled_hmm import SampledHMM\n return SampledHMM(estimated_hmm, sampled_hmms)", "response": "r Bayesian HMM based on sampling the observed data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the sum of arr assuming arr is in the log domain.", "response": "def logsumexp(arr, axis=0):\n \"\"\"Computes the sum of arr assuming arr is in the log domain.\n\n Returns log(sum(exp(arr))) while minimizing the possibility of\n over/underflow.\n\n Examples\n --------\n\n >>> import numpy as np\n >>> from sklearn.utils.extmath import logsumexp\n >>> a = np.arange(10)\n >>> np.log(np.sum(np.exp(a)))\n 9.4586297444267107\n >>> logsumexp(a)\n 9.4586297444267107\n \"\"\"\n arr = np.rollaxis(arr, axis)\n # Use the max to normalize, as with the log this is what accumulates\n # the less errors\n vmax = arr.max(axis=0)\n out = np.log(np.sum(np.exp(arr - vmax), axis=0))\n out += vmax\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _ensure_sparse_format(spmatrix, accept_sparse, dtype, order, copy,\n force_all_finite):\n \"\"\"Convert a sparse matrix to a given format.\n\n Checks the sparse format of spmatrix and converts if necessary.\n\n Parameters\n ----------\n spmatrix : scipy sparse matrix\n Input to validate and convert.\n\n accept_sparse : string, list of string or None (default=None)\n String[s] representing allowed sparse matrix formats ('csc',\n 'csr', 'coo', 'dok', 'bsr', 'lil', 'dia'). None means that sparse\n matrix input will raise an error. If the input is sparse but not in\n the allowed format, it will be converted to the first listed format.\n\n dtype : string, type or None (default=none)\n Data type of result. If None, the dtype of the input is preserved.\n\n order : 'F', 'C' or None (default=None)\n Whether an array will be forced to be fortran or c-style.\n\n copy : boolean (default=False)\n Whether a forced copy will be triggered. If copy=False, a copy might\n be triggered by a conversion.\n\n force_all_finite : boolean (default=True)\n Whether to raise an error on np.inf and np.nan in X.\n\n Returns\n -------\n spmatrix_converted : scipy sparse matrix.\n Matrix that is ensured to have an allowed type.\n \"\"\"\n if accept_sparse is None:\n raise TypeError('A sparse matrix was passed, but dense '\n 'data is required. Use X.toarray() to '\n 'convert to a dense numpy array.')\n sparse_type = spmatrix.format\n if dtype is None:\n dtype = spmatrix.dtype\n if sparse_type in accept_sparse:\n # correct type\n if dtype == spmatrix.dtype:\n # correct dtype\n if copy:\n spmatrix = spmatrix.copy()\n else:\n # convert dtype\n spmatrix = spmatrix.astype(dtype)\n else:\n # create new\n spmatrix = spmatrix.asformat(accept_sparse[0]).astype(dtype)\n if force_all_finite:\n if not hasattr(spmatrix, \"data\"):\n warnings.warn(\"Can't check %s sparse matrix for nan or inf.\"\n % spmatrix.format)\n else:\n _assert_all_finite(spmatrix.data)\n if hasattr(spmatrix, \"data\"):\n spmatrix.data = np.array(spmatrix.data, copy=False, order=order)\n return spmatrix", "response": "Convert a sparse matrix to a given format."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_array(array, accept_sparse=None, dtype=\"numeric\", order=None,\n copy=False, force_all_finite=True, ensure_2d=True,\n allow_nd=False, ensure_min_samples=1, ensure_min_features=1):\n \"\"\"Input validation on an array, list, sparse matrix or similar.\n\n By default, the input is converted to an at least 2nd numpy array.\n If the dtype of the array is object, attempt converting to float,\n raising on failure.\n\n Parameters\n ----------\n array : object\n Input object to check / convert.\n\n accept_sparse : string, list of string or None (default=None)\n String[s] representing allowed sparse matrix formats, such as 'csc',\n 'csr', etc. None means that sparse matrix input will raise an error.\n If the input is sparse but not in the allowed format, it will be\n converted to the first listed format.\n\n dtype : string, type or None (default=\"numeric\")\n Data type of result. If None, the dtype of the input is preserved.\n If \"numeric\", dtype is preserved unless array.dtype is object.\n\n order : 'F', 'C' or None (default=None)\n Whether an array will be forced to be fortran or c-style.\n\n copy : boolean (default=False)\n Whether a forced copy will be triggered. If copy=False, a copy might\n be triggered by a conversion.\n\n force_all_finite : boolean (default=True)\n Whether to raise an error on np.inf and np.nan in X.\n\n ensure_2d : boolean (default=True)\n Whether to make X at least 2d.\n\n allow_nd : boolean (default=False)\n Whether to allow X.ndim > 2.\n\n ensure_min_samples : int (default=1)\n Make sure that the array has a minimum number of samples in its first\n axis (rows for a 2D array). Setting to 0 disables this check.\n\n ensure_min_features : int (default=1)\n Make sure that the 2D array has some minimum number of features\n (columns). The default value of 1 rejects empty datasets.\n This check is only enforced when the input data has effectively 2\n dimensions or is originally 1D and ``ensure_2d`` is True. Setting to 0\n disables this check.\n\n Returns\n -------\n X_converted : object\n The converted and validated X.\n \"\"\"\n if isinstance(accept_sparse, str):\n accept_sparse = [accept_sparse]\n\n # store whether originally we wanted numeric dtype\n dtype_numeric = dtype == \"numeric\"\n\n if sp.issparse(array):\n if dtype_numeric:\n dtype = None\n array = _ensure_sparse_format(array, accept_sparse, dtype, order,\n copy, force_all_finite)\n else:\n if ensure_2d:\n array = np.atleast_2d(array)\n if dtype_numeric:\n if hasattr(array, \"dtype\") and getattr(array.dtype, \"kind\", None) == \"O\":\n # if input is object, convert to float.\n dtype = np.float64\n else:\n dtype = None\n array = np.array(array, dtype=dtype, order=order, copy=copy)\n # make sure we actually converted to numeric:\n if dtype_numeric and array.dtype.kind == \"O\":\n array = array.astype(np.float64)\n if not allow_nd and array.ndim >= 3:\n raise ValueError(\"Found array with dim %d. Expected <= 2\" %\n array.ndim)\n if force_all_finite:\n _assert_all_finite(array)\n\n shape_repr = _shape_repr(array.shape)\n if ensure_min_samples > 0:\n n_samples = _num_samples(array)\n if n_samples < ensure_min_samples:\n raise ValueError(\"Found array with %d sample(s) (shape=%s) while a\"\n \" minimum of %d is required.\"\n % (n_samples, shape_repr, ensure_min_samples))\n\n if ensure_min_features > 0 and array.ndim == 2:\n n_features = array.shape[1]\n if n_features < ensure_min_features:\n raise ValueError(\"Found array with %d feature(s) (shape=%s) while\"\n \" a minimum of %d is required.\"\n % (n_features, shape_repr, ensure_min_features))\n return array", "response": "Input validation on an array."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef beta_confidence_intervals(ci_X, ntrials, ci=0.95):\n # Compute low and high confidence interval for symmetric CI about mean.\n ci_low = 0.5 - ci/2;\n ci_high = 0.5 + ci/2;\n\n # Compute for every element of ci_X.\n from scipy.stats import beta\n Plow = ci_X * 0.0;\n Phigh = ci_X * 0.0;\n for i in range(ci_X.shape[0]):\n for j in range(ci_X.shape[1]):\n Plow[i,j] = beta.ppf(ci_low, a = ci_X[i,j] * ntrials, b = (1-ci_X[i,j]) * ntrials);\n Phigh[i,j] = beta.ppf(ci_high, a = ci_X[i,j] * ntrials, b = (1-ci_X[i,j]) * ntrials);\n\n return [Plow, Phigh]", "response": "Returns a random set of confidence intervals for beta distributions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef empirical_confidence_interval(sample, interval=0.95):\n # Sort sample in increasing order.\n sample = np.sort(sample)\n\n # Determine sample size.\n N = len(sample)\n\n # Compute low and high indices.\n low_index = int(np.round((N-1) * (0.5 - interval/2))) + 1\n high_index = int(np.round((N-1) * (0.5 + interval/2))) + 1\n\n # Compute low and high.\n low = sample[low_index]\n high = sample[high_index]\n\n return [low, high]", "response": "Compute specified symmetric confidence interval for empirical sample."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_latex_table(sampled_hmm, conf=0.95, dt=1, time_unit='ms', obs_name='force', obs_units='pN',\n caption='', outfile=None):\n \"\"\"\n Generate a LaTeX column-wide table showing various computed properties and uncertainties.\n\n Parameters\n ----------\n conf : float\n confidence interval. Use 0.68 for 1 sigma, 0.95 for 2 sigma etc.\n\n \"\"\"\n # check input\n from bhmm.hmm.generic_sampled_hmm import SampledHMM\n from bhmm.hmm.gaussian_hmm import SampledGaussianHMM\n assert issubclass(sampled_hmm.__class__, SampledHMM), 'sampled_hmm ist not a SampledHMM'\n\n # confidence interval\n sampled_hmm.set_confidence(conf)\n # dt\n dt = float(dt)\n # nstates\n nstates = sampled_hmm.nstates\n\n table = \"\"\"\n\\\\begin{table}\n \\\\begin{tabular*}{\\columnwidth}{@{\\extracolsep{\\\\fill}}lcc}\n \\hline\n {\\\\bf Property} & {\\\\bf Symbol} & {\\\\bf Value} \\\\\\\\\n \\hline\n \"\"\"\n # Stationary probability.\n p = sampled_hmm.stationary_distribution_mean\n p_lo, p_hi = sampled_hmm.stationary_distribution_conf\n for i in range(nstates):\n if i == 0:\n table += '\\t\\tEquilibrium probability '\n table += '\\t\\t& $\\pi_{%d}$ & $%0.3f_{\\:%0.3f}^{\\:%0.3f}$ \\\\\\\\' % (i+1, p[i], p_lo[i], p_hi[i]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n # Transition probabilities.\n P = sampled_hmm.transition_matrix_mean\n P_lo, P_hi = sampled_hmm.transition_matrix_conf\n for i in range(nstates):\n for j in range(nstates):\n if i == 0 and j == 0:\n table += '\\t\\tTransition probability ($\\Delta t = $%s) ' % (str(dt)+' '+time_unit)\n table += '\\t\\t& $T_{%d%d}$ & $%0.4f_{\\:%0.4f}^{\\:%0.4f}$ \\\\\\\\' % (i+1, j+1, P[i, j], P_lo[i, j], P_hi[i, j]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n # Transition rates via pseudogenerator.\n K = P - np.eye(sampled_hmm.nstates)\n K /= dt\n K_lo = P_lo - np.eye(sampled_hmm.nstates)\n K_lo /= dt\n K_hi = P_hi - np.eye(sampled_hmm.nstates)\n K_hi /= dt\n for i in range(nstates):\n for j in range(nstates):\n if i == 0 and j == 0:\n table += '\\t\\tTransition rate (%s$^{-1}$) ' % time_unit\n if i != j:\n table += '\\t\\t& $k_{%d%d}$ & $%2.4f_{\\:%2.4f}^{\\:%2.4f}$ \\\\\\\\' % (i+1, j+1, K[i, j], K_lo[i, j], K_hi[i, j]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n # State mean lifetimes.\n l = sampled_hmm.lifetimes_mean\n l *= dt\n l_lo, l_hi = sampled_hmm.lifetimes_conf\n l_lo *= dt\n l_hi *= dt\n for i in range(nstates):\n if i == 0:\n table += '\\t\\tState mean lifetime (%s) ' % time_unit\n table += '\\t\\t& $t_{%d}$ & $%.3f_{\\:%.3f}^{\\:%.3f}$ \\\\\\\\' % (i+1, l[i], l_lo[i], l_hi[i]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n # State relaxation timescales.\n t = sampled_hmm.timescales_mean\n t *= dt\n t_lo, t_hi = sampled_hmm.timescales_conf\n t_lo *= dt\n t_hi *= dt\n for i in range(nstates-1):\n if i == 0:\n table += '\\t\\tRelaxation time (%s) ' % time_unit\n table += '\\t\\t& $\\\\tau_{%d}$ & $%.3f_{\\:%.3f}^{\\:%.3f}$ \\\\\\\\' % (i+1, t[i], t_lo[i], t_hi[i]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n if issubclass(sampled_hmm.__class__, SampledGaussianHMM):\n table += '\\t\\t\\hline' + '\\n'\n\n # State mean forces.\n m = sampled_hmm.means_mean\n m_lo, m_hi = sampled_hmm.means_conf\n for i in range(nstates):\n if i == 0:\n table += '\\t\\tState %s mean (%s) ' % (obs_name, obs_units)\n table += '\\t\\t& $\\mu_{%d}$ & $%.3f_{\\:%.3f}^{\\:%.3f}$ \\\\\\\\' % (i+1, m[i], m_lo[i], m_hi[i]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n # State force standard deviations.\n s = sampled_hmm.sigmas_mean\n s_lo, s_hi = sampled_hmm.sigmas_conf\n for i in range(nstates):\n if i == 0:\n table += '\\t\\tState %s std dev (%s) ' % (obs_name, obs_units)\n table += '\\t\\t& $s_{%d}$ & $%.3f_{\\:%.3f}^{\\:%.3f}$ \\\\\\\\' % (i+1, s[i], s_lo[i], s_hi[i]) + '\\n'\n table += '\\t\\t\\hline' + '\\n'\n\n table += \"\"\"\n \\\\hline\n \\\\end{tabular*}\n \\\\caption{{\\\\bf %s}}\n\\\\end{table}\n \"\"\" % caption\n\n # write to file if wanted:\n if outfile is not None:\n f = open(outfile, 'w')\n f.write(table)\n f.close()\n\n return table", "response": "Generates LaTeX table for the given SampledHMM."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the mean and alpha - confidence interval of the given sample set.", "response": "def confidence_interval(data, alpha):\n \"\"\"\n Computes the mean and alpha-confidence interval of the given sample set\n\n Parameters\n ----------\n data : ndarray\n a 1D-array of samples\n alpha : float in [0,1]\n the confidence level, i.e. percentage of data included in the interval\n \n Returns\n -------\n [m,l,r] where m is the mean of the data, and (l,r) are the m-alpha/2 and m+alpha/2 \n confidence interval boundaries.\n \"\"\"\n if alpha < 0 or alpha > 1:\n raise ValueError('Not a meaningful confidence level: '+str(alpha))\n \n # compute mean\n m = np.mean(data)\n # sort data\n sdata = np.sort(data)\n # index of the mean\n im = np.searchsorted(sdata, m)\n if im == 0 or im == len(sdata):\n pm = im\n else:\n pm = (im-1) + (m-sdata[im-1]) / (sdata[im]-sdata[im-1])\n # left interval boundary\n pl = pm - alpha * pm\n il1 = max(0, int(math.floor(pl)))\n il2 = min(len(sdata)-1, int(math.ceil(pl)))\n l = sdata[il1] + (pl - il1)*(sdata[il2] - sdata[il1])\n # right interval boundary\n pr = pm + alpha * (len(data)-im)\n ir1 = max(0, int(math.floor(pr)))\n ir2 = min(len(sdata)-1, int(math.ceil(pr)))\n r = sdata[ir1] + (pr - ir1)*(sdata[ir2] - sdata[ir1])\n\n # return\n return m, l, r"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the connection status both locally and remotely.", "response": "def status(self, remote=False):\n \"\"\"\n Return the connection status, both locally and remotely.\n\n The local connection status is a dictionary that gives:\n * the count of multiple queries sent to the server.\n * the count of single queries sent to the server.\n * the count of actions sent to the server.\n * the count of actions executed successfully by the server.\n * the count of actions queued to go to the server.\n\n The remote connection status includes whether the server is live,\n as well as data about version and build. The server data is\n cached, unless the remote flag is specified.\n\n :param remote: whether to query the server for its latest status\n :return: tuple of status dicts: (local, server).\n \"\"\"\n if remote:\n components = urlparse.urlparse(self.endpoint)\n try:\n result = self.session.get(components[0] + \"://\" + components[1] + \"/status\", timeout=self.timeout)\n except Exception as e:\n if self.logger: self.logger.debug(\"Failed to connect to server for status: %s\", e)\n result = None\n if result and result.status_code == 200:\n self.server_status = result.json()\n self.server_status[\"endpoint\"] = self.endpoint\n elif result:\n if self.logger: self.logger.debug(\"Server status response not understandable: Status: %d, Body: %s\",\n result.status_code, result.text)\n self.server_status = {\"endpoint\": self.endpoint,\n \"status\": (\"Unexpected HTTP status \" + str(result.status_code) + \" at: \" +\n strftime(\"%d %b %Y %H:%M:%S +0000\", gmtime()))}\n else:\n self.server_status = {\"endpoint\": self.endpoint,\n \"status\": \"Unreachable at: \" + strftime(\"%d %b %Y %H:%M:%S +0000\", gmtime())}\n return self.local_status, self.server_status"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef query_single(self, object_type, url_params, query_params=None):\n # type: (str, list, dict) -> dict\n \"\"\"\n Query for a single object.\n :param object_type: string query type (e.g., \"users\" or \"groups\")\n :param url_params: required list of strings to provide as additional URL components\n :param query_params: optional dictionary of query options\n :return: the found object (a dictionary), which is empty if none were found\n \"\"\"\n # Server API convention (v2) is that the pluralized object type goes into the endpoint\n # but the object type is the key in the response dictionary for the returned object.\n self.local_status[\"single-query-count\"] += 1\n query_type = object_type + \"s\" # poor man's plural\n query_path = \"/organizations/{}/{}\".format(self.org_id, query_type)\n for component in url_params if url_params else []:\n query_path += \"/\" + urlparse.quote(component, safe='/@')\n if query_params: query_path += \"?\" + urlparse.urlencode(query_params)\n try:\n result = self.make_call(query_path)\n body = result.json()\n except RequestError as re:\n if re.result.status_code == 404:\n if self.logger: self.logger.debug(\"Ran %s query: %s %s (0 found)\",\n object_type, url_params, query_params)\n return {}\n else:\n raise re\n if body.get(\"result\") == \"success\":\n value = body.get(object_type, {})\n if self.logger: self.logger.debug(\"Ran %s query: %s %s (1 found)\", object_type, url_params, query_params)\n return value\n else:\n raise ClientError(\"OK status but no 'success' result\", result)", "response": "Query for a single object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef query_multiple(self, object_type, page=0, url_params=None, query_params=None):\n # type: (str, int, list, dict) -> tuple\n \"\"\"\n Query for a page of objects. Defaults to the (0-based) first page.\n Sadly, the sort order is undetermined.\n :param object_type: string constant query type: either \"user\" or \"group\")\n :param page: numeric page (0-based) of results to get (up to 200 in a page)\n :param url_params: optional list of strings to provide as additional URL components\n :param query_params: optional dictionary of query options\n :return: tuple (list of returned dictionaries (one for each query result), bool for whether this is last page)\n \"\"\"\n # As of 2017-10-01, we are moving to to different URLs for user and user-group queries,\n # and these endpoints have different conventions for pagination. For the time being,\n # we are also preserving the more general \"group\" query capability.\n self.local_status[\"multiple-query-count\"] += 1\n if object_type in (\"user\", \"group\"):\n query_path = \"/{}s/{}/{:d}\".format(object_type, self.org_id, page)\n if url_params: query_path += \"/\" + \"/\".join([urlparse.quote(c) for c in url_params])\n if query_params: query_path += \"?\" + urlparse.urlencode(query_params)\n elif object_type == \"user-group\":\n query_path = \"/{}/user-groups\".format(self.org_id)\n if url_params: query_path += \"/\" + \"/\".join([urlparse.quote(c) for c in url_params])\n query_path += \"?page={:d}\".format(page+1)\n if query_params: query_path += \"&\" + urlparse.urlencode(query_params)\n else:\n raise ArgumentError(\"Unknown query object type ({}): must be 'user' or 'group'\".format(object_type))\n try:\n result = self.make_call(query_path)\n body = result.json()\n except RequestError as re:\n if re.result.status_code == 404:\n if self.logger: self.logger.debug(\"Ran %s query: %s %s (0 found)\",\n object_type, url_params, query_params)\n return [], True\n else:\n raise re\n if object_type in (\"user\", \"group\"):\n if body.get(\"result\") == \"success\":\n values = body.get(object_type + \"s\", [])\n last_page = body.get(\"lastPage\", False)\n if self.logger: self.logger.debug(\"Ran multi-%s query: %s %s (page %d: %d found)\",\n object_type, url_params, query_params, page, len(values))\n return values, last_page\n else:\n raise ClientError(\"OK status but no 'success' result\", result)\n elif object_type == \"user-group\":\n page_number = result.headers.get(\"X-Current-Page\", \"1\")\n page_count = result.headers.get(\"X-Page-Count\", \"1\")\n if self.logger: self.logger.debug(\"Ran multi-group query: %s %s (page %d: %d found)\",\n url_params, query_params, page, len(body))\n return body, int(page_number) >= int(page_count)\n else:\n # this would actually be caught above, but we use a parallel construction in both places\n # to make it easy to add query object types\n raise ArgumentError(\"Unknown query object type ({}): must be 'user' or 'group'\".format(object_type))", "response": "Query for multiple objects."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute a single action on a single object.", "response": "def execute_single(self, action, immediate=False):\n \"\"\"\n Execute a single action (containing commands on a single object).\n Normally, since actions are batched so as to be most efficient about execution,\n but if you want this command sent immediately (and all prior queued commands\n sent earlier in this command's batch), specify a True value for the immediate flag.\n\n Since any command can fill the current batch, your command may be submitted\n even if you don't specify the immediate flag. So don't think of this always\n being a queue call if immedidate=False.\n\n Returns the number of actions in the queue, that got sent, and that executed successfully.\n\n :param action: the Action to be executed\n :param immediate: whether the Action should be executed immediately\n :return: the number of actions in the queue, that got sent, and that executed successfully.\n \"\"\"\n return self.execute_multiple([action], immediate=immediate)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes multiple actions on a single object.", "response": "def execute_multiple(self, actions, immediate=True):\n \"\"\"\n Execute multiple Actions (each containing commands on a single object).\n Normally, the actions are sent for execution immediately (possibly preceded\n by earlier queued commands), but if you are going for maximum efficiency\n you can set immeediate=False which will cause the connection to wait\n and batch as many actions as allowed in each server call.\n\n Since any command can fill the current batch, one or more of your commands may be submitted\n even if you don't specify the immediate flag. So don't think of this call as always\n being a queue call when immedidate=False.\n\n Returns the number of actions left in the queue, that got sent, and that executed successfully.\n\n NOTE: This is where we throttle the number of commands per action. So the number\n of actions we were given may not be the same as the number we queue or send to the server.\n \n NOTE: If the server gives us a response we don't understand, we note that and continue\n processing as usual. Then, at the end of the batch, we throw in order to warn the client\n that we had a problem understanding the server.\n\n :param actions: the list of Action objects to be executed\n :param immediate: whether to immediately send them to the server\n :return: tuple: the number of actions in the queue, that got sent, and that executed successfully.\n \"\"\"\n # throttling part 1: split up each action into smaller actions, as needed\n # optionally split large lists of groups in add/remove commands (if action supports it)\n split_actions = []\n exceptions = []\n for a in actions:\n if len(a.commands) == 0:\n if self.logger: self.logger.warning(\"Sending action with no commands: %s\", a.frame)\n # maybe_split_groups is a UserAction attribute, so the call may throw an AttributeError\n try:\n if a.maybe_split_groups(self.throttle_groups):\n if self.logger: self.logger.debug(\n \"Throttling actions %s to have a maximum of %d entries in group lists.\",\n a.frame, self.throttle_groups)\n except AttributeError:\n pass\n if len(a.commands) > self.throttle_commands:\n if self.logger: self.logger.debug(\"Throttling action %s to have a maximum of %d commands.\",\n a.frame, self.throttle_commands)\n split_actions += a.split(self.throttle_commands)\n else:\n split_actions.append(a)\n actions = self.action_queue + split_actions\n # throttling part 2: execute the action list in batches, as needed\n sent = completed = 0\n batch_size = self.throttle_actions\n min_size = 1 if immediate else batch_size\n while len(actions) >= min_size:\n batch, actions = actions[0:batch_size], actions[batch_size:]\n if self.logger: self.logger.debug(\"Executing %d actions (%d remaining).\", len(batch), len(actions))\n sent += len(batch)\n try:\n completed += self._execute_batch(batch)\n except Exception as e:\n exceptions.append(e)\n self.action_queue = actions\n self.local_status[\"actions-queued\"] = queued = len(actions)\n self.local_status[\"actions-sent\"] += sent\n self.local_status[\"actions-completed\"] += completed\n if exceptions:\n raise BatchError(exceptions, queued, sent, completed)\n return queued, sent, completed"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes a batch of Actions.", "response": "def _execute_batch(self, actions):\n \"\"\"\n Execute a single batch of Actions.\n For each action that has a problem, we annotate the action with the\n error information for that action, and we return the number of\n successful actions in the batch.\n :param actions: the list of Action objects to be executed\n :return: count of successful actions\n \"\"\"\n wire_form = [a.wire_dict() for a in actions]\n if self.test_mode:\n result = self.make_call(\"/action/%s?testOnly=true\" % self.org_id, wire_form)\n else:\n result = self.make_call(\"/action/%s\" % self.org_id, wire_form)\n body = result.json()\n if body.get(\"errors\", None) is None:\n if body.get(\"result\") != \"success\":\n if self.logger: self.logger.warning(\"Server action result: no errors, but no success:\\n%s\", body)\n return len(actions)\n try:\n if body.get(\"result\") == \"success\":\n if self.logger: self.logger.warning(\"Server action result: errors, but success report:\\n%s\", body)\n for error in body[\"errors\"]:\n actions[error[\"index\"]].report_command_error(error)\n except:\n raise ClientError(str(body), result)\n return body.get(\"completed\", 0)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_call(self, path, body=None, delete=False):\n if body:\n request_body = json.dumps(body)\n def call():\n return self.session.post(self.endpoint + path, auth=self.auth, data=request_body, timeout=self.timeout)\n else:\n if not delete:\n def call():\n return self.session.get(self.endpoint + path, auth=self.auth, timeout=self.timeout)\n else:\n def call():\n return self.session.delete(self.endpoint + path, auth=self.auth, timeout=self.timeout)\n\n start_time = time()\n result = None\n for num_attempts in range(1, self.retry_max_attempts + 1):\n try:\n result = call()\n if result.status_code in [200,201,204]:\n return result\n elif result.status_code in [429, 502, 503, 504]:\n if self.logger: self.logger.warning(\"UMAPI timeout...service unavailable (code %d on try %d)\",\n result.status_code, num_attempts)\n retry_wait = 0\n if \"Retry-After\" in result.headers:\n advice = result.headers[\"Retry-After\"]\n advised_time = parsedate_tz(advice)\n if advised_time is not None:\n # header contains date\n retry_wait = int(mktime_tz(advised_time) - time())\n else:\n # header contains delta seconds\n retry_wait = int(advice)\n if retry_wait <= 0:\n # use exponential back-off with random delay\n delay = randint(0, self.retry_random_delay)\n retry_wait = (int(pow(2, num_attempts - 1)) * self.retry_first_delay) + delay\n elif 201 <= result.status_code < 400:\n raise ClientError(\"Unexpected HTTP Status {:d}: {}\".format(result.status_code, result.text), result)\n elif 400 <= result.status_code < 500:\n raise RequestError(result)\n else:\n raise ServerError(result)\n except requests.Timeout:\n if self.logger: self.logger.warning(\"UMAPI connection timeout...(%d seconds on try %d)\",\n self.timeout, num_attempts)\n retry_wait = 0\n result = None\n if num_attempts < self.retry_max_attempts:\n if retry_wait > 0:\n if self.logger: self.logger.warning(\"Next retry in %d seconds...\", retry_wait)\n sleep(retry_wait)\n else:\n if self.logger: self.logger.warning(\"Immediate retry...\")\n total_time = int(time() - start_time)\n if self.logger: self.logger.error(\"UMAPI timeout...giving up after %d attempts (%d seconds).\",\n self.retry_max_attempts, total_time)\n raise UnavailableError(self.retry_max_attempts, total_time, result)", "response": "Make a single UMAPI call with error handling and retry on temporary failure."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef paginate(query, org_id, max_pages=maxsize, max_records=maxsize):\n page_count = 0\n record_count = 0\n records = []\n while page_count < max_pages and record_count < max_records:\n res = make_call(query, org_id, page_count)\n page_count += 1\n # the following incredibly ugly piece of code is very fragile.\n # the problem is that we are a \"dumb helper\" that doesn't understand\n # the semantics of the UMAPI or know which query we were given.\n if \"groups\" in res:\n records += res[\"groups\"]\n elif \"users\" in res:\n records += res[\"users\"]\n record_count = len(records)\n if res.get(\"lastPage\"):\n break\n return records", "response": "Paginate through all results of a UMAPI query\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmake a single UMAPI call with error handling and server - controlled throttling.", "response": "def make_call(query, org_id, page):\n \"\"\"\n Make a single UMAPI call with error handling and server-controlled throttling.\n (Adapted from sample code at https://www.adobe.io/products/usermanagement/docs/samples#retry)\n :param query: a query method from a UMAPI instance (callable as a function)\n :param org_id: the organization being queried\n :param page: the page number of the desired result set\n :return: the json (dictionary) received from the server (if any)\n \"\"\"\n wait_time = 0\n num_attempts = 0\n\n while num_attempts < retry_max_attempts:\n if wait_time > 0:\n sleep(wait_time)\n wait_time = 0\n try:\n num_attempts += 1\n return query(org_id, page)\n except UMAPIRetryError as e:\n logger.warning(\"UMAPI service temporarily unavailable (attempt %d) -- %s\", num_attempts, e.res.status_code)\n if \"Retry-After\" in e.res.headers:\n advice = e.res.headers[\"Retry-After\"]\n advised_time = parsedate_tz(advice)\n if advised_time is not None:\n # header contains date\n wait_time = int(mktime_tz(advised_time) - time())\n else:\n # header contains delta seconds\n wait_time = int(advice)\n if wait_time <= 0:\n # use exponential back-off with random delay\n delay = randint(0, retry_random_delay_max)\n wait_time = (int(pow(2, num_attempts)) * retry_exponential_backoff_factor) + delay\n logger.warning(\"Next retry in %d seconds...\", wait_time)\n continue\n except UMAPIRequestError as e:\n logger.warning(\"UMAPI error processing request -- %s\", e.code)\n return {}\n except UMAPIError as e:\n logger.warning(\"HTTP error processing request -- %s: %s\", e.res.status_code, e.res.text)\n return {}\n logger.error(\"UMAPI timeout...giving up on results page %d after %d attempts.\", page, retry_max_attempts)\n return {}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef do(self, **kwargs):\n # add \"create\" / \"add\" / \"removeFrom\" first\n for k, v in list(six.iteritems(kwargs)):\n if k.startswith(\"create\") or k.startswith(\"addAdobe\") or k.startswith(\"removeFrom\"):\n self.commands.append({k: v})\n del kwargs[k]\n\n # now do the other actions, in a canonical order (to avoid py2/py3 variations)\n for k, v in sorted(six.iteritems(kwargs)):\n if k in ['add', 'remove']:\n self.commands.append({k: {\"product\": v}})\n else:\n self.commands.append({k: v})\n return self", "response": "This method is used to add commands to the list of available user - defined commands."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef split(self, max_commands):\n a_prior = Action(**self.frame)\n a_prior.commands = list(self.commands)\n self.split_actions = [a_prior]\n while len(a_prior.commands) > max_commands:\n a_next = Action(**self.frame)\n a_prior.commands, a_next.commands = a_prior.commands[0:max_commands], a_prior.commands[max_commands:]\n self.split_actions.append(a_next)\n a_prior = a_next\n return self.split_actions", "response": "Split this action into an equivalent list of actions each of which have at most max_commands commands."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd commands at the end of the sequence.", "response": "def append(self, **kwargs):\n \"\"\"\n Add commands at the end of the sequence.\n\n Be careful: because this runs in Python 2.x, the order of the kwargs dict may not match\n the order in which the args were specified. Thus, if you care about specific ordering,\n you must make multiple calls to append in that order. Luckily, append returns\n the Action so you can compose easily: Action(...).append(...).append(...).\n See also insert, below.\n :param kwargs: the key/value pairs to add\n :return: the action\n \"\"\"\n for k, v in six.iteritems(kwargs):\n self.commands.append({k: v})\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef insert(self, **kwargs):\n for k, v in six.iteritems(kwargs):\n self.commands.insert(0, {k: v})\n return self", "response": "Insert commands at the beginning of the sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreport a server error executing a command.", "response": "def report_command_error(self, error_dict):\n \"\"\"\n Report a server error executing a command.\n\n We keep track of the command's position in the command list,\n and we add annotation of what the command was, to the error.\n :param error_dict: The server's error dict for the error encountered\n \"\"\"\n error = dict(error_dict)\n error[\"command\"] = self.commands[error_dict[\"step\"]]\n error[\"target\"] = self.frame\n del error[\"index\"] # throttling can change which action this was in the batch\n del error[\"step\"] # throttling can change which step this was in the action\n self.errors.append(error)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of commands that encountered execution errors with the error.", "response": "def execution_errors(self):\n \"\"\"\n Return a list of commands that encountered execution errors, with the error.\n\n Each dictionary entry gives the command dictionary and the error dictionary\n :return: list of commands that gave errors, with their error information\n \"\"\"\n if self.split_actions:\n # throttling split this action, get errors from the split\n return [dict(e) for s in self.split_actions for e in s.errors]\n else:\n return [dict(e) for e in self.errors]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef maybe_split_groups(self, max_groups):\n split_commands = []\n # return True if we split at least once\n maybe_split = False\n valid_step_keys = ['add', 'addRoles', 'remove']\n for command in self.commands:\n # commands are assumed to contain a single key\n step_key, step_args = next(six.iteritems(command))\n if step_key not in valid_step_keys or not isinstance(step_args, dict):\n split_commands.append(command)\n continue\n new_commands = [command]\n while True:\n new_command = {step_key: {}}\n for group_type, groups in six.iteritems(command[step_key]):\n if len(groups) > max_groups:\n command[step_key][group_type], new_command[step_key][group_type] = \\\n groups[0:max_groups], groups[max_groups:]\n if new_command[step_key]:\n new_commands.append(new_command)\n command = new_command\n maybe_split = True\n else:\n break\n split_commands += new_commands\n self.commands = split_commands\n return maybe_split", "response": "Check if group lists in add or remove directives should be split and split them if needed."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reload(self):\n self._results = []\n self._next_item_index = 0\n self._next_page_index = 0\n self._last_page_seen = False", "response": "Reload the results from the server side."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfetch the next page of the query.", "response": "def _next_page(self):\n \"\"\"\n Fetch the next page of the query.\n \"\"\"\n if self._last_page_seen:\n raise StopIteration\n new, self._last_page_seen = self.conn.query_multiple(self.object_type, self._next_page_index,\n self.url_params, self.query_params)\n self._next_page_index += 1\n if len(new) == 0:\n self._last_page_seen = True # don't bother with next page if nothing was returned\n else:\n self._results += new"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef all_results(self):\n while not self._last_page_seen:\n self._next_page()\n self._next_item_index = len(self._results)\n return list(self._results)", "response": "Eagerly fetch all the results."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _fetch_result(self):\n self._result = self.conn.query_single(self.object_type, self.url_params, self.query_params)", "response": "Fetch the object from the server."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create(self, first_name=None, last_name=None, country=None, email=None,\n on_conflict=IfAlreadyExistsOptions.ignoreIfAlreadyExists):\n \"\"\"\n Create the user on the Adobe back end.\n See [Issue 32](https://github.com/adobe-apiplatform/umapi-client.py/issues/32): because\n we retry create calls if they time out, the default conflict handling for creation is to ignore the\n create call if the user already exists (possibly from an earlier call that timed out).\n :param first_name: (optional) user first name\n :param last_name: (optional) user last name\n :param country: (optional except for Federated ID) user 2-letter ISO country code\n :param email: user email, if not already specified at create time\n :param on_conflict: IfAlreadyExistsOption (or string name thereof) controlling creation of existing users\n :return: the User, so you can do User(...).create(...).add_to_groups(...)\n \"\"\"\n # first validate the params: email, on_conflict, first_name, last_name, country\n create_params = {}\n if email is None:\n if not self.email:\n raise ArgumentError(\"You must specify email when creating a user\")\n elif self.email is None:\n self.email = email\n elif self.email.lower() != email.lower():\n raise ArgumentError(\"Specified email (%s) doesn't match user's email (%s)\" % (email, self.email))\n create_params[\"email\"] = self.email\n if on_conflict in IfAlreadyExistsOptions.__members__:\n on_conflict = IfAlreadyExistsOptions[on_conflict]\n if on_conflict not in IfAlreadyExistsOptions:\n raise ArgumentError(\"on_conflict must be one of {}\".format([o.name for o in IfAlreadyExistsOptions]))\n if on_conflict != IfAlreadyExistsOptions.errorIfAlreadyExists:\n create_params[\"option\"] = on_conflict.name\n if first_name: create_params[\"firstname\"] = first_name # per issue #54 now allowed for all identity types\n if last_name: create_params[\"lastname\"] = last_name # per issue #54 now allowed for all identity types\n if country: create_params[\"country\"] = country # per issue #55 should not be defaulted\n\n # each type is created using a different call\n if self.id_type == IdentityTypes.adobeID:\n return self.insert(addAdobeID=dict(**create_params))\n elif self.id_type == IdentityTypes.enterpriseID:\n return self.insert(createEnterpriseID=dict(**create_params))\n else:\n return self.insert(createFederatedID=dict(**create_params))", "response": "Create a user on the Adobe back end."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the user s attributes.", "response": "def update(self, email=None, username=None, first_name=None, last_name=None, country=None):\n \"\"\"\n Update values on an existing user. See the API docs for what kinds of update are possible.\n :param email: new email for this user\n :param username: new username for this user\n :param first_name: new first name for this user\n :param last_name: new last name for this user\n :param country: new country for this user\n :return: the User, so you can do User(...).update(...).add_to_groups(...)\n \"\"\"\n if username and self.id_type != IdentityTypes.federatedID:\n raise ArgumentError(\"You cannot set username except for a federated ID\")\n if username and '@' in username and not email:\n raise ArgumentError(\"Cannot update email-type username when email is not specified\")\n if email and username and email.lower() == username.lower():\n raise ArgumentError(\"Specify just email to set both email and username for a federated ID\")\n updates = {}\n for k, v in six.iteritems(dict(email=email, username=username,\n firstname=first_name, lastname=last_name,\n country=country)):\n if v:\n updates[k] = v\n return self.append(update=updates)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding user to some groups.", "response": "def add_to_groups(self, groups=None, all_groups=False, group_type=None):\n \"\"\"\n Add user to some (typically PLC) groups. Note that, if you add to no groups, the effect\n is simply to do an \"add to organization Everybody group\", so we let that be done.\n :param groups: list of group names the user should be added to\n :param all_groups: a boolean meaning add to all (don't specify groups or group_type in this case)\n :param group_type: the type of group (defaults to \"product\")\n :return: the User, so you can do User(...).add_to_groups(...).add_role(...)\n \"\"\"\n if all_groups:\n if groups or group_type:\n raise ArgumentError(\"When adding to all groups, do not specify specific groups or types\")\n glist = \"all\"\n else:\n if not groups:\n groups = []\n if not group_type:\n group_type = GroupTypes.product\n elif group_type in GroupTypes.__members__:\n group_type = GroupTypes[group_type]\n if group_type not in GroupTypes:\n raise ArgumentError(\"You must specify a GroupType value for argument group_type\")\n glist = {group_type.name: [group for group in groups]}\n return self.append(add=glist)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_from_groups(self, groups=None, all_groups=False, group_type=None):\n if all_groups:\n if groups or group_type:\n raise ArgumentError(\"When removing from all groups, do not specify specific groups or types\")\n glist = \"all\"\n else:\n if not groups:\n raise ArgumentError(\"You must specify groups from which to remove the user\")\n if not group_type:\n group_type = GroupTypes.product\n elif group_type in GroupTypes.__members__:\n group_type = GroupTypes[group_type]\n if group_type not in GroupTypes:\n raise ArgumentError(\"You must specify a GroupType value for argument group_type\")\n glist = {group_type.name: [group for group in groups]}\n return self.append(remove=glist)", "response": "Removes a user from some PLC groups or all of them."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_role(self, groups=None, role_type=RoleTypes.admin):\n if not groups:\n raise ArgumentError(\"You must specify groups to which to add the role for this user\")\n if role_type in RoleTypes.__members__:\n role_type = RoleTypes[role_type]\n if role_type not in RoleTypes:\n raise ArgumentError(\"You must specify a RoleType value for argument role_type\")\n glist = {role_type.name: [group for group in groups]}\n return self.append(addRoles=glist)", "response": "Add a role to a user"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_role(self, groups=None, role_type=RoleTypes.admin):\n if not groups:\n raise ArgumentError(\"You must specify groups from which to remove the role for this user\")\n if role_type in RoleTypes.__members__:\n role_type = RoleTypes[role_type]\n if role_type not in RoleTypes:\n raise ArgumentError(\"You must specify a RoleType value for argument role_type\")\n glist = {role_type.name: [group for group in groups]}\n return self.append(removeRoles=glist)", "response": "Removes a user from a role of some groups."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef remove_from_organization(self, delete_account=False):\n self.append(removeFromOrg={\"deleteAccount\": True if delete_account else False})\n return None", "response": "Removes a user from the organization s list of visible users."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef delete_account(self):\n if self.id_type == IdentityTypes.adobeID:\n raise ArgumentError(\"You cannot delete an Adobe ID account.\")\n self.append(removeFromDomain={})\n return None", "response": "Deletes a user s account."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate the group name", "response": "def _validate(cls, group_name):\n \"\"\"\n Validates the group name\n Input values must be strings (standard or unicode). Throws ArgumentError if any input is invalid\n :param group_name: name of group\n \"\"\"\n if group_name and not cls._group_name_regex.match(group_name):\n raise ArgumentError(\"'%s': Illegal group name\" % (group_name,))\n if group_name and len(group_name) > 255:\n raise ArgumentError(\"'%s': Group name is too long\" % (group_name,))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_to_products(self, products=None, all_products=False):\n if all_products:\n if products:\n raise ArgumentError(\"When adding to all products, do not specify specific products\")\n plist = \"all\"\n else:\n if not products:\n raise ArgumentError(\"You must specify products to which to add the user group\")\n plist = {GroupTypes.productConfiguration.name: [product for product in products]}\n return self.append(add=plist)", "response": "Add a user group to some product license configuration groups or all of them."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_from_products(self, products=None, all_products=False):\n if all_products:\n if products:\n raise ArgumentError(\"When removing from all products, do not specify specific products\")\n plist = \"all\"\n else:\n if not products:\n raise ArgumentError(\"You must specify products from which to remove the user group\")\n plist = {GroupTypes.productConfiguration.name: [product for product in products]}\n return self.append(remove=plist)", "response": "Removes a user group from some product license configuration groups or all of them."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_users(self, users=None):\n if not users:\n raise ArgumentError(\"You must specify emails for users to add to the user group\")\n ulist = {\"user\": [user for user in users]}\n return self.append(add=ulist)", "response": "Adds users to the user group."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_users(self, users=None):\n if not users:\n raise ArgumentError(\"You must specify emails for users to remove from the user group\")\n ulist = {\"user\": [user for user in users]}\n return self.append(remove=ulist)", "response": "Removes users from the user group."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef scale2x(self, surface):\n assert(self._scale == 2)\n return self._pygame.transform.scale2x(surface)", "response": "Returns the scale of the image in x - plane."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsmooths scaling using MMX or SSE extensions.", "response": "def smoothscale(self, surface):\n \"\"\"\n Smooth scaling using MMX or SSE extensions if available\n \"\"\"\n return self._pygame.transform.smoothscale(surface, self._output_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the identity of the current object.", "response": "def identity(self, surface):\n \"\"\"\n Fast scale operation that does not sample the results\n \"\"\"\n return self._pygame.transform.scale(surface, self._output_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef led_matrix(self, surface):\n scale = self._led_on.get_width()\n w, h = self._input_size\n pix = self._pygame.PixelArray(surface)\n img = self._pygame.Surface((w * scale, h * scale))\n\n for y in range(h):\n for x in range(w):\n led = self._led_on if pix[x, y] & 0xFFFFFF > 0 else self._led_off\n img.blit(led, (x * scale, y * scale))\n\n return img", "response": "Transforms the input surface into an LED matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts RGB values to nearest equivalent xterm - 256 color.", "response": "def rgb2short(r, g, b):\n \"\"\"\n Converts RGB values to the nearest equivalent xterm-256 color.\n \"\"\"\n # Using list of snap points, convert RGB value to cube indexes\n r, g, b = [len(tuple(s for s in snaps if s < x)) for x in (r, g, b)]\n\n # Simple colorcube transform\n return (r * 36) + (g * 6) + b + 16"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_surface(self, image, alpha=1.0):\n assert(0.0 <= alpha <= 1.0)\n if alpha < 1.0:\n im = image.convert(\"RGBA\")\n black = Image.new(im.mode, im.size, \"black\")\n im = Image.blend(black, im, alpha)\n else:\n im = image.convert(\"RGB\")\n\n mode = im.mode\n size = im.size\n data = im.tobytes()\n del im\n\n surface = self._pygame.image.fromstring(data, size, mode)\n return self._transform(surface)", "response": "Converts a Pillow. Image into a pygame. Surface."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef display(self, image):\n assert(image.size == self.size)\n self._last_image = image\n\n self._count += 1\n filename = self._file_template.format(self._count)\n image = self.preprocess(image)\n surface = self.to_surface(image, alpha=self._contrast)\n logger.debug(\"Writing: {0}\".format(filename))\n self._pygame.image.save(surface, filename)", "response": "Takes a : py : mod : PIL. Image and dumps it to a numbered PNG file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef display(self, image):\n assert(image.size == self.size)\n self._last_image = image\n\n image = self.preprocess(image)\n surface = self.to_surface(image, alpha=self._contrast)\n rawbytes = self._pygame.image.tostring(surface, \"RGB\", False)\n im = Image.frombytes(\"RGB\", surface.get_size(), rawbytes)\n self._images.append(im)\n\n self._count += 1\n logger.debug(\"Recording frame: {0}\".format(self._count))\n\n if self._max_frames and self._count >= self._max_frames:\n sys.exit(0)", "response": "Displays an image in the current thread."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndisplays the image to the pygame display surface.", "response": "def display(self, image):\n \"\"\"\n Takes a :py:mod:`PIL.Image` and renders it to a pygame display surface.\n \"\"\"\n assert(image.size == self.size)\n self._last_image = image\n\n image = self.preprocess(image)\n self._clock.tick(self._fps)\n self._pygame.event.pump()\n\n if self._abort():\n self._pygame.quit()\n sys.exit()\n\n surface = self.to_surface(image, alpha=self._contrast)\n if self._screen is None:\n self._screen = self._pygame.display.set_mode(surface.get_size())\n self._screen.blit(surface, (0, 0))\n self._pygame.display.flip()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncount the number of black pixels in a rendered character.", "response": "def _char_density(self, c, font=ImageFont.load_default()):\n \"\"\"\n Count the number of black pixels in a rendered character.\n \"\"\"\n image = Image.new('1', font.getsize(c), color=255)\n draw = ImageDraw.Draw(image)\n draw.text((0, 0), c, fill=\"white\", font=font)\n return collections.Counter(image.getdata())[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates an ascii art.", "response": "def _generate_art(self, image, width, height):\n \"\"\"\n Return an iterator that produces the ascii art.\n \"\"\"\n # Characters aren't square, so scale the output by the aspect ratio of a charater\n height = int(height * self._char_width / float(self._char_height))\n image = image.resize((width, height), Image.ANTIALIAS).convert(\"RGB\")\n\n for (r, g, b) in image.getdata():\n greyscale = int(0.299 * r + 0.587 * g + 0.114 * b)\n ch = self._chars[int(greyscale / 255. * (len(self._chars) - 1) + 0.5)]\n yield (ch, rgb2short(r, g, b))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef display(self, image):\n assert(image.size == self.size)\n self._last_image = image\n\n surface = self.to_surface(self.preprocess(image), alpha=self._contrast)\n rawbytes = self._pygame.image.tostring(surface, \"RGB\", False)\n image = Image.frombytes(\"RGB\", surface.get_size(), rawbytes)\n\n scr_width = self._stdscr.getmaxyx()[1]\n scale = float(scr_width) / image.width\n\n self._stdscr.erase()\n self._stdscr.move(0, 0)\n try:\n for (ch, color) in self._generate_art(image, int(image.width * scale), int(image.height * scale)):\n self._stdscr.addstr(ch, curses.color_pair(color))\n\n except curses.error:\n # End of screen reached\n pass\n\n self._stdscr.refresh()", "response": "Displays an image as ASCII - art."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating an ascii art.", "response": "def _generate_art(self, image, width, height):\n \"\"\"\n Return an iterator that produces the ascii art.\n \"\"\"\n image = image.resize((width, height), Image.ANTIALIAS).convert(\"RGB\")\n pixels = list(image.getdata())\n\n for y in range(0, height - 1, 2):\n for x in range(width):\n i = y * width + x\n bg = rgb2short(*(pixels[i]))\n fg = rgb2short(*(pixels[i + width]))\n yield (fg, bg)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprints the CSI command.", "response": "def _CSI(self, cmd):\n \"\"\"\n Control sequence introducer\n \"\"\"\n sys.stdout.write('\\x1b[')\n sys.stdout.write(cmd)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndisplays the image as an ASCII - block.", "response": "def display(self, image):\n \"\"\"\n Takes a :py:mod:`PIL.Image` and renders it to the current terminal as\n ASCII-blocks.\n \"\"\"\n assert(image.size == self.size)\n self._last_image = image\n\n surface = self.to_surface(self.preprocess(image), alpha=self._contrast)\n rawbytes = self._pygame.image.tostring(surface, \"RGB\", False)\n image = Image.frombytes(\"RGB\", surface.get_size(), rawbytes)\n\n scr_width = self._terminal_size()[1]\n scale = float(scr_width) / image.width\n\n self._CSI('1;1H') # Move to top/left\n\n for (fg, bg) in self._generate_art(image, int(image.width * scale), int(image.height * scale)):\n self._CSI('38;5;{0};48;5;{1}m'.format(fg, bg))\n sys.stdout.write('\u2584')\n\n self._CSI('0m')\n sys.stdout.flush()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nyielding successive n - sized chunks from l.", "response": "def chunk_sequence(sequence, chunk_length):\n \"\"\"Yield successive n-sized chunks from l.\"\"\"\n for index in range(0, len(sequence), chunk_length):\n yield sequence[index:index + chunk_length]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving a collection of routes returns a new set containing only valid routes.", "response": "def _filter_invalid_routes(routes, board, railroad):\n \"\"\"\n Given a collection of routes, returns a new set containing only valid routes. Invalid routes removed:\n - contain less than 2 cities, or\n - go through Chicago using an impassable exit\n - only contain Chicago as a station, but don't use the correct exit path\n\n This fltering after the fact keeps the path finding algorithm simpler. It allows groups of 3 cells to be considered\n (important for the Chicago checks), which would be tricky, since the algorithm operates on pairs of cells (at the\n time of writing).\n \"\"\"\n chicago_space = board.get_space(CHICAGO_CELL)\n\n chicago_neighbor_cells = [cell for cell in CHICAGO_CELL.neighbors.values() if cell != CHICAGO_CONNECTIONS_CELL]\n stations = board.stations(railroad.name)\n\n # A sieve style filter. If a condition isn't met, iteration continues to the next item. Items meeting all conditions\n # are added to valid_routes at the end of the loop iteration.\n valid_routes = set()\n for route in routes:\n # A route must connect at least 2 cities.\n if len(route.cities) < 2:\n continue\n\n # A route cannot run from east to east\n if isinstance(route.cities[0], EastTerminalCity) and isinstance(route.cities[-1], EastTerminalCity):\n continue\n\n # If the route goes through Chicago and isn't [C5, D6], ensure the path it took either contains its station or is unblocked\n if route.contains_cell(CHICAGO_CONNECTIONS_CELL) and len(route.cities) != 2:\n # Finds the subroute which starts at Chicago and is 3 tiles long. That is, it will go [C5, D6, chicago exit]\n all_chicago_subroutes = [subroute for subroute in route.subroutes(CHICAGO_CONNECTIONS_CELL) if len(subroute) == 3]\n chicago_subroute = all_chicago_subroutes[0] if all_chicago_subroutes else None\n for cell in chicago_neighbor_cells:\n chicago_exit = chicago_subroute and chicago_subroute.contains_cell(cell)\n if chicago_exit and chicago_space.passable(cell, railroad):\n break\n else:\n continue\n\n # Each route must contain at least 1 station\n stations_on_route = [station for station in stations if route.contains_cell(station.cell)]\n if not stations_on_route:\n continue\n # If the only station is Chicago, the path must be [D6, C5], or exit through the appropriate side.\n elif [CHICAGO_CELL] == [station.cell for station in stations_on_route]:\n exit_cell = board.get_space(CHICAGO_CELL).get_station_exit_cell(stations_on_route[0])\n chicago_exit_route = Route.create([chicago_space, board.get_space(exit_cell)])\n if not (len(route) == 2 and route.contains_cell(CHICAGO_CONNECTIONS_CELL)) and not route.overlap(chicago_exit_route):\n continue\n\n valid_routes.add(route)\n\n return valid_routes"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef find(path,\n level=None,\n message=None,\n time_lower=None, time_upper=None,\n case_sensitive=False): # pragma: no cover\n \"\"\"\n Filter log message.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u6839\u636elevel\u540d\u79f0, message\u4e2d\u7684\u5173\u952e\u5b57, \u548clog\u7684\u65f6\u95f4\u7684\u533a\u95f4, \u7b5b\u9009\u51fa\u76f8\u5173\u7684\u65e5\u5fd7\n \"\"\"\n if level:\n level = level.upper() # level name has to be capitalized.\n\n if not case_sensitive:\n message = message.lower()\n\n with open(path, \"r\") as f:\n result = Result(path=path,\n level=level, message=message,\n time_lower=time_lower, time_upper=time_upper,\n case_sensitive=case_sensitive,\n )\n\n for line in f:\n try:\n _time, _level, _message = [i.strip() for i in line.split(\";\")]\n\n if level:\n if _level != level:\n continue\n\n if time_lower:\n if _time < time_lower:\n continue\n\n if time_upper:\n if _time > time_upper:\n continue\n\n if message:\n if not case_sensitive:\n _message = _message.lower()\n\n if message not in _message:\n continue\n\n result.lines.append(line)\n except Exception as e:\n print(e)\n\n return result", "response": "Find a log file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_logger_by_name(name=None, rand_name=False, charset=Charset.HEX):\n if rand_name:\n name = rand_str(charset)\n logger = logging.getLogger(name)\n return logger", "response": "Get a logger by name."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef debug(self, msg, indent=0, **kwargs):\n return self.logger.debug(self._indent(msg, indent), **kwargs)", "response": "invoke self. logger. debug"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef info(self, msg, indent=0, **kwargs):\n return self.logger.info(self._indent(msg, indent), **kwargs)", "response": "invoke self. info. debug"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef warning(self, msg, indent=0, **kwargs):\n return self.logger.warning(self._indent(msg, indent), **kwargs)", "response": "invoke self. logger. warning"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ninvoking self. logger. error", "response": "def error(self, msg, indent=0, **kwargs):\n \"\"\"invoke ``self.logger.error``\"\"\"\n return self.logger.error(self._indent(msg, indent), **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef critical(self, msg, indent=0, **kwargs):\n return self.logger.critical(self._indent(msg, indent), **kwargs)", "response": "invoke self. logger. critical"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprint a message to console indent format may apply.", "response": "def show(self, msg, indent=0, style=\"\", **kwargs):\n \"\"\"\n Print message to console, indent format may apply.\n \"\"\"\n if self.enable_verbose:\n new_msg = self.MessageTemplate.with_style.format(\n indent=self.tab * indent,\n style=style,\n msg=msg,\n )\n print(new_msg, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remove_all_handler(self):\n for handler in self.logger.handlers[:]:\n self.logger.removeHandler(handler)\n self._handler_cache.append(handler)", "response": "Remove all file handlers associated with this logger."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef recover_all_handler(self):\n for handler in self._handler_cache:\n self.logger.addHandler(handler)\n self._handler_cache = list()", "response": "Re - add all file handlers that have been removed from the logger."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_protobuf(cls, proto: LinkItemProto) -> LinkItem:\n if proto.name == '':\n raise ValueError(\"name of LinkItem does not exist or is empty inside the protobuf.\")\n return cls(proto.name, Timestamp.ToDatetime(proto.time))", "response": "Constructor from protobuf.\n\n :param proto: protobuf structure\n :type proto: ~unidown.plugin.protobuf.link_item_pb2.LinkItemProto\n :return: the LinkItem\n :rtype: ~unidown.plugin.link_item.LinkItem\n :raises ValueError: name of LinkItem does not exist inside the protobuf or is empty"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_protobuf(self) -> LinkItemProto:\n result = LinkItemProto()\n result.name = self._name\n result.time.CopyFrom(datetime_to_timestamp(self._time))\n return result", "response": "Create protobuf item.\n\n :return: protobuf structure\n :rtype: ~unidown.plugin.protobuf.link_item_pb2.LinkItemProto"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update(self, fname):\n ltfh = FileHandler(fname)\n self._log.addHandler(ltfh)", "response": "Adds a handler to save to a file. Includes debug stuff."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_dir_rec(path: Path):\n if not path.exists() or not path.is_dir():\n return\n for sub in path.iterdir():\n if sub.is_dir():\n delete_dir_rec(sub)\n else:\n sub.unlink()\n path.rmdir()", "response": "Delete a folder recursively."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_dir_rec(path: Path):\n if not path.exists():\n Path.mkdir(path, parents=True, exist_ok=True)", "response": "Create a folder recursively."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts datetime to protobuf. timestamp. Timestamp", "response": "def datetime_to_timestamp(time: datetime) -> Timestamp:\n \"\"\"\n Convert datetime to protobuf.timestamp.\n\n :param time: time\n :type time: ~datetime.datetime\n :return: protobuf.timestamp\n :rtype: ~google.protobuf.timestamp_pb2.Timestamp\n \"\"\"\n protime = Timestamp()\n protime.FromDatetime(time)\n return protime"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprint all registered plugins and checks if they can be loaded or not.", "response": "def print_plugin_list(plugins: Dict[str, pkg_resources.EntryPoint]):\n \"\"\"\n Prints all registered plugins and checks if they can be loaded or not.\n\n :param plugins: plugins\n :type plugins: Dict[str, ~pkg_resources.EntryPoint]\n \"\"\"\n for trigger, entry_point in plugins.items():\n try:\n plugin_class = entry_point.load()\n version = str(plugin_class._info.version)\n print(\n f\"{trigger} (ok)\\n\"\n f\" {version}\"\n )\n except Exception:\n print(\n f\"{trigger} (failed)\"\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndetermine whether two windows overlap.", "response": "def overlap(xl1, yl1, nx1, ny1, xl2, yl2, nx2, ny2):\n \"\"\"\n Determines whether two windows overlap\n \"\"\"\n return (xl2 < xl1+nx1 and xl2+nx2 > xl1 and\n yl2 < yl1+ny1 and yl2+ny2 > yl1)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef saveJSON(g, data, backup=False):\n if not backup:\n fname = filedialog.asksaveasfilename(\n defaultextension='.json',\n filetypes=[('json files', '.json'), ],\n initialdir=g.cpars['app_directory']\n )\n else:\n fname = os.path.join(os.path.expanduser('~/.hdriver'), 'app.json')\n\n if not fname:\n g.clog.warn('Aborted save to disk')\n return False\n\n with open(fname, 'w') as of:\n of.write(\n json.dumps(data, sort_keys=True, indent=4,\n separators=(',', ': '))\n )\n g.clog.info('Saved setup to' + fname)\n return True", "response": "Saves the current setup in JSON compatible dictionary format to disk."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nposting the current setup to the camera and data servers.", "response": "def postJSON(g, data):\n \"\"\"\n Posts the current setup to the camera and data servers.\n\n g : hcam_drivers.globals.Container\n Container with globals\n\n data : dict\n The current setup in JSON compatible dictionary format.\n \"\"\"\n g.clog.debug('Entering postJSON')\n\n # encode data as json\n json_data = json.dumps(data).encode('utf-8')\n\n # Send the xml to the server\n url = urllib.parse.urljoin(g.cpars['hipercam_server'], g.SERVER_POST_PATH)\n g.clog.debug('Server URL = ' + url)\n\n opener = urllib.request.build_opener()\n g.clog.debug('content length = ' + str(len(json_data)))\n req = urllib.request.Request(url, data=json_data, headers={'Content-type': 'application/json'})\n response = opener.open(req, timeout=15).read()\n g.rlog.debug('Server response: ' + response.decode())\n csr = ReadServer(response, status_msg=False)\n if not csr.ok:\n g.clog.warn('Server response was not OK')\n g.rlog.warn('postJSON response: ' + response.decode())\n g.clog.warn('Server error = ' + csr.err)\n return False\n\n # now try to setup nodding server if appropriate\n if g.cpars['telins_name'] == 'GTC':\n url = urllib.parse.urljoin(g.cpars['gtc_offset_server'], 'setup')\n g.clog.debug('Offset Server URL = ' + url)\n opener = urllib.request.build_opener()\n try:\n req = urllib.request.Request(url, data=json_data, headers={'Content-type': 'application/json'})\n response = opener.open(req, timeout=5).read().decode()\n except Exception as err:\n g.clog.warn('Could not communicate with GTC offsetter')\n g.clog.warn(str(err))\n return False\n\n g.rlog.info('Offset Server Response: ' + response)\n if not json.loads(response)['status'] == 'OK':\n g.clog.warn('Offset Server response was not OK')\n return False\n\n g.clog.debug('Leaving postJSON')\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef createJSON(g, full=True):\n data = dict()\n if 'gps_attached' not in g.cpars:\n data['gps_attached'] = 1\n else:\n data['gps_attached'] = 1 if g.cpars['gps_attached'] else 0\n data['appdata'] = g.ipars.dumpJSON()\n data['user'] = g.rpars.dumpJSON()\n if full:\n data['hardware'] = g.ccd_hw.dumpJSON()\n data['tcs'] = g.info.dumpJSON()\n\n if g.cpars['telins_name'].lower() == 'gtc' and has_corba:\n try:\n s = get_telescope_server()\n data['gtc_headers'] = dict(\n create_header_from_telpars(s.getTelescopeParams())\n )\n except:\n g.clog.warn('cannot get GTC headers from telescope server')\n return data", "response": "Create JSON compatible dictionary from current settings"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupload a table of TCS data to the server and returns True if successful False otherwise", "response": "def insertFITSHDU(g):\n \"\"\"\n Uploads a table of TCS data to the servers, which is appended onto a run.\n\n Arguments\n ---------\n g : hcam_drivers.globals.Container\n the Container object of application globals\n \"\"\"\n if not g.cpars['hcam_server_on']:\n g.clog.warn('insertFITSHDU: servers are not active')\n return False\n\n run_number = getRunNumber(g)\n tcs_table = g.info.tcs_table\n\n g.clog.info('Adding TCS table data to run{:04d}.fits'.format(run_number))\n url = g.cpars['hipercam_server'] + 'addhdu'\n try:\n fd = StringIO()\n ascii.write(tcs_table, format='ecsv', output=fd)\n files = {'file': fd.getvalue()}\n r = requests.post(url, data={'run': 'run{:04d}.fits'.format(run_number)},\n files=files)\n fd.close()\n rs = ReadServer(r.content, status_msg=False)\n if rs.ok:\n g.clog.info('Response from server was OK')\n return True\n else:\n g.clog.warn('Response from server was not OK')\n g.clog.warn('Reason: ' + rs.err)\n return False\n except Exception as err:\n g.clog.warn('insertFITSHDU failed')\n g.clog.warn(str(err))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes a command on the NGC controller and returns True if the command was successful False otherwise", "response": "def execCommand(g, command, timeout=10):\n \"\"\"\n Executes a command by sending it to the rack server\n\n Arguments:\n g : hcam_drivers.globals.Container\n the Container object of application globals\n command : (string)\n the command (see below)\n\n Possible commands are:\n\n start : starts a run\n stop : stops a run\n abort : aborts a run\n online : bring ESO control server online and power up hardware\n off : put ESO control server in idle state and power down\n standby : server can communicate, but child processes disabled\n reset : resets the NGC controller front end\n\n Returns True/False according to whether the command\n succeeded or not.\n \"\"\"\n if not g.cpars['hcam_server_on']:\n g.clog.warn('execCommand: servers are not active')\n return False\n\n try:\n url = g.cpars['hipercam_server'] + command\n g.clog.info('execCommand, command = \"' + command + '\"')\n response = urllib.request.urlopen(url, timeout=timeout)\n rs = ReadServer(response.read(), status_msg=False)\n\n g.rlog.info('Server response =\\n' + rs.resp())\n if rs.ok:\n g.clog.info('Response from server was OK')\n return True\n else:\n g.clog.warn('Response from server was not OK')\n g.clog.warn('Reason: ' + rs.err)\n return False\n except urllib.error.URLError as err:\n g.clog.warn('execCommand failed')\n g.clog.warn(str(err))\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef isRunActive(g):\n if g.cpars['hcam_server_on']:\n url = g.cpars['hipercam_server'] + 'summary'\n response = urllib.request.urlopen(url, timeout=2)\n rs = ReadServer(response.read(), status_msg=True)\n if not rs.ok:\n raise DriverError('isRunActive error: ' + str(rs.err))\n if rs.state == 'idle':\n return False\n elif rs.state == 'active':\n return True\n else:\n raise DriverError('isRunActive error, state = ' + rs.state)\n else:\n raise DriverError('isRunActive error: servers are not active')", "response": "Checks if a run is active"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npoll the data server to find the current frame number.", "response": "def getFrameNumber(g):\n \"\"\"\n Polls the data server to find the current frame number.\n\n Throws an exceotion if it cannot determine it.\n \"\"\"\n if not g.cpars['hcam_server_on']:\n raise DriverError('getRunNumber error: servers are not active')\n url = g.cpars['hipercam_server'] + 'status/DET.FRAM2.NO'\n response = urllib.request.urlopen(url, timeout=2)\n rs = ReadServer(response.read(), status_msg=False)\n try:\n msg = rs.msg\n except:\n raise DriverError('getFrameNumber error: no message found')\n try:\n frame_no = int(msg.split()[1])\n except:\n raise DriverError('getFrameNumber error: invalid msg ' + msg)\n return frame_no"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npoll the data server to find the current run number. Throws exceptions if it can t determine it.", "response": "def getRunNumber(g):\n \"\"\"\n Polls the data server to find the current run number. Throws\n exceptions if it can't determine it.\n \"\"\"\n if not g.cpars['hcam_server_on']:\n raise DriverError('getRunNumber error: servers are not active')\n url = g.cpars['hipercam_server'] + 'summary'\n response = urllib.request.urlopen(url, timeout=2)\n rs = ReadServer(response.read(), status_msg=True)\n if rs.ok:\n return rs.run\n else:\n raise DriverError('getRunNumber error: ' + str(rs.err))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef checkSimbad(g, target, maxobj=5, timeout=5):\n url = 'http://simbad.u-strasbg.fr/simbad/sim-script'\n q = 'set limit ' + str(maxobj) + \\\n '\\nformat object form1 \"Target: %IDLIST(1) | %COO(A D;ICRS)\"\\nquery ' \\\n + target\n query = urllib.parse.urlencode({'submit': 'submit script', 'script': q})\n resp = urllib.request.urlopen(url, query.encode(), timeout)\n data = False\n error = False\n results = []\n for line in resp:\n line = line.decode()\n if line.startswith('::data::'):\n data = True\n if line.startswith('::error::'):\n error = True\n if data and line.startswith('Target:'):\n name, coords = line[7:].split(' | ')\n results.append(\n {'Name': name.strip(), 'Position': coords.strip(),\n 'Frame': 'ICRS'})\n resp.close()\n\n if error and len(results):\n g.clog.warn('drivers.check: Simbad: there appear to be some ' +\n 'results but an error was unexpectedly raised.')\n return results", "response": "Sends a request to Simbad to check whether a target is recognised. Returns with a list of results or raises an exception if it times out."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning the thread and stores the tracebacks and the exception in the fifo.", "response": "def run(self):\n \"\"\"\n Version of run that traps Exceptions and stores\n them in the fifo\n \"\"\"\n try:\n threading.Thread.run(self)\n except Exception:\n t, v, tb = sys.exc_info()\n error = traceback.format_exception_only(t, v)[0][:-1]\n tback = (self.name + ' Traceback (most recent call last):\\n' +\n ''.join(traceback.format_tb(tb)))\n self.fifo.put((self.name, error, tback))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef addStyle(w):\n # access global container in root widget\n root = get_root(w)\n g = root.globals\n fsize = g.cpars['font_size']\n family = g.cpars['font_family']\n\n # Default font\n g.DEFAULT_FONT = font.nametofont(\"TkDefaultFont\")\n g.DEFAULT_FONT.configure(size=fsize, weight='bold', family=family)\n w.option_add('*Font', g.DEFAULT_FONT)\n\n # Menu font\n g.MENU_FONT = font.nametofont(\"TkMenuFont\")\n g.MENU_FONT.configure(family=family)\n w.option_add('*Menu.Font', g.MENU_FONT)\n\n # Entry font\n g.ENTRY_FONT = font.nametofont(\"TkTextFont\")\n g.ENTRY_FONT.configure(size=fsize, family=family)\n w.option_add('*Entry.Font', g.ENTRY_FONT)\n\n # position and size\n # root.geometry(\"320x240+325+200\")\n\n # Default colours. Note there is a difference between\n # specifying 'background' with a capital B or lowercase b\n w.option_add('*background', g.COL['main'])\n w.option_add('*HighlightBackground', g.COL['main'])\n w.config(background=g.COL['main'])", "response": "Adds the styles of the GUI element w to the global fonts and colours."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef init(main_dir: Path, logfile_path: Path, log_level: str):\n dynamic_data.reset()\n dynamic_data.init_dirs(main_dir, logfile_path)\n\n dynamic_data.check_dirs()\n\n tools.create_dir_rec(dynamic_data.MAIN_DIR)\n tools.create_dir_rec(dynamic_data.TEMP_DIR)\n tools.create_dir_rec(dynamic_data.DOWNLOAD_DIR)\n tools.create_dir_rec(dynamic_data.SAVESTAT_DIR)\n tools.create_dir_rec(Path.resolve(dynamic_data.LOGFILE_PATH).parent)\n dynamic_data.LOG_LEVEL = log_level\n logging.basicConfig(filename=dynamic_data.LOGFILE_PATH, filemode='a', level=dynamic_data.LOG_LEVEL,\n format='%(asctime)s.%(msecs)03d | %(levelname)s - %(name)s | %(module)s.%(funcName)s: %('\n 'message)s',\n datefmt='%Y.%m.%d %H:%M:%S')\n logging.captureWarnings(True)\n\n cores = multiprocessing.cpu_count()\n dynamic_data.USING_CORES = min(4, max(1, cores - 1))\n\n info = f\"{static_data.NAME} {static_data.VERSION}\\n\\n\" \\\n f\"System: {platform.system()} - {platform.version()} - {platform.machine()} - {cores} cores\\n\" \\\n f\"Python: {platform.python_version()} - {' - '.join(platform.python_build())}\\n\" \\\n f\"Arguments: main={main_dir.resolve()} | logfile={logfile_path.resolve()} | loglevel={log_level}\\n\" \\\n f\"Using cores: {dynamic_data.USING_CORES}\\n\\n\"\n with dynamic_data.LOGFILE_PATH.open(mode='w', encoding=\"utf8\") as writer:\n writer.write(info)\n\n dynamic_data.AVAIL_PLUGINS = APlugin.get_plugins()", "response": "Initialize the _downloader. TODO.\n\n :param main_dir: main directory\n :type main_dir: ~pathlib.Path\n :param logfile_path: logfile path\n :type logfile_path: ~pathlib.Path\n :param log_level: logging level\n :type log_level: str"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef download_from_plugin(plugin: APlugin):\n # get last update date\n plugin.log.info('Get last update')\n plugin.update_last_update()\n # load old save state\n save_state = plugin.load_save_state()\n if plugin.last_update <= save_state.last_update:\n plugin.log.info('No update. Nothing to do.')\n return\n # get download links\n plugin.log.info('Get download links')\n plugin.update_download_links()\n # compare with save state\n down_link_item_dict = plugin.get_updated_data(save_state.link_item_dict)\n plugin.log.info('Compared with save state: ' + str(len(plugin.download_data)))\n if not down_link_item_dict:\n plugin.log.info('No new data. Nothing to do.')\n return\n # download new/updated data\n plugin.log.info(f\"Download new {plugin.unit}s: {len(down_link_item_dict)}\")\n plugin.download(down_link_item_dict, plugin.download_path, 'Download new ' + plugin.unit + 's', plugin.unit)\n # check which downloads are succeeded\n succeed_link_item_dict, lost_link_item_dict = plugin.check_download(down_link_item_dict, plugin.download_path)\n plugin.log.info(f\"Downloaded: {len(succeed_link_item_dict)}/{len(down_link_item_dict)}\")\n # update savestate link_item_dict with succeeded downloads dict\n plugin.log.info('Update savestate')\n plugin.update_dict(save_state.link_item_dict, succeed_link_item_dict)\n # write new savestate\n plugin.log.info('Write savestate')\n plugin.save_save_state(save_state.link_item_dict)", "response": "Download and check if all data is downloaded and if so update the site with the latest update time."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef run(plugin_name: str, options: List[str] = None) -> PluginState:\n if options is None:\n options = []\n\n if plugin_name not in dynamic_data.AVAIL_PLUGINS:\n msg = 'Plugin ' + plugin_name + ' was not found.'\n logging.error(msg)\n print(msg)\n return PluginState.NOT_FOUND\n\n try:\n plugin_class = dynamic_data.AVAIL_PLUGINS[plugin_name].load()\n plugin = plugin_class(options)\n except Exception:\n msg = 'Plugin ' + plugin_name + ' crashed while loading.'\n logging.exception(msg)\n print(msg + ' Check log for more information.')\n return PluginState.LOAD_CRASH\n else:\n logging.info('Loaded plugin: ' + plugin_name)\n\n try:\n download_from_plugin(plugin)\n plugin.clean_up()\n except PluginException as ex:\n msg = f\"Plugin {plugin.name} stopped working. Reason: {'unknown' if (ex.msg == '') else ex.msg}\"\n logging.error(msg)\n print(msg)\n return PluginState.RUN_FAIL\n except Exception:\n msg = 'Plugin ' + plugin.name + ' crashed.'\n logging.exception(msg)\n print(msg + ' Check log for more information.')\n return PluginState.RUN_CRASH\n else:\n logging.info(plugin.name + ' ends without errors.')\n return PluginState.END_SUCCESS", "response": "Runs a plugin so use the download routine and clean up after."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking for app updates and print them if they are available.", "response": "def check_update():\n \"\"\"\n Check for app updates and print/log them.\n \"\"\"\n logging.info('Check for app updates.')\n try:\n update = updater.check_for_app_updates()\n except Exception:\n logging.exception('Check for updates failed.')\n return\n if update:\n print(\"!!! UPDATE AVAILABLE !!!\\n\"\n \"\" + static_data.PROJECT_URL + \"\\n\\n\")\n logging.info(\"Update available: \" + static_data.PROJECT_URL)\n else:\n logging.info(\"No update available.\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndownloading the version tag from remote and return the newest version.", "response": "def get_newest_app_version() -> Version:\n \"\"\"\n Download the version tag from remote.\n\n :return: version from remote\n :rtype: ~packaging.version.Version\n \"\"\"\n with urllib3.PoolManager(cert_reqs='CERT_REQUIRED', ca_certs=certifi.where()) as p_man:\n pypi_json = p_man.urlopen('GET', static_data.PYPI_JSON_URL).data.decode('utf-8')\n releases = json.loads(pypi_json).get('releases', [])\n online_version = Version('0.0.0')\n for release in releases:\n cur_version = Version(release)\n if not cur_version.is_prerelease:\n online_version = max(online_version, cur_version)\n return online_version"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndumping current parameters to JSON compatible dictionary.", "response": "def dumpJSON(self):\n \"\"\"\n Encodes current parameters to JSON compatible dictionary\n \"\"\"\n numexp = self.number.get()\n expTime, _, _, _, _ = self.timing()\n if numexp == 0:\n numexp = -1\n\n data = dict(\n numexp=self.number.value(),\n app=self.app.value(),\n led_flsh=self.led(),\n dummy_out=self.dummy(),\n fast_clks=self.fastClk(),\n readout=self.readSpeed(),\n dwell=self.expose.value(),\n exptime=expTime,\n oscan=self.oscan(),\n oscany=self.oscany(),\n xbin=self.wframe.xbin.value(),\n ybin=self.wframe.ybin.value(),\n multipliers=self.nmult.getall(),\n clear=self.clear()\n )\n\n # only allow nodding in clear mode, even if GUI has got confused\n if data['clear'] and self.nodPattern:\n data['nodpattern'] = self.nodPattern\n\n # no mixing clear and multipliers, no matter what GUI says\n if data['clear']:\n data['multipliers'] = [1 for i in self.nmult.getall()]\n\n # add window mode\n if not self.isFF():\n if self.isDrift():\n # no clear, multipliers or oscan in drift\n for setting in ('clear', 'oscan', 'oscany'):\n data[setting] = 0\n data['multipliers'] = [1 for i in self.nmult.getall()]\n\n for iw, (xsl, xsr, ys, nx, ny) in enumerate(self.wframe):\n data['x{}start_left'.format(iw+1)] = xsl\n data['x{}start_right'.format(iw+1)] = xsr\n data['y{}start'.format(iw+1)] = ys\n data['y{}size'.format(iw+1)] = ny\n data['x{}size'.format(iw+1)] = nx\n else:\n # no oscany in window mode\n data['oscany'] = 0\n\n for iw, (xsll, xsul, xslr, xsur, ys, nx, ny) in enumerate(self.wframe):\n data['x{}start_upperleft'.format(iw+1)] = xsul\n data['x{}start_lowerleft'.format(iw+1)] = xsll\n data['x{}start_upperright'.format(iw+1)] = xsur\n data['x{}start_lowerright'.format(iw+1)] = xslr\n data['y{}start'.format(iw+1)] = ys\n data['x{}size'.format(iw+1)] = nx\n data['y{}size'.format(iw+1)] = ny\n return data"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef loadJSON(self, json_string):\n g = get_root(self).globals\n data = json.loads(json_string)['appdata']\n # first set the parameters which change regardless of mode\n # number of exposures\n numexp = data.get('numexp', 0)\n if numexp == -1:\n numexp = 0\n self.number.set(numexp)\n # Overscan (x, y)\n if 'oscan' in data:\n self.oscan.set(data['oscan'])\n if 'oscany' in data:\n self.oscan.set(data['oscany'])\n # LED setting\n self.led.set(data.get('led_flsh', 0))\n # Dummy output enabled\n self.dummy.set(data.get('dummy_out', 0))\n # Fast clocking option?\n self.fastClk.set(data.get('fast_clks', 0))\n # readout speed\n self.readSpeed.set(data.get('readout', 'Slow'))\n # dwell\n dwell = data.get('dwell', 0)\n self.expose.set(str(float(dwell)))\n\n # multipliers\n mult_values = data.get('multipliers',\n (1, 1, 1, 1, 1))\n self.nmult.setall(mult_values)\n\n # look for nodpattern in data\n nodPattern = data.get('nodpattern', {})\n if nodPattern and g.cpars['telins_name'] == 'GTC':\n self.nodPattern = nodPattern\n self.nod.set(True)\n self.clear.set(True)\n else:\n self.nodPattern = {}\n self.nod.set(False)\n\n # binning\n self.quad_frame.xbin.set(data.get('xbin', 1))\n self.quad_frame.ybin.set(data.get('ybin', 1))\n self.drift_frame.xbin.set(data.get('xbin', 1))\n self.drift_frame.ybin.set(data.get('ybin', 1))\n\n # now for the behaviour which depends on mode\n if 'app' in data:\n self.app.set(data['app'])\n app = data['app']\n\n if app == 'Drift':\n # disable clear mode in drift\n self.clear.set(0)\n # only one pair allowed\n self.wframe.npair.set(1)\n\n # set the window pair values\n labels = ('x1start_left', 'y1start',\n 'x1start_right', 'x1size',\n 'y1size')\n if not all(label in data for label in labels):\n raise DriverError('Drift mode application missing window params')\n # now actually set them\n self.wframe.xsl[0].set(data['x1start_left'])\n self.wframe.xsr[0].set(data['x1start_right'])\n self.wframe.ys[0].set(data['y1start'])\n self.wframe.nx[0].set(data['x1size'])\n self.wframe.ny[0].set(data['y1size'])\n self.wframe.check()\n\n elif app == 'FullFrame':\n # enable clear mode if set\n self.clear.set(data.get('clear', 0))\n\n elif app == 'Windows':\n # enable clear mode if set\n self.clear.set(data.get('clear', 0))\n nquad = 0\n for nw in range(2):\n labels = ('x{0}start_lowerleft y{0}start x{0}start_upperleft x{0}start_upperright ' +\n 'x{0}start_lowerright x{0}size y{0}size').format(nw+1).split()\n if all(label in data for label in labels):\n xsll = data[labels[0]]\n xslr = data[labels[4]]\n xsul = data[labels[2]]\n xsur = data[labels[3]]\n ys = data[labels[1]]\n nx = data[labels[5]]\n ny = data[labels[6]]\n self.wframe.xsll[nw].set(xsll)\n self.wframe.xslr[nw].set(xslr)\n self.wframe.xsul[nw].set(xsul)\n self.wframe.xsur[nw].set(xsur)\n self.wframe.ys[nw].set(ys)\n self.wframe.nx[nw].set(nx)\n self.wframe.ny[nw].set(ny)\n nquad += 1\n else:\n break\n self.wframe.nquad.set(nquad)\n self.wframe.check()", "response": "Load the current state from a JSON string."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check(self, *args):\n status = True\n g = get_root(self).globals\n\n # clear errors on binning (may be set later if FF)\n xbinw, ybinw = self.wframe.xbin, self.wframe.ybin\n xbinw.config(bg=g.COL['main'])\n ybinw.config(bg=g.COL['main'])\n\n # keep binning factors of drift mode and windowed mode up to date\n oframe, aframe = ((self.quad_frame, self.drift_frame) if self.drift_frame.winfo_ismapped()\n else (self.drift_frame, self.quad_frame))\n xbin, ybin = aframe.xbin.value(), aframe.ybin.value()\n oframe.xbin.set(xbin)\n oframe.ybin.set(ybin)\n\n if not self.frozen:\n if self.clear() or self.isDrift():\n # disable nmult in clear or drift mode\n self.nmult.disable()\n else:\n self.nmult.enable()\n\n if self.isDrift():\n self.clearLab.config(state='disable')\n self.nodLab.config(state='disable')\n if not self.drift_frame.winfo_ismapped():\n self.quad_frame.grid_forget()\n self.drift_frame.grid(row=10, column=0, columnspan=3,\n sticky=tk.W+tk.N)\n\n if not self.frozen:\n self.oscany.config(state='disable')\n self.oscan.config(state='disable')\n self.clear.config(state='disable')\n self.nod.config(state='disable')\n self.wframe.enable()\n status = self.wframe.check()\n\n elif self.isFF():\n # special case check of binning from window frame\n if 1024 % xbin != 0:\n status = False\n xbinw.config(bg=g.COL['error'])\n elif (1024 // xbin) % 4 != 0:\n status = False\n xbinw.config(bg=g.COL['error'])\n if 512 % ybin != 0:\n status = False\n ybinw.config(bg=g.COL['error'])\n\n if not self.quad_frame.winfo_ismapped():\n self.drift_frame.grid_forget()\n self.quad_frame.grid(row=10, column=0, columnspan=3,\n sticky=tk.W+tk.N)\n\n self.clearLab.config(state='normal')\n if g.cpars['telins_name'] == 'GTC':\n self.nodLab.config(state='normal')\n else:\n self.nodLab.config(state='disable')\n if not self.frozen:\n self.oscany.config(state='normal')\n self.oscan.config(state='normal')\n self.clear.config(state='normal')\n if g.cpars['telins_name'] == 'GTC':\n self.nod.config(state='normal')\n else:\n self.nod.config(state='disable')\n self.wframe.disable()\n\n else:\n self.clearLab.config(state='normal')\n if g.cpars['telins_name'] == 'GTC':\n self.nodLab.config(state='normal')\n else:\n self.nodLab.config(state='disable')\n if not self.quad_frame.winfo_ismapped():\n self.drift_frame.grid_forget()\n self.quad_frame.grid(row=10, column=0, columnspan=3,\n sticky=tk.W+tk.N)\n\n if not self.frozen:\n self.oscany.config(state='disable')\n self.oscan.config(state='normal')\n self.clear.config(state='normal')\n if g.cpars['telins_name'] == 'GTC':\n self.nod.config(state='normal')\n else:\n self.nod.config(state='disable')\n self.wframe.enable()\n status = self.wframe.check()\n\n # exposure delay\n if self.expose.ok():\n self.expose.config(bg=g.COL['main'])\n else:\n self.expose.config(bg=g.COL['warn'])\n status = False\n\n # don't allow binning other than 1, 2 in overscan or prescan mode\n if self.oscan() or self.oscany():\n if xbin not in (1, 2):\n status = False\n xbinw.config(bg=g.COL['error'])\n if ybin not in (1, 2):\n status = False\n ybinw.config(bg=g.COL['error'])\n\n # disable clear if nodding enabled. re-enable if not drift\n if not self.frozen:\n if self.nod() or self.nodPattern:\n self.clear.config(state='disabled')\n self.clearLab.config(state='disabled')\n elif not self.isDrift():\n self.clear.config(state='normal')\n self.clearLab.config(state='normal')\n\n # allow posting if parameters are OK. update count and SN estimates too\n if status:\n if (g.cpars['hcam_server_on'] and g.cpars['eso_server_online'] and\n g.observe.start['state'] == 'disabled' and\n not isRunActive(g)):\n g.observe.start.enable()\n g.count.update()\n else:\n g.observe.start.disable()\n\n return status", "response": "Checks validity of the instrument parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfreezes all settings so they cannot be altered", "response": "def freeze(self):\n \"\"\"\n Freeze all settings so they cannot be altered\n \"\"\"\n self.app.disable()\n self.clear.disable()\n self.nod.disable()\n self.led.disable()\n self.dummy.disable()\n self.readSpeed.disable()\n self.expose.disable()\n self.number.disable()\n self.wframe.disable(everything=True)\n self.nmult.disable()\n self.frozen = True"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a string suitable to send off to rtplot when the current window parameters are OK. Returns null string if the current window parameters are not OK.", "response": "def getRtplotWins(self):\n \"\"\"\"\n Returns a string suitable to sending off to rtplot when\n it asks for window parameters. Returns null string '' if\n the windows are not OK. This operates on the basis of\n trying to send something back, even if it might not be\n OK as a window setup. Note that we have to take care\n here not to update any GUI components because this is\n called outside of the main thread.\n \"\"\"\n try:\n if self.isFF():\n return 'fullframe\\r\\n'\n elif self.isDrift():\n xbin = self.wframe.xbin.value()\n ybin = self.wframe.ybin.value()\n nwin = 2*self.wframe.npair.value()\n ret = str(xbin) + ' ' + str(ybin) + ' ' + str(nwin) + '\\r\\n'\n for xsl, xsr, ys, nx, ny in self.wframe:\n ret += '{:d} {:d} {:d} {:d}\\r\\n'.format(\n xsl, ys, nx, ny\n )\n ret += '{:d} {:d} {:d} {:d}'.format(\n xsr, ys, nx, ny\n )\n return ret\n else:\n xbin = self.wframe.xbin.value()\n ybin = self.wframe.ybin.value()\n nwin = 4*self.wframe.nquad.value()\n ret = str(xbin) + ' ' + str(ybin) + ' ' + str(nwin) + '\\r\\n'\n for xsll, xsul, xslr, xsur, ys, nx, ny in self.wframe:\n ret += '{:d} {:d} {:d} {:d}\\r\\n'.format(\n xsll, ys, nx, ny\n )\n ret += '{:d} {:d} {:d} {:d}\\r\\n'.format(\n xsul, 1025 - ys - ny, nx, ny\n )\n ret += '{:d} {:d} {:d} {:d}\\r\\n'.format(\n xslr, ys, nx, ny\n )\n ret += '{:d} {:d} {:d} {:d}\\r\\n'.format(\n xsur, 1025 - ys - ny, nx, ny\n )\n return ret\n except:\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef timing(self):\n # drift mode y/n?\n isDriftMode = self.isDrift()\n # FF y/n?\n isFF = self.isFF()\n\n # Set the readout speed\n readSpeed = self.readSpeed()\n\n if readSpeed == 'Fast' and self.dummy():\n video = VIDEO_FAST\n elif readSpeed == 'Slow' and self.dummy():\n video = VIDEO_SLOW\n elif not self.dummy():\n video = VIDEO_SLOW_SE\n else:\n raise DriverError('InstPars.timing: readout speed = ' +\n readSpeed + ' not recognised.')\n\n if self.fastClk():\n DUMP_TIME = DUMP_TIME_FAST\n VCLOCK_FRAME = VCLOCK_FAST\n VCLOCK_STORAGE = VCLOCK_FAST\n HCLOCK = HCLOCK_FAST\n else:\n DUMP_TIME = DUMP_TIME_SLOW\n VCLOCK_FRAME = VCLOCK_FRAME_SLOW\n VCLOCK_STORAGE = VCLOCK_STORAGE_SLOW\n HCLOCK = HCLOCK_SLOW\n\n # clear chip on/off?\n lclear = not isDriftMode and self.clear()\n\n # overscan read or not\n oscan = not isDriftMode and self.oscan()\n oscany = not isDriftMode and self.oscany()\n\n # get exposure delay\n expose = self.expose.value()\n\n # window parameters\n xbin = self.wframe.xbin.value()\n ybin = self.wframe.ybin.value()\n if isDriftMode:\n nwin = 1 # number of windows per output\n dys = self.wframe.ys[0].value() - 1\n dnx = self.wframe.nx[0].value()\n dny = self.wframe.ny[0].value()\n dxsl = self.wframe.xsl[0].value()\n dxsr = self.wframe.xsr[0].value()\n # differential shift needed to line both\n # windows up with the edge of the chip\n diffshift = abs(dxsl - 1 - (2*FFX - dxsr - dnx + 1))\n elif isFF:\n nwin = 1\n ys, nx, ny = [0], [1024], [512]\n else:\n ys, nx, ny = [], [], []\n xse, xsf, xsg, xsh = [], [], [], []\n nwin = self.wframe.nquad.value()\n for xsll, xsul, xslr, xsur, ysv, nxv, nyv in self.wframe:\n xse.append(xsll - 1)\n xsf.append(2049 - xslr - nxv)\n xsg.append(2049 - xsur - nxv)\n xsh.append(xsul - 1)\n ys.append(ysv-1)\n nx.append(nxv)\n ny.append(nyv)\n\n # convert timing parameters to seconds\n expose_delay = expose\n\n # clear chip by VCLOCK-ing the image and area and dumping storage area (x5)\n if lclear:\n clear_time = 5*(FFY*VCLOCK_FRAME + FFY*DUMP_TIME)\n else:\n clear_time = 0.0\n\n if isDriftMode:\n # for drift mode, we need the number of windows in the pipeline\n # and the pipeshift\n nrows = FFY # number of rows in storage area\n pnwin = int(((nrows / dny) + 1)/2)\n pshift = nrows - (2*pnwin-1)*dny\n frame_transfer = (dny+dys)*VCLOCK_FRAME\n\n yshift = [dys*VCLOCK_STORAGE]\n\n # After placing the window adjacent to the serial register, the\n # register must be cleared by clocking out the entire register,\n # taking FFX hclocks.\n line_clear = [0.]\n if yshift[0] != 0:\n line_clear[0] = DUMP_TIME\n\n # to calculate number of HCLOCKS needed to read a line in\n # drift mode we have to account for the diff shifts and dumping.\n # first perform diff shifts\n # for now we need this *2 (for quadrants E, H or F, G)\n numhclocks = 2*diffshift\n # now add the amount of clocks needed to get\n # both windows to edge of chip\n if dxsl - 1 > 2*FFX - dxsr - dnx + 1:\n # it was the left window that got the diff shift,\n # so the number of hclocks increases by the amount\n # needed to get the RH window to the edge\n numhclocks += 2*FFX - dxsr - dnx + 1\n else:\n # vice versa\n numhclocks += dxsl - 1\n # now we actually clock the windows themselves\n numhclocks += dnx\n # finally, we need to hclock the additional pre-scan pixels\n numhclocks += 2*PRSCX\n\n # here is the total time to read the whole line\n line_read = [VCLOCK_STORAGE*ybin + numhclocks*HCLOCK +\n video*dnx/xbin + DUMP_TIME + 2*SETUP_READ]\n\n readout = [(dny/ybin) * line_read[0]]\n elif isFF:\n # move entire image into storage area\n frame_transfer = FFY*VCLOCK_FRAME + DUMP_TIME\n\n yshift = [0]\n line_clear = [0]\n\n numhclocks = FFX + PRSCX\n line_read = [VCLOCK_STORAGE*ybin + numhclocks*HCLOCK +\n video*nx[0]/xbin + SETUP_READ]\n if oscan:\n line_read[0] += video*PRSCX/xbin\n nlines = ny[0]/ybin if not oscany else (ny[0] + 8/ybin)\n readout = [nlines*line_read[0]]\n else:\n # windowed mode\n # move entire image into storage area\n frame_transfer = FFY*VCLOCK_FRAME + DUMP_TIME\n\n # dump rows in storage area up to start of the window without changing the\n # image area.\n yshift = nwin*[0.]\n yshift[0] = ys[0]*DUMP_TIME\n for nw in range(1, nwin):\n yshift[nw] = (ys[nw]-ys[nw-1]-ny[nw-1])*DUMP_TIME\n\n line_clear = nwin*[0.]\n # Naidu always dumps the serial register, in windowed mode\n # regardless of whether we need to or not\n for nw in range(nwin):\n line_clear[nw] = DUMP_TIME\n\n # calculate how long it takes to shift one row into the serial\n # register shift along serial register and then read out the data.\n # total number of hclocks needs to account for diff shifts of\n # windows, carried out in serial\n numhclocks = nwin*[0]\n for nw in range(nwin):\n common_shift = min(xse[nw], xsf[nw], xsg[nw], xsh[nw])\n diffshifts = sum((xs-common_shift for xs in (xse[nw], xsf[nw], xsg[nw], xsh[nw])))\n numhclocks[nw] = 2*PRSCX + common_shift + diffshifts + nx[nw]\n\n line_read = nwin*[0.]\n # line read includes vclocking a row, all the hclocks, digitising pixels and dumping serial register\n # when windows are read out.\n for nw in range(nwin):\n line_read[nw] = (VCLOCK_STORAGE*ybin + numhclocks[nw]*HCLOCK +\n video*nx[nw]/xbin + 2*SETUP_READ + DUMP_TIME)\n if oscan:\n line_read[nw] += video*PRSCX/xbin\n\n # multiply time to shift one row into serial register by\n # number of rows for total readout time\n readout = nwin*[0.]\n for nw in range(nwin):\n nlines = ny[nw]/ybin if not oscany else (ny[nw] + 8/ybin)\n readout[nw] = nlines * line_read[nw]\n\n # now get the total time to read out one exposure.\n cycleTime = expose_delay + clear_time + frame_transfer\n if isDriftMode:\n cycleTime += pshift*VCLOCK_STORAGE + yshift[0] + line_clear[0] + readout[0]\n else:\n for nw in range(nwin):\n cycleTime += yshift[nw] + line_clear[nw] + readout[nw]\n\n # use 5sec estimate for nod time\n # TODO: replace with accurate estimate\n if self.nod() and lclear:\n cycleTime += 5\n elif self.nod():\n g = get_root(self).globals\n g.clog.warn('ERR: dithering enabled with clear mode off')\n\n frameRate = 1.0/cycleTime\n expTime = expose_delay if lclear else cycleTime - frame_transfer\n deadTime = cycleTime - expTime\n dutyCycle = 100.0*expTime/cycleTime\n return (expTime, deadTime, cycleTime, dutyCycle, frameRate)", "response": "Return a tuple of the current set of timing information for the current set of instrument parameters."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef loadJSON(self, json_string):\n g = get_root(self).globals\n user = json.loads(json_string)['user']\n\n def setField(widget, field):\n val = user.get(field)\n if val is not None:\n widget.set(val)\n\n setField(self.prog_ob.obid, 'OB')\n setField(self.target, 'target')\n setField(self.prog_ob.progid, 'ID')\n setField(self.pi, 'PI')\n setField(self.observers, 'Observers')\n setField(self.comment, 'comment')\n setField(self.filter, 'filters')\n setField(g.observe.rtype, 'flags')", "response": "Sets the values of the run parameters given a JSON string"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dumpJSON(self):\n g = get_root(self).globals\n dtype = g.observe.rtype()\n if dtype == 'bias':\n target = 'BIAS'\n elif dtype == 'flat':\n target = 'FLAT'\n elif dtype == 'dark':\n target = 'DARK'\n else:\n target = self.target.value()\n\n return dict(\n target=target,\n ID=self.prog_ob.progid.value(),\n PI=self.pi.value(),\n OB='{:04d}'.format(self.prog_ob.obid.value()),\n Observers=self.observers.value(),\n comment=self.comment.value(),\n flags=dtype,\n filters=self.filter.value()\n )", "response": "Dumps current parameters to JSON compatible dictionary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck the validity of the run parameters. Returns True if the flag is False otherwise.", "response": "def check(self, *args):\n \"\"\"\n Checks the validity of the run parameters. Returns\n flag (True = OK), and a message which indicates the\n nature of the problem if the flag is False.\n \"\"\"\n\n ok = True\n msg = ''\n g = get_root(self).globals\n dtype = g.observe.rtype()\n expert = g.cpars['expert_level'] > 0\n\n if dtype == 'bias' or dtype == 'flat' or dtype == 'dark':\n self.pi.configure(state='disable')\n self.prog_ob.configure(state='disable')\n self.target.disable()\n else:\n if expert:\n self.pi.configure(state='normal')\n self.prog_ob.configure(state='normal')\n self.prog_ob.enable()\n else:\n self.prog_ob.configure(state='disable')\n self.pi.configure(state='disable')\n self.prog_ob.disable()\n self.target.enable()\n\n if g.cpars['require_run_params']:\n if self.target.ok():\n self.target.entry.config(bg=g.COL['main'])\n else:\n self.target.entry.config(bg=g.COL['error'])\n ok = False\n msg += 'Target name field cannot be blank\\n'\n\n if dtype == 'data caution' or \\\n dtype == 'data' or dtype == 'technical':\n\n if self.prog_ob.ok():\n self.prog_ob.config(bg=g.COL['main'])\n else:\n self.prog_ob.config(bg=g.COL['error'])\n ok = False\n msg += 'Programme or OB ID field cannot be blank\\n'\n\n if self.pi.ok():\n self.pi.config(bg=g.COL['main'])\n else:\n self.pi.config(bg=g.COL['error'])\n ok = False\n msg += 'Principal Investigator field cannot be blank\\n'\n\n if self.observers.ok():\n self.observers.config(bg=g.COL['main'])\n else:\n self.observers.config(bg=g.COL['error'])\n ok = False\n msg += 'Observers field cannot be blank'\n return (ok, msg)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef freeze(self):\n self.target.disable()\n self.filter.configure(state='disable')\n self.prog_ob.configure(state='disable')\n self.pi.configure(state='disable')\n self.observers.configure(state='disable')\n self.comment.configure(state='disable')", "response": "Freeze all settings so that they can t be altered"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef unfreeze(self):\n g = get_root(self).globals\n self.filter.configure(state='normal')\n dtype = g.observe.rtype()\n if dtype == 'data caution' or dtype == 'data' or dtype == 'technical':\n self.prog_ob.configure(state='normal')\n self.pi.configure(state='normal')\n self.target.enable()\n self.observers.configure(state='normal')\n self.comment.configure(state='normal')", "response": "Unfreeze all settings so that they can be altered"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if the current observing parameters are OK.", "response": "def checkUpdate(self, *args):\n \"\"\"\n Updates values after first checking instrument parameters are OK.\n This is not integrated within update to prevent ifinite recursion\n since update gets called from ipars.\n \"\"\"\n g = get_root(self).globals\n if not self.check():\n g.clog.warn('Current observing parameters are not valid.')\n return False\n\n if not g.ipars.check():\n g.clog.warn('Current instrument parameters are not valid.')\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the current instance of the assessment is OK.", "response": "def check(self):\n \"\"\"\n Checks values\n \"\"\"\n status = True\n g = get_root(self).globals\n if self.mag.ok():\n self.mag.config(bg=g.COL['main'])\n else:\n self.mag.config(bg=g.COL['warn'])\n status = False\n\n if self.airmass.ok():\n self.airmass.config(bg=g.COL['main'])\n else:\n self.airmass.config(bg=g.COL['warn'])\n status = False\n\n if self.seeing.ok():\n self.seeing.config(bg=g.COL['main'])\n else:\n self.seeing.config(bg=g.COL['warn'])\n status = False\n\n return status"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update(self, *args):\n g = get_root(self).globals\n expTime, deadTime, cycleTime, dutyCycle, frameRate = g.ipars.timing()\n\n total, peak, peakSat, peakWarn, ston, ston3 = \\\n self.counts(expTime, cycleTime)\n\n if cycleTime < 0.01:\n self.cadence.config(text='{0:7.5f} s'.format(cycleTime))\n elif cycleTime < 0.1:\n self.cadence.config(text='{0:6.4f} s'.format(cycleTime))\n elif cycleTime < 1.:\n self.cadence.config(text='{0:5.3f} s'.format(cycleTime))\n elif cycleTime < 10.:\n self.cadence.config(text='{0:4.2f} s'.format(cycleTime))\n elif cycleTime < 100.:\n self.cadence.config(text='{0:4.1f} s'.format(cycleTime))\n elif cycleTime < 1000.:\n self.cadence.config(text='{0:4.0f} s'.format(cycleTime))\n else:\n self.cadence.config(text='{0:5.0f} s'.format(cycleTime))\n\n if expTime < 0.01:\n self.exposure.config(text='{0:7.5f} s'.format(expTime))\n elif expTime < 0.1:\n self.exposure.config(text='{0:6.4f} s'.format(expTime))\n elif expTime < 1.:\n self.exposure.config(text='{0:5.3f} s'.format(expTime))\n elif expTime < 10.:\n self.exposure.config(text='{0:4.2f} s'.format(expTime))\n elif expTime < 100.:\n self.exposure.config(text='{0:4.1f} s'.format(expTime))\n elif expTime < 1000.:\n self.exposure.config(text='{0:4.0f} s'.format(expTime))\n else:\n self.exposure.config(text='{0:5.0f} s'.format(expTime))\n\n self.duty.config(text='{0:4.1f} %'.format(dutyCycle))\n self.peak.config(text='{0:d} cts'.format(int(round(peak))))\n if peakSat:\n self.peak.config(bg=g.COL['error'])\n elif peakWarn:\n self.peak.config(bg=g.COL['warn'])\n else:\n self.peak.config(bg=g.COL['main'])\n\n self.total.config(text='{0:d} cts'.format(int(round(total))))\n self.ston.config(text='{0:.1f}'.format(ston))\n self.ston3.config(text='{0:.1f}'.format(ston3))", "response": "Updates the values of the cadence and exposure objects for the current instance of the base class."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef counts(self, expTime, cycleTime, ap_scale=1.6, ndiv=5):\n\n # code directly translated from Java equivalent.\n g = get_root(self).globals\n\n # Set the readout speed\n readSpeed = g.ipars.readSpeed()\n if readSpeed == 'Fast':\n gain = GAIN_FAST\n read = RNO_FAST\n elif readSpeed == 'Slow':\n gain = GAIN_SLOW\n read = RNO_SLOW\n else:\n raise DriverError('CountsFrame.counts: readout speed = ' +\n readSpeed + ' not recognised.')\n\n xbin, ybin = g.ipars.wframe.xbin.value(), g.ipars.wframe.ybin.value()\n\n # calculate SN info.\n zero, sky, skyTot, darkTot = 0., 0., 0., 0.\n total, peak, correct, signal, readTot, seeing = 0., 0., 0., 0., 0., 0.\n noise, narcsec, npix, signalToNoise3 = 1., 0., 0., 0.\n\n tinfo = g.TINS[g.cpars['telins_name']]\n filtnam = self.filter.value()\n\n zero = tinfo['zerop'][filtnam]\n mag = self.mag.value()\n seeing = self.seeing.value()\n sky = g.SKY[self.moon.value()][filtnam]\n airmass = self.airmass.value()\n plateScale = tinfo['plateScale']\n\n # calculate expected electrons\n total = 10.**((zero-mag-airmass*g.EXTINCTION[filtnam])/2.5)*expTime\n\n # compute fraction that fall in central pixel\n # assuming target exactly at its centre. Do this\n # by splitting each pixel of the central (potentially\n # binned) pixel into ndiv * ndiv points at\n # which the seeing profile is added. sigma is the\n # RMS seeing in terms of pixels.\n sigma = seeing/g.EFAC/plateScale\n\n sum = 0.\n for iyp in range(ybin):\n yoff = -ybin/2.+iyp\n for ixp in range(xbin):\n xoff = -xbin/2.+ixp\n for iys in range(ndiv):\n y = (yoff + (iys+0.5)/ndiv)/sigma\n for ixs in range(ndiv):\n x = (xoff + (ixs+0.5)/ndiv)/sigma\n sum += math.exp(-(x*x+y*y)/2.)\n peak = total*sum/(2.*math.pi*sigma**2*ndiv**2)\n\n # Work out fraction of flux in aperture with radius AP_SCALE*seeing\n correct = 1. - math.exp(-(g.EFAC*ap_scale)**2/2.)\n\n # expected sky e- per arcsec\n skyPerArcsec = 10.**((zero-sky)/2.5)*expTime\n # skyPerPixel = skyPerArcsec*plateScale**2*xbin*ybin\n narcsec = math.pi*(ap_scale*seeing)**2\n skyTot = skyPerArcsec*narcsec\n npix = math.pi*(ap_scale*seeing/plateScale)**2/xbin/ybin\n\n signal = correct*total # in electrons\n darkTot = npix*DARK_E*expTime # in electrons\n readTot = npix*read**2 # in electrons\n\n # noise, in electrons\n noise = math.sqrt(readTot + darkTot + skyTot + signal)\n\n # Now compute signal-to-noise in 3 hour seconds run\n signalToNoise3 = signal/noise*math.sqrt(3*3600./cycleTime)\n\n # convert from electrons to counts\n total /= gain\n peak /= gain\n\n warn = 25000\n sat = 60000\n\n peakSat = peak > sat\n peakWarn = peak > warn\n\n return (total, peak, peakSat, peakWarn, signal/noise, signalToNoise3)", "response": "Compute the counts per pixel total counts sky counts and signal to noise per exposure and aperture radius."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndisable the button if in non - expert mode.", "response": "def disable(self):\n \"\"\"\n Disable the button, if in non-expert mode.\n \"\"\"\n w.ActButton.disable(self)\n g = get_root(self).globals\n if self._expert:\n self.config(bg=g.COL['start'])\n else:\n self.config(bg=g.COL['startD'])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setExpert(self):\n w.ActButton.setExpert(self)\n g = get_root(self).globals\n self.config(bg=g.COL['start'])", "response": "Turn on expert status whereby the button is always enabled regardless of its activity status."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef act(self):\n g = get_root(self).globals\n # check binning against overscan\n msg = \"\"\"\n HiperCAM has an o/scan of 50 pixels.\n Your binning does not fit into this\n region. Some columns will contain a\n mix of o/scan and data.\n\n Click OK if you wish to continue.\"\"\"\n if g.ipars.oscan():\n xbin, ybin = g.ipars.wframe.xbin.value(), g.ipars.wframe.ybin.value()\n if xbin not in (1, 2, 5, 10) or ybin not in (1, 2, 5, 10):\n if not messagebox.askokcancel('Binning alert', msg):\n return False\n\n # Check instrument pars are OK\n if not g.ipars.check():\n g.clog.warn('Invalid instrument parameters; save failed.')\n return False\n\n # create JSON to post\n data = createJSON(g)\n\n # POST\n try:\n success = postJSON(g, data)\n if not success:\n raise Exception('postJSON returned False')\n except Exception as err:\n g.clog.warn(\"Failed to post data to servers\")\n g.clog.warn(str(err))\n return False\n\n # Is nod enabled? Should we start GTC offsetter?\n try:\n success = startNodding(g, data)\n if not success:\n raise Exception('Failed to start dither: response was false')\n except Exception as err:\n g.clog.warn(\"Failed to start GTC offsetter\")\n g.clog.warn(str(err))\n return False\n\n # START\n try:\n success = execCommand(g, 'start')\n if not success:\n raise Exception(\"Start command failed: check server response\")\n except Exception as err:\n g.clog.warn('Failed to start run')\n g.clog.warn(str(err))\n return False\n\n # Send first offset if nodding enabled.\n # Initial trigger is sent after first offset, otherwise we'll hang indefinitely\n try:\n success = forceNod(g, data)\n if not success:\n raise Exception('Failed to send intitial offset and trigger - exposure will be paused indefinitely')\n except Exception as err:\n g.clog.warn('Run is paused indefinitely')\n g.clog.warn('use \"ngcbCmd seq start\" to fix')\n g.clog.warn(str(err))\n\n # Run successfully started.\n # enable stop button, disable Start\n # also make inactive until RunType select box makes active again\n # start run timer\n # finally, clear table which stores TCS info during this run\n self.disable()\n self.run_type_set = False\n g.observe.stop.enable()\n g.info.timer.start()\n g.info.clear_tcs_table()\n return True", "response": "Actually asks if user has permission to start a new entry in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef act(self):\n g = get_root(self).globals\n fname = filedialog.askopenfilename(\n defaultextension='.json',\n filetypes=[('json files', '.json'), ('fits files', '.fits')],\n initialdir=g.cpars['app_directory'])\n if not fname:\n g.clog.warn('Aborted load from disk')\n return False\n\n # load json\n if fname.endswith('.json'):\n with open(fname) as ifname:\n json_string = ifname.read()\n else:\n json_string = jsonFromFits(fname)\n\n # load up the instrument settings\n g.ipars.loadJSON(json_string)\n\n # load up the run parameters\n g.rpars.loadJSON(json_string)\n\n return True", "response": "Carries out the action associated with the Load button."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncarry out the action associated with the Save button.", "response": "def act(self):\n \"\"\"\n Carries out the action associated with the Save button\n \"\"\"\n g = get_root(self).globals\n g.clog.info('\\nSaving current application to disk')\n\n # check instrument parameters\n if not g.ipars.check():\n g.clog.warn('Invalid instrument parameters; save failed.')\n return False\n\n # check run parameters\n rok, msg = g.rpars.check()\n if not rok:\n g.clog.warn('Invalid run parameters; save failed.')\n g.clog.warn(msg)\n return False\n\n # Get data to save\n data = createJSON(g, full=False)\n\n # Save to disk\n if saveJSON(g, data):\n # modify buttons\n g.observe.load.enable()\n g.observe.unfreeze.disable()\n\n # unfreeze the instrument and run params\n g.ipars.unfreeze()\n g.rpars.unfreeze()\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef act(self):\n g = get_root(self).globals\n g.ipars.unfreeze()\n g.rpars.unfreeze()\n g.observe.load.enable()\n self.disable()", "response": "Carries out the action associated with the Unfreeze button"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the expert level of the user s current user.", "response": "def setExpertLevel(self):\n \"\"\"\n Set expert level\n \"\"\"\n g = get_root(self).globals\n level = g.cpars['expert_level']\n\n # now set whether buttons are permanently enabled or not\n if level == 0 or level == 1:\n self.load.setNonExpert()\n self.save.setNonExpert()\n self.unfreeze.setNonExpert()\n self.start.setNonExpert()\n self.stop.setNonExpert()\n\n elif level == 2:\n self.load.setExpert()\n self.save.setExpert()\n self.unfreeze.setExpert()\n self.start.setExpert()\n self.stop.setExpert()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef deep_getattr(clsdict: Dict[str, Any],\n bases: Tuple[Type[object], ...],\n name: str,\n default: Any = _missing) -> Any:\n \"\"\"\n Acts just like ``getattr`` would on a constructed class object, except this operates\n on the pre-construction class dictionary and base classes. In other words, first we\n look for the attribute in the class dictionary, and then we search all the base\n classes (in method resolution order), finally returning the default value if the\n attribute was not found in any of the class dictionary or base classes (or it raises\n ``AttributeError`` if no default was given).\n \"\"\"\n value = clsdict.get(name, _missing)\n if value != _missing:\n return value\n for base in bases:\n value = getattr(base, name, _missing)\n if value != _missing:\n return value\n if default != _missing:\n return default\n raise AttributeError(name)", "response": "This function operates on a class dictionary and base classes and returns the value of the attribute name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getattr(self, name, default: Any = _missing):\n return deep_getattr(self.clsdict, self.bases, name, default)", "response": "Get the attribute of the base class."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the fully qualified name of the class.", "response": "def qualname(self) -> str:\n \"\"\"\n Returns the fully qualified name of the class-under-construction, if possible,\n otherwise just the class name.\n \"\"\"\n if self.module:\n return self.module + '.' + self.name\n return self.name"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_abstract(self) -> bool:\n meta_value = getattr(self.clsdict.get('Meta'), 'abstract', False)\n return self.clsdict.get(ABSTRACT_ATTR, meta_value) is True", "response": "Whether or not the class - under - construction was declared as abstract."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_value(self, Meta: Type[object], base_classes_meta, mcs_args: McsArgs) -> Any:\n value = self.default\n if self.inherit and base_classes_meta is not None:\n value = getattr(base_classes_meta, self.name, value)\n if Meta is not None:\n value = getattr(Meta, self.name, value)\n return value", "response": "Returns the value for self. name given the class - under - construction s class\n Meta and McsArgs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of MetaOption instances that this factory supports.", "response": "def _get_meta_options(self) -> List[MetaOption]:\n \"\"\"\n Returns a list of :class:`MetaOption` instances that this factory supports.\n \"\"\"\n return [option if isinstance(option, MetaOption) else option()\n for option in self._options]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _contribute_to_class(self, mcs_args: McsArgs):\n self._mcs_args = mcs_args\n\n Meta = mcs_args.clsdict.pop('Meta', None) # type: Type[object]\n base_classes_meta = mcs_args.getattr('Meta', None) # type: MetaOptionsFactory\n\n mcs_args.clsdict['Meta'] = self # must come before _fill_from_meta, because\n # some meta options may depend upon having\n # access to the values of earlier meta options\n self._fill_from_meta(Meta, base_classes_meta, mcs_args)\n\n for option in self._get_meta_options():\n option_value = getattr(self, option.name, None)\n option.contribute_to_class(mcs_args, option_value)", "response": "This method is called by _parse_and_contribute_to_class."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfill the instance with the values from the Meta object.", "response": "def _fill_from_meta(self, Meta: Type[object], base_classes_meta, mcs_args: McsArgs):\n \"\"\"\n Iterate over our supported meta options, and set attributes on the factory\n instance (self) for each meta option's name/value. Raises ``TypeError`` if\n we discover any unsupported meta options on the class-under-construction's\n ``class Meta``.\n \"\"\"\n # Exclude private/protected fields from the Meta\n meta_attrs = {} if not Meta else {k: v for k, v in vars(Meta).items()\n if not k.startswith('_')}\n\n for option in self._get_meta_options():\n existing = getattr(self, option.name, None)\n if existing and not (existing in self._allowed_properties\n and not isinstance(existing, property)):\n raise RuntimeError(\"Can't override field {name}.\"\n \"\".format(name=option.name))\n value = option.get_value(Meta, base_classes_meta, mcs_args)\n option.check_value(value, mcs_args)\n meta_attrs.pop(option.name, None)\n if option.name != '_':\n setattr(self, option.name, value)\n\n if meta_attrs:\n # Only allow attributes on the Meta that have a respective MetaOption\n raise TypeError(\n '`class Meta` for {cls} got unknown attribute(s) {attrs}'.format(\n cls=mcs_args.name,\n attrs=', '.join(sorted(meta_attrs.keys()))))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a reference to an object using CORBA", "response": "def get_object(self, binding_name, cls):\n \"\"\"\n Get a reference to a remote object using CORBA\n \"\"\"\n return self._state.get_object(self, binding_name, cls)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting a reference to a remote object using CORBA.", "response": "def get_object(conn, binding_name, object_cls):\n \"\"\"\n Get a reference to a remote object using CORBA\n \"\"\"\n try:\n obj = conn.rootContext.resolve(binding_name)\n narrowed = obj._narrow(object_cls)\n except CORBA.TRANSIENT:\n raise IOError('Attempt to retrieve object failed')\n\n if narrowed is None:\n raise IOError('Attempt to retrieve object got a different class of object')\n\n return narrowed"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set(self, num):\n self._value = str(int(num))\n self._variable.set(self._value)", "response": "Sets the current value equal to num\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_key_release_repeat(self, *dummy):\n self.has_prev_key_release = self.after_idle(self.on_key_release, dummy)", "response": "Avoid repeated trigger of callback.\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _pollMouse(self):\n if self._mouseJustPressed:\n delay = 300\n self._mouseJustPressed = False\n else:\n delay = 100\n\n if self._leftMousePressed:\n self.add(1)\n self.after_id = self.after(delay, self._pollMouse)\n\n if self._shiftLeftMousePressed:\n self.add(10)\n self.after_id = self.after(delay, self._pollMouse)\n\n if self._rightMousePressed:\n self.sub(1)\n self.after_id = self.after(delay, self._pollMouse)\n\n if self._shiftRightMousePressed:\n self.sub(10)\n self.after_id = self.after(delay, self._pollMouse)", "response": "Polls at the mouse and adds a slight delay at the mouse end."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_bind(self):\n IntegerEntry.set_bind(self)\n self.bind('', lambda e: self.set(0))", "response": "Sets the key bindings for this entry"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate(self, value):\n try:\n # trap blank fields here\n if not self.blank or value:\n v = int(value)\n if v < 0:\n return None\n return value\n except ValueError:\n return None", "response": "Validate the value of the recordstore entry."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd num to the current value of the attribute.", "response": "def add(self, num):\n \"\"\"\n Adds num to the current value\n \"\"\"\n try:\n val = self.value() + num\n except:\n val = num\n self.set(max(0, val))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sub(self, num):\n try:\n val = self.value() - num\n except:\n val = -num\n self.set(max(0, val))", "response": "Subtracts num from the current value"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ok(self):\n try:\n v = int(self._value)\n if v < 0:\n return False\n else:\n return True\n except:\n return False", "response": "Returns True if OK to use False otherwise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the key bindings for this entry", "response": "def set_bind(self):\n \"\"\"\n Sets key bindings -- we need this more than once\n \"\"\"\n IntegerEntry.set_bind(self)\n self.bind('', lambda e: self.set(self.imin))\n self.bind('', lambda e: self.set(self.imax))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating the value of the recordset.", "response": "def validate(self, value):\n \"\"\"\n Applies the validation criteria.\n Returns value, new value, or None if invalid.\n\n Overload this in derived classes.\n \"\"\"\n try:\n # trap blank fields here\n if not self.blank or value:\n v = int(value)\n if v < self.imin or v > self.imax:\n return None\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding num to the current value", "response": "def add(self, num):\n \"\"\"\n Adds num to the current value\n \"\"\"\n try:\n val = self.value() + num\n except:\n val = num\n self.set(min(self.imax, max(self.imin, val)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if OK to use False otherwise", "response": "def ok(self):\n \"\"\"\n Returns True if OK to use, else False\n \"\"\"\n try:\n v = int(self._value)\n if v < self.imin or v > self.imax:\n return False\n else:\n return True\n except:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the key bindings for this instance.", "response": "def set_bind(self):\n \"\"\"\n Sets key bindings -- we need this more than once\n \"\"\"\n RangedInt.set_bind(self)\n self.unbind('')\n self.unbind('')\n self.bind('', lambda e: self.set(self._min()))\n self.bind('', lambda e: self.set(self._max()))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add(self, num):\n try:\n val = self.value() + num\n except:\n val = num\n\n chunk = self.mfac.value()\n if val % chunk > 0:\n if num > 0:\n val = chunk*(val // chunk + 1)\n elif num < 0:\n val = chunk*(val // chunk)\n\n val = max(self._min(), min(self._max(), val))\n self.set(val)", "response": "Adds num to the current value jumping up the next next"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ok(self):\n try:\n v = int(self._value)\n chunk = self.mfac.value()\n if v < self.imin or v > self.imax or (v % chunk != 0):\n return False\n else:\n return True\n except:\n return False", "response": "Returns True if OK to use False otherwise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset key bindings for the current locale.", "response": "def set_bind(self):\n \"\"\"\n Sets key bindings -- we need this more than once\n \"\"\"\n IntegerEntry.set_bind(self)\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n\n self.bind('', lambda e: self.add(1))\n self.bind('', lambda e: self.sub(1))\n self.bind('', lambda e: self.add(1))\n self.bind('', lambda e: self.sub(1))\n self.bind('', self._enter)\n self.bind('', lambda e: self.set(self.allowed[0]))\n self.bind('', lambda e: self.set(self.allowed[-1]))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nunbinds the key bindings for the current entry.", "response": "def set_unbind(self):\n \"\"\"\n Unsets key bindings -- we need this more than once\n \"\"\"\n IntegerEntry.set_unbind(self)\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')\n self.unbind('')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvalidating the value of the national product of the current user s log entry.", "response": "def validate(self, value):\n \"\"\"\n Applies the validation criteria.\n Returns value, new value, or None if invalid.\n\n Overload this in derived classes.\n \"\"\"\n try:\n v = int(value)\n if v not in self.allowed:\n return None\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the current value to num", "response": "def set(self, num):\n \"\"\"\n Sets current value to num\n \"\"\"\n if self.validate(num) is not None:\n self.index = self.allowed.index(num)\n IntegerEntry.set(self, num)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding num to the current value of the set.", "response": "def add(self, num):\n \"\"\"\n Adds num to the current value\n \"\"\"\n self.index = max(0, min(len(self.allowed)-1, self.index+num))\n self.set(self.allowed[self.index])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nvalidates the value of the key.", "response": "def validate(self, value):\n \"\"\"\n Applies the validation criteria.\n Returns value, new value, or None if invalid.\n\n Overload this in derived classes.\n \"\"\"\n try:\n # trap blank fields here\n if not self.blank or value:\n float(value)\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the current value equal to num", "response": "def set(self, num):\n \"\"\"\n Sets the current value equal to num\n \"\"\"\n self._value = str(round(float(num), self.nplaces))\n self._variable.set(self._value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting the bound methods for this object to the values of the current FloatEntry.", "response": "def set_bind(self):\n \"\"\"\n Sets key bindings -- we need this more than once\n \"\"\"\n FloatEntry.set_bind(self)\n self.bind('', lambda e: self.set(self.fmin))\n self.bind('', lambda e: self.set(self.fmax))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates the value of the record ID field.", "response": "def validate(self, value):\n \"\"\"\n Applies the validation criteria.\n Returns value, new value, or None if invalid.\n\n Overload this in derived classes.\n \"\"\"\n try:\n # trap blank fields here\n if not self.blank or value:\n v = float(value)\n if (self.allowzero and v != 0 and v < self.fmin) or \\\n (not self.allowzero and v < self.fmin) or v > self.fmax:\n return None\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds num to the current value", "response": "def add(self, num):\n \"\"\"\n Adds num to the current value\n \"\"\"\n try:\n val = self.value() + num\n except:\n val = num\n self.set(min(self.fmax, max(self.fmin, val)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if OK to use False otherwise", "response": "def ok(self):\n \"\"\"\n Returns True if OK to use, else False\n \"\"\"\n try:\n v = float(self._value)\n if v < self.fmin or v > self.fmax:\n return False\n else:\n return True\n except:\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate the value of the recordstore.", "response": "def validate(self, value):\n \"\"\"\n This prevents setting any value more precise than 0.00001\n \"\"\"\n try:\n # trap blank fields here\n if value:\n v = float(value)\n if (v != 0 and v < self.fmin) or v > self.fmax:\n return None\n if abs(round(100000*v)-100000*v) > 1.e-12:\n return None\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_min(self, fmin):\n if round(100000*fmin) != 100000*fmin:\n raise DriverError('utils.widgets.Expose.set_min: ' +\n 'fmin must be a multiple of 0.00001')\n self.fmin = fmin\n self.set(self.fmin)", "response": "Updates minimum value of the log entry."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisabling the button if in non - expert mode unset its activity flag come - what - may.", "response": "def disable(self):\n \"\"\"\n Disable the button, if in non-expert mode;\n unset its activity flag come-what-may.\n \"\"\"\n if not self._expert:\n self.config(state='disable')\n self._active = False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nturns off expert status whereby to allow a button to be disabled", "response": "def setNonExpert(self):\n \"\"\"\n Turns off 'expert' status whereby to allow a button to be disabled\n \"\"\"\n self._expert = False\n if self._active:\n self.enable()\n else:\n self.disable()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate the value of the key as specified by the user.", "response": "def validate(self, value):\n \"\"\"\n Applies the validation criteria.\n Returns value, new value, or None if invalid.\n \"\"\"\n try:\n coord.Angle(value, unit=self.unit)\n return value\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set(self, num):\n self._value = coord.Angle(num, unit=u.deg)\n self._variable.set(self.as_string())", "response": "Sets the current value equal to num\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add(self, quantity):\n newvalue = self._value + quantity\n self.set(newvalue.deg)", "response": "Adds an angle to the value\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsubtracting an angle from the value", "response": "def sub(self, quantity):\n \"\"\"\n Subtracts an angle from the value\n \"\"\"\n newvalue = self._value - quantity\n self.set(newvalue.deg)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ok(self):\n try:\n coord.Angle(self._value, unit=u.deg)\n return True\n except ValueError:\n return False", "response": "Returns True if OK to use False otherwise"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _callback(self, *dummy):\n # retrieve the value from the Entry\n value = self._variable.get()\n # run the validation. Returns None if no good\n newvalue = self.validate(value)\n if newvalue is None:\n # Invalid: restores previously stored value\n # no checker run.\n self._variable.set(self.as_string())\n else:\n # Store new value\n self._value = coord.Angle(value, unit=self.unit)\n if self.checker:\n self.checker(*dummy)", "response": "This is the callback function that is called when the value of the Entry is changed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncarrying out the action associated with Stop button", "response": "def act(self):\n \"\"\"\n Carries out the action associated with Stop button\n \"\"\"\n g = get_root(self).globals\n g.clog.debug('Stop pressed')\n\n # Stop exposure meter\n # do this first, so timer doesn't also try to enable idle mode\n g.info.timer.stop()\n\n def stop_in_background():\n try:\n self.stopping = True\n if execCommand(g, 'abort'):\n self.stopped_ok = True\n else:\n g.clog.warn('Failed to stop run')\n self.stopped_ok = False\n self.stopping = False\n except Exception as err:\n g.clog.warn('Failed to stop run. Error = ' + str(err))\n self.stopping = False\n self.stopped_ok = False\n\n # stopping can take a while during which the GUI freezes so run in\n # background.\n t = threading.Thread(target=stop_in_background)\n t.daemon = True\n t.start()\n self.after(500, self.check)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks the status of the stop exposure command", "response": "def check(self):\n \"\"\"\n Checks the status of the stop exposure command\n This is run in background and can take a few seconds\n \"\"\"\n g = get_root(self).globals\n if self.stopped_ok:\n # Exposure stopped OK; modify buttons\n self.disable()\n\n # try and write FITS table before enabling start button, otherwise\n # a new start will clear table\n try:\n insertFITSHDU(g)\n except Exception as err:\n g.clog.warn('Could not add FITS Table to run')\n g.clog.warn(str(err))\n\n g.observe.start.enable()\n g.setup.powerOn.disable()\n g.setup.powerOff.enable()\n\n # Report that run has stopped\n g.clog.info('Run stopped')\n\n # enable idle mode now run has stopped\n g.clog.info('Setting chips to idle')\n idle = {'appdata': {'app': 'Idle'}}\n try:\n success = postJSON(g, idle)\n if not success:\n raise Exception('postJSON returned false')\n except Exception as err:\n g.clog.warn('Failed to enable idle mode')\n g.clog.warn(str(err))\n\n g.clog.info('Stopping offsets (if running')\n try:\n success = stopNodding(g)\n if not success:\n raise Exception('Failed to stop dithering: response was false')\n except Exception as err:\n g.clog.warn('Failed to stop GTC offset script')\n g.clog.warn(str(err))\n\n return True\n\n elif self.stopping:\n # Exposure in process of stopping\n # Disable lots of buttons\n self.disable()\n g.observe.start.disable()\n g.setup.powerOn.disable()\n g.setup.powerOff.disable()\n\n # wait a second before trying again\n self.after(500, self.check)\n\n else:\n self.enable()\n g.observe.start.disable()\n g.setup.powerOn.disable()\n g.setup.powerOff.disable()\n # Start exposure meter\n g.info.timer.start()\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef modver(self, *args):\n g = get_root(self).globals\n if self.ok():\n tname = self.val.get()\n if tname in self.successes:\n # known to be in simbad\n self.verify.config(bg=g.COL['start'])\n elif tname in self.failures:\n # known not to be in simbad\n self.verify.config(bg=g.COL['stop'])\n else:\n # not known whether in simbad\n self.verify.config(bg=g.COL['main'])\n self.verify.config(state='normal')\n else:\n self.verify.config(bg=g.COL['main'])\n self.verify.config(state='disable')\n\n if self.callback is not None:\n self.callback()", "response": "Change the status of the current user to the status of the current verify button."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncarry out the action associated with Verify button", "response": "def act(self):\n \"\"\"\n Carries out the action associated with Verify button\n \"\"\"\n tname = self.val.get()\n g = get_root(self).globals\n g.clog.info('Checking ' + tname + ' in simbad')\n try:\n ret = checkSimbad(g, tname)\n if len(ret) == 0:\n self.verify.config(bg=g.COL['stop'])\n g.clog.warn('No matches to \"' + tname + '\" found.')\n if tname not in self.failures:\n self.failures.append(tname)\n elif len(ret) == 1:\n self.verify.config(bg=g.COL['start'])\n g.clog.info(tname + ' verified OK in simbad')\n g.clog.info('Primary simbad name = ' + ret[0]['Name'])\n if tname not in self.successes:\n self.successes.append(tname)\n else:\n g.clog.warn('More than one match to \"' + tname + '\" found')\n self.verify.config(bg=g.COL['stop'])\n if tname not in self.failures:\n self.failures.append(tname)\n except urllib.error.URLError:\n g.clog.warn('Simbad lookup timed out')\n except socket.timeout:\n g.clog.warn('Simbad lookup timed out')"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the expert level of the current object.", "response": "def setExpertLevel(self):\n \"\"\"\n Set expert level\n \"\"\"\n g = get_root(self).globals\n level = g.cpars['expert_level']\n\n # first define which buttons are visible\n if level == 0:\n # simple layout\n for button in self.all_buttons:\n button.grid_forget()\n\n # then re-grid the two simple ones\n self.powerOn.grid(row=0, column=0)\n self.powerOff.grid(row=0, column=1)\n\n elif level == 1 or level == 2:\n # first remove all possible buttons\n for button in self.all_buttons:\n button.grid_forget()\n\n # restore detailed layout\n self.cldcOn.grid(row=0, column=1)\n self.cldcOff.grid(row=1, column=1)\n self.seqStart.grid(row=2, column=1)\n self.seqStop.grid(row=3, column=1)\n self.ngcOnline.grid(row=0, column=0)\n self.ngcOff.grid(row=1, column=0)\n self.ngcStandby.grid(row=2, column=0)\n self.ngcReset.grid(row=3, column=0)\n\n # now set whether buttons are permanently enabled or not\n if level == 0 or level == 1:\n for button in self.all_buttons:\n button.setNonExpert()\n\n elif level == 2:\n for button in self.all_buttons:\n button.setExpert()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the expertise level of the current item", "response": "def setExpertLevel(self):\n \"\"\"\n Modifies widget according to expertise level, which in this\n case is just matter of hiding or revealing the button to\n set CCD temps\n \"\"\"\n g = get_root(self).globals\n level = g.cpars['expert_level']\n if level == 0:\n if self.val.get() == 'CCD TECs':\n self.val.set('Observe')\n self._changed()\n self.tecs.grid_forget()\n else:\n self.tecs.grid(row=0, column=3, sticky=tk.W)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef start(self):\n self.startTime = time.time()\n self.configure(text='{0: self.lastRiset and now > self.lastAstro):\n # Only re-compute rise and setting times when necessary,\n # and only re-compute when both rise/set and astro\n # twilight times have gone by\n\n # For sunrise and set we set the horizon down to match a\n # standard amount of refraction at the horizon and subtract size of disc\n horizon = -64*u.arcmin\n sunset = calc_riseset(now, 'sun', self.obs, 'next', 'setting', horizon)\n sunrise = calc_riseset(now, 'sun', self.obs, 'next', 'rising', horizon)\n\n # Astro twilight: geometric centre at -18 deg\n horizon = -18*u.deg\n astroset = calc_riseset(now, 'sun', self.obs, 'next', 'setting', horizon)\n astrorise = calc_riseset(now, 'sun', self.obs, 'next', 'rising', horizon)\n\n if sunrise > sunset:\n # In the day time we report the upcoming sunset and\n # end of evening twilight\n self.lriset.configure(text='Sets:', font=g.DEFAULT_FONT)\n self.lastRiset = sunset\n self.lastAstro = astroset\n\n elif astrorise > astroset and astrorise < sunrise:\n # During evening twilight, we report the sunset just\n # passed and end of evening twilight\n self.lriset.configure(text='Sets:', font=g.DEFAULT_FONT)\n horizon = -64*u.arcmin\n self.lastRiset = calc_riseset(now, 'sun', self.obs, 'previous', 'setting', horizon)\n self.lastAstro = astroset\n\n elif astrorise > astroset and astrorise < sunrise:\n # During night, report upcoming start of morning\n # twilight and sunrise\n self.lriset.configure(text='Rises:',\n font=g.DEFAULT_FONT)\n horizon = -64*u.arcmin\n self.lastRiset = sunrise\n self.lastAstro = astrorise\n\n else:\n # During morning twilight report start of twilight\n # just passed and upcoming sunrise\n self.lriset.configure(text='Rises:',\n font=g.DEFAULT_FONT)\n horizon = -18*u.deg\n self.lastRiset = sunrise\n self.lastAstro = calc_riseset(now, 'sun', self.obs, 'previous', 'rising', horizon)\n\n # Configure the corresponding text fields\n self.riset.configure(\n text=self.lastRiset.datetime.strftime(\"%H:%M:%S\")\n )\n self.astro.configure(\n text=self.lastAstro.datetime.strftime(\"%H:%M:%S\")\n )\n\n except Exception as err:\n # catchall\n g.clog.warn('AstroFrame.update: error = ' + str(err))\n\n # run again after 100 milli-seconds\n self.after(100, self.update)", "response": "Updates the internal state of the object with the current state of the object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the values of the window pairs.", "response": "def check(self):\n \"\"\"\n Checks the values of the window pairs. If any problems are found, it\n flags them by changing the background colour.\n\n Returns (status, synced)\n\n status : flag for whether parameters are viable at all\n synced : flag for whether the windows are synchronised.\n \"\"\"\n\n status = True\n synced = True\n\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n npair = self.npair.value()\n\n g = get_root(self).globals\n # individual pair checks\n for xslw, xsrw, ysw, nxw, nyw in zip(self.xsl[:npair], self.xsr[:npair],\n self.ys[:npair], self.nx[:npair],\n self.ny[:npair]):\n xslw.config(bg=g.COL['main'])\n xsrw.config(bg=g.COL['main'])\n ysw.config(bg=g.COL['main'])\n nxw.config(bg=g.COL['main'])\n nyw.config(bg=g.COL['main'])\n status = status if xslw.ok() else False\n status = status if xsrw.ok() else False\n status = status if ysw.ok() else False\n status = status if nxw.ok() else False\n status = status if nyw.ok() else False\n xsl = xslw.value()\n xsr = xsrw.value()\n ys = ysw.value()\n nx = nxw.value()\n ny = nyw.value()\n\n # Are unbinned dimensions consistent with binning factors?\n if nx is None or nx % xbin != 0:\n nxw.config(bg=g.COL['error'])\n status = False\n elif (nx // xbin) % 4 != 0:\n \"\"\"\n The NGC collects pixel data in chunks before transmission.\n As a result, to avoid loss of data from frames, the binned\n x-size must be a multiple of 4.\n \"\"\"\n nxw.config(bg=g.COL['error'])\n status = False\n\n if ny is None or ny % ybin != 0:\n nyw.config(bg=g.COL['error'])\n status = False\n\n # overlap checks\n if xsl is None or xsr is None or xsl >= xsr:\n xsrw.config(bg=g.COL['error'])\n status = False\n\n if xsl is None or xsr is None or nx is None or xsl + nx > xsr:\n xsrw.config(bg=g.COL['error'])\n status = False\n\n # Are the windows synchronised? This means that they would\n # be consistent with the pixels generated were the whole CCD\n # to be binned by the same factors. If relevant values are not\n # set, we count that as \"synced\" because the purpose of this is\n # to enable / disable the sync button and we don't want it to be\n # enabled just because xs or ys are not set.\n perform_check = all([param is not None for param in (xsl, xsr, ys, nx, ny)])\n if (perform_check and\n ((xsl - 1) % xbin != 0 or (xsr - 1025) % xbin != 0 or\n (ys - 1) % ybin != 0)):\n synced = False\n\n # Range checks\n if xsl is None or nx is None or xsl + nx - 1 > xslw.imax:\n xslw.config(bg=g.COL['error'])\n status = False\n\n if xsr is None or nx is None or xsr + nx - 1 > xsrw.imax:\n xsrw.config(bg=g.COL['error'])\n status = False\n\n if ys is None or ny is None or ys + ny - 1 > ysw.imax:\n ysw.config(bg=g.COL['error'])\n status = False\n\n # Pair overlap checks. Compare one pair with the next one in the\n # same quadrant (if there is one). Only bother if we have survived\n # so far, which saves a lot of checks\n if status:\n for index in range(npair-2):\n ys1 = self.ys[index].value()\n ny1 = self.ny[index].value()\n\n ysw2 = self.ys[index+2]\n ys2 = ysw2.value()\n if ys1 + ny1 > ys2:\n ysw2.config(bg=g.COL['error'])\n status = False\n\n if synced:\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.disable()\n else:\n if not self.frozen:\n self.sbutt.enable()\n self.sbutt.config(bg=g.COL['warn'])\n\n return status"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsynchronise the settings. This means that the pixel start values are shifted downwards so that they are synchronised with a full-frame binned version. This does nothing if the binning factors == 1.", "response": "def sync(self):\n \"\"\"\n Synchronise the settings. This means that the pixel start\n values are shifted downwards so that they are synchronised\n with a full-frame binned version. This does nothing if the\n binning factors == 1.\n \"\"\"\n\n # needs some mods for ultracam ??\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n n = 0\n for xsl, xsr, ys, nx, ny in self:\n if xbin > 1:\n xsl = xbin*((xsl-1)//xbin)+1\n self.xsl[n].set(xsl)\n xsr = xbin*((xsr-1025)//xbin)+1025\n self.xsr[n].set(xsr)\n\n if ybin > 1:\n ys = ybin*((ys-1)//ybin)+1\n self.ys[n].set(ys)\n\n n += 1\n g = get_root(self).globals\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.config(state='disable')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef freeze(self):\n for xsl, xsr, ys, nx, ny in \\\n zip(self.xsl, self.xsr,\n self.ys, self.nx, self.ny):\n xsl.disable()\n xsr.disable()\n ys.disable()\n nx.disable()\n ny.disable()\n self.npair.disable()\n self.xbin.disable()\n self.ybin.disable()\n self.sbutt.disable()\n self.frozen = True", "response": "Freeze all settings so they can t be altered"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef disable(self, everything=False):\n self.freeze()\n if not everything:\n self.xbin.enable()\n self.ybin.enable()\n self.frozen = False", "response": "Disable all but possibly not binning."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nenable WinPair settings for the current instance of this class.", "response": "def enable(self):\n \"\"\"\n Enables WinPair settings\n \"\"\"\n npair = self.npair.value()\n for label, xsl, xsr, ys, nx, ny in \\\n zip(self.label[:npair], self.xsl[:npair], self.xsr[:npair],\n self.ys[:npair], self.nx[:npair], self.ny[:npair]):\n label.config(state='normal')\n xsl.enable()\n xsr.enable()\n ys.enable()\n nx.enable()\n ny.enable()\n\n for label, xsl, xsr, ys, nx, ny in \\\n zip(self.label[npair:], self.xsl[npair:], self.xsr[npair:],\n self.ys[npair:], self.nx[npair:], self.ny[npair:]):\n label.config(state='disable')\n xsl.disable()\n xsr.disable()\n ys.disable()\n nx.disable()\n ny.disable()\n\n self.npair.enable()\n self.xbin.enable()\n self.ybin.enable()\n self.sbutt.enable()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check(self):\n status = synced = True\n\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n nquad = self.nquad.value()\n\n g = get_root(self).globals\n # individual window checks\n for (xsllw, xsulw, xslrw, xsurw, ysw, nxw, nyw) in zip(\n self.xsll[:nquad],\n self.xsul[:nquad], self.xslr[:nquad],\n self.xsur[:nquad], self.ys[:nquad], self.nx[:nquad], self.ny[:nquad]):\n\n all_fields = (xsllw, xsulw, xslrw, xsurw, ysw, nxw, nyw)\n for field in all_fields:\n field.config(bg=g.COL['main'])\n status = status if field.ok() else False\n\n xsll = xsllw.value()\n xsul = xsulw.value()\n xslr = xslrw.value()\n xsur = xsurw.value()\n ys = ysw.value()\n nx = nxw.value()\n ny = nyw.value()\n\n # Are unbinned dimensions consistent with binning factors?\n if nx is None or nx % xbin != 0:\n nxw.config(bg=g.COL['error'])\n status = False\n elif (nx // xbin) % 4 != 0:\n \"\"\"\n The NGC collects pixel data in chunks before transmission.\n As a result, to avoid loss of data from frames, the binned\n x-size must be a multiple of 4.\n \"\"\"\n nxw.config(bg=g.COL['error'])\n status = False\n\n if ny is None or ny % ybin != 0:\n nyw.config(bg=g.COL['error'])\n status = False\n\n # overlap checks in x direction\n if xsll is None or xslr is None or xsll >= xslr:\n xslrw.config(bg=g.COL['error'])\n status = False\n if xsul is None or xsur is None or xsul >= xsur:\n xsurw.config(bg=g.COL['error'])\n status = False\n if nx is None or xsll is None or xsll + nx > xslr:\n xslrw.config(bg=g.COL['error'])\n status = False\n if xsul is None or nx is None or xsul + nx > xsur:\n xsurw.config(bg=g.COL['error'])\n status = False\n\n # Are the windows synchronised? This means that they would\n # be consistent with the pixels generated were the whole CCD\n # to be binned by the same factors. If relevant values are not\n # set, we count that as \"synced\" because the purpose of this is\n # to enable / disable the sync button and we don't want it to be\n # enabled just because xs or ys are not set.\n perform_check = all([param is not None for param in (\n xsll, xslr, ys, nx, ny\n )])\n if (perform_check and ((xsll - 1) % xbin != 0 or (xslr - 1025) % xbin != 0 or\n (ys - 1) % ybin != 0)):\n synced = False\n\n perform_check = all([param is not None for param in (\n xsul, xsur, ys, nx, ny\n )])\n if (perform_check and ((xsul - 1) % xbin != 0 or (xsur - 1025) % xbin != 0 or\n (ys - 1) % ybin != 0)):\n synced = False\n\n # Range checks\n rchecks = ((xsll, nx, xsllw), (xslr, nx, xslrw),\n (xsul, nx, xsulw), (xsur, nx, xsurw),\n (ys, ny, ysw))\n for check in rchecks:\n val, size, widg = check\n if val is None or size is None or val + size - 1 > widg.imax:\n widg.config(bg=g.COL['error'])\n status = False\n\n # Quad overlap checks. Compare one quad with the next one\n # in the same quadrant if there is one. Only bother if we\n # have survived so far, which saves a lot of checks.\n if status:\n for index in range(nquad-1):\n ys1 = self.ys[index].value()\n ny1 = self.ny[index].value()\n ysw2 = self.ys[index+1]\n ys2 = ysw2.value()\n if any([thing is None for thing in (ys1, ny1, ys2)]) or ys1 + ny1 > ys2:\n ysw2.config(bg=g.COL['error'])\n status = False\n\n if synced:\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.disable()\n else:\n if not self.frozen:\n self.sbutt.enable()\n self.sbutt.config(bg=g.COL['warn'])\n return status", "response": "Checks the values of the window quads. If any problems are found it checks the values of the window quads. If any problems are found it checks the values of the offending window by changing the background colour."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsynchronises the settings. This routine changes the window settings so that the pixel start values are shifted downwards until they are synchronised with a full-frame binned version. This does nothing if the binning factor is 1.", "response": "def sync(self):\n \"\"\"\n Synchronise the settings.\n\n This routine changes the window settings so that the pixel start\n values are shifted downwards until they are synchronised with a\n full-frame binned version. This does nothing if the binning factor\n is 1.\n \"\"\"\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n if xbin == 1 and ybin == 1:\n self.sbutt.config(state='disable')\n return\n\n for n, (xsll, xsul, xslr, xsur, ys, nx, ny) in enumerate(self):\n if (xsll-1) % xbin != 0:\n xsll = xbin * ((xsll-1)//xbin)+1\n self.xsll[n].set(xsll)\n if (xsul-1) % xbin != 0:\n xsul = xbin * ((xsul-1)//xbin)+1\n self.xsul[n].set(xsul)\n if (xslr-1025) % xbin != 0:\n xslr = xbin * ((xslr-1025)//xbin)+1025\n self.xslr[n].set(xslr)\n if (xsur-1025) % xbin != 0:\n xsur = xbin * ((xsur-1025)//xbin)+1025\n self.xsur[n].set(xsur)\n\n if ybin > 1 and (ys-1) % ybin != 0:\n ys = ybin*((ys-1)//ybin)+1\n self.ys[n].set(ys)\n\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.config(state='disable')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef freeze(self):\n for fields in zip(self.xsll, self.xsul, self.xslr, self.xsur,\n self.ys, self.nx, self.ny):\n for field in fields:\n field.disable()\n self.nquad.disable()\n self.xbin.disable()\n self.ybin.disable()\n self.sbutt.disable()\n self.frozen = True", "response": "Freeze all settings in this instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check(self):\n\n status = True\n synced = True\n\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n nwin = self.nwin.value()\n\n # individual window checks\n g = get_root(self).globals\n for xsw, ysw, nxw, nyw in \\\n zip(self.xs[:nwin], self.ys[:nwin],\n self.nx[:nwin], self.ny[:nwin]):\n\n xsw.config(bg=g.COL['main'])\n ysw.config(bg=g.COL['main'])\n nxw.config(bg=g.COL['main'])\n nyw.config(bg=g.COL['main'])\n status = status if xsw.ok() else False\n status = status if ysw.ok() else False\n status = status if nxw.ok() else False\n status = status if nyw.ok() else False\n xs = xsw.value()\n ys = ysw.value()\n nx = nxw.value()\n ny = nyw.value()\n\n # Are unbinned dimensions consistent with binning factors?\n if nx is None or nx % xbin != 0:\n nxw.config(bg=g.COL['error'])\n status = False\n elif (nx // xbin) % 4 != 0:\n \"\"\"\n The NGC collects pixel data in chunks before transmission.\n As a result, to avoid loss of data from frames, the binned\n x-size must be a multiple of 4.\n \"\"\"\n nxw.config(bg=g.COL['error'])\n status = False\n\n if ny is None or ny % ybin != 0:\n nyw.config(bg=g.COL['error'])\n status = False\n\n # Are the windows synchronised? This means that they\n # would be consistent with the pixels generated were\n # the whole CCD to be binned by the same factors\n # If relevant values are not set, we count that as\n # \"synced\" because the purpose of this is to enable\n # / disable the sync button and we don't want it to be\n # enabled just because xs or ys are not set.\n if (xs is not None and ys is not None and nx is not None and\n ny is not None):\n if (xs < 1025 and ((xs - 1) % xbin != 0 or (ys - 1) % ybin != 0)\n or ((xs-1025) % xbin != 0 or (ys - 1) % ybin != 0)):\n synced = False\n\n # Range checks\n if xs is None or nx is None or xs + nx - 1 > xsw.imax:\n xsw.config(bg=g.COL['error'])\n status = False\n\n if ys is None or ny is None or ys + ny - 1 > ysw.imax:\n ysw.config(bg=g.COL['error'])\n status = False\n\n # Overlap checks. Compare each window with the next one, requiring\n # no y overlap and that the second is higher than the first\n if status:\n n1 = 0\n for ysw1, nyw1 in zip(self.ys[:nwin-1], self.ny[:nwin-1]):\n\n ys1 = ysw1.value()\n ny1 = nyw1.value()\n\n n1 += 1\n ysw2 = self.ys[n1]\n\n ys2 = ysw2.value()\n\n if ys2 < ys1 + ny1:\n ysw2.config(bg=g.COL['error'])\n status = False\n\n if synced:\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.disable()\n else:\n if not self.frozen:\n self.sbutt.enable()\n self.sbutt.config(bg=g.COL['warn'])\n\n return status", "response": "Checks the values of the active windows and returns status and flag for whether parameters are viable."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsynchronising the settings. This is a no - op if the pixel start is not binned.", "response": "def sync(self, *args):\n \"\"\"\n Synchronise the settings. This means that the pixel start\n values are shifted downwards so that they are synchronised\n with a full-frame binned version. This does nothing if the\n binning factor == 1\n \"\"\"\n xbin = self.xbin.value()\n ybin = self.ybin.value()\n n = 0\n for xs, ys, nx, ny in self:\n if xbin > 1 and xs % xbin != 1:\n if xs < 1025:\n xs = xbin*((xs-1)//xbin)+1\n else:\n xs = xbin*((xs-1025)//xbin)+1025\n self.xs[n].set(xs)\n\n if ybin > 1 and ys % ybin != 1:\n ys = ybin*((ys-1)//ybin)+1\n self.ys[n].set(ys)\n\n n += 1\n self.sbutt.config(bg=g.COL['main'])\n self.sbutt.config(state='disable')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfreezes all settings so they can t be altered", "response": "def freeze(self):\n \"\"\"\n Freeze all settings so they can't be altered\n \"\"\"\n for xs, ys, nx, ny in \\\n zip(self.xs, self.ys, self.nx, self.ny):\n xs.disable()\n ys.disable()\n nx.disable()\n ny.disable()\n self.nwin.disable()\n self.xbin.disable()\n self.ybin.disable()\n self.sbutt.disable()\n self.frozen = True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nenabling all settings for the current instance of the class.", "response": "def enable(self):\n \"\"\"\n Enables all settings\n \"\"\"\n nwin = self.nwin.value()\n for label, xs, ys, nx, ny in \\\n zip(self.label[:nwin], self.xs[:nwin], self.ys[:nwin],\n self.nx[:nwin], self.ny[:nwin]):\n label.config(state='normal')\n xs.enable()\n ys.enable()\n nx.enable()\n ny.enable()\n\n for label, xs, ys, nx, ny in \\\n zip(self.label[nwin:], self.xs[nwin:], self.ys[nwin:],\n self.nx[nwin:], self.ny[nwin:]):\n label.config(state='disable')\n xs.disable()\n ys.disable()\n nx.disable()\n ny.disable()\n\n self.nwin.enable()\n self.xbin.enable()\n self.ybin.enable()\n self.sbutt.enable()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if the download of the given dict was successful.", "response": "def check_download(self, link_item_dict: Dict[str, LinkItem], folder: Path, log: bool = True) -> Tuple[\n Dict[str, LinkItem], Dict[str, LinkItem]]:\n \"\"\"\n Check if the download of the given dict was successful. No proving if the content of the file is correct too.\n\n :param link_item_dict: dict which to check\n :type link_item_dict: Dict[str, ~unidown.plugin.link_item.LinkItem]\n :param folder: folder where the downloads are saved\n :type folder: ~pathlib.Path\n :param log: if the lost items should be logged\n :type log: bool\n :return: succeeded and lost dicts\n :rtype: Tuple[Dict[str, ~unidown.plugin.link_item.LinkItem], Dict[str, ~unidown.plugin.link_item.LinkItem]]\n \"\"\"\n succeed = {link: item for link, item in link_item_dict.items() if folder.joinpath(item.name).is_file()}\n lost = {link: item for link, item in link_item_dict.items() if link not in succeed}\n\n if lost and log:\n for link, item in lost.items():\n self.log.error(f\"Not downloaded: {self.info.host+link} - {item.name}\")\n\n return succeed, lost"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndelete all data related to the plugin.", "response": "def delete_data(self):\n \"\"\"\n Delete everything which is related to the plugin. **Do not use if you do not know what you do!**\n \"\"\"\n self.clean_up()\n tools.delete_dir_rec(self._download_path)\n if self._save_state_file.exists():\n self._save_state_file.unlink()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef download_as_file(self, url: str, folder: Path, name: str, delay: float = 0) -> str:\n while folder.joinpath(name).exists(): # TODO: handle already existing files\n self.log.warning('already exists: ' + name)\n name = name + '_d'\n\n with self._downloader.request('GET', url, preload_content=False, retries=urllib3.util.retry.Retry(3)) as reader:\n if reader.status == 200:\n with folder.joinpath(name).open(mode='wb') as out_file:\n out_file.write(reader.data)\n else:\n raise HTTPError(f\"{url} | {reader.status}\")\n\n if delay > 0:\n time.sleep(delay)\n\n return url", "response": "Download the given url to the given target folder."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndownloads the given LinkItem dict to the given folder.", "response": "def download(self, link_item_dict: Dict[str, LinkItem], folder: Path, desc: str, unit: str, delay: float = 0) -> \\\n List[str]:\n \"\"\"\n .. warning::\n\n The parameters may change in future versions. (e.g. change order and accept another host)\n\n Download the given LinkItem dict from the plugins host, to the given path. Proceeded with multiple connections\n :attr:`~unidown.plugin.a_plugin.APlugin._simul_downloads`. After\n :func:`~unidown.plugin.a_plugin.APlugin.check_download` is recommend.\n\n This function don't use an internal `link_item_dict`, `delay` or `folder` directly set in options or instance\n vars, because it can be used aside of the normal download routine inside the plugin itself for own things.\n As of this it still needs access to the logger, so a staticmethod is not possible.\n\n :param link_item_dict: data which gets downloaded\n :type link_item_dict: Dict[str, ~unidown.plugin.link_item.LinkItem]\n :param folder: target download folder\n :type folder: ~pathlib.Path\n :param desc: description of the progressbar\n :type desc: str\n :param unit: unit of the download, shown in the progressbar\n :type unit: str\n :param delay: delay between the downloads in seconds\n :type delay: float\n :return: list of urls of downloads without errors\n :rtype: List[str]\n \"\"\"\n if 'delay' in self._options:\n delay = self._options['delay']\n # TODO: add other optional host?\n if not link_item_dict:\n return []\n\n job_list = []\n with ThreadPoolExecutor(max_workers=self._simul_downloads) as executor:\n for link, item in link_item_dict.items():\n job = executor.submit(self.download_as_file, link, folder, item.name, delay)\n job_list.append(job)\n\n pbar = tqdm(as_completed(job_list), total=len(job_list), desc=desc, unit=unit, leave=True, mininterval=1,\n ncols=100, disable=dynamic_data.DISABLE_TQDM)\n for _ in pbar:\n pass\n\n download_without_errors = []\n for job in job_list:\n try:\n download_without_errors.append(job.result())\n except HTTPError as ex:\n self.log.warning(\"Failed to download: \" + str(ex))\n # Todo: connection lost handling (check if the connection to the server itself is lost)\n\n return download_without_errors"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _create_save_state(self, link_item_dict: Dict[str, LinkItem]) -> SaveState:\n return SaveState(dynamic_data.SAVE_STATE_VERSION, self.info, self.last_update, link_item_dict)", "response": "Create protobuf savestate of the module and the given data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsave meta data about the downloaded things and the plugin to file.", "response": "def save_save_state(self, data_dict: Dict[str, LinkItem]): # TODO: add progressbar\n \"\"\"\n Save meta data about the downloaded things and the plugin to file.\n\n :param data_dict: data\n :type data_dict: Dict[link, ~unidown.plugin.link_item.LinkItem]\n \"\"\"\n json_data = json_format.MessageToJson(self._create_save_state(data_dict).to_protobuf())\n with self._save_state_file.open(mode='w', encoding=\"utf8\") as writer:\n writer.write(json_data)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_save_state(self) -> SaveState:\n if not self._save_state_file.exists():\n self.log.info(\"No savestate file found.\")\n return SaveState(dynamic_data.SAVE_STATE_VERSION, self.info, datetime(1970, 1, 1), {})\n\n savestat_proto = \"\"\n with self._save_state_file.open(mode='r', encoding=\"utf8\") as data_file:\n try:\n savestat_proto = json_format.Parse(data_file.read(), SaveStateProto(), ignore_unknown_fields=False)\n except ParseError:\n raise PluginException(\n f\"Broken savestate json. Please fix or delete (you may lose data in this case) the file: {self._save_state_file}\")\n\n try:\n save_state = SaveState.from_protobuf(savestat_proto)\n except ValueError as ex:\n raise PluginException(f\"Could not parse the protobuf {self._save_state_file}: {ex}\")\n else:\n del savestat_proto\n\n if save_state.version != dynamic_data.SAVE_STATE_VERSION:\n raise PluginException(\"Different save state version handling is not implemented yet.\")\n\n if save_state.plugin_info.version != self.info.version:\n raise PluginException(\"Different plugin version handling is not implemented yet.\")\n\n if save_state.plugin_info.name != self.name:\n raise PluginException(\"Save state plugin ({name}) does not match the current ({cur_name}).\".format(\n name=save_state.plugin_info.name, cur_name=self.name))\n return save_state", "response": "Loads the savestate of the plugin."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget links who need to be downloaded by comparing old and new data.", "response": "def get_updated_data(self, old_data: Dict[str, LinkItem]) -> Dict[str, LinkItem]:\n \"\"\"\n Get links who needs to be downloaded by comparing old and the new data.\n\n :param old_data: old data\n :type old_data: Dict[str, ~unidown.plugin.link_item.LinkItem]\n :return: data which is newer or dont exist in the old one\n :rtype: Dict[str, ~unidown.plugin.link_item.LinkItem]\n \"\"\"\n if not self.download_data:\n return {}\n new_link_item_dict = {}\n for link, link_item in tqdm(self.download_data.items(), desc=\"Compare with save\", unit=\"item\", leave=True,\n mininterval=1, ncols=100, disable=dynamic_data.DISABLE_TQDM):\n # TODO: add methode to log lost items, which are in old but not in new\n # if link in new_link_item_dict: # TODO: is ever false, since its the key of a dict: move to the right place\n # self.log.warning(\"Duplicate: \" + link + \" - \" + new_link_item_dict[link] + \" : \" + link_item)\n\n # if the new_data link does not exists in old_data or new_data time is newer\n if (link not in old_data) or (link_item.time > old_data[link].time):\n new_link_item_dict[link] = link_item\n\n return new_link_item_dict"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_dict(self, base: Dict[str, LinkItem], new: Dict[str, LinkItem]):\n if logging.INFO >= logging.getLevelName(dynamic_data.LOG_LEVEL): # TODO: logging here or outside\n for link, item in new.items():\n if link in base:\n self.log.info('Actualize item: ' + link + ' | ' + str(base[link]) + ' -> ' + str(item))\n base.update(new)", "response": "Update the base dict with the new dict."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_options_dic(self, options: List[str]) -> Dict[str, str]:\n options_dic = {}\n for option in options:\n cur_option = option.split(\"=\")\n if len(cur_option) != 2:\n self.log.warning(f\"'{option}' is not valid and will be ignored.\")\n options_dic[cur_option[0]] = cur_option[1]\n return options_dic", "response": "Convert the option list to a dictionary where the key is the option and the value is the related option."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget all available plugins for unidown.", "response": "def get_plugins() -> Dict[str, pkg_resources.EntryPoint]:\n \"\"\"\n Get all available plugins for unidown.\n\n :return: plugin name list\n :rtype: Dict[str, ~pkg_resources.EntryPoint]\n \"\"\"\n return {entry.name: entry for entry in pkg_resources.iter_entry_points('unidown.plugin')}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _equation_of_time(t):\n\n # Julian centuries since J2000.0\n T = (t - Time(\"J2000\")).to(u.year).value / 100\n\n # obliquity of ecliptic (Meeus 1998, eq 22.2)\n poly_pars = (84381.448, 46.8150, 0.00059, 0.001813)\n eps = u.Quantity(polyval(T, poly_pars), u.arcsec)\n y = np.tan(eps/2)**2\n\n # Sun's mean longitude (Meeus 1998, eq 25.2)\n poly_pars = (280.46646, 36000.76983, 0.0003032)\n L0 = u.Quantity(polyval(T, poly_pars), u.deg)\n\n # Sun's mean anomaly (Meeus 1998, eq 25.3)\n poly_pars = (357.52911, 35999.05029, 0.0001537)\n M = u.Quantity(polyval(T, poly_pars), u.deg)\n\n # eccentricity of Earth's orbit (Meeus 1998, eq 25.4)\n poly_pars = (0.016708634, -0.000042037, -0.0000001267)\n e = polyval(T, poly_pars)\n\n # equation of time, radians (Meeus 1998, eq 28.3)\n eot = (y * np.sin(2*L0) - 2*e*np.sin(M) + 4*e*y*np.sin(M)*np.cos(2*L0) -\n 0.5*y**2 * np.sin(4*L0) - 5*e**2 * np.sin(2*M)/4) * u.rad\n return eot.to(u.hourangle)", "response": "Find the difference between apparent and mean solar time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a Local Sidereal Time to an astropy. time. Time object.", "response": "def _astropy_time_from_LST(t, LST, location, prev_next):\n \"\"\"\n Convert a Local Sidereal Time to an astropy Time object.\n\n The local time is related to the LST through the RA of the Sun.\n This routine uses this relationship to convert a LST to an astropy\n time object.\n\n Returns\n -------\n ret1 : `~astropy.time.Time`\n time corresponding to LST\n \"\"\"\n # now we need to figure out time to return from LST\n raSun = coord.get_sun(t).ra\n\n # calculate Greenwich Apparent Solar Time, which we will use as ~UTC for now\n with warnings.catch_warnings():\n warnings.simplefilter('ignore')\n # ignore astropy deprecation warnings\n lon = location.longitude\n solarTime = LST - raSun + 12*u.hourangle - lon\n\n # assume this is on the same day as supplied time, and fix later\n first_guess = Time(\n u.d*int(t.mjd) + u.hour*solarTime.wrap_at('360d').hour,\n format='mjd'\n )\n\n # Equation of time is difference between GAST and UTC\n eot = _equation_of_time(first_guess)\n first_guess = first_guess - u.hour * eot.value\n\n if prev_next == 'next':\n # if 'next', we want time to be greater than given time\n mask = first_guess < t\n rise_set_time = first_guess + mask * u.sday\n else:\n # if 'previous', we want time to be less than given time\n mask = first_guess > t\n rise_set_time = first_guess - mask * u.sday\n return rise_set_time"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate next rise set of target.", "response": "def calc_riseset(t, target_name, location, prev_next, rise_set, horizon):\n \"\"\"\n Time at next rise/set of ``target``.\n\n Parameters\n ----------\n t : `~astropy.time.Time` or other (see below)\n Time of observation. This will be passed in as the first argument to\n the `~astropy.time.Time` initializer, so it can be anything that\n `~astropy.time.Time` will accept (including a `~astropy.time.Time`\n object)\n\n target_name : str\n 'moon' or 'sun'\n\n location : `~astropy.coordinates.EarthLocation`\n Observatory location\n\n prev_next : str - either 'previous' or 'next'\n Test next rise/set or previous rise/set\n\n rise_set : str - either 'rising' or 'setting'\n Compute prev/next rise or prev/next set\n\n location : `~astropy.coordinates.EarthLocation`\n Location of observer\n\n horizon : `~astropy.units.Quantity`\n Degrees above/below actual horizon to use\n for calculating rise/set times (i.e.,\n -6 deg horizon = civil twilight, etc.)\n\n Returns\n -------\n ret1 : `~astropy.time.Time`\n Time of rise/set\n \"\"\"\n target = coord.get_body(target_name, t)\n t0 = _rise_set_trig(t, target, location, prev_next, rise_set)\n grid = t0 + np.linspace(-4*u.hour, 4*u.hour, 10)\n altaz_frame = coord.AltAz(obstime=grid, location=location)\n target = coord.get_body(target_name, grid)\n altaz = target.transform_to(altaz_frame)\n time_limits, altitude_limits = _horiz_cross(altaz.obstime, altaz.alt,\n rise_set, horizon)\n return _two_point_interp(time_limits, altitude_limits, horizon)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _horiz_cross(t, alt, rise_set, horizon=0*u.degree):\n if rise_set == 'rising':\n # Find index where altitude goes from below to above horizon\n condition = (alt[:-1] < horizon) * (alt[1:] > horizon)\n elif rise_set == 'setting':\n # Find index where altitude goes from above to below horizon\n condition = (alt[:-1] > horizon) * (alt[1:] < horizon)\n\n if np.count_nonzero(condition) == 0:\n warnmsg = ('Target does not cross horizon={} within '\n '8 hours of trigonometric estimate'.format(horizon))\n warnings.warn(warnmsg)\n\n # Fill in missing time with MAGIC_TIME\n time_inds = np.nan\n times = [np.nan, np.nan]\n altitudes = [np.nan, np.nan]\n else:\n time_inds = np.nonzero(condition)[0][0]\n times = t[time_inds:time_inds+2]\n altitudes = alt[time_inds:time_inds+2]\n\n return times, altitudes", "response": "Calculate the time and altitude crossing of a base time and altitudes."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _two_point_interp(times, altitudes, horizon=0*u.deg):\n if not isinstance(times, Time):\n return MAGIC_TIME\n else:\n slope = (altitudes[1] - altitudes[0])/(times[1].jd - times[0].jd)\n return Time(times[1].jd - ((altitudes[1] - horizon)/slope).value,\n format='jd')", "response": "Do linear interpolation between two times and altitudes at\n two times and horizon."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init_dirs(main_dir: Path, logfilepath: Path):\n global MAIN_DIR, TEMP_DIR, DOWNLOAD_DIR, SAVESTAT_DIR, LOGFILE_PATH\n MAIN_DIR = main_dir\n TEMP_DIR = MAIN_DIR.joinpath(Path('temp/'))\n DOWNLOAD_DIR = MAIN_DIR.joinpath(Path('downloads/'))\n SAVESTAT_DIR = MAIN_DIR.joinpath(Path('savestates/'))\n LOGFILE_PATH = MAIN_DIR.joinpath(logfilepath)", "response": "Initialize the main directories."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nresetting all dynamic variables to the default values.", "response": "def reset():\n \"\"\"\n Reset all dynamic variables to the default values.\n \"\"\"\n global MAIN_DIR, TEMP_DIR, DOWNLOAD_DIR, SAVESTAT_DIR, LOGFILE_PATH, USING_CORES, LOG_LEVEL, DISABLE_TQDM, \\\n SAVE_STATE_VERSION\n MAIN_DIR = Path('./')\n TEMP_DIR = MAIN_DIR.joinpath(Path('temp/'))\n DOWNLOAD_DIR = MAIN_DIR.joinpath(Path('downloads/'))\n SAVESTAT_DIR = MAIN_DIR.joinpath(Path('savestates/'))\n LOGFILE_PATH = MAIN_DIR.joinpath(Path('UniDown.log'))\n\n USING_CORES = 1\n LOG_LEVEL = 'INFO'\n DISABLE_TQDM = False\n\n SAVE_STATE_VERSION = Version('1')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_dirs():\n dirs = [MAIN_DIR, TEMP_DIR, DOWNLOAD_DIR, SAVESTAT_DIR]\n for directory in dirs:\n if directory.exists() and not directory.is_dir():\n raise FileExistsError(str(directory.resolve()) + \" cannot be used as a directory.\")", "response": "Checks the directories if they exist."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_hstring(hs):\n # split the string on = and /, also stripping whitespace and annoying quotes\n name, value, comment = yield_three(\n [val.strip().strip(\"'\") for val in filter(None, re.split(\"[=/]+\", hs))]\n )\n\n # if comment has a slash in it, put it back together\n try:\n len(comment)\n except:\n pass\n else:\n comment = '/'.join(comment)\n return name, value, comment", "response": "Parse a single item from the telescope server into name value comment."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_header_from_telpars(telpars):\n # pars is a list of strings describing tel info in FITS\n # style, each entry in the list is a different class of\n # thing (weather, telescope, instrument etc).\n\n # first, we munge it into a single list of strings, each one\n # describing a single item whilst also stripping whitespace\n pars = [val.strip() for val in (';').join(telpars).split(';')\n if val.strip() != '']\n\n # apply parse_hstring to everything in pars\n with warnings.catch_warnings():\n warnings.simplefilter('ignore', fits.verify.VerifyWarning)\n hdr = fits.Header(map(parse_hstring, pars))\n\n return hdr", "response": "Create a list of fits header items from GTC telescope pars."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding a row with current values to GTC table with current values", "response": "def add_gtc_header_table_row(t, telpars):\n \"\"\"\n Add a row with current values to GTC table\n\n Arguments\n ---------\n t : `~astropy.table.Table`\n The table to append row to\n telpars : list\n list returned by server call to getTelescopeParams\n \"\"\"\n now = Time.now().mjd\n hdr = create_header_from_telpars(telpars)\n\n # make dictionary of vals to put in table\n vals = {k: v for k, v in hdr.items() if k in VARIABLE_GTC_KEYS}\n vals['MJD'] = now\n # store LST as hourangle\n vals['LST'] = Longitude(vals['LST'], unit=u.hour).hourangle\n t.add_row(vals)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef from_protobuf(cls, proto: PluginInfoProto) -> PluginInfo:\n if proto.name == \"\":\n raise ValueError(\"name of PluginInfo does not exist or is empty inside the protobuf.\")\n elif proto.version == \"\":\n raise ValueError(\"version of PluginInfo does not exist or is empty inside the protobuf.\")\n elif proto.host == \"\":\n raise ValueError(\"host of PluginInfo does not exist or is empty inside the protobuf.\")\n return cls(proto.name, proto.version, proto.host)", "response": "Constructor from protobuf.\n\n :param proto: protobuf structure\n :type proto: ~unidown.plugin.protobuf.plugin_info_pb2.PluginInfoProto\n :return: the PluginInfo\n :rtype: ~unidown.plugin.plugin_info.PluginInfo\n :raises ValueError: name of PluginInfo does not exist or is empty inside the protobuf\n :raises ValueError: version of PluginInfo does not exist or is empty inside the protobuf\n :raises ValueError: host of PluginInfo does not exist or is empty inside the protobuf"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating protobuf item. :return: protobuf structure :rtype: ~unidown.plugin.protobuf.link_item_pb2.PluginInfoProto", "response": "def to_protobuf(self) -> PluginInfoProto:\n \"\"\"\n Create protobuf item.\n\n :return: protobuf structure\n :rtype: ~unidown.plugin.protobuf.link_item_pb2.PluginInfoProto\n \"\"\"\n proto = PluginInfoProto()\n proto.name = self.name\n proto.version = str(self.version)\n proto.host = self.host\n return proto"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_protobuf(cls, proto: SaveStateProto) -> SaveState:\n data_dict = {}\n for key, link_item in proto.data.items():\n data_dict[key] = LinkItem.from_protobuf(link_item)\n if proto.version == \"\":\n raise ValueError(\"version of SaveState does not exist or is empty inside the protobuf.\")\n try:\n version = Version(proto.version)\n except InvalidVersion:\n raise InvalidVersion(f\"Plugin version is not PEP440 conform: {proto.version}\")\n return cls(version, PluginInfo.from_protobuf(proto.plugin_info), Timestamp.ToDatetime(proto.last_update),\n data_dict)", "response": "Constructs a SaveState from a SaveState protobuf."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_protobuf(self) -> SaveStateProto:\n result = SaveStateProto()\n result.version = str(self.version)\n result.last_update.CopyFrom(datetime_to_timestamp(self.last_update))\n result.plugin_info.CopyFrom(self.plugin_info.to_protobuf())\n for key, link_item in self.link_item_dict.items():\n result.data[key].CopyFrom(link_item.to_protobuf())\n return result", "response": "Create protobuf item.\n\n :return: protobuf structure\n :rtype: ~unidown.plugin.protobuf.save_state_pb2.SaveStateProto"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the definite article for a given word.", "response": "def definite_article(word, gender=MALE, role=SUBJECT):\n \"\"\" Returns the definite article (der/die/das/die) for a given word.\n \"\"\"\n return article_definite.get((gender[:1].lower(), role[:3].lower()))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef indefinite_article(word, gender=MALE, role=SUBJECT):\n return article_indefinite.get((gender[:1].lower(), role[:3].lower()))", "response": "Returns the indefinite article for a given word."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the indefinite or definite article for the given word.", "response": "def article(word, function=INDEFINITE, gender=MALE, role=SUBJECT):\n \"\"\" Returns the indefinite (ein) or definite (der/die/das/die) article for the given word.\n \"\"\"\n return function == DEFINITE \\\n and definite_article(word, gender, role) \\\n or indefinite_article(word, gender, role)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a string with the article + the word.", "response": "def referenced(word, article=INDEFINITE, gender=MALE, role=SUBJECT):\n \"\"\" Returns a string with the article + the word.\n \"\"\"\n return \"%s %s\" % (_article(word, article, gender, role), word)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef gender(word, pos=NOUN):\n w = word.lower()\n if pos == NOUN:\n # Default rules (baseline = 32%).\n if w.endswith(gender_masculine):\n return MASCULINE\n if w.endswith(gender_feminine):\n return FEMININE\n if w.endswith(gender_neuter):\n return NEUTER\n # Majority vote.\n for g in gender_majority_vote:\n if w.endswith(gender_majority_vote[g]):\n return g", "response": "Returns the gender of the given word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the plural of a given word.", "response": "def pluralize(word, pos=NOUN, gender=MALE, role=SUBJECT, custom={}):\n \"\"\" Returns the plural of a given word.\n The inflection is based on probability rather than gender and role.\n \"\"\"\n w = word.lower().capitalize()\n if word in custom:\n return custom[word]\n if pos == NOUN:\n for a, b in plural_inflections:\n if w.endswith(a):\n return w[:-len(a)] + b\n # Default rules (baseline = 69%).\n if w.startswith(\"ge\"):\n return w\n if w.endswith(\"gie\"):\n return w\n if w.endswith(\"e\"):\n return w + \"n\"\n if w.endswith(\"ien\"):\n return w[:-2] + \"um\"\n if w.endswith((\"au\", \"ein\", \"eit\", \"er\", \"en\", \"el\", \"chen\", \"mus\", u\"t\u00e4t\", \"tik\", \"tum\", \"u\")):\n return w\n if w.endswith((\"ant\", \"ei\", \"enz\", \"ion\", \"ist\", \"or\", \"schaft\", \"tur\", \"ung\")):\n return w + \"en\"\n if w.endswith(\"in\"):\n return w + \"nen\"\n if w.endswith(\"nis\"):\n return w + \"se\"\n if w.endswith((\"eld\", \"ild\", \"ind\")):\n return w + \"er\"\n if w.endswith(\"o\"):\n return w + \"s\"\n if w.endswith(\"a\"):\n return w[:-1] + \"en\"\n # Inflect common umlaut vowels: Kopf => K\u00f6pfe.\n if w.endswith((\"all\", \"and\", \"ang\", \"ank\", \"atz\", \"auf\", \"ock\", \"opf\", \"uch\", \"uss\")):\n umlaut = w[-3]\n umlaut = umlaut.replace(\"a\", u\"\u00e4\")\n umlaut = umlaut.replace(\"o\", u\"\u00f6\")\n umlaut = umlaut.replace(\"u\", u\"\u00fc\")\n return w[:-3] + umlaut + w[-2:] + \"e\"\n for a, b in (\n (\"ag\", u\"\u00e4ge\"), \n (\"ann\", u\"\u00e4nner\"), \n (\"aum\", u\"\u00e4ume\"), \n (\"aus\", u\"\u00e4user\"), \n (\"zug\", u\"z\u00fcge\")):\n if w.endswith(a):\n return w[:-len(a)] + b\n return w + \"e\"\n return w"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef singularize(word, pos=NOUN, gender=MALE, role=SUBJECT, custom={}):\n w = word.lower().capitalize()\n if word in custom:\n return custom[word]\n if word in singular:\n return singular[word]\n if pos == NOUN:\n for a, b in singular_inflections:\n if w.endswith(a):\n return w[:-len(a)] + b\n # Default rule: strip known plural suffixes (baseline = 51%).\n for suffix in (\"nen\", \"en\", \"n\", \"e\", \"er\", \"s\"):\n if w.endswith(suffix):\n w = w[:-len(suffix)]\n break\n # Corrections (these add about 1% accuracy):\n if w.endswith((\"rr\", \"rv\", \"nz\")):\n return w + \"e\"\n return w\n return w", "response": "Returns the singular of a given word."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef attributive(adjective, gender=MALE, role=SUBJECT, article=None):\n w, g, c, a = \\\n adjective.lower(), gender[:1].lower(), role[:3].lower(), article and article.lower() or None\n if w in adjective_attributive:\n return adjective_attributive[w]\n if a is None \\\n or a in (\"mir\", \"dir\", \"ihm\") \\\n or a in (\"ein\", \"etwas\", \"mehr\") \\\n or a.startswith((\"all\", \"mehrer\", \"wenig\", \"viel\")):\n return w + adjectives_strong.get((g, c), \"\")\n if a.startswith((\"ein\", \"kein\")) \\\n or a.startswith((\"mein\", \"dein\", \"sein\", \"ihr\", \"Ihr\", \"unser\", \"euer\")):\n return w + adjectives_mixed.get((g, c), \"\")\n if a in (\"arm\", \"alt\", \"all\", \"der\", \"die\", \"das\", \"den\", \"dem\", \"des\") \\\n or a.startswith((\n \"derselb\", \"derjenig\", \"jed\", \"jeglich\", \"jen\", \"manch\", \n \"dies\", \"solch\", \"welch\")):\n return w + adjectives_weak.get((g, c), \"\")\n # Default to strong inflection.\n return w + adjectives_strong.get((g, c), \"\")", "response": "Returns the attributive for the given adjective."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef predicative(adjective):\n w = adjective.lower()\n if len(w) > 3:\n for suffix in (\"em\", \"en\", \"er\", \"es\", \"e\"):\n if w.endswith(suffix):\n b = w[:max(-len(suffix), -(len(w)-3))]\n if b.endswith(\"bl\"): # plausibles => plausibel\n b = b[:-1] + \"el\"\n if b.endswith(\"pr\"): # propres => proper\n b = b[:-1] + \"er\"\n return b\n return w", "response": "Returns the predicative adjective."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the comparative or superlative form of the given adjective.", "response": "def grade(adjective, suffix=COMPARATIVE):\n \"\"\" Returns the comparative or superlative form of the given (inflected) adjective.\n \"\"\"\n b = predicative(adjective)\n # gro\u00df => gro\u00dft, sch\u00f6n => sch\u00f6nst\n if suffix == SUPERLATIVE and b.endswith((\"s\", u\"\u00df\")):\n suffix = suffix[1:]\n # gro\u00dfe => gro\u00dfere, sch\u00f6nes => sch\u00f6neres\n return adjective[:len(b)] + suffix + adjective[len(b):]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfinds the base form of the given verb.", "response": "def find_lemma(self, verb):\n \"\"\" Returns the base form of the given inflected verb, using a rule-based approach.\n \"\"\"\n v = verb.lower()\n # Common prefixes: be-finden and emp-finden probably inflect like finden.\n if not (v.startswith(\"ge\") and v.endswith(\"t\")): # Probably gerund.\n for prefix in prefixes:\n if v.startswith(prefix) and v[len(prefix):] in self.inflections:\n return prefix + self.inflections[v[len(prefix):]]\n # Common sufixes: setze nieder => niedersetzen.\n b, suffix = \" \" in v and v.split()[:2] or (v, \"\")\n # Infinitive -ln: trommeln.\n if b.endswith((\"ln\", \"rn\")):\n return b\n # Lemmatize regular inflections.\n for x in (\"test\", \"est\", \"end\", \"ten\", \"tet\", \"en\", \"et\", \"te\", \"st\", \"e\", \"t\"):\n if b.endswith(x): b = b[:-len(x)]; break\n # Subjunctive: hielte => halten, schnitte => schneiden.\n for x, y in (\n (\"ieb\", \"eib\"), ( \"ied\", \"eid\"), ( \"ief\", \"auf\" ), ( \"ieg\", \"eig\" ), (\"iel\", \"alt\"), \n (\"ien\", \"ein\"), (\"iess\", \"ass\"), (u\"ie\u00df\", u\"a\u00df\" ), ( \"iff\", \"eif\" ), (\"iss\", \"eiss\"), \n (u\"i\u00df\", u\"ei\u00df\"), ( \"it\", \"eid\"), ( \"oss\", \"iess\"), (u\"\u00f6ss\", \"iess\")):\n if b.endswith(x): b = b[:-len(x)] + y; break\n b = b.replace(\"eeiss\", \"eiss\")\n b = b.replace(\"eeid\", \"eit\")\n # Subjunctive: wechselte => wechseln\n if not b.endswith((\"e\", \"l\")) and not (b.endswith(\"er\") and len(b) >= 3 and not b[-3] in VOWELS):\n b = b + \"e\"\n # abknallst != abknalln => abknallen\n if b.endswith((\"hl\", \"ll\", \"ul\", \"eil\")):\n b = b + \"e\"\n # Strip ge- from (likely) gerund:\n if b.startswith(\"ge\") and v.endswith(\"t\"):\n b = b[2:]\n # Corrections (these add about 1.5% accuracy):\n if b.endswith((\"lnde\", \"rnde\")):\n b = b[:-3]\n if b.endswith((\"ae\", \"al\", u\"\u00f6e\", u\"\u00fce\")):\n b = b.rstrip(\"e\") + \"te\"\n if b.endswith(u\"\u00e4l\"):\n b = b + \"e\"\n return suffix + b + \"n\""} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the lexeme of a verb.", "response": "def find_lexeme(self, verb):\n \"\"\" For a regular verb (base form), returns the forms using a rule-based approach.\n \"\"\"\n v = verb.lower()\n # Stem = infinitive minus -en, -ln, -rn.\n b = b0 = re.sub(\"en$\", \"\", re.sub(\"ln$\", \"l\", re.sub(\"rn$\", \"r\", v)))\n # Split common prefixes.\n x, x1, x2 = \"\", \"\", \"\"\n for prefix in prefix_separable:\n if v.startswith(prefix):\n b, x = b[len(prefix):], prefix\n x1 = (\" \" + x).rstrip()\n x2 = x + \"ge\"\n break\n # Present tense 1sg and subjunctive -el: handeln => ich handle, du handlest.\n pl = b.endswith(\"el\") and b[:-2]+\"l\" or b\n # Present tense 1pl -el: handeln => wir handeln\n pw = v.endswith((\"ln\", \"rn\")) and v or b+\"en\"\n # Present tense ending in -d or -t gets -e:\n pr = b.endswith((\"d\", \"t\")) and b+\"e\" or b\n # Present tense 2sg gets -st, unless stem ends with -s or -z.\n p2 = pr.endswith((\"s\",\"z\")) and pr+\"t\" or pr+\"st\"\n # Present participle: spiel + -end, arbeiten + -d:\n pp = v.endswith((\"en\", \"ln\", \"rn\")) and v+\"d\" or v+\"end\"\n # Past tense regular:\n pt = encode_sz(pr) + \"t\"\n # Past participle: haushalten => hausgehalten\n ge = (v.startswith(prefix_inseparable) or b.endswith((\"r\",\"t\"))) and pt or \"ge\"+pt\n ge = x and x+\"ge\"+pt or ge\n # Present subjunctive: stem + -e, -est, -en, -et:\n s1 = encode_sz(pl)\n # Past subjunctive: past (usually with Umlaut) + -e, -est, -en, -et:\n s2 = encode_sz(pt)\n # Construct the lexeme:\n lexeme = a = [\n v, \n pl+\"e\"+x1, p2+x1, pr+\"t\"+x1, pw+x1, pr+\"t\"+x1, pp, # present\n pt+\"e\"+x1, pt+\"est\"+x1, pt+\"e\"+x1, pt+\"en\"+x1, pt+\"et\"+x1, ge, # past\n b+\"e\"+x1, pr+\"t\"+x1, x+pw, # imperative\n s1+\"e\"+x1, s1+\"est\"+x1, s1+\"en\"+x1, s1+\"et\"+x1, # subjunctive I\n s2+\"e\"+x1, s2+\"est\"+x1, s2+\"en\"+x1, s2+\"et\"+x1 # subjunctive II\n ]\n # Encode Eszett (\u00df) and attempt to retrieve from the lexicon.\n # Decode Eszett for present and imperative.\n if encode_sz(v) in self:\n a = self[encode_sz(v)]\n a = [decode_sz(v) for v in a[:7]] + a[7:13] + [decode_sz(v) for v in a[13:20]] + a[20:]\n # Since the lexicon does not contain imperative for all verbs, don't simply return it.\n # Instead, update the rule-based lexeme with inflections from the lexicon.\n return [a[i] or lexeme[i] for i in range(len(a))]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tenses(self, verb, parse=True):\n tenses = _Verbs.tenses(self, verb, parse)\n if len(tenses) == 0:\n # auswirkte => wirkte aus\n for prefix in prefix_separable:\n if verb.startswith(prefix):\n tenses = _Verbs.tenses(self, verb[len(prefix):] + \" \" + prefix, parse)\n break\n return tenses", "response": "Returns a list of possible tenses for the given verb."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_words_from_dataset(dataset):\n # Words may be either a string or a list of tokens. Return an iterator\n # of tokens accordingly\n def tokenize(words):\n if isinstance(words, basestring):\n return word_tokenize(words, include_punc=False)\n else:\n return words\n all_words = chain.from_iterable(tokenize(words) for words, _ in dataset)\n return set(all_words)", "response": "Return a set of all words in a dataset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef basic_extractor(document, train_set):\n word_features = _get_words_from_dataset(train_set)\n tokens = _get_document_tokens(document)\n features = dict(((u'contains({0})'.format(word), (word in tokens))\n for word in word_features))\n return features", "response": "A basic document feature extractor that returns a dict indicating what\n words in train_set are contained in document."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _read_data(self, dataset, format=None):\n # Attempt to detect file format if \"format\" isn't specified\n if not format:\n format_class = formats.detect(dataset)\n else:\n if format not in formats.AVAILABLE.keys():\n raise ValueError(\"'{0}' format not supported.\".format(format))\n format_class = formats.AVAILABLE[format]\n return format_class(dataset).to_iterable()", "response": "Reads a data file and returns and iterable that can be used as\n testing or training data."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef extract_features(self, text):\n # Feature extractor may take one or two arguments\n try:\n return self.feature_extractor(text, self.train_set)\n except (TypeError, AttributeError):\n return self.feature_extractor(text)", "response": "Extracts features from a body of text."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef train(self, *args, **kwargs):\n try:\n self.classifier = self.nltk_class.train(self.train_features,\n *args, **kwargs)\n return self.classifier\n except AttributeError:\n raise ValueError(\"NLTKClassifier must have a nltk_class\"\n \" variable that is not None.\")", "response": "Train the classifier with a labeled feature set and return the classifier. Takes the same arguments as the wrapped NLTK class and returns the classifier."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nclassify the text. :param str text: A string of text.", "response": "def classify(self, text):\n \"\"\"Classifies the text.\n\n :param str text: A string of text.\n\n \"\"\"\n text_features = self.extract_features(text)\n return self.classifier.classify(text_features)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the accuracy on a test set.", "response": "def accuracy(self, test_set, format=None):\n \"\"\"Compute the accuracy on a test set.\n\n :param test_set: A list of tuples of the form ``(text, label)``, or a\n filename.\n :param format: If ``test_set`` is a filename, the file format, e.g.\n ``\"csv\"`` or ``\"json\"``. If ``None``, will attempt to detect the\n file format.\n\n \"\"\"\n if isinstance(test_set, basestring): # test_set is a filename\n test_data = self._read_data(test_set)\n else: # test_set is a list of tuples\n test_data = test_set\n test_features = [(self.extract_features(d), c) for d, c in test_data]\n return nltk.classify.accuracy(self.classifier, test_features)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update(self, new_data, *args, **kwargs):\n '''Update the classifier with new training data and re-trains the\n classifier.\n\n :param new_data: New data as a list of tuples of the form\n ``(text, label)``.\n '''\n self.train_set += new_data\n self.train_features = [(self.extract_features(d), c)\n for d, c in self.train_set]\n try:\n self.classifier = self.nltk_class.train(self.train_features,\n *args, **kwargs)\n except AttributeError: # Descendant has not defined nltk_class\n raise ValueError(\"NLTKClassifier must have a nltk_class\"\n \" variable that is not None.\")\n return True", "response": "Update the classifier with new training data and re - traains the nltk_class."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef prob_classify(self, text):\n text_features = self.extract_features(text)\n return self.classifier.prob_classify(text_features)", "response": "Return the label probability distribution for classifying a string\n of text."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef train(self, *args, **kwargs):\n self.classifier = self.nltk_class.train(self.positive_features,\n self.unlabeled_features,\n self.positive_prob_prior)\n return self.classifier", "response": "Train the classifier with a labeled and unlabeled feature sets and\n return the classifier. Takes the same arguments as the wrapped NLTK class and kwargs are passed to the wrapped NLTK class."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update(self, new_positive_data=None,\n new_unlabeled_data=None, positive_prob_prior=0.5,\n *args, **kwargs):\n '''Update the classifier with new data and re-trains the\n classifier.\n\n :param new_positive_data: List of new, labeled strings.\n :param new_unlabeled_data: List of new, unlabeled strings.\n '''\n self.positive_prob_prior = positive_prob_prior\n if new_positive_data:\n self.positive_set += new_positive_data\n self.positive_features += [self.extract_features(d)\n for d in new_positive_data]\n if new_unlabeled_data:\n self.unlabeled_set += new_unlabeled_data\n self.unlabeled_features += [self.extract_features(d)\n for d in new_unlabeled_data]\n self.classifier = self.nltk_class.train(self.positive_features,\n self.unlabeled_features,\n self.positive_prob_prior,\n *args, **kwargs)\n return True", "response": "Update the classifier with new data and re - traains the nltk_classifier."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prob_classify(self, text):\n feats = self.extract_features(text)\n return self.classifier.prob_classify(feats)", "response": "Return the label probability distribution for classifying a string\n of text."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of tuples that are the lemma and tag.", "response": "def lemmatize(self, text):\n \"\"\"Return a list of (lemma, tag) tuples.\n\n :param str text: A string.\n\n \"\"\"\n #: Do not process empty strings (Issue #3)\n if text.strip() == \"\":\n return []\n parsed_sentences = self._parse_text(text)\n _lemmalist = []\n for s in parsed_sentences:\n tokens = s.split()\n for i, t in enumerate(tokens):\n #: Filter empty tokens from the parser output (Issue #5)\n #: This only happens if parser input is improperly tokenized\n #: e.g. if there are empty strings in the list of tokens ['A', '', '.']\n if t.startswith('/'):\n continue\n w, tag, phrase, role, lemma = t.split('/')\n # The lexicon uses Swiss spelling: \"ss\" instead of \"\u00df\".\n lemma = lemma.replace(u\"\u00df\", \"ss\")\n # Reverse previous replacement\n lemma = lemma.strip().replace(\"forwardslash\", \"/\")\n if w[0].isupper() and i > 0:\n lemma = lemma.title()\n elif tag.startswith(\"N\") and i == 0:\n lemma = lemma.title()\n # Todo: Check if it makes sense to treat '/' as punctuation\n # (especially for sentiment analysis it might be interesting\n # to treat it as OR ('oder')).\n if w in string.punctuation or lemma == '/':\n continue\n else:\n lemma = lemma\n\n _lemmalist.append((lemma, tag))\n return _lemmalist"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a string and return a list of parsed sentences.", "response": "def _parse_text(self, text):\n \"\"\"Parse text (string) and return list of parsed sentences (strings).\n\n Each sentence consists of space separated token elements and the\n token format returned by the PatternParser is WORD/TAG/PHRASE/ROLE/LEMMA\n (separated by a forward slash '/')\n\n :param str text: A string.\n\n \"\"\"\n # Fix for issue #1\n text = text.replace(\"/\", \" FORWARDSLASH \")\n _tokenized = \" \".join(self.tokenizer.tokenize(text))\n parsed_text = pattern_parse(_tokenized, tokenize=False, lemmata=True)\n return parsed_text.split('\\n')"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _match(string, pattern):\n p = pattern\n try:\n if p[:1] == WILDCARD and (p[-1:] == WILDCARD and p[1:-1] in string or string.endswith(p[1:])):\n return True\n if p[-1:] == WILDCARD and not p[-2:-1] == \"\\\\\" and string.startswith(p[:-1]):\n return True\n if p == string:\n return True\n if WILDCARD in p[1:-1]:\n p = p.split(WILDCARD)\n return string.startswith(p[0]) and string.endswith(p[-1])\n except:\n # For performance, calling isinstance() last is 10% faster for plain strings.\n if isinstance(p, regexp):\n return p.search(string) is not None\n return False", "response": "Returns True if the pattern matches the given word string."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of unique items in the iterable.", "response": "def unique(iterable):\n \"\"\" Returns a list copy in which each item occurs only once (in-order).\n \"\"\"\n seen = set()\n return [x for x in iterable if x not in seen and not seen.add(x)]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef product(*args, **kwargs):\n p = [[]]\n for iterable in map(tuple, args) * kwargs.get(\"repeat\", 1):\n p = [x + [y] for x in p for y in iterable]\n for p in p:\n yield tuple(p)", "response": "Yields all permutations with replacement"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all possible variations of a sequence with optional items.", "response": "def variations(iterable, optional=lambda x: False):\n \"\"\" Returns all possible variations of a sequence with optional items.\n \"\"\"\n # For example: variations([\"A?\", \"B?\", \"C\"], optional=lambda s: s.endswith(\"?\"))\n # defines a sequence where constraint A and B are optional:\n # [(\"A?\", \"B?\", \"C\"), (\"B?\", \"C\"), (\"A?\", \"C\"), (\"C\")]\n iterable = tuple(iterable)\n # Create a boolean sequence where True means optional:\n # (\"A?\", \"B?\", \"C\") => [True, True, False]\n o = [optional(x) for x in iterable]\n # Find all permutations of the boolean sequence:\n # [True, False, True], [True, False, False], [False, False, True], [False, False, False].\n # Map to sequences of constraints whose index in the boolean sequence yields True.\n a = set()\n for p in product([False, True], repeat=sum(o)):\n p = list(p)\n v = [b and (b and p.pop(0)) for b in o]\n v = tuple(iterable[i] for i in xrange(len(v)) if not v[i])\n a.add(v)\n # Longest-first.\n return sorted(a, cmp=lambda x, y: len(y) - len(x))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a Pattern object from the given string or regular expression.", "response": "def compile(pattern, *args, **kwargs):\n \"\"\" Returns a Pattern from the given string or regular expression.\n Recently compiled patterns are kept in cache\n (if they do not use taxonomies, which are mutable dicts).\n \"\"\"\n id, p = repr(pattern) + repr(args), pattern\n if id in _cache and not kwargs:\n return _cache[id]\n if isinstance(pattern, basestring):\n p = Pattern.fromstring(pattern, *args, **kwargs)\n if isinstance(pattern, regexp):\n p = Pattern([Constraint(words=[pattern], taxonomy=kwargs.get(\"taxonomy\", TAXONOMY))], *args, **kwargs)\n if len(_cache) > _CACHE_SIZE:\n _cache.clear()\n if isinstance(p, Pattern) and not kwargs:\n _cache[id] = p\n if isinstance(p, Pattern):\n return p\n else:\n raise TypeError(\"can't compile '%s' object\" % pattern.__class__.__name__)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef scan(pattern, string, *args, **kwargs):\n return compile(pattern, *args, **kwargs).scan(string)", "response": "Returns True if pattern. search ( string ) may yield matches."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmatches a pattern against a given sentence.", "response": "def match(pattern, sentence, *args, **kwargs):\n \"\"\" Returns the first match found in the given sentence, or None.\n \"\"\"\n return compile(pattern, *args, **kwargs).match(sentence)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch for all matches in the given sentence.", "response": "def search(pattern, sentence, *args, **kwargs):\n \"\"\" Returns a list of all matches found in the given sentence.\n \"\"\"\n return compile(pattern, *args, **kwargs).search(sentence)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef push(self, kv):\n if kv[0] in self: \n self.__delitem__(kv[0])\n self.__setitem__(kv[0], kv[1])", "response": "Adds a new item from the given key - value - tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef append(self, term, type=None, value=None):\n term = self._normalize(term)\n type = self._normalize(type)\n self.setdefault(term, (odict(), odict()))[0].push((type, True))\n self.setdefault(type, (odict(), odict()))[1].push((term, True))\n self._values[term] = value", "response": "Appends the given term to the taxonomy and tags it as the given type."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef classify(self, term, **kwargs):\n term = self._normalize(term)\n if dict.__contains__(self, term):\n return self[term][0].keys()[-1]\n # If the term is not in the dictionary, check the classifiers.\n # Returns the first term in the list returned by a classifier.\n for classifier in self.classifiers:\n # **kwargs are useful if the classifier requests extra information,\n # for example the part-of-speech tag.\n v = classifier.parents(term, **kwargs)\n if v:\n return v[0]", "response": "Returns the most recently added semantic type for the given term."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of all semantic types for the given term. If recursive = True traverses parents up to the root.", "response": "def parents(self, term, recursive=False, **kwargs):\n \"\"\" Returns a list of all semantic types for the given term.\n If recursive=True, traverses parents up to the root.\n \"\"\"\n def dfs(term, recursive=False, visited={}, **kwargs):\n if term in visited: # Break on cyclic relations.\n return []\n visited[term], a = True, []\n if dict.__contains__(self, term):\n a = self[term][0].keys()\n for classifier in self.classifiers:\n a.extend(classifier.parents(term, **kwargs) or [])\n if recursive:\n for w in a: a += dfs(w, recursive, visited, **kwargs)\n return a\n return unique(dfs(self._normalize(term), recursive, {}, **kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef value(self, term, **kwargs):\n term = self._normalize(term)\n if term in self._values:\n return self._values[term]\n for classifier in self.classifiers:\n v = classifier.value(term, **kwargs)\n if v is not None:\n return v", "response": "Returns the value of the given term."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a string containing a set of constraints.", "response": "def fromstring(cls, s, **kwargs):\n \"\"\" Returns a new Constraint from the given string.\n Uppercase words indicate either a tag (\"NN\", \"JJ\", \"VP\")\n or a taxonomy term (e.g., \"PRODUCT\", \"PERSON\").\n Syntax:\n ( defines an optional constraint, e.g., \"(JJ)\".\n [ defines a constraint with spaces, e.g., \"[Mac OS X | Windows Vista]\".\n _ is converted to spaces, e.g., \"Windows_Vista\".\n | separates different options, e.g., \"ADJP|ADVP\".\n ! can be used as a word prefix to disallow it.\n * can be used as a wildcard character, e.g., \"soft*|JJ*\".\n ? as a suffix defines a constraint that is optional, e.g., \"JJ?\".\n + as a suffix defines a constraint that can span multiple words, e.g., \"JJ+\".\n ^ as a prefix defines a constraint that can only match the first word.\n These characters need to be escaped if used as content: \"\\(\".\n \"\"\"\n C = cls(**kwargs)\n s = s.strip()\n s = s.strip(\"{}\")\n s = s.strip()\n for i in range(3):\n # Wrapping order of control characters is ignored:\n # (NN+) == (NN)+ == NN?+ == NN+? == [NN+?] == [NN]+?\n if s.startswith(\"^\"):\n s = s[1: ]; C.first = True\n if s.endswith(\"+\") and not s.endswith(\"\\+\"):\n s = s[0:-1]; C.multiple = True\n if s.endswith(\"?\") and not s.endswith(\"\\?\"):\n s = s[0:-1]; C.optional = True\n if s.startswith(\"(\") and s.endswith(\")\"):\n s = s[1:-1]; C.optional = True\n if s.startswith(\"[\") and s.endswith(\"]\"):\n s = s[1:-1]\n s = re.sub(r\"^\\\\\\^\", \"^\", s)\n s = re.sub(r\"\\\\\\+$\", \"+\", s)\n s = s.replace(\"\\_\", \"&uscore;\")\n s = s.replace(\"_\",\" \")\n s = s.replace(\"&uscore;\", \"_\")\n s = s.replace(\"&lparen;\", \"(\")\n s = s.replace(\"&rparen;\", \")\")\n s = s.replace(\"[\", \"[\")\n s = s.replace(\"]\", \"]\")\n s = s.replace(\"&lcurly;\", \"{\")\n s = s.replace(\"&rcurly;\", \"}\")\n s = s.replace(\"\\(\", \"(\")\n s = s.replace(\"\\)\", \")\") \n s = s.replace(\"\\[\", \"[\")\n s = s.replace(\"\\]\", \"]\") \n s = s.replace(\"\\{\", \"{\")\n s = s.replace(\"\\}\", \"}\") \n s = s.replace(\"\\*\", \"*\")\n s = s.replace(\"\\?\", \"?\") \n s = s.replace(\"\\+\", \"+\")\n s = s.replace(\"\\^\", \"^\")\n s = s.replace(\"\\|\", \"⊢\")\n s = s.split(\"|\")\n s = [v.replace(\"⊢\", \"|\").strip() for v in s]\n for v in s:\n C._append(v)\n return C"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef match(self, word):\n # If the constraint has a custom function it must return True.\n if self.custom is not None and self.custom(word) is False:\n return False\n # If the constraint can only match the first word, Word.index must be 0.\n if self.first and word.index > 0:\n return False\n # If the constraint defines excluded options, Word can not match any of these.\n if self.exclude and self.exclude.match(word):\n return False\n # If the constraint defines allowed tags, Word.tag needs to match one of these.\n if self.tags:\n if find(lambda w: _match(word.tag, w), self.tags) is None:\n return False\n # If the constraint defines allowed chunks, Word.chunk.tag needs to match one of these.\n if self.chunks:\n ch = word.chunk and word.chunk.tag or None\n if find(lambda w: _match(ch, w), self.chunks) is None:\n return False\n # If the constraint defines allowed role, Word.chunk.tag needs to match one of these.\n if self.roles:\n R = word.chunk and [r2 for r1, r2 in word.chunk.relations] or []\n if find(lambda w: w in R, self.roles) is None:\n return False\n # If the constraint defines allowed words,\n # Word.string.lower() OR Word.lemma needs to match one of these.\n b = True # b==True when word in constraint (or Constraints.words=[]).\n if len(self.words) + len(self.taxa) > 0:\n s1 = word.string.lower()\n s2 = word.lemma\n b = False\n for w in itertools.chain(self.words, self.taxa):\n # If the constraint has a word with spaces (e.g., a proper noun),\n # compare it to the entire chunk.\n try:\n if \" \" in w and (s1 in w or s2 and s2 in w or \"*\" in w):\n s1 = word.chunk and word.chunk.string.lower() or s1\n s2 = word.chunk and \" \".join([x or \"\" for x in word.chunk.lemmata]) or s2\n except:\n s1 = s1\n s2 = None\n # Compare the word to the allowed words (which can contain wildcards).\n if _match(s1, w):\n b=True; break\n # Compare the word lemma to the allowed words, e.g.,\n # if \"was\" is not in the constraint, perhaps \"be\" is, which is a good match.\n if s2 and _match(s2, w):\n b=True; break\n # If the constraint defines allowed taxonomy terms,\n # and the given word did not match an allowed word, traverse the taxonomy.\n # The search goes up from the given word to its parents in the taxonomy.\n # This is faster than traversing all the children of terms in Constraint.taxa.\n # The drawback is that:\n # 1) Wildcards in the taxonomy are not detected (use classifiers instead),\n # 2) Classifier.children() has no effect, only Classifier.parent().\n if self.taxa and (not self.words or (self.words and not b)):\n for s in (\n word.string, # \"ants\"\n word.lemma, # \"ant\"\n word.chunk and word.chunk.string or None, # \"army ants\"\n word.chunk and \" \".join([x or \"\" for x in word.chunk.lemmata]) or None): # \"army ant\"\n if s is not None:\n if self.taxonomy.case_sensitive is False:\n s = s.lower()\n # Compare ancestors of the word to each term in Constraint.taxa.\n for p in self.taxonomy.parents(s, recursive=True):\n if find(lambda s: p==s, self.taxa): # No wildcards.\n return True\n return b", "response": "Returns True if the given word is part of the constraint False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new Pattern from the given string.", "response": "def fromstring(cls, s, *args, **kwargs):\n \"\"\" Returns a new Pattern from the given string.\n Constraints are separated by a space.\n If a constraint contains a space, it must be wrapped in [].\n \"\"\"\n s = s.replace(\"\\(\", \"&lparen;\")\n s = s.replace(\"\\)\", \"&rparen;\")\n s = s.replace(\"\\[\", \"[\")\n s = s.replace(\"\\]\", \"]\")\n s = s.replace(\"\\{\", \"&lcurly;\")\n s = s.replace(\"\\}\", \"&rcurly;\")\n p = []\n i = 0\n for m in re.finditer(r\"\\[.*?\\]|\\(.*?\\)\", s):\n # Spaces in a range encapsulated in square brackets are encoded.\n # \"[Windows Vista]\" is one range, don't split on space.\n p.append(s[i:m.start()])\n p.append(s[m.start():m.end()].replace(\" \", \"&space;\")); i=m.end()\n p.append(s[i:])\n s = \"\".join(p) \n s = s.replace(\"][\", \"] [\")\n s = s.replace(\")(\", \") (\")\n s = s.replace(\"\\|\", \"⊢\")\n s = re.sub(r\"\\s+\\|\\s+\", \"|\", s) \n s = re.sub(r\"\\s+\", \" \", s)\n s = re.sub(r\"\\{\\s+\", \"{\", s)\n s = re.sub(r\"\\s+\\}\", \"}\", s)\n s = s.split(\" \")\n s = [v.replace(\"&space;\",\" \") for v in s]\n P = cls([], *args, **kwargs)\n G, O, i = [], [], 0\n for s in s:\n constraint = Constraint.fromstring(s.strip(\"{}\"), taxonomy=kwargs.get(\"taxonomy\", TAXONOMY))\n constraint.index = len(P.sequence)\n P.sequence.append(constraint)\n # Push a new group on the stack if string starts with \"{\".\n # Parse constraint from string, add it to all open groups.\n # Pop latest group from stack if string ends with \"}\".\n # Insert groups in opened-first order (i).\n while s.startswith(\"{\"):\n s = s[1:]\n G.append((i, [])); i+=1\n O.append([])\n for g in G:\n g[1].append(constraint)\n while s.endswith(\"}\"):\n s = s[:-1]\n if G: O[G[-1][0]] = G[-1][1]; G.pop()\n P.groups = [g for g in O if g]\n return P"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef scan(self, string):\n # In the following example, first scan the string for \"good\" and \"bad\":\n # p = Pattern.fromstring(\"good|bad NN\")\n # for s in open(\"parsed.txt\"):\n # if p.scan(s):\n # s = Sentence(s)\n # m = p.search(s)\n # if m:\n # print(m)\n w = (constraint.words for constraint in self.sequence if not constraint.optional)\n w = itertools.chain(*w)\n w = [w.strip(WILDCARD) for w in w if WILDCARD not in w[1:-1]]\n if w and not any(w in string.lower() for w in w):\n return False\n return True", "response": "Returns True if search ( Sentence ( string ) may yield matches."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of all matches found in the given sentence.", "response": "def search(self, sentence):\n \"\"\" Returns a list of all matches found in the given sentence.\n \"\"\"\n if sentence.__class__.__name__ == \"Sentence\":\n pass\n elif isinstance(sentence, list) or sentence.__class__.__name__ == \"Text\":\n a=[]; [a.extend(self.search(s)) for s in sentence]; return a\n elif isinstance(sentence, basestring):\n sentence = Sentence(sentence)\n elif isinstance(sentence, Match) and len(sentence) > 0:\n sentence = sentence[0].sentence.slice(sentence[0].index, sentence[-1].index + 1)\n a = []\n v = self._variations()\n u = {}\n m = self.match(sentence, _v=v)\n while m:\n a.append(m)\n m = self.match(sentence, start=m.words[-1].index+1, _v=v, _u=u)\n return a"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the first match found in the given sentence or None.", "response": "def match(self, sentence, start=0, _v=None, _u=None):\n \"\"\" Returns the first match found in the given sentence, or None.\n \"\"\"\n if sentence.__class__.__name__ == \"Sentence\":\n pass\n elif isinstance(sentence, list) or sentence.__class__.__name__ == \"Text\":\n return find(lambda m,s: m is not None, ((self.match(s, start, _v), s) for s in sentence))[0]\n elif isinstance(sentence, basestring):\n sentence = Sentence(sentence)\n elif isinstance(sentence, Match) and len(sentence) > 0:\n sentence = sentence[0].sentence.slice(sentence[0].index, sentence[-1].index + 1)\n # Variations (_v) further down the list may match words more to the front.\n # We need to check all of them. Unmatched variations are blacklisted (_u).\n # Pattern.search() calls Pattern.match() with a persistent blacklist (1.5x faster).\n a = []\n for sequence in (_v is not None and _v or self._variations()):\n if _u is not None and id(sequence) in _u:\n continue\n m = self._match(sequence, sentence, start)\n if m is not None:\n a.append((m.words[0].index, len(m.words), m))\n if m is not None and m.words[0].index == start:\n return m\n if m is None and _u is not None:\n _u[id(sequence)] = False\n # Return the leftmost-longest.\n if len(a) > 0:\n return sorted(a)[0][-1]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef constraint(self, word):\n if word.index in self._map1:\n return self._map1[word.index]", "response": "Returns the constraint that matches the given Word or None if no constraint is found."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of constraints that match the given Chunk.", "response": "def constraints(self, chunk):\n \"\"\" Returns a list of constraints that match the given Chunk.\n \"\"\"\n a = [self._map1[w.index] for w in chunk.words if w.index in self._map1]\n b = []; [b.append(constraint) for constraint in a if constraint not in b]\n return b"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef constituents(self, constraint=None):\n # Select only words that match the given constraint.\n # Note: this will only work with constraints from Match.pattern.sequence.\n W = self.words\n n = len(self.pattern.sequence)\n if isinstance(constraint, (int, Constraint)):\n if isinstance(constraint, int):\n i = constraint \n i = i<0 and i%n or i\n else:\n i = self.pattern.sequence.index(constraint)\n W = self._map2.get(i,[])\n W = [self.words[i-self.words[0].index] for i in W] \n if isinstance(constraint, (list, tuple)):\n W = []; [W.extend(self._map2.get(j<0 and j%n or j,[])) for j in constraint]\n W = [self.words[i-self.words[0].index] for i in W]\n W = unique(W)\n a = []\n i = 0\n while i < len(W):\n w = W[i]\n if w.chunk and W[i:i+len(w.chunk)] == w.chunk.words:\n i += len(w.chunk) - 1\n a.append(w.chunk)\n else:\n a.append(w)\n i += 1\n return a", "response": "Returns a list of Word and Chunk objects where words have been grouped into their chunks whenever possible. Optionally returns only chunks and words that match given constraint."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of Word objects that match the given group.", "response": "def group(self, index, chunked=False):\n \"\"\" Returns a list of Word objects that match the given group.\n With chunked=True, returns a list of Word + Chunk objects - see Match.constituents().\n A group consists of consecutive constraints wrapped in { }, e.g.,\n search(\"{JJ JJ} NN\", Sentence(parse(\"big black cat\"))).group(1) => big black.\n \"\"\"\n if index < 0 or index > len(self.pattern.groups):\n raise IndexError(\"no such group\")\n if index > 0 and index <= len(self.pattern.groups):\n g = self.pattern.groups[index-1]\n if index == 0:\n g = self.pattern.sequence\n if chunked is True:\n return Group(self, self.constituents(constraint=[self.pattern.sequence.index(x) for x in g]))\n return Group(self, [w for w in self.words if self.constraint(w) in g])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef analyze(self, text):\n if self.lemmatize:\n text = self._lemmatize(text)\n return self.RETURN_TYPE(*pattern_sentiment(text))", "response": "Return the sentiment as a tuple of the form polarity subjectivity"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert an STTS tag to a universal tag.", "response": "def stts2universal(token, tag):\n \"\"\" Converts an STTS tag to a universal tag.\n For example: ohne/APPR => ohne/PREP\n \"\"\"\n if tag in (\"KON\", \"KOUI\", \"KOUS\", \"KOKOM\"):\n return (token, CONJ)\n if tag in (\"PTKZU\", \"PTKNEG\", \"PTKVZ\", \"PTKANT\"):\n return (token, PRT)\n if tag in (\"PDF\", \"PDAT\", \"PIS\", \"PIAT\", \"PIDAT\", \"PPER\", \"PPOS\", \"PPOSAT\"): \n return (token, PRON)\n if tag in (\"PRELS\", \"PRELAT\", \"PRF\", \"PWS\", \"PWAT\", \"PWAV\", \"PAV\"):\n return (token, PRON)\n return penntreebank2universal(*stts2penntreebank(token, tag))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef find_lemmata(tokens):\n for token in tokens:\n word, pos, lemma = token[0], token[1], token[0]\n if pos.startswith((\"DT\", \"JJ\")):\n lemma = predicative(word) \n if pos == \"NNS\":\n lemma = singularize(word)\n if pos.startswith((\"VB\", \"MD\")):\n lemma = conjugate(word, INFINITIVE) or word\n token.append(lemma.lower())\n return tokens", "response": "Annotates the tokens with lemmata for plural nouns and conjugated verbs and\n where each token is a [ word part - of - speech list."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tree(s, token=[WORD, POS, CHUNK, PNP, REL, LEMMA]):\n return Text(s, token)", "response": "Returns a parsed Text from the given parsed string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tag(s, tokenize=True, encoding=\"utf-8\", **kwargs):\n tags = []\n for sentence in parse(s, tokenize, True, False, False, False, encoding, **kwargs).split():\n for token in sentence:\n tags.append((token[0], token[1]))\n return tags", "response": "Returns a list of ( token tag ) - tuples from the given string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef keywords(s, top=10, **kwargs):\n return parser.find_keywords(s, top=top, frequency=parser.frequency)", "response": "Returns a sorted list of keywords in the given string."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tokenize(self, text, include_punc=True, nested=False):\n self.tokens = [\n w for w in (\n self.word_tokenize(\n s,\n include_punc) for s in self.sent_tokenize(text))]\n if nested:\n return self.tokens\n else:\n return list(chain.from_iterable(self.tokens))", "response": "Tokenize a string of text into a list of word tokens."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sent_tokenize(self, text, **kwargs):\n sentences = self.sent_tok.tokenize(\n text,\n realign_boundaries=kwargs.get(\n \"realign_boundaries\",\n True))\n return sentences", "response": "Tokenizes a text into sentences."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sent_tokenize(self, text, **kwargs):\n\n sentences = find_sentences(text,\n punctuation=kwargs.get(\n \"punctuation\",\n PUNCTUATION),\n abbreviations=kwargs.get(\n \"abbreviations\",\n ABBREVIATIONS_DE),\n replace=kwargs.get(\"replace\", replacements),\n linebreak=r\"\\n{2,}\")\n return sentences", "response": "Returns a list of sentences that are sent to the given text."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tokenize(self, text, include_punc=True, **kwargs):\n return self.tokenizer.word_tokenize(text, include_punc, **kwargs)", "response": "Tokenize a string of text into a list of word tokens."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse(self, text):\n #: Do not process empty strings (Issue #3)\n if text.strip() == \"\":\n return \"\"\n #: Do not process strings consisting of a single punctuation mark (Issue #4)\n elif text.strip() in PUNCTUATION:\n _sym = text.strip()\n if _sym in tuple('.?!'):\n _tag = \".\"\n else:\n _tag = _sym\n if self.lemmata:\n return \"{0}/{1}/O/O/{0}\".format(_sym, _tag)\n else:\n return \"{0}/{1}/O/O\".format(_sym, _tag)\n if self.tokenize:\n _tokenized = \" \".join(self.tokenizer.tokenize(text))\n else:\n _tokenized = text\n\n _parsed = pattern_parse(_tokenized,\n # text is tokenized before it is passed on to\n # pattern.de.parse\n tokenize=False,\n tags=self.tags, chunks=self.chunks,\n relations=self.relations, lemmata=self.lemmata,\n encoding=self.encoding, tagset=self.tagset)\n if self.pprint:\n _parsed = pattern_pprint(_parsed)\n\n return _parsed", "response": "Parses the text and returns a dictionary of the related attributes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nextract a list of noun phrases for a body of text.", "response": "def extract(self, text):\n \"\"\"Return a list of noun phrases (strings) for a body of text.\n\n :param str text: A string.\n\n \"\"\"\n _extracted = []\n if text.strip() == \"\":\n return _extracted\n parsed_sentences = self._parse_text(text)\n for s in parsed_sentences:\n tokens = s.split()\n new_np = []\n for t in tokens:\n w, tag, phrase, role = t.split('/')\n # exclude some parser errors (e.g. VB within NP),\n # extend startswith tuple if necessary\n if 'NP' in phrase and not self._is_verb(w, tag):\n if len(new_np) > 0 and w.lower() in START_NEW_NP:\n _extracted.append(\" \".join(new_np))\n new_np = [w]\n else:\n # normalize capitalisation of sentence starters, except\n # for nouns\n new_np.append(w.lower() if tokens[0].startswith(w) and\n not tag.startswith('N') else w)\n else:\n if len(new_np) > 0:\n _extracted.append(\" \".join(new_np))\n new_np = []\n return self._filter_extracted(_extracted)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfilter insignificant words for key noun phrase extraction.", "response": "def _filter_extracted(self, extracted_list):\n \"\"\"Filter insignificant words for key noun phrase extraction.\n\n determiners, relative pronouns, reflexive pronouns\n In general, pronouns are not useful, as you need context to know what they refer to.\n Most of the pronouns, however, are filtered out by blob.noun_phrase method's\n np length (>1) filter\n\n :param list extracted_list: A list of noun phrases extracted from parser output.\n\n \"\"\"\n _filtered = []\n for np in extracted_list:\n _np = np.split()\n if _np[0] in INSIGNIFICANT:\n _np.pop(0)\n try:\n if _np[-1] in INSIGNIFICANT:\n _np.pop(-1)\n # e.g. 'welcher die ...'\n if _np[0] in INSIGNIFICANT:\n _np.pop(0)\n except IndexError:\n _np = []\n if len(_np) > 0:\n _filtered.append(\" \".join(_np))\n return _filtered"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse text and return list of parsed sentences.", "response": "def _parse_text(self, text):\n \"\"\"Parse text (string) and return list of parsed sentences (strings).\n\n Each sentence consists of space separated token elements and the\n token format returned by the PatternParser is WORD/TAG/PHRASE/ROLE/(LEMMA)\n (separated by a forward slash '/')\n\n :param str text: A string.\n\n \"\"\"\n if isinstance(self.tokenizer, PatternTokenizer):\n parsed_text = pattern_parse(text, tokenize=True, lemmata=False)\n else:\n _tokenized = []\n _sentences = sent_tokenize(text, tokenizer=self.tokenizer)\n for s in _sentences:\n _tokenized.append(\" \".join(self.tokenizer.tokenize(s)))\n parsed_text = pattern_parse(\n _tokenized,\n tokenize=False,\n lemmata=False)\n return parsed_text.split('\\n')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tag(self, sentence, tokenize=True):\n #: Do not process empty strings (Issue #3)\n if sentence.strip() == \"\":\n return []\n #: Do not process strings consisting of a single punctuation mark (Issue #4)\n elif sentence.strip() in PUNCTUATION:\n if self.include_punc:\n _sym = sentence.strip()\n if _sym in tuple('.?!'):\n _tag = \".\"\n else:\n _tag = _sym\n return [(_sym, _tag)]\n else:\n return []\n if tokenize:\n _tokenized = \" \".join(self.tokenizer.tokenize(sentence))\n sentence = _tokenized\n # Sentence is tokenized before it is passed on to pattern.de.tag\n # (i.e. it is either submitted tokenized or if )\n _tagged = pattern_tag(sentence, tokenize=False,\n encoding=self.encoding,\n tagset=self.tagset)\n if self.include_punc:\n return _tagged\n else:\n _tagged = [\n (word, t) for word, t in _tagged if not PUNCTUATION_REGEX.match(\n unicode(t))]\n return _tagged", "response": "Tag a string sentence."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef decode_string(v, encoding=\"utf-8\"):\n if isinstance(encoding, basestring):\n encoding = ((encoding,),) + ((\"windows-1252\",), (\"utf-8\", \"ignore\"))\n if isinstance(v, binary_type):\n for e in encoding:\n try:\n return v.decode(*e)\n except:\n pass\n return v\n return unicode(v)", "response": "Returns the given value as a Unicode string ( if possible )."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the given value as a Python byte string ( if possible.", "response": "def encode_string(v, encoding=\"utf-8\"):\n \"\"\"Returns the given value as a Python byte string (if possible).\"\"\"\n if isinstance(encoding, basestring):\n encoding = ((encoding,),) + ((\"windows-1252\",), (\"utf-8\", \"ignore\"))\n if isinstance(v, unicode):\n for e in encoding:\n try:\n return v.encode(*e)\n except:\n pass\n return v\n return str(v)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _shutil_which(cmd, mode=os.F_OK | os.X_OK, path=None):\n # Check that a given file can be accessed with the correct mode.\n # Additionally check that `file` is not a directory, as on Windows\n # directories pass the os.access check.\n def _access_check(fn, mode):\n return (os.path.exists(fn) and os.access(fn, mode)\n and not os.path.isdir(fn))\n\n # If we're given a path with a directory part, look it up directly rather\n # than referring to PATH directories. This includes checking relative to the\n # current directory, e.g. ./script\n if os.path.dirname(cmd):\n if _access_check(cmd, mode):\n return cmd\n return None\n\n if path is None:\n path = os.environ.get(\"PATH\", os.defpath)\n if not path:\n return None\n path = path.split(os.pathsep)\n\n if sys.platform == \"win32\":\n # The current directory takes precedence on Windows.\n if not os.curdir in path:\n path.insert(0, os.curdir)\n\n # PATHEXT is necessary to check on Windows.\n pathext = os.environ.get(\"PATHEXT\", \"\").split(os.pathsep)\n # See if the given file matches any of the expected path extensions.\n # This will allow us to short circuit when given \"python.exe\".\n # If it does match, only test that one, otherwise we have to try\n # others.\n if any([cmd.lower().endswith(ext.lower()) for ext in pathext]):\n files = [cmd]\n else:\n files = [cmd + ext for ext in pathext]\n else:\n # On other platforms you don't have things like PATHEXT to tell you\n # what file suffixes are executable, so just pass on cmd as-is.\n files = [cmd]\n\n seen = set()\n for dir in path:\n normdir = os.path.normcase(dir)\n if normdir not in seen:\n seen.add(normdir)\n for thefile in files:\n name = os.path.join(dir, thefile)\n if _access_check(name, mode):\n return name\n return None", "response": "Returns the path which conforms to the given command mode and a PATH string."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef translate(self, from_lang=None, to=\"de\"):\n if from_lang is None:\n from_lang = self.translator.detect(self.string)\n return self.translator.translate(self.string,\n from_lang=from_lang, to_lang=to)", "response": "Translate the word to another language using Google s Translate API."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef lemmatize(self):\n _lemmatizer = PatternParserLemmatizer(tokenizer=NLTKPunktTokenizer())\n # WordList object --> Sentence.string\n # add a period (improves parser accuracy)\n _raw = \" \".join(self) + \".\"\n _lemmas = _lemmatizer.lemmatize(_raw)\n return self.__class__([Word(l, t) for l, t in _lemmas])", "response": "Return the lemma of each word in this WordList. This method uses NLTKPunktTokenizer to get all lemmatization results for all words in this WordList."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tokenize(self, tokenizer=None):\n t = tokenizer if tokenizer is not None else self.tokenizer\n return WordList(t.tokenize(self.raw))", "response": "Return a list of tokens using tokenizer."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of noun phrases for this blob.", "response": "def noun_phrases(self):\n \"\"\"Returns a list of noun phrases for this blob.\"\"\"\n return WordList([phrase.strip()\n for phrase in self.np_extractor.extract(self.raw)\n if len(phrase.split()) > 1])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pos_tags(self):\n return [(Word(word, pos_tag=t), unicode(t))\n for word, t in self.pos_tagger.tag(self.raw)\n # new keyword PatternTagger(include_punc=False)\n # if not PUNCTUATION_REGEX.match(unicode(t))\n ]", "response": "Returns a list of tuples of the form word POS tag."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dict(self):\n return {\n 'raw': self.raw,\n 'start_index': self.start_index,\n 'end_index': self.end_index,\n 'stripped': self.stripped,\n 'noun_phrases': self.noun_phrases,\n 'polarity': self.polarity,\n 'subjectivity': self.subjectivity,\n }", "response": "The dict representation of this sentence."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of word tokens. This excludes punctuation characters.", "response": "def words(self):\n \"\"\"Return a list of word tokens. This excludes punctuation characters.\n If you want to include punctuation characters, access the ``tokens``\n property.\n\n :returns: A :class:`WordList ` of word tokens.\n\n \"\"\"\n return WordList(\n word_tokenize(self.raw, self.tokenizer, include_punc=False))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sentiment(self):\n #: Enhancement Issue #2\n #: adapted from 'textblob.en.sentiments.py'\n #: Return type declaration\n _RETURN_TYPE = namedtuple('Sentiment', ['polarity', 'subjectivity'])\n _polarity = 0\n _subjectivity = 0\n for s in self.sentences:\n _polarity += s.polarity\n _subjectivity += s.subjectivity\n try:\n polarity = _polarity / len(self.sentences)\n except ZeroDivisionError:\n polarity = 0.0\n try:\n subjectivity = _subjectivity / len(self.sentences)\n except ZeroDivisionError:\n subjectivity = 0.0\n return _RETURN_TYPE(polarity, subjectivity)", "response": "Return a namedtuple containing the polarity and subjectivity of the current object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_json(self, *args, **kwargs):\n return json.dumps(self.serialized, *args, **kwargs)", "response": "Return a json representation of this blob. Takes the same\n arguments as json. dumps."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _create_sentence_objects(self):\n sentence_objects = []\n sentences = sent_tokenize(self.raw, tokenizer=self.tokenizer)\n char_index = 0 # Keeps track of character index within the blob\n for sent in sentences:\n # Compute the start and end indices of the sentence\n # within the blob. This only works if the sentence splitter\n # does not perform any character replacements or changes to\n # white space.\n # Working: NLTKPunktTokenizer\n # Not working: PatternTokenizer\n try:\n start_index = self.raw.index(sent, char_index)\n char_index += len(sent)\n end_index = start_index + len(sent)\n except ValueError:\n start_index = None\n end_index = None\n # Sentences share the same models as their parent blob\n s = Sentence(\n sent,\n start_index=start_index,\n end_index=end_index,\n tokenizer=self.tokenizer,\n np_extractor=self.np_extractor,\n pos_tagger=self.pos_tagger,\n analyzer=self.analyzer,\n parser=self.parser,\n classifier=self.classifier)\n sentence_objects.append(s)\n return sentence_objects", "response": "Returns a list of Sentence objects from the raw text."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of n - grams from the given string.", "response": "def ngrams(string, n=3, punctuation=PUNCTUATION, continuous=False):\n \"\"\" Returns a list of n-grams (tuples of n successive words) from the given string.\n Alternatively, you can supply a Text or Sentence object.\n With continuous=False, n-grams will not run over sentence markers (i.e., .!?).\n Punctuation marks are stripped from words.\n \"\"\"\n def strip_punctuation(s, punctuation=set(punctuation)):\n return [w for w in s if (isinstance(w, Word) and w.string or w) not in punctuation]\n if n <= 0:\n return []\n if isinstance(string, basestring):\n s = [strip_punctuation(s.split(\" \")) for s in tokenize(string)]\n if isinstance(string, Sentence):\n s = [strip_punctuation(string)]\n if isinstance(string, Text):\n s = [strip_punctuation(s) for s in string]\n if continuous:\n s = [sum(s, [])]\n g = []\n for s in s:\n #s = [None] + s + [None]\n g.extend([tuple(s[i:i+n]) for i in range(len(s)-n+1)])\n return g"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a string with no more than n repeated characters.", "response": "def deflood(s, n=3):\n \"\"\" Returns the string with no more than n repeated characters, e.g.,\n deflood(\"NIIIICE!!\", n=1) => \"Nice!\"\n deflood(\"nice.....\", n=3) => \"nice...\"\n \"\"\"\n if n == 0:\n return s[0:0]\n return re.sub(r\"((.)\\2{%s,})\" % (n-1), lambda m: m.group(1)[0] * n, s)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pprint(string, token=[WORD, POS, CHUNK, PNP], column=4):\n if isinstance(string, basestring):\n print(\"\\n\\n\".join([table(sentence, fill=column) for sentence in Text(string, token)]))\n if isinstance(string, Text):\n print(\"\\n\\n\".join([table(sentence, fill=column) for sentence in string]))\n if isinstance(string, Sentence):\n print(table(string, fill=column))", "response": "Pretty - prints the output of Parser. parse as a table with outlined columns."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads the file at the given path and returns an iterator over the lines in the file at the given path stripping comments and decoding each line to Unicode.", "response": "def _read(path, encoding=\"utf-8\", comment=\";;;\"):\n \"\"\" Returns an iterator over the lines in the file at the given path,\n strippping comments and decoding each line to Unicode.\n \"\"\"\n if path:\n if isinstance(path, basestring) and os.path.exists(path):\n # From file path.\n if PY2:\n f = codecs.open(path, 'r', encoding='utf-8')\n else:\n f = open(path, 'r', encoding='utf-8')\n elif isinstance(path, basestring):\n # From string.\n f = path.splitlines()\n else:\n # From file or buffer.\n f = path\n for i, line in enumerate(f):\n line = line.strip(codecs.BOM_UTF8) if i == 0 and isinstance(line, binary_type) else line\n line = line.strip()\n line = decode_utf8(line, encoding)\n if not line or (comment and line.startswith(comment)):\n continue\n yield line\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef penntreebank2universal(token, tag):\n if tag.startswith((\"NNP-\", \"NNPS-\")):\n return (token, \"%s-%s\" % (NOUN, tag.split(\"-\")[-1]))\n if tag in (\"NN\", \"NNS\", \"NNP\", \"NNPS\", \"NP\"):\n return (token, NOUN)\n if tag in (\"MD\", \"VB\", \"VBD\", \"VBG\", \"VBN\", \"VBP\", \"VBZ\"):\n return (token, VERB)\n if tag in (\"JJ\", \"JJR\", \"JJS\"):\n return (token, ADJ)\n if tag in (\"RB\", \"RBR\", \"RBS\", \"WRB\"):\n return (token, ADV)\n if tag in (\"PRP\", \"PRP$\", \"WP\", \"WP$\"):\n return (token, PRON)\n if tag in (\"DT\", \"PDT\", \"WDT\", \"EX\"):\n return (token, DET)\n if tag in (\"IN\",):\n return (token, PREP)\n if tag in (\"CD\",):\n return (token, NUM)\n if tag in (\"CC\",):\n return (token, CONJ)\n if tag in (\"UH\",):\n return (token, INTJ)\n if tag in (\"POS\", \"RP\", \"TO\"):\n return (token, PRT)\n if tag in (\"SYM\", \"LS\", \".\", \"!\", \"?\", \",\", \":\", \"(\", \")\", \"\\\"\", \"#\", \"$\"):\n return (token, PUNC)\n return (token, X)", "response": "Returns a tuple with a simplified universal part - of - speech tag."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_tokens(string, punctuation=PUNCTUATION, abbreviations=ABBREVIATIONS, replace=replacements, linebreak=r\"\\n{2,}\"):\n # Handle periods separately.\n punctuation = tuple(punctuation.replace(\".\", \"\"))\n # Handle replacements (contractions).\n for a, b in replace.items():\n string = re.sub(a, b, string)\n # Handle Unicode quotes.\n if isinstance(string, unicode):\n string = string.replace(u\"\u201c\", u\" \u201c \")\n string = string.replace(u\"\u201d\", u\" \u201d \")\n string = string.replace(u\"\u2018\", u\" \u2018 \")\n string = string.replace(u\"\u2019\", u\" \u2019 \")\n # Collapse whitespace.\n string = re.sub(\"\\r\\n\", \"\\n\", string)\n string = re.sub(linebreak, \" %s \" % EOS, string)\n string = re.sub(r\"\\s+\", \" \", string)\n tokens = []\n # Handle punctuation marks.\n for t in TOKEN.findall(string+\" \"):\n if len(t) > 0:\n tail = []\n while t.startswith(punctuation) and \\\n not t in replace:\n # Split leading punctuation.\n if t.startswith(punctuation):\n tokens.append(t[0]); t=t[1:]\n while t.endswith(punctuation+(\".\",)) and \\\n not t in replace:\n # Split trailing punctuation.\n if t.endswith(punctuation):\n tail.append(t[-1]); t=t[:-1]\n # Split ellipsis (...) before splitting period.\n if t.endswith(\"...\"):\n tail.append(\"...\"); t=t[:-3].rstrip(\".\")\n # Split period (if not an abbreviation).\n if t.endswith(\".\"):\n if t in abbreviations or \\\n RE_ABBR1.match(t) is not None or \\\n RE_ABBR2.match(t) is not None or \\\n RE_ABBR3.match(t) is not None:\n break\n else:\n tail.append(t[-1]); t=t[:-1]\n if t != \"\":\n tokens.append(t)\n tokens.extend(reversed(tail))\n # Handle sentence breaks (periods, quotes, parenthesis).\n sentences, i, j = [[]], 0, 0\n while j < len(tokens):\n if tokens[j] in (\"...\", \".\", \"!\", \"?\", EOS):\n while j < len(tokens) \\\n and tokens[j] in (\"'\", \"\\\"\", u\"\u201d\", u\"\u2019\", \"...\", \".\", \"!\", \"?\", \")\", EOS):\n if tokens[j] in (\"'\", \"\\\"\") and sentences[-1].count(tokens[j]) % 2 == 0:\n break # Balanced quotes.\n j += 1\n sentences[-1].extend(t for t in tokens[i:j] if t != EOS)\n sentences.append([])\n i = j\n j += 1\n # Handle emoticons.\n sentences[-1].extend(tokens[i:j])\n sentences = (\" \".join(s) for s in sentences if len(s) > 0)\n sentences = (RE_SARCASM.sub(\"(!)\", s) for s in sentences)\n sentences = [RE_EMOTICONS.sub(\n lambda m: m.group(1).replace(\" \", \"\") + m.group(2), s) for s in sentences]\n return sentences", "response": "Find all tokens in a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndefault morphological tagging rules for English based on word suffixes.", "response": "def _suffix_rules(token, tag=\"NN\"):\n \"\"\" Default morphological tagging rules for English, based on word suffixes.\n \"\"\"\n if isinstance(token, (list, tuple)):\n token, tag = token\n if token.endswith(\"ing\"):\n tag = \"VBG\"\n if token.endswith(\"ly\"):\n tag = \"RB\"\n if token.endswith(\"s\") and not token.endswith((\"is\", \"ous\", \"ss\")):\n tag = \"NNS\"\n if token.endswith((\"able\", \"al\", \"ful\", \"ible\", \"ient\", \"ish\", \"ive\", \"less\", \"tic\", \"ous\")) or \"-\" in token:\n tag = \"JJ\"\n if token.endswith(\"ed\"):\n tag = \"VBN\"\n if token.endswith((\"ate\", \"ify\", \"ise\", \"ize\")):\n tag = \"VBP\"\n return [token, tag]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_tags(tokens, lexicon={}, model=None, morphology=None, context=None, entities=None, default=(\"NN\", \"NNP\", \"CD\"), language=\"en\", map=None, **kwargs):\n tagged = []\n # Tag known words.\n for i, token in enumerate(tokens):\n tagged.append([token, lexicon.get(token, i == 0 and lexicon.get(token.lower()) or None)])\n # Tag unknown words.\n for i, (token, tag) in enumerate(tagged):\n prev, next = (None, None), (None, None)\n if i > 0:\n prev = tagged[i-1]\n if i < len(tagged) - 1:\n next = tagged[i+1]\n if tag is None or token in (model is not None and model.unknown or ()):\n # Use language model (i.e., SLP).\n if model is not None:\n tagged[i] = model.apply([token, None], prev, next)\n # Use NNP for capitalized words (except in German).\n elif token.istitle() and language != \"de\":\n tagged[i] = [token, default[1]]\n # Use CD for digits and numbers.\n elif CD.match(token) is not None:\n tagged[i] = [token, default[2]]\n # Use suffix rules (e.g., -ly = RB).\n elif morphology is not None:\n tagged[i] = morphology.apply([token, default[0]], prev, next)\n # Use suffix rules (English default).\n elif language == \"en\":\n tagged[i] = _suffix_rules([token, default[0]])\n # Use most frequent tag (NN).\n else:\n tagged[i] = [token, default[0]]\n # Tag words by context.\n if context is not None and model is None:\n tagged = context.apply(tagged)\n # Tag named entities.\n if entities is not None:\n tagged = entities.apply(tagged)\n # Map tags with a custom function.\n if map is not None:\n tagged = [list(map(token, tag)) or [token, default[0]] for token, tag in tagged]\n return tagged", "response": "Find tags for a list of tokens."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind the chunks of a given language.", "response": "def find_chunks(tagged, language=\"en\"):\n \"\"\" The input is a list of [token, tag]-items.\n The output is a list of [token, tag, chunk]-items:\n The/DT nice/JJ fish/NN is/VBZ dead/JJ ./. =>\n The/DT/B-NP nice/JJ/I-NP fish/NN/I-NP is/VBZ/B-VP dead/JJ/B-ADJP ././O\n \"\"\"\n chunked = [x for x in tagged]\n tags = \"\".join(\"%s%s\" % (tag, SEPARATOR) for token, tag in tagged)\n # Use Germanic or Romance chunking rules according to given language.\n for tag, rule in CHUNKS[int(language in (\"ca\", \"es\", \"pt\", \"fr\", \"it\", \"pt\", \"ro\"))]:\n for m in rule.finditer(tags):\n # Find the start of chunks inside the tags-string.\n # Number of preceding separators = number of preceding tokens.\n i = m.start()\n j = tags[:i].count(SEPARATOR)\n n = m.group(0).count(SEPARATOR)\n for k in range(j, j+n):\n if len(chunked[k]) == 3:\n continue\n if len(chunked[k]) < 3:\n # A conjunction or comma cannot be start of a chunk.\n if k == j and chunked[k][1] in (\"CC\", \"CJ\", \",\"):\n j += 1\n # Mark first token in chunk with B-.\n elif k == j:\n chunked[k].append(\"B-\" + tag)\n # Mark other tokens in chunk with I-.\n else:\n chunked[k].append(\"I-\" + tag)\n # Mark chinks (tokens outside of a chunk) with O-.\n for chink in filter(lambda x: len(x) < 3, chunked):\n chink.append(\"O\")\n # Post-processing corrections.\n for i, (word, tag, chunk) in enumerate(chunked):\n if tag.startswith(\"RB\") and chunk == \"B-NP\":\n # \"Perhaps you\" => ADVP + NP\n # \"Really nice work\" => NP\n # \"Really, nice work\" => ADVP + O + NP\n if i < len(chunked)-1 and not chunked[i+1][1].startswith(\"JJ\"):\n chunked[i+0][2] = \"B-ADVP\"\n chunked[i+1][2] = \"B-NP\"\n if i < len(chunked)-1 and chunked[i+1][1] in (\"CC\", \"CJ\", \",\"):\n chunked[i+1][2] = \"O\"\n if i < len(chunked)-2 and chunked[i+1][2] == \"O\":\n chunked[i+2][2] = \"B-NP\"\n return chunked"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfinds prepositions of the items in a list of tokens.", "response": "def find_prepositions(chunked):\n \"\"\" The input is a list of [token, tag, chunk]-items.\n The output is a list of [token, tag, chunk, preposition]-items.\n PP-chunks followed by NP-chunks make up a PNP-chunk.\n \"\"\"\n # Tokens that are not part of a preposition just get the O-tag.\n for ch in chunked:\n ch.append(\"O\")\n for i, chunk in enumerate(chunked):\n if chunk[2].endswith(\"PP\") and chunk[-1] == \"O\":\n # Find PP followed by other PP, NP with nouns and pronouns, VP with a gerund.\n if i < len(chunked)-1 and \\\n (chunked[i+1][2].endswith((\"NP\", \"PP\")) or \\\n chunked[i+1][1] in (\"VBG\", \"VBN\")):\n chunk[-1] = \"B-PNP\"\n pp = True\n for ch in chunked[i+1:]:\n if not (ch[2].endswith((\"NP\", \"PP\")) or ch[1] in (\"VBG\", \"VBN\")):\n break\n if ch[2].endswith(\"PP\") and pp:\n ch[-1] = \"I-PNP\"\n if not ch[2].endswith(\"PP\"):\n ch[-1] = \"I-PNP\"\n pp = False\n return chunked"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_relations(chunked):\n tag = lambda token: token[2].split(\"-\")[-1] # B-NP => NP\n # Group successive tokens with the same chunk-tag.\n chunks = []\n for token in chunked:\n if len(chunks) == 0 \\\n or token[2].startswith(\"B-\") \\\n or tag(token) != tag(chunks[-1][-1]):\n chunks.append([])\n chunks[-1].append(token+[\"O\"])\n # If a VP is preceded by a NP, the NP is tagged as NP-SBJ-(id).\n # If a VP is followed by a NP, the NP is tagged as NP-OBJ-(id).\n # Chunks that are not part of a relation get an O-tag.\n id = 0\n for i, chunk in enumerate(chunks):\n if tag(chunk[-1]) == \"VP\" and i > 0 and tag(chunks[i-1][-1]) == \"NP\":\n if chunk[-1][-1] == \"O\":\n id += 1\n for token in chunk:\n token[-1] = \"VP-\" + str(id)\n for token in chunks[i-1]:\n token[-1] += \"*NP-SBJ-\" + str(id)\n token[-1] = token[-1].lstrip(\"O-*\")\n if tag(chunk[-1]) == \"VP\" and i < len(chunks)-1 and tag(chunks[i+1][-1]) == \"NP\":\n if chunk[-1][-1] == \"O\":\n id += 1\n for token in chunk:\n token[-1] = \"VP-\" + str(id)\n for token in chunks[i+1]:\n token[-1] = \"*NP-OBJ-\" + str(id)\n token[-1] = token[-1].lstrip(\"O-*\")\n # This is more a proof-of-concept than useful in practice:\n # PP-LOC = be + in|at + the|my\n # PP-DIR = go + to|towards + the|my\n for i, chunk in enumerate(chunks):\n if 0 < i < len(chunks)-1 and len(chunk) == 1 and chunk[-1][-1] == \"O\":\n t0, t1, t2 = chunks[i-1][-1], chunks[i][0], chunks[i+1][0] # previous / current / next\n if tag(t1) == \"PP\" and t2[1] in (\"DT\", \"PR\", \"PRP$\"):\n if t0[0] in BE and t1[0] in (\"in\", \"at\") : t1[-1] = \"PP-LOC\"\n if t0[0] in GO and t1[0] in (\"to\", \"towards\") : t1[-1] = \"PP-DIR\"\n related = []; [related.extend(chunk) for chunk in chunks]\n return related", "response": "This function finds the relation between two noun phrase tokens and a list of tokens."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a sorted list of keywords in the given string.", "response": "def find_keywords(string, parser, top=10, frequency={}, **kwargs):\n \"\"\" Returns a sorted list of keywords in the given string.\n The given parser (e.g., pattern.en.parser) is used to identify noun phrases.\n The given frequency dictionary can be a reference corpus,\n with relative document frequency (df, 0.0-1.0) for each lemma, \n e.g., {\"the\": 0.8, \"cat\": 0.1, ...}\n \"\"\"\n lemmata = kwargs.pop(\"lemmata\", kwargs.pop(\"stem\", True))\n # Parse the string and extract noun phrases (NP).\n chunks = []\n wordcount = 0\n for sentence in parser.parse(string, chunks=True, lemmata=lemmata).split():\n for w in sentence: # [\"cats\", \"NNS\", \"I-NP\", \"O\", \"cat\"]\n if w[2] == \"B-NP\":\n chunks.append([w])\n wordcount += 1\n elif w[2] == \"I-NP\" and w[1][:3] == chunks[-1][-1][1][:3] == \"NNP\":\n chunks[-1][-1][+0] += \" \" + w[+0] # Collapse NNPs: \"Ms Kitty\".\n chunks[-1][-1][-1] += \" \" + w[-1]\n elif w[2] == \"I-NP\":\n chunks[-1].append(w)\n wordcount += 1\n # Rate the nouns in noun phrases.\n m = {}\n for i, chunk in enumerate(chunks):\n head = True\n if parser.language not in (\"ca\", \"es\", \"pt\", \"fr\", \"it\", \"pt\", \"ro\"):\n # Head of \"cat hair\" => \"hair\".\n # Head of \"poils de chat\" => \"poils\".\n chunk = list(reversed(chunk))\n for w in chunk:\n if w[1].startswith(\"NN\"):\n if lemmata:\n k = w[-1]\n else:\n k = w[0].lower()\n if not k in m:\n m[k] = [0.0, set(), 1.0, 1.0, 1.0]\n # Higher score for chunks that appear more frequently.\n m[k][0] += 1 / float(wordcount)\n # Higher score for chunks that appear in more contexts (semantic centrality).\n m[k][1].add(\" \".join(map(lambda x: x[0], chunk)).lower())\n # Higher score for chunks at the start (25%) of the text.\n m[k][2] += 1 if float(i) / len(chunks) <= 0.25 else 0\n # Higher score for chunks not in a prepositional phrase.\n m[k][3] += 1 if w[3] == \"O\" else 0\n # Higher score for chunk head.\n m[k][4] += 1 if head else 0\n head = False\n # Rate tf-idf if a frequency dict is given.\n for k in m:\n if frequency:\n df = frequency.get(k, 0.0)\n df = max(df, 1e-10)\n df = log(1.0 / df, 2.71828)\n else:\n df = 1.0\n m[k][0] = max(1e-10, m[k][0] * df)\n m[k][1] = 1 + float(len(m[k][1]))\n # Sort candidates alphabetically by total score\n # The harmonic mean will emphasize tf-idf score.\n hmean = lambda a: len(a) / sum(1.0 / x for x in a)\n m = [(hmean(m[k]), k) for k in m]\n m = sorted(m, key=lambda x: x[1])\n m = sorted(m, key=lambda x: x[0], reverse=True)\n m = [k for score, k in m]\n return m[:top]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tense_id(*args, **kwargs):\n # Unpack tense given as a tuple, e.g., tense((PRESENT, 1, SG)):\n if len(args) == 1 and isinstance(args[0], (list, tuple)):\n if args[0] not in ((PRESENT, PARTICIPLE), (PAST, PARTICIPLE)):\n args = args[0]\n # No parameters defaults to tense=INFINITIVE, tense=PRESENT otherwise.\n if len(args) == 0 and len(kwargs) == 0:\n t = INFINITIVE\n else:\n t = PRESENT\n # Set default values.\n tense = kwargs.get(\"tense\" , args[0] if len(args) > 0 else t)\n person = kwargs.get(\"person\" , args[1] if len(args) > 1 else 3) or None\n number = kwargs.get(\"number\" , args[2] if len(args) > 2 else SINGULAR)\n mood = kwargs.get(\"mood\" , args[3] if len(args) > 3 else INDICATIVE)\n aspect = kwargs.get(\"aspect\" , args[4] if len(args) > 4 else IMPERFECTIVE)\n negated = kwargs.get(\"negated\", args[5] if len(args) > 5 else False)\n # Disambiguate wrong order of parameters.\n if mood in (PERFECTIVE, IMPERFECTIVE):\n mood, aspect = INDICATIVE, mood\n # Disambiguate INFINITIVE.\n # Disambiguate PARTICIPLE, IMPERFECT, PRETERITE.\n # These are often considered to be tenses but are in fact tense + aspect.\n if tense == INFINITIVE:\n person = number = mood = aspect = None; negated=False\n if tense in ((PRESENT, PARTICIPLE), PRESENT+PARTICIPLE, PARTICIPLE, GERUND):\n tense, aspect = PRESENT, PROGRESSIVE\n if tense in ((PAST, PARTICIPLE), PAST+PARTICIPLE):\n tense, aspect = PAST, PROGRESSIVE\n if tense == IMPERFECT:\n tense, aspect = PAST, IMPERFECTIVE\n if tense == PRETERITE:\n tense, aspect = PAST, PERFECTIVE\n if aspect in (CONTINUOUS, PARTICIPLE, GERUND):\n aspect = PROGRESSIVE\n if aspect == PROGRESSIVE:\n person = number = None\n # Disambiguate CONDITIONAL.\n # In Spanish, the conditional is regarded as an indicative tense.\n if tense == CONDITIONAL and mood == INDICATIVE:\n tense, mood = PRESENT, CONDITIONAL\n # Disambiguate aliases: \"pl\" =>\n # (PRESENT, None, PLURAL, INDICATIVE, IMPERFECTIVE, False).\n return TENSES_ID.get(tense.lower(),\n TENSES_ID.get((tense, person, number, mood, aspect, negated)))", "response": "Returns the tense id for a given set of parameters."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the value of the given function in the given language module.", "response": "def _multilingual(function, *args, **kwargs):\n \"\"\" Returns the value from the function with the given name in the given language module.\n By default, language=\"en\".\n \"\"\"\n return getattr(_module(kwargs.pop(\"language\", \"en\")), function)(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a tuple of language and confidence for the given string.", "response": "def language(s):\n \"\"\" Returns a (language, confidence)-tuple for the given string.\n \"\"\"\n s = decode_utf8(s)\n s = set(w.strip(PUNCTUATION) for w in s.replace(\"'\", \"' \").split())\n n = float(len(s) or 1)\n p = {}\n for xx in LANGUAGES:\n lexicon = _module(xx).__dict__[\"lexicon\"]\n p[xx] = sum(1 for w in s if w in lexicon) / n\n return max(p.items(), key=lambda kv: (kv[1], int(kv[0] == \"en\")))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _lazy(self, method, *args):\n if list.__len__(self) == 0:\n self.load()\n setattr(self, method, types.MethodType(getattr(list, method), self))\n return getattr(list, method)(self, *args)", "response": "Wrapper for lazylist. method."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntrain the model to predict the given tag for the given token.", "response": "def train(self, token, tag, previous=None, next=None):\n \"\"\" Trains the model to predict the given tag for the given token,\n in context of the given previous and next (token, tag)-tuples.\n \"\"\"\n self._classifier.train(self._v(token, previous, next), type=tag)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef classify(self, token, previous=None, next=None, **kwargs):\n return self._classifier.classify(self._v(token, previous, next), **kwargs)", "response": "Classifies the given token in context of the given previous and next tags."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a tuple of the given token and tag in context of the given previous and next.", "response": "def apply(self, token, previous=(None, None), next=(None, None)):\n \"\"\" Returns a (token, tag)-tuple for the given token,\n in context of the given previous and next (token, tag)-tuples.\n \"\"\"\n return [token[0], self._classifier.classify(self._v(token[0], previous, next))]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _v(self, token, previous=None, next=None):\n def f(v, s1, s2):\n if s2: \n v[s1 + \" \" + s2] = 1\n p, n = previous, next\n p = (\"\", \"\") if not p else (p[0] or \"\", p[1] or \"\")\n n = (\"\", \"\") if not n else (n[0] or \"\", n[1] or \"\")\n v = {}\n f(v, \"b\", \"b\") # Bias.\n f(v, \"h\", token[0]) # Capitalization.\n f(v, \"w\", token[-6:] if token not in self.known or token in self.unknown else \"\")\n f(v, \"x\", token[-3:]) # Word suffix.\n f(v, \"-x\", p[0][-3:]) # Word suffix left.\n f(v, \"+x\", n[0][-3:]) # Word suffix right.\n f(v, \"-t\", p[1]) # Tag left.\n f(v, \"-+\", p[1] + n[1]) # Tag left + right.\n f(v, \"+t\", n[1]) # Tag right.\n return v", "response": "Returns a training vector for the given word tag - tuple and its context."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\napplying lexical rules to the given token.", "response": "def apply(self, token, previous=(None, None), next=(None, None)):\n \"\"\" Applies lexical rules to the given token, which is a [word, tag] list.\n \"\"\"\n w = token[0]\n for r in self:\n if r[1] in self._cmd: # Rule = ly hassuf 2 RB x\n f, x, pos, cmd = bool(0), r[0], r[-2], r[1].lower()\n if r[2] in self._cmd: # Rule = NN s fhassuf 1 NNS x\n f, x, pos, cmd = bool(1), r[1], r[-2], r[2].lower().lstrip(\"f\")\n if f and token[1] != r[0]:\n continue\n if (cmd == \"word\" and x == w) \\\n or (cmd == \"char\" and x in w) \\\n or (cmd == \"haspref\" and w.startswith(x)) \\\n or (cmd == \"hassuf\" and w.endswith(x)) \\\n or (cmd == \"addpref\" and x + w in self.known) \\\n or (cmd == \"addsuf\" and w + x in self.known) \\\n or (cmd == \"deletepref\" and w.startswith(x) and w[len(x):] in self.known) \\\n or (cmd == \"deletesuf\" and w.endswith(x) and w[:-len(x)] in self.known) \\\n or (cmd == \"goodleft\" and x == next[0]) \\\n or (cmd == \"goodright\" and x == previous[0]):\n token[1] = pos\n return token"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninsert a new rule that assigns the given tag to words with the given affix.", "response": "def insert(self, i, tag, affix, cmd=\"hassuf\", tagged=None):\n \"\"\" Inserts a new rule that assigns the given tag to words with the given affix,\n e.g., Morphology.append(\"RB\", \"-ly\").\n \"\"\"\n if affix.startswith(\"-\") and affix.endswith(\"-\"):\n affix, cmd = affix[+1:-1], \"char\"\n if affix.startswith(\"-\"):\n affix, cmd = affix[+1:-0], \"hassuf\"\n if affix.endswith(\"-\"):\n affix, cmd = affix[+0:-1], \"haspref\"\n if tagged:\n r = [tagged, affix, \"f\"+cmd.lstrip(\"f\"), tag, \"x\"]\n else:\n r = [affix, cmd.lstrip(\"f\"), tag, \"x\"]\n lazylist.insert(self, i, r)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\napply contextual rules to the given list of tokens.", "response": "def apply(self, tokens):\n \"\"\" Applies contextual rules to the given list of tokens,\n where each token is a [word, tag] list.\n \"\"\"\n o = [(\"STAART\", \"STAART\")] * 3 # Empty delimiters for look ahead/back.\n t = o + tokens + o\n for i, token in enumerate(t):\n for r in self:\n if token[1] == \"STAART\":\n continue\n if token[1] != r[0] and r[0] != \"*\":\n continue\n cmd, x, y = r[2], r[3], r[4] if len(r) > 4 else \"\"\n cmd = cmd.lower()\n if (cmd == \"prevtag\" and x == t[i-1][1]) \\\n or (cmd == \"nexttag\" and x == t[i+1][1]) \\\n or (cmd == \"prev2tag\" and x == t[i-2][1]) \\\n or (cmd == \"next2tag\" and x == t[i+2][1]) \\\n or (cmd == \"prev1or2tag\" and x in (t[i-1][1], t[i-2][1])) \\\n or (cmd == \"next1or2tag\" and x in (t[i+1][1], t[i+2][1])) \\\n or (cmd == \"prev1or2or3tag\" and x in (t[i-1][1], t[i-2][1], t[i-3][1])) \\\n or (cmd == \"next1or2or3tag\" and x in (t[i+1][1], t[i+2][1], t[i+3][1])) \\\n or (cmd == \"surroundtag\" and x == t[i-1][1] and y == t[i+1][1]) \\\n or (cmd == \"curwd\" and x == t[i+0][0]) \\\n or (cmd == \"prevwd\" and x == t[i-1][0]) \\\n or (cmd == \"nextwd\" and x == t[i+1][0]) \\\n or (cmd == \"prev1or2wd\" and x in (t[i-1][0], t[i-2][0])) \\\n or (cmd == \"next1or2wd\" and x in (t[i+1][0], t[i+2][0])) \\\n or (cmd == \"prevwdtag\" and x == t[i-1][0] and y == t[i-1][1]) \\\n or (cmd == \"nextwdtag\" and x == t[i+1][0] and y == t[i+1][1]) \\\n or (cmd == \"wdprevtag\" and x == t[i-1][1] and y == t[i+0][0]) \\\n or (cmd == \"wdnexttag\" and x == t[i+0][0] and y == t[i+1][1]) \\\n or (cmd == \"wdand2aft\" and x == t[i+0][0] and y == t[i+2][0]) \\\n or (cmd == \"wdand2tagbfr\" and x == t[i-2][1] and y == t[i+0][0]) \\\n or (cmd == \"wdand2tagaft\" and x == t[i+0][0] and y == t[i+2][1]) \\\n or (cmd == \"lbigram\" and x == t[i-1][0] and y == t[i+0][0]) \\\n or (cmd == \"rbigram\" and x == t[i+0][0] and y == t[i+1][0]) \\\n or (cmd == \"prevbigram\" and x == t[i-2][1] and y == t[i-1][1]) \\\n or (cmd == \"nextbigram\" and x == t[i+1][1] and y == t[i+2][1]):\n t[i] = [t[i][0], r[1]]\n return t[len(o):-len(o)]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef insert(self, i, tag1, tag2, cmd=\"prevtag\", x=None, y=None):\n if \" < \" in tag1 and not x and not y:\n tag1, x = tag1.split(\" < \"); cmd=\"prevtag\"\n if \" > \" in tag1 and not x and not y:\n x, tag1 = tag1.split(\" > \"); cmd=\"nexttag\"\n lazylist.insert(self, i, [tag1, tag2, cmd, x or \"\", y or \"\"])", "response": "Inserts a new rule that updates words with tag1 to tag2 given constraints x and y."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\napplies the named entity recognizer to the given list of tokens.", "response": "def apply(self, tokens):\n \"\"\" Applies the named entity recognizer to the given list of tokens,\n where each token is a [word, tag] list.\n \"\"\"\n # Note: we could also scan for patterns, e.g.,\n # \"my|his|her name is|was *\" => NNP-PERS.\n i = 0\n while i < len(tokens):\n w = tokens[i][0].lower()\n if RE_ENTITY1.match(w) \\\n or RE_ENTITY2.match(w) \\\n or RE_ENTITY3.match(w):\n tokens[i][1] = self.tag\n if w in self:\n for e in self[w]:\n # Look ahead to see if successive words match the named entity.\n e, tag = (e[:-1], \"-\"+e[-1].upper()) if e[-1] in self._cmd else (e, \"\")\n b = True\n for j, e in enumerate(e):\n if i + j >= len(tokens) or tokens[i+j][0].lower() != e:\n b = False; break\n if b:\n for token in tokens[i:i+j+1]:\n token[1] = token[1] if token[1].startswith(self.tag) else self.tag\n token[1] += tag\n i += j\n break\n i += 1\n return tokens"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nappend a named entity to the lexicon.", "response": "def append(self, entity, name=\"pers\"):\n \"\"\" Appends a named entity to the lexicon,\n e.g., Entities.append(\"Hooloovoo\", \"PERS\")\n \"\"\"\n e = map(lambda s: s.lower(), entity.split(\" \") + [name])\n self.setdefault(e[0], []).append(e)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a sorted list of keywords in the given string.", "response": "def find_keywords(self, string, **kwargs):\n \"\"\" Returns a sorted list of keywords in the given string.\n \"\"\"\n return find_keywords(string,\n parser = self,\n top = kwargs.pop(\"top\", 10),\n frequency = kwargs.pop(\"frequency\", {}), **kwargs\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_tokens(self, string, **kwargs):\n # \"The cat purs.\" => [\"The cat purs .\"]\n return find_tokens(string,\n punctuation = kwargs.get( \"punctuation\", PUNCTUATION),\n abbreviations = kwargs.get(\"abbreviations\", ABBREVIATIONS),\n replace = kwargs.get( \"replace\", replacements),\n linebreak = r\"\\n{2,}\")", "response": "Returns a list of sentences from the given string."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nannotate the given list of tokens with part - of - speech tags.", "response": "def find_tags(self, tokens, **kwargs):\n \"\"\" Annotates the given list of tokens with part-of-speech tags.\n Returns a list of tokens, where each token is now a [word, tag]-list.\n \"\"\"\n # [\"The\", \"cat\", \"purs\"] => [[\"The\", \"DT\"], [\"cat\", \"NN\"], [\"purs\", \"VB\"]]\n return find_tags(tokens,\n lexicon = kwargs.get( \"lexicon\", self.lexicon or {}),\n model = kwargs.get( \"model\", self.model),\n morphology = kwargs.get(\"morphology\", self.morphology),\n context = kwargs.get( \"context\", self.context),\n entities = kwargs.get( \"entities\", self.entities),\n language = kwargs.get( \"language\", self.language),\n default = kwargs.get( \"default\", self.default),\n map = kwargs.get( \"map\", None))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nannotating the given list of tokens with chunk tags. Several tags can be added for example chunk + preposition tags.", "response": "def find_chunks(self, tokens, **kwargs):\n \"\"\" Annotates the given list of tokens with chunk tags.\n Several tags can be added, for example chunk + preposition tags.\n \"\"\"\n # [[\"The\", \"DT\"], [\"cat\", \"NN\"], [\"purs\", \"VB\"]] =>\n # [[\"The\", \"DT\", \"B-NP\"], [\"cat\", \"NN\", \"I-NP\"], [\"purs\", \"VB\", \"B-VP\"]]\n return find_prepositions(\n find_chunks(tokens,\n language = kwargs.get(\"language\", self.language)))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse(self, s, tokenize=True, tags=True, chunks=True, relations=False, lemmata=False, encoding=\"utf-8\", **kwargs):\n # Tokenizer.\n if tokenize is True:\n s = self.find_tokens(s, **kwargs)\n if isinstance(s, (list, tuple)):\n s = [isinstance(s, basestring) and s.split(\" \") or s for s in s]\n if isinstance(s, basestring):\n s = [s.split(\" \") for s in s.split(\"\\n\")]\n # Unicode.\n for i in range(len(s)):\n for j in range(len(s[i])):\n if isinstance(s[i][j], str):\n s[i][j] = decode_string(s[i][j], encoding)\n # Tagger (required by chunker, labeler & lemmatizer).\n if tags or chunks or relations or lemmata:\n s[i] = self.find_tags(s[i], **kwargs)\n else:\n s[i] = [[w] for w in s[i]]\n # Chunker.\n if chunks or relations:\n s[i] = self.find_chunks(s[i], **kwargs)\n # Labeler.\n if relations:\n s[i] = self.find_labels(s[i], **kwargs)\n # Lemmatizer.\n if lemmata:\n s[i] = self.find_lemmata(s[i], **kwargs)\n # Slash-formatted tagged string.\n # With collapse=False (or split=True), returns raw list\n # (this output is not usable by tree.Text).\n if not kwargs.get(\"collapse\", True) \\\n or kwargs.get(\"split\", False):\n return s\n # Construct TaggedString.format.\n # (this output is usable by tree.Text).\n format = [\"word\"]\n if tags:\n format.append(\"part-of-speech\")\n if chunks:\n format.extend((\"chunk\", \"preposition\"))\n if relations:\n format.append(\"relation\")\n if lemmata:\n format.append(\"lemma\")\n # Collapse raw list.\n # Sentences are separated by newlines, tokens by spaces, tags by slashes.\n # Slashes in words are encoded with &slash;\n for i in range(len(s)):\n for j in range(len(s[i])):\n s[i][j][0] = s[i][j][0].replace(\"/\", \"&slash;\")\n s[i][j] = \"/\".join(s[i][j])\n s[i] = \" \".join(s[i])\n s = \"\\n\".join(s)\n s = TaggedString(s, format, language=kwargs.get(\"language\", self.language))\n return s", "response": "Takes a string and returns a tagged Unicode string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef split(self, sep=TOKENS):\n if sep != TOKENS:\n return unicode.split(self, sep)\n if len(self) == 0:\n return []\n return [[[x.replace(\"&slash;\", \"/\") for x in token.split(\"/\")]\n for token in sentence.split(\" \")]\n for sentence in unicode.split(self, \"\\n\")]", "response": "Returns a list of sentences where each sentence is a list of tokens where each token is a list of word + tags."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef TENSES(self):\n a = set(TENSES[id] for id in self._format)\n a = a.union(set(TENSES[id] for id in self._default.keys()))\n a = a.union(set(TENSES[id] for id in self._default.values()))\n a = sorted(x[:-2] for x in a if x[-2] is False) # Exclude negation.\n return a", "response": "Yields a list of tenses for this language excluding negations."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the infinitive form of the given verb or None if no such verb exists.", "response": "def lemma(self, verb, parse=True):\n \"\"\" Returns the infinitive form of the given verb, or None.\n \"\"\"\n if dict.__len__(self) == 0:\n self.load()\n if verb.lower() in self._inverse:\n return self._inverse[verb.lower()]\n if verb in self._inverse:\n return self._inverse[verb]\n if parse is True: # rule-based\n return self.find_lemma(verb)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lexeme(self, verb, parse=True):\n a = []\n b = self.lemma(verb, parse=parse)\n if b in self:\n a = [x for x in self[b] if x != \"\"]\n elif parse is True: # rule-based\n a = self.find_lexeme(b)\n u = []; [u.append(x) for x in a if x not in u]\n return u", "response": "Returns a list of all possible inflections of the given verb."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninflect the verb and returns the given tense.", "response": "def conjugate(self, verb, *args, **kwargs):\n \"\"\" Inflects the verb and returns the given tense (or None).\n For example: be\n - Verbs.conjugate(\"is\", INFINITVE) => be\n - Verbs.conjugate(\"be\", PRESENT, 1, SINGULAR) => I am\n - Verbs.conjugate(\"be\", PRESENT, 1, PLURAL) => we are\n - Verbs.conjugate(\"be\", PAST, 3, SINGULAR) => he was\n - Verbs.conjugate(\"be\", PAST, aspect=PROGRESSIVE) => been\n - Verbs.conjugate(\"be\", PAST, person=1, negated=True) => I wasn't\n \"\"\"\n id = tense_id(*args, **kwargs)\n # Get the tense index from the format description (or a default).\n i1 = self._format.get(id)\n i2 = self._format.get(self._default.get(id))\n i3 = self._format.get(self._default.get(self._default.get(id)))\n b = self.lemma(verb, parse=kwargs.get(\"parse\", True))\n v = []\n # Get the verb lexeme and return the requested index.\n if b in self:\n v = self[b]\n for i in (i1, i2, i3):\n if i is not None and 0 <= i < len(v) and v[i]:\n return v[i]\n if kwargs.get(\"parse\", True) is True: # rule-based\n v = self.find_lexeme(b)\n for i in (i1, i2, i3):\n if i is not None and 0 <= i < len(v) and v[i]:\n return v[i]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of possible tenses for the given verb.", "response": "def tenses(self, verb, parse=True):\n \"\"\" Returns a list of possible tenses for the given inflected verb.\n \"\"\"\n verb = verb.lower()\n a = set()\n b = self.lemma(verb, parse=parse)\n v = []\n if b in self:\n v = self[b]\n elif parse is True: # rule-based\n v = self.find_lexeme(b)\n # For each tense in the verb lexeme that matches the given tense,\n # 1) retrieve the tense tuple,\n # 2) retrieve the tense tuples for which that tense is a default.\n for i, tense in enumerate(v):\n if tense == verb:\n for id, index in self._format.items():\n if i == index:\n a.add(id)\n for id1, id2 in self._default.items():\n if id2 in a:\n a.add(id1)\n for id1, id2 in self._default.items():\n if id2 in a:\n a.add(id1)\n a = (TENSES[id][:-2] for id in a)\n a = Tenses(sorted(a))\n return a"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load(self, path=None):\n # \n # \n if not path:\n path = self._path\n if not os.path.exists(path):\n return\n words, synsets, labels = {}, {}, {}\n xml = cElementTree.parse(path)\n xml = xml.getroot()\n for w in xml.findall(\"word\"):\n if self._confidence is None \\\n or self._confidence <= float(w.attrib.get(\"confidence\", 0.0)):\n w, pos, p, s, i, label, synset = (\n w.attrib.get(\"form\"),\n w.attrib.get(\"pos\"),\n w.attrib.get(\"polarity\", 0.0),\n w.attrib.get(\"subjectivity\", 0.0),\n w.attrib.get(\"intensity\", 1.0),\n w.attrib.get(\"label\"),\n w.attrib.get(self._synset) # wordnet_id, cornetto_id, ...\n )\n psi = (float(p), float(s), float(i))\n if w:\n words.setdefault(w, {}).setdefault(pos, []).append(psi)\n if w and label:\n labels[w] = label\n if synset:\n synsets.setdefault(synset, []).append(psi)\n self._language = xml.attrib.get(\"language\", self._language)\n # Average scores of all word senses per part-of-speech tag.\n for w in words:\n words[w] = dict((pos, [avg(each) for each in zip(*psi)]) for pos, psi in words[w].items())\n # Average scores of all part-of-speech tags.\n for w, pos in list(words.items()):\n words[w][None] = [avg(each) for each in zip(*pos.values())]\n # Average scores of all synonyms per synset.\n for id, psi in synsets.items():\n synsets[id] = [avg(each) for each in zip(*psi)]\n dict.update(self, words)\n dict.update(self.labeler, labels)\n dict.update(self._synsets, synsets)", "response": "Loads the Sentiment. xml file with sentiment annotations."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a tuple of polarity subjectivity - tuple for the given synset id.", "response": "def synset(self, id, pos=ADJECTIVE):\n \"\"\" Returns a (polarity, subjectivity)-tuple for the given synset id.\n For example, the adjective \"horrible\" has id 193480 in WordNet:\n Sentiment.synset(193480, pos=\"JJ\") => (-0.6, 1.0, 1.0).\n \"\"\"\n id = str(id).zfill(8)\n if not id.startswith((\"n-\", \"v-\", \"a-\", \"r-\")):\n if pos == NOUN:\n id = \"n-\" + id\n if pos == VERB:\n id = \"v-\" + id\n if pos == ADJECTIVE:\n id = \"a-\" + id\n if pos == ADVERB:\n id = \"r-\" + id\n if dict.__len__(self) == 0:\n self.load()\n try:\n return tuple(self._synsets[id])[:2]\n except KeyError: # Some WordNet id's are not zero padded.\n return tuple(self._synsets.get(re.sub(r\"-0+\", \"-\", id), (0.0, 0.0))[:2])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef assessments(self, words=[], negation=True):\n a = []\n m = None # Preceding modifier (i.e., adverb or adjective).\n n = None # Preceding negation (e.g., \"not beautiful\").\n for w, pos in words:\n # Only assess known words, preferably by part-of-speech tag.\n # Including unknown words (polarity 0.0 and subjectivity 0.0) lowers the average.\n if w is None:\n continue\n if w in self and pos in self[w]:\n p, s, i = self[w][pos]\n # Known word not preceded by a modifier (\"good\").\n if m is None:\n a.append(dict(w=[w], p=p, s=s, i=i, n=1, x=self.labeler.get(w)))\n # Known word preceded by a modifier (\"really good\").\n if m is not None:\n a[-1][\"w\"].append(w)\n a[-1][\"p\"] = max(-1.0, min(p * a[-1][\"i\"], +1.0))\n a[-1][\"s\"] = max(-1.0, min(s * a[-1][\"i\"], +1.0))\n a[-1][\"i\"] = i\n a[-1][\"x\"] = self.labeler.get(w)\n # Known word preceded by a negation (\"not really good\").\n if n is not None:\n a[-1][\"w\"].insert(0, n)\n a[-1][\"i\"] = 1.0 / a[-1][\"i\"]\n a[-1][\"n\"] = -1\n # Known word may be a negation.\n # Known word may be modifying the next word (i.e., it is a known adverb).\n m = None\n n = None\n if pos and pos in self.modifiers or any(map(self[w].__contains__, self.modifiers)):\n m = (w, pos)\n if negation and w in self.negations:\n n = w\n else:\n # Unknown word may be a negation (\"not good\").\n if negation and w in self.negations:\n n = w\n # Unknown word. Retain negation across small words (\"not a good\").\n elif n and len(w.strip(\"'\")) > 1:\n n = None\n # Unknown word may be a negation preceded by a modifier (\"really not good\").\n if n is not None and m is not None and (pos in self.modifiers or self.modifier(m[0])):\n a[-1][\"w\"].append(n)\n a[-1][\"n\"] = -1\n n = None\n # Unknown word. Retain modifier across small words (\"really is a good\").\n elif m and len(w) > 2:\n m = None\n # Exclamation marks boost previous word.\n if w == \"!\" and len(a) > 0:\n a[-1][\"w\"].append(\"!\")\n a[-1][\"p\"] = max(-1.0, min(a[-1][\"p\"] * 1.25, +1.0))\n # Exclamation marks in parentheses indicate sarcasm.\n if w == \"(!)\":\n a.append(dict(w=[w], p=0.0, s=1.0, i=1.0, n=1, x=IRONY))\n # EMOTICONS: {(\"grin\", +1.0): set((\":-D\", \":D\"))}\n if w.isalpha() is False and len(w) <= 5 and w not in PUNCTUATION: # speedup\n for (type, p), e in EMOTICONS.items():\n if w in map(lambda e: e.lower(), e):\n a.append(dict(w=[w], p=p, s=1.0, i=1.0, n=1, x=MOOD))\n break\n for i in range(len(a)):\n w = a[i][\"w\"]\n p = a[i][\"p\"]\n s = a[i][\"s\"]\n n = a[i][\"n\"]\n x = a[i][\"x\"]\n # \"not good\" = slightly bad, \"not bad\" = slightly good.\n a[i] = (w, p * -0.5 if n < 0 else p, s, x)\n return a", "response": "Returns a list of tuples for the given list of words."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef annotate(self, word, pos=None, polarity=0.0, subjectivity=0.0, intensity=1.0, label=None):\n w = self.setdefault(word, {})\n w[pos] = w[None] = (polarity, subjectivity, intensity)\n if label:\n self.labeler[word] = label", "response": "Annotates the given word with polarity subjectivity and intensity scores."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef train(self, s, path=\"spelling.txt\"):\n model = {}\n for w in re.findall(\"[a-z]+\", s.lower()):\n model[w] = w in model and model[w] + 1 or 1\n model = (\"%s %s\" % (k, v) for k, v in sorted(model.items()))\n model = \"\\n\".join(model)\n f = open(path, \"w\")\n f.write(model)\n f.close()", "response": "Counts the words in the given string and saves the probabilities at the given path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a set of words with edit distance 1 from the given word.", "response": "def _edit1(self, w):\n \"\"\" Returns a set of words with edit distance 1 from the given word.\n \"\"\"\n # Of all spelling errors, 80% is covered by edit distance 1.\n # Edit distance 1 = one character deleted, swapped, replaced or inserted.\n split = [(w[:i], w[i:]) for i in range(len(w) + 1)]\n delete, transpose, replace, insert = (\n [a + b[1:] for a, b in split if b],\n [a + b[1] + b[0] + b[2:] for a, b in split if len(b) > 1],\n [a + c + b[1:] for a, b in split for c in Spelling.ALPHA if b],\n [a + c + b[0:] for a, b in split for c in Spelling.ALPHA]\n )\n return set(delete + transpose + replace + insert)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a set of words with edit distance 2 from the given word.", "response": "def _edit2(self, w):\n \"\"\" Returns a set of words with edit distance 2 from the given word\n \"\"\"\n # Of all spelling errors, 99% is covered by edit distance 2.\n # Only keep candidates that are actually known words (20% speedup).\n return set(e2 for e1 in self._edit1(w) for e2 in self._edit1(e1) if e2 in self)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef suggest(self, w):\n if len(self) == 0:\n self.load()\n if len(w) == 1:\n return [(w, 1.0)] # I\n if w in PUNCTUATION:\n return [(w, 1.0)] # .?!\n if w.replace(\".\", \"\").isdigit():\n return [(w, 1.0)] # 1.5\n candidates = self._known([w]) \\\n or self._known(self._edit1(w)) \\\n or self._known(self._edit2(w)) \\\n or [w]\n candidates = [(self.get(c, 0.0), c) for c in candidates]\n s = float(sum(p for p, w in candidates) or 1)\n candidates = sorted(((p / s, w) for p, w in candidates), reverse=True)\n candidates = [(w.istitle() and x.title() or x, p) for p, x in candidates] # case-sensitive\n return candidates", "response": "Return a list of ( word confidence ) spelling corrections for the given word."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of tuples where the i - th tuple contains the i - th element from each of the argument sequences or iterables.", "response": "def zip(*args, **kwargs):\n \"\"\" Returns a list of tuples, where the i-th tuple contains the i-th element \n from each of the argument sequences or iterables (or default if too short).\n \"\"\"\n args = [list(iterable) for iterable in args]\n n = max(map(len, args))\n v = kwargs.get(\"default\", None)\n return _zip(*[i + [v] * (n - len(i)) for i in args])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef chunked(sentence):\n # For example, to construct a training vector with the head of previous chunks as a feature.\n # Doing this with Sentence.chunks would discard the punctuation marks and conjunctions\n # (Sentence.chunks only yields Chunk objects), which amy be useful features.\n chunks = []\n for word in sentence:\n if word.chunk is not None:\n if len(chunks) == 0 or chunks[-1] != word.chunk:\n chunks.append(word.chunk)\n else:\n ch = Chink(sentence)\n ch.append(word.copy(ch))\n chunks.append(ch)\n return chunks", "response": "Returns a list of Chunk and Chink objects from the given sentence."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tree(string, token=[WORD, POS, CHUNK, PNP, REL, ANCHOR, LEMMA]):\n return Text(string, token)", "response": "Transforms the output of parse into a Text object."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the string with XML - safe special characters.", "response": "def xml_encode(string):\n \"\"\" Returns the string with XML-safe special characters.\n \"\"\"\n string = string.replace(\"&\", \"&\")\n string = string.replace(\"<\", \"<\")\n string = string.replace(\">\", \">\")\n string = string.replace(\"\\\"\",\""\")\n string = string.replace(SLASH, \"/\")\n return string"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xml_decode(string):\n string = string.replace(\"&\", \"&\")\n string = string.replace(\"<\", \"<\")\n string = string.replace(\">\", \">\")\n string = string.replace(\""\",\"\\\"\")\n string = string.replace(\"/\", SLASH)\n return string", "response": "Returns the string with special characters decoded."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_xml(sentence, tab=\"\\t\", id=\"\"):\n uid = lambda *parts: \"\".join([str(id), _UID_SEPARATOR ]+[str(x) for x in parts]).lstrip(_UID_SEPARATOR)\n push = lambda indent: indent+tab # push() increases the indentation.\n pop = lambda indent: indent[:-len(tab)] # pop() decreases the indentation.\n indent = tab\n xml = []\n # Start the sentence element:\n # \n xml.append('<%s%s %s=\"%s\" %s=\"%s\">' % (\n XML_SENTENCE,\n XML_ID and \" %s=\\\"%s\\\"\" % (XML_ID, str(id)) or \"\",\n XML_TOKEN, \", \".join(sentence.token),\n XML_LANGUAGE, sentence.language\n ))\n # Collect chunks that are PNP anchors and assign id.\n anchors = {}\n for chunk in sentence.chunks:\n if chunk.attachments:\n anchors[chunk.start] = len(anchors) + 1\n # Traverse all words in the sentence.\n for word in sentence.words:\n chunk = word.chunk\n pnp = word.chunk and word.chunk.pnp or None\n # Start the PNP element if the chunk is the first chunk in PNP:\n # \n if pnp and pnp.start == chunk.start:\n a = pnp.anchor and ' %s=\"%s\"' % (XML_OF, uid(\"A\", anchors.get(pnp.anchor.start, \"\"))) or \"\"\n xml.append(indent + '<%s %s=\"PNP\"%s>' % (XML_CHUNK, XML_TYPE, a))\n indent = push(indent)\n # Start the chunk element if the word is the first word in the chunk:\n # \n if chunk and chunk.start == word.index:\n if chunk.relations:\n # Create the shortest possible attribute values for multiple relations, \n # e.g., [(1,\"OBJ\"),(2,\"OBJ\")]) => relation=\"OBJ\" id=\"1|2\"\n r1 = unzip(0, chunk.relations) # Relation id's.\n r2 = unzip(1, chunk.relations) # Relation roles.\n r1 = [x is None and \"-\" or uid(x) for x in r1]\n r2 = [x is None and \"-\" or x for x in r2]\n r1 = not len(unique(r1)) == 1 and \"|\".join(r1) or (r1+[None])[0]\n r2 = not len(unique(r2)) == 1 and \"|\".join(r2) or (r2+[None])[0]\n xml.append(indent + '<%s%s%s%s%s%s>' % (\n XML_CHUNK,\n chunk.type and ' %s=\"%s\"' % (XML_TYPE, chunk.type) or \"\",\n chunk.relations and chunk.role != None and ' %s=\"%s\"' % (XML_RELATION, r2) or \"\",\n chunk.relation and chunk.type == \"VP\" and ' %s=\"%s\"' % (XML_ID, uid(chunk.relation)) or \"\",\n chunk.relation and chunk.type != \"VP\" and ' %s=\"%s\"' % (XML_OF, r1) or \"\",\n chunk.attachments and ' %s=\"%s\"' % (XML_ANCHOR, uid(\"A\",anchors[chunk.start])) or \"\"\n ))\n indent = push(indent)\n # Words outside of a chunk are wrapped in a tag:\n # \n if not chunk:\n xml.append(indent + '<%s>' % XML_CHINK)\n indent = push(indent)\n # Add the word element:\n # eat\n xml.append(indent + '<%s%s%s%s>%s' % (\n XML_WORD,\n word.type and ' %s=\"%s\"' % (XML_TYPE, xml_encode(word.type)) or '',\n word.lemma and ' %s=\"%s\"' % (XML_LEMMA, xml_encode(word.lemma)) or '',\n (\" \"+\" \".join(['%s=\"%s\"' % (k,v) for k,v in word.custom_tags.items() if v != None])).rstrip(),\n xml_encode(unicode(word)),\n XML_WORD\n ))\n if not chunk:\n # Close the element if outside of a chunk.\n indent = pop(indent); xml.append(indent + \"\" % XML_CHINK)\n if chunk and chunk.stop-1 == word.index:\n # Close the element if this is the last word in the chunk.\n indent = pop(indent); xml.append(indent + \"\" % XML_CHUNK)\n if pnp and pnp.stop-1 == word.index:\n # Close the PNP element if this is the last word in the PNP.\n indent = pop(indent); xml.append(indent + \"\" % XML_CHUNK)\n xml.append(\"\" % XML_SENTENCE)\n # Return as a plain str.\n return \"\\n\".join(xml).encode(\"utf-8\")", "response": "Parses the given sentence as an XML - string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_string(xml):\n string = \"\"\n # Traverse all the elements in the XML.\n dom = XML(xml)\n for sentence in dom(XML_SENTENCE):\n _anchors.clear() # Populated by calling _parse_tokens().\n _attachments.clear() # Populated by calling _parse_tokens().\n # Parse the language from .\n language = sentence.get(XML_LANGUAGE, \"en\")\n # Parse the token tag format from .\n # This information is returned in TokenString.tags,\n # so the format and order of the token tags is retained when exporting/importing as XML.\n format = sentence.get(XML_TOKEN, [WORD, POS, CHUNK, PNP, REL, ANCHOR, LEMMA])\n format = not isinstance(format, basestring) and format or format.replace(\" \",\"\").split(\",\")\n # Traverse all and elements in the sentence.\n # Find the elements inside and create tokens.\n tokens = []\n for chunk in sentence:\n tokens.extend(_parse_tokens(chunk, format))\n # Attach PNP's to their anchors.\n # Keys in _anchors have linked anchor chunks (each chunk is a list of tokens).\n # The keys correspond to the keys in _attachments, which have linked PNP chunks.\n if ANCHOR in format:\n A, P, a, i = _anchors, _attachments, 1, format.index(ANCHOR)\n for id in sorted(A.keys()):\n for token in A[id]:\n token[i] += \"-\"+\"-\".join([\"A\"+str(a+p) for p in range(len(P[id]))])\n token[i] = token[i].strip(\"O-\")\n for p, pnp in enumerate(P[id]):\n for token in pnp: \n token[i] += \"-\"+\"P\"+str(a+p)\n token[i] = token[i].strip(\"O-\")\n a += len(P[id])\n # Collapse the tokens to string.\n # Separate multiple sentences with a new line.\n tokens = [\"/\".join([tag for tag in token]) for token in tokens]\n tokens = \" \".join(tokens)\n string += tokens + \"\\n\"\n # Return a TokenString, which is a unicode string that transforms easily\n # into a plain str, a list of tokens, or a Sentence.\n try:\n if MBSP: from mbsp import TokenString\n return TokenString(string.strip(), tags=format, language=language)\n except:\n return TaggedString(string.strip(), tags=format, language=language)", "response": "Parses the given XML representation into a TokenString or TaggedString."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_tokens(chunk, format=[WORD, POS, CHUNK, PNP, REL, ANCHOR, LEMMA]):\n tokens = []\n # Only process and elements, \n # text nodes in between return an empty list.\n if not (chunk.tag == XML_CHUNK or chunk.tag == XML_CHINK):\n return []\n type = chunk.get(XML_TYPE, \"O\")\n if type == \"PNP\":\n # For, , recurse all the child chunks inside the PNP.\n for ch in chunk:\n tokens.extend(_parse_tokens(ch, format))\n # Tag each of them as part of the PNP.\n if PNP in format:\n i = format.index(PNP)\n for j, token in enumerate(tokens):\n token[i] = (j==0 and \"B-\" or \"I-\") + \"PNP\"\n # Store attachments so we can construct anchor id's in parse_string().\n # This has to be done at the end, when all the chunks have been found.\n a = chunk.get(XML_OF).split(_UID_SEPARATOR)[-1]\n if a:\n _attachments.setdefault(a, [])\n _attachments[a].append(tokens)\n return tokens\n # For , the relation is VP-1.\n # For , the relation is NP-OBJ-1.\n relation = _parse_relation(chunk, type)\n # Process all of the elements in the chunk, for example:\n # pizza => [pizza, NN, I-NP, O, NP-OBJ-1, O, pizza]\n for word in filter(lambda n: n.tag == XML_WORD, chunk):\n tokens.append(_parse_token(word, chunk=type, relation=relation, format=format))\n # Add the IOB chunk tags:\n # words at the start of a chunk are marked with B-, words inside with I-.\n if CHUNK in format:\n i = format.index(CHUNK)\n for j, token in enumerate(tokens):\n token[i] = token[i] != \"O\" and ((j==0 and \"B-\" or \"I-\") + token[i]) or \"O\"\n # The chunk can be the anchor of one or more PNP chunks.\n # Store anchors so we can construct anchor id's in parse_string().\n a = chunk.get(XML_ANCHOR, \"\").split(_UID_SEPARATOR)[-1]\n if a: \n _anchors[a] = tokens\n return tokens", "response": "Parses a list of tokens from the given XML chunk element."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the relation string from the given chunk element.", "response": "def _parse_relation(chunk, type=\"O\"):\n \"\"\" Returns a string of the roles and relations parsed from the given element.\n The chunk type (which is part of the relation string) can be given as parameter.\n \"\"\"\n r1 = chunk.get(XML_RELATION)\n r2 = chunk.get(XML_ID, chunk.get(XML_OF))\n r1 = [x != \"-\" and x or None for x in r1.split(\"|\")] or [None]\n r2 = [x != \"-\" and x or None for x in r2.split(\"|\")] or [None]\n r2 = [x is not None and x.split(_UID_SEPARATOR )[-1] or x for x in r2]\n if len(r1) < len(r2): r1 = r1 + r1 * (len(r2)-len(r1)) # [1] [\"SBJ\", \"OBJ\"] => \"SBJ-1;OBJ-1\"\n if len(r2) < len(r1): r2 = r2 + r2 * (len(r1)-len(r2)) # [2,4] [\"OBJ\"] => \"OBJ-2;OBJ-4\"\n return \";\".join([\"-\".join([x for x in (type, r1, r2) if x]) for r1, r2 in zip(r1, r2)])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses a word element into a list of token tags.", "response": "def _parse_token(word, chunk=\"O\", pnp=\"O\", relation=\"O\", anchor=\"O\", \n format=[WORD, POS, CHUNK, PNP, REL, ANCHOR, LEMMA]):\n \"\"\" Returns a list of token tags parsed from the given element.\n Tags that are not attributes in a (e.g., relation) can be given as parameters.\n \"\"\"\n tags = []\n for tag in format:\n if tag == WORD : tags.append(xml_decode(word.value))\n elif tag == POS : tags.append(xml_decode(word.get(XML_TYPE, \"O\")))\n elif tag == CHUNK : tags.append(chunk)\n elif tag == PNP : tags.append(pnp)\n elif tag == REL : tags.append(relation)\n elif tag == ANCHOR : tags.append(anchor)\n elif tag == LEMMA : tags.append(xml_decode(word.get(XML_LEMMA, \"\")))\n else:\n # Custom tags when the parser has been extended, see also Word.custom_tags{}.\n tags.append(xml_decode(word.get(tag, \"O\")))\n return tags"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef nltk_tree(sentence):\n from nltk import tree\n def do_pnp(pnp):\n # Returns the PNPChunk (and the contained Chunk objects) in NLTK bracket format.\n s = ' '.join([do_chunk(ch) for ch in pnp.chunks])\n return '(PNP %s)' % s\n \n def do_chunk(ch):\n # Returns the Chunk in NLTK bracket format. Recurse attached PNP's.\n s = ' '.join(['(%s %s)' % (w.pos, w.string) for w in ch.words])\n s+= ' '.join([do_pnp(pnp) for pnp in ch.attachments])\n return '(%s %s)' % (ch.type, s)\n \n T = ['(S']\n v = [] # PNP's already visited.\n for ch in sentence.chunked():\n if not ch.pnp and isinstance(ch, Chink):\n T.append('(%s %s)' % (ch.words[0].pos, ch.words[0].string))\n elif not ch.pnp:\n T.append(do_chunk(ch))\n #elif ch.pnp not in v:\n elif ch.pnp.anchor is None and ch.pnp not in v:\n # The chunk is part of a PNP without an anchor.\n T.append(do_pnp(ch.pnp))\n v.append(ch.pnp)\n T.append(')')\n return tree.bracket_parse(' '.join(T))", "response": "Returns an NLTK nltk. tree. Tree object from the given Sentence."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a DOT - formatted string that can be visualized as a graph in GraphViz.", "response": "def graphviz_dot(sentence, font=\"Arial\", colors=BLUE):\n \"\"\" Returns a dot-formatted string that can be visualized as a graph in GraphViz.\n \"\"\"\n s = 'digraph sentence {\\n'\n s += '\\tranksep=0.75;\\n'\n s += '\\tnodesep=0.15;\\n'\n s += '\\tnode [penwidth=1, fontname=\"%s\", shape=record, margin=0.1, height=0.35];\\n' % font\n s += '\\tedge [penwidth=1];\\n'\n s += '\\t{ rank=same;\\n'\n # Create node groups for words, chunks and PNP chunks.\n for w in sentence.words:\n s += '\\t\\tword%s [label=\"%s|%s\"%s];\\n' % (w.index, w.string, w.type, _colorize(w, colors))\n for w in sentence.words[:-1]:\n # Invisible edges forces the words into the right order:\n s += '\\t\\tword%s -> word%s [color=none];\\n' % (w.index, w.index+1)\n s += '\\t}\\n'\n s += '\\t{ rank=same;\\n' \n for i, ch in enumerate(sentence.chunks):\n s += '\\t\\tchunk%s [label=\"%s\"%s];\\n' % (i+1, \"-\".join([x for x in (\n ch.type, ch.role, str(ch.relation or '')) if x]) or '-', _colorize(ch, colors))\n for i, ch in enumerate(sentence.chunks[:-1]):\n # Invisible edges forces the chunks into the right order:\n s += '\\t\\tchunk%s -> chunk%s [color=none];\\n' % (i+1, i+2)\n s += '}\\n'\n s += '\\t{ rank=same;\\n'\n for i, ch in enumerate(sentence.pnp):\n s += '\\t\\tpnp%s [label=\"PNP\"%s];\\n' % (i+1, _colorize(ch, colors))\n s += '\\t}\\n'\n s += '\\t{ rank=same;\\n S [shape=circle, margin=0.25, penwidth=2]; }\\n'\n # Connect words to chunks.\n # Connect chunks to PNP or S.\n for i, ch in enumerate(sentence.chunks):\n for w in ch:\n s += '\\tword%s -> chunk%s;\\n' % (w.index, i+1)\n if ch.pnp:\n s += '\\tchunk%s -> pnp%s;\\n' % (i+1, sentence.pnp.index(ch.pnp)+1)\n else:\n s += '\\tchunk%s -> S;\\n' % (i+1)\n if ch.type == 'VP':\n # Indicate related chunks with a dotted\n for r in ch.related:\n s += '\\tchunk%s -> chunk%s [style=dotted, arrowhead=none];\\n' % (\n i+1, sentence.chunks.index(r)+1)\n # Connect PNP to anchor chunk or S.\n for i, ch in enumerate(sentence.pnp):\n if ch.anchor:\n s += '\\tpnp%s -> chunk%s;\\n' % (i+1, sentence.chunks.index(ch.anchor)+1)\n s += '\\tpnp%s -> S [color=none];\\n' % (i+1)\n else:\n s += '\\tpnp%s -> S;\\n' % (i+1)\n s += \"}\"\n return s"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a string where the tags of tokens in the sentence are organized in outlined columns.", "response": "def table(sentence, fill=1, placeholder=\"-\"):\n \"\"\" Returns a string where the tags of tokens in the sentence are organized in outlined columns.\n \"\"\"\n tags = [WORD, POS, IOB, CHUNK, ROLE, REL, PNP, ANCHOR, LEMMA]\n tags += [tag for tag in sentence.token if tag not in tags]\n def format(token, tag):\n # Returns the token tag as a string.\n if tag == WORD : s = token.string\n elif tag == POS : s = token.type\n elif tag == IOB : s = token.chunk and (token.index == token.chunk.start and \"B\" or \"I\")\n elif tag == CHUNK : s = token.chunk and token.chunk.type\n elif tag == ROLE : s = token.chunk and token.chunk.role\n elif tag == REL : s = token.chunk and token.chunk.relation and str(token.chunk.relation)\n elif tag == PNP : s = token.chunk and token.chunk.pnp and token.chunk.pnp.type\n elif tag == ANCHOR : s = token.chunk and token.chunk.anchor_id\n elif tag == LEMMA : s = token.lemma\n else : s = token.custom_tags.get(tag)\n return s or placeholder\n def outline(column, fill=1, padding=3, align=\"left\"):\n # Add spaces to each string in the column so they line out to the highest width.\n n = max([len(x) for x in column]+[fill])\n if align == \"left\" : return [x+\" \"*(n-len(x))+\" \"*padding for x in column]\n if align == \"right\" : return [\" \"*(n-len(x))+x+\" \"*padding for x in column]\n \n # Gather the tags of the tokens in the sentece per column.\n # If the IOB-tag is I-, mark the chunk tag with \"^\".\n # Add the tag names as headers in each column.\n columns = [[format(token, tag) for token in sentence] for tag in tags]\n columns[3] = [columns[3][i]+(iob == \"I\" and \" ^\" or \"\") for i, iob in enumerate(columns[2])]\n del columns[2]\n for i, header in enumerate(['word', 'tag', 'chunk', 'role', 'id', 'pnp', 'anchor', 'lemma']+tags[9:]):\n columns[i].insert(0, \"\")\n columns[i].insert(0, header.upper())\n # The left column (the word itself) is outlined to the right,\n # and has extra spacing so that words across sentences line out nicely below each other.\n for i, column in enumerate(columns):\n columns[i] = outline(column, fill+10*(i==0), align=(\"left\",\"right\")[i==0])\n # Anchor column is useful in MBSP but not in pattern.en.\n if not MBSP:\n del columns[6] \n # Create a string with one row (i.e., one token) per line.\n return \"\\n\".join([\"\".join([x[i] for x in columns]) for i in range(len(columns[0]))])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tags(self):\n # See also. Sentence.__repr__().\n ch, I,O,B = self.chunk, INSIDE+\"-\", OUTSIDE, BEGIN+\"-\"\n tags = [OUTSIDE for i in range(len(self.sentence.token))]\n for i, tag in enumerate(self.sentence.token): # Default: [WORD, POS, CHUNK, PNP, RELATION, ANCHOR, LEMMA]\n if tag == WORD:\n tags[i] = encode_entities(self.string)\n elif tag == POS and self.type:\n tags[i] = self.type\n elif tag == CHUNK and ch and ch.type:\n tags[i] = (self == ch[0] and B or I) + ch.type\n elif tag == PNP and self.pnp:\n tags[i] = (self == self.pnp[0] and B or I) + \"PNP\"\n elif tag == REL and ch and len(ch.relations) > 0:\n tags[i] = [\"-\".join([str(x) for x in [ch.type]+list(reversed(r)) if x]) for r in ch.relations]\n tags[i] = \"*\".join(tags[i])\n elif tag == ANCHOR and ch:\n tags[i] = ch.anchor_id or OUTSIDE\n elif tag == LEMMA:\n tags[i] = encode_entities(self.lemma or \"\")\n elif tag in self.custom_tags:\n tags[i] = self.custom_tags.get(tag) or OUTSIDE\n return tags", "response": "Yields a list of all the token tags as they appeared when the word was parsed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the next word in the sentence with the given type.", "response": "def next(self, type=None):\n \"\"\" Returns the next word in the sentence with the given type.\n \"\"\"\n i = self.index + 1\n s = self.sentence\n while i < len(s):\n if type in (s[i].type, None):\n return s[i]\n i += 1"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the previous word in the sentence with the given type.", "response": "def previous(self, type=None):\n \"\"\" Returns the next previous word in the sentence with the given type.\n \"\"\"\n i = self.index - 1\n s = self.sentence\n while i > 0:\n if type in (s[i].type, None):\n return s[i]\n i -= 1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nyielding the first entry in the list.", "response": "def head(self):\n \"\"\" Yields the head of the chunk (usually, the last word in the chunk).\n \"\"\"\n if self.type == \"NP\" and any(w.type.startswith(\"NNP\") for w in self):\n w = find(lambda w: w.type.startswith(\"NNP\"), reversed(self))\n elif self.type == \"NP\": # \"the cat\" => \"cat\"\n w = find(lambda w: w.type.startswith(\"NN\"), reversed(self))\n elif self.type == \"VP\": # \"is watching\" => \"watching\"\n w = find(lambda w: w.type.startswith(\"VB\"), reversed(self))\n elif self.type == \"PP\": # \"from up on\" => \"from\"\n w = find(lambda w: w.type.startswith((\"IN\", \"PP\")), self)\n elif self.type == \"PNP\": # \"from up on the roof\" => \"roof\"\n w = find(lambda w: w.type.startswith(\"NN\"), reversed(self))\n else:\n w = None\n if w is None:\n w = self[-1]\n return w"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nyields a list of all chunks in the sentence with the same relation id.", "response": "def related(self):\n \"\"\" Yields a list of all chunks in the sentence with the same relation id.\n \"\"\"\n return [ch for ch in self.sentence.chunks \n if ch != self and intersects(unzip(0, ch.relations), unzip(0, self.relations))]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nyield the anchor tag as parsed from the original token.", "response": "def anchor_id(self):\n \"\"\" Yields the anchor tag as parsed from the original token.\n Chunks that are anchors have a tag with an \"A\" prefix (e.g., \"A1\").\n Chunks that are PNP attachmens (or chunks inside a PNP) have \"P\" (e.g., \"P1\").\n Chunks inside a PNP can be both anchor and attachment (e.g., \"P1-A2\"),\n as in: \"clawed/A1 at/P1 mice/P1-A2 in/P2 the/P2 wall/P2\"\n \"\"\"\n id = \"\"\n f = lambda ch: filter(lambda k: self.sentence._anchors[k] == ch, self.sentence._anchors)\n if self.pnp and self.pnp.anchor:\n id += \"-\" + \"-\".join(f(self.pnp))\n if self.anchor:\n id += \"-\" + \"-\".join(f(self))\n if self.attachments:\n id += \"-\" + \"-\".join(f(self))\n return id.strip(\"-\") or None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of all the modifiers that are part of the VP.", "response": "def modifiers(self):\n \"\"\" For verb phrases (VP), yields a list of the nearest adjectives and adverbs.\n \"\"\"\n if self._modifiers is None:\n # Iterate over all the chunks and attach modifiers to their VP-anchor.\n is_modifier = lambda ch: ch.type in (\"ADJP\", \"ADVP\") and ch.relation is None\n for chunk in self.sentence.chunks:\n chunk._modifiers = []\n for chunk in filter(is_modifier, self.sentence.chunks):\n anchor = chunk.nearest(\"VP\")\n if anchor: anchor._modifiers.append(chunk)\n return self._modifiers"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef nearest(self, type=\"VP\"):\n candidate, d = None, len(self.sentence.chunks)\n if isinstance(self, PNPChunk):\n i = self.sentence.chunks.index(self.chunks[0])\n else:\n i = self.sentence.chunks.index(self)\n for j, chunk in enumerate(self.sentence.chunks):\n if chunk.type.startswith(type) and abs(i-j) < d:\n candidate, d = chunk, abs(i-j)\n return candidate", "response": "Returns the nearest chunk in the sentence with the given type."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef next(self, type=None):\n i = self.stop\n s = self.sentence\n while i < len(s):\n if s[i].chunk is not None and type in (s[i].chunk.type, None):\n return s[i].chunk\n i += 1", "response": "Returns the next chunk in the sentence with the given type."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the previous chunk in the sentence with the given type.", "response": "def previous(self, type=None):\n \"\"\" Returns the next previous chunk in the sentence with the given type.\n \"\"\"\n i = self.start - 1\n s = self.sentence\n while i > 0:\n if s[i].chunk is not None and type in (s[i].chunk.type, None):\n return s[i].chunk\n i -= 1"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nappend a new word to the internal list of words.", "response": "def append(self, word, lemma=None, type=None, chunk=None, role=None, relation=None, pnp=None, anchor=None, iob=None, custom={}):\n \"\"\" Appends the next word to the sentence / chunk / preposition.\n For example: Sentence.append(\"clawed\", \"claw\", \"VB\", \"VP\", role=None, relation=1)\n - word : the current word,\n - lemma : the canonical form of the word,\n - type : part-of-speech tag for the word (NN, JJ, ...),\n - chunk : part-of-speech tag for the chunk this word is part of (NP, VP, ...),\n - role : the chunk's grammatical role (SBJ, OBJ, ...),\n - relation : an id shared by other related chunks (e.g., SBJ-1 <=> VP-1),\n - pnp : PNP if this word is in a prepositional noun phrase (B- prefix optional),\n - iob : BEGIN if the word marks the start of a new chunk,\n INSIDE (optional) if the word is part of the previous chunk,\n - custom : a dictionary of (tag, value)-items for user-defined word tags.\n \"\"\"\n self._do_word(word, lemma, type) # Append Word object.\n self._do_chunk(chunk, role, relation, iob) # Append Chunk, or add last word to last chunk.\n self._do_conjunction()\n self._do_relation()\n self._do_pnp(pnp, anchor)\n self._do_anchor(anchor)\n self._do_custom(custom)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing a tagged token representation into a list of arguments.", "response": "def parse_token(self, token, tags=[WORD, POS, CHUNK, PNP, REL, ANCHOR, LEMMA]):\n \"\"\" Returns the arguments for Sentence.append() from a tagged token representation.\n The order in which token tags appear can be specified.\n The default order is (separated by slashes): \n - word, \n - part-of-speech, \n - (IOB-)chunk, \n - (IOB-)preposition, \n - chunk(-relation)(-role), \n - anchor, \n - lemma.\n Examples:\n The/DT/B-NP/O/NP-SBJ-1/O/the\n cats/NNS/I-NP/O/NP-SBJ-1/O/cat\n clawed/VBD/B-VP/O/VP-1/A1/claw\n at/IN/B-PP/B-PNP/PP/P1/at\n the/DT/B-NP/I-PNP/NP/P1/the\n sofa/NN/I-NP/I-PNP/NP/P1/sofa\n ././O/O/O/O/.\n Returns a (word, lemma, type, chunk, role, relation, preposition, anchor, iob, custom)-tuple,\n which can be passed to Sentence.append(): Sentence.append(*Sentence.parse_token(\"cats/NNS/NP\"))\n The custom value is a dictionary of (tag, value)-items of unrecognized tags in the token.\n \"\"\"\n p = { WORD: \"\", \n POS: None, \n IOB: None,\n CHUNK: None,\n PNP: None,\n REL: None,\n ROLE: None,\n ANCHOR: None,\n LEMMA: None }\n # Split the slash-formatted token into separate tags in the given order.\n # Decode &slash; characters (usually in words and lemmata).\n # Assume None for missing tags (except the word itself, which defaults to an empty string).\n custom = {}\n for k, v in izip(tags, token.split(\"/\")):\n if SLASH0 in v:\n v = v.replace(SLASH, \"/\")\n if k not in p:\n custom[k] = None\n if v != OUTSIDE or k == WORD or k == LEMMA: # \"type O negative\" => \"O\" != OUTSIDE.\n (p if k not in custom else custom)[k] = v\n # Split IOB-prefix from the chunk tag:\n # B- marks the start of a new chunk, \n # I- marks inside of a chunk.\n ch = p[CHUNK]\n if ch is not None and ch.startswith((\"B-\", \"I-\")):\n p[IOB], p[CHUNK] = ch[:1], ch[2:] # B-NP\n # Split the role from the relation:\n # NP-SBJ-1 => relation id is 1 and role is SBJ, \n # VP-1 => relation id is 1 with no role.\n # Tokens may be tagged with multiple relations (e.g., NP-OBJ-1*NP-OBJ-3).\n if p[REL] is not None:\n ch, p[REL], p[ROLE] = self._parse_relation(p[REL])\n # Infer a missing chunk tag from the relation tag (e.g., NP-SBJ-1 => NP).\n # For PP relation tags (e.g., PP-CLR-1), the first chunk is PP, the following chunks NP.\n if ch == \"PP\" \\\n and self._previous \\\n and self._previous[REL] == p[REL] \\\n and self._previous[ROLE] == p[ROLE]: \n ch = \"NP\"\n if p[CHUNK] is None and ch != OUTSIDE:\n p[CHUNK] = ch\n self._previous = p\n # Return the tags in the right order for Sentence.append().\n return p[WORD], p[LEMMA], p[POS], p[CHUNK], p[ROLE], p[REL], p[PNP], p[ANCHOR], p[IOB], custom"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing the token relation tag and returns the chunk relation role and relation id.", "response": "def _parse_relation(self, tag):\n \"\"\" Parses the chunk tag, role and relation id from the token relation tag.\n - VP => VP, [], []\n - VP-1 => VP, [1], [None]\n - ADJP-PRD => ADJP, [None], [PRD]\n - NP-SBJ-1 => NP, [1], [SBJ]\n - NP-OBJ-1*NP-OBJ-2 => NP, [1,2], [OBJ,OBJ]\n - NP-SBJ;NP-OBJ-1 => NP, [1,1], [SBJ,OBJ]\n \"\"\"\n chunk, relation, role = None, [], []\n if \";\" in tag:\n # NP-SBJ;NP-OBJ-1 => 1 relates to both SBJ and OBJ.\n id = tag.split(\"*\")[0][-2:]\n id = id if id.startswith(\"-\") else \"\"\n tag = tag.replace(\";\", id + \"*\")\n if \"*\" in tag:\n tag = tag.split(\"*\")\n else:\n tag = [tag]\n for s in tag:\n s = s.split(\"-\")\n n = len(s)\n if n == 1: \n chunk = s[0]\n if n == 2: \n chunk = s[0]; relation.append(s[1]); role.append(None)\n if n >= 3: \n chunk = s[0]; relation.append(s[2]); role.append(s[1])\n if n > 1:\n id = relation[-1]\n if id.isdigit():\n relation[-1] = int(id)\n else:\n # Correct \"ADJP-PRD\":\n # (ADJP, [PRD], [None]) => (ADJP, [None], [PRD])\n relation[-1], role[-1] = None, id\n return chunk, relation, role"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a new Word to the sentence.", "response": "def _do_word(self, word, lemma=None, type=None):\n \"\"\" Adds a new Word to the sentence.\n Other Sentence._do_[tag] functions assume a new word has just been appended.\n \"\"\"\n # Improve 3rd person singular \"'s\" lemma to \"be\", e.g., as in \"he's fine\".\n if lemma == \"'s\" and type in (\"VB\", \"VBZ\"):\n lemma = \"be\"\n self.words.append(Word(self, word, lemma, type, index=len(self.words)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _do_chunk(self, type, role=None, relation=None, iob=None):\n if (type is None or type == OUTSIDE) and \\\n (role is None or role == OUTSIDE) and (relation is None or relation == OUTSIDE):\n return\n if iob != BEGIN \\\n and self.chunks \\\n and self.chunks[-1].type == type \\\n and self._relation == (relation, role) \\\n and self.words[-2].chunk is not None: # \"one, two\" => \"one\" & \"two\" different chunks.\n self.chunks[-1].append(self.words[-1])\n else:\n ch = Chunk(self, [self.words[-1]], type, role, relation)\n self.chunks.append(ch)\n self._relation = (relation, role)", "response": "Add a new Chunk to the sentence or adds the last word to the previous chunk."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _do_relation(self):\n if self.chunks:\n ch = self.chunks[-1]\n for relation, role in ch.relations:\n if role == \"SBJ\" or role == \"OBJ\":\n self.relations[role][relation] = ch\n if ch.type in (\"VP\",):\n self.relations[ch.type][ch.relation] = ch", "response": "Attaches subjects objects and verbs to the last chunk of the list of related resources."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nattaching prepositional noun phrases. Identifies PNP's from either the PNP tag or the P-attachment tag. This does not determine the PP-anchor, it only groups words in a PNP chunk.", "response": "def _do_pnp(self, pnp, anchor=None):\n \"\"\" Attaches prepositional noun phrases.\n Identifies PNP's from either the PNP tag or the P-attachment tag.\n This does not determine the PP-anchor, it only groups words in a PNP chunk.\n \"\"\"\n if anchor or pnp and pnp.endswith(\"PNP\"):\n if anchor is not None:\n m = find(lambda x: x.startswith(\"P\"), anchor)\n else:\n m = None\n if self.pnp \\\n and pnp \\\n and pnp != OUTSIDE \\\n and pnp.startswith(\"B-\") is False \\\n and self.words[-2].pnp is not None:\n self.pnp[-1].append(self.words[-1])\n elif m is not None and m == self._attachment:\n self.pnp[-1].append(self.words[-1])\n else:\n ch = PNPChunk(self, [self.words[-1]], type=\"PNP\")\n self.pnp.append(ch) \n self._attachment = m"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _do_anchor(self, anchor):\n if anchor:\n for x in anchor.split(\"-\"):\n A, P = None, None\n if x.startswith(\"A\") and len(self.chunks) > 0: # anchor\n A, P = x, x.replace(\"A\",\"P\")\n self._anchors[A] = self.chunks[-1]\n if x.startswith(\"P\") and len(self.pnp) > 0: # attachment (PNP)\n A, P = x.replace(\"P\",\"A\"), x\n self._anchors[P] = self.pnp[-1]\n if A in self._anchors and P in self._anchors and not self._anchors[P].anchor:\n pnp = self._anchors[P]\n pnp.anchor = self._anchors[A]\n pnp.anchor.attachments.append(pnp)", "response": "Collects anchors and attachments in a dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _do_conjunction(self, _and=(\"and\", \"e\", \"en\", \"et\", \"und\", \"y\")):\n w = self.words\n if len(w) > 2 and w[-2].type == \"CC\" and w[-2].chunk is None:\n cc = w[-2].string.lower() in _and and AND or OR\n ch1 = w[-3].chunk\n ch2 = w[-1].chunk\n if ch1 is not None and \\\n ch2 is not None:\n ch1.conjunctions.append(ch2, cc)\n ch2.conjunctions.append(ch1, cc)", "response": "Add a conjunction to the list of possible conjunctions."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, index, tag=LEMMA):\n if tag == WORD:\n return self.words[index]\n if tag == LEMMA:\n return self.words[index].lemma\n if tag == POS:\n return self.words[index].type\n if tag == CHUNK:\n return self.words[index].chunk\n if tag == PNP:\n return self.words[index].pnp\n if tag == REL:\n ch = self.words[index].chunk; return ch and ch.relation\n if tag == ROLE:\n ch = self.words[index].chunk; return ch and ch.role\n if tag == ANCHOR:\n ch = self.words[index].pnp; return ch and ch.anchor\n if tag in self.words[index].custom_tags:\n return self.words[index].custom_tags[tag]\n return None", "response": "Returns a tag for the word at the given index."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nyields the words in the Sentence and lemmata.", "response": "def loop(self, *tags):\n \"\"\" Iterates over the tags in the entire Sentence,\n For example, Sentence.loop(POS, LEMMA) yields tuples of the part-of-speech tags and lemmata. \n Possible tags: WORD, LEMMA, POS, CHUNK, PNP, RELATION, ROLE, ANCHOR or a custom word tag.\n Any order or combination of tags can be supplied.\n \"\"\"\n for i in range(len(self.words)):\n yield tuple([self.get(i, tag=tag) for tag in tags])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef indexof(self, value, tag=WORD):\n match = lambda a, b: a.endswith(\"*\") and b.startswith(a[:-1]) or a==b\n indices = []\n for i in range(len(self.words)):\n if match(value, unicode(self.get(i, tag))):\n indices.append(i)\n return indices", "response": "Returns the indices of the words in the sentence where the given token tag equals the string."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a portion of the sentence from word start index to word stop index.", "response": "def slice(self, start, stop):\n \"\"\" Returns a portion of the sentence from word start index to word stop index.\n The returned slice is a subclass of Sentence and a deep copy.\n \"\"\"\n s = Slice(token=self.token, language=self.language)\n for i, word in enumerate(self.words[start:stop]):\n # The easiest way to copy (part of) a sentence\n # is by unpacking all of the token tags and passing them to Sentence.append().\n p0 = word.string # WORD\n p1 = word.lemma # LEMMA\n p2 = word.type # POS\n p3 = word.chunk is not None and word.chunk.type or None # CHUNK\n p4 = word.pnp is not None and \"PNP\" or None # PNP\n p5 = word.chunk is not None and unzip(0, word.chunk.relations) or None # REL \n p6 = word.chunk is not None and unzip(1, word.chunk.relations) or None # ROLE\n p7 = word.chunk and word.chunk.anchor_id or None # ANCHOR\n p8 = word.chunk and word.chunk.start == start+i and BEGIN or None # IOB\n p9 = word.custom_tags # User-defined tags.\n # If the given range does not contain the chunk head, remove the chunk tags.\n if word.chunk is not None and (word.chunk.stop > stop):\n p3, p4, p5, p6, p7, p8 = None, None, None, None, None, None\n # If the word starts the preposition, add the IOB B-prefix (i.e., B-PNP).\n if word.pnp is not None and word.pnp.start == start+i:\n p4 = BEGIN+\"-\"+\"PNP\"\n # If the given range does not contain the entire PNP, remove the PNP tags.\n # The range must contain the entire PNP, \n # since it starts with the PP and ends with the chunk head (and is meaningless without these).\n if word.pnp is not None and (word.pnp.start < start or word.chunk.stop > stop):\n p4, p7 = None, None\n s.append(word=p0, lemma=p1, type=p2, chunk=p3, pnp=p4, relation=p5, role=p6, anchor=p7, iob=p8, custom=p9)\n s.parent = self\n s._start = start\n return s"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef constituents(self, pnp=False):\n a = []\n for word in self.words:\n if pnp and word.pnp is not None:\n if len(a) == 0 or a[-1] != word.pnp:\n a.append(word.pnp)\n elif word.chunk is not None:\n if len(a) == 0 or a[-1] != word.chunk:\n a.append(word.chunk)\n else:\n a.append(word)\n return a", "response": "Returns an in - order list of mixed Chunk and Word objects. With pnp = True also contains PNPChunk objects whenever possible."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_xml(cls, xml):\n s = parse_string(xml)\n return Sentence(s.split(\"\\n\")[0], token=s.tags, language=s.language)", "response": "Returns a new Sentence from the given XML string."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nyields the sentence as an XML - formatted string.", "response": "def xml(self):\n \"\"\" Yields the sentence as an XML-formatted string (plain bytestring, UTF-8 encoded).\n All the sentences in the XML are wrapped in a element.\n \"\"\"\n xml = []\n xml.append('' % XML_ENCODING.get(self.encoding, self.encoding))\n xml.append(\"<%s>\" % XML_TEXT)\n xml.extend([sentence.xml for sentence in self])\n xml.append(\"\" % XML_TEXT)\n return \"\\n\".join(xml)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract credits from AUTHORS. rst", "response": "def get_credits():\n \"\"\"Extract credits from `AUTHORS.rst`\"\"\"\n credits = read(os.path.join(_HERE, \"AUTHORS.rst\")).split(\"\\n\")\n from_index = credits.index(\"Active Contributors\")\n credits = \"\\n\".join(credits[from_index + 2:])\n return credits"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rst2markdown_github(path_to_rst, path_to_md, pandoc=\"pandoc\"):\n _proc = subprocess.Popen([pandoc, \"-f\", \"rst\",\n \"-t\", \"markdown_github\",\n #\"-o\", path_to_md,\n path_to_rst],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n print(\"Converting README.rst to markdown_github, \"\n \"using 'pandoc' ...\")\n _stdout, _stderr = _proc.communicate()\n with _open(path_to_md, \"w\", encoding=\"utf-8\") as _md:\n _skip_newline = False\n for line in _stdout.decode('utf-8').split(os.linesep):\n line = re.sub(\"``` sourceCode\", \"``` python\", line)\n if line.startswith(\"[![\"):\n _md.write(line)\n _md.write(\"\\n\")\n if not line.startswith((\"[![LICENSE\")):\n _skip_newline = True\n elif _skip_newline and line == \"\":\n _skip_newline = False\n continue\n else:\n _md.write(line)\n _md.write(\"\\n\")\n\n if _stderr:\n print(\"pandoc.exe STDERR: \", _stderr)\n if os.path.isfile(path_to_md) and os.stat(path_to_md).st_size > 0:\n print(\"README.rst converted and saved as: {}\".format(path_to_md))", "response": "Converts README. rst to markdown_github."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextracts HELP information from `` -h | --help`` message **Input** * ``$ -h | --help`` * ``$ cd && make help`` **Output** * ``docs/src/console_help_xy.rst``", "response": "def console_help2rst(cwd, help_cmd, path_to_rst, rst_title,\n format_as_code=False):\n \"\"\"\n Extract HELP information from `` -h | --help`` message\n\n **Input**\n\n * ``$ -h | --help``\n * ``$ cd && make help``\n\n **Output**\n\n * ``docs/src/console_help_xy.rst``\n\n \"\"\"\n generated_time_str = \"\"\"\n\n ::\n\n generated: {0}\n\n\"\"\".format(time.strftime(\"%d %B %Y - %H:%M\"))\n\n with _open(path_to_rst, \"w\", encoding='utf-8') as f:\n print(\"File\", f)\n print(\"cwd\", cwd)\n print(\"help_cmd\", help_cmd)\n os.chdir(cwd)\n _proc = subprocess.Popen(\n help_cmd,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n universal_newlines=True,)\n help_msg = _proc.stdout.readlines()\n f.write(get_rst_title(\n rst_title,\n \"-\",\n overline=True))\n f.write(generated_time_str)\n if \"README\" in path_to_rst:\n help_msg = \"\".join(help_msg[10:])\n #help_msg = PACKAGE_DOCSTRING + help_msg\n for line in help_msg:\n # exclude directory walk messages of 'make'\n if line.strip().startswith(\"make[1]:\"):\n print(\"skipped line: {}\".format(line))\n # exclude warning messages\n elif line.strip().startswith(\"\\x1b[1m\"):\n print(\"skipped line: {}\".format(line))\n # exclude warning messages on Windows (without ``colorama``)\n elif line.strip().startswith(\"Using fallback version of '\"):\n print(\"skipped line: {}\".format(line))\n else:\n # correctly indent tips in 'make help'\n if line.strip().startswith(\"-->\"):\n f.write(3 * \"\\t\")\n if format_as_code:\n f.write(\"\\t\" + line.strip())\n f.write(\"\\n\")\n else:\n f.write(line)\n\n f.write(\"\\n\")\n if \"README\" in path_to_rst:\n f.write(get_rst_title(\"Credits\", \"^\"))\n f.write(get_credits())\n\n print(\"\\ncmd:{} in dir:{} --> RST generated:\\n\\t{}\\n\\n\".format(\n help_cmd, cwd, path_to_rst))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the documentation for the current project.", "response": "def update_docs(readme=True, makefiles=True):\n \"\"\"Update documentation (ready for publishing new release)\n\n Usually called by ``make docs``\n\n :param bool make_doc: generate DOC page from Makefile help messages\n\n \"\"\"\n if readme:\n _pandoc = get_external_executable(\"pandoc\")\n rst2markdown_github(os.path.join(_HERE, \"README.rst\"),\n os.path.join(_HERE, \"README.md\"),\n pandoc=_pandoc)\n\n if makefiles:\n _make = get_external_executable(\"make\")\n project_makefile_dir = os.path.abspath(_HERE)\n project_makefile_rst = os.path.join(\n _HERE,\n 'docs',\n 'src',\n 'project_makefile.rst')\n docs_makefile_dir = os.path.join(_HERE, 'docs', 'src')\n docs_makefile_rst = os.path.join(\n _HERE,\n 'docs',\n 'src',\n 'docs_makefile.rst')\n\n #: ``help2rst_queue`` stores tuples of\n #: ``(cwd, help_cmd, path_to_rst_file, rst_title_of_new_file)``\n help2rst_queue = [\n (project_makefile_dir, [_make, \"help\"], project_makefile_rst,\n \"Project ``Makefile``\"),\n\n (docs_makefile_dir, [_make, \"help\"], docs_makefile_rst,\n \"Documentation ``Makefile``\")]\n\n for cwd, help_cmd, outfile, title in help2rst_queue:\n console_help2rst(\n cwd,\n help_cmd,\n outfile,\n title,\n format_as_code=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns root mean square value of f", "response": "def rms(self, x, params=()):\n \"\"\" Returns root mean square value of f(x, params) \"\"\"\n internal_x, internal_params = self.pre_process(np.asarray(x),\n np.asarray(params))\n if internal_params.ndim > 1:\n raise NotImplementedError(\"Parameters should be constant.\")\n result = np.empty(internal_x.size//self.nx)\n for idx in range(internal_x.shape[0]):\n result[idx] = np.sqrt(np.mean(np.square(self.f_cb(\n internal_x[idx, :], internal_params))))\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef solve_series(self, x0, params, varied_data, varied_idx,\n internal_x0=None, solver=None, propagate=True, **kwargs):\n \"\"\" Solve system for a set of parameters in which one is varied\n\n Parameters\n ----------\n x0 : array_like\n Guess (subject to ``self.post_processors``)\n params : array_like\n Parameter values\n vaired_data : array_like\n Numerical values of the varied parameter.\n varied_idx : int or str\n Index of the varied parameter (indexing starts at 0).\n If ``self.par_by_name`` this should be the name (str) of the varied\n parameter.\n internal_x0 : array_like (default: None)\n Guess (*not* subject to ``self.post_processors``).\n Overrides ``x0`` when given.\n solver : str or callback\n See :meth:`solve`.\n propagate : bool (default: True)\n Use last successful solution as ``x0`` in consecutive solves.\n \\\\*\\\\*kwargs :\n Keyword arguments pass along to :meth:`solve`.\n\n Returns\n -------\n xout : array\n Of shape ``(varied_data.size, x0.size)``.\n info_dicts : list of dictionaries\n Dictionaries each containing keys such as containing 'success', 'nfev', 'njev' etc.\n\n \"\"\"\n if self.x_by_name and isinstance(x0, dict):\n x0 = [x0[k] for k in self.names]\n if self.par_by_name:\n if isinstance(params, dict):\n params = [params[k] for k in self.param_names]\n if isinstance(varied_idx, str):\n varied_idx = self.param_names.index(varied_idx)\n\n new_params = np.atleast_1d(np.array(params, dtype=np.float64))\n xout = np.empty((len(varied_data), len(x0)))\n self.internal_xout = np.empty_like(xout)\n self.internal_params_out = np.empty((len(varied_data),\n len(new_params)))\n info_dicts = []\n new_x0 = np.array(x0, dtype=np.float64) # copy\n conds = kwargs.get('initial_conditions', None) # see ConditionalNeqSys\n for idx, value in enumerate(varied_data):\n try:\n new_params[varied_idx] = value\n except TypeError:\n new_params = value # e.g. type(new_params) == int\n if conds is not None:\n kwargs['initial_conditions'] = conds\n x, info_dict = self.solve(new_x0, new_params, internal_x0, solver,\n **kwargs)\n if propagate:\n if info_dict['success']:\n try:\n # See ChainedNeqSys.solve\n new_x0 = info_dict['x_vecs'][0]\n internal_x0 = info_dict['internal_x_vecs'][0]\n conds = info_dict['intermediate_info'][0].get(\n 'conditions', None)\n except:\n new_x0 = x\n internal_x0 = None\n conds = info_dict.get('conditions', None)\n xout[idx, :] = x\n self.internal_xout[idx, :] = self.internal_x\n self.internal_params_out[idx, :] = self.internal_params\n info_dicts.append(info_dict)\n return xout, info_dicts", "response": "Solve system for a set of parameters in which one is varied."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nplot the results from the solve_series method.", "response": "def plot_series(self, xres, varied_data, varied_idx, **kwargs):\n \"\"\" Plots the results from :meth:`solve_series`.\n\n Parameters\n ----------\n xres : array\n Of shape ``(varied_data.size, self.nx)``.\n varied_data : array\n See :meth:`solve_series`.\n varied_idx : int or str\n See :meth:`solve_series`.\n \\\\*\\\\*kwargs :\n Keyword arguments passed to :func:`pyneqsys.plotting.plot_series`.\n\n \"\"\"\n for attr in 'names latex_names'.split():\n if kwargs.get(attr, None) is None:\n kwargs[attr] = getattr(self, attr)\n ax = plot_series(xres, varied_data, **kwargs)\n if self.par_by_name and isinstance(varied_idx, str):\n varied_idx = self.param_names.index(varied_idx)\n if self.latex_param_names:\n ax.set_xlabel('$%s$' % self.latex_param_names[varied_idx])\n elif self.param_names:\n ax.set_xlabel(self.param_names[varied_idx])\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef plot_series_residuals(self, xres, varied_data, varied_idx, params, **kwargs):\n nf = len(self.f_cb(*self.pre_process(xres[0], params)))\n xerr = np.empty((xres.shape[0], nf))\n new_params = np.array(params)\n\n for idx, row in enumerate(xres):\n new_params[varied_idx] = varied_data[idx]\n xerr[idx, :] = self.f_cb(*self.pre_process(row, params))\n return self.plot_series(xerr, varied_data, varied_idx, **kwargs)", "response": "Analogous to plot_series but will plot residuals."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plot_series_residuals_internal(self, varied_data, varied_idx, **kwargs):\n nf = len(self.f_cb(*self.pre_process(\n self.internal_xout[0], self.internal_params_out[0])))\n xerr = np.empty((self.internal_xout.shape[0], nf))\n for idx, (res, params) in enumerate(zip(self.internal_xout, self.internal_params_out)):\n xerr[idx, :] = self.f_cb(res, params)\n return self.plot_series(xerr, varied_data, varied_idx, **kwargs)", "response": "Analogous to plot_series but for internal residuals from last run."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef solve_and_plot_series(self, x0, params, varied_data, varied_idx, solver=None, plot_kwargs=None,\n plot_residuals_kwargs=None, **kwargs):\n \"\"\" Solve and plot for a series of a varied parameter.\n\n Convenience method, see :meth:`solve_series`, :meth:`plot_series` &\n :meth:`plot_series_residuals_internal` for more information.\n \"\"\"\n sol, nfo = self.solve_series(\n x0, params, varied_data, varied_idx, solver=solver, **kwargs)\n ax_sol = self.plot_series(sol, varied_data, varied_idx, info=nfo,\n **(plot_kwargs or {}))\n\n extra = dict(ax_sol=ax_sol, info=nfo)\n if plot_residuals_kwargs:\n extra['ax_resid'] = self.plot_series_residuals_internal(\n varied_data, varied_idx, info=nfo,\n **(plot_residuals_kwargs or {})\n )\n return sol, extra", "response": "Solve and plot for a varied parameter."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nuse internally for transformation of variables.", "response": "def pre_process(self, x0, params=()):\n \"\"\" Used internally for transformation of variables. \"\"\"\n # Should be used by all methods matching \"solve_*\"\n if self.x_by_name and isinstance(x0, dict):\n x0 = [x0[k] for k in self.names]\n if self.par_by_name and isinstance(params, dict):\n params = [params[k] for k in self.param_names]\n for pre_processor in self.pre_processors:\n x0, params = pre_processor(x0, params)\n return x0, np.atleast_1d(params)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef post_process(self, xout, params_out):\n # Should be used by all methods matching \"solve_*\"\n for post_processor in self.post_processors:\n xout, params_out = post_processor(xout, params_out)\n return xout, params_out", "response": "Used internally for transformation of variables."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef solve(self, x0, params=(), internal_x0=None, solver=None, attached_solver=None, **kwargs):\n if not isinstance(solver, (tuple, list)):\n solver = [solver]\n if not isinstance(attached_solver, (tuple, list)):\n attached_solver = [attached_solver] + [None]*(len(solver) - 1)\n _x0, self.internal_params = self.pre_process(x0, params)\n for solv, attached_solv in zip(solver, attached_solver):\n if internal_x0 is not None:\n _x0 = internal_x0\n elif self.internal_x0_cb is not None:\n _x0 = self.internal_x0_cb(x0, params)\n\n nfo = self._get_solver_cb(solv, attached_solv)(_x0, **kwargs)\n _x0 = nfo['x'].copy()\n self.internal_x = _x0\n x0 = self.post_process(self.internal_x, self.internal_params)[0]\n return x0, nfo", "response": "Solve with user specified solver choice."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _solve_scipy(self, intern_x0, tol=1e-8, method=None, **kwargs):\n from scipy.optimize import root\n if method is None:\n if self.nf > self.nx:\n method = 'lm'\n elif self.nf == self.nx:\n method = 'hybr'\n else:\n raise ValueError('Underdetermined problem')\n if 'band' in kwargs:\n raise ValueError(\"Set 'band' at initialization instead.\")\n if 'args' in kwargs:\n raise ValueError(\"Set 'args' as params in initialization instead.\")\n\n new_kwargs = kwargs.copy()\n if self.band is not None:\n warnings.warn(\"Band argument ignored (see SciPy docs)\")\n new_kwargs['band'] = self.band\n new_kwargs['args'] = self.internal_params\n\n return root(self.f_cb, intern_x0, jac=self.j_cb, method=method, tol=tol, **new_kwargs)", "response": "Solve the current state of a set of unsatisfied iCal objects using scipy. optimize. root."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef solve(self, x0, params=(), internal_x0=None, solver=None,\n conditional_maxiter=20, initial_conditions=None, **kwargs):\n \"\"\" Solve the problem (systems of equations)\n\n Parameters\n ----------\n x0 : array\n Guess.\n params : array\n See :meth:`NeqSys.solve`.\n internal_x0 : array\n See :meth:`NeqSys.solve`.\n solver : str or callable or iterable of such.\n See :meth:`NeqSys.solve`.\n conditional_maxiter : int\n Maximum number of switches between conditions.\n initial_conditions : iterable of bools\n Corresponding conditions to ``x0``\n \\\\*\\\\*kwargs :\n Keyword arguments passed on to :meth:`NeqSys.solve`.\n\n \"\"\"\n if initial_conditions is not None:\n conds = initial_conditions\n else:\n conds = self.get_conds(x0, params, initial_conditions)\n idx, nfev, njev = 0, 0, 0\n while idx < conditional_maxiter:\n neqsys = self.neqsys_factory(conds)\n x0, info = neqsys.solve(x0, params, internal_x0, solver, **kwargs)\n if idx == 0:\n internal_x0 = None\n nfev += info['nfev']\n njev += info.get('njev', 0)\n new_conds = self.get_conds(x0, params, conds)\n if new_conds == conds:\n break\n else:\n conds = new_conds\n idx += 1\n if idx == conditional_maxiter:\n raise Exception(\"Solving failed, conditional_maxiter reached\")\n self.internal_x = info['x']\n self.internal_params = neqsys.internal_params\n result = {\n 'x': info['x'],\n 'success': info['success'],\n 'conditions': conds,\n 'nfev': nfev,\n 'njev': njev,\n }\n if 'fun' in info:\n result['fun'] = info['fun']\n return x0, result", "response": "Solve the problem for a set of conditions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsolving a system using a user - specified solver.", "response": "def solve(guess_a, guess_b, power, solver='scipy'):\n \"\"\" Constructs a pyneqsys.symbolic.SymbolicSys instance and returns from its ``solve`` method. \"\"\"\n # The problem is 2 dimensional so we need 2 symbols\n x = sp.symbols('x:2', real=True)\n # There is a user specified parameter ``p`` in this problem:\n p = sp.Symbol('p', real=True, negative=False, integer=True)\n # Our system consists of 2-non-linear equations:\n f = [x[0] + (x[0] - x[1])**p/2 - 1,\n (x[1] - x[0])**p/2 + x[1]]\n # We construct our ``SymbolicSys`` instance by passing variables, equations and parameters:\n neqsys = SymbolicSys(x, f, [p]) # (this will derive the Jacobian symbolically)\n\n # Finally we solve the system using user-specified ``solver`` choice:\n return neqsys.solve([guess_a, guess_b], [power], solver=solver)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots the values of the varied parameter vector vs the varied parameter vector.", "response": "def plot_series(xres, varied_data, indices=None, info=None,\n fail_vline=None, plot_kwargs_cb=None,\n ls=('-', '--', ':', '-.'),\n c=('k', 'r', 'g', 'b', 'c', 'm', 'y'), labels=None,\n ax=None, names=None, latex_names=None):\n \"\"\" Plot the values of the solution vector vs the varied parameter.\n\n Parameters\n ----------\n xres : array\n Solution vector of shape ``(varied_data.size, x0.size)``.\n varied_data : array\n Numerical values of the varied parameter.\n indices : iterable of integers, optional\n Indices of variables to be plotted. default: all\n fail_vline : bool\n Show vertical lines where the solver failed.\n plot_kwargs_cb : callable\n Takes the index as single argument, returns a dict passed to\n the plotting function\n ls : iterable of str\n Linestyles.\n c : iterable of str\n Colors.\n labels : iterable of str\n ax : matplotlib Axes instance\n names : iterable of str\n latex_names : iterable of str\n\n \"\"\"\n import matplotlib.pyplot as plt\n\n if indices is None:\n indices = range(xres.shape[1])\n\n if fail_vline is None:\n if info is None:\n fail_vline = False\n else:\n fail_vline = True\n\n if ax is None:\n ax = plt.subplot(1, 1, 1)\n\n if labels is None:\n labels = names if latex_names is None else ['$%s$' % ln.strip('$') for ln in latex_names]\n\n if plot_kwargs_cb is None:\n def plot_kwargs_cb(idx, labels=None):\n kwargs = {'ls': ls[idx % len(ls)],\n 'c': c[idx % len(c)]}\n if labels:\n kwargs['label'] = labels[idx]\n return kwargs\n else:\n plot_kwargs_cb = plot_kwargs_cb or (lambda idx: {})\n\n for idx in indices:\n ax.plot(varied_data, xres[:, idx], **plot_kwargs_cb(idx, labels=labels))\n\n if fail_vline:\n for i, nfo in enumerate(info):\n if not nfo['success']:\n ax.axvline(varied_data[i], c='k', ls='--')\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nplace a legend box outside a matplotlib Axes instance.", "response": "def mpl_outside_legend(ax, **kwargs):\n \"\"\" Places a legend box outside a matplotlib Axes instance. \"\"\"\n box = ax.get_position()\n ax.set_position([box.x0, box.y0, box.width * 0.75, box.height])\n # Put a legend to the right of the current axis\n ax.legend(loc='upper left', bbox_to_anchor=(1, 1), **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntransforming a linear system to reduced row - echelon form.", "response": "def linear_rref(A, b, Matrix=None, S=None):\n \"\"\" Transform a linear system to reduced row-echelon form\n\n Transforms both the matrix and right-hand side of a linear\n system of equations to reduced row echelon form\n\n Parameters\n ----------\n A : Matrix-like\n Iterable of rows.\n b : iterable\n\n Returns\n -------\n A', b' - transformed versions\n\n \"\"\"\n if Matrix is None:\n from sympy import Matrix\n if S is None:\n from sympy import S\n mat_rows = [_map2l(S, list(row) + [v]) for row, v in zip(A, b)]\n aug = Matrix(mat_rows)\n raug, pivot = aug.rref()\n nindep = len(pivot)\n return raug[:nindep, :-1], raug[:nindep, -1]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef linear_exprs(A, x, b=None, rref=False, Matrix=None):\n if b is None:\n b = [0]*len(x)\n if rref:\n rA, rb = linear_rref(A, b, Matrix)\n if Matrix is None:\n from sympy import Matrix\n return [lhs - rhs for lhs, rhs in zip(rA * Matrix(len(x), 1, x), rb)]\n else:\n return [sum([x0*x1 for x0, x1 in zip(row, x)]) - v\n for row, v in zip(A, b)]", "response": "Linear expression of A."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_callback(cls, cb, nx=None, nparams=None, **kwargs):\n if kwargs.get('x_by_name', False):\n if 'names' not in kwargs:\n raise ValueError(\"Need ``names`` in kwargs.\")\n if nx is None:\n nx = len(kwargs['names'])\n elif nx != len(kwargs['names']):\n raise ValueError(\"Inconsistency between nx and length of ``names``.\")\n if kwargs.get('par_by_name', False):\n if 'param_names' not in kwargs:\n raise ValueError(\"Need ``param_names`` in kwargs.\")\n if nparams is None:\n nparams = len(kwargs['param_names'])\n elif nparams != len(kwargs['param_names']):\n raise ValueError(\"Inconsistency between ``nparam`` and length of ``param_names``.\")\n\n if nparams is None:\n nparams = 0\n\n if nx is None:\n raise ValueError(\"Need ``nx`` of ``names`` together with ``x_by_name==True``.\")\n be = Backend(kwargs.pop('backend', None))\n x = be.real_symarray('x', nx)\n p = be.real_symarray('p', nparams)\n _x = dict(zip(kwargs['names'], x)) if kwargs.get('x_by_name', False) else x\n _p = dict(zip(kwargs['param_names'], p)) if kwargs.get('par_by_name', False) else p\n try:\n exprs = cb(_x, _p, be)\n except TypeError:\n exprs = _ensure_3args(cb)(_x, _p, be)\n return cls(x, exprs, p, backend=be, **kwargs)", "response": "Generate a new instance of a SymbolicSys instance from a callback function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_jac(self):\n if self._jac is True:\n if self.band is None:\n f = self.be.Matrix(self.nf, 1, self.exprs)\n _x = self.be.Matrix(self.nx, 1, self.x)\n return f.jacobian(_x)\n else:\n # Banded\n return self.be.Matrix(banded_jacobian(\n self.exprs, self.x, *self.band))\n elif self._jac is False:\n return False\n else:\n return self._jac", "response": "Returns the jacobian of the expressions"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a TransformedSys instance from a callback.", "response": "def from_callback(cls, cb, transf_cbs, nx, nparams=0, pre_adj=None,\n **kwargs):\n \"\"\" Generate a TransformedSys instance from a callback\n\n Parameters\n ----------\n cb : callable\n Should have the signature ``cb(x, p, backend) -> list of exprs``.\n The callback ``cb`` should return *untransformed* expressions.\n transf_cbs : pair or iterable of pairs of callables\n Callables for forward- and backward-transformations. Each\n callable should take a single parameter (expression) and\n return a single expression.\n nx : int\n Number of unkowns.\n nparams : int\n Number of parameters.\n pre_adj : callable, optional\n To tweak expression prior to transformation. Takes a\n sinlge argument (expression) and return a single argument\n rewritten expression.\n \\\\*\\\\*kwargs :\n Keyword arguments passed on to :class:`TransformedSys`. See also\n :class:`SymbolicSys` and :class:`pyneqsys.NeqSys`.\n\n Examples\n --------\n >>> import sympy as sp\n >>> transformed = TransformedSys.from_callback(lambda x, p, be: [\n ... x[0]*x[1] - p[0],\n ... be.exp(-x[0]) + be.exp(-x[1]) - p[0]**-2\n ... ], (sp.log, sp.exp), 2, 1)\n ...\n\n\n \"\"\"\n be = Backend(kwargs.pop('backend', None))\n x = be.real_symarray('x', nx)\n p = be.real_symarray('p', nparams)\n try:\n transf = [(transf_cbs[idx][0](xi),\n transf_cbs[idx][1](xi))\n for idx, xi in enumerate(x)]\n except TypeError:\n transf = zip(_map2(transf_cbs[0], x), _map2(transf_cbs[1], x))\n try:\n exprs = cb(x, p, be)\n except TypeError:\n exprs = _ensure_3args(cb)(x, p, be)\n return cls(x, _map2l(pre_adj, exprs), transf, p, backend=be, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite binary header data to a file handle. This method writes exactly 512 bytes to the beginning of the given file handle. Parameters ---------- handle : file handle The given handle will be reset to 0 using `seek` and then 512 bytes will be written to describe the parameters in this Header. The handle must be writeable.", "response": "def write(self, handle):\n '''Write binary header data to a file handle.\n\n This method writes exactly 512 bytes to the beginning of the given file\n handle.\n\n Parameters\n ----------\n handle : file handle\n The given handle will be reset to 0 using `seek` and then 512 bytes\n will be written to describe the parameters in this Header. The\n handle must be writeable.\n '''\n handle.seek(0)\n handle.write(struct.pack(self.BINARY_FORMAT,\n self.parameter_block,\n 0x50,\n self.point_count,\n self.analog_count,\n self.first_frame,\n self.last_frame,\n self.max_gap,\n self.scale_factor,\n self.data_block,\n self.analog_per_frame,\n self.frame_rate,\n b'',\n self.long_event_labels and 0x3039 or 0x0,\n self.label_block,\n b''))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read(self, handle):\n '''Read and parse binary header data from a file handle.\n\n This method reads exactly 512 bytes from the beginning of the given file\n handle.\n\n Parameters\n ----------\n handle : file handle\n The given handle will be reset to 0 using `seek` and then 512 bytes\n will be read to initialize the attributes in this Header. The handle\n must be readable.\n\n Raises\n ------\n AssertionError\n If the magic byte from the header is not 80 (the C3D magic value).\n '''\n handle.seek(0)\n (self.parameter_block,\n magic,\n self.point_count,\n self.analog_count,\n self.first_frame,\n self.last_frame,\n self.max_gap,\n self.scale_factor,\n self.data_block,\n self.analog_per_frame,\n self.frame_rate,\n _,\n self.long_event_labels,\n self.label_block,\n _) = struct.unpack(self.BINARY_FORMAT, handle.read(512))\n\n assert magic == 80, 'C3D magic {} != 80 !'.format(magic)", "response": "Read and parse binary header data from a file handle."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the number of bytes needed to store this parameter.", "response": "def binary_size(self):\n '''Return the number of bytes needed to store this parameter.'''\n return (\n 1 + # group_id\n 2 + # next offset marker\n 1 + len(self.name.encode('utf-8')) + # size of name and name bytes\n 1 + # data size\n 1 + len(self.dimensions) + # size of dimensions and dimension bytes\n self.total_bytes + # data\n 1 + len(self.desc.encode('utf-8')) # size of desc and desc bytes\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites binary data for this parameter to a file handle.", "response": "def write(self, group_id, handle):\n '''Write binary data for this parameter to a file handle.\n\n Parameters\n ----------\n group_id : int\n The numerical ID of the group that holds this parameter.\n handle : file handle\n An open, writable, binary file handle.\n '''\n name = self.name.encode('utf-8')\n handle.write(struct.pack('bb', len(name), group_id))\n handle.write(name)\n handle.write(struct.pack(' -1\n points[~valid, 3:5] = -1\n c = raw[valid, 3].astype(np.uint16)\n\n # fourth value is floating-point (scaled) error estimate\n points[valid, 3] = (c & 0xff).astype(float) * scale\n\n # fifth value is number of bits set in camera-observation byte\n points[valid, 4] = sum((c & (1 << k)) >> k for k in range(8, 17))\n\n if self.header.analog_count > 0:\n n = self.header.analog_count\n raw = np.fromstring(self._handle.read(n * analog_bytes),\n dtype=analog_dtype,\n count=n).reshape((-1, self.analog_used)).T\n analog = (raw.astype(float) - offsets) * scales * gen_scale\n\n if copy:\n yield frame_no, points.copy(), analog.copy()\n else:\n yield frame_no, points, analog", "response": "Iterate over the data frames from our C3D file handle."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _pad_block(self, handle):\n '''Pad the file with 0s to the end of the next block boundary.'''\n extra = handle.tell() % 512\n if extra:\n handle.write(b'\\x00' * (512 - extra))", "response": "Pad the file with 0s to the end of the next block boundary."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites the metadata to a given file handle.", "response": "def _write_metadata(self, handle):\n '''Write metadata to a file handle.\n\n Parameters\n ----------\n handle : file\n Write metadata and C3D motion frames to the given file handle. The\n writer does not close the handle.\n '''\n self.check_metadata()\n\n # header\n self.header.write(handle)\n self._pad_block(handle)\n assert handle.tell() == 512\n\n # groups\n handle.write(struct.pack(\n 'BBBB', 0, 0, self.parameter_blocks(), PROCESSOR_INTEL))\n id_groups = sorted(\n (i, g) for i, g in self.groups.items() if isinstance(i, int))\n for group_id, group in id_groups:\n group.write(group_id, handle)\n\n # padding\n self._pad_block(handle)\n while handle.tell() != 512 * (self.header.data_block - 1):\n handle.write(b'\\x00' * 512)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _write_frames(self, handle):\n '''Write our frame data to the given file handle.\n\n Parameters\n ----------\n handle : file\n Write metadata and C3D motion frames to the given file handle. The\n writer does not close the handle.\n '''\n assert handle.tell() == 512 * (self.header.data_block - 1)\n scale = abs(self.point_scale)\n is_float = self.point_scale < 0\n point_dtype = [np.int16, np.float32][is_float]\n point_scale = [scale, 1][is_float]\n point_format = 'if'[is_float]\n raw = np.empty((self.point_used, 4), point_dtype)\n for points, analog in self._frames:\n valid = points[:, 3] > -1\n raw[~valid, 3] = -1\n raw[valid, :3] = points[valid, :3] / self._point_scale\n raw[valid, 3] = (\n ((points[valid, 4]).astype(np.uint8) << 8) |\n (points[valid, 3] / scale).astype(np.uint16)\n )\n point = array.array(point_format)\n point.extend(raw.flatten())\n point.tofile(handle)\n analog = array.array(point_format)\n analog.extend(analog)\n analog.tofile(handle)\n self._pad_block(handle)", "response": "Write our frame data to the given file handle."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write(self, handle):\n '''Write metadata and point + analog frames to a file handle.\n\n Parameters\n ----------\n handle : file\n Write metadata and C3D motion frames to the given file handle. The\n writer does not close the handle.\n '''\n if not self._frames:\n return\n\n def add(name, desc, bpe, format, bytes, *dimensions):\n group.add_param(name,\n desc=desc,\n bytes_per_element=bpe,\n bytes=struct.pack(format, bytes),\n dimensions=list(dimensions))\n\n def add_str(name, desc, bytes, *dimensions):\n group.add_param(name,\n desc=desc,\n bytes_per_element=-1,\n bytes=bytes.encode('utf-8'),\n dimensions=list(dimensions))\n\n def add_empty_array(name, desc, bpe):\n group.add_param(name, desc=desc, bytes_per_element=bpe, dimensions=[0])\n\n points, analog = self._frames[0]\n ppf = len(points)\n\n # POINT group\n group = self.add_group(1, 'POINT', 'POINT group')\n add('USED', 'Number of 3d markers', 2, '0 it will limit the amount of prefixes returned. filter_proto: Can be: - None: Returns both ipv4 and ipv6 - 4: Returns only ipv4 - 6: Retruns only ipv6 Returns: A list of prefixes sorted by sum_bytes. For example: [ {'key': '192.168.1.0/25', 'sum_bytes': 3000, 'as_dst': 345}, {'key': '192.213.1.0/25', 'sum_bytes': 2000, 'as_dst': 123}, {'key': '231.168.1.0/25', 'sum_bytes': 1000, 'as_dst': 321}, ]", "response": "def aggregate_per_prefix(self, start_time, end_time, limit=0, net_masks='', exclude_net_masks=False, filter_proto=None):\n \"\"\" Given a time range aggregates bytes per prefix.\n\n Args:\n start_time: A string representing the starting time of the time range\n end_time: A string representing the ending time of the time range\n limit: An optional integer. If it's >0 it will limit the amount of prefixes returned.\n filter_proto: Can be:\n - None: Returns both ipv4 and ipv6\n - 4: Returns only ipv4\n - 6: Retruns only ipv6\n\n Returns:\n A list of prefixes sorted by sum_bytes. For example:\n\n [\n {'key': '192.168.1.0/25', 'sum_bytes': 3000, 'as_dst': 345},\n {'key': '192.213.1.0/25', 'sum_bytes': 2000, 'as_dst': 123},\n {'key': '231.168.1.0/25', 'sum_bytes': 1000, 'as_dst': 321},\n ]\n \"\"\"\n if net_masks == '':\n net_mask_filter = ''\n elif not exclude_net_masks:\n net_mask_filter = 'AND mask_dst IN ({})'.format(net_masks)\n elif exclude_net_masks:\n net_mask_filter = 'AND mask_dst NOT IN ({})'.format(net_masks)\n\n if filter_proto is None:\n proto_filter = ''\n elif int(filter_proto) == 4:\n proto_filter = 'AND ip_dst NOT LIKE \"%:%\"'\n elif int(filter_proto) == 6:\n proto_filter = 'AND ip_dst LIKE \"%:%\"'\n\n query = ''' SELECT ip_dst||'/'||mask_dst as key, SUM(bytes) as sum_bytes, as_dst\n from acct\n WHERE\n datetime(stamp_updated) BETWEEN datetime(?) AND datetime(?, \"+1 second\")\n {}\n {}\n GROUP by ip_dst,mask_dst\n ORDER BY SUM(bytes) DESC\n '''.format(net_mask_filter, proto_filter)\n\n if limit > 0:\n query += 'LIMIT %d' % limit\n\n return self._execute_query(query, [start_time, end_time])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a time range aggregates bytes per ASNs. Args: start_time: A string representing the starting time of the time range end_time: A string representing the ending time of the time range Returns: A list of prefixes sorted by sum_bytes. For example: [ {'key': '6500', 'sum_bytes': 3000}, {'key': '2310', 'sum_bytes': 2000}, {'key': '8182', 'sum_bytes': 1000}, ]", "response": "def aggregate_per_as(self, start_time, end_time):\n \"\"\" Given a time range aggregates bytes per ASNs.\n\n Args:\n start_time: A string representing the starting time of the time range\n end_time: A string representing the ending time of the time range\n\n Returns:\n A list of prefixes sorted by sum_bytes. For example:\n\n [\n {'key': '6500', 'sum_bytes': 3000},\n {'key': '2310', 'sum_bytes': 2000},\n {'key': '8182', 'sum_bytes': 1000},\n ]\n \"\"\"\n\n query = ''' SELECT as_dst as key, SUM(bytes) as sum_bytes\n from acct\n WHERE\n datetime(stamp_updated) BETWEEN datetime(?) AND datetime(?, \"+1 second\")\n GROUP by as_dst\n ORDER BY SUM(bytes) DESC;\n '''\n\n return self._execute_query(query, [start_time, end_time])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreplace vector images with fake ones.", "response": "def _workaround_no_vector_images(project):\n \"\"\"Replace vector images with fake ones.\"\"\"\n RED = (255, 0, 0)\n PLACEHOLDER = kurt.Image.new((32, 32), RED)\n for scriptable in [project.stage] + project.sprites:\n for costume in scriptable.costumes:\n if costume.image.format == \"SVG\":\n yield \"%s - %s\" % (scriptable.name, costume.name)\n costume.image = PLACEHOLDER"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _workaround_no_stage_specific_variables(project):\n for (name, var) in project.stage.variables.items():\n yield \"variable %s\" % name\n for (name, _list) in project.stage.lists.items():\n yield \"list %s\" % name\n project.variables.update(project.stage.variables)\n project.lists.update(project.stage.lists)\n project.stage.variables = {}\n project.stage.lists = {}", "response": "Make Stage - specific variables global ( move them to Project"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nregisters a new KurtPlugin.", "response": "def register(cls, plugin):\n \"\"\"Register a new :class:`KurtPlugin`.\n\n Once registered, the plugin can be used by :class:`Project`, when:\n\n * :attr:`Project.load` sees a file with the right extension\n\n * :attr:`Project.convert` is called with the format as a parameter\n\n \"\"\"\n cls.plugins[plugin.name] = plugin\n\n # make features\n plugin.features = map(Feature.get, plugin.features)\n\n # fix blocks\n blocks = []\n for pbt in plugin.blocks:\n if pbt:\n pbt = pbt.copy()\n pbt.format = plugin.name\n blocks.append(pbt)\n plugin.blocks = blocks\n\n # add blocks\n new_blocks = filter(None, plugin.blocks)\n for pbt in new_blocks:\n for bt in cls.blocks:\n if (bt.has_command(pbt.command) or\n bt.has_command(pbt._match)):\n bt._add_conversion(plugin.name, pbt)\n break\n else:\n if pbt._match:\n raise ValueError, \"Couldn't match %r\" % pbt._match\n cls.blocks.append(kurt.BlockType(pbt))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the first format plugin whose attributes match kwargs.", "response": "def get_plugin(cls, name=None, **kwargs):\n \"\"\"Returns the first format plugin whose attributes match kwargs.\n\n For example::\n\n get_plugin(extension=\"scratch14\")\n\n Will return the :class:`KurtPlugin` whose :attr:`extension\n ` attribute is ``\"scratch14\"``.\n\n The :attr:`name ` is used as the ``format`` parameter\n to :attr:`Project.load` and :attr:`Project.save`.\n\n :raises: :class:`ValueError` if the format doesn't exist.\n\n :returns: :class:`KurtPlugin`\n\n \"\"\"\n if isinstance(name, KurtPlugin):\n return name\n\n if 'extension' in kwargs:\n kwargs['extension'] = kwargs['extension'].lower()\n if name:\n kwargs[\"name\"] = name\n if not kwargs:\n raise ValueError, \"No arguments\"\n\n for plugin in cls.plugins.values():\n for name in kwargs:\n if getattr(plugin, name) != kwargs[name]:\n break\n else:\n return plugin\n\n raise ValueError, \"Unknown format %r\" % kwargs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef block_by_command(cls, command):\n for block in cls.blocks:\n if block.has_command(command):\n return block", "response": "Return the block with the given command. Returns None if the block is not found."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef blocks_by_text(cls, text):\n text = kurt.BlockType._strip_text(text)\n matches = []\n for block in cls.blocks:\n for pbt in block.conversions:\n if pbt.stripped_text == text:\n matches.append(block)\n break\n return matches", "response": "Return a list of blocks matching the given text."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef clean_up(scripts):\n scripts_with_pos = [s for s in scripts if s.pos]\n scripts_with_pos.sort(key=lambda s: (s.pos[1], s.pos[0]))\n scripts = scripts_with_pos + [s for s in scripts if not s.pos]\n\n y = 20\n for script in scripts:\n script.pos = (20, y)\n if isinstance(script, kurt.Script):\n y += stack_height(script.blocks)\n elif isinstance(script, kurt.Comment):\n y += 14\n y += 15", "response": "Clean up the given list of scripts in - place so none of the scripts overlap."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of classes in a module that have a classID attribute.", "response": "def obj_classes_from_module(module):\n \"\"\"Return a list of classes in a module that have a 'classID' attribute.\"\"\"\n for name in dir(module):\n if not name.startswith('_'):\n cls = getattr(module, name)\n if getattr(cls, 'classID', None):\n yield (name, cls)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndecode obj table entries into a single object.", "response": "def decode_network(objects):\n \"\"\"Return root object from ref-containing obj table entries\"\"\"\n def resolve_ref(obj, objects=objects):\n if isinstance(obj, Ref):\n # first entry is 1\n return objects[obj.index - 1]\n else:\n return obj\n\n # Reading the ObjTable backwards somehow makes more sense.\n for i in xrange(len(objects)-1, -1, -1):\n obj = objects[i]\n\n if isinstance(obj, Container):\n obj.update((k, resolve_ref(v)) for (k, v) in obj.items())\n\n elif isinstance(obj, Dictionary):\n obj.value = dict(\n (resolve_ref(field), resolve_ref(value))\n for (field, value) in obj.value.items()\n )\n\n elif isinstance(obj, dict):\n obj = dict(\n (resolve_ref(field), resolve_ref(value))\n for (field, value) in obj.items()\n )\n\n elif isinstance(obj, list):\n obj = [resolve_ref(field) for field in obj]\n\n elif isinstance(obj, Form):\n for field in obj.value:\n value = getattr(obj, field)\n value = resolve_ref(value)\n setattr(obj, field, value)\n\n elif isinstance(obj, ContainsRefs):\n obj.value = [resolve_ref(field) for field in obj.value]\n\n objects[i] = obj\n\n for obj in objects:\n if isinstance(obj, Form):\n obj.built()\n\n root = objects[0]\n return root"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encode_network(root):\n orig_objects = []\n objects = []\n\n def get_ref(value, objects=objects):\n \"\"\"Returns the index of the given object in the object table,\n adding it if needed.\n\n \"\"\"\n value = PythonicAdapter(Pass)._encode(value, None)\n # Convert strs to FixedObjects here to make sure they get encoded\n # correctly\n\n if isinstance(value, (Container, FixedObject)):\n if getattr(value, '_tmp_index', None):\n index = value._tmp_index\n else:\n objects.append(value)\n index = len(objects)\n value._tmp_index = index\n orig_objects.append(value) # save the object so we can\n # strip the _tmp_indexes later\n return Ref(index)\n else:\n return value # Inline value\n\n def fix_fields(obj):\n obj = PythonicAdapter(Pass)._encode(obj, None)\n # Convert strs to FixedObjects here to make sure they get encoded\n # correctly\n\n if isinstance(obj, Container):\n obj.update((k, get_ref(v)) for (k, v) in obj.items()\n if k != 'class_name')\n fixed_obj = obj\n\n elif isinstance(obj, Dictionary):\n fixed_obj = obj.__class__(dict(\n (get_ref(field), get_ref(value))\n for (field, value) in obj.value.items()\n ))\n\n elif isinstance(obj, dict):\n fixed_obj = dict(\n (get_ref(field), get_ref(value))\n for (field, value) in obj.items()\n )\n\n elif isinstance(obj, list):\n fixed_obj = [get_ref(field) for field in obj]\n\n elif isinstance(obj, Form):\n fixed_obj = obj.__class__(**dict(\n (field, get_ref(value))\n for (field, value) in obj.value.items()\n ))\n\n elif isinstance(obj, ContainsRefs):\n fixed_obj = obj.__class__([get_ref(field)\n for field in obj.value])\n\n else:\n return obj\n\n fixed_obj._made_from = obj\n return fixed_obj\n\n root = PythonicAdapter(Pass)._encode(root, None)\n\n i = 0\n objects = [root]\n root._tmp_index = 1\n while i < len(objects):\n objects[i] = fix_fields(objects[i])\n i += 1\n\n for obj in orig_objects:\n obj._tmp_index = None\n # Strip indexes off objects in case we save again later\n\n return objects", "response": "Yields the object table entries from the object network."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nyield ref - containing obj table entries from object network", "response": "def encode_network(root):\n \"\"\"Yield ref-containing obj table entries from object network\"\"\"\n def fix_values(obj):\n if isinstance(obj, Container):\n obj.update((k, get_ref(v)) for (k, v) in obj.items()\n if k != 'class_name')\n fixed_obj = obj\n\n elif isinstance(obj, Dictionary):\n fixed_obj = obj.__class__(dict(\n (get_ref(field), get_ref(value))\n for (field, value) in obj.value.items()\n ))\n\n elif isinstance(obj, dict):\n fixed_obj = dict(\n (get_ref(field), get_ref(value))\n for (field, value) in obj.items()\n )\n\n elif isinstance(obj, list):\n fixed_obj = [get_ref(field) for field in obj]\n\n elif isinstance(obj, Form):\n fixed_obj = obj.__class__(**dict(\n (field, get_ref(value))\n for (field, value) in obj.value.items()\n ))\n\n elif isinstance(obj, ContainsRefs):\n fixed_obj = obj.__class__([get_ref(field)\n for field in obj.value])\n\n else:\n return obj\n\n fixed_obj._made_from = obj\n return fixed_obj\n\n objects = []\n\n def get_ref(obj, objects=objects):\n obj = PythonicAdapter(Pass)._encode(obj, None)\n\n if isinstance(obj, (FixedObject, Container)):\n if getattr(obj, '_index', None):\n index = obj._index\n else:\n objects.append(None)\n obj._index = index = len(objects)\n objects[index - 1] = fix_values(obj)\n return Ref(index)\n else:\n return obj # Inline value\n\n get_ref(root)\n\n for obj in objects:\n if getattr(obj, '_index', None):\n del obj._index\n return objects"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef decode_obj_table(table_entries, plugin):\n entries = []\n for entry in table_entries:\n if isinstance(entry, Container):\n assert not hasattr(entry, '__recursion_lock__')\n user_obj_def = plugin.user_objects[entry.classID]\n assert entry.version == user_obj_def.version\n entry = Container(class_name=entry.classID,\n **dict(zip(user_obj_def.defaults.keys(),\n entry.values)))\n entries.append(entry)\n\n return decode_network(entries)", "response": "Converts user - class objects to a list of user - class objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning list of obj table entries. Converts user - class objects into Container objects.", "response": "def encode_obj_table(root, plugin):\n \"\"\"Return list of obj table entries. Converts user-class objects\"\"\"\n entries = encode_network(root)\n\n table_entries = []\n for entry in entries:\n if isinstance(entry, Container):\n assert not hasattr(entry, '__recursion_lock__')\n user_obj_def = plugin.user_objects[entry.class_name]\n attrs = OrderedDict()\n for (key, default) in user_obj_def.defaults.items():\n attrs[key] = entry.get(key, default)\n entry = Container(classID=entry.class_name,\n length=len(attrs),\n version=user_obj_def.version,\n values=attrs.values())\n table_entries.append(entry)\n return table_entries"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nencoding a class to a lower - level object using the class own to_construct function.", "response": "def _encode(self, obj, context):\n \"\"\"Encodes a class to a lower-level object using the class' own\n to_construct function.\n If no such function is defined, returns the object unchanged.\n \"\"\"\n func = getattr(obj, 'to_construct', None)\n if callable(func):\n return func(context)\n else:\n return obj"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitialises a new Python class from a construct using the mapping passed to the adapter.", "response": "def _decode(self, obj, context):\n \"\"\"Initialises a new Python class from a construct using the mapping\n passed to the adapter.\n \"\"\"\n cls = self._get_class(obj.classID)\n return cls.from_construct(obj, context)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write_file(self, name, contents):\n # TODO: find a way to make ZipFile accept a file object.\n zi = zipfile.ZipInfo(name)\n zi.date_time = time.localtime(time.time())[:6]\n zi.compress_type = zipfile.ZIP_DEFLATED\n zi.external_attr = 0777 << 16L\n self.zip_file.writestr(zi, contents)", "response": "Write file contents string into archive."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads a MIDI file into a sequence of MIDI objects.", "response": "def load_midi_file(path):\n \"\"\"Yield (pitch, start_beat, end_beat) for each note in midi file.\"\"\"\n\n midi_notes = []\n def register_note(track, channel, pitch, velocity, start, end):\n midi_notes.append((pitch, start, end))\n midi.register_note = register_note\n\n global m\n m = midi.MidiFile()\n m.open(midi_path)\n m.read()\n m.close()\n\n for (pitch, start, end) in midi_notes:\n start /= m.ticksPerQuarterNote\n end /= m.ticksPerQuarterNote\n yield (pitch, start, end)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of images and sounds from a v14 scriptable.", "response": "def get_media(self, v14_scriptable):\n \"\"\"Return (images, sounds)\"\"\"\n images = []\n sounds = []\n for media in v14_scriptable.media:\n if media.class_name == 'SoundMedia':\n sounds.append(media)\n elif media.class_name == 'ImageMedia':\n images.append(media)\n return (images, sounds)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndecode a run - length encoded ByteArray and returns a Bitmap. The ByteArray decompresses to a sequence of 32 - bit values which are stored as a byte string.", "response": "def from_byte_array(cls, bytes_):\n \"\"\"Decodes a run-length encoded ByteArray and returns a Bitmap.\n The ByteArray decompresses to a sequence of 32-bit values, which are\n stored as a byte string. (The specific encoding depends on Form.depth.)\n \"\"\"\n runs = cls._length_run_coding.parse(bytes_)\n pixels = (run.pixels for run in runs.data)\n data = \"\".join(itertools.chain.from_iterable(pixels))\n return cls(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_string(cls, width, height, rgba_string):\n # Convert RGBA string to ARGB\n raw = \"\"\n for i in range(0, len(rgba_string), 4):\n raw += rgba_string[i+3] # alpha\n raw += rgba_string[i:i+3] # rgb\n\n assert len(rgba_string) == width * height * 4\n\n return Form(\n width = width,\n height = height,\n depth = 32,\n bits = Bitmap(raw),\n )", "response": "Returns a Form with 32 - bit RGBA pixels\n Accepts string containing raw RGBA color values\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef open_file(path):\n if sys.platform=='win32':\n os.startfile(path)\n #subprocess.Popen(['start', path], shell= True)\n \n elif sys.platform=='darwin':\n subprocess.Popen(['open', path])\n \n else:\n try:\n subprocess.Popen(['xdg-open', path])\n except OSError:\n pass", "response": "Opens Explorer or Finder with given path depending on platform"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_file_path(self, path):\n self.file_path.delete(0, END)\n self.file_path.insert(0, path)", "response": "Update the file_path Entry widget"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load(cls, path, format=None):\n path_was_string = isinstance(path, basestring)\n if path_was_string:\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n if format is None:\n plugin = kurt.plugin.Kurt.get_plugin(extension=extension)\n if not plugin:\n raise UnknownFormat(extension)\n fp = open(path, \"rb\")\n else:\n fp = path\n assert format, \"Format is required\"\n plugin = kurt.plugin.Kurt.get_plugin(format)\n\n if not plugin:\n raise ValueError, \"Unknown format %r\" % format\n\n project = plugin.load(fp)\n if path_was_string:\n fp.close()\n project.convert(plugin)\n if isinstance(path, basestring):\n project.path = path\n if not project.name:\n project.name = name\n return project", "response": "Load a Kurt project from file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new Project instance deep - copying all the attributes.", "response": "def copy(self):\n \"\"\"Return a new Project instance, deep-copying all the attributes.\"\"\"\n p = Project()\n p.name = self.name\n p.path = self.path\n p._plugin = self._plugin\n p.stage = self.stage.copy()\n p.stage.project = p\n\n for sprite in self.sprites:\n s = sprite.copy()\n s.project = p\n p.sprites.append(s)\n\n for actor in self.actors:\n if isinstance(actor, Sprite):\n p.actors.append(p.get_sprite(actor.name))\n else:\n a = actor.copy()\n if isinstance(a, Watcher):\n if isinstance(a.target, Project):\n a.target = p\n elif isinstance(a.target, Stage):\n a.target = p.stage\n else:\n a.target = p.get_sprite(a.target.name)\n p.actors.append(a)\n\n p.variables = dict((n, v.copy()) for (n, v) in self.variables.items())\n p.lists = dict((n, l.copy()) for (n, l) in self.lists.items())\n p.thumbnail = self.thumbnail\n p.tempo = self.tempo\n p.notes = self.notes\n p.author = self.author\n return p"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting the project in - place to a different file format.", "response": "def convert(self, format):\n \"\"\"Convert the project in-place to a different file format.\n\n Returns a list of :class:`UnsupportedFeature` objects, which may give\n warnings about the conversion.\n\n :param format: :attr:`KurtFileFormat.name` eg. ``\"scratch14\"``.\n\n :raises: :class:`ValueError` if the format doesn't exist.\n\n \"\"\"\n self._plugin = kurt.plugin.Kurt.get_plugin(format)\n return list(self._normalize())"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsaving the project to file.", "response": "def save(self, path=None, debug=False):\n \"\"\"Save project to file.\n\n :param path: Path or file pointer.\n\n If you pass a file pointer, you're responsible for closing\n it.\n\n If path is not given, the :attr:`path` attribute is used,\n usually the original path given to :attr:`load()`.\n\n If `path` has the extension of an existing plugin, the\n project will be converted using :attr:`convert`.\n Otherwise, the extension will be replaced with the\n extension of the current plugin.\n\n (Note that log output for the conversion will be printed\n to stdout. If you want to deal with the output, call\n :attr:`convert` directly.)\n\n If the path ends in a folder instead of a file, the\n filename is based on the project's :attr:`name`.\n\n :param debug: If true, return debugging information from the format\n plugin instead of the path.\n\n :raises: :py:class:`ValueError` if there's no path or name.\n\n :returns: path to the saved file.\n\n \"\"\"\n\n p = self.copy()\n plugin = p._plugin\n\n # require path\n p.path = path or self.path\n if not p.path:\n raise ValueError, \"path is required\"\n\n if isinstance(p.path, basestring):\n # split path\n (folder, filename) = os.path.split(p.path)\n (name, extension) = os.path.splitext(filename)\n\n # get plugin from extension\n if path: # only if not using self.path\n try:\n plugin = kurt.plugin.Kurt.get_plugin(extension=extension)\n except ValueError:\n pass\n\n # build output path\n if not name:\n name = _clean_filename(self.name)\n if not name:\n raise ValueError, \"name is required\"\n filename = name + plugin.extension\n p.path = os.path.join(folder, filename)\n\n # open\n fp = open(p.path, \"wb\")\n else:\n fp = p.path\n path = None\n\n if not plugin:\n raise ValueError, \"must convert project to a format before saving\"\n\n for m in p.convert(plugin):\n print m\n result = p._save(fp)\n if path:\n fp.close()\n return result if debug else p.path"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _normalize(self):\n\n unique_sprite_names = set(sprite.name for sprite in self.sprites)\n if len(unique_sprite_names) < len(self.sprites):\n raise ValueError, \"Sprite names must be unique\"\n\n # sync self.sprites and self.actors\n for sprite in self.sprites:\n if sprite not in self.actors:\n self.actors.append(sprite)\n for actor in self.actors:\n if isinstance(actor, Sprite):\n if actor not in self.sprites:\n raise ValueError, \\\n \"Can't have sprite on stage that isn't in sprites\"\n\n # normalize Scriptables\n self.stage._normalize()\n for sprite in self.sprites:\n sprite._normalize()\n\n # normalize actors\n for actor in self.actors:\n if not isinstance(actor, Scriptable):\n actor._normalize()\n\n # make Watchers if needed\n for thing in [self, self.stage] + self.sprites:\n for (name, var) in thing.variables.items():\n if not var.watcher:\n var.watcher = kurt.Watcher(thing,\n kurt.Block(\"var\", name), is_visible=False)\n self.actors.append(var.watcher)\n for (name, list_) in thing.lists.items():\n if not list_.watcher:\n list_.watcher = kurt.Watcher(thing,\n kurt.Block(\"list\", name), is_visible=False)\n self.actors.append(list_.watcher)\n\n # notes - line endings\n self.notes = self.notes.replace(\"\\r\\n\", \"\\n\").replace(\"\\r\", \"\\n\")\n\n # convert scripts\n def convert_block(block):\n # convert block\n try:\n if isinstance(block.type, CustomBlockType):\n if \"Custom Blocks\" not in self._plugin.features:\n raise BlockNotSupported(\n \"%s doesn't support custom blocks\"\n % self._plugin.display_name)\n\n else: # BlockType\n pbt = block.type.convert(self._plugin)\n except BlockNotSupported, err:\n err.message += \". Caused by: %r\" % block\n err.block = block\n err.scriptable = scriptable\n err.args = (err.message,)\n if getattr(block.type, '_workaround', None):\n block = block.type._workaround(block)\n if not block:\n raise\n else:\n raise\n\n # convert args\n args = []\n for arg in block.args:\n if isinstance(arg, Block):\n arg = convert_block(arg)\n elif isinstance(arg, list):\n arg = map(convert_block, arg)\n args.append(arg)\n block.args = args\n\n return block\n\n for scriptable in [self.stage] + self.sprites:\n for script in scriptable.scripts:\n if isinstance(script, Script):\n script.blocks = map(convert_block, script.blocks)\n\n # workaround unsupported features\n for feature in kurt.plugin.Feature.FEATURES.values():\n if feature not in self._plugin.features:\n for x in feature.workaround(self):\n yield UnsupportedFeature(feature, x)\n\n # normalize supported features\n for feature in self._plugin.features:\n feature.normalize(self)", "response": "Convert the project to a standardised form for the current plugin."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef copy(self, o=None):\n if o is None: o = self.__class__(self.project)\n o.scripts = [s.copy() for s in self.scripts]\n o.variables = dict((n, v.copy()) for (n, v) in self.variables.items())\n o.lists = dict((n, l.copy()) for (n, l) in self.lists.items())\n o.costumes = [c.copy() for c in self.costumes]\n o.sounds = [s.copy() for s in self.sounds]\n o.costume_index = self.costume_index\n o.volume = self.volume\n return o", "response": "Return a new instance deep - copying all the attributes."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the given code and add it to the scripts list.", "response": "def parse(self, text):\n \"\"\"Parse the given code and add it to :attr:`scripts`.\n\n The syntax matches :attr:`Script.stringify()`. See :mod:`kurt.text` for\n reference.\n\n \"\"\"\n self.scripts.append(kurt.text.parse(text, self))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new instance deep - copying all the attributes.", "response": "def copy(self):\n \"\"\"Return a new instance, deep-copying all the attributes.\"\"\"\n o = self.__class__(self.project, self.name)\n Scriptable.copy(self, o)\n o.position = tuple(self.position)\n o.direction = self.direction\n o.rotation_style = self.rotation_style\n o.size = self.size\n o.is_draggable = self.is_draggable\n o.is_visible = self.is_visible\n return o"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a copy of the current instance with the same attributes.", "response": "def copy(self):\n \"\"\"Return a new instance with the same attributes.\"\"\"\n o = self.__class__(self.target,\n self.block.copy(),\n self.style,\n self.is_visible,\n self.pos)\n o.slider_min = self.slider_min\n o.slider_max = self.slider_max\n return o"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef kind(self):\n if self.block.type.has_command('readVariable'):\n return 'variable'\n elif self.block.type.has_command('contentsOfList:'):\n return 'list'\n else:\n return 'block'", "response": "The type of value to watch based on the block type."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef value(self):\n if self.kind == 'variable':\n return self.target.variables[self.block.args[0]]\n elif self.kind == 'list':\n return self.target.lists[self.block.args[0]]", "response": "Return the value of the attribute."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stringify(self):\n hexcode = \"#\"\n for x in self.value:\n part = hex(x)[2:]\n if len(part) < 2: part = \"0\" + part\n hexcode += part\n return hexcode", "response": "Returns the color value in hexcode format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef options(self, scriptable=None):\n options = list(Insert.KIND_OPTIONS.get(self.kind, []))\n if scriptable:\n if self.kind == 'var':\n options += scriptable.variables.keys()\n options += scriptable.project.variables.keys()\n elif self.kind == 'list':\n options += scriptable.lists.keys()\n options += scriptable.project.lists.keys()\n elif self.kind == 'costume':\n options += [c.name for c in scriptable.costumes]\n elif self.kind == 'backdrop':\n options += [c.name for c in scriptable.project.stage.costumes]\n elif self.kind == 'sound':\n options += [c.name for c in scriptable.sounds]\n options += [c.name for c in scriptable.project.stage.sounds]\n elif self.kind in ('spriteOnly', 'spriteOrMouse', 'spriteOrStage',\n 'touching'):\n options += [s.name for s in scriptable.project.sprites]\n elif self.kind == 'attribute':\n pass # TODO\n elif self.kind == 'broadcast':\n options += list(set(scriptable.project.get_broadcasts()))\n return options", "response": "Return a list of valid options to a menu insert given a\n Scriptable for context."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stripped_text(self):\n return BaseBlockType._strip_text(\n self.text % tuple((i.default if i.shape == 'inline' else '%s')\n for i in self.inserts))", "response": "The text with spaces and inserts removed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning text with spaces and inserts removed.", "response": "def _strip_text(text):\n \"\"\"Returns text with spaces and inserts removed.\"\"\"\n text = re.sub(r'[ ,?:]|%s', \"\", text.lower())\n for chr in \"-%\":\n new_text = text.replace(chr, \"\")\n if new_text:\n text = new_text\n return text.lower()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn True if any of the inserts have the given shape.", "response": "def has_insert(self, shape):\n \"\"\"Returns True if any of the inserts have the given shape.\"\"\"\n for insert in self.inserts:\n if insert.shape == shape:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _add_conversion(self, plugin, pbt):\n assert self.shape == pbt.shape\n assert len(self.inserts) == len(pbt.inserts)\n for (i, o) in zip(self.inserts, pbt.inserts):\n assert i.shape == o.shape\n assert i.kind == o.kind\n assert i.unevaluated == o.unevaluated\n if plugin not in self._plugins:\n self._plugins[plugin] = pbt", "response": "Add a new conversion."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert(self, plugin=None):\n if plugin:\n plugin = kurt.plugin.Kurt.get_plugin(plugin)\n if plugin.name in self._plugins:\n return self._plugins[plugin.name]\n else:\n err = BlockNotSupported(\"%s doesn't have %r\" %\n (plugin.display_name, self))\n err.block_type = self\n raise err\n else:\n return self.conversions[0]", "response": "Return a : class : PluginBlockType for the given plugin name."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the plugin supports this block.", "response": "def has_conversion(self, plugin):\n \"\"\"Return True if the plugin supports this block.\"\"\"\n plugin = kurt.plugin.Kurt.get_plugin(plugin)\n return plugin.name in self._plugins"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef has_command(self, command):\n for pbt in self._plugins.values():\n if pbt.command == command:\n return True\n return False", "response": "Returns True if any of the plugins have the given command."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a : class : BlockType instance from the given parameter.", "response": "def get(cls, block_type):\n \"\"\"Return a :class:`BlockType` instance from the given parameter.\n\n * If it's already a BlockType instance, return that.\n\n * If it exactly matches the command on a :class:`PluginBlockType`,\n return the corresponding BlockType.\n\n * If it loosely matches the text on a PluginBlockType, return the\n corresponding BlockType.\n\n * If it's a PluginBlockType instance, look for and return the\n corresponding BlockType.\n\n \"\"\"\n if isinstance(block_type, (BlockType, CustomBlockType)):\n return block_type\n\n if isinstance(block_type, PluginBlockType):\n block_type = block_type.command\n\n block = kurt.plugin.Kurt.block_by_command(block_type)\n if block:\n return block\n\n blocks = kurt.plugin.Kurt.blocks_by_text(block_type)\n for block in blocks: # check the blocks' commands map to unique blocks\n if kurt.plugin.Kurt.block_by_command(\n block.convert().command) != blocks[0]:\n raise ValueError(\n \"ambigious block text %r, use one of %r instead\" %\n (block_type, [b.convert().command for b in blocks]))\n\n if blocks:\n return blocks[0]\n\n raise UnknownBlock, repr(block_type)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef copy(self):\n args = []\n for arg in self.args:\n if isinstance(arg, Block):\n arg = arg.copy()\n elif isinstance(arg, list):\n arg = [b.copy() for b in arg]\n args.append(arg)\n return Block(self.type, *args)", "response": "Return a new Block instance with the same attributes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a new instance with the same attributes.", "response": "def copy(self):\n \"\"\"Return a new instance with the same attributes.\"\"\"\n return self.__class__([b.copy() for b in self.blocks],\n tuple(self.pos) if self.pos else None)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load(self, path):\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n return Costume(name, Image.load(path))", "response": "Load a Costume from an image file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pil_image(self):\n if not self._pil_image:\n if self._format == \"SVG\":\n raise VectorImageError(\"can't rasterise vector images\")\n self._pil_image = PIL.Image.open(StringIO(self.contents))\n return self._pil_image", "response": "A : class : PIL. Image instance containing the image data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the format of the image file.", "response": "def format(self):\n \"\"\"The format of the image file.\n\n An uppercase string corresponding to the\n :attr:`PIL.ImageFile.ImageFile.format` attribute. Valid values include\n ``\"JPEG\"`` and ``\"PNG\"``.\n\n \"\"\"\n if self._format:\n return self._format\n elif self.pil_image:\n return self.pil_image.format"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the size of the image.", "response": "def size(self):\n \"\"\"``(width, height)`` in pixels.\"\"\"\n if self._size and not self._pil_image:\n return self._size\n else:\n return self.pil_image.size"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads image from file.", "response": "def load(cls, path):\n \"\"\"Load image from file.\"\"\"\n assert os.path.exists(path), \"No such file: %r\" % path\n\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n\n image = Image(None)\n image._path = path\n image._format = Image.image_format(extension)\n\n return image"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef convert(self, *formats):\n for format in formats:\n format = Image.image_format(format)\n if self.format == format:\n return self\n else:\n return self._convert(format)", "response": "Convert an image to one of the given formats."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _convert(self, format):\n if self.format == format:\n return self\n else:\n image = Image(self.pil_image)\n image._format = format\n return image", "response": "Return a new Image instance with the given format."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving the image to file path.", "response": "def save(self, path):\n \"\"\"Save image to file path.\n\n The image format is guessed from the extension. If path has no\n extension, the image's :attr:`format` is used.\n\n :returns: Path to the saved file.\n\n \"\"\"\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n\n if not name:\n raise ValueError, \"name is required\"\n\n if extension:\n format = Image.image_format(extension)\n else:\n format = self.format\n filename = name + self.extension\n path = os.path.join(folder, filename)\n\n image = self.convert(format)\n if image._contents:\n f = open(path, \"wb\")\n f.write(image._contents)\n f.close()\n else:\n image.pil_image.save(path, format)\n\n return path"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef new(self, size, fill):\n return Image(PIL.Image.new(\"RGB\", size, fill))", "response": "Return a new Image instance filled with a color."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef resize(self, size):\n return Image(self.pil_image.resize(size, PIL.Image.ANTIALIAS))", "response": "Return a new Image instance with the given size."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new Image with the given image pasted on top.", "response": "def paste(self, other):\n \"\"\"Return a new Image with the given image pasted on top.\n\n This image will show through transparent areas of the given image.\n\n \"\"\"\n r, g, b, alpha = other.pil_image.split()\n pil_image = self.pil_image.copy()\n pil_image.paste(other.pil_image, mask=alpha)\n return kurt.Image(pil_image)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load(self, path):\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n return Sound(name, Waveform.load(path))", "response": "Load a sound from a wave file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save(self, path):\n (folder, filename) = os.path.split(path)\n if not filename:\n filename = _clean_filename(self.name)\n path = os.path.join(folder, filename)\n return self.waveform.save(path)", "response": "Save the sound to a wave file at the given path."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef contents(self):\n if not self._contents:\n if self._path:\n # Read file into memory so we don't run out of file descriptors\n f = open(self._path, \"rb\")\n self._contents = f.read()\n f.close()\n return self._contents", "response": "The raw file contents as a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a wave. Wave_read instance from the wave module.", "response": "def _wave(self):\n \"\"\"Return a wave.Wave_read instance from the ``wave`` module.\"\"\"\n try:\n return wave.open(StringIO(self.contents))\n except wave.Error, err:\n err.message += \"\\nInvalid wave file: %s\" % self\n err.args = (err.message,)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading Waveform from file.", "response": "def load(cls, path):\n \"\"\"Load Waveform from file.\"\"\"\n assert os.path.exists(path), \"No such file: %r\" % path\n\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n\n wave = Waveform(None)\n wave._path = path\n return wave"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self, path):\n (folder, filename) = os.path.split(path)\n (name, extension) = os.path.splitext(filename)\n\n if not name:\n raise ValueError, \"name is required\"\n\n path = os.path.join(folder, name + self.extension)\n f = open(path, \"wb\")\n f.write(self.contents)\n f.close()\n\n return path", "response": "Save the waveform to file path as a WAV file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsort the given list in the way that humans expect.", "response": "def sort_nicely(l):\n \"\"\"Sort the given list in the way that humans expect.\"\"\"\n convert = lambda text: int(text) if text.isdigit() else text\n alphanum_key = lambda key: [convert(c) for c in re.split('([0-9]+)', key)]\n l.sort(key=alphanum_key)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nopens a new channel on this connection.", "response": "def open_channel(self):\n \"\"\"\n Open a new channel on this connection.\n\n This method is a :ref:`coroutine `.\n\n :return: The new :class:`Channel` object.\n \"\"\"\n if self._closing:\n raise ConnectionClosed(\"Closed by application\")\n if self.closed.done():\n raise self.closed.exception()\n\n channel = yield from self.channel_factory.open()\n return channel"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncloses the connection by handshaking with the server.", "response": "def close(self):\n \"\"\"\n Close the connection by handshaking with the server.\n\n This method is a :ref:`coroutine `.\n \"\"\"\n if not self.is_closed():\n self._closing = True\n # Let the ConnectionActor do the actual close operations.\n # It will do the work on CloseOK\n self.sender.send_Close(\n 0, 'Connection closed by application', 0, 0)\n try:\n yield from self.synchroniser.wait(spec.ConnectionCloseOK)\n except AMQPConnectionError:\n # For example if both sides want to close or the connection\n # is closed.\n pass\n else:\n if self._closing:\n log.warn(\"Called `close` on already closing connection...\")\n # finish all pending tasks\n yield from self.protocol.heartbeat_monitor.wait_closed()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nhandle a Poison Pill frame.", "response": "def handle_PoisonPillFrame(self, frame):\n \"\"\" Is sent in case protocol lost connection to server.\"\"\"\n # Will be delivered after Close or CloseOK handlers. It's for channels,\n # so ignore it.\n if self.connection.closed.done():\n return\n # If connection was not closed already - we lost connection.\n # Protocol should already be closed\n self._close_all(frame.exception)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef handle_ConnectionClose(self, frame):\n # Notify server we are OK to close.\n self.sender.send_CloseOK()\n\n exc = ConnectionClosed(frame.payload.reply_text,\n frame.payload.reply_code)\n self._close_all(exc)\n # This will not abort transport, it will try to flush remaining data\n # asynchronously, as stated in `asyncio` docs.\n self.protocol.close()", "response": "Handle a ConnectionClose frame."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef publish(self, message, routing_key, *, mandatory=True):\n self.sender.send_BasicPublish(self.name, routing_key, mandatory, message)", "response": "Publish a message on the exchange to be asynchronously delivered to queues."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes the exchange. This method is a :ref:`coroutine `. :keyword bool if_unused: If true, the exchange will only be deleted if it has no queues bound to it.", "response": "def delete(self, *, if_unused=True):\n \"\"\"\n Delete the exchange.\n\n This method is a :ref:`coroutine `.\n\n :keyword bool if_unused: If true, the exchange will only be deleted if\n it has no queues bound to it.\n \"\"\"\n self.sender.send_ExchangeDelete(self.name, if_unused)\n yield from self.synchroniser.wait(spec.ExchangeDeleteOK)\n self.reader.ready()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reject(self, *, requeue=True):\n self.sender.send_BasicReject(self.delivery_tag, requeue)", "response": "Reject the message.\n\n :keyword bool requeue: if true, the broker will attempt to requeue the\n message and deliver it to an alternate consumer."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef declare_exchange(self, name, type, *, durable=True, auto_delete=False,\n passive=False, internal=False, nowait=False,\n arguments=None):\n \"\"\"\n Declare an :class:`Exchange` on the broker. If the exchange does not exist, it will be created.\n\n This method is a :ref:`coroutine `.\n\n :param str name: the name of the exchange.\n :param str type: the type of the exchange\n (usually one of ``'fanout'``, ``'direct'``, ``'topic'``, or ``'headers'``)\n :keyword bool durable: If true, the exchange will be re-created when\n the server restarts.\n :keyword bool auto_delete: If true, the exchange will be\n deleted when the last queue is un-bound from it.\n :keyword bool passive: If `true` and exchange with such a name does\n not exist it will raise a :class:`exceptions.NotFound`. If `false`\n server will create it. Arguments ``durable``, ``auto_delete`` and\n ``internal`` are ignored if `passive=True`.\n :keyword bool internal: If true, the exchange cannot be published to\n directly; it can only be bound to other exchanges.\n :keyword bool nowait: If true, the method will not wait for declare-ok\n to arrive and return right away.\n :keyword dict arguments: Table of optional parameters for extensions to\n the AMQP protocol. See :ref:`extensions`.\n\n :return: the new :class:`Exchange` object.\n \"\"\"\n if name == '':\n return exchange.Exchange(self.reader, self.synchroniser, self.sender, name, 'direct', True, False, False)\n\n if not VALID_EXCHANGE_NAME_RE.match(name):\n raise ValueError(\n \"Invalid exchange name.\\n\"\n \"Valid names consist of letters, digits, hyphen, underscore, \"\n \"period, or colon, and do not begin with 'amq.'\")\n\n self.sender.send_ExchangeDeclare(\n name, type, passive, durable, auto_delete, internal, nowait,\n arguments or {})\n if not nowait:\n yield from self.synchroniser.wait(spec.ExchangeDeclareOK)\n self.reader.ready()\n ex = exchange.Exchange(\n self.reader, self.synchroniser, self.sender, name, type, durable,\n auto_delete, internal)\n return ex", "response": "Declare an exchange on the broker."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndeclare a queue on the broker.", "response": "def declare_queue(self, name='', *, durable=True, exclusive=False,\n auto_delete=False, passive=False,\n nowait=False, arguments=None):\n \"\"\"\n Declare a queue on the broker. If the queue does not exist, it will be created.\n\n This method is a :ref:`coroutine `.\n\n :param str name: the name of the queue.\n Supplying a name of '' will create a queue with a unique name of the server's choosing.\n :keyword bool durable: If true, the queue will be re-created when the server restarts.\n :keyword bool exclusive: If true, the queue can only be accessed by the current connection,\n and will be deleted when the connection is closed.\n :keyword bool auto_delete: If true, the queue will be deleted when the last consumer is cancelled.\n If there were never any conusmers, the queue won't be deleted.\n :keyword bool passive: If true and queue with such a name does not\n exist it will raise a :class:`exceptions.NotFound` instead of\n creating it. Arguments ``durable``, ``auto_delete`` and\n ``exclusive`` are ignored if ``passive=True``.\n :keyword bool nowait: If true, will not wait for a declare-ok to arrive.\n :keyword dict arguments: Table of optional parameters for extensions to the AMQP protocol. See :ref:`extensions`.\n\n :return: The new :class:`Queue` object.\n \"\"\"\n q = yield from self.queue_factory.declare(\n name, durable, exclusive, auto_delete, passive, nowait,\n arguments if arguments is not None else {})\n return q"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_qos(self, prefetch_size=0, prefetch_count=0, apply_globally=False):\n self.sender.send_BasicQos(prefetch_size, prefetch_count, apply_globally)\n yield from self.synchroniser.wait(spec.BasicQosOK)\n self.reader.ready()", "response": "Set the quality of service for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclose the channel by handshaking with the server.", "response": "def close(self):\n \"\"\"\n Close the channel by handshaking with the server.\n\n This method is a :ref:`coroutine `.\n \"\"\"\n # If we aren't already closed ask for server to close\n if not self.is_closed():\n self._closing = True\n # Let the ChannelActor do the actual close operations.\n # It will do the work on CloseOK\n self.sender.send_Close(\n 0, 'Channel closed by application', 0, 0)\n try:\n yield from self.synchroniser.wait(spec.ChannelCloseOK)\n except AMQPError:\n # For example if both sides want to close or the connection\n # is closed.\n pass\n else:\n if self._closing:\n log.warn(\"Called `close` on already closing channel...\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef handle_ChannelClose(self, frame):\n # By docs:\n # The response to receiving a Close after sending Close must be to\n # send Close-Ok.\n #\n # No need for additional checks\n\n self.sender.send_CloseOK()\n exc = exceptions._get_exception_type(frame.payload.reply_code)\n self._close_all(exc)", "response": "Handle a channel close."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nhandles a ChannelCloseOK frame.", "response": "def handle_ChannelCloseOK(self, frame):\n \"\"\" AMQP server closed channel as per our request \"\"\"\n assert self.channel._closing, \"received a not expected CloseOk\"\n # Release the `close` method's future\n self.synchroniser.notify(spec.ChannelCloseOK)\n\n exc = ChannelClosed()\n self._close_all(exc)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbind a queue to an exchange with the supplied routing key.", "response": "def bind(self, exchange, routing_key, *, arguments=None):\n \"\"\"\n Bind a queue to an exchange, with the supplied routing key.\n\n This action 'subscribes' the queue to the routing key; the precise meaning of this\n varies with the exchange type.\n\n This method is a :ref:`coroutine `.\n\n :param asynqp.Exchange exchange: the :class:`Exchange` to bind to\n :param str routing_key: the routing key under which to bind\n :keyword dict arguments: Table of optional parameters for extensions to the AMQP protocol. See :ref:`extensions`.\n\n :return: The new :class:`QueueBinding` object\n \"\"\"\n if self.deleted:\n raise Deleted(\"Queue {} was deleted\".format(self.name))\n if not exchange:\n raise InvalidExchangeName(\"Can't bind queue {} to the default exchange\".format(self.name))\n\n self.sender.send_QueueBind(self.name, exchange.name, routing_key, arguments or {})\n yield from self.synchroniser.wait(spec.QueueBindOK)\n b = QueueBinding(self.reader, self.sender, self.synchroniser, self, exchange, routing_key)\n self.reader.ready()\n return b"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef consume(self, callback, *, no_local=False, no_ack=False, exclusive=False, arguments=None):\n if self.deleted:\n raise Deleted(\"Queue {} was deleted\".format(self.name))\n\n self.sender.send_BasicConsume(self.name, no_local, no_ack, exclusive, arguments or {})\n tag = yield from self.synchroniser.wait(spec.BasicConsumeOK)\n consumer = Consumer(\n tag, callback, self.sender, self.synchroniser, self.reader,\n loop=self._loop)\n self.consumers.add_consumer(consumer)\n self.reader.ready()\n return consumer", "response": "Start a consumer on the queue. Messages will be delivered asynchronously to the consumer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, *, no_ack=False):\n if self.deleted:\n raise Deleted(\"Queue {} was deleted\".format(self.name))\n\n self.sender.send_BasicGet(self.name, no_ack)\n tag_msg = yield from self.synchroniser.wait(spec.BasicGetOK, spec.BasicGetEmpty)\n\n if tag_msg is not None:\n consumer_tag, msg = tag_msg\n assert consumer_tag is None\n else:\n msg = None\n self.reader.ready()\n return msg", "response": "Synchronously get a message from the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef purge(self):\n self.sender.send_QueuePurge(self.name)\n yield from self.synchroniser.wait(spec.QueuePurgeOK)\n self.reader.ready()", "response": "Purge all undelivered messages from the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes the queue. This method is a :ref:`coroutine `. :keyword bool if_unused: If true, the queue will only be deleted if it has no consumers. :keyword bool if_empty: If true, the queue will only be deleted if it has no unacknowledged messages.", "response": "def delete(self, *, if_unused=True, if_empty=True):\n \"\"\"\n Delete the queue.\n\n This method is a :ref:`coroutine `.\n\n :keyword bool if_unused: If true, the queue will only be deleted\n if it has no consumers.\n :keyword bool if_empty: If true, the queue will only be deleted if\n it has no unacknowledged messages.\n \"\"\"\n if self.deleted:\n raise Deleted(\"Queue {} was already deleted\".format(self.name))\n\n self.sender.send_QueueDelete(self.name, if_unused, if_empty)\n yield from self.synchroniser.wait(spec.QueueDeleteOK)\n self.deleted = True\n self.reader.ready()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nunbind the queue from the exchange.", "response": "def unbind(self, arguments=None):\n \"\"\"\n Unbind the queue from the exchange.\n\n This method is a :ref:`coroutine `.\n \"\"\"\n if self.deleted:\n raise Deleted(\"Queue {} was already unbound from exchange {}\".format(self.queue.name, self.exchange.name))\n\n self.sender.send_QueueUnbind(self.queue.name, self.exchange.name, self.routing_key, arguments or {})\n yield from self.synchroniser.wait(spec.QueueUnbindOK)\n self.deleted = True\n self.reader.ready()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cancel(self):\n self.sender.send_BasicCancel(self.tag)\n try:\n yield from self.synchroniser.wait(spec.BasicCancelOK)\n except AMQPError:\n pass\n else:\n # No need to call ready if channel closed.\n self.reader.ready()\n self.cancelled = True\n self.cancelled_future.set_result(self)\n if hasattr(self.callback, 'on_cancel'):\n self.callback.on_cancel()", "response": "Cancel the consumer and stop recieving messages."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef hello_world():\n # connect to the RabbitMQ broker\n connection = yield from asynqp.connect('localhost', 5672, username='guest', password='guest')\n\n # Open a communications channel\n channel = yield from connection.open_channel()\n\n # Create a queue and an exchange on the broker\n exchange = yield from channel.declare_exchange('test.exchange', 'direct')\n queue = yield from channel.declare_queue('test.queue')\n\n # Bind the queue to the exchange, so the queue will get messages published to the exchange\n yield from queue.bind(exchange, 'routing.key')\n\n # If you pass in a dict it will be automatically converted to JSON\n msg = asynqp.Message({'hello': 'world'})\n exchange.publish(msg, 'routing.key')\n\n # Synchronously get a message from the queue\n received_message = yield from queue.get()\n print(received_message.json()) # get JSON from incoming messages easily\n\n # Acknowledge a delivered message\n received_message.ack()\n\n yield from channel.close()\n yield from connection.close()", "response": "Send a hello world message and then reads it from the queue."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nkill all ongoing requests and clear the internal state.", "response": "def killall(self, exc):\n \"\"\" Connection/Channel was closed. All subsequent and ongoing requests\n should raise an error\n \"\"\"\n self.connection_exc = exc\n # Set an exception for all others\n for method, futs in self._futures.items():\n for fut in futs:\n if fut.done():\n continue\n fut.set_exception(exc)\n self._futures.clear()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef connect(host='localhost',\n port=5672,\n username='guest', password='guest',\n virtual_host='/',\n on_connection_close=None, *,\n loop=None, sock=None, **kwargs):\n \"\"\"\n Connect to an AMQP server on the given host and port.\n\n Log in to the given virtual host using the supplied credentials.\n This function is a :ref:`coroutine `.\n\n :param str host: the host server to connect to.\n :param int port: the port which the AMQP server is listening on.\n :param str username: the username to authenticate with.\n :param str password: the password to authenticate with.\n :param str virtual_host: the AMQP virtual host to connect to.\n :param func on_connection_close: function called after connection lost.\n :keyword BaseEventLoop loop: An instance of :class:`~asyncio.BaseEventLoop` to use.\n (Defaults to :func:`asyncio.get_event_loop()`)\n :keyword socket sock: A :func:`~socket.socket` instance to use for the connection.\n This is passed on to :meth:`loop.create_connection() `.\n If ``sock`` is supplied then ``host`` and ``port`` will be ignored.\n\n Further keyword arguments are passed on to :meth:`loop.create_connection() `.\n\n This function will set TCP_NODELAY on TCP and TCP6 sockets either on supplied ``sock`` or created one.\n\n :return: the :class:`Connection` object.\n \"\"\"\n from .protocol import AMQP\n from .routing import Dispatcher\n from .connection import open_connection\n\n loop = asyncio.get_event_loop() if loop is None else loop\n\n if sock is None:\n kwargs['host'] = host\n kwargs['port'] = port\n else:\n kwargs['sock'] = sock\n\n dispatcher = Dispatcher()\n\n def protocol_factory():\n return AMQP(dispatcher, loop, close_callback=on_connection_close)\n transport, protocol = yield from loop.create_connection(protocol_factory, **kwargs)\n\n # RPC-like applications require TCP_NODELAY in order to acheive\n # minimal response time. Actually, this library send data in one\n # big chunk and so this will not affect TCP-performance.\n sk = transport.get_extra_info('socket')\n # 1. Unfortunatelly we cannot check socket type (sk.type == socket.SOCK_STREAM). https://bugs.python.org/issue21327\n # 2. Proto remains zero, if not specified at creation of socket\n if (sk.family in (socket.AF_INET, socket.AF_INET6)) and (sk.proto in (0, socket.IPPROTO_TCP)):\n sk.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)\n\n connection_info = {\n 'username': username,\n 'password': password,\n 'virtual_host': virtual_host\n }\n connection = yield from open_connection(\n loop, transport, protocol, dispatcher, connection_info)\n return connection", "response": "Connect to an AMQP server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef connect_and_open_channel(host='localhost',\n port=5672,\n username='guest', password='guest',\n virtual_host='/',\n on_connection_close=None, *,\n loop=None, **kwargs):\n \"\"\"\n Connect to an AMQP server and open a channel on the connection.\n This function is a :ref:`coroutine `.\n\n Parameters of this function are the same as :func:`connect`.\n\n :return: a tuple of ``(connection, channel)``.\n\n Equivalent to::\n\n connection = yield from connect(host, port, username, password, virtual_host, on_connection_close, loop=loop, **kwargs)\n channel = yield from connection.open_channel()\n return connection, channel\n \"\"\"\n connection = yield from connect(host, port, username, password, virtual_host, on_connection_close, loop=loop, **kwargs)\n channel = yield from connection.open_channel()\n return connection, channel", "response": "Connect to an AMQP server and open a channel on the connection."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef heartbeat_timeout(self):\n assert not self._closed, \"Did we not stop heartbeat_monitor on close?\"\n log.error(\"Heartbeat time out\")\n poison_exc = ConnectionLostError('Heartbeat timed out')\n poison_frame = frames.PoisonPillFrame(poison_exc)\n self.dispatcher.dispatch_all(poison_frame)\n # Spec says to just close socket without ConnectionClose handshake.\n self.close()", "response": "Called by heartbeat_monitor when heartbeat_timeout expires."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntaking a python object and convert it to the format Imgur expects.", "response": "def convert_general(value):\n \"\"\"Take a python object and convert it to the format Imgur expects.\"\"\"\n if isinstance(value, bool):\n return \"true\" if value else \"false\"\n elif isinstance(value, list):\n value = [convert_general(item) for item in value]\n value = convert_to_imgur_list(value)\n elif isinstance(value, Integral):\n return str(value)\n elif 'pyimgur' in str(type(value)):\n return str(getattr(value, 'id', value))\n return value"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts the parameters to the format Imgur expects.", "response": "def to_imgur_format(params):\n \"\"\"Convert the parameters to the format Imgur expects.\"\"\"\n if params is None:\n return None\n return dict((k, convert_general(val)) for (k, val) in params.items())"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nhandles a few expected values for rendering the current choice.", "response": "def render(self, name, value, attrs=None, *args, **kwargs):\n \"\"\"Handle a few expected values for rendering the current choice.\n\n Extracts the state name from StateWrapper and State object.\n \"\"\"\n if isinstance(value, base.StateWrapper):\n state_name = value.state.name\n elif isinstance(value, base.State):\n state_name = value.name\n else:\n state_name = str(value)\n return super(StateSelect, self).render(name, state_name, attrs, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef contribute_to_class(self, cls, name):\n super(StateField, self).contribute_to_class(cls, name)\n\n parent_property = getattr(cls, self.name, None)\n setattr(cls, self.name, StateFieldProperty(self, parent_property))", "response": "Contributes the state to a Model."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the DB - stored value into a Python value.", "response": "def to_python(self, value):\n \"\"\"Converts the DB-stored value into a Python value.\"\"\"\n if isinstance(value, base.StateWrapper):\n res = value\n else:\n if isinstance(value, base.State):\n state = value\n elif value is None:\n state = self.workflow.initial_state\n else:\n try:\n state = self.workflow.states[value]\n except KeyError:\n raise exceptions.ValidationError(self.error_messages['invalid'])\n res = base.StateWrapper(state, self.workflow)\n\n if res.state not in self.workflow.states:\n raise exceptions.ValidationError(self.error_messages['invalid'])\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a value to DB storage.", "response": "def get_db_prep_value(self, value, connection, prepared=False):\n \"\"\"Convert a value to DB storage.\n\n Returns the state name.\n \"\"\"\n if not prepared:\n value = self.get_prep_value(value)\n return value.state.name"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef value_to_string(self, obj):\n statefield = self.to_python(self.value_from_object(obj))\n return statefield.state.name", "response": "Convert a field value to a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvalidating that a given value is a valid option for a given model instance.", "response": "def validate(self, value, model_instance):\n \"\"\"Validate that a given value is a valid option for a given model instance.\n\n Args:\n value (xworkflows.base.StateWrapper): The base.StateWrapper returned by to_python.\n model_instance: A WorkflowEnabled instance\n \"\"\"\n if not isinstance(value, base.StateWrapper):\n raise exceptions.ValidationError(self.error_messages['wrong_type'] % value)\n elif not value.workflow == self.workflow:\n raise exceptions.ValidationError(self.error_messages['wrong_workflow'] % value.workflow)\n elif value.state not in self.workflow.states:\n raise exceptions.ValidationError(self.error_messages['invalid_state'] % value.state)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind workflow definition in a WorkflowEnabled definition.", "response": "def _find_workflows(mcs, attrs):\n \"\"\"Find workflow definition(s) in a WorkflowEnabled definition.\n\n This method overrides the default behavior from xworkflows in order to\n use our custom StateField objects.\n \"\"\"\n workflows = {}\n for k, v in attrs.items():\n if isinstance(v, StateField):\n workflows[k] = v\n return workflows"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncaching for fetching the actual log model object once django is loaded.", "response": "def _get_log_model_class(self):\n \"\"\"Cache for fetching the actual log model object once django is loaded.\n\n Otherwise, import conflict occur: WorkflowEnabled imports \n which tries to import all models to retrieve the proper model class.\n \"\"\"\n if self.log_model_class is not None:\n return self.log_model_class\n\n app_label, model_label = self.log_model.rsplit('.', 1)\n self.log_model_class = apps.get_model(app_label, model_label)\n return self.log_model_class"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlog the transition into the database.", "response": "def db_log(self, transition, from_state, instance, *args, **kwargs):\n \"\"\"Logs the transition into the database.\"\"\"\n if self.log_model:\n model_class = self._get_log_model_class()\n\n extras = {}\n for db_field, transition_arg, default in model_class.EXTRA_LOG_ATTRIBUTES:\n extras[db_field] = kwargs.get(transition_arg, default)\n\n return model_class.log_transition(\n modified_object=instance,\n transition=transition.name,\n from_state=from_state.name,\n to_state=transition.target.name,\n **extras)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef clean_readme(fname):\n with codecs.open(fname, 'r', 'utf-8') as f:\n return ''.join(\n re.sub(r':\\w+:`([^`]+?)( <[^<>]+>)?`', r'``\\1``', line)\n for line in f\n if not (line.startswith('.. currentmodule') or line.startswith('.. toctree'))\n )", "response": "Cleanup README. rst for proper PyPI formatting."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_album_or_image(json, imgur):\n if json['is_album']:\n return Gallery_album(json, imgur, has_fetched=False)\n return Gallery_image(json, imgur)", "response": "Return a gallery image or album depending on what the json represent."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef refresh(self):\n resp = self._imgur._send_request(self._INFO_URL)\n self._populate(resp)\n self._has_fetched = True", "response": "Refresh the attributes of this object to the newest values."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds images to the album.", "response": "def add_images(self, images):\n \"\"\"\n Add images to the album.\n\n :param images: A list of the images we want to add to the album. Can be\n Image objects, ids or a combination of the two. Images that you\n cannot add (non-existing or not owned by you) will not cause\n exceptions, but fail silently.\n \"\"\"\n url = self._imgur._base_url + \"/3/album/{0}/add\".format(self.id)\n params = {'ids': images}\n return self._imgur._send_request(url, needs_auth=True, params=params,\n method=\"POST\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nremoving images from the album.", "response": "def remove_images(self, images):\n \"\"\"\n Remove images from the album.\n\n :param images: A list of the images we want to remove from the album.\n Can be Image objects, ids or a combination of the two. Images that\n you cannot remove (non-existing, not owned by you or not part of\n album) will not cause exceptions, but fail silently.\n \"\"\"\n url = (self._imgur._base_url + \"/3/album/{0}/\"\n \"remove_images\".format(self._delete_or_id_hash))\n # NOTE: Returns True and everything seem to be as it should in testing.\n # Seems most likely to be upstream bug.\n params = {'ids': images}\n return self._imgur._send_request(url, params=params, method=\"DELETE\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the images in this album.", "response": "def set_images(self, images):\n \"\"\"\n Set the images in this album.\n\n :param images: A list of the images we want the album to contain.\n Can be Image objects, ids or a combination of the two. Images that\n images that you cannot set (non-existing or not owned by you) will\n not cause exceptions, but fail silently.\n \"\"\"\n url = (self._imgur._base_url + \"/3/album/\"\n \"{0}/\".format(self._delete_or_id_hash))\n params = {'ids': images}\n return self._imgur._send_request(url, needs_auth=True, params=params,\n method=\"POST\")"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsubmit this item to the gallery.", "response": "def submit_to_gallery(self, title, bypass_terms=False):\n \"\"\"\n Add this to the gallery.\n\n Require that the authenticated user has accepted gallery terms and\n verified their email.\n\n :param title: The title of the new gallery item.\n :param bypass_terms: If the user has not accepted Imgur's terms yet,\n this method will return an error. Set this to True to by-pass the\n terms.\n \"\"\"\n url = self._imgur._base_url + \"/3/gallery/{0}\".format(self.id)\n payload = {'title': title, 'terms': '1' if bypass_terms else '0'}\n self._imgur._send_request(url, needs_auth=True, params=payload,\n method='POST')\n item = self._imgur.get_gallery_album(self.id)\n _change_object(self, item)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update(self, title=None, description=None, images=None, cover=None,\n layout=None, privacy=None):\n \"\"\"\n Update the album's information.\n\n Arguments with the value None will retain their old values.\n\n :param title: The title of the album.\n :param description: A description of the album.\n :param images: A list of the images we want the album to contain.\n Can be Image objects, ids or a combination of the two. Images that\n images that you cannot set (non-existing or not owned by you) will\n not cause exceptions, but fail silently.\n :param privacy: The albums privacy level, can be public, hidden or\n secret.\n :param cover: The id of the cover image.\n :param layout: The way the album is displayed, can be blog, grid,\n horizontal or vertical.\n \"\"\"\n url = (self._imgur._base_url + \"/3/album/\"\n \"{0}\".format(self._delete_or_id_hash))\n is_updated = self._imgur._send_request(url, params=locals(),\n method='POST')\n if is_updated:\n self.title = title or self.title\n self.description = description or self.description\n self.layout = layout or self.layout\n self.privacy = privacy or self.privacy\n if cover is not None:\n self.cover = (cover if isinstance(cover, Image)\n else Image({'id': cover}, self._imgur,\n has_fetched=False))\n if images:\n self.images = [img if isinstance(img, Image) else\n Image({'id': img}, self._imgur, False)\n for img in images]\n return is_updated", "response": "Update the album s information."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_replies(self):\n url = self._imgur._base_url + \"/3/comment/{0}/replies\".format(self.id)\n json = self._imgur._send_request(url)\n child_comments = json['children']\n return [Comment(com, self._imgur) for com in child_comments]", "response": "Get the replies to this comment."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes a top - level comment to this.", "response": "def comment(self, text):\n \"\"\"\n Make a top-level comment to this.\n\n :param text: The comment text.\n \"\"\"\n url = self._imgur._base_url + \"/3/comment\"\n payload = {'image_id': self.id, 'comment': text}\n resp = self._imgur._send_request(url, params=payload, needs_auth=True,\n method='POST')\n return Comment(resp, imgur=self._imgur, has_fetched=False)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef downvote(self):\n url = self._imgur._base_url + \"/3/gallery/{0}/vote/down\".format(self.id)\n return self._imgur._send_request(url, needs_auth=True, method='POST')", "response": "Downvote the item with the highest priority."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a list of the top - level comments.", "response": "def get_comments(self):\n \"\"\"Get a list of the top-level comments.\"\"\"\n url = self._imgur._base_url + \"/3/gallery/{0}/comments\".format(self.id)\n resp = self._imgur._send_request(url)\n return [Comment(com, self._imgur) for com in resp]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nremoving this image from the gallery.", "response": "def remove_from_gallery(self):\n \"\"\"Remove this image from the gallery.\"\"\"\n url = self._imgur._base_url + \"/3/gallery/{0}\".format(self.id)\n self._imgur._send_request(url, needs_auth=True, method='DELETE')\n if isinstance(self, Image):\n item = self._imgur.get_image(self.id)\n else:\n item = self._imgur.get_album(self.id)\n _change_object(self, item)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef download(self, path='', name=None, overwrite=False, size=None):\n def save_as(filename):\n local_path = os.path.join(path, filename)\n if os.path.exists(local_path) and not overwrite:\n raise Exception(\"Trying to save as {0}, but file \"\n \"already exists.\".format(local_path))\n with open(local_path, 'wb') as out_file:\n out_file.write(resp.content)\n return local_path\n valid_sizes = {'small_square': 's', 'big_square': 'b',\n 'small_thumbnail': 't', 'medium_thumbnail': 'm',\n 'large_thumbnail': 'l', 'huge_thumbnail': 'h'}\n if size is not None:\n size = size.lower().replace(' ', '_')\n if size not in valid_sizes:\n raise LookupError('Invalid size. Valid options are: {0}'.format(\n \", \" .join(valid_sizes.keys())))\n suffix = valid_sizes.get(size, '')\n base, sep, ext = self.link.rpartition('.')\n resp = requests.get(base + suffix + sep + ext)\n if name or self.title:\n try:\n return save_as((name or self.title) + suffix + sep + ext)\n except IOError:\n pass\n # Invalid filename\n return save_as(self.id + suffix + sep + ext)", "response": "Download the image.\n\n :param path: The image will be downloaded to the folder specified at\n path, if path is None (default) then the current working directory\n will be used.\n :param name: The name the image will be stored as (not including file\n extension). If name is None, then the title of the image will be\n used. If the image doesn't have a title, it's id will be used. Note\n that if the name given by name or title is an invalid filename,\n then the hash will be used as the name instead.\n :param overwrite: If True overwrite already existing file with the same\n name as what we want to save the file as.\n :param size: Instead of downloading the image in it's original size, we\n can choose to instead download a thumbnail of it. Options are\n 'small_square', 'big_square', 'small_thumbnail',\n 'medium_thumbnail', 'large_thumbnail' or 'huge_thumbnail'.\n\n :returns: Name of the new file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the image with a new title and or description.", "response": "def update(self, title=None, description=None):\n \"\"\"Update the image with a new title and/or description.\"\"\"\n url = (self._imgur._base_url + \"/3/image/\"\n \"{0}\".format(self._delete_or_id_hash))\n is_updated = self._imgur._send_request(url, params=locals(),\n method='POST')\n if is_updated:\n self.title = title or self.title\n self.description = description or self.description\n return is_updated"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _send_request(self, url, needs_auth=False, **kwargs):\n # TODO: Add automatic test for timed_out access_tokens and\n # automatically refresh it before carrying out the request.\n if self.access_token is None and needs_auth:\n # TODO: Use inspect to insert name of method in error msg.\n raise Exception(\"Authentication as a user is required to use this \"\n \"method.\")\n if self.access_token is None:\n # Not authenticated as a user. Use anonymous access.\n auth = {'Authorization': 'Client-ID {0}'.format(self.client_id)}\n else:\n auth = {'Authorization': 'Bearer {0}'.format(self.access_token)}\n if self.mashape_key:\n auth.update({'X-Mashape-Key': self.mashape_key})\n content = []\n is_paginated = False\n if 'limit' in kwargs:\n is_paginated = True\n limit = kwargs['limit'] or self.DEFAULT_LIMIT\n del kwargs['limit']\n page = 0\n base_url = url\n url.format(page)\n kwargs['authentication'] = auth\n while True:\n result = request.send_request(url, verify=self.verify, **kwargs)\n new_content, ratelimit_info = result\n if is_paginated and new_content and limit > len(new_content):\n content += new_content\n page += 1\n url = base_url.format(page)\n else:\n if is_paginated:\n content = (content + new_content)[:limit]\n else:\n content = new_content\n break\n # Note: When the cache is implemented, it's important that the\n # ratelimit info doesn't get updated with the ratelimit info in the\n # cache since that's likely incorrect.\n for key, value in ratelimit_info.items():\n setattr(self, key[2:].replace('-', '_'), value)\n return content", "response": "Send a request to the Imgur API."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the authorization url that s needed to authorize as a user.", "response": "def authorization_url(self, response, state=\"\"):\n \"\"\"\n Return the authorization url that's needed to authorize as a user.\n\n :param response: Can be either code or pin. If it's code the user will\n be redirected to your redirect url with the code as a get parameter\n after authorizing your application. If it's pin then after\n authorizing your application, the user will instead be shown a pin\n on Imgurs website. Both code and pin are used to get an\n access_token and refresh token with the exchange_code and\n exchange_pin functions respectively.\n :param state: This optional parameter indicates any state which may be\n useful to your application upon receipt of the response. Imgur\n round-trips this parameter, so your application receives the same\n value it sent. Possible uses include redirecting the user to the\n correct resource in your site, nonces, and\n cross-site-request-forgery mitigations.\n \"\"\"\n return AUTHORIZE_URL.format(self._base_url, self.client_id, response, state)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nchange the current authentication.", "response": "def change_authentication(self, client_id=None, client_secret=None,\n access_token=None, refresh_token=None):\n \"\"\"Change the current authentication.\"\"\"\n # TODO: Add error checking so you cannot change client_id and retain\n # access_token. Because that doesn't make sense.\n self.client_id = client_id or self.client_id\n self.client_secret = client_secret or self.client_secret\n self.access_token = access_token or self.access_token\n self.refresh_token = refresh_token or self.refresh_token"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_album(self, title=None, description=None, images=None,\n cover=None):\n \"\"\"\n Create a new Album.\n\n :param title: The title of the album.\n :param description: The albums description.\n :param images: A list of the images that will be added to the album\n after it's created. Can be Image objects, ids or a combination of\n the two. Images that you cannot add (non-existing or not owned by\n you) will not cause exceptions, but fail silently.\n :param cover: The id of the image you want as the albums cover image.\n\n :returns: The newly created album.\n \"\"\"\n url = self._base_url + \"/3/album/\"\n payload = {'ids': images, 'title': title,\n 'description': description, 'cover': cover}\n resp = self._send_request(url, params=payload, method='POST')\n return Album(resp, self, has_fetched=False)", "response": "Create a new Album."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexchanging one - use code for an access_token and request_token.", "response": "def exchange_code(self, code):\n \"\"\"Exchange one-use code for an access_token and request_token.\"\"\"\n params = {'client_id': self.client_id,\n 'client_secret': self.client_secret,\n 'grant_type': 'authorization_code',\n 'code': code}\n result = self._send_request(EXCHANGE_URL.format(self._base_url),\n params=params, method='POST',\n data_field=None)\n self.access_token = result['access_token']\n self.refresh_token = result['refresh_token']\n return self.access_token, self.refresh_token"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef exchange_pin(self, pin):\n params = {'client_id': self.client_id,\n 'client_secret': self.client_secret,\n 'grant_type': 'pin',\n 'pin': pin}\n result = self._send_request(EXCHANGE_URL.format(self._base_url),\n params=params, method='POST',\n data_field=None)\n self.access_token = result['access_token']\n self.refresh_token = result['refresh_token']\n return self.access_token, self.refresh_token", "response": "Exchange one - use pin for an access_token and request_token."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_album(self, id):\n url = self._base_url + \"/3/album/{0}\".format(id)\n json = self._send_request(url)\n return Album(json, self)", "response": "Return information about this album."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_at_url(self, url):\n class NullDevice():\n def write(self, string):\n pass\n\n def get_gallery_item(id):\n \"\"\"\n Special helper method to get gallery items.\n\n The problem is that it's impossible to distinguish albums and\n images from each other based on the url. And there isn't a common\n url endpoints that return either a Gallery_album or a Gallery_image\n depending on what the id represents. So the only option is to\n assume it's a Gallery_image and if we get an exception then try\n Gallery_album. Gallery_image is attempted first because there is\n the most of them.\n \"\"\"\n try:\n # HACK: Problem is that send_request prints the error message\n # from Imgur when it encounters an error. This is nice because\n # this error message is more descriptive than just the status\n # code that Requests give. But since we first assume the id\n # belong to an image, it means we will get an error whenever\n # the id belongs to an album. The following code temporarily\n # disables stdout to avoid give a cryptic and incorrect error.\n\n # Code for disabling stdout is from\n # http://coreygoldberg.blogspot.dk/2009/05/\n # python-redirect-or-turn-off-stdout-and.html\n original_stdout = sys.stdout # keep a reference to STDOUT\n sys.stdout = NullDevice() # redirect the real STDOUT\n return self.get_gallery_image(id)\n # TODO: Add better error codes so I don't have to do a catch-all\n except Exception:\n return self.get_gallery_album(id)\n finally:\n sys.stdout = original_stdout # turn STDOUT back on\n\n if not self.is_imgur_url(url):\n return None\n\n objects = {'album': {'regex': \"a/(?P[\\w.]*?)$\",\n 'method': self.get_album},\n 'comment': {'regex': \"gallery/\\w*/comment/(?P[\\w.]*?)$\",\n 'method': self.get_comment},\n 'gallery': {'regex': \"(gallery|r/\\w*?)/(?P[\\w.]*?)$\",\n 'method': get_gallery_item},\n # Valid image extensions: http://imgur.com/faq#types\n # All are between 3 and 4 chars long.\n 'image': {'regex': \"(?P[\\w.]*?)(\\\\.\\w{3,4})?$\",\n 'method': self.get_image},\n 'user': {'regex': \"user/(?P[\\w.]*?)$\",\n 'method': self.get_user}\n }\n parsed_url = urlparse(url)\n for obj_type, values in objects.items():\n regex_result = re.match('/' + values['regex'], parsed_url.path)\n if regex_result is not None:\n obj_id = regex_result.group('id')\n initial_object = values['method'](obj_id)\n if obj_type == 'image':\n try:\n # A better version might be to ping the url where the\n # gallery_image should be with a requests.head call. If\n # we get a 200 returned, then that means it exists and\n # this becomes less hacky.\n original_stdout = sys.stdout\n sys.stdout = NullDevice()\n if getattr(initial_object, 'section', None):\n sub = initial_object.section\n return self.get_subreddit_image(sub, obj_id)\n return self.get_gallery_image(obj_id)\n except Exception:\n pass\n finally:\n sys.stdout = original_stdout\n return initial_object", "response": "Return a object representing the content at the given url."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_comment(self, id):\n url = self._base_url + \"/3/comment/{0}\".format(id)\n json = self._send_request(url)\n return Comment(json, self)", "response": "Return information about this comment."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a list of albums and gallery images.", "response": "def get_gallery(self, section='hot', sort='viral', window='day',\n show_viral=True, limit=None):\n \"\"\"\n Return a list of gallery albums and gallery images.\n\n :param section: hot | top | user - defaults to hot.\n :param sort: viral | time - defaults to viral.\n :param window: Change the date range of the request if the section is\n \"top\", day | week | month | year | all, defaults to day.\n :param show_viral: true | false - Show or hide viral images from the\n 'user' section. Defaults to true.\n :param limit: The number of items to return.\n \"\"\"\n url = (self._base_url + \"/3/gallery/{}/{}/{}/{}?showViral=\"\n \"{}\".format(section, sort, window, '{}', show_viral))\n resp = self._send_request(url, limit=limit)\n return [_get_album_or_image(thing, self) for thing in resp]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_gallery_album(self, id):\n url = self._base_url + \"/3/gallery/album/{0}\".format(id)\n resp = self._send_request(url)\n return Gallery_album(resp, self)", "response": "Get a gallery album by id"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_gallery_image(self, id):\n url = self._base_url + \"/3/gallery/image/{0}\".format(id)\n resp = self._send_request(url)\n return Gallery_image(resp, self)", "response": "Get a gallery image by id"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a Image object representing the image with the given id.", "response": "def get_image(self, id):\n \"\"\"Return a Image object representing the image with the given id.\"\"\"\n url = self._base_url + \"/3/image/{0}\".format(id)\n resp = self._send_request(url)\n return Image(resp, self)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting a message object for given id.", "response": "def get_message(self, id):\n \"\"\"\n Return a Message object for given id.\n\n :param id: The id of the message object to return.\n \"\"\"\n url = self._base_url + \"/3/message/{0}\".format(id)\n resp = self._send_request(url)\n return Message(resp, self)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_notification(self, id):\n url = self._base_url + \"/3/notification/{0}\".format(id)\n resp = self._send_request(url)\n return Notification(resp, self)", "response": "Get a specific notification object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_memes_gallery(self, sort='viral', window='week', limit=None):\n url = (self._base_url + \"/3/gallery/g/memes/{0}/{1}/{2}\".format(\n sort, window, '{}'))\n resp = self._send_request(url, limit=limit)\n return [_get_album_or_image(thing, self) for thing in resp]", "response": "Get a list of gallery albums or images submitted to the memes gallery"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_subreddit_gallery(self, subreddit, sort='time', window='top',\n limit=None):\n \"\"\"\n Return a list of gallery albums/images submitted to a subreddit.\n\n A subreddit is a subsection of the website www.reddit.com, where users\n can, among other things, post images.\n\n :param subreddit: A valid subreddit name.\n :param sort: time | top - defaults to top.\n :param window: Change the date range of the request if the section is\n \"top\", day | week | month | year | all, defaults to day.\n :param limit: The number of items to return.\n \"\"\"\n url = (self._base_url + \"/3/gallery/r/{0}/{1}/{2}/{3}\".format(\n subreddit, sort, window, '{}'))\n resp = self._send_request(url, limit=limit)\n return [_get_album_or_image(thing, self) for thing in resp]", "response": "Return a list of gallery albums or images submitted to a subreddit."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the Gallery_image with the id submitted to subreddit gallery", "response": "def get_subreddit_image(self, subreddit, id):\n \"\"\"\n Return the Gallery_image with the id submitted to subreddit gallery\n\n :param subreddit: The subreddit the image has been submitted to.\n :param id: The id of the image we want.\n \"\"\"\n url = self._base_url + \"/3/gallery/r/{0}/{1}\".format(subreddit, id)\n resp = self._send_request(url)\n return Gallery_image(resp, self)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_user(self, username):\n url = self._base_url + \"/3/account/{0}\".format(username)\n json = self._send_request(url)\n return User(json, self)", "response": "Get a User object for this username."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrefreshing the access_token. The self.access_token attribute will be updated with the value of the new access_token which will also be returned.", "response": "def refresh_access_token(self):\n \"\"\"\n Refresh the access_token.\n\n The self.access_token attribute will be updated with the value of the\n new access_token which will also be returned.\n \"\"\"\n if self.client_secret is None:\n raise Exception(\"client_secret must be set to execute \"\n \"refresh_access_token.\")\n if self.refresh_token is None:\n raise Exception(\"refresh_token must be set to execute \"\n \"refresh_access_token.\")\n params = {'client_id': self.client_id,\n 'client_secret': self.client_secret,\n 'grant_type': 'refresh_token',\n 'refresh_token': self.refresh_token}\n result = self._send_request(REFRESH_URL.format(self._base_url),\n params=params, method='POST',\n data_field=None)\n self.access_token = result['access_token']\n return self.access_token"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef search_gallery(self, q):\n url = self._base_url + \"/3/gallery/search?q={0}\".format(q)\n resp = self._send_request(url)\n return [_get_album_or_image(thing, self) for thing in resp]", "response": "Search the gallery with the given query string."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nuploading an image at either path or url.", "response": "def upload_image(self, path=None, url=None, title=None, description=None,\n album=None):\n \"\"\"\n Upload the image at either path or url.\n\n :param path: The path to the image you want to upload.\n :param url: The url to the image you want to upload.\n :param title: The title the image will have when uploaded.\n :param description: The description the image will have when uploaded.\n :param album: The album the image will be added to when uploaded. Can\n be either a Album object or it's id. Leave at None to upload\n without adding to an Album, adding it later is possible.\n Authentication as album owner is necessary to upload to an album\n with this function.\n\n :returns: An Image object representing the uploaded image.\n \"\"\"\n if bool(path) == bool(url):\n raise LookupError(\"Either path or url must be given.\")\n if path:\n with open(path, 'rb') as image_file:\n binary_data = image_file.read()\n image = b64encode(binary_data)\n else:\n image = url\n\n payload = {'album_id': album, 'image': image,\n 'title': title, 'description': description}\n\n resp = self._send_request(self._base_url + \"/3/image\",\n params=payload, method='POST')\n # TEMPORARY HACK:\n # On 5-08-2013 I noticed Imgur now returned enough information from\n # this call to fully populate the Image object. However those variables\n # that matched arguments were always None, even if they had been given.\n # See https://groups.google.com/forum/#!topic/imgur/F3uVb55TMGo\n resp['title'] = title\n resp['description'] = description\n if album is not None:\n resp['album'] = (Album({'id': album}, self, False) if not\n isinstance(album, Album) else album)\n return Image(resp, self)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the message thread this Message is in.", "response": "def get_thread(self):\n \"\"\"Return the message thread this Message is in.\"\"\"\n url = (self._imgur._base_url + \"/3/message/{0}/thread\".format(\n self.first_message.id))\n resp = self._imgur._send_request(url)\n return [Message(msg, self._imgur) for msg in resp]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreplying to this message.", "response": "def reply(self, body):\n \"\"\"\n Reply to this message.\n\n This is a convenience method calling User.send_message. See it for more\n information on usage. Note that both recipient and reply_to are given\n by using this convenience method.\n\n :param body: The body of the message.\n \"\"\"\n return self.author.send_message(body=body, reply_to=self.id)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates the settings of the user.", "response": "def change_settings(self, bio=None, public_images=None,\n messaging_enabled=None, album_privacy=None,\n accepted_gallery_terms=None):\n \"\"\"\n Update the settings for the user.\n\n :param bio: A basic description filled out by the user, is displayed in\n the gallery profile page.\n :param public_images: Set the default privacy setting of the users\n images. If True images are public, if False private.\n :param messaging_enabled: Set to True to enable messaging.\n :param album_privacy: The default privacy level of albums created by\n the user. Can be public, hidden or secret.\n :param accepted_gallery_terms: The user agreement to Imgur Gallery\n terms. Necessary before the user can submit to the gallery.\n \"\"\"\n # NOTE: album_privacy should maybe be renamed to default_privacy\n # NOTE: public_images is a boolean, despite the documentation saying it\n # is a string.\n url = self._imgur._base_url + \"/3/account/{0}/settings\".format(self.name)\n resp = self._imgur._send_request(url, needs_auth=True, params=locals(),\n method='POST')\n return resp"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of the user s albums.", "response": "def get_albums(self, limit=None):\n \"\"\"\n Return a list of the user's albums.\n\n Secret and hidden albums are only returned if this is the logged-in\n user.\n \"\"\"\n url = (self._imgur._base_url + \"/3/account/{0}/albums/{1}\".format(self.name,\n '{}'))\n resp = self._imgur._send_request(url, limit=limit)\n return [Album(alb, self._imgur, False) for alb in resp]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_favorites(self):\n url = self._imgur._base_url + \"/3/account/{0}/favorites\".format(self.name)\n resp = self._imgur._send_request(url, needs_auth=True)\n return [_get_album_or_image(thing, self._imgur) for thing in resp]", "response": "Return the users favorited images."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_gallery_favorites(self):\n url = (self._imgur._base_url + \"/3/account/{0}/gallery_favorites\".format(\n self.name))\n resp = self._imgur._send_request(url)\n return [Image(img, self._imgur) for img in resp]", "response": "Get a list of the images in the gallery this user has favorited."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_gallery_profile(self):\n url = (self._imgur._base_url + \"/3/account/{0}/\"\n \"gallery_profile\".format(self.name))\n return self._imgur._send_request(url)", "response": "Return the users gallery profile."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef has_verified_email(self):\n url = (self._imgur._base_url + \"/3/account/{0}/\"\n \"verifyemail\".format(self.name))\n return self._imgur._send_request(url, needs_auth=True)", "response": "Returns True if the user has verified that the email he has given is legit?"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_images(self, limit=None):\n url = (self._imgur._base_url + \"/3/account/{0}/\"\n \"images/{1}\".format(self.name, '{}'))\n resp = self._imgur._send_request(url, limit=limit)\n return [Image(img, self._imgur) for img in resp]", "response": "Return all of the images associated with the user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn all messages sent to this user formatted as a notification.", "response": "def get_messages(self, new=True):\n \"\"\"\n Return all messages sent to this user, formatted as a notification.\n\n :param new: False for all notifications, True for only non-viewed\n notifications.\n \"\"\"\n url = (self._imgur._base_url + \"/3/account/{0}/notifications/\"\n \"messages\".format(self.name))\n result = self._imgur._send_request(url, params=locals(),\n needs_auth=True)\n return [Notification(msg_dict, self._imgur, has_fetched=True) for\n msg_dict in result]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn all the notifications for this user.", "response": "def get_notifications(self, new=True):\n \"\"\"Return all the notifications for this user.\"\"\"\n url = (self._imgur._base_url + \"/3/account/{0}/\"\n \"notifications\".format(self.name))\n resp = self._imgur._send_request(url, params=locals(), needs_auth=True)\n msgs = [Message(msg_dict, self._imgur, has_fetched=True) for msg_dict\n in resp['messages']]\n replies = [Comment(com_dict, self._imgur, has_fetched=True) for\n com_dict in resp['replies']]\n return {'messages': msgs, 'replies': replies}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_replies(self, new=True):\n url = (self._imgur._base_url + \"/3/account/{0}/\"\n \"notifications/replies\".format(self.name))\n return self._imgur._send_request(url, needs_auth=True)", "response": "Return all replies for this user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn current settings. Only accessible if authenticated as the user.", "response": "def get_settings(self):\n \"\"\"\n Returns current settings.\n\n Only accessible if authenticated as the user.\n \"\"\"\n url = self._imgur._base_url + \"/3/account/{0}/settings\".format(self.name)\n return self._imgur._send_request(url)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of the images a user has submitted to the gallery.", "response": "def get_submissions(self, limit=None):\n \"\"\"Return a list of the images a user has submitted to the gallery.\"\"\"\n url = (self._imgur._base_url + \"/3/account/{0}/submissions/\"\n \"{1}\".format(self.name, '{}'))\n resp = self._imgur._send_request(url, limit=limit)\n return [_get_album_or_image(thing, self._imgur) for thing in resp]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending a message to this user from the logged in user.", "response": "def send_message(self, body, subject=None, reply_to=None):\n \"\"\"\n Send a message to this user from the logged in user.\n\n :param body: The body of the message.\n :param subject: The subject of the message. Note that if the this\n message is a reply, then the subject of the first message will be\n used instead.\n :param reply_to: Messages can either be replies to other messages or\n start a new message thread. If this is None it will start a new\n message thread. If it's a Message object or message_id, then the\n new message will be sent as a reply to the reply_to message.\n \"\"\"\n url = self._imgur._base_url + \"/3/message\"\n parent_id = reply_to.id if isinstance(reply_to, Message) else reply_to\n payload = {'recipient': self.name, 'body': body, 'subject': subject,\n 'parent_id': parent_id}\n self._imgur._send_request(url, params=payload, needs_auth=True,\n method='POST')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_verification_email(self):\n url = (self._imgur._base_url + \"/3/account/{0}\"\n \"/verifyemail\".format(self.name))\n self._imgur._send_request(url, needs_auth=True, method='POST')", "response": "Send verification email to this users email address."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhook the current excepthook to the backtrace.", "response": "def hook(reverse=False,\n align=False,\n strip_path=False,\n enable_on_envvar_only=False,\n on_tty=False,\n conservative=False,\n styles=None,\n tb=None,\n tpe=None,\n value=None):\n \"\"\"Hook the current excepthook to the backtrace.\n\n If `align` is True, all parts (line numbers, file names, etc..) will be\n aligned to the left according to the longest entry.\n\n If `strip_path` is True, only the file name will be shown, not its full\n path.\n\n If `enable_on_envvar_only` is True, only if the environment variable\n `ENABLE_BACKTRACE` is set, backtrace will be activated.\n\n If `on_tty` is True, backtrace will be activated only if you're running\n in a readl terminal (i.e. not piped, redirected, etc..).\n\n If `convervative` is True, the traceback will have more seemingly original\n style (There will be no alignment by default, 'File', 'line' and 'in'\n prefixes and will ignore any styling provided by the user.)\n\n See https://github.com/nir0s/backtrace/blob/master/README.md for\n information on `styles`.\n \"\"\"\n if enable_on_envvar_only and 'ENABLE_BACKTRACE' not in os.environ:\n return\n\n isatty = getattr(sys.stderr, 'isatty', lambda: False)\n if on_tty and not isatty():\n return\n\n if conservative:\n styles = CONVERVATIVE_STYLES\n align = align or False\n elif styles:\n for k in STYLES.keys():\n styles[k] = styles.get(k, STYLES[k])\n else:\n styles = STYLES\n\n # For Windows\n colorama.init()\n\n def backtrace_excepthook(tpe, value, tb=None):\n # Don't know if we're getting traceback or traceback entries.\n # We'll try to parse a traceback object.\n try:\n traceback_entries = traceback.extract_tb(tb)\n except AttributeError:\n traceback_entries = tb\n parser = _Hook(traceback_entries, align, strip_path, conservative)\n\n tpe = tpe if isinstance(tpe, str) else tpe.__name__\n tb_message = styles['backtrace'].format('Traceback ({0}):'.format(\n 'Most recent call ' + ('first' if reverse else 'last'))) + \\\n Style.RESET_ALL\n err_message = styles['error'].format(tpe + ': ' + str(value)) + \\\n Style.RESET_ALL\n\n if reverse:\n parser.reverse()\n\n _flush(tb_message)\n backtrace = parser.generate_backtrace(styles)\n backtrace.insert(0 if reverse else len(backtrace), err_message)\n for entry in backtrace:\n _flush(entry)\n\n if tb:\n backtrace_excepthook(tpe=tpe, value=value, tb=tb)\n else:\n sys.excepthook = backtrace_excepthook"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextracts a traceback from the input text.", "response": "def _extract_traceback(text):\n \"\"\"Receive a list of strings representing the input from stdin and return\n the restructured backtrace.\n\n This iterates over the output and once it identifies a hopefully genuine\n identifier, it will start parsing output.\n In the case the input includes a reraise (a Python 3 case), the primary\n traceback isn't handled, only the reraise.\n\n Each of the traceback lines are then handled two lines at a time for each\n stack object.\n\n Note that all parts of each stack object are stripped from newlines and\n spaces to keep the output clean.\n \"\"\"\n capture = False\n entries = []\n all_else = []\n ignore_trace = False\n\n # In python 3, a traceback may includes output from a reraise.\n # e.g, an exception is captured and reraised with another exception.\n # This marks that we should ignore\n if text.count(TRACEBACK_IDENTIFIER) == 2:\n ignore_trace = True\n\n for index, line in enumerate(text):\n if TRACEBACK_IDENTIFIER in line:\n if ignore_trace:\n ignore_trace = False\n continue\n capture = True\n # We're not capturing and making sure we only read lines\n # with spaces since, after the initial identifier, all traceback lines\n # contain a prefix spacing.\n elif capture and line.startswith(' '):\n if index % 2 == 0:\n # Line containing a file, line and module.\n line = line.strip().strip('\\n')\n next_line = text[index + 1].strip('\\n')\n entries.append(line + ', ' + next_line)\n elif capture:\n # Line containing the module call.\n entries.append(line)\n break\n else:\n # Add everything else after the traceback.\n all_else.append(line)\n\n traceback_entries = []\n\n # Build the traceback structure later passed for formatting.\n for index, line in enumerate(entries[:-2]):\n # TODO: This should be done in a _parse_entry function\n element = line.split(',')\n element[0] = element[0].strip().lstrip('File').strip(' \"')\n element[1] = element[1].strip().lstrip('line').strip()\n element[2] = element[2].strip().lstrip('in').strip()\n traceback_entries.append(tuple(element))\n return traceback_entries, all_else"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_backtrace(self, styles):\n backtrace = []\n for entry in self.entries:\n backtrace.append(self.rebuild_entry(entry, styles))\n\n # Get the lenght of the longest string for each field of an entry\n lengths = self.align_all(backtrace) if self.align else [1, 1, 1, 1]\n\n aligned_backtrace = []\n for entry in backtrace:\n aligned_backtrace.append(self.align_entry(entry, lengths))\n return aligned_backtrace", "response": "Return the potentially aligned traceback for the current set of entries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nclassifying a list of sources and splits them according to their class.", "response": "def classify_catalog(catalog):\n \"\"\"\n Look at a list of sources and split them according to their class.\n\n Parameters\n ----------\n catalog : iterable\n A list or iterable object of {SimpleSource, IslandSource, OutputSource} objects, possibly mixed.\n Any other objects will be silently ignored.\n\n Returns\n -------\n components : list\n List of sources of type OutputSource\n\n islands : list\n List of sources of type IslandSource\n\n simples : list\n List of source of type SimpleSource\n \"\"\"\n components = []\n islands = []\n simples = []\n for source in catalog:\n if isinstance(source, OutputSource):\n components.append(source)\n elif isinstance(source, IslandSource):\n islands.append(source)\n elif isinstance(source, SimpleSource):\n simples.append(source)\n return components, islands, simples"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef island_itergen(catalog):\n # reverse sort so that we can pop the last elements and get an increasing island number\n catalog = sorted(catalog)\n catalog.reverse()\n group = []\n\n # using pop and keeping track of the list length ourselves is faster than\n # constantly asking for len(catalog)\n src = catalog.pop()\n c_len = len(catalog)\n isle_num = src.island\n while c_len >= 0:\n if src.island == isle_num:\n group.append(src)\n c_len -= 1\n if c_len <0:\n # we have just added the last item from the catalog\n # and there are no more to pop\n yield group\n else:\n src = catalog.pop()\n else:\n isle_num += 1\n # maybe there are no sources in this island so skip it\n if group == []:\n continue\n yield group\n group = []\n return", "response": "Iterate over a catalog of sources and return an island worth of sources at a time."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsanitizes the attributes of the object so that they can print properly.", "response": "def _sanitise(self):\n \"\"\"\n Convert attributes of type npumpy.float32 to numpy.float64 so that they will print properly.\n \"\"\"\n for k in self.__dict__:\n if isinstance(self.__dict__[k], np.float32): # np.float32 has a broken __str__ method\n self.__dict__[k] = np.float64(self.__dict__[k])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning an ordered list of the source attributes", "response": "def as_list(self):\n \"\"\"\n Return an *ordered* list of the source attributes\n \"\"\"\n self._sanitise()\n l = []\n for name in self.names:\n l.append(getattr(self, name))\n return l"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding one or more circles to this region.", "response": "def add_circles(self, ra_cen, dec_cen, radius, depth=None):\n \"\"\"\n Add one or more circles to this region\n\n Parameters\n ----------\n ra_cen, dec_cen, radius : float or list\n The center and radius of the circle or circles to add to this region.\n\n depth : int\n The depth at which the given circles will be inserted.\n\n \"\"\"\n if depth is None or depth > self.maxdepth:\n depth = self.maxdepth\n try:\n sky = list(zip(ra_cen, dec_cen))\n rad = radius\n except TypeError:\n sky = [[ra_cen, dec_cen]]\n rad = [radius]\n sky = np.array(sky)\n rad = np.array(rad)\n vectors = self.sky2vec(sky)\n for vec, r in zip(vectors, rad):\n pix = hp.query_disc(2**depth, vec, r, inclusive=True, nest=True)\n self.add_pixels(pix, depth)\n self._renorm()\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_poly(self, positions, depth=None):\n if not (len(positions) >= 3): raise AssertionError(\"A minimum of three coordinate pairs are required\")\n\n if depth is None or depth > self.maxdepth:\n depth = self.maxdepth\n\n ras, decs = np.array(list(zip(*positions)))\n sky = self.radec2sky(ras, decs)\n pix = hp.query_polygon(2**depth, self.sky2vec(sky), inclusive=True, nest=True)\n self.add_pixels(pix, depth)\n self._renorm()\n return", "response": "Adds a single polygon to this region."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd one or more HEALPix pixels to this region.", "response": "def add_pixels(self, pix, depth):\n \"\"\"\n Add one or more HEALPix pixels to this region.\n\n Parameters\n ----------\n pix : int or iterable\n The pixels to be added\n\n depth : int\n The depth at which the pixels are added.\n \"\"\"\n if depth not in self.pixeldict:\n self.pixeldict[depth] = set()\n self.pixeldict[depth].update(set(pix))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates the total area of the region.", "response": "def get_area(self, degrees=True):\n \"\"\"\n Calculate the total area represented by this region.\n\n Parameters\n ----------\n degrees : bool\n If True then return the area in square degrees, otherwise use steradians.\n Default = True.\n\n Returns\n -------\n area : float\n The area of the region.\n \"\"\"\n area = 0\n for d in range(1, self.maxdepth+1):\n area += len(self.pixeldict[d])*hp.nside2pixarea(2**d, degrees=degrees)\n return area"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _demote_all(self):\n # only do the calculations if the demoted list is empty\n if len(self.demoted) == 0:\n pd = self.pixeldict\n for d in range(1, self.maxdepth):\n for p in pd[d]:\n pd[d+1].update(set((4*p, 4*p+1, 4*p+2, 4*p+3)))\n pd[d] = set() # clear the pixels from this level\n self.demoted = pd[d+1]\n return", "response": "Convert the multi - depth pixeldict into a single set of pixels at the deepest layer."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nremaking the pixel dictionary, merging groups of pixels at level N into a single pixel at level N-1", "response": "def _renorm(self):\n \"\"\"\n Remake the pixel dictionary, merging groups of pixels at level N into a single pixel\n at level N-1\n \"\"\"\n self.demoted = set()\n # convert all to lowest level\n self._demote_all()\n # now promote as needed\n for d in range(self.maxdepth, 2, -1):\n plist = self.pixeldict[d].copy()\n for p in plist:\n if p % 4 == 0:\n nset = set((p, p+1, p+2, p+3))\n if p+1 in plist and p+2 in plist and p+3 in plist:\n # remove the four pixels from this level\n self.pixeldict[d].difference_update(nset)\n # add a new pixel to the next level up\n self.pixeldict[d-1].add(p/4)\n self.demoted = set()\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sky_within(self, ra, dec, degin=False):\n sky = self.radec2sky(ra, dec)\n\n if degin:\n sky = np.radians(sky)\n\n theta_phi = self.sky2ang(sky)\n # Set values that are nan to be zero and record a mask\n mask = np.bitwise_not(np.logical_and.reduce(np.isfinite(theta_phi), axis=1))\n theta_phi[mask, :] = 0\n\n theta, phi = theta_phi.transpose()\n pix = hp.ang2pix(2**self.maxdepth, theta, phi, nest=True)\n pixelset = self.get_demoted()\n result = np.in1d(pix, list(pixelset))\n # apply the mask and set the shonky values to False\n result[mask] = False\n return result", "response": "Test whether a given sky position is within this region."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef union(self, other, renorm=True):\n # merge the pixels that are common to both\n for d in range(1, min(self.maxdepth, other.maxdepth)+1):\n self.add_pixels(other.pixeldict[d], d)\n\n # if the other region is at higher resolution, then include a degraded version of the remaining pixels.\n if self.maxdepth < other.maxdepth:\n for d in range(self.maxdepth+1, other.maxdepth+1):\n for p in other.pixeldict[d]:\n # promote this pixel to self.maxdepth\n pp = p/4**(d-self.maxdepth)\n self.pixeldict[self.maxdepth].add(pp)\n if renorm:\n self._renorm()\n return", "response": "Add another Region by performing union on their pixlists."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef without(self, other):\n # work only on the lowest level\n # TODO: Allow this to be done for regions with different depths.\n if not (self.maxdepth == other.maxdepth): raise AssertionError(\"Regions must have the same maxdepth\")\n self._demote_all()\n opd = set(other.get_demoted())\n self.pixeldict[self.maxdepth].difference_update(opd)\n self._renorm()\n return", "response": "Subtract another Region from this one by performing a difference operation on their pixlists."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef intersect(self, other):\n # work only on the lowest level\n # TODO: Allow this to be done for regions with different depths.\n if not (self.maxdepth == other.maxdepth): raise AssertionError(\"Regions must have the same maxdepth\")\n self._demote_all()\n opd = set(other.get_demoted())\n self.pixeldict[self.maxdepth].intersection_update(opd)\n self._renorm()\n return", "response": "Combine with another Region by performing intersection on their pixlists."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef symmetric_difference(self, other):\n # work only on the lowest level\n # TODO: Allow this to be done for regions with different depths.\n if not (self.maxdepth == other.maxdepth): raise AssertionError(\"Regions must have the same maxdepth\")\n self._demote_all()\n opd = set(other.get_demoted())\n self.pixeldict[self.maxdepth].symmetric_difference_update(opd)\n self._renorm()\n return", "response": "Combine with another Region by performing the symmetric difference of their pixlists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a ds9 region file that represents this region as a set of diamonds.", "response": "def write_reg(self, filename):\n \"\"\"\n Write a ds9 region file that represents this region as a set of diamonds.\n\n Parameters\n ----------\n filename : str\n File to write\n \"\"\"\n with open(filename, 'w') as out:\n for d in range(1, self.maxdepth+1):\n for p in self.pixeldict[d]:\n line = \"fk5; polygon(\"\n # the following int() gets around some problems with np.int64 that exist prior to numpy v 1.8.1\n vectors = list(zip(*hp.boundaries(2**d, int(p), step=1, nest=True)))\n positions = []\n for sky in self.vec2sky(np.array(vectors), degrees=True):\n ra, dec = sky\n pos = SkyCoord(ra/15, dec, unit=(u.degree, u.degree))\n positions.append(pos.ra.to_string(sep=':', precision=2))\n positions.append(pos.dec.to_string(sep=':', precision=2))\n line += ','.join(positions)\n line += \")\"\n print(line, file=out)\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef write_fits(self, filename, moctool=''):\n datafile = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data', 'MOC.fits')\n hdulist = fits.open(datafile)\n cols = fits.Column(name='NPIX', array=self._uniq(), format='1K')\n tbhdu = fits.BinTableHDU.from_columns([cols])\n hdulist[1] = tbhdu\n hdulist[1].header['PIXTYPE'] = ('HEALPIX ', 'HEALPix magic code')\n hdulist[1].header['ORDERING'] = ('NUNIQ ', 'NUNIQ coding method')\n hdulist[1].header['COORDSYS'] = ('C ', 'ICRS reference frame')\n hdulist[1].header['MOCORDER'] = (self.maxdepth, 'MOC resolution (best order)')\n hdulist[1].header['MOCTOOL'] = (moctool, 'Name of the MOC generator')\n hdulist[1].header['MOCTYPE'] = ('CATALOG', 'Source type (IMAGE or CATALOG)')\n hdulist[1].header['MOCID'] = (' ', 'Identifier of the collection')\n hdulist[1].header['ORIGIN'] = (' ', 'MOC origin')\n time = datetime.datetime.utcnow()\n hdulist[1].header['DATE'] = (datetime.datetime.strftime(time, format=\"%Y-%m-%dT%H:%m:%SZ\"), 'MOC creation date')\n hdulist.writeto(filename, overwrite=True)\n return", "response": "Write a FITS file representing the MOC of this region to a file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a list of all the pixels that cover this region.", "response": "def _uniq(self):\n \"\"\"\n Create a list of all the pixels that cover this region.\n This list contains overlapping pixels of different orders.\n\n Returns\n -------\n pix : list\n A list of HEALPix pixel numbers.\n \"\"\"\n pd = []\n for d in range(1, self.maxdepth):\n pd.extend(map(lambda x: int(4**(d+1) + x), self.pixeldict[d]))\n return sorted(pd)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert ra dec to sky coordinates.", "response": "def radec2sky(ra, dec):\n \"\"\"\n Convert [ra], [dec] to [(ra[0], dec[0]),....]\n and also ra,dec to [(ra,dec)] if ra/dec are not iterable\n\n Parameters\n ----------\n ra, dec : float or iterable\n Sky coordinates\n\n Returns\n -------\n sky : numpy.array\n array of (ra,dec) coordinates.\n \"\"\"\n try:\n sky = np.array(list(zip(ra, dec)))\n except TypeError:\n sky = np.array([(ra, dec)])\n return sky"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sky2ang(sky):\n try:\n theta_phi = sky.copy()\n except AttributeError as _:\n theta_phi = np.array(sky)\n theta_phi[:, [1, 0]] = theta_phi[:, [0, 1]]\n theta_phi[:, 0] = np.pi/2 - theta_phi[:, 0]\n # # force 0<=theta<=2pi\n # theta_phi[:, 0] -= 2*np.pi*(theta_phi[:, 0]//(2*np.pi))\n # # and now -pi<=theta<=pi\n # theta_phi[:, 0] -= (theta_phi[:, 0] > np.pi)*2*np.pi\n return theta_phi", "response": "Convert ra dec coordinates to theta and phi coordinates."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sky2vec(cls, sky):\n theta_phi = cls.sky2ang(sky)\n theta, phi = map(np.array, list(zip(*theta_phi)))\n vec = hp.ang2vec(theta, phi)\n return vec", "response": "Convert the sky positions in to 3d - vectors on the unit sphere."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef vec2sky(cls, vec, degrees=False):\n theta, phi = hp.vec2ang(vec)\n ra = phi\n dec = np.pi/2-theta\n\n if degrees:\n ra = np.degrees(ra)\n dec = np.degrees(dec)\n return cls.radec2sky(ra, dec)", "response": "Convert vector into sky coordinates ra dec"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new WCSHelper object from the given header.", "response": "def from_header(cls, header, beam=None, lat=None):\n \"\"\"\n Create a new WCSHelper class from the given header.\n\n Parameters\n ----------\n header : `astropy.fits.HDUHeader` or string\n The header to be used to create the WCS helper\n\n beam : :class:`AegeanTools.fits_image.Beam` or None\n The synthesized beam. If the supplied beam is None then one is constructed form the header.\n\n lat : float\n The latitude of the telescope.\n\n Returns\n -------\n obj : :class:`AegeanTools.wcs_helpers.WCSHelper`\n A helper object.\n \"\"\"\n try:\n wcs = pywcs.WCS(header, naxis=2)\n except: # TODO: figure out what error is being thrown\n wcs = pywcs.WCS(str(header), naxis=2)\n\n if beam is None:\n beam = get_beam(header)\n else:\n beam = beam\n\n if beam is None:\n logging.critical(\"Cannot determine beam information\")\n\n _, pixscale = get_pixinfo(header)\n refpix = (header['CRPIX1'], header['CRPIX2'])\n return cls(wcs, beam, pixscale, refpix, lat)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_file(cls, filename, beam=None):\n header = fits.getheader(filename)\n return cls.from_header(header, beam)", "response": "Create a new WCSHelper class from a given FITS file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts pixel coordinates into sky coordinates.", "response": "def pix2sky(self, pixel):\n \"\"\"\n Convert pixel coordinates into sky coordinates.\n\n Parameters\n ----------\n pixel : (float, float)\n The (x,y) pixel coordinates\n\n Returns\n -------\n sky : (float, float)\n The (ra,dec) sky coordinates in degrees\n\n \"\"\"\n x, y = pixel\n # wcs and pyfits have oposite ideas of x/y\n return self.wcs.wcs_pix2world([[y, x]], 1)[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert sky coordinates into pixel coordinates.", "response": "def sky2pix(self, pos):\n \"\"\"\n Convert sky coordinates into pixel coordinates.\n\n Parameters\n ----------\n pos : (float, float)\n The (ra, dec) sky coordinates (degrees)\n\n Returns\n -------\n pixel : (float, float)\n The (x,y) pixel coordinates\n\n \"\"\"\n pixel = self.wcs.wcs_world2pix([pos], 1)\n # wcs and pyfits have oposite ideas of x/y\n return [pixel[0][1], pixel[0][0]]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sky2pix_vec(self, pos, r, pa):\n ra, dec = pos\n x, y = self.sky2pix(pos)\n a = translate(ra, dec, r, pa)\n locations = self.sky2pix(a)\n x_off, y_off = locations\n a = np.sqrt((x - x_off) ** 2 + (y - y_off) ** 2)\n theta = np.degrees(np.arctan2((y_off - y), (x_off - x)))\n return x, y, a, theta", "response": "Convert a vector from sky to pixel coordinates."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving and input position and vector in pixel coordinates calculate the equivalent position and vector in sky coordinates.", "response": "def pix2sky_vec(self, pixel, r, theta):\n \"\"\"\n Given and input position and vector in pixel coordinates, calculate\n the equivalent position and vector in sky coordinates.\n\n Parameters\n ----------\n pixel : (int,int)\n origin of vector in pixel coordinates\n r : float\n magnitude of vector in pixels\n theta : float\n angle of vector in degrees\n\n Returns\n -------\n ra, dec : float\n The (ra, dec) of the origin point (degrees).\n r, pa : float\n The magnitude and position angle of the vector (degrees).\n \"\"\"\n ra1, dec1 = self.pix2sky(pixel)\n x, y = pixel\n a = [x + r * np.cos(np.radians(theta)),\n y + r * np.sin(np.radians(theta))]\n locations = self.pix2sky(a)\n ra2, dec2 = locations\n a = gcd(ra1, dec1, ra2, dec2)\n pa = bear(ra1, dec1, ra2, dec2)\n return ra1, dec1, a, pa"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts an ellipse from sky to pixel coordinates.", "response": "def sky2pix_ellipse(self, pos, a, b, pa):\n \"\"\"\n Convert an ellipse from sky to pixel coordinates.\n\n Parameters\n ----------\n pos : (float, float)\n The (ra, dec) of the ellipse center (degrees).\n a, b, pa: float\n The semi-major axis, semi-minor axis and position angle of the ellipse (degrees).\n\n Returns\n -------\n x,y : float\n The (x, y) pixel coordinates of the ellipse center.\n sx, sy : float\n The major and minor axes (FWHM) in pixels.\n theta : float\n The rotation angle of the ellipse (degrees).\n theta = 0 corresponds to the ellipse being aligned with the x-axis.\n\n \"\"\"\n ra, dec = pos\n x, y = self.sky2pix(pos)\n\n x_off, y_off = self.sky2pix(translate(ra, dec, a, pa))\n sx = np.hypot((x - x_off), (y - y_off))\n theta = np.arctan2((y_off - y), (x_off - x))\n\n x_off, y_off = self.sky2pix(translate(ra, dec, b, pa - 90))\n sy = np.hypot((x - x_off), (y - y_off))\n theta2 = np.arctan2((y_off - y), (x_off - x)) - np.pi / 2\n\n # The a/b vectors are perpendicular in sky space, but not always in pixel space\n # so we have to account for this by calculating the angle between the two vectors\n # and modifying the minor axis length\n defect = theta - theta2\n sy *= abs(np.cos(defect))\n\n return x, y, sx, sy, np.degrees(theta)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts an ellipse from pixel to sky coordinates.", "response": "def pix2sky_ellipse(self, pixel, sx, sy, theta):\n \"\"\"\n Convert an ellipse from pixel to sky coordinates.\n\n Parameters\n ----------\n pixel : (float, float)\n The (x, y) coordinates of the center of the ellipse.\n sx, sy : float\n The major and minor axes (FHWM) of the ellipse, in pixels.\n theta : float\n The rotation angle of the ellipse (degrees).\n theta = 0 corresponds to the ellipse being aligned with the x-axis.\n\n Returns\n -------\n ra, dec : float\n The (ra, dec) coordinates of the center of the ellipse (degrees).\n\n a, b : float\n The semi-major and semi-minor axis of the ellipse (degrees).\n\n pa : float\n The position angle of the ellipse (degrees).\n \"\"\"\n ra, dec = self.pix2sky(pixel)\n x, y = pixel\n v_sx = [x + sx * np.cos(np.radians(theta)),\n y + sx * np.sin(np.radians(theta))]\n ra2, dec2 = self.pix2sky(v_sx)\n major = gcd(ra, dec, ra2, dec2)\n pa = bear(ra, dec, ra2, dec2)\n\n v_sy = [x + sy * np.cos(np.radians(theta - 90)),\n y + sy * np.sin(np.radians(theta - 90))]\n ra2, dec2 = self.pix2sky(v_sy)\n minor = gcd(ra, dec, ra2, dec2)\n pa2 = bear(ra, dec, ra2, dec2) - 90\n\n # The a/b vectors are perpendicular in sky space, but not always in pixel space\n # so we have to account for this by calculating the angle between the two vectors\n # and modifying the minor axis length\n defect = pa - pa2\n minor *= abs(np.cos(np.radians(defect)))\n return ra, dec, major, minor, pa"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the beam in pixels at the given location in pixel coordinates.", "response": "def get_pixbeam_pixel(self, x, y):\n \"\"\"\n Determine the beam in pixels at the given location in pixel coordinates.\n\n Parameters\n ----------\n x , y : float\n The pixel coordinates at which the beam is determined.\n\n Returns\n -------\n beam : :class:`AegeanTools.fits_image.Beam`\n A beam object, with a/b/pa in pixel coordinates.\n \"\"\"\n ra, dec = self.pix2sky((x, y))\n return self.get_pixbeam(ra, dec)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the beam object at the given sky location.", "response": "def get_beam(self, ra, dec):\n \"\"\"\n Determine the beam at the given sky location.\n\n Parameters\n ----------\n ra, dec : float\n The sky coordinates at which the beam is determined.\n\n Returns\n -------\n beam : :class:`AegeanTools.fits_image.Beam`\n A beam object, with a/b/pa in sky coordinates\n \"\"\"\n # check to see if we need to scale the major axis based on the declination\n if self.lat is None:\n factor = 1\n else:\n # this works if the pa is zero. For non-zero pa it's a little more difficult\n factor = np.cos(np.radians(dec - self.lat))\n return Beam(self.beam.a / factor, self.beam.b, self.beam.pa)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_pixbeam(self, ra, dec):\n\n if ra is None:\n ra, dec = self.pix2sky(self.refpix)\n pos = [ra, dec]\n\n beam = self.get_beam(ra, dec)\n _, _, major, minor, theta = self.sky2pix_ellipse(pos, beam.a, beam.b, beam.pa)\n\n if major < minor:\n major, minor = minor, major\n theta -= 90\n if theta < -180:\n theta += 180\n if not np.isfinite(theta):\n theta = 0\n if not all(np.isfinite([major, minor, theta])):\n beam = None\n else:\n beam = Beam(major, minor, theta)\n return beam", "response": "Returns the beam in pixels at the given location in sky coordinates."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate the area of the synthesized beam in square degrees.", "response": "def get_beamarea_deg2(self, ra, dec):\n \"\"\"\n Calculate the area of the synthesized beam in square degrees.\n\n Parameters\n ----------\n ra, dec : float\n The sky coordinates at which the calculation is made.\n\n Returns\n -------\n area : float\n The beam area in square degrees.\n \"\"\"\n barea = abs(self.beam.a * self.beam.b * np.pi) # in deg**2 at reference coords\n if self.lat is not None:\n barea /= np.cos(np.radians(dec - self.lat))\n return barea"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the beam area in square pixels at the given sky coordinates.", "response": "def get_beamarea_pix(self, ra, dec):\n \"\"\"\n Calculate the beam area in square pixels.\n\n Parameters\n ----------\n ra, dec : float\n The sky coordinates at which the calculation is made\n dec\n\n Returns\n -------\n area : float\n The beam area in square pixels.\n \"\"\"\n parea = abs(self.pixscale[0] * self.pixscale[1]) # in deg**2 at reference coords\n barea = self.get_beamarea_deg2(ra, dec)\n return barea / parea"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the GCD sky separation between two pixels.", "response": "def sky_sep(self, pix1, pix2):\n \"\"\"\n calculate the GCD sky separation (degrees) between two pixels.\n\n Parameters\n ----------\n pix1, pix2 : (float, float)\n The (x,y) pixel coordinates for the two positions.\n\n Returns\n -------\n dist : float\n The distance between the two points (degrees).\n \"\"\"\n pos1 = self.pix2sky(pix1)\n pos2 = self.pix2sky(pix2)\n sep = gcd(pos1[0], pos1[1], pos2[0], pos2[1])\n return sep"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_psf_sky(self, ra, dec):\n # If we don't have a psf map then we just fall back to using the beam\n # from the fits header (including ZA scaling)\n if self.data is None:\n beam = self.wcshelper.get_beam(ra, dec)\n return beam.a, beam.b, beam.pa\n\n x, y = self.sky2pix([ra, dec])\n # We leave the interpolation in the hands of whoever is making these images\n # clamping the x,y coords at the image boundaries just makes sense\n x = int(np.clip(x, 0, self.data.shape[1] - 1))\n y = int(np.clip(y, 0, self.data.shape[2] - 1))\n psf_sky = self.data[:, x, y]\n return psf_sky", "response": "Determine the local psf at a given sky location."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_psf_pix(self, ra, dec):\n psf_sky = self.get_psf_sky(ra, dec)\n psf_pix = self.wcshelper.sky2pix_ellipse([ra, dec], psf_sky[0], psf_sky[1], psf_sky[2])[2:]\n return psf_pix", "response": "Returns the local psf at a given sky location."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_pixbeam(self, ra, dec):\n # If there is no psf image then just use the fits header (plus lat scaling) from the wcshelper\n if self.data is None:\n return self.wcshelper.get_pixbeam(ra, dec)\n # get the beam from the psf image data\n psf = self.get_psf_pix(ra, dec)\n if not np.all(np.isfinite(psf)):\n log.warn(\"PSF requested, returned Null\")\n return None\n return Beam(psf[0], psf[1], psf[2])", "response": "Get the beam at the location specified in pixel coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the beam at the given sky position.", "response": "def get_beam(self, ra, dec):\n \"\"\"\n Get the psf as a :class:`AegeanTools.fits_image.Beam` object.\n\n Parameters\n ----------\n ra, dec : float\n The sky position (degrees).\n\n Returns\n -------\n beam : :class:`AegeanTools.fits_image.Beam`\n The psf at the given location.\n \"\"\"\n if self.data is None:\n return self.wcshelper.beam\n else:\n psf = self.get_psf_sky(ra, dec)\n if not all(np.isfinite(psf)):\n return None\n return Beam(psf[0], psf[1], psf[2])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the area of the beam in square pixels.", "response": "def get_beamarea_pix(self, ra, dec):\n \"\"\"\n Calculate the area of the beam in square pixels.\n\n Parameters\n ----------\n ra, dec : float\n The sky position (degrees).\n\n Returns\n -------\n area : float\n The area of the beam in square pixels.\n \"\"\"\n beam = self.get_pixbeam(ra, dec)\n if beam is None:\n return 0\n return beam.a * beam.b * np.pi"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the area of the beam in square degrees.", "response": "def get_beamarea_deg2(self, ra, dec):\n\n \"\"\"\n Calculate the area of the beam in square degrees.\n\n Parameters\n ----------\n ra, dec : float\n The sky position (degrees).\n\n Returns\n -------\n area : float\n The area of the beam in square degrees.\n \"\"\"\n beam = self.get_beam(ra, dec)\n if beam is None:\n return 0\n return beam.a * beam.b * np.pi"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_start_point(self):\n for i, row in enumerate(self.data):\n for j, _ in enumerate(row):\n if self.data[i, j] != 0: # or not np.isfinite(self.data[i,j]):\n return i, j", "response": "Find the first location in our array that is not empty"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef step(self, x, y):\n up_left = self.solid(x - 1, y - 1)\n up_right = self.solid(x, y - 1)\n down_left = self.solid(x - 1, y)\n down_right = self.solid(x, y)\n\n state = 0\n self.prev = self.next\n # which cells are filled?\n if up_left:\n state |= 1\n if up_right:\n state |= 2\n if down_left:\n state |= 4\n if down_right:\n state |= 8\n\n # what is the next step?\n if state in [1, 5, 13]:\n self.next = self.UP\n elif state in [2, 3, 7]:\n self.next = self.RIGHT\n elif state in [4, 12, 14]:\n self.next = self.LEFT\n elif state in [8, 10, 11]:\n self.next = self.DOWN\n elif state == 6:\n if self.prev == self.UP:\n self.next = self.LEFT\n else:\n self.next = self.RIGHT\n elif state == 9:\n if self.prev == self.RIGHT:\n self.next = self.UP\n else:\n self.next = self.DOWN\n else:\n self.next = self.NOWHERE\n return", "response": "Move from the current location to the next location."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndetermining if the pixel x y is nonzero.", "response": "def solid(self, x, y):\n \"\"\"\n Determine whether the pixel x,y is nonzero\n\n Parameters\n ----------\n x, y : int\n The pixel of interest.\n\n Returns\n -------\n solid : bool\n True if the pixel is not zero.\n \"\"\"\n if not(0 <= x < self.xsize) or not(0 <= y < self.ysize):\n return False\n if self.data[x, y] == 0:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef walk_perimeter(self, startx, starty):\n # checks\n startx = max(startx, 0)\n startx = min(startx, self.xsize)\n starty = max(starty, 0)\n starty = min(starty, self.ysize)\n\n points = []\n\n x, y = startx, starty\n\n while True:\n self.step(x, y)\n if 0 <= x <= self.xsize and 0 <= y <= self.ysize:\n points.append((x, y))\n if self.next == self.UP:\n y -= 1\n elif self.next == self.LEFT:\n x -= 1\n elif self.next == self.DOWN:\n y += 1\n elif self.next == self.RIGHT:\n x += 1\n # stop if we meet some kind of error\n elif self.next == self.NOWHERE:\n break\n # stop when we return to the starting location\n if x == startx and y == starty:\n break\n return points", "response": "Walk the perimeter of a region and return the path taken."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmarching about and trace the outline of our object", "response": "def do_march(self):\n \"\"\"\n March about and trace the outline of our object\n\n Returns\n -------\n perimeter : list\n The pixels on the perimeter of the region [[x1, y1], ...]\n \"\"\"\n x, y = self.find_start_point()\n perimeter = self.walk_perimeter(x, y)\n return perimeter"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nblank all the pixels within the given perimeter.", "response": "def _blank_within(self, perimeter):\n \"\"\"\n Blank all the pixels within the given perimeter.\n\n Parameters\n ----------\n perimeter : list\n The perimeter of the region.\n\n \"\"\"\n # Method:\n # scan around the perimeter filling 'up' from each pixel\n # stopping when we reach the other boundary\n for p in perimeter:\n # if we are on the edge of the data then there is nothing to fill\n if p[0] >= self.data.shape[0] or p[1] >= self.data.shape[1]:\n continue\n # if this pixel is blank then don't fill\n if self.data[p] == 0:\n continue\n\n # blank this pixel\n self.data[p] = 0\n\n # blank until we reach the other perimeter\n for i in range(p[1]+1, self.data.shape[1]):\n q = p[0], i\n # stop when we reach another part of the perimeter\n if q in perimeter:\n break\n # fill everything in between, even inclusions\n self.data[q] = 0\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef elliptical_gaussian(x, y, amp, xo, yo, sx, sy, theta):\n try:\n sint, cost = math.sin(np.radians(theta)), math.cos(np.radians(theta))\n except ValueError as e:\n if 'math domain error' in e.args:\n sint, cost = np.nan, np.nan\n xxo = x - xo\n yyo = y - yo\n exp = (xxo * cost + yyo * sint) ** 2 / sx ** 2 \\\n + (xxo * sint - yyo * cost) ** 2 / sy ** 2\n exp *= -1. / 2\n return amp * np.exp(exp)", "response": "Generate a 2d Gaussian model with the given parameters."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Cmatrix(x, y, sx, sy, theta):\n C = np.vstack([elliptical_gaussian(x, y, 1, i, j, sx, sy, theta) for i, j in zip(x, y)])\n return C", "response": "Construct a correlation matrix corresponding to the data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Bmatrix(C):\n # this version of finding the square root of the inverse matrix\n # suggested by Cath Trott\n L, Q = eigh(C)\n # force very small eigenvalues to have some minimum non-zero value\n minL = 1e-9*L[-1]\n L[L < minL] = minL\n S = np.diag(1 / np.sqrt(L))\n B = Q.dot(S)\n return B", "response": "Calculate a matrix which is effectively the square root of the correlation matrix C."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef jacobian(pars, x, y):\n\n matrix = []\n\n for i in range(pars['components'].value):\n prefix = \"c{0}_\".format(i)\n amp = pars[prefix + 'amp'].value\n xo = pars[prefix + 'xo'].value\n yo = pars[prefix + 'yo'].value\n sx = pars[prefix + 'sx'].value\n sy = pars[prefix + 'sy'].value\n theta = pars[prefix + 'theta'].value\n\n # The derivative with respect to component i doesn't depend on any other components\n # thus the model should not contain the other components\n model = elliptical_gaussian(x, y, amp, xo, yo, sx, sy, theta)\n\n # precompute for speed\n sint = np.sin(np.radians(theta))\n cost = np.cos(np.radians(theta))\n xxo = x - xo\n yyo = y - yo\n xcos, ycos = xxo * cost, yyo * cost\n xsin, ysin = xxo * sint, yyo * sint\n\n if pars[prefix + 'amp'].vary:\n dmds = model / amp\n matrix.append(dmds)\n\n if pars[prefix + 'xo'].vary:\n dmdxo = cost * (xcos + ysin) / sx ** 2 + sint * (xsin - ycos) / sy ** 2\n dmdxo *= model\n matrix.append(dmdxo)\n\n if pars[prefix + 'yo'].vary:\n dmdyo = sint * (xcos + ysin) / sx ** 2 - cost * (xsin - ycos) / sy ** 2\n dmdyo *= model\n matrix.append(dmdyo)\n\n if pars[prefix + 'sx'].vary:\n dmdsx = model / sx ** 3 * (xcos + ysin) ** 2\n matrix.append(dmdsx)\n\n if pars[prefix + 'sy'].vary:\n dmdsy = model / sy ** 3 * (xsin - ycos) ** 2\n matrix.append(dmdsy)\n\n if pars[prefix + 'theta'].vary:\n dmdtheta = model * (sy ** 2 - sx ** 2) * (xsin - ycos) * (xcos + ysin) / sx ** 2 / sy ** 2\n matrix.append(dmdtheta)\n\n return np.array(matrix)", "response": "This function calculates the Jacobian of an elliptical gaussian model that contains multiple Gaussians and returns the Jacobian of the model."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrap around the jacobian function to get the output of the lmfit. jacobian function.", "response": "def lmfit_jacobian(pars, x, y, errs=None, B=None, emp=False):\n \"\"\"\n Wrapper around :func:`AegeanTools.fitting.jacobian` and :func:`AegeanTools.fitting.emp_jacobian`\n which gives the output in a format that is required for lmfit.\n\n Parameters\n ----------\n pars : lmfit.Model\n The model parameters\n\n x, y : list\n Locations at which the jacobian is being evaluated\n\n errs : list\n a vector of 1\\sigma errors (optional). Default = None\n\n B : 2d-array\n a B-matrix (optional) see :func:`AegeanTools.fitting.Bmatrix`\n\n emp : bool\n If true the use the empirical Jacobian, otherwise use the analytical one.\n Default = False.\n\n Returns\n -------\n j : 2d-array\n A Jacobian.\n\n See Also\n --------\n :func:`AegeanTools.fitting.Bmatrix`\n :func:`AegeanTools.fitting.jacobian`\n :func:`AegeanTools.fitting.emp_jacobian`\n\n \"\"\"\n if emp:\n matrix = emp_jacobian(pars, x, y)\n else:\n # calculate in the normal way\n matrix = jacobian(pars, x, y)\n # now munge this to be as expected for lmfit\n matrix = np.vstack(matrix)\n\n if errs is not None:\n matrix /= errs\n # matrix = matrix.dot(errs)\n\n if B is not None:\n matrix = matrix.dot(B)\n\n matrix = np.transpose(matrix)\n return matrix"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef hessian(pars, x, y):\n j = 0 # keeping track of the number of variable parameters\n # total number of variable parameters\n ntvar = np.sum([pars[k].vary for k in pars.keys() if k != 'components'])\n # construct an empty matrix of the correct size\n hmat = np.zeros((ntvar, ntvar, x.shape[0], x.shape[1]))\n npvar = 0\n\n for i in range(pars['components'].value):\n prefix = \"c{0}_\".format(i)\n amp = pars[prefix + 'amp'].value\n xo = pars[prefix + 'xo'].value\n yo = pars[prefix + 'yo'].value\n sx = pars[prefix + 'sx'].value\n sy = pars[prefix + 'sy'].value\n theta = pars[prefix + 'theta'].value\n\n amp_var = pars[prefix + 'amp'].vary\n xo_var = pars[prefix + 'xo'].vary\n yo_var = pars[prefix + 'yo'].vary\n sx_var = pars[prefix + 'sx'].vary\n sy_var = pars[prefix + 'sy'].vary\n theta_var = pars[prefix + 'theta'].vary\n\n # precomputed for speed\n model = elliptical_gaussian(x, y, amp, xo, yo, sx, sy, theta)\n sint = np.sin(np.radians(theta))\n sin2t = np.sin(np.radians(2*theta))\n cost = np.cos(np.radians(theta))\n cos2t = np.cos(np.radians(2*theta))\n sx2 = sx**2\n sy2 = sy**2\n xxo = x-xo\n yyo = y-yo\n xcos, ycos = xxo*cost, yyo*cost\n xsin, ysin = xxo*sint, yyo*sint\n\n if amp_var:\n k = npvar # second round of keeping track of variable params\n # H(amp,amp)/G = 0\n hmat[j][k] = 0\n k += 1\n\n if xo_var:\n # H(amp,xo)/G = 1.0*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t))/(amp*sx**2*sy**2)\n hmat[j][k] = (xsin - ycos)*sint/sy2 + (xcos + ysin)*cost/sx2\n hmat[j][k] *= model\n k += 1\n\n if yo_var:\n # H(amp,yo)/G = 1.0*(-sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*cos(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t))/(amp*sx**2*sy**2)\n hmat[j][k] = -(xsin - ycos)*cost/sy2 + (xcos + ysin)*sint/sx2\n hmat[j][k] *= model/amp\n k += 1\n\n if sx_var:\n # H(amp,sx)/G = 1.0*((x - xo)*cos(t) + (y - yo)*sin(t))**2/(amp*sx**3)\n hmat[j][k] = (xcos + ysin)**2\n hmat[j][k] *= model/(amp*sx**3)\n k += 1\n\n if sy_var:\n # H(amp,sy) = 1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))**2/(amp*sy**3)\n hmat[j][k] = (xsin - ycos)**2\n hmat[j][k] *= model/(amp*sy**3)\n k += 1\n\n if theta_var:\n # H(amp,t) = (-1.0*sx**2 + sy**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))*((x - xo)*cos(t) + (y - yo)*sin(t))/(amp*sx**2*sy**2)\n hmat[j][k] = (xsin - ycos)*(xcos + ysin)\n hmat[j][k] *= sy2-sx2\n hmat[j][k] *= model/(amp*sx2*sy2)\n # k += 1\n j += 1\n\n if xo_var:\n k = npvar\n if amp_var:\n # H(xo,amp)/G = H(amp,xo)\n hmat[j][k] = hmat[k][j]\n k += 1\n\n # if xo_var:\n # H(xo,xo)/G = 1.0*(-sx**2*sy**2*(sx**2*sin(t)**2 + sy**2*cos(t)**2) + (sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t))**2)/(sx**4*sy**4)\n hmat[j][k] = -sx2*sy2*(sx2*sint**2 + sy2*cost**2)\n hmat[j][k] += (sx2*(xsin - ycos)*sint + sy2*(xcos + ysin)*cost)**2\n hmat[j][k] *= model/ (sx2**2*sy2**2)\n k += 1\n\n if yo_var:\n # H(xo,yo)/G = 1.0*(sx**2*sy**2*(sx**2 - sy**2)*sin(2*t)/2 - (sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*cos(t) - sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t)))/(sx**4*sy**4)\n hmat[j][k] = sx2*sy2*(sx2 - sy2)*sin2t/2\n hmat[j][k] -= (sx2*(xsin - ycos)*sint + sy2*(xcos + ysin)*cost)*(sx2*(xsin -ycos)*cost - sy2*(xcos + ysin)*sint)\n hmat[j][k] *= model / (sx**4*sy**4)\n k += 1\n\n if sx_var:\n # H(xo,sx) = ((x - xo)*cos(t) + (y - yo)*sin(t))*(-2.0*sx**2*sy**2*cos(t) + 1.0*((x - xo)*cos(t) + (y - yo)*sin(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t)))/(sx**5*sy**2)\n hmat[j][k] = (xcos + ysin)\n hmat[j][k] *= -2*sx2*sy2*cost + (xcos + ysin)*(sx2*(xsin - ycos)*sint + sy2*(xcos + ysin)*cost)\n hmat[j][k] *= model / (sx**5*sy2)\n k += 1\n\n if sy_var:\n # H(xo,sy) = ((x - xo)*sin(t) + (-y + yo)*cos(t))*(-2.0*sx**2*sy**2*sin(t) + 1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t)))/(sx2*sy**5)\n hmat[j][k] = (xsin - ycos)\n hmat[j][k] *= -2*sx2*sy2*sint + (xsin - ycos)*(sx2*(xsin - ycos)*sint + sy2*(xcos + ysin)*cost)\n hmat[j][k] *= model/(sx2*sy**5)\n k += 1\n\n if theta_var:\n # H(xo,t) = 1.0*(sx**2*sy**2*(sx**2 - sy**2)*(x*sin(2*t) - xo*sin(2*t) - y*cos(2*t) + yo*cos(2*t)) + (-sx**2 + 1.0*sy**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))*((x - xo)*cos(t) + (y - yo)*sin(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*sin(t) + sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*cos(t)))/(sx**4*sy**4)\n # second part\n hmat[j][k] = (sy2-sx2)*(xsin - ycos)*(xcos + ysin)\n hmat[j][k] *= sx2*(xsin -ycos)*sint + sy2*(xcos + ysin)*cost\n # first part\n hmat[j][k] += sx2*sy2*(sx2 - sy2)*(xxo*sin2t -yyo*cos2t)\n hmat[j][k] *= model/(sx**4*sy**4)\n # k += 1\n j += 1\n\n if yo_var:\n k = npvar\n if amp_var:\n # H(yo,amp)/G = H(amp,yo)\n hmat[j][k] = hmat[0][2]\n k += 1\n\n if xo_var:\n # H(yo,xo)/G = H(xo,yo)/G\n hmat[j][k] =hmat[1][2]\n k += 1\n\n # if yo_var:\n # H(yo,yo)/G = 1.0*(-sx**2*sy**2*(sx**2*cos(t)**2 + sy**2*sin(t)**2) + (sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*cos(t) - sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t))**2)/(sx**4*sy**4)\n hmat[j][k] = (sx2*(xsin - ycos)*cost - sy2*(xcos + ysin)*sint)**2 / (sx2**2*sy2**2)\n hmat[j][k] -= cost**2/sy2 + sint**2/sx2\n hmat[j][k] *= model\n k += 1\n\n if sx_var:\n # H(yo,sx)/G = -((x - xo)*cos(t) + (y - yo)*sin(t))*(2.0*sx**2*sy**2*sin(t) + 1.0*((x - xo)*cos(t) + (y - yo)*sin(t))*(sx**2*((x - xo)*sin(t) - (y - yo)*cos(t))*cos(t) - sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t)))/(sx**5*sy**2)\n hmat[j][k] = -1*(xcos + ysin)\n hmat[j][k] *= 2*sx2*sy2*sint + (xcos + ysin)*(sx2*(xsin - ycos)*cost - sy2*(xcos + ysin)*sint)\n hmat[j][k] *= model/(sx**5*sy2)\n k += 1\n\n if sy_var:\n # H(yo,sy)/G = ((x - xo)*sin(t) + (-y + yo)*cos(t))*(2.0*sx**2*sy**2*cos(t) - 1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*cos(t) - sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t)))/(sx**2*sy**5)\n hmat[j][k] = (xsin -ycos)\n hmat[j][k] *= 2*sx2*sy2*cost - (xsin - ycos)*(sx2*(xsin - ycos)*cost - sy2*(xcos + ysin)*sint)\n hmat[j][k] *= model/(sx2*sy**5)\n k += 1\n\n if theta_var:\n # H(yo,t)/G = 1.0*(sx**2*sy**2*(sx**2*(-x*cos(2*t) + xo*cos(2*t) - y*sin(2*t) + yo*sin(2*t)) + sy**2*(x*cos(2*t) - xo*cos(2*t) + y*sin(2*t) - yo*sin(2*t))) + (1.0*sx**2 - sy**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))*((x - xo)*cos(t) + (y - yo)*sin(t))*(sx**2*((x - xo)*sin(t) + (-y + yo)*cos(t))*cos(t) - sy**2*((x - xo)*cos(t) + (y - yo)*sin(t))*sin(t)))/(sx**4*sy**4)\n hmat[j][k] = (sx2 - sy2)*(xsin - ycos)*(xcos + ysin)\n hmat[j][k] *= (sx2*(xsin - ycos)*cost - sy2*(xcos + ysin)*sint)\n hmat[j][k] += sx2*sy2*(sx2-sy2)*(-x*cos2t + xo*cos2t - y*sin2t + yo*sin2t)\n hmat[j][k] *= model/(sx**4*sy**4)\n # k += 1\n j += 1\n\n if sx_var:\n k = npvar\n if amp_var:\n # H(sx,amp)/G = H(amp,sx)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n\n if xo_var:\n # H(sx,xo)/G = H(xo,sx)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n\n if yo_var:\n # H(sx,yo)/G = H(yo/sx)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n\n # if sx_var:\n # H(sx,sx)/G = (-3.0*sx**2 + 1.0*((x - xo)*cos(t) + (y - yo)*sin(t))**2)*((x - xo)*cos(t) + (y - yo)*sin(t))**2/sx**6\n hmat[j][k] = -3*sx2 + (xcos + ysin)**2\n hmat[j][k] *= (xcos + ysin)**2\n hmat[j][k] *= model/sx**6\n k += 1\n\n if sy_var:\n # H(sx,sy)/G = 1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))**2*((x - xo)*cos(t) + (y - yo)*sin(t))**2/(sx**3*sy**3)\n hmat[j][k] = (xsin - ycos)**2 * (xcos + ysin)**2\n hmat[j][k] *= model/(sx**3*sy**3)\n k += 1\n\n if theta_var:\n # H(sx,t)/G = (-2.0*sx**2*sy**2 + 1.0*(-sx**2 + sy**2)*((x - xo)*cos(t) + (y - yo)*sin(t))**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))*((x - xo)*cos(t) + (y - yo)*sin(t))/(sx**5*sy**2)\n hmat[j][k] = -2*sx2*sy2 + (sy2 - sx2)*(xcos + ysin)**2\n hmat[j][k] *= (xsin -ycos)*(xcos + ysin)\n hmat[j][k] *= model/(sx**5*sy**2)\n # k += 1\n j += 1\n\n if sy_var:\n k = npvar\n if amp_var:\n # H(sy,amp)/G = H(amp,sy)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if xo_var:\n # H(sy,xo)/G = H(xo,sy)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if yo_var:\n # H(sy,yo)/G = H(yo/sy)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if sx_var:\n # H(sy,sx)/G = H(sx,sy)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n\n # if sy_var:\n # H(sy,sy)/G = (-3.0*sy**2 + 1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))**2/sy**6\n hmat[j][k] = -3*sy2 + (xsin - ycos)**2\n hmat[j][k] *= (xsin - ycos)**2\n hmat[j][k] *= model/sy**6\n k += 1\n\n if theta_var:\n # H(sy,t)/G = (2.0*sx**2*sy**2 + 1.0*(-sx**2 + sy**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))**2)*((x - xo)*sin(t) + (-y + yo)*cos(t))*((x - xo)*cos(t) + (y - yo)*sin(t))/(sx**2*sy**5)\n hmat[j][k] = 2*sx2*sy2 + (sy2 - sx2)*(xsin - ycos)**2\n hmat[j][k] *= (xsin - ycos)*(xcos + ysin)\n hmat[j][k] *= model/(sx**2*sy**5)\n # k += 1\n j += 1\n\n if theta_var:\n k = npvar\n if amp_var:\n # H(t,amp)/G = H(amp,t)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if xo_var:\n # H(t,xo)/G = H(xo,t)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if yo_var:\n # H(t,yo)/G = H(yo/t)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if sx_var:\n # H(t,sx)/G = H(sx,t)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n if sy_var:\n # H(t,sy)/G = H(sy,t)/G\n hmat[j][k] = hmat[k][j]\n k += 1\n # if theta_var:\n # H(t,t)/G = (sx**2*sy**2*(sx**2*(((x - xo)*sin(t) + (-y + yo)*cos(t))**2 - 1.0*((x - xo)*cos(t) + (y - yo)*sin(t))**2) + sy**2*(-1.0*((x - xo)*sin(t) + (-y + yo)*cos(t))**2 + ((x - xo)*cos(t) + (y - yo)*sin(t))**2)) + (sx**2 - 1.0*sy**2)**2*((x - xo)*sin(t) + (-y + yo)*cos(t))**2*((x - xo)*cos(t) + (y - yo)*sin(t))**2)/(sx**4*sy**4)\n hmat[j][k] = sx2*sy2\n hmat[j][k] *= sx2*((xsin - ycos)**2 - (xcos + ysin)**2) + sy2*((xcos + ysin)**2 - (xsin - ycos)**2)\n hmat[j][k] += (sx2 - sy2)**2*(xsin - ycos)**2*(xcos + ysin)**2\n hmat[j][k] *= model/(sx**4*sy**4)\n # j += 1\n\n # save the number of variables for the next iteration\n # as we need to start our indexing at this number\n npvar = k\n return np.array(hmat)", "response": "Create a hessian matrix corresponding to the source model pars."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the hessian matrix empirically.", "response": "def emp_hessian(pars, x, y):\n \"\"\"\n Calculate the hessian matrix empirically.\n Create a hessian matrix corresponding to the source model 'pars'\n Only parameters that vary will contribute to the hessian.\n Thus there will be a total of nvar x nvar entries, each of which is a\n len(x) x len(y) array.\n\n Parameters\n ----------\n pars : lmfit.Parameters\n The model\n x, y : list\n locations at which to evaluate the Hessian\n\n Returns\n -------\n h : np.array\n Hessian. Shape will be (nvar, nvar, len(x), len(y))\n\n Notes\n -----\n Uses :func:`AegeanTools.fitting.emp_jacobian` to calculate the first order derivatives.\n\n See Also\n --------\n :func:`AegeanTools.fitting.hessian`\n \"\"\"\n eps = 1e-5\n matrix = []\n for i in range(pars['components'].value):\n model = emp_jacobian(pars, x, y)\n prefix = \"c{0}_\".format(i)\n for p in ['amp', 'xo', 'yo', 'sx', 'sy', 'theta']:\n if pars[prefix+p].vary:\n pars[prefix+p].value += eps\n dm2didj = emp_jacobian(pars, x, y) - model\n matrix.append(dm2didj/eps)\n pars[prefix+p].value -= eps\n matrix = np.array(matrix)\n return matrix"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef nan_acf(noise):\n corr = np.zeros(noise.shape)\n ix,jx = noise.shape\n for i in range(ix):\n si_min = slice(i, None, None)\n si_max = slice(None, ix-i, None)\n for j in range(jx):\n sj_min = slice(j, None, None)\n sj_max = slice(None, jx-j, None)\n if np.all(np.isnan(noise[si_min, sj_min])) or np.all(np.isnan(noise[si_max, sj_max])):\n corr[i, j] = np.nan\n else:\n corr[i, j] = np.nansum(noise[si_min, sj_min] * noise[si_max, sj_max])\n # return the normalised acf\n return corr / np.nanmax(corr)", "response": "Calculate the autocorrelation function of the noise image that contains nans\n ."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the expected bias on each of the parameters in the model.", "response": "def RB_bias(data, pars, ita=None, acf=None):\n \"\"\"\n Calculate the expected bias on each of the parameters in the model pars.\n Only parameters that are allowed to vary will have a bias.\n Calculation follows the description of Refrieger & Brown 1998 (cite).\n\n\n Parameters\n ----------\n data : 2d-array\n data that was fit\n\n pars : lmfit.Parameters\n The model\n\n ita : 2d-array\n The ita matrix (optional).\n\n acf : 2d-array\n The acf for the data.\n\n Returns\n -------\n bias : array\n The bias on each of the parameters\n \"\"\"\n log.info(\"data {0}\".format(data.shape))\n nparams = np.sum([pars[k].vary for k in pars.keys() if k != 'components'])\n # masked pixels\n xm, ym = np.where(np.isfinite(data))\n # all pixels\n x, y = np.indices(data.shape)\n # Create the jacobian as an AxN array accounting for the masked pixels\n j = np.array(np.vsplit(lmfit_jacobian(pars, xm, ym).T, nparams)).reshape(nparams, -1)\n\n h = hessian(pars, x, y)\n # mask the hessian to be AxAxN array\n h = h[:, :, xm, ym]\n Hij = np.einsum('ik,jk', j, j)\n Dij = np.linalg.inv(Hij)\n Bijk = np.einsum('ip,jkp', j, h)\n Eilkm = np.einsum('il,km', Dij, Dij)\n\n Cimn_1 = -1 * np.einsum('krj,ir,km,jn', Bijk, Dij, Dij, Dij)\n Cimn_2 = -1./2 * np.einsum('rkj,ir,km,jn', Bijk, Dij, Dij, Dij)\n Cimn = Cimn_1 + Cimn_2\n\n if ita is None:\n # N is the noise (data-model)\n N = data - ntwodgaussian_lmfit(pars)(x, y)\n if acf is None:\n acf = nan_acf(N)\n ita = make_ita(N, acf=acf)\n log.info('acf.shape {0}'.format(acf.shape))\n log.info('acf[0] {0}'.format(acf[0]))\n log.info('ita.shape {0}'.format(ita.shape))\n log.info('ita[0] {0}'.format(ita[0]))\n\n # Included for completeness but not required\n\n # now mask/ravel the noise\n # N = N[np.isfinite(N)].ravel()\n # Pi = np.einsum('ip,p', j, N)\n # Qij = np.einsum('ijp,p', h, N)\n\n Vij = np.einsum('ip,jq,pq', j, j, ita)\n Uijk = np.einsum('ip,jkq,pq', j, h, ita)\n\n bias_1 = np.einsum('imn, mn', Cimn, Vij)\n bias_2 = np.einsum('ilkm, mlk', Eilkm, Uijk)\n bias = bias_1 + bias_2\n log.info('bias {0}'.format(bias))\n return bias"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bias_correct(params, data, acf=None):\n bias = RB_bias(data, params, acf=acf)\n i = 0\n for p in params:\n if 'theta' in p:\n continue\n if params[p].vary:\n params[p].value -= bias[i]\n i += 1\n return", "response": "Calculate and apply a bias correction to the given fit parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates the parameter errors for a fitted source based on the conditions of the source.", "response": "def condon_errors(source, theta_n, psf=None):\n \"\"\"\n Calculate the parameter errors for a fitted source\n using the description of Condon'97\n All parameters are assigned errors, assuming that all params were fit.\n If some params were held fixed then these errors are overestimated.\n\n Parameters\n ----------\n source : :class:`AegeanTools.models.SimpleSource`\n The source which was fit.\n\n theta_n : float or None\n A measure of the beam sampling. (See Condon'97).\n\n psf : :class:`AegeanTools.fits_image.Beam`\n The psf at the location of the source.\n\n Returns\n -------\n None\n\n \"\"\"\n\n # indices for the calculation or rho\n alphas = {'amp': (3. / 2, 3. / 2),\n 'major': (5. / 2, 1. / 2),\n 'xo': (5. / 2, 1. / 2),\n 'minor': (1. / 2, 5. / 2),\n 'yo': (1. / 2, 5. / 2),\n 'pa': (1. / 2, 5. / 2)}\n\n major = source.a / 3600. # degrees\n minor = source.b / 3600. # degrees\n phi = np.radians(source.pa) # radians\n if psf is not None:\n beam = psf.get_beam(source.ra, source.dec)\n if beam is not None:\n theta_n = np.hypot(beam.a, beam.b)\n print(beam, theta_n)\n\n if theta_n is None:\n source.err_a = source.err_b = source.err_peak_flux = source.err_pa = source.err_int_flux = 0.0\n return\n\n smoothing = major * minor / (theta_n ** 2)\n factor1 = (1 + (major / theta_n))\n factor2 = (1 + (minor / theta_n))\n snr = source.peak_flux / source.local_rms\n # calculation of rho2 depends on the parameter being used so we lambda this into a function\n rho2 = lambda x: smoothing / 4 * factor1 ** alphas[x][0] * factor2 ** alphas[x][1] * snr ** 2\n\n source.err_peak_flux = source.peak_flux * np.sqrt(2 / rho2('amp'))\n source.err_a = major * np.sqrt(2 / rho2('major')) * 3600. # arcsec\n source.err_b = minor * np.sqrt(2 / rho2('minor')) * 3600. # arcsec\n\n err_xo2 = 2. / rho2('xo') * major ** 2 / (8 * np.log(2)) # Condon'97 eq 21\n err_yo2 = 2. / rho2('yo') * minor ** 2 / (8 * np.log(2))\n source.err_ra = np.sqrt(err_xo2 * np.sin(phi)**2 + err_yo2 * np.cos(phi)**2)\n source.err_dec = np.sqrt(err_xo2 * np.cos(phi)**2 + err_yo2 * np.sin(phi)**2)\n\n if (major == 0) or (minor == 0):\n source.err_pa = ERR_MASK\n # if major/minor are very similar then we should not be able to figure out what pa is.\n elif abs(2 * (major-minor) / (major+minor)) < 0.01:\n source.err_pa = ERR_MASK\n else:\n source.err_pa = np.degrees(np.sqrt(4 / rho2('pa')) * (major * minor / (major ** 2 - minor ** 2)))\n\n # integrated flux error\n err2 = (source.err_peak_flux / source.peak_flux) ** 2\n err2 += (theta_n ** 2 / (major * minor)) * ((source.err_a / source.a) ** 2 + (source.err_b / source.b) ** 2)\n source.err_int_flux = source.int_flux * np.sqrt(err2)\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef errors(source, model, wcshelper):\n\n # if the source wasn't fit then all errors are -1\n if source.flags & (flags.NOTFIT | flags.FITERR):\n source.err_peak_flux = source.err_a = source.err_b = source.err_pa = ERR_MASK\n source.err_ra = source.err_dec = source.err_int_flux = ERR_MASK\n return source\n # copy the errors from the model\n prefix = \"c{0}_\".format(source.source)\n err_amp = model[prefix + 'amp'].stderr\n xo, yo = model[prefix + 'xo'].value, model[prefix + 'yo'].value\n err_xo = model[prefix + 'xo'].stderr\n err_yo = model[prefix + 'yo'].stderr\n\n sx, sy = model[prefix + 'sx'].value, model[prefix + 'sy'].value\n err_sx = model[prefix + 'sx'].stderr\n err_sy = model[prefix + 'sy'].stderr\n\n theta = model[prefix + 'theta'].value\n err_theta = model[prefix + 'theta'].stderr\n\n source.err_peak_flux = err_amp\n pix_errs = [err_xo, err_yo, err_sx, err_sy, err_theta]\n\n log.debug(\"Pix errs: {0}\".format(pix_errs))\n\n ref = wcshelper.pix2sky([xo, yo])\n # check to see if the reference position has a valid WCS coordinate\n # It is possible for this to fail, even if the ra/dec conversion works elsewhere\n if not all(np.isfinite(ref)):\n source.flags |= flags.WCSERR\n source.err_peak_flux = source.err_a = source.err_b = source.err_pa = ERR_MASK\n source.err_ra = source.err_dec = source.err_int_flux = ERR_MASK\n return source\n\n # position errors\n if model[prefix + 'xo'].vary and model[prefix + 'yo'].vary \\\n and all(np.isfinite([err_xo, err_yo])):\n offset = wcshelper.pix2sky([xo + err_xo, yo + err_yo])\n source.err_ra = gcd(ref[0], ref[1], offset[0], ref[1])\n source.err_dec = gcd(ref[0], ref[1], ref[0], offset[1])\n else:\n source.err_ra = source.err_dec = -1\n\n if model[prefix + 'theta'].vary and np.isfinite(err_theta):\n # pa error\n off1 = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n off2 = wcshelper.pix2sky(\n [xo + sx * np.cos(np.radians(theta + err_theta)), yo + sy * np.sin(np.radians(theta + err_theta))])\n source.err_pa = abs(bear(ref[0], ref[1], off1[0], off1[1]) - bear(ref[0], ref[1], off2[0], off2[1]))\n else:\n source.err_pa = ERR_MASK\n\n if model[prefix + 'sx'].vary and model[prefix + 'sy'].vary \\\n and all(np.isfinite([err_sx, err_sy])):\n # major axis error\n ref = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n offset = wcshelper.pix2sky(\n [xo + (sx + err_sx) * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n source.err_a = gcd(ref[0], ref[1], offset[0], offset[1]) * 3600\n\n # minor axis error\n ref = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta + 90)), yo + sy * np.sin(np.radians(theta + 90))])\n offset = wcshelper.pix2sky(\n [xo + sx * np.cos(np.radians(theta + 90)), yo + (sy + err_sy) * np.sin(np.radians(theta + 90))])\n source.err_b = gcd(ref[0], ref[1], offset[0], offset[1]) * 3600\n else:\n source.err_a = source.err_b = ERR_MASK\n\n sqerr = 0\n sqerr += (source.err_peak_flux / source.peak_flux) ** 2 if source.err_peak_flux > 0 else 0\n sqerr += (source.err_a / source.a) ** 2 if source.err_a > 0 else 0\n sqerr += (source.err_b / source.b) ** 2 if source.err_b > 0 else 0\n if sqerr == 0:\n source.err_int_flux = ERR_MASK\n else:\n source.err_int_flux = abs(source.int_flux * np.sqrt(sqerr))\n\n return source", "response": "Convert pixel based errors into sky coord errors"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting pixel based errors into sky coord errors Uses covariance matrix for ra dec errors Uses a linear approach to calculate the error in the source obejct.", "response": "def new_errors(source, model, wcshelper): # pragma: no cover\n \"\"\"\n Convert pixel based errors into sky coord errors\n Uses covariance matrix for ra/dec errors\n and calculus approach to a/b/pa errors\n\n Parameters\n ----------\n source : :class:`AegeanTools.models.SimpleSource`\n The source which was fit.\n\n model : lmfit.Parameters\n The model which was fit.\n\n wcshelper : :class:`AegeanTools.wcs_helpers.WCSHelper`\n WCS information.\n\n Returns\n -------\n source : :class:`AegeanTools.models.SimpleSource`\n The modified source obejct.\n\n \"\"\"\n\n # if the source wasn't fit then all errors are -1\n if source.flags & (flags.NOTFIT | flags.FITERR):\n source.err_peak_flux = source.err_a = source.err_b = source.err_pa = ERR_MASK\n source.err_ra = source.err_dec = source.err_int_flux = ERR_MASK\n return source\n # copy the errors/values from the model\n prefix = \"c{0}_\".format(source.source)\n err_amp = model[prefix + 'amp'].stderr\n xo, yo = model[prefix + 'xo'].value, model[prefix + 'yo'].value\n err_xo = model[prefix + 'xo'].stderr\n err_yo = model[prefix + 'yo'].stderr\n\n sx, sy = model[prefix + 'sx'].value, model[prefix + 'sy'].value\n err_sx = model[prefix + 'sx'].stderr\n err_sy = model[prefix + 'sy'].stderr\n\n theta = model[prefix + 'theta'].value\n err_theta = model[prefix + 'theta'].stderr\n\n # the peak flux error doesn't need to be converted, just copied\n source.err_peak_flux = err_amp\n\n pix_errs = [err_xo, err_yo, err_sx, err_sy, err_theta]\n\n # check for inf/nan errors -> these sources have poor fits.\n if not all(a is not None and np.isfinite(a) for a in pix_errs):\n source.flags |= flags.FITERR\n source.err_peak_flux = source.err_a = source.err_b = source.err_pa = ERR_MASK\n source.err_ra = source.err_dec = source.err_int_flux = ERR_MASK\n return source\n\n # calculate the reference coordinate\n ref = wcshelper.pix2sky([xo, yo])\n # check to see if the reference position has a valid WCS coordinate\n # It is possible for this to fail, even if the ra/dec conversion works elsewhere\n if not all(np.isfinite(ref)):\n source.flags |= flags.WCSERR\n source.err_peak_flux = source.err_a = source.err_b = source.err_pa = ERR_MASK\n source.err_ra = source.err_dec = source.err_int_flux = ERR_MASK\n return source\n\n # calculate position errors by transforming the error ellipse\n if model[prefix + 'xo'].vary and model[prefix + 'yo'].vary:\n # determine the error ellipse from the Jacobian\n mat = model.covar[1:3, 1:3]\n if not(np.all(np.isfinite(mat))):\n source.err_ra = source.err_dec = ERR_MASK\n else:\n (a, b), e = np.linalg.eig(mat)\n pa = np.degrees(np.arctan2(*e[0]))\n # transform this ellipse into sky coordinates\n _, _, major, minor, pa = wcshelper.pix2sky_ellipse([xo, yo], a, b, pa)\n\n # now determine the radius of the ellipse along the ra/dec directions.\n source.err_ra = major*minor / np.hypot(major*np.sin(np.radians(pa)), minor*np.cos(np.radians(pa)))\n source.err_dec = major*minor / np.hypot(major*np.cos(np.radians(pa)), minor*np.sin(np.radians(pa)))\n else:\n source.err_ra = source.err_dec = -1\n\n if model[prefix + 'theta'].vary:\n # pa error\n off1 = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n # offset by 1 degree\n off2 = wcshelper.pix2sky(\n [xo + sx * np.cos(np.radians(theta + 1)), yo + sy * np.sin(np.radians(theta + 1))])\n # scale the initial theta error by this amount\n source.err_pa = abs(bear(ref[0], ref[1], off1[0], off1[1]) - bear(ref[0], ref[1], off2[0], off2[1])) * err_theta\n else:\n source.err_pa = ERR_MASK\n\n if model[prefix + 'sx'].vary and model[prefix + 'sy'].vary:\n # major axis error\n ref = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n # offset by 0.1 pixels\n offset = wcshelper.pix2sky(\n [xo + (sx + 0.1) * np.cos(np.radians(theta)), yo + sy * np.sin(np.radians(theta))])\n source.err_a = gcd(ref[0], ref[1], offset[0], offset[1])/0.1 * err_sx * 3600\n\n # minor axis error\n ref = wcshelper.pix2sky([xo + sx * np.cos(np.radians(theta + 90)), yo + sy * np.sin(np.radians(theta + 90))])\n # offset by 0.1 pixels\n offset = wcshelper.pix2sky(\n [xo + sx * np.cos(np.radians(theta + 90)), yo + (sy + 0.1) * np.sin(np.radians(theta + 90))])\n source.err_b = gcd(ref[0], ref[1], offset[0], offset[1])/0.1*err_sy * 3600\n else:\n source.err_a = source.err_b = ERR_MASK\n sqerr = 0\n sqerr += (source.err_peak_flux / source.peak_flux) ** 2 if source.err_peak_flux > 0 else 0\n sqerr += (source.err_a / source.a) ** 2 if source.err_a > 0 else 0\n sqerr += (source.err_b / source.b) ** 2 if source.err_b > 0 else 0\n source.err_int_flux = abs(source.int_flux * np.sqrt(sqerr))\n\n return source"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a function that calculates the model given by params at pixel coordinates x y.", "response": "def ntwodgaussian_lmfit(params):\n \"\"\"\n Convert an lmfit.Parameters object into a function which calculates the model.\n\n\n Parameters\n ----------\n params : lmfit.Parameters\n Model parameters, can have multiple components.\n\n Returns\n -------\n model : func\n A function f(x,y) that will compute the model.\n \"\"\"\n\n def rfunc(x, y):\n \"\"\"\n Compute the model given by params, at pixel coordinates x,y\n\n Parameters\n ----------\n x, y : numpy.ndarray\n The x/y pixel coordinates at which the model is being evaluated\n\n Returns\n -------\n result : numpy.ndarray\n Model\n \"\"\"\n result = None\n for i in range(params['components'].value):\n prefix = \"c{0}_\".format(i)\n # I hope this doesn't kill our run time\n amp = np.nan_to_num(params[prefix + 'amp'].value)\n xo = params[prefix + 'xo'].value\n yo = params[prefix + 'yo'].value\n sx = params[prefix + 'sx'].value\n sy = params[prefix + 'sy'].value\n theta = params[prefix + 'theta'].value\n if result is not None:\n result += elliptical_gaussian(x, y, amp, xo, yo, sx, sy, theta)\n else:\n result = elliptical_gaussian(x, y, amp, xo, yo, sx, sy, theta)\n return result\n\n return rfunc"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef do_lmfit(data, params, B=None, errs=None, dojac=True):\n # copy the params so as not to change the initial conditions\n # in case we want to use them elsewhere\n params = copy.deepcopy(params)\n data = np.array(data)\n mask = np.where(np.isfinite(data))\n\n def residual(params, **kwargs):\n \"\"\"\n The residual function required by lmfit\n\n Parameters\n ----------\n params: lmfit.Params\n The parameters of the model being fit\n\n Returns\n -------\n result : numpy.ndarray\n Model - Data\n \"\"\"\n f = ntwodgaussian_lmfit(params) # A function describing the model\n model = f(*mask) # The actual model\n if B is None:\n return model - data[mask]\n else:\n return (model - data[mask]).dot(B)\n\n if dojac:\n result = lmfit.minimize(residual, params, kws={'x': mask[0], 'y': mask[1], 'B': B, 'errs': errs}, Dfun=lmfit_jacobian)\n else:\n result = lmfit.minimize(residual, params, kws={'x': mask[0], 'y': mask[1], 'B': B, 'errs': errs})\n\n # Remake the residual so that it is once again (model - data)\n if B is not None:\n result.residual = result.residual.dot(inv(B))\n return result, params", "response": "Fit the model to the data and return the result."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef covar_errors(params, data, errs, B, C=None):\n\n mask = np.where(np.isfinite(data))\n\n # calculate the proper parameter errors and copy them across.\n if C is not None:\n try:\n J = lmfit_jacobian(params, mask[0], mask[1], errs=errs)\n covar = np.transpose(J).dot(inv(C)).dot(J)\n onesigma = np.sqrt(np.diag(inv(covar)))\n except (np.linalg.linalg.LinAlgError, ValueError) as _:\n C = None\n\n if C is None:\n try:\n J = lmfit_jacobian(params, mask[0], mask[1], B=B, errs=errs)\n covar = np.transpose(J).dot(J)\n onesigma = np.sqrt(np.diag(inv(covar)))\n except (np.linalg.linalg.LinAlgError, ValueError) as _:\n onesigma = [-2] * len(mask[0])\n\n for i in range(params['components'].value):\n prefix = \"c{0}_\".format(i)\n j = 0\n for p in ['amp', 'xo', 'yo', 'sx', 'sy', 'theta']:\n if params[prefix + p].vary:\n params[prefix + p].stderr = onesigma[j]\n j += 1\n\n return params", "response": "This function calculates the errors of the parameters that were fit with lmfit and replaces the errorsof the parameters with the 1 \\ sigma errors calculated using the covariance matrix."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef barrier(events, sid, kind='neighbour'):\n events[sid].set()\n # only wait for the neighbours\n if kind=='neighbour':\n if sid > 0:\n logging.debug(\"{0} is waiting for {1}\".format(sid, sid - 1))\n events[sid - 1].wait()\n if sid < len(bkg_events) - 1:\n logging.debug(\"{0} is waiting for {1}\".format(sid, sid + 1))\n events[sid + 1].wait()\n # wait for all\n else:\n [e.wait() for e in events]\n return", "response": "A multiprocessing barrier for a single event."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sigmaclip(arr, lo, hi, reps=3):\n clipped = np.array(arr)[np.isfinite(arr)]\n\n if len(clipped) < 1:\n return np.nan, np.nan\n\n std = np.std(clipped)\n mean = np.mean(clipped)\n for _ in range(int(reps)):\n clipped = clipped[np.where(clipped > mean-std*lo)]\n clipped = clipped[np.where(clipped < mean+std*hi)]\n pstd = std\n if len(clipped) < 1:\n break\n std = np.std(clipped)\n mean = np.mean(clipped)\n if 2*abs(pstd-std)/(pstd+std) < 0.2:\n break\n return mean, std", "response": "Perform sigma clipping on an array ignoring non finite values."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sigma_filter(filename, region, step_size, box_size, shape, domask, sid):\n\n ymin, ymax = region\n logging.debug('rows {0}-{1} starting at {2}'.format(ymin, ymax, strftime(\"%Y-%m-%d %H:%M:%S\", gmtime())))\n\n # cut out the region of interest plus 1/2 the box size, but clip to the image size\n data_row_min = max(0, ymin - box_size[0]//2)\n data_row_max = min(shape[0], ymax + box_size[0]//2)\n\n # Figure out how many axes are in the datafile\n NAXIS = fits.getheader(filename)[\"NAXIS\"]\n\n with fits.open(filename, memmap=True) as a:\n if NAXIS == 2:\n data = a[0].section[data_row_min:data_row_max, 0:shape[1]]\n elif NAXIS == 3:\n data = a[0].section[0, data_row_min:data_row_max, 0:shape[1]]\n elif NAXIS == 4:\n data = a[0].section[0, 0, data_row_min:data_row_max, 0:shape[1]]\n else:\n logging.error(\"Too many NAXIS for me {0}\".format(NAXIS))\n logging.error(\"fix your file to be more sane\")\n raise Exception(\"Too many NAXIS\")\n\n row_len = shape[1]\n\n logging.debug('data size is {0}'.format(data.shape))\n\n def box(r, c):\n \"\"\"\n calculate the boundaries of the box centered at r,c\n with size = box_size\n \"\"\"\n r_min = max(0, r - box_size[0] // 2)\n r_max = min(data.shape[0] - 1, r + box_size[0] // 2)\n c_min = max(0, c - box_size[1] // 2)\n c_max = min(data.shape[1] - 1, c + box_size[1] // 2)\n return r_min, r_max, c_min, c_max\n\n # set up a grid of rows/cols at which we will compute the bkg/rms\n rows = list(range(ymin-data_row_min, ymax-data_row_min, step_size[0]))\n rows.append(ymax-data_row_min)\n cols = list(range(0, shape[1], step_size[1]))\n cols.append(shape[1])\n\n # store the computed bkg/rms in this smaller array\n vals = np.zeros(shape=(len(rows),len(cols)))\n\n for i, row in enumerate(rows):\n for j, col in enumerate(cols):\n r_min, r_max, c_min, c_max = box(row, col)\n new = data[r_min:r_max, c_min:c_max]\n new = np.ravel(new)\n bkg, _ = sigmaclip(new, 3, 3)\n vals[i,j] = bkg\n\n # indices of all the pixels within our region\n gr, gc = np.mgrid[ymin-data_row_min:ymax-data_row_min, 0:shape[1]]\n\n logging.debug(\"Interpolating bkg to sharemem\")\n ifunc = RegularGridInterpolator((rows, cols), vals)\n for i in range(gr.shape[0]):\n row = np.array(ifunc((gr[i], gc[i])), dtype=np.float32)\n start_idx = np.ravel_multi_index((ymin+i, 0), shape)\n end_idx = start_idx + row_len\n ibkg[start_idx:end_idx] = row # np.ctypeslib.as_ctypes(row)\n del ifunc\n logging.debug(\" ... done writing bkg\")\n\n # signal that the bkg is done for this region, and wait for neighbours\n barrier(bkg_events, sid)\n\n logging.debug(\"{0} background subtraction\".format(sid))\n for i in range(data_row_max - data_row_min):\n start_idx = np.ravel_multi_index((data_row_min + i, 0), shape)\n end_idx = start_idx + row_len\n data[i, :] = data[i, :] - ibkg[start_idx:end_idx]\n # reset/recycle the vals array\n vals[:] = 0\n\n for i, row in enumerate(rows):\n for j, col in enumerate(cols):\n r_min, r_max, c_min, c_max = box(row, col)\n new = data[r_min:r_max, c_min:c_max]\n new = np.ravel(new)\n _ , rms = sigmaclip(new, 3, 3)\n vals[i,j] = rms\n\n logging.debug(\"Interpolating rm to sharemem rms\")\n ifunc = RegularGridInterpolator((rows, cols), vals)\n for i in range(gr.shape[0]):\n row = np.array(ifunc((gr[i], gc[i])), dtype=np.float32)\n start_idx = np.ravel_multi_index((ymin+i, 0), shape)\n end_idx = start_idx + row_len\n irms[start_idx:end_idx] = row # np.ctypeslib.as_ctypes(row)\n del ifunc\n logging.debug(\" .. done writing rms\")\n\n if domask:\n barrier(mask_events, sid)\n logging.debug(\"applying mask\")\n for i in range(gr.shape[0]):\n mask = np.where(np.bitwise_not(np.isfinite(data[i + ymin-data_row_min,:])))[0]\n for j in mask:\n idx = np.ravel_multi_index((i + ymin,j),shape)\n ibkg[idx] = np.nan\n irms[idx] = np.nan\n logging.debug(\" ... done applying mask\")\n logging.debug('rows {0}-{1} finished at {2}'.format(ymin, ymax, strftime(\"%Y-%m-%d %H:%M:%S\", gmtime())))\n return", "response": "Calculate the background and rms for a sub region of an image."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the background and noise images corresponding to the input file. The calculation is done via a box-car approach and uses multiple cores and shared memory. Parameters ---------- filename : str Filename to be filtered. step_size : (int, int) Step size for the filter. box_size : (int, int) Box size for the filter. cores : int Number of cores to use. If None then use all available. nslice : int The image will be divided into this many horizontal stripes for processing. Default = None = equal to cores shape : (int, int) The shape of the image in the given file. domask : bool True(Default) = copy data mask to output. Returns ------- bkg, rms : numpy.ndarray The interpolated background and noise images.", "response": "def filter_mc_sharemem(filename, step_size, box_size, cores, shape, nslice=None, domask=True):\n \"\"\"\n Calculate the background and noise images corresponding to the input file.\n The calculation is done via a box-car approach and uses multiple cores and shared memory.\n\n Parameters\n ----------\n filename : str\n Filename to be filtered.\n\n step_size : (int, int)\n Step size for the filter.\n\n box_size : (int, int)\n Box size for the filter.\n\n cores : int\n Number of cores to use. If None then use all available.\n\n nslice : int\n The image will be divided into this many horizontal stripes for processing.\n Default = None = equal to cores\n\n shape : (int, int)\n The shape of the image in the given file.\n\n domask : bool\n True(Default) = copy data mask to output.\n\n Returns\n -------\n bkg, rms : numpy.ndarray\n The interpolated background and noise images.\n \"\"\"\n\n if cores is None:\n cores = multiprocessing.cpu_count()\n if (nslice is None) or (cores==1):\n nslice = cores\n\n img_y, img_x = shape\n # initialise some shared memory\n global ibkg\n # bkg = np.ctypeslib.as_ctypes(np.empty(shape, dtype=np.float32))\n # ibkg = multiprocessing.sharedctypes.Array(bkg._type_, bkg, lock=True)\n ibkg = multiprocessing.Array('f', img_y*img_x)\n\n global irms\n #rms = np.ctypeslib.as_ctypes(np.empty(shape, dtype=np.float32))\n #irms = multiprocessing.sharedctypes.Array(rms._type_, rms, lock=True)\n irms = multiprocessing.Array('f', img_y * img_x)\n\n logging.info(\"using {0} cores\".format(cores))\n logging.info(\"using {0} stripes\".format(nslice))\n\n if nslice > 1:\n # box widths should be multiples of the step_size, and not zero\n width_y = int(max(img_y/nslice/step_size[1], 1) * step_size[1])\n\n # locations of the box edges\n ymins = list(range(0, img_y, width_y))\n ymaxs = list(range(width_y, img_y, width_y))\n ymaxs.append(img_y)\n else:\n ymins = [0]\n ymaxs = [img_y]\n\n logging.debug(\"ymins {0}\".format(ymins))\n logging.debug(\"ymaxs {0}\".format(ymaxs))\n\n # create an event per stripe\n global bkg_events, mask_events\n bkg_events = [multiprocessing.Event() for _ in range(len(ymaxs))]\n mask_events = [multiprocessing.Event() for _ in range(len(ymaxs))]\n\n args = []\n for i, region in enumerate(zip(ymins, ymaxs)):\n args.append((filename, region, step_size, box_size, shape, domask, i))\n\n # start a new process for each task, hopefully to reduce residual memory use\n pool = multiprocessing.Pool(processes=cores, maxtasksperchild=1)\n try:\n # chunksize=1 ensures that we only send a single task to each process\n pool.map_async(_sf2, args, chunksize=1).get(timeout=10000000)\n except KeyboardInterrupt:\n logging.error(\"Caught keyboard interrupt\")\n pool.close()\n sys.exit(1)\n pool.close()\n pool.join()\n bkg = np.reshape(np.array(ibkg[:], dtype=np.float32), shape)\n rms = np.reshape(np.array(irms[:], dtype=np.float32), shape)\n del ibkg, irms\n return bkg, rms"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfilter an image into a single image.", "response": "def filter_image(im_name, out_base, step_size=None, box_size=None, twopass=False, cores=None, mask=True, compressed=False, nslice=None):\n \"\"\"\n Create a background and noise image from an input image.\n Resulting images are written to `outbase_bkg.fits` and `outbase_rms.fits`\n\n Parameters\n ----------\n im_name : str or HDUList\n Image to filter. Either a string filename or an astropy.io.fits.HDUList.\n out_base : str\n The output filename base. Will be modified to make _bkg and _rms files.\n step_size : (int,int)\n Tuple of the x,y step size in pixels\n box_size : (int,int)\n The size of the box in piexls\n twopass : bool\n Perform a second pass calculation to ensure that the noise is not contaminated by the background.\n Default = False\n cores : int\n Number of CPU corse to use.\n Default = all available\n nslice : int\n The image will be divided into this many horizontal stripes for processing.\n Default = None = equal to cores\n mask : bool\n Mask the output array to contain np.nna wherever the input array is nan or not finite.\n Default = true\n compressed : bool\n Return a compressed version of the background/noise images.\n Default = False\n\n Returns\n -------\n None\n\n \"\"\"\n\n header = fits.getheader(im_name)\n shape = (header['NAXIS2'],header['NAXIS1'])\n\n if step_size is None:\n if 'BMAJ' in header and 'BMIN' in header:\n beam_size = np.sqrt(abs(header['BMAJ']*header['BMIN']))\n if 'CDELT1' in header:\n pix_scale = np.sqrt(abs(header['CDELT1']*header['CDELT2']))\n elif 'CD1_1' in header:\n pix_scale = np.sqrt(abs(header['CD1_1']*header['CD2_2']))\n if 'CD1_2' in header and 'CD2_1' in header:\n if header['CD1_2'] != 0 or header['CD2_1']!=0:\n logging.warning(\"CD1_2 and/or CD2_1 are non-zero and I don't know what to do with them\")\n logging.warning(\"Ingoring them\")\n else:\n logging.warning(\"Cannot determine pixel scale, assuming 4 pixels per beam\")\n pix_scale = beam_size/4.\n # default to 4x the synthesized beam width\n step_size = int(np.ceil(4*beam_size/pix_scale))\n else:\n logging.info(\"BMAJ and/or BMIN not in fits header.\")\n logging.info(\"Assuming 4 pix/beam, so we have step_size = 16 pixels\")\n step_size = 16\n step_size = (step_size, step_size)\n\n if box_size is None:\n # default to 6x the step size so we have ~ 30beams\n box_size = (step_size[0]*6, step_size[1]*6)\n\n if compressed:\n if not step_size[0] == step_size[1]:\n step_size = (min(step_size), min(step_size))\n logging.info(\"Changing grid to be {0} so we can compress the output\".format(step_size))\n\n logging.info(\"using grid_size {0}, box_size {1}\".format(step_size,box_size))\n logging.info(\"on data shape {0}\".format(shape))\n bkg, rms = filter_mc_sharemem(im_name, step_size=step_size, box_size=box_size, cores=cores, shape=shape, nslice=nslice, domask=mask)\n logging.info(\"done\")\n\n bkg_out = '_'.join([os.path.expanduser(out_base), 'bkg.fits'])\n rms_out = '_'.join([os.path.expanduser(out_base), 'rms.fits'])\n\n\n # add a comment to the fits header\n header['HISTORY'] = 'BANE {0}-({1})'.format(__version__, __date__)\n\n # compress\n if compressed:\n hdu = fits.PrimaryHDU(bkg)\n hdu.header = copy.deepcopy(header)\n hdulist = fits.HDUList([hdu])\n compress(hdulist, step_size[0], bkg_out)\n hdulist[0].header = copy.deepcopy(header)\n hdulist[0].data = rms\n compress(hdulist, step_size[0], rms_out)\n return\n\n write_fits(bkg, header, bkg_out)\n write_fits(rms, header, rms_out)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncombining data and a fits header to write a fits file.", "response": "def write_fits(data, header, file_name):\n \"\"\"\n Combine data and a fits header to write a fits file.\n\n Parameters\n ----------\n data : numpy.ndarray\n The data to be written.\n\n header : astropy.io.fits.hduheader\n The header for the fits file.\n\n file_name : string\n The file to write\n\n Returns\n -------\n None\n \"\"\"\n hdu = fits.PrimaryHDU(data)\n hdu.header = header\n hdulist = fits.HDUList([hdu])\n hdulist.writeto(file_name, overwrite=True)\n logging.info(\"Wrote {0}\".format(file_name))\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting sexegessimal RA string into a float in degrees.", "response": "def dec2dec(dec):\n \"\"\"\n Convert sexegessimal RA string into a float in degrees.\n\n Parameters\n ----------\n dec : string\n A string separated representing the Dec.\n Expected format is `[+- ]hh:mm[:ss.s]`\n Colons can be replaced with any whit space character.\n\n Returns\n -------\n dec : float\n The Dec in degrees.\n \"\"\"\n d = dec.replace(':', ' ').split()\n if len(d) == 2:\n d.append(0.0)\n if d[0].startswith('-') or float(d[0]) < 0:\n return float(d[0]) - float(d[1]) / 60.0 - float(d[2]) / 3600.0\n return float(d[0]) + float(d[1]) / 60.0 + float(d[2]) / 3600.0"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dec2dms(x):\n if not np.isfinite(x):\n return 'XX:XX:XX.XX'\n if x < 0:\n sign = '-'\n else:\n sign = '+'\n x = abs(x)\n d = int(math.floor(x))\n m = int(math.floor((x - d) * 60))\n s = float(( (x - d) * 60 - m) * 60)\n return '{0}{1:02d}:{2:02d}:{3:05.2f}'.format(sign, d, m, s)", "response": "Convert decimal degrees into sexagessimal string in degrees."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dec2hms(x):\n if not np.isfinite(x):\n return 'XX:XX:XX.XX'\n # wrap negative RA's\n if x < 0:\n x += 360\n x /= 15.0\n h = int(x)\n x = (x - h) * 60\n m = int(x)\n s = (x - m) * 60\n return '{0:02d}:{1:02d}:{2:05.2f}'.format(h, m, s)", "response": "Convert decimal degrees into sexagessimal string in hours."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the great circle distance between two points using the haversine formula.", "response": "def gcd(ra1, dec1, ra2, dec2):\n \"\"\"\n Calculate the great circle distance between to points using the haversine formula [1]_.\n\n\n Parameters\n ----------\n ra1, dec1, ra2, dec2 : float\n The coordinates of the two points of interest.\n Units are in degrees.\n\n Returns\n -------\n dist : float\n The distance between the two points in degrees.\n\n Notes\n -----\n This duplicates the functionality of astropy but is faster as there is no creation of SkyCoords objects.\n\n .. [1] `Haversine formula `_\n \"\"\"\n # TODO: Vincenty formula see - https://en.wikipedia.org/wiki/Great-circle_distance\n dlon = ra2 - ra1\n dlat = dec2 - dec1\n a = np.sin(np.radians(dlat) / 2) ** 2\n a += np.cos(np.radians(dec1)) * np.cos(np.radians(dec2)) * np.sin(np.radians(dlon) / 2) ** 2\n sep = np.degrees(2 * np.arcsin(np.minimum(1, np.sqrt(a))))\n return sep"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the bearing of point 2 from point 1 along a great circle.", "response": "def bear(ra1, dec1, ra2, dec2):\n \"\"\"\n Calculate the bearing of point 2 from point 1 along a great circle.\n The bearing is East of North and is in [0, 360), whereas position angle is also East of North but (-180,180]\n\n Parameters\n ----------\n ra1, dec1, ra2, dec2 : float\n The sky coordinates (degrees) of the two points.\n\n Returns\n -------\n bear : float\n The bearing of point 2 from point 1 (degrees).\n \"\"\"\n rdec1 = np.radians(dec1)\n rdec2 = np.radians(dec2)\n rdlon = np.radians(ra2-ra1)\n y = np.sin(rdlon) * np.cos(rdec2)\n x = np.cos(rdec1) * np.sin(rdec2)\n x -= np.sin(rdec1) * np.cos(rdec2) * np.cos(rdlon)\n return np.degrees(np.arctan2(y, x))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef translate(ra, dec, r, theta):\n factor = np.sin(np.radians(dec)) * np.cos(np.radians(r))\n factor += np.cos(np.radians(dec)) * np.sin(np.radians(r)) * np.cos(np.radians(theta))\n dec_out = np.degrees(np.arcsin(factor))\n\n y = np.sin(np.radians(theta)) * np.sin(np.radians(r)) * np.cos(np.radians(dec))\n x = np.cos(np.radians(r)) - np.sin(np.radians(dec)) * np.sin(np.radians(dec_out))\n ra_out = ra + np.degrees(np.arctan2(y, x))\n return ra_out, dec_out", "response": "Translate a given point a distance r in the initial direction theta along a great circle."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the distance between two points on the Rhumb line.", "response": "def dist_rhumb(ra1, dec1, ra2, dec2):\n \"\"\"\n Calculate the Rhumb line distance between two points [1]_.\n A Rhumb line between two points is one which follows a constant bearing.\n\n Parameters\n ----------\n ra1, dec1, ra2, dec2 : float\n The position of the two points (degrees).\n\n Returns\n -------\n dist : float\n The distance between the two points along a line of constant bearing.\n\n Notes\n -----\n .. [1] `Rhumb line `_\n \"\"\"\n # verified against website to give correct results\n phi1 = np.radians(dec1)\n phi2 = np.radians(dec2)\n dphi = phi2 - phi1\n lambda1 = np.radians(ra1)\n lambda2 = np.radians(ra2)\n dpsi = np.log(np.tan(np.pi / 4 + phi2 / 2) / np.tan(np.pi / 4 + phi1 / 2))\n if dpsi < 1e-12:\n q = np.cos(phi1)\n else:\n q = dpsi / dphi\n dlambda = lambda2 - lambda1\n if dlambda > np.pi:\n dlambda -= 2 * np.pi\n dist = np.hypot(dphi, q * dlambda)\n return np.degrees(dist)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bear_rhumb(ra1, dec1, ra2, dec2):\n # verified against website to give correct results\n phi1 = np.radians(dec1)\n phi2 = np.radians(dec2)\n lambda1 = np.radians(ra1)\n lambda2 = np.radians(ra2)\n dlambda = lambda2 - lambda1\n\n dpsi = np.log(np.tan(np.pi / 4 + phi2 / 2) / np.tan(np.pi / 4 + phi1 / 2))\n\n theta = np.arctan2(dlambda, dpsi)\n return np.degrees(theta)", "response": "Calculate the bearing of point 2 from point 1 along a Rhumb line."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef translate_rhumb(ra, dec, r, theta):\n # verified against website to give correct results\n # with the help of http://williams.best.vwh.net/avform.htm#Rhumb\n delta = np.radians(r)\n phi1 = np.radians(dec)\n phi2 = phi1 + delta * np.cos(np.radians(theta))\n dphi = phi2 - phi1\n\n if abs(dphi) < 1e-9:\n q = np.cos(phi1)\n else:\n dpsi = np.log(np.tan(np.pi / 4 + phi2 / 2) / np.tan(np.pi / 4 + phi1 / 2))\n q = dphi / dpsi\n\n lambda1 = np.radians(ra)\n dlambda = delta * np.sin(np.radians(theta)) / q\n lambda2 = lambda1 + dlambda\n\n ra_out = np.degrees(lambda2)\n dec_out = np.degrees(phi2)\n return ra_out, dec_out", "response": "Translate a given point a distance r in the initial direction theta along a Rhumb line."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a Galactic l and b to fk5 ra and dec", "response": "def galactic2fk5(l, b):\n \"\"\"\n Convert galactic l/b to fk5 ra/dec\n\n Parameters\n ----------\n l, b : float\n Galactic coordinates in radians.\n\n Returns\n -------\n ra, dec : float\n FK5 ecliptic coordinates in radians.\n \"\"\"\n a = SkyCoord(l, b, unit=(u.radian, u.radian), frame='galactic')\n return a.fk5.ra.radian, a.fk5.dec.radian"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmasking a 2d image array such that pixels within region are set to nan.", "response": "def mask_plane(data, wcs, region, negate=False):\n \"\"\"\n Mask a 2d image (data) such that pixels within 'region' are set to nan.\n\n Parameters\n ----------\n data : 2d-array\n Image array.\n\n wcs : astropy.wcs.WCS\n WCS for the image in question.\n\n region : :class:`AegeanTools.regions.Region`\n A region within which the image pixels will be masked.\n\n negate : bool\n If True then pixels *outside* the region are masked.\n Default = False.\n\n Returns\n -------\n masked : 2d-array\n The original array, but masked as required.\n \"\"\"\n # create an array but don't set the values (they are random)\n indexes = np.empty((data.shape[0]*data.shape[1], 2), dtype=int)\n # since I know exactly what the index array needs to look like i can construct\n # it faster than list comprehension would allow\n # we do this only once and then recycle it\n idx = np.array([(j, 0) for j in range(data.shape[1])])\n j = data.shape[1]\n for i in range(data.shape[0]):\n idx[:, 1] = i\n indexes[i*j:(i+1)*j] = idx\n\n # put ALL the pixles into our vectorized functions and minimise our overheads\n ra, dec = wcs.wcs_pix2world(indexes, 1).transpose()\n bigmask = region.sky_within(ra, dec, degin=True)\n if not negate:\n bigmask = np.bitwise_not(bigmask)\n # rework our 1d list into a 2d array\n bigmask = bigmask.reshape(data.shape)\n # and apply the mask\n data[bigmask] = np.nan\n return data"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mask_file(regionfile, infile, outfile, negate=False):\n # Check that the input file is accessible and then open it\n if not os.path.exists(infile): raise AssertionError(\"Cannot locate fits file {0}\".format(infile))\n im = pyfits.open(infile)\n if not os.path.exists(regionfile): raise AssertionError(\"Cannot locate region file {0}\".format(regionfile))\n region = Region.load(regionfile)\n try:\n wcs = pywcs.WCS(im[0].header, naxis=2)\n except: # TODO: figure out what error is being thrown\n wcs = pywcs.WCS(str(im[0].header), naxis=2)\n\n if len(im[0].data.shape) > 2:\n data = np.squeeze(im[0].data)\n else:\n data = im[0].data\n\n print(data.shape)\n if len(data.shape) == 3:\n for plane in range(data.shape[0]):\n mask_plane(data[plane], wcs, region, negate)\n else:\n mask_plane(data, wcs, region, negate)\n im[0].data = data\n im.writeto(outfile, overwrite=True)\n logging.info(\"Wrote {0}\".format(outfile))\n return", "response": "Masks a single image in a single region."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mask_table(region, table, negate=False, racol='ra', deccol='dec'):\n inside = region.sky_within(table[racol], table[deccol], degin=True)\n if not negate:\n mask = np.bitwise_not(inside)\n else:\n mask = inside\n return table[mask]", "response": "Masks a given table by removing all the rows with ra or dec inside the given region."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef mask_catalog(regionfile, infile, outfile, negate=False, racol='ra', deccol='dec'):\n logging.info(\"Loading region from {0}\".format(regionfile))\n region = Region.load(regionfile)\n logging.info(\"Loading catalog from {0}\".format(infile))\n table = load_table(infile)\n masked_table = mask_table(region, table, negate=negate, racol=racol, deccol=deccol)\n write_table(masked_table, outfile)\n return", "response": "Apply a region file as a mask to a catalog."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef mim2reg(mimfile, regfile):\n region = Region.load(mimfile)\n region.write_reg(regfile)\n logging.info(\"Converted {0} -> {1}\".format(mimfile, regfile))\n return", "response": "Convert a MIMAS region into a DS9 region."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a MIMAS region into a FITS region.", "response": "def mim2fits(mimfile, fitsfile):\n \"\"\"\n Convert a MIMAS region (.mim) file into a MOC region (.fits) file.\n\n Parameters\n ----------\n mimfile : str\n Input file in MIMAS format.\n\n fitsfile : str\n Output file.\n \"\"\"\n region = Region.load(mimfile)\n region.write_fits(fitsfile, moctool='MIMAS {0}-{1}'.format(__version__, __date__))\n logging.info(\"Converted {0} -> {1}\".format(mimfile, fitsfile))\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a string that describes a box in ds9 format into a polygon", "response": "def box2poly(line):\n \"\"\"\n Convert a string that describes a box in ds9 format, into a polygon that is given by the corners of the box\n\n Parameters\n ----------\n line : str\n A string containing a DS9 region command for a box.\n\n Returns\n -------\n poly : [ra, dec, ...]\n The corners of the box in clockwise order from top left.\n \"\"\"\n words = re.split('[(\\s,)]', line)\n ra = words[1]\n dec = words[2]\n width = words[3]\n height = words[4]\n if \":\" in ra:\n ra = Angle(ra, unit=u.hour)\n else:\n ra = Angle(ra, unit=u.degree)\n dec = Angle(dec, unit=u.degree)\n width = Angle(float(width[:-1])/2, unit=u.arcsecond) # strip the \"\n height = Angle(float(height[:-1])/2, unit=u.arcsecond) # strip the \"\n center = SkyCoord(ra, dec)\n tl = center.ra.degree+width.degree, center.dec.degree+height.degree\n tr = center.ra.degree-width.degree, center.dec.degree+height.degree\n bl = center.ra.degree+width.degree, center.dec.degree-height.degree\n br = center.ra.degree-width.degree, center.dec.degree-height.degree\n return np.ravel([tl, tr, br, bl]).tolist()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses a string that describes a circle in ds9 format.", "response": "def circle2circle(line):\n \"\"\"\n Parse a string that describes a circle in ds9 format.\n\n Parameters\n ----------\n line : str\n A string containing a DS9 region command for a circle.\n\n Returns\n -------\n circle : [ra, dec, radius]\n The center and radius of the circle.\n \"\"\"\n words = re.split('[(,\\s)]', line)\n ra = words[1]\n dec = words[2]\n radius = words[3][:-1] # strip the \"\n if \":\" in ra:\n ra = Angle(ra, unit=u.hour)\n else:\n ra = Angle(ra, unit=u.degree)\n dec = Angle(dec, unit=u.degree)\n radius = Angle(radius, unit=u.arcsecond)\n return [ra.degree, dec.degree, radius.degree]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses a string containing a DS9 region command for a polygon and return a list of coordinates.", "response": "def poly2poly(line):\n \"\"\"\n Parse a string of text containing a DS9 description of a polygon.\n\n This function works but is not very robust due to the constraints of healpy.\n\n Parameters\n ----------\n line : str\n A string containing a DS9 region command for a polygon.\n\n Returns\n -------\n poly : [ra, dec, ...]\n The coordinates of the polygon.\n \"\"\"\n words = re.split('[(\\s,)]', line)\n ras = np.array(words[1::2])\n decs = np.array(words[2::2])\n coords = []\n for ra, dec in zip(ras, decs):\n if ra.strip() == '' or dec.strip() == '':\n continue\n if \":\" in ra:\n pos = SkyCoord(Angle(ra, unit=u.hour), Angle(dec, unit=u.degree))\n else:\n pos = SkyCoord(Angle(ra, unit=u.degree), Angle(dec, unit=u.degree))\n # only add this point if it is some distance from the previous one\n coords.extend([pos.ra.degree, pos.dec.degree])\n return coords"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reg2mim(regfile, mimfile, maxdepth):\n logging.info(\"Reading regions from {0}\".format(regfile))\n lines = (l for l in open(regfile, 'r') if not l.startswith('#'))\n poly = []\n circles = []\n for line in lines:\n if line.startswith('box'):\n poly.append(box2poly(line))\n elif line.startswith('circle'):\n circles.append(circle2circle(line))\n elif line.startswith('polygon'):\n logging.warning(\"Polygons break a lot, but I'll try this one anyway.\")\n poly.append(poly2poly(line))\n else:\n logging.warning(\"Not sure what to do with {0}\".format(line[:-1]))\n container = Dummy(maxdepth=maxdepth)\n container.include_circles = circles\n container.include_polygons = poly\n\n region = combine_regions(container)\n save_region(region, mimfile)\n return", "response": "Parse a DS9 region file and write a MIMAS region file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef combine_regions(container):\n # create empty region\n region = Region(container.maxdepth)\n\n # add/rem all the regions from files\n for r in container.add_region:\n logging.info(\"adding region from {0}\".format(r))\n r2 = Region.load(r[0])\n region.union(r2)\n\n for r in container.rem_region:\n logging.info(\"removing region from {0}\".format(r))\n r2 = Region.load(r[0])\n region.without(r2)\n\n\n # add circles\n if len(container.include_circles) > 0:\n for c in container.include_circles:\n circles = np.radians(np.array(c))\n if container.galactic:\n l, b, radii = circles.reshape(3, circles.shape[0]//3)\n ras, decs = galactic2fk5(l, b)\n else:\n ras, decs, radii = circles.reshape(3, circles.shape[0]//3)\n region.add_circles(ras, decs, radii)\n\n # remove circles\n if len(container.exclude_circles) > 0:\n for c in container.exclude_circles:\n r2 = Region(container.maxdepth)\n circles = np.radians(np.array(c))\n if container.galactic:\n l, b, radii = circles.reshape(3, circles.shape[0]//3)\n ras, decs = galactic2fk5(l, b)\n else:\n ras, decs, radii = circles.reshape(3, circles.shape[0]//3)\n r2.add_circles(ras, decs, radii)\n region.without(r2)\n\n # add polygons\n if len(container.include_polygons) > 0:\n for p in container.include_polygons:\n poly = np.radians(np.array(p))\n poly = poly.reshape((poly.shape[0]//2, 2))\n region.add_poly(poly)\n\n # remove polygons\n if len(container.exclude_polygons) > 0:\n for p in container.include_polygons:\n poly = np.array(np.radians(p))\n r2 = Region(container.maxdepth)\n r2.add_poly(poly)\n region.without(r2)\n\n return region", "response": "Returns a new region that is the combination of those specified in the container."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef intersect_regions(flist):\n if len(flist) < 2:\n raise Exception(\"Require at least two regions to perform intersection\")\n a = Region.load(flist[0])\n for b in [Region.load(f) for f in flist[1:]]:\n a.intersect(b)\n return a", "response": "Returns a region which is the intersection of all regions described in the given list of file names."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsaves the given region to a file.", "response": "def save_region(region, filename):\n \"\"\"\n Save the given region to a file\n\n Parameters\n ----------\n region : :class:`AegeanTools.regions.Region`\n A region.\n\n filename : str\n Output file name.\n \"\"\"\n region.save(filename)\n logging.info(\"Wrote {0}\".format(filename))\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a MIMAS region into an image.", "response": "def save_as_image(region, filename):\n \"\"\"\n Convert a MIMAS region (.mim) file into a image (eg .png)\n\n Parameters\n ----------\n region : :class:`AegeanTools.regions.Region`\n Region of interest.\n\n filename : str\n Output filename.\n \"\"\"\n import healpy as hp\n pixels = list(region.get_demoted())\n order = region.maxdepth\n m = np.arange(hp.nside2npix(2**order))\n m[:] = 0\n m[pixels] = 1\n hp.write_map(filename, m, nest=True, coord='C')\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning some pixel information based on the given header", "response": "def get_pixinfo(header):\n \"\"\"\n Return some pixel information based on the given hdu header\n pixarea - the area of a single pixel in deg2\n pixscale - the side lengths of a pixel (assuming they are square)\n\n Parameters\n ----------\n header : HDUHeader or dict\n FITS header information\n\n Returns\n -------\n pixarea : float\n The are of a single pixel at the reference location, in square degrees.\n\n pixscale : (float, float)\n The pixel scale in degrees, at the reference location.\n\n Notes\n -----\n The reference location is not always at the image center, and the pixel scale/area may\n change over the image, depending on the projection.\n \"\"\"\n if all(a in header for a in [\"CDELT1\", \"CDELT2\"]):\n pixarea = abs(header[\"CDELT1\"]*header[\"CDELT2\"])\n pixscale = (header[\"CDELT1\"], header[\"CDELT2\"])\n elif all(a in header for a in [\"CD1_1\", \"CD1_2\", \"CD2_1\", \"CD2_2\"]):\n pixarea = abs(header[\"CD1_1\"]*header[\"CD2_2\"]\n - header[\"CD1_2\"]*header[\"CD2_1\"])\n pixscale = (header[\"CD1_1\"], header[\"CD2_2\"])\n if not (header[\"CD1_2\"] == 0 and header[\"CD2_1\"] == 0):\n log.warning(\"Pixels don't appear to be square -> pixscale is wrong\")\n elif all(a in header for a in [\"CD1_1\", \"CD2_2\"]):\n pixarea = abs(header[\"CD1_1\"]*header[\"CD2_2\"])\n pixscale = (header[\"CD1_1\"], header[\"CD2_2\"])\n else:\n log.critical(\"cannot determine pixel area, using zero EVEN THOUGH THIS IS WRONG!\")\n pixarea = 0\n pixscale = (0, 0)\n return pixarea, pixscale"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a beam object from a header.", "response": "def get_beam(header):\n \"\"\"\n Create a :class:`AegeanTools.fits_image.Beam` object from a fits header.\n\n BPA may be missing but will be assumed to be zero.\n\n if BMAJ or BMIN are missing then return None instead of a beam object.\n\n Parameters\n ----------\n header : HDUHeader\n The fits header.\n\n Returns\n -------\n beam : :class:`AegeanTools.fits_image.Beam`\n Beam object, with a, b, and pa in degrees.\n \"\"\"\n\n if \"BPA\" not in header:\n log.warning(\"BPA not present in fits header, using 0\")\n bpa = 0\n else:\n bpa = header[\"BPA\"]\n\n if \"BMAJ\" not in header:\n log.warning(\"BMAJ not present in fits header.\")\n bmaj = None\n else:\n bmaj = header[\"BMAJ\"]\n\n if \"BMIN\" not in header:\n log.warning(\"BMIN not present in fits header.\")\n bmin = None\n else:\n bmin = header[\"BMIN\"]\n if None in [bmaj, bmin, bpa]:\n return None\n beam = Beam(bmaj, bmin, bpa)\n return beam"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfix the AIPS header.", "response": "def fix_aips_header(header):\n \"\"\"\n Search through an image header. If the keywords BMAJ/BMIN/BPA are not set,\n but there are AIPS history cards, then we can populate the BMAJ/BMIN/BPA.\n Fix the header if possible, otherwise don't. Either way, don't complain.\n\n\n Parameters\n ----------\n header : HDUHeader\n Fits header which may or may not have AIPS history cards.\n\n Returns\n -------\n header : HDUHeader\n A header which has BMAJ, BMIN, and BPA keys, as well as a new HISTORY card.\n \"\"\"\n if 'BMAJ' in header and 'BMIN' in header and 'BPA' in header:\n # The header already has the required keys so there is nothing to do\n return header\n aips_hist = [a for a in header['HISTORY'] if a.startswith(\"AIPS\")]\n if len(aips_hist) == 0:\n # There are no AIPS history items to process\n return header\n for a in aips_hist:\n if \"BMAJ\" in a:\n # this line looks like\n # 'AIPS CLEAN BMAJ= 1.2500E-02 BMIN= 1.2500E-02 BPA= 0.00'\n words = a.split()\n bmaj = float(words[3])\n bmin = float(words[5])\n bpa = float(words[7])\n break\n else:\n # there are AIPS cards but there is no BMAJ/BMIN/BPA\n return header\n header['BMAJ'] = bmaj\n header['BMIN'] = bmin\n header['BPA'] = bpa\n header['HISTORY'] = 'Beam information AIPS->fits by AegeanTools'\n return header"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the image data.", "response": "def set_pixels(self, pixels):\n \"\"\"\n Set the image data.\n Will not work if the new image has a different shape than the current image.\n\n Parameters\n ----------\n pixels : numpy.ndarray\n New image data\n\n Returns\n -------\n None\n \"\"\"\n if not (pixels.shape == self._pixels.shape):\n raise AssertionError(\"Shape mismatch between pixels supplied {0} and existing image pixels {1}\".format(pixels.shape,self._pixels.shape))\n self._pixels = pixels\n # reset this so that it is calculated next time the function is called\n self._rms = None\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_background_rms(self):\n # TODO: return a proper background RMS ignoring the sources\n # This is an approximate method suggested by PaulH.\n # I have no idea where this magic 1.34896 number comes from...\n if self._rms is None:\n # Get the pixels values without the NaNs\n data = numpy.extract(self.hdu.data > -9999999, self.hdu.data)\n p25 = scipy.stats.scoreatpercentile(data, 25)\n p75 = scipy.stats.scoreatpercentile(data, 75)\n iqr = p75 - p25\n self._rms = iqr / 1.34896\n return self._rms", "response": "Calculate the rms of the background image."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pix2sky(self, pixel):\n pixbox = numpy.array([pixel, pixel])\n skybox = self.wcs.all_pix2world(pixbox, 1)\n return [float(skybox[0][0]), float(skybox[0][1])]", "response": "Get the sky coordinates for a given image pixel."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sky2pix(self, skypos):\n skybox = [skypos, skypos]\n pixbox = self.wcs.all_world2pix(skybox, 1)\n return [float(pixbox[0][0]), float(pixbox[0][1])]", "response": "Get the pixel coordinates for a given sky position."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_sources(filename):\n catalog = catalogs.table_to_source_list(catalogs.load_table(filename))\n logging.info(\"read {0} sources from {1}\".format(len(catalog), filename))\n return catalog", "response": "Load all the sources in a file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef search_beam(hdulist):\n header = hdulist[0].header\n history = header['HISTORY']\n history_str = str(history)\n #AIPS CLEAN BMAJ= 1.2500E-02 BMIN= 1.2500E-02 BPA= 0.00\n if 'BMAJ' in history_str:\n return True\n else:\n return False", "response": "Search the beam info from the HISTORY header"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fix_shape(source):\n if source.a < source.b:\n source.a, source.b = source.b, source.a\n source.err_a, source.err_b = source.err_b, source.err_a\n source.pa += 90\n return", "response": "Ensure that a > = b for a given source object."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the limit of theta in the base sequence.", "response": "def theta_limit(theta):\n \"\"\"\n Angle theta is periodic with period pi.\n Constrain theta such that -pi/2 np.pi / 2:\n theta -= np.pi\n return theta"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a telescope name into a latitude.", "response": "def scope2lat(telescope):\n \"\"\"\n Convert a telescope name into a latitude\n returns None when the telescope is unknown.\n\n Parameters\n ----------\n telescope : str\n Acronym (name) of telescope, eg MWA.\n\n Returns\n -------\n lat : float\n The latitude of the telescope.\n\n Notes\n -----\n These values were taken from wikipedia so have varying precision/accuracy\n \"\"\"\n scopes = {'MWA': -26.703319,\n \"ATCA\": -30.3128,\n \"VLA\": 34.0790,\n \"LOFAR\": 52.9088,\n \"KAT7\": -30.721,\n \"MEERKAT\": -30.721,\n \"PAPER\": -30.7224,\n \"GMRT\": 19.096516666667,\n \"OOTY\": 11.383404,\n \"ASKAP\": -26.7,\n \"MOST\": -35.3707,\n \"PARKES\": -32.999944,\n \"WSRT\": 52.914722,\n \"AMILA\": 52.16977,\n \"AMISA\": 52.164303,\n \"ATA\": 40.817,\n \"CHIME\": 49.321,\n \"CARMA\": 37.28044,\n \"DRAO\": 49.321,\n \"GBT\": 38.433056,\n \"LWA\": 34.07,\n \"ALMA\": -23.019283,\n \"FAST\": 25.6525\n }\n if telescope.upper() in scopes:\n return scopes[telescope.upper()]\n else:\n log = logging.getLogger(\"Aegean\")\n log.warn(\"Telescope {0} is unknown\".format(telescope))\n log.warn(\"integrated fluxes may be incorrect\")\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_cores(cores):\n cores = min(multiprocessing.cpu_count(), cores)\n if six.PY3:\n log = logging.getLogger(\"Aegean\")\n log.info(\"Multi-cores not supported in python 3+, using one core\")\n return 1\n try:\n queue = pprocess.Queue(limit=cores, reuse=1)\n except: # TODO: figure out what error is being thrown\n cores = 1\n else:\n try:\n _ = queue.manage(pprocess.MakeReusable(fix_shape))\n except:\n cores = 1\n return cores", "response": "Determine how many cores we can use."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_aux_files(basename):\n base = os.path.splitext(basename)[0]\n files = {\"bkg\": base + \"_bkg.fits\",\n \"rms\": base + \"_rms.fits\",\n \"mask\": base + \".mim\",\n \"cat\": base + \"_comp.fits\",\n \"psf\": base + \"_psf.fits\"}\n\n for k in files.keys():\n if not os.path.exists(files[k]):\n files[k] = None\n return files", "response": "Returns a dict of all the aux files that are associated with this filename."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating an island of a single image and return a list of islands.", "response": "def _gen_flood_wrap(self, data, rmsimg, innerclip, outerclip=None, domask=False):\n \"\"\"\n Generator function.\n Segment an image into islands and return one island at a time.\n\n Needs to work for entire image, and also for components within an island.\n\n Parameters\n ----------\n data : 2d-array\n Image array.\n\n rmsimg : 2d-array\n Noise image.\n\n innerclip, outerclip :float\n Seed (inner) and flood (outer) clipping values.\n\n domask : bool\n If True then look for a region mask in globals, only return islands that are within the region.\n Default = False.\n\n Yields\n ------\n data_box : 2d-array\n A island of sources with subthreshold values masked.\n\n xmin, xmax, ymin, ymax : int\n The corners of the data_box within the initial data array.\n \"\"\"\n\n if outerclip is None:\n outerclip = innerclip\n\n # compute SNR image (data has already been background subtracted)\n snr = abs(data) / rmsimg\n # mask of pixles that are above the outerclip\n a = snr >= outerclip\n # segmentation a la scipy\n l, n = label(a)\n f = find_objects(l)\n\n if n == 0:\n self.log.debug(\"There are no pixels above the clipping limit\")\n return\n self.log.debug(\"{1} Found {0} islands total above flood limit\".format(n, data.shape))\n # Yield values as before, though they are not sorted by flux\n for i in range(n):\n xmin, xmax = f[i][0].start, f[i][0].stop\n ymin, ymax = f[i][1].start, f[i][1].stop\n if np.any(snr[xmin:xmax, ymin:ymax] > innerclip): # obey inner clip constraint\n # self.log.info(\"{1} Island {0} is above the inner clip limit\".format(i, data.shape))\n data_box = copy.copy(data[xmin:xmax, ymin:ymax]) # copy so that we don't blank the master data\n data_box[np.where(\n snr[xmin:xmax, ymin:ymax] < outerclip)] = np.nan # blank pixels that are outside the outerclip\n data_box[np.where(l[xmin:xmax, ymin:ymax] != i + 1)] = np.nan # blank out other summits\n # check if there are any pixels left unmasked\n if not np.any(np.isfinite(data_box)):\n # self.log.info(\"{1} Island {0} has no non-masked pixels\".format(i,data.shape))\n continue\n if domask and (self.global_data.region is not None):\n y, x = np.where(snr[xmin:xmax, ymin:ymax] >= outerclip)\n # convert indices of this sub region to indices in the greater image\n yx = list(zip(y + ymin, x + xmin))\n ra, dec = self.global_data.wcshelper.wcs.wcs_pix2world(yx, 1).transpose()\n mask = self.global_data.region.sky_within(ra, dec, degin=True)\n # if there are no un-masked pixels within the region then we skip this island.\n if not np.any(mask):\n continue\n self.log.debug(\"Mask {0}\".format(mask))\n # self.log.info(\"{1} Island {0} will be fit\".format(i, data.shape))\n yield data_box, xmin, xmax, ymin, ymax"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef estimate_lmfit_parinfo(self, data, rmsimg, curve, beam, innerclip, outerclip=None, offsets=(0, 0),\n max_summits=None):\n \"\"\"\n Estimates the number of sources in an island and returns initial parameters for the fit as well as\n limits on those parameters.\n\n Parameters\n ----------\n data : 2d-array\n (sub) image of flux values. Background should be subtracted.\n\n rmsimg : 2d-array\n Image of 1sigma values\n\n curve : 2d-array\n Image of curvature values [-1,0,+1]\n\n beam : :class:`AegeanTools.fits_image.Beam`\n The beam information for the image.\n\n innerclip, outerclip : float\n Inerr and outer level for clipping (sigmas).\n\n offsets : (int, int)\n The (x,y) offset of data within it's parent image\n\n max_summits : int\n If not None, only this many summits/components will be fit. More components may be\n present in the island, but subsequent components will not have free parameters.\n\n Returns\n -------\n model : lmfit.Parameters\n The initial estimate of parameters for the components within this island.\n \"\"\"\n\n debug_on = self.log.isEnabledFor(logging.DEBUG)\n is_flag = 0\n global_data = self.global_data\n\n # check to see if this island is a negative peak since we need to treat such cases slightly differently\n isnegative = max(data[np.where(np.isfinite(data))]) < 0\n if isnegative:\n self.log.debug(\"[is a negative island]\")\n\n if outerclip is None:\n outerclip = innerclip\n\n self.log.debug(\" - shape {0}\".format(data.shape))\n\n if not data.shape == curve.shape:\n self.log.error(\"data and curvature are mismatched\")\n self.log.error(\"data:{0} curve:{1}\".format(data.shape, curve.shape))\n raise AssertionError()\n\n # For small islands we can't do a 6 param fit\n # Don't count the NaN values as part of the island\n non_nan_pix = len(data[np.where(np.isfinite(data))].ravel())\n if 4 <= non_nan_pix <= 6:\n self.log.debug(\"FIXED2PSF\")\n is_flag |= flags.FIXED2PSF\n elif non_nan_pix < 4:\n self.log.debug(\"FITERRSMALL!\")\n is_flag |= flags.FITERRSMALL\n else:\n is_flag = 0\n if debug_on:\n self.log.debug(\" - size {0}\".format(len(data.ravel())))\n\n if min(data.shape) <= 2 or (is_flag & flags.FITERRSMALL) or (is_flag & flags.FIXED2PSF):\n # 1d islands or small islands only get one source\n if debug_on:\n self.log.debug(\"Tiny summit detected\")\n self.log.debug(\"{0}\".format(data))\n summits = [[data, 0, data.shape[0], 0, data.shape[1]]]\n # and are constrained to be point sources\n is_flag |= flags.FIXED2PSF\n else:\n if isnegative:\n # the summit should be able to include all pixels within the island not just those above innerclip\n kappa_sigma = np.where(curve > 0.5, np.where(data + outerclip * rmsimg < 0, data, np.nan), np.nan)\n else:\n kappa_sigma = np.where(-1 * curve > 0.5, np.where(data - outerclip * rmsimg > 0, data, np.nan), np.nan)\n summits = list(self._gen_flood_wrap(kappa_sigma, np.ones(kappa_sigma.shape), 0, domask=False))\n\n params = lmfit.Parameters()\n i = 0\n summits_considered = 0\n # This can happen when the image contains regions of nans\n # the data/noise indicate an island, but the curvature doesn't back it up.\n if len(summits) < 1:\n self.log.debug(\"Island has {0} summits\".format(len(summits)))\n return None\n\n # add summits in reverse order of peak SNR - ie brightest first\n for summit, xmin, xmax, ymin, ymax in sorted(summits, key=lambda x: np.nanmax(-1. * abs(x[0]))):\n summits_considered += 1\n summit_flag = is_flag\n if debug_on:\n self.log.debug(\n \"Summit({5}) - shape:{0} x:[{1}-{2}] y:[{3}-{4}]\".format(summit.shape, ymin, ymax, xmin, xmax, i))\n try:\n if isnegative:\n amp = np.nanmin(summit)\n xpeak, ypeak = np.unravel_index(np.nanargmin(summit), summit.shape)\n else:\n amp = np.nanmax(summit)\n xpeak, ypeak = np.unravel_index(np.nanargmax(summit), summit.shape)\n except ValueError as e:\n if \"All-NaN\" in e.message:\n self.log.warn(\"Summit of nan's detected - this shouldn't happen\")\n continue\n else:\n raise e\n\n if debug_on:\n self.log.debug(\" - max is {0:f}\".format(amp))\n self.log.debug(\" - peak at {0},{1}\".format(xpeak, ypeak))\n yo = ypeak + ymin\n xo = xpeak + xmin\n\n # Summits are allowed to include pixels that are between the outer and inner clip\n # This means that sometimes we get a summit that has all it's pixels below the inner clip\n # So we test for that here.\n snr = np.nanmax(abs(data[xmin:xmax + 1, ymin:ymax + 1] / rmsimg[xmin:xmax + 1, ymin:ymax + 1]))\n if snr < innerclip:\n self.log.debug(\"Summit has SNR {0} < innerclip {1}: skipping\".format(snr, innerclip))\n continue\n\n # allow amp to be 5% or (innerclip) sigma higher\n # TODO: the 5% should depend on the beam sampling\n # note: when innerclip is 400 this becomes rather stupid\n if amp > 0:\n amp_min, amp_max = 0.95 * min(outerclip * rmsimg[xo, yo], amp), amp * 1.05 + innerclip * rmsimg[xo, yo]\n else:\n amp_max, amp_min = 0.95 * max(-outerclip * rmsimg[xo, yo], amp), amp * 1.05 - innerclip * rmsimg[xo, yo]\n\n if debug_on:\n self.log.debug(\"a_min {0}, a_max {1}\".format(amp_min, amp_max))\n\n pixbeam = global_data.psfhelper.get_pixbeam_pixel(yo + offsets[0], xo + offsets[1])\n if pixbeam is None:\n self.log.debug(\" Summit has invalid WCS/Beam - Skipping.\")\n continue\n\n # set a square limit based on the size of the pixbeam\n xo_lim = 0.5 * np.hypot(pixbeam.a, pixbeam.b)\n yo_lim = xo_lim\n\n yo_min, yo_max = yo - yo_lim, yo + yo_lim\n # if yo_min == yo_max: # if we have a 1d summit then allow the position to vary by +/-0.5pix\n # yo_min, yo_max = yo_min - 0.5, yo_max + 0.5\n\n xo_min, xo_max = xo - xo_lim, xo + xo_lim\n # if xo_min == xo_max: # if we have a 1d summit then allow the position to vary by +/-0.5pix\n # xo_min, xo_max = xo_min - 0.5, xo_max + 0.5\n\n # the size of the island\n xsize = data.shape[0]\n ysize = data.shape[1]\n\n # initial shape is the psf\n sx = pixbeam.a * FWHM2CC\n sy = pixbeam.b * FWHM2CC\n\n # lmfit does silly things if we start with these two parameters being equal\n sx = max(sx, sy * 1.01)\n\n # constraints are based on the shape of the island\n # sx,sy can become flipped so we set the min/max account for this\n sx_min, sx_max = sy * 0.8, max((max(xsize, ysize) + 1) * math.sqrt(2) * FWHM2CC, sx * 1.1)\n sy_min, sy_max = sy * 0.8, max((max(xsize, ysize) + 1) * math.sqrt(2) * FWHM2CC, sx * 1.1)\n\n theta = pixbeam.pa # Degrees\n flag = summit_flag\n\n # check to see if we are going to fit this component\n if max_summits is not None:\n maxxed = i >= max_summits\n else:\n maxxed = False\n\n # components that are not fit need appropriate flags\n if maxxed:\n summit_flag |= flags.NOTFIT\n summit_flag |= flags.FIXED2PSF\n\n if debug_on:\n self.log.debug(\" - var val min max | min max\")\n self.log.debug(\" - amp {0} {1} {2} \".format(amp, amp_min, amp_max))\n self.log.debug(\" - xo {0} {1} {2} \".format(xo, xo_min, xo_max))\n self.log.debug(\" - yo {0} {1} {2} \".format(yo, yo_min, yo_max))\n self.log.debug(\" - sx {0} {1} {2} | {3} {4}\".format(sx, sx_min, sx_max, sx_min * CC2FHWM,\n sx_max * CC2FHWM))\n self.log.debug(\" - sy {0} {1} {2} | {3} {4}\".format(sy, sy_min, sy_max, sy_min * CC2FHWM,\n sy_max * CC2FHWM))\n self.log.debug(\" - theta {0} {1} {2}\".format(theta, -180, 180))\n self.log.debug(\" - flags {0}\".format(flag))\n self.log.debug(\" - fit? {0}\".format(not maxxed))\n\n # TODO: figure out how incorporate the circular constraint on sx/sy\n prefix = \"c{0}_\".format(i)\n params.add(prefix + 'amp', value=amp, min=amp_min, max=amp_max, vary=not maxxed)\n params.add(prefix + 'xo', value=xo, min=float(xo_min), max=float(xo_max), vary=not maxxed)\n params.add(prefix + 'yo', value=yo, min=float(yo_min), max=float(yo_max), vary=not maxxed)\n\n if summit_flag & flags.FIXED2PSF > 0:\n psf_vary = False\n else:\n psf_vary = not maxxed\n params.add(prefix + 'sx', value=sx, min=sx_min, max=sx_max, vary=psf_vary)\n params.add(prefix + 'sy', value=sy, min=sy_min, max=sy_max, vary=psf_vary)\n params.add(prefix + 'theta', value=theta, vary=psf_vary)\n params.add(prefix + 'flags', value=summit_flag, vary=False)\n\n # starting at zero allows the maj/min axes to be fit better.\n # if params[prefix + 'theta'].vary:\n # params[prefix + 'theta'].value = 0\n\n i += 1\n if debug_on:\n self.log.debug(\"Estimated sources: {0}\".format(i))\n # remember how many components are fit.\n params.add('components', value=i, vary=False)\n # params.components=i\n if params['components'].value < 1:\n self.log.debug(\"Considered {0} summits, accepted {1}\".format(summits_considered, i))\n return params", "response": "Estimate the number of sources in an island and returns initial parameters for the fit."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef result_to_components(self, result, model, island_data, isflags):\n global_data = self.global_data\n\n # island data\n isle_num = island_data.isle_num\n idata = island_data.i\n xmin, xmax, ymin, ymax = island_data.offsets\n\n box = slice(int(xmin), int(xmax)), slice(int(ymin), int(ymax))\n rms = global_data.rmsimg[box]\n bkg = global_data.bkgimg[box]\n residual = np.median(result.residual), np.std(result.residual)\n is_flag = isflags\n\n sources = []\n j = 0\n for j in range(model['components'].value):\n src_flags = is_flag\n source = OutputSource()\n source.island = isle_num\n source.source = j\n self.log.debug(\" component {0}\".format(j))\n prefix = \"c{0}_\".format(j)\n xo = model[prefix + 'xo'].value\n yo = model[prefix + 'yo'].value\n sx = model[prefix + 'sx'].value\n sy = model[prefix + 'sy'].value\n theta = model[prefix + 'theta'].value\n amp = model[prefix + 'amp'].value\n src_flags |= model[prefix + 'flags'].value\n\n # these are goodness of fit statistics for the entire island.\n source.residual_mean = residual[0]\n source.residual_std = residual[1]\n # set the flags\n source.flags = src_flags\n\n # #pixel pos within island +\n # island offset within region +\n # region offset within image +\n # 1 for luck\n # (pyfits->fits conversion = luck)\n x_pix = xo + xmin + 1\n y_pix = yo + ymin + 1\n # update the source xo/yo so the error calculations can be done correctly\n # Note that you have to update the max or the value you set will be clipped at the max allowed value\n model[prefix + 'xo'].set(value=x_pix, max=np.inf)\n model[prefix + 'yo'].set(value=y_pix, max=np.inf)\n # ------ extract source parameters ------\n\n # fluxes\n # the background is taken from background map\n # Clamp the pixel location to the edge of the background map\n y = max(min(int(round(y_pix - ymin)), bkg.shape[1] - 1), 0)\n x = max(min(int(round(x_pix - xmin)), bkg.shape[0] - 1), 0)\n source.background = bkg[x, y]\n source.local_rms = rms[x, y]\n source.peak_flux = amp\n\n # all params are in degrees\n source.ra, source.dec, source.a, source.b, source.pa = global_data.wcshelper.pix2sky_ellipse((x_pix, y_pix),\n sx * CC2FHWM,\n sy * CC2FHWM,\n theta)\n source.a *= 3600 # arcseconds\n source.b *= 3600\n # force a>=b\n fix_shape(source)\n # limit the pa to be in (-90,90]\n source.pa = pa_limit(source.pa)\n\n # if one of these values are nan then there has been some problem with the WCS handling\n if not all(np.isfinite((source.ra, source.dec, source.a, source.b, source.pa))):\n src_flags |= flags.WCSERR\n # negative degrees is valid for RA, but I don't want them.\n if source.ra < 0:\n source.ra += 360\n source.ra_str = dec2hms(source.ra)\n source.dec_str = dec2dms(source.dec)\n\n # calculate integrated flux\n source.int_flux = source.peak_flux * sx * sy * CC2FHWM ** 2 * np.pi\n # scale Jy/beam -> Jy using the area of the beam\n source.int_flux /= global_data.psfhelper.get_beamarea_pix(source.ra, source.dec)\n\n # Calculate errors for params that were fit (as well as int_flux)\n errors(source, model, global_data.wcshelper)\n\n source.flags = src_flags\n # add psf info\n local_beam = global_data.psfhelper.get_beam(source.ra, source.dec)\n if local_beam is not None:\n source.psf_a = local_beam.a * 3600\n source.psf_b = local_beam.b * 3600\n source.psf_pa = local_beam.pa\n else:\n source.psf_a = 0\n source.psf_b = 0\n source.psf_pa = 0\n sources.append(source)\n self.log.debug(source)\n\n if global_data.blank:\n outerclip = island_data.scalars[1]\n idx, idy = np.where(abs(idata) - outerclip * rms > 0)\n idx += xmin\n idy += ymin\n self.global_data.img._pixels[[idx, idy]] = np.nan\n\n # calculate the integrated island flux if required\n if island_data.doislandflux:\n _, outerclip, _ = island_data.scalars\n self.log.debug(\"Integrated flux for island {0}\".format(isle_num))\n kappa_sigma = np.where(abs(idata) - outerclip * rms > 0, idata, np.NaN)\n self.log.debug(\"- island shape is {0}\".format(kappa_sigma.shape))\n\n source = IslandSource()\n source.flags = 0\n source.island = isle_num\n source.components = j + 1\n source.peak_flux = np.nanmax(kappa_sigma)\n # check for negative islands\n if source.peak_flux < 0:\n source.peak_flux = np.nanmin(kappa_sigma)\n self.log.debug(\"- peak flux {0}\".format(source.peak_flux))\n\n # positions and background\n if np.isfinite(source.peak_flux):\n positions = np.where(kappa_sigma == source.peak_flux)\n else: # if a component has been refit then it might have flux = np.nan\n positions = [[kappa_sigma.shape[0] / 2], [kappa_sigma.shape[1] / 2]]\n xy = positions[0][0] + xmin, positions[1][0] + ymin\n radec = global_data.wcshelper.pix2sky(xy)\n source.ra = radec[0]\n\n # convert negative ra's to positive ones\n if source.ra < 0:\n source.ra += 360\n\n source.dec = radec[1]\n source.ra_str = dec2hms(source.ra)\n source.dec_str = dec2dms(source.dec)\n source.background = bkg[positions[0][0], positions[1][0]]\n source.local_rms = rms[positions[0][0], positions[1][0]]\n source.x_width, source.y_width = idata.shape\n source.pixels = int(sum(np.isfinite(kappa_sigma).ravel() * 1.0))\n source.extent = [xmin, xmax, ymin, ymax]\n\n # TODO: investigate what happens when the sky coords are skewed w.r.t the pixel coords\n # calculate the area of the island as a fraction of the area of the bounding box\n bl = global_data.wcshelper.pix2sky([xmax, ymin])\n tl = global_data.wcshelper.pix2sky([xmax, ymax])\n tr = global_data.wcshelper.pix2sky([xmin, ymax])\n height = gcd(tl[0], tl[1], bl[0], bl[1])\n width = gcd(tl[0], tl[1], tr[0], tr[1])\n area = height * width\n source.area = area * source.pixels / source.x_width / source.y_width # area is in deg^2\n\n # create contours\n msq = MarchingSquares(idata)\n source.contour = [(a[0] + xmin, a[1] + ymin) for a in msq.perimeter]\n # calculate the maximum angular size of this island, brute force method\n source.max_angular_size = 0\n for i, pos1 in enumerate(source.contour):\n radec1 = global_data.wcshelper.pix2sky(pos1)\n for j, pos2 in enumerate(source.contour[i:]):\n radec2 = global_data.wcshelper.pix2sky(pos2)\n dist = gcd(radec1[0], radec1[1], radec2[0], radec2[1])\n if dist > source.max_angular_size:\n source.max_angular_size = dist\n source.pa = bear(radec1[0], radec1[1], radec2[0], radec2[1])\n source.max_angular_size_anchors = [pos1[0], pos1[1], pos2[0], pos2[1]]\n\n self.log.debug(\"- peak position {0}, {1} [{2},{3}]\".format(source.ra_str, source.dec_str, positions[0][0],\n positions[1][0]))\n\n # integrated flux\n beam_area = global_data.psfhelper.get_beamarea_deg2(source.ra, source.dec) # beam in deg^2\n # get_beamarea_pix(source.ra, source.dec) # beam is in pix^2\n isize = source.pixels # number of non zero pixels\n self.log.debug(\"- pixels used {0}\".format(isize))\n source.int_flux = np.nansum(kappa_sigma) # total flux Jy/beam\n self.log.debug(\"- sum of pixles {0}\".format(source.int_flux))\n source.int_flux *= beam_area # total flux in Jy\n self.log.debug(\"- integrated flux {0}\".format(source.int_flux))\n eta = erf(np.sqrt(-1 * np.log(abs(source.local_rms * outerclip / source.peak_flux)))) ** 2\n self.log.debug(\"- eta {0}\".format(eta))\n source.eta = eta\n source.beam_area = beam_area\n\n # I don't know how to calculate this error so we'll set it to nan\n source.err_int_flux = np.nan\n sources.append(source)\n return sources", "response": "Convert fitting results into a set of components and islands."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_globals(self, filename, hdu_index=0, bkgin=None, rmsin=None, beam=None, verb=False, rms=None, bkg=None, cores=1,\n do_curve=True, mask=None, lat=None, psf=None, blank=False, docov=True, cube_index=None):\n \"\"\"\n Populate the global_data object by loading or calculating the various components\n\n Parameters\n ----------\n filename : str or HDUList\n Main image which source finding is run on\n\n hdu_index : int\n HDU index of the image within the fits file, default is 0 (first)\n\n bkgin, rmsin : str or HDUList\n background and noise image filename or HDUList\n\n beam : :class:`AegeanTools.fits_image.Beam`\n Beam object representing the synthsized beam. Will replace what is in the FITS header.\n\n verb : bool\n Verbose. Write extra lines to INFO level log.\n\n rms, bkg : float\n A float that represents a constant rms/bkg levels for the entire image.\n Default = None, which causes the rms/bkg to be loaded or calculated.\n\n cores : int\n Number of cores to use if different from what is autodetected.\n\n\n do_curve : bool\n If True a curvature map will be created, default=True.\n\n mask : str or :class:`AegeanTools.regions.Region`\n filename or Region object\n\n lat : float\n Latitude of the observing telescope (declination of zenith)\n\n psf : str or HDUList\n Filename or HDUList of a psf image\n\n blank : bool\n True = blank output image where islands are found.\n Default = False.\n\n docov : bool\n True = use covariance matrix in fitting.\n Default = True.\n\n cube_index : int\n For an image cube, which slice to use.\n\n \"\"\"\n # don't reload already loaded data\n if self.global_data.img is not None:\n return\n img = FitsImage(filename, hdu_index=hdu_index, beam=beam, cube_index=cube_index)\n beam = img.beam\n\n debug = logging.getLogger('Aegean').isEnabledFor(logging.DEBUG)\n\n if mask is None:\n self.global_data.region = None\n else:\n # allow users to supply and object instead of a filename\n if isinstance(mask, Region):\n self.global_data.region = mask\n elif os.path.exists(mask):\n self.log.info(\"Loading mask from {0}\".format(mask))\n self.global_data.region = Region.load(mask)\n else:\n self.log.error(\"File {0} not found for loading\".format(mask))\n self.global_data.region = None\n\n self.global_data.wcshelper = WCSHelper.from_header(img.get_hdu_header(), beam, lat)\n self.global_data.psfhelper = PSFHelper(psf, self.global_data.wcshelper)\n\n self.global_data.beam = self.global_data.wcshelper.beam\n self.global_data.img = img\n self.global_data.data_pix = img.get_pixels()\n self.global_data.dtype = type(self.global_data.data_pix[0][0])\n self.global_data.bkgimg = np.zeros(self.global_data.data_pix.shape, dtype=self.global_data.dtype)\n self.global_data.rmsimg = np.zeros(self.global_data.data_pix.shape, dtype=self.global_data.dtype)\n self.global_data.pixarea = img.pixarea\n self.global_data.dcurve = None\n\n if do_curve:\n self.log.info(\"Calculating curvature\")\n # calculate curvature but store it as -1,0,+1\n dcurve = np.zeros(self.global_data.data_pix.shape, dtype=np.int8)\n peaks = scipy.ndimage.filters.maximum_filter(self.global_data.data_pix, size=3)\n troughs = scipy.ndimage.filters.minimum_filter(self.global_data.data_pix, size=3)\n pmask = np.where(self.global_data.data_pix == peaks)\n tmask = np.where(self.global_data.data_pix == troughs)\n dcurve[pmask] = -1\n dcurve[tmask] = 1\n self.global_data.dcurve = dcurve\n\n # if either of rms or bkg images are not supplied then calculate them both\n if not (rmsin and bkgin):\n if verb:\n self.log.info(\"Calculating background and rms data\")\n self._make_bkg_rms(mesh_size=20, forced_rms=rms, forced_bkg=bkg, cores=cores)\n\n # replace the calculated images with input versions, if the user has supplied them.\n if bkgin:\n if verb:\n self.log.info(\"Loading background data from file {0}\".format(bkgin))\n self.global_data.bkgimg = self._load_aux_image(img, bkgin)\n if rmsin:\n if verb:\n self.log.info(\"Loading rms data from file {0}\".format(rmsin))\n self.global_data.rmsimg = self._load_aux_image(img, rmsin)\n\n # subtract the background image from the data image and save\n if verb and debug:\n self.log.debug(\"Data max is {0}\".format(img.get_pixels()[np.isfinite(img.get_pixels())].max()))\n self.log.debug(\"Doing background subtraction\")\n img.set_pixels(img.get_pixels() - self.global_data.bkgimg)\n self.global_data.data_pix = img.get_pixels()\n if verb and debug:\n self.log.debug(\"Data max is {0}\".format(img.get_pixels()[np.isfinite(img.get_pixels())].max()))\n\n self.global_data.blank = blank\n self.global_data.docov = docov\n\n # Default to false until I can verify that this is working\n self.global_data.dobias = False\n\n # check if the WCS is galactic\n if 'lon' in self.global_data.img._header['CTYPE1'].lower():\n self.log.info(\"Galactic coordinates detected and noted\")\n SimpleSource.galactic = True\n return", "response": "Loads the global_data object from a FITS file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating and save the background and RMS maps as FITS files.", "response": "def save_background_files(self, image_filename, hdu_index=0, bkgin=None, rmsin=None, beam=None, rms=None, bkg=None, cores=1,\n outbase=None):\n \"\"\"\n Generate and save the background and RMS maps as FITS files.\n They are saved in the current directly as aegean-background.fits and aegean-rms.fits.\n\n Parameters\n ----------\n image_filename : str or HDUList\n Input image.\n\n hdu_index : int\n If fits file has more than one hdu, it can be specified here.\n Default = 0.\n\n bkgin, rmsin : str or HDUList\n Background and noise image filename or HDUList\n\n beam : :class:`AegeanTools.fits_image.Beam`\n Beam object representing the synthsized beam. Will replace what is in the FITS header.\n\n\n rms, bkg : float\n A float that represents a constant rms/bkg level for the entire image.\n Default = None, which causes the rms/bkg to be loaded or calculated.\n\n cores : int\n Number of cores to use if different from what is autodetected.\n\n outbase : str\n Basename for output files.\n\n \"\"\"\n\n self.log.info(\"Saving background / RMS maps\")\n # load image, and load/create background/rms images\n self.load_globals(image_filename, hdu_index=hdu_index, bkgin=bkgin, rmsin=rmsin, beam=beam, verb=True, rms=rms, bkg=bkg,\n cores=cores, do_curve=True)\n img = self.global_data.img\n bkgimg, rmsimg = self.global_data.bkgimg, self.global_data.rmsimg\n curve = np.array(self.global_data.dcurve, dtype=bkgimg.dtype)\n # mask these arrays have the same mask the same as the data\n mask = np.where(np.isnan(self.global_data.data_pix))\n bkgimg[mask] = np.NaN\n rmsimg[mask] = np.NaN\n curve[mask] = np.NaN\n\n # Generate the new FITS files by copying the existing HDU and assigning new data.\n # This gives the new files the same WCS projection and other header fields.\n new_hdu = img.hdu\n # Set the ORIGIN to indicate Aegean made this file\n new_hdu.header[\"ORIGIN\"] = \"Aegean {0}-({1})\".format(__version__, __date__)\n for c in ['CRPIX3', 'CRPIX4', 'CDELT3', 'CDELT4', 'CRVAL3', 'CRVAL4', 'CTYPE3', 'CTYPE4']:\n if c in new_hdu.header:\n del new_hdu.header[c]\n\n if outbase is None:\n outbase, _ = os.path.splitext(os.path.basename(image_filename))\n noise_out = outbase + '_rms.fits'\n background_out = outbase + '_bkg.fits'\n curve_out = outbase + '_crv.fits'\n snr_out = outbase + '_snr.fits'\n\n new_hdu.data = bkgimg\n new_hdu.writeto(background_out, overwrite=True)\n self.log.info(\"Wrote {0}\".format(background_out))\n\n new_hdu.data = rmsimg\n new_hdu.writeto(noise_out, overwrite=True)\n self.log.info(\"Wrote {0}\".format(noise_out))\n\n new_hdu.data = curve\n new_hdu.writeto(curve_out, overwrite=True)\n self.log.info(\"Wrote {0}\".format(curve_out))\n\n new_hdu.data = self.global_data.data_pix / rmsimg\n new_hdu.writeto(snr_out, overwrite=True)\n self.log.info(\"Wrote {0}\".format(snr_out))\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef save_image(self, outname):\n hdu = self.global_data.img.hdu\n hdu.data = self.global_data.img._pixels\n hdu.header[\"ORIGIN\"] = \"Aegean {0}-({1})\".format(__version__, __date__)\n # delete some axes that we aren't going to need\n for c in ['CRPIX3', 'CRPIX4', 'CDELT3', 'CDELT4', 'CRVAL3', 'CRVAL4', 'CTYPE3', 'CTYPE4']:\n if c in hdu.header:\n del hdu.header[c]\n hdu.writeto(outname, overwrite=True)\n self.log.info(\"Wrote {0}\".format(outname))\n return", "response": "Save the image data to a file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating an rms image and a bkg image. Parameters ---------- mesh_size : int Number of beams per box default = 20 forced_rms : float The rms of the image. If None: calculate the rms level (default). Otherwise assume a constant rms. forced_bkg : float The background level of the image. If None: calculate the background level (default). Otherwise assume a constant background. cores: int Number of cores to use if different from what is autodetected.", "response": "def _make_bkg_rms(self, mesh_size=20, forced_rms=None, forced_bkg=None, cores=None):\n \"\"\"\n Calculate an rms image and a bkg image.\n\n Parameters\n ----------\n mesh_size : int\n Number of beams per box default = 20\n\n forced_rms : float\n The rms of the image.\n If None: calculate the rms level (default).\n Otherwise assume a constant rms.\n\n forced_bkg : float\n The background level of the image.\n If None: calculate the background level (default).\n Otherwise assume a constant background.\n\n cores: int\n Number of cores to use if different from what is autodetected.\n\n \"\"\"\n if (forced_rms is not None):\n self.log.info(\"Forcing rms = {0}\".format(forced_rms))\n self.global_data.rmsimg[:] = forced_rms\n if (forced_bkg is not None):\n self.log.info(\"Forcing bkg = {0}\".format(forced_bkg))\n self.global_data.bkgimg[:] = forced_bkg\n\n # If we known both the rms and the bkg then there is nothing to compute\n if (forced_rms is not None) and (forced_bkg is not None):\n return\n\n data = self.global_data.data_pix\n beam = self.global_data.beam\n\n img_x, img_y = data.shape\n xcen = int(img_x / 2)\n ycen = int(img_y / 2)\n\n # calculate a local beam from the center of the data\n pixbeam = self.global_data.psfhelper.get_pixbeam_pixel(xcen, ycen)\n if pixbeam is None:\n self.log.error(\"Cannot determine the beam shape at the image center\")\n sys.exit(1)\n\n width_x = mesh_size * max(abs(math.cos(np.radians(pixbeam.pa)) * pixbeam.a),\n abs(math.sin(np.radians(pixbeam.pa)) * pixbeam.b))\n width_x = int(width_x)\n width_y = mesh_size * max(abs(math.sin(np.radians(pixbeam.pa)) * pixbeam.a),\n abs(math.cos(np.radians(pixbeam.pa)) * pixbeam.b))\n width_y = int(width_y)\n\n self.log.debug(\"image size x,y:{0},{1}\".format(img_x, img_y))\n self.log.debug(\"beam: {0}\".format(beam))\n self.log.debug(\"mesh width (pix) x,y: {0},{1}\".format(width_x, width_y))\n\n # box centered at image center then tilling outwards\n xstart = int(xcen - width_x / 2) % width_x # the starting point of the first \"full\" box\n ystart = int(ycen - width_y / 2) % width_y\n\n xend = img_x - int(img_x - xstart) % width_x # the end point of the last \"full\" box\n yend = img_y - int(img_y - ystart) % width_y\n\n xmins = [0]\n xmins.extend(list(range(xstart, xend, width_x)))\n xmins.append(xend)\n\n xmaxs = [xstart]\n xmaxs.extend(list(range(xstart + width_x, xend + 1, width_x)))\n xmaxs.append(img_x)\n\n ymins = [0]\n ymins.extend(list(range(ystart, yend, width_y)))\n ymins.append(yend)\n\n ymaxs = [ystart]\n ymaxs.extend(list(range(ystart + width_y, yend + 1, width_y)))\n ymaxs.append(img_y)\n\n # if the image is smaller than our ideal mesh size, just use the whole image instead\n if width_x >= img_x:\n xmins = [0]\n xmaxs = [img_x]\n if width_y >= img_y:\n ymins = [0]\n ymaxs = [img_y]\n\n if cores > 1:\n # set up the queue\n queue = pprocess.Queue(limit=cores, reuse=1)\n estimate = queue.manage(pprocess.MakeReusable(self._estimate_bkg_rms))\n # populate the queue\n for xmin, xmax in zip(xmins, xmaxs):\n for ymin, ymax in zip(ymins, ymaxs):\n estimate(ymin, ymax, xmin, xmax)\n else:\n queue = []\n for xmin, xmax in zip(xmins, xmaxs):\n for ymin, ymax in zip(ymins, ymaxs):\n queue.append(self._estimate_bkg_rms(xmin, xmax, ymin, ymax))\n\n # only copy across the bkg/rms if they are not already set\n # queue can only be traversed once so we have to put the if inside the loop\n for ymin, ymax, xmin, xmax, bkg, rms in queue:\n if (forced_rms is None):\n self.global_data.rmsimg[ymin:ymax, xmin:xmax] = rms\n if (forced_rms is None):\n self.global_data.bkgimg[ymin:ymax, xmin:xmax] = bkg\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nestimating the background noise mean and RMS.", "response": "def _estimate_bkg_rms(self, xmin, xmax, ymin, ymax):\n \"\"\"\n Estimate the background noise mean and RMS.\n The mean is estimated as the median of data.\n The RMS is estimated as the IQR of data / 1.34896.\n\n Parameters\n ----------\n xmin, xmax, ymin, ymax : int\n The bounding region over which the bkg/rms will be calculated.\n\n Returns\n -------\n ymin, ymax, xmin, xmax : int\n A copy of the input parameters\n\n bkg, rms : float\n The calculated background and noise.\n \"\"\"\n data = self.global_data.data_pix[ymin:ymax, xmin:xmax]\n pixels = np.extract(np.isfinite(data), data).ravel()\n if len(pixels) < 4:\n bkg, rms = np.NaN, np.NaN\n else:\n pixels.sort()\n p25 = pixels[int(pixels.size / 4)]\n p50 = pixels[int(pixels.size / 2)]\n p75 = pixels[int(pixels.size / 4 * 3)]\n iqr = p75 - p25\n bkg, rms = p50, iqr / 1.34896\n # return the input and output data so we know what we are doing\n # when compiling the results of multiple processes\n return ymin, ymax, xmin, xmax, bkg, rms"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _load_aux_image(self, image, auxfile):\n auximg = FitsImage(auxfile, beam=self.global_data.beam).get_pixels()\n if auximg.shape != image.get_pixels().shape:\n self.log.error(\"file {0} is not the same size as the image map\".format(auxfile))\n self.log.error(\"{0}= {1}, image = {2}\".format(auxfile, auximg.shape, image.get_pixels().shape))\n sys.exit(1)\n return auximg", "response": "Load a main image and auxiliary file and make sure that it is the same shape as the main image."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrefit islands on a group of components.", "response": "def _refit_islands(self, group, stage, outerclip=None, istart=0):\n \"\"\"\n Do island refitting (priorized fitting) on a group of islands.\n\n Parameters\n ----------\n group : list\n A list of components grouped by island.\n\n stage : int\n Refitting stage.\n\n outerclip : float\n Ignored, placed holder for future development.\n\n istart : int\n The starting island number.\n\n Returns\n -------\n sources : list\n List of sources (and islands).\n \"\"\"\n global_data = self.global_data\n sources = []\n\n data = global_data.data_pix\n rmsimg = global_data.rmsimg\n\n for inum, isle in enumerate(group, start=istart):\n self.log.debug(\"-=-\")\n self.log.debug(\"input island = {0}, {1} components\".format(isle[0].island, len(isle)))\n\n # set up the parameters for each of the sources within the island\n i = 0\n params = lmfit.Parameters()\n shape = data.shape\n xmin, ymin = shape\n xmax = ymax = 0\n\n # island_mask = []\n src_valid_psf = None\n # keep track of the sources that are actually being refit\n # this may be a subset of all sources in the island\n included_sources = []\n for src in isle:\n pixbeam = global_data.psfhelper.get_pixbeam(src.ra, src.dec)\n # find the right pixels from the ra/dec\n source_x, source_y = global_data.wcshelper.sky2pix([src.ra, src.dec])\n source_x -= 1\n source_y -= 1\n x = int(round(source_x))\n y = int(round(source_y))\n\n self.log.debug(\"pixel location ({0:5.2f},{1:5.2f})\".format(source_x, source_y))\n # reject sources that are outside the image bounds, or which have nan data/rms values\n if not 0 <= x < shape[0] or not 0 <= y < shape[1] or \\\n not np.isfinite(data[x, y]) or \\\n not np.isfinite(rmsimg[x, y]) or \\\n pixbeam is None:\n self.log.debug(\"Source ({0},{1}) not within usable region: skipping\".format(src.island, src.source))\n continue\n else:\n # Keep track of the last source to have a valid psf so that we can use it later on\n src_valid_psf = src\n # determine the shape parameters in pixel values\n _, _, sx, sy, theta = global_data.wcshelper.sky2pix_ellipse([src.ra, src.dec], src.a / 3600,\n src.b / 3600, src.pa)\n sx *= FWHM2CC\n sy *= FWHM2CC\n\n self.log.debug(\"Source shape [sky coords] {0:5.2f}x{1:5.2f}@{2:05.2f}\".format(src.a, src.b, src.pa))\n self.log.debug(\"Source shape [pixel coords] {0:4.2f}x{1:4.2f}@{2:05.2f}\".format(sx, sy, theta))\n\n # choose a region that is 2x the major axis of the source, 4x semimajor axis a\n width = 4 * sx\n ywidth = int(round(width)) + 1\n xwidth = int(round(width)) + 1\n\n # adjust the size of the island to include this source\n xmin = min(xmin, max(0, x - xwidth / 2))\n ymin = min(ymin, max(0, y - ywidth / 2))\n xmax = max(xmax, min(shape[0], x + xwidth / 2 + 1))\n ymax = max(ymax, min(shape[1], y + ywidth / 2 + 1))\n\n s_lims = [0.8 * min(sx, pixbeam.b * FWHM2CC), max(sy, sx) * 1.25]\n\n # Set up the parameters for the fit, including constraints\n prefix = \"c{0}_\".format(i)\n params.add(prefix + 'amp', value=src.peak_flux, vary=True)\n # for now the xo/yo are locations within the main image, we correct this later\n params.add(prefix + 'xo', value=source_x, min=source_x - sx / 2., max=source_x + sx / 2.,\n vary=stage >= 2)\n params.add(prefix + 'yo', value=source_y, min=source_y - sy / 2., max=source_y + sy / 2.,\n vary=stage >= 2)\n params.add(prefix + 'sx', value=sx, min=s_lims[0], max=s_lims[1], vary=stage >= 3)\n params.add(prefix + 'sy', value=sy, min=s_lims[0], max=s_lims[1], vary=stage >= 3)\n params.add(prefix + 'theta', value=theta, vary=stage >= 3)\n params.add(prefix + 'flags', value=0, vary=False)\n # this source is being refit so add it to the list\n included_sources.append(src)\n i += 1\n\n # TODO: Allow this mask to be used in conjunction with the FWHM mask that is defined further on\n # # Use pixels above outerclip sigmas..\n # if outerclip>=0:\n # mask = np.where(data[xmin:xmax,ymin:ymax]-outerclip*rmsimg[xmin:xmax,ymin:ymax]>0)\n # else: # negative outer clip means use all the pixels\n # mask = np.where(data[xmin:xmax,ymin:ymax])\n #\n # # convert the pixel indices to be pixels within the parent data set\n # xmask = mask[0] + xmin\n # ymask = mask[1] + ymin\n # island_mask.extend(zip(xmask,ymask))\n\n if i == 0:\n self.log.debug(\"No sources found in island {0}\".format(src.island))\n continue\n params.add('components', value=i, vary=False)\n # params.components = i\n self.log.debug(\" {0} components being fit\".format(i))\n # now we correct the xo/yo positions to be relative to the sub-image\n self.log.debug(\"xmxxymyx {0} {1} {2} {3}\".format(xmin, xmax, ymin, ymax))\n for i in range(params['components'].value):\n prefix = \"c{0}_\".format(i)\n params[prefix + 'xo'].value -= xmin\n params[prefix + 'xo'].min -= xmin\n params[prefix + 'xo'].max -= xmin\n params[prefix + 'yo'].value -= ymin\n params[prefix + 'yo'].min -= ymin\n params[prefix + 'yo'].max -= ymin\n # self.log.debug(params)\n # don't fit if there are no sources\n if params['components'].value < 1:\n self.log.info(\"Island {0} has no components\".format(src.island))\n continue\n\n # this .copy() will stop us from modifying the parent region when we later apply our mask.\n idata = data[int(xmin):int(xmax), int(ymin):int(ymax)].copy()\n # now convert these back to indices within the idata region\n # island_mask = np.array([(x-xmin, y-ymin) for x, y in island_mask])\n\n allx, ally = np.indices(idata.shape)\n # mask to include pixels that are withn the FWHM of the sources being fit\n mask_params = copy.deepcopy(params)\n for i in range(mask_params['components'].value):\n prefix = 'c{0}_'.format(i)\n mask_params[prefix + 'amp'].value = 1\n mask_model = ntwodgaussian_lmfit(mask_params)\n mask = np.where(mask_model(allx.ravel(), ally.ravel()) <= 0.1)\n mask = allx.ravel()[mask], ally.ravel()[mask]\n del mask_params\n\n idata[mask] = np.nan\n\n mx, my = np.where(np.isfinite(idata))\n non_nan_pix = len(mx)\n total_pix = len(allx.ravel())\n self.log.debug(\"island extracted:\")\n self.log.debug(\" x[{0}:{1}] y[{2}:{3}]\".format(xmin, xmax, ymin, ymax))\n self.log.debug(\" max = {0}\".format(np.nanmax(idata)))\n self.log.debug(\n \" total {0}, masked {1}, not masked {2}\".format(total_pix, total_pix - non_nan_pix, non_nan_pix))\n\n # Check to see that each component has some data within the central 3x3 pixels of it's location\n # If not then we don't fit that component\n for i in range(params['components'].value):\n prefix = \"c{0}_\".format(i)\n # figure out a box around the center of this\n cx, cy = params[prefix + 'xo'].value, params[prefix + 'yo'].value # central pixel coords\n self.log.debug(\" comp {0}\".format(i))\n self.log.debug(\" x0, y0 {0} {1}\".format(cx, cy))\n xmx = int(round(np.clip(cx + 2, 0, idata.shape[0])))\n xmn = int(round(np.clip(cx - 1, 0, idata.shape[0])))\n ymx = int(round(np.clip(cy + 2, 0, idata.shape[1])))\n ymn = int(round(np.clip(cy - 1, 0, idata.shape[1])))\n square = idata[xmn:xmx, ymn:ymx]\n # if there are no not-nan pixels in this region then don't vary any parameters\n if not np.any(np.isfinite(square)):\n self.log.debug(\" not fitting component {0}\".format(i))\n params[prefix + 'amp'].value = np.nan\n for p in ['amp', 'xo', 'yo', 'sx', 'sy', 'theta']:\n params[prefix + p].vary = False\n params[prefix + p].stderr = np.nan # this results in an error of -1 later on\n params[prefix + 'flags'].value |= flags.NOTFIT\n\n # determine the number of free parameters and if we have enough data for a fit\n nfree = np.count_nonzero([params[p].vary for p in params.keys()])\n self.log.debug(params)\n if nfree < 1:\n self.log.debug(\" Island has no components to fit\")\n result = DummyLM()\n model = params\n else:\n if non_nan_pix < nfree:\n self.log.debug(\"More free parameters {0} than available pixels {1}\".format(nfree, non_nan_pix))\n if non_nan_pix >= params['components'].value:\n self.log.debug(\"Fixing all parameters except amplitudes\")\n for p in params.keys():\n if 'amp' not in p:\n params[p].vary = False\n else:\n self.log.debug(\" no not-masked pixels, skipping\")\n continue\n\n # do the fit\n # if the pixel beam is not valid, then recalculate using the location of the last source to have a valid psf\n if pixbeam is None:\n if src_valid_psf is not None:\n pixbeam = global_data.psfhelper.get_pixbeam(src_valid_psf.ra, src_valid_psf.dec)\n else:\n self.log.critical(\"Cannot determine pixel beam\")\n fac = 1 / np.sqrt(2)\n if self.global_data.docov:\n C = Cmatrix(mx, my, pixbeam.a * FWHM2CC * fac, pixbeam.b * FWHM2CC * fac, pixbeam.pa)\n B = Bmatrix(C)\n else:\n C = B = None\n errs = np.nanmax(rmsimg[int(xmin):int(xmax), int(ymin):int(ymax)])\n result, _ = do_lmfit(idata, params, B=B)\n model = covar_errors(result.params, idata, errs=errs, B=B, C=C)\n\n # convert the results to a source object\n offsets = (xmin, xmax, ymin, ymax)\n # TODO allow for island fluxes in the refitting.\n island_data = IslandFittingData(inum, i=idata, offsets=offsets, doislandflux=False, scalars=(4, 4, None))\n new_src = self.result_to_components(result, model, island_data, src.flags)\n\n\n for ns, s in zip(new_src, included_sources):\n # preserve the uuid so we can do exact matching between catalogs\n ns.uuid = s.uuid\n\n # flag the sources as having been priorized\n ns.flags |= flags.PRIORIZED\n\n # if the position wasn't fit then copy the errors from the input catalog\n if stage < 2:\n ns.err_ra = s.err_ra\n ns.err_dec = s.err_dec\n ns.flags |= flags.FIXED2PSF\n\n # if the shape wasn't fit then copy the errors from the input catalog\n if stage < 3:\n ns.err_a = s.err_a\n ns.err_b = s.err_b\n ns.err_pa = s.err_pa\n sources.extend(new_src)\n return sources"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _fit_island(self, island_data):\n global_data = self.global_data\n\n # global data\n dcurve = global_data.dcurve\n rmsimg = global_data.rmsimg\n\n # island data\n isle_num = island_data.isle_num\n idata = island_data.i\n innerclip, outerclip, max_summits = island_data.scalars\n xmin, xmax, ymin, ymax = island_data.offsets\n\n # get the beam parameters at the center of this island\n midra, middec = global_data.wcshelper.pix2sky([0.5 * (xmax + xmin), 0.5 * (ymax + ymin)])\n beam = global_data.psfhelper.get_psf_pix(midra, middec)\n del middec, midra\n\n icurve = dcurve[xmin:xmax, ymin:ymax]\n rms = rmsimg[xmin:xmax, ymin:ymax]\n\n is_flag = 0\n pixbeam = global_data.psfhelper.get_pixbeam_pixel((xmin + xmax) / 2., (ymin + ymax) / 2.)\n if pixbeam is None:\n # This island is not 'on' the sky, ignore it\n return []\n\n self.log.debug(\"=====\")\n self.log.debug(\"Island ({0})\".format(isle_num))\n\n params = self.estimate_lmfit_parinfo(idata, rms, icurve, beam, innerclip, outerclip, offsets=[xmin, ymin],\n max_summits=max_summits)\n\n # islands at the edge of a region of nans\n # result in no components\n if params is None or params['components'].value < 1:\n return []\n\n self.log.debug(\"Rms is {0}\".format(np.shape(rms)))\n self.log.debug(\"Isle is {0}\".format(np.shape(idata)))\n self.log.debug(\" of which {0} are masked\".format(sum(np.isnan(idata).ravel() * 1)))\n\n # Check that there is enough data to do the fit\n mx, my = np.where(np.isfinite(idata))\n non_blank_pix = len(mx)\n free_vars = len([1 for a in params.keys() if params[a].vary])\n if non_blank_pix < free_vars or free_vars == 0:\n self.log.debug(\"Island {0} doesn't have enough pixels to fit the given model\".format(isle_num))\n self.log.debug(\"non_blank_pix {0}, free_vars {1}\".format(non_blank_pix, free_vars))\n result = DummyLM()\n model = params\n is_flag |= flags.NOTFIT\n else:\n # Model is the fitted parameters\n fac = 1 / np.sqrt(2)\n if self.global_data.docov:\n C = Cmatrix(mx, my, pixbeam.a * FWHM2CC * fac, pixbeam.b * FWHM2CC * fac, pixbeam.pa)\n B = Bmatrix(C)\n else:\n C = B = None\n self.log.debug(\n \"C({0},{1},{2},{3},{4})\".format(len(mx), len(my), pixbeam.a * FWHM2CC, pixbeam.b * FWHM2CC, pixbeam.pa))\n errs = np.nanmax(rms)\n self.log.debug(\"Initial params\")\n self.log.debug(params)\n result, _ = do_lmfit(idata, params, B=B)\n if not result.errorbars:\n is_flag |= flags.FITERR\n # get the real (sky) parameter errors\n model = covar_errors(result.params, idata, errs=errs, B=B, C=C)\n\n if self.global_data.dobias and self.global_data.docov:\n x, y = np.indices(idata.shape)\n acf = elliptical_gaussian(x, y, 1, 0, 0, pixbeam.a * FWHM2CC * fac, pixbeam.b * FWHM2CC * fac,\n pixbeam.pa)\n bias_correct(model, idata, acf=acf * errs ** 2)\n\n if not result.success:\n is_flag |= flags.FITERR\n\n self.log.debug(\"Final params\")\n self.log.debug(model)\n\n # convert the fitting results to a list of sources [and islands]\n sources = self.result_to_components(result, model, island_data, is_flag)\n\n return sources", "response": "This function is called by the island fit method. It is called by the island fit method."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nexecuting fitting on a list of islands and return a list of sources.", "response": "def _fit_islands(self, islands):\n \"\"\"\n Execute fitting on a list of islands\n This function just wraps around fit_island, so that when we do multiprocesing\n a single process will fit multiple islands before returning results.\n\n\n Parameters\n ----------\n islands : list of :class:`AegeanTools.models.IslandFittingData`\n The islands to be fit.\n\n Returns\n -------\n sources : list\n The sources that were fit.\n \"\"\"\n self.log.debug(\"Fitting group of {0} islands\".format(len(islands)))\n sources = []\n for island in islands:\n res = self._fit_island(island)\n sources.extend(res)\n return sources"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_sources_in_image(self, filename, hdu_index=0, outfile=None, rms=None, bkg=None, max_summits=None, innerclip=5,\n outerclip=4, cores=None, rmsin=None, bkgin=None, beam=None, doislandflux=False,\n nopositive=False, nonegative=False, mask=None, lat=None, imgpsf=None, blank=False,\n docov=True, cube_index=None):\n \"\"\"\n Run the Aegean source finder.\n\n\n Parameters\n ----------\n filename : str or HDUList\n Image filename or HDUList.\n\n hdu_index : int\n The index of the FITS HDU (extension).\n\n outfile : str\n file for printing catalog (NOT a table, just a text file of my own design)\n\n rms : float\n Use this rms for the entire image (will also assume that background is 0)\n\n max_summits : int\n Fit up to this many components to each island (extras are included but not fit)\n\n innerclip, outerclip : float\n The seed (inner) and flood (outer) clipping level (sigmas).\n\n cores : int\n Number of CPU cores to use. None means all cores.\n\n rmsin, bkgin : str or HDUList\n Filename or HDUList for the noise and background images.\n If either are None, then it will be calculated internally.\n\n beam : (major, minor, pa)\n Floats representing the synthesised beam (degrees).\n Replaces whatever is given in the FITS header.\n If the FITS header has no BMAJ/BMIN then this is required.\n\n doislandflux : bool\n If True then each island will also be characterized.\n\n nopositive, nonegative : bool\n Whether to return positive or negative sources.\n Default nopositive=False, nonegative=True.\n\n mask : str\n The filename of a region file created by MIMAS.\n Islands outside of this region will be ignored.\n\n lat : float\n The latitude of the telescope (declination of zenith).\n\n imgpsf : str or HDUList\n Filename or HDUList for a psf image.\n\n blank : bool\n Cause the output image to be blanked where islands are found.\n\n docov : bool\n If True then include covariance matrix in the fitting process. (default=True)\n\n cube_index : int\n For image cubes, cube_index determines which slice is used.\n\n Returns\n -------\n sources : list\n List of sources found.\n \"\"\"\n\n # Tell numpy to be quiet\n np.seterr(invalid='ignore')\n if cores is not None:\n if not (cores >= 1): raise AssertionError(\"cores must be one or more\")\n\n self.load_globals(filename, hdu_index=hdu_index, bkgin=bkgin, rmsin=rmsin, beam=beam, rms=rms, bkg=bkg, cores=cores,\n verb=True, mask=mask, lat=lat, psf=imgpsf, blank=blank, docov=docov, cube_index=cube_index)\n global_data = self.global_data\n rmsimg = global_data.rmsimg\n data = global_data.data_pix\n\n self.log.info(\"beam = {0:5.2f}'' x {1:5.2f}'' at {2:5.2f}deg\".format(\n global_data.beam.a * 3600, global_data.beam.b * 3600, global_data.beam.pa))\n # stop people from doing silly things.\n if outerclip > innerclip:\n outerclip = innerclip\n self.log.info(\"seedclip={0}\".format(innerclip))\n self.log.info(\"floodclip={0}\".format(outerclip))\n\n isle_num = 0\n\n if cores == 1: # single-threaded, no parallel processing\n queue = []\n else:\n queue = pprocess.Queue(limit=cores, reuse=1)\n fit_parallel = queue.manage(pprocess.MakeReusable(self._fit_islands))\n\n island_group = []\n group_size = 20\n for i, xmin, xmax, ymin, ymax in self._gen_flood_wrap(data, rmsimg, innerclip, outerclip, domask=True):\n # ignore empty islands\n # This should now be impossible to trigger\n if np.size(i) < 1:\n self.log.warn(\"Empty island detected, this should be imposisble.\")\n continue\n isle_num += 1\n scalars = (innerclip, outerclip, max_summits)\n offsets = (xmin, xmax, ymin, ymax)\n island_data = IslandFittingData(isle_num, i, scalars, offsets, doislandflux)\n # If cores==1 run fitting in main process. Otherwise build up groups of islands\n # and submit to queue for subprocesses. Passing a group of islands is more\n # efficient than passing single islands to the subprocesses.\n if cores == 1:\n res = self._fit_island(island_data)\n queue.append(res)\n else:\n island_group.append(island_data)\n # If the island group is full queue it for the subprocesses to fit\n if len(island_group) >= group_size:\n fit_parallel(island_group)\n island_group = []\n\n # The last partially-filled island group also needs to be queued for fitting\n if len(island_group) > 0:\n fit_parallel(island_group)\n\n # Write the output to the output file\n if outfile:\n print(header.format(\"{0}-({1})\".format(__version__, __date__), filename), file=outfile)\n print(OutputSource.header, file=outfile)\n\n sources = []\n for srcs in queue:\n if srcs: # ignore empty lists\n for src in srcs:\n # ignore sources that we have been told to ignore\n if (src.peak_flux > 0 and nopositive) or (src.peak_flux < 0 and nonegative):\n continue\n sources.append(src)\n if outfile:\n print(str(src), file=outfile)\n self.sources.extend(sources)\n return sources", "response": "This function will find sources in an image."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntaking an input catalog, and image, and optional background/noise images fit the flux and ra/dec for each of the given sources, keeping the morphology fixed if doregroup is true the groups will be recreated based on a matching radius/probability. if doregroup is false then the islands of the input catalog will be preserved. Multiple cores can be specified, and will be used. Parameters ---------- filename : str or HDUList Image filename or HDUList. catalogue : str or list Input catalogue file name or list of OutputSource objects. hdu_index : int The index of the FITS HDU (extension). outfile : str file for printing catalog (NOT a table, just a text file of my own design) rmsin, bkgin : str or HDUList Filename or HDUList for the noise and background images. If either are None, then it will be calculated internally. cores : int Number of CPU cores to use. None means all cores. rms : float Use this rms for the entire image (will also assume that background is 0) beam : (major, minor, pa) Floats representing the synthesised beam (degrees). Replaces whatever is given in the FITS header. If the FITS header has no BMAJ/BMIN then this is required. lat : float The latitude of the telescope (declination of zenith). imgpsf : str or HDUList Filename or HDUList for a psf image. catpsf : str or HDUList Filename or HDUList for the catalogue psf image. stage : int Refitting stage ratio : float If not None - ratio of image psf to catalog psf, otherwise interpret from catalogue or image if possible innerclip, outerclip : float The seed (inner) and flood (outer) clipping level (sigmas). docov : bool If True then include covariance matrix in the fitting process. (default=True) cube_index : int For image cubes, slice determines which slice is used. Returns ------- sources : list List of sources measured.", "response": "def priorized_fit_islands(self, filename, catalogue, hdu_index=0, outfile=None, bkgin=None, rmsin=None, cores=1,\n rms=None, bkg=None, beam=None, lat=None, imgpsf=None, catpsf=None, stage=3, ratio=None, outerclip=3,\n doregroup=True, docov=True, cube_index=None):\n \"\"\"\n Take an input catalog, and image, and optional background/noise images\n fit the flux and ra/dec for each of the given sources, keeping the morphology fixed\n\n if doregroup is true the groups will be recreated based on a matching radius/probability.\n if doregroup is false then the islands of the input catalog will be preserved.\n\n Multiple cores can be specified, and will be used.\n\n\n Parameters\n ----------\n filename : str or HDUList\n Image filename or HDUList.\n\n catalogue : str or list\n Input catalogue file name or list of OutputSource objects.\n\n hdu_index : int\n The index of the FITS HDU (extension).\n\n outfile : str\n file for printing catalog (NOT a table, just a text file of my own design)\n\n rmsin, bkgin : str or HDUList\n Filename or HDUList for the noise and background images.\n If either are None, then it will be calculated internally.\n\n cores : int\n Number of CPU cores to use. None means all cores.\n\n rms : float\n Use this rms for the entire image (will also assume that background is 0)\n\n beam : (major, minor, pa)\n Floats representing the synthesised beam (degrees).\n Replaces whatever is given in the FITS header.\n If the FITS header has no BMAJ/BMIN then this is required.\n\n lat : float\n The latitude of the telescope (declination of zenith).\n\n imgpsf : str or HDUList\n Filename or HDUList for a psf image.\n\n catpsf : str or HDUList\n Filename or HDUList for the catalogue psf image.\n\n stage : int\n Refitting stage\n\n ratio : float\n If not None - ratio of image psf to catalog psf, otherwise interpret from catalogue or image if possible\n\n innerclip, outerclip : float\n The seed (inner) and flood (outer) clipping level (sigmas).\n\n docov : bool\n If True then include covariance matrix in the fitting process. (default=True)\n\n cube_index : int\n For image cubes, slice determines which slice is used.\n\n\n Returns\n -------\n sources : list\n List of sources measured.\n\n \"\"\"\n\n from AegeanTools.cluster import regroup\n\n self.load_globals(filename, hdu_index=hdu_index, bkgin=bkgin, rmsin=rmsin, rms=rms, bkg=bkg, cores=cores, verb=True,\n do_curve=False, beam=beam, lat=lat, psf=imgpsf, docov=docov, cube_index=cube_index)\n\n global_data = self.global_data\n far = 10 * global_data.beam.a # degrees\n # load the table and convert to an input source list\n if isinstance(catalogue, six.string_types):\n input_table = load_table(catalogue)\n input_sources = np.array(table_to_source_list(input_table))\n else:\n input_sources = np.array(catalogue)\n\n if len(input_sources) < 1:\n self.log.debug(\"No input sources for priorized fitting\")\n return []\n\n # reject sources with missing params\n ok = True\n for param in ['ra', 'dec', 'peak_flux', 'a', 'b', 'pa']:\n if np.isnan(getattr(input_sources[0], param)):\n self.log.info(\"Source 0, is missing param '{0}'\".format(param))\n ok = False\n if not ok:\n self.log.error(\"Missing parameters! Not fitting.\")\n self.log.error(\"Maybe your table is missing or mis-labeled columns?\")\n return []\n del ok\n\n src_mask = np.ones(len(input_sources), dtype=bool)\n\n # check to see if the input catalog contains psf information\n has_psf = getattr(input_sources[0], 'psf_a', None) is not None\n\n # the input sources are the initial conditions for our fits.\n # Expand each source size if needed.\n\n # If ratio is provided we just the psf by this amount\n if ratio is not None:\n self.log.info(\"Using ratio of {0} to scale input source shapes\".format(ratio))\n far *= ratio\n for i, src in enumerate(input_sources):\n # Sources with an unknown psf are rejected as they are either outside the image\n # or outside the region covered by the psf\n skybeam = global_data.psfhelper.get_beam(src.ra, src.dec)\n if skybeam is None:\n src_mask[i] = False\n self.log.info(\"Excluding source ({0.island},{0.source}) due to lack of psf knowledge\".format(src))\n continue\n # the new source size is the previous size, convolved with the expanded psf\n src.a = np.sqrt(src.a ** 2 + (skybeam.a * 3600) ** 2 * (1 - 1 / ratio ** 2))\n src.b = np.sqrt(src.b ** 2 + (skybeam.b * 3600) ** 2 * (1 - 1 / ratio ** 2))\n # source with funky a/b are also rejected\n if not np.all(np.isfinite((src.a, src.b))):\n self.log.info(\"Excluding source ({0.island},{0.source}) due to funky psf ({0.a},{0.b},{0.pa})\".format(src))\n src_mask[i] = False\n\n # if we know the psf from the input catalogue (has_psf), or if it was provided via a psf map\n # then we use that psf.\n elif catpsf is not None or has_psf:\n if catpsf is not None:\n self.log.info(\"Using catalog PSF from {0}\".format(catpsf))\n psf_helper = PSFHelper(catpsf, None) # might need to set the WCSHelper to be not None\n else:\n self.log.info(\"Using catalog PSF from input catalog\")\n psf_helper = None\n for i, src in enumerate(input_sources):\n if (src.psf_a <=0) or (src.psf_b <=0):\n src_mask[i] = False\n self.log.info(\"Excluding source ({0.island},{0.source}) due to psf_a/b <=0\".format(src))\n continue\n if has_psf:\n catbeam = Beam(src.psf_a / 3600, src.psf_b / 3600, src.psf_pa)\n else:\n catbeam = psf_helper.get_beam(src.ra, src.dec)\n imbeam = global_data.psfhelper.get_beam(src.ra, src.dec)\n # If either of the above are None then we skip this source.\n if catbeam is None or imbeam is None:\n src_mask[i] = False\n self.log.info(\"Excluding source ({0.island},{0.source}) due to lack of psf knowledge\".format(src))\n continue\n\n # TODO: The following assumes that the various psf's are scaled versions of each other\n # and makes no account for differing position angles. This needs to be checked and/or addressed.\n\n # deconvolve the source shape from the catalogue psf\n src.a = (src.a / 3600) ** 2 - catbeam.a ** 2 + imbeam.a ** 2 # degrees\n\n # clip the minimum source shape to be the image psf\n if src.a < 0:\n src.a = imbeam.a * 3600 # arcsec\n else:\n src.a = np.sqrt(src.a) * 3600 # arcsec\n\n src.b = (src.b / 3600) ** 2 - catbeam.b ** 2 + imbeam.b ** 2\n if src.b < 0:\n src.b = imbeam.b * 3600 # arcsec\n else:\n src.b = np.sqrt(src.b) * 3600 # arcsec\n else:\n self.log.info(\"Not scaling input source sizes\")\n\n self.log.info(\"{0} sources in catalog\".format(len(input_sources)))\n self.log.info(\"{0} sources accepted\".format(sum(src_mask)))\n\n if len(src_mask) < 1:\n self.log.debug(\"No sources accepted for priorized fitting\")\n return []\n\n input_sources = input_sources[src_mask]\n # redo the grouping if required\n if doregroup:\n groups = regroup(input_sources, eps=np.sqrt(2), far=far)\n else:\n groups = list(island_itergen(input_sources))\n\n if cores == 1: # single-threaded, no parallel processing\n queue = []\n else:\n queue = pprocess.Queue(limit=cores, reuse=1)\n fit_parallel = queue.manage(pprocess.MakeReusable(self._refit_islands))\n\n sources = []\n island_group = []\n group_size = 20\n\n for i, island in enumerate(groups):\n island_group.append(island)\n # If the island group is full queue it for the subprocesses to fit\n if len(island_group) >= group_size:\n if cores > 1:\n fit_parallel(island_group, stage, outerclip, istart=i)\n else:\n res = self._refit_islands(island_group, stage, outerclip, istart=i)\n queue.append(res)\n island_group = []\n\n # The last partially-filled island group also needs to be queued for fitting\n if len(island_group) > 0:\n if cores > 1:\n fit_parallel(island_group, stage, outerclip, istart=i)\n else:\n res = self._refit_islands(island_group, stage, outerclip, istart=i)\n queue.append(res)\n\n # now unpack the fitting results in to a list of sources\n for s in queue:\n sources.extend(s)\n\n sources = sorted(sources)\n\n # Write the output to the output file\n if outfile:\n print(header.format(\"{0}-({1})\".format(__version__, __date__), filename), file=outfile)\n print(OutputSource.header, file=outfile)\n\n components = 0\n for source in sources:\n if isinstance(source, OutputSource):\n components += 1\n if outfile:\n print(str(source), file=outfile)\n\n self.log.info(\"fit {0} components\".format(components))\n self.sources.extend(sources)\n return sources"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_table_formats(files):\n cont = True\n formats = get_table_formats()\n for t in files.split(','):\n _, ext = os.path.splitext(t)\n ext = ext[1:].lower()\n if ext not in formats:\n cont = False\n log.warn(\"Format not supported for {0} ({1})\".format(t, ext))\n if not cont:\n log.error(\"Invalid table format specified.\")\n return cont", "response": "Checks if a list of files are of a recognizable output type."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting a list of all the file formats that are supported for writing.", "response": "def show_formats():\n \"\"\"\n Print a list of all the file formats that are supported for writing.\n The file formats are determined by their extensions.\n\n Returns\n -------\n None\n \"\"\"\n fmts = {\n \"ann\": \"Kvis annotation\",\n \"reg\": \"DS9 regions file\",\n \"fits\": \"FITS Binary Table\",\n \"csv\": \"Comma separated values\",\n \"tab\": \"tabe separated values\",\n \"tex\": \"LaTeX table format\",\n \"html\": \"HTML table\",\n \"vot\": \"VO-Table\",\n \"xml\": \"VO-Table\",\n \"db\": \"Sqlite3 database\",\n \"sqlite\": \"Sqlite3 database\"}\n supported = get_table_formats()\n print(\"Extension | Description | Supported?\")\n for k in sorted(fmts.keys()):\n print(\"{0:10s} {1:24s} {2}\".format(k, fmts[k], k in supported))\n return"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_meta_data(meta=None):\n if meta is None:\n meta = {}\n if 'DATE' not in meta:\n meta['DATE'] = strftime(\"%Y-%m-%d %H:%M:%S\", gmtime())\n if 'PROGRAM' not in meta:\n meta['PROGRAM'] = \"AegeanTools.catalogs\"\n meta['PROGVER'] = \"{0}-({1})\".format(__version__, __date__)\n return meta", "response": "Modify the metadata dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_catalog(filename, catalog, meta=None, prefix=None):\n ascii_table_formats = {'csv': 'csv', 'tab': 'tab', 'tex': 'latex', 'html': 'html'}\n # .ann and .reg are handled by me\n meta = update_meta_data(meta)\n extension = os.path.splitext(filename)[1][1:].lower()\n if extension in ['ann', 'reg']:\n writeAnn(filename, catalog, extension)\n elif extension in ['db', 'sqlite']:\n writeDB(filename, catalog, meta)\n elif extension in ['hdf5', 'fits', 'vo', 'vot', 'xml']:\n write_catalog(filename, catalog, extension, meta, prefix=prefix)\n elif extension in ascii_table_formats.keys():\n write_catalog(filename, catalog, fmt=ascii_table_formats[extension], meta=meta, prefix=prefix)\n else:\n log.warning(\"extension not recognised {0}\".format(extension))\n log.warning(\"You get tab format\")\n write_catalog(filename, catalog, fmt='tab', prefix=prefix)\n return", "response": "Save a catalogue of sources using filename as a model."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads a catalogue and extract the source positions", "response": "def load_catalog(filename):\n \"\"\"\n Load a catalogue and extract the source positions (only)\n\n Parameters\n ----------\n filename : str\n Filename to read. Supported types are csv, tab, tex, vo, vot, and xml.\n\n Returns\n -------\n catalogue : list\n A list of [ (ra, dec), ...]\n\n \"\"\"\n supported = get_table_formats()\n\n fmt = os.path.splitext(filename)[-1][1:].lower() # extension sans '.'\n\n if fmt in ['csv', 'tab', 'tex'] and fmt in supported:\n log.info(\"Reading file {0}\".format(filename))\n t = ascii.read(filename)\n catalog = list(zip(t.columns['ra'], t.columns['dec']))\n\n elif fmt in ['vo', 'vot', 'xml'] and fmt in supported:\n log.info(\"Reading file {0}\".format(filename))\n t = parse_single_table(filename)\n catalog = list(zip(t.array['ra'].tolist(), t.array['dec'].tolist()))\n\n else:\n log.info(\"Assuming ascii format, reading first two columns\")\n lines = [a.strip().split() for a in open(filename, 'r').readlines() if not a.startswith('#')]\n try:\n catalog = [(float(a[0]), float(a[1])) for a in lines]\n except:\n log.error(\"Expecting two columns of floats but failed to parse\")\n log.error(\"Catalog file {0} not loaded\".format(filename))\n raise Exception(\"Could not determine file format\")\n\n return catalog"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_table(filename):\n supported = get_table_formats()\n\n fmt = os.path.splitext(filename)[-1][1:].lower() # extension sans '.'\n\n if fmt in ['csv', 'tab', 'tex'] and fmt in supported:\n log.info(\"Reading file {0}\".format(filename))\n t = ascii.read(filename)\n elif fmt in ['vo', 'vot', 'xml', 'fits', 'hdf5'] and fmt in supported:\n log.info(\"Reading file {0}\".format(filename))\n t = Table.read(filename)\n else:\n log.error(\"Table format not recognized or supported\")\n log.error(\"{0} [{1}]\".format(filename, fmt))\n raise Exception(\"Table format not recognized or supported\")\n return t", "response": "Load a table from a given file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting a table to a file.", "response": "def write_table(table, filename):\n \"\"\"\n Write a table to a file.\n\n Parameters\n ----------\n table : Table\n Table to be written\n\n filename : str\n Destination for saving table.\n\n Returns\n -------\n None\n \"\"\"\n try:\n if os.path.exists(filename):\n os.remove(filename)\n table.write(filename)\n log.info(\"Wrote {0}\".format(filename))\n except Exception as e:\n if \"Format could not be identified\" not in e.message:\n raise e\n else:\n fmt = os.path.splitext(filename)[-1][1:].lower() # extension sans '.'\n raise Exception(\"Cannot auto-determine format for {0}\".format(fmt))\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef table_to_source_list(table, src_type=OutputSource):\n source_list = []\n if table is None:\n return source_list\n\n for row in table:\n # Initialise our object\n src = src_type()\n # look for the columns required by our source object\n for param in src_type.names:\n if param in table.colnames:\n # copy the value to our object\n val = row[param]\n # hack around float32's broken-ness\n if isinstance(val, np.float32):\n val = np.float64(val)\n setattr(src, param, val)\n # save this object to our list of sources\n source_list.append(src)\n return source_list", "response": "Convert a table of data into a list of sources."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write_catalog(filename, catalog, fmt=None, meta=None, prefix=None):\n if meta is None:\n meta = {}\n\n if prefix is None:\n pre=''\n else:\n pre = prefix + '_'\n\n def writer(filename, catalog, fmt=None):\n \"\"\"\n construct a dict of the data\n this method preserves the data types in the VOTable\n \"\"\"\n tab_dict = {}\n name_list = []\n for name in catalog[0].names:\n col_name = name\n if catalog[0].galactic:\n if name.startswith('ra'):\n col_name = 'lon'+name[2:]\n elif name.endswith('ra'):\n col_name = name[:-2] + 'lon'\n elif name.startswith('dec'):\n col_name = 'lat'+name[3:]\n elif name.endswith('dec'):\n col_name = name[:-3] + 'lat'\n col_name = pre + col_name\n tab_dict[col_name] = [getattr(c, name, None) for c in catalog]\n name_list.append(col_name)\n t = Table(tab_dict, meta=meta)\n # re-order the columns\n t = t[[n for n in name_list]]\n\n if fmt is not None:\n if fmt in [\"vot\", \"vo\", \"xml\"]:\n vot = from_table(t)\n # description of this votable\n vot.description = repr(meta)\n writetoVO(vot, filename)\n elif fmt in ['hdf5']:\n t.write(filename, path='data', overwrite=True)\n elif fmt in ['fits']:\n writeFITSTable(filename, t)\n else:\n ascii.write(t, filename, fmt, overwrite=True)\n else:\n ascii.write(t, filename, overwrite=True)\n return\n\n # sort the sources into types and then write them out individually\n components, islands, simples = classify_catalog(catalog)\n\n if len(components) > 0:\n new_name = \"{1}{0}{2}\".format('_comp', *os.path.splitext(filename))\n writer(new_name, components, fmt)\n log.info(\"wrote {0}\".format(new_name))\n if len(islands) > 0:\n new_name = \"{1}{0}{2}\".format('_isle', *os.path.splitext(filename))\n writer(new_name, islands, fmt)\n log.info(\"wrote {0}\".format(new_name))\n if len(simples) > 0:\n new_name = \"{1}{0}{2}\".format('_simp', *os.path.splitext(filename))\n writer(new_name, simples, fmt)\n log.info(\"wrote {0}\".format(new_name))\n return", "response": "Writes a list of sources to a file with format determined by extension."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a table into a FITSTable and then write to disk.", "response": "def writeFITSTable(filename, table):\n \"\"\"\n Convert a table into a FITSTable and then write to disk.\n\n Parameters\n ----------\n filename : str\n Filename to write.\n\n table : Table\n Table to write.\n\n Returns\n -------\n None\n\n Notes\n -----\n Due to a bug in numpy, `int32` and `float32` are converted to `int64` and `float64` before writing.\n \"\"\"\n def FITSTableType(val):\n \"\"\"\n Return the FITSTable type corresponding to each named parameter in obj\n \"\"\"\n if isinstance(val, bool):\n types = \"L\"\n elif isinstance(val, (int, np.int64, np.int32)):\n types = \"J\"\n elif isinstance(val, (float, np.float64, np.float32)):\n types = \"E\"\n elif isinstance(val, six.string_types):\n types = \"{0}A\".format(len(val))\n else:\n log.warning(\"Column {0} is of unknown type {1}\".format(val, type(val)))\n log.warning(\"Using 5A\")\n types = \"5A\"\n return types\n\n cols = []\n for name in table.colnames:\n cols.append(fits.Column(name=name, format=FITSTableType(table[name][0]), array=table[name]))\n cols = fits.ColDefs(cols)\n tbhdu = fits.BinTableHDU.from_columns(cols)\n for k in table.meta:\n tbhdu.header['HISTORY'] = ':'.join((k, table.meta[k]))\n tbhdu.writeto(filename, overwrite=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef writeIslandContours(filename, catalog, fmt='reg'):\n if fmt != 'reg':\n log.warning(\"Format {0} not yet supported\".format(fmt))\n log.warning(\"not writing anything\")\n return\n\n out = open(filename, 'w')\n print(\"#Aegean island contours\", file=out)\n print(\"#AegeanTools.catalogs version {0}-({1})\".format(__version__, __date__), file=out)\n line_fmt = 'image;line({0},{1},{2},{3})'\n text_fmt = 'fk5; text({0},{1}) # text={{{2}}}'\n mas_fmt = 'image; line({1},{0},{3},{2}) #color = yellow'\n x_fmt = 'image; point({1},{0}) # point=x'\n for c in catalog:\n contour = c.contour\n if len(contour) > 1:\n for p1, p2 in zip(contour[:-1], contour[1:]):\n print(line_fmt.format(p1[1] + 0.5, p1[0] + 0.5, p2[1] + 0.5, p2[0] + 0.5), file=out)\n print(line_fmt.format(contour[-1][1] + 0.5, contour[-1][0] + 0.5, contour[0][1] + 0.5,\n contour[0][0] + 0.5), file=out)\n # comment out lines that have invalid ra/dec (WCS problems)\n if np.nan in [c.ra, c.dec]:\n print('#', end=' ', file=out)\n # some islands may not have anchors because they don't have any contours\n if len(c.max_angular_size_anchors) == 4:\n print(text_fmt.format(c.ra, c.dec, c.island), file=out)\n print(mas_fmt.format(*[a + 0.5 for a in c.max_angular_size_anchors]), file=out)\n for p1, p2 in c.pix_mask:\n # DS9 uses 1-based instead of 0-based indexing\n print(x_fmt.format(p1 + 1, p2 + 1), file=out)\n out.close()\n return", "response": "Writes an output file in ds9. reg format that outlines the boundaries of each island."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef writeIslandBoxes(filename, catalog, fmt):\n if fmt not in ['reg', 'ann']:\n log.warning(\"Format not supported for island boxes{0}\".format(fmt))\n return # fmt not supported\n\n out = open(filename, 'w')\n print(\"#Aegean Islands\", file=out)\n print(\"#Aegean version {0}-({1})\".format(__version__, __date__), file=out)\n\n if fmt == 'reg':\n print(\"IMAGE\", file=out)\n box_fmt = 'box({0},{1},{2},{3}) #{4}'\n else:\n print(\"COORD P\", file=out)\n box_fmt = 'box P {0} {1} {2} {3} #{4}'\n\n for c in catalog:\n # x/y swap for pyfits/numpy translation\n ymin, ymax, xmin, xmax = c.extent\n # +1 for array/image offset\n xcen = (xmin + xmax) / 2.0 + 1\n # + 0.5 in each direction to make lines run 'between' DS9 pixels\n xwidth = xmax - xmin + 1\n ycen = (ymin + ymax) / 2.0 + 1\n ywidth = ymax - ymin + 1\n print(box_fmt.format(xcen, ycen, xwidth, ywidth, c.island), file=out)\n out.close()\n return", "response": "Writes an output file in ds9. reg or kvis. ann format that contains bounding boxes for all the islands."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting an annotation file that can be read by Kvis or DS9.", "response": "def writeAnn(filename, catalog, fmt):\n \"\"\"\n Write an annotation file that can be read by Kvis (.ann) or DS9 (.reg).\n Uses ra/dec from catalog.\n Draws ellipses if bmaj/bmin/pa are in catalog. Draws 30\" circles otherwise.\n\n Only :class:`AegeanTools.models.OutputSource` will appear in the annotation file\n unless there are none, in which case :class:`AegeanTools.models.SimpleSource` (if present)\n will be written. If any :class:`AegeanTools.models.IslandSource` objects are present then\n an island contours file will be written.\n\n Parameters\n ----------\n filename : str\n Output filename base.\n\n catalog : list\n List of sources.\n\n fmt : ['ann', 'reg']\n Output file type.\n\n Returns\n -------\n None\n\n See Also\n --------\n AegeanTools.catalogs.writeIslandContours\n \"\"\"\n if fmt not in ['reg', 'ann']:\n log.warning(\"Format not supported for island boxes{0}\".format(fmt))\n return # fmt not supported\n\n components, islands, simples = classify_catalog(catalog)\n if len(components) > 0:\n cat = sorted(components)\n suffix = \"comp\"\n elif len(simples) > 0:\n cat = simples\n suffix = \"simp\"\n else:\n cat = []\n\n if len(cat) > 0:\n ras = [a.ra for a in cat]\n decs = [a.dec for a in cat]\n if not hasattr(cat[0], 'a'): # a being the variable that I used for bmaj.\n bmajs = [30 / 3600.0 for a in cat]\n bmins = bmajs\n pas = [0 for a in cat]\n else:\n bmajs = [a.a / 3600.0 for a in cat]\n bmins = [a.b / 3600.0 for a in cat]\n pas = [a.pa for a in cat]\n\n names = [a.__repr__() for a in cat]\n if fmt == 'ann':\n new_file = re.sub('.ann$', '_{0}.ann'.format(suffix), filename)\n out = open(new_file, 'w')\n print(\"#Aegean version {0}-({1})\".format(__version__, __date__), file=out)\n print('PA SKY', file=out)\n print('FONT hershey12', file=out)\n print('COORD W', file=out)\n formatter = \"ELLIPSE W {0} {1} {2} {3} {4:+07.3f} #{5}\\nTEXT W {0} {1} {5}\"\n else: # reg\n new_file = re.sub('.reg$', '_{0}.reg'.format(suffix), filename)\n out = open(new_file, 'w')\n print(\"#Aegean version {0}-({1})\".format(__version__, __date__), file=out)\n print(\"fk5\", file=out)\n formatter = 'ellipse {0} {1} {2:.9f}d {3:.9f}d {4:+07.3f}d # text=\"{5}\"'\n # DS9 has some strange ideas about position angle\n pas = [a - 90 for a in pas]\n\n for ra, dec, bmaj, bmin, pa, name in zip(ras, decs, bmajs, bmins, pas, names):\n # comment out lines that have invalid or stupid entries\n if np.nan in [ra, dec, bmaj, bmin, pa] or bmaj >= 180:\n print('#', end=' ', file=out)\n print(formatter.format(ra, dec, bmaj, bmin, pa, name), file=out)\n out.close()\n log.info(\"wrote {0}\".format(new_file))\n if len(islands) > 0:\n if fmt == 'reg':\n new_file = re.sub('.reg$', '_isle.reg', filename)\n elif fmt == 'ann':\n log.warning('kvis islands are currently not working')\n return\n else:\n log.warning('format {0} not supported for island annotations'.format(fmt))\n return\n writeIslandContours(new_file, islands, fmt)\n log.info(\"wrote {0}\".format(new_file))\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef writeDB(filename, catalog, meta=None):\n\n def sqlTypes(obj, names):\n \"\"\"\n Return the sql type corresponding to each named parameter in obj\n \"\"\"\n types = []\n for n in names:\n val = getattr(obj, n)\n if isinstance(val, bool):\n types.append(\"BOOL\")\n elif isinstance(val, (int, np.int64, np.int32)):\n types.append(\"INT\")\n elif isinstance(val, (float, np.float64, np.float32)): # float32 is bugged and claims not to be a float\n types.append(\"FLOAT\")\n elif isinstance(val, six.string_types):\n types.append(\"VARCHAR\")\n else:\n log.warning(\"Column {0} is of unknown type {1}\".format(n, type(n)))\n log.warning(\"Using VARCHAR\")\n types.append(\"VARCHAR\")\n return types\n\n if os.path.exists(filename):\n log.warning(\"overwriting {0}\".format(filename))\n os.remove(filename)\n conn = sqlite3.connect(filename)\n db = conn.cursor()\n # determine the column names by inspecting the catalog class\n for t, tn in zip(classify_catalog(catalog), [\"components\", \"islands\", \"simples\"]):\n if len(t) < 1:\n continue #don't write empty tables\n col_names = t[0].names\n col_types = sqlTypes(t[0], col_names)\n stmnt = ','.join([\"{0} {1}\".format(a, b) for a, b in zip(col_names, col_types)])\n db.execute('CREATE TABLE {0} ({1})'.format(tn, stmnt))\n stmnt = 'INSERT INTO {0} ({1}) VALUES ({2})'.format(tn, ','.join(col_names), ','.join(['?' for i in col_names]))\n # expend the iterators that are created by python 3+\n data = list(map(nulls, list(r.as_list() for r in t)))\n db.executemany(stmnt, data)\n log.info(\"Created table {0}\".format(tn))\n # metadata add some meta data\n db.execute(\"CREATE TABLE meta (key VARCHAR, val VARCHAR)\")\n for k in meta:\n db.execute(\"INSERT INTO meta (key, val) VALUES (?,?)\", (k, meta[k]))\n conn.commit()\n log.info(db.execute(\"SELECT name FROM sqlite_master WHERE type='table';\").fetchall())\n conn.close()\n log.info(\"Wrote file {0}\".format(filename))\n return", "response": "Writes a sqlite3 database containing one table for each source type and each source type in the catalog."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the normalised distance between two sources.", "response": "def norm_dist(src1, src2):\n \"\"\"\n Calculate the normalised distance between two sources.\n Sources are elliptical Gaussians.\n\n The normalised distance is calculated as the GCD distance between the centers,\n divided by quadrature sum of the radius of each ellipse along a line joining the two ellipses.\n\n For ellipses that touch at a single point, the normalized distance will be 1/sqrt(2).\n\n Parameters\n ----------\n src1, src2 : object\n The two positions to compare. Objects must have the following parameters: (ra, dec, a, b, pa).\n\n Returns\n -------\n dist: float\n The normalised distance.\n\n \"\"\"\n if np.all(src1 == src2):\n return 0\n dist = gcd(src1.ra, src1.dec, src2.ra, src2.dec) # degrees\n\n # the angle between the ellipse centers\n phi = bear(src1.ra, src1.dec, src2.ra, src2.dec) # Degrees\n # Calculate the radius of each ellipse along a line that joins their centers.\n r1 = src1.a*src1.b / np.hypot(src1.a * np.sin(np.radians(phi - src1.pa)),\n src1.b * np.cos(np.radians(phi - src1.pa)))\n r2 = src2.a*src2.b / np.hypot(src2.a * np.sin(np.radians(180 + phi - src2.pa)),\n src2.b * np.cos(np.radians(180 + phi - src2.pa)))\n R = dist / (np.hypot(r1, r2) / 3600)\n return R"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sky_dist(src1, src2):\n\n if np.all(src1 == src2):\n return 0\n return gcd(src1.ra, src1.dec, src2.ra, src2.dec)", "response": "Returns the approximate circle distance between two sources."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndoing a pairwise comparison of all sources and determine if they have a normalized distance within eps.", "response": "def pairwise_ellpitical_binary(sources, eps, far=None):\n \"\"\"\n Do a pairwise comparison of all sources and determine if they have a normalized distance within\n eps.\n\n Form this into a matrix of shape NxN.\n\n\n Parameters\n ----------\n sources : list\n A list of sources (objects with parameters: ra,dec,a,b,pa)\n\n eps : float\n Normalised distance constraint.\n\n far : float\n If sources have a dec that differs by more than this amount then they are considered to be not matched.\n This is a short-cut around performing GCD calculations.\n\n Returns\n -------\n prob : numpy.ndarray\n A 2d array of True/False.\n\n See Also\n --------\n :func:`AegeanTools.cluster.norm_dist`\n \"\"\"\n if far is None:\n far = max(a.a/3600 for a in sources)\n l = len(sources)\n distances = np.zeros((l, l), dtype=bool)\n for i in range(l):\n for j in range(i, l):\n if i == j:\n distances[i, j] = False\n continue\n src1 = sources[i]\n src2 = sources[j]\n if src2.dec - src1.dec > far:\n break\n if abs(src2.ra - src1.ra)*np.cos(np.radians(src1.dec)) > far:\n continue\n distances[i, j] = norm_dist(src1, src2) > eps\n distances[j, i] = distances[i, j]\n return distances"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of islands that are vectorized within a catalog.", "response": "def regroup_vectorized(srccat, eps, far=None, dist=norm_dist):\n \"\"\"\n Regroup the islands of a catalog according to their normalised distance.\n\n Assumes srccat is recarray-like for efficiency.\n Return a list of island groups.\n\n Parameters\n ----------\n srccat : np.rec.arry or pd.DataFrame\n Should have the following fields[units]:\n ra[deg],dec[deg], a[arcsec],b[arcsec],pa[deg], peak_flux[any]\n eps : float\n maximum normalised distance within which sources are considered to be\n grouped\n far : float\n (degrees) sources that are further than this distance apart will not\n be grouped, and will not be tested.\n Default = 0.5.\n dist : func\n a function that calculates the distance between a source and each\n element of an array of sources.\n Default = :func:`AegeanTools.cluster.norm_dist`\n\n Returns\n -------\n islands : list of lists\n Each island contians integer indices for members from srccat\n (in descending dec order).\n \"\"\"\n if far is None:\n far = 0.5 # 10*max(a.a/3600 for a in srccat)\n\n # most negative declination first\n # XXX: kind='mergesort' ensures stable sorting for determinism.\n # Do we need this?\n order = np.argsort(srccat.dec, kind='mergesort')[::-1]\n # TODO: is it better to store groups as arrays even if appends are more\n # costly?\n groups = [[order[0]]]\n for idx in order[1:]:\n rec = srccat[idx]\n # TODO: Find out if groups are big enough for this to give us a speed\n # gain. If not, get distance to all entries in groups above\n # decmin simultaneously.\n decmin = rec.dec - far\n for group in reversed(groups):\n # when an island's largest (last) declination is smaller than\n # decmin, we don't need to look at any more islands\n if srccat.dec[group[-1]] < decmin:\n # new group\n groups.append([idx])\n rafar = far / np.cos(np.radians(rec.dec))\n group_recs = np.take(srccat, group, mode='clip')\n group_recs = group_recs[abs(rec.ra - group_recs.ra) <= rafar]\n if len(group_recs) and dist(rec, group_recs).min() < eps:\n group.append(idx)\n break\n else:\n # new group\n groups.append([idx])\n\n # TODO?: a more numpy-like interface would return only an array providing\n # the mapping:\n # group_idx = np.empty(len(srccat), dtype=int)\n # for i, group in enumerate(groups):\n # group_idx[group] = i\n # return group_idx\n return groups"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef regroup(catalog, eps, far=None, dist=norm_dist):\n\n if isinstance(catalog, str):\n table = load_table(catalog)\n srccat = table_to_source_list(table)\n else:\n try:\n srccat = catalog\n _ = catalog[0].ra, catalog[0].dec, catalog[0].a, catalog[0].b, catalog[0].pa, catalog[0].peak_flux\n\n except AttributeError as e:\n log.error(\"catalog is not understood.\")\n log.error(\"catalog: Should be a list of objects with the following properties[units]:\\n\" +\n \"ra[deg],dec[deg], a[arcsec],b[arcsec],pa[deg], peak_flux[any]\")\n raise e\n\n log.info(\"Regrouping islands within catalog\")\n log.debug(\"Calculating distances\")\n\n if far is None:\n far = 0.5 # 10*max(a.a/3600 for a in srccat)\n\n srccat_array = np.rec.fromrecords(\n [(s.ra, s.dec, s.a, s.b, s.pa, s.peak_flux)\n for s in srccat],\n names=['ra', 'dec', 'a', 'b', 'pa', 'peak_flux'])\n groups = regroup_vectorized(srccat_array, eps=eps, far=far, dist=dist)\n groups = [[srccat[idx] for idx in group]\n for group in groups]\n\n islands = []\n # now that we have the groups, we relabel the sources to have (island,component) in flux order\n # note that the order of sources within an island list is not changed - just their labels\n for isle, group in enumerate(groups):\n for comp, src in enumerate(sorted(group, key=lambda x: -1*x.peak_flux)):\n src.island = isle\n src.source = comp\n islands.append(group)\n return islands", "response": "Regroups the islands of a catalog according to their normalised distance."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads a file from disk and return an HDUList if it is already an HDUList", "response": "def load_file_or_hdu(filename):\n \"\"\"\n Load a file from disk and return an HDUList\n If filename is already an HDUList return that instead\n\n Parameters\n ----------\n filename : str or HDUList\n File or HDU to be loaded\n\n Returns\n -------\n hdulist : HDUList\n \"\"\"\n if isinstance(filename, fits.HDUList):\n hdulist = filename\n else:\n hdulist = fits.open(filename, ignore_missing_end=True)\n return hdulist"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncompresses a file using decimation factor.", "response": "def compress(datafile, factor, outfile=None):\n \"\"\"\n Compress a file using decimation.\n\n Parameters\n ----------\n datafile : str or HDUList\n Input data to be loaded. (HDUList will be modified if passed).\n\n factor : int\n Decimation factor.\n\n outfile : str\n File to be written. Default = None, which means don't write a file.\n\n Returns\n -------\n hdulist : HDUList\n A decimated HDUList\n\n See Also\n --------\n :func:`AegeanTools.fits_interp.expand`\n \"\"\"\n if not (factor > 0 and isinstance(factor, int)):\n logging.error(\"factor must be a positive integer\")\n return None\n\n hdulist = load_file_or_hdu(datafile)\n\n header = hdulist[0].header\n data = np.squeeze(hdulist[0].data)\n cx, cy = data.shape[0], data.shape[1]\n\n nx = cx // factor\n ny = cy // factor\n # check to see if we will have some residual data points\n lcx = cx % factor\n lcy = cy % factor\n if lcx > 0:\n nx += 1\n if lcy > 0:\n ny += 1\n # decimate the data\n new_data = np.empty((nx + 1, ny + 1))\n new_data[:nx, :ny] = data[::factor, ::factor]\n # copy the last row/col across\n new_data[-1, :ny] = data[-1, ::factor]\n new_data[:nx, -1] = data[::factor, -1]\n new_data[-1, -1] = data[-1, -1]\n\n # TODO: Figure out what to do when CD2_1 and CD1_2 are non-zero\n if 'CDELT1' in header:\n header['CDELT1'] *= factor\n elif 'CD1_1' in header:\n header['CD1_1'] *= factor\n else:\n logging.error(\"Error: Can't find CDELT1 or CD1_1\")\n return None\n if 'CDELT2' in header:\n header['CDELT2'] *= factor\n elif \"CD2_2\" in header:\n header['CD2_2'] *= factor\n else:\n logging.error(\"Error: Can't find CDELT2 or CD2_2\")\n return None\n # Move the reference pixel so that the WCS is correct\n header['CRPIX1'] = (header['CRPIX1'] + factor - 1) / factor\n header['CRPIX2'] = (header['CRPIX2'] + factor - 1) / factor\n\n # Update the header so that we can do the correct interpolation later on\n header['BN_CFAC'] = (factor, \"Compression factor (grid size) used by BANE\")\n header['BN_NPX1'] = (header['NAXIS1'], 'original NAXIS1 value')\n header['BN_NPX2'] = (header['NAXIS2'], 'original NAXIS2 value')\n header['BN_RPX1'] = (lcx, 'Residual on axis 1')\n header['BN_RPX2'] = (lcy, 'Residual on axis 2')\n header['HISTORY'] = \"Compressed by a factor of {0}\".format(factor)\n\n # save the changes\n hdulist[0].data = np.array(new_data, dtype=np.float32)\n hdulist[0].header = header\n if outfile is not None:\n hdulist.writeto(outfile, overwrite=True)\n logging.info(\"Wrote: {0}\".format(outfile))\n return hdulist"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexpands and interpolate the given data file using the given method.", "response": "def expand(datafile, outfile=None):\n \"\"\"\n Expand and interpolate the given data file using the given method.\n Datafile can be a filename or an HDUList\n\n It is assumed that the file has been compressed and that there are `BN_?` keywords in the\n fits header that describe how the compression was done.\n\n Parameters\n ----------\n datafile : str or HDUList\n filename or HDUList of file to work on\n\n outfile : str\n filename to write to (default = None)\n\n Returns\n -------\n hdulist : HDUList\n HDUList of the expanded data.\n\n See Also\n --------\n :func:`AegeanTools.fits_interp.compress`\n\n \"\"\"\n hdulist = load_file_or_hdu(datafile)\n\n header = hdulist[0].header\n data = hdulist[0].data\n # Check for the required key words, only expand if they exist\n if not all(a in header for a in ['BN_CFAC', 'BN_NPX1', 'BN_NPX2', 'BN_RPX1', 'BN_RPX2']):\n return hdulist\n\n factor = header['BN_CFAC']\n (gx, gy) = np.mgrid[0:header['BN_NPX2'], 0:header['BN_NPX1']]\n # fix the last column of the grid to account for residuals\n lcx = header['BN_RPX2']\n lcy = header['BN_RPX1']\n\n rows = (np.arange(data.shape[0]) + int(lcx/factor))*factor\n cols = (np.arange(data.shape[1]) + int(lcy/factor))*factor\n\n # Do the interpolation\n hdulist[0].data = np.array(RegularGridInterpolator((rows,cols), data)((gx, gy)), dtype=np.float32)\n\n # update the fits keywords so that the WCS is correct\n header['CRPIX1'] = (header['CRPIX1'] - 1) * factor + 1\n header['CRPIX2'] = (header['CRPIX2'] - 1) * factor + 1\n\n if 'CDELT1' in header:\n header['CDELT1'] /= factor\n elif 'CD1_1' in header:\n header['CD1_1'] /= factor\n else:\n logging.error(\"Error: Can't find CD1_1 or CDELT1\")\n return None\n\n if 'CDELT2' in header:\n header['CDELT2'] /= factor\n elif \"CD2_2\" in header:\n header['CD2_2'] /= factor\n else:\n logging.error(\"Error: Can't find CDELT2 or CD2_2\")\n return None\n\n header['HISTORY'] = 'Expanded by factor {0}'.format(factor)\n\n # don't need these any more so delete them.\n del header['BN_CFAC'], header['BN_NPX1'], header['BN_NPX2'], header['BN_RPX1'], header['BN_RPX2']\n hdulist[0].header = header\n if outfile is not None:\n hdulist.writeto(outfile, overwrite=True)\n logging.info(\"Wrote: {0}\".format(outfile))\n return hdulist"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nchange autocommit mode of the current database entry.", "response": "def change_autocommit_mode(self, switch):\n \"\"\"\n Strip and make a string case insensitive and ensure it is either 'true' or 'false'.\n\n If neither, prompt user for either value.\n When 'true', return True, and when 'false' return False.\n \"\"\"\n parsed_switch = switch.strip().lower()\n if not parsed_switch in ['true', 'false']:\n self.send_response(\n self.iopub_socket, 'stream', {\n 'name': 'stderr',\n 'text': 'autocommit must be true or false.\\n\\n'\n }\n )\n\n switch_bool = (parsed_switch == 'true')\n committed = self.switch_autocommit(switch_bool)\n message = (\n 'committed current transaction & ' if committed else '' +\n 'switched autocommit mode to ' +\n str(self._autocommit)\n )\n self.send_response(\n self.iopub_socket, 'stream', {\n 'name': 'stderr',\n 'text': message,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an ASCII Armor format for the given binary data.", "response": "def armor(data, versioned=True):\n \"\"\"\n Returns a string in ASCII Armor format, for the given binary data. The\n output of this is compatiple with pgcrypto's armor/dearmor functions.\n \"\"\"\n template = '-----BEGIN PGP MESSAGE-----\\n%(headers)s%(body)s\\n=%(crc)s\\n-----END PGP MESSAGE-----'\n body = base64.b64encode(data)\n # The 24-bit CRC should be in big-endian, strip off the first byte (it's already masked in crc24).\n crc = base64.b64encode(struct.pack('>L', crc24(data))[1:])\n return template % {\n 'headers': 'Version: django-pgcrypto %s\\n\\n' % __version__ if versioned else '\\n',\n 'body': body.decode('ascii'),\n 'crc': crc.decode('ascii'),\n }"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dearmor(text, verify=True):\n lines = text.strip().split('\\n')\n data_lines = []\n check_data = None\n started = False\n in_body = False\n for line in lines:\n if line.startswith('-----BEGIN'):\n started = True\n elif line.startswith('-----END'):\n break\n elif started:\n if in_body:\n if line.startswith('='):\n # Once we get the checksum data, we're done.\n check_data = line[1:5].encode('ascii')\n break\n else:\n # This is part of the base64-encoded data.\n data_lines.append(line)\n else:\n if line.strip():\n # This is a header line, which we basically ignore for now.\n pass\n else:\n # The data starts after an empty line.\n in_body = True\n b64_str = ''.join(data_lines)\n # Python 3's b64decode expects bytes, not a string. We know base64 is ASCII, though.\n data = base64.b64decode(b64_str.encode('ascii'))\n if verify and check_data:\n # The 24-bit CRC is in big-endian, so we add a null byte to the beginning.\n crc = struct.unpack('>L', b'\\0' + base64.b64decode(check_data))[0]\n if crc != crc24(data):\n raise BadChecksumError()\n return data", "response": "Given a string in ASCII Armor format returns the decoded binary data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unpad(text, block_size):\n end = len(text)\n if end == 0:\n return text\n padch = ord_safe(text[end - 1])\n if padch > block_size:\n # If the last byte value is larger than the block size, it's not padded.\n return text\n while end > 0 and ord_safe(text[end - 1]) in (0, padch):\n end -= 1\n return text[:end]", "response": "Unpads the given text into a block of size block_size."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pad(text, block_size, zero=False):\n num = block_size - (len(text) % block_size)\n ch = b'\\0' if zero else chr(num).encode('latin-1')\n return text + (ch * num)", "response": "Given a text string and a block size pads the text with zeros to the specified block size."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef deconstruct(self):\n name, path, args, kwargs = super(BaseEncryptedField, self).deconstruct()\n kwargs.update({\n #'key': self.cipher_key,\n 'cipher': self.cipher_name,\n 'charset': self.charset,\n 'check_armor': self.check_armor,\n 'versioned': self.versioned,\n })\n return name, path, args, kwargs", "response": "Deconstruct the field for Django 1. 7 + migrations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_cipher(self):\n return self.cipher_class.new(self.cipher_key, self.cipher_class.MODE_CBC, b'\\0' * self.cipher_class.block_size)", "response": "Returns a new Cipher object for each time we want to encrypt or decrypt."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_packages_by_root_package(where):\n root_package = os.path.basename(where)\n packages = [ \"%s.%s\" % (root_package, sub_package)\n for sub_package in find_packages(where)]\n packages.insert(0, root_package)\n return packages", "response": "Better than excluding everything that is not needed collect only what is needed."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nclicks is a framework to simplify writing composable commands for command-line tools. This package extends the click_ functionality by adding support for commands that use configuration files. .. _click: https://click.pocoo.org/ EXAMPLE: A configuration file, like: .. code-block:: INI # -- FILE: foo.ini [foo] flag = yes name = Alice and Bob numbers = 1 4 9 16 25 filenames = foo/xxx.txt bar/baz/zzz.txt [person.alice] name = Alice birthyear = 1995 [person.bob] name = Bob birthyear = 2001 can be processed with: .. code-block:: python # EXAMPLE:", "response": "def make_long_description(marker=None, intro=None):\n \"\"\"\n click_ is a framework to simplify writing composable commands for\n command-line tools. This package extends the click_ functionality\n by adding support for commands that use configuration files.\n\n .. _click: https://click.pocoo.org/\n\n EXAMPLE:\n\n A configuration file, like:\n\n .. code-block:: INI\n\n # -- FILE: foo.ini\n [foo]\n flag = yes\n name = Alice and Bob\n numbers = 1 4 9 16 25\n filenames = foo/xxx.txt\n bar/baz/zzz.txt\n\n [person.alice]\n name = Alice\n birthyear = 1995\n\n [person.bob]\n name = Bob\n birthyear = 2001\n\n can be processed with:\n\n .. code-block:: python\n\n # EXAMPLE:\n \"\"\"\n if intro is None:\n intro = inspect.getdoc(make_long_description)\n\n with open(\"README.rst\", \"r\") as infile:\n line = infile.readline()\n while not line.strip().startswith(marker):\n line = infile.readline()\n\n # -- COLLECT REMAINING: Usage example\n contents = infile.read()\n\n text = intro +\"\\n\" + contents\n return text"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pubsub_pop_message(self, deadline=None):\n if not self.subscribed:\n excep = ClientError(\"you must subscribe before using \"\n \"pubsub_pop_message\")\n raise tornado.gen.Return(excep)\n reply = None\n try:\n reply = self._reply_list.pop(0)\n raise tornado.gen.Return(reply)\n except IndexError:\n pass\n if deadline is not None:\n td = timedelta(seconds=deadline)\n yield self._condition.wait(timeout=td)\n else:\n yield self._condition.wait()\n try:\n reply = self._reply_list.pop(0)\n except IndexError:\n pass\n raise tornado.gen.Return(reply)", "response": "Pops a message for a subscribed client."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate the slopes and directions based on Tarboton s algorithm.", "response": "def _tarboton_slopes_directions(data, dX, dY, facets, ang_adj):\n \"\"\"\n Calculate the slopes and directions based on the 8 sections from\n Tarboton http://www.neng.usu.edu/cee/faculty/dtarb/96wr03137.pdf\n \"\"\"\n shp = np.array(data.shape) - 1\n\n\n direction = np.full(data.shape, FLAT_ID_INT, 'float64')\n mag = np.full(data.shape, FLAT_ID_INT, 'float64')\n\n slc0 = [slice(1, -1), slice(1, -1)]\n for ind in xrange(8):\n\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(1 + e1[0], shp[0] + e1[0]),\n slice(1 + e1[1], shp[1] + e1[1])]\n slc2 = [slice(1 + e2[0], shp[0] + e2[0]),\n slice(1 + e2[1], shp[1] + e2[1])]\n\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp)\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n\n # %%Now do the edges\n # if the edge is lower than the interior, we need to copy the value\n # from the interior (as an approximation)\n ids1 = (direction[:, 1] > np.pi / 2) \\\n & (direction[:, 1] < 3 * np.pi / 2)\n direction[ids1, 0] = direction[ids1, 1]\n mag[ids1, 0] = mag[ids1, 1]\n ids1 = (direction[:, -2] < np.pi / 2) \\\n | (direction[:, -2] > 3 * np.pi / 2)\n direction[ids1, -1] = direction[ids1, -2]\n mag[ids1, -1] = mag[ids1, -2]\n ids1 = (direction[1, :] > 0) & (direction[1, :] < np.pi)\n direction[0, ids1] = direction[1, ids1]\n mag[0, ids1] = mag[1, ids1]\n ids1 = (direction[-2, :] > np.pi) & (direction[-2, :] < 2 * np.pi)\n direction[-1, ids1] = direction[-2, ids1]\n mag[-1, ids1] = mag[-2, ids1]\n\n # Now update the edges in case they are higher than the interior (i.e.\n # look at the downstream angle)\n\n # left edge\n slc0 = [slice(1, -1), slice(0, 1)]\n for ind in [0, 1, 6, 7]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(1 + e1[0], shp[0] + e1[0]), slice(e1[1], 1 + e1[1])]\n slc2 = [slice(1 + e2[0], shp[0] + e2[0]), slice(e2[1], 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp)\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # right edge\n slc0 = [slice(1, -1), slice(-1, None)]\n for ind in [2, 3, 4, 5]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(1 + e1[0], shp[0] + e1[0]),\n slice(shp[1] + e1[1], shp[1] + 1 + e1[1])]\n slc2 = [slice(1 + e2[0], shp[0] + e2[0]),\n slice(shp[1] + e2[1], shp[1] + 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp)\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # top edge\n slc0 = [slice(0, 1), slice(1, -1)]\n for ind in [4, 5, 6, 7]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(e1[0], 1 + e1[0]), slice(1 + e1[1], shp[1] + e1[1])]\n slc2 = [slice(e2[0], 1 + e2[0]), slice(1 + e2[1], shp[1] + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'top')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # bottom edge\n slc0 = [slice(-1, None), slice(1, -1)]\n for ind in [0, 1, 2, 3]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(shp[0] + e1[0], shp[0] + 1 + e1[0]),\n slice(1 + e1[1], shp[1] + e1[1])]\n slc2 = [slice(shp[0] + e2[0], shp[0] + 1 + e2[0]),\n slice(1 + e2[1], shp[1] + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'bot')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # top-left corner\n slc0 = [slice(0, 1), slice(0, 1)]\n for ind in [6, 7]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(e1[0], 1 + e1[0]), slice(e1[1], 1 + e1[1])]\n slc2 = [slice(e2[0], 1 + e2[0]), slice(e2[1], 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'top')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # top-right corner\n slc0 = [slice(0, 1), slice(-1, None)]\n for ind in [4, 5]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(e1[0], 1 + e1[0]),\n slice(shp[1] + e1[1], shp[1] + 1 + e1[1])]\n slc2 = [slice(e2[0], 1 + e2[0]),\n slice(shp[1] + e2[1], shp[1] + 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'top')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # bottom-left corner\n slc0 = [slice(-1, None), slice(0, 1)]\n for ind in [0, 1]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(shp[0] + e1[0], shp[0] + 1 + e1[0]),\n slice(e1[1], 1 + e1[1])]\n slc2 = [slice(shp[0] + e2[0], shp[0] + 1 + e2[0]),\n slice(e2[1], 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'bot')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n # bottom-right corner\n slc0 = [slice(-1, None), slice(-1, None)]\n for ind in [3, 4]:\n e1 = facets[ind][1]\n e2 = facets[ind][2]\n ang = ang_adj[ind]\n slc1 = [slice(shp[0] + e1[0], shp[0] + 1 + e1[0]),\n slice(shp[1] + e1[1], shp[1] + 1 + e1[1])]\n slc2 = [slice(shp[0] + e2[0], shp[0] + 1 + e2[0]),\n slice(shp[1] + e2[1], shp[1] + 1 + e2[1])]\n d1, d2, theta = _get_d1_d2(dX, dY, ind, e1, e2, shp, 'bot')\n mag, direction = _calc_direction(data, mag, direction, ang, d1, d2,\n theta, slc0, slc1, slc2)\n\n mag[mag > 0] = np.sqrt(mag[mag > 0])\n\n return mag, direction"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _calc_direction(data, mag, direction, ang, d1, d2, theta,\n slc0, slc1, slc2):\n \"\"\"\n This function gives the magnitude and direction of the slope based on\n Tarboton's D_\\infty method. This is a helper-function to\n _tarboton_slopes_directions\n \"\"\"\n \n data0 = data[slc0]\n data1 = data[slc1]\n data2 = data[slc2]\n\n s1 = (data0 - data1) / d1\n s2 = (data1 - data2) / d2\n s1_2 = s1**2\n \n sd = (data0 - data2) / np.sqrt(d1**2 + d2**2)\n r = np.arctan2(s2, s1)\n rad2 = s1_2 + s2**2\n\n # Handle special cases\n # should be on diagonal\n b_s1_lte0 = s1 <= 0\n b_s2_lte0 = s2 <= 0\n b_s1_gt0 = s1 > 0\n b_s2_gt0 = s2 > 0\n\n I1 = (b_s1_lte0 & b_s2_gt0) | (r > theta)\n if I1.any():\n rad2[I1] = sd[I1] ** 2\n r[I1] = theta.repeat(I1.shape[1], 1)[I1]\n \n I2 = (b_s1_gt0 & b_s2_lte0) | (r < 0) # should be on straight section\n if I2.any():\n rad2[I2] = s1_2[I2]\n r[I2] = 0\n \n I3 = b_s1_lte0 & (b_s2_lte0 | (b_s2_gt0 & (sd <= 0))) # upslope or flat\n rad2[I3] = -1\n\n I4 = rad2 > mag[slc0]\n if I4.any():\n mag[slc0][I4] = rad2[I4]\n direction[slc0][I4] = r[I4] * ang[1] + ang[0] * np.pi/2\n\n return mag, direction", "response": "This function calculates the magnitude and direction of the current Tarboton object based on Tarboton s D_\\infty method."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get(self, key, side):\n return getattr(self, side).ravel()[self.keys[key]]", "response": "Returns an edge given a particular key\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nassigning data on the ith tile to the field of that tile.", "response": "def set_i(self, i, data, field, side):\n \"\"\" Assigns data on the i'th tile to the data 'field' of the 'side'\n edge of that tile\n \"\"\"\n edge = self.get_i(i, side)\n setattr(edge, field, data[edge.slice])"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nassign data on the key tile to all the edges that are on the key tile.", "response": "def set_sides(self, key, data, field, local=False):\n \"\"\"\n Assign data on the 'key' tile to all the edges\n \"\"\"\n for side in ['left', 'right', 'top', 'bottom']:\n self.set(key, data, field, side, local)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nassign data to the edge on the neihboring tile which is on the neighbor_side of the key tile.", "response": "def set_neighbor_data(self, neighbor_side, data, key, field):\n \"\"\"\n Assign data from the 'key' tile to the edge on the\n neighboring tile which is on the 'neighbor_side' of the 'key' tile.\n The data is assigned to the 'field' attribute of the neihboring tile's\n edge.\n \"\"\"\n i = self.keys[key]\n found = False\n sides = []\n if 'left' in neighbor_side:\n if i % self.n_cols == 0:\n return None\n i -= 1\n sides.append('right')\n found = True\n if 'right' in neighbor_side:\n if i % self.n_cols == self.n_cols - 1:\n return None\n i += 1\n sides.append('left')\n found = True\n if 'top' in neighbor_side:\n sides.append('bottom')\n i -= self.n_cols\n found = True\n if 'bottom' in neighbor_side:\n sides.append('top')\n i += self.n_cols\n found = True\n if not found:\n print \"Side '%s' not found\" % neighbor_side\n # Check if i is in range\n if i < 0 or i >= self.n_chunks:\n return None\n # Otherwise, set the data\n for side in sides:\n self.set_i(i, data, field, side)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_all_neighbors_data(self, data, done, key):\n # The order of this for loop is important because the topleft gets\n # it's data from the left neighbor, which should have already been\n # updated...\n for side in ['left', 'right', 'top', 'bottom', 'topleft',\n 'topright', 'bottomleft', 'bottomright']:\n self.set_neighbor_data(side, data, key, 'data')\n# self.set_neighbor_data(side, todo, key, 'todo')\n self.set_neighbor_data(side, done, key, 'done')", "response": "Assign this data to all the neighbors of the key tile s data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fill_n_todo(self):\n left = self.left\n right = self.right\n top = self.top\n bottom = self.bottom\n for i in xrange(self.n_chunks):\n self.n_todo.ravel()[i] = np.sum([left.ravel()[i].n_todo,\n right.ravel()[i].n_todo,\n top.ravel()[i].n_todo,\n bottom.ravel()[i].n_todo])", "response": "Calculate and record the number of edge pixels left to do on each tile\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fill_n_done(self):\n left = self.left\n right = self.right\n top = self.top\n bottom = self.bottom\n for i in xrange(self.n_chunks):\n self.n_done.ravel()[i] = np.sum([left.ravel()[i].n_done,\n right.ravel()[i].n_done,\n top.ravel()[i].n_done,\n bottom.ravel()[i].n_done])", "response": "Calculates and record the number of edge pixels that are done one each\n tile."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the percentage of edge pixels that would be done if the tile was reprocessed. This is done for each tile.", "response": "def fill_percent_done(self):\n \"\"\"\n Calculate the percentage of edge pixels that would be done if the tile\n was reprocessed. This is done for each tile.\n \"\"\"\n left = self.left\n right = self.right\n top = self.top\n bottom = self.bottom\n for i in xrange(self.n_chunks):\n self.percent_done.ravel()[i] = \\\n np.sum([left.ravel()[i].percent_done,\n right.ravel()[i].percent_done,\n top.ravel()[i].percent_done,\n bottom.ravel()[i].percent_done])\n self.percent_done.ravel()[i] /= \\\n np.sum([left.ravel()[i].percent_done > 0,\n right.ravel()[i].percent_done > 0,\n top.ravel()[i].percent_done > 0,\n bottom.ravel()[i].percent_done > 0, 1e-16])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fill_array(self, array, field, add=False, maximize=False):\n self.fix_shapes()\n for i in xrange(self.n_chunks):\n for side in ['left', 'right', 'top', 'bottom']:\n edge = getattr(self, side).ravel()[i]\n if add:\n array[edge.slice] += getattr(edge, field)\n elif maximize:\n array[edge.slice] = np.maximum(array[edge.slice],\n getattr(edge, field))\n else:\n array[edge.slice] = getattr(edge, field)\n return array", "response": "Given a full array with the data on the while image fill it with the data on the while image edges."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfixing the shape of the data fields on edges.", "response": "def fix_shapes(self):\n \"\"\"\n Fixes the shape of the data fields on edges. Left edges should be\n column vectors, and top edges should be row vectors, for example.\n \"\"\"\n for i in xrange(self.n_chunks):\n for side in ['left', 'right', 'top', 'bottom']:\n edge = getattr(self, side).ravel()[i]\n if side in ['left', 'right']:\n shp = [edge.todo.size, 1]\n else:\n shp = [1, edge.todo.size]\n edge.done = edge.done.reshape(shp)\n edge.data = edge.data.reshape(shp)\n edge.todo = edge.todo.reshape(shp)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_best_candidate(self):\n self.fill_percent_done()\n i_b = np.argmax(self.percent_done.ravel())\n if self.percent_done.ravel()[i_b] <= 0:\n return None\n\n # check for ties\n I = self.percent_done.ravel() == self.percent_done.ravel()[i_b]\n if I.sum() == 1:\n return i_b\n else:\n I2 = np.argmax(self.max_elev.ravel()[I])\n return I.nonzero()[0][I2]", "response": "Determine which tile when processed would complete the largest\n percentage of unresolved edge pixels."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef save_array(self, array, name=None, partname=None, rootpath='.',\n raw=False, as_int=True):\n \"\"\"\n Standard array saving routine\n\n Parameters\n -----------\n array : array\n Array to save to file\n name : str, optional\n Default 'array.tif'. Filename of array to save. Over-writes\n partname.\n partname : str, optional\n Part of the filename to save (with the coordinates appended)\n rootpath : str, optional\n Default '.'. Which directory to save file\n raw : bool, optional\n Default False. If true will save a .npz of the array. If false,\n will save a geotiff\n as_int : bool, optional\n Default True. If true will save array as an integer array (\n excellent compression). If false will save as float array.\n \"\"\"\n if name is None and partname is not None:\n fnl_file = self.get_full_fn(partname, rootpath)\n tmp_file = os.path.join(rootpath, partname,\n self.get_fn(partname + '_tmp'))\n elif name is not None:\n fnl_file = name\n tmp_file = fnl_file + '_tmp.tiff'\n else:\n fnl_file = 'array.tif'\n if not raw:\n s_file = self.elev.clone_traits()\n s_file.raster_data = np.ma.masked_array(array)\n count = 10\n while count > 0 and (s_file.raster_data.mask.sum() > 0 \\\n or np.isnan(s_file.raster_data).sum() > 0):\n s_file.inpaint()\n count -= 1\n\n s_file.export_to_geotiff(tmp_file)\n\n if as_int:\n cmd = \"gdalwarp -multi -wm 2000 -co BIGTIFF=YES -of GTiff -co compress=lzw -ot Int16 -co TILED=YES -wo OPTIMIZE_SIZE=YES -r near -t_srs %s %s %s\" \\\n % (self.save_projection, tmp_file, fnl_file)\n else:\n cmd = \"gdalwarp -multi -wm 2000 -co BIGTIFF=YES -of GTiff -co compress=lzw -co TILED=YES -wo OPTIMIZE_SIZE=YES -r near -t_srs %s %s %s\" \\\n % (self.save_projection, tmp_file, fnl_file)\n print \"<<\"*4, cmd, \">>\"*4\n subprocess.call(cmd)\n os.remove(tmp_file)\n else:\n np.savez_compressed(fnl_file, array)", "response": "Standard array saving routine."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsaves the upstream contributing area to a file", "response": "def save_uca(self, rootpath, raw=False, as_int=False):\n \"\"\" Saves the upstream contributing area to a file\n \"\"\"\n self.save_array(self.uca, None, 'uca', rootpath, raw, as_int=as_int)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsave the topographic wetness index to a file.", "response": "def save_twi(self, rootpath, raw=False, as_int=True):\n \"\"\" Saves the topographic wetness index to a file\n \"\"\"\n self.twi = np.ma.masked_array(self.twi, mask=self.twi <= 0,\n fill_value=-9999)\n # self.twi = self.twi.filled()\n self.twi[self.flats] = 0\n self.twi.mask[self.flats] = True\n # self.twi = self.flats\n self.save_array(self.twi, None, 'twi', rootpath, raw, as_int=as_int)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef save_slope(self, rootpath, raw=False, as_int=False):\n self.save_array(self.mag, None, 'mag', rootpath, raw, as_int=as_int)", "response": "Saves the magnitude of the slope to a file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsave the direction of the slope to a file", "response": "def save_direction(self, rootpath, raw=False, as_int=False):\n \"\"\" Saves the direction of the slope to a file\n \"\"\"\n self.save_array(self.direction, None, 'ang', rootpath, raw, as_int=as_int)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_outputs(self, rootpath='.', raw=False):\n self.save_twi(rootpath, raw)\n self.save_uca(rootpath, raw)\n self.save_slope(rootpath, raw)\n self.save_direction(rootpath, raw)", "response": "Saves TWI UCA magnitude and direction of slope to files."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_array(self, fn, name):\n\n if os.path.exists(fn + '.npz'):\n array = np.load(fn + '.npz')\n try:\n setattr(self, name, array['arr_0'])\n except Exception, e:\n print e\n finally:\n array.close()\n\n else:\n raise RuntimeError(\"File %s does not exist.\" % (fn + '.npz'))", "response": "Loads an array from a file and sets the attribute name to the value of the attribute name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the edges of the array that gives the nominal size of the array and the overlap of the chunks.", "response": "def _get_chunk_edges(self, NN, chunk_size, chunk_overlap):\n \"\"\"\n Given the size of the array, calculate and array that gives the\n edges of chunks of nominal size, with specified overlap\n Parameters\n ----------\n NN : int\n Size of array\n chunk_size : int\n Nominal size of chunks (chunk_size < NN)\n chunk_overlap : int\n Number of pixels chunks will overlap\n Returns\n -------\n start_id : array\n The starting id of a chunk. start_id[i] gives the starting id of\n the i'th chunk\n end_id : array\n The ending id of a chunk. end_id[i] gives the ending id of\n the i'th chunk\n \"\"\"\n left_edge = np.arange(0, NN - chunk_overlap, chunk_size)\n left_edge[1:] -= chunk_overlap\n right_edge = np.arange(0, NN - chunk_overlap, chunk_size)\n right_edge[:-1] = right_edge[1:] + chunk_overlap\n right_edge[-1] = NN\n right_edge = np.minimum(right_edge, NN)\n return left_edge, right_edge"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nassigns data from a chunk to a full array.", "response": "def _assign_chunk(self, data, arr1, arr2, te, be, le, re, ovr, add=False):\n \"\"\"\n Assign data from a chunk to the full array. The data in overlap regions\n will not be assigned to the full array\n\n Parameters\n -----------\n data : array\n Unused array (except for shape) that has size of full tile\n arr1 : array\n Full size array to which data will be assigned\n arr2 : array\n Chunk-sized array from which data will be assigned\n te : int\n Top edge id\n be : int\n Bottom edge id\n le : int\n Left edge id\n re : int\n Right edge id\n ovr : int\n The number of pixels in the overlap\n add : bool, optional\n Default False. If true, the data in arr2 will be added to arr1,\n otherwise data in arr2 will overwrite data in arr1\n \"\"\"\n \n if te == 0:\n i1 = 0\n else:\n i1 = ovr\n \n if be == data.shape[0]:\n i2 = 0\n i2b = None\n else:\n i2 = -ovr\n i2b = -ovr\n \n if le == 0:\n j1 = 0\n else:\n j1 = ovr\n \n if re == data.shape[1]:\n j2 = 0\n j2b = None\n else:\n j2 = -ovr\n j2b = -ovr\n\n if add:\n arr1[te+i1:be+i2, le+j1:re+j2] += arr2[i1:i2b, j1:j2b]\n else:\n arr1[te+i1:be+i2, le+j1:re+j2] = arr2[i1:i2b, j1:j2b]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the magnitude and direction of slopes and fills self. data", "response": "def calc_slopes_directions(self, plotflag=False):\n \"\"\"\n Calculates the magnitude and direction of slopes and fills\n self.mag, self.direction\n \"\"\"\n \n # TODO minimum filter behavior with nans?\n \n # fill/interpolate flats first\n if self.fill_flats:\n data = np.ma.filled(self.data.astype('float64'), np.nan)\n filled = data.copy()\n \n edge = np.ones_like(data, bool)\n edge[1:-1, 1:-1] = False\n\n if self.fill_flats_below_sea: sea_mask = data != 0\n else: sea_mask = data > 0\n flat = (spndi.minimum_filter(data, (3, 3)) >= data) & sea_mask\n\n flats, n = spndi.label(flat, structure=FLATS_KERNEL3)\n objs = spndi.find_objects(flats)\n\n for i, _obj in enumerate(objs):\n obj = grow_obj(_obj, data.shape)\n self._fill_flat(data[obj], filled[obj], flats[obj]==i+1, edge[obj])\n\n self.data = np.ma.masked_array(filled, mask=np.isnan(filled)).astype(self.data.dtype)\n\n # %% Calculate the slopes and directions based on the 8 sections from\n # Tarboton http://www.neng.usu.edu/cee/faculty/dtarb/96wr03137.pdf\n if self.data.shape[0] <= self.chunk_size_slp_dir and \\\n self.data.shape[1] <= self.chunk_size_slp_dir:\n print \"starting slope/direction calculation\"\n self.mag, self.direction = self._slopes_directions(\n self.data, self.dX, self.dY, 'tarboton')\n # Find the flat regions. This is mostly simple (look for mag < 0),\n # but the downstream pixel at the edge of a flat will have a\n # calcuable angle which will not be accurate. We have to also find\n # these edges and set their magnitude to -1 (that is, the flat_id)\n\n self.find_flats()\n else:\n self.direction = np.full(self.data.shape, FLAT_ID_INT, 'float64')\n self.mag = np.full(self.data.shape, FLAT_ID_INT, 'float64')\n self.flats = np.zeros(self.data.shape, bool)\n top_edge, bottom_edge = \\\n self._get_chunk_edges(self.data.shape[0],\n self.chunk_size_slp_dir,\n self.chunk_overlap_slp_dir)\n left_edge, right_edge = \\\n self._get_chunk_edges(self.data.shape[1],\n self.chunk_size_slp_dir,\n self.chunk_overlap_slp_dir)\n ovr = self.chunk_overlap_slp_dir\n count = 1\n for te, be in zip(top_edge, bottom_edge):\n for le, re in zip(left_edge, right_edge):\n print \"starting slope/direction calculation for chunk\", \\\n count, \"[%d:%d, %d:%d]\" % (te, be, le, re)\n count += 1\n mag, direction = \\\n self._slopes_directions(self.data[te:be, le:re],\n self.dX[te:be-1],\n self.dY[te:be-1])\n\n flats = self._find_flats_edges(self.data[te:be, le:re],\n mag, direction)\n\n direction[flats] = FLAT_ID_INT\n mag[flats] = FLAT_ID_INT\n self._assign_chunk(self.data, self.mag, mag,\n te, be, le, re, ovr)\n self._assign_chunk(self.data, self.direction, direction,\n te, be, le, re, ovr)\n self._assign_chunk(self.data, self.flats, flats,\n te, be, le, re, ovr)\n\n if plotflag:\n self._plot_debug_slopes_directions()\n\n gc.collect() # Just in case\n return self.mag, self.direction"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _slopes_directions(self, data, dX, dY, method='tarboton'):\n # %%\n if method == 'tarboton':\n return self._tarboton_slopes_directions(data, dX, dY)\n elif method == 'central':\n return self._central_slopes_directions(data, dX, dY)", "response": "Wrapper to pick between various algorithms"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _tarboton_slopes_directions(self, data, dX, dY):\n\n return _tarboton_slopes_directions(data, dX, dY,\n self.facets, self.ang_adj)", "response": "Calculate the slopes and directions based on Tarboton s section 3. 1. 1."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _central_slopes_directions(self, data, dX, dY):\n shp = np.array(data.shape) - 1\n\n direction = np.full(data.shape, FLAT_ID_INT, 'float64')\n mag = np.full(direction, FLAT_ID_INT, 'float64')\n\n ind = 0\n d1, d2, theta = _get_d1_d2(dX, dY, ind, [0, 1], [1, 1], shp)\n s2 = (data[0:-2, 1:-1] - data[2:, 1:-1]) / d2\n s1 = -(data[1:-1, 0:-2] - data[1:-1, 2:]) / d1\n direction[1:-1, 1:-1] = np.arctan2(s2, s1) + np.pi\n mag = np.sqrt(s1**2 + s2**2)\n\n return mag, direction", "response": "Calculates the magnitude and direction of slopes using central difference"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfind the flats that are adjacent to the downstream side of the flat.", "response": "def _find_flats_edges(self, data, mag, direction):\n \"\"\"\n Extend flats 1 square downstream\n Flats on the downstream side of the flat might find a valid angle,\n but that doesn't mean that it's a correct angle. We have to find\n these and then set them equal to a flat\n \"\"\"\n\n i12 = np.arange(data.size).reshape(data.shape)\n\n flat = mag == FLAT_ID_INT\n flats, n = spndi.label(flat, structure=FLATS_KERNEL3)\n objs = spndi.find_objects(flats)\n\n f = flat.ravel()\n d = data.ravel()\n for i, _obj in enumerate(objs):\n region = flats[_obj] == i+1\n I = i12[_obj][region]\n J = get_adjacent_index(I, data.shape, data.size)\n f[J] = d[J] == d[I[0]]\n\n flat = f.reshape(data.shape)\n return flat"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef calc_uca(self, plotflag=False, edge_init_data=None, uca_init=None):\n if self.direction is None:\n self.calc_slopes_directions()\n\n # Initialize the upstream area\n uca_edge_init = np.zeros(self.data.shape, 'float64')\n uca_edge_done = np.zeros(self.data.shape, bool)\n uca_edge_todo = np.zeros(self.data.shape, bool)\n edge_init_done, edge_init_todo = None, None\n if edge_init_data is not None:\n edge_init_data, edge_init_done, edge_init_todo = edge_init_data\n slices = {'left': [slice(None), slice(0, 1)],\n 'right': [slice(None), slice(-1, None)],\n 'top': [slice(0, 1), slice(None)],\n 'bottom': [slice(-1, None), slice(None)]}\n for key, val in slices.iteritems():\n # To initialize and edge it needs to have data and be finished\n uca_edge_done[val] += \\\n edge_init_done[key].reshape(uca_edge_init[val].shape)\n uca_edge_init[val] = \\\n edge_init_data[key].reshape(uca_edge_init[val].shape)\n uca_edge_init[val][~uca_edge_done[val]] = 0\n uca_edge_todo[val] += \\\n edge_init_todo[key].reshape(uca_edge_init[val].shape)\n\n if uca_init is None:\n self.uca = np.full(self.data.shape, FLAT_ID_INT, 'float64')\n else:\n self.uca = uca_init.astype('float64')\n\n if self.data.shape[0] <= self.chunk_size_uca and \\\n self.data.shape[1] <= self.chunk_size_uca:\n if uca_init is None:\n print \"Starting uca calculation\"\n res = self._calc_uca_chunk(self.data, self.dX, self.dY,\n self.direction, self.mag,\n self.flats,\n area_edges=uca_edge_init,\n plotflag=plotflag)\n self.edge_todo = res[1]\n self.edge_done = res[2]\n self.uca = res[0]\n else:\n print \"Starting edge resolution round: \",\n # last return value will be None: edge_\n area, e2doi, edone, _ = \\\n self._calc_uca_chunk_update(self.data, self.dX, self.dY,\n self.direction, self.mag,\n self.flats,\n area_edges=uca_edge_init,\n edge_todo=uca_edge_todo,\n edge_done=uca_edge_done)\n self.uca += area\n self.edge_todo = e2doi\n self.edge_done = edone\n\n else:\n top_edge, bottom_edge = \\\n self._get_chunk_edges(self.data.shape[0], self.chunk_size_uca,\n self.chunk_overlap_uca)\n left_edge, right_edge = \\\n self._get_chunk_edges(self.data.shape[1], self.chunk_size_uca,\n self.chunk_overlap_uca)\n ovr = self.chunk_overlap_uca\n\n # Initialize the edge_todo and done arrays\n edge_todo = np.zeros(self.data.shape, bool)\n edge_todo_tile = np.zeros(self.data.shape, bool)\n edge_not_done_tile = np.zeros(self.data.shape, bool)\n edge_done = np.zeros(self.data.shape, bool)\n\n tile_edge = TileEdge(top_edge, bottom_edge, left_edge,\n right_edge, ovr,\n self.elev.grid_coordinates.x_axis,\n self.elev.grid_coordinates.y_axis, self.data)\n count = 1\n\n # Mask out the edges because we're just trying to resolve the\n # internal edge conflicts\n self.data.mask[:, 0] = True\n self.data.mask[:, -1] = True\n self.data.mask[0, :] = True\n self.data.mask[-1, :] = True\n\n # if 1: # uca_init == None:\n print \"Starting uca calculation for chunk: \",\n # %%\n for te, be in zip(top_edge, bottom_edge):\n for le, re in zip(left_edge, right_edge):\n print count, \"[%d:%d, %d:%d]\" % (te, be, le, re),\n count += 1\n area, e2doi, edone, e2doi_no_mask, e2o_no_mask = \\\n self._calc_uca_chunk(self.data[te:be, le:re],\n self.dX[te:be-1],\n self.dY[te:be-1],\n self.direction[te:be, le:re],\n self.mag[te:be, le:re],\n self.flats[te:be, le:re],\n area_edges=uca_edge_init[te:be, le:re],\n plotflag=plotflag,\n edge_todo_i_no_mask=uca_edge_todo[te:be, le:re])\n self._assign_chunk(self.data, self.uca, area,\n te, be, le, re, ovr)\n edge_todo[te:be, le:re] += e2doi\n edge_not_done_tile[te:be, le:re] += e2o_no_mask\n # if this tile is on the edge of the domain, we actually\n # want to keep the edge information\n # UPDATE: I don't think we actually need this here as it\n # will be handled by chunk update ???\n self._assign_chunk(self.data, edge_todo_tile, e2doi_no_mask,\n te, be, le, re, ovr)\n# if te == top_edge[0] or be == bottom_edge[-1] \\\n# or le == left_edge[0] or re == right_edge[-1]:\n# edge_todo_tile[te:be, le:re] = e2doi\n self._assign_chunk(self.data, edge_done, edone,\n te, be, le, re, ovr)\n tile_edge.set_all_neighbors_data(self.uca,\n edge_done,\n (te, be, le, re))\n tile_edge.set_sides((te, be, le, re), e2doi, 'todo',\n local=True)\n # %%\n print '..Done'\n # This needs to be much more sophisticated because we have to\n # follow the tile's edge value through the interior.\n # Since we have to do that anyway, we might as well recompute\n # the UCA from scratch. So the above branch does that. The branch\n # below would be more efficient if we can get it working.\n if 0: # else:\n # need to populate tile_edge somehow\n edge_todo_tile = uca_edge_todo & ~uca_edge_done\n edge_not_done_tile = edge_todo_tile.copy()\n for te, be in zip(top_edge, bottom_edge):\n for le, re in zip(\n [left_edge[0], left_edge[-1]],\n [right_edge[0], right_edge[-1]]):\n e2doi = uca_edge_todo[te:be, le:re]\n tiledata = uca_edge_init[te:be, le:re]\n tile_edge.set_sides((te, be, le, re), tiledata, 'data',\n local=True)\n tiledone = uca_edge_done[te:be, le:re]\n tile_edge.set_sides((te, be, le, re), tiledone, 'done',\n local=True)\n for te, be in zip([top_edge[0], top_edge[-1]],\n [bottom_edge[0], bottom_edge[-1]]):\n for le, re in zip(left_edge, right_edge):\n e2doi = uca_edge_todo[te:be, le:re]\n tiledata = uca_edge_init[te:be, le:re]\n tile_edge.set_sides((te, be, le, re), tiledata, 'data',\n local=True)\n tiledone = uca_edge_done[te:be, le:re]\n tile_edge.set_sides((te, be, le, re), tiledone, 'done',\n local=True)\n\n if not self.resolve_edges:\n # This branch is probably horribly broken (but it might have\n # always been that way)\n self.tile_edge = tile_edge\n self.edge_todo = edge_todo\n self.edge_done = edge_done\n return self.uca\n\n # ## RESOLVING EDGES ## #\n\n # Get a good starting tile for the iteration\n i = tile_edge.find_best_candidate()\n tile_edge.fix_shapes()\n# dbug = np.zeros_like(self.uca)\n print \"Starting edge resolution round: \",\n count = 0\n i_old = -1\n while i is not None and i != i_old:\n count += 1\n print count, '(%d) .' % i,\n # %%\n te, be, le, re = tile_edge.coords[i]\n data, dX, dY, direction, mag, flats = \\\n [self.data[te:be, le:re],\n self.dX[te:be-1], self.dY[te:be-1],\n self.direction[te:be, le:re],\n self.mag[te:be, le:re], self.flats[te:be, le:re]]\n area, e2doi, edone, e2doi_tile = self._calc_uca_chunk_update(\n data, dX, dY, direction, mag, flats, tile_edge, i,\n edge_todo=edge_not_done_tile[te:be, le:re])\n self._assign_chunk(self.data, self.uca, area,\n te, be, le, re, ovr, add=True)\n self._assign_chunk(self.data, edge_done, edone,\n te, be, le, re, ovr)\n tile_edge.set_all_neighbors_data(self.uca,\n edge_done, (te, be, le, re))\n\n try:\n edge_not_done_tile[te:be, le:re] += e2doi_tile\n except:\n import ipdb; ipdb.set_trace() # BREAKPOINT\n tile_edge.set_sides((te, be, le, re), e2doi, 'todo',\n local=True)\n i_old = i\n i = tile_edge.find_best_candidate()\n # Debugging plots below. Feel free to uncomment for debugging\n# def drawgrid():\n# ax = gca();\n# ax.set_xticks(np.linspace(-0.5, 63.5, 9))\n# ax.set_yticks(np.linspace(-0.5, 63.5, 9))\n# grid(lw=2, ls='-', c=(0.5, 0.5, 0.5))\n# figure(1);clf();imshow((self.uca), interpolation='none');colorbar(); title(\"uca\" + str(i_old) + \" \" + str(i));drawgrid()\n# figure(2);clf();imshow(area, interpolation='none');colorbar(); title(\"local area\" + str(i_old) + \" \" + str(i))\n## edge_todo[:] = 0\n## edge_todo = tile_edge.fill_array(edge_todo, 'todo', add=True)\n# figure(3);clf();imshow(edge_todo*1.0 + edge_done*2.0, interpolation='none');colorbar(); title(\"todo\" + str(i_old) + \" \" + str(i));clim(0, 3)\n# edge_todo[:] = 0\n# edge_todo = tile_edge.fill_array(edge_todo, 'coulddo', add=True)\n# figure(3);clf();imshow(edge_todo*1.0 + edge_done*2.0, interpolation='none');colorbar(); title(\"todo\" + str(i_old) + \" \" + str(i));clim(0, 3);drawgrid()\n# figure(4);clf();imshow(tile_edge.percent_done, interpolation='none');colorbar(); title(\"percent done\" + str(i_old) + \" \" + str(i));clim(0, 1)\n# dbug[:] = 0\n# dbug = tile_edge.fill_array(dbug, 'coulddo', maximize=False)\n# dbug[dbug > 0] -= (self.uca - ref_area)[dbug > 0]\n# figure(5);clf();imshow(dbug, interpolation='none');colorbar(); title(\"data diff\" + str(i_old) + \" \" + str(i));drawgrid()\n# dbug = (self.uca - area1)\n# figure(6);clf();imshow(np.log10(np.abs(dbug)), interpolation='none');colorbar(); title(\"uca diff\" + str(i_old) + \" \" + str(i));drawgrid()\n # %%\n self.tile_edge = tile_edge\n self.edge_todo = edge_todo_tile\n self.edge_done = ~edge_not_done_tile\n print '..Done'\n\n # Fix the very last pixel on the edges\n self.fix_edge_pixels(edge_init_data, edge_init_done, edge_init_todo)\n\n gc.collect() # Just in case\n return self.uca", "response": "Calculates the upstream contributing area of the key - value entry."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fix_edge_pixels(self, edge_init_data, edge_init_done, edge_init_todo):\n data, dX, dY, direction, flats = \\\n self.data, self.dX, self.dY, self.direction, self.flats\n sides = ['left', 'right', 'top', 'bottom']\n slices_o = [[slice(None), slice(1, 2)], [slice(None), slice(-2, -1)],\n [slice(1, 2), slice(None)], [slice(-2, -1), slice(None)]]\n slices_d = [[slice(None), slice(0, 1)], [slice(None), slice(-1, None)],\n [slice(0, 1), slice(None)], [slice(-1, None), slice(None)]]\n\n # The first set of edges will have contributions from two nodes whereas\n # the second set of edges will only have contributinos from one node\n indices = {'left': [[3, 4], [2, 5]], 'right': [[0, 7], [1, 6]],\n 'top': [[1, 2], [0, 3]], 'bottom': [[5, 6], [4, 7]]}\n\n # Figure out which section the drainage goes towards, and what\n # proportion goes to the straight-sided (as opposed to diagonal) node.\n\n for side, slice_o, slice_d in zip(sides, slices_o, slices_d):\n section, proportion = \\\n self._calc_uca_section_proportion(data[slice_o],\n dX[slice_o[0]],\n dY[slice_o[0]],\n direction[slice_o],\n flats[slice_o])\n # self-initialize:\n if side in ['left', 'right']:\n self.uca[slice_d] = \\\n np.concatenate(([dX[slice_d[0]][0] * dY[slice_d[0]][0]],\n dX[slice_d[0]] * dY[slice_d[0]]))\\\n .reshape(self.uca[slice_d].shape)\n else:\n self.uca[slice_d] = dX[slice_d[0]][0] * dY[slice_d[0]][0]\n for e in range(2):\n for i in indices[side][e]:\n ed = self.facets[i][2]\n ids = section == i\n if e == 0:\n self.uca[slice_d][ids] += self.uca[slice_o][ids] \\\n * proportion[ids]\n self.uca[slice_d][ids] += \\\n np.roll(np.roll(self.uca[slice_o] * (1 - proportion),\n ed[0], 0),\n ed[1], 1)[ids]\n if e == 1:\n self.uca[slice_d][ids] += \\\n np.roll(np.roll(self.uca[slice_o] * (proportion),\n ed[0], 0),\n ed[1], 1)[ids]\n\n # Finally, add the edge data from adjacent tiles\n if edge_init_done is not None:\n ids = edge_init_done[side] # > 0\n if side in ['left', 'right']:\n self.uca[slice_d][ids, :] = \\\n edge_init_data[side][ids][:, None]\n else:\n self.uca[slice_d][:, ids] = edge_init_data[side][ids]", "response": "This function fixes the pixels on the very edge of the tile."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _calc_uca_chunk_update(self, data, dX, dY, direction, mag, flats,\n tile_edge=None, i=None,\n area_edges=None, edge_todo=None, edge_done=None,\n plotflag=False):\n \"\"\"\n Calculates the upstream contributing area due to contributions from\n the edges only.\n \"\"\"\n # %%\n\n sides = ['left', 'right', 'top', 'bottom']\n slices = [[slice(None), slice(0, 1)], [slice(None), slice(-1, None)],\n [slice(0, 1), slice(None)], [slice(-1, None), slice(None)]]\n # Figure out which section the drainage goes towards, and what\n # proportion goes to the straight-sided (as opposed to diagonal) node.\n\n section, proportion = self._calc_uca_section_proportion(\n data, dX, dY, direction, flats)\n\n # Build the drainage or adjacency matrix\n A = self._mk_adjacency_matrix(section, proportion, flats, data, mag, dX, dY)\n if CYTHON:\n B = A\n C = A.tocsr()\n if not CYTHON:\n A = A.tocoo()\n\n ids = np.zeros(data.shape, bool)\n area = np.zeros(data.shape, 'float64')\n # Set the ids to the edges that are now done, and initialize the\n # edge area\n if tile_edge is not None:\n if edge_todo is not None:\n edge_todo_tile = edge_todo\n else:\n edge_todo_tile = None\n edge_todo = np.zeros(data.shape, bool)\n for side, slice0 in zip(sides, slices):\n edge = getattr(tile_edge, side).ravel()[i]\n ids[slice0] = edge.done & edge.coulddo\n # only add area from the finished edges\n area[slice0] = edge.data * edge.done * edge.coulddo\n edge_todo[slice0] = edge.todo & ~edge.done\n elif area_edges is not None and edge_todo is not None \\\n and edge_done is not None:\n area[:, 0] = area_edges[:, 0]\n area[:, -1] = area_edges[:, -1]\n area[-1, :] = area_edges[-1, :]\n area[0, :] = area_edges[0, :]\n\n # Initialize starting ids\n ids = edge_done & edge_todo\n edge_todo = edge_todo & ~edge_done\n edge_todo_tile = None\n else:\n raise RuntimeError(\"Need to specify either tile_edge or area_edges\"\n \"in _calc_uca_chunk_update\")\n\n ids = ids.ravel()\n ids0 = ids.copy()\n\n area[flats] = np.nan\n edge_done = ~edge_todo\n\n edge_todo_i = edge_todo.copy()\n ids_old = np.zeros_like(ids)\n\n # I need this to keep track of when I have to add the area, and when\n # I have to replace the area.\n ids_i = np.arange(ids.size)\n\n done = np.ones(data.shape, bool)\n done.ravel()[ids] = False\n\n # Now we have to advance done through the mesh to figure out which\n # contributions matter (i.e. what's done already)\n def drain_pixels_done(ids, arr, rows_A, cols_A):\n ids_old = ids.copy()\n ids_old[:] = False\n # If I use ids.sum() > 0 then I might get stuck in circular\n # references.\n while (ids - ids_old).sum() > 0:\n # %%\n print \"x\",\n ids_old = ids.copy()\n ids_todo = ids_i[ids.ravel()]\n ids[:] = False\n for id_todo in ids_todo:\n rows = cols_A == id_todo\n rows_id = rows_A[rows]\n ids[rows_id] += arr.ravel()[rows_id] is True\n arr.ravel()[rows_id] = False # Set second arrival new id\n\n return arr\n\n if CYTHON:\n a = cyutils.drain_connections(\n done.ravel(), ids, B.indptr, B.indices, set_to=False)\n done = a.reshape(done.shape).astype(bool)\n else:\n done = drain_pixels_done(ids, done, A.row, A.col)\n\n done[data.mask] = True # deal with no-data values\n #\n ids = ids0.copy()\n # Set all the edges to \"done\" for ids0. This ensures that no edges\n # will ever be updated, whether they are done or not.\n ids0 = ids0.reshape(data.shape)\n ids0[:, 0] = True\n ids0[:, -1] = True\n ids0[0, :] = True\n ids0[-1, :] = True\n ids0 = ids0.ravel()\n\n ids_old[:] = 0\n\n # %%\n def drain_area(ids, area, done, rows_A, cols_A, data_A,\n edge_todo_tile):\n ids_old = ids.copy()\n ids_old[:] = False\n # If I use ids.sum() > 0 then I might get stuck in\n # circular references.\n while (ids - ids_old).sum() > 0:\n # %%\n print \"o\",\n ids_old = ids.copy()\n done.ravel()[ids] = True\n ids_todo = ids_i[ids.ravel()]\n ids[:] = False\n for id_todo in ids_todo:\n rows = cols_A == id_todo\n rows_id = rows_A[rows]\n factor = data_A[rows]\n # not allowed to modify edge values\n edge_filter_ids = ~ids0[rows_id]\n factor = factor[edge_filter_ids]\n rows_id = rows_id[edge_filter_ids]\n area.ravel()[rows_id] += area.ravel()[id_todo] * factor\n if edge_todo_tile is not None:\n edge_todo_tile.ravel()[rows_id] += \\\n edge_todo_tile.ravel()[id_todo] * factor\n # Figure out of this cell that just received a contribution\n # should give its contribution to what's next... i.e. make\n # sure all inputs have been added together\n for row_id in rows_id: # this is the 'waiting' part\n cols = cols_A[rows_A == row_id]\n ids[row_id] += (~(done.ravel()[cols])).sum() == 0\n# for col in cols:\n# print 'row', row_id, 'col', col, 'done', done.ravel()[col]\n # Follow the drainage along. New candidates are cells that\n # just changed\n #ids = (track_id_old.ravel() == -1) \\\n # & (track_id_old.ravel() != track_id.ravel())\n# done.ravel()[ids] = True\n # figure(7);clf();imshow(ids.reshape(mag.shape), interpolation='none')\n # figure(8);clf();imshow(area, interpolation='none');colorbar()\n # figure(9);clf();imshow(done, interpolation='none');colorbar()\n # figure(10);clf();imshow(a + area - b, interpolation='none');colorbar()\n #%%\n # self._plot_connectivity(A, data=data)\n return area, done, edge_todo_tile\n\n if CYTHON:\n if edge_todo_tile is not None:\n a, b, c, d = cyutils.drain_area(area.ravel(),\n done.ravel(),\n ids,\n B.indptr, B.indices, B.data,\n C.indptr, C.indices,\n area.shape[0], area.shape[1],\n edge_todo_tile.astype('float64').ravel(),\n skip_edge=True)\n edge_todo_tile = c.reshape(edge_todo_tile.shape)\n else:\n a, b, c, d = cyutils.drain_area(area.ravel(),\n done.ravel(),\n ids,\n B.indptr, B.indices, B.data,\n C.indptr, C.indices,\n area.shape[0], area.shape[1],\n skip_edge=True)\n area = a.reshape(area.shape)\n done = b.reshape(done.shape)\n\n else:\n area, done, edge_todo_tile = \\\n drain_area(ids, area, done, A.row, A.col, A.data,\n edge_todo_tile)\n\n # Rather unfortunately, we still have to follow through the boolean\n # edge_todo matrix...\n ids = edge_todo.copy().ravel()\n # %%\n\n def drain_pixels_todo(ids, arr, rows_A, cols_A):\n ids_old = ids.copy()\n ids_old[:] = False\n # If I use ids.sum() > 0 then I might get stuck in\n # circular references.\n while (ids - ids_old).sum() > 0:\n # %%\n print \"x\",\n ids_old = ids.copy()\n# edge_todo_old = arr.copy()\n ids_todo = ids_i[ids.ravel()]\n ids[:] = False\n for id_todo in ids_todo:\n rows = cols_A == id_todo\n rows_id = rows_A[rows]\n ids[rows_id] += arr.ravel()[rows_id] == False\n arr.ravel()[rows_id] = True # Set new id of second arrival\n\n# #Follow the drainage along. New candidates are cells that just changed\n# ids = (edge_todo_old.ravel() != arr.ravel())\n return arr\n \n if CYTHON:\n a = cyutils.drain_connections(edge_todo.ravel(),\n ids, B.indptr, B.indices,\n set_to=True)\n edge_todo = a.reshape(edge_todo.shape).astype(bool)\n else:\n edge_todo = drain_pixels_todo(ids, edge_todo, A.row, A.col)\n\n area[flats] = np.nan\n edge_done = ~edge_todo\n\n return area, edge_todo_i, edge_done, edge_todo_tile", "response": "Calculates the upstream contributing area due to contributions from the edges only."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the upstream contributing area for the interior and the upstream contributing area for the interior and the upstream contributing area for the interior and the upstream contributing area for the interior and the upstream contributing area for the interior.", "response": "def _calc_uca_chunk(self, data, dX, dY, direction, mag, flats,\n area_edges, plotflag=False, edge_todo_i_no_mask=True):\n \"\"\"\n Calculates the upstream contributing area for the interior, and\n includes edge contributions if they are provided through area_edges.\n \"\"\"\n # %%\n # Figure out which section the drainage goes towards, and what\n # proportion goes to the straight-sided (as opposed to diagonal) node.\n section, proportion = self._calc_uca_section_proportion(\n data, dX, dY, direction, flats)\n\n # Build the drainage or adjacency matrix\n A = self._mk_adjacency_matrix(section, proportion, flats, data, mag, dX, dY)\n if CYTHON:\n B = A.tocsr()\n\n colsum = np.array(A.sum(1)).ravel()\n ids = colsum == 0 # If no one drains into me\n\n area = (dX * dY)\n # Record minimum area\n min_area = np.nanmin(area)\n self.twi_min_area = min(self.twi_min_area, min_area)\n\n area = np.concatenate((area[0:1], area)).reshape(area.size+1, 1)\n area = area.repeat(data.shape[1], 1)\n # Set the edge areas to zero, will add those contributions later\n area[:, 0] = area_edges[:, 0]\n area[:, -1] = area_edges[:, -1]\n area[-1, :] = area_edges[-1, :]\n area[0, :] = area_edges[0, :]\n # These edges are done, they have been drained already\n ids[area_edges.ravel() > 0] = True\n\n done = np.zeros(data.shape, bool)\n done.ravel()[ids] = True\n # deal with no-data values\n done[1:-1, 1:-1] = done[1:-1, 1:-1] | data.mask[1:-1, 1:-1]\n\n # Check the inlet edges\n edge_todo = np.zeros_like(done)\n ids_ed = np.arange(data.size).reshape(data.shape)\n # left\n edge_todo[:, 0] = (A[:, ids_ed[:, 0]].sum(0) > 0) \\\n & (area_edges[:, 0] == 0)\n edge_todo[:, -1] = (A[:, ids_ed[:, -1]].sum(0) > 0) \\\n & (area_edges[:, -1] == 0)\n edge_todo[0, :] = (A[:, ids_ed[0, :]].sum(0) > 0) \\\n & (area_edges[0, :] == 0)\n edge_todo[-1, :] = (A[:, ids_ed[-1, :]].sum(0) > 0) \\\n & (area_edges[-1, :] == 0)\n\n # Will do the tile-level doneness\n edge_todo_i_no_mask = edge_todo.copy() & edge_todo_i_no_mask\n edge_todo_no_mask = edge_todo_i_no_mask.copy() # tile-level doneness\n edge_todo[data.mask] = False # Don't do masked areas\n # Initialize done edges\n edge_todo_i = edge_todo.copy()\n ids_old = np.zeros_like(ids)\n # %%\n count = 1\n \n if CYTHON:\n area_ = area.ravel()\n done_ = done.ravel()\n edge_todo_ = edge_todo.astype('float64').ravel()\n edge_todo_no_mask_ = edge_todo_no_mask.astype('float64').ravel()\n data_ = data.ravel()\n\n while (np.any(~done) and count < self.circular_ref_maxcount):\n print \".\",\n count += 1\n if CYTHON:\n area_, done_, edge_todo_, edge_todo_no_mask_ = cyutils.drain_area(area_,\n done_, ids,\n A.indptr, A.indices, A.data, B.indptr, B.indices,\n area.shape[0], area.shape[1],\n edge_todo_, edge_todo_no_mask_)\n else:\n # If I use ids.sum() > 0 then I might get stuck in\n # circular references.\n while (ids - ids_old).sum() > 0:\n # %%\n ids_old = ids.copy()\n ids, area, done, edge_todo = \\\n self._drain_step(A, ids, area, done, edge_todo)\n # figure(1);clf();imshow(area, interpolation='none');colorbar()\n # figure(2);clf();imshow(ids.reshape(area.shape), interpolation='none');colorbar()\n # figure(3);clf();imshow(done, interpolation='none');colorbar()\n done_ = done.ravel()\n #%%\n ids[:] = False\n max_elev = (data_ * (~done_)).max()\n ids[((data_ * (~done_) - max_elev) / max_elev > -0.01)] = True\n\n if CYTHON:\n area = area_.reshape(area.shape)\n done = done_.reshape(done.shape)\n edge_todo = edge_todo_.reshape(edge_todo.shape).astype(bool)\n edge_todo_no_mask = edge_todo_no_mask_.reshape(edge_todo_no_mask.shape).astype(bool)\n\n area[flats] = np.nan\n\n edge_done = ~edge_todo\n edge_done[data.mask] = True # Don't do masked areas\n\n if self.apply_uca_limit_edges:\n # 2x because of bifurcations (maybe should be more than 2x, but\n # should be ok\n edge_done[area > self.uca_saturation_limit * 2 * min_area] = True\n\n # %%\n if plotflag:\n # TODO DTYPE\n self._plot_connectivity(A, (done.astype('float64') is False)\n + flats.astype('float64') * 2, [0, 3])\n return area, edge_todo_i, edge_done, edge_todo_i_no_mask, edge_todo_no_mask"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _drain_step(self, A, ids, area, done, edge_todo):\n # Only drain to cells that have a contribution\n A_todo = A[:, ids.ravel()]\n colsum = np.array(A_todo.sum(1)).ravel()\n # Only touch cells that actually receive a contribution\n # during this stage\n ids_new = colsum != 0\n # Is it possible that I may drain twice from my own cell?\n # -- No, I don't think so...\n # Is it possible that other cells may drain into me in\n # multiple iterations -- yes\n # Then say I check for when I'm done ensures that I don't drain until\n # everyone has drained into me\n area.ravel()[ids_new] += (A_todo[ids_new, :]\n * (area.ravel()[ids].ravel()))\n edge_todo.ravel()[ids_new] += (A_todo[ids_new, :]\n * (edge_todo.ravel()[ids].ravel()))\n # Figure out what's left to do.\n done.ravel()[ids] = True\n\n colsum = A * (~done.ravel())\n ids = colsum == 0\n # Figure out the new-undrained ids\n ids = ids & (~done.ravel())\n\n return ids, area, done, edge_todo", "response": "Do a single step of the upstream contributing area calculation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _calc_uca_section_proportion(self, data, dX, dY, direction, flats):\n shp = np.array(data.shape) - 1\n\n facets = self.facets\n adjust = self.ang_adj[:, 1]\n\n d1, d2, theta = _get_d1_d2(dX, dY, 0, facets[0][1], facets[0][2], shp)\n if dX.size > 1:\n theta = np.row_stack((theta[0, :], theta, theta[-1, :]))\n # Which quadrant am I in?\n section = ((direction / np.pi * 2.0) // 1).astype('int8') # TODO DTYPE\n # Gets me in the quadrant\n quadrant = (direction - np.pi / 2.0 * section)\n proportion = np.full_like(quadrant, np.nan)\n # Now which section within the quadrant\n section = section * 2 \\\n + (quadrant > theta.repeat(data.shape[1], 1)) * (section % 2 == 0) \\\n + (quadrant > (np.pi/2 - theta.repeat(data.shape[1], 1))) \\\n * (section % 2 == 1) # greater than because of ties resolution b4\n\n # %% Calculate proportion\n # As a side note, it's crazy to me how:\n # _get_d1_d2 needs to use indices 0, 3, 4, 7,\n # section uses even/odd (i.e. % 2)\n # proportion uses indices (0, 1, 4, 5) {ALl of them different! ARG!}\n I1 = (section == 0) | (section == 1) | (section == 4) | (section == 5)\n # I1 = section % 2 == 0\n I = I1 & (quadrant <= theta.repeat(data.shape[1], 1))\n proportion[I] = quadrant[I] / theta.repeat(data.shape[1], 1)[I]\n I = I1 & (quadrant > theta.repeat(data.shape[1], 1))\n proportion[I] = (quadrant[I] - theta.repeat(data.shape[1], 1)[I]) \\\n / (np.pi / 2 - theta.repeat(data.shape[1], 1)[I])\n I = (~I1) & (quadrant <= (np.pi / 2 - theta.repeat(data.shape[1], 1)))\n proportion[I] = (quadrant[I]) \\\n / (np.pi / 2 - theta.repeat(data.shape[1], 1)[I])\n I = (~I1) & (quadrant > (np.pi / 2 - theta.repeat(data.shape[1], 1)))\n proportion[I] = (quadrant[I] - (np.pi / 2 - theta.repeat(data.shape[1], 1)[I])) \\\n / (theta.repeat(data.shape[1], 1)[I])\n # %%Finish Proportion Calculation\n section[flats] = FLAT_ID_INT\n proportion[flats] = FLAT_ID\n\n section[section == 8] = 0 # Fence-post error correction\n proportion = (1 + adjust[section]) / 2.0 - adjust[section] * proportion\n\n return section, proportion", "response": "Calculate the uca section proportion of the drainage"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _mk_adjacency_matrix(self, section, proportion, flats, elev, mag, dX, dY):\n shp = section.shape\n mat_data = np.row_stack((proportion, 1 - proportion))\n NN = np.prod(shp)\n i12 = np.arange(NN).reshape(shp)\n j1 = - np.ones_like(i12)\n j2 = - np.ones_like(i12)\n\n # make the connectivity for the non-flats/pits\n j1, j2 = self._mk_connectivity(section, i12, j1, j2)\n j = np.row_stack((j1, j2))\n i = np.row_stack((i12, i12))\n \n # connectivity for flats/pits\n if self.drain_pits:\n pit_i, pit_j, pit_prop, flats, mag = \\\n self._mk_connectivity_pits(i12, flats, elev, mag, dX, dY)\n\n j = np.concatenate([j.ravel(), pit_j]).astype('int64')\n i = np.concatenate([i.ravel(), pit_i]).astype('int64')\n mat_data = np.concatenate([mat_data.ravel(), pit_prop])\n\n elif self.drain_flats:\n j1, j2, mat_data, flat_i, flat_j, flat_prop = \\\n self._mk_connectivity_flats(\n i12, j1, j2, mat_data, flats, elev, mag)\n\n j = np.concatenate([j.ravel(), flat_j]).astype('int64')\n i = np.concatenate([i.ravel(), flat_j]).astype('int64')\n mat_data = np.concatenate([mat_data.ravel(), flat_prop])\n\n\n\n # This prevents no-data values, remove connections when not present,\n # and makes sure that floating point precision errors do not\n # create circular references where a lower elevation cell drains\n # to a higher elevation cell\n I = ~np.isnan(mat_data) & (j != -1) & (mat_data > 1e-8) \\\n & (elev.ravel()[j] <= elev.ravel()[i])\n\n mat_data = mat_data[I]\n j = j[I]\n i = i[I]\n\n # %%Make the matrix and initialize\n # What is A? The row i area receives area contributions from the\n # entries in its columns. If all the entries in my columns have\n # drained, then I can drain.\n A = sps.csc_matrix((mat_data.ravel(),\n np.row_stack((j.ravel(), i.ravel()))),\n shape=(NN, NN))\n normalize = np.array(A.sum(0) + 1e-16).squeeze()\n A = np.dot(A, sps.diags(1/normalize, 0))\n\n return A", "response": "Calculates the adjacency matrix for the internal state of the logarithmic entry."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _mk_connectivity_flats(self, i12, j1, j2, mat_data, flats, elev, mag):\n nn, mm = flats.shape\n NN = np.prod(flats.shape)\n # Label the flats\n assigned, n_flats = spndi.label(flats, FLATS_KERNEL3)\n\n flat_ids, flat_coords, flat_labelsf = _get_flat_ids(assigned)\n flat_j = [None] * n_flats\n flat_prop = [None] * n_flats\n flat_i = [None] * n_flats\n\n # Temporary array to find the flats\n edges = np.zeros_like(flats)\n # %% Calcute the flat drainage\n warn_flats = []\n for ii in xrange(n_flats):\n ids_flats = flat_ids[flat_coords[ii]:flat_coords[ii+1]]\n edges[:] = 0\n j = ids_flats % mm\n i = ids_flats // mm\n for iii in [-1, 0, 1]:\n for jjj in [-1, 0, 1]:\n i_2 = i + iii\n j_2 = j + jjj\n ids_tmp = (i_2 >= 0) & (j_2 >= 0) & (i_2 < nn) & (j_2 < mm)\n edges[i_2[ids_tmp], j_2[ids_tmp]] += \\\n FLATS_KERNEL3[iii+1, jjj+1]\n edges.ravel()[ids_flats] = 0\n ids_edge = np.argwhere(edges.ravel()).squeeze()\n\n flat_elev_loc = elev.ravel()[ids_flats]\n # It is possble for the edges to merge 2 flats, so we need to\n # take the lower elevation to avoid large circular regions\n flat_elev = flat_elev_loc.min()\n\n loc_elev = elev.ravel()[ids_edge]\n # Filter out any elevations larger than the flat elevation\n # TODO: Figure out if this should be <= or <\n I_filt = loc_elev < flat_elev\n try:\n loc_elev = loc_elev[I_filt]\n loc_slope = mag.ravel()[ids_edge][I_filt]\n except: # If this is fully masked out (i.e. inside a no-data area)\n loc_elev = np.array([])\n loc_slope = np.array([])\n\n loc_dx = self.dX.mean()\n\n # Now I have to figure out if I should just use the minimum or\n # distribute amongst many pixels on the flat boundary\n n = len(loc_slope)\n if n == 0: # Flat does not have anywhere to drain\n # Let's see if the flat goes to the edge. If yes, we'll just\n # distribute the area along the edge.\n ids_flat_on_edge = ((ids_flats % mag.shape[1]) == 0) | \\\n ((ids_flats % mag.shape[1]) == (mag.shape[1] - 1)) | \\\n (ids_flats <= mag.shape[1]) | \\\n (ids_flats >= (mag.shape[1] * (mag.shape[0] - 1)))\n if ids_flat_on_edge.sum() == 0:\n warn_flats.append(ii)\n continue\n\n drain_ids = ids_flats[ids_flat_on_edge]\n loc_proportions = mag.ravel()[ids_flats[ids_flat_on_edge]]\n loc_proportions /= loc_proportions.sum()\n\n ids_flats = ids_flats[~ids_flat_on_edge]\n # This flat is entirely on the edge of the image\n if len(ids_flats) == 0:\n # therefore, whatever drains into it is done.\n continue\n flat_elev_loc = flat_elev_loc[~ids_flat_on_edge]\n else: # Flat has a place to drain to\n min_edges = np.zeros(loc_slope.shape, bool)\n min_edges[np.argmin(loc_slope)] = True\n # Add to the min edges any edge that is within an error\n # tolerance as small as the minimum\n min_edges = (loc_slope + loc_slope * loc_dx / 2) \\\n >= loc_slope[min_edges]\n\n drain_ids = ids_edge[I_filt][min_edges]\n\n loc_proportions = loc_slope[min_edges]\n loc_proportions /= loc_proportions.sum()\n\n # Now distribute the connectivity amongst the chosen elevations\n # proportional to their slopes\n\n # First, let all the the ids in the flats drain to 1\n # flat id (for ease)\n one_id = np.zeros(ids_flats.size, bool)\n one_id[np.argmin(flat_elev_loc)] = True\n\n j1.ravel()[ids_flats[~one_id]] = ids_flats[one_id]\n mat_data.ravel()[ids_flats[~one_id]] = 1\n # Negative indices will be eliminated before making the matix\n j2.ravel()[ids_flats[~one_id]] = -1\n mat_data.ravel()[ids_flats[~one_id] + NN] = 0\n\n # Now drain the 1 flat to the drains\n j1.ravel()[ids_flats[one_id]] = drain_ids[0]\n mat_data.ravel()[ids_flats[one_id]] = loc_proportions[0]\n if len(drain_ids) > 1:\n j2.ravel()[ids_flats[one_id]] = drain_ids[1]\n mat_data.ravel()[ids_flats[one_id] + NN] = loc_proportions[1]\n\n if len(loc_proportions > 2):\n flat_j[ii] = drain_ids[2:]\n flat_prop[ii] = loc_proportions[2:]\n flat_i[ii] = np.ones(drain_ids[2:].size, 'int64') * ids_flats[one_id]\n try:\n flat_j = np.concatenate([fj for fj in flat_j if fj is not None])\n flat_prop = \\\n np.concatenate([fp for fp in flat_prop if fp is not None])\n flat_i = np.concatenate([fi for fi in flat_i if fi is not None])\n except:\n flat_j = np.array([], 'int64')\n flat_prop = np.array([], 'float64')\n flat_i = np.array([], 'int64')\n\n if len(warn_flats) > 0:\n warnings.warn(\"Warning %d flats had no place\" % len(warn_flats) +\n \" to drain to --> these are pits (check pit-remove\"\n \"algorithm).\")\n return j1, j2, mat_data, flat_i, flat_j, flat_prop", "response": "This function calculates the connectivity between two flat regions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the topographic wetness index and saves the result in self. twi.", "response": "def calc_twi(self):\n \"\"\"\n Calculates the topographic wetness index and saves the result in\n self.twi.\n\n Returns\n -------\n twi : array\n Array giving the topographic wetness index at each pixel\n \"\"\"\n if self.uca is None:\n self.calc_uca()\n gc.collect() # Just in case\n min_area = self.twi_min_area\n min_slope = self.twi_min_slope\n twi = self.uca.copy()\n if self.apply_twi_limits_on_uca:\n twi[twi > self.uca_saturation_limit * min_area] = \\\n self.uca_saturation_limit * min_area\n gc.collect() # Just in case\n twi = np.log((twi) / (self.mag + min_slope))\n # apply the cap\n if self.apply_twi_limits:\n twi_sat_value = \\\n np.log(self.uca_saturation_limit * min_area / min_slope)\n twi[twi > twi_sat_value] = twi_sat_value\n # multiply by 10 for better integer resolution when storing\n self.twi = twi * 10\n\n gc.collect() # Just in case\n return twi"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _plot_connectivity(self, A, data=None, lims=[None, None]):\n\n if data is None:\n data = self.data\n\n B = A.tocoo()\n self._plot_connectivity_helper(B.col, B.row, B.data, data, lims)", "response": "A debug function used to plot the adjacency matrix A."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _plot_connectivity_helper(self, ii, ji, mat_datai, data, lims=[1, 8]):\n from matplotlib.pyplot import quiver, colorbar, clim, matshow\n I = ~np.isnan(mat_datai) & (ji != -1) & (mat_datai >= 0)\n mat_data = mat_datai[I]\n j = ji[I]\n i = ii[I]\n x = i.astype(float) % data.shape[1]\n y = i.astype(float) // data.shape[1]\n x1 = (j.astype(float) % data.shape[1]).ravel()\n y1 = (j.astype(float) // data.shape[1]).ravel()\n nx = (x1 - x)\n ny = (y1 - y)\n matshow(data, cmap='gist_rainbow'); colorbar(); clim(lims)\n quiver(x, y, nx, ny, mat_data.ravel(), angles='xy', scale_units='xy',\n scale=1, cmap='bone')\n colorbar(); clim([0, 1])", "response": "A debug function used to plot the adjacency matrix and connectivity matrix."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _plot_debug_slopes_directions(self):\n # %%\n from matplotlib.pyplot import matshow, colorbar, clim, title\n\n matshow(self.direction / np.pi * 180); colorbar(); clim(0, 360)\n title('Direction')\n\n mag2, direction2 = self._central_slopes_directions()\n matshow(direction2 / np.pi * 180.0); colorbar(); clim(0, 360)\n title('Direction (central difference)')\n\n matshow(self.mag); colorbar()\n title('Magnitude')\n matshow(mag2); colorbar(); title(\"Magnitude (Central difference)\")\n\n # %%\n # Compare to Taudem\n filename = self.file_name\n os.chdir('testtiff')\n try:\n os.remove('test_ang.tif')\n os.remove('test_slp.tif')\n except:\n pass\n cmd = ('dinfflowdir -fel \"%s\" -ang \"%s\" -slp \"%s\"' %\n (os.path.split(filename)[-1], 'test_ang.tif', 'test_slp.tif'))\n taudem._run(cmd)\n\n td_file = GdalReader(file_name='test_ang.tif')\n td_ang, = td_file.raster_layers\n td_file2 = GdalReader(file_name='test_slp.tif')\n td_mag, = td_file2.raster_layers\n os.chdir('..')\n\n matshow(td_ang.raster_data / np.pi*180); clim(0, 360); colorbar()\n title('Taudem direction')\n matshow(td_mag.raster_data); colorbar()\n title('Taudem magnitude')\n\n matshow(self.data); colorbar()\n title('The test data (elevation)')\n\n diff = (td_ang.raster_data - self.direction) / np.pi * 180.0\n diff[np.abs(diff) > 300] = np.nan\n matshow(diff); colorbar(); clim([-1, 1])\n title('Taudem direction - calculated Direction')\n\n # normalize magnitudes\n mag2 = td_mag.raster_data\n mag2 /= np.nanmax(mag2)\n mag = self.mag.copy()\n mag /= np.nanmax(mag)\n matshow(mag - mag2); colorbar()\n title('Taudem magnitude - calculated magnitude')\n del td_file\n del td_file2\n del td_ang\n del td_mag", "response": "A debug function to plot the direction calculated in various ways."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clean(ctx, dry_run=False):\n basedir = ctx.sphinx.destdir or \"build/docs\"\n cleanup_dirs([basedir], dry_run=dry_run)", "response": "Cleanup generated document artifacts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding docs with sphinx - build", "response": "def build(ctx, builder=\"html\", options=\"\"):\n \"\"\"Build docs with sphinx-build\"\"\"\n sourcedir = ctx.config.sphinx.sourcedir\n destdir = Path(ctx.config.sphinx.destdir or \"build\")/builder\n destdir = destdir.abspath()\n with cd(sourcedir):\n destdir_relative = Path(\".\").relpathto(destdir)\n command = \"sphinx-build {opts} -b {builder} {sourcedir} {destdir}\" \\\n .format(builder=builder, sourcedir=\".\",\n destdir=destdir_relative, opts=options)\n ctx.run(command)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef browse(ctx):\n page_html = Path(ctx.config.sphinx.destdir)/\"html\"/\"index.html\"\n if not page_html.exists():\n build(ctx, builder=\"html\")\n assert page_html.exists()\n open_cmd = \"open\" # -- WORKS ON: MACOSX\n if sys.platform.startswith(\"win\"):\n open_cmd = \"start\"\n ctx.run(\"{open} {page_html}\".format(open=open_cmd, page_html=page_html))", "response": "Open documentation in web browser."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(ctx, dest=\"docs.html\", format=\"html\"):\n print(\"STEP: Generate docs in HTML format\")\n build(ctx, builder=format)\n\n print(\"STEP: Save docs under %s/\" % dest)\n source_dir = Path(ctx.config.sphinx.destdir)/format\n Path(dest).rmtree_p()\n source_dir.copytree(dest)\n\n # -- POST-PROCESSING: Polish up.\n for part in [ \".buildinfo\", \".doctrees\" ]:\n partpath = Path(dest)/part\n if partpath.isdir():\n partpath.rmtree_p()\n elif partpath.exists():\n partpath.remove_p()", "response": "Save docs under destination directory."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the tile neighbors based on filenames.", "response": "def find_neighbors(neighbors, coords, I, source_files, f, sides):\n \"\"\"Find the tile neighbors based on filenames\n\n Parameters\n -----------\n neighbors : dict\n Dictionary that stores the neighbors. Format is\n neighbors[\"source_file_name\"][\"side\"] = \"neighbor_source_file_name\"\n coords : list\n List of coordinates determined from the filename.\n See :py:func:`utils.parse_fn`\n I : array\n Sort index. Different sorting schemes will speed up when neighbors\n are found\n source_files : list\n List of strings of source file names\n f : callable\n Function that determines if two tiles are neighbors based on their\n coordinates. f(c1, c2) returns True if tiles are neighbors\n sides : list\n List of 2 strings that give the \"side\" where tiles are neighbors.\n\n Returns\n -------\n neighbors : dict\n Dictionary of neighbors\n\n Notes\n -------\n For example, if Tile1 is to the left of Tile2, then\n neighbors['Tile1']['right'] = 'Tile2'\n neighbors['Tile2']['left'] = 'Tile1'\n \"\"\"\n for i, c1 in enumerate(coords):\n me = source_files[I[i]]\n # If the left neighbor has already been found...\n if neighbors[me][sides[0]] != '':\n continue\n # could try coords[i:] (+ fixes) for speed if it becomes a problem\n for j, c2 in enumerate(coords):\n if f(c1, c2):\n # then tiles are neighbors neighbors\n neigh = source_files[I[j]]\n neighbors[me][sides[0]] = neigh\n neighbors[neigh][sides[1]] = me\n break\n return neighbors"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the data and done arrays for the neighbors of the given elevation filename.", "response": "def set_neighbor_data(self, elev_fn, dem_proc, interp=None):\n \"\"\"\n From the elevation filename, we can figure out and load the data and\n done arrays.\n \"\"\"\n if interp is None:\n interp = self.build_interpolator(dem_proc)\n opp = {'top': 'bottom', 'left': 'right'}\n for key in self.neighbors[elev_fn].keys():\n tile = self.neighbors[elev_fn][key]\n if tile == '':\n continue\n oppkey = key\n for me, neigh in opp.iteritems():\n if me in key:\n oppkey = oppkey.replace(me, neigh)\n else:\n oppkey = oppkey.replace(neigh, me)\n opp_edge = self.neighbors[tile][oppkey]\n if opp_edge == '':\n continue\n\n interp.values = dem_proc.uca[::-1, :]\n# interp.values[:, 0] = np.ravel(dem_proc.uca) # for other interp.\n # for the top-left tile we have to set the bottom and right edges\n # of that tile, so two edges for those tiles\n for key_ed in oppkey.split('-'):\n self.edges[tile][key_ed].set_data('data', interp)\n\n interp.values = dem_proc.edge_done[::-1, :].astype(float)\n# interp.values[:, 0] = np.ravel(dem_proc.edge_done)\n for key_ed in oppkey.split('-'):\n self.edges[tile][key_ed].set_data('done', interp)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the todo of the edge", "response": "def update_edge_todo(self, elev_fn, dem_proc):\n \"\"\"\n Can figure out how to update the todo based on the elev filename\n \"\"\"\n for key in self.edges[elev_fn].keys():\n self.edges[elev_fn][key].set_data('todo', data=dem_proc.edge_todo)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the edges of the node with the given elev_fn and dem_proc.", "response": "def update_edges(self, elev_fn, dem_proc):\n \"\"\"\n After finishing a calculation, this will update the neighbors and the\n todo for that tile\n \"\"\"\n interp = self.build_interpolator(dem_proc)\n self.update_edge_todo(elev_fn, dem_proc)\n self.set_neighbor_data(elev_fn, dem_proc, interp)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_edge_init_data(self, fn, save_path=None):\n\n edge_init_data = {key: self.edges[fn][key].get('data') for key in\n self.edges[fn].keys()}\n edge_init_done = {key: self.edges[fn][key].get('done') for key in\n self.edges[fn].keys()}\n edge_init_todo = {key: self.edges[fn][key].get('todo') for key in\n self.edges[fn].keys()}\n return edge_init_data, edge_init_done, edge_init_todo", "response": "Creates the initialization data from the edge structure\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_best_candidate(self, elev_source_files=None):\n self.fill_percent_done()\n i_b = np.argmax(self.percent_done.values())\n if self.percent_done.values()[i_b] <= 0:\n return None\n\n # check for ties\n I = np.array(self.percent_done.values()) == \\\n self.percent_done.values()[i_b]\n if I.sum() == 1:\n pass # no ties\n else:\n I2 = np.argmax(np.array(self.max_elev.values())[I])\n i_b = I.nonzero()[0][I2]\n\n # Make sure the apples are still apples\n assert(np.array(self.max_elev.keys())[I][I2]\n == np.array(self.percent_done.keys())[I][I2])\n\n if elev_source_files is not None:\n fn = self.percent_done.keys()[i_b]\n lckfn = _get_lockfile_name(fn)\n if os.path.exists(lckfn): # another process is working on it\n # Find a different Candidate\n i_alt = np.argsort(self.percent_done.values())[::-1]\n for i in i_alt:\n fn = self.percent_done.keys()[i]\n lckfn = _get_lockfile_name(fn)\n if not os.path.exists(lckfn):\n break\n # Get and return the index\n i_b = elev_source_files.index(fn)\n\n return i_b", "response": "Find the best candidate for the current thread based on the updated edge information."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process_twi(self, index=None, do_edges=False, skip_uca_twi=False):\n if index is not None:\n elev_source_files = [self.elev_source_files[index]]\n else:\n elev_source_files = self.elev_source_files\n for i, esfile in enumerate(elev_source_files):\n try:\n fn, status = self.calculate_twi(esfile,\n save_path=self.save_path,\n do_edges=do_edges,\n skip_uca_twi=skip_uca_twi)\n if index is None:\n self.twi_status[i] = status\n else:\n self.twi_status[index] = status\n except:\n lckfn = _get_lockfile_name(esfile)\n try:\n os.remove(lckfn)\n except:\n pass\n traceback.print_exc()\n print traceback.format_exc()\n if index is None:\n self.twi_status[i] = \"Error \" + traceback.format_exc()\n else:\n self.twi_status[index] = \"Error \" + traceback.format_exc()", "response": "Processes the TWI and returns the TWI status of the UCA and TWI."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calculate_twi(self, esfile, save_path, use_cache=True, do_edges=False,\n skip_uca_twi=False):\n \"\"\"\n Calculates twi for supplied elevation file\n\n Parameters\n -----------\n esfile : str\n Path to elevation file to be processed\n save_path: str\n Root path to location where TWI will be saved. TWI will be saved in\n a subdirectory 'twi'.\n use_cache : bool (optional)\n Default True. If a temporary file exists (from a previous run),\n the cached file will be used. Otherwise, if False, existing files\n will be recomputed\n do_edges : bool (optional)\n See :py:func:`process_twi` for details on this argument.\n skip_uca_twi : bool (optional)\n Skips the calculation of the UCA and TWI (only calculates the\n magnitude and direction)\n \"\"\"\n if os.path.exists(os.path.join(save_path, 'tile_edge.pkl')) and \\\n self.tile_edge is None:\n with open(os.path.join(save_path, 'tile_edge.pkl'), 'r') as fid:\n self.tile_edge = cPickle.load(fid)\n elif self.tile_edge is None:\n self.tile_edge = TileEdgeFile(self.elev_source_files, save_path)\n with open(os.path.join(save_path, 'tile_edge.pkl'), 'wb') as fid:\n cPickle.dump(self.tile_edge, fid)\n\n\n status = 'Success' # optimism\n # Check if file is locked\n lckfn = _get_lockfile_name(esfile)\n coords = parse_fn(esfile)\n fn = get_fn_from_coords(coords, 'twi')\n print '*'*79\n if skip_uca_twi:\n print '*'*10, fn, 'Slope Calculation starting...:', '*'*10\n else:\n print '*'*10, fn, 'TWI Calculation starting...:', '*'*10\n print '*'*79\n if os.path.exists(lckfn): # another process is working on it\n print fn, 'is locked'\n return fn, \"Locked\"\n else: # lock this tile\n fid = file(lckfn, 'w')\n fid.close()\n\n dem_proc = DEMProcessor(esfile)\n # check if the slope already exists for the file. If yes, we should\n # move on to the next tile without doing anything else\n if skip_uca_twi \\\n and os.path.exists(dem_proc.get_full_fn('mag', save_path)\n + '.npz') \\\n and os.path.exists(dem_proc.get_full_fn('ang', save_path)\n + '.npz'):\n print dem_proc.get_full_fn('mag', save_path) + '.npz', 'already exists'\n print dem_proc.get_full_fn('ang', save_path) + '.npz', 'already exists'\n # remove lock file\n os.remove(lckfn)\n return fn, 'Cached: Slope'\n # check if the twi already exists for the file. If not in the edge\n # resolution round, we should move on to the next tile\n if os.path.exists(dem_proc.get_full_fn('twi', save_path)) \\\n and (do_edges is False):\n print dem_proc.get_full_fn('twi', save_path), 'already exists'\n # remove lock file\n os.remove(lckfn)\n return fn, 'Cached'\n\n # only calculate the slopes and direction if they do not exist in cache\n fn_ang = dem_proc.get_full_fn('ang', save_path)\n fn_mag = dem_proc.get_full_fn('mag', save_path)\n if os.path.exists(fn_ang + '.npz') and os.path.exists(fn_mag + '.npz')\\\n and not self.overwrite_cache:\n dem_proc.load_direction(fn_ang)\n dem_proc.load_slope(fn_mag)\n dem_proc.find_flats()\n else:\n if os.path.exists(fn_ang + '.npz') and os.path_exists(fn_mag + '.npz')\\\n and self.overwrite_cache:\n os.remove(fn_ang)\n os.remove(fn_mag)\n dem_proc.calc_slopes_directions()\n dem_proc.save_slope(save_path, raw=True)\n dem_proc.save_direction(save_path, raw=True)\n if self._DEBUG:\n dem_proc.save_slope(save_path, as_int=False)\n dem_proc.save_direction(save_path, as_int=False)\n\n if skip_uca_twi:\n # remove lock file\n os.remove(lckfn)\n return fn, status + \":mag-dir-only\"\n\n fn_uca = dem_proc.get_full_fn('uca', save_path)\n fn_uca_ec = dem_proc.get_full_fn('uca_edge_corrected', save_path)\n fn_twi = dem_proc.get_full_fn('twi', save_path)\n\n # check if edge structure exists for this tile and initialize\n edge_init_data, edge_init_done, edge_init_todo = \\\n self.tile_edge.get_edge_init_data(esfile, save_path)\n\n # Check if uca data exists (if yes, we are in the\n # edge-resolution round)\n uca_init = None\n if os.path.exists(fn_uca + '.npz'):\n if os.path.exists(fn_uca_ec + '.npz'):\n dem_proc.load_uca(fn_uca_ec)\n else:\n dem_proc.load_uca(fn_uca)\n uca_init = dem_proc.uca\n\n if do_edges or uca_init is None:\n dem_proc.calc_uca(uca_init=uca_init,\n edge_init_data=[edge_init_data, edge_init_done,\n edge_init_todo])\n\n if uca_init is None:\n dem_proc.save_uca(save_path, raw=True)\n if self._DEBUG:\n # Also save a geotiff for debugging\n dem_proc.save_uca(save_path, as_int=False)\n else:\n if os.path.exists(fn_uca_ec):\n os.remove(fn_uca_ec)\n dem_proc.save_array(dem_proc.uca, None, 'uca_edge_corrected',\n save_path, raw=True)\n if self._DEBUG:\n dem_proc.save_array(dem_proc.uca, None, 'uca_edge_corrected',\n save_path, as_int=False)\n # Saving Edge Data, and updating edges\n self.tile_edge.update_edges(esfile, dem_proc)\n\n dem_proc.calc_twi()\n if os.path.exists(fn_twi):\n os.remove(fn_twi)\n dem_proc.save_twi(save_path, raw=False)\n\n # clean up for in case\n gc.collect()\n\n # remove lock file\n os.remove(lckfn)\n # Save last-used dem_proc for debugging purposes\n if self._DEBUG:\n self.dem_proc = dem_proc\n return fn, status", "response": "Calculates the TWI for the supplied elevation file and saves it in the specified path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprocess the command for all the elevation sources in the source directory.", "response": "def process_command(self, command, save_name='custom', index=None):\n \"\"\"\n Processes the hillshading\n\n Parameters\n -----------\n index : int/slice (optional)\n Default: None - process all tiles in source directory. Otherwise,\n will only process the index/indices of the files as listed in\n self.elev_source_files\n\n \"\"\"\n if index is not None:\n elev_source_files = [self.elev_source_files[index]]\n else:\n elev_source_files = self.elev_source_files\n save_root = os.path.join(self.save_path, save_name)\n if not os.path.exists(save_root):\n os.makedirs(save_root)\n\n for i, esfile in enumerate(elev_source_files):\n try:\n status = 'Success' # optimism\n # Check if file is locked\n lckfn = _get_lockfile_name(esfile)\n coords = parse_fn(esfile)\n fn = get_fn_from_coords(coords, save_name)\n fn = os.path.join(save_root, fn)\n if os.path.exists(lckfn): # another process is working on it\n print fn, 'is locked'\n status = 'locked'\n elif os.path.exists(fn):\n print fn, 'already exists'\n status = 'cached'\n else: # lock this tile\n print fn, '... calculating ', save_name\n fid = file(lckfn, 'w')\n fid.close()\n\n # Calculate the custom process for this tile\n status = command(esfile, fn)\n\n os.remove(lckfn)\n\n if index is None:\n self.custom_status[i] = status\n else:\n self.custom_status[index] = status\n except:\n lckfn = _get_lockfile_name(esfile)\n try:\n os.remove(lckfn)\n except:\n pass\n traceback.print_exc()\n print traceback.format_exc()\n if index is None:\n self.custom_status[i] = \"Error \" + traceback.format_exc()\n else:\n self.custom_status[index] = \"Error \" + traceback.format_exc()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngive a list of file paths for elevation files, this function will rename those files to the format required by the pyDEM package. This assumes a .tif extension. Parameters ----------- files : list A list of strings of the paths to the elevation files that will be renamed name : str (optional) Default = None. A suffix to the filename. For example _suffix.tif Notes ------ The files are renamed in the same directory as the original file locations", "response": "def rename_files(files, name=None):\n \"\"\"\n Given a list of file paths for elevation files, this function will rename\n those files to the format required by the pyDEM package.\n\n This assumes a .tif extension.\n\n Parameters\n -----------\n files : list\n A list of strings of the paths to the elevation files that will be\n renamed\n name : str (optional)\n Default = None. A suffix to the filename. For example\n _suffix.tif\n\n Notes\n ------\n The files are renamed in the same directory as the original file locations\n \"\"\"\n for fil in files:\n elev_file = GdalReader(file_name=fil)\n elev, = elev_file.raster_layers\n fn = get_fn(elev, name)\n del elev_file\n del elev\n fn = os.path.join(os.path.split(fil)[0], fn)\n os.rename(fil, fn)\n print \"Renamed\", fil, \"to\", fn"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetermine the standard filename for a given GeoTIFF Layer.", "response": "def get_fn(elev, name=None):\n \"\"\"\n Determines the standard filename for a given GeoTIFF Layer.\n\n Parameters\n -----------\n elev : GdalReader.raster_layer\n A raster layer from the GdalReader object.\n name : str (optional)\n An optional suffix to the filename.\n Returns\n -------\n fn : str\n The standard _.tif with suffix (if supplied)\n \"\"\"\n gcs = elev.grid_coordinates\n coords = [gcs.LLC.lat, gcs.LLC.lon, gcs.URC.lat, gcs.URC.lon]\n return get_fn_from_coords(coords, name)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a set of coordinates returns the standard filename.", "response": "def get_fn_from_coords(coords, name=None):\n \"\"\" Given a set of coordinates, returns the standard filename.\n\n Parameters\n -----------\n coords : list\n [LLC.lat, LLC.lon, URC.lat, URC.lon]\n name : str (optional)\n An optional suffix to the filename.\n\n Returns\n -------\n fn : str\n The standard _.tif with suffix (if supplied)\n \"\"\"\n NS1 = [\"S\", \"N\"][coords[0] > 0]\n EW1 = [\"W\", \"E\"][coords[1] > 0]\n NS2 = [\"S\", \"N\"][coords[2] > 0]\n EW2 = [\"W\", \"E\"][coords[3] > 0]\n new_name = \"%s%0.3g%s%0.3g_%s%0.3g%s%0.3g\" % \\\n (NS1, coords[0], EW1, coords[1], NS2, coords[2], EW2, coords[3])\n if name is not None:\n new_name += '_' + name\n return new_name.replace('.', 'o') + '.tif'"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nextracts the change in x and y coordinates from the geotiff layer. Presently only supports WGS - 84 files.", "response": "def mk_dx_dy_from_geotif_layer(geotif):\n \"\"\"\n Extracts the change in x and y coordinates from the geotiff file. Presently\n only supports WGS-84 files.\n \"\"\"\n ELLIPSOID_MAP = {'WGS84': 'WGS-84'}\n ellipsoid = ELLIPSOID_MAP[geotif.grid_coordinates.wkt]\n d = distance(ellipsoid=ellipsoid)\n dx = geotif.grid_coordinates.x_axis\n dy = geotif.grid_coordinates.y_axis\n dX = np.zeros((dy.shape[0]-1))\n for j in xrange(len(dX)):\n dX[j] = d.measure((dy[j+1], dx[1]), (dy[j+1], dx[0])) * 1000 # km2m\n dY = np.zeros((dy.shape[0]-1))\n for i in xrange(len(dY)):\n dY[i] = d.measure((dy[i], 0), (dy[i+1], 0)) * 1000 # km2m\n return dX, dY"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new geotiff file object and return a handle to it and driver", "response": "def mk_geotiff_obj(raster, fn, bands=1, gdal_data_type=gdal.GDT_Float32,\n lat=[46, 45], lon=[-73, -72]):\n \"\"\"\n Creates a new geotiff file objects using the WGS84 coordinate system, saves\n it to disk, and returns a handle to the python file object and driver\n\n Parameters\n ------------\n raster : array\n Numpy array of the raster data to be added to the object\n fn : str\n Name of the geotiff file\n bands : int (optional)\n See :py:func:`gdal.GetDriverByName('Gtiff').Create\n gdal_data : gdal.GDT_\n Gdal data type (see gdal.GDT_...)\n lat : list\n northern lat, southern lat\n lon : list\n [western lon, eastern lon]\n \"\"\"\n NNi, NNj = raster.shape\n driver = gdal.GetDriverByName('GTiff')\n obj = driver.Create(fn, NNj, NNi, bands, gdal_data_type)\n pixel_height = -np.abs(lat[0] - lat[1]) / (NNi - 1.0)\n pixel_width = np.abs(lon[0] - lon[1]) / (NNj - 1.0)\n obj.SetGeoTransform([lon[0], pixel_width, 0, lat[0], 0, pixel_height])\n srs = osr.SpatialReference()\n srs.SetWellKnownGeogCS('WGS84')\n obj.SetProjection(srs.ExportToWkt())\n obj.GetRasterBand(1).WriteArray(raster)\n return obj, driver"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sortrows(a, i=0, index_out=False, recurse=True):\n I = np.argsort(a[:, i])\n a = a[I, :]\n # We recursively call sortrows to make sure it is sorted best by every\n # column\n if recurse & (len(a[0]) > i + 1):\n for b in np.unique(a[:, i]):\n ids = a[:, i] == b\n colids = range(i) + range(i+1, len(a[0]))\n a[np.ix_(ids, colids)], I2 = sortrows(a[np.ix_(ids, colids)],\n 0, True, True)\n I[ids] = I[np.nonzero(ids)[0][I2]]\n\n if index_out:\n return a, I\n else:\n return a", "response": "Sorts array a in descending order by column i."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns indices of 2d adjacent to those in I. Helper function for get_border.", "response": "def get_adjacent_index(I, shape, size):\n \"\"\"\n Find indices 2d-adjacent to those in I. Helper function for get_border*.\n\n Parameters\n ----------\n I : np.ndarray(dtype=int)\n indices in the flattened region\n shape : tuple(int, int)\n region shape\n size : int\n region size (technically computable from shape)\n\n Returns\n -------\n J : np.ndarray(dtype=int)\n indices orthogonally and diagonally adjacent to I\n\n \"\"\"\n\n m, n = shape\n In = I % n\n bL = In != 0\n bR = In != n-1\n \n J = np.concatenate([\n # orthonally adjacent\n I - n,\n I[bL] - 1,\n I[bR] + 1,\n I + n,\n\n # diagonally adjacent\n I[bL] - n-1,\n I[bR] - n+1,\n I[bL] + n-1,\n I[bR] + n+1])\n\n # remove indices outside the array\n J = J[(J>=0) & (J chunk_max_size:\n # we have enough data at first iteration\n # => fast path optimization\n view = self._get_pointer_or_memoryview(data,\n data_length)\n self.appendleft(view[chunk_max_size:])\n return view[:chunk_max_size]\n else:\n # no single iteration fast path optimization :-(\n # let's use a WriteBuffer to build the result chunk\n chunk_write_buffer = WriteBuffer()\n else:\n # not first iteration\n if chunk_write_buffer._total_length + data_length \\\n > chunk_max_size:\n view = self._get_pointer_or_memoryview(data,\n data_length)\n limit = chunk_max_size - \\\n chunk_write_buffer._total_length - data_length\n self.appendleft(view[limit:])\n data = view[:limit]\n chunk_write_buffer.append(data)\n if chunk_write_buffer._total_length >= chunk_max_size:\n break\n except IndexError:\n # the buffer is empty (so no memoryview inside)\n self._has_view = False\n break\n first_iteration = False\n return chunk_write_buffer._tobytes()", "response": "Pops a chunk of the given max size."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef with_name(self, name):\n if not self.name:\n raise ValueError(\"%r has an empty name\" % (self,))\n return self._from_parsed_parts(self._drv, self._root,\n self._parts[:-1] + [name])", "response": "Return a new path with the file name changed."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef with_suffix(self, suffix):\n # XXX if suffix is None, should the current suffix be removed?\n drv, root, parts = self._flavour.parse_parts((suffix,))\n if drv or root or len(parts) != 1:\n raise ValueError(\"Invalid suffix %r\" % (suffix))\n suffix = parts[0]\n if not suffix.startswith('.'):\n raise ValueError(\"Invalid suffix %r\" % (suffix))\n name = self.name\n if not name:\n raise ValueError(\"%r has an empty name\" % (self,))\n old_suffix = self.suffix\n if not old_suffix:\n name = name + suffix\n else:\n name = name[:-len(old_suffix)] + suffix\n return self._from_parsed_parts(self._drv, self._root,\n self._parts[:-1] + [name])", "response": "Return a new path with the file suffix changed or added."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _raw_open(self, flags, mode=0o777):\n return self._accessor.open(self, flags, mode)", "response": "Open the file pointed by this path and return a file descriptor."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\niterating over the files in this directory. Does not yield any result for the special paths....", "response": "def iterdir(self):\n \"\"\"Iterate over the files in this directory. Does not yield any\n result for the special paths '.' and '..'.\n \"\"\"\n for name in self._accessor.listdir(self):\n if name in ('.', '..'):\n # Yielding a path object for these makes little sense\n continue\n yield self._make_child_relpath(name)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef absolute(self):\n # XXX untested yet!\n if self.is_absolute():\n return self\n # FIXME this must defer to the specific flavour (and, under Windows,\n # use nt._getfullpathname())\n obj = self._from_parts([os.getcwd()] + self._parts, init=False)\n obj._init(template=self)\n return obj", "response": "Return an absolute version of this path."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef resolve(self):\n s = self._flavour.resolve(self)\n if s is None:\n # No symlink resolution => for consistency, raise an error if\n # the path doesn't exist or is forbidden\n self.stat()\n s = str(self.absolute())\n # Now we have no symlinks in the path, it's safe to normalize it.\n normed = self._flavour.pathmod.normpath(s)\n obj = self._from_parts((normed,), init=False)\n obj._init(template=self)\n return obj", "response": "Resolve the path and return a new object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nopens the file pointed by this path and return a file object.", "response": "def open(self, mode='r', buffering=-1, encoding=None,\n errors=None, newline=None):\n \"\"\"\n Open the file pointed by this path and return a file object, as\n the built-in open() function does.\n \"\"\"\n if sys.version_info >= (3, 3):\n return io.open(str(self), mode, buffering, encoding, errors, newline,\n opener=self._opener)\n else:\n return io.open(str(self), mode, buffering, encoding, errors, newline)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef replace(self, target):\n if sys.version_info < (3, 3):\n raise NotImplementedError(\"replace() is only available \"\n \"with Python 3.3 and later\")\n self._accessor.replace(self, target)", "response": "Rename this path to the given path clobbering the existing one."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef symlink_to(self, target, target_is_directory=False):\n self._accessor.symlink(target, self, target_is_directory)", "response": "Make this path a symlink pointing to the given path."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if this path is a symbolic link.", "response": "def is_symlink(self):\n \"\"\"\n Whether this path is a symbolic link.\n \"\"\"\n try:\n return S_ISLNK(self.lstat().st_mode)\n except OSError as e:\n if e.errno != ENOENT:\n raise\n # Path doesn't exist\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_block_device(self):\n try:\n return S_ISBLK(self.stat().st_mode)\n except OSError as e:\n if e.errno != ENOENT:\n raise\n # Path doesn't exist or is a broken symlink\n # (see https://bitbucket.org/pitrou/pathlib/issue/12/)\n return False", "response": "Returns True if this path is a block device."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_char_device(self):\n try:\n return S_ISCHR(self.stat().st_mode)\n except OSError as e:\n if e.errno != ENOENT:\n raise\n # Path doesn't exist or is a broken symlink\n # (see https://bitbucket.org/pitrou/pathlib/issue/12/)\n return False", "response": "Return True if this path is a character device."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a GridCoordinates object for the given upper left corner and lower right corner.", "response": "def grid_coords_from_corners(upper_left_corner, lower_right_corner, size):\n ''' Points are the outer edges of the UL and LR pixels. Size is rows, columns.\n GC projection type is taken from Points. '''\n assert upper_left_corner.wkt == lower_right_corner.wkt\n geotransform = np.array([upper_left_corner.lon, -(upper_left_corner.lon - lower_right_corner.lon) / float(size[1]), 0,\n upper_left_corner.lat, 0, -(upper_left_corner.lat - lower_right_corner.lat) / float(size[0])])\n return GridCoordinates(geotransform=geotransform,\n wkt=upper_left_corner.wkt,\n y_size=size[0],\n x_size=size[1])"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn True if the GC s overlap.", "response": "def intersects(self, other_grid_coordinates):\n \"\"\" returns True if the GC's overlap. \"\"\"\n ogc = other_grid_coordinates # alias\n # for explanation: http://stackoverflow.com/questions/306316/determine-if-two-rectangles-overlap-each-other\n # Note the flipped y-coord in this coord system.\n ax1, ay1, ax2, ay2 = self.ULC.lon, self.ULC.lat, self.LRC.lon, self.LRC.lat\n bx1, by1, bx2, by2 = ogc.ULC.lon, ogc.ULC.lat, ogc.LRC.lon, ogc.LRC.lat\n if ((ax1 <= bx2) and (ax2 >= bx1) and (ay1 >= by2) and (ay2 <= by1)):\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unique_str(self):\n\n unique_str = \"_\".join([\"%.3f\" % f for f in self.geotransform] +\n [\"%d\" % d for d in self.x_size, self.y_size]\n )\n if self.date is not None:\n unique_str += '_' + str(self.date)\n if self.time is not None:\n unique_str += '_' + str(self.time)\n return unique_str.replace(':', '_')", "response": "A string that uniquely represents this object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the x axis of the image.", "response": "def _get_x_axis(self):\n \"\"\"See http://www.gdal.org/gdal_datamodel.html for details.\"\"\"\n # 0,0 is top/left top top/left pixel. Actual x/y coord of that pixel are (.5,.5).\n x_centers = np.linspace(.5, self.x_size - .5, self.x_size)\n y_centers = x_centers * 0\n return (self.geotransform[0]\n + self.geotransform[1] * x_centers\n + self.geotransform[2] * y_centers)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_y_axis(self):\n # 0,0 is top/left top top/left pixel. Actual x/y coord of that pixel are (.5,.5).\n y_centers = np.linspace(.5, self.y_size - .5, self.y_size)\n x_centers = y_centers * 0\n return (self.geotransform[3]\n + self.geotransform[4] * x_centers\n + self.geotransform[5] * y_centers)", "response": "Get the y axis of the logarithmic data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting pixel coordinates to projection coordinates.", "response": "def raster_to_projection_coords(self, pixel_x, pixel_y):\n \"\"\" Use pixel centers when appropriate.\n See documentation for the GDAL function GetGeoTransform for details. \"\"\"\n h_px_py = np.array([1, pixel_x, pixel_y])\n gt = np.array([[1, 0, 0], self.geotransform[0:3], self.geotransform[3:6]])\n arr = np.inner(gt, h_px_py)\n return arr[2], arr[1]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef projection_to_raster_coords(self, lat, lon):\n r_px_py = np.array([1, lon, lat])\n tg = inv(np.array([[1, 0, 0], self.geotransform[0:3], self.geotransform[3:6]]))\n return np.inner(tg, r_px_py)[1:]", "response": "Returns pixel centers.\n See documentation for the GDAL function GetGeoTransform for details."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreplace masked - out elements in an array using an iterative image inpainting algorithm.", "response": "def inpaint(self):\n \"\"\" Replace masked-out elements in an array using an iterative image inpainting algorithm. \"\"\"\n\n import inpaint\n filled = inpaint.replace_nans(np.ma.filled(self.raster_data, np.NAN).astype(np.float32), 3, 0.01, 2)\n self.raster_data = np.ma.masked_invalid(filled)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef interp_value(self, lat, lon, indexed=False):\n (px, py) = self.grid_coordinates.projection_to_raster_coords(lat, lon)\n if indexed:\n return self.raster_data[round(py), round(px)]\n else:\n# from scipy.interpolate import interp2d\n# f_interp = interp2d(self.grid_coordinates.x_axis, self.grid_coordinates.y_axis, self.raster_data, bounds_error=True)\n# return f_interp(lon, lat)[0]\n from scipy.ndimage import map_coordinates\n ret = map_coordinates(self.raster_data, [[py], [px]], order=1) # linear interp\n return ret[0]", "response": "Interpolate the pixel value in the raster data."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_connected_client(self):\n if self.__sem is not None:\n yield self.__sem.acquire()\n client = None\n newly_created, client = self._get_client_from_pool_or_make_it()\n if newly_created:\n res = yield client.connect()\n if not res:\n LOG.warning(\"can't connect to %s\", client.title)\n raise tornado.gen.Return(\n ClientError(\"can't connect to %s\" % client.title))\n raise tornado.gen.Return(client)", "response": "Gets a connected Client object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_client_nowait(self):\n if self.__sem is not None:\n if self.__sem._value == 0:\n return None\n self.__sem.acquire()\n _, client = self._get_client_from_pool_or_make_it()\n return client", "response": "Gets a Client object that is not yet available."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a ContextManagerFuture that is yielded in a with statement.", "response": "def connected_client(self):\n \"\"\"Returns a ContextManagerFuture to be yielded in a with statement.\n\n Returns:\n A ContextManagerFuture object.\n\n Examples:\n >>> with (yield pool.connected_client()) as client:\n # client is a connected tornadis.Client instance\n # it will be automatically released to the pool thanks to\n # the \"with\" keyword\n reply = yield client.call(\"PING\")\n \"\"\"\n future = self.get_connected_client()\n cb = functools.partial(self._connected_client_release_cb, future)\n return ContextManagerFuture(future, cb)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef release_client(self, client):\n if isinstance(client, Client):\n if not self._is_expired_client(client):\n LOG.debug('Client is not expired. Adding back to pool')\n self.__pool.append(client)\n elif client.is_connected():\n LOG.debug('Client is expired and connected. Disconnecting')\n client.disconnect()\n if self.__sem is not None:\n self.__sem.release()", "response": "Releases a client object to the pool."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndisconnects all pooled client objects.", "response": "def destroy(self):\n \"\"\"Disconnects all pooled client objects.\"\"\"\n while True:\n try:\n client = self.__pool.popleft()\n if isinstance(client, Client):\n client.disconnect()\n except IndexError:\n break"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef preconnect(self, size=-1):\n if size == -1 and self.max_size == -1:\n raise ClientError(\"size=-1 not allowed with pool max_size=-1\")\n limit = min(size, self.max_size) if size != -1 else self.max_size\n clients = yield [self.get_connected_client() for _ in range(0, limit)]\n for client in clients:\n self.release_client(client)", "response": "Connects some or all redis clients inside the pool."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef setup_path(invoke_minversion=None):\n # print(\"INVOKE.tasks: setup_path\")\n if not os.path.isdir(TASKS_VENDOR_DIR):\n print(\"SKIP: TASKS_VENDOR_DIR=%s is missing\" % TASKS_VENDOR_DIR)\n return\n elif os.path.abspath(TASKS_VENDOR_DIR) in sys.path:\n # -- SETUP ALREADY DONE:\n # return\n pass\n\n use_vendor_bundles = os.environ.get(\"INVOKE_TASKS_USE_VENDOR_BUNDLES\", \"no\")\n if need_vendor_bundles(invoke_minversion):\n use_vendor_bundles = \"yes\"\n\n if use_vendor_bundles == \"yes\":\n syspath_insert(0, os.path.abspath(TASKS_VENDOR_DIR))\n if setup_path_for_bundle(INVOKE_BUNDLE, pos=1):\n import invoke\n bundle_path = os.path.relpath(INVOKE_BUNDLE, os.getcwd())\n print(\"USING: %s (version: %s)\" % (bundle_path, invoke.__version__))\n else:\n # -- BEST-EFFORT: May rescue something\n syspath_append(os.path.abspath(TASKS_VENDOR_DIR))\n setup_path_for_bundle(INVOKE_BUNDLE, pos=len(sys.path))\n\n if DEBUG_SYSPATH:\n for index, p in enumerate(sys.path):\n print(\" %d. %s\" % (index, p))", "response": "Setup python search and add TASKS_VENDOR_DIR if available."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef require_invoke_minversion(min_version, verbose=False):\n # -- REQUIRES: sys.path is setup and contains invoke\n try:\n import invoke\n invoke_version = invoke.__version__\n except ImportError:\n invoke_version = \"__NOT_INSTALLED\"\n\n if invoke_version < min_version:\n message = \"REQUIRE: invoke.version >= %s (but was: %s)\" % \\\n (min_version, invoke_version)\n message += \"\\nUSE: pip install invoke>=%s\" % min_version\n raise VersionRequirementError(message)\n\n INVOKE_VERSION = os.environ.get(\"INVOKE_VERSION\", None)\n if verbose and not INVOKE_VERSION:\n os.environ[\"INVOKE_VERSION\"] = invoke_version\n print(\"USING: invoke.version=%s\" % invoke_version)", "response": "Ensures that the current version of invoke is less than min_version."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef matches_section(section_name):\n section_names = section_name\n if isinstance(section_name, six.string_types):\n section_names = [section_name]\n elif not isinstance(section_name, (list, tuple)):\n raise ValueError(\"%r (expected: string, strings)\" % section_name)\n\n def decorator(cls):\n class_section_names = getattr(cls, \"section_names\", None)\n if class_section_names is None:\n cls.section_names = list(section_names)\n else:\n # -- BETTER SUPPORT: For multiple decorators\n # @matches_section(\"foo\")\n # @matches_section(\"bar.*\")\n # class Example(SectionSchema):\n # pass\n # assert Example.section_names == [\"foo\", \"bar.*\"]\n approved = [name for name in section_names\n if name not in cls.section_names]\n cls.section_names = approved + cls.section_names\n return cls\n return decorator", "response": "Decorator for SectionSchema classes to define the mapping between a config section schema class and one or more config sections with matching name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef assign_param_names(cls=None, param_class=None):\n if param_class is None:\n param_class = Param\n\n def decorate_class(cls):\n for name, value in select_params_from_section_schema(cls, param_class,\n deep=True):\n # -- ANNOTATE PARAM: By assigning its name\n if not value.name:\n value.name = name\n return cls\n\n # -- DECORATOR LOGIC:\n if cls is None:\n # -- CASE: @assign_param_names\n # -- CASE: @assign_param_names(...)\n return decorate_class\n else:\n # -- CASE: @assign_param_names class X: ...\n # -- CASE: assign_param_names(my_class)\n # -- CASE: my_class = assign_param_names(my_class)\n return decorate_class(cls)", "response": "Class decorator to assign parameter names to instances of Param."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef select_params_from_section_schema(section_schema, param_class=Param,\n deep=False):\n \"\"\"Selects the parameters of a config section schema.\n\n :param section_schema: Configuration file section schema to use.\n :return: Generator of params\n \"\"\"\n # pylint: disable=invalid-name\n for name, value in inspect.getmembers(section_schema):\n if name.startswith(\"__\") or value is None:\n continue # pragma: no cover\n elif inspect.isclass(value) and deep:\n # -- CASE: class => SELF-CALL (recursively).\n # pylint: disable= bad-continuation\n cls = value\n for name, value in select_params_from_section_schema(cls,\n param_class=param_class, deep=True):\n yield (name, value)\n elif isinstance(value, param_class):\n yield (name, value)", "response": "Selects the parameters of a config file section schema."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_config_section(config_section, section_schema):\n storage = {}\n for name, param in select_params_from_section_schema(section_schema):\n value = config_section.get(name, None)\n if value is None:\n if param.default is None:\n continue\n value = param.default\n else:\n value = param.parse(value)\n # -- DIAGNOSTICS:\n # print(\" %s = %s\" % (name, repr(value)))\n storage[name] = value\n return storage", "response": "Parse a config file section by using its schema and description."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate all available configuration file names.", "response": "def generate_configfile_names(config_files, config_searchpath=None):\n \"\"\"Generates all configuration file name combinations to read.\n\n .. sourcecode::\n\n # -- ALGORITHM:\n # First basenames/directories are prefered and override other files.\n for config_path in reversed(config_searchpath):\n for config_basename in reversed(config_files):\n config_fname = os.path.join(config_path, config_basename)\n if os.path.isfile(config_fname):\n yield config_fname\n\n :param config_files: List of config file basenames.\n :param config_searchpath: List of directories to look for config files.\n :return: List of available configuration file names (as generator)\n \"\"\"\n if config_searchpath is None:\n config_searchpath = [\".\"]\n\n for config_path in reversed(config_searchpath):\n for config_basename in reversed(config_files):\n config_fname = os.path.join(config_path, config_basename)\n if os.path.isfile(config_fname):\n # MAYBE: yield os.path.normpath(config_fname)\n yield config_fname"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef select_config_sections(configfile_sections, desired_section_patterns):\n for section_name in configfile_sections:\n for desired_section_pattern in desired_section_patterns:\n if fnmatch(section_name, desired_section_pattern):\n yield section_name", "response": "Select a subset of the sections in a configuration file by using\n a list of section names of list of section name patters\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef matches_section(cls, section_name, supported_section_names=None):\n if supported_section_names is None:\n supported_section_names = getattr(cls, \"section_names\", None)\n\n # pylint: disable=invalid-name\n for supported_section_name_or_pattern in supported_section_names:\n if fnmatch(section_name, supported_section_name_or_pattern):\n return True\n # -- OTHERWISE:\n return False", "response": "Indicates if this schema can be applied to a config section."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nderives support config section names from config section schemas.", "response": "def collect_config_sections_from_schemas(cls, config_section_schemas=None):\n # pylint: disable=invalid-name\n \"\"\"Derive support config section names from config section schemas.\n If no :param:`config_section_schemas` are provided, the schemas from\n this class are used (normally defined in the DerivedClass).\n\n :param config_section_schemas: List of config section schema classes.\n :return: List of config section names or name patterns (as string).\n \"\"\"\n if config_section_schemas is None:\n config_section_schemas = cls.config_section_schemas\n\n collected = []\n for schema in config_section_schemas:\n collected.extend(schema.section_names)\n # -- MAYBE BETTER:\n # for name in schema.section_names:\n # if name not in collected:\n # collected.append(name)\n return collected"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprocessing the config section and store the extracted data in the param storage.", "response": "def process_config_section(cls, config_section, storage):\n \"\"\"Process the config section and store the extracted data in\n the param:`storage` (as outgoing param).\n \"\"\"\n # -- CONCEPT:\n # if not storage:\n # # -- INIT DATA: With default parts.\n # storage.update(dict(_PERSONS={}))\n\n schema = cls.select_config_schema_for(config_section.name)\n if not schema:\n message = \"No schema found for: section=%s\"\n raise LookupError(message % config_section.name)\n\n # -- PARSE AND STORE CONFIG SECTION:\n section_storage = cls.select_storage_for(config_section.name, storage)\n section_data = parse_config_section(config_section, schema)\n section_storage.update(section_data)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef select_config_schema_for(cls, section_name):\n # pylint: disable=cell-var-from-loop, redefined-outer-name\n for section_schema in cls.config_section_schemas:\n schema_matches = getattr(section_schema, \"matches_section\", None)\n if schema_matches is None:\n # -- OTHER SCHEMA CLASS: Reuse SectionSchema functionality.\n schema_matches = lambda name: SectionSchema.matches_section(\n name, section_schema.section_names)\n\n if schema_matches(section_name):\n return section_schema\n return None", "response": "Select the config schema that matches the config section."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nselecting the data storage for a config section within the given storage.", "response": "def select_storage_for(cls, section_name, storage):\n \"\"\"Selects the data storage for a config section within the\n :param:`storage`. The primary config section is normally merged into\n the :param:`storage`.\n\n :param section_name: Config section (name) to process.\n :param storage: Data storage to use.\n :return: :param:`storage` or a part of it (as section storage).\n \"\"\"\n section_storage = storage\n storage_name = cls.get_storage_name_for(section_name)\n if storage_name:\n section_storage = storage.get(storage_name, None)\n if section_storage is None:\n section_storage = storage[storage_name] = dict()\n return section_storage"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncleaning up all the files and directories in the current order.", "response": "def clean_all(ctx, dry_run=False):\n \"\"\"Clean up everything, even the precious stuff.\n NOTE: clean task is executed first.\n \"\"\"\n cleanup_dirs(ctx.clean_all.directories or [], dry_run=dry_run)\n cleanup_dirs(ctx.clean_all.extra_directories or [], dry_run=dry_run)\n cleanup_files(ctx.clean_all.files or [], dry_run=dry_run)\n cleanup_files(ctx.clean_all.extra_files or [], dry_run=dry_run)\n execute_cleanup_tasks(ctx, cleanup_all_tasks, dry_run=dry_run)\n clean(ctx, dry_run=dry_run)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclean python related files and directories.", "response": "def clean_python(ctx, dry_run=False):\n \"\"\"Cleanup python related files/dirs: *.pyc, *.pyo, ...\"\"\"\n # MAYBE NOT: \"**/__pycache__\"\n cleanup_dirs([\"build\", \"dist\", \"*.egg-info\", \"**/__pycache__\"],\n dry_run=dry_run)\n if not dry_run:\n ctx.run(\"py.cleanup\")\n cleanup_files([\"**/*.pyc\", \"**/*.pyo\", \"**/*$py.class\"], dry_run=dry_run)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving files or files selected by patterns.", "response": "def cleanup_files(patterns, dry_run=False, workdir=\".\"):\n \"\"\"Remove files or files selected by file patterns.\n Skips removal if file does not exist.\n\n :param patterns: File patterns, like \"**/*.pyc\" (as list).\n :param dry_run: Dry-run mode indicator (as bool).\n :param workdir: Current work directory (default=\".\")\n \"\"\"\n current_dir = Path(workdir)\n python_basedir = Path(Path(sys.executable).dirname()).joinpath(\"..\").abspath()\n error_message = None\n error_count = 0\n for file_pattern in patterns:\n for file_ in path_glob(file_pattern, current_dir):\n if file_.abspath().startswith(python_basedir):\n # -- PROTECT CURRENTLY USED VIRTUAL ENVIRONMENT:\n continue\n\n if dry_run:\n print(\"REMOVE: %s (dry-run)\" % file_)\n else:\n print(\"REMOVE: %s\" % file_)\n try:\n file_.remove_p()\n except os.error as e:\n message = \"%s: %s\" % (e.__class__.__name__, e)\n print(message + \" basedir: \"+ python_basedir)\n error_count += 1\n if not error_message:\n error_message = message\n if False and error_message:\n class CleanupError(RuntimeError): pass\n raise CleanupError(error_message)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef path_glob(pattern, current_dir=None):\n if not current_dir:\n current_dir = pathlib.Path.cwd()\n elif not isinstance(current_dir, pathlib.Path):\n # -- CASE: string, path.Path (string-like)\n current_dir = pathlib.Path(str(current_dir))\n\n for p in current_dir.glob(pattern):\n yield Path(str(p))", "response": "Use pathlib for ant - like patterns like. py"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stack_call(self, *args):\n self.pipelined_args.append(args)\n self.number_of_stacked_calls = self.number_of_stacked_calls + 1", "response": "Stack a redis command inside the object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef connect(self):\n if self.is_connected() or self.is_connecting():\n raise tornado.gen.Return(True)\n if self.unix_domain_socket is None:\n self.__socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n if self.tcp_nodelay:\n self.__socket.setsockopt(socket.IPPROTO_TCP,\n socket.TCP_NODELAY, 1)\n else:\n if not os.path.exists(self.unix_domain_socket):\n LOG.warning(\"can't connect to %s, file does not exist\",\n self.unix_domain_socket)\n raise tornado.gen.Return(False)\n self.__socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n self.__socket.setblocking(0)\n self.__periodic_callback.start()\n try:\n LOG.debug(\"connecting to %s...\", self._redis_server())\n self._state.set_connecting()\n if self.unix_domain_socket is None:\n self.__socket.connect((self.host, self.port))\n else:\n self.__socket.connect(self.unix_domain_socket)\n except socket.error as e:\n if (errno_from_exception(e) not in _ERRNO_INPROGRESS and\n errno_from_exception(e) not in _ERRNO_WOULDBLOCK):\n self.disconnect()\n LOG.warning(\"can't connect to %s\", self._redis_server())\n raise tornado.gen.Return(False)\n self.__socket_fileno = self.__socket.fileno()\n self._register_or_update_event_handler()\n yield self._state.get_changed_state_future()\n if not self.is_connected():\n LOG.warning(\"can't connect to %s\", self._redis_server())\n raise tornado.gen.Return(False)\n else:\n LOG.debug(\"connected to %s\", self._redis_server())\n self.__socket_fileno = self.__socket.fileno()\n self._state.set_connected()\n self._register_or_update_event_handler()\n raise tornado.gen.Return(True)", "response": "Connects the object to the redis server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef disconnect(self):\n if not self.is_connected() and not self.is_connecting():\n return\n LOG.debug(\"disconnecting from %s...\", self._redis_server())\n self.__periodic_callback.stop()\n try:\n self._ioloop.remove_handler(self.__socket_fileno)\n self._listened_events = 0\n except Exception:\n pass\n self.__socket_fileno = -1\n try:\n self.__socket.close()\n except Exception:\n pass\n self._state.set_disconnected()\n self._close_callback()\n LOG.debug(\"disconnected from %s\", self._redis_server())", "response": "Disconnects the object from the Redis server."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite some data to the internal connection.", "response": "def write(self, data):\n \"\"\"Buffers some data to be sent to the host:port in a non blocking way.\n\n So the data is always buffered and not sent on the socket in a\n synchronous way.\n\n You can give a WriteBuffer as parameter. The internal Connection\n WriteBuffer will be extended with this one (without copying).\n\n Args:\n data (str or WriteBuffer): string (or WriteBuffer) to write to\n the host:port.\n \"\"\"\n if isinstance(data, WriteBuffer):\n self._write_buffer.append(data)\n else:\n if len(data) > 0:\n self._write_buffer.append(data)\n if self.aggressive_write:\n self._handle_write()\n if self._write_buffer._total_length > 0:\n self._register_or_update_event_handler(write=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef surrogate_escape(error):\n chars = error.object[error.start:error.end]\n assert len(chars) == 1\n val = ord(chars)\n val += 0xdc00\n return __builtin__.unichr(val), error.end", "response": "Simulate the Python 3 surrogateescape handler but for Python 2 only."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _always_unicode(cls, path):\n if PY3 or isinstance(path, text_type):\n return path\n return path.decode(sys.getfilesystemencoding(), 'surrogateescape')", "response": "Ensure the path as retrieved from a Python API is a proper Unicode string."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef namebase(self):\n base, ext = self.module.splitext(self.name)\n return base", "response": "The same as name but with one file extension stripped off."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dirs(self, pattern=None):\n return [p for p in self.listdir(pattern) if p.isdir()]", "response": "Returns a list of all directories in this directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef files(self, pattern=None):\n\n return [p for p in self.listdir(pattern) if p.isfile()]", "response": "D. files - > List of the files in this directory."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef walkdirs(self, pattern=None, errors='strict'):\n if errors not in ('strict', 'warn', 'ignore'):\n raise ValueError(\"invalid errors parameter\")\n\n try:\n dirs = self.dirs()\n except Exception:\n if errors == 'ignore':\n return\n elif errors == 'warn':\n warnings.warn(\n \"Unable to list directory '%s': %s\"\n % (self, sys.exc_info()[1]),\n TreeWalkWarning)\n return\n else:\n raise\n\n for child in dirs:\n if pattern is None or child.fnmatch(pattern):\n yield child\n for subsubdir in child.walkdirs(pattern, errors):\n yield subsubdir", "response": "A generator function that iterates over all directories in the directory and returns only directories whose names match the pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef walkfiles(self, pattern=None, errors='strict'):\n if errors not in ('strict', 'warn', 'ignore'):\n raise ValueError(\"invalid errors parameter\")\n\n try:\n childList = self.listdir()\n except Exception:\n if errors == 'ignore':\n return\n elif errors == 'warn':\n warnings.warn(\n \"Unable to list directory '%s': %s\"\n % (self, sys.exc_info()[1]),\n TreeWalkWarning)\n return\n else:\n raise\n\n for child in childList:\n try:\n isfile = child.isfile()\n isdir = not isfile and child.isdir()\n except:\n if errors == 'ignore':\n continue\n elif errors == 'warn':\n warnings.warn(\n \"Unable to access '%s': %s\"\n % (self, sys.exc_info()[1]),\n TreeWalkWarning)\n continue\n else:\n raise\n\n if isfile:\n if pattern is None or child.fnmatch(pattern):\n yield child\n elif isdir:\n for f in child.walkfiles(pattern, errors):\n yield f", "response": "A generator function that returns all files in the directory and directories that match the pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef open(self, *args, **kwargs):\n with io_error_compat():\n return io.open(self, *args, **kwargs)", "response": "Open this file and return a corresponding file object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_lines(self, lines, encoding=None, errors='strict',\n linesep=os.linesep, append=False):\n r\"\"\" Write the given lines of text to this file.\n\n By default this overwrites any existing file at this path.\n\n This puts a platform-specific newline sequence on every line.\n See `linesep` below.\n\n `lines` - A list of strings.\n\n `encoding` - A Unicode encoding to use. This applies only if\n `lines` contains any Unicode strings.\n\n `errors` - How to handle errors in Unicode encoding. This\n also applies only to Unicode strings.\n\n linesep - The desired line-ending. This line-ending is\n applied to every line. If a line already has any\n standard line ending (``'\\r'``, ``'\\n'``, ``'\\r\\n'``,\n ``u'\\x85'``, ``u'\\r\\x85'``, ``u'\\u2028'``), that will\n be stripped off and this will be used instead. The\n default is os.linesep, which is platform-dependent\n (``'\\r\\n'`` on Windows, ``'\\n'`` on Unix, etc.).\n Specify ``None`` to write the lines as-is, like\n :meth:`file.writelines`.\n\n Use the keyword argument ``append=True`` to append lines to the\n file. The default is to overwrite the file.\n\n .. warning ::\n\n When you use this with Unicode data, if the encoding of the\n existing data in the file is different from the encoding\n you specify with the `encoding=` parameter, the result is\n mixed-encoding data, which can really confuse someone trying\n to read the file later.\n \"\"\"\n with self.open('ab' if append else 'wb') as f:\n for l in lines:\n isUnicode = isinstance(l, text_type)\n if linesep is not None:\n pattern = U_NL_END if isUnicode else NL_END\n l = pattern.sub('', l) + linesep\n if isUnicode:\n l = l.encode(encoding or sys.getdefaultencoding(), errors)\n f.write(l)", "response": "r Write the given lines of text to the file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nliking mkdir but does not raise an exception.", "response": "def mkdir_p(self, mode=0o777):\n \"\"\" Like :meth:`mkdir`, but does not raise an exception if the\n directory already exists. \"\"\"\n try:\n self.mkdir(mode)\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.EEXIST:\n raise\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef makedirs_p(self, mode=0o777):\n try:\n self.makedirs(mode)\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.EEXIST:\n raise\n return self", "response": "Like makedirs but does not raise an exception."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rmdir_p(self):\n try:\n self.rmdir()\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.ENOTEMPTY and e.errno != errno.EEXIST:\n raise\n return self", "response": "Like rmdir but does not raise an exception if the directory is not empty."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef removedirs_p(self):\n try:\n self.removedirs()\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.ENOTEMPTY and e.errno != errno.EEXIST:\n raise\n return self", "response": "Like removedirs but does not raise an exception if the the\n directory is not empty."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_p(self):\n try:\n self.unlink()\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.ENOENT:\n raise\n return self", "response": "Like remove but does not raise an exception if the the\n file does not exist."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rmtree_p(self):\n try:\n self.rmtree()\n except OSError:\n _, e, _ = sys.exc_info()\n if e.errno != errno.ENOENT:\n raise\n return self", "response": "Like rmtree but does not raise an exception if the directory does not exist."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef merge_tree(self, dst, symlinks=False, *args, **kwargs):\n update = kwargs.pop('update', False)\n with tempdir() as _temp_dir:\n # first copy the tree to a stage directory to support\n # the parameters and behavior of copytree.\n stage = _temp_dir / str(hash(self))\n self.copytree(stage, symlinks, *args, **kwargs)\n # now copy everything from the stage directory using\n # the semantics of dir_util.copy_tree\n dir_util.copy_tree(stage, dst, preserve_symlinks=symlinks,\n update=update)", "response": "Copy all contents of self to dst overwriting existing\n contents in dst."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconnecting the client object to redis.", "response": "def connect(self):\n \"\"\"Connects the client object to redis.\n\n It's safe to use this method even if you are already connected.\n Note: this method is useless with autoconnect mode (default).\n\n Returns:\n a Future object with True as result if the connection was ok.\n \"\"\"\n if self.is_connected():\n raise tornado.gen.Return(True)\n cb1 = self._read_callback\n cb2 = self._close_callback\n self.__callback_queue = collections.deque()\n self._reply_list = []\n self.__reader = hiredis.Reader(replyError=ClientError)\n kwargs = self.connection_kwargs\n self.__connection = Connection(cb1, cb2, **kwargs)\n connection_status = yield self.__connection.connect()\n if connection_status is not True:\n # nothing left to do here, return\n raise tornado.gen.Return(False)\n if self.password is not None:\n authentication_status = yield self._call('AUTH', self.password)\n if authentication_status != b'OK':\n # incorrect password, return back the result\n LOG.warning(\"impossible to connect: bad password\")\n self.__connection.disconnect()\n raise tornado.gen.Return(False)\n if self.db != 0:\n db_status = yield self._call('SELECT', self.db)\n if db_status != b'OK':\n LOG.warning(\"can't select db %s\", self.db)\n raise tornado.gen.Return(False)\n raise tornado.gen.Return(True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _close_callback(self):\n while True:\n try:\n callback = self.__callback_queue.popleft()\n callback(ConnectionError(\"closed connection\"))\n except IndexError:\n break\n if self.subscribed:\n # pubsub clients\n self._reply_list.append(ConnectionError(\"closed connection\"))\n self._condition.notify_all()", "response": "Callback called when redis closed the connection."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_callback(self, data=None):\n try:\n if data is not None:\n self.__reader.feed(data)\n while True:\n reply = self.__reader.gets()\n if reply is not False:\n try:\n callback = self.__callback_queue.popleft()\n # normal client (1 reply = 1 callback)\n callback(reply)\n except IndexError:\n # pubsub clients\n self._reply_list.append(reply)\n self._condition.notify_all()\n else:\n break\n except hiredis.ProtocolError:\n # something nasty occured (corrupt stream => no way to recover)\n LOG.warning(\"corrupted stream => disconnect\")\n self.disconnect()", "response": "Callback called when some data is read on the socket."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef call(self, *args, **kwargs):\n if not self.is_connected():\n if self.autoconnect:\n # We use this method only when we are not contected\n # to void performance penaly due to gen.coroutine decorator\n return self._call_with_autoconnect(*args, **kwargs)\n else:\n error = ConnectionError(\"you are not connected and \"\n \"autoconnect=False\")\n return tornado.gen.maybe_future(error)\n return self._call(*args, **kwargs)", "response": "Calls a redis command and returns a Future of the reply."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall a redis command and waits for the reply and call a callback.", "response": "def async_call(self, *args, **kwargs):\n \"\"\"Calls a redis command, waits for the reply and call a callback.\n\n Following options are available (not part of the redis command itself):\n\n - callback\n Function called (with the result as argument) when the result\n is available. If not set, the reply is silently discarded. In\n case of errors, the callback is called with a\n TornadisException object as argument.\n\n Args:\n *args: full redis command as variable length argument list or\n a Pipeline object (as a single argument).\n **kwargs: options as keyword parameters.\n\n Examples:\n\n >>> def cb(result):\n pass\n >>> client.async_call(\"HSET\", \"key\", \"field\", \"val\", callback=cb)\n \"\"\"\n def after_autoconnect_callback(future):\n if self.is_connected():\n self._call(*args, **kwargs)\n else:\n # FIXME\n pass\n\n if 'callback' not in kwargs:\n kwargs['callback'] = discard_reply_cb\n if not self.is_connected():\n if self.autoconnect:\n connect_future = self.connect()\n cb = after_autoconnect_callback\n self.__connection._ioloop.add_future(connect_future, cb)\n else:\n error = ConnectionError(\"you are not connected and \"\n \"autoconnect=False\")\n kwargs['callback'](error)\n else:\n self._call(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nformats arguments into the redis protocol", "response": "def format_args_in_redis_protocol(*args):\n \"\"\"Formats arguments into redis protocol...\n\n This function makes and returns a string/buffer corresponding to\n given arguments formated with the redis protocol.\n\n integer, text, string or binary types are automatically converted\n (using utf8 if necessary).\n\n More informations about the protocol: http://redis.io/topics/protocol\n\n Args:\n *args: full redis command as variable length argument list\n\n Returns:\n binary string (arguments in redis protocol)\n\n Examples:\n >>> format_args_in_redis_protocol(\"HSET\", \"key\", \"field\", \"value\")\n '*4\\r\\n$4\\r\\nHSET\\r\\n$3\\r\\nkey\\r\\n$5\\r\\nfield\\r\\n$5\\r\\nvalue\\r\\n'\n \"\"\"\n buf = WriteBuffer()\n l = \"*%d\\r\\n\" % len(args) # noqa: E741\n if six.PY2:\n buf.append(l)\n else: # pragma: no cover\n buf.append(l.encode('utf-8'))\n for arg in args:\n if isinstance(arg, six.text_type):\n # it's a unicode string in Python2 or a standard (unicode)\n # string in Python3, let's encode it in utf-8 to get raw bytes\n arg = arg.encode('utf-8')\n elif isinstance(arg, six.string_types):\n # it's a basestring in Python2 => nothing to do\n pass\n elif isinstance(arg, six.binary_type): # pragma: no cover\n # it's a raw bytes string in Python3 => nothing to do\n pass\n elif isinstance(arg, six.integer_types):\n tmp = \"%d\" % arg\n if six.PY2:\n arg = tmp\n else: # pragma: no cover\n arg = tmp.encode('utf-8')\n elif isinstance(arg, WriteBuffer):\n # it's a WriteBuffer object => nothing to do\n pass\n else:\n raise Exception(\"don't know what to do with %s\" % type(arg))\n l = \"$%d\\r\\n\" % len(arg) # noqa: E741\n if six.PY2:\n buf.append(l)\n else: # pragma: no cover\n buf.append(l.encode('utf-8'))\n buf.append(arg)\n buf.append(b\"\\r\\n\")\n return buf"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _done_callback(self, wrapped):\n if wrapped.exception():\n self.set_exception(wrapped.exception())\n else:\n self.set_result(wrapped.result())", "response": "Internal done callback to set the result of the object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate the URL in the LyricWikia format", "response": "def create_url(artist, song):\n \"\"\"Create the URL in the LyricWikia format\"\"\"\n return (__BASE_URL__ +\n '/wiki/{artist}:{song}'.format(artist=urlize(artist),\n song=urlize(song)))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_lyrics(artist, song, linesep='\\n', timeout=None):\n return get_all_lyrics(artist, song, linesep, timeout)[0]", "response": "Retrieve the lyrics of the song and return the first one in case\n multiple versions are available."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves a list of all the lyrics versions of a song.", "response": "def get_all_lyrics(artist, song, linesep='\\n', timeout=None):\n \"\"\"Retrieve a list of all the lyrics versions of a song.\"\"\"\n url = create_url(artist, song)\n response = _requests.get(url, timeout=timeout)\n soup = _BeautifulSoup(response.content, \"html.parser\")\n lyricboxes = soup.findAll('div', {'class': 'lyricbox'})\n\n if not lyricboxes:\n raise LyricsNotFound('Cannot download lyrics')\n\n for lyricbox in lyricboxes:\n for br in lyricbox.findAll('br'):\n br.replace_with(linesep)\n\n return [lyricbox.text.strip() for lyricbox in lyricboxes]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nopening an ARF file creating as necessary.", "response": "def open_file(name, mode=None, driver=None, libver=None, userblock_size=None, **kwargs):\n \"\"\"Open an ARF file, creating as necessary.\n\n Use this instead of h5py.File to ensure that root-level attributes and group\n creation property lists are set correctly.\n\n \"\"\"\n import sys\n import os\n from h5py import h5p\n from h5py._hl import files\n\n try:\n # If the byte string doesn't match the default\n # encoding, just pass it on as-is. Note Unicode\n # objects can always be encoded.\n name = name.encode(sys.getfilesystemencoding())\n except (UnicodeError, LookupError):\n pass\n exists = os.path.exists(name)\n try:\n fcpl = h5p.create(h5p.FILE_CREATE)\n fcpl.set_link_creation_order(\n h5p.CRT_ORDER_TRACKED | h5p.CRT_ORDER_INDEXED)\n except AttributeError:\n # older version of h5py\n fp = files.File(name, mode=mode, driver=driver,\n libver=libver, **kwargs)\n else:\n fapl = files.make_fapl(driver, libver, **kwargs)\n fp = files.File(files.make_fid(name, mode, userblock_size, fapl, fcpl))\n\n if not exists and fp.mode == 'r+':\n set_attributes(fp,\n arf_library='python',\n arf_library_version=__version__,\n arf_version=spec_version)\n return fp"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new ARF entry under a group.", "response": "def create_entry(group, name, timestamp, **attributes):\n \"\"\"Create a new ARF entry under group, setting required attributes.\n\n An entry is an abstract collection of data which all refer to the same time\n frame. Data can include physiological recordings, sound recordings, and\n derived data such as spike times and labels. See add_data() for information\n on how data are stored.\n\n name -- the name of the new entry. any valid python string.\n\n timestamp -- timestamp of entry (datetime object, or seconds since\n January 1, 1970). Can be an integer, a float, or a tuple\n of integers (seconds, microsceconds)\n\n Additional keyword arguments are set as attributes on created entry.\n\n Returns: newly created entry object\n\n \"\"\"\n # create group using low-level interface to store creation order\n from h5py import h5p, h5g, _hl\n try:\n gcpl = h5p.create(h5p.GROUP_CREATE)\n gcpl.set_link_creation_order(\n h5p.CRT_ORDER_TRACKED | h5p.CRT_ORDER_INDEXED)\n except AttributeError:\n grp = group.create_group(name)\n else:\n name, lcpl = group._e(name, lcpl=True)\n grp = _hl.group.Group(h5g.create(group.id, name, lcpl=lcpl, gcpl=gcpl))\n set_uuid(grp, attributes.pop(\"uuid\", None))\n set_attributes(grp,\n timestamp=convert_timestamp(timestamp),\n **attributes)\n return grp"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_dataset(group, name, data, units='', datatype=DataTypes.UNDEFINED,\n chunks=True, maxshape=None, compression=None,\n **attributes):\n \"\"\"Create an ARF dataset under group, setting required attributes\n\n Required arguments:\n name -- the name of dataset in which to store the data\n data -- the data to store\n\n Data can be of the following types:\n\n * sampled data: an N-D numerical array of measurements\n * \"simple\" event data: a 1-D array of times\n * \"complex\" event data: a 1-D array of records, with field 'start' required\n\n Optional arguments:\n datatype -- a code defining the nature of the data in the channel\n units -- channel units (optional for sampled data, otherwise required)\n sampling_rate -- required for sampled data and event data with units=='samples'\n\n Arguments passed to h5py:\n maxshape -- make the node resizable up to this shape. Use None for axes that\n need to be unlimited.\n chunks -- specify the chunk size. The optimal chunk size depends on the\n intended use of the data. For single-channel sampled data the\n auto-chunking (True) is probably best.\n compression -- compression strategy. Can be 'gzip', 'szip', 'lzf' or an integer\n in range(10) specifying gzip(N). Only gzip is really portable.\n\n Additional arguments are set as attributes on the created dataset\n\n Returns the created dataset\n \"\"\"\n from numpy import asarray\n srate = attributes.get('sampling_rate', None)\n # check data validity before doing anything\n if not hasattr(data, 'dtype'):\n data = asarray(data)\n if data.dtype.kind in ('S', 'O', 'U'):\n raise ValueError(\n \"data must be in array with numeric or compound type\")\n if data.dtype.kind == 'V':\n if 'start' not in data.dtype.names:\n raise ValueError(\"complex event data requires 'start' field\")\n if not isinstance(units, (list, tuple)):\n raise ValueError(\"complex event data requires sequence of units\")\n if not len(units) == len(data.dtype.names):\n raise ValueError(\"number of units doesn't match number of fields\")\n if units == '':\n if srate is None or not srate > 0:\n raise ValueError(\n \"unitless data assumed time series and requires sampling_rate attribute\")\n elif units == 'samples':\n if srate is None or not srate > 0:\n raise ValueError(\n \"data with units of 'samples' requires sampling_rate attribute\")\n # NB: can't really catch case where sampled data has units but doesn't\n # have sampling_rate attribute\n\n dset = group.create_dataset(\n name, data=data, maxshape=maxshape, chunks=chunks, compression=compression)\n set_attributes(dset, units=units, datatype=datatype, **attributes)\n return dset", "response": "Create an ARF dataset under the specified group."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_table(group, name, dtype, **attributes):\n dset = group.create_dataset(\n name, shape=(0,), dtype=dtype, maxshape=(None,))\n set_attributes(dset, **attributes)\n return dset", "response": "Create a new array dataset under group with compound datatype and maxshape = None."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef append_data(dset, data):\n N = data.shape[0] if hasattr(data, 'shape') else 1\n if N == 0:\n return\n oldlen = dset.shape[0]\n newlen = oldlen + N\n dset.resize(newlen, axis=0)\n dset[oldlen:] = data", "response": "Append data to dset along axis 0."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the ARF version attribute of file for compatibility.", "response": "def check_file_version(file):\n \"\"\"Check the ARF version attribute of file for compatibility.\n\n Raises DeprecationWarning for backwards-incompatible files, FutureWarning\n for (potentially) forwards-incompatible files, and UserWarning for files\n that may not have been created by an ARF library.\n\n Returns the version for the file\n\n \"\"\"\n from distutils.version import StrictVersion as Version\n try:\n ver = file.attrs.get('arf_version', None)\n if ver is None:\n ver = file.attrs['arf_library_version']\n except KeyError:\n raise UserWarning(\n \"Unable to determine ARF version for {0.filename};\"\n \"created by another program?\".format(file))\n try:\n # if the attribute is stored as a string, it's ascii-encoded\n ver = ver.decode(\"ascii\")\n except (LookupError, AttributeError):\n pass\n # should be backwards compatible after 1.1\n file_version = Version(ver)\n if file_version < Version('1.1'):\n raise DeprecationWarning(\n \"ARF library {} may have trouble reading file \"\n \"version {} (< 1.1)\".format(version, file_version))\n elif file_version >= Version('3.0'):\n raise FutureWarning(\n \"ARF library {} may be incompatible with file \"\n \"version {} (>= 3.0)\".format(version, file_version))\n return file_version"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_attributes(node, overwrite=True, **attributes):\n aset = node.attrs\n for k, v in attributes.items():\n if not overwrite and k in aset:\n pass\n elif v is None:\n if k in aset:\n del aset[k]\n else:\n aset[k] = v", "response": "Set multiple attributes on a node."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef keys_by_creation(group):\n from h5py import h5\n out = []\n try:\n group._id.links.iterate(\n out.append, idx_type=h5.INDEX_CRT_ORDER, order=h5.ITER_INC)\n except (AttributeError, RuntimeError):\n # pre 2.2 shim\n def f(name):\n if name.find(b'/', 1) == -1:\n out.append(name)\n group._id.links.visit(\n f, idx_type=h5.INDEX_CRT_ORDER, order=h5.ITER_INC)\n return map(group._d, out)", "response": "Returns a sequence of links in group in order of creation."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts an ARF timestamp from an object to an ARF timestamp.", "response": "def convert_timestamp(obj):\n \"\"\"Make an ARF timestamp from an object.\n\n Argument can be a datetime.datetime object, a time.struct_time, an integer,\n a float, or a tuple of integers. The returned value is a numpy array with\n the integer number of seconds since the Epoch and any additional\n microseconds.\n\n Note that because floating point values are approximate, the conversion\n between float and integer tuple may not be reversible.\n\n \"\"\"\n import numbers\n from datetime import datetime\n from time import mktime, struct_time\n from numpy import zeros\n\n out = zeros(2, dtype='int64')\n if isinstance(obj, datetime):\n out[0] = mktime(obj.timetuple())\n out[1] = obj.microsecond\n elif isinstance(obj, struct_time):\n out[0] = mktime(obj)\n elif isinstance(obj, numbers.Integral):\n out[0] = obj\n elif isinstance(obj, numbers.Real):\n out[0] = obj\n out[1] = (obj - out[0]) * 1e6\n else:\n try:\n out[:2] = obj[:2]\n except:\n raise TypeError(\"unable to convert %s to timestamp\" % obj)\n return out"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting an ARF timestamp to a datetime. datetime object", "response": "def timestamp_to_datetime(timestamp):\n \"\"\"Convert an ARF timestamp to a datetime.datetime object (naive local time)\"\"\"\n from datetime import datetime, timedelta\n obj = datetime.fromtimestamp(timestamp[0])\n return obj + timedelta(microseconds=int(timestamp[1]))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_uuid(obj, uuid=None):\n from uuid import uuid4, UUID\n if uuid is None:\n uuid = uuid4()\n elif isinstance(uuid, bytes):\n if len(uuid) == 16:\n uuid = UUID(bytes=uuid)\n else:\n uuid = UUID(hex=uuid)\n\n if \"uuid\" in obj.attrs:\n del obj.attrs[\"uuid\"]\n obj.attrs.create(\"uuid\", str(uuid).encode('ascii'), dtype=\"|S36\")", "response": "Set the uuid attribute of an HDF5 object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_uuid(obj):\n # TODO: deprecate null uuid ret val\n from uuid import UUID\n try:\n uuid = obj.attrs['uuid']\n except KeyError:\n return UUID(int=0)\n # convert to unicode for python 3\n try:\n uuid = uuid.decode('ascii')\n except (LookupError, AttributeError):\n pass\n return UUID(uuid)", "response": "Return the uuid for obj or null uuid if none is set"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef count_children(obj, type=None):\n if type is None:\n return len(obj)\n else:\n # there doesn't appear to be any hdf5 function for getting this\n # information without inspecting each child, which makes this somewhat\n # slow\n return sum(1 for x in obj if obj.get(x, getclass=True) is type)", "response": "Return the number of children of obj optionally restricting by class"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a dict keyed by value", "response": "def _todict(cls):\n \"\"\" generate a dict keyed by value \"\"\"\n return dict((getattr(cls, attr), attr) for attr in dir(cls) if not attr.startswith('_'))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_template(template_name,fields=None):\n '''get_template will return a template in the template folder,\n with some substitutions (eg, {'{{ graph | safe }}':\"fill this in!\"}\n '''\n template = None\n if not template_name.endswith('.html'):\n template_name = \"%s.html\" %(template_name)\n here = \"%s/cli/app/templates\" %(get_installdir())\n template_path = \"%s/%s\" %(here,template_name)\n if os.path.exists(template_path):\n template = ''.join(read_file(template_path))\n if fields is not None:\n for tag,sub in fields.items():\n template = template.replace(tag,sub)\n return template", "response": "get_template will return a template in the template folder with some substitutions"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef container_similarity_vector(container1=None,packages_set=None,by=None):\n '''container similarity_vector is similar to compare_packages, but intended\n to compare a container object (singularity image or singularity hub container)\n to a list of packages. If packages_set is not provided, the default used is \n 'docker-os'. This can be changed to 'docker-library', or if the user wants a custom\n list, should define custom_set.\n :param container1: singularity image or singularity hub container.\n :param packages_set: a name of a package set, provided are docker-os and docker-library\n :by: metrics to compare by (files.txt and or folders.txt)\n '''\n\n if by == None:\n by = ['files.txt']\n\n if not isinstance(by,list):\n by = [by]\n if not isinstance(packages_set,list):\n packages_set = [packages_set]\n\n comparisons = dict()\n\n for b in by:\n bot.debug(\"Starting comparisons for %s\" %b)\n df = pandas.DataFrame(columns=packages_set)\n for package2 in packages_set:\n sim = calculate_similarity(container1=container1,\n image_package2=package2,\n by=b)[b]\n \n name1 = os.path.basename(package2).replace('.img.zip','')\n bot.debug(\"container vs. %s: %s\" %(name1,sim))\n df.loc[\"container\",package2] = sim\n df.columns = [os.path.basename(x).replace('.img.zip','') for x in df.columns.tolist()]\n comparisons[b] = df\n return comparisons", "response": "container similarity_vector is similar to compare_packages but intended\n to compare a container object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compare_containers(container1=None, container2=None):\n\n '''compare_containers will generate a data structure with common and unique files to\n two images. If environmental variable SINGULARITY_HUB is set, will use container\n database objects.\n :param container1: first container for comparison\n :param container2: second container for comparison if either not defined must include\n default compares just files\n '''\n\n # Get files and folders for each\n container1_guts = get_container_contents(split_delim=\"\\n\",\n container=container1)['all']\n container2_guts = get_container_contents(split_delim=\"\\n\",\n container=container2)['all']\n\n # Do the comparison for each metric\n return compare_lists(container1_guts, container2_guts)", "response": "This function will generate a data structure with common and unique files to\n two images."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef compare_lists(list1,list2):\n '''compare lists is the lowest level that drives compare_containers and\n compare_packages. It returns a comparison object (dict) with the unique,\n total, and intersecting things between two lists\n :param list1: the list for container1\n :param list2: the list for container2\n '''\n intersect = list(set(list1).intersection(list2))\n unique1 = list(set(list1).difference(list2))\n unique2 = list(set(list2).difference(list1))\n\n # Return data structure\n comparison = {\"intersect\":intersect,\n \"unique1\": unique1,\n \"unique2\": unique2,\n \"total1\": len(list1),\n \"total2\": len(list2)}\n return comparison", "response": "compare lists is the lowest level that drives compare_containers and compare_packages. It returns a dict with the unique total and intersecting things between two lists\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npackages a node into a container", "response": "def package_node(root=None, name=None):\n '''package node aims to package a (present working node) for a user into\n a container. This assumes that the node is a single partition.\n \n :param root: the root of the node to package, default is /\n :param name: the name for the image. If not specified, will use machine's\n\n psutil.disk_partitions()\n\n '''\n\n if name is None:\n name = platform.node()\n\n if root is None:\n root = \"/\"\n\n tmpdir = tempfile.mkdtemp()\n image = \"%s/%s.tgz\" %(tmpdir,name)\n\n print(\"Preparing to package root %s into %s\" %(root,name))\n cmd = [\"tar\",\"--one-file-system\",\"-czvSf\", image, root,\"--exclude\",image]\n output = run_command(cmd)\n return image"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_build_template(template_name,params=None,to_file=None):\n '''get_build template returns a string or file for a particular build template, which is\n intended to build a version of a Singularity image on a cloud resource.\n :param template_name: the name of the template to retrieve in build/scripts\n :param params: (if needed) a dictionary of parameters to substitute in the file\n :param to_file: if defined, will write to file. Default returns string.\n '''\n base = get_installdir()\n template_folder = \"%s/build/scripts\" %(base)\n template_file = \"%s/%s\" %(template_folder,template_name)\n if os.path.exists(template_file):\n bot.debug(\"Found template %s\" %template_file)\n\n # Implement when needed - substitute params here\n # Will need to read in file instead of copying below\n # if params != None:\n \n if to_file is not None:\n shutil.copyfile(template_file,to_file)\n bot.debug(\"Template file saved to %s\" %to_file)\n return to_file\n\n # If the user wants a string\n content = ''.join(read_file(template_file)) \n return content\n\n\n else:\n bot.warning(\"Template %s not found.\" %template_file)", "response": "get_build template returns a string or file for a particular build template"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sniff_extension(file_path,verbose=True):\n '''sniff_extension will attempt to determine the file type based on the extension,\n and return the proper mimetype\n :param file_path: the full path to the file to sniff\n :param verbose: print stuff out\n '''\n mime_types = { \"xls\": 'application/vnd.ms-excel',\n \"xlsx\": 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',\n \"xml\": 'text/xml',\n \"ods\": 'application/vnd.oasis.opendocument.spreadsheet',\n \"csv\": 'text/plain',\n \"tmpl\": 'text/plain',\n \"pdf\": 'application/pdf',\n \"php\": 'application/x-httpd-php',\n \"jpg\": 'image/jpeg',\n \"png\": 'image/png',\n \"gif\": 'image/gif',\n \"bmp\": 'image/bmp',\n \"txt\": 'text/plain',\n \"doc\": 'application/msword',\n \"js\": 'text/js',\n \"swf\": 'application/x-shockwave-flash',\n \"mp3\": 'audio/mpeg',\n \"zip\": 'application/zip',\n \"simg\": 'application/zip',\n \"rar\": 'application/rar',\n \"tar\": 'application/tar',\n \"arj\": 'application/arj',\n \"cab\": 'application/cab',\n \"html\": 'text/html',\n \"htm\": 'text/html',\n \"default\": 'application/octet-stream',\n \"folder\": 'application/vnd.google-apps.folder',\n \"img\" : \"application/octet-stream\" }\n\n ext = os.path.basename(file_path).split('.')[-1]\n\n mime_type = mime_types.get(ext,None)\n\n if mime_type == None:\n mime_type = mime_types['txt']\n\n if verbose==True:\n bot.info(\"%s --> %s\" %(file_path, mime_type))\n\n return mime_type", "response": "sniff_extension will attempt to determine the file type based on the extension and return the proper mimetype"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef zip_up(file_list,zip_name,output_folder=None):\n '''zip_up will zip up some list of files into a package (.zip)\n :param file_list: a list of files to include in the zip.\n :param output_folder: the output folder to create the zip in. If not \n :param zip_name: the name of the zipfile to return.\n specified, a temporary folder will be given.\n '''\n tmpdir = tempfile.mkdtemp()\n \n # Make a new archive \n output_zip = \"%s/%s\" %(tmpdir,zip_name)\n zf = zipfile.ZipFile(output_zip, \"w\", zipfile.ZIP_DEFLATED, allowZip64=True)\n\n # Write files to zip, depending on type\n for filename,content in file_list.items():\n\n bot.debug(\"Adding %s to package...\" %filename)\n\n # If it's the files list, move files into the archive\n if filename.lower() == \"files\":\n if not isinstance(content,list): \n content = [content]\n for copyfile in content:\n zf.write(copyfile,os.path.basename(copyfile))\n os.remove(copyfile)\n\n else:\n\n output_file = \"%s/%s\" %(tmpdir, filename)\n \n # If it's a list, write to new file, and save\n if isinstance(content,list):\n write_file(output_file,\"\\n\".join(content))\n \n # If it's a dict, save to json\n elif isinstance(content,dict):\n write_json(content,output_file)\n\n # If bytes, need to decode\n elif isinstance(content,bytes):\n write_file(output_file,content.decode('utf-8'))\n \n # String or other\n else: \n output_file = write_file(output_file,content)\n\n if os.path.exists(output_file):\n zf.write(output_file,filename)\n os.remove(output_file)\n\n # Close the zip file \n zf.close()\n\n if output_folder is not None:\n shutil.copyfile(output_zip,\"%s/%s\"%(output_folder,zip_name))\n shutil.rmtree(tmpdir)\n output_zip = \"%s/%s\"%(output_folder,zip_name)\n\n return output_zip", "response": "This function will zip up some list of files into a single file in a folder."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_container_contents(container, split_delim=None):\n '''get_container_contents will return a list of folders and or files\n for a container. The environmental variable SINGULARITY_HUB being set\n means that container objects are referenced instead of packages\n :param container: the container to get content for\n :param gets: a list of file names to return, without parent folders\n :param split_delim: if defined, will split text by split delimiter\n '''\n\n # We will look for everything in guts, then return it\n guts = dict()\n\n SINGULARITY_HUB = os.environ.get('SINGULARITY_HUB',\"False\")\n\n # Visualization deployed local or elsewhere\n if SINGULARITY_HUB == \"False\":\n file_obj,tar = get_image_tar(container) \n guts = extract_guts(image_path=container, tar=tar)\n delete_image_tar(file_obj, tar)\n\n # Visualization deployed by singularity hub\n else: \n \n # user has provided a package, but not a container\n if container == None:\n guts = load_package(image_package,get=gets)\n\n # user has provided a container, but not a package\n else:\n for sfile in container.files:\n for gut_key in gets: \n if os.path.basename(sfile['name']) == gut_key:\n if split_delim == None:\n guts[gut_key] = requests.get(sfile['mediaLink']).text\n else:\n guts[gut_key] = requests.get(sfile['mediaLink']).text.split(split_delim)\n\n return guts", "response": "get_container_contents will return a list of folders or files\n for a container"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_image_file_hash(image_path):\n '''get_image_hash will return an md5 hash of the file based on a criteria level.\n :param level: one of LOW, MEDIUM, HIGH\n :param image_path: full path to the singularity image\n '''\n hasher = hashlib.md5()\n with open(image_path, \"rb\") as f:\n for chunk in iter(lambda: f.read(4096), b\"\"):\n hasher.update(chunk)\n return hasher.hexdigest()", "response": "get_image_hash will return an md5 hash of the file based on a criteria level"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef container_difference(container=None,container_subtract=None,image_package=None,\n image_package_subtract=None,comparison=None):\n '''container_difference will return a data structure to render an html \n tree (graph) of the differences between two images or packages. The second\n container is subtracted from the first\n :param container: the primary container object (to subtract from)\n :param container_subtract: the second container object to remove\n :param image_package: a zipped package for image 1, created with package\n :param image_package_subtract: a zipped package for subtraction image, created with package\n :param comparison: the comparison result object for the tree. If provided,\n will skip over function to obtain it.\n '''\n if comparison == None:\n comparison = compare_containers(container1=container,\n container2=container_subtract,\n image_package1=image_package,\n image_package2=image_package_subtract,\n by=['files.txt','folders.txt'])\n\n files = comparison[\"files.txt\"]['unique1']\n folders = comparison['folders.txt']['unique1']\n tree = make_container_tree(folders=folders,\n files=files)\n return tree", "response": "container_difference will return a data structure to render an html \n tree"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef container_similarity(container1=None,container2=None,image_package1=None,\n image_package2=None,comparison=None):\n '''container_sim will return a data structure to render an html tree \n (graph) of the intersection (commonalities) between two images or packages\n :param container1: the first container object\n :param container2: the second container object if either not defined, need\n :param image_package1: a packaged container1 (produced by package)\n :param image_package2: a packaged container2 (produced by package)\n :param comparison: the comparison result object for the tree. If provided,\n will skip over function to obtain it.\n '''\n if comparison == None:\n comparison = compare_containers(container1=container1,\n container2=container2,\n image_package1=image_package1,\n image_package2=image_package2,\n by=['files.txt','folders.txt'])\n files = comparison[\"files.txt\"]['intersect']\n folders = comparison['folders.txt']['intersect']\n tree = make_container_tree(folders=folders,\n files=files)\n return tree", "response": "container_sim will return a data structure to render an html tree \n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef container_tree(container=None,image_package=None):\n '''tree will render an html tree (graph) of a container\n '''\n\n guts = get_container_contents(container=container,\n image_package=image_package,\n split_delim=\"\\n\")\n\n # Make the tree and return it\n tree = make_container_tree(folders = guts[\"folders.txt\"],\n files = guts['files.txt'])\n return tree", "response": "will render an html tree of a container"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_container_tree(folders,files,path_delim=\"/\",parse_files=True):\n '''make_container_tree will convert a list of folders and files into a json structure that represents a graph.\n :param folders: a list of folders in the image\n :param files: a list of files in the folder\n :param parse_files: return 'files' lookup in result, to associate ID of node with files (default True)\n :param path_delim: the path delimiter, default is '/'\n '''\n nodes = {} # first we will make a list of nodes\n lookup = {}\n count = 1 # count will hold an id for nodes\n max_depth = 0\n for folder in folders:\n if folder != \".\":\n folder = re.sub(\"^[.]/\",\"\",folder)\n path_components = folder.split(path_delim)\n for p in range(len(path_components)):\n path_component = path_components[p]\n fullpath = path_delim.join(path_components[0:p+1])\n # Have we created the node yet?\n if fullpath not in lookup:\n lookup[fullpath] = count\n node = {\"id\":count,\"name\":path_component,\"path\":fullpath,\"level\":p,\"children\":[]}\n count +=1\n # Did we find a deeper level?\n if p > max_depth:\n max_depth = p\n # Does the node have a parent?\n if p==0: # base node, no parent\n parent_id = 0\n else: # look up the parent id\n parent_path = path_delim.join(path_components[0:p])\n parent_id = lookup[parent_path] \n node[\"parent\"] = parent_id\n nodes[node['id']] = node \n \n # Now make the graph, we simply append children to their parents\n seen = []\n graph = []\n iters = list(range(max_depth+1)) # 0,1,2,3...\n iters.reverse() # ...3,2,1,0\n iters.pop() # remove 0\n for level in iters:\n children = {x:y for x,y in nodes.items() if y['level'] == level}\n seen = seen + [y['id'] for x,y in children.items()]\n nodes = {x:y for x,y in nodes.items() if y['id'] not in seen}\n for node_id,child_node in children.items():\n if node_id == 0: #base node\n graph[node_id] = child_node\n else:\n parent_id = child_node['parent']\n nodes[parent_id][\"children\"].append(child_node)\n\n # Now add the parents to graph, with name as main lookup\n for parent,parent_info in nodes.items():\n graph.append(parent_info)\n graph = {\"name\":\"base\",\"children\":graph}\n result = {\"graph\":graph,\"lookup\":lookup,\"depth\":max_depth+1}\n\n # Parse files to include in tree\n if parse_files == True:\n file_lookup = {}\n for filey in files:\n filey = re.sub(\"^[.]/\",\"\",filey)\n filepath,filename = os.path.split(filey)\n if filepath in lookup:\n folder_id = lookup[filepath]\n if folder_id in file_lookup:\n file_lookup[folder_id].append(filename)\n else:\n file_lookup[folder_id] = [filename]\n elif filepath == '': # base folder\n if 0 in file_lookup:\n file_lookup[0].append(filename)\n else:\n file_lookup[0] = [filename]\n result['files'] = file_lookup\n\n return result", "response": "This function will convert a list of folders and files into a json structure that represents a graph."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef make_package_tree(matrix=None,labels=None,width=25,height=10,title=None,font_size=None):\n '''make package tree will make a dendrogram comparing a matrix of packages\n :param matrix: a pandas df of packages, with names in index and columns\n :param labels: a list of labels corresponding to row names, will be\n pulled from rows if not defined\n :param title: a title for the plot, if not defined, will be left out.\n :returns a plot that can be saved with savefig\n '''\n from matplotlib import pyplot as plt\n from scipy.cluster.hierarchy import (\n dendrogram, \n linkage\n )\n\n if font_size is None:\n font_size = 8.\n\n from scipy.cluster.hierarchy import cophenet\n from scipy.spatial.distance import pdist\n\n if not isinstance(matrix,pandas.DataFrame):\n bot.info(\"No pandas DataFrame (matrix) of similarities defined, will use default.\")\n matrix = compare_packages()['files.txt']\n title = 'Docker Library Similarity to Base OS'\n\n Z = linkage(matrix, 'ward')\n c, coph_dists = cophenet(Z, pdist(matrix))\n\n if labels == None:\n labels = matrix.index.tolist()\n\n plt.figure(figsize=(width, height))\n\n if title != None:\n plt.title(title)\n\n plt.xlabel('image index')\n plt.ylabel('distance')\n dendrogram(Z,\n leaf_rotation=90., # rotates the x axis labels\n leaf_font_size=font_size, # font size for the x axis labels\n labels=labels)\n return plt", "response": "make a package tree"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_interactive_tree(matrix=None,labels=None):\n '''make interactive tree will return complete html for an interactive tree\n :param title: a title for the plot, if not defined, will be left out.\n '''\n from scipy.cluster.hierarchy import (\n dendrogram, \n linkage,\n to_tree\n )\n\n d3 = None\n from scipy.cluster.hierarchy import cophenet\n from scipy.spatial.distance import pdist\n\n if isinstance(matrix,pandas.DataFrame):\n Z = linkage(matrix, 'ward') # clusters\n T = to_tree(Z, rd=False)\n\n if labels == None:\n labels = matrix.index.tolist()\n lookup = dict(zip(range(len(labels)), labels))\n\n # Create a dendrogram object without plotting\n dend = dendrogram(Z,no_plot=True,\n orientation=\"right\",\n leaf_rotation=90., # rotates the x axis labels\n leaf_font_size=8., # font size for the x axis labels\n labels=labels)\n\n d3 = dict(children=[], name=\"root\")\n add_node(T, d3)\n label_tree(d3[\"children\"][0],lookup)\n else:\n bot.warning('Please provide data as pandas Data Frame.')\n return d3", "response": "make an interactive tree"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_node(node, parent):\n '''add_node will add a node to it's parent\n '''\n newNode = dict(node_id=node.id, children=[])\n parent[\"children\"].append(newNode)\n if node.left: add_node(node.left, newNode)\n if node.right: add_node(node.right, newNode)", "response": "add_node will add a node to it s parent\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef label_tree(n,lookup):\n '''label tree will again recursively label the tree\n :param n: the root node, usually d3['children'][0]\n :param lookup: the node/id lookup\n '''\n if len(n[\"children\"]) == 0:\n leaves = [lookup[n[\"node_id\"]]]\n else:\n leaves = reduce(lambda ls, c: ls + label_tree(c,lookup), n[\"children\"], [])\n del n[\"node_id\"]\n n[\"name\"] = name = \"|||\".join(sorted(map(str, leaves)))\n return leaves", "response": "label tree will recursively label the tree"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nextract app will extract metadata for one or more apps", "response": "def extract_apps(image, app_names):\n ''' extract app will extract metadata for one or more apps\n Parameters\n ==========\n image: the absolute path to the image\n app_name: the name of the app under /scif/apps\n\n '''\n apps = dict()\n\n if isinstance(app_names, tuple):\n app_names = list(app_names)\n if not isinstance(app_names, list):\n app_names = [app_names]\n if len(app_names) == 0:\n return apps\n\n for app_name in app_names:\n metadata = dict()\n\n # Inspect: labels, env, runscript, tests, help\n try:\n inspection = json.loads(Client.inspect(image, app=app_name))\n del inspection['data']['attributes']['deffile']\n metadata['inspect'] = inspection\n\n # If illegal characters prevent load, not much we can do\n except:\n pass\n apps[app_name] = metadata\n return apps"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef download_repo(repo_url, destination, commit=None):\n '''download_repo\n :param repo_url: the url of the repo to clone from\n :param destination: the full path to the destination for the repo\n '''\n command = \"git clone %s %s\" % (repo_url, destination)\n os.system(command)\n return destination", "response": "downloads the repo from the specified url to the specified destination"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets tags will return a list of tags that describe the software in an image", "response": "def get_tags(container=None,\n search_folders=None,\n file_list=None,\n return_unique=True):\n\n '''get tags will return a list of tags that describe the software in an image,\n meaning inside of a paricular folder. If search_folder is not defined, uses lib\n :param container: if provided, will use container as image. Can also provide\n :param image_package: if provided, can be used instead of container\n :param search_folders: specify one or more folders to look for tags \n :param file_list: the list of files\n :param return_unique: return unique files in folders. Default True.\n Default is 'bin'\n\n ::notes\n \n The algorithm works as follows:\n 1) first compare package to set of base OS (provided with shub)\n 2) subtract the most similar os from image, leaving \"custom\" files\n 3) organize custom files into dict based on folder name\n 4) return search_folders as tags\n\n '''\n if file_list is None:\n file_list = get_container_contents(container, split_delim='\\n')['all']\n\n if search_folders == None:\n search_folders = 'bin'\n\n if not isinstance(search_folders,list):\n search_folders = [search_folders]\n\n tags = []\n for search_folder in search_folders:\n for file_name in file_list:\n if search_folder in file_name:\n tags.append(file_name)\n\n if return_unique == True:\n tags = list(set(tags))\n return tags"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfiling counts will return a list of files that match one or more regular expressions.", "response": "def file_counts(container=None, \n patterns=None, \n image_package=None, \n file_list=None):\n\n '''file counts will return a list of files that match one or more regular expressions.\n if no patterns is defined, a default of readme is used. All patterns and files are made\n case insensitive.\n\n Parameters\n ==========\n :param container: if provided, will use container as image. Can also provide\n :param image_package: if provided, can be used instead of container\n :param patterns: one or more patterns (str or list) of files to search for.\n :param diff: the difference between a container and it's parent OS from get_diff\n if not provided, will be generated.\n\n '''\n if file_list is None:\n file_list = get_container_contents(container, split_delim='\\n')['all']\n\n if patterns == None:\n patterns = 'readme'\n\n if not isinstance(patterns,list):\n patterns = [patterns]\n\n count = 0\n for pattern in patterns:\n count += len([x for x in file_list if re.search(pattern.lower(),x.lower())])\n bot.info(\"Total files matching patterns is %s\" %count)\n return count"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef extension_counts(container=None, file_list=None, return_counts=True):\n '''extension counts will return a dictionary with counts of file extensions for\n an image.\n :param container: if provided, will use container as image. Can also provide\n :param image_package: if provided, can be used instead of container\n :param file_list: the complete list of files\n :param return_counts: return counts over dict with files. Default True\n '''\n if file_list is None:\n file_list = get_container_contents(container, split_delim='\\n')['all']\n\n extensions = dict()\n for item in file_list:\n filename,ext = os.path.splitext(item)\n if ext == '':\n if return_counts == False:\n extensions = update_dict(extensions,'no-extension',item)\n else:\n extensions = update_dict_sum(extensions,'no-extension')\n else:\n if return_counts == False:\n extensions = update_dict(extensions,ext,item)\n else:\n extensions = update_dict_sum(extensions,ext)\n\n return extensions", "response": "returns a dictionary with counts of file extensions for a container and an image."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nassesses the differences between two images on each level of returns a dictionary with files that are the same different and an overall score.", "response": "def assess_differences(image_file1,\n image_file2,\n levels=None,\n version=None,\n size_heuristic=False,\n guts1=None,\n guts2=None):\n\n '''assess_differences will compare two images on each level of \n reproducibility, returning for each level a dictionary with files\n that are the same, different, and an overall score.\n :param size_heuristic: if True, assess root owned files based on size\n :param guts1,guts2: the result (dict with sizes,roots,etc) from get_content_hashes\n '''\n if levels is None:\n levels = get_levels(version=version)\n\n reports = dict()\n scores = dict()\n\n for level_name, level_filter in levels.items():\n contenders = []\n different = []\n setdiff = []\n same = 0\n\n # Compare the dictionary of file:hash between two images, and get root owned lookup\n if guts1 is None:\n guts1 = get_content_hashes(image_path=image_file1,\n level_filter=level_filter)\n # tag_root=True\n # include_sizes=True\n \n if guts2 is None:\n guts2 = get_content_hashes(image_path=image_file2,\n level_filter=level_filter)\n \n print(level_name)\n files = list(set(list(guts1['hashes'].keys()) + list(guts2['hashes'].keys())))\n\n for file_name in files:\n\n # If it's not in one or the other\n if file_name not in guts1['hashes'] or file_name not in guts2['hashes']:\n setdiff.append(file_name)\n\n else:\n if guts1['hashes'][file_name] == guts2['hashes'][file_name]:\n same+=1\n else:\n\n # If the file is root owned, we compare based on size\n if size_heuristic == True:\n if guts1['root_owned'][file_name] or guts2['root_owned'][file_name]:\n if guts1['sizes'][file_name] == guts2['sizes'][file_name]: \n same+=1\n else:\n different.append(file_name)\n else:\n # Otherwise, we can assess the bytes content by reading it\n contenders.append(file_name)\n else:\n contenders.append(file_name)\n\n # If the user wants identical (meaning extraction order and timestamps)\n if level_name == \"IDENTICAL\":\n different = different + contenders\n\n # Otherwise we need to check based on byte content\n else: \n if len(contenders) > 0:\n for rogue in contenders:\n hashy1 = extract_content(image_file1, rogue, return_hash=True)\n hashy2 = extract_content(image_file2, rogue, return_hash=True)\n \n # If we can't compare, we use size as a heuristic\n if hashy1 is None or hashy2 is None: # if one is symlink, could be None\n different.append(file_name) \n elif len(hashy1) == 0 or len(hashy2) == 0:\n if guts1['sizes'][file_name] == guts2['sizes'][file_name]: \n same+=1\n else:\n different.append(file_name) \n elif hashy1 != hashy2:\n different.append(rogue)\n else:\n same+=1\n\n # We use a similar Jacaard coefficient, twice the shared information in the numerator \n # (the intersection, same), as a proportion of the total summed files\n union = len(guts1['hashes']) + len(guts2['hashes'])\n\n report = {'difference': setdiff,\n 'intersect_different': different,\n 'same':same,\n 'union': union}\n \n if union == 0:\n scores[level_name] = 0\n else:\n scores[level_name] = 2*(same) / union\n reports[level_name] = report\n\n reports['scores'] = scores\n return reports"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef include_file(member,file_filter):\n '''include_file will look at a path and determine\n if it matches a regular expression from a level\n '''\n member_path = member.name.replace('.','',1)\n\n if len(member_path) == 0:\n return False\n\n # Does the filter skip it explicitly?\n if \"skip_files\" in file_filter:\n if member_path in file_filter['skip_files']:\n return False\n\n # Include explicitly?\n if \"include_files\" in file_filter:\n if member_path in file_filter['include_files']:\n return True\n\n # Regular expression?\n if \"regexp\" in file_filter:\n if re.search(file_filter[\"regexp\"],member_path):\n return True\n return False", "response": "Returns True if the member matches a regular expression from a level\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_root_owned(member):\n '''assess if a file is root owned, meaning \"root\" or user/group \n id of 0'''\n if member.uid == 0 or member.gid == 0:\n return True\n elif member.uname == 'root' or member.gname == 'root':\n return True\n return False", "response": "assess if a file is root owned meaning root or user / group \n id of 0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndetermine if the filter wants the file to be read for content.", "response": "def assess_content(member,file_filter):\n '''Determine if the filter wants the file to be read for content.\n In the case of yes, we would then want to add the content to the\n hash and not the file object.\n '''\n member_path = member.name.replace('.','',1)\n\n if len(member_path) == 0:\n return False\n\n # Does the filter skip it explicitly?\n if \"skip_files\" in file_filter:\n if member_path in file_filter['skip_files']:\n return False\n\n if \"assess_content\" in file_filter:\n if member_path in file_filter['assess_content']:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_custom_level(regexp=None,description=None,skip_files=None,include_files=None):\n '''get_custom_level will generate a custom level for the user, \n based on a regular expression. If used outside the context of tarsum, the user\n can generate their own named and described filters.\n :param regexp: must be defined, the file filter regular expression\n :param description: optional description\n '''\n if regexp == None:\n regexp = \".\"\n if description is None:\n description = \"This is a custom filter generated by the user.\"\n \n custom = {\"description\":description,\n \"regexp\":regexp}\n\n # Include extra files?\n if include_files is not None:\n if not isinstance(include_files,set):\n include_files = set(include_files)\n custom['include_files'] = include_files\n\n # Skip files?\n if skip_files is not None:\n if not isinstance(skip_files,set):\n skip_files = set(skip_files)\n custom['skip_files'] = skip_files\n\n return custom", "response": "This function returns a custom level for a user based on a regular expression."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmodifies level is intended to add or modify a content type.", "response": "def modify_level(level,field,values,append=True):\n '''modify level is intended to add / modify a content type.\n Default content type is list, meaning the entry is appended.\n If you set append to False, the content will be overwritten\n For any other content type, the entry is overwritten.\n '''\n field = field.lower()\n valid_fields = ['regexp','skip_files','include_files']\n if field not in valid_fields:\n bot.warning(\"%s is not a valid field, skipping. Choices are %s\" %(field,\",\".join(valid_fields)))\n return level\n if append:\n if not isinstance(values,list):\n values = [values]\n if field in level:\n level[field] = level[field] + values\n else:\n level[field] = values\n else:\n level[field] = values\n\n level = make_level_set(level)\n\n return level"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_levels(version=None):\n '''get_levels returns a dictionary of levels (key) and values (dictionaries with\n descriptions and regular expressions for files) for the user. \n :param version: the version of singularity to use (default is 2.2)\n :param include_files: files to add to the level, only relvant if\n '''\n valid_versions = ['2.3','2.2']\n\n if version is None:\n version = \"2.3\" \n version = str(version)\n\n if version not in valid_versions:\n bot.error(\"Unsupported version %s, valid versions are %s\" %(version,\n \",\".join(valid_versions)))\n\n levels_file = os.path.abspath(os.path.join(get_installdir(),\n 'analysis',\n 'reproduce',\n 'data',\n 'reproduce_levels.json'))\n levels = read_json(levels_file)\n if version == \"2.2\":\n # Labels not added until 2.3\n del levels['LABELS']\n\n levels = make_levels_set(levels)\n\n return levels", "response": "get_levels returns a dictionary of levels descriptions and regular expressions for files for the user."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes set efficient will convert all lists of items in levels to a set", "response": "def make_levels_set(levels):\n '''make set efficient will convert all lists of items\n in levels to a set to speed up operations'''\n for level_key,level_filters in levels.items():\n levels[level_key] = make_level_set(level_filters)\n return levels"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake level set will convert one level into a set", "response": "def make_level_set(level):\n '''make level set will convert one level into\n a set'''\n new_level = dict()\n for key,value in level.items():\n if isinstance(value,list):\n new_level[key] = set(value)\n else:\n new_level[key] = value\n return new_level"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextracting the file guts from an in memory tarfile.", "response": "def extract_guts(image_path,\n tar,\n file_filter=None,\n tag_root=True,\n include_sizes=True):\n\n '''extract the file guts from an in memory tarfile. The file is not closed.\n This should not be done for large images.\n '''\n if file_filter is None:\n file_filter = get_level('IDENTICAL')\n\n results = dict()\n digest = dict()\n allfiles = []\n\n if tag_root:\n roots = dict()\n\n if include_sizes: \n sizes = dict()\n\n for member in tar:\n member_name = member.name.replace('.','',1)\n allfiles.append(member_name)\n included = False\n if member.isdir() or member.issym():\n continue\n elif assess_content(member,file_filter):\n digest[member_name] = extract_content(image_path, member.name, return_hash=True)\n included = True\n elif include_file(member,file_filter):\n hasher = hashlib.md5()\n buf = member.tobuf()\n hasher.update(buf)\n digest[member_name] = hasher.hexdigest()\n included = True\n if included:\n if include_sizes:\n sizes[member_name] = member.size\n if tag_root:\n roots[member_name] = is_root_owned(member)\n\n results['all'] = allfiles\n results['hashes'] = digest\n if include_sizes:\n results['sizes'] = sizes\n if tag_root:\n results['root_owned'] = roots\n return results"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting an in memory tar of an image", "response": "def get_memory_tar(image_path):\n '''get an in memory tar of an image. Use carefully, not as reliable\n as get_image_tar\n '''\n byte_array = Client.image.export(image_path)\n file_object = io.BytesIO(byte_array)\n tar = tarfile.open(mode=\"r|*\", fileobj=file_object)\n return (file_object,tar)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting an image tar either written in memory or to the file system.", "response": "def get_image_tar(image_path):\n '''get an image tar, either written in memory or to\n the file system. file_obj will either be the file object,\n or the file itself.\n '''\n bot.debug('Generate file system tar...') \n file_obj = Client.image.export(image_path=image_path)\n if file_obj is None:\n bot.error(\"Error generating tar, exiting.\")\n sys.exit(1)\n tar = tarfile.open(file_obj)\n return file_obj, tar"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef delete_image_tar(file_obj, tar):\n '''delete image tar will close a file object (if extracted into\n memory) or delete from the file system (if saved to disk)'''\n try:\n file_obj.close()\n except:\n tar.close()\n if os.path.exists(file_obj):\n os.remove(file_obj)\n deleted = True\n bot.debug('Deleted temporary tar.') \n return deleted", "response": "delete image tar will close a file object or delete from the file system"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef extract_content(image_path, member_name, return_hash=False):\n '''extract_content will extract content from an image using cat.\n If hash=True, a hash sum is returned instead\n '''\n if member_name.startswith('./'):\n member_name = member_name.replace('.','',1)\n if return_hash:\n hashy = hashlib.md5()\n\n try:\n content = Client.execute(image_path,'cat %s' %(member_name))\n except:\n return None\n\n if not isinstance(content,bytes):\n content = content.encode('utf-8')\n content = bytes(content)\n\n # If permissions don't allow read, return None\n if len(content) == 0:\n return None\n if return_hash:\n hashy.update(content)\n return hashy.hexdigest()\n return content", "response": "extract_content will extract content from an image using cat."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_build(build_dir, params, verbose=True):\n '''run_build takes a build directory and params dictionary, and does the following:\n - downloads repo to a temporary directory\n - changes branch or commit, if needed\n - creates and bootstraps singularity image from Singularity file\n - returns a dictionary with: \n image (path), metadata (dict)\n\n The following must be included in params: \n spec_file, repo_url, branch, commit\n\n '''\n\n # Download the repository\n\n download_repo(repo_url=params['repo_url'],\n destination=build_dir)\n\n os.chdir(build_dir)\n\n if params['branch'] != None:\n bot.info('Checking out branch %s' %params['branch'])\n os.system('git checkout %s' %(params['branch']))\n else:\n params['branch'] = \"master\"\n\n\n # Set the debug level\n\n Client.debug = params['debug']\n\n # Commit\n\n if params['commit'] not in [None,'']:\n bot.info('Checking out commit %s' %params['commit'])\n os.system('git checkout %s .' %(params['commit']))\n\n # From here on out commit is used as a unique id, if we don't have one, we use current\n else:\n params['commit'] = os.popen('git log -n 1 --pretty=format:\"%H\"').read()\n bot.warning(\"commit not specified, setting to current %s\" %params['commit'])\n\n # Dump some params for the builder, in case it fails after this\n passing_params = \"/tmp/params.pkl\"\n pickle.dump(params, open(passing_params,'wb'))\n\n # Now look for spec file\n if os.path.exists(params['spec_file']):\n bot.info(\"Found spec file %s in repository\" %params['spec_file'])\n\n # If the user has a symbolic link\n if os.path.islink(params['spec_file']):\n bot.info(\"%s is a symbolic link.\" %params['spec_file'])\n params['spec_file'] = os.path.realpath(params['spec_file'])\n\n # START TIMING\n start_time = datetime.now()\n\n # Secure Build\n image = Client.build(recipe=params['spec_file'],\n build_folder=build_dir,\n isolated=True)\n\n # Save has for metadata (also is image name)\n version = get_image_file_hash(image)\n params['version'] = version\n pickle.dump(params, open(passing_params,'wb'))\n\n # Rename image to be hash\n finished_image = \"%s/%s.simg\" %(os.path.dirname(image), version)\n image = shutil.move(image, finished_image)\n\n final_time = (datetime.now() - start_time).seconds\n bot.info(\"Final time of build %s seconds.\" %final_time) \n\n # Did the container build successfully?\n test_result = test_container(image)\n if test_result['return_code'] != 0:\n bot.error(\"Image failed to build, cancelling.\")\n sys.exit(1)\n\n # Get singularity version\n singularity_version = Client.version()\n Client.debug = False\n inspect = Client.inspect(image) # this is a string\n Client.debug = params['debug']\n\n # Get information on apps\n Client.debug = False\n app_names = Client.apps(image)\n Client.debug = params['debug']\n apps = extract_apps(image, app_names)\n \n metrics = {'build_time_seconds': final_time,\n 'singularity_version': singularity_version,\n 'singularity_python_version': singularity_python_version, \n 'inspect': inspect,\n 'version': version,\n 'apps': apps}\n \n output = {'image':image,\n 'metadata':metrics,\n 'params':params }\n\n return output\n\n else:\n\n # Tell the user what is actually there\n present_files = glob(\"*\")\n bot.error(\"Build file %s not found in repository\" %params['spec_file'])\n bot.info(\"Found files are %s\" %\"\\n\".join(present_files))\n # Params have been exported, will be found by log\n sys.exit(1)", "response": "This function downloads the repository and builds the singularity image from Singularity and creates the singularity image from Singularity."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_build_data(build_dir, data, secret, \n response_url=None,clean_up=True):\n '''finish build sends the build and data (response) to a response url\n :param build_dir: the directory of the build\n :response_url: where to send the response. If None, won't send\n :param data: the data object to send as a post\n :param clean_up: If true (default) removes build directory\n '''\n # Send with Authentication header\n body = '%s|%s|%s|%s|%s' %(data['container_id'],\n data['commit'],\n data['branch'],\n data['token'],\n data['tag']) \n\n signature = generate_header_signature(secret=secret,\n payload=body,\n request_type=\"push\")\n\n headers = {'Authorization': signature }\n\n if response_url is not None:\n finish = requests.post(response_url,data=data, headers=headers)\n bot.debug(\"RECEIVE POST TO SINGULARITY HUB ---------------------\")\n bot.debug(finish.status_code)\n bot.debug(finish.reason)\n else:\n bot.warning(\"response_url set to None, skipping sending of build.\")\n\n if clean_up == True:\n shutil.rmtree(build_dir)\n\n # Delay a bit, to give buffer between bringing instance down\n time.sleep(20)", "response": "finish build sends the build and data to a response url"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_build_close(params,response_url):\n '''send build close sends a final response (post) to the server to bring down\n the instance. The following must be included in params:\n\n repo_url, logfile, repo_id, secret, log_file, token\n '''\n # Finally, package everything to send back to shub\n response = {\"log\": json.dumps(params['log_file']),\n \"repo_url\": params['repo_url'],\n \"logfile\": params['logfile'],\n \"repo_id\": params['repo_id'],\n \"container_id\": params['container_id']}\n\n body = '%s|%s|%s|%s|%s' %(params['container_id'],\n params['commit'],\n params['branch'],\n params['token'],\n params['tag']) \n\n signature = generate_header_signature(secret=params['token'],\n payload=body,\n request_type=\"finish\")\n\n headers = {'Authorization': signature }\n\n finish = requests.post(response_url,data=response, headers=headers)\n bot.debug(\"FINISH POST TO SINGULARITY HUB ---------------------\")\n bot.debug(finish.status_code)\n bot.debug(finish.reason)\n return finish", "response": "send build close to the server"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_unicode_dict(input_dict):\n '''remove unicode keys and values from dict, encoding in utf8\n '''\n if isinstance(input_dict, collections.Mapping):\n return dict(map(remove_unicode_dict, input_dict.iteritems()))\n elif isinstance(input_dict, collections.Iterable):\n return type(input_dict)(map(remove_unicode_dict, input_dict))\n else:\n return input_dict", "response": "remove unicode keys and values from dict encoding in utf8\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_dict(input_dict,key,value):\n '''update_dict will update lists in a dictionary. If the key is not included,\n if will add as new list. If it is, it will append.\n :param input_dict: the dict to update\n :param value: the value to update with\n '''\n if key in input_dict:\n input_dict[key].append(value)\n else:\n input_dict[key] = [value]\n return input_dict", "response": "update_dict will update lists in a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_google_service(service_type=None,version=None):\n '''\n get_url will use the requests library to get a url\n :param service_type: the service to get (default is storage)\n :param version: version to use (default is v1)\n '''\n if service_type == None:\n service_type = \"storage\"\n if version == None:\n version = \"v1\"\n\n credentials = GoogleCredentials.get_application_default()\n return build(service_type, version, credentials=credentials)", "response": "get_url will use the requests library to get a url"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef upload_file(storage_service,bucket,bucket_path,file_name,verbose=True):\n '''get_folder will return the folder with folder_name, and if create=True,\n will create it if not found. If folder is found or created, the metadata is\n returned, otherwise None is returned\n :param storage_service: the drive_service created from get_storage_service\n :param bucket: the bucket object from get_bucket\n :param file_name: the name of the file to upload\n :param bucket_path: the path to upload to\n '''\n # Set up path on bucket\n upload_path = \"%s/%s\" %(bucket['id'],bucket_path)\n if upload_path[-1] != '/':\n upload_path = \"%s/\" %(upload_path)\n upload_path = \"%s%s\" %(upload_path,os.path.basename(file_name))\n body = {'name': upload_path }\n # Create media object with correct mimetype\n if os.path.exists(file_name):\n mimetype = sniff_extension(file_name,verbose=verbose)\n media = http.MediaFileUpload(file_name,\n mimetype=mimetype,\n resumable=True)\n request = storage_service.objects().insert(bucket=bucket['id'], \n body=body,\n predefinedAcl=\"publicRead\",\n media_body=media)\n result = request.execute()\n return result\n bot.warning('%s requested for upload does not exist, skipping' %file_name)", "response": "upload a file to the specified bucket"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_image_path(repo_url, trailing_path):\n '''get_image_path will determine an image path based on a repo url, removing\n any token, and taking into account urls that end with .git.\n :param repo_url: the repo url to parse:\n :param trailing_path: the trailing path (commit then hash is common)\n '''\n repo_url = repo_url.split('@')[-1].strip()\n if repo_url.endswith('.git'):\n repo_url = repo_url[:-4]\n return \"%s/%s\" %(re.sub('^http.+//www[.]','',repo_url), trailing_path)", "response": "This function will determine the path to the image file based on a repo url and the trailing path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run_build(logfile='/tmp/.shub-log'):\n\n '''run_build will generate the Singularity build from a spec_file from a repo_url.\n\n If no arguments are required, the metadata api is queried for the values.\n\n :param build_dir: directory to do the build in. If not specified, will use temporary. \n :param spec_file: the spec_file name to use, assumed to be in git repo\n :param repo_url: the url to download the repo from\n :param repo_id: the repo_id to uniquely identify the repo (in case name changes)\n :param commit: the commit to checkout. If none provided, will use most recent.\n :param bucket_name: the name of the bucket to send files to\n :param verbose: print out extra details as we go (default True) \n :param token: a token to send back to the server to authenticate the collection\n :param secret: a secret to match to the correct container\n :param response_url: the build url to send the response back to. Should also come\n from metadata. If not specified, no response is sent\n :param branch: the branch to checkout for the build.\n\n :: note: this function is currently configured to work with Google Compute\n Engine metadata api, and should (will) be customized if needed to work elsewhere \n\n '''\n\n # If we are building the image, this will not be set\n go = get_build_metadata(key='dobuild')\n if go == None:\n sys.exit(0)\n\n # If the user wants debug, this will be set\n debug = True\n enable_debug = get_build_metadata(key='debug')\n if enable_debug == None:\n debug = False\n bot.info('DEBUG %s' %debug)\n\n # Uaw /tmp for build directory\n build_dir = tempfile.mkdtemp()\n\n # Get variables from the instance metadata API\n metadata = [{'key': 'repo_url', 'value': None },\n {'key': 'repo_id', 'value': None },\n {'key': 'response_url', 'value': None },\n {'key': 'bucket_name', 'value': None },\n {'key': 'tag', 'value': None },\n {'key': 'container_id', 'value': None },\n {'key': 'commit', 'value': None },\n {'key': 'token', 'value': None},\n {'key': 'branch', 'value': None },\n {'key': 'spec_file', 'value': None},\n {'key': 'logging_url', 'value': None },\n {'key': 'logfile', 'value': logfile }]\n\n # Obtain values from build\n bot.log(\"BUILD PARAMETERS:\")\n params = get_build_params(metadata)\n params['debug'] = debug\n \n # Default spec file is Singularity\n if params['spec_file'] == None:\n params['spec_file'] = \"Singularity\"\n \n if params['bucket_name'] == None:\n params['bucket_name'] = \"singularityhub\"\n\n if params['tag'] == None:\n params['tag'] = \"latest\"\n\n output = run_build_main(build_dir=build_dir,\n params=params)\n\n # Output includes:\n finished_image = output['image']\n metadata = output['metadata']\n params = output['params'] \n\n # Upload image package files to Google Storage\n if os.path.exists(finished_image):\n bot.info(\"%s successfully built\" %finished_image)\n dest_dir = tempfile.mkdtemp(prefix='build')\n\n # The path to the images on google drive will be the github url/commit folder\n trailing_path = \"%s/%s\" %(params['commit'], params['version'])\n image_path = get_image_path(params['repo_url'], trailing_path) \n # commits are no longer unique\n # storage is by commit\n\n build_files = [finished_image]\n bot.info(\"Sending image to storage:\") \n bot.info('\\n'.join(build_files))\n\n # Start the storage service, retrieve the bucket\n storage_service = get_google_service() # default is \"storage\" \"v1\"\n bucket = get_bucket(storage_service,params[\"bucket_name\"])\n\n # For each file, upload to storage\n files = []\n for build_file in build_files:\n bot.info(\"Uploading %s to storage...\" %build_file)\n storage_file = upload_file(storage_service,\n bucket=bucket,\n bucket_path=image_path,\n file_name=build_file) \n files.append(storage_file)\n \n # Finally, package everything to send back to shub\n response = {\"files\": json.dumps(files),\n \"repo_url\": params['repo_url'],\n \"commit\": params['commit'],\n \"repo_id\": params['repo_id'],\n \"branch\": params['branch'],\n \"tag\": params['tag'],\n \"container_id\": params['container_id'],\n \"spec_file\":params['spec_file'],\n \"token\": params['token'],\n \"metadata\": json.dumps(metadata)}\n\n # Did the user specify a specific log file?\n custom_logfile = get_build_metadata('logfile')\n if custom_logfile is not None:\n logfile = custom_logfile \n response['logfile'] = logfile\n\n # Send final build data to instance\n send_build_data(build_dir=build_dir,\n response_url=params['response_url'],\n secret=params['token'],\n data=response)\n\n # Dump final params, for logger to retrieve\n passing_params = \"/tmp/params.pkl\"\n pickle.dump(params,open(passing_params,'wb'))", "response": "This function will generate the Singularity build from a spec_file from a repo_url and then run it."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_build_metadata(key):\n '''get_build_metadata will return metadata about an instance from within it.\n :param key: the key to look up\n '''\n headers = {\"Metadata-Flavor\":\"Google\"}\n url = \"http://metadata.google.internal/computeMetadata/v1/instance/attributes/%s\" %(key) \n response = requests.get(url=url,headers=headers)\n if response.status_code == 200:\n return response.text\n return None", "response": "get_build_metadata will return metadata about an instance from within it"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_build_params(metadata):\n '''get_build_params uses get_build_metadata to retrieve corresponding meta data values for a build\n :param metadata: a list, each item a dictionary of metadata, in format:\n metadata = [{'key': 'repo_url', 'value': repo_url },\n {'key': 'repo_id', 'value': repo_id },\n {'key': 'credential', 'value': credential },\n {'key': 'response_url', 'value': response_url },\n {'key': 'token', 'value': token},\n {'key': 'commit', 'value': commit }]\n\n '''\n params = dict()\n for item in metadata:\n if item['value'] == None:\n response = get_build_metadata(key=item['key'])\n item['value'] = response\n params[item['key']] = item['value']\n if item['key'] not in ['token', 'secret', 'credential']:\n bot.info('%s is set to %s' %(item['key'],item['value'])) \n return params", "response": "uses get_build_metadata to retrieve corresponding meta data values for a build\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rsync(*args, **kwargs):\n kwargs.setdefault('capture', False)\n replacements = dict(\n host_string=\"{user}@{host}\".format(\n user=env.instance.config.get('user', 'root'),\n host=env.instance.config.get(\n 'host', env.instance.config.get(\n 'ip', env.instance.uid))))\n args = [x.format(**replacements) for x in args]\n ssh_info = env.instance.init_ssh_key()\n ssh_info.pop('host')\n ssh_info.pop('user')\n ssh_args = env.instance.ssh_args_from_info(ssh_info)\n cmd_parts = ['rsync']\n cmd_parts.extend(['-e', \"ssh %s\" % shjoin(ssh_args)])\n cmd_parts.extend(args)\n cmd = shjoin(cmd_parts)\n return local(cmd, **kwargs)", "response": "wrapper around the rsync command."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbootstrapping an EC2 instance that has been booted into an AMI from http://www. daemonology. net", "response": "def bootstrap(**kwargs):\n \"\"\" Bootstrap an EC2 instance that has been booted into an AMI from http://www.daemonology.net/freebsd-on-ec2/\n Note: deprecated, current AMI images are basically pre-bootstrapped, they just need to be configured.\n \"\"\"\n # the user for the image is `ec2-user`, there is no sudo, but we can su to root w/o password\n original_host = env.host_string\n env.host_string = 'ec2-user@%s' % env.instance.uid\n bootstrap_files = env.instance.config.get('bootstrap-files', 'bootstrap-files')\n put('%s/authorized_keys' % bootstrap_files, '/tmp/authorized_keys')\n put(join(bsdploy_path, 'enable_root_login_on_daemonology.sh'), '/tmp/', mode='0775')\n run(\"\"\"su root -c '/tmp/enable_root_login_on_daemonology.sh'\"\"\")\n # revert back to root\n env.host_string = original_host\n # give sshd a chance to restart\n sleep(2)\n run('rm /tmp/enable_root_login_on_daemonology.sh')\n\n # allow overwrites from the commandline\n env.instance.config.update(kwargs)\n\n bu = BootstrapUtils()\n bu.ssh_keys = None\n bu.upload_authorized_keys = False\n bu.bootstrap_files_yaml = 'daemonology-files.yml'\n bu.print_bootstrap_files()\n\n bu.create_bootstrap_directories()\n bu.upload_bootstrap_files({})\n # we need to install python here, because there is no way to install it via\n # ansible playbooks\n bu.install_pkg('/', chroot=False, packages=['python27'])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbootstrapping the FreeBSD droplet into a new root.", "response": "def bootstrap(**kwargs):\n \"\"\"Digital Oceans FreeBSD droplets are pretty much already pre-bootstrapped,\n including having python2.7 and sudo etc. pre-installed.\n the only thing we need to change is to allow root to login (without a password)\n enable pf and ensure it is running\n \"\"\"\n\n bu = BootstrapUtils()\n # (temporarily) set the user to `freebsd`\n original_host = env.host_string\n env.host_string = 'freebsd@%s' % env.instance.uid\n # copy DO bsdclout-init results:\n if bu.os_release.startswith('10'):\n sudo(\"\"\"cat /etc/rc.digitalocean.d/droplet.conf > /etc/rc.conf\"\"\")\n sudo(\"\"\"sysrc zfs_enable=YES\"\"\")\n sudo(\"\"\"sysrc sshd_enable=YES\"\"\")\n # enable and start pf\n sudo(\"\"\"sysrc pf_enable=YES\"\"\")\n sudo(\"\"\"sysrc -f /boot/loader.conf pfload=YES\"\"\")\n sudo('kldload pf', warn_only=True)\n sudo('''echo 'pass in all' > /etc/pf.conf''')\n sudo('''echo 'pass out all' >> /etc/pf.conf''')\n sudo('''chmod 644 /etc/pf.conf''')\n sudo('service pf start')\n # overwrite sshd_config, because the DO version only contains defaults\n # and a line explicitly forbidding root to log in\n sudo(\"\"\"echo 'PermitRootLogin without-password' > /etc/ssh/sshd_config\"\"\")\n # additionally, make sure the root user is unlocked!\n sudo('pw unlock root')\n # overwrite the authorized keys for root, because DO creates its entries to explicitly\n # disallow root login\n bootstrap_files = env.instance.config.get('bootstrap-files', 'bootstrap-files')\n put(path.abspath(path.join(env['config_base'], bootstrap_files, 'authorized_keys')), '/tmp/authorized_keys', use_sudo=True)\n sudo('''mv /tmp/authorized_keys /root/.ssh/''')\n sudo('''chown root:wheel /root/.ssh/authorized_keys''')\n\n sudo(\"\"\"service sshd fastreload\"\"\")\n # revert back to root\n env.host_string = original_host\n # give sshd a chance to restart\n sleep(2)\n # clean up DO cloudinit leftovers\n run(\"rm -f /etc/rc.d/digitalocean\")\n run(\"rm -rf /etc/rc.digitalocean.d\")\n run(\"rm -rf /usr/local/bsd-cloudinit/\")\n run(\"pkg remove -y avahi-autoipd || true\")\n\n # allow overwrites from the commandline\n env.instance.config.update(kwargs)\n\n bu.ssh_keys = None\n bu.upload_authorized_keys = False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef bootstrap_files(self):\n bootstrap_file_yamls = [\n abspath(join(self.default_template_path, self.bootstrap_files_yaml)),\n abspath(join(self.custom_template_path, self.bootstrap_files_yaml))]\n bootstrap_files = dict()\n if self.upload_authorized_keys:\n bootstrap_files['authorized_keys'] = BootstrapFile(self, 'authorized_keys', **{\n 'directory': '/mnt/root/.ssh',\n 'directory_mode': '0600',\n 'remote': '/mnt/root/.ssh/authorized_keys',\n 'fallback': [\n '~/.ssh/identity.pub',\n '~/.ssh/id_rsa.pub',\n '~/.ssh/id_dsa.pub',\n '~/.ssh/id_ecdsa.pub']})\n for bootstrap_file_yaml in bootstrap_file_yamls:\n if not exists(bootstrap_file_yaml):\n continue\n with open(bootstrap_file_yaml) as f:\n info = yaml.load(f, Loader=SafeLoader)\n if info is None:\n continue\n for k, v in info.items():\n bootstrap_files[k] = BootstrapFile(self, k, **v)\n\n for bf in bootstrap_files.values():\n if not exists(bf.local) and bf.raw_fallback:\n if not bf.existing_fallback:\n print(\"Found no public key in %s, you have to create '%s' manually\" % (expanduser('~/.ssh'), bf.local))\n sys.exit(1)\n print(\"The '%s' file is missing.\" % bf.local)\n for path in bf.existing_fallback:\n yes = env.instance.config.get('bootstrap-yes', False)\n if yes or yesno(\"Should we generate it using the key in '%s'?\" % path):\n if not exists(bf.expected_path):\n os.mkdir(bf.expected_path)\n with open(bf.local, 'wb') as out:\n with open(path, 'rb') as f:\n out.write(f.read())\n break\n else:\n # answered no to all options\n sys.exit(1)\n\n if not bf.check():\n print('Cannot find %s' % bf.local)\n sys.exit(1)\n\n packages_path = join(self.download_path, 'packages')\n if exists(packages_path):\n for dirpath, dirnames, filenames in os.walk(packages_path):\n path = dirpath.split(packages_path)[1][1:]\n for filename in filenames:\n if not filename.endswith('.txz'):\n continue\n bootstrap_files[join(path, filename)] = BootstrapFile(\n self, join(path, filename), **dict(\n local=join(packages_path, join(path, filename)),\n remote=join('/mnt/var/cache/pkg/All', filename),\n encrypted=False))\n\n if self.ssh_keys is not None:\n for ssh_key_name, ssh_key_options in list(self.ssh_keys):\n ssh_key = join(self.custom_template_path, ssh_key_name)\n if exists(ssh_key):\n pub_key_name = '%s.pub' % ssh_key_name\n pub_key = '%s.pub' % ssh_key\n if not exists(pub_key):\n print(\"Public key '%s' for '%s' missing.\" % (pub_key, ssh_key))\n sys.exit(1)\n bootstrap_files[ssh_key_name] = BootstrapFile(\n self, ssh_key_name, **dict(\n local=ssh_key,\n remote='/mnt/etc/ssh/%s' % ssh_key_name,\n mode=0600))\n bootstrap_files[pub_key_name] = BootstrapFile(\n self, pub_key_name, **dict(\n local=pub_key,\n remote='/mnt/etc/ssh/%s' % pub_key_name,\n mode=0644))\n if hasattr(env.instance, 'get_vault_lib'):\n vaultlib = env.instance.get_vault_lib()\n for bf in bootstrap_files.values():\n if bf.encrypted is None and exists(bf.local):\n with open(bf.local) as f:\n data = f.read()\n bf.info['encrypted'] = vaultlib.is_encrypted(data)\n return bootstrap_files", "response": "This method will create a bootstrap file for the FreeBSD installation."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the name of the disk devices that are suitable installation targets by subtracting CDROM - and USB devices from the list of total mounts.", "response": "def devices(self):\n \"\"\" computes the name of the disk devices that are suitable\n installation targets by subtracting CDROM- and USB devices\n from the list of total mounts.\n \"\"\"\n install_devices = self.install_devices\n if 'bootstrap-system-devices' in env.instance.config:\n devices = set(env.instance.config['bootstrap-system-devices'].split())\n else:\n devices = set(self.sysctl_devices)\n for sysctl_device in self.sysctl_devices:\n for install_device in install_devices:\n if install_device.startswith(sysctl_device):\n devices.remove(sysctl_device)\n return devices"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndownload bootstrap assets to control host.", "response": "def fetch_assets(self):\n \"\"\" download bootstrap assets to control host.\n If present on the control host they will be uploaded to the target host during bootstrapping.\n \"\"\"\n # allow overwrites from the commandline\n packages = set(\n env.instance.config.get('bootstrap-packages', '').split())\n packages.update(['python27'])\n cmd = env.instance.config.get('bootstrap-local-download-cmd', 'wget -c -O \"{0.local}\" \"{0.url}\"')\n items = sorted(self.bootstrap_files.items())\n for filename, asset in items:\n if asset.url:\n if not exists(dirname(asset.local)):\n os.makedirs(dirname(asset.local))\n local(cmd.format(asset))\n if filename == 'packagesite.txz':\n # add packages to download\n items.extend(self._fetch_packages(asset.local, packages))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a string representation of the given response object.", "response": "def res_to_str(res):\n \"\"\"\n :param res: :class:`requests.Response` object\n\n Parse the given request and generate an informative string from it\n \"\"\"\n if 'Authorization' in res.request.headers:\n res.request.headers['Authorization'] = \"*****\"\n return \"\"\"\n####################################\nurl = %s\nheaders = %s\n-------- data sent -----------------\n%s\n------------------------------------\n@@@@@ response @@@@@@@@@@@@@@@@\nheaders = %s\ncode = %d\nreason = %s\n--------- data received ------------\n%s\n------------------------------------\n####################################\n\"\"\" % (res.url,\n str(res.request.headers),\n OLD_REQ and res.request.data or res.request.body,\n res.headers,\n res.status_code,\n res.reason,\n res.text)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse a resource definition and return all the info extracted from the resource section of the apipie", "response": "def parse_resource_definition(resource_name, resource_dct):\n \"\"\"\n Returns all the info extracted from a resource section of the apipie json\n\n :param resource_name: Name of the resource that is defined by the section\n :param resrouce_dict: Dictionary as generated by apipie of the resource\n definition\n \"\"\"\n new_dict = {\n '__module__': resource_dct.get('__module__', __name__),\n '__doc__': resource_dct['full_description'],\n '_resource_name': resource_name,\n '_own_methods': set(),\n '_conflicting_methods': [],\n }\n\n # methods in foreign_methods are meant for other resources,\n # that is, the url and the resource field do not match /api/{resource}\n foreign_methods = {}\n\n # as defined per apipie gem, each method can have more than one api,\n # for example, /api/hosts can have the GET /api/hosts api and the GET\n # /api/hosts/:id api or DELETE /api/hosts\n for method in resource_dct['methods']:\n # set the docstring if it only has one api\n if not new_dict['__doc__'] and len(method['apis']) == 1:\n new_dict['__doc__'] = \\\n method['apis'][0]['short_description']\n for api in method['apis']:\n api = MethodAPIDescription(resource_name, method, api)\n\n if api.resource != resource_name:\n # this means that the json apipie passed says that an\n # endpoint in the form: /api/{resource}/* belongs to\n # {different_resource}, we just put it under {resource}\n # later, storing it under _foreign_methods for now as we\n # might not have parsed {resource} yet\n functions = foreign_methods.setdefault(api.resource, {})\n if api.name in functions:\n old_api = functions.get(api.name).defs\n # show only in debug the repeated but identical definitions\n log_method = logger.warning\n if api.url == old_api.url:\n log_method = logger.debug\n\n log_method(\n \"There is a conflict trying to redefine a method \"\n \"for a foreign resource (%s): \\n\"\n \"\\tresource:\\n\"\n \"\\tapipie_resource: %s\\n\"\n \"\\tnew_api: %s\\n\"\n \"\\tnew_url: %s\\n\"\n \"\\told_api: %s\\n\"\n \"\\told_url: %s\",\n api.name,\n resource_name,\n pprint.pformat(api),\n api.url,\n pprint.pformat(old_api),\n old_api.url,\n )\n new_dict['_conflicting_methods'].append(api)\n continue\n functions[api.name] = api.generate_func()\n\n else:\n # it's an own method, resource and url match\n if api.name in new_dict['_own_methods']:\n old_api = new_dict.get(api.name).defs\n log_method = logger.warning\n # show only in debug the repeated but identical definitions\n if api.url == old_api.url:\n log_method = logger.debug\n\n log_method(\n \"There is a conflict trying to redefine method \"\n \"(%s): \\n\"\n \"\\tapipie_resource: %s\\n\"\n \"\\tnew_api: %s\\n\"\n \"\\tnew_url: %s\\n\"\n \"\\told_api: %s\\n\"\n \"\\told_url: %s\",\n api.name,\n resource_name,\n pprint.pformat(api),\n api.url,\n pprint.pformat(old_api),\n old_api.url,\n )\n new_dict['_conflicting_methods'].append(api)\n continue\n new_dict['_own_methods'].add(api.name)\n new_dict[api.name] = api.generate_func()\n\n return new_dict, foreign_methods"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the resource name from the given URL.", "response": "def parse_resource_from_url(self, url):\n \"\"\"\n Returns the appropriate resource name for the given URL.\n\n :param url: API URL stub, like: '/api/hosts'\n :return: Resource name, like 'hosts', or None if not found\n \"\"\"\n # special case for the api root\n if url == '/api':\n return 'api'\n elif url == '/katello':\n return 'katello'\n\n match = self.resource_pattern.match(url)\n if match:\n return match.groupdict().get('resource', None)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_name(self):\n if self.url.count(':') > 1:\n # /api/one/two/:three/four -> two_:three_four\n base_name = self.url.split('/', 3)[-1].replace('/', '_')[1:]\n # :one_two_three -> two_three\n if base_name.startswith(':'):\n base_name = base_name.split('_')[-1]\n # one_:two_three_:four_five -> one_three_five\n base_name = re.sub('_:[^/]+', '', base_name)\n # in case that the last term was a parameter\n if base_name.endswith('_'):\n base_name = base_name[:-1]\n # one_two_three -> one_two_method_three\n base_name = (\n '_' + self._method['name']\n ).join(base_name.rsplit('_', 1))\n else:\n base_name = self._method['name']\n if base_name == 'import':\n base_name = 'import_'\n if self._apipie_resource != self.resource:\n return '%s_%s' % (self._apipie_resource, base_name)\n else:\n return base_name", "response": "Get the name of the current object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_func(self, as_global=False):\n keywords = []\n params_def = []\n params_doc = \"\"\n original_names = {}\n\n params = dict(\n (param['name'], param)\n for param in self.params\n )\n\n # parse the url required params, as sometimes they are skipped in the\n # parameters list of the definition\n for param in self.url_params:\n if param not in params:\n param = {\n 'name': param,\n 'required': True,\n 'description': '',\n 'validator': '',\n }\n params[param['name']] = param\n else:\n params[param]['required'] = True\n\n # split required and non-required params for the definition\n req_params = []\n nonreq_params = []\n for param in six.itervalues(params):\n if param['required']:\n req_params.append(param)\n else:\n nonreq_params.append(param)\n\n for param in req_params + nonreq_params:\n params_doc += self.create_param_doc(param) + \"\\n\"\n local_name = param['name']\n # some params collide with python keywords, that's why we do\n # this switch (and undo it inside the function we generate)\n if param['name'] == 'except':\n local_name = 'except_'\n original_names[local_name] = param['name']\n keywords.append(local_name)\n if param['required']:\n params_def.append(\"%s\" % local_name)\n else:\n params_def.append(\"%s=None\" % local_name)\n\n func_head = 'def {0}(self, {1}):'.format(\n as_global and self.get_global_method_name() or self.name,\n ', '.join(params_def)\n )\n code_body = (\n ' _vars_ = locals()\\n'\n ' _url = self._fill_url(\"{url}\", _vars_, {url_params})\\n'\n ' _original_names = {original_names}\\n'\n ' _kwargs = dict((_original_names[k], _vars_[k])\\n'\n ' for k in {keywords} if _vars_[k])\\n'\n ' return self._foreman.do_{http_method}(_url, _kwargs)')\n code_body = code_body.format(\n http_method=self.http_method.lower(),\n url=self.url,\n url_params=self.url_params,\n keywords=keywords,\n original_names=original_names,\n )\n\n code = [\n func_head,\n ' \"\"\"',\n self.short_desc,\n '',\n params_doc,\n ' \"\"\"',\n code_body,\n ]\n\n code = '\\n'.join(code)\n\n six.exec_(code)\n\n function = locals()[self.name]\n # to ease debugging, all the funcs have the definitions attached\n setattr(function, 'defs', self)\n return function", "response": "Generates function for specific method and using specific api"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_param_doc(cls, param, prefix=None):\n desc = cls.exclude_html_reg.sub('', param['description']).strip()\n if not desc:\n desc = \"\"\n name = param['name']\n if prefix:\n name = \"%s[%s]\" % (prefix, name)\n doc_ = \":param %s: %s; %s\" % (name, desc, param['validator'])\n if param['required']:\n doc_ += \" (REQUIRED)\"\n else:\n doc_ += \" (OPTIONAL)\"\n for param in param.get('params', []):\n doc_ += \"\\n\" + cls.create_param_doc(param, name)\n return doc_", "response": "Generate documentation for single parameter of function\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convert_plugin_def(http_method, funcs):\n methods = []\n if http_method not in ('GET', 'PUT', 'POST', 'DELETE'):\n logger.error(\n 'Plugin load failure, HTTP method %s unsupported.',\n http_method,\n )\n return methods\n for fname, params in six.iteritems(funcs):\n method = {\n 'apis': [{'short_description': 'no-doc'}],\n 'params': [],\n }\n method['apis'][0]['http_method'] = http_method\n method['apis'][0]['api_url'] = '/api/' + fname\n method['name'] = fname\n for pname, pdef in six.iteritems(params):\n param = {\n 'name': pname,\n 'validator': \"Must be %s\" % pdef['ptype'],\n 'description': '',\n 'required': pdef['required'],\n }\n method['params'].append(param)\n methods.append(method)\n return methods", "response": "This function parses the plugin definition dict for a\n plugin and extracts the relevant information."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_current_version(repo_path):\n repo = dulwich.repo.Repo(repo_path)\n tags = get_tags(repo)\n maj_version = 0\n feat_version = 0\n fix_version = 0\n\n for commit_sha, children in reversed(\n get_children_per_first_parent(repo_path).items()\n ):\n commit = get_repo_object(repo, commit_sha)\n maj_version, feat_version, fix_version = get_version(\n commit=commit,\n tags=tags,\n maj_version=maj_version,\n feat_version=feat_version,\n fix_version=fix_version,\n children=children,\n )\n\n return '%s.%s.%s' % (maj_version, feat_version, fix_version)", "response": "Given a repo will return the current version of the current version"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_authors(repo_path, from_commit):\n repo = dulwich.repo.Repo(repo_path)\n refs = get_refs(repo)\n start_including = False\n authors = set()\n\n if from_commit is None:\n start_including = True\n\n for commit_sha, children in reversed(\n get_children_per_first_parent(repo_path).items()\n ):\n commit = get_repo_object(repo, commit_sha)\n if (\n start_including or commit_sha.startswith(from_commit) or\n fuzzy_matches_refs(from_commit, refs.get(commit_sha, []))\n ):\n authors.add(commit.author.decode())\n for child in children:\n authors.add(child.author.decode())\n\n start_including = True\n\n return '\\n'.join(sorted(authors))", "response": "Given a repo and optionally a base revision to start from will return a list of authors."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nemit a spout Tuple message.", "response": "def emit(\n self, tup, tup_id=None, stream=None, direct_task=None, need_task_ids=False\n ):\n \"\"\"Emit a spout Tuple message.\n\n :param tup: the Tuple to send to Storm, should contain only\n JSON-serializable data.\n :type tup: list or tuple\n :param tup_id: the ID for the Tuple. Leave this blank for an\n unreliable emit.\n :type tup_id: str\n :param stream: ID of the stream this Tuple should be emitted to.\n Leave empty to emit to the default stream.\n :type stream: str\n :param direct_task: the task to send the Tuple to if performing a\n direct emit.\n :type direct_task: int\n :param need_task_ids: indicate whether or not you'd like the task IDs\n the Tuple was emitted (default: ``False``).\n :type need_task_ids: bool\n\n :returns: ``None``, unless ``need_task_ids=True``, in which case it will\n be a ``list`` of task IDs that the Tuple was sent to if. Note\n that when specifying direct_task, this will be equal to\n ``[direct_task]``.\n \"\"\"\n return super(Spout, self).emit(\n tup,\n tup_id=tup_id,\n stream=stream,\n direct_task=direct_task,\n need_task_ids=need_task_ids,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalls when a bolt acknowledges a tuple in the topology.", "response": "def ack(self, tup_id):\n \"\"\"Called when a bolt acknowledges a Tuple in the topology.\n\n :param tup_id: the ID of the Tuple that has been fully acknowledged in\n the topology.\n :type tup_id: str\n \"\"\"\n self.failed_tuples.pop(tup_id, None)\n try:\n del self.unacked_tuples[tup_id]\n except KeyError:\n self.logger.error(\"Received ack for unknown tuple ID: %r\", tup_id)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fail(self, tup_id):\n saved_args = self.unacked_tuples.get(tup_id)\n if saved_args is None:\n self.logger.error(\"Received fail for unknown tuple ID: %r\", tup_id)\n return\n tup, stream, direct_task, need_task_ids = saved_args\n if self.failed_tuples[tup_id] < self.max_fails:\n self.emit(\n tup,\n tup_id=tup_id,\n stream=stream,\n direct_task=direct_task,\n need_task_ids=need_task_ids,\n )\n self.failed_tuples[tup_id] += 1\n else:\n # Just pretend we got an ack when we exceed retry limit\n self.logger.info(\n \"Acking tuple ID %r after it exceeded retry limit \" \"(%r)\",\n tup_id,\n self.max_fails,\n )\n self.ack(tup_id)", "response": "Called when a Tuple fails in the topology."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef emit(\n self, tup, tup_id=None, stream=None, direct_task=None, need_task_ids=False\n ):\n \"\"\"Emit a spout Tuple & add metadata about it to `unacked_tuples`.\n\n In order for this to work, `tup_id` is a required parameter.\n\n See :meth:`Bolt.emit`.\n \"\"\"\n if tup_id is None:\n raise ValueError(\n \"You must provide a tuple ID when emitting with a \"\n \"ReliableSpout in order for the tuple to be \"\n \"tracked.\"\n )\n args = (tup, stream, direct_task, need_task_ids)\n self.unacked_tuples[tup_id] = args\n return super(ReliableSpout, self).emit(\n tup,\n tup_id=tup_id,\n stream=stream,\n direct_task=direct_task,\n need_task_ids=need_task_ids,\n )", "response": "Emit a single entry in the unacked_tuples dictionary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remote_pdb_handler(signum, frame):\n try:\n from remote_pdb import RemotePdb\n\n rdb = RemotePdb(host=\"127.0.0.1\", port=0)\n rdb.set_trace(frame=frame)\n except ImportError:\n log.warning(\n \"remote_pdb unavailable. Please install remote_pdb to \"\n \"allow remote debugging.\"\n )\n # Restore signal handler for later\n signal.signal(signum, remote_pdb_handler)", "response": "Handler to drop us into a remote debugger upon receiving SIGUSR1"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef emit(self, record):\n try:\n msg = self.format(record)\n level = _STORM_LOG_LEVELS.get(record.levelname.lower(), _STORM_LOG_INFO)\n self.serializer.send_message(\n {\"command\": \"log\", \"msg\": str(msg), \"level\": level}\n )\n except Exception:\n self.handleError(record)", "response": "Emit a record to Storm."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _setup_component(self, storm_conf, context):\n self.topology_name = storm_conf.get(\"topology.name\", \"\")\n self.task_id = context.get(\"taskid\", \"\")\n self.component_name = context.get(\"componentid\")\n # If using Storm before 0.10.0 componentid is not available\n if self.component_name is None:\n self.component_name = context.get(\"task->component\", {}).get(\n str(self.task_id), \"\"\n )\n self.debug = storm_conf.get(\"topology.debug\", False)\n self.storm_conf = storm_conf\n self.context = context\n\n # Set up logging\n self.logger = logging.getLogger(\".\".join((__name__, self.component_name)))\n log_path = self.storm_conf.get(\"pystorm.log.path\")\n log_file_name = self.storm_conf.get(\n \"pystorm.log.file\",\n \"pystorm_{topology_name}\" \"_{component_name}\" \"_{task_id}\" \"_{pid}.log\",\n )\n root_log = logging.getLogger()\n log_level = self.storm_conf.get(\"pystorm.log.level\", \"info\")\n if log_path:\n max_bytes = self.storm_conf.get(\"pystorm.log.max_bytes\", 1000000) # 1 MB\n backup_count = self.storm_conf.get(\"pystorm.log.backup_count\", 10)\n log_file = join(\n log_path,\n (\n log_file_name.format(\n topology_name=self.topology_name,\n component_name=self.component_name,\n task_id=self.task_id,\n pid=self.pid,\n )\n ),\n )\n handler = RotatingFileHandler(\n log_file, maxBytes=max_bytes, backupCount=backup_count\n )\n log_format = self.storm_conf.get(\n \"pystorm.log.format\",\n \"%(asctime)s - %(name)s - \" \"%(levelname)s - %(message)s\",\n )\n else:\n self.log(\n \"pystorm StormHandler logging enabled, so all messages at \"\n 'levels greater than \"pystorm.log.level\" ({}) will be sent'\n \" to Storm.\".format(log_level)\n )\n handler = StormHandler(self.serializer)\n log_format = self.storm_conf.get(\n \"pystorm.log.format\", \"%(asctime)s - %(name)s - \" \"%(message)s\"\n )\n formatter = logging.Formatter(log_format)\n log_level = _PYTHON_LOG_LEVELS.get(log_level, logging.INFO)\n if self.debug:\n # potentially override logging that was provided if\n # topology.debug was set to true\n log_level = logging.DEBUG\n handler.setLevel(log_level)\n handler.setFormatter(formatter)\n root_log.addHandler(handler)\n self.logger.setLevel(log_level)\n logging.getLogger(\"pystorm\").setLevel(log_level)\n # Redirect stdout to ensure that print statements/functions\n # won't disrupt the multilang protocol\n if self.serializer.output_stream == sys.stdout:\n sys.stdout = LogStream(logging.getLogger(\"pystorm.stdout\"))", "response": "Setup the component with the given storm_conf and context."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread and process an initial handshake message from Storm.", "response": "def read_handshake(self):\n \"\"\"Read and process an initial handshake message from Storm.\"\"\"\n msg = self.read_message()\n pid_dir, _conf, _context = msg[\"pidDir\"], msg[\"conf\"], msg[\"context\"]\n\n # Write a blank PID file out to the pidDir\n open(join(pid_dir, str(self.pid)), \"w\").close()\n self.send_message({\"pid\": self.pid})\n\n return _conf, _context"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a message to Storm via stdout.", "response": "def send_message(self, message):\n \"\"\"Send a message to Storm via stdout.\"\"\"\n if not isinstance(message, dict):\n logger = self.logger if self.logger else log\n logger.error(\n \"%s.%d attempted to send a non dict message to Storm: \" \"%r\",\n self.component_name,\n self.pid,\n message,\n )\n return\n self.serializer.send_message(message)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreport an exception back to Storm via logging.", "response": "def raise_exception(self, exception, tup=None):\n \"\"\"Report an exception back to Storm via logging.\n\n :param exception: a Python exception.\n :param tup: a :class:`Tuple` object.\n \"\"\"\n if tup:\n message = (\n \"Python {exception_name} raised while processing Tuple \"\n \"{tup!r}\\n{traceback}\"\n )\n else:\n message = \"Python {exception_name} raised\\n{traceback}\"\n message = message.format(\n exception_name=exception.__class__.__name__, tup=tup, traceback=format_exc()\n )\n self.send_message({\"command\": \"error\", \"msg\": str(message)})\n self.send_message({\"command\": \"sync\"})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlogging a message to Storm optionally providing a logging level.", "response": "def log(self, message, level=None):\n \"\"\"Log a message to Storm optionally providing a logging level.\n\n :param message: the log message to send to Storm.\n :type message: str\n :param level: the logging level that Storm should use when writing the\n ``message``. Can be one of: trace, debug, info, warn, or\n error (default: ``info``).\n :type level: str\n\n .. warning::\n\n This will send your message to Storm regardless of what level you\n specify. In almost all cases, you are better of using\n ``Component.logger`` and not setting ``pystorm.log.path``, because\n that will use a :class:`pystorm.component.StormHandler` to do the\n filtering on the Python side (instead of on the Java side after taking\n the time to serialize your message and send it to Storm).\n \"\"\"\n level = _STORM_LOG_LEVELS.get(level, _STORM_LOG_INFO)\n self.send_message({\"command\": \"log\", \"msg\": str(message), \"level\": level})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nemitting a new tuple to a stream.", "response": "def emit(\n self,\n tup,\n tup_id=None,\n stream=None,\n anchors=None,\n direct_task=None,\n need_task_ids=False,\n ):\n \"\"\"Emit a new Tuple to a stream.\n\n :param tup: the Tuple payload to send to Storm, should contain only\n JSON-serializable data.\n :type tup: :class:`list` or :class:`pystorm.component.Tuple`\n :param tup_id: the ID for the Tuple. If omitted by a\n :class:`pystorm.spout.Spout`, this emit will be\n unreliable.\n :type tup_id: str\n :param stream: the ID of the stream to emit this Tuple to. Specify\n ``None`` to emit to default stream.\n :type stream: str\n :param anchors: IDs the Tuples (or\n :class:`pystorm.component.Tuple` instances)\n which the emitted Tuples should be anchored to. This is\n only passed by :class:`pystorm.bolt.Bolt`.\n :type anchors: list\n :param direct_task: the task to send the Tuple to.\n :type direct_task: int\n :param need_task_ids: indicate whether or not you'd like the task IDs\n the Tuple was emitted (default: ``False``).\n :type need_task_ids: bool\n\n :returns: ``None``, unless ``need_task_ids=True``, in which case it will\n be a ``list`` of task IDs that the Tuple was sent to if. Note\n that when specifying direct_task, this will be equal to\n ``[direct_task]``.\n \"\"\"\n if not isinstance(tup, (list, tuple)):\n raise TypeError(\n \"All Tuples must be either lists or tuples, \"\n \"received {!r} instead.\".format(type(tup))\n )\n\n msg = {\"command\": \"emit\", \"tuple\": tup}\n downstream_task_ids = None\n\n if anchors is not None:\n msg[\"anchors\"] = anchors\n if tup_id is not None:\n msg[\"id\"] = tup_id\n if stream is not None:\n msg[\"stream\"] = stream\n if direct_task is not None:\n msg[\"task\"] = direct_task\n if need_task_ids:\n downstream_task_ids = [direct_task]\n\n if not need_task_ids:\n # only need to send on False, Storm's default is True\n msg[\"need_task_ids\"] = need_task_ids\n\n if need_task_ids and direct_task is None:\n # Use both locks so we ensure send_message and read_task_ids are for\n # same emit\n with self._reader_lock, self._writer_lock:\n self.send_message(msg)\n downstream_task_ids = self.read_task_ids()\n # No locks necessary in simple case because serializer will acquire\n # write lock itself\n else:\n self.send_message(msg)\n\n return downstream_task_ids"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _wrap_stream(stream):\n if hasattr(stream, \"buffer\"):\n return io.TextIOWrapper(stream.buffer, encoding=\"utf-8\")\n elif hasattr(stream, \"readable\"):\n return io.TextIOWrapper(stream, encoding=\"utf-8\")\n # Python 2.x stdin and stdout are just files\n else:\n return io.open(stream.fileno(), mode=stream.mode, encoding=\"utf-8\")", "response": "Returns a TextIOWrapper around the given stream that handles UTF - 8\n encoding and decoding."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads a message from the input stream and returns it as a dict.", "response": "def read_message(self):\n \"\"\"The Storm multilang protocol consists of JSON messages followed by\n a newline and \"end\\n\".\n\n All of Storm's messages (for either bolts or spouts) should be of the\n form::\n\n '\\\\nend\\\\n'\n\n Command example, an incoming Tuple to a bolt::\n\n '{ \"id\": \"-6955786537413359385\", \"comp\": \"1\", \"stream\": \"1\", \"task\": 9, \"tuple\": [\"snow white and the seven dwarfs\", \"field2\", 3]}\\\\nend\\\\n'\n\n Command example for a spout to emit its next Tuple::\n\n '{\"command\": \"next\"}\\\\nend\\\\n'\n\n Example, the task IDs a prior emit was sent to::\n\n '[12, 22, 24]\\\\nend\\\\n'\n\n The edge case of where we read ``''`` from ``input_stream`` indicating\n EOF, usually means that communication with the supervisor has been\n severed.\n \"\"\"\n msg = \"\"\n num_blank_lines = 0\n while True:\n # readline will return trailing \\n so that output is unambigious, we\n # should only have line == '' if we're at EOF\n with self._reader_lock:\n line = self.input_stream.readline()\n if line == \"end\\n\":\n break\n elif line == \"\":\n raise StormWentAwayError()\n elif line == \"\\n\":\n num_blank_lines += 1\n if num_blank_lines % 1000 == 0:\n log.warn(\n \"While trying to read a command or pending task \"\n \"ID, Storm has instead sent %s '\\\\n' messages.\",\n num_blank_lines,\n )\n continue\n\n msg = \"{}{}\\n\".format(msg, line[0:-1])\n\n try:\n return json.loads(msg)\n except Exception:\n log.error(\"JSON decode error for message: %r\", msg, exc_info=True)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nserializing to JSON a message dictionary.", "response": "def serialize_dict(self, msg_dict):\n \"\"\"Serialize to JSON a message dictionary.\"\"\"\n serialized = json.dumps(msg_dict, namedtuple_as_object=False)\n if PY2:\n serialized = serialized.decode(\"utf-8\")\n serialized = \"{}\\nend\\n\".format(serialized)\n return serialized"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading a tuple from the pipe to Storm.", "response": "def read_tuple(self):\n \"\"\"Read a tuple from the pipe to Storm.\"\"\"\n cmd = self.read_command()\n source = cmd[\"comp\"]\n stream = cmd[\"stream\"]\n values = cmd[\"tuple\"]\n val_type = self._source_tuple_types[source].get(stream)\n return Tuple(\n cmd[\"id\"],\n source,\n stream,\n cmd[\"task\"],\n tuple(values) if val_type is None else val_type(*values),\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nemit a new tuple to Storm.", "response": "def emit(\n self, tup, stream=None, anchors=None, direct_task=None, need_task_ids=False\n ):\n \"\"\"Emit a new Tuple to a stream.\n\n :param tup: the Tuple payload to send to Storm, should contain only\n JSON-serializable data.\n :type tup: :class:`list` or :class:`pystorm.component.Tuple`\n :param stream: the ID of the stream to emit this Tuple to. Specify\n ``None`` to emit to default stream.\n :type stream: str\n :param anchors: IDs the Tuples (or :class:`pystorm.component.Tuple`\n instances) which the emitted Tuples should be anchored\n to. If ``auto_anchor`` is set to ``True`` and\n you have not specified ``anchors``, ``anchors`` will be\n set to the incoming/most recent Tuple ID(s).\n :type anchors: list\n :param direct_task: the task to send the Tuple to.\n :type direct_task: int\n :param need_task_ids: indicate whether or not you'd like the task IDs\n the Tuple was emitted (default: ``False``).\n :type need_task_ids: bool\n\n :returns: ``None``, unless ``need_task_ids=True``, in which case it will\n be a ``list`` of task IDs that the Tuple was sent to if. Note\n that when specifying direct_task, this will be equal to\n ``[direct_task]``.\n \"\"\"\n if anchors is None:\n anchors = self._current_tups if self.auto_anchor else []\n anchors = [a.id if isinstance(a, Tuple) else a for a in anchors]\n\n return super(Bolt, self).emit(\n tup,\n stream=stream,\n anchors=anchors,\n direct_task=direct_task,\n need_task_ids=need_task_ids,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nindicates that processing of a Tuple has succeeded.", "response": "def ack(self, tup):\n \"\"\"Indicate that processing of a Tuple has succeeded.\n\n :param tup: the Tuple to acknowledge.\n :type tup: :class:`str` or :class:`pystorm.component.Tuple`\n \"\"\"\n tup_id = tup.id if isinstance(tup, Tuple) else tup\n self.send_message({\"command\": \"ack\", \"id\": tup_id})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nindicating that processing of a Tuple has failed.", "response": "def fail(self, tup):\n \"\"\"Indicate that processing of a Tuple has failed.\n\n :param tup: the Tuple to fail (its ``id`` if ``str``).\n :type tup: :class:`str` or :class:`pystorm.component.Tuple`\n \"\"\"\n tup_id = tup.id if isinstance(tup, Tuple) else tup\n self.send_message({\"command\": \"fail\", \"id\": tup_id})"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nhandling an exception raised while running the run loop.", "response": "def _handle_run_exception(self, exc):\n \"\"\"Process an exception encountered while running the ``run()`` loop.\n\n Called right before program exits.\n \"\"\"\n if len(self._current_tups) == 1:\n tup = self._current_tups[0]\n self.raise_exception(exc, tup)\n if self.auto_fail:\n self.fail(tup)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmodifies emit that will not return task IDs after emitting.", "response": "def emit(self, tup, **kwargs):\n \"\"\"Modified emit that will not return task IDs after emitting.\n\n See :class:`pystorm.component.Bolt` for more information.\n\n :returns: ``None``.\n \"\"\"\n kwargs[\"need_task_ids\"] = False\n return super(BatchingBolt, self).emit(tup, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_tick(self, tick_tup):\n self._tick_counter += 1\n # ACK tick Tuple immediately, since it's just responsible for counter\n self.ack(tick_tup)\n if self._tick_counter > self.ticks_between_batches and self._batches:\n self.process_batches()\n self._tick_counter = 0", "response": "Increment tick counter and call process_batch for all current Tuples."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\niterates through all batches and process them and ack.", "response": "def process_batches(self):\n \"\"\"Iterate through all batches, call process_batch on them, and ack.\n\n Separated out for the rare instances when we want to subclass\n BatchingBolt and customize what mechanism causes batches to be\n processed.\n \"\"\"\n for key, batch in iteritems(self._batches):\n self._current_tups = batch\n self._current_key = key\n self.process_batch(key, batch)\n if self.auto_ack:\n for tup in batch:\n self.ack(tup)\n # Set current batch to [] so that we know it was acked if a\n # later batch raises an exception\n self._current_key = None\n self._batches[key] = []\n self._batches = defaultdict(list)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process(self, tup):\n # Append latest Tuple to batches\n group_key = self.group_key(tup)\n self._batches[group_key].append(tup)", "response": "Group non - tick Tuples into batches by group_key."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _handle_run_exception(self, exc):\n self.raise_exception(exc, self._current_tups)\n\n if self.auto_fail:\n failed = set()\n for key, batch in iteritems(self._batches):\n # Only wipe out batches other than current for exit_on_exception\n if self.exit_on_exception or key == self._current_key:\n for tup in batch:\n self.fail(tup)\n failed.add(tup.id)\n\n # Fail current batch or tick Tuple if we have one\n for tup in self._current_tups:\n if tup.id not in failed:\n self.fail(tup)\n\n # Reset current batch info\n self._batches[self._current_key] = []\n self._current_key = None", "response": "Handle an exception raised while running the run loop."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _batch_entry_run(self):\n time.sleep(self.secs_between_batches)\n with self._batch_lock:\n self.process_batches()", "response": "The inside of _batch_entry s infinite loop."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _batch_entry(self):\n try:\n while True:\n self._batch_entry_run()\n except:\n self.exc_info = sys.exc_info()\n os.kill(self.pid, signal.SIGUSR1)", "response": "Entry point for the batcher thread."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _run(self):\n tup = self.read_tuple()\n with self._batch_lock:\n self._current_tups = [tup]\n if self.is_heartbeat(tup):\n self.send_message({\"command\": \"sync\"})\n elif self.is_tick(tup):\n self.process_tick(tup)\n else:\n self.process(tup)\n # reset so that we don't accidentally fail the wrong Tuples\n # if a successive call to read_tuple fails\n self._current_tups = []", "response": "This method is called by the BatchingBolt s main thread to process the current Tuples\n ."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nserializing a message dictionary and write it to the output stream.", "response": "def send_message(self, msg_dict):\n \"\"\"Serialize a message dictionary and write it to the output stream.\"\"\"\n with self._writer_lock:\n try:\n self.output_stream.flush()\n self.output_stream.write(self.serialize_dict(msg_dict))\n self.output_stream.flush()\n except IOError:\n raise StormWentAwayError()\n except:\n log.exception(\"Failed to send message: %r\", msg_dict)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting the FFI result to Python data structures.", "response": "def _void_array_to_list(restuple, _func, _args):\n \"\"\" Convert the FFI result to Python data structures \"\"\"\n shape = (restuple.e.len, 1)\n array_size = np.prod(shape)\n mem_size = 8 * array_size\n\n array_str_e = string_at(restuple.e.data, mem_size)\n array_str_n = string_at(restuple.n.data, mem_size)\n\n ls_e = np.frombuffer(array_str_e, float, array_size).tolist()\n ls_n = np.frombuffer(array_str_n, float, array_size).tolist()\n\n return ls_e, ls_n"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads a data file and return it as a list of lines.", "response": "def load_data_file(filename, encoding='utf-8'):\n \"\"\"Load a data file and return it as a list of lines.\n\n Parameters:\n filename: The name of the file (no directories included).\n encoding: The file encoding. Defaults to utf-8.\n\n \"\"\"\n data = pkgutil.get_data(PACKAGE_NAME, os.path.join(DATA_DIR, filename))\n return data.decode(encoding).splitlines()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads the word and character mapping data into a dictionary.", "response": "def _load_data():\n \"\"\"Load the word and character mapping data into a dictionary.\n\n In the data files, each line is formatted like this:\n HANZI PINYIN_READING/PINYIN_READING\n\n So, lines need to be split by '\\t' and then the Pinyin readings need to be\n split by '/'.\n\n \"\"\"\n data = {}\n for name, file_name in (('words', 'hanzi_pinyin_words.tsv'),\n ('characters', 'hanzi_pinyin_characters.tsv')):\n # Split the lines by tabs: [[hanzi, pinyin]...].\n lines = [line.split('\\t') for line in\n dragonmapper.data.load_data_file(file_name)]\n # Make a dictionary: {hanzi: [pinyin, pinyin]...}.\n data[name] = {hanzi: pinyin.split('/') for hanzi, pinyin in lines}\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the Pinyin reading for a Chinese word.", "response": "def _hanzi_to_pinyin(hanzi):\n \"\"\"Return the Pinyin reading for a Chinese word.\n\n If the given string *hanzi* matches a CC-CEDICT word, the return value is\n formatted like this: [WORD_READING1, WORD_READING2, ...]\n\n If the given string *hanzi* doesn't match a CC-CEDICT word, the return\n value is formatted like this: [[CHAR_READING1, CHAR_READING2 ...], ...]\n\n When returning character readings, if a character wasn't recognized, the\n original character is returned, e.g. [[CHAR_READING1, ...], CHAR, ...]\n\n \"\"\"\n try:\n return _HANZI_PINYIN_MAP['words'][hanzi]\n except KeyError:\n return [_CHARACTERS.get(character, character) for character in hanzi]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nencloses a reading within a container.", "response": "def _enclose_readings(container, readings):\n \"\"\"Enclose a reading within a container, e.g. '[]'.\"\"\"\n container_start, container_end = tuple(container)\n enclosed_readings = '%(container_start)s%(readings)s%(container_end)s' % {\n 'container_start': container_start, 'container_end': container_end,\n 'readings': readings}\n return enclosed_readings"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_pinyin(s, delimiter=' ', all_readings=False, container='[]',\n accented=True):\n \"\"\"Convert a string's Chinese characters to Pinyin readings.\n\n *s* is a string containing Chinese characters. *accented* is a\n boolean value indicating whether to return accented or numbered Pinyin\n readings.\n\n *delimiter* is the character used to indicate word boundaries in *s*.\n This is used to differentiate between words and characters so that a more\n accurate reading can be returned.\n\n *all_readings* is a boolean value indicating whether or not to return all\n possible readings in the case of words/characters that have multiple\n readings. *container* is a two character string that is used to\n enclose words/characters if *all_readings* is ``True``. The default\n ``'[]'`` is used like this: ``'[READING1/READING2]'``.\n\n Characters not recognized as Chinese are left untouched.\n\n \"\"\"\n hanzi = s\n pinyin = ''\n\n # Process the given string.\n while hanzi:\n\n # Get the next match in the given string.\n match = re.search('[^%s%s]+' % (delimiter, zhon.hanzi.punctuation),\n hanzi)\n\n # There are no more matches, but the string isn't finished yet.\n if match is None and hanzi:\n pinyin += hanzi\n break\n\n match_start, match_end = match.span()\n\n # Process the punctuation marks that occur before the match.\n if match_start > 0:\n pinyin += hanzi[0:match_start]\n\n # Get the Chinese word/character readings.\n readings = _hanzi_to_pinyin(match.group())\n\n # Process the returned word readings.\n if match.group() in _WORDS:\n if all_readings:\n reading = _enclose_readings(container,\n _READING_SEPARATOR.join(readings))\n else:\n reading = readings[0]\n pinyin += reading\n\n # Process the returned character readings.\n else:\n # Process each character individually.\n for character in readings:\n # Don't touch unrecognized characters.\n if isinstance(character, str):\n pinyin += character\n # Format multiple readings.\n elif isinstance(character, list) and all_readings:\n pinyin += _enclose_readings(\n container, _READING_SEPARATOR.join(character))\n # Select and format the most common reading.\n elif isinstance(character, list) and not all_readings:\n # Add an apostrophe to separate syllables.\n if (pinyin and character[0][0] in zhon.pinyin.vowels and\n pinyin[-1] in zhon.pinyin.lowercase):\n pinyin += \"'\"\n pinyin += character[0]\n\n # Move ahead in the given string.\n hanzi = hanzi[match_end:]\n\n if accented:\n return pinyin\n else:\n return accented_to_numbered(pinyin)", "response": "Convert a string s Chinese characters to Pinyin readings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a string s Chinese characters to Zhuyin readings.", "response": "def to_zhuyin(s, delimiter=' ', all_readings=False, container='[]'):\n \"\"\"Convert a string's Chinese characters to Zhuyin readings.\n\n *s* is a string containing Chinese characters.\n\n *delimiter* is the character used to indicate word boundaries in *s*.\n This is used to differentiate between words and characters so that a more\n accurate reading can be returned.\n\n *all_readings* is a boolean value indicating whether or not to return all\n possible readings in the case of words/characters that have multiple\n readings. *container* is a two character string that is used to\n enclose words/characters if *all_readings* is ``True``. The default\n ``'[]'`` is used like this: ``'[READING1/READING2]'``.\n\n Characters not recognized as Chinese are left untouched.\n\n \"\"\"\n numbered_pinyin = to_pinyin(s, delimiter, all_readings, container, False)\n zhuyin = pinyin_to_zhuyin(numbered_pinyin)\n return zhuyin"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_ipa(s, delimiter=' ', all_readings=False, container='[]'):\n numbered_pinyin = to_pinyin(s, delimiter, all_readings, container, False)\n ipa = pinyin_to_ipa(numbered_pinyin)\n return ipa", "response": "Convert a string s Chinese characters to IPA."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nloading the transcription mapping data into a dictionary.", "response": "def _load_data():\n \"\"\"Load the transcription mapping data into a dictionary.\"\"\"\n lines = dragonmapper.data.load_data_file('transcriptions.csv')\n pinyin_map, zhuyin_map, ipa_map = {}, {}, {}\n for line in lines:\n p, z, i = line.split(',')\n pinyin_map[p] = {'Zhuyin': z, 'IPA': i}\n zhuyin_map[z] = {'Pinyin': p, 'IPA': i}\n ipa_map[i] = {'Pinyin': p, 'Zhuyin': z}\n return pinyin_map, zhuyin_map, ipa_map"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a numbered Pinyin vowel to an accented Pinyin vowel.", "response": "def _numbered_vowel_to_accented(vowel, tone):\n \"\"\"Convert a numbered Pinyin vowel to an accented Pinyin vowel.\"\"\"\n if isinstance(tone, int):\n tone = str(tone)\n return _PINYIN_TONES[vowel + tone]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts an accented Pinyin vowel to a numbered Pinyin vowel.", "response": "def _accented_vowel_to_numbered(vowel):\n \"\"\"Convert an accented Pinyin vowel to a numbered Pinyin vowel.\"\"\"\n for numbered_vowel, accented_vowel in _PINYIN_TONES.items():\n if vowel == accented_vowel:\n return tuple(numbered_vowel)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the syllable and tone of a numbered Pinyin syllable.", "response": "def _parse_numbered_syllable(unparsed_syllable):\n \"\"\"Return the syllable and tone of a numbered Pinyin syllable.\"\"\"\n tone_number = unparsed_syllable[-1]\n if not tone_number.isdigit():\n syllable, tone = unparsed_syllable, '5'\n elif tone_number == '0':\n syllable, tone = unparsed_syllable[:-1], '5'\n elif tone_number in '12345':\n syllable, tone = unparsed_syllable[:-1], tone_number\n else:\n raise ValueError(\"Invalid syllable: %s\" % unparsed_syllable)\n return syllable, tone"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _parse_accented_syllable(unparsed_syllable):\n if unparsed_syllable[0] == '\\u00B7':\n # Special case for middle dot tone mark.\n return unparsed_syllable[1:], '5'\n for character in unparsed_syllable:\n if character in _ACCENTED_VOWELS:\n vowel, tone = _accented_vowel_to_numbered(character)\n return unparsed_syllable.replace(character, vowel), tone\n return unparsed_syllable, '5'", "response": "Return the syllable and tone of an accented Pinyin syllable."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the syllable and tone of a Zhuyin syllable.", "response": "def _parse_zhuyin_syllable(unparsed_syllable):\n \"\"\"Return the syllable and tone of a Zhuyin syllable.\"\"\"\n zhuyin_tone = unparsed_syllable[-1]\n if zhuyin_tone in zhon.zhuyin.characters:\n syllable, tone = unparsed_syllable, '1'\n elif zhuyin_tone in zhon.zhuyin.marks:\n for tone_number, tone_mark in _ZHUYIN_TONES.items():\n if zhuyin_tone == tone_mark:\n syllable, tone = unparsed_syllable[:-1], tone_number\n else:\n raise ValueError(\"Invalid syllable: %s\" % unparsed_syllable)\n\n return syllable, tone"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_ipa_syllable(unparsed_syllable):\n ipa_tone = re.search('[%(marks)s]+' % {'marks': _IPA_MARKS},\n unparsed_syllable)\n if not ipa_tone:\n syllable, tone = unparsed_syllable, '5'\n else:\n for tone_number, tone_mark in _IPA_TONES.items():\n if ipa_tone.group() == tone_mark:\n tone = tone_number\n break\n syllable = unparsed_syllable[0:ipa_tone.start()]\n return syllable, tone", "response": "Return the syllable and tone of an IPA syllable."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrestore a lowercase string s characters to their original case.", "response": "def _restore_case(s, memory):\n \"\"\"Restore a lowercase string's characters to their original case.\"\"\"\n cased_s = []\n for i, c in enumerate(s):\n if i + 1 > len(memory):\n break\n cased_s.append(c if memory[i] else c.upper())\n return ''.join(cased_s)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a numbered Pinyin syllable to an accented Pinyin syllable.", "response": "def numbered_syllable_to_accented(s):\n \"\"\"Convert numbered Pinyin syllable *s* to an accented Pinyin syllable.\n\n It implements the following algorithm to determine where to place tone\n marks:\n\n 1. If the syllable has an 'a', 'e', or 'o' (in that order), put the\n tone mark over that vowel.\n 2. Otherwise, put the tone mark on the last vowel.\n\n \"\"\"\n if s == 'r5':\n return 'r' # Special case for 'r' suffix.\n\n lowercase_syllable, case_memory = _lower_case(s)\n syllable, tone = _parse_numbered_syllable(lowercase_syllable)\n syllable = syllable.replace('v', '\\u00fc')\n if re.search('[%s]' % _UNACCENTED_VOWELS, syllable) is None:\n return s\n if 'a' in syllable:\n accented_a = _numbered_vowel_to_accented('a', tone)\n accented_syllable = syllable.replace('a', accented_a)\n elif 'e' in syllable:\n accented_e = _numbered_vowel_to_accented('e', tone)\n accented_syllable = syllable.replace('e', accented_e)\n elif 'o' in syllable:\n accented_o = _numbered_vowel_to_accented('o', tone)\n accented_syllable = syllable.replace('o', accented_o)\n else:\n vowel = syllable[max(map(syllable.rfind, _UNACCENTED_VOWELS))]\n accented_vowel = _numbered_vowel_to_accented(vowel, tone)\n accented_syllable = syllable.replace(vowel, accented_vowel)\n return _restore_case(accented_syllable, case_memory)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert accented Pinyin syllable s to a numbered Pinyin syllable.", "response": "def accented_syllable_to_numbered(s):\n \"\"\"Convert accented Pinyin syllable *s* to a numbered Pinyin syllable.\"\"\"\n if s[0] == '\\u00B7':\n lowercase_syllable, case_memory = _lower_case(s[1:])\n lowercase_syllable = '\\u00B7' + lowercase_syllable\n else:\n lowercase_syllable, case_memory = _lower_case(s)\n numbered_syllable, tone = _parse_accented_syllable(lowercase_syllable)\n return _restore_case(numbered_syllable, case_memory) + tone"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a Pinyin syllable to a Zhuyin syllable.", "response": "def pinyin_syllable_to_zhuyin(s):\n \"\"\"Convert Pinyin syllable *s* to a Zhuyin syllable.\"\"\"\n pinyin_syllable, tone = _parse_pinyin_syllable(s)\n try:\n zhuyin_syllable = _PINYIN_MAP[pinyin_syllable.lower()]['Zhuyin']\n except KeyError:\n raise ValueError('Not a valid syllable: %s' % s)\n return zhuyin_syllable + _ZHUYIN_TONES[tone]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pinyin_syllable_to_ipa(s):\n pinyin_syllable, tone = _parse_pinyin_syllable(s)\n try:\n ipa_syllable = _PINYIN_MAP[pinyin_syllable.lower()]['IPA']\n except KeyError:\n raise ValueError('Not a valid syllable: %s' % s)\n return ipa_syllable + _IPA_TONES[tone]", "response": "Convert Pinyin syllable to an IPA syllable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a Zhuyin syllable to a numbered Pinyin syllable.", "response": "def _zhuyin_syllable_to_numbered(s):\n \"\"\"Convert Zhuyin syllable *s* to a numbered Pinyin syllable.\"\"\"\n zhuyin_syllable, tone = _parse_zhuyin_syllable(s)\n try:\n pinyin_syllable = _ZHUYIN_MAP[zhuyin_syllable]['Pinyin']\n except KeyError:\n raise ValueError('Not a valid syllable: %s' % s)\n return pinyin_syllable + tone"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert an IPA syllable to a numbered Pinyin syllable.", "response": "def _ipa_syllable_to_numbered(s):\n \"\"\"Convert IPA syllable *s* to a numbered Pinyin syllable.\"\"\"\n ipa_syllable, tone = _parse_ipa_syllable(s)\n try:\n pinyin_syllable = _IPA_MAP[ipa_syllable]['Pinyin']\n except KeyError:\n raise ValueError('Not a valid syllable: %s' % s)\n return pinyin_syllable + tone"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a string s syllables to a different transcription system.", "response": "def _convert(s, re_pattern, syllable_function, add_apostrophes=False,\n remove_apostrophes=False, separate_syllables=False):\n \"\"\"Convert a string's syllables to a different transcription system.\"\"\"\n original = s\n new = ''\n while original:\n match = re.search(re_pattern, original, re.IGNORECASE | re.UNICODE)\n if match is None and original:\n # There are no more matches, but the given string isn't fully\n # processed yet.\n new += original\n break\n match_start, match_end = match.span()\n if match_start > 0: # Handle extra characters before matched syllable.\n if (new and remove_apostrophes and match_start == 1 and\n original[0] == \"'\"):\n pass # Remove the apostrophe between Pinyin syllables.\n if separate_syllables: # Separate syllables by a space.\n new += ' '\n else:\n new += original[0:match_start]\n else: # Matched syllable starts immediately.\n if new and separate_syllables: # Separate syllables by a space.\n new += ' '\n elif (new and add_apostrophes and\n match.group()[0].lower() in _UNACCENTED_VOWELS):\n new += \"'\"\n # Convert the matched syllable.\n new += syllable_function(match.group())\n original = original[match_end:]\n return new"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef numbered_to_accented(s):\n return _convert(s, zhon.pinyin.syllable, numbered_syllable_to_accented,\n add_apostrophes=True)", "response": "Convert all numbered Pinyin syllables in * s * to accented Pinyin."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts all Pinyin syllables in s to Zhuyin.", "response": "def pinyin_to_zhuyin(s):\n \"\"\"Convert all Pinyin syllables in *s* to Zhuyin.\n\n Spaces are added between connected syllables and syllable-separating\n apostrophes are removed.\n\n \"\"\"\n return _convert(s, zhon.pinyin.syllable, pinyin_syllable_to_zhuyin,\n remove_apostrophes=True, separate_syllables=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts all Pinyin syllables in s to IPA.", "response": "def pinyin_to_ipa(s):\n \"\"\"Convert all Pinyin syllables in *s* to IPA.\n\n Spaces are added between connected syllables and syllable-separating\n apostrophes are removed.\n\n \"\"\"\n return _convert(s, zhon.pinyin.syllable, pinyin_syllable_to_ipa,\n remove_apostrophes=True, separate_syllables=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef zhuyin_to_pinyin(s, accented=True):\n if accented:\n function = _zhuyin_syllable_to_accented\n else:\n function = _zhuyin_syllable_to_numbered\n return _convert(s, zhon.zhuyin.syllable, function)", "response": "Convert all Zhuyin syllables in s to Pinyin syllables."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts all IPA syllables in s to Pinyin syllables.", "response": "def ipa_to_pinyin(s, accented=True):\n \"\"\"Convert all IPA syllables in *s* to Pinyin.\n\n If *accented* is ``True``, diacritics are added to the Pinyin syllables. If\n it's ``False``, numbers are used to indicate tone.\n\n \"\"\"\n if accented:\n function = _ipa_syllable_to_accented\n else:\n function = _ipa_syllable_to_numbered\n return _convert(s, _IPA_SYLLABLE, function)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a Chinese transcription string to a Pinyin syllab.", "response": "def to_pinyin(s, accented=True):\n \"\"\"Convert *s* to Pinyin.\n\n If *accented* is ``True``, diacritics are added to the Pinyin syllables. If\n it's ``False``, numbers are used to indicate tone.\n\n \"\"\"\n identity = identify(s)\n if identity == PINYIN:\n if _has_accented_vowels(s):\n return s if accented else accented_to_numbered(s)\n else:\n return numbered_to_accented(s) if accented else s\n elif identity == ZHUYIN:\n return zhuyin_to_pinyin(s, accented=accented)\n elif identity == IPA:\n return ipa_to_pinyin(s, accented=accented)\n else:\n raise ValueError(\"String is not a valid Chinese transcription.\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert string s to Zhuyin.", "response": "def to_zhuyin(s):\n \"\"\"Convert *s* to Zhuyin.\"\"\"\n identity = identify(s)\n if identity == ZHUYIN:\n return s\n elif identity == PINYIN:\n return pinyin_to_zhuyin(s)\n elif identity == IPA:\n return ipa_to_zhuyin(s)\n else:\n raise ValueError(\"String is not a valid Chinese transcription.\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_ipa(s):\n identity = identify(s)\n if identity == IPA:\n return s\n elif identity == PINYIN:\n return pinyin_to_ipa(s)\n elif identity == ZHUYIN:\n return zhuyin_to_ipa(s)\n else:\n raise ValueError(\"String is not a valid Chinese transcription.\")", "response": "Convert string s to IPA."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _is_pattern_match(re_pattern, s):\n match = re.match(re_pattern, s, re.I)\n return match.group() == s if match else False", "response": "Check if a re pattern expression matches an entire string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if *s* consists of valid Pinyin.", "response": "def is_pinyin(s):\n \"\"\"Check if *s* consists of valid Pinyin.\"\"\"\n re_pattern = ('(?:%(word)s|[ \\t%(punctuation)s])+' %\n {'word': zhon.pinyin.word,\n 'punctuation': zhon.pinyin.punctuation})\n return _is_pattern_match(re_pattern, s)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if a string s contains Zhuyin - compatible characters.", "response": "def is_zhuyin_compatible(s):\n \"\"\"Checks if *s* is consists of Zhuyin-compatible characters.\n\n This does not check if *s* contains valid Zhuyin syllables; for that\n see :func:`is_zhuyin`.\n\n Besides Zhuyin characters and tone marks, spaces are also accepted.\n This function checks that all characters in *s* exist in\n :data:`zhon.zhuyin.characters`, :data:`zhon.zhuyin.marks`, or ``' '``.\n\n \"\"\"\n printable_zhuyin = zhon.zhuyin.characters + zhon.zhuyin.marks + ' '\n return _is_pattern_match('[%s]+' % printable_zhuyin, s)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if *s* consists of valid Chinese IPA.", "response": "def is_ipa(s):\n \"\"\"Check if *s* consists of valid Chinese IPA.\"\"\"\n re_pattern = ('(?:%(syllable)s|[ \\t%(punctuation)s])+' %\n {'syllable': _IPA_SYLLABLE,\n 'punctuation': zhon.pinyin.punctuation})\n return _is_pattern_match(re_pattern, s)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nidentifying a given string s transcription system.", "response": "def identify(s):\n \"\"\"Identify a given string's transcription system.\n\n *s* is the string to identify. The string is checked to see if its\n contents are valid Pinyin, Zhuyin, or IPA. The :data:`PINYIN`,\n :data:`ZHUYIN`, and :data:`IPA` constants are returned to indicate the\n string's identity.\n If *s* is not a valid transcription system, then :data:`UNKNOWN` is\n returned.\n\n When checking for valid Pinyin or Zhuyin, testing is done on a syllable\n level, not a character level. For example, just because a string is\n composed of characters used in Pinyin, doesn't mean that it will identify\n as Pinyin; it must actually consist of valid Pinyin syllables. The same\n applies for Zhuyin.\n\n When checking for IPA, testing is only done on a character level. In other\n words, a string just needs to consist of Chinese IPA characters in order\n to identify as IPA.\n\n \"\"\"\n if is_pinyin(s):\n return PINYIN\n elif is_zhuyin(s):\n return ZHUYIN\n elif is_ipa(s):\n return IPA\n else:\n return UNKNOWN"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\naccept an objective function for optimization.", "response": "def prepare(self, f):\n \"\"\"Accept an objective function for optimization.\"\"\"\n self.g = autograd.grad(f)\n self.h = autograd.hessian(f)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef optimize(self, x0, target):\n x = x0\n for _ in range(self.maxiter):\n delta = np.linalg.solve(self.h(x, target), -self.g(x, target))\n x = x + delta\n if np.linalg.norm(delta) < self.tol:\n break\n return x", "response": "Calculate an optimum argument of an objective function."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates an optimum argument of an objective function.", "response": "def optimize(self, x0, target):\n \"\"\"Calculate an optimum argument of an objective function.\"\"\"\n x = x0\n for i in range(self.maxiter):\n g = self.g(x, target)\n h = self.h(x, target)\n if i == 0:\n alpha = 0\n m = g\n else:\n alpha = - np.dot(m, np.dot(h, g)) / np.dot(m, np.dot(h, m))\n m = g + np.dot(alpha, m)\n t = - np.dot(m, g) / np.dot(m, np.dot(h, m))\n delta = np.dot(t, m)\n x = x + delta\n if np.linalg.norm(delta) < self.tol:\n break\n return x"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate an optimum argument of an objective function.", "response": "def optimize(self, angles0, target):\n \"\"\"Calculate an optimum argument of an objective function.\"\"\"\n def new_objective(angles):\n return self.f(angles, target)\n\n return scipy.optimize.minimize(\n new_objective,\n angles0,\n **self.optimizer_opt).x"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates an optimum argument of an objective function.", "response": "def optimize(self, angles0, target):\n \"\"\"Calculate an optimum argument of an objective function.\"\"\"\n def new_objective(angles):\n a = angles - angles0\n if isinstance(self.smooth_factor, (np.ndarray, list)):\n if len(a) == len(self.smooth_factor):\n return (self.f(angles, target) +\n np.sum(self.smooth_factor * np.power(a, 2)))\n else:\n raise ValueError('len(smooth_factor) != number of joints')\n else:\n return (self.f(angles, target) +\n self.smooth_factor * np.sum(np.power(a, 2)))\n\n return scipy.optimize.minimize(\n new_objective,\n angles0,\n **self.optimizer_opt).x"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates a position of the end - effector and return it.", "response": "def solve(self, angles):\n \"\"\"Calculate a position of the end-effector and return it.\"\"\"\n return reduce(\n lambda a, m: np.dot(m, a),\n reversed(self._matrices(angles)),\n np.array([0., 0., 0., 1.])\n )[:3]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates joint angles and returns it.", "response": "def solve(self, angles0, target):\n \"\"\"Calculate joint angles and returns it.\"\"\"\n return self.optimizer.optimize(np.array(angles0), target)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef matrix(self, _):\n x, y, z = self.coord\n return np.array([\n [1., 0., 0., x],\n [0., 1., 0., y],\n [0., 0., 1., z],\n [0., 0., 0., 1.]\n ])", "response": "Return translation matrix in homogeneous coordinates."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef matrix(self, angle):\n _rot_mat = {\n 'x': self._x_rot,\n 'y': self._y_rot,\n 'z': self._z_rot\n }\n return _rot_mat[self.axis](angle)", "response": "Return rotation matrix in homogeneous coordinates."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting a logger to send debug messages to the current session", "response": "def set_logger(self, logger):\n \"\"\"\n Set a logger to send debug messages to\n\n Parameters\n ----------\n logger : `Logger `_\n A python logger used to get debugging output from this module.\n \"\"\"\n self.__logger = logger\n self.session.set_logger(self.__logger)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef version(self):\n this_path = os.path.dirname(os.path.realpath(__file__))\n version_file = os.path.join(this_path, 'VERSION')\n return open(version_file).read().strip()", "response": "Returns the version number of the Lending Club Investor tool\n Returns -------\n string A version number of the Lending Club Investor tool\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nattempts to authenticate the user.", "response": "def authenticate(self, email=None, password=None):\n \"\"\"\n Attempt to authenticate the user.\n\n Parameters\n ----------\n email : string\n The email of a user on Lending Club\n password : string\n The user's password, for authentication.\n\n Returns\n -------\n boolean\n True if the user authenticated or raises an exception if not\n\n Raises\n ------\n session.AuthenticationError\n If authentication failed\n session.NetworkError\n If a network error occurred\n \"\"\"\n if self.session.authenticate(email, password):\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the cash balance available for investing the account.", "response": "def get_cash_balance(self):\n \"\"\"\n Returns the account cash balance available for investing\n\n Returns\n -------\n float\n The cash balance in your account.\n \"\"\"\n cash = False\n try:\n response = self.session.get('/browse/cashBalanceAj.action')\n json_response = response.json()\n\n if self.session.json_success(json_response):\n self.__log('Cash available: {0}'.format(json_response['cashBalance']))\n cash_value = json_response['cashBalance']\n\n # Convert currency to float value\n # Match values like $1,000.12 or 1,0000$\n cash_match = re.search('^[^0-9]?([0-9\\.,]+)[^0-9]?', cash_value)\n if cash_match:\n cash_str = cash_match.group(1)\n cash_str = cash_str.replace(',', '')\n cash = float(cash_str)\n else:\n self.__log('Could not get cash balance: {0}'.format(response.text))\n\n except Exception as e:\n self.__log('Could not get the cash balance on the account: Error: {0}\\nJSON: {1}'.format(str(e), response.text))\n raise e\n\n return cash"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget your list of named portfolios from the lendingclub. com base object.", "response": "def get_portfolio_list(self, names_only=False):\n \"\"\"\n Get your list of named portfolios from the lendingclub.com\n\n Parameters\n ----------\n names_only : boolean, optional\n If set to True, the function will return a list of portfolio names, instead of portfolio objects\n\n Returns\n -------\n list\n A list of portfolios (or names, if `names_only` is True)\n \"\"\"\n folios = []\n response = self.session.get('/data/portfolioManagement?method=getLCPortfolios')\n json_response = response.json()\n\n # Get portfolios and create a list of names\n if self.session.json_success(json_response):\n folios = json_response['results']\n\n if names_only is True:\n for i, folio in enumerate(folios):\n folios[i] = folio['portfolioName']\n\n return folios"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef assign_to_portfolio(self, portfolio_name, loan_id, order_id):\n response = None\n\n assert type(loan_id) == type(order_id), \"Both loan_id and order_id need to be the same type\"\n assert type(loan_id) in (int, list), \"loan_id and order_id can only be int or list types\"\n assert type(loan_id) is int or (type(loan_id) is list and len(loan_id) == len(order_id)), \"If order_id and loan_id are lists, they both need to be the same length\"\n\n # Data\n post = {\n 'loan_id': loan_id,\n 'record_id': loan_id,\n 'order_id': order_id\n }\n query = {\n 'method': 'createLCPortfolio',\n 'lcportfolio_name': portfolio_name\n }\n\n # Is it an existing portfolio\n existing = self.get_portfolio_list()\n for folio in existing:\n if folio['portfolioName'] == portfolio_name:\n query['method'] = 'addToLCPortfolio'\n\n # Send\n response = self.session.post('/data/portfolioManagement', query=query, data=post)\n json_response = response.json()\n\n # Failed\n if not self.session.json_success(json_response):\n raise LendingClubError('Could not assign order to portfolio \\'{0}\\''.format(portfolio_name), response)\n\n # Success\n else:\n\n # Assigned to another portfolio, for some reason, raise warning\n if 'portfolioName' in json_response and json_response['portfolioName'] != portfolio_name:\n raise LendingClubError('Added order to portfolio \"{0}\" - NOT - \"{1}\", and I don\\'t know why'.format(json_response['portfolioName'], portfolio_name))\n\n # Assigned to the correct portfolio\n else:\n self.__log('Added order to portfolio \"{0}\"'.format(portfolio_name))\n\n return True\n\n return False", "response": "Assign a note to a named portfolio."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsearches for a list of notes that can be invested in.", "response": "def search(self, filters=None, start_index=0, limit=100):\n \"\"\"\n Search for a list of notes that can be invested in.\n (similar to searching for notes in the Browse section on the site)\n\n Parameters\n ----------\n filters : lendingclub.filters.*, optional\n The filter to use to search for notes. If no filter is passed, a wildcard search\n will be performed.\n start_index : int, optional\n The result index to start on. By default only 100 records will be returned at a time, so use this\n to start at a later index in the results. For example, to get results 200 - 300, set `start_index` to 200.\n (default is 0)\n limit : int, optional\n The number of results to return per request. (default is 100)\n\n Returns\n -------\n dict\n A dictionary object with the list of matching loans under the `loans` key.\n \"\"\"\n assert filters is None or isinstance(filters, Filter), 'filter is not a lendingclub.filters.Filter'\n\n # Set filters\n if filters:\n filter_string = filters.search_string()\n else:\n filter_string = 'default'\n payload = {\n 'method': 'search',\n 'filter': filter_string,\n 'startindex': start_index,\n 'pagesize': limit\n }\n\n # Make request\n response = self.session.post('/browse/browseNotesAj.action', data=payload)\n json_response = response.json()\n\n if self.session.json_success(json_response):\n results = json_response['searchresult']\n\n # Normalize results by converting loanGUID -> loan_id\n for loan in results['loans']:\n loan['loan_id'] = int(loan['loanGUID'])\n\n # Validate that fractions do indeed match the filters\n if filters is not None:\n filters.validate(results['loans'])\n\n return results\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild a list of loan notes that are diversified by your min_percent request and max_percent request.", "response": "def build_portfolio(self, cash, max_per_note=25, min_percent=0, max_percent=20, filters=None, automatically_invest=False, do_not_clear_staging=False):\n \"\"\"\n Returns a list of loan notes that are diversified by your min/max percent request and filters.\n One way to invest in these loan notes, is to start an order and use add_batch to add all the\n loan fragments to them. (see examples)\n\n Parameters\n ----------\n cash : int\n The total amount you want to invest across a portfolio of loans (at least $25).\n max_per_note : int, optional\n The maximum dollar amount you want to invest per note. Must be a multiple of 25\n min_percent : int, optional\n THIS IS NOT PER NOTE, but the minimum average percent of return for the entire portfolio.\n max_percent : int, optional\n THIS IS NOT PER NOTE, but the maxmimum average percent of return for the entire portfolio.\n filters : lendingclub.filters.*, optional\n The filters to use to search for portfolios\n automatically_invest : boolean, optional\n If you want the tool to create an order and automatically invest in the portfolio that matches your filter.\n (default False)\n do_not_clear_staging : boolean, optional\n Similar to automatically_invest, don't do this unless you know what you're doing.\n Setting this to True stops the method from clearing the loan staging area before returning\n\n Returns\n -------\n dict\n A dict representing a new portfolio or False if nothing was found.\n If `automatically_invest` was set to `True`, the dict will contain an `order_id` key with\n the ID of the completed investment order.\n\n Notes\n -----\n **The min/max_percent parameters**\n\n When searching for portfolios, these parameters will match a portfolio of loan notes which have\n an **AVERAGE** percent return between these values. If there are multiple portfolio matches, the\n one closes to the max percent will be chosen.\n\n Examples\n --------\n Here we want to invest $400 in a portfolio with only B, C, D and E grade notes with an average overall return between 17% - 19%. This similar to finding a portfolio in the 'Invest' section on lendingclub.com::\n\n >>> from lendingclub import LendingClub\n >>> from lendingclub.filters import Filter\n >>> lc = LendingClub()\n >>> lc.authenticate()\n Email:test@test.com\n Password:\n True\n >>> filters = Filter() # Set the search filters (only B, C, D and E grade notes)\n >>> filters['grades']['C'] = True\n >>> filters['grades']['D'] = True\n >>> filters['grades']['E'] = True\n >>> lc.get_cash_balance() # See the cash you have available for investing\n 463.80000000000001\n\n >>> portfolio = lc.build_portfolio(400, # Invest $400 in a portfolio...\n min_percent=17.0, # Return percent average between 17 - 19%\n max_percent=19.0,\n max_per_note=50, # As much as $50 per note\n filters=filters) # Search using your filters\n\n >>> len(portfolio['loan_fractions']) # See how many loans are in this portfolio\n 16\n >>> loans_notes = portfolio['loan_fractions']\n >>> order = lc.start_order() # Start a new order\n >>> order.add_batch(loans_notes) # Add the loan notes to the order\n >>> order.execute() # Execute the order\n 1861880\n\n Here we do a similar search, but automatically invest the found portfolio. **NOTE** This does not allow\n you to review the portfolio before you invest in it.\n\n >>> from lendingclub import LendingClub\n >>> from lendingclub.filters import Filter\n >>> lc = LendingClub()\n >>> lc.authenticate()\n Email:test@test.com\n Password:\n True\n # Filter shorthand\n >>> filters = Filter({'grades': {'B': True, 'C': True, 'D': True, 'E': True}})\n >>> lc.get_cash_balance() # See the cash you have available for investing\n 463.80000000000001\n\n >>> portfolio = lc.build_portfolio(400,\n min_percent=17.0,\n max_percent=19.0,\n max_per_note=50,\n filters=filters,\n automatically_invest=True) # Same settings, except invest immediately\n\n >>> portfolio['order_id'] # See order ID\n 1861880\n \"\"\"\n assert filters is None or isinstance(filters, Filter), 'filter is not a lendingclub.filters.Filter'\n assert max_per_note >= 25, 'max_per_note must be greater than or equal to 25'\n\n # Set filters\n if filters:\n filter_str = filters.search_string()\n else:\n filter_str = 'default'\n\n # Start a new order\n self.session.clear_session_order()\n\n # Make request\n payload = {\n 'amount': cash,\n 'max_per_note': max_per_note,\n 'filter': filter_str\n }\n self.__log('POST VALUES -- amount: {0}, max_per_note: {1}, filter: ...'.format(cash, max_per_note))\n response = self.session.post('/portfolio/lendingMatchOptionsV2.action', data=payload)\n json_response = response.json()\n\n # Options were found\n if self.session.json_success(json_response) and 'lmOptions' in json_response:\n options = json_response['lmOptions']\n\n # Nothing found\n if type(options) is not list or json_response['numberTicks'] == 0:\n self.__log('No lending portfolios were returned with your search')\n return False\n\n # Choose an investment option based on the user's min/max values\n i = 0\n match_index = -1\n match_option = None\n for option in options:\n\n # A perfect match\n if option['percentage'] == max_percent:\n match_option = option\n match_index = i\n break\n\n # Over the max\n elif option['percentage'] > max_percent:\n break\n\n # Higher than the minimum percent and the current matched option\n elif option['percentage'] >= min_percent and (match_option is None or match_option['percentage'] < option['percentage']):\n match_option = option\n match_index = i\n\n i += 1\n\n # Nothing matched\n if match_option is None:\n self.__log('No portfolios matched your percentage requirements')\n return False\n\n # Mark this portfolio for investing (in order to get a list of all notes)\n payload = {\n 'order_amount': cash,\n 'lending_match_point': match_index,\n 'lending_match_version': 'v2'\n }\n self.session.get('/portfolio/recommendPortfolio.action', query=payload)\n\n # Get all loan fractions\n payload = {\n 'method': 'getPortfolio'\n }\n response = self.session.get('/data/portfolio', query=payload)\n json_response = response.json()\n\n # Extract fractions from response\n fractions = []\n if 'loanFractions' in json_response:\n fractions = json_response['loanFractions']\n\n # Normalize by converting loanFractionAmount to invest_amount\n for frac in fractions:\n frac['invest_amount'] = frac['loanFractionAmount']\n\n # Raise error if amount is greater than max_per_note\n if frac['invest_amount'] > max_per_note:\n raise LendingClubError('ERROR: LendingClub tried to invest ${0} in a loan note. Your max per note is set to ${1}. Portfolio investment canceled.'.format(frac['invest_amount'], max_per_note))\n\n if len(fractions) == 0:\n self.__log('The selected portfolio didn\\'t have any loans')\n return False\n match_option['loan_fractions'] = fractions\n\n # Validate that fractions do indeed match the filters\n if filters is not None:\n filters.validate(fractions)\n\n # Not investing -- reset portfolio search session and return\n if automatically_invest is not True:\n if do_not_clear_staging is not True:\n self.session.clear_session_order()\n\n # Invest in this porfolio\n elif automatically_invest is True: # just to be sure\n order = self.start_order()\n\n # This should probably only be ever done here...ever.\n order._Order__already_staged = True\n order._Order__i_know_what_im_doing = True\n\n order.add_batch(match_option['loan_fractions'])\n order_id = order.execute()\n match_option['order_id'] = order_id\n\n return match_option\n else:\n raise LendingClubError('Could not find any portfolio options that match your filters', response)\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_note(self, note_id):\n\n index = 0\n while True:\n notes = self.my_notes(start_index=index, sort_by='noteId')\n\n if notes['result'] != 'success':\n break\n\n # If the first note has a higher ID, we've passed it\n if notes['loans'][0]['noteId'] > note_id:\n break\n\n # If the last note has a higher ID, it could be in this record set\n if notes['loans'][-1]['noteId'] >= note_id:\n for note in notes['loans']:\n if note['noteId'] == note_id:\n return note\n\n index += 100\n\n return False", "response": "Get a specific loan note by ID."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsearches for the notes that are invested in.", "response": "def search_my_notes(self, loan_id=None, order_id=None, grade=None, portfolio_name=None, status=None, term=None):\n \"\"\"\n Search for notes you are invested in. Use the parameters to define how to search.\n Passing no parameters is the same as calling `my_notes(get_all=True)`\n\n Parameters\n ----------\n loan_id : int, optional\n Search for notes for a specific loan. Since a loan is broken up into a pool of notes, it's possible\n to invest multiple notes in a single loan\n order_id : int, optional\n Search for notes from a particular investment order.\n grade : {A, B, C, D, E, F, G}, optional\n Match by a particular loan grade\n portfolio_name : string, optional\n Search for notes in a portfolio with this name (case sensitive)\n status : string, {issued, in-review, in-funding, current, charged-off, late, in-grace-period, fully-paid}, optional\n The funding status string.\n term : {60, 36}, optional\n Term length, either 60 or 36 (for 5 year and 3 year, respectively)\n\n Returns\n -------\n dict\n A dictionary with a list of matching notes on the `loans` key\n \"\"\"\n assert grade is None or type(grade) is str, 'grade must be a string'\n assert portfolio_name is None or type(portfolio_name) is str, 'portfolio_name must be a string'\n\n index = 0\n found = []\n sort_by = 'orderId' if order_id is not None else 'loanId'\n group_id = order_id if order_id is not None else loan_id # first match by order, then by loan\n\n # Normalize grade\n if grade is not None:\n grade = grade[0].upper()\n\n # Normalize status\n if status is not None:\n status = re.sub('[^a-zA-Z\\-]', ' ', status.lower()) # remove all non alpha characters\n status = re.sub('days', ' ', status) # remove days\n status = re.sub('\\s+', '-', status.strip()) # replace spaces with dash\n status = re.sub('(^-+)|(-+$)', '', status)\n\n while True:\n notes = self.my_notes(start_index=index, sort_by=sort_by)\n\n if notes['result'] != 'success':\n break\n\n # If the first note has a higher ID, we've passed it\n if group_id is not None and notes['loans'][0][sort_by] > group_id:\n break\n\n # If the last note has a higher ID, it could be in this record set\n if group_id is None or notes['loans'][-1][sort_by] >= group_id:\n for note in notes['loans']:\n\n # Order ID, no match\n if order_id is not None and note['orderId'] != order_id:\n continue\n\n # Loan ID, no match\n if loan_id is not None and note['loanId'] != loan_id:\n continue\n\n # Grade, no match\n if grade is not None and note['rate'][0] != grade:\n continue\n\n # Portfolio, no match\n if portfolio_name is not None and note['portfolioName'][0] != portfolio_name:\n continue\n\n # Term, no match\n if term is not None and note['loanLength'] != term:\n continue\n\n # Status\n if status is not None:\n # Normalize status message\n nstatus = re.sub('[^a-zA-Z\\-]', ' ', note['status'].lower()) # remove all non alpha characters\n nstatus = re.sub('days', ' ', nstatus) # remove days\n nstatus = re.sub('\\s+', '-', nstatus.strip()) # replace spaces with dash\n nstatus = re.sub('(^-+)|(-+$)', '', nstatus)\n\n # No match\n if nstatus != status:\n continue\n\n # Must be a match\n found.append(note)\n\n index += 100\n\n return found"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add(self, loan_id, amount):\n assert amount > 0 and amount % 25 == 0, 'Amount must be a multiple of 25'\n assert type(amount) in (float, int), 'Amount must be a number'\n\n if type(loan_id) is dict:\n loan = loan_id\n assert 'loan_id' in loan and type(loan['loan_id']) is int, 'loan_id must be a number or dictionary containing a loan_id value'\n loan_id = loan['loan_id']\n\n assert type(loan_id) in [str, unicode, int], 'Loan ID must be an integer number or a string'\n self.loans[loan_id] = amount", "response": "Add a loan and amount to your order."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_batch(self, loans, batch_amount=None):\n assert batch_amount is None or batch_amount % 25 == 0, 'batch_amount must be a multiple of 25'\n\n # Add each loan\n assert type(loans) is list, 'The loans property must be a list. (not {0})'.format(type(loans))\n for loan in loans:\n loan_id = loan\n amount = batch_amount\n\n # Extract ID and amount from loan dict\n if type(loan) is dict:\n assert 'loan_id' in loan, 'Each loan dict must have a loan_id value'\n assert batch_amount or 'invest_amount' in loan, 'Could not determine how much to invest in loan {0}'.format(loan['loan_id'])\n\n loan_id = loan['loan_id']\n if amount is None and 'invest_amount' in loan:\n amount = loan['invest_amount']\n\n assert amount is not None, 'Could not determine how much to invest in loan {0}'.format(loan_id)\n assert amount % 25 == 0, 'Amount to invest must be a multiple of 25 (loan_id: {0})'.format(loan_id)\n\n self.add(loan_id, amount)", "response": "Adds a batch of loans to your order."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef execute(self, portfolio_name=None):\n assert self.order_id == 0, 'This order has already been place. Start a new order.'\n assert len(self.loans) > 0, 'There aren\\'t any loans in your order'\n\n # Place the order\n self.__stage_order()\n token = self.__get_strut_token()\n self.order_id = self.__place_order(token)\n\n self.__log('Order #{0} was successfully submitted'.format(self.order_id))\n\n # Assign to portfolio\n if portfolio_name:\n return self.assign_to_portfolio(portfolio_name)\n\n return self.order_id", "response": "Place the order with LendingClub and assign the loan notes to the portfolio."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nassigns all the notes in this order to a portfolio.", "response": "def assign_to_portfolio(self, portfolio_name=None):\n \"\"\"\n Assign all the notes in this order to a portfolio\n\n Parameters\n ----------\n portfolio_name -- The name of the portfolio to assign it to (new or existing)\n\n Raises\n ------\n LendingClubError\n\n Returns\n -------\n boolean\n True on success\n \"\"\"\n assert self.order_id > 0, 'You need to execute this order before you can assign to a portfolio.'\n\n # Get loan IDs as a list\n loan_ids = self.loans.keys()\n\n # Make a list of 1 order ID per loan\n order_ids = [self.order_id]*len(loan_ids)\n\n return self.lc.assign_to_portfolio(portfolio_name, loan_ids, order_ids)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstage all the loans in the batch to the order session", "response": "def __stage_order(self):\n \"\"\"\n Add all the loans to the LC order session\n \"\"\"\n\n # Skip staging...probably not a good idea...you've been warned\n if self.__already_staged is True and self.__i_know_what_im_doing is True:\n self.__log('Not staging the order...I hope you know what you\\'re doing...'.format(len(self.loans)))\n return\n\n self.__log('Staging order for {0} loan notes...'.format(len(self.loans)))\n\n # Create a fresh order session\n self.lc.session.clear_session_order()\n\n #\n # Stage all the loans to the order\n #\n loan_ids = self.loans.keys()\n self.__log('Staging loans {0}'.format(loan_ids))\n\n # LendingClub requires you to search for the loans before you can stage them\n f = FilterByLoanID(loan_ids)\n results = self.lc.search(f, limit=len(self.loans))\n if len(results['loans']) == 0 or results['totalRecords'] != len(self.loans):\n raise LendingClubError('Could not stage the loans. The number of loans in your batch does not match totalRecords. {0} != {1}'.format(len(self.loans), results['totalRecords']), results)\n\n # Stage each loan\n for loan_id, amount in self.loans.iteritems():\n payload = {\n 'method': 'addToPortfolio',\n 'loan_id': loan_id,\n 'loan_amount': amount,\n 'remove': 'false'\n }\n response = self.lc.session.get('/data/portfolio', query=payload)\n json_response = response.json()\n\n # Ensure it was successful before moving on\n if not self.lc.session.json_success(json_response):\n raise LendingClubError('Could not stage loan {0} on the order: {1}'.format(loan_id, response.text), response)\n\n #\n # Add all staged loans to the order\n #\n payload = {\n 'method': 'addToPortfolioNew'\n }\n response = self.lc.session.get('/data/portfolio', query=payload)\n json_response = response.json()\n\n if self.lc.session.json_success(json_response):\n self.__log(json_response['message'])\n return True\n else:\n raise self.__log('Could not add loans to the order: {0}'.format(response.text))\n raise LendingClubError('Could not add loans to the order', response.text)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the struts token from the place order HTML.", "response": "def __get_strut_token(self):\n \"\"\"\n Move the staged loan notes to the order stage and get the struts token\n from the place order HTML.\n The order will not be placed until calling _confirm_order()\n\n Returns\n -------\n dict\n A dict with the token name and value\n \"\"\"\n\n try:\n # Move to the place order page and get the struts token\n\n response = self.lc.session.get('/portfolio/placeOrder.action')\n soup = BeautifulSoup(response.text, \"html5lib\")\n\n\n # Example HTML with the stuts token:\n \"\"\"\n \n \n \"\"\"\n # 'struts.token.name' defines the field name with the token value\n\n strut_tag = None\n strut_token_name = soup.find('input', {'name': 'struts.token.name'})\n if strut_token_name and strut_token_name['value'].strip():\n\n # Get form around the strut.token.name element\n form = soup.form # assumed\n for parent in strut_token_name.parents:\n if parent and parent.name == 'form':\n form = parent\n break\n\n # Get strut token value\n strut_token_name = strut_token_name['value']\n strut_tag = soup.find('input', {'name': strut_token_name})\n if strut_tag and strut_tag['value'].strip():\n return {'name': strut_token_name, 'value': strut_tag['value'].strip()}\n\n # No strut token found\n self.__log('No struts token! HTML: {0}'.format(response.text))\n raise LendingClubError('No struts token. Please report this error.', response)\n\n except Exception as e:\n self.__log('Could not get struts token. Error message: {0}'.format(str(e)))\n raise LendingClubError('Could not get struts token. Error message: {0}'.format(str(e)))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplaces the order in the struts token.", "response": "def __place_order(self, token):\n \"\"\"\n Use the struts token to place the order.\n\n Parameters\n ----------\n token : string\n The struts token received from the place order page\n\n Returns\n -------\n int\n The completed order ID.\n \"\"\"\n order_id = 0\n response = None\n\n if not token or token['value'] == '':\n raise LendingClubError('The token parameter is False, None or unknown.')\n\n # Process order confirmation page\n try:\n # Place the order\n payload = {}\n if token:\n payload['struts.token.name'] = token['name']\n payload[token['name']] = token['value']\n response = self.lc.session.post('/portfolio/orderConfirmed.action', data=payload)\n\n # Process HTML for the order ID\n html = response.text\n soup = BeautifulSoup(html, 'html5lib')\n\n # Order num\n order_field = soup.find(id='order_id')\n if order_field:\n order_id = int(order_field['value'])\n\n # Did not find an ID\n if order_id == 0:\n self.__log('An investment order was submitted, but a confirmation ID could not be determined')\n raise LendingClubError('No order ID was found when placing the order.', response)\n else:\n return order_id\n\n except Exception as e:\n raise LendingClubError('Could not place the order: {0}'.format(str(e)), response)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if the time since the last HTTP request is under the session timeout limit. If it is too long since the last HTTP request attempt to authenticate again.", "response": "def __continue_session(self):\n \"\"\"\n Check if the time since the last HTTP request is under the\n session timeout limit. If it's been too long since the last request\n attempt to authenticate again.\n \"\"\"\n now = time.time()\n diff = abs(now - self.last_request_time)\n timeout_sec = self.session_timeout * 60 # convert minutes to seconds\n\n if diff >= timeout_sec:\n self.__log('Session timed out, attempting to authenticate')\n self.authenticate()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_url(self, path):\n url = '{0}{1}'.format(self.base_url, path)\n url = re.sub('([^:])//', '\\\\1/', url) # Remove double slashes\n return url", "response": "Builds a LendingClub URL from a path."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef authenticate(self, email=None, password=None):\n\n # Get email and password\n if email is None:\n email = self.email\n else:\n self.email = email\n\n if password is None:\n password = self.__pass\n else:\n self.__pass = password\n\n # Get them from the user\n if email is None:\n email = raw_input('Email:')\n self.email = email\n if password is None:\n password = getpass.getpass()\n self.__pass = password\n\n self.__log('Attempting to authenticate: {0}'.format(self.email))\n\n # Start session\n self.__session = requests.Session()\n self.__session.headers = {\n 'Referer': 'https://www.lendingclub.com/',\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 Safari/537.31'\n }\n\n # Set last request time to now\n self.last_request_time = time.time()\n\n # Send login request to LC\n payload = {\n 'login_email': email,\n 'login_password': password\n }\n response = self.post('/account/login.action', data=payload, redirects=False)\n\n # Get URL redirect URL and save the last part of the path as the endpoint\n response_url = response.url\n if response.status_code == 302:\n response_url = response.headers['location']\n endpoint = response_url.split('/')[-1]\n\n # Debugging\n self.__log('Status code: {0}'.format(response.status_code))\n self.__log('Redirected to: {0}'.format(response_url))\n self.__log('Cookies: {0}'.format(str(response.cookies.keys())))\n\n # Show query and data that the server received\n if 'x-echo-query' in response.headers:\n self.__log('Query: {0}'.format(response.headers['x-echo-query']))\n if 'x-echo-data' in response.headers:\n self.__log('Data: {0}'.format(response.headers['x-echo-data']))\n\n # Parse any errors from the HTML\n soup = BeautifulSoup(response.text, \"html5lib\")\n errors = soup.find(id='master_error-list')\n if errors:\n errors = errors.text.strip()\n\n # Remove extra spaces and newlines from error message\n errors = re.sub('\\t+', '', errors)\n errors = re.sub('\\s*\\n+\\s*', ' * ', errors)\n\n if errors == '':\n errors = None\n\n # Raise error\n if errors is not None:\n raise AuthenticationError(errors)\n\n # Redirected back to the login page...must be an error\n if endpoint == 'login.action':\n raise AuthenticationError('Unknown! Redirected back to the login page without an error message')\n\n return True", "response": "Authenticate with LendingClub and preserve the user session for future requests."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_site_available(self):\n try:\n response = requests.head(self.base_url)\n status = response.status_code\n return 200 <= status < 400 # Returns true if the status code is greater than 200 and less than 400\n except Exception:\n return False", "response": "Returns True if we can access the site with the given ID."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef request(self, method, path, query=None, data=None, redirects=True):\n\n # Check session time\n self.__continue_session()\n\n try:\n url = self.build_url(path)\n method = method.upper()\n\n self.__log('{0} request to: {1}'.format(method, url))\n\n if method == 'POST':\n request = self.__session.post(url, params=query, data=data, allow_redirects=redirects)\n elif method == 'GET':\n request = self.__session.get(url, params=query, data=data, allow_redirects=redirects)\n elif method == 'HEAD':\n request = self.__session.head(url, params=query, data=data, allow_redirects=redirects)\n elif method == 'DELETE':\n request = self.__session.delete(url, params=query, data=data, allow_redirects=redirects)\n else:\n raise SessionError('{0} is not a supported HTTP method'.format(method))\n\n self.last_response = request\n\n self.__log('Status code: {0}'.format(request.status_code))\n\n # Update session time\n self.last_request_time = time.time()\n\n except (RequestException, ConnectionError, TooManyRedirects, HTTPError) as e:\n raise NetworkError('{0} failed to: {1}'.format(method, url), e)\n except Timeout:\n raise NetworkError('{0} request timed out: {1}'.format(method, url), e)\n\n return request", "response": "Send a HTTP request to the LendingClub."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrap for request function for POST requests.", "response": "def post(self, path, query=None, data=None, redirects=True):\n \"\"\"\n POST request wrapper for :func:`request()`\n \"\"\"\n return self.request('POST', path, query, data, redirects)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwraps for request method for GET requests.", "response": "def get(self, path, query=None, redirects=True):\n \"\"\"\n GET request wrapper for :func:`request()`\n \"\"\"\n return self.request('GET', path, query, None, redirects)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef head(self, path, query=None, data=None, redirects=True):\n return self.request('HEAD', path, query, None, redirects)", "response": "HEAD request wrapper for the request method."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef json_success(self, json):\n if type(json) is dict and 'result' in json and json['result'] == 'success':\n return True\n return False", "response": "Check the JSON response object for the success flag"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmerging dictionary objects recursively, by only updating keys existing in to_dict", "response": "def __merge_values(self, from_dict, to_dict):\n \"\"\"\n Merge dictionary objects recursively, by only updating keys existing in to_dict\n \"\"\"\n for key, value in from_dict.iteritems():\n\n # Only if the key already exists\n if key in to_dict:\n\n # Make sure the values are the same datatype\n assert type(to_dict[key]) is type(from_dict[key]), 'Data type for {0} is incorrect: {1}, should be {2}'.format(key, type(from_dict[key]), type(to_dict[key]))\n\n # Recursively dive into the next dictionary\n if type(to_dict[key]) is dict:\n to_dict[key] = self.__merge_values(from_dict[key], to_dict[key])\n\n # Replace value\n else:\n to_dict[key] = from_dict[key]\n\n return to_dict"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __normalize_grades(self):\n\n if 'grades' in self and self['grades']['All'] is True:\n for grade in self['grades']:\n if grade != 'All' and self['grades'][grade] is True:\n self['grades']['All'] = False\n break", "response": "Adjust the grades list."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadjust the funding progress filter to be a factor of 10", "response": "def __normalize_progress(self):\n \"\"\"\n Adjust the funding progress filter to be a factor of 10\n \"\"\"\n\n progress = self['funding_progress']\n if progress % 10 != 0:\n progress = round(float(progress) / 10)\n progress = int(progress) * 10\n\n self['funding_progress'] = progress"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __normalize(self):\n\n # Don't normalize if we're already normalizing or intializing\n if self.__normalizing is True or self.__initialized is False:\n return\n\n self.__normalizing = True\n self.__normalize_grades()\n self.__normalize_progress()\n self.__normalizing = False", "response": "Adjusts the values of the filters to be correct."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_one(self, loan):\n assert type(loan) is dict, 'loan parameter must be a dictionary object'\n\n # Map the loan value keys to the filter keys\n req = {\n 'loanGUID': 'loan_id',\n 'loanGrade': 'grade',\n 'loanLength': 'term',\n 'loanUnfundedAmount': 'progress',\n 'loanAmountRequested': 'progress',\n 'alreadyInvestedIn': 'exclude_existing',\n 'purpose': 'loan_purpose',\n }\n\n # Throw an error if the loan does not contain one of the criteria keys that this filter has\n for key, criteria in req.iteritems():\n if criteria in self and key not in loan:\n raise FilterValidationError('Loan does not have a \"{0}\" value.'.format(key), loan, criteria)\n\n # Loan ID\n if 'loan_id' in self:\n loan_ids = str(self['loan_id']).split(',')\n if str(loan['loanGUID']) not in loan_ids:\n raise FilterValidationError('Did not meet filter criteria for loan ID. {0} does not match {1}'.format(loan['loanGUID'], self['loan_id']), loan=loan, criteria='loan ID')\n\n # Grade\n grade = loan['loanGrade'][0] # Extract the letter portion of the loan\n if 'grades' in self and self['grades']['All'] is not True:\n if grade not in self['grades']:\n raise FilterValidationError('Loan grade \"{0}\" is unknown'.format(grade), loan, 'grade')\n elif self['grades'][grade] is False:\n raise FilterValidationError(loan=loan, criteria='grade')\n\n # Term\n if 'term' in self and self['term'] is not None:\n if loan['loanLength'] == 36 and self['term']['Year3'] is False:\n raise FilterValidationError(loan=loan, criteria='loan term')\n elif loan['loanLength'] == 60 and self['term']['Year5'] is False:\n raise FilterValidationError(loan=loan, criteria='loan term')\n\n # Progress\n if 'funding_progress' in self:\n loan_progress = (1 - (loan['loanUnfundedAmount'] / loan['loanAmountRequested'])) * 100\n if self['funding_progress'] > loan_progress:\n raise FilterValidationError(loan=loan, criteria='funding progress')\n\n # Exclude existing\n if 'exclude_existing' in self:\n if self['exclude_existing'] is True and loan['alreadyInvestedIn'] is True:\n raise FilterValidationError(loan=loan, criteria='exclude loans you are invested in')\n\n # Loan purpose (either an array or single value)\n if 'loan_purpose' in self and loan['purpose'] is not False:\n purpose = self['loan_purpose']\n if type(purpose) is not dict:\n purpose = {purpose: True}\n\n if 'All' not in purpose or purpose['All'] is False:\n if loan['purpose'] not in purpose:\n raise FilterValidationError(loan=loan, criteria='loan purpose')\n\n return True", "response": "Validate a single loan record against the filters."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the JSON string that LendingClub expects for it s search", "response": "def search_string(self):\n \"\"\"\"\n Returns the JSON string that LendingClub expects for it's search\n \"\"\"\n self.__normalize()\n\n # Get the template\n tmpl_source = unicode(open(self.tmpl_file).read())\n\n # Process template\n compiler = Compiler()\n template = compiler.compile(tmpl_source)\n out = template(self)\n if not out:\n return False\n out = ''.join(out)\n\n #\n # Cleanup output and remove all extra space\n #\n\n # remove extra spaces\n out = re.sub('\\n', '', out)\n out = re.sub('\\s{3,}', ' ', out)\n\n # Remove hanging commas i.e: [1, 2,]\n out = re.sub(',\\s*([}\\\\]])', '\\\\1', out)\n\n # Space between brackets i.e: ], [\n out = re.sub('([{\\\\[}\\\\]])(,?)\\s*([{\\\\[}\\\\]])', '\\\\1\\\\2\\\\3', out)\n\n # Cleanup spaces around [, {, }, ], : and , characters\n out = re.sub('\\s*([{\\\\[\\\\]}:,])\\s*', '\\\\1', out)\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef all_filters(lc):\n\n filters = []\n response = lc.session.get('/browse/getSavedFiltersAj.action')\n json_response = response.json()\n\n # Load all filters\n if lc.session.json_success(json_response):\n for saved in json_response['filters']:\n filters.append(SavedFilter(lc, saved['id']))\n\n return filters", "response": "Get a list of all your saved filters"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load(self):\n\n # Attempt to load the saved filter\n payload = {\n 'id': self.id\n }\n response = self.lc.session.get('/browse/getSavedFilterAj.action', query=payload)\n self.response = response\n json_response = response.json()\n\n if self.lc.session.json_success(json_response) and json_response['filterName'] != 'No filters':\n self.name = json_response['filterName']\n\n #\n # Parse out the filter JSON string manually from the response JSON.\n # If the filter JSON is modified at all, or any value is out of order,\n # LendingClub will reject the filter and perform a wildcard search instead,\n # without any error. So we need to retain the filter JSON value exactly how it is given to us.\n #\n text = response.text\n\n # Cut off everything before \"filter\": [...]\n text = re.sub('\\n', '', text)\n text = re.sub('^.*?,\\s*[\"\\']filter[\"\\']:\\s*\\[(.*)', '[\\\\1', text)\n\n # Now loop through the string until we find the end of the filter block\n # This is a simple parser that keeps track of block elements, quotes and\n # escape characters\n blockTracker = []\n blockChars = {\n '[': ']',\n '{': '}'\n }\n inQuote = False\n lastChar = None\n json_text = \"\"\n for char in text:\n json_text += char\n\n # Escape char\n if char == '\\\\':\n if lastChar == '\\\\':\n lastChar = ''\n else:\n lastChar = char\n continue\n\n # Quotes\n if char == \"'\" or char == '\"':\n if inQuote is False: # Starting a quote block\n inQuote = char\n elif inQuote == char: # Ending a quote block\n inQuote = False\n lastChar = char\n continue\n\n # Start of a block\n if char in blockChars.keys():\n blockTracker.insert(0, blockChars[char])\n\n # End of a block, remove from block path\n elif len(blockTracker) > 0 and char == blockTracker[0]:\n blockTracker.pop(0)\n\n # No more blocks in the tracker which means we're at the end of the filter block\n if len(blockTracker) == 0 and lastChar is not None:\n break\n\n lastChar = char\n\n # Verify valid JSON\n try:\n if json_text.strip() == '':\n raise SavedFilterError('A saved filter could not be found for ID {0}'.format(self.id), response)\n\n json_test = json.loads(json_text)\n\n # Make sure it looks right\n assert type(json_test) is list, 'Expecting a list, instead received a {0}'.format(type(json_test))\n assert 'm_id' in json_test[0], 'Expecting a \\'m_id\\' property in each filter'\n assert 'm_value' in json_test[0], 'Expecting a \\'m_value\\' property in each filter'\n\n self.json = json_test\n except Exception as e:\n raise SavedFilterError('Could not parse filter from the JSON response: {0}'.format(str(e)))\n\n self.json_text = json_text\n self.__analyze()\n\n else:\n raise SavedFilterError('A saved filter could not be found for ID {0}'.format(self.id), response)", "response": "Load the filter from the server and store it in the self. name attribute."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __analyze(self):\n filter_values = {}\n\n # ID to filter name mapping\n name_map = {\n 10: 'grades',\n 11: 'loan_purpose',\n 13: 'approved',\n 15: 'funding_progress',\n 38: 'exclude_existing',\n 39: 'term',\n 43: 'keyword'\n }\n\n if self.json is not None:\n filters = self.json\n\n for f in filters:\n if 'm_id' in f:\n name = f['m_id']\n\n # Get the name to represent this filter\n if f['m_id'] in name_map:\n name = name_map[f['m_id']]\n\n # Get values\n if 'm_value' in f:\n raw_values = f['m_value']\n value = {}\n\n # No value, skip it\n if raw_values is None:\n continue\n\n # Loop through multiple values\n if type(raw_values) is list:\n\n # A single non string value, is THE value\n if len(raw_values) == 1 and type(raw_values[0]['value']) not in [str, unicode]:\n value = raw_values[0]['value']\n\n # Create a dict of values: name = True\n for val in raw_values:\n if type(val['value']) in [str, unicode]:\n value[val['value']] = True\n\n # A single value\n else:\n value = raw_values\n\n # Normalize grades array\n if name == 'grades':\n if 'All' not in value:\n value['All'] = False\n\n # Add filter value\n filter_values[name] = value\n dict.__setitem__(self, name, value)\n\n return filter_values", "response": "Analyze the filter JSON and attempt to parse out the individual filters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _float_copy_to_out(out, origin):\n if out is None:\n out = origin / 1 # The division forces cast to a floating point type\n elif out is not origin:\n np.copyto(out, origin)\n return out", "response": "Copy origin to out and return it."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _u_centered_imp(a, out=None):\n out = _float_copy_to_out(out, a)\n\n dim = np.size(a, 0)\n\n u_mu = np.sum(a) / ((dim - 1) * (dim - 2))\n sum_cols = np.sum(a, 0, keepdims=True)\n sum_rows = np.sum(a, 1, keepdims=True)\n u_mu_cols = np.ones((dim, 1)).dot(sum_cols / (dim - 2))\n u_mu_rows = (sum_rows / (dim - 2)).dot(np.ones((1, dim)))\n\n # Do one operation at a time, to improve broadcasting memory usage.\n out -= u_mu_rows\n out -= u_mu_cols\n out += u_mu\n\n # The diagonal is zero\n out[np.eye(dim, dtype=bool)] = 0\n\n return out", "response": "Real implementation of u_centered."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef u_product(a, b):\n n = np.size(a, 0)\n\n return np.sum(a * b) / (n * (n - 3))", "response": "Returns the inner product of two arrays a and b."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef u_complementary_projection(a):\n proj = u_projection(a)\n\n def projection(a):\n \"\"\"\n Orthogonal projection over the complementary space.\n\n This function was returned by :code:`u_complementary_projection`.\n The complete usage information is in the documentation of\n :code:`u_complementary_projection`.\n\n See Also\n --------\n u_complementary_projection\n\n \"\"\"\n return a - proj(a)\n\n return projection", "response": "r Returns the orthogonal projection function over a matrix."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _distance_matrix_generic(x, centering, exponent=1):\n _check_valid_dcov_exponent(exponent)\n\n x = _transform_to_2d(x)\n\n # Calculate distance matrices\n a = distances.pairwise_distances(x, exponent=exponent)\n\n # Double centering\n a = centering(a, out=a)\n\n return a", "response": "Compute a centered distance matrix given a matrix."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nscale a random vector for using the affinely invariant measures", "response": "def _af_inv_scaled(x):\n \"\"\"Scale a random vector for using the affinely invariant measures\"\"\"\n x = _transform_to_2d(x)\n\n cov_matrix = np.atleast_2d(np.cov(x, rowvar=False))\n\n cov_matrix_power = _mat_sqrt_inv(cov_matrix)\n\n return x.dot(cov_matrix_power)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the partial distance covariance estimator for the given random vectors x y and z.", "response": "def partial_distance_covariance(x, y, z):\n \"\"\"\n Partial distance covariance estimator.\n\n Compute the estimator for the partial distance covariance of the\n random vectors corresponding to :math:`x` and :math:`y` with respect\n to the random variable corresponding to :math:`z`.\n\n Parameters\n ----------\n x: array_like\n First random vector. The columns correspond with the individual random\n variables while the rows are individual instances of the random vector.\n y: array_like\n Second random vector. The columns correspond with the individual random\n variables while the rows are individual instances of the random vector.\n z: array_like\n Random vector with respect to which the partial distance covariance\n is computed. The columns correspond with the individual random\n variables while the rows are individual instances of the random vector.\n\n Returns\n -------\n numpy scalar\n Value of the estimator of the partial distance covariance.\n\n See Also\n --------\n partial_distance_correlation\n\n Examples\n --------\n >>> import numpy as np\n >>> import dcor\n >>> a = np.array([[1, 2, 3, 4],\n ... [5, 6, 7, 8],\n ... [9, 10, 11, 12],\n ... [13, 14, 15, 16]])\n >>> b = np.array([[1], [0], [0], [1]])\n >>> c = np.array([[1, 3, 4],\n ... [5, 7, 8],\n ... [9, 11, 15],\n ... [13, 15, 16]])\n >>> dcor.partial_distance_covariance(a, a, c) # doctest: +ELLIPSIS\n 0.0024298...\n >>> dcor.partial_distance_covariance(a, b, c)\n 0.0347030...\n >>> dcor.partial_distance_covariance(b, b, c)\n 0.4956241...\n\n \"\"\"\n a = _u_distance_matrix(x)\n b = _u_distance_matrix(y)\n c = _u_distance_matrix(z)\n\n proj = u_complementary_projection(c)\n\n return u_product(proj(a), proj(b))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef partial_distance_correlation(x, y, z): # pylint:disable=too-many-locals\n a = _u_distance_matrix(x)\n b = _u_distance_matrix(y)\n c = _u_distance_matrix(z)\n\n aa = u_product(a, a)\n bb = u_product(b, b)\n cc = u_product(c, c)\n ab = u_product(a, b)\n ac = u_product(a, c)\n bc = u_product(b, c)\n\n denom_sqr = aa * bb\n r_xy = ab / _sqrt(denom_sqr) if denom_sqr != 0 else denom_sqr\n r_xy = np.clip(r_xy, -1, 1)\n\n denom_sqr = aa * cc\n r_xz = ac / _sqrt(denom_sqr) if denom_sqr != 0 else denom_sqr\n r_xz = np.clip(r_xz, -1, 1)\n\n denom_sqr = bb * cc\n r_yz = bc / _sqrt(denom_sqr) if denom_sqr != 0 else denom_sqr\n r_yz = np.clip(r_yz, -1, 1)\n\n denom = _sqrt(1 - r_xz ** 2) * _sqrt(1 - r_yz ** 2)\n\n return (r_xy - r_xz * r_yz) / denom if denom != 0 else denom", "response": "Partial distance correlation estimator."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _energy_distance_from_distance_matrices(\n distance_xx, distance_yy, distance_xy):\n \"\"\"Compute energy distance with precalculated distance matrices.\"\"\"\n return (2 * np.mean(distance_xy) - np.mean(distance_xx) -\n np.mean(distance_yy))", "response": "Compute energy distance with precalculated distance matrices."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes generic squared stats.", "response": "def _distance_sqr_stats_naive_generic(x, y, matrix_centered, product,\n exponent=1):\n \"\"\"Compute generic squared stats.\"\"\"\n a = matrix_centered(x, exponent=exponent)\n b = matrix_centered(y, exponent=exponent)\n\n covariance_xy_sqr = product(a, b)\n variance_x_sqr = product(a, a)\n variance_y_sqr = product(b, b)\n\n denominator_sqr = np.absolute(variance_x_sqr * variance_y_sqr)\n denominator = _sqrt(denominator_sqr)\n\n # Comparisons using a tolerance can change results if the\n # covariance has a similar order of magnitude\n if denominator == 0.0:\n correlation_xy_sqr = 0.0\n else:\n correlation_xy_sqr = covariance_xy_sqr / denominator\n\n return Stats(covariance_xy=covariance_xy_sqr,\n correlation_xy=correlation_xy_sqr,\n variance_x=variance_x_sqr,\n variance_y=variance_y_sqr)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _distance_correlation_sqr_naive(x, y, exponent=1):\n return _distance_sqr_stats_naive_generic(\n x, y,\n matrix_centered=_distance_matrix,\n product=mean_product,\n exponent=exponent).correlation_xy", "response": "Biased distance correlation estimator between two matrices."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _u_distance_correlation_sqr_naive(x, y, exponent=1):\n return _distance_sqr_stats_naive_generic(\n x, y,\n matrix_centered=_u_distance_matrix,\n product=u_product,\n exponent=exponent).correlation_xy", "response": "Bias - correcteded distance correlation estimator between two matrices."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _can_use_fast_algorithm(x, y, exponent=1):\n return (_is_random_variable(x) and _is_random_variable(y) and\n x.shape[0] > 3 and y.shape[0] > 3 and exponent == 1)", "response": "Check if the fast algorithm for distance stats can be used."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _distance_covariance_sqr_fast_generic(\n x, y, unbiased=False): # pylint:disable=too-many-locals\n # This function has many locals so it can be compared\n # with the original algorithm.\n \"\"\"Fast algorithm for the squared distance covariance.\"\"\"\n x = np.asarray(x)\n y = np.asarray(y)\n\n x = np.ravel(x)\n y = np.ravel(y)\n\n n = x.shape[0]\n assert n > 3\n assert n == y.shape[0]\n temp = range(n)\n\n # Step 1\n ix0 = np.argsort(x)\n vx = x[ix0]\n\n ix = np.zeros(n, dtype=int)\n ix[ix0] = temp\n\n iy0 = np.argsort(y)\n vy = y[iy0]\n\n iy = np.zeros(n, dtype=int)\n iy[iy0] = temp\n\n # Step 2\n sx = np.cumsum(vx)\n sy = np.cumsum(vy)\n\n # Step 3\n alpha_x = ix\n alpha_y = iy\n\n beta_x = sx[ix] - vx[ix]\n beta_y = sy[iy] - vy[iy]\n\n # Step 4\n x_dot = np.sum(x)\n y_dot = np.sum(y)\n\n # Step 5\n a_i_dot = x_dot + (2 * alpha_x - n) * x - 2 * beta_x\n b_i_dot = y_dot + (2 * alpha_y - n) * y - 2 * beta_y\n\n sum_ab = np.sum(a_i_dot * b_i_dot)\n\n # Step 6\n a_dot_dot = 2 * np.sum(alpha_x * x) - 2 * np.sum(beta_x)\n b_dot_dot = 2 * np.sum(alpha_y * y) - 2 * np.sum(beta_y)\n\n # Step 7\n gamma_1 = _partial_sum_2d(x, y, np.ones(n, dtype=x.dtype))\n gamma_x = _partial_sum_2d(x, y, x)\n gamma_y = _partial_sum_2d(x, y, y)\n gamma_xy = _partial_sum_2d(x, y, x * y)\n\n # Step 8\n aijbij = np.sum(x * y * gamma_1 + gamma_xy - x * gamma_y - y * gamma_x)\n\n if unbiased:\n d3 = (n - 3)\n d2 = (n - 2)\n d1 = (n - 1)\n else:\n d3 = d2 = d1 = n\n\n # Step 9\n d_cov = (aijbij / n / d3 - 2 * sum_ab / n / d2 / d3 +\n a_dot_dot / n * b_dot_dot / d1 / d2 / d3)\n\n return d_cov", "response": "Fast algorithm for the squared distance covariance."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _distance_stats_sqr_fast_generic(x, y, dcov_function):\n covariance_xy_sqr = dcov_function(x, y)\n variance_x_sqr = dcov_function(x, x)\n variance_y_sqr = dcov_function(y, y)\n denominator_sqr_signed = variance_x_sqr * variance_y_sqr\n denominator_sqr = np.absolute(denominator_sqr_signed)\n denominator = _sqrt(denominator_sqr)\n\n # Comparisons using a tolerance can change results if the\n # covariance has a similar order of magnitude\n if denominator == 0.0:\n correlation_xy_sqr = denominator.dtype.type(0)\n else:\n correlation_xy_sqr = covariance_xy_sqr / denominator\n\n return Stats(covariance_xy=covariance_xy_sqr,\n correlation_xy=correlation_xy_sqr,\n variance_x=variance_x_sqr,\n variance_y=variance_y_sqr)", "response": "Compute the distance stats using the fast algorithm."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef distance_covariance_sqr(x, y, **kwargs):\n if _can_use_fast_algorithm(x, y, **kwargs):\n return _distance_covariance_sqr_fast(x, y)\n else:\n return _distance_covariance_sqr_naive(x, y, **kwargs)", "response": "Calculates the squared distance covariance between two random vectors."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef distance_stats_sqr(x, y, **kwargs):\n if _can_use_fast_algorithm(x, y, **kwargs):\n return _distance_stats_sqr_fast(x, y)\n else:\n return _distance_sqr_stats_naive_generic(\n x, y,\n matrix_centered=_distance_matrix,\n product=mean_product,\n **kwargs)", "response": "This function computes the squared distance covariance and squared distance correlation between two random vectors and the individual squared distance variances."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef u_distance_stats_sqr(x, y, **kwargs):\n if _can_use_fast_algorithm(x, y, **kwargs):\n return _u_distance_stats_sqr_fast(x, y)\n else:\n return _distance_sqr_stats_naive_generic(\n x, y,\n matrix_centered=_u_distance_matrix,\n product=u_product,\n **kwargs)", "response": "This function computes the unbiased un"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef distance_stats(x, y, **kwargs):\n return Stats(*[_sqrt(s) for s in distance_stats_sqr(x, y, **kwargs)])", "response": "This function computes the distance covariance and distance correlation between two random vectors and the individual distance variances."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef u_distance_correlation_sqr(x, y, **kwargs):\n if _can_use_fast_algorithm(x, y, **kwargs):\n return _u_distance_correlation_sqr_fast(x, y)\n else:\n return _u_distance_correlation_sqr_naive(x, y, **kwargs)", "response": "Returns the squared distance correlation between two random vectors."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef distance_correlation_af_inv_sqr(x, y):\n x = _af_inv_scaled(x)\n y = _af_inv_scaled(y)\n\n correlation = distance_correlation_sqr(x, y)\n return 0 if np.isnan(correlation) else correlation", "response": "Calculates the squared affinely invariant distance correlation between two random vectors."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pairwise(function, x, y=None, **kwargs):\n\n return _pairwise_imp(function, x, y, **kwargs)", "response": "A function that computes the dependency measure between each pair of elements."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _jit(function):\n import sys\n\n compiled = numba.jit(function)\n\n if hasattr(sys, '_called_from_test'):\n return function\n else: # pragma: no cover\n return compiled", "response": "Compile a function using a jit compiler."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns square root of an ndarray. This sqrt function for ndarrays tries to use the exponentiation operator if the objects stored do not supply a sqrt method.", "response": "def _sqrt(x):\n \"\"\"\n Return square root of an ndarray.\n\n This sqrt function for ndarrays tries to use the exponentiation operator\n if the objects stored do not supply a sqrt method.\n\n \"\"\"\n x = np.clip(x, a_min=0, a_max=None)\n\n try:\n return np.sqrt(x)\n except AttributeError:\n exponent = 0.5\n\n try:\n exponent = np.take(x, 0).from_float(exponent)\n except AttributeError:\n pass\n\n return x ** exponent"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _transform_to_2d(t):\n t = np.asarray(t)\n\n dim = len(t.shape)\n assert dim <= 2\n\n if dim < 2:\n t = np.atleast_2d(t).T\n\n return t", "response": "Convert vectors to column matrices to always have a 2d shape."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _can_be_double(x):\n return ((np.issubdtype(x.dtype, np.floating) and\n x.dtype.itemsize <= np.dtype(float).itemsize) or\n (np.issubdtype(x.dtype, np.signedinteger) and\n np.can_cast(x, float)))", "response": "Return True if the array can be safely converted to double."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cdist_scipy(x, y, exponent=1):\n metric = 'euclidean'\n\n if exponent != 1:\n metric = 'sqeuclidean'\n\n distances = _spatial.distance.cdist(x, y, metric=metric)\n\n if exponent != 1:\n distances **= exponent / 2\n\n return distances", "response": "Pairwise distance between the points in two sets."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _pdist(x, exponent=1):\n if _can_be_double(x):\n return _pdist_scipy(x, exponent)\n else:\n return _cdist_naive(x, x, exponent)", "response": "Pairwise distance between points in a set."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _cdist(x, y, exponent=1):\n if _can_be_double(x) and _can_be_double(y):\n return _cdist_scipy(x, y, exponent)\n else:\n return _cdist_naive(x, y, exponent)", "response": "Pairwise distance between points in two sets."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef respond(self,\n content=EmptyValue,\n content_type=EmptyValue,\n always_hash_content=True,\n ext=None):\n \"\"\"\n Respond to the request.\n\n This generates the :attr:`mohawk.Receiver.response_header`\n attribute.\n\n :param content=EmptyValue: Byte string of response body that will be sent.\n :type content=EmptyValue: str\n\n :param content_type=EmptyValue: content-type header value for response.\n :type content_type=EmptyValue: str\n\n :param always_hash_content=True:\n When True, ``content`` and ``content_type`` must be provided.\n Read :ref:`skipping-content-checks` to learn more.\n :type always_hash_content=True: bool\n\n :param ext=None:\n An external `Hawk`_ string. If not None, this value will be\n signed so that the sender can trust it.\n :type ext=None: str\n\n .. _`Hawk`: https://github.com/hueniverse/hawk\n \"\"\"\n\n log.debug('generating response header')\n\n resource = Resource(url=self.resource.url,\n credentials=self.resource.credentials,\n ext=ext,\n app=self.parsed_header.get('app', None),\n dlg=self.parsed_header.get('dlg', None),\n method=self.resource.method,\n content=content,\n content_type=content_type,\n always_hash_content=always_hash_content,\n nonce=self.parsed_header['nonce'],\n timestamp=self.parsed_header['ts'])\n\n mac = calculate_mac('response', resource, resource.gen_content_hash())\n\n self.response_header = self._make_header(resource, mac,\n additional_keys=['ext'])\n return self.response_header", "response": "This method generates the response header for the current resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef calculate_payload_hash(payload, algorithm, content_type):\n p_hash = hashlib.new(algorithm)\n\n parts = []\n parts.append('hawk.' + str(HAWK_VER) + '.payload\\n')\n parts.append(parse_content_type(content_type) + '\\n')\n parts.append(payload or '')\n parts.append('\\n')\n\n for i, p in enumerate(parts):\n # Make sure we are about to hash binary strings.\n if not isinstance(p, six.binary_type):\n p = p.encode('utf8')\n p_hash.update(p)\n parts[i] = p\n\n log.debug('calculating payload hash from:\\n{parts}'\n .format(parts=pprint.pformat(parts)))\n\n return b64encode(p_hash.digest())", "response": "Calculates a hash for a given payload."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating a message authorization code ( MAC ).", "response": "def calculate_mac(mac_type, resource, content_hash):\n \"\"\"Calculates a message authorization code (MAC).\"\"\"\n normalized = normalize_string(mac_type, resource, content_hash)\n log.debug(u'normalized resource for mac calc: {norm}'\n .format(norm=normalized))\n digestmod = getattr(hashlib, resource.credentials['algorithm'])\n\n # Make sure we are about to hash binary strings.\n\n if not isinstance(normalized, six.binary_type):\n normalized = normalized.encode('utf8')\n key = resource.credentials['key']\n if not isinstance(key, six.binary_type):\n key = key.encode('ascii')\n\n result = hmac.new(key, normalized, digestmod)\n return b64encode(result.digest())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate a message authorization code for a timestamp.", "response": "def calculate_ts_mac(ts, credentials):\n \"\"\"Calculates a message authorization code (MAC) for a timestamp.\"\"\"\n normalized = ('hawk.{hawk_ver}.ts\\n{ts}\\n'\n .format(hawk_ver=HAWK_VER, ts=ts))\n log.debug(u'normalized resource for ts mac calc: {norm}'\n .format(norm=normalized))\n digestmod = getattr(hashlib, credentials['algorithm'])\n\n if not isinstance(normalized, six.binary_type):\n normalized = normalized.encode('utf8')\n key = credentials['key']\n if not isinstance(key, six.binary_type):\n key = key.encode('ascii')\n\n result = hmac.new(key, normalized, digestmod)\n return b64encode(result.digest())"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nserializes mac_type and resource into a HAWK string.", "response": "def normalize_string(mac_type, resource, content_hash):\n \"\"\"Serializes mac_type and resource into a HAWK string.\"\"\"\n\n normalized = [\n 'hawk.' + str(HAWK_VER) + '.' + mac_type,\n normalize_header_attr(resource.timestamp),\n normalize_header_attr(resource.nonce),\n normalize_header_attr(resource.method or ''),\n normalize_header_attr(resource.name or ''),\n normalize_header_attr(resource.host),\n normalize_header_attr(resource.port),\n normalize_header_attr(content_hash or '')\n ]\n\n # The blank lines are important. They follow what the Node Hawk lib does.\n\n normalized.append(normalize_header_attr(resource.ext or ''))\n\n if resource.app:\n normalized.append(normalize_header_attr(resource.app))\n normalized.append(normalize_header_attr(resource.dlg or ''))\n\n # Add trailing new line.\n normalized.append('')\n\n normalized = '\\n'.join(normalized)\n\n return normalized"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse an authorization header into a single object.", "response": "def parse_authorization_header(auth_header):\n \"\"\"\n Example Authorization header:\n\n 'Hawk id=\"dh37fgj492je\", ts=\"1367076201\", nonce=\"NPHgnG\", ext=\"and\n welcome!\", mac=\"CeWHy4d9kbLGhDlkyw2Nh3PJ7SDOdZDa267KH4ZaNMY=\"'\n \"\"\"\n if len(auth_header) > MAX_LENGTH:\n raise BadHeaderValue('Header exceeds maximum length of {max_length}'.format(\n max_length=MAX_LENGTH))\n\n # Make sure we have a unicode object for consistency.\n if isinstance(auth_header, six.binary_type):\n auth_header = auth_header.decode('utf8')\n\n scheme, attributes_string = auth_header.split(' ', 1)\n\n if scheme.lower() != 'hawk':\n raise HawkFail(\"Unknown scheme '{scheme}' when parsing header\"\n .format(scheme=scheme))\n\n\n attributes = {}\n\n def replace_attribute(match):\n \"\"\"Extract the next key=\"value\"-pair in the header.\"\"\"\n key = match.group('key')\n value = match.group('value')\n if key not in allowable_header_keys:\n raise HawkFail(\"Unknown Hawk key '{key}' when parsing header\"\n .format(key=key))\n validate_header_attr(value, name=key)\n if key in attributes:\n raise BadHeaderValue('Duplicate key in header: {key}'.format(key=key))\n attributes[key] = value\n\n # Iterate over all the key=\"value\"-pairs in the header, replace them with\n # an empty string, and store the extracted attribute in the attributes\n # dict. Correctly formed headers will then leave nothing unparsed ('').\n unparsed_header = HAWK_HEADER_RE.sub(replace_attribute, attributes_string)\n if unparsed_header != '':\n raise BadHeaderValue(\"Couldn't parse Hawk header\", unparsed_header)\n\n log.debug('parsed Hawk header: {header} into: \\n{parsed}'\n .format(header=auth_header, parsed=pprint.pformat(attributes)))\n return attributes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a bewit identifier for the resource as a string.", "response": "def get_bewit(resource):\n \"\"\"\n Returns a bewit identifier for the resource as a string.\n\n :param resource:\n Resource to generate a bewit for\n :type resource: `mohawk.base.Resource`\n \"\"\"\n if resource.method != 'GET':\n raise ValueError('bewits can only be generated for GET requests')\n if resource.nonce != '':\n raise ValueError('bewits must use an empty nonce')\n mac = calculate_mac(\n 'bewit',\n resource,\n None,\n )\n\n if isinstance(mac, six.binary_type):\n mac = mac.decode('ascii')\n\n if resource.ext is None:\n ext = ''\n else:\n validate_header_attr(resource.ext, name='ext')\n ext = resource.ext\n\n # b64encode works only with bytes in python3, but all of our parameters are\n # in unicode, so we need to encode them. The cleanest way to do this that\n # works in both python 2 and 3 is to use string formatting to get a\n # unicode string, and then explicitly encode it to bytes.\n inner_bewit = u\"{id}\\\\{exp}\\\\{mac}\\\\{ext}\".format(\n id=resource.credentials['id'],\n exp=resource.timestamp,\n mac=mac,\n ext=ext,\n )\n inner_bewit_bytes = inner_bewit.encode('ascii')\n bewit_bytes = urlsafe_b64encode(inner_bewit_bytes)\n # Now decode the resulting bytes back to a unicode string\n return bewit_bytes.decode('ascii')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing a base64 encoded bewit string into a bewittuple.", "response": "def parse_bewit(bewit):\n \"\"\"\n Returns a `bewittuple` representing the parts of an encoded bewit string.\n This has the following named attributes:\n (id, expiration, mac, ext)\n\n :param bewit:\n A base64 encoded bewit string\n :type bewit: str\n \"\"\"\n decoded_bewit = b64decode(bewit).decode('ascii')\n bewit_parts = decoded_bewit.split(\"\\\\\")\n if len(bewit_parts) != 4:\n raise InvalidBewit('Expected 4 parts to bewit: %s' % decoded_bewit)\n return bewittuple(*bewit_parts)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef strip_bewit(url):\n m = re.search('[?&]bewit=([^&]+)', url)\n if not m:\n raise InvalidBewit('no bewit data found')\n bewit = m.group(1)\n stripped_url = url[:m.start()] + url[m.end():]\n return bewit, stripped_url", "response": "Strips the bewit parameter out of a url."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if a given bewit is valid and raises a subclass of HawkFail otherwise.", "response": "def check_bewit(url, credential_lookup, now=None):\n \"\"\"\n Validates the given bewit.\n\n Returns True if the resource has a valid bewit parameter attached,\n or raises a subclass of HawkFail otherwise.\n\n :param credential_lookup:\n Callable to look up the credentials dict by sender ID.\n The credentials dict must have the keys:\n ``id``, ``key``, and ``algorithm``.\n See :ref:`receiving-request` for an example.\n :type credential_lookup: callable\n\n :param now=None:\n Unix epoch time for the current time to determine if bewit has expired.\n If None, then the current time as given by utc_now() is used.\n :type now=None: integer\n \"\"\"\n raw_bewit, stripped_url = strip_bewit(url)\n bewit = parse_bewit(raw_bewit)\n try:\n credentials = credential_lookup(bewit.id)\n except LookupError:\n raise CredentialsLookupError('Could not find credentials for ID {0}'\n .format(bewit.id))\n\n res = Resource(url=stripped_url,\n method='GET',\n credentials=credentials,\n timestamp=bewit.expiration,\n nonce='',\n ext=bewit.ext,\n )\n mac = calculate_mac('bewit', res, None)\n mac = mac.decode('ascii')\n\n if not strings_match(mac, bewit.mac):\n raise MacMismatch('bewit with mac {bewit_mac} did not match expected mac {expected_mac}'\n .format(bewit_mac=bewit.mac,\n expected_mac=mac))\n\n # Check that the timestamp isn't expired\n if now is None:\n # TODO: Add offset/skew\n now = utc_now()\n if int(bewit.expiration) < now:\n # TODO: Refactor TokenExpired to handle this better\n raise TokenExpired('bewit with UTC timestamp {ts} has expired; '\n 'it was compared to {now}'\n .format(ts=bewit.expiration, now=now),\n localtime_in_seconds=now,\n www_authenticate=''\n )\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\naccepting a response to the Hawk server.", "response": "def accept_response(self,\n response_header,\n content=EmptyValue,\n content_type=EmptyValue,\n accept_untrusted_content=False,\n localtime_offset_in_seconds=0,\n timestamp_skew_in_seconds=default_ts_skew_in_seconds,\n **auth_kw):\n \"\"\"\n Accept a response to this request.\n\n :param response_header:\n A `Hawk`_ ``Server-Authorization`` header\n such as one created by :class:`mohawk.Receiver`.\n :type response_header: str\n\n :param content=EmptyValue: Byte string of the response body received.\n :type content=EmptyValue: str\n\n :param content_type=EmptyValue:\n Content-Type header value of the response received.\n :type content_type=EmptyValue: str\n\n :param accept_untrusted_content=False:\n When True, allow responses that do not hash their content.\n Read :ref:`skipping-content-checks` to learn more.\n :type accept_untrusted_content=False: bool\n\n :param localtime_offset_in_seconds=0:\n Seconds to add to local time in case it's out of sync.\n :type localtime_offset_in_seconds=0: float\n\n :param timestamp_skew_in_seconds=60:\n Max seconds until a message expires. Upon expiry,\n :class:`mohawk.exc.TokenExpired` is raised.\n :type timestamp_skew_in_seconds=60: float\n\n .. _`Hawk`: https://github.com/hueniverse/hawk\n \"\"\"\n log.debug('accepting response {header}'\n .format(header=response_header))\n\n parsed_header = parse_authorization_header(response_header)\n\n resource = Resource(ext=parsed_header.get('ext', None),\n content=content,\n content_type=content_type,\n # The following response attributes are\n # in reference to the original request,\n # not to the reponse header:\n timestamp=self.req_resource.timestamp,\n nonce=self.req_resource.nonce,\n url=self.req_resource.url,\n method=self.req_resource.method,\n app=self.req_resource.app,\n dlg=self.req_resource.dlg,\n credentials=self.credentials,\n seen_nonce=self.seen_nonce)\n\n self._authorize(\n 'response', parsed_header, resource,\n # Per Node lib, a responder macs the *sender's* timestamp.\n # It does not create its own timestamp.\n # I suppose a slow response could time out here. Maybe only check\n # mac failures, not timeouts?\n their_timestamp=resource.timestamp,\n timestamp_skew_in_seconds=timestamp_skew_in_seconds,\n localtime_offset_in_seconds=localtime_offset_in_seconds,\n accept_untrusted_content=accept_untrusted_content,\n **auth_kw)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dict of the current state of the instance.", "response": "def current_state(self):\n \"\"\"\n Returns a ``field -> value`` dict of the current state of the instance.\n \"\"\"\n field_names = set()\n [field_names.add(f.name) for f in self._meta.local_fields]\n [field_names.add(f.attname) for f in self._meta.local_fields]\n return dict([(field_name, getattr(self, field_name)) for field_name in field_names])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if the user was persisted in its old state.", "response": "def was_persisted(self):\n \"\"\"\n Returns true if the instance was persisted (saved) in its old\n state.\n\n Examples::\n\n >>> user = User()\n >>> user.save()\n >>> user.was_persisted()\n False\n\n >>> user = User.objects.get(pk=1)\n >>> user.delete()\n >>> user.was_persisted()\n True\n \"\"\"\n pk_name = self._meta.pk.name\n return bool(self.old_state()[pk_name])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _trim(cls, s):\n reverse = s[::-1]\n idx = 0\n for i in range(0, len(reverse)):\n if reverse[i] == \":\":\n idx += 1\n else:\n break\n\n # Return the substring after all trailing colons,\n # reversed back to its original character order.\n new_s = reverse[idx: len(reverse)]\n return new_s[::-1]", "response": "Removes trailing colons from the URI back to the first non - colon."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the structure to store the input type of system associated with CPE Name components of CPE Name.", "response": "def _create_cpe_parts(self, system, components):\n \"\"\"\n Create the structure to store the input type of system associated\n with components of CPE Name (hardware, operating system and software).\n\n :param string system: type of system associated with CPE Name\n :param dict components: CPE Name components to store\n :returns: None\n :exception: KeyError - incorrect system\n \"\"\"\n\n if system not in CPEComponent.SYSTEM_VALUES:\n errmsg = \"Key '{0}' is not exist\".format(system)\n raise ValueError(errmsg)\n\n elements = []\n elements.append(components)\n\n pk = CPE._system_and_parts[system]\n self[pk] = elements"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the list of Component objects of the attribute in CPE Name", "response": "def _get_attribute_components(self, att):\n \"\"\"\n Returns the component list of input attribute.\n\n :param string att: Attribute name to get\n :returns: List of Component objects of the attribute in CPE Name\n :rtype: list\n :exception: ValueError - invalid attribute name\n \"\"\"\n\n lc = []\n\n if not CPEComponent.is_valid_attribute(att):\n errmsg = \"Invalid attribute name '{0}' is not exist\".format(att)\n raise ValueError(errmsg)\n\n for pk in CPE.CPE_PART_KEYS:\n elements = self.get(pk)\n for elem in elements:\n lc.append(elem.get(att))\n\n return lc"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npack the values of the five arguments into the simple edition component and return the string representation of the edition attribute.", "response": "def _pack_edition(self):\n \"\"\"\n Pack the values of the five arguments into the simple edition\n component. If all the values are blank, just return a blank.\n\n :returns: \"edition\", \"sw_edition\", \"target_sw\", \"target_hw\" and \"other\"\n attributes packed in a only value\n :rtype: string\n :exception: TypeError - incompatible version with pack operation\n \"\"\"\n\n COMP_KEYS = (CPEComponent.ATT_EDITION,\n CPEComponent.ATT_SW_EDITION,\n CPEComponent.ATT_TARGET_SW,\n CPEComponent.ATT_TARGET_HW,\n CPEComponent.ATT_OTHER)\n\n separator = CPEComponent2_3_URI_edpacked.SEPARATOR_COMP\n\n packed_ed = []\n packed_ed.append(separator)\n\n for ck in COMP_KEYS:\n lc = self._get_attribute_components(ck)\n if len(lc) > 1:\n # Incompatible version 1.1, there are two or more elements\n # in CPE Name\n errmsg = \"Incompatible version {0} with URI\".format(\n self.VERSION)\n raise TypeError(errmsg)\n\n comp = lc[0]\n if (isinstance(comp, CPEComponentUndefined) or\n isinstance(comp, CPEComponentEmpty) or\n isinstance(comp, CPEComponentAnyValue)):\n\n value = \"\"\n elif (isinstance(comp, CPEComponentNotApplicable)):\n value = CPEComponent2_3_URI.VALUE_NA\n else:\n # Component has some value; transform this original value\n # in URI value\n value = comp.as_uri_2_3()\n\n # Save the value of edition attribute\n if ck == CPEComponent.ATT_EDITION:\n ed = value\n\n # Packed the value of component\n packed_ed.append(value)\n packed_ed.append(separator)\n\n # Del the last separator\n packed_ed_str = \"\".join(packed_ed[:-1])\n\n only_ed = []\n only_ed.append(separator)\n only_ed.append(ed)\n only_ed.append(separator)\n only_ed.append(separator)\n only_ed.append(separator)\n only_ed.append(separator)\n\n only_ed_str = \"\".join(only_ed)\n\n if (packed_ed_str == only_ed_str):\n # All the extended attributes are blank,\n # so don't do any packing, just return ed\n return ed\n else:\n # Otherwise, pack the five values into a simple string\n # prefixed and internally delimited with the tilde\n return packed_ed_str"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_uri_2_3(self):\n\n uri = []\n uri.append(\"cpe:/\")\n\n ordered_comp_parts = {\n 0: CPEComponent.ATT_PART,\n 1: CPEComponent.ATT_VENDOR,\n 2: CPEComponent.ATT_PRODUCT,\n 3: CPEComponent.ATT_VERSION,\n 4: CPEComponent.ATT_UPDATE,\n 5: CPEComponent.ATT_EDITION,\n 6: CPEComponent.ATT_LANGUAGE}\n\n # Indicates if the previous component must be set depending on the\n # value of current component\n set_prev_comp = False\n prev_comp_list = []\n\n for i in range(0, len(ordered_comp_parts)):\n ck = ordered_comp_parts[i]\n lc = self._get_attribute_components(ck)\n\n if len(lc) > 1:\n # Incompatible version 1.1, there are two or more elements\n # in CPE Name\n errmsg = \"Incompatible version {0} with URI\".format(\n self.VERSION)\n raise TypeError(errmsg)\n\n if ck == CPEComponent.ATT_EDITION:\n # Call the pack() helper function to compute the proper\n # binding for the edition element\n v = self._pack_edition()\n if not v:\n set_prev_comp = True\n prev_comp_list.append(CPEComponent2_3_URI.VALUE_ANY)\n continue\n else:\n comp = lc[0]\n\n if (isinstance(comp, CPEComponentEmpty) or\n isinstance(comp, CPEComponentAnyValue)):\n\n # Logical value any\n v = CPEComponent2_3_URI.VALUE_ANY\n\n elif isinstance(comp, CPEComponentNotApplicable):\n\n # Logical value not applicable\n v = CPEComponent2_3_URI.VALUE_NA\n elif isinstance(comp, CPEComponentUndefined):\n set_prev_comp = True\n prev_comp_list.append(CPEComponent2_3_URI.VALUE_ANY)\n continue\n else:\n # Get the value of component encoded in URI\n v = comp.as_uri_2_3()\n\n # Append v to the URI and add a separator\n uri.append(v)\n uri.append(CPEComponent2_3_URI.SEPARATOR_COMP)\n\n if set_prev_comp:\n # Set the previous attribute as logical value any\n v = CPEComponent2_3_URI.VALUE_ANY\n pos_ini = max(len(uri) - len(prev_comp_list) - 1, 1)\n increment = 2 # Count of inserted values\n\n for p, val in enumerate(prev_comp_list):\n pos = pos_ini + (p * increment)\n uri.insert(pos, v)\n uri.insert(pos + 1, CPEComponent2_3_URI.SEPARATOR_COMP)\n\n set_prev_comp = False\n prev_comp_list = []\n\n # Return the URI string, with trailing separator trimmed\n return CPE._trim(\"\".join(uri[:-1]))", "response": "Returns the CPE Name as URI string of version 2. 3."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the CPE Name as Well - Formed Name string of version 2. 3.", "response": "def as_wfn(self):\n \"\"\"\n Returns the CPE Name as Well-Formed Name string of version 2.3.\n\n :return: CPE Name as WFN string\n :rtype: string\n :exception: TypeError - incompatible version\n \"\"\"\n\n from .cpe2_3_wfn import CPE2_3_WFN\n\n wfn = []\n wfn.append(CPE2_3_WFN.CPE_PREFIX)\n\n for i in range(0, len(CPEComponent.ordered_comp_parts)):\n ck = CPEComponent.ordered_comp_parts[i]\n lc = self._get_attribute_components(ck)\n\n if len(lc) > 1:\n # Incompatible version 1.1, there are two or more elements\n # in CPE Name\n errmsg = \"Incompatible version {0} with WFN\".format(\n self.VERSION)\n raise TypeError(errmsg)\n\n else:\n comp = lc[0]\n\n v = []\n v.append(ck)\n v.append(\"=\")\n\n if isinstance(comp, CPEComponentAnyValue):\n\n # Logical value any\n v.append(CPEComponent2_3_WFN.VALUE_ANY)\n\n elif isinstance(comp, CPEComponentNotApplicable):\n\n # Logical value not applicable\n v.append(CPEComponent2_3_WFN.VALUE_NA)\n\n elif (isinstance(comp, CPEComponentUndefined) or\n isinstance(comp, CPEComponentEmpty)):\n # Do not set the attribute\n continue\n else:\n # Get the simple value of WFN of component\n v.append('\"')\n v.append(comp.as_wfn())\n v.append('\"')\n\n # Append v to the WFN and add a separator\n wfn.append(\"\".join(v))\n wfn.append(CPEComponent2_3_WFN.SEPARATOR_COMP)\n\n # Del the last separator\n wfn = wfn[:-1]\n\n # Return the WFN string\n wfn.append(CPE2_3_WFN.CPE_SUFFIX)\n\n return \"\".join(wfn)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the CPE Name as formatted string of version 2. 3.", "response": "def as_fs(self):\n \"\"\"\n Returns the CPE Name as formatted string of version 2.3.\n\n :returns: CPE Name as formatted string\n :rtype: string\n :exception: TypeError - incompatible version\n \"\"\"\n\n fs = []\n fs.append(\"cpe:2.3:\")\n\n for i in range(0, len(CPEComponent.ordered_comp_parts)):\n ck = CPEComponent.ordered_comp_parts[i]\n lc = self._get_attribute_components(ck)\n\n if len(lc) > 1:\n # Incompatible version 1.1, there are two or more elements\n # in CPE Name\n errmsg = \"Incompatible version {0} with formatted string\".format(\n self.VERSION)\n raise TypeError(errmsg)\n\n else:\n comp = lc[0]\n\n if (isinstance(comp, CPEComponentUndefined) or\n isinstance(comp, CPEComponentEmpty) or\n isinstance(comp, CPEComponentAnyValue)):\n\n # Logical value any\n v = CPEComponent2_3_FS.VALUE_ANY\n\n elif isinstance(comp, CPEComponentNotApplicable):\n\n # Logical value not applicable\n v = CPEComponent2_3_FS.VALUE_NA\n else:\n # Get the value of component encoded in formatted string\n v = comp.as_fs()\n\n # Append v to the formatted string then add a separator.\n fs.append(v)\n fs.append(CPEComponent2_3_FS.SEPARATOR_COMP)\n\n # Return the formatted string\n return CPE._trim(\"\".join(fs[:-1]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if the character c is an alphanumeric character.", "response": "def _is_alphanum(cls, c):\n \"\"\"\n Returns True if c is an uppercase letter, a lowercase letter,\n a digit or an underscore, otherwise False.\n\n :param string c: Character to check\n :returns: True if char is alphanumeric or an underscore,\n False otherwise\n :rtype: boolean\n\n TEST: a wrong character\n >>> c = \"#\"\n >>> CPEComponentSimple._is_alphanum(c)\n False\n \"\"\"\n\n alphanum_rxc = re.compile(CPEComponentSimple._ALPHANUM_PATTERN)\n return (alphanum_rxc.match(c) is not None)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _pct_encode_uri(cls, c):\n\n CPEComponentSimple.spechar_to_pce['-'] = c # bound without encoding\n CPEComponentSimple.spechar_to_pce['.'] = c # bound without encoding\n\n return CPEComponentSimple.spechar_to_pce[c]", "response": "Return the appropriate percent - encoding of character c."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _is_valid_language(self):\n\n comp_str = self._encoded_value.lower()\n lang_rxc = re.compile(CPEComponentSimple._LANGTAG_PATTERN)\n return lang_rxc.match(comp_str) is not None", "response": "Returns True if the value of component in attribute language is valid False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if the value of component in attribute part is valid False otherwise.", "response": "def _is_valid_part(self):\n \"\"\"\n Return True if the value of component in attribute \"part\" is valid,\n and otherwise False.\n\n :returns: True if value of component is valid, False otherwise\n :rtype: boolean\n \"\"\"\n\n comp_str = self._encoded_value.lower()\n part_rxc = re.compile(CPEComponentSimple._PART_PATTERN)\n return part_rxc.match(comp_str) is not None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse the value of component and return the component s value.", "response": "def _parse(self, comp_att):\n \"\"\"\n Check if the value of component is correct in the attribute \"comp_att\".\n\n :param string comp_att: attribute associated with value of component\n :returns: None\n :exception: ValueError - incorrect value of component\n \"\"\"\n\n errmsg = \"Invalid attribute '{0}'\".format(comp_att)\n\n if not CPEComponent.is_valid_attribute(comp_att):\n raise ValueError(errmsg)\n\n comp_str = self._encoded_value\n\n errmsg = \"Invalid value of attribute '{0}': {1}\".format(\n comp_att, comp_str)\n\n # Check part (system type) value\n if comp_att == CPEComponentSimple.ATT_PART:\n if not self._is_valid_part():\n raise ValueError(errmsg)\n\n # Check language value\n elif comp_att == CPEComponentSimple.ATT_LANGUAGE:\n if not self._is_valid_language():\n raise ValueError(errmsg)\n\n # Check edition value\n elif comp_att == CPEComponentSimple.ATT_EDITION:\n if not self._is_valid_edition():\n raise ValueError(errmsg)\n\n # Check other type of component value\n elif not self._is_valid_value():\n raise ValueError(errmsg)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef as_fs(self):\n\n s = self._standard_value\n result = []\n idx = 0\n while (idx < len(s)):\n\n c = s[idx] # get the idx'th character of s\n if c != \"\\\\\":\n # unquoted characters pass thru unharmed\n result.append(c)\n else:\n # Escaped characters are examined\n nextchr = s[idx + 1]\n\n if (nextchr == \".\") or (nextchr == \"-\") or (nextchr == \"_\"):\n # the period, hyphen and underscore pass unharmed\n result.append(nextchr)\n idx += 1\n else:\n # all others retain escaping\n result.append(\"\\\\\")\n result.append(nextchr)\n idx += 2\n continue\n idx += 1\n\n return \"\".join(result)", "response": "Returns the value of component encoded as a string."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the value of component encoded as URI string.", "response": "def as_uri_2_3(self):\n \"\"\"\n Returns the value of component encoded as URI string.\n\n Scans an input string s and applies the following transformations:\n\n - Pass alphanumeric characters thru untouched\n - Percent-encode quoted non-alphanumerics as needed\n - Unquoted special characters are mapped to their special forms.\n\n :returns: URI string associated with component\n :rtype: string\n \"\"\"\n\n s = self._standard_value\n result = []\n idx = 0\n while (idx < len(s)):\n thischar = s[idx] # get the idx'th character of s\n\n # alphanumerics (incl. underscore) pass untouched\n if (CPEComponentSimple._is_alphanum(thischar)):\n result.append(thischar)\n idx += 1\n continue\n\n # escape character\n if (thischar == \"\\\\\"):\n idx += 1\n nxtchar = s[idx]\n result.append(CPEComponentSimple._pct_encode_uri(nxtchar))\n idx += 1\n continue\n\n # Bind the unquoted '?' special character to \"%01\".\n if (thischar == \"?\"):\n result.append(\"%01\")\n\n # Bind the unquoted '*' special character to \"%02\".\n if (thischar == \"*\"):\n result.append(\"%02\")\n\n idx += 1\n\n return \"\".join(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the value of component. By default the component has a simple value.", "response": "def set_value(self, comp_str, comp_att):\n \"\"\"\n Set the value of component. By default, the component has a simple\n value.\n\n :param string comp_str: new value of component\n :param string comp_att: attribute associated with value of component\n :returns: None\n :exception: ValueError - incorrect value of component\n \"\"\"\n\n old_value = self._encoded_value\n self._encoded_value = comp_str\n\n # Check the value of component\n try:\n self._parse(comp_att)\n except ValueError:\n # Restore old value of component\n self._encoded_value = old_value\n raise\n\n # Convert encoding value to standard value (WFN)\n self._decode()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the characters of string s to standard value (WFN value). Inspect each character in value of component. Copy quoted characters, with their escaping, into the result. Look for unquoted non alphanumerics and if not \"*\" or \"?\", add escaping. :exception: ValueError - invalid character in value of component", "response": "def _decode(self):\n \"\"\"\n Convert the characters of string s to standard value (WFN value).\n Inspect each character in value of component. Copy quoted characters,\n with their escaping, into the result. Look for unquoted non\n alphanumerics and if not \"*\" or \"?\", add escaping.\n\n :exception: ValueError - invalid character in value of component\n \"\"\"\n\n result = []\n idx = 0\n s = self._encoded_value\n embedded = False\n\n errmsg = []\n errmsg.append(\"Invalid character '\")\n\n while (idx < len(s)):\n c = s[idx] # get the idx'th character of s\n errmsg.append(c)\n errmsg.append(\"'\")\n errmsg_str = \"\".join(errmsg)\n\n if (CPEComponentSimple._is_alphanum(c)):\n # Alphanumeric characters pass untouched\n result.append(c)\n idx += 1\n embedded = True\n continue\n\n if c == \"\\\\\":\n # Anything quoted in the bound string stays quoted\n # in the unbound string.\n result.append(s[idx: idx + 2])\n idx += 2\n embedded = True\n continue\n\n if (c == CPEComponent2_3_FS.WILDCARD_MULTI):\n # An unquoted asterisk must appear at the beginning or\n # end of the string.\n if (idx == 0) or (idx == (len(s) - 1)):\n result.append(c)\n idx += 1\n embedded = True\n continue\n else:\n raise ValueError(errmsg_str)\n\n if (c == CPEComponent2_3_FS.WILDCARD_ONE):\n # An unquoted question mark must appear at the beginning or\n # end of the string, or in a leading or trailing sequence:\n # - ? legal at beginning or end\n # - embedded is false, so must be preceded by ?\n # - embedded is true, so must be followed by ?\n if (((idx == 0) or (idx == (len(s) - 1))) or\n ((not embedded) and (s[idx - 1] == CPEComponent2_3_FS.WILDCARD_ONE)) or\n (embedded and (s[idx + 1] == CPEComponent2_3_FS.WILDCARD_ONE))):\n result.append(c)\n idx += 1\n embedded = False\n continue\n else:\n raise ValueError(errmsg_str)\n\n # all other characters must be quoted\n result.append(\"\\\\\")\n result.append(c)\n idx += 1\n embedded = True\n\n self._standard_value = \"\".join(result)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse(self):\n\n # CPE Name must not have whitespaces\n if (self.cpe_str.find(\" \") != -1):\n errmsg = \"Bad-formed CPE Name: it must not have whitespaces\"\n raise ValueError(errmsg)\n\n # Partitioning of CPE Name in parts\n parts_match = CPE1_1._parts_rxc.match(self.cpe_str)\n\n # ################################\n # Validation of CPE Name parts #\n # ################################\n\n if (parts_match is None):\n errmsg = \"Bad-formed CPE Name: not correct definition of CPE Name parts\"\n raise ValueError(errmsg)\n\n CPE_PART_KEYS = (CPE.KEY_HW, CPE.KEY_OS, CPE.KEY_APP)\n\n for pk in CPE_PART_KEYS:\n # Get part content\n part = parts_match.group(pk)\n elements = []\n\n if (part is not None):\n # Part of CPE Name defined\n\n # ###############################\n # Validation of part elements #\n # ###############################\n\n # semicolon (;) is used to separate the part elements\n for part_elem in part.split(CPE1_1.ELEMENT_SEPARATOR):\n j = 1\n\n # ####################################\n # Validation of element components #\n # ####################################\n\n components = dict()\n\n # colon (:) is used to separate the element components\n for elem_comp in part_elem.split(CPEComponent1_1.SEPARATOR_COMP):\n comp_att = CPEComponent.ordered_comp_parts[j]\n\n if elem_comp == CPEComponent1_1.VALUE_EMPTY:\n comp = CPEComponentEmpty()\n else:\n try:\n comp = CPEComponent1_1(elem_comp, comp_att)\n except ValueError:\n errmsg = \"Bad-formed CPE Name: not correct value: {0}\".format(\n elem_comp)\n raise ValueError(errmsg)\n\n # Identification of component name\n components[comp_att] = comp\n\n j += 1\n\n # Adds the components of version 2.3 of CPE not defined\n # in version 1.1\n for idx in range(j, len(CPEComponent.ordered_comp_parts)):\n comp_att = CPEComponent.ordered_comp_parts[idx]\n components[comp_att] = CPEComponentUndefined()\n\n # Get the type of system associated with CPE Name and\n # store it in element as component\n if (pk == CPE.KEY_HW):\n components[CPEComponent.ATT_PART] = CPEComponent1_1(\n CPEComponent.VALUE_PART_HW, CPEComponent.ATT_PART)\n elif (pk == CPE.KEY_OS):\n components[CPEComponent.ATT_PART] = CPEComponent1_1(\n CPEComponent.VALUE_PART_OS, CPEComponent.ATT_PART)\n elif (pk == CPE.KEY_APP):\n components[CPEComponent.ATT_PART] = CPEComponent1_1(\n CPEComponent.VALUE_PART_APP, CPEComponent.ATT_PART)\n\n # Store the element identified\n elements.append(components)\n\n # Store the part identified\n self[pk] = elements\n\n self[CPE.KEY_UNDEFINED] = []", "response": "Checks if the CPE Name is valid."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the values of attribute att_name of CPE Name.", "response": "def get_attribute_values(self, att_name):\n \"\"\"\n Returns the values of attribute \"att_name\" of CPE Name.\n By default a only element in each part.\n\n :param string att_name: Attribute name to get\n :returns: List of attribute values\n :rtype: list\n :exception: ValueError - invalid attribute name\n \"\"\"\n\n lc = []\n\n if not CPEComponent.is_valid_attribute(att_name):\n errmsg = \"Invalid attribute name: {0}\".format(att_name)\n raise ValueError(errmsg)\n\n for pk in CPE.CPE_PART_KEYS:\n elements = self.get(pk)\n for elem in elements:\n comp = elem.get(att_name)\n\n if (isinstance(comp, CPEComponentEmpty) or\n isinstance(comp, CPEComponentUndefined)):\n\n value = CPEComponent1_1.VALUE_EMPTY\n else:\n value = comp.get_value()\n\n lc.append(value)\n return lc"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_value(self, comp_str):\n\n self._is_negated = False\n self._encoded_value = comp_str\n self._standard_value = super(\n CPEComponent2_3_URI_edpacked, self)._decode()", "response": "Set the value of component."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndecoding the encoded value of component into a standard value.", "response": "def _decode(self):\n \"\"\"\n Convert the encoded value of component to standard value (WFN value).\n \"\"\"\n\n s = self._encoded_value\n elements = s.replace('~', '').split('!')\n dec_elements = []\n\n for elem in elements:\n result = []\n idx = 0\n while (idx < len(elem)):\n # Get the idx'th character of s\n c = elem[idx]\n\n if (c in CPEComponent1_1.NON_STANDARD_VALUES):\n # Escape character\n result.append(\"\\\\\")\n result.append(c)\n else:\n # Do nothing\n result.append(c)\n\n idx += 1\n dec_elements.append(\"\".join(result))\n\n self._standard_value = dec_elements"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if the value of generic attribute is valid and otherwise False.", "response": "def _is_valid_value(self):\n \"\"\"\n Return True if the value of component in generic attribute is valid,\n and otherwise False.\n\n :returns: True if value is valid, False otherwise\n :rtype: boolean\n \"\"\"\n\n comp_str = self._encoded_value\n\n value_pattern = []\n value_pattern.append(\"^((\")\n value_pattern.append(\"~[\")\n value_pattern.append(CPEComponent1_1._STRING)\n value_pattern.append(\"]+\")\n value_pattern.append(\")|(\")\n value_pattern.append(\"[\")\n value_pattern.append(CPEComponent1_1._STRING)\n value_pattern.append(\"]+(![\")\n value_pattern.append(CPEComponent1_1._STRING)\n value_pattern.append(\"]+)*\")\n value_pattern.append(\"))$\")\n\n value_rxc = re.compile(\"\".join(value_pattern))\n return value_rxc.match(comp_str) is not None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_value(self, comp_str, comp_att):\n\n super(CPEComponent1_1, self).set_value(comp_str, comp_att)\n self._is_negated = comp_str.startswith('~')", "response": "Set the value of component. By default the component has a simple\n value."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _create_component(cls, att, value):\n\n if value == CPEComponent2_3_URI.VALUE_UNDEFINED:\n comp = CPEComponentUndefined()\n elif (value == CPEComponent2_3_URI.VALUE_ANY or\n value == CPEComponent2_3_URI.VALUE_EMPTY):\n comp = CPEComponentAnyValue()\n elif (value == CPEComponent2_3_URI.VALUE_NA):\n comp = CPEComponentNotApplicable()\n else:\n comp = CPEComponentNotApplicable()\n try:\n comp = CPEComponent2_3_URI(value, att)\n except ValueError:\n errmsg = \"Invalid value of attribute '{0}': {1} \".format(att,\n value)\n raise ValueError(errmsg)\n\n return comp", "response": "Create a component object with value value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nunpacking the edition attribute and sets the attributes in wfn accordingly.", "response": "def _unpack_edition(cls, value):\n \"\"\"\n Unpack its elements and set the attributes in wfn accordingly.\n Parse out the five elements:\n\n ~ edition ~ software edition ~ target sw ~ target hw ~ other\n\n :param string value: Value of edition attribute\n :returns: Dictionary with parts of edition attribute\n :exception: ValueError - invalid value of edition attribute\n \"\"\"\n\n components = value.split(CPEComponent2_3_URI.SEPARATOR_PACKED_EDITION)\n d = dict()\n\n ed = components[1]\n sw_ed = components[2]\n t_sw = components[3]\n t_hw = components[4]\n oth = components[5]\n\n ck = CPEComponent.ATT_EDITION\n d[ck] = CPE2_3_URI._create_component(ck, ed)\n ck = CPEComponent.ATT_SW_EDITION\n d[ck] = CPE2_3_URI._create_component(ck, sw_ed)\n ck = CPEComponent.ATT_TARGET_SW\n d[ck] = CPE2_3_URI._create_component(ck, t_sw)\n ck = CPEComponent.ATT_TARGET_HW\n d[ck] = CPE2_3_URI._create_component(ck, t_hw)\n ck = CPEComponent.ATT_OTHER\n d[ck] = CPE2_3_URI._create_component(ck, oth)\n\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses the CPE Name and returns a dictionary of components and edition parts.", "response": "def _parse(self):\n \"\"\"\n Checks if the CPE Name is valid.\n\n :returns: None\n :exception: ValueError - bad-formed CPE Name\n \"\"\"\n\n # CPE Name must not have whitespaces\n if (self._str.find(\" \") != -1):\n msg = \"Bad-formed CPE Name: it must not have whitespaces\"\n raise ValueError(msg)\n\n # Partitioning of CPE Name\n parts_match = CPE2_3_URI._parts_rxc.match(self._str)\n\n # Validation of CPE Name parts\n if (parts_match is None):\n msg = \"Bad-formed CPE Name: validation of parts failed\"\n raise ValueError(msg)\n\n components = dict()\n edition_parts = dict()\n\n for ck in CPEComponent.CPE_COMP_KEYS:\n value = parts_match.group(ck)\n\n try:\n if (ck == CPEComponent.ATT_EDITION and value is not None):\n if value[0] == CPEComponent2_3_URI.SEPARATOR_PACKED_EDITION:\n # Unpack the edition part\n edition_parts = CPE2_3_URI._unpack_edition(value)\n else:\n comp = CPE2_3_URI._create_component(ck, value)\n else:\n comp = CPE2_3_URI._create_component(ck, value)\n except ValueError:\n errmsg = \"Bad-formed CPE Name: not correct value '{0}'\".format(\n value)\n raise ValueError(errmsg)\n else:\n components[ck] = comp\n\n components = dict(components, **edition_parts)\n\n # Adds the components of version 2.3 of CPE not defined in version 2.2\n for ck2 in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n if ck2 not in components.keys():\n components[ck2] = CPEComponentUndefined()\n\n # Exchange the undefined values in middle attributes of CPE Name for\n # logical value ANY\n check_change = True\n\n # Start in the last attribute specififed in CPE Name\n for ck in CPEComponent.CPE_COMP_KEYS[::-1]:\n if ck in components:\n comp = components[ck]\n if check_change:\n check_change = ((ck != CPEComponent.ATT_EDITION) and\n (comp == CPEComponentUndefined()) or\n (ck == CPEComponent.ATT_EDITION and\n (len(edition_parts) == 0)))\n elif comp == CPEComponentUndefined():\n comp = CPEComponentAnyValue()\n\n components[ck] = comp\n\n # Storage of CPE Name\n part_comp = components[CPEComponent.ATT_PART]\n if isinstance(part_comp, CPEComponentLogical):\n elements = []\n elements.append(components)\n self[CPE.KEY_UNDEFINED] = elements\n else:\n # Create internal structure of CPE Name in parts:\n # one of them is filled with identified components,\n # the rest are empty\n system = parts_match.group(CPEComponent.ATT_PART)\n if system in CPEComponent.SYSTEM_VALUES:\n self._create_cpe_parts(system, components)\n else:\n self._create_cpe_parts(CPEComponent.VALUE_PART_UNDEFINED,\n components)\n\n # Fills the empty parts of internal structure of CPE Name\n for pk in CPE.CPE_PART_KEYS:\n if pk not in self.keys():\n # Empty part\n self[pk] = []"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef as_wfn(self):\n\n if self._str.find(CPEComponent2_3_URI.SEPARATOR_PACKED_EDITION) == -1:\n # Edition unpacked, only show the first seven components\n\n wfn = []\n wfn.append(CPE2_3_WFN.CPE_PREFIX)\n\n for ck in CPEComponent.CPE_COMP_KEYS:\n lc = self._get_attribute_components(ck)\n\n if len(lc) > 1:\n # Incompatible version 1.1, there are two or more elements\n # in CPE Name\n errmsg = \"Incompatible version {0} with WFN\".format(\n self.VERSION)\n raise TypeError(errmsg)\n\n else:\n comp = lc[0]\n\n v = []\n v.append(ck)\n v.append(\"=\")\n\n if (isinstance(comp, CPEComponentUndefined) or\n isinstance(comp, CPEComponentEmpty)):\n\n # Do not set the attribute\n continue\n\n elif isinstance(comp, CPEComponentAnyValue):\n\n # Logical value any\n v.append(CPEComponent2_3_WFN.VALUE_ANY)\n\n elif isinstance(comp, CPEComponentNotApplicable):\n\n # Logical value not applicable\n v.append(CPEComponent2_3_WFN.VALUE_NA)\n\n else:\n # Get the value of WFN of component\n v.append('\"')\n v.append(comp.as_wfn())\n v.append('\"')\n\n # Append v to the WFN and add a separator\n wfn.append(\"\".join(v))\n wfn.append(CPEComponent2_3_WFN.SEPARATOR_COMP)\n\n # Del the last separator\n wfn = wfn[:-1]\n\n # Return the WFN string\n wfn.append(CPE2_3_WFN.CPE_SUFFIX)\n\n return \"\".join(wfn)\n\n else:\n # Shows all components\n return super(CPE2_3_URI, self).as_wfn()", "response": "Returns the CPE Name as Well - Formed Name string of version 2. 3."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts the characters of character in value of component to standard value (WFN value). This function scans the value of component and returns a copy with all percent-encoded characters decoded. :exception: ValueError - invalid character in value of component", "response": "def _decode(self):\n \"\"\"\n Convert the characters of character in value of component to standard\n value (WFN value).\n This function scans the value of component and returns a copy\n with all percent-encoded characters decoded.\n\n :exception: ValueError - invalid character in value of component\n \"\"\"\n\n result = []\n idx = 0\n s = self._encoded_value\n embedded = False\n\n errmsg = []\n errmsg.append(\"Invalid value: \")\n\n while (idx < len(s)):\n errmsg.append(s)\n errmsg_str = \"\".join(errmsg)\n\n # Get the idx'th character of s\n c = s[idx]\n\n # Deal with dot, hyphen and tilde: decode with quoting\n if ((c == '.') or (c == '-') or (c == '~')):\n result.append(\"\\\\\")\n result.append(c)\n idx += 1\n embedded = True # a non-%01 encountered\n continue\n\n if (c != '%'):\n result.append(c)\n idx += 1\n embedded = True # a non-%01 encountered\n continue\n\n # we get here if we have a substring starting w/ '%'\n form = s[idx: idx + 3] # get the three-char sequence\n\n if form == CPEComponent2_3_URI.WILDCARD_ONE:\n # If %01 legal at beginning or end\n # embedded is false, so must be preceded by %01\n # embedded is true, so must be followed by %01\n if (((idx == 0) or (idx == (len(s)-3))) or\n ((not embedded) and (s[idx - 3:idx] == CPEComponent2_3_URI.WILDCARD_ONE)) or\n (embedded and (len(s) >= idx + 6) and (s[idx + 3:idx + 6] == CPEComponent2_3_URI.WILDCARD_ONE))):\n\n # A percent-encoded question mark is found\n # at the beginning or the end of the string,\n # or embedded in sequence as required.\n # Decode to unquoted form.\n result.append(CPEComponent2_3_WFN.WILDCARD_ONE)\n idx += 3\n continue\n else:\n raise ValueError(errmsg_str)\n\n elif form == CPEComponent2_3_URI.WILDCARD_MULTI:\n if ((idx == 0) or (idx == (len(s) - 3))):\n # Percent-encoded asterisk is at the beginning\n # or the end of the string, as required.\n # Decode to unquoted form.\n result.append(CPEComponent2_3_WFN.WILDCARD_MULTI)\n else:\n raise ValueError(errmsg_str)\n\n elif form in CPEComponent2_3_URI.pce_char_to_decode.keys():\n value = CPEComponent2_3_URI.pce_char_to_decode[form]\n result.append(value)\n\n else:\n errmsg.append(\"Invalid percent-encoded character: \")\n errmsg.append(s)\n raise ValueError(\"\".join(errmsg))\n\n idx += 3\n embedded = True # a non-%01 encountered.\n\n self._standard_value = \"\".join(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the input value of attribute edition is valid and otherwise False.", "response": "def _is_valid_edition(self):\n \"\"\"\n Return True if the input value of attribute \"edition\" is valid,\n and otherwise False.\n\n :returns: True if value is valid, False otherwise\n :rtype: boolean\n \"\"\"\n\n comp_str = self._standard_value[0]\n\n packed = []\n packed.append(\"(\")\n packed.append(CPEComponent2_3_URI.SEPARATOR_PACKED_EDITION)\n packed.append(CPEComponent2_3_URI._string)\n packed.append(\"){5}\")\n\n value_pattern = []\n value_pattern.append(\"^(\")\n value_pattern.append(CPEComponent2_3_URI._string)\n value_pattern.append(\"|\")\n value_pattern.append(\"\".join(packed))\n value_pattern.append(\")$\")\n\n value_rxc = re.compile(\"\".join(value_pattern))\n return value_rxc.match(comp_str) is not None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _is_valid_language(self):\n\n def check_generic_language(self, value):\n \"\"\"\n Check possible values in language part\n when region part exists or not in language value.\n\n Possible values of language attribute: a=letter\n | *a\n | *aa\n | aa\n | aaa\n | ?a\n | ?aa\n | ??\n | ??a\n | ???\n \"\"\"\n lang_pattern = []\n lang_pattern.append(\"^(\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"[a-z]{1,2}\")\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"(([a-z][a-z]?)|(\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"(\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"|[a-z])?))\")\n lang_pattern.append(\"|([a-z]{2,3}))$\")\n\n lang_rxc = re.compile(\"\".join(lang_pattern))\n\n return lang_rxc.match(value)\n\n def check_language_without_region(self, value):\n \"\"\"\n Check possible values in language part\n when region part not exist in language value.\n\n Possible values of language attribute: a=letter\n | a?\n | aa?\n | a??\n | a*\n | aa*\n | aaa*\n | *a*\n | *a?\n | ?a*\n | ?a?\n \"\"\"\n lang_pattern = []\n lang_pattern.append(\"^([a-z]\")\n lang_pattern.append(\"([a-z](\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"|\")\n lang_pattern.append(\"([a-z]\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"))\")\n lang_pattern.append(\"|\")\n lang_pattern.append(\"\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"(\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\")?\")\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\")|\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\"[a-z](\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\")\")\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"[a-z](\\\\\")\n lang_pattern.append(self.WILDCARD_MULTI)\n lang_pattern.append(\"|\\\\\")\n lang_pattern.append(self.WILDCARD_ONE)\n lang_pattern.append(\")\")\n lang_pattern.append(\")$\")\n\n lang_rxc = re.compile(\"\".join(lang_pattern))\n\n return lang_rxc.match(value)\n\n def check_region_with_language(self, value):\n \"\"\"\n Check possible values in region part when language part exists.\n\n Possible values of language attribute: a=letter, 1=digit\n | *\n | a*\n | a?\n | aa\n | ??\n | 1*\n | 1??\n | 11*\n | 11?\n | 111\n | ???\n \"\"\"\n region_pattern = []\n region_pattern.append(\"^(\")\n region_pattern.append(\"(\\\\\")\n region_pattern.append(self.WILDCARD_MULTI)\n region_pattern.append(\")|((\\\\\")\n region_pattern.append(self.WILDCARD_ONE)\n region_pattern.append(\"){2,3})|([a-z]([a-z]|\\\\\")\n region_pattern.append(self.WILDCARD_MULTI)\n region_pattern.append(\"|\\\\\")\n region_pattern.append(self.WILDCARD_ONE)\n region_pattern.append(\"))|([0-9](\\\\\")\n region_pattern.append(self.WILDCARD_MULTI)\n region_pattern.append(\"|\\\\\")\n region_pattern.append(self.WILDCARD_ONE)\n region_pattern.append(\"(\\\\\")\n region_pattern.append(self.WILDCARD_ONE)\n region_pattern.append(\")?|[0-9][0-9\\\\\")\n region_pattern.append(self.WILDCARD_MULTI)\n region_pattern.append(\"\\\\\")\n region_pattern.append(self.WILDCARD_ONE)\n region_pattern.append(\"])))$\")\n\n region_rxc = re.compile(\"\".join(region_pattern))\n return region_rxc.match(region)\n\n def check_region_without_language(self, value):\n \"\"\"\n Check possible values in region part when language part not exist.\n\n Possible values of language attribute: 1=digit\n | *111\n | *11\n | *1\n \"\"\"\n region_pattern = []\n region_pattern.append(\"^(\")\n region_pattern.append(\"(\\\\\")\n region_pattern.append(self.WILDCARD_MULTI)\n region_pattern.append(\"[0-9])\")\n region_pattern.append(\"([0-9]([0-9])?)?\")\n region_pattern.append(\")$\")\n\n region_rxc = re.compile(\"\".join(region_pattern))\n return region_rxc.match(region)\n\n comp_str = self._encoded_value.lower()\n\n # Value with wildcards; separate language and region of value\n parts = comp_str.split(self.SEPARATOR_LANG)\n language = parts[0]\n region_exists = len(parts) == 2\n\n # Check the language part\n if check_generic_language(self, language) is not None:\n # Valid language, check region part\n if region_exists:\n # Region part exists; check it\n region = parts[1]\n return (check_region_with_language(self, region) is not None)\n else:\n # Not region part\n return True\n elif check_language_without_region(self, language) is not None:\n # Language without region; region part should not exist\n return not region_exists\n else:\n # Language part not exist; check region part\n region = parts[0]\n return check_region_without_language(self, region) is not None", "response": "Checks if the value of component in attribute language is valid and if it is valid."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if the value of component in attribute part is valid and if otherwise False.", "response": "def _is_valid_part(self):\n \"\"\"\n Return True if the value of component in attribute \"part\" is valid,\n and otherwise False.\n\n :returns: True if value of component is valid, False otherwise\n :rtype: boolean\n \"\"\"\n\n comp_str = self._encoded_value\n\n # Check if value of component do not have wildcard\n if ((comp_str.find(self.WILDCARD_ONE) == -1) and\n (comp_str.find(self.WILDCARD_MULTI) == -1)):\n\n return super(CPEComponent2_3, self)._is_valid_part()\n\n # Compilation of regular expression associated with value of part\n part_pattern = \"^(\\{0}|\\{1})$\".format(self.WILDCARD_ONE,\n self.WILDCARD_MULTI)\n part_rxc = re.compile(part_pattern)\n\n return part_rxc.match(comp_str) is not None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _compare(cls, source, target):\n\n if (CPESet2_3._is_string(source)):\n source = source.lower()\n if (CPESet2_3._is_string(target)):\n target = target.lower()\n\n # In this specification, unquoted wildcard characters in the target\n # yield an undefined result\n\n if (CPESet2_3._is_string(target) and\n CPESet2_3._contains_wildcards(target)):\n return CPESet2_3.LOGICAL_VALUE_UNDEFINED\n\n # If source and target attribute values are equal,\n # then the result is EQUAL\n if (source == target):\n return CPESet2_3.LOGICAL_VALUE_EQUAL\n\n # If source attribute value is ANY, then the result is SUPERSET\n if (source == CPEComponent2_3_WFN.VALUE_ANY):\n return CPESet2_3.LOGICAL_VALUE_SUPERSET\n\n # If target attribute value is ANY, then the result is SUBSET\n if (target == CPEComponent2_3_WFN.VALUE_ANY):\n return CPESet2_3.LOGICAL_VALUE_SUBSET\n\n # If either source or target attribute value is NA\n # then the result is DISJOINT\n isSourceNA = source == CPEComponent2_3_WFN.VALUE_NA\n isTargetNA = target == CPEComponent2_3_WFN.VALUE_NA\n\n if (isSourceNA or isTargetNA):\n return CPESet2_3.LOGICAL_VALUE_DISJOINT\n\n # If we get to this point, we are comparing two strings\n return CPESet2_3._compare_strings(source, target)", "response": "Compares two values associated with a attribute of two WFNs and returns the attribute comparison relation."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompare a source string to a target string, and addresses the condition in which the source string includes unquoted special characters. It performs a simple regular expression match, with the assumption that (as required) unquoted special characters appear only at the beginning and/or the end of the source string. It also properly differentiates between unquoted and quoted special characters. :param string source: First string value :param string target: Second string value :returns: The comparison relation among input strings. :rtype: int", "response": "def _compare_strings(cls, source, target):\n \"\"\"\n Compares a source string to a target string,\n and addresses the condition in which the source string\n includes unquoted special characters.\n\n It performs a simple regular expression match,\n with the assumption that (as required) unquoted special characters\n appear only at the beginning and/or the end of the source string.\n\n It also properly differentiates between unquoted and quoted\n special characters.\n\n :param string source: First string value\n :param string target: Second string value\n :returns: The comparison relation among input strings.\n :rtype: int\n \"\"\"\n\n start = 0\n end = len(source)\n begins = 0\n ends = 0\n\n # Reading of initial wildcard in source\n if source.startswith(CPEComponent2_3_WFN.WILDCARD_MULTI):\n # Source starts with \"*\"\n start = 1\n begins = -1\n else:\n while ((start < len(source)) and\n source.startswith(CPEComponent2_3_WFN.WILDCARD_ONE,\n start, start)):\n # Source starts with one or more \"?\"\n start += 1\n begins += 1\n\n # Reading of final wildcard in source\n if (source.endswith(CPEComponent2_3_WFN.WILDCARD_MULTI) and\n CPESet2_3._is_even_wildcards(source, end - 1)):\n\n # Source ends in \"*\"\n end -= 1\n ends = -1\n else:\n while ((end > 0) and\n source.endswith(CPEComponent2_3_WFN.WILDCARD_ONE, end - 1, end) and\n CPESet2_3._is_even_wildcards(source, end - 1)):\n\n # Source ends in \"?\"\n end -= 1\n ends += 1\n\n source = source[start: end]\n index = -1\n leftover = len(target)\n\n while (leftover > 0):\n index = target.find(source, index + 1)\n if (index == -1):\n break\n escapes = target.count(\"\\\\\", 0, index)\n if ((index > 0) and (begins != -1) and\n (begins < (index - escapes))):\n\n break\n\n escapes = target.count(\"\\\\\", index + 1, len(target))\n leftover = len(target) - index - escapes - len(source)\n if ((leftover > 0) and ((ends != -1) and (leftover > ends))):\n continue\n\n return CPESet2_3.LOGICAL_VALUE_SUPERSET\n\n return CPESet2_3.LOGICAL_VALUE_DISJOINT"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if the string contains any unquoted special characters and otherwise False.", "response": "def _contains_wildcards(cls, s):\n \"\"\"\n Return True if the string contains any unquoted special characters\n (question-mark or asterisk), otherwise False.\n\n Ex: _contains_wildcards(\"foo\") => FALSE\n Ex: _contains_wildcards(\"foo\\?\") => FALSE\n Ex: _contains_wildcards(\"foo?\") => TRUE\n Ex: _contains_wildcards(\"\\*bar\") => FALSE\n Ex: _contains_wildcards(\"*bar\") => TRUE\n\n :param string s: string to check\n :returns: True if string contains any unquoted special characters,\n False otherwise.\n :rtype: boolean\n\n This function is a support function for _compare().\n \"\"\"\n\n idx = s.find(\"*\")\n if idx != -1:\n if idx == 0:\n return True\n else:\n if s[idx - 1] != \"\\\\\":\n return True\n\n idx = s.find(\"?\")\n if idx != -1:\n if idx == 0:\n return True\n else:\n if s[idx - 1] != \"\\\\\":\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _is_even_wildcards(cls, str, idx):\n\n result = 0\n while ((idx > 0) and (str[idx - 1] == \"\\\\\")):\n idx -= 1\n result += 1\n\n isEvenNumber = (result % 2) == 0\n return isEvenNumber", "response": "Checks if the string str contains even wildcards."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _is_string(cls, arg):\n\n isAny = arg == CPEComponent2_3_WFN.VALUE_ANY\n isNa = arg == CPEComponent2_3_WFN.VALUE_NA\n\n return not (isAny or isNa)", "response": "Checks if the given string value is a string value."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compare_wfns(cls, source, target):\n\n # Compare results using the get() function in WFN\n for att in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n value_src = source.get_attribute_values(att)[0]\n if value_src.find('\"') > -1:\n # Not a logical value: del double quotes\n value_src = value_src[1:-1]\n\n value_tar = target.get_attribute_values(att)[0]\n if value_tar.find('\"') > -1:\n # Not a logical value: del double quotes\n value_tar = value_tar[1:-1]\n\n yield (att, CPESet2_3._compare(value_src, value_tar))", "response": "Compares two set of WFNs and returns a generator of pairwise attribute - value - comparison results."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cpe_disjoint(cls, source, target):\n\n # If any pairwise comparison returned DISJOINT then\n # the overall name relationship is DISJOINT\n for att, result in CPESet2_3.compare_wfns(source, target):\n isDisjoint = result == CPESet2_3.LOGICAL_VALUE_DISJOINT\n if isDisjoint:\n return True\n return False", "response": "Checks if the set - theoretic relation between two WFNs is DISJOINT."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompare two set - theoretic WFNs and returns True if the set - theoretic relation between the names is EQUAL otherwise False.", "response": "def cpe_equal(cls, source, target):\n \"\"\"\n Compares two WFNs and returns True if the set-theoretic relation\n between the names is EQUAL.\n\n :param CPE2_3_WFN source: first WFN CPE Name\n :param CPE2_3_WFN target: seconds WFN CPE Name\n :returns: True if the set relation between source and target\n is EQUAL, otherwise False.\n :rtype: boolean\n \"\"\"\n\n # If any pairwise comparison returned EQUAL then\n # the overall name relationship is EQUAL\n for att, result in CPESet2_3.compare_wfns(source, target):\n isEqual = result == CPESet2_3.LOGICAL_VALUE_EQUAL\n if not isEqual:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if the set - theoretic relation between source and target is SUBSET otherwise False.", "response": "def cpe_subset(cls, source, target):\n \"\"\"\n Compares two WFNs and returns True if the set-theoretic relation\n between the names is (non-proper) SUBSET.\n\n :param CPE2_3_WFN source: first WFN CPE Name\n :param CPE2_3_WFN target: seconds WFN CPE Name\n :returns: True if the set relation between source and target\n is SUBSET, otherwise False.\n :rtype: boolean\n \"\"\"\n\n # If any pairwise comparison returned something other than SUBSET\n # or EQUAL, then SUBSET is False.\n for att, result in CPESet2_3.compare_wfns(source, target):\n isSubset = result == CPESet2_3.LOGICAL_VALUE_SUBSET\n isEqual = result == CPESet2_3.LOGICAL_VALUE_EQUAL\n if (not isSubset) and (not isEqual):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cpe_superset(cls, source, target):\n\n # If any pairwise comparison returned something other than SUPERSET\n # or EQUAL, then SUPERSET is False.\n for att, result in CPESet2_3.compare_wfns(source, target):\n isSuperset = result == CPESet2_3.LOGICAL_VALUE_SUPERSET\n isEqual = result == CPESet2_3.LOGICAL_VALUE_EQUAL\n if (not isSuperset) and (not isEqual):\n return False\n\n return True", "response": "Returns True if the set - theoretic relation between two WFNs is SUPERSET otherwise False."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a CPE element to the set if not already.", "response": "def append(self, cpe):\n \"\"\"\n Adds a CPE element to the set if not already.\n Only WFN CPE Names are valid, so this function converts the input CPE\n object of version 2.3 to WFN style.\n\n :param CPE cpe: CPE Name to store in set\n :returns: None\n :exception: ValueError - invalid version of CPE Name\n \"\"\"\n\n if cpe.VERSION != CPE2_3.VERSION:\n errmsg = \"CPE Name version {0} not valid, version 2.3 expected\".format(\n cpe.VERSION)\n raise ValueError(errmsg)\n\n for k in self.K:\n if cpe._str == k._str:\n return None\n\n if isinstance(cpe, CPE2_3_WFN):\n self.K.append(cpe)\n else:\n # Convert the CPE Name to WFN\n wfn = CPE2_3_WFN(cpe.as_wfn())\n self.K.append(wfn)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if X matches any member of K False otherwise.", "response": "def name_match(self, wfn):\n \"\"\"\n Accepts a set of CPE Names K and a candidate CPE Name X. It returns\n 'True' if X matches any member of K, and 'False' otherwise.\n\n :param CPESet self: A set of m known CPE Names K = {K1, K2, \u2026, Km}.\n :param CPE cpe: A candidate CPE Name X.\n :returns: True if X matches K, otherwise False.\n :rtype: boolean\n \"\"\"\n\n for N in self.K:\n if CPESet2_3.cpe_superset(wfn, N):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if the CPE Name is valid.", "response": "def _parse(self):\n \"\"\"\n Checks if the CPE Name is valid.\n\n :returns: None\n :exception: ValueError - bad-formed CPE Name\n \"\"\"\n\n # Check prefix and initial bracket of WFN\n if self._str[0:5] != CPE2_3_WFN.CPE_PREFIX:\n errmsg = \"Bad-formed CPE Name: WFN prefix not found\"\n raise ValueError(errmsg)\n\n # Check final backet\n if self._str[-1:] != \"]\":\n errmsg = \"Bad-formed CPE Name: final bracket of WFN not found\"\n raise ValueError(errmsg)\n\n content = self._str[5:-1]\n\n if content != \"\":\n # Dictionary with pairs attribute-value\n components = dict()\n\n # Split WFN in components\n list_component = content.split(CPEComponent2_3_WFN.SEPARATOR_COMP)\n\n # Adds the defined components\n for e in list_component:\n # Whitespace not valid in component names and values\n if e.find(\" \") != -1:\n msg = \"Bad-formed CPE Name: WFN with too many whitespaces\"\n raise ValueError(msg)\n\n # Split pair attribute-value\n pair = e.split(CPEComponent2_3_WFN.SEPARATOR_PAIR)\n att_name = pair[0]\n att_value = pair[1]\n\n # Check valid attribute name\n if att_name not in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n msg = \"Bad-formed CPE Name: invalid attribute name '{0}'\".format(\n att_name)\n raise ValueError(msg)\n\n if att_name in components:\n # Duplicate attribute\n msg = \"Bad-formed CPE Name: attribute '{0}' repeated\".format(\n att_name)\n raise ValueError(msg)\n\n if not (att_value.startswith('\"') and\n att_value.endswith('\"')):\n\n # Logical value\n strUpper = att_value.upper()\n if strUpper == CPEComponent2_3_WFN.VALUE_ANY:\n comp = CPEComponentAnyValue()\n elif strUpper == CPEComponent2_3_WFN.VALUE_NA:\n comp = CPEComponentNotApplicable()\n else:\n msg = \"Invalid logical value '{0}'\".format(att_value)\n raise ValueError(msg)\n\n elif att_value.startswith('\"') and att_value.endswith('\"'):\n # String value\n comp = CPEComponent2_3_WFN(att_value, att_name)\n\n else:\n # Bad value\n msg = \"Bad-formed CPE Name: invalid value '{0}'\".format(\n att_value)\n raise ValueError(msg)\n\n components[att_name] = comp\n\n # Adds the undefined components\n for ck in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n if ck not in components:\n components[ck] = CPEComponentUndefined()\n\n # #######################\n # Storage of CPE Name #\n # #######################\n\n part_comp = components[CPEComponent.ATT_PART]\n if isinstance(part_comp, CPEComponentLogical):\n elements = []\n elements.append(components)\n self[CPE.KEY_UNDEFINED] = elements\n else:\n # Create internal structure of CPE Name in parts:\n # one of them is filled with identified components,\n # the rest are empty\n part_value = part_comp.get_value()\n # Del double quotes of value\n system = part_value[1:-1]\n if system in CPEComponent.SYSTEM_VALUES:\n self._create_cpe_parts(system, components)\n else:\n self._create_cpe_parts(CPEComponent.VALUE_PART_UNDEFINED,\n components)\n\n # Fills the empty parts of internal structure of CPE Name\n for pk in CPE.CPE_PART_KEYS:\n if pk not in self.keys():\n self[pk] = []"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef language_match(self, cpeset, cpel_dom=None):\n\n # Root element tag\n TAG_ROOT = '#document'\n # A container for child platform definitions\n TAG_PLATSPEC = 'cpe:platform-specification'\n\n # Information about a platform definition\n TAG_PLATFORM = 'cpe:platform'\n TAG_LOGITEST = 'cpe:logical-test'\n TAG_CPE = 'cpe:fact-ref'\n\n # Tag attributes\n ATT_NAME = 'name'\n ATT_OP = 'operator'\n ATT_NEGATE = 'negate'\n\n # Attribute values\n ATT_OP_AND = 'AND'\n ATT_OP_OR = 'OR'\n ATT_NEGATE_TRUE = 'TRUE'\n\n if cpel_dom is None:\n cpel_dom = self.document\n\n # Identify the root element\n if cpel_dom.nodeName == TAG_ROOT or cpel_dom.nodeName == TAG_PLATSPEC:\n for node in cpel_dom.childNodes:\n if node.nodeName == TAG_PLATSPEC:\n return self.language_match(cpeset, node)\n if node.nodeName == TAG_PLATFORM:\n return self.language_match(cpeset, node)\n\n # Identify a platform element\n elif cpel_dom.nodeName == TAG_PLATFORM:\n for node in cpel_dom.childNodes:\n if node.nodeName == TAG_LOGITEST:\n return self.language_match(cpeset, node)\n\n # Identify a CPE element\n elif cpel_dom.nodeName == TAG_CPE:\n cpename = cpel_dom.getAttribute(ATT_NAME)\n c = CPE2_2(cpename)\n\n # Try to match a CPE name with CPE set\n return cpeset.name_match(c)\n\n # Identify a logical operator element\n elif cpel_dom.nodeName == TAG_LOGITEST:\n count = 0\n len = 0\n answer = False\n\n for node in cpel_dom.childNodes:\n if node.nodeName.find(\"#\") == 0:\n continue\n len = len + 1\n if self.language_match(cpeset, node):\n count = count + 1\n\n operator = cpel_dom.getAttribute(ATT_OP).upper()\n\n if operator == ATT_OP_AND:\n if count == len:\n answer = True\n elif operator == ATT_OP_OR:\n if count > 0:\n answer = True\n\n operator_not = cpel_dom.getAttribute(ATT_NEGATE)\n if operator_not:\n if operator_not.upper() == ATT_NEGATE_TRUE:\n answer = not answer\n\n return answer\n else:\n return False", "response": "Returns True if the expression matches with the set False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a CPE Name to the set if not already.", "response": "def append(self, cpe):\n \"\"\"\n Adds a CPE Name to the set if not already.\n\n :param CPE cpe: CPE Name to store in set\n :returns: None\n :exception: ValueError - invalid version of CPE Name\n\n\n TEST:\n\n >>> from .cpeset2_2 import CPESet2_2\n >>> from .cpe2_2 import CPE2_2\n >>> uri1 = 'cpe:/h:hp'\n >>> c1 = CPE2_2(uri1)\n >>> s = CPESet2_2()\n >>> s.append(c1)\n \"\"\"\n\n if cpe.VERSION != CPE.VERSION_2_2:\n errmsg = \"CPE Name version {0} not valid, version 2.2 expected\".format(\n cpe.VERSION)\n raise ValueError(errmsg)\n\n for k in self.K:\n if cpe.cpe_str == k.cpe_str:\n return None\n\n self.K.append(cpe)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_value(self, comp_str, comp_att):\n\n # Del double quotes of value\n str = comp_str[1:-1]\n self._standard_value = str\n\n # Parse the value\n super(CPEComponent2_3_WFN, self).set_value(str, comp_att)", "response": "Set the value of component."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a CPE Name to the set if not already.", "response": "def append(self, cpe):\n \"\"\"\n Adds a CPE Name to the set if not already.\n\n :param CPE cpe: CPE Name to store in set\n :returns: None\n :exception: ValueError - invalid version of CPE Name\n\n TEST:\n\n >>> from .cpeset1_1 import CPESet1_1\n >>> from .cpe1_1 import CPE1_1\n >>> uri1 = 'cpe://microsoft:windows:xp!vista'\n >>> c1 = CPE1_1(uri1)\n >>> s = CPESet1_1()\n >>> s.append(c1)\n \"\"\"\n\n if cpe.VERSION != CPE.VERSION_1_1:\n msg = \"CPE Name version {0} not valid, version 1.1 expected\".format(\n cpe.VERSION)\n raise ValueError(msg)\n\n for k in self.K:\n if cpe.cpe_str == k.cpe_str:\n return None\n\n self.K.append(cpe)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntest if a set of known instances of a candidate CPE Name X matches a set of known instances of a candidate CPE Name X and returns True if the candidate can be shown to be otherwise False.", "response": "def name_match(self, cpe):\n \"\"\"\n Accepts a set of known instances of CPE Names and a candidate CPE Name,\n and returns 'True' if the candidate can be shown to be\n an instance based on the content of the known instances.\n Otherwise, it returns 'False'.\n\n :param CPESet self: A set of m known CPE Names K = {K1, K2, \u2026, Km}.\n :param CPE cpe: A candidate CPE Name X.\n :returns: True if X matches K, otherwise False.\n :rtype: boolean\n\n TEST: matching with identical CPE in set\n\n >>> from .cpe1_1 import CPE1_1\n >>> from .cpeset1_1 import CPESet1_1\n >>> uri1 = 'cpe://microsoft:windows:xp!vista'\n >>> uri2 = 'cpe:/cisco::3825;cisco:2:44/cisco:ios:12.3:enterprise'\n >>> c1 = CPE1_1(uri1)\n >>> c2 = CPE1_1(uri2)\n >>> s = CPESet1_1()\n >>> s.append(c1)\n >>> s.append(c2)\n >>> s.name_match(c2)\n True\n\n \"\"\"\n\n # An empty set not matching with any CPE\n if len(self) == 0:\n return False\n\n # If input CPE Name string is in set of CPE Name strings\n # not do searching more because there is a matching\n for k in self.K:\n if (k.cpe_str == cpe.cpe_str):\n return True\n\n # There are not a CPE Name string in set equal to\n # input CPE Name string\n match = False\n\n for p in CPE.CPE_PART_KEYS:\n elems_cpe = cpe.get(p)\n for ec in elems_cpe:\n # Search of element of part of input CPE\n\n # Each element ec of input cpe[p] is compared with\n # each element ek of k[p] in set K\n\n for k in self.K:\n elems_k = k.get(p)\n\n for ek in elems_k:\n # Matching\n\n # Each component in element ec is compared with\n # each component in element ek\n for ck in CPEComponent.CPE_COMP_KEYS:\n comp_cpe = ec.get(ck)\n comp_k = ek.get(ck)\n\n match = comp_k in comp_cpe\n\n if not match:\n # Search compoment in another element ek[p]\n break\n\n # Component analyzed\n\n if match:\n # Element matched\n break\n if match:\n break\n # Next element in part in \"cpe\"\n\n if not match:\n # cpe part not match with parts in set\n return False\n\n # Next part in input CPE Name\n\n # All parts in input CPE Name matched\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse(self):\n\n # CPE Name must not have whitespaces\n if (self._str.find(\" \") != -1):\n msg = \"Bad-formed CPE Name: it must not have whitespaces\"\n raise ValueError(msg)\n\n # Partitioning of CPE Name\n parts_match = CPE2_3_FS._parts_rxc.match(self._str)\n\n # Validation of CPE Name parts\n if (parts_match is None):\n msg = \"Bad-formed CPE Name: validation of parts failed\"\n raise ValueError(msg)\n\n components = dict()\n parts_match_dict = parts_match.groupdict()\n\n for ck in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n if ck in parts_match_dict:\n value = parts_match.group(ck)\n\n if (value == CPEComponent2_3_FS.VALUE_ANY):\n comp = CPEComponentAnyValue()\n elif (value == CPEComponent2_3_FS.VALUE_NA):\n comp = CPEComponentNotApplicable()\n else:\n try:\n comp = CPEComponent2_3_FS(value, ck)\n except ValueError:\n errmsg = \"Bad-formed CPE Name: not correct value: {0}\".format(\n value)\n raise ValueError(errmsg)\n else:\n errmsg = \"Component {0} should be specified\".format(ck)\n raise ValueError(ck)\n\n components[ck] = comp\n\n # #######################\n # Storage of CPE Name #\n # #######################\n\n part_comp = components[CPEComponent.ATT_PART]\n if isinstance(part_comp, CPEComponentLogical):\n elements = []\n elements.append(components)\n self[CPE.KEY_UNDEFINED] = elements\n else:\n # Create internal structure of CPE Name in parts:\n # one of them is filled with identified components,\n # the rest are empty\n system = parts_match.group(CPEComponent.ATT_PART)\n if system in CPEComponent.SYSTEM_VALUES:\n self._create_cpe_parts(system, components)\n else:\n self._create_cpe_parts(CPEComponent.VALUE_PART_UNDEFINED,\n components)\n\n # Fills the empty parts of internal structure of CPE Name\n for pk in CPE.CPE_PART_KEYS:\n if pk not in self.keys():\n # Empty part\n self[pk] = []", "response": "Checks if the CPE Name is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the values of attribute att_name of CPE Name. By default a only element in each part.", "response": "def get_attribute_values(self, att_name):\n \"\"\"\n Returns the values of attribute \"att_name\" of CPE Name.\n By default a only element in each part.\n\n :param string att_name: Attribute name to get\n :returns: List of attribute values\n :rtype: list\n :exception: ValueError - invalid attribute name\n \"\"\"\n\n lc = []\n\n if not CPEComponent.is_valid_attribute(att_name):\n errmsg = \"Invalid attribute name: {0}\".format(att_name)\n raise ValueError(errmsg)\n\n for pk in CPE.CPE_PART_KEYS:\n elements = self.get(pk)\n for elem in elements:\n comp = elem.get(att_name)\n\n if isinstance(comp, CPEComponentAnyValue):\n value = CPEComponent2_3_FS.VALUE_ANY\n\n elif isinstance(comp, CPEComponentNotApplicable):\n value = CPEComponent2_3_FS.VALUE_NA\n\n else:\n value = comp.get_value()\n\n lc.append(value)\n return lc"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if the CPE Name is valid.", "response": "def _parse(self):\n \"\"\"\n Checks if CPE Name is valid.\n\n :returns: None\n :exception: ValueError - bad-formed CPE Name\n \"\"\"\n\n # CPE Name must not have whitespaces\n if (self._str.find(\" \") != -1):\n msg = \"Bad-formed CPE Name: it must not have whitespaces\"\n raise ValueError(msg)\n\n # Partitioning of CPE Name\n parts_match = CPE2_2._parts_rxc.match(self._str)\n\n # Validation of CPE Name parts\n if (parts_match is None):\n msg = \"Bad-formed CPE Name: validation of parts failed\"\n raise ValueError(msg)\n\n components = dict()\n parts_match_dict = parts_match.groupdict()\n\n for ck in CPEComponent.CPE_COMP_KEYS:\n if ck in parts_match_dict:\n value = parts_match.group(ck)\n\n if (value == CPEComponent2_2.VALUE_UNDEFINED):\n comp = CPEComponentUndefined()\n elif (value == CPEComponent2_2.VALUE_EMPTY):\n comp = CPEComponentEmpty()\n else:\n try:\n comp = CPEComponent2_2(value, ck)\n except ValueError:\n errmsg = \"Bad-formed CPE Name: not correct value: {0}\".format(\n value)\n raise ValueError(errmsg)\n else:\n # Component not exist in this version of CPE\n comp = CPEComponentUndefined()\n\n components[ck] = comp\n\n # Adds the components of version 2.3 of CPE not defined in version 2.2\n for ck2 in CPEComponent.CPE_COMP_KEYS_EXTENDED:\n if ck2 not in components.keys():\n components[ck2] = CPEComponentUndefined()\n\n # #######################\n # Storage of CPE Name #\n # #######################\n\n # If part component is undefined, store it in the part without name\n if components[CPEComponent.ATT_PART] == CPEComponentUndefined():\n system = CPEComponent.VALUE_PART_UNDEFINED\n else:\n system = parts_match.group(CPEComponent.ATT_PART)\n\n self._create_cpe_parts(system, components)\n\n # Adds the undefined parts\n for sys in CPEComponent.SYSTEM_VALUES:\n if sys != system:\n pk = CPE._system_and_parts[sys]\n self[pk] = []"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the CPE Name as WFN string of version 2. 3.", "response": "def as_wfn(self):\n \"\"\"\n Returns the CPE Name as WFN string of version 2.3.\n Only shows the first seven components.\n\n :return: CPE Name as WFN string\n :rtype: string\n :exception: TypeError - incompatible version\n \"\"\"\n\n wfn = []\n wfn.append(CPE2_3_WFN.CPE_PREFIX)\n\n for ck in CPEComponent.CPE_COMP_KEYS:\n lc = self._get_attribute_components(ck)\n\n comp = lc[0]\n\n if (isinstance(comp, CPEComponentUndefined) or\n isinstance(comp, CPEComponentEmpty)):\n\n # Do not set the attribute\n continue\n else:\n v = []\n v.append(ck)\n v.append(\"=\")\n\n # Get the value of WFN of component\n v.append('\"')\n v.append(comp.as_wfn())\n v.append('\"')\n\n # Append v to the WFN and add a separator\n wfn.append(\"\".join(v))\n wfn.append(CPEComponent2_3_WFN.SEPARATOR_COMP)\n\n # Del the last separator\n wfn = wfn[:-1]\n\n # Return the WFN string\n wfn.append(CPE2_3_WFN.CPE_SUFFIX)\n\n return \"\".join(wfn)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if wfn is a proper superset of any of the names in cpeset False otherwise.", "response": "def _fact_ref_eval(cls, cpeset, wfn):\n \"\"\"\n Returns True if wfn is a non-proper superset (True superset\n or equal to) any of the names in cpeset, otherwise False.\n\n :param CPESet cpeset: list of CPE bound Names.\n :param CPE2_3_WFN wfn: WFN CPE Name.\n :returns: True if wfn is a non-proper superset any of the names in cpeset, otherwise False\n :rtype: boolean\n \"\"\"\n\n for n in cpeset:\n # Need to convert each n from bound form to WFN\n if (CPESet2_3.cpe_superset(wfn, n)):\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the result (True, False, Error) of performing the specified check, unless the check isn\u0092t supported, in which case it returns False. Error is a catch-all for all results other than True and False. :param string cpel_dom: XML infoset for the check_fact_ref element. :returns: result of performing the specified check :rtype: boolean or error", "response": "def _check_fact_ref_eval(cls, cpel_dom):\n \"\"\"\n Returns the result (True, False, Error) of performing the specified\n check, unless the check isn\u0092t supported, in which case it returns\n False. Error is a catch-all for all results other than True and\n False.\n\n :param string cpel_dom: XML infoset for the check_fact_ref element.\n :returns: result of performing the specified check\n :rtype: boolean or error\n \"\"\"\n\n CHECK_SYSTEM = \"check-system\"\n CHECK_LOCATION = \"check-location\"\n CHECK_ID = \"check-id\"\n\n checksystemID = cpel_dom.getAttribute(CHECK_SYSTEM)\n if (checksystemID == \"http://oval.mitre.org/XMLSchema/ovaldefinitions-5\"):\n # Perform an OVAL check.\n # First attribute is the URI of an OVAL definitions file.\n # Second attribute is an OVAL definition ID.\n return CPELanguage2_3._ovalcheck(cpel_dom.getAttribute(CHECK_LOCATION),\n cpel_dom.getAttribute(CHECK_ID))\n\n if (checksystemID == \"http://scap.nist.gov/schema/ocil/2\"):\n # Perform an OCIL check.\n # First attribute is the URI of an OCIL questionnaire file.\n # Second attribute is OCIL questionnaire ID.\n return CPELanguage2_3._ocilcheck(cpel_dom.getAttribute(CHECK_LOCATION),\n cpel_dom.getAttribute(CHECK_ID))\n\n # Can add additional check systems here, with each returning a\n # True, False, or Error value\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nunbind a bound form to a WFN object.", "response": "def _unbind(cls, boundname):\n \"\"\"\n Unbinds a bound form to a WFN.\n\n :param string boundname: CPE name\n :returns: WFN object associated with boundname.\n :rtype: CPE2_3_WFN\n \"\"\"\n\n try:\n fs = CPE2_3_FS(boundname)\n except:\n # CPE name is not formatted string\n try:\n uri = CPE2_3_URI(boundname)\n except:\n # CPE name is not URI but WFN\n return CPE2_3_WFN(boundname)\n else:\n return CPE2_3_WFN(uri.as_wfn())\n else:\n return CPE2_3_WFN(fs.as_wfn())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning True if the expression can be satisfied by the language matching with the set.", "response": "def language_match(self, cpeset, cpel_dom=None):\n \"\"\"\n Accepts a set of known CPE Names and an expression in the CPE language,\n and delivers the answer True if the expression matches with the set.\n Otherwise, it returns False.\n\n :param CPELanguage self: An expression in the CPE Applicability\n Language, represented as the XML infoset for the platform element.\n :param CPESet cpeset: CPE set object to match with self expression.\n :param string cpel_dom: An expression in the CPE Applicability\n Language, represented as DOM tree.\n :returns: True if self expression can be satisfied by language matching\n against cpeset, False otherwise.\n :rtype: boolean\n \"\"\"\n\n # Root element tag\n TAG_ROOT = '#document'\n # A container for child platform definitions\n TAG_PLATSPEC = 'cpe:platform-specification'\n\n # Information about a platform definition\n TAG_PLATFORM = 'cpe:platform'\n TAG_LOGITEST = 'cpe:logical-test'\n TAG_CPE = 'cpe:fact-ref'\n TAG_CHECK_CPE = 'check-fact-ref'\n\n # Tag attributes\n ATT_NAME = 'name'\n ATT_OP = 'operator'\n ATT_NEGATE = 'negate'\n\n # Attribute values\n ATT_OP_AND = 'AND'\n ATT_OP_OR = 'OR'\n ATT_NEGATE_TRUE = 'TRUE'\n\n # Constant associated with an error in language matching\n ERROR = 2\n\n if cpel_dom is None:\n cpel_dom = self.document\n\n # Identify the root element\n if cpel_dom.nodeName == TAG_ROOT or cpel_dom.nodeName == TAG_PLATSPEC:\n for node in cpel_dom.childNodes:\n if node.nodeName == TAG_PLATSPEC:\n return self.language_match(cpeset, node)\n if node.nodeName == TAG_PLATFORM:\n return self.language_match(cpeset, node)\n\n # Identify a platform element\n elif cpel_dom.nodeName == TAG_PLATFORM:\n # Parse through E's elements and ignore all but logical-test\n for node in cpel_dom.childNodes:\n if node.nodeName == TAG_LOGITEST:\n # Call the function again, but with logical-test\n # as the root element\n return self.language_match(cpeset, node)\n\n # Identify a CPE element\n elif cpel_dom.nodeName == TAG_CPE:\n # fact-ref's name attribute is a bound name,\n # so we unbind it to a WFN before passing it\n cpename = cpel_dom.getAttribute(ATT_NAME)\n wfn = CPELanguage2_3._unbind(cpename)\n return CPELanguage2_3._fact_ref_eval(cpeset, wfn)\n\n # Identify a check of CPE names (OVAL, OCIL...)\n elif cpel_dom.nodeName == TAG_CHECK_CPE:\n return CPELanguage2_3._check_fact_ref_Eval(cpel_dom)\n\n # Identify a logical operator element\n elif cpel_dom.nodeName == TAG_LOGITEST:\n count = 0\n len = 0\n answer = False\n\n for node in cpel_dom.childNodes:\n if node.nodeName.find(\"#\") == 0:\n continue\n len = len + 1\n result = self.language_match(cpeset, node)\n if result:\n count = count + 1\n elif result == ERROR:\n answer = ERROR\n\n operator = cpel_dom.getAttribute(ATT_OP).upper()\n\n if operator == ATT_OP_AND:\n if count == len:\n answer = True\n elif operator == ATT_OP_OR:\n if count > 0:\n answer = True\n\n operator_not = cpel_dom.getAttribute(ATT_NEGATE)\n if operator_not:\n if ((operator_not.upper() == ATT_NEGATE_TRUE) and\n (answer != ERROR)):\n answer = not answer\n\n return answer\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef name_match(self, cpe):\n\n # An empty set not matching with any CPE\n if len(self) == 0:\n return False\n\n # If input CPE Name string is in set of CPE Name strings\n # not do searching more because there is a matching\n for k in self.K:\n if (k.cpe_str == cpe.cpe_str):\n return True\n\n # If \"cpe\" is an empty CPE Name any system matches\n if len(cpe) == 0:\n return True\n\n # There are not a CPE Name string in set equal to\n # input CPE Name string\n match = False\n\n for p in CPE.CPE_PART_KEYS:\n elems_cpe = cpe.get(p)\n for ec in elems_cpe:\n # Search of element of part of input CPE\n\n # Each element ec of input cpe[p] is compared with\n # each element ek of k[p] in set K\n\n for k in self.K:\n if (len(k) >= len(cpe)):\n elems_k = k.get(p)\n\n for ek in elems_k:\n # Matching\n\n # Each component in element ec is compared with\n # each component in element ek\n for c in range(0, len(cpe)):\n key = CPEComponent.ordered_comp_parts[c]\n comp_cpe = ec.get(key)\n comp_k = ek.get(key)\n match = comp_k in comp_cpe\n\n if not match:\n # Search compoment in another element ek[p]\n break\n\n # Component analyzed\n\n if match:\n # Element matched\n break\n if match:\n break\n # Next element in part in \"cpe\"\n\n if not match:\n # cpe part not match with parts in set\n return False\n\n # Next part in input CPE Name\n\n # All parts in input CPE Name matched\n return True", "response": "Returns True if the set of known instances of a candidate CPE Name X matches the set of known instances of a candidate CPE Name X."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndecode the encoded value of component into a standard value.", "response": "def _decode(self):\n \"\"\"\n Convert the encoded value of component to standard value (WFN value).\n \"\"\"\n\n result = []\n idx = 0\n s = self._encoded_value\n\n while (idx < len(s)):\n # Get the idx'th character of s\n c = s[idx]\n\n if (c in CPEComponent2_2.NON_STANDARD_VALUES):\n # Escape character\n result.append(\"\\\\\")\n result.append(c)\n else:\n # Do nothing\n result.append(c)\n\n idx += 1\n\n self._standard_value = \"\".join(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if the set of ids form a single connected component. Returns True if the set of ids form a single connected component False otherwise", "response": "def is_component(w, ids):\n \"\"\"Check if the set of ids form a single connected component\n\n Parameters\n ----------\n\n w : spatial weights boject\n\n ids : list\n identifiers of units that are tested to be a single connected\n component\n\n\n Returns\n -------\n\n True : if the list of ids represents a single connected component\n\n False : if the list of ids forms more than a single connected component\n\n \"\"\"\n\n components = 0\n marks = dict([(node, 0) for node in ids])\n q = []\n for node in ids:\n if marks[node] == 0:\n components += 1\n q.append(node)\n if components > 1:\n return False\n while q:\n node = q.pop()\n marks[node] = components\n others = [neighbor for neighbor in w.neighbors[node]\n if neighbor in ids]\n for other in others:\n if marks[other] == 0 and other not in q:\n q.append(other)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if a node is contiguity based on a set of neighbors.", "response": "def check_contiguity(w, neighbors, leaver):\n \"\"\"Check if contiguity is maintained if leaver is removed from neighbors\n\n\n Parameters\n ----------\n\n w : spatial weights object\n simple contiguity based weights\n neighbors : list\n nodes that are to be checked if they form a single \\\n connected component\n leaver : id\n a member of neighbors to check for removal\n\n\n Returns\n -------\n\n True : if removing leaver from neighbors does not break contiguity\n of remaining set\n in neighbors\n False : if removing leaver from neighbors breaks contiguity\n\n Example\n -------\n\n Setup imports and a 25x25 spatial weights matrix on a 5x5 square region.\n\n >>> import libpysal as lps\n >>> w = lps.weights.lat2W(5, 5)\n\n Test removing various areas from a subset of the region's areas. In the\n first case the subset is defined as observations 0, 1, 2, 3 and 4. The\n test shows that observations 0, 1, 2 and 3 remain connected even if\n observation 4 is removed.\n\n >>> check_contiguity(w,[0,1,2,3,4],4)\n True\n >>> check_contiguity(w,[0,1,2,3,4],3)\n False\n >>> check_contiguity(w,[0,1,2,3,4],0)\n True\n >>> check_contiguity(w,[0,1,2,3,4],1)\n False\n >>>\n \"\"\"\n\n ids = neighbors[:]\n ids.remove(leaver)\n return is_component(w, ids)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nasking user a yes or no question and return their response as a boolean.", "response": "def confirm(question, assume_yes=True):\n \"\"\"\n Ask user a yes/no question and return their response as a boolean.\n\n ``question`` should be a simple, grammatically complete question such as\n \"Do you wish to continue?\", and will have a string similar to ``\" [Y/n] \"``\n appended automatically. This function will *not* append a question mark for\n you.\n\n By default, when the user presses Enter without typing anything, \"yes\" is\n assumed. This can be changed by specifying ``affirmative=False``.\n\n .. note::\n If the user does not supplies input that is (case-insensitively) equal\n to \"y\", \"yes\", \"n\" or \"no\", they will be re-prompted until they do.\n\n :param str question: The question part of the prompt.\n :param bool assume_yes:\n Whether to assume the affirmative answer by default. Default value:\n ``True``.\n\n :returns: A `bool`.\n \"\"\"\n # Set up suffix\n if assume_yes:\n suffix = \"Y/n\"\n else:\n suffix = \"y/N\"\n # Loop till we get something we like\n # TODO: maybe don't do this? It can be annoying. Turn into 'q'-for-quit?\n while True:\n # TODO: ensure that this is Ctrl-C friendly, ISTR issues with\n # raw_input/input on some Python versions blocking KeyboardInterrupt.\n response = input(\"{0} [{1}] \".format(question, suffix))\n response = response.lower().strip() # Normalize\n # Default\n if not response:\n return assume_yes\n # Yes\n if response in [\"y\", \"yes\"]:\n return True\n # No\n if response in [\"n\", \"no\"]:\n return False\n # Didn't get empty, yes or no, so complain and loop\n err = \"I didn't understand you. Please specify '(y)es' or '(n)o'.\"\n print(err, file=sys.stderr)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef jsonapi(f):\n @wraps(f)\n def wrapper(*args, **kwargs):\n rv = f(*args, **kwargs)\n return make_json_response(rv)\n return wrapper", "response": "Declare the view as a JSON API method"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _unpack(c, tmp, package, version, git_url=None):\n real_version = version[:]\n source = None\n if git_url:\n pass\n # git clone into tempdir\n # git checkout \n # set target to checkout\n # if version does not look SHA-ish:\n # in the checkout, obtain SHA from that branch\n # set real_version to that value\n else:\n cwd = os.getcwd()\n print(\"Moving into temp dir %s\" % tmp)\n os.chdir(tmp)\n try:\n # Nab from index. Skip wheels; we want to unpack an sdist.\n flags = \"--download=. --build=build --no-use-wheel\"\n cmd = \"pip install %s %s==%s\" % (flags, package, version)\n c.run(cmd)\n # Identify basename\n # TODO: glob is bad here because pip install --download gets all\n # dependencies too! ugh. Figure out best approach for that.\n globs = []\n globexpr = \"\"\n for extension, opener in (\n (\"zip\", \"unzip\"),\n (\"tgz\", \"tar xzvf\"),\n (\"tar.gz\", \"tar xzvf\"),\n ):\n globexpr = \"*.{0}\".format(extension)\n globs = glob(globexpr)\n if globs:\n break\n archive = os.path.basename(globs[0])\n source, _, _ = archive.rpartition(\".{0}\".format(extension))\n c.run(\"{0} {1}\".format(opener, globexpr))\n finally:\n os.chdir(cwd)\n return real_version, source", "response": "Download + unpack given package into temp dir tmp and return the real_version and source directory."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef vendorize(\n c,\n distribution,\n version,\n vendor_dir,\n package=None,\n git_url=None,\n license=None,\n):\n \"\"\"\n Vendorize Python package ``distribution`` at version/SHA ``version``.\n\n Specify the vendor folder (e.g. ``/vendor``) as ``vendor_dir``.\n\n For Crate/PyPI releases, ``package`` should be the name of the software\n entry on those sites, and ``version`` should be a specific version number.\n E.g. ``vendorize('lexicon', '0.1.2')``.\n\n For Git releases, ``package`` should be the name of the package folder\n within the checkout that needs to be vendorized and ``version`` should be a\n Git identifier (branch, tag, SHA etc.) ``git_url`` must also be given,\n something suitable for ``git clone ``.\n\n For SVN releases: xxx.\n\n For packages where the distribution name is not the same as the package\n directory name, give ``package='name'``.\n\n By default, no explicit license seeking is done -- we assume the license\n info is in file headers or otherwise within the Python package vendorized.\n This is not always true; specify ``license=/path/to/license/file`` to\n trigger copying of a license into the vendored folder from the\n checkout/download (relative to its root.)\n \"\"\"\n with tmpdir() as tmp:\n package = package or distribution\n target = os.path.join(vendor_dir, package)\n # Unpack source\n real_version, source = _unpack(c, tmp, distribution, version, git_url)\n abs_source = os.path.join(tmp, source)\n source_package = os.path.join(abs_source, package)\n # Ensure source package exists\n if not os.path.exists(source_package):\n rel_package = os.path.join(source, package)\n raise ValueError(\"Source package %s doesn't exist!\" % rel_package)\n # Nuke target if exists\n if os.path.exists(target):\n print(\"Removing pre-existing vendorized folder %s\" % target)\n rmtree(target)\n # Perform the copy\n print(\"Copying %s => %s\" % (source_package, target))\n copytree(source_package, target)\n # Explicit license if needed\n if license:\n copy(os.path.join(abs_source, license), target)", "response": "Vendorize a Python package into a specific version of a Crate or PyPI package."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a passworded sudo - capable user.", "response": "def make_sudouser(c):\n \"\"\"\n Create a passworded sudo-capable user.\n\n Used by other tasks to execute the test suite so sudo tests work.\n \"\"\"\n user = c.travis.sudo.user\n password = c.travis.sudo.password\n # --create-home because we need a place to put conf files, keys etc\n # --groups travis because we must be in the Travis group to access the\n # (created by Travis for us) virtualenv and other contents within\n # /home/travis.\n c.sudo(\"useradd {0} --create-home --groups travis\".format(user))\n # Password 'mypass' also arbitrary\n c.run(\"echo {0}:{1} | sudo chpasswd\".format(user, password))\n # Set up new (glob-sourced) sudoers conf file for our user; easier than\n # attempting to mutate or overwrite main sudoers conf.\n conf = \"/etc/sudoers.d/passworded\"\n cmd = \"echo '{0} ALL=(ALL:ALL) PASSWD:ALL' > {1}\".format(user, conf)\n c.sudo('sh -c \"{0}\"'.format(cmd))\n # Grant travis group write access to /home/travis as some integration tests\n # may try writing conf files there. (TODO: shouldn't running the tests via\n # 'sudo -H' mean that's no longer necessary?)\n c.sudo(\"chmod g+w /home/travis\")"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef make_sshable(c):\n user = c.travis.sudo.user\n home = \"~{0}\".format(user)\n # Run sudo() as the new sudo user; means less chown'ing, etc.\n c.config.sudo.user = user\n ssh_dir = \"{0}/.ssh\".format(home)\n # TODO: worth wrapping in 'sh -c' and using '&&' instead of doing this?\n for cmd in (\"mkdir {0}\", \"chmod 0700 {0}\"):\n c.sudo(cmd.format(ssh_dir, user))\n c.sudo('ssh-keygen -f {0}/id_rsa -N \"\"'.format(ssh_dir))\n c.sudo(\"cp {0}/{{id_rsa.pub,authorized_keys}}\".format(ssh_dir))", "response": "Create a new SSH directory and authorized_hosts for the current user."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun some command under Travis - oriented sudo subshell.", "response": "def sudo_run(c, command):\n \"\"\"\n Run some command under Travis-oriented sudo subshell/virtualenv.\n\n :param str command:\n Command string to run, e.g. ``inv coverage``, ``inv integration``, etc.\n (Does not necessarily need to be an Invoke task, but...)\n \"\"\"\n # NOTE: explicit shell wrapper because sourcing the venv works best here;\n # test tasks currently use their own subshell to call e.g. 'pytest --blah',\n # so the tactic of '$VIRTUAL_ENV/bin/inv coverage' doesn't help - only that\n # intermediate process knows about the venv!\n cmd = \"source $VIRTUAL_ENV/bin/activate && {}\".format(command)\n c.sudo('bash -c \"{0}\"'.format(cmd), user=c.travis.sudo.user)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef blacken(c):\n if not PYTHON.startswith(\"3.6\"):\n msg = \"Not blackening, since Python {} != Python 3.6\".format(PYTHON)\n print(msg, file=sys.stderr)\n return\n # Install, allowing config override of hardcoded default version\n config = c.config.get(\"travis\", {}).get(\"black\", {})\n version = config.get(\"version\", \"18.5b0\")\n c.run(\"pip install black=={}\".format(version))\n # Execute our blacken task, with diff + check, which will both error\n # and emit diffs.\n checks.blacken(c, check=True, diff=True)", "response": "Blacken the base node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the concordance binary tree for a given set of nodes and their associated values.", "response": "def _calc(self, x, y):\n \"\"\"\n List based implementation of binary tree algorithm for concordance\n measure after :cite:`Christensen2005`.\n\n \"\"\"\n x = np.array(x)\n y = np.array(y)\n n = len(y)\n perm = list(range(n))\n perm.sort(key=lambda a: (x[a], y[a]))\n vals = y[perm]\n ExtraY = 0\n ExtraX = 0\n ACount = 0\n BCount = 0\n CCount = 0\n DCount = 0\n ECount = 0\n DCount = 0\n Concordant = 0\n Discordant = 0\n # ids for left child\n li = [None] * (n - 1)\n # ids for right child\n ri = [None] * (n - 1)\n # number of left descendants for a node\n ld = np.zeros(n)\n # number of values equal to value i\n nequal = np.zeros(n)\n\n for i in range(1, n):\n NumBefore = 0\n NumEqual = 1\n root = 0\n x0 = x[perm[i - 1]]\n y0 = y[perm[i - 1]]\n x1 = x[perm[i]]\n y1 = y[perm[i]]\n if x0 != x1:\n DCount = 0\n ECount = 1\n else:\n if y0 == y1:\n ECount += 1\n else:\n DCount += ECount\n ECount = 1\n root = 0\n inserting = True\n while inserting:\n current = y[perm[i]]\n if current > y[perm[root]]:\n # right branch\n NumBefore += 1 + ld[root] + nequal[root]\n if ri[root] is None:\n # insert as right child to root\n ri[root] = i\n inserting = False\n else:\n root = ri[root]\n elif current < y[perm[root]]:\n # increment number of left descendants\n ld[root] += 1\n if li[root] is None:\n # insert as left child to root\n li[root] = i\n inserting = False\n else:\n root = li[root]\n elif current == y[perm[root]]:\n NumBefore += ld[root]\n NumEqual += nequal[root] + 1\n nequal[root] += 1\n inserting = False\n\n ACount = NumBefore - DCount\n BCount = NumEqual - ECount\n CCount = i - (ACount + BCount + DCount + ECount - 1)\n ExtraY += DCount\n ExtraX += BCount\n Concordant += ACount\n Discordant += CCount\n\n cd = Concordant + Discordant\n num = Concordant - Discordant\n tau = num / np.sqrt((cd + ExtraX) * (cd + ExtraY))\n v = (4. * n + 10) / (9. * n * (n - 1))\n z = tau / np.sqrt(v)\n pval = erfc(np.abs(z) / 1.4142136) # follow scipy\n return tau, pval, Concordant, Discordant, ExtraX, ExtraY"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwraps function to decorate a function", "response": "def decorator(self, func):\n \"\"\" Wrapper function to decorate a function \"\"\"\n if inspect.isfunction(func):\n func._methodview = self\n elif inspect.ismethod(func):\n func.__func__._methodview = self\n else:\n raise AssertionError('Can only decorate function and methods, {} given'.format(func))\n return func"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef matches(self, verb, params):\n return (self.ifset is None or self.ifset <= params) and \\\n (self.ifnset is None or self.ifnset.isdisjoint(params)) and \\\n (self.methods is None or verb in self.methods)", "response": "Test if the method matches the provided set of arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _match_view(self, method, route_params):\n method = method.upper()\n route_params = frozenset(k for k, v in route_params.items() if v is not None)\n\n for view_name, info in self.methods_map[method].items():\n if info.matches(method, route_params):\n return getattr(self, view_name)\n else:\n return None", "response": "Detect a view matching the method and route_params."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nregistering the view with an URL rule", "response": "def route_as_view(cls, app, name, rules, *class_args, **class_kwargs):\n \"\"\" Register the view with an URL route\n :param app: Flask application\n :type app: flask.Flask|flask.Blueprint\n :param name: Unique view name\n :type name: str\n :param rules: List of route rules to use\n :type rules: Iterable[str|werkzeug.routing.Rule]\n :param class_args: Args to pass to object constructor\n :param class_kwargs: KwArgs to pass to object constructor\n :return: View callable\n :rtype: Callable\n \"\"\"\n view = super(MethodView, cls).as_view(name, *class_args, **class_kwargs)\n for rule in rules:\n app.add_url_rule(rule, view_func=view)\n return view"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the steady state probability vector for a regular Markov transition matrix P.", "response": "def steady_state(P):\n \"\"\"\n Calculates the steady state probability vector for a regular Markov\n transition matrix P.\n\n Parameters\n ----------\n P : array\n (k, k), an ergodic Markov transition probability matrix.\n\n Returns\n -------\n : array\n (k, ), steady state distribution.\n\n Examples\n --------\n Taken from :cite:`Kemeny1967`. Land of Oz example where the states are\n Rain, Nice and Snow, so there is 25 percent chance that if it\n rained in Oz today, it will snow tomorrow, while if it snowed today in\n Oz there is a 50 percent chance of snow again tomorrow and a 25\n percent chance of a nice day (nice, like when the witch with the monkeys\n is melting).\n\n >>> import numpy as np\n >>> from giddy.ergodic import steady_state\n >>> p=np.array([[.5, .25, .25],[.5,0,.5],[.25,.25,.5]])\n >>> steady_state(p)\n array([0.4, 0.2, 0.4])\n\n Thus, the long run distribution for Oz is to have 40 percent of the\n days classified as Rain, 20 percent as Nice, and 40 percent as Snow\n (states are mutually exclusive).\n\n \"\"\"\n\n v, d = la.eig(np.transpose(P))\n d = np.array(d)\n\n # for a regular P maximum eigenvalue will be 1\n mv = max(v)\n # find its position\n i = v.tolist().index(mv)\n\n row = abs(d[:, i])\n\n # normalize eigenvector corresponding to the eigenvalue 1\n return row / sum(row)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fmpt(P):\n\n P = np.matrix(P)\n k = P.shape[0]\n A = np.zeros_like(P)\n ss = steady_state(P).reshape(k, 1)\n for i in range(k):\n A[:, i] = ss\n A = A.transpose()\n I = np.identity(k)\n Z = la.inv(I - P + A)\n E = np.ones_like(Z)\n A_diag = np.diag(A)\n A_diag = A_diag + (A_diag == 0)\n D = np.diag(1. / A_diag)\n Zdg = np.diag(np.diag(Z))\n M = (I - Z + E * Zdg) * D\n return np.array(M)", "response": "Calculates the matrix of first mean passage times for an ergodic Markov transition probability matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _converge(c):\n #\n # Data/state gathering\n #\n\n # Get data about current repo context: what branch are we on & what kind of\n # release does it appear to represent?\n branch, release_type = _release_line(c)\n # Short-circuit if type is undefined; we can't do useful work for that.\n if release_type is Release.UNDEFINED:\n raise UndefinedReleaseType(\n \"You don't seem to be on a release-related branch; \"\n \"why are you trying to cut a release?\"\n )\n # Parse our changelog so we can tell what's released and what's not.\n # TODO: below needs to go in something doc-y somewhere; having it in a\n # non-user-facing subroutine docstring isn't visible enough.\n \"\"\"\n .. note::\n Requires that one sets the ``packaging.changelog_file`` configuration\n option; it should be a relative or absolute path to your\n ``changelog.rst`` (or whatever it's named in your project).\n \"\"\"\n # TODO: allow skipping changelog if not using Releases since we have no\n # other good way of detecting whether a changelog needs/got an update.\n # TODO: chdir to sphinx.source, import conf.py, look at\n # releases_changelog_name - that way it will honor that setting and we can\n # ditch this explicit one instead. (and the docstring above)\n changelog = parse_changelog(\n c.packaging.changelog_file, load_extensions=True\n )\n # Get latest appropriate changelog release and any unreleased issues, for\n # current line\n line_release, issues = _release_and_issues(changelog, branch, release_type)\n # Also get latest overall release, sometimes that matters (usually only\n # when latest *appropriate* release doesn't exist yet)\n overall_release = _versions_from_changelog(changelog)[-1]\n # Obtain the project's main package & its version data\n current_version = load_version(c)\n # Grab all git tags\n tags = _get_tags(c)\n\n state = Lexicon(\n {\n \"branch\": branch,\n \"release_type\": release_type,\n \"changelog\": changelog,\n \"latest_line_release\": Version(line_release)\n if line_release\n else None,\n \"latest_overall_release\": overall_release, # already a Version\n \"unreleased_issues\": issues,\n \"current_version\": Version(current_version),\n \"tags\": tags,\n }\n )\n # Version number determinations:\n # - latest actually-released version\n # - the next version after that for current branch\n # - which of the two is the actual version we're looking to converge on,\n # depends on current changelog state.\n latest_version, next_version = _latest_and_next_version(state)\n state.latest_version = latest_version\n state.next_version = next_version\n state.expected_version = latest_version\n if state.unreleased_issues:\n state.expected_version = next_version\n\n #\n # Logic determination / convergence\n #\n\n actions = Lexicon()\n\n # Changelog: needs new release entry if there are any unreleased issues for\n # current branch's line.\n # TODO: annotate with number of released issues [of each type?] - so not\n # just \"up to date!\" but \"all set (will release 3 features & 5 bugs)\"\n actions.changelog = Changelog.OKAY\n if release_type in (Release.BUGFIX, Release.FEATURE) and issues:\n actions.changelog = Changelog.NEEDS_RELEASE\n\n # Version file: simply whether version file equals the target version.\n # TODO: corner case of 'version file is >1 release in the future', but\n # that's still wrong, just would be a different 'bad' status output.\n actions.version = VersionFile.OKAY\n if state.current_version != state.expected_version:\n actions.version = VersionFile.NEEDS_BUMP\n\n # Git tag: similar to version file, except the check is existence of tag\n # instead of comparison to file contents. We even reuse the\n # 'expected_version' variable wholesale.\n actions.tag = Tag.OKAY\n if state.expected_version not in state.tags:\n actions.tag = Tag.NEEDS_CUTTING\n\n #\n # Return\n #\n\n return actions, state", "response": "Return the state of the current release."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef status(c):\n # TODO: wants some holistic \"you don't actually HAVE any changes to\n # release\" final status - i.e. all steps were at no-op status.\n actions, state = _converge(c)\n table = []\n # NOTE: explicit 'sensible' sort (in rough order of how things are usually\n # modified, and/or which depend on one another, e.g. tags are near the end)\n for component in \"changelog version tag\".split():\n table.append((component.capitalize(), actions[component].value))\n print(tabulate(table))\n return actions, state", "response": "Print current release version changelog tag etc."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef prepare(c):\n # Print dry-run/status/actions-to-take data & grab programmatic result\n # TODO: maybe expand the enum-based stuff to have values that split up\n # textual description, command string, etc. See the TODO up by their\n # definition too, re: just making them non-enum classes period.\n # TODO: otherwise, we at least want derived eg changelog/version/etc paths\n # transmitted from status() into here...\n actions, state = status(c)\n # TODO: unless nothing-to-do in which case just say that & exit 0\n if not confirm(\"Take the above actions?\"):\n sys.exit(\"Aborting.\")\n\n # TODO: factor out what it means to edit a file:\n # - $EDITOR or explicit expansion of it in case no shell involved\n # - pty=True and hide=False, because otherwise things can be bad\n # - what else?\n\n # Changelog! (pty for non shite editing, eg vim sure won't like non-pty)\n if actions.changelog is Changelog.NEEDS_RELEASE:\n # TODO: identify top of list and inject a ready-made line? Requires vim\n # assumption...GREAT opportunity for class/method based tasks!\n cmd = \"$EDITOR {0.packaging.changelog_file}\".format(c)\n c.run(cmd, pty=True, hide=False)\n # TODO: add a step for checking reqs.txt / setup.py vs virtualenv contents\n # Version file!\n if actions.version == VersionFile.NEEDS_BUMP:\n # TODO: suggest the bump and/or overwrite the entire file? Assumes a\n # specific file format. Could be bad for users which expose __version__\n # but have other contents as well.\n version_file = os.path.join(\n _find_package(c),\n c.packaging.get(\"version_module\", \"_version\") + \".py\",\n )\n cmd = \"$EDITOR {0}\".format(version_file)\n c.run(cmd, pty=True, hide=False)\n if actions.tag == Tag.NEEDS_CUTTING:\n # Commit, if necessary, so the tag includes everything.\n # NOTE: this strips out untracked files. effort.\n cmd = 'git status --porcelain | egrep -v \"^\\\\?\"'\n if c.run(cmd, hide=True, warn=True).ok:\n c.run(\n 'git commit -am \"Cut {0}\"'.format(state.expected_version),\n hide=False,\n )\n # Tag!\n c.run(\"git tag {0}\".format(state.expected_version), hide=False)", "response": "Prepare the contents of the a\n ."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexamining current repo state to determine what type of release to prep.", "response": "def _release_line(c):\n \"\"\"\n Examine current repo state to determine what type of release to prep.\n\n :returns:\n A two-tuple of ``(branch-name, line-type)`` where:\n\n - ``branch-name`` is the current branch name, e.g. ``1.1``, ``master``,\n ``gobbledygook`` (or, usually, ``HEAD`` if not on a branch).\n - ``line-type`` is a symbolic member of `.Release` representing what\n \"type\" of release the line appears to be for:\n\n - ``Release.BUGFIX`` if on a bugfix/stable release line, e.g.\n ``1.1``.\n - ``Release.FEATURE`` if on a feature-release branch (typically\n ``master``).\n - ``Release.UNDEFINED`` if neither of those appears to apply\n (usually means on some unmerged feature/dev branch).\n \"\"\"\n # TODO: I don't _think_ this technically overlaps with Releases (because\n # that only ever deals with changelog contents, and therefore full release\n # version numbers) but in case it does, move it there sometime.\n # TODO: this and similar calls in this module may want to be given an\n # explicit pointer-to-git-repo option (i.e. if run from outside project\n # context).\n # TODO: major releases? or are they big enough events we don't need to\n # bother with the script? Also just hard to gauge - when is master the next\n # 1.x feature vs 2.0?\n branch = c.run(\"git rev-parse --abbrev-ref HEAD\", hide=True).stdout.strip()\n type_ = Release.UNDEFINED\n if BUGFIX_RE.match(branch):\n type_ = Release.BUGFIX\n if FEATURE_RE.match(branch):\n type_ = Release.FEATURE\n return branch, type_"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns all released versions from given changelog dict sorted.", "response": "def _versions_from_changelog(changelog):\n \"\"\"\n Return all released versions from given ``changelog``, sorted.\n\n :param dict changelog:\n A changelog dict as returned by ``releases.util.parse_changelog``.\n\n :returns: A sorted list of `semantic_version.Version` objects.\n \"\"\"\n versions = [Version(x) for x in changelog if BUGFIX_RELEASE_RE.match(x)]\n return sorted(versions)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns most recent release and issue numbers for a given branch.", "response": "def _release_and_issues(changelog, branch, release_type):\n \"\"\"\n Return most recent branch-appropriate release, if any, and its contents.\n\n :param dict changelog:\n Changelog contents, as returned by ``releases.util.parse_changelog``.\n\n :param str branch:\n Branch name.\n\n :param release_type:\n Member of `Release`, e.g. `Release.FEATURE`.\n\n :returns:\n Two-tuple of release (``str``) and issues (``list`` of issue numbers.)\n\n If there is no latest release for the given branch (e.g. if it's a\n feature or master branch), it will be ``None``.\n \"\"\"\n # Bugfix lines just use the branch to find issues\n bucket = branch\n # Features need a bit more logic\n if release_type is Release.FEATURE:\n bucket = _latest_feature_bucket(changelog)\n # Issues is simply what's in the bucket\n issues = changelog[bucket]\n # Latest release is undefined for feature lines\n release = None\n # And requires scanning changelog, for bugfix lines\n if release_type is Release.BUGFIX:\n versions = [text_type(x) for x in _versions_from_changelog(changelog)]\n release = [x for x in versions if x.startswith(bucket)][-1]\n return release, issues"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn sorted list of release - style tags as semver objects.", "response": "def _get_tags(c):\n \"\"\"\n Return sorted list of release-style tags as semver objects.\n \"\"\"\n tags_ = []\n for tagstr in c.run(\"git tag\", hide=True).stdout.strip().split(\"\\n\"):\n try:\n tags_.append(Version(tagstr))\n # Ignore anything non-semver; most of the time they'll be non-release\n # tags, and even if they are, we can't reason about anything\n # non-semver anyways.\n # TODO: perhaps log these to DEBUG\n except ValueError:\n pass\n # Version objects sort semantically\n return sorted(tags_)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntry to find the One True Package for this project.", "response": "def _find_package(c):\n \"\"\"\n Try to find 'the' One True Package for this project.\n\n Mostly for obtaining the ``_version`` file within it.\n\n Uses the ``packaging.package`` config setting if defined. If not defined,\n fallback is to look for a single top-level Python package (directory\n containing ``__init__.py``). (This search ignores a small blacklist of\n directories like ``tests/``, ``vendor/`` etc.)\n \"\"\"\n # TODO: is there a way to get this from the same place setup.py does w/o\n # setup.py barfing (since setup() runs at import time and assumes CLI use)?\n configured_value = c.get(\"packaging\", {}).get(\"package\", None)\n if configured_value:\n return configured_value\n # TODO: tests covering this stuff here (most logic tests simply supply\n # config above)\n packages = [\n path\n for path in os.listdir(\".\")\n if (\n os.path.isdir(path)\n and os.path.exists(os.path.join(path, \"__init__.py\"))\n and path not in (\"tests\", \"integration\", \"sites\", \"vendor\")\n )\n ]\n if not packages:\n sys.exit(\"Unable to find a local Python package!\")\n if len(packages) > 1:\n sys.exit(\"Found multiple Python packages: {0!r}\".format(packages))\n return packages[0]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef build(c, sdist=True, wheel=False, directory=None, python=None, clean=True):\n # Config hooks\n config = c.config.get(\"packaging\", {})\n # TODO: update defaults to be None, then flip the below so non-None runtime\n # beats config.\n sdist = config.get(\"sdist\", sdist)\n wheel = config.get(\"wheel\", wheel)\n python = config.get(\"python\", python or \"python\") # buffalo buffalo\n # Sanity\n if not sdist and not wheel:\n sys.exit(\n \"You said no sdists and no wheels...\"\n \"what DO you want to build exactly?\"\n )\n # Directory path/arg logic\n if not directory:\n directory = \"\" # os.path.join() doesn't like None\n dist_dir = os.path.join(directory, \"dist\")\n dist_arg = \"-d {0}\".format(dist_dir)\n build_dir = os.path.join(directory, \"build\")\n build_arg = \"-b {0}\".format(build_dir)\n # Clean\n if clean:\n if os.path.exists(build_dir):\n rmtree(build_dir)\n # NOTE: not cleaning dist_dir, since this may be called >1 time within\n # publish() trying to build up multiple wheels/etc.\n # TODO: separate clean-build/clean-dist args? Meh\n # Build\n parts = [python, \"setup.py\"]\n if sdist:\n parts.extend((\"sdist\", dist_arg))\n if wheel:\n # Manually execute build in case we are using a custom build dir.\n # Doesn't seem to be a way to tell bdist_wheel to do this directly.\n parts.extend((\"build\", build_arg))\n parts.extend((\"bdist_wheel\", dist_arg))\n c.run(\" \".join(parts))", "response": "Build a new sdist and wheel archives."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npublishing a single entry to PyPI or index of choice.", "response": "def publish(\n c,\n sdist=True,\n wheel=False,\n index=None,\n sign=False,\n dry_run=False,\n directory=None,\n dual_wheels=False,\n alt_python=None,\n check_desc=False,\n):\n \"\"\"\n Publish code to PyPI or index of choice.\n\n All parameters save ``dry_run`` and ``directory`` honor config settings of\n the same name, under the ``packaging`` tree. E.g. say\n ``.configure({'packaging': {'wheel': True}})`` to force building wheel\n archives by default.\n\n :param bool sdist:\n Whether to upload sdists/tgzs.\n\n :param bool wheel:\n Whether to upload wheels (requires the ``wheel`` package from PyPI).\n\n :param str index:\n Custom upload index/repository name. See ``upload`` help for details.\n\n :param bool sign:\n Whether to sign the built archive(s) via GPG.\n\n :param bool dry_run:\n Skip actual publication step if ``True``.\n\n This also prevents cleanup of the temporary build/dist directories, so\n you can examine the build artifacts.\n\n :param str directory:\n Base directory within which will live the ``dist/`` and ``build/``\n directories.\n\n Defaults to a temporary directory which is cleaned up after the run\n finishes.\n\n :param bool dual_wheels:\n When ``True``, builds individual wheels for Python 2 and Python 3.\n\n Useful for situations where you can't build universal wheels, but still\n want to distribute for both interpreter versions.\n\n Requires that you have a useful ``python3`` (or ``python2``, if you're\n on Python 3 already) binary in your ``$PATH``. Also requires that this\n other python have the ``wheel`` package installed in its\n ``site-packages``; usually this will mean the global site-packages for\n that interpreter.\n\n See also the ``alt_python`` argument.\n\n :param str alt_python:\n Path to the 'alternate' Python interpreter to use when\n ``dual_wheels=True``.\n\n When ``None`` (the default) will be ``python3`` or ``python2``,\n depending on the currently active interpreter.\n\n :param bool check_desc:\n Whether to run ``setup.py check -r -s`` (uses ``readme_renderer``)\n before trying to publish - catches long_description bugs. Default:\n ``False``.\n \"\"\"\n # Don't hide by default, this step likes to be verbose most of the time.\n c.config.run.hide = False\n # Config hooks\n config = c.config.get(\"packaging\", {})\n index = config.get(\"index\", index)\n sign = config.get(\"sign\", sign)\n dual_wheels = config.get(\"dual_wheels\", dual_wheels)\n check_desc = config.get(\"check_desc\", check_desc)\n # Initial sanity check, if needed. Will die usefully.\n if check_desc:\n c.run(\"python setup.py check -r -s\")\n # Build, into controlled temp dir (avoids attempting to re-upload old\n # files)\n with tmpdir(skip_cleanup=dry_run, explicit=directory) as tmp:\n # Build default archives\n build(c, sdist=sdist, wheel=wheel, directory=tmp)\n # Build opposing interpreter archive, if necessary\n if dual_wheels:\n if not alt_python:\n alt_python = \"python2\"\n if sys.version_info[0] == 2:\n alt_python = \"python3\"\n build(c, sdist=False, wheel=True, directory=tmp, python=alt_python)\n # Do the thing!\n upload(c, directory=tmp, index=index, sign=sign, dry_run=dry_run)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef upload(c, directory, index=None, sign=False, dry_run=False):\n # Obtain list of archive filenames, then ensure any wheels come first\n # so their improved metadata is what PyPI sees initially (otherwise, it\n # only honors the sdist's lesser data).\n archives = list(\n itertools.chain.from_iterable(\n glob(os.path.join(directory, \"dist\", \"*.{0}\".format(extension)))\n for extension in (\"whl\", \"tar.gz\")\n )\n )\n # Sign each archive in turn\n # TODO: twine has a --sign option; but the below is still nice insofar\n # as it lets us dry-run, generate for web upload when pypi's API is\n # being cranky, etc. Figure out which is better.\n if sign:\n prompt = \"Please enter GPG passphrase for signing: \"\n input_ = StringIO(getpass.getpass(prompt) + \"\\n\")\n gpg_bin = find_gpg(c)\n if not gpg_bin:\n sys.exit(\n \"You need to have one of `gpg`, `gpg1` or `gpg2` \"\n \"installed to GPG-sign!\"\n )\n for archive in archives:\n cmd = \"{0} --detach-sign -a --passphrase-fd 0 {{0}}\".format(\n gpg_bin\n ) # noqa\n c.run(cmd.format(archive), in_stream=input_)\n input_.seek(0) # So it can be replayed by subsequent iterations\n # Upload\n parts = [\"twine\", \"upload\"]\n if index:\n index_arg = \"--repository {0}\".format(index)\n if index:\n parts.append(index_arg)\n paths = archives[:]\n if sign:\n paths.append(os.path.join(directory, \"dist\", \"*.asc\"))\n parts.extend(paths)\n cmd = \" \".join(parts)\n if dry_run:\n print(\"Would publish via: {0}\".format(cmd))\n print(\"Files that would be published:\")\n c.run(\"ls -l {0}\".format(\" \".join(paths)))\n else:\n c.run(cmd)", "response": "Upload all build artifacts in a directory."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates ransom spatial permutations for inference on LISA vectors.", "response": "def permute(self, permutations=99, alternative='two.sided'):\n \"\"\"\n Generate ransom spatial permutations for inference on LISA vectors.\n\n Parameters\n ----------\n permutations : int, optional\n Number of random permutations of observations.\n alternative : string, optional\n Type of alternative to form in generating p-values.\n Options are: `two-sided` which tests for difference between observed\n counts and those obtained from the permutation distribution;\n `positive` which tests the alternative that the focal unit and its\n lag move in the same direction over time; `negative` which tests\n that the focal unit and its lag move in opposite directions over\n the interval.\n \"\"\"\n rY = self.Y.copy()\n idxs = np.arange(len(rY))\n counts = np.zeros((permutations, len(self.counts)))\n for m in range(permutations):\n np.random.shuffle(idxs)\n res = self._calc(rY[idxs, :], self.w, self.k)\n counts[m] = res['counts']\n self.counts_perm = counts\n self.larger_perm = np.array(\n [(counts[:, i] >= self.counts[i]).sum() for i in range(self.k)])\n self.smaller_perm = np.array(\n [(counts[:, i] <= self.counts[i]).sum() for i in range(self.k)])\n self.expected_perm = counts.mean(axis=0)\n self.alternative = alternative\n\n # pvalue logic\n # if P is the proportion that are as large for a one sided test (larger\n # than), then\n # p=P.\n #\n # For a two-tailed test, if P < .5, p = 2 * P, else, p = 2(1-P)\n # Source: Rayner, J. C. W., O. Thas, and D. J. Best. 2009. \"Appendix B:\n # Parametric Bootstrap P-Values.\" In Smooth Tests of Goodness of Fit,\n # 247. John Wiley and Sons.\n # Note that the larger and smaller counts would be complements (except\n # for the shared equality, for\n # a given bin in the circular histogram. So we only need one of them.\n\n # We report two-sided p-values for each bin as the default\n # since a priori there could # be different alternatives for each bin\n # depending on the problem at hand.\n\n alt = alternative.upper()\n if alt == 'TWO.SIDED':\n P = (self.larger_perm + 1) / (permutations + 1.)\n mask = P < 0.5\n self.p = mask * 2 * P + (1 - mask) * 2 * (1 - P)\n elif alt == 'POSITIVE':\n # NE, SW sectors are higher, NW, SE are lower\n POS = _POS8\n if self.k == 4:\n POS = _POS4\n L = (self.larger_perm + 1) / (permutations + 1.)\n S = (self.smaller_perm + 1) / (permutations + 1.)\n P = POS * L + (1 - POS) * S\n self.p = P\n elif alt == 'NEGATIVE':\n # NE, SW sectors are lower, NW, SE are higher\n NEG = _NEG8\n if self.k == 4:\n NEG = _NEG4\n L = (self.larger_perm + 1) / (permutations + 1.)\n S = (self.smaller_perm + 1) / (permutations + 1.)\n P = NEG * L + (1 - NEG) * S\n self.p = P\n else:\n print(('Bad option for alternative: %s.' % alternative))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting the rose diagram of the current object.", "response": "def plot(self, attribute=None, ax=None, **kwargs):\n \"\"\"\n Plot the rose diagram.\n\n Parameters\n ----------\n attribute : (n,) ndarray, optional\n Variable to specify colors of the colorbars.\n ax : Matplotlib Axes instance, optional\n If given, the figure will be created inside this axis.\n Default =None. Note, this axis should have a polar projection.\n **kwargs : keyword arguments, optional\n Keywords used for creating and designing the plot.\n Note: 'c' and 'color' cannot be passed when attribute is not None\n\n Returns\n -------\n fig : Matplotlib Figure instance\n Moran scatterplot figure\n ax : matplotlib Axes instance\n Axes in which the figure is plotted\n\n \"\"\"\n\n from splot.giddy import dynamic_lisa_rose\n fig, ax = dynamic_lisa_rose(self, attribute=attribute,\n ax=ax, **kwargs)\n return fig, ax"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plot_origin(self): # TODO add attribute option to color vectors\n import matplotlib.cm as cm\n import matplotlib.pyplot as plt\n ax = plt.subplot(111)\n xlim = [self._dx.min(), self._dx.max()]\n ylim = [self._dy.min(), self._dy.max()]\n for x, y in zip(self._dx, self._dy):\n xs = [0, x]\n ys = [0, y]\n plt.plot(xs, ys, '-b') # TODO change this to scale with attribute\n plt.axis('equal')\n plt.xlim(xlim)\n plt.ylim(ylim)", "response": "Plot the LISA values starting from the same origin."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plot_vectors(self, arrows=True):\n\n from splot.giddy import dynamic_lisa_vectors\n\n fig, ax = dynamic_lisa_vectors(self, arrows=arrows)\n return fig, ax", "response": "Plots the vectors of positional transition of LISA values in a scatterplot."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _clean(c):\n if isdir(c.sphinx.target):\n rmtree(c.sphinx.target)", "response": "Clean the target directory of the Nuke docs build."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nopening build target s index. html in a browser.", "response": "def _browse(c):\n \"\"\"\n Open build target's index.html in a browser (using 'open').\n \"\"\"\n index = join(c.sphinx.target, c.sphinx.target_file)\n c.run(\"open {0}\".format(index))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nbuild the project s Sphinx docs.", "response": "def build(\n c,\n clean=False,\n browse=False,\n nitpick=False,\n opts=None,\n source=None,\n target=None,\n):\n \"\"\"\n Build the project's Sphinx docs.\n \"\"\"\n if clean:\n _clean(c)\n if opts is None:\n opts = \"\"\n if nitpick:\n opts += \" -n -W -T\"\n cmd = \"sphinx-build{0} {1} {2}\".format(\n (\" \" + opts) if opts else \"\",\n source or c.sphinx.source,\n target or c.sphinx.target,\n )\n c.run(cmd, pty=True)\n if browse:\n _browse(c)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tree(c):\n ignore = \".git|*.pyc|*.swp|dist|*.egg-info|_static|_build|_templates\"\n c.run('tree -Ca -I \"{0}\" {1}'.format(ignore, c.sphinx.source))", "response": "Display documentation contents with the tree program."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbuilds both doc sites w/ maxed nitpicking.", "response": "def sites(c):\n \"\"\"\n Build both doc sites w/ maxed nitpicking.\n \"\"\"\n # TODO: This is super lolzy but we haven't actually tackled nontrivial\n # in-Python task calling yet, so we do this to get a copy of 'our' context,\n # which has been updated with the per-collection config data of the\n # docs/www subcollections.\n docs_c = Context(config=c.config.clone())\n www_c = Context(config=c.config.clone())\n docs_c.update(**docs.configuration())\n www_c.update(**www.configuration())\n # Must build both normally first to ensure good intersphinx inventory files\n # exist =/ circular dependencies ahoy! Do it quietly to avoid pulluting\n # output; only super-serious errors will bubble up.\n # TODO: wants a 'temporarily tweak context settings' contextmanager\n # TODO: also a fucking spinner cuz this confuses me every time I run it\n # when the docs aren't already prebuilt\n docs_c[\"run\"].hide = True\n www_c[\"run\"].hide = True\n docs[\"build\"](docs_c)\n www[\"build\"](www_c)\n docs_c[\"run\"].hide = False\n www_c[\"run\"].hide = False\n # Run the actual builds, with nitpick=True (nitpicks + tracebacks)\n docs[\"build\"](docs_c, nitpick=True)\n www[\"build\"](www_c, nitpick=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef watch_docs(c):\n # TODO: break back down into generic single-site version, then create split\n # tasks as with docs/www above. Probably wants invoke#63.\n\n # NOTE: 'www'/'docs' refer to the module level sub-collections. meh.\n\n # Readme & WWW triggers WWW\n www_c = Context(config=c.config.clone())\n www_c.update(**www.configuration())\n www_handler = make_handler(\n ctx=www_c,\n task_=www[\"build\"],\n regexes=[r\"\\./README.rst\", r\"\\./sites/www\"],\n ignore_regexes=[r\".*/\\..*\\.swp\", r\"\\./sites/www/_build\"],\n )\n\n # Code and docs trigger API\n docs_c = Context(config=c.config.clone())\n docs_c.update(**docs.configuration())\n regexes = [r\"\\./sites/docs\"]\n package = c.get(\"packaging\", {}).get(\"package\", None)\n if package is None:\n package = c.get(\"tests\", {}).get(\"package\", None)\n if package:\n regexes.append(r\"\\./{}/\".format(package))\n api_handler = make_handler(\n ctx=docs_c,\n task_=docs[\"build\"],\n regexes=regexes,\n ignore_regexes=[r\".*/\\..*\\.swp\", r\"\\./sites/docs/_build\"],\n )\n\n observe(www_handler, api_handler)", "response": "Watch both doc trees & rebuild them if files change."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_lower(matrix):\n n = matrix.shape[0]\n lowerlist = []\n for i in range(n):\n for j in range(n):\n if i > j:\n lowerlist.append(matrix[i, j])\n veclen = n * (n - 1) / 2\n lowvec = np.reshape(np.array(lowerlist), (int(veclen), 1))\n return lowvec", "response": "Flattens the lower part of an n x n matrix into an n x n x 1 vector."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef blacken(\n c, line_length=79, folders=None, check=False, diff=False, find_opts=None\n):\n r\"\"\"\n Run black on the current source tree (all ``.py`` files).\n\n .. warning::\n ``black`` only runs on Python 3.6 or above. (However, it can be\n executed against Python 2 compatible code.)\n\n :param int line_length:\n Line length argument. Default: ``79``.\n :param list folders:\n List of folders (or, on the CLI, an argument that can be given N times)\n to search within for ``.py`` files. Default: ``[\".\"]``. Honors the\n ``blacken.folders`` config option.\n :param bool check:\n Whether to run ``black --check``. Default: ``False``.\n :param bool diff:\n Whether to run ``black --diff``. Default: ``False``.\n :param str find_opts:\n Extra option string appended to the end of the internal ``find``\n command. For example, skip a vendor directory with ``\"-and -not -path\n ./vendor\\*\"``, add ``-mtime N``, or etc. Honors the\n ``blacken.find_opts`` config option.\n\n .. versionadded:: 1.2\n .. versionchanged:: 1.4\n Added the ``find_opts`` argument.\n \"\"\"\n config = c.config.get(\"blacken\", {})\n default_folders = [\".\"]\n configured_folders = config.get(\"folders\", default_folders)\n folders = folders or configured_folders\n\n default_find_opts = \"\"\n configured_find_opts = config.get(\"find_opts\", default_find_opts)\n find_opts = find_opts or configured_find_opts\n\n black_command_line = \"black -l {}\".format(line_length)\n if check:\n black_command_line = \"{} --check\".format(black_command_line)\n if diff:\n black_command_line = \"{} --diff\".format(black_command_line)\n if find_opts:\n find_opts = \" {}\".format(find_opts)\n else:\n find_opts = \"\"\n\n cmd = \"find {} -name '*.py'{} | xargs {}\".format(\n \" \".join(folders), find_opts, black_command_line\n )\n c.run(cmd, pty=True)", "response": "r Runs black on the current source tree."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef normalize_response_value(rv):\n status = headers = None\n if isinstance(rv, tuple):\n rv, status, headers = rv + (None,) * (3 - len(rv))\n return rv, status, headers", "response": "Normalizes the response value into a 3 - tuple"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmake a JsonResponse object from a response.", "response": "def make_json_response(rv):\n \"\"\" Make JsonResponse\n :param rv: Response: the object to encode, or tuple (response, status, headers)\n :type rv: tuple|*\n :rtype: JsonResponse\n \"\"\"\n # Tuple of (response, status, headers)\n rv, status, headers = normalize_response_value(rv)\n\n # JsonResponse\n if isinstance(rv, JsonResponse):\n return rv\n\n # Data\n return JsonResponse(rv, status, headers)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a Markov - based mobility index.", "response": "def markov_mobility(p, measure=\"P\", ini=None):\n \"\"\"\n Markov-based mobility index.\n\n Parameters\n ----------\n p : array\n (k, k), Markov transition probability matrix.\n measure : string\n If measure= \"P\",\n :math:`M_{P} = \\\\frac{m-\\sum_{i=1}^m P_{ii}}{m-1}`;\n if measure = \"D\",\n :math:`M_{D} = 1 - |\\det(P)|`,\n where :math:`\\det(P)` is the determinant of :math:`P`;\n if measure = \"L2\",\n :math:`M_{L2} = 1 - |\\lambda_2|`,\n where :math:`\\lambda_2` is the second largest eigenvalue of\n :math:`P`;\n if measure = \"B1\",\n :math:`M_{B1} = \\\\frac{m-m \\sum_{i=1}^m \\pi_i P_{ii}}{m-1}`,\n where :math:`\\pi` is the initial income distribution;\n if measure == \"B2\",\n :math:`M_{B2} = \\\\frac{1}{m-1} \\sum_{i=1}^m \\sum_{\n j=1}^m \\pi_i P_{ij} |i-j|`,\n where :math:`\\pi` is the initial income distribution.\n ini : array\n (k,), initial distribution. Need to be specified if\n measure = \"B1\" or \"B2\". If not,\n the initial distribution would be treated as a uniform\n distribution.\n\n Returns\n -------\n mobi : float\n Mobility value.\n\n Notes\n -----\n The mobility indices are based on :cite:`Formby:2004fk`.\n\n Examples\n --------\n >>> import numpy as np\n >>> import libpysal\n >>> import mapclassify as mc\n >>> from giddy.markov import Markov\n >>> from giddy.mobility import markov_mobility\n >>> f = libpysal.io.open(libpysal.examples.get_path(\"usjoin.csv\"))\n >>> pci = np.array([f.by_col[str(y)] for y in range(1929,2010)])\n >>> q5 = np.array([mc.Quantiles(y).yb for y in pci]).transpose()\n >>> m = Markov(q5)\n >>> m.p\n array([[0.91011236, 0.0886392 , 0.00124844, 0. , 0. ],\n [0.09972299, 0.78531856, 0.11080332, 0.00415512, 0. ],\n [0. , 0.10125 , 0.78875 , 0.1075 , 0.0025 ],\n [0. , 0.00417827, 0.11977716, 0.79805014, 0.07799443],\n [0. , 0. , 0.00125156, 0.07133917, 0.92740926]])\n\n (1) Estimate Shorrock1 mobility index:\n\n >>> mobi_1 = markov_mobility(m.p, measure=\"P\")\n >>> print(\"{:.5f}\".format(mobi_1))\n 0.19759\n\n (2) Estimate Shorrock2 mobility index:\n\n >>> mobi_2 = markov_mobility(m.p, measure=\"D\")\n >>> print(\"{:.5f}\".format(mobi_2))\n 0.60685\n\n (3) Estimate Sommers and Conlisk mobility index:\n\n >>> mobi_3 = markov_mobility(m.p, measure=\"L2\")\n >>> print(\"{:.5f}\".format(mobi_3))\n 0.03978\n\n (4) Estimate Bartholomew1 mobility index (note that the initial\n distribution should be given):\n\n >>> ini = np.array([0.1,0.2,0.2,0.4,0.1])\n >>> mobi_4 = markov_mobility(m.p, measure = \"B1\", ini=ini)\n >>> print(\"{:.5f}\".format(mobi_4))\n 0.22777\n\n (5) Estimate Bartholomew2 mobility index (note that the initial\n distribution should be given):\n\n >>> ini = np.array([0.1,0.2,0.2,0.4,0.1])\n >>> mobi_5 = markov_mobility(m.p, measure = \"B2\", ini=ini)\n >>> print(\"{:.5f}\".format(mobi_5))\n 0.04637\n\n \"\"\"\n\n p = np.array(p)\n k = p.shape[1]\n if measure == \"P\":\n t = np.trace(p)\n mobi = (k - t) / (k - 1)\n elif measure == \"D\":\n mobi = 1 - abs(la.det(p))\n elif measure == \"L2\":\n w, v = la.eig(p)\n eigen_value_abs = abs(w)\n mobi = 1 - np.sort(eigen_value_abs)[-2]\n elif measure == \"B1\":\n if ini is None:\n ini = 1.0 / k * np.ones(k)\n mobi = (k - k * np.sum(ini * np.diag(p))) / (k - 1)\n elif measure == \"B2\":\n mobi = 0\n if ini is None:\n ini = 1.0 / k * np.ones(k)\n for i in range(k):\n for j in range(k):\n mobi = mobi + ini[i] * p[i, j] * abs(i - j)\n mobi = mobi / (k - 1)\n\n return mobi"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef prais(pmat):\n pmat = np.array(pmat)\n pr = 1 - np.diag(pmat)\n return pr", "response": "Returns a Prais conditional mobility measure for each class in the k - class hierarchy."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntests for homogeneity of Markov transition probabilities across regimes.", "response": "def homogeneity(transition_matrices, regime_names=[], class_names=[],\n title=\"Markov Homogeneity Test\"):\n \"\"\"\n Test for homogeneity of Markov transition probabilities across regimes.\n\n Parameters\n ----------\n transition_matrices : list\n of transition matrices for regimes, all matrices must\n have same size (r, c). r is the number of rows in the\n transition matrix and c is the number of columns in\n the transition matrix.\n regime_names : sequence\n Labels for the regimes.\n class_names : sequence\n Labels for the classes/states of the Markov chain.\n title : string\n name of test.\n\n Returns\n -------\n : implicit\n an instance of Homogeneity_Results.\n \"\"\"\n\n return Homogeneity_Results(transition_matrices, regime_names=regime_names,\n class_names=class_names, title=title)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates sojourn time based on a given transition probability matrix.", "response": "def sojourn_time(p):\n \"\"\"\n Calculate sojourn time based on a given transition probability matrix.\n\n Parameters\n ----------\n p : array\n (k, k), a Markov transition probability matrix.\n\n Returns\n -------\n : array\n (k, ), sojourn times. Each element is the expected time a Markov\n chain spends in each states before leaving that state.\n\n Notes\n -----\n Refer to :cite:`Ibe2009` for more details on sojourn times for Markov\n chains.\n\n Examples\n --------\n >>> from giddy.markov import sojourn_time\n >>> import numpy as np\n >>> p = np.array([[.5, .25, .25], [.5, 0, .5], [.25, .25, .5]])\n >>> sojourn_time(p)\n array([2., 1., 2.])\n\n \"\"\"\n p = np.asarray(p)\n pii = p.diagonal()\n\n if not (1 - pii).all():\n print(\"Sojourn times are infinite for absorbing states!\")\n return 1 / (1 - pii)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetect spillover locations for diffusion in LISA Markov.", "response": "def spillover(self, quadrant=1, neighbors_on=False):\n \"\"\"\n Detect spillover locations for diffusion in LISA Markov.\n\n Parameters\n ----------\n quadrant : int\n which quadrant in the scatterplot should form the core\n of a cluster.\n neighbors_on : binary\n If false, then only the 1st order neighbors of a core\n location are included in the cluster.\n If true, neighbors of cluster core 1st order neighbors\n are included in the cluster.\n\n Returns\n -------\n results : dictionary\n two keys - values pairs:\n 'components' - array (n, t)\n values are integer ids (starting at 1) indicating which\n component/cluster observation i in period t belonged to.\n 'spillover' - array (n, t-1)\n binary values indicating if the location was a\n spill-over location that became a new member of a\n previously existing cluster.\n\n Examples\n --------\n >>> import libpysal\n >>> from giddy.markov import LISA_Markov\n >>> f = libpysal.io.open(libpysal.examples.get_path(\"usjoin.csv\"))\n >>> years = list(range(1929, 2010))\n >>> pci = np.array([f.by_col[str(y)] for y in years]).transpose()\n >>> w = libpysal.io.open(libpysal.examples.get_path(\"states48.gal\")).read()\n >>> np.random.seed(10)\n >>> lm_random = LISA_Markov(pci, w, permutations=99)\n >>> r = lm_random.spillover()\n >>> (r['components'][:, 12] > 0).sum()\n 17\n >>> (r['components'][:, 13]>0).sum()\n 23\n >>> (r['spill_over'][:,12]>0).sum()\n 6\n\n Including neighbors of core neighbors\n >>> rn = lm_random.spillover(neighbors_on=True)\n >>> (rn['components'][:, 12] > 0).sum()\n 26\n >>> (rn[\"components\"][:, 13] > 0).sum()\n 34\n >>> (rn[\"spill_over\"][:, 12] > 0).sum()\n 8\n\n \"\"\"\n n, k = self.q.shape\n if self.permutations:\n spill_over = np.zeros((n, k - 1))\n components = np.zeros((n, k))\n i2id = {} # handle string keys\n for key in list(self.w.neighbors.keys()):\n idx = self.w.id2i[key]\n i2id[idx] = key\n sig_lisas = (self.q == quadrant) \\\n * (self.p_values <= self.significance_level)\n sig_ids = [np.nonzero(\n sig_lisas[:, i])[0].tolist() for i in range(k)]\n\n neighbors = self.w.neighbors\n for t in range(k - 1):\n s1 = sig_ids[t]\n s2 = sig_ids[t + 1]\n g1 = Graph(undirected=True)\n for i in s1:\n for neighbor in neighbors[i2id[i]]:\n g1.add_edge(i2id[i], neighbor, 1.0)\n if neighbors_on:\n for nn in neighbors[neighbor]:\n g1.add_edge(neighbor, nn, 1.0)\n components1 = g1.connected_components(op=gt)\n components1 = [list(c.nodes) for c in components1]\n g2 = Graph(undirected=True)\n for i in s2:\n for neighbor in neighbors[i2id[i]]:\n g2.add_edge(i2id[i], neighbor, 1.0)\n if neighbors_on:\n for nn in neighbors[neighbor]:\n g2.add_edge(neighbor, nn, 1.0)\n components2 = g2.connected_components(op=gt)\n components2 = [list(c.nodes) for c in components2]\n c2 = []\n c1 = []\n for c in components2:\n c2.extend(c)\n for c in components1:\n c1.extend(c)\n\n new_ids = [j for j in c2 if j not in c1]\n spill_ids = []\n for j in new_ids:\n # find j's component in period 2\n cj = [c for c in components2 if j in c][0]\n # for members of j's component in period 2, check if they\n # belonged to any components in period 1\n for i in cj:\n if i in c1:\n spill_ids.append(j)\n break\n for spill_id in spill_ids:\n id = self.w.id2i[spill_id]\n spill_over[id, t] = 1\n for c, component in enumerate(components1):\n for i in component:\n ii = self.w.id2i[i]\n components[ii, t] = c + 1\n results = {}\n results['components'] = components\n results['spill_over'] = spill_over\n return results\n\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets entity property names", "response": "def get_entity_propnames(entity):\n \"\"\" Get entity property names\n\n :param entity: Entity\n :type entity: sqlalchemy.ext.declarative.api.DeclarativeMeta\n :returns: Set of entity property names\n :rtype: set\n \"\"\"\n ins = entity if isinstance(entity, InstanceState) else inspect(entity)\n return set(\n ins.mapper.column_attrs.keys() + # Columns\n ins.mapper.relationships.keys() # Relationships\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_entity_loaded_propnames(entity):\n ins = inspect(entity)\n keynames = get_entity_propnames(ins)\n\n # If the entity is not transient -- exclude unloaded keys\n # Transient entities won't load these anyway, so it's safe to include all columns and get defaults\n if not ins.transient:\n keynames -= ins.unloaded\n\n # If the entity is expired -- reload expired attributes as well\n # Expired attributes are usually unloaded as well!\n if ins.expired:\n keynames |= ins.expired_attributes\n\n # Finish\n return keynames", "response": "Get the set of property names that are loaded in the entity."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new version with a minor number greater than self s. version .", "response": "def next_minor(self):\n \"\"\"\n Return a Version whose minor number is one greater than self's.\n\n .. note::\n The new Version will always have a zeroed-out bugfix/tertiary version\n number, because the \"next minor release\" of e.g. 1.2.1 is 1.3.0, not\n 1.3.1.\n \"\"\"\n clone = self.clone()\n clone.minor += 1\n clone.patch = 0\n return clone"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_encoding(name, encoding_to_check, alternative_encoding, source=None):\n assert name is not None\n\n if encoding_to_check not in (alternative_encoding, 'chardet', None):\n try:\n ''.encode(encoding_to_check)\n except LookupError:\n raise pygount.common.OptionError(\n '{0} is \"{1}\" but must be \"{2}\" or a known Python encoding'.format(\n name, encoding_to_check, alternative_encoding),\n source)", "response": "Checks that the encoding of the given name is valid Python encoding."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lines(text):\n assert text is not None\n assert '\\r' not in text\n previous_newline_index = 0\n newline_index = text.find('\\n')\n while newline_index != -1:\n yield text[previous_newline_index:newline_index]\n previous_newline_index = newline_index + 1\n newline_index = text.find('\\n', previous_newline_index)\n last_line = text[previous_newline_index:]\n if last_line != '':\n yield last_line", "response": "Generator function to yield lines stored in a\n ."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef matching_number_line_and_regex(source_lines, generated_regexes, max_line_count=15):\n initial_numbers_and_lines = enumerate(itertools.islice(source_lines, max_line_count))\n matching_number_line_and_regexps = (\n (number, line, matching_regex)\n for number, line in initial_numbers_and_lines\n for matching_regex in generated_regexes\n if matching_regex.match(line)\n )\n possible_first_matching_number_line_and_regexp = list(\n itertools.islice(matching_number_line_and_regexps, 1))\n result = (possible_first_matching_number_line_and_regexp + [None])[0]\n return result", "response": "Returns the first line and its regex that matches the source code."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _pythonized_comments(tokens):\n is_after_colon = True\n for token_type, token_text in tokens:\n if is_after_colon and (token_type in pygments.token.String):\n token_type = pygments.token.Comment\n elif token_text == ':':\n is_after_colon = True\n elif token_type not in pygments.token.Comment:\n is_whitespace = len(token_text.rstrip(' \\f\\n\\r\\t')) == 0\n if not is_whitespace:\n is_after_colon = False\n yield token_type, token_text", "response": "Yields the pythonized comments for the given list of tokens."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encoding_for(source_path, encoding='automatic', fallback_encoding=None):\n assert encoding is not None\n\n if encoding == 'automatic':\n with open(source_path, 'rb') as source_file:\n heading = source_file.read(128)\n result = None\n if len(heading) == 0:\n # File is empty, assume a dummy encoding.\n result = 'utf-8'\n if result is None:\n # Check for known BOMs.\n for bom, encoding in _BOM_TO_ENCODING_MAP.items():\n if heading[:len(bom)] == bom:\n result = encoding\n break\n if result is None:\n # Look for common headings that indicate the encoding.\n ascii_heading = heading.decode('ascii', errors='replace')\n ascii_heading = ascii_heading.replace('\\r\\n', '\\n')\n ascii_heading = ascii_heading.replace('\\r', '\\n')\n ascii_heading = '\\n'.join(ascii_heading.split('\\n')[:2]) + '\\n'\n coding_magic_match = _CODING_MAGIC_REGEX.match(ascii_heading)\n if coding_magic_match is not None:\n result = coding_magic_match.group('encoding')\n else:\n first_line = ascii_heading.split('\\n')[0]\n xml_prolog_match = _XML_PROLOG_REGEX.match(first_line)\n if xml_prolog_match is not None:\n result = xml_prolog_match.group('encoding')\n elif encoding == 'chardet':\n assert _detector is not None, \\\n 'without chardet installed, encoding=\"chardet\" must be rejected before calling encoding_for()'\n _detector.reset()\n with open(source_path, 'rb') as source_file:\n for line in source_file.readlines():\n _detector.feed(line)\n if _detector.done:\n break\n result = _detector.result['encoding']\n if result is None:\n _log.warning(\n '%s: chardet cannot determine encoding, assuming fallback encoding %s',\n source_path, fallback_encoding)\n result = fallback_encoding\n else:\n # Simply use the specified encoding.\n result = encoding\n if result is None:\n # Encoding 'automatic' or 'chardet' failed to detect anything.\n if fallback_encoding is not None:\n # If defined, use the fallback encoding.\n result = fallback_encoding\n else:\n try:\n # Attempt to read the file as UTF-8.\n with open(source_path, 'r', encoding='utf-8') as source_file:\n source_file.read()\n result = 'utf-8'\n except UnicodeDecodeError:\n # UTF-8 did not work out, use the default as last resort.\n result = DEFAULT_FALLBACK_ENCODING\n _log.debug('%s: no fallback encoding specified, using %s', source_path, result)\n\n assert result is not None\n return result", "response": "Returns the encoding used by the text file stored in source_path."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef has_lexer(source_path):\n result = bool(pygments.lexers.find_lexer_class_for_filename(source_path))\n if not result:\n suffix = os.path.splitext(os.path.basename(source_path))[1].lstrip('.')\n result = suffix in _SUFFIX_TO_FALLBACK_LEXER_MAP\n return result", "response": "Returns True if there is a lexer for source_path."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nanalyzes the source code in the specified source file.", "response": "def source_analysis(\n source_path, group, encoding='automatic', fallback_encoding='cp1252',\n generated_regexes=pygount.common.regexes_from(DEFAULT_GENERATED_PATTERNS_TEXT),\n duplicate_pool=None):\n \"\"\"\n Analysis for line counts in source code stored in ``source_path``.\n\n :param source_path:\n :param group: name of a logical group the sourc code belongs to, e.g. a\n package.\n :param encoding: encoding according to :func:`encoding_for`\n :param fallback_encoding: fallback encoding according to\n :func:`encoding_for`\n :return: a :class:`SourceAnalysis`\n \"\"\"\n assert encoding is not None\n assert generated_regexes is not None\n\n result = None\n lexer = None\n source_code = None\n source_size = os.path.getsize(source_path)\n if source_size == 0:\n _log.info('%s: is empty', source_path)\n result = pseudo_source_analysis(source_path, group, SourceState.empty)\n elif is_binary_file(source_path):\n _log.info('%s: is binary', source_path)\n result = pseudo_source_analysis(source_path, group, SourceState.binary)\n elif not has_lexer(source_path):\n _log.info('%s: unknown language', source_path)\n result = pseudo_source_analysis(source_path, group, SourceState.unknown)\n elif duplicate_pool is not None:\n duplicate_path = duplicate_pool.duplicate_path(source_path)\n if duplicate_path is not None:\n _log.info('%s: is a duplicate of %s', source_path, duplicate_path)\n result = pseudo_source_analysis(source_path, group, SourceState.duplicate, duplicate_path)\n if result is None:\n if encoding in ('automatic', 'chardet'):\n encoding = encoding_for(source_path, encoding, fallback_encoding)\n try:\n with open(source_path, 'r', encoding=encoding) as source_file:\n source_code = source_file.read()\n except (LookupError, OSError, UnicodeError) as error:\n _log.warning('cannot read %s using encoding %s: %s', source_path, encoding, error)\n result = pseudo_source_analysis(source_path, group, SourceState.error, error)\n if result is None:\n lexer = guess_lexer(source_path, source_code)\n assert lexer is not None\n if (result is None) and (len(generated_regexes) != 0):\n number_line_and_regex = matching_number_line_and_regex(\n pygount.common.lines(source_code), generated_regexes\n )\n if number_line_and_regex is not None:\n number, _, regex = number_line_and_regex\n message = 'line {0} matches {1}'.format(number, regex)\n _log.info('%s: is generated code because %s', source_path, message)\n result = pseudo_source_analysis(source_path, group, SourceState.generated, message)\n if result is None:\n assert lexer is not None\n assert source_code is not None\n language = lexer.name\n if ('xml' in language.lower()) or (language == 'Genshi'):\n dialect = pygount.xmldialect.xml_dialect(source_path, source_code)\n if dialect is not None:\n language = dialect\n _log.info('%s: analyze as %s using encoding %s', source_path, language, encoding)\n mark_to_count_map = {'c': 0, 'd': 0, 'e': 0, 's': 0}\n for line_parts in _line_parts(lexer, source_code):\n mark_to_increment = 'e'\n for mark_to_check in ('d', 's', 'c'):\n if mark_to_check in line_parts:\n mark_to_increment = mark_to_check\n mark_to_count_map[mark_to_increment] += 1\n result = SourceAnalysis(\n path=source_path,\n language=language,\n group=group,\n code=mark_to_count_map['c'],\n documentation=mark_to_count_map['d'],\n empty=mark_to_count_map['e'],\n string=mark_to_count_map['s'],\n state=SourceState.analyzed.name,\n state_info=None,\n )\n\n assert result is not None\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef polynomial(img, mask, inplace=False, replace_all=False,\r\n max_dev=1e-5, max_iter=20, order=2):\r\n '''\r\n replace all masked values\r\n calculate flatField from 2d-polynomal fit filling\r\n all high gradient areas within averaged fit-image\r\n\r\n returns flatField, average background level, fitted image, valid indices mask\r\n '''\r\n if inplace:\r\n out = img\r\n else:\r\n out = img.copy()\r\n lastm = 0\r\n for _ in range(max_iter):\r\n out2 = polyfit2dGrid(out, mask, order=order, copy=not inplace,\r\n replace_all=replace_all)\r\n if replace_all:\r\n out = out2\r\n break\r\n res = (np.abs(out2 - out)).mean()\r\n print('residuum: ', res)\r\n if res < max_dev:\r\n out = out2\r\n break\r\n out = out2\r\n mask = _highGrad(out)\r\n\r\n m = mask.sum()\r\n if m == lastm or m == img.size:\r\n break\r\n lastm = m\r\n out = np.clip(out, 0, 1, out=out) # if inplace else None)\r\n return out", "response": "polyfit2dGrid - 2d polynomial fit"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the error distribution of a single event in a single time series.", "response": "def errorDist(scale, measExpTime, n_events_in_expTime,\r\n event_duration, std,\r\n points_per_time=100, n_repetitions=300):\r\n '''\r\n TODO\r\n '''\r\n ntimes = 10\r\n s1 = measExpTime * scale * 10\r\n # exp. time factor 1/16-->16:\r\n p2 = np.logspace(-4, 4, 18, base=2)\r\n\r\n t = np.linspace(0, s1, ntimes * points_per_time * s1)\r\n\r\n err = None\r\n for rr in range(n_repetitions):\r\n\r\n f = _flux(t, n_events_in_expTime, event_duration, std)\r\n\r\n e = np.array([_capture(f, t, measExpTime, pp) for pp in p2])\r\n if err is None:\r\n err = e\r\n else:\r\n err += e\r\n err /= (rr + 1)\r\n \r\n # normalize, so that error==1 at 1:\r\n try:\r\n fac = findXAt(err, p2, 1)\r\n except:\r\n fac = 1\r\n\r\n err /= fac\r\n return p2, err, t, f"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef exampleSignals(std=1, dur1=1, dur2=3, dur3=0.2,\r\n n1=0.5, n2=0.5, n3=2):\r\n '''\r\n std ... standard deviation of every signal\r\n dur1...dur3 --> event duration per second\r\n n1...n3 --> number of events per second\r\n '''\r\n np.random.seed(123)\r\n t = np.linspace(0, 10, 100)\r\n\r\n f0 = _flux(t, n1, dur1, std, offs=0)\r\n f1 = _flux(t, n2, dur2, std, offs=0)\r\n f2 = _flux(t, n3, dur3, std, offs=0)\r\n return t,f0,f1,f2", "response": "std ... standard deviation of every signal\r\n dur1...dur3 --> event duration per second\r\n n1...n3 --> number of events per second"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new array of the same size as the original array.", "response": "def _flux(t, n, duration, std, offs=1):\r\n '''\r\n returns Gaussian shaped signal fluctuations [events]\r\n \r\n t --> times to calculate event for\r\n n --> numbers of events per sec\r\n duration --> event duration per sec\r\n std --> std of event if averaged over time\r\n offs --> event offset\r\n '''\r\n duration *= len(t) / t[-1]\r\n duration = int(max(duration, 1))\r\n\r\n pos = []\r\n n *= t[-1]\r\n pp = np.arange(len(t))\r\n valid = np.ones_like(t, dtype=bool)\r\n for _ in range(int(round(n))):\r\n try:\r\n ppp = np.random.choice(pp[valid], 1)[0]\r\n pos.append(ppp)\r\n valid[max(0, ppp - duration - 1):ppp + duration + 1] = False\r\n except ValueError:\r\n break\r\n sign = np.random.randint(0, 2, len(pos))\r\n sign[sign == 0] = -1\r\n\r\n out = np.zeros_like(t)\r\n\r\n amps = np.random.normal(loc=0, scale=1, size=len(pos))\r\n\r\n if duration > 2:\r\n evt = gaussian(duration, duration)\r\n evt -= evt[0]\r\n else:\r\n evt = np.ones(shape=duration)\r\n\r\n\r\n for s, p, a in zip(sign, pos, amps):\r\n pp = duration\r\n if p + duration > len(out):\r\n pp = len(out) - p\r\n\r\n out[p:p + pp] = s * a * evt[:pp]\r\n\r\n out /= out.std() / std\r\n out += offs\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _capture(f, t, t0, factor):\r\n '''\r\n capture signal and return its standard deviation\r\n #TODO: more detail\r\n '''\r\n n_per_sec = len(t) / t[-1]\r\n\r\n # len of one split:\r\n n = int(t0 * factor * n_per_sec)\r\n s = len(f) // n\r\n m = s * n\r\n f = f[:m]\r\n ff = np.split(f, s)\r\n m = np.mean(ff, axis=1)\r\n\r\n return np.std(m)", "response": "capture signal and return its standard deviation"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef genericCameraMatrix(shape, angularField=60):\r\n '''\r\n Return a generic camera matrix\r\n [[fx, 0, cx],\r\n [ 0, fy, cy],\r\n [ 0, 0, 1]]\r\n for a given image shape\r\n '''\r\n # http://nghiaho.com/?page_id=576\r\n # assume that the optical centre is in the middle:\r\n cy = int(shape[0] / 2)\r\n cx = int(shape[1] / 2)\r\n\r\n # assume that the FOV is 60 DEG (webcam)\r\n fx = fy = cx / np.tan(angularField / 2 * np.pi /\r\n 180) # camera focal length\r\n # see\r\n # http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html\r\n return np.array([[fx, 0, cx],\r\n [0, fy, cy],\r\n [0, 0, 1]\r\n ], dtype=np.float32)", "response": "Returns a generic camera matrix for a given image shape"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef standardDeviation2d(img, ksize=5, blurred=None):\r\n '''\r\n calculate the spatial resolved standard deviation \r\n for a given 2d array\r\n \r\n ksize -> kernel size\r\n \r\n blurred(optional) -> with same ksize gaussian filtered image\r\n setting this parameter reduces processing time\r\n '''\r\n if ksize not in (list, tuple):\r\n ksize = (ksize,ksize)\r\n\r\n if blurred is None:\r\n blurred = gaussian_filter(img, ksize)\r\n else:\r\n assert blurred.shape == img.shape\r\n \r\n std = np.empty_like(img)\r\n \r\n _calc(img, ksize[0], ksize[1], blurred, std)\r\n \r\n return std", "response": "calculate the spatial resolved standard deviation of a given 2d array"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef maskedFilter(arr, mask, ksize=30, fill_mask=True,\r\n fn='median'):\r\n '''\r\n fn['mean', 'median']\r\n\r\n fill_mask=True:\r\n replaced masked areas with filtered results\r\n\r\n fill_mask=False:\r\n masked areas are ignored\r\n '''\r\n\r\n if fill_mask:\r\n mask1 = mask\r\n out = arr\r\n else:\r\n mask1 = ~mask\r\n out = np.full_like(arr, fill_value=np.nan)\r\n mask2 = ~mask\r\n\r\n if fn == 'mean':\r\n _calcMean(arr, mask1, mask2, out, ksize // 2)\r\n else:\r\n buff = np.empty(shape=(ksize * ksize), dtype=arr.dtype)\r\n _calcMedian(arr, mask1, mask2, out, ksize // 2, buff)\r\n return out", "response": "Returns a new array with masked areas replaced with filtered results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef vignettingFromDifferentObjects(imgs, bg):\r\n '''\r\n Extract vignetting from a set of images\r\n containing different devices\r\n The devices spatial inhomogeneities are averaged\r\n\r\n This method is referred as 'Method C' in \r\n ---\r\n K.Bedrich, M.Bokalic et al.:\r\n ELECTROLUMINESCENCE IMAGING OF PV DEVICES:\r\n ADVANCED FLAT FIELD CALIBRATION,2017\r\n ---\r\n '''\r\n\r\n f = FlatFieldFromImgFit(imgs, bg)\r\n return f.result, f.mask", "response": "Extract vignetting from a set of images containing different devices"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef SNR_IEC(i1, i2, ibg=0, allow_color_images=False):\r\n '''\r\n Calculate the averaged signal-to-noise ratio SNR50\r\n as defined by IEC NP 60904-13\r\n\r\n needs 2 reference EL images and one background image\r\n '''\r\n # ensure images are type float64 (double precision):\r\n i1 = np.asfarray(i1)\r\n i2 = np.asfarray(i2)\r\n if ibg is not 0:\r\n ibg = np.asfarray(ibg)\r\n assert i1.shape == ibg.shape, 'all input images need to have the same resolution'\r\n\r\n assert i1.shape == i2.shape, 'all input images need to have the same resolution'\r\n if not allow_color_images:\r\n assert i1.ndim == 2, 'Images need to be in grayscale according to the IEC standard'\r\n\r\n # SNR calculation as defined in 'IEC TS 60904-13':\r\n signal = 0.5 * (i1 + i2) - ibg\r\n noise = 0.5**0.5 * np.abs(i1 - i2) * ((2 / np.pi)**-0.5)\r\n if signal.ndim == 3: # color\r\n signal = np.average(signal, axis=2, weights=(0.114, 0.587, 0.299))\r\n noise = np.average(noise, axis=2, weights=(0.114, 0.587, 0.299))\r\n signal = signal.sum()\r\n noise = noise.sum()\r\n return signal / noise", "response": "Calculate the average signal - to - noise ratio of two images."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrotate the image by the given angle", "response": "def _rotate(img, angle):\r\n '''\r\n angle [DEG]\r\n '''\r\n s = img.shape\r\n if angle == 0:\r\n return img\r\n else:\r\n M = cv2.getRotationMatrix2D((s[1] // 2,\r\n s[0] // 2), angle, 1)\r\n return cv2.warpAffine(img, M, (s[1], s[0]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _findOverlap(self, img_rgb, overlap, overlapDeviation,\r\n rotation, rotationDeviation):\r\n '''\r\n return offset(x,y) which fit best self._base_img\r\n through template matching\r\n '''\r\n # get gray images\r\n if len(img_rgb.shape) != len(img_rgb.shape):\r\n raise Exception(\r\n 'number of channels(colors) for both images different')\r\n if overlapDeviation == 0 and rotationDeviation == 0:\r\n return (0, overlap, rotation)\r\n\r\n s = self.base_img_rgb.shape\r\n ho = int(round(overlap * 0.5))\r\n overlap = int(round(overlap))\r\n # create two image cuts to compare:\r\n imgcut = self.base_img_rgb[s[0] - overlapDeviation - overlap:, :]\r\n template = img_rgb[:overlap, ho:s[1] - ho]\r\n\r\n def fn(angle):\r\n rotTempl = self._rotate(template, angle)\r\n # Apply template Matching\r\n fn.res = cv2.matchTemplate(rotTempl.astype(np.float32),\r\n imgcut.astype(np.float32),\r\n cv2.TM_CCORR_NORMED)\r\n return 1 / fn.res.mean()\r\n\r\n if rotationDeviation == 0:\r\n angle = rotation\r\n fn(rotation)\r\n\r\n else:\r\n # find best rotation angle:\r\n angle = brent(fn, brack=(rotation - rotationDeviation,\r\n rotation + rotationDeviation))\r\n\r\n loc = cv2.minMaxLoc(fn.res)[-1]\r\n\r\n offsx = int(round(loc[0] - ho))\r\n offsy = overlapDeviation + overlap - loc[1]\r\n return offsx, offsy, angle", "response": "find the best image that fits best self. _base_img"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nestimating the noise level function over images at the same position", "response": "def estimateFromImages(imgs1, imgs2=None, mn_mx=None, nbins=100):\r\n '''\r\n estimate the noise level function as stDev over image intensity\r\n from a set of 2 image groups \r\n images at the same position have to show\r\n the identical setup, so\r\n imgs1[i] - imgs2[i] = noise\r\n '''\r\n if imgs2 is None:\r\n imgs2 = [None] * len(imgs1)\r\n else:\r\n assert len(imgs1) == len(imgs2)\r\n\r\n y_vals = np.empty((len(imgs1), nbins))\r\n w_vals = np.zeros((len(imgs1), nbins))\r\n\r\n if mn_mx is None:\r\n print('estimating min and max image value')\r\n mn = 1e6\r\n mx = -1e6\r\n # get min and max image value checking all first images:\r\n for n, i1 in enumerate(imgs1):\r\n print('%s/%s' % (n + 1, len(imgs1)))\r\n i1 = imread(i1)\r\n mmn, mmx = _getMinMax(i1)\r\n mn = min(mn, mmn)\r\n mx = mx = max(mx, mmx)\r\n print('--> min(%s), max(%s)' % (mn, mx))\r\n else:\r\n mn, mx = mn_mx\r\n\r\n x = None\r\n print('get noise level function')\r\n for n, (i1, i2) in enumerate(zip(imgs1, imgs2)):\r\n print('%s/%s' % (n + 1, len(imgs1)))\r\n\r\n i1 = imread(i1)\r\n if i2 is not None:\r\n i2 = imread(i2)\r\n\r\n x, y, weights, _ = calcNLF(i1, i2, mn_mx_nbins=(mn, mx, nbins), x=x)\r\n y_vals[n] = y\r\n w_vals[n] = weights\r\n\r\n # filter empty places:\r\n filledPos = np.sum(w_vals, axis=0) != 0\r\n w_vals = w_vals[:, filledPos]\r\n y_vals = y_vals[:, filledPos]\r\n x = x[filledPos]\r\n\r\n y_avg = np.average(np.nan_to_num(y_vals),\r\n weights=w_vals,\r\n axis=0)\r\n w_vals = np.sum(w_vals, axis=0)\r\n w_vals /= w_vals.sum()\r\n\r\n fitParams, fn, i = _evaluate(x, y_avg, w_vals)\r\n return x, fn, y_avg, y_vals, w_vals, fitParams, i"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nevaluate the parameters of the necessary by function through curve fitting", "response": "def _evaluate(x, y, weights):\r\n '''\r\n get the parameters of the, needed by 'function'\r\n through curve fitting\r\n '''\r\n i = _validI(x, y, weights)\r\n xx = x[i]\r\n y = y[i]\r\n\r\n try:\r\n fitParams = _fit(xx, y)\r\n # bound noise fn to min defined y value:\r\n minY = function(xx[0], *fitParams)\r\n fitParams = np.insert(fitParams, 0, minY)\r\n fn = lambda x, minY=minY: boundedFunction(x, *fitParams)\r\n except RuntimeError:\r\n print(\r\n \"couldn't fit noise function with filtered indices, use polynomial fit instead\")\r\n fitParams = None\r\n fn = smooth(xx, y, weights[i])\r\n return fitParams, fn, i"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlimit [ function ] to a minimum y value", "response": "def boundedFunction(x, minY, ax, ay):\r\n '''\r\n limit [function] to a minimum y value \r\n '''\r\n y = function(x, ax, ay)\r\n return np.maximum(np.nan_to_num(y), minY)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns indices that have enough data points and are not erroneous", "response": "def _validI(x, y, weights):\r\n '''\r\n return indices that have enough data points and are not erroneous \r\n '''\r\n # density filter:\r\n i = np.logical_and(np.isfinite(y), weights > np.median(weights))\r\n # filter outliers:\r\n try:\r\n grad = np.abs(np.gradient(y[i]))\r\n max_gradient = 4 * np.median(grad)\r\n i[i][grad > max_gradient] = False\r\n except (IndexError, ValueError):\r\n pass\r\n return i"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsmooths the NLF with respect to the given weights", "response": "def smooth(x, y, weights):\r\n '''\r\n in case the NLF cannot be described by \r\n a square root function\r\n commit bounded polynomial interpolation\r\n '''\r\n # Spline hard to smooth properly, therefore solfed with\r\n # bounded polynomal interpolation\r\n # ext=3: no extrapolation, but boundary value\r\n# return UnivariateSpline(x, y, w=weights,\r\n# s=len(y)*weights.max()*100, ext=3)\r\n\r\n# return np.poly1d(np.polyfit(x,y,w=weights,deg=2))\r\n p = np.polyfit(x, y, w=weights, deg=2)\r\n if np.any(np.isnan(p)):\r\n # couldn't even do polynomial fit\r\n # as last option: assume constant noise\r\n my = np.average(y, weights=weights)\r\n return lambda x: my\r\n return lambda xint: np.poly1d(p)(np.clip(xint, x[0], x[-1]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef oneImageNLF(img, img2=None, signal=None):\r\n '''\r\n Estimate the NLF from one or two images of the same kind\r\n '''\r\n x, y, weights, signal = calcNLF(img, img2, signal)\r\n _, fn, _ = _evaluate(x, y, weights)\r\n return fn, signal", "response": "Estimate the NLF from one or two images of the same kind"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the a range of image intensities that most pixels are in with", "response": "def _getMinMax(img):\r\n '''\r\n Get the a range of image intensities\r\n that most pixels are in with\r\n '''\r\n av = np.mean(img)\r\n std = np.std(img)\r\n # define range for segmentation:\r\n mn = av - 3 * std\r\n mx = av + 3 * std\r\n\r\n return max(img.min(), mn, 0), min(img.max(), mx)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the noise level function for a single image and a signal.", "response": "def calcNLF(img, img2=None, signal=None, mn_mx_nbins=None, x=None,\r\n averageFn='AAD',\r\n signalFromMultipleImages=False):\r\n '''\r\n Calculate the noise level function (NLF) as f(intensity)\r\n using one or two image.\r\n The approach for this work is published in JPV##########\r\n\r\n img2 - 2nd image taken under same conditions\r\n used to estimate noise via image difference\r\n\r\n signalFromMultipleImages - whether the signal is an average of multiple\r\n images and not just got from one median filtered image\r\n '''\r\n # CONSTANTS:\r\n # factor Root mead square to average-absolute-difference:\r\n F_RMS2AAD = (2 / np.pi)**-0.5\r\n F_NOISE_WITH_MEDIAN = 1 + (1 / 3**2)\r\n N_BINS = 100\r\n MEDIAN_KERNEL_SIZE = 3\r\n\r\n def _averageAbsoluteDeviation(d):\r\n return np.mean(np.abs(d)) * F_RMS2AAD\r\n\r\n def _rootMeanSquare(d):\r\n return (d**2).mean()**0.5\r\n\r\n if averageFn == 'AAD':\r\n averageFn = _averageAbsoluteDeviation\r\n else:\r\n averageFn = _rootMeanSquare\r\n\r\n img = np.asfarray(img)\r\n\r\n if img2 is None:\r\n if signal is None:\r\n signal = median_filter(img, MEDIAN_KERNEL_SIZE)\r\n if signalFromMultipleImages:\r\n diff = img - signal\r\n else:\r\n # difference between the filtered and original image:\r\n diff = (img - signal) * F_NOISE_WITH_MEDIAN\r\n\r\n else:\r\n img2 = np.asfarray(img2)\r\n diff = (img - img2)\r\n # 2**0.5 because noise is subtracted by noise\r\n # and variance of sum = sum of variance:\r\n # var(immg1-img2)~2*var(img)\r\n # std(2*var) = 2**0.5*var**0.5\r\n diff /= 2**0.5\r\n if signal is None:\r\n signal = median_filter(0.5 * (img + img2), MEDIAN_KERNEL_SIZE)\r\n\r\n if mn_mx_nbins is not None:\r\n mn, mx, nbins = mn_mx_nbins\r\n min_len = 0\r\n else:\r\n mn, mx = _getMinMax(signal)\r\n s = img.shape\r\n min_len = int(s[0] * s[1] * 1e-3)\r\n if min_len < 1:\r\n min_len = 5\r\n # number of bins/different intensity ranges to analyse:\r\n nbins = N_BINS\r\n if mx - mn < nbins:\r\n nbins = int(mx - mn)\r\n # bin width:\r\n step = (mx - mn) / nbins\r\n\r\n # empty arrays:\r\n y = np.empty(shape=nbins)\r\n set_x = False\r\n if x is None:\r\n set_x = True\r\n x = np.empty(shape=nbins)\r\n # give bins with more samples more weight:\r\n weights = np.zeros(shape=nbins)\r\n\r\n # cur step:\r\n m = mn\r\n for n in range(nbins):\r\n # get indices of all pixel with in a bin:\r\n ind = np.logical_and(signal >= m, signal <= m + step)\r\n m += step\r\n d = diff[ind]\r\n ld = len(d)\r\n if ld >= min_len:\r\n weights[n] = ld\r\n # average absolute deviation (AAD),\r\n # scaled to RMS:\r\n y[n] = averageFn(d)\r\n if set_x:\r\n x[n] = m - 0.5 * step\r\n\r\n return x, y, weights, signal"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a new array with the polynomial fitted values in the order specified by the mask and order specified.", "response": "def polyfit2dGrid(arr, mask=None, order=3, replace_all=False,\r\n copy=True, outgrid=None):\r\n '''\r\n replace all masked values with polynomial fitted ones\r\n '''\r\n s0,s1 = arr.shape\r\n if mask is None:\r\n if outgrid is None:\r\n y,x = np.mgrid[:float(s0),:float(s1)]\r\n p = polyfit2d(x.flatten(),y.flatten(),arr.flatten(),order)\r\n return polyval2d(x,y, p, dtype=arr.dtype)\r\n mask = np.zeros_like(arr, dtype=bool)\r\n elif mask.sum() == 0 and not replace_all and outgrid is None:\r\n\r\n return arr\r\n valid = ~mask\r\n y,x = np.where(valid)\r\n z = arr[valid]\r\n p = polyfit2d(x,y,z,order)\r\n if outgrid is not None:\r\n yy,xx = outgrid\r\n else:\r\n if replace_all:\r\n yy,xx = np.mgrid[:float(s0),:float(s1)]\r\n else:\r\n yy,xx = np.where(mask)\r\n new = polyval2d(xx,yy, p, dtype=arr.dtype)\r\n \r\n if outgrid is not None or replace_all:\r\n return new\r\n \r\n if copy:\r\n arr = arr.copy()\r\n arr[mask] = new\r\n return arr"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef minimumLineInArray(arr, relative=False, f=0,\r\n refinePosition=True,\r\n max_pos=100,\r\n return_pos_arr=False,\r\n # order=2\r\n ):\r\n '''\r\n find closest minimum position next to middle line\r\n relative: return position relative to middle line\r\n f: relative decrease (0...1) - setting this value close to one will\r\n discriminate positions further away from the center\r\n ##order: 2 for cubic refinement\r\n '''\r\n s0, s1 = arr.shape[:2]\r\n if max_pos >= s1:\r\n x = np.arange(s1)\r\n else:\r\n # take fewer positions within 0->(s1-1)\r\n x = np.rint(np.linspace(0, s1 - 1, min(max_pos, s1))).astype(int)\r\n res = np.empty((s0, s0), dtype=float)\r\n\r\n _lineSumXY(x, res, arr, f)\r\n\r\n if return_pos_arr:\r\n return res\r\n\r\n # best integer index\r\n i, j = np.unravel_index(np.nanargmin(res), res.shape)\r\n\r\n if refinePosition:\r\n try:\r\n sub = res[i - 1:i + 2, j - 1:j + 2]\r\n ii, jj = center_of_mass(sub)\r\n if not np.isnan(ii):\r\n i += (ii - 1)\r\n if not np.isnan(jj):\r\n j += (jj - 1)\r\n except TypeError:\r\n pass\r\n\r\n\r\n if not relative:\r\n return i, j\r\n\r\n hs = (s0 - 1) / 2\r\n return i - hs, j - hs", "response": "Returns the minimum position in an array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef highPassFilter(self, threshold):\r\n '''\r\n remove all low frequencies by setting a square in the middle of the\r\n Fourier transformation of the size (2*threshold)^2 to zero\r\n threshold = 0...1\r\n '''\r\n if not threshold:\r\n return\r\n rows, cols = self.img.shape\r\n tx = int(cols * threshold)\r\n ty = int(rows * threshold)\r\n # middle:\r\n crow, ccol = rows // 2, cols // 2\r\n # square in the middle to zero\r\n self.fshift[crow - tx:crow + tx, ccol - ty:ccol + ty] = 0", "response": "Remove all low frequencies by setting a square in the middle of the Fourier transformation of the size ( 2 * threshold^2 to zero"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reconstructImage(self):\r\n '''\r\n do inverse Fourier transform and return result\r\n '''\r\n f_ishift = np.fft.ifftshift(self.fshift)\r\n return np.real(np.fft.ifft2(f_ishift))", "response": "reconstruct the image from the Fourier transform and return the real image"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the histogram of Oriented Gradients", "response": "def hog(image, orientations=8, ksize=(5, 5)):\r\n '''\r\n returns the Histogram of Oriented Gradients\r\n\r\n :param ksize: convolution kernel size as (y,x) - needs to be odd\r\n :param orientations: number of orientations in between rad=0 and rad=pi\r\n\r\n similar to http://scikit-image.org/docs/dev/auto_examples/plot_hog.html\r\n but faster and with less options\r\n '''\r\n s0, s1 = image.shape[:2]\r\n\r\n # speed up the process through saving generated kernels:\r\n try:\r\n k = hog.kernels[str(ksize) + str(orientations)]\r\n except KeyError:\r\n k = _mkConvKernel(ksize, orientations)\r\n hog.kernels[str(ksize) + str(orientations)] = k\r\n\r\n out = np.empty(shape=(s0, s1, orientations))\r\n image[np.isnan(image)] = 0\r\n\r\n for i in range(orientations):\r\n out[:, :, i] = convolve(image, k[i])\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvisualizes HOG as polynomial around cell center and radius of each sub array.", "response": "def visualize(hog, grid=(10, 10), radCircle=None):\r\n '''\r\n visualize HOG as polynomial around cell center\r\n for [grid] * cells\r\n '''\r\n s0, s1, nang = hog.shape\r\n angles = np.linspace(0, np.pi, nang + 1)[:-1]\r\n # center of each sub array:\r\n cx, cy = s0 // (2 * grid[0]), s1 // (2 * grid[1])\r\n # max. radius of polynomial around cenetr:\r\n rx, ry = cx, cy\r\n # for drawing a position indicator (circle):\r\n if radCircle is None:\r\n radCircle = max(1, rx // 10)\r\n # output array:\r\n out = np.zeros((s0, s1), dtype=np.uint8)\r\n # point of polynomial:\r\n pts = np.empty(shape=(1, 2 * nang, 2), dtype=np.int32)\r\n # takes grid[0]*grid[1] sample HOG values:\r\n samplesHOG = subCell2DFnArray(hog, lambda arr: arr[cx, cy], grid)\r\n mxHOG = samplesHOG.max()\r\n # sub array slices:\r\n slices = list(subCell2DSlices(out, grid))\r\n m = 0\r\n for m, hhh in enumerate(samplesHOG.reshape(grid[0] * grid[1], nang)):\r\n hhmax = hhh.max()\r\n hh = hhh / hhmax\r\n sout = out[slices[m][2:4]]\r\n for n, (o, a) in enumerate(zip(hh, angles)):\r\n pts[0, n, 0] = cx + np.cos(a) * o * rx\r\n pts[0, n, 1] = cy + np.sin(a) * o * ry\r\n pts[0, n + nang, 0] = cx + np.cos(a + np.pi) * o * rx\r\n pts[0, n + nang, 1] = cy + np.sin(a + np.pi) * o * ry\r\n\r\n cv2.fillPoly(sout, pts, int(255 * hhmax / mxHOG))\r\n cv2.circle(sout, (cx, cy), radCircle, 0, thickness=-1)\r\n\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a new array with the same shape as the array arr.", "response": "def postProcessing(arr, method='KW replace + Gauss', mask=None):\r\n '''\r\n Post process measured flat field [arr].\r\n Depending on the measurement, different\r\n post processing [method]s are beneficial.\r\n The available methods are presented in\r\n ---\r\n K.Bedrich, M.Bokalic et al.:\r\n ELECTROLUMINESCENCE IMAGING OF PV DEVICES:\r\n ADVANCED FLAT FIELD CALIBRATION,2017\r\n ---\r\n\r\n methods:\r\n 'POLY replace' --> replace [arr] with a 2d polynomial fit\r\n 'KW replace' --> ... a fitted Kang-Weiss function\r\n 'AoV replace' --> ... a fitted Angle-of-view function\r\n\r\n 'POLY repair' --> same as above but either replacing empty\r\n 'KW repair' areas of smoothing out high gradient\r\n 'AoV repair' variations (POLY only)\r\n\r\n 'KW repair + Gauss' --> same as 'KW replace' with additional \r\n 'KW repair + Median' Gaussian or Median filter\r\n\r\n mask:\r\n None/2darray(bool) --> array of same shape ar [arr] indicating\r\n invalid or empty positions\r\n '''\r\n assert method in ppMETHODS, \\\r\n 'post processing method (%s) must be one of %s' % (method, ppMETHODS)\r\n\r\n if method == 'POLY replace':\r\n return polyfit2dGrid(arr, mask, order=2, replace_all=True)\r\n\r\n elif method == 'KW replace':\r\n return function(arr, mask, replace_all=True)\r\n\r\n elif method == 'POLY repair':\r\n return polynomial(arr, mask, replace_all=False)\r\n\r\n elif method == 'KW repair':\r\n return function(arr, mask, replace_all=False)\r\n\r\n elif method == 'KW repair + Median':\r\n return median_filter(function(arr, mask, replace_all=False),\r\n min(method.shape) // 20)\r\n elif method == 'KW repair + Gauss':\r\n return gaussian_filter(function(arr, mask, replace_all=False),\r\n min(arr.shape) // 20)\r\n\r\n elif method == 'AoV repair':\r\n return function(arr, mask, fn=lambda XY, a:\r\n angleOfView(XY, method.shape, a=a), guess=(0.01),\r\n down_scale_factor=1)\r\n\r\n elif method == 'AoV replace':\r\n return function(arr, mask, fn=lambda XY, a:\r\n angleOfView(XY, arr.shape, a=a), guess=(0.01),\r\n replace_all=True, down_scale_factor=1)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rmBorder(img, border=None):\r\n '''\r\n border [None], if images are corrected and device ends at\r\n image border\r\n [one number] (like 50), \r\n if there is an equally spaced border\r\n around the device\r\n [two tuples] like ((50,60),(1500,900))\r\n means ((Xfrom,Yfrom),(Xto, Yto))\r\n [four tuples] like ((x0,y0),(x1,y1),...(x3,y3))\r\n '''\r\n if border is None:\r\n pass\r\n elif len(border) == 2:\r\n s0 = slice(border[0][1], border[1][1])\r\n s1 = slice(border[0][0], border[1][0])\r\n img = img[s0, s1]\r\n elif len(border) == 4:\r\n # eval whether border values are orthogonal:\r\n x = np.unique(border[:, 0])\r\n y = np.unique(border[:, 1])\r\n if len(x) == 2 and len(y) == 2:\r\n s0 = slice(y[0], y[1])\r\n s1 = slice(x[0], x[1])\r\n img = img[s0, s1]\r\n else:\r\n # edges are irregular:\r\n img = simplePerspectiveTransform(img, border)\r\n else:\r\n raise Exception('[border] input wrong')\r\n return img", "response": "Remove the border of an image."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding an image to the list of entries in the cache.", "response": "def addImage(self, image, mask=None):\r\n '''\r\n #########\r\n mask -- optional\r\n '''\r\n self._last_diff = diff = image - self.noSTE\r\n\r\n ste = diff > self.threshold\r\n removeSinglePixels(ste)\r\n\r\n self.mask_clean = clean = ~ste\r\n\r\n if mask is not None:\r\n clean = np.logical_and(mask, clean)\r\n\r\n self.mma.update(image, clean)\r\n\r\n if self.save_ste_indices:\r\n self.mask_STE += ste\r\n\r\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef relativeAreaSTE(self):\r\n '''\r\n return STE area - relative to image area\r\n '''\r\n s = self.noSTE.shape\r\n return np.sum(self.mask_STE) / (s[0] * s[1])", "response": "Returns the STE area relative to image area"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef intensityDistributionSTE(self, bins=10, range=None):\r\n '''\r\n return distribution of STE intensity\r\n '''\r\n v = np.abs(self._last_diff[self.mask_STE])\r\n return np.histogram(v, bins, range)", "response": "Returns the STE intensity distribution of the last set of STE"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef toUIntArray(img, dtype=None, cutNegative=True, cutHigh=True,\r\n range=None, copy=True):\r\n '''\r\n transform a float to an unsigned integer array of a fitting dtype\r\n adds an offset, to get rid of negative values\r\n\r\n range = (min, max) - scale values between given range\r\n\r\n cutNegative - all values <0 will be set to 0\r\n cutHigh - set to False to rather scale values to fit\r\n '''\r\n mn, mx = None, None\r\n if range is not None:\r\n mn, mx = range\r\n\r\n if dtype is None:\r\n if mx is None:\r\n mx = np.nanmax(img)\r\n dtype = np.uint16 if mx > 255 else np.uint8\r\n\r\n dtype = np.dtype(dtype)\r\n if dtype == img.dtype:\r\n return img\r\n\r\n # get max px value:\r\n b = {'uint8': 255,\r\n 'uint16': 65535,\r\n 'uint32': 4294967295,\r\n 'uint64': 18446744073709551615}[dtype.name]\r\n\r\n if copy:\r\n img = img.copy()\r\n\r\n if range is not None:\r\n img = np.asfarray(img)\r\n img -= mn\r\n # img[img<0]=0\r\n # print np.nanmin(img), np.nanmax(img), mn, mx, range, b\r\n\r\n img *= b / (mx - mn)\r\n # print np.nanmin(img), np.nanmax(img), mn, mx, range, b\r\n img = np.clip(img, 0, b)\r\n\r\n else:\r\n if cutNegative:\r\n img[img < 0] = 0\r\n else:\r\n # add an offset to all values:\r\n mn = np.min(img)\r\n if mn < 0:\r\n img -= mn # set minimum to 0\r\n\r\n if cutHigh:\r\n #ind = img > b\r\n img[img > b] = b\r\n else:\r\n # scale values\r\n mx = np.nanmax(img)\r\n img = np.asfarray(img) * (float(b) / mx)\r\n\r\n img = img.astype(dtype)\r\n\r\n# if range is not None and cutHigh:\r\n# img[ind] = b\r\n return img", "response": "transform a float to an unsigned integer array of a fitting dtype cutNegative cutHigh - all values < 0 will be set to False to get rid of negative values from fit\r\n range - all values from fit\r\n range - all values from fit\r\n range - all values from fit\r\n range - all values from fit\r\n range - all values from fit\r\n range - all values from fit\r\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef toFloatArray(img):\r\n '''\r\n transform an unsigned integer array into a\r\n float array of the right size\r\n '''\r\n _D = {1: np.float32, # uint8\r\n 2: np.float32, # uint16\r\n 4: np.float64, # uint32\r\n 8: np.float64} # uint64\r\n return img.astype(_D[img.itemsize])", "response": "transform an unsigned integer array into a float array of the right size\r\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncasts array to the next higher integer array if dtype = unsigned integer", "response": "def toNoUintArray(arr):\r\n '''\r\n cast array to the next higher integer array\r\n if dtype=unsigned integer\r\n '''\r\n d = arr.dtype\r\n if d.kind == 'u':\r\n arr = arr.astype({1: np.int16,\r\n 2: np.int32,\r\n 4: np.int64}[d.itemsize])\r\n return arr"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a single image to grayscale.", "response": "def toGray(img):\r\n '''\r\n weights see\r\n https://en.wikipedia.org/wiki/Grayscale#Colorimetric_.28luminance-prese\r\n http://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor\r\n '''\r\n return np.average(img, axis=-1, weights=(0.299, # red\r\n 0.587, # green\r\n 0.114) # blue\r\n ).astype(img.dtype)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the normalized RGB space RGB / intensity", "response": "def rgChromaticity(img):\r\n '''\r\n returns the normalized RGB space (RGB/intensity)\r\n see https://en.wikipedia.org/wiki/Rg_chromaticity\r\n '''\r\n out = _calc(img)\r\n if img.dtype == np.uint8:\r\n out = (255 * out).astype(np.uint8)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the monochromatic wavelength of the given image.", "response": "def monochromaticWavelength(img):\r\n '''\r\n TODO##########\r\n '''\r\n # peak wave lengths: https://en.wikipedia.org/wiki/RGB_color_model\r\n out = _calc(img)\r\n\r\n peakWavelengths = (570, 540, 440) # (r,g,b)\r\n# s = sum(peakWavelengths)\r\n for n, p in enumerate(peakWavelengths):\r\n out[..., n] *= p\r\n return out.sum(axis=2)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rot90(img):\r\n '''\r\n rotate one or multiple grayscale or color images 90 degrees\r\n '''\r\n s = img.shape\r\n if len(s) == 3:\r\n if s[2] in (3, 4): # color image\r\n out = np.empty((s[1], s[0], s[2]), dtype=img.dtype)\r\n for i in range(s[2]):\r\n out[:, :, i] = np.rot90(img[:, :, i])\r\n else: # mutliple grayscale\r\n out = np.empty((s[0], s[2], s[1]), dtype=img.dtype)\r\n for i in range(s[0]):\r\n out[i] = np.rot90(img[i])\r\n elif len(s) == 2: # one grayscale\r\n out = np.rot90(img)\r\n elif len(s) == 4 and s[3] in (3, 4): # multiple color\r\n out = np.empty((s[0], s[2], s[1], s[3]), dtype=img.dtype)\r\n for i in range(s[0]): # for each img\r\n for j in range(s[3]): # for each channel\r\n out[i, :, :, j] = np.rot90(img[i, :, :, j])\r\n else:\r\n NotImplemented\r\n return out", "response": "rotate one or multiple grayscale or color images 90 degrees\r\nInsights"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef applyColorMap(gray, cmap='flame'):\r\n '''\r\n like cv2.applyColorMap(im_gray, cv2.COLORMAP_*) but with different color maps\r\n '''\r\n # TODO:implement more cmaps\r\n if cmap != 'flame':\r\n raise NotImplemented\r\n # TODO: make better\r\n mx = 256 # if gray.dtype==np.uint8 else 65535\r\n lut = np.empty(shape=(256, 3))\r\n cmap = (\r\n # taken from pyqtgraph GradientEditorItem\r\n (0, (0, 0, 0)),\r\n (0.2, (7, 0, 220)),\r\n (0.5, (236, 0, 134)),\r\n (0.8, (246, 246, 0)),\r\n (1.0, (255, 255, 255))\r\n )\r\n # build lookup table:\r\n lastval, lastcol = cmap[0]\r\n for step, col in cmap[1:]:\r\n val = int(step * mx)\r\n for i in range(3):\r\n lut[lastval:val, i] = np.linspace(\r\n lastcol[i], col[i], val - lastval)\r\n\r\n lastcol = col\r\n lastval = val\r\n\r\n s0, s1 = gray.shape\r\n out = np.empty(shape=(s0, s1, 3), dtype=np.uint8)\r\n\r\n for i in range(3):\r\n out[..., i] = cv2.LUT(gray, lut[:, i])\r\n return out", "response": "Like cv2. applyColorMap but with different color maps"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _insertDateIndex(date, l):\r\n '''\r\n returns the index to insert the given date in a list\r\n where each items first value is a date\r\n '''\r\n return next((i for i, n in enumerate(l) if n[0] < date), len(l))", "response": "Returns the index to insert the given date in a list\r\n where each item is a date"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the index of given or best fitting date in list l", "response": "def _getFromDate(l, date):\r\n '''\r\n returns the index of given or best fitting date\r\n '''\r\n try:\r\n date = _toDate(date)\r\n i = _insertDateIndex(date, l) - 1\r\n if i == -1:\r\n return l[0]\r\n return l[i]\r\n except (ValueError, TypeError):\r\n # ValueError: date invalid / TypeError: date = None\r\n return l[0]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dates(self, typ, light=None):\r\n '''\r\n Args:\r\n typ: type of calibration to look for. See .coeffs.keys() for all types available\r\n light (Optional[str]): restrict to calibrations, done given light source\r\n\r\n Returns:\r\n list: All calibration dates available for given typ\r\n '''\r\n try:\r\n d = self._getDate(typ, light)\r\n return [self._toDateStr(c[0]) for c in d]\r\n except KeyError:\r\n return []", "response": "Returns a list of dates available for the given type"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef infos(self, typ, light=None, date=None):\r\n '''\r\n Args:\r\n typ: type of calibration to look for. See .coeffs.keys() for all types available\r\n date (Optional[str]): date of calibration\r\n\r\n Returns:\r\n list: all infos available for given typ\r\n '''\r\n d = self._getDate(typ, light)\r\n if date is None:\r\n return [c[1] for c in d]\r\n # TODO: not struct time, but time in ms since epoch\r\n return _getFromDate(d, date)[1]", "response": "Returns a list of infos available for a given type and light."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn an overview of all calibrations of all calibrations of all the calendars in the calendar.", "response": "def overview(self):\r\n '''\r\n Returns:\r\n str: an overview covering all calibrations \r\n infos and shapes\r\n '''\r\n c = self.coeffs\r\n out = 'camera name: %s' % c['name']\r\n out += '\\nmax value: %s' % c['depth']\r\n out += '\\nlight spectra: %s' % c['light spectra']\r\n\r\n out += '\\ndark current:'\r\n for (date, info, (slope, intercept), error) in c['dark current']:\r\n out += '\\n\\t date: %s' % self._toDateStr(date)\r\n out += '\\n\\t\\t info: %s; slope:%s, intercept:%s' % (\r\n info, slope.shape, intercept.shape)\r\n\r\n out += '\\nflat field:'\r\n for light, vals in c['flat field'].items():\r\n out += '\\n\\t light: %s' % light\r\n for (date, info, arr, error) in vals:\r\n out += '\\n\\t\\t date: %s' % self._toDateStr(date)\r\n out += '\\n\\t\\t\\t info: %s; array:%s' % (info, arr.shape)\r\n\r\n out += '\\nlens:'\r\n for light, vals in c['lens'].items():\r\n out += '\\n\\t light: %s' % light\r\n for (date, info, coeffs) in vals:\r\n out += '\\n\\t\\t date: %s' % self._toDateStr(date)\r\n out += '\\n\\t\\t\\t info: %s; coeffs:%s' % (info, coeffs)\r\n\r\n out += '\\nnoise:'\r\n for (date, info, nlf_coeff, error) in c['noise']:\r\n out += '\\n\\t date: %s' % self._toDateStr(date)\r\n out += '\\n\\t\\t info: %s; coeffs:%s' % (info, nlf_coeff)\r\n\r\n out += '\\nPoint spread function:'\r\n for light, vals in c['psf'].items():\r\n out += '\\n\\t light: %s' % light\r\n for (date, info, psf) in vals:\r\n out += '\\n\\t\\t date: %s' % self._toDateStr(date)\r\n out += '\\n\\t\\t\\t info: %s; shape:%s' % (info, psf.shape)\r\n\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setCamera(self, camera_name, bit_depth=16):\r\n '''\r\n Args:\r\n camera_name (str): Name of the camera\r\n bit_depth (int): depth (bit) of the camera sensor\r\n '''\r\n self.coeffs['name'] = camera_name\r\n self.coeffs['depth'] = bit_depth", "response": "Sets the name and depth of the camera sensor with the name camera_name and bit_depth."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef addDarkCurrent(self, slope, intercept=None, date=None, info='', error=None):\r\n '''\r\n Args:\r\n slope (np.array)\r\n intercept (np.array)\r\n error (numpy.array)\r\n slope (float): dPx/dExposureTime[sec]\r\n error (float): absolute\r\n date (str): \"DD Mon YY\" e.g. \"30 Nov 16\"\r\n '''\r\n date = _toDate(date)\r\n\r\n self._checkShape(slope)\r\n self._checkShape(intercept)\r\n\r\n d = self.coeffs['dark current']\r\n if intercept is None:\r\n data = slope\r\n else:\r\n data = (slope, intercept)\r\n d.insert(_insertDateIndex(date, d), [date, info, data, error])", "response": "Adds a dark current to the calendar."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef addNoise(self, nlf_coeff, date=None, info='', error=None):\r\n '''\r\n Args:\r\n nlf_coeff (list)\r\n error (float): absolute\r\n info (str): additional information\r\n date (str): \"DD Mon YY\" e.g. \"30 Nov 16\"\r\n '''\r\n date = _toDate(date)\r\n d = self.coeffs['noise']\r\n d.insert(_insertDateIndex(date, d), [date, info, nlf_coeff, error])", "response": "Adds noise to the entry in the dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef addFlatField(self, arr, date=None, info='', error=None,\r\n light_spectrum='visible'):\r\n '''\r\n light_spectrum = light, IR ...\r\n '''\r\n self._registerLight(light_spectrum)\r\n self._checkShape(arr)\r\n date = _toDate(date)\r\n f = self.coeffs['flat field']\r\n if light_spectrum not in f:\r\n f[light_spectrum] = []\r\n f[light_spectrum].insert(_insertDateIndex(date, f[light_spectrum]),\r\n [date, info, arr, error])", "response": "Add a flat field to the log."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd lens to the list of available lights", "response": "def addLens(self, lens, date=None, info='', light_spectrum='visible'):\r\n '''\r\n lens -> instance of LensDistortion or saved file\r\n '''\r\n self._registerLight(light_spectrum)\r\n date = _toDate(date)\r\n\r\n if not isinstance(lens, LensDistortion):\r\n l = LensDistortion()\r\n l.readFromFile(lens)\r\n lens = l\r\n\r\n f = self.coeffs['lens']\r\n if light_spectrum not in f:\r\n f[light_spectrum] = []\r\n f[light_spectrum].insert(_insertDateIndex(date, f[light_spectrum]),\r\n [date, info, lens.coeffs])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove all old calibration entries except the youngest one.", "response": "def clearOldCalibrations(self, date=None):\r\n '''\r\n if not only a specific date than remove all except of the youngest calibration\r\n '''\r\n self.coeffs['dark current'] = [self.coeffs['dark current'][-1]]\r\n self.coeffs['noise'] = [self.coeffs['noise'][-1]]\r\n\r\n for light in self.coeffs['flat field']:\r\n self.coeffs['flat field'][light] = [\r\n self.coeffs['flat field'][light][-1]]\r\n for light in self.coeffs['lens']:\r\n self.coeffs['lens'][light] = [self.coeffs['lens'][light][-1]]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntransposing all calibration arrays in case different array shape orders were used", "response": "def transpose(self):\r\n '''\r\n transpose all calibration arrays\r\n in case different array shape orders were used (x,y) vs. (y,x)\r\n '''\r\n def _t(item):\r\n if type(item) == list:\r\n for n, it in enumerate(item):\r\n if type(it) == tuple:\r\n it = list(it)\r\n item[n] = it\r\n if type(it) == list:\r\n _t(it)\r\n if isinstance(it, np.ndarray) and it.shape == s:\r\n item[n] = it.T\r\n\r\n s = self.coeffs['shape']\r\n\r\n for item in self.coeffs.values():\r\n if type(item) == dict:\r\n for item2 in item.values():\r\n _t(item2)\r\n else:\r\n _t(item)\r\n\r\n self.coeffs['shape'] = s[::-1]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _correctNoise(self, image):\r\n '''\r\n denoise using non-local-means\r\n with guessing best parameters\r\n '''\r\n from skimage.restoration import denoise_nl_means # save startup time\r\n image[np.isnan(image)] = 0 # otherwise result =nan\r\n out = denoise_nl_means(image,\r\n patch_size=7,\r\n patch_distance=11,\r\n #h=signalStd(image) * 0.1\r\n )\r\n\r\n return out", "response": "Corrects the image using non - local - means\r\n with guessing best parameters"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncorrect dark current image", "response": "def _correctDarkCurrent(self, image, exposuretime, bgImages, date):\r\n '''\r\n open OR calculate a background image: f(t)=m*t+n\r\n '''\r\n # either exposureTime or bgImages has to be given\r\n# if exposuretime is not None or bgImages is not None:\r\n print('... remove dark current')\r\n\r\n if bgImages is not None:\r\n\r\n if (type(bgImages) in (list, tuple) or\r\n (isinstance(bgImages, np.ndarray) and\r\n bgImages.ndim == 3)):\r\n if len(bgImages) > 1:\r\n # if multiple images are given: do STE removal:\r\n nlf = self.noise_level_function\r\n bg = SingleTimeEffectDetection(\r\n bgImages, nStd=4,\r\n noise_level_function=nlf).noSTE\r\n else:\r\n bg = imread(bgImages[0])\r\n else:\r\n bg = imread(bgImages)\r\n else:\r\n bg = self.calcDarkCurrent(exposuretime, date)\r\n self.temp['bg'] = bg\r\n image -= bg"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _correctArtefacts(self, image, threshold):\r\n '''\r\n Apply a thresholded median replacing high gradients \r\n and values beyond the boundaries\r\n '''\r\n image = np.nan_to_num(image)\r\n medianThreshold(image, threshold, copy=False)\r\n return image", "response": "Apply a thresholded median replacing high gradients \r\n and values beyond the boundaries"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getCoeff(self, name, light=None, date=None):\r\n '''\r\n try to get calibration for right light source, but\r\n use another if they is none existent\r\n '''\r\n d = self.coeffs[name]\r\n\r\n try:\r\n c = d[light]\r\n except KeyError:\r\n try:\r\n k, i = next(iter(d.items()))\r\n if light is not None:\r\n print(\r\n 'no calibration found for [%s] - using [%s] instead' % (light, k))\r\n except StopIteration:\r\n return None\r\n c = i\r\n except TypeError:\r\n # coeff not dependent on light source\r\n c = d\r\n return _getFromDate(c, date)", "response": "get the most recent calibration for a given name light and date"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a tuple of the smoothed and mask of the objects in the objectVignettingSeparation objects.", "response": "def vignettingFromRandomSteps(imgs, bg, inPlane_scale_factor=None,\r\n debugFolder=None, **kwargs):\r\n '''\r\n important: first image should shown most iof the device\r\n because it is used as reference\r\n '''\r\n # TODO: inPlane_scale_factor\r\n if debugFolder:\r\n debugFolder = PathStr(debugFolder)\r\n\r\n s = ObjectVignettingSeparation(imgs[0], bg, **kwargs)\r\n for img in imgs[1:]:\r\n fit = s.addImg(img)\r\n\r\n if debugFolder and fit is not False:\r\n imwrite(debugFolder.join('fit_%s.tiff' % len(s.fits)), fit)\r\n\r\n if debugFolder:\r\n imwrite(debugFolder.join('init.tiff'), s.flatField)\r\n\r\n smoothed_ff, mask, flatField, obj = s.separate()\r\n\r\n if debugFolder:\r\n imwrite(debugFolder.join('object.tiff'), obj)\r\n imwrite(debugFolder.join('flatfield.tiff'), flatField, dtype=float)\r\n imwrite(debugFolder.join('flatfield_smoothed.tiff'), smoothed_ff,\r\n dtype=float)\r\n\r\n return smoothed_ff, mask"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef addImg(self, img, maxShear=0.015, maxRot=100, minMatches=12,\r\n borderWidth=3): # borderWidth=100\r\n \"\"\"\r\n Args:\r\n img (path or array): image containing the same object as in the reference image\r\n Kwargs:\r\n maxShear (float): In order to define a good fit, refect higher shear values between\r\n this and the reference image\r\n maxRot (float): Same for rotation\r\n minMatches (int): Minimum of mating points found in both, this and the reference image\r\n \"\"\"\r\n try:\r\n fit, img, H, H_inv, nmatched = self._fitImg(img)\r\n except Exception as e:\r\n print(e)\r\n return\r\n\r\n # CHECK WHETHER FIT IS GOOD ENOUGH:\r\n (translation, rotation, scale, shear) = decompHomography(H)\r\n print('Homography ...\\n\\ttranslation: %s\\n\\trotation: %s\\n\\tscale: %s\\n\\tshear: %s'\r\n % (translation, rotation, scale, shear))\r\n if (nmatched > minMatches\r\n and abs(shear) < maxShear\r\n and abs(rotation) < maxRot):\r\n print('==> img added')\r\n # HOMOGRAPHY:\r\n self.Hs.append(H)\r\n # INVERSE HOMOGRSAPHY\r\n self.Hinvs.append(H_inv)\r\n # IMAGES WARPED TO THE BASE IMAGE\r\n self.fits.append(fit)\r\n # ADD IMAGE TO THE INITIAL flatField ARRAY:\r\n i = img > self.signal_ranges[-1][0]\r\n\r\n # remove borders (that might have erroneous light):\r\n i = minimum_filter(i, borderWidth)\r\n\r\n self._ff_mma.update(img, i)\r\n\r\n # create fit img mask:\r\n mask = fit < self.signal_ranges[-1][0]\r\n mask = maximum_filter(mask, borderWidth)\r\n # IGNORE BORDER\r\n r = self.remove_border_size\r\n if r:\r\n mask[:r, :] = 1\r\n mask[-r:, :] = 1\r\n mask[:, -r:] = 1\r\n mask[:, :r] = 1\r\n self._fit_masks.append(mask)\r\n\r\n # image added\r\n return fit\r\n return False", "response": "Add an image to the list of related objects."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef error(self, nCells=15):\r\n '''\r\n calculate the standard deviation of all fitted images, \r\n averaged to a grid\r\n '''\r\n s0, s1 = self.fits[0].shape\r\n aR = s0 / s1\r\n if aR > 1:\r\n ss0 = int(nCells)\r\n ss1 = int(ss0 / aR)\r\n else:\r\n ss1 = int(nCells)\r\n ss0 = int(ss1 * aR)\r\n L = len(self.fits)\r\n\r\n arr = np.array(self.fits)\r\n arr[np.array(self._fit_masks)] = np.nan\r\n avg = np.tile(np.nanmean(arr, axis=0), (L, 1, 1))\r\n arr = (arr - avg) / avg\r\n\r\n out = np.empty(shape=(L, ss0, ss1))\r\n\r\n with warnings.catch_warnings():\r\n warnings.simplefilter(\"ignore\", category=RuntimeWarning)\r\n\r\n for n, f in enumerate(arr):\r\n out[n] = subCell2DFnArray(f, np.nanmean, (ss0, ss1))\r\n\r\n return np.nanmean(out**2)**0.5", "response": "calculate the standard deviation of all fitted images averaged to a grid\r\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _fitImg(self, img):\r\n '''\r\n fit perspective and size of the input image to the reference image\r\n '''\r\n img = imread(img, 'gray')\r\n if self.bg is not None:\r\n img = cv2.subtract(img, self.bg)\r\n\r\n if self.lens is not None:\r\n img = self.lens.correct(img, keepSize=True)\r\n\r\n (H, _, _, _, _, _, _, n_matches) = self.findHomography(img)\r\n H_inv = self.invertHomography(H)\r\n\r\n s = self.obj_shape\r\n fit = cv2.warpPerspective(img, H_inv, (s[1], s[0]))\r\n return fit, img, H, H_inv, n_matches", "response": "fit the image to the reference image"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _findObject(self, img):\r\n '''\r\n Create a bounding box around the object within an image\r\n '''\r\n from imgProcessor.imgSignal import signalMinimum\r\n # img is scaled already\r\n i = img > signalMinimum(img) # img.max()/2.5\r\n # filter noise, single-time-effects etc. from mask:\r\n i = minimum_filter(i, 4)\r\n return boundingBox(i)", "response": "Find the object within an image"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef filterVerticalLines(arr, min_line_length=4):\r\n gy = arr.shape[0]\r\n gx = arr.shape[1]\r\n mn = min_line_length-1\r\n for i in range(gy):\r\n for j in range(gx):\r\n if arr[i,j]:\r\n for d in range(min_line_length):\r\n if not arr[i+d,j]:\r\n break\r\n if d == mn:\r\n d = 0\r\n while True:\r\n if not arr[i+d,j]:\r\n break\r\n arr[i+d,j] = 0\r\n d +=1", "response": "Remove vertical lines in boolean array if linelength < min_line_length"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the vignetting of a single image", "response": "def vignetting(xy, f=100, alpha=0, rot=0, tilt=0, cx=50, cy=50):\r\n '''\r\n Vignetting equation using the KANG-WEISS-MODEL\r\n see http://research.microsoft.com/en-us/um/people/sbkang/publications/eccv00.pdf \r\n\r\n f - focal length\r\n alpha - coefficient in the geometric vignetting factor\r\n tilt - tilt angle of a planar scene\r\n rot - rotation angle of a planar scene\r\n cx - image center, x\r\n cy - image center, y\r\n '''\r\n x, y = xy\r\n # distance to image center:\r\n dist = ((x - cx)**2 + (y - cy)**2)**0.5\r\n\r\n # OFF_AXIS ILLUMINATION FACTOR:\r\n A = 1.0 / (1 + (dist / f)**2)**2\r\n # GEOMETRIC FACTOR:\r\n if alpha != 0:\r\n G = (1 - alpha * dist)\r\n else:\r\n G = 1\r\n # TILT FACTOR:\r\n if tilt != 0:\r\n T = tiltFactor((x, y), f, tilt, rot, (cy, cx))\r\n else:\r\n T = 1\r\n return A * G * T"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the tilt factor of a single image", "response": "def tiltFactor(xy, f, tilt, rot, center=None):\r\n '''\r\n this function is extra to only cover vignetting through perspective distortion\r\n\r\n f - focal length [px]\r\n tau - tilt angle of a planar scene [radian]\r\n rot - rotation angle of a planar scene [radian]\r\n '''\r\n x, y = xy\r\n arr = np.cos(tilt) * (\r\n 1 + (np.tan(tilt) / f) * (\r\n x * np.sin(rot) - y * np.cos(rot)))**3\r\n return arr"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef imgAverage(images, copy=True):\r\n '''\r\n returns an image average\r\n\r\n works on many, also unloaded images\r\n minimises RAM usage\r\n '''\r\n i0 = images[0]\r\n out = imread(i0, dtype='float')\r\n if copy and id(i0) == id(out):\r\n out = out.copy()\r\n\r\n for i in images[1:]:\r\n out += imread(i, dtype='float')\r\n out /= len(images)\r\n return out", "response": "Returns an image average of the images in the order they appear."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a meshgrid that is offset within the image", "response": "def offsetMeshgrid(offset, grid, shape):\r\n '''\r\n Imagine you have cell averages [grid] on an image.\r\n the top-left position of [grid] within the image \r\n can be variable [offset]\r\n \r\n offset(x,y) \r\n e.g.(0,0) if no offset\r\n grid(nx,ny) resolution of smaller grid\r\n shape(x,y) -> output shape \r\n \r\n returns meshgrid to be used to upscale [grid] to [shape] resolution\r\n ''' \r\n g0,g1 = grid\r\n s0,s1 = shape\r\n o0, o1 = offset\r\n #rescale to small grid:\r\n o0 = - o0/ s0 * (g0-1)\r\n o1 = - o1/ s1 * (g1-1)\r\n\r\n xx,yy = np.meshgrid(np.linspace(o1, o1+g1-1, s1),\r\n np.linspace(o0,o0+g0-1, s0))\r\n return yy,xx"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef poisson(x, a, b, c, d=0):\r\n '''\r\n Poisson function\r\n a -> height of the curve's peak\r\n b -> position of the center of the peak\r\n c -> standard deviation \r\n d -> offset\r\n '''\r\n from scipy.misc import factorial #save startup time\r\n lamb = 1\r\n X = (x/(2*c)).astype(int)\r\n return a * (( lamb**X/factorial(X)) * np.exp(-lamb) ) +d", "response": "Returns the maximal Poisson distance of a peak with respect to a center of the peak b and c."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rotate(image, angle, interpolation=cv2.INTER_CUBIC,\r\n borderMode=cv2.BORDER_REFLECT, borderValue=0):\r\n '''\r\n angle [deg]\r\n '''\r\n s0, s1 = image.shape\r\n image_center = (s0 - 1) / 2., (s1 - 1) / 2.\r\n rot_mat = cv2.getRotationMatrix2D(image_center, angle, 1.0)\r\n result = cv2.warpAffine(image, rot_mat, image.shape,\r\n flags=interpolation, borderMode=borderMode,\r\n borderValue=borderValue)\r\n return result", "response": "rotate an image by a given angle"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef adjustUncertToExposureTime(facExpTime, uncertMap, evtLenMap):\r\n '''\r\n Adjust image uncertainty (measured at exposure time t0)\r\n to new exposure time\r\n \r\n facExpTime --> new exp.time / reference exp.time =(t/t0)\r\n uncertMap --> 2d array mapping image uncertainty\r\n \r\n evtLen --> 2d array mapping event duration within image [sec]\r\n event duration is relative to exposure time\r\n e.g. duration = 2 means event is 2x longer than \r\n exposure time\r\n \r\n More information can be found at ...\r\n ----\r\n K.Bedrich: Quantitative Electroluminescence Imaging, PhD Thesis, 2017\r\n Subsection 5.1.4.3: Exposure Time Dependency\r\n ----\r\n '''\r\n\r\n #fit parameters, obtained from ####[simulateUncertDependencyOnExpTime]\r\n params = np.array( \r\n #a facExpTime f_0 f_inf \r\n [[ 2.63017121e+00, 3.05873627e-01, 1.00000000e+01, 2.78233309e-01],\r\n [ 2.26467931e+00, 2.86206621e-01, 8.01396977e+00, 2.04089232e-01],\r\n [ 1.27361168e+00, 5.18377189e-01, 3.04180084e+00, 2.61396338e-01],\r\n [ 7.34546040e-01, 7.34549823e-01, 1.86507345e+00, 2.77563156e-01],\r\n [ 3.82715618e-01, 9.32410141e-01, 1.34510254e+00, 2.91228149e-01],\r\n [ 1.71166071e-01, 1.14092885e+00, 1.11243702e+00, 3.07947386e-01],\r\n [ 6.13455410e-02, 1.43802520e+00, 1.02995065e+00, 3.93920802e-01],\r\n [ 1.65383071e-02, 1.75605076e+00, 1.00859395e+00, 5.02132321e-01],\r\n [ 4.55800114e-03, 1.99855711e+00, 9.98819118e-01, 5.99572776e-01]])\r\n \r\n #event duration relative to exposure time:(1/16...16)\r\n dur = np.array([ 0.0625, 0.125 , 0.25 , \r\n 0.5 , 1. , 2. ,\r\n 4. , 8. , 16. ])\r\n #get factors from interpolation:\r\n a = UnivariateSpline(dur, params[:, 0], k=3, s=0)\r\n b = UnivariateSpline(dur, params[:, 1], k=3, s=0)\r\n start = UnivariateSpline(dur, params[:, 2], k=3, s=0)\r\n end = UnivariateSpline(dur, params[:, 3], k=3, s=0)\r\n p0 = a(evtLenMap), b(evtLenMap), start(evtLenMap), end(evtLenMap)\r\n #uncertainty for new exposure time:\r\n out = uncertMap * _fitfn(facExpTime, *p0)\r\n # everywhere where there ARE NO EVENTS --> scale uncert. as if would\r\n # be normal distributed:\r\n i = evtLenMap == 0\r\n out[i] = uncertMap[i] * (1 / facExpTime)**0.5\r\n return out", "response": "This function adjusts the image uncertainty to the exposure time of the given event."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting images to a video file.", "response": "def videoWrite(path, imgs, levels=None, shape=None, frames=15,\r\n annotate_names=None,\r\n lut=None, updateFn=None):\r\n '''\r\n TODO\r\n '''\r\n frames = int(frames)\r\n if annotate_names is not None:\r\n assert len(annotate_names) == len(imgs)\r\n\r\n if levels is None:\r\n if imgs[0].dtype == np.uint8:\r\n levels = 0, 255\r\n elif imgs[0].dtype == np.uint16:\r\n levels = 0, 2**16 - 1\r\n else:\r\n levels = np.min(imgs), np.max(imgs)\r\n\r\n fourcc = cv2.VideoWriter_fourcc(*'XVID')\r\n h, w = imgs.shape[1:3]\r\n\r\n if shape and shape != (h, w):\r\n h, w = shape\r\n imgs = [cv2.resize(i, (w, h)) for i in imgs]\r\n\r\n assert path[-3:] in ('avi',\r\n 'png'), 'video export only supports *.avi or *.png'\r\n isVideo = path[-3:] == 'avi'\r\n if isVideo:\r\n cap = cv2.VideoCapture(0)\r\n # im.ndim==4)\r\n out = cv2.VideoWriter(path, fourcc, frames, (w, h), isColor=1)\r\n\r\n times = np.linspace(0, len(imgs) - 1, len(imgs) * frames)\r\n interpolator = LinearInterpolateImageStack(imgs)\r\n\r\n if lut is not None:\r\n lut = lut(imgs[0])\r\n\r\n for n, time in enumerate(times):\r\n if updateFn:\r\n # update progress:\r\n updateFn.emit(100 * n / len(times))\r\n image = interpolator(time)\r\n\r\n cimg = makeRGBA(image, lut=lut,\r\n levels=levels)[0]\r\n cimg = cv2.cvtColor(cimg, cv2.COLOR_RGBA2BGR)\r\n\r\n if annotate_names:\r\n text = annotate_names[n // frames]\r\n alpha = 0.5\r\n org = (0, cimg.shape[0])\r\n fontFace = cv2.FONT_HERSHEY_PLAIN\r\n fontScale = 2\r\n thickness = 3\r\n putTextAlpha(cimg, text, alpha, org, fontFace, fontScale,\r\n (0, 255, 0), thickness\r\n )\r\n\r\n if isVideo:\r\n out.write(cimg)\r\n else:\r\n cv2.imwrite('%s_%i_%.3f.png' % (path[:-4], n, time), cimg)\r\n\r\n if isVideo:\r\n cap.release()\r\n out.release()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef imread(img, color=None, dtype=None):\r\n '''\r\n dtype = 'noUint', uint8, float, 'float', ...\r\n '''\r\n COLOR2CV = {'gray': cv2.IMREAD_GRAYSCALE,\r\n 'all': cv2.IMREAD_COLOR,\r\n None: cv2.IMREAD_ANYCOLOR\r\n }\r\n c = COLOR2CV[color]\r\n if callable(img):\r\n img = img()\r\n elif isinstance(img, string_types):\r\n # from_file = True\r\n # try:\r\n # ftype = img[img.find('.'):]\r\n # img = READERS[ftype](img)[0]\r\n # except KeyError:\r\n # open with openCV\r\n # grey - 8 bit\r\n if dtype in (None, \"noUint\") or np.dtype(dtype) != np.uint8:\r\n c |= cv2.IMREAD_ANYDEPTH\r\n img2 = cv2.imread(img, c)\r\n if img2 is None:\r\n raise IOError(\"image '%s' is not existing\" % img)\r\n img = img2\r\n\r\n elif color == 'gray' and img.ndim == 3: # multi channel img like rgb\r\n # cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) #cannot handle float64\r\n img = toGray(img)\r\n # transform array to uint8 array due to openCV restriction\r\n if dtype is not None:\r\n if isinstance(img, np.ndarray):\r\n img = _changeArrayDType(img, dtype, cutHigh=False)\r\n\r\n return img", "response": "Returns the image with the specified color and dtype."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd an image to the image list and roi - upper left corner of the image", "response": "def addImg(self, img, roi=None):\r\n '''\r\n img - background, flat field, ste corrected image\r\n roi - [(x1,y1),...,(x4,y4)] - boundaries where points are\r\n '''\r\n self.img = imread(img, 'gray')\r\n s0, s1 = self.img.shape\r\n\r\n if roi is None:\r\n roi = ((0, 0), (s0, 0), (s0, s1), (0, s1))\r\n\r\n k = self.kernel_size\r\n hk = k // 2\r\n\r\n # mask image\r\n img2 = self.img.copy() # .astype(int)\r\n\r\n mask = np.zeros(self.img.shape)\r\n cv2.fillConvexPoly(mask, np.asarray(roi, dtype=np.int32), color=1)\r\n mask = mask.astype(bool)\r\n im = img2[mask]\r\n\r\n bg = im.mean() # assume image average with in roi == background\r\n mask = ~mask\r\n img2[mask] = -1\r\n\r\n # find points from local maxima:\r\n self.points = np.zeros(shape=(self.max_points, 2), dtype=int)\r\n thresh = 0.8 * bg + 0.2 * im.max()\r\n\r\n _findPoints(img2, thresh, self.min_dist, self.points)\r\n self.points = self.points[:np.argmin(self.points, axis=0)[0]]\r\n\r\n # correct point position, to that every point is over max value:\r\n for n, p in enumerate(self.points):\r\n sub = self.img[p[1] - hk:p[1] + hk + 1, p[0] - hk:p[0] + hk + 1]\r\n i, j = np.unravel_index(np.nanargmax(sub), sub.shape)\r\n self.points[n] += [j - hk, i - hk]\r\n\r\n # remove points that are too close to their neighbour or the border\r\n mask = maximum_filter(mask, hk)\r\n i = np.ones(self.points.shape[0], dtype=bool)\r\n for n, p in enumerate(self.points):\r\n if mask[p[1], p[0]]: # too close to border\r\n i[n] = False\r\n else:\r\n # too close to other points\r\n for pp in self.points[n + 1:]:\r\n if norm(p - pp) < hk + 1:\r\n i[n] = False\r\n isum = i.sum()\r\n ll = len(i) - isum\r\n print('found %s points' % isum)\r\n if ll:\r\n print(\r\n 'removed %s points (too close to border or other points)' %\r\n ll)\r\n self.points = self.points[i]\r\n\r\n# self.n_points += len(self.points)\r\n\r\n # for finding best peak position:\r\n# def fn(xy,cx,cy):#par\r\n# (x,y) = xy\r\n# return 1-(((x-cx)**2 + (y-cy)**2)*(1/8)).flatten()\r\n\r\n# x,y = np.mgrid[-2:3,-2:3]\r\n# x = x.flatten()\r\n# y = y.flatten()\r\n # for shifting peak:\r\n xx, yy = np.mgrid[0:k, 0:k]\r\n xx = xx.astype(float)\r\n yy = yy.astype(float)\r\n\r\n self.subs = []\r\n\r\n\r\n# import pylab as plt\r\n# plt.figure(20)\r\n# img = self.drawPoints()\r\n# plt.imshow(img, interpolation='none')\r\n# # plt.figure(21)\r\n# # plt.imshow(sub2, interpolation='none')\r\n# plt.show()\r\n\r\n #thresh = 0.8*bg + 0.1*im.max()\r\n for i, p in enumerate(self.points):\r\n sub = self.img[p[1] - hk:p[1] + hk + 1,\r\n p[0] - hk:p[0] + hk + 1].astype(float)\r\n sub2 = sub.copy()\r\n\r\n mean = sub2.mean()\r\n mx = sub2.max()\r\n sub2[sub2 < 0.5 * (mean + mx)] = 0 # only select peak\r\n try:\r\n # SHIFT SUB ARRAY to align peak maximum exactly in middle:\r\n # only eval a 5x5 array in middle of sub:\r\n # peak = sub[hk-3:hk+4,hk-3:hk+4]#.copy()\r\n\r\n # peak -= peak.min()\r\n # peak/=peak.max()\r\n # peak = peak.flatten()\r\n # fit paraboloid to get shift in x,y:\r\n # p, _ = curve_fit(fn, (x,y), peak, (0,0))\r\n c0, c1 = center_of_mass(sub2)\r\n\r\n# print (p,c0,c1,hk)\r\n\r\n #coords = np.array([xx+p[0],yy+p[1]])\r\n coords = np.array([xx + (c0 - hk), yy + (c1 - hk)])\r\n\r\n #print (c0,c1)\r\n\r\n #import pylab as plt\r\n #plt.imshow(sub2, interpolation='none')\r\n\r\n # shift array:\r\n sub = map_coordinates(sub, coords,\r\n mode='nearest').reshape(k, k)\r\n # plt.figure(2)\r\n #plt.imshow(sub, interpolation='none')\r\n # plt.show()\r\n\r\n #normalize:\r\n bg = 0.25* ( sub[0].mean() + sub[-1].mean() \r\n + sub[:,0].mean() + sub[:,-1].mean())\r\n \r\n sub-=bg\r\n sub /= sub.max()\r\n\r\n# import pylab as plt\r\n# plt.figure(20)\r\n# plt.imshow(sub, interpolation='none')\r\n# # plt.figure(21)\r\n# # plt.imshow(sub2, interpolation='none')\r\n# plt.show()\r\n\r\n self._psf += sub\r\n\r\n if self.calc_std:\r\n self.subs.append(sub)\r\n except ValueError:\r\n pass"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nstitch 2 images vertically together. Smooth the overlap area of both images with a linear fade from img1 to img2 @param img1: numpy.2dArray @param img2: numpy.2dArray of the same shape[1,2] as img1 @param overlap: number of pixels both images overlap @returns: stitched-image", "response": "def linearBlend(img1, img2, overlap, backgroundColor=None):\r\n '''\r\n Stitch 2 images vertically together.\r\n Smooth the overlap area of both images with a linear fade from img1 to img2\r\n\r\n @param img1: numpy.2dArray\r\n @param img2: numpy.2dArray of the same shape[1,2] as img1\r\n @param overlap: number of pixels both images overlap\r\n @returns: stitched-image\r\n '''\r\n (sizex, sizey) = img1.shape[:2]\r\n\r\n overlapping = True\r\n if overlap < 0:\r\n overlapping = False\r\n overlap = -overlap\r\n\r\n # linear transparency change:\r\n alpha = np.tile(np.expand_dims(np.linspace(1, 0, overlap), 1), sizey)\r\n if len(img2.shape) == 3: # multi channel img like rgb\r\n # make alpha 3d with n channels\r\n alpha = np.dstack(([alpha for _ in range(img2.shape[2])]))\r\n if overlapping:\r\n img1_cut = img1[sizex - overlap:sizex, :]\r\n img2_cut = img2[0:overlap, :]\r\n else:\r\n # take average of last 5 rows:\r\n img1_cut = np.tile(img1[-min(sizex, 5):, :].mean(\r\n axis=0), (overlap, 1)).reshape(alpha.shape)\r\n img2_cut = np.tile(img2[:min(img2.shape[0], 5), :].mean(\r\n axis=0), (overlap, 1)).reshape(alpha.shape)\r\n\r\n # fill intermediate area as mixture of both images\r\n #################bg transparent############\r\n inter = (img1_cut * alpha + img2_cut * (1 - alpha)).astype(img1.dtype)\r\n # set background areas to value of respective other img:\r\n if backgroundColor is not None:\r\n mask = np.logical_and(img1_cut == backgroundColor,\r\n img2_cut != backgroundColor)\r\n inter[mask] = img2_cut[mask]\r\n mask = np.logical_and(img2_cut == backgroundColor,\r\n img1_cut != backgroundColor)\r\n\r\n inter[mask] = img1_cut[mask]\r\n if not overlapping:\r\n overlap = 0\r\n return np.vstack((img1[0:sizex - overlap, :],\r\n inter,\r\n img2[overlap:, :]))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef interpolate2dStructuredPointSpreadIDW(grid, mask, kernel=15, power=2,\r\n maxIter=1e5, copy=True):\r\n '''\r\n same as interpolate2dStructuredIDW but using the point spread method\r\n this is faster if there are bigger connected masked areas and the border\r\n length is smaller\r\n\r\n replace all values in [grid] indicated by [mask]\r\n with the inverse distance weighted interpolation of all values within\r\n px+-kernel\r\n\r\n [power] -> distance weighting factor: 1/distance**[power]\r\n [copy] -> False: a bit faster, but modifies 'grid' and 'mask'\r\n '''\r\n assert grid.shape == mask.shape, 'grid and mask shape are different'\r\n\r\n border = np.zeros(shape=mask.shape, dtype=np.bool)\r\n if copy:\r\n # copy mask as well because if will be modified later:\r\n mask = mask.copy()\r\n grid = grid.copy()\r\n return _calc(grid, mask, border, kernel, power, maxIter)", "response": "same as interpolate2dStructuredPointSpreadIDW but using the point spread method"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef SNR(img1, img2=None, bg=None,\r\n noise_level_function=None,\r\n constant_noise_level=False,\r\n imgs_to_be_averaged=False):\r\n '''\r\n Returns a signal-to-noise-map\r\n uses algorithm as described in BEDRICH 2016 JPV (not jet published)\r\n\r\n :param constant_noise_level: True, to assume noise to be constant\r\n :param imgs_to_be_averaged: True, if SNR is for average(img1, img2)\r\n '''\r\n # dark current subtraction:\r\n img1 = np.asfarray(img1)\r\n if bg is not None:\r\n img1 = img1 - bg\r\n\r\n # SIGNAL:\r\n if img2 is not None:\r\n img2_exists = True\r\n img2 = np.asfarray(img2) - bg\r\n # signal as average on both images\r\n signal = 0.5 * (img1 + img2)\r\n else:\r\n img2_exists = False\r\n signal = img1\r\n # denoise:\r\n signal = median_filter(signal, 3)\r\n\r\n # NOISE\r\n if constant_noise_level:\r\n # CONSTANT NOISE\r\n if img2_exists:\r\n d = img1 - img2\r\n # 0.5**0.5 because of sum of variances\r\n noise = 0.5**0.5 * np.mean(np.abs((d))) * F_RMS2AAD\r\n else:\r\n d = (img1 - signal) * F_NOISE_WITH_MEDIAN\r\n noise = np.mean(np.abs(d)) * F_RMS2AAD\r\n else:\r\n # NOISE LEVEL FUNCTION\r\n if noise_level_function is None:\r\n noise_level_function, _ = oneImageNLF(img1, img2, signal)\r\n noise = noise_level_function(signal)\r\n noise[noise < 1] = 1 # otherwise SNR could be higher than image value\r\n\r\n if imgs_to_be_averaged:\r\n # SNR will be higher if both given images are supposed to be averaged:\r\n # factor of noise reduction if SNR if for average(img1, img2):\r\n noise *= 0.5**0.5\r\n\r\n # BACKGROUND estimation and removal if background not given:\r\n if bg is None:\r\n bg = getBackgroundLevel(img1)\r\n signal -= bg\r\n snr = signal / noise\r\n\r\n # limit to 1, saying at these points signal=noise:\r\n snr[snr < 1] = 1\r\n return snr", "response": "Returns a signal - to - noise - map for the two images img1 and img2."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sortCorners(corners):\r\n '''\r\n sort the corners of a given quadrilateral of the type\r\n corners : [ [xi,yi],... ]\r\n\r\n to an anti-clockwise order starting with the bottom left corner\r\n\r\n or (if plotted as image where y increases to the bottom):\r\n clockwise, starting top left\r\n '''\r\n corners = np.asarray(corners)\r\n # bring edges in order:\r\n corners2 = corners[ConvexHull(corners).vertices]\r\n \r\n if len(corners2) == 3:\r\n # sometimes ConvexHull one point is missing because it is\r\n # within the hull triangle\r\n # find the right position of set corner as the minimum perimeter\r\n # built with that point as different indices\r\n for c in corners:\r\n if c not in corners2:\r\n break\r\n perimeter = []\r\n for n in range(0, 4):\r\n corners3 = np.insert(corners2, n, c, axis=0)\r\n perimeter.append(\r\n np.linalg.norm(\r\n np.diff(\r\n corners3,\r\n axis=0),\r\n axis=1).sum())\r\n n = np.argmin(perimeter)\r\n corners2 = np.insert(corners2, n, c, axis=0)\r\n\r\n # find the edge with the right angle to the quad middle:\r\n mn = corners2.mean(axis=0)\r\n d = (corners2 - mn)\r\n ascent = np.arctan2(d[:, 1], d[:, 0])\r\n bl = np.abs(BL_ANGLE + ascent).argmin()\r\n # build a index list starting with bl:\r\n i = list(range(bl, 4))\r\n i.extend(list(range(0, bl)))\r\n return corners2[i]", "response": "sort the corners of a given quadrilateral of the type\r\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns an array with contains the closest distance to the next positive value given in arr within a given kernel size", "response": "def closestDirectDistance(arr, ksize=30, dtype=np.uint16):\r\n '''\r\n return an array with contains the closest distance to the next positive\r\n value given in arr within a given kernel size\r\n '''\r\n\r\n out = np.zeros_like(arr, dtype=dtype)\r\n _calc(out, arr, ksize)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef closestConnectedDistance(target, walls=None,\r\n max_len_border_line=500,\r\n max_n_path=100,\r\n concentrate_every_n_pixel=1):\r\n '''\r\n returns an array with contains the closest distance from every pixel\r\n the next position where target == 1\r\n\r\n [walls] binary 2darray - e.g. walls in a labyrinth that have to be surrounded in order to get to the target\r\n\r\n [target] binary 2darray - positions given by 1\r\n\r\n [concentrate_every_n_pixel] often the distance of neighbour pixels is similar\r\n to speed up calculation set this value to e.g. 3 to calculate only\r\n the distance for every 3. pixel and interpolate in between\r\n\r\n recommended are values up to 3-5\r\n\r\n [max_len_border_line]\r\n this function calculates distances travelled using region growth\r\n e.g.\r\n\r\n 0123\r\n 1123\r\n 2223\r\n 3333\r\n\r\n the last steps (e.g. for all steps 3 border_line=7) are stored in an array of limited\r\n length defined in 'max_len_border_line'\r\n\r\n [max_n_path]\r\n how many paths are possible between every pixel and the target\r\n only needed if fast==False\r\n '''\r\n c = concentrate_every_n_pixel\r\n assert c >= 1\r\n if walls is None:\r\n walls = np.zeros_like(target, dtype=bool)\r\n s = target.shape\r\n dt = np.uint16\r\n if max(target.shape) < 200:\r\n dt = np.uint8\r\n out = np.zeros((s[0] // c, s[1] // c), dtype=dt)\r\n # temporary arrays:\r\n growth = np.zeros_like(target, dtype=dt)\r\n res = np.empty(shape=3, dtype=dt)\r\n\r\n steps = np.empty(shape=(max_len_border_line, 2), dtype=dt)\r\n new_steps = np.empty(shape=(max_len_border_line, 2), dtype=dt)\r\n\r\n # run calculation:\r\n _calc(growth, out, walls, target, steps, new_steps,\r\n res, concentrate_every_n_pixel)\r\n\r\n if c > 1:\r\n # if concentrate_every_n_pixel > 1\r\n # the resized output array\r\n # will have wrong values close to the wall\r\n # therefore substitute all wall value (-1)\r\n # with an average of their closest neighbours\r\n interpolate2dStructuredIDW(out, out == 0)\r\n out = cv2.resize(out, s[::-1])\r\n out[walls] = 0\r\n return out", "response": "Returns a binary 2darray with the closest distance from every pixel to the target."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfills [res] with [distance to next position where target == 1, x coord., y coord. of that position in target] using region growth i,j -> pixel position growth -> a work array, needed to measure the distance steps, new_steps -> current and last positions of the region growth steps using this instead of looking for the right step position in [growth] should speed up the process", "response": "def _grow(growth, walls, target, i, j, steps, new_steps, res):\r\n '''\r\n fills [res] with [distance to next position where target == 1,\r\n x coord.,\r\n y coord. of that position in target]\r\n using region growth\r\n\r\n i,j -> pixel position\r\n growth -> a work array, needed to measure the distance\r\n steps, new_steps -> current and last positions of the region growth steps\r\n using this instead of looking for the right step position in [growth]\r\n should speed up the process\r\n '''\r\n\r\n # clean array:\r\n growth[:] = 0\r\n\r\n if target[i, j]:\r\n # pixel is in target\r\n res[0] = 1\r\n res[1] = i\r\n res[2] = j\r\n return\r\n\r\n step = 1\r\n s0, s1 = growth.shape\r\n step_len = 1\r\n new_step_ind = 0\r\n\r\n steps[new_step_ind, 0] = i\r\n steps[new_step_ind, 1] = j\r\n growth[i, j] = 1\r\n\r\n while True:\r\n for n in range(step_len):\r\n i, j = steps[n]\r\n for ii, jj in DIRECT_NEIGHBOURS:\r\n pi = i + ii\r\n pj = j + jj\r\n\r\n # if in image:\r\n if 0 <= pi < s0 and 0 <= pj < s1:\r\n # is growth array is empty and there are no walls:\r\n # fill growth with current step\r\n if growth[pi, pj] == 0 and not walls[pi, pj]:\r\n growth[pi, pj] = step\r\n if target[pi, pj]:\r\n # found destination\r\n res[0] = 1\r\n res[1] = pi\r\n res[2] = pj\r\n return\r\n\r\n new_steps[new_step_ind, 0] = pi\r\n new_steps[new_step_ind, 1] = pj\r\n new_step_ind += 1\r\n\r\n if new_step_ind == 0:\r\n # couldn't populate any more because growth is full\r\n # and all possible steps are gone\r\n res[0] = 0\r\n return\r\n\r\n step += 1\r\n steps, new_steps = new_steps, steps\r\n step_len = new_step_ind\r\n new_step_ind = 0"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef polylinesFromBinImage(img, minimum_cluster_size=6,\r\n remove_small_obj_size=3,\r\n reconnect_size=3,\r\n max_n_contours=None, max_len_contour=None,\r\n copy=True):\r\n '''\r\n return a list of arrays of un-branching contours\r\n\r\n img -> (boolean) array \r\n\r\n optional:\r\n ---------\r\n minimum_cluster_size -> minimum number of pixels connected together to build a contour\r\n\r\n ##search_kernel_size -> TODO\r\n ##min_search_kernel_moment -> TODO\r\n\r\n numeric:\r\n -------------\r\n max_n_contours -> maximum number of possible contours in img\r\n max_len_contour -> maximum contour length\r\n\r\n '''\r\n assert minimum_cluster_size > 1\r\n assert reconnect_size % 2, 'ksize needs to be odd'\r\n\r\n # assert search_kernel_size == 0 or search_kernel_size > 2 and search_kernel_size%2, 'kernel size needs to be odd'\r\n # assume array size parameters, is not given:\r\n if max_n_contours is None:\r\n max_n_contours = max(img.shape)\r\n if max_len_contour is None:\r\n max_len_contour = sum(img.shape[:2])\r\n # array containing coord. of all contours:\r\n contours = np.zeros(shape=(max_n_contours, max_len_contour, 2),\r\n dtype=np.uint16) # if not search_kernel_size else np.float32)\r\n\r\n if img.dtype != np.bool:\r\n img = img.astype(bool)\r\n elif copy:\r\n img = img.copy()\r\n\r\n if remove_small_obj_size:\r\n remove_small_objects(img, remove_small_obj_size,\r\n connectivity=2, in_place=True)\r\n if reconnect_size:\r\n # remove gaps\r\n maximum_filter(img, reconnect_size, output=img)\r\n # reduce contour width to 1\r\n img = skeletonize(img)\r\n\r\n n_contours = _populateContoursArray(img, contours, minimum_cluster_size)\r\n contours = contours[:n_contours]\r\n\r\n l = []\r\n for c in contours:\r\n ind = np.zeros(shape=len(c), dtype=bool)\r\n _getValidInd(c, ind)\r\n # remove all empty spaces:\r\n l.append(c[ind])\r\n return l", "response": "Returns a list of arrays of contour arrays from a binary image."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the cumulative density function of a given array or a given position", "response": "def cdf(arr, pos=None):\r\n '''\r\n Return the cumulative density function of a given array or\r\n its intensity at a given position (0-1)\r\n '''\r\n\r\n r = (arr.min(), arr.max())\r\n hist, bin_edges = np.histogram(arr, bins=2 * int(r[1] - r[0]), range=r)\r\n hist = np.asfarray(hist) / hist.sum()\r\n cdf = np.cumsum(hist)\r\n if pos is None:\r\n return cdf\r\n i = np.argmax(cdf > pos)\r\n return bin_edges[i]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef subCell2DGenerator(arr, shape, d01=None, p01=None):\r\n '''Generator to access evenly sized sub-cells in a 2d array\r\n\r\n Args:\r\n shape (tuple): number of sub-cells in y,x e.g. (10,15)\r\n d01 (tuple, optional): cell size in y and x\r\n p01 (tuple, optional): position of top left edge\r\n\r\n Returns:\r\n int: 1st index\r\n int: 2nd index\r\n array: sub array\r\n\r\n Example:\r\n\r\n >>> a = np.array([[[0,1],[1,2]],[[2,3],[3,4]]])\r\n >>> gen = subCell2DGenerator(a,(2,2))\r\n >>> for i,j, sub in gen: print( i,j, sub )\r\n 0 0 [[[0 1]]]\r\n 0 1 [[[1 2]]]\r\n 1 0 [[[2 3]]]\r\n 1 1 [[[3 4]]]\r\n '''\r\n for i, j, s0, s1 in subCell2DSlices(arr, shape, d01, p01):\r\n yield i, j, arr[s0, s1]", "response": "Generator to access evenly sized sub - cells in a 2d array"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef subCell2DSlices(arr, shape, d01=None, p01=None):\r\n '''Generator to access evenly sized sub-cells in a 2d array\r\n\r\n Args:\r\n shape (tuple): number of sub-cells in y,x e.g. (10,15)\r\n d01 (tuple, optional): cell size in y and x\r\n p01 (tuple, optional): position of top left edge\r\n\r\n Returns:\r\n int: 1st index\r\n int: 2nd index\r\n slice: first dimension\r\n slice: 1st dimension\r\n '''\r\n if p01 is not None:\r\n yinit, xinit = p01\r\n else:\r\n xinit, yinit = 0, 0\r\n\r\n x, y = xinit, yinit\r\n g0, g1 = shape\r\n s0, s1 = arr.shape[:2]\r\n\r\n if d01 is not None:\r\n d0, d1 = d01\r\n else:\r\n d0, d1 = s0 / g0, s1 / g1\r\n\r\n y1 = d0 + yinit\r\n for i in range(g0):\r\n for j in range(g1):\r\n x1 = x + d1\r\n yield (i, j, slice(max(0, _rint(y)),\r\n max(0, _rint(y1))),\r\n slice(max(0, _rint(x)),\r\n max(0, _rint(x1))))\r\n x = x1\r\n y = y1\r\n y1 = y + d0\r\n x = xinit", "response": "Generator to access evenly sized sub - cells in a 2d array"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef subCell2DCoords(*args, **kwargs):\r\n '''Same as subCell2DSlices but returning coordinates\r\n\r\n Example:\r\n g = subCell2DCoords(arr, shape)\r\n for x, y in g:\r\n plt.plot(x, y)\r\n '''\r\n for _, _, s0, s1 in subCell2DSlices(*args, **kwargs):\r\n yield ((s1.start, s1.start, s1.stop),\r\n (s0.start, s0.stop, s0.stop))", "response": "Same as subCell2DSlices but returning coordinates"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns array where every cell equals result of fn on all sub - arrays", "response": "def subCell2DFnArray(arr, fn, shape, dtype=None, **kwargs):\r\n '''\r\n Return array where every cell is the output of a given cell function\r\n\r\n Args:\r\n fn (function): ...to be executed on all sub-arrays\r\n\r\n Returns:\r\n array: value of every cell equals result of fn(sub-array)\r\n\r\n Example: \r\n mx = subCell2DFnArray(myArray, np.max, (10,6) )\r\n - -> here mx is a 2d array containing all cell maxima\r\n '''\r\n\r\n sh = list(arr.shape)\r\n sh[:2] = shape\r\n out = np.empty(sh, dtype=dtype)\r\n for i, j, c in subCell2DGenerator(arr, shape, **kwargs):\r\n out[i, j] = fn(c)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef defocusThroughDepth(u, uf, f, fn, k=2.355):\r\n '''\r\n return the defocus (mm std) through DOF\r\n \r\n u -> scene point (depth value)\r\n uf -> in-focus position (the distance at which the scene point should be placed in order to be focused)\r\n f -> focal length\r\n k -> camera dependent constant (transferring blur circle to PSF), 2.335 would be FHWD of 2dgaussian\r\n fn --> f-number (relative aperture) \r\n \r\n equation (3) taken from http://linkinghub.elsevier.com/retrieve/pii/S0031320312004736\r\n Pertuz et.al. \"Analysis of focus measure operators for shape-from-focus\"\r\n \r\n all parameter should be in same physical unit [mm]\r\n \r\n !! assumes spatial invariant blur\r\n '''\r\n # A = f/fn \r\n return (k/fn) * (f**2*abs(u-uf)) / (u*(uf-f))", "response": "Returns the defocus through a scene point uf f and fn"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextending array for convolution", "response": "def extendArrayForConvolution(arr, kernelXY, \r\n modex='reflect', \r\n modey='reflect'):\r\n '''\r\n extends a given array right right border handling\r\n for convolution\r\n -->in opposite to skimage and skipy this function \r\n allows to chose different mode = ('reflect', 'wrap')\r\n in x and y direction\r\n \r\n only supports 'warp' and 'reflect' at the moment \r\n '''\r\n (kx, ky) = kernelXY\r\n kx//=2\r\n ky//=2\r\n \r\n #indexing 0:-0 leads to empty arrays and not the whole thing\r\n #make it easy with assuming ksize=1 and removing extra size later:\r\n nokx = kx == 0 \r\n noky = ky == 0 \r\n if nokx:\r\n kx = 1\r\n if noky:\r\n ky = 1 \r\n \r\n s0,s1 = arr.shape\r\n \r\n assert ky < s0\r\n assert kx < s1\r\n \r\n arr2 = np.zeros((s0+2*ky, s1+2*kx), dtype=arr.dtype)\r\n if kx == 0:\r\n kx = None\r\n arr2[ky:-ky,kx:-kx]=arr\r\n \r\n #original array:\r\n t = arr[:ky] #TOP\r\n rb = arr[-1:-ky-1:-1] #reverse bottom\r\n rt = arr[ky-1::-1] #reverse top\r\n rr = arr[:,-1:-kx-1:-1] #reverse right\r\n l = arr[:,:kx] #left\r\n# rtl = arr[ky-1::-1,kx-1::-1]\r\n\r\n #filter array:\r\n tm2 = arr2[:ky , kx:-kx] #TOP-MIDDLE\r\n bm2 = arr2[-ky:, kx:-kx] #BOTTOM-MIDDLE\r\n tl2 = arr2[:ky , :kx] #TOP-LEFT\r\n bl2 = arr2[-ky:, :kx] #BOTTOM-LEFT\r\n tr2 = arr2[:ky:, -kx:]#TOP-RIGHT\r\n br2 = arr2[-ky:, -kx:]#TOP-RIGHT\r\n \r\n #fill edges:\r\n if modey == 'warp':\r\n tm2[:] = t\r\n bm2[:] = rb\r\n \r\n tl2[:] = arr2[2*ky:ky:-1,:kx]\r\n bl2[:] = arr2[-ky-1:-2*ky-1:-1,:kx]\r\n #TODO: do other options!!! \r\n elif modey == 'reflect':\r\n tm2[:] = rt\r\n bm2[:] = rb\r\n if modex =='reflect':\r\n tl2[:] = arr[ky-1::-1,kx-1::-1]\r\n bl2[:] = arr[-1:-ky-1:-1,kx-1::-1]\r\n \r\n tr2[:] = arr[:ky,-kx:][::-1,::-1]\r\n br2[:] = arr[-ky:,-kx:][::-1,::-1]\r\n \r\n else:#warp\r\n tl2[:] = arr[ky-1::-1 , -kx:]\r\n bl2[:] = arr[-1:-ky-1:-1 , -kx:]\r\n tr2[:] = arr[ky-1::-1 , :kx]\r\n br2[:] = arr[-1:-ky-1:-1 , :kx]\r\n \r\n else:\r\n raise Exception('modey not supported')\r\n \r\n if modex == 'wrap':\r\n arr2[ky:-ky,kx-1::-1] = rr\r\n arr2[ky:-ky,-kx:] = l \r\n elif modex == 'reflect':\r\n arr2[ky:-ky,:kx] = l[:,::-1]\r\n arr2[ky:-ky,-kx:] = rr \r\n else:\r\n raise Exception('modex not supported')\r\n\r\n if nokx:\r\n arr2 = arr2[:,1:-1]\r\n if noky:\r\n arr2 = arr2[1:-1] \r\n \r\n return arr2"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef calibrate(self, board_size=(8, 6), method='Chessboard', images=[],\r\n max_images=100, sensorSize_mm=None,\r\n detect_sensible=True):\r\n '''\r\n sensorSize_mm - (width, height) [mm] Physical size of the sensor\r\n '''\r\n self._coeffs = {}\r\n self.opts = {'foundPattern': [], # whether pattern could be found for image\r\n 'size': board_size,\r\n 'imgs': [], # list of either npArrsays or img paths\r\n # list or 2d coords. of found pattern features (e.g.\r\n # chessboard corners)\r\n 'imgPoints': []\r\n }\r\n\r\n self._detect_sensible = detect_sensible\r\n\r\n self.method = {'Chessboard': self._findChessboard,\r\n 'Symmetric circles': self._findSymmetricCircles,\r\n 'Asymmetric circles': self._findAsymmetricCircles,\r\n 'Manual': None\r\n # TODO: 'Image grid':FindGridInImage\r\n }[method]\r\n\r\n self.max_images = max_images\r\n self.findCount = 0\r\n self.apertureSize = sensorSize_mm\r\n\r\n self.objp = self._mkObjPoints(board_size)\r\n\r\n if method == 'Asymmetric circles':\r\n # this pattern have its points (every 2. row) displaced, so:\r\n i = self.objp[:, 1] % 2 == 1\r\n self.objp[:, 0] *= 2\r\n self.objp[i, 0] += 1\r\n\r\n # Arrays to store object points and image points from all the images.\r\n self.objpoints = [] # 3d point in real world space\r\n # self.imgpoints = [] # 2d points in image plane.\r\n self.mapx, self.mapy = None, None\r\n\r\n # from matplotlib import pyplot as plt\r\n for n, i in enumerate(images):\r\n print('working on image %s' % n)\r\n if self.addImg(i):\r\n print('OK')", "response": "Create a new instance of the appropriate class based on the parameters of the specified method."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef addPoints(self, points, board_size=None):\r\n '''\r\n add corner points directly instead of extracting them from\r\n image\r\n points = ( (0,1), (...),... ) [x,y]\r\n '''\r\n self.opts['foundPattern'].append(True)\r\n self.findCount += 1\r\n if board_size is not None:\r\n self.objpoints.append(self._mkObjPoints(board_size))\r\n else:\r\n self.objpoints.append(self.objp)\r\n s0 = points.shape[0]\r\n\r\n self.opts['imgPoints'].append(np.asarray(points).reshape(\r\n s0, 1, 2).astype(np.float32))", "response": "add corner points directly to extract them from image\r\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setImgShape(self, shape):\r\n '''\r\n image shape must be known for calculating camera matrix\r\n if method==Manual and addPoints is used instead of addImg\r\n this method must be called before .coeffs are obtained\r\n '''\r\n self.img = type('Dummy', (object,), {})\r\n# if imgProcessor.ARRAYS_ORDER_IS_XY:\r\n# self.img.shape = shape[::-1]\r\n# else:\r\n self.img.shape = shape", "response": "image shape must be known for calculating camera matrix\r\n if method==Manual and addPoints is used instead of addImg\r\n this method must be called before .coeffs are obtained"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a new image using a continous stream", "response": "def addImgStream(self, img):\r\n '''\r\n add images using a continous stream\r\n - stop when max number of images is reached\r\n '''\r\n if self.findCount > self.max_images:\r\n raise EnoughImages('have enough images')\r\n return self.addImg(img)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef addImg(self, img):\r\n '''\r\n add one chessboard image for detection lens distortion\r\n '''\r\n # self.opts['imgs'].append(img)\r\n\r\n self.img = imread(img, 'gray', 'uint8')\r\n\r\n didFindCorners, corners = self.method()\r\n self.opts['foundPattern'].append(didFindCorners)\r\n\r\n if didFindCorners:\r\n self.findCount += 1\r\n self.objpoints.append(self.objp)\r\n self.opts['imgPoints'].append(corners)\r\n return didFindCorners", "response": "add one chessboard image for detection lens distortion"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the distortion coeffs in a formated string", "response": "def getCoeffStr(self):\r\n '''\r\n get the distortion coeffs in a formated string\r\n '''\r\n txt = ''\r\n for key, val in self.coeffs.items():\r\n txt += '%s = %s\\n' % (key, val)\r\n return txt"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef drawChessboard(self, img=None):\r\n '''\r\n draw a grid fitting to the last added image\r\n on this one or an extra image\r\n img == None\r\n ==False -> draw chessbord on empty image\r\n ==img\r\n '''\r\n assert self.findCount > 0, 'cannot draw chessboard if nothing found'\r\n if img is None:\r\n img = self.img\r\n elif isinstance(img, bool) and not img:\r\n img = np.zeros(shape=(self.img.shape), dtype=self.img.dtype)\r\n else:\r\n img = imread(img, dtype='uint8')\r\n gray = False\r\n if img.ndim == 2:\r\n gray = True\r\n # need a color 8 bit image\r\n img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)\r\n # Draw and display the corners\r\n cv2.drawChessboardCorners(img, self.opts['size'],\r\n self.opts['imgPoints'][-1],\r\n self.opts['foundPattern'][-1])\r\n if gray:\r\n img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\r\n return img", "response": "draw a chessboard on the last added image\r\n - > image with the most recent found pattern"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites the distortion coeffs to file", "response": "def writeToFile(self, filename, saveOpts=False):\r\n '''\r\n write the distortion coeffs to file\r\n saveOpts --> Whether so save calibration options (and not just results)\r\n '''\r\n try:\r\n if not filename.endswith('.%s' % self.ftype):\r\n filename += '.%s' % self.ftype\r\n s = {'coeffs': self.coeffs}\r\n if saveOpts:\r\n s['opts'] = self.opts\r\n# else:\r\n# s['opts':{}]\r\n np.savez(filename, **s)\r\n return filename\r\n except AttributeError:\r\n raise Exception(\r\n 'need to calibrate camera before calibration can be saved to file')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading the distortion coeffs from file", "response": "def readFromFile(self, filename):\r\n '''\r\n read the distortion coeffs from file\r\n '''\r\n s = dict(np.load(filename))\r\n try:\r\n self.coeffs = s['coeffs'][()]\r\n except KeyError:\r\n #LEGENCY - remove\r\n self.coeffs = s\r\n try:\r\n self.opts = s['opts'][()]\r\n except KeyError:\r\n pass\r\n return self.coeffs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef undistortPoints(self, points, keepSize=False):\r\n '''\r\n points --> list of (x,y) coordinates\r\n '''\r\n s = self.img.shape\r\n cam = self.coeffs['cameraMatrix']\r\n d = self.coeffs['distortionCoeffs']\r\n\r\n pts = np.asarray(points, dtype=np.float32)\r\n if pts.ndim == 2:\r\n pts = np.expand_dims(pts, axis=0)\r\n\r\n (newCameraMatrix, roi) = cv2.getOptimalNewCameraMatrix(cam,\r\n d, s[::-1], 1,\r\n s[::-1])\r\n if not keepSize:\r\n xx, yy = roi[:2]\r\n pts[0, 0] -= xx\r\n pts[0, 1] -= yy\r\n\r\n return cv2.undistortPoints(pts,\r\n cam, d, P=newCameraMatrix)", "response": "Returns a list of undistorted points."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving lens distortion from given image keepSize = True keepSize = False borderValue = 0", "response": "def correct(self, image, keepSize=False, borderValue=0):\r\n '''\r\n remove lens distortion from given image\r\n '''\r\n image = imread(image)\r\n (h, w) = image.shape[:2]\r\n mapx, mapy = self.getUndistortRectifyMap(w, h)\r\n self.img = cv2.remap(image, mapx, mapy, cv2.INTER_LINEAR,\r\n borderMode=cv2.BORDER_CONSTANT,\r\n borderValue=borderValue\r\n )\r\n if not keepSize:\r\n xx, yy, ww, hh = self.roi\r\n self.img = self.img[yy: yy + hh, xx: xx + ww]\r\n return self.img"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef getCameraParams(self):\r\n '''\r\n value positions based on \r\n http://docs.opencv.org/modules/imgproc/doc/geometric_transformations.html#cv.InitUndistortRectifyMap\r\n '''\r\n c = self.coeffs['cameraMatrix']\r\n fx = c[0][0]\r\n fy = c[1][1]\r\n cx = c[0][2]\r\n cy = c[1][2]\r\n k1, k2, p1, p2, k3 = tuple(self.coeffs['distortionCoeffs'].tolist()[0])\r\n return fx, fy, cx, cy, k1, k2, k3, p1, p2", "response": "Returns a tuple of the two elements of the camera matrix and the two elements of the camera matrix."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of standard uncertainties for the image and the image sharpness.", "response": "def standardUncertainties(self, sharpness=0.5):\r\n '''\r\n sharpness -> image sharpness // std of Gaussian PSF [px]\r\n\r\n returns a list of standard uncertainties for the x and y component:\r\n (1x,2x), (1y, 2y), (intensity:None)\r\n 1. px-size-changes(due to deflection)\r\n 2. reprojection error\r\n '''\r\n height, width = self.coeffs['shape']\r\n fx, fy = self.getDeflection(width, height)\r\n # is RMSE of imgPoint-projectedPoints\r\n r = self.coeffs['reprojectionError']\r\n t = (sharpness**2 + r**2)**0.5\r\n return fx * t, fy * t"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntakes a binary image and returns the edges of the object inside .", "response": "def edgesFromBoolImg(arr, dtype=None):\r\n '''\r\n takes a binary image (usually a mask)\r\n and returns the edges of the object inside\r\n '''\r\n out = np.zeros_like(arr, dtype=dtype)\r\n _calc(arr, out)\r\n _calc(arr.T, out.T)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef draw_matches(img1, kp1, img2, kp2, matches, color=None, thickness=2, r=15):\r\n # We're drawing them side by side. Get dimensions accordingly.\r\n # Handle both color and grayscale images.\r\n if len(img1.shape) == 3:\r\n new_shape = (max(img1.shape[0], img2.shape[0]), img1.shape[\r\n 1] + img2.shape[1], img1.shape[2])\r\n elif len(img1.shape) == 2:\r\n new_shape = (\r\n max(img1.shape[0], img2.shape[0]), img1.shape[1] + img2.shape[1])\r\n new_img = np.zeros(new_shape, type(img1.flat[0]))\r\n # Place images onto the new image.\r\n new_img[0:img1.shape[0], 0:img1.shape[1]] = img1\r\n new_img[0:img2.shape[0], img1.shape[1]\r\n :img1.shape[1] + img2.shape[1]] = img2\r\n\r\n # Draw lines between matches. Make sure to offset kp coords in second\r\n # image appropriately.\r\n if color:\r\n c = color\r\n for m in matches:\r\n # Generate random color for RGB/BGR and grayscale images as needed.\r\n if not color:\r\n c = np.random.randint(0, 256, 3) if len(\r\n img1.shape) == 3 else np.random.randint(0, 256)\r\n # So the keypoint locs are stored as a tuple of floats. cv2.line(), like most other things,\r\n # wants locs as a tuple of ints.\r\n end1 = tuple(np.round(kp1[m.trainIdx].pt).astype(int))\r\n end2 = tuple(np.round(kp2[m.queryIdx].pt).astype(\r\n int) + np.array([img1.shape[1], 0]))\r\n cv2.line(new_img, end1, end2, c, thickness)\r\n cv2.circle(new_img, end1, r, c, thickness)\r\n cv2.circle(new_img, end2, r, c, thickness)\r\n return new_img", "response": "Draws lines between two matching keypoints."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef findHomography(self, img, drawMatches=False):\r\n '''\r\n Find homography of the image through pattern\r\n comparison with the base image\r\n '''\r\n print(\"\\t Finding points...\")\r\n # Find points in the next frame\r\n img = self._prepareImage(img)\r\n features, descs = self.detector.detectAndCompute(img, None)\r\n\r\n ######################\r\n # TODO: CURRENTLY BROKEN IN OPENCV3.1 - WAITNG FOR NEW RELEASE 3.2\r\n# matches = self.matcher.knnMatch(descs,#.astype(np.float32),\r\n# self.base_descs,\r\n# k=3)\r\n# print(\"\\t Match Count: \", len(matches))\r\n# matches_subset = self._filterMatches(matches)\r\n\r\n # its working alternative (for now):\r\n bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)\r\n matches_subset = bf.match(descs, self.base_descs)\r\n\r\n ######################\r\n# matches = bf.knnMatch(descs,self.base_descs, k=2)\r\n# # Apply ratio test\r\n# matches_subset = []\r\n# medDist = np.median([m.distance for m in matches])\r\n# matches_subset = [m for m in matches if m.distance < medDist]\r\n# for m in matches:\r\n# print(m.distance)\r\n# for m,n in matches:\r\n# if m.distance < 0.75*n.distance:\r\n# matches_subset.append([m])\r\n\r\n if not len(matches_subset):\r\n raise Exception('no matches found')\r\n print(\"\\t Filtered Match Count: \", len(matches_subset))\r\n\r\n distance = sum([m.distance for m in matches_subset])\r\n print(\"\\t Distance from Key Image: \", distance)\r\n\r\n averagePointDistance = distance / (len(matches_subset))\r\n print(\"\\t Average Distance: \", averagePointDistance)\r\n\r\n kp1 = []\r\n kp2 = []\r\n\r\n for match in matches_subset:\r\n kp1.append(self.base_features[match.trainIdx])\r\n kp2.append(features[match.queryIdx])\r\n\r\n # /self._fH #scale with _fH, if image was resized\r\n\r\n p1 = np.array([k.pt for k in kp1])\r\n p2 = np.array([k.pt for k in kp2]) # /self._fH\r\n\r\n H, status = cv2.findHomography(p1, p2,\r\n cv2.RANSAC, # method\r\n 5.0 # max reprojection error (1...10)\r\n )\r\n if status is None:\r\n raise Exception('no homography found')\r\n else:\r\n inliers = np.sum(status)\r\n print('%d / %d inliers/matched' % (inliers, len(status)))\r\n inlierRatio = inliers / len(status)\r\n if self.minInlierRatio > inlierRatio or inliers < self.minInliers:\r\n raise Exception('bad fit!')\r\n\r\n # scale with _fH, if image was resized\r\n # see\r\n # http://answers.opencv.org/question/26173/the-relationship-between-homography-matrix-and-scaling-images/\r\n s = np.eye(3, 3)\r\n s[0, 0] = 1 / self._fH\r\n s[1, 1] = 1 / self._fH\r\n H = s.dot(H).dot(np.linalg.inv(s))\r\n\r\n if drawMatches:\r\n # s0,s1 = img.shape\r\n # out = np.empty(shape=(s0,s1,3), dtype=np.uint8)\r\n img = draw_matches(self.base8bit, self.base_features, img, features,\r\n matches_subset[:20], # None,#out,\r\n # flags=2\r\n thickness=5\r\n )\r\n\r\n return (H, inliers, inlierRatio, averagePointDistance,\r\n img, features,\r\n descs, len(matches_subset))", "response": "Find homography of the image through pattern\r\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef patCircles(s0):\r\n '''make circle array'''\r\n arr = np.zeros((s0,s0), dtype=np.uint8)\r\n col = 255\r\n for rad in np.linspace(s0,s0/7.,10):\r\n cv2.circle(arr, (0,0), int(round(rad)), color=col, \r\n thickness=-1, lineType=cv2.LINE_AA )\r\n if col:\r\n col = 0\r\n else:\r\n col = 255\r\n \r\n\r\n return arr.astype(float)", "response": "make a circle array"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake line pattern for the given number of times", "response": "def patCrossLines(s0):\r\n '''make line pattern'''\r\n arr = np.zeros((s0,s0), dtype=np.uint8)\r\n col = 255\r\n t = int(s0/100.)\r\n for pos in np.logspace(0.01,1,10):\r\n pos = int(round((pos-0.5)*s0/10.))\r\n cv2.line(arr, (0,pos), (s0,pos), color=col, \r\n thickness=t, lineType=cv2.LINE_AA )\r\n cv2.line(arr, (pos,0), (pos,s0), color=col, \r\n thickness=t, lineType=cv2.LINE_AA )\r\n\r\n return arr.astype(float)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef removeSinglePixels(img):\r\n '''\r\n img - boolean array\r\n remove all pixels that have no neighbour\r\n '''\r\n\r\n gx = img.shape[0]\r\n gy = img.shape[1]\r\n\r\n for i in range(gx):\r\n for j in range(gy):\r\n\r\n if img[i, j]:\r\n\r\n found_neighbour = False\r\n for ii in range(max(0, i - 1), min(gx, i + 2)):\r\n for jj in range(max(0, j - 1), min(gy, j + 2)):\r\n\r\n if ii == i and jj == j:\r\n continue\r\n\r\n if img[ii, jj]:\r\n found_neighbour = True\r\n break\r\n if found_neighbour:\r\n break\r\n\r\n if not found_neighbour:\r\n img[i, j] = 0", "response": "remove all pixels that have no neighbour\r\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninterpolating a 2D array into a 2D array with a cross average.", "response": "def interpolate2dStructuredCrossAvg(grid, mask, kernel=15, power=2):\r\n '''\r\n #######\r\n usefull if large empty areas need to be filled\r\n\r\n '''\r\n\r\n vals = np.empty(shape=4, dtype=grid.dtype)\r\n dist = np.empty(shape=4, dtype=np.uint16)\r\n weights = np.empty(shape=4, dtype=np.float32)\r\n valid = np.empty(shape=4, dtype=bool)\r\n\r\n return _calc(grid, mask, power, kernel, vals, dist, weights, valid)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef growPositions(ksize):\r\n '''\r\n return all positions around central point (0,0) \r\n for a given kernel size \r\n positions grow from smallest to biggest distances\r\n \r\n returns [positions] and [distances] from central cell\r\n \r\n '''\r\n i = ksize*2+1\r\n kk = np.ones( (i, i), dtype=bool)\r\n x,y = np.where(kk)\r\n pos = np.empty(shape=(i,i,2), dtype=int)\r\n pos[:,:,0]=x.reshape(i,i)-ksize\r\n pos[:,:,1]=y.reshape(i,i)-ksize\r\n\r\n dist = np.fromfunction(lambda x,y: ((x-ksize)**2\r\n +(y-ksize)**2)**0.5, (i,i))\r\n\r\n pos = np.dstack(\r\n np.unravel_index(\r\n np.argsort(dist.ravel()), (i, i)))[0,1:]\r\n\r\n pos0 = pos[:,0]\r\n pos1 = pos[:,1]\r\n\r\n return pos-ksize, dist[pos0, pos1]", "response": "grows all positions around central point"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a QImage to a numpy array.", "response": "def qImageToArray(qimage, dtype = 'array'):\r\n \"\"\"Convert QImage to numpy.ndarray. The dtype defaults to uint8\r\n for QImage.Format_Indexed8 or `bgra_dtype` (i.e. a record array)\r\n for 32bit color images. You can pass a different dtype to use, or\r\n 'array' to get a 3D uint8 array for color images.\"\"\"\r\n\r\n result_shape = (qimage.height(), qimage.width())\r\n temp_shape = (qimage.height(),\r\n qimage.bytesPerLine() * 8 // qimage.depth())\r\n if qimage.format() in (QtGui.QImage.Format_ARGB32_Premultiplied,\r\n QtGui.QImage.Format_ARGB32,\r\n QtGui.QImage.Format_RGB32):\r\n if dtype == 'rec':\r\n dtype = np.dtype({'b': (np.uint8, 0),\r\n 'g': (np.uint8, 1),\r\n 'r': (np.uint8, 2),\r\n 'a': (np.uint8, 3)})\r\n elif dtype == 'array':\r\n dtype = np.uint8\r\n result_shape += (4, )\r\n temp_shape += (4, )\r\n elif qimage.format() == QtGui.QImage.Format_Indexed8:\r\n dtype = np.uint8\r\n else:\r\n raise ValueError(\"qimage2numpy only supports 32bit and 8bit images\")\r\n # FIXME: raise error if alignment does not match\r\n buf = qimage.bits().asstring(qimage.byteCount())\r\n result = np.frombuffer(buf, dtype).reshape(temp_shape)\r\n if result_shape != temp_shape:\r\n result = result[:,:result_shape[1]]\r\n if qimage.format() == QtGui.QImage.Format_RGB32 and dtype == np.uint8:\r\n #case byteorder == 'little'\r\n result = result[...,:3]\r\n #byteorder == 'big' -> get ARGB\r\n result = result[...,::-1]\r\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a new array with variable ksize in y", "response": "def varYSizeGaussianFilter(arr, stdyrange, stdx=0,\r\n modex='wrap', modey='reflect'):\r\n '''\r\n applies gaussian_filter on input array\r\n but allowing variable ksize in y\r\n \r\n stdyrange(int) -> maximum ksize - ksizes will increase from 0 to given value\r\n stdyrange(tuple,list) -> minimum and maximum size as (mn,mx)\r\n stdyrange(np.array) -> all different ksizes in y\r\n '''\r\n assert arr.ndim == 2, 'only works on 2d arrays at the moment'\r\n \r\n s0 = arr.shape[0]\r\n \r\n #create stdys:\r\n if isinstance(stdyrange, np.ndarray):\r\n assert len(stdyrange)==s0, '[stdyrange] needs to have same length as [arr]'\r\n stdys = stdyrange\r\n else:\r\n if type(stdyrange) not in (list, tuple):\r\n stdyrange = (0,stdyrange)\r\n mn,mx = stdyrange\r\n stdys = np.linspace(mn,mx,s0)\r\n \r\n #prepare array for convolution:\r\n kx = int(stdx*2.5)\r\n kx += 1-kx%2\r\n ky = int(mx*2.5)\r\n ky += 1-ky%2\r\n arr2 = extendArrayForConvolution(arr, (kx, ky), modex, modey)\r\n \r\n #create convolution kernels:\r\n inp = np.zeros((ky,kx))\r\n inp[ky//2, kx//2] = 1\r\n kernels = np.empty((s0,ky,kx))\r\n for i in range(s0):\r\n stdy = stdys[i]\r\n kernels[i] = gaussian_filter(inp, (stdy,stdx))\r\n\r\n out = np.empty_like(arr)\r\n _2dConvolutionYdependentKernel(arr2, out, kernels)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef numbaGaussian2d(psf, sy, sx):\r\n '''\r\n 2d Gaussian to be used in numba code\r\n '''\r\n ps0, ps1 = psf.shape\r\n c0,c1 = ps0//2, ps1//2\r\n ssx = 2*sx**2\r\n ssy = 2*sy**2\r\n for i in range(ps0):\r\n for j in range(ps1):\r\n psf[i,j]=exp( -( (i-c0)**2/ssy\r\n +(j-c1)**2/ssx) )\r\n psf/=psf.sum()", "response": "Returns a 2d Gaussian that is used in numba code"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef estimateBackgroundLevel(img, image_is_artefact_free=False, \r\n min_rel_size=0.05, max_abs_size=11):\r\n '''\r\n estimate background level through finding the most homogeneous area\r\n and take its average\r\n \r\n min_size - relative size of the examined area\r\n '''\r\n\r\n s0,s1 = img.shape[:2]\r\n s = min(max_abs_size, int(max(s0,s1)*min_rel_size))\r\n arr = np.zeros(shape=(s0-2*s, s1-2*s), dtype=img.dtype)\r\n \r\n #fill arr:\r\n _spatialStd(img, arr, s)\r\n #most homogeneous area:\r\n i,j = np.unravel_index(arr.argmin(), arr.shape)\r\n sub = img[int(i+0.5*s):int(i+s*1.5), \r\n int(j+s*0.5):int(j+s*1.5)]\r\n\r\n return np.median(sub)", "response": "estimate background level through finding the most homogeneous area and take its average"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the angular dependent EL emissivity of a PV module", "response": "def EL_Si_module():\r\n '''\r\n returns angular dependent EL emissivity of a PV module\r\n \r\n calculated of nanmedian(persp-corrected EL module/reference module)\r\n \r\n published in K. Bedrich: Quantitative Electroluminescence Measurement on PV devices\r\n PhD Thesis, 2017\r\n '''\r\n arr = np.array([\r\n [2.5, 1.00281 ],\r\n [7.5, 1.00238 ],\r\n [12.5, 1.00174],\r\n [17.5, 1.00204 ],\r\n [22.5, 1.00054 ],\r\n [27.5, 0.998255],\r\n [32.5, 0.995351],\r\n [37.5, 0.991246],\r\n [42.5, 0.985304],\r\n [47.5, 0.975338],\r\n [52.5, 0.960455],\r\n [57.5, 0.937544],\r\n [62.5, 0.900607],\r\n [67.5, 0.844636],\r\n [72.5, 0.735028],\r\n [77.5, 0.57492 ],\r\n [82.5, 0.263214],\r\n [87.5, 0.123062]\r\n ])\r\n\r\n angles = arr[:,0]\r\n vals = arr[:,1]\r\n\r\n vals[vals>1]=1\r\n return angles, vals"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreflect temperature for 250DEG Glass published in IEC 62446-3 TS: Photovoltaic (PV) systems - Requirements for testing, documentation and maintenance - Part 3: Outdoor infrared thermography of photovoltaic modules and plants p Page 12", "response": "def TG_glass():\r\n '''\r\n reflected temperature for 250DEG Glass\r\n published in IEC 62446-3 TS: Photovoltaic (PV) systems \r\n - Requirements for testing, documentation and maintenance \r\n - Part 3: Outdoor infrared thermography of photovoltaic modules \r\n and plants p Page 12\r\n '''\r\n vals = np.array([(80,0.88),\r\n (75,0.88),\r\n (70,0.88),\r\n (65,0.88),\r\n (60,0.88),\r\n (55,0.88),\r\n (50,0.87),\r\n (45,0.86),\r\n (40,0.85),\r\n (35,0.83),\r\n (30,0.80),\r\n (25,0.76),\r\n (20,0.7),\r\n (15,0.60),\r\n (10,0.44)])\r\n #invert angle reference:\r\n vals[:,0]=90-vals[:,0]\r\n #make emissivity relative:\r\n vals[:,1]/=vals[0,1]\r\n return vals[:,0], vals[:,1]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the sensitivity of the homogeneously illuminated images.", "response": "def sensitivity(imgs, bg=None):\r\n '''\r\n Extract pixel sensitivity from a set of homogeneously illuminated images\r\n\r\n This method is detailed in Section 5 of:\r\n ---\r\n K.Bedrich, M.Bokalic et al.:\r\n ELECTROLUMINESCENCE IMAGING OF PV DEVICES:\r\n ADVANCED FLAT FIELD CALIBRATION,2017\r\n ---\r\n\r\n '''\r\n bg = getBackground(bg)\r\n for n, i in enumerate(imgs):\r\n i = imread(i, dtype=float)\r\n i -= bg\r\n smooth = fastMean(median_filter(i, 3))\r\n i /= smooth\r\n if n == 0:\r\n out = i\r\n else:\r\n out += i\r\n out /= (n + 1)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef navierStokes2d(u, v, p, dt, nt, rho, nu, \r\n boundaryConditionUV, \r\n boundardConditionP, nit=100):\r\n '''\r\n solves the Navier-Stokes equation for incompressible flow\r\n one a regular 2d grid\r\n \r\n u,v,p --> initial velocity(u,v) and pressure(p) maps\r\n \r\n dt --> time step\r\n nt --> number of time steps to caluclate\r\n \r\n rho, nu --> material constants\r\n \r\n nit --> number of iteration to solve the pressure field\r\n '''\r\n #next u, v, p maps:\r\n un = np.empty_like(u)\r\n vn = np.empty_like(v)\r\n pn = np.empty_like(p)\r\n #poisson equation ==> laplace term = b[source term]\r\n b = np.zeros_like(p)\r\n\r\n ny,nx = p.shape\r\n #cell size:\r\n dx = 2 / (nx - 1)\r\n dy = 2 / (ny - 1)\r\n \r\n #next time step:\r\n for _ in range(nt):\r\n un[:] = u\r\n vn[:] = v\r\n #pressure\r\n _buildB(b, rho, dt, u, v, dx, dy)\r\n for _ in range(nit):\r\n _pressurePoisson(p, pn, dx, dy, b)\r\n boundardConditionP(p)\r\n #UV\r\n _calcUV(u, v, un, p,vn, dt, dx, dy, rho, nu)\r\n boundaryConditionUV(u,v)\r\n\r\n return u, v, p", "response": "Returns a new 2d grid with the same initial velocity and pressure."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd noise to the image and return the new image", "response": "def addNoise(img, snr=25, rShot=0.5):\r\n '''\r\n adds Gaussian (thermal) and shot noise to [img]\r\n\r\n [img] is assumed to be noise free\r\n\r\n [rShot] - shot noise ratio relative to all noise \r\n '''\r\n s0, s1 = img.shape[:2]\r\n\r\n m = img.mean()\r\n if np.isnan(m):\r\n m = np.nanmean(img)\r\n assert m != 0, 'image mean cannot be zero'\r\n\r\n img = img / m\r\n noise = np.random.normal(size=s0 * s1).reshape(s0, s1)\r\n if rShot > 0:\r\n noise *= (rShot * img**0.5 + 1)\r\n noise /= np.nanstd(noise)\r\n noise[np.isnan(noise)] = 0\r\n return m * (img + noise / snr)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning an array of the given shape coarse - maximum of the given array.", "response": "def coarseMaximum(arr, shape):\r\n '''\r\n return an array of [shape]\r\n where every cell equals the localised maximum of the given array [arr]\r\n at the same (scalled) position\r\n '''\r\n ss0, ss1 = shape\r\n s0, s1 = arr.shape\r\n\r\n pos0 = linspace2(0, s0, ss0, dtype=int)\r\n pos1 = linspace2(0, s1, ss1, dtype=int)\r\n\r\n k0 = pos0[0]\r\n k1 = pos1[0]\r\n\r\n out = np.empty(shape, dtype=arr.dtype)\r\n _calc(arr, out, pos0, pos1, k0, k1, ss0, ss1)\r\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the angle of view of the aperture given the specified parameters", "response": "def angleOfView(XY, shape=None, a=None, f=None, D=None, center=None):\r\n '''\r\n Another vignetting equation from:\r\n M. Koentges, M. Siebert, and D. Hinken, \"Quantitative analysis of PV-modules by electroluminescence images for quality control\"\r\n 2009\r\n f --> Focal length\r\n D --> Diameter of the aperture\r\n BOTH, D AND f NEED TO HAVE SAME UNIT [PX, mm ...]\r\n a --> Angular aperture\r\n \r\n center -> optical center [y,x]\r\n '''\r\n if a is None:\r\n assert f is not None and D is not None\r\n #https://en.wikipedia.org/wiki/Angular_aperture\r\n a = 2*np.arctan2(D/2,f)\r\n \r\n x,y = XY\r\n\r\n try:\r\n c0,c1 = center\r\n except:\r\n s0,s1 = shape\r\n c0,c1 = s0/2, s1/2\r\n\r\n rx = (x-c0)**2\r\n ry = (y-c1)**2 \r\n\r\n return 1 / (1+np.tan(a)*((rx+ry)/c0))**0.5"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the angle of view between two camera modules.", "response": "def angleOfView2(x,y, b, x0=None,y0=None):\r\n '''\r\n Corrected AngleOfView equation by Koentges (via mail from 14/02/2017)\r\n b --> distance between the camera and the module in m\r\n x0 --> viewable with in the module plane of the camera in m\r\n y0 --> viewable height in the module plane of the camera in m\r\n x,y --> pixel position [m] from top left\r\n '''\r\n if x0 is None:\r\n x0 = x[-1,-1]\r\n if y0 is None:\r\n y0 = y[-1,-1] \r\n return np.cos( np.arctan( np.sqrt(\r\n ( (x-x0/2)**2+(y-y0/2)**2 ) ) /b ) )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a 2d grid of nCells lines from a list of edges.", "response": "def gridLinesFromVertices(edges, nCells, subgrid=None, dtype=float):\r\n \"\"\"\r\n ###TODO REDO TXT\r\n\r\n OPTIONAL:\r\n subgrid = ([x],[y]) --> relative positions\r\n e.g. subgrid = ( (0.3,0.7), () )\r\n --> two subgrid lines in x - nothing in y\r\n\r\n Returns: \r\n horiz,vert -> arrays of (x,y) poly-lines\r\n\r\n\r\n if subgrid != None, Returns:\r\n horiz,vert, subhoriz, subvert\r\n\r\n\r\n #######\r\n creates a regular 2d grid from given edge points (4*(x0,y0))\r\n and number of cells in x and y\r\n\r\n Returns:\r\n tuple(4lists): horizontal and vertical lines as (x0,y0,x1,y1)\r\n \"\"\"\r\n\r\n nx, ny = nCells\r\n\r\n y, x = np.mgrid[0.:ny + 1, 0.:nx + 1]\r\n\r\n src = np.float32([[0, 0], [nx, 0], [nx, ny], [0, ny]])\r\n dst = sortCorners(edges).astype(np.float32)\r\n\r\n homography = cv2.getPerspectiveTransform(src, dst)\r\n\r\n pts = np.float32((x.flatten(), y.flatten())).T\r\n pts = pts.reshape(1, *pts.shape)\r\n\r\n pts2 = cv2.perspectiveTransform(pts, homography)[0]\r\n\r\n horiz = pts2.reshape(ny + 1, nx + 1, 2)\r\n vert = np.swapaxes(horiz, 0, 1)\r\n\r\n subh, subv = [], []\r\n if subgrid is not None:\r\n sh, sv = subgrid\r\n\r\n if len(sh):\r\n subh = np.empty(shape=(ny * len(sh), nx + 1, 2), dtype=np.float32)\r\n last_si = 0\r\n for n, si in enumerate(sh):\r\n spts = pts[:, :-(nx + 1)]\r\n spts[..., 1] += si - last_si\r\n last_si = si\r\n spts2 = cv2.perspectiveTransform(spts, homography)[0]\r\n subh[n::len(sh)] = spts2.reshape(ny, nx + 1, 2)\r\n if len(sv):\r\n subv = np.empty(shape=(ny + 1, nx * len(sv), 2), dtype=np.float32)\r\n last_si = 0\r\n sspts = pts.reshape(1, ny + 1, nx + 1, 2)\r\n sspts = sspts[:, :, :-1]\r\n\r\n sspts = sspts.reshape(1, (ny + 1) * nx, 2)\r\n for n, si in enumerate(sv):\r\n sspts[..., 0] += si - last_si\r\n last_si = si\r\n spts2 = cv2.perspectiveTransform(sspts, homography)[0]\r\n subv[:, n::len(sv)] = spts2.reshape(ny + 1, nx, 2)\r\n subv = np.swapaxes(subv, 0, 1)\r\n return [horiz, vert, subh, subv]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the most recent object resolution as [ line pairs / mm ] where MTF = 50% of the given time frame.", "response": "def MTF50(self, MTFx,MTFy):\r\n '''\r\n return object resolution as [line pairs/mm]\r\n where MTF=50%\r\n see http://www.imatest.com/docs/sharpness/\r\n '''\r\n if self.mtf_x is None:\r\n self.MTF()\r\n f = UnivariateSpline(self.mtf_x, self.mtf_y-0.5)\r\n return f.roots()[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef MTF(self, px_per_mm):\r\n '''\r\n px_per_mm = cam_resolution / image_size\r\n '''\r\n res = 100 #numeric resolution\r\n r = 4 #range +-r*std\r\n \r\n #size of 1 px:\r\n px_size = 1 / px_per_mm\r\n \r\n #standard deviation of the point-spread-function (PSF) as normal distributed:\r\n std = self.std*px_size #transform standard deviation from [px] to [mm]\r\n\r\n x = np.linspace(-r*std,r*std, res)\r\n #line spread function:\r\n lsf = self.gaussian1d(x, 1, 0, std)\r\n #MTF defined as Fourier transform of the line spread function:\r\n #abs() because result is complex\r\n y = abs(np.fft.fft(lsf)) \r\n #normalize fft so that max = 1\r\n y /= np.max(y)\r\n #step length between xn and xn+1\r\n dstep = r*std/res\r\n # Fourier frequencies - here: line pairs(cycles) per mm\r\n freq = np.fft.fftfreq(lsf.size, dstep)\r\n #limit mtf between [0-px_per_mm]:\r\n i = np.argmax(freq>px_per_mm)\r\n self.mtf_x = freq[:i]\r\n self.mtf_y = y[:i]\r\n return self.mtf_x, self.mtf_y", "response": "Returns the MTF and the MTF_y values for the most recent image."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the intensity based uncertainty due to the unsharpness of the image as standard deviation", "response": "def uncertaintyMap(self, psf, method='convolve', fitParams=None):\r\n '''\r\n return the intensity based uncertainty due to the unsharpness of the image\r\n as standard deviation\r\n \r\n method = ['convolve' , 'unsupervised_wiener']\r\n latter one also returns the reconstructed image (deconvolution)\r\n '''\r\n\r\n #ignore background:\r\n #img[img<0]=0\r\n ###noise should not influence sharpness uncertainty:\r\n ##img = median_filter(img, 3)\r\n\r\n # decrease noise in order not to overestimate result:\r\n img = scaleSignal(self.img, fitParams=fitParams)\r\n\r\n if method == 'convolve':\r\n #print 'convolve'\r\n blurred = convolve2d(img, psf, 'same')\r\n m = abs(img-blurred) / abs(img + blurred)\r\n m = np.nan_to_num(m)\r\n m*=self.std**2\r\n m[m>1]=1\r\n self.blur_distortion = m\r\n np.save('blurred', blurred)\r\n return m\r\n else:\r\n restored = unsupervised_wiener(img, psf)[0]\r\n m = abs(img-restored) / abs(img + restored)\r\n m = np.nan_to_num(m)\r\n m*=self.std**2\r\n m[m>1]=1\r\n self.blur_distortion = m\r\n return m, restored"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef stdDev(self):\r\n '''\r\n get the standard deviation \r\n from the PSF is evaluated as 2d Gaussian\r\n '''\r\n if self._corrPsf is None:\r\n self.psf()\r\n p = self._corrPsf.copy()\r\n mn = p.min()\r\n p[p<0.05*p.max()] = mn\r\n p-=mn\r\n p/=p.sum()\r\n \r\n x,y = self._psfGridCoords()\r\n x = x.flatten()\r\n y = y.flatten()\r\n\r\n guess = (1,1,0)\r\n\r\n param, _ = curve_fit(self._fn, (x,y), p.flatten(), guess)\r\n\r\n self._fitParam = param \r\n stdx,stdy = param[:2]\r\n self._std = (stdx+stdy) / 2\r\n \r\n return self._std", "response": "get the standard deviation of the current time - domain entry from the PSF is evaluated as 2d Gaussian\r\n self. _corrPsf is None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninterpolates all values in a 2D structured IDW grid with inverse distance weighted interpolation of all values in a 2D structured IDW grid", "response": "def interpolate2dStructuredIDW(grid, mask, kernel=15, power=2, fx=1, fy=1):\r\n '''\r\n replace all values in [grid] indicated by [mask]\r\n with the inverse distance weighted interpolation of all values within \r\n px+-kernel\r\n [power] -> distance weighting factor: 1/distance**[power]\r\n\r\n '''\r\n weights = np.empty(shape=((2*kernel+1,2*kernel+1)))\r\n for xi in range(-kernel,kernel+1):\r\n for yi in range(-kernel,kernel+1):\r\n dist = ((fx*xi)**2+(fy*yi)**2)\r\n if dist:\r\n weights[xi+kernel,yi+kernel] = 1 / dist**(0.5*power)\r\n\r\n return _calc(grid, mask, kernel, weights)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef temporalSignalStability(imgs, times, down_scale_factor=1):\r\n '''\r\n (Electroluminescence) signal is not stable over time\r\n especially next to cracks.\r\n This function takes a set of images\r\n and returns parameters, needed to transform uncertainty \r\n to other exposure times using [adjustUncertToExposureTime]\r\n \r\n \r\n return [signal uncertainty] obtained from linear fit to [imgs]\r\n [average event length] \r\n [ascent],[offset] of linear fit\r\n \r\n --------\r\n [imgs] --> corrected EL images captured in sequence\r\n \r\n [times] --> absolute measurement times of all [imgs]\r\n e.g. every image was taken every 60 sec, then \r\n times=60,120,180... \r\n [down_scale_factor] --> down scale [imgs] to speed up process\r\n -------\r\n\r\n More information can be found at ...\r\n ----\r\n K.Bedrich: Quantitative Electroluminescence Imaging, PhD Thesis, 2017\r\n Subsection 5.1.4.3: Exposure Time Dependency\r\n ----\r\n '''\r\n imgs = np.asarray(imgs)\r\n s0, s1, s2 = imgs.shape\r\n\r\n #down scale imgs to speed up process:\r\n if down_scale_factor > 1:\r\n s1 //= down_scale_factor\r\n s2 //= down_scale_factor\r\n imgs2 = np.empty(shape=(s0, s1, s2))\r\n for n, c in enumerate(imgs):\r\n imgs2[n] = cv2.resize(c, (s2, s1), interpolation=cv2.INTER_AREA)\r\n imgs = imgs2\r\n \r\n # linear fit for every point in image set:\r\n ascent, offset, error = linRegressUsingMasked2dArrays(\r\n times, imgs, calcError=True)\r\n \r\n # functionally obtained [imgs]:\r\n fn_imgs = np.array([offset + t * ascent for t in times])\r\n #difference between [imgs] for fit result:\r\n diff = imgs - fn_imgs\r\n diff = median_filter(diff, 5)\r\n\r\n error_t = np.tile(error, (s0, 1, 1))\r\n # find events: \r\n evt = (np.abs(diff) > 0.5 * error_t) \r\n # calc average event length:\r\n avlen = _calcAvgLen(evt, np.empty(shape=evt.shape[1:]))\r\n \r\n #cannot calc event length smaller exposure time, so:\r\n i = avlen == 0\r\n avlen = maskedFilter(avlen, mask=i, fn='mean', ksize=7, fill_mask=False)\r\n # remove single px:\r\n i = maximum_filter(i, 3)\r\n avlen[i] = 0\r\n avlen = maximum_filter(avlen, 3)\r\n\r\n i = avlen == 0\r\n avlen = median_filter(avlen, 3)\r\n avlen[i] = 0\r\n\r\n return error, avlen, ascent, offset", "response": "This function takes a set of images and returns a set of time - dependent Electrouminescence signal parameters needed to transform the uncertainty of each image to other exposure times using adjustUncertToExposureTime."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef vignettingFromSpotAverage(\r\n images, bgImages=None, averageSpot=True, thresh=None):\r\n '''\r\n [images] --> list of images containing\r\n small bright spots generated by the same \r\n device images at different positions within image plane\r\n depending on the calibrated waveband the device can be \r\n a LCD display or PV 1-cell mini module\r\n\r\n This method is referred as 'Method B' in\r\n ---\r\n K.Bedrich, M.Bokalic et al.:\r\n ELECTROLUMINESCENCE IMAGING OF PV DEVICES:\r\n ADVANCED FLAT FIELD CALIBRATION,2017\r\n ---\r\n\r\n Args:\r\n averageSpot(bool): True: take only the average intensity of each spot\r\n thresh(float): marks the minimum spot value \r\n (estimated with Otsus method otherwise)\r\n\r\n Returns:\r\n * array to be post processed\r\n * image mask containing valid positions\r\n '''\r\n\r\n fitimg, mask = None, None\r\n mx = 0\r\n for c, img in enumerate(images):\r\n print('%s/%s' % (c + 1, len(images)))\r\n\r\n if c == 0:\r\n avgBg = getBackground2(bgImages, img)\r\n img = imread(img, dtype=float)\r\n img -= avgBg\r\n # init:\r\n if fitimg is None:\r\n fitimg = np.zeros_like(img)\r\n mask = np.zeros_like(img, dtype=bool)\r\n # find spot:\r\n if thresh is None:\r\n t = threshold_otsu(img)\r\n else:\r\n t = thresh\r\n\r\n # take brightest spot\r\n spots, n = label(minimum_filter(img > t, 3),\r\n background=0, return_num=True)\r\n spot_sizes = [(spots == i).sum() for i in range(1, n + 1)]\r\n\r\n try:\r\n spot = (spots == np.argmax(spot_sizes) + 1)\r\n except ValueError:\r\n print(\"couldn't find spot in image\")\r\n continue\r\n\r\n if averageSpot:\r\n spot = np.rint(center_of_mass(spot)).astype(int)\r\n mx2 = img[spot].max()\r\n else:\r\n mx2 = img[spot].mean()\r\n fitimg[spot] = img[spot]\r\n mask[spot] = 1\r\n\r\n if mx2 > mx:\r\n mx = mx2\r\n\r\n # scale [0...1]:\r\n fitimg /= mx\r\n\r\n return fitimg, mask", "response": "Returns a new image mask containing the small bright spots generated by the same \r\nAttributeNames image mask containing valid positions and background images that are at different positions within the same \r\nAttributeNames image mask containing valid positions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsimulating a synthetic image using the given parameters and returns a new list of synthetic images.", "response": "def simulateSytematicError(N_SAMPLES=5, N_IMAGES=10,\r\n SHOW_DETECTED_PATTERN=True, # GRAYSCALE=False,\r\n HEIGHT=500, PLOT_RESULTS=True, PLOT_ERROR_ARRAY=True,\r\n CAMERA_PARAM=None, PERSPECTIVE=True, ROTATION=True,\r\n RELATIVE_PATTERN_SIZE=0.5, POSITION=True,\r\n NOISE=25, BLUR=(3, 3), PATTERNS=None):\r\n '''\r\n Simulates a lens calibration using synthetic images\r\n * images are rendered under the given HEIGHT resolution\r\n * noise and smoothing is applied\r\n * perspective and position errors are applied\r\n * images are deformed using the given CAMERA_PARAM\r\n\r\n * the detected camera parameters are used to calculate the error to the given ones\r\n\r\n\r\n simulation\r\n -----------\r\n N_IMAGES -> number of images to take for a camera calibration\r\n N_SAMPLES -> number of camera calibrations of each pattern type\r\n\r\n output\r\n --------\r\n SHOW_DETECTED_PATTERN: print each image and detected pattern to screen\r\n PLOT_RESULTS: plot boxplots of the mean error and std of the camera parameters\r\n PLOT_ERROR_ARRAY: plot position error for the lens correction\r\n\r\n pattern\r\n --------\r\n this simulation tests the openCV standard patterns: chess board, asymmetric and symmetric circles\r\n\r\n GRAYSCALE: whether to load the pattern as gray scale\r\n RELATIVE_PATTERN_SIZE: the relative size of the pattern within the image (0.4->40%)\r\n PERSPECTIVE: [True] -> enable perspective distortion\r\n ROTATION: [True] -> enable rotation of the pattern\r\n BLUR: False or (sizex,sizey), like (3,3)\r\n\r\n CAMERA_PARAM: camera calibration parameters as [fx,fy,cx,cy,k1,k2,k3,p1,p2]\r\n\r\n\r\n '''\r\n print(\r\n 'calculate systematic error of the implemented calibration algorithms')\r\n\r\n # LOCATION OF PATTERN IMAGES\r\n folder = MEDIA_PATH\r\n\r\n if PATTERNS is None:\r\n PATTERNS = ('Chessboard', 'Asymmetric circles', 'Symmetric circles')\r\n\r\n patterns = OrderedDict(( # n of inner corners\r\n ('Chessboard', ((6, 9), 'chessboard_pattern_a3.svg')),\r\n ('Asymmetric circles', ((4, 11), 'acircles_pattern_a3.svg')),\r\n ('Symmetric circles', ((8, 11), 'circles_pattern_a3.svg')),\r\n ))\r\n # REMOVE PATTERNS THAT ARE NOT TO BE TESTED:\r\n [patterns.pop(key) for key in patterns if key not in PATTERNS]\r\n\r\n if SHOW_DETECTED_PATTERN:\r\n cv2.namedWindow('Pattern', cv2.WINDOW_NORMAL)\r\n # number of positive detected patterns:\r\n success = []\r\n # list[N_SAMPLES] of random camera parameters\r\n fx, fy, cx, cy, k1, k2, k3, p1, p2 = [], [], [], [], [], [], [], [], []\r\n # list[Method, N_SAMPLES] of given-detected parameters:\r\n errl, fxl, fyl, cxl, cyl, k1l, k2l, k3l, p1l, p2l = [\r\n ], [], [], [], [], [], [], [], [], []\r\n # list[Method, N_SAMPLES] of magnitude(difference of displacement vector\r\n # array):\r\n dxl = []\r\n dyl = []\r\n # maintain aspect ratio of din a4, a3...:\r\n aspect_ratio_DIN = 2.0**0.5\r\n width = int(round(HEIGHT / aspect_ratio_DIN))\r\n\r\n if CAMERA_PARAM is None:\r\n CAMERA_PARAM = [\r\n HEIGHT, HEIGHT, HEIGHT / 2, width / 2, 0.0, 0.01, 0.1, 0.01, 0.001]\r\n\r\n # ???CREATE N DIFFERENT RANDOM LENS ERRORS:\r\n for n in range(N_SAMPLES):\r\n # TODO: RANDOMIZE CAMERA ERROR??\r\n fx.append(CAMERA_PARAM[0]) # * np.random.uniform(1, 2) )\r\n fy.append(CAMERA_PARAM[1]) # * np.random.uniform(1, 2) )\r\n cx.append(CAMERA_PARAM[2]) # * np.random.uniform(0.9, 1.1) )\r\n cy.append(CAMERA_PARAM[3]) # * np.random.uniform(0.9, 1.1) )\r\n k1.append(CAMERA_PARAM[4]) # + np.random.uniform(-1, 1)*0.1)\r\n k2.append(CAMERA_PARAM[5]) # + np.random.uniform(-1, 1)*0.01)\r\n p1.append(CAMERA_PARAM[6]) # + np.random.uniform(0, 1)*0.1)\r\n p2.append(CAMERA_PARAM[7]) # + np.random.uniform(0, 1)*0.01)\r\n k3.append(CAMERA_PARAM[8]) # + np.random.uniform(0, 1)*0.001)\r\n\r\n L = LensDistortion()\r\n # FOR EVERY METHOD:\r\n for method, (board_size, filename) in patterns.items():\r\n\r\n f = folder.join(filename)\r\n\r\n # LOAD THE SVG FILE, AND SAVE IT WITH NEW RESOLUTION:\r\n svg = QtSvg.QSvgRenderer(f)\r\n image = QtGui.QImage(width * 4, HEIGHT * 4, QtGui.QImage.Format_ARGB32)\r\n image.fill(QtCore.Qt.white)\r\n # Get QPainter that paints to the image\r\n painter = QtGui.QPainter(image)\r\n svg.render(painter)\r\n # Save, image format based on file extension\r\n# f = \"rendered.png\"\r\n# image.save(f)\r\n#\r\n# if GRAYSCALE:\r\n# img = cv2.imread(f, cv2.IMREAD_GRAYSCALE)\r\n# else:\r\n# img = cv2.imread(f)\r\n\r\n img = qImageToArray(image)\r\n\r\n success.append([])\r\n fxl.append([])\r\n errl.append([])\r\n fyl.append([])\r\n cxl.append([])\r\n cyl.append([])\r\n k1l.append([])\r\n k2l.append([])\r\n k3l.append([])\r\n p1l.append([])\r\n p2l.append([])\r\n\r\n dxl.append([])\r\n dyl.append([])\r\n\r\n imgHeight, imgWidth = img.shape[0], img.shape[1]\r\n\r\n for n in range(N_SAMPLES):\r\n L.calibrate(board_size, method)\r\n\r\n print('SET PARAMS:', fx[n], fy[n], cx[n],\r\n cy[n], k1[n], k2[n], k3[n], p1[n], p2[n])\r\n L.setCameraParams(\r\n fx[n], fy[n], cx[n], cy[n], k1[n], k2[n], k3[n], p1[n], p2[n])\r\n L._coeffs['shape'] = (imgHeight, imgWidth)\r\n\r\n hw = imgWidth * 0.5\r\n hh = imgHeight * 0.5\r\n for m in range(N_IMAGES):\r\n pts1 = np.float32([[hw, hh + 100],\r\n [hw - 100, hh - 100],\r\n [hw + 100, hh - 100]])\r\n pts2 = pts1.copy()\r\n if ROTATION:\r\n rotatePolygon(pts2, np.random.randint(0, 2 * np.pi))\r\n\r\n if PERSPECTIVE:\r\n # CREATE A RANDOM PERSPECTIVE:\r\n pts2 += np.random.randint(-hw *\r\n 0.05, hh * 0.05, size=(3, 2))\r\n # MAKE SURE THAT THE PATTERN IS FULLY WITHIN THE IMAGE:\r\n pts2 *= RELATIVE_PATTERN_SIZE\r\n\r\n # MOVE TO THE CENTER\r\n pts2[:, 0] += hw * (1 - RELATIVE_PATTERN_SIZE)\r\n pts2[:, 1] += hh * (1 - RELATIVE_PATTERN_SIZE)\r\n\r\n if POSITION:\r\n f = ((2 * np.random.rand(2)) - 1)\r\n pts2[:, 0] += hw * 0.7 * f[0] * (1 - RELATIVE_PATTERN_SIZE)\r\n pts2[:, 1] += hh * 0.7 * f[1] * (1 - RELATIVE_PATTERN_SIZE)\r\n # EXEC PERSPECTICE, POSITION, ROTATION:\r\n M = cv2.getAffineTransform(pts1, pts2)\r\n img_warped = cv2.warpAffine(\r\n img, M, (imgWidth, imgHeight), borderValue=(230, 230, 230))\r\n # DOWNSCALE IMAGE AGAIN - UPSCALING AND DOWNSCALING SHOULD BRING THE ERRROR\r\n # WARPING DOWN\r\n img_warped = cv2.resize(img_warped, (width, HEIGHT))\r\n # CREATE THE LENS DISTORTION:\r\n mapx, mapy = L.getDistortRectifyMap(width, HEIGHT)\r\n # print 664, mapx.shape\r\n img_distorted = cv2.remap(\r\n img_warped, mapx, mapy, cv2.INTER_LINEAR, borderValue=(230, 230, 230))\r\n\r\n# img_distorted[img_distorted==0]=20\r\n# img_distorted[img_distorted>100]=230\r\n if BLUR:\r\n img_distorted = cv2.blur(img_distorted, BLUR)\r\n if NOISE:\r\n # soften, black and white more gray, and add noise\r\n img_distorted = img_distorted.astype(np.int16)\r\n img_distorted += (np.random.rand(*img_distorted.shape) *\r\n NOISE).astype(img_distorted.dtype)\r\n img_distorted = np.clip(\r\n img_distorted, 0, 255).astype(np.uint8)\r\n# plt.imshow(img_distorted)\r\n# plt.show()\r\n found = L.addImg(img_distorted)\r\n\r\n if SHOW_DETECTED_PATTERN and found:\r\n img_distorted = L.drawChessboard(img_distorted)\r\n cv2.imshow('Pattern', img_distorted)\r\n cv2.waitKey(1)\r\n\r\n success[-1].append(L.findCount)\r\n try:\r\n L._coeffs = None\r\n errl[-1].append(L.coeffs['reprojectionError'])\r\n\r\n L.correct(img_distorted)\r\n\r\n c = L.getCameraParams()\r\n print('GET PARAMS:', c)\r\n\r\n fxl[-1].append(fx[n] - c[0])\r\n fyl[-1].append(fy[n] - c[1])\r\n cxl[-1].append(cx[n] - c[2])\r\n cyl[-1].append(cy[n] - c[3])\r\n k1l[-1].append(k1[n] - c[4])\r\n k2l[-1].append(k2[n] - c[5])\r\n k3l[-1].append(k3[n] - c[6])\r\n p1l[-1].append(p1[n] - c[7])\r\n p2l[-1].append(p2[n] - c[8])\r\n\r\n if PLOT_ERROR_ARRAY:\r\n dx = (mapx - L.mapx) / 2\r\n dy = (mapy - L.mapy) / 2\r\n dxl[-1].append(dx)\r\n dyl[-1].append(dy)\r\n\r\n except NothingFound:\r\n print(\r\n \"Couldn't create a calibration because no patterns were detected\")\r\n\r\n del painter\r\n\r\n # AVERAGE SAMPLES AND GET STD\r\n dx_std, dx_mean = [], []\r\n dy_std, dy_mean = [], []\r\n mag = []\r\n std = []\r\n for patterndx, patterndy in zip(dxl, dyl):\r\n x = np.mean(patterndx, axis=0)\r\n dx_mean.append(x)\r\n y = np.mean(patterndy, axis=0)\r\n dy_mean.append(y)\r\n x = np.std(patterndx, axis=0)\r\n mag.append((x**2 + y**2)**0.5)\r\n dx_std.append(x)\r\n y = np.std(patterndy, axis=0)\r\n dy_std.append(y)\r\n std.append((x**2 + y**2)**0.5)\r\n\r\n # PLOT\r\n p = len(patterns)\r\n if PLOT_RESULTS:\r\n fig, axs = plt.subplots(nrows=2, ncols=5)\r\n\r\n axs = np.array(axs).ravel()\r\n for ax, typ, tname in zip(axs,\r\n (success, fxl, fyl, cxl, cyl,\r\n k1l, k2l, k3l, p1l, p2l),\r\n ('Success rate', 'fx', 'fy', 'cx',\r\n 'cy', 'k1', 'k2', 'k3', 'p1', 'p2')\r\n ):\r\n ax.set_title(tname)\r\n # , showmeans=True, meanline=True)#labels=patterns.keys())\r\n ax.boxplot(typ, notch=0, sym='+', vert=1, whis=1.5)\r\n # , ha=ha[n])\r\n ax.set_xticklabels(patterns.keys(), rotation=40, fontsize=8)\r\n\r\n if PLOT_ERROR_ARRAY:\r\n\r\n mmin = np.min(mag)\r\n mmax = np.max(mag)\r\n smin = np.min(std)\r\n smax = np.max(std)\r\n\r\n plt.figure()\r\n for n, pattern in enumerate(patterns.keys()):\r\n\r\n plt.subplot(int('2%s%s' % (p, n + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(mag[n], origin='upper', vmin=mmin, vmax=mmax)\r\n if n == p - 1:\r\n plt.colorbar(label='Average')\r\n\r\n plt.subplot(int('2%s%s' % (p, n + p + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(std[n], origin='upper', vmin=smin, vmax=smax)\r\n if n == p - 1:\r\n plt.colorbar(label='Standard deviation')\r\n\r\n fig = plt.figure()\r\n fig.suptitle('Individually scaled')\r\n for n, pattern in enumerate(patterns.keys()):\r\n # downscale - show max 30 arrows each dimension\r\n sy, sx = dx_mean[n].shape\r\n ix = int(sx / 15)\r\n if ix < 1:\r\n ix = 1\r\n iy = int(sy / 15)\r\n if iy < 1:\r\n iy = 1\r\n\r\n Y, X = np.meshgrid(np.arange(0, sy, iy), np.arange(0, sx, ix))\r\n\r\n plt.subplot(int('2%s%s' % (p, n + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(mag[n], origin='upper')\r\n plt.colorbar()\r\n plt.quiver(\r\n X, Y, dy_mean[n][::ix, ::iy] * 20, dx_mean[n][::ix, ::iy] * 20)\r\n\r\n plt.subplot(int('2%s%s' % (p, n + p + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(std[n], origin='upper')\r\n plt.colorbar()\r\n # plt.quiver(X,Y,dx_std[n][::ix,::iy]*50, dy_std[n][::ix,::iy]*10)\r\n\r\n #############################################\r\n fig = plt.figure()\r\n fig.suptitle('Spatial uncertainty + deflection')\r\n\r\n for n, pattern in enumerate(patterns.keys()):\r\n L.calibrate(board_size, method)\r\n # there is alot of additional calc thats not necassary:\r\n L.setCameraParams(\r\n fx[0], fy[0], cx[0], cy[0], k1[0], k2[0], k3[0], p1[0], p2[0])\r\n L._coeffs['shape'] = (imgHeight, imgWidth)\r\n L._coeffs['reprojectionError'] = np.mean(errl[n])\r\n\r\n\r\n# deflection_x, deflection_y = L.getDeflection(width, HEIGHT)\r\n# deflection_x += dx_mean[n]\r\n# deflection_y += dy_mean[n]\r\n\r\n ux, uy = L.standardUncertainties()\r\n\r\n plt.subplot(int('2%s%s' % (p, n + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(mag[n], origin='upper')\r\n plt.colorbar()\r\n\r\n # DEFLECTION\r\n plt.subplot(int('2%s%s' % (p, n + p + 1)), axisbg='g')\r\n plt.title(pattern)\r\n plt.imshow(np.linalg.norm([ux, uy], axis=0), origin='upper')\r\n plt.colorbar()\r\n # DEFL: VECTORS\r\n # downscale - show max 30 arrows each dimension\r\n sy, sx = dx_mean[n].shape\r\n ix = int(sx / 15)\r\n if ix < 1:\r\n ix = 1\r\n iy = int(sy / 15)\r\n if iy < 1:\r\n iy = 1\r\n Y, X = np.meshgrid(np.arange(0, sy, iy), np.arange(0, sx, ix))\r\n plt.quiver(X, Y, ux[::ix, ::iy] * 20, uy[::ix, ::iy] * 20)\r\n\r\n if PLOT_ERROR_ARRAY or PLOT_RESULTS:\r\n plt.show()\r\n\r\n return dx_mean, dy_mean"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plotSet(imgDir, posExTime, outDir, show_legend,\r\n show_plots, save_to_file, ftype):\r\n '''\r\n creates plots showing both found GAUSSIAN peaks, the histogram, a smoothed histogram\r\n from all images within [imgDir]\r\n\r\n posExTime - position range of the exposure time in the image name e.g.: img_30s.jpg -> (4,5)\r\n outDir - dirname to save the output images\r\n show_legend - True/False\r\n show_plots - display the result on screen\r\n save_to_file - save the result to file\r\n ftype - file type of the output images\r\n '''\r\n xvals = []\r\n hist = []\r\n peaks = []\r\n exTimes = []\r\n max_border = 0\r\n\r\n if not imgDir.exists():\r\n raise Exception(\"image dir doesn't exist\")\r\n\r\n for n, f in enumerate(imgDir):\r\n print(f)\r\n try:\r\n # if imgDir.join(f).isfile():\r\n img = imgDir.join(f)\r\n s = FitHistogramPeaks(img)\r\n xvals.append(s.xvals)\r\n hist.append(s.yvals)\r\n# smoothedHist.append(s.yvals2)\r\n peaks.append(s.fitValues())\r\n\r\n if s.border() > max_border:\r\n max_border = s.plotBorder()\r\n\r\n exTimes.append(float(f[posExTime[0]:posExTime[1] + 1]))\r\n except:\r\n pass\r\n nx = 2\r\n ny = int(len(hist) // nx) + len(hist) % nx\r\n\r\n fig, ax = plt.subplots(ny, nx)\r\n\r\n # flatten 2d-ax list:\r\n if nx > 1:\r\n ax = [list(i) for i in zip(*ax)] # transpose 2d-list\r\n axx = []\r\n for xa in ax:\r\n for ya in xa:\r\n axx.append(ya)\r\n ax = axx\r\n\r\n for x, h, p, e, a in zip(xvals, hist, peaks, exTimes, ax):\r\n\r\n a.plot(x, h, label='histogram', thickness=3)\r\n# l1 = a.plot(x, s, label='smoothed')\r\n for n, pi in enumerate(p):\r\n l2 = a.plot(x, pi, label='peak %s' % n, thickness=6)\r\n a.set_xlim(xmin=0, xmax=max_border)\r\n a.set_title('%s s' % e)\r\n\r\n# plt.setp([l1,l2], linewidth=2)#, linestyle='--', color='r') # set\r\n# both to dashed\r\n\r\n l1 = ax[0].legend() # loc='upper center', bbox_to_anchor=(0.7, 1.05),\r\n l1.draw_frame(False)\r\n\r\n plt.xlabel('pixel value')\r\n plt.ylabel('number of pixels')\r\n\r\n fig = plt.gcf()\r\n fig.set_size_inches(7 * nx, 3 * ny)\r\n\r\n if save_to_file:\r\n p = PathStr(outDir).join('result').setFiletype(ftype)\r\n plt.savefig(p, bbox_inches='tight')\r\n\r\n if show_plots:\r\n plt.show()", "response": "Plots a set of GAUSSIAN peaks and a smoothed histogram of all images within the image dir."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\naverages multiple images of a homogeneous device imaged directly in front the camera lens. if [bg_imgs] are not given, background level is extracted from 1% of the cumulative intensity distribution of the averaged [imgs] This measurement method is referred as 'Method A' in --- K.Bedrich, M.Bokalic et al.: ELECTROLUMINESCENCE IMAGING OF PV DEVICES: ADVANCED FLAT FIELD CALIBRATION,2017 ---", "response": "def flatFieldFromCloseDistance(imgs, bg_imgs=None):\r\n '''\r\n Average multiple images of a homogeneous device\r\n imaged directly in front the camera lens.\r\n\r\n if [bg_imgs] are not given, background level is extracted\r\n from 1% of the cumulative intensity distribution\r\n of the averaged [imgs]\r\n\r\n This measurement method is referred as 'Method A' in\r\n ---\r\n K.Bedrich, M.Bokalic et al.:\r\n ELECTROLUMINESCENCE IMAGING OF PV DEVICES:\r\n ADVANCED FLAT FIELD CALIBRATION,2017\r\n ---\r\n '''\r\n img = imgAverage(imgs)\r\n bg = getBackground2(bg_imgs, img)\r\n img -= bg\r\n img = toGray(img)\r\n mx = median_filter(img[::10, ::10], 3).max()\r\n img /= mx\r\n return img"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flatFieldFromCloseDistance2(images, bgImages=None, calcStd=False,\r\n nlf=None, nstd=6):\r\n '''\r\n Same as [flatFieldFromCloseDistance]. Differences are:\r\n ... single-time-effect removal included\r\n ... returns the standard deviation of the image average [calcStd=True]\r\n\r\n Optional:\r\n -----------\r\n calcStd -> set to True to also return the standard deviation\r\n nlf -> noise level function (callable)\r\n nstd -> artefact needs to deviate more than [nstd] to be removed\r\n '''\r\n\r\n if len(images) > 1:\r\n\r\n # start with brightest images\r\n def fn(img):\r\n img = imread(img)\r\n s0, s1 = img.shape[:2]\r\n # rough approx. of image brightness:\r\n return -img[::s0 // 10, ::s1 // 10].min()\r\n\r\n images = sorted(images, key=lambda i: fn(i))\r\n\r\n avgBg = getBackground2(bgImages, images[1])\r\n\r\n i0 = imread(images[0], dtype=float) - avgBg\r\n i1 = imread(images[1], dtype=float) - avgBg\r\n\r\n if nlf is None:\r\n nlf = oneImageNLF(i0, i1)[0]\r\n\r\n det = SingleTimeEffectDetection(\r\n (i0, i1), nlf, nStd=nstd, calcVariance=calcStd)\r\n\r\n for i in images[1:]:\r\n i = imread(i)\r\n # exclude erroneously darker areas:\r\n thresh = det.noSTE - nlf(det.noSTE) * nstd\r\n mask = i > thresh\r\n # filter STE:\r\n det.addImage(i, mask)\r\n\r\n ma = det.noSTE\r\n\r\n else:\r\n ma = imread(images[0], dtype=float) - avgBg\r\n\r\n # fast artifact free maximum:\r\n mx = median_filter(ma[::10, ::10], 3).max()\r\n\r\n if calcStd:\r\n return ma / mx, det.mma.var**0.5 / mx\r\n\r\n return ma / mx", "response": "Returns the flat field from close distance images."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the signal - to - noise ratio of the given images.", "response": "def SNR_hinken(imgs, bg=0, roi=None):\r\n '''\r\n signal-to-noise ratio (SNR) as mean(images) / std(images)\r\n as defined in Hinken et.al. 2011 (DOI: 10.1063/1.3541766)\r\n \r\n works on unloaded images\r\n no memory overload if too many images are given\r\n '''\r\n mean = None\r\n M = len(imgs)\r\n if bg is not 0:\r\n bg = imread(bg)[roi]\r\n if roi is not None:\r\n bg = bg[roi]\r\n #calc mean:\r\n for i in imgs:\r\n img = imread(i).asfarray()\r\n if roi is not None:\r\n img = img[roi]\r\n img -= bg\r\n if mean is None:\r\n #init\r\n mean = np.zeros_like(img)\r\n std = np.zeros_like(img)\r\n mean += img\r\n del img\r\n mean /= M\r\n #calc std of mean:\r\n for i in imgs:\r\n img = imread(i).asfarray()\r\n if roi is not None:\r\n img = img[roi]\r\n img -= bg\r\n std += (mean - img)**2\r\n del img\r\n std = (std / M)**0.5\r\n return mean.mean() / std.mean()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntransforming at maximum 8 bool layers --> 2d arrays dtype = bool int array", "response": "def boolMasksToImage(masks):\r\n '''\r\n Transform at maximum 8 bool layers --> 2d arrays, dtype=(bool,int)\r\n to one 8bit image\r\n '''\r\n assert len(masks) <= 8, 'can only transform up to 8 masks into image'\r\n masks = np.asarray(masks, dtype=np.uint8)\r\n assert masks.ndim == 3, 'layers need to be stack of 2d arrays'\r\n return np.packbits(masks, axis=0)[0].T"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninverse of [ boolMasksToImage ]", "response": "def imageToBoolMasks(arr):\r\n '''inverse of [boolMasksToImage]'''\r\n assert arr.dtype == np.uint8, 'image needs to be dtype=uint8'\r\n masks = np.unpackbits(arr).reshape(*arr.shape, 8)\r\n return np.swapaxes(masks, 2, 0)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the aspect ratio of a new object from a list of corners.", "response": "def calcAspectRatioFromCorners(corners, in_plane=False):\r\n '''\r\n simple and better alg. than below\r\n in_plane -> whether object has no tilt, but only rotation and translation\r\n '''\r\n\r\n q = corners\r\n l0 = [q[0, 0], q[0, 1], q[1, 0], q[1, 1]]\r\n l1 = [q[0, 0], q[0, 1], q[-1, 0], q[-1, 1]]\r\n\r\n l2 = [q[2, 0], q[2, 1], q[3, 0], q[3, 1]]\r\n l3 = [q[2, 0], q[2, 1], q[1, 0], q[1, 1]]\r\n\r\n a1 = line.length(l0) / line.length(l1)\r\n a2 = line.length(l2) / line.length(l3)\r\n\r\n if in_plane:\r\n # take aspect ration from more rectangular corner\r\n if (abs(0.5 * np.pi - abs(line.angle2(l0, l1)))\r\n < abs(0.5 * np.pi - abs(line.angle2(l2, l3)))):\r\n return a1\r\n else:\r\n return a2\r\n\r\n return 0.5 * (a1 + a2)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef putTextAlpha(img, text, alpha, org, fontFace, fontScale, color,\r\n thickness): # , lineType=None\r\n '''\r\n Extends cv2.putText with [alpha] argument\r\n '''\r\n\r\n x, y = cv2.getTextSize(text, fontFace,\r\n fontScale, thickness)[0]\r\n\r\n ox, oy = org\r\n\r\n imgcut = img[oy - y - 3:oy, ox:ox + x]\r\n\r\n if img.ndim == 3:\r\n txtarr = np.zeros(shape=(y + 3, x, 3), dtype=np.uint8)\r\n else:\r\n txtarr = np.zeros(shape=(y + 3, x), dtype=np.uint8)\r\n\r\n cv2.putText(txtarr, text, (0, y), fontFace,\r\n fontScale, color,\r\n thickness=thickness\r\n #, lineType=lineType\r\n )\r\n\r\n cv2.addWeighted(txtarr, alpha, imgcut, 1, 0, imgcut, -1)\r\n return img", "response": "Adds text to image with alpha color."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbuilds a file path from the given list of paths and return the contents.", "response": "def read(*paths):\r\n \"\"\"Build a file path from *paths* and return the contents.\"\"\"\r\n try:\r\n f_name = os.path.join(*paths)\r\n with open(f_name, 'r') as f:\r\n return f.read()\r\n except IOError:\r\n print('%s not existing ... skipping' % f_name)\r\n return ''"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a DarkCurrentMap that has the average background images with same exposure time.", "response": "def averageSameExpTimes(imgs_path):\r\n '''\r\n average background images with same exposure time\r\n '''\r\n firsts = imgs_path[:2]\r\n imgs = imgs_path[2:]\r\n for n, i in enumerate(firsts):\r\n firsts[n] = np.asfarray(imread(i))\r\n d = DarkCurrentMap(firsts)\r\n for i in imgs:\r\n i = imread(i)\r\n d.addImg(i)\r\n return d.map()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the linear interpolation function for the given set of images.", "response": "def getLinearityFunction(expTimes, imgs, mxIntensity=65535, min_ascent=0.001,\r\n ):\r\n '''\r\n returns offset, ascent \r\n of image(expTime) = offset + ascent*expTime\r\n '''\r\n # TODO: calculate [min_ascent] from noise function\r\n # instead of having it as variable\r\n\r\n ascent, offset, error = linRegressUsingMasked2dArrays(\r\n expTimes, imgs, imgs > mxIntensity)\r\n\r\n ascent[np.isnan(ascent)] = 0\r\n # remove low frequent noise:\r\n if min_ascent > 0:\r\n i = ascent < min_ascent\r\n offset[i] += (0.5 * (np.min(expTimes) + np.max(expTimes))) * ascent[i]\r\n ascent[i] = 0\r\n\r\n return offset, ascent, error"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsort image paths for same exposure time", "response": "def sortForSameExpTime(expTimes, img_paths): # , excludeSingleImg=True):\r\n '''\r\n return image paths sorted for same exposure time\r\n '''\r\n d = {}\r\n for e, i in zip(expTimes, img_paths):\r\n if e not in d:\r\n d[e] = []\r\n d[e].append(i)\r\n# for key in list(d.keys()):\r\n# if len(d[key]) == 1:\r\n# print('have only one image of exposure time [%s]' % key)\r\n# print('--> exclude that one')\r\n# d.pop(key)\r\n d = OrderedDict(sorted(d.items()))\r\n return list(d.keys()), list(d.values())"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the exposure times and image averages for each exposure time returns exposure times and image averages for each exposure time returns exposure times imgs averages for each exposure time imgs_p returns imgs_p", "response": "def getDarkCurrentAverages(exposuretimes, imgs):\r\n '''\r\n return exposure times, image averages for each exposure time\r\n '''\r\n x, imgs_p = sortForSameExpTime(exposuretimes, imgs)\r\n s0, s1 = imgs[0].shape\r\n\r\n imgs = np.empty(shape=(len(x), s0, s1),\r\n dtype=imgs[0].dtype)\r\n for i, ip in zip(imgs, imgs_p):\r\n if len(ip) == 1:\r\n i[:] = ip[0]\r\n else:\r\n i[:] = averageSameExpTimes(ip)\r\n return x, imgs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getDarkCurrentFunction(exposuretimes, imgs, **kwargs):\r\n '''\r\n get dark current function from given images and exposure times\r\n '''\r\n exposuretimes, imgs = getDarkCurrentAverages(exposuretimes, imgs)\r\n offs, ascent, rmse = getLinearityFunction(exposuretimes, imgs, **kwargs)\r\n return offs, ascent, rmse", "response": "get dark current function from given images and exposure times"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a subimage aligned along given line.", "response": "def alignImageAlongLine(img, line, height=15, length=None,\r\n zoom=1, fast=False, borderValue=0):\r\n '''\r\n return a sub image aligned along given line\r\n\r\n @param img - numpy.2darray input image to get subimage from\r\n @param line - list of 2 points [x0,y0,x1,y1])\r\n @param height - height of output array in y\r\n @param length - width of output array\r\n @param zoom - zoom factor\r\n @param fast - speed up calculation using nearest neighbour interpolation\r\n @returns transformed image as numpy.2darray with found line as in the middle\r\n '''\r\n\r\n height = int(round(height))\r\n if height % 2 == 0: # ->is even number\r\n height += 1 # only take uneven numbers to have line in middle\r\n if length is None:\r\n length = int(round(ln.length(line)))\r\n hh = (height - 1)\r\n ll = (length - 1)\r\n\r\n # end points of the line:\r\n p0 = np.array(line[0:2], dtype=float)\r\n p1 = np.array(line[2:], dtype=float)\r\n # p2 is above middle of p0,p1:\r\n norm = np.array(ln.normal(line))\r\n if not ln.isHoriz(line):\r\n norm *= -1\r\n\r\n p2 = (p0 + p1) * 0.5 + norm * hh * 0.5\r\n middleY = hh / 2\r\n pp0 = [0, middleY]\r\n pp1 = [ll, middleY]\r\n pp2 = [ll * 0.5, hh]\r\n\r\n pts1 = np.array([p0, p1, p2], dtype=np.float32)\r\n pts2 = np.array([pp0, pp1, pp2], dtype=np.float32)\r\n\r\n if zoom != 1:\r\n length = int(round(length * zoom))\r\n height = int(round(height * zoom))\r\n pts2 *= zoom\r\n\r\n # TRANSFORM:\r\n M = cv2.getAffineTransform(pts1, pts2)\r\n dst = cv2.warpAffine(\r\n img, M, (length, height),\r\n flags=cv2.INTER_NEAREST if fast else cv2.INTER_LINEAR,\r\n borderValue=borderValue)\r\n return dst"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the estimated standard deviation map from the changes of neighbouring pixels from a center pixel within a point spread function within a center pixel within a point spread function within a center pixel within a point spread function within a center pixel within a point spread function in x and y taken from the ( sx sy ).", "response": "def positionToIntensityUncertainty(image, sx, sy, kernelSize=None):\r\n '''\r\n calculates the estimated standard deviation map from the changes\r\n of neighbouring pixels from a center pixel within a point spread function\r\n defined by a std.dev. in x and y taken from the (sx, sy) maps \r\n\r\n sx,sy -> either 2d array of same shape as [image]\r\n of single values\r\n '''\r\n psf_is_const = not isinstance(sx, np.ndarray)\r\n if not psf_is_const:\r\n assert image.shape == sx.shape == sy.shape, \\\r\n \"Image and position uncertainty maps need to have same size\"\r\n if kernelSize is None:\r\n kernelSize = _kSizeFromStd(max(sx.max(), sy.max()))\r\n else:\r\n assert type(sx) in (int, float) and type(sx) in (int, float), \\\r\n \"Image and position uncertainty values need to be int OR float\"\r\n if kernelSize is None:\r\n kernelSize = _kSizeFromStd(max(sx, sy))\r\n\r\n if image.dtype.kind == 'u':\r\n image = image.astype(int) # otherwise stack overflow through uint\r\n size = kernelSize // 2\r\n if size < 1:\r\n size = 1\r\n kernelSize = 1 + 2 * size\r\n # array to be filled by individual psf of every pixel:\r\n psf = np.zeros((kernelSize, kernelSize))\r\n # intensity uncertainty as stdev:\r\n sint = np.zeros(image.shape)\r\n if psf_is_const:\r\n _calc_constPSF(image, sint, sx, sy, psf, size)\r\n else:\r\n _calc_variPSF(image, sint, sx, sy, psf, size)\r\n return sint"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _coarsenImage(image, f):\r\n '''\r\n seems to be a more precise (but slower)\r\n way to down-scale an image\r\n '''\r\n from skimage.morphology import square\r\n from skimage.filters import rank\r\n from skimage.transform._warps import rescale\r\n selem = square(f)\r\n arri = rank.mean(image, selem=selem)\r\n return rescale(arri, 1 / f, order=0)", "response": "Returns a new image with the given number of coarsening components."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef positionToIntensityUncertaintyForPxGroup(image, std, y0, y1, x0, x1):\r\n '''\r\n like positionToIntensityUncertainty\r\n but calculated average uncertainty for an area [y0:y1,x0:x1]\r\n '''\r\n fy, fx = y1 - y0, x1 - x0\r\n if fy != fx:\r\n raise Exception('averaged area need to be square ATM')\r\n image = _coarsenImage(image, fx)\r\n k = _kSizeFromStd(std)\r\n y0 = int(round(y0 / fy))\r\n x0 = int(round(x0 / fx))\r\n arr = image[y0 - k:y0 + k, x0 - k:x0 + k]\r\n U = positionToIntensityUncertainty(arr, std / fx, std / fx)\r\n return U[k:-k, k:-k]", "response": "Like positionToIntensityUncertainty but calculated average uncertainty for an area [ y0 x0 x1"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef nan_maximum_filter(arr, ksize):\r\n '''\r\n same as scipy.filters.maximum_filter\r\n but working excluding nans\r\n '''\r\n out = np.empty_like(arr)\r\n _calc(arr, out, ksize//2)\r\n return out", "response": "same as scipy. filters. maximum_filter\r\n but working excluding nan"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fastFilter(arr, ksize=30, every=None, resize=True, fn='median',\r\n interpolation=cv2.INTER_LANCZOS4,\r\n smoothksize=0,\r\n borderMode=cv2.BORDER_REFLECT):\r\n '''\r\n fn['nanmean', 'mean', 'nanmedian', 'median']\r\n \r\n a fast 2d filter for large kernel sizes that also \r\n works with nans\r\n the computation speed is increased because only 'every'nsth position \r\n within the median kernel is evaluated\r\n '''\r\n if every is None:\r\n every = max(ksize//3, 1)\r\n else:\r\n assert ksize >= 3*every\r\n s0,s1 = arr.shape[:2]\r\n \r\n ss0 = s0//every\r\n every = s0//ss0\r\n ss1 = s1//every\r\n \r\n out = np.full((ss0+1,ss1+1), np.nan)\r\n \r\n c = {'median':_calcMedian,\r\n 'nanmedian':_calcNanMedian,\r\n 'nanmean':_calcNanMean,\r\n 'mean':_calcMean,\r\n }[fn]\r\n ss0,ss1 = c(arr, out, ksize, every)\r\n out = out[:ss0,:ss1]\r\n \r\n if smoothksize:\r\n out = gaussian_filter(out, smoothksize)\r\n \r\n \r\n if not resize:\r\n return out\r\n return cv2.resize(out, arr.shape[:2][::-1],\r\n interpolation=interpolation)", "response": "fast filter for large kernel sizes"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef elbin(filename):\r\n '''\r\n Read EL images (*.elbin) created by the RELTRON EL Software\r\n http://www.reltron.com/Products/Solar.html\r\n '''\r\n# arrs = []\r\n labels = []\r\n\r\n # These are all exposure times [s] to be selectable:\r\n TIMES = (0.3, 0.4, 0.6, 0.8, 1.2, 1.6, 2.4, 3.2, 4.8, 6.4, 9.6, 12.8, 19.2,\r\n 25.6, 38.4, 51.2, 76.8, 102.6, 153.6, 204.6, 307.2, 409.8, 614.4,\r\n 819., 1228.8, 1638.6, 3276.6, 5400., 8100., 12168., 18216., 27324.,\r\n 41004., 61488., 92268.)\r\n\r\n with open(filename, 'rb') as f:\r\n # image shape and number:\r\n height, width, frames = np.frombuffer(f.read(4 * 3), dtype=np.uint32)\r\n arrs = np.empty((frames, width, height), dtype=np.uint16)\r\n for i in range(frames):\r\n # read header between all frames:\r\n current, voltage = np.frombuffer(f.read(8 * 2), dtype=np.float64)\r\n i_time = np.frombuffer(f.read(4), dtype=np.uint32)[0]\r\n time = TIMES[i_time]\r\n # read image:\r\n arr = np.frombuffer(f.read(width * height * 2), dtype=np.uint16)\r\n arrs[i] = arr.reshape(width, height)\r\n # last row is all zeros in all imgs\r\n # print arr[:,:-1]\r\n\r\n# arrs.append(arr)\r\n labels.append({'exposure time[s]': time,\r\n 'current[A]': current,\r\n 'voltage[V]': voltage})\r\n return arrs, labels", "response": "Reads the EL images created by the RELTRON EL Software and returns a list of tuples each containing the current time and the number of times in the image."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef gaussian2d(xy, sx, sy, mx=0, my=0, rho=0, amp=1, offs=0):\r\n '''\r\n see http://en.wikipedia.org/wiki/Multivariate_normal_distribution\r\n # probability density function of a vector [x,y]\r\n sx,sy -> sigma (standard deviation)\r\n mx,my: mue (mean position)\r\n rho: correlation between x and y\r\n '''\r\n x,y = xy\r\n return offs+amp*( \r\n 1/(2*np.pi*sx*sy*(1-(rho**2))**0.5) *\r\n np.exp( (-1/(2*(1-rho**2))) *\r\n (\r\n ( (x-mx)**2/sx**2 )\r\n + ( (y-my)**2/sy**2 )\r\n - ( ( 2*rho*(x-mx)*(y-my)) / (sx*sy) )\r\n )\r\n )\r\n )", "response": "Returns the 2D gaussian distribution of a vector x y"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fitImg(self, img_rgb):\r\n '''\r\n fit perspective and size of the input image to the base image\r\n '''\r\n H = self.pattern.findHomography(img_rgb)[0]\r\n H_inv = self.pattern.invertHomography(H)\r\n s = self.img_orig.shape\r\n warped = cv2.warpPerspective(img_rgb, H_inv, (s[1], s[0]))\r\n return warped", "response": "fit perspective and size of the input image to the base image"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef scaleSignalCut(img, ratio, nbins=100):\r\n '''\r\n scaling img cutting x percent of top and bottom part of histogram\r\n '''\r\n start, stop = scaleSignalCutParams(img, ratio, nbins)\r\n img = img - start\r\n img /= (stop - start)\r\n return img", "response": "scale the img cutting x percent of top and bottom part of histogram"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef scaleSignal(img, fitParams=None,\r\n backgroundToZero=False, reference=None):\r\n '''\r\n scale the image between...\r\n\r\n backgroundToZero=True -> 0 (average background) and 1 (maximum signal)\r\n backgroundToZero=False -> signal+-3std\r\n\r\n reference -> reference image -- scale image to fit this one\r\n\r\n returns:\r\n scaled image\r\n '''\r\n img = imread(img)\r\n if reference is not None:\r\n # def fn(ii, m,n):\r\n # return ii*m+n\r\n # curve_fit(fn, img[::10,::10], ref[::10,::10])\r\n\r\n low, high = signalRange(img, fitParams)\r\n low2, high2 = signalRange(reference)\r\n img = np.asfarray(img)\r\n ampl = (high2 - low2) / (high - low)\r\n img -= low\r\n img *= ampl\r\n img += low2\r\n return img\r\n else:\r\n offs, div = scaleParams(img, fitParams, backgroundToZero)\r\n img = np.asfarray(img) - offs\r\n img /= div\r\n print('offset: %s, divident: %s' % (offs, div))\r\n return img", "response": "scale the image between... and 1 - 3std"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getBackgroundRange(fitParams):\r\n '''\r\n return minimum, average, maximum of the background peak\r\n '''\r\n smn, _, _ = getSignalParameters(fitParams)\r\n\r\n bg = fitParams[0]\r\n _, avg, std = bg\r\n bgmn = max(0, avg - 3 * std)\r\n\r\n if avg + 4 * std < smn:\r\n bgmx = avg + 4 * std\r\n if avg + 3 * std < smn:\r\n bgmx = avg + 3 * std\r\n if avg + 2 * std < smn:\r\n bgmx = avg + 2 * std\r\n else:\r\n bgmx = avg + std\r\n return bgmn, avg, bgmx", "response": "Returns the minimum average maximum of the background peak peak\r\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if there is background", "response": "def hasBackground(fitParams):\r\n '''\r\n compare the height of putative bg and signal peak\r\n if ratio if too height assume there is no background\r\n '''\r\n signal = getSignalPeak(fitParams)\r\n bg = getBackgroundPeak(fitParams)\r\n if signal == bg:\r\n return False\r\n r = signal[0] / bg[0]\r\n if r < 1:\r\n r = 1 / r\r\n return r < 100"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef signalMinimum2(img, bins=None):\r\n '''\r\n minimum position between signal and background peak\r\n '''\r\n f = FitHistogramPeaks(img, bins=bins)\r\n i = signalPeakIndex(f.fitParams)\r\n spos = f.fitParams[i][1]\r\n# spos = getSignalPeak(f.fitParams)[1]\r\n# bpos = getBackgroundPeak(f.fitParams)[1]\r\n bpos = f.fitParams[i - 1][1]\r\n ind = np.logical_and(f.xvals > bpos, f.xvals < spos)\r\n try:\r\n i = np.argmin(f.yvals[ind])\r\n return f.xvals[ind][i]\r\n except ValueError as e:\r\n if bins is None:\r\n return signalMinimum2(img, bins=400)\r\n else:\r\n raise e", "response": "Returns the minimum position between signal and background peak."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef signalMinimum(img, fitParams=None, n_std=3):\r\n '''\r\n intersection between signal and background peak\r\n '''\r\n if fitParams is None:\r\n fitParams = FitHistogramPeaks(img).fitParams\r\n assert len(fitParams) > 1, 'need 2 peaks so get minimum signal'\r\n\r\n i = signalPeakIndex(fitParams)\r\n signal = fitParams[i]\r\n bg = getBackgroundPeak(fitParams)\r\n smn = signal[1] - n_std * signal[2]\r\n bmx = bg[1] + n_std * bg[2]\r\n if smn > bmx:\r\n return smn\r\n # peaks are overlapping\r\n # define signal min. as intersection between both Gaussians\r\n\r\n def solve(p1, p2):\r\n s1, m1, std1 = p1\r\n s2, m2, std2 = p2\r\n a = (1 / (2 * std1**2)) - (1 / (2 * std2**2))\r\n b = (m2 / (std2**2)) - (m1 / (std1**2))\r\n c = (m1**2 / (2 * std1**2)) - (m2**2 / (2 * std2**2)) - \\\r\n np.log(((std2 * s1) / (std1 * s2)))\r\n return np.roots([a, b, c])\r\n i = solve(bg, signal)\r\n try:\r\n return i[np.logical_and(i > bg[1], i < signal[1])][0]\r\n except IndexError:\r\n # this error shouldn't occur... well\r\n return max(smn, bmx)", "response": "get minimum signal peak"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the minimum average maximum of the signal peak and the signal peak peak.", "response": "def getSignalParameters(fitParams, n_std=3):\r\n '''\r\n return minimum, average, maximum of the signal peak\r\n '''\r\n signal = getSignalPeak(fitParams)\r\n mx = signal[1] + n_std * signal[2]\r\n mn = signal[1] - n_std * signal[2]\r\n if mn < fitParams[0][1]:\r\n mn = fitParams[0][1] # set to bg\r\n return mn, signal[1], mx"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef equalizeImage(img, save_path=None, name_additive='_eqHist'):\r\n '''\r\n Equalize the histogram (contrast) of an image\r\n works with RGB/multi-channel images\r\n and flat-arrays\r\n\r\n @param img - image_path or np.array\r\n @param save_path if given output images will be saved there\r\n @param name_additive if given this additive will be appended to output images\r\n\r\n @return output images if input images are numpy.arrays and no save_path is given\r\n @return None elsewise\r\n '''\r\n\r\n if isinstance(img, string_types):\r\n img = PathStr(img)\r\n if not img.exists():\r\n raise Exception(\"image path doesn't exist\")\r\n img_name = img.basename().replace('.', '%s.' % name_additive)\r\n if save_path is None:\r\n save_path = img.dirname()\r\n img = cv2.imread(img)\r\n\r\n if img.dtype != np.dtype('uint8'):\r\n # openCV cannot work with float arrays or uint > 8bit\r\n eqFn = _equalizeHistogram\r\n else:\r\n eqFn = cv2.equalizeHist\r\n if len(img.shape) == 3: # multi channel img like rgb\r\n for i in range(img.shape[2]):\r\n img[:, :, i] = eqFn(img[:, :, i])\r\n else: # grey scale image\r\n img = eqFn(img)\r\n if save_path:\r\n img_name = PathStr(save_path).join(img_name)\r\n cv2.imwrite(img_name, img)\r\n return img", "response": "Equalize the histogram of an image"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef localizedMaximum(img, thresh=0, min_increase=0, max_length=0, dtype=bool):\r\n '''\r\n Returns the local maximum of a given 2d array\r\n\r\n\r\n thresh -> if given, ignore all values below that value\r\n\r\n max_length -> limit length between value has to vary > min_increase\r\n\r\n >>> a = np.array([[0,1,2,3,2,1,0], \\\r\n [0,1,2,2,3,1,0], \\\r\n [0,1,1,2,2,3,0], \\\r\n [0,1,1,2,1,1,0], \\\r\n [0,0,0,1,1,0,0]])\r\n\r\n >>> print localizedMaximum(a, dtype=int)\r\n [[0 1 1 1 0 1 0]\r\n [0 0 0 0 1 0 0]\r\n [0 0 0 1 0 1 0]\r\n [0 0 1 1 0 1 0]\r\n [0 0 0 1 0 0 0]]\r\n '''\r\n # because numba cannot create arrays:\r\n out = np.zeros(shape=img.shape, dtype=dtype)\r\n # first iterate all rows:\r\n _calc(img, out, thresh, min_increase, max_length)\r\n # that all columns:\r\n _calc(img.T, out.T, thresh, min_increase, max_length)\r\n return out", "response": "Returns the localized maximum of a given 2d array"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the reference of the object", "response": "def setReference(self, ref):\r\n '''\r\n ref ... either quad, grid, homography or reference image\r\n\r\n quad --> list of four image points(x,y) marking the edges of the quad\r\n to correct\r\n homography --> h. matrix to correct perspective distortion\r\n referenceImage --> image of same object without perspective distortion\r\n '''\r\n# self.maps = {}\r\n self.quad = None\r\n# self.refQuad = None\r\n self._camera_position = None\r\n self._homography = None\r\n self._homography_is_fixed = True\r\n# self.tvec, self.rvec = None, None\r\n self._pose = None\r\n\r\n # evaluate input:\r\n if isinstance(ref, np.ndarray) and ref.shape == (3, 3):\r\n # REF IS HOMOGRAPHY\r\n self._homography = ref\r\n # REF IS QUAD\r\n elif len(ref) == 4:\r\n self.quad = sortCorners(ref)\r\n\r\n # TODO: cleanup # only need to call once - here\r\n o = self.obj_points # no property any more\r\n\r\n # REF IS IMAGE\r\n else:\r\n self.ref = imread(ref)\r\n# self._refshape = ref.shape[:2]\r\n self.pattern = PatternRecognition(self.ref)\r\n self._homography_is_fixed = False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\napply perspective distortion ion to image and return a new image.", "response": "def distort(self, img, rotX=0, rotY=0, quad=None):\r\n '''\r\n Apply perspective distortion ion self.img\r\n angles are in DEG and need to be positive to fit into image\r\n\r\n '''\r\n self.img = imread(img)\r\n # fit old image to self.quad:\r\n corr = self.correct(self.img)\r\n\r\n s = self.img.shape\r\n if quad is None:\r\n wquad = (self.quad - self.quad.mean(axis=0)).astype(float)\r\n\r\n win_width = s[1]\r\n win_height = s[0]\r\n # project quad:\r\n for n, q in enumerate(wquad):\r\n p = Point3D(q[0], q[1], 0).rotateX(-rotX).rotateY(-rotY)\r\n p = p.project(win_width, win_height, s[1], s[1])\r\n wquad[n] = (p.x, p.y)\r\n wquad = sortCorners(wquad)\r\n # scale result so that longest side of quad and wquad are equal\r\n w = wquad[:, 0].max() - wquad[:, 0].min()\r\n h = wquad[:, 1].max() - wquad[:, 1].min()\r\n scale = min(s[1] / w, s[0] / h)\r\n # scale:\r\n wquad = (wquad * scale).astype(int)\r\n else:\r\n wquad = sortCorners(quad)\r\n wquad -= wquad.min(axis=0)\r\n\r\n lx = corr.shape[1]\r\n ly = corr.shape[0]\r\n\r\n objP = np.array([\r\n [0, 0],\r\n [lx, 0],\r\n [lx, ly],\r\n [0, ly],\r\n ], dtype=np.float32)\r\n\r\n homography = cv2.getPerspectiveTransform(\r\n wquad.astype(np.float32), objP)\r\n # distort corr:\r\n w = wquad[:, 0].max() - wquad[:, 0].min()\r\n h = wquad[:, 1].max() - wquad[:, 1].min()\r\n #(int(w),int(h))\r\n dist = cv2.warpPerspective(corr, homography, (int(w), int(h)),\r\n flags=cv2.INTER_CUBIC | cv2.WARP_INVERSE_MAP)\r\n\r\n # move middle of dist to middle of the old quad:\r\n bg = np.zeros(shape=s)\r\n rmn = (bg.shape[0] / 2, bg.shape[1] / 2)\r\n\r\n ss = dist.shape\r\n mn = (ss[0] / 2, ss[1] / 2) # wquad.mean(axis=0)\r\n ref = (int(rmn[0] - mn[0]), int(rmn[1] - mn[1]))\r\n\r\n bg[ref[0]:ss[0] + ref[0], ref[1]:ss[1] + ref[1]] = dist\r\n\r\n # finally move quad into right position:\r\n self.quad = wquad\r\n self.quad += (ref[1], ref[0])\r\n self.img = bg\r\n self._homography = None\r\n self._poseFromQuad()\r\n\r\n if self.opts['do_correctIntensity']:\r\n tf = self.tiltFactor()\r\n if self.img.ndim == 3:\r\n for col in range(self.img.shape[2]):\r\n self.img[..., col] *= tf\r\n else:\r\n # tf = np.tile(tf, (1,1,self.img.shape[2]))\r\n self.img = self.img * tf\r\n\r\n return self.img"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef correctGrid(self, img, grid):\r\n '''\r\n grid -> array of polylines=((p0x,p0y),(p1x,p1y),,,)\r\n '''\r\n\r\n self.img = imread(img)\r\n h = self.homography # TODO: cleanup only needed to get newBorder attr.\r\n\r\n if self.opts['do_correctIntensity']:\r\n self.img = self.img / self._getTiltFactor(self.img.shape)\r\n\r\n s0, s1 = grid.shape[:2]\r\n n0, n1 = s0 - 1, s1 - 1\r\n\r\n snew = self._newBorders\r\n b = self.opts['border']\r\n\r\n sx, sy = (snew[0] - 2 * b) // n0, (snew[1] - 2 * b) // n1\r\n\r\n out = np.empty(snew[::-1], dtype=self.img.dtype)\r\n\r\n def warp(ix, iy, objP, outcut):\r\n shape = outcut.shape[::-1]\r\n quad = grid[ix:ix + 2,\r\n iy:iy + 2].reshape(4, 2)[np.array([0, 2, 3, 1])]\r\n hcell = cv2.getPerspectiveTransform(\r\n quad.astype(np.float32), objP)\r\n cv2.warpPerspective(self.img, hcell, shape, outcut,\r\n flags=cv2.INTER_LANCZOS4,\r\n **self.opts['cv2_opts'])\r\n return quad\r\n\r\n objP = np.array([[0, 0],\r\n [sx, 0],\r\n [sx, sy],\r\n [0, sy]], dtype=np.float32)\r\n # INNER CELLS\r\n for ix in range(1, n0 - 1):\r\n for iy in range(1, n1 - 1):\r\n sub = out[iy * sy + b: (iy + 1) * sy + b,\r\n ix * sx + b: (ix + 1) * sx + b]\r\n# warp(ix, iy, objP, sub)\r\n\r\n shape = sub.shape[::-1]\r\n quad = grid[ix:ix + 2,\r\n iy:iy + 2].reshape(4, 2)[np.array([0, 2, 3, 1])]\r\n# print(quad, objP)\r\n\r\n hcell = cv2.getPerspectiveTransform(\r\n quad.astype(np.float32), objP)\r\n cv2.warpPerspective(self.img, hcell, shape, sub,\r\n flags=cv2.INTER_LANCZOS4,\r\n **self.opts['cv2_opts'])\r\n\r\n# return out\r\n # TOP CELLS\r\n objP[:, 1] += b\r\n for ix in range(1, n0 - 1):\r\n warp(ix, 0, objP, out[: sy + b,\r\n ix * sx + b: (ix + 1) * sx + b])\r\n # BOTTOM CELLS\r\n objP[:, 1] -= b\r\n for ix in range(1, n0 - 1):\r\n iy = (n1 - 1)\r\n y = iy * sy + b\r\n x = ix * sx + b\r\n warp(ix, iy, objP, out[y: y + sy + b, x: x + sx])\r\n # LEFT CELLS\r\n objP[:, 0] += b\r\n for iy in range(1, n1 - 1):\r\n y = iy * sy + b\r\n warp(0, iy, objP, out[y: y + sy, : sx + b])\r\n # RIGHT CELLS\r\n objP[:, 0] -= b\r\n ix = (n0 - 1)\r\n x = ix * sx + b\r\n for iy in range(1, n1 - 1):\r\n y = iy * sy + b\r\n warp(ix, iy, objP, out[y: y + sy, x: x + sx + b])\r\n # BOTTOM RIGHT CORNER\r\n warp(n0 - 1, n1 - 1, objP, out[-sy - b - 1:, x: x + sx + b])\r\n# #TOP LEFT CORNER\r\n objP += (b, b)\r\n warp(0, 0, objP, out[0: sy + b, 0: sx + b])\r\n # TOP RIGHT CORNER\r\n objP[:, 0] -= b\r\n# x = (n0-1)*sx+b\r\n warp(n0 - 1, 0, objP, out[: sy + b, x: x + sx + b])\r\n# #BOTTOM LEFT CORNER\r\n objP += (b, -b)\r\n warp(0, n1 - 1, objP, out[-sy - b - 1:, : sx + b])\r\n return out", "response": "Returns a new image with the new image and the new border."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncorrect the image using the perspective distortion transformation and tilt factor.", "response": "def correct(self, img):\r\n '''\r\n ...from perspective distortion:\r\n --> perspective transformation\r\n --> apply tilt factor (view factor) correction \r\n '''\r\n print(\"CORRECT PERSPECTIVE ...\")\r\n self.img = imread(img)\r\n\r\n if not self._homography_is_fixed:\r\n self._homography = None\r\n h = self.homography\r\n\r\n if self.opts['do_correctIntensity']:\r\n tf = self.tiltFactor()\r\n self.img = np.asfarray(self.img)\r\n if self.img.ndim == 3:\r\n for col in range(self.img.shape[2]):\r\n self.img[..., col] /= tf\r\n else:\r\n self.img = self.img / tf\r\n warped = cv2.warpPerspective(self.img,\r\n h,\r\n self._newBorders[::-1],\r\n flags=cv2.INTER_LANCZOS4,\r\n **self.opts['cv2_opts'])\r\n return warped"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef camera_position(self, pose=None):\r\n '''\r\n returns camera position in world coordinates using self.rvec and self.tvec\r\n from http://stackoverflow.com/questions/14515200/python-opencv-solvepnp-yields-wrong-translation-vector\r\n '''\r\n if pose is None:\r\n pose = self.pose()\r\n t, r = pose\r\n return -np.matrix(cv2.Rodrigues(r)[0]).T * np.matrix(t)", "response": "Returns the camera position in world coordinates using self. rvec and self. tvec"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef viewAngle(self, **kwargs):\r\n '''\r\n calculate view factor between one small and one finite surface\r\n vf =1/pi * integral(cos(beta1)*cos(beta2)/s**2) * dA\r\n according to VDI heatatlas 2010 p961\r\n '''\r\n v0 = self.cam2PlaneVectorField(**kwargs)\r\n # obj cannot be behind camera\r\n v0[2][v0[2] < 0] = np.nan\r\n\r\n _t, r = self.pose()\r\n n = self.planeSfN(r)\r\n # because of different x,y orientation:\r\n n[2] *= -1\r\n# beta2 = vectorAngle(v0, vectorToField(n) )\r\n beta2 = vectorAngle(v0, n)\r\n return beta2", "response": "calculate view factor between one small and one finite surface\r\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn foreground (quad) mask", "response": "def foreground(self, quad=None):\r\n '''return foreground (quad) mask'''\r\n fg = np.zeros(shape=self._newBorders[::-1], dtype=np.uint8)\r\n if quad is None:\r\n quad = self.quad\r\n else:\r\n quad = quad.astype(np.int32)\r\n cv2.fillConvexPoly(fg, quad, 1)\r\n return fg.astype(bool)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tiltFactor(self, midpointdepth=None,\r\n printAvAngle=False):\r\n '''\r\n get tilt factor from inverse distance law\r\n https://en.wikipedia.org/wiki/Inverse-square_law\r\n '''\r\n # TODO: can also be only def. with FOV, rot, tilt\r\n beta2 = self.viewAngle(midpointdepth=midpointdepth)\r\n try:\r\n angles, vals = getattr(\r\n emissivity_vs_angle, self.opts['material'])()\r\n except AttributeError:\r\n raise AttributeError(\"material[%s] is not in list of know materials: %s\" % (\r\n self.opts['material'], [o[0] for o in getmembers(emissivity_vs_angle)\r\n if isfunction(o[1])]))\r\n if printAvAngle:\r\n avg_angle = beta2[self.foreground()].mean()\r\n print('angle: %s DEG' % np.degrees(avg_angle))\r\n\r\n # use averaged angle instead of beta2 to not overemphasize correction\r\n normEmissivity = np.clip(\r\n InterpolatedUnivariateSpline(\r\n np.radians(angles), vals)(beta2), 0, 1)\r\n return normEmissivity", "response": "get tilt factor from inverse distance law\r\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef standardUncertainties(self, focal_Length_mm, f_number, midpointdepth=1000,\r\n focusAtYX=None,\r\n # sigma_best_focus=0,\r\n # quad_pos_err=0,\r\n shape=None,\r\n uncertainties=(0, 0)):\r\n '''\r\n focusAtXY - image position with is in focus\r\n if not set it is assumed that the image middle is in focus\r\n sigma_best_focus - standard deviation of the PSF\r\n within the best focus (default blur)\r\n uncertainties - contibutors for standard uncertainty\r\n these need to be perspective transformed to fit the new \r\n image shape\r\n '''\r\n # TODO: consider quad_pos_error\r\n # (also influences intensity corr map)\r\n\r\n if shape is None:\r\n s = self.img.shape\r\n else:\r\n s = shape\r\n\r\n # 1. DEFOCUS DUE TO DEPTH OF FIELD\r\n ##################################\r\n depthMap = self.depthMap(midpointdepth)\r\n if focusAtYX is None:\r\n # assume image middle is in-focus:\r\n focusAtYX = s[0] // 2, s[1] // 2\r\n infocusDepth = depthMap[focusAtYX]\r\n depthOfField_blur = defocusThroughDepth(\r\n depthMap, infocusDepth, focal_Length_mm, f_number, k=2.335)\r\n\r\n # 2. INCREAASED PIXEL SIZE DUE TO INTERPOLATION BETWEEN\r\n # PIXELS MOVED APARD\r\n ######################################################\r\n # index maps:\r\n py, px = np.mgrid[0:s[0], 0:s[1]]\r\n # warped index maps:\r\n wx = cv2.warpPerspective(np.asfarray(px), self.homography,\r\n self._newBorders,\r\n borderValue=np.nan,\r\n flags=cv2.INTER_LANCZOS4)\r\n wy = cv2.warpPerspective(np.asfarray(py), self.homography,\r\n self._newBorders,\r\n borderValue=np.nan,\r\n flags=cv2.INTER_LANCZOS4)\r\n\r\n pxSizeFactorX = 1 / np.abs(np.gradient(wx)[1])\r\n pxSizeFactorY = 1 / np.abs(np.gradient(wy)[0])\r\n\r\n # WARP ALL FIELD TO NEW PERSPECTIVE AND MULTIPLY WITH PXSIZE FACTOR:\r\n depthOfField_blur = cv2.warpPerspective(\r\n depthOfField_blur, self.homography, self._newBorders,\r\n borderValue=np.nan,\r\n )\r\n\r\n # perspective transform given uncertainties:\r\n warpedU = []\r\n for u in uncertainties:\r\n # warpedU.append([])\r\n # for i in u:\r\n # print i, type(i), isinstance(i, np.ndarray)\r\n if isinstance(u, np.ndarray) and u.size > 1:\r\n u = cv2.warpPerspective(u, self.homography,\r\n self._newBorders,\r\n borderValue=np.nan,\r\n flags=cv2.INTER_LANCZOS4) # *f\r\n\r\n else:\r\n # multiply with area ratio: after/before perspective warp\r\n u *= self.areaRatio\r\n\r\n warpedU.append(u)\r\n\r\n # given uncertainties after warp:\r\n ux, uy = warpedU\r\n\r\n ux = pxSizeFactorX * (ux**2 + depthOfField_blur**2)**0.5\r\n uy = pxSizeFactorY * (uy**2 + depthOfField_blur**2)**0.5\r\n\r\n # TODO: remove depthOfField_blur,fx,fy from return\r\n return ux, uy, depthOfField_blur, pxSizeFactorX, pxSizeFactorY", "response": "Returns a new image with the standard uncertainty."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nestimating the pose of the object plane using quad", "response": "def _poseFromQuad(self, quad=None):\r\n '''\r\n estimate the pose of the object plane using quad\r\n setting:\r\n self.rvec -> rotation vector\r\n self.tvec -> translation vector\r\n '''\r\n if quad is None:\r\n quad = self.quad\r\n if quad.ndim == 3:\r\n quad = quad[0]\r\n # http://answers.opencv.org/question/1073/what-format-does-cv2solvepnp-use-for-points-in/\r\n # Find the rotation and translation vectors.\r\n img_pn = np.ascontiguousarray(quad[:, :2],\r\n dtype=np.float32).reshape((4, 1, 2))\r\n\r\n obj_pn = self.obj_points - self.obj_points.mean(axis=0)\r\n retval, rvec, tvec = cv2.solvePnP(\r\n obj_pn,\r\n img_pn,\r\n self.opts['cameraMatrix'],\r\n self.opts['distCoeffs'],\r\n flags=cv2.SOLVEPNP_P3P # because exactly four points are given\r\n )\r\n if not retval:\r\n print(\"Couln't estimate pose\")\r\n return tvec, rvec"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndraw the quad into given img", "response": "def drawQuad(self, img=None, quad=None, thickness=30):\r\n '''\r\n Draw the quad into given img \r\n '''\r\n if img is None:\r\n img = self.img\r\n if quad is None:\r\n quad = self.quad\r\n q = np.int32(quad)\r\n c = int(img.max())\r\n cv2.line(img, tuple(q[0]), tuple(q[1]), c, thickness)\r\n cv2.line(img, tuple(q[1]), tuple(q[2]), c, thickness)\r\n cv2.line(img, tuple(q[2]), tuple(q[3]), c, thickness)\r\n cv2.line(img, tuple(q[3]), tuple(q[0]), c, thickness)\r\n return img"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndraws the 3d coordinate axes into a given image", "response": "def draw3dCoordAxis(self, img=None, thickness=8):\r\n '''\r\n draw the 3d coordinate axes into given image\r\n if image == False:\r\n create an empty image\r\n '''\r\n if img is None:\r\n img = self.img\r\n elif img is False:\r\n img = np.zeros(shape=self.img.shape, dtype=self.img.dtype)\r\n else:\r\n img = imread(img)\r\n # project 3D points to image plane:\r\n # self.opts['obj_width_mm'], self.opts['obj_height_mm']\r\n w, h = self.opts['new_size']\r\n axis = np.float32([[0.5 * w, 0.5 * h, 0],\r\n [w, 0.5 * h, 0],\r\n [0.5 * w, h, 0],\r\n [0.5 * w, 0.5 * h, -0.5 * w]])\r\n t, r = self.pose()\r\n imgpts = cv2.projectPoints(axis, r, t,\r\n self.opts['cameraMatrix'],\r\n self.opts['distCoeffs'])[0]\r\n\r\n mx = int(img.max())\r\n origin = tuple(imgpts[0].ravel())\r\n cv2.line(img, origin, tuple(imgpts[1].ravel()), (0, 0, mx), thickness)\r\n cv2.line(img, origin, tuple(imgpts[2].ravel()), (0, mx, 0), thickness)\r\n cv2.line(\r\n img, origin, tuple(imgpts[3].ravel()), (mx, 0, 0), thickness * 2)\r\n return img"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the size of a rectangle in perspective distortion in [ px ].", "response": "def _calcQuadSize(corners, aspectRatio):\r\n '''\r\n return the size of a rectangle in perspective distortion in [px]\r\n DEBUG: PUT THAT BACK IN??::\r\n if aspectRatio is not given is will be determined\r\n '''\r\n if aspectRatio > 1: # x is bigger -> reduce y\r\n x_length = PerspectiveCorrection._quadXLength(corners)\r\n y = x_length / aspectRatio\r\n return x_length, y\r\n else: # y is bigger -> reduce x\r\n y_length = PerspectiveCorrection._quadYLength(corners)\r\n x = y_length * aspectRatio\r\n return x, y_length"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef linearToPolar(img, center=None,\r\n final_radius=None,\r\n initial_radius=None,\r\n phase_width=None,\r\n interpolation=cv2.INTER_AREA, maps=None,\r\n borderValue=0, borderMode=cv2.BORDER_REFLECT, **opts):\r\n '''\r\n map a 2d (x,y) Cartesian array to a polar (r, phi) array\r\n using opencv.remap\r\n '''\r\n if maps is None:\r\n mapY, mapX = linearToPolarMaps(img.shape[:2], center, final_radius,\r\n initial_radius, phase_width)\r\n else:\r\n mapY, mapX = maps\r\n\r\n o = {'interpolation': interpolation,\r\n 'borderValue': borderValue,\r\n 'borderMode': borderMode}\r\n o.update(opts)\r\n\r\n return cv2.remap(img, mapY, mapX, **o)", "response": "Returns a 2d array of polar coordinates using opencv. remap"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef polarToLinear(img, shape=None, center=None, maps=None,\r\n interpolation=cv2.INTER_AREA,\r\n borderValue=0, borderMode=cv2.BORDER_REFLECT, **opts):\r\n '''\r\n map a 2d polar (r, phi) polar array to a Cartesian (x,y) array\r\n using opencv.remap\r\n '''\r\n\r\n if maps is None:\r\n mapY, mapX = polarToLinearMaps(img.shape[:2], shape, center)\r\n else:\r\n mapY, mapX = maps\r\n\r\n o = {'interpolation': interpolation,\r\n 'borderValue': borderValue,\r\n 'borderMode': borderMode}\r\n o.update(opts)\r\n\r\n return cv2.remap(img, mapY, mapX, **o)", "response": "Returns a 2d array of polar coordinates with a linear interpolation"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef varianceOfLaplacian(img):\r\n ''''LAPV' algorithm (Pech2000)'''\r\n lap = cv2.Laplacian(img, ddepth=-1)#cv2.cv.CV_64F)\r\n stdev = cv2.meanStdDev(lap)[1]\r\n s = stdev[0]**2\r\n return s[0]", "response": "Calculates the variance of Laplacian."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the normalized graylevel variance of the image.", "response": "def normalizedGraylevelVariance(img):\r\n ''''GLVN' algorithm (Santos97)'''\r\n mean, stdev = cv2.meanStdDev(img)\r\n s = stdev[0]**2 / mean[0]\r\n return s[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef linePlot(img, x0, y0, x1, y1, resolution=None, order=3):\r\n '''\r\n returns [img] intensity values along line\r\n defined by [x0, y0, x1, y1]\r\n \r\n resolution ... number or data points to evaluate\r\n order ... interpolation precision\r\n '''\r\n if resolution is None:\r\n resolution = int( ((x1-x0)**2 + (y1-y0)**2 )**0.5 )\r\n \r\n if order == 0:\r\n x = np.linspace(x0, x1, resolution, dtype=int)\r\n y = np.linspace(y0, y1, resolution, dtype=int)\r\n return img[y, x]\r\n\r\n x = np.linspace(x0, x1, resolution)\r\n y = np.linspace(y0, y1, resolution)\r\n return map_coordinates(img, np.vstack((y,x)), order=order)", "response": "Returns a line plot of the image along a given line defined by [ x0 y0 x1 y1."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flatFieldFromFunction(self):\r\n '''\r\n calculate flatField from fitting vignetting function to averaged fit-image\r\n returns flatField, average background level, fitted image, valid indices mask\r\n '''\r\n fitimg, mask = self._prepare()\r\n mask = ~mask\r\n\r\n s0, s1 = fitimg.shape\r\n #f-value, alpha, fx, cx, cy\r\n guess = (s1 * 0.7, 0, 1, s0 / 2, s1 / 2)\r\n\r\n # set assume normal plane - no tilt and rotation:\r\n fn = lambda xy, f, alpha, fx, cx, cy: vignetting((xy[0] * fx, xy[1]), f, alpha,\r\n cx=cx, cy=cy)\r\n\r\n# mask = fitimg>0.5\r\n\r\n flatfield = fit2dArrayToFn(fitimg, fn, mask=mask,\r\n guess=guess, output_shape=self._orig_shape)[0]\r\n\r\n return flatfield, self.bglevel / self._n, fitimg, mask", "response": "calculate flatField from fitting vignetting function to averaged fit - image returns flatField average background level fitted image valid indices mask"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating flatField from 2d - polaral fit filling all high gradient areas within averaged fit - image returns flatField average background level fitted image valid indices mask", "response": "def flatFieldFromFit(self):\r\n '''\r\n calculate flatField from 2d-polynomal fit filling\r\n all high gradient areas within averaged fit-image\r\n\r\n returns flatField, average background level, fitted image, valid indices mask\r\n '''\r\n fitimg, mask = self._prepare()\r\n\r\n out = fitimg.copy()\r\n lastm = 0\r\n\r\n for _ in range(10):\r\n out = polyfit2dGrid(out, mask, 2)\r\n mask = highGrad(out)\r\n m = mask.sum()\r\n if m == lastm:\r\n break\r\n lastm = m\r\n\r\n out = np.clip(out, 0.1, 1)\r\n\r\n out = resize(out, self._orig_shape, mode='reflect')\r\n return out, self.bglevel / self._n, fitimg, mask"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_vhdl_file(fname):\n '''Parse a named VHDL file\n \n Args:\n fname(str): Name of file to parse\n Returns:\n Parsed objects.\n '''\n with open(fname, 'rt') as fh:\n text = fh.read()\n return parse_vhdl(text)", "response": "Parse a named VHDL file\n \n"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_vhdl(text):\n '''Parse a text buffer of VHDL code\n\n Args:\n text(str): Source code to parse\n Returns:\n Parsed objects.\n '''\n lex = VhdlLexer\n \n name = None\n kind = None\n saved_type = None\n end_param_group = False\n cur_package = None\n\n metacomments = []\n parameters = []\n param_items = []\n\n generics = []\n ports = []\n sections = []\n port_param_index = 0\n last_item = None\n array_range_start_pos = 0\n\n objects = []\n \n for pos, action, groups in lex.run(text):\n if action == 'metacomment':\n realigned = re.sub(r'^#+', lambda m: ' ' * len(m.group(0)), groups[0])\n if last_item is None:\n metacomments.append(realigned)\n else:\n last_item.desc = realigned\n if action == 'section_meta':\n sections.append((port_param_index, groups[0]))\n\n elif action == 'function':\n kind = 'function'\n name = groups[0]\n param_items = []\n parameters = []\n elif action == 'procedure':\n kind = 'procedure'\n name = groups[0]\n param_items = []\n parameters = []\n elif action == 'param':\n if end_param_group:\n # Complete previous parameters\n for i in param_items:\n parameters.append(i)\n param_items = []\n end_param_group = False\n\n param_items.append(VhdlParameter(groups[1]))\n elif action == 'param_type':\n mode, ptype = groups\n \n if mode is not None:\n mode = mode.strip()\n \n for i in param_items: # Set mode and type for all pending parameters\n i.mode = mode\n i.data_type = ptype\n\n end_param_group = True\n\n elif action == 'param_default':\n for i in param_items:\n i.default_value = groups[0]\n\n elif action == 'end_subprogram':\n # Complete last parameters\n for i in param_items:\n parameters.append(i)\n \n if kind == 'function':\n vobj = VhdlFunction(name, cur_package, parameters, groups[0], metacomments)\n else:\n vobj = VhdlProcedure(name, cur_package, parameters, metacomments)\n \n objects.append(vobj)\n \n metacomments = []\n parameters = []\n param_items = []\n kind = None\n name = None\n\n elif action == 'component':\n kind = 'component'\n name = groups[0]\n generics = []\n ports = []\n param_items = []\n sections = []\n port_param_index = 0\n\n elif action == 'generic_param':\n param_items.append(groups[0])\n\n elif action == 'generic_param_type':\n ptype = groups[0]\n \n for i in param_items:\n generics.append(VhdlParameter(i, 'in', ptype))\n param_items = []\n last_item = generics[-1]\n\n elif action == 'port_param':\n param_items.append(groups[0])\n port_param_index += 1\n\n elif action == 'port_param_type':\n mode, ptype = groups\n\n for i in param_items:\n ports.append(VhdlParameter(i, mode, ptype))\n\n param_items = []\n last_item = ports[-1]\n\n elif action == 'port_array_param_type':\n mode, ptype = groups\n array_range_start_pos = pos[1]\n\n elif action == 'array_range_end':\n arange = text[array_range_start_pos:pos[0]+1]\n\n for i in param_items:\n ports.append(VhdlParameter(i, mode, ptype + arange))\n\n param_items = []\n last_item = ports[-1]\n\n elif action == 'end_component':\n vobj = VhdlComponent(name, cur_package, ports, generics, dict(sections), metacomments)\n objects.append(vobj)\n last_item = None\n metacomments = []\n\n elif action == 'package':\n objects.append(VhdlPackage(groups[0]))\n cur_package = groups[0]\n kind = None\n name = None\n\n elif action == 'type':\n saved_type = groups[0]\n\n elif action in ('array_type', 'file_type', 'access_type', 'record_type', 'range_type', 'enum_type', 'incomplete_type'):\n vobj = VhdlType(saved_type, cur_package, action, metacomments)\n objects.append(vobj)\n kind = None\n name = None\n metacomments = []\n\n elif action == 'subtype':\n vobj = VhdlSubtype(groups[0], cur_package, groups[1], metacomments)\n objects.append(vobj)\n kind = None\n name = None\n metacomments = []\n\n elif action == 'constant':\n vobj = VhdlConstant(groups[0], cur_package, groups[1], metacomments)\n objects.append(vobj)\n kind = None\n name = None\n metacomments = []\n\n return objects", "response": "Parse a text buffer of VHDL code into a list of Parsed objects."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef subprogram_prototype(vo):\n '''Generate a canonical prototype string\n \n Args:\n vo (VhdlFunction, VhdlProcedure): Subprogram object\n Returns:\n Prototype string.\n '''\n\n plist = '; '.join(str(p) for p in vo.parameters)\n\n if isinstance(vo, VhdlFunction):\n if len(vo.parameters) > 0:\n proto = 'function {}({}) return {};'.format(vo.name, plist, vo.return_type)\n else:\n proto = 'function {} return {};'.format(vo.name, vo.return_type)\n\n else: # procedure\n proto = 'procedure {}({});'.format(vo.name, plist)\n\n return proto", "response": "Generate a canonical prototype string for a sequence of subprogram objects."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a signature string for a sequence of subprogram objects.", "response": "def subprogram_signature(vo, fullname=None):\n '''Generate a signature string\n \n Args:\n vo (VhdlFunction, VhdlProcedure): Subprogram object\n Returns:\n Signature string.\n '''\n\n if fullname is None:\n fullname = vo.name\n\n if isinstance(vo, VhdlFunction):\n plist = ','.join(p.data_type for p in vo.parameters)\n sig = '{}[{} return {}]'.format(fullname, plist, vo.return_type)\n else: # procedure\n plist = ','.join(p.data_type for p in vo.parameters)\n sig = '{}[{}]'.format(fullname, plist)\n\n return sig"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nextracts object declarations from a text buffer containing a single object.", "response": "def extract_objects_from_source(self, text, type_filter=None):\n '''Extract object declarations from a text buffer\n\n Args:\n text (str): Source code to parse\n type_filter (class, optional): Object class to filter results\n Returns:\n List of parsed objects.\n '''\n objects = parse_vhdl(text)\n self._register_array_types(objects)\n\n if type_filter:\n objects = [o for o in objects if isinstance(o, type_filter)]\n\n return objects"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_array(self, data_type):\n '''Check if a type is a known array type\n \n Args:\n data_type (str): Name of type to check\n Returns:\n True if ``data_type`` is a known array type.\n '''\n\n # Split off any brackets\n data_type = data_type.split('[')[0].strip()\n\n return data_type.lower() in self.array_types", "response": "Check if a type is a known array type."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads file of previously extracted data types", "response": "def load_array_types(self, fname):\n '''Load file of previously extracted data types\n \n Args:\n fname (str): Name of file to load array database from\n '''\n type_defs = ''\n with open(fname, 'rt') as fh:\n type_defs = fh.read()\n\n try:\n type_defs = ast.literal_eval(type_defs)\n except SyntaxError:\n type_defs = {}\n\n self._add_array_types(type_defs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving array type registry to a file", "response": "def save_array_types(self, fname):\n '''Save array type registry to a file\n \n Args:\n fname (str): Name of file to save array database to\n '''\n type_defs = {'arrays': sorted(list(self.array_types))}\n with open(fname, 'wt') as fh:\n pprint(type_defs, stream=fh)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _register_array_types(self, objects):\n '''Add array type definitions to internal registry\n \n Args:\n objects (list of VhdlType or VhdlSubtype): Array types to track\n '''\n # Add all array types directly\n types = [o for o in objects if isinstance(o, VhdlType) and o.type_of == 'array_type']\n for t in types:\n self.array_types.add(t.name)\n\n subtypes = {o.name:o.base_type for o in objects if isinstance(o, VhdlSubtype)}\n\n # Find all subtypes of an array type\n for k,v in subtypes.iteritems():\n while v in subtypes: # Follow subtypes of subtypes\n v = subtypes[v]\n if v in self.array_types:\n self.array_types.add(k)", "response": "Add array type definitions to internal registry\n \n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd array type definitions from a list of source files to internal registry", "response": "def register_array_types_from_sources(self, source_files):\n '''Add array type definitions from a file list to internal registry\n\n Args:\n source_files (list of str): Files to parse for array definitions\n '''\n for fname in source_files:\n if is_vhdl(fname):\n self._register_array_types(self.extract_objects(fname))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun lexer rules against a source text.", "response": "def run(self, text):\n '''Run lexer rules against a source text\n\n Args:\n text (str): Text to apply lexer to\n\n Yields:\n A sequence of lexer matches.\n '''\n\n stack = ['root']\n pos = 0\n\n patterns = self.tokens[stack[-1]]\n\n while True:\n for pat, action, new_state in patterns:\n m = pat.match(text, pos)\n if m:\n if action:\n #print('## MATCH: {} -> {}'.format(m.group(), action))\n yield (pos, m.end()-1), action, m.groups()\n\n pos = m.end()\n\n if new_state:\n if isinstance(new_state, int): # Pop states\n del stack[new_state:]\n else:\n stack.append(new_state)\n\n #print('## CHANGE STATE:', pos, new_state, stack)\n patterns = self.tokens[stack[-1]]\n\n break\n\n else:\n try:\n if text[pos] == '\\n':\n pos += 1\n continue\n pos += 1\n except IndexError:\n break"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_package_version(verfile):\n '''Scan the script for the version string'''\n version = None\n with open(verfile) as fh:\n try:\n version = [line.split('=')[1].strip().strip(\"'\") for line in fh if \\\n line.startswith('__version__')][0]\n except IndexError:\n pass\n return version", "response": "Scan the script for the version string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse a named Verilog file", "response": "def parse_verilog_file(fname):\n '''Parse a named Verilog file\n \n Args:\n fname (str): File to parse.\n Returns:\n List of parsed objects.\n '''\n with open(fname, 'rt') as fh:\n text = fh.read()\n return parse_verilog(text)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing a text buffer of Verilog code into a list of objects.", "response": "def parse_verilog(text):\n '''Parse a text buffer of Verilog code\n\n Args:\n text (str): Source code to parse\n Returns:\n List of parsed objects.\n '''\n lex = VerilogLexer\n\n name = None\n kind = None\n saved_type = None\n mode = 'input'\n ptype = 'wire'\n\n metacomments = []\n parameters = []\n param_items = []\n\n generics = []\n ports = collections.OrderedDict()\n sections = []\n port_param_index = 0\n last_item = None\n array_range_start_pos = 0\n\n objects = []\n\n for pos, action, groups in lex.run(text):\n if action == 'metacomment':\n if last_item is None:\n metacomments.append(groups[0])\n else:\n last_item.desc = groups[0]\n\n if action == 'section_meta':\n sections.append((port_param_index, groups[0]))\n\n elif action == 'module':\n kind = 'module'\n name = groups[0]\n generics = []\n ports = collections.OrderedDict()\n param_items = []\n sections = []\n port_param_index = 0\n\n elif action == 'parameter_start':\n net_type, vec_range = groups\n\n new_ptype = ''\n if net_type is not None:\n new_ptype += net_type\n\n if vec_range is not None:\n new_ptype += ' ' + vec_range\n\n ptype = new_ptype\n\n elif action == 'param_item':\n generics.append(VerilogParameter(groups[0], 'in', ptype))\n\n elif action == 'module_port_start':\n new_mode, net_type, signed, vec_range = groups\n\n new_ptype = ''\n if net_type is not None:\n new_ptype += net_type\n\n if signed is not None:\n new_ptype += ' ' + signed\n\n if vec_range is not None:\n new_ptype += ' ' + vec_range\n\n # Complete pending items\n for i in param_items:\n ports[i] = VerilogParameter(i, mode, ptype)\n\n param_items = []\n if len(ports) > 0:\n last_item = next(reversed(ports))\n\n # Start with new mode\n mode = new_mode\n ptype = new_ptype\n\n elif action == 'port_param':\n ident = groups[0]\n\n param_items.append(ident)\n port_param_index += 1\n\n elif action == 'end_module':\n # Finish any pending ports\n for i in param_items:\n ports[i] = VerilogParameter(i, mode, ptype)\n\n vobj = VerilogModule(name, ports.values(), generics, dict(sections), metacomments)\n objects.append(vobj)\n last_item = None\n metacomments = []\n\n return objects"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nextract objects from a source file", "response": "def extract_objects(self, fname, type_filter=None):\n '''Extract objects from a source file\n\n Args:\n fname(str): Name of file to read from\n type_filter (class, optional): Object class to filter results\n Returns:\n List of objects extracted from the file.\n '''\n objects = []\n if fname in self.object_cache:\n objects = self.object_cache[fname]\n else:\n with io.open(fname, 'rt', encoding='utf-8') as fh:\n text = fh.read()\n objects = parse_verilog(text)\n self.object_cache[fname] = objects\n\n if type_filter:\n objects = [o for o in objects if isinstance(o, type_filter)]\n\n return objects"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef extract_objects_from_source(self, text, type_filter=None):\n '''Extract object declarations from a text buffer\n\n Args:\n text (str): Source code to parse\n type_filter (class, optional): Object class to filter results\n Returns:\n List of parsed objects.\n '''\n objects = parse_verilog(text)\n\n if type_filter:\n objects = [o for o in objects if isinstance(o, type_filter)]\n\n return objects", "response": "Extract object declarations from a text buffer containing a verilog file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload schema from a JSON file", "response": "def load_json_from_file(file_path):\n \"\"\"Load schema from a JSON file\"\"\"\n try:\n with open(file_path) as f:\n json_data = json.load(f)\n except ValueError as e:\n raise ValueError('Given file {} is not a valid JSON file: {}'.format(file_path, e))\n else:\n return json_data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload schema from JSON string", "response": "def load_json_from_string(string):\n \"\"\"Load schema from JSON string\"\"\"\n try:\n json_data = json.loads(string)\n except ValueError as e:\n raise ValueError('Given string is not valid JSON: {}'.format(e))\n else:\n return json_data"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfetches the given uri and return the contents of the response.", "response": "def fetch(method, uri, params_prefix=None, **params):\n \"\"\"Fetch the given uri and return the contents of the response.\"\"\"\n params = urlencode(_prepare_params(params, params_prefix))\n binary_params = params.encode('ASCII')\n\n # build the HTTP request\n url = \"https://%s/%s.xml\" % (CHALLONGE_API_URL, uri)\n req = Request(url, binary_params)\n req.get_method = lambda: method\n\n # use basic authentication\n user, api_key = get_credentials()\n auth_handler = HTTPBasicAuthHandler()\n auth_handler.add_password(\n realm=\"Application\",\n uri=req.get_full_url(),\n user=user,\n passwd=api_key\n )\n opener = build_opener(auth_handler)\n\n try:\n response = opener.open(req)\n except HTTPError as e:\n if e.code != 422:\n raise\n # wrap up application-level errors\n doc = ElementTree.parse(e).getroot()\n if doc.tag != \"errors\":\n raise\n errors = [e.text for e in doc]\n raise ChallongeException(*errors)\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching the given uri and return the root Element of the response.", "response": "def fetch_and_parse(method, uri, params_prefix=None, **params):\n \"\"\"Fetch the given uri and return the root Element of the response.\"\"\"\n doc = ElementTree.parse(fetch(method, uri, params_prefix, **params))\n return _parse(doc.getroot())"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse(root):\n if root.tag == \"nil-classes\":\n return []\n elif root.get(\"type\") == \"array\":\n return [_parse(child) for child in root]\n\n d = {}\n for child in root:\n type = child.get(\"type\") or \"string\"\n\n if child.get(\"nil\"):\n value = None\n elif type == \"boolean\":\n value = True if child.text.lower() == \"true\" else False\n elif type == \"dateTime\":\n value = iso8601.parse_date(child.text)\n elif type == \"decimal\":\n value = decimal.Decimal(child.text)\n elif type == \"integer\":\n value = int(child.text)\n else:\n value = child.text\n\n d[child.tag] = value\n return d", "response": "Recursively convert an Element into python data types"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _prepare_params(dirty_params, prefix=None):\n params = {}\n for k, v in dirty_params.items():\n if hasattr(v, \"isoformat\"):\n v = v.isoformat()\n elif isinstance(v, bool):\n # challonge.com only accepts lowercase true/false\n v = str(v).lower()\n\n if prefix:\n params[\"%s[%s]\" % (prefix, k)] = v\n else:\n params[k] = v\n\n return params", "response": "Prepares parameters to be sent to challonge. com."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexpand descendants from list of branches as TreeOfContents objs", "response": "def expandDescendants(self, branches):\n \"\"\"\n Expand descendants from list of branches\n\n :param list branches: list of immediate children as TreeOfContents objs\n :return: list of all descendants\n \"\"\"\n return sum([b.descendants() for b in branches], []) + \\\n [b.source for b in branches]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing top level of markdown tree of contents.", "response": "def parseBranches(self, descendants):\n \"\"\"\n Parse top level of markdown\n\n :param list elements: list of source objects\n :return: list of filtered TreeOfContents objects\n \"\"\"\n parsed, parent, cond = [], False, lambda b: (b.string or '').strip()\n for branch in filter(cond, descendants):\n if self.getHeadingLevel(branch) == self.depth:\n parsed.append({'root':branch.string, 'source':branch})\n parent = True\n elif not parent:\n parsed.append({'root':branch.string, 'source':branch})\n else:\n parsed[-1].setdefault('descendants', []).append(branch)\n return [TOC(depth=self.depth+1, **kwargs) for kwargs in parsed]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a TreeOfContents object from a markdown string.", "response": "def fromMarkdown(md, *args, **kwargs):\n \"\"\"\n Creates abstraction using path to file\n\n :param str path: path to markdown file\n :return: TreeOfContents object\n \"\"\"\n return TOC.fromHTML(markdown(md, *args, **kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fromHTML(html, *args, **kwargs):\n source = BeautifulSoup(html, 'html.parser', *args, **kwargs)\n return TOC('[document]',\n source=source,\n descendants=source.children)", "response": "Creates a TreeOfContents object from a HTML string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an attachment from data.", "response": "def from_data(cls, type, **data):\n \"\"\"Create an attachment from data.\n\n :param str type: attachment type\n :param kwargs data: additional attachment data\n :return: an attachment subclass object\n :rtype: `~groupy.api.attachments.Attachment`\n \"\"\"\n try:\n return cls._types[type](**data)\n except KeyError:\n return cls(type=type, **data)\n except TypeError as e:\n error = 'could not create {!r} attachment'.format(type)\n raise TypeError('{}: {}'.format(error, e.args[0]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new image attachment from an image file.", "response": "def from_file(self, fp):\n \"\"\"Create a new image attachment from an image file.\n\n :param file fp: a file object containing binary image data\n :return: an image attachment\n :rtype: :class:`~groupy.api.attachments.Image`\n \"\"\"\n image_urls = self.upload(fp)\n return Image(image_urls['url'], source_url=image_urls['picture_url'])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef upload(self, fp):\n url = utils.urljoin(self.url, 'pictures')\n response = self.session.post(url, data=fp.read())\n image_urls = response.data\n return image_urls", "response": "Upload image data to the image service."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef download(self, image, url_field='url', suffix=None):\n url = getattr(image, url_field)\n if suffix is not None:\n url = '.'.join(url, suffix)\n response = self.session.get(url)\n return response.content", "response": "Download the binary data of an image attachment."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef download_preview(self, image, url_field='url'):\n return self.download(image, url_field=url_field, suffix='preview')", "response": "Downloads the binary data of an image attachment at preview size."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef download_large(self, image, url_field='url'):\n return self.download(image, url_field=url_field, suffix='large')", "response": "Downloads the binary data of an image attachment at large size."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndownloads the binary data of an image attachment at avatar size.", "response": "def download_avatar(self, image, url_field='url'):\n \"\"\"Downlaod the binary data of an image attachment at avatar size.\n\n :param str url_field: the field of the image with the right URL\n :return: binary image data\n :rtype: bytes\n\n \"\"\"\n return self.download(image, url_field=url_field, suffix='avatar')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef autopage(self):\n while self.items:\n yield from self.items\n self.items = self.fetch_next()", "response": "Iterate through results from all pages."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetect which listing mode of the given params.", "response": "def detect_mode(cls, **params):\n \"\"\"Detect which listing mode of the given params.\n\n :params kwargs params: the params\n :return: one of the available modes\n :rtype: str\n :raises ValueError: if multiple modes are detected\n \"\"\"\n modes = []\n for mode in cls.modes:\n if params.get(mode) is not None:\n modes.append(mode)\n if len(modes) > 1:\n error_message = 'ambiguous mode, must be one of {}'\n modes_csv = ', '.join(list(cls.modes))\n raise ValueError(error_message.format(modes_csv))\n return modes[0] if modes else cls.default_mode"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_next_page_params(self):\n if self.items:\n index = self.get_last_item_index()\n self.params[self.mode] = self.get_next_page_param(self.items[index])", "response": "Set the params so that the next page is fetched."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list(self):\n params = {'user': self.user_id}\n response = self.session.get(self.url, params=params)\n blocks = response.data['blocks']\n return [Block(self, **block) for block in blocks]", "response": "List the users you have blocked."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if there is a block between you and the given user.", "response": "def between(self, other_user_id):\n \"\"\"Check if there is a block between you and the given user.\n\n :return: ``True`` if the given user has been blocked\n :rtype: bool\n \"\"\"\n params = {'user': self.user_id, 'otherUser': other_user_id}\n response = self.session.get(self.url, params=params)\n return response.data['between']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef block(self, other_user_id):\n params = {'user': self.user_id, 'otherUser': other_user_id}\n response = self.session.post(self.url, params=params)\n block = response.data['block']\n return Block(self, **block)", "response": "Block the given user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nunblocking the given user.", "response": "def unblock(self, other_user_id):\n \"\"\"Unblock the given user.\n\n :param str other_user_id: the ID of the user to unblock\n :return: ``True`` if successful\n :rtype: bool\n \"\"\"\n params = {'user': self.user_id, 'otherUser': other_user_id}\n response = self.session.delete(self.url, params=params)\n return response.ok"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef list(self, page=1, per_page=10):\n return pagers.ChatList(self, self._raw_list, per_page=per_page,\n page=page)", "response": "List a page of chats."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list(self, before_id=None, since_id=None, after_id=None, limit=20):\n return pagers.MessageList(self, self._raw_list, before_id=before_id,\n after_id=after_id, since_id=since_id,\n limit=limit)", "response": "Return a page of group messages."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef list_since(self, message_id, limit=None):\n return self.list(since_id=message_id, limit=limit)", "response": "Return a page of group messages created since a message."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a page of group messages created after a message.", "response": "def list_after(self, message_id, limit=None):\n \"\"\"Return a page of group messages created after a message.\n\n This is used to page forwards through messages.\n\n :param str message_id: the ID of a message\n :param int limit: maximum number of messages per page\n :return: group messages\n :rtype: :class:`~groupy.pagers.MessageList`\n \"\"\"\n return self.list(after_id=message_id, limit=limit)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef list_all_before(self, message_id, limit=None):\n return self.list_before(message_id, limit=limit).autopage()", "response": "Return all group messages created before a message."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all group messages created after a message.", "response": "def list_all_after(self, message_id, limit=None):\n \"\"\"Return all group messages created after a message.\n\n :param str message_id: the ID of a message\n :param int limit: maximum number of messages per page\n :return: group messages\n :rtype: generator\n \"\"\"\n return self.list_after(message_id, limit=limit).autopage()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create(self, text=None, attachments=None, source_guid=None):\n message = {\n 'source_guid': source_guid or str(time.time()),\n }\n\n if text is not None:\n message['text'] = text\n\n if attachments is not None:\n message['attachments'] = [a.to_json() for a in attachments]\n\n payload = {'message': message}\n response = self.session.post(self.url, json=payload)\n message = response.data['message']\n return Message(self, **message)", "response": "Create a new message in the group."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a page of direct messages.", "response": "def list(self, before_id=None, since_id=None, **kwargs):\n \"\"\"Return a page of direct messages.\n\n The messages come in reversed order (newest first). Note you can only\n provide _one_ of ``before_id``, ``since_id``.\n\n :param str before_id: message ID for paging backwards\n :param str since_id: message ID for most recent messages since\n :return: direct messages\n :rtype: :class:`~groupy.pagers.MessageList`\n \"\"\"\n return pagers.MessageList(self, self._raw_list, before_id=before_id,\n since_id=since_id, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_all(self, before_id=None, since_id=None, **kwargs):\n return self.list(before_id=before_id, since_id=since_id, **kwargs).autopage()", "response": "Return all direct messages."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add(self, nickname, email=None, phone_number=None, user_id=None):\n member = {\n 'nickname': nickname,\n 'email': email,\n 'phone_number': phone_number,\n 'user_id': user_id,\n }\n return self.add_multiple(member)", "response": "Add a user to the group."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_multiple(self, *users):\n guid = uuid.uuid4()\n for i, user_ in enumerate(users):\n user_['guid'] = '{}-{}'.format(guid, i)\n\n payload = {'members': users}\n url = utils.urljoin(self.url, 'add')\n response = self.session.post(url, json=payload)\n return MembershipRequest(self, *users, group_id=self.group_id,\n **response.data)", "response": "Adds multiple users to the group at once."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check(self, results_id):\n path = 'results/{}'.format(results_id)\n url = utils.urljoin(self.url, path)\n response = self.session.get(url)\n if response.status_code == 503:\n raise exceptions.ResultsNotReady(response)\n if response.status_code == 404:\n raise exceptions.ResultsExpired(response)\n return response.data['members']", "response": "Check for results of a membership request."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update(self, nickname=None, **kwargs):\n url = self.url + 'hips/update'\n payload = {\n 'membership': {\n 'nickname': nickname,\n },\n }\n payload['membership'].update(kwargs)\n response = self.session.post(url, json=payload)\n return Member(self, self.group_id, **response.data)", "response": "Update your own membership."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove(self, membership_id):\n path = '{}/remove'.format(membership_id)\n url = utils.urljoin(self.url, path)\n payload = {'membership_id': membership_id}\n response = self.session.post(url, json=payload)\n return response.ok", "response": "Removes a member from the group."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef post(self, text=None, attachments=None, source_guid=None):\n return self.messages.create(text=text, attachments=attachments,\n source_guid=source_guid)", "response": "Post a direct message to the user."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds the member to another group.", "response": "def add_to_group(self, group_id, nickname=None):\n \"\"\"Add the member to another group.\n\n If a nickname is not provided the member's current nickname is used.\n\n :param str group_id: the group_id of a group\n :param str nickname: a new nickname\n :return: a membership request\n :rtype: :class:`MembershipRequest`\n \"\"\"\n if nickname is None:\n nickname = self.nickname\n memberships = Memberships(self.manager.session, group_id=group_id)\n return memberships.add(nickname, user_id=self.user_id)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_if_ready(self):\n try:\n results = self.manager.check(self.results_id)\n except exceptions.ResultsNotReady as e:\n self._is_ready = False\n self._not_ready_exception = e\n except exceptions.ResultsExpired as e:\n self._is_ready = True\n self._expired_exception = e\n else:\n failures = self.get_failed_requests(results)\n members = self.get_new_members(results)\n self.results = self.__class__.Results(list(members), list(failures))\n self._is_ready = True\n self._not_ready_exception = None", "response": "Check for and fetch the results if ready."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the requests that failed.", "response": "def get_failed_requests(self, results):\n \"\"\"Return the requests that failed.\n\n :param results: the results of a membership request check\n :type results: :class:`list`\n :return: the failed requests\n :rtype: generator\n \"\"\"\n data = {member['guid']: member for member in results}\n for request in self.requests:\n if request['guid'] not in data:\n yield request"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_new_members(self, results):\n for member in results:\n guid = member.pop('guid')\n yield Member(self.manager, self.group_id, **member)\n member['guid'] = guid", "response": "Return the newly added members."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn True if the results are ready.", "response": "def is_ready(self, check=True):\n \"\"\"Return ``True`` if the results are ready.\n\n If you pass ``check=False``, no attempt is made to check again for\n results.\n\n :param bool check: whether to query for the results\n :return: ``True`` if the results are ready\n :rtype: bool\n \"\"\"\n if not self._is_ready and check:\n self.check_if_ready()\n return self._is_ready"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\npolling the membership request for the result set to become ready.", "response": "def poll(self, timeout=30, interval=2):\n \"\"\"Return the results when they become ready.\n\n :param int timeout: the maximum time to wait for the results\n :param float interval: the number of seconds between checks\n :return: the membership request result\n :rtype: :class:`~groupy.api.memberships.MembershipResult.Results`\n \"\"\"\n time.sleep(interval)\n start = time.time()\n while time.time() - start < timeout and not self.is_ready():\n time.sleep(interval)\n return self.get()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get(self):\n if self._expired_exception:\n raise self._expired_exception\n if self._not_ready_exception:\n raise self._not_ready_exception\n return self.results", "response": "Return the results now."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list(self, page=1, per_page=10, omit=None):\n return pagers.GroupList(self, self._raw_list, page=page,\n per_page=per_page, omit=omit)", "response": "List the groups by page."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef list_all(self, per_page=10, omit=None):\n return self.list(per_page=per_page, omit=omit).autopage()", "response": "List all groups.\n\n Since the order of groups is determined by recent activity, this is the\n recommended way to obtain a list of all groups. See\n :func:`~groupy.api.groups.Groups.list` for details about ``omit``.\n\n :param int per_page: number of groups per page\n :param int omit: a comma-separated list of fields to exclude\n :return: a list of groups\n :rtype: :class:`~groupy.pagers.GroupList`"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_former(self):\n url = utils.urljoin(self.url, 'former')\n response = self.session.get(url)\n return [Group(self, **group) for group in response.data]", "response": "List all former groups."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a single group by ID.", "response": "def get(self, id):\n \"\"\"Get a single group by ID.\n\n :param str id: a group ID\n :return: a group\n :rtype: :class:`~groupy.api.groups.Group`\n \"\"\"\n url = utils.urljoin(self.url, id)\n response = self.session.get(url)\n return Group(self, **response.data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create(self, name, description=None, image_url=None, share=None, **kwargs):\n payload = {\n 'name': name,\n 'description': description,\n 'image_url': image_url,\n 'share': share,\n }\n payload.update(kwargs)\n response = self.session.post(self.url, json=payload)\n return Group(self, **response.data)", "response": "Create a new group."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, id, name=None, description=None, image_url=None,\n office_mode=None, share=None, **kwargs):\n \"\"\"Update the details of a group.\n\n .. note::\n\n There are significant bugs in this endpoint!\n 1. not providing ``name`` produces 400: \"Topic can't be blank\"\n 2. not providing ``office_mode`` produces 500: \"sql: Scan error on\n column index 14: sql/driver: couldn't convert () into\n type bool\"\n\n Note that these issues are \"handled\" automatically when calling\n update on a :class:`~groupy.api.groups.Group` object.\n\n :param str id: group ID\n :param str name: group name (140 characters maximum)\n :param str description: short description (255 characters maximum)\n :param str image_url: GroupMe image service URL\n :param bool office_mode: (undocumented)\n :param bool share: whether to generate a share URL\n :return: an updated group\n :rtype: :class:`~groupy.api.groups.Group`\n \"\"\"\n path = '{}/update'.format(id)\n url = utils.urljoin(self.url, path)\n payload = {\n 'name': name,\n 'description': description,\n 'image_url': image_url,\n 'office_mode': office_mode,\n 'share': share,\n }\n payload.update(kwargs)\n response = self.session.post(url, json=payload)\n return Group(self, **response.data)", "response": "Update the details of a group."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndestroying a group s ID .", "response": "def destroy(self, id):\n \"\"\"Destroy a group.\n\n :param str id: a group ID\n :return: ``True`` if successful\n :rtype: bool\n \"\"\"\n path = '{}/destroy'.format(id)\n url = utils.urljoin(self.url, path)\n response = self.session.post(url)\n return response.ok"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\njoining a group using a share token.", "response": "def join(self, group_id, share_token):\n \"\"\"Join a group using a share token.\n\n :param str group_id: the group_id of a group\n :param str share_token: the share token\n :return: the group\n :rtype: :class:`~groupy.api.groups.Group`\n \"\"\"\n path = '{}/join/{}'.format(group_id, share_token)\n url = utils.urljoin(self.url, path)\n response = self.session.post(url)\n group = response.data['group']\n return Group(self, **group)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rejoin(self, group_id):\n url = utils.urljoin(self.url, 'join')\n payload = {'group_id': group_id}\n response = self.session.post(url, json=payload)\n return Group(self, **response.data)", "response": "Rejoin a former group."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nchange the owner of a group.", "response": "def change_owners(self, group_id, owner_id):\n \"\"\"Change the owner of a group.\n\n .. note:: you must be the owner to change owners\n\n :param str group_id: the group_id of a group\n :param str owner_id: the ID of the new owner\n :return: the result\n :rtype: :class:`~groupy.api.groups.ChangeOwnersResult`\n \"\"\"\n url = utils.urljoin(self.url, 'change_owners')\n payload = {\n 'requests': [{\n 'group_id': group_id,\n 'owner_id': owner_id,\n }],\n }\n response = self.session.post(url, json=payload)\n result, = response.data['results'] # should be exactly one\n return ChangeOwnersResult(**result)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the details of the group.", "response": "def update(self, name=None, description=None, image_url=None,\n office_mode=None, share=None, **kwargs):\n \"\"\"Update the details of the group.\n\n :param str name: group name (140 characters maximum)\n :param str description: short description (255 characters maximum)\n :param str image_url: GroupMe image service URL\n :param bool office_mode: (undocumented)\n :param bool share: whether to generate a share URL\n :return: an updated group\n :rtype: :class:`~groupy.api.groups.Group`\n \"\"\"\n # note we default to the current values for name and office_mode as a\n # work-around for issues with the group update endpoint\n if name is None:\n name = self.name\n if office_mode is None:\n office_mode = self.office_mode\n return self.manager.update(id=self.id, name=name, description=description,\n image_url=image_url, office_mode=office_mode,\n share=share, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef refresh_from_server(self):\n group = self.manager.get(id=self.id)\n self.__init__(self.manager, **group.data)", "response": "Refresh the group from the server in place."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new bot in a particular group.", "response": "def create_bot(self, name, avatar_url=None, callback_url=None, dm_notification=None,\n **kwargs):\n \"\"\"Create a new bot in a particular group.\n\n :param str name: bot name\n :param str avatar_url: the URL of an image to use as an avatar\n :param str callback_url: a POST-back URL for each new message\n :param bool dm_notification: whether to POST-back for direct messages?\n :return: the new bot\n :rtype: :class:`~groupy.api.bots.Bot`\n \"\"\"\n return self._bots.create(name=name, group_id=self.group_id,\n avatar_url=avatar_url, callback_url=callback_url,\n dm_notification=dm_notification)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_membership(self):\n user_id = self._user.me['user_id']\n for member in self.members:\n if member.user_id == user_id:\n return member\n raise exceptions.MissingMembershipError(self.group_id, user_id)", "response": "Get your membership.\n\n Note that your membership may not exist. For example, you do not have\n a membership in a former group. Also, the group returned by the API\n when rejoining a former group does not contain your membership. You\n must call :func:`refresh_from_server` to update the list of members.\n\n :return: your membership in the group\n :rtype: :class:`~groupy.api.memberships.Member`\n :raises groupy.exceptions.MissingMembershipError: if your membership is\n not in the group data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating your own membership. Note that this fails on former groups. :param str nickname: new nickname :return: updated membership :rtype: :class:`~groupy.api.members.Member`", "response": "def update_membership(self, nickname=None, **kwargs):\n \"\"\"Update your own membership.\n\n Note that this fails on former groups.\n\n :param str nickname: new nickname\n :return: updated membership\n :rtype: :class:`~groupy.api.members.Member`\n \"\"\"\n return self.memberships.update(nickname=nickname, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef urljoin(base, path=None):\n # /foo/bar + baz makes /foo/bar/baz instead of /foo/baz\n if path is None:\n url = base\n else:\n if not base.endswith('/'):\n base += '/'\n url = urllib.parse.urljoin(base, str(path))\n return url", "response": "Join a base url with a relative path."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_share_url(share_url):\n *__, group_id, share_token = share_url.rstrip('/').split('/')\n return group_id, share_token", "response": "Parse a group s share url."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn an RFC 3339 timestamp.", "response": "def get_rfc3339(when):\n \"\"\"Return an RFC 3339 timestamp.\n\n :param datetime.datetime when: a datetime in UTC\n :return: RFC 3339 timestamp\n :rtype: str\n \"\"\"\n microseconds = format(when.microsecond, '04d')[:4]\n rfc3339 = '%Y-%m-%dT%H:%M:%S.{}Z'\n return when.strftime(rfc3339.format(microseconds))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a filter from keyword arguments.", "response": "def make_filter(**tests):\n \"\"\"Create a filter from keyword arguments.\"\"\"\n tests = [AttrTest(k, v) for k, v in tests.items()]\n return Filter(tests)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfind exactly one match in the list of objects.", "response": "def find(self, objects):\n \"\"\"Find exactly one match in the list of objects.\n\n :param objects: objects to filter\n :type objects: :class:`list`\n :return: the one matching object\n :raises groupy.exceptions.NoMatchesError: if no objects match\n :raises groupy.exceptions.MultipleMatchesError: if multiple objects match\n \"\"\"\n matches = list(self.__call__(objects))\n if not matches:\n raise exceptions.NoMatchesError(objects, self.tests)\n elif len(matches) > 1:\n raise exceptions.MultipleMatchesError(objects, self.tests,\n matches=matches)\n return matches[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of your bots.", "response": "def list(self):\n \"\"\"Return a list of bots.\n\n :return: all of your bots\n :rtype: :class:`list`\n \"\"\"\n response = self.session.get(self.url)\n return [Bot(self, **bot) for bot in response.data]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new bot in a particular group.", "response": "def create(self, name, group_id, avatar_url=None, callback_url=None,\n dm_notification=None, **kwargs):\n \"\"\"Create a new bot in a particular group.\n\n :param str name: bot name\n :param str group_id: the group_id of a group\n :param str avatar_url: the URL of an image to use as an avatar\n :param str callback_url: a POST-back URL for each new message\n :param bool dm_notification: whether to POST-back for direct messages?\n :return: the new bot\n :rtype: :class:`~groupy.api.bots.Bot`\n \"\"\"\n payload = {\n 'bot': {\n 'name': name,\n 'group_id': group_id,\n 'avatar_url': avatar_url,\n 'callback_url': callback_url,\n 'dm_notification': dm_notification,\n },\n }\n payload['bot'].update(kwargs)\n response = self.session.post(self.url, json=payload)\n bot = response.data['bot']\n return Bot(self, **bot)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef post(self, bot_id, text, attachments=None):\n url = utils.urljoin(self.url, 'post')\n payload = dict(bot_id=bot_id, text=text)\n\n if attachments:\n payload['attachments'] = [a.to_json() for a in attachments]\n\n response = self.session.post(url, json=payload)\n return response.ok", "response": "Post a new message as a bot to its room."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndestroys a bot. :param str bot_id: the ID of the bot to destroy :return: ``True`` if successful :rtype: bool", "response": "def destroy(self, bot_id):\n \"\"\"Destroy a bot.\n\n :param str bot_id: the ID of the bot to destroy\n :return: ``True`` if successful\n :rtype: bool\n \"\"\"\n url = utils.urljoin(self.url, 'destroy')\n payload = {'bot_id': bot_id}\n response = self.session.post(url, json=payload)\n return response.ok"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nposting a message as the bot.", "response": "def post(self, text, attachments=None):\n \"\"\"Post a message as the bot.\n\n :param str text: the text of the message\n :param attachments: a list of attachments\n :type attachments: :class:`list`\n :return: ``True`` if successful\n :rtype: bool\n \"\"\"\n return self.manager.post(self.bot_id, text, attachments)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nflattens a nested sequence.", "response": "def flatten_until(is_leaf, xs):\n \"\"\"\n Flatten a nested sequence. A sequence could be a nested list of lists\n or tuples or a combination of both\n\n :param is_leaf: Predicate. Predicate to determine whether an item\n in the iterable `xs` is a leaf node or not.\n :param xs: Iterable. Nested lists or tuples\n :return: list.\n \"\"\"\n\n def _flatten_until(items):\n if isinstance(Iterable, items) and not is_leaf(items):\n for item in items:\n for i in _flatten_until(item):\n yield i\n else:\n yield items\n\n return list(_flatten_until(xs))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling the function f by flipping the first two positional arguments", "response": "def flip(f):\n \"\"\"\n Calls the function f by flipping the first two positional\n arguments\n \"\"\"\n\n def wrapped(*args, **kwargs):\n return f(*flip_first_two(args), **kwargs)\n\n f_spec = make_func_curry_spec(f)\n\n return curry_by_spec(f_spec, wrapped)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cachier(stale_after=None, next_time=False, pickle_reload=True,\n mongetter=None):\n \"\"\"A persistent, stale-free memoization decorator.\n\n The positional and keyword arguments to the wrapped function must be\n hashable (i.e. Python's immutable built-in objects, not mutable\n containers). Also, notice that since objects which are instances of\n user-defined classes are hashable but all compare unequal (their hash\n value is their id), equal objects across different sessions will not yield\n identical keys.\n\n Arguments\n ---------\n stale_after (optional) : datetime.timedelta\n The time delta afterwhich a cached result is considered stale. Calls\n made after the result goes stale will trigger a recalculation of the\n result, but whether a stale or fresh result will be returned is\n determined by the optional next_time argument.\n next_time (optional) : bool\n If set to True, a stale result will be returned when finding one, not\n waiting for the calculation of the fresh result to return. Defaults to\n False.\n pickle_reload (optional) : bool\n If set to True, in-memory cache will be reloaded on each cache read,\n enabling different threads to share cache. Should be set to False for\n faster reads in single-thread programs. Defaults to True.\n mongetter (optional) : callable\n A callable that takes no arguments and returns a pymongo.Collection\n object with writing permissions. If unset a local pickle cache is used\n instead.\n \"\"\"\n # print('Inside the wrapper maker')\n # print('mongetter={}'.format(mongetter))\n # print('stale_after={}'.format(stale_after))\n # print('next_time={}'.format(next_time))\n\n if mongetter:\n core = _MongoCore(mongetter, stale_after, next_time)\n else:\n core = _PickleCore( # pylint: disable=R0204\n stale_after, next_time, pickle_reload)\n\n def _cachier_decorator(func):\n core.set_func(func)\n\n @wraps(func)\n def func_wrapper(*args, **kwds): # pylint: disable=C0111,R0911\n # print('Inside general wrapper for {}.'.format(func.__name__))\n ignore_cache = kwds.pop('ignore_cache', False)\n overwrite_cache = kwds.pop('overwrite_cache', False)\n verbose_cache = kwds.pop('verbose_cache', False)\n _print = lambda x: None\n if verbose_cache:\n _print = print\n if ignore_cache:\n return func(*args, **kwds)\n key, entry = core.get_entry(args, kwds)\n if overwrite_cache:\n return _calc_entry(core, key, func, args, kwds)\n if entry is not None: # pylint: disable=R0101\n _print('Entry found.')\n if entry.get('value', None) is not None:\n _print('Cached result found.')\n if stale_after:\n now = datetime.datetime.now()\n if now - entry['time'] > stale_after:\n _print('But it is stale... :(')\n if entry['being_calculated']:\n if next_time:\n _print('Returning stale.')\n return entry['value'] # return stale val\n _print('Already calc. Waiting on change.')\n try:\n return core.wait_on_entry_calc(key)\n except RecalculationNeeded:\n return _calc_entry(core, key, func, args, kwds)\n if next_time:\n _print('Async calc and return stale')\n try:\n core.mark_entry_being_calculated(key)\n _get_executor().submit(\n _function_thread, core, key, func,\n args, kwds)\n finally:\n core.mark_entry_not_calculated(key)\n return entry['value']\n _print('Calling decorated function and waiting')\n return _calc_entry(core, key, func, args, kwds)\n _print('And it is fresh!')\n return entry['value']\n if entry['being_calculated']:\n _print('No value but being calculated. Waiting.')\n try:\n return core.wait_on_entry_calc(key)\n except RecalculationNeeded:\n return _calc_entry(core, key, func, args, kwds)\n _print('No entry found. No current calc. Calling like a boss.')\n return _calc_entry(core, key, func, args, kwds)\n\n def clear_cache():\n \"\"\"Clear the cache.\"\"\"\n core.clear_cache()\n\n def clear_being_calculated():\n \"\"\"Marks all entries in this cache as not being calculated.\"\"\"\n core.clear_being_calculated()\n\n func_wrapper.clear_cache = clear_cache\n func_wrapper.clear_being_calculated = clear_being_calculated\n return func_wrapper\n\n return _cachier_decorator", "response": "A persistent stale - free memoization decorator."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsearching for UD s definition ID and return list of UrbanDefinition objects.", "response": "def defineID(defid):\n \"\"\"Search for UD's definition ID and return list of UrbanDefinition objects.\n\n Keyword arguments:\n defid -- definition ID to search for (int or str)\n \"\"\"\n json = _get_urban_json(UD_DEFID_URL + urlquote(str(defid)))\n return _parse_urban_json(json)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck that a non - Python dependency is installed.", "response": "def has_external_dependency(name):\n 'Check that a non-Python dependency is installed.'\n for directory in os.environ['PATH'].split(':'):\n if os.path.exists(os.path.join(directory, name)):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntest VTK interface and repair of Stanford Bunny Mesh", "response": "def with_vtk(plot=True):\n \"\"\" Tests VTK interface and mesh repair of Stanford Bunny Mesh \"\"\"\n mesh = vtki.PolyData(bunny_scan)\n meshfix = pymeshfix.MeshFix(mesh)\n if plot:\n print('Plotting input mesh')\n meshfix.plot()\n meshfix.repair()\n if plot:\n print('Plotting repaired mesh')\n meshfix.plot()\n\n return meshfix.mesh"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_arrays(self, v, f):\n # Check inputs\n if not isinstance(v, np.ndarray):\n try:\n v = np.asarray(v, np.float)\n if v.ndim != 2 and v.shape[1] != 3:\n raise Exception('Invalid vertex format. Shape ' +\n 'should be (npoints, 3)')\n except BaseException:\n raise Exception(\n 'Unable to convert vertex input to valid numpy array')\n\n if not isinstance(f, np.ndarray):\n try:\n f = np.asarray(f, ctypes.c_int)\n if f.ndim != 2 and f.shape[1] != 3:\n raise Exception('Invalid face format. ' +\n 'Shape should be (nfaces, 3)')\n except BaseException:\n raise Exception('Unable to convert face input to valid' +\n ' numpy array')\n\n self.v = v\n self.f = f", "response": "Loads triangular mesh from vertex and face numpy arrays."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef mesh(self):\n triangles = np.empty((self.f.shape[0], 4))\n triangles[:, -3:] = self.f\n triangles[:, 0] = 3\n return vtki.PolyData(self.v, triangles, deep=False)", "response": "Return the surface mesh"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef plot(self, show_holes=True):\n if show_holes:\n edges = self.mesh.extract_edges(boundary_edges=True,\n feature_edges=False,\n manifold_edges=False)\n\n plotter = vtki.Plotter()\n plotter.add_mesh(self.mesh, label='mesh')\n plotter.add_mesh(edges, 'r', label='edges')\n plotter.plot()\n\n else:\n self.mesh.plot(show_edges=True)", "response": "Plot the mesh.\n\n Parameters\n ----------\n show_holes : bool, optional\n Shows boundaries. Default True"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nperforms a mesh repair using MeshFix s default repair process.", "response": "def repair(self, verbose=False, joincomp=False,\n remove_smallest_components=True):\n \"\"\"Performs mesh repair using MeshFix's default repair\n process.\n\n Parameters\n ----------\n verbose : bool, optional\n Enables or disables debug printing. Disabled by default.\n\n joincomp : bool, optional\n Attempts to join nearby open components.\n\n remove_smallest_components : bool, optional\n Remove all but the largest isolated component from the\n mesh before beginning the repair process. Default True\n\n Notes\n -----\n Vertex and face arrays are updated inplace. Access them with:\n meshfix.v\n meshfix.f\n \"\"\"\n assert self.f.shape[1] == 3, 'Face array must contain three columns'\n assert self.f.ndim == 2, 'Face array must be 2D'\n self.v, self.f = _meshfix.clean_from_arrays(self.v, self.f,\n verbose, joincomp,\n remove_smallest_components)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting a surface mesh to disk.", "response": "def write(self, filename, binary=True):\n \"\"\"Writes a surface mesh to disk.\n\n Written file may be an ASCII or binary ply, stl, or vtk mesh\n file.\n\n Parameters\n ----------\n filename : str\n Filename of mesh to be written. Filetype is inferred from\n the extension of the filename unless overridden with\n ftype. Can be one of the following types (.ply, .stl,\n .vtk)\n\n ftype : str, optional\n Filetype. Inferred from filename unless specified with a\n three character string. Can be one of the following:\n 'ply', 'stl', or 'vtk'.\n\n Notes\n -----\n Binary files write much faster than ASCII.\n \"\"\"\n self.mesh.write(filename, binary)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nscrapes a URL optionally with parameters.", "response": "def scrape(url, params=None, user_agent=None):\n '''\n Scrape a URL optionally with parameters.\n This is effectively a wrapper around urllib2.urlopen.\n '''\n\n headers = {}\n\n if user_agent:\n headers['User-Agent'] = user_agent\n\n data = params and six.moves.urllib.parse.urlencode(params) or None\n req = six.moves.urllib.request.Request(url, data=data, headers=headers)\n f = six.moves.urllib.request.urlopen(req)\n\n text = f.read()\n f.close()\n\n return text"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts pdf file to xml file", "response": "def pdftoxml(pdfdata, options=\"\"):\n \"\"\"converts pdf file to xml file\"\"\"\n pdffout = tempfile.NamedTemporaryFile(suffix='.pdf')\n pdffout.write(pdfdata)\n pdffout.flush()\n\n xmlin = tempfile.NamedTemporaryFile(mode='r', suffix='.xml')\n tmpxml = xmlin.name # \"temph.xml\"\n cmd = 'pdftohtml -xml -nodrm -zoom 1.5 -enc UTF-8 -noframes %s \"%s\" \"%s\"' % (\n options, pdffout.name, os.path.splitext(tmpxml)[0])\n # can't turn off output, so throw away even stderr yeuch\n cmd = cmd + \" >/dev/null 2>&1\"\n os.system(cmd)\n\n pdffout.close()\n #xmlfin = open(tmpxml)\n xmldata = xmlin.read()\n xmlin.close()\n return xmldata.decode('utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting an arbitrary SQL query given by query returning a list of OrderedDicts.", "response": "def execute(query, data=None):\n \"\"\"\n Execute an arbitrary SQL query given by query, returning any\n results as a list of OrderedDicts. A list of values can be supplied as an,\n additional argument, which will be substituted into question marks in the\n query.\n \"\"\"\n connection = _State.connection()\n _State.new_transaction()\n\n if data is None:\n data = []\n\n result = connection.execute(query, data)\n\n _State.table = None\n _State.metadata = None\n try:\n del _State.table_pending\n except AttributeError:\n pass\n\n if not result.returns_rows:\n return {u'data': [], u'keys': []}\n\n return {u'data': result.fetchall(), u'keys': list(result.keys())}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef select(query, data=None):\n connection = _State.connection()\n _State.new_transaction()\n if data is None:\n data = []\n\n result = connection.execute('select ' + query, data)\n\n rows = []\n for row in result:\n rows.append(dict(list(row.items())))\n\n return rows", "response": "Perform a sql select statement with the given query and return any results as a list of OrderedDicts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving the given data to the specified table.", "response": "def save(unique_keys, data, table_name='swdata'):\n \"\"\"\n Save the given data to the table specified by `table_name`\n (which defaults to 'swdata'). The data must be a mapping\n or an iterable of mappings. Unique keys is a list of keys that exist\n for all rows and for which a unique index will be created.\n \"\"\"\n\n _set_table(table_name)\n\n connection = _State.connection()\n\n if isinstance(data, Mapping):\n # Is a single datum\n data = [data]\n elif not isinstance(data, Iterable):\n raise TypeError(\"Data must be a single mapping or an iterable \"\n \"of mappings\")\n\n insert = _State.table.insert(prefixes=['OR REPLACE'])\n for row in data:\n if not isinstance(row, Mapping):\n raise TypeError(\"Elements of data must be mappings, got {}\".format(\n type(row)))\n fit_row(connection, row, unique_keys)\n connection.execute(insert.values(row))\n _State.check_last_committed()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the table to work on.", "response": "def _set_table(table_name):\n \"\"\"\n Specify the table to work on.\n \"\"\"\n _State.connection()\n _State.reflect_metadata()\n _State.table = sqlalchemy.Table(table_name, _State.metadata,\n extend_existing=True)\n\n if list(_State.table.columns.keys()) == []:\n _State.table_pending = True\n else:\n _State.table_pending = False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef show_tables():\n _State.connection()\n _State.reflect_metadata()\n metadata = _State.metadata\n\n response = select('name, sql from sqlite_master where type=\"table\"')\n\n return {row['name']: row['sql'] for row in response}", "response": "Return the names of the tables currently in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsaves a variable to the table specified by _State. vars_table_name.", "response": "def save_var(name, value):\n \"\"\"\n Save a variable to the table specified by _State.vars_table_name. Key is\n the name of the variable, and value is the value.\n \"\"\"\n connection = _State.connection()\n _State.reflect_metadata()\n\n vars_table = sqlalchemy.Table(\n _State.vars_table_name, _State.metadata,\n sqlalchemy.Column('name', sqlalchemy.types.Text, primary_key=True),\n sqlalchemy.Column('value_blob', sqlalchemy.types.LargeBinary),\n sqlalchemy.Column('type', sqlalchemy.types.Text),\n keep_existing=True\n )\n\n vars_table.create(bind=connection, checkfirst=True)\n\n column_type = get_column_type(value)\n\n if column_type == sqlalchemy.types.LargeBinary:\n value_blob = value\n else:\n value_blob = unicode(value).encode('utf-8')\n\n values = dict(name=name,\n value_blob=value_blob,\n # value_blob=Blob(value),\n type=column_type.__visit_name__.lower())\n\n vars_table.insert(prefixes=['OR REPLACE']).values(**values).execute()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the value of a variable in the _State. vars_table_name.", "response": "def get_var(name, default=None):\n \"\"\"\n Returns the variable with the provided key from the\n table specified by _State.vars_table_name.\n \"\"\"\n alchemytypes = {\"text\": lambda x: x.decode('utf-8'),\n \"big_integer\": lambda x: int(x),\n \"date\": lambda x: x.decode('utf-8'),\n \"datetime\": lambda x: x.decode('utf-8'),\n \"float\": lambda x: float(x),\n \"large_binary\": lambda x: x,\n \"boolean\": lambda x: x==b'True'}\n\n connection = _State.connection()\n _State.new_transaction()\n\n if _State.vars_table_name not in list(_State.metadata.tables.keys()):\n return None\n\n table = sqlalchemy.Table(_State.vars_table_name, _State.metadata)\n s = sqlalchemy.select([table.c.value_blob, table.c.type])\n s = s.where(table.c.name == name)\n result = connection.execute(s).fetchone()\n\n if not result:\n return None\n\n return alchemytypes[result[1]](result[0])\n\n # This is to do the variable type conversion through the SQL engine\n execute = connection.execute\n execute(\"CREATE TEMPORARY TABLE _sw_tmp ('value' {})\".format(result.type))\n execute(\"INSERT INTO _sw_tmp VALUES (:value)\", value=result.value_blob)\n var = execute('SELECT value FROM _sw_tmp').fetchone().value\n execute(\"DROP TABLE _sw_tmp\")\n return var.decode('utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a new index of the columns in column_names where column_names is a list of strings.", "response": "def create_index(column_names, unique=False):\n \"\"\"\n Create a new index of the columns in column_names, where column_names is\n a list of strings. If unique is True, it will be a\n unique index.\n \"\"\"\n connection = _State.connection()\n _State.reflect_metadata()\n table_name = _State.table.name\n\n table = _State.table\n\n index_name = re.sub(r'[^a-zA-Z0-9]', '', table_name) + '_'\n index_name += '_'.join(re.sub(r'[^a-zA-Z0-9]', '', x)\n for x in column_names)\n\n if unique:\n index_name += '_unique'\n\n columns = []\n for column_name in column_names:\n columns.append(table.columns[column_name])\n\n current_indices = [x.name for x in table.indexes]\n index = sqlalchemy.schema.Index(index_name, *columns, unique=unique)\n if index.name not in current_indices:\n index.create(bind=_State.engine)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fit_row(connection, row, unique_keys):\n new_columns = []\n for column_name, column_value in list(row.items()):\n new_column = sqlalchemy.Column(column_name,\n get_column_type(column_value))\n\n if not column_name in list(_State.table.columns.keys()):\n new_columns.append(new_column)\n _State.table.append_column(new_column)\n\n if _State.table_pending:\n create_table(unique_keys)\n return\n\n for new_column in new_columns:\n add_column(connection, new_column)", "response": "Takes a row and checks to make sure it fits in the columns of the current table. Adds the required columns."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_table(unique_keys):\n _State.new_transaction()\n _State.table.create(bind=_State.engine, checkfirst=True)\n if unique_keys != []:\n create_index(unique_keys, unique=True)\n _State.table_pending = False\n _State.reflect_metadata()", "response": "Create the table in the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding a column to the current table.", "response": "def add_column(connection, column):\n \"\"\"\n Add a column to the current table.\n \"\"\"\n stmt = alembic.ddl.base.AddColumn(_State.table.name, column)\n connection.execute(stmt)\n _State.reflect_metadata()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef drop():\n # Ensure the connection is up\n _State.connection()\n _State.table.drop(checkfirst=True)\n _State.metadata.remove(_State.table)\n _State.table = None\n _State.new_transaction()", "response": "Drop the current table if it exists."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextracts attributes and attaches them to element.", "response": "def attach_attrs_table(key, value, fmt, meta):\n \"\"\"Extracts attributes and attaches them to element.\"\"\"\n\n # We can't use attach_attrs_factory() because Table is a block-level element\n if key in ['Table']:\n assert len(value) == 5\n caption = value[0] # caption, align, x, head, body\n\n # Set n to the index where the attributes start\n n = 0\n while n < len(caption) and not \\\n (caption[n]['t'] == 'Str' and caption[n]['c'].startswith('{')):\n n += 1\n\n try:\n attrs = extract_attrs(caption, n)\n value.insert(0, attrs)\n except (ValueError, IndexError):\n pass"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprocess the attributed tables.", "response": "def process_tables(key, value, fmt, meta):\n \"\"\"Processes the attributed tables.\"\"\"\n\n global has_unnumbered_tables # pylint: disable=global-statement\n\n # Process block-level Table elements\n if key == 'Table':\n\n # Inspect the table\n if len(value) == 5: # Unattributed, bail out\n has_unnumbered_tables = True\n if fmt in ['latex']:\n return [RawBlock('tex', r'\\begin{no-prefix-table-caption}'),\n Table(*value),\n RawBlock('tex', r'\\end{no-prefix-table-caption}')]\n return None\n\n # Process the table\n table = _process_table(value, fmt)\n\n # Context-dependent output\n attrs = table['attrs']\n if table['is_unnumbered']:\n if fmt in ['latex']:\n return [RawBlock('tex', r'\\begin{no-prefix-table-caption}'),\n AttrTable(*value),\n RawBlock('tex', r'\\end{no-prefix-table-caption}')]\n\n elif fmt in ['latex']:\n if table['is_tagged']: # Code in the tags\n tex = '\\n'.join([r'\\let\\oldthetable=\\thetable',\n r'\\renewcommand\\thetable{%s}'%\\\n references[attrs[0]]])\n pre = RawBlock('tex', tex)\n tex = '\\n'.join([r'\\let\\thetable=\\oldthetable',\n r'\\addtocounter{table}{-1}'])\n post = RawBlock('tex', tex)\n return [pre, AttrTable(*value), post]\n elif table['is_unreferenceable']:\n attrs[0] = '' # The label isn't needed any further\n elif fmt in ('html', 'html5') and LABEL_PATTERN.match(attrs[0]):\n # Insert anchor\n anchor = RawBlock('html', ''%attrs[0])\n return [anchor, AttrTable(*value)]\n elif fmt == 'docx':\n # As per http://officeopenxml.com/WPhyperlink.php\n bookmarkstart = \\\n RawBlock('openxml',\n ''\n %attrs[0])\n bookmarkend = \\\n RawBlock('openxml', '')\n return [bookmarkstart, AttrTable(*value), bookmarkend]\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfilter the document AST and returns the resulting XML tree.", "response": "def main():\n \"\"\"Filters the document AST.\"\"\"\n\n # pylint: disable=global-statement\n global PANDOCVERSION\n global AttrTable\n\n # Get the output format and document\n fmt = args.fmt\n doc = json.loads(STDIN.read())\n\n # Initialize pandocxnos\n # pylint: disable=too-many-function-args\n PANDOCVERSION = pandocxnos.init(args.pandocversion, doc)\n\n # Element primitives\n AttrTable = elt('Table', 6)\n\n # Chop up the doc\n meta = doc['meta'] if PANDOCVERSION >= '1.18' else doc[0]['unMeta']\n blocks = doc['blocks'] if PANDOCVERSION >= '1.18' else doc[1:]\n\n # Process the metadata variables\n process(meta)\n\n # First pass\n detach_attrs_table = detach_attrs_factory(Table)\n insert_secnos = insert_secnos_factory(Table)\n delete_secnos = delete_secnos_factory(Table)\n altered = functools.reduce(lambda x, action: walk(x, action, fmt, meta),\n [attach_attrs_table, insert_secnos,\n process_tables, delete_secnos,\n detach_attrs_table], blocks)\n\n # Second pass\n process_refs = process_refs_factory(references.keys())\n replace_refs = replace_refs_factory(references,\n use_cleveref_default, False,\n plusname if not capitalize else\n [name.title() for name in plusname],\n starname, 'table')\n altered = functools.reduce(lambda x, action: walk(x, action, fmt, meta),\n [repair_refs, process_refs, replace_refs],\n altered)\n\n # Insert supporting TeX\n if fmt in ['latex']:\n\n rawblocks = []\n\n if has_unnumbered_tables:\n rawblocks += [RawBlock('tex', TEX0),\n RawBlock('tex', TEX1),\n RawBlock('tex', TEX2)]\n\n if captionname != 'Table':\n rawblocks += [RawBlock('tex', TEX3 % captionname)]\n\n insert_rawblocks = insert_rawblocks_factory(rawblocks)\n\n altered = functools.reduce(lambda x, action: walk(x, action, fmt, meta),\n [insert_rawblocks], altered)\n\n # Update the doc\n if PANDOCVERSION >= '1.18':\n doc['blocks'] = altered\n else:\n doc = doc[:1] + altered\n\n # Dump the results\n json.dump(doc, STDOUT)\n\n # Flush stdout\n STDOUT.flush()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef request(url, xml):\n \n r = _urllib.Request(url, xml)\n r.add_header('Content-Type', 'application/xml')\n r.add_header('User-Agent', 'Clockwork Python wrapper/1.0')\n \n result = {}\n try:\n f = _urllib.urlopen(r)\n except URLError as error:\n raise clockwork_exceptions.HttpException(\"Error connecting to clockwork server: %s\" % error)\n \n result['data'] = f.read()\n result['status'] = f.getcode()\n \n if hasattr(f, 'headers'):\n result['etag'] = f.headers.get('ETag')\n result['lastmodified'] = f.headers.get('Last-Modified')\n if f.headers.get('content\u2212encoding', '') == 'gzip':\n result['data'] = gzip.GzipFile(fileobj=StringIO(result['data'])).read()\n if hasattr(f, 'url'):\n result['url'] = f.url\n result['status'] = 200\n f.close()\n \n if result['status'] != 200:\n raise clockwork_exceptions.HttpException(\"Error connecting to clockwork server - status code %s\" % result['status'])\n \n return result", "response": "Make a http request to the clockwork server"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck the balance fot this account.", "response": "def get_balance(self):\n \"\"\"Check the balance fot this account.\n Returns a dictionary containing:\n account_type: The account type\n balance: The balance remaining on the account\n currency: The currency used for the account balance. Assume GBP in not set\"\"\"\n\n xml_root = self.__init_xml('Balance')\n\n response = clockwork_http.request(BALANCE_URL, etree.tostring(xml_root, encoding='utf-8'))\n data_etree = etree.fromstring(response['data'])\n\n err_desc = data_etree.find('ErrDesc')\n if err_desc is not None:\n raise clockwork_exceptions.ApiException(err_desc.text, data_etree.find('ErrNo').text)\n\n result = {}\n result['account_type'] = data_etree.find('AccountType').text\n result['balance'] = data_etree.find('Balance').text\n result['currency'] = data_etree.find('Currency').text\n return result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send(self, messages):\n\n tmpSms = SMS(to='', message='')\n if str(type(messages)) == str(type(tmpSms)):\n messages = [messages]\n\n xml_root = self.__init_xml('Message')\n wrapper_id = 0\n\n for m in messages:\n m.wrapper_id = wrapper_id\n msg = self.__build_sms_data(m)\n sms = etree.SubElement(xml_root, 'SMS')\n for sms_element in msg:\n element = etree.SubElement(sms, sms_element)\n element.text = msg[sms_element]\n\n # print etree.tostring(xml_root)\n response = clockwork_http.request(SMS_URL, etree.tostring(xml_root, encoding='utf-8'))\n response_data = response['data']\n\n # print response_data\n data_etree = etree.fromstring(response_data)\n\n # Check for general error\n err_desc = data_etree.find('ErrDesc')\n if err_desc is not None:\n raise clockwork_exceptions.ApiException(err_desc.text, data_etree.find('ErrNo').text)\n\n # Return a consistent object\n results = []\n for sms in data_etree:\n matching_sms = next((s for s in messages if str(s.wrapper_id) == sms.find('WrapperID').text), None)\n new_result = SMSResponse(\n sms = matching_sms,\n id = '' if sms.find('MessageID') is None else sms.find('MessageID').text,\n error_code = 0 if sms.find('ErrNo') is None else sms.find('ErrNo').text,\n error_message = '' if sms.find('ErrDesc') is None else sms.find('ErrDesc').text,\n success = True if sms.find('ErrNo') is None else (sms.find('ErrNo').text == 0)\n )\n results.append(new_result)\n\n if len(results) > 1:\n return results\n\n return results[0]", "response": "Send a list of SMS messages or an array of SMS messages"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __init_xml(self, rootElementTag):\n xml_root = etree.Element(rootElementTag)\n key = etree.SubElement(xml_root, \"Key\")\n key.text = self.apikey\n return xml_root", "response": "Init a etree element and pop a key in there"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __build_sms_data(self, message):\n\n attributes = {}\n\n attributes_to_translate = {\n 'to' : 'To',\n 'message' : 'Content',\n 'client_id' : 'ClientID',\n 'concat' : 'Concat',\n 'from_name': 'From',\n 'invalid_char_option' : 'InvalidCharOption',\n 'truncate' : 'Truncate',\n 'wrapper_id' : 'WrapperId'\n }\n\n for attr in attributes_to_translate:\n val_to_use = None\n if hasattr(message, attr):\n val_to_use = getattr(message, attr)\n if val_to_use is None and hasattr(self, attr):\n val_to_use = getattr(self, attr)\n if val_to_use is not None:\n attributes[attributes_to_translate[attr]] = str(val_to_use)\n\n return attributes", "response": "Build a dictionary of SMS message elements"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pstats2entries(data):\n # Each entry's key is a tuple of (filename, line number, function name)\n entries = {}\n allcallers = {}\n\n # first pass over stats to build the list of entry instances\n for code_info, call_info in data.stats.items():\n # build a fake code object\n code = Code(*code_info)\n\n # build a fake entry object. entry.calls will be filled during the\n # second pass over stats\n cc, nc, tt, ct, callers = call_info\n entry = Entry(code, callcount=cc, reccallcount=nc - cc, inlinetime=tt,\n totaltime=ct, calls=[])\n\n # collect the new entry\n entries[code_info] = entry\n allcallers[code_info] = list(callers.items())\n\n # second pass of stats to plug callees into callers\n for entry in entries.values():\n entry_label = cProfile.label(entry.code)\n entry_callers = allcallers.get(entry_label, [])\n for entry_caller, call_info in entry_callers:\n cc, nc, tt, ct = call_info\n subentry = Subentry(entry.code, callcount=cc, reccallcount=nc - cc,\n inlinetime=tt, totaltime=ct)\n # entry_caller has the same form as code_info\n entries[entry_caller].calls.append(subentry)\n\n return list(entries.values())", "response": "Convert serialized pstats to a list of raw entries."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_installed(prog):\n with open(os.devnull, 'w') as devnull:\n try:\n if os.name == 'nt':\n retcode = subprocess.call(['where', prog], stdout=devnull)\n else:\n retcode = subprocess.call(['which', prog], stdout=devnull)\n except OSError as e:\n # If where or which doesn't exist, a \"ENOENT\" error will occur (The\n # FileNotFoundError subclass on Python 3).\n if e.errno != errno.ENOENT:\n raise\n retcode = 1\n\n return retcode == 0", "response": "Return whether or not a given executable is installed on the machine."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes the kcachegrind tool on the calltree.", "response": "def main():\n \"\"\"Execute the converter using parameters provided on the command line\"\"\"\n\n parser = argparse.ArgumentParser()\n parser.add_argument('-o', '--outfile', metavar='output_file_path',\n help=\"Save calltree stats to \")\n parser.add_argument('-i', '--infile', metavar='input_file_path',\n help=\"Read Python stats from \")\n parser.add_argument('-k', '--kcachegrind',\n help=\"Run the kcachegrind tool on the converted data\",\n action=\"store_true\")\n parser.add_argument('-r', '--run-script',\n nargs=argparse.REMAINDER,\n metavar=('scriptfile', 'args'),\n dest='script',\n help=\"Name of the Python script to run to collect\"\n \" profiling data\")\n args = parser.parse_args()\n\n outfile = args.outfile\n\n if args.script is not None:\n # collect profiling data by running the given script\n if not args.outfile:\n outfile = '%s.log' % os.path.basename(args.script[0])\n\n fd, tmp_path = tempfile.mkstemp(suffix='.prof', prefix='pyprof2calltree')\n os.close(fd)\n try:\n cmd = [\n sys.executable,\n '-m', 'cProfile',\n '-o', tmp_path,\n ]\n cmd.extend(args.script)\n subprocess.check_call(cmd)\n\n kg = CalltreeConverter(tmp_path)\n finally:\n os.remove(tmp_path)\n\n elif args.infile is not None:\n # use the profiling data from some input file\n if not args.outfile:\n outfile = '%s.log' % os.path.basename(args.infile)\n\n if args.infile == outfile:\n # prevent name collisions by appending another extension\n outfile += \".log\"\n\n kg = CalltreeConverter(pstats.Stats(args.infile))\n\n else:\n # at least an input file or a script to run is required\n parser.print_usage()\n sys.exit(2)\n\n if args.outfile is not None or not args.kcachegrind:\n # user either explicitly required output file or requested by not\n # explicitly asking to launch kcachegrind\n sys.stderr.write(\"writing converted data to: %s\\n\" % outfile)\n with open(outfile, 'w') as f:\n kg.output(f)\n\n if args.kcachegrind:\n sys.stderr.write(\"launching kcachegrind\\n\")\n kg.visualize()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef convert(profiling_data, outputfile):\n converter = CalltreeConverter(profiling_data)\n if is_basestring(outputfile):\n with open(outputfile, \"w\") as f:\n converter.output(f)\n else:\n converter.output(outputfile)", "response": "convert profiling_data to calltree format and dump it to outputfile"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef output(self, out_file):\n self.out_file = out_file\n out_file.write('event: ns : Nanoseconds\\n')\n out_file.write('events: ns\\n')\n self._output_summary()\n for entry in sorted(self.entries, key=_entry_sort_key):\n self._output_entry(entry)", "response": "Write the converted entries to out_file"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlaunch kcachegrind on the converted entries.", "response": "def visualize(self):\n \"\"\"Launch kcachegrind on the converted entries.\n\n One of the executables listed in KCACHEGRIND_EXECUTABLES\n must be present in the system path.\n \"\"\"\n\n available_cmd = None\n for cmd in KCACHEGRIND_EXECUTABLES:\n if is_installed(cmd):\n available_cmd = cmd\n break\n\n if available_cmd is None:\n sys.stderr.write(\"Could not find kcachegrind. Tried: %s\\n\" %\n \", \".join(KCACHEGRIND_EXECUTABLES))\n return\n\n if self.out_file is None:\n fd, outfile = tempfile.mkstemp(\".log\", \"pyprof2calltree\")\n use_temp_file = True\n else:\n outfile = self.out_file.name\n use_temp_file = False\n\n try:\n if use_temp_file:\n with io.open(fd, \"w\") as f:\n self.output(f)\n subprocess.call([available_cmd, outfile])\n finally:\n # clean the temporary file\n if use_temp_file:\n os.remove(outfile)\n self.out_file = None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_app(self, reference_app=None):\n\n if reference_app is not None:\n return reference_app\n\n if self.app is not None:\n return self.app\n\n ctx = stack.top\n\n if ctx is not None:\n return ctx.app\n\n raise RuntimeError('Application not registered on Bouncer'\n ' instance and no application bound'\n ' to current context')", "response": "Helper method that implements the logic to look up an application."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitializes the Flask - Bouncer extension for the specified application.", "response": "def init_app(self, app, **kwargs):\n \"\"\" Initializes the Flask-Bouncer extension for the specified application.\n\n :param app: The application.\n \"\"\"\n self.app = app\n\n self._init_extension()\n\n self.app.before_request(self.check_implicit_rules)\n\n if kwargs.get('ensure_authorization', False):\n self.app.after_request(self.check_authorization)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_authorization(self, response):\n if not hasattr(request, '_authorized'):\n raise Unauthorized\n elif not request._authorized:\n raise Unauthorized\n return response", "response": "checks that an authorization call has been made during the request"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_implicit_rules(self):\n if not self.request_is_managed_by_flask_classy():\n return\n\n if self.method_is_explictly_overwritten():\n return\n\n class_name, action = request.endpoint.split(':')\n clazz = [classy_class for classy_class in self.flask_classy_classes if classy_class.__name__ == class_name][0]\n Condition(action, clazz.__target_model__).test()", "response": "Check implicit rules for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rotate_texture(texture, rotation, x_offset=0.5, y_offset=0.5):\n x, y = texture\n x = x.copy() - x_offset\n y = y.copy() - y_offset\n angle = np.radians(rotation)\n x_rot = x * np.cos(angle) + y * np.sin(angle)\n y_rot = x * -np.sin(angle) + y * np.cos(angle)\n return x_rot + x_offset, y_rot + y_offset", "response": "Rotates the given texture by a given angle."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fit_texture(layer):\n x, y = layer\n x = (x - np.nanmin(x)) / (np.nanmax(x) - np.nanmin(x))\n y = (y - np.nanmin(y)) / (np.nanmax(y) - np.nanmin(y))\n return x, y", "response": "Fits a layer into a texture by scaling each axis to 0 1."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_valid_solution(solution, graph):\n expected = Counter(\n i for (i, _) in graph.iter_starts_with_index()\n if i < graph.get_disjoint(i)\n )\n actual = Counter(\n min(i, graph.get_disjoint(i))\n for i in solution\n )\n\n difference = Counter(expected)\n difference.subtract(actual)\n difference = {k: v for k, v in difference.items() if v != 0}\n if difference:\n print('Solution is not valid!'\n 'Difference in node counts (expected - actual): {}'.format(difference))\n return False\n return True", "response": "Check that the solution is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a solution into a list of paths suitable for rendering.", "response": "def get_route_from_solution(solution, graph):\n \"\"\"Converts a solution (a list of node indices) into a list\n of paths suitable for rendering.\"\"\"\n\n # As a guard against comparing invalid \"solutions\",\n # ensure that this solution is valid.\n assert check_valid_solution(solution, graph)\n\n return [graph.get_path(i) for i in solution]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef branching_turtle_generator(turtle_program, turn_amount=DEFAULT_TURN,\n initial_angle=DEFAULT_INITIAL_ANGLE, resolution=1):\n \"\"\"Given a turtle program, creates a generator of turtle positions.\n \n The state of the turtle consists of its position and angle.\n The turtle starts at the position ``(0, 0)`` facing up. Each character in the\n turtle program is processed in order and causes an update to the state.\n The position component of the state is yielded at each state change. A\n ``(nan, nan)`` separator is emitted between state changes for which no line\n should be drawn.\n\n The turtle program consists of the following commands:\n\n - Any letter in ``ABCDEFGHIJ`` means \"move forward one unit and draw a path\"\n - Any letter in ``abcdefghij`` means \"move forward\" (no path)\n - The character ``-`` means \"move counter-clockwise\"\n - The character ``+`` means \"move clockwise\"\n - The character ``[`` means \"push a copy of the current state to the stack\"\n - The character ``]`` means \"pop a state from the stack and return there\"\n - All other characters are silently ignored (this is useful when producing\n programs with L-Systems)\n \n Args:\n turtle_program (str): a string or generator representing the turtle program\n turn_amount (float): how much the turn commands should change the angle\n initial_angle (float): if provided, the turtle starts at this angle (degrees)\n resolution (int): if provided, interpolate this many points along each visible\n line\n\n Yields:\n pair: The next coordinate pair, or ``(nan, nan)`` as a path separator.\n \"\"\"\n saved_states = list()\n state = (0, 0, DEFAULT_INITIAL_ANGLE)\n yield (0, 0)\n\n for command in turtle_program:\n x, y, angle = state\n\n if command in FORWARD_COMMANDS:\n new_x = x - np.cos(np.radians(angle))\n new_y = y + np.sin(np.radians(angle))\n state = (new_x, new_y, angle)\n\n if command not in VISIBLE_FORWARD_COMMANDS:\n yield (np.nan, np.nan)\n yield (state[0], state[1])\n else:\n dx = new_x - x\n dy = new_y - y\n for frac in (1 - np.flipud(np.linspace(0, 1, resolution, False))):\n yield (x + frac * dx, y + frac * dy)\n\n elif command == CW_TURN_COMMAND:\n state = (x, y, angle + turn_amount)\n\n elif command == CCW_TURN_COMMAND:\n state = (x, y, angle - turn_amount)\n\n elif command == PUSH_STATE_COMMAND:\n saved_states.append(state)\n\n elif command == POP_STATE_COMMAND:\n state = saved_states.pop()\n yield (np.nan, np.nan)\n x, y, _ = state\n yield (x, y)", "response": "Generates a generator of turtle positions and lines that are branched at each location."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmaking a texture from a turtle program.", "response": "def turtle_to_texture(turtle_program, turn_amount=DEFAULT_TURN,\n initial_angle=DEFAULT_INITIAL_ANGLE, resolution=1):\n \"\"\"Makes a texture from a turtle program.\n\n Args:\n turtle_program (str): a string representing the turtle program; see the\n docstring of `branching_turtle_generator` for more details\n turn_amount (float): amount to turn in degrees\n initial_angle (float): initial orientation of the turtle\n resolution (int): if provided, interpolation amount for visible lines\n\n Returns:\n texture: A texture.\n \"\"\"\n generator = branching_turtle_generator(\n turtle_program, turn_amount, initial_angle, resolution)\n return texture_from_generator(generator)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform_sequence(sequence, transformations):\n for c in sequence:\n for k in transformations.get(c, c):\n yield k", "response": "Applies a given set of substitution rules to the given string or generator."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchaining a transformation a given number of times.", "response": "def transform_multiple(sequence, transformations, iterations):\n \"\"\"Chains a transformation a given number of times.\n\n Args:\n sequence (str): a string or generator onto which transformations are applied\n transformations (dict): a dictionary mapping each char to the string that is\n substituted for it when the rule is applied\n iterations (int): how many times to repeat the transformation\n\n Yields:\n str: the next character in the output sequence.\n \"\"\"\n for _ in range(iterations):\n sequence = transform_sequence(sequence, transformations)\n return sequence"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a texture by running transformations on a turtle program that is a turtle system.", "response": "def l_system(axiom, transformations, iterations=1, angle=45, resolution=1):\n \"\"\"Generates a texture by running transformations on a turtle program.\n\n First, the given transformations are applied to the axiom. This is\n repeated `iterations` times. Then, the output is run as a turtle\n program to get a texture, which is returned.\n\n For more background see: https://en.wikipedia.org/wiki/L-system\n\n Args:\n axiom (str): the axiom of the Lindenmeyer system (a string)\n transformations (dict): a dictionary mapping each char to the string that is\n substituted for it when the rule is applied\n iterations (int): the number of times to apply the transformations\n angle (float): the angle to use for turns when interpreting the string\n as a turtle graphics program\n resolution (int): the number of midpoints to create in each turtle step\n\n Returns:\n A texture\n \"\"\"\n turtle_program = transform_multiple(axiom, transformations, iterations)\n return turtle_to_texture(turtle_program, angle, resolution=resolution)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_path(self, i):\n index = (i - 1) // 2\n reverse = (i - 1) % 2\n path = self.paths[index]\n if reverse:\n return path.reversed()\n else:\n return path", "response": "Returns the path corresponding to the node i."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the distance between the end of path i and the start of path j.", "response": "def cost(self, i, j):\n \"\"\"Returns the distance between the end of path i\n and the start of path j.\"\"\"\n return dist(self.endpoints[i][1], self.endpoints[j][0])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the starting coordinates of node i as a pair and the end coordinates iff end is True.", "response": "def get_coordinates(self, i, end=False):\n \"\"\"Returns the starting coordinates of node i as a pair,\n or the end coordinates iff end is True.\"\"\"\n if end:\n endpoint = self.endpoints[i][1]\n else:\n endpoint = self.endpoints[i][0]\n return (endpoint.real, endpoint.imag)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef iter_starts_with_index(self):\n for i in range(1, len(self.endpoints)):\n yield i, self.get_coordinates(i)", "response": "Returns a generator over ( index start coordinate ) pairs and the original location of the resource."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef iter_disjunctions(self):\n for i in range(1, len(self.endpoints), 2):\n yield [i, self.get_disjoint(i)]", "response": "Returns an iterator over the disjoints of the cluster - level entries."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\npreviewing a plot in a jupyter notebook.", "response": "def show_plot(plot, width=PREVIEW_WIDTH, height=PREVIEW_HEIGHT):\n \"\"\"Preview a plot in a jupyter notebook.\n\n Args:\n plot (list): the plot to display (list of layers)\n width (int): the width of the preview\n height (int): the height of the preview\n \n Returns:\n An object that renders in Jupyter as the provided plot\n \"\"\"\n return SVG(data=plot_to_svg(plot, width, height))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef calculate_view_box(layers, aspect_ratio, margin=DEFAULT_VIEW_BOX_MARGIN):\n min_x = min(np.nanmin(x) for x, y in layers)\n max_x = max(np.nanmax(x) for x, y in layers)\n min_y = min(np.nanmin(y) for x, y in layers)\n max_y = max(np.nanmax(y) for x, y in layers)\n height = max_y - min_y\n width = max_x - min_x\n\n if height > width * aspect_ratio:\n adj_height = height * (1. + margin)\n adj_width = adj_height / aspect_ratio\n else:\n adj_width = width * (1. + margin)\n adj_height = adj_width * aspect_ratio\n\n width_buffer = (adj_width - width) / 2.\n height_buffer = (adj_height - height) / 2.\n\n return (\n min_x - width_buffer,\n min_y - height_buffer,\n adj_width,\n adj_height\n )", "response": "Calculates the size of the SVG viewBox to use."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating an SVG path from a given layer.", "response": "def _layer_to_path_gen(layer):\n \"\"\"Generates an SVG path from a given layer.\n\n Args:\n layer (layer): the layer to convert\n\n Yields:\n str: the next component of the path\n \"\"\"\n draw = False\n for x, y in zip(*layer):\n if np.isnan(x) or np.isnan(y):\n draw = False\n elif not draw:\n yield 'M {} {}'.format(x, y)\n draw = True\n else:\n yield 'L {} {}'.format(x, y)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a list of layers into an SVG document.", "response": "def plot_to_svg(plot, width, height, unit=''):\n \"\"\"Converts a plot (list of layers) into an SVG document.\n\n Args:\n plot (list): list of layers that make up the plot\n width (float): the width of the resulting image\n height (float): the height of the resulting image\n unit (str): the units of the resulting image if not pixels\n\n Returns:\n str: A stringified XML document representing the image\n \"\"\"\n flipped_plot = [(x, -y) for x, y in plot]\n aspect_ratio = height / width\n view_box = calculate_view_box(flipped_plot, aspect_ratio=aspect_ratio)\n view_box_str = '{} {} {} {}'.format(*view_box)\n stroke_thickness = STROKE_THICKNESS * (view_box[2])\n\n svg = ET.Element('svg', attrib={\n 'xmlns': 'http://www.w3.org/2000/svg',\n 'xmlns:inkscape': 'http://www.inkscape.org/namespaces/inkscape',\n 'width': '{}{}'.format(width, unit),\n 'height': '{}{}'.format(height, unit),\n 'viewBox': view_box_str})\n\n for i, layer in enumerate(flipped_plot):\n group = ET.SubElement(svg, 'g', attrib={\n 'inkscape:label': '{}-layer'.format(i),\n 'inkscape:groupmode': 'layer',\n })\n\n color = PLOT_COLORS[i % len(PLOT_COLORS)]\n ET.SubElement(group, 'path', attrib={\n 'style': 'stroke-width: {}; stroke: {};'.format(stroke_thickness, color),\n 'fill': 'none',\n 'd': layer_to_path(layer)\n })\n\n try:\n return ET.tostring(svg, encoding='unicode')\n except LookupError:\n # Python 2.x\n return ET.tostring(svg)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_plot(plot, filename, width=DEFAULT_PAGE_WIDTH, height=DEFAULT_PAGE_HEIGHT, unit=DEFAULT_PAGE_UNIT):\n svg = plot_to_svg(plot, width, height, unit)\n with open(filename, 'w') as outfile:\n outfile.write(svg)", "response": "Writes a plot SVG to a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndraw a layer on the given matplotlib axis.", "response": "def draw_layer(ax, layer):\n \"\"\"Draws a layer on the given matplotlib axis.\n\n Args:\n ax (axis): the matplotlib axis to draw on\n layer (layer): the layers to plot\n \"\"\"\n ax.set_aspect('equal', 'datalim')\n ax.plot(*layer)\n ax.axis('off')"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef map_texture_to_surface(texture, surface):\n texture_x, texture_y = texture\n surface_h, surface_w = surface.shape\n\n surface_x = np.clip(\n np.int32(surface_w * texture_x - 1e-9), 0, surface_w - 1)\n surface_y = np.clip(\n np.int32(surface_h * texture_y - 1e-9), 0, surface_h - 1)\n\n surface_z = surface[surface_y, surface_x]\n return surface_z", "response": "Returns the values on a surface for each point in the specified texture."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new texture by adding z - values to an existing texture and projecting it.", "response": "def project_texture(texture_xy, texture_z, angle=DEFAULT_ANGLE):\n \"\"\"Creates a texture by adding z-values to an existing texture and projecting.\n\n When working with surfaces there are two ways to accomplish the same thing:\n\n 1. project the surface and map a texture to the projected surface\n 2. map a texture to the surface, and then project the result\n\n The first method, which does not use this function, is preferred because\n it is easier to do occlusion removal that way. This function is provided\n for cases where you do not wish to generate a surface (and don't care about\n occlusion removal.)\n\n Args:\n texture_xy (texture): the texture to project\n texture_z (np.array): the Z-values to use in the projection\n angle (float): the angle to project at, in degrees (0 = overhead, 90 = side view)\n\n Returns:\n layer: A layer.\n \"\"\"\n z_coef = np.sin(np.radians(angle))\n y_coef = np.cos(np.radians(angle))\n surface_x, surface_y = texture\n return (surface_x, -surface_y * y_coef + surface_z * z_coef)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the height of the surface when projected at the given angle.", "response": "def project_surface(surface, angle=DEFAULT_ANGLE):\n \"\"\"Returns the height of the surface when projected at the given angle.\n\n Args:\n surface (surface): the surface to project\n angle (float): the angle at which to project the surface\n\n Returns:\n surface: A projected surface.\n \"\"\"\n z_coef = np.sin(np.radians(angle))\n y_coef = np.cos(np.radians(angle))\n\n surface_height, surface_width = surface.shape\n slope = np.tile(np.linspace(0., 1., surface_height), [surface_width, 1]).T\n\n return slope * y_coef + surface * z_coef"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmap a texture onto a surface then projects it onto a 2D layer.", "response": "def project_texture_on_surface(texture, surface, angle=DEFAULT_ANGLE):\n \"\"\"Maps a texture onto a surface, then projects to 2D and returns a layer.\n\n Args:\n texture (texture): the texture to project\n surface (surface): the surface to project onto\n angle (float): the projection angle in degrees (0 = top-down, 90 = side view)\n\n Returns:\n layer: A layer.\n \"\"\"\n projected_surface = project_surface(surface, angle)\n texture_x, _ = texture\n texture_y = map_texture_to_surface(texture, projected_surface)\n return texture_x, texture_y"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove hidden parts from a projected surface.", "response": "def _remove_hidden_parts(projected_surface):\n \"\"\"Removes parts of a projected surface that are not visible.\n\n Args:\n projected_surface (surface): the surface to use\n\n Returns:\n surface: A projected surface.\n \"\"\"\n surface = np.copy(projected_surface)\n surface[~_make_occlusion_mask(projected_surface)] = np.nan\n return surface"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprojects a texture onto a surface with occluded areas removed.", "response": "def project_and_occlude_texture(texture, surface, angle=DEFAULT_ANGLE):\n \"\"\"Projects a texture onto a surface with occluded areas removed.\n\n Args:\n texture (texture): the texture to map to the projected surface\n surface (surface): the surface to project\n angle (float): the angle to project at, in degrees (0 = overhead, 90 = side view)\n\n Returns:\n layer: A layer.\n \"\"\"\n projected_surface = project_surface(surface, angle)\n projected_surface = _remove_hidden_parts(projected_surface)\n texture_y = map_texture_to_surface(texture, projected_surface)\n texture_x, _ = texture\n return texture_x, texture_y"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmakes a texture consisting of a given number of horizontal lines.", "response": "def make_lines_texture(num_lines=10, resolution=50):\n \"\"\"Makes a texture consisting of a given number of horizontal lines.\n\n Args:\n num_lines (int): the number of lines to draw\n resolution (int): the number of midpoints on each line\n\n Returns:\n A texture.\n \"\"\"\n x, y = np.meshgrid(\n np.hstack([np.linspace(0, 1, resolution), np.nan]),\n np.linspace(0, 1, num_lines),\n )\n \n y[np.isnan(x)] = np.nan\n return x.flatten(), y.flatten()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a texture consisting of a grid of horizontal and vertical lines.", "response": "def make_grid_texture(num_h_lines=10, num_v_lines=10, resolution=50):\n \"\"\"Makes a texture consisting of a grid of vertical and horizontal lines.\n\n Args:\n num_h_lines (int): the number of horizontal lines to draw\n num_v_lines (int): the number of vertical lines to draw\n resolution (int): the number of midpoints to draw on each line\n\n Returns:\n A texture.\n \"\"\"\n x_h, y_h = make_lines_texture(num_h_lines, resolution)\n y_v, x_v = make_lines_texture(num_v_lines, resolution)\n return np.concatenate([x_h, x_v]), np.concatenate([y_h, y_v])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a texture consisting of a spiral from the origin.", "response": "def make_spiral_texture(spirals=6.0, ccw=False, offset=0.0, resolution=1000):\n \"\"\"Makes a texture consisting of a spiral from the origin.\n\n Args:\n spirals (float): the number of rotations to make\n ccw (bool): make spirals counter-clockwise (default is clockwise)\n offset (float): if non-zero, spirals start offset by this amount\n resolution (int): number of midpoints along the spiral\n\n Returns:\n A texture.\n \"\"\"\n dist = np.sqrt(np.linspace(0., 1., resolution))\n if ccw:\n direction = 1.\n else:\n direction = -1.\n angle = dist * spirals * np.pi * 2. * direction\n spiral_texture = (\n (np.cos(angle) * dist / 2.) + 0.5,\n (np.sin(angle) * dist / 2.) + 0.5\n )\n return spiral_texture"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_hex_texture(grid_size = 2, resolution=1):\n grid_x, grid_y = np.meshgrid(\n np.arange(grid_size),\n np.arange(grid_size)\n )\n ROOT_3_OVER_2 = np.sqrt(3) / 2\n ONE_HALF = 0.5\n \n grid_x = (grid_x * np.sqrt(3) + (grid_y % 2) * ROOT_3_OVER_2).flatten()\n grid_y = grid_y.flatten() * 1.5\n \n grid_points = grid_x.shape[0]\n \n x_offsets = np.interp(np.arange(4 * resolution),\n np.arange(4) * resolution, [\n ROOT_3_OVER_2,\n 0.,\n -ROOT_3_OVER_2,\n -ROOT_3_OVER_2,\n ])\n y_offsets = np.interp(np.arange(4 * resolution),\n np.arange(4) * resolution, [\n -ONE_HALF,\n -1.,\n -ONE_HALF,\n ONE_HALF\n ])\n \n tmx = 4 * resolution\n x_t = np.tile(grid_x, (tmx, 1)) + x_offsets.reshape((tmx, 1))\n y_t = np.tile(grid_y, (tmx, 1)) + y_offsets.reshape((tmx, 1))\n \n x_t = np.vstack([x_t, np.tile(np.nan, (1, grid_x.size))])\n y_t = np.vstack([y_t, np.tile(np.nan, (1, grid_y.size))])\n \n return fit_texture((x_t.flatten('F'), y_t.flatten('F')))", "response": "Makes a texture consisting on a grid of hexagons."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_noise_surface(dims=DEFAULT_DIMS, blur=10, seed=None):\n if seed is not None:\n np.random.seed(seed)\n\n return gaussian_filter(np.random.normal(size=dims), blur)", "response": "Makes a random noise surface"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_gradients(dims=DEFAULT_DIMS):\n return np.meshgrid(\n np.linspace(0.0, 1.0, dims[0]),\n np.linspace(0.0, 1.0, dims[1])\n )", "response": "Makes a pair of gradients to generate textures from numpy primitives."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_sine_surface(dims=DEFAULT_DIMS, offset=0.5, scale=1.0):\n gradients = (np.array(make_gradients(dims)) - offset) * scale * np.pi\n return np.sin(np.linalg.norm(gradients, axis=0))", "response": "Makes a surface from the 3D sine function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_bubble_surface(dims=DEFAULT_DIMS, repeat=3):\n gradients = make_gradients(dims)\n return (\n np.sin((gradients[0] - 0.5) * repeat * np.pi) *\n np.sin((gradients[1] - 0.5) * repeat * np.pi))", "response": "Makes a surface from the product of sine functions on each axis."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef vrp_solver(path_graph, initial_solution=None, runtime_seconds=60):\n # Create the VRP routing model. The 1 means we are only looking\n # for a single path.\n routing = pywrapcp.RoutingModel(path_graph.num_nodes(),\n 1, path_graph.ORIGIN)\n\n # For every path node, add a disjunction so that we do not also\n # draw its reverse.\n for disjunction in path_graph.iter_disjunctions():\n routing.AddDisjunction(disjunction)\n\n # Wrap the distance function so that it converts to an integer,\n # as or-tools requires. Values are multiplied by COST_MULTIPLIER\n # prior to conversion to reduce the loss of precision.\n COST_MULTIPLIER = 1e4\n\n def distance(i, j):\n return int(path_graph.cost(i, j) * COST_MULTIPLIER)\n routing.SetArcCostEvaluatorOfAllVehicles(distance)\n\n start_time = time()\n\n def found_solution():\n t = time() - start_time\n cost = routing.CostVar().Max() / COST_MULTIPLIER\n print('\\rBest solution at {} seconds has cost {} '.format(\n int(t), cost), end='')\n routing.AddAtSolutionCallback(found_solution)\n\n # If we weren't supplied with a solution initially, construct one by taking\n # all of the paths in their original direction, in their original order.\n if not initial_solution:\n initial_solution = [i for i, _ in path_graph.iter_disjunctions()]\n\n # Compute the cost of the initial solution. This is the number we hope to\n # improve on.\n initial_assignment = routing.ReadAssignmentFromRoutes([initial_solution],\n True)\n # print('Initial distance:',\n # initial_assignment.ObjectiveValue() / COST_MULTIPLIER)\n\n # Set the parameters of the search.\n search_parameters = pywrapcp.RoutingModel.DefaultSearchParameters()\n search_parameters.time_limit_ms = runtime_seconds * 1000\n search_parameters.local_search_metaheuristic = (\n routing_enums_pb2.LocalSearchMetaheuristic.GUIDED_LOCAL_SEARCH)\n\n # Run the optimizer and report the final distance.\n assignment = routing.SolveFromAssignmentWithParameters(initial_assignment,\n search_parameters)\n print()\n #print('Final distance:', assignment.ObjectiveValue() / COST_MULTIPLIER)\n\n # Iterate over the result to produce a list to return as the solution.\n solution = []\n index = routing.Start(0)\n while not routing.IsEnd(index):\n index = assignment.Value(routing.NextVar(index))\n node = routing.IndexToNode(index)\n if node != 0:\n # For compatibility with the greedy solution, exclude the origin.\n solution.append(node)\n return solution", "response": "Solve a path using or - tools Vehicle Routing Problem solver."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef init_controller(url):\n # pylint: disable=global-statement\n global _VERA_CONTROLLER\n created = False\n if _VERA_CONTROLLER is None:\n _VERA_CONTROLLER = VeraController(url)\n created = True\n _VERA_CONTROLLER.start()\n return [_VERA_CONTROLLER, created]", "response": "Initialize a controller.\n\n Provides a single global controller for applications that can't do this\n themselves"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform a data_request and return the result.", "response": "def data_request(self, payload, timeout=TIMEOUT):\n \"\"\"Perform a data_request and return the result.\"\"\"\n request_url = self.base_url + \"/data_request\"\n return requests.get(request_url, timeout=timeout, params=payload)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_simple_devices_info(self):\n j = self.data_request({'id': 'sdata'}).json()\n\n self.scenes = []\n items = j.get('scenes')\n\n for item in items:\n self.scenes.append(VeraScene(item, self))\n\n if j.get('temperature'):\n self.temperature_units = j.get('temperature')\n\n self.categories = {}\n cats = j.get('categories')\n\n for cat in cats:\n self.categories[cat.get('id')] = cat.get('name')\n\n self.device_id_map = {}\n\n devs = j.get('devices')\n for dev in devs:\n dev['categoryName'] = self.categories.get(dev.get('category'))\n self.device_id_map[dev.get('id')] = dev", "response": "Get basic device info from Vera."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsearch the list of connected devices by name.", "response": "def get_device_by_name(self, device_name):\n \"\"\"Search the list of connected devices by name.\n\n device_name param is the string name of the device\n \"\"\"\n\n # Find the device for the vera device name we are interested in\n found_device = None\n for device in self.get_devices():\n if device.name == device_name:\n found_device = device\n # found the first (and should be only) one so we will finish\n break\n\n if found_device is None:\n logger.debug('Did not find device with {}'.format(device_name))\n\n return found_device"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch the list of connected devices by ID.", "response": "def get_device_by_id(self, device_id):\n \"\"\"Search the list of connected devices by ID.\n\n device_id param is the integer ID of the device\n \"\"\"\n\n # Find the device for the vera device name we are interested in\n found_device = None\n for device in self.get_devices():\n if device.device_id == device_id:\n found_device = device\n # found the first (and should be only) one so we will finish\n break\n\n if found_device is None:\n logger.debug('Did not find device with {}'.format(device_id))\n\n return found_device"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget list of connected devices.", "response": "def get_devices(self, category_filter=''):\n \"\"\"Get list of connected devices.\n\n category_filter param is an array of strings\n \"\"\"\n # pylint: disable=too-many-branches\n\n # the Vera rest API is a bit rough so we need to make 2 calls to get\n # all the info e need\n self.get_simple_devices_info()\n\n j = self.data_request({'id': 'status', 'output_format': 'json'}).json()\n\n self.devices = []\n items = j.get('devices')\n\n for item in items:\n item['deviceInfo'] = self.device_id_map.get(item.get('id'))\n if item.get('deviceInfo'):\n device_category = item.get('deviceInfo').get('category')\n if device_category == CATEGORY_DIMMER:\n device = VeraDimmer(item, self)\n elif ( device_category == CATEGORY_SWITCH or\n device_category == CATEGORY_VERA_SIREN):\n device = VeraSwitch(item, self)\n elif device_category == CATEGORY_THERMOSTAT:\n device = VeraThermostat(item, self)\n elif device_category == CATEGORY_LOCK:\n device = VeraLock(item, self)\n elif device_category == CATEGORY_CURTAIN:\n device = VeraCurtain(item, self)\n elif device_category == CATEGORY_ARMABLE:\n device = VeraBinarySensor(item, self)\n elif (device_category == CATEGORY_SENSOR or\n device_category == CATEGORY_HUMIDITY_SENSOR or\n device_category == CATEGORY_TEMPERATURE_SENSOR or\n device_category == CATEGORY_LIGHT_SENSOR or\n device_category == CATEGORY_POWER_METER or\n device_category == CATEGORY_UV_SENSOR):\n device = VeraSensor(item, self)\n elif (device_category == CATEGORY_SCENE_CONTROLLER or\n device_category == CATEGORY_REMOTE):\n device = VeraSceneController(item, self)\n elif device_category == CATEGORY_GARAGE_DOOR:\n device = VeraGarageDoor(item, self)\n else:\n device = VeraDevice(item, self)\n self.devices.append(device)\n if (device.is_armable and not (\n device_category == CATEGORY_SWITCH or\n device_category == CATEGORY_VERA_SIREN or\n device_category == CATEGORY_CURTAIN or\n device_category == CATEGORY_GARAGE_DOOR)):\n self.devices.append(VeraArmableDevice(item, self))\n else:\n self.devices.append(VeraDevice(item, self))\n\n if not category_filter:\n return self.devices\n\n devices = []\n for device in self.devices:\n if (device.category_name is not None and\n device.category_name != '' and\n device.category_name in category_filter):\n devices.append(device)\n return devices"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef refresh_data(self):\n j = self.data_request({'id': 'sdata'}).json()\n\n self.temperature_units = j.get('temperature', 'C')\n self.model = j.get('model')\n self.version = j.get('version')\n self.serial_number = j.get('serial_number')\n\n categories = {}\n cats = j.get('categories')\n\n for cat in cats:\n categories[cat.get('id')] = cat.get('name')\n\n device_id_map = {}\n\n devs = j.get('devices')\n for dev in devs:\n dev['categoryName'] = categories.get(dev.get('category'))\n device_id_map[dev.get('id')] = dev\n\n return device_id_map", "response": "Refresh data from Vera device."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef map_services(self):\n # the Vera rest API is a bit rough so we need to make 2 calls\n # to get all the info e need\n self.get_simple_devices_info()\n\n j = self.data_request({'id': 'status', 'output_format': 'json'}).json()\n\n service_map = {}\n\n items = j.get('devices')\n\n for item in items:\n service_map[item.get('id')] = item.get('states')\n\n self.device_services_map = service_map", "response": "Get full Vera device service info."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget data since last timestamp.", "response": "def get_changed_devices(self, timestamp):\n \"\"\"Get data since last timestamp.\n\n This is done via a blocking call, pass NONE for initial state.\n \"\"\"\n if timestamp is None:\n payload = {}\n else:\n payload = {\n 'timeout': SUBSCRIPTION_WAIT,\n 'minimumdelay': SUBSCRIPTION_MIN_WAIT\n }\n payload.update(timestamp)\n # double the timeout here so requests doesn't timeout before vera\n payload.update({\n 'id': 'lu_sdata',\n })\n\n logger.debug(\"get_changed_devices() requesting payload %s\", str(payload))\n r = self.data_request(payload, TIMEOUT*2)\n r.raise_for_status()\n\n # If the Vera disconnects before writing a full response (as lu_sdata\n # will do when interrupted by a Luup reload), the requests module will\n # happily return 200 with an empty string. So, test for empty response,\n # so we don't rely on the JSON parser to throw an exception.\n if r.text == \"\":\n raise PyveraError(\"Empty response from Vera\")\n\n # Catch a wide swath of what the JSON parser might throw, within\n # reason. Unfortunately, some parsers don't specifically return\n # json.decode.JSONDecodeError, but so far most seem to derive what\n # they do throw from ValueError, so that's helpful.\n try:\n result = r.json()\n except ValueError as ex:\n raise PyveraError(\"JSON decode error: \" + str(ex))\n\n if not ( type(result) is dict\n and 'loadtime' in result and 'dataversion' in result ):\n raise PyveraError(\"Unexpected/garbled response from Vera\")\n\n # At this point, all good. Update timestamp and return change data.\n device_data = result.get('devices')\n timestamp = {\n 'loadtime': result.get('loadtime'),\n 'dataversion': result.get('dataversion')\n }\n return [device_data, timestamp]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_service_value(self, service_id, set_name, parameter_name, value):\n payload = {\n 'id': 'lu_action',\n 'action': 'Set' + set_name,\n 'serviceId': service_id,\n parameter_name: value\n }\n result = self.vera_request(**payload)\n logger.debug(\"set_service_value: \"\n \"result of vera_request with payload %s: %s\",\n payload, result.text)", "response": "Set a variable on the vera device."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncall a Vera service.", "response": "def call_service(self, service_id, action):\n \"\"\"Call a Vera service.\n\n This will call the Vera api to change device state.\n \"\"\"\n result = self.vera_request(id='action', serviceId=service_id,\n action=action)\n logger.debug(\"call_service: \"\n \"result of vera_request with id %s: %s\", service_id,\n result.text)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_cache_value(self, name, value):\n dev_info = self.json_state.get('deviceInfo')\n if dev_info.get(name.lower()) is None:\n logger.error(\"Could not set %s for %s (key does not exist).\",\n name, self.name)\n logger.error(\"- dictionary %s\", dev_info)\n return\n dev_info[name.lower()] = str(value)", "response": "Set a variable in the local state dictionary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset a variable in the local complex state dictionary.", "response": "def set_cache_complex_value(self, name, value):\n \"\"\"Set a variable in the local complex state dictionary.\n\n This does not change the physical device. Useful if you want the\n device state to refect a new value which has not yet updated from\n Vera.\n \"\"\"\n for item in self.json_state.get('states'):\n if item.get('variable') == name:\n item['value'] = str(value)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a value from the service dictionaries.", "response": "def get_complex_value(self, name):\n \"\"\"Get a value from the service dictionaries.\n\n It's best to use get_value if it has the data you require since\n the vera subscription only updates data in dev_info.\n \"\"\"\n for item in self.json_state.get('states'):\n if item.get('variable') == name:\n return item.get('value')\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting a case - sensitive key value from the dev_info area.", "response": "def get_strict_value(self, name):\n \"\"\"Get a case-sensitive keys value from the dev_info area.\n \"\"\"\n dev_info = self.json_state.get('deviceInfo')\n return dev_info.get(name, None)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrefreshing a value from the service dictionaries.", "response": "def refresh_complex_value(self, name):\n \"\"\"Refresh a value from the service dictionaries.\n\n It's best to use get_value / refresh if it has the data you need.\n \"\"\"\n for item in self.json_state.get('states'):\n if item.get('variable') == name:\n service_id = item.get('service')\n result = self.vera_request(**{\n 'id': 'variableget',\n 'output_format': 'json',\n 'DeviceNum': self.device_id,\n 'serviceId': service_id,\n 'Variable': name\n })\n item['value'] = result.text\n return item.get('value')\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrefreshing the dev_info data used by get_value.", "response": "def refresh(self):\n \"\"\"Refresh the dev_info data used by get_value.\n\n Only needed if you're not using subscriptions.\n \"\"\"\n j = self.vera_request(id='sdata', output_format='json').json()\n devices = j.get('devices')\n for device_data in devices:\n if device_data.get('id') == self.device_id:\n self.update(device_data)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the dev_info data from a dictionary.", "response": "def update(self, params):\n \"\"\"Update the dev_info data from a dictionary.\n\n Only updates if it already exists in the device.\n \"\"\"\n dev_info = self.json_state.get('deviceInfo')\n dev_info.update({k: params[k] for k in params if dev_info.get(k)})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget level from vera.", "response": "def level(self):\n \"\"\"Get level from vera.\"\"\"\n # Used for dimmers, curtains\n # Have seen formats of 10, 0.0 and \"0%\"!\n level = self.get_value('level')\n try:\n return int(float(level))\n except (TypeError, ValueError):\n pass\n try:\n return int(level.strip('%'))\n except (TypeError, AttributeError, ValueError):\n pass\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the switch state also update local state.", "response": "def set_switch_state(self, state):\n \"\"\"Set the switch state, also update local state.\"\"\"\n self.set_service_value(\n self.switch_service,\n 'Target',\n 'newTargetValue',\n state)\n self.set_cache_value('Status', state)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_switched_on(self, refresh=False):\n if refresh:\n self.refresh()\n return self.get_brightness(refresh) > 0", "response": "Check if dimmer is switched on."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting dimmer brightness from Vera level property.", "response": "def get_brightness(self, refresh=False):\n \"\"\"Get dimmer brightness.\n\n Refresh data from Vera if refresh is True, otherwise use local cache.\n Refresh is only needed if you're not using subscriptions.\n Converts the Vera level property for dimmable lights from a percentage\n to the 0 - 255 scale used by HA.\n \"\"\"\n if refresh:\n self.refresh()\n brightness = 0\n percent = self.level\n if percent > 0:\n brightness = round(percent * 2.55)\n return int(brightness)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets dimmer brightness. Converts the Vera level property for dimmable lights from a percentage to the 0 - 255 scale used by HA.", "response": "def set_brightness(self, brightness):\n \"\"\"Set dimmer brightness.\n\n Converts the Vera level property for dimmable lights from a percentage\n to the 0 - 255 scale used by HA.\n \"\"\"\n percent = 0\n if brightness > 0:\n percent = round(brightness / 2.55)\n\n self.set_service_value(\n self.dimmer_service,\n 'LoadLevelTarget',\n 'newLoadlevelTarget',\n percent)\n self.set_cache_value('level', percent)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_color_index(self, colors, refresh=False):\n if refresh:\n self.refresh_complex_value('SupportedColors')\n\n sup = self.get_complex_value('SupportedColors')\n if sup is None:\n return None\n\n sup = sup.split(',')\n if not set(colors).issubset(sup):\n return None\n\n return [sup.index(c) for c in colors]", "response": "Get color index.\n\n Refresh data from Vera if refresh is True, otherwise use local cache."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_color(self, refresh=False):\n if refresh:\n self.refresh_complex_value('CurrentColor')\n\n ci = self.get_color_index(['R', 'G', 'B'], refresh)\n cur = self.get_complex_value('CurrentColor')\n if ci is None or cur is None:\n return None\n\n try:\n val = [cur.split(',')[c] for c in ci]\n return [int(v.split('=')[1]) for v in val]\n except IndexError:\n return None", "response": "Get color.\n\n Refresh data from Vera if refresh is True, otherwise use local cache."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_armed_state(self, state):\n self.set_service_value(\n self.security_sensor_service,\n 'Armed',\n 'newArmedValue',\n state)\n self.set_cache_value('Armed', state)", "response": "Set the armed state also update local state."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_switched_on(self, refresh=False):\n if refresh:\n self.refresh()\n val = self.get_value('Armed')\n return val == '1'", "response": "Check if the user is armed on."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if the current state of the current instance is open.", "response": "def is_open(self, refresh=False):\n \"\"\"Get curtains state.\n\n Refresh data from Vera if refresh is True, otherwise use local cache.\n Refresh is only needed if you're not using subscriptions.\n \"\"\"\n if refresh:\n self.refresh()\n return self.get_level(refresh) > 0"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the open level of the curtains.", "response": "def set_level(self, level):\n \"\"\"Set open level of the curtains.\n\n Scale is 0-100\n \"\"\"\n self.set_service_value(\n self.dimmer_service,\n 'LoadLevelTarget',\n 'newLoadlevelTarget',\n level)\n\n self.set_cache_value('level', level)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the last used PIN user id", "response": "def get_last_user(self, refresh=False):\n \"\"\"Get the last used PIN user id\"\"\"\n if refresh:\n self.refresh_complex_value('sl_UserCode')\n val = self.get_complex_value(\"sl_UserCode\")\n # Syntax string: UserID=\"\" UserName=\"\"\n # See http://wiki.micasaverde.com/index.php/Luup_UPnP_Variables_and_Actions#DoorLock1\n\n try:\n # Get the UserID=\"\" and UserName=\"\" fields separately\n raw_userid, raw_username = val.split(' ')\n # Get the right hand value without quotes of UserID=\"\"\n userid = raw_userid.split('=')[1].split('\"')[1]\n # Get the right hand value without quotes of UserName=\"\"\n username = raw_username.split('=')[1].split('\"')[1]\n except Exception as ex:\n logger.error('Got unsupported user string {}: {}'.format(val, ex))\n return None\n\n return ( userid, username )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_pin_codes(self, refresh=False):\n if refresh:\n self.refresh()\n val = self.get_value(\"pincodes\")\n\n # val syntax string: next_available_user_code_id\\tuser_code_id,active,date_added,date_used,PIN_code,name;\\t...\n # See (outdated) http://wiki.micasaverde.com/index.php/Luup_UPnP_Variables_and_Actions#DoorLock1\n\n # Remove the trailing tab\n # ignore the version and next available at the start\n # and split out each set of code attributes\n raw_code_list = []\n try:\n raw_code_list = val.rstrip().split('\\t')[1:]\n except Exception as ex:\n logger.error('Got unsupported string {}: {}'.format(val, ex))\n\n # Loop to create a list of codes\n codes = []\n for code in raw_code_list:\n\n try:\n # Strip off trailing semicolon\n # Create a list from csv\n code_addrs = code.split(';')[0].split(',')\n\n # Get the code ID (slot) and see if it should have values\n slot, active = code_addrs[:2]\n if active != '0':\n # Since it has additional attributes, get the remaining ones\n _, _, pin, name = code_addrs[2:]\n # And add them as a tuple to the list\n codes.append((slot, name, pin))\n except Exception as ex:\n logger.error('Problem parsing pin code string {}: {}'.format(code, ex))\n \n return codes", "response": "Get the list of PIN codes that can be found with self. get_complex_value ( PIN_CODES )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_temperature(self, temp):\n\n self.set_service_value(\n self.thermostat_setpoint,\n 'CurrentSetpoint',\n 'NewCurrentSetpoint',\n temp)\n\n self.set_cache_value('setpoint', temp)", "response": "Set current goal temperature"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting current goal temperature", "response": "def get_current_goal_temperature(self, refresh=False):\n \"\"\"Get current goal temperature / setpoint\"\"\"\n if refresh:\n self.refresh()\n try:\n return float(self.get_value('setpoint'))\n except (TypeError, ValueError):\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_hvac_mode(self, mode):\n self.set_service_value(\n self.thermostat_operating_service,\n 'ModeTarget',\n 'NewModeTarget',\n mode)\n self.set_cache_value('mode', mode)", "response": "Set the hvac mode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_fan_mode(self, mode):\n self.set_service_value(\n self.thermostat_fan_service,\n 'Mode',\n 'NewMode',\n mode)\n self.set_cache_value('fanmode', mode)", "response": "Set the fan mode"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_last_scene_id(self, refresh=False):\n if refresh:\n self.refresh_complex_value('LastSceneID')\n self.refresh_complex_value('sl_CentralScene')\n val = self.get_complex_value('LastSceneID') or self.get_complex_value('sl_CentralScene')\n return val", "response": "Get the last scene id from Vera."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_last_scene_time(self, refresh=False):\n if refresh:\n self.refresh_complex_value('LastSceneTime')\n val = self.get_complex_value('LastSceneTime')\n return val", "response": "Get last scene time from Vera"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef vera_request(self, **kwargs):\n request_payload = {\n 'output_format': 'json',\n 'SceneNum': self.scene_id,\n }\n request_payload.update(kwargs)\n\n return self.vera_controller.data_request(request_payload)", "response": "Perfom a vera_request for this scene."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nactivates a Vera scene.", "response": "def activate(self):\n \"\"\"Activate a Vera scene.\n\n This will call the Vera api to activate a scene.\n \"\"\"\n payload = {\n 'id': 'lu_action',\n 'action': 'RunScene',\n 'serviceId': self.scene_service\n }\n result = self.vera_request(**payload)\n logger.debug(\"activate: \"\n \"result of vera_request with payload %s: %s\",\n payload, result.text)\n\n self._active = True"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrefresh the data used by get_value.", "response": "def refresh(self):\n \"\"\"Refresh the data used by get_value.\n\n Only needed if you're not using subscriptions.\n \"\"\"\n j = self.vera_request(id='sdata', output_format='json').json()\n scenes = j.get('scenes')\n for scene_data in scenes:\n if scene_data.get('id') == self.scene_id:\n self.update(scene_data)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nregister a callback for a specific device.", "response": "def register(self, device, callback):\n \"\"\"Register a callback.\n\n device: device to be updated by subscription\n callback: callback for notification of changes\n \"\"\"\n if not device:\n logger.error(\"Received an invalid device: %r\", device)\n return\n\n logger.debug(\"Subscribing to events for %s\", device.name)\n self._devices[device.vera_device_id].append(device)\n self._callbacks[device].append(callback)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving a registered callback.", "response": "def unregister(self, device, callback):\n \"\"\"Remove a registered a callback.\n\n device: device that has the subscription\n callback: callback used in original registration\n \"\"\"\n if not device:\n logger.error(\"Received an invalid device: %r\", device)\n return\n\n logger.debug(\"Removing subscription for {}\".format(device.name))\n self._callbacks[device].remove(callback)\n self._devices[device.vera_device_id].remove(device)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstart a thread to handle Vera blocked polling.", "response": "def start(self):\n \"\"\"Start a thread to handle Vera blocked polling.\"\"\"\n self._poll_thread = threading.Thread(target=self._run_poll_server,\n name='Vera Poll Thread')\n self._poll_thread.deamon = True\n self._poll_thread.start()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format_timestamp(ts):\n tz_info = tz.tzutc()\n return datetime.fromtimestamp(ts, tz=tz_info).strftime(\"%Y-%m-%dT%H:%M:%S.000Z\")", "response": "Format the UTC timestamp for Elasticsearch\n eg. 2014 - 07 - 09T08 : 37.000Z"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _pick_level(cls, btc_amount):\n for size, level in cls.TICKER_LEVEL:\n if btc_amount < size:\n return level\n return cls.TICKER_LEVEL[-1][1]", "response": "Choose the level of the item based on the amount specified."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef format_tsv_line(source, edge, target, value=None, metadata=None):\n return '{source}\\t{edge}\\t{target}\\t{value}\\t{metadata}'.format(\n source=source,\n edge=edge,\n target=target,\n value='{:.4f}'.format(value) if value is not None else '',\n metadata=metadata or ''\n ).rstrip(' \\t')", "response": "Render a single line for TSV file with data flow described by the user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrender a. dot file with graph definition from a given set of data.", "response": "def format_graphviz_lines(lines):\n \"\"\"\n Render a .dot file with graph definition from a given set of data\n\n :type lines list[dict]\n :rtype: str\n \"\"\"\n # first, prepare the unique list of all nodes (sources and targets)\n lines_nodes = set()\n\n for line in lines:\n lines_nodes.add(line['source'])\n lines_nodes.add(line['target'])\n\n # generate a list of all nodes and their names for graphviz graph\n nodes = OrderedDict()\n\n for i, node in enumerate(sorted(lines_nodes)):\n nodes[node] = 'n{}'.format(i+1)\n\n # print(lines_nodes, nodes)\n\n graph = list()\n\n # some basic style definition\n # https://graphviz.gitlab.io/_pages/doc/info/lang.html\n graph.append('digraph G {')\n\n # https://graphviz.gitlab.io/_pages/doc/info/shapes.html#record\n graph.append('\\tgraph [ center=true, margin=0.75, nodesep=0.5, ranksep=0.75, rankdir=LR ];')\n graph.append('\\tnode [ shape=box, style=\"rounded,filled\" width=0, height=0, '\n 'fontname=Helvetica, fontsize=11 ];')\n graph.append('\\tedge [ fontname=Helvetica, fontsize=9 ];')\n\n # emit nodes definition\n graph.append('\\n\\t// nodes')\n\n # https://www.graphviz.org/doc/info/colors.html#brewer\n group_colors = dict()\n\n for label, name in nodes.items():\n if ':' in label:\n (group, label) = str(label).split(':', 1)\n\n # register a new group for coloring\n if group not in group_colors:\n group_colors[group] = len(group_colors.keys()) + 1\n else:\n group = None\n\n label = escape_graphviz_entry(label)\n\n graph.append('\\t{name} [label=\"{label}\"{group}];'.format(\n name=name,\n label=\"{}\\\\n{}\".format(group, label) if group is not None else label,\n group=' group=\"{}\" colorscheme=pastel28 color={}'.format(\n group, group_colors[group]) if group is not None else ''\n ))\n\n # now, connect the nodes\n graph.append('\\n\\t// edges')\n for line in lines:\n label = line.get('metadata', '')\n\n graph.append('\\t{source} -> {target} [{label}];'.format(\n source=nodes[line['source']],\n target=nodes[line['target']],\n label='label=\"{}\"'.format(escape_graphviz_entry(label)) if label != '' else ''\n ))\n\n graph.append('}')\n\n return '\\n'.join(graph)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef logs_map_and_reduce(logs, _map, _reduce):\n keys = []\n mapped_count = Counter()\n mapped = defaultdict(list)\n\n # first map all entries\n for log in logs:\n key = _map(log)\n mapped[key].append(log)\n mapped_count[key] += 1\n\n if key not in keys:\n keys.append(key)\n\n # the most common mapped item\n top_count = mapped_count.most_common(1).pop()[1]\n\n # now reduce mapped items\n reduced = []\n\n # keep the order under control\n for key in keys:\n entries = mapped[key]\n # print(key, entries)\n\n # add \"value\" field to each reduced item (1.0 will be assigned to the most \"common\" item)\n item = _reduce(entries)\n item['value'] = 1. * len(entries) / top_count\n\n reduced.append(item)\n\n # print(mapped)\n return reduced", "response": "map and reduce the log list"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncloses the remote chromium instance. This command is normally executed as part of the class destructor. It can be called early without issue, but calling ANY class functions after the remote chromium instance is shut down will have unknown effects. Note that if you are rapidly creating and destroying ChromeController instances, you may need to *explicitly* call this before destruction.", "response": "def close_chromium(self):\n\t\t'''\n\t\tClose the remote chromium instance.\n\n\t\tThis command is normally executed as part of the class destructor.\n\t\tIt can be called early without issue, but calling ANY class functions\n\t\tafter the remote chromium instance is shut down will have unknown effects.\n\n\t\tNote that if you are rapidly creating and destroying ChromeController instances,\n\t\tyou may need to *explicitly* call this before destruction.\n\t\t'''\n\t\tif self.cr_proc:\n\t\t\ttry:\n\t\t\t\tif 'win' in sys.platform:\n\t\t\t\t\tself.__close_internal_windows()\n\t\t\t\telse:\n\t\t\t\t\tself.__close_internal_linux()\n\t\t\texcept Exception as e:\n\t\t\t\tfor line in traceback.format_exc().split(\"\\n\"):\n\t\t\t\t\tself.log.error(line)\n\n\n\n\t\tACTIVE_PORTS.discard(self.port)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef connect(self, tab_key):\n\n\t\tassert self.tablist is not None\n\n\t\ttab_idx = self._get_tab_idx_for_key(tab_key)\n\n\t\tif not self.tablist:\n\t\t\tself.tablist = self.fetch_tablist()\n\n\t\tfor fails in range(9999):\n\t\t\ttry:\n\t\t\t\t# If we're one past the end of the tablist, we need to create a new tab\n\t\t\t\tif tab_idx is None:\n\t\t\t\t\tself.log.debug(\"Creating new tab (%s active)\", len(self.tablist))\n\t\t\t\t\tself.__create_new_tab(tab_key)\n\n\t\t\t\tself.__connect_to_tab(tab_key)\n\t\t\t\tbreak\n\n\t\t\texcept cr_exceptions.ChromeConnectFailure as e:\n\t\t\t\tif fails > 6:\n\t\t\t\t\tself.log.error(\"Failed to fetch tab websocket URL after %s retries. Aborting!\", fails)\n\t\t\t\t\traise e\n\t\t\t\tself.log.info(\"Tab may not have started yet (%s tabs active). Recreating.\", len(self.tablist))\n\t\t\t\t# self.log.info(\"Tag: %s\", self.tablist[tab_idx])\n\n\n\t\t\t\t# For reasons I don't understand, sometimes a new tab doesn't get a websocket\n\t\t\t\t# debugger URL. Anyways, we close and re-open the tab if that happens.\n\t\t\t\t# TODO: Handle the case when this happens on the first tab. I think closing the first\n\t\t\t\t# tab will kill chromium.\n\t\t\t\tself.__close_tab(tab_key)", "response": "Open a websocket connection to remote browser."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nclose websocket connection to remote browser.", "response": "def close_websockets(self):\n\t\t\"\"\" Close websocket connection to remote browser.\"\"\"\n\t\tself.log.info(\"Websocket Teardown called\")\n\t\tfor key in list(self.soclist.keys()):\n\t\t\tif self.soclist[key]:\n\t\t\t\tself.soclist[key].close()\n\t\t\tself.soclist.pop(key)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fetch_tablist(self):\n\t\t# find websocket endpoint\n\t\ttry:\n\t\t\tresponse = requests.get(\"http://%s:%s/json\" % (self.host, self.port))\n\t\texcept requests.exceptions.ConnectionError:\n\t\t\traise cr_exceptions.ChromeConnectFailure(\"Failed to fetch configuration json from browser!\")\n\n\t\ttablist = json.loads(response.text)\n\n\t\treturn tablist", "response": "Connect to host : port and request list of open tabs return list of dicts of data about open tabs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a command to the remote chromium instance.", "response": "def send(self, command, tab_key, params=None):\n\t\t'''\n\t\tSend command `command` with optional parameters `params` to the\n\t\tremote chrome instance.\n\n\t\tThe command `id` is automatically added to the outgoing message.\n\n\t\treturn value is the command id, which can be used to match a command\n\t\tto it's associated response.\n\t\t'''\n\t\tself.__check_open_socket(tab_key)\n\n\t\tsent_id = self.msg_id\n\n\t\tcommand = {\n\t\t\t\t\"id\": self.msg_id,\n\t\t\t\t\"method\": command,\n\t\t\t}\n\n\t\tif params:\n\t\t\tcommand[\"params\"] = params\n\t\tnavcom = json.dumps(command)\n\n\n\t\t# self.log.debug(\"\t\tSending: '%s'\", navcom)\n\t\ttry:\n\t\t\tself.soclist[tab_key].send(navcom)\n\t\texcept (socket.timeout, websocket.WebSocketTimeoutException):\n\t\t\traise cr_exceptions.ChromeCommunicationsError(\"Failure sending command to chromium.\")\n\t\texcept websocket.WebSocketConnectionClosedException:\n\t\t\traise cr_exceptions.ChromeCommunicationsError(\"Websocket appears to have been closed. Is the\"\n\t\t\t\t\" remote chromium instance dead?\")\n\n\n\t\tself.msg_id += 1\n\t\treturn sent_id"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef recv_filtered(self, keycheck, tab_key, timeout=30, message=None):\n\t\t'''\n\t\tReceive a filtered message, using the callable `keycheck` to filter received messages\n\t\tfor content.\n\n\t\t`keycheck` is expected to be a callable that takes a single parameter (the decoded response\n\t\tfrom chromium), and returns a boolean (true, if the command is the one filtered for, or false\n\t\tif the command is not the one filtered for).\n\n\t\tThis is used internally, for example, by `recv()`, to filter the response for a specific ID:\n\n\t\t```\n\t\t\tdef check_func(message):\n\t\t\t\tif message_id is None:\n\t\t\t\t\treturn True\n\t\t\t\tif \"id\" in message:\n\t\t\t\t\treturn message['id'] == message_id\n\t\t\t\treturn False\n\t\t\treturn self.recv_filtered(check_func, timeout)\n\n\t\t```\n\n\t\tNote that the function is defined dynamically, and `message_id` is captured via closure.\n\n\t\t'''\n\n\n\t\tself.__check_open_socket(tab_key)\n\n\t\t# First, check if the message has already been received.\n\t\tfor idx in range(len(self.messages[tab_key])):\n\t\t\tif keycheck(self.messages[tab_key][idx]):\n\t\t\t\treturn self.messages[tab_key].pop(idx)\n\n\t\ttimeout_at = time.time() + timeout\n\t\twhile 1:\n\t\t\ttmp = self.___recv(tab_key)\n\t\t\tif keycheck(tmp):\n\t\t\t\treturn tmp\n\t\t\telse:\n\t\t\t\tself.messages[tab_key].append(tmp)\n\n\t\t\tif time.time() > timeout_at:\n\t\t\t\tif message:\n\t\t\t\t\traise cr_exceptions.ChromeResponseNotReceived(\"Failed to receive response in recv_filtered() (%s)\" % message)\n\t\t\t\telse:\n\t\t\t\t\traise cr_exceptions.ChromeResponseNotReceived(\"Failed to receive response in recv_filtered()\")\n\t\t\telse:\n\t\t\t\ttime.sleep(0.005)", "response": "This method is used to filter received messages using a callable that takes a single parameter and returns a boolean value that indicates whether the command was filtered for the given message."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreceiving a all messages matching a filter, using the callable `keycheck` to filter received messages for content. This function will *ALWAY* block for at least `timeout` seconds. If chromium is for some reason continuously streaming responses, it may block forever! `keycheck` is expected to be a callable that takes a single parameter (the decoded response from chromium), and returns a boolean (true, if the command is the one filtered for, or false if the command is not the one filtered for). ``` def check_func(message): if message_id is None: return True if \"id\" in message: return message['id'] == message_id return False return self.recv_filtered(check_func, timeout) ``` Note that the function is defined dynamically, and `message_id` is captured via closure.", "response": "def recv_all_filtered(self, keycheck, tab_key, timeout=0.5):\n\t\t'''\n\t\tReceive a all messages matching a filter, using the callable `keycheck` to filter received messages\n\t\tfor content.\n\n\t\tThis function will *ALWAY* block for at least `timeout` seconds.\n\n\t\tIf chromium is for some reason continuously streaming responses, it may block forever!\n\n\t\t`keycheck` is expected to be a callable that takes a single parameter (the decoded response\n\t\tfrom chromium), and returns a boolean (true, if the command is the one filtered for, or false\n\t\tif the command is not the one filtered for).\n\n\t\t```\n\t\t\tdef check_func(message):\n\t\t\t\tif message_id is None:\n\t\t\t\t\treturn True\n\t\t\t\tif \"id\" in message:\n\t\t\t\t\treturn message['id'] == message_id\n\t\t\t\treturn False\n\t\t\treturn self.recv_filtered(check_func, timeout)\n\n\t\t```\n\n\t\tNote that the function is defined dynamically, and `message_id` is captured via closure.\n\n\t\t'''\n\n\n\t\tself.__check_open_socket(tab_key)\n\t\t# First, check if the message has already been received.\n\t\tret = [tmp for tmp in self.messages[tab_key] if keycheck(tmp)]\n\t\tself.messages[tab_key] = [tmp for tmp in self.messages[tab_key] if not keycheck(tmp)]\n\n\t\tself.log.debug(\"Waiting for all messages from the socket\")\n\t\ttimeout_at = time.time() + timeout\n\t\twhile 1:\n\t\t\ttmp = self.___recv(tab_key, timeout=timeout)\n\t\t\tif keycheck(tmp):\n\t\t\t\tret.append(tmp)\n\t\t\telse:\n\t\t\t\tself.messages[tab_key].append(tmp)\n\n\t\t\tif time.time() > timeout_at:\n\t\t\t\treturn ret\n\t\t\telse:\n\t\t\t\tself.log.debug(\"Sleeping: %s, %s\" % (timeout_at, time.time()))\n\t\t\t\ttime.sleep(0.005)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef drain(self, tab_key):\n\t\t'''\n\t\tReturn all messages in waiting for the websocket connection.\n\t\t'''\n\t\tself.log.debug(\"Draining transport\")\n\t\tret = []\n\t\twhile len(self.messages[tab_key]):\n\t\t\tret.append(self.messages[tab_key].pop(0))\n\n\t\tself.log.debug(\"Polling socket\")\n\n\t\ttmp = self.___recv(tab_key)\n\t\twhile tmp is not None:\n\t\t\tret.append(tmp)\n\t\t\ttmp = self.___recv(tab_key)\n\n\t\tself.log.debug(\"Drained %s messages\", len(ret))\n\t\treturn ret", "response": "Return all messages in waiting for the websocket connection."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fetch(url, binary, outfile, noprint, rendered):\n\t'''\n\tFetch a specified URL's content, and output it to the console.\n\t'''\n\twith chrome_context.ChromeContext(binary=binary) as cr:\n\t\tresp = cr.blocking_navigate_and_get_source(url)\n\t\tif rendered:\n\t\t\tresp['content'] = cr.get_rendered_page_source()\n\t\t\tresp['binary'] = False\n\t\t\tresp['mimie'] = 'text/html'\n\n\tif not noprint:\n\t\tif resp['binary'] is False:\n\t\t\tprint(resp['content'])\n\t\telse:\n\t\t\tprint(\"Response is a binary file\")\n\t\t\tprint(\"Cannot print!\")\n\n\tif outfile:\n\t\twith open(outfile, \"wb\") as fp:\n\t\t\tif resp['binary']:\n\t\t\t\tfp.write(resp['content'])\n\t\t\telse:\n\t\t\t\tfp.write(resp['content'].encode(\"UTF-8\"))", "response": "Fetch a specified URL s content and output it to a file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Memory_setPressureNotificationsSuppressed(self, suppressed):\n\t\tassert isinstance(suppressed, (bool,)\n\t\t ), \"Argument 'suppressed' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t suppressed)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Memory.setPressureNotificationsSuppressed', suppressed=suppressed)\n\t\treturn subdom_funcs", "response": "This method is used to set the suppressed memory pressure notifications in all processes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Page_addScriptToEvaluateOnLoad(self, scriptSource):\n\t\tassert isinstance(scriptSource, (str,)\n\t\t ), \"Argument 'scriptSource' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scriptSource)\n\t\tsubdom_funcs = self.synchronous_command('Page.addScriptToEvaluateOnLoad',\n\t\t scriptSource=scriptSource)\n\t\treturn subdom_funcs", "response": "This method is used to add a script to the page to evaluate on load."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Page_addScriptToEvaluateOnNewDocument(self, source):\n\t\tassert isinstance(source, (str,)\n\t\t ), \"Argument 'source' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t source)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Page.addScriptToEvaluateOnNewDocument', source=source)\n\t\treturn subdom_funcs", "response": "This method is used to add a script to the current document."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: Page.setAutoAttachToCreatedPages Domain: Page Method name: setAutoAttachToCreatedPages WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'autoAttach' (type: boolean) -> If true, browser will open a new inspector window for every page created from this one. No return value. Description: Controls whether browser will open a new inspector window for connected pages.", "response": "def Page_setAutoAttachToCreatedPages(self, autoAttach):\n\t\t\"\"\"\n\t\tFunction path: Page.setAutoAttachToCreatedPages\n\t\t\tDomain: Page\n\t\t\tMethod name: setAutoAttachToCreatedPages\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'autoAttach' (type: boolean) -> If true, browser will open a new inspector window for every page created from this one.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Controls whether browser will open a new inspector window for connected pages.\n\t\t\"\"\"\n\t\tassert isinstance(autoAttach, (bool,)\n\t\t ), \"Argument 'autoAttach' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t autoAttach)\n\t\tsubdom_funcs = self.synchronous_command('Page.setAutoAttachToCreatedPages',\n\t\t autoAttach=autoAttach)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: Page.setAdBlockingEnabled Domain: Page Method name: setAdBlockingEnabled WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'enabled' (type: boolean) -> Whether to block ads. No return value. Description: Enable Chrome's experimental ad filter on all sites.", "response": "def Page_setAdBlockingEnabled(self, enabled):\n\t\t\"\"\"\n\t\tFunction path: Page.setAdBlockingEnabled\n\t\t\tDomain: Page\n\t\t\tMethod name: setAdBlockingEnabled\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'enabled' (type: boolean) -> Whether to block ads.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Enable Chrome's experimental ad filter on all sites.\n\t\t\"\"\"\n\t\tassert isinstance(enabled, (bool,)\n\t\t ), \"Argument 'enabled' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t enabled)\n\t\tsubdom_funcs = self.synchronous_command('Page.setAdBlockingEnabled',\n\t\t enabled=enabled)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: Page.navigateToHistoryEntry Domain: Page Method name: navigateToHistoryEntry WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'entryId' (type: integer) -> Unique id of the entry to navigate to. No return value. Description: Navigates current page to the given history entry.", "response": "def Page_navigateToHistoryEntry(self, entryId):\n\t\t\"\"\"\n\t\tFunction path: Page.navigateToHistoryEntry\n\t\t\tDomain: Page\n\t\t\tMethod name: navigateToHistoryEntry\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'entryId' (type: integer) -> Unique id of the entry to navigate to.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Navigates current page to the given history entry.\n\t\t\"\"\"\n\t\tassert isinstance(entryId, (int,)\n\t\t ), \"Argument 'entryId' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t entryId)\n\t\tsubdom_funcs = self.synchronous_command('Page.navigateToHistoryEntry',\n\t\t entryId=entryId)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npage deleteCookie - Deletes a browser cookie with given name domain and path.", "response": "def Page_deleteCookie(self, cookieName, url):\n\t\t\"\"\"\n\t\tFunction path: Page.deleteCookie\n\t\t\tDomain: Page\n\t\t\tMethod name: deleteCookie\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'cookieName' (type: string) -> Name of the cookie to remove.\n\t\t\t\t\t'url' (type: string) -> URL to match cooke domain and path.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Deletes browser cookie with given name, domain and path.\n\t\t\"\"\"\n\t\tassert isinstance(cookieName, (str,)\n\t\t ), \"Argument 'cookieName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t cookieName)\n\t\tassert isinstance(url, (str,)\n\t\t ), \"Argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t url)\n\t\tsubdom_funcs = self.synchronous_command('Page.deleteCookie', cookieName=\n\t\t cookieName, url=url)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Page_getResourceContent(self, frameId, url):\n\t\tassert isinstance(url, (str,)\n\t\t ), \"Argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t url)\n\t\tsubdom_funcs = self.synchronous_command('Page.getResourceContent',\n\t\t frameId=frameId, url=url)\n\t\treturn subdom_funcs", "response": "Function path: Page.getResourceContent\n\t\t\tDomain: Page\n\t\t\tMethod name: getResourceContent\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'frameId' (type: FrameId) -> Frame id to get resource for.\n\t\t\t\t\t'url' (type: string) -> URL of the resource to get content for.\n\t\t\tReturns:\n\t\t\t\t'content' (type: string) -> Resource content.\n\t\t\t\t'base64Encoded' (type: boolean) -> True, if content was served as base64.\n\t\t\n\t\t\tDescription: Returns content of the given resource."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Page_searchInResource(self, frameId, url, query, **kwargs):\n\t\tassert isinstance(url, (str,)\n\t\t ), \"Argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t url)\n\t\tassert isinstance(query, (str,)\n\t\t ), \"Argument 'query' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t query)\n\t\tif 'caseSensitive' in kwargs:\n\t\t\tassert isinstance(kwargs['caseSensitive'], (bool,)\n\t\t\t ), \"Optional argument 'caseSensitive' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['caseSensitive'])\n\t\tif 'isRegex' in kwargs:\n\t\t\tassert isinstance(kwargs['isRegex'], (bool,)\n\t\t\t ), \"Optional argument 'isRegex' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['isRegex'])\n\t\texpected = ['caseSensitive', 'isRegex']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['caseSensitive', 'isRegex']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Page.searchInResource', frameId=\n\t\t frameId, url=url, query=query, **kwargs)\n\t\treturn subdom_funcs", "response": "Function path: Page.searchInResource\n\t\t\tDomain: Page\n\t\t\tMethod name: searchInResource\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'frameId' (type: FrameId) -> Frame id for resource to search in.\n\t\t\t\t\t'url' (type: string) -> URL of the resource to search in.\n\t\t\t\t\t'query' (type: string) -> String to search for.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'caseSensitive' (type: boolean) -> If true, search is case sensitive.\n\t\t\t\t\t'isRegex' (type: boolean) -> If true, treats string parameter as regex.\n\t\t\tReturns:\n\t\t\t\t'result' (type: array) -> List of search matches.\n\t\t\n\t\t\tDescription: Searches for given string in resource content."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef Page_setDocumentContent(self, frameId, html):\n\t\tassert isinstance(html, (str,)\n\t\t ), \"Argument 'html' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t html)\n\t\tsubdom_funcs = self.synchronous_command('Page.setDocumentContent',\n\t\t frameId=frameId, html=html)\n\t\treturn subdom_funcs", "response": "Page. setDocumentContent - Sets given markup as the document s HTML."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npage setDeviceMetricsOverride - Sets the device screen dimensions and scale factor of the resulting view image.", "response": "def Page_setDeviceMetricsOverride(self, width, height, deviceScaleFactor,\n\t mobile, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Page.setDeviceMetricsOverride\n\t\t\tDomain: Page\n\t\t\tMethod name: setDeviceMetricsOverride\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'width' (type: integer) -> Overriding width value in pixels (minimum 0, maximum 10000000). 0 disables the override.\n\t\t\t\t\t'height' (type: integer) -> Overriding height value in pixels (minimum 0, maximum 10000000). 0 disables the override.\n\t\t\t\t\t'deviceScaleFactor' (type: number) -> Overriding device scale factor value. 0 disables the override.\n\t\t\t\t\t'mobile' (type: boolean) -> Whether to emulate mobile device. This includes viewport meta tag, overlay scrollbars, text autosizing and more.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'scale' (type: number) -> Scale to apply to resulting view image.\n\t\t\t\t\t'screenWidth' (type: integer) -> Overriding screen width value in pixels (minimum 0, maximum 10000000).\n\t\t\t\t\t'screenHeight' (type: integer) -> Overriding screen height value in pixels (minimum 0, maximum 10000000).\n\t\t\t\t\t'positionX' (type: integer) -> Overriding view X position on screen in pixels (minimum 0, maximum 10000000).\n\t\t\t\t\t'positionY' (type: integer) -> Overriding view Y position on screen in pixels (minimum 0, maximum 10000000).\n\t\t\t\t\t'dontSetVisibleSize' (type: boolean) -> Do not set visible view size, rely upon explicit setVisibleSize call.\n\t\t\t\t\t'screenOrientation' (type: Emulation.ScreenOrientation) -> Screen orientation override.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Overrides the values of device screen dimensions (window.screen.width, window.screen.height, window.innerWidth, window.innerHeight, and \"device-width\"/\"device-height\"-related CSS media query results).\n\t\t\"\"\"\n\t\tassert isinstance(width, (int,)\n\t\t ), \"Argument 'width' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t width)\n\t\tassert isinstance(height, (int,)\n\t\t ), \"Argument 'height' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t height)\n\t\tassert isinstance(deviceScaleFactor, (float, int)\n\t\t ), \"Argument 'deviceScaleFactor' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t deviceScaleFactor)\n\t\tassert isinstance(mobile, (bool,)\n\t\t ), \"Argument 'mobile' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t mobile)\n\t\tif 'scale' in kwargs:\n\t\t\tassert isinstance(kwargs['scale'], (float, int)\n\t\t\t ), \"Optional argument 'scale' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['scale'])\n\t\tif 'screenWidth' in kwargs:\n\t\t\tassert isinstance(kwargs['screenWidth'], (int,)\n\t\t\t ), \"Optional argument 'screenWidth' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['screenWidth'])\n\t\tif 'screenHeight' in kwargs:\n\t\t\tassert isinstance(kwargs['screenHeight'], (int,)\n\t\t\t ), \"Optional argument 'screenHeight' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['screenHeight'])\n\t\tif 'positionX' in kwargs:\n\t\t\tassert isinstance(kwargs['positionX'], (int,)\n\t\t\t ), \"Optional argument 'positionX' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['positionX'])\n\t\tif 'positionY' in kwargs:\n\t\t\tassert isinstance(kwargs['positionY'], (int,)\n\t\t\t ), \"Optional argument 'positionY' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['positionY'])\n\t\tif 'dontSetVisibleSize' in kwargs:\n\t\t\tassert isinstance(kwargs['dontSetVisibleSize'], (bool,)\n\t\t\t ), \"Optional argument 'dontSetVisibleSize' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['dontSetVisibleSize'])\n\t\texpected = ['scale', 'screenWidth', 'screenHeight', 'positionX',\n\t\t 'positionY', 'dontSetVisibleSize', 'screenOrientation']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['scale', 'screenWidth', 'screenHeight', 'positionX', 'positionY', 'dontSetVisibleSize', 'screenOrientation']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Page.setDeviceMetricsOverride',\n\t\t width=width, height=height, deviceScaleFactor=deviceScaleFactor,\n\t\t mobile=mobile, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npaging setDeviceOrientationOverride - Allows you to override the Device Orientation.", "response": "def Page_setDeviceOrientationOverride(self, alpha, beta, gamma):\n\t\t\"\"\"\n\t\tFunction path: Page.setDeviceOrientationOverride\n\t\t\tDomain: Page\n\t\t\tMethod name: setDeviceOrientationOverride\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'alpha' (type: number) -> Mock alpha\n\t\t\t\t\t'beta' (type: number) -> Mock beta\n\t\t\t\t\t'gamma' (type: number) -> Mock gamma\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Overrides the Device Orientation.\n\t\t\"\"\"\n\t\tassert isinstance(alpha, (float, int)\n\t\t ), \"Argument 'alpha' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t alpha)\n\t\tassert isinstance(beta, (float, int)\n\t\t ), \"Argument 'beta' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t beta)\n\t\tassert isinstance(gamma, (float, int)\n\t\t ), \"Argument 'gamma' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t gamma)\n\t\tsubdom_funcs = self.synchronous_command('Page.setDeviceOrientationOverride',\n\t\t alpha=alpha, beta=beta, gamma=gamma)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Page_screencastFrameAck(self, sessionId):\n\t\tassert isinstance(sessionId, (int,)\n\t\t ), \"Argument 'sessionId' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t sessionId)\n\t\tsubdom_funcs = self.synchronous_command('Page.screencastFrameAck',\n\t\t sessionId=sessionId)\n\t\treturn subdom_funcs", "response": "This method is used to acknowledge a screencast frame received by the frontend. This method is used to acknowledge a screencast frame received by the frontend."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef Overlay_setShowPaintRects(self, result):\n\t\tassert isinstance(result, (bool,)\n\t\t ), \"Argument 'result' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t result)\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setShowPaintRects',\n\t\t result=result)\n\t\treturn subdom_funcs", "response": "This method is used to set the show paint rectangles on the overlay."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: Overlay.setShowDebugBorders Domain: Overlay Method name: setShowDebugBorders Parameters: Required arguments: 'show' (type: boolean) -> True for showing debug borders No return value. Description: Requests that backend shows debug borders on layers", "response": "def Overlay_setShowDebugBorders(self, show):\n\t\t\"\"\"\n\t\tFunction path: Overlay.setShowDebugBorders\n\t\t\tDomain: Overlay\n\t\t\tMethod name: setShowDebugBorders\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'show' (type: boolean) -> True for showing debug borders\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Requests that backend shows debug borders on layers\n\t\t\"\"\"\n\t\tassert isinstance(show, (bool,)\n\t\t ), \"Argument 'show' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t show)\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setShowDebugBorders',\n\t\t show=show)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Overlay_setShowFPSCounter(self, show):\n\t\tassert isinstance(show, (bool,)\n\t\t ), \"Argument 'show' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t show)\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setShowFPSCounter', show\n\t\t =show)\n\t\treturn subdom_funcs", "response": "This method is used to set the show flag on the FPS counter."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: Overlay.setShowScrollBottleneckRects Domain: Overlay Method name: setShowScrollBottleneckRects Parameters: Required arguments: 'show' (type: boolean) -> True for showing scroll bottleneck rects No return value. Description: Requests that backend shows scroll bottleneck rects", "response": "def Overlay_setShowScrollBottleneckRects(self, show):\n\t\t\"\"\"\n\t\tFunction path: Overlay.setShowScrollBottleneckRects\n\t\t\tDomain: Overlay\n\t\t\tMethod name: setShowScrollBottleneckRects\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'show' (type: boolean) -> True for showing scroll bottleneck rects\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Requests that backend shows scroll bottleneck rects\n\t\t\"\"\"\n\t\tassert isinstance(show, (bool,)\n\t\t ), \"Argument 'show' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t show)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Overlay.setShowScrollBottleneckRects', show=show)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Overlay_setShowViewportSizeOnResize(self, show):\n\t\tassert isinstance(show, (bool,)\n\t\t ), \"Argument 'show' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t show)\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setShowViewportSizeOnResize'\n\t\t , show=show)\n\t\treturn subdom_funcs", "response": "This command is used to set the viewport size on resize."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Overlay_setSuspended(self, suspended):\n\t\tassert isinstance(suspended, (bool,)\n\t\t ), \"Argument 'suspended' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t suspended)\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setSuspended', suspended\n\t\t =suspended)\n\t\treturn subdom_funcs", "response": "This method is used to set the suspended flag of the overlay."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: Overlay.setInspectMode Domain: Overlay Method name: setInspectMode Parameters: Required arguments: 'mode' (type: InspectMode) -> Set an inspection mode. Optional arguments: 'highlightConfig' (type: HighlightConfig) -> A descriptor for the highlight appearance of hovered-over nodes. May be omitted if enabled == false. No return value. Description: Enters the 'inspect' mode. In this mode, elements that user is hovering over are highlighted. Backend then generates 'inspectNodeRequested' event upon element selection.", "response": "def Overlay_setInspectMode(self, mode, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Overlay.setInspectMode\n\t\t\tDomain: Overlay\n\t\t\tMethod name: setInspectMode\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'mode' (type: InspectMode) -> Set an inspection mode.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'highlightConfig' (type: HighlightConfig) -> A descriptor for the highlight appearance of hovered-over nodes. May be omitted if enabled == false.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Enters the 'inspect' mode. In this mode, elements that user is hovering over are highlighted. Backend then generates 'inspectNodeRequested' event upon element selection.\n\t\t\"\"\"\n\t\texpected = ['highlightConfig']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['highlightConfig']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Overlay.setInspectMode', mode=\n\t\t mode, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfunctioning path: Overlay.highlightRect Domain: Overlay Method name: highlightRect Parameters: Required arguments: 'x' (type: integer) -> X coordinate 'y' (type: integer) -> Y coordinate 'width' (type: integer) -> Rectangle width 'height' (type: integer) -> Rectangle height Optional arguments: 'color' (type: DOM.RGBA) -> The highlight fill color (default: transparent). 'outlineColor' (type: DOM.RGBA) -> The highlight outline color (default: transparent). No return value. Description: Highlights given rectangle. Coordinates are absolute with respect to the main frame viewport.", "response": "def Overlay_highlightRect(self, x, y, width, height, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Overlay.highlightRect\n\t\t\tDomain: Overlay\n\t\t\tMethod name: highlightRect\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'x' (type: integer) -> X coordinate\n\t\t\t\t\t'y' (type: integer) -> Y coordinate\n\t\t\t\t\t'width' (type: integer) -> Rectangle width\n\t\t\t\t\t'height' (type: integer) -> Rectangle height\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'color' (type: DOM.RGBA) -> The highlight fill color (default: transparent).\n\t\t\t\t\t'outlineColor' (type: DOM.RGBA) -> The highlight outline color (default: transparent).\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Highlights given rectangle. Coordinates are absolute with respect to the main frame viewport.\n\t\t\"\"\"\n\t\tassert isinstance(x, (int,)\n\t\t ), \"Argument 'x' must be of type '['int']'. Received type: '%s'\" % type(x\n\t\t )\n\t\tassert isinstance(y, (int,)\n\t\t ), \"Argument 'y' must be of type '['int']'. Received type: '%s'\" % type(y\n\t\t )\n\t\tassert isinstance(width, (int,)\n\t\t ), \"Argument 'width' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t width)\n\t\tassert isinstance(height, (int,)\n\t\t ), \"Argument 'height' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t height)\n\t\texpected = ['color', 'outlineColor']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['color', 'outlineColor']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Overlay.highlightRect', x=x, y=y,\n\t\t width=width, height=height, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Overlay_highlightQuad(self, quad, **kwargs):\n\t\texpected = ['color', 'outlineColor']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['color', 'outlineColor']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Overlay.highlightQuad', quad=\n\t\t quad, **kwargs)\n\t\treturn subdom_funcs", "response": "This command is used to highlight a given quad."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef Overlay_highlightNode(self, highlightConfig, **kwargs):\n\t\texpected = ['nodeId', 'backendNodeId', 'objectId']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['nodeId', 'backendNodeId', 'objectId']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Overlay.highlightNode',\n\t\t highlightConfig=highlightConfig, **kwargs)\n\t\treturn subdom_funcs", "response": "This method is used to highlight a node in the overlay."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Overlay_highlightFrame(self, frameId, **kwargs):\n\t\texpected = ['contentColor', 'contentOutlineColor']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['contentColor', 'contentOutlineColor']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Overlay.highlightFrame', frameId\n\t\t =frameId, **kwargs)\n\t\treturn subdom_funcs", "response": "This command is used to highlight the owner element of a frame."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: Emulation.setPageScaleFactor Domain: Emulation Method name: setPageScaleFactor WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'pageScaleFactor' (type: number) -> Page scale factor. No return value. Description: Sets a specified page scale factor.", "response": "def Emulation_setPageScaleFactor(self, pageScaleFactor):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setPageScaleFactor\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setPageScaleFactor\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'pageScaleFactor' (type: number) -> Page scale factor.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets a specified page scale factor.\n\t\t\"\"\"\n\t\tassert isinstance(pageScaleFactor, (float, int)\n\t\t ), \"Argument 'pageScaleFactor' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t pageScaleFactor)\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setPageScaleFactor',\n\t\t pageScaleFactor=pageScaleFactor)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: Emulation.setVisibleSize Domain: Emulation Method name: setVisibleSize WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'width' (type: integer) -> Frame width (DIP). 'height' (type: integer) -> Frame height (DIP). No return value. Description: Resizes the frame/viewport of the page. Note that this does not affect the frame's container (e.g. browser window). Can be used to produce screenshots of the specified size. Not supported on Android.", "response": "def Emulation_setVisibleSize(self, width, height):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setVisibleSize\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setVisibleSize\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'width' (type: integer) -> Frame width (DIP).\n\t\t\t\t\t'height' (type: integer) -> Frame height (DIP).\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Resizes the frame/viewport of the page. Note that this does not affect the frame's container (e.g. browser window). Can be used to produce screenshots of the specified size. Not supported on Android.\n\t\t\"\"\"\n\t\tassert isinstance(width, (int,)\n\t\t ), \"Argument 'width' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t width)\n\t\tassert isinstance(height, (int,)\n\t\t ), \"Argument 'height' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t height)\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setVisibleSize', width\n\t\t =width, height=height)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning path: Emulation.setScriptExecutionDisabled Domain: Emulation Method name: setScriptExecutionDisabled WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'value' (type: boolean) -> Whether script execution should be disabled in the page. No return value. Description: Switches script execution in the page.", "response": "def Emulation_setScriptExecutionDisabled(self, value):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setScriptExecutionDisabled\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setScriptExecutionDisabled\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'value' (type: boolean) -> Whether script execution should be disabled in the page.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Switches script execution in the page.\n\t\t\"\"\"\n\t\tassert isinstance(value, (bool,)\n\t\t ), \"Argument 'value' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Emulation.setScriptExecutionDisabled', value=value)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nemulates the given media for CSS media queries.", "response": "def Emulation_setEmulatedMedia(self, media):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setEmulatedMedia\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setEmulatedMedia\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'media' (type: string) -> Media type to emulate. Empty string disables the override.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Emulates the given media for CSS media queries.\n\t\t\"\"\"\n\t\tassert isinstance(media, (str,)\n\t\t ), \"Argument 'media' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t media)\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setEmulatedMedia',\n\t\t media=media)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfunctioning path: Emulation.setCPUThrottlingRate Domain: Emulation Method name: setCPUThrottlingRate WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'rate' (type: number) -> Throttling rate as a slowdown factor (1 is no throttle, 2 is 2x slowdown, etc). No return value. Description: Enables CPU throttling to emulate slow CPUs.", "response": "def Emulation_setCPUThrottlingRate(self, rate):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setCPUThrottlingRate\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setCPUThrottlingRate\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'rate' (type: number) -> Throttling rate as a slowdown factor (1 is no throttle, 2 is 2x slowdown, etc).\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Enables CPU throttling to emulate slow CPUs.\n\t\t\"\"\"\n\t\tassert isinstance(rate, (float, int)\n\t\t ), \"Argument 'rate' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t rate)\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setCPUThrottlingRate',\n\t\t rate=rate)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Emulation_setVirtualTimePolicy(self, policy, **kwargs):\n\t\tif 'budget' in kwargs:\n\t\t\tassert isinstance(kwargs['budget'], (int,)\n\t\t\t ), \"Optional argument 'budget' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['budget'])\n\t\texpected = ['budget']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['budget']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setVirtualTimePolicy',\n\t\t policy=policy, **kwargs)\n\t\treturn subdom_funcs", "response": "Emulation. setVirtualTimePolicy - Sets the current virtual time policy."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: Emulation.setNavigatorOverrides Domain: Emulation Method name: setNavigatorOverrides WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'platform' (type: string) -> The platform navigator.platform should return. No return value. Description: Overrides value returned by the javascript navigator object.", "response": "def Emulation_setNavigatorOverrides(self, platform):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setNavigatorOverrides\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setNavigatorOverrides\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'platform' (type: string) -> The platform navigator.platform should return.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Overrides value returned by the javascript navigator object.\n\t\t\"\"\"\n\t\tassert isinstance(platform, (str,)\n\t\t ), \"Argument 'platform' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t platform)\n\t\tsubdom_funcs = self.synchronous_command('Emulation.setNavigatorOverrides',\n\t\t platform=platform)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: Emulation.setDefaultBackgroundColorOverride Domain: Emulation Method name: setDefaultBackgroundColorOverride WARNING: This function is marked 'Experimental'! Parameters: Optional arguments: 'color' (type: DOM.RGBA) -> RGBA of the default background color. If not specified, any existing override will be cleared. No return value. Description: Sets or clears an override of the default background color of the frame. This override is used if the content does not specify one.", "response": "def Emulation_setDefaultBackgroundColorOverride(self, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Emulation.setDefaultBackgroundColorOverride\n\t\t\tDomain: Emulation\n\t\t\tMethod name: setDefaultBackgroundColorOverride\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'color' (type: DOM.RGBA) -> RGBA of the default background color. If not specified, any existing override will be cleared.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets or clears an override of the default background color of the frame. This override is used if the content does not specify one.\n\t\t\"\"\"\n\t\texpected = ['color']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['color']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Emulation.setDefaultBackgroundColorOverride', **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Security_handleCertificateError(self, eventId, action):\n\t\tassert isinstance(eventId, (int,)\n\t\t ), \"Argument 'eventId' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t eventId)\n\t\tsubdom_funcs = self.synchronous_command('Security.handleCertificateError',\n\t\t eventId=eventId, action=action)\n\t\treturn subdom_funcs", "response": "This method is used to handle a certificate error event. It is used to handle the certificate error event."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: Security.setOverrideCertificateErrors Domain: Security Method name: setOverrideCertificateErrors Parameters: Required arguments: 'override' (type: boolean) -> If true, certificate errors will be overridden. No return value. Description: Enable/disable overriding certificate errors. If enabled, all certificate error events need to be handled by the DevTools client and should be answered with handleCertificateError commands.", "response": "def Security_setOverrideCertificateErrors(self, override):\n\t\t\"\"\"\n\t\tFunction path: Security.setOverrideCertificateErrors\n\t\t\tDomain: Security\n\t\t\tMethod name: setOverrideCertificateErrors\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'override' (type: boolean) -> If true, certificate errors will be overridden.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Enable/disable overriding certificate errors. If enabled, all certificate error events need to be handled by the DevTools client and should be answered with handleCertificateError commands.\n\t\t\"\"\"\n\t\tassert isinstance(override, (bool,)\n\t\t ), \"Argument 'override' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t override)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Security.setOverrideCertificateErrors', override=override)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Audits_getEncodedResponse(self, requestId, encoding, **kwargs):\n\t\tassert isinstance(encoding, (str,)\n\t\t ), \"Argument 'encoding' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t encoding)\n\t\tif 'quality' in kwargs:\n\t\t\tassert isinstance(kwargs['quality'], (float, int)\n\t\t\t ), \"Optional argument 'quality' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['quality'])\n\t\tif 'sizeOnly' in kwargs:\n\t\t\tassert isinstance(kwargs['sizeOnly'], (bool,)\n\t\t\t ), \"Optional argument 'sizeOnly' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['sizeOnly'])\n\t\texpected = ['quality', 'sizeOnly']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['quality', 'sizeOnly']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Audits.getEncodedResponse',\n\t\t requestId=requestId, encoding=encoding, **kwargs)\n\t\treturn subdom_funcs", "response": "Audits. getEncodedResponse method. Returns the encoded response body and size if it was re - encoded."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Network_setUserAgentOverride(self, userAgent):\n\t\tassert isinstance(userAgent, (str,)\n\t\t ), \"Argument 'userAgent' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t userAgent)\n\t\tsubdom_funcs = self.synchronous_command('Network.setUserAgentOverride',\n\t\t userAgent=userAgent)\n\t\treturn subdom_funcs", "response": "This method allows overriding user agent with the given string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfunctioning path: Network.setBlockedURLs Domain: Network Method name: setBlockedURLs WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'urls' (type: array) -> URL patterns to block. Wildcards ('*') are allowed. No return value. Description: Blocks URLs from loading.", "response": "def Network_setBlockedURLs(self, urls):\n\t\t\"\"\"\n\t\tFunction path: Network.setBlockedURLs\n\t\t\tDomain: Network\n\t\t\tMethod name: setBlockedURLs\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'urls' (type: array) -> URL patterns to block. Wildcards ('*') are allowed.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Blocks URLs from loading.\n\t\t\"\"\"\n\t\tassert isinstance(urls, (list, tuple)\n\t\t ), \"Argument 'urls' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t urls)\n\t\tsubdom_funcs = self.synchronous_command('Network.setBlockedURLs', urls=urls)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: Network.getCookies Domain: Network Method name: getCookies WARNING: This function is marked 'Experimental'! Parameters: Optional arguments: 'urls' (type: array) -> The list of URLs for which applicable cookies will be fetched Returns: 'cookies' (type: array) -> Array of cookie objects. Description: Returns all browser cookies for the current URL. Depending on the backend support, will return detailed cookie information in the cookies field.", "response": "def Network_getCookies(self, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Network.getCookies\n\t\t\tDomain: Network\n\t\t\tMethod name: getCookies\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'urls' (type: array) -> The list of URLs for which applicable cookies will be fetched\n\t\t\tReturns:\n\t\t\t\t'cookies' (type: array) -> Array of cookie objects.\n\t\t\n\t\t\tDescription: Returns all browser cookies for the current URL. Depending on the backend support, will return detailed cookie information in the cookies field.\n\t\t\"\"\"\n\t\tif 'urls' in kwargs:\n\t\t\tassert isinstance(kwargs['urls'], (list, tuple)\n\t\t\t ), \"Optional argument 'urls' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t\t kwargs['urls'])\n\t\texpected = ['urls']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['urls']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Network.getCookies', **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Network_deleteCookies(self, name, **kwargs):\n\t\tassert isinstance(name, (str,)\n\t\t ), \"Argument 'name' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t name)\n\t\tif 'url' in kwargs:\n\t\t\tassert isinstance(kwargs['url'], (str,)\n\t\t\t ), \"Optional argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['url'])\n\t\tif 'domain' in kwargs:\n\t\t\tassert isinstance(kwargs['domain'], (str,)\n\t\t\t ), \"Optional argument 'domain' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['domain'])\n\t\tif 'path' in kwargs:\n\t\t\tassert isinstance(kwargs['path'], (str,)\n\t\t\t ), \"Optional argument 'path' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['path'])\n\t\texpected = ['url', 'domain', 'path']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['url', 'domain', 'path']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Network.deleteCookies', name=\n\t\t name, **kwargs)\n\t\treturn subdom_funcs", "response": "Function Network. deleteCookies Network. deleteCookiesNetworkMethod returns value of Network object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Network_setCookie(self, name, value, **kwargs):\n\t\tassert isinstance(name, (str,)\n\t\t ), \"Argument 'name' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t name)\n\t\tassert isinstance(value, (str,)\n\t\t ), \"Argument 'value' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tif 'url' in kwargs:\n\t\t\tassert isinstance(kwargs['url'], (str,)\n\t\t\t ), \"Optional argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['url'])\n\t\tif 'domain' in kwargs:\n\t\t\tassert isinstance(kwargs['domain'], (str,)\n\t\t\t ), \"Optional argument 'domain' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['domain'])\n\t\tif 'path' in kwargs:\n\t\t\tassert isinstance(kwargs['path'], (str,)\n\t\t\t ), \"Optional argument 'path' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['path'])\n\t\tif 'secure' in kwargs:\n\t\t\tassert isinstance(kwargs['secure'], (bool,)\n\t\t\t ), \"Optional argument 'secure' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['secure'])\n\t\tif 'httpOnly' in kwargs:\n\t\t\tassert isinstance(kwargs['httpOnly'], (bool,)\n\t\t\t ), \"Optional argument 'httpOnly' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['httpOnly'])\n\t\texpected = ['url', 'domain', 'path', 'secure', 'httpOnly', 'sameSite',\n\t\t 'expires']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['url', 'domain', 'path', 'secure', 'httpOnly', 'sameSite', 'expires']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Network.setCookie', name=name,\n\t\t value=value, **kwargs)\n\t\treturn subdom_funcs", "response": "Function path: Network.setCookie\n\t\t\tDomain: Network\n\t\t\tMethod name: setCookie\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'name' (type: string) -> Cookie name.\n\t\t\t\t\t'value' (type: string) -> Cookie value.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'url' (type: string) -> The request-URI to associate with the setting of the cookie. This value can affect the default domain and path values of the created cookie.\n\t\t\t\t\t'domain' (type: string) -> Cookie domain.\n\t\t\t\t\t'path' (type: string) -> Cookie path.\n\t\t\t\t\t'secure' (type: boolean) -> True if cookie is secure.\n\t\t\t\t\t'httpOnly' (type: boolean) -> True if cookie is http-only.\n\t\t\t\t\t'sameSite' (type: CookieSameSite) -> Cookie SameSite type.\n\t\t\t\t\t'expires' (type: TimeSinceEpoch) -> Cookie expiration date, session cookie if not set\n\t\t\tReturns:\n\t\t\t\t'success' (type: boolean) -> True if successfully set cookie.\n\t\t\n\t\t\tDescription: Sets a cookie with the given cookie data; may overwrite equivalent cookies if they exist."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: Network.setCookies Domain: Network Method name: setCookies WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'cookies' (type: array) -> Cookies to be set. No return value. Description: Sets given cookies.", "response": "def Network_setCookies(self, cookies):\n\t\t\"\"\"\n\t\tFunction path: Network.setCookies\n\t\t\tDomain: Network\n\t\t\tMethod name: setCookies\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'cookies' (type: array) -> Cookies to be set.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets given cookies.\n\t\t\"\"\"\n\t\tassert isinstance(cookies, (list, tuple)\n\t\t ), \"Argument 'cookies' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t cookies)\n\t\tsubdom_funcs = self.synchronous_command('Network.setCookies', cookies=cookies\n\t\t )\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning path: Network.emulateNetworkConditions Domain: Network Method name: emulateNetworkConditions Parameters: Required arguments: 'offline' (type: boolean) -> True to emulate internet disconnection. 'latency' (type: number) -> Minimum latency from request sent to response headers received (ms). 'downloadThroughput' (type: number) -> Maximal aggregated download throughput (bytes/sec). -1 disables download throttling. 'uploadThroughput' (type: number) -> Maximal aggregated upload throughput (bytes/sec). -1 disables upload throttling. Optional arguments: 'connectionType' (type: ConnectionType) -> Connection type if known. No return value. Description: Activates emulation of network conditions.", "response": "def Network_emulateNetworkConditions(self, offline, latency,\n\t downloadThroughput, uploadThroughput, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Network.emulateNetworkConditions\n\t\t\tDomain: Network\n\t\t\tMethod name: emulateNetworkConditions\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'offline' (type: boolean) -> True to emulate internet disconnection.\n\t\t\t\t\t'latency' (type: number) -> Minimum latency from request sent to response headers received (ms).\n\t\t\t\t\t'downloadThroughput' (type: number) -> Maximal aggregated download throughput (bytes/sec). -1 disables download throttling.\n\t\t\t\t\t'uploadThroughput' (type: number) -> Maximal aggregated upload throughput (bytes/sec). -1 disables upload throttling.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'connectionType' (type: ConnectionType) -> Connection type if known.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Activates emulation of network conditions.\n\t\t\"\"\"\n\t\tassert isinstance(offline, (bool,)\n\t\t ), \"Argument 'offline' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t offline)\n\t\tassert isinstance(latency, (float, int)\n\t\t ), \"Argument 'latency' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t latency)\n\t\tassert isinstance(downloadThroughput, (float, int)\n\t\t ), \"Argument 'downloadThroughput' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t downloadThroughput)\n\t\tassert isinstance(uploadThroughput, (float, int)\n\t\t ), \"Argument 'uploadThroughput' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t uploadThroughput)\n\t\texpected = ['connectionType']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['connectionType']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Network.emulateNetworkConditions',\n\t\t offline=offline, latency=latency, downloadThroughput=\n\t\t downloadThroughput, uploadThroughput=uploadThroughput, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: Network.setCacheDisabled Domain: Network Method name: setCacheDisabled Parameters: Required arguments: 'cacheDisabled' (type: boolean) -> Cache disabled state. No return value. Description: Toggles ignoring cache for each request. If true, cache will not be used.", "response": "def Network_setCacheDisabled(self, cacheDisabled):\n\t\t\"\"\"\n\t\tFunction path: Network.setCacheDisabled\n\t\t\tDomain: Network\n\t\t\tMethod name: setCacheDisabled\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'cacheDisabled' (type: boolean) -> Cache disabled state.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Toggles ignoring cache for each request. If true, cache will not be used.\n\t\t\"\"\"\n\t\tassert isinstance(cacheDisabled, (bool,)\n\t\t ), \"Argument 'cacheDisabled' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t cacheDisabled)\n\t\tsubdom_funcs = self.synchronous_command('Network.setCacheDisabled',\n\t\t cacheDisabled=cacheDisabled)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Network_setBypassServiceWorker(self, bypass):\n\t\tassert isinstance(bypass, (bool,)\n\t\t ), \"Argument 'bypass' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t bypass)\n\t\tsubdom_funcs = self.synchronous_command('Network.setBypassServiceWorker',\n\t\t bypass=bypass)\n\t\treturn subdom_funcs", "response": "This method is used to set the bypass of service worker for each request."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: Network.getCertificate Domain: Network Method name: getCertificate WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'origin' (type: string) -> Origin to get certificate for. Returns: 'tableNames' (type: array) -> No description Description: Returns the DER-encoded certificate.", "response": "def Network_getCertificate(self, origin):\n\t\t\"\"\"\n\t\tFunction path: Network.getCertificate\n\t\t\tDomain: Network\n\t\t\tMethod name: getCertificate\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'origin' (type: string) -> Origin to get certificate for.\n\t\t\tReturns:\n\t\t\t\t'tableNames' (type: array) -> No description\n\t\t\n\t\t\tDescription: Returns the DER-encoded certificate.\n\t\t\"\"\"\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tsubdom_funcs = self.synchronous_command('Network.getCertificate', origin=\n\t\t origin)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Network_setRequestInterceptionEnabled(self, enabled, **kwargs):\n\t\tassert isinstance(enabled, (bool,)\n\t\t ), \"Argument 'enabled' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t enabled)\n\t\tif 'patterns' in kwargs:\n\t\t\tassert isinstance(kwargs['patterns'], (list, tuple)\n\t\t\t ), \"Optional argument 'patterns' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t\t kwargs['patterns'])\n\t\texpected = ['patterns']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['patterns']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Network.setRequestInterceptionEnabled', enabled=enabled, **kwargs)\n\t\treturn subdom_funcs", "response": "This function is used to enable or disable requests intercepting."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: Network.continueInterceptedRequest Domain: Network Method name: continueInterceptedRequest WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'interceptionId' (type: InterceptionId) -> No description Optional arguments: 'errorReason' (type: ErrorReason) -> If set this causes the request to fail with the given reason. Passing Aborted for requests marked with isNavigationRequest also cancels the navigation. Must not be set in response to an authChallenge. 'rawResponse' (type: string) -> If set the requests completes using with the provided base64 encoded raw response, including HTTP status line and headers etc... Must not be set in response to an authChallenge. 'url' (type: string) -> If set the request url will be modified in a way that's not observable by page. Must not be set in response to an authChallenge. 'method' (type: string) -> If set this allows the request method to be overridden. Must not be set in response to an authChallenge. 'postData' (type: string) -> If set this allows postData to be set. Must not be set in response to an authChallenge. 'headers' (type: Headers) -> If set this allows the request headers to be changed. Must not be set in response to an authChallenge. 'authChallengeResponse' (type: AuthChallengeResponse) -> Response to a requestIntercepted with an authChallenge. Must not be set otherwise. No return value. Description: Response to Network.requestIntercepted which either modifies the request to continue with any modifications, or blocks it, or completes it with the provided response bytes. If a network fetch occurs as a result which encounters a redirect an additional Network.requestIntercepted event will be sent with the same InterceptionId.", "response": "def Network_continueInterceptedRequest(self, interceptionId, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Network.continueInterceptedRequest\n\t\t\tDomain: Network\n\t\t\tMethod name: continueInterceptedRequest\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'interceptionId' (type: InterceptionId) -> No description\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'errorReason' (type: ErrorReason) -> If set this causes the request to fail with the given reason. Passing Aborted for requests marked with isNavigationRequest also cancels the navigation. Must not be set in response to an authChallenge.\n\t\t\t\t\t'rawResponse' (type: string) -> If set the requests completes using with the provided base64 encoded raw response, including HTTP status line and headers etc... Must not be set in response to an authChallenge.\n\t\t\t\t\t'url' (type: string) -> If set the request url will be modified in a way that's not observable by page. Must not be set in response to an authChallenge.\n\t\t\t\t\t'method' (type: string) -> If set this allows the request method to be overridden. Must not be set in response to an authChallenge.\n\t\t\t\t\t'postData' (type: string) -> If set this allows postData to be set. Must not be set in response to an authChallenge.\n\t\t\t\t\t'headers' (type: Headers) -> If set this allows the request headers to be changed. Must not be set in response to an authChallenge.\n\t\t\t\t\t'authChallengeResponse' (type: AuthChallengeResponse) -> Response to a requestIntercepted with an authChallenge. Must not be set otherwise.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Response to Network.requestIntercepted which either modifies the request to continue with any modifications, or blocks it, or completes it with the provided response bytes. If a network fetch occurs as a result which encounters a redirect an additional Network.requestIntercepted event will be sent with the same InterceptionId.\n\t\t\"\"\"\n\t\tif 'rawResponse' in kwargs:\n\t\t\tassert isinstance(kwargs['rawResponse'], (str,)\n\t\t\t ), \"Optional argument 'rawResponse' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['rawResponse'])\n\t\tif 'url' in kwargs:\n\t\t\tassert isinstance(kwargs['url'], (str,)\n\t\t\t ), \"Optional argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['url'])\n\t\tif 'method' in kwargs:\n\t\t\tassert isinstance(kwargs['method'], (str,)\n\t\t\t ), \"Optional argument 'method' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['method'])\n\t\tif 'postData' in kwargs:\n\t\t\tassert isinstance(kwargs['postData'], (str,)\n\t\t\t ), \"Optional argument 'postData' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['postData'])\n\t\texpected = ['errorReason', 'rawResponse', 'url', 'method', 'postData',\n\t\t 'headers', 'authChallengeResponse']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['errorReason', 'rawResponse', 'url', 'method', 'postData', 'headers', 'authChallengeResponse']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Network.continueInterceptedRequest',\n\t\t interceptionId=interceptionId, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef Database_executeSQL(self, databaseId, query):\n\t\tassert isinstance(query, (str,)\n\t\t ), \"Argument 'query' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t query)\n\t\tsubdom_funcs = self.synchronous_command('Database.executeSQL', databaseId\n\t\t =databaseId, query=query)\n\t\treturn subdom_funcs", "response": "Function path: Database.executeSQL\n\t\t\tDomain: Database\n\t\t\tMethod name: executeSQL\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'databaseId' (type: DatabaseId) -> No description\n\t\t\t\t\t'query' (type: string) -> No description\n\t\t\tReturns:\n\t\t\t\t'columnNames' (type: array) -> No description\n\t\t\t\t'values' (type: array) -> No description\n\t\t\t\t'sqlError' (type: Error) -> No description"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef IndexedDB_requestDatabaseNames(self, securityOrigin):\n\t\tassert isinstance(securityOrigin, (str,)\n\t\t ), \"Argument 'securityOrigin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t securityOrigin)\n\t\tsubdom_funcs = self.synchronous_command('IndexedDB.requestDatabaseNames',\n\t\t securityOrigin=securityOrigin)\n\t\treturn subdom_funcs", "response": "This method is used to request the names of databases for a given security origin."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef IndexedDB_requestData(self, securityOrigin, databaseName,\n\t objectStoreName, indexName, skipCount, pageSize, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: IndexedDB.requestData\n\t\t\tDomain: IndexedDB\n\t\t\tMethod name: requestData\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'securityOrigin' (type: string) -> Security origin.\n\t\t\t\t\t'databaseName' (type: string) -> Database name.\n\t\t\t\t\t'objectStoreName' (type: string) -> Object store name.\n\t\t\t\t\t'indexName' (type: string) -> Index name, empty string for object store data requests.\n\t\t\t\t\t'skipCount' (type: integer) -> Number of records to skip.\n\t\t\t\t\t'pageSize' (type: integer) -> Number of records to fetch.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'keyRange' (type: KeyRange) -> Key range.\n\t\t\tReturns:\n\t\t\t\t'objectStoreDataEntries' (type: array) -> Array of object store data entries.\n\t\t\t\t'hasMore' (type: boolean) -> If true, there are more entries to fetch in the given range.\n\t\t\n\t\t\tDescription: Requests data from object store or index.\n\t\t\"\"\"\n\t\tassert isinstance(securityOrigin, (str,)\n\t\t ), \"Argument 'securityOrigin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t securityOrigin)\n\t\tassert isinstance(databaseName, (str,)\n\t\t ), \"Argument 'databaseName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t databaseName)\n\t\tassert isinstance(objectStoreName, (str,)\n\t\t ), \"Argument 'objectStoreName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t objectStoreName)\n\t\tassert isinstance(indexName, (str,)\n\t\t ), \"Argument 'indexName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t indexName)\n\t\tassert isinstance(skipCount, (int,)\n\t\t ), \"Argument 'skipCount' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t skipCount)\n\t\tassert isinstance(pageSize, (int,)\n\t\t ), \"Argument 'pageSize' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t pageSize)\n\t\texpected = ['keyRange']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['keyRange']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('IndexedDB.requestData',\n\t\t securityOrigin=securityOrigin, databaseName=databaseName,\n\t\t objectStoreName=objectStoreName, indexName=indexName, skipCount=\n\t\t skipCount, pageSize=pageSize, **kwargs)\n\t\treturn subdom_funcs", "response": "This method is used to make a request to the data of an object store or index. This method is used to make a request to the data of an object store or index."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef IndexedDB_clearObjectStore(self, securityOrigin, databaseName,\n\t objectStoreName):\n\t\t\"\"\"\n\t\tFunction path: IndexedDB.clearObjectStore\n\t\t\tDomain: IndexedDB\n\t\t\tMethod name: clearObjectStore\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'securityOrigin' (type: string) -> Security origin.\n\t\t\t\t\t'databaseName' (type: string) -> Database name.\n\t\t\t\t\t'objectStoreName' (type: string) -> Object store name.\n\t\t\tReturns:\n\t\t\n\t\t\tDescription: Clears all entries from an object store.\n\t\t\"\"\"\n\t\tassert isinstance(securityOrigin, (str,)\n\t\t ), \"Argument 'securityOrigin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t securityOrigin)\n\t\tassert isinstance(databaseName, (str,)\n\t\t ), \"Argument 'databaseName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t databaseName)\n\t\tassert isinstance(objectStoreName, (str,)\n\t\t ), \"Argument 'objectStoreName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t objectStoreName)\n\t\tsubdom_funcs = self.synchronous_command('IndexedDB.clearObjectStore',\n\t\t securityOrigin=securityOrigin, databaseName=databaseName,\n\t\t objectStoreName=objectStoreName)\n\t\treturn subdom_funcs", "response": "This is the IndexedDB clearObjectStore operation. This operation is idempotent and will return a list of all the entries in an object store."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef IndexedDB_deleteDatabase(self, securityOrigin, databaseName):\n\t\tassert isinstance(securityOrigin, (str,)\n\t\t ), \"Argument 'securityOrigin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t securityOrigin)\n\t\tassert isinstance(databaseName, (str,)\n\t\t ), \"Argument 'databaseName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t databaseName)\n\t\tsubdom_funcs = self.synchronous_command('IndexedDB.deleteDatabase',\n\t\t securityOrigin=securityOrigin, databaseName=databaseName)\n\t\treturn subdom_funcs", "response": "This method deletes a database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: CacheStorage.requestCacheNames Domain: CacheStorage Method name: requestCacheNames Parameters: Required arguments: 'securityOrigin' (type: string) -> Security origin. Returns: 'caches' (type: array) -> Caches for the security origin. Description: Requests cache names.", "response": "def CacheStorage_requestCacheNames(self, securityOrigin):\n\t\t\"\"\"\n\t\tFunction path: CacheStorage.requestCacheNames\n\t\t\tDomain: CacheStorage\n\t\t\tMethod name: requestCacheNames\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'securityOrigin' (type: string) -> Security origin.\n\t\t\tReturns:\n\t\t\t\t'caches' (type: array) -> Caches for the security origin.\n\t\t\n\t\t\tDescription: Requests cache names.\n\t\t\"\"\"\n\t\tassert isinstance(securityOrigin, (str,)\n\t\t ), \"Argument 'securityOrigin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t securityOrigin)\n\t\tsubdom_funcs = self.synchronous_command('CacheStorage.requestCacheNames',\n\t\t securityOrigin=securityOrigin)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef CacheStorage_requestEntries(self, cacheId, skipCount, pageSize):\n\t\tassert isinstance(skipCount, (int,)\n\t\t ), \"Argument 'skipCount' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t skipCount)\n\t\tassert isinstance(pageSize, (int,)\n\t\t ), \"Argument 'pageSize' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t pageSize)\n\t\tsubdom_funcs = self.synchronous_command('CacheStorage.requestEntries',\n\t\t cacheId=cacheId, skipCount=skipCount, pageSize=pageSize)\n\t\treturn subdom_funcs", "response": "Cache Storage requestEntries operation."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction path: CacheStorage.deleteEntry Domain: CacheStorage Method name: deleteEntry Parameters: Required arguments: 'cacheId' (type: CacheId) -> Id of cache where the entry will be deleted. 'request' (type: string) -> URL spec of the request. No return value. Description: Deletes a cache entry.", "response": "def CacheStorage_deleteEntry(self, cacheId, request):\n\t\t\"\"\"\n\t\tFunction path: CacheStorage.deleteEntry\n\t\t\tDomain: CacheStorage\n\t\t\tMethod name: deleteEntry\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'cacheId' (type: CacheId) -> Id of cache where the entry will be deleted.\n\t\t\t\t\t'request' (type: string) -> URL spec of the request.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Deletes a cache entry.\n\t\t\"\"\"\n\t\tassert isinstance(request, (str,)\n\t\t ), \"Argument 'request' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t request)\n\t\tsubdom_funcs = self.synchronous_command('CacheStorage.deleteEntry',\n\t\t cacheId=cacheId, request=request)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef CacheStorage_requestCachedResponse(self, cacheId, requestURL):\n\t\tassert isinstance(requestURL, (str,)\n\t\t ), \"Argument 'requestURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t requestURL)\n\t\tsubdom_funcs = self.synchronous_command('CacheStorage.requestCachedResponse',\n\t\t cacheId=cacheId, requestURL=requestURL)\n\t\treturn subdom_funcs", "response": "This method is used to request the response from the cache entry."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: DOMStorage.setDOMStorageItem Domain: DOMStorage Method name: setDOMStorageItem Parameters: Required arguments: 'storageId' (type: StorageId) -> No description 'key' (type: string) -> No description 'value' (type: string) -> No description No return value.", "response": "def DOMStorage_setDOMStorageItem(self, storageId, key, value):\n\t\t\"\"\"\n\t\tFunction path: DOMStorage.setDOMStorageItem\n\t\t\tDomain: DOMStorage\n\t\t\tMethod name: setDOMStorageItem\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'storageId' (type: StorageId) -> No description\n\t\t\t\t\t'key' (type: string) -> No description\n\t\t\t\t\t'value' (type: string) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(key, (str,)\n\t\t ), \"Argument 'key' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t key)\n\t\tassert isinstance(value, (str,)\n\t\t ), \"Argument 'value' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command('DOMStorage.setDOMStorageItem',\n\t\t storageId=storageId, key=key, value=value)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef DOMStorage_removeDOMStorageItem(self, storageId, key):\n\t\tassert isinstance(key, (str,)\n\t\t ), \"Argument 'key' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t key)\n\t\tsubdom_funcs = self.synchronous_command('DOMStorage.removeDOMStorageItem',\n\t\t storageId=storageId, key=key)\n\t\treturn subdom_funcs", "response": "description: Remove a DOM storage item."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nqueries selector on a given node.", "response": "def DOM_querySelector(self, nodeId, selector):\n\t\t\"\"\"\n\t\tFunction path: DOM.querySelector\n\t\t\tDomain: DOM\n\t\t\tMethod name: querySelector\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the node to query upon.\n\t\t\t\t\t'selector' (type: string) -> Selector string.\n\t\t\tReturns:\n\t\t\t\t'nodeId' (type: NodeId) -> Query selector result.\n\t\t\n\t\t\tDescription: Executes querySelector on a given node.\n\t\t\"\"\"\n\t\tassert isinstance(selector, (str,)\n\t\t ), \"Argument 'selector' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t selector)\n\t\tsubdom_funcs = self.synchronous_command('DOM.querySelector', nodeId=\n\t\t nodeId, selector=selector)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: DOM.setNodeName Domain: DOM Method name: setNodeName Parameters: Required arguments: 'nodeId' (type: NodeId) -> Id of the node to set name for. 'name' (type: string) -> New node's name. Returns: 'nodeId' (type: NodeId) -> New node's id. Description: Sets node name for a node with given id.", "response": "def DOM_setNodeName(self, nodeId, name):\n\t\t\"\"\"\n\t\tFunction path: DOM.setNodeName\n\t\t\tDomain: DOM\n\t\t\tMethod name: setNodeName\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the node to set name for.\n\t\t\t\t\t'name' (type: string) -> New node's name.\n\t\t\tReturns:\n\t\t\t\t'nodeId' (type: NodeId) -> New node's id.\n\t\t\n\t\t\tDescription: Sets node name for a node with given id.\n\t\t\"\"\"\n\t\tassert isinstance(name, (str,)\n\t\t ), \"Argument 'name' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t name)\n\t\tsubdom_funcs = self.synchronous_command('DOM.setNodeName', nodeId=nodeId,\n\t\t name=name)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef DOM_setNodeValue(self, nodeId, value):\n\t\tassert isinstance(value, (str,)\n\t\t ), \"Argument 'value' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command('DOM.setNodeValue', nodeId=nodeId,\n\t\t value=value)\n\t\treturn subdom_funcs", "response": "This method is used to set the value of a node."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef DOM_setAttributeValue(self, nodeId, name, value):\n\t\tassert isinstance(name, (str,)\n\t\t ), \"Argument 'name' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t name)\n\t\tassert isinstance(value, (str,)\n\t\t ), \"Argument 'value' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command('DOM.setAttributeValue', nodeId=\n\t\t nodeId, name=name, value=value)\n\t\treturn subdom_funcs", "response": "Function path: DOM.setAttributeValue\n\t\t\tDomain: DOM\n\t\t\tMethod name: setAttributeValue\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the element to set attribute for.\n\t\t\t\t\t'name' (type: string) -> Attribute name.\n\t\t\t\t\t'value' (type: string) -> Attribute value.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets attribute for an element with given id."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: DOM.setAttributesAsText Domain: DOM Method name: setAttributesAsText Parameters: Required arguments: 'nodeId' (type: NodeId) -> Id of the element to set attributes for. 'text' (type: string) -> Text with a number of attributes. Will parse this text using HTML parser. Optional arguments: 'name' (type: string) -> Attribute name to replace with new attributes derived from text in case text parsed successfully. No return value. Description: Sets attributes on element with given id. This method is useful when user edits some existing attribute value and types in several attribute name/value pairs.", "response": "def DOM_setAttributesAsText(self, nodeId, text, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: DOM.setAttributesAsText\n\t\t\tDomain: DOM\n\t\t\tMethod name: setAttributesAsText\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the element to set attributes for.\n\t\t\t\t\t'text' (type: string) -> Text with a number of attributes. Will parse this text using HTML parser.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'name' (type: string) -> Attribute name to replace with new attributes derived from text in case text parsed successfully.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets attributes on element with given id. This method is useful when user edits some existing attribute value and types in several attribute name/value pairs.\n\t\t\"\"\"\n\t\tassert isinstance(text, (str,)\n\t\t ), \"Argument 'text' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t text)\n\t\tif 'name' in kwargs:\n\t\t\tassert isinstance(kwargs['name'], (str,)\n\t\t\t ), \"Optional argument 'name' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['name'])\n\t\texpected = ['name']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['name']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('DOM.setAttributesAsText', nodeId\n\t\t =nodeId, text=text, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: DOM.setOuterHTML Domain: DOM Method name: setOuterHTML Parameters: Required arguments: 'nodeId' (type: NodeId) -> Id of the node to set markup for. 'outerHTML' (type: string) -> Outer HTML markup to set. No return value. Description: Sets node HTML markup, returns new node id.", "response": "def DOM_setOuterHTML(self, nodeId, outerHTML):\n\t\t\"\"\"\n\t\tFunction path: DOM.setOuterHTML\n\t\t\tDomain: DOM\n\t\t\tMethod name: setOuterHTML\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the node to set markup for.\n\t\t\t\t\t'outerHTML' (type: string) -> Outer HTML markup to set.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets node HTML markup, returns new node id.\n\t\t\"\"\"\n\t\tassert isinstance(outerHTML, (str,)\n\t\t ), \"Argument 'outerHTML' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t outerHTML)\n\t\tsubdom_funcs = self.synchronous_command('DOM.setOuterHTML', nodeId=nodeId,\n\t\t outerHTML=outerHTML)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef DOM_getSearchResults(self, searchId, fromIndex, toIndex):\n\t\tassert isinstance(searchId, (str,)\n\t\t ), \"Argument 'searchId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t searchId)\n\t\tassert isinstance(fromIndex, (int,)\n\t\t ), \"Argument 'fromIndex' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t fromIndex)\n\t\tassert isinstance(toIndex, (int,)\n\t\t ), \"Argument 'toIndex' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t toIndex)\n\t\tsubdom_funcs = self.synchronous_command('DOM.getSearchResults', searchId=\n\t\t searchId, fromIndex=fromIndex, toIndex=toIndex)\n\t\treturn subdom_funcs", "response": "This method returns the search results from the given search session identifier."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions path: DOM.discardSearchResults Domain: DOM Method name: discardSearchResults WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'searchId' (type: string) -> Unique search session identifier. No return value. Description: Discards search results from the session with the given id. getSearchResults should no longer be called for that search.", "response": "def DOM_discardSearchResults(self, searchId):\n\t\t\"\"\"\n\t\tFunction path: DOM.discardSearchResults\n\t\t\tDomain: DOM\n\t\t\tMethod name: discardSearchResults\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'searchId' (type: string) -> Unique search session identifier.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Discards search results from the session with the given id. getSearchResults should no longer be called for that search.\n\t\t\"\"\"\n\t\tassert isinstance(searchId, (str,)\n\t\t ), \"Argument 'searchId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t searchId)\n\t\tsubdom_funcs = self.synchronous_command('DOM.discardSearchResults',\n\t\t searchId=searchId)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef DOM_pushNodeByPathToFrontend(self, path):\n\t\tassert isinstance(path, (str,)\n\t\t ), \"Argument 'path' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t path)\n\t\tsubdom_funcs = self.synchronous_command('DOM.pushNodeByPathToFrontend',\n\t\t path=path)\n\t\treturn subdom_funcs", "response": "This method is used to push a node to the frontend."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning path: DOM.pushNodesByBackendIdsToFrontend Domain: DOM Method name: pushNodesByBackendIdsToFrontend WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'backendNodeIds' (type: array) -> The array of backend node ids. Returns: 'nodeIds' (type: array) -> The array of ids of pushed nodes that correspond to the backend ids specified in backendNodeIds. Description: Requests that a batch of nodes is sent to the caller given their backend node ids.", "response": "def DOM_pushNodesByBackendIdsToFrontend(self, backendNodeIds):\n\t\t\"\"\"\n\t\tFunction path: DOM.pushNodesByBackendIdsToFrontend\n\t\t\tDomain: DOM\n\t\t\tMethod name: pushNodesByBackendIdsToFrontend\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'backendNodeIds' (type: array) -> The array of backend node ids.\n\t\t\tReturns:\n\t\t\t\t'nodeIds' (type: array) -> The array of ids of pushed nodes that correspond to the backend ids specified in backendNodeIds.\n\t\t\n\t\t\tDescription: Requests that a batch of nodes is sent to the caller given their backend node ids.\n\t\t\"\"\"\n\t\tassert isinstance(backendNodeIds, (list, tuple)\n\t\t ), \"Argument 'backendNodeIds' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t backendNodeIds)\n\t\tsubdom_funcs = self.synchronous_command('DOM.pushNodesByBackendIdsToFrontend'\n\t\t , backendNodeIds=backendNodeIds)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfunctions path: DOM.copyTo Domain: DOM Method name: copyTo WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'nodeId' (type: NodeId) -> Id of the node to copy. 'targetNodeId' (type: NodeId) -> Id of the element to drop the copy into. Optional arguments: 'insertBeforeNodeId' (type: NodeId) -> Drop the copy before this node (if absent, the copy becomes the last child of targetNodeId). Returns: 'nodeId' (type: NodeId) -> Id of the node clone. Description: Creates a deep copy of the specified node and places it into the target container before the given anchor.", "response": "def DOM_copyTo(self, nodeId, targetNodeId, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: DOM.copyTo\n\t\t\tDomain: DOM\n\t\t\tMethod name: copyTo\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: NodeId) -> Id of the node to copy.\n\t\t\t\t\t'targetNodeId' (type: NodeId) -> Id of the element to drop the copy into.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'insertBeforeNodeId' (type: NodeId) -> Drop the copy before this node (if absent, the copy becomes the last child of targetNodeId).\n\t\t\tReturns:\n\t\t\t\t'nodeId' (type: NodeId) -> Id of the node clone.\n\t\t\n\t\t\tDescription: Creates a deep copy of the specified node and places it into the target container before the given anchor.\n\t\t\"\"\"\n\t\texpected = ['insertBeforeNodeId']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['insertBeforeNodeId']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('DOM.copyTo', nodeId=nodeId,\n\t\t targetNodeId=targetNodeId, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef DOM_setFileInputFiles(self, files, **kwargs):\n\t\tassert isinstance(files, (list, tuple)\n\t\t ), \"Argument 'files' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t files)\n\t\texpected = ['nodeId', 'backendNodeId', 'objectId']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['nodeId', 'backendNodeId', 'objectId']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('DOM.setFileInputFiles', files=\n\t\t files, **kwargs)\n\t\treturn subdom_funcs", "response": "This method is used to set the files for the given file input element."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: DOM.getNodeForLocation Domain: DOM Method name: getNodeForLocation WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'x' (type: integer) -> X coordinate. 'y' (type: integer) -> Y coordinate. Optional arguments: 'includeUserAgentShadowDOM' (type: boolean) -> False to skip to the nearest non-UA shadow root ancestor (default: false). Returns: 'nodeId' (type: NodeId) -> Id of the node at given coordinates. Description: Returns node id at given location.", "response": "def DOM_getNodeForLocation(self, x, y, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: DOM.getNodeForLocation\n\t\t\tDomain: DOM\n\t\t\tMethod name: getNodeForLocation\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'x' (type: integer) -> X coordinate.\n\t\t\t\t\t'y' (type: integer) -> Y coordinate.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'includeUserAgentShadowDOM' (type: boolean) -> False to skip to the nearest non-UA shadow root ancestor (default: false).\n\t\t\tReturns:\n\t\t\t\t'nodeId' (type: NodeId) -> Id of the node at given coordinates.\n\t\t\n\t\t\tDescription: Returns node id at given location.\n\t\t\"\"\"\n\t\tassert isinstance(x, (int,)\n\t\t ), \"Argument 'x' must be of type '['int']'. Received type: '%s'\" % type(x\n\t\t )\n\t\tassert isinstance(y, (int,)\n\t\t ), \"Argument 'y' must be of type '['int']'. Received type: '%s'\" % type(y\n\t\t )\n\t\tif 'includeUserAgentShadowDOM' in kwargs:\n\t\t\tassert isinstance(kwargs['includeUserAgentShadowDOM'], (bool,)\n\t\t\t ), \"Optional argument 'includeUserAgentShadowDOM' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['includeUserAgentShadowDOM'])\n\t\texpected = ['includeUserAgentShadowDOM']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['includeUserAgentShadowDOM']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('DOM.getNodeForLocation', x=x, y=\n\t\t y, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef DOM_describeNode(self, **kwargs):\n\t\tif 'depth' in kwargs:\n\t\t\tassert isinstance(kwargs['depth'], (int,)\n\t\t\t ), \"Optional argument 'depth' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['depth'])\n\t\tif 'pierce' in kwargs:\n\t\t\tassert isinstance(kwargs['pierce'], (bool,)\n\t\t\t ), \"Optional argument 'pierce' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['pierce'])\n\t\texpected = ['nodeId', 'backendNodeId', 'objectId', 'depth', 'pierce']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['nodeId', 'backendNodeId', 'objectId', 'depth', 'pierce']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('DOM.describeNode', **kwargs)\n\t\treturn subdom_funcs", "response": "This method returns a description of a node."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef CSS_setStyleSheetText(self, styleSheetId, text):\n\t\tassert isinstance(text, (str,)\n\t\t ), \"Argument 'text' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t text)\n\t\tsubdom_funcs = self.synchronous_command('CSS.setStyleSheetText',\n\t\t styleSheetId=styleSheetId, text=text)\n\t\treturn subdom_funcs", "response": "This command is executed when the user changes the text of a stylesheet."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef CSS_setRuleSelector(self, styleSheetId, range, selector):\n\t\tassert isinstance(selector, (str,)\n\t\t ), \"Argument 'selector' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t selector)\n\t\tsubdom_funcs = self.synchronous_command('CSS.setRuleSelector',\n\t\t styleSheetId=styleSheetId, range=range, selector=selector)\n\t\treturn subdom_funcs", "response": "This method is used to set the rule selector in a style sheet."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef CSS_setKeyframeKey(self, styleSheetId, range, keyText):\n\t\tassert isinstance(keyText, (str,)\n\t\t ), \"Argument 'keyText' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t keyText)\n\t\tsubdom_funcs = self.synchronous_command('CSS.setKeyframeKey',\n\t\t styleSheetId=styleSheetId, range=range, keyText=keyText)\n\t\treturn subdom_funcs", "response": "This command is used to set the keyframe rule key text."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef CSS_setStyleTexts(self, edits):\n\t\tassert isinstance(edits, (list, tuple)\n\t\t ), \"Argument 'edits' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t edits)\n\t\tsubdom_funcs = self.synchronous_command('CSS.setStyleTexts', edits=edits)\n\t\treturn subdom_funcs", "response": "This method is used to set the style texts of the selected elements in the editor."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfunction path: CSS.setMediaText Domain: CSS Method name: setMediaText Parameters: Required arguments: 'styleSheetId' (type: StyleSheetId) -> No description 'range' (type: SourceRange) -> No description 'text' (type: string) -> No description Returns: 'media' (type: CSSMedia) -> The resulting CSS media rule after modification. Description: Modifies the rule selector.", "response": "def CSS_setMediaText(self, styleSheetId, range, text):\n\t\t\"\"\"\n\t\tFunction path: CSS.setMediaText\n\t\t\tDomain: CSS\n\t\t\tMethod name: setMediaText\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'styleSheetId' (type: StyleSheetId) -> No description\n\t\t\t\t\t'range' (type: SourceRange) -> No description\n\t\t\t\t\t'text' (type: string) -> No description\n\t\t\tReturns:\n\t\t\t\t'media' (type: CSSMedia) -> The resulting CSS media rule after modification.\n\t\t\n\t\t\tDescription: Modifies the rule selector.\n\t\t\"\"\"\n\t\tassert isinstance(text, (str,)\n\t\t ), \"Argument 'text' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t text)\n\t\tsubdom_funcs = self.synchronous_command('CSS.setMediaText', styleSheetId=\n\t\t styleSheetId, range=range, text=text)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: CSS.addRule Domain: CSS Method name: addRule Parameters: Required arguments: 'styleSheetId' (type: StyleSheetId) -> The css style sheet identifier where a new rule should be inserted. 'ruleText' (type: string) -> The text of a new rule. 'location' (type: SourceRange) -> Text position of a new rule in the target style sheet. Returns: 'rule' (type: CSSRule) -> The newly created rule. Description: Inserts a new rule with the given ruleText in a stylesheet with given styleSheetId, at the position specified by location.", "response": "def CSS_addRule(self, styleSheetId, ruleText, location):\n\t\t\"\"\"\n\t\tFunction path: CSS.addRule\n\t\t\tDomain: CSS\n\t\t\tMethod name: addRule\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'styleSheetId' (type: StyleSheetId) -> The css style sheet identifier where a new rule should be inserted.\n\t\t\t\t\t'ruleText' (type: string) -> The text of a new rule.\n\t\t\t\t\t'location' (type: SourceRange) -> Text position of a new rule in the target style sheet.\n\t\t\tReturns:\n\t\t\t\t'rule' (type: CSSRule) -> The newly created rule.\n\t\t\n\t\t\tDescription: Inserts a new rule with the given ruleText in a stylesheet with given styleSheetId, at the position specified by location.\n\t\t\"\"\"\n\t\tassert isinstance(ruleText, (str,)\n\t\t ), \"Argument 'ruleText' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t ruleText)\n\t\tsubdom_funcs = self.synchronous_command('CSS.addRule', styleSheetId=\n\t\t styleSheetId, ruleText=ruleText, location=location)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef CSS_forcePseudoState(self, nodeId, forcedPseudoClasses):\n\t\tassert isinstance(forcedPseudoClasses, (list, tuple)\n\t\t ), \"Argument 'forcedPseudoClasses' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t forcedPseudoClasses)\n\t\tsubdom_funcs = self.synchronous_command('CSS.forcePseudoState', nodeId=\n\t\t nodeId, forcedPseudoClasses=forcedPseudoClasses)\n\t\treturn subdom_funcs", "response": "This method allows you to force the pseudo state of an element."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef CSS_setEffectivePropertyValueForNode(self, nodeId, propertyName, value):\n\t\tassert isinstance(propertyName, (str,)\n\t\t ), \"Argument 'propertyName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t propertyName)\n\t\tassert isinstance(value, (str,)\n\t\t ), \"Argument 'value' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'CSS.setEffectivePropertyValueForNode', nodeId=nodeId, propertyName=\n\t\t propertyName, value=value)\n\t\treturn subdom_funcs", "response": "This method is used to set the value of the given property on the given node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: DOMSnapshot.getSnapshot Domain: DOMSnapshot Method name: getSnapshot Parameters: Required arguments: 'computedStyleWhitelist' (type: array) -> Whitelist of computed styles to return. Returns: 'domNodes' (type: array) -> The nodes in the DOM tree. The DOMNode at index 0 corresponds to the root document. 'layoutTreeNodes' (type: array) -> The nodes in the layout tree. 'computedStyles' (type: array) -> Whitelisted ComputedStyle properties for each node in the layout tree. Description: Returns a document snapshot, including the full DOM tree of the root node (including iframes, template contents, and imported documents) in a flattened array, as well as layout and white-listed computed style information for the nodes. Shadow DOM in the returned DOM tree is flattened.", "response": "def DOMSnapshot_getSnapshot(self, computedStyleWhitelist):\n\t\t\"\"\"\n\t\tFunction path: DOMSnapshot.getSnapshot\n\t\t\tDomain: DOMSnapshot\n\t\t\tMethod name: getSnapshot\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'computedStyleWhitelist' (type: array) -> Whitelist of computed styles to return.\n\t\t\tReturns:\n\t\t\t\t'domNodes' (type: array) -> The nodes in the DOM tree. The DOMNode at index 0 corresponds to the root document.\n\t\t\t\t'layoutTreeNodes' (type: array) -> The nodes in the layout tree.\n\t\t\t\t'computedStyles' (type: array) -> Whitelisted ComputedStyle properties for each node in the layout tree.\n\t\t\n\t\t\tDescription: Returns a document snapshot, including the full DOM tree of the root node (including iframes, template contents, and imported documents) in a flattened array, as well as layout and white-listed computed style information for the nodes. Shadow DOM in the returned DOM tree is flattened. \n\t\t\"\"\"\n\t\tassert isinstance(computedStyleWhitelist, (list, tuple)\n\t\t ), \"Argument 'computedStyleWhitelist' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t computedStyleWhitelist)\n\t\tsubdom_funcs = self.synchronous_command('DOMSnapshot.getSnapshot',\n\t\t computedStyleWhitelist=computedStyleWhitelist)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: DOMDebugger.setDOMBreakpoint Domain: DOMDebugger Method name: setDOMBreakpoint Parameters: Required arguments: 'nodeId' (type: DOM.NodeId) -> Identifier of the node to set breakpoint on. 'type' (type: DOMBreakpointType) -> Type of the operation to stop upon. No return value. Description: Sets breakpoint on particular operation with DOM.", "response": "def DOMDebugger_setDOMBreakpoint(self, nodeId, type):\n\t\t\"\"\"\n\t\tFunction path: DOMDebugger.setDOMBreakpoint\n\t\t\tDomain: DOMDebugger\n\t\t\tMethod name: setDOMBreakpoint\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: DOM.NodeId) -> Identifier of the node to set breakpoint on.\n\t\t\t\t\t'type' (type: DOMBreakpointType) -> Type of the operation to stop upon.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets breakpoint on particular operation with DOM.\n\t\t\"\"\"\n\t\tsubdom_funcs = self.synchronous_command('DOMDebugger.setDOMBreakpoint',\n\t\t nodeId=nodeId, type=type)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: DOMDebugger.removeDOMBreakpoint Domain: DOMDebugger Method name: removeDOMBreakpoint Parameters: Required arguments: 'nodeId' (type: DOM.NodeId) -> Identifier of the node to remove breakpoint from. 'type' (type: DOMBreakpointType) -> Type of the breakpoint to remove. No return value. Description: Removes DOM breakpoint that was set using setDOMBreakpoint.", "response": "def DOMDebugger_removeDOMBreakpoint(self, nodeId, type):\n\t\t\"\"\"\n\t\tFunction path: DOMDebugger.removeDOMBreakpoint\n\t\t\tDomain: DOMDebugger\n\t\t\tMethod name: removeDOMBreakpoint\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'nodeId' (type: DOM.NodeId) -> Identifier of the node to remove breakpoint from.\n\t\t\t\t\t'type' (type: DOMBreakpointType) -> Type of the breakpoint to remove.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Removes DOM breakpoint that was set using setDOMBreakpoint.\n\t\t\"\"\"\n\t\tsubdom_funcs = self.synchronous_command('DOMDebugger.removeDOMBreakpoint',\n\t\t nodeId=nodeId, type=type)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: DOMDebugger.setInstrumentationBreakpoint Domain: DOMDebugger Method name: setInstrumentationBreakpoint WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'eventName' (type: string) -> Instrumentation name to stop on. No return value. Description: Sets breakpoint on particular native event.", "response": "def DOMDebugger_setInstrumentationBreakpoint(self, eventName):\n\t\t\"\"\"\n\t\tFunction path: DOMDebugger.setInstrumentationBreakpoint\n\t\t\tDomain: DOMDebugger\n\t\t\tMethod name: setInstrumentationBreakpoint\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'eventName' (type: string) -> Instrumentation name to stop on.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets breakpoint on particular native event.\n\t\t\"\"\"\n\t\tassert isinstance(eventName, (str,)\n\t\t ), \"Argument 'eventName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t eventName)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'DOMDebugger.setInstrumentationBreakpoint', eventName=eventName)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: DOMDebugger.removeInstrumentationBreakpoint Domain: DOMDebugger Method name: removeInstrumentationBreakpoint WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'eventName' (type: string) -> Instrumentation name to stop on. No return value. Description: Removes breakpoint on particular native event.", "response": "def DOMDebugger_removeInstrumentationBreakpoint(self, eventName):\n\t\t\"\"\"\n\t\tFunction path: DOMDebugger.removeInstrumentationBreakpoint\n\t\t\tDomain: DOMDebugger\n\t\t\tMethod name: removeInstrumentationBreakpoint\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'eventName' (type: string) -> Instrumentation name to stop on.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Removes breakpoint on particular native event.\n\t\t\"\"\"\n\t\tassert isinstance(eventName, (str,)\n\t\t ), \"Argument 'eventName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t eventName)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'DOMDebugger.removeInstrumentationBreakpoint', eventName=eventName)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef DOMDebugger_setXHRBreakpoint(self, url):\n\t\tassert isinstance(url, (str,)\n\t\t ), \"Argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t url)\n\t\tsubdom_funcs = self.synchronous_command('DOMDebugger.setXHRBreakpoint',\n\t\t url=url)\n\t\treturn subdom_funcs", "response": "setXHRBreakpoint - Sets breakpoint on XMLHttpRequest."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: DOMDebugger.removeXHRBreakpoint Domain: DOMDebugger Method name: removeXHRBreakpoint Parameters: Required arguments: 'url' (type: string) -> Resource URL substring. No return value. Description: Removes breakpoint from XMLHttpRequest.", "response": "def DOMDebugger_removeXHRBreakpoint(self, url):\n\t\t\"\"\"\n\t\tFunction path: DOMDebugger.removeXHRBreakpoint\n\t\t\tDomain: DOMDebugger\n\t\t\tMethod name: removeXHRBreakpoint\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'url' (type: string) -> Resource URL substring.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Removes breakpoint from XMLHttpRequest.\n\t\t\"\"\"\n\t\tassert isinstance(url, (str,)\n\t\t ), \"Argument 'url' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t url)\n\t\tsubdom_funcs = self.synchronous_command('DOMDebugger.removeXHRBreakpoint',\n\t\t url=url)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Target_setDiscoverTargets(self, discover):\n\t\tassert isinstance(discover, (bool,)\n\t\t ), \"Argument 'discover' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t discover)\n\t\tsubdom_funcs = self.synchronous_command('Target.setDiscoverTargets',\n\t\t discover=discover)\n\t\treturn subdom_funcs", "response": "Target. setDiscoverTargets - Sets whether to discover available targets and notify via targetCreated and targetInfoChanged events."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: Target.setAutoAttach Domain: Target Method name: setAutoAttach Parameters: Required arguments: 'autoAttach' (type: boolean) -> Whether to auto-attach to related targets. 'waitForDebuggerOnStart' (type: boolean) -> Whether to pause new targets when attaching to them. Use Runtime.runIfWaitingForDebugger to run paused targets. No return value. Description: Controls whether to automatically attach to new targets which are considered to be related to this one. When turned on, attaches to all existing related targets as well. When turned off, automatically detaches from all currently attached targets.", "response": "def Target_setAutoAttach(self, autoAttach, waitForDebuggerOnStart):\n\t\t\"\"\"\n\t\tFunction path: Target.setAutoAttach\n\t\t\tDomain: Target\n\t\t\tMethod name: setAutoAttach\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'autoAttach' (type: boolean) -> Whether to auto-attach to related targets.\n\t\t\t\t\t'waitForDebuggerOnStart' (type: boolean) -> Whether to pause new targets when attaching to them. Use Runtime.runIfWaitingForDebugger to run paused targets.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Controls whether to automatically attach to new targets which are considered to be related to this one. When turned on, attaches to all existing related targets as well. When turned off, automatically detaches from all currently attached targets.\n\t\t\"\"\"\n\t\tassert isinstance(autoAttach, (bool,)\n\t\t ), \"Argument 'autoAttach' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t autoAttach)\n\t\tassert isinstance(waitForDebuggerOnStart, (bool,)\n\t\t ), \"Argument 'waitForDebuggerOnStart' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t waitForDebuggerOnStart)\n\t\tsubdom_funcs = self.synchronous_command('Target.setAutoAttach',\n\t\t autoAttach=autoAttach, waitForDebuggerOnStart=waitForDebuggerOnStart)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: Target.setAttachToFrames Domain: Target Method name: setAttachToFrames Parameters: Required arguments: 'value' (type: boolean) -> Whether to attach to frames. No return value.", "response": "def Target_setAttachToFrames(self, value):\n\t\t\"\"\"\n\t\tFunction path: Target.setAttachToFrames\n\t\t\tDomain: Target\n\t\t\tMethod name: setAttachToFrames\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'value' (type: boolean) -> Whether to attach to frames.\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(value, (bool,)\n\t\t ), \"Argument 'value' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t value)\n\t\tsubdom_funcs = self.synchronous_command('Target.setAttachToFrames', value\n\t\t =value)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Target_setRemoteLocations(self, locations):\n\t\tassert isinstance(locations, (list, tuple)\n\t\t ), \"Argument 'locations' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t locations)\n\t\tsubdom_funcs = self.synchronous_command('Target.setRemoteLocations',\n\t\t locations=locations)\n\t\treturn subdom_funcs", "response": "This method is used to set the remote locations of the target."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfunctions path: Target.sendMessageToTarget Domain: Target Method name: sendMessageToTarget Parameters: Required arguments: 'message' (type: string) -> No description Optional arguments: 'sessionId' (type: SessionID) -> Identifier of the session. 'targetId' (type: TargetID) -> Deprecated. No return value. Description: Sends protocol message over session with given id.", "response": "def Target_sendMessageToTarget(self, message, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Target.sendMessageToTarget\n\t\t\tDomain: Target\n\t\t\tMethod name: sendMessageToTarget\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'message' (type: string) -> No description\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'sessionId' (type: SessionID) -> Identifier of the session.\n\t\t\t\t\t'targetId' (type: TargetID) -> Deprecated.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sends protocol message over session with given id.\n\t\t\"\"\"\n\t\tassert isinstance(message, (str,)\n\t\t ), \"Argument 'message' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t message)\n\t\texpected = ['sessionId', 'targetId']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['sessionId', 'targetId']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Target.sendMessageToTarget',\n\t\t message=message, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path : ServiceWorker. unregister Domain : ServiceWorker Method name : unregister Parameters arguments : ServiceWorker Parameter type : str", "response": "def ServiceWorker_unregister(self, scopeURL):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.unregister\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: unregister\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'scopeURL' (type: string) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(scopeURL, (str,)\n\t\t ), \"Argument 'scopeURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scopeURL)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.unregister',\n\t\t scopeURL=scopeURL)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ServiceWorker_updateRegistration(self, scopeURL):\n\t\tassert isinstance(scopeURL, (str,)\n\t\t ), \"Argument 'scopeURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scopeURL)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.updateRegistration',\n\t\t scopeURL=scopeURL)\n\t\treturn subdom_funcs", "response": "Function path : ServiceWorker. updateRegistration"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: ServiceWorker.startWorker Domain: ServiceWorker Method name: startWorker Parameters: Required arguments: 'scopeURL' (type: string) -> No description No return value.", "response": "def ServiceWorker_startWorker(self, scopeURL):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.startWorker\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: startWorker\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'scopeURL' (type: string) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(scopeURL, (str,)\n\t\t ), \"Argument 'scopeURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scopeURL)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.startWorker',\n\t\t scopeURL=scopeURL)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: ServiceWorker.skipWaiting Domain: ServiceWorker Method name: skipWaiting Parameters: Required arguments: 'scopeURL' (type: string) -> No description No return value.", "response": "def ServiceWorker_skipWaiting(self, scopeURL):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.skipWaiting\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: skipWaiting\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'scopeURL' (type: string) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(scopeURL, (str,)\n\t\t ), \"Argument 'scopeURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scopeURL)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.skipWaiting',\n\t\t scopeURL=scopeURL)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ServiceWorker_stopWorker(self, versionId):\n\t\tassert isinstance(versionId, (str,)\n\t\t ), \"Argument 'versionId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t versionId)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.stopWorker',\n\t\t versionId=versionId)\n\t\treturn subdom_funcs", "response": "This method is used to stop the Worker with the given versionId."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: ServiceWorker.inspectWorker Domain: ServiceWorker Method name: inspectWorker Parameters: Required arguments: 'versionId' (type: string) -> No description No return value.", "response": "def ServiceWorker_inspectWorker(self, versionId):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.inspectWorker\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: inspectWorker\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'versionId' (type: string) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(versionId, (str,)\n\t\t ), \"Argument 'versionId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t versionId)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.inspectWorker',\n\t\t versionId=versionId)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: ServiceWorker.setForceUpdateOnPageLoad Domain: ServiceWorker Method name: setForceUpdateOnPageLoad Parameters: Required arguments: 'forceUpdateOnPageLoad' (type: boolean) -> No description No return value.", "response": "def ServiceWorker_setForceUpdateOnPageLoad(self, forceUpdateOnPageLoad):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.setForceUpdateOnPageLoad\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: setForceUpdateOnPageLoad\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'forceUpdateOnPageLoad' (type: boolean) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(forceUpdateOnPageLoad, (bool,)\n\t\t ), \"Argument 'forceUpdateOnPageLoad' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t forceUpdateOnPageLoad)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'ServiceWorker.setForceUpdateOnPageLoad', forceUpdateOnPageLoad=\n\t\t forceUpdateOnPageLoad)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ServiceWorker_deliverPushMessage(self, origin, registrationId, data):\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tassert isinstance(registrationId, (str,)\n\t\t ), \"Argument 'registrationId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t registrationId)\n\t\tassert isinstance(data, (str,)\n\t\t ), \"Argument 'data' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t data)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.deliverPushMessage',\n\t\t origin=origin, registrationId=registrationId, data=data)\n\t\treturn subdom_funcs", "response": "This method is used to deliver a push message to the specified registrationId."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: ServiceWorker.dispatchSyncEvent Domain: ServiceWorker Method name: dispatchSyncEvent Parameters: Required arguments: 'origin' (type: string) -> No description 'registrationId' (type: string) -> No description 'tag' (type: string) -> No description 'lastChance' (type: boolean) -> No description No return value.", "response": "def ServiceWorker_dispatchSyncEvent(self, origin, registrationId, tag,\n\t lastChance):\n\t\t\"\"\"\n\t\tFunction path: ServiceWorker.dispatchSyncEvent\n\t\t\tDomain: ServiceWorker\n\t\t\tMethod name: dispatchSyncEvent\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'origin' (type: string) -> No description\n\t\t\t\t\t'registrationId' (type: string) -> No description\n\t\t\t\t\t'tag' (type: string) -> No description\n\t\t\t\t\t'lastChance' (type: boolean) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tassert isinstance(registrationId, (str,)\n\t\t ), \"Argument 'registrationId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t registrationId)\n\t\tassert isinstance(tag, (str,)\n\t\t ), \"Argument 'tag' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t tag)\n\t\tassert isinstance(lastChance, (bool,)\n\t\t ), \"Argument 'lastChance' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t lastChance)\n\t\tsubdom_funcs = self.synchronous_command('ServiceWorker.dispatchSyncEvent',\n\t\t origin=origin, registrationId=registrationId, tag=tag, lastChance=\n\t\t lastChance)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Input_setIgnoreInputEvents(self, ignore):\n\t\tassert isinstance(ignore, (bool,)\n\t\t ), \"Argument 'ignore' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t ignore)\n\t\tsubdom_funcs = self.synchronous_command('Input.setIgnoreInputEvents',\n\t\t ignore=ignore)\n\t\treturn subdom_funcs", "response": "This command is used to set ignore input events."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Input_dispatchMouseEvent(self, type, x, y, **kwargs):\n\t\tassert isinstance(type, (str,)\n\t\t ), \"Argument 'type' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t type)\n\t\tassert isinstance(x, (float, int)\n\t\t ), \"Argument 'x' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t x)\n\t\tassert isinstance(y, (float, int)\n\t\t ), \"Argument 'y' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t y)\n\t\tif 'modifiers' in kwargs:\n\t\t\tassert isinstance(kwargs['modifiers'], (int,)\n\t\t\t ), \"Optional argument 'modifiers' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['modifiers'])\n\t\tif 'button' in kwargs:\n\t\t\tassert isinstance(kwargs['button'], (str,)\n\t\t\t ), \"Optional argument 'button' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['button'])\n\t\tif 'clickCount' in kwargs:\n\t\t\tassert isinstance(kwargs['clickCount'], (int,)\n\t\t\t ), \"Optional argument 'clickCount' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['clickCount'])\n\t\tif 'deltaX' in kwargs:\n\t\t\tassert isinstance(kwargs['deltaX'], (float, int)\n\t\t\t ), \"Optional argument 'deltaX' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['deltaX'])\n\t\tif 'deltaY' in kwargs:\n\t\t\tassert isinstance(kwargs['deltaY'], (float, int)\n\t\t\t ), \"Optional argument 'deltaY' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['deltaY'])\n\t\texpected = ['modifiers', 'timestamp', 'button', 'clickCount', 'deltaX',\n\t\t 'deltaY']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['modifiers', 'timestamp', 'button', 'clickCount', 'deltaX', 'deltaY']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Input.dispatchMouseEvent', type=\n\t\t type, x=x, y=y, **kwargs)\n\t\treturn subdom_funcs", "response": "Input. dispatchMouseEvent - Dispatches a mouse event to the page."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: Input.dispatchTouchEvent Domain: Input Method name: dispatchTouchEvent WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'type' (type: string) -> Type of the touch event. TouchEnd and TouchCancel must not contain any touch points, while TouchStart and TouchMove must contains at least one. 'touchPoints' (type: array) -> Active touch points on the touch device. One event per any changed point (compared to previous touch event in a sequence) is generated, emulating pressing/moving/releasing points one by one. Optional arguments: 'modifiers' (type: integer) -> Bit field representing pressed modifier keys. Alt=1, Ctrl=2, Meta/Command=4, Shift=8 (default: 0). 'timestamp' (type: TimeSinceEpoch) -> Time at which the event occurred. No return value. Description: Dispatches a touch event to the page.", "response": "def Input_dispatchTouchEvent(self, type, touchPoints, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Input.dispatchTouchEvent\n\t\t\tDomain: Input\n\t\t\tMethod name: dispatchTouchEvent\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'type' (type: string) -> Type of the touch event. TouchEnd and TouchCancel must not contain any touch points, while TouchStart and TouchMove must contains at least one.\n\t\t\t\t\t'touchPoints' (type: array) -> Active touch points on the touch device. One event per any changed point (compared to previous touch event in a sequence) is generated, emulating pressing/moving/releasing points one by one.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'modifiers' (type: integer) -> Bit field representing pressed modifier keys. Alt=1, Ctrl=2, Meta/Command=4, Shift=8 (default: 0).\n\t\t\t\t\t'timestamp' (type: TimeSinceEpoch) -> Time at which the event occurred.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Dispatches a touch event to the page.\n\t\t\"\"\"\n\t\tassert isinstance(type, (str,)\n\t\t ), \"Argument 'type' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t type)\n\t\tassert isinstance(touchPoints, (list, tuple)\n\t\t ), \"Argument 'touchPoints' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t touchPoints)\n\t\tif 'modifiers' in kwargs:\n\t\t\tassert isinstance(kwargs['modifiers'], (int,)\n\t\t\t ), \"Optional argument 'modifiers' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['modifiers'])\n\t\texpected = ['modifiers', 'timestamp']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['modifiers', 'timestamp']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Input.dispatchTouchEvent', type=\n\t\t type, touchPoints=touchPoints, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef Input_synthesizePinchGesture(self, x, y, scaleFactor, **kwargs):\n\t\tassert isinstance(x, (float, int)\n\t\t ), \"Argument 'x' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t x)\n\t\tassert isinstance(y, (float, int)\n\t\t ), \"Argument 'y' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t y)\n\t\tassert isinstance(scaleFactor, (float, int)\n\t\t ), \"Argument 'scaleFactor' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t scaleFactor)\n\t\tif 'relativeSpeed' in kwargs:\n\t\t\tassert isinstance(kwargs['relativeSpeed'], (int,)\n\t\t\t ), \"Optional argument 'relativeSpeed' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t\t kwargs['relativeSpeed'])\n\t\texpected = ['relativeSpeed', 'gestureSourceType']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['relativeSpeed', 'gestureSourceType']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Input.synthesizePinchGesture', x\n\t\t =x, y=y, scaleFactor=scaleFactor, **kwargs)\n\t\treturn subdom_funcs", "response": "Input. synthesizePinchGesture - Synthesizes a pinch gesture over a time period."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nlayers Tree load snapshot operation.", "response": "def LayerTree_loadSnapshot(self, tiles):\n\t\t\"\"\"\n\t\tFunction path: LayerTree.loadSnapshot\n\t\t\tDomain: LayerTree\n\t\t\tMethod name: loadSnapshot\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'tiles' (type: array) -> An array of tiles composing the snapshot.\n\t\t\tReturns:\n\t\t\t\t'snapshotId' (type: SnapshotId) -> The id of the snapshot.\n\t\t\n\t\t\tDescription: Returns the snapshot identifier.\n\t\t\"\"\"\n\t\tassert isinstance(tiles, (list, tuple)\n\t\t ), \"Argument 'tiles' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t tiles)\n\t\tsubdom_funcs = self.synchronous_command('LayerTree.loadSnapshot', tiles=tiles\n\t\t )\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef Tracing_recordClockSyncMarker(self, syncId):\n\t\tassert isinstance(syncId, (str,)\n\t\t ), \"Argument 'syncId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t syncId)\n\t\tsubdom_funcs = self.synchronous_command('Tracing.recordClockSyncMarker',\n\t\t syncId=syncId)\n\t\treturn subdom_funcs", "response": "Trace. recordClockSyncMarker - Record a clock sync marker in the trace."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction path: Animation.setPlaybackRate Domain: Animation Method name: setPlaybackRate Parameters: Required arguments: 'playbackRate' (type: number) -> Playback rate for animations on page No return value. Description: Sets the playback rate of the document timeline.", "response": "def Animation_setPlaybackRate(self, playbackRate):\n\t\t\"\"\"\n\t\tFunction path: Animation.setPlaybackRate\n\t\t\tDomain: Animation\n\t\t\tMethod name: setPlaybackRate\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'playbackRate' (type: number) -> Playback rate for animations on page\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Sets the playback rate of the document timeline.\n\t\t\"\"\"\n\t\tassert isinstance(playbackRate, (float, int)\n\t\t ), \"Argument 'playbackRate' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t playbackRate)\n\t\tsubdom_funcs = self.synchronous_command('Animation.setPlaybackRate',\n\t\t playbackRate=playbackRate)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Animation_getCurrentTime(self, id):\n\t\tassert isinstance(id, (str,)\n\t\t ), \"Argument 'id' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t id)\n\t\tsubdom_funcs = self.synchronous_command('Animation.getCurrentTime', id=id)\n\t\treturn subdom_funcs", "response": "Animation. getCurrentTime - Returns the current time of the animation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Animation_setPaused(self, animations, paused):\n\t\tassert isinstance(animations, (list, tuple)\n\t\t ), \"Argument 'animations' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t animations)\n\t\tassert isinstance(paused, (bool,)\n\t\t ), \"Argument 'paused' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t paused)\n\t\tsubdom_funcs = self.synchronous_command('Animation.setPaused', animations\n\t\t =animations, paused=paused)\n\t\treturn subdom_funcs", "response": "This method is used to set the paused state of a set of animations."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Animation_setTiming(self, animationId, duration, delay):\n\t\tassert isinstance(animationId, (str,)\n\t\t ), \"Argument 'animationId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t animationId)\n\t\tassert isinstance(duration, (float, int)\n\t\t ), \"Argument 'duration' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t duration)\n\t\tassert isinstance(delay, (float, int)\n\t\t ), \"Argument 'delay' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t delay)\n\t\tsubdom_funcs = self.synchronous_command('Animation.setTiming',\n\t\t animationId=animationId, duration=duration, delay=delay)\n\t\treturn subdom_funcs", "response": "This method is used to set the timing of an animation node."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Animation_seekAnimations(self, animations, currentTime):\n\t\tassert isinstance(animations, (list, tuple)\n\t\t ), \"Argument 'animations' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t animations)\n\t\tassert isinstance(currentTime, (float, int)\n\t\t ), \"Argument 'currentTime' must be of type '['float', 'int']'. Received type: '%s'\" % type(\n\t\t currentTime)\n\t\tsubdom_funcs = self.synchronous_command('Animation.seekAnimations',\n\t\t animations=animations, currentTime=currentTime)\n\t\treturn subdom_funcs", "response": "This method is used to seek an animation to a particular time."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: Animation.releaseAnimations Domain: Animation Method name: releaseAnimations Parameters: Required arguments: 'animations' (type: array) -> List of animation ids to seek. No return value. Description: Releases a set of animations to no longer be manipulated.", "response": "def Animation_releaseAnimations(self, animations):\n\t\t\"\"\"\n\t\tFunction path: Animation.releaseAnimations\n\t\t\tDomain: Animation\n\t\t\tMethod name: releaseAnimations\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'animations' (type: array) -> List of animation ids to seek.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Releases a set of animations to no longer be manipulated.\n\t\t\"\"\"\n\t\tassert isinstance(animations, (list, tuple)\n\t\t ), \"Argument 'animations' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t animations)\n\t\tsubdom_funcs = self.synchronous_command('Animation.releaseAnimations',\n\t\t animations=animations)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Animation_resolveAnimation(self, animationId):\n\t\tassert isinstance(animationId, (str,)\n\t\t ), \"Argument 'animationId' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t animationId)\n\t\tsubdom_funcs = self.synchronous_command('Animation.resolveAnimation',\n\t\t animationId=animationId)\n\t\treturn subdom_funcs", "response": "This method returns the remote object of the Animation with the given id."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Storage_clearDataForOrigin(self, origin, storageTypes):\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tassert isinstance(storageTypes, (str,)\n\t\t ), \"Argument 'storageTypes' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t storageTypes)\n\t\tsubdom_funcs = self.synchronous_command('Storage.clearDataForOrigin',\n\t\t origin=origin, storageTypes=storageTypes)\n\t\treturn subdom_funcs", "response": "This method clears storage for the specified origin."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef Storage_getUsageAndQuota(self, origin):\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tsubdom_funcs = self.synchronous_command('Storage.getUsageAndQuota',\n\t\t origin=origin)\n\t\treturn subdom_funcs", "response": "This method returns the usage and quota in bytes."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Storage_trackCacheStorageForOrigin(self, origin):\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tsubdom_funcs = self.synchronous_command('Storage.trackCacheStorageForOrigin',\n\t\t origin=origin)\n\t\treturn subdom_funcs", "response": "This method is used to track the state of a cache storage list for a given origin."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Storage_untrackCacheStorageForOrigin(self, origin):\n\t\tassert isinstance(origin, (str,)\n\t\t ), \"Argument 'origin' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t origin)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Storage.untrackCacheStorageForOrigin', origin=origin)\n\t\treturn subdom_funcs", "response": "This method unregisters a cache storage origin from receiving notifications for cache storage."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfunctioning path: Log.startViolationsReport Domain: Log Method name: startViolationsReport Parameters: Required arguments: 'config' (type: array) -> Configuration for violations. No return value. Description: start violation reporting.", "response": "def Log_startViolationsReport(self, config):\n\t\t\"\"\"\n\t\tFunction path: Log.startViolationsReport\n\t\t\tDomain: Log\n\t\t\tMethod name: startViolationsReport\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'config' (type: array) -> Configuration for violations.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: start violation reporting.\n\t\t\"\"\"\n\t\tassert isinstance(config, (list, tuple)\n\t\t ), \"Argument 'config' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t config)\n\t\tsubdom_funcs = self.synchronous_command('Log.startViolationsReport',\n\t\t config=config)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning path: Tethering.bind Domain: Tethering Method name: bind Parameters: Required arguments: 'port' (type: integer) -> Port number to bind. No return value. Description: Request browser port binding.", "response": "def Tethering_bind(self, port):\n\t\t\"\"\"\n\t\tFunction path: Tethering.bind\n\t\t\tDomain: Tethering\n\t\t\tMethod name: bind\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'port' (type: integer) -> Port number to bind.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Request browser port binding.\n\t\t\"\"\"\n\t\tassert isinstance(port, (int,)\n\t\t ), \"Argument 'port' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t port)\n\t\tsubdom_funcs = self.synchronous_command('Tethering.bind', port=port)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning path: Tethering.unbind Domain: Tethering Method name: unbind Parameters: Required arguments: 'port' (type: integer) -> Port number to unbind. No return value. Description: Request browser port unbinding.", "response": "def Tethering_unbind(self, port):\n\t\t\"\"\"\n\t\tFunction path: Tethering.unbind\n\t\t\tDomain: Tethering\n\t\t\tMethod name: unbind\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'port' (type: integer) -> Port number to unbind.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Request browser port unbinding.\n\t\t\"\"\"\n\t\tassert isinstance(port, (int,)\n\t\t ), \"Argument 'port' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t port)\n\t\tsubdom_funcs = self.synchronous_command('Tethering.unbind', port=port)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions path: Browser.setWindowBounds Domain: Browser Method name: setWindowBounds Parameters: Required arguments: 'windowId' (type: WindowID) -> Browser window id. 'bounds' (type: Bounds) -> New window bounds. The 'minimized', 'maximized' and 'fullscreen' states cannot be combined with 'left', 'top', 'width' or 'height'. Leaves unspecified fields unchanged. No return value. Description: Set position and/or size of the browser window.", "response": "def Browser_setWindowBounds(self, windowId, bounds):\n\t\t\"\"\"\n\t\tFunction path: Browser.setWindowBounds\n\t\t\tDomain: Browser\n\t\t\tMethod name: setWindowBounds\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'windowId' (type: WindowID) -> Browser window id.\n\t\t\t\t\t'bounds' (type: Bounds) -> New window bounds. The 'minimized', 'maximized' and 'fullscreen' states cannot be combined with 'left', 'top', 'width' or 'height'. Leaves unspecified fields unchanged.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Set position and/or size of the browser window.\n\t\t\"\"\"\n\t\tsubdom_funcs = self.synchronous_command('Browser.setWindowBounds',\n\t\t windowId=windowId, bounds=bounds)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfunction path: Runtime.evaluate Domain: Runtime Method name: evaluate Parameters: Required arguments: 'expression' (type: string) -> Expression to evaluate. Optional arguments: 'objectGroup' (type: string) -> Symbolic group name that can be used to release multiple objects. 'includeCommandLineAPI' (type: boolean) -> Determines whether Command Line API should be available during the evaluation. 'silent' (type: boolean) -> In silent mode exceptions thrown during evaluation are not reported and do not pause execution. Overrides setPauseOnException state. 'contextId' (type: ExecutionContextId) -> Specifies in which execution context to perform evaluation. If the parameter is omitted the evaluation will be performed in the context of the inspected page. 'returnByValue' (type: boolean) -> Whether the result is expected to be a JSON object that should be sent by value. 'generatePreview' (type: boolean) -> Whether preview should be generated for the result. 'userGesture' (type: boolean) -> Whether execution should be treated as initiated by user in the UI. 'awaitPromise' (type: boolean) -> Whether execution should await for resulting value and return once awaited promise is resolved. Returns: 'result' (type: RemoteObject) -> Evaluation result. 'exceptionDetails' (type: ExceptionDetails) -> Exception details. Description: Evaluates expression on global object.", "response": "def Runtime_evaluate(self, expression, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Runtime.evaluate\n\t\t\tDomain: Runtime\n\t\t\tMethod name: evaluate\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'expression' (type: string) -> Expression to evaluate.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'objectGroup' (type: string) -> Symbolic group name that can be used to release multiple objects.\n\t\t\t\t\t'includeCommandLineAPI' (type: boolean) -> Determines whether Command Line API should be available during the evaluation.\n\t\t\t\t\t'silent' (type: boolean) -> In silent mode exceptions thrown during evaluation are not reported and do not pause execution. Overrides setPauseOnException state.\n\t\t\t\t\t'contextId' (type: ExecutionContextId) -> Specifies in which execution context to perform evaluation. If the parameter is omitted the evaluation will be performed in the context of the inspected page.\n\t\t\t\t\t'returnByValue' (type: boolean) -> Whether the result is expected to be a JSON object that should be sent by value.\n\t\t\t\t\t'generatePreview' (type: boolean) -> Whether preview should be generated for the result.\n\t\t\t\t\t'userGesture' (type: boolean) -> Whether execution should be treated as initiated by user in the UI.\n\t\t\t\t\t'awaitPromise' (type: boolean) -> Whether execution should await for resulting value and return once awaited promise is resolved.\n\t\t\tReturns:\n\t\t\t\t'result' (type: RemoteObject) -> Evaluation result.\n\t\t\t\t'exceptionDetails' (type: ExceptionDetails) -> Exception details.\n\t\t\n\t\t\tDescription: Evaluates expression on global object.\n\t\t\"\"\"\n\t\tassert isinstance(expression, (str,)\n\t\t ), \"Argument 'expression' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t expression)\n\t\tif 'objectGroup' in kwargs:\n\t\t\tassert isinstance(kwargs['objectGroup'], (str,)\n\t\t\t ), \"Optional argument 'objectGroup' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['objectGroup'])\n\t\tif 'includeCommandLineAPI' in kwargs:\n\t\t\tassert isinstance(kwargs['includeCommandLineAPI'], (bool,)\n\t\t\t ), \"Optional argument 'includeCommandLineAPI' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['includeCommandLineAPI'])\n\t\tif 'silent' in kwargs:\n\t\t\tassert isinstance(kwargs['silent'], (bool,)\n\t\t\t ), \"Optional argument 'silent' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['silent'])\n\t\tif 'returnByValue' in kwargs:\n\t\t\tassert isinstance(kwargs['returnByValue'], (bool,)\n\t\t\t ), \"Optional argument 'returnByValue' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['returnByValue'])\n\t\tif 'generatePreview' in kwargs:\n\t\t\tassert isinstance(kwargs['generatePreview'], (bool,)\n\t\t\t ), \"Optional argument 'generatePreview' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['generatePreview'])\n\t\tif 'userGesture' in kwargs:\n\t\t\tassert isinstance(kwargs['userGesture'], (bool,)\n\t\t\t ), \"Optional argument 'userGesture' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['userGesture'])\n\t\tif 'awaitPromise' in kwargs:\n\t\t\tassert isinstance(kwargs['awaitPromise'], (bool,)\n\t\t\t ), \"Optional argument 'awaitPromise' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['awaitPromise'])\n\t\texpected = ['objectGroup', 'includeCommandLineAPI', 'silent', 'contextId',\n\t\t 'returnByValue', 'generatePreview', 'userGesture', 'awaitPromise']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['objectGroup', 'includeCommandLineAPI', 'silent', 'contextId', 'returnByValue', 'generatePreview', 'userGesture', 'awaitPromise']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Runtime.evaluate', expression=\n\t\t expression, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction path: Runtime.callFunctionOn Domain: Runtime Method name: callFunctionOn Parameters: Required arguments: 'functionDeclaration' (type: string) -> Declaration of the function to call. Optional arguments: 'objectId' (type: RemoteObjectId) -> Identifier of the object to call function on. Either objectId or executionContextId should be specified. 'arguments' (type: array) -> Call arguments. All call arguments must belong to the same JavaScript world as the target object. 'silent' (type: boolean) -> In silent mode exceptions thrown during evaluation are not reported and do not pause execution. Overrides setPauseOnException state. 'returnByValue' (type: boolean) -> Whether the result is expected to be a JSON object which should be sent by value. 'generatePreview' (type: boolean) -> Whether preview should be generated for the result. 'userGesture' (type: boolean) -> Whether execution should be treated as initiated by user in the UI. 'awaitPromise' (type: boolean) -> Whether execution should await for resulting value and return once awaited promise is resolved. 'executionContextId' (type: ExecutionContextId) -> Specifies execution context which global object will be used to call function on. Either executionContextId or objectId should be specified. 'objectGroup' (type: string) -> Symbolic group name that can be used to release multiple objects. If objectGroup is not specified and objectId is, objectGroup will be inherited from object. Returns: 'result' (type: RemoteObject) -> Call result. 'exceptionDetails' (type: ExceptionDetails) -> Exception details. Description: Calls function with given declaration on the given object. Object group of the result is inherited from the target object.", "response": "def Runtime_callFunctionOn(self, functionDeclaration, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Runtime.callFunctionOn\n\t\t\tDomain: Runtime\n\t\t\tMethod name: callFunctionOn\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'functionDeclaration' (type: string) -> Declaration of the function to call.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'objectId' (type: RemoteObjectId) -> Identifier of the object to call function on. Either objectId or executionContextId should be specified.\n\t\t\t\t\t'arguments' (type: array) -> Call arguments. All call arguments must belong to the same JavaScript world as the target object.\n\t\t\t\t\t'silent' (type: boolean) -> In silent mode exceptions thrown during evaluation are not reported and do not pause execution. Overrides setPauseOnException state.\n\t\t\t\t\t'returnByValue' (type: boolean) -> Whether the result is expected to be a JSON object which should be sent by value.\n\t\t\t\t\t'generatePreview' (type: boolean) -> Whether preview should be generated for the result.\n\t\t\t\t\t'userGesture' (type: boolean) -> Whether execution should be treated as initiated by user in the UI.\n\t\t\t\t\t'awaitPromise' (type: boolean) -> Whether execution should await for resulting value and return once awaited promise is resolved.\n\t\t\t\t\t'executionContextId' (type: ExecutionContextId) -> Specifies execution context which global object will be used to call function on. Either executionContextId or objectId should be specified.\n\t\t\t\t\t'objectGroup' (type: string) -> Symbolic group name that can be used to release multiple objects. If objectGroup is not specified and objectId is, objectGroup will be inherited from object.\n\t\t\tReturns:\n\t\t\t\t'result' (type: RemoteObject) -> Call result.\n\t\t\t\t'exceptionDetails' (type: ExceptionDetails) -> Exception details.\n\t\t\n\t\t\tDescription: Calls function with given declaration on the given object. Object group of the result is inherited from the target object.\n\t\t\"\"\"\n\t\tassert isinstance(functionDeclaration, (str,)\n\t\t ), \"Argument 'functionDeclaration' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t functionDeclaration)\n\t\tif 'arguments' in kwargs:\n\t\t\tassert isinstance(kwargs['arguments'], (list, tuple)\n\t\t\t ), \"Optional argument 'arguments' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t\t kwargs['arguments'])\n\t\tif 'silent' in kwargs:\n\t\t\tassert isinstance(kwargs['silent'], (bool,)\n\t\t\t ), \"Optional argument 'silent' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['silent'])\n\t\tif 'returnByValue' in kwargs:\n\t\t\tassert isinstance(kwargs['returnByValue'], (bool,)\n\t\t\t ), \"Optional argument 'returnByValue' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['returnByValue'])\n\t\tif 'generatePreview' in kwargs:\n\t\t\tassert isinstance(kwargs['generatePreview'], (bool,)\n\t\t\t ), \"Optional argument 'generatePreview' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['generatePreview'])\n\t\tif 'userGesture' in kwargs:\n\t\t\tassert isinstance(kwargs['userGesture'], (bool,)\n\t\t\t ), \"Optional argument 'userGesture' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['userGesture'])\n\t\tif 'awaitPromise' in kwargs:\n\t\t\tassert isinstance(kwargs['awaitPromise'], (bool,)\n\t\t\t ), \"Optional argument 'awaitPromise' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['awaitPromise'])\n\t\tif 'objectGroup' in kwargs:\n\t\t\tassert isinstance(kwargs['objectGroup'], (str,)\n\t\t\t ), \"Optional argument 'objectGroup' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['objectGroup'])\n\t\texpected = ['objectId', 'arguments', 'silent', 'returnByValue',\n\t\t 'generatePreview', 'userGesture', 'awaitPromise',\n\t\t 'executionContextId', 'objectGroup']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['objectId', 'arguments', 'silent', 'returnByValue', 'generatePreview', 'userGesture', 'awaitPromise', 'executionContextId', 'objectGroup']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Runtime.callFunctionOn',\n\t\t functionDeclaration=functionDeclaration, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfunctions path: Runtime.releaseObjectGroup Domain: Runtime Method name: releaseObjectGroup Parameters: Required arguments: 'objectGroup' (type: string) -> Symbolic object group name. No return value. Description: Releases all remote objects that belong to a given group.", "response": "def Runtime_releaseObjectGroup(self, objectGroup):\n\t\t\"\"\"\n\t\tFunction path: Runtime.releaseObjectGroup\n\t\t\tDomain: Runtime\n\t\t\tMethod name: releaseObjectGroup\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'objectGroup' (type: string) -> Symbolic object group name.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Releases all remote objects that belong to a given group.\n\t\t\"\"\"\n\t\tassert isinstance(objectGroup, (str,)\n\t\t ), \"Argument 'objectGroup' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t objectGroup)\n\t\tsubdom_funcs = self.synchronous_command('Runtime.releaseObjectGroup',\n\t\t objectGroup=objectGroup)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: Runtime.setCustomObjectFormatterEnabled Domain: Runtime Method name: setCustomObjectFormatterEnabled WARNING: This function is marked 'Experimental'! Parameters: Required arguments: 'enabled' (type: boolean) -> No description No return value.", "response": "def Runtime_setCustomObjectFormatterEnabled(self, enabled):\n\t\t\"\"\"\n\t\tFunction path: Runtime.setCustomObjectFormatterEnabled\n\t\t\tDomain: Runtime\n\t\t\tMethod name: setCustomObjectFormatterEnabled\n\t\t\n\t\t\tWARNING: This function is marked 'Experimental'!\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'enabled' (type: boolean) -> No description\n\t\t\tNo return value.\n\t\t\n\t\t\"\"\"\n\t\tassert isinstance(enabled, (bool,)\n\t\t ), \"Argument 'enabled' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t enabled)\n\t\tsubdom_funcs = self.synchronous_command(\n\t\t 'Runtime.setCustomObjectFormatterEnabled', enabled=enabled)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Runtime_compileScript(self, expression, sourceURL, persistScript, **kwargs\n\t ):\n\t\t\"\"\"\n\t\tFunction path: Runtime.compileScript\n\t\t\tDomain: Runtime\n\t\t\tMethod name: compileScript\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'expression' (type: string) -> Expression to compile.\n\t\t\t\t\t'sourceURL' (type: string) -> Source url to be set for the script.\n\t\t\t\t\t'persistScript' (type: boolean) -> Specifies whether the compiled script should be persisted.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'executionContextId' (type: ExecutionContextId) -> Specifies in which execution context to perform script run. If the parameter is omitted the evaluation will be performed in the context of the inspected page.\n\t\t\tReturns:\n\t\t\t\t'scriptId' (type: ScriptId) -> Id of the script.\n\t\t\t\t'exceptionDetails' (type: ExceptionDetails) -> Exception details.\n\t\t\n\t\t\tDescription: Compiles expression.\n\t\t\"\"\"\n\t\tassert isinstance(expression, (str,)\n\t\t ), \"Argument 'expression' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t expression)\n\t\tassert isinstance(sourceURL, (str,)\n\t\t ), \"Argument 'sourceURL' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t sourceURL)\n\t\tassert isinstance(persistScript, (bool,)\n\t\t ), \"Argument 'persistScript' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t persistScript)\n\t\texpected = ['executionContextId']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['executionContextId']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Runtime.compileScript',\n\t\t expression=expression, sourceURL=sourceURL, persistScript=\n\t\t persistScript, **kwargs)\n\t\treturn subdom_funcs", "response": "Runtime. compileScript - Executes a script."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Runtime_runScript(self, scriptId, **kwargs):\n\t\tif 'objectGroup' in kwargs:\n\t\t\tassert isinstance(kwargs['objectGroup'], (str,)\n\t\t\t ), \"Optional argument 'objectGroup' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t\t kwargs['objectGroup'])\n\t\tif 'silent' in kwargs:\n\t\t\tassert isinstance(kwargs['silent'], (bool,)\n\t\t\t ), \"Optional argument 'silent' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['silent'])\n\t\tif 'includeCommandLineAPI' in kwargs:\n\t\t\tassert isinstance(kwargs['includeCommandLineAPI'], (bool,)\n\t\t\t ), \"Optional argument 'includeCommandLineAPI' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['includeCommandLineAPI'])\n\t\tif 'returnByValue' in kwargs:\n\t\t\tassert isinstance(kwargs['returnByValue'], (bool,)\n\t\t\t ), \"Optional argument 'returnByValue' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['returnByValue'])\n\t\tif 'generatePreview' in kwargs:\n\t\t\tassert isinstance(kwargs['generatePreview'], (bool,)\n\t\t\t ), \"Optional argument 'generatePreview' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['generatePreview'])\n\t\tif 'awaitPromise' in kwargs:\n\t\t\tassert isinstance(kwargs['awaitPromise'], (bool,)\n\t\t\t ), \"Optional argument 'awaitPromise' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['awaitPromise'])\n\t\texpected = ['executionContextId', 'objectGroup', 'silent',\n\t\t 'includeCommandLineAPI', 'returnByValue', 'generatePreview',\n\t\t 'awaitPromise']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['executionContextId', 'objectGroup', 'silent', 'includeCommandLineAPI', 'returnByValue', 'generatePreview', 'awaitPromise']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Runtime.runScript', scriptId=\n\t\t scriptId, **kwargs)\n\t\treturn subdom_funcs", "response": "Runtime. runScript - Runs a script in a given context."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Debugger_setBreakpointsActive(self, active):\n\t\tassert isinstance(active, (bool,)\n\t\t ), \"Argument 'active' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t active)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setBreakpointsActive',\n\t\t active=active)\n\t\treturn subdom_funcs", "response": "This method is used to set the active state of all breakpoints on the page."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Debugger_setSkipAllPauses(self, skip):\n\t\tassert isinstance(skip, (bool,)\n\t\t ), \"Argument 'skip' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t skip)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setSkipAllPauses', skip\n\t\t =skip)\n\t\treturn subdom_funcs", "response": "This method is used to set the skip pauses of all pages."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: Debugger.setScriptSource Domain: Debugger Method name: setScriptSource Parameters: Required arguments: 'scriptId' (type: Runtime.ScriptId) -> Id of the script to edit. 'scriptSource' (type: string) -> New content of the script. Optional arguments: 'dryRun' (type: boolean) -> If true the change will not actually be applied. Dry run may be used to get result description without actually modifying the code. Returns: 'callFrames' (type: array) -> New stack trace in case editing has happened while VM was stopped. 'stackChanged' (type: boolean) -> Whether current call stack was modified after applying the changes. 'asyncStackTrace' (type: Runtime.StackTrace) -> Async stack trace, if any. 'exceptionDetails' (type: Runtime.ExceptionDetails) -> Exception details if any. Description: Edits JavaScript source live.", "response": "def Debugger_setScriptSource(self, scriptId, scriptSource, **kwargs):\n\t\t\"\"\"\n\t\tFunction path: Debugger.setScriptSource\n\t\t\tDomain: Debugger\n\t\t\tMethod name: setScriptSource\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'scriptId' (type: Runtime.ScriptId) -> Id of the script to edit.\n\t\t\t\t\t'scriptSource' (type: string) -> New content of the script.\n\t\t\t\tOptional arguments:\n\t\t\t\t\t'dryRun' (type: boolean) -> If true the change will not actually be applied. Dry run may be used to get result description without actually modifying the code.\n\t\t\tReturns:\n\t\t\t\t'callFrames' (type: array) -> New stack trace in case editing has happened while VM was stopped.\n\t\t\t\t'stackChanged' (type: boolean) -> Whether current call stack was modified after applying the changes.\n\t\t\t\t'asyncStackTrace' (type: Runtime.StackTrace) -> Async stack trace, if any.\n\t\t\t\t'exceptionDetails' (type: Runtime.ExceptionDetails) -> Exception details if any.\n\t\t\n\t\t\tDescription: Edits JavaScript source live.\n\t\t\"\"\"\n\t\tassert isinstance(scriptSource, (str,)\n\t\t ), \"Argument 'scriptSource' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t scriptSource)\n\t\tif 'dryRun' in kwargs:\n\t\t\tassert isinstance(kwargs['dryRun'], (bool,)\n\t\t\t ), \"Optional argument 'dryRun' must be of type '['bool']'. Received type: '%s'\" % type(\n\t\t\t kwargs['dryRun'])\n\t\texpected = ['dryRun']\n\t\tpassed_keys = list(kwargs.keys())\n\t\tassert all([(key in expected) for key in passed_keys]\n\t\t ), \"Allowed kwargs are ['dryRun']. Passed kwargs: %s\" % passed_keys\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setScriptSource',\n\t\t scriptId=scriptId, scriptSource=scriptSource, **kwargs)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions path: Debugger.setPauseOnExceptions Domain: Debugger Method name: setPauseOnExceptions Parameters: Required arguments: 'state' (type: string) -> Pause on exceptions mode. No return value. Description: Defines pause on exceptions state. Can be set to stop on all exceptions, uncaught exceptions or no exceptions. Initial pause on exceptions state is none.", "response": "def Debugger_setPauseOnExceptions(self, state):\n\t\t\"\"\"\n\t\tFunction path: Debugger.setPauseOnExceptions\n\t\t\tDomain: Debugger\n\t\t\tMethod name: setPauseOnExceptions\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'state' (type: string) -> Pause on exceptions mode.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Defines pause on exceptions state. Can be set to stop on all exceptions, uncaught exceptions or no exceptions. Initial pause on exceptions state is none.\n\t\t\"\"\"\n\t\tassert isinstance(state, (str,)\n\t\t ), \"Argument 'state' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t state)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setPauseOnExceptions',\n\t\t state=state)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfunction path: Debugger.setVariableValue Domain: Debugger Method name: setVariableValue Parameters: Required arguments: 'scopeNumber' (type: integer) -> 0-based number of scope as was listed in scope chain. Only 'local', 'closure' and 'catch' scope types are allowed. Other scopes could be manipulated manually. 'variableName' (type: string) -> Variable name. 'newValue' (type: Runtime.CallArgument) -> New variable value. 'callFrameId' (type: CallFrameId) -> Id of callframe that holds variable. No return value. Description: Changes value of variable in a callframe. Object-based scopes are not supported and must be mutated manually.", "response": "def Debugger_setVariableValue(self, scopeNumber, variableName, newValue,\n\t callFrameId):\n\t\t\"\"\"\n\t\tFunction path: Debugger.setVariableValue\n\t\t\tDomain: Debugger\n\t\t\tMethod name: setVariableValue\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'scopeNumber' (type: integer) -> 0-based number of scope as was listed in scope chain. Only 'local', 'closure' and 'catch' scope types are allowed. Other scopes could be manipulated manually.\n\t\t\t\t\t'variableName' (type: string) -> Variable name.\n\t\t\t\t\t'newValue' (type: Runtime.CallArgument) -> New variable value.\n\t\t\t\t\t'callFrameId' (type: CallFrameId) -> Id of callframe that holds variable.\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Changes value of variable in a callframe. Object-based scopes are not supported and must be mutated manually.\n\t\t\"\"\"\n\t\tassert isinstance(scopeNumber, (int,)\n\t\t ), \"Argument 'scopeNumber' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t scopeNumber)\n\t\tassert isinstance(variableName, (str,)\n\t\t ), \"Argument 'variableName' must be of type '['str']'. Received type: '%s'\" % type(\n\t\t variableName)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setVariableValue',\n\t\t scopeNumber=scopeNumber, variableName=variableName, newValue=newValue,\n\t\t callFrameId=callFrameId)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction path: Debugger.setAsyncCallStackDepth Domain: Debugger Method name: setAsyncCallStackDepth Parameters: Required arguments: 'maxDepth' (type: integer) -> Maximum depth of async call stacks. Setting to 0 will effectively disable collecting async call stacks (default). No return value. Description: Enables or disables async call stacks tracking.", "response": "def Debugger_setAsyncCallStackDepth(self, maxDepth):\n\t\t\"\"\"\n\t\tFunction path: Debugger.setAsyncCallStackDepth\n\t\t\tDomain: Debugger\n\t\t\tMethod name: setAsyncCallStackDepth\n\t\t\n\t\t\tParameters:\n\t\t\t\tRequired arguments:\n\t\t\t\t\t'maxDepth' (type: integer) -> Maximum depth of async call stacks. Setting to 0 will effectively disable collecting async call stacks (default).\n\t\t\tNo return value.\n\t\t\n\t\t\tDescription: Enables or disables async call stacks tracking.\n\t\t\"\"\"\n\t\tassert isinstance(maxDepth, (int,)\n\t\t ), \"Argument 'maxDepth' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t maxDepth)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setAsyncCallStackDepth',\n\t\t maxDepth=maxDepth)\n\t\treturn subdom_funcs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Debugger_setBlackboxPatterns(self, patterns):\n\t\tassert isinstance(patterns, (list, tuple)\n\t\t ), \"Argument 'patterns' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t patterns)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setBlackboxPatterns',\n\t\t patterns=patterns)\n\t\treturn subdom_funcs", "response": "This function is used to set patterns in the script url for blackboxing."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef Debugger_setBlackboxedRanges(self, scriptId, positions):\n\t\tassert isinstance(positions, (list, tuple)\n\t\t ), \"Argument 'positions' must be of type '['list', 'tuple']'. Received type: '%s'\" % type(\n\t\t positions)\n\t\tsubdom_funcs = self.synchronous_command('Debugger.setBlackboxedRanges',\n\t\t scriptId=scriptId, positions=positions)\n\t\treturn subdom_funcs", "response": "This method is used to set blackboxed ranges in a script."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef Profiler_setSamplingInterval(self, interval):\n\t\tassert isinstance(interval, (int,)\n\t\t ), \"Argument 'interval' must be of type '['int']'. Received type: '%s'\" % type(\n\t\t interval)\n\t\tsubdom_funcs = self.synchronous_command('Profiler.setSamplingInterval',\n\t\t interval=interval)\n\t\treturn subdom_funcs", "response": "This method is used to change CPU profiler sampling interval in microseconds."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngive a set of headers, update both the user-agent and additional headers for the remote browser. header_args must be a dict. Keys are the names of the corresponding HTTP header. return value is a 2-tuple of the results of the user-agent update, as well as the extra headers update. If no 'User-Agent' key is present in the new headers, the first item in the tuple will be None", "response": "def update_headers(self, header_args):\n\t\t'''\n\t\tGiven a set of headers, update both the user-agent\n\t\tand additional headers for the remote browser.\n\n\t\theader_args must be a dict. Keys are the names of\n\t\tthe corresponding HTTP header.\n\n\t\treturn value is a 2-tuple of the results of the user-agent\n\t\tupdate, as well as the extra headers update.\n\t\tIf no 'User-Agent' key is present in the new headers,\n\t\tthe first item in the tuple will be None\n\n\t\t'''\n\t\tassert isinstance(header_args, dict), \"header_args must be a dict, passed type was %s\" \\\n\t\t\t% (type(header_args), )\n\n\t\tua = header_args.pop('User-Agent', None)\n\t\tret_1 = None\n\t\tif ua:\n\t\t\tret_1 = self.Network_setUserAgentOverride(userAgent=ua)\n\n\n\t\tret_2 = self.Network_setExtraHTTPHeaders(headers = header_args)\n\n\t\treturn (ret_1, ret_2)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_cookies(self):\n\t\t'''\n\t\tRetreive the cookies from the remote browser.\n\n\t\tReturn value is a list of http.cookiejar.Cookie() instances.\n\t\tThese can be directly used with the various http.cookiejar.XXXCookieJar\n\t\tcookie management classes.\n\t\t'''\n\t\tret = self.Network_getAllCookies()\n\n\t\tassert 'result' in ret, \"No return value in function response!\"\n\t\tassert 'cookies' in ret['result'], \"No 'cookies' key in function response\"\n\n\t\tcookies = []\n\t\tfor raw_cookie in ret['result']['cookies']:\n\n\t\t\t# Chromium seems to support the following key values for the cookie dict:\n\t\t\t# \t\"name\"\n\t\t\t# \t\"value\"\n\t\t\t# \t\"domain\"\n\t\t\t# \t\"path\"\n\t\t\t# \t\"expires\"\n\t\t\t# \t\"httpOnly\"\n\t\t\t# \t\"session\"\n\t\t\t# \t\"secure\"\n\t\t\t#\n\t\t\t# This seems supported by the fact that the underlying chromium cookie implementation has\n\t\t\t# the following members:\n\t\t\t# std::string name_;\n\t\t\t# std::string value_;\n\t\t\t# std::string domain_;\n\t\t\t# std::string path_;\n\t\t\t# base::Time creation_date_;\n\t\t\t# base::Time expiry_date_;\n\t\t\t# base::Time last_access_date_;\n\t\t\t# bool secure_;\n\t\t\t# bool httponly_;\n\t\t\t# CookieSameSite same_site_;\n\t\t\t# CookiePriority priority_;\n\t\t\t#\n\t\t\t# See chromium/net/cookies/canonical_cookie.h for more.\n\t\t\t#\n\t\t\t# I suspect the python cookie implementation is derived exactly from the standard, while the\n\t\t\t# chromium implementation is more of a practically derived structure.\n\n\t\t\t# Network.setCookie\n\n\t\t\tbaked_cookie = http.cookiejar.Cookie(\n\t\t\t\t\t# We assume V0 cookies, principally because I don't think I've /ever/ actually encountered a V1 cookie.\n\t\t\t\t\t# Chromium doesn't seem to specify it.\n\t\t\t\t\tversion = 0,\n\n\t\t\t\t\tname = raw_cookie['name'],\n\t\t\t\t\tvalue = raw_cookie['value'],\n\t\t\t\t\tport = None,\n\t\t\t\t\tport_specified = False,\n\t\t\t\t\tdomain = raw_cookie['domain'],\n\t\t\t\t\tdomain_specified = True,\n\t\t\t\t\tdomain_initial_dot = False,\n\t\t\t\t\tpath = raw_cookie['path'],\n\t\t\t\t\tpath_specified = False,\n\t\t\t\t\tsecure = raw_cookie['secure'],\n\t\t\t\t\texpires = raw_cookie['expires'],\n\t\t\t\t\tdiscard = raw_cookie['session'],\n\t\t\t\t\tcomment = None,\n\t\t\t\t\tcomment_url = None,\n\t\t\t\t\trest = {\"httponly\":\"%s\" % raw_cookie['httpOnly']},\n\t\t\t\t\trfc2109 = False\n\t\t\t\t)\n\t\t\tcookies.append(baked_cookie)\n\n\t\treturn cookies", "response": "Return the cookies from the remote browser."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting a cookie on the remote chromium instance.", "response": "def set_cookie(self, cookie):\n\t\t'''\n\t\tAdd a cookie to the remote chromium instance.\n\n\t\tPassed value `cookie` must be an instance of `http.cookiejar.Cookie()`.\n\t\t'''\n\n\t\t# Function path: Network.setCookie\n\t\t# Domain: Network\n\t\t# Method name: setCookie\n\t\t# WARNING: This function is marked 'Experimental'!\n\t\t# Parameters:\n\t\t# Required arguments:\n\t\t# 'url' (type: string) -> The request-URI to associate with the setting of the cookie. This value can affect the default domain and path values of the created cookie.\n\t\t# 'name' (type: string) -> The name of the cookie.\n\t\t# 'value' (type: string) -> The value of the cookie.\n\t\t# Optional arguments:\n\t\t# 'domain' (type: string) -> If omitted, the cookie becomes a host-only cookie.\n\t\t# 'path' (type: string) -> Defaults to the path portion of the url parameter.\n\t\t# 'secure' (type: boolean) -> Defaults ot false.\n\t\t# 'httpOnly' (type: boolean) -> Defaults to false.\n\t\t# 'sameSite' (type: CookieSameSite) -> Defaults to browser default behavior.\n\t\t# 'expirationDate' (type: Timestamp) -> If omitted, the cookie becomes a session cookie.\n\t\t# Returns:\n\t\t# 'success' (type: boolean) -> True if successfully set cookie.\n\n\t\t# Description: Sets a cookie with the given cookie data; may overwrite equivalent cookies if they exist.\n\n\t\tassert isinstance(cookie, http.cookiejar.Cookie), 'The value passed to `set_cookie` must be an instance of http.cookiejar.Cookie().' + \\\n\t\t\t' Passed: %s (\"%s\").' % (type(cookie), cookie)\n\n\t\t# Yeah, the cookielib stores this attribute as a string, despite it containing a\n\t\t# boolean value. No idea why.\n\t\tis_http_only = str(cookie.get_nonstandard_attr('httponly', 'False')).lower() == \"true\"\n\n\n\t\t# I'm unclear what the \"url\" field is actually for. A cookie only needs the domain and\n\t\t# path component to be fully defined. Considering the API apparently allows the domain and\n\t\t# path parameters to be unset, I think it forms a partially redundant, with some\n\t\t# strange interactions with mode-changing between host-only and more general\n\t\t# cookies depending on what's set where.\n\t\t# Anyways, given we need a URL for the API to work properly, we produce a fake\n\t\t# host url by building it out of the relevant cookie properties.\n\t\tfake_url = urllib.parse.urlunsplit((\n\t\t\t\t\"http\" if is_http_only else \"https\", # Scheme\n\t\t\t\tcookie.domain, # netloc\n\t\t\t\tcookie.path, # path\n\t\t\t\t'', # query\n\t\t\t\t'', # fragment\n\t\t\t))\n\n\t\tparams = {\n\t\t\t\t'url' : fake_url,\n\n\t\t\t\t'name' : cookie.name,\n\t\t\t\t'value' : cookie.value if cookie.value else \"\",\n\t\t\t\t'domain' : cookie.domain,\n\t\t\t\t'path' : cookie.path,\n\t\t\t\t'secure' : cookie.secure,\n\t\t\t\t'expires' : float(cookie.expires) if cookie.expires else float(2**32),\n\n\t\t\t\t'httpOnly' : is_http_only,\n\n\t\t\t\t# The \"sameSite\" flag appears to be a chromium-only extension for controlling\n\t\t\t\t# cookie sending in non-first-party contexts. See:\n\t\t\t\t# https://bugs.chromium.org/p/chromium/issues/detail?id=459154\n\t\t\t\t# Anyways, we just use the default here, whatever that is.\n\t\t\t\t# sameSite = cookie.xxx\n\t\t\t}\n\n\t\tret = self.Network_setCookie(**params)\n\n\t\treturn ret"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the URL of the current window.", "response": "def get_current_url(self):\n\t\t'''\n\t\tProbe the remote session for the current window URL.\n\n\t\tThis is primarily used to do things like unwrap redirects,\n\t\tor circumvent outbound url wrappers.\n\n\t\t'''\n\t\tres = self.Page_getNavigationHistory()\n\t\tassert 'result' in res\n\t\tassert 'currentIndex' in res['result']\n\t\tassert 'entries' in res['result']\n\n\t\treturn res['result']['entries'][res['result']['currentIndex']]['url']"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the title and current url from the remote session.", "response": "def get_page_url_title(self):\n\t\t'''\n\t\tGet the title and current url from the remote session.\n\n\t\tReturn is a 2-tuple: (page_title, page_url).\n\n\t\t'''\n\n\t\tcr_tab_id = self.transport._get_cr_tab_meta_for_key(self.tab_id)['id']\n\t\ttargets = self.Target_getTargets()\n\n\t\tassert 'result' in targets\n\t\tassert 'targetInfos' in targets['result']\n\n\t\tfor tgt in targets['result']['targetInfos']:\n\t\t\tif tgt['targetId'] == cr_tab_id:\n\t\t\t\t# {\n\t\t\t\t# \t'title': 'Page Title 1',\n\t\t\t\t# \t'targetId': '9d2c503c-e39e-42cc-b950-96db073918ee',\n\t\t\t\t# \t'attached': True,\n\t\t\t\t# \t'url': 'http://localhost:47181/with_title_1',\n\t\t\t\t# \t'type': 'page'\n\t\t\t\t# }\n\n\t\t\t\ttitle = tgt['title']\n\t\t\t\tcur_url = tgt['url']\n\t\t\t\treturn title, cur_url"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting a javascript string in the context of the browser tab.", "response": "def execute_javascript(self, *args, **kwargs):\n\t\t'''\n\t\tExecute a javascript string in the context of the browser tab.\n\t\t'''\n\n\t\tret = self.__exec_js(*args, **kwargs)\n\t\treturn ret"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_element(self, search):\n\n\t\t'''\n\t\tDOM_performSearch(self, query, includeUserAgentShadowDOM)\n\t\tPython Function: DOM_performSearch\n\t\t Domain: DOM\n\t\t Method name: performSearch\n\n\t\t WARNING: This function is marked 'Experimental'!\n\n\t\t Parameters:\n\t\t 'query' (type: string) -> Plain text or query selector or XPath search query.\n\t\t 'includeUserAgentShadowDOM' (type: boolean) -> True to search in user agent shadow DOM.\n\t\t Returns:\n\t\t 'searchId' (type: string) -> Unique search session identifier.\n\t\t 'resultCount' (type: integer) -> Number of search results.\n\t\t Description: Searches for a given string in the DOM tree. Use getSearchResults to access search results or cancelSearch to end this search session.\n\n\t\tPython Function: DOM_getSearchResults\n\t\t Domain: DOM\n\t\t Method name: getSearchResults\n\n\t\t WARNING: This function is marked 'Experimental'!\n\n\t\t Parameters:\n\t\t 'searchId' (type: string) -> Unique search session identifier.\n\t\t 'fromIndex' (type: integer) -> Start index of the search result to be returned.\n\t\t 'toIndex' (type: integer) -> End index of the search result to be returned.\n\t\t Returns:\n\t\t 'nodeIds' (type: array) -> Ids of the search result nodes.\n\t\t Description: Returns search results from given fromIndex to given toIndex from the sarch with the given identifier.\n\n\t\tDOM_discardSearchResults(self, searchId)\n\t\tPython Function: DOM_discardSearchResults\n\t\t Domain: DOM\n\t\t Method name: discardSearchResults\n\n\t\t WARNING: This function is marked 'Experimental'!\n\n\t\t Parameters:\n\t\t 'searchId' (type: string) -> Unique search session identifier.\n\t\t No return value.\n\t\t Description: Discards search results from the session with the given id. getSearchResults should no longer be called for that search.\n\t\t'''\n\n\t\tres = self.DOM_performSearch(search, includeUserAgentShadowDOM=False)\n\t\tassert 'result' in res\n\t\tassert 'searchId' in res['result']\n\t\tsearchid = res['result']['searchId']\n\t\tres_cnt = res['result']['resultCount']\n\t\tself.log.debug(\"%s\", res)\n\t\tself.log.debug(\"%s\", searchid)\n\n\t\tif res_cnt == 0:\n\t\t\treturn None\n\n\t\titems = self.DOM_getSearchResults(searchId=searchid, fromIndex=0, toIndex=res_cnt)\n\n\t\tself.log.debug(\"Results:\")\n\t\tself.log.debug(\"%s\", items)", "response": "This method searches the DOM tree for a given search string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the response body from Network_getResponseBody and return it as a dict.", "response": "def get_unpacked_response_body(self, requestId, mimetype=\"application/unknown\"):\n\t\t'''\n\t\tReturn a unpacked, decoded resposne body from Network_getResponseBody()\n\t\t'''\n\t\tcontent = self.Network_getResponseBody(requestId)\n\n\t\tassert 'result' in content\n\t\tresult = content['result']\n\n\t\tassert 'base64Encoded' in result\n\t\tassert 'body' in result\n\n\t\tif result['base64Encoded']:\n\t\t\tcontent = base64.b64decode(result['body'])\n\t\telse:\n\t\t\tcontent = result['body']\n\n\t\tself.log.info(\"Navigate complete. Received %s byte response with type %s.\", len(content), mimetype)\n\n\t\treturn {'binary' : result['base64Encoded'], 'mimetype' : mimetype, 'content' : content}"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef handle_page_location_changed(self, timeout=None):\n\t\t'''\n\t\tIf the chrome tab has internally redirected (generally because jerberscript), this\n\t\twill walk the page navigation responses and attempt to fetch the response body for\n\t\tthe tab's latest location.\n\t\t'''\n\n\t\t# In general, this is often called after other mechanisms have confirmed\n\t\t# that the tab has already navigated. As such, we want to not wait a while\n\t\t# to discover something went wrong, so use a timeout that basically just\n\t\t# results in checking the available buffer, and nothing else.\n\t\tif not timeout:\n\t\t\ttimeout = 0.1\n\n\t\tself.log.debug(\"We may have redirected. Checking.\")\n\n\t\tmessages = self.transport.recv_all_filtered(filter_funcs.capture_loading_events, tab_key=self.tab_id)\n\t\tif not messages:\n\t\t\traise ChromeError(\"Couldn't track redirect! No idea what to do!\")\n\n\t\tlast_message = messages[-1]\n\t\tself.log.info(\"Probably a redirect! New content url: '%s'\", last_message['params']['documentURL'])\n\n\t\tresp = self.transport.recv_filtered(filter_funcs.network_response_recieved_for_url(last_message['params']['documentURL'], last_message['params']['frameId']), tab_key=self.tab_id)\n\t\tresp = resp['params']\n\n\t\tctype = 'application/unknown'\n\n\t\tresp_response = resp['response']\n\n\t\tif 'mimeType' in resp_response:\n\t\t\tctype = resp_response['mimeType']\n\t\tif 'headers' in resp_response and 'content-type' in resp_response['headers']:\n\t\t\tctype = resp_response['headers']['content-type'].split(\";\")[0]\n\n\t\t# We assume the last document request was the redirect.\n\t\t# This is /probably/ kind of a poor practice, but what the hell.\n\t\t# I have no idea what this would do if there are non-html documents (or if that can even happen.)\n\t\treturn self.get_unpacked_response_body(last_message['params']['requestId'], mimetype=ctype)", "response": "This method is called when the page location has changed."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndoing a blocking navigate to url `url`, and then extract the response body and return that. This effectively returns the *unrendered* page content that's sent over the wire. As such, if the page does any modification of the contained markup during rendering (via javascript), this function will not reflect the changes made by the javascript. The rendered page content can be retreived by calling `get_rendered_page_source()`. Due to the remote api structure, accessing the raw content after the content has been loaded is not possible, so any task requiring the raw content must be careful to request it before it actually navigates to said content. Return value is a dictionary with two keys: { 'binary' : (boolean, true if content is binary, false if not) 'content' : (string of bytestring, depending on whether `binary` is true or not) }", "response": "def blocking_navigate_and_get_source(self, url, timeout=DEFAULT_TIMEOUT_SECS):\n\t\t'''\n\t\tDo a blocking navigate to url `url`, and then extract the\n\t\tresponse body and return that.\n\n\t\tThis effectively returns the *unrendered* page content that's sent over the wire. As such,\n\t\tif the page does any modification of the contained markup during rendering (via javascript), this\n\t\tfunction will not reflect the changes made by the javascript.\n\n\t\tThe rendered page content can be retreived by calling `get_rendered_page_source()`.\n\n\t\tDue to the remote api structure, accessing the raw content after the content has been loaded\n\t\tis not possible, so any task requiring the raw content must be careful to request it\n\t\tbefore it actually navigates to said content.\n\n\t\tReturn value is a dictionary with two keys:\n\t\t{\n\t\t\t'binary' : (boolean, true if content is binary, false if not)\n\t\t\t'content' : (string of bytestring, depending on whether `binary` is true or not)\n\t\t}\n\n\t\t'''\n\n\n\t\tresp = self.blocking_navigate(url, timeout)\n\t\tassert 'requestId' in resp\n\t\tassert 'response' in resp\n\t\t# self.log.debug('blocking_navigate Response %s', pprint.pformat(resp))\n\n\t\tctype = 'application/unknown'\n\n\t\tresp_response = resp['response']\n\n\t\tif 'mimeType' in resp_response:\n\t\t\tctype = resp_response['mimeType']\n\t\tif 'headers' in resp_response and 'content-type' in resp_response['headers']:\n\t\t\tctype = resp_response['headers']['content-type'].split(\";\")[0]\n\n\t\tself.log.debug(\"Trying to get response body\")\n\t\ttry:\n\t\t\tret = self.get_unpacked_response_body(resp['requestId'], mimetype=ctype)\n\t\texcept ChromeError:\n\t\t\tret = self.handle_page_location_changed(timeout)\n\n\t\treturn ret"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_rendered_page_source(self, dom_idle_requirement_secs=3, max_wait_timeout=30):\n\t\t'''\n\t\tGet the HTML markup for the current page.\n\n\t\tThis is done by looking up the root DOM node, and then requesting the outer HTML\n\t\tfor that node ID.\n\n\t\tThis calls return will reflect any modifications made by javascript to the\n\t\tpage. For unmodified content, use `blocking_navigate_and_get_source()`\n\n\t\tdom_idle_requirement_secs specifies the period of time for which there must have been no\n\t\tDOM modifications before treating the rendered output as \"final\". This call will therefore block for\n\t\tat least dom_idle_requirement_secs seconds.\n\t\t'''\n\n\t\t# There are a bunch of events which generally indicate a page is still doing *things*.\n\t\t# I have some concern about how this will handle things like advertisements, which\n\t\t# basically load crap forever. That's why we have the max_wait_timeout.\n\t\ttarget_events = [\n\t\t\t\"Page.frameResized\",\n\t\t\t\"Page.frameStartedLoading\",\n\t\t\t\"Page.frameNavigated\",\n\t\t\t\"Page.frameAttached\",\n\t\t\t\"Page.frameStoppedLoading\",\n\t\t\t\"Page.frameScheduledNavigation\",\n\t\t\t\"Page.domContentEventFired\",\n\t\t\t\"Page.frameClearedScheduledNavigation\",\n\t\t\t\"Page.loadEventFired\",\n\t\t\t\"DOM.documentUpdated\",\n\t\t\t\"DOM.childNodeInserted\",\n\t\t\t\"DOM.childNodeRemoved\",\n\t\t\t\"DOM.childNodeCountUpdated\",\n\t\t]\n\n\t\tstart_time = time.time()\n\t\ttry:\n\t\t\twhile 1:\n\t\t\t\tif time.time() - start_time > max_wait_timeout:\n\t\t\t\t\tself.log.debug(\"Page was not idle after waiting %s seconds. Giving up and extracting content now.\", max_wait_timeout)\n\t\t\t\tself.transport.recv_filtered(filter_funcs.wait_for_methods(target_events),\n\t\t\t\t\ttab_key=self.tab_id, timeout=dom_idle_requirement_secs)\n\n\t\texcept ChromeResponseNotReceived:\n\t\t\t# We timed out, the DOM is probably idle.\n\t\t\tpass\n\n\n\n\t\t# We have to find the DOM root node ID\n\t\tdom_attr = self.DOM_getDocument(depth=-1, pierce=False)\n\t\tassert 'result' in dom_attr\n\t\tassert 'root' in dom_attr['result']\n\t\tassert 'nodeId' in dom_attr['result']['root']\n\n\t\t# Now, we have the root node ID.\n\t\troot_node_id = dom_attr['result']['root']['nodeId']\n\n\t\t# Use that to get the HTML for the specified node\n\t\tresponse = self.DOM_getOuterHTML(nodeId=root_node_id)\n\n\t\tassert 'result' in response\n\t\tassert 'outerHTML' in response['result']\n\t\treturn response['result']['outerHTML']", "response": "Get the rendered HTML markup for the current page."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef take_screeshot(self):\n\t\t'''\n\t\tTake a screenshot of the virtual viewport content.\n\n\t\tReturn value is a png image as a bytestring.\n\t\t'''\n\t\tresp = self.Page_captureScreenshot()\n\t\tassert 'result' in resp\n\t\tassert 'data' in resp['result']\n\t\timgdat = base64.b64decode(resp['result']['data'])\n\t\treturn imgdat", "response": "Take a screenshot of the virtual viewport content."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef blocking_navigate(self, url, timeout=DEFAULT_TIMEOUT_SECS):\n\t\t'''\n\t\tDo a blocking navigate to url `url`.\n\n\t\tThis function triggers a navigation, and then waits for the browser\n\t\tto claim the page has finished loading.\n\n\t\tRoughly, this corresponds to the javascript `DOMContentLoaded` event,\n\t\tmeaning the dom for the page is ready.\n\n\n\t\tInternals:\n\n\t\tA navigation command results in a sequence of events:\n\n\t\t - Page.frameStartedLoading\" (with frameid)\n\t\t - Page.frameStoppedLoading\" (with frameid)\n\t\t - Page.loadEventFired\" (not attached to an ID)\n\n\t\tTherefore, this call triggers a navigation option,\n\t\tand then waits for the expected set of response event messages.\n\n\t\t'''\n\n\t\tself.transport.flush(tab_key=self.tab_id)\n\n\t\tret = self.Page_navigate(url = url)\n\n\t\tassert(\"result\" in ret), \"Missing return content\"\n\t\tassert(\"frameId\" in ret['result']), \"Missing 'frameId' in return content\"\n\t\tassert(\"loaderId\" in ret['result']), \"Missing 'loaderId' in return content\"\n\n\t\texpected_id = ret['result']['frameId']\n\t\tloader_id = ret['result']['loaderId']\n\n\t\ttry:\n\t\t\tself.log.debug(\"Waiting for frame navigated command response.\")\n\t\t\tself.transport.recv_filtered(filter_funcs.check_frame_navigated_command(expected_id), tab_key=self.tab_id, timeout=timeout)\n\t\t\tself.log.debug(\"Waiting for frameStartedLoading response.\")\n\t\t\tself.transport.recv_filtered(filter_funcs.check_frame_load_command(\"Page.frameStartedLoading\"), tab_key=self.tab_id, timeout=timeout)\n\t\t\tself.log.debug(\"Waiting for frameStoppedLoading response.\")\n\t\t\tself.transport.recv_filtered(filter_funcs.check_frame_load_command(\"Page.frameStoppedLoading\"), tab_key=self.tab_id, timeout=timeout)\n\t\t\t# self.transport.recv_filtered(check_load_event_fired, tab_key=self.tab_id, timeout=timeout)\n\n\t\t\tself.log.debug(\"Waiting for responseReceived response.\")\n\t\t\tresp = self.transport.recv_filtered(filter_funcs.network_response_recieved_for_url(url=None, expected_id=expected_id), tab_key=self.tab_id, timeout=timeout)\n\n\t\t\tif resp is None:\n\t\t\t\traise ChromeNavigateTimedOut(\"Blocking navigate timed out!\")\n\n\t\t\treturn resp['params']\n\t\t# The `Page.frameNavigated ` event does not get fired for non-markup responses.\n\t\t# Therefore, if we timeout on waiting for that, check to see if we received a binary response.\n\t\texcept ChromeResponseNotReceived:\n\t\t\t# So this is basically broken, fix is https://bugs.chromium.org/p/chromium/issues/detail?id=831887\n\t\t\t# but that bug report isn't fixed yet.\n\t\t\t# Siiiigh.\n\t\t\tself.log.warning(\"Failed to receive expected response to navigate command. Checking if response is a binary object.\")\n\t\t\tresp = self.transport.recv_filtered(\n\t\t\t\tkeycheck = filter_funcs.check_frame_loader_command(\n\t\t\t\t\t\tmethod_name = \"Network.responseReceived\",\n\t\t\t\t\t\tloader_id = loader_id\n\t\t\t\t\t),\n\t\t\t\ttab_key = self.tab_id,\n\t\t\t\ttimeout = timeout)\n\n\t\t\treturn resp['params']", "response": "This method is used to send a blocking navigate command to url."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tab(self, netloc=None, url=None, extra_id=None, use_tid=False):\n\t\t'''\n\t\tGet a chromium tab from the pool, optionally one that has an association with a specific netloc/URL.\n\n\t\tIf no url or netloc is specified, the per-thread identifier will be used.\n\t\tIf `extra_id` is specified, it's stringified value will be mixed into the pool key\n\t\tIf `use_tid` is true, the per-thread identifier will be mixed into the pool key.\n\n\t\tIn all cases, the tab pool is a least-recently-used cache, so the tab that has been accessed the\n\t\tleast recently will be automatically closed if a new tab is requested, and there are already\n\t\t`tab_pool_max_size` tabs created.\n\n\t\t'''\n\t\tassert self.alive, \"Chrome has been shut down! Cannot continue!\"\n\t\tif not netloc and url:\n\t\t\tnetloc = urllib.parse.urlparse(url).netloc\n\t\t\tself.log.debug(\"Getting tab for netloc: %s (url: %s)\", netloc, url)\n\t\t# Coerce to string type so even if it's none, it doesn't hurt anything.\n\t\tkey = str(netloc)\n\t\tif extra_id:\n\t\t\tkey += \" \" + str(extra_id)\n\t\tif use_tid or not key:\n\t\t\tkey += \" \" + str(threading.get_ident())\n\n\t\tif self.__started_pid != os.getpid():\n\t\t\tself.log.error(\"TabPooledChromium instances are not safe to share across multiple processes.\")\n\t\t\tself.log.error(\"Please create a new in each separate multiprocesssing process.\")\n\t\t\traise RuntimeError(\"TabPooledChromium instances are not safe to share across multiple processes.\")\n\n\t\twith self.__counter_lock:\n\t\t\tself.__active_tabs.setdefault(key, 0)\n\t\t\tself.__active_tabs[key] += 1\n\t\t\tif self.__active_tabs[key] > 1:\n\t\t\t\tself.log.warning(\"Tab with key %s checked out more then once simultaneously\")\n\n\t\ttry:\n\t\t\tlock, tab = self.__tab_cache[key]\n\t\t\twith lock:\n\t\t\t\tyield tab\n\t\tfinally:\n\n\t\t\twith self.__counter_lock:\n\t\t\t\tself.__active_tabs[key] -= 1\n\t\t\t\tif self.__active_tabs[key] == 0:\n\t\t\t\t\tself.__active_tabs.pop(key)", "response": "Get a chromium tab from the pool optionally one that has an association with a specific netloc and URL."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a function to be run when the parent process dies", "response": "def on_parent_exit(signame):\n \"\"\"\n Return a function to be run in a child process which will trigger SIGNAME\n to be sent when the parent process dies\n \"\"\"\n signum = getattr(signal, signame)\n def set_parent_exit_signal():\n # http://linux.die.net/man/2/prctl\n result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)\n if result != 0:\n raise PrCtlError('prctl failed with error code %s' % result)\n return set_parent_exit_signal"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ChromeContext(*args, **kwargs):\n\t'''\n\tContext manager for conveniently handling the lifetime of the underlying chromium instance.\n\n\tIn general, this should be the preferred way to use an instance of `ChromeRemoteDebugInterface`.\n\n\tAll parameters are forwarded through to the underlying ChromeRemoteDebugInterface() constructor.\n\t'''\n\tlog = logging.getLogger(\"Main.ChromeController.ChromeContext\")\n\tchrome_created = False\n\ttry:\n\t\tchrome_instance = ChromeRemoteDebugInterface(*args, **kwargs)\n\t\tchrome_created = True\n\t\tlog.info(\"Entering chrome context\")\n\t\tyield chrome_instance\n\texcept Exception as e:\n\n\t\tlog.error(\"Exception in chrome context!\")\n\t\tfor line in traceback.format_exc().split(\"\\n\"):\n\t\t\tlog.error(line)\n\t\traise e\n\n\tfinally:\n\t\tlog.info(\"Exiting chrome context\")\n\t\tif chrome_created:\n\t\t\tchrome_instance.close()", "response": "A context manager for conveniently handling the lifetime of a chrome instance."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nforwarding a command to the remote chrome instance via the transport connection, and check the return for an error. If the command resulted in an error, a `ChromeController.ChromeError` is raised, with the error string containing the response from the remote chrome instance describing the problem and it's cause. Otherwise, the decoded json data-structure returned from the remote instance is returned.", "response": "def synchronous_command(self, *args, **kwargs):\n\t\t'''\n\t\tForward a command to the remote chrome instance via the transport\n\t\tconnection, and check the return for an error.\n\n\t\tIf the command resulted in an error, a `ChromeController.ChromeError` is raised,\n\t\twith the error string containing the response from the remote\n\t\tchrome instance describing the problem and it's cause.\n\n\t\tOtherwise, the decoded json data-structure returned from the remote instance is\n\t\treturned.\n\n\t\t'''\n\n\t\tself.transport.check_process_ded()\n\t\tret = self.transport.synchronous_command(tab_key=self.tab_id, *args, **kwargs)\n\t\tself.transport.check_process_ded()\n\t\tself.__check_ret(ret)\n\t\tself.transport.check_process_ded()\n\t\treturn ret"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef drain_transport(self):\n\t\t'''\n\t\t\"Drain\" the transport connection.\n\n\t\tThis command simply returns all waiting messages sent from the remote chrome\n\t\tinstance. This can be useful when waiting for a specific asynchronous message\n\t\tfrom chrome, but higher level calls are better suited for managing wait-for-message\n\t\ttype needs.\n\n\t\t'''\n\t\tself.transport.check_process_ded()\n\t\tret = self.transport.drain(tab_key=self.tab_id)\n\t\tself.transport.check_process_ded()\n\t\treturn ret", "response": "Drain the transport connection."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef resetLoggingLocks():\n\t'''\n\tThis function is a HACK!\n\n\tBasically, if we fork() while a logging lock is held, the lock\n\tis /copied/ while in the acquired state. However, since we've\n\tforked, the thread that acquired the lock no longer exists,\n\tso it can never unlock the lock, and we end up blocking\n\tforever.\n\n\tTherefore, we manually enter the logging module, and forcefully\n\trelease all the locks it holds.\n\n\tTHIS IS NOT SAFE (or thread-safe).\n\tBasically, it MUST be called right after a process\n\tstarts, and no where else.\n\t'''\n\ttry:\n\t\tlogging._releaseLock()\n\texcept RuntimeError:\n\t\tpass # The lock is already released\n\n\t# Iterate over the root logger hierarchy, and\n\t# force-free all locks.\n\t# if logging.Logger.root\n\tfor handler in logging.Logger.manager.loggerDict.values():\n\t\tif hasattr(handler, \"lock\") and handler.lock:\n\t\t\ttry:\n\t\t\t\thandler.lock.release()\n\t\t\texcept RuntimeError:\n\t\t\t\tpass", "response": "This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK! This function is a HACK!"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nemits a record to the output stream.", "response": "def stream_emit(self, record, source_name):\n\t\t\"\"\"\n\t\tEmit a record.\n\n\t\tIf a formatter is specified, it is used to format the record.\n\t\tThe record is then written to the stream with a trailing newline. If\n\t\texception information is present, it is formatted using\n\t\ttraceback.print_exception and appended to the stream. If the stream\n\t\thas an 'encoding' attribute, it is used to determine how to do the\n\t\toutput to the stream.\n\t\t\"\"\"\n\n\t\tif not source_name in self.output_streams:\n\t\t\tout_path = os.path.abspath(\"./logs\")\n\t\t\tlogpath = ansi_escape.sub('', source_name.replace(\"/\", \";\").replace(\":\", \";\").replace(\"?\", \"-\"))\n\t\t\tfilename = \"log {path}.txt\".format(path=logpath)\n\t\t\tprint(\"Opening output log file for path: %s\" % filename)\n\t\t\tself.output_streams[source_name] = open(os.path.join(out_path, filename), self.mode, encoding=self.encoding)\n\n\t\tstream = self.output_streams[source_name]\n\t\ttry:\n\t\t\tmsg = self.format(record)\n\t\t\tstream.write(msg)\n\t\t\tstream.write(self.terminator)\n\t\t\tstream.flush()\n\t\t\tself.flush()\n\t\texcept Exception:\n\t\t\tself.handleError(record)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nemit a record. If the stream was not opened because 'delay' was specified in the constructor, open it before calling the superclass's emit.", "response": "def emit(self, record):\n\t\t\"\"\"\n\t\tEmit a record.\n\n\t\tIf the stream was not opened because 'delay' was specified in the\n\t\tconstructor, open it before calling the superclass's emit.\n\t\t\"\"\"\n\t\tfailures = 0\n\t\twhile failures < 3:\n\t\t\ttry:\n\t\t\t\tself.stream_emit(record, record.name)\n\t\t\t\tbreak\n\t\t\texcept:\n\t\t\t\tfailures += 1\n\t\telse:\n\t\t\ttraceback.print_stack()\n\t\t\tprint(\"Error writing to file?\")\n\n\n\t\tself.close()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_suite(self, suite, **kwargs):\n return self.test_runner(\n verbosity=self.verbosity,\n failfast=self.failfast,\n no_colour=self.no_colour,\n ).run(suite)", "response": "This is the version from Django 1. 7."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a private key of key_length bits and attaches it to the object as the private_key variable.", "response": "def generate_private_key(self):\n \"\"\"\n Generates a private key of key_length bits and attaches it to the object as the __private_key variable.\n\n :return: void\n :rtype: void\n \"\"\"\n key_length = self.key_length // 8 + 8\n key = 0\n\n try:\n key = int.from_bytes(rng(key_length), byteorder='big')\n except:\n key = int(hex(rng(key_length)), base=16)\n\n self.__private_key = key"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates public key. :return: void :rtype: void", "response": "def generate_public_key(self):\n \"\"\"\n Generates public key.\n\n :return: void\n :rtype: void\n \"\"\"\n self.public_key = pow(self.generator,\n self.__private_key,\n self.prime)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a shared secret from the other party s public key.", "response": "def generate_shared_secret(self, other_public_key, echo_return_key=False):\n \"\"\"\n Generates shared secret from the other party's public key.\n\n :param other_public_key: Other party's public key\n :type other_public_key: int\n :param echo_return_key: Echo return shared key\n :type bool\n :return: void\n :rtype: void\n \"\"\"\n if self.verify_public_key(other_public_key) is False:\n raise MalformedPublicKey\n\n self.shared_secret = pow(other_public_key,\n self.__private_key,\n self.prime)\n\n shared_secret_as_bytes = self.shared_secret.to_bytes(self.shared_secret.bit_length() // 8 + 1, byteorder='big')\n\n _h = sha256()\n _h.update(bytes(shared_secret_as_bytes))\n\n self.shared_key = _h.hexdigest()\n\n if echo_return_key is True:\n return self.shared_key"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef requires_public_key(func):\n\n def func_wrapper(self, *args, **kwargs):\n if hasattr(self, \"public_key\"):\n func(self, *args, **kwargs)\n else:\n self.generate_public_key()\n func(self, *args, **kwargs)\n\n return func_wrapper", "response": "Decorator for functions that require a public key to be defined."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprint the debug flag if set to True in the config. yaml file.", "response": "def print_debug(*args, **kwargs):\n \"\"\"\n Print if and only if the debug flag is set true in the config.yaml file.\n\n Args:\n args : var args of print arguments.\n\n \"\"\"\n if WTF_CONFIG_READER.get(\"debug\", False) == True:\n print(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting the current elapsed time.", "response": "def print_time(self, message=\"Time is now: \", print_frame_info=True):\n \"\"\"\n Print the current elapsed time.\n\n Kwargs:\n message (str) : Message to prefix the time stamp.\n print_frame_info (bool) : Add frame info to the print message.\n\n \"\"\"\n if print_frame_info:\n frame_info = inspect.getouterframes(inspect.currentframe())[1]\n print(message, (datetime.now() - self.start_time), frame_info)\n else:\n print(message, (datetime.now() - self.start_time))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if resource at URL is fetchable.", "response": "def check_url(url):\n '''\n Check if resource at URL is fetchable. (by trying to fetch it and checking for 200 status.\n\n Args:\n url (str): Url to check.\n\n Returns:\n Returns a tuple of {True/False, response code}\n\n '''\n request = urllib2.Request(url)\n try:\n response = urlopen(request)\n return True, response.code\n except urllib2.HTTPError as e:\n return False, e.code"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_base_url(webdriver):\n current_url = webdriver.current_url\n try:\n return re.findall(\"^[^/]+//[^/$]+\", current_url)[0]\n except:\n raise RuntimeError(\n u(\"Unable to process base url: {0}\").format(current_url))", "response": "Get the current base URL."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_browser_datetime(webdriver):\n js_stmt = \"\"\"\n var wtf_get_date = new Date();\n return {'month':wtf_get_date.getMonth(), \n 'day':wtf_get_date.getDate(), \n 'year':wtf_get_date.getFullYear(),\n 'hours':wtf_get_date.getHours(),\n 'minutes':wtf_get_date.getMinutes(),\n 'seconds':wtf_get_date.getSeconds(),\n 'milliseconds':wtf_get_date.getMilliseconds()};\n \"\"\"\n browser_date = webdriver.execute_script(js_stmt)\n return datetime(int(browser_date['year']),\n int(browser_date['month']) + 1, # javascript months start at 0 \n int(browser_date['day']),\n int(browser_date['hours']),\n int(browser_date['minutes']),\n int(browser_date['seconds']),\n int(browser_date['milliseconds']))", "response": "Get the current date and time on the web browser as a Python datetime object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if a web driver is mobile", "response": "def is_webdriver_mobile(webdriver):\n \"\"\"\n Check if a web driver if mobile.\n\n Args:\n webdriver (WebDriver): Selenium webdriver.\n\n \"\"\"\n browser = webdriver.capabilities['browserName']\n\n if (browser == u('iPhone') or \n browser == u('android')):\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_webdriver_ios(webdriver):\n browser = webdriver.capabilities['browserName']\n\n if (browser == u('iPhone') or \n browser == u('iPad')):\n return True\n else:\n return False", "response": "Check if a web driver is iOS"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a row into a dictionary of key - value pairs.", "response": "def row_to_dictionary(header_row_web_element, row_webelement):\n \"\"\"\n Converts a row into a dictionary of key/values.\n (Note: assumes all rows/columns have uniform cells. Does not \n account for any row or column spans)\n \n Args:\n header_row_web_element (WebElement): WebElement reference to the column headers.\n row_webelement (WebElement): WebElement reference to row.\n \n Returns:\n Returns a dictionary object containing keys consistenting of the column headers \n and values consisting of the row contents.\n\n Usage::\n\n self.webdriver.get(\"http://the-internet.herokuapp.com/tables\")\n\n header = self.webdriver.find_element_by_css_selector(\"#table1 thead tr\")\n target_row = self.webdriver.find_element_by_css_selector(\"#table1 tbody tr\")\n \n row_values = WebUtils.row_to_dictionary(header, target_row)\n row_values == {'Last Name': 'Smith', \n 'Due': '$50.00',\n 'First Name': 'John', \n 'Web Site': 'http://www.jsmith.com', \n 'Action': 'edit delete', \n 'Email': 'jsmith@gmail.com'}\n\n \"\"\"\n headers = header_row_web_element.find_elements_by_tag_name(\"th\")\n data_cells = row_webelement.find_elements_by_tag_name(\"td\")\n \n value_dictionary = {}\n for i in range(len(data_cells)):\n value_dictionary[ headers[i].text ] = data_cells[i].text\n\n return value_dictionary"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an instance of BrowserStandBy and immediately return a running instance.", "response": "def start_standby(cls, webdriver=None, max_time=WTF_TIMEOUT_MANAGER.EPIC, sleep=5):\n \"\"\"\n Create an instance of BrowserStandBy() and immediately return a running instance.\n\n This is best used in a 'with' block.\n\n Example::\n\n with BrowserStandBy.start_standby():\n # Now browser is in standby, you can do a bunch of stuff with in this block.\n # ...\n\n # We are now outside the block, and the browser standby has ended.\n\n \"\"\"\n return cls(webdriver=webdriver, max_time=max_time, sleep=sleep, _autostart=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef start(self):\n self._end_time = datetime.now() + timedelta(seconds=self._max_time)\n self._thread = Thread(target=lambda: self.__stand_by_loop())\n self._keep_running = True\n self._thread.start()\n return self", "response": "Start standing by. A periodic command like 'current_url' will be sent to the \n webdriver instance to prevent it from timing out."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a temp path.", "response": "def temp_path(file_name=None):\n \"\"\"\n Gets a temp path.\n\n Kwargs:\n file_name (str) : if file name is specified, it gets appended to the temp dir.\n\n Usage::\n\n temp_file_path = temp_path(\"myfile\")\n copyfile(\"myfile\", temp_file_path) # copies 'myfile' to '/tmp/myfile'\n\n \"\"\"\n\n if file_name is None:\n file_name = generate_timestamped_string(\"wtf_temp_file\")\n\n return os.path.join(tempfile.gettempdir(), file_name)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a temporary file with the given name.", "response": "def create_temp_file(file_name=None, string_or_another_file=\"\"):\n \"\"\"\n Creates a temp file using a given name. Temp files are placed in the Project/temp/ \n directory. Any temp files being created with an existing temp file, will be \n overridden. This is useful for testing uploads, where you would want to create a \n temporary file with a desired name, upload it, then delete the file when you're \n done.\n\n Kwargs:\n file_name (str): Name of file\n string_or_another_file: Contents to set this file to. If this is set to a file, \n it will copy that file. If this is set to a string, then \n it will write this string to the temp file.\n\n Return: \n str - Returns the file path to the generated temp file.\n\n Usage::\n\n temp_file_path = create_temp_file(\"mytestfile\", \"The nimble fox jumps over the lazy dog.\")\n file_obj = open(temp_file_path)\n os.remove(temp_file_path)\n\n \"\"\"\n temp_file_path = temp_path(file_name)\n if isinstance(string_or_another_file, file):\n # attempt to read it as a file.\n temp_file = open(temp_file_path, \"wb\")\n temp_file.write(string_or_another_file.read())\n else:\n # handle as a string type if we can't handle as a file.\n temp_file = codecs.open(temp_file_path, \"w+\", \"utf-8\")\n temp_file.write(string_or_another_file)\n\n temp_file.close()\n return temp_file_path"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndownloading a URL contents to a tempfile.", "response": "def download_to_tempfile(url, file_name=None, extension=None):\n \"\"\"\n Downloads a URL contents to a tempfile. This is useful for testing downloads.\n It will download the contents of a URL to a tempfile, which you then can \n open and use to validate the downloaded contents.\n\n Args:\n url (str) : URL of the contents to download.\n\n Kwargs:\n file_name (str): Name of file.\n extension (str): Extension to use.\n\n Return:\n str - Returns path to the temp file.\n\n \"\"\"\n\n if not file_name:\n file_name = generate_timestamped_string(\"wtf_temp_file\")\n\n if extension:\n file_path = temp_path(file_name + extension)\n else:\n ext = \"\"\n try:\n ext = re.search(u\"\\\\.\\\\w+$\", file_name).group(0)\n except:\n pass\n file_path = temp_path(file_name + ext)\n\n webFile = urllib.urlopen(url)\n localFile = open(file_path, 'w')\n localFile.write(webFile.read())\n webFile.close()\n localFile.close()\n\n return file_path"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_page(cls, webdriver=None, **kwargs):\n if not webdriver:\n webdriver = WTF_WEBDRIVER_MANAGER.get_driver()\n return PageFactory.create_page(cls, webdriver=webdriver, **kwargs)", "response": "Class method short cut to call PageFactory. create_page on itself using a webdriver."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_page(page_object_class_or_interface,\n webdriver=None, **kwargs):\n \"\"\"\n Instantiate a page object from a given Interface or Abstract class.\n\n Args:\n page_object_class_or_interface (Class): PageObject class, AbstractBaseClass, or \n Interface to attempt to consturct.\n\n Kwargs:\n webdriver (WebDriver): Selenium Webdriver to use to instantiate the page. If none \n is provided, then it was use the default from \n WTF_WEBDRIVER_MANAGER\n\n Returns:\n PageObject\n\n Raises:\n NoMatchingPageError\n\n Instantiating a Page from PageObject from class usage::\n\n my_page_instance = PageFactory.create_page(MyPageClass)\n\n\n Instantiating a Page from an Interface or base class::\n\n import pages.mysite.* # Make sure you import classes first, or else PageFactory will not know about it.\n my_page_instance = PageFactory.create_page(MyPageInterfaceClass)\n\n\n Instantiating a Page from a list of classes.::\n\n my_page_instance = PageFactory.create_page([PossiblePage1, PossiblePage2])\n\n\n Note: It'll only be able to detect pages that are imported. To it's best to \n do an import of all pages implementing a base class or the interface inside the \n __init__.py of the package directory. \n\n \"\"\"\n if not webdriver:\n webdriver = WTF_WEBDRIVER_MANAGER.get_driver()\n\n # will be used later when tracking best matched page.\n current_matched_page = None\n\n # used to track if there is a valid page object within the set of PageObjects searched.\n was_validate_called = False\n\n # Walk through all classes if a list was passed.\n if type(page_object_class_or_interface) == list:\n subclasses = []\n for page_class in page_object_class_or_interface:\n # attempt to instantiate class.\n page = PageFactory.__instantiate_page_object(page_class,\n webdriver,\n **kwargs)\n\n if isinstance(page, PageObject):\n was_validate_called = True\n if (current_matched_page == None or page > current_matched_page):\n current_matched_page = page\n\n elif page is True:\n was_validate_called = True\n\n # check for subclasses\n subclasses += PageFactory.__itersubclasses(page_class)\n else:\n # A single class was passed in, try to instantiate the class.\n page_class = page_object_class_or_interface\n page = PageFactory.__instantiate_page_object(page_class,\n webdriver,\n **kwargs)\n # Check if we got a valid PageObject back.\n if isinstance(page, PageObject):\n was_validate_called = True\n current_matched_page = page\n elif page is True:\n was_validate_called = True\n\n # check for subclasses\n subclasses = PageFactory.__itersubclasses(\n page_object_class_or_interface)\n\n # Iterate over subclasses of the passed in classes to see if we have a\n # better match.\n for pageClass in subclasses:\n try:\n page = PageFactory.__instantiate_page_object(pageClass,\n webdriver,\n **kwargs)\n # If we get a valid PageObject match, check to see if the ranking is higher\n # than our current PageObject.\n if isinstance(page, PageObject):\n was_validate_called = True\n if current_matched_page == None or page > current_matched_page:\n current_matched_page = page\n elif page is True:\n was_validate_called = True\n\n except InvalidPageError as e:\n _wtflog.debug(\"InvalidPageError: %s\", e)\n pass # This happens when the page fails check.\n except TypeError as e:\n _wtflog.debug(\"TypeError: %s\", e)\n # this happens when it tries to instantiate the original\n # abstract class.\n pass\n except Exception as e:\n _wtflog.debug(\"Exception during page instantiation: %s\", e)\n # Unexpected exception.\n raise e\n\n # If no matching classes.\n if not isinstance(current_matched_page, PageObject):\n # Check that there is at least 1 valid page object that was passed in.\n if was_validate_called is False:\n raise TypeError(\"Neither the PageObjects nor it's subclasses have implemented \" + \n \"'PageObject._validate(self, webdriver)'.\")\n\n try:\n current_url = webdriver.current_url\n raise NoMatchingPageError(u(\"There's, no matching classes to this page. URL:{0}\")\n .format(current_url))\n except:\n raise NoMatchingPageError(u(\"There's, no matching classes to this page. \"))\n else:\n return current_matched_page", "response": "Instantiate a Page object from a given class or interface."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nattempting to instantiate a page object.", "response": "def __instantiate_page_object(page_obj_class, webdriver, **kwargs):\n \"\"\"\n Attempts to instantiate a page object.\n\n Args:\n page_obj_class (PageObject) - PageObject to instantiate.\n webdriver (WebDriver) - Selenium webdriver to associate with the PageObject\n \n Returns:\n PageObject - If page object instantiation succeeded.\n True - If page object instantiation failed, but validation was called.\n None - If validation did not occur.\n\n \"\"\"\n try:\n page = page_obj_class(webdriver, **kwargs)\n return page\n except InvalidPageError:\n # This happens when the page fails check.\n # Means validate was implemented, but the check didn't pass.\n return True\n except TypeError:\n # this happens when it tries to instantiate the original abstract\n # class, or a PageObject where _validate() was not implemented.\n return False\n except Exception as e:\n # Unexpected exception.\n raise e"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\niterate over all subclasses of a given class.", "response": "def __itersubclasses(cls, _seen=None):\n \"\"\"\n \n Credit goes to: http://code.activestate.com/recipes/576949-find-all-subclasses-of-a-given-class/\n\n itersubclasses(cls)\n\n Generator over all subclasses of a given class, in depth first order.\n\n >>> list(itersubclasses(int)) == [bool]\n True\n >>> class A(object): pass\n >>> class B(A): pass\n >>> class C(A): pass\n >>> class D(B,C): pass\n >>> class E(D): pass\n >>> \n >>> for cls in itersubclasses(A):\n ... print(cls.__name__)\n B\n D\n E\n C\n >>> # get ALL (new-style) classes currently defined\n >>> [cls.__name__ for cls in itersubclasses(object)] #doctest: +ELLIPSIS\n ['type', ...'tuple', ...]\n \"\"\"\n if not isinstance(cls, type):\n raise TypeError(u('Argument ({0}) passed to PageFactory does not appear to be a valid Class.').format(cls),\n \"Check to make sure the first parameter is an PageObject class, interface, or mixin.\")\n if _seen is None:\n _seen = set()\n try:\n subs = cls.__subclasses__()\n except TypeError: # fails only when cls is type\n subs = cls.__subclasses__(cls)\n for sub in subs:\n if sub not in _seen:\n _seen.add(sub)\n yield sub\n for sub in PageFactory.__itersubclasses(sub, _seen):\n yield sub"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if all CSS selectors passed in are found.", "response": "def check_css_selectors(webdriver, *selectors):\n \"\"\"Returns true if all CSS selectors passed in is found. This can be used \n to quickly validate a page.\n\n Args:\n webdriver (Webdriver) : Selenium Webdriver instance\n selectors (str) : N number of CSS selectors strings to match against the page.\n\n Returns:\n True, False - if the page matches all selectors.\n\n Usage Example::\n\n # Checks for a Form with id='loginForm' and a button with class 'login'\n if not PageObjectUtils.check_css_selectors(\"form#loginForm\", \"button.login\"):\n raise InvalidPageError(\"This is not the login page.\")\n\n You can use this within a PageObject's `_validate_page(webdriver)` method for \n validating pages.\n \"\"\"\n for selector in selectors:\n try:\n webdriver.find_element_by_css_selector(selector)\n except:\n return False # A selector failed.\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef wait_until_page_loaded(page_obj_class,\n webdriver=None,\n timeout=WTF_TIMEOUT_MANAGER.NORMAL,\n sleep=0.5,\n bad_page_classes=[],\n message=None,\n **kwargs):\n \"\"\"\n Waits until the page is loaded.\n\n Args:\n page_obj_class (Class) : PageObject class\n\n Kwargs:\n webdriver (Webdriver) : Selenium Webdriver. Default uses WTF_WEBDRIVER_MANAGER's instance.\n timeout (number) : Number of seconds to wait to allow the page to load.\n sleep (number) : Number of seconds to wait between polling.\n bad_page_classes (list) : List of PageObject classes to fail if matched. For example, ServerError page.\n message (string) : Use your own message with PageLoadTimeoutError raised.\n\n Returns:\n PageObject\n\n Raises:\n PageUtilOperationTimeoutError : Timeout occurred before the desired PageObject was matched.\n BadPageEncounteredError : One or more of the PageObject in the specified 'bad_page_classes' list \n was matched.\n\n\n Usage Example:: \n webdriver.get(\"http://www.mysite.com/login\")\n # Wait up to 60 seconds for the page to load.\n login_page = wait_until_page_loaded(LoginPage, timeout=60, [ServerErrorPage])\n\n This will wait for the login_page to load, then return a LoginPage() PageObject.\n\n \"\"\"\n if not webdriver:\n webdriver = WTF_WEBDRIVER_MANAGER.get_driver()\n\n # convert this param to list if not already.\n if type(bad_page_classes) != list:\n bad_page_classes = [bad_page_classes]\n\n end_time = datetime.now() + timedelta(seconds=timeout)\n last_exception = None\n while datetime.now() < end_time:\n # Check to see if we're at our target page.\n try:\n page = PageFactory.create_page(\n page_obj_class, webdriver=webdriver, **kwargs)\n return page\n except Exception as e:\n _wtflog.debug(\"Encountered exception: %s \", e)\n last_exception = e\n\n # Check to see if we're at one of those labled 'Bad' pages.\n for bad_page_class in bad_page_classes:\n try:\n PageFactory.create_page(\n bad_page_class, webdriver=webdriver, **kwargs)\n # if the if/else statement succeeds, than we have an error.\n raise BadPageEncounteredError(\n u(\"Encountered a bad page. \") + bad_page_class.__name__)\n except BadPageEncounteredError as e:\n raise e\n except:\n pass # We didn't hit a bad page class yet.\n # sleep till the next iteration.\n time.sleep(sleep)\n\n print \"Unable to construct page, last exception\", last_exception\n\n # Attempt to get current URL to assist in debugging\n try:\n current_url = webdriver.current_url\n except:\n # unable to get current URL, could be a webdriver for a non-webpage like mobile app.\n current_url = None\n\n if message:\n err_msg = u(message) + u(\"{page}:{url}\")\\\n .format(page=PageUtils.__get_name_for_class__(page_obj_class),\n url=current_url)\n else:\n err_msg = u(\"Timed out while waiting for {page} to load. Url:{url}\")\\\n .format(page=PageUtils.__get_name_for_class__(page_obj_class),\n url=current_url)\n raise PageLoadTimeoutError(err_msg)", "response": "Waits until a page is loaded."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwaiting until the page is ready.", "response": "def wait_until_page_ready(page_object, timeout=WTF_TIMEOUT_MANAGER.NORMAL):\n \"\"\"Waits until document.readyState == Complete (e.g. ready to execute javascript commands)\n\n Args:\n page_object (PageObject) : PageObject class\n\n Kwargs:\n timeout (number) : timeout period\n \"\"\"\n try:\n do_until(lambda: page_object.webdriver.execute_script(\"return document.readyState\").lower()\n == 'complete', timeout)\n except wait_utils.OperationTimeoutError:\n raise PageUtilOperationTimeoutError(\n \"Timeout occurred while waiting for page to be ready.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_asset_path(self, filename):\n if os.path.exists(os.path.join(self._asset_path, filename)):\n return os.path.join(self._asset_path, filename)\n else:\n raise AssetNotFoundError(\n u(\"Cannot find asset: {0}\").format(filename))", "response": "Get the full system path of a given asset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a Selenium WebDriver instance based on config settings.", "response": "def create_webdriver(self, testname=None):\n '''\n Creates an instance of Selenium webdriver based on config settings.\n This should only be called by a shutdown hook. Do not call directly within \n a test.\n\n Kwargs:\n testname: Optional test name to pass, this gets appended to the test name \n sent to selenium grid.\n\n Returns:\n WebDriver - Selenium Webdriver instance.\n\n '''\n try:\n driver_type = self._config_reader.get(\n self.DRIVER_TYPE_CONFIG)\n except:\n driver_type = self.DRIVER_TYPE_LOCAL\n _wtflog.warn(\"%s setting is missing from config. Using default setting, %s\",\n self.DRIVER_TYPE_CONFIG, driver_type)\n\n if driver_type == self.DRIVER_TYPE_REMOTE:\n # Create desired capabilities.\n self.webdriver = self.__create_remote_webdriver_from_config(\n testname=testname)\n else:\n # handle as local webdriver\n self.webdriver = self.__create_driver_from_browser_config()\n try:\n self.webdriver.maximize_window()\n except:\n # wait a short period and try again.\n time.sleep(self._timeout_mgr.BRIEF)\n try:\n self.webdriver.maximize_window()\n except Exception as e:\n if (isinstance(e, WebDriverException) and\n \"implemented\" in e.msg.lower()):\n pass # Maximizing window not supported by this webdriver.\n else:\n _wtflog.warn(\"Unable to maxmize browser window. \" + \n \"It may be possible the browser did not instantiate correctly. % s\",\n e)\n\n return self.webdriver"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __create_driver_from_browser_config(self):\n '''\n Reads the config value for browser type.\n '''\n try:\n browser_type = self._config_reader.get(\n WebDriverFactory.BROWSER_TYPE_CONFIG)\n except KeyError:\n _wtflog(\"%s missing is missing from config file. Using defaults\",\n WebDriverFactory.BROWSER_TYPE_CONFIG)\n browser_type = WebDriverFactory.FIREFOX\n\n # Special Chrome Sauce\n options = webdriver.ChromeOptions()\n options.add_experimental_option(\"excludeSwitches\", [\"ignore-certificate-errors\"])\n options.add_argument(\"always-authorize-plugins\")\n\n browser_type_dict = {\n self.CHROME: lambda: webdriver.Chrome(\n self._config_reader.get(WebDriverFactory.CHROME_DRIVER_PATH),\n chrome_options=options),\n self.FIREFOX: lambda: webdriver.Firefox(),\n self.INTERNETEXPLORER: lambda: webdriver.Ie(),\n self.OPERA: lambda: webdriver.Opera(),\n self.PHANTOMJS: lambda: self.__create_phantom_js_driver(),\n self.SAFARI: lambda: self.__create_safari_driver()\n }\n\n try:\n return browser_type_dict[browser_type]()\n except KeyError:\n raise TypeError(\n u(\"Unsupported Browser Type {0}\").format(browser_type))", "response": "Reads the config value for browser type."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __create_safari_driver(self):\n '''\n Creates an instance of Safari webdriver.\n '''\n # Check for selenium jar env file needed for safari driver.\n if not os.getenv(self.__SELENIUM_SERVER_JAR_ENV):\n # If not set, check if we have a config setting for it.\n try:\n selenium_server_path = self._config_reader.get(\n self.SELENIUM_SERVER_LOCATION)\n self._env_vars[\n self.__SELENIUM_SERVER_JAR_ENV] = selenium_server_path\n except KeyError:\n raise RuntimeError(u(\"Missing selenium server path config {0}.\").format(\n self.SELENIUM_SERVER_LOCATION))\n\n return webdriver.Safari()", "response": "Creates an instance of Safari webdriver."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate an instance of PhantomJS driver.", "response": "def __create_phantom_js_driver(self):\n '''\n Creates an instance of PhantomJS driver.\n '''\n try:\n return webdriver.PhantomJS(executable_path=self._config_reader.get(self.PHANTOMEJS_EXEC_PATH),\n service_args=['--ignore-ssl-errors=true'])\n except KeyError:\n return webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true'])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a remote webdriver instance from the config file.", "response": "def __create_remote_webdriver_from_config(self, testname=None):\n '''\n Reads the config value for browser type.\n '''\n desired_capabilities = self._generate_desired_capabilities(testname)\n \n remote_url = self._config_reader.get(\n WebDriverFactory.REMOTE_URL_CONFIG)\n\n # Instantiate remote webdriver.\n driver = webdriver.Remote(\n desired_capabilities=desired_capabilities,\n command_executor=remote_url\n )\n\n # Log IP Address of node if configured, so it can be used to\n # troubleshoot issues if they occur.\n log_driver_props = \\\n self._config_reader.get(\n WebDriverFactory.LOG_REMOTEDRIVER_PROPS, default_value=False\n ) in [True, \"true\", \"TRUE\", \"True\"]\n if \"wd/hub\" in remote_url and log_driver_props:\n try:\n grid_addr = remote_url[:remote_url.index(\"wd/hub\")]\n info_request_response = urllib2.urlopen(\n grid_addr + \"grid/api/testsession?session=\" + driver.session_id, \"\", 5000)\n node_info = info_request_response.read()\n _wtflog.info(\n u(\"RemoteWebdriver using node: \") + u(node_info).strip())\n except:\n # Unable to get IP Address of remote webdriver.\n # This happens with many 3rd party grid providers as they don't want you accessing info on nodes on\n # their internal network.\n pass\n\n return driver"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncleans up webdrivers created during execution.", "response": "def clean_up_webdrivers(self):\n '''\n Clean up webdrivers created during execution.\n '''\n # Quit webdrivers.\n _wtflog.info(\"WebdriverManager: Cleaning up webdrivers\")\n try:\n if self.__use_shutdown_hook:\n for key in self.__registered_drivers.keys():\n for driver in self.__registered_drivers[key]:\n try:\n _wtflog.debug(\n \"Shutdown hook closing Webdriver for thread: %s\", key)\n driver.quit()\n except:\n pass\n except:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclose current running instance of Webdriver.", "response": "def close_driver(self):\n \"\"\"\n Close current running instance of Webdriver.\n\n Usage::\n\n driver = WTF_WEBDRIVER_MANAGER.new_driver()\n driver.get(\"http://the-internet.herokuapp.com\")\n WTF_WEBDRIVER_MANAGER.close_driver()\n \"\"\"\n channel = self.__get_channel()\n driver = self.__get_driver_for_channel(channel)\n if self.__config.get(self.REUSE_BROWSER, True):\n # If reuse browser is set, we'll avoid closing it and just clear out the cookies,\n # and reset the location.\n try:\n driver.delete_all_cookies()\n # check to see if webdriver is still responding\n driver.get(\"about:blank\")\n except:\n pass\n\n if driver is not None:\n try:\n driver.quit()\n except:\n pass\n\n self.__unregister_driver(channel)\n if driver in self.__registered_drivers[channel]:\n self.__registered_drivers[channel].remove(driver)\n\n self.webdriver = None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_driver(self):\n '''\n Get an already running instance of Webdriver. If there is none, it will create one.\n\n Returns:\n Webdriver - Selenium Webdriver instance.\n\n Usage::\n\n driver = WTF_WEBDRIVER_MANAGER.new_driver()\n driver.get(\"http://the-internet.herokuapp.com\")\n same_driver = WTF_WEBDRIVER_MANAGER.get_driver()\n print(driver is same_driver) # True\n '''\n driver = self.__get_driver_for_channel(self.__get_channel())\n if driver is None:\n driver = self.new_driver()\n\n return driver", "response": "Get an already running instance of Webdriver."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef new_driver(self, testname=None):\n '''\n Used at a start of a test to get a new instance of WebDriver. If the \n 'resuebrowser' setting is true, it will use a recycled WebDriver instance\n with delete_all_cookies() called.\n\n Kwargs:\n testname (str) - Optional test name to pass to Selenium Grid. Helpful for \n labeling tests on 3rd party WebDriver cloud providers.\n\n Returns:\n Webdriver - Selenium Webdriver instance.\n\n Usage::\n\n driver = WTF_WEBDRIVER_MANAGER.new_driver()\n driver.get(\"http://the-internet.herokuapp.com\")\n '''\n channel = self.__get_channel()\n\n # Get reference for the current driver.\n driver = self.__get_driver_for_channel(channel)\n\n if self.__config.get(WebDriverManager.REUSE_BROWSER, True):\n\n if driver is None:\n driver = self._webdriver_factory.create_webdriver(\n testname=testname)\n\n # Register webdriver so it can be retrieved by the manager and\n # cleaned up after exit.\n self.__register_driver(channel, driver)\n else:\n try:\n # Attempt to get the browser to a pristine state as possible when we are\n # reusing this for another test.\n driver.delete_all_cookies()\n # check to see if webdriver is still responding\n driver.get(\"about:blank\")\n except:\n # In the case the browser is unhealthy, we should kill it\n # and serve a new one.\n try:\n if driver.is_online():\n driver.quit()\n except:\n pass\n\n driver = self._webdriver_factory.create_webdriver(\n testname=testname)\n self.__register_driver(channel, driver)\n\n else:\n # Attempt to tear down any existing webdriver.\n if driver is not None:\n try:\n driver.quit()\n except:\n pass\n self.__unregister_driver(channel)\n driver = self._webdriver_factory.create_webdriver(\n testname=testname)\n self.__register_driver(channel, driver)\n\n return driver", "response": "Create a new instance of Selenium Webdriver."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nregister webdriver to a channel.", "response": "def __register_driver(self, channel, webdriver):\n \"Register webdriver to a channel.\"\n\n # Add to list of webdrivers to cleanup.\n if not self.__registered_drivers.has_key(channel):\n self.__registered_drivers[channel] = [] # set to new empty array\n\n self.__registered_drivers[channel].append(webdriver)\n\n # Set singleton instance for the channel\n self.__webdriver[channel] = webdriver"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __get_channel(self):\n \"Get the channel to register webdriver to.\"\n if self.__config.get(WebDriverManager.ENABLE_THREADING_SUPPORT, False):\n channel = current_thread().ident\n else:\n channel = 0\n\n return channel", "response": "Get the channel to register webdriver to."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef take_screenshot(webdriver, file_name):\n folder_location = os.path.join(ProjectUtils.get_project_root(),\n WebScreenShotUtil.SCREEN_SHOT_LOCATION)\n\n WebScreenShotUtil.__capture_screenshot(\n webdriver, folder_location, file_name + \".png\")", "response": "Captures a screenshot.\n\n Args:\n webdriver (WebDriver) - Selenium webdriver.\n file_name (str) - File name to save screenshot as."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef take_reference_screenshot(webdriver, file_name):\n folder_location = os.path.join(ProjectUtils.get_project_root(),\n WebScreenShotUtil.REFERENCE_SCREEN_SHOT_LOCATION)\n\n WebScreenShotUtil.__capture_screenshot(\n webdriver, folder_location, file_name + \".png\")", "response": "Takes a screenshot as a reference screenshot."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_project_root(cls):\n '''\n Return path of the project directory. Use this method for getting paths relative to the project.\n However, for data, it's recommended you use WTF_DATA_MANAGER and for assets it's recommended \n to use WTF_ASSET_MANAGER, which are already singleton instances that manger the /data, and /assets \n folder for you.\n\n Returns:\n str - path of project root directory.\n '''\n if(cls.__root_folder__ != None):\n return cls.__root_folder__\n\n # Check for enviornment variable override.\n try:\n cls.__root_folder__ = os.environ[cls.WTF_HOME_CONFIG_ENV_VAR]\n except KeyError:\n # Means WTF_HOME override isn't specified.\n pass\n\n # Search starting from the current working directory and traverse up parent directories for the\n # hidden file denoting the project root folder.\n path = os.getcwd()\n seperator_matches = re.finditer(\"/|\\\\\\\\\", path)\n\n paths_to_search = [path]\n for match in seperator_matches:\n p = path[:match.start()]\n paths_to_search.insert(0, p)\n\n for path in paths_to_search:\n target_path = os.path.join(path, cls.__WTF_ROOT_FOLDER_FILE)\n if os.path.isfile(target_path):\n cls.__root_folder__ = path\n return cls.__root_folder__\n\n raise RuntimeError(u(\"Missing root project folder locator file '.wtf_root_folder'.\")\n + u(\"Check to make sure this file is located in your project directory.\"))", "response": "Get the project root directory."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef do_until(lambda_expr, timeout=WTF_TIMEOUT_MANAGER.NORMAL, sleep=0.5, message=None):\n '''\n A retry wrapper that'll keep performing the action until it succeeds.\n (main differnce between do_until and wait_until is do_until will keep trying \n until a value is returned, while wait until will wait until the function \n evaluates True.)\n\n Args:\n lambda_expr (lambda) : Expression to evaluate.\n\n Kwargs: \n timeout (number): Timeout period in seconds.\n sleep (number) : Sleep time to wait between iterations\n message (str) : Provide a message for TimeoutError raised.\n\n Returns:\n The value of the evaluated lambda expression.\n\n Usage::\n\n do_until(lambda: driver.find_element_by_id(\"save\").click(),\n timeout=30,\n sleep=0.5)\n\n Is equivalent to:\n\n end_time = datetime.now() + timedelta(seconds=30)\n while datetime.now() < end_time:\n try:\n return driver.find_element_by_id(\"save\").click()\n except:\n pass\n time.sleep(0.5)\n raise OperationTimeoutError()\n '''\n __check_condition_parameter_is_function(lambda_expr)\n\n end_time = datetime.now() + timedelta(seconds=timeout)\n last_exception = None\n while datetime.now() < end_time:\n try:\n return lambda_expr()\n except Exception as e:\n last_exception = e\n time.sleep(sleep)\n\n if message:\n raise OperationTimeoutError(message, last_exception)\n else:\n raise OperationTimeoutError(\"Operation timed out.\", last_exception)", "response": "A retry wrapper that will keep performing the action until the function evaluates True."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwaits until condition is met and returns the result.", "response": "def wait_and_ignore(condition, timeout=WTF_TIMEOUT_MANAGER.NORMAL, sleep=0.5):\n '''\n Waits wrapper that'll wait for the condition to become true, but will \n not error if the condition isn't met.\n\n Args:\n condition (lambda) - Lambda expression to wait for to evaluate to True.\n\n Kwargs:\n timeout (number) : Maximum number of seconds to wait.\n sleep (number) : Sleep time to wait between iterations.\n\n Example::\n\n wait_and_ignore(lambda: driver.find_element_by_id(\"success\").is_displayed(), \n timeout=30,\n sleep=0.5)\n\n is equivalent to::\n\n end_time = datetime.now() + timedelta(seconds=30)\n while datetime.now() < end_time:\n try:\n if driver.find_element_by_id(\"success\").is_displayed():\n break;\n except:\n pass\n time.sleep(0.5)\n '''\n try:\n return wait_until(condition, timeout, sleep)\n except:\n pass"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwaits until a condition is true.", "response": "def wait_until(condition, timeout=WTF_TIMEOUT_MANAGER.NORMAL, sleep=0.5, pass_exceptions=False, message=None):\n '''\n Waits wrapper that'll wait for the condition to become true.\n (main differnce between do_until and wait_until is do_until will keep trying \n until a value is returned, while wait until will wait until the function \n evaluates True.)\n\n Args:\n condition (lambda) - Lambda expression to wait for to evaluate to True.\n\n Kwargs:\n timeout (number) : Maximum number of seconds to wait.\n sleep (number) : Sleep time to wait between iterations.\n pass_exceptions (bool) : If set true, any exceptions raised will be re-raised up the chain.\n Normally exceptions are ignored.\n message (str) : Optional message to pass into OperationTimeoutError if the wait times out.\n\n Example::\n\n wait_until(lambda: driver.find_element_by_id(\"success\").is_displayed(), \n timeout=30,\n sleep=0.5)\n\n is equivalent to::\n\n end_time = datetime.now() + timedelta(seconds=30)\n did_succeed = False\n while datetime.now() < end_time:\n try:\n if driver.find_element_by_id(\"success\").is_displayed():\n did_succeed = True\n break;\n except:\n pass\n time.sleep(0.5)\n if not did_succeed:\n raise OperationTimeoutError()\n '''\n __check_condition_parameter_is_function(condition)\n\n last_exception = None\n end_time = datetime.now() + timedelta(seconds=timeout)\n while datetime.now() < end_time:\n try:\n if condition():\n return\n except Exception as e:\n if pass_exceptions:\n raise e\n else:\n last_exception = e\n time.sleep(sleep)\n\n if message:\n if last_exception:\n raise OperationTimeoutError(message, e)\n else:\n raise OperationTimeoutError(message)\n else:\n if last_exception:\n raise OperationTimeoutError(\"Operation timed out.\", e)\n else:\n raise OperationTimeoutError(\"Operation timed out.\")"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch for an email by Subject. Returns True or False.", "response": "def check_email_exists_by_subject(self, subject, match_recipient=None):\n \"\"\"\n Searches for Email by Subject. Returns True or False.\n\n Args:\n subject (str): Subject to search for.\n\n Kwargs:\n match_recipient (str) : Recipient to match exactly. (don't care if not specified)\n\n Returns: \n True - email found, False - email not found\n\n \"\"\"\n # Select inbox to fetch the latest mail on server.\n self._mail.select(\"inbox\")\n\n try:\n matches = self.__search_email_by_subject(subject, match_recipient)\n if len(matches) <= 0:\n return False\n else:\n return True\n except Exception as e:\n raise e"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsearch for Email by Subject. Returns a list of Integers representing the email s UIDs if matching subjects is found.", "response": "def find_emails_by_subject(self, subject, limit=50, match_recipient=None):\n \"\"\"\n Searches for Email by Subject. Returns email's imap message IDs \n as a list if matching subjects is found.\n\n Args:\n subject (str) - Subject to search for.\n\n Kwargs:\n limit (int) - Limit search to X number of matches, default 50\n match_recipient (str) - Recipient to exactly (don't care if not specified)\n\n Returns:\n list - List of Integers representing imap message UIDs.\n\n \"\"\"\n # Select inbox to fetch the latest mail on server.\n self._mail.select(\"inbox\")\n\n matching_uids = self.__search_email_by_subject(\n subject, match_recipient)\n\n return matching_uids"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfetches contents of an email.", "response": "def get_email_message(self, message_uid, message_type=\"text/plain\"):\n \"\"\"\n Fetch contents of email.\n\n Args:\n message_uid (int): IMAP Message UID number.\n\n Kwargs:\n message_type: Can be 'text' or 'html'\n\n \"\"\"\n self._mail.select(\"inbox\")\n result = self._mail.uid('fetch', message_uid, \"(RFC822)\")\n msg = email.message_from_string(result[1][0][1])\n\n try:\n # Try to handle as multipart message first.\n for part in msg.walk():\n if part.get_content_type() == message_type:\n return part.get_payload(decode=True)\n except:\n # handle as plain text email\n return msg.get_payload(decode=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the a set of emails matching each regular expression passed in against the (RFC822) content. Args: *args: list of regular expressions. Kwargs: limit (int) - Limit to how many of the most resent emails to search through. date (datetime) - If specified, it will filter avoid checking messages older than this date.", "response": "def raw_search(self, *args, **kwargs):\n \"\"\"\n Find the a set of emails matching each regular expression passed in against the (RFC822) content.\n\n Args:\n *args: list of regular expressions.\n\n Kwargs:\n limit (int) - Limit to how many of the most resent emails to search through.\n date (datetime) - If specified, it will filter avoid checking messages older \n than this date.\n\n \"\"\"\n limit = 50\n try:\n limit = kwargs['limit']\n except KeyError:\n pass\n\n # Get first X messages.\n self._mail.select(\"inbox\")\n\n # apply date filter.\n try:\n date = kwargs['date']\n date_str = date.strftime(\"%d-%b-%Y\")\n _, email_ids = self._mail.search(None, '(SINCE \"%s\")' % date_str)\n except KeyError:\n _, email_ids = self._mail.search(None, 'ALL')\n\n # Above call returns email IDs as an array containing 1 str\n email_ids = email_ids[0].split()\n\n matching_uids = []\n for _ in range(1, min(limit, len(email_ids))):\n email_id = email_ids.pop()\n rfc_body = self._mail.fetch(email_id, \"(RFC822)\")[1][0][1]\n\n match = True\n for expr in args:\n if re.search(expr, rfc_body) is None:\n match = False\n break\n\n if match:\n uid = re.search(\n \"UID\\\\D*(\\\\d+)\\\\D*\", self._mail.fetch(email_id, 'UID')[1][0]).group(1)\n matching_uids.append(uid)\n\n return matching_uids"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a list of message numbers", "response": "def __search_email_by_subject(self, subject, match_recipient):\n \"Get a list of message numbers\"\n if match_recipient is None:\n _, data = self._mail.uid('search',\n None,\n '(HEADER SUBJECT \"{subject}\")'\n .format(subject=subject))\n\n uid_list = data[0].split()\n return uid_list\n else:\n _, data = self._mail.uid('search',\n None,\n '(HEADER SUBJECT \"{subject}\" TO \"{recipient}\")'\n .format(subject=subject, recipient=match_recipient))\n\n filtered_list = []\n uid_list = data[0].split()\n for uid in uid_list:\n # Those hard coded indexes [1][0][1] is a hard reference to the message email message headers\n # that's burried in all those wrapper objects that's associated\n # with fetching a message.\n to_addr = re.search(\n \"[^-]To: (.*)\", self._mail.uid('fetch', uid, \"(RFC822)\")[1][0][1]).group(1).strip()\n\n if (to_addr == match_recipient or to_addr == \"<{0}>\".format(match_recipient)):\n # Add matching entry to the list.\n filtered_list.append(uid)\n\n return filtered_list"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the value from the yaml config file based on the key.", "response": "def get(self, key, default_value=__NoDefaultSpecified__):\n '''\n Gets the value from the yaml config based on the key.\n\n No type casting is performed, any type casting should be \n performed by the caller.\n\n Args:\n key (str) - Config setting key.\n\n Kwargs:\n default_value - Default value to return if config is not specified.\n\n Returns:\n Returns value stored in config file.\n\n '''\n # First attempt to get the var from OS enviornment.\n os_env_string = ConfigReader.ENV_PREFIX + key\n os_env_string = os_env_string.replace(\".\", \"_\")\n if type(os.getenv(os_env_string)) != NoneType:\n return os.getenv(os_env_string)\n\n # Otherwise search through config files.\n for data_map in self._dataMaps:\n try:\n if \".\" in key:\n # this is a multi levl string\n namespaces = key.split(\".\")\n temp_var = data_map\n for name in namespaces:\n temp_var = temp_var[name]\n return temp_var\n else:\n value = data_map[key]\n return value\n except (AttributeError, TypeError, KeyError):\n pass\n\n if default_value == self.__NoDefaultSpecified__:\n raise KeyError(u(\"Key '{0}' does not exist\").format(key))\n else:\n return default_value"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_page_object(page_name, url):\n \"Generate page object from URL\"\n\n # Attempt to extract partial URL for verification.\n url_with_path = u('^.*//[^/]+([^?]+)?|$')\n try:\n match = re.match(url_with_path, url)\n partial_url = match.group(1)\n print(\"Using partial URL for location verification. \", partial_url)\n except:\n # use full url since we couldn't extract a partial.\n partial_url = url\n print(\"Could not find usable partial url, using full url.\", url)\n\n # Attempt to map input objects.\n print(\"Processing page source...\")\n response = urllib2.urlopen(url)\n html = response.read()\n input_tags_expr = u('<\\s*input[^>]*>')\n input_tag_iter = re.finditer(input_tags_expr, html, re.IGNORECASE)\n\n objectmap = \"\"\n print(\"Creating object map for tags...\")\n for input_tag_match in input_tag_iter:\n if not \"hidden\" in input_tag_match.group(0):\n try:\n print(\"processing\", input_tag_match.group(0))\n obj_map_entry = _process_input_tag(input_tag_match.group(0))\n objectmap += u(\" \") + obj_map_entry + \"\\n\"\n except Exception as e:\n print(e)\n # we failed to process it, nothing more we can do.\n pass\n\n return _page_object_template_.contents.format(date=datetime.now(),\n url=url,\n pagename=page_name,\n partialurl=partial_url,\n objectmap=objectmap)", "response": "Generate page object from URL"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets data path. Args: filename (string) : Name of file inside of /data folder to retrieve. Kwargs: env_prefix (string) : Name of subfolder, ex: 'qa' will find files in /data/qa Returns: String - path to file. Usage:: open(WTF_DATA_MANAGER.get_data_path('testdata.csv') Note: WTF_DATA_MANAGER is a provided global instance of DataManager", "response": "def get_data_path(self, filename, env_prefix=None):\n \"\"\"\n Get data path.\n\n Args:\n filename (string) : Name of file inside of /data folder to retrieve.\n\n Kwargs:\n env_prefix (string) : Name of subfolder, ex: 'qa' will find files in /data/qa\n\n Returns:\n String - path to file.\n\n Usage::\n\n open(WTF_DATA_MANAGER.get_data_path('testdata.csv')\n\n Note: WTF_DATA_MANAGER is a provided global instance of DataManager\n\n \"\"\"\n if env_prefix == None:\n target_file = filename\n else:\n target_file = os.path.join(env_prefix, filename)\n\n if os.path.exists(os.path.join(self._data_path, target_file)):\n return os.path.join(self._data_path, target_file)\n else:\n raise DataNotFoundError(\n u(\"Cannot find data file: {0}\").format(target_file))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef next(self):\n try:\n entry = {}\n row = self._csv_reader.next()\n for i in range(0, len(row)):\n entry[self._headers[i]] = row[i]\n\n return entry\n except Exception as e:\n # close our file when we're done reading.\n self._file.close()\n raise e", "response": "Gets next entry as a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_element_by_selectors(webdriver, *selectors):\n # perform initial check to verify selectors are valid by statements.\n for selector in selectors:\n (by_method, value) = selector\n if not WebElementSelector.__is_valid_by_type(by_method):\n raise BadSelectorError(\n u(\"Selectors should be of type selenium.webdriver.common.by.By\"))\n if type(value) != str:\n raise BadSelectorError(\n u(\"Selectors should be of type selenium.webdriver.common.by.By\"))\n\n selectors_used = []\n for selector in selectors:\n (by_method, value) = selector\n selectors_used.append(\n u(\"{by}:{value}\").format(by=by_method, value=value))\n try:\n return webdriver.find_element(by=by_method, value=value)\n except:\n pass\n\n raise ElementNotSelectableException(\n u(\"Unable to find elements using:\") + u(\",\").join(selectors_used))", "response": "Utility method makes it easier to find an element using multiple selectors. This method will return a new element if the element is found."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef wait_until_element_not_visible(webdriver, locator_lambda_expression,\n timeout=WTF_TIMEOUT_MANAGER.NORMAL, sleep=0.5):\n \"\"\"\n Wait for a WebElement to disappear.\n\n Args:\n webdriver (Webdriver) - Selenium Webdriver\n locator (lambda) - Locator lambda expression.\n\n Kwargs:\n timeout (number) - timeout period\n sleep (number) - sleep period between intervals.\n\n \"\"\"\n # Wait for loading progress indicator to go away.\n try:\n stoptime = datetime.now() + timedelta(seconds=timeout)\n while datetime.now() < stoptime:\n element = WebDriverWait(webdriver, WTF_TIMEOUT_MANAGER.BRIEF).until(\n locator_lambda_expression)\n if element.is_displayed():\n time.sleep(sleep)\n else:\n break\n except TimeoutException:\n pass", "response": "Wait until an element is not visible."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_image_loaded(webdriver, webelement):\n '''\n Check if an image (in an image tag) is loaded.\n Note: This call will not work against background images. Only Images in tags.\n\n Args:\n webelement (WebElement) - WebDriver web element to validate.\n\n '''\n script = (u(\"return arguments[0].complete && type of arguments[0].naturalWidth != \\\"undefined\\\" \") + \n u(\"&& arguments[0].naturalWidth > 0\"))\n try:\n return webdriver.execute_script(script, webelement)\n except:\n return False", "response": "Check if an image is loaded."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a time - stamped string.", "response": "def generate_timestamped_string(subject=\"test\", number_of_random_chars=4):\n \"\"\"\n Generate time-stamped string. Format as follows...\n\n `2013-01-31_14:12:23_SubjectString_a3Zg`\n\n\n Kwargs:\n subject (str): String to use as subject.\n number_of_random_chars (int) : Number of random characters to append.\n\n\n This method is helpful for creating unique names with timestamps in them so \n when you have to troubleshoot an issue, the name is easier to find.::\n\n self.project_name = generate_timestamped_string(\"project\")\n new_project_page.create_project(project_name)\n\n \"\"\"\n random_str = generate_random_string(number_of_random_chars)\n timestamp = generate_timestamp()\n return u\"{timestamp}_{subject}_{random_str}\".format(timestamp=timestamp,\n subject=subject,\n random_str=random_str)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generate_random_string(number_of_random_chars=8, character_set=string.ascii_letters):\n return u('').join(random.choice(character_set)\n for _ in range(number_of_random_chars))", "response": "Generates a series of random characters."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting the date format pattern used by Weka to the date format pattern used by Python s datetime. strftime.", "response": "def convert_weka_to_py_date_pattern(p):\n \"\"\"\n Converts the date format pattern used by Weka to the date format pattern used by Python's datetime.strftime().\n \"\"\"\n # https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior\n # https://www.cs.waikato.ac.nz/ml/weka/arff.html\n p = p.replace('yyyy', r'%Y')\n p = p.replace('MM', r'%m')\n p = p.replace('dd', r'%d')\n p = p.replace('HH', r'%H')\n p = p.replace('mm', r'%M')\n p = p.replace('ss', r'%S')\n return p"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_attribute_value(self, name, index):\n if index == MISSING:\n return\n elif self.attribute_types[name] in NUMERIC_TYPES:\n at = self.attribute_types[name]\n if at == TYPE_INTEGER:\n return int(index)\n return Decimal(str(index))\n else:\n assert self.attribute_types[name] == TYPE_NOMINAL\n cls_index, cls_value = index.split(':')\n #return self.attribute_data[name][index-1]\n if cls_value != MISSING:\n assert cls_value in self.attribute_data[name], \\\n 'Predicted value \"%s\" but only values %s are allowed.' \\\n % (cls_value, ', '.join(self.attribute_data[name]))\n return cls_value", "response": "Returns the value associated with the given value index."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading an ARFF File from a file.", "response": "def load(cls, filename, schema_only=False):\n \"\"\"\n Load an ARFF File from a file.\n \"\"\"\n o = open(filename)\n s = o.read()\n a = cls.parse(s, schema_only=schema_only)\n if not schema_only:\n a._filename = filename\n o.close()\n return a"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nparses an ARFF File already loaded into a string.", "response": "def parse(cls, s, schema_only=False):\n \"\"\"\n Parse an ARFF File already loaded into a string.\n \"\"\"\n a = cls()\n a.state = 'comment'\n a.lineno = 1\n for l in s.splitlines():\n a.parseline(l)\n a.lineno += 1\n if schema_only and a.state == 'data':\n # Don't parse data if we're only loading the schema.\n break\n return a"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a deep copy of the instance.", "response": "def copy(self, schema_only=False):\n \"\"\"\n Creates a deepcopy of the instance.\n If schema_only is True, the data will be excluded from the copy.\n \"\"\"\n o = type(self)()\n o.relation = self.relation\n o.attributes = list(self.attributes)\n o.attribute_types = self.attribute_types.copy()\n o.attribute_data = self.attribute_data.copy()\n if not schema_only:\n o.comment = list(self.comment)\n o.data = copy.deepcopy(self.data)\n return o"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nopen the arff structure to a file.", "response": "def open_stream(self, class_attr_name=None, fn=None):\n \"\"\"\n Save an arff structure to a file, leaving the file object\n open for writing of new data samples.\n This prevents you from directly accessing the data via Python,\n but when generating a huge file, this prevents all your data\n from being stored in memory.\n \"\"\"\n if fn:\n self.fout_fn = fn\n else:\n fd, self.fout_fn = tempfile.mkstemp()\n os.close(fd)\n self.fout = open(self.fout_fn, 'w')\n if class_attr_name:\n self.class_attr_name = class_attr_name\n self.write(fout=self.fout, schema_only=True)\n self.write(fout=self.fout, data_only=True)\n self.fout.flush()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nterminates an open stream and returns the filename of the file containing the streamed data.", "response": "def close_stream(self):\n \"\"\"\n Terminates an open stream and returns the filename\n of the file containing the streamed data.\n \"\"\"\n if self.fout:\n fout = self.fout\n fout_fn = self.fout_fn\n self.fout.flush()\n self.fout.close()\n self.fout = None\n self.fout_fn = None\n return fout_fn"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving an arff structure to a file.", "response": "def save(self, filename=None):\n \"\"\"\n Save an arff structure to a file.\n \"\"\"\n filename = filename or self._filename\n o = open(filename, 'w')\n o.write(self.write())\n o.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a single data line into a string.", "response": "def write_line(self, d, fmt=SPARSE):\n \"\"\"\n Converts a single data line to a string.\n \"\"\"\n \n def smart_quote(s):\n if isinstance(s, basestring) and ' ' in s and s[0] != '\"':\n s = '\"%s\"' % s\n return s\n \n if fmt == DENSE:\n #TODO:fix\n assert not isinstance(d, dict), NotImplemented\n line = []\n for e, a in zip(d, self.attributes):\n at = self.attribute_types[a]\n if at in NUMERIC_TYPES:\n line.append(str(e))\n elif at == TYPE_STRING:\n line.append(self.esc(e))\n elif at == TYPE_NOMINAL:\n line.append(e)\n else:\n raise Exception(\"Type \" + at + \" not supported for writing!\")\n s = ','.join(map(str, line))\n return s\n elif fmt == SPARSE:\n line = []\n \n # Convert flat row into dictionary.\n if isinstance(d, (list, tuple)):\n d = dict(zip(self.attributes, d))\n for k in d:\n at = self.attribute_types.get(k)\n if isinstance(d[k], Value):\n continue\n elif d[k] == MISSING:\n d[k] = Str(d[k])\n elif at in (TYPE_NUMERIC, TYPE_REAL):\n d[k] = Num(d[k])\n elif at == TYPE_STRING:\n d[k] = Str(d[k])\n elif at == TYPE_INTEGER:\n d[k] = Int(d[k])\n elif at == TYPE_NOMINAL:\n d[k] = Nom(d[k])\n elif at == TYPE_DATE:\n d[k] = Date(d[k])\n else:\n raise Exception('Unknown type: %s' % at)\n\n for i, name in enumerate(self.attributes):\n v = d.get(name)\n if v is None:\n# print 'Skipping attribute with None value:', name\n continue\n elif v == MISSING or (isinstance(v, Value) and v.value == MISSING):\n v = MISSING\n elif isinstance(v, String):\n v = '\"%s\"' % v.value\n elif isinstance(v, Date):\n date_format = self.attribute_data.get(name, DEFAULT_DATE_FORMAT)\n date_format = convert_weka_to_py_date_pattern(date_format)\n if isinstance(v.value, basestring):\n _value = dateutil.parser.parse(v.value)\n else:\n assert isinstance(v.value, (date, datetime))\n _value = v.value\n v.value = v = _value.strftime(date_format)\n elif isinstance(v, Value):\n v = v.value\n\n if v != MISSING and self.attribute_types[name] == TYPE_NOMINAL and str(v) not in map(str, self.attribute_data[name]):\n pass\n else:\n line.append('%i %s' % (i, smart_quote(v)))\n\n if len(line) == 1 and MISSING in line[-1]:\n # Skip lines with nothing other than a missing class.\n return\n elif not line:\n # Don't write blank lines.\n return\n return '{' + (', '.join(line)) + '}'\n else:\n raise Exception('Uknown format: %s' % (fmt,))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write(self,\n fout=None,\n fmt=SPARSE,\n schema_only=False,\n data_only=False):\n \"\"\"\n Write an arff structure to a string.\n \"\"\"\n assert not (schema_only and data_only), 'Make up your mind.'\n assert fmt in FORMATS, 'Invalid format \"%s\". Should be one of: %s' % (fmt, ', '.join(FORMATS))\n close = False\n if fout is None:\n close = True\n fout = StringIO()\n if not data_only:\n print('% ' + re.sub(\"\\n\", \"\\n% \", '\\n'.join(self.comment)), file=fout)\n print(\"@relation \" + self.relation, file=fout)\n self.write_attributes(fout=fout)\n if not schema_only:\n print(\"@data\", file=fout)\n for d in self.data:\n line_str = self.write_line(d, fmt=fmt)\n if line_str:\n print(line_str, file=fout)\n if isinstance(fout, StringIO) and close:\n return fout.getvalue()", "response": "Write an arff structure to a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndefine a new attribute. atype has to be one of integer real numeric string date nominal or date. data can be a dictionary of data type and value.", "response": "def define_attribute(self, name, atype, data=None):\n \"\"\"\n Define a new attribute. atype has to be one of 'integer', 'real', 'numeric', 'string', 'date' or 'nominal'.\n For nominal attributes, pass the possible values as data.\n For date attributes, pass the format as data.\n \"\"\"\n self.attributes.append(name)\n assert atype in TYPES, \"Unknown type '%s'. Must be one of: %s\" % (atype, ', '.join(TYPES),)\n self.attribute_types[name] = atype\n self.attribute_data[name] = data"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprint an overview of the ARFF file.", "response": "def dump(self):\n \"\"\"Print an overview of the ARFF file.\"\"\"\n print(\"Relation \" + self.relation)\n print(\" With attributes\")\n for n in self.attributes:\n if self.attribute_types[n] != TYPE_NOMINAL:\n print(\" %s of type %s\" % (n, self.attribute_types[n]))\n else:\n print(\" \" + n + \" of type nominal with values \" + ', '.join(self.attribute_data[n]))\n for d in self.data:\n print(d)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsorts the attributes names alphabetically except for the class attribute which is kept last.", "response": "def alphabetize_attributes(self):\n \"\"\"\n Orders attributes names alphabetically, except for the class attribute, which is kept last.\n \"\"\"\n self.attributes.sort(key=lambda name: (name == self.class_attr_name, name))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rgb_to_hsl(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n\n minVal = min(r, g, b) # min RGB value\n maxVal = max(r, g, b) # max RGB value\n\n l = (maxVal + minVal) / 2.0\n if minVal==maxVal:\n return (0.0, 0.0, l) # achromatic (gray)\n\n d = maxVal - minVal # delta RGB value\n\n if l < 0.5: s = d / (maxVal + minVal)\n else: s = d / (2.0 - maxVal - minVal)\n\n dr, dg, db = [(maxVal-val) / d for val in (r, g, b)]\n\n if r==maxVal:\n h = db - dg\n elif g==maxVal:\n h = 2.0 + dr - db\n else:\n h = 4.0 + dg - dr\n\n h = (h*60.0) % 360.0\n return (h, s, l)", "response": "Convert the color from RGB coordinates to HSL."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef hsl_to_rgb(h, s=None, l=None):\n if type(h) in [list,tuple]:\n h, s, l = h\n\n if s==0: return (l, l, l) # achromatic (gray)\n\n if l<0.5: n2 = l * (1.0 + s)\n else: n2 = l+s - (l*s)\n\n n1 = (2.0 * l) - n2\n\n h /= 60.0\n hueToRgb = _hue_to_rgb\n r = hueToRgb(n1, n2, h + 2)\n g = hueToRgb(n1, n2, h)\n b = hueToRgb(n1, n2, h - 2)\n\n return (r, g, b)", "response": "Convert the color from HSL coordinates to RGB."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rgb_to_hsv(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n\n v = float(max(r, g, b))\n d = v - min(r, g, b)\n if d==0: return (0.0, 0.0, v)\n s = d / v\n\n dr, dg, db = [(v - val) / d for val in (r, g, b)]\n\n if r==v:\n h = db - dg # between yellow & magenta\n elif g==v:\n h = 2.0 + dr - db # between cyan & yellow\n else: # b==v\n h = 4.0 + dg - dr # between magenta & cyan\n\n h = (h*60.0) % 360.0\n return (h, s, v)", "response": "Convert the color from RGB coordinates to HSV."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef hsv_to_rgb(h, s=None, v=None):\n if type(h) in [list,tuple]:\n h, s, v = h\n\n if s==0: return (v, v, v) # achromatic (gray)\n\n h /= 60.0\n h = h % 6.0\n\n i = int(h)\n f = h - i\n if not(i&1): f = 1-f # if i is even\n\n m = v * (1.0 - s)\n n = v * (1.0 - (s * f))\n\n if i==0: return (v, n, m)\n if i==1: return (n, v, m)\n if i==2: return (m, v, n)\n if i==3: return (m, n, v)\n if i==4: return (n, m, v)\n return (v, m, n)", "response": "Convert the color from RGB coordinates to HSV."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rgb_to_yiq(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n\n y = (r * 0.29895808) + (g * 0.58660979) + (b *0.11443213)\n i = (r * 0.59590296) - (g * 0.27405705) - (b *0.32184591)\n q = (r * 0.21133576) - (g * 0.52263517) + (b *0.31129940)\n return (y, i, q)", "response": "Convert the color from RGB to YIQ."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the color from YIQ coordinates to RGB.", "response": "def yiq_to_rgb(y, i=None, q=None):\n \"\"\"Convert the color from YIQ coordinates to RGB.\n\n Parameters:\n :y:\n Tte Y component value [0...1]\n :i:\n The I component value [0...1]\n :q:\n The Q component value [0...1]\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n >>> '({}, {}, {})'.format(*[round(v, 6) for v in yiq_to_rgb(0.592263, 0.458874, -0.0499818)])\n '(1.0, 0.5, 1e-06)'\n\n \"\"\"\n if type(y) in [list,tuple]:\n y, i, q = y\n r = y + (i * 0.9562) + (q * 0.6210)\n g = y - (i * 0.2717) - (q * 0.6485)\n b = y - (i * 1.1053) + (q * 1.7020)\n return (r, g, b)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rgb_to_yuv(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n\n y = (r * 0.29900) + (g * 0.58700) + (b * 0.11400)\n u = -(r * 0.14713) - (g * 0.28886) + (b * 0.43600)\n v = (r * 0.61500) - (g * 0.51499) - (b * 0.10001)\n return (y, u, v)", "response": "Convert the color from RGB coordinates to YUV."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert the color from YUV coordinates to RGB.", "response": "def yuv_to_rgb(y, u=None, v=None):\n \"\"\"Convert the color from YUV coordinates to RGB.\n\n Parameters:\n :y:\n The Y component value [0...1]\n :u:\n The U component value [-0.436...0.436]\n :v:\n The V component value [-0.615...0.615]\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n >>> '(%g, %g, %g)' % yuv_to_rgb(0.5925, -0.2916, 0.3575)\n '(0.999989, 0.500015, -6.3276e-05)'\n\n \"\"\"\n if type(y) in [list,tuple]:\n y, u, v = y\n r = y + (v * 1.13983)\n g = y - (u * 0.39465) - (v * 0.58060)\n b = y + (u * 2.03211)\n return (r, g, b)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rgb_to_xyz(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n\n r, g, b = [((v <= 0.03928) and [v / 12.92] or [((v+0.055) / 1.055) **2.4])[0] for v in (r, g, b)]\n\n x = (r * 0.4124) + (g * 0.3576) + (b * 0.1805)\n y = (r * 0.2126) + (g * 0.7152) + (b * 0.0722)\n z = (r * 0.0193) + (g * 0.1192) + (b * 0.9505)\n return (x, y, z)", "response": "Convert the color from sRGB to XYZ."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef xyz_to_rgb(x, y=None, z=None):\n if type(x) in [list,tuple]:\n x, y, z = x\n r = (x * 3.2406255) - (y * 1.5372080) - (z * 0.4986286)\n g = -(x * 0.9689307) + (y * 1.8757561) + (z * 0.0415175)\n b = (x * 0.0557101) - (y * 0.2040211) + (z * 1.0569959)\n return tuple((((v <= _srgbGammaCorrInv) and [v * 12.92] or [(1.055 * (v ** (1/2.4))) - 0.055])[0] for v in (r, g, b)))", "response": "Converts the color from CIE XYZ coordinates to sRGB."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts the color from CIE XYZ to lab.", "response": "def xyz_to_lab(x, y=None, z=None, wref=_DEFAULT_WREF):\n \"\"\"Convert the color from CIE XYZ to CIE L*a*b*.\n\n Parameters:\n :x:\n The X component value [0...1]\n :y:\n The Y component value [0...1]\n :z:\n The Z component value [0...1]\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n The color as an (L, a, b) tuple in the range:\n L[0...100],\n a[-1...1],\n b[-1...1]\n\n >>> '(%g, %g, %g)' % xyz_to_lab(0.488941, 0.365682, 0.0448137)\n '(66.9518, 0.430841, 0.739692)'\n\n >>> '(%g, %g, %g)' % xyz_to_lab(0.488941, 0.365682, 0.0448137, WHITE_REFERENCE['std_D50'])\n '(66.9518, 0.41166, 0.67282)'\n\n \"\"\"\n if type(x) in [list,tuple]:\n x, y, z = x\n # White point correction\n x /= wref[0]\n y /= wref[1]\n z /= wref[2]\n\n # Nonlinear distortion and linear transformation\n x, y, z = [((v > 0.008856) and [v**_oneThird] or [(7.787 * v) + _sixteenHundredsixteenth])[0] for v in (x, y, z)]\n\n # Vector scaling\n l = (116 * y) - 16\n a = 5.0 * (x - y)\n b = 2.0 * (y - z)\n\n return (l, a, b)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef lab_to_xyz(l, a=None, b=None, wref=_DEFAULT_WREF):\n if type(l) in [list,tuple]:\n l, a, b = l\n y = (l + 16) / 116\n x = (a / 5.0) + y\n z = y - (b / 2.0)\n return tuple((((v > 0.206893) and [v**3] or [(v - _sixteenHundredsixteenth) / 7.787])[0] * w for v, w in zip((x, y, z), wref)))", "response": "Convert the color from CIE L a * b * to CIE 1931 XYZ."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cmyk_to_cmy(c, m=None, y=None, k=None):\n if type(c) in [list,tuple]:\n c, m, y, k = c\n mk = 1-k\n return ((c*mk + k), (m*mk + k), (y*mk + k))", "response": "Convert the color from CMYK coordinates to CMY."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the color from CMY coordinates to CMYK.", "response": "def cmy_to_cmyk(c, m=None, y=None):\n \"\"\"Convert the color from CMY coordinates to CMYK.\n\n Parameters:\n :c:\n The Cyan component value [0...1]\n :m:\n The Magenta component value [0...1]\n :y:\n The Yellow component value [0...1]\n\n Returns:\n The color as an (c, m, y, k) tuple in the range:\n c[0...1],\n m[0...1],\n y[0...1],\n k[0...1]\n\n >>> '(%g, %g, %g, %g)' % cmy_to_cmyk(1, 0.66, 0.5)\n '(1, 0.32, 0, 0.5)'\n\n \"\"\"\n if type(c) in [list,tuple]:\n c, m, y = c\n k = min(c, m, y)\n if k==1.0: return (0.0, 0.0, 0.0, 1.0)\n mk = 1.0-k\n return ((c-k) / mk, (m-k) / mk, (y-k) / mk, k)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert the color from RGB coordinates to CMY.", "response": "def rgb_to_cmy(r, g=None, b=None):\n \"\"\"Convert the color from RGB coordinates to CMY.\n\n Parameters:\n :r:\n The Red component value [0...1]\n :g:\n The Green component value [0...1]\n :b:\n The Blue component value [0...1]\n\n Returns:\n The color as an (c, m, y) tuple in the range:\n c[0...1],\n m[0...1],\n y[0...1]\n\n >>> rgb_to_cmy(1, 0.5, 0)\n (0, 0.5, 1)\n\n \"\"\"\n if type(r) in [list,tuple]:\n r, g, b = r\n return (1-r, 1-g, 1-b)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cmy_to_rgb(c, m=None, y=None):\n if type(c) in [list,tuple]:\n c, m, y = c\n return (1-c, 1-m, 1-y)", "response": "Convert the color from CMY coordinates to RGB."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rgb_to_ints(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n return tuple(int(round(v*255)) for v in (r, g, b))", "response": "Convert the color in the standard [ 0... 255 ] range to ints in the [ 0... 255 ] range."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert ints in the 0... 255 range to the standard [ 0... 1 ] range.", "response": "def ints_to_rgb(r, g=None, b=None):\n \"\"\"Convert ints in the [0...255] range to the standard [0...1] range.\n\n Parameters:\n :r:\n The Red component value [0...255]\n :g:\n The Green component value [0...255]\n :b:\n The Blue component value [0...255]\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n >>> '(%g, %g, %g)' % ints_to_rgb((255, 128, 0))\n '(1, 0.501961, 0)'\n\n \"\"\"\n if type(r) in [list,tuple]:\n r, g, b = r\n return tuple(float(v) / 255.0 for v in [r, g, b])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rgb_to_html(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n return '#%02x%02x%02x' % tuple((min(round(v*255), 255) for v in (r, g, b)))", "response": "Convert the color from rgb to html."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert the HTML color to RGB.", "response": "def html_to_rgb(html):\n \"\"\"Convert the HTML color to (r, g, b).\n\n Parameters:\n :html:\n the HTML definition of the color (#RRGGBB or #RGB or a color name).\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n Throws:\n :ValueError:\n If html is neither a known color name or a hexadecimal RGB\n representation.\n\n >>> '(%g, %g, %g)' % html_to_rgb('#ff8000')\n '(1, 0.501961, 0)'\n >>> '(%g, %g, %g)' % html_to_rgb('ff8000')\n '(1, 0.501961, 0)'\n >>> '(%g, %g, %g)' % html_to_rgb('#f60')\n '(1, 0.4, 0)'\n >>> '(%g, %g, %g)' % html_to_rgb('f60')\n '(1, 0.4, 0)'\n >>> '(%g, %g, %g)' % html_to_rgb('lemonchiffon')\n '(1, 0.980392, 0.803922)'\n\n \"\"\"\n html = html.strip().lower()\n if html[0]=='#':\n html = html[1:]\n elif html in NAMED_COLOR:\n html = NAMED_COLOR[html][1:]\n\n if len(html)==6:\n rgb = html[:2], html[2:4], html[4:]\n elif len(html)==3:\n rgb = ['%c%c' % (v,v) for v in html]\n else:\n raise ValueError(\"input #%s is not in #RRGGBB format\" % html)\n\n return tuple(((int(n, 16) / 255.0) for n in rgb))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rgb_to_pil(r, g=None, b=None):\n if type(r) in [list,tuple]:\n r, g, b = r\n r, g, b = [min(int(round(v*255)), 255) for v in (r, g, b)]\n return (b << 16) + (g << 8) + r", "response": "Convert the color from RGB to PIL - compatible integer."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting the color from a PIL - compatible integer to RGB.", "response": "def pil_to_rgb(pil):\n \"\"\"Convert the color from a PIL-compatible integer to RGB.\n\n Parameters:\n pil: a PIL compatible color representation (0xBBGGRR)\n Returns:\n The color as an (r, g, b) tuple in the range:\n the range:\n r: [0...1]\n g: [0...1]\n b: [0...1]\n\n >>> '(%g, %g, %g)' % pil_to_rgb(0x0080ff)\n '(1, 0.501961, 0)'\n\n \"\"\"\n r = 0xff & pil\n g = 0xff & (pil >> 8)\n b = 0xff & (pil >> 16)\n return tuple((v / 255.0 for v in (r, g, b)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _websafe_component(c, alt=False):\n # This sucks, but floating point between 0 and 1 is quite fuzzy...\n # So we just change the scale a while to make the equality tests\n # work, otherwise it gets wrong at some decimal far to the right.\n sc = c * 100.0\n\n # If the color is already safe, return it straight away\n d = sc % 20\n if d==0: return c\n\n # Get the lower and upper safe values\n l = sc - d\n u = l + 20\n\n # Return the 'closest' value according to the alt flag\n if alt:\n if (sc-l) >= (u-sc): return l/100.0\n else: return u/100.0\n else:\n if (sc-l) >= (u-sc): return u/100.0\n else: return l/100.0", "response": "Convert a color component to its web safe equivalent."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting the color from RGB to web safe RGB", "response": "def rgb_to_websafe(r, g=None, b=None, alt=False):\n \"\"\"Convert the color from RGB to 'web safe' RGB\n\n Parameters:\n :r:\n The Red component value [0...1]\n :g:\n The Green component value [0...1]\n :b:\n The Blue component value [0...1]\n :alt:\n If True, use the alternative color instead of the nearest one.\n Can be used for dithering.\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n >>> '(%g, %g, %g)' % rgb_to_websafe(1, 0.55, 0.0)\n '(1, 0.6, 0)'\n\n \"\"\"\n if type(r) in [list,tuple]:\n r, g, b = r\n websafeComponent = _websafe_component\n return tuple((websafeComponent(v, alt) for v in (r, g, b)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert the color from RGB to its greyscale equivalent.", "response": "def rgb_to_greyscale(r, g=None, b=None):\n \"\"\"Convert the color from RGB to its greyscale equivalent\n\n Parameters:\n :r:\n The Red component value [0...1]\n :g:\n The Green component value [0...1]\n :b:\n The Blue component value [0...1]\n\n Returns:\n The color as an (r, g, b) tuple in the range:\n the range:\n r[0...1],\n g[0...1],\n b[0...1]\n\n >>> '(%g, %g, %g)' % rgb_to_greyscale(1, 0.8, 0)\n '(0.6, 0.6, 0.6)'\n\n \"\"\"\n if type(r) in [list,tuple]:\n r, g, b = r\n v = (r + g + b) / 3.0\n return (v, v, v)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmap a hue on the RGB color wheel to Itten s RYB wheel.", "response": "def rgb_to_ryb(hue):\n \"\"\"Maps a hue on the RGB color wheel to Itten's RYB wheel.\n\n Parameters:\n :hue:\n The hue on the RGB color wheel [0...360]\n\n Returns:\n An approximation of the corresponding hue on Itten's RYB wheel.\n\n >>> rgb_to_ryb(15)\n 26.0\n\n \"\"\"\n d = hue % 15\n i = int(hue / 15)\n x0 = _RybWheel[i]\n x1 = _RybWheel[i+1]\n return x0 + (x1-x0) * d / 15"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ryb_to_rgb(hue):\n d = hue % 15\n i = int(hue / 15)\n x0 = _RgbWheel[i]\n x1 = _RgbWheel[i+1]\n return x0 + (x1-x0) * d / 15", "response": "Maps a hue on Itten s RYB color wheel to the standard RGB wheel."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_rgb(r, g, b, alpha=1.0, wref=_DEFAULT_WREF):\n return Color((r, g, b), 'rgb', alpha, wref)", "response": "Create a new instance based on the specifed RGB values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_hsl(h, s, l, alpha=1.0, wref=_DEFAULT_WREF):\n return Color((h, s, l), 'hsl', alpha, wref)", "response": "Create a new instance based on the specifed HSL values."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new instance based on the specifed HSV values.", "response": "def from_hsv(h, s, v, alpha=1.0, wref=_DEFAULT_WREF):\n \"\"\"Create a new instance based on the specifed HSV values.\n\n Parameters:\n :h:\n The Hus component value [0...1]\n :s:\n The Saturation component value [0...1]\n :v:\n The Value component [0...1]\n :alpha:\n The color transparency [0...1], default is opaque\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_hsv(30, 1, 1)\n Color(1.0, 0.5, 0.0, 1.0)\n >>> Color.from_hsv(30, 1, 1, 0.5)\n Color(1.0, 0.5, 0.0, 0.5)\n\n \"\"\"\n h2, s, l = rgb_to_hsl(*hsv_to_rgb(h, s, v))\n return Color((h, s, l), 'hsl', alpha, wref)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a new instance based on the specifed YIQ values.", "response": "def from_yiq(y, i, q, alpha=1.0, wref=_DEFAULT_WREF):\n \"\"\"Create a new instance based on the specifed YIQ values.\n\n Parameters:\n :y:\n The Y component value [0...1]\n :i:\n The I component value [0...1]\n :q:\n The Q component value [0...1]\n :alpha:\n The color transparency [0...1], default is opaque\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_yiq(0.5922, 0.45885,-0.05)\n Color(0.999902, 0.499955, -6.7e-05, 1.0)\n >>> Color.from_yiq(0.5922, 0.45885,-0.05, 0.5)\n Color(0.999902, 0.499955, -6.7e-05, 0.5)\n\n \"\"\"\n return Color(yiq_to_rgb(y, i, q), 'rgb', alpha, wref)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_yuv(y, u, v, alpha=1.0, wref=_DEFAULT_WREF):\n return Color(yuv_to_rgb(y, u, v), 'rgb', alpha, wref)", "response": "Create a new instance based on the specifed YUV values."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_xyz(x, y, z, alpha=1.0, wref=_DEFAULT_WREF):\n return Color(xyz_to_rgb(x, y, z), 'rgb', alpha, wref)", "response": "Create a new Color instance based on the specifed CIE - XYZ values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new Color instance based on the specifed CIE - LAB values.", "response": "def from_lab(l, a, b, alpha=1.0, wref=_DEFAULT_WREF):\n \"\"\"Create a new instance based on the specifed CIE-LAB values.\n\n Parameters:\n :l:\n The L component [0...100]\n :a:\n The a component [-1...1]\n :b:\n The a component [-1...1]\n :alpha:\n The color transparency [0...1], default is opaque\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_lab(66.951823, 0.43084105, 0.73969231)\n Color(1.0, 0.5, -0.0, 1.0)\n >>> Color.from_lab(66.951823, 0.41165967, 0.67282012, wref=WHITE_REFERENCE['std_D50'])\n Color(1.0, 0.5, -0.0, 1.0)\n >>> Color.from_lab(66.951823, 0.43084105, 0.73969231, 0.5)\n Color(1.0, 0.5, -0.0, 0.5)\n >>> Color.from_lab(66.951823, 0.41165967, 0.67282012, 0.5, WHITE_REFERENCE['std_D50'])\n Color(1.0, 0.5, -0.0, 0.5)\n\n \"\"\"\n return Color(xyz_to_rgb(*lab_to_xyz(l, a, b, wref)), 'rgb', alpha, wref)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_cmy(c, m, y, alpha=1.0, wref=_DEFAULT_WREF):\n return Color(cmy_to_rgb(c, m, y), 'rgb', alpha, wref)", "response": "Create a new instance based on the specifed CMY values."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new instance based on the specifed CMYK values.", "response": "def from_cmyk(c, m, y, k, alpha=1.0, wref=_DEFAULT_WREF):\n \"\"\"Create a new instance based on the specifed CMYK values.\n\n Parameters:\n :c:\n The Cyan component value [0...1]\n :m:\n The Magenta component value [0...1]\n :y:\n The Yellow component value [0...1]\n :k:\n The Black component value [0...1]\n :alpha:\n The color transparency [0...1], default is opaque\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_cmyk(1, 0.32, 0, 0.5)\n Color(0.0, 0.34, 0.5, 1.0)\n >>> Color.from_cmyk(1, 0.32, 0, 0.5, 0.5)\n Color(0.0, 0.34, 0.5, 0.5)\n\n \"\"\"\n return Color(cmy_to_rgb(*cmyk_to_cmy(c, m, y, k)), 'rgb', alpha, wref)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_html(html, alpha=1.0, wref=_DEFAULT_WREF):\n return Color(html_to_rgb(html), 'rgb', alpha, wref)", "response": "Create a new instance based on the specifed HTML color definition."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new instance based on the specified PIL color representation.", "response": "def from_pil(pil, alpha=1.0, wref=_DEFAULT_WREF):\n \"\"\"Create a new instance based on the specifed PIL color.\n\n Parameters:\n :pil:\n A PIL compatible color representation (0xBBGGRR)\n :alpha:\n The color transparency [0...1], default is opaque\n :wref:\n The whitepoint reference, default is 2\u00b0 D65.\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_pil(0x0080ff)\n Color(1.0, 0.501961, 0.0, 1.0)\n >>> Color.from_pil(0x0080ff, 0.5)\n Color(1.0, 0.501961, 0.0, 0.5)\n\n \"\"\"\n return Color(pil_to_rgb(pil), 'rgb', alpha, wref)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef with_white_ref(self, wref, labAsRef=False):\n if labAsRef:\n l, a, b = self.lab\n return Color.from_lab(l, a, b, self.__a, wref)\n else:\n return Color(self.__rgb, 'rgb', self.__a, wref)", "response": "Create a new instance based on this one with a new white reference."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef desaturate(self, level):\n h, s, l = self.__hsl\n return Color((h, max(s - level, 0), l), 'hsl', self.__a, self.__wref)", "response": "Create a new instance based on this one but less saturated."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef websafe_dither(self):\n return (\n Color(rgb_to_websafe(*self.__rgb), 'rgb', self.__a, self.__wref),\n Color(rgb_to_websafe(alt=True, *self.__rgb), 'rgb', self.__a, self.__wref))", "response": "Return the two websafe colors nearest to this one."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new instance which is the complementary color of this one.", "response": "def complementary_color(self, mode='ryb'):\n \"\"\"Create a new instance which is the complementary color of this one.\n\n Parameters:\n :mode:\n Select which color wheel to use for the generation (ryb/rgb).\n\n\n Returns:\n A grapefruit.Color instance.\n\n >>> Color.from_hsl(30, 1, 0.5).complementary_color(mode='rgb')\n Color(0.0, 0.5, 1.0, 1.0)\n >>> Color.from_hsl(30, 1, 0.5).complementary_color(mode='rgb').hsl\n (210.0, 1.0, 0.5)\n\n \"\"\"\n h, s, l = self.__hsl\n\n if mode == 'ryb': h = rgb_to_ryb(h)\n h = (h+180)%360\n if mode == 'ryb': h = ryb_to_rgb(h)\n\n return Color((h, s, l), 'hsl', self.__a, self.__wref)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef make_analogous_scheme(self, angle=30, mode='ryb'):\n h, s, l = self.__hsl\n\n if mode == 'ryb': h = rgb_to_ryb(h)\n h += 360\n h1 = (h - angle) % 360\n h2 = (h + angle) % 360\n if mode == 'ryb':\n h1 = ryb_to_rgb(h1)\n h2 = ryb_to_rgb(h2)\n\n return (Color((h1, s, l), 'hsl', self.__a, self.__wref),\n Color((h2, s, l), 'hsl', self.__a, self.__wref))", "response": "Return two colors analogous to this one."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef blend(self, other, percent=0.5):\n dest = 1.0 - percent\n rgb = tuple(((u * percent) + (v * dest) for u, v in zip(self.__rgb, other.__rgb)))\n a = (self.__a * percent) + (other.__a * dest)\n return Color(rgb, 'rgb', a, self.__wref)", "response": "blends this color with the other one."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_parser():\n parser = ArgumentParser(description='a command-line web scraping tool')\n parser.add_argument('query', metavar='QUERY', type=str, nargs='*',\n help='URLs/files to scrape')\n parser.add_argument('-a', '--attributes', type=str, nargs='*',\n help='extract text using tag attributes')\n parser.add_argument('-all', '--crawl-all', help='crawl all pages',\n action='store_true')\n parser.add_argument('-c', '--crawl', type=str, nargs='*',\n help='regexp rules for following new pages')\n parser.add_argument('-C', '--clear-cache', help='clear requests cache',\n action='store_true')\n parser.add_argument('--csv', help='write files as csv',\n action='store_true')\n parser.add_argument('-cs', '--cache-size', type=int, nargs='?',\n help='size of page cache (default: 1000)',\n default=1000)\n parser.add_argument('-f', '--filter', type=str, nargs='*',\n help='regexp rules for filtering text')\n parser.add_argument('--html', help='write files as HTML',\n action='store_true')\n parser.add_argument('-i', '--images', action='store_true',\n help='save page images')\n parser.add_argument('-m', '--multiple', help='save to multiple files',\n action='store_true')\n parser.add_argument('-max', '--max-crawls', type=int,\n help='max number of pages to crawl')\n parser.add_argument('-n', '--nonstrict', action='store_true',\n help='allow crawler to visit any domain')\n parser.add_argument('-ni', '--no-images', action='store_true',\n help='do not save page images')\n parser.add_argument('-no', '--no-overwrite', action='store_true',\n help='do not overwrite files if they exist')\n parser.add_argument('-o', '--out', type=str, nargs='*',\n help='specify outfile names')\n parser.add_argument('-ow', '--overwrite', action='store_true',\n help='overwrite a file if it exists')\n parser.add_argument('-p', '--pdf', help='write files as pdf',\n action='store_true')\n parser.add_argument('-pt', '--print', help='print text output',\n action='store_true')\n parser.add_argument('-q', '--quiet', help='suppress program output',\n action='store_true')\n parser.add_argument('-s', '--single', help='save to a single file',\n action='store_true')\n parser.add_argument('-t', '--text', help='write files as text',\n action='store_true')\n parser.add_argument('-v', '--version', help='display current version',\n action='store_true')\n parser.add_argument('-x', '--xpath', type=str, nargs='?',\n help='filter HTML using XPath')\n return parser", "response": "Parse command - line arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrite scraped or local file(s ) in desired format.", "response": "def write_files(args, infilenames, outfilename):\n \"\"\"Write scraped or local file(s) in desired format.\n\n Keyword arguments:\n args -- program arguments (dict)\n infilenames -- names of user-inputted and/or downloaded files (list)\n outfilename -- name of output file (str)\n\n Remove PART(#).html files after conversion unless otherwise specified.\n \"\"\"\n write_actions = {'print': utils.print_text,\n 'pdf': utils.write_pdf_files,\n 'csv': utils.write_csv_files,\n 'text': utils.write_text_files}\n try:\n for action in iterkeys(write_actions):\n if args[action]:\n write_actions[action](args, infilenames, outfilename)\n finally:\n if args['urls'] and not args['html']:\n utils.remove_part_files()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite to a single output file and or a subdirectory.", "response": "def write_single_file(args, base_dir, crawler):\n \"\"\"Write to a single output file and/or subdirectory.\"\"\"\n if args['urls'] and args['html']:\n # Create a directory to save PART.html files in\n domain = utils.get_domain(args['urls'][0])\n if not args['quiet']:\n print('Storing html files in {0}/'.format(domain))\n utils.mkdir_and_cd(domain)\n\n infilenames = []\n for query in args['query']:\n if query in args['files']:\n infilenames.append(query)\n elif query.strip('/') in args['urls']:\n if args['crawl'] or args['crawl_all']:\n # Crawl and save HTML files/image files to disk\n infilenames += crawler.crawl_links(query)\n else:\n raw_resp = utils.get_raw_resp(query)\n if raw_resp is None:\n return False\n\n prev_part_num = utils.get_num_part_files()\n utils.write_part_file(args, query, raw_resp)\n curr_part_num = prev_part_num + 1\n infilenames += utils.get_part_filenames(curr_part_num, prev_part_num)\n\n # Convert output or leave as PART.html files\n if args['html']:\n # HTML files have been written already, so return to base directory\n os.chdir(base_dir)\n else:\n # Write files to text or pdf\n if infilenames:\n if args['out']:\n outfilename = args['out'][0]\n else:\n outfilename = utils.get_single_outfilename(args)\n if outfilename:\n write_files(args, infilenames, outfilename)\n else:\n utils.remove_part_files()\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting to multiple output files and or subdirectories.", "response": "def write_multiple_files(args, base_dir, crawler):\n \"\"\"Write to multiple output files and/or subdirectories.\"\"\"\n for i, query in enumerate(args['query']):\n if query in args['files']:\n # Write files\n if args['out'] and i < len(args['out']):\n outfilename = args['out'][i]\n else:\n outfilename = '.'.join(query.split('.')[:-1])\n write_files(args, [query], outfilename)\n elif query in args['urls']:\n # Scrape/crawl urls\n domain = utils.get_domain(query)\n if args['html']:\n # Create a directory to save PART.html files in\n if not args['quiet']:\n print('Storing html files in {0}/'.format(domain))\n utils.mkdir_and_cd(domain)\n\n if args['crawl'] or args['crawl_all']:\n # Crawl and save HTML files/image files to disk\n infilenames = crawler.crawl_links(query)\n else:\n raw_resp = utils.get_raw_resp(query)\n if raw_resp is None:\n return False\n\n # Saves page as PART.html file\n prev_part_num = utils.get_num_part_files()\n utils.write_part_file(args, query, raw_resp)\n curr_part_num = prev_part_num + 1\n infilenames = utils.get_part_filenames(curr_part_num, prev_part_num)\n\n # Convert output or leave as PART.html files\n if args['html']:\n # HTML files have been written already, so return to base dir\n os.chdir(base_dir)\n else:\n # Write files to text or pdf\n if infilenames:\n if args['out'] and i < len(args['out']):\n outfilename = args['out'][i]\n else:\n outfilename = utils.get_outfilename(query, domain)\n write_files(args, infilenames, outfilename)\n else:\n sys.stderr.write('Failed to retrieve content from {0}.\\n'\n .format(query))\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsplit query input into local files and URLs.", "response": "def split_input(args):\n \"\"\"Split query input into local files and URLs.\"\"\"\n args['files'] = []\n args['urls'] = []\n for arg in args['query']:\n if os.path.isfile(arg):\n args['files'].append(arg)\n else:\n args['urls'].append(arg.strip('/'))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef detect_output_type(args):\n if not args['single'] and not args['multiple']:\n # Save to multiple files if multiple files/URLs entered\n if len(args['query']) > 1 or len(args['out']) > 1:\n args['multiple'] = True\n else:\n args['single'] = True", "response": "Detect whether to save to a single or multiple files."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprompting user for filetype if none specified.", "response": "def prompt_filetype(args):\n \"\"\"Prompt user for filetype if none specified.\"\"\"\n valid_types = ('print', 'text', 'csv', 'pdf', 'html')\n if not any(args[x] for x in valid_types):\n try:\n filetype = input('Print or save output as ({0}): '\n .format(', '.join(valid_types))).lower()\n while filetype not in valid_types:\n filetype = input('Invalid entry. Choose from ({0}): '\n .format(', '.join(valid_types))).lower()\n except (KeyboardInterrupt, EOFError):\n return\n args[filetype] = True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef prompt_save_images(args):\n if args['images'] or args['no_images']:\n return\n\n if (args['pdf'] or args['html']) and (args['crawl'] or args['crawl_all']):\n save_msg = ('Choosing to save images will greatly slow the'\n ' crawling process.\\nSave images anyways? (y/n): ')\n try:\n save_images = utils.confirm_input(input(save_msg))\n except (KeyboardInterrupt, EOFError):\n return\n\n args['images'] = save_images\n args['no_images'] = not save_images", "response": "Prompt user to save images when crawling."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef command_line_runner():\n parser = get_parser()\n args = vars(parser.parse_args())\n if args['version']:\n print(__version__)\n return\n if args['clear_cache']:\n utils.clear_cache()\n print('Cleared {0}.'.format(utils.CACHE_DIR))\n return\n if not args['query']:\n parser.print_help()\n return\n\n # Enable cache unless user sets environ variable SCRAPE_DISABLE_CACHE\n if not os.getenv('SCRAPE_DISABLE_CACHE'):\n utils.enable_cache()\n\n # Save images unless user sets environ variable SCRAPE_DISABLE_IMGS\n if os.getenv('SCRAPE_DISABLE_IMGS'):\n args['no_images'] = True\n\n # Prompt user for filetype if none specified\n prompt_filetype(args)\n\n # Prompt user to save images when crawling (for pdf and HTML formats)\n prompt_save_images(args)\n\n # Scrape webpage content\n scrape(args)", "response": "Handle command - line interaction."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading a trained classifier from the raw Weka model format.", "response": "def load_raw(cls, model_fn, schema, *args, **kwargs):\n \"\"\"\n Loads a trained classifier from the raw Weka model format.\n Must specify the model schema and classifier name, since\n these aren't currently deduced from the model format.\n \"\"\"\n c = cls(*args, **kwargs)\n c.schema = schema.copy(schema_only=True)\n c._model_data = open(model_fn, 'rb').read()\n return c"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntrains the classifier with new training data and test data.", "response": "def train(self, training_data, testing_data=None, verbose=False):\n \"\"\"\n Updates the classifier with new data.\n \"\"\"\n model_fn = None\n training_fn = None\n clean_training = False\n testing_fn = None\n clean_testing = False\n try:\n \n # Validate training data.\n if isinstance(training_data, basestring):\n assert os.path.isfile(training_data)\n training_fn = training_data\n else:\n assert isinstance(training_data, arff.ArffFile)\n fd, training_fn = tempfile.mkstemp(suffix='.arff')\n os.close(fd)\n with open(training_fn, 'w') as fout:\n fout.write(training_data.write())\n clean_training = True\n assert training_fn\n \n # Validate testing data.\n if testing_data:\n if isinstance(testing_data, basestring):\n assert os.path.isfile(testing_data)\n testing_fn = testing_data\n else:\n assert isinstance(testing_data, arff.ArffFile)\n fd, testing_fn = tempfile.mkstemp(suffix='.arff')\n os.close(fd)\n with open(testing_fn, 'w') as fout:\n fout.write(testing_data.write())\n clean_testing = True\n else:\n testing_fn = training_fn\n assert testing_fn\n \n # Validate model file.\n fd, model_fn = tempfile.mkstemp()\n os.close(fd)\n if self._model_data:\n fout = open(model_fn, 'wb')\n fout.write(self._model_data)\n fout.close()\n \n # Call Weka Jar.\n args = dict(\n CP=CP,\n classifier_name=self.name,\n model_fn=model_fn,\n training_fn=training_fn,\n testing_fn=testing_fn,\n ckargs=self._get_ckargs_str(),\n )\n if self._model_data:\n # Load existing model.\n cmd = (\n \"java -cp %(CP)s %(classifier_name)s -l \\\"%(model_fn)s\\\" \"\n \"-t \\\"%(training_fn)s\\\" -T \\\"%(testing_fn)s\\\" -d \\\"%(model_fn)s\\\"\") % args\n else:\n # Create new model file.\n cmd = (\n \"java -cp %(CP)s %(classifier_name)s -t \\\"%(training_fn)s\\\" \"\n \"-T \\\"%(testing_fn)s\\\" -d \\\"%(model_fn)s\\\" %(ckargs)s\") % args\n if verbose:\n print(cmd)\n p = Popen(\n cmd,\n shell=True,\n stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=sys.platform != \"win32\")\n stdin, stdout, stderr = (p.stdin, p.stdout, p.stderr)\n stdout_str = stdout.read()\n stderr_str = stderr.read()\n \n self.last_training_stdout = stdout_str\n self.last_training_stderr = stderr_str\n \n if verbose:\n print('stdout:')\n print(stdout_str)\n print('stderr:')\n print(stderr_str)\n # exclude \"Warning\" lines not to raise an error for a simple warning\n stderr_str = '\\n'.join(l for l in stderr_str.decode('utf8').split('\\n') if not \"Warning\" in l)\n if stderr_str:\n raise TrainingError(stderr_str)\n \n # Save schema.\n if not self.schema:\n self.schema = arff.ArffFile.load(training_fn, schema_only=True).copy(schema_only=True)\n \n # Save model.\n with open(model_fn, 'rb') as fin:\n self._model_data = fin.read()\n assert self._model_data\n finally:\n # Cleanup files.\n if model_fn:\n os.remove(model_fn)\n if training_fn and clean_training:\n os.remove(training_fn)\n if testing_fn and clean_testing:\n os.remove(testing_fn)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef predict(self, query_data, verbose=False, distribution=False, cleanup=True):\n model_fn = None\n query_fn = None\n clean_query = False\n stdout = None\n try:\n \n # Validate query data.\n if isinstance(query_data, basestring):\n assert os.path.isfile(query_data)\n query_fn = query_data\n else:\n #assert isinstance(query_data, arff.ArffFile) #TODO: doesn't work in Python 3.*?\n assert type(query_data).__name__ == 'ArffFile', 'Must be of type ArffFile, not \"%s\"' % type(query_data).__name__\n fd, query_fn = tempfile.mkstemp(suffix='.arff')\n if verbose:\n print('writing', query_fn)\n os.close(fd)\n open(query_fn, 'w').write(query_data.write())\n clean_query = True\n assert query_fn\n \n # Validate model file.\n fd, model_fn = tempfile.mkstemp()\n os.close(fd)\n assert self._model_data, \"You must train this classifier before predicting.\"\n fout = open(model_fn, 'wb')\n fout.write(self._model_data)\n fout.close()\n \n# print(open(model_fn).read()\n# print(open(query_fn).read()\n # Call Weka Jar.\n args = dict(\n CP=CP,\n classifier_name=self.name,\n model_fn=model_fn,\n query_fn=query_fn,\n #ckargs = self._get_ckargs_str(),\n distribution=('-distribution' if distribution else ''),\n )\n cmd = (\"java -cp %(CP)s %(classifier_name)s -p 0 %(distribution)s -l \\\"%(model_fn)s\\\" -T \\\"%(query_fn)s\\\"\") % args\n if verbose:\n print(cmd)\n p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)\n stdin, stdout, stderr = (p.stdin, p.stdout, p.stderr)\n stdout_str = stdout.read()\n stderr_str = stderr.read()\n if verbose:\n print('stdout:')\n print(stdout_str)\n print('stderr:')\n print(stderr_str)\n if stderr_str:\n raise PredictionError(stderr_str)\n \n if stdout_str:\n # inst# actual predicted error prediction\n #header = 'inst,actual,predicted,error'.split(',')\n query = arff.ArffFile.load(query_fn)\n query_variables = [\n query.attributes[i]\n for i, v in enumerate(query.data[0])\n if v == arff.MISSING]\n if not query_variables:\n query_variables = [query.attributes[-1]]\n# assert query_variables, \\\n# \"There must be at least one query variable in the query.\"\n if verbose:\n print('query_variables:', query_variables)\n header = 'predicted'.split(',')\n # sample line: 1 1:? 4:36 + 1\n \n # Expected output without distribution:\n #=== Predictions on test data ===\n #\n # inst# actual predicted error prediction\n # 1 1:? 11:Acer_tr + 1\n\n #=== Predictions on test data ===\n #\n # inst# actual predicted error\n # 1 ? 7 ? \n\n #=== Predictions on test data ===\n #\n # inst# actual predicted error prediction\n # 1 1:? 1:0 0.99 \n # 2 1:? 1:0 0.99 \n # 3 1:? 1:0 0.99 \n # 4 1:? 1:0 0.99 \n # 5 1:? 1:0 0.99 \n\n # Expected output with distribution:\n #=== Predictions on test data ===\n #\n # inst# actual predicted error distribution\n # 1 1:? 11:Acer_tr + 0,0,0,0,0,0,0,0,0,0,*1,0,0,0,0,0...\n\n # Expected output with simple format:\n # inst# actual predicted error\n # 1 ? -3.417 ? \n\n\n q = re.findall(\n r'J48 pruned tree\\s+\\-+:\\s+([0-9]+)\\s+',\n stdout_str.decode('utf-8'), re.MULTILINE|re.DOTALL)\n if q:\n class_label = q[0]\n prob = 1.0\n yield PredictionResult(\n actual=None,\n predicted=class_label,\n probability=prob,)\n elif re.findall(r'error\\s+(?:distribution|prediction)', stdout_str.decode('utf-8')):\n # Check for distribution output.\n matches = re.findall(\n r\"^\\s*[0-9\\.]+\\s+[a-zA-Z0-9\\.\\?\\:]+\\s+(?P[a-zA-Z0-9_\\.\\?\\:]+)\\s+\\+?\\s+(?P[a-zA-Z0-9\\.\\?\\,\\*]+)\",\n stdout_str.decode('utf-8'),\n re.MULTILINE)\n assert matches, (\"No results found matching distribution pattern in stdout: %s\") % stdout_str\n for match in matches:\n prediction, prob = match\n class_index, class_label = prediction.split(':')\n class_index = int(class_index)\n if distribution:\n # Convert list of probabilities into a hash linking the prob\n # to the associated class value.\n prob = dict(zip(\n query.attribute_data[query.attributes[-1]],\n map(float, prob.replace('*', '').split(','))))\n else:\n prob = float(prob)\n class_label = query.attribute_data[query.attributes[-1]][class_index-1]\n yield PredictionResult(\n actual=None,\n predicted=class_label,\n probability=prob,)\n else:\n # Otherwise, assume a simple output.\n matches = re.findall(\n # inst# actual predicted \n r\"^\\s*([0-9\\.]+)\\s+([a-zA-Z0-9\\-\\.\\?\\:]+)\\s+([a-zA-Z0-9\\-_\\.\\?\\:]+)\\s+\",\n stdout_str.decode('utf-8'),\n re.MULTILINE)\n assert matches, \"No results found matching simple pattern in stdout: %s\" % stdout_str\n #print('matches:',len(matches)\n for match in matches:\n inst, actual, predicted = match\n class_name = query.attributes[-1]\n actual_value = query.get_attribute_value(class_name, actual)\n predicted_value = query.get_attribute_value(class_name, predicted)\n yield PredictionResult(\n actual=actual_value,\n predicted=predicted_value,\n probability=None,)\n finally:\n # Cleanup files.\n if cleanup:\n if model_fn:\n self._model_data = open(model_fn, 'rb').read()\n os.remove(model_fn)\n if query_fn and clean_query:\n os.remove(query_fn)", "response": "Predict the value of a class by running a Weka model and predicting it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a ratio of classifiers that were able to be trained successfully.", "response": "def get_training_coverage(self):\n \"\"\"\n Returns a ratio of classifiers that were able to be trained successfully.\n \"\"\"\n total = len(self.training_results)\n i = sum(1 for data in self.training_results.values() if not isinstance(data, basestring))\n return i/float(total)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_new_links(self, url, resp):\n links_on_page = resp.xpath('//a/@href')\n links = [utils.clean_url(u, url) for u in links_on_page]\n\n # Remove non-links through filtering by protocol\n links = [x for x in links if utils.check_protocol(x)]\n\n # Restrict new URLs by the domain of the input URL\n if not self.args['nonstrict']:\n domain = utils.get_domain(url)\n links = [x for x in links if utils.get_domain(x) == domain]\n\n # Filter URLs by regex keywords, if any\n if self.args['crawl']:\n links = utils.re_filter(links, self.args['crawl'])\n return links", "response": "Get new links from a URL and filter them."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef page_crawled(self, page_resp):\n page_text = utils.parse_text(page_resp)\n page_hash = utils.hash_text(''.join(page_text))\n if page_hash not in self.page_cache:\n utils.cache_page(self.page_cache, page_hash, self.args['cache_size'])\n return False\n return True", "response": "Check if a page has been crawled by hashing its text content."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncrawls the page responses and creates PART. html files.", "response": "def crawl_links(self, seed_url=None):\n \"\"\"Find new links given a seed URL and follow them breadth-first.\n\n Save page responses as PART.html files.\n Return the PART.html filenames created during crawling.\n \"\"\"\n if seed_url is not None:\n self.seed_url = seed_url\n\n if self.seed_url is None:\n sys.stderr.write('Crawling requires a seed URL.\\n')\n return []\n\n prev_part_num = utils.get_num_part_files()\n crawled_links = set()\n uncrawled_links = OrderedSet()\n\n uncrawled_links.add(self.seed_url)\n try:\n while uncrawled_links:\n # Check limit on number of links and pages to crawl\n if self.limit_reached(len(crawled_links)):\n break\n url = uncrawled_links.pop(last=False)\n\n # Remove protocol, fragments, etc. to get unique URLs\n unique_url = utils.remove_protocol(utils.clean_url(url))\n if unique_url not in crawled_links:\n raw_resp = utils.get_raw_resp(url)\n if raw_resp is None:\n if not self.args['quiet']:\n sys.stderr.write('Failed to parse {0}.\\n'.format(url))\n continue\n\n resp = lh.fromstring(raw_resp)\n if self.page_crawled(resp):\n continue\n\n crawled_links.add(unique_url)\n new_links = self.get_new_links(url, resp)\n uncrawled_links.update(new_links)\n if not self.args['quiet']:\n print('Crawled {0} (#{1}).'.format(url, len(crawled_links)))\n\n # Write page response to PART.html file\n utils.write_part_file(self.args, url, raw_resp, resp, len(crawled_links))\n except (KeyboardInterrupt, EOFError):\n pass\n\n curr_part_num = utils.get_num_part_files()\n return utils.get_part_filenames(curr_part_num, prev_part_num)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget available proxies to use with requests library.", "response": "def get_proxies():\n \"\"\"Get available proxies to use with requests library.\"\"\"\n proxies = getproxies()\n filtered_proxies = {}\n for key, value in proxies.items():\n if key.startswith('http://'):\n if not value.startswith('http://'):\n filtered_proxies[key] = 'http://{0}'.format(value)\n else:\n filtered_proxies[key] = value\n return filtered_proxies"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget webpage response as an lxml. html. HtmlElement object.", "response": "def get_resp(url):\n \"\"\"Get webpage response as an lxml.html.HtmlElement object.\"\"\"\n try:\n headers = {'User-Agent': random.choice(USER_AGENTS)}\n try:\n request = requests.get(url, headers=headers, proxies=get_proxies())\n except MissingSchema:\n url = add_protocol(url)\n request = requests.get(url, headers=headers, proxies=get_proxies())\n return lh.fromstring(request.text.encode('utf-8') if PY2 else request.text)\n except Exception:\n sys.stderr.write('Failed to retrieve {0}.\\n'.format(url))\n raise"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nenable requests library cache.", "response": "def enable_cache():\n \"\"\"Enable requests library cache.\"\"\"\n try:\n import requests_cache\n except ImportError as err:\n sys.stderr.write('Failed to enable cache: {0}\\n'.format(str(err)))\n return\n if not os.path.exists(CACHE_DIR):\n os.makedirs(CACHE_DIR)\n requests_cache.install_cache(CACHE_FILE)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn MD5 hash of a string.", "response": "def hash_text(text):\n \"\"\"Return MD5 hash of a string.\"\"\"\n md5 = hashlib.md5()\n md5.update(text)\n return md5.hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a page to the page cache.", "response": "def cache_page(page_cache, page_hash, cache_size):\n \"\"\"Add a page to the page cache.\"\"\"\n page_cache.append(page_hash)\n if len(page_cache) > cache_size:\n page_cache.pop(0)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef re_filter(text, regexps):\n if not regexps:\n return text\n\n matched_text = []\n compiled_regexps = [re.compile(x) for x in regexps]\n for line in text:\n if line in matched_text:\n continue\n\n for regexp in compiled_regexps:\n found = regexp.search(line)\n if found and found.group():\n matched_text.append(line)\n\n return matched_text or text", "response": "Filter text using regular expressions."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_whitespace(text):\n clean_text = []\n curr_line = ''\n # Remove any newlines that follow two lines of whitespace consecutively\n # Also remove whitespace at start and end of text\n while text:\n if not curr_line:\n # Find the first line that is not whitespace and add it\n curr_line = text.pop(0)\n while not curr_line.strip() and text:\n curr_line = text.pop(0)\n if curr_line.strip():\n clean_text.append(curr_line)\n else:\n # Filter the rest of the lines\n curr_line = text.pop(0)\n if not text:\n # Add the final line if it is not whitespace\n if curr_line.strip():\n clean_text.append(curr_line)\n continue\n\n if curr_line.strip():\n clean_text.append(curr_line)\n else:\n # If the current line is whitespace then make sure there is\n # no more than one consecutive line of whitespace following\n if not text[0].strip():\n if len(text) > 1 and text[1].strip():\n clean_text.append(curr_line)\n else:\n clean_text.append(curr_line)\n\n # Now filter each individual line for extraneous whitespace\n cleaner_text = []\n for line in clean_text:\n clean_line = ' '.join(line.split())\n if not clean_line.strip():\n clean_line += '\\n'\n cleaner_text.append(clean_line)\n return cleaner_text", "response": "Remove unnecessary whitespace while keeping logical structure."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_text(infile, xpath=None, filter_words=None, attributes=None):\n infiles = []\n text = []\n if xpath is not None:\n infile = parse_html(infile, xpath)\n if isinstance(infile, list):\n if isinstance(infile[0], lh.HtmlElement):\n infiles = list(infile)\n else:\n text = [line + '\\n' for line in infile]\n elif isinstance(infile, lh.HtmlElement):\n infiles = [infile]\n else:\n text = [infile]\n else:\n infiles = [infile]\n\n if attributes is not None:\n attributes = [clean_attr(x) for x in attributes]\n attributes = [x for x in attributes if x]\n else:\n attributes = ['text()']\n\n if not text:\n text_xpath = '//*[not(self::script) and not(self::style)]'\n for attr in attributes:\n for infile in infiles:\n if isinstance(infile, lh.HtmlElement):\n new_text = infile.xpath('{0}/{1}'.format(text_xpath, attr))\n else:\n # re.split preserves delimiters place in the list\n new_text = [x for x in re.split('(\\n)', infile) if x]\n text += new_text\n\n if filter_words is not None:\n text = re_filter(text, filter_words)\n return [''.join(x for x in line if x in string.printable)\n for line in remove_whitespace(text) if line]", "response": "Parse text using XPath regex keywords and tag attributes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_parsed_text(args, infilename):\n parsed_text = []\n if infilename.endswith('.html'):\n # Convert HTML to lxml object for content parsing\n html = lh.fromstring(read_files(infilename))\n text = None\n else:\n html = None\n text = read_files(infilename)\n\n if html is not None:\n parsed_text = parse_text(html, args['xpath'], args['filter'],\n args['attributes'])\n elif text is not None:\n parsed_text = parse_text(text, args['xpath'], args['filter'])\n else:\n if not args['quiet']:\n sys.stderr.write('Failed to parse text from {0}.\\n'\n .format(infilename))\n return parsed_text", "response": "Parse and return text content of infiles."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_html(infile, xpath):\n if not isinstance(infile, lh.HtmlElement):\n infile = lh.fromstring(infile)\n infile = infile.xpath(xpath)\n if not infile:\n raise ValueError('XPath {0} returned no results.'.format(xpath))\n return infile", "response": "Filter HTML using XPath."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef clean_url(url, base_url=None):\n parsed_url = urlparse(url)\n\n fragment = '{url.fragment}'.format(url=parsed_url)\n if fragment:\n url = url.split(fragment)[0]\n\n # Identify internal URLs and fix their format\n netloc = '{url.netloc}'.format(url=parsed_url)\n if base_url is not None and not netloc:\n parsed_base = urlparse(base_url)\n split_base = '{url.scheme}://{url.netloc}{url.path}/'.format(url=parsed_base)\n url = urljoin(split_base, url)\n netloc = '{url.netloc}'.format(url=urlparse(url))\n\n if 'www.' in netloc:\n url = url.replace(netloc, netloc.replace('www.', ''))\n return url.rstrip(string.punctuation)", "response": "Remove www and fragments from internal URLs and remove www."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd com suffix to URL if none found.", "response": "def add_url_suffix(url):\n \"\"\"Add .com suffix to URL if none found.\"\"\"\n url = url.rstrip('/')\n if not has_suffix(url):\n return '{0}.com'.format(url)\n return url"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconstructing the output filename from domain and end of path.", "response": "def get_outfilename(url, domain=None):\n \"\"\"Construct the output filename from domain and end of path.\"\"\"\n if domain is None:\n domain = get_domain(url)\n\n path = '{url.path}'.format(url=urlparse(url))\n if '.' in path:\n tail_url = path.split('.')[-2]\n else:\n tail_url = path\n\n if tail_url:\n if '/' in tail_url:\n tail_pieces = [x for x in tail_url.split('/') if x]\n tail_url = tail_pieces[-1]\n\n # Keep length of return string below or equal to max_len\n max_len = 24\n if domain:\n max_len -= (len(domain) + 1)\n if len(tail_url) > max_len:\n if '-' in tail_url:\n tail_pieces = [x for x in tail_url.split('-') if x]\n tail_url = tail_pieces.pop(0)\n if len(tail_url) > max_len:\n tail_url = tail_url[:max_len]\n else:\n # Add as many tail pieces that can fit\n tail_len = 0\n for piece in tail_pieces:\n tail_len += len(piece)\n if tail_len <= max_len:\n tail_url += '-' + piece\n else:\n break\n else:\n tail_url = tail_url[:max_len]\n\n if domain:\n return '{0}-{1}'.format(domain, tail_url).lower()\n return tail_url\n return domain.lower()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nuses first possible entry in query as filename.", "response": "def get_single_outfilename(args):\n \"\"\"Use first possible entry in query as filename.\"\"\"\n for arg in args['query']:\n if arg in args['files']:\n return ('.'.join(arg.split('.')[:-1])).lower()\n for url in args['urls']:\n if arg.strip('/') in url:\n domain = get_domain(url)\n return get_outfilename(url, domain)\n sys.stderr.write('Failed to construct a single out filename.\\n')\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmodifying filename to have a unique numerical identifier.", "response": "def modify_filename_id(filename):\n \"\"\"Modify filename to have a unique numerical identifier.\"\"\"\n split_filename = os.path.splitext(filename)\n id_num_re = re.compile('(\\(\\d\\))')\n id_num = re.findall(id_num_re, split_filename[-2])\n if id_num:\n new_id_num = int(id_num[-1].lstrip('(').rstrip(')')) + 1\n\n # Reconstruct filename with incremented id and its extension\n filename = ''.join((re.sub(id_num_re, '({0})'.format(new_id_num),\n split_filename[-2]), split_filename[-1]))\n else:\n split_filename = os.path.splitext(filename)\n\n # Reconstruct filename with new id and its extension\n filename = ''.join(('{0} (2)'.format(split_filename[-2]),\n split_filename[-1]))\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef overwrite_file_check(args, filename):\n if not args['overwrite'] and os.path.exists(filename):\n # Confirm overwriting of the file, or modify filename\n if args['no_overwrite']:\n overwrite = False\n else:\n try:\n overwrite = confirm_input(input('Overwrite {0}? (yes/no): '\n .format(filename)))\n except (KeyboardInterrupt, EOFError):\n sys.exit()\n if not overwrite:\n new_filename = modify_filename_id(filename)\n while os.path.exists(new_filename):\n new_filename = modify_filename_id(new_filename)\n return new_filename\n return filename", "response": "If filename exists overwrite or modify it to be unique."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprint text content of infiles to stdout.", "response": "def print_text(args, infilenames, outfilename=None):\n \"\"\"Print text content of infiles to stdout.\n\n Keyword arguments:\n args -- program arguments (dict)\n infilenames -- names of user-inputted and/or downloaded files (list)\n outfilename -- only used for interface purposes (None)\n \"\"\"\n for infilename in infilenames:\n parsed_text = get_parsed_text(args, infilename)\n if parsed_text:\n for line in parsed_text:\n print(line)\n print('')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting pdf files to disk using pdfkit.", "response": "def write_pdf_files(args, infilenames, outfilename):\n \"\"\"Write pdf file(s) to disk using pdfkit.\n\n Keyword arguments:\n args -- program arguments (dict)\n infilenames -- names of user-inputted and/or downloaded files (list)\n outfilename -- name of output pdf file (str)\n \"\"\"\n if not outfilename.endswith('.pdf'):\n outfilename = outfilename + '.pdf'\n outfilename = overwrite_file_check(args, outfilename)\n\n options = {}\n try:\n if args['multiple']:\n # Multiple files are written one at a time, so infilenames will\n # never contain more than one file here\n infilename = infilenames[0]\n if not args['quiet']:\n print('Attempting to write to {0}.'.format(outfilename))\n else:\n options['quiet'] = None\n\n if args['xpath']:\n # Process HTML with XPath before writing\n html = parse_html(read_files(infilename), args['xpath'])\n if isinstance(html, list):\n if isinstance(html[0], str):\n pk.from_string('\\n'.join(html), outfilename,\n options=options)\n else:\n pk.from_string('\\n'.join(lh.tostring(x) for x in html),\n outfilename, options=options)\n elif isinstance(html, str):\n pk.from_string(html, outfilename, options=options)\n else:\n pk.from_string(lh.tostring(html), outfilename,\n options=options)\n else:\n pk.from_file(infilename, outfilename, options=options)\n elif args['single']:\n if not args['quiet']:\n print('Attempting to write {0} page(s) to {1}.'\n .format(len(infilenames), outfilename))\n else:\n options['quiet'] = None\n\n if args['xpath']:\n # Process HTML with XPath before writing\n html = parse_html(read_files(infilenames), args['xpath'])\n if isinstance(html, list):\n if isinstance(html[0], str):\n pk.from_string('\\n'.join(html), outfilename,\n options=options)\n else:\n pk.from_string('\\n'.join(lh.tostring(x) for x in html),\n outfilename, options=options)\n elif isinstance(html, str):\n pk.from_string(html, outfilename, options=options)\n else:\n pk.from_string(lh.tostring(html), outfilename,\n options=options)\n else:\n pk.from_file(infilenames, outfilename, options=options)\n return True\n except (OSError, IOError) as err:\n sys.stderr.write('An error occurred while writing {0}:\\n{1}'\n .format(outfilename, str(err)))\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting csv files to disk.", "response": "def write_csv_files(args, infilenames, outfilename):\n \"\"\"Write csv file(s) to disk.\n\n Keyword arguments:\n args -- program arguments (dict)\n infilenames -- names of user-inputted and/or downloaded files (list)\n outfilename -- name of output text file (str)\n \"\"\"\n def csv_convert(line):\n \"\"\"Strip punctuation and insert commas\"\"\"\n clean_line = []\n for word in line.split(' '):\n clean_line.append(word.strip(string.punctuation))\n return ', '.join(clean_line)\n\n if not outfilename.endswith('.csv'):\n outfilename = outfilename + '.csv'\n outfilename = overwrite_file_check(args, outfilename)\n\n all_text = [] # Text must be aggregated if writing to a single output file\n for i, infilename in enumerate(infilenames):\n parsed_text = get_parsed_text(args, infilename)\n if parsed_text:\n if args['multiple']:\n if not args['quiet']:\n print('Attempting to write to {0}.'.format(outfilename))\n\n csv_text = [csv_convert(x) for x in parsed_text]\n print(csv_text)\n write_file(csv_text, outfilename)\n elif args['single']:\n all_text += parsed_text\n # Newline added between multiple files being aggregated\n if len(infilenames) > 1 and i < len(infilenames) - 1:\n all_text.append('\\n')\n\n # Write all text to a single output file\n if args['single'] and all_text:\n if not args['quiet']:\n print('Attempting to write {0} page(s) to {1}.'\n .format(len(infilenames), outfilename))\n\n csv_text = [csv_convert(x) for x in all_text]\n print(csv_text)\n write_file(csv_text, outfilename)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_text_files(args, infilenames, outfilename):\n if not outfilename.endswith('.txt'):\n outfilename = outfilename + '.txt'\n outfilename = overwrite_file_check(args, outfilename)\n\n all_text = [] # Text must be aggregated if writing to a single output file\n for i, infilename in enumerate(infilenames):\n parsed_text = get_parsed_text(args, infilename)\n if parsed_text:\n if args['multiple']:\n if not args['quiet']:\n print('Attempting to write to {0}.'.format(outfilename))\n write_file(parsed_text, outfilename)\n elif args['single']:\n all_text += parsed_text\n # Newline added between multiple files being aggregated\n if len(infilenames) > 1 and i < len(infilenames) - 1:\n all_text.append('\\n')\n\n # Write all text to a single output file\n if args['single'] and all_text:\n if not args['quiet']:\n print('Attempting to write {0} page(s) to {1}.'\n .format(len(infilenames), outfilename))\n write_file(all_text, outfilename)", "response": "Write text files to disk."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a single file to disk.", "response": "def write_file(data, outfilename):\n \"\"\"Write a single file to disk.\"\"\"\n if not data:\n return False\n try:\n with open(outfilename, 'w') as outfile:\n for line in data:\n if line:\n outfile.write(line)\n return True\n except (OSError, IOError) as err:\n sys.stderr.write('An error occurred while writing {0}:\\n{1}'\n .format(outfilename, str(err)))\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the number of PART. html files currently saved to disk.", "response": "def get_num_part_files():\n \"\"\"Get the number of PART.html files currently saved to disk.\"\"\"\n num_parts = 0\n for filename in os.listdir(os.getcwd()):\n if filename.startswith('PART') and filename.endswith('.html'):\n num_parts += 1\n return num_parts"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_part_images(url, raw_html, html, filename):\n save_dirname = '{0}_files'.format(os.path.splitext(filename)[0])\n if not os.path.exists(save_dirname):\n os.makedirs(save_dirname)\n images = html.xpath('//img/@src')\n internal_image_urls = [x for x in images if x.startswith('/')]\n\n headers = {'User-Agent': random.choice(USER_AGENTS)}\n for img_url in images:\n img_name = img_url.split('/')[-1]\n if \"?\" in img_name:\n img_name = img_name.split('?')[0]\n if not os.path.splitext(img_name)[1]:\n img_name = '{0}.jpeg'.format(img_name)\n\n try:\n full_img_name = os.path.join(save_dirname, img_name)\n with open(full_img_name, 'wb') as img:\n if img_url in internal_image_urls:\n # Internal images need base url added\n full_img_url = '{0}{1}'.format(url.rstrip('/'), img_url)\n else:\n # External image\n full_img_url = img_url\n img_content = requests.get(full_img_url, headers=headers,\n proxies=get_proxies()).content\n img.write(img_content)\n raw_html = raw_html.replace(escape(img_url), full_img_name)\n except (OSError, IOError):\n pass\n time.sleep(random.uniform(0, 0.5)) # Slight delay between downloads\n return raw_html", "response": "Write image files associated with a PART. html file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_part_file(args, url, raw_html, html=None, part_num=None):\n if part_num is None:\n part_num = get_num_part_files() + 1\n filename = 'PART{0}.html'.format(part_num)\n\n # Decode bytes to string in Python 3 versions\n if not PY2 and isinstance(raw_html, bytes):\n raw_html = raw_html.encode('ascii', 'ignore')\n\n # Convert html to an lh.HtmlElement object for parsing/saving images\n if html is None:\n html = lh.fromstring(raw_html)\n\n # Parse HTML if XPath entered\n if args['xpath']:\n raw_html = parse_html(html, args['xpath'])\n if isinstance(raw_html, list):\n if not isinstance(raw_html[0], lh.HtmlElement):\n raise ValueError('XPath should return an HtmlElement object.')\n else:\n if not isinstance(raw_html, lh.HtmlElement):\n raise ValueError('XPath should return an HtmlElement object.')\n\n # Write HTML and possibly images to disk\n if raw_html:\n if not args['no_images'] and (args['pdf'] or args['html']):\n raw_html = write_part_images(url, raw_html, html, filename)\n with open(filename, 'w') as part:\n if not isinstance(raw_html, list):\n raw_html = [raw_html]\n if isinstance(raw_html[0], lh.HtmlElement):\n for elem in raw_html:\n part.write(lh.tostring(elem))\n else:\n for line in raw_html:\n part.write(line)", "response": "Write a PART. html file to disk images in PART_files directory."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_part_filenames(num_parts=None, start_num=0):\n if num_parts is None:\n num_parts = get_num_part_files()\n return ['PART{0}.html'.format(i) for i in range(start_num+1, num_parts+1)]", "response": "Get numbered PART. html filenames."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_files(filenames):\n if isinstance(filenames, list):\n for filename in filenames:\n with open(filename, 'r') as infile:\n return infile.read()\n else:\n with open(filenames, 'r') as infile:\n return infile.read()", "response": "Read a file into memory."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nremoves PART ( # )_files directory containing images from disk.", "response": "def remove_part_images(filename):\n \"\"\"Remove PART(#)_files directory containing images from disk.\"\"\"\n dirname = '{0}_files'.format(os.path.splitext(filename)[0])\n if os.path.exists(dirname):\n shutil.rmtree(dirname)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving PART. html files and image directories from disk.", "response": "def remove_part_files(num_parts=None):\n \"\"\"Remove PART(#).html files and image directories from disk.\"\"\"\n filenames = get_part_filenames(num_parts)\n for filename in filenames:\n remove_part_images(filename)\n remove_file(filename)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck user input for yes no or an exit signal.", "response": "def confirm_input(user_input):\n \"\"\"Check user input for yes, no, or an exit signal.\"\"\"\n if isinstance(user_input, list):\n user_input = ''.join(user_input)\n\n try:\n u_inp = user_input.lower().strip()\n except AttributeError:\n u_inp = user_input\n\n # Check for exit signal\n if u_inp in ('q', 'quit', 'exit'):\n sys.exit()\n if u_inp in ('y', 'yes'):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchanges directory and / or create it if necessary.", "response": "def mkdir_and_cd(dirname):\n \"\"\"Change directory and/or create it if necessary.\"\"\"\n if not os.path.exists(dirname):\n os.makedirs(dirname)\n os.chdir(dirname)\n else:\n os.chdir(dirname)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert(data, in_format, out_format, name=None, pretty=False):\n # Decide on a json formatter depending on desired prettiness\n dumps = json.dumps if pretty else json.compress\n\n # Shortcut for avoiding pybel dependency\n if not has_ob and in_format == 'json' and out_format == 'json':\n return dumps(json.loads(data) if is_string(data) else data)\n elif not has_ob:\n raise ImportError(\"Chemical file format conversion requires pybel.\")\n\n # These use the open babel library to interconvert, with additions for json\n if in_format == 'json':\n mol = json_to_pybel(json.loads(data) if is_string(data) else data)\n elif in_format == 'pybel':\n mol = data\n else:\n mol = pybel.readstring(in_format, data)\n\n # Infer structure in cases where the input format has no specification\n if not mol.OBMol.HasNonZeroCoords():\n mol.make3D()\n\n # Make P1 if that's a thing, recalculating bonds in process\n if in_format == 'mmcif' and hasattr(mol, 'unitcell'):\n mol.unitcell.FillUnitCell(mol.OBMol)\n mol.OBMol.ConnectTheDots()\n mol.OBMol.PerceiveBondOrders()\n\n mol.OBMol.Center()\n\n if out_format == 'pybel':\n return mol\n elif out_format == 'object':\n return pybel_to_json(mol, name)\n elif out_format == 'json':\n return dumps(pybel_to_json(mol, name))\n else:\n return mol.write(out_format)", "response": "Converts a string representing the chemical file in in_format to out_format."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef json_to_pybel(data, infer_bonds=False):\n obmol = ob.OBMol()\n obmol.BeginModify()\n for atom in data['atoms']:\n obatom = obmol.NewAtom()\n obatom.SetAtomicNum(table.GetAtomicNum(str(atom['element'])))\n obatom.SetVector(*atom['location'])\n if 'label' in atom:\n pd = ob.OBPairData()\n pd.SetAttribute('_atom_site_label')\n pd.SetValue(atom['label'])\n obatom.CloneData(pd)\n\n # If there is no bond data, try to infer them\n if 'bonds' not in data or not data['bonds']:\n if infer_bonds:\n obmol.ConnectTheDots()\n obmol.PerceiveBondOrders()\n # Otherwise, use the bonds in the data set\n else:\n for bond in data['bonds']:\n if 'atoms' not in bond:\n continue\n obmol.AddBond(bond['atoms'][0] + 1, bond['atoms'][1] + 1,\n bond['order'])\n\n # Check for unit cell data\n if 'unitcell' in data:\n uc = ob.OBUnitCell()\n uc.SetData(*(ob.vector3(*v) for v in data['unitcell']))\n uc.SetSpaceGroup('P1')\n obmol.CloneData(uc)\n obmol.EndModify()\n\n mol = pybel.Molecule(obmol)\n\n # Add partial charges\n if 'charge' in data['atoms'][0]:\n mol.OBMol.SetPartialChargesPerceived()\n for atom, pyatom in zip(data['atoms'], mol.atoms):\n pyatom.OBAtom.SetPartialCharge(atom['charge'])\n\n return mol", "response": "Converts a python data structure to a Python object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a pybel. Molecule to json.", "response": "def pybel_to_json(molecule, name=None):\n \"\"\"Converts a pybel molecule to json.\n\n Args:\n molecule: An instance of `pybel.Molecule`\n name: (Optional) If specified, will save a \"name\" property\n Returns:\n A Python dictionary containing atom and bond data\n \"\"\"\n # Save atom element type and 3D location.\n atoms = [{'element': table.GetSymbol(atom.atomicnum),\n 'location': list(atom.coords)}\n for atom in molecule.atoms]\n # Recover auxiliary data, if exists\n for json_atom, pybel_atom in zip(atoms, molecule.atoms):\n if pybel_atom.partialcharge != 0:\n json_atom['charge'] = pybel_atom.partialcharge\n if pybel_atom.OBAtom.HasData('_atom_site_label'):\n obatom = pybel_atom.OBAtom\n json_atom['label'] = obatom.GetData('_atom_site_label').GetValue()\n if pybel_atom.OBAtom.HasData('color'):\n obatom = pybel_atom.OBAtom\n json_atom['color'] = obatom.GetData('color').GetValue()\n\n # Save number of bonds and indices of endpoint atoms\n bonds = [{'atoms': [b.GetBeginAtom().GetIndex(),\n b.GetEndAtom().GetIndex()],\n 'order': b.GetBondOrder()}\n for b in ob.OBMolBondIter(molecule.OBMol)]\n output = {'atoms': atoms, 'bonds': bonds, 'units': {}}\n\n # If there's unit cell data, save it to the json output\n if hasattr(molecule, 'unitcell'):\n uc = molecule.unitcell\n output['unitcell'] = [[v.GetX(), v.GetY(), v.GetZ()]\n for v in uc.GetCellVectors()]\n density = (sum(atom.atomicmass for atom in molecule.atoms) /\n (uc.GetCellVolume() * 0.6022))\n output['density'] = density\n output['units']['density'] = 'kg / L'\n\n # Save the formula to json. Use Hill notation, just to have a standard.\n element_count = Counter(table.GetSymbol(a.atomicnum) for a in molecule)\n hill_count = []\n for element in ['C', 'H']:\n if element in element_count:\n hill_count += [(element, element_count[element])]\n del element_count[element]\n hill_count += sorted(element_count.items())\n\n # If it's a crystal, then reduce the Hill formula\n div = (reduce(gcd, (c[1] for c in hill_count))\n if hasattr(molecule, 'unitcell') else 1)\n\n output['formula'] = ''.join(n if c / div == 1 else '%s%d' % (n, c / div)\n for n, c in hill_count)\n output['molecular_weight'] = molecule.molwt / div\n output['units']['molecular_weight'] = 'g / mol'\n\n # If the input has been given a name, add that\n if name:\n output['name'] = name\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfire when an unserializable object is hit.", "response": "def default(self, obj):\n \"\"\"Fired when an unserializable object is hit.\"\"\"\n if hasattr(obj, '__dict__'):\n return obj.__dict__.copy()\n elif HAS_NUMPY and isinstance(obj, np.ndarray):\n return obj.copy().tolist()\n else:\n raise TypeError((\"Object of type {:s} with value of {:s} is not \"\n \"JSON serializable\").format(type(obj), repr(obj)))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(data, format='auto', size=(400, 300), drawing_type='ball and stick',\n camera_type='perspective', shader='lambert', display_html=True,\n element_properties=None, show_save=False):\n \"\"\"Draws an interactive 3D visualization of the inputted chemical.\n\n Args:\n data: A string or file representing a chemical.\n format: The format of the `data` variable (default is 'auto').\n size: Starting dimensions of visualization, in pixels.\n drawing_type: Specifies the molecular representation. Can be 'ball and\n stick', 'wireframe', or 'space filling'.\n camera_type: Can be 'perspective' or 'orthographic'.\n shader: Specifies shading algorithm to use. Can be 'toon', 'basic',\n 'phong', or 'lambert'.\n display_html: If True (default), embed the html in a IPython display.\n If False, return the html as a string.\n element_properites: A dictionary providing color and radius information\n for custom elements or overriding the defaults in imolecule.js\n show_save: If True, displays a save icon for rendering molecule as an\n image.\n\n The `format` can be any value specified by Open Babel\n (http://openbabel.org/docs/2.3.1/FileFormats/Overview.html). The 'auto'\n option uses the extension for files (ie. my_file.mol -> mol) and defaults\n to SMILES (smi) for strings.\n \"\"\"\n # Catch errors on string-based input before getting js involved\n draw_options = ['ball and stick', 'wireframe', 'space filling']\n camera_options = ['perspective', 'orthographic']\n shader_options = ['toon', 'basic', 'phong', 'lambert']\n if drawing_type not in draw_options:\n raise Exception(\"Invalid drawing type! Please use one of: \" +\n \", \".join(draw_options))\n if camera_type not in camera_options:\n raise Exception(\"Invalid camera type! Please use one of: \" +\n \", \".join(camera_options))\n if shader not in shader_options:\n raise Exception(\"Invalid shader! Please use one of: \" +\n \", \".join(shader_options))\n\n json_mol = generate(data, format)\n if element_properties is None:\n element_properties = dict()\n json_element_properties = to_json(element_properties)\n div_id = uuid.uuid4()\n html = \"\"\"
\n \"\"\" % (div_id, local_path[:-3], remote_path[:-3],\n div_id, size[0], size[1], drawing_type,\n camera_type, shader,\n 'true' if show_save else 'false',\n json_element_properties,\n json_mol, size[0], size[1])\n\n # Execute js and display the results in a div (see script for more)\n if display_html:\n try:\n __IPYTHON__\n except NameError:\n # We're running outside ipython, let's generate a static HTML and\n # show it in the browser\n import shutil\n import webbrowser\n from tempfile import mkdtemp\n from time import time\n try: # Python 3\n from urllib.parse import urljoin\n from urllib.request import pathname2url\n except ImportError: # Python 2\n from urlparse import urljoin\n from urllib import pathname2url\n from tornado import template\n\n t = template.Loader(file_path).load('viewer.template')\n html = t.generate(title=\"imolecule\", json_mol=json_mol,\n drawing_type=drawing_type, shader=shader,\n camera_type=camera_type,\n json_element_properties=json_element_properties)\n\n tempdir = mkdtemp(prefix='imolecule_{:.0f}_'.format(time()))\n\n html_filename = os.path.join(tempdir, 'index.html')\n with open(html_filename, 'wb') as f:\n f.write(html)\n\n libs = (('server', 'css', 'chosen.css'),\n ('server', 'css', 'server.css'),\n ('js', 'jquery-1.11.1.min.js'),\n ('server', 'js', 'chosen.jquery.min.js'),\n ('js', 'build', 'imolecule.min.js'))\n for lib in libs:\n shutil.copy(os.path.join(file_path, *lib), tempdir)\n\n html_file_url = urljoin('file:', pathname2url(html_filename))\n\n print('Opening html file: {}'.format(html_file_url))\n webbrowser.open(html_file_url)\n else:\n # We're running in ipython: display widget\n display(HTML(html))\n else:\n return html", "response": "Draws an interactive 3D visualization of the inputted chemical."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef generate(data, format=\"auto\"):\n # Support both files and strings and attempt to infer file type\n try:\n with open(data) as in_file:\n if format == 'auto':\n format = data.split('.')[-1]\n data = in_file.read()\n except:\n if format == 'auto':\n format = 'smi'\n return format_converter.convert(data, format, 'json')", "response": "Converts input chemical formats to json and optimizes structure."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_json(data, compress=False):\n return json.compress(data) if compress else json.dumps(data)", "response": "Converts the output of generate(... ) to json."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start_server():\n parser = argparse.ArgumentParser(description=\"Opens a browser-based \"\n \"client that interfaces with the \"\n \"chemical format converter.\")\n parser.add_argument('--debug', action=\"store_true\", help=\"Prints all \"\n \"transmitted data streams.\")\n parser.add_argument('--port', type=int, default=8000, help=\"The port \"\n \"on which to serve the website.\")\n parser.add_argument('--timeout', type=int, default=5, help=\"The maximum \"\n \"time, in seconds, allowed for a process to run \"\n \"before returning an error.\")\n parser.add_argument('--workers', type=int, default=2, help=\"The number of \"\n \"worker processes to use with the server.\")\n parser.add_argument('--no-browser', action=\"store_true\", help=\"Disables \"\n \"opening a browser window on startup.\")\n global args\n args = parser.parse_args()\n\n if args.debug:\n logging.getLogger().setLevel(logging.DEBUG)\n\n handlers = [(r'/', IndexHandler), (r'/websocket', WebSocket),\n (r'/static/(.*)', tornado.web.StaticFileHandler,\n {'path': os.path.normpath(os.path.dirname(__file__))})]\n application = tornado.web.Application(handlers)\n application.listen(args.port)\n\n if not args.no_browser:\n webbrowser.open('http://localhost:%d/' % args.port, new=2)\n\n try:\n tornado.ioloop.IOLoop.instance().start()\n except KeyboardInterrupt:\n sys.stderr.write(\"Received keyboard interrupt. Stopping server.\\n\")\n tornado.ioloop.IOLoop.instance().stop()\n sys.exit(1)", "response": "Starts up the imolecule server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nevaluate the function pointed to by json - rpc.", "response": "def on_message(self, message):\n \"\"\"Evaluates the function pointed to by json-rpc.\"\"\"\n json_rpc = json.loads(message)\n logging.log(logging.DEBUG, json_rpc)\n\n if self.pool is None:\n self.pool = multiprocessing.Pool(processes=args.workers)\n\n # Spawn a process to protect the server against segfaults\n async = self.pool.apply_async(_worker_process, [json_rpc])\n try:\n result = async.get(timeout=args.timeout)\n error = 0\n except multiprocessing.TimeoutError:\n result = (\"File format conversion timed out! This is due \"\n \"either to a large input file or a segmentation \"\n \"fault in the underlying open babel library.\")\n error = 1\n self.pool.terminate()\n self.pool = multiprocessing.Pool(processes=args.workers)\n except Exception:\n result = traceback.format_exc()\n error = 1\n logging.log(logging.DEBUG, result)\n\n self.write_message(json.dumps({'result': result, 'error': error,\n 'id': json_rpc['id']},\n separators=(',', ':')))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef nearly_eq(valA, valB, maxf=None, minf=None, epsilon=None):\n '''\n implementation based on:\n\n http://floating-point-gui.de/errors/comparison/\n '''\n\n if valA == valB:\n return True\n\n if maxf is None:\n maxf = float_info.max\n\n if minf is None:\n minf = float_info.min\n\n if epsilon is None:\n epsilon = float_info.epsilon\n\n absA = abs(valA)\n absB = abs(valB)\n delta = abs(valA - valB)\n\n if (valA == 0) or (valB == 0) or (delta < minf):\n return delta < (epsilon * minf)\n\n return (delta / min(absA + absB, maxf)) < (epsilon * 2)", "response": "returns True if valA == valB and false otherwise"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _convert(cls, other, ignoreScalars=False):\n '''\n :other: Point or point equivalent\n :ignorescalars: optional boolean\n :return: Point\n\n Class private method for converting 'other' into a Point\n subclasss. If 'other' already is a Point subclass, nothing\n is done. If ignoreScalars is True and other is a float or int\n type, a TypeError exception is raised.\n '''\n if ignoreScalars:\n if isinstance(other, (int, float)):\n msg = \"unable to convert {} to {}\".format(other, cls.__name__)\n raise TypeError(msg)\n\n return cls(other) if not issubclass(type(other), cls) else other", "response": "Convert other into a Point class."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of three Point objects representing the head of a unit vector.", "response": "def units(cls, scale=1):\n '''\n :scale: optional integer scaling factor\n :return: list of three Point subclass\n\n Returns three points whose coordinates are the head of a\n unit vector from the origin ( conventionally i, j and k).\n\n '''\n return [cls(x=scale), cls(y=scale), cls(z=scale)]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new Point subclass with the coordinates picked from a Gaussian distribution with mean mu and standard deviation sigma.", "response": "def gaussian(cls, mu=0, sigma=1):\n '''\n :mu: mean\n :sigma: standard deviation\n :return: Point subclass\n\n Returns a point whose coordinates are picked from a Gaussian\n distribution with mean 'mu' and standard deviation 'sigma'.\n See random.gauss for further explanation of those parameters.\n '''\n return cls(random.gauss(mu, sigma),\n random.gauss(mu, sigma),\n random.gauss(mu, sigma))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new Point with random x y and z coordinates bounded by origin and radius around origin.", "response": "def random(cls, origin=None, radius=1):\n '''\n :origin: optional Point or point equivalent\n :radius: optional float, radius around origin\n :return: Point subclass\n\n Returns a point with random x, y and z coordinates bounded by\n the sphere defined by (origin,radius).\n\n If a sphere is not supplied, a unit sphere at the origin is\n used by default.\n '''\n\n p = cls(origin)\n\n r = random.uniform(0, radius)\n u = random.uniform(0, Two_Pi)\n v = random.uniform(-Half_Pi, Half_Pi)\n\n r_cosv = r * math.cos(v)\n\n p.x += r_cosv * math.cos(u)\n p.y += r_cosv * math.sin(u)\n p.z += radius * math.sin(v)\n\n return p"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _unary_(self, func, inplace=False):\n '''\n :func: unary function to apply to each coordinate\n :inplace: optional boolean\n :return: Point\n\n Implementation private method.\n\n All of the unary operations funnel thru this method\n to reduce cut-and-paste code and enforce consistent\n behavior of unary ops.\n\n Applies 'func' to self and returns the result.\n\n The expected call signature of 'func' is f(a)\n\n If 'inplace' is True, the results are stored in 'self',\n otherwise the results will be stored in a new object.\n\n Returns a Point.\n\n '''\n dst = self if inplace else self.__class__(self)\n dst.x = func(dst.x)\n dst.y = func(dst.y)\n dst.z = func(dst.z)\n return dst", "response": "Internal function that applies a function to each coordinate and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cross(self, other):\n '''\n :other: Point or point equivalent\n :return: float\n\n Vector cross product of points U (self) and V (other), computed:\n\n U x V = (u1*i + u2*j + u3*k) x (v1*i + v2*j + v3*k)\n s1 = u2v3 - u3v2\n s2 = u3v1 - u1v3\n s3 = u1v2 - u2v1\n\n U x V = s1 + s2 + s3\n\n Returns a float.\n '''\n\n b = self.__class__._convert(other)\n\n return sum([(self.y * b.z) - (self.z * b.y),\n (self.z * b.x) - (self.x * b.z),\n (self.x * b.y) - (self.y * b.x)])", "response": "Return the cross product of two sets of points U x V."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef isBetween(self, a, b, axes='xyz'):\n '''\n :a: Point or point equivalent\n :b: Point or point equivalent\n :axis: optional string\n :return: float\n\n Checks the coordinates specified in 'axes' of 'self' to\n determine if they are bounded by 'a' and 'b'. The range\n is inclusive of end-points.\n\n Returns boolean.\n '''\n a = self.__class__._convert(a)\n b = self.__class__._convert(b)\n\n fn = lambda k: (self[k] >= min(a[k], b[k])) and (\n self[k] <= max(a[k], b[k]))\n\n return all(fn(axis) for axis in axes)", "response": "Checks if the coordinates specified in axes of self to\n are bounded by a and b."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the angle of rotation around a given point.", "response": "def ccw(self, b, c, axis='z'):\n '''\n :b: Point or point equivalent\n :c: Point or point equivalent\n :axis: optional string or integer in set('x',0,'y',1,'z',2)\n :return: float\n\n CCW - Counter Clockwise\n\n Returns an integer signifying the direction of rotation around 'axis'\n described by the angle [b, self, c].\n\n > 0 : counter-clockwise\n 0 : points are collinear\n < 0 : clockwise\n\n Returns an integer.\n\n Raises ValueError if axis is not in 'xyz'.\n '''\n bsuba = b - self\n csuba = c - self\n\n if axis in ['z', 2]:\n return (bsuba.x * csuba.y) - (bsuba.y * csuba.x)\n\n if axis in ['y', 1]:\n return (bsuba.x * csuba.z) - (bsuba.z * csuba.x)\n\n if axis in ['x', 0]:\n return (bsuba.y * csuba.z) - (bsuba.z * csuba.y)\n\n msg = \"invalid axis '{!r}', must be one of {}\".format(axis, self._keys)\n\n raise ValueError(msg)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef isCCW(self, b, c, axis='z'):\n '''\n :b: Point or point equivalent\n :c: Point or point equivalent\n :axis: optional string or integer in set('x',0,'y',1,'z',2)\n :return: boolean\n\n True if the angle determined by a,self,b around 'axis'\n describes a counter-clockwise rotation, otherwise False.\n\n Raises CollinearPoints if self, b, c are collinear.\n '''\n\n result = self.ccw(b, c, axis)\n\n if result == 0:\n raise CollinearPoints(b, self, c)\n\n return result > 0", "response": "Returns True if the angle determined by a self b c around axis is counter - clockwise rotation otherwise False."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef isCollinear(self, b, c):\n '''\n :b: Point or point equivalent\n :c: Point or point equivalent\n :return: boolean\n\n True if 'self' is collinear with 'b' and 'c', otherwise False.\n '''\n\n return all(self.ccw(b, c, axis) == 0 for axis in self._keys)", "response": "Return True if self is collinear with b and c otherwise False."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrotate the current Point around a given origin.", "response": "def rotate2d(self, theta, origin=None, axis='z', radians=False):\n '''\n :theta: float radians to rotate self around origin\n :origin: optional Point, defaults to 0,0,0\n\n Returns a Point rotated by :theta: around :origin:.\n '''\n\n origin = Point._convert(origin)\n\n delta = self - origin\n\n p = Point(origin)\n\n if not radians:\n theta = math.radians(theta)\n\n cosT = math.cos(theta)\n sinT = math.sin(theta)\n\n if axis == 'z':\n p.x += (cosT * delta.x) - (sinT * delta.y)\n p.y += (sinT * delta.x) + (cosT * delta.y)\n return p\n\n if axis == 'y':\n p.z += (cosT * delta.z) - (sinT * delta.x)\n p.x += (sinT * delta.z) + (cosT * delta.x)\n return p\n\n if axis == 'x':\n p.y += (cosT * delta.y) - (sinT * delta.z)\n p.z += (sinT * delta.y) + (cosT * delta.z)\n return p\n\n raise KeyError('unknown axis {}, expecting x, y or z'.format(axis))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef withAngles(cls, origin=None, base=1, alpha=None,\n beta=None, gamma=None, inDegrees=False):\n '''\n :origin: optional Point\n :alpha: optional float describing length of the side opposite A\n :beta: optional float describing length of the side opposite B\n :gamma: optional float describing length of the side opposite C\n :return: Triangle initialized with points comprising the triangle\n with the specified angles.\n '''\n raise NotImplementedError(\"withAngles\")", "response": "Returns a Triangle initialized with points comprising the triangle\n with the specified angles."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef heronsArea(self):\n '''\n Heron's forumla for computing the area of a triangle, float.\n\n Performance note: contains a square root.\n\n '''\n s = self.semiperimeter\n\n return math.sqrt(s * ((s - self.a) * (s - self.b) * (s - self.c)))", "response": "Computes the area of a triangle in the Heron s forumla for computing the area of a triangle."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the radius of the circumcenter of the triangle.", "response": "def circumradius(self):\n '''\n Distance from the circumcenter to all the verticies in\n the Triangle, float.\n\n '''\n return (self.a * self.b * self.c) / (self.area * 4)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef altitudes(self):\n '''\n A list of the altitudes of each vertex [AltA, AltB, AltC], list of\n floats.\n\n An altitude is the shortest distance from a vertex to the side\n opposite of it.\n\n '''\n A = self.area * 2\n\n return [A / self.a, A / self.b, A / self.c]", "response": "A list of the altitudes of each vertex"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntruing iff two side lengths are equal boolean.", "response": "def isIsosceles(self):\n '''\n True iff two side lengths are equal, boolean.\n '''\n return (self.a == self.b) or (self.a == self.c) or (self.b == self.c)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the center point of the ellipse equidistant from foci Point class.", "response": "def center(self):\n '''\n Center point of the ellipse, equidistant from foci, Point class.\\n\n Defaults to the origin.\n '''\n try:\n return self._center\n except AttributeError:\n pass\n self._center = Point()\n return self._center"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the radius of the ellipse Point class.", "response": "def radius(self):\n '''\n Radius of the ellipse, Point class.\n '''\n try:\n return self._radius\n except AttributeError:\n pass\n self._radius = Point(1, 1, 0)\n return self._radius"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if the major axis is parallel to the X axis boolean.", "response": "def xAxisIsMajor(self):\n '''\n Returns True if the major axis is parallel to the X axis, boolean.\n '''\n return max(self.radius.x, self.radius.y) == self.radius.x"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if the minor axis is parallel to the X axis boolean.", "response": "def xAxisIsMinor(self):\n '''\n Returns True if the minor axis is parallel to the X axis, boolean.\n '''\n return min(self.radius.x, self.radius.y) == self.radius.x"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if the major axis is parallel to the Y axis boolean.", "response": "def yAxisIsMajor(self):\n '''\n Returns True if the major axis is parallel to the Y axis, boolean.\n '''\n return max(self.radius.x, self.radius.y) == self.radius.y"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the minor axis is parallel to the Y axis boolean.", "response": "def yAxisIsMinor(self):\n '''\n Returns True if the minor axis is parallel to the Y axis, boolean.\n '''\n return min(self.radius.x, self.radius.y) == self.radius.y"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef a_neg(self):\n '''\n Negative antipodal point on the major axis, Point class.\n\n '''\n na = Point(self.center)\n\n if self.xAxisIsMajor:\n na.x -= self.majorRadius\n else:\n na.y -= self.majorRadius\n return na", "response": "Negative antipodal point on the major axis Point class."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef b_neg(self):\n '''\n Negative antipodal point on the minor axis, Point class.\n '''\n nb = Point(self.center)\n\n if self.xAxisIsMinor:\n nb.x -= self.minorRadius\n else:\n nb.y -= self.minorRadius\n return nb", "response": "Negative antipodal point on the minor axis Point class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the first focus of the ellipse Point class.", "response": "def focus0(self):\n '''\n First focus of the ellipse, Point class.\n\n '''\n f = Point(self.center)\n\n if self.xAxisIsMajor:\n f.x -= self.linearEccentricity\n else:\n f.y -= self.linearEccentricity\n return f"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef circumcircleForTriangle(cls, triangle):\n '''\n :param: triangle - Triangle class\n :return: Circle class\n\n Returns the circle where every vertex in the input triangle is\n on the radius of that circle.\n\n '''\n\n if triangle.isRight:\n # circumcircle origin is the midpoint of the hypotenues\n o = triangle.hypotenuse.midpoint\n r = o.distance(triangle.A)\n return cls(o, r)\n\n # otherwise\n # 1. find the normals to two sides\n # 2. translate them to the midpoints of those two sides\n # 3. intersect those lines for center of circumcircle\n # 4. radius is distance from center to any vertex in the triangle\n\n abn = triangle.AB.normal\n abn += triangle.AB.midpoint\n\n acn = triangle.AC.normal\n acn += triangle.AC.midpoint\n\n o = abn.intersection(acn)\n r = o.distance(triangle.A)\n return cls(o, r)", "response": "Returns the circle that is the center of the circumcircle of the input triangle."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True iff self. center. distance ( other. center ) <= self. radius + other. radius", "response": "def doesIntersect(self, other):\n '''\n :param: other - Circle class\n\n Returns True iff:\n self.center.distance(other.center) <= self.radius+other.radius\n '''\n\n otherType = type(other)\n\n if issubclass(otherType, Ellipse):\n distance = self.center.distance(other.center)\n radiisum = self.radius + other.radius\n return distance <= radiisum\n\n if issubclass(otherType, Line):\n raise NotImplementedError('doesIntersect,other is Line class')\n\n raise TypeError(\"unknown type '{t}'\".format(t=otherType))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a Line normal to this Line.", "response": "def normal(self):\n '''\n :return: Line\n\n Returns a Line normal (perpendicular) to this Line.\n '''\n\n d = self.B - self.A\n\n return Line([-d.y, d.x], [d.y, -d.x])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flip(self):\n '''\n :returns: None\n\n Swaps the positions of A and B.\n '''\n tmp = self.A.xyz\n self.A = self.B\n self.B = tmp", "response": "Flips the positions of A and B."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True iff self and other are both Line subclasses and Line objects.", "response": "def doesIntersect(self, other):\n '''\n :param: other - Line subclass\n :return: boolean\n\n Returns True iff:\n ccw(self.A,self.B,other.A) * ccw(self.A,self.B,other.B) <= 0\n and\n ccw(other.A,other.B,self.A) * ccw(other.A,other.B,self.B) <= 0\n\n '''\n if self.A.ccw(self.B, other.A) * self.A.ccw(self.B, other.B) > 0:\n return False\n\n if other.A.ccw(other.B, self.A) * other.A.ccw(other.B, self.B) > 0:\n return False\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the intersection of two Line objects.", "response": "def intersection(self, other):\n '''\n :param: other - Line subclass\n :return: Point subclass\n\n Returns a Point object with the coordinates of the intersection\n between the current line and the other line.\n\n Will raise Parallel() if the two lines are parallel.\n Will raise Collinear() if the two lines are collinear.\n '''\n\n if self.isCollinear(other):\n msg = '{!r} and {!r} are collinear'\n raise CollinearLines(msg.format(self, other))\n\n d0 = self.A - self.B\n d1 = other.A - other.B\n\n denominator = (d0.x * d1.y) - (d0.y * d1.x)\n\n if denominator == 0:\n msg = '{!r} and {!r} are parallel'\n raise ParallelLines(msg.format(self, other))\n\n cp0 = self.A.cross(self.B)\n cp1 = other.A.cross(other.B)\n\n x_num = (cp0 * d1.x) - (d0.x * cp1)\n y_num = (cp0 * d1.y) - (d0.y * cp1)\n\n p = Point(x_num / denominator, y_num / denominator)\n\n if p in self and p in other:\n return p\n\n msg = \"found point {!r} but not in {!r} and {!r}\"\n raise ParallelLines(msg.format(p, self, other))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the distance from the line to the given point.", "response": "def distanceFromPoint(self, point):\n '''\n :param: point - Point subclass\n :return: float\n\n Distance from the line to the given point.\n '''\n # XXX planar distance, doesn't take into account z ?\n d = self.m\n n = (d.y * point.x) - (d.x * point.y) + self.A.cross(self.B)\n return abs(n / self.A.distance(self.B))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the angle measured between two lines in radians", "response": "def radiansBetween(self, other):\n '''\n :param: other - Line subclass\n :return: float\n\n Returns the angle measured between two lines in radians\n with a range of [0, 2 * math.pi].\n\n '''\n # a dot b = |a||b| * cos(theta)\n # a dot b / |a||b| = cos(theta)\n # cos-1(a dot b / |a||b|) = theta\n\n # translate each line so that it passes through the origin and\n # produce a new point whose distance (magnitude) from the\n # origin is 1.\n #\n\n a = Point.unit(self.A, self.B)\n b = Point.unit(other.A, other.B)\n\n # in a perfect world, after unit: |A| = |B| = 1\n # which is a noop when dividing the dot product of A,B\n # but sometimes the lengths are different.\n #\n # let's just assume things are perfect and the lengths equal 1.\n\n return math.acos(a.dot(b))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef FloatProperty(name, default=0.0, readonly=False, docs=None):\n '''\n :name: string - property name\n :default: float - property default value\n :readonly: boolean - if True, setter method is NOT generated\n\n Returns a property object that can be used to initialize a\n class instance variable as a property.\n '''\n\n private_name = '_' + name\n\n def getf(self):\n if not hasattr(self, private_name):\n setattr(self, private_name, default)\n return getattr(self, private_name)\n\n if readonly:\n setf = None\n else:\n def setf(self, newValue):\n def epsilon_set(v):\n # epsilon_set: creates a float from v unless that\n # float is less than epsilon, which will\n # be considered effectively zero.\n fv = float(v)\n return 0.0 if nearly_zero(fv) else fv\n\n try:\n setattr(self, private_name, epsilon_set(newValue))\n return\n except TypeError:\n pass\n\n if isinstance(newValue, collections.Mapping):\n try:\n setattr(self, private_name, epsilon_set(newValue[name]))\n except KeyError:\n pass\n return\n\n if isinstance(newValue, collections.Iterable):\n try:\n setattr(self, private_name, epsilon_set(newValue[0]))\n return\n except (IndexError, TypeError):\n pass\n\n try:\n mapping = vars(newValue)\n setattr(self, private_name, epsilon_set(mapping[name]))\n return\n except (TypeError, KeyError):\n pass\n\n if newValue is None:\n setattr(self, private_name, epsilon_set(default))\n return\n\n raise ValueError(newValue)\n\n return property(getf, setf, None, docs)", "response": "Returns a property that can be used to initialize a new object variable."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef randomSizeAndLocation(cls, radius, widthLimits,\n heightLimits, origin=None):\n '''\n :param: radius - float\n :param: widthLimits - iterable of floats with length >= 2\n :param: heightLimits - iterable of floats with length >= 2\n :param: origin - optional Point subclass\n :return: Rectangle\n '''\n\n r = cls(widthLimits, heightLimits, origin)\n\n r.origin = Point.randomLocation(radius, origin)", "response": "Returns a Rectangle with random size and location."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef randomSize(cls, widthLimits, heightLimits, origin=None):\n '''\n :param: widthLimits - iterable of integers with length >= 2\n :param: heightLimits - iterable of integers with length >= 2\n :param: origin - optional Point subclass\n :return: Rectangle\n '''\n\n r = cls(0, 0, origin)\n\n r.w = random.randint(widthLimits[0], widthLimits[1])\n r.h = random.randint(heightLimits[0], heightLimits[1])\n\n return r", "response": "Returns a random Rectangle object with width and height limits."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef randomLocation(cls, radius, width, height, origin=None):\n '''\n :param: radius - float\n :param: width - float\n :param: height - float\n :param: origin - optional Point subclass\n :return: Rectangle\n '''\n return cls(width,\n height,\n Point.randomLocation(radius, origin))", "response": "Returns a new Rectangle containing random locations."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the origin of the rectangle.", "response": "def origin(self):\n '''\n Point describing the origin of the rectangle. Defaults to (0,0,0).\n '''\n try:\n return self._origin\n except AttributeError:\n pass\n self._origin = Point()\n return self._origin"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef B(self):\n '''\n Point whose coordinates are (maxX,minY,origin.z), Point.\n '''\n return Point(self.maxX, self.minY, self.origin.z)", "response": "Return a new Point object with the same coordinates as this one."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new Point object that is the center of the current point.", "response": "def C(self):\n '''\n Point whose coordinates are (maxX,maxY,origin.z), Point.\n '''\n return Point(self.maxX, self.maxY, self.origin.z)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the D coordinate of the current Point.", "response": "def D(self):\n '''\n Point whose coordinates are (minX,maxY,origin.Z), Point.\n '''\n return Point(self.minX, self.maxY, self.origin.z)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the center of the current Point.", "response": "def center(self):\n '''\n Point whose coordinates are (midX,midY,origin.z), Point.\n '''\n return Point(self.midX, self.midY, self.origin.z)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nscales the rectangle s width and height by dx and dy.", "response": "def scale(self, dx=1.0, dy=1.0):\n '''\n :param: dx - optional float\n :param: dy - optional float\n\n Scales the rectangle's width and height by dx and dy.\n\n '''\n self.width *= dx\n self.height *= dy"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef containsPoint(self, point, Zorder=False):\n '''\n :param: point - Point subclass\n :param: Zorder - optional Boolean\n\n Is true if the point is contain in the rectangle or\n along the rectangle's edges.\n\n If Zorder is True, the method will check point.z for\n equality with the rectangle origin's Z coordinate.\n\n '''\n if not point.isBetweenX(self.A, self.B):\n return False\n if not point.isBetweenY(self.A, self.D):\n return False\n\n if Zorder:\n return point.z == self.origin.z\n\n return True", "response": "Returns True if the point is contained in the rectangle or False otherwise."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a new Triangle with random coordinates in the circle described by origin and radius.", "response": "def random(cls, origin=None, radius=1):\n '''\n :origin: - optional Point subclass\n :radius: - optional float\n :return: Triangle\n\n Creates a triangle with random coordinates in the circle\n described by (origin,radius). If origin is unspecified, (0,0)\n is assumed. If the radius is unspecified, 1.0 is assumed.\n\n '''\n # XXX no collinearity checks, possible to generate a\n # line (not likely, just possible).\n #\n pts = set()\n while len(pts) < 3:\n p = Point.random(origin, radius)\n pts.add(p)\n return cls(pts)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new avec triangle initialized with points comprising a equilateral triangle.", "response": "def equilateral(cls, origin=None, side=1):\n '''\n :origin: optional Point\n :side: optional float describing triangle side length\n :return: Triangle initialized with points comprising a\n equilateral triangle.\n\n XXX equilateral triangle definition\n\n '''\n o = Point(origin)\n\n base = o.x + side\n h = 0.5 * Sqrt_3 * side + o.y\n \n return cls(o, [base, o.y], [base / 2, h])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new isosceles triangle initialized with points comprising a .", "response": "def isosceles(cls, origin=None, base=1, alpha=90):\n '''\n :origin: optional Point\n :base: optional float describing triangle base length\n :return: Triangle initialized with points comprising a\n isosceles triangle.\n\n XXX isoceles triangle definition\n\n '''\n o = Point(origin)\n base = o.x + base\n\n return cls(o, [base, o.y], [base / 2, o.y + base])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef C(self):\n '''\n Third vertex of triangle, Point subclass.\n\n '''\n try:\n return self._C\n except AttributeError:\n pass\n self._C = Point(0, 1)\n return self._C", "response": "Return the Third vertex of triangle Point subclass."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef AC(self):\n '''\n Vertices A and C, list.\n\n '''\n try:\n return self._AC\n except AttributeError:\n pass\n self._AC = [self.A, self.C]\n return self._AC", "response": "A list of A and C vertices."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef CA(self):\n '''\n Vertices C and A, list.\n\n '''\n try:\n return self._CA\n except AttributeError:\n pass\n self._CA = [self.C, self.A]\n return self._CA", "response": "A list of all possible unique names for the current set of resources."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef CB(self):\n '''\n Vertices C and B, list.\n\n '''\n try:\n return self._CB\n except AttributeError:\n pass\n self._CB = [self.C, self.B]\n return self._CB", "response": "Returns a list of the n - ary items that are in the order of the two sets."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef segments(self):\n '''\n A list of the Triangle's line segments [AB, BC, AC], list.\n\n '''\n return [Segment(self.AB),\n Segment(self.BC),\n Segment(self.AC)]", "response": "A list of the Triangle s line segments [ AB BC AC ]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef circumcenter(self):\n '''\n The intersection of the median perpendicular bisectors, Point.\n\n The center of the circumscribed circle, which is the circle that\n passes through all vertices of the triangle.\n\n https://en.wikipedia.org/wiki/Circumscribed_circle#Cartesian_coordinates_2\n\n BUG: only finds the circumcenter in the XY plane\n '''\n\n if self.isRight:\n return self.hypotenuse.midpoint\n\n if self.A.isOrigin:\n t = self\n else:\n # translate triangle to origin\n t = Triangle(self.A - self.A, self.B - self.A, self.C - self.A)\n # XXX translation would be easier by defining add and sub for points\n # t = self - self.A \n\n if not t.A.isOrigin:\n raise ValueError('failed to translate {} to origin'.format(t))\n\n BmulC = t.B * t.C.yx\n\n d = 2 * (BmulC.x - BmulC.y)\n\n bSqSum = sum((t.B ** 2).xy)\n cSqSum = sum((t.C ** 2).xy)\n\n x = (((t.C.y * bSqSum) - (t.B.y * cSqSum)) / d) + self.A.x\n y = (((t.B.x * cSqSum) - (t.C.x * bSqSum)) / d) + self.A.y\n\n return Point(x, y)", "response": "Return the center of the circumscribed circle."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntrue if all sides of the triangle are the same length.", "response": "def isEquilateral(self):\n '''\n True if all sides of the triangle are the same length.\n\n All equilateral triangles are also isosceles.\n All equilateral triangles are also acute.\n\n '''\n if not nearly_eq(self.a, self.b):\n return False\n\n if not nearly_eq(self.b, self.c):\n return False\n\n return nearly_eq(self.a, self.c)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a new Triangle with flipped side.", "response": "def swap(self, side='AB', inplace=False):\n '''\n :side: - optional string\n :inplace: - optional boolean\n :return: Triangle with flipped side.\n\n The optional side paramater should have one of three values:\n AB, BC, or AC.\n\n Changes the order of the triangle's points, swapping the\n specified points. Doing so will change the results of isCCW\n and ccw.\n\n '''\n try:\n flipset = {'AB': (self.B.xyz, self.A.xyz, self.C.xyz),\n 'BC': (self.A.xyz, self.C.xyz, self.B.xyz),\n 'AC': (self.C.xyz, self.B.xyz, self.A.xyz)}[side]\n except KeyError as e:\n raise KeyError(str(e))\n\n if inplace:\n self.ABC = flipset\n return self\n\n return Triangle(flipset)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True iff any of the segments in self intersects any segment in other.", "response": "def doesIntersect(self, other):\n '''\n :param: other - Triangle or Line subclass\n :return: boolean\n\n Returns True iff:\n Any segment in self intersects any segment in other.\n\n '''\n otherType = type(other)\n\n if issubclass(otherType, Triangle):\n for s in self.segments.values():\n for q in other.segments.values():\n if s.doesIntersect(q):\n return True\n return False\n\n if issubclass(otherType, Line):\n for s in self.segments.values():\n if s.doesIntersect(other):\n return True\n return False\n\n msg = \"expecting Line or Triangle subclasses, got '{}'\"\n\n raise TypeError(msg.format(otherType))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef perimeter(self):\n '''\n Sum of the length of all sides, float.\n '''\n return sum([a.distance(b) for a, b in self.pairs()])", "response": "Returns the total length of all sides of the current set."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef vl_dsift(data, fast=False, norm=False, bounds=None, size=3, step=1,\n window_size=None, float_descriptors=False,\n verbose=False, matlab_style=True):\n '''\n Dense sift descriptors from an image.\n\n Returns:\n frames: num_frames x (2 or 3) matrix of x, y, (norm)\n descrs: num_frames x 128 matrix of descriptors\n '''\n if not matlab_style:\n import warnings\n warnings.warn(\"matlab_style=False gets different results than matlab, \"\n \"not sure why or how incorrect they are.\")\n\n order = 'F' if matlab_style else 'C'\n data = as_float_image(data, dtype=np.float32, order=order)\n if data.ndim != 2:\n raise TypeError(\"data should be a 2d array\")\n\n if window_size is not None:\n assert np.isscalar(window_size) and window_size >= 0\n\n # construct the dsift object\n M, N = data.shape\n dsift_p = vl_dsift_new_basic(M, N, step, size)\n\n try:\n dsift = dsift_p.contents\n\n # set parameters\n if bounds is not None:\n if matlab_style:\n y0, x0, y1, x1 = bounds # transposed\n else:\n x0, y0, x1, y1 = bounds\n dsift.boundMinX = int(max(x0, 0))\n dsift.boundMinY = int(max(y0, 0))\n dsift.boundMaxX = int(min(x1, M - 1))\n dsift.boundMaxY = int(min(y1, N - 1))\n _vl_dsift_update_buffers(dsift_p)\n\n dsift.useFlatWindow = fast\n\n if window_size is not None:\n dsift.windowSize = window_size\n\n # get calculated parameters\n descr_size = dsift.descrSize\n num_frames = dsift.numFrames\n geom = dsift.geom\n\n if verbose:\n pr = lambda *a, **k: print('vl_dsift:', *a, **k)\n pr(\"image size [W, H] = [{}, {}]\".format(N, M))\n x0 = dsift.boundMinX + 1\n y0 = dsift.boundMinY + 1\n x1 = dsift.boundMaxX + 1\n y1 = dsift.boundMaxY + 1\n bound_args = [y0, x0, y1, x1] if matlab_style else [x0, y0, x1, y1]\n pr(\"bounds: [minX,minY,maxX,maxY] = [{}, {}, {}, {}]\"\n .format(*bound_args))\n pr(\"subsampling steps: stepX={}, stepY={}\".format(\n dsift.stepX, dsift.stepY))\n pr(\"num bins: [numBinT, numBinX, numBinY] = [{}, {}, {}]\"\n .format(geom.numBinT, geom.numBinX, geom.numBinY))\n pr(\"descriptor size: {}\".format(descr_size))\n pr(\"bin sizes: [binSizeX, binSizeY] = [{}, {}]\".format(\n geom.binSizeX, geom.binSizeY))\n pr(\"flat window: {}\".format(bool(fast)))\n pr(\"window size: {}\".format(dsift.windowSize))\n pr(\"num of features: {}\".format(num_frames))\n\n # do the actual processing\n vl_dsift_process(dsift_p, data)\n\n # copy frames' locations, norms out\n # the frames are a structure of just 4 doubles (VLDsiftKeypoint),\n # which luckily looks exactly like an array of doubles. :)\n # NOTE: this might be platform/compiler-dependent...but it works with\n # the provided binaries on os x, at least\n frames_p = cast(dsift.frames, c_double_p)\n frames_p_a = npc.as_array(frames_p, shape=(num_frames, 4))\n cols = [1, 0] if matlab_style else [0, 1]\n if norm:\n cols.append(3)\n frames = np.require(frames_p_a[:, cols], requirements=['C', 'O'])\n\n # copy descriptors into a new array\n descrs_p = npc.as_array(dsift.descrs, shape=(num_frames, descr_size))\n descrs = descrs_p * 512\n assert descrs.flags.owndata\n np.minimum(descrs, 255, out=descrs)\n if not float_descriptors:\n descrs = descrs.astype(np.uint8) # TODO: smarter about copying?\n if matlab_style:\n new_order = np.empty(descr_size, dtype=int)\n vl_dsift_transpose_descriptor(new_order, np.arange(descr_size),\n geom.numBinT, geom.numBinX, geom.numBinY)\n descrs = descrs[:, new_order]\n # the old, super-slow way:\n ## # gross pointer arithmetic to get the relevant descriptor\n ## descrs_addr = addressof(descrs.contents)\n ## descrs_step = descr_size * sizeof(c_float)\n ##\n ## for k in range(num_frames):\n ## out_frames[:2, k] = [frames[k].y + 1, frames[k].x + 1]\n ## if norm: # there's an implied / 2 in norm, because of clipping\n ## out_frames[2, k] = frames[k].norm\n ##\n ## # gross pointer arithmetic to get the relevant descriptor\n ## the_descr = cast(descrs_addr + k * descrs_step, c_float_p)\n ## transposed = vl_dsift_transpose_descriptor(\n ## the_descr,\n ## geom.numBinT, geom.numBinX, geom.numBinY)\n ## out_descrs[:, k] = np.minimum(512. * transposed, 255.)\n\n return frames, descrs\n\n finally:\n vl_dsift_delete(dsift_p)", "response": "Create a new DSIFT object from an image."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rgb2gray(img):\n T = np.linalg.inv(np.array([\n [1.0, 0.956, 0.621],\n [1.0, -0.272, -0.647],\n [1.0, -1.106, 1.703],\n ]))\n r_c, g_c, b_c = T[0]\n r, g, b = np.rollaxis(as_float_image(img), axis=-1)\n return r_c * r + g_c * g + b_c * b", "response": "Converts an RGB image to grayscale using matlab s algorithm."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rgb2hsv(arr):\n arr = np.asanyarray(arr)\n if arr.ndim != 3 or arr.shape[2] != 3:\n raise ValueError(\"the input array must have a shape == (.,.,3)\")\n arr = as_float_image(arr)\n\n out = np.empty_like(arr)\n\n # -- V channel\n out_v = arr.max(-1)\n\n # -- S channel\n delta = arr.ptp(-1)\n # Ignore warning for zero divided by zero\n old_settings = np.seterr(invalid='ignore')\n out_s = delta / out_v\n out_s[delta == 0.] = 0.\n\n # -- H channel\n # red is max\n idx = (arr[:, :, 0] == out_v)\n out[idx, 0] = (arr[idx, 1] - arr[idx, 2]) / delta[idx]\n\n # green is max\n idx = (arr[:, :, 1] == out_v)\n out[idx, 0] = 2. + (arr[idx, 2] - arr[idx, 0]) / delta[idx]\n\n # blue is max\n idx = (arr[:, :, 2] == out_v)\n out[idx, 0] = 4. + (arr[idx, 0] - arr[idx, 1]) / delta[idx]\n out_h = (out[:, :, 0] / 6.) % 1.\n out_h[delta == 0.] = 0.\n\n np.seterr(**old_settings)\n\n # -- output\n out[:, :, 0] = out_h\n out[:, :, 1] = out_s\n out[:, :, 2] = out_v\n\n # remove NaN\n out[np.isnan(out)] = 0\n return out", "response": "Converts an RGB image to HSV using scikit - image s algorithm."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse an SCI command response into an ElementTree Element", "response": "def _parse_command_response(response):\n \"\"\"Parse an SCI command response into ElementTree XML\n\n This is a helper method that takes a Requests Response object\n of an SCI command response and will parse it into an ElementTree Element\n representing the root of the XML response.\n\n :param response: The requests response object\n :return: An ElementTree Element that is the root of the response XML\n :raises ResponseParseError: If the response XML is not well formed\n \"\"\"\n try:\n root = ET.fromstring(response.text)\n except ET.ParseError:\n raise ResponseParseError(\n \"Unexpected response format, could not parse XML. Response: {}\".format(response.text))\n\n return root"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_error_tree(error):\n errinf = ErrorInfo(error.get('id'), None)\n if error.text is not None:\n errinf.message = error.text\n else:\n desc = error.find('./desc')\n if desc is not None:\n errinf.message = desc.text\n return errinf", "response": "Parse an error ElementTree Node to create an ErrorInfo object"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_data(self):\n\n target = DeviceTarget(self.device_id)\n return self._fssapi.get_file(target, self.path)[self.device_id]", "response": "Get the contents of this file\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef delete(self):\n target = DeviceTarget(self.device_id)\n return self._fssapi.delete_file(target, self.path)[self.device_id]", "response": "Delete this file from the device"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlisting the contents of this directory and return a LsInfo object that contains directories and files that are not yet modified.", "response": "def list_contents(self):\n \"\"\"List the contents of this directory\n\n :return: A LsInfo object that contains directories and files\n :rtype: :class:`~.LsInfo` or :class:`~.ErrorInfo`\n\n Here is an example usage::\n\n # let dirinfo be a DirectoryInfo object\n ldata = dirinfo.list_contents()\n if isinstance(ldata, ErrorInfo):\n # Do some error handling\n logger.warn(\"Error listing file info: (%s) %s\", ldata.errno, ldata.message)\n # It's of type LsInfo\n else:\n # Look at all the files\n for finfo in ldata.files:\n logger.info(\"Found file %s of size %s\", finfo.path, finfo.size)\n # Look at all the directories\n for dinfo in ldata.directories:\n logger.info(\"Found directory %s of last modified %s\", dinfo.path, dinfo.last_modified)\n \"\"\"\n target = DeviceTarget(self.device_id)\n return self._fssapi.list_files(target, self.path)[self.device_id]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the server response for this ls command and return a list of directories and files on the device.", "response": "def parse_response(cls, response, device_id=None, fssapi=None, **kwargs):\n \"\"\"Parse the server response for this ls command\n\n This will parse xml of the following form::\n\n \n \n ...\n \n ...\n \n\n or with an error::\n\n \n \n \n\n :param response: The XML root of the response for an ls command\n :type response: :class:`xml.etree.ElementTree.Element`\n :param device_id: The device id of the device this ls response came from\n :param fssapi: A reference to a :class:`~FileSystemServiceAPI` for use with the\n :class:`~FileInfo` and :class:`~DirectoryInfo` objects for future commands\n :return: An :class:`~LsInfo` object containing the list of directories and files on\n the device or an :class:`~ErrorInfo` if the xml contained an error\n \"\"\"\n if response.tag != cls.command_name:\n raise ResponseParseError(\n \"Received response of type {}, LsCommand can only parse responses of type {}\".format(response.tag,\n cls.command_name))\n\n if fssapi is None:\n raise FileSystemServiceException(\"fssapi is required to parse an LsCommand response\")\n if device_id is None:\n raise FileSystemServiceException(\"device_id is required to parse an LsCommand response\")\n\n error = response.find('./error')\n if error is not None:\n return _parse_error_tree(error)\n\n hash_type = response.get('hash')\n dirs = []\n files = []\n\n # Get each file listed in this response\n for myfile in response.findall('./file'):\n fi = FileInfo(fssapi,\n device_id,\n myfile.get('path'),\n int(myfile.get('last_modified')),\n int(myfile.get('size')),\n myfile.get('hash'),\n hash_type)\n files.append(fi)\n # Get each directory listed for this device\n for mydir in response.findall('./dir'):\n di = DirectoryInfo(fssapi,\n device_id,\n mydir.get('path'),\n int(mydir.get('last_modified')))\n dirs.append(di)\n return LsInfo(directories=dirs, files=files)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_response(cls, response, **kwargs):\n if response.tag != cls.command_name:\n raise ResponseParseError(\n \"Received response of type {}, GetCommand can only parse responses of type {}\".format(response.tag,\n cls.command_name))\n\n error = response.find('./error')\n if error is not None:\n return _parse_error_tree(error)\n\n text = response.find('./data').text\n if text:\n return base64.b64decode(six.b(text))\n else:\n return six.b('')", "response": "Parse the server response for this get file command\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_response(cls, response, **kwargs):\n if response.tag != cls.command_name:\n raise ResponseParseError(\n \"Received response of type {}, PutCommand can only parse responses of type {}\".format(response.tag,\n cls.command_name))\n error = response.find('./error')\n if error is not None:\n return _parse_error_tree(error)\n\n return None", "response": "Parse the server response for this put file command and return the corresponding object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_command_block(self, target, command_block):\n root = _parse_command_response(\n self._sci_api.send_sci(\"file_system\", target, command_block.get_command_string()))\n\n out_dict = {}\n for device in root.findall('./file_system/device'):\n device_id = device.get('id')\n results = []\n for command in device.find('./commands'):\n for command_class in FILE_SYSTEM_COMMANDS:\n if command_class.command_name == command.tag.lower():\n results.append(command_class.parse_response(command, fssapi=self, device_id=device_id))\n out_dict[device_id] = results\n return out_dict", "response": "Send an arbitrary file system command block to a target."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef list_files(self, target, path, hash='any'):\n command_block = FileSystemServiceCommandBlock()\n command_block.add_command(LsCommand(path, hash=hash))\n root = _parse_command_response(\n self._sci_api.send_sci(\"file_system\", target, command_block.get_command_string()))\n\n out_dict = {}\n\n # At this point the XML we have is of the form\n # \n # \n # \n # \n # \n # \n # ...\n # \n # ...\n # \n # \n # \n # \n # \n # \n # \n # ...\n # \n # ...\n # \n # \n # \n # ...\n # \n # \n\n # Here we will get each of the XML trees rooted at the device nodes\n for device in root.findall('./file_system/device'):\n device_id = device.get('id')\n error = device.find('./error')\n if error is not None:\n out_dict[device_id] = _parse_error_tree(error)\n else:\n linfo = LsCommand.parse_response(device.find('./commands/ls'), device_id=device_id, fssapi=self)\n out_dict[device_id] = linfo\n return out_dict", "response": "List all files and directories on the target and return a dictionary containing the files and directories and directories and error messages."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_file(self, target, path, offset=None, length=None):\n command_block = FileSystemServiceCommandBlock()\n command_block.add_command(GetCommand(path, offset, length))\n root = _parse_command_response(\n self._sci_api.send_sci(\"file_system\", target, command_block.get_command_string()))\n out_dict = {}\n for device in root.findall('./file_system/device'):\n device_id = device.get('id')\n error = device.find('./error')\n if error is not None:\n out_dict[device_id] = _parse_error_tree(error)\n else:\n data = GetCommand.parse_response(device.find('./commands/get_file'))\n out_dict[device_id] = data\n return out_dict", "response": "Retrieves the contents of a file on the device."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nputting data into a file on the devicecloud server.", "response": "def put_file(self, target, path, file_data=None, server_file=None, offset=None, truncate=False):\n \"\"\"Put data into a file on the device\n\n :param target: The device(s) to be targeted with this request\n :type target: :class:`devicecloud.sci.TargetABC` or list of :class:`devicecloud.sci.TargetABC` instances\n :param path: The path on the target to the file to write to. If the file already exists it will be overwritten.\n :param file_data: A `six.binary_type` containing the data to put into the file\n :param server_file: The path to a file on the devicecloud server containing the data to put into the file on the\n device\n :param offset: Start writing bytes to the file at this position, if None start at the beginning\n :param truncate: Boolean, if True after bytes are done being written end the file their even if previous data\n exists beyond it. If False, leave any existing data in place.\n :return: A dictionary with keys being device ids and value being None if successful or an :class:`~.ErrorInfo`\n if the operation failed on that device\n :raises: :class:`~.FileSystemServiceException` if either both file_data and server_file are specified or\n neither are specified\n :raises: :class:`~.ResponseParseError` If the SCI response has unrecognized formatting\n \"\"\"\n\n command_block = FileSystemServiceCommandBlock()\n command_block.add_command(PutCommand(path, file_data, server_file, offset, truncate))\n\n root = _parse_command_response(self._sci_api.send_sci(\"file_system\", target, command_block.get_command_string()))\n out_dict = {}\n for device in root.findall('./file_system/device'):\n device_id = device.get('id')\n error = device.find('./error')\n if error is not None:\n out_dict[device_id] = _parse_error_tree(error)\n else:\n out_dict[device_id] = PutCommand.parse_response(device.find('./commands/put_file'))\n\n return out_dict"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndeleting a file from a specific target.", "response": "def delete_file(self, target, path):\n \"\"\"Delete a file from a device\n\n :param target: The device(s) to be targeted with this request\n :type target: :class:`devicecloud.sci.TargetABC` or list of :class:`devicecloud.sci.TargetABC` instances\n :param path: The path on the target to the file to delete.\n :return: A dictionary with keys being device ids and value being None if successful or an :class:`~.ErrorInfo`\n if the operation failed on that device\n :raises: :class:`~.ResponseParseError` If the SCI response has unrecognized formatting\n \"\"\"\n command_block = FileSystemServiceCommandBlock()\n command_block.add_command(DeleteCommand(path))\n root = _parse_command_response(self._sci_api.send_sci(\"file_system\", target, command_block.get_command_string()))\n\n out_dict = {}\n for device in root.findall('./file_system/device'):\n device_id = device.get('id')\n error = device.find('./error')\n if error is not None:\n out_dict[device_id] = _parse_error_tree(error)\n else:\n out_dict[device_id] = DeleteCommand.parse_response(device.find('./commands/rm'))\n return out_dict"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget all files and directories that have been modified since a given time.", "response": "def get_modified_items(self, target, path, last_modified_cutoff):\n \"\"\"Get all files and directories from a path on the device modified since a given time\n\n :param target: The device(s) to be targeted with this request\n :type target: :class:`devicecloud.sci.TargetABC` or list of :class:`devicecloud.sci.TargetABC` instances\n :param path: The path on the target to the directory to check for modified files.\n :param last_modified_cutoff: The time (as Unix epoch time) to get files modified since\n :type last_modified_cutoff: int\n :return: A dictionary where the key is a device id and the value is either an :class:`~.ErrorInfo` if there\n was a problem with the operation or a :class:`~.LsInfo` with the items modified since the\n specified date\n \"\"\"\n file_list = self.list_files(target, path)\n out_dict = {}\n for device_id, device_data in six.iteritems(file_list):\n if isinstance(device_data, ErrorInfo):\n out_dict[device_id] = device_data\n else:\n files = []\n dirs = []\n for cur_file in device_data.files:\n if cur_file.last_modified > last_modified_cutoff:\n files.append(cur_file)\n\n for cur_dir in device_data.directories:\n if cur_dir.last_modified > last_modified_cutoff:\n dirs.append(cur_dir)\n out_dict[device_id] = LsInfo(directories=dirs, files=files)\n\n return out_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks if a path refers to an existing path on the devicecloud", "response": "def exists(self, target, path, path_sep=\"/\"):\n \"\"\"Check if path refers to an existing path on the device\n\n :param target: The device(s) to be targeted with this request\n :type target: :class:`devicecloud.sci.TargetABC` or list of :class:`devicecloud.sci.TargetABC` instances\n :param path: The path on the target to check for existence.\n :param path_sep: The path separator of the device\n :return: A dictionary where the key is a device id and the value is either an :class:`~.ErrorInfo` if there\n was a problem with the operation or a boolean with the existence status of the path on that device\n \"\"\"\n if path.endswith(path_sep):\n path = path[:-len(path_sep)]\n par_dir, filename = path.rsplit(path_sep, 1)\n file_list = self.list_files(target, par_dir)\n out_dict = {}\n for device_id, device_data in six.iteritems(file_list):\n if isinstance(device_data, ErrorInfo):\n out_dict[device_id] = device_data\n else:\n out_dict[device_id] = False\n for cur_file in device_data.files:\n if cur_file.path == path:\n out_dict[device_id] = True\n for cur_dir in device_data.directories:\n if cur_dir.path == path:\n out_dict[device_id] = True\n\n return out_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_devices(self, condition=None, page_size=1000):\n\n condition = validate_type(condition, type(None), Expression, *six.string_types)\n page_size = validate_type(page_size, *six.integer_types)\n\n params = {\"embed\": \"true\"}\n if condition is not None:\n params[\"condition\"] = condition.compile()\n\n for device_json in self._conn.iter_json_pages(\"/ws/DeviceCore\", page_size=page_size, **params):\n yield Device(self._conn, self._sci, device_json)", "response": "Get all devices in this cloud account that match the condition."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_group_tree_root(self, page_size=1000):\n\n # first pass, build mapping\n group_map = {} # map id -> group\n page_size = validate_type(page_size, *six.integer_types)\n for group in self.get_groups(page_size=page_size):\n group_map[group.get_id()] = group\n\n # second pass, find root and populate list of children for each node\n root = None\n for group_id, group in group_map.items():\n if group.is_root():\n root = group\n else:\n parent = group_map[group.get_parent_id()]\n parent.add_child(group)\n return root", "response": "Return the root group for this accounts group tree."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_groups(self, condition=None, page_size=1000):\n query_kwargs = {}\n if condition is not None:\n query_kwargs[\"condition\"] = condition.compile()\n for group_data in self._conn.iter_json_pages(\"/ws/Group\", page_size=page_size, **query_kwargs):\n yield Group.from_json(group_data)", "response": "Returns an iterator over all groups in this device cloud account."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprovisioning multiple devices with a single API call", "response": "def provision_devices(self, devices):\n \"\"\"Provision multiple devices with a single API call\n\n This method takes an iterable of dictionaries where the values in the dictionary are\n expected to match the arguments of a call to :meth:`provision_device`. The\n contents of each dictionary will be validated.\n\n :param list devices: An iterable of dictionaries each containing information about\n a device to be provision. The form of the dictionary should match the keyword\n arguments taken by :meth:`provision_device`.\n :raises DeviceCloudHttpException: If there is an unexpected error reported by Device Cloud.\n :raises ValueError: If any input fields are known to have a bad form.\n :return: A list of dictionaries in the form described for :meth:`provision_device` in the\n order matching the requested device list. Note that it is possible for there to\n be mixed success and error when provisioning multiple devices.\n\n \"\"\"\n # Validate all the input for each device provided\n sio = six.StringIO()\n\n def write_tag(tag, val):\n sio.write(\"<{tag}>{val}\".format(tag=tag, val=val))\n\n def maybe_write_element(tag, val):\n if val is not None:\n write_tag(tag, val)\n return True\n return False\n\n sio.write(\"\")\n for d in devices:\n sio.write(\"\")\n\n mac_address = d.get(\"mac_address\")\n device_id = d.get(\"device_id\")\n imei = d.get(\"imei\")\n if mac_address is not None:\n write_tag(\"devMac\", mac_address)\n elif device_id is not None:\n write_tag(\"devConnectwareId\", device_id)\n elif imei is not None:\n write_tag(\"devCellularModemId\", imei)\n else:\n raise ValueError(\"mac_address, device_id, or imei must be provided for device %r\" % d)\n\n # Write optional elements if present.\n maybe_write_element(\"grpPath\", d.get(\"group_path\"))\n maybe_write_element(\"dpUserMetaData\", d.get(\"metadata\"))\n maybe_write_element(\"dpTags\", d.get(\"tags\"))\n maybe_write_element(\"dpMapLong\", d.get(\"map_long\"))\n maybe_write_element(\"dpMapLat\", d.get(\"map_lat\"))\n maybe_write_element(\"dpContact\", d.get(\"contact\"))\n maybe_write_element(\"dpDescription\", d.get(\"description\"))\n\n sio.write(\"\")\n sio.write(\"\")\n\n # Send the request, set the Accept XML as a nicety\n results = []\n response = self._conn.post(\"/ws/DeviceCore\", sio.getvalue(), headers={'Accept': 'application/xml'})\n root = ET.fromstring(response.content) # tag is root of response\n for child in root:\n if child.tag.lower() == \"location\":\n results.append({\n \"error\": False,\n \"error_msg\": None,\n \"location\": child.text\n })\n else: # we expect \"error\" but handle generically\n results.append({\n \"error\": True,\n \"location\": None,\n \"error_msg\": child.text\n })\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef from_json(cls, json_data):\n # Example Data:\n # { \"grpId\": \"11817\", \"grpName\": \"7603_Digi\", \"grpDescription\": \"7603_Digi root group\",\n # \"grpPath\": \"\\/7603_Digi\\/\", \"grpParentId\": \"1\"}\n return cls(\n group_id=json_data[\"grpId\"],\n name=json_data[\"grpName\"],\n description=json_data.get(\"grpDescription\", \"\"),\n path=json_data[\"grpPath\"],\n parent_id=json_data[\"grpParentId\"],\n )", "response": "Build and return a new Group object from json data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef print_subtree(self, fobj=sys.stdout, level=0):\n fobj.write(\"{}{!r}\\n\".format(\" \" * (level * 2), self))\n for child in self.get_children():\n child.print_subtree(fobj, level + 1)", "response": "Print this group node and the subtree rooted at it"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_device_json(self, use_cached=True):\n if not use_cached:\n devicecore_data = self._conn.get_json(\n \"/ws/DeviceCore/{}\".format(self.get_device_id()))\n self._device_json = devicecore_data[\"items\"][0] # should only be 1\n return self._device_json", "response": "Get the JSON metadata for this device as a python data structure."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_tags(self, use_cached=True):\n device_json = self.get_device_json(use_cached)\n potential_tags = device_json.get(\"dpTags\")\n if potential_tags:\n return list(filter(None, potential_tags.split(\",\")))\n else:\n return []", "response": "Get the list of tags for this device"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if the device is currrently connect and False if not.", "response": "def is_connected(self, use_cached=True):\n \"\"\"Return True if the device is currrently connect and False if not\"\"\"\n device_json = self.get_device_json(use_cached)\n return int(device_json.get(\"dpConnectionStatus\")) > 0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_connectware_id(self, use_cached=True):\n device_json = self.get_device_json(use_cached)\n return device_json.get(\"devConnectwareId\")", "response": "Get the connectware id of this device"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets this device s device id", "response": "def get_device_id(self, use_cached=True):\n \"\"\"Get this device's device id\"\"\"\n device_json = self.get_device_json(use_cached)\n return device_json[\"id\"].get(\"devId\")"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the last known IP of this device", "response": "def get_ip(self, use_cached=True):\n \"\"\"Get the last known IP of this device\"\"\"\n device_json = self.get_device_json(use_cached)\n return device_json.get(\"dpLastKnownIp\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the MAC address of this device", "response": "def get_mac(self, use_cached=True):\n \"\"\"Get the MAC address of this device\"\"\"\n device_json = self.get_device_json(use_cached)\n return device_json.get(\"devMac\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the last 4 characters in the device mac address hex", "response": "def get_mac_last4(self, use_cached=True):\n \"\"\"Get the last 4 characters in the device mac address hex (e.g. 00:40:9D:58:17:5B -> 175B)\n\n This is useful for use as a short reference to the device. It is not guaranteed to\n be unique (obviously) but will often be if you don't have too many devices.\n\n \"\"\"\n chunks = self.get_mac(use_cached).split(\":\")\n mac4 = \"%s%s\" % (chunks[-2], chunks[-1])\n return mac4.upper()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_registration_dt(self, use_cached=True):\n device_json = self.get_device_json(use_cached)\n start_date_iso8601 = device_json.get(\"devRecordStartDate\")\n if start_date_iso8601:\n return iso8601_to_dt(start_date_iso8601)\n else:\n return None", "response": "Get the datetime of when this device was added to Device Cloud"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_latlon(self, use_cached=True):\n device_json = self.get_device_json(use_cached)\n lat = device_json.get(\"dpMapLat\")\n lon = device_json.get(\"dpMapLong\")\n return (float(lat) if lat else None,\n float(lon) if lon else None, )", "response": "Get a tuple with device latitude and longitude... these may be None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_to_group(self, group_path):\n\n if self.get_group_path() != group_path:\n post_data = ADD_GROUP_TEMPLATE.format(connectware_id=self.get_connectware_id(),\n group_path=group_path)\n self._conn.put('/ws/DeviceCore', post_data)\n\n # Invalidate cache\n self._device_json = None", "response": "Add a device to a group"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a tag to existing device tags. This method will not add a duplicate, if already in the list. :param new_tags: the tag(s) to be added. new_tags can be a comma-separated string or list", "response": "def add_tag(self, new_tags):\n \"\"\"Add a tag to existing device tags. This method will not add a duplicate, if already in the list.\n\n :param new_tags: the tag(s) to be added. new_tags can be a comma-separated string or list\n \"\"\"\n\n tags = self.get_tags()\n orig_tag_cnt = len(tags)\n # print(\"self.get_tags() {}\".format(tags))\n\n if isinstance(new_tags, six.string_types):\n new_tags = new_tags.split(',')\n # print(\"spliting tags :: {}\".format(new_tags))\n\n for tag in new_tags:\n if not tag in tags:\n tags.append(tag.strip())\n\n if len(tags) > orig_tag_cnt:\n xml_tags = escape(\",\".join(tags))\n post_data = TAGS_TEMPLATE.format(connectware_id=self.get_connectware_id(),\n tags=xml_tags)\n self._conn.put('/ws/DeviceCore', post_data)\n\n # Invalidate cache\n self._device_json = None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove a tag from existing device tags", "response": "def remove_tag(self, tag):\n \"\"\"Remove tag from existing device tags\n\n :param tag: the tag to be removed from the list\n\n :raises ValueError: If tag does not exist in list\n \"\"\"\n\n tags = self.get_tags()\n tags.remove(tag)\n\n post_data = TAGS_TEMPLATE.format(connectware_id=self.get_connectware_id(),\n tags=escape(\",\".join(tags)))\n self._conn.put('/ws/DeviceCore', post_data)\n\n # Invalidate cache\n self._device_json = None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the hostname that this connection is associated with", "response": "def hostname(self):\n \"\"\"Get the hostname that this connection is associated with\"\"\"\n from six.moves.urllib.parse import urlparse\n return urlparse(self._base_url).netloc.split(':', 1)[0]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef iter_json_pages(self, path, page_size=1000, **params):\n path = validate_type(path, *six.string_types)\n page_size = validate_type(page_size, *six.integer_types)\n\n offset = 0\n remaining_size = 1 # just needs to be non-zero\n while remaining_size > 0:\n reqparams = {\"start\": offset, \"size\": page_size}\n reqparams.update(params)\n response = self.get_json(path, params=reqparams)\n offset += page_size\n remaining_size = int(response.get(\"remainingSize\", \"0\"))\n for item_json in response.get(\"items\", []):\n yield item_json", "response": "Return an iterator over JSON items from a paginated resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get(self, path, **kwargs):\n url = self._make_url(path)\n return self._make_request(\"GET\", url, **kwargs)", "response": "Perform an HTTP GET request against Device Cloud with this accounts\n credentials and base url."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_json(self, path, **kwargs):\n\n url = self._make_url(path)\n headers = kwargs.setdefault('headers', {})\n headers.update({'Accept': 'application/json'})\n response = self._make_request(\"GET\", url, **kwargs)\n return json.loads(response.text)", "response": "Perform an HTTP GET request with JSON headers of the specified path against Device Cloud with this accounts\n credentials and base url."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform an HTTP POST request against Device Cloud with this accounts credentials and base url.", "response": "def post(self, path, data, **kwargs):\n \"\"\"Perform an HTTP POST request of the specified path in Device Cloud\n\n Make an HTTP POST request against Device Cloud with this accounts\n credentials and base url. This method uses the\n `requests `_ library\n `request method `_\n and all keyword arguments will be passed on to that method.\n\n :param str path: Device Cloud path to POST\n :param int retries: The number of times the request should be retried if an\n unsuccessful response is received. Most likely, you should leave this at 0.\n :param data: The data to be posted in the body of the POST request (see docs for\n ``requests.post``\n :raises DeviceCloudHttpException: if a non-success response to the request is received\n from Device Cloud\n :returns: A requests ``Response`` object\n\n \"\"\"\n url = self._make_url(path)\n return self._make_request(\"POST\", url, data=data, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef put(self, path, data, **kwargs):\n\n url = self._make_url(path)\n return self._make_request(\"PUT\", url, data=data, **kwargs)", "response": "Perform an HTTP PUT request against Device Cloud with this accounts\n credentials and base url."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nperform an HTTP DELETE request against Device Cloud with this accounts credentials and base url.", "response": "def delete(self, path, retries=DEFAULT_THROTTLE_RETRIES, **kwargs):\n \"\"\"Perform an HTTP DELETE request of the specified path in Device Cloud\n\n Make an HTTP DELETE request against Device Cloud with this accounts\n credentials and base url. This method uses the\n `requests `_ library\n `request method `_\n and all keyword arguments will be passed on to that method.\n\n :param str path: Device Cloud path to DELETE\n :param int retries: The number of times the request should be retried if an\n unsuccessful response is received. Most likely, you should leave this at 0.\n :raises DeviceCloudHttpException: if a non-success response to the request is received\n from Device Cloud\n :returns: A requests ``Response`` object\n\n \"\"\"\n url = self._make_url(path)\n return self._make_request(\"DELETE\", url, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef streams(self):\n if self._streams_api is None:\n self._streams_api = self.get_streams_api()\n return self._streams_api", "response": "Property providing access to the streams API"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef filedata(self):\n if self._filedata_api is None:\n self._filedata_api = self.get_filedata_api()\n return self._filedata_api", "response": "Property providing access to the FileDataAPI"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef devicecore(self):\n if self._devicecore_api is None:\n self._devicecore_api = self.get_devicecore_api()\n return self._devicecore_api", "response": "Property providing access to the device core API"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a DeviceCoreAPI object bound to this device cloud instance", "response": "def get_devicecore_api(self):\n \"\"\"Returns a :class:`.DeviceCoreAPI` bound to this device cloud instance\n\n This provides access to the same API as :attr:`.DeviceCloud.devicecore` but will create\n a new object (with a new cache) each time called.\n\n :return: devicecore API object bound to this device cloud account\n :rtype: :class:`.DeviceCoreAPI`\n\n \"\"\"\n from devicecloud.devicecore import DeviceCoreAPI\n\n return DeviceCoreAPI(self._conn, self.get_sci_api())"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nqueries an asynchronous SCI job by ID This is useful for getting the job information from the async queue.", "response": "def get_async_job(self, job_id):\n \"\"\"Query an asynchronous SCI job by ID\n\n This is useful if the job was not created with send_sci_async().\n\n :param int job_id: The job ID to query\n :returns: The SCI response from GETting the job information\n \"\"\"\n uri = \"/ws/sci/{0}\".format(job_id)\n # TODO: do parsing here?\n return self._conn.get(uri)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_sci_async(self, operation, target, payload, **sci_options):\n sci_options['synchronous'] = False\n resp = self.send_sci(operation, target, payload, **sci_options)\n dom = ET.fromstring(resp.content)\n job_element = dom.find('.//jobId')\n if job_element is None:\n return\n job_id = int(job_element.text)\n return AsyncRequestProxy(job_id, self._conn)", "response": "Send an asynchronous SCI request and wrap the job in an AsyncRequestProxy."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend an SCI request to one or more targets.", "response": "def send_sci(self, operation, target, payload, reply=None, synchronous=None, sync_timeout=None,\n cache=None, allow_offline=None, wait_for_reconnect=None):\n \"\"\"Send SCI request to 1 or more targets\n\n :param str operation: The operation is one of {send_message, update_firmware, disconnect, query_firmware_targets,\n file_system, data_service, and reboot}\n :param target: The device(s) to be targeted with this request\n :type target: :class:`~.TargetABC` or list of :class:`~.TargetABC` instances\n\n TODO: document other params\n\n \"\"\"\n if not isinstance(payload, six.string_types) and not isinstance(payload, six.binary_type):\n raise TypeError(\"payload is required to be a string or bytes\")\n\n # validate targets and bulid targets xml section\n try:\n iter(target)\n targets = target\n except TypeError:\n targets = [target, ]\n if not all(isinstance(t, TargetABC) for t in targets):\n raise TypeError(\"Target(s) must each be instances of TargetABC\")\n targets_xml = \"\".join(t.to_xml() for t in targets)\n\n # reply argument\n if not isinstance(reply, (type(None), six.string_types)):\n raise TypeError(\"reply must be either None or a string\")\n if reply is not None:\n reply_xml = ' reply=\"{}\"'.format(reply)\n else:\n reply_xml = ''\n\n # synchronous argument\n if not isinstance(synchronous, (type(None), bool)):\n raise TypeError(\"synchronous expected to be either None or a boolean\")\n if synchronous is not None:\n synchronous_xml = ' synchronous=\"{}\"'.format('true' if synchronous else 'false')\n else:\n synchronous_xml = ''\n\n # sync_timeout argument\n # TODO: What units is syncTimeout in? seconds?\n if sync_timeout is not None and not isinstance(sync_timeout, six.integer_types):\n raise TypeError(\"sync_timeout expected to either be None or a number\")\n if sync_timeout is not None:\n sync_timeout_xml = ' syncTimeout=\"{}\"'.format(sync_timeout)\n else:\n sync_timeout_xml = ''\n\n # cache argument\n if not isinstance(cache, (type(None), bool)):\n raise TypeError(\"cache expected to either be None or a boolean\")\n if cache is not None:\n cache_xml = ' cache=\"{}\"'.format('true' if cache else 'false')\n else:\n cache_xml = ''\n\n # allow_offline argument\n if not isinstance(allow_offline, (type(None), bool)):\n raise TypeError(\"allow_offline is expected to be either None or a boolean\")\n if allow_offline is not None:\n allow_offline_xml = ' allowOffline=\"{}\"'.format('true' if allow_offline else 'false')\n else:\n allow_offline_xml = ''\n\n # wait_for_reconnect argument\n if not isinstance(wait_for_reconnect, (type(None), bool)):\n raise TypeError(\"wait_for_reconnect expected to be either None or a boolean\")\n if wait_for_reconnect is not None:\n wait_for_reconnect_xml = ' waitForReconnect=\"{}\"'.format('true' if wait_for_reconnect else 'false')\n else:\n wait_for_reconnect_xml = ''\n\n full_request = SCI_TEMPLATE.format(\n operation=operation,\n targets=targets_xml,\n reply=reply_xml,\n synchronous=synchronous_xml,\n sync_timeout=sync_timeout_xml,\n cache=cache_xml,\n allow_offline=allow_offline_xml,\n wait_for_reconnect=wait_for_reconnect_xml,\n payload=payload\n )\n\n # TODO: do parsing here?\n return self._conn.post(\"/ws/sci\", full_request)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite to stream using fmt and value", "response": "def conditional_write(strm, fmt, value, *args, **kwargs):\n \"\"\"Write to stream using fmt and value if value is not None\"\"\"\n if value is not None:\n strm.write(fmt.format(value, *args, **kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngive an ISO8601 string as returned by Device Cloud convert to a datetime object", "response": "def iso8601_to_dt(iso8601):\n \"\"\"Given an ISO8601 string as returned by Device Cloud, convert to a datetime object\"\"\"\n # We could just use arrow.get() but that is more permissive than we actually want.\n # Internal (but still public) to arrow is the actual parser where we can be\n # a bit more specific\n parser = DateTimeParser()\n try:\n arrow_dt = arrow.Arrow.fromdatetime(parser.parse_iso(iso8601))\n return arrow_dt.to('utc').datetime\n except ParserError as pe:\n raise ValueError(\"Provided was not a valid ISO8601 string: %r\" % pe)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting input to either None or a datetime object.", "response": "def to_none_or_dt(input):\n \"\"\"Convert ``input`` to either None or a datetime object\n\n If the input is None, None will be returned.\n If the input is a datetime object, it will be converted to a datetime\n object with UTC timezone info. If the datetime object is naive, then\n this method will assume the object is specified according to UTC and\n not local or some other timezone.\n If the input to the function is a string, this method will attempt to\n parse the input as an ISO-8601 formatted string.\n\n :param input: Input data (expected to be either str, None, or datetime object)\n :return: datetime object from input or None if already None\n :rtype: datetime or None\n\n \"\"\"\n if input is None:\n return input\n elif isinstance(input, datetime.datetime):\n arrow_dt = arrow.Arrow.fromdatetime(input, input.tzinfo or 'utc')\n return arrow_dt.to('utc').datetime\n if isinstance(input, six.string_types):\n # try to convert from ISO8601\n return iso8601_to_dt(input)\n else:\n raise TypeError(\"Not a string, NoneType, or datetime object\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an ISO - 8601 formatted string from the provided datetime object", "response": "def isoformat(dt):\n \"\"\"Return an ISO-8601 formatted string from the provided datetime object\"\"\"\n if not isinstance(dt, datetime.datetime):\n raise TypeError(\"Must provide datetime.datetime object to isoformat\")\n\n if dt.tzinfo is None:\n raise ValueError(\"naive datetime objects are not allowed beyond the library boundaries\")\n\n return dt.isoformat().replace(\"+00:00\", \"Z\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a generator over all filedata matching the condition.", "response": "def get_filedata(self, condition=None, page_size=1000):\n \"\"\"Return a generator over all results matching the provided condition\n\n :param condition: An :class:`.Expression` which defines the condition\n which must be matched on the filedata that will be retrieved from\n file data store. If a condition is unspecified, the following condition\n will be used ``fd_path == '~/'``. This condition will match all file\n data in this accounts \"home\" directory (a sensible root).\n :type condition: :class:`.Expression` or None\n :param int page_size: The number of results to fetch in a single page. Regardless\n of the size specified, :meth:`.get_filedata` will continue to fetch pages\n and yield results until all items have been fetched.\n :return: Generator yielding :class:`.FileDataObject` instances matching the\n provided conditions.\n\n \"\"\"\n\n condition = validate_type(condition, type(None), Expression, *six.string_types)\n page_size = validate_type(page_size, *six.integer_types)\n if condition is None:\n condition = (fd_path == \"~/\") # home directory\n\n params = {\"embed\": \"true\", \"condition\": condition.compile()}\n for fd_json in self._conn.iter_json_pages(\"/ws/FileData\", page_size=page_size, **params):\n yield FileDataObject.from_json(self, fd_json)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting a file to the file data store at the given path.", "response": "def write_file(self, path, name, data, content_type=None, archive=False,\n raw=False):\n \"\"\"Write a file to the file data store at the given path\n\n :param str path: The path (directory) into which the file should be written.\n :param str name: The name of the file to be written.\n :param data: The binary data that should be written into the file.\n :type data: str (Python2) or bytes (Python3)\n :param content_type: The content type for the data being written to the file. May\n be left unspecified.\n :type content_type: str or None\n :param bool archive: If true, history will be retained for various revisions of this\n file. If this is not required, leave as false.\n :param bool raw: If true, skip the FileData XML headers (necessary for binary files)\n\n \"\"\"\n path = validate_type(path, *six.string_types)\n name = validate_type(name, *six.string_types)\n data = validate_type(data, six.binary_type)\n content_type = validate_type(content_type, type(None), *six.string_types)\n archive_str = \"true\" if validate_type(archive, bool) else \"false\"\n\n if not path.startswith(\"/\"):\n path = \"/\" + path\n if not path.endswith(\"/\"):\n path += \"/\"\n name = name.lstrip(\"/\")\n\n sio = six.moves.StringIO()\n if not raw:\n if six.PY3:\n base64_encoded_data = base64.encodebytes(data).decode('utf-8')\n else:\n base64_encoded_data = base64.encodestring(data)\n\n sio.write(\"\")\n if content_type is not None:\n sio.write(\"{}\".format(content_type))\n sio.write(\"file\")\n sio.write(\"{}\".format(base64_encoded_data))\n sio.write(\"{}\".format(archive_str))\n sio.write(\"\")\n else:\n sio.write(data)\n\n params = {\n \"type\": \"file\",\n \"archive\": archive_str\n }\n self._conn.put(\n \"/ws/FileData{path}{name}\".format(path=path, name=name),\n sio.getvalue(),\n params=params)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes a file or directory from the filedata store", "response": "def delete_file(self, path):\n \"\"\"Delete a file or directory from the filedata store\n\n This method removes a file or directory (recursively) from\n the filedata store.\n\n :param path: The path of the file or directory to remove\n from the file data store.\n\n \"\"\"\n path = validate_type(path, *six.string_types)\n if not path.startswith(\"/\"):\n path = \"/\" + path\n\n self._conn.delete(\"/ws/FileData{path}\".format(path=path))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef walk(self, root=\"~/\"):\n root = validate_type(root, *six.string_types)\n\n directories = []\n files = []\n\n # fd_path is real picky\n query_fd_path = root\n if not query_fd_path.endswith(\"/\"):\n query_fd_path += \"/\"\n\n for fd_object in self.get_filedata(fd_path == query_fd_path):\n if fd_object.get_type() == \"directory\":\n directories.append(fd_object)\n else:\n files.append(fd_object)\n\n # Yield the walk results for this level of the tree\n yield (root, directories, files)\n\n # recurse on each directory and yield results up the chain\n for directory in directories:\n for dirpath, directories, files in self.walk(directory.get_full_path()):\n yield (dirpath, directories, files)", "response": "Emulation of os. walk behavior against Device Cloud filedata store\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_data(self):\n # NOTE: we assume that the \"embed\" option is used\n base64_data = self._json_data.get(\"fdData\")\n if base64_data is None:\n return None\n else:\n # need to convert to bytes() with python 3\n return base64.decodestring(six.b(base64_data))", "response": "Get the data associated with this filedata object"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef write_file(self, *args, **kwargs):\n return self._fdapi.write_file(self.get_path(), *args, **kwargs)", "response": "Write a file into this directory"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_tcp_monitor(self, topics, batch_size=1, batch_duration=0,\n compression='gzip', format_type='json'):\n \"\"\"Creates a TCP Monitor instance in Device Cloud for a given list of topics\n\n :param topics: a string list of topics (e.g. ['DeviceCore[U]',\n 'FileDataCore']).\n :param batch_size: How many Msgs received before sending data.\n :param batch_duration: How long to wait before sending batch if it\n does not exceed batch_size.\n :param compression: Compression value (i.e. 'gzip').\n :param format_type: What format server should send data in (i.e. 'xml' or 'json').\n\n Returns an object of the created Monitor\n \"\"\"\n\n monitor_xml = \"\"\"\\\n \n {topics}\n {batch_size}\n {format_type}\n tcp\n {compression}\n \n \"\"\".format(\n topics=','.join(topics),\n batch_size=batch_size,\n batch_duration=batch_duration,\n format_type=format_type,\n compression=compression,\n )\n monitor_xml = textwrap.dedent(monitor_xml)\n\n response = self._conn.post(\"/ws/Monitor\", monitor_xml)\n location = ET.fromstring(response.text).find('.//location').text\n monitor_id = int(location.split('/')[-1])\n return TCPDeviceCloudMonitor(self._conn, monitor_id, self._tcp_client_manager)", "response": "Creates a new TCP Monitor instance for a given list of topics."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a Monitor instance in Device Cloud for a given list of topics.", "response": "def create_http_monitor(self, topics, transport_url, transport_token=None, transport_method='PUT', connect_timeout=0,\n response_timeout=0, batch_size=1, batch_duration=0, compression='none', format_type='json'):\n \"\"\"Creates a HTTP Monitor instance in Device Cloud for a given list of topics\n\n :param topics: a string list of topics (e.g. ['DeviceCore[U]',\n 'FileDataCore']).\n :param transport_url: URL of the customer web server.\n :param transport_token: Credentials for basic authentication in the following format: username:password\n :param transport_method: HTTP method to use for sending data: PUT or POST. The default is PUT.\n :param connect_timeout: A value of 0 means use the system default of 5000 (5 seconds).\n :param response_timeout: A value of 0 means use the system default of 5000 (5 seconds).\n :param batch_size: How many Msgs received before sending data.\n :param batch_duration: How long to wait before sending batch if it\n does not exceed batch_size.\n :param compression: Compression value (i.e. 'gzip').\n :param format_type: What format server should send data in (i.e. 'xml' or 'json').\n\n Returns an object of the created Monitor\n \"\"\"\n\n monitor_xml = \"\"\"\\\n \n {topics}\n {batch_size}\n {format_type}\n http\n {transport_url}\n {transport_token}\n {transport_method}\n {connect_timeout}\n {response_timeout}\n {compression}\n \n \"\"\".format(\n topics=','.join(topics),\n transport_url=transport_url,\n transport_token=transport_token,\n transport_method=transport_method,\n connect_timeout=connect_timeout,\n response_timeout=response_timeout,\n batch_size=batch_size,\n batch_duration=batch_duration,\n format_type=format_type,\n compression=compression,\n )\n monitor_xml = textwrap.dedent(monitor_xml)\n\n response = self._conn.post(\"/ws/Monitor\", monitor_xml)\n location = ET.fromstring(response.text).find('.//location').text\n monitor_id = int(location.split('/')[-1])\n return HTTPDeviceCloudMonitor(self._conn, monitor_id)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_monitors(self, condition=None, page_size=1000):\n req_kwargs = {}\n if condition:\n req_kwargs['condition'] = condition.compile()\n for monitor_data in self._conn.iter_json_pages(\"/ws/Monitor\", **req_kwargs):\n yield DeviceCloudMonitor.from_json(self._conn, monitor_data, self._tcp_client_manager)", "response": "Returns an iterator over all monitors matching the provided condition."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_monitor(self, topics):\n for monitor in self.get_monitors(MON_TOPIC_ATTR == \",\".join(topics)):\n return monitor # return the first one, even if there are multiple\n return None", "response": "Attempts to find a Monitor in device cloud that matches the provided topics. Returns None if no Monitor is found."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_decoder_method(stream_type):\n if stream_type is not None:\n return DSTREAM_TYPE_MAP.get(stream_type.upper(), (lambda x: x, lambda x: x))[0]\n else:\n return lambda x: x", "response": "Returns a function that returns the python type converter function for the given stream type."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a list of all the streams in the system", "response": "def _get_streams(self, uri_suffix=None):\n \"\"\"Clear and update internal cache of stream objects\"\"\"\n # TODO: handle paging, perhaps change this to be a generator\n if uri_suffix is not None and not uri_suffix.startswith('/'):\n uri_suffix = '/' + uri_suffix\n elif uri_suffix is None:\n uri_suffix = \"\"\n streams = {}\n response = self._conn.get_json(\"/ws/DataStream{}\".format(uri_suffix))\n for stream_data in response[\"items\"]:\n stream_id = stream_data[\"streamId\"]\n stream = DataStream(self._conn, stream_id, stream_data)\n streams[stream_id] = stream\n return streams"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new data stream on Device Cloud This method will attempt to create a new data stream on Device Cloud. This method will only succeed if the stream does not already exist. :param str stream_id: The path/id of the stream being created on Device Cloud. :param str data_type: The type of this stream. This must be in the set `{ INTEGER, LONG, FLOAT, DOUBLE, STRING, BINARY, UNKNOWN }`. These values are available in constants like :attr:`~STREAM_TYPE_INTEGER`. :param str description: An optional description of this stream. See :meth:`~DataStream.get_description`. :param int data_ttl: The TTL for data points in this stream. See :meth:`~DataStream.get_data_ttl`. :param int rollup_ttl: The TTL for performing rollups on data. See :meth:~DataStream.get_rollup_ttl`. :param str units: Units for data in this stream. See :meth:`~DataStream.get_units`", "response": "def create_stream(self, stream_id, data_type, description=None, data_ttl=None,\n rollup_ttl=None, units=None):\n \"\"\"Create a new data stream on Device Cloud\n\n This method will attempt to create a new data stream on Device Cloud.\n This method will only succeed if the stream does not already exist.\n\n :param str stream_id: The path/id of the stream being created on Device Cloud.\n :param str data_type: The type of this stream. This must be in the set\n `{ INTEGER, LONG, FLOAT, DOUBLE, STRING, BINARY, UNKNOWN }`. These values are\n available in constants like :attr:`~STREAM_TYPE_INTEGER`.\n :param str description: An optional description of this stream. See :meth:`~DataStream.get_description`.\n :param int data_ttl: The TTL for data points in this stream. See :meth:`~DataStream.get_data_ttl`.\n :param int rollup_ttl: The TTL for performing rollups on data. See :meth:~DataStream.get_rollup_ttl`.\n :param str units: Units for data in this stream. See :meth:`~DataStream.get_units`\n\n \"\"\"\n\n stream_id = validate_type(stream_id, *six.string_types)\n data_type = validate_type(data_type, type(None), *six.string_types)\n if isinstance(data_type, *six.string_types):\n data_type = str(data_type).upper()\n if not data_type in (set([None, ]) | set(list(DSTREAM_TYPE_MAP.keys()))):\n raise ValueError(\"data_type %r is not valid\" % data_type)\n description = validate_type(description, type(None), *six.string_types)\n data_ttl = validate_type(data_ttl, type(None), *six.integer_types)\n rollup_ttl = validate_type(rollup_ttl, type(None), *six.integer_types)\n units = validate_type(units, type(None), *six.string_types)\n\n sio = StringIO()\n sio.write(\"\")\n conditional_write(sio, \"{}\", stream_id)\n conditional_write(sio, \"{}\", data_type)\n conditional_write(sio, \"{}\", description)\n conditional_write(sio, \"{}\", data_ttl)\n conditional_write(sio, \"{}\", rollup_ttl)\n conditional_write(sio, \"{}\", units)\n sio.write(\"\")\n\n self._conn.post(\"/ws/DataStream\", sio.getvalue())\n logger.info(\"Data stream (%s) created successfully\", stream_id)\n stream = DataStream(self._conn, stream_id)\n return stream"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_stream_if_exists(self, stream_id):\n stream = self.get_stream(stream_id)\n try:\n stream.get_data_type(use_cached=True)\n except NoSuchStreamException:\n return None\n else:\n return stream", "response": "Return a reference to a stream with the given stream_id if it exists."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform a bulk write (or set of writes) of a collection of data points This method takes a list (or other iterable) of datapoints and writes them to Device Cloud in an efficient manner, minimizing the number of HTTP requests that need to be made. As this call is performed from outside the context of any particular stream, each DataPoint object passed in must include information about the stream into which the point should be written. If all data points being written are for the same stream, you may want to consider using :meth:`~DataStream.bulk_write_datapoints` instead. Example:: datapoints = [] for i in range(300): datapoints.append(DataPoint( stream_id=\"my/stream%d\" % (i % 3), data_type=STREAM_TYPE_INTEGER, units=\"meters\", data=i, )) dc.streams.bulk_write_datapoints(datapoints) Depending on the size of the list of datapoints provided, this method may need to make multiple calls to Device Cloud (in chunks of 250). :param list datapoints: a list of datapoints to be written to Device Cloud :raises TypeError: if a list of datapoints is not provided :raises ValueError: if any of the provided data points do not have all required information (such as information about the stream) :raises DeviceCloudHttpException: in the case of an unexpected error in communicating with Device Cloud.", "response": "def bulk_write_datapoints(self, datapoints):\n \"\"\"Perform a bulk write (or set of writes) of a collection of data points\n\n This method takes a list (or other iterable) of datapoints and writes them\n to Device Cloud in an efficient manner, minimizing the number of HTTP\n requests that need to be made.\n\n As this call is performed from outside the context of any particular stream,\n each DataPoint object passed in must include information about the stream\n into which the point should be written.\n\n If all data points being written are for the same stream, you may want to\n consider using :meth:`~DataStream.bulk_write_datapoints` instead.\n\n Example::\n\n datapoints = []\n for i in range(300):\n datapoints.append(DataPoint(\n stream_id=\"my/stream%d\" % (i % 3),\n data_type=STREAM_TYPE_INTEGER,\n units=\"meters\",\n data=i,\n ))\n dc.streams.bulk_write_datapoints(datapoints)\n\n Depending on the size of the list of datapoints provided, this method may\n need to make multiple calls to Device Cloud (in chunks of 250).\n\n :param list datapoints: a list of datapoints to be written to Device Cloud\n :raises TypeError: if a list of datapoints is not provided\n :raises ValueError: if any of the provided data points do not have all required\n information (such as information about the stream)\n :raises DeviceCloudHttpException: in the case of an unexpected error in communicating\n with Device Cloud.\n\n \"\"\"\n datapoints = list(datapoints) # effectively performs validation that we have the right type\n for dp in datapoints:\n if not isinstance(dp, DataPoint):\n raise TypeError(\"All items in the datapoints list must be DataPoints\")\n if dp.get_stream_id() is None:\n raise ValueError(\"stream_id must be set on all datapoints\")\n\n remaining_datapoints = datapoints\n while remaining_datapoints:\n # take up to 250 points and post them until complete\n this_chunk_of_datapoints = remaining_datapoints[:MAXIMUM_DATAPOINTS_PER_POST]\n remaining_datapoints = remaining_datapoints[MAXIMUM_DATAPOINTS_PER_POST:]\n\n # Build XML list containing data for all points\n datapoints_out = StringIO()\n datapoints_out.write(\"\")\n for dp in this_chunk_of_datapoints:\n datapoints_out.write(dp.to_xml())\n datapoints_out.write(\"\")\n\n # And send the HTTP Post\n self._conn.post(\"/ws/DataPoint\", datapoints_out.getvalue())\n logger.info('DataPoint batch of %s datapoints written', len(this_chunk_of_datapoints))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new DataPoint object from a JSON data stream.", "response": "def from_json(cls, stream, json_data):\n \"\"\"Create a new DataPoint object from device cloud JSON data\n\n :param DataStream stream: The :class:`~DataStream` out of which this data is coming\n :param dict json_data: Deserialized JSON data from Device Cloud about this device\n :raises ValueError: if the data is malformed\n :return: (:class:`~DataPoint`) newly created :class:`~DataPoint`\n\n \"\"\"\n type_converter = _get_decoder_method(stream.get_data_type())\n data = type_converter(json_data.get(\"data\"))\n return cls(\n # these are actually properties of the stream, not the data point\n stream_id=stream.get_stream_id(),\n data_type=stream.get_data_type(),\n units=stream.get_units(),\n\n # and these are part of the data point itself\n data=data,\n description=json_data.get(\"description\"),\n timestamp=json_data.get(\"timestampISO\"),\n server_timestamp=json_data.get(\"serverTimestampISO\"),\n quality=json_data.get(\"quality\"),\n location=json_data.get(\"location\"),\n dp_id=json_data.get(\"id\"),\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_rollup_json(cls, stream, json_data):\n dp = cls.from_json(stream, json_data)\n\n # Special handling for timestamp\n timestamp = isoformat(dc_utc_timestamp_to_dt(int(json_data.get(\"timestamp\"))))\n\n # Special handling for data, all rollup data is float type\n type_converter = _get_decoder_method(stream.get_data_type())\n data = type_converter(float(json_data.get(\"data\")))\n\n # Update the special fields\n dp.set_timestamp(timestamp)\n dp.set_data(data)\n return dp", "response": "Create a new data point from a rollup json data stream."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the stream id associated with this data point", "response": "def set_stream_id(self, stream_id):\n \"\"\"Set the stream id associated with this data point\"\"\"\n stream_id = validate_type(stream_id, type(None), *six.string_types)\n if stream_id is not None:\n stream_id = stream_id.lstrip('/')\n self._stream_id = stream_id"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_description(self, description):\n self._description = validate_type(description, type(None), *six.string_types)", "response": "Set the description for this data point"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_quality(self, quality):\n if isinstance(quality, *six.string_types):\n quality = int(quality)\n elif isinstance(quality, float):\n quality = int(quality)\n\n self._quality = validate_type(quality, type(None), *six.integer_types)", "response": "Set the quality for this sample"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_location(self, location):\n if location is None:\n self._location = location\n\n elif isinstance(location, *six.string_types): # from device cloud, convert from csv\n parts = str(location).split(\",\")\n if len(parts) == 3:\n self._location = tuple(map(float, parts))\n return\n else:\n raise ValueError(\"Location string %r has unexpected format\" % location)\n\n # TODO: could maybe try to allow any iterable but this covers the most common cases\n elif (isinstance(location, (tuple, list))\n and len(location) == 3\n and all([isinstance(x, (float, six.integer_types)) for x in location])):\n self._location = tuple(map(float, location)) # coerce ints to float\n else:\n raise TypeError(\"Location must be None or 3-tuple of floats\")\n\n self._location = location", "response": "Set the location for this data point."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_data_type(self, data_type):\n validate_type(data_type, type(None), *six.string_types)\n if isinstance(data_type, *six.string_types):\n data_type = str(data_type).upper()\n if not data_type in ({None} | set(DSTREAM_TYPE_MAP.keys())):\n raise ValueError(\"Provided data type not in available set of types\")\n self._data_type = data_type", "response": "Set the data type for the current entry point."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the unit for this data point", "response": "def set_units(self, unit):\n \"\"\"Set the unit for this data point\n\n Unit, as with data_type, are actually associated with the stream and not\n the individual data point. As such, changing this within a stream is\n not encouraged. Setting the unit on the data point is useful when the\n stream might be created with the write of a data point.\n\n \"\"\"\n self._units = validate_type(unit, type(None), *six.string_types)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_xml(self):\n type_converter = _get_encoder_method(self._data_type)\n # Convert from python native to device cloud\n encoded_data = type_converter(self._data)\n\n out = StringIO()\n out.write(\"\")\n out.write(\"{}\".format(self.get_stream_id()))\n out.write(\"{}\".format(encoded_data))\n conditional_write(out, \"{}\", self.get_description())\n if self.get_timestamp() is not None:\n out.write(\"{}\".format(isoformat(self.get_timestamp())))\n conditional_write(out, \"{}\", self.get_quality())\n if self.get_location() is not None:\n out.write(\"%s\" % \",\".join(map(str, self.get_location())))\n conditional_write(out, \"{}\", self.get_data_type())\n conditional_write(out, \"{}\", self.get_units())\n out.write(\"\")\n return out.getvalue()", "response": "Convert this datapoint into a form suitable for pushing to device cloud cloud."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nretrieve metadata about this stream from Device Cloud", "response": "def _get_stream_metadata(self, use_cached):\n \"\"\"Retrieve metadata about this stream from Device Cloud\"\"\"\n if self._cached_data is None or not use_cached:\n try:\n self._cached_data = self._conn.get_json(\"/ws/DataStream/%s\" % self._stream_id)[\"items\"][0]\n except DeviceCloudHttpException as http_exception:\n if http_exception.response.status_code == 404:\n raise NoSuchStreamException(\"Stream with id %r has not been created\" % self._stream_id)\n raise http_exception\n return self._cached_data"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the data type of this data stream.", "response": "def get_data_type(self, use_cached=True):\n \"\"\"Get the data type of this stream if it exists\n\n The data type is the type of data stored in this data stream. Valid types include:\n\n * INTEGER - data can be represented with a network (= big-endian) 32-bit two's-complement integer. Data\n with this type maps to a python int.\n * LONG - data can be represented with a network (= big-endian) 64-bit two's complement integer. Data\n with this type maps to a python int.\n * FLOAT - data can be represented with a network (= big-endian) 32-bit IEEE754 floating point. Data\n with this type maps to a python float.\n * DOUBLE - data can be represented with a network (= big-endian) 64-bit IEEE754 floating point. Data\n with this type maps to a python float.\n * STRING - UTF-8. Data with this type map to a python string\n * BINARY - Data with this type map to a python string.\n * UNKNOWN - Data with this type map to a python string.\n\n :param bool use_cached: If False, the function will always request the latest from Device Cloud.\n If True, the device will not make a request if it already has cached data.\n :return: The data type of this stream as a string\n :rtype: str\n\n \"\"\"\n dtype = self._get_stream_metadata(use_cached).get(\"dataType\")\n if dtype is not None:\n dtype = dtype.upper()\n return dtype"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the data TTL for this stream.", "response": "def get_data_ttl(self, use_cached=True):\n \"\"\"Retrieve the dataTTL for this stream\n\n The dataTtl is the time to live (TTL) in seconds for data points stored in the data stream.\n A data point expires after the configured amount of time and is automatically deleted.\n\n :param bool use_cached: If False, the function will always request the latest from Device Cloud.\n If True, the device will not make a request if it already has cached data.\n :raises devicecloud.DeviceCloudHttpException: in the case of an unexpected http error\n :raises devicecloud.streams.NoSuchStreamException: if this stream has not yet been created\n :return: The dataTtl associated with this stream in seconds\n :rtype: int or None\n\n \"\"\"\n\n data_ttl_text = self._get_stream_metadata(use_cached).get(\"dataTtl\")\n return int(data_ttl_text)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the rollupTtl associated with this stream in seconds.", "response": "def get_rollup_ttl(self, use_cached=True):\n \"\"\"Retrieve the rollupTtl for this stream\n\n The rollupTtl is the time to live (TTL) in seconds for the aggregate roll-ups of data points\n stored in the stream. A roll-up expires after the configured amount of time and is\n automatically deleted.\n\n :param bool use_cached: If False, the function will always request the latest from Device Cloud.\n If True, the device will not make a request if it already has cached data.\n :raises devicecloud.DeviceCloudHttpException: in the case of an unexpected http error\n :raises devicecloud.streams.NoSuchStreamException: if this stream has not yet been created\n :return: The rollupTtl associated with this stream in seconds\n :rtype: int or None\n\n \"\"\"\n rollup_ttl_text = self._get_stream_metadata(use_cached).get(\"rollupTtl\")\n return int(rollup_ttl_text)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the current value written to a stream.", "response": "def get_current_value(self, use_cached=False):\n \"\"\"Return the most recent DataPoint value written to a stream\n\n The current value is the last recorded data point for this stream.\n\n :param bool use_cached: If False, the function will always request the latest from Device Cloud.\n If True, the device will not make a request if it already has cached data.\n :raises devicecloud.DeviceCloudHttpException: in the case of an unexpected http error\n :raises devicecloud.streams.NoSuchStreamException: if this stream has not yet been created\n :return: The most recent value written to this stream (or None if nothing has been written)\n :rtype: :class:`~DataPoint` or None\n\n \"\"\"\n current_value = self._get_stream_metadata(use_cached).get(\"currentValue\")\n if current_value:\n return DataPoint.from_json(self, current_value)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef delete(self):\n try:\n self._conn.delete(\"/ws/DataStream/{}\".format(self.get_stream_id()))\n except DeviceCloudHttpException as http_excpeption:\n if http_excpeption.response.status_code == 404:\n raise NoSuchStreamException() # this branch is present, but the DC appears to just return 200 again\n else:\n raise http_excpeption", "response": "Delete this stream from Device Cloud along with its history\n\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndelete the provided datapoint from this stream", "response": "def delete_datapoint(self, datapoint):\n \"\"\"Delete the provided datapoint from this stream\n\n :raises devicecloud.DeviceCloudHttpException: in the case of an unexpected http error\n\n \"\"\"\n datapoint = validate_type(datapoint, DataPoint)\n self._conn.delete(\"/ws/DataPoint/{stream_id}/{datapoint_id}\".format(\n stream_id=self.get_stream_id(),\n datapoint_id=datapoint.get_id(),\n ))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndeletes datapoints from this stream between the provided start and end times.", "response": "def delete_datapoints_in_time_range(self, start_dt=None, end_dt=None):\n \"\"\"Delete datapoints from this stream between the provided start and end times\n\n If neither a start or end time is specified, all data points in the stream\n will be deleted.\n\n :param start_dt: The datetime after which data points should be deleted or None\n if all data points from the beginning of time should be deleted.\n :param end_dt: The datetime before which data points should be deleted or None\n if all data points until the current time should be deleted.\n :raises devicecloud.DeviceCloudHttpException: in the case of an unexpected http error\n\n \"\"\"\n start_dt = to_none_or_dt(validate_type(start_dt, datetime.datetime, type(None)))\n end_dt = to_none_or_dt(validate_type(end_dt, datetime.datetime, type(None)))\n\n params = {}\n if start_dt is not None:\n params['startTime'] = isoformat(start_dt)\n if end_dt is not None:\n params['endTime'] = isoformat(end_dt)\n\n self._conn.delete(\"/ws/DataPoint/{stream_id}{querystring}\".format(\n stream_id=self.get_stream_id(),\n querystring=\"?\" + urllib.parse.urlencode(params) if params else \"\",\n ))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite some raw data to a stream using the DataPoint API", "response": "def write(self, datapoint):\n \"\"\"Write some raw data to a stream using the DataPoint API\n\n This method will mutate the datapoint provided to populate it with information\n available from the stream as it is available (but without making any new HTTP\n requests). For instance, we will add in information about the stream data\n type if it is available so that proper type conversion happens.\n\n Values already set on the datapoint will not be overridden (except for path)\n\n :param DataPoint datapoint: The :class:`.DataPoint` that should be written to Device Cloud\n\n \"\"\"\n if not isinstance(datapoint, DataPoint):\n raise TypeError(\"First argument must be a DataPoint object\")\n\n datapoint._stream_id = self.get_stream_id()\n if self._cached_data is not None and datapoint.get_data_type() is None:\n datapoint._data_type = self.get_data_type()\n\n self._conn.post(\"/ws/DataPoint/{}\".format(self.get_stream_id()), datapoint.to_xml())"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, start_time=None, end_time=None, use_client_timeline=True, newest_first=True,\n rollup_interval=None, rollup_method=None, timezone=None, page_size=1000):\n \"\"\"Read one or more DataPoints from a stream\n\n .. warning::\n The data points from Device Cloud is a paged data set. When iterating over the\n result set there could be delays when we hit the end of a page. If this is undesirable,\n the caller should collect all results into a data structure first before iterating over\n the result set.\n\n :param start_time: The start time for the window of data points to read. None means\n that we should start with the oldest data available.\n :type start_time: :class:`datetime.datetime` or None\n :param end_time: The end time for the window of data points to read. None means\n that we should include all points received until this point in time.\n :type end_time: :class:`datetime.datetime` or None\n :param bool use_client_timeline: If True, the times used will be those provided by\n clients writing data points into the cloud (which also default to server time\n if the a timestamp was not included by the client). This is usually what you\n want. If False, the server timestamp will be used which records when the data\n point was received.\n :param bool newest_first: If True, results will be ordered from newest to oldest (descending order).\n If False, results will be returned oldest to newest.\n :param rollup_interval: the roll-up interval that should be used if one is desired at all. Rollups\n will not be performed if None is specified for the interval. Valid roll-up interval values\n are None, \"half\", \"hourly\", \"day\", \"week\", and \"month\". See `DataPoints documentation\n `_\n for additional details on these values.\n :type rollup_interval: str or None\n :param rollup_method: The aggregation applied to values in the points within the specified\n rollup_interval. Available methods are None, \"sum\", \"average\", \"min\", \"max\", \"count\", and\n \"standarddev\". See `DataPoint documentation\n `_\n for additional details on these values.\n :type rollup_method: str or None\n :param timezone: timezone for calculating roll-ups. This determines roll-up interval\n boundaries and only applies to roll-ups of a day or larger (for example, day,\n week, or month). Note that it does not apply to the startTime and endTime parameters.\n See the `Timestamps `_\n and `Supported Time Zones `_\n sections for more information.\n :type timezone: str or None\n :param int page_size: The number of results that we should attempt to retrieve from the\n device cloud in each page. Generally, this can be left at its default value unless\n you have a good reason to change the parameter for performance reasons.\n :returns: A generator object which one can iterate over the DataPoints read.\n\n \"\"\"\n\n is_rollup = False\n if (rollup_interval is not None) or (rollup_method is not None):\n is_rollup = True\n numeric_types = [\n STREAM_TYPE_INTEGER,\n STREAM_TYPE_LONG,\n STREAM_TYPE_FLOAT,\n STREAM_TYPE_DOUBLE,\n STREAM_TYPE_STRING,\n STREAM_TYPE_BINARY,\n STREAM_TYPE_UNKNOWN,\n ]\n\n if self.get_data_type(use_cached=True) not in numeric_types:\n raise InvalidRollupDatatype('Rollups only support numerical DataPoints')\n\n # Validate function inputs\n start_time = to_none_or_dt(validate_type(start_time, datetime.datetime, type(None)))\n end_time = to_none_or_dt(validate_type(end_time, datetime.datetime, type(None)))\n use_client_timeline = validate_type(use_client_timeline, bool)\n newest_first = validate_type(newest_first, bool)\n rollup_interval = validate_type(rollup_interval, type(None), *six.string_types)\n if not rollup_interval in {None,\n ROLLUP_INTERVAL_HALF,\n ROLLUP_INTERVAL_HOUR,\n ROLLUP_INTERVAL_DAY,\n ROLLUP_INTERVAL_WEEK,\n ROLLUP_INTERVAL_MONTH, }:\n raise ValueError(\"Invalid rollup_interval %r provided\" % (rollup_interval, ))\n rollup_method = validate_type(rollup_method, type(None), *six.string_types)\n if not rollup_method in {None,\n ROLLUP_METHOD_SUM,\n ROLLUP_METHOD_AVERAGE,\n ROLLUP_METHOD_MIN,\n ROLLUP_METHOD_MAX,\n ROLLUP_METHOD_COUNT,\n ROLLUP_METHOD_STDDEV}:\n raise ValueError(\"Invalid rollup_method %r provided\" % (rollup_method, ))\n timezone = validate_type(timezone, type(None), *six.string_types)\n page_size = validate_type(page_size, *six.integer_types)\n\n # Remember that there could be multiple pages of data and we want to provide\n # in iterator over the result set. To start the process out, we need to make\n # an initial request without a page cursor. We should get one in response to\n # our first request which we will use to page through the result set\n query_parameters = {\n 'timeline': 'client' if use_client_timeline else 'server',\n 'order': 'descending' if newest_first else 'ascending',\n 'size': page_size\n }\n if start_time is not None:\n query_parameters[\"startTime\"] = isoformat(start_time)\n if end_time is not None:\n query_parameters[\"endTime\"] = isoformat(end_time)\n if rollup_interval is not None:\n query_parameters[\"rollupInterval\"] = rollup_interval\n if rollup_method is not None:\n query_parameters[\"rollupMethod\"] = rollup_method\n if timezone is not None:\n query_parameters[\"timezone\"] = timezone\n\n result_size = page_size\n while result_size == page_size:\n # request the next page of data or first if pageCursor is not set as query param\n try:\n result = self._conn.get_json(\"/ws/DataPoint/{stream_id}?{query_params}\".format(\n stream_id=self.get_stream_id(),\n query_params=urllib.parse.urlencode(query_parameters)\n ))\n except DeviceCloudHttpException as http_exception:\n if http_exception.response.status_code == 404:\n raise NoSuchStreamException()\n raise http_exception\n\n result_size = int(result[\"resultSize\"]) # how many are actually included here?\n query_parameters[\"pageCursor\"] = result.get(\"pageCursor\") # will not be present if result set is empty\n for item_info in result.get(\"items\", []):\n if is_rollup:\n data_point = DataPoint.from_rollup_json(self, item_info)\n else:\n data_point = DataPoint.from_json(self, item_info)\n yield data_point", "response": "Reads one or more data points from a Device Cloud stream."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a single - quoted and escaped version of value", "response": "def _quoted(value):\n \"\"\"Return a single-quoted and escaped (percent-encoded) version of value\n\n This function will also perform transforms of known data types to a representation\n that will be handled by Device Cloud. For instance, datetime objects will be\n converted to ISO8601.\n\n \"\"\"\n if isinstance(value, datetime.datetime):\n value = isoformat(to_none_or_dt(value))\n else:\n value = str(value)\n\n return \"'{}'\".format(value)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compile(self):\n return \"{lhs}{sep}{rhs}\".format(\n lhs=self.lhs.compile(),\n sep=self.sep,\n rhs=self.rhs.compile(),\n )", "response": "Compile this expression into a query string"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef compile(self):\n return \"{attribute}{sep}{value}\".format(\n attribute=self.attribute,\n sep=self.sep,\n value=_quoted(self.value)\n )", "response": "Compile this expression into a query string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread a message header from the socket and return the response type.", "response": "def _read_msg_header(session):\n \"\"\"\n Perform a read on input socket to consume headers and then return\n a tuple of message type, message length.\n\n :param session: Push Session to read data for.\n\n Returns response type (i.e. PUBLISH_MESSAGE) if header was completely\n read, otherwise None if header was not completely read.\n \"\"\"\n try:\n data = session.socket.recv(6 - len(session.data))\n if len(data) == 0: # No Data on Socket. Likely closed.\n return NO_DATA\n session.data += data\n # Data still not completely read.\n if len(session.data) < 6:\n return INCOMPLETE\n\n except ssl.SSLError:\n # This can happen when select gets triggered\n # for an SSL socket and data has not yet been\n # read.\n return INCOMPLETE\n\n session.message_length = struct.unpack('!i', session.data[2:6])[0]\n response_type = struct.unpack('!H', session.data[0:2])[0]\n\n # Clear out session data as header is consumed.\n session.data = six.b(\"\")\n return response_type"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads a message from the socket and return the payload and block_id in a tuple.", "response": "def _read_msg(session):\n \"\"\"\n Perform a read on input socket to consume message and then return the\n payload and block_id in a tuple.\n\n :param session: Push Session to read data for.\n \"\"\"\n if len(session.data) == session.message_length:\n # Data Already completely read. Return\n return True\n\n try:\n data = session.socket.recv(session.message_length - len(session.data))\n if len(data) == 0:\n raise PushException(\"No Data on Socket!\")\n session.data += data\n except ssl.SSLError:\n # This can happen when select gets triggered\n # for an SSL socket and data has not yet been\n # read. Wait for it to get triggered again.\n return False\n\n # Whether or not all data was read.\n return len(session.data) == session.message_length"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send_connection_request(self):\n try:\n self.log.info(\"Sending ConnectionRequest for Monitor %s.\"\n % self.monitor_id)\n # Send connection request and perform a receive to ensure\n # request is authenticated.\n # Protocol Version = 1.\n payload = struct.pack('!H', 0x01)\n # Username Length.\n payload += struct.pack('!H', len(self.client.username))\n # Username.\n payload += six.b(self.client.username)\n # Password Length.\n payload += struct.pack('!H', len(self.client.password))\n # Password.\n payload += six.b(self.client.password)\n # Monitor ID.\n payload += struct.pack('!L', int(self.monitor_id))\n\n # Header 6 Bytes : Type [2 bytes] & Length [4 Bytes]\n # ConnectionRequest is Type 0x01.\n data = struct.pack(\"!HL\", CONNECTION_REQUEST, len(payload))\n\n # The full payload.\n data += payload\n\n # Send Connection Request.\n self.socket.send(data)\n\n # Set a 60 second blocking on recv, if we don't get any data\n # within 60 seconds, timeout which will throw an exception.\n self.socket.settimeout(60)\n\n # Should receive 10 bytes with ConnectionResponse.\n response = self.socket.recv(10)\n\n # Make socket blocking.\n self.socket.settimeout(0)\n\n if len(response) != 10:\n raise PushException(\"Length of Connection Request Response \"\n \"(%d) is not 10.\" % len(response))\n\n # Type\n response_type = int(struct.unpack(\"!H\", response[0:2])[0])\n if response_type != CONNECTION_RESPONSE:\n raise PushException(\n \"Connection Response Type (%d) is not \"\n \"ConnectionResponse Type (%d).\" % (response_type, CONNECTION_RESPONSE))\n\n status_code = struct.unpack(\"!H\", response[6:8])[0]\n self.log.info(\"Got ConnectionResponse for Monitor %s. Status %s.\"\n % (self.monitor_id, status_code))\n if status_code != STATUS_OK:\n raise PushException(\"Connection Response Status Code (%d) is \"\n \"not STATUS_OK (%d).\" % (status_code, STATUS_OK))\n except Exception as exception:\n # TODO(posborne): This is bad! It isn't necessarily a socket exception!\n # Likely a socket exception, close it and raise an exception.\n self.socket.close()\n self.socket = None\n raise exception", "response": "Send a ConnectionRequest to the iDigi server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nstart a TCP connection to Device Cloud and sends a ConnectionRequest message.", "response": "def start(self):\n \"\"\"Creates a TCP connection to Device Cloud and sends a ConnectionRequest message\"\"\"\n self.log.info(\"Starting Insecure Session for Monitor %s\" % self.monitor_id)\n if self.socket is not None:\n raise Exception(\"Socket already established for %s.\" % self)\n\n try:\n self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n self.socket.connect((self.client.hostname, PUSH_OPEN_PORT))\n self.socket.setblocking(0)\n except socket.error as exception:\n self.socket.close()\n self.socket = None\n raise\n\n self.send_connection_request()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstop the session and close the socket associated with this session.", "response": "def stop(self):\n \"\"\"Stop/Close this session\n\n Close the socket associated with this session and puts Session\n into a state such that it can be re-established later.\n \"\"\"\n if self.socket is not None:\n self.socket.close()\n self.socket = None\n self.data = None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstarting the SSL connection to the iDigi Server and sends a ConnectionRequest message.", "response": "def start(self):\n \"\"\"\n Creates a SSL connection to the iDigi Server and sends a\n ConnectionRequest message.\n \"\"\"\n self.log.info(\"Starting SSL Session for Monitor %s.\"\n % self.monitor_id)\n if self.socket is not None:\n raise Exception(\"Socket already established for %s.\" % self)\n\n try:\n # Create socket, wrap in SSL and connect.\n self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n # Validate that certificate server uses matches what we expect.\n if self.ca_certs is not None:\n self.socket = ssl.wrap_socket(self.socket,\n cert_reqs=ssl.CERT_REQUIRED,\n ca_certs=self.ca_certs)\n else:\n self.socket = ssl.wrap_socket(self.socket)\n\n self.socket.connect((self.client.hostname, PUSH_SECURE_PORT))\n self.socket.setblocking(0)\n except Exception as exception:\n self.socket.close()\n self.socket = None\n raise exception\n\n self.send_connection_request()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _consume_queue(self):\n while True:\n session, block_id, raw_data = self._queue.get()\n data = json.loads(raw_data.decode('utf-8')) # decode as JSON\n try:\n result = session.callback(data)\n if result is None:\n self.log.warn(\"Callback %r returned None, expected boolean. Messages \"\n \"are not marked as received unless True is returned\", session.callback)\n elif result:\n # Send a Successful PublishMessageReceived with the\n # block id sent in request\n if self._write_queue is not None:\n response_message = struct.pack('!HHH',\n PUBLISH_MESSAGE_RECEIVED,\n block_id, 200)\n self._write_queue.put((session.socket, response_message))\n except Exception as exception:\n self.log.exception(exception)\n\n self._queue.task_done()", "response": "Continually blocks until data is received from the internal queue and sends a PublishMessageReceived\n to the client."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef queue_callback(self, session, block_id, data):\n self._queue.put((session, block_id, data))", "response": "Queues up a callback event to occur for a given session with the given\n payload data. Will block if the queue is full."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _restart_session(self, session):\n # remove old session key, if socket is None, that means the\n # session was closed by user and there is no need to restart.\n if session.socket is not None:\n self.log.info(\"Attempting restart session for Monitor Id %s.\"\n % session.monitor_id)\n del self.sessions[session.socket.fileno()]\n session.stop()\n session.start()\n self.sessions[session.socket.fileno()] = session", "response": "Restarts and re - establishes the session."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites to the master socket.", "response": "def _writer(self):\n \"\"\"\n Indefinitely checks the writer queue for data to write\n to socket.\n \"\"\"\n while not self.closed:\n try:\n sock, data = self._write_queue.get(timeout=0.1)\n self._write_queue.task_done()\n sock.send(data)\n except Empty:\n pass # nothing to write after timeout\n except socket.error as err:\n if err.errno == errno.EBADF:\n self._clean_dead_sessions()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntraverses sessions to determine if any sockets were removed (indicates a stopped session). In these cases, remove the session.", "response": "def _clean_dead_sessions(self):\n \"\"\"\n Traverses sessions to determine if any sockets\n were removed (indicates a stopped session).\n In these cases, remove the session.\n \"\"\"\n for sck in list(self.sessions.keys()):\n session = self.sessions[sck]\n if session.socket is None:\n del self.sessions[sck]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _select(self):\n try:\n while not self.closed:\n try:\n inputready = select.select(self.sessions.keys(), [], [], 0.1)[0]\n for sock in inputready:\n session = self.sessions[sock]\n sck = session.socket\n\n if sck is None:\n # Socket has since been deleted, continue\n continue\n\n # If no defined message length, nothing has been\n # consumed yet, parse the header.\n if session.message_length == 0:\n # Read header information before receiving rest of\n # message.\n response_type = _read_msg_header(session)\n if response_type == NO_DATA:\n # No data could be read, assume socket closed.\n if session.socket is not None:\n self.log.error(\"Socket closed for Monitor %s.\" % session.monitor_id)\n self._restart_session(session)\n continue\n elif response_type == INCOMPLETE:\n # More Data to be read. Continue.\n continue\n elif response_type != PUBLISH_MESSAGE:\n self.log.warn(\"Response Type (%x) does not match PublishMessage (%x)\"\n % (response_type, PUBLISH_MESSAGE))\n continue\n\n try:\n if not _read_msg(session):\n # Data not completely read, continue.\n continue\n except PushException as err:\n # If Socket is None, it was closed,\n # otherwise it was closed when it shouldn't\n # have been restart it.\n session.data = six.b(\"\")\n session.message_length = 0\n\n if session.socket is None:\n del self.sessions[sck]\n else:\n self.log.exception(err)\n self._restart_session(session)\n continue\n\n # We received full payload,\n # clear session data and parse it.\n data = session.data\n session.data = six.b(\"\")\n session.message_length = 0\n block_id = struct.unpack('!H', data[0:2])[0]\n compression = struct.unpack('!B', data[4:5])[0]\n payload = data[10:]\n\n if compression == 0x01:\n # Data is compressed, uncompress it.\n payload = zlib.decompress(payload)\n\n # Enqueue payload into a callback queue to be\n # invoked\n self._callback_pool.queue_callback(session, block_id, payload)\n except select.error as err:\n # Evaluate sessions if we get a bad file descriptor, if\n # socket is gone, delete the session.\n if err.args[0] == errno.EBADF:\n self._clean_dead_sessions()\n except Exception as err:\n self.log.exception(err)\n finally:\n for session in self.sessions.values():\n if session is not None:\n session.stop()", "response": "Performs a socket select and returns a list of the unique identifiers."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialize the IO and Writer threads.", "response": "def _init_threads(self):\n \"\"\"Initializes the IO and Writer threads\"\"\"\n if self._io_thread is None:\n self._io_thread = Thread(target=self._select)\n self._io_thread.start()\n\n if self._writer_thread is None:\n self._writer_thread = Thread(target=self._writer)\n self._writer_thread.start()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_session(self, callback, monitor_id):\n self.log.info(\"Creating Session for Monitor %s.\" % monitor_id)\n session = SecurePushSession(callback, monitor_id, self, self._ca_certs) \\\n if self._secure else PushSession(callback, monitor_id, self)\n\n session.start()\n self.sessions[session.socket.fileno()] = session\n\n self._init_threads()\n return session", "response": "Creates and returns a PushSession instance based on the input monitor_id and callback."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef stop(self):\n if self._io_thread is not None:\n self.log.info(\"Waiting for I/O thread to stop...\")\n self.closed = True\n self._io_thread.join()\n\n if self._writer_thread is not None:\n self.log.info(\"Waiting for Writer Thread to stop...\")\n self.closed = True\n self._writer_thread.join()\n\n self.log.info(\"All worker threads stopped.\")", "response": "Stops all session activity. Blocks until io and writer threads dies\n ."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplot the original data in a graph above the plot of the dtwed data", "response": "def plotF0(fromTuple, toTuple, mergeTupleList, fnFullPath):\n '''\n Plots the original data in a graph above the plot of the dtw'ed data\n '''\n _matplotlibCheck()\n \n plt.hold(True)\n\n fig, (ax0) = plt.subplots(nrows=1)\n\n # Old data\n plot1 = ax0.plot(fromTuple[0], fromTuple[1], color='red',\n linewidth=2, label=\"From\")\n plot2 = ax0.plot(toTuple[0], toTuple[1], color='blue',\n linewidth=2, label=\"To\")\n ax0.set_title(\"Plot of F0 Morph\")\n plt.ylabel('Pitch (hz)')\n plt.xlabel('Time (s)')\n\n # Merge data\n colorValue = 0\n colorStep = 255.0 / len(mergeTupleList)\n for timeList, valueList in mergeTupleList:\n colorValue += colorStep\n hexValue = \"#%02x0000\" % int(255 - colorValue)\n if int(colorValue) == 255:\n ax0.plot(timeList, valueList, color=hexValue, linewidth=1,\n label=\"Merged line, final iteration\")\n else:\n ax0.plot(timeList, valueList, color=hexValue, linewidth=1)\n\n plt.legend(loc=1, borderaxespad=0.)\n# plt.legend([plot1, plot2, plot3], [\"From\", \"To\", \"Merged line\"])\n\n plt.savefig(fnFullPath, dpi=300, bbox_inches='tight')\n plt.close(fig)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the pitch for the given data for the given tier", "response": "def getPitchForIntervals(data, tgFN, tierName):\n '''\n Preps data for use in f0Morph\n '''\n tg = tgio.openTextgrid(tgFN)\n data = tg.tierDict[tierName].getValuesInIntervals(data)\n data = [dataList for _, dataList in data]\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecodes a string in the numbering format of pinyin to text with the appropriate tone marks (\"n\u01d0h\u01ceo\").", "response": "def decode(s):\n \"\"\"\n Converts text in the numbering format of pinyin (\"ni3hao3\") to text with the \n appropriate tone marks (\"n\u01d0h\u01ceo\").\n \"\"\"\n\n s = s.lower()\n r = \"\"\n t = \"\"\n for c in s:\n if c >= 'a' and c <= 'z':\n t += c\n elif c == ':':\n try:\n if t[-1] == 'u':\n t = t[:-1] + u\"\\u00fc\"\n except:\n pass\n else:\n if c >= '0' and c <= '5':\n tone = int(c) % 5\n if tone != 0:\n m = re.search(u\"[aoeiuv\\u00fc]+\", t)\n if m is None:\n t += c\n elif len(m.group(0)) == 1:\n t = t[:m.start(0)] + PinyinToneMark[tone][PinyinToneMark[0].index(m.group(0))] + t[m.end(0):]\n else:\n if 'a' in t:\n t = t.replace(\"a\", PinyinToneMark[tone][0])\n elif 'o' in t:\n t = t.replace(\"o\", PinyinToneMark[tone][1])\n elif 'e' in t:\n t = t.replace(\"e\", PinyinToneMark[tone][2])\n elif t.endswith(\"ui\"):\n t = t.replace(\"i\", PinyinToneMark[tone][3])\n elif t.endswith(\"iu\"):\n t = t.replace(\"u\", PinyinToneMark[tone][4])\n else:\n t += \"!\"\n r += t\n t = \"\"\n r += t\n return r"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadjust peak height of the peak.", "response": "def adjustPeakHeight(self, heightAmount):\n '''\n Adjust peak height\n \n The foot of the accent is left unchanged and intermediate\n values are linearly scaled\n '''\n if heightAmount == 0:\n return\n \n pitchList = [f0V for _, f0V in self.pointList]\n minV = min(pitchList)\n maxV = max(pitchList)\n scale = lambda x, y: x + y * (x - minV) / float(maxV - minV)\n \n self.pointList = [(timeV, scale(f0V, heightAmount))\n for timeV, f0V in self.pointList]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a plateau to the current object.", "response": "def addPlateau(self, plateauAmount, pitchSampFreq=None):\n '''\n Add a plateau\n \n A negative plateauAmount will move the peak backwards.\n A positive plateauAmount will move the peak forwards.\n \n All points on the side of the peak growth will also get moved.\n i.e. the slope of the peak does not change. The accent gets\n wider instead.\n \n If pitchSampFreq=None, the plateau will only be specified by\n the start and end points of the plateau\n '''\n if plateauAmount == 0:\n return\n \n maxPoint = self.pointList[self.peakI]\n \n # Define the plateau\n if pitchSampFreq is not None:\n numSteps = abs(int(plateauAmount / pitchSampFreq))\n timeChangeList = [stepV * pitchSampFreq\n for stepV in\n range(0, numSteps + 1)]\n else:\n timeChangeList = [plateauAmount, ]\n \n # Shift the side being pushed by the plateau\n if plateauAmount < 0: # Plateau moves left of the peak\n leftSide = self.pointList[:self.peakI]\n rightSide = self.pointList[self.peakI:]\n \n plateauPoints = [(maxPoint[0] + timeChange, maxPoint[1])\n for timeChange in timeChangeList]\n leftSide = [(timeV + plateauAmount, f0V)\n for timeV, f0V in leftSide]\n self.netLeftShift += plateauAmount\n \n elif plateauAmount > 0: # Plateau moves right of the peak\n leftSide = self.pointList[:self.peakI + 1]\n rightSide = self.pointList[self.peakI + 1:]\n \n plateauPoints = [(maxPoint[0] + timeChange, maxPoint[1])\n for timeChange in timeChangeList]\n rightSide = [(timeV + plateauAmount, f0V)\n for timeV, f0V in rightSide]\n self.netRightShift += plateauAmount\n \n self.pointList = leftSide + plateauPoints + rightSide"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmoves the whole accent earlier or later or later AttributeNames", "response": "def shiftAccent(self, shiftAmount):\n '''\n Move the whole accent earlier or later\n '''\n if shiftAmount == 0:\n return\n \n self.pointList = [(time + shiftAmount, pitch)\n for time, pitch in self.pointList]\n \n # Update shift amounts\n if shiftAmount < 0:\n self.netLeftShift += shiftAmount\n elif shiftAmount >= 0:\n self.netRightShift += shiftAmount"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes points from another list that overlap with points in this list.", "response": "def deleteOverlapping(self, targetList):\n '''\n Erase points from another list that overlap with points in this list\n '''\n start = self.pointList[0][0]\n stop = self.pointList[-1][0]\n \n if self.netLeftShift < 0:\n start += self.netLeftShift\n \n if self.netRightShift > 0:\n stop += self.netRightShift\n \n targetList = _deletePoints(targetList, start, stop)\n \n return targetList"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nintegrates the pitch values of the accent into a larger pitch contour", "response": "def reintegrate(self, fullPointList):\n '''\n Integrates the pitch values of the accent into a larger pitch contour\n '''\n # Erase the original region of the accent\n fullPointList = _deletePoints(fullPointList, self.minT, self.maxT)\n \n # Erase the new region of the accent\n fullPointList = self.deleteOverlapping(fullPointList)\n \n # Add the accent into the full pitch list\n outputPointList = fullPointList + self.pointList\n outputPointList.sort()\n \n return outputPointList"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a file with crappy encoding to a new file.", "response": "def convert(filename, new_filename=None, overwrite=False, to_encoding='utf-8', force=True):\n \"\"\" Convert file with crappy encoding to a new proper encoding (or vice versa if you wish).\n\n filename -- the name, partial path or full path of the file you want to encode to a new encoding\n new_filename -- (optional) the name of the new file to be generated using the new encoding\n overwrite -- if `new_filename` is omitted, set this to True to change the supplied file's \n encoding and not bother creating a new file (be careful! loss of information is likely)\n to_encoding -- the name of the encoding you wish to convert to (utf-8 by default)\n force -- Encode even if the current file is already in the correct encoding.\n \"\"\"\n logging.info('Opening file %s' % filename)\n f = open(filename)\n detection = chardet.detect(f.read())\n f.close()\n encoding = detection.get('encoding')\n confidence = detection.get('confidence')\n logging.info('I think it is %s with %.1f%% confidence' % (encoding, confidence * 100.0))\n\n delete_original = bool(new_filename) == False and overwrite\n if not new_filename or new_filename == filename:\n # use the current filename, but add the encoding to the name (while keeping extension intact)\n base_name, ext = os.path.splitext(filename)\n new_filename = base_name + '_%s' % to_encoding + ext\n\n if not encoding.lower() == to_encoding.lower():\n logging.info('Converting to %s with iconv...' % to_encoding)\n else:\n logging.info('Already in correct encoding.')\n if force:\n logging.info('Going ahead anyway, because force == True (the force is strong with this one)')\n else:\n logging.warning('Stopping. Use force = True if you want to force the encoding.')\n return None\n\n # command example: iconv -f gb18030 -t utf-8 chs.srt > chs-utf8.srt\n # \"iconv\" does not support -o parameter now and use stdout to instead.\n with open(new_filename, 'w') as output_file:\n p = subprocess.Popen(['iconv', '-f', encoding, '-t', to_encoding + \"//IGNORE\", \\\n os.path.abspath(filename)], shell=False, \\\n stdout=output_file, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)\n retval = p.wait()\n\n if delete_original and os.path.isfile(new_filename):\n os.remove(filename)\n os.rename(new_filename, filename)\n new_filename = filename\n\n return new_filename"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndetects the encoding of a file.", "response": "def detect(filename, include_confidence=False):\n \"\"\"\n Detect the encoding of a file.\n\n Returns only the predicted current encoding as a string.\n\n If `include_confidence` is True, \n Returns tuple containing: (str encoding, float confidence)\n \"\"\"\n f = open(filename)\n detection = chardet.detect(f.read())\n f.close()\n encoding = detection.get('encoding')\n confidence = detection.get('confidence')\n if include_confidence:\n return (encoding, confidence)\n return encoding"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _t(unistr, charset_from, charset_to):\n # if type(unistr) is str:\n # try:\n # unistr = unistr.decode('utf-8')\n # # Python 3 returns AttributeError when .decode() is called on a str\n # # This means it is already unicode.\n # except AttributeError:\n # pass\n # try:\n # if type(unistr) is not unicode:\n # return unistr\n # # Python 3 returns NameError because unicode is not a type.\n # except NameError:\n # pass\n\n chars = []\n for c in unistr:\n idx = charset_from.find(c)\n chars.append(charset_to[idx] if idx!=-1 else c)\n return u''.join(chars)", "response": "This function is a unexposed function is responsibility for translation internal."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef identify(text):\n filtered_text = set(list(text)).intersection(ALL_CHARS)\n if len(filtered_text) is 0:\n return None\n if filtered_text.issubset(SHARED_CHARS):\n return EITHER\n if filtered_text.issubset(TRAD_CHARS):\n return TRAD\n if filtered_text.issubset(SIMP_CHARS):\n return SIMP\n if filtered_text.difference(TRAD_CHARS).issubset(SIMP_CHARS):\n return BOTH", "response": "Identify whether a string is simplified or traditional Chinese."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a sequence relative to the main sequence", "response": "def makeSequenceRelative(absVSequence):\n '''\n Puts every value in a list on a continuum between 0 and 1\n\n Also returns the min and max values (to reverse the process)\n '''\n\n if len(absVSequence) < 2 or len(set(absVSequence)) == 1:\n raise RelativizeSequenceException(absVSequence)\n\n minV = min(absVSequence)\n maxV = max(absVSequence)\n relativeSeq = [(value - minV) / (maxV - minV) for value in absVSequence]\n\n return relativeSeq, minV, maxV"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes every value in a sequence absolute", "response": "def makeSequenceAbsolute(relVSequence, minV, maxV):\n '''\n Makes every value in a sequence absolute\n '''\n\n return [(value * (maxV - minV)) + minV for value in relVSequence]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving normal pitch tier data puts the times on a scale from 0 to 1 Returns the start and end times of the process", "response": "def _makeTimingRelative(absoluteDataList):\n '''\n Given normal pitch tier data, puts the times on a scale from 0 to 1\n\n Input is a list of tuples of the form\n ([(time1, pitch1), (time2, pitch2),...]\n\n Also returns the start and end time so that the process can be reversed\n '''\n\n timingSeq = [row[0] for row in absoluteDataList]\n valueSeq = [list(row[1:]) for row in absoluteDataList]\n\n relTimingSeq, startTime, endTime = makeSequenceRelative(timingSeq)\n \n relDataList = [tuple([time, ] + row) for time, row\n in zip(relTimingSeq, valueSeq)]\n\n return relDataList, startTime, endTime"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _makeTimingAbsolute(relativeDataList, startTime, endTime):\n '''\n Maps values from 0 to 1 to the provided start and end time\n\n Input is a list of tuples of the form\n ([(time1, pitch1), (time2, pitch2),...]\n '''\n\n timingSeq = [row[0] for row in relativeDataList]\n valueSeq = [list(row[1:]) for row in relativeDataList]\n \n absTimingSeq = makeSequenceAbsolute(timingSeq, startTime, endTime)\n\n absDataList = [tuple([time, ] + row) for time, row\n in zip(absTimingSeq, valueSeq)]\n\n return absDataList", "response": "Make a list of tuples of the form time1 pitch1 time2 pitch3..."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _getSmallestDifference(inputList, targetVal):\n '''\n Returns the value in inputList that is closest to targetVal\n \n Iteratively splits the dataset in two, so it should be pretty fast\n '''\n targetList = inputList[:]\n retVal = None\n while True:\n # If we're down to one value, stop iterating\n if len(targetList) == 1:\n retVal = targetList[0]\n break\n halfPoint = int(len(targetList) / 2.0) - 1\n a = targetList[halfPoint]\n b = targetList[halfPoint + 1]\n \n leftDiff = abs(targetVal - a)\n rightDiff = abs(targetVal - b)\n \n # If the distance is 0, stop iterating, the targetVal is present\n # in the inputList\n if leftDiff == 0 or rightDiff == 0:\n retVal = targetVal\n break\n \n # Look at left half or right half\n if leftDiff < rightDiff:\n targetList = targetList[:halfPoint + 1]\n else:\n targetList = targetList[halfPoint + 1:]\n \n return retVal", "response": "Returns the value in inputList that is closest to targetVal Iteratively splits the dataset in two so it should be pretty fast"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _getNearestMappingIndexList(fromValList, toValList):\n '''\n Finds the indicies for data points that are closest to each other.\n\n The inputs should be in relative time, scaled from 0 to 1\n e.g. if you have [0, .1, .5., .9] and [0, .1, .2, 1]\n will output [0, 1, 1, 2]\n '''\n\n indexList = []\n for fromTimestamp in fromValList:\n smallestDiff = _getSmallestDifference(toValList, fromTimestamp)\n i = toValList.index(smallestDiff)\n indexList.append(i)\n\n return indexList", "response": "Returns the list of indices that are closest to each other in the input list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nyielding the data points from fromList into toList using the values 0 to 1 in stepList.", "response": "def morphDataLists(fromList, toList, stepList):\n '''\n Iteratively morph fromList into toList using the values 0 to 1 in stepList\n \n stepList: a value of 0 means no change and a value of 1 means a complete\n change to the other value\n '''\n\n # If there are more than 1 pitch value, then we align the data in\n # relative time.\n # Each data point comes with a timestamp. The earliest timestamp is 0\n # and the latest timestamp is 1. Using this method, for each relative\n # timestamp in the source list, we find the closest relative timestamp\n # in the target list. Just because two pitch values have the same index\n # in the source and target lists does not mean that they correspond to\n # the same speech event.\n fromListRel, fromStartTime, fromEndTime = _makeTimingRelative(fromList)\n toListRel = _makeTimingRelative(toList)[0]\n\n # If fromList has more points, we'll have flat areas\n # If toList has more points, we'll might miss peaks or valleys\n fromTimeList = [dataTuple[0] for dataTuple in fromListRel]\n toTimeList = [dataTuple[0] for dataTuple in toListRel]\n indexList = _getNearestMappingIndexList(fromTimeList, toTimeList)\n alignedToPitchRel = [toListRel[i] for i in indexList]\n\n for stepAmount in stepList:\n newPitchList = []\n\n # Perform the interpolation\n for fromTuple, toTuple in zip(fromListRel, alignedToPitchRel):\n fromTime, fromValue = fromTuple\n toTime, toValue = toTuple\n\n # i + 1 b/c i_0 = 0 = no change\n newValue = fromValue + (stepAmount * (toValue - fromValue))\n newTime = fromTime + (stepAmount * (toTime - fromTime))\n\n newPitchList.append((newTime, newValue))\n\n newPitchList = _makeTimingAbsolute(newPitchList, fromStartTime,\n fromEndTime)\n\n yield stepAmount, newPitchList"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadjusting the values in fromPitchList to have the same average as toPitchList Because other manipulations can alter the average pitch, morphing the pitch is the last pitch manipulation that should be done After the morphing, the code removes any values below zero, thus the final average might not match the target average.", "response": "def morphAveragePitch(fromDataList, toDataList):\n '''\n Adjusts the values in fromPitchList to have the same average as toPitchList\n \n Because other manipulations can alter the average pitch, morphing the pitch\n is the last pitch manipulation that should be done\n \n After the morphing, the code removes any values below zero, thus the\n final average might not match the target average.\n '''\n \n timeList, fromPitchList = zip(*fromDataList)\n toPitchList = [pitchVal for _, pitchVal in toDataList]\n \n # Zero pitch values aren't meaningful, so filter them out if they are\n # in the dataset\n fromListNoZeroes = [val for val in fromPitchList if val > 0]\n fromAverage = sum(fromListNoZeroes) / float(len(fromListNoZeroes))\n \n toListNoZeroes = [val for val in toPitchList if val > 0]\n toAverage = sum(toListNoZeroes) / float(len(toListNoZeroes))\n \n newPitchList = [val - fromAverage + toAverage for val in fromPitchList]\n \n# finalAverage = sum(newPitchList) / float(len(newPitchList))\n \n # Removing zeroes and negative pitch values\n retDataList = [(time, pitchVal) for time, pitchVal\n in zip(timeList, newPitchList)\n if pitchVal > 0]\n \n return retDataList"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef morphRange(fromDataList, toDataList):\n '''\n Changes the scale of values in one distribution to that of another\n \n ie The maximum value in fromDataList will be set to the maximum value in\n toDataList. The 75% largest value in fromDataList will be set to the\n 75% largest value in toDataList, etc.\n \n Small sample sizes will yield results that are not very meaningful\n '''\n \n # Isolate and sort pitch values\n fromPitchList = [dataTuple[1] for dataTuple in fromDataList]\n toPitchList = [dataTuple[1] for dataTuple in toDataList]\n \n fromPitchListSorted = sorted(fromPitchList)\n toPitchListSorted = sorted(toPitchList)\n \n # Bin pitch values between 0 and 1\n fromListRel = makeSequenceRelative(fromPitchListSorted)[0]\n toListRel = makeSequenceRelative(toPitchListSorted)[0]\n \n # Find each values closest equivalent in the other list\n indexList = _getNearestMappingIndexList(fromListRel, toListRel)\n \n # Map the source pitch to the target pitch value\n # Pitch value -> get sorted position -> get corresponding position in\n # target list -> get corresponding pitch value = the new pitch value\n retList = []\n for time, pitch in fromDataList:\n fromI = fromPitchListSorted.index(pitch)\n toI = indexList[fromI]\n newPitch = toPitchListSorted[toI]\n \n retList.append((time, newPitch))\n \n return retList", "response": "This function morphs two sets of values in one distribution to that of another."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef quadraticInterpolation(valueList2d, numDegrees, n,\n startTime=None, endTime=None):\n '''\n Generates a series of points on a smooth curve that cross the given points\n \n numDegrees - the degrees of the fitted polynomial\n - the curve gets weird if this value is too high for the input\n n - number of points to output\n startTime/endTime/n - n points will be generated at evenly spaced\n intervals between startTime and endTime\n '''\n _numpyCheck()\n \n x, y = zip(*valueList2d)\n \n if startTime is None:\n startTime = x[0]\n if endTime is None:\n endTime = x[-1]\n \n polyFunc = np.poly1d(np.polyfit(x, y, numDegrees))\n \n newX = np.linspace(startTime, endTime, n)\n \n retList = [(n, polyFunc(n)) for n in newX]\n \n return retList", "response": "This function generates a series of points on a smooth curve that crosses the given valueList2d."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getIntervals(fn, tierName, filterFunc=None,\n includeUnlabeledRegions=False):\n '''\n Get information about the 'extract' tier, used by several merge scripts\n '''\n\n tg = tgio.openTextgrid(fn)\n \n tier = tg.tierDict[tierName]\n if includeUnlabeledRegions is True:\n tier = tgio._fillInBlanks(tier)\n\n entryList = tier.entryList\n if filterFunc is not None:\n entryList = [entry for entry in entryList if filterFunc(entry)]\n\n return entryList", "response": "Get information about the extract tier used by several merge scripts\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nusing praat to morph duration in one file to duration in another Praat uses the PSOLA algorithm", "response": "def changeDuration(fromWavFN, durationParameters, stepList, outputName,\n outputMinPitch, outputMaxPitch, praatEXE):\n '''\n Uses praat to morph duration in one file to duration in another\n\n Praat uses the PSOLA algorithm\n '''\n\n rootPath = os.path.split(fromWavFN)[0]\n\n # Prep output directories\n outputPath = join(rootPath, \"duration_resynthesized_wavs\")\n utils.makeDir(outputPath)\n \n durationTierPath = join(rootPath, \"duration_tiers\")\n utils.makeDir(durationTierPath)\n\n fromWavDuration = audio_scripts.getSoundFileDuration(fromWavFN)\n\n durationParameters = copy.deepcopy(durationParameters)\n # Pad any gaps with values of 1 (no change in duration)\n \n # No need to stretch out any pauses at the beginning\n if durationParameters[0][0] != 0:\n tmpVar = (0, durationParameters[0][0] - PRAAT_TIME_DIFF, 1)\n durationParameters.insert(0, tmpVar)\n\n # Or the end\n if durationParameters[-1][1] < fromWavDuration:\n durationParameters.append((durationParameters[-1][1] + PRAAT_TIME_DIFF,\n fromWavDuration, 1))\n\n # Create the praat script for doing duration manipulation\n for stepAmount in stepList:\n durationPointList = []\n for start, end, ratio in durationParameters:\n percentChange = 1 + (ratio - 1) * stepAmount\n durationPointList.append((start, percentChange))\n durationPointList.append((end, percentChange))\n \n outputPrefix = \"%s_%0.3g\" % (outputName, stepAmount)\n durationTierFN = join(durationTierPath,\n \"%s.DurationTier\" % outputPrefix)\n outputWavFN = join(outputPath, \"%s.wav\" % outputPrefix)\n durationTier = dataio.PointObject2D(durationPointList, dataio.DURATION,\n 0, fromWavDuration)\n durationTier.save(durationTierFN)\n \n praat_scripts.resynthesizeDuration(praatEXE,\n fromWavFN,\n durationTierFN,\n outputWavFN,\n outputMinPitch, outputMaxPitch)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting intervals for source audio files and target audio files and return the duration parameters of the morphological audio files.", "response": "def getMorphParameters(fromTGFN, toTGFN, tierName,\n filterFunc=None, useBlanks=False):\n '''\n Get intervals for source and target audio files\n \n Use this information to find out how much to stretch/shrink each source\n interval.\n \n The target values are based on the contents of toTGFN.\n '''\n \n if filterFunc is None:\n filterFunc = lambda entry: True # Everything is accepted\n \n fromEntryList = utils.getIntervals(fromTGFN, tierName,\n includeUnlabeledRegions=useBlanks)\n toEntryList = utils.getIntervals(toTGFN, tierName,\n includeUnlabeledRegions=useBlanks)\n\n fromEntryList = [entry for entry in fromEntryList if filterFunc(entry)]\n toEntryList = [entry for entry in toEntryList if filterFunc(entry)]\n\n assert(len(fromEntryList) == len(toEntryList))\n\n durationParameters = []\n for fromEntry, toEntry in zip(fromEntryList, toEntryList):\n fromStart, fromEnd = fromEntry[:2]\n toStart, toEnd = toEntry[:2]\n\n # Praat will ignore a second value appearing at the same time as\n # another so we give each start a tiny offset to distinguish intervals\n # that start and end at the same point\n toStart += PRAAT_TIME_DIFF\n fromStart += PRAAT_TIME_DIFF\n \n ratio = (toEnd - toStart) / float((fromEnd - fromStart))\n durationParameters.append((fromStart, fromEnd, ratio))\n \n return durationParameters"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the list of parameters that can be used to calculate the duration of the audio files in a tier.", "response": "def getManipulatedParamaters(tgFN, tierName, modFunc,\n filterFunc=None, useBlanks=False):\n '''\n Get intervals for source and target audio files\n \n Use this information to find out how much to stretch/shrink each source\n interval.\n \n The target values are based on modfunc.\n '''\n \n fromExtractInfo = utils.getIntervals(tgFN, tierName, filterFunc,\n useBlanks)\n \n durationParameters = []\n for fromInfoTuple in fromExtractInfo:\n fromStart, fromEnd = fromInfoTuple[:2]\n toStart, toEnd = modFunc(fromStart), modFunc(fromEnd)\n\n # Praat will ignore a second value appearing at the same time as\n # another so we give each start a tiny offset to distinguish intervals\n # that start and end at the same point\n toStart += PRAAT_TIME_DIFF\n fromStart += PRAAT_TIME_DIFF\n\n ratio = (toEnd - toStart) / float((fromEnd - fromStart))\n\n ratioTuple = (fromStart, fromEnd, ratio)\n durationParameters.append(ratioTuple)\n\n return durationParameters"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef textgridMorphDuration(fromTGFN, toTGFN):\n '''\n A convenience function. Morphs interval durations of one tg to another.\n \n This assumes the two textgrids have the same number of segments.\n '''\n fromTG = tgio.openTextgrid(fromTGFN)\n toTG = tgio.openTextgrid(toTGFN)\n adjustedTG = tgio.Textgrid()\n\n for tierName in fromTG.tierNameList:\n fromTier = fromTG.tierDict[tierName]\n toTier = toTG.tierDict[tierName]\n adjustedTier = fromTier.morph(toTier)\n adjustedTG.addTier(adjustedTier)\n\n return adjustedTG", "response": "A convenience function that returns a new tg that is a copy of the fromTGFN and toTGFN."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the duration of a wav file in seconds", "response": "def getSoundFileDuration(fn):\n '''\n Returns the duration of a wav file (in seconds)\n '''\n audiofile = wave.open(fn, \"r\")\n \n params = audiofile.getparams()\n framerate = params[2]\n nframes = params[3]\n \n duration = float(nframes) / framerate\n return duration"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef split_text(text, include_part_of_speech=False, strip_english=False, strip_numbers=False):\n\n if not include_part_of_speech:\n seg_list = pseg.cut(text)\n if strip_english:\n seg_list = filter(lambda x: not contains_english(x), seg_list)\n if strip_numbers:\n seg_list = filter(lambda x: not _is_number(x), seg_list)\n return list(map(lambda i: i.word, seg_list))\n else:\n seg_list = pseg.cut(text)\n objs = map(lambda w: (w.word, w.flag), seg_list)\n if strip_english:\n objs = filter(lambda x: not contains_english(x[0]), objs)\n if strip_english:\n objs = filter(lambda x: not _is_number(x[0]), objs)\n return objs\n\n # if was_traditional:\n # seg_list = map(tradify, seg_list)\n\n return list(seg_list)", "response": "u Split Chinese text at word boundaries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if a string contains a special character.", "response": "def is_special_atom(cron_atom, span):\n \"\"\"\n Returns a boolean indicating whether or not the string can be parsed by\n parse_atom to produce a static set. In the process of examining the\n string, the syntax of any special character uses is also checked.\n \"\"\"\n for special_char in ('%', '#', 'L', 'W'):\n if special_char not in cron_atom:\n continue\n\n if special_char == '#':\n if span != DAYS_OF_WEEK:\n raise ValueError(\"\\\"#\\\" invalid where used.\")\n elif not VALIDATE_POUND.match(cron_atom):\n raise ValueError(\"\\\"#\\\" syntax incorrect.\")\n elif special_char == \"W\":\n if span != DAYS_OF_MONTH:\n raise ValueError(\"\\\"W\\\" syntax incorrect.\")\n elif not(VALIDATE_W.match(cron_atom) and int(cron_atom[:-1]) > 0):\n raise ValueError(\"Invalid use of \\\"W\\\".\")\n elif special_char == \"L\":\n if span not in L_FIELDS:\n raise ValueError(\"\\\"L\\\" invalid where used.\")\n elif span == DAYS_OF_MONTH:\n if cron_atom != \"L\":\n raise ValueError(\"\\\"L\\\" must be alone in days of month.\")\n elif span == DAYS_OF_WEEK:\n if not VALIDATE_L_IN_DOW.match(cron_atom):\n raise ValueError(\"\\\"L\\\" syntax incorrect.\")\n elif special_char == \"%\":\n if not(cron_atom[1:].isdigit() and int(cron_atom[1:]) > 1):\n raise ValueError(\"\\\"%\\\" syntax incorrect.\")\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing a cron - style range of the internal list of the available cron - tables.", "response": "def parse_atom(parse, minmax):\n \"\"\"\n Returns a set containing valid values for a given cron-style range of\n numbers. The 'minmax' arguments is a two element iterable containing the\n inclusive upper and lower limits of the expression.\n\n Examples:\n >>> parse_atom(\"1-5\",(0,6))\n set([1, 2, 3, 4, 5])\n\n >>> parse_atom(\"*/6\",(0,23))\n set([0, 6, 12, 18])\n\n >>> parse_atom(\"18-6/4\",(0,23))\n set([18, 22, 0, 4])\n\n >>> parse_atom(\"*/9\",(0,23))\n set([0, 9, 18])\n \"\"\"\n parse = parse.strip()\n increment = 1\n if parse == '*':\n return set(xrange(minmax[0], minmax[1] + 1))\n elif parse.isdigit():\n # A single number still needs to be returned as a set\n value = int(parse)\n if value >= minmax[0] and value <= minmax[1]:\n return set((value,))\n else:\n raise ValueError(\"\\\"%s\\\" is not within valid range.\" % parse)\n elif '-' in parse or '/' in parse:\n divide = parse.split('/')\n subrange = divide[0]\n if len(divide) == 2:\n # Example: 1-3/5 or */7 increment should be 5 and 7 respectively\n increment = int(divide[1])\n\n if '-' in subrange:\n # Example: a-b\n prefix, suffix = [int(n) for n in subrange.split('-')]\n if prefix < minmax[0] or suffix > minmax[1]:\n raise ValueError(\"\\\"%s\\\" is not within valid range.\" % parse)\n elif subrange.isdigit():\n # Handle offset increments e.g. 5/15 to run at :05, :20, :35, and :50\n return set(xrange(int(subrange), minmax[1] + 1, increment))\n elif subrange == '*':\n # Include all values with the given range\n prefix, suffix = minmax\n else:\n raise ValueError(\"Unrecognized symbol \\\"%s\\\"\" % subrange)\n\n if prefix < suffix:\n # Example: 7-10\n return set(xrange(prefix, suffix + 1, increment))\n else:\n # Example: 12-4/2; (12, 12 + n, ..., 12 + m*n) U (n_0, ..., 4)\n noskips = list(xrange(prefix, minmax[1] + 1))\n noskips += list(xrange(minmax[0], suffix + 1))\n return set(noskips[::increment])\n else:\n raise ValueError(\"Atom \\\"%s\\\" not in a recognized format.\" % parse)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef compute_numtab(self):\n self.numerical_tab = []\n\n for field_str, span in zip(self.string_tab, FIELD_RANGES):\n split_field_str = field_str.split(',')\n if len(split_field_str) > 1 and \"*\" in split_field_str:\n raise ValueError(\"\\\"*\\\" must be alone in a field.\")\n\n unified = set()\n for cron_atom in split_field_str:\n # parse_atom only handles static cases\n if not(is_special_atom(cron_atom, span)):\n unified.update(parse_atom(cron_atom, span))\n\n self.numerical_tab.append(unified)\n\n if self.string_tab[2] == \"*\" and self.string_tab[4] != \"*\":\n self.numerical_tab[2] = set()\n elif self.string_tab[4] == \"*\" and self.string_tab[2] != \"*\":\n self.numerical_tab[4] = set()", "response": "Recomputes the sets for the static ranges of the trigger time."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_trigger(self, date_tuple, utc_offset=0):\n year, month, day, hour, mins = date_tuple\n given_date = datetime.date(year, month, day)\n zeroday = datetime.date(*self.epoch[:3])\n last_dom = calendar.monthrange(year, month)[-1]\n dom_matched = True\n\n # In calendar and datetime.date.weekday, Monday = 0\n given_dow = (datetime.date.weekday(given_date) + 1) % 7\n first_dow = (given_dow + 1 - day) % 7\n\n # Figure out how much time has passed from the epoch to the given date\n utc_diff = utc_offset - self.epoch[5]\n mod_delta_yrs = year - self.epoch[0]\n mod_delta_mon = month - self.epoch[1] + mod_delta_yrs * 12\n mod_delta_day = (given_date - zeroday).days\n mod_delta_hrs = hour - self.epoch[3] + mod_delta_day * 24 + utc_diff\n mod_delta_min = mins - self.epoch[4] + mod_delta_hrs * 60\n\n # Makes iterating through like components easier.\n quintuple = zip(\n (mins, hour, day, month, given_dow),\n self.numerical_tab,\n self.string_tab,\n (mod_delta_min, mod_delta_hrs, mod_delta_day, mod_delta_mon,\n mod_delta_day),\n FIELD_RANGES)\n\n for value, valid_values, field_str, delta_t, field_type in quintuple:\n # All valid, static values for the fields are stored in sets\n if value in valid_values:\n continue\n\n # The following for loop implements the logic for context\n # sensitive and epoch sensitive constraints. break statements,\n # which are executed when a match is found, lead to a continue\n # in the outer loop. If there are no matches found, the given date\n # does not match expression constraints, so the function returns\n # False as seen at the end of this for...else... construct.\n for cron_atom in field_str.split(','):\n if cron_atom[0] == '%':\n if not(delta_t % int(cron_atom[1:])):\n break\n\n elif '#' in cron_atom:\n D, N = int(cron_atom[0]), int(cron_atom[2])\n # Computes Nth occurence of D day of the week\n if (((D - first_dow) % 7) + 1 + 7 * (N - 1)) == day:\n break\n\n elif cron_atom[-1] == 'W':\n target = min(int(cron_atom[:-1]), last_dom)\n lands_on = (first_dow + target - 1) % 7\n if lands_on == 0:\n # Shift from Sun. to Mon. unless Mon. is next month\n if target < last_dom:\n target += 1\n else:\n target -= 2\n elif lands_on == 6:\n # Shift from Sat. to Fri. unless Fri. in prior month\n if target > 1:\n target -= 1\n else:\n target += 2\n\n # Break if the day is correct, and target is a weekday\n if target == day and (first_dow + target) % 7 > 1:\n break\n\n elif cron_atom[-1] == 'L':\n # In dom field, L means the last day of the month\n target = last_dom\n\n if field_type == DAYS_OF_WEEK:\n # Calculates the last occurence of given day of week\n desired_dow = int(cron_atom[:-1])\n target = (((desired_dow - first_dow) % 7) + 29)\n if target > last_dom:\n target -= 7\n\n if target == day:\n break\n else:\n # See 2010.11.15 of CHANGELOG\n if field_type == DAYS_OF_MONTH and self.string_tab[4] != '*':\n dom_matched = False\n continue\n elif field_type == DAYS_OF_WEEK and self.string_tab[2] != '*':\n # If we got here, then days of months validated so it does\n # not matter that days of the week failed.\n return dom_matched\n\n # None of the expressions matched which means this field fails\n return False\n\n # Arriving at this point means the date landed within the constraints\n # of all fields; the associated trigger should be fired.\n return True", "response": "Checks if the trigger is active at the given time."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef show(self):\n for rule in self.rules_list:\n result = \", \".join([str(check) for check, deny in rule])\n print(result)", "response": "Show the structure of self. rules_list only for debug."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run(self):\n failed_result = None\n for rule in self.rules_list:\n for check, deny in rule:\n if not check():\n failed_result = (False, deny)\n break\n else:\n return (True, None)\n return failed_result", "response": "Run self. rules_list.\n\n Return True if one rule channel has been passed otherwise return False and the deny method of the last failed rule."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_fraction(self, value):\n if value < 0:\n value *= -1\n value = min(value, 1)\n if self.horizontal:\n width = int(self.width * value)\n height = self.height\n else:\n width = self.width\n height = int(self.height * value)\n self.canvas.coords(self.meter, self.xpos, self.ypos,\n self.xpos + width, self.ypos + height)", "response": "Set the meter indicator. Value should be between 0 and 1."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_status(self):\n try:\n # all this may fail if the connection to the fritzbox is down\n self.update_connection_status()\n self.max_stream_rate.set(self.get_stream_rate_str())\n self.ip.set(self.status.external_ip)\n self.uptime.set(self.status.str_uptime)\n upstream, downstream = self.status.transmission_rate\n except IOError:\n # here we inform the user about being unable to\n # update the status informations\n pass\n else:\n # max_downstream and max_upstream may be zero if the\n # fritzbox is configured as ip-client.\n if self.max_downstream > 0:\n self.in_meter.set_fraction(\n 1.0 * downstream / self.max_downstream)\n if self.max_upstream > 0:\n self.out_meter.set_fraction(1.0 * upstream / self.max_upstream)\n self.update_traffic_info()\n self.after(1000, self.update_status)", "response": "Update status informations in tkinter window."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef format_num(num, unit='bytes'):\n if unit == 'bytes':\n extension = 'B'\n else:\n # if it's not bytes, it's bits\n extension = 'Bit'\n for dimension in (unit, 'K', 'M', 'G', 'T'):\n if num < 1024:\n if dimension == unit:\n return '%3.1f %s' % (num, dimension)\n return '%3.1f %s%s' % (num, dimension, extension)\n num /= 1024\n return '%3.1f P%s' % (num, extension)", "response": "Returns a human readable string of a byte - value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_headers(content_disposition, location=None, relaxed=False):\n\n LOGGER.debug(\n 'Content-Disposition %r, Location %r', content_disposition, location)\n\n if content_disposition is None:\n return ContentDisposition(location=location)\n\n # Both alternatives seem valid.\n if False:\n # Require content_disposition to be ascii bytes (0-127),\n # or characters in the ascii range\n content_disposition = ensure_charset(content_disposition, 'ascii')\n else:\n # We allow non-ascii here (it will only be parsed inside of\n # qdtext, and rejected by the grammar if it appears in\n # other places), although parsing it can be ambiguous.\n # Parsing it ensures that a non-ambiguous filename* value\n # won't get dismissed because of an unrelated ambiguity\n # in the filename parameter. But it does mean we occasionally\n # give less-than-certain values for some legacy senders.\n content_disposition = ensure_charset(content_disposition, 'iso-8859-1')\n\n # Check the caller already did LWS-folding (normally done\n # when separating header names and values; RFC 2616 section 2.2\n # says it should be done before interpretation at any rate).\n # Hopefully space still means what it should in iso-8859-1.\n # This check is a bit stronger that LWS folding, it will\n # remove CR and LF even if they aren't part of a CRLF.\n # However http doesn't allow isolated CR and LF in headers outside\n # of LWS.\n\n if relaxed:\n # Relaxed has two effects (so far):\n # the grammar allows a final ';' in the header;\n # we do LWS-folding, and possibly normalise other broken\n # whitespace, instead of rejecting non-lws-safe text.\n # XXX Would prefer to accept only the quoted whitespace\n # case, rather than normalising everything.\n content_disposition = normalize_ws(content_disposition)\n parser = content_disposition_value_relaxed\n else:\n # Turns out this is occasionally broken: two spaces inside\n # a quoted_string's qdtext. Firefox and Chrome save the two spaces.\n if not is_lws_safe(content_disposition):\n raise ValueError(\n content_disposition, 'Contains nonstandard whitespace')\n\n parser = content_disposition_value\n\n try:\n parsed = parser.parse(content_disposition)\n except FullFirstMatchException:\n return ContentDisposition(location=location)\n return ContentDisposition(\n disposition=parsed[0], assocs=parsed[1:], location=location)", "response": "Parse a Content - Disposition header value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nbuild a ContentDisposition from a requests response.", "response": "def parse_requests_response(response, **kwargs):\n \"\"\"Build a ContentDisposition from a requests (PyPI) response.\n \"\"\"\n\n return parse_headers(\n response.headers.get('content-disposition'), response.url, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a Content - Disposition header for a given filename.", "response": "def build_header(\n filename, disposition='attachment', filename_compat=None\n):\n \"\"\"Generate a Content-Disposition header for a given filename.\n\n For legacy clients that don't understand the filename* parameter,\n a filename_compat value may be given.\n It should either be ascii-only (recommended) or iso-8859-1 only.\n In the later case it should be a character string\n (unicode in Python 2).\n\n Options for generating filename_compat (only useful for legacy clients):\n - ignore (will only send filename*);\n - strip accents using unicode's decomposing normalisations,\n which can be done from unicode data (stdlib), and keep only ascii;\n - use the ascii transliteration tables from Unidecode (PyPI);\n - use iso-8859-1\n Ignore is the safest, and can be used to trigger a fallback\n to the document location (which can be percent-encoded utf-8\n if you control the URLs).\n\n See https://tools.ietf.org/html/rfc6266#appendix-D\n \"\"\"\n\n # While this method exists, it could also sanitize the filename\n # by rejecting slashes or other weirdness that might upset a receiver.\n\n if disposition != 'attachment':\n assert is_token(disposition)\n\n rv = disposition\n\n if is_token(filename):\n rv += '; filename=%s' % (filename, )\n return rv\n elif is_ascii(filename) and is_lws_safe(filename):\n qd_filename = qd_quote(filename)\n rv += '; filename=\"%s\"' % (qd_filename, )\n if qd_filename == filename:\n # RFC 6266 claims some implementations are iffy on qdtext's\n # backslash-escaping, we'll include filename* in that case.\n return rv\n elif filename_compat:\n if is_token(filename_compat):\n rv += '; filename=%s' % (filename_compat, )\n else:\n assert is_lws_safe(filename_compat)\n rv += '; filename=\"%s\"' % (qd_quote(filename_compat), )\n\n # alnum are already considered always-safe, but the rest isn't.\n # Python encodes ~ when it shouldn't, for example.\n rv += \"; filename*=utf-8''%s\" % (percent_encode(\n filename, safe=attr_chars_nonalnum, encoding='utf-8'), )\n\n # This will only encode filename_compat, if it used non-ascii iso-8859-1.\n return rv.encode('iso-8859-1')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef filename_sanitized(self, extension, default_filename='file'):\n\n assert extension\n assert extension[0] != '.'\n assert default_filename\n assert '.' not in default_filename\n extension = '.' + extension\n\n fname = self.filename_unsafe\n if fname is None:\n fname = default_filename\n fname = posixpath.basename(fname)\n fname = os.path.basename(fname)\n fname = fname.lstrip('.')\n if not fname:\n fname = default_filename\n if not fname.endswith(extension):\n fname = fname + extension\n return fname", "response": "Returns a filename that is safer to use on the filesystem."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef transmission_rate(self):\n sent = self.bytes_sent\n received = self.bytes_received\n traffic_call = time.time()\n time_delta = traffic_call - self.last_traffic_call\n upstream = int(1.0 * (sent - self.last_bytes_sent)/time_delta)\n downstream = int(1.0 * (received - self.last_bytes_received)/time_delta)\n self.last_bytes_sent = sent\n self.last_bytes_received = received\n self.last_traffic_call = traffic_call\n return upstream, downstream", "response": "Returns the upstream downstream values as a tuple in bytes per\n second. Use this for periodical calling."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a tuple of human readable transmission rates in bytes.", "response": "def str_transmission_rate(self):\n \"\"\"Returns a tuple of human readable transmission rates in bytes.\"\"\"\n upstream, downstream = self.transmission_rate\n return (\n fritztools.format_num(upstream),\n fritztools.format_num(downstream)\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef max_bit_rate(self):\n status = self.fc.call_action('WANCommonInterfaceConfig',\n 'GetCommonLinkProperties')\n downstream = status['NewLayer1DownstreamMaxBitRate']\n upstream = status['NewLayer1UpstreamMaxBitRate']\n return upstream, downstream", "response": "Returns a tuple with the maximun upstream - and downstream - rate of the given connection."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a human readable max bit rate of the given connection.", "response": "def str_max_bit_rate(self):\n \"\"\"\n Returns a human readable maximun upstream- and downstream-rate\n of the given connection. The rate is given in bits/sec.\n \"\"\"\n upstream, downstream = self.max_bit_rate\n return (\n fritztools.format_rate(upstream, unit='bits'),\n fritztools.format_rate(downstream, unit ='bits')\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _body_builder(self, kwargs):\n p = {\n 'action_name': self.name,\n 'service_type': self.service_type,\n 'arguments': '',\n }\n if kwargs:\n arguments = [\n self.argument_template % {'name': k, 'value': v}\n for k, v in kwargs.items()\n ]\n p['arguments'] = ''.join(arguments)\n body = self.body_template.strip() % p\n return body", "response": "Helper method to construct the appropriate SOAP - body to call a\n FritzBox - Service."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef execute(self, **kwargs):\n headers = self.header.copy()\n headers['soapaction'] = '%s#%s' % (self.service_type, self.name)\n data = self.envelope.strip() % self._body_builder(kwargs)\n url = 'http://%s:%s%s' % (self.address, self.port, self.control_url)\n auth = None\n if self.password:\n auth=HTTPDigestAuth(self.user, self.password)\n response = requests.post(url, data=data, headers=headers, auth=auth)\n # lxml needs bytes, therefore response.content (not response.text)\n result = self.parse_response(response.content)\n return result", "response": "Calls the FritzBox action and returns a dictionary with the arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_response(self, response):\n result = {}\n root = etree.fromstring(response)\n for argument in self.arguments.values():\n try:\n value = root.find('.//%s' % argument.name).text\n except AttributeError:\n # will happen by searching for in-parameters and by\n # parsing responses with status_code != 200\n continue\n if argument.data_type.startswith('ui'):\n try:\n value = int(value)\n except ValueError:\n # should not happen\n value = None\n result[argument.name] = value\n return result", "response": "Parses the response from a FritzBox action - call into a dictionary with the received arguments - value pairs."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_modelname(self):\n xpath = '%s/%s' % (self.nodename('device'), self.nodename('modelName'))\n return self.root.find(xpath).text", "response": "Returns the FritzBox model name."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_services(self):\n result = []\n nodes = self.root.iterfind(\n './/ns:service', namespaces={'ns': self.namespace})\n for node in nodes:\n result.append(FritzService(\n node.find(self.nodename('serviceType')).text,\n node.find(self.nodename('controlURL')).text,\n node.find(self.nodename('SCPDURL')).text))\n return result", "response": "Returns a list of FritzService - objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _read_state_variables(self):\n nodes = self.root.iterfind(\n './/ns:stateVariable', namespaces={'ns': self.namespace})\n for node in nodes:\n key = node.find(self.nodename('name')).text\n value = node.find(self.nodename('dataType')).text\n self.state_variables[key] = value", "response": "Reads the stateVariable information from the xml - file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of FritzAction instances.", "response": "def get_actions(self):\n \"\"\"Returns a list of FritzAction instances.\"\"\"\n self._read_state_variables()\n actions = []\n nodes = self.root.iterfind(\n './/ns:action', namespaces={'ns': self.namespace})\n for node in nodes:\n action = FritzAction(self.service.service_type,\n self.service.control_url)\n action.name = node.find(self.nodename('name')).text\n action.arguments = self._get_arguments(node)\n actions.append(action)\n return actions"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a dictionary of arguments for the given action_node.", "response": "def _get_arguments(self, action_node):\n \"\"\"\n Returns a dictionary of arguments for the given action_node.\n \"\"\"\n arguments = {}\n argument_nodes = action_node.iterfind(\n r'./ns:argumentList/ns:argument', namespaces={'ns': self.namespace})\n for argument_node in argument_nodes:\n argument = self._get_argument(argument_node)\n arguments[argument.name] = argument\n return arguments"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a FritzActionArgument instance for the given argument_node.", "response": "def _get_argument(self, argument_node):\n \"\"\"\n Returns a FritzActionArgument instance for the given argument_node.\n \"\"\"\n argument = FritzActionArgument()\n argument.name = argument_node.find(self.nodename('name')).text\n argument.direction = argument_node.find(self.nodename('direction')).text\n rsv = argument_node.find(self.nodename('relatedStateVariable')).text\n # TODO: track malformed xml-nodes (i.e. misspelled)\n argument.data_type = self.state_variables.get(rsv, None)\n return argument"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads and evaluate the igddesc. xml file and the tr64desc. xml file.", "response": "def _read_descriptions(self, password):\n \"\"\"\n Read and evaluate the igddesc.xml file\n and the tr64desc.xml file if a password is given.\n \"\"\"\n descfiles = [FRITZ_IGD_DESC_FILE]\n if password:\n descfiles.append(FRITZ_TR64_DESC_FILE)\n for descfile in descfiles:\n parser = FritzDescParser(self.address, self.port, descfile)\n if not self.modelname:\n self.modelname = parser.get_modelname()\n services = parser.get_services()\n self._read_services(services)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _read_services(self, services):\n for service in services:\n parser = FritzSCDPParser(self.address, self.port, service)\n actions = parser.get_actions()\n service.actions = {action.name: action for action in actions}\n self.services[service.name] = service", "response": "Get actions from services."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef actionnames(self):\n actions = []\n for service_name in sorted(self.services.keys()):\n action_names = self.services[service_name].actions.keys()\n for action_name in sorted(action_names):\n actions.append((service_name, action_name))\n return actions", "response": "Returns a alphabetical sorted list of tuples with all known\n service - and action - names."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of tuples with all known arguments for the given service and action.", "response": "def get_action_arguments(self, service_name, action_name):\n \"\"\"\n Returns a list of tuples with all known arguments for the given\n service- and action-name combination. The tuples contain the\n argument-name, direction and data_type.\n \"\"\"\n return self.services[service_name].actions[action_name].info"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef call_action(self, service_name, action_name, **kwargs):\n action = self.services[service_name].actions[action_name]\n return action.execute(**kwargs)", "response": "Executes the given action. Raise a KeyError on unkown actions."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_hosts_info(self):\n result = []\n index = 0\n while index < self.host_numbers:\n host = self.get_generic_host_entry(index)\n result.append({\n 'ip': host['NewIPAddress'],\n 'name': host['NewHostName'],\n 'mac': host['NewMACAddress'],\n 'status': host['NewActive']})\n index += 1\n return result", "response": "Returns a list of dicts with information about the known hosts."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_executable(executable):\n '''\n Finds executable in PATH\n\n Returns:\n string or None\n '''\n logger = logging.getLogger(__name__)\n logger.debug(\"Checking executable '%s'...\", executable)\n executable_path = _find_executable(executable)\n found = executable_path is not None\n if found:\n logger.debug(\"Executable '%s' found: '%s'\", executable, executable_path)\n else:\n logger.debug(\"Executable '%s' not found\", executable)\n return executable_path", "response": "Finds executable in PATH and returns the path to the executable."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_network_connection(server, port):\n '''\n Checks if jasper can connect a network server.\n Arguments:\n server -- (optional) the server to connect with (Default:\n \"www.google.com\")\n Returns:\n True or False\n '''\n logger = logging.getLogger(__name__)\n logger.debug(\"Checking network connection to server '%s'...\", server)\n try:\n # see if we can resolve the host name -- tells us if there is\n # a DNS listening\n host = socket.gethostbyname(server)\n # connect to the host -- tells us if the host is actually\n # reachable\n sock = socket.create_connection((host, port), 2)\n sock.close()\n except Exception: # pragma: no cover\n logger.debug(\"Network connection not working\")\n return False\n logger.debug(\"Network connection working\")\n return True", "response": "Checks if jasper can connect to a network server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if a python package or module is importable.", "response": "def check_python_import(package_or_module):\n '''\n Checks if a python package or module is importable.\n Arguments:\n package_or_module -- the package or module name to check\n Returns:\n True or False\n '''\n logger = logging.getLogger(__name__)\n logger.debug(\"Checking python import '%s'...\", package_or_module)\n loader = pkgutil.get_loader(package_or_module)\n found = loader is not None\n if found:\n logger.debug(\"Python %s '%s' found\",\n \"package\" if loader.is_package(package_or_module)\n else \"module\", package_or_module)\n else: # pragma: no cover\n logger.debug(\"Python import '%s' not found\", package_or_module)\n return found"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef inject(self):\n with open(self.script_url, \"r\", encoding=\"utf8\") as f:\n self.selenium.execute_script(f.read())", "response": "Recursively injects aXe into all iframes and the top level document."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(self, context=None, options=None):\n template = (\n \"var callback = arguments[arguments.length - 1];\"\n + \"axe.run(%s).then(results => callback(results))\"\n )\n args = \"\"\n\n # If context parameter is passed, add to args\n if context is not None:\n args += \"%r\" % context\n # Add comma delimiter only if both parameters are passed\n if context is not None and options is not None:\n args += \",\"\n # If options parameter is passed, add to args\n if options is not None:\n args += \"%s\" % options\n\n command = template % args\n response = self.selenium.execute_async_script(command)\n return response", "response": "Run axe against the current page."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef report(self, violations):\n string = \"\"\n string += \"Found \" + str(len(violations)) + \" accessibility violations:\"\n for violation in violations:\n string += (\n \"\\n\\n\\nRule Violated:\\n\"\n + violation[\"id\"]\n + \" - \"\n + violation[\"description\"]\n + \"\\n\\tURL: \"\n + violation[\"helpUrl\"]\n + \"\\n\\tImpact Level: \"\n + violation[\"impact\"]\n + \"\\n\\tTags:\"\n )\n for tag in violation[\"tags\"]:\n string += \" \" + tag\n string += \"\\n\\tElements Affected:\"\n i = 1\n for node in violation[\"nodes\"]:\n for target in node[\"target\"]:\n string += \"\\n\\t\" + str(i) + \") Target: \" + target\n i += 1\n for item in node[\"all\"]:\n string += \"\\n\\t\\t\" + item[\"message\"]\n for item in node[\"any\"]:\n string += \"\\n\\t\\t\" + item[\"message\"]\n for item in node[\"none\"]:\n string += \"\\n\\t\\t\" + item[\"message\"]\n string += \"\\n\\n\\n\"\n return string", "response": "Returns a readable string of accessibility violations found."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write_results(self, data, name=None):\n\n if name:\n filepath = os.path.abspath(name)\n else:\n filepath = os.path.join(os.path.getcwd(), \"results.json\")\n\n with open(filepath, \"w\", encoding=\"utf8\") as f:\n try:\n f.write(unicode(json.dumps(data, indent=4)))\n except NameError:\n f.write(json.dumps(data, indent=4))", "response": "Write JSON to file with the specified name."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating an instance of an engine.", "response": "def create_engine(engine, options=None, defaults=None):\n '''\n Creates an instance of an engine.\n There is a two-stage instantiation process with engines.\n\n 1. ``options``:\n The keyword options to instantiate the engine class\n 2. ``defaults``:\n The default configuration for the engine (options often depends on instantiated TTS engine)\n '''\n if engine not in _ENGINE_MAP.keys():\n raise TTSError('Unknown engine %s' % engine)\n\n options = options or {}\n defaults = defaults or {}\n einst = _ENGINE_MAP[engine](**options)\n einst.configure_default(**defaults)\n return einst"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef classify(self, txt):\n '''\n Classifies text by language. Uses preferred_languages weighting.\n '''\n ranks = []\n for lang, score in langid.rank(txt):\n if lang in self.preferred_languages:\n score += self.preferred_factor\n ranks.append((lang, score))\n ranks.sort(key=lambda x: x[1], reverse=True)\n return ranks[0][0]", "response": "Classifies text by language. Uses preferred_languages weighting."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndetermines the preferred engine for a language.", "response": "def get_engine_for_lang(self, lang):\n '''\n Determines the preferred engine/voice for a language.\n '''\n for eng in self.engines:\n if lang in eng.languages.keys():\n return eng\n raise TTSError('Could not match language')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef say(self, txt, lang=None):\n '''\n Says the text.\n\n if ``lang`` is ``None``, then uses ``classify()`` to detect language.\n '''\n lang = lang or self.classify(txt)\n self.get_engine_for_lang(lang).say(txt, language=lang)", "response": "Says the text.\n\n if ``lang`` is ``None``, then uses ``classify()`` to detect language."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting default configuration. Raises TTSError on error.", "response": "def configure_default(self, **_options):\n '''\n Sets default configuration.\n\n Raises TTSError on error.\n '''\n language, voice, voiceinfo, options = self._configure(**_options)\n self.languages_options[language] = (voice, options)\n self.default_language = language\n self.default_options = options"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets language - specific configuration.", "response": "def configure(self, **_options):\n '''\n Sets language-specific configuration.\n\n Raises TTSError on error.\n '''\n language, voice, voiceinfo, options = self._configure(**_options)\n self.languages_options[language] = (voice, options)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef say(self, phrase, **_options):\n '''\n Says the phrase, optionally allows to select/override any voice options.\n '''\n language, voice, voiceinfo, options = self._configure(**_options)\n self._logger.debug(\"Saying '%s' with '%s'\", phrase, self.SLUG)\n self._say(phrase, language, voice, voiceinfo, options)", "response": "Says the phrase with the specified options."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef play(self, filename, translate=False): # pragma: no cover\n '''\n Plays the sounds.\n\n :filename: The input file name\n :translate: If True, it runs it through audioread which will translate from common compression formats to raw WAV.\n '''\n # FIXME: Use platform-independent and async audio-output here\n # PyAudio looks most promising, too bad about:\n # --allow-external PyAudio --allow-unverified PyAudio\n if translate:\n with tempfile.NamedTemporaryFile(suffix='.wav', delete=False) as f:\n fname = f.name\n with audioread.audio_open(filename) as f:\n with contextlib.closing(wave.open(fname, 'w')) as of:\n of.setnchannels(f.channels)\n of.setframerate(f.samplerate)\n of.setsampwidth(2)\n for buf in f:\n of.writeframes(buf)\n filename = fname\n\n if winsound:\n winsound.PlaySound(str(filename), winsound.SND_FILENAME)\n else:\n cmd = ['aplay', str(filename)]\n self._logger.debug('Executing %s', ' '.join([pipes.quote(arg) for arg in cmd]))\n subprocess.call(cmd)\n\n if translate:\n os.remove(fname)", "response": "Plays the sounds.\n\n :filename: The input file name\n :translate: If True, it runs it through audioread which will translate from common compression formats to raw WAV."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the appropriate collectd server.", "response": "def getCollectDServer(queue, cfg):\n \"\"\"Get the appropriate collectd server (multi processed or not)\"\"\"\n server = CollectDServerMP if cfg.collectd_workers > 1 else CollectDServer\n return server(queue, cfg)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _hashes_match(self, a, b):\n if len(a) != len(b):\n return False\n diff = 0\n if six.PY2:\n a = bytearray(a)\n b = bytearray(b)\n for x, y in zip(a, b):\n diff |= x ^ y\n return not diff", "response": "Constant time comparison of bytes for py3 strings for py2"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_sites(*basin_ids):\n\n # Resolve basin ids to HUC8s if needed\n basins = []\n for basin in basin_ids:\n if basin.isdigit() and len(basin) == 8:\n basins.append(basin)\n else:\n from climata.huc8 import get_huc8\n basins.extend(get_huc8(basin))\n\n # Load sites with data since 1900\n sites = StationMetaIO(\n basin=basins,\n parameter=list(elems.keys()),\n start_date='1900-01-01',\n end_date=date.today(),\n meta=ALL_META_FIELDS,\n )\n\n # Load all sites (to get sites without data)\n seen_sites = [site.uid for site in sites]\n nodata_sites = [\n site for site in StationMetaIO(basin=basins)\n if site.uid not in seen_sites\n ]\n\n # Determine the following from the site lists:\n seen_auths = set() # Which authority codes are actually used by any site\n seen_elems = set() # Which elems actually have data in any site\n ranges = {} # The overall period of record for each site\n for site in sites:\n\n for auth in site.sids.keys():\n seen_auths.add(auth)\n\n start, end = None, None\n for elem in site.valid_daterange:\n s, e = site.valid_daterange[elem]\n seen_elems.add(elem)\n if s is None or e is None:\n continue\n if start is None or s < start:\n start = s\n if end is None or e > end:\n end = e\n ranges[site.uid] = [start, end]\n\n # Check for authority codes that might not be in sites with data\n for site in nodata_sites:\n for auth in site.sids.keys():\n seen_auths.add(auth)\n\n # Print CSV headers (FIXME: use CsvFileIO for this?)\n seen_auths = sorted(seen_auths)\n seen_elems = sorted(seen_elems)\n print(\",\".join(\n ['ACIS uid', 'name']\n + seen_auths\n + ['latitude', 'longitude', 'start', 'end', 'years']\n + [elems[elem]['desc'] for elem in seen_elems]\n ))\n\n # Print sites with data\n for site in sites:\n\n # Determine if elems are available for entire period or shorter range\n start, end = ranges[site.uid]\n if start and end:\n years = end.year - start.year + 1\n elem_ranges = []\n for elem in seen_elems:\n estart, eend = site.valid_daterange[elem]\n if estart is None:\n erange = \"\"\n elif estart == start and eend == end:\n erange = \"period\"\n else:\n erange = \"%s to %s\" % (estart.date(), eend.date())\n elem_ranges.append(erange)\n\n # Output CSV row\n print(\",\".join(map(\n str,\n [site.uid, site.name]\n + [site.sids.get(auth, \"\") for auth in seen_auths]\n + [site.latitude, site.longitude]\n + [start.date(), end.date(), years]\n + elem_ranges\n )))\n\n # Print CSV rows for sites without data\n for site in nodata_sites:\n print(\",\".join(map(\n str,\n [site.uid, site.name]\n + [site.sids.get(auth, \"\") for auth in seen_auths]\n + [site.latitude, site.longitude]\n + [\"NO DATA\"]\n )))", "response": "Load metadata for all sites in basin_ids."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn all 8 - digit HUCs matching the given prefix", "response": "def get_huc8(prefix):\n \"\"\"\n Return all HUC8s matching the given prefix (e.g. 1801) or basin name\n (e.g. Klamath)\n \"\"\"\n if not prefix.isdigit():\n # Look up hucs by name\n name = prefix\n prefix = None\n for row in hucs:\n if row.basin.lower() == name.lower():\n # Use most general huc if two have the same name\n if prefix is None or len(row.huc) < len(prefix):\n prefix = row.huc\n\n if prefix is None:\n return []\n\n huc8s = []\n for row in hucs:\n # Return all 8-digit hucs with given prefix\n if len(row.huc) == 8 and row.huc.startswith(prefix):\n huc8s.append(row.huc)\n return huc8s"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert ACIS 'll' value into separate latitude and longitude.", "response": "def parse(self):\n \"\"\"\n Convert ACIS 'll' value into separate latitude and longitude.\n \"\"\"\n super(AcisIO, self).parse()\n\n # This is more of a \"mapping\" step than a \"parsing\" step, but mappers\n # only allow one-to-one mapping from input fields to output fields.\n for row in self.data:\n if 'meta' in row:\n row = row['meta']\n if 'll' in row:\n row['longitude'], row['latitude'] = row['ll']\n del row['ll']"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmaps the value of a resource field to a dictionary of values.", "response": "def map_value(self, field, value):\n \"\"\"\n Clean up some values returned from the web service.\n (overrides wq.io.mappers.BaseMapper)\n \"\"\"\n\n if field == 'sids':\n # Site identifiers are returned as \"[id] [auth_id]\";\n # Map to auth name for easier usability\n ids = {}\n for idinfo in value:\n id, auth = idinfo.split(' ')\n auth = AUTHORITY_BY_ID[auth]\n ids[auth['name']] = id\n return ids\n\n if field == 'valid_daterange':\n # Date ranges for each element are returned in an array\n # (sorted by the order the elements were were requested);\n # Convert to dictionary with element id as key\n elems, complex = self.getlist('parameter')\n ranges = {}\n for elem, val in zip(elems, value):\n if val:\n start, end = val\n ranges[elem] = (parse_date(start), parse_date(end))\n else:\n ranges[elem] = None, None\n return ranges\n return value"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the list of fields that are used for each station in the data IO.", "response": "def get_field_names(self):\n \"\"\"\n ACIS web service returns \"meta\" and \"data\" for each station;\n Use meta attributes as field names\n \"\"\"\n field_names = super(StationDataIO, self).get_field_names()\n if set(field_names) == set(['meta', 'data']):\n meta_fields = list(self.data[0]['meta'].keys())\n if set(meta_fields) < set(self.getvalue('meta')):\n meta_fields = self.getvalue('meta')\n field_names = list(meta_fields) + ['data']\n return field_names"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef usable_item(self, data):\n\n # Use metadata as item\n item = data['meta']\n\n # Add nested IO for data\n elems, elems_is_complex = self.getlist('parameter')\n if elems_is_complex:\n elems = [elem['name'] for elem in elems]\n\n add, add_is_complex = self.getlist('add')\n item['data'] = DataIO(\n data=data['data'],\n parameter=elems,\n add=add,\n start_date=self.getvalue('start_date'),\n end_date=self.getvalue('end_date'),\n )\n\n # TupleMapper will convert item to namedtuple\n return super(StationDataIO, self).usable_item(item)", "response": "Use meta and data attributes as item values and add nested IOs for iterating over data"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads MultiStnData data from a list of dictionaries.", "response": "def load_data(self, data):\n \"\"\"\n MultiStnData data results are arrays without explicit dates;\n Infer time series based on start date.\n \"\"\"\n\n dates = fill_date_range(self.start_date, self.end_date)\n for row, date in zip(data, dates):\n data = {'date': date}\n if self.add:\n # If self.add is set, results will contain additional\n # attributes (e.g. flags). In that case, create one row per\n # result, with attributes \"date\", \"elem\", \"value\", and one for\n # each item in self.add.\n for elem, vals in zip(self.parameter, row):\n data['elem'] = elem\n for add, val in zip(['value'] + self.add, vals):\n data[add] = val\n yield data\n else:\n # Otherwise, return one row per date, with \"date\" and each\n # element's value as attributes.\n for elem, val in zip(self.parameter, row):\n # namedtuple doesn't like numeric field names\n if elem.isdigit():\n elem = \"e%s\" % elem\n data[elem] = val\n yield data"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of field names depending on self. add setting.", "response": "def get_field_names(self):\n \"\"\"\n Different field names depending on self.add setting (see load_data)\n For BaseIO\n \"\"\"\n if self.add:\n return ['date', 'elem', 'value'] + [flag for flag in self.add]\n else:\n field_names = ['date']\n for elem in self.parameter:\n # namedtuple doesn't like numeric field names\n if elem.isdigit():\n elem = \"e%s\" % elem\n field_names.append(elem)\n return field_names"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning fills the date_list with the dates in the given date range.", "response": "def fill_date_range(start_date, end_date, date_format=None):\n \"\"\"\n Function accepts start date, end date, and format (if dates are strings)\n and returns a list of Python dates.\n \"\"\"\n\n if date_format:\n start_date = datetime.strptime(start_date, date_format).date()\n end_date = datetime.strptime(end_date, date_format).date()\n date_list = []\n while start_date <= end_date:\n date_list.append(start_date)\n start_date = start_date + timedelta(days=1)\n return date_list"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nenforcing rules and return parsed value", "response": "def parse(self, value):\n \"\"\"\n Enforce rules and return parsed value\n \"\"\"\n if self.required and value is None:\n raise ValueError(\"%s is required!\" % self.name)\n elif self.ignored and value is not None:\n warn(\"%s is ignored for this class!\" % self.name)\n elif not self.multi and isinstance(value, (list, tuple)):\n if len(value) > 1:\n raise ValueError(\n \"%s does not accept multiple values!\" % self.name\n )\n return value[0]\n elif self.multi and value is not None:\n if not isinstance(value, (list, tuple)):\n return [value]\n return value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing value into a base - 10 value.", "response": "def parse(self, value):\n \"\"\"\n Parse date\n \"\"\"\n value = super(DateOpt, self).parse(value)\n if value is None:\n return None\n if isinstance(value, str):\n value = self.parse_date(value)\n if isinstance(value, datetime) and self.date_only:\n value = value.date()\n return value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_filter_options(cls):\n attr = '_filter_options_%s' % id(cls)\n\n options = getattr(cls, attr, {})\n if options:\n return options\n\n for key in dir(cls):\n val = getattr(cls, key)\n if isinstance(val, FilterOpt):\n options[key] = val\n\n setattr(cls, attr, options)\n return options", "response": "Get all filter options defined on class and superclasses."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves given property from class or instance ensuring it is a list.", "response": "def getlist(self, name):\n \"\"\"\n Retrieve given property from class/instance, ensuring it is a list.\n Also determine whether the list contains simple text/numeric values or\n nested dictionaries (a \"complex\" list)\n \"\"\"\n value = self.getvalue(name)\n complex = {}\n\n def str_value(val):\n # TODO: nonlocal complex\n if isinstance(val, dict):\n complex['complex'] = True\n return val\n else:\n return str(val)\n\n if value is None:\n pass\n else:\n value = [str_value(val) for val in as_list(value)]\n\n return value, bool(complex)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the value of a parameter in a dictionary into.", "response": "def set_param(self, into, name):\n \"\"\"\n Set parameter key, noting whether list value is \"complex\"\n \"\"\"\n value, complex = self.getlist(name)\n if value is not None:\n into[name] = value\n return complex"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_params(self):\n params = {}\n complex = False\n\n for name, opt in self.filter_options.items():\n if opt.ignored:\n continue\n if self.set_param(params, name):\n complex = True\n return params, complex", "response": "Get parameters for web service noting whether any are complex"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dictionary of all parameters for this loader.", "response": "def params(self):\n \"\"\"\n URL parameters for wq.io.loaders.NetLoader\n \"\"\"\n params, complex = self.get_params()\n url_params = self.default_params.copy()\n url_params.update(self.serialize_params(params, complex))\n return url_params"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef serialize_params(self, params, complex=False):\n if complex:\n # See climata.acis for an example implementation\n raise NotImplementedError(\"Cannot serialize %s!\" % params)\n else:\n # Simpler queries can use traditional URL parameters\n return {\n self.get_url_param(key): ','.join(val)\n for key, val in params.items()\n }", "response": "Serialize parameter names and values to a dict ready for urlencode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_inches(self):\n ''' convert the measurement to inches '''\n if self._obs_value in self.MISSING:\n return 'MISSING'\n if self._obs_units == self.MILLIMETERS:\n return round(self.INCH_CONVERSION_FACTOR * self._obs_value, 4)", "response": "convert the measurement to inches"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef formatted(self):\n ''' print a nicely formatted output of this report '''\n\n return \"\"\"\nWeather Station: %s (%s, %s)\nElevation: %s m\nTime: %s UTC\nAir Temperature: %s C (%s F)\nWind Speed: %s m/s (%s mph)\nWind Direction: %s\nPresent Weather Obs: %s\nPrecipitation: %s\nCloud Coverage: %s oktas\nCloud Summation: %s\nSolar Irradiance: %s \n \"\"\" % (self.weather_station, self.latitude, self.longitude,\n self.elevation, self.datetime, self.air_temperature,\n self.air_temperature.get_fahrenheit(), self.wind_speed,\n self.wind_speed.get_miles(), self.wind_direction,\n str(self.present_weather), str(self.precipitation),\n str(self.sky_cover), str(self.sky_cover_summation),\n str(self.solar_irradiance))", "response": "print a nicely formatted output of this report"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef loads(self, noaa_string):\n ''' load in a report (or set) from a string '''\n self.raw = noaa_string\n self.weather_station = noaa_string[4:10]\n self.wban = noaa_string[10:15]\n expected_length = int(noaa_string[0:4]) + self.PREAMBLE_LENGTH\n actual_length = len(noaa_string)\n if actual_length != expected_length:\n msg = \"Non matching lengths. Expected %d, got %d\" % (expected_length,\n actual_length)\n raise ish_reportException(msg)\n\n try:\n self.datetime = datetime.strptime(noaa_string[15:27], '%Y%m%d%H%M')\n except ValueError:\n ''' some cases, we get 2400 hours, which is really the next day, so \n this is a workaround for those cases '''\n time = noaa_string[15:27]\n time = time.replace(\"2400\", \"2300\")\n self.datetime = datetime.strptime(time, '%Y%m%d%H%M')\n self.datetime += timedelta(hours=1)\n\n self.datetime = self.datetime.replace(tzinfo=pytz.UTC)\n\n self.report_type = ReportType(noaa_string[41:46].strip())\n\n self.latitude = float(noaa_string[28:34]) / self.GEO_SCALE\n self.longitude = float(noaa_string[34:41]) / self.GEO_SCALE\n self.elevation = int(noaa_string[46:51])\n\n ''' other mandatory fields '''\n self.wind_direction = Direction(noaa_string[60:63],\n Direction.RADIANS,\n noaa_string[63:64])\n self.wind_observation_direction_type = noaa_string[64:64]\n self.wind_speed = Speed(int(noaa_string[65:69]) / float(self.SPEED_SCALE),\n Speed.METERSPERSECOND,\n noaa_string[69:70])\n self.sky_ceiling = Distance(int(noaa_string[70:75]),\n Distance.METERS,\n noaa_string[75:76])\n self.sky_ceiling_determination = noaa_string[76:77]\n self.visibility_distance = Distance(int(noaa_string[78:84]),\n Distance.METERS,\n noaa_string[84:85]) \n self.visibility_variability = noaa_string[85:86]\n self.visibility_variability_quality = noaa_string[86:87]\n\n self.air_temperature = Temperature(int(noaa_string[87:92]) / self.TEMPERATURE_SCALE,\n Units.CELSIUS,\n noaa_string[92:93])\n self.dew_point = Temperature(int(noaa_string[93:98]) / self.TEMPERATURE_SCALE,\n Units.CELSIUS,\n noaa_string[98:99])\n\n self.humidity = Humidity(str(self.air_temperature), str(self.dew_point))\n self.sea_level_pressure = Pressure(int(noaa_string[99:104])/self.PRESSURE_SCALE,\n Pressure.HECTOPASCALS,\n noaa_string[104:104])\n\n ''' handle the additional fields '''\n additional = noaa_string[105:108]\n if additional == 'ADD':\n position = 108\n while position < expected_length:\n try:\n (position, (addl_code, addl_string)) = self._get_component(noaa_string,\n position)\n self._additional[addl_code] = addl_string\n except ish_reportException as err:\n ''' this catches when we move to remarks section '''\n break\n\n ''' handle the remarks section if it exists '''\n try:\n position = noaa_string.index('REM', 108) \n self._get_remarks_component(noaa_string, position)\n except (ish_reportException, ValueError) as err:\n ''' this catches when we move to EQD section '''\n\n return self", "response": "loads in a report from a string"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the remarks from the string into the _remarks dict.", "response": "def _get_remarks_component(self, string, initial_pos):\n ''' Parse the remarks into the _remarks dict '''\n remarks_code = string[initial_pos:initial_pos + self.ADDR_CODE_LENGTH]\n if remarks_code != 'REM':\n raise ish_reportException(\"Parsing remarks. Expected REM but got %s.\" % (remarks_code,))\n\n expected_length = int(string[0:4]) + self.PREAMBLE_LENGTH\n position = initial_pos + self.ADDR_CODE_LENGTH\n while position < expected_length:\n key = string[position:position + self.ADDR_CODE_LENGTH]\n if key == 'EQD':\n break\n chars_to_read = string[position + self.ADDR_CODE_LENGTH:position + \\\n (self.ADDR_CODE_LENGTH * 2)]\n chars_to_read = int(chars_to_read)\n position += (self.ADDR_CODE_LENGTH * 2)\n string_value = string[position:position + chars_to_read]\n self._remarks[key] = string_value\n position += chars_to_read"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_component(self, string, initial_pos):\n ''' given a string and a position, return both an updated position and\n either a Component Object or a String back to the caller '''\n add_code = string[initial_pos:initial_pos + self.ADDR_CODE_LENGTH]\n \n if add_code == 'REM':\n raise ish_reportException(\"This is a remarks record\")\n if add_code == 'EQD':\n raise ish_reportException(\"This is EQD record\")\n\n initial_pos += self.ADDR_CODE_LENGTH \n try:\n useable_map = self.MAP[add_code]\n except:\n raise BaseException(\"Cannot find code %s in string %s (%d).\" % (add_code, string, initial_pos))\n\n # if there is no defined length, then read next three chars to get it\n # this only applies to REM types, which have 3 chars for the type, then variable\n if useable_map[1] is False:\n chars_to_read = string[initial_pos + self.ADDR_CODE_LENGTH:initial_pos + \\\n (self.ADDR_CODE_LENGTH * 2)]\n chars_to_read = int(chars_to_read)\n initial_pos += (self.ADDR_CODE_LENGTH * 2)\n else:\n chars_to_read = useable_map[1]\n\n new_position = initial_pos + chars_to_read\n string_value = string[initial_pos:new_position]\n\n try:\n object_value = useable_map[2]()\n object_value.loads(string_value)\n except IndexError as err:\n object_value = string_value\n\n return (new_position, [add_code, object_value])", "response": "given a string and a position return both an updated position and a Component Object or a String back to the caller"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading a string into the internal list of reports", "response": "def loads(self, string):\n ''' load from a string '''\n for line in string.split(\"\\n\"):\n if len(line) < 10:\n continue\n\n try:\n report = ish_report()\n report.loads(line)\n self._reports.append(report)\n except BaseException as exp:\n ''' don't complain TOO much '''\n logging.warning('unable to load report, error: %s' % exp)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns only specific weather observations", "response": "def get_observations(self):\n ''' return only specific weather observations (FM types) and\n ignore the summary of day reports '''\n return [rpt for rpt in self._reports if rpt.report_type in self.OBS_TYPES]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting the measurement to inches", "response": "def get_miles(self):\n ''' convert the measurement to inches '''\n if self._obs_value in self.MISSING:\n return 'MISSING'\n if self._obs_units == self.METERSPERSECOND:\n return round(2.23694 * self._obs_value, 4)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating the python source files and copy them to the mo path.", "response": "def do_pot(self):\n \"\"\"\n Sync the template with the python code.\n \"\"\"\n files_to_translate = []\n log.debug(\"Collecting python sources for pot ...\")\n for source_path in self._source_paths:\n for source_path in self._iter_suffix(path=source_path, suffix=\".py\"):\n log.debug(\"... add to pot: {source}\".format(source=str(source_path)))\n files_to_translate.append(str(source_path))\n for system_file in self.SYSTEM_SOURCE_FILES:\n files_to_translate.append(str(self._system_path / system_file))\n # FIXME: use separate domain for system source translations? Nerge them when generating mo's?\n log.debug(\"Finished collection sources.\")\n pot_path = (self._po_path / self._basename).with_suffix(\".pot\")\n command = [\"xgettext\", \"--keyword=_\", \"--keyword=_translate\",\n \"--output={output}\".format(output=str(pot_path))]\n command.extend(files_to_translate)\n check_call(command)\n log.debug(\"pot file \\\"{pot}\\\" created!\".format(pot=str(pot_path)))\n\n pot_copy_path = self._mo_path / pot_path.name\n log.debug(\"Copying pot file to mo path: {pot_copy_path}\".format(pot_copy_path=str(pot_copy_path)))\n shutil.copy(str(pot_path), str(pot_copy_path))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating all the po files with the data in the pot reference file.", "response": "def do_po(self):\n \"\"\"\n Update all po files with the data in the pot reference file.\n \"\"\"\n log.debug(\"Start updating po files ...\")\n pot_path = (self._po_path / self._basename).with_suffix(\".pot\")\n for po_dir_path in self._iter_po_dir():\n po_path = (po_dir_path / self._basename).with_suffix(\".po\")\n if po_path.exists():\n log.debug(\"update {po}:\".format(po=str(po_path)))\n check_call([\"msgmerge\", \"-U\", str(po_path), str(pot_path)])\n else:\n log.debug(\"create {po}:\".format(po=str(po_path)))\n check_call([\"msginit\", \"-i\", str(pot_path), \"-o\", str(po_path), \"--no-translator\"])\n po_copy_path = self._mo_path / po_path.parent.name / po_path.name\n po_copy_path.parent.mkdir(exist_ok=True)\n log.debug(\"Copying po file to mo path: {po_copy_path}\".format(po_copy_path=str(po_copy_path)))\n\n shutil.copy(str(po_path), str(po_copy_path))\n log.debug(\"All po files updated\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef do_mo(self):\n log.debug(\"Start updating mo files ...\")\n for po_dir_path in self._iter_po_dir():\n po_path = (po_dir_path / self._basename).with_suffix(\".po\")\n lc_path = self._mo_path / po_dir_path.name / \"LC_MESSAGES\"\n lc_path.mkdir(parents=True, exist_ok=True)\n mo_path = (lc_path / self._basename).with_suffix(\".mo\")\n log.debug(\"Creating from {po}: {mo}\".format(po=str(po_path), mo=str(mo_path)))\n check_call([\"msgfmt\", str(po_path), \"-o\", str(mo_path)])\n log.debug(\"All mo files updated\")", "response": "Generate the MO files for all the po files."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning the worker and consumes messages from RabbitMQ.", "response": "def run(self) -> None:\n \"\"\"Runs the worker and consumes messages from RabbitMQ.\n Returns only after `shutdown()` is called.\n\n \"\"\"\n if self._logging_level:\n logging.basicConfig(\n level=getattr(logging, self._logging_level.upper()),\n format=\"%(levelname).1s %(name)s.%(funcName)s:%(lineno)d - %(message)s\")\n\n signal.signal(signal.SIGINT, self._handle_sigint)\n signal.signal(signal.SIGTERM, self._handle_sigterm)\n if platform.system() != 'Windows':\n # These features will not be available on Windows, but that is OK.\n # Read this issue for more details:\n # https://github.com/cenkalti/kuyruk/issues/54\n signal.signal(signal.SIGHUP, self._handle_sighup)\n signal.signal(signal.SIGUSR1, self._handle_sigusr1)\n signal.signal(signal.SIGUSR2, self._handle_sigusr2)\n\n self._started_at = os.times().elapsed\n\n for t in self._threads:\n t.start()\n\n try:\n signals.worker_start.send(self.kuyruk, worker=self)\n self._consume_messages()\n signals.worker_shutdown.send(self.kuyruk, worker=self)\n finally:\n self.shutdown_pending.set()\n for t in self._threads:\n t.join()\n\n logger.debug(\"End run worker\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess the message received from the queue.", "response": "def _process_message(self, message: amqp.Message) -> None:\n \"\"\"Processes the message received from the queue.\"\"\"\n if self.shutdown_pending.is_set():\n return\n\n try:\n if isinstance(message.body, bytes):\n message.body = message.body.decode()\n description = json.loads(message.body)\n except Exception:\n logger.error(\"Cannot decode message. Dropping. Message: %r\", message.body)\n traceback.print_exc()\n message.channel.basic_reject(message.delivery_tag, requeue=False)\n else:\n logger.info(\"Processing task: %r\", description)\n self._process_description(message, description)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _apply_task(task: Task, args: Tuple, kwargs: Dict[str, Any]) -> Any:\n if args is None:\n args = ()\n if kwargs is None:\n kwargs = {}\n\n start = monotonic()\n try:\n return task.apply(*args, **kwargs)\n finally:\n delta = monotonic() - start\n logger.info(\"%s finished in %i seconds.\" % (task.name, delta))", "response": "Logs the time spent while running the task."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncount down from MAX_WORKER_RUN_TIME. When it reaches zero sutdown gracefully.", "response": "def _shutdown_timer(self) -> None:\n \"\"\"Counts down from MAX_WORKER_RUN_TIME. When it reaches zero sutdown\n gracefully.\n\n \"\"\"\n remaining = self._max_run_time - self.uptime\n if not self.shutdown_pending.wait(remaining):\n logger.warning('Run time reached zero')\n self.shutdown()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling by the process when the process has lost the connection to RabbitMQ.", "response": "def _handle_sighup(self, signum: int, frame: Any) -> None:\n \"\"\"Used internally to fail the task when connection to RabbitMQ is\n lost during the execution of the task.\n\n \"\"\"\n logger.warning(\"Catched SIGHUP\")\n exc_info = self._heartbeat_exc_info\n self._heartbeat_exc_info = None\n # Format exception info to see in tools like Sentry.\n formatted_exception = ''.join(traceback.format_exception(*exc_info)) # noqa\n raise HeartbeatError(exc_info)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the default folder where user - specific data is stored.", "response": "def configuration_get_default_folder():\n \"\"\"\n Return the default folder where user-specific data is stored.\n This depends of the system on which Python is running,\n :return: path to the user-specific configuration data folder\n \"\"\"\n system = platform.system()\n if system == 'Linux':\n # https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html\n sys_config_path = Path(os.getenv('XDG_CONFIG_HOME', os.path.expanduser(\"~/.config\")))\n elif system == 'Windows':\n sys_config_path = Path(os.getenv('APPDATA', ''))\n else:\n log.error('Unknown system: \"{system}\" (using default configuration path)'.format(system=system))\n sys_config_path = Path()\n log.debug('User-specific system configuration folder=\"{sys_config_path}\"'.format(\n sys_config_path=sys_config_path))\n sys_config = sys_config_path / PROJECT_TITLE\n log.debug('User-specific {project} configuration folder=\"{sys_config}\"'.format(\n project=PROJECT_TITLE, sys_config=sys_config))\n return sys_config"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a new Language instance from a locale string", "response": "def from_locale(cls, locale):\n \"\"\"\n Create a new Language instance from a locale string\n :param locale: locale as string\n :return: Language instance with instance.locale() == locale if locale is valid else instance of Unknown Language\n \"\"\"\n locale = str(locale)\n if locale is 'unknown':\n return UnknownLanguage(locale)\n try:\n return cls._from_xyz('locale', locale)\n except NotALanguageException:\n log.warning('Unknown locale: {}'.format(locale))\n return UnknownLanguage(locale)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new Language instance from an ISO639 string.", "response": "def from_xx(cls, xx):\n \"\"\"\n Create a new Language instance from a ISO639 string\n :param xx: ISO639 as string\n :return: Language instance with instance.xx() == xx if xx is valid else instance of UnknownLanguage\n \"\"\"\n xx = str(xx).lower()\n if xx is 'unknown':\n return UnknownLanguage(xx)\n try:\n return cls._from_xyz('ISO639', xx)\n except NotALanguageException:\n log.warning('Unknown ISO639: {}'.format(xx))\n return UnknownLanguage(xx)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_xxx(cls, xxx):\n xxx = str(xxx).lower()\n if xxx is 'unknown':\n return UnknownLanguage(xxx)\n try:\n return cls._from_xyz('LanguageID', xxx)\n except NotALanguageException:\n log.warning('Unknown LanguageId: {}'.format(xxx))\n return UnknownLanguage(xxx)", "response": "Create a new Language instance from a LanguageID string."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new Language instance from a name as string", "response": "def from_name(cls, name):\n \"\"\"\n Create a new Language instance from a name as string\n :param name: name as string\n :return: Language instance with instance.name() == name if name is valid else instance of UnknownLanguage\n \"\"\"\n name = str(name).lower()\n if name is 'unknown' or name is _('unknown'):\n return UnknownLanguage(name)\n try:\n return cls._from_xyz('LanguageName', name)\n except NotALanguageException:\n log.warning('Unknown LanguageName: {}'.format(name))\n return UnknownLanguage(name)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _from_xyz(cls, xyzkey, xyzvalue):\n if xyzvalue == 'unknown' or xyzvalue == _('unknown'):\n return UnknownLanguage(xyzvalue)\n for lang_id, lang_data in enumerate(LANGUAGES):\n for data_value in lang_data[xyzkey]:\n if xyzvalue == data_value.lower():\n return cls(lang_id)\n raise NotALanguageException(xyzvalue, 'Illegal language {}: {}'.format(xyzkey, xyzvalue))", "response": "Private helper function to create a new Language instance from a given language key and value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntrying to create a Language instance from an unknown language.", "response": "def from_unknown(cls, value, xx=False, xxx=False, locale=False, name=False):\n \"\"\"\n Try to create a Language instance having only some limited data about the Language.\n If no corresponding Language is found, a NotALanguageException is thrown.\n :param value: data known about the language as string\n :param xx: True if the value may be a locale\n :param xxx: True if the value may be a LanguageID\n :param locale: True if the value may be a locale\n :param name: True if the value may be a LanguageName\n :return: Language Instance if a matching Language was found\n \"\"\"\n # Use 2 lists instead of dict ==> order known\n keys = ['ISO639', 'LanguageID', 'locale', 'LanguageName']\n truefalses = [xx, xxx, locale, name]\n value = value.lower()\n for key, doKey in zip(keys, truefalses):\n if doKey:\n try:\n return cls._from_xyz(key, value)\n except NotALanguageException:\n pass\n raise NotALanguageException(value, 'Illegal language \"{}\"'.format(value))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_file(cls, filepath, chunk_size=None):\n log.debug('Language.from_file: \"{}\", chunk={} ...'.format(filepath, chunk_size))\n with filepath.open('rb') as f:\n data = f.read(-1 if chunk_size is None else chunk_size)\n data_ascii = asciify(data)\n lang_xx = langdetect_detect(data_ascii)\n lang = cls.from_xx(lang_xx)\n log.debug('... result language={}'.format(lang))\n return lang", "response": "Try to determine the language of a text file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef onFolderTreeClicked(self, proxyIndex):\n if not proxyIndex.isValid():\n return\n\n index = self.proxyFileModel.mapToSource(proxyIndex)\n settings = QSettings()\n folder_path = self.fileModel.filePath(index)\n settings.setValue('mainwindow/workingDirectory', folder_path)", "response": "What to do when a Folder in the tree is clicked"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a message to the queue.", "response": "def send_to_queue(\n self,\n args: Tuple=(),\n kwargs: Dict[str, Any]={},\n host: str=None,\n wait_result: Union[int, float]=None,\n message_ttl: Union[int, float]=None,\n ) -> Any:\n \"\"\"\n Sends a message to the queue.\n A worker will run the task's function when it receives the message.\n\n :param args: Arguments that will be passed to task on execution.\n :param kwargs: Keyword arguments that will be passed to task\n on execution.\n :param host: Send this task to specific host. ``host`` will be\n appended to the queue name. If ``host`` is \"localhost\", hostname\n of the server will be appended to the queue name.\n :param wait_result:\n Wait for result from worker for ``wait_result`` seconds.\n If timeout occurs,\n :class:`~kuyruk.exceptions.ResultTimeout` is raised.\n If excecption occurs in worker,\n :class:`~kuyruk.exceptions.RemoteException` is raised.\n :param message_ttl:\n If set, message will be destroyed in queue after ``message_ttl``\n seconds.\n :return: Result from worker if ``wait_result`` is set,\n else :const:`None`.\n\n \"\"\"\n if self.kuyruk.config.EAGER:\n # Run the task in current process\n result = self.apply(*args, **kwargs)\n return result if wait_result else None\n\n logger.debug(\"Task.send_to_queue args=%r, kwargs=%r\", args, kwargs)\n queue = self._queue_for_host(host)\n description = self._get_description(args, kwargs)\n self._send_signal(signals.task_presend, args=args, kwargs=kwargs, description=description)\n\n body = json.dumps(description)\n msg = amqp.Message(body=body)\n if wait_result:\n # Use direct reply-to feature from RabbitMQ:\n # https://www.rabbitmq.com/direct-reply-to.html\n msg.properties['reply_to'] = 'amq.rabbitmq.reply-to'\n\n if message_ttl:\n msg.properties['expiration'] = str(int(message_ttl * 1000))\n\n with self.kuyruk.channel() as ch:\n if wait_result:\n result = Result(ch.connection)\n ch.basic_consume(queue='amq.rabbitmq.reply-to', no_ack=True, callback=result.process_message)\n\n ch.queue_declare(queue=queue, durable=True, auto_delete=False)\n ch.basic_publish(msg, exchange=\"\", routing_key=queue)\n self._send_signal(signals.task_postsend, args=args, kwargs=kwargs, description=description)\n\n if wait_result:\n return result.wait(wait_result)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_description(self, args: Tuple, kwargs: Dict[str, Any]) -> Dict[str, Any]:\n return {\n 'id': uuid1().hex,\n 'args': args,\n 'kwargs': kwargs,\n 'module': self._module_name,\n 'function': self.f.__name__,\n 'sender_hostname': socket.gethostname(),\n 'sender_pid': os.getpid(),\n 'sender_cmd': ' '.join(sys.argv),\n 'sender_timestamp': datetime.utcnow().isoformat()[:19],\n }", "response": "Return the dictionary to be sent to the queue."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef apply(self, *args: Any, **kwargs: Any) -> Any:\n def send_signal(sig: Signal, **extra: Any) -> None:\n self._send_signal(sig, args=args, kwargs=kwargs, **extra)\n\n logger.debug(\"Applying %r, args=%r, kwargs=%r\", self, args, kwargs)\n\n send_signal(signals.task_preapply)\n try:\n tries = 1 + self.retry\n while 1:\n tries -= 1\n send_signal(signals.task_prerun)\n try:\n with time_limit(self.max_run_time or 0):\n return self.f(*args, **kwargs)\n except Exception:\n send_signal(signals.task_error, exc_info=sys.exc_info())\n if tries <= 0:\n raise\n else:\n break\n finally:\n send_signal(signals.task_postrun)\n except Exception:\n send_signal(signals.task_failure, exc_info=sys.exc_info())\n raise\n else:\n send_signal(signals.task_success)\n finally:\n send_signal(signals.task_postapply)", "response": "Calls the wrapped function and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _module_name(self) -> str:\n name = self.f.__module__\n if name == '__main__':\n return importer.main_module_name()\n return name", "response": "Return the name of the wrapped function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef i18n_install(lc=None):\n log.debug('i18n_install( {lc} ) called.'.format(lc=lc))\n if lc is None:\n lc = i18n_system_locale()\n if lc is None:\n log.debug('i18n_install(): installing NullTranslations')\n translator = gettext.NullTranslations()\n else:\n child_locales = i18n_support_locale(lc) # Call i18n_support_locale to log the supported locales\n\n log.debug('i18n_install(): installing gettext.translation(domain={domain}, localedir={localedir}, '\n 'languages={languages}, fallback={fallback})'.format(domain=project.PROJECT_TITLE.lower(),\n localedir=i18n_get_path(),\n languages=child_locales,\n fallback=True))\n translator = gettext.translation(\n domain=project.PROJECT_TITLE.lower(), localedir=str(i18n_get_path()),\n languages=child_locales, fallback=True)\n translator.install(names=['ngettext'])", "response": "Install internationalization support for the clients using the specified locale."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef i18n_system_locale():\n log.debug('i18n_system_locale() called')\n lc, encoding = locale.getlocale()\n log.debug('locale.getlocale() = (lc=\"{lc}\", encoding=\"{encoding}).'.format(lc=lc, encoding=encoding))\n if lc is None:\n lc, encoding = locale.getdefaultlocale()\n log.debug('locale.getdefaultlocale() = (lc=\"{lc}\", encoding=\"{encoding}).'.format(lc=lc, encoding=encoding))\n return lc", "response": "Return the system locale as a string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef i18n_locale_fallbacks_calculate(lc):\n log.debug('i18n_locale_fallbacks_calculate( locale=\"{locale}\" ) called'.format(locale=lc))\n locales = []\n lc_original = lc\n while lc:\n locales.append(lc)\n rindex = max([lc.rfind(separator) for separator in ['@', '_', '-', '.']])\n if rindex == -1:\n break\n lc = lc[:rindex]\n log.debug('i18n_locale_fallbacks_calculate( lc=\"{lc}\" ) = {locales}'.format(lc=lc_original, locales=locales))\n return locales", "response": "Calculate all child locales from a locale."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding out whether a locale is supported by the given locale.", "response": "def i18n_support_locale(lc_parent):\n \"\"\"\n Find out whether lc is supported. Returns all child locales (and eventually lc) which do have support.\n :param lc_parent: Locale for which we want to know the child locales that are supported\n :return: list of supported locales\n \"\"\"\n log.debug('i18n_support_locale( locale=\"{locale}\" ) called'.format(locale=lc_parent))\n lc_childs = i18n_locale_fallbacks_calculate(lc_parent)\n locales = []\n\n locale_path = i18n_get_path()\n mo_file = '{project}.mo'.format(project=project.PROJECT_TITLE.lower())\n\n for lc in lc_childs:\n lc_mo_path = locale_path / lc / 'LC_MESSAGES' / mo_file\n log.debug('Locale data \"{lc_mo_path}\" exists? ...'.format(lc_mo_path=lc_mo_path))\n if lc_mo_path.is_file():\n log.debug('... Yes! \"{locale_path}\" contains {mo_file}.'.format(locale_path=locale_path, mo_file=mo_file))\n locales.append(lc)\n else:\n log.debug('... No')\n\n log.debug('i18n_support_locale( lc=\"{lc}\" ) = {locales}'.format(lc=lc_parent, locales=locales))\n return locales"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef i18n_get_path():\n local_locale_path = client_get_path() / 'locale'\n if platform.system() == 'Linux':\n if local_locale_path.exists():\n return local_locale_path\n else:\n return Path('/usr/share/locale')\n else:\n return local_locale_path", "response": "Get path to the internationalization data."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of all locales that have internationalization data for this program", "response": "def i18n_get_supported_locales():\n \"\"\"\n List all locales that have internationalization data for this program\n :return: List of locales\n \"\"\"\n locale_path = i18n_get_path()\n log.debug('Scanning translation files .mo in locale path: {}'.format(locale_path))\n langs = []\n mo_file = '{project}.mo'.format(project=project.PROJECT_TITLE.lower())\n for lc in locale_path.iterdir():\n lc_mo_path = lc / 'LC_MESSAGES' / mo_file\n if lc_mo_path.exists():\n langs.append(lc.name)\n log.debug('Detected: {langs}'.format(langs=langs))\n return langs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_object(self, obj: Union[str, Any]) -> None:\n if isinstance(obj, str):\n obj = importer.import_object_str(obj)\n\n for key in dir(obj):\n if key.isupper():\n value = getattr(obj, key)\n self._setattr(key, value)\n\n logger.info(\"Config is loaded from object: %r\", obj)", "response": "Load values from an object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload values from a dict.", "response": "def from_dict(self, d: Dict[str, Any]) -> None:\n \"\"\"Load values from a dict.\"\"\"\n for key, value in d.items():\n if key.isupper():\n self._setattr(key, value)\n\n logger.info(\"Config is loaded from dict: %r\", d)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload values from a Python file.", "response": "def from_pyfile(self, filename: str) -> None:\n \"\"\"Load values from a Python file.\"\"\"\n globals_ = {} # type: Dict[str, Any]\n locals_ = {} # type: Dict[str, Any]\n with open(filename, \"rb\") as f:\n exec(compile(f.read(), filename, 'exec'), globals_, locals_)\n\n for key, value in locals_.items():\n if (key.isupper() and not isinstance(value, types.ModuleType)):\n self._setattr(key, value)\n\n logger.info(\"Config is loaded from file: %s\", filename)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload values from environment variables.", "response": "def from_env_vars(self) -> None:\n \"\"\"Load values from environment variables.\n Keys must start with `KUYRUK_`.\"\"\"\n for key, value in os.environ.items():\n if key.startswith('KUYRUK_'):\n key = key[7:]\n if hasattr(Config, key):\n try:\n value = ast.literal_eval(value)\n except (ValueError, SyntaxError):\n pass\n\n self._setattr(key, value)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_method_call(node, method_name):\n\n if not isinstance(node, nodes.Call):\n return False\n\n if isinstance(node.node, nodes.Getattr):\n # e.g. foo.bar()\n method = node.node.attr\n\n elif isinstance(node.node, nodes.Name):\n # e.g. bar()\n method = node.node.name\n\n elif isinstance(node.node, nodes.Getitem):\n # e.g. foo[\"bar\"]()\n method = node.node.arg.value\n\n else:\n return False\n\n if isinstance(method_name, (list, tuple)):\n return method in method_name\n\n return method == method_name", "response": "Returns True if node is a method call for method_name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_loop_helper(node):\n return hasattr(node, 'node') and isinstance(node.node, nodes.Name) and node.node.name == 'loop'", "response": "Returns True is node is a loop helper e. g. loop. index or loop. first"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the generated JavaScript code.", "response": "def get_output(self):\n \"\"\"\n Returns the generated JavaScript code.\n\n Returns:\n str\n \"\"\"\n # generate the JS function string\n template_function = TEMPLATE_WRAPPER.format(\n function_name=self.js_function_name,\n template_code=self.output.getvalue()\n ).strip()\n\n # get the correct module format template\n module_format = JS_MODULE_FORMATS[self.js_module_format]\n\n # generate the module code\n return module_format(self.dependencies, template_function)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_depencency_var_name(self, dependency):\n for dep_path, var_name in self.dependencies:\n if dep_path == dependency:\n return var_name", "response": "Returns the variable name assigned to the given dependency."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds the given dependency and returns the variable name to use to access it.", "response": "def _add_dependency(self, dependency, var_name=None):\n \"\"\"\n Adds the given dependency and returns the variable name to use to access it. If `var_name`\n is not given then a random one will be created.\n\n Args:\n dependency (str):\n var_name (str, optional):\n\n Returns:\n str\n \"\"\"\n if var_name is None:\n var_name = next(self.temp_var_names)\n # Don't add duplicate dependencies\n if (dependency, var_name) not in self.dependencies:\n self.dependencies.append((dependency, var_name))\n return var_name"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess an extends block e. g. some. template. jinja", "response": "def _process_extends(self, node, **kwargs):\n \"\"\"\n Processes an extends block e.g. `{% extends \"some/template.jinja\" %}`\n \"\"\"\n\n # find all the blocks in this template\n for b in self.ast.find_all(nodes.Block):\n\n # if not already in `child_blocks` then this is the first time a\n # block with this name has been encountered.\n if b.name not in self.child_blocks:\n self.child_blocks[b.name] = b\n else:\n\n # otherwise we have seen this block before, so we need to find the last\n # super_block and add the block from this template to the end.\n block = self.child_blocks.get(b.name)\n while hasattr(block, 'super_block'):\n block = block.super_block\n block.super_block = b\n\n # load the parent template\n parent_template = JinjaToJS(template_root=self.template_root,\n template_name=node.template.value,\n js_module_format=self.js_module_format,\n runtime_path=self.runtime_path,\n include_prefix=self.include_prefix,\n include_ext=self.include_ext,\n child_blocks=self.child_blocks,\n dependencies=self.dependencies)\n\n # add the parent templates output to the current output\n self.output.write(parent_template.output.getvalue())\n\n # Raise an exception so we stop parsing this template\n raise ExtendsException"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _process_block(self, node, **kwargs):\n\n # check if this node already has a 'super_block' attribute\n if not hasattr(node, 'super_block'):\n\n # since it doesn't it must be the last block in the inheritance chain\n node.super_block = None\n\n # see if there has been a child block defined - if there is this\n # will be the first block in the inheritance chain\n child_block = self.child_blocks.get(node.name)\n\n if child_block:\n\n # we have child nodes so we need to set `node` as the\n # super of the last one in the chain\n last_block = child_block\n while hasattr(last_block, 'super_block'):\n last_block = child_block.super_block\n\n # once we have found it, set this node as it's super block\n last_block.super_block = node\n\n # this is the node we want to process as it's the first in the inheritance chain\n node = child_block\n\n # process the block passing the it's super along, if this block\n # calls super() it will be handled by `_process_call`\n for n in node.body:\n self._process_node(n, super_block=node.super_block, **kwargs)", "response": "Processes a block e. g. my_block endblock"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _process_output(self, node, **kwargs):\n for n in node.nodes:\n self._process_node(n, **kwargs)", "response": "Processes an output node which will contain things like Name and TemplateData nodes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprocess a TemplateData node.", "response": "def _process_templatedata(self, node, **_):\n \"\"\"\n Processes a `TemplateData` node, this is just a bit of as-is text\n to be written to the output.\n \"\"\"\n\n # escape double quotes\n value = re.sub('\"', r'\\\\\"', node.data)\n\n # escape new lines\n value = re.sub('\\n', r'\\\\n', value)\n\n # append value to the result\n self.output.write('__result += \"' + value + '\";')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _process_name(self, node, **kwargs):\n\n with self._interpolation():\n with self._python_bool_wrapper(**kwargs):\n\n if node.name not in self.stored_names and node.ctx != 'store':\n self.output.write(self.context_name)\n self.output.write('.')\n\n if node.ctx == 'store':\n self.stored_names.add(node.name)\n\n self.output.write(node.name)", "response": "Processes a Name node."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _process_getattr(self, node, **kwargs):\n\n with self._interpolation():\n with self._python_bool_wrapper(**kwargs) as new_kwargs:\n if is_loop_helper(node):\n self._process_loop_helper(node, **new_kwargs)\n else:\n self._process_node(node.node, **new_kwargs)\n self.output.write('.')\n self.output.write(node.attr)", "response": "Processes a GetAttr node. e. g. foo. bar."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprocess a GetItem node e. g. foo. bar.", "response": "def _process_getitem(self, node, **kwargs):\n \"\"\"\n Processes a `GetItem` node e.g. {{ foo[\"bar\"] }}\n \"\"\"\n\n with self._interpolation():\n with self._python_bool_wrapper(**kwargs) as new_kwargs:\n self._process_node(node.node, **new_kwargs)\n\n if isinstance(node.arg, nodes.Slice):\n self.output.write('.slice(')\n\n if node.arg.step is not None:\n raise Exception('The step argument is not supported when slicing.')\n\n if node.arg.start is None:\n self.output.write('0')\n else:\n self._process_node(node.arg.start, **new_kwargs)\n\n if node.arg.stop is None:\n self.output.write(')')\n else:\n self.output.write(',')\n self._process_node(node.arg.stop, **new_kwargs)\n self.output.write(')')\n else:\n self.output.write('[')\n self._process_node(node.arg, **new_kwargs)\n self.output.write(']')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _process_for(self, node, **kwargs):\n\n # since a for loop can introduce new names into the context\n # we need to remember the ones that existed outside the loop\n previous_stored_names = self.stored_names.copy()\n\n with self._execution():\n self.output.write('__runtime.each(')\n\n if is_method_call(node.iter, dict.keys.__name__):\n self.output.write('Object.keys(')\n\n self._process_node(node.iter, **kwargs)\n\n if is_method_call(node.iter, dict.keys.__name__):\n self.output.write(')')\n\n self.output.write(',')\n self.output.write('function')\n self.output.write('(')\n\n # javascript iterations put the value first, then the key\n if isinstance(node.target, nodes.Tuple):\n if len(node.target.items) > 2:\n raise Exception('De-structuring more than 2 items is not supported.')\n\n for i, item in enumerate(reversed(node.target.items)):\n self._process_node(item, **kwargs)\n if i < len(node.target.items) - 1:\n self.output.write(',')\n else:\n self._process_node(node.target, **kwargs)\n\n self.output.write(')')\n self.output.write('{')\n\n if node.test:\n self.output.write('if (!(')\n self._process_node(node.test, **kwargs)\n self.output.write(')) { return; }')\n\n assigns = node.target.items if isinstance(node.target, nodes.Tuple) else [node.target]\n\n with self._scoped_variables(assigns, **kwargs):\n for n in node.body:\n self._process_node(n, **kwargs)\n\n with self._execution():\n self.output.write('}')\n self.output.write(')')\n self.output.write(';')\n\n # restore the stored names\n self.stored_names = previous_stored_names", "response": "Processes a for loop. e. g. for loops."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _process_math(self, node, math_operator=None, function=None, **kwargs):\n\n with self._interpolation():\n if function:\n self.output.write(function)\n self.output.write('(')\n\n self._process_node(node.left, **kwargs)\n self.output.write(math_operator)\n self._process_node(node.right, **kwargs)\n\n if function:\n self.output.write(')')", "response": "Processes a math node e. g. Div Sub Add Mul etc..."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _process_loop_helper(self, node, **kwargs):\n\n if node.attr == LOOP_HELPER_INDEX:\n self.output.write('(arguments[1] + 1)')\n elif node.attr == LOOP_HELPER_INDEX_0:\n self.output.write('arguments[1]')\n elif node.attr == LOOP_HELPER_FIRST:\n self.output.write('(arguments[1] == 0)')\n elif node.attr == LOOP_HELPER_LAST:\n self.output.write('(arguments[1] == arguments[2].length - 1)')\n elif node.attr == LOOP_HELPER_LENGTH:\n self.output.write('arguments[2].length')", "response": "Processes a loop helper e. g. loop. first or loop. last."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _execution(self):\n\n did_start_executing = False\n\n if self.state == STATE_DEFAULT:\n did_start_executing = True\n self.state = STATE_EXECUTING\n\n def close():\n if did_start_executing and self.state == STATE_EXECUTING:\n self.state = STATE_DEFAULT\n\n yield close\n close()", "response": "Context manager for executing some JavaScript inside a template."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _scoped_variables(self, nodes_list, **kwargs):\n\n tmp_vars = []\n for node in nodes_list:\n\n is_assign_node = isinstance(node, nodes.Assign)\n name = node.target.name if is_assign_node else node.name\n\n # create a temp variable name\n tmp_var = next(self.temp_var_names)\n\n # save previous context value\n with self._execution():\n\n # save the current value of this name\n self.output.write('var %s = %s.%s;' % (tmp_var, self.context_name, name))\n\n # add new value to context\n self.output.write('%s.%s = ' % (self.context_name, name))\n\n if is_assign_node:\n self._process_node(node.node, **kwargs)\n else:\n self.output.write(node.name)\n\n self.output.write(';')\n\n tmp_vars.append((tmp_var, name))\n\n yield\n\n # restore context\n for tmp_var, name in tmp_vars:\n with self._execution():\n self.output.write('%s.%s = %s;' % (self.context_name, name, tmp_var))", "response": "Context manager for creating scoped variables defined by the nodes in nodes_list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndownloading an url to a local file.", "response": "def download_raw(url, local_path, callback):\n \"\"\"\n Download an url to a local file.\n :param url: url of the file to download\n :param local_path: path where the downloaded file should be saved\n :param callback: instance of ProgressCallback\n :return: True is succeeded\n \"\"\"\n log.debug('download_raw(url={url}, local_path={local_path})'.format(url=url, local_path=local_path))\n raw_progress = RawDownloadProgress(callback)\n reporthook = raw_progress.get_report_hook()\n try:\n log.debug('urlretrieve(url={url}, local_path={local_path}) ...'.format(url=url, local_path=local_path))\n urlretrieve(url=url, filename=local_path, reporthook=reporthook)\n log.debug('... SUCCEEDED')\n callback.finish(True)\n return True\n except URLError:\n log.exception('... FAILED')\n callback.finish(False)\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a callback function suitable for using reporthook argument of urllib. request. urlretrieve", "response": "def get_report_hook(self):\n \"\"\"\n Return a callback function suitable for using reporthook argument of urllib(.request).urlretrieve\n :return: function object\n \"\"\"\n def report_hook(chunkNumber, chunkSize, totalSize):\n if totalSize != -1 and not self._callback.range_initialized():\n log.debug('Initializing range: [{},{}]'.format(0, totalSize))\n self._callback.set_range(0, totalSize)\n self._chunkNumber = chunkNumber\n self._total += chunkSize\n if self._total > totalSize:\n # The chunk size can be bigger than the file\n self._total = totalSize\n self._callback.update(self._total)\n\n return report_hook"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the minimum and maximum value of the cache entry.", "response": "def set_range(self, minimum, maximum):\n \"\"\"\n Set a range.\n The range is passed unchanged to the rangeChanged member function.\n :param minimum: minimum value of the range (None if no percentage is required)\n :param maximum: maximum value of the range (None if no percentage is required)\n \"\"\"\n self._min = minimum\n self._max = maximum\n self.on_rangeChange(minimum, maximum)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new child ProgressCallback instance.", "response": "def get_child_progress(self, parent_min, parent_max):\n \"\"\"\n Create a new child ProgressCallback.\n Minimum and maximum values of the child are mapped to parent_min and parent_max of this parent ProgressCallback.\n :param parent_min: minimum value of the child is mapped to parent_min of this parent ProgressCallback\n :param parent_max: maximum value of the child is mapped to parent_max of this parent ProgressCallback\n :return: instance of SubProgressCallback\n \"\"\"\n return SubProgressCallback(parent=self, parent_min=parent_min, parent_max=parent_max)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall this function to inform that an update is available. This function does NOT call finish when value == maximum. :param value: The current index/position of the action. (Should be, but must not be, in the range [min, max]) :param args: extra positional arguments to pass on :param kwargs: extra keyword arguments to pass on", "response": "def update(self, value, *args, **kwargs):\n \"\"\"\n Call this function to inform that an update is available.\n This function does NOT call finish when value == maximum.\n :param value: The current index/position of the action. (Should be, but must not be, in the range [min, max])\n :param args: extra positional arguments to pass on\n :param kwargs: extra keyword arguments to pass on\n \"\"\"\n log.debug('update(value={value}, args={args}, kwargs={kwargs})'.format(value=value, args=args, kwargs=kwargs))\n self.on_update(value, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling this function to inform that the operation is finished.", "response": "def finish(self, *args, **kwargs):\n \"\"\"\n Call this function to inform that the operation is finished.\n :param args: extra positional arguments to pass on\n :param kwargs: extra keyword arguments to pass on\n \"\"\"\n log.debug('finish(args={args}, kwargs={kwargs})'.format(args=args, kwargs=kwargs))\n self.on_finish(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef on_update(self, value, *args, **kwargs):\n parent_value = self._parent_min\n if self._max != self._min:\n sub_progress = (value - self._min) / (self._max - self._min)\n parent_value = self._parent_min + sub_progress * (self._parent_max - self._parent_min)\n self._parent.update(parent_value, *args, **kwargs)", "response": "Informs the parent of progress."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlogging-in to the Server using username and password. Returns True if successful and False if not.", "response": "def _login(self, username=\"\", password=\"\"):\n \"\"\"Login to the Server using username/password,\n empty parameters means an anonymously login\n Returns True if login sucessful, and False if not.\n \"\"\"\n self.log.debug(\"----------------\")\n self.log.debug(\"Logging in (username: %s)...\" % username)\n\n def run_query():\n return self._xmlrpc_server.LogIn(\n username, password, self.language, self.user_agent)\n\n info = self._safe_exec(run_query, None)\n if info is None:\n self._token = None\n return False\n\n self.log.debug(\"Login ended in %s with status: %s\" %\n (info['seconds'], info['status']))\n\n if info['status'] == \"200 OK\":\n self.log.debug(\"Session ID: %s\" % info['token'])\n self.log.debug(\"----------------\")\n self._token = info['token']\n return True\n else:\n # force token reset\n self.log.debug(\"----------------\")\n self._token = None\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _logout(self):\n self.log.debug(\"Logging out from session ID: %s\" % self._token)\n try:\n info = self._xmlrpc_server.LogOut(self._token)\n self.log.debug(\"Logout ended in %s with status: %s\" %\n (info['seconds'], info['status']))\n except ProtocolError as e:\n self.log.debug(\"error in HTTP/HTTPS transport layer\")\n raise\n except Fault as e:\n self.log.debug(\"error in xml-rpc server\")\n raise\n except:\n self.log.exception(\"Connection to the server failed/other error\")\n raise\n finally:\n # force token reset\n self._token = None", "response": "Logout from current session"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the program arguments.", "response": "def parse_arguments(args=None):\n \"\"\"\n Parse the program arguments.\n :return: argparse.Namespace object with the parsed arguments\n \"\"\"\n parser = get_argument_parser()\n\n # Autocomplete arguments\n autocomplete(parser)\n\n ns = parser.parse_args(args=args)\n return ArgumentSettings(\n program=ArgumentProgramSettings(\n log=ArgumentLogSettings(\n path=None,\n level=ns.loglevel,\n ),\n settings=ArgumentSettingsSettings(\n path=ns.settings_path,\n ),\n client=ArgumentClientSettings(\n type=ns.client_type,\n cli=ArgumentClientCliSettings(\n interactive=False,\n ),\n gui=ArgumentClientGuiSettings(\n ),\n ),\n ),\n search=ArgumentSearchSettings(\n recursive=ns.recursive,\n working_directory=ns.video_path,\n ),\n filter=FilterSettings(\n languages=ns.languages,\n ),\n download=DownloadSettings(\n rename_strategy=ns.rename_strategy,\n ),\n providers=ns.providers,\n proxy=ns.proxy,\n test=ns.test,\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_argument_parser():\n parser = argparse.ArgumentParser(description=project.get_description(),\n epilog=_('Visit us at {website}.').format(website=project.WEBSITE_MAIN))\n\n parser.add_argument('--version', action='version',\n version='{project} {version}'.format(project=project.PROJECT_TITLE,\n version=project.PROJECT_VERSION_STR))\n parser.add_argument('-T', '--test', dest='test',\n action='store_true', default=False,\n help=argparse.SUPPRESS)\n parser.add_argument('-V', '--video', dest='video_path', default=None, metavar='PATH',\n nargs=argparse.ONE_OR_MORE, action=PathsAction,\n help=_('Full path to your video(s).'))\n parser.add_argument('-s', '--settings', dest='settings_path', type=Path, default=None, metavar='FILE',\n help=_('Set the settings file.'))\n parser.add_argument('-l', '--lang', dest='languages', metavar='LANGUAGE',\n default=[UnknownLanguage.create_generic()],\n nargs=argparse.ONE_OR_MORE, action=LanguagesAction,\n help=_('Set the preferred subtitle language(s) for download and upload.'))\n\n # interface options\n interface_group = parser.add_argument_group(_('interface'), _('Change settings of the interface'))\n guicli = interface_group.add_mutually_exclusive_group()\n guicli.add_argument('-g', '--gui', dest='client_type',\n action='store_const', const=ClientType.GUI,\n help=_('Run application in GUI mode. This is the default.'))\n guicli.add_argument('-c', '--cli', dest='client_type',\n action='store_const', const=ClientType.CLI,\n help=_('Run application in CLI mode.'))\n parser.set_defaults(client_type=ClientType.GUI)\n\n # logger options\n loggroup = parser.add_argument_group(_('logging'), _('Change the amount of logging done.'))\n loglvlex = loggroup.add_mutually_exclusive_group()\n loglvlex.add_argument('-d', '--debug', dest='loglevel',\n action='store_const', const=logging.DEBUG,\n help=_('Print log messages of debug severity and higher to stderr.'))\n loglvlex.add_argument('-w', '--warning', dest='loglevel',\n action='store_const', const=logging.WARNING,\n help=_('Print log messages of warning severity and higher to stderr. This is the default.'))\n loglvlex.add_argument('-e', '--error', dest='loglevel',\n action='store_const', const=logging.ERROR,\n help=_('Print log messages of error severity and higher to stderr.'))\n loglvlex.add_argument('-q', '--quiet', dest='loglevel',\n action='store_const', const=LOGGING_LOGNOTHING,\n help=_('Don\\'t log anything to stderr.'))\n loggroup.set_defaults(loglevel=logging.WARNING)\n\n loggroup.add_argument('--log', dest='logfile', metavar='FILE', type=Path,\n default=None, help=_('Path name of the log file.'))\n\n # cli options\n cli_group = parser.add_argument_group(_('cli'), _('Change the behavior of the command line interface.'))\n cli_group.add_argument('-i', '--interactive', dest='interactive',\n action='store_true', default=False,\n help=_('Prompt user when decisions need to be done.'))\n cli_group.add_argument('-r', '--recursive', dest='recursive',\n action='store_true', default=False,\n help=_('Search for subtitles recursively.'))\n\n operation_group = cli_group.add_mutually_exclusive_group()\n operation_group.add_argument('-D', '--download', dest='operation', action='store_const', const=CliAction.DOWNLOAD,\n help=_('Download subtitle(s). This is the default.'))\n operation_group.add_argument('-U', '--upload', dest='operation', action='store_const', const=CliAction.UPLOAD,\n help=_('Upload subtitle(s).'))\n # operation_group.add_argument('-L', '--list', dest='operation', action='store_const', const=CliAction.LIST,\n # help=_('List available subtitle(s) without downloading.'))\n parser.set_defaults(operation=CliAction.DOWNLOAD)\n\n rename_group = cli_group.add_mutually_exclusive_group()\n rename_group.add_argument('--rename-online', dest='rename_strategy', action='store_const',\n const=SubtitleRenameStrategy.ONLINE,\n help=_('Use the on-line subtitle filename as name for the downloaded subtitles. '\n 'This is the default.'))\n rename_group.add_argument('--rename-video', dest='rename_strategy', action='store_const',\n const=SubtitleRenameStrategy.VIDEO,\n help=_('Use the local video filename as name for the downloaded subtitle.'))\n rename_group.add_argument('--rename-lang', dest='rename_strategy', action='store_const',\n const=SubtitleRenameStrategy.VIDEO_LANG,\n help=_('Use the local video filename + language as name for the downloaded subtitle.'))\n rename_group.add_argument('--rename-uploader', dest='rename_strategy', action='store_const',\n const=SubtitleRenameStrategy.VIDEO_LANG_UPLOADER,\n help=_('Use the local video filename + uploader + language '\n 'as name for the downloaded subtitle.'))\n parser.set_defaults(rename_strategy=SubtitleRenameStrategy.ONLINE)\n\n # online options\n online_group = parser.add_argument_group('online', 'Change parameters related to the online provider.')\n online_group.add_argument('-P', '--proxy', dest='proxy', default=None, action=ProxyAction,\n help=_('Proxy to use on internet connections.'))\n online_group.add_argument('--provider', dest='providers', metavar='NAME [KEY1=VALUE1 [KEY2=VALUE2 [...]]]',\n nargs=argparse.ONE_OR_MORE, default=None, action=ProviderAction,\n help=_('Enable and configure a provider.'))\n\n return parser", "response": "Returns an argument parser that can parse program arguments."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrap functions with this decorator to convert them to *tasks*. After wrapping, calling the function will send a message to a queue instead of running the function. :param queue: Queue name for the tasks. :param kwargs: Keyword arguments will be passed to :class:`~kuyruk.Task` constructor. :return: Callable :class:`~kuyruk.Task` object wrapping the original function.", "response": "def task(self, queue: str = 'kuyruk', **kwargs: Any) -> Callable:\n \"\"\"\n Wrap functions with this decorator to convert them to *tasks*.\n After wrapping, calling the function will send a message to\n a queue instead of running the function.\n\n :param queue: Queue name for the tasks.\n :param kwargs: Keyword arguments will be passed to\n :class:`~kuyruk.Task` constructor.\n :return: Callable :class:`~kuyruk.Task` object wrapping the original\n function.\n\n \"\"\"\n def wrapper(f: Callable) -> Task:\n return Task(f, self, queue, **kwargs)\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a new channel from a new connection as a context manager.", "response": "def channel(self) -> Iterator[amqp.Channel]:\n \"\"\"Returns a new channel from a new connection as a context manager.\"\"\"\n with self.connection() as conn:\n ch = conn.channel()\n logger.info('Opened new channel')\n with _safe_close(ch):\n yield ch"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef connection(self) -> Iterator[amqp.Connection]:\n TCP_USER_TIMEOUT = 18 # constant is available on Python 3.6+.\n socket_settings = {TCP_USER_TIMEOUT: self.config.TCP_USER_TIMEOUT}\n\n if sys.platform.startswith('darwin'):\n del socket_settings[TCP_USER_TIMEOUT]\n\n conn = amqp.Connection(\n host=\"%s:%s\" % (self.config.RABBIT_HOST, self.config.RABBIT_PORT),\n userid=self.config.RABBIT_USER,\n password=self.config.RABBIT_PASSWORD,\n virtual_host=self.config.RABBIT_VIRTUAL_HOST,\n connect_timeout=self.config.RABBIT_CONNECT_TIMEOUT,\n read_timeout=self.config.RABBIT_READ_TIMEOUT,\n write_timeout=self.config.RABBIT_WRITE_TIMEOUT,\n socket_settings=socket_settings,\n heartbeat=self.config.RABBIT_HEARTBEAT,\n )\n conn.connect()\n logger.info('Connected to RabbitMQ')\n with _safe_close(conn):\n yield conn", "response": "Returns an iterator over the connection to RabbitMQ."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef scan_videopath(videopath, callback, recursive=False):\n log.debug('scan_videopath(videopath=\"{videopath}\", recursive={recursive})'.format(\n videopath=videopath, recursive=recursive))\n if not videopath.exists():\n log.debug('\"{videopath}\" does not exist'.format(videopath=videopath))\n raise IllegalPathException(path=videopath)\n if videopath.is_dir():\n log.debug('\"{videopath}\" is a directory'.format(videopath=videopath))\n return __scan_folder(videopath, callback=callback, recursive=recursive)\n elif videopath.is_file():\n log.debug('\"{videopath}\" is a file'.format(videopath=videopath))\n videopath_dir = videopath.parent\n [all_subs, _] = filter_files_extensions(videopath_dir.iterdir(), [SUBTITLES_EXT, VIDEOS_EXT])\n [_, video] = filter_files_extensions([videopath], [SUBTITLES_EXT, VIDEOS_EXT])\n sub_videos = [all_subs, video]\n path_subvideos = {videopath_dir: sub_videos}\n return merge_path_subvideo(path_subvideos, callback)\n else:\n log.debug('\"{videopath}\" is of unknown type'.format(videopath=videopath))\n return [], []", "response": "Scan the videopath string for video files and return a tuple with list of videos and list of subtitles that match the video files."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __scan_folder(folder_path, callback, recursive=False):\n log.debug('__scan_folder(folder_path=\"{folder_path}\", recursive={recursive})'.format(folder_path=folder_path,\n recursive=recursive))\n path_subvideos = {}\n # FIXME: a folder named 'movie.avi' is also considered a movie. Fix this.\n if recursive:\n for dir_path, _, files in os.walk(str(folder_path)):\n log.debug('walking current directory:\"{}\"'.format(dir_path))\n path_files = [Path(dir_path) / file for file in files]\n sub_videos = filter_files_extensions(path_files, [SUBTITLES_EXT, VIDEOS_EXT])\n path_subvideos[dir_path] = sub_videos\n else:\n files = [folder_path / f for f in folder_path.iterdir() if f.is_file()] # filter(lambda f: (folder_path / f).is_file(), folder_path.iterdir())\n sub_videos = filter_files_extensions(files, [SUBTITLES_EXT, VIDEOS_EXT])\n path_subvideos[folder_path] = sub_videos\n return merge_path_subvideo(path_subvideos, callback)", "response": "Scan a folder for videos and subtitles."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmerges subtitles into videos.", "response": "def merge_path_subvideo(path_subvideos, callback):\n \"\"\"\n Merge subtitles into videos.\n :param path_subvideos: a dict with paths as key and a list of lists of videos and subtitles\n :param callback: Instance of ProgressCallback\n :return: tuple with list of videos and list of subtitles (videos have matched subtitles)\n \"\"\"\n log.debug('merge_path_subvideo(path_subvideos=<#paths={nb_paths}>)'.format(nb_paths=len(path_subvideos)))\n # FIXME: add logging\n nb_videos = sum([len(subvids[1]) for subvids in path_subvideos.values()])\n\n all_videos = []\n all_subtitles = []\n\n callback.set_range(0, nb_videos)\n\n vid_i = 0\n callback.update(vid_i)\n for path, subvideos in path_subvideos.items():\n [subs_path, vids_path] = subvideos\n subtitles = [LocalSubtitleFile(filepath=sub_path) for sub_path in subs_path]\n all_subtitles.extend(subtitles)\n for vid_path in vids_path:\n try:\n video = VideoFile(vid_path)\n except NotAVideoException:\n continue\n all_videos.append(video)\n\n for subtitle in subtitles:\n if subtitle.matches_video_filename(video):\n video.add_subtitle(subtitle)\n video.get_subtitles().add_candidates(subtitles)\n\n vid_i += 1\n callback.update(vid_i)\n callback.finish(True)\n return all_videos, all_subtitles"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef filter_files_extensions(files, extension_lists):\n log.debug('filter_files_extensions: files=\"{}\"'.format(files))\n result = [[] for _ in extension_lists]\n for file in files:\n ext = file.suffix[1:].lower()\n for ext_i, ext_list in enumerate(extension_lists):\n if ext in ext_list:\n result[ext_i].append(file)\n log.debug('filter_files_extensions result:{}'.format(result))\n return result", "response": "Filter the files in buckets according to extension_lists."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef detect_language_filename(cls, filename):\n log.debug('detect_language(filename=\"{}\") ...'.format(filename))\n root, _ = os.path.splitext(filename)\n fn_lang = cls.DETECT_LANGUAGE_REGEX.findall(root)\n if fn_lang:\n language_part = fn_lang[0]\n try:\n lang = Language.from_unknown(language_part, xx=True, xxx=True)\n log.debug('... SUCCESS: detected from filename: {lang}'.format(lang=lang))\n return lang\n except NotALanguageException:\n pass\n else:\n log.debug('... FAIL: could not detect from filename')\n return UnknownLanguage.create_generic()", "response": "Detect the language of a filename."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef matches_video_filename(self, video):\n\n vid_fn = video.get_filename()\n vid_base, _ = os.path.splitext(vid_fn)\n vid_base = vid_base.lower()\n\n sub_fn = self.get_filename()\n sub_base, _ = os.path.splitext(sub_fn)\n sub_base = sub_base.lower()\n\n log.debug('matches_filename(subtitle=\"{sub_filename}\", video=\"{vid_filename}\") ...'.format(\n sub_filename=sub_fn, vid_filename=vid_fn))\n\n matches = sub_base == vid_base\n\n lang = None\n if not matches:\n if sub_base.startswith(vid_base):\n sub_rest = sub_base[len(vid_base):]\n while len(sub_rest) > 0:\n if sub_rest[0].isalnum():\n break\n sub_rest = sub_rest[1:]\n try:\n lang = Language.from_unknown(sub_rest, xx=True, xxx=True)\n matches = True\n except NotALanguageException:\n matches = False\n\n if matches:\n log.debug('... matches (language={language})'.format(language=lang))\n else:\n log.debug('... does not match')\n return matches", "response": "Detect whether the filename of videofile matches with this SubtitleFile."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninstall logger that will write to file.", "response": "def logging_file_install(path):\n \"\"\"\n Install logger that will write to file. If this function has already installed a handler, replace it.\n :param path: path to the log file, Use None for default file location.\n \"\"\"\n if path is None:\n path = configuration_get_default_folder() / LOGGING_DEFAULTNAME\n\n if not path.parent.exists():\n log.error('File logger installation FAILED!')\n log.error('The directory of the log file does not exist.')\n return\n\n formatter = logging.Formatter(LOGGING_FORMAT)\n logger = logging.getLogger()\n\n logger.removeHandler(LOGGING_HANDLERS['file'])\n\n logFileHandler = logging.handlers.RotatingFileHandler(filename=str(path),\n mode='a',\n maxBytes=LOGGING_MAXBYTES,\n backupCount=LOGGING_BACKUPCOUNT)\n logFileHandler.setLevel(logging.DEBUG)\n logFileHandler.setFormatter(formatter)\n\n LOGGING_HANDLERS['file'] = logFileHandler\n\n logger.addHandler(logFileHandler)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef logging_stream_install(loglevel):\n formatter = logging.Formatter(LOGGING_FORMAT)\n logger = logging.getLogger()\n\n logger.removeHandler(LOGGING_HANDLERS['stream'])\n\n if loglevel == LOGGING_LOGNOTHING:\n streamHandler = None\n else:\n streamHandler = logging.StreamHandler()\n streamHandler.setLevel(loglevel)\n streamHandler.setFormatter(formatter)\n\n LOGGING_HANDLERS['stream'] = streamHandler\n\n if streamHandler:\n logger.addHandler(streamHandler)", "response": "Installs a stream handler that will output to stderr."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse a TLE string into an object.", "response": "def parseFromDelimitedString(obj, buf, offset=0):\n \"\"\"\n Stanford CoreNLP uses the Java \"writeDelimitedTo\" function, which\n writes the size (and offset) of the buffer before writing the object.\n This function handles parsing this message starting from offset 0.\n\n @returns how many bytes of @buf were consumed.\n \"\"\"\n size, pos = _DecodeVarint(buf, offset)\n obj.ParseFromString(buf[offset+pos:offset+pos+size])\n return pos+size"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef writeToDelimitedString(obj, stream=None):\n if stream is None:\n stream = BytesIO()\n\n _EncodeVarint(stream.write, obj.ByteSize(), True)\n stream.write(obj.SerializeToString())\n return stream", "response": "Writes a unicode object to a delimited string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_text(sentence):\n text = \"\"\n for i, tok in enumerate(sentence.token):\n if i != 0:\n text += tok.before\n text += tok.word\n return text", "response": "Converts a Sentence protobuf to a string from\n its tokens."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nshowing a message in the bottom part of the splash screen.", "response": "def showMessage(self, message, *args):\n \"\"\"\n Public method to show a message in the bottom part of the splashscreen.\n\n @param message message to be shown (string or QString)\n \"\"\"\n QSplashScreen.showMessage(\n self, message, Qt.AlignBottom | Qt.AlignRight | Qt.AlignAbsolute, QColor(Qt.white))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing a video at a given path and return a Metadata object.", "response": "def parse_path(path):\n \"\"\"\n Parse a video at filepath, using pymediainfo framework.\n :param path: path of video to parse as string\n \"\"\"\n import pymediainfo\n\n metadata = Metadata()\n log.debug('pymediainfo: parsing \"{path}\" ...'.format(path=path))\n parseRes = pymediainfo.MediaInfo.parse(str(path))\n log.debug('... parsing FINISHED')\n for track in parseRes.tracks:\n log.debug('... found track type: \"{track_type}\"'.format(track_type=track.track_type))\n if track.track_type == 'Video':\n duration_ms = track.duration\n framerate = track.frame_rate\n framecount = track.frame_count\n log.debug('mode={mode}'.format(mode=track.frame_rate_mode))\n if duration_ms is None or framerate is None:\n log.debug('... Video track does not have duration and/or framerate.')\n continue\n log.debug('... duration = {duration_ms} ms, framerate = {framerate} fps'.format(duration_ms=duration_ms,\n framerate=framerate))\n metadata.add_metadata(\n MetadataVideoTrack(\n duration_ms=duration_ms,\n framerate=float(framerate),\n framecount=framecount\n )\n )\n return metadata"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_size(self):\n if self._size is None:\n self._size = self._filepath.stat().st_size\n return self._size", "response": "Get size of this VideoFile in bytes\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the hash of this local videofile", "response": "def get_osdb_hash(self):\n \"\"\"\n Get the hash of this local videofile\n :return: hash as string\n \"\"\"\n if self._osdb_hash is None:\n self._osdb_hash = self._calculate_osdb_hash()\n return self._osdb_hash"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _calculate_osdb_hash(self):\n log.debug('_calculate_OSDB_hash() of \"{path}\" ...'.format(path=self._filepath))\n f = self._filepath.open(mode='rb')\n\n file_size = self.get_size()\n\n longlong_format = 'Q' # unsigned long long little endian\n size_longlong = struct.calcsize(longlong_format)\n\n block_size = min(file_size, 64 << 10) # 64kiB\n block_size = block_size & ~(size_longlong - 1) # lower round on multiple of longlong\n\n nb_longlong = block_size // size_longlong\n fmt = '<{nbll}{member_format}'.format(\n nbll=nb_longlong,\n member_format=longlong_format)\n\n hash_int = file_size\n\n buffer = f.read(block_size)\n list_longlong = struct.unpack(fmt, buffer)\n hash_int += sum(list_longlong)\n\n f.seek(-block_size, os.SEEK_END)\n buffer = f.read(block_size)\n list_longlong = struct.unpack(fmt, buffer)\n hash_int += sum(list_longlong)\n\n f.close()\n hash_str = '{:016x}'.format(hash_int)[-16:]\n log.debug('hash(\"{}\")={}'.format(self.get_filepath(), hash_str))\n return hash_str", "response": "Calculate OSDB hash of this VideoFile"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nimport a module by its name.", "response": "def import_module(name: str) -> ModuleType:\n \"\"\"Import module by it's name from following places in order:\n - main module\n - current working directory\n - Python path\n\n \"\"\"\n logger.debug(\"Importing module: %s\", name)\n if name == main_module_name():\n return main_module\n\n return importlib.import_module(name)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main_module_name() -> str:\n if not hasattr(main_module, '__file__'):\n # running from interactive shell\n return None\n\n main_filename = os.path.basename(main_module.__file__)\n module_name, ext = os.path.splitext(main_filename)\n return module_name", "response": "Returns main module and module name pair."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write_stream(src_file, destination_path):\n with open(destination_path, 'wb') as destination_file:\n shutil.copyfileobj(fsrc=src_file, fdst=destination_file)", "response": "Write the file - like src_file object to the string destination_path"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds necessary directories based on a list of file paths", "response": "def build_dirs(files):\n '''\n Build necessary directories based on a list of file paths\n '''\n for i in files:\n if type(i) is list:\n build_dirs(i)\n continue\n else:\n if len(i['path']) > 1:\n addpath = os.path.join(os.getcwd(), *i['path'][:-1])\n subdirs = all_subdirs(os.getcwd())\n if addpath and addpath not in subdirs:\n os.makedirs(addpath)\n print 'just made path', addpath"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nasks the user which files in file_list he or she is interested in. Return indices for the files in file_list", "response": "def get_want_file_pos(file_list):\n '''\n Ask the user which files in file_list he or she is interested in.\n Return indices for the files inside file_list\n '''\n want_file_pos = []\n print '\\nFiles contained:\\n'\n for i in file_list:\n print(os.path.join(*i['path']))\n while 1:\n all_answer = raw_input('\\nDo you want all these files? (y/n): ')\n if all_answer in ('y', 'n'):\n break\n if all_answer == 'y':\n want_file_pos = range(len(file_list))\n return want_file_pos\n if all_answer == 'n':\n for j, tfile in enumerate(file_list):\n while 1:\n file_answer = raw_input('Do you want {}? '\n '(y/n): '.format(os.path.join\n (*tfile['path'])))\n\n if file_answer in ('y', 'n'):\n break\n if file_answer == 'y':\n want_file_pos.append(j)\n print \"Here are all the files you want:\"\n for k in want_file_pos:\n print os.path.join(*file_list[k]['path'])\n return want_file_pos"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the starting position of a list of files by iteratively summing their lengths", "response": "def get_file_starts(file_list):\n '''\n Return the starting position (in bytes) of a list of files by\n iteratively summing their lengths\n '''\n starts = []\n total = 0\n for i in file_list:\n starts.append(total)\n total += i['length']\n print starts\n return starts"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_rightmost_index(byte_index=0, file_starts=[0]):\n\n '''\n Retrieve the highest-indexed file that starts at or before byte_index.\n '''\n i = 1\n while i <= len(file_starts):\n start = file_starts[-i]\n if start <= byte_index:\n return len(file_starts) - i\n else:\n i += 1\n else:\n raise Exception('byte_index lower than all file_starts')", "response": "Retrieve the index of the rightmost element in a file list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_next_want_file(self, byte_index, block):\n '''\n Returns the leftmost file in the user's list of wanted files\n (want_file_pos). If the first file it finds isn't in the list,\n it will keep searching until the length of 'block' is exceeded.\n '''\n while block:\n rightmost = get_rightmost_index(byte_index=byte_index,\n file_starts=self.file_starts)\n if rightmost in self.want_file_pos:\n return rightmost, byte_index, block\n else:\n file_start = (self.file_starts\n [rightmost])\n file_length = self.file_list[rightmost]['length']\n bytes_rem = file_start + file_length - byte_index\n if len(block) > bytes_rem:\n block = block[bytes_rem:]\n byte_index = byte_index + bytes_rem\n else:\n block = ''\n else:\n return None", "response": "Returns the next wanted file in the user s list of wanted files."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends the state of the BTC at the time the visualizer connects and initializing it.", "response": "def vis_init(self):\n '''\n Sends the state of the BTC at the time the visualizer connects,\n initializing it.\n '''\n init_dict = {}\n init_dict['kind'] = 'init'\n assert len(self.want_file_pos) == len(self.heads_and_tails)\n init_dict['want_file_pos'] = self.want_file_pos\n init_dict['files'] = self.file_list\n init_dict['heads_and_tails'] = self.heads_and_tails\n init_dict['num_pieces'] = self.num_pieces\n self.broadcast(init_dict)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a message to the visualizer or enqueue for later ArcGIS", "response": "def broadcast(self, data_dict):\n '''\n Send to the visualizer (if there is one) or enqueue for later\n '''\n if self.vis_socket:\n self.queued_messages.append(data_dict)\n self.send_all_updates()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef build_payload(self):\n '''\n Builds the payload that will be sent in tracker_request\n '''\n payload = {}\n hashed_info = hashlib.sha1(tparser.bencode(self.torrent_dict['info']))\n self.hash_string = hashed_info.digest()\n self.peer_id = ('-DR' + VERSION +\n ''.join(random.sample(ALPHANUM, 13)))\n assert len(self.peer_id) == 20\n payload['info_hash'] = self.hash_string\n payload['peer_id'] = self.peer_id\n payload['port'] = self.port\n payload['uploaded'] = 0\n payload['downloaded'] = 0\n payload['left'] = self.length\n payload['compact'] = 1\n payload['supportcrypto'] = 1\n payload['event'] = 'started'\n return payload", "response": "Builds the payload that will be sent in tracker_request\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend the initial request to the tracker", "response": "def tracker_request(self):\n '''\n Sends the initial request to the tracker, compiling list of all peers\n announcing to the tracker\n '''\n\n assert self.torrent_dict['info']\n payload = self.build_payload()\n\n if self.torrent_dict['announce'].startswith('udp'):\n raise Exception('need to deal with UDP')\n\n else:\n self.r = requests.get(self.torrent_dict['announce'],\n params=payload)\n\n # Decoding response from tracker\n self.tracker_response = tparser.bdecode(self.r.content)\n self.get_peer_ips()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating list of peer IPs from tracker response.", "response": "def get_peer_ips(self):\n '''\n Generates list of peer IPs from tracker response. Note: not all of\n these IPs might be good, which is why we only init peer objects for\n the subset that respond to handshake\n '''\n presponse = [ord(i) for i in self.tracker_response['peers']]\n while presponse:\n peer_ip = (('.'.join(str(x) for x in presponse[0:4]),\n 256 * presponse[4] + presponse[5]))\n if peer_ip not in self.peer_ips:\n self.peer_ips.append(peer_ip)\n presponse = presponse[6:]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef handshake_peers(self):\n '''\n pstrlen = length of pstr as one byte\n pstr = BitTorrent protocol\n reserved = chr(0)*8\n info_hash = 20-byte hash above (aka self.hash_string)\n peer_id = 20-byte string\n '''\n\n pstr = 'BitTorrent protocol'\n pstrlen = len(pstr)\n info_hash = self.hash_string\n peer_id = self.peer_id\n\n packet = ''.join([chr(pstrlen), pstr, chr(0) * 8, info_hash,\n peer_id])\n print \"Here's my packet {}\".format(repr(packet))\n # TODO -- add some checks in here so that I'm talking\n # to a maximum of 30 peers\n\n # TODO -- think about why i'm deleting self.peer_ips.\n # What was the point of it? Why won't I need it?\n # Think about what we're doing -- using this list to create\n # new peer objects. Should make this functional, that way I\n # can also call when I get new peers.\n for i in self.peer_ips:\n if len(self.peer_dict) >= 30:\n break\n s = socket.socket()\n s.setblocking(True)\n s.settimeout(0.5)\n try:\n s.connect(i)\n except socket.timeout:\n print '{} timed out on connect'.format(s.fileno())\n continue\n except socket.error:\n print '{} threw a socket error'.format(s.fileno())\n continue\n except:\n raise Exception\n s.send(packet)\n try:\n data = s.recv(68) # Peer's handshake - len from docs\n if data:\n print 'From {} received: {}'.format(s.fileno(), repr(data))\n self.initpeer(s)\n except:\n print '{} timed out on recv'.format(s.fileno())\n continue\n else:\n self.peer_ips = []", "response": "This function is called by the peer_ip_list method to create new I\n objects."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new peer object for a nvalid socket and adds it to reactor s select list", "response": "def initpeer(self, sock):\n '''\n Creates a new peer object for a nvalid socket and adds it to reactor's\n listen list\n '''\n location_json = requests.request(\"GET\", \"http://freegeoip.net/json/\"\n + sock.getpeername()[0]).content\n location = json.loads(location_json)\n tpeer = peer.Peer(sock, self.reactor, self, location)\n self.peer_dict[sock] = tpeer\n self.reactor.select_list.append(tpeer)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read(self):\n try:\n bytes = self.sock.recv(self.max_size)\n except:\n self.torrent.kill_peer(self)\n return\n '''\n Chain of events:\n - process_input\n - check save_state and read length, id, and message accordingly\n - if we have a piece (really a block), we piece.save it out\n inside call to ppiece\n - If we've completed a piece we:\n - Tell the switchboard to write it out\n - init a new piece\n '''\n if len(bytes) == 0:\n print 'Got 0 bytes from fileno {}.'.format(self.fileno())\n self.torrent.kill_peer(self)\n self.process_input(bytes)", "response": "Read a new entry from the switchboard."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ppiece(self, content):\n '''\n Process a piece that we've received from a peer, writing it out to\n one or more files\n '''\n piece_index, byte_begin = struct.unpack('!ii', content[0:8])\n\n # TODO -- figure out a better way to catch this error.\n # How is piece_index getting swapped out from under me?\n if piece_index != self.piece.index:\n return\n\n assert byte_begin % REQUEST_SIZE == 0\n block_begin = byte_begin / REQUEST_SIZE\n block = content[8:]\n self.piece.save(index=block_begin, bytes=block)\n if self.piece.complete:\n piece_bytes = self.piece.get_bytes()\n if self.piece.index == self.torrent.last_piece:\n piece_bytes = piece_bytes[:self.torrent.last_piece_length]\n if hashlib.sha1(piece_bytes).digest() == (self.torrent.torrent_dict\n ['info']['pieces']\n [20 * piece_index:20 *\n piece_index + 20]):\n\n print 'hash matches'\n\n # Take care of visualizer stuff\n piece_dict = {'kind': 'piece', 'peer': self.sock.getpeername(),\n 'piece_index': piece_index}\n self.torrent.switchboard.broadcast(piece_dict)\n\n print ('writing piece {}. Length is '\n '{}').format(repr(piece_bytes)[:10] + '...',\n len(piece_bytes))\n\n # Write out\n byte_index = piece_index * self.torrent.piece_length\n self.piece = self.init_piece()\n self.request_all()\n self.torrent.switchboard.write(byte_index, piece_bytes)\n self.torrent.switchboard.mark_off(piece_index)\n print self.torrent.switchboard.bitfield\n if self.torrent.switchboard.complete:\n print '\\nDownload complete\\n'\n self.reactor.is_running = False\n else:\n print \"Bad data -- hash doesn't match. Discarding piece.\"\n self.piece = self.init_piece()\n self.request_all()", "response": "Process a piece that we ve received from a peer writing it out to a new file and one or more files."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the connection status of the data store.", "response": "def is_connected(self):\n \"\"\" Returns the connection status of the data store.\n\n Returns:\n bool: ``True`` if the data store is connected to the MongoDB server.\n \"\"\"\n if self._client is not None:\n try:\n self._client.server_info()\n except ConnectionFailure:\n return False\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef connect(self):\n mongodb_args = {\n 'host': self.host,\n 'port': self.port,\n 'username': self._username,\n 'password': self._password,\n 'authSource': self._auth_source,\n 'serverSelectionTimeoutMS': self._connect_timeout\n }\n\n if self._auth_mechanism is not None:\n mongodb_args['authMechanism'] = self._auth_mechanism\n\n self._client = MongoClient(**mongodb_args)\n\n if self._handle_reconnect:\n self._client = MongoClientProxy(self._client)", "response": "Establishes a connection to the MongoDB server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef exists(self, workflow_id):\n try:\n db = self._client[self.database]\n col = db[WORKFLOW_DATA_COLLECTION_NAME]\n return col.find_one({\"_id\": ObjectId(workflow_id)}) is not None\n\n except ConnectionFailure:\n raise DataStoreNotConnected()", "response": "Checks whether a document with the specified workflow id already exists in the data store."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a new document to the data store and returns its id.", "response": "def add(self, payload=None):\n \"\"\" Adds a new document to the data store and returns its id.\n\n Args:\n payload (dict): Dictionary of initial data that should be stored\n in the new document in the meta section.\n\n Raises:\n DataStoreNotConnected: If the data store is not connected to the server.\n\n Returns:\n str: The id of the newly created document.\n \"\"\"\n try:\n db = self._client[self.database]\n col = db[WORKFLOW_DATA_COLLECTION_NAME]\n return str(col.insert_one({\n DataStoreDocumentSection.Meta:\n payload if isinstance(payload, dict) else {},\n DataStoreDocumentSection.Data: {}\n }).inserted_id)\n\n except ConnectionFailure:\n raise DataStoreNotConnected()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove a document specified by its id from the data store.", "response": "def remove(self, workflow_id):\n \"\"\" Removes a document specified by its id from the data store.\n\n All associated GridFs documents are deleted as well.\n\n Args:\n workflow_id (str): The id of the document that represents a workflow run.\n\n Raises:\n DataStoreNotConnected: If the data store is not connected to the server.\n \"\"\"\n try:\n db = self._client[self.database]\n fs = GridFSProxy(GridFS(db.unproxied_object))\n\n for grid_doc in fs.find({\"workflow_id\": workflow_id},\n no_cursor_timeout=True):\n fs.delete(grid_doc._id)\n\n col = db[WORKFLOW_DATA_COLLECTION_NAME]\n return col.delete_one({\"_id\": ObjectId(workflow_id)})\n\n except ConnectionFailure:\n raise DataStoreNotConnected()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get(self, workflow_id):\n try:\n db = self._client[self.database]\n fs = GridFSProxy(GridFS(db.unproxied_object))\n return DataStoreDocument(db[WORKFLOW_DATA_COLLECTION_NAME], fs, workflow_id)\n\n except ConnectionFailure:\n raise DataStoreNotConnected()", "response": "Returns the document for the given workflow id."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the field specified by its key from the specified section. This method access the specified section of the workflow document and returns the value for the given key. Args: key (str): The key pointing to the value that should be retrieved. It supports MongoDB's dot notation for nested fields. default: The default value that is returned if the key does not exist. section (DataStoreDocumentSection): The section from which the data should be retrieved. Returns: object: The value from the field that the specified key is pointing to. If the key does not exist, the default value is returned. If no default value is provided and the key does not exist ``None`` is returned.", "response": "def get(self, key, default=None, *, section=DataStoreDocumentSection.Data):\n \"\"\" Return the field specified by its key from the specified section.\n\n This method access the specified section of the workflow document and returns the\n value for the given key.\n\n Args:\n key (str): The key pointing to the value that should be retrieved. It supports\n MongoDB's dot notation for nested fields.\n default: The default value that is returned if the key does not exist.\n section (DataStoreDocumentSection): The section from which the data should\n be retrieved.\n\n Returns:\n object: The value from the field that the specified key is pointing to. If the\n key does not exist, the default value is returned. If no default value\n is provided and the key does not exist ``None`` is returned.\n \"\"\"\n key_notation = '.'.join([section, key])\n try:\n return self._decode_value(self._data_from_dotnotation(key_notation, default))\n except KeyError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstores a value under the specified key in the given section of the document. This method stores a value into the specified section of the workflow data store document. Any existing value is overridden. Before storing a value, any linked GridFS document under the specified key is deleted. Args: key (str): The key pointing to the value that should be stored/updated. It supports MongoDB's dot notation for nested fields. value: The value that should be stored/updated. section (DataStoreDocumentSection): The section from which the data should be retrieved. Returns: bool: ``True`` if the value could be set/updated, otherwise ``False``.", "response": "def set(self, key, value, *, section=DataStoreDocumentSection.Data):\n \"\"\" Store a value under the specified key in the given section of the document.\n\n This method stores a value into the specified section of the workflow data store\n document. Any existing value is overridden. Before storing a value, any linked\n GridFS document under the specified key is deleted.\n\n Args:\n key (str): The key pointing to the value that should be stored/updated.\n It supports MongoDB's dot notation for nested fields.\n value: The value that should be stored/updated.\n section (DataStoreDocumentSection): The section from which the data should\n be retrieved.\n\n Returns:\n bool: ``True`` if the value could be set/updated, otherwise ``False``.\n \"\"\"\n key_notation = '.'.join([section, key])\n\n try:\n self._delete_gridfs_data(self._data_from_dotnotation(key_notation,\n default=None))\n except KeyError:\n logger.info('Adding new field {} to the data store'.format(key_notation))\n\n result = self._collection.update_one(\n {\"_id\": ObjectId(self._workflow_id)},\n {\n \"$set\": {\n key_notation: self._encode_value(value)\n },\n \"$currentDate\": {\"lastModified\": True}\n }\n )\n return result.modified_count == 1"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds a value to a list in the specified section of the document.", "response": "def push(self, key, value, *, section=DataStoreDocumentSection.Data):\n \"\"\" Appends a value to a list in the specified section of the document.\n\n Args:\n key (str): The key pointing to the value that should be stored/updated.\n It supports MongoDB's dot notation for nested fields.\n value: The value that should be appended to a list in the data store.\n section (DataStoreDocumentSection): The section from which the data should\n be retrieved.\n\n Returns:\n bool: ``True`` if the value could be appended, otherwise ``False``.\n \"\"\"\n key_notation = '.'.join([section, key])\n result = self._collection.update_one(\n {\"_id\": ObjectId(self._workflow_id)},\n {\n \"$push\": {\n key_notation: self._encode_value(value)\n },\n \"$currentDate\": {\"lastModified\": True}\n }\n )\n return result.modified_count == 1"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef extend(self, key, values, *, section=DataStoreDocumentSection.Data):\n key_notation = '.'.join([section, key])\n if not isinstance(values, list):\n return False\n\n result = self._collection.update_one(\n {\"_id\": ObjectId(self._workflow_id)},\n {\n \"$push\": {\n key_notation: {\"$each\": self._encode_value(values)}\n },\n \"$currentDate\": {\"lastModified\": True}\n }\n )\n return result.modified_count == 1", "response": "Extends a list in the data store with the elements of values."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the MongoDB data for the specified key using dot notation.", "response": "def _data_from_dotnotation(self, key, default=None):\n \"\"\" Returns the MongoDB data from a key using dot notation.\n\n Args:\n key (str): The key to the field in the workflow document. Supports MongoDB's\n dot notation for embedded fields.\n default (object): The default value that is returned if the key\n does not exist.\n\n Returns:\n object: The data for the specified key or the default value.\n \"\"\"\n if key is None:\n raise KeyError('NoneType is not a valid key!')\n\n doc = self._collection.find_one({\"_id\": ObjectId(self._workflow_id)})\n if doc is None:\n return default\n\n for k in key.split('.'):\n doc = doc[k]\n\n return doc"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nencoding the value into a GridFS object.", "response": "def _encode_value(self, value):\n \"\"\" Encodes the value such that it can be stored into MongoDB.\n\n Any primitive types are stored directly into MongoDB, while non-primitive types\n are pickled and stored as GridFS objects. The id pointing to a GridFS object\n replaces the original value.\n\n Args:\n value (object): The object that should be encoded for storing in MongoDB.\n\n Returns:\n object: The encoded value ready to be stored in MongoDB.\n \"\"\"\n if isinstance(value, (int, float, str, bool, datetime)):\n return value\n elif isinstance(value, list):\n return [self._encode_value(item) for item in value]\n elif isinstance(value, dict):\n result = {}\n for key, item in value.items():\n result[key] = self._encode_value(item)\n return result\n else:\n return self._gridfs.put(Binary(pickle.dumps(value)),\n workflow_id=self._workflow_id)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndecode the value into a Python object.", "response": "def _decode_value(self, value):\n \"\"\" Decodes the value by turning any binary data back into Python objects.\n\n The method searches for ObjectId values, loads the associated binary data from\n GridFS and returns the decoded Python object.\n\n Args:\n value (object): The value that should be decoded.\n\n Raises:\n DataStoreDecodingError: An ObjectId was found but the id is not a valid\n GridFS id.\n DataStoreDecodeUnknownType: The type of the specified value is unknown.\n\n Returns:\n object: The decoded value as a valid Python object.\n \"\"\"\n if isinstance(value, (int, float, str, bool, datetime)):\n return value\n elif isinstance(value, list):\n return [self._decode_value(item) for item in value]\n elif isinstance(value, dict):\n result = {}\n for key, item in value.items():\n result[key] = self._decode_value(item)\n return result\n elif isinstance(value, ObjectId):\n if self._gridfs.exists({\"_id\": value}):\n return pickle.loads(self._gridfs.get(value).read())\n else:\n raise DataStoreGridfsIdInvalid()\n else:\n raise DataStoreDecodeUnknownType()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndeletes all GridFS data that is linked by fields in the specified data.", "response": "def _delete_gridfs_data(self, data):\n \"\"\" Delete all GridFS data that is linked by fields in the specified data.\n\n Args:\n data: The data that is parsed for MongoDB ObjectIDs. The linked GridFs object\n for any ObjectID is deleted.\n \"\"\"\n if isinstance(data, ObjectId):\n if self._gridfs.exists({\"_id\": data}):\n self._gridfs.delete(data)\n else:\n raise DataStoreGridfsIdInvalid()\n elif isinstance(data, list):\n for item in data:\n self._delete_gridfs_data(item)\n elif isinstance(data, dict):\n for key, item in data.items():\n self._delete_gridfs_data(item)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Homecall call for a given callsign.", "response": "def get_homecall(callsign):\n \"\"\"Strips off country prefixes (HC2/DH1TW) and activity suffixes (DH1TW/P).\n\n Args:\n callsign (str): Amateur Radio callsign\n\n Returns:\n str: callsign without country/activity pre/suffixes\n\n Raises:\n ValueError: No callsign found in string\n\n Example:\n The following code retrieves the home call for \"HC2/DH1TW/P\"\n\n >>> from pyhamtools import LookupLib, Callinfo\n >>> my_lookuplib = LookupLib(lookuptype=\"countryfile\")\n >>> cic = Callinfo(my_lookuplib)\n >>> cic.get_homecall(\"HC2/DH1TW/P\")\n DH1TW\n\n \"\"\"\n\n callsign = callsign.upper()\n homecall = re.search('[\\d]{0,1}[A-Z]{1,2}\\d([A-Z]{1,4}|\\d{3,3}|\\d{1,3}[A-Z])[A-Z]{0,5}', callsign)\n if homecall:\n homecall = homecall.group(0)\n return homecall\n else:\n raise ValueError"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _iterate_prefix(self, callsign, timestamp=timestamp_now):\n prefix = callsign\n\n if re.search('(VK|AX|VI)9[A-Z]{3}', callsign): #special rule for VK9 calls\n if timestamp > datetime(2006,1,1, tzinfo=UTC):\n prefix = callsign[0:3]+callsign[4:5]\n\n while len(prefix) > 0:\n try:\n return self._lookuplib.lookup_prefix(prefix, timestamp)\n except KeyError:\n prefix = prefix.replace(' ', '')[:-1]\n continue\n raise KeyError", "response": "iterate over the prefix in the database"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndismantles a callsign by analyzing the callsign s identity and returns a dictionary of the information that can be used to identify the callsign.", "response": "def _dismantle_callsign(self, callsign, timestamp=timestamp_now):\n \"\"\" try to identify the callsign's identity by analyzing it in the following order:\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Raises:\n KeyError: Callsign could not be identified\n\n\n \"\"\"\n entire_callsign = callsign.upper()\n\n if re.search('[/A-Z0-9\\-]{3,15}', entire_callsign): # make sure the call has at least 3 characters\n\n if re.search('\\-\\d{1,3}$', entire_callsign): # cut off any -10 / -02 appendixes\n callsign = re.sub('\\-\\d{1,3}$', '', entire_callsign)\n\n if re.search('/[A-Z0-9]{1,4}/[A-Z0-9]{1,4}$', callsign):\n callsign = re.sub('/[A-Z0-9]{1,4}$', '', callsign) # cut off 2. appendix DH1TW/HC2/P -> DH1TW/HC2\n\n # multiple character appendix (callsign/xxx)\n if re.search('[A-Z0-9]{4,10}/[A-Z0-9]{2,4}$', callsign): # case call/xxx, but ignoring /p and /m or /5\n appendix = re.search('/[A-Z0-9]{2,4}$', callsign)\n appendix = re.sub('/', '', appendix.group(0))\n self._logger.debug(\"appendix: \" + appendix)\n\n if appendix == 'MM': # special case Martime Mobile\n #self._mm = True\n return {\n 'adif': 999,\n 'continent': '',\n 'country': 'MARITIME MOBILE',\n 'cqz': 0,\n 'latitude': 0.0,\n 'longitude': 0.0\n }\n elif appendix == 'AM': # special case Aeronautic Mobile\n return {\n 'adif': 998,\n 'continent': '',\n 'country': 'AIRCAFT MOBILE',\n 'cqz': 0,\n 'latitude': 0.0,\n 'longitude': 0.0\n }\n elif appendix == 'QRP': # special case QRP\n callsign = re.sub('/QRP', '', callsign)\n return self._iterate_prefix(callsign, timestamp)\n elif appendix == 'QRPP': # special case QRPP\n callsign = re.sub('/QRPP', '', callsign)\n return self._iterate_prefix(callsign, timestamp)\n elif appendix == 'BCN': # filter all beacons\n callsign = re.sub('/BCN', '', callsign)\n data = self._iterate_prefix(callsign, timestamp).copy()\n data[const.BEACON] = True\n return data\n elif appendix == \"LH\": # Filter all Lighthouses\n callsign = re.sub('/LH', '', callsign)\n return self._iterate_prefix(callsign, timestamp)\n elif re.search('[A-Z]{3}', appendix): #case of US county(?) contest N3HBX/UAL\n callsign = re.sub('/[A-Z]{3}$', '', callsign)\n return self._iterate_prefix(callsign, timestamp)\n\n else:\n # check if the appendix is a valid country prefix\n return self._iterate_prefix(re.sub('/', '', appendix), timestamp)\n\n # Single character appendix (callsign/x)\n elif re.search('/[A-Z0-9]$', callsign): # case call/p or /b /m or /5 etc.\n appendix = re.search('/[A-Z0-9]$', callsign)\n appendix = re.sub('/', '', appendix.group(0))\n\n if appendix == 'B': # special case Beacon\n callsign = re.sub('/B', '', callsign)\n data = self._iterate_prefix(callsign, timestamp).copy()\n data[const.BEACON] = True\n return data\n\n elif re.search('\\d$', appendix):\n area_nr = re.search('\\d$', appendix).group(0)\n callsign = re.sub('/\\d$', '', callsign) #remove /number\n if len(re.findall(r'\\d+', callsign)) == 1: #call has just on digit e.g. DH1TW\n callsign = re.sub('[\\d]+', area_nr, callsign)\n else: # call has several digits e.g. 7N4AAL\n pass # no (two) digit prefix contries known where appendix would change entitiy\n return self._iterate_prefix(callsign, timestamp)\n\n else:\n return self._iterate_prefix(callsign, timestamp)\n\n # regular callsigns, without prefix or appendix\n elif re.match('^[\\d]{0,1}[A-Z]{1,2}\\d([A-Z]{1,4}|\\d{3,3}|\\d{1,3}[A-Z])[A-Z]{0,5}$', callsign):\n return self._iterate_prefix(callsign, timestamp)\n\n # callsigns with prefixes (xxx/callsign)\n elif re.search('^[A-Z0-9]{1,4}/', entire_callsign):\n pfx = re.search('^[A-Z0-9]{1,4}/', entire_callsign)\n pfx = re.sub('/', '', pfx.group(0))\n #make sure that the remaining part is actually a callsign (avoid: OZ/JO81)\n rest = re.search('/[A-Z0-9]+', entire_callsign)\n rest = re.sub('/', '', rest.group(0))\n if re.match('^[\\d]{0,1}[A-Z]{1,2}\\d([A-Z]{1,4}|\\d{3,3}|\\d{1,3}[A-Z])[A-Z]{0,5}$', rest):\n return self._iterate_prefix(pfx)\n\n if entire_callsign in callsign_exceptions:\n return self._iterate_prefix(callsign_exceptions[entire_callsign])\n\n self._logger.debug(\"Could not decode \" + callsign)\n raise KeyError(\"Callsign could not be decoded\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_all(self, callsign, timestamp=timestamp_now):\n callsign_data = self._lookup_callsign(callsign, timestamp)\n\n try:\n cqz = self._lookuplib.lookup_zone_exception(callsign, timestamp)\n callsign_data[const.CQZ] = cqz\n except KeyError:\n pass\n\n return callsign_data", "response": "Lookup a callsign and return all available data for the specified date."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if a callsign is valid for a given date.", "response": "def is_valid_callsign(self, callsign, timestamp=timestamp_now):\n \"\"\" Checks if a callsign is valid\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n bool: True / False\n\n Example:\n The following checks if \"DH1TW\" is a valid callsign\n\n >>> from pyhamtools import LookupLib, Callinfo\n >>> my_lookuplib = LookupLib(lookuptype=\"countryfile\")\n >>> cic = Callinfo(my_lookuplib)\n >>> cic.is_valid_callsign(\"DH1TW\")\n True\n\n \"\"\"\n try:\n if self.get_all(callsign, timestamp):\n return True\n except KeyError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns Latitude and Longitude for a callsign.", "response": "def get_lat_long(self, callsign, timestamp=timestamp_now):\n \"\"\" Returns Latitude and Longitude for a callsign\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n dict: Containing Latitude and Longitude\n\n Raises:\n KeyError: No data found for callsign\n\n Example:\n The following code returns Latitude & Longitude for \"DH1TW\"\n\n >>> from pyhamtools import LookupLib, Callinfo\n >>> my_lookuplib = LookupLib(lookuptype=\"countryfile\")\n >>> cic = Callinfo(my_lookuplib)\n >>> cic.get_lat_long(\"DH1TW\")\n {\n 'latitude': 51.0,\n 'longitude': -10.0\n }\n\n Note:\n Unfortunately, in most cases the returned Latitude and Longitude are not very precise.\n Clublog and Country-files.com use the country's capital coordinates in most cases, if no\n dedicated entry in the database exists. Best results will be retrieved with QRZ.com Lookup.\n\n \"\"\"\n callsign_data = self.get_all(callsign, timestamp=timestamp)\n return {\n const.LATITUDE: callsign_data[const.LATITUDE],\n const.LONGITUDE: callsign_data[const.LONGITUDE]\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the CQ Zone of a callsign", "response": "def get_cqz(self, callsign, timestamp=timestamp_now):\n \"\"\" Returns CQ Zone of a callsign\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n int: containing the callsign's CQ Zone\n\n Raises:\n KeyError: no CQ Zone found for callsign\n\n \"\"\"\n return self.get_all(callsign, timestamp)[const.CQZ]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the ITU Zone of a callsign", "response": "def get_ituz(self, callsign, timestamp=timestamp_now):\n \"\"\" Returns ITU Zone of a callsign\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n int: containing the callsign's CQ Zone\n\n Raises:\n KeyError: No ITU Zone found for callsign\n\n Note:\n Currently, only Country-files.com lookup database contains ITU Zones\n\n \"\"\"\n return self.get_all(callsign, timestamp)[const.ITUZ]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_country_name(self, callsign, timestamp=timestamp_now):\n return self.get_all(callsign, timestamp)[const.COUNTRY]", "response": "Returns the name of the country that the callsign is located in"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_adif_id(self, callsign, timestamp=timestamp_now):\n return self.get_all(callsign, timestamp)[const.ADIF]", "response": "Returns the ADIF id of a callsign s country."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the continent Identifier of a callsign", "response": "def get_continent(self, callsign, timestamp=timestamp_now):\n \"\"\" Returns the continent Identifier of a callsign\n\n Args:\n callsign (str): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n str: continent identified\n\n Raises:\n KeyError: No Continent found for callsign\n\n Note:\n The following continent identifiers are used:\n\n - EU: Europe\n - NA: North America\n - SA: South America\n - AS: Asia\n - AF: Africa\n - OC: Oceania\n - AN: Antarctica\n \"\"\"\n return self.get_all(callsign, timestamp)[const.CONTINENT]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_indices(lst, element):\n result = []\n offset = -1\n while True:\n try:\n offset = lst.index(element, offset+1)\n except ValueError:\n return result\n result.append(offset)", "response": "Returns the indices for all occurrences of element in lst."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a fully initialised workflow object from a workflow script.", "response": "def from_name(cls, name, *, queue=DefaultJobQueueName.Workflow,\n clear_data_store=True, arguments=None):\n \"\"\" Create a workflow object from a workflow script.\n\n Args:\n name (str): The name of the workflow script.\n queue (str): Name of the queue the workflow should be scheduled to.\n clear_data_store (bool): Remove any documents created during the workflow\n run in the data store after the run.\n arguments (dict): Dictionary of additional arguments that are ingested\n into the data store prior to the execution of the workflow.\n\n Returns:\n Workflow: A fully initialised workflow object\n \"\"\"\n new_workflow = cls(queue=queue, clear_data_store=clear_data_store)\n new_workflow.load(name, arguments=arguments)\n return new_workflow"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nimports the workflow script and loads all known objects into the data store.", "response": "def load(self, name, *, arguments=None, validate_arguments=True, strict_dag=False):\n \"\"\" Import the workflow script and load all known objects.\n\n The workflow script is treated like a module and imported\n into the Python namespace. After the import, the method looks\n for instances of known classes and stores a reference for further\n use in the workflow object.\n\n Args:\n name (str): The name of the workflow script.\n arguments (dict): Dictionary of additional arguments that are ingested\n into the data store prior to the execution of the workflow.\n validate_arguments (bool): Whether to check that all required arguments have\n been supplied.\n strict_dag (bool): If true then the loaded workflow module must contain an\n instance of Dag.\n\n Raises:\n WorkflowArgumentError: If the workflow requires arguments to be set that\n were not supplied to the workflow.\n WorkflowImportError: If the import of the workflow fails.\n \"\"\"\n arguments = {} if arguments is None else arguments\n\n try:\n workflow_module = importlib.import_module(name)\n\n dag_present = False\n\n # extract objects of specific types from the workflow module\n for key, obj in workflow_module.__dict__.items():\n if isinstance(obj, Dag):\n self._dags_blueprint[obj.name] = obj\n dag_present = True\n elif isinstance(obj, Parameters):\n self._parameters.extend(obj)\n\n self._name = name\n self._docstring = inspect.getdoc(workflow_module)\n del sys.modules[name]\n\n if strict_dag and not dag_present:\n raise WorkflowImportError(\n 'Workflow does not include a dag {}'.format(name))\n\n if validate_arguments:\n missing_parameters = self._parameters.check_missing(arguments)\n if len(missing_parameters) > 0:\n raise WorkflowArgumentError(\n 'The following parameters are required ' +\n 'by the workflow, but are missing: {}'.format(\n ', '.join(missing_parameters)))\n\n self._provided_arguments = arguments\n\n except (TypeError, ImportError):\n logger.error('Cannot import workflow {}'.format(name))\n raise WorkflowImportError('Cannot import workflow {}'.format(name))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun all autostart dags in the workflow.", "response": "def run(self, config, data_store, signal_server, workflow_id):\n \"\"\" Run all autostart dags in the workflow.\n\n Only the dags that are flagged as autostart are started.\n\n Args:\n config (Config): Reference to the configuration object from which the\n settings for the workflow are retrieved.\n data_store (DataStore): A DataStore object that is fully initialised and\n connected to the persistent data storage.\n signal_server (Server): A signal Server object that receives requests\n from dags and tasks.\n workflow_id (str): A unique workflow id that represents this workflow run\n \"\"\"\n self._workflow_id = workflow_id\n self._celery_app = create_app(config)\n\n # pre-fill the data store with supplied arguments\n args = self._parameters.consolidate(self._provided_arguments)\n for key, value in args.items():\n data_store.get(self._workflow_id).set(key, value)\n\n # start all dags with the autostart flag set to True\n for name, dag in self._dags_blueprint.items():\n if dag.autostart:\n self._queue_dag(name)\n\n # as long as there are dags in the list keep running\n while self._dags_running:\n if config.workflow_polling_time > 0.0:\n sleep(config.workflow_polling_time)\n\n # handle new requests from dags, tasks and the library (e.g. cli, web)\n for i in range(MAX_SIGNAL_REQUESTS):\n request = signal_server.receive()\n if request is None:\n break\n\n try:\n response = self._handle_request(request)\n if response is not None:\n signal_server.send(response)\n else:\n signal_server.restore(request)\n except (RequestActionUnknown, RequestFailed):\n signal_server.send(Response(success=False, uid=request.uid))\n\n # remove any dags and their result data that finished running\n for name, dag in list(self._dags_running.items()):\n if dag.ready():\n if self._celery_app.conf.result_expires == 0:\n dag.forget()\n del self._dags_running[name]\n elif dag.failed():\n self._stop_workflow = True\n\n # remove the signal entry\n signal_server.clear()\n\n # delete all entries in the data_store under this workflow id, if requested\n if self._clear_data_store:\n data_store.remove(self._workflow_id)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _queue_dag(self, name, *, data=None):\n if self._stop_workflow:\n return None\n\n if name not in self._dags_blueprint:\n raise DagNameUnknown()\n\n new_dag = copy.deepcopy(self._dags_blueprint[name])\n new_dag.workflow_name = self.name\n self._dags_running[new_dag.name] = self._celery_app.send_task(\n JobExecPath.Dag, args=(new_dag, self._workflow_id, data),\n queue=new_dag.queue, routing_key=new_dag.queue)\n\n return new_dag.name", "response": "Add a new dag to the queue."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling an incoming request by forwarding it to the appropriate method.", "response": "def _handle_request(self, request):\n \"\"\" Handle an incoming request by forwarding it to the appropriate method.\n\n Args:\n request (Request): Reference to a request object containing the\n incoming request.\n\n Raises:\n RequestActionUnknown: If the action specified in the request is not known.\n\n Returns:\n Response: A response object containing the response from the method handling\n the request.\n \"\"\"\n if request is None:\n return Response(success=False, uid=request.uid)\n\n action_map = {\n 'start_dag': self._handle_start_dag,\n 'stop_workflow': self._handle_stop_workflow,\n 'join_dags': self._handle_join_dags,\n 'stop_dag': self._handle_stop_dag,\n 'is_dag_stopped': self._handle_is_dag_stopped\n }\n\n if request.action in action_map:\n return action_map[request.action](request)\n else:\n raise RequestActionUnknown()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _handle_start_dag(self, request):\n dag_name = self._queue_dag(name=request.payload['name'],\n data=request.payload['data'])\n return Response(success=dag_name is not None, uid=request.uid,\n payload={'dag_name': dag_name})", "response": "The handler for the start_dag request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _handle_stop_dag(self, request):\n if (request.payload['name'] is not None) and \\\n (request.payload['name'] not in self._stop_dags):\n self._stop_dags.append(request.payload['name'])\n return Response(success=True, uid=request.uid)", "response": "The handler for the stop_dag request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _handle_is_dag_stopped(self, request):\n return Response(success=True,\n uid=request.uid,\n payload={\n 'is_stopped': request.payload['dag_name'] in self._stop_dags\n })", "response": "The handler for the dag_stopped request."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start_dag(self, dag, *, data=None):\n return self._client.send(\n Request(\n action='start_dag',\n payload={'name': dag.name if isinstance(dag, Dag) else dag,\n 'data': data if isinstance(data, MultiTaskData) else None}\n )\n ).payload['dag_name']", "response": "Starts execution of a dag."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwaits for the specified dags to terminate.", "response": "def join_dags(self, names=None):\n \"\"\" Wait for the specified dags to terminate.\n\n This function blocks until the specified dags terminate. If no dags are specified\n wait for all dags of the workflow, except the dag of the task calling this signal,\n to terminate.\n\n Args:\n names (list): The names of the dags that have to terminate.\n\n Returns:\n bool: True if all the signal was sent successfully.\n \"\"\"\n return self._client.send(\n Request(\n action='join_dags',\n payload={'names': names}\n )\n ).success"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a stop signal to the specified dag or the dag that hosts this task.", "response": "def stop_dag(self, name=None):\n \"\"\" Send a stop signal to the specified dag or the dag that hosts this task.\n\n Args:\n name str: The name of the dag that should be stopped. If no name is given the\n dag that hosts this task is stopped.\n\n Upon receiving the stop signal, the dag will not queue any new tasks and wait\n for running tasks to terminate.\n\n Returns:\n bool: True if the signal was sent successfully.\n \"\"\"\n return self._client.send(\n Request(\n action='stop_dag',\n payload={'name': name if name is not None else self._dag_name}\n )\n ).success"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_stopped(self):\n resp = self._client.send(\n Request(\n action='is_dag_stopped',\n payload={'dag_name': self._dag_name}\n )\n )\n return resp.payload['is_stopped']", "response": "Check whether the task is stopped."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef event_stream(app, *, filter_by_prefix=None):\n q = Queue()\n\n def handle_event(event):\n if filter_by_prefix is None or\\\n (filter_by_prefix is not None and\n event['type'].startswith(filter_by_prefix)):\n q.put(event)\n\n def receive_events():\n with app.connection() as connection:\n recv = app.events.Receiver(connection, handlers={\n '*': handle_event\n })\n\n recv.capture(limit=None, timeout=None, wakeup=True)\n\n t = threading.Thread(target=receive_events)\n t.start()\n\n while True:\n yield q.get(block=True)", "response": "A generator function that returns celery events."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate an event model from a celery event.", "response": "def create_event_model(event):\n \"\"\" Factory function that turns a celery event into an event object.\n\n Args:\n event (dict): A dictionary that represents a celery event.\n\n Returns:\n object: An event object representing the received event.\n\n Raises:\n JobEventTypeUnsupported: If an unsupported celery job event was received.\n WorkerEventTypeUnsupported: If an unsupported celery worker event was received.\n EventTypeUnknown: If an unknown event type (neither job nor worker) was received.\n \"\"\"\n if event['type'].startswith('task'):\n factory = {\n JobEventName.Started: JobStartedEvent,\n JobEventName.Succeeded: JobSucceededEvent,\n JobEventName.Stopped: JobStoppedEvent,\n JobEventName.Aborted: JobAbortedEvent\n }\n if event['type'] in factory:\n return factory[event['type']].from_event(event)\n else:\n raise JobEventTypeUnsupported(\n 'Unsupported event type {}'.format(event['type']))\n elif event['type'].startswith('worker'):\n raise WorkerEventTypeUnsupported(\n 'Unsupported event type {}'.format(event['type']))\n else:\n raise EventTypeUnknown('Unknown event type {}'.format(event['type']))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef config_required(f):\n def new_func(obj, *args, **kwargs):\n if 'config' not in obj:\n click.echo(_style(obj.get('show_color', False),\n 'Could not find a valid configuration file!',\n fg='red', bold=True))\n raise click.Abort()\n else:\n return f(obj, *args, **kwargs)\n return update_wrapper(new_func, f)", "response": "Decorator that checks whether a configuration file is set."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef ingest_config_obj(ctx, *, silent=True):\n try:\n ctx.obj['config'] = Config.from_file(ctx.obj['config_path'])\n except ConfigLoadError as err:\n click.echo(_style(ctx.obj['show_color'], str(err), fg='red', bold=True))\n if not silent:\n raise click.Abort()", "response": "Ingest the configuration object into the click context."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncommand line client for lightflow.", "response": "def cli(ctx, config, no_color):\n \"\"\" Command line client for lightflow. A lightweight, high performance pipeline\n system for synchrotrons.\n\n Lightflow is being developed at the Australian Synchrotron.\n \"\"\"\n ctx.obj = {\n 'show_color': not no_color if no_color is not None else True,\n 'config_path': config\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a default configuration file.", "response": "def config_default(dest):\n \"\"\" Create a default configuration file.\n\n \\b\n DEST: Path or file name for the configuration file.\n \"\"\"\n conf_path = Path(dest).resolve()\n if conf_path.is_dir():\n conf_path = conf_path / LIGHTFLOW_CONFIG_NAME\n\n conf_path.write_text(Config.default())\n click.echo('Configuration written to {}'.format(conf_path))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists the current configuration.", "response": "def config_list(ctx):\n \"\"\" List the current configuration. \"\"\"\n ingest_config_obj(ctx, silent=False)\n click.echo(json.dumps(ctx.obj['config'].to_dict(), indent=4))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef config_examples(dest, user_dir):\n examples_path = Path(lightflow.__file__).parents[1] / 'examples'\n if examples_path.exists():\n dest_path = Path(dest).resolve()\n if not user_dir:\n dest_path = dest_path / 'examples'\n\n if dest_path.exists():\n if not click.confirm('Directory already exists. Overwrite existing files?',\n default=True, abort=True):\n return\n else:\n dest_path.mkdir()\n\n for example_file in examples_path.glob('*.py'):\n shutil.copy(str(example_file), str(dest_path / example_file.name))\n click.echo('Copied examples to {}'.format(str(dest_path)))\n else:\n click.echo('The examples source path does not exist')", "response": "Copy the examples to a directory."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef workflow_list(obj):\n try:\n for wf in list_workflows(config=obj['config']):\n click.echo('{:23} {}'.format(\n _style(obj['show_color'], wf.name, bold=True),\n wf.docstring.split('\\n')[0] if wf.docstring is not None else ''))\n except WorkflowDefinitionError as e:\n click.echo(_style(obj['show_color'],\n 'The graph {} in workflow {} is not a directed acyclic graph'.\n format(e.graph_name, e.workflow_name), fg='red', bold=True))", "response": "List all available workflows."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a workflow to the queue.", "response": "def workflow_start(obj, queue, keep_data, name, workflow_args):\n \"\"\" Send a workflow to the queue.\n\n \\b\n NAME: The name of the workflow that should be started.\n WORKFLOW_ARGS: Workflow arguments in the form key1=value1 key2=value2.\n \"\"\"\n try:\n start_workflow(name=name,\n config=obj['config'],\n queue=queue,\n clear_data_store=not keep_data,\n store_args=dict([arg.split('=', maxsplit=1)\n for arg in workflow_args]))\n except (WorkflowArgumentError, WorkflowImportError) as e:\n click.echo(_style(obj['show_color'],\n 'An error occurred when trying to start the workflow',\n fg='red', bold=True))\n click.echo('{}'.format(e))\n except WorkflowDefinitionError as e:\n click.echo(_style(obj['show_color'],\n 'The graph {} in workflow {} is not a directed acyclic graph'.\n format(e.graph_name, e.workflow_name), fg='red', bold=True))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstop one or more workflows.", "response": "def workflow_stop(obj, names):\n \"\"\" Stop one or more running workflows.\n\n \\b\n NAMES: The names, ids or job ids of the workflows that should be stopped.\n Leave empty to stop all running workflows.\n \"\"\"\n if len(names) == 0:\n msg = 'Would you like to stop all workflows?'\n else:\n msg = '\\n{}\\n\\n{}'.format('\\n'.join(names),\n 'Would you like to stop these jobs?')\n\n if click.confirm(msg, default=True, abort=True):\n stop_workflow(obj['config'], names=names if len(names) > 0 else None)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef workflow_status(obj, details):\n show_colors = obj['show_color']\n config_cli = obj['config'].cli\n\n if details:\n temp_form = '{:>{}} {:20} {:25} {:25} {:38} {}'\n else:\n temp_form = '{:>{}} {:20} {:25} {} {} {}'\n\n click.echo('\\n')\n click.echo(temp_form.format(\n 'Status',\n 12,\n 'Name',\n 'Start Time',\n 'ID' if details else '',\n 'Job' if details else '',\n 'Arguments'\n ))\n click.echo('-' * (138 if details else 75))\n\n def print_jobs(jobs, *, label='', color='green'):\n for job in jobs:\n start_time = job.start_time if job.start_time is not None else 'unknown'\n\n click.echo(temp_form.format(\n _style(show_colors, label, fg=color, bold=True),\n 25 if show_colors else 12,\n job.name,\n start_time.replace(tzinfo=pytz.utc).astimezone().strftime(\n config_cli['time_format']),\n job.workflow_id if details else '',\n job.id if details else '',\n ','.join(['{}={}'.format(k, v) for k, v in job.arguments.items()]))\n )\n\n # running jobs\n print_jobs(list_jobs(config=obj['config'],\n status=JobStatus.Active,\n filter_by_type=JobType.Workflow),\n label='Running', color='green')\n\n # scheduled jobs\n print_jobs(list_jobs(config=obj['config'],\n status=JobStatus.Scheduled,\n filter_by_type=JobType.Workflow),\n label='Scheduled', color='blue')\n\n # registered jobs\n print_jobs(list_jobs(config=obj['config'],\n status=JobStatus.Registered,\n filter_by_type=JobType.Workflow),\n label='Registered', color='yellow')\n\n # reserved jobs\n print_jobs(list_jobs(config=obj['config'],\n status=JobStatus.Reserved,\n filter_by_type=JobType.Workflow),\n label='Reserved', color='yellow')", "response": "Show the status of the workflows."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef worker_start(obj, queues, name, celery_args):\n try:\n start_worker(queues=queues.split(','),\n config=obj['config'],\n name=name,\n celery_args=celery_args)\n except DataStoreNotConnected:\n click.echo(_style(obj['show_color'],\n 'Cannot connect to the Data Store server. Is the server running?',\n fg='red', bold=True))", "response": "Start a worker process."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstop running workers. \\b WORKER_IDS: The IDs of the worker that should be stopped or none to stop them all.", "response": "def worker_stop(obj, worker_ids):\n \"\"\" Stop running workers.\n\n \\b\n WORKER_IDS: The IDs of the worker that should be stopped or none to stop them all.\n \"\"\"\n if len(worker_ids) == 0:\n msg = 'Would you like to stop all workers?'\n else:\n msg = '\\n{}\\n\\n{}'.format('\\n'.join(worker_ids),\n 'Would you like to stop these workers?')\n\n if click.confirm(msg, default=True, abort=True):\n stop_worker(obj['config'],\n worker_ids=list(worker_ids) if len(worker_ids) > 0 else None)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nshow the status of all running workers.", "response": "def worker_status(obj, filter_queues, details):\n \"\"\" Show the status of all running workers. \"\"\"\n show_colors = obj['show_color']\n\n f_queues = filter_queues.split(',') if filter_queues is not None else None\n\n workers = list_workers(config=obj['config'], filter_by_queues=f_queues)\n if len(workers) == 0:\n click.echo('No workers are running at the moment.')\n return\n\n for ws in workers:\n click.echo('{} {}'.format(_style(show_colors, 'Worker:', fg='blue', bold=True),\n _style(show_colors, ws.name, fg='blue')))\n click.echo('{:23} {}'.format(_style(show_colors, '> pid:', bold=True), ws.pid))\n\n if details:\n click.echo('{:23} {}'.format(_style(show_colors, '> concurrency:', bold=True),\n ws.concurrency))\n click.echo('{:23} {}'.format(_style(show_colors, '> processes:', bold=True),\n ', '.join(str(p) for p in ws.process_pids)))\n click.echo('{:23} {}://{}:{}/{}'.format(_style(show_colors, '> broker:',\n bold=True),\n ws.broker.transport,\n ws.broker.hostname,\n ws.broker.port,\n ws.broker.virtual_host))\n\n click.echo('{:23} {}'.format(_style(show_colors, '> queues:', bold=True),\n ', '.join([q.name for q in ws.queues])))\n\n if details:\n click.echo('{:23} {}'.format(_style(show_colors, '> job count:', bold=True),\n ws.job_count))\n\n jobs = list_jobs(config=obj['config'], filter_by_worker=ws.name)\n click.echo('{:23} [{}]'.format(_style(show_colors, '> jobs:', bold=True),\n len(jobs) if len(jobs) > 0 else 'No tasks'))\n\n for job in jobs:\n click.echo('{:15} {} {}'.format(\n '',\n _style(show_colors, '{}'.format(job.name),\n bold=True, fg=JOB_COLOR[job.type]),\n _style(show_colors, '({}) [{}] <{}> on {}'.format(\n job.type, job.workflow_id, job.id, job.worker_pid),\n fg=JOB_COLOR[job.type])))\n\n click.echo('\\n')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting the worker and workflow event stream.", "response": "def monitor(ctx, details):\n \"\"\" Show the worker and workflow event stream. \"\"\"\n ingest_config_obj(ctx, silent=False)\n\n show_colors = ctx.obj['show_color']\n\n event_display = {\n JobEventName.Started: {'color': 'blue', 'label': 'started'},\n JobEventName.Succeeded: {'color': 'green', 'label': 'succeeded'},\n JobEventName.Stopped: {'color': 'yellow', 'label': 'stopped'},\n JobEventName.Aborted: {'color': 'red', 'label': 'aborted'}\n }\n\n click.echo('\\n')\n click.echo('{:>10} {:>12} {:25} {:18} {:16} {:28} {}'.format(\n 'Status',\n 'Type',\n 'Name',\n 'Duration (sec)',\n 'Queue' if details else '',\n 'Workflow ID' if details else '',\n 'Worker' if details else ''))\n\n click.echo('-' * (136 if details else 65))\n\n for event in workflow_events(ctx.obj['config']):\n evt_disp = event_display[event.event]\n click.echo('{:>{}} {:>{}} {:25} {:18} {:16} {:28} {}'.format(\n _style(show_colors, evt_disp['label'], fg=evt_disp['color']),\n 20 if show_colors else 10,\n _style(show_colors, event.type, bold=True, fg=JOB_COLOR[event.type]),\n 24 if show_colors else 12,\n event.name,\n '{0:.3f}'.format(event.duration) if event.duration is not None else '',\n event.queue if details else '',\n event.workflow_id if details else '',\n event.hostname if details else ''))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning an extension by its name.", "response": "def ext(obj, ext_name, ext_args):\n \"\"\" Run an extension by its name.\n\n \\b\n EXT_NAME: The name of the extension.\n EXT_ARGS: Arguments that are passed to the extension.\n \"\"\"\n try:\n mod = import_module('lightflow_{}.__main__'.format(ext_name))\n mod.main(ext_args)\n except ImportError as err:\n click.echo(_style(obj['show_color'],\n 'An error occurred when trying to call the extension',\n fg='red', bold=True))\n click.echo('{}'.format(err))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _style(enabled, text, **kwargs):\n if enabled:\n return click.style(text, **kwargs)\n else:\n return text", "response": "Helper function to enable or disable styling of the base item in a node tree."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef freq_to_band(freq):\n\n band = None\n mode = None\n if ((freq >= 135) and (freq <= 138)):\n band = 2190\n mode = const.CW\n elif ((freq >= 1800) and (freq <= 2000)):\n band = 160\n if ((freq >= 1800) and (freq < 1838)):\n mode = const.CW\n elif ((freq >= 1838) and (freq < 1840)):\n mode = const.DIGITAL\n elif ((freq >= 1840) and (freq < 2000)):\n mode = const.LSB\n elif ((freq >= 3500) and (freq <= 4000)):\n band = 80\n if ((freq >= 3500) and (freq < 3580)):\n mode = const.CW\n elif ((freq >= 3580) and (freq < 3600)):\n mode = const.DIGITAL\n elif ((freq >= 3600) and (freq < 4000)):\n mode = const.LSB\n elif ((freq >= 5000) and (freq <= 5500)):\n band = 60\n elif ((freq >= 7000) and (freq <= 7300)):\n band = 40\n if ((freq >= 7000) and (freq < 7040)):\n mode = const.CW\n elif ((freq >= 7040) and (freq < 7050)):\n mode = const.DIGITAL\n elif ((freq >= 7050) and (freq < 7300)):\n mode = const.LSB\n elif ((freq >= 10100) and (freq <= 10150)):\n band = 30\n if ((freq >= 10100) and (freq < 10140)):\n mode = const.CW\n elif ((freq >= 10140) and (freq < 10150)):\n mode = const.DIGITAL\n elif ((freq >= 14000) and (freq <= 14350)):\n band = 20\n if ((freq >= 14000) and (freq < 14070)):\n mode = const.CW\n elif ((freq >= 14070) and (freq < 14099)):\n mode = const.DIGITAL\n elif ((freq >= 14100) and (freq < 14350)):\n mode = const.USB\n elif ((freq >= 18068) and (freq <= 18268)):\n band = 17\n if ((freq >= 18068) and (freq < 18095)):\n mode = const.CW\n elif ((freq >= 18095) and (freq < 18110)):\n mode = const.DIGITAL\n elif ((freq >= 18110) and (freq < 18268)):\n mode = const.USB\n elif ((freq >= 21000) and (freq <= 21450)):\n band = 15\n if ((freq >= 21000) and (freq < 21070)):\n mode = const.CW\n elif ((freq >= 21070) and (freq < 21150)):\n mode = const.DIGITAL\n elif ((freq >= 21150) and (freq < 21450)):\n mode = const.USB\n elif ((freq >= 24890) and (freq <= 24990)):\n band = 12\n if ((freq >= 24890) and (freq < 24915)):\n mode = const.CW\n elif ((freq >= 24915) and (freq < 24930)):\n mode = const.DIGITAL\n elif ((freq >= 24930) and (freq < 24990)):\n mode = const.USB\n elif ((freq >= 28000) and (freq <= 29700)):\n band = 10\n if ((freq >= 28000) and (freq < 28070)):\n mode = const.CW\n elif ((freq >= 28070) and (freq < 28190)):\n mode = const.DIGITAL\n elif ((freq >= 28300) and (freq < 29700)):\n mode = const.USB\n elif ((freq >= 50000) and (freq <= 54000)):\n band = 6\n if ((freq >= 50000) and (freq < 50100)):\n mode = const.CW\n elif ((freq >= 50100) and (freq < 50500)):\n mode = const.USB\n elif ((freq >= 50500) and (freq < 51000)):\n mode = const.DIGITAL\n elif ((freq >= 70000) and (freq <= 71000)):\n band = 4\n mode = None\n elif ((freq >= 144000) and (freq <= 148000)):\n band = 2\n if ((freq >= 144000) and (freq < 144150)):\n mode = const.CW\n elif ((freq >= 144150) and (freq < 144400)):\n mode = const.USB\n elif ((freq >= 144400) and (freq < 148000)):\n mode = None\n elif ((freq >= 220000) and (freq <= 226000)):\n band = 1.25 #1.25m\n mode = None\n elif ((freq >= 420000) and (freq <= 470000)):\n band = 0.7 #70cm\n mode = None\n elif ((freq >= 902000) and (freq <= 928000)):\n band = 0.33 #33cm US\n mode = None\n elif ((freq >= 1200000) and (freq <= 1300000)):\n band = 0.23 #23cm\n mode = None\n elif ((freq >= 2390000) and (freq <= 2450000)):\n band = 0.13 #13cm\n mode = None\n elif ((freq >= 3300000) and (freq <= 3500000)):\n band = 0.09 #9cm\n mode = None\n elif ((freq >= 5650000) and (freq <= 5850000)):\n band = 0.053 #5.3cm\n mode = None\n elif ((freq >= 10000000) and (freq <= 10500000)):\n band = 0.03 #3cm\n mode = None\n elif ((freq >= 24000000) and (freq <= 24050000)):\n band = 0.0125 #1,25cm\n mode = None\n elif ((freq >= 47000000) and (freq <= 47200000)):\n band = 0.0063 #6,3mm\n mode = None\n else:\n raise KeyError\n\n return {\"band\": band, \"mode\": mode}", "response": "converts a Frequency [ kHz ] into the band and mode according to the IARU bandplan\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_app(config):\n\n # configure the celery logging system with the lightflow settings\n setup_logging.connect(partial(_initialize_logging, config), weak=False)\n task_postrun.connect(partial(_cleanup_workflow, config), weak=False)\n\n # patch Celery to use cloudpickle instead of pickle for serialisation\n patch_celery()\n\n # create the main celery app and load the configuration\n app = Celery('lightflow')\n app.conf.update(**config.celery)\n\n # overwrite user supplied settings to make sure celery works with lightflow\n app.conf.update(\n task_serializer='pickle',\n accept_content=['pickle'],\n result_serializer='pickle',\n task_default_queue=DefaultJobQueueName.Task\n )\n\n if isinstance(app.conf.include, list):\n app.conf.include.extend(LIGHTFLOW_INCLUDE)\n else:\n if len(app.conf.include) > 0:\n raise ConfigOverwriteError(\n 'The content in the include config will be overwritten')\n app.conf.include = LIGHTFLOW_INCLUDE\n\n return app", "response": "Create a fully configured Celery application object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _cleanup_workflow(config, task_id, args, **kwargs):\n from lightflow.models import Workflow\n if isinstance(args[0], Workflow):\n if config.celery['result_expires'] == 0:\n AsyncResult(task_id).forget()", "response": "Cleanup the results of a workflow when it finished."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a single celery task on a worker.", "response": "def execute_task(self, task, workflow_id, data=None):\n \"\"\" Celery task that runs a single task on a worker.\n\n Args:\n self (Task): Reference to itself, the celery task object.\n task (BaseTask): Reference to the task object that performs the work\n in its run() method.\n workflow_id (string): The unique ID of the workflow run that started this task.\n data (MultiTaskData): An optional MultiTaskData object that contains the data\n that has been passed down from upstream tasks.\n \"\"\"\n start_time = datetime.utcnow()\n\n store_doc = DataStore(**self.app.user_options['config'].data_store,\n auto_connect=True).get(workflow_id)\n store_loc = 'log.{}.tasks.{}'.format(task.dag_name, task.name)\n\n def handle_callback(message, event_type, exc=None):\n msg = '{}: {}'.format(message, str(exc)) if exc is not None else message\n\n # set the logging level\n if event_type == JobEventName.Stopped:\n logger.warning(msg)\n elif event_type == JobEventName.Aborted:\n logger.error(msg)\n else:\n logger.info(msg)\n\n current_time = datetime.utcnow()\n\n # store provenance information about a task\n if event_type != JobEventName.Started:\n duration = (current_time - start_time).total_seconds()\n\n store_doc.set(key='{}.end_time'.format(store_loc),\n value=current_time,\n section=DataStoreDocumentSection.Meta)\n\n store_doc.set(key='{}.duration'.format(store_loc),\n value=duration,\n section=DataStoreDocumentSection.Meta)\n else:\n # store provenance information about a task\n store_doc.set(key='{}.start_time'.format(store_loc),\n value=start_time,\n section=DataStoreDocumentSection.Meta)\n\n store_doc.set(key='{}.worker'.format(store_loc),\n value=self.request.hostname,\n section=DataStoreDocumentSection.Meta)\n\n store_doc.set(key='{}.queue'.format(store_loc),\n value=task.queue,\n section=DataStoreDocumentSection.Meta)\n duration = None\n\n # send custom celery event\n self.send_event(event_type,\n job_type=JobType.Task,\n name=task.name,\n queue=task.queue,\n time=current_time,\n workflow_id=workflow_id,\n duration=duration)\n\n # store job specific meta information wth the job\n self.update_state(meta={'name': task.name,\n 'queue': task.queue,\n 'type': JobType.Task,\n 'workflow_id': workflow_id})\n\n # send start celery event\n handle_callback('Start task <{}>'.format(task.name), JobEventName.Started)\n\n # run the task and capture the result\n return task._run(\n data=data,\n store=store_doc,\n signal=TaskSignal(Client(\n SignalConnection(**self.app.user_options['config'].signal, auto_connect=True),\n request_key=workflow_id),\n task.dag_name),\n context=TaskContext(task.name, task.dag_name, task.workflow_name,\n workflow_id, self.request.hostname),\n success_callback=partial(handle_callback,\n message='Complete task <{}>'.format(task.name),\n event_type=JobEventName.Succeeded),\n stop_callback=partial(handle_callback,\n message='Stop task <{}>'.format(task.name),\n event_type=JobEventName.Stopped),\n abort_callback=partial(handle_callback,\n message='Abort workflow <{}> by task <{}>'.format(\n task.workflow_name, task.name),\n event_type=JobEventName.Aborted))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_celery(cls, broker_dict):\n return BrokerStats(\n hostname=broker_dict['hostname'],\n port=broker_dict['port'],\n transport=broker_dict['transport'],\n virtual_host=broker_dict['virtual_host']\n )", "response": "Create a BrokerStats object from a dictionary returned by celery."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_dict(self):\n return {\n 'hostname': self.hostname,\n 'port': self.port,\n 'transport': self.transport,\n 'virtual_host': self.virtual_host\n }", "response": "Return a dictionary of the broker stats."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a WorkerStats object from a dictionary returned by celery.", "response": "def from_celery(cls, name, worker_dict, queues):\n \"\"\" Create a WorkerStats object from the dictionary returned by celery.\n\n Args:\n name (str): The name of the worker.\n worker_dict (dict): The dictionary as returned by celery.\n queues (list): A list of QueueStats objects that represent the queues this\n worker is listening on.\n\n Returns:\n WorkerStats: A fully initialized WorkerStats object.\n \"\"\"\n return WorkerStats(\n name=name,\n broker=BrokerStats.from_celery(worker_dict['broker']),\n pid=worker_dict['pid'],\n process_pids=worker_dict['pool']['processes'],\n concurrency=worker_dict['pool']['max-concurrency'],\n job_count=worker_dict['pool']['writes']['total'],\n queues=queues\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_dict(self):\n return {\n 'name': self.name,\n 'broker': self.broker.to_dict(),\n 'pid': self.pid,\n 'process_pids': self.process_pids,\n 'concurrency': self.concurrency,\n 'job_count': self.job_count,\n 'queues': [q.to_dict() for q in self.queues]\n }", "response": "Return a dictionary of the worker stats."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a JobStats object from a dictionary returned by celery.", "response": "def from_celery(cls, worker_name, job_dict, celery_app):\n \"\"\" Create a JobStats object from the dictionary returned by celery.\n\n Args:\n worker_name (str): The name of the worker this jobs runs on.\n job_dict (dict): The dictionary as returned by celery.\n celery_app: Reference to a celery application object.\n\n Returns:\n JobStats: A fully initialized JobStats object.\n \"\"\"\n if not isinstance(job_dict, dict) or 'id' not in job_dict:\n raise JobStatInvalid('The job description is missing important fields.')\n\n async_result = AsyncResult(id=job_dict['id'], app=celery_app)\n a_info = async_result.info if isinstance(async_result.info, dict) else None\n\n return JobStats(\n name=a_info.get('name', '') if a_info is not None else '',\n job_id=job_dict['id'],\n job_type=a_info.get('type', '') if a_info is not None else '',\n workflow_id=a_info.get('workflow_id', '') if a_info is not None else '',\n queue=a_info.get('queue', '') if a_info is not None else '',\n start_time=a_info.get('start_time', None) if a_info is not None else None,\n arguments=a_info.get('arguments', {}) if a_info is not None else {},\n acknowledged=job_dict['acknowledged'],\n func_name=job_dict['type'],\n hostname=job_dict['hostname'],\n worker_name=worker_name,\n worker_pid=job_dict['worker_pid'],\n routing_key=job_dict['delivery_info']['routing_key']\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a dictionary of the job stats.", "response": "def to_dict(self):\n \"\"\" Return a dictionary of the job stats.\n\n Returns:\n dict: Dictionary of the stats.\n \"\"\"\n return {\n 'name': self.name,\n 'id': self.id,\n 'type': self.type,\n 'workflow_id': self.workflow_id,\n 'queue': self.queue,\n 'start_time': self.start_time,\n 'arguments': self.arguments,\n 'acknowledged': self.acknowledged,\n 'func_name': self.func_name,\n 'hostname': self.hostname,\n 'worker_name': self.worker_name,\n 'worker_pid': self.worker_pid,\n 'routing_key': self.routing_key\n }"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_event(cls, event):\n return cls(\n uuid=event['uuid'],\n job_type=event['job_type'],\n event_type=event['type'],\n queue=event['queue'],\n hostname=event['hostname'],\n pid=event['pid'],\n name=event['name'],\n workflow_id=event['workflow_id'],\n event_time=event['time'],\n duration=event['duration']\n )", "response": "Create a JobEvent object from a dictionary returned by celery."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef start_workflow(name, config, *, queue=DefaultJobQueueName.Workflow,\n clear_data_store=True, store_args=None):\n \"\"\" Start a single workflow by sending it to the workflow queue.\n\n Args:\n name (str): The name of the workflow that should be started. Refers to the\n name of the workflow file without the .py extension.\n config (Config): Reference to the configuration object from which the\n settings for the workflow are retrieved.\n queue (str): Name of the queue the workflow should be scheduled to.\n clear_data_store (bool): Remove any documents created during the workflow\n run in the data store after the run.\n store_args (dict): Dictionary of additional arguments that are ingested into the\n data store prior to the execution of the workflow.\n Returns:\n str: The ID of the workflow job.\n Raises:\n WorkflowArgumentError: If the workflow requires arguments to be set in store_args\n that were not supplied to the workflow.\n WorkflowImportError: If the import of the workflow fails.\n \"\"\"\n try:\n wf = Workflow.from_name(name,\n queue=queue,\n clear_data_store=clear_data_store,\n arguments=store_args)\n except DirectedAcyclicGraphInvalid as e:\n raise WorkflowDefinitionError(workflow_name=name,\n graph_name=e.graph_name)\n\n celery_app = create_app(config)\n result = celery_app.send_task(JobExecPath.Workflow,\n args=(wf,), queue=queue, routing_key=queue)\n return result.id", "response": "Starts a single workflow by sending it to the workflow queue."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stop_workflow(config, *, names=None):\n jobs = list_jobs(config, filter_by_type=JobType.Workflow)\n\n if names is not None:\n filtered_jobs = []\n for job in jobs:\n if (job.id in names) or (job.name in names) or (job.workflow_id in names):\n filtered_jobs.append(job)\n else:\n filtered_jobs = jobs\n\n success = []\n failed = []\n for job in filtered_jobs:\n client = Client(SignalConnection(**config.signal, auto_connect=True),\n request_key=job.workflow_id)\n\n if client.send(Request(action='stop_workflow')).success:\n success.append(job)\n else:\n failed.append(job)\n\n return success, failed", "response": "Stop one or more workflows."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list_workflows(config):\n workflows = []\n for path in config.workflows:\n filenames = glob.glob(os.path.join(os.path.abspath(path), '*.py'))\n\n for filename in filenames:\n module_name = os.path.splitext(os.path.basename(filename))[0]\n workflow = Workflow()\n try:\n workflow.load(module_name, validate_arguments=False, strict_dag=True)\n workflows.append(workflow)\n except DirectedAcyclicGraphInvalid as e:\n raise WorkflowDefinitionError(workflow_name=module_name,\n graph_name=e.graph_name)\n except WorkflowImportError:\n continue\n\n return workflows", "response": "List all available workflows."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of Jobs that are in the given status.", "response": "def list_jobs(config, *, status=JobStatus.Active,\n filter_by_type=None, filter_by_worker=None):\n \"\"\" Return a list of Celery jobs.\n\n Args:\n config (Config): Reference to the configuration object from which the\n settings are retrieved.\n status (JobStatus): The status of the jobs that should be returned.\n filter_by_type (list): Restrict the returned jobs to the types in this list.\n filter_by_worker (list): Only return jobs that were registered, reserved or are\n running on the workers given in this list of worker names. Using\n this option will increase the performance.\n\n Returns:\n list: A list of JobStats.\n \"\"\"\n celery_app = create_app(config)\n\n # option to filter by the worker (improves performance)\n if filter_by_worker is not None:\n inspect = celery_app.control.inspect(\n destination=filter_by_worker if isinstance(filter_by_worker, list)\n else [filter_by_worker])\n else:\n inspect = celery_app.control.inspect()\n\n # get active, registered or reserved jobs\n if status == JobStatus.Active:\n job_map = inspect.active()\n elif status == JobStatus.Registered:\n job_map = inspect.registered()\n elif status == JobStatus.Reserved:\n job_map = inspect.reserved()\n elif status == JobStatus.Scheduled:\n job_map = inspect.scheduled()\n else:\n job_map = None\n\n if job_map is None:\n return []\n\n result = []\n for worker_name, jobs in job_map.items():\n for job in jobs:\n try:\n job_stats = JobStats.from_celery(worker_name, job, celery_app)\n\n if (filter_by_type is None) or (job_stats.type == filter_by_type):\n result.append(job_stats)\n except JobStatInvalid:\n pass\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a generator that yields workflow events.", "response": "def events(config):\n \"\"\" Return a generator that yields workflow events.\n\n For every workflow event that is sent from celery this generator yields an event\n object.\n\n Args:\n config (Config): Reference to the configuration object from which the\n settings are retrieved.\n\n Returns:\n generator: A generator that returns workflow events.\n\n \"\"\"\n celery_app = create_app(config)\n\n for event in event_stream(celery_app, filter_by_prefix='task'):\n try:\n yield create_event_model(event)\n except JobEventTypeUnsupported:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(self):\n read_stdout = partial(self._read_output, stream=self._process.stdout,\n callback=self._callback_stdout,\n output_file=self._stdout_file)\n\n read_stderr = partial(self._read_output, stream=self._process.stderr,\n callback=self._callback_stderr,\n output_file=self._stderr_file)\n\n # capture the process output as long as the process is active\n try:\n while self._process.poll() is None:\n result_stdout = read_stdout()\n result_stderr = read_stderr()\n\n if not result_stdout and not result_stderr:\n sleep(self._refresh_time)\n\n # read remaining lines\n while read_stdout():\n pass\n\n while read_stderr():\n pass\n\n except (StopTask, AbortWorkflow) as exc:\n self._exc_obj = exc", "response": "Runs the process and returns the current state of the application."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _read_output(self, stream, callback, output_file):\n if (callback is None and output_file is None) or stream.closed:\n return False\n\n line = stream.readline()\n if line:\n if callback is not None:\n callback(line.decode(),\n self._data, self._store, self._signal, self._context)\n\n if output_file is not None:\n output_file.write(line)\n\n return True\n else:\n return False", "response": "Read the output of the process and execute the callback and save the output."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef run(self, data, store, signal, context, **kwargs):\n params = self.params.eval(data, store, exclude=['command'])\n\n capture_stdout = self._callback_stdout is not None or params.capture_stdout\n capture_stderr = self._callback_stderr is not None or params.capture_stderr\n\n stdout_file = TemporaryFile() if params.capture_stdout else None\n stderr_file = TemporaryFile() if params.capture_stderr else None\n\n stdout = PIPE if capture_stdout else None\n stderr = PIPE if capture_stderr else None\n\n # change the user or group under which the process should run\n if params.user is not None or params.group is not None:\n pre_exec = self._run_as(params.user, params.group)\n else:\n pre_exec = None\n\n # call the command\n proc = Popen(self.params.eval_single('command', data, store),\n cwd=params.cwd, shell=True, env=params.env,\n preexec_fn=pre_exec, stdout=stdout, stderr=stderr,\n stdin=PIPE if params.stdin is not None else None)\n\n # if input is available, send it to the process\n if params.stdin is not None:\n proc.stdin.write(params.stdin.encode(sys.getfilesystemencoding()))\n\n # send a notification that the process has been started\n try:\n if self._callback_process is not None:\n self._callback_process(proc.pid, data, store, signal, context)\n except (StopTask, AbortWorkflow):\n proc.terminate()\n raise\n\n # send the output handling to a thread\n if capture_stdout or capture_stderr:\n output_reader = BashTaskOutputReader(proc, stdout_file, stderr_file,\n self._callback_stdout,\n self._callback_stderr,\n params.refresh_time,\n data, store, signal, context)\n output_reader.start()\n else:\n output_reader = None\n\n # wait for the process to complete and watch for a stop signal\n while proc.poll() is None or\\\n (output_reader is not None and output_reader.is_alive()):\n sleep(params.refresh_time)\n if signal.is_stopped:\n proc.terminate()\n\n if output_reader is not None:\n output_reader.join()\n data = output_reader.data\n\n # if a stop or abort exception was raised, stop the bash process and re-raise\n if output_reader.exc_obj is not None:\n if proc.poll() is None:\n proc.terminate()\n raise output_reader.exc_obj\n\n # send a notification that the process has completed\n if self._callback_end is not None:\n if stdout_file is not None:\n stdout_file.seek(0)\n if stderr_file is not None:\n stderr_file.seek(0)\n\n self._callback_end(proc.returncode, stdout_file, stderr_file,\n data, store, signal, context)\n\n if stdout_file is not None:\n stdout_file.close()\n\n if stderr_file is not None:\n stderr_file.close()\n\n return Action(data)", "response": "This method is called by the main run method of the Python task. It runs the command and sends the output to the process."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _run_as(user, group):\n def wrapper():\n if user is not None:\n os.setuid(user)\n if group is not None:\n os.setgid(group)\n return wrapper", "response": "Function wrapper that sets the user and group for the process"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert the specified value to the type given by the option.", "response": "def convert(self, value):\n \"\"\" Convert the specified value to the type of the option.\n\n Args:\n value: The value that should be converted.\n\n Returns:\n The value with the type given by the option.\n \"\"\"\n if self._type is str:\n return str(value)\n elif self._type is int:\n try:\n return int(value)\n except (UnicodeError, ValueError):\n raise WorkflowArgumentError('Cannot convert {} to int'.format(value))\n elif self._type is float:\n try:\n return float(value)\n except (UnicodeError, ValueError):\n raise WorkflowArgumentError('Cannot convert {} to float'.format(value))\n elif self._type is bool:\n if isinstance(value, bool):\n return bool(value)\n value = value.lower()\n if value in ('true', '1', 'yes', 'y'):\n return True\n elif value in ('false', '0', 'no', 'n'):\n return False\n raise WorkflowArgumentError('Cannot convert {} to bool'.format(value))\n else:\n return value"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef check_missing(self, args):\n return [opt.name for opt in self\n if (opt.name not in args) and (opt.default is None)]", "response": "Checks that all options that are not specified in the passed arguments are present in the current configuration."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef consolidate(self, args):\n result = dict(args)\n\n for opt in self:\n if opt.name in result:\n result[opt.name] = opt.convert(result[opt.name])\n else:\n if opt.default is not None:\n result[opt.name] = opt.convert(opt.default)\n\n return result", "response": "Consolidate the provided arguments."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef define(self, schema, *, validate=True):\n self._schema = schema\n if validate:\n self.validate(self.make_graph(self._schema))", "response": "Define the task graph definition."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(self, config, workflow_id, signal, *, data=None):\n graph = self.make_graph(self._schema)\n\n # pre-checks\n self.validate(graph)\n\n if config is None:\n raise ConfigNotDefinedError()\n\n # create the celery app for submitting tasks\n celery_app = create_app(config)\n\n # the task queue for managing the current state of the tasks\n tasks = []\n stopped = False\n\n # add all tasks without predecessors to the task list\n for task in nx.topological_sort(graph):\n task.workflow_name = self.workflow_name\n task.dag_name = self.name\n if len(list(graph.predecessors(task))) == 0:\n task.state = TaskState.Waiting\n tasks.append(task)\n\n def set_task_completed(completed_task):\n \"\"\" For each completed task, add all successor tasks to the task list.\n If they are not in the task list yet, flag them as 'waiting'.\n \"\"\"\n completed_task.state = TaskState.Completed\n for successor in graph.successors(completed_task):\n if successor not in tasks:\n successor.state = TaskState.Waiting\n tasks.append(successor)\n\n # process the task queue as long as there are tasks in it\n while tasks:\n if not stopped:\n stopped = signal.is_stopped\n\n # delay the execution by the polling time\n if config.dag_polling_time > 0.0:\n sleep(config.dag_polling_time)\n\n for i in range(len(tasks) - 1, -1, -1):\n task = tasks[i]\n\n # for each waiting task, wait for all predecessor tasks to be\n # completed. Then check whether the task should be skipped by\n # interrogating the predecessor tasks.\n if task.is_waiting:\n if stopped:\n task.state = TaskState.Stopped\n else:\n pre_tasks = list(graph.predecessors(task))\n if all([p.is_completed for p in pre_tasks]):\n\n # check whether the task should be skipped\n run_task = task.has_to_run or len(pre_tasks) == 0\n for pre in pre_tasks:\n if run_task:\n break\n\n # predecessor task is skipped and flag should\n # not be propagated\n if pre.is_skipped and not pre.propagate_skip:\n run_task = True\n\n # limits of a non-skipped predecessor task\n if not pre.is_skipped:\n if pre.celery_result.result.limit is not None:\n if task.name in [\n n.name if isinstance(n, BaseTask) else n\n for n in pre.celery_result.result.limit]:\n run_task = True\n else:\n run_task = True\n\n task.is_skipped = not run_task\n\n # send the task to celery or, if skipped, mark it as completed\n if task.is_skipped:\n set_task_completed(task)\n else:\n # compose the input data from the predecessor tasks\n # output. Data from skipped predecessor tasks do not\n # contribute to the input data\n if len(pre_tasks) == 0:\n input_data = data\n else:\n input_data = MultiTaskData()\n for pt in [p for p in pre_tasks if not p.is_skipped]:\n slot = graph[pt][task]['slot']\n input_data.add_dataset(\n pt.name,\n pt.celery_result.result.data.default_dataset,\n aliases=[slot] if slot is not None else None)\n\n task.state = TaskState.Running\n task.celery_result = celery_app.send_task(\n JobExecPath.Task,\n args=(task, workflow_id, input_data),\n queue=task.queue,\n routing_key=task.queue\n )\n\n # flag task as completed\n elif task.is_running:\n if task.celery_completed:\n set_task_completed(task)\n elif task.celery_failed:\n task.state = TaskState.Aborted\n signal.stop_workflow()\n\n # cleanup task results that are not required anymore\n elif task.is_completed:\n if all([s.is_completed or s.is_stopped or s.is_aborted\n for s in graph.successors(task)]):\n if celery_app.conf.result_expires == 0:\n task.clear_celery_result()\n tasks.remove(task)\n\n # cleanup and remove stopped and aborted tasks\n elif task.is_stopped or task.is_aborted:\n if celery_app.conf.result_expires == 0:\n task.clear_celery_result()\n tasks.remove(task)", "response": "Runs the dag by calling the tasks in the correct order."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate(self, graph):\n if not nx.is_directed_acyclic_graph(graph):\n raise DirectedAcyclicGraphInvalid(graph_name=self._name)", "response": "Validate the graph by checking whether it is a directed acyclic graph."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconstructing a graph object from a given schema.", "response": "def make_graph(schema):\n \"\"\" Construct the task graph (dag) from a given schema.\n\n Parses the graph schema definition and creates the task graph. Tasks are the\n vertices of the graph and the connections defined in the schema become the edges.\n\n A key in the schema dict represents a parent task and the value one or more\n children:\n {parent: [child]} or {parent: [child1, child2]}\n\n The data output of one task can be routed to a labelled input slot of successor\n tasks using a dictionary instead of a list for the children:\n {parent: {child1: 'positive', child2: 'negative'}}\n\n An empty slot name or None skips the creation of a labelled slot:\n {parent: {child1: '', child2: None}}\n\n The underlying graph library creates nodes automatically, when an edge between\n non-existing nodes is created.\n\n Args:\n schema (dict): A dictionary with the schema definition.\n\n Returns:\n DiGraph: A reference to the fully constructed graph object.\n\n Raises:\n DirectedAcyclicGraphUndefined: If the schema is not defined.\n \"\"\"\n if schema is None:\n raise DirectedAcyclicGraphUndefined()\n\n # sanitize the input schema such that it follows the structure:\n # {parent: {child_1: slot_1, child_2: slot_2, ...}, ...}\n sanitized_schema = {}\n for parent, children in schema.items():\n child_dict = {}\n if children is not None:\n if isinstance(children, list):\n if len(children) > 0:\n child_dict = {child: None for child in children}\n else:\n child_dict = {None: None}\n elif isinstance(children, dict):\n for child, slot in children.items():\n child_dict[child] = slot if slot != '' else None\n else:\n child_dict = {children: None}\n else:\n child_dict = {None: None}\n\n sanitized_schema[parent] = child_dict\n\n # build the graph from the sanitized schema\n graph = nx.DiGraph()\n for parent, children in sanitized_schema.items():\n for child, slot in children.items():\n if child is not None:\n graph.add_edge(parent, child, slot=slot)\n else:\n graph.add_node(parent)\n\n return graph"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge(self, dataset):\n def merge_data(source, dest):\n for key, value in source.items():\n if isinstance(value, dict):\n merge_data(value, dest.setdefault(key, {}))\n else:\n dest[key] = value\n return dest\n\n merge_data(dataset.data, self._data)\n\n for h in dataset.task_history:\n if h not in self._task_history:\n self._task_history.append(h)", "response": "Merges the specified dataset on top of the existing data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a new dataset to the MultiTaskData.", "response": "def add_dataset(self, task_name, dataset=None, *, aliases=None):\n \"\"\" Add a new dataset to the MultiTaskData.\n\n Args:\n task_name (str): The name of the task from which the dataset was received.\n dataset (TaskData): The dataset that should be added.\n aliases (list): A list of aliases that should be registered with the dataset.\n \"\"\"\n self._datasets.append(dataset if dataset is not None else TaskData())\n last_index = len(self._datasets) - 1\n self._aliases[task_name] = last_index\n\n if aliases is not None:\n for alias in aliases:\n self._aliases[alias] = last_index\n\n if len(self._datasets) == 1:\n self._default_index = 0"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_alias(self, alias, index):\n if index >= len(self._datasets):\n raise DataInvalidIndex('A dataset with index {} does not exist'.format(index))\n self._aliases[alias] = index", "response": "Adds an alias pointing to the given index."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef flatten(self, in_place=True):\n new_dataset = TaskData()\n\n for i, dataset in enumerate(self._datasets):\n if i != self._default_index:\n new_dataset.merge(dataset)\n\n new_dataset.merge(self.default_dataset)\n\n # point all aliases to the new, single dataset\n new_aliases = {alias: 0 for alias, _ in self._aliases.items()}\n\n # replace existing datasets or return a new MultiTaskData object\n if in_place:\n self._datasets = [new_dataset]\n self._aliases = new_aliases\n self._default_index = 0\n else:\n return MultiTaskData(dataset=new_dataset, aliases=list(new_aliases.keys()))", "response": "Returns a new MultiTaskData object containing all datasets and all aliases."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the default dataset by its alias.", "response": "def set_default_by_alias(self, alias):\n \"\"\" Set the default dataset by its alias.\n\n After changing the default dataset, all calls without explicitly specifying the\n dataset by index or alias will be redirected to this dataset.\n\n Args:\n alias (str): The alias of the dataset that should be made the default.\n\n Raises:\n DataInvalidAlias: If the alias does not represent a valid dataset.\n \"\"\"\n if alias not in self._aliases:\n raise DataInvalidAlias('A dataset with alias {} does not exist'.format(alias))\n\n self._default_index = self._aliases[alias]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the default dataset by its index.", "response": "def set_default_by_index(self, index):\n \"\"\" Set the default dataset by its index.\n\n After changing the default dataset, all calls without explicitly specifying the\n dataset by index or alias will be redirected to this dataset.\n\n Args:\n index (int): The index of the dataset that should be made the default.\n\n Raises:\n DataInvalidIndex: If the index does not represent a valid dataset.\n \"\"\"\n if index >= len(self._datasets):\n raise DataInvalidIndex('A dataset with index {} does not exist'.format(index))\n\n self._default_index = index"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dataset by its alias.", "response": "def get_by_alias(self, alias):\n \"\"\" Return a dataset by its alias.\n\n Args:\n alias (str): The alias of the dataset that should be returned.\n\n Raises:\n DataInvalidAlias: If the alias does not represent a valid dataset.\n \"\"\"\n if alias not in self._aliases:\n raise DataInvalidAlias('A dataset with alias {} does not exist'.format(alias))\n\n return self.get_by_index(self._aliases[alias])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_by_index(self, index):\n if index >= len(self._datasets):\n raise DataInvalidIndex('A dataset with index {} does not exist'.format(index))\n\n return self._datasets[index]", "response": "Return a dataset by its index."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the task context content as a dictionary.", "response": "def to_dict(self):\n \"\"\" Return the task context content as a dictionary. \"\"\"\n return {\n 'task_name': self.task_name,\n 'dag_name': self.dag_name,\n 'workflow_name': self.workflow_name,\n 'workflow_id': self.workflow_id,\n 'worker_hostname': self.worker_hostname\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstarting a worker process.", "response": "def start_worker(queues, config, *, name=None, celery_args=None, check_datastore=True):\n \"\"\" Start a worker process.\n\n Args:\n queues (list): List of queue names this worker accepts jobs from.\n config (Config): Reference to the configuration object from which the\n settings for the worker are retrieved.\n name (string): Unique name for the worker. The hostname template variables from\n Celery can be used. If not given, a unique name is created.\n celery_args (list): List of additional Celery worker command line arguments.\n Please note that this depends on the version of Celery used and might change.\n Use with caution.\n check_datastore (bool): Set to True to check whether the data store is available\n prior to starting the worker.\n \"\"\"\n celery_app = create_app(config)\n\n if check_datastore:\n with DataStore(**config.data_store,\n auto_connect=True, handle_reconnect=False) as ds:\n celery_app.user_options['datastore_info'] = ds.server_info\n\n argv = [\n 'worker',\n '-n={}'.format(uuid4() if name is None else name),\n '--queues={}'.format(','.join(queues))\n ]\n\n argv.extend(celery_args or [])\n\n celery_app.steps['consumer'].add(WorkerLifecycle)\n celery_app.user_options['config'] = config\n celery_app.worker_main(argv)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stop_worker(config, *, worker_ids=None):\n if worker_ids is not None and not isinstance(worker_ids, list):\n worker_ids = [worker_ids]\n\n celery_app = create_app(config)\n celery_app.control.shutdown(destination=worker_ids)", "response": "Stop a worker process."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of all available workers.", "response": "def list_workers(config, *, filter_by_queues=None):\n \"\"\" Return a list of all available workers.\n\n Args:\n config (Config): Reference to the configuration object from which the\n settings are retrieved.\n filter_by_queues (list): Restrict the returned workers to workers that listen to\n at least one of the queue names in this list.\n\n Returns:\n list: A list of WorkerStats objects.\n \"\"\"\n celery_app = create_app(config)\n worker_stats = celery_app.control.inspect().stats()\n queue_stats = celery_app.control.inspect().active_queues()\n\n if worker_stats is None:\n return []\n\n workers = []\n for name, w_stat in worker_stats.items():\n queues = [QueueStats.from_celery(q_stat) for q_stat in queue_stats[name]]\n\n add_worker = filter_by_queues is None\n if not add_worker:\n for queue in queues:\n if queue.name in filter_by_queues:\n add_worker = True\n break\n\n if add_worker:\n workers.append(WorkerStats.from_celery(name, w_stat, queues))\n\n return workers"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef eval(self, data, data_store, *, exclude=None):\n exclude = [] if exclude is None else exclude\n\n result = {}\n for key, value in self.items():\n if key in exclude:\n continue\n\n if value is not None and callable(value):\n result[key] = value(data, data_store)\n else:\n result[key] = value\n return TaskParameters(result)", "response": "Evaluate the callable parameters of the current object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef eval_single(self, key, data, data_store):\n if key in self:\n value = self[key]\n if value is not None and callable(value):\n return value(data, data_store)\n else:\n return value\n else:\n raise AttributeError()", "response": "Evaluate the value of a single parameter taking into account callables."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_lotw_users(**kwargs):\n\n url = \"\"\n\n lotw = {}\n\n try:\n url = kwargs['url']\n except KeyError:\n # url = \"http://wd5eae.org/LoTW_Data.txt\"\n url = \"https://lotw.arrl.org/lotw-user-activity.csv\"\n\n try:\n result = requests.get(url)\n except (ConnectionError, HTTPError, Timeout) as e:\n raise IOError(e)\n\n error_count = 0\n\n if result.status_code == requests.codes.ok:\n for el in result.text.split():\n data = el.split(\",\")\n try:\n lotw[data[0]] = datetime.strptime(data[1], '%Y-%m-%d')\n except ValueError as e:\n error_count += 1\n if error_count > 10:\n raise ValueError(\"more than 10 wrongly formatted datasets \" + str(e))\n\n else:\n raise IOError(\"HTTP Error: \" + str(result.status_code))\n\n return lotw", "response": "Download the latest offical list of the LOTW User Activity of the World and return a dictionary containing the callsign date of the last LOTW upload"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndownloads the latest offical list of Clublog __ users.", "response": "def get_clublog_users(**kwargs):\n \"\"\"Download the latest offical list of `Clublog`__ users.\n\n Args:\n url (str, optional): Download URL\n\n Returns:\n dict: Dictionary containing (if data available) the fields:\n firstqso, lastqso, last-lotw, lastupload (datetime),\n locator (string) and oqrs (boolean)\n\n Raises:\n IOError: When network is unavailable, file can't be downloaded or processed\n\n Example:\n The following example downloads the Clublog user list and returns a dictionary with the data of HC2/AL1O:\n\n >>> from pyhamtools.qsl import get_clublog_users\n >>> clublog = get_lotw_users()\n >>> clublog['HC2/AL1O']\n {'firstqso': datetime.datetime(2012, 1, 1, 19, 59, 27),\n 'last-lotw': datetime.datetime(2013, 5, 9, 1, 56, 23),\n 'lastqso': datetime.datetime(2013, 5, 5, 6, 39, 3),\n 'lastupload': datetime.datetime(2013, 5, 8, 15, 0, 6),\n 'oqrs': True}\n\n .. _CLUBLOG: https://secure.clublog.org\n __ CLUBLOG_\n\n \"\"\"\n\n url = \"\"\n\n clublog = {}\n\n try:\n url = kwargs['url']\n except KeyError:\n url = \"https://secure.clublog.org/clublog-users.json.zip\"\n\n try:\n result = requests.get(url)\n except (ConnectionError, HTTPError, Timeout) as e:\n raise IOError(e)\n\n\n if result.status_code != requests.codes.ok:\n raise IOError(\"HTTP Error: \" + str(result.status_code))\n\n zip_file = zipfile.ZipFile(BytesIO(result.content))\n files = zip_file.namelist()\n cl_json_unzipped = zip_file.read(files[0]).decode('utf8').replace(\"'\", '\"')\n\n cl_data = json.loads(cl_json_unzipped, encoding='UTF-8')\n\n error_count = 0\n\n for call, call_data in iteritems(cl_data):\n try:\n data = {}\n if \"firstqso\" in call_data:\n if call_data[\"firstqso\"] != None:\n data[\"firstqso\"] = datetime.strptime(call_data[\"firstqso\"], '%Y-%m-%d %H:%M:%S')\n if \"lastqso\" in call_data:\n if call_data[\"lastqso\"] != None:\n data[\"lastqso\"] = datetime.strptime(call_data[\"lastqso\"], '%Y-%m-%d %H:%M:%S')\n if \"last-lotw\" in call_data:\n if call_data[\"last-lotw\"] != None:\n data[\"last-lotw\"] = datetime.strptime(call_data[\"last-lotw\"], '%Y-%m-%d %H:%M:%S')\n if \"lastupload\" in call_data:\n if call_data[\"lastupload\"] != None:\n data[\"lastupload\"] = datetime.strptime(call_data[\"lastupload\"], '%Y-%m-%d %H:%M:%S')\n if \"locator\" in call_data:\n if call_data[\"locator\"] != None:\n data[\"locator\"] = call_data[\"locator\"]\n if \"oqrs\" in call_data:\n if call_data[\"oqrs\"] != None:\n data[\"oqrs\"] = call_data[\"oqrs\"]\n clublog[call] = data\n except TypeError: #some date fields contain null instead of a valid datetime string - we ignore them\n print(\"Ignoring invalid type in data:\", call, call_data)\n pass\n except ValueError: #some date fiels are invalid. we ignore them for the moment\n print(\"Ignoring invalid data:\", call, call_data)\n pass\n\n return clublog"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndownloads the latest official list of EQSL. cc __ users. The list of users can be found here_.", "response": "def get_eqsl_users(**kwargs):\n \"\"\"Download the latest official list of `EQSL.cc`__ users. The list of users can be found here_.\n\n Args:\n url (str, optional): Download URL\n\n Returns:\n list: List containing the callsigns of EQSL users (unicode)\n\n Raises:\n IOError: When network is unavailable, file can't be downloaded or processed\n\n Example:\n The following example downloads the EQSL user list and checks if DH1TW is a user:\n\n >>> from pyhamtools.qsl import get_eqsl_users\n >>> mylist = get_eqsl_users()\n >>> try:\n >>> mylist.index('DH1TW')\n >>> except ValueError as e:\n >>> print e\n 'DH1TW' is not in list\n\n .. _here: http://www.eqsl.cc/QSLCard/DownloadedFiles/AGMemberlist.txt\n\n \"\"\"\n\n url = \"\"\n\n eqsl = []\n\n try:\n url = kwargs['url']\n except KeyError:\n url = \"http://www.eqsl.cc/QSLCard/DownloadedFiles/AGMemberlist.txt\"\n\n try:\n result = requests.get(url)\n except (ConnectionError, HTTPError, Timeout) as e:\n raise IOError(e)\n\n if result.status_code == requests.codes.ok:\n eqsl = re.sub(\"^List.+UTC\", \"\", result.text)\n eqsl = eqsl.upper().split()\n else:\n raise IOError(\"HTTP Error: \" + str(result.status_code))\n\n return eqsl"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef copy_data_in_redis(self, redis_prefix, redis_instance):\n\n if redis_instance is not None:\n self._redis = redis_instance\n\n if self._redis is None:\n raise AttributeError(\"redis_instance is missing\")\n\n if redis_prefix is None:\n raise KeyError(\"redis_prefix is missing\")\n\n if self._lookuptype == \"clublogxml\" or self._lookuptype == \"countryfile\":\n\n self._push_dict_to_redis(self._entities, redis_prefix, \"_entity_\")\n\n self._push_dict_index_to_redis(self._callsign_exceptions_index, redis_prefix, \"_call_ex_index_\")\n self._push_dict_to_redis(self._callsign_exceptions, redis_prefix, \"_call_ex_\")\n\n self._push_dict_index_to_redis(self._prefixes_index, redis_prefix, \"_prefix_index_\")\n self._push_dict_to_redis(self._prefixes, redis_prefix, \"_prefix_\")\n\n self._push_dict_index_to_redis(self._invalid_operations_index, redis_prefix, \"_inv_op_index_\")\n self._push_dict_to_redis(self._invalid_operations, redis_prefix, \"_inv_op_\")\n\n self._push_dict_index_to_redis(self._zone_exceptions_index, redis_prefix, \"_zone_ex_index_\")\n self._push_dict_to_redis(self._zone_exceptions, redis_prefix, \"_zone_ex_\")\n\n return True", "response": "Copy the complete lookup data into Redis."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lookup_entity(self, entity=None):\n if self._lookuptype == \"clublogxml\":\n entity = int(entity)\n if entity in self._entities:\n return self._strip_metadata(self._entities[entity])\n else:\n raise KeyError\n\n elif self._lookuptype == \"redis\":\n if self._redis_prefix is None:\n raise KeyError (\"redis_prefix is missing\")\n #entity = str(entity)\n json_data = self._redis.get(self._redis_prefix + \"_entity_\" + str(entity))\n if json_data is not None:\n my_dict = self._deserialize_data(json_data)\n return self._strip_metadata(my_dict)\n\n elif self._lookuptype == \"qrz\":\n result = self._lookup_qrz_dxcc(entity, self._apikey)\n return result\n\n # no matching case\n raise KeyError", "response": "Returns the lookup data of an ADIF Entity with the specified ID."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving not needed metadata from a dictionary.", "response": "def _strip_metadata(self, my_dict):\n \"\"\"\n Create a copy of dict and remove not needed data\n \"\"\"\n new_dict = copy.deepcopy(my_dict)\n if const.START in new_dict:\n del new_dict[const.START]\n if const.END in new_dict:\n del new_dict[const.END]\n if const.WHITELIST in new_dict:\n del new_dict[const.WHITELIST]\n if const.WHITELIST_START in new_dict:\n del new_dict[const.WHITELIST_START]\n if const.WHITELIST_END in new_dict:\n del new_dict[const.WHITELIST_END]\n return new_dict"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the data for a specific callsign.", "response": "def lookup_callsign(self, callsign=None, timestamp=timestamp_now):\n \"\"\"\n Returns lookup data if an exception exists for a callsign\n\n Args:\n callsign (string): Amateur radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n dict: Dictionary containing the country specific data of the callsign\n\n Raises:\n KeyError: No matching callsign found\n APIKeyMissingError: API Key for Clublog missing or incorrect\n\n Example:\n The following code queries the the online Clublog API for the callsign \"VK9XO\" on a specific date.\n\n >>> from pyhamtools import LookupLib\n >>> from datetime import datetime\n >>> import pytz\n >>> my_lookuplib = LookupLib(lookuptype=\"clublogapi\", apikey=\"myapikey\")\n >>> timestamp = datetime(year=1962, month=7, day=7, tzinfo=pytz.UTC)\n >>> print my_lookuplib.lookup_callsign(\"VK9XO\", timestamp)\n {\n 'country': u'CHRISTMAS ISLAND',\n 'longitude': 105.7,\n 'cqz': 29,\n 'adif': 35,\n 'latitude': -10.5,\n 'continent': u'OC'\n }\n\n Note:\n This method is available for\n\n - clublogxml\n - clublogapi\n - countryfile\n - qrz.com\n - redis\n\n\n \"\"\"\n callsign = callsign.strip().upper()\n\n if self._lookuptype == \"clublogapi\":\n callsign_data = self._lookup_clublogAPI(callsign=callsign, timestamp=timestamp, apikey=self._apikey)\n if callsign_data[const.ADIF]==1000:\n raise KeyError\n else:\n return callsign_data\n\n elif self._lookuptype == \"clublogxml\" or self._lookuptype == \"countryfile\":\n\n return self._check_data_for_date(callsign, timestamp, self._callsign_exceptions, self._callsign_exceptions_index)\n\n elif self._lookuptype == \"redis\":\n\n data_dict, index = self._get_dicts_from_redis(\"_call_ex_\", \"_call_ex_index_\", self._redis_prefix, callsign)\n return self._check_data_for_date(callsign, timestamp, data_dict, index)\n\n # no matching case\n elif self._lookuptype == \"qrz\":\n return self._lookup_qrz_callsign(callsign, self._apikey, self._apiv)\n\n raise KeyError(\"unknown Callsign\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_dicts_from_redis(self, name, index_name, redis_prefix, item):\n r = self._redis\n data_dict = {}\n data_index_dict = {}\n\n if redis_prefix is None:\n raise KeyError (\"redis_prefix is missing\")\n\n if r.scard(redis_prefix + index_name + str(item)) > 0:\n data_index_dict[str(item)] = r.smembers(redis_prefix + index_name + str(item))\n\n for i in data_index_dict[item]:\n json_data = r.get(redis_prefix + name + str(int(i)))\n data_dict[i] = self._deserialize_data(json_data)\n\n return (data_dict, data_index_dict)\n\n raise KeyError (\"No Data found in Redis for \"+ item)", "response": "Retrieve the data of an item from redis and put it in an index and data dictionary to match the common query interface."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _check_data_for_date(self, item, timestamp, data_dict, data_index_dict):\n\n if item in data_index_dict:\n for item in data_index_dict[item]:\n\n # startdate < timestamp\n if const.START in data_dict[item] and not const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp:\n item_data = copy.deepcopy(data_dict[item])\n del item_data[const.START]\n return item_data\n\n # enddate > timestamp\n elif not const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.END] > timestamp:\n item_data = copy.deepcopy(data_dict[item])\n del item_data[const.END]\n return item_data\n\n # startdate > timestamp > enddate\n elif const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp \\\n and data_dict[item][const.END] > timestamp:\n item_data = copy.deepcopy(data_dict[item])\n del item_data[const.START]\n del item_data[const.END]\n return item_data\n\n # no startdate or enddate available\n elif not const.START in data_dict[item] and not const.END in data_dict[item]:\n return data_dict[item]\n\n raise KeyError", "response": "Checks if the data for the given item is found in the index and returns the data that is stored in the index."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if the callsign is invalid for a given timestamp.", "response": "def _check_inv_operation_for_date(self, item, timestamp, data_dict, data_index_dict):\n \"\"\"\n Checks if the callsign is marked as an invalid operation for a given timestamp.\n In case the operation is invalid, True is returned. Otherwise a KeyError is raised.\n \"\"\"\n\n if item in data_index_dict:\n for item in data_index_dict[item]:\n\n # startdate < timestamp\n if const.START in data_dict[item] and not const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp:\n return True\n\n # enddate > timestamp\n elif not const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.END] > timestamp:\n return True\n\n # startdate > timestamp > enddate\n elif const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp \\\n and data_dict[item][const.END] > timestamp:\n return True\n\n # no startdate or enddate available\n elif not const.START in data_dict[item] and not const.END in data_dict[item]:\n return True\n\n raise KeyError"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef lookup_prefix(self, prefix, timestamp=timestamp_now):\n\n prefix = prefix.strip().upper()\n\n if self._lookuptype == \"clublogxml\" or self._lookuptype == \"countryfile\":\n\n return self._check_data_for_date(prefix, timestamp, self._prefixes, self._prefixes_index)\n\n elif self._lookuptype == \"redis\":\n\n data_dict, index = self._get_dicts_from_redis(\"_prefix_\", \"_prefix_index_\", self._redis_prefix, prefix)\n return self._check_data_for_date(prefix, timestamp, data_dict, index)\n\n # no matching case\n raise KeyError", "response": "Returns the lookup data for a specific prefix"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if an operation is known as invalid for a given callsign.", "response": "def is_invalid_operation(self, callsign, timestamp=datetime.utcnow().replace(tzinfo=UTC)):\n \"\"\"\n Returns True if an operations is known as invalid\n\n Args:\n callsign (string): Amateur Radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n bool: True if a record exists for this callsign (at the given time)\n\n Raises:\n KeyError: No matching callsign found\n APIKeyMissingError: API Key for Clublog missing or incorrect\n\n Example:\n The following code checks the Clublog XML database if the operation is valid for two dates.\n\n >>> from pyhamtools import LookupLib\n >>> from datetime import datetime\n >>> import pytz\n >>> my_lookuplib = LookupLib(lookuptype=\"clublogxml\", apikey=\"myapikey\")\n >>> print my_lookuplib.is_invalid_operation(\"5W1CFN\")\n True\n >>> try:\n >>> timestamp = datetime(year=2012, month=1, day=31).replace(tzinfo=pytz.UTC)\n >>> my_lookuplib.is_invalid_operation(\"5W1CFN\", timestamp)\n >>> except KeyError:\n >>> print \"Seems to be invalid operation before 31.1.2012\"\n Seems to be an invalid operation before 31.1.2012\n\n Note:\n This method is available for\n\n - clublogxml\n - redis\n\n \"\"\"\n\n callsign = callsign.strip().upper()\n\n if self._lookuptype == \"clublogxml\":\n\n return self._check_inv_operation_for_date(callsign, timestamp, self._invalid_operations, self._invalid_operations_index)\n\n elif self._lookuptype == \"redis\":\n\n data_dict, index = self._get_dicts_from_redis(\"_inv_op_\", \"_inv_op_index_\", self._redis_prefix, callsign)\n return self._check_inv_operation_for_date(callsign, timestamp, data_dict, index)\n\n #no matching case\n raise KeyError"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _check_zone_exception_for_date(self, item, timestamp, data_dict, data_index_dict):\n if item in data_index_dict:\n for item in data_index_dict[item]:\n\n # startdate < timestamp\n if const.START in data_dict[item] and not const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp:\n return data_dict[item][const.CQZ]\n\n # enddate > timestamp\n elif not const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.END] > timestamp:\n return data_dict[item][const.CQZ]\n\n # startdate > timestamp > enddate\n elif const.START in data_dict[item] and const.END in data_dict[item]:\n if data_dict[item][const.START] < timestamp \\\n and data_dict[item][const.END] > timestamp:\n return data_dict[item][const.CQZ]\n\n # no startdate or enddate available\n elif not const.START in data_dict[item] and not const.END in data_dict[item]:\n return data_dict[item][const.CQZ]\n\n raise KeyError", "response": "Checks the index and data for a cq - zone exception for the specified date."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a CQ Zone exception for the given callsign at the given time.", "response": "def lookup_zone_exception(self, callsign, timestamp=datetime.utcnow().replace(tzinfo=UTC)):\n \"\"\"\n Returns a CQ Zone if an exception exists for the given callsign\n\n Args:\n callsign (string): Amateur radio callsign\n timestamp (datetime, optional): datetime in UTC (tzinfo=pytz.UTC)\n\n Returns:\n int: Value of the the CQ Zone exception which exists for this callsign (at the given time)\n\n Raises:\n KeyError: No matching callsign found\n APIKeyMissingError: API Key for Clublog missing or incorrect\n\n Example:\n The following code checks the Clublog XML database if a CQ Zone exception exists for the callsign DP0GVN.\n\n >>> from pyhamtools import LookupLib\n >>> my_lookuplib = LookupLib(lookuptype=\"clublogxml\", apikey=\"myapikey\")\n >>> print my_lookuplib.lookup_zone_exception(\"DP0GVN\")\n 38\n\n The prefix \"DP\" It is assigned to Germany, but the station is located in Antarctica, and therefore\n in CQ Zone 38\n\n Note:\n This method is available for\n\n - clublogxml\n - redis\n\n \"\"\"\n\n callsign = callsign.strip().upper()\n\n if self._lookuptype == \"clublogxml\":\n\n return self._check_zone_exception_for_date(callsign, timestamp, self._zone_exceptions, self._zone_exceptions_index)\n\n elif self._lookuptype == \"redis\":\n\n data_dict, index = self._get_dicts_from_redis(\"_zone_ex_\", \"_zone_ex_index_\", self._redis_prefix, callsign)\n return self._check_zone_exception_for_date(callsign, timestamp, data_dict, index)\n\n #no matching case\n raise KeyError"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the lookup object for Clublog Online API", "response": "def _lookup_clublogAPI(self, callsign=None, timestamp=timestamp_now, url=\"https://secure.clublog.org/dxcc\", apikey=None):\n \"\"\" Set up the Lookup object for Clublog Online API\n \"\"\"\n\n params = {\"year\" : timestamp.strftime(\"%Y\"),\n \"month\" : timestamp.strftime(\"%m\"),\n \"day\" : timestamp.strftime(\"%d\"),\n \"hour\" : timestamp.strftime(\"%H\"),\n \"minute\" : timestamp.strftime(\"%M\"),\n \"api\" : apikey,\n \"full\" : \"1\",\n \"call\" : callsign\n }\n\n if sys.version_info.major == 3:\n encodeurl = url + \"?\" + urllib.parse.urlencode(params)\n else:\n encodeurl = url + \"?\" + urllib.urlencode(params)\n response = requests.get(encodeurl, timeout=5)\n\n if not self._check_html_response(response):\n raise LookupError\n\n jsonLookup = response.json()\n lookup = {}\n\n for item in jsonLookup:\n if item == \"Name\": lookup[const.COUNTRY] = jsonLookup[\"Name\"]\n elif item == \"DXCC\": lookup[const.ADIF] = int(jsonLookup[\"DXCC\"])\n elif item == \"Lon\": lookup[const.LONGITUDE] = float(jsonLookup[\"Lon\"])*(-1)\n elif item == \"Lat\": lookup[const.LATITUDE] = float(jsonLookup[\"Lat\"])\n elif item == \"CQZ\": lookup[const.CQZ] = int(jsonLookup[\"CQZ\"])\n elif item == \"Continent\": lookup[const.CONTINENT] = jsonLookup[\"Continent\"]\n\n if lookup[const.ADIF] == 0:\n raise KeyError\n else:\n return lookup"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _lookup_qrz_callsign(self, callsign=None, apikey=None, apiv=\"1.3.3\"):\n\n if apikey is None:\n raise AttributeError(\"Session Key Missing\")\n\n callsign = callsign.upper()\n\n response = self._request_callsign_info_from_qrz(callsign, apikey, apiv)\n\n root = BeautifulSoup(response.text, \"html.parser\")\n lookup = {}\n\n if root.error:\n\n if re.search('Not found', root.error.text, re.I): #No data available for callsign\n raise KeyError(root.error.text)\n\n #try to get a new session key and try to request again\n elif re.search('Session Timeout', root.error.text, re.I) or re.search('Invalid session key', root.error.text, re.I):\n apikey = self._get_qrz_session_key(self._username, self._pwd)\n response = self._request_callsign_info_from_qrz(callsign, apikey, apiv)\n root = BeautifulSoup(response.text, \"html.parser\")\n\n #if this fails again, raise error\n if root.error:\n\n if re.search('Not found', root.error.text, re.I): #No data available for callsign\n raise KeyError(root.error.text)\n else:\n raise AttributeError(root.error.text) #most likely session key invalid\n else:\n #update API Key ob Lookup object\n self._apikey = apikey\n\n else:\n raise AttributeError(root.error.text) #most likely session key missing\n\n if root.callsign is None:\n raise ValueError\n\n if root.callsign.call:\n lookup[const.CALLSIGN] = root.callsign.call.text\n if root.callsign.xref:\n lookup[const.XREF] = root.callsign.xref.text\n if root.callsign.aliases:\n lookup[const.ALIASES] = root.callsign.aliases.text.split(',')\n if root.callsign.dxcc:\n lookup[const.ADIF] = int(root.callsign.dxcc.text)\n if root.callsign.fname:\n lookup[const.FNAME] = root.callsign.fname.text\n if root.callsign.find(\"name\"):\n lookup[const.NAME] = root.callsign.find('name').get_text()\n if root.callsign.addr1:\n lookup[const.ADDR1] = root.callsign.addr1.text\n if root.callsign.addr2:\n lookup[const.ADDR2] = root.callsign.addr2.text\n if root.callsign.state:\n lookup[const.STATE] = root.callsign.state.text\n if root.callsign.zip:\n lookup[const.ZIPCODE] = root.callsign.zip.text\n if root.callsign.country:\n lookup[const.COUNTRY] = root.callsign.country.text\n if root.callsign.ccode:\n lookup[const.CCODE] = int(root.callsign.ccode.text)\n if root.callsign.lat:\n lookup[const.LATITUDE] = float(root.callsign.lat.text)\n if root.callsign.lon:\n lookup[const.LONGITUDE] = float(root.callsign.lon.text)\n if root.callsign.grid:\n lookup[const.LOCATOR] = root.callsign.grid.text\n if root.callsign.county:\n lookup[const.COUNTY] = root.callsign.county.text\n if root.callsign.fips:\n lookup[const.FIPS] = int(root.callsign.fips.text) # check type\n if root.callsign.land:\n lookup[const.LAND] = root.callsign.land.text\n if root.callsign.efdate:\n try:\n lookup[const.EFDATE] = datetime.strptime(root.callsign.efdate.text, '%Y-%m-%d').replace(tzinfo=UTC)\n except ValueError:\n self._logger.debug(\"[QRZ.com] efdate: Invalid DateTime; \" + callsign + \" \" + root.callsign.efdate.text)\n if root.callsign.expdate:\n try:\n lookup[const.EXPDATE] = datetime.strptime(root.callsign.expdate.text, '%Y-%m-%d').replace(tzinfo=UTC)\n except ValueError:\n self._logger.debug(\"[QRZ.com] expdate: Invalid DateTime; \" + callsign + \" \" + root.callsign.expdate.text)\n if root.callsign.p_call:\n lookup[const.P_CALL] = root.callsign.p_call.text\n if root.callsign.find('class'):\n lookup[const.LICENSE_CLASS] = root.callsign.find('class').get_text()\n if root.callsign.codes:\n lookup[const.CODES] = root.callsign.codes.text\n if root.callsign.qslmgr:\n lookup[const.QSLMGR] = root.callsign.qslmgr.text\n if root.callsign.email:\n lookup[const.EMAIL] = root.callsign.email.text\n if root.callsign.url:\n lookup[const.URL] = root.callsign.url.text\n if root.callsign.u_views:\n lookup[const.U_VIEWS] = int(root.callsign.u_views.text)\n if root.callsign.bio:\n lookup[const.BIO] = root.callsign.bio.text\n if root.callsign.biodate:\n try:\n lookup[const.BIODATE] = datetime.strptime(root.callsign.biodate.text, '%Y-%m-%d %H:%M:%S').replace(tzinfo=UTC)\n except ValueError:\n self._logger.warning(\"[QRZ.com] biodate: Invalid DateTime; \" + callsign)\n if root.callsign.image:\n lookup[const.IMAGE] = root.callsign.image.text\n if root.callsign.imageinfo:\n lookup[const.IMAGE_INFO] = root.callsign.imageinfo.text\n if root.callsign.serial:\n lookup[const.SERIAL] = long(root.callsign.serial.text)\n if root.callsign.moddate:\n try:\n lookup[const.MODDATE] = datetime.strptime(root.callsign.moddate.text, '%Y-%m-%d %H:%M:%S').replace(tzinfo=UTC)\n except ValueError:\n self._logger.warning(\"[QRZ.com] moddate: Invalid DateTime; \" + callsign)\n if root.callsign.MSA:\n lookup[const.MSA] = int(root.callsign.MSA.text)\n if root.callsign.AreaCode:\n lookup[const.AREACODE] = int(root.callsign.AreaCode.text)\n if root.callsign.TimeZone:\n lookup[const.TIMEZONE] = int(root.callsign.TimeZone.text)\n if root.callsign.GMTOffset:\n lookup[const.GMTOFFSET] = float(root.callsign.GMTOffset.text)\n if root.callsign.DST:\n if root.callsign.DST.text == \"Y\":\n lookup[const.DST] = True\n else:\n lookup[const.DST] = False\n if root.callsign.eqsl:\n if root.callsign.eqsl.text == \"1\":\n lookup[const.EQSL] = True\n else:\n lookup[const.EQSL] = False\n if root.callsign.mqsl:\n if root.callsign.mqsl.text == \"1\":\n lookup[const.MQSL] = True\n else:\n lookup[const.MQSL] = False\n if root.callsign.cqzone:\n lookup[const.CQZ] = int(root.callsign.cqzone.text)\n if root.callsign.ituzone:\n lookup[const.ITUZ] = int(root.callsign.ituzone.text)\n if root.callsign.born:\n lookup[const.BORN] = int(root.callsign.born.text)\n if root.callsign.user:\n lookup[const.USER_MGR] = root.callsign.user.text\n if root.callsign.lotw:\n if root.callsign.lotw.text == \"1\":\n lookup[const.LOTW] = True\n else:\n lookup[const.LOTW] = False\n if root.callsign.iota:\n lookup[const.IOTA] = root.callsign.iota.text\n if root.callsign.geoloc:\n lookup[const.GEOLOC] = root.callsign.geoloc.text\n\n # if sys.version_info >= (2,):\n # for item in lookup:\n # if isinstance(lookup[item], unicode):\n # print item, repr(lookup[item])\n return lookup", "response": "Perform a QRZ callsign lookup against the QRZ. com XML API."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading and process the ClublogXML file.", "response": "def _load_clublogXML(self,\n url=\"https://secure.clublog.org/cty.php\",\n apikey=None,\n cty_file=None):\n \"\"\" Load and process the ClublogXML file either as a download or from file\n \"\"\"\n\n if self._download:\n cty_file = self._download_file(\n url = url,\n apikey = apikey)\n else:\n cty_file = self._lib_filename\n\n header = self._extract_clublog_header(cty_file)\n cty_file = self._remove_clublog_xml_header(cty_file)\n cty_dict = self._parse_clublog_xml(cty_file)\n\n self._entities = cty_dict[\"entities\"]\n self._callsign_exceptions = cty_dict[\"call_exceptions\"]\n self._prefixes = cty_dict[\"prefixes\"]\n self._invalid_operations = cty_dict[\"invalid_operations\"]\n self._zone_exceptions = cty_dict[\"zone_exceptions\"]\n\n self._callsign_exceptions_index = cty_dict[\"call_exceptions_index\"]\n self._prefixes_index = cty_dict[\"prefixes_index\"]\n self._invalid_operations_index = cty_dict[\"invalid_operations_index\"]\n self._zone_exceptions_index = cty_dict[\"zone_exceptions_index\"]\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _load_countryfile(self,\n url=\"https://www.country-files.com/cty/cty.plist\",\n country_mapping_filename=\"countryfilemapping.json\",\n cty_file=None):\n \"\"\" Load and process the ClublogXML file either as a download or from file\n \"\"\"\n\n cwdFile = os.path.abspath(os.path.join(os.getcwd(), country_mapping_filename))\n pkgFile = os.path.abspath(os.path.join(os.path.dirname(__file__), country_mapping_filename))\n\n # from cwd\n if os.path.exists(cwdFile):\n # country mapping files contains the ADIF identifiers of a particular\n # country since the country-files do not provide this information (only DXCC id)\n country_mapping_filename = cwdFile\n # from package\n elif os.path.exists(pkgFile):\n country_mapping_filename = pkgFile\n else:\n country_mapping_filename = None\n\n if self._download:\n cty_file = self._download_file(url=url)\n else:\n cty_file = os.path.abspath(cty_file)\n\n cty_dict = self._parse_country_file(cty_file, country_mapping_filename)\n self._callsign_exceptions = cty_dict[\"exceptions\"]\n self._prefixes = cty_dict[\"prefixes\"]\n self._callsign_exceptions_index = cty_dict[\"exceptions_index\"]\n self._prefixes_index = cty_dict[\"prefixes_index\"]\n\n return True", "response": "Loads and processes the ClublogXML file for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndownload lookup files either from Clublog or Country - files. com and return the filename of the downloaded file.", "response": "def _download_file(self, url, apikey=None):\n \"\"\" Download lookup files either from Clublog or Country-files.com\n \"\"\"\n import gzip\n import tempfile\n\n cty = {}\n cty_date = \"\"\n cty_file_path = None\n\n filename = None\n\n # download file\n if apikey: # clublog\n response = requests.get(url+\"?api=\"+apikey, timeout=10)\n else: # country-files.com\n response = requests.get(url, timeout=10)\n\n if not self._check_html_response(response):\n raise LookupError\n\n #Clublog Webserver Header\n if \"Content-Disposition\" in response.headers:\n f = re.search('filename=\".+\"', response.headers[\"Content-Disposition\"])\n if f:\n f = f.group(0)\n filename = re.search('\".+\"', f).group(0).replace('\"', '')\n\n #Country-files.org webserver header\n else:\n f = re.search('/.{4}plist$', url)\n if f:\n f = f.group(0)\n filename = f[1:]\n\n if not filename:\n filename = \"cty_\" + self._generate_random_word(5)\n\n download_file_path = os.path.join(tempfile.gettempdir(), filename)\n with open(download_file_path, \"wb\") as download_file:\n download_file.write(response.content)\n self._logger.debug(str(download_file_path) + \" successfully downloaded\")\n\n # unzip file, if gz\n if os.path.splitext(download_file_path)[1][1:] == \"gz\":\n\n download_file = gzip.open(download_file_path, \"r\")\n try:\n cty_file_path = os.path.join(os.path.splitext(download_file_path)[0])\n with open(cty_file_path, \"wb\") as cty_file:\n cty_file.write(download_file.read())\n self._logger.debug(str(cty_file_path) + \" successfully extracted\")\n finally:\n download_file.close()\n else:\n cty_file_path = download_file_path\n\n return cty_file_path"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _extract_clublog_header(self, cty_xml_filename):\n\n cty_header = {}\n\n try:\n with open(cty_xml_filename, \"r\") as cty:\n raw_header = cty.readline()\n\n cty_date = re.search(\"date='.+'\", raw_header)\n if cty_date:\n cty_date = cty_date.group(0).replace(\"date=\", \"\").replace(\"'\", \"\")\n cty_date = datetime.strptime(cty_date[:19], '%Y-%m-%dT%H:%M:%S')\n cty_date.replace(tzinfo=UTC)\n cty_header[\"Date\"] = cty_date\n\n cty_ns = re.search(\"xmlns='.+[']\", raw_header)\n if cty_ns:\n cty_ns = cty_ns.group(0).replace(\"xmlns=\", \"\").replace(\"'\", \"\")\n cty_header['NameSpace'] = cty_ns\n\n if len(cty_header) == 2:\n self._logger.debug(\"Header successfully retrieved from CTY File\")\n elif len(cty_header) < 2:\n self._logger.warning(\"Header could only be partically retrieved from CTY File\")\n self._logger.warning(\"Content of Header: \")\n for key in cty_header:\n self._logger.warning(str(key)+\": \"+str(cty_header[key]))\n return cty_header\n\n except Exception as e:\n self._logger.error(\"Clublog CTY File could not be opened / modified\")\n self._logger.error(\"Error Message: \" + str(e))\n return", "response": "Extract the header of the Clublog XML File"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves the header of the Clublog XML File to make it properly parseable for the python ElementTree XML parser", "response": "def _remove_clublog_xml_header(self, cty_xml_filename):\n \"\"\"\n remove the header of the Clublog XML File to make it\n properly parseable for the python ElementTree XML parser\n \"\"\"\n import tempfile\n\n try:\n with open(cty_xml_filename, \"r\") as f:\n content = f.readlines()\n\n cty_dir = tempfile.gettempdir()\n cty_name = os.path.split(cty_xml_filename)[1]\n cty_xml_filename_no_header = os.path.join(cty_dir, \"NoHeader_\"+cty_name)\n\n with open(cty_xml_filename_no_header, \"w\") as f:\n f.writelines(\"\\n\\r\")\n f.writelines(content[1:])\n\n self._logger.debug(\"Header successfully modified for XML Parsing\")\n return cty_xml_filename_no_header\n\n except Exception as e:\n self._logger.error(\"Clublog CTY could not be opened / modified\")\n self._logger.error(\"Error Message: \" + str(e))\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_clublog_xml(self, cty_xml_filename):\n\n entities = {}\n call_exceptions = {}\n prefixes = {}\n invalid_operations = {}\n zone_exceptions = {}\n\n call_exceptions_index = {}\n prefixes_index = {}\n invalid_operations_index = {}\n zone_exceptions_index = {}\n\n cty_tree = ET.parse(cty_xml_filename)\n root = cty_tree.getroot()\n\n #retrieve ADIF Country Entities\n cty_entities = cty_tree.find(\"entities\")\n self._logger.debug(\"total entities: \" + str(len(cty_entities)))\n if len(cty_entities) > 1:\n for cty_entity in cty_entities:\n try:\n entity = {}\n for item in cty_entity:\n if item.tag == \"name\":\n entity[const.COUNTRY] = unicode(item.text)\n self._logger.debug(unicode(item.text))\n elif item.tag == \"prefix\":\n entity[const.PREFIX] = unicode(item.text)\n elif item.tag == \"deleted\":\n if item.text == \"TRUE\":\n entity[const.DELETED] = True\n else:\n entity[const.DELETED] = False\n elif item.tag == \"cqz\":\n entity[const.CQZ] = int(item.text)\n elif item.tag == \"cont\":\n entity[const.CONTINENT] = unicode(item.text)\n elif item.tag == \"long\":\n entity[const.LONGITUDE] = float(item.text)\n elif item.tag == \"lat\":\n entity[const.LATITUDE] = float(item.text)\n elif item.tag == \"start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n entity[const.START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n entity[const.END] = dt.replace(tzinfo=UTC)\n elif item.tag == \"whitelist\":\n if item.text == \"TRUE\":\n entity[const.WHITELIST] = True\n else:\n entity[const.WHITELIST] = False\n elif item.tag == \"whitelist_start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n entity[const.WHITELIST_START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"whitelist_end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n entity[const.WHITELIST_END] = dt.replace(tzinfo=UTC)\n except AttributeError:\n self._logger.error(\"Error while processing: \")\n entities[int(cty_entity[0].text)] = entity\n self._logger.debug(str(len(entities))+\" Entities added\")\n else:\n raise Exception(\"No Country Entities detected in XML File\")\n\n\n cty_exceptions = cty_tree.find(\"exceptions\")\n if len(cty_exceptions) > 1:\n for cty_exception in cty_exceptions:\n call_exception = {}\n for item in cty_exception:\n if item.tag == \"call\":\n call = str(item.text)\n if call in call_exceptions_index.keys():\n call_exceptions_index[call].append(int(cty_exception.attrib[\"record\"]))\n else:\n call_exceptions_index[call] = [int(cty_exception.attrib[\"record\"])]\n elif item.tag == \"entity\":\n call_exception[const.COUNTRY] = unicode(item.text)\n elif item.tag == \"adif\":\n call_exception[const.ADIF] = int(item.text)\n elif item.tag == \"cqz\":\n call_exception[const.CQZ] = int(item.text)\n elif item.tag == \"cont\":\n call_exception[const.CONTINENT] = unicode(item.text)\n elif item.tag == \"long\":\n call_exception[const.LONGITUDE] = float(item.text)\n elif item.tag == \"lat\":\n call_exception[const.LATITUDE] = float(item.text)\n elif item.tag == \"start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n call_exception[const.START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n call_exception[const.END] = dt.replace(tzinfo=UTC)\n call_exceptions[int(cty_exception.attrib[\"record\"])] = call_exception\n\n self._logger.debug(str(len(call_exceptions))+\" Exceptions added\")\n self._logger.debug(str(len(call_exceptions_index))+\" unique Calls in Index \")\n\n else:\n raise Exception(\"No Exceptions detected in XML File\")\n\n\n cty_prefixes = cty_tree.find(\"prefixes\")\n if len(cty_prefixes) > 1:\n for cty_prefix in cty_prefixes:\n prefix = {}\n for item in cty_prefix:\n pref = None\n if item.tag == \"call\":\n\n #create index for this prefix\n call = str(item.text)\n if call in prefixes_index.keys():\n prefixes_index[call].append(int(cty_prefix.attrib[\"record\"]))\n else:\n prefixes_index[call] = [int(cty_prefix.attrib[\"record\"])]\n if item.tag == \"entity\":\n prefix[const.COUNTRY] = unicode(item.text)\n elif item.tag == \"adif\":\n prefix[const.ADIF] = int(item.text)\n elif item.tag == \"cqz\":\n prefix[const.CQZ] = int(item.text)\n elif item.tag == \"cont\":\n prefix[const.CONTINENT] = unicode(item.text)\n elif item.tag == \"long\":\n prefix[const.LONGITUDE] = float(item.text)\n elif item.tag == \"lat\":\n prefix[const.LATITUDE] = float(item.text)\n elif item.tag == \"start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n prefix[const.START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n prefix[const.END] = dt.replace(tzinfo=UTC)\n prefixes[int(cty_prefix.attrib[\"record\"])] = prefix\n\n self._logger.debug(str(len(prefixes))+\" Prefixes added\")\n self._logger.debug(str(len(prefixes_index))+\" unique Prefixes in Index\")\n else:\n raise Exception(\"No Prefixes detected in XML File\")\n\n cty_inv_operations = cty_tree.find(\"invalid_operations\")\n if len(cty_inv_operations) > 1:\n for cty_inv_operation in cty_inv_operations:\n invalid_operation = {}\n for item in cty_inv_operation:\n call = None\n if item.tag == \"call\":\n call = str(item.text)\n if call in invalid_operations_index.keys():\n invalid_operations_index[call].append(int(cty_inv_operation.attrib[\"record\"]))\n else:\n invalid_operations_index[call] = [int(cty_inv_operation.attrib[\"record\"])]\n\n elif item.tag == \"start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n invalid_operation[const.START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n invalid_operation[const.END] = dt.replace(tzinfo=UTC)\n invalid_operations[int(cty_inv_operation.attrib[\"record\"])] = invalid_operation\n\n self._logger.debug(str(len(invalid_operations))+\" Invalid Operations added\")\n self._logger.debug(str(len(invalid_operations_index))+\" unique Calls in Index\")\n else:\n raise Exception(\"No records for invalid operations detected in XML File\")\n\n\n cty_zone_exceptions = cty_tree.find(\"zone_exceptions\")\n if len(cty_zone_exceptions) > 1:\n for cty_zone_exception in cty_zone_exceptions:\n zoneException = {}\n for item in cty_zone_exception:\n call = None\n if item.tag == \"call\":\n call = str(item.text)\n if call in zone_exceptions_index.keys():\n zone_exceptions_index[call].append(int(cty_zone_exception.attrib[\"record\"]))\n else:\n zone_exceptions_index[call] = [int(cty_zone_exception.attrib[\"record\"])]\n\n elif item.tag == \"zone\":\n zoneException[const.CQZ] = int(item.text)\n elif item.tag == \"start\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n zoneException[const.START] = dt.replace(tzinfo=UTC)\n elif item.tag == \"end\":\n dt = datetime.strptime(item.text[:19], '%Y-%m-%dT%H:%M:%S')\n zoneException[const.END] = dt.replace(tzinfo=UTC)\n zone_exceptions[int(cty_zone_exception.attrib[\"record\"])] = zoneException\n\n self._logger.debug(str(len(zone_exceptions))+\" Zone Exceptions added\")\n self._logger.debug(str(len(zone_exceptions_index))+\" unique Calls in Index\")\n else:\n raise Exception(\"No records for zone exceptions detected in XML File\")\n\n result = {\n \"entities\" : entities,\n \"call_exceptions\" : call_exceptions,\n \"prefixes\" : prefixes,\n \"invalid_operations\" : invalid_operations,\n \"zone_exceptions\" : zone_exceptions,\n \"prefixes_index\" : prefixes_index,\n \"call_exceptions_index\" : call_exceptions_index,\n \"invalid_operations_index\" : invalid_operations_index,\n \"zone_exceptions_index\" : zone_exceptions_index,\n }\n return result", "response": "Parse the content of a clublog XML file and return the dictionary of the ADIF Country Entity IDs and their values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _parse_country_file(self, cty_file, country_mapping_filename=None):\n\n import plistlib\n\n cty_list = None\n entities = {}\n exceptions = {}\n prefixes = {}\n\n exceptions_index = {}\n prefixes_index = {}\n\n exceptions_counter = 0\n prefixes_counter = 0\n\n mapping = None\n\n with open(country_mapping_filename, \"r\") as f:\n mapping = json.loads(f.read(),encoding='UTF-8')\n\n cty_list = plistlib.readPlist(cty_file)\n\n for item in cty_list:\n entry = {}\n call = str(item)\n entry[const.COUNTRY] = unicode(cty_list[item][\"Country\"])\n if mapping:\n entry[const.ADIF] = int(mapping[cty_list[item][\"Country\"]])\n entry[const.CQZ] = int(cty_list[item][\"CQZone\"])\n entry[const.ITUZ] = int(cty_list[item][\"ITUZone\"])\n entry[const.CONTINENT] = unicode(cty_list[item][\"Continent\"])\n entry[const.LATITUDE] = float(cty_list[item][\"Latitude\"])\n entry[const.LONGITUDE] = float(cty_list[item][\"Longitude\"])*(-1)\n\n if cty_list[item][\"ExactCallsign\"]:\n if call in exceptions_index.keys():\n exceptions_index[call].append(exceptions_counter)\n else:\n exceptions_index[call] = [exceptions_counter]\n exceptions[exceptions_counter] = entry\n exceptions_counter += 1\n else:\n if call in prefixes_index.keys():\n prefixes_index[call].append(prefixes_counter)\n else:\n prefixes_index[call] = [prefixes_counter]\n prefixes[prefixes_counter] = entry\n prefixes_counter += 1\n\n self._logger.debug(str(len(prefixes))+\" Prefixes added\")\n self._logger.debug(str(len(prefixes_index))+\" Prefixes in Index\")\n self._logger.debug(str(len(exceptions))+\" Exceptions added\")\n self._logger.debug(str(len(exceptions_index))+\" Exceptions in Index\")\n\n result = {\n \"prefixes\" : prefixes,\n \"exceptions\" : exceptions,\n \"prefixes_index\" : prefixes_index,\n \"exceptions_index\" : exceptions_index,\n }\n\n return result", "response": "Parse the content of a PLIST file from country - files. com and return the parsed values in dictionaries."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _generate_random_word(self, length):\n return ''.join(random.choice(string.ascii_lowercase) for _ in range(length))", "response": "Generates a random word of the given length."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _check_html_response(self, response):\n\n error1 = \"Access to this form requires a valid API key. For more info see: http://www.clublog.org/need_api.php\"\n error2 = \"Invalid or missing API Key\"\n\n if response.status_code == requests.codes.ok:\n return True\n else:\n err_str = \"HTTP Status Code: \" + str(response.status_code) + \" HTTP Response: \" + str(response.text)\n self._logger.error(err_str)\n if response.status_code == 403:\n raise APIKeyMissingError\n else:\n raise LookupError(err_str)", "response": "Checks if the response from the API Key is OK and if it is not raise LookupError"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _serialize_data(self, my_dict):\n new_dict = {}\n for item in my_dict:\n if isinstance(my_dict[item], datetime):\n new_dict[item] = my_dict[item].strftime('%Y-%m-%d%H:%M:%S')\n else:\n new_dict[item] = str(my_dict[item])\n\n return json.dumps(new_dict)", "response": "Serialize a Dictionary into JSON"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the names of all callable attributes of an object", "response": "def get_methods(*objs):\n \"\"\" Return the names of all callable attributes of an object\"\"\"\n return set(\n attr\n for obj in objs\n for attr in dir(obj)\n if not attr.startswith('_') and callable(getattr(obj, attr))\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new Config object from a configuration file.", "response": "def from_file(cls, filename, *, strict=True):\n \"\"\" Create a new Config object from a configuration file.\n\n Args:\n filename (str): The location and name of the configuration file.\n strict (bool): If true raises a ConfigLoadError when the configuration\n cannot be found.\n\n Returns:\n An instance of the Config class.\n\n Raises:\n ConfigLoadError: If the configuration cannot be found.\n \"\"\"\n config = cls()\n config.load_from_file(filename, strict=strict)\n return config"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_from_file(self, filename=None, *, strict=True):\n self.set_to_default()\n\n if filename:\n self._update_from_file(filename)\n else:\n if LIGHTFLOW_CONFIG_ENV not in os.environ:\n if os.path.isfile(os.path.join(os.getcwd(), LIGHTFLOW_CONFIG_NAME)):\n self._update_from_file(\n os.path.join(os.getcwd(), LIGHTFLOW_CONFIG_NAME))\n elif os.path.isfile(expand_env_var('~/{}'.format(LIGHTFLOW_CONFIG_NAME))):\n self._update_from_file(\n expand_env_var('~/{}'.format(LIGHTFLOW_CONFIG_NAME)))\n else:\n if strict:\n raise ConfigLoadError('Could not find the configuration file.')\n else:\n self._update_from_file(expand_env_var(os.environ[LIGHTFLOW_CONFIG_ENV]))\n\n self._update_python_paths()", "response": "Loads the configuration from a file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_from_dict(self, conf_dict=None):\n self.set_to_default()\n self._update_dict(self._config, conf_dict)\n self._update_python_paths()", "response": "Load the configuration from a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _update_from_file(self, filename):\n if os.path.exists(filename):\n try:\n with open(filename, 'r') as config_file:\n yaml_dict = yaml.safe_load(config_file.read())\n if yaml_dict is not None:\n self._update_dict(self._config, yaml_dict)\n except IsADirectoryError:\n raise ConfigLoadError(\n 'The specified configuration file is a directory not a file')\n else:\n raise ConfigLoadError('The config file {} does not exist'.format(filename))", "response": "Updates the dictionary with the values from a file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _update_python_paths(self):\n for path in self._config['workflows'] + self._config['libraries']:\n if os.path.isdir(os.path.abspath(path)):\n if path not in sys.path:\n sys.path.append(path)\n else:\n raise ConfigLoadError(\n 'Workflow directory {} does not exist'.format(path))", "response": "Append the workflow and libraries paths to the PYTHONPATH."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef decode_char_spot(raw_string):\n\n data = {}\n\n # Spotter callsign\n if re.match('[A-Za-z0-9\\/]+[:$]', raw_string[6:15]):\n data[const.SPOTTER] = re.sub(':', '', re.match('[A-Za-z0-9\\/]+[:$]', raw_string[6:15]).group(0))\n else:\n raise ValueError\n\n if re.search('[0-9\\.]{5,12}', raw_string[10:25]):\n data[const.FREQUENCY] = float(re.search('[0-9\\.]{5,12}', raw_string[10:25]).group(0))\n else:\n raise ValueError\n\n data[const.DX] = re.sub('[^A-Za-z0-9\\/]+', '', raw_string[26:38])\n data[const.COMMENT] = re.sub('[^\\sA-Za-z0-9\\.,;\\#\\+\\-!\\?\\$\\(\\)@\\/]+', ' ', raw_string[39:69]).strip()\n data[const.TIME] = datetime.now().replace(tzinfo=UTC)\n\n return data", "response": "Decode a single character spot into pieces and return a dict with the spot data"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndecoding PC11 message which usually contains DX Spots", "response": "def decode_pc11_message(raw_string):\n \"\"\"Decode PC11 message, which usually contains DX Spots\"\"\"\n\n data = {}\n spot = raw_string.split(\"^\")\n data[const.FREQUENCY] = float(spot[1])\n data[const.DX] = spot[2]\n data[const.TIME] = datetime.fromtimestamp(mktime(strptime(spot[3]+\" \"+spot[4][:-1], \"%d-%b-%Y %H%M\")))\n data[const.COMMENT] = spot[5]\n data[const.SPOTTER] = spot[6]\n data[\"node\"] = spot[7]\n data[\"raw_spot\"] = raw_string\n return data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef decode_pc23_message(raw_string):\n\n data = {}\n wcy = raw_string.split(\"^\")\n data[const.R] = int(wcy[1])\n data[const.expk] = int(wcy[2])\n data[const.CALLSIGN] = wcy[3]\n data[const.A] = wcy[4]\n data[const.SFI] = wcy[5]\n data[const.K] = wcy[6]\n data[const.AURORA] = wcy[7]\n data[\"node\"] = wcy[7]\n data[\"ip\"] = wcy[8]\n data[\"raw_data\"] = raw_string\n return data", "response": "Decode PC23 Message which usually contains WCY"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _run(self, data, store, signal, context, *,\n success_callback=None, stop_callback=None, abort_callback=None):\n \"\"\" The internal run method that decorates the public run method.\n\n This method makes sure data is being passed to and from the task.\n\n Args:\n data (MultiTaskData): The data object that has been passed from the\n predecessor task.\n store (DataStoreDocument): The persistent data store object that allows the\n task to store data for access across the current\n workflow run.\n signal (TaskSignal): The signal object for tasks. It wraps the construction\n and sending of signals into easy to use methods.\n context (TaskContext): The context in which the tasks runs.\n success_callback: This function is called when the task completed successfully\n stop_callback: This function is called when a StopTask exception was raised.\n abort_callback: This function is called when an AbortWorkflow exception\n was raised.\n\n Raises:\n TaskReturnActionInvalid: If the return value of the task is not\n an Action object.\n\n Returns:\n Action: An Action object containing the data that should be passed on\n to the next task and optionally a list of successor tasks that\n should be executed.\n \"\"\"\n if data is None:\n data = MultiTaskData()\n data.add_dataset(self._name)\n\n try:\n if self._callback_init is not None:\n self._callback_init(data, store, signal, context)\n\n result = self.run(data, store, signal, context)\n\n if self._callback_finally is not None:\n self._callback_finally(TaskStatus.Success, data, store, signal, context)\n\n if success_callback is not None:\n success_callback()\n\n # the task should be stopped and optionally all successor tasks skipped\n except StopTask as err:\n if self._callback_finally is not None:\n self._callback_finally(TaskStatus.Stopped, data, store, signal, context)\n\n if stop_callback is not None:\n stop_callback(exc=err)\n\n result = Action(data, limit=[]) if err.skip_successors else None\n\n # the workflow should be stopped immediately\n except AbortWorkflow as err:\n if self._callback_finally is not None:\n self._callback_finally(TaskStatus.Aborted, data, store, signal, context)\n\n if abort_callback is not None:\n abort_callback(exc=err)\n\n result = None\n signal.stop_workflow()\n\n # catch any other exception, call the finally callback, then re-raise\n except:\n if self._callback_finally is not None:\n self._callback_finally(TaskStatus.Error, data, store, signal, context)\n\n signal.stop_workflow()\n raise\n\n # handle the returned data (either implicitly or as an returned Action object) by\n # flattening all, possibly modified, input datasets in the MultiTask data down to\n # a single output dataset.\n if result is None:\n data.flatten(in_place=True)\n data.add_task_history(self.name)\n return Action(data)\n else:\n if not isinstance(result, Action):\n raise TaskReturnActionInvalid()\n\n result.data.flatten(in_place=True)\n result.data.add_task_history(self.name)\n return result", "response": "This method is called by the internal run method of the internal run method. It wraps the run method to make sure that the data object passed to the internal run method is passed to the internal run method."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef latlong_to_locator (latitude, longitude):\n\n if longitude >= 180 or longitude <= -180:\n raise ValueError\n\n if latitude >= 90 or latitude <= -90:\n raise ValueError\n\n longitude += 180;\n latitude +=90;\n\n locator = chr(ord('A') + int(longitude / 20))\n locator += chr(ord('A') + int(latitude / 10))\n locator += chr(ord('0') + int((longitude % 20) / 2))\n locator += chr(ord('0') + int(latitude % 10))\n locator += chr(ord('A') + int((longitude - int(longitude / 2) * 2) / (2 / 24)))\n locator += chr(ord('A') + int((latitude - int(latitude / 1) * 1 ) / (1 / 24)))\n\n return locator", "response": "Converts WGS84 coordinates into Maidenhead Locator"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef locator_to_latlong (locator):\n\n locator = locator.upper()\n\n if len(locator) == 5 or len(locator) < 4:\n raise ValueError\n\n if ord(locator[0]) > ord('R') or ord(locator[0]) < ord('A'):\n raise ValueError\n\n if ord(locator[1]) > ord('R') or ord(locator[1]) < ord('A'):\n raise ValueError\n\n if ord(locator[2]) > ord('9') or ord(locator[2]) < ord('0'):\n raise ValueError\n\n if ord(locator[3]) > ord('9') or ord(locator[3]) < ord('0'):\n raise ValueError\n\n if len(locator) == 6:\n if ord(locator[4]) > ord('X') or ord(locator[4]) < ord('A'):\n raise ValueError\n if ord (locator[5]) > ord('X') or ord(locator[5]) < ord('A'):\n raise ValueError\n\n longitude = (ord(locator[0]) - ord('A')) * 20 - 180\n latitude = (ord(locator[1]) - ord('A')) * 10 - 90\n longitude += (ord(locator[2]) - ord('0')) * 2\n latitude += (ord(locator[3]) - ord('0'))\n\n if len(locator) == 6:\n longitude += ((ord(locator[4])) - ord('A')) * (2 / 24)\n latitude += ((ord(locator[5])) - ord('A')) * (1 / 24)\n\n # move to center of subsquare\n longitude += 1 / 24\n latitude += 0.5 / 24\n\n else:\n # move to center of square\n longitude += 1;\n latitude += 0.5;\n\n return latitude, longitude", "response": "Converts a Maidenhead locator in the corresponding WGS84 coordinates into a Latitude and Longitude tuple."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the distance between two Maidenhead locators in km", "response": "def calculate_distance(locator1, locator2):\n \"\"\"calculates the (shortpath) distance between two Maidenhead locators\n\n Args:\n locator1 (string): Locator, either 4 or 6 characters\n locator2 (string): Locator, either 4 or 6 characters\n\n Returns:\n float: Distance in km\n\n Raises:\n ValueError: When called with wrong or invalid input arg\n AttributeError: When args are not a string\n\n Example:\n The following calculates the distance between two Maidenhead locators in km\n\n >>> from pyhamtools.locator import calculate_distance\n >>> calculate_distance(\"JN48QM\", \"QF67bf\")\n 16466.413\n\n \"\"\"\n\n R = 6371 #earh radius\n lat1, long1 = locator_to_latlong(locator1)\n lat2, long2 = locator_to_latlong(locator2)\n\n d_lat = radians(lat2) - radians(lat1)\n d_long = radians(long2) - radians(long1)\n\n r_lat1 = radians(lat1)\n r_long1 = radians(long1)\n r_lat2 = radians(lat2)\n r_long2 = radians(long2)\n\n a = sin(d_lat/2) * sin(d_lat/2) + cos(r_lat1) * cos(r_lat2) * sin(d_long/2) * sin(d_long/2)\n c = 2 * atan2(sqrt(a), sqrt(1-a))\n d = R * c #distance in km\n\n return d;"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the longpath distance between two Maidenhead locators in km", "response": "def calculate_distance_longpath(locator1, locator2):\n \"\"\"calculates the (longpath) distance between two Maidenhead locators\n\n Args:\n locator1 (string): Locator, either 4 or 6 characters\n locator2 (string): Locator, either 4 or 6 characters\n\n Returns:\n float: Distance in km\n\n Raises:\n ValueError: When called with wrong or invalid input arg\n AttributeError: When args are not a string\n\n Example:\n The following calculates the longpath distance between two Maidenhead locators in km\n\n >>> from pyhamtools.locator import calculate_distance_longpath\n >>> calculate_distance_longpath(\"JN48QM\", \"QF67bf\")\n 23541.5867\n\n \"\"\"\n\n c = 40008 #[km] earth circumference\n sp = calculate_distance(locator1, locator2)\n\n return c - sp"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calculate_heading(locator1, locator2):\n\n lat1, long1 = locator_to_latlong(locator1)\n lat2, long2 = locator_to_latlong(locator2)\n\n r_lat1 = radians(lat1)\n r_lon1 = radians(long1)\n\n r_lat2 = radians(lat2)\n r_lon2 = radians(long2)\n\n d_lon = radians(long2 - long1)\n\n b = atan2(sin(d_lon)*cos(r_lat2),cos(r_lat1)*sin(r_lat2)-sin(r_lat1)*cos(r_lat2)*cos(d_lon)) # bearing calc\n bd = degrees(b)\n br,bn = divmod(bd+360,360) # the bearing remainder and final bearing\n\n return bn", "response": "Calculates the heading from the first to the second locator\n and returns the heading in degree."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef calculate_heading_longpath(locator1, locator2):\n\n heading = calculate_heading(locator1, locator2)\n\n lp = (heading + 180)%360\n\n return lp", "response": "calculates the long path heading from the first to the second locator"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef calculate_sunrise_sunset(locator, calc_date=datetime.utcnow()):\n morning_dawn = None\n sunrise = None\n evening_dawn = None\n sunset = None\n\n latitude, longitude = locator_to_latlong(locator)\n\n if type(calc_date) != datetime:\n raise ValueError\n\n sun = ephem.Sun()\n home = ephem.Observer()\n\n home.lat = str(latitude)\n home.long = str(longitude)\n home.date = calc_date\n\n sun.compute(home)\n\n try:\n nextrise = home.next_rising(sun)\n nextset = home.next_setting(sun)\n\n home.horizon = '-6'\n beg_twilight = home.next_rising(sun, use_center=True)\n end_twilight = home.next_setting(sun, use_center=True)\n\n morning_dawn = beg_twilight.datetime()\n sunrise = nextrise.datetime()\n\n evening_dawn = nextset.datetime()\n sunset = end_twilight.datetime()\n\n #if sun never sets or rises (e.g. at polar circles)\n except ephem.AlwaysUpError as e:\n morning_dawn = None\n sunrise = None\n evening_dawn = None\n sunset = None\n except ephem.NeverUpError as e:\n morning_dawn = None\n sunrise = None\n evening_dawn = None\n sunset = None\n\n result = {}\n result['morning_dawn'] = morning_dawn\n result['sunrise'] = sunrise\n result['evening_dawn'] = evening_dawn\n result['sunset'] = sunset\n\n if morning_dawn:\n result['morning_dawn'] = morning_dawn.replace(tzinfo=UTC)\n if sunrise:\n result['sunrise'] = sunrise.replace(tzinfo=UTC)\n if evening_dawn:\n result['evening_dawn'] = evening_dawn.replace(tzinfo=UTC)\n if sunset:\n result['sunset'] = sunset.replace(tzinfo=UTC)\n return result", "response": "Calculates the next sunset and sunrise for a Maidenhead locator at a give date & time."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nencoding Python objects into a byte stream using cloudpickle.", "response": "def cloudpickle_dumps(obj, dumper=cloudpickle.dumps):\n \"\"\" Encode Python objects into a byte stream using cloudpickle. \"\"\"\n return dumper(obj, protocol=serialization.pickle_protocol)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef patch_celery():\n registry = serialization.registry\n serialization.pickle = cloudpickle\n registry.unregister('pickle')\n registry.register('pickle', cloudpickle_dumps, cloudpickle_loads,\n content_type='application/x-python-serialize',\n content_encoding='binary')\n\n import celery.worker as worker\n import celery.concurrency.asynpool as asynpool\n worker.state.pickle = cloudpickle\n asynpool._pickle = cloudpickle\n\n import billiard.common\n billiard.common.pickle = cloudpickle\n billiard.common.pickle_dumps = cloudpickle_dumps\n billiard.common.pickle_loads = cloudpickle_loads", "response": "Monkey patch Celery to use cloudpickle instead of pickle."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconnects to the redis database.", "response": "def connect(self):\n \"\"\" Connects to the redis database. \"\"\"\n self._connection = StrictRedis(\n host=self._host,\n port=self._port,\n db=self._database,\n password=self._password)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef receive(self):\n pickled_request = self._connection.connection.lpop(self._request_key)\n return pickle.loads(pickled_request) if pickled_request is not None else None", "response": "Returns a single request from the list of requests and returns it."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send(self, response):\n self._connection.connection.set('{}:{}'.format(SIGNAL_REDIS_PREFIX, response.uid),\n pickle.dumps(response))", "response": "Send a response back to the client that issued a request."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef restore(self, request):\n self._connection.connection.rpush(self._request_key, pickle.dumps(request))", "response": "Push the request back onto the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send(self, request):\n self._connection.connection.rpush(self._request_key, pickle.dumps(request))\n resp_key = '{}:{}'.format(SIGNAL_REDIS_PREFIX, request.uid)\n\n while True:\n if self._connection.polling_time > 0.0:\n sleep(self._connection.polling_time)\n\n response_data = self._connection.connection.get(resp_key)\n if response_data is not None:\n self._connection.connection.delete(resp_key)\n break\n\n return pickle.loads(response_data)", "response": "Send a request to the server and wait for its response."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef verify_pattern(pattern):\n\n regex = re.compile(\"^!?[a-zA-Z]+$|[*]{1,2}$\")\n\n def __verify_pattern__(__pattern__):\n if not __pattern__:\n return False\n elif __pattern__[0] == \"!\":\n return __verify_pattern__(__pattern__[1:])\n elif __pattern__[0] == \"[\" and __pattern__[-1] == \"]\":\n return all(__verify_pattern__(p) for p in __pattern__[1:-1].split(\",\"))\n else:\n return regex.match(__pattern__)\n return all(__verify_pattern__(p) for p in pattern.split(\"/\"))", "response": "Verifies if pattern for matching and finding fulfill expected structure."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef print_tree(sent, token_attr):\n def __print_sent__(token, attr):\n print(\"{\", end=\" \")\n [__print_sent__(t, attr) for t in token.lefts]\n print(u\"%s->%s(%s)\" % (token,token.dep_,token.tag_ if not attr else getattr(token, attr)), end=\"\")\n [__print_sent__(t, attr) for t in token.rights]\n print(\"}\", end=\" \")\n return __print_sent__(sent.root, token_attr)", "response": "Prints a tree of sentences using token_attr from token"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmatches given sentence with given pattern.", "response": "def match_tree(sentence, pattern):\n \"\"\"Matches given sentence with provided pattern.\n\n :param sentence: sentence from Spacy(see: http://spacy.io/docs/#doc-spans-sents) representing complete statement\n :param pattern: pattern to which sentence will be compared\n\n :return: True if sentence match to pattern, False otherwise\n\n :raises: PatternSyntaxException: if pattern has wrong syntax\n\n \"\"\"\n\n if not verify_pattern(pattern):\n raise PatternSyntaxException(pattern)\n\n def _match_node(t, p):\n pat_node = p.pop(0) if p else \"\"\n return not pat_node or (_match_token(t, pat_node, False) and _match_edge(t.children,p))\n\n def _match_edge(edges,p):\n pat_edge = p.pop(0) if p else \"\"\n if not pat_edge:\n return True\n elif not edges:\n return False\n else:\n for (t) in edges:\n if (_match_token(t, pat_edge, True)) and _match_node(t, list(p)):\n return True\n elif pat_edge == \"**\" and _match_edge(t.children, [\"**\"] + p):\n return True\n return False\n return _match_node(sentence.root, pattern.split(\"/\"))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_tokens(sentence, pattern):\n\n if not verify_pattern(pattern):\n raise PatternSyntaxException(pattern)\n\n def _match_node(t, p, tokens):\n pat_node = p.pop(0) if p else \"\"\n res = not pat_node or (_match_token(t, pat_node, False) and (not p or _match_edge(t.children, p, tokens)))\n if res and not p:\n tokens.append(t)\n return res\n\n def _match_edge(edges,p, tokens):\n pat_edge = p.pop(0) if p else \"\"\n if pat_edge:\n for (t) in edges:\n if _match_token(t, pat_edge, True):\n _match_node(t, list(p), tokens)\n if pat_edge == \"**\":\n _match_edge(t.children, [\"**\"] + p, tokens)\n result_tokens = []\n _match_node(sentence.root, pattern.split(\"/\"), result_tokens)\n return result_tokens", "response": "Find all tokens from parts of sentence fitted to pattern."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsplits a string using a single - character delimter.", "response": "def split (s, delimter, trim = True, limit = 0): # pragma: no cover\n\t\t\"\"\"\n\t\tSplit a string using a single-character delimter\n\t\t@params:\n\t\t\t`s`: the string\n\t\t\t`delimter`: the single-character delimter\n\t\t\t`trim`: whether to trim each part. Default: True\n\t\t@examples:\n\t\t\t```python\n\t\t\tret = split(\"'a,b',c\", \",\")\n\t\t\t# ret == [\"'a,b'\", \"c\"]\n\t\t\t# ',' inside quotes will be recognized.\n\t\t\t```\n\t\t@returns:\n\t\t\tThe list of substrings\n\t\t\"\"\"\n\t\tret = []\n\t\tspecial1 = ['(', ')', '[', ']', '{', '}']\n\t\tspecial2 = ['\\'', '\"']\n\t\tspecial3 = '\\\\'\n\t\tflags1 = [0, 0, 0]\n\t\tflags2 = [False, False]\n\t\tflags3 = False\n\t\tstart = 0\n\t\tnlim = 0\n\t\tfor i, c in enumerate(s):\n\t\t\tif c == special3:\n\t\t\t\t# next char is escaped\n\t\t\t\tflags3 = not flags3\n\t\t\telif not flags3:\n\t\t\t\t# no escape\n\t\t\t\tif c in special1:\n\t\t\t\t\tindex = special1.index(c)\n\t\t\t\t\tif index % 2 == 0:\n\t\t\t\t\t\tflags1[int(index/2)] += 1\n\t\t\t\t\telse:\n\t\t\t\t\t\tflags1[int(index/2)] -= 1\n\t\t\t\telif c in special2:\n\t\t\t\t\tindex = special2.index(c)\n\t\t\t\t\tflags2[index] = not flags2[index]\n\t\t\t\telif c == delimter and not any(flags1) and not any(flags2):\n\t\t\t\t\tr = s[start:i]\n\t\t\t\t\tif trim: r = r.strip()\n\t\t\t\t\tret.append(r)\n\t\t\t\t\tstart = i + 1\n\t\t\t\t\tnlim = nlim + 1\n\t\t\t\t\tif limit and nlim >= limit:\n\t\t\t\t\t\tbreak\n\t\t\telse:\n\t\t\t\t# escaping closed\n\t\t\t\tflags3 = False\n\t\tr = s[start:]\n\t\tif trim: r = r.strip()\n\t\tret.append(r)\n\t\treturn ret"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrendering this template by applying it to context.", "response": "def render(self, **context):\n\t\t\"\"\"\n\t\tRender this template by applying it to `context`.\n\t\t@params:\n\t\t\t`context`: a dictionary of values to use in this rendering.\n\t\t@returns:\n\t\t\tThe rendered string\n\t\t\"\"\"\n\t\t# Make the complete context we'll use.\n\t\tlocalns = self.envs.copy()\n\t\tlocalns.update(context)\n\n\t\ttry:\n\t\t\texec(str(self.code), None, localns)\n\t\t\treturn localns[Liquid.COMPLIED_RENDERED_STR]\n\t\texcept Exception:\n\t\t\tstacks = list(reversed(traceback.format_exc().splitlines()))\n\t\t\tfor stack in stacks:\n\t\t\t\tstack = stack.strip()\n\t\t\t\tif stack.startswith('File \"\"'):\n\t\t\t\t\tlineno = int(stack.split(', ')[1].split()[-1])\n\t\t\t\t\tsource = []\n\t\t\t\t\tif 'NameError:' in stacks[0]:\n\t\t\t\t\t\tsource.append('Do you forget to provide the data?')\n\n\t\t\t\t\timport math\n\t\t\t\t\tsource.append('\\nCompiled source (use debug mode to see full source):')\n\t\t\t\t\tsource.append('---------------------------------------------------')\n\t\t\t\t\tnlines = len(self.code.codes)\n\t\t\t\t\tnbit = int(math.log(nlines, 10)) + 3\n\t\t\t\t\tfor i, line in enumerate(self.code.codes):\n\t\t\t\t\t\tif i - 7 > lineno or i + 9 < lineno: continue\n\t\t\t\t\t\tif i + 1 != lineno:\n\t\t\t\t\t\t\tsource.append(' ' + (str(i+1) + '.').ljust(nbit) + str(line).rstrip())\n\t\t\t\t\t\telse:\n\t\t\t\t\t\t\tsource.append('* ' + (str(i+1) + '.').ljust(nbit) + str(line).rstrip())\n\n\t\t\t\t\traise LiquidRenderError(\n\t\t\t\t\t\tstacks[0], \n\t\t\t\t\t\trepr(self.code.codes[lineno - 1]) + \n\t\t\t\t\t\t'\\n' + '\\n'.join(source) + \n\t\t\t\t\t\t'\\n\\nPREVIOUS EXCEPTION:\\n------------------\\n' + \n\t\t\t\t\t\t'\\n'.join(stacks) + '\\n' +\n\t\t\t\t\t\t'\\nCONTEXT:\\n------------------\\n' +\n\t\t\t\t\t\t'\\n'.join(\n\t\t\t\t\t\t\t' ' + key + ': ' + str(val) \n\t\t\t\t\t\t\tfor key, val in localns.items() if not key.startswith('_liquid_') and not key.startswith('__')\n\t\t\t\t\t\t) + '\\n'\n\t\t\t\t\t)\n\t\t\traise"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a line of source to the code.", "response": "def addLine(self, line):\n\t\t\"\"\"\n\t\tAdd a line of source to the code.\n\t\tIndentation and newline will be added for you, don't provide them.\n\t\t@params:\n\t\t\t`line`: The line to add\n\t\t\"\"\"\n\t\tif not isinstance(line, LiquidLine):\n\t\t\tline = LiquidLine(line)\n\t\tline.ndent = self.ndent\n\t\tself.codes.append(line)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting or set the logging level.", "response": "def level(self, lvl=None):\n '''Get or set the logging level.'''\n if not lvl:\n return self._lvl\n self._lvl = self._parse_level(lvl)\n self.stream.setLevel(self._lvl)\n logging.root.setLevel(self._lvl)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfetches currency conversion rate from the database", "response": "def get_rate_from_db(currency: str) -> Decimal:\n \"\"\"\n Fetch currency conversion rate from the database\n \"\"\"\n from .models import ConversionRate\n try:\n rate = ConversionRate.objects.get_rate(currency)\n except ConversionRate.DoesNotExist: # noqa\n raise ValueError('No conversion rate for %s' % (currency, ))\n return rate.rate"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets conversion rate for a given currency", "response": "def get_conversion_rate(from_currency: str, to_currency: str) -> Decimal:\n \"\"\"\n Get conversion rate to use in exchange\n \"\"\"\n reverse_rate = False\n if to_currency == BASE_CURRENCY:\n # Fetch exchange rate for base currency and use 1 / rate for conversion\n rate_currency = from_currency\n reverse_rate = True\n else:\n rate_currency = to_currency\n rate = get_rate_from_db(rate_currency)\n\n if reverse_rate:\n conversion_rate = Decimal(1) / rate\n else:\n conversion_rate = rate\n return conversion_rate"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef exchange_currency(\n base: T, to_currency: str, *, conversion_rate: Decimal=None) -> T:\n \"\"\"\n Exchanges Money, TaxedMoney and their ranges to the specified currency.\n get_rate parameter is a callable taking single argument (target currency)\n that returns proper conversion rate\n \"\"\"\n if base.currency == to_currency:\n return base\n if base.currency != BASE_CURRENCY and to_currency != BASE_CURRENCY:\n # Exchange to base currency first\n base = exchange_currency(base, BASE_CURRENCY)\n\n if conversion_rate is None:\n conversion_rate = get_conversion_rate(base.currency, to_currency)\n\n if isinstance(base, Money):\n return Money(base.amount * conversion_rate, currency=to_currency)\n if isinstance(base, MoneyRange):\n return MoneyRange(\n exchange_currency(\n base.start, to_currency, conversion_rate=conversion_rate),\n exchange_currency(\n base.stop, to_currency, conversion_rate=conversion_rate))\n if isinstance(base, TaxedMoney):\n return TaxedMoney(\n exchange_currency(\n base.net, to_currency, conversion_rate=conversion_rate),\n exchange_currency(\n base.gross, to_currency, conversion_rate=conversion_rate))\n if isinstance(base, TaxedMoneyRange):\n return TaxedMoneyRange(\n exchange_currency(\n base.start, to_currency, conversion_rate=conversion_rate),\n exchange_currency(\n base.stop, to_currency, conversion_rate=conversion_rate))\n\n # base.currency was set but we don't know how to exchange given type\n raise TypeError('Unknown base for exchange_currency: %r' % (base,))", "response": "Exchange currency from base to another currency."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef normalize(location_name, preserve_commas=False):\n def replace(match):\n if preserve_commas and ',' in match.group(0):\n return ','\n return ' '\n return NORMALIZATION_RE.sub(replace, location_name).strip().lower()", "response": "Normalize a location name by stripping punctuation and collapsing\n runs of whitespace and returning the normalized name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating precipitation statistics for the cascade model while aggregating hourly observations", "response": "def calc_precipitation_stats(self, months=None, avg_stats=True, percentile=50):\n \"\"\"\n Calculates precipitation statistics for the cascade model while aggregating hourly observations\n\n Parameters\n ----------\n months : Months for each seasons to be used for statistics (array of numpy array, default=1-12, e.g., [np.arange(12) + 1])\n avg_stats : average statistics for all levels True/False (default=True)\n percentile : percentil for splitting the dataset in small and high intensities (default=50)\n \n \"\"\"\n if months is None:\n months = [np.arange(12) + 1]\n\n self.precip.months = months\n self.precip.stats = melodist.build_casc(self.data, months=months, avg_stats=avg_stats, percentile=percentile)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates statistics in order to derive diurnal patterns of wind speed", "response": "def calc_wind_stats(self):\n \"\"\"\n Calculates statistics in order to derive diurnal patterns of wind speed\n \"\"\"\n a, b, t_shift = melodist.fit_cosine_function(self.data.wind)\n self.wind.update(a=a, b=b, t_shift=t_shift)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate statistics in order to derive diurnal patterns of relative humidity.", "response": "def calc_humidity_stats(self):\n \"\"\"\n Calculates statistics in order to derive diurnal patterns of relative humidity.\n \"\"\"\n a1, a0 = melodist.calculate_dewpoint_regression(self.data, return_stats=False)\n self.hum.update(a0=a0, a1=a1)\n self.hum.kr = 12\n\n self.hum.month_hour_precip_mean = melodist.calculate_month_hour_precip_mean(self.data)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef calc_temperature_stats(self):\n self.temp.max_delta = melodist.get_shift_by_data(self.data.temp, self._lon, self._lat, self._timezone)\n self.temp.mean_course = melodist.util.calculate_mean_daily_course_by_month(self.data.temp, normalize=True)", "response": "Calculates statistics in order to derive diurnal patterns of temperature"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating statistics in order to derive solar radiation from sunshine duration or solar temperature.", "response": "def calc_radiation_stats(self, data_daily=None, day_length=None, how='all'):\n \"\"\"\n Calculates statistics in order to derive solar radiation from sunshine duration or\n minimum/maximum temperature.\n\n Parameters\n ----------\n data_daily : DataFrame, optional\n Daily data from the associated ``Station`` object.\n\n day_length : Series, optional\n Day lengths as calculated by ``calc_sun_times``.\n \"\"\"\n assert how in ('all', 'seasonal', 'monthly')\n\n self.glob.mean_course = melodist.util.calculate_mean_daily_course_by_month(self.data.glob)\n\n if data_daily is not None:\n pot_rad = melodist.potential_radiation(\n melodist.util.hourly_index(data_daily.index),\n self._lon, self._lat, self._timezone)\n pot_rad_daily = pot_rad.resample('D').mean()\n obs_rad_daily = self.data.glob.resample('D').mean()\n\n if how == 'all':\n month_ranges = [np.arange(12) + 1]\n elif how == 'seasonal':\n month_ranges = [[3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 1, 2]]\n elif how == 'monthly':\n month_ranges = zip(np.arange(12) + 1)\n\n def myisin(s, v):\n return pd.Series(s).isin(v).values\n\n def extract_months(s, months):\n return s[myisin(s.index.month, months)]\n\n if 'ssd' in data_daily and day_length is not None:\n for months in month_ranges:\n a, b = melodist.fit_angstroem_params(\n extract_months(data_daily.ssd, months),\n extract_months(day_length, months),\n extract_months(pot_rad_daily, months),\n extract_months(obs_rad_daily, months),\n )\n\n for month in months:\n self.glob.angstroem.loc[month] = a, b\n\n if 'tmin' in data_daily and 'tmax' in data_daily:\n df = pd.DataFrame(\n data=dict(\n tmin=data_daily.tmin,\n tmax=data_daily.tmax,\n pot_rad=pot_rad_daily,\n obs_rad=obs_rad_daily,\n )\n ).dropna(how='any')\n\n for months in month_ranges:\n a, c = melodist.fit_bristow_campbell_params(\n extract_months(df.tmin, months),\n extract_months(df.tmax, months),\n extract_months(df.pot_rad, months),\n extract_months(df.obs_rad, months),\n )\n\n for month in months:\n self.glob.bristcamp.loc[month] = a, c"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_json(self, filename=None):\n def json_encoder(obj):\n if isinstance(obj, pd.DataFrame) or isinstance(obj, pd.Series):\n if isinstance(obj.index, pd.core.index.MultiIndex):\n obj = obj.reset_index() # convert MultiIndex to columns\n\n return json.loads(obj.to_json(date_format='iso'))\n elif isinstance(obj, melodist.cascade.CascadeStatistics):\n return obj.__dict__\n elif isinstance(obj, np.ndarray):\n return obj.tolist()\n else:\n raise TypeError('%s not supported' % type(obj))\n\n d = dict(\n temp=self.temp,\n wind=self.wind,\n precip=self.precip,\n hum=self.hum,\n glob=self.glob\n )\n\n j = json.dumps(d, default=json_encoder, indent=4)\n\n if filename is None:\n return j\n else:\n with open(filename, 'w') as f:\n f.write(j)", "response": "Exports statistical data to a JSON formatted file"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_json(cls, filename):\n def json_decoder(d):\n if 'p01' in d and 'pxx' in d: # we assume this is a CascadeStatistics object\n return melodist.cascade.CascadeStatistics.from_dict(d)\n\n return d\n\n with open(filename) as f:\n d = json.load(f, object_hook=json_decoder)\n\n stats = cls()\n\n stats.temp.update(d['temp'])\n stats.hum.update(d['hum'])\n stats.precip.update(d['precip'])\n stats.wind.update(d['wind'])\n stats.glob.update(d['glob'])\n\n if stats.temp.max_delta is not None:\n stats.temp.max_delta = pd.read_json(json.dumps(stats.temp.max_delta), typ='series').sort_index()\n\n if stats.temp.mean_course is not None:\n mc = pd.read_json(json.dumps(stats.temp.mean_course), typ='frame').sort_index()[np.arange(1, 12 + 1)]\n stats.temp.mean_course = mc.sort_index()[np.arange(1, 12 + 1)]\n\n if stats.hum.month_hour_precip_mean is not None:\n mhpm = pd.read_json(json.dumps(stats.hum.month_hour_precip_mean), typ='frame').sort_index()\n mhpm = mhpm.set_index(['level_0', 'level_1', 'level_2']) # convert to MultiIndex\n mhpm = mhpm.squeeze() # convert to Series\n mhpm = mhpm.rename_axis([None, None, None]) # remove index labels\n stats.hum.month_hour_precip_mean = mhpm\n\n for var in ('angstroem', 'bristcamp', 'mean_course'):\n if stats.glob[var] is not None:\n stats.glob[var] = pd.read_json(json.dumps(stats.glob[var])).sort_index()\n\n if stats.glob.mean_course is not None:\n stats.glob.mean_course = stats.glob.mean_course[np.arange(1, 12 + 1)]\n\n return stats", "response": "Imports statistical data from a JSON formatted file containing statistics."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disaggregate_radiation(data_daily,\r\n sun_times=None,\r\n pot_rad=None,\r\n method='pot_rad',\r\n angstr_a=0.25,\r\n angstr_b=0.5,\r\n bristcamp_a=0.75,\r\n bristcamp_c=2.4,\r\n mean_course=None):\r\n \"\"\"general function for radiation disaggregation\r\n\r\n Args:\r\n daily_data: daily values\r\n sun_times: daily dataframe including results of the util.sun_times function\r\n pot_rad: hourly dataframe including potential radiation\r\n method: keyword specifying the disaggregation method to be used\r\n angstr_a: parameter a of the Angstrom model (intercept)\r\n angstr_b: parameter b of the Angstrom model (slope)\r\n mean_course: monthly values of the mean hourly radiation course\r\n \r\n Returns:\r\n Disaggregated hourly values of shortwave radiation.\r\n \"\"\"\r\n # check if disaggregation method has a valid value\r\n if method not in ('pot_rad', 'pot_rad_via_ssd', 'pot_rad_via_bc', 'mean_course'):\r\n raise ValueError('Invalid option')\r\n\r\n glob_disagg = pd.Series(index=melodist.util.hourly_index(data_daily.index))\r\n\r\n if method == 'mean_course':\r\n assert mean_course is not None\r\n\r\n pot_rad = pd.Series(index=glob_disagg.index)\r\n pot_rad[:] = mean_course.unstack().loc[list(zip(pot_rad.index.month, pot_rad.index.hour))].values\r\n else:\r\n assert pot_rad is not None\r\n\r\n pot_rad_daily = pot_rad.resample('D').mean()\r\n\r\n if method in ('pot_rad', 'mean_course'):\r\n globalrad = data_daily.glob\r\n elif method == 'pot_rad_via_ssd':\r\n # in this case use the Angstrom model\r\n globalrad = pd.Series(index=data_daily.index, data=0.)\r\n dates = sun_times.index[sun_times.daylength > 0] # account for polar nights\r\n globalrad[dates] = angstroem(data_daily.ssd[dates], sun_times.daylength[dates],\r\n pot_rad_daily[dates], angstr_a, angstr_b)\r\n elif method == 'pot_rad_via_bc':\r\n # using data from Bristow-Campbell model\r\n globalrad = bristow_campbell(data_daily.tmin, data_daily.tmax, pot_rad_daily, bristcamp_a, bristcamp_c)\r\n\r\n globalrad_equal = globalrad.reindex(pot_rad.index, method='ffill') # hourly values (replicate daily mean value for each hour)\r\n pot_rad_daily_equal = pot_rad_daily.reindex(pot_rad.index, method='ffill')\r\n glob_disagg = pot_rad / pot_rad_daily_equal * globalrad_equal\r\n glob_disagg[glob_disagg < 1e-2] = 0.\r\n\r\n return glob_disagg", "response": "function to disaggregate radiation by day"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a DataFrame containing the total potential radiation for a specific location and time.", "response": "def potential_radiation(dates, lon, lat, timezone, terrain_slope=0, terrain_slope_azimuth=0,\r\n cloud_fraction=0, split=False):\r\n \"\"\"\r\n Calculate potential shortwave radiation for a specific location and time.\r\n\r\n This routine calculates global radiation as described in:\r\n Liston, G. E. and Elder, K. (2006): A Meteorological Distribution System for\r\n High-Resolution Terrestrial Modeling (MicroMet), J. Hydrometeorol., 7, 217\u2013234.\r\n\r\n Corrections for eccentricity are carried out following:\r\n Paltridge, G.W., Platt, C.M.R., 1976. Radiative processes in Meteorology and Climatology.\r\n Elsevier Scientific Publishing Company, Amsterdam, Oxford, New York.\r\n\r\n Parameters\r\n ----------\r\n dates : DatetimeIndex or array-like\r\n The dates for which potential radiation shall be calculated\r\n lon : float\r\n Longitude (degrees)\r\n lat : float\r\n Latitude (degrees)\r\n timezone : float\r\n Time zone\r\n terrain_slope : float, default 0\r\n Terrain slope as defined in Liston & Elder (2006) (eq. 12)\r\n terrain_slope_azimuth : float, default 0\r\n Terrain slope azimuth as defined in Liston & Elder (2006) (eq. 13)\r\n cloud_fraction : float, default 0\r\n Cloud fraction between 0 and 1\r\n split : boolean, default False\r\n If True, return a DataFrame containing direct and diffuse radiation,\r\n otherwise return a Series containing total radiation\r\n \"\"\"\r\n solar_constant = 1367.\r\n days_per_year = 365.25\r\n tropic_of_cancer = np.deg2rad(23.43697)\r\n solstice = 173.0\r\n\r\n dates = pd.DatetimeIndex(dates)\r\n dates_hour = np.array(dates.hour)\r\n dates_minute = np.array(dates.minute)\r\n day_of_year = np.array(dates.dayofyear)\r\n\r\n # compute solar decline in rad\r\n solar_decline = tropic_of_cancer * np.cos(2.0 * np.pi * (day_of_year - solstice) / days_per_year)\r\n\r\n # compute the sun hour angle in rad\r\n standard_meridian = timezone * 15.\r\n delta_lat_time = (lon - standard_meridian) * 24. / 360.\r\n hour_angle = np.pi * (((dates_hour + dates_minute / 60. + delta_lat_time) / 12.) - 1.)\r\n\r\n # get solar zenith angle\r\n cos_solar_zenith = (np.sin(solar_decline) * np.sin(np.deg2rad(lat))\r\n + np.cos(solar_decline) * np.cos(np.deg2rad(lat)) * np.cos(hour_angle))\r\n cos_solar_zenith = cos_solar_zenith.clip(min=0)\r\n solar_zenith_angle = np.arccos(cos_solar_zenith)\r\n\r\n # compute transmissivities for direct and diffus radiation using cloud fraction\r\n transmissivity_direct = (0.6 + 0.2 * cos_solar_zenith) * (1.0 - cloud_fraction)\r\n transmissivity_diffuse = (0.3 + 0.1 * cos_solar_zenith) * cloud_fraction\r\n\r\n # modify solar constant for eccentricity\r\n beta = 2. * np.pi * (day_of_year / days_per_year)\r\n radius_ratio = (1.00011 + 0.034221 * np.cos(beta) + 0.00128 * np.sin(beta)\r\n + 0.000719 * np.cos(2. * beta) + 0.000077 * np.sin(2 * beta))\r\n solar_constant_times_radius_ratio = solar_constant * radius_ratio\r\n\r\n mu = np.arcsin(np.cos(solar_decline) * np.sin(hour_angle) / np.sin(solar_zenith_angle))\r\n cosi = (np.cos(terrain_slope) * cos_solar_zenith\r\n + np.sin(terrain_slope) * np.sin(solar_zenith_angle) * np.cos(mu - terrain_slope_azimuth))\r\n\r\n # get total shortwave radiation\r\n direct_radiation = solar_constant_times_radius_ratio * transmissivity_direct * cosi\r\n diffuse_radiation = solar_constant_times_radius_ratio * transmissivity_diffuse * cos_solar_zenith\r\n direct_radiation = direct_radiation.clip(min=0)\r\n\r\n df = pd.DataFrame(index=dates, data=dict(direct=direct_radiation, diffuse=diffuse_radiation))\r\n\r\n if split:\r\n return df\r\n else:\r\n return df.direct + df.diffuse"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate potential shortwave radiation based on minimum and maximum temperature This routine calculates global radiation as described in: Bristow, Keith L., and Gaylon S. Campbell: On the relationship between incoming solar radiation and daily maximum and minimum temperature. Agricultural and forest meteorology 31.2 (1984): 159-166. Args: daily_data: time series (daily data) including at least minimum and maximum temeprature pot_rad_daily: mean potential daily radiation A: parameter A of the Bristow-Campbell model C: parameter C of the Bristow-Campbell model Returns: series of potential shortwave radiation", "response": "def bristow_campbell(tmin, tmax, pot_rad_daily, A, C):\r\n \"\"\"calculates potential shortwave radiation based on minimum and maximum temperature\r\n\r\n This routine calculates global radiation as described in:\r\n Bristow, Keith L., and Gaylon S. Campbell: On the relationship between\r\n incoming solar radiation and daily maximum and minimum temperature.\r\n Agricultural and forest meteorology 31.2 (1984): 159-166.\r\n\r\n Args:\r\n daily_data: time series (daily data) including at least minimum and maximum temeprature\r\n pot_rad_daily: mean potential daily radiation\r\n A: parameter A of the Bristow-Campbell model\r\n C: parameter C of the Bristow-Campbell model\r\n Returns:\r\n series of potential shortwave radiation\r\n \"\"\"\r\n\r\n assert tmin.index.equals(tmax.index)\r\n\r\n temp = pd.DataFrame(data=dict(tmin=tmin, tmax=tmax))\r\n temp = temp.reindex(pd.DatetimeIndex(start=temp.index[0], end=temp.index[-1], freq='D'))\r\n temp['tmin_nextday'] = temp.tmin\r\n temp.tmin_nextday.iloc[:-1] = temp.tmin.iloc[1:].values\r\n\r\n temp = temp.loc[tmin.index]\r\n pot_rad_daily = pot_rad_daily.loc[tmin.index]\r\n\r\n dT = temp.tmax - (temp.tmin + temp.tmin_nextday) / 2\r\n\r\n dT_m_avg = dT.groupby(dT.index.month).mean()\r\n B = 0.036 * np.exp(-0.154 * dT_m_avg[temp.index.month])\r\n B.index = temp.index\r\n\r\n if isinstance(A, pd.Series):\r\n months = temp.index.month\r\n A = A.loc[months].values\r\n C = C.loc[months].values\r\n\r\n transmissivity = A * (1 - np.exp(-B * dT**C))\r\n R0 = transmissivity * pot_rad_daily\r\n\r\n return R0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit_bristow_campbell_params(tmin, tmax, pot_rad_daily, obs_rad_daily):\r\n def bc_absbias(ac):\r\n return np.abs(np.mean(bristow_campbell(df.tmin, df.tmax, df.pot, ac[0], ac[1]) - df.obs))\r\n\r\n df = pd.DataFrame(data=dict(tmin=tmin, tmax=tmax, pot=pot_rad_daily, obs=obs_rad_daily)).dropna(how='any')\r\n res = scipy.optimize.minimize(bc_absbias, [0.75, 2.4]) # i.e. we minimize the absolute bias\r\n\r\n return res.x", "response": "Fits the A and C parameters for the Bristow & Campbell model using observed daily solar radiation and observed daily radiation."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef angstroem(ssd, day_length, pot_rad_daily, a, b):\r\n if isinstance(a, pd.Series):\r\n months = ssd.index.month\r\n a = a.loc[months].values\r\n b = b.loc[months].values\r\n\r\n glob_day = (a + b * ssd / day_length) * pot_rad_daily\r\n\r\n return glob_day", "response": "Calculates mean daily radiation from observed sunshine duration according to Angstroem model."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit_angstroem_params(ssd, day_length, pot_rad_daily, obs_rad_daily):\r\n df = pd.DataFrame(data=dict(ssd=ssd, day_length=day_length, pot=pot_rad_daily, obs=obs_rad_daily)).dropna(how='any')\r\n\r\n def angstroem_opt(x, a, b):\r\n return angstroem(x[0], x[1], x[2], a, b)\r\n\r\n x = np.array([df.ssd, df.day_length, df.pot])\r\n popt, pcov = scipy.optimize.curve_fit(angstroem_opt, x, df.obs, p0=[0.25, 0.75])\r\n\r\n return popt", "response": "Fits the a and b parameters for the Angstroem model using observed daily solar radiation and mean daily."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef register(name):\n def decorator(class_):\n if name in known_resolvers:\n raise ValueError('duplicate resolver name \"%s\"' % name)\n known_resolvers[name] = class_\n return decorator", "response": "Return a decorator that registers the decorated class as a\nCTYPE resolver with the given name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_resolver(order=None, options=None, modules=None):\n if not known_resolvers:\n from . import resolvers as carmen_resolvers\n modules = [carmen_resolvers] + (modules or [])\n for module in modules:\n for loader, name, _ in pkgutil.iter_modules(module.__path__):\n full_name = module.__name__ + '.' + name\n loader.find_module(full_name).load_module(full_name)\n if order is None:\n order = ('place', 'geocode', 'profile')\n else:\n order = tuple(order)\n if options is None:\n options = {}\n resolvers = []\n for resolver_name in order:\n if resolver_name not in known_resolvers:\n raise ValueError('unknown resolver name \"%s\"' % resolver_name)\n resolvers.append((\n resolver_name,\n known_resolvers[resolver_name](**options.get(resolver_name, {}))))\n return ResolverCollection(resolvers)", "response": "Return a location resolver."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load_locations(self, location_file=None):\n if location_file is None:\n contents = pkgutil.get_data(__package__, 'data/locations.json')\n contents_string = contents.decode(\"ascii\")\n locations = contents_string.split('\\n')\n else:\n from .cli import open_file\n with open_file(location_file, 'rb') as input:\n locations = input.readlines()\n \n for location_string in locations:\n if location_string.strip():\n location = Location(known=True, **json.loads(location_string))\n self.location_id_to_location[location.id] = location\n self.add_location(location)", "response": "Load locations into this resolver from the given location_file which should contain one JSON object per line containing a location."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a tuple containing a canonicalized version of this location s country state county and city names.", "response": "def canonical(self):\n \"\"\"Return a tuple containing a canonicalized version of this\n location's country, state, county, and city names.\"\"\"\n try:\n return tuple(map(lambda x: x.lower(), self.name()))\n except:\n return tuple([x.lower() for x in self.name()])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a tuple containing this location s country state county and city names.", "response": "def name(self):\n \"\"\"Return a tuple containing this location's country, state,\n county, and city names.\"\"\"\n try:\n return tuple(\n getattr(self, x) if getattr(self, x) else u''\n for x in ('country', 'state', 'county', 'city'))\n except:\n return tuple(\n getattr(self, x) if getattr(self, x) else ''\n for x in ('country', 'state', 'county', 'city'))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a location representing the administrative unit above the one represented by this location.", "response": "def parent(self):\n \"\"\"Return a location representing the administrative unit above\n the one represented by this location.\"\"\"\n if self.city:\n return Location(\n country=self.country, state=self.state, county=self.county)\n if self.county:\n return Location(country=self.country, state=self.state)\n if self.state:\n return Location(country=self.country)\n return Location()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef disaggregate_humidity(data_daily, method='equal', temp=None,\n a0=None, a1=None, kr=None,\n month_hour_precip_mean=None, preserve_daily_mean=False):\n \"\"\"general function for humidity disaggregation\n\n Args:\n daily_data: daily values\n method: keyword specifying the disaggregation method to be used\n temp: hourly temperature time series (necessary for some methods)\n kr: parameter for linear_dewpoint_variation method (6 or 12)\n month_hour_precip_mean: [month, hour, precip(y/n)] categorical mean values\n preserve_daily_mean: if True, correct the daily mean values of the disaggregated\n data with the observed daily means.\n\n Returns:\n Disaggregated hourly values of relative humidity.\n \"\"\"\n assert method in ('equal',\n 'minimal',\n 'dewpoint_regression',\n 'min_max',\n 'linear_dewpoint_variation',\n 'month_hour_precip_mean'), 'Invalid option'\n\n if method == 'equal':\n hum_disagg = melodist.distribute_equally(data_daily.hum)\n elif method in ('minimal', 'dewpoint_regression', 'linear_dewpoint_variation'):\n if method == 'minimal':\n a0 = 0\n a1 = 1\n\n assert a0 is not None and a1 is not None, 'a0 and a1 must be specified'\n tdew_daily = a0 + a1 * data_daily.tmin\n\n tdew = melodist.distribute_equally(tdew_daily)\n\n if method == 'linear_dewpoint_variation':\n assert kr is not None, 'kr must be specified'\n assert kr in (6, 12), 'kr must be 6 or 12'\n tdew_delta = 0.5 * np.sin((temp.index.hour + 1) * np.pi / kr - 3. * np.pi / 4.) # eq. (21) from Debele et al. (2007)\n\n tdew_nextday = tdew.shift(-24)\n tdew_nextday.iloc[-24:] = tdew.iloc[-24:] # copy the last day\n\n # eq. (20) from Debele et al. (2007):\n # (corrected - the equation is wrong both in Debele et al. (2007) and Bregaglio et al. (2010) - it should\n # be (T_dp,day)_(d+1) - (T_dp,day)_d instead of the other way around)\n tdew += temp.index.hour / 24. * (tdew_nextday - tdew) + tdew_delta\n\n sat_vap_press_tdew = util.vapor_pressure(tdew, 100)\n sat_vap_press_t = util.vapor_pressure(temp, 100)\n hum_disagg = pd.Series(index=temp.index, data=100 * sat_vap_press_tdew / sat_vap_press_t)\n elif method == 'min_max':\n assert 'hum_min' in data_daily.columns and 'hum_max' in data_daily.columns, \\\n 'Minimum and maximum humidity must be present in data frame'\n\n hmin = melodist.distribute_equally(data_daily.hum_min)\n hmax = melodist.distribute_equally(data_daily.hum_max)\n tmin = melodist.distribute_equally(data_daily.tmin)\n tmax = melodist.distribute_equally(data_daily.tmax)\n\n hum_disagg = hmax + (temp - tmin) / (tmax - tmin) * (hmin - hmax)\n elif method == 'month_hour_precip_mean':\n assert month_hour_precip_mean is not None\n\n precip_equal = melodist.distribute_equally(data_daily.precip) # daily precipitation equally distributed to hourly values\n hum_disagg = pd.Series(index=precip_equal.index)\n locs = list(zip(hum_disagg.index.month, hum_disagg.index.hour, precip_equal > 0))\n hum_disagg[:] = month_hour_precip_mean.loc[locs].values\n\n if preserve_daily_mean:\n daily_mean_df = pd.DataFrame(data=dict(obs=data_daily.hum, disagg=hum_disagg.resample('D').mean()))\n bias = melodist.util.distribute_equally(daily_mean_df.disagg - daily_mean_df.obs)\n bias = bias.fillna(0)\n hum_disagg -= bias\n\n return hum_disagg.clip(0, 100)", "response": "disaggregation of daily values of relative humidity."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cosine_function(x, a, b, t_shift):\n\n mean_wind, t = x\n return a * mean_wind * np.cos(np.pi * (t - t_shift) / 12) + b * mean_wind", "response": "This function calculates the cosine function of the windspeed accroding to the cosine function\nCTYPE"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfit a cosine function to observed hourly windspeed data AttributeNames is a list of dicts where each dictionary contains the key and the value of the observed hourly windspeed is the observed hourly windspeed of the wind.", "response": "def fit_cosine_function(wind):\n \"\"\"fits a cosine function to observed hourly windspeed data\n\n Args:\n wind: observed hourly windspeed data\n \n Returns:\n parameters needed to generate diurnal features of windspeed using a cosine function\n \"\"\"\n wind_daily = wind.groupby(wind.index.date).mean()\n wind_daily_hourly = pd.Series(index=wind.index, data=wind_daily.loc[wind.index.date].values) # daily values evenly distributed over the hours\n\n df = pd.DataFrame(data=dict(daily=wind_daily_hourly, hourly=wind)).dropna(how='any')\n x = np.array([df.daily, df.index.hour])\n popt, pcov = scipy.optimize.curve_fit(_cosine_function, x, df.hourly)\n\n return popt"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads smet data and returns the data in required dataformat.", "response": "def read_smet(filename, mode):\n \"\"\"Reads smet data and returns the data in required dataformat (pd df)\n\n See https://models.slf.ch/docserver/meteoio/SMET_specifications.pdf\n for further details on the specifications of this file format.\n\n Parameters\n ----\n filename : SMET file to read\n mode : \"d\" for daily and \"h\" for hourly input\n\n Returns\n ----\n [header, data]\n header: header as dict\n data : data as pd df\n \"\"\"\n\n # dictionary\n # based on smet spec V.1.1 and self defined\n # daily data\n dict_d = {'TA': 'tmean',\n 'TMAX': 'tmax', # no spec\n 'TMIN': 'tmin', # no spec\n 'PSUM': 'precip',\n 'ISWR': 'glob', # no spec\n 'RH': 'hum',\n 'VW': 'wind'}\n\n # hourly data\n dict_h = {'TA': 'temp',\n 'PSUM': 'precip',\n 'ISWR': 'glob', # no spec\n 'RH': 'hum',\n 'VW': 'wind'}\n\n with open(filename) as f:\n in_header = False\n data_start = None\n header = collections.OrderedDict()\n\n for line_num, line in enumerate(f):\n\n if line.strip() == '[HEADER]':\n in_header = True\n continue\n elif line.strip() == '[DATA]':\n data_start = line_num + 1\n break\n\n if in_header:\n line_split = line.split('=')\n k = line_split[0].strip()\n v = line_split[1].strip()\n header[k] = v\n\n # get column names\n columns = header['fields'].split()\n multiplier = [float(x) for x in header['units_multiplier'].split()][1:]\n\n data = pd.read_table(\n filename,\n sep=r'\\s+',\n na_values=[-999],\n skiprows=data_start,\n names=columns,\n index_col='timestamp',\n parse_dates=True,\n )\n\n data = data*multiplier\n\n del data.index.name\n\n # rename columns\n if mode == \"d\":\n data = data.rename(columns=dict_d)\n if mode == \"h\":\n data = data.rename(columns=dict_h)\n\n return header, data"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread the DWD file and returns the data in required dataformat", "response": "def read_dwd(filename, metadata, mode=\"d\", skip_last=True):\n \"\"\"Reads dwd (German Weather Service) data and returns the data in required\n dataformat (pd df)\n\n Parameters\n ----\n filename : DWD file to read (full path) / list of hourly files (RR+TU+FF) \n metadata : corresponding DWD metadata file to read\n mode : \"d\" for daily and \"h\" for hourly input\n skip_last : boolen, skips last line due to file format\n\n Returns\n ----\n [header, data]\n header: header as dict\n data : data as pd df\n \"\"\"\n\n def read_single_dwd(filename, metadata, mode, skip_last):\n # Param names {'DWD':'dissag_def'}\n dict_d = {'LUFTTEMPERATUR': 'tmean',\n 'LUFTTEMPERATUR_MINIMUM': 'tmin', # no spec\n 'LUFTTEMPERATUR_MAXIMUM': 'tmax', # no spec\n 'NIEDERSCHLAGSHOEHE': 'precip',\n 'GLOBAL_KW_J': 'glob', # no spec\n 'REL_FEUCHTE': 'hum',\n 'WINDGESCHWINDIGKEIT': 'wind',\n 'SONNENSCHEINDAUER': 'sun_h'}\n\n # ---read meta------------------\n meta = pd.read_csv(\n metadata,\n sep=';'\n )\n\n # remove whitespace from header columns\n meta.rename(columns=lambda x: x.strip(), inplace=True)\n\n header = {\"Stations_id\": meta.Stations_id[meta.last_valid_index()],\n \"Stationsname\": meta.Stationsname[meta.last_valid_index()],\n # workaround for colnames with . (Geogr.Breite)\n \"Breite\": meta.iloc[meta.last_valid_index(), 2], # DezDeg\n \"Laenge\": meta.iloc[meta.last_valid_index(), 3] # DezDeg\n }\n\n # ---read data------------------\n if skip_last is not None:\n num_lines = sum(1 for line in open(filename))\n skip_last = [num_lines-1]\n\n # hourly data must be parsed by custom definition\n if mode == \"d\":\n data = pd.read_csv(\n filename,\n sep=';',\n na_values='-999',\n index_col=' MESS_DATUM',\n parse_dates=True,\n skiprows=skip_last\n )\n\n # hourly data must be parsed by custom definition\n if mode == \"h\":\n def date_parser(date_time):\n hour = date_time[8:10]\n day = date_time[6:8]\n month = date_time[4:6]\n year = date_time[0:4]\n minute = '00'\n sec = '00'\n return pd.Timestamp('%s-%s-%s %s:%s:%s' % (year, month, day, hour, minute, sec))\n\n data = pd.read_csv(\n filename,\n sep=';',\n na_values='-999',\n index_col=' MESS_DATUM',\n date_parser=date_parser,\n skiprows=skip_last\n )\n\n # remove whitespace from header columns\n data.rename(columns=lambda x: x.strip(), inplace=True)\n\n # rename to dissag definition\n data = data.rename(columns=dict_d)\n # get colums which are not defined\n drop = [col for col in data.columns if col not in dict_d.values()]\n # delete columns\n data = data.drop(drop, axis=1)\n\n # convert temperatures to Kelvin (+273.15)\n if 'tmin' in data.columns:\n data[\"tmin\"] = data[\"tmin\"] + 273.15\n if 'tmax' in data.columns:\n data[\"tmax\"] = data[\"tmax\"] + 273.15\n if 'tmean' in data.columns:\n data[\"tmean\"] = data[\"tmean\"] + 273.15\n if 'temp' in data.columns:\n data[\"temp\"] = data[\"temp\"] + 273.15\n\n return header, data\n\n if type(filename) == list:\n i = 1\n for file in filename:\n header, data_h = read_single_dwd(file, metadata, mode, skip_last)\n\n if i == 1:\n data = data_h\n else:\n data = data.join(data_h, how='outer')\n i += 1\n\n else:\n header, data = read_single_dwd(filename, metadata, mode, skip_last)\n\n return header, data"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting a smet file", "response": "def write_smet(filename, data, metadata, nodata_value=-999, mode='h', check_nan=True):\n \"\"\"writes smet files\n\n Parameters\n ----\n filename : filename/loction of output\n data : data to write as pandas df\n metadata: header to write input as dict\n nodata_value: Nodata Value to write/use\n mode: defines if to write daily (\"d\") or continuos data (default 'h')\n check_nan: will check if only nans in data and if true will not write this colums (default True)\n \"\"\"\n\n # dictionary\n # based on smet spec V.1.1 and selfdefined\n # daily data\n dict_d= {'tmean':'TA',\n 'tmin':'TMAX', #no spec\n 'tmax':'TMIN', #no spec\n 'precip':'PSUM',\n 'glob':'ISWR', #no spec\n 'hum':'RH',\n 'wind':'VW'\n }\n\n #hourly data\n dict_h= {'temp':'TA',\n 'precip':'PSUM',\n 'glob':'ISWR', #no spec\n 'hum':'RH',\n 'wind':'VW'\n }\n \n #rename columns\n if mode == \"d\":\n data = data.rename(columns=dict_d)\n if mode == \"h\":\n data = data.rename(columns=dict_h)\n\n if check_nan: \n #get all colums with data\n datas_in = data.sum().dropna().to_frame().T\n #get colums with no datas\n drop = [data_nan for data_nan in data.columns if data_nan not in datas_in] \n #delete columns\n data = data.drop(drop, axis=1)\n \n with open(filename, 'w') as f:\n\n #preparing data\n #converte date_times to SMET timestamps\n if mode == \"d\":\n t = '%Y-%m-%dT00:00'\n if mode == \"h\":\n t = '%Y-%m-%dT%H:%M'\n\n data['timestamp'] = [d.strftime(t) for d in data.index]\n \n cols = data.columns.tolist()\n cols = cols[-1:] + cols[:-1]\n data = data[cols]\n\n\n #metadatas update\n metadata['fields'] = ' '.join(data.columns)\n metadata[\"units_multiplier\"] = len(metadata['fields'].split())*\"1 \"\n\n #writing data\n #metadata\n f.write('SMET 1.1 ASCII\\n')\n f.write('[HEADER]\\n')\n\n for k, v in metadata.items():\n f.write('{} = {}\\n'.format(k, v))\n\n #data\n f.write('[DATA]\\n')\n\n data_str = data.fillna(nodata_value).to_string(\n header=False,\n index=False,\n float_format=lambda x: '{:.2f}'.format(x),\n )\n\n f.write(data_str)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading a single KNMI s meteorological time series file and returns pandas data frame including time series", "response": "def read_single_knmi_file(filename):\n \"\"\"reads a single file of KNMI's meteorological time series\n\n data availability: www.knmi.nl/nederland-nu/klimatologie/uurgegevens\n\n Args:\n filename: the file to be opened\n\n Returns:\n pandas data frame including time series\n \"\"\"\n hourly_data_obs_raw = pd.read_csv(\n filename,\n parse_dates=[['YYYYMMDD', 'HH']],\n date_parser=lambda yyyymmdd, hh: pd.datetime(int(str(yyyymmdd)[0:4]),\n int(str(yyyymmdd)[4:6]),\n int(str(yyyymmdd)[6:8]),\n int(hh) - 1),\n skiprows=31,\n skipinitialspace=True,\n na_values='',\n keep_date_col=True,\n )\n\n hourly_data_obs_raw.index = hourly_data_obs_raw['YYYYMMDD_HH']\n hourly_data_obs_raw.index = hourly_data_obs_raw.index + pd.Timedelta(hours=1)\n\n columns_hourly = ['temp', 'precip', 'glob', 'hum', 'wind', 'ssd']\n\n hourly_data_obs = pd.DataFrame(\n index=hourly_data_obs_raw.index,\n columns=columns_hourly,\n data=dict(\n temp=hourly_data_obs_raw['T'] / 10 + 273.15,\n precip=hourly_data_obs_raw['RH'] / 10,\n glob=hourly_data_obs_raw['Q'] * 10000 / 3600.,\n hum=hourly_data_obs_raw['U'],\n wind=hourly_data_obs_raw['FH'] / 10,\n ssd=hourly_data_obs_raw['SQ'] * 6,\n ),\n )\n # remove negative values\n negative_values = hourly_data_obs['precip'] < 0.0\n hourly_data_obs.loc[negative_values, 'precip'] = 0.0\n return hourly_data_obs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads files from a directory and merges the time series with the time series.", "response": "def read_knmi_dataset(directory):\n \"\"\"Reads files from a directory and merges the time series\n\n Please note: For each station, a separate directory must be provided!\n data availability: www.knmi.nl/nederland-nu/klimatologie/uurgegevens\n\n Args:\n directory: directory including the files\n\n Returns:\n pandas data frame including time series\n \"\"\"\n filemask = '%s*.txt' % directory\n filelist = glob.glob(filemask)\n\n columns_hourly = ['temp', 'precip', 'glob', 'hum', 'wind', 'ssd']\n ts = pd.DataFrame(columns=columns_hourly)\n\n first_call = True\n for file_i in filelist:\n print(file_i)\n current = read_single_knmi_file(file_i)\n if(first_call):\n ts = current\n first_call = False\n else:\n ts = pd.concat([ts, current])\n return ts"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate sunrise sunset for each day.", "response": "def calc_sun_times(self):\n \"\"\"\n Computes the times of sunrise, solar noon, and sunset for each day.\n \"\"\"\n\n self.sun_times = melodist.util.get_sun_times(self.data_daily.index, self.lon, self.lat, self.timezone)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disaggregate_wind(self, method='equal'):\n self.data_disagg.wind = melodist.disaggregate_wind(self.data_daily.wind, method=method, **self.statistics.wind)", "response": "Disaggregate wind speed.\n\n Parameters\n ----------\n method : str, optional\n Disaggregation method.\n\n ``equal``\n Mean daily wind speed is duplicated for the 24 hours of the day. (Default)\n\n ``cosine``\n Distributes daily mean wind speed using a cosine function derived from hourly\n observations.\n\n ``random``\n Draws random numbers to distribute wind speed (usually not conserving the\n daily average)."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disaggregate_temperature(self, method='sine_min_max', min_max_time='fix', mod_nighttime=False):\n self.data_disagg.temp = melodist.disaggregate_temperature(\n self.data_daily,\n method=method,\n min_max_time=min_max_time,\n max_delta=self.statistics.temp.max_delta,\n mean_course=self.statistics.temp.mean_course,\n sun_times=self.sun_times,\n mod_nighttime=mod_nighttime\n )", "response": "Disaggregate air temperature.\n\n Parameters\n ----------\n method : str, optional\n Disaggregation method.\n\n ``sine_min_max``\n Hourly temperatures follow a sine function preserving daily minimum\n and maximum values. (Default)\n\n ``sine_mean``\n Hourly temperatures follow a sine function preserving the daily mean\n value and the diurnal temperature range.\n\n ``sine``\n Same as ``sine_min_max``.\n\n ``mean_course_min_max``\n Hourly temperatures follow an observed average course (calculated for each month),\n preserving daily minimum and maximum values.\n\n ``mean_course_mean``\n Hourly temperatures follow an observed average course (calculated for each month),\n preserving the daily mean value and the diurnal temperature range.\n\n min_max_time : str, optional\n Method to determine the time of minimum and maximum temperature.\n\n ``fix``:\n Minimum/maximum temperature are assumed to occur at 07:00/14:00 local time.\n\n ``sun_loc``:\n Minimum/maximum temperature are assumed to occur at sunrise / solar noon + 2 h.\n\n ``sun_loc_shift``:\n Minimum/maximum temperature are assumed to occur at sunrise / solar noon + monthly mean shift.\n\n mod_nighttime : bool, optional\n Use linear interpolation between minimum and maximum temperature."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef disaggregate_radiation(self, method='pot_rad', pot_rad=None):\n if self.sun_times is None:\n self.calc_sun_times()\n\n if pot_rad is None and method != 'mean_course':\n pot_rad = melodist.potential_radiation(self.data_disagg.index, self.lon, self.lat, self.timezone)\n\n self.data_disagg.glob = melodist.disaggregate_radiation(\n self.data_daily,\n sun_times=self.sun_times,\n pot_rad=pot_rad,\n method=method,\n angstr_a=self.statistics.glob.angstroem.a,\n angstr_b=self.statistics.glob.angstroem.b,\n bristcamp_a=self.statistics.glob.bristcamp.a,\n bristcamp_c=self.statistics.glob.bristcamp.c,\n mean_course=self.statistics.glob.mean_course\n )", "response": "Disaggregate solar radiation.\n\n Parameters\n ----------\n method : str, optional\n Disaggregation method.\n\n ``pot_rad``\n Calculates potential clear-sky hourly radiation and scales it according to the\n mean daily radiation. (Default)\n\n ``pot_rad_via_ssd``\n Calculates potential clear-sky hourly radiation and scales it according to the\n observed daily sunshine duration.\n\n ``pot_rad_via_bc``\n Calculates potential clear-sky hourly radiation and scales it according to daily\n minimum and maximum temperature.\n\n ``mean_course``\n Hourly radiation follows an observed average course (calculated for each month).\n\n pot_rad : Series, optional\n Hourly values of potential solar radiation. If ``None``, calculated internally."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef interpolate(self, column_hours, method='linear', limit=24, limit_direction='both', **kwargs):\n kwargs = dict(kwargs, method=method, limit=limit, limit_direction=limit_direction)\n data = melodist.util.prepare_interpolation_data(self.data_daily, column_hours)\n return data.interpolate(**kwargs)", "response": "Wrapper function for pandas. Series. interpolate that can be used to disaggregate values for the daily calendar."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef head(self, n=10, by=None, **kwargs):\n col, id_col = self._query_helper(by=by)\n\n select = (\"SELECT %s FROM %s ORDER BY %s ASC LIMIT %d\" %\n (col, self.table.name, id_col, n))\n\n return self._db.query(select, **kwargs)", "response": "Get the first n entries for a given Table or Column."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef last(self, n=10, by=None, **kwargs):\n return self.tail(n=n, by=by, **kwargs)", "response": "Return the last n entries in the set."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef where(self, where_string, **kwargs):\n col, id_col = self._query_helper(by=None)\n\n where_string = str(where_string) # Coerce here, for .__contains___\n where_operators = [\"=\", \">\", \"<\", \"LIKE\", \"like\"]\n if np.any([where_string.__contains__(w) for w in where_operators]):\n select = (\"SELECT %s FROM %s WHERE %s\" %\n (col, self.table.name, where_string))\n else:\n select = (\"SELECT %s FROM %s WHERE %s = %s\" %\n (col, self.table.name, id_col, where_string))\n\n return self._db.query(select, **kwargs)", "response": "Select from a given Table or Column with the specified WHERE clause."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query(self, sql_query, return_as=\"dataframe\"):\n if isinstance(sql_query, str):\n pass\n elif isinstance(sql_query, unicode):\n sql_query = str(sql_query)\n else:\n raise QueryDbError(\"query() requires a str or unicode input.\")\n\n query = sqlalchemy.sql.text(sql_query)\n\n if return_as.upper() in [\"DF\", \"DATAFRAME\"]:\n return self._to_df(query, self._engine)\n elif return_as.upper() in [\"RESULT\", \"RESULTPROXY\"]:\n with self._engine.connect() as conn:\n result = conn.execute(query)\n return result\n else:\n raise QueryDbError(\"Other return types not implemented.\")", "response": "Executes a raw SQL query against the SQL database and returns the result as a DataFrame or None."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _set_metadata(self):\n meta = QueryDbMeta()\n with self._engine.connect() as conn:\n meta.bind = conn\n meta.reflect()\n self._meta = meta\n\n # Set an inspect attribute, whose subattributes\n # return individual tables / columns. Tables and columns\n # are special classes with .last() and other convenience methods\n self.inspect = QueryDbAttributes()\n for table in self._meta.tables:\n setattr(self.inspect, table,\n QueryDbOrm(self._meta.tables[table], self))\n\n table_attr = getattr(self.inspect, table)\n table_cols = table_attr.table.columns\n\n for col in table_cols.keys():\n setattr(table_attr, col,\n QueryDbOrm(table_cols[col], self))\n\n # Finally add some summary info:\n # Table name\n # Primary Key item or list\n # N of Cols\n # Distinct Col Values (class so NVARCHAR(20) and NVARCHAR(30) are not different)\n primary_keys = table_attr.table.primary_key.columns.keys()\n self._summary_info.append((\n table,\n primary_keys[0] if len(primary_keys) == 1 else primary_keys,\n len(table_cols),\n len(set([x.type.__class__ for x in table_cols.values()])),\n ))", "response": "Set the metadata of the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_shift_by_data(temp_hourly, lon, lat, time_zone):\n '''function to get max temp shift (monthly) by hourly data\n \n Parameters\n ----\n hourly_data_obs : observed hourly data \n lat : latitude in DezDeg\n lon : longitude in DezDeg\n time_zone: timezone\n '''\n daily_index = temp_hourly.resample('D').mean().index\n sun_times = melodist.util.get_sun_times(daily_index, lon, lat, time_zone)\n\n idxmax = temp_hourly.groupby(temp_hourly.index.date).idxmax()\n idxmax.index = pd.to_datetime(idxmax.index)\n max_temp_hour_obs = idxmax.dropna().apply(lambda d: d.hour)\n max_temp_hour_pot = sun_times.sunnoon\n max_delta = max_temp_hour_obs - max_temp_hour_pot\n mean_monthly_delta = max_delta.groupby(max_delta.index.month).mean()\n\n return mean_monthly_delta", "response": "function to get max temp shift by observed hourly data \n"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the daily values in a single hour by equally distributing the daily values.", "response": "def distribute_equally(daily_data, divide=False):\n \"\"\"Obtains hourly values by equally distributing the daily values.\n\n Args:\n daily_data: daily values\n divide: if True, divide resulting values by the number of hours in\n order to preserve the daily sum (required e.g. for precipitation).\n\n Returns:\n Equally distributed hourly values.\n \"\"\"\n\n index = hourly_index(daily_data.index)\n hourly_data = daily_data.reindex(index)\n hourly_data = hourly_data.groupby(hourly_data.index.day).transform(\n lambda x: x.fillna(method='ffill', limit=23))\n\n if divide:\n hourly_data /= 24\n\n return hourly_data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating vapor pressure from temperature and humidity after Sonntag 1990.", "response": "def vapor_pressure(temp, hum):\n \"\"\"\n Calculates vapor pressure from temperature and humidity after Sonntag (1990).\n\n Args:\n temp: temperature values\n hum: humidity value(s). Can be scalar (e.g. for calculating saturation vapor pressure).\n\n Returns:\n Vapor pressure in hPa.\n \"\"\"\n\n if np.isscalar(hum):\n hum = np.zeros(temp.shape) + hum\n\n assert(temp.shape == hum.shape)\n\n positives = np.array(temp >= 273.15)\n vap_press = np.zeros(temp.shape) * np.nan\n vap_press[positives] = 6.112 * np.exp((17.62 * (temp[positives] - 273.15)) / (243.12 + (temp[positives] - 273.15))) * hum[positives] / 100.\n vap_press[~positives] = 6.112 * np.exp((22.46 * (temp[~positives] - 273.15)) / (272.62 + (temp[~positives] - 273.15))) * hum[~positives] / 100.\n\n return vap_press"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dewpoint_temperature(temp, hum):\n assert(temp.shape == hum.shape)\n\n vap_press = vapor_pressure(temp, hum)\n\n positives = np.array(temp >= 273.15)\n dewpoint_temp = temp.copy() * np.nan\n dewpoint_temp[positives] = 243.12 * np.log(vap_press[positives] / 6.112) / (17.62 - np.log(vap_press[positives] / 6.112))\n dewpoint_temp[~positives] = 272.62 * np.log(vap_press[~positives] / 6.112) / (22.46 - np.log(vap_press[~positives] / 6.112))\n\n return dewpoint_temp + 273.15", "response": "computes the dewpoint temperature in the Konch tree"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef linregress(x, y, return_stats=False):\n a1, a0, r_value, p_value, stderr = scipy.stats.linregress(x, y)\n\n retval = a1, a0\n if return_stats:\n retval += r_value, p_value, stderr\n\n return retval", "response": "linear regression calculation of x and y"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the times of sunrise sunnoon and sunset for each day.", "response": "def get_sun_times(dates, lon, lat, time_zone):\n \"\"\"Computes the times of sunrise, solar noon, and sunset for each day.\n\n Parameters\n ----\n dates: datetime\n lat : latitude in DecDeg\n lon : longitude in DecDeg\n time_zone : timezone\n \n\n Returns\n ----\n DataFrame: [sunrise, sunnoon, sunset, day length] in dec hours\n \"\"\"\n\n df = pd.DataFrame(index=dates, columns=['sunrise', 'sunnoon', 'sunset', 'daylength'])\n\n doy = np.array([(d - d.replace(day=1, month=1)).days + 1 for d in df.index]) # day of year\n\n # Day angle and declination after Bourges (1985):\n day_angle_b = np.deg2rad((360. / 365.25) * (doy - 79.346))\n \n declination = np.deg2rad(\n 0.3723 + 23.2567 * np.sin(day_angle_b) - 0.7580 * np.cos(day_angle_b)\n + 0.1149 * np.sin(2*day_angle_b) + 0.3656 * np.cos(2*day_angle_b)\n - 0.1712 * np.sin(3*day_angle_b) + 0.0201 * np.cos(3*day_angle_b)\n )\n \n # Equation of time with day angle after Spencer (1971):\n day_angle_s = 2 * np.pi * (doy - 1) / 365.\n eq_time = 12. / np.pi * (\n 0.000075 +\n 0.001868 * np.cos( day_angle_s) - 0.032077 * np.sin( day_angle_s) -\n 0.014615 * np.cos(2*day_angle_s) - 0.040849 * np.sin(2*day_angle_s)\n )\n \n #\n standard_meridian = time_zone * 15.\n delta_lat_time = (lon - standard_meridian) * 24. / 360.\n \n omega_nul_arg = -np.tan(np.deg2rad(lat)) * np.tan(declination)\n omega_nul = np.arccos(omega_nul_arg)\n sunrise = 12. * (1. - (omega_nul) / np.pi) - delta_lat_time - eq_time\n sunset = 12. * (1. + (omega_nul) / np.pi) - delta_lat_time - eq_time\n\n # as an approximation, solar noon is independent of the below mentioned\n # cases:\n sunnoon = 12. * (1.) - delta_lat_time - eq_time\n \n # $kf 2015-11-13: special case midnight sun and polar night\n # CASE 1: MIDNIGHT SUN\n # set sunrise and sunset to values that would yield the maximum day\n # length even though this a crude assumption\n pos = omega_nul_arg < -1\n sunrise[pos] = sunnoon[pos] - 12\n sunset[pos] = sunnoon[pos] + 12\n\n # CASE 2: POLAR NIGHT\n # set sunrise and sunset to values that would yield the minmum day\n # length even though this a crude assumption\n pos = omega_nul_arg > 1\n sunrise[pos] = sunnoon[pos]\n sunset[pos] = sunnoon[pos]\n\n daylength = sunset - sunrise\n \n # adjust if required\n sunrise[sunrise < 0] += 24\n sunset[sunset > 24] -= 24\n\n df.sunrise = sunrise\n df.sunnoon = sunnoon\n df.sunset = sunset\n df.daylength = daylength\n\n return df"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if a given dataframe contains gaps and returns the number of gaps that occur in the screen.", "response": "def detect_gaps(dataframe, timestep, print_all=False, print_max=5, verbose=True):\n \"\"\"checks if a given dataframe contains gaps and returns the number of gaps\n\n This funtion checks if a dataframe contains any gaps for a given temporal\n resolution that needs to be specified in seconds. The number of gaps\n detected in the dataframe is returned.\n\n Args:\n dataframe: A pandas dataframe object with index defined as datetime\n timestep (int): The temporal resolution of the time series in seconds\n (e.g., 86400 for daily values)\n print_all (bool, opt): Lists every gap on the screen\n print_mx (int, opt): The maximum number of gaps listed on the screen in\n order to avoid a decrease in performance if numerous gaps occur\n verbose (bool, opt): Enables/disables output to the screen\n\n Returns:\n The number of gaps as integer. Negative values indicate errors.\n \"\"\"\n gcount = 0\n msg_counter = 0\n warning_printed = False\n try:\n n = len(dataframe.index)\n except:\n print('Error: Invalid dataframe.')\n return -1\n for i in range(0, n):\n if(i > 0):\n time_diff = dataframe.index[i] - dataframe.index[i-1]\n if(time_diff.delta/1E9 != timestep):\n gcount += 1\n if print_all or (msg_counter <= print_max - 1):\n if verbose:\n print('Warning: Gap in time series found between %s and %s' % (dataframe.index[i-1], dataframe.index[i]))\n msg_counter += 1\n if msg_counter == print_max and verbose and not warning_printed:\n print('Waring: Only the first %i gaps have been listed. Try to increase print_max parameter to show more details.' % msg_counter)\n warning_printed = True\n if verbose:\n print('%i gaps found in total.' % (gcount))\n return gcount"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntruncating a given dataframe to full days only This funtion truncates a given pandas dataframe (time series) to full days only, thus dropping leading and tailing hours of incomplete days. Please note that this methodology only applies to hourly time series. Args: dataframe: A pandas dataframe object with index defined as datetime shift (unsigned int, opt): First hour of daily recordings. For daily recordings of precipitation gages, 8 would be the first hour of the subsequent day of recordings since daily totals are usually recorded at 7. Omit defining this parameter if you intend to pertain recordings to 0-23h. Returns: A dataframe with full days only.", "response": "def drop_incomplete_days(dataframe, shift=0):\n \"\"\"truncates a given dataframe to full days only\n\n This funtion truncates a given pandas dataframe (time series) to full days\n only, thus dropping leading and tailing hours of incomplete days. Please\n note that this methodology only applies to hourly time series.\n\n Args:\n dataframe: A pandas dataframe object with index defined as datetime\n shift (unsigned int, opt): First hour of daily recordings. For daily\n recordings of precipitation gages, 8 would be the first hour of\n the subsequent day of recordings since daily totals are\n usually recorded at 7. Omit defining this parameter if you intend\n to pertain recordings to 0-23h.\n\n Returns:\n A dataframe with full days only.\n \"\"\"\n dropped = 0\n if shift > 23 or shift < 0:\n print(\"Invalid shift parameter setting! Using defaults.\")\n shift = 0\n first = shift\n last = first - 1\n if last < 0:\n last += 24\n try:\n # todo: move this checks to a separate function\n n = len(dataframe.index)\n except:\n print('Error: Invalid dataframe.')\n return dataframe\n \n delete = list() \n \n # drop heading lines if required\n for i in range(0, n):\n if dataframe.index.hour[i] == first and dataframe.index.minute[i] == 0:\n break\n else:\n delete.append(i)\n dropped += 1\n\n # drop tailing lines if required\n for i in range(n-1, 0, -1):\n if dataframe.index.hour[i] == last and dataframe.index.minute[i] == 0:\n break\n else:\n delete.append(i)\n dropped += 1\n # print(\"The following rows have been dropped (%i in total):\" % dropped)\n # print(delete)\n return dataframe.drop(dataframe.index[[delete]])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef daily_from_hourly(df):\n\n df_daily = pd.DataFrame()\n\n if 'temp' in df:\n df_daily['temp'] = df.temp.resample('D').mean()\n df_daily['tmin'] = df.temp.groupby(df.temp.index.date).min()\n df_daily['tmax'] = df.temp.groupby(df.temp.index.date).max()\n\n if 'precip' in df:\n df_daily['precip'] = df.precip.resample('D').sum()\n\n if 'glob' in df:\n df_daily['glob'] = df.glob.resample('D').mean()\n\n if 'hum' in df:\n df_daily['hum'] = df.hum.resample('D').mean()\n\n if 'hum' in df:\n df_daily['hum_min'] = df.hum.groupby(df.hum.index.date).min()\n\n if 'hum' in df:\n df_daily['hum_max'] = df.hum.groupby(df.hum.index.date).max()\n\n if 'wind' in df:\n df_daily['wind'] = df.wind.resample('D').mean()\n\n if 'ssd' in df:\n df_daily['ssd'] = df.ssd.resample('D').sum() / 60 # minutes to hours\n\n df_daily.index.name = None\n return df_daily", "response": "Aggregates data according to the characteristics\n of each variable"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef precip_master_station(precip_daily,\n master_precip_hourly,\n zerodiv):\n \"\"\"Disaggregate precipitation based on the patterns of a master station\n\n Parameters\n -----------\n precip_daily : pd.Series\n daily data\n master_precip_hourly : pd.Series\n observed hourly data of the master station\n zerodiv : str\n method to deal with zero division by key \"uniform\" --> uniform\n distribution\n \"\"\"\n\n precip_hourly = pd.Series(index=melodist.util.hourly_index(precip_daily.index))\n\n # set some parameters for cosine function\n for index_d, precip in precip_daily.iteritems():\n\n # get hourly data of the day\n index = index_d.date().isoformat()\n precip_h = master_precip_hourly[index]\n\n # calc rel values and multiply by daily sums\n # check for zero division\n if precip_h.sum() != 0 and precip_h.sum() != np.isnan(precip_h.sum()):\n precip_h_rel = (precip_h / precip_h.sum()) * precip\n\n else:\n # uniform option will preserve daily data by uniform distr\n if zerodiv == 'uniform':\n precip_h_rel = (1/24) * precip\n\n else:\n precip_h_rel = 0\n\n # write the disaggregated day to data\n precip_hourly[index] = precip_h_rel\n\n return precip_hourly", "response": "Disaggregate precipitation based on the patterns of a master station and observed hourly data of the master station."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\naggregating highly resolved precipitation data and creates statistics for the cascade model.", "response": "def aggregate_precipitation(vec_data,hourly=True, percentile=50):\n \"\"\"Aggregates highly resolved precipitation data and creates statistics\n\n Parameters\n ----------\n vec_data : pd.Series\n hourly (hourly=True) OR 5-min values \n\n Returns\n -------\n output : cascade object\n representing statistics of the cascade model\n \"\"\"\n cascade_opt = cascade.CascadeStatistics()\n cascade_opt.percentile = percentile\n\n # length of input time series\n n_in = len(vec_data)\n n_out = np.floor(n_in/2)\n\n # alternative:\n # 1st step: new time series\n vec_time = vec_data.index\n vdn0 = []\n vtn0 = []\n j = 0\n for i in range(0, n_in):\n if np.mod(i, 2) != 0:\n vdn0.append(vec_data.precip.values[i-1] + vec_data.precip.values[i])\n vtn0.append(vec_time[i])\n j = j+1\n\n vdn = pd.DataFrame(index=vtn0, data={'precip': vdn0})\n\n # length of new time series\n n_out = len(vdn)\n\n # series of box types:\n vbtype = np.zeros((n_out, ), dtype=np.int)\n\n # fields for empirical probabilities\n # counts\n nb = np.zeros((2, 4))\n nbxx = np.zeros((2, 4))\n\n # class boundaries for histograms\n # wclassbounds = np.linspace(0, 1, num=8)\n wlower = np.array([0,\n 0.1429,\n 0.2857,\n 0.4286,\n 0.5714,\n 0.7143,\n 0.8571]) # wclassbounds[0:7]\n wupper = np.array([0.1429,\n 0.2857,\n 0.4286,\n 0.5714,\n 0.7143,\n 0.8571,\n 1.0]) # wclassbounds[1:8]\n\n # evaluate mean rainfall intensity for wet boxes\n # these values should be determined during the aggregation phase!!!!!\n # mean volume threshold\n meanvol = np.percentile(vdn.precip[vdn.precip > 0.],\n cascade_opt.percentile) # np.mean(vdn.precip[vdn.precip>0.])\n cascade_opt.threshold = np.array([meanvol])\n\n # 2nd step: classify boxes at the upper level\n for i in range(0, n_out):\n if vdn.precip.values[i] > 0.: # rain?\n if i == 0: # only starting or isolated\n if vdn.precip.values[i+1] > 0.:\n vbtype[i] = cascade.BoxTypes.starting\n else:\n vbtype[i] = cascade.BoxTypes.isolated\n elif i == n_out-1: # only ending or isolated\n if vdn.precip.values[i-1] > 0.:\n vbtype[i] = cascade.BoxTypes.ending\n else:\n vbtype[i] = cascade.BoxTypes.isolated\n else: # neither at at the end nor at the beginning\n if vdn.precip.values[i-1] == 0. and vdn.precip.values[i+1] == 0.:\n vbtype[i] = cascade.BoxTypes.isolated\n if vdn.precip.values[i-1] == 0. and vdn.precip.values[i+1] > 0.:\n vbtype[i] = cascade.BoxTypes.starting\n if vdn.precip.values[i-1] > 0. and vdn.precip.values[i+1] > 0.:\n vbtype[i] = cascade.BoxTypes.enclosed\n if vdn.precip.values[i-1] > 0. and vdn.precip.values[i+1] == 0.:\n vbtype[i] = cascade.BoxTypes.ending\n else:\n vbtype[i] = cascade.BoxTypes.dry # no rain\n\n # 3rd step: examine branching\n j = 0\n for i in range(0, n_in):\n if np.mod(i, 2) != 0:\n if vdn.precip.values[j] > 0:\n if vdn.precip.values[j] > meanvol:\n belowabove = 1 # above mean\n else:\n belowabove = 0 # below mean\n\n nb[belowabove, vbtype[j]-1] += 1\n\n if vec_data.precip.values[i-1] > 0 and vec_data.precip.values[i] == 0:\n # P(1/0)\n cascade_opt.p10[belowabove, vbtype[j]-1] += 1\n\n if vec_data.precip.values[i-1] == 0 and vec_data.precip.values[i] > 0:\n # P(0/1)\n cascade_opt.p01[belowabove, vbtype[j]-1] += 1\n\n if vec_data.precip.values[i-1] > 0 and vec_data.precip.values[i] > 0:\n # P(x/x)\n cascade_opt.pxx[belowabove, vbtype[j]-1] += 1\n\n nbxx[belowabove, vbtype[j]-1] += 1\n\n # weights\n r1 = vec_data.precip.values[i-1]\n r2 = vec_data.precip.values[i]\n wxxval = r1 / (r1 + r2)\n\n # Test\n if abs(r1+r2-vdn.precip.values[j]) > 1.E-3:\n print('i=' + str(i) + ', j=' + str(j) +\n ', r1=' + str(r1) + \", r2=\" + str(r2) +\n \", Summe=\" + str(vdn.precip.values[j]))\n print(vec_data.index[i])\n print(vdn.index[j])\n print('error')\n return cascade_opt, vdn\n\n for k in range(0, 7):\n if wxxval > wlower[k] and wxxval <= wupper[k]:\n cascade_opt.wxx[k, belowabove, vbtype[j]-1] += 1\n break\n j = j + 1\n\n # 4th step: transform counts to percentages\n cascade_opt.p01 = cascade_opt.p01 / nb\n cascade_opt.p10 = cascade_opt.p10 / nb\n cascade_opt.pxx = cascade_opt.pxx / nb\n\n with np.errstate(divide='ignore', invalid='ignore'): # do not issue warnings here when dividing by zero, this is handled below\n for k in range(0, 7):\n cascade_opt.wxx[k, :, :] = cascade_opt.wxx[k, :, :] / nbxx[:, :]\n\n # In some cases, the time series are too short for deriving statistics.\n\n if (np.isnan(cascade_opt.p01).any() or\n np.isnan(cascade_opt.p10).any() or\n np.isnan(cascade_opt.pxx).any()):\n print(\"ERROR (branching probabilities):\")\n print(\"Invalid statistics. Default values will be returned. \"\n \"Try to use longer time series or apply statistics \"\n \"derived for another station.\")\n cascade_opt.fill_with_sample_data()\n\n # For some box types, the corresponding probabilities might yield nan.\n # If this happens, nan values will be replaced by 1/7 in order to provide\n # valid values for disaggregation.\n if np.isnan(cascade_opt.wxx).any():\n print(\"Warning (weighting probabilities):\")\n print(\"The derived cascade statistics are not valid as some \"\n \"probabilities are undefined! \", end=\"\")\n print(\"Try to use longer time series that might be more \"\n \"appropriate for deriving statistics. \", end=\"\")\n print(\"As a workaround, default values according to equally \"\n \"distributed probabilities \", end=\"\")\n print(\"will be applied...\", end=\"\")\n cascade_opt.wxx[np.isnan(cascade_opt.wxx)] = 1.0 / 7.0\n wxx = np.zeros((2, 4))\n for k in range(0, 7):\n wxx[:, :] += cascade_opt.wxx[k, :, :]\n if wxx.any() > 1.001 or wxx.any() < 0.999:\n print(\"failed! Using default values!\")\n cascade_opt.fill_with_sample_data()\n else:\n print(\"OK!\")\n\n return cascade_opt, vdn"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the seasonal data.", "response": "def seasonal_subset(dataframe,\n months='all'):\n '''Get the seasonal data.\n\n Parameters\n ----------\n dataframe : pd.DataFrame\n months: int, str\n Months to use for statistics, or 'all' for 1-12 (default='all')\n '''\n\n if isinstance(months, str) and months == 'all':\n months = np.arange(12) + 1\n\n for month_num, month in enumerate(months):\n df_cur = dataframe[dataframe.index.month == month]\n\n if month_num == 0:\n df = df_cur\n else:\n df = df.append(df_cur)\n\n return df.sort_index()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbuilding the cascade statistics of observed data for disaggregation based on the given date and hourly.", "response": "def build_casc(ObsData, hourly=True,level=9,\n months=None,\n avg_stats=True,\n percentile=50):\n '''Builds the cascade statistics of observed data for disaggregation\n\n Parameters\n -----------\n ObsData : pd.Series\n hourly=True -> hourly obs data\n else -> 5min data (disaggregation level=9 (default), 10, 11)\n \n months : numpy array of ints\n Months for each seasons to be used for statistics (array of\n numpy array, default=1-12, e.g., [np.arange(12) + 1])\n avg_stats : bool\n average statistics for all levels True/False (default=True)\n percentile : int, float\n percentile for splitting the dataset in small and high\n intensities (default=50)\n\n Returns\n -------\n list_seasonal_casc :\n list holding the results\n '''\n\n list_seasonal_casc = list()\n\n if months is None:\n months = [np.arange(12) + 1]\n\n # Parameter estimation for each season\n for cur_months in months:\n vdn = seasonal_subset(ObsData, cur_months)\n if len(ObsData.precip[np.isnan(ObsData.precip)]) > 0:\n ObsData.precip[np.isnan(ObsData.precip)] = 0\n\n casc_opt = melodist.cascade.CascadeStatistics()\n casc_opt.percentile = percentile\n list_casc_opt = list()\n \n count = 0\n \n if hourly:\n aggre_level = 5\n else:\n aggre_level = level\n \n thresholds = np.zeros(aggre_level) #np.array([0., 0., 0., 0., 0.])\n \n for i in range(0, aggre_level):\n # aggregate the data\n casc_opt_i, vdn = aggregate_precipitation(vdn, hourly, \\\n percentile=percentile)\n thresholds[i] = casc_opt_i.threshold\n copy_of_casc_opt_i = copy.copy(casc_opt_i)\n list_casc_opt.append(copy_of_casc_opt_i)\n n_vdn = len(vdn)\n casc_opt_i * n_vdn # level related weighting\n casc_opt + casc_opt_i # add to total statistics\n count = count + n_vdn\n casc_opt * (1. / count) # transfer weighted matrices to probabilities\n casc_opt.threshold = thresholds\n \n # statistics object\n if avg_stats:\n # in this case, the average statistics will be applied for all levels likewise\n stat_obj = casc_opt\n else:\n # for longer time series, separate statistics might be more appropriate\n # level dependent statistics will be assumed\n stat_obj = list_casc_opt\n\n list_seasonal_casc.append(stat_obj)\n\n return list_seasonal_casc"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreplaces in valid emojis in a string where a valid emoji is between ::", "response": "def replace(cls, replacement_string):\n \"\"\"Add in valid emojis in a string where a valid emoji is between ::\"\"\"\n e = cls()\n\n def _replace_emoji(match):\n val = match.group(1)\n if val in e:\n return e._image_string(match.group(1))\n else:\n return match.group(0)\n\n return e._pattern.sub(_replace_emoji, replacement_string)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef replace_unicode(cls, replacement_string):\n e = cls()\n output = []\n surrogate_character = None\n\n if settings.EMOJI_REPLACE_HTML_ENTITIES:\n replacement_string = cls.replace_html_entities(replacement_string)\n\n for i, character in enumerate(replacement_string):\n if character in cls._unicode_modifiers:\n continue\n\n # Check whether this is the first character in a Unicode\n # surrogate pair when Python doesn't have wide Unicode\n # support.\n #\n # Is there any reason to do this even if Python got wide\n # support enabled?\n if(not UNICODE_WIDE and not surrogate_character and\n ord(character) >= UNICODE_SURROGATE_MIN and\n ord(character) <= UNICODE_SURROGATE_MAX):\n surrogate_character = character\n continue\n\n if surrogate_character:\n character = convert_unicode_surrogates(\n surrogate_character + character\n )\n surrogate_character = None\n\n name = e.name_for(character)\n if name:\n if settings.EMOJI_ALT_AS_UNICODE:\n character = e._image_string(name, alt=character)\n else:\n character = e._image_string(name)\n\n output.append(character)\n\n return ''.join(output)", "response": "This method will iterate over every character in replacement_string and replace it with an image just like replace."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef replace_html_entities(cls, replacement_string):\n def _hex_to_unicode(hex_code):\n if PYTHON3:\n hex_code = '{0:0>8}'.format(hex_code)\n as_int = struct.unpack('>i', bytes.fromhex(hex_code))[0]\n return '{0:c}'.format(as_int)\n else:\n return hex_to_unicode(hex_code)\n\n def _replace_integer_entity(match):\n hex_val = hex(int(match.group(1)))\n\n return _hex_to_unicode(hex_val.replace('0x', ''))\n\n def _replace_hex_entity(match):\n return _hex_to_unicode(match.group(1))\n\n # replace integer code points, A\n replacement_string = re.sub(\n cls._html_entities_integer_unicode_regex,\n _replace_integer_entity,\n replacement_string\n )\n # replace hex code points, A\n replacement_string = re.sub(\n cls._html_entities_hex_unicode_regex,\n _replace_hex_entity,\n replacement_string\n )\n\n return replacement_string", "response": "Replaces HTML escaped unicode entities with their unicode\n equivalent."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _convert_to_unicode(string):\n codepoints = []\n for character in string.split('-'):\n if character in BLACKLIST_UNICODE:\n next\n\n codepoints.append(\n '\\U{0:0>8}'.format(character).decode('unicode-escape')\n )\n\n return codepoints", "response": "This method converts a string to unicode."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _delete_file(configurator, path):\n path = os.path.join(configurator.target_directory, path)\n os.remove(path)\n try:\n os.removedirs(os.path.dirname(path))\n except OSError:\n pass", "response": "Delete file and remove its directories if empty"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninserts an item into the list of existing manifest items.", "response": "def _insert_manifest_item(configurator, key, item):\n \"\"\" Insert an item in the list of an existing manifest key \"\"\"\n with _open_manifest(configurator) as f:\n manifest = f.read()\n if item in ast.literal_eval(manifest).get(key, []):\n return\n pattern = \"\"\"([\"']{}[\"']:\\\\s*\\\\[)\"\"\".format(key)\n repl = \"\"\"\\\\1\\n '{}',\"\"\".format(item)\n manifest = re.sub(pattern, repl, manifest, re.MULTILINE)\n with _open_manifest(configurator, \"w\") as f:\n f.write(manifest)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _read_requirements(filename):\n with open(filename) as requirements_file:\n contents = requirements_file.read()\n return [line.strip() for line in contents.splitlines() if _is_requirement(line)]", "response": "Parses a file for pip installation requirements."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nassigns a permission to a group", "response": "def assign_perm(perm, group):\n \"\"\"\n Assigns a permission to a group\n \"\"\"\n if not isinstance(perm, Permission):\n try:\n app_label, codename = perm.split('.', 1)\n except ValueError:\n raise ValueError(\"For global permissions, first argument must be in\"\n \" format: 'app_label.codename' (is %r)\" % perm)\n perm = Permission.objects.get(content_type__app_label=app_label, codename=codename)\n\n group.permissions.add(perm)\n return perm"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving a permission from a group", "response": "def remove_perm(perm, group):\n \"\"\"\n Removes a permission from a group\n \"\"\"\n if not isinstance(perm, Permission):\n try:\n app_label, codename = perm.split('.', 1)\n except ValueError:\n raise ValueError(\"For global permissions, first argument must be in\"\n \" format: 'app_label.codename' (is %r)\" % perm)\n perm = Permission.objects.get(content_type__app_label=app_label, codename=codename)\n\n group.permissions.remove(perm)\n return"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_list_class(context, list):\n return \"list_%s_%s\" % (list.model._meta.app_label, list.model._meta.model_name)", "response": "Returns the class to use for the passed in list."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nformats a date in the current timezone if one is specified", "response": "def format_datetime(time):\n \"\"\"\n Formats a date, converting the time to the user timezone if one is specified\n \"\"\"\n user_time_zone = timezone.get_current_timezone()\n if time.tzinfo is None:\n time = time.replace(tzinfo=pytz.utc)\n user_time_zone = pytz.timezone(getattr(settings, 'USER_TIME_ZONE', 'GMT'))\n\n time = time.astimezone(user_time_zone)\n return time.strftime(\"%b %d, %Y %H:%M\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the value of the passed in field from the object s ListView.", "response": "def get_value_from_view(context, field):\n \"\"\"\n Responsible for deriving the displayed value for the passed in 'field'.\n\n This first checks for a particular method on the ListView, then looks for a method\n on the object, then finally treats it as an attribute.\n \"\"\"\n view = context['view']\n obj = None\n if 'object' in context:\n obj = context['object']\n\n value = view.lookup_field_value(context, obj, field)\n\n # it's a date\n if type(value) == datetime:\n return format_datetime(value)\n\n return value"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the class of the given field.", "response": "def get_class(context, field, obj=None):\n \"\"\"\n Looks up the class for this field\n \"\"\"\n view = context['view']\n return view.lookup_field_class(field, obj, \"field_\" + field)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_label(context, field, obj=None):\n view = context['view']\n return view.lookup_field_label(context, field, obj)", "response": "Returns the right label for the passed in field."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the link to the given field.", "response": "def get_field_link(context, field, obj=None):\n \"\"\"\n Determine what the field link should be for the given field, object pair\n \"\"\"\n view = context['view']\n return view.lookup_field_link(context, field, obj)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_permissions_app_name():\n global permissions_app_name\n\n if not permissions_app_name:\n permissions_app_name = getattr(settings, 'PERMISSIONS_APP', None)\n\n if not permissions_app_name:\n app_names_with_models = [a.name for a in apps.get_app_configs() if a.models_module is not None]\n if app_names_with_models:\n permissions_app_name = app_names_with_models[-1]\n\n return permissions_app_name", "response": "Gets the app after which smartmin permissions should be installed."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking the role and permissions for the passed in role.", "response": "def check_role_permissions(role, permissions, current_permissions):\n \"\"\"\n Checks the the passed in role (can be user, group or AnonymousUser) has all the passed\n in permissions, granting them if necessary.\n \"\"\"\n role_permissions = []\n\n # get all the current permissions, we'll remove these as we verify they should still be granted\n for permission in permissions:\n splits = permission.split(\".\")\n if len(splits) != 2 and len(splits) != 3:\n sys.stderr.write(\" invalid permission %s, ignoring\\n\" % permission)\n continue\n\n app = splits[0]\n codenames = []\n\n if len(splits) == 2:\n codenames.append(splits[1])\n else:\n (object, action) = splits[1:]\n\n # if this is a wildcard, then query our database for all the permissions that exist on this object\n if action == '*':\n for perm in Permission.objects.filter(codename__startswith=\"%s_\" % object, content_type__app_label=app):\n codenames.append(perm.codename)\n # otherwise, this is an error, continue\n else:\n sys.stderr.write(\" invalid permission %s, ignoring\\n\" % permission)\n continue\n\n if len(codenames) == 0:\n continue\n\n for codename in codenames:\n # the full codename for this permission\n full_codename = \"%s.%s\" % (app, codename)\n\n # this marks all the permissions which should remain\n role_permissions.append(full_codename)\n\n try:\n assign_perm(full_codename, role)\n except ObjectDoesNotExist:\n pass\n # sys.stderr.write(\" unknown permission %s, ignoring\\n\" % permission)\n\n # remove any that are extra\n for permission in current_permissions:\n if isinstance(permission, str):\n key = permission\n else:\n key = \"%s.%s\" % (permission.content_type.app_label, permission.codename)\n\n if key not in role_permissions:\n remove_perm(key, role)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_all_group_permissions(sender, **kwargs):\n if not is_permissions_app(sender):\n return\n\n config = getattr(settings, 'GROUP_PERMISSIONS', dict())\n\n # for each of our items\n for name, permissions in config.items():\n # get or create the group\n (group, created) = Group.objects.get_or_create(name=name)\n if created:\n pass\n\n check_role_permissions(group, permissions, group.permissions.all())", "response": "Checks that all the permissions specified in settings. py are set for our groups."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_permission(content_type, permission):\n # build our permission slug\n codename = \"%s_%s\" % (content_type.model, permission)\n\n # sys.stderr.write(\"Checking %s permission for %s\\n\" % (permission, content_type.name))\n\n # does it already exist\n if not Permission.objects.filter(content_type=content_type, codename=codename):\n Permission.objects.create(content_type=content_type,\n codename=codename,\n name=\"Can %s %s\" % (permission, content_type.name))", "response": "Adds the passed in permission to a content type."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(self, commit=True):\n is_new_user = self.instance.pk is None\n\n user = super(UserForm, self).save(commit)\n\n # new users should be made active by default\n if is_new_user:\n user.is_active = True\n\n # if we had a new password set, use it\n new_pass = self.cleaned_data['new_password']\n if new_pass:\n user.set_password(new_pass)\n if commit:\n user.save()\n\n return user", "response": "Save the user instance."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsmarts url function for a node.", "response": "def smart_url(url, obj=None):\n \"\"\"\n URLs that start with @ are reversed, using the passed in arguments.\n\n Otherwise a straight % substitution is applied.\n \"\"\"\n if url.find(\"@\") >= 0:\n (args, value) = url.split('@')\n\n if args:\n val = getattr(obj, args, None)\n return reverse(value, args=[val])\n else:\n return reverse(value)\n else:\n if obj is None:\n return url\n else:\n return url % obj.id"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef derive_single_object_url_pattern(slug_url_kwarg, path, action):\n if slug_url_kwarg:\n return r'^%s/%s/(?P<%s>[^/]+)/$' % (path, action, slug_url_kwarg)\n else:\n return r'^%s/%s/(?P\\d+)/$' % (path, action)", "response": "Derives the regular expression used to match single object views."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if the current user has permission for this view.", "response": "def has_permission(self, request, *args, **kwargs):\n \"\"\"\n Figures out if the current user has permissions for this view.\n \"\"\"\n self.kwargs = kwargs\n self.args = args\n self.request = request\n\n if not getattr(self, 'permission', None):\n return True\n else:\n return request.user.has_perm(self.permission)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\noverloading to check permissions if appropriate", "response": "def dispatch(self, request, *args, **kwargs):\n \"\"\"\n Overloaded to check permissions if appropriate\n \"\"\"\n def wrapper(request, *args, **kwargs):\n if not self.has_permission(request, *args, **kwargs):\n path = urlquote(request.get_full_path())\n login_url = kwargs.pop('login_url', settings.LOGIN_URL)\n redirect_field_name = kwargs.pop('redirect_field_name', REDIRECT_FIELD_NAME)\n return HttpResponseRedirect(\"%s?%s=%s\" % (login_url, redirect_field_name, path))\n else:\n response = self.pre_process(request, *args, **kwargs)\n if not response:\n return super(SmartView, self).dispatch(request, *args, **kwargs)\n else:\n return response\n\n return wrapper(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lookup_obj_attribute(self, obj, field):\n curr_field = field.encode('ascii', 'ignore').decode(\"utf-8\")\n rest = None\n\n if field.find('.') >= 0:\n curr_field = field.split('.')[0]\n rest = '.'.join(field.split('.')[1:])\n\n # next up is the object itself\n obj_field = getattr(obj, curr_field, None)\n\n # if it is callable, do so\n if obj_field and getattr(obj_field, '__call__', None):\n obj_field = obj_field()\n\n if obj_field and rest:\n return self.lookup_obj_attribute(obj_field, rest)\n else:\n return obj_field", "response": "Look for a field s value from the passed in object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlooking up the field value for the passed in object and field name. Note that this method is actually called from a template, but this provides a hook for subclasses to modify behavior if they wish to do so. This may be used for example to change the display value of a variable depending on other variables within our context.", "response": "def lookup_field_value(self, context, obj, field):\n \"\"\"\n Looks up the field value for the passed in object and field name.\n\n Note that this method is actually called from a template, but this provides a hook\n for subclasses to modify behavior if they wish to do so.\n\n This may be used for example to change the display value of a variable depending on\n other variables within our context.\n \"\"\"\n curr_field = field.encode('ascii', 'ignore').decode(\"utf-8\")\n\n # if this isn't a subfield, check the view to see if it has a get_ method\n if field.find('.') == -1:\n # view supercedes all, does it have a 'get_' method for this obj\n view_method = getattr(self, 'get_%s' % curr_field, None)\n if view_method:\n return view_method(obj)\n\n return self.lookup_obj_attribute(obj, field)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef lookup_field_label(self, context, field, default=None):\n # if this is a subfield, strip off everything but the last field name\n if field.find('.') >= 0:\n return self.lookup_field_label(context, field.split('.')[-1], default)\n\n label = None\n\n # is there a label specified for this field\n if field in self.field_config and 'label' in self.field_config[field]:\n label = self.field_config[field]['label']\n\n # if we were given a default, use that\n elif default:\n label = default\n\n # check our model\n else:\n for model_field in self.model._meta.fields:\n if model_field.name == field:\n return model_field.verbose_name.title()\n\n # otherwise, derive it from our field name\n if label is None:\n label = self.derive_field_label(field)\n\n return label", "response": "Look up the label for a field in the context."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlooking up the help text for the passed in field.", "response": "def lookup_field_help(self, field, default=None):\n \"\"\"\n Looks up the help text for the passed in field.\n \"\"\"\n help = None\n\n # is there a label specified for this field\n if field in self.field_config and 'help' in self.field_config[field]:\n help = self.field_config[field]['help']\n\n # if we were given a default, use that\n elif default:\n help = default\n\n # try to see if there is a description on our model\n elif hasattr(self, 'model'):\n for model_field in self.model._meta.fields:\n if model_field.name == field:\n help = model_field.help_text\n break\n\n return help"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nlooks up any additional class we should include when rendering this field", "response": "def lookup_field_class(self, field, obj=None, default=None):\n \"\"\"\n Looks up any additional class we should include when rendering this field\n \"\"\"\n css = \"\"\n\n # is there a class specified for this field\n if field in self.field_config and 'class' in self.field_config[field]:\n css = self.field_config[field]['class']\n\n # if we were given a default, use that\n elif default:\n css = default\n\n return css"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_template_names(self):\n templates = []\n if getattr(self, 'template_name', None):\n templates.append(self.template_name)\n\n if getattr(self, 'default_template', None):\n templates.append(self.default_template)\n else:\n templates = super(SmartView, self).get_template_names()\n\n return templates", "response": "Returns the names of the template to use for this request."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nderive the list of fields from the instance.", "response": "def derive_fields(self):\n \"\"\"\n Default implementation\n \"\"\"\n fields = []\n if self.fields:\n fields.append(self.fields)\n\n return fields"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_context_data(self, **kwargs):\n context = super(SmartView, self).get_context_data(**kwargs)\n\n # derive our field config\n self.field_config = self.derive_field_config()\n\n # add our fields\n self.fields = self.derive_fields()\n\n # build up our current parameter string, EXCLUSIVE of our page. These\n # are used to build pagination URLs\n url_params = \"?\"\n order_params = \"\"\n for key in self.request.GET.keys():\n if key != 'page' and key != 'pjax' and (len(key) == 0 or key[0] != '_'):\n for value in self.request.GET.getlist(key):\n url_params += \"%s=%s&\" % (key, urlquote(value))\n elif key == '_order':\n order_params = \"&\".join([\"%s=%s\" % (key, _) for _ in self.request.GET.getlist(key)])\n\n context['url_params'] = url_params\n context['order_params'] = order_params + \"&\"\n context['pjax'] = self.pjax\n\n # set our blocks\n context['blocks'] = dict()\n\n # stuff it all in our context\n context['fields'] = self.fields\n context['view'] = self\n context['field_config'] = self.field_config\n\n context['title'] = self.derive_title()\n\n # and any extra context the user specified\n context.update(self.extra_context)\n\n # by default, our base is 'base.html', but we might be pjax\n base_template = \"base.html\"\n if 'pjax' in self.request.GET or 'pjax' in self.request.POST:\n base_template = \"smartmin/pjax.html\"\n\n if 'HTTP_X_PJAX' in self.request.META:\n base_template = \"smartmin/pjax.html\"\n\n context['base_template'] = base_template\n\n # set our refresh if we have one\n refresh = self.derive_refresh()\n if refresh:\n context['refresh'] = refresh\n\n return context", "response": "This method supplements the normal context data by adding our fields and labels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef render_to_response(self, context, **response_kwargs):\n # should we actually render in json?\n if '_format' in self.request.GET and self.request.GET['_format'] == 'json':\n return JsonResponse(self.as_json(context), safe=False)\n\n # otherwise, return normally\n else:\n return super(SmartView, self).render_to_response(context)", "response": "Render the object to a response."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef derive_fields(self):\n if self.fields:\n return list(self.fields)\n\n else:\n fields = []\n for field in self.object._meta.fields:\n fields.append(field.name)\n\n # only exclude? then remove those items there\n exclude = self.derive_exclude()\n\n # remove any excluded fields\n fields = [field for field in fields if field not in exclude]\n\n return fields", "response": "Derives our fields from our object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_context_data(self, **kwargs):\n context = super(SmartDeleteView, self).get_context_data(**kwargs)\n context['name_field'] = self.name_field\n context['cancel_url'] = self.get_cancel_url()\n return context", "response": "Add in the name field to use for the name field"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef derive_title(self):\n title = super(SmartListView, self).derive_title()\n\n if not title:\n return force_text(self.model._meta.verbose_name_plural).title()\n else:\n return title", "response": "Derives our title from our list\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef derive_link_fields(self, context):\n if self.link_fields is not None:\n return self.link_fields\n\n else:\n link_fields = set()\n if self.fields:\n for field in self.fields:\n if field != 'is_active':\n link_fields.add(field)\n break\n\n return link_fields", "response": "This method is used to derive which fields should be linked."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef lookup_field_orderable(self, field):\n try:\n self.model._meta.get_field_by_name(field)\n return True\n except Exception:\n # that field doesn't exist, so not sortable\n return False", "response": "Returns whether the passed in field is sortable or not."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_queryset(self, **kwargs):\n queryset = self.derive_queryset(**kwargs)\n\n return self.order_queryset(queryset)", "response": "Gets our queryset. This takes care of filtering if there are any\n fields to filter by."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef derive_ordering(self):\n if '_order' in self.request.GET:\n return self.request.GET['_order']\n elif self.default_order:\n return self.default_order\n else:\n return None", "response": "Derives the order of the items in the current queryset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nordering the passed in queryset using the _order query parameter.", "response": "def order_queryset(self, queryset):\n \"\"\"\n Orders the passed in queryset, returning a new queryset in response. By default uses the _order query\n parameter.\n \"\"\"\n order = self.derive_ordering()\n\n # if we get our order from the request\n # make sure it is a valid field in the list\n if '_order' in self.request.GET:\n if order.lstrip('-') not in self.derive_fields():\n order = None\n\n if order:\n # if our order is a single string, convert to a simple list\n if isinstance(order, str):\n order = (order,)\n\n queryset = queryset.order_by(*order)\n\n return queryset"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef render_to_response(self, context, **response_kwargs):\n # is this a select2 format response?\n if self.request.GET.get('_format', 'html') == 'select2':\n\n results = []\n for obj in context['object_list']:\n result = None\n if hasattr(obj, 'as_select2'):\n result = obj.as_select2()\n\n if not result:\n result = dict(id=obj.pk, text=\"%s\" % obj)\n\n results.append(result)\n\n json_data = dict(results=results, err='nil', more=context['page_obj'].has_next())\n return JsonResponse(json_data)\n # otherwise, return normally\n else:\n return super(SmartListView, self).render_to_response(context)", "response": "Render the list to a JSON response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn an instance of the form to be used in this view.", "response": "def get_form(self):\n \"\"\"\n Returns an instance of the form to be used in this view.\n \"\"\"\n self.form = super(SmartFormMixin, self).get_form()\n\n fields = list(self.derive_fields())\n\n # apply our field filtering on our form class\n exclude = self.derive_exclude()\n exclude += self.derive_readonly()\n\n # remove any excluded fields\n for field in exclude:\n if field in self.form.fields:\n del self.form.fields[field]\n\n if fields is not None:\n # filter out our form fields\n remove = [name for name in self.form.fields.keys() if name not in fields]\n for name in remove:\n del self.form.fields[name]\n\n # stuff in our referer as the default location for where to return\n location = forms.CharField(widget=forms.widgets.HiddenInput(), required=False)\n\n if ('HTTP_REFERER' in self.request.META):\n location.initial = self.request.META['HTTP_REFERER']\n\n # add the location to our form fields\n self.form.fields['loc'] = location\n\n if fields:\n fields.append('loc')\n\n # provides a hook to programmatically customize fields before rendering\n for (name, field) in self.form.fields.items():\n field = self.customize_form_field(name, field)\n self.form.fields[name] = field\n\n return self.form"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef customize_form_field(self, name, field):\n if isinstance(field, forms.fields.DateField) and isinstance(field.widget, forms.widgets.DateInput):\n field.widget = widgets.DatePickerWidget()\n field.input_formats = [field.widget.input_format[1]] + list(field.input_formats)\n\n if isinstance(field, forms.fields.ImageField) and isinstance(field.widget, forms.widgets.ClearableFileInput):\n field.widget = widgets.ImageThumbnailWidget()\n\n return field", "response": "Customize the form field."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lookup_field_label(self, context, field, default=None):\n default = None\n\n for form_field in self.form:\n if form_field.name == field:\n default = form_field.label\n break\n\n return super(SmartFormMixin, self).lookup_field_label(context, field, default=default)", "response": "Look up the field label for the passed in field name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lookup_field_help(self, field, default=None):\n default = None\n\n for form_field in self.form:\n if form_field.name == field:\n default = form_field.help_text\n break\n\n return super(SmartFormMixin, self).lookup_field_help(field, default=default)", "response": "Look up the help text for the passed in field."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef derive_readonly(self):\n readonly = list(self.readonly)\n for key, value in self.field_config.items():\n if 'readonly' in value and value['readonly']:\n readonly.append(key)\n\n return readonly", "response": "Derives the list of readonly items from the field_config"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the form class to use in this view", "response": "def get_form_class(self):\n \"\"\"\n Returns the form class to use in this view\n \"\"\"\n if self.form_class:\n form_class = self.form_class\n\n else:\n if self.model is not None:\n # If a model has been explicitly provided, use it\n model = self.model\n elif hasattr(self, 'object') and self.object is not None:\n # If this view is operating on a single object, use\n # the class of that object\n model = self.object.__class__\n else:\n # Try to get a queryset and extract the model class\n # from that\n model = self.get_queryset().model\n\n # run time parameters when building our form\n factory_kwargs = self.get_factory_kwargs()\n form_class = model_forms.modelform_factory(model, **factory_kwargs)\n\n return form_class"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a dictionary of keyword arguments that can be passed to the form factory function.", "response": "def get_factory_kwargs(self):\n \"\"\"\n Let's us specify any extra parameters we might want to call for our form factory.\n\n These can include: 'form', 'fields', 'exclude' or 'formfield_callback'\n \"\"\"\n params = dict()\n\n exclude = self.derive_exclude()\n exclude += self.derive_readonly()\n\n if self.fields:\n fields = list(self.fields)\n for ex in exclude:\n if ex in fields:\n fields.remove(ex)\n\n params['fields'] = fields\n\n if exclude:\n params['exclude'] = exclude\n\n return params"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the url that was stuffed in ourCOOKIE and the object that was created.", "response": "def get_success_url(self):\n \"\"\"\n By default we use the referer that was stuffed in our\n form when it was created\n \"\"\"\n if self.success_url:\n # if our smart url references an object, pass that in\n if self.success_url.find('@') > 0:\n return smart_url(self.success_url, self.object)\n else:\n return smart_url(self.success_url, None)\n\n elif 'loc' in self.form.cleaned_data:\n return self.form.cleaned_data['loc']\n\n raise ImproperlyConfigured(\"No redirect location found, override get_success_url to not use redirect urls\")"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_form_kwargs(self):\n kwargs = super(SmartFormMixin, self).get_form_kwargs()\n kwargs['initial'] = self.derive_initial()\n return kwargs", "response": "Override this to include only those fields specified in the form."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nderives our title from our object", "response": "def derive_title(self):\n \"\"\"\n Derives our title from our object\n \"\"\"\n if not self.title:\n return _(\"Create %s\") % force_text(self.model._meta.verbose_name).title()\n else:\n return self.title"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the permission to use for the passed in action", "response": "def permission_for_action(self, action):\n \"\"\"\n Returns the permission to use for the passed in action\n \"\"\"\n return \"%s.%s_%s\" % (self.app_name.lower(), self.model_name.lower(), action)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the template to use for the passed in action", "response": "def template_for_action(self, action):\n \"\"\"\n Returns the template to use for the passed in action\n \"\"\"\n return \"%s/%s_%s.html\" % (self.module_name.lower(), self.model_name.lower(), action)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the reverse name for this action", "response": "def url_name_for_action(self, action):\n \"\"\"\n Returns the reverse name for this action\n \"\"\"\n return \"%s.%s_%s\" % (self.module_name.lower(), self.model_name.lower(), action)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef view_for_action(self, action):\n # this turns replace_foo into ReplaceFoo and read into Read\n class_name = \"\".join([word.capitalize() for word in action.split(\"_\")])\n view = None\n\n # see if we have a custom class defined for this action\n if hasattr(self, class_name):\n # return that one\n view = getattr(self, class_name)\n\n # no model set? set it ourselves\n if not getattr(view, 'model', None):\n view.model = self.model\n\n # no permission and we are supposed to set them, do so\n if not hasattr(view, 'permission') and self.permissions:\n view.permission = self.permission_for_action(action)\n\n # set our link URL based on read and update\n if not getattr(view, 'link_url', None):\n if 'read' in self.actions:\n view.link_url = 'id@%s' % self.url_name_for_action('read')\n elif 'update' in self.actions:\n view.link_url = 'id@%s' % self.url_name_for_action('update')\n\n # if we can't infer a link URL then view class must override lookup_field_link\n if not getattr(view, 'link_url', None) and 'lookup_field_link' not in view.__dict__:\n view.link_fields = ()\n\n # set add_button based on existence of Create view if add_button not explicitly set\n if action == 'list' and getattr(view, 'add_button', None) is None:\n view.add_button = 'create' in self.actions\n\n # set edit_button based on existence of Update view if edit_button not explicitly set\n if action == 'read' and getattr(view, 'edit_button', None) is None:\n view.edit_button = 'update' in self.actions\n\n # if update or create, set success url if not set\n if not getattr(view, 'success_url', None) and (action == 'update' or action == 'create'):\n view.success_url = '@%s' % self.url_name_for_action('list')\n\n # otherwise, use our defaults\n else:\n options = dict(model=self.model)\n\n # if this is an update or create, and we have a list view, then set the default to that\n if action == 'update' or action == 'create' and 'list' in self.actions:\n options['success_url'] = '@%s' % self.url_name_for_action('list')\n\n # set permissions if appropriate\n if self.permissions:\n options['permission'] = self.permission_for_action(action)\n\n if action == 'create':\n view = type(str(\"%sCreateView\" % self.model_name), (SmartCreateView,), options)\n\n elif action == 'read':\n if 'update' in self.actions:\n options['edit_button'] = True\n\n view = type(str(\"%sReadView\" % self.model_name), (SmartReadView,), options)\n\n elif action == 'update':\n if 'delete' in self.actions:\n options['delete_url'] = 'id@%s' % self.url_name_for_action('delete')\n\n view = type(str(\"%sUpdateView\" % self.model_name), (SmartUpdateView,), options)\n\n elif action == 'delete':\n if 'list' in self.actions:\n options['cancel_url'] = '@%s' % self.url_name_for_action('list')\n options['redirect_url'] = '@%s' % self.url_name_for_action('list')\n\n elif 'update' in self.actions:\n options['cancel_url'] = '@%s' % self.url_name_for_action('update')\n\n view = type(str(\"%sDeleteView\" % self.model_name), (SmartDeleteView,), options)\n\n elif action == 'list':\n if 'read' in self.actions:\n options['link_url'] = 'id@%s' % self.url_name_for_action('read')\n elif 'update' in self.actions:\n options['link_url'] = 'id@%s' % self.url_name_for_action('update')\n else:\n options['link_fields'] = ()\n\n if 'create' in self.actions:\n options['add_button'] = True\n\n view = type(str(\"%sListView\" % self.model_name), (SmartListView,), options)\n\n elif action == 'csv_import':\n options['model'] = ImportTask\n view = type(str(\"%sCSVImportView\" % self.model_name), (SmartCSVImportView,), options)\n\n if not view:\n # couldn't find a view? blow up\n raise Exception(\"No view found for action: %s\" % action)\n\n # set the url name for this view\n view.url_name = self.url_name_for_action(action)\n\n # no template set for it? set one based on our action and app name\n if not getattr(view, 'template_name', None):\n view.template_name = self.template_for_action(action)\n\n view.crudl = self\n\n return view", "response": "Returns the appropriate view class for the passed in action."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pattern_for_view(self, view, action):\n # if this view knows how to define a URL pattern, call that\n if getattr(view, 'derive_url_pattern', None):\n return view.derive_url_pattern(self.path, action)\n\n # otherwise take our best guess\n else:\n return r'^%s/%s/$' % (self.path, action)", "response": "Returns the URL pattern for the passed in action."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the appropriate URLs for this object.", "response": "def as_urlpatterns(self):\n \"\"\"\n Creates the appropriate URLs for this object.\n \"\"\"\n urls = []\n\n # for each of our actions\n for action in self.actions:\n view_class = self.view_for_action(action)\n view_pattern = self.pattern_for_view(view_class, action)\n name = self.url_name_for_action(action)\n urls.append(url(view_pattern, view_class.as_view(), name=name))\n\n return urls"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload all migrations in the order they would be applied to a clean database.", "response": "def load_migrations(self): # pragma: no cover\n \"\"\"\n Loads all migrations in the order they would be applied to a clean database\n \"\"\"\n executor = MigrationExecutor(connection=None)\n\n # create the forwards plan Django would follow on an empty database\n plan = executor.migration_plan(executor.loader.graph.leaf_nodes(), clean_start=True)\n\n if self.verbosity >= 2:\n for migration, _ in plan:\n self.stdout.write(\" > %s\" % migration)\n\n return [m[0] for m in plan]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nextracting SQL operations from the given migrations.", "response": "def extract_operations(self, migrations):\n \"\"\"\n Extract SQL operations from the given migrations\n \"\"\"\n operations = []\n\n for migration in migrations:\n for operation in migration.operations:\n if isinstance(operation, RunSQL):\n statements = sqlparse.parse(dedent(operation.sql))\n\n for statement in statements:\n operation = SqlObjectOperation.parse(statement)\n if operation:\n operations.append(operation)\n\n if self.verbosity >= 2:\n self.stdout.write(\" > % -100s (%s)\" % (operation, migration))\n\n return operations"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef normalize_operations(self, operations):\n normalized = OrderedDict()\n\n for operation in operations:\n op_key = (operation.sql_type, operation.obj_name)\n\n # do we already have an operation for this object?\n if op_key in normalized:\n if self.verbosity >= 2:\n self.stdout.write(\" < %s\" % normalized[op_key])\n\n del normalized[op_key]\n\n # don't add DROP operations for objects not previously created\n if operation.is_create:\n normalized[op_key] = operation\n elif self.verbosity >= 2:\n self.stdout.write(\" < %s\" % operation)\n\n return normalized.values()", "response": "Normalizes SQL operations for the object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_type_dumps(self, operations, preserve_order, output_dir):\n by_type = {SqlType.INDEX: [], SqlType.FUNCTION: [], SqlType.TRIGGER: []}\n for operation in operations:\n by_type[operation.sql_type].append(operation)\n\n # optionally sort each operation list by the object name\n if not preserve_order:\n for obj_type, ops in by_type.items():\n by_type[obj_type] = sorted(ops, key=lambda o: o.obj_name)\n\n if by_type[SqlType.INDEX]:\n self.write_dump('indexes', by_type[SqlType.INDEX], output_dir)\n if by_type[SqlType.FUNCTION]:\n self.write_dump('functions', by_type[SqlType.FUNCTION], output_dir)\n if by_type[SqlType.TRIGGER]:\n self.write_dump('triggers', by_type[SqlType.TRIGGER], output_dir)", "response": "Splits the list of SQL operations by type and dumps them to separate files\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns this Widget rendered as HTML as a Unicode string.", "response": "def render(self, name, value, attrs=None, renderer=None):\n \"\"\"\n Returns this Widget rendered as HTML, as a Unicode string.\n\n The 'value' given is not guaranteed to be valid input, so subclass\n implementations should program defensively.\n \"\"\"\n html = ''\n html += '%s' % value\n html += '' % (escape(name), escape(value))\n return mark_safe(html)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_atom_data(data_api, data_setters, atom_names, element_names, atom_charges, group_atom_ind):\n atom_name = atom_names[group_atom_ind]\n element = element_names[group_atom_ind]\n charge = atom_charges[group_atom_ind]\n alternative_location_id = data_api.alt_loc_list[data_api.atom_counter]\n serial_number = data_api.atom_id_list[data_api.atom_counter]\n x = data_api.x_coord_list[data_api.atom_counter]\n y = data_api.y_coord_list[data_api.atom_counter]\n z = data_api.z_coord_list[data_api.atom_counter]\n occupancy = data_api.occupancy_list[data_api.atom_counter]\n temperature_factor = data_api.b_factor_list[data_api.atom_counter]\n data_setters.set_atom_info(atom_name, serial_number, alternative_location_id,\n x, y, z, occupancy, temperature_factor, element, charge)", "response": "Adds the atom data to the data_setters class"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_group_bonds(data_setters, bond_indices, bond_orders):\n for bond_index in range(len(bond_orders)):\n data_setters.set_group_bond(bond_indices[bond_index*2],bond_indices[bond_index*2+1],bond_orders[bond_index])", "response": "Adds the bonds to the group."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_group(data_api, data_setters, group_index):\n group_type_ind = data_api.group_type_list[group_index]\n atom_count = len(data_api.group_list[group_type_ind][\"atomNameList\"])\n insertion_code = data_api.ins_code_list[group_index]\n data_setters.set_group_info(data_api.group_list[group_type_ind][\"groupName\"],\n data_api.group_id_list[group_index], insertion_code,\n data_api.group_list[group_type_ind][\"chemCompType\"],\n atom_count, data_api.num_bonds,\n data_api.group_list[group_type_ind][\"singleLetterCode\"],\n data_api.sequence_index_list[group_index],\n data_api.sec_struct_list[group_index])\n for group_atom_ind in range(atom_count):\n add_atom_data(data_api, data_setters,\n data_api.group_list[group_type_ind][\"atomNameList\"],\n data_api.group_list[group_type_ind][\"elementList\"],\n data_api.group_list[group_type_ind][\"formalChargeList\"],\n group_atom_ind)\n data_api.atom_counter +=1\n add_group_bonds(data_setters,\n data_api.group_list[group_type_ind][\"bondAtomList\"],\n data_api.group_list[group_type_ind][\"bondOrderList\"])\n return atom_count", "response": "Adds the data for a whole group to the group_list"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_chain_info(data_api, data_setters, chain_index):\n chain_id = data_api.chain_id_list[chain_index]\n chain_name = data_api.chain_name_list[chain_index]\n num_groups = data_api.groups_per_chain[chain_index]\n data_setters.set_chain_info(chain_id, chain_name, num_groups)\n next_ind = data_api.group_counter + num_groups\n last_ind = data_api.group_counter\n for group_ind in range(last_ind, next_ind):\n add_group(data_api, data_setters, group_ind)\n data_api.group_counter +=1\n data_api.chain_counter+=1", "response": "Add the data for a whole chain."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_atomic_information(data_api, data_setters):\n for model_chains in data_api.chains_per_model:\n data_setters.set_model_info(data_api.model_counter, model_chains)\n tot_chains_this_model = data_api.chain_counter + model_chains\n last_chain_counter = data_api.chain_counter\n for chain_index in range(last_chain_counter, tot_chains_this_model):\n add_chain_info(data_api, data_setters, chain_index)\n data_api.model_counter+=1", "response": "Add all the atomic information to the data_api"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate the bioassembly data.", "response": "def generate_bio_assembly(data_api, struct_inflator):\n \"\"\"Generate the bioassembly data.\n :param data_api the interface to the decoded data\n :param struct_inflator the interface to put the data into the client object\"\"\"\n bioassembly_count = 0\n for bioassembly in data_api.bio_assembly:\n bioassembly_count += 1\n for transform in bioassembly[\"transformList\"]:\n struct_inflator.set_bio_assembly_trans(bioassembly_count,\n transform[\"chainIndexList\"],\n transform[\"matrix\"])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_inter_group_bonds(data_api, struct_inflator):\n for i in range(len(data_api.bond_order_list)):\n struct_inflator.set_inter_group_bond(data_api.bond_atom_list[i * 2],\n data_api.bond_atom_list[i * 2 + 1],\n data_api.bond_order_list[i])", "response": "Generate inter group bonds."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_header_info(data_api, struct_inflator):\n struct_inflator.set_header_info(data_api.r_free,\n data_api.r_work,\n data_api.resolution,\n data_api.title,\n data_api.deposition_date,\n data_api.release_date,\n data_api.experimental_methods)", "response": "Add ancilliary header information to the structure."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding the crystallographic data to the structure.", "response": "def add_xtalographic_info(data_api, struct_inflator):\n \"\"\"\t Add the crystallographic data to the structure.\n\t :param data_api the interface to the decoded data\n\t :param struct_inflator the interface to put the data into the client object\"\"\"\n if data_api.unit_cell == None and data_api.space_group is not None:\n struct_inflator.set_xtal_info(data_api.space_group,\n constants.UNKNOWN_UNIT_CELL)\n elif data_api.unit_cell is not None and data_api.space_group is None:\n struct_inflator.set_xtal_info(constants.UNKNOWN_SPACE_GROUP,\n data_api.unit_cell)\n elif data_api.unit_cell is None and data_api.space_group is None:\n struct_inflator.set_xtal_info(constants.UNKNOWN_SPACE_GROUP,\n constants.UNKNOWN_UNIT_CELL)\n else:\n struct_inflator.set_xtal_info(data_api.space_group,\n data_api.unit_cell)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_entity_info( data_api, struct_inflator):\n for entity in data_api.entity_list:\n struct_inflator.set_entity_info(entity[\"chainIndexList\"],\n entity[\"sequence\"],\n entity[\"description\"],\n entity[\"type\"])", "response": "Add the entity info to the structure."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning to get a unique list of groups.", "response": "def get_unique_groups(input_list):\n \"\"\"Function to get a unique list of groups.\"\"\"\n out_list = []\n for item in input_list:\n if item not in out_list:\n out_list.append(item)\n return out_list"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert_to_dict(self):\n out_dict = {}\n out_dict[\"groupName\"] = self.group_name\n out_dict[\"atomNameList\"] = self.atom_name_list\n out_dict[\"elementList\"] = self.element_list\n out_dict[\"bondOrderList\"] = self.bond_order_list\n out_dict[\"bondAtomList\"] = self.bond_atom_list\n out_dict[\"formalChargeList\"] = self.charge_list\n out_dict[\"singleLetterCode\"] = self.single_letter_code\n out_dict[\"chemCompType\"] = self.group_type\n return out_dict", "response": "Convert the group object to a dictionary"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating an atom object and set the information.", "response": "def set_atom_info(self, atom_name, serial_number, alternative_location_id,\n x, y, z, occupancy, temperature_factor, element, charge):\n \"\"\"Create an atom object an set the information.\n :param atom_name: the atom name, e.g. CA for this atom\n :param serial_number: the serial id of the atom (e.g. 1)\n :param alternative_location_id: the alternative location id for the atom, if present\n :param x: the x coordiante of the atom\n :param y: the y coordinate of the atom\n :param z: the z coordinate of the atom\n :param occupancy: the occupancy of the atom\n :param temperature_factor: the temperature factor of the atom\n :param element: the element of the atom, e.g. C for carbon. According to IUPAC. Calcium is Ca\n :param charge: the formal atomic charge of the atom\n \"\"\"\n raise NotImplementedError"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the group information for a specific entry in the chemical dictionary.", "response": "def set_group_info(self, group_name, group_number, insertion_code,\n group_type, atom_count, bond_count, single_letter_code,\n sequence_index, secondary_structure_type):\n \"\"\"Set the information for a group\n :param group_name: the name of this group,e.g. LYS\n :param group_number: the residue number of this group\n :param insertion_code: the insertion code for this group\n :param group_type: a string indicating the type of group (as found in the chemcomp dictionary.\n Empty string if none available.\n :param atom_count: the number of atoms in the group\n :param bond_count: the number of unique bonds in the group\n :param single_letter_code: the single letter code of the group\n :param sequence_index: the index of this group in the sequence defined by the enttiy\n :param secondary_structure_type: the type of secondary structure used (types are according to DSSP and\n number to type mappings are defined in the specification)\n \"\"\"\n raise NotImplementedError"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the header information for the structure containing the structure containing the header information.", "response": "def set_header_info(self, r_free, r_work, resolution, title,\n deposition_date, release_date, experimental_methods):\n \"\"\"Sets the header information.\n :param r_free: the measured R-Free for the structure\n :param r_work: the measure R-Work for the structure\n :param resolution: the resolution of the structure\n :param title: the title of the structure\n :param deposition_date: the deposition date of the structure\n :param release_date: the release date of the structure\n :param experimnetal_methods: the list of experimental methods in the structure\n \"\"\"\n raise NotImplementedError"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encode_data(self):\n output_data = {}\n output_data[\"groupTypeList\"] = encode_array(self.group_type_list, 4, 0)\n output_data[\"xCoordList\"] = encode_array(self.x_coord_list, 10, 1000)\n output_data[\"yCoordList\"] = encode_array(self.y_coord_list, 10, 1000)\n output_data[\"zCoordList\"] = encode_array(self.z_coord_list, 10, 1000)\n output_data[\"bFactorList\"] = encode_array(self.b_factor_list, 10, 100)\n output_data[\"occupancyList\"] = encode_array(self.occupancy_list, 9, 100)\n output_data[\"atomIdList\"] = encode_array(self.atom_id_list, 8, 0)\n output_data[\"altLocList\"] = encode_array(self.alt_loc_list, 6, 0)\n output_data[\"insCodeList\"] = encode_array(self.ins_code_list, 6, 0)\n output_data[\"groupIdList\"] = encode_array(self.group_id_list, 8, 0)\n output_data[\"groupList\"] = self.group_list\n output_data[\"sequenceIndexList\"] = encode_array(self.sequence_index_list, 8, 0)\n output_data[\"chainNameList\"] = encode_array(self.chain_name_list, 5, 4)\n output_data[\"chainIdList\"] = encode_array(self.chain_id_list, 5, 4)\n output_data[\"bondAtomList\"] = encode_array(self.bond_atom_list, 4, 0)\n output_data[\"bondOrderList\"] = encode_array(self.bond_order_list, 2, 0)\n output_data[\"secStructList\"] = encode_array(self.sec_struct_list, 2, 0)\n output_data[\"chainsPerModel\"] = self.chains_per_model\n output_data[\"groupsPerChain\"] = self.groups_per_chain\n output_data[\"spaceGroup\"] = self.space_group\n output_data[\"mmtfVersion\"] = self.mmtf_version\n output_data[\"mmtfProducer\"] = self.mmtf_producer\n output_data[\"structureId\"] = self.structure_id\n output_data[\"entityList\"] = self.entity_list\n output_data[\"bioAssemblyList\"] = self.bio_assembly\n output_data[\"rFree\"] = self.r_free\n output_data[\"rWork\"] = self.r_work\n output_data[\"resolution\"] = self.resolution\n output_data[\"title\"] = self.title\n output_data[\"experimentalMethods\"] = self.experimental_methods\n output_data[\"depositionDate\"] = self.deposition_date\n output_data[\"releaseDate\"] = self.release_date\n output_data[\"unitCell\"] = self.unit_cell\n output_data[\"numBonds\"] = self.num_bonds\n output_data[\"numChains\"] = self.num_chains\n output_data[\"numModels\"] = self.num_models\n output_data[\"numAtoms\"] = self.num_atoms\n output_data[\"numGroups\"] = self.num_groups\n return output_data", "response": "Encode the data back into a dict."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitialises the structure object.", "response": "def init_structure(self, total_num_bonds, total_num_atoms,\n total_num_groups, total_num_chains, total_num_models,\n structure_id):\n \"\"\"Initialise the structure object.\n :param total_num_bonds: the number of bonds in the structure\n :param total_num_atoms: the number of atoms in the structure\n :param total_num_groups: the number of groups in the structure\n :param total_num_chains: the number of chains in the structure\n :param total_num_models: the number of models in the structure\n :param structure_id the: id of the structure (e.g. PDB id)\n \"\"\"\n self.mmtf_version = constants.MMTF_VERSION\n self.mmtf_producer = constants.PRODUCER\n self.num_atoms = total_num_atoms\n self.num_bonds = total_num_bonds\n self.num_groups = total_num_groups\n self.num_chains = total_num_chains\n self.num_models = total_num_models\n self.structure_id = structure_id\n # initialise the arrays\n self.x_coord_list = []\n self.y_coord_list = []\n self.z_coord_list = []\n self.group_type_list = []\n self.entity_list = []\n self.b_factor_list = []\n self.occupancy_list = []\n self.atom_id_list = []\n self.alt_loc_list = []\n self.ins_code_list = []\n self.group_id_list = []\n self.sequence_index_list = []\n self.group_list = []\n self.chain_name_list = []\n self.chain_id_list = []\n self.bond_atom_list = []\n self.bond_order_list = []\n self.sec_struct_list = []\n self.chains_per_model = []\n self.groups_per_chain = []\n self.current_group = None\n self.bio_assembly = []"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate an atom object and set the information.", "response": "def set_atom_info(self, atom_name, serial_number, alternative_location_id,\n x, y, z, occupancy, temperature_factor, element, charge):\n \"\"\"Create an atom object an set the information.\n :param atom_name: the atom name, e.g. CA for this atom\n :param serial_number: the serial id of the atom (e.g. 1)\n :param alternative_location_id: the alternative location id for the atom, if present\n :param x: the x coordiante of the atom\n :param y: the y coordinate of the atom\n :param z: the z coordinate of the atom\n :param occupancy: the occupancy of the atom\n :param temperature_factor: the temperature factor of the atom\n :param element: the element of the atom, e.g. C for carbon. According to IUPAC. Calcium is Ca\n :param charge: the formal atomic charge of the atom\n \"\"\"\n self.x_coord_list.append(x)\n self.y_coord_list.append(y)\n self.z_coord_list.append(z)\n self.atom_id_list.append(serial_number)\n self.alt_loc_list.append(alternative_location_id)\n self.occupancy_list.append(occupancy)\n self.b_factor_list.append(temperature_factor)\n ## Now add the group level data\n self.current_group.atom_name_list.append(atom_name)\n self.current_group.charge_list.append(charge)\n self.current_group.element_list.append(element)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the chain information.", "response": "def set_chain_info(self, chain_id, chain_name, num_groups):\n \"\"\"Set the chain information.\n :param chain_id: the asym chain id from mmCIF\n :param chain_name: the auth chain id from mmCIF\n :param num_groups: the number of groups this chain has\n \"\"\"\n self.chain_id_list.append(chain_id)\n self.chain_name_list.append(chain_name)\n self.groups_per_chain.append(num_groups)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_entity_info(self, chain_indices, sequence, description, entity_type):\n self.entity_list.append(make_entity_dict(chain_indices,sequence,description,entity_type))", "response": "Set the entity level information for the structure."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the group level information for a group", "response": "def set_group_info(self, group_name, group_number, insertion_code,\n group_type, atom_count, bond_count, single_letter_code,\n sequence_index, secondary_structure_type):\n \"\"\"Set the information for a group\n :param group_name: the name of this group,e.g. LYS\n :param group_number: the residue number of this group\n :param insertion_code: the insertion code for this group\n :param group_type: a string indicating the type of group (as found in the chemcomp dictionary.\n Empty string if none available.\n :param atom_count: the number of atoms in the group\n :param bond_count: the number of unique bonds in the group\n :param single_letter_code: the single letter code of the group\n :param sequence_index: the index of this group in the sequence defined by the enttiy\n :param secondary_structure_type: the type of secondary structure used (types are according to DSSP and\n number to type mappings are defined in the specification)\n \"\"\"\n # Add the group to the overall list - unless it's the first time round\n if self.current_group is not None:\n self.group_list.append(self.current_group)\n\n # Add the group level information\n self.group_id_list.append(group_number)\n self.ins_code_list.append(insertion_code)\n self.sequence_index_list.append(sequence_index)\n self.sec_struct_list.append(secondary_structure_type)\n self.current_group = Group()\n self.current_group.group_name = group_name\n self.current_group.group_type = group_type\n self.current_group.single_letter_code = single_letter_code"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the crystallographic information for the structure", "response": "def set_xtal_info(self, space_group, unit_cell):\n \"\"\"Set the crystallographic information for the structure\n :param space_group: the space group name, e.g. \"P 21 21 21\"\n :param unit_cell: an array of length 6 with the unit cell parameters in order: a, b, c, alpha, beta, gamma\n \"\"\"\n self.space_group = space_group\n self.unit_cell = unit_cell"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_header_info(self, r_free, r_work, resolution, title,\n deposition_date, release_date, experimental_methods):\n \"\"\"Sets the header information.\n :param r_free: the measured R-Free for the structure\n :param r_work: the measure R-Work for the structure\n :param resolution: the resolution of the structure\n :param title: the title of the structure\n :param deposition_date: the deposition date of the structure\n :param release_date: the release date of the structure\n :param experimnetal_methods: the list of experimental methods in the structure\n \"\"\"\n self.r_free = r_free\n self.r_work = r_work\n self.resolution = resolution\n self.title = title\n self.deposition_date = deposition_date\n self.release_date = release_date\n self.experimental_methods = experimental_methods", "response": "Sets the header information for the structure containing the entries in the structure containing the header information."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_bio_assembly_trans(self, bio_assembly_index, input_chain_indices, input_transform):\n this_bioass = None\n for bioass in self.bio_assembly:\n if bioass['name'] == str(bio_assembly_index):\n this_bioass = bioass\n break\n if not this_bioass:\n this_bioass = {\"name\": str(bio_assembly_index), 'transformList': []}\n else:\n self.bio_assembly.remove(this_bioass)\n this_bioass['transformList'].append({'chainIndexList':input_chain_indices,'matrix': input_transform})\n self.bio_assembly.append(this_bioass)", "response": "Set the Bioassembly transformation information. A single bioassembly can have multiple transforms."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd bonds within a group.", "response": "def set_group_bond(self, atom_index_one, atom_index_two, bond_order):\n \"\"\"Add bonds within a group.\n :param atom_index_one: the integer atom index (in the group) of the first partner in the bond\n :param atom_index_two: the integer atom index (in the group) of the second partner in the bond\n :param bond_order: the integer bond order\n \"\"\"\n self.current_group.bond_atom_list.append(atom_index_one)\n self.current_group.bond_atom_list.append(atom_index_two)\n self.current_group.bond_order_list.append(bond_order)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_inter_group_bond(self, atom_index_one, atom_index_two, bond_order):\n self.bond_atom_list.append(atom_index_one)\n self.bond_atom_list.append(atom_index_two)\n self.bond_order_list.append(bond_order)", "response": "Add bonds between groups. A bond is added to the list of bonds that are connected to the first and second partner in the bond. A bond is added to the list of bonds that are connected to the second partner in the bond."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run_length_encode(in_array):\n if(len(in_array)==0):\n return []\n curr_ans = in_array[0]\n out_array = [curr_ans]\n counter = 1\n for in_int in in_array[1:]:\n if in_int == curr_ans:\n counter+=1\n else:\n out_array.append(counter)\n out_array.append(in_int)\n curr_ans = in_int\n counter = 1\n # Add the final counter\n out_array.append(counter)\n return out_array", "response": "A function to run length decode an integer array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delta_encode(in_array):\n if(len(in_array)==0):\n return []\n curr_ans = in_array[0]\n out_array = [curr_ans]\n for in_int in in_array[1:]:\n out_array.append(in_int-curr_ans)\n curr_ans = in_int\n return out_array", "response": "A function to delta decode an integer array."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decode_array(input_array):\n codec, length, param, input_array = parse_header(input_array)\n return codec_dict[codec].decode(input_array, param)", "response": "Parses the input byte array and then decodes using the codec and the appropirate parameter."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencode the array using the method and then add the header to the fornt", "response": "def encode_array(input_array, codec, param):\n \"\"\"Encode the array using the method and then add the header to this array.\n\n :param input_array: the array to be encoded\n :param codec: the integer index of the codec to use\n :param param: the integer parameter to use in the function\n :return an array with the header added to the fornt\"\"\"\n return add_header(codec_dict[codec].encode(input_array, param), codec, len(input_array), param)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_length_decode(in_array):\n switch=False\n out_array=[]\n for item in in_array:\n if switch==False:\n this_item = item\n switch=True\n else:\n switch=False\n out_array.extend([this_item]*int(item))\n return out_array", "response": "A function to run length decode an int array."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delta_decode(in_array):\n if len(in_array) == 0:\n return []\n this_ans = in_array[0]\n out_array = [this_ans]\n for i in range(1, len(in_array)):\n this_ans += in_array[i]\n out_array.append(this_ans)\n return out_array", "response": "A function to delta decode an int array."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a byte array into an integer array.", "response": "def convert_bytes_to_ints(in_bytes, num):\n \"\"\"Convert a byte array into an integer array. The number of bytes forming an integer\n is defined by num\n\n :param in_bytes: the input bytes\n :param num: the number of bytes per int\n :return the integer array\"\"\"\n dt = numpy.dtype('>i' + str(num))\n return numpy.frombuffer(in_bytes, dt)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef decode_chain_list(in_bytes):\n bstrings = numpy.frombuffer(in_bytes, numpy.dtype('S' + str(mmtf.utils.constants.CHAIN_LEN)))\n return [s.decode(\"ascii\").strip(mmtf.utils.constants.NULL_BYTE) for s in bstrings]", "response": "Convert a list of bytes to a list of strings. Each string is of length mmtf. CHAIN_LEN\n ."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nunpacking an array of integers using recursive indexing.", "response": "def recursive_index_decode(int_array, max=32767, min=-32768):\n \"\"\"Unpack an array of integers using recursive indexing.\n\n :param int_array: the input array of integers\n :param max: the maximum integer size\n :param min: the minimum integer size\n :return the array of integers after recursive index decoding\"\"\"\n out_arr = []\n decoded_val = 0\n for item in int_array.tolist():\n if item==max or item==min:\n decoded_val += item\n else:\n decoded_val += item\n out_arr.append(decoded_val)\n decoded_val = 0\n return numpy.asarray(out_arr,dtype=numpy.int32)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef decode_data(self, input_data):\n self.group_type_list = decode_array(input_data[\"groupTypeList\"])\n self.x_coord_list = decode_array(input_data[\"xCoordList\"])\n self.y_coord_list = decode_array(input_data[\"yCoordList\"])\n self.z_coord_list = decode_array(input_data[\"zCoordList\"])\n if \"bFactorList\" in input_data:\n self.b_factor_list = decode_array(input_data[\"bFactorList\"])\n else:\n self.b_factor_list = []\n if \"occupancyList\" in input_data:\n self.occupancy_list = decode_array(input_data[\"occupancyList\"])\n else:\n self.occupancy_list = []\n if \"atomIdList\" in input_data:\n self.atom_id_list = decode_array(input_data[\"atomIdList\"])\n else:\n self.atom_id_list = []\n if \"altLocList\" in input_data:\n self.alt_loc_list = decode_array(input_data[\"altLocList\"])\n else:\n self.alt_loc_list = []\n if \"insCodeList\" in input_data:\n self.ins_code_list = decode_array(input_data[\"insCodeList\"])\n else:\n self.ins_code_list = []\n self.group_id_list = decode_array(input_data[\"groupIdList\"])\n self.group_list = input_data[\"groupList\"]\n if \"sequenceIndexList\" in input_data:\n self.sequence_index_list = decode_array(input_data[\"sequenceIndexList\"])\n else:\n self.sequence_index_list = []\n self.chains_per_model = input_data[\"chainsPerModel\"]\n self.groups_per_chain = input_data[\"groupsPerChain\"]\n if \"chainNameList\" in input_data:\n self.chain_name_list = decode_array(input_data[\"chainNameList\"])\n else:\n self.chain_name_list = []\n self.chain_id_list = decode_array(input_data[\"chainIdList\"])\n if \"spaceGroup\" in input_data:\n self.space_group = input_data[\"spaceGroup\"]\n else:\n self.space_group = None\n if \"bondAtomList\" in input_data:\n self.bond_atom_list = decode_array(input_data[\"bondAtomList\"])\n else:\n self.bond_atom_list = None\n if \"bondOrderList\" in input_data:\n self.bond_order_list = decode_array(input_data[\"bondOrderList\"])\n else:\n self.bond_order_list = None\n if sys.version_info[0] < 3:\n if \"mmtfVersion\" in input_data:\n self.mmtf_version = input_data[\"mmtfVersion\"]\n else:\n self.mmtf_version = None\n if \"mmtfProducer\" in input_data:\n self.mmtf_producer = input_data[\"mmtfProducer\"]\n else:\n self.mmtf_producer = None\n if \"structureId\" in input_data:\n self.structure_id = input_data[\"structureId\"]\n else:\n self.structure_id = None\n else:\n if \"mmtfVersion\" in input_data:\n self.mmtf_version = input_data[\"mmtfVersion\"]\n else:\n self.mmtf_version = None\n if \"mmtfProducer\" in input_data:\n self.mmtf_producer = input_data[\"mmtfProducer\"]\n else:\n self.mmtf_producer = None\n if \"structureId\" in input_data:\n self.structure_id = input_data[\"structureId\"]\n else:\n self.structure_id = None\n if \"title\" in input_data:\n if sys.version_info[0] < 3:\n self.title = input_data[\"title\"]\n else:\n self.title = input_data[\"title\"]\n if \"experimentalMethods\" in input_data:\n self.experimental_methods = input_data[\"experimentalMethods\"]\n else:\n self.experimental_methods = None\n if \"depositionDate\" in input_data:\n self.deposition_date = input_data[\"depositionDate\"]\n else:\n self.deposition_date = None\n if \"releaseDate\" in input_data:\n self.release_date = input_data[\"releaseDate\"]\n else:\n self.release_date = None\n if \"entityList\" in input_data:\n self.entity_list = input_data[\"entityList\"]\n else:\n self.entity_list = []\n if \"bioAssemblyList\" in input_data:\n self.bio_assembly = input_data[\"bioAssemblyList\"]\n else:\n self.bio_assembly = []\n if \"rFree\" in input_data:\n self.r_free = input_data[\"rFree\"]\n else:\n self.r_free = None\n if \"rWork\" in input_data:\n self.r_work = input_data[\"rWork\"]\n else:\n self.r_work = None\n if \"resolution\" in input_data:\n self.resolution = input_data[\"resolution\"]\n else:\n self.resolution = None\n if \"unitCell\" in input_data:\n self.unit_cell = input_data[\"unitCell\"]\n else:\n self.unit_cell = None\n if \"secStructList\" in input_data:\n self.sec_struct_list = decode_array(input_data[\"secStructList\"])\n # Now all the numbers to defien the\n self.num_bonds = int(input_data[\"numBonds\"])\n self.num_chains = int(input_data[\"numChains\"])\n self.num_models = int(input_data[\"numModels\"])\n self.num_atoms = int(input_data[\"numAtoms\"])\n self.num_groups = int(input_data[\"numGroups\"])", "response": "Function to decode the input data and place it onto the class."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pass_data_on(self, data_setters):\n data_setters.init_structure(self.num_bonds, len(self.x_coord_list), len(self.group_type_list),\n len(self.chain_id_list), len(self.chains_per_model), self.structure_id)\n decoder_utils.add_entity_info(self, data_setters)\n decoder_utils.add_atomic_information(self, data_setters)\n decoder_utils.add_header_info(self, data_setters)\n decoder_utils.add_xtalographic_info(self, data_setters)\n decoder_utils.generate_bio_assembly(self, data_setters)\n decoder_utils.add_inter_group_bonds(self, data_setters)\n data_setters.finalize_structure()", "response": "Write the data from the getters to the setters."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _internet_on(address):\n try:\n urllib2.urlopen(address, timeout=1)\n return True\n except urllib2.URLError as err:\n return False", "response": "Checks to see if the internet is on by pinging the given IP or address."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_mmtf(file_path, input_data, input_function):\n mmtf_encoder = MMTFEncoder()\n pass_data_on(input_data, input_function, mmtf_encoder)\n mmtf_encoder.write_file(file_path)", "response": "API function to write data as MMTF to a file"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the msgpack unpacked data given a PDB id.", "response": "def get_raw_data_from_url(pdb_id, reduced=False):\n \"\"\"\" Get the msgpack unpacked data given a PDB id.\n\n :param pdb_id: the input PDB id\n :return the unpacked data (a dict) \"\"\"\n url = get_url(pdb_id,reduced)\n request = urllib2.Request(url)\n request.add_header('Accept-encoding', 'gzip')\n response = urllib2.urlopen(request)\n if response.info().get('Content-Encoding') == 'gzip':\n data = ungzip_data(response.read())\n else:\n data = response.read()\n return _unpack(data)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a decoded API to the data from a file path.", "response": "def parse(file_path):\n \"\"\"Return a decoded API to the data from a file path.\n\n :param file_path: the input file path. Data is not entropy compressed (e.g. gzip)\n :return an API to decoded data \"\"\"\n newDecoder = MMTFDecoder()\n with open(file_path, \"rb\") as fh:\n newDecoder.decode_data(_unpack(fh))\n return newDecoder"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_gzip(file_path):\n newDecoder = MMTFDecoder()\n newDecoder.decode_data(_unpack(gzip.open(file_path, \"rb\")))\n return newDecoder", "response": "Return a decoded API to the data from a gzip compressed file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ungzip_data(input_data):\n buf = StringIO(input_data)\n f = gzip.GzipFile(fileobj=buf)\n return f", "response": "Returns a string of data after gzip decoding\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_header(input_array):\n codec = struct.unpack(mmtf.utils.constants.NUM_DICT[4], input_array[0:4])[0]\n length = struct.unpack(mmtf.utils.constants.NUM_DICT[4], input_array[4:8])[0]\n param = struct.unpack(mmtf.utils.constants.NUM_DICT[4], input_array[8:12])[0]\n return codec,length,param,input_array[12:]", "response": "Parse the header and return it along with the input array minus the header."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_header(input_array, codec, length, param):\n return struct.pack(mmtf.utils.constants.NUM_DICT[4], codec) + \\\n struct.pack(mmtf.utils.constants.NUM_DICT[4], length) + \\\n struct.pack(mmtf.utils.constants.NUM_DICT[4], param) + input_array", "response": "Add the header to the appropriate array."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a byte array into an integer array.", "response": "def convert_bytes_to_ints(in_bytes, num):\n \"\"\"Convert a byte array into an integer array. The number of bytes forming an integer\n is defined by num\n :param in_bytes: the input bytes\n :param num: the number of bytes per int\n :return the integer array\"\"\"\n out_arr = []\n for i in range(len(in_bytes)//num):\n val = in_bytes[i * num:i * num + num]\n unpacked = struct.unpack(mmtf.utils.constants.NUM_DICT[num], val)\n out_arr.append(unpacked[0])\n return out_arr"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert_ints_to_bytes(in_ints, num):\n out_bytes= b\"\"\n for val in in_ints:\n out_bytes+=struct.pack(mmtf.utils.constants.NUM_DICT[num], val)\n return out_bytes", "response": "Convert an integer array into a byte array."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef decode_chain_list(in_bytes):\n tot_strings = len(in_bytes) // mmtf.utils.constants.CHAIN_LEN\n out_strings = []\n for i in range(tot_strings):\n out_s = in_bytes[i * mmtf.utils.constants.CHAIN_LEN:i * mmtf.utils.constants.CHAIN_LEN + mmtf.utils.constants.CHAIN_LEN]\n out_strings.append(out_s.decode(\"ascii\").strip(mmtf.utils.constants.NULL_BYTE))\n return out_strings", "response": "Convert a list of bytes to a list of strings. Each string is of length mmtf. CHAIN_LEN\n ."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a list of strings to a list of byte arrays.", "response": "def encode_chain_list(in_strings):\n \"\"\"Convert a list of strings to a list of byte arrays.\n\n :param in_strings: the input strings\n :return the encoded list of byte arrays\"\"\"\n out_bytes = b\"\"\n for in_s in in_strings:\n out_bytes+=in_s.encode('ascii')\n for i in range(mmtf.utils.constants.CHAIN_LEN -len(in_s)):\n out_bytes+= mmtf.utils.constants.NULL_BYTE.encode('ascii')\n return out_bytes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npack an integer array using recursive indexing.", "response": "def recursive_index_encode(int_array, max=32767, min=-32768):\n \"\"\"Pack an integer array using recursive indexing.\n\n :param int_array: the input array of integers\n :param max: the maximum integer size\n :param min: the minimum integer size\n :return the array of integers after recursive index encoding\"\"\"\n out_arr = []\n for curr in int_array:\n if curr >= 0 :\n while curr >= max:\n out_arr.append(max)\n curr -= max\n else:\n while curr <= min:\n out_arr.append(min)\n curr += int(math.fabs(min))\n out_arr.append(curr)\n return out_arr"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nunpack an array of integers using recursive indexing.", "response": "def recursive_index_decode(int_array, max=32767, min=-32768):\n \"\"\"Unpack an array of integers using recursive indexing.\n :param int_array: the input array of integers\n :param max: the maximum integer size\n :param min: the minimum integer size\n :return the array of integers after recursive index decoding\"\"\"\n out_arr = []\n encoded_ind = 0\n while encoded_ind < len(int_array):\n decoded_val = 0\n while int_array[encoded_ind]==max or int_array[encoded_ind]==min:\n decoded_val += int_array[encoded_ind]\n encoded_ind+=1\n if int_array[encoded_ind]==0:\n break\n decoded_val += int_array[encoded_ind]\n encoded_ind+=1\n out_arr.append(decoded_val)\n return out_arr"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds and return an optimizer for the rosenbrock function.", "response": "def build(algo, init):\n '''Build and return an optimizer for the rosenbrock function.\n\n In downhill, an optimizer can be constructed using the build() top-level\n function. This function requires several Theano quantities such as the loss\n being optimized and the parameters to update during optimization.\n '''\n x = theano.shared(np.array(init, FLOAT), name='x')\n n = 0.1 * RandomStreams().normal((len(init) - 1, ))\n monitors = []\n if len(init) == 2:\n # this gives us access to the x and y locations during optimization.\n monitors.extend([('x', x[:-1].sum()), ('y', x[1:].sum())])\n return downhill.build(\n algo,\n loss=(n + 100 * (x[1:] - x[:-1] ** 2) ** 2 + (1 - x[:-1]) ** 2).sum(),\n params=[x],\n monitors=monitors,\n monitor_gradients=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning an optimizer on the rosenbrock function. Return xs ys and losses.", "response": "def build_and_trace(algo, init, limit=100, **kwargs):\n '''Run an optimizer on the rosenbrock function. Return xs, ys, and losses.\n\n In downhill, optimization algorithms can be iterated over to progressively\n minimize the loss. At each iteration, the optimizer yields a dictionary of\n monitor values that were computed during that iteration. Here we build an\n optimizer and then run it for a fixed number of iterations.\n '''\n kw = dict(min_improvement=0, patience=0, max_gradient_norm=100)\n kw.update(kwargs)\n xs, ys, loss = [], [], []\n for tm, _ in build(algo, init).iterate([[]], **kw):\n if len(init) == 2:\n xs.append(tm['x'])\n ys.append(tm['y'])\n loss.append(tm['loss'])\n if len(loss) == limit:\n break\n # Return the optimization up to any failure of patience.\n return xs[:-9], ys[:-9], loss[-9]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nminimizing a loss function with respect to some symbolic parameters.", "response": "def minimize(loss, train, valid=None, params=None, inputs=None, algo='rmsprop',\n updates=(), monitors=(), monitor_gradients=False, batch_size=32,\n train_batches=None, valid_batches=None, **kwargs):\n '''Minimize a loss function with respect to some symbolic parameters.\n\n Additional keyword arguments are passed to the underlying :class:`Optimizer\n ` instance.\n\n Parameters\n ----------\n loss : Theano expression\n Loss function to minimize. This must be a scalar-valued expression.\n train : :class:`Dataset `, ndarray, or callable\n Dataset to use for computing gradient updates.\n valid : :class:`Dataset `, ndarray, or callable, optional\n Dataset to use for validating the minimization process. The training\n dataset is used if this is not provided.\n params : list of Theano variables, optional\n Symbolic variables to adjust to minimize the loss. If not given, these\n will be computed automatically by walking the computation graph.\n inputs : list of Theano variables, optional\n Symbolic variables required to compute the loss. If not given, these\n will be computed automatically by walking the computation graph.\n algo : str, optional\n Name of the minimization algorithm to use. Must be one of the strings\n that can be passed to :func:`build`. Defaults to ``'rmsprop'``.\n updates : list of update pairs, optional\n A list of pairs providing updates for the internal of the loss\n computation. Normally this is empty, but it can be provided if the loss,\n for example, requires an update to an internal random number generator.\n monitors : dict or sequence of (str, Theano expression) tuples, optional\n Additional values to monitor during optimization. These must be provided\n as either a sequence of (name, expression) tuples, or as a dictionary\n mapping string names to Theano expressions.\n monitor_gradients : bool, optional\n If True, add monitors to log the norms of the parameter gradients during\n optimization. Defaults to False.\n batch_size : int, optional\n Size of batches provided by datasets. Defaults to 32.\n train_batches : int, optional\n Number of batches of training data to iterate over during one pass of\n optimization. Defaults to None, which uses the entire training dataset.\n valid_batches : int, optional\n Number of batches of validation data to iterate over during one pass of\n validation. Defaults to None, which uses the entire validation dataset.\n\n Returns\n -------\n train_monitors : dict\n A dictionary mapping monitor names to monitor values. This dictionary\n will always contain the ``'loss'`` key, giving the value of the loss\n evaluated on the training dataset.\n valid_monitors : dict\n A dictionary mapping monitor names to monitor values, evaluated on the\n validation dataset. This dictionary will always contain the ``'loss'``\n key, giving the value of the loss function. Because validation is not\n always computed after every optimization update, these monitor values\n may be \"stale\"; however, they will always contain the most recently\n computed values.\n '''\n if not isinstance(train, Dataset):\n train = Dataset(\n train,\n name='train',\n batch_size=batch_size,\n iteration_size=train_batches,\n )\n if valid is not None and not isinstance(valid, Dataset):\n valid = Dataset(\n valid,\n name='valid',\n batch_size=batch_size,\n iteration_size=valid_batches,\n )\n return build(\n algo,\n loss=loss,\n params=params,\n inputs=inputs,\n updates=updates,\n monitors=monitors,\n monitor_gradients=monitor_gradients,\n ).minimize(train, valid, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a legend label for an optimization run.", "response": "def make_label(loss, key):\n '''Create a legend label for an optimization run.'''\n algo, rate, mu, half, reg = key\n slots, args = ['{:.3f}', '{}', 'm={:.3f}'], [loss, algo, mu]\n if algo in 'SGD NAG RMSProp Adam ESGD'.split():\n slots.append('lr={:.2e}')\n args.append(rate)\n if algo in 'RMSProp ADADELTA ESGD'.split():\n slots.append('rmsh={}')\n args.append(half)\n slots.append('rmsr={:.2e}')\n args.append(reg)\n return ' '.join(slots).format(*args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\niterating over batches in the dataset and return them as a generator.", "response": "def iterate(self, shuffle=True):\n '''Iterate over batches in the dataset.\n\n This method generates ``iteration_size`` batches from the dataset and\n then returns.\n\n Parameters\n ----------\n shuffle : bool, optional\n Shuffle the batches in this dataset if the iteration reaches the end\n of the batch list. Defaults to True.\n\n Yields\n ------\n batches : data batches\n A sequence of batches---often from a training, validation, or test\n dataset.\n '''\n for _ in range(self.iteration_size):\n if self._callable is not None:\n yield self._callable()\n else:\n yield self._next_batch(shuffle)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef shared_like(param, suffix, init=0):\n '''Create a Theano shared variable like an existing parameter.\n\n Parameters\n ----------\n param : Theano variable\n Theano variable to use for shape information.\n suffix : str\n Suffix to append to the parameter's name for the new variable.\n init : float or ndarray, optional\n Initial value of the shared variable. Defaults to 0.\n\n Returns\n -------\n shared : Theano shared variable\n A new shared variable with the same shape and data type as ``param``.\n '''\n return theano.shared(np.zeros_like(param.get_value()) + init,\n name='{}_{}'.format(param.name, suffix),\n broadcastable=param.broadcastable)", "response": "Create a Theano shared variable like an existing parameter."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_inputs_and_params(node):\n '''Walk a computation graph and extract root variables.\n\n Parameters\n ----------\n node : Theano expression\n A symbolic Theano expression to walk.\n\n Returns\n -------\n inputs : list Theano variables\n A list of candidate inputs for this graph. Inputs are nodes in the graph\n with no parents that are not shared and are not constants.\n params : list of Theano shared variables\n A list of candidate parameters for this graph. Parameters are nodes in\n the graph that are shared variables.\n '''\n queue, seen, inputs, params = [node], set(), set(), set()\n while queue:\n node = queue.pop()\n seen.add(node)\n queue.extend(p for p in node.get_parents() if p not in seen)\n if not node.get_parents():\n if isinstance(node, theano.compile.SharedVariable):\n params.add(node)\n elif not isinstance(node, TT.Constant):\n inputs.add(node)\n return list(inputs), list(params)", "response": "Walk a computation graph and extract root variables."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlogs a message to the console.", "response": "def log(msg, *args, **kwargs):\n '''Log a message to the console.\n\n Parameters\n ----------\n msg : str\n A string to display on the console. This can contain {}-style\n formatting commands; the remaining positional and keyword arguments\n will be used to fill them in.\n '''\n now = datetime.datetime.now()\n module = 'downhill'\n if _detailed_callsite:\n caller = inspect.stack()[1]\n parts = caller.filename.replace('.py', '').split('/')\n module = '{}:{}'.format(\n '.'.join(parts[parts.index('downhill')+1:]), caller.lineno)\n click.echo(' '.join((\n click.style(now.strftime('%Y%m%d'), fg='blue'),\n click.style(now.strftime('%H%M%S'), fg='cyan'),\n click.style(module, fg='magenta'),\n msg.format(*args, **kwargs),\n )))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef log_param(name, value):\n '''Log a parameter value to the console.\n\n Parameters\n ----------\n name : str\n Name of the parameter being logged.\n value : any\n Value of the parameter being logged.\n '''\n log('setting {} = {}', click.style(str(name)),\n click.style(str(value), fg='yellow'))", "response": "Log a parameter value to the console."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads the MNIST digits dataset.", "response": "def load_mnist():\n '''Load the MNIST digits dataset.'''\n mnist = skdata.mnist.dataset.MNIST()\n mnist.meta # trigger download if needed.\n\n def arr(n, dtype):\n arr = mnist.arrays[n]\n return arr.reshape((len(arr), -1)).astype(dtype)\n train_images = arr('train_images', np.float32) / 128 - 1\n train_labels = arr('train_labels', np.uint8)\n return ((train_images[:50000], train_labels[:50000, 0]),\n (train_images[50000:], train_labels[50000:, 0]))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconstructs an optimizer by name.", "response": "def build(algo, loss, params=None, inputs=None, updates=(), monitors=(),\n monitor_gradients=False):\n '''Construct an optimizer by name.\n\n Parameters\n ----------\n algo : str\n The name of the optimization algorithm to build.\n loss : Theano expression\n Loss function to minimize. This must be a scalar-valued expression.\n params : list of Theano variables, optional\n Symbolic variables to adjust to minimize the loss. If not given, these\n will be computed automatically by walking the computation graph.\n inputs : list of Theano variables, optional\n Symbolic variables required to compute the loss. If not given, these\n will be computed automatically by walking the computation graph.\n updates : list of update pairs, optional\n A list of pairs providing updates for the internal of the loss\n computation. Normally this is empty, but it can be provided if the loss,\n for example, requires an update to an internal random number generator.\n monitors : dict or sequence of (str, Theano expression) tuples, optional\n Additional values to monitor during optimization. These must be provided\n as either a sequence of (name, expression) tuples, or as a dictionary\n mapping string names to Theano expressions.\n monitor_gradients : bool, optional\n If True, add monitors to log the norms of the parameter gradients during\n optimization. Defaults to False.\n\n Returns\n -------\n optimizer : :class:`Optimizer`\n An optimizer instance.\n '''\n return Optimizer.build(algo, loss, params, inputs,\n updates=updates, monitors=monitors,\n monitor_gradients=monitor_gradients)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _compile(self, **kwargs):\n '''Compile the Theano functions for evaluating and updating our model.\n '''\n util.log('compiling evaluation function')\n self.f_eval = theano.function(self._inputs,\n self._monitor_exprs,\n updates=self._updates,\n name='evaluation')\n label = self.__class__.__name__\n util.log('compiling {} optimizer'.format(click.style(label, fg='red')))\n updates = list(self._updates) + list(self.get_updates(**kwargs))\n self.f_step = theano.function(self._inputs,\n self._monitor_exprs,\n updates=updates,\n name=label)", "response": "Compile the Theano functions for evaluating and updating our model."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nyield the set of parameter updates for performing optimization.", "response": "def get_updates(self, **kwargs):\n '''Get parameter update expressions for performing optimization.\n\n Keyword arguments can be applied here to set any of the global\n optimizer attributes.\n\n Yields\n ------\n updates : (parameter, expression) tuples\n A sequence of parameter updates to be applied during optimization.\n '''\n self._prepare(**kwargs)\n for param, grad in self._differentiate():\n for var, update in self._get_updates_for(param, grad):\n # For auxiliary variables, updates are meant to replace the\n # existing variable value.\n if var != param:\n yield var, update\n continue\n # If momentum is disabled, just apply the parameter delta.\n if self.momentum == 0:\n yield var, param - update\n continue\n # Momentum is enabled, so we keep track of velocity here.\n vel_tm1 = util.shared_like(param, 'vel')\n vel_t = util.as_float(self.momentum) * vel_tm1 - update\n if self.nesterov:\n # see http://arxiv.org/pdf/1212.0901v2.pdf (eq 7) and\n # https://github.com/lisa-lab/pylearn2/pull/136#issuecomment-10381617\n mom_sqr = util.as_float(self.momentum ** 2)\n mom_inc = util.as_float(1 + self.momentum)\n vel_t = mom_sqr * vel_tm1 - mom_inc * update\n yield vel_tm1, vel_t\n yield param, param + vel_t"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a sequence of gradients for our parameters.", "response": "def _differentiate(self, params=None):\n '''Return a sequence of gradients for our parameters.\n\n If this optimizer has been configured with a gradient norm limit, or\n with elementwise gradient clipping, this method applies the appropriate\n rescaling and clipping operations before returning the gradient.\n\n Parameters\n ----------\n params : list of Theano variables, optional\n Return the gradient with respect to these parameters. Defaults to\n all parameters that the optimizer knows about.\n\n Yields\n ------\n pairs : (param, grad) tuples\n Generates a sequence of tuples representing each of the parameters\n requested and the corresponding Theano gradient expressions.\n '''\n if params is None:\n params = self._params\n for param, grad in zip(params, TT.grad(self._loss, params)):\n if self.max_gradient_elem > 0:\n limit = util.as_float(self.max_gradient_elem)\n yield param, TT.clip(grad, -limit, limit)\n elif self.max_gradient_norm > 0:\n norm = TT.sqrt((grad * grad).sum())\n limit = util.as_float(self.max_gradient_norm)\n yield param, grad * TT.minimum(1, limit / norm)\n else:\n yield param, grad"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the values of the parameters to the given target values.", "response": "def set_params(self, targets=None):\n '''Set the values of the parameters to the given target values.\n\n Parameters\n ----------\n targets : sequence of ndarray, optional\n Arrays for setting the parameters of our model. If this is not\n provided, the current best parameters for this optimizer will be\n used.\n '''\n if not isinstance(targets, (list, tuple)):\n targets = self._best_params\n for param, target in zip(self._params, targets):\n param.set_value(target)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlog the state of the optimizer on the console.", "response": "def _log(self, monitors, iteration, label='', suffix=''):\n '''Log the state of the optimizer on the console.\n\n Parameters\n ----------\n monitors : OrderedDict\n A dictionary of monitor names mapped to values. These names and\n values are what is being logged.\n iteration : int\n Optimization iteration that we are logging.\n label : str, optional\n A label for the name of the optimizer creating the log line.\n Defaults to the name of the current class.\n suffix : str, optional\n A suffix to add to the end of the log line, if any.\n '''\n label = label or self.__class__.__name__\n fields = (('{}={:.6f}').format(k, v) for k, v in monitors.items())\n util.log('{} {} {}{}'.format(label, iteration, ' '.join(fields), suffix))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef evaluate(self, dataset):\n '''Evaluate the current model parameters on a dataset.\n\n Parameters\n ----------\n dataset : :class:`Dataset `\n A set of data to use for evaluating the model.\n\n Returns\n -------\n monitors : OrderedDict\n A dictionary mapping monitor names to values. Monitors are\n quantities of interest during optimization---for example, loss\n function, accuracy, or whatever the optimization task requires.\n '''\n if dataset is None:\n values = [self.f_eval()]\n else:\n values = [self.f_eval(*x) for x in dataset]\n monitors = zip(self._monitor_names, np.mean(values, axis=0))\n return collections.OrderedDict(monitors)", "response": "Evaluate the current model parameters on a dataset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset up properties for optimization.", "response": "def _prepare(self, **kwargs):\n '''Set up properties for optimization.\n\n This method can be overridden by base classes to provide parameters that\n are specific to a particular optimization technique (e.g., setting up a\n learning rate value).\n '''\n self.learning_rate = util.as_float(kwargs.pop('learning_rate', 1e-4))\n self.momentum = kwargs.pop('momentum', 0)\n self.nesterov = kwargs.pop('nesterov', False)\n self.patience = kwargs.get('patience', 5)\n self.validate_every = kwargs.pop('validate_every', 10)\n self.min_improvement = kwargs.pop('min_improvement', 0)\n self.max_gradient_norm = kwargs.pop('max_gradient_norm', 0)\n self.max_gradient_elem = kwargs.pop('max_gradient_elem', 0)\n\n util.log_param('patience', self.patience)\n util.log_param('validate_every', self.validate_every)\n util.log_param('min_improvement', self.min_improvement)\n util.log_param('max_gradient_norm', self.max_gradient_norm)\n util.log_param('max_gradient_elem', self.max_gradient_elem)\n util.log_param('learning_rate', self.learning_rate)\n util.log_param('momentum', self.momentum)\n util.log_param('nesterov', self.nesterov)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef iterate(self, train=None, valid=None, max_updates=None, **kwargs):\n r'''Optimize a loss iteratively using a training and validation dataset.\n\n This method yields a series of monitor values to the caller. After every\n optimization epoch, a pair of monitor dictionaries is generated: one\n evaluated on the training dataset during the epoch, and another\n evaluated on the validation dataset at the most recent validation epoch.\n\n The validation monitors might not be updated during every optimization\n iteration; in this case, the most recent validation monitors will be\n yielded along with the training monitors.\n\n Additional keyword arguments supplied here will set the global\n optimizer attributes.\n\n Parameters\n ----------\n train : sequence or :class:`Dataset `\n A set of training data for computing updates to model parameters.\n valid : sequence or :class:`Dataset `\n A set of validation data for computing monitor values and\n determining when the loss has stopped improving. Defaults to the\n training data.\n max_updates : int, optional\n If specified, halt optimization after this many gradient updates\n have been processed. If not provided, uses early stopping to decide\n when to halt.\n\n Yields\n ------\n train_monitors : dict\n A dictionary mapping monitor names to values, evaluated on the\n training dataset.\n valid_monitors : dict\n A dictionary containing monitor values evaluated on the validation\n dataset.\n '''\n self._compile(**kwargs)\n\n if valid is None:\n valid = train\n iteration = 0\n training = validation = None\n while max_updates is None or iteration < max_updates:\n if not iteration % self.validate_every:\n try:\n validation = self.evaluate(valid)\n except KeyboardInterrupt:\n util.log('interrupted!')\n break\n if self._test_patience(validation):\n util.log('patience elapsed!')\n break\n try:\n training = self._step(train)\n except KeyboardInterrupt:\n util.log('interrupted!')\n break\n iteration += 1\n self._log(training, iteration)\n yield training, validation\n self.set_params('best')", "response": "This method is used to iterate over the training and validation data for a single iteration of the optimization."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\noptimizes our loss exhaustively.", "response": "def minimize(self, *args, **kwargs):\n '''Optimize our loss exhaustively.\n\n This method is a thin wrapper over the :func:`iterate` method. It simply\n exhausts the iterative optimization process and returns the final\n monitor values.\n\n Returns\n -------\n train_monitors : dict\n A dictionary mapping monitor names to values, evaluated on the\n training dataset.\n valid_monitors : dict\n A dictionary containing monitor values evaluated on the validation\n dataset.\n '''\n monitors = None\n for monitors in self.iterate(*args, **kwargs):\n pass\n return monitors"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadvance the state of the optimizer by one step.", "response": "def _step(self, dataset):\n '''Advance the state of the optimizer by one step.\n\n Parameters\n ----------\n dataset : :class:`Dataset `\n A dataset for optimizing the model.\n\n Returns\n -------\n train_monitors : dict\n A dictionary mapping monitor names to values.\n '''\n if dataset is None:\n values = [self.f_step()]\n else:\n values = [self.f_step(*x) for x in dataset]\n return collections.OrderedDict(\n zip(self._monitor_names, np.mean(values, axis=0)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if the given method will accept the given number of arguments AttributeNames", "response": "def accept_arguments(method, number_of_arguments=1):\n \"\"\"Returns True if the given method will accept the given number of arguments\n\n method: the method to perform introspection on\n number_of_arguments: the number_of_arguments\n \"\"\"\n if 'method' in method.__class__.__name__:\n number_of_arguments += 1\n func = getattr(method, 'im_func', getattr(method, '__func__'))\n func_defaults = getattr(func, 'func_defaults', getattr(func, '__defaults__'))\n number_of_defaults = func_defaults and len(func_defaults) or 0\n elif method.__class__.__name__ == 'function':\n func_defaults = getattr(method, 'func_defaults', getattr(method, '__defaults__'))\n number_of_defaults = func_defaults and len(func_defaults) or 0\n\n coArgCount = getattr(method, 'func_code', getattr(method, '__code__')).co_argcount\n if(coArgCount >= number_of_arguments and coArgCount - number_of_defaults <= number_of_arguments):\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nemit a signal causing all slot methods connected with the signal to be called with the given value.", "response": "def emit(self, signal, value=None, gather=False):\n \"\"\"Emits a signal, causing all slot methods connected with the signal to be called (optionally w/ related value)\n\n signal: the name of the signal to emit, must be defined in the classes 'signals' list.\n value: the value to pass to all connected slot methods.\n gather: if set, causes emit to return a list of all slot results\n \"\"\"\n results = [] if gather else True\n if hasattr(self, 'connections') and signal in self.connections:\n for condition, values in self.connections[signal].items():\n if condition is None or condition == value or (callable(condition) and condition(value)):\n for slot, transform in values.items():\n if transform is not None:\n if callable(transform):\n used_value = transform(value)\n elif isinstance(transform, str):\n used_value = transform.format(value=value)\n else:\n used_value = transform\n else:\n used_value = value\n\n if used_value is not None:\n if(accept_arguments(slot, 1)):\n result = slot(used_value)\n elif(accept_arguments(slot, 0)):\n result = slot()\n else:\n result = ''\n else:\n result = slot()\n\n if gather:\n results.append(result)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndefines a connection between this object and another object.", "response": "def connect(self, signal, slot, transform=None, condition=None):\n \"\"\"Defines a connection between this objects signal and another objects slot\n\n signal: the signal this class will emit, to cause the slot method to be called\n receiver: the object containing the slot method to be called\n slot: the slot method to call\n transform: an optional value override to pass into the slot method as the first variable\n condition: only call the slot if the value emitted matches the required value or calling required returns True\n \"\"\"\n if not signal in self.signals:\n print(\"WARNING: {0} is trying to connect a slot to an undefined signal: {1}\".format(self.__class__.__name__,\n str(signal)))\n return\n\n if not hasattr(self, 'connections'):\n self.connections = {}\n connection = self.connections.setdefault(signal, {})\n connection = connection.setdefault(condition, {})\n connection[slot] = transform"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves all connections between this object signal and connected slot ( s", "response": "def disconnect(self, signal=None, slot=None, transform=None, condition=None):\n \"\"\"Removes connection(s) between this objects signal and connected slot(s)\n\n signal: the signal this class will emit, to cause the slot method to be called\n receiver: the object containing the slot method to be called\n slot: the slot method or function to call\n transform: an optional value override to pass into the slot method as the first variable\n condition: only call the slot method if the value emitted matches this condition\n \"\"\"\n if slot:\n self.connections[signal][condition].pop(slot, None)\n elif condition is not None:\n self.connections[signal].pop(condition, None)\n elif signal:\n self.connections.pop(signal, None)\n else:\n delattr(self, 'connections')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the context data for a given app. Can be an ID or a case insensitive name.", "response": "def get(self, key):\n \"\"\" Returns context data for a given app, can be an ID or a case insensitive name \"\"\"\n keystr = str(key)\n res = None\n\n try:\n res = self.ctx[keystr]\n except KeyError:\n for k, v in self.ctx.items():\n if \"name\" in v and v[\"name\"].lower() == keystr.lower():\n res = v\n break\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef quality(self):\n try:\n qid = int((self.tool_metadata or {}).get(\"quality\", 0))\n except:\n qid = 0\n\n # We might be able to get the quality strings from the item's tags\n internal_name, name = \"normal\", \"Normal\"\n if self.tags:\n tags = {x.get('category'): x for x in self.tags}\n if 'Quality' in tags:\n internal_name, name = tags['Quality'].get('internal_name'), tags['Quality'].get('name')\n\n return qid, internal_name, name", "response": "Return the quality of the item."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the current API key.", "response": "def get(cls):\n \"\"\"Get the current API key.\n if one has not been given via 'set' the env var STEAMODD_API_KEY will\n be checked instead.\n \"\"\"\n apikey = cls.__api_key or cls.__api_key_env_var\n\n if apikey:\n return apikey\n else:\n raise APIKeyMissingError(\"API key not set\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef call(self):\n data = self._downloader.download()\n\n # Only try to pass errors arg if supported\n if sys.version >= \"2.7\":\n data = data.decode(\"utf-8\", errors=\"ignore\")\n else:\n data = data.decode(\"utf-8\")\n\n self.update(json.loads(data))\n self._fetched = True", "response": "Make the API call again and fetch fresh data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _attribute_definition(self, attrid):\n attrs = self._schema[\"attributes\"]\n\n try:\n # Make a new dict to avoid side effects\n return dict(attrs[attrid])\n except KeyError:\n attr_names = self._schema[\"attribute_names\"]\n attrdef = attrs.get(attr_names.get(str(attrid).lower()))\n\n if not attrdef:\n return None\n else:\n return dict(attrdef)", "response": "Returns the attribute definition dict of a given attribute ID"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _quality_definition(self, qid):\n qualities = self._schema[\"qualities\"]\n\n try:\n return qualities[qid]\n except KeyError:\n qid = self._schema[\"quality_names\"].get(str(qid).lower(), 0)\n return qualities.get(qid, (qid, \"normal\", \"Normal\"))", "response": "Returns the ID and localized name of the given quality"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all attributes in the schema", "response": "def attributes(self):\n \"\"\" Returns all attributes in the schema \"\"\"\n attrs = self._schema[\"attributes\"]\n return [item_attribute(attr) for attr in sorted(attrs.values(),\n key=operator.itemgetter(\"defindex\"))]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef origin_id_to_name(self, origin):\n try:\n oid = int(origin)\n except (ValueError, TypeError):\n return None\n\n return self.origins.get(oid)", "response": "Returns a localized origin name for a given ID"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of attributes that are available in the current object.", "response": "def attributes(self):\n \"\"\" Returns a list of attributes \"\"\"\n\n overridden_attrs = self._attributes\n sortmap = {\"neutral\": 1, \"positive\": 2,\n \"negative\": 3}\n\n sortedattrs = list(overridden_attrs.values())\n sortedattrs.sort(key=operator.itemgetter(\"defindex\"))\n sortedattrs.sort(key=lambda t: sortmap.get(t.get(\"effect_type\",\n \"neutral\"), 99))\n return [item_attribute(theattr) for theattr in sortedattrs]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a dict of classes that have the item equipped and in what slot", "response": "def equipped(self):\n \"\"\" Returns a dict of classes that have the item equipped and in what slot \"\"\"\n equipped = self._item.get(\"equipped\", [])\n\n # WORKAROUND: 0 is probably an off-by-one error\n # WORKAROUND: 65535 actually serves a purpose (according to Valve)\n return dict([(eq[\"class\"], eq[\"slot\"]) for eq in equipped if eq[\"class\"] != 0 and eq[\"slot\"] != 65535])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of classes that can be used by the item.", "response": "def equipable_classes(self):\n \"\"\" Returns a list of classes that _can_ use the item. \"\"\"\n sitem = self._schema_item\n\n return [c for c in sitem.get(\"used_by_classes\", self.equipped.keys()) if c]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the item in the container if there is one.", "response": "def contents(self):\n \"\"\" Returns the item in the container, if there is one.\n This will be a standard item object. \"\"\"\n rawitem = self._item.get(\"contained_item\")\n if rawitem:\n return self.__class__(rawitem, self._schema)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the item s rank if it has one.", "response": "def rank(self):\n \"\"\"\n Returns the item's rank (if it has one)\n as a dict that includes required score, name, and level.\n \"\"\"\n\n if self._rank != {}:\n # Don't bother doing attribute lookups again\n return self._rank\n\n try:\n # The eater determining the rank\n levelkey, typename, count = self.kill_eaters[0]\n except IndexError:\n # Apparently no eater available\n self._rank = None\n return None\n\n rankset = self._ranks.get(levelkey,\n [{\"level\": 0,\n \"required_score\": 0,\n \"name\": \"Strange\"}])\n\n for rank in rankset:\n self._rank = rank\n if count < rank[\"required_score\"]:\n break\n\n return self._rank"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of all styles defined for the item", "response": "def available_styles(self):\n \"\"\" Returns a list of all styles defined for the item \"\"\"\n styles = self._schema_item.get(\"styles\", [])\n\n return list(map(operator.itemgetter(\"name\"), styles))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef formatted_value(self):\n # TODO: Cleanup all of this, it's just weird and unnatural maths\n val = self.value\n pval = val\n ftype = self.value_type\n\n if ftype == \"percentage\":\n pval = int(round(val * 100))\n\n if self.type == \"negative\":\n pval = 0 - (100 - pval)\n else:\n pval -= 100\n elif ftype == \"additive_percentage\":\n pval = int(round(val * 100))\n elif ftype == \"inverted_percentage\":\n pval = 100 - int(round(val * 100))\n\n # Can't remember what workaround this was, is it needed?\n if self.type == \"negative\":\n if self.value > 1:\n pval = 0 - pval\n elif ftype == \"additive\" or ftype == \"particle_index\" or ftype == \"account_id\":\n if int(val) == val:\n pval = int(val)\n elif ftype == \"date\":\n d = time.gmtime(int(val))\n pval = time.strftime(\"%Y-%m-%d %H:%M:%S\", d)\n\n return u\"{0}\".format(pval)", "response": "Returns a formatted value for the current object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a formatted description string for the current object.", "response": "def formatted_description(self):\n \"\"\" Returns a formatted description string (%s* tokens replaced) or None if unavailable \"\"\"\n desc = self.description\n\n if desc:\n return desc.replace(\"%s1\", self.formatted_value)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef value_type(self):\n redundantprefix = \"value_is_\"\n vtype = self._attribute.get(\"description_format\")\n\n if vtype and vtype.startswith(redundantprefix):\n return vtype[len(redundantprefix):]\n else:\n return vtype", "response": "The type of the value in the item."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef account_info(self):\n account_info = self._attribute.get(\"account_info\")\n if account_info:\n return {\"persona\": account_info.get(\"personaname\", \"\"),\n \"id64\": account_info[\"steamid\"]}\n else:\n return None", "response": "A dict with two keys persona and id64."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a dict containing tags and their localized labels as values", "response": "def tags(self):\n \"\"\" Returns a dict containing tags and their localized labels as values \"\"\"\n return dict([(t, self._catalog.tags.get(t, t)) for t in self._asset.get(\"tags\", [])])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the user s vanity url if it exists None otherwise.", "response": "def vanity(self):\n \"\"\" Returns the user's vanity url if it exists, None otherwise \"\"\"\n purl = self.profile_url.strip('/')\n if purl.find(\"/id/\") != -1:\n return os.path.basename(purl)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the creation date as a localtime. struct_time struct if public", "response": "def creation_date(self):\n \"\"\" Returns the account creation date as a localtime time.struct_time\n struct if public\"\"\"\n timestamp = self._prof.get(\"timecreated\")\n if timestamp:\n return time.localtime(timestamp)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef current_game(self):\n obj = self._prof\n gameid = obj.get(\"gameid\")\n gameserverip = obj.get(\"gameserverip\")\n gameextrainfo = obj.get(\"gameextrainfo\")\n return (int(gameid) if gameid else None, gameserverip, gameextrainfo)", "response": "Returns a tuple of 3 elements each of which may be None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the user s profile level", "response": "def level(self):\n \"\"\"\n Returns the the user's profile level, note that this runs a separate\n request because the profile level data isn't in the standard player summary\n output even though it should be. Which is also why it's not implemented\n as a separate class. You won't need this output and not the profile output\n \"\"\"\n\n level_key = \"player_level\"\n\n if level_key in self._api[\"response\"]:\n return self._api[\"response\"][level_key]\n\n try:\n lvl = api.interface(\"IPlayerService\").GetSteamLevel(steamid=self.id64)[\"response\"][level_key]\n self._api[\"response\"][level_key] = lvl\n return lvl\n except:\n return -1"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_def(cls, obj):\n prof = cls(obj[\"steamid\"])\n prof._cache = obj\n\n return prof", "response": "Builds a profile object from a raw player summary object"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_status_withheld(self, status_id, user_id, countries):\n logger.info('Status %s withheld for user %s', status_id, user_id)\n return True", "response": "Called when a status is withheld"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_disconnect(self, code, stream_name, reason):\n logger.error('Disconnect message: %s %s %s', code, stream_name, reason)\n return True", "response": "Called when a disconnect is received"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling when a non - 200 status code is returned", "response": "def on_error(self, status_code):\n \"\"\"Called when a non-200 status code is returned\"\"\"\n logger.error('Twitter returned error code %s', status_code)\n self.error = status_code\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_exception(self, exception):\n logger.error('Exception from stream!', exc_info=True)\n self.streaming_exception = exception", "response": "Called when an exception occurs in the streaming thread"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if the list of tracked terms has changed. Returns True if changed otherwise False.", "response": "def check(self):\n \"\"\"\n Checks if the list of tracked terms has changed.\n Returns True if changed, otherwise False.\n \"\"\"\n\n new_tracking_terms = self.update_tracking_terms()\n\n terms_changed = False\n\n # any deleted terms?\n if self._tracking_terms_set > new_tracking_terms:\n logging.debug(\"Some tracking terms removed\")\n terms_changed = True\n\n # any added terms?\n elif self._tracking_terms_set < new_tracking_terms:\n logging.debug(\"Some tracking terms added\")\n terms_changed = True\n\n # Go ahead and store for later\n self._tracking_terms_set = new_tracking_terms\n\n # If the terms changed, we need to restart the stream\n return terms_changed"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_tracking_terms(self):\n import codecs\n with codecs.open(self.filename,\"r\", encoding='utf8') as input:\n # read all the lines\n lines = input.readlines()\n\n # build a set of terms\n new_terms = set()\n for line in lines:\n line = line.strip()\n if len(line):\n new_terms.add(line)\n\n return set(new_terms)", "response": "Update the tracking terms of the current node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninterrupt running process, and provide a python prompt for interactive debugging.", "response": "def launch_debugger(frame, stream=None):\n \"\"\"\n Interrupt running process, and provide a python prompt for\n interactive debugging.\n \"\"\"\n\n d = {'_frame': frame} # Allow access to frame object.\n d.update(frame.f_globals) # Unless shadowed by global\n d.update(frame.f_locals)\n\n import code, traceback\n\n i = code.InteractiveConsole(d)\n message = \"Signal received : entering python shell.\\nTraceback:\\n\"\n message += ''.join(traceback.format_stack(frame))\n i.interact(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_debug_listener(stream):\n\n def debugger(sig, frame):\n launch_debugger(frame, stream)\n\n if hasattr(signal, 'SIGUSR1'):\n signal.signal(signal.SIGUSR1, debugger)\n else:\n logger.warn(\"Cannot set SIGUSR1 signal for debug mode.\")", "response": "Break into a debugger if receives the SIGUSR1 signal"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting up signal handlers for handling SIGTERM and SIGINT.", "response": "def set_terminate_listeners(stream):\n \"\"\"Die on SIGTERM or SIGINT\"\"\"\n\n def stop(signum, frame):\n terminate(stream.listener)\n\n # Installs signal handlers for handling SIGINT and SIGTERM\n # gracefully.\n signal.signal(signal.SIGINT, stop)\n signal.signal(signal.SIGTERM, stop)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a tweepy auth object", "response": "def get_tweepy_auth(twitter_api_key,\n twitter_api_secret,\n twitter_access_token,\n twitter_access_token_secret):\n \"\"\"Make a tweepy auth object\"\"\"\n auth = tweepy.OAuthHandler(twitter_api_key, twitter_api_secret)\n auth.set_access_token(twitter_access_token, twitter_access_token_secret)\n return auth"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates the listener that prints tweets", "response": "def construct_listener(outfile=None):\n \"\"\"Create the listener that prints tweets\"\"\"\n if outfile is not None:\n if os.path.exists(outfile):\n raise IOError(\"File %s already exists\" % outfile)\n \n outfile = open(outfile, 'wb')\n\n return PrintingListener(out=outfile)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstarting and maintain the streaming connection...", "response": "def begin_stream_loop(stream, poll_interval):\n \"\"\"Start and maintain the streaming connection...\"\"\"\n while should_continue():\n try:\n stream.start_polling(poll_interval)\n except Exception as e:\n # Infinite restart\n logger.error(\"Exception while polling. Restarting in 1 second.\", exc_info=True)\n time.sleep(1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprinting out some tweets", "response": "def on_status(self, status):\n \"\"\"Print out some tweets\"\"\"\n self.out.write(json.dumps(status))\n self.out.write(os.linesep)\n\n self.received += 1\n return not self.terminate"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting out the current status and reset the counter", "response": "def print_status(self):\n \"\"\"Print out the current tweet rate and reset the counter\"\"\"\n tweets = self.received\n now = time.time()\n diff = now - self.since\n self.since = now\n self.received = 0\n if diff > 0:\n logger.info(\"Receiving tweets at %s tps\", tweets / diff)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef start_polling(self, interval):\n interval = float(interval)\n\n self.polling = True\n\n # clear the stored list of terms - we aren't tracking any\n self.term_checker.reset()\n\n logger.info(\"Starting polling for changes to the track list\")\n while self.polling:\n\n loop_start = time()\n\n self.update_stream()\n self.handle_exceptions()\n\n # wait for the interval unless interrupted, compensating for time elapsed in the loop\n elapsed = time() - loop_start\n sleep(max(0.1, interval - elapsed))\n\n logger.warning(\"Term poll ceased!\")", "response": "Start polling for term updates and streaming."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nrestarts the stream with the current list of tracking terms.", "response": "def update_stream(self):\n \"\"\"\n Restarts the stream with the current list of tracking terms.\n \"\"\"\n\n need_to_restart = False\n\n # If we think we are running, but something has gone wrong in the streaming thread\n # Restart it.\n if self.stream is not None and not self.stream.running:\n logger.warning(\"Stream exists but isn't running\")\n self.listener.error = False\n self.listener.streaming_exception = None\n need_to_restart = True\n\n # Check if the tracking list has changed\n if self.term_checker.check():\n logger.info(\"Terms have changed\")\n need_to_restart = True\n\n # If we aren't running and we are allowing unfiltered streams\n if self.stream is None and self.unfiltered:\n need_to_restart = True\n\n if not need_to_restart:\n return\n\n logger.info(\"Restarting stream...\")\n \n # Stop any old stream\n self.stop_stream()\n\n # Start a new stream\n self.start_stream()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstarting a new stream with teh current tracking terms", "response": "def start_stream(self):\n \"\"\"Starts a stream with teh current tracking terms\"\"\"\n\n tracking_terms = self.term_checker.tracking_terms()\n\n if len(tracking_terms) > 0 or self.unfiltered:\n # we have terms to track, so build a new stream\n self.stream = tweepy.Stream(self.auth, self.listener,\n stall_warnings=True,\n timeout=90,\n retry_count=self.retry_count)\n\n if len(tracking_terms) > 0:\n logger.info(\"Starting new twitter stream with %s terms:\", len(tracking_terms))\n logger.info(\" %s\", repr(tracking_terms))\n \n # Launch it in a new thread\n self.stream.filter(track=tracking_terms, async=True, languages=self.languages)\n else:\n logger.info(\"Starting new unfiltered stream\")\n self.stream.sample(async=True, languages=self.languages)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop_stream(self):\n\n if self.stream is not None:\n # There is a streaming thread\n\n logger.warning(\"Stopping twitter stream...\")\n self.stream.disconnect()\n\n self.stream = None\n\n # wait a few seconds to allow the streaming to actually stop\n sleep(self.STOP_TIMEOUT)", "response": "Stops the current stream."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef enrich(self, tweet):\n tweet = urlize_tweet(expand_tweet_urls(tweet))\n # parses created_at \"Wed Aug 27 13:08:45 +0000 2008\"\n\n if settings.USE_TZ:\n tweet['datetime'] = datetime.strptime(tweet['created_at'], '%a %b %d %H:%M:%S +0000 %Y').replace(tzinfo=timezone.utc)\n else:\n tweet['datetime'] = datetime.strptime(tweet['created_at'], '%a %b %d %H:%M:%S +0000 %Y')\n\n return tweet", "response": "Apply the local presentation logic to the fetched data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_user_cache_key(**kwargs):\n key = 'get_tweets_%s' % ('_'.join([str(kwargs[key]) for key in sorted(kwargs) if kwargs[key]]))\n not_allowed = re.compile('[^%s]' % ''.join([chr(i) for i in range(33, 128)]))\n key = not_allowed.sub('', key)\n return key", "response": "Generate suitable key to cache twitter tag context"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_search_cache_key(prefix, *args):\n key = '%s_%s' % (prefix, '_'.join([str(arg) for arg in args if arg]))\n not_allowed = re.compile('[^%s]' % ''.join([chr(i) for i in range(33, 128)]))\n key = not_allowed.sub('', key)\n return key", "response": "Generate suitable key to cache twitter tag context"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef urlize_tweet(tweet):\n text = tweet.get('html', tweet['text'])\n for hash in tweet['entities']['hashtags']:\n text = text.replace('#%s' % hash['text'], TWITTER_HASHTAG_URL % (quote(hash['text'].encode(\"utf-8\")), hash['text']))\n for mention in tweet['entities']['user_mentions']:\n text = text.replace('@%s' % mention['screen_name'], TWITTER_USERNAME_URL % (quote(mention['screen_name']), mention['screen_name']))\n tweet['html'] = text\n return tweet", "response": "Turn the text in a tweet to Twitter hyperlinks"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef expand_tweet_urls(tweet):\n if 'retweeted_status' in tweet:\n text = 'RT @{user}: {text}'.format(user=tweet['retweeted_status']['user']['screen_name'],\n text=tweet['retweeted_status']['text'])\n urls = tweet['retweeted_status']['entities']['urls']\n else:\n text = tweet['text']\n urls = tweet['entities']['urls']\n\n for url in urls:\n text = text.replace(url['url'], '%s' % (url['expanded_url'], url['display_url']))\n tweet['html'] = text\n return tweet", "response": "Replace shortened URLs with long URLs in the twitter status and add the RT flag."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef safe_power(a, b):\n if abs(a) > MAX_POWER or abs(b) > MAX_POWER:\n raise ValueError('Number too high!')\n return a ** b", "response": "safe_power - Calculates the power of two numbers"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nperforming a rabin-miller primality test :param p: Number to test :return: Bool of whether num is prime", "response": "def rabin_miller(p):\n \"\"\"\n Performs a rabin-miller primality test\n\n :param p: Number to test\n :return: Bool of whether num is prime\n \"\"\"\n # From this stackoverflow answer: https://codegolf.stackexchange.com/questions/26739/super-speedy-totient-function\n if p < 2:\n return False\n if p != 2 and p & 1 == 0:\n return False\n s = p - 1\n while s & 1 == 0:\n s >>= 1\n for x in range(10):\n a = random.randrange(p - 1) + 1\n temp = s\n mod = pow(a, temp, p)\n while temp != p - 1 and mod != 1 and mod != p - 1:\n mod = (mod * mod) % p\n temp = temp * 2\n if mod != p - 1 and temp % 2 == 0:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef zero_width_split(pattern, string):\n splits = list((m.start(), m.end()) for m in regex.finditer(pattern, string, regex.VERBOSE))\n starts = [0] + [i[1] for i in splits]\n ends = [i[0] for i in splits] + [len(string)]\n return [string[start:end] for start, end in zip(starts, ends)]", "response": "Split a string on a regex that only matches zero - width strings\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef roll_group(group):\n group = regex.match(r'^(\\d*)d(\\d+)$', group, regex.IGNORECASE)\n num_of_dice = int(group[1]) if group[1] != '' else 1\n type_of_dice = int(group[2])\n assert num_of_dice > 0\n\n result = []\n for i in range(num_of_dice):\n result.append(random.randint(1, type_of_dice))\n return result", "response": "Rolls a group of dice in 2d6 3d10 d12 etc. format\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef num_equal(result, operator, comparator):\n if operator == '<':\n return len([x for x in result if x < comparator])\n elif operator == '>':\n return len([x for x in result if x > comparator])\n elif operator == '=':\n return len([x for x in result if x == comparator])\n else:\n raise ValueError", "response": "Returns the number of elements in a list that pass a comparison on a dice roll\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef roll_dice(roll, *, functions=True, floats=True):\n roll = ''.join(roll.split())\n roll = regex.sub(r'(?<=d)%', '100', roll, regex.IGNORECASE)\n roll = roll.replace('^', '**')\n roll = zero_width_split(r'((?<=[\\(\\),%^\\/+*-])(?=.))|((?<=.)(?=[\\(\\),%^\\/+*-]))', roll) # Split the string on the boundary between operators and other chars\n\n string = []\n\n results = []\n\n for group in roll:\n if group in '()/=<>,%^+*-' or group in DEFAULT_FUNCTIONS: #Append operators without modification\n results.append(group)\n string.append(group)\n continue\n try:\n explode = regex.match(r'^((\\d*)d(\\d+))!$', group, regex.IGNORECASE) # Regex for exploding dice, ie. 2d10!, 4d100!, d12!, etc.\n\n specific_explode = regex.match(r'^((\\d*)d(\\d+))!(\\d+)$', group) # Regex for exploding dice on specific number, ie. d20!10 or d12!4\n\n comparison_explode = regex.match(r'^((\\d*)d(\\d+))!([<>])(\\d+)$', group, regex.IGNORECASE) # Regex for exploding dice with a comparison, ie. d20!>10, d6!<2\n\n penetrate = regex.match(r'^((\\d*)d(\\d+))!p$', group, regex.IGNORECASE) # Penetrating dice are the same as exploding except any dice after the initial number are added with a -1 penalty\n\n specific_penetrate = regex.match(r'^((\\d*)d(\\d+))!p(\\d+)$', group, regex.IGNORECASE) # See above\n\n comparison_penetrate = regex.match(r'^((\\d*)d(\\d+))!p([<>])(\\d+)$', group, regex.IGNORECASE) # See above\n\n reroll = regex.match(r'^((\\d*)d(\\d+))([Rr])$', group, regex.IGNORECASE) # Reroll on a one, matches 1d6R, 4d12r, etc.\n\n specific_reroll = regex.match(r'^((\\d*)d(\\d+))([Rr])(\\d+)$', group, regex.IGNORECASE) # Reroll on a specific number\n\n comparison_reroll = regex.match(r'^((\\d*)d(\\d+))([Rr])([<>])(\\d+)$', group, regex.IGNORECASE) # Reroll on a comparison\n\n success_comparison = regex.match(r'^((?:\\d*)d(\\d+))([<>])(\\d+)$', group, regex.IGNORECASE) # Regex for dice with comparison, ie. 2d10>4, 5d3<2, etc.\n\n success_fail_comparison = regex.match(r'^((?:\\d*)d(\\d+))(?|((<)(\\d+)f(>)(\\d+))|((>)(\\d+)f(<)(\\d+)))$', group, regex.IGNORECASE) # Regex for dice with success comparison and failure comparison.\n\n keep = regex.match(r'^((?:\\d*)d\\d+)([Kk])(\\d*)$', group, regex.IGNORECASE) # Regex for keeping a number of dice, ie. 2d10K, 2d10k3, etc.\n\n drop = regex.match(r'^((?:\\d*)d\\d+)([Xx])(\\d*)$', group, regex.IGNORECASE) # As above but with dropping dice and X\n\n individual = regex.match(r'^((\\d*)d(\\d+))([asm])(\\d+)$', group, regex.IGNORECASE) #Regex for rolling dice with a modifier attached to each roll\n\n normal = regex.match(r'^((\\d*)d(\\d+))$', group, regex.IGNORECASE) # Regex for normal dice rolls\n\n literal = regex.match(r'^(\\d+)(?!\\.)$', group, regex.IGNORECASE) # Regex for number literals.\n\n float_literal = regex.match(r'^(\\.\\d+)|(\\d+.\\d+)$', group, regex.IGNORECASE) # Regex for floats\n\n if explode is not None: # Handle exploding dice without a comparison modifier.\n type_of_dice = int(explode[3])\n\n result = []\n last_result = roll_group(explode[1])\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', type_of_dice)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice)) # Reroll dice\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', type_of_dice) # Check how many dice we have to reroll again\n\n results.append(sum(result))\n roll = ','.join([('!' + str(i) if i == type_of_dice else str(i)) for i in result]) # Build a string of the dice rolls, adding an exclamation mark before every roll that resulted in an explosion.\n string.append('[%s]' % roll)\n\n elif specific_explode is not None: # Handle exploding dice without a comparison modifier.\n type_of_dice = int(specific_explode[3])\n\n comparator = int(specific_explode[4])\n\n assert 0 < comparator <= type_of_dice\n\n result = []\n last_result = roll_group(specific_explode[1])\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', comparator)\n\n results.append(sum(result))\n roll = ','.join([('!' + str(i) if i == comparator else str(i)) for i in result]) # Build a string of the dice rolls, adding an exclamation mark before every roll that resulted in an explosion.\n string.append('[%s]' % roll)\n\n elif comparison_explode is not None: # Handle exploding dice with a comparison modifier\n type_of_dice = int(comparison_explode[3])\n\n comparator = int(comparison_explode[5])\n\n if comparison_explode[4] == '>': # Ensure comparison is within bounds\n assert 0 < comparator < type_of_dice\n else:\n assert 1 < comparator <= type_of_dice\n\n result = []\n last_result = roll_group(comparison_explode[1])\n result.extend(last_result)\n if comparison_explode[4] == '>':\n number_to_roll = num_equal(last_result, '>', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '>', comparator)\n roll = ','.join([('!' + str(i) if i > comparator else str(i)) for i in\n result]) # Same as on other explodes except with a > or < comparison\n\n else:\n number_to_roll = num_equal(last_result, '<', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '<', comparator)\n roll = ','.join([('!' + str(i) if i < comparator else str(i)) for i in\n result]) # Same as on other explodes except with a > or < comparison\n\n results.append(sum(result))\n string.append('[%s]' % roll)\n\n elif penetrate is not None: # Handle penetrating dice without a comparison modifier.\n type_of_dice = int(penetrate[3])\n\n first_num = int(penetrate[2])\n\n result = []\n last_result = roll_group(penetrate[1])\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', type_of_dice)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', type_of_dice)\n\n pre_result = result[:first_num] # Add the first rolls with no modifier\n pre_result.extend([x - 1 for x in result[first_num:]]) # Add the second rolls with a -1 modifier\n\n results.append(sum(pre_result))\n\n roll = ','.join(['!' + str(i) if i == type_of_dice else str(i) for i in result[:first_num]]) # Add the first numbers, without the -1 but with a ! when roll is penetration\n roll += (',' if len(pre_result) > first_num else '') # Only add the comma in between if there's at least one penetration\n roll += ','.join([('!' + str(i) + '-1' if i == type_of_dice else str(i) + '-1') for i in result[first_num:]]) # Add the penetration dice with the '-1' tacked on the end\n string.append('[%s]' % roll)\n\n elif specific_penetrate is not None: # Handle penetrating dice without a comparison modifier.\n type_of_dice = int(specific_penetrate[3])\n\n first_num = int(specific_penetrate[2])\n\n comparator = int(specific_penetrate[4])\n\n assert 0 < comparator <= type_of_dice\n\n result = []\n last_result = roll_group(specific_penetrate[1])\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '=', comparator)\n\n pre_result = result[:first_num] # Same as normal penetration\n pre_result.extend([x - 1 for x in result[first_num:]])\n\n results.append(sum(pre_result))\n\n roll = ','.join(['!' + str(i) if i == comparator else str(i) for i in result[:first_num]]) # Same as above\n roll += (',' if len(pre_result) > first_num else '')\n roll += ','.join([('!' + str(i) + '-1' if i == comparator else str(i) + '-1') for i in result[first_num:]])\n string.append('[%s]' % roll)\n\n elif comparison_penetrate is not None: # Handle penetrating dice without a comparison modifier.\n type_of_dice = int(comparison_penetrate[3])\n\n comparator = int(comparison_penetrate[5])\n\n first_num = int(comparison_penetrate[2])\n\n if comparison_penetrate[4] == '>': # Ensure comparison is within bounds\n assert 0 < comparator < type_of_dice\n else:\n assert 1 < comparator <= type_of_dice\n\n result = []\n last_result = roll_group(comparison_penetrate[1])\n result.extend(last_result)\n\n # Do penetration based on more than or less than sign.\n if comparison_penetrate[4] == '>':\n number_to_roll = num_equal(last_result, '>', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '>', comparator)\n\n else:\n number_to_roll = num_equal(last_result, '<', comparator)\n while number_to_roll != 0:\n last_result = roll_group(str(number_to_roll) + 'd' + str(type_of_dice))\n result.extend(last_result)\n number_to_roll = num_equal(last_result, '<', comparator)\n\n pre_result = result[:first_num]\n pre_result.extend([x - 1 for x in result[first_num:]])\n results.append(sum(pre_result))\n\n if comparison_penetrate[4] == '>':\n roll = ','.join(\n ['!' + str(i) if i > comparator else str(i) for i in result[:first_num]]) # Same as above\n roll += (',' if len(pre_result) > first_num else '')\n roll += ','.join(\n [('!' + str(i) + '-1' if i > comparator else str(i) + '-1') for i in result[first_num:]])\n else:\n roll = ','.join(\n ['!' + str(i) if i < comparator else str(i) for i in result[:first_num]]) # Same as above\n roll += (',' if len(pre_result) > first_num else '')\n roll += ','.join(\n [('!' + str(i) + '-1' if i < comparator else str(i) + '-1') for i in result[first_num:]])\n string.append('[%s]' % roll)\n\n elif reroll is not None: # Handle rerolling dice without a comparison modifier (ie. on 1)\n type_of_dice = int(reroll[3])\n\n result_strings = []\n roll_strings = []\n result = roll_group(reroll[1])\n repeat = True if reroll[4] == 'R' else False # Reroll just once or infinite number of times\n\n if repeat: #Handle rerolling the dice and building a string of all the rerolled ones\n for i in range(len(result)):\n prev = [result[i]]\n while result[i] == 1:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n else:\n for i in range(len(result)):\n prev = [result[i]]\n if result[i] == 1:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n results.append(sum(result))\n for roll_string in roll_strings:\n roll_string.reverse()\n result_strings.append('%s' % roll_string[0] + ('~' if len(roll_string) > 1 else '') + '~'.join(roll_string[1:])) #Build the string\n\n roll = ','.join(result_strings)\n string.append('[%s]' % roll)\n\n elif specific_reroll is not None: # Handle rerolling dice on a specific number, see reroll\n type_of_dice = int(specific_reroll[3])\n comparator = int(specific_reroll[5])\n\n assert 0 < comparator <= type_of_dice # Ensure comparison is within bounds\n\n result_strings = []\n roll_strings = []\n result = roll_group(specific_reroll[1])\n repeat = True if specific_reroll[4] == 'R' else False\n\n if repeat:\n for i in range(len(result)):\n prev = [result[i]]\n while result[i] == comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n else:\n for i in range(len(result)):\n prev = [result[i]]\n if result[i] == comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n results.append(sum(result))\n for roll_string in roll_strings:\n roll_string.reverse()\n result_strings.append('%s' % roll_string[0] + ('~' if len(roll_string) > 1 else '') + '~'.join(roll_string[1:]))\n\n roll = ','.join(result_strings)\n string.append('[%s]' % roll)\n\n elif comparison_reroll is not None: # Handle rerolling dice with a comparison modifier.\n type_of_dice = int(comparison_reroll[3])\n comparator = int(comparison_reroll[6])\n\n if comparison_reroll[5] == '>': # Ensure comparison is within bounds\n assert 0 < comparator < type_of_dice\n else:\n assert 1 < comparator <= type_of_dice\n\n result_strings = []\n roll_strings = []\n result = roll_group(comparison_reroll[1])\n repeat = True if comparison_reroll[4] == 'R' else False\n if comparison_reroll[5] == '>':\n if repeat:\n for i in range(len(result)):\n prev = [result[i]]\n while result[i] > comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n else:\n for i in range(len(result)):\n prev = [result[i]]\n if result[i] > comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n else:\n if repeat:\n for i in range(len(result)):\n prev = [result[i]]\n while result[i] < comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n else:\n for i in range(len(result)):\n prev = [result[i]]\n if result[i] < comparator:\n result[i] = random.randint(1, type_of_dice)\n prev.append(result[i])\n\n roll_strings.append([str(x) for x in prev])\n\n results.append(sum(result))\n for roll_string in roll_strings:\n roll_string.reverse()\n result_strings.append('%s' % roll_string[0] + ('~' if len(roll_string) > 1 else '') + '~'.join(roll_string[1:]))\n\n roll = ','.join(result_strings)\n string.append('[%s]' % roll)\n\n elif success_comparison is not None:\n group_result = roll_group(success_comparison[1])\n result = []\n result_string = []\n\n type_of_dice = int(success_comparison[2])\n\n comparator = int(success_comparison[4])\n\n if success_comparison[3] == '>': # Ensure comparison is within bounds\n assert 0 < comparator < type_of_dice\n else:\n assert 1 < comparator <= type_of_dice\n\n for die in group_result:\n if success_comparison[3] == '>':\n result.append(1 if die > comparator else 0)\n result_string.append('!' + str(die) if die > comparator else str(die))\n else:\n result.append(1 if die < comparator else 0)\n result_string.append('!' + str(die) if die < comparator else str(die))\n\n results.append(sum(result))\n roll = ','.join(result_string) # Craft the string, adding an exclamation mark before every string that passed the comparison.\n string.append('[%s]' % roll)\n\n elif success_fail_comparison is not None:\n group_result = roll_group(success_fail_comparison[1])\n\n result = []\n result_string = []\n\n type_of_dice = int(success_fail_comparison[2])\n success_comp = int(success_fail_comparison[5])\n fail_comp = int(success_fail_comparison[7])\n\n # Ensure both comparisons are within bounds\n if success_fail_comparison[4] == '>':\n assert 0 < success_comp < type_of_dice\n assert 1 < fail_comp <= type_of_dice\n else:\n assert 1 < success_comp <= type_of_dice\n assert 0 < fail_comp < type_of_dice\n\n for die in group_result:\n if success_fail_comparison[4] == '>': # Get the actual list of successes and fails with both comparisons\n if die > success_comp:\n result.append(1)\n result_string.append('!' + str(die))\n elif die < fail_comp:\n result.append(-1)\n result_string.append('*' + str(die))\n else:\n result.append(0)\n result_string.append(str(die))\n else:\n if die < success_comp:\n result.append(1)\n result_string.append('!' + str(die))\n elif die > fail_comp:\n result.append(-1)\n result_string.append('*' + str(die))\n else:\n result.append(0)\n result_string.append(str(die))\n\n results.append(sum(result)) #\n roll = ','.join(result_string)\n string.append('[%s]' % roll)\n\n elif keep is not None: # Handle rolling dice and keeping the x highest or lowest values\n group_result = roll_group(keep[1])\n group_result.sort(reverse=True if keep[\n 2] == 'K' else False) # Uppercase is keep highest and lowercase is keep lowest.\n\n num_to_keep = int(keep[3] if keep[3] != '' else 1)\n assert 1 <= num_to_keep < len(group_result)\n\n results.append(sum(group_result[:num_to_keep]))\n roll = ','.join([str(i) for i in group_result[\n :num_to_keep]]) + ' ~~ ' # This time format the string with all kept rolls on the left and dropped rolls on the right\n roll += ','.join([str(i) for i in group_result[num_to_keep:]])\n string.append('[%s]' % roll)\n\n elif drop is not None:\n group_result = roll_group(drop[1])\n group_result.sort(reverse=True if drop[2] == 'X' else False) # Same thing as keep dice\n\n num_to_drop = int(drop[3] if drop[3] != '' else 1)\n assert 1 <= num_to_drop < len(group_result)\n\n results.append(sum(group_result[:num_to_drop]))\n roll = ','.join([str(i) for i in group_result[num_to_drop:]]) + ' ~~ ' # Same as above.\n roll += ','.join([str(i) for i in group_result[:num_to_drop]])\n string.append('[%s]' % roll)\n\n elif individual is not None:\n group_result = roll_group(individual[1])\n result = []\n for i, j in enumerate(group_result): #add to each roll\n if individual[4] == 'a':\n result.append(j + int(individual[5]))\n\n elif individual[4] == 's':\n result.append(j - int(individual[5]))\n\n elif individual[4] == 'm':\n result.append(j * int(individual[5]))\n\n else:\n raise ValueError\n results.append(sum(result))\n roll = ','.join([str(x) + individual[4] + individual[5] for x in group_result]) #Create string with the modifier on each roll\n string.append('[%s]' % roll)\n\n elif normal is not None:\n group_result = roll_group(group)\n results.append(sum(group_result))\n roll = ','.join([str(i) for i in group_result])\n string.append('[%s]' % roll)\n\n elif literal is not None:\n results.append(int(literal[1])) # Just append the integer value\n string.append(literal[1])\n\n elif float_literal is not None:\n if floats:\n results.append(float(group))\n string.append(group)\n else:\n raise TypeError\n else:\n raise Exception\n\n except Exception:\n raise DiceGroupException('\"%s\" is not a valid dicegroup.' % group)\n\n parser = SimpleEval(floats=floats, functions=functions) #The parser object parses the dice rolls and functions\n try:\n final_result = parser.eval(''.join([str(x) for x in results])) #Call the parser to parse into one value\n if not floats:\n final_result = int(final_result)\n except Exception:\n raise DiceOperatorException('Error parsing operators and or functions')\n\n #Create explanation string and remove extraneous spaces\n explanation = ''.join(string)\n explanation = zero_width_split(r\"\"\"((?<=[\\/%^+])(?![\\/,]))| # Split between /, %, ^, and +\n ((?\n\n Return a list of destination files.\n \"\"\"\n dest_files = []\n for filename in files:\n leaf_dest_folder = os.path.join(destination, os.path.dirname(filename))\n if not os.path.exists(leaf_dest_folder):\n os.makedirs(leaf_dest_folder)\n dest_file = os.path.join(destination, filename)\n bash(\"git show :{filename} > {dest_file}\".format(\n filename=filename,\n dest_file=dest_file)\n )\n dest_files.append(os.path.realpath(dest_file))\n return dest_files"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets all python files from the list of files.", "response": "def filter_python_files(files):\n \"Get all python files from the list of files.\"\n py_files = []\n for f in files:\n # If we end in .py, or if we don't have an extension and file says that\n # we are a python script, then add us to the list\n extension = os.path.splitext(f)[-1]\n\n if extension:\n if extension == '.py':\n py_files.append(f)\n elif 'python' in open(f, 'r').readline():\n py_files.append(f)\n elif 'python script' in bash('file {}'.format(f)).value().lower():\n py_files.append(f)\n\n return py_files"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the configuration of a specific plugin.", "response": "def configuration(self, plugin):\n \"\"\"\n Get plugin configuration.\n\n Return a tuple of (on|off|default, args)\n \"\"\"\n conf = self.config.get(plugin, \"default;\").split(';')\n if len(conf) == 1:\n conf.append('')\n return tuple(conf)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking frosted errors in the code base.", "response": "def run(files, temp_folder):\n \"Check frosted errors in the code base.\"\n try:\n import frosted # NOQA\n except ImportError:\n return NO_FROSTED_MSG\n\n py_files = filter_python_files(files)\n cmd = 'frosted {0}'.format(' '.join(py_files))\n\n return bash(cmd).value()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run(files, temp_folder):\n try:\n import isort # NOQA\n except ImportError:\n return NO_ISORT_MSG\n\n py_files = filter_python_files(files)\n\n # --quiet because isort >= 4.1 outputs its logo in the console by default.\n return bash('isort -df --quiet {0}'.format(' '.join(py_files))).value()", "response": "Run isort on a list of python files."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck we re not committing to a blocked branch", "response": "def run(files, temp_folder, arg=None):\n \"Check we're not committing to a blocked branch\"\n parser = get_parser()\n argos = parser.parse_args(arg.split())\n\n current_branch = bash('git symbolic-ref HEAD').value()\n current_branch = current_branch.replace('refs/heads/', '').strip()\n if current_branch in argos.branches:\n return (\"Branch '{0}' is blocked from being \"\n \"committed to.\".format(current_branch))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck coding convention of the code base.", "response": "def run(files, temp_folder, arg=None):\n \"Check coding convention of the code base.\"\n try:\n import pylint\n except ImportError:\n return NO_PYLINT_MSG\n\n # set default level of threshold\n arg = arg or SCORE\n\n py_files = filter_python_files(files)\n if not py_files:\n return False\n\n str_py_files = \" \".join(py_files)\n cmd = \"{0} {1}\".format(PYLINT_CMD, str_py_files)\n output = bash(cmd).value()\n\n if 'rated' not in output:\n return False\n score = float(re.search(\"(\\d.\\d\\d)/10\", output).group(1))\n if score >= float(arg):\n return False\n return (\"Pylint appreciated your {0} as {1},\"\n \"required threshold is {2}\".format(PYLINT_TARGET, score, arg)\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck to see if python files are py3 compatible", "response": "def run(files, temp_folder):\n \"Check to see if python files are py3 compatible\"\n errors = []\n for py_file in filter_python_files(files):\n # We only want to show errors if we CAN'T compile to py3.\n # but we want to show all the errors at once.\n b = bash('python3 -m py_compile {0}'.format(py_file))\n if b.stderr:\n b = bash('2to3-2.7 {file}'.format(file=py_file))\n errors.append(b.value())\n return \"\\n\".join(errors)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck flake8 errors in the code base.", "response": "def run(files, temp_folder):\n \"Check flake8 errors in the code base.\"\n try:\n import flake8 # NOQA\n except ImportError:\n return NO_FLAKE_MSG\n try:\n from flake8.engine import get_style_guide\n except ImportError:\n # We're on a new version of flake8\n from flake8.api.legacy import get_style_guide\n\n py_files = filter_python_files(files)\n if not py_files:\n return\n DEFAULT_CONFIG = join(temp_folder, get_config_file())\n\n with change_folder(temp_folder):\n flake8_style = get_style_guide(config_file=DEFAULT_CONFIG)\n out, err = StringIO(), StringIO()\n with redirected(out, err):\n flake8_style.check_files(py_files)\n return out.getvalue().strip() + err.getvalue().strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nputs two strategies to a classic battle of wits.", "response": "def tictactoe(w, i, player, opponent, grid=None):\n \"Put two strategies to a classic battle of wits.\"\n grid = grid or empty_grid\n while True:\n w.render_to_terminal(w.array_from_text(view(grid)))\n if is_won(grid):\n print(whose_move(grid), \"wins.\")\n break\n if not successors(grid):\n print(\"A draw.\")\n break\n grid = player(w, i, grid)\n player, opponent = opponent, player"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a function like f that remembers and reuses results of past calls.", "response": "def memo(f):\n \"Return a function like f that remembers and reuses results of past calls.\"\n table = {}\n def memo_f(*args):\n try:\n return table[args]\n except KeyError:\n table[args] = value = f(*args)\n return value\n return memo_f"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef human_play(w, i, grid):\n \"Just ask for a move.\"\n plaint = ''\n prompt = whose_move(grid) + \" move? [1-9] \"\n while True:\n w.render_to_terminal(w.array_from_text(view(grid)\n + '\\n\\n' + plaint + prompt))\n key = c = i.next()\n try:\n move = int(key)\n except ValueError:\n pass\n else:\n if 1 <= move <= 9:\n successor = apply_move(grid, from_human_move(move))\n if successor: return successor\n plaint = (\"Hey, that's illegal. Give me one of these digits:\\n\\n\"\n + (grid_format\n % tuple(move if apply_move(grid, from_human_move(move)) else '-'\n for move in range(1, 10))\n + '\\n\\n'))", "response": "Just ask for a move."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplay like Spock except breaking ties by drunk_value.", "response": "def max_play(w, i, grid):\n \"Play like Spock, except breaking ties by drunk_value.\"\n return min(successors(grid),\n key=lambda succ: (evaluate(succ), drunk_value(succ)))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the expected value to the player if both players play at random.", "response": "def drunk_value(grid):\n \"Return the expected value to the player if both players play at random.\"\n if is_won(grid): return -1\n succs = successors(grid)\n return -average(map(drunk_value, succs)) if succs else 0"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef evaluate(grid):\n \"Return the value for the player to move, assuming perfect play.\"\n if is_won(grid): return -1\n succs = successors(grid)\n return -min(map(evaluate, succs)) if succs else 0", "response": "Return the value for the player to move assuming perfect play."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndoes the latest move win the game?", "response": "def is_won(grid):\n \"Did the latest move win the game?\"\n p, q = grid\n return any(way == (way & q) for way in ways_to_win)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef apply_move(grid, move):\n \"Try to move: return a new grid, or None if illegal.\"\n p, q = grid\n bit = 1 << move\n return (q, p | bit) if 0 == (bit & (p | q)) else None", "response": "Try to move : return a new grid or None if illegal."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns two results : the player s mark and their opponents.", "response": "def player_marks(grid):\n \"Return two results: the player's mark and their opponent's.\"\n p, q = grid\n return 'XO' if sum(player_bits(p)) == sum(player_bits(q)) else 'OX'"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nshow a grid human - readably.", "response": "def view(grid):\n \"Show a grid human-readably.\"\n p_mark, q_mark = player_marks(grid)\n return grid_format % tuple(p_mark if by_p else q_mark if by_q else '.'\n for by_p, by_q in zip(*map(player_bits, grid)))"}