{"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_by_name(cls, name):\n try:\n return cls.query.filter_by(name=name).one()\n except NoResultFound:\n return None", "response": "Query group by a group name."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nquery by a list of group names.", "response": "def query_by_names(cls, names):\n \"\"\"Query group by a list of group names.\n\n :param list names: List of the group names.\n :returns: Query object.\n \"\"\"\n assert isinstance(names, list)\n return cls.query.filter(cls.name.in_(names))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query_by_user(cls, user, with_pending=False, eager=False):\n q1 = Group.query.join(Membership).filter_by(user_id=user.get_id())\n if not with_pending:\n q1 = q1.filter_by(state=MembershipState.ACTIVE)\n if eager:\n q1 = q1.options(joinedload(Group.members))\n\n q2 = Group.query.join(GroupAdmin).filter_by(\n admin_id=user.get_id(), admin_type=resolve_admin_type(user))\n if eager:\n q2 = q2.options(joinedload(Group.members))\n\n query = q1.union(q2).with_entities(Group.id)\n\n return Group.query.filter(Group.id.in_(query))", "response": "Query group by user."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef search(cls, query, q):\n return query.filter(Group.name.like('%{0}%'.format(q)))", "response": "Modify query as so include only specific group names."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_member(self, user, state=MembershipState.ACTIVE):\n return Membership.create(self, user, state)", "response": "Invite a user to a group."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninvite a user to a group.", "response": "def invite(self, user, admin=None):\n \"\"\"Invite a user to a group (should be done by admins).\n\n Wrapper around ``add_member()`` to ensure proper membership state.\n\n :param user: User to invite.\n :param admin: Admin doing the action. If provided, user is only invited\n if the object is an admin for this group. Default: None.\n :returns: Newly created Membership or None.\n \"\"\"\n if admin is None or self.is_admin(admin):\n return self.add_member(user, state=MembershipState.PENDING_USER)\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninvites users to a group by emails.", "response": "def invite_by_emails(self, emails):\n \"\"\"Invite users to a group by emails.\n\n :param list emails: Emails of users that shall be invited.\n :returns list: Newly created Memberships or Nones.\n \"\"\"\n assert emails is None or isinstance(emails, list)\n\n results = []\n\n for email in emails:\n try:\n user = User.query.filter_by(email=email).one()\n results.append(self.invite(user))\n except NoResultFound:\n results.append(None)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsubscribing a user to a group.", "response": "def subscribe(self, user):\n \"\"\"Subscribe a user to a group (done by users).\n\n Wrapper around ``add_member()`` which checks subscription policy.\n\n :param user: User to subscribe.\n :returns: Newly created Membership or None.\n \"\"\"\n if self.subscription_policy == SubscriptionPolicy.OPEN:\n return self.add_member(user)\n elif self.subscription_policy == SubscriptionPolicy.APPROVAL:\n return self.add_member(user, state=MembershipState.PENDING_ADMIN)\n elif self.subscription_policy == SubscriptionPolicy.CLOSED:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nverifies if given user is a group member.", "response": "def is_member(self, user, with_pending=False):\n \"\"\"Verify if given user is a group member.\n\n :param user: User to be checked.\n :param bool with_pending: Whether to include pending users or not.\n :returns: True or False.\n \"\"\"\n m = Membership.get(self, user)\n if m is not None:\n if with_pending:\n return True\n elif m.state == MembershipState.ACTIVE:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef can_see_members(self, user):\n if self.privacy_policy == PrivacyPolicy.PUBLIC:\n return True\n elif self.privacy_policy == PrivacyPolicy.MEMBERS:\n return self.is_member(user) or self.is_admin(user)\n elif self.privacy_policy == PrivacyPolicy.ADMINS:\n return self.is_admin(user)", "response": "Determine if given user can see other group members."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndetermining if a user can invite people to a group.", "response": "def can_invite_others(self, user):\n \"\"\"Determine if user can invite people to a group.\n\n Be aware that this check is independent from the people (users) which\n are going to be invited. The checked user is the one who invites\n someone, NOT who is going to be invited.\n\n :param user: User to be checked.\n :returns: True or False.\n \"\"\"\n if self.is_managed:\n return False\n elif self.is_admin(user):\n return True\n elif self.subscription_policy != SubscriptionPolicy.CLOSED:\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(cls, group, user):\n try:\n m = cls.query.filter_by(user_id=user.get_id(), group=group).one()\n return m\n except Exception:\n return None", "response": "Get the membership for given user and group."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfilters a query result.", "response": "def _filter(cls, query, state=MembershipState.ACTIVE, eager=None):\n \"\"\"Filter a query result.\"\"\"\n query = query.filter_by(state=state)\n\n eager = eager or []\n for field in eager:\n query = query.options(joinedload(field))\n\n return query"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query_by_user(cls, user, **kwargs):\n return cls._filter(\n cls.query.filter_by(user_id=user.get_id()),\n **kwargs\n )", "response": "Get a user s memberships."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef query_invitations(cls, user, eager=False):\n if eager:\n eager = [Membership.group]\n return cls.query_by_user(user, state=MembershipState.PENDING_USER,\n eager=eager)", "response": "Get all invitations for given user."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef query_requests(cls, admin, eager=False):\n # Get direct pending request\n if hasattr(admin, 'is_superadmin') and admin.is_superadmin:\n q1 = GroupAdmin.query.with_entities(\n GroupAdmin.group_id)\n else:\n q1 = GroupAdmin.query_by_admin(admin).with_entities(\n GroupAdmin.group_id)\n q2 = Membership.query.filter(\n Membership.state == MembershipState.PENDING_ADMIN,\n Membership.id_group.in_(q1),\n )\n\n # Get request from admin groups your are member of\n q3 = Membership.query_by_user(\n user=admin, state=MembershipState.ACTIVE\n ).with_entities(Membership.id_group)\n q4 = GroupAdmin.query.filter(\n GroupAdmin.admin_type == 'Group', GroupAdmin.admin_id.in_(q3)\n ).with_entities(GroupAdmin.group_id)\n q5 = Membership.query.filter(\n Membership.state == MembershipState.PENDING_ADMIN,\n Membership.id_group.in_(q4))\n\n query = q2.union(q5)\n\n return query", "response": "Get all pending group requests."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef query_by_group(cls, group_or_id, with_invitations=False, **kwargs):\n if isinstance(group_or_id, Group):\n id_group = group_or_id.id\n else:\n id_group = group_or_id\n\n if not with_invitations:\n return cls._filter(\n cls.query.filter_by(id_group=id_group),\n **kwargs\n )\n else:\n return cls.query.filter(\n Membership.id_group == id_group,\n db.or_(\n Membership.state == MembershipState.PENDING_USER,\n Membership.state == MembershipState.ACTIVE\n )\n )", "response": "Get a group s members."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef search(cls, query, q):\n query = query.join(User).filter(\n User.email.like('%{0}%'.format(q)),\n )\n return query", "response": "Modify query as so include only specific members."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmodifies query as so to order the results.", "response": "def order(cls, query, field, s):\n \"\"\"Modify query as so to order the results.\n\n :param query: Query object.\n :param str s: Orderinig: ``asc`` or ``desc``.\n :returs: Query object.\n \"\"\"\n if s == 'asc':\n query = query.order_by(asc(field))\n elif s == 'desc':\n query = query.order_by(desc(field))\n return query"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create(cls, group, user, state=MembershipState.ACTIVE):\n with db.session.begin_nested():\n membership = cls(\n user_id=user.get_id(),\n id_group=group.id,\n state=state,\n )\n db.session.add(membership)\n return membership", "response": "Create a new membership."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new group admin.", "response": "def create(cls, group, admin):\n \"\"\"Create a new group admin.\n\n :param group: Group object.\n :param admin: Admin object.\n :returns: Newly created GroupAdmin object.\n :raises: IntegrityError\n \"\"\"\n with db.session.begin_nested():\n obj = cls(\n group=group,\n admin=admin,\n )\n db.session.add(obj)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget specific GroupAdmin object.", "response": "def get(cls, group, admin):\n \"\"\"Get specific GroupAdmin object.\"\"\"\n try:\n ga = cls.query.filter_by(\n group=group, admin_id=admin.get_id(),\n admin_type=resolve_admin_type(admin)).one()\n return ga\n except Exception:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef delete(cls, group, admin):\n with db.session.begin_nested():\n obj = cls.query.filter(\n cls.admin == admin, cls.group == group).one()\n db.session.delete(obj)", "response": "Delete admin from group."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting all groups for a specific admin.", "response": "def query_by_admin(cls, admin):\n \"\"\"Get all groups for for a specific admin.\"\"\"\n return cls.query.filter_by(\n admin_type=resolve_admin_type(admin), admin_id=admin.get_id())"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query_admins_by_group_ids(cls, groups_ids=None):\n assert groups_ids is None or isinstance(groups_ids, list)\n\n query = db.session.query(\n Group.id, func.count(GroupAdmin.id)\n ).join(\n GroupAdmin\n ).group_by(\n Group.id\n )\n\n if groups_ids:\n query = query.filter(Group.id.in_(groups_ids))\n\n return query", "response": "Get count of admins per group."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef all(self):\n '''\n Get all social newtworks profiles\n '''\n\n response = self.api.get(url=PATHS['GET_PROFILES'])\n\n for raw_profile in response:\n self.append(Profile(self.api, raw_profile))\n\n return self", "response": "Get all social newtworks profiles"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a new Profiles object containing only the items chosen by some criteria", "response": "def filter(self, **kwargs):\n '''\n Based on some criteria, filter the profiles and return a new Profiles\n Manager containing only the chosen items\n\n If the manager doen't have any items, get all the profiles from Buffer\n '''\n\n if not len(self):\n self.all()\n\n new_list = filter(lambda item: [True for arg in kwargs if item[arg] == kwargs[arg]] != [], self)\n\n return Profiles(self.api, new_list)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dorun(SNR=20, sweeps=20, burn=8, noise_samples=10):\n radii = np.linspace(2,10,8, endpoint=False)\n crbs, vals, errs = [], [], []\n\n for radius in radii:\n print 'radius', radius\n s,im = pxint(radius=radius, factor=4)\n goodstate = s.state.copy()\n\n common.set_image(s, im, 1.0/SNR)\n tcrb = crb(s)\n tval, terr = sample(s, im, 1.0/SNR, N=noise_samples, sweeps=sweeps, burn=burn)\n crbs.append(tcrb)\n vals.append(tval)\n errs.append(terr)\n\n return np.array(crbs), np.array(vals), np.array(errs), radii", "response": "This function is used to plot the errors introduced by pixelation so we plot the CRB for the current node and the image of the next node."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the skew parameter for direction d", "response": "def _skew(self, x, z, d=0):\n \"\"\" returns the kurtosis parameter for direction d, d=0 is rho, d=1 is z \"\"\"\n # get the top bound determined by the kurtosis\n kval = (np.tanh(self._poly(z, self._kurtosis_coeffs(d)))+1)/12.\n bdpoly = np.array([\n -1.142468e+04, 3.0939485e+03, -2.0283568e+02,\n -2.1047846e+01, 3.79808487e+00, 1.19679781e-02\n ])\n top = np.polyval(bdpoly, kval)\n\n # limit the skewval to be 0 -> top val\n skew = self._poly(z, self._skew_coeffs(d))\n skewval = top*(np.tanh(skew) + 1) - top\n\n return skewval*(3*x - x**3)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the kurtosis parameter for direction d", "response": "def _kurtosis(self, x, z, d=0):\n \"\"\" returns the kurtosis parameter for direction d, d=0 is rho, d=1 is z \"\"\"\n val = self._poly(z, self._kurtosis_coeffs(d))\n return (np.tanh(val)+1)/12.*(3 - 6*x**2 + x**4)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfits a separation of two parties into a single edge", "response": "def fit_edge(separation, radius=5.0, samples=100, imsize=64, sigma=0.05, axis='z'):\n \"\"\"\n axis is 'z' or 'xy'\n seps = np.linspace(0,2,20) 'z'\n seps = np.linspace(-2,2,20) 'xy'\n \"\"\"\n terrors = []\n berrors = []\n crbs = []\n\n for sep in separation:\n print '='*79\n print 'sep =', sep,\n\n s = init.create_two_particle_state(imsize, radius=radius, delta=sep, sigma=0.05,\n axis='z', psfargs={'params': (2.0,1.0,4.0), 'error': 1e-8},\n stateargs={'sigmapad': True, 'pad': const.PAD})\n\n # move off of a pixel edge (cheating for trackpy)\n d = np.array([0,0.5,0.5])\n s.obj.pos -= d\n s.reset()\n\n # move the particles to the edge\n bl = s.blocks_particle(0)\n s.update(bl[0], np.array([s.pad+radius]))\n\n bl = s.blocks_particle(1)\n s.update(bl[0], np.array([s.pad-radius]))\n\n if axis == 'z':\n bl = s.blocks_particle(1)\n s.update(bl[0], s.state[bl[0]]-sep)\n s.model_to_true_image()\n\n if axis == 'xy':\n bl = s.blocks_particle(1)\n s.update(bl[2], s.state[bl[2]]+sep)\n s.model_to_true_image()\n\n # save where the particles were originally so we can jiggle\n p = s.state[s.b_pos].reshape(-1,3).copy()\n\n print p[0], p[1]\n # calculate the CRB for this configuration\n bl = s.explode(s.b_pos)\n crbs.append(np.sqrt(np.diag(np.linalg.inv(s.fisher_information(blocks=bl)))).reshape(-1,3))\n\n # calculate the featuring errors\n tmp_tp, tmp_bf = [],[]\n for i in xrange(samples):\n print i\n bench.jiggle_particles(s, pos=p, sig=0.3, mask=np.array([1,1,1]))\n t = bench.trackpy(s)\n b = bench.bamfpy_positions(s, sweeps=15)\n\n tmp_tp.append(bench.error(s, t))\n tmp_bf.append(bench.error(s, b))\n terrors.append(tmp_tp)\n berrors.append(tmp_bf)\n\n return np.array(crbs), np.array(terrors), np.array(berrors)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef zjitter(jitter=0.0, radius=5):\n psfsize = np.array([2.0, 1.0, 3.0])\n\n # create a base image of one particle\n s0 = init.create_single_particle_state(imsize=4*radius, \n radius=radius, psfargs={'params': psfsize, 'error': 1e-6})\n sl = np.s_[s0.pad:-s0.pad,s0.pad:-s0.pad,s0.pad:-s0.pad]\n\n # add up a bunch of trajectories\n finalimage = 0*s0.get_model_image()[sl]\n position = 0*s0.obj.pos[0]\n\n for i in xrange(finalimage.shape[0]):\n offset = jitter*np.random.randn(3)*np.array([1,0,0])\n s0.obj.pos[0] = np.array(s0.image.shape)/2 + offset\n s0.reset()\n\n finalimage[i] = s0.get_model_image()[sl][i]\n position += s0.obj.pos[0]\n\n position /= float(finalimage.shape[0])\n\n # place that into a new image at the expected parameters\n s = init.create_single_particle_state(imsize=4*radius, sigma=0.05,\n radius=radius, psfargs={'params': psfsize, 'error': 1e-6})\n s.reset()\n\n # measure the true inferred parameters\n return s, finalimage, position", "response": "create a new state and a base image of one particle and a random position"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dorun(SNR=20, njitters=20, samples=10, noise_samples=10, sweeps=20, burn=10):\n jitters = np.logspace(-6, np.log10(0.5), njitters)\n crbs, vals, errs, poss = [], [], [], []\n\n for i,t in enumerate(jitters):\n print '###### jitter', i, t\n\n for j in xrange(samples):\n print 'image', j, '|', \n s,im,pos = zjitter(jitter=t)\n\n # typical image\n common.set_image(s, im, 1.0/SNR)\n crbs.append(crb(s))\n\n val, err = sample(s, im, 1.0/SNR, N=noise_samples, sweeps=sweeps, burn=burn)\n poss.append(pos)\n vals.append(val)\n errs.append(err)\n\n\n shape0 = (njitters, samples, -1)\n shape1 = (njitters, samples, noise_samples, -1)\n\n crbs = np.array(crbs).reshape(shape0)\n vals = np.array(vals).reshape(shape1)\n errs = np.array(errs).reshape(shape1)\n poss = np.array(poss).reshape(shape0)\n\n return [crbs, vals, errs, poss, jitters]", "response": "This function is used to plot the errors introduced by pixelation so we plot the crbs and values and errors for the next time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the detailed information on individual interactions with the social media update such as favorites retweets and likes.", "response": "def interactions(self):\n '''\n Returns the detailed information on individual interactions with the social\n media update such as favorites, retweets and likes.\n '''\n\n interactions = []\n url = PATHS['GET_INTERACTIONS'] % self.id\n\n response = self.api.get(url=url)\n for interaction in response['interactions']:\n interactions.append(ResponseObject(interaction))\n\n self.__interactions = interactions\n\n return self.__interactions"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef edit(self, text, media=None, utc=None, now=None):\n '''\n Edit an existing, individual status update.\n '''\n\n url = PATHS['EDIT'] % self.id\n\n post_data = \"text=%s&\" % text\n\n if now:\n post_data += \"now=%s&\" % now\n\n if utc:\n post_data += \"utc=%s&\" % utc\n\n if media:\n media_format = \"media[%s]=%s&\"\n\n for media_type, media_item in media.iteritems():\n post_data += media_format % (media_type, media_item)\n\n response = self.api.post(url=url, data=post_data)\n\n return Update(api=self.api, raw_response=response['update'])", "response": "Edit an existing individual status update."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef publish(self):\n '''\n Immediately shares a single pending update and recalculates times for\n updates remaining in the queue.\n '''\n\n url = PATHS['PUBLISH'] % self.id\n return self.api.post(url=url)", "response": "Publishes a single pending update and recalculates times for all updates remaining in the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef move_to_top(self):\n '''\n Move an existing status update to the top of the queue and recalculate\n times for all updates in the queue. Returns the update with its new\n posting time.\n '''\n\n url = PATHS['MOVE_TO_TOP'] % self.id\n\n response = self.api.post(url=url)\n return Update(api=self.api, raw_response=response)", "response": "Move an existing status update to the top of the queue and recalculate the time for all updates in the queue. Returns the updated update with the new\n posting time."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove the noise poles from a 2d noise distribution to show that affects the real-space noise picture. noise -- fftshifted 2d array of q values poles -- N,2 list of pole locations. the last index is in the order y,x as determined by mpl interactive plots for example: poles = np.array([[190,277], [227,253], [233, 256]]", "response": "def pole_removal(noise, poles=None, sig=3):\n \"\"\"\n Remove the noise poles from a 2d noise distribution to show that affects\n the real-space noise picture. \n \n noise -- fftshifted 2d array of q values\n\n poles -- N,2 list of pole locations. the last index is in the order y,x as\n determined by mpl interactive plots\n\n for example: poles = np.array([[190,277], [227,253], [233, 256]]\n \"\"\"\n center = np.array(noise.shape)/2\n\n v = np.rollaxis(\n np.array(\n np.meshgrid(*(np.arange(s) for s in noise.shape), indexing='ij')\n ), 0, 3\n ).astype(\"float\")\n filter = np.zeros_like(noise, dtype='float')\n\n for p in poles:\n for pp in [p, center - (p-center)]:\n dist = ((v-pp)**2).sum(axis=-1)\n filter += np.exp(-dist / (2*sig**2))\n\n filter[filter > 1] = 1\n return noise*(1-filter)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pending(self):\n '''\n Returns an array of updates that are currently in the buffer for an\n individual social media profile.\n '''\n\n pending_updates = []\n url = PATHS['GET_PENDING'] % self.profile_id\n\n response = self.api.get(url=url)\n for update in response['updates']:\n pending_updates.append(Update(api=self.api, raw_response=update))\n\n self.__pending = pending_updates\n\n return self.__pending", "response": "Returns an array of updates that are currently in the buffer for an individual social media profile."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sent(self):\n '''\n Returns an array of updates that have been sent from the buffer for an\n individual social media profile.\n '''\n\n sent_updates = []\n url = PATHS['GET_SENT'] % self.profile_id\n\n response = self.api.get(url=url)\n for update in response['updates']:\n sent_updates.append(Update(api=self.api, raw_response=update))\n\n self.__sent = sent_updates\n\n return self.__sent", "response": "Returns an array of updates that have been sent from the buffer for an individual social media profile."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef shuffle(self, count=None, utc=None):\n '''\n Randomize the order at which statuses for the specified social media\n profile will be sent out of the buffer.\n '''\n\n url = PATHS['SHUFFLE'] % self.profile_id\n\n post_data = ''\n if count:\n post_data += 'count=%s&' % count\n if utc:\n post_data += 'utc=%s' % utc\n\n return self.api.post(url=url, data=post_data)", "response": "Randomize the order at which statuses for the specified social media offer will be sent out of the buffer."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef reorder(self, updates_ids, offset=None, utc=None):\n '''\n Edit the order at which statuses for the specified social media profile will\n be sent out of the buffer.\n '''\n\n url = PATHS['REORDER'] % self.profile_id\n\n order_format = \"order[]=%s&\"\n post_data = ''\n\n if offset:\n post_data += 'offset=%s&' % offset\n\n if utc:\n post_data += 'utc=%s&' % utc\n\n for update in updates_ids:\n post_data += order_format % update\n\n return self.api.post(url=url, data=post_data)", "response": "This method is used to reorder the items in a social media profile."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate one or more new status updates.", "response": "def new(self, text, shorten=None, now=None, top=None, media=None, when=None):\n '''\n Create one or more new status updates.\n '''\n\n url = PATHS['CREATE']\n\n post_data = \"text=%s&\" % text\n post_data += \"profile_ids[]=%s&\" % self.profile_id\n\n if shorten:\n post_data += \"shorten=%s&\" % shorten\n\n if now:\n post_data += \"now=%s&\" % now\n\n if top:\n post_data += \"top=%s&\" % top\n\n if when:\n post_data += \"scheduled_at=%s&\" % str(when)\n\n if media:\n media_format = \"media[%s]=%s&\"\n\n for media_type, media_item in media.iteritems():\n post_data += media_format % (media_type, media_item)\n\n response = self.api.post(url=url, data=post_data)\n new_update = Update(api=self.api, raw_response=response['updates'][0])\n\n self.append(new_update)\n\n return new_update"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the logging level of the handlers.", "response": "def set_level(self, level='info', handlers=None):\n \"\"\"\n Set the logging level (which types of logs are actually printed / recorded)\n to one of ['debug', 'info', 'warn', 'error', 'fatal'] in that order\n of severity\n \"\"\"\n for h in self.get_handlers(handlers):\n h.setLevel(levels[level])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_formatter(self, formatter='standard', handlers=None):\n for h in self.get_handlers(handlers):\n h.setFormatter(logging.Formatter(formatters[formatter]))", "response": "Set the text format of messages to one of the pre - deterministic forms"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a new handler to the logging system if not present already.", "response": "def add_handler(self, name='console-color', level='info', formatter='standard', **kwargs):\n \"\"\"\n Add another handler to the logging system if not present already.\n Available handlers are currently: ['console-bw', 'console-color', 'rotating-log']\n \"\"\"\n # make sure the the log file has a name\n if name == 'rotating-log' and 'filename' not in kwargs:\n kwargs.update({'filename': self.logfilename})\n\n # make sure the the log file has a name\n if name == 'stringio' and 'stringio' not in kwargs:\n kwargs.update({'stringio': StringIO.StringIO()})\n\n handler = types[name](**kwargs)\n self.add_handler_raw(handler, name, level=level, formatter=formatter)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_handler(self, name):\n if name in self.handlers:\n self.log.removeHandler(self.handlers[name])", "response": "Remove a handler from the logging system if present."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the verbosity level of a certain log handler or all handlers.", "response": "def set_verbosity(self, verbosity='vvv', handlers=None):\n \"\"\"\n Set the verbosity level of a certain log handler or of all handlers.\n\n Parameters\n ----------\n verbosity : 'v' to 'vvvvv'\n the level of verbosity, more v's is more verbose\n\n handlers : string, or list of strings\n handler names can be found in ``peri.logger.types.keys()``\n Current set is::\n\n ['console-bw', 'console-color', 'rotating-log']\n \"\"\"\n self.verbosity = sanitize(verbosity)\n self.set_level(v2l[verbosity], handlers=handlers)\n self.set_formatter(v2f[verbosity], handlers=handlers)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef normalize(im, invert=False, scale=None, dtype=np.float64):\n if dtype not in {np.float16, np.float32, np.float64}:\n raise ValueError('dtype must be numpy.float16, float32, or float64.')\n out = im.astype('float').copy()\n\n scale = scale or (0.0, 255.0)\n l, u = (float(i) for i in scale)\n out = (out - l) / (u - l)\n if invert:\n out = -out + (out.max() + out.min())\n return out.astype(dtype)", "response": "Normalize a field to a range of exposure values."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a centered boolean mask of a 3D sphere", "response": "def generate_sphere(radius):\n \"\"\"Generates a centered boolean mask of a 3D sphere\"\"\"\n rint = np.ceil(radius).astype('int')\n t = np.arange(-rint, rint+1, 1)\n x,y,z = np.meshgrid(t, t, t, indexing='ij')\n r = np.sqrt(x*x + y*y + z*z)\n sphere = r < radius\n return sphere"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef harris_feature(im, region_size=5, to_return='harris', scale=0.05):\n ndim = im.ndim\n #1. Gradient of image\n grads = [nd.sobel(im, axis=i) for i in range(ndim)]\n #2. Corner response matrix\n matrix = np.zeros((ndim, ndim) + im.shape)\n for a in range(ndim):\n for b in range(ndim):\n matrix[a,b] = nd.filters.gaussian_filter(grads[a]*grads[b],\n region_size)\n if to_return == 'matrix':\n return matrix\n #3. Trace, determinant\n trc = np.trace(matrix, axis1=0, axis2=1)\n det = np.linalg.det(matrix.T).T\n if to_return == 'trace-determinant':\n return trc, det\n else:\n #4. Harris detector:\n harris = det - scale*trc*trc\n return harris", "response": "Harris - motivated feature detection on a d - dimensional image."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nidentifies slabs in an image. Functions by running a Harris-inspired edge detection on the image, thresholding the edge, then clustering. Parameters ---------- im : numpy.ndarray 3D array of the image to analyze. sigma : Float, optional Gaussian blurring kernel to remove non-slab features such as noise and particles. Default is 5. region_size : Int, optional The size of region for Harris corner featuring. Default is 10 masscut : Float, optional The minimum number of pixels for a feature to be identified as a slab. Default is 1e4; should be smaller for smaller images. asdict : Bool, optional Set to True to return a list of dicts, with keys of ``'theta'`` and ``'phi'`` as rotation angles about the x- and z- axes, and of ``'zpos'`` for the z-position, i.e. a list of dicts which can be unpacked into a :class:``peri.comp.objs.Slab`` Returns ------- [poses, normals] : numpy.ndarray The positions and normals of each slab in the image; ``poses[i]`` and ``normals[i]`` are the ``i``th slab. Returned if ``asdict`` is False [list] A list of dictionaries. Returned if ``asdict`` is True", "response": "def identify_slab(im, sigma=5., region_size=10, masscut=1e4, asdict=False):\n \"\"\"\n Identifies slabs in an image.\n\n Functions by running a Harris-inspired edge detection on the image,\n thresholding the edge, then clustering.\n\n Parameters\n ----------\n im : numpy.ndarray\n 3D array of the image to analyze.\n sigma : Float, optional\n Gaussian blurring kernel to remove non-slab features such as\n noise and particles. Default is 5.\n region_size : Int, optional\n The size of region for Harris corner featuring. Default is 10\n masscut : Float, optional\n The minimum number of pixels for a feature to be identified as\n a slab. Default is 1e4; should be smaller for smaller images.\n asdict : Bool, optional\n Set to True to return a list of dicts, with keys of ``'theta'``\n and ``'phi'`` as rotation angles about the x- and z- axes, and\n of ``'zpos'`` for the z-position, i.e. a list of dicts which\n can be unpacked into a :class:``peri.comp.objs.Slab``\n\n Returns\n -------\n [poses, normals] : numpy.ndarray\n The positions and normals of each slab in the image; ``poses[i]``\n and ``normals[i]`` are the ``i``th slab. Returned if ``asdict``\n is False\n [list]\n A list of dictionaries. Returned if ``asdict`` is True\n \"\"\"\n #1. edge detect:\n fim = nd.filters.gaussian_filter(im, sigma)\n trc, det = harris_feature(fim, region_size, to_return='trace-determinant')\n #we want an edge == not a corner, so one eigenvalue is high and\n #one is low compared to the other.\n #So -- trc high, normalized det low:\n dnrm = det / (trc*trc)\n trc_cut = otsu_threshold(trc)\n det_cut = otsu_threshold(dnrm)\n slabs = (trc > trc_cut) & (dnrm < det_cut)\n labeled, nslabs = nd.label(slabs)\n #masscuts:\n masses = [(labeled == i).sum() for i in range(1, nslabs+1)]\n good = np.array([m > masscut for m in masses])\n inds = np.nonzero(good)[0] + 1 #+1 b/c the lowest label is the bkg\n #Slabs are identifiied, now getting the coords:\n poses = np.array(nd.measurements.center_of_mass(trc, labeled, inds))\n #normals from eigenvectors of the covariance matrix\n normals = []\n z = np.arange(im.shape[0]).reshape(-1,1,1).astype('float')\n y = np.arange(im.shape[1]).reshape(1,-1,1).astype('float')\n x = np.arange(im.shape[2]).reshape(1,1,-1).astype('float')\n #We also need to identify the direction of the normal:\n gim = [nd.sobel(fim, axis=i) for i in range(fim.ndim)]\n for i, p in zip(range(1, nslabs+1), poses):\n wts = trc * (labeled == i)\n wts /= wts.sum()\n zc, yc, xc = [xi-pi for xi, pi in zip([z,y,x],p.squeeze())]\n cov = [[np.sum(xi*xj*wts) for xi in [zc,yc,xc]] for xj in [zc,yc,xc]]\n vl, vc = np.linalg.eigh(cov)\n #lowest eigenvalue is the normal:\n normal = vc[:,0]\n #Removing the sign ambiguity:\n gn = np.sum([n*g[tuple(p.astype('int'))] for g,n in zip(gim, normal)])\n normal *= np.sign(gn)\n normals.append(normal)\n if asdict:\n get_theta = lambda n: -np.arctan2(n[1], -n[0])\n get_phi = lambda n: np.arcsin(n[2])\n return [{'zpos':p[0], 'angles':(get_theta(n), get_phi(n))}\n for p, n in zip(poses, normals)]\n else:\n return poses, np.array(normals)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nplot the errors for a single crystal.", "response": "def plot_errors_single(rad, crb, errors, labels=['trackpy', 'peri']):\n fig = pl.figure()\n comps = ['z', 'y', 'x']\n markers = ['o', '^', '*']\n colors = COLORS\n\n for i in reversed(range(3)):\n pl.plot(rad, crb[:,0,i], lw=2.5, label='CRB-'+comps[i], color=colors[i])\n\n for c, (error, label) in enumerate(zip(errors, labels)):\n mu = np.sqrt((error**2).mean(axis=1))[:,0,:]\n std = np.std(np.sqrt((error**2)), axis=1)[:,0,:]\n\n for i in reversed(range(len(mu[0]))):\n pl.plot(rad, mu[:,i], marker=markers[c], color=colors[i], lw=0, label=label+\"-\"+comps[i], ms=13)\n\n pl.ylim(1e-3, 8e0)\n pl.semilogy()\n pl.legend(loc='upper left', ncol=3, numpoints=1, prop={\"size\": 16})\n pl.xlabel(r\"radius (pixels)\")\n pl.ylabel(r\"CRB / $\\Delta$ (pixels)\")\n \"\"\" \n ax = fig.add_axes([0.6, 0.6, 0.28, 0.28])\n ax.plot(rad, crb[:,0,:], lw=2.5)\n for c, error in enumerate(errors):\n mu = np.sqrt((error**2).mean(axis=1))[:,0,:]\n std = np.std(np.sqrt((error**2)), axis=1)[:,0,:]\n\n for i in range(len(mu[0])):\n ax.errorbar(rad, mu[:,i], yerr=std[:,i], fmt=markers[c], color=colors[i], lw=1)\n ax.set_ylim(-0.1, 1.5)\n ax.grid('off')\n \"\"\""} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sphere_analytical_gaussian(dr, a, alpha=0.2765):\n term1 = 0.5*(erf((dr+2*a)/(alpha*np.sqrt(2))) + erf(-dr/(alpha*np.sqrt(2))))\n term2 = np.sqrt(0.5/np.pi)*(alpha/(dr+a+1e-10)) * (\n np.exp(-0.5*dr**2/alpha**2) - np.exp(-0.5*(dr+2*a)**2/alpha**2)\n )\n return term1 - term2", "response": "Analytically calculate the sphere s functional form by convolving the solar system with first order approximation to the sinc a Gaussian."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nseeing sphere_analytical_gaussian_exact. I trimmed to terms from the functional form that are essentially zero (1e-8) for r0 > cut (~1.5), a fine approximation for these platonic anyway.", "response": "def sphere_analytical_gaussian_trim(dr, a, alpha=0.2765, cut=1.6):\n \"\"\"\n See sphere_analytical_gaussian_exact.\n\n I trimmed to terms from the functional form that are essentially zero (1e-8)\n for r0 > cut (~1.5), a fine approximation for these platonic anyway.\n \"\"\"\n m = np.abs(dr) <= cut\n\n # only compute on the relevant scales\n rr = dr[m]\n t = -rr/(alpha*np.sqrt(2))\n q = 0.5*(1 + erf(t)) - np.sqrt(0.5/np.pi)*(alpha/(rr+a+1e-10)) * np.exp(-t*t)\n\n # fill in the grid, inside the interpolation and outside where values are constant\n ans = 0*dr\n ans[m] = q\n ans[dr > cut] = 0\n ans[dr < -cut] = 1\n return ans"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nseeing sphere_analytical_gaussian_trim, but implemented in C with fast erf and exp approximations found at Abramowitz and Stegun: Handbook of Mathematical Functions A Fast, Compact Approximation of the Exponential Function The default cut 1.25 was chosen based on the accuracy of fast_erf", "response": "def sphere_analytical_gaussian_fast(dr, a, alpha=0.2765, cut=1.20):\n \"\"\"\n See sphere_analytical_gaussian_trim, but implemented in C with\n fast erf and exp approximations found at\n Abramowitz and Stegun: Handbook of Mathematical Functions\n A Fast, Compact Approximation of the Exponential Function\n\n The default cut 1.25 was chosen based on the accuracy of fast_erf\n \"\"\"\n\n code = \"\"\"\n double coeff1 = 1.0/(alpha*sqrt(2.0));\n double coeff2 = sqrt(0.5/pi)*alpha;\n\n for (int i=0; i -cut){\n double t = -dri*coeff1;\n ans[i] = 0.5*(1+fast_erf(t)) - coeff2/(dri+a+1e-10) * fast_exp(-t*t);\n } else {\n ans[i] = 0.0*(dri > cut) + 1.0*(dri < -cut);\n }\n }\n \"\"\"\n\n shape = r.shape\n r = r.flatten()\n N = self.N\n ans = r*0\n pi = np.pi\n\n inline(code, arg_names=['dr', 'a', 'alpha', 'cut', 'ans', 'pi', 'N'],\n support_code=functions, verbose=0)\n return ans.reshape(shape)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the surface of a cubic interpolant constrained to be a sphere.", "response": "def sphere_constrained_cubic(dr, a, alpha):\n \"\"\"\n Sphere generated by a cubic interpolant constrained to be (1,0) on\n (r0-sqrt(3)/2, r0+sqrt(3)/2), the size of the cube in the (111) direction.\n \"\"\"\n sqrt3 = np.sqrt(3)\n\n b_coeff = a*0.5/sqrt3*(1 - 0.6*sqrt3*alpha)/(0.15 + a*a)\n rscl = np.clip(dr, -0.5*sqrt3, 0.5*sqrt3)\n\n a, d = rscl + 0.5*sqrt3, rscl - 0.5*sqrt3\n return alpha*d*a*rscl + b_coeff*d*a - d/sqrt3"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nperforms an iterative method to calculate the effective sphere that conserves volume.", "response": "def exact_volume_sphere(rvec, pos, radius, zscale=1.0, volume_error=1e-5,\n function=sphere_analytical_gaussian, max_radius_change=1e-2, args=()):\n \"\"\"\n Perform an iterative method to calculate the effective sphere that perfectly\n (up to the volume_error) conserves volume. Return the resulting image\n \"\"\"\n vol_goal = 4./3*np.pi*radius**3 / zscale\n rprime = radius\n\n dr = inner(rvec, pos, rprime, zscale=zscale)\n t = function(dr, rprime, *args)\n for i in range(MAX_VOLUME_ITERATIONS):\n vol_curr = np.abs(t.sum())\n if np.abs(vol_goal - vol_curr)/vol_goal < volume_error:\n break\n\n rprime = rprime + 1.0*(vol_goal - vol_curr) / (4*np.pi*rprime**2)\n\n if np.abs(rprime - radius)/radius > max_radius_change:\n break\n\n dr = inner(rvec, pos, rprime, zscale=zscale)\n t = function(dr, rprime, *args)\n\n return t"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the update tile surrounding particle n", "response": "def _tile(self, n):\n \"\"\"Get the update tile surrounding particle `n` \"\"\"\n pos = self._trans(self.pos[n])\n return Tile(pos, pos).pad(self.support_pad)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _p2i(self, param):\n g = param.split('-')\n if len(g) == 3:\n return g[2], int(g[1])\n else:\n raise ValueError('`param` passed as incorrect format')", "response": "Convert parameter to indices returns coord index"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninitializing all objects and draw particles", "response": "def initialize(self):\n \"\"\"Start from scratch and initialize all objects / draw self.particles\"\"\"\n self.particles = np.zeros(self.shape.shape, dtype=self.float_precision)\n\n for p0, arg0 in zip(self.pos, self._drawargs()):\n self._draw_particle(p0, *listify(arg0))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _vps(self, inds):\n return [j for j in inds if j >= 0 and j < self.N]", "response": "Clips a list of inds to be on [ 0 self. N )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntranslate index info to parameter name", "response": "def _i2p(self, ind, coord):\n \"\"\" Translate index info to parameter name \"\"\"\n return '-'.join([self.param_prefix, str(ind), coord])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the amount of support size required for a particular update.", "response": "def get_update_tile(self, params, values):\n \"\"\" Get the amount of support size required for a particular update.\"\"\"\n doglobal, particles = self._update_type(params)\n if doglobal:\n return self.shape.copy()\n\n # 1) store the current parameters of interest\n values0 = self.get_values(params)\n # 2) calculate the current tileset\n tiles0 = [self._tile(n) for n in particles]\n\n # 3) update to newer parameters and calculate tileset\n self.set_values(params, values)\n tiles1 = [self._tile(n) for n in particles]\n\n # 4) revert parameters & return union of all tiles\n self.set_values(params, values0)\n return Tile.boundingtile(tiles0 + tiles1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the particles field given new parameter values", "response": "def update(self, params, values):\n \"\"\"\n Update the particles field given new parameter values\n \"\"\"\n #1. Figure out if we're going to do a global update, in which\n # case we just draw from scratch.\n global_update, particles = self._update_type(params)\n\n # if we are doing a global update, everything must change, so\n # starting fresh will be faster instead of add subtract\n if global_update:\n self.set_values(params, values)\n self.initialize()\n return\n\n # otherwise, update individual particles. delete the current versions\n # of the particles update the particles, and redraw them anew at the\n # places given by (params, values)\n oldargs = self._drawargs()\n for n in particles:\n self._draw_particle(self.pos[n], *listify(oldargs[n]), sign=-1)\n\n self.set_values(params, values)\n\n newargs = self._drawargs()\n for n in particles:\n self._draw_particle(self.pos[n], *listify(newargs[n]), sign=+1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting position and radius of one or more particles", "response": "def param_particle(self, ind):\n \"\"\" Get position and radius of one or more particles \"\"\"\n ind = self._vps(listify(ind))\n return [self._i2p(i, j) for i in ind for j in ['z', 'y', 'x', 'a']]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef param_particle_pos(self, ind):\n ind = self._vps(listify(ind))\n return [self._i2p(i, j) for i in ind for j in ['z', 'y', 'x']]", "response": "Get position of one or more particles"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef param_particle_rad(self, ind):\n ind = self._vps(listify(ind))\n return [self._i2p(i, 'a') for i in ind]", "response": "Get radius of one or more particles"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_particle(self, pos, rad):\n rad = listify(rad)\n # add some zero mass particles to the list (same as not having these\n # particles in the image, which is true at this moment)\n inds = np.arange(self.N, self.N+len(rad))\n self.pos = np.vstack([self.pos, pos])\n self.rad = np.hstack([self.rad, np.zeros(len(rad))])\n\n # update the parameters globally\n self.setup_variables()\n self.trigger_parameter_change()\n\n # now request a drawing of the particle plz\n params = self.param_particle_rad(inds)\n self.trigger_update(params, rad)\n return inds", "response": "Add a particle or list of particles given by a list of positions and radii."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_particle(self, inds):\n if self.rad.shape[0] == 0:\n return\n\n inds = listify(inds)\n\n # Here's the game plan:\n # 1. get all positions and sizes of particles that we will be\n # removing (to return to user)\n # 2. redraw those particles to 0.0 radius\n # 3. remove the particles and trigger changes\n # However, there is an issue -- if there are two particles at opposite\n # ends of the image, it will be significantly slower than usual\n pos = self.pos[inds].copy()\n rad = self.rad[inds].copy()\n\n self.trigger_update(self.param_particle_rad(inds), np.zeros(len(inds)))\n\n self.pos = np.delete(self.pos, inds, axis=0)\n self.rad = np.delete(self.rad, inds, axis=0)\n\n # update the parameters globally\n self.setup_variables()\n self.trigger_parameter_change()\n return np.array(pos).reshape(-1,3), np.array(rad).reshape(-1)", "response": "Removes the particle at the given indices. Returns the positions and radius of the particle at the given indices."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _update_type(self, params):\n dozscale = False\n particles = []\n for p in listify(params):\n typ, ind = self._p2i(p)\n particles.append(ind)\n dozscale = dozscale or typ == 'zscale'\n particles = set(particles)\n return dozscale, particles", "response": "Returns the dozscale and particle list of update"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _tile(self, n):\n zsc = np.array([1.0/self.zscale, 1, 1])\n pos, rad = self.pos[n], self.rad[n]\n pos = self._trans(pos)\n return Tile(pos - zsc*rad, pos + zsc*rad).pad(self.support_pad)", "response": "Get the tile surrounding particle n"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update(self, params, values):\n # radparams = self.param_radii()\n params = listify(params)\n values = listify(values)\n for i, p in enumerate(params):\n # if (p in radparams) & (values[i] < 0):\n if (p[-2:] == '-a') and (values[i] < 0):\n values[i] = 0.0\n super(PlatonicSpheresCollection, self).update(params, values)", "response": "Calls an update but clips radii to be > 0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rmatrix(self):\n t = self.param_dict[self.lbl_theta]\n r0 = np.array([ [np.cos(t), -np.sin(t), 0],\n [np.sin(t), np.cos(t), 0],\n [0, 0, 1]])\n\n p = self.param_dict[self.lbl_phi]\n r1 = np.array([ [np.cos(p), 0, np.sin(p)],\n [0, 1, 0],\n [-np.sin(p), 0, np.cos(p)]])\n return np.dot(r1, r0)", "response": "Generate the composite rotation matrix that rotates the slab normal."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef j2(x):\n to_return = 2./(x+1e-15)*j1(x) - j0(x)\n to_return[x==0] = 0\n return to_return", "response": "A fast j2 function that returns the value of x."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef calc_pts_hg(npts=20):\n pts_hg, wts_hg = np.polynomial.hermite.hermgauss(npts*2)\n pts_hg = pts_hg[npts:]\n wts_hg = wts_hg[npts:] * np.exp(pts_hg*pts_hg)\n return pts_hg, wts_hg", "response": "Calculates the quadrature points for even functions"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef calc_pts_lag(npts=20):\n scl = { 15:0.072144,\n 20:0.051532,\n 25:0.043266}[npts]\n pts0, wts0 = np.polynomial.laguerre.laggauss(npts)\n pts = np.sinh(pts0*scl)\n wts = scl*wts0*np.cosh(pts0*scl)*np.exp(pts0)\n return pts, wts", "response": "Calculates Gauss - Laguerre quadrature points rescaled for a single line scan."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the wavefront distortion due to z at a given cos_theta and z.", "response": "def f_theta(cos_theta, zint, z, n2n1=0.95, sph6_ab=None, **kwargs):\n \"\"\"\n Returns the wavefront aberration for an aberrated, defocused lens.\n\n Calculates the portions of the wavefront distortion due to z, theta\n only, for a lens with defocus and spherical aberration induced by\n coverslip mismatch. (The rho portion can be analytically integrated\n to Bessels.)\n\n Parameters\n ----------\n cos_theta : numpy.ndarray.\n The N values of cos(theta) at which to compute f_theta.\n zint : Float\n The position of the lens relative to the interface.\n z : numpy.ndarray\n The M z-values to compute f_theta at. `z.size` is unrelated\n to `cos_theta.size`\n n2n1: Float, optional\n The ratio of the index of the immersed medium to the optics.\n Default is 0.95\n sph6_ab : Float or None, optional\n Set sph6_ab to a nonzero value to add residual 6th-order\n spherical aberration that is proportional to sph6_ab. Default\n is None (i.e. doesn't calculate).\n\n Returns\n -------\n wvfront : numpy.ndarray\n The aberrated wavefront, as a function of theta and z.\n Shape is [z.size, cos_theta.size]\n \"\"\"\n wvfront = (np.outer(np.ones_like(z)*zint, cos_theta) -\n np.outer(zint+z, csqrt(n2n1**2-1+cos_theta**2)))\n if (sph6_ab is not None) and (not np.isnan(sph6_ab)):\n sec2_theta = 1.0/(cos_theta*cos_theta)\n wvfront += sph6_ab * (sec2_theta-1)*(sec2_theta-2)*cos_theta\n #Ensuring evanescent waves are always suppressed:\n if wvfront.dtype == np.dtype('complex128'):\n wvfront.imag = -np.abs(wvfront.imag)\n return wvfront"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_Kprefactor(z, cos_theta, zint=100.0, n2n1=0.95, get_hdet=False,\n **kwargs):\n \"\"\"\n Returns a prefactor in the electric field integral.\n\n This is an internal function called by get_K. The returned prefactor\n in the integrand is independent of which integral is being called;\n it is a combination of the exp(1j*phase) and apodization.\n\n Parameters\n ----------\n z : numpy.ndarray\n The values of z (distance along optical axis) at which to\n calculate the prefactor. Size is unrelated to the size of\n `cos_theta`\n cos_theta : numpy.ndarray\n The values of cos(theta) (i.e. position on the incoming\n focal spherical wavefront) at which to calculate the\n prefactor. Size is unrelated to the size of `z`\n zint : Float, optional\n The position of the optical interface, in units of 1/k.\n Default is 100.\n n2n1 : Float, optional\n The ratio of the index mismatch between the optics (n1) and\n the sample (n2). Default is 0.95\n get_hdet : Bool, optional\n Set to True to calculate the detection prefactor vs the\n illumination prefactor (i.e. False to include apodization).\n Default is False\n\n Returns\n -------\n numpy.ndarray\n The prefactor, of size [`z.size`, `cos_theta.size`], sampled\n at the values [`z`, `cos_theta`]\n \"\"\"\n phase = f_theta(cos_theta, zint, z, n2n1=n2n1, **kwargs)\n to_return = np.exp(-1j*phase)\n if not get_hdet:\n to_return *= np.outer(np.ones_like(z),np.sqrt(cos_theta))\n\n return to_return", "response": "Calculates the prefactor of the illumination for the given order of the electric field."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the symmetric and asymmetric portions of a confocal PSF.", "response": "def get_hsym_asym(rho, z, get_hdet=False, include_K3_det=True, **kwargs):\n \"\"\"\n Calculates the symmetric and asymmetric portions of a confocal PSF.\n\n Parameters\n ----------\n rho : numpy.ndarray\n Rho in cylindrical coordinates, in units of 1/k.\n z : numpy.ndarray\n Z in cylindrical coordinates, in units of 1/k. Must be the\n same shape as `rho`\n get_hdet : Bool, optional\n Set to True to get the detection portion of the psf; False\n to get the illumination portion of the psf. Default is True\n include_K3_det : Bool, optional.\n Flag to not calculate the `K3' component for the detection\n PSF, corresponding to (I think) a low-aperature focusing\n lens and no z-polarization of the focused light. Default\n is True, i.e. calculates the K3 component as if the focusing\n lens is high-aperture\n\n Other Parameters\n ----------------\n alpha : Float, optional\n The acceptance angle of the lens, on (0,pi/2). Default is 1.\n zint : Float, optional\n The distance of the len's unaberrated focal point from the\n optical interface, in units of 1/k. Default is 100.\n n2n1 : Float, optional\n The ratio n2/n1 of the index mismatch between the sample\n (index n2) and the optical train (index n1). Must be on\n [0,inf) but should be near 1. Default is 0.95\n\n Returns\n -------\n hsym : numpy.ndarray\n `rho`.shape numpy.array of the symmetric portion of the PSF\n hasym : numpy.ndarray\n `rho`.shape numpy.array of the asymmetric portion of the PSF\n \"\"\"\n\n K1, Kprefactor = get_K(rho, z, K=1, get_hdet=get_hdet, Kprefactor=None,\n return_Kprefactor=True, **kwargs)\n K2 = get_K(rho, z, K=2, get_hdet=get_hdet, Kprefactor=Kprefactor,\n return_Kprefactor=False, **kwargs)\n\n if get_hdet and not include_K3_det:\n K3 = 0*K1\n else:\n K3 = get_K(rho, z, K=3, get_hdet=get_hdet, Kprefactor=Kprefactor,\n return_Kprefactor=False, **kwargs)\n\n hsym = K1*K1.conj() + K2*K2.conj() + 0.5*(K3*K3.conj())\n hasym= K1*K2.conj() + K2*K1.conj() + 0.5*(K3*K3.conj())\n\n return hsym.real, hasym.real"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the perfect - pinhole PSF for a set of points x y z.", "response": "def calculate_pinhole_psf(x, y, z, kfki=0.89, zint=100.0, normalize=False,\n **kwargs):\n \"\"\"\n Calculates the perfect-pinhole PSF, for a set of points (x,y,z).\n\n Parameters\n -----------\n x : numpy.ndarray\n The x-coordinate of the PSF in units of 1/ the wavevector of\n the incoming light.\n y : numpy.ndarray\n The y-coordinate.\n z : numpy.ndarray\n The z-coordinate.\n kfki : Float\n The (scalar) ratio of wavevectors of the outgoing light to the\n incoming light. Default is 0.89.\n zint : Float\n The (scalar) distance from the interface, in units of\n 1/k_incoming. Default is 100.0\n normalize : Bool\n Set to True to normalize the psf correctly, accounting for\n intensity variations with depth. This will give a psf that does\n not sum to 1.\n\n Other Parameters\n ----------------\n alpha : Float\n The opening angle of the lens. Default is 1.\n n2n1 : Float\n The ratio of the index in the 2nd medium to that in the first.\n Default is 0.95\n\n Returns\n -------\n psf : numpy.ndarray, of shape x.shape\n\n Comments\n --------\n (1) The PSF is not necessarily centered on the z=0 pixel, since the\n calculation includes the shift.\n\n (2) If you want z-varying illumination of the psf then set\n normalize=True. This does the normalization by doing:\n hsym, hasym /= hsym.sum()\n hdet /= hdet.sum()\n and then calculating the psf that way. So if you want the\n intensity to be correct you need to use a large-ish array of\n roughly equally spaced points. Or do it manually by calling\n get_hsym_asym()\n \"\"\"\n rho = np.sqrt(x**2 + y**2)\n phi = np.arctan2(y, x)\n\n hsym, hasym = get_hsym_asym(rho, z, zint=zint, get_hdet=False, **kwargs)\n hdet, toss = get_hsym_asym(rho*kfki, z*kfki, zint=kfki*zint, get_hdet=True, **kwargs)\n\n if normalize:\n hasym /= hsym.sum()\n hsym /= hsym.sum()\n hdet /= hdet.sum()\n\n return (hsym + np.cos(2*phi)*hasym)*hdet"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate a set of points and weights for a set of polydisp outgoing wavevectors.", "response": "def get_polydisp_pts_wts(kfki, sigkf, dist_type='gaussian', nkpts=3):\n \"\"\"\n Calculates a set of Gauss quadrature points & weights for polydisperse\n light.\n\n Returns a list of points and weights of the final wavevector's distri-\n bution, in units of the initial wavevector.\n\n Parameters\n ----------\n kfki : Float\n The mean of the polydisperse outgoing wavevectors.\n sigkf : Float\n The standard dev. of the polydisperse outgoing wavevectors.\n dist_type : {`gaussian`, `gamma`}, optional\n The distribution, gaussian or gamma, of the wavevectors.\n Default is `gaussian`\n nkpts : Int, optional\n The number of quadrature points to use. Default is 3\n Returns\n -------\n kfkipts : numpy.ndarray\n The Gauss quadrature points at which to calculate kfki.\n wts : numpy.ndarray\n The associated Gauss quadrature weights.\n \"\"\"\n if dist_type.lower() == 'gaussian':\n pts, wts = np.polynomial.hermite.hermgauss(nkpts)\n kfkipts = np.abs(kfki + sigkf*np.sqrt(2)*pts)\n elif dist_type.lower() == 'laguerre' or dist_type.lower() == 'gamma':\n k_scale = sigkf**2/kfki\n associated_order = kfki**2/sigkf**2 - 1\n #Associated Laguerre with alpha >~170 becomes numerically unstable, so:\n max_order=150\n if associated_order > max_order or associated_order < (-1+1e-3):\n warnings.warn('Numerically unstable sigk, clipping', RuntimeWarning)\n associated_order = np.clip(associated_order, -1+1e-3, max_order)\n kfkipts, wts = la_roots(nkpts, associated_order)\n kfkipts *= k_scale\n else:\n raise ValueError('dist_type must be either gaussian or laguerre')\n return kfkipts, wts/wts.sum()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the PSF for a set of points x y z.", "response": "def calculate_polychrome_pinhole_psf(x, y, z, normalize=False, kfki=0.889,\n sigkf=0.1, zint=100., nkpts=3, dist_type='gaussian', **kwargs):\n \"\"\"\n Calculates the perfect-pinhole PSF, for a set of points (x,y,z).\n\n Parameters\n -----------\n x : numpy.ndarray\n The x-coordinate of the PSF in units of 1/ the wavevector of\n the incoming light.\n y : numpy.ndarray\n The y-coordinate.\n z : numpy.ndarray\n The z-coordinate.\n kfki : Float\n The mean ratio of the outgoing light's wavevector to the incoming\n light's. Default is 0.89.\n sigkf : Float\n Standard deviation of kfki; the distribution of the light values\n will be approximately kfki +- sigkf.\n zint : Float\n The (scalar) distance from the interface, in units of\n 1/k_incoming. Default is 100.0\n dist_type: The distribution type of the polychromatic light.\n Can be one of 'laguerre'/'gamma' or 'gaussian.' If 'gaussian'\n the resulting k-values are taken in absolute value. Default\n is 'gaussian.'\n normalize : Bool\n Set to True to normalize the psf correctly, accounting for\n intensity variations with depth. This will give a psf that does\n not sum to 1. Default is False.\n\n Other Parameters\n ----------------\n alpha : Float\n The opening angle of the lens. Default is 1.\n n2n1 : Float\n The ratio of the index in the 2nd medium to that in the first.\n Default is 0.95\n\n Returns\n -------\n psf : numpy.ndarray, of shape x.shape\n\n Comments\n --------\n (1) The PSF is not necessarily centered on the z=0 pixel, since the\n calculation includes the shift.\n\n (2) If you want z-varying illumination of the psf then set\n normalize=True. This does the normalization by doing:\n hsym, hasym /= hsym.sum()\n hdet /= hdet.sum()\n and then calculating the psf that way. So if you want the\n intensity to be correct you need to use a large-ish array of\n roughly equally spaced points. Or do it manually by calling\n get_hsym_asym()\n \"\"\"\n #0. Setup\n kfkipts, wts = get_polydisp_pts_wts(kfki, sigkf, dist_type=dist_type,\n nkpts=nkpts)\n rho = np.sqrt(x**2 + y**2)\n phi = np.arctan2(y, x)\n\n #1. Hilm\n hsym, hasym = get_hsym_asym(rho, z, zint=zint, get_hdet=False, **kwargs)\n hilm = (hsym + np.cos(2*phi)*hasym)\n\n #2. Hdet\n hdet_func = lambda kfki: get_hsym_asym(rho*kfki, z*kfki,\n zint=kfki*zint, get_hdet=True, **kwargs)[0]\n inner = [wts[a] * hdet_func(kfkipts[a]) for a in range(nkpts)]\n hdet = np.sum(inner, axis=0)\n\n #3. Normalize and return\n if normalize:\n hilm /= hilm.sum()\n hdet /= hdet.sum()\n psf = hdet * hilm\n return psf if normalize else psf / psf.sum()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_psf_scalar(x, y, z, kfki=1., zint=100.0, normalize=False, **kwargs):\n\n rho = np.sqrt(x**2 + y**2)\n phi = np.arctan2(y, x)\n\n K1 = get_K(rho, z, K=1,zint=zint,get_hdet=True, **kwargs)\n hilm = np.real( K1*K1.conj() )\n\n if np.abs(kfki - 1.0) > 1e-13:\n Kdet = get_K(rho*kfki, z*kfki, K=1, zint=zint*kfki, get_hdet=True,\n **kwargs)\n hdet = np.real( Kdet*Kdet.conj() )\n else:\n hdet = hilm.copy()\n\n if normalize:\n hilm /= hsym.sum()\n hdet /= hdet.sum()\n psf = hilm * hdet\n else:\n psf = hilm * hdet\n # psf /= psf.sum()\n\n return psf", "response": "Calculates the scalar PSF of a specific entry in the system."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the illumination PSF for a line - scanning confocal with the illuminating light.", "response": "def calculate_linescan_ilm_psf(y,z, polar_angle=0., nlpts=1,\n pinhole_width=1, use_laggauss=False, **kwargs):\n \"\"\"\n Calculates the illumination PSF for a line-scanning confocal with the\n confocal line oriented along the x direction.\n\n Parameters\n ----------\n y : numpy.ndarray\n The y points (in-plane, perpendicular to the line direction)\n at which to evaluate the illumination PSF, in units of 1/k.\n Arbitrary shape.\n z : numpy.ndarray\n The z points (optical axis) at which to evaluate the illum-\n ination PSF, in units of 1/k. Must be the same shape as `y`\n polar_angle : Float, optional\n The angle of the illuminating light's polarization with\n respect to the line's orientation along x. Default is 0.\n pinhole_width : Float, optional\n The width of the geometric image of the line projected onto\n the sample, in units of 1/k. Default is 1. The perfect line\n image is assumed to be a Gaussian. If `nlpts` is set to 1,\n the line will always be of zero width.\n nlpts : Int, optional\n The number of points to use for Hermite-gauss quadrature over\n the line's width. Default is 1, corresponding to a zero-width\n line.\n use_laggauss : Bool, optional\n Set to True to use a more-accurate sinh'd Laguerre-Gauss\n quadrature for integration over the line's length (more accurate\n in the same amount of time). Default is False for backwards\n compatibility. FIXME what did we do here?\n\n Other Parameters\n ----------------\n alpha : Float, optional\n The acceptance angle of the lens, on (0,pi/2). Default is 1.\n zint : Float, optional\n The distance of the len's unaberrated focal point from the\n optical interface, in units of 1/k. Default is 100.\n n2n1 : Float, optional\n The ratio n2/n1 of the index mismatch between the sample\n (index n2) and the optical train (index n1). Must be on\n [0,inf) but should be near 1. Default is 0.95\n\n Returns\n -------\n hilm : numpy.ndarray\n The line illumination, of the same shape as y and z.\n \"\"\"\n if use_laggauss:\n x_vals, wts = calc_pts_lag()\n else:\n x_vals, wts = calc_pts_hg()\n\n #I'm assuming that y,z are already some sort of meshgrid\n xg, yg, zg = [np.zeros( list(y.shape) + [x_vals.size] ) for a in range(3)]\n hilm = np.zeros(xg.shape)\n\n for a in range(x_vals.size):\n xg[...,a] = x_vals[a]\n yg[...,a] = y.copy()\n zg[...,a] = z.copy()\n\n y_pinhole, wts_pinhole = np.polynomial.hermite.hermgauss(nlpts)\n y_pinhole *= np.sqrt(2)*pinhole_width\n wts_pinhole /= np.sqrt(np.pi)\n\n #Pinhole hermgauss first:\n for yp, wp in zip(y_pinhole, wts_pinhole):\n rho = np.sqrt(xg*xg + (yg-yp)*(yg-yp))\n phi = np.arctan2(yg,xg)\n\n hsym, hasym = get_hsym_asym(rho,zg,get_hdet = False, **kwargs)\n hilm += wp*(hsym + np.cos(2*(phi-polar_angle))*hasym)\n\n #Now line hermgauss\n for a in range(x_vals.size):\n hilm[...,a] *= wts[a]\n\n return hilm.sum(axis=-1)*2."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef calculate_linescan_psf(x, y, z, normalize=False, kfki=0.889, zint=100.,\n polar_angle=0., wrap=True, **kwargs):\n \"\"\"\n Calculates the point spread function of a line-scanning confocal.\n\n Make x,y,z __1D__ numpy.arrays, with x the direction along the\n scan line. (to make the calculation faster since I dont' need the line\n ilm for each x).\n\n Parameters\n ----------\n x : numpy.ndarray\n _One_dimensional_ array of the x grid points (along the line\n illumination) at which to evaluate the psf. In units of\n 1/k_incoming.\n y : numpy.ndarray\n _One_dimensional_ array of the y grid points (in plane,\n perpendicular to the line illumination) at which to evaluate\n the psf. In units of 1/k_incoming.\n z : numpy.ndarray\n _One_dimensional_ array of the z grid points (along the\n optical axis) at which to evaluate the psf. In units of\n 1/k_incoming.\n normalize : Bool, optional\n Set to True to include the effects of PSF normalization on\n the image intensity. Default is False.\n kfki : Float, optional\n The ratio of the final light's wavevector to the incoming.\n Default is 0.889\n zint : Float, optional\n The position of the optical interface, in units of 1/k_incoming\n Default is 100.\n wrap : Bool, optional\n If True, wraps the psf calculation for speed, assuming that\n the input x, y are regularly-spaced points. If x,y are not\n regularly spaced then `wrap` must be set to False. Default is True.\n polar_angle : Float, optional\n The polarization angle of the light (radians) with respect to\n the line direction (x). Default is 0.\n\n Other Parameters\n ----------------\n alpha : Float\n The opening angle of the lens. Default is 1.\n n2n1 : Float\n The ratio of the index in the 2nd medium to that in the first.\n Default is 0.95\n\n Returns\n -------\n numpy.ndarray\n A 3D- numpy.array of the point-spread function. Indexing is\n psf[x,y,z]; shape is [x.size, y,size, z.size]\n \"\"\"\n\n #0. Set up vecs\n if wrap:\n xpts = vec_to_halfvec(x)\n ypts = vec_to_halfvec(y)\n x3, y3, z3 = np.meshgrid(xpts, ypts, z, indexing='ij')\n else:\n x3,y3,z3 = np.meshgrid(x, y, z, indexing='ij')\n rho3 = np.sqrt(x3*x3 + y3*y3)\n\n #1. Hilm\n if wrap:\n y2,z2 = np.meshgrid(ypts, z, indexing='ij')\n hilm0 = calculate_linescan_ilm_psf(y2, z2, zint=zint,\n polar_angle=polar_angle, **kwargs)\n if ypts[0] == 0:\n hilm = np.append(hilm0[-1:0:-1], hilm0, axis=0)\n else:\n hilm = np.append(hilm0[::-1], hilm0, axis=0)\n else:\n y2,z2 = np.meshgrid(y, z, indexing='ij')\n hilm = calculate_linescan_ilm_psf(y2, z2, zint=zint,\n polar_angle=polar_angle, **kwargs)\n\n #2. Hdet\n if wrap:\n #Lambda function that ignores its args but still returns correct values\n func = lambda *args: get_hsym_asym(rho3*kfki, z3*kfki, zint=kfki*zint,\n get_hdet=True, **kwargs)[0]\n hdet = wrap_and_calc_psf(xpts, ypts, z, func)\n else:\n hdet, toss = get_hsym_asym(rho3*kfki, z3*kfki, zint=kfki*zint,\n get_hdet=True, **kwargs)\n\n if normalize:\n hilm /= hilm.sum()\n hdet /= hdet.sum()\n\n for a in range(x.size):\n hdet[a] *= hilm\n\n return hdet if normalize else hdet / hdet.sum()", "response": "Calculates the point spread function of a line - scanning confocal."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef calculate_polychrome_linescan_psf(x, y, z, normalize=False, kfki=0.889,\n sigkf=0.1, zint=100., nkpts=3, dist_type='gaussian', wrap=True,\n **kwargs):\n \"\"\"\n Calculates the point spread function of a line-scanning confocal with\n polydisperse dye emission.\n\n Make x,y,z __1D__ numpy.arrays, with x the direction along the\n scan line. (to make the calculation faster since I dont' need the line\n ilm for each x).\n\n Parameters\n ----------\n x : numpy.ndarray\n _One_dimensional_ array of the x grid points (along the line\n illumination) at which to evaluate the psf. In units of\n 1/k_incoming.\n y : numpy.ndarray\n _One_dimensional_ array of the y grid points (in plane,\n perpendicular to the line illumination) at which to evaluate\n the psf. In units of 1/k_incoming.\n z : numpy.ndarray\n _One_dimensional_ array of the z grid points (along the\n optical axis) at which to evaluate the psf. In units of\n 1/k_incoming.\n normalize : Bool, optional\n Set to True to include the effects of PSF normalization on\n the image intensity. Default is False.\n kfki : Float, optional\n The mean of the ratio of the final light's wavevector to the\n incoming. Default is 0.889\n sigkf : Float, optional\n The standard deviation of the ratio of the final light's\n wavevector to the incoming. Default is 0.1\n zint : Float, optional\n The position of the optical interface, in units of 1/k_incoming\n Default is 100.\n dist_type : {`gaussian`, `gamma`}, optional\n The distribution of the outgoing light. If 'gaussian' the\n resulting k-values are taken in absolute value. Default\n is `gaussian`\n wrap : Bool, optional\n If True, wraps the psf calculation for speed, assuming that\n the input x, y are regularly-spaced points. If x,y are not\n regularly spaced then `wrap` must be set to False. Default is True.\n\n Other Parameters\n ----------------\n polar_angle : Float, optional\n The polarization angle of the light (radians) with respect to\n the line direction (x). Default is 0.\n alpha : Float\n The opening angle of the lens. Default is 1.\n n2n1 : Float\n The ratio of the index in the 2nd medium to that in the first.\n Default is 0.95\n\n Returns\n -------\n numpy.ndarray\n A 3D- numpy.array of the point-spread function. Indexing is\n psf[x,y,z]; shape is [x.size, y,size, z.size]\n Notes\n -----\n Neither distribution type is perfect. If sigkf/k0 is big (>0.5ish)\n then part of the Gaussian is negative. To avoid issues an abs() is\n taken, but then the actual mean and variance are not what is\n supplied. Conversely, if sigkf/k0 is small (<0.0815), then the\n requisite associated Laguerre quadrature becomes unstable. To\n prevent this sigkf/k0 is effectively clipped to be > 0.0815.\n \"\"\"\n kfkipts, wts = get_polydisp_pts_wts(kfki, sigkf, dist_type=dist_type,\n nkpts=nkpts)\n\n #0. Set up vecs\n if wrap:\n xpts = vec_to_halfvec(x)\n ypts = vec_to_halfvec(y)\n x3, y3, z3 = np.meshgrid(xpts, ypts, z, indexing='ij')\n else:\n x3,y3,z3 = np.meshgrid(x, y, z, indexing='ij')\n rho3 = np.sqrt(x3*x3 + y3*y3)\n\n #1. Hilm\n if wrap:\n y2,z2 = np.meshgrid(ypts, z, indexing='ij')\n hilm0 = calculate_linescan_ilm_psf(y2, z2, zint=zint, **kwargs)\n if ypts[0] == 0:\n hilm = np.append(hilm0[-1:0:-1], hilm0, axis=0)\n else:\n hilm = np.append(hilm0[::-1], hilm0, axis=0)\n else:\n y2,z2 = np.meshgrid(y, z, indexing='ij')\n hilm = calculate_linescan_ilm_psf(y2, z2, zint=zint, **kwargs)\n\n #2. Hdet\n if wrap:\n #Lambda function that ignores its args but still returns correct values\n func = lambda x,y,z, kfki=1.: get_hsym_asym(rho3*kfki, z3*kfki,\n zint=kfki*zint, get_hdet=True, **kwargs)[0]\n hdet_func = lambda kfki: wrap_and_calc_psf(xpts,ypts,z, func, kfki=kfki)\n else:\n hdet_func = lambda kfki: get_hsym_asym(rho3*kfki, z3*kfki,\n zint=kfki*zint, get_hdet=True, **kwargs)[0]\n #####\n inner = [wts[a] * hdet_func(kfkipts[a]) for a in range(nkpts)]\n hdet = np.sum(inner, axis=0)\n\n if normalize:\n hilm /= hilm.sum()\n hdet /= hdet.sum()\n for a in range(x.size):\n hdet[a] *= hilm\n\n return hdet if normalize else hdet / hdet.sum()", "response": "Calculates the point spread function of a line - scanning confocal with a poly - chrome line - scanning confocal with a single line - scanning system."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrap a point - spread function in x and y and z and return a numpy array of the wrapped and calculated psf.", "response": "def wrap_and_calc_psf(xpts, ypts, zpts, func, **kwargs):\n \"\"\"\n Wraps a point-spread function in x and y.\n\n Speeds up psf calculations by a factor of 4 for free / some broadcasting\n by exploiting the x->-x, y->-y symmetry of a psf function. Pass x and y\n as the positive (say) values of the coordinates at which to evaluate func,\n and it will return the function sampled at [x[::-1]] + x. Note it is not\n wrapped in z.\n\n Parameters\n ----------\n xpts : numpy.ndarray\n 1D N-element numpy.array of the x-points to evaluate func at.\n ypts : numpy.ndarray\n y-points to evaluate func at.\n zpts : numpy.ndarray\n z-points to evaluate func at.\n func : function\n The function to evaluate and wrap around. Syntax must be\n func(x,y,z, **kwargs)\n **kwargs : Any parameters passed to the function.\n\n Outputs\n -------\n to_return : numpy.ndarray\n The wrapped and calculated psf, of shape\n [2*x.size - x0, 2*y.size - y0, z.size], where x0=1 if x[0]=0, etc.\n\n Notes\n -----\n The coordinates should be something like numpy.arange(start, stop, diff),\n with start near 0. If x[0]==0, all of x is calcualted but only x[1:]\n is wrapped (i.e. it works whether or not x[0]=0).\n\n This doesn't work directly for a linescan psf because the illumination\n portion is not like a grid. However, the illumination and detection\n are already combined with wrap_and_calc in calculate_linescan_psf etc.\n \"\"\"\n #1. Checking that everything is hunky-dory:\n for t in [xpts,ypts,zpts]:\n if len(t.shape) != 1:\n raise ValueError('xpts,ypts,zpts must be 1D.')\n\n dx = 1 if xpts[0]==0 else 0\n dy = 1 if ypts[0]==0 else 0\n\n xg,yg,zg = np.meshgrid(xpts,ypts,zpts, indexing='ij')\n xs, ys, zs = [ pts.size for pts in [xpts,ypts,zpts] ]\n to_return = np.zeros([2*xs-dx, 2*ys-dy, zs])\n\n #2. Calculate:\n up_corner_psf = func(xg,yg,zg, **kwargs)\n\n to_return[xs-dx:,ys-dy:,:] = up_corner_psf.copy() #x>0, y>0\n if dx == 0:\n to_return[:xs-dx,ys-dy:,:] = up_corner_psf[::-1,:,:].copy() #x<0, y>0\n else:\n to_return[:xs-dx,ys-dy:,:] = up_corner_psf[-1:0:-1,:,:].copy() #x<0, y>0\n if dy == 0:\n to_return[xs-dx:,:ys-dy,:] = up_corner_psf[:,::-1,:].copy() #x>0, y<0\n else:\n to_return[xs-dx:,:ys-dy,:] = up_corner_psf[:,-1:0:-1,:].copy() #x>0, y<0\n if (dx == 0) and (dy == 0):\n to_return[:xs-dx,:ys-dy,:] = up_corner_psf[::-1,::-1,:].copy() #x<0,y<0\n elif (dx == 0) and (dy != 0):\n to_return[:xs-dx,:ys-dy,:] = up_corner_psf[::-1,-1:0:-1,:].copy() #x<0,y<0\n elif (dy == 0) and (dx != 0):\n to_return[:xs-dx,:ys-dy,:] = up_corner_psf[-1:0:-1,::-1,:].copy() #x<0,y<0\n else: #dx==1 and dy==1\n to_return[:xs-dx,:ys-dy,:] = up_corner_psf[-1:0:-1,-1:0:-1,:].copy()#x<0,y<0\n\n return to_return"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntransforms a vector np. arange - N M dx to np. arange - N M dx", "response": "def vec_to_halfvec(vec):\n \"\"\"Transforms a vector np.arange(-N, M, dx) to np.arange(min(|vec|), max(N,M),dx)]\"\"\"\n d = vec[1:] - vec[:-1]\n if ((d/d.mean()).std() > 1e-14) or (d.mean() < 0):\n raise ValueError('vec must be np.arange() in increasing order')\n dx = d.mean()\n lowest = np.abs(vec).min()\n highest = np.abs(vec).max()\n return np.arange(lowest, highest + 0.1*dx, dx).astype(vec.dtype)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a scalar a to a list and all iterables to list as well.", "response": "def listify(a):\n \"\"\"\n Convert a scalar ``a`` to a list and all iterables to list as well.\n\n Examples\n --------\n >>> listify(0)\n [0]\n\n >>> listify([1,2,3])\n [1, 2, 3]\n\n >>> listify('a')\n ['a']\n\n >>> listify(np.array([1,2,3]))\n [1, 2, 3]\n\n >>> listify('string')\n ['string']\n \"\"\"\n if a is None:\n return []\n elif not isinstance(a, (tuple, list, np.ndarray)):\n return [a]\n return list(a)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of objects that are not in a single element list.", "response": "def delistify(a, b=None):\n \"\"\"\n If a single element list, extract the element as an object, otherwise\n leave as it is.\n\n Examples\n --------\n >>> delistify('string')\n 'string'\n\n >>> delistify(['string'])\n 'string'\n\n >>> delistify(['string', 'other'])\n ['string', 'other']\n\n >>> delistify(np.array([1.0]))\n 1.0\n\n >>> delistify([1, 2, 3])\n [1, 2, 3]\n \"\"\"\n if isinstance(b, (tuple, list, np.ndarray)):\n if isinstance(a, (tuple, list, np.ndarray)):\n return type(b)(a)\n return type(b)([a])\n else:\n if isinstance(a, (tuple, list, np.ndarray)) and len(a) == 1:\n return a[0]\n return a\n return a"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef aN(a, dim=3, dtype='int'):\n if not hasattr(a, '__iter__'):\n return np.array([a]*dim, dtype=dtype)\n return np.array(a).astype(dtype)", "response": "Convert an integer or iterable list to numpy array of length dim."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cdd(d, k):\n if not isinstance(k, list):\n k = [k]\n for i in k:\n if i in d:\n d.pop(i)", "response": "Conditionally delete key or list of keys from dict d"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef patch_docs(subclass, superclass):\n funcs0 = inspect.getmembers(subclass, predicate=inspect.ismethod)\n funcs1 = inspect.getmembers(superclass, predicate=inspect.ismethod)\n\n funcs1 = [f[0] for f in funcs1]\n\n for name, func in funcs0:\n if name.startswith('_'):\n continue\n\n if name not in funcs1:\n continue\n\n if func.__doc__ is None:\n func = getattr(subclass, name)\n func.__func__.__doc__ = getattr(superclass, name).__func__.__doc__", "response": "Apply the documentation from subclass to superclass by filling the documentation from all overridden member function docstrings with those from the\n parent class\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef indir(path):\n cwd = os.getcwd()\n\n try:\n os.chdir(path)\n yield\n except Exception as e:\n raise\n finally:\n os.chdir(cwd)", "response": "Context manager for switching the current path of the process."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\narrays slicer object for this tile", "response": "def slicer(self):\n \"\"\"\n Array slicer object for this tile\n\n >>> Tile((2,3)).slicer\n (slice(0, 2, None), slice(0, 3, None))\n\n >>> np.arange(10)[Tile((4,)).slicer]\n array([0, 1, 2, 3])\n \"\"\"\n return tuple(np.s_[l:r] for l,r in zip(*self.bounds))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef oslicer(self, tile):\n mask = None\n vecs = tile.coords(form='meshed')\n for v in vecs:\n v[self.slicer] = -1\n mask = mask & (v > 0) if mask is not None else (v>0)\n return tuple(np.array(i).astype('int') for i in zip(*[v[mask] for v in vecs]))", "response": "Opposite slicer the outer part wrt to a field"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef kcenter(self):\n return np.array([\n np.abs(np.fft.fftshift(np.fft.fftfreq(q))).argmin()\n for q in self.shape\n ]).astype('float')", "response": "Return the frequency center of the tile"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef corners(self):\n corners = []\n for ind in itertools.product(*((0,1),)*self.dim):\n ind = np.array(ind)\n corners.append(self.l + ind*self.r)\n return np.array(corners)", "response": "Iterate the vector of all corners of the hyperrectangles"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nformats a 3d vector field in certain ways see coords for a description of each formatting method.", "response": "def _format_vector(self, vecs, form='broadcast'):\n \"\"\"\n Format a 3d vector field in certain ways, see `coords` for a description\n of each formatting method.\n \"\"\"\n if form == 'meshed':\n return np.meshgrid(*vecs, indexing='ij')\n elif form == 'vector':\n vecs = np.meshgrid(*vecs, indexing='ij')\n return np.rollaxis(np.array(np.broadcast_arrays(*vecs)),0,self.dim+1)\n elif form == 'flat':\n return vecs\n else:\n return [v[self._coord_slicers[i]] for i,v in enumerate(vecs)]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef coords(self, norm=False, form='broadcast'):\n if norm is False:\n norm = 1\n if norm is True:\n norm = np.array(self.shape)\n norm = aN(norm, self.dim, dtype='float')\n\n v = list(np.arange(self.l[i], self.r[i]) / norm[i] for i in range(self.dim))\n return self._format_vector(v, form=form)", "response": "Returns the coordinates of the tile."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef kvectors(self, norm=False, form='broadcast', real=False, shift=False):\n if norm is False:\n norm = 1\n if norm is True:\n norm = np.array(self.shape)\n norm = aN(norm, self.dim, dtype='float')\n\n v = list(np.fft.fftfreq(self.shape[i])/norm[i] for i in range(self.dim))\n\n if shift:\n v = list(np.fft.fftshift(t) for t in v)\n\n if real:\n v[-1] = v[-1][:(self.shape[-1]+1)//2]\n\n return self._format_vector(v, form=form)", "response": "Return the kvectors associated with this tile."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef contains(self, items, pad=0):\n o = ((items >= self.l-pad) & (items < self.r+pad))\n if len(o.shape) == 2:\n o = o.all(axis=-1)\n elif len(o.shape) == 1:\n o = o.all()\n return o", "response": "Test whether the given items are contained within this tile."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntranslating a tile by an amount dr", "response": "def translate(self, dr):\n \"\"\"\n Translate a tile by an amount dr\n\n >>> Tile(5).translate(1)\n Tile [1, 1, 1] -> [6, 6, 6] ([5, 5, 5])\n \"\"\"\n tile = self.copy()\n tile.l += dr\n tile.r += dr\n return tile"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npadding this tile by an equal amount on each side as specified by pad", "response": "def pad(self, pad):\n \"\"\"\n Pad this tile by an equal amount on each side as specified by pad\n\n >>> Tile(10).pad(2)\n Tile [-2, -2, -2] -> [12, 12, 12] ([14, 14, 14])\n\n >>> Tile(10).pad([1,2,3])\n Tile [-1, -2, -3] -> [11, 12, 13] ([12, 14, 16])\n \"\"\"\n tile = self.copy()\n tile.l -= pad\n tile.r += pad\n return tile"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef overhang(self, tile):\n ll = np.abs(amin(self.l - tile.l, aN(0, dim=self.dim)))\n rr = np.abs(amax(self.r - tile.r, aN(0, dim=self.dim)))\n return ll, rr", "response": "Get the left and right absolute overflow of a given tile"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reflect_overhang(self, clip):\n orig = self.copy()\n tile = self.copy()\n\n hangl, hangr = tile.overhang(clip)\n tile = tile.pad(hangl)\n tile = tile.pad(hangr)\n\n inner = Tile.intersection([clip, orig])\n outer = Tile.intersection([clip, tile])\n return inner, outer", "response": "Compute the overhang and reflect it internally so respect periodic\n padding rules. Returns both the inner tile and the outer tile with necessary pad."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a filtered image after applying the Fourier - space filters", "response": "def filtered_image(self, im):\n \"\"\"Returns a filtered image after applying the Fourier-space filters\"\"\"\n q = np.fft.fftn(im)\n for k,v in self.filters:\n q[k] -= v\n return np.real(np.fft.ifftn(q))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_filter(self, slices, values):\n self.filters = [[sl,values[sl]] for sl in slices]", "response": "Sets the Fourier - space filters for the image."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the image from the file and perform any transforms to get a loaded image", "response": "def load_image(self):\n \"\"\" Read the file and perform any transforms to get a loaded image \"\"\"\n try:\n image = initializers.load_tiff(self.filename)\n image = initializers.normalize(\n image, invert=self.invert, scale=self.exposure,\n dtype=self.float_precision\n )\n except IOError as e:\n log.error(\"Could not find image '%s'\" % self.filename)\n raise e\n\n return image"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the minimum and maximum of the image in this object.", "response": "def get_scale(self):\n \"\"\"\n If exposure was not set in the __init__, get the exposure associated\n with this RawImage so that it may be used in other\n :class:`~peri.util.RawImage`. This is useful for transferring exposure\n parameters to a series of images.\n\n Returns\n -------\n exposure : tuple of floats\n The (emin, emax) which get mapped to (0, 1)\n \"\"\"\n if self.exposure is not None:\n return self.exposure\n\n raw = initializers.load_tiff(self.filename)\n return raw.min(), raw.max()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_scale_from_raw(raw, scaled):\n t0, t1 = scaled.min(), scaled.max()\n r0, r1 = float(raw.min()), float(raw.max())\n\n rmin = (t1*r0 - t0*r1) / (t1 - t0)\n rmax = (r1 - r0) / (t1 - t0) + rmin\n return (rmin, rmax)", "response": "This function takes a raw image and a scaled version of the same image and returns the exposure parameters associated with those images."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update(self, value=0):\n self._deltas.append(time.time())\n\n self.value = value\n self._percent = 100.0 * self.value / self.num\n\n if self.bar:\n self._bars = self._bar_symbol*int(np.round(self._percent / 100. * self._barsize))\n\n if (len(self._deltas) < 2) or (self._deltas[-1] - self._deltas[-2]) > 1e-1:\n self._estimate_time()\n self._draw()\n\n if self.value == self.num:\n self.end()", "response": "Update the value of the progress and update the progress bar."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks that the modelstr and varmap are consistent across the many components.", "response": "def check_consistency(self):\n \"\"\"\n Make sure that the required comps are included in the list of\n components supplied by the user. Also check that the parameters are\n consistent across the many components.\n \"\"\"\n error = False\n regex = re.compile('([a-zA-Z_][a-zA-Z0-9_]*)')\n\n # there at least must be the full model, not necessarily partial updates\n if 'full' not in self.modelstr:\n raise ModelError(\n 'Model must contain a `full` key describing '\n 'the entire image formation'\n )\n\n # Check that the two model descriptors are consistent\n for name, eq in iteritems(self.modelstr):\n var = regex.findall(eq)\n for v in var:\n # remove the derivative signs if there (dP -> P)\n v = re.sub(r\"^d\", '', v)\n if v not in self.varmap:\n log.error(\n \"Variable '%s' (eq. '%s': '%s') not found in category map %r\" %\n (v, name, eq, self.varmap)\n )\n error = True\n\n if error:\n raise ModelError('Inconsistent varmap and modelstr descriptions')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_inputs(self, comps):\n error = False\n compcats = [c.category for c in comps]\n\n # Check that the components are all provided, given the categories\n for k, v in iteritems(self.varmap):\n if k not in self.modelstr['full']:\n log.warn('Component (%s : %s) not used in model.' % (k,v))\n\n if v not in compcats:\n log.error('Map component (%s : %s) not found in list of components.' % (k,v))\n error = True\n\n if error:\n raise ModelError('Component list incomplete or incorrect')", "response": "Check that the list of components comp is compatible with both the varmap and modelstr for this Model\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the model corresponding to a variation wrt a category. For example is H + B.", "response": "def get_difference_model(self, category):\n \"\"\"\n Get the equation corresponding to a variation wrt category. For example\n if::\n\n modelstr = {\n 'full' :'H(I) + B',\n 'dH' : 'dH(I)',\n 'dI' : 'H(dI)',\n 'dB' : 'dB'\n }\n\n varmap = {'H': 'psf', 'I': 'obj', 'B': 'bkg'}\n\n then ``get_difference_model('obj') == modelstr['dI'] == 'H(dI)'``\n \"\"\"\n name = self.diffname(self.ivarmap[category])\n return self.modelstr.get(name)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmapping component function funcname result into model variables dictionary for use in eval of the model.", "response": "def map_vars(self, comps, funcname='get', diffmap=None, **kwargs):\n \"\"\"\n Map component function ``funcname`` result into model variables\n dictionary for use in eval of the model. If ``diffmap`` is provided then\n that symbol is translated into 'd'+diffmap.key and is replaced by\n diffmap.value. ``**kwargs` are passed to the ``comp.funcname(**kwargs)``.\n \"\"\"\n out = {}\n diffmap = diffmap or {}\n\n for c in comps:\n cat = c.category\n\n if cat in diffmap:\n symbol = self.diffname(self.ivarmap[cat])\n out[symbol] = diffmap[cat]\n else:\n symbol = self.ivarmap[cat]\n out[symbol] = getattr(c, funcname)(**kwargs)\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef evaluate(self, comps, funcname='get', diffmap=None, **kwargs):\n evar = self.map_vars(comps, funcname, diffmap=diffmap)\n\n if diffmap is None:\n return eval(self.get_base_model(), evar)\n else:\n compname = list(diffmap.keys())[0]\n return eval(self.get_difference_model(compname), evar)", "response": "Evaluate the model of a set of components and return the output of a model."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lbl(axis, label, size=22):\n at = AnchoredText(label, loc=2, prop=dict(size=size), frameon=True)\n at.patch.set_boxstyle(\"round,pad=0.,rounding_size=0.0\")\n #bb = axis.get_yaxis_transform()\n #at = AnchoredText(label,\n # loc=3, prop=dict(size=18), frameon=True,\n # bbox_to_anchor=(-0.5,1),#(-.255, 0.90),\n # bbox_transform=bb,#axis.transAxes\n # )\n axis.add_artist(at)", "response": "Put a figure label in an axis"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generative_model(s,x,y,z,r, factor=1.1):\n pl.close('all')\n\n slicez = int(round(z.mean()))\n slicex = s.image.shape[2]//2\n slicer1 = np.s_[slicez,s.pad:-s.pad,s.pad:-s.pad]\n slicer2 = np.s_[s.pad:-s.pad,s.pad:-s.pad,slicex]\n center = (slicez, s.image.shape[1]//2, slicex)\n\n fig = pl.figure(figsize=(factor*13,factor*10))\n\n #=========================================================================\n #=========================================================================\n gs1 = ImageGrid(fig, rect=[0.0, 0.6, 1.0, 0.35], nrows_ncols=(1,3),\n axes_pad=0.1)\n ax_real = gs1[0]\n ax_fake = gs1[1]\n ax_diff = gs1[2]\n\n diff = s.get_model_image() - s.image\n ax_real.imshow(s.image[slicer1], cmap=pl.cm.bone_r)\n ax_real.set_xticks([])\n ax_real.set_yticks([])\n ax_real.set_title(\"Confocal image\", fontsize=24)\n ax_fake.imshow(s.get_model_image()[slicer1], cmap=pl.cm.bone_r)\n ax_fake.set_xticks([])\n ax_fake.set_yticks([])\n ax_fake.set_title(\"Model image\", fontsize=24)\n ax_diff.imshow(diff[slicer1], cmap=pl.cm.RdBu, vmin=-0.1, vmax=0.1)\n ax_diff.set_xticks([])\n ax_diff.set_yticks([])\n ax_diff.set_title(\"Difference\", fontsize=24)\n\n #=========================================================================\n #=========================================================================\n gs2 = ImageGrid(fig, rect=[0.1, 0.0, 0.4, 0.55], nrows_ncols=(3,2),\n axes_pad=0.1)\n ax_plt1 = fig.add_subplot(gs2[0])\n ax_plt2 = fig.add_subplot(gs2[1])\n ax_ilm1 = fig.add_subplot(gs2[2])\n ax_ilm2 = fig.add_subplot(gs2[3])\n ax_psf1 = fig.add_subplot(gs2[4])\n ax_psf2 = fig.add_subplot(gs2[5])\n\n c = int(z.mean()), int(y.mean())+s.pad, int(x.mean())+s.pad\n if s.image.shape[0] > 2*s.image.shape[1]//3:\n w = s.image.shape[2] - 2*s.pad\n h = 2*w//3\n else:\n h = s.image.shape[0] - 2*s.pad\n w = 3*h//2\n\n w,h = w//2, h//2\n xyslice = np.s_[slicez, c[1]-h:c[1]+h, c[2]-w:c[2]+w]\n yzslice = np.s_[c[0]-h:c[0]+h, c[1]-w:c[1]+w, slicex]\n\n #h = s.image.shape[2]/2 - s.image.shape[0]/2\n #slicer2 = np.s_[s.pad:-s.pad, s.pad:-s.pad, slicex]\n #slicer3 = np.s_[slicez, s.pad+h:-s.pad-h, s.pad:-s.pad]\n\n ax_plt1.imshow(1-s.obj.get_field()[xyslice], cmap=pl.cm.bone_r, vmin=0, vmax=1)\n ax_plt1.set_xticks([])\n ax_plt1.set_yticks([])\n ax_plt1.set_ylabel(\"Platonic\", fontsize=22)\n ax_plt1.set_title(\"x-y\", fontsize=24)\n ax_plt2.imshow(1-s._platonic_image()[yzslice], cmap=pl.cm.bone_r, vmin=0, vmax=1)\n ax_plt2.set_xticks([])\n ax_plt2.set_yticks([])\n ax_plt2.set_title(\"y-z\", fontsize=24)\n\n ax_ilm1.imshow(s.ilm.get_field()[xyslice], cmap=pl.cm.bone_r)\n ax_ilm1.set_xticks([])\n ax_ilm1.set_yticks([])\n ax_ilm1.set_ylabel(\"ILM\", fontsize=22)\n ax_ilm2.imshow(s.ilm.get_field()[yzslice], cmap=pl.cm.bone_r)\n ax_ilm2.set_xticks([])\n ax_ilm2.set_yticks([])\n\n t = s.ilm.get_field().copy()\n t *= 0\n t[c] = 1\n s.psf.set_tile(util.Tile(t.shape))\n psf = (s.psf.execute(t)+5e-5)**0.1\n\n ax_psf1.imshow(psf[xyslice], cmap=pl.cm.bone)\n ax_psf1.set_xticks([])\n ax_psf1.set_yticks([])\n ax_psf1.set_ylabel(\"PSF\", fontsize=22)\n ax_psf2.imshow(psf[yzslice], cmap=pl.cm.bone)\n ax_psf2.set_xticks([])\n ax_psf2.set_yticks([])\n\n #=========================================================================\n #=========================================================================\n ax_zoom = fig.add_axes([0.48, 0.018, 0.45, 0.52])\n\n #s.model_to_true_image()\n im = s.image[slicer1]\n sh = np.array(im.shape)\n cx = x.mean()\n cy = y.mean()\n\n extent = [0,sh[0],0,sh[1]]\n ax_zoom.set_xticks([])\n ax_zoom.set_yticks([])\n ax_zoom.imshow(im, extent=extent, cmap=pl.cm.bone_r)\n ax_zoom.set_xlim(cx-12, cx+12)\n ax_zoom.set_ylim(cy-12, cy+12)\n ax_zoom.set_title(\"Sampled positions\", fontsize=24)\n ax_zoom.hexbin(x,y, gridsize=32, mincnt=0, cmap=pl.cm.hot)\n\n zoom1 = zoomed_inset_axes(ax_zoom, 30, loc=3)\n zoom1.imshow(im, extent=extent, cmap=pl.cm.bone_r)\n zoom1.set_xlim(cx-1.0/6, cx+1.0/6)\n zoom1.set_ylim(cy-1.0/6, cy+1.0/6)\n zoom1.hexbin(x,y,gridsize=32, mincnt=5, cmap=pl.cm.hot)\n zoom1.set_xticks([])\n zoom1.set_yticks([])\n zoom1.hlines(cy-1.0/6 + 1.0/32, cx-1.0/6+5e-2, cx-1.0/6+5e-2+1e-1, lw=3)\n zoom1.text(cx-1.0/6 + 1.0/24, cy-1.0/6+5e-2, '0.1px')\n mark_inset(ax_zoom, zoom1, loc1=2, loc2=4, fc=\"none\", ec=\"0.0\")", "response": "Generate a model for the state s."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef examine_unexplained_noise(state, bins=1000, xlim=(-10,10)):\n r = state.residuals\n q = np.fft.fftn(r)\n #Get the expected values of `sigma`:\n calc_sig = lambda x: np.sqrt(np.dot(x,x) / x.size)\n rh, xr = np.histogram(r.ravel() / calc_sig(r.ravel()), bins=bins,\n density=True)\n bigq = np.append(q.real.ravel(), q.imag.ravel())\n qh, xq = np.histogram(bigq / calc_sig(q.real.ravel()), bins=bins,\n density=True)\n xr = 0.5*(xr[1:] + xr[:-1])\n xq = 0.5*(xq[1:] + xq[:-1])\n\n gauss = lambda t : np.exp(-t*t*0.5) / np.sqrt(2*np.pi)\n\n plt.figure(figsize=[16,8])\n axes = []\n for a, (x, r, lbl) in enumerate([[xr, rh, 'Real'], [xq, qh, 'Fourier']]):\n ax = plt.subplot(1,2,a+1)\n ax.semilogy(x, r, label='Data')\n ax.plot(x, gauss(x), label='Gauss Fit', scalex=False, scaley=False)\n ax.set_xlabel('Residuals value $r/\\sigma$')\n ax.set_ylabel('Probability $P(r/\\sigma)$')\n ax.legend(loc='upper right')\n ax.set_title('{}-Space'.format(lbl))\n ax.set_xlim(xlim)\n axes.append(ax)\n return axes", "response": "Examines a state s residuals in real and Fourier space with a Gaussian."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compare_data_model_residuals(s, tile, data_vmin='calc', data_vmax='calc',\n res_vmin=-0.1, res_vmax=0.1, edgepts='calc', do_imshow=True,\n data_cmap=plt.cm.bone, res_cmap=plt.cm.RdBu):\n \"\"\"\n Compare the data, model, and residuals of a state.\n\n Makes an image of any 2D slice of a state that compares the data,\n model, and residuals. The upper left portion of the image is the raw\n data, the central portion the model, and the lower right portion the\n image. Either plots the image using plt.imshow() or returns a\n np.ndarray of the image pixels for later use.\n\n Parameters\n ----------\n st : peri.ImageState object\n The state to plot.\n tile : peri.util.Tile object\n The slice of the image to plot. Can be any xy, xz, or yz\n projection, but it must return a valid 2D slice (the slice is\n squeezed internally).\n\n data_vmin : {Float, `calc`}, optional\n vmin for the imshow for the data and generative model (shared).\n Default is 'calc' = 0.5(data.min() + model.min())\n data_vmax : {Float, `calc`}, optional\n vmax for the imshow for the data and generative model (shared).\n Default is 'calc' = 0.5(data.max() + model.max())\n res_vmin : Float, optional\n vmin for the imshow for the residuals. Default is -0.1\n Default is 'calc' = 0.5(data.min() + model.min())\n res_vmax : Float, optional\n vmax for the imshow for the residuals. Default is +0.1\n edgepts : {Nested list-like, Float, 'calc'}, optional.\n The vertices of the triangles which determine the splitting of\n the image. The vertices are at (image corner, (edge, y), and\n (x,edge), where edge is the appropriate edge of the image.\n edgepts[0] : (x,y) points for the upper edge\n edgepts[1] : (x,y) points for the lower edge\n where `x` is the coordinate along the image's 0th axis and `y`\n along the images 1st axis. Default is 'calc,' which calculates\n edge points by splitting the image into 3 regions of equal\n area. If edgepts is a float scalar, calculates the edge points\n based on a constant fraction of distance from the edge.\n do_imshow : Bool\n If True, imshow's and returns the returned handle.\n If False, returns the array as a [M,N,4] array.\n data_cmap : matplotlib colormap instance\n The colormap to use for the data and model.\n res_cmap : matplotlib colormap instance\n The colormap to use for the residuals.\n\n Returns\n -------\n image : {matplotlib.pyplot.AxesImage, numpy.ndarray}\n If `do_imshow` == True, the returned handle from imshow.\n If `do_imshow` == False, an [M,N,4] np.ndarray of the image\n pixels.\n \"\"\"\n # This could be modified to alpha the borderline... or to embiggen\n # the image and slice it more finely\n residuals = s.residuals[tile.slicer].squeeze()\n data = s.data[tile.slicer].squeeze()\n model = s.model[tile.slicer].squeeze()\n if data.ndim != 2:\n raise ValueError('tile does not give a 2D slice')\n\n im = np.zeros([data.shape[0], data.shape[1], 4])\n if data_vmin == 'calc':\n data_vmin = 0.5*(data.min() + model.min())\n if data_vmax == 'calc':\n data_vmax = 0.5*(data.max() + model.max())\n\n #1. Get masks:\n upper_mask, center_mask, lower_mask = trisect_image(im.shape, edgepts)\n\n #2. Get colorbar'd images\n gm = data_cmap(center_data(model, data_vmin, data_vmax))\n dt = data_cmap(center_data(data, data_vmin, data_vmax))\n rs = res_cmap(center_data(residuals, res_vmin, res_vmax))\n\n for a in range(4):\n im[:,:,a][upper_mask] = rs[:,:,a][upper_mask]\n im[:,:,a][center_mask] = gm[:,:,a][center_mask]\n im[:,:,a][lower_mask] = dt[:,:,a][lower_mask]\n if do_imshow:\n return plt.imshow(im)\n else:\n return im", "response": "Compare the data model and residuals of a state."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntrisects an image into 3 triangular portions.", "response": "def trisect_image(imshape, edgepts='calc'):\n \"\"\"\n Returns 3 masks that trisect an image into 3 triangular portions.\n\n Parameters\n ----------\n imshape : 2-element list-like of ints\n The shape of the image. Elements after the first 2 are ignored.\n\n edgepts : Nested list-like, float, or `calc`, optional.\n The vertices of the triangles which determine the splitting of\n the image. The vertices are at (image corner, (edge, y), and\n (x,edge), where edge is the appropriate edge of the image.\n edgepts[0] : (x,y) points for the upper edge\n edgepts[1] : (x,y) points for the lower edge\n where `x` is the coordinate along the image's 0th axis and `y`\n along the images 1st axis. Default is 'calc,' which calculates\n edge points by splitting the image into 3 regions of equal\n area. If edgepts is a float scalar, calculates the edge points\n based on a constant fraction of distance from the edge.\n\n Returns\n -------\n upper_mask : numpy.ndarray\n Boolean array; True in the image's upper region.\n center_mask : numpy.ndarray\n Boolean array; True in the image's center region.\n lower_mask : numpy.ndarray\n Boolean array; True in the image's lower region.\n \"\"\"\n im_x, im_y = np.meshgrid(np.arange(imshape[0]), np.arange(imshape[1]),\n indexing='ij')\n if np.size(edgepts) == 1:\n #Gets equal-area sections, at sqrt(2/3) of the sides\n f = np.sqrt(2./3.) if edgepts == 'calc' else edgepts\n # f = np.sqrt(2./3.)\n lower_edge = (imshape[0] * (1-f), imshape[1] * f)\n upper_edge = (imshape[0] * f, imshape[1] * (1-f))\n else:\n upper_edge, lower_edge = edgepts\n\n #1. Get masks\n lower_slope = lower_edge[1] / max(float(imshape[0] - lower_edge[0]), 1e-9)\n upper_slope = (imshape[1] - upper_edge[1]) / float(upper_edge[0])\n #and the edge points are the x or y intercepts\n lower_intercept = -lower_slope * lower_edge[0]\n upper_intercept = upper_edge[1]\n lower_mask = im_y < (im_x * lower_slope + lower_intercept)\n upper_mask = im_y > (im_x * upper_slope + upper_intercept)\n\n center_mask= -(lower_mask | upper_mask)\n return upper_mask, center_mask, lower_mask"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef center_data(data, vmin, vmax):\n ans = data - vmin\n ans /= (vmax - vmin)\n return np.clip(ans, 0, 1)", "response": "Clips data on vmin vmax then rescales to 1"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef crb_compare(state0, samples0, state1, samples1, crb0=None, crb1=None,\n zlayer=None, xlayer=None):\n \"\"\"\n To run, do:\n\n s,h = pickle...\n s1,h1 = pickle...\n i.e. /media/scratch/bamf/vacancy/vacancy_zoom-1.tif_t002.tif-featured-v2.pkl\n i.e. /media/scratch/bamf/frozen-particles/0.tif-featured-full.pkl\n crb0 = diag_crb_particles(s); crb1 = diag_crb_particles(s1)\n crb_compare(s,h[-25:],s1,h1[-25:], crb0, crb1)\n \"\"\"\n s0 = state0\n s1 = state1\n h0 = np.array(samples0)\n h1 = np.array(samples1)\n\n slicez = zlayer or s0.image.shape[0]//2\n slicex = xlayer or s0.image.shape[2]//2\n slicer1 = np.s_[slicez,s0.pad:-s0.pad,s0.pad:-s0.pad]\n slicer2 = np.s_[s0.pad:-s0.pad,s0.pad:-s0.pad,slicex]\n center = (slicez, s0.image.shape[1]//2, slicex)\n\n mu0 = h0.mean(axis=0)\n mu1 = h1.mean(axis=0)\n\n std0 = h0.std(axis=0)\n std1 = h1.std(axis=0)\n\n mask0 = (s0.state[s0.b_typ]==1.) & (\n analyze.trim_box(s0, mu0[s0.b_pos].reshape(-1,3)))\n mask1 = (s1.state[s1.b_typ]==1.) & (\n analyze.trim_box(s1, mu1[s1.b_pos].reshape(-1,3)))\n active0 = np.arange(s0.N)[mask0]#s0.state[s0.b_typ]==1.]\n active1 = np.arange(s1.N)[mask1]#s1.state[s1.b_typ]==1.]\n\n pos0 = mu0[s0.b_pos].reshape(-1,3)[active0]\n pos1 = mu1[s1.b_pos].reshape(-1,3)[active1]\n rad0 = mu0[s0.b_rad][active0]\n rad1 = mu1[s1.b_rad][active1]\n\n link = analyze.nearest(pos0, pos1)\n dpos = pos0 - pos1[link]\n drad = rad0 - rad1[link]\n\n drift = dpos.mean(axis=0)\n log.info('drift {}'.format(drift))\n\n dpos -= drift\n\n fig = pl.figure(figsize=(24,10))\n\n #=========================================================================\n #=========================================================================\n gs0 = ImageGrid(fig, rect=[0.02, 0.4, 0.4, 0.60], nrows_ncols=(2,3), axes_pad=0.1)\n\n lbl(gs0[0], 'A')\n for i,slicer in enumerate([slicer1, slicer2]):\n ax_real = gs0[3*i+0]\n ax_fake = gs0[3*i+1]\n ax_diff = gs0[3*i+2]\n\n diff0 = s0.get_model_image() - s0.image\n diff1 = s1.get_model_image() - s1.image\n a = (s0.image - s1.image)\n b = (s0.get_model_image() - s1.get_model_image())\n c = (diff0 - diff1)\n\n ptp = 0.7*max([np.abs(a).max(), np.abs(b).max(), np.abs(c).max()])\n cmap = pl.cm.RdBu_r\n ax_real.imshow(a[slicer], cmap=cmap, vmin=-ptp, vmax=ptp)\n ax_real.set_xticks([])\n ax_real.set_yticks([])\n ax_fake.imshow(b[slicer], cmap=cmap, vmin=-ptp, vmax=ptp)\n ax_fake.set_xticks([])\n ax_fake.set_yticks([])\n ax_diff.imshow(c[slicer], cmap=cmap, vmin=-ptp, vmax=ptp)#cmap=pl.cm.RdBu, vmin=-1.0, vmax=1.0)\n ax_diff.set_xticks([])\n ax_diff.set_yticks([])\n\n if i == 0:\n ax_real.set_title(r\"$\\Delta$ Confocal image\", fontsize=24)\n ax_fake.set_title(r\"$\\Delta$ Model image\", fontsize=24)\n ax_diff.set_title(r\"$\\Delta$ Difference\", fontsize=24)\n ax_real.set_ylabel('x-y')\n else:\n ax_real.set_ylabel('x-z')\n\n #=========================================================================\n #=========================================================================\n gs1 = GridSpec(1,3, left=0.05, bottom=0.125, right=0.42, top=0.37,\n wspace=0.15, hspace=0.05)\n\n spos0 = std0[s0.b_pos].reshape(-1,3)[active0]\n spos1 = std1[s1.b_pos].reshape(-1,3)[active1]\n srad0 = std0[s0.b_rad][active0]\n srad1 = std1[s1.b_rad][active1]\n\n def hist(ax, vals, bins, *args, **kwargs):\n y,x = np.histogram(vals, bins=bins)\n x = (x[1:] + x[:-1])/2\n y /= len(vals)\n ax.plot(x,y, *args, **kwargs)\n\n def pp(ind, tarr, tsim, tcrb, var='x'):\n bins = 10**np.linspace(-3, 0.0, 30)\n bin2 = 10**np.linspace(-3, 0.0, 100)\n bins = np.linspace(0.0, 0.2, 30)\n bin2 = np.linspace(0.0, 0.2, 100)\n xlim = (0.0, 0.12)\n #xlim = (1e-3, 1e0)\n ylim = (1e-2, 30)\n\n ticks = ticker.FuncFormatter(lambda x, pos: '{:0.0f}'.format(np.log10(x)))\n scaler = lambda x: x #np.log10(x)\n\n ax_crb = pl.subplot(gs1[0,ind])\n ax_crb.hist(scaler(np.abs(tarr)), bins=bins,\n normed=True, alpha=0.7, histtype='stepfilled', lw=1)\n ax_crb.hist(scaler(np.abs(tcrb)).ravel(), bins=bin2,\n normed=True, alpha=1.0, histtype='step', ls='solid', lw=1.5, color='k')\n ax_crb.hist(scaler(np.abs(tsim).ravel()), bins=bin2,\n normed=True, alpha=1.0, histtype='step', lw=3)\n ax_crb.set_xlabel(r\"$\\Delta = |%s(t_1) - %s(t_0)|$\" % (var,var), fontsize=24)\n #ax_crb.semilogx()\n ax_crb.set_xlim(xlim)\n #ax_crb.semilogy()\n #ax_crb.set_ylim(ylim)\n #ax_crb.xaxis.set_major_formatter(ticks)\n ax_crb.grid(b=False, which='both', axis='both')\n\n if ind == 0:\n lbl(ax_crb, 'B')\n ax_crb.set_ylabel(r\"$P(\\Delta)$\")\n else:\n ax_crb.set_yticks([])\n\n ax_crb.locator_params(axis='x', nbins=3)\n\n f,g = 1.5, 1.95\n sim = f*sim_crb_diff(spos0[:,1], spos1[:,1][link])\n crb = g*sim_crb_diff(crb0[0][:,1][active0], crb1[0][:,1][active1][link])\n pp(0, dpos[:,1], sim, crb, 'x')\n\n sim = f*sim_crb_diff(spos0[:,0], spos1[:,0][link])\n crb = g*sim_crb_diff(crb0[0][:,0][active0], crb1[0][:,0][active1][link])\n pp(1, dpos[:,0], sim, crb, 'z')\n\n sim = f*sim_crb_diff(srad0, srad1[link])\n crb = g*sim_crb_diff(crb0[1][active0], crb1[1][active1][link])\n pp(2, drad, sim, crb, 'a')\n\n #ax_crb_r.locator_params(axis='both', nbins=3)\n #gs1.tight_layout(fig)\n\n #=========================================================================\n #=========================================================================\n gs2 = GridSpec(2,2, left=0.48, bottom=0.12, right=0.99, top=0.95,\n wspace=0.35, hspace=0.35)\n\n ax_hist = pl.subplot(gs2[0,0])\n ax_hist.hist(std0[s0.b_pos], bins=np.logspace(-3.0, 0, 50), alpha=0.7, label='POS', histtype='stepfilled')\n ax_hist.hist(std0[s0.b_rad], bins=np.logspace(-3.0, 0, 50), alpha=0.7, label='RAD', histtype='stepfilled')\n ax_hist.set_xlim((10**-3.0, 1))\n ax_hist.semilogx()\n ax_hist.set_xlabel(r\"$\\bar{\\sigma}$\")\n ax_hist.set_ylabel(r\"$P(\\bar{\\sigma})$\")\n ax_hist.legend(loc='upper right')\n lbl(ax_hist, 'C')\n\n imdiff = ((s0.get_model_image() - s0.image)/s0._sigma_field)[s0.image_mask==1.].ravel()\n mu = imdiff.mean()\n #sig = imdiff.std()\n #print mu, sig\n x = np.linspace(-5,5,10000)\n\n ax_diff = pl.subplot(gs2[0,1])\n ax_diff.plot(x, 1.0/np.sqrt(2*np.pi) * np.exp(-(x-mu)**2 / 2), '-', alpha=0.7, color='k', lw=2)\n ax_diff.hist(imdiff, bins=1000, histtype='step', alpha=0.7, normed=True)\n ax_diff.semilogy()\n ax_diff.set_ylabel(r\"$P(\\delta)$\")\n ax_diff.set_xlabel(r\"$\\delta = (M_i - d_i)/\\sigma_i$\")\n ax_diff.locator_params(axis='x', nbins=5)\n ax_diff.grid(b=False, which='minor', axis='y')\n ax_diff.set_xlim(-5, 5)\n ax_diff.set_ylim(1e-4, 1e0)\n lbl(ax_diff, 'D')\n\n pos = mu0[s0.b_pos].reshape(-1,3)\n rad = mu0[s0.b_rad]\n mask = analyze.trim_box(s0, pos)\n pos = pos[mask]\n rad = rad[mask]\n\n gx, gy = analyze.gofr(pos, rad, mu0[s0.b_zscale][0], resolution=5e-2,mask_start=0.5)\n mask = gx < 5\n gx = gx[mask]\n gy = gy[mask]\n ax_gofr = pl.subplot(gs2[1,0])\n ax_gofr.plot(gx, gy, '-', lw=1)\n ax_gofr.set_xlabel(r\"$r/a$\")\n ax_gofr.set_ylabel(r\"$g(r/a)$\")\n ax_gofr.locator_params(axis='both', nbins=5)\n #ax_gofr.semilogy()\n lbl(ax_gofr, 'E')\n\n gx, gy = analyze.gofr(pos, rad, mu0[s0.b_zscale][0], method='surface')\n mask = gx < 5\n gx = gx[mask]\n gy = gy[mask]\n gy[gy <= 0.] = gy[gy>0].min()\n ax_gofrs = pl.subplot(gs2[1,1])\n ax_gofrs.plot(gx, gy, '-', lw=1)\n ax_gofrs.set_xlabel(r\"$r/a$\")\n ax_gofrs.set_ylabel(r\"$g_{\\rm{surface}}(r/a)$\")\n ax_gofrs.locator_params(axis='both', nbins=5)\n ax_gofrs.grid(b=False, which='minor', axis='y')\n #ax_gofrs.semilogy()\n lbl(ax_gofrs, 'F')\n\n ylim = ax_gofrs.get_ylim()\n ax_gofrs.set_ylim(gy.min(), ylim[1])", "response": "Compare two states and returns the crb of the two states."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nplots two parts of the ortho view the two sections of the ortho view.", "response": "def twoslice(field, center=None, size=6.0, cmap='bone_r', vmin=0, vmax=1,\n orientation='vertical', figpad=1.09, off=0.01):\n \"\"\"\n Plot two parts of the ortho view, the two sections given by ``orientation``.\n \"\"\"\n center = center or [i//2 for i in field.shape]\n slices = []\n for i,c in enumerate(center):\n blank = [np.s_[:]]*len(center)\n blank[i] = c\n slices.append(tuple(blank))\n\n z,y,x = [float(i) for i in field.shape]\n w = float(x + z)\n h = float(y + z)\n\n def show(field, ax, slicer, transpose=False):\n tmp = field[slicer] if not transpose else field[slicer].T\n ax.imshow(\n tmp, cmap=cmap, interpolation='nearest',\n vmin=vmin, vmax=vmax\n )\n ax.set_xticks([])\n ax.set_yticks([])\n ax.grid('off')\n\n if orientation.startswith('v'):\n # rect = l,b,w,h\n log.info('{} {} {} {} {} {}'.format(x, y, z, w, h, x/h))\n r = x/h\n q = y/h\n f = 1 / (1 + 3*off)\n fig = pl.figure(figsize=(size*r, size*f))\n ax1 = fig.add_axes((off, f*(1-q)+2*off, f, f*q))\n ax2 = fig.add_axes((off, off, f, f*(1-q)))\n\n show(field, ax1, slices[0])\n show(field, ax2, slices[1])\n else:\n # rect = l,b,w,h\n r = y/w\n q = x/w\n f = 1 / (1 + 3*off)\n fig = pl.figure(figsize=(size*f, size*r))\n ax1 = fig.add_axes((off, off, f*q, f))\n ax2 = fig.add_axes((2*off+f*q, off, f*(1-q), f))\n\n show(field, ax1, slices[0])\n show(field, ax2, slices[2], transpose=True)\n\n return fig, ax1, ax2"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef circles(st, layer, axis, ax=None, talpha=1.0, cedge='white', cface='white'):\n pos = st.obj_get_positions()\n rad = st.obj_get_radii()\n shape = st.ishape.shape.tolist()\n shape.pop(axis) #shape is now the shape of the image\n if ax is None:\n fig = plt.figure()\n axisbg = 'white' if cface == 'black' else 'black'\n sx, sy = ((1,shape[1]/float(shape[0])) if shape[0] > shape[1] else\n (shape[0]/float(shape[1]), 1))\n ax = fig.add_axes((0,0, sx, sy), axisbg=axisbg)\n # get the index of the particles we want to include\n particles = np.arange(len(pos))[np.abs(pos[:,axis] - layer) < rad]\n\n # for each of these particles display the effective radius\n # in the proper place\n scale = 1.0 #np.max(shape).astype('float')\n for i in particles:\n p = pos[i].copy()\n r = 2*np.sqrt(rad[i]**2 - (p[axis] - layer)**2)\n #CIRCLE IS IN FIGURE COORDINATES!!!\n if axis==0:\n ix = 1; iy = 2\n elif axis == 1:\n ix = 0; iy = 2\n elif axis==2:\n ix = 0; iy = 1\n c = Circle((p[ix]/scale, p[iy]/scale), radius=r/2/scale, fc=cface,\n ec=cedge, alpha=talpha)\n ax.add_patch(c)\n # plt.axis([0,1,0,1])\n plt.axis('equal') #circles not ellipses\n return ax", "response": "Returns a set of circles corresponding to a slice through the platonic\n structure."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a two particle state and compare it to featuring using a single particle guess", "response": "def missing_particle(separation=0.0, radius=RADIUS, SNR=20):\n \"\"\" create a two particle state and compare it to featuring using a single particle guess \"\"\"\n # create a base image of one particle\n s = init.create_two_particle_state(imsize=6*radius+4, axis='x', sigma=1.0/SNR,\n delta=separation, radius=radius, stateargs={'varyn': True}, psfargs={'error': 1e-6})\n s.obj.typ[1] = 0.\n s.reset()\n\n return s, s.obj.pos.copy()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating a random approximation to J by returning J only at the given indices.", "response": "def get_rand_Japprox(s, params, num_inds=1000, include_cost=False, **kwargs):\n \"\"\"\n Calculates a random approximation to J by returning J only at a\n set of random pixel/voxel locations.\n\n Parameters\n ----------\n s : :class:`peri.states.State`\n The state to calculate J for.\n params : List\n The list of parameter names to calculate the gradient of.\n num_inds : Int, optional.\n The number of pix/voxels at which to calculate the random\n approximation to J. Default is 1000.\n include_cost : Bool, optional\n Set to True to append a finite-difference measure of the full\n cost gradient onto the returned J.\n\n Other Parameters\n ----------------\n All kwargs parameters get passed to s.gradmodel only.\n\n Returns\n -------\n J : numpy.ndarray\n [d, num_inds] array of J, at the given indices.\n\n return_inds : numpy.ndarray or slice\n [num_inds] element array or slice(0, None) of the model\n indices at which J was evaluated.\n \"\"\"\n start_time = time.time()\n tot_pix = s.residuals.size\n if num_inds < tot_pix:\n inds = np.random.choice(tot_pix, size=num_inds, replace=False)\n slicer = None\n return_inds = np.sort(inds)\n else:\n inds = None\n return_inds = slice(0, None)\n slicer = [slice(0, None)]*len(s.residuals.shape)\n if include_cost:\n Jact, ge = s.gradmodel_e(params=params, inds=inds, slicer=slicer,flat=False,\n **kwargs)\n Jact *= -1\n J = [Jact, ge]\n else:\n J = -s.gradmodel(params=params, inds=inds, slicer=slicer, flat=False,\n **kwargs)\n CLOG.debug('J:\\t%f' % (time.time()-start_time))\n return J, return_inds"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of the global parameter names of the state s.", "response": "def name_globals(s, remove_params=None):\n \"\"\"\n Returns a list of the global parameter names.\n\n Parameters\n ----------\n s : :class:`peri.states.ImageState`\n The state to name the globals of.\n remove_params : Set or None\n A set of unique additional parameters to remove from the globals\n list.\n\n Returns\n -------\n all_params : list\n The list of the global parameter names, with each of\n remove_params removed.\n \"\"\"\n all_params = s.params\n for p in s.param_particle(np.arange(s.obj_get_positions().shape[0])):\n all_params.remove(p)\n if remove_params is not None:\n for p in set(remove_params):\n all_params.remove(p)\n return all_params"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the number of pixels to use for a given state and parameters.", "response": "def get_num_px_jtj(s, nparams, decimate=1, max_mem=1e9, min_redundant=20):\n \"\"\"\n Calculates the number of pixels to use for J at a given memory usage.\n\n Tries to pick a number of pixels as (size of image / `decimate`).\n However, clips this to a maximum size and minimum size to ensure that\n (1) too much memory isn't used and (2) J has enough elements so that\n the inverse of JTJ will be well-conditioned.\n\n Parameters\n ----------\n s : :class:`peri.states.State`\n The state on which to calculate J.\n nparams : Int\n The number of parameters that will be included in J.\n decimate : Int, optional\n The amount to decimate the number of pixels in the image by,\n i.e. tries to pick num_px = size of image / decimate.\n Default is 1\n max_mem : Numeric, optional\n The maximum allowed memory, in bytes, for J to occupy at\n double-precision. Default is 1e9.\n min_redundant : Int, optional\n The number of pixels must be at least `min_redundant` *\n `nparams`. If not, an error is raised. Default is 20\n\n Returns\n -------\n num_px : Int\n The number of pixels at which to calcualte J.\n \"\"\"\n #1. Max for a given max_mem:\n px_mem = int(max_mem // 8 // nparams) #1 float = 8 bytes\n #2. num_pix for a given redundancy\n px_red = min_redundant*nparams\n #3. And # desired for decimation\n px_dec = s.residuals.size//decimate\n\n if px_red > px_mem:\n raise RuntimeError('Insufficient max_mem for desired redundancy.')\n num_px = np.clip(px_dec, px_red, px_mem).astype('int')\n return num_px"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef vectorize_damping(params, damping=1.0, increase_list=[['psf-', 1e4]]):\n damp_vec = np.ones(len(params)) * damping\n for nm, fctr in increase_list:\n for a in range(damp_vec.size):\n if nm in params[a]:\n damp_vec[a] *= fctr\n return damp_vec", "response": "Vectorizes the damping of a set of parameters."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef low_mem_sq(m, step=100000):\n if not m.flags.c_contiguous:\n raise ValueError('m must be C ordered for this to work with less mem.')\n # -- can make this even faster with pre-allocating arrays, but not worth it\n # right now\n # mmt = np.zeros([m.shape[0], m.shape[0]]) #6us\n # mt_tmp = np.zeros([step, m.shape[0]])\n # for a in range(0, m.shape[1], step):\n # mx = min(a+step, m.shape[1])\n # mt_tmp[:mx-a,:] = m.T[a:mx]\n # # np.dot(m_tmp, m.T, out=mmt[a:mx])\n # # np.dot(m, m[a:mx].T, out=mmt[:, a:mx])\n # np.dot(m[:,a:mx], mt_tmp[:mx], out=mmt)\n # return mmt\n mmt = np.zeros([m.shape[0], m.shape[0]]) #6us\n # m_tmp = np.zeros([step, m.shape[1]])\n for a in range(0, m.shape[0], step):\n mx = min(a+step, m.shape[1])\n # m_tmp[:] = m[a:mx]\n # np.dot(m_tmp, m.T, out=mmt[a:mx])\n mmt[:, a:mx] = np.dot(m, m[a:mx].T)\n return mmt", "response": "Compute the low mem vector for a given set of tables."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinding the particles in a tile as numpy. ndarray of ints.", "response": "def find_particles_in_tile(positions, tile):\n \"\"\"\n Finds the particles in a tile, as numpy.ndarray of ints.\n\n Parameters\n ----------\n positions : `numpy.ndarray`\n [N,3] array of the particle positions to check in the tile\n tile : :class:`peri.util.Tile` instance\n Tile of the region inside which to check for particles.\n\n Returns\n -------\n numpy.ndarray, int\n The indices of the particles in the tile.\n \"\"\"\n bools = tile.contains(positions)\n return np.arange(bools.size)[bools]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef separate_particles_into_groups(s, region_size=40, bounds=None,\n doshift=False):\n \"\"\"\n Separates particles into convenient groups for optimization.\n\n Given a state, returns a list of groups of particles. Each group of\n particles are located near each other in the image. Every particle\n located in the desired region is contained in exactly 1 group.\n\n Parameters\n ----------\n s : :class:`peri.states.ImageState`\n The peri state to find particles in.\n region_size : Int or 3-element list-like of ints, optional\n The size of the box. Groups particles into boxes of shape\n (region_size[0], region_size[1], region_size[2]). If region_size\n is a scalar, the box is a cube of length region_size.\n Default is 40.\n bounds : 2-element list-like of 3-element lists, optional\n The sub-region of the image over which to look for particles.\n bounds[0]: The lower-left corner of the image region.\n bounds[1]: The upper-right corner of the image region.\n Default (None -> ([0,0,0], s.oshape.shape)) is a box of the entire\n image size, i.e. the default places every particle in the image\n somewhere in the groups.\n doshift : {True, False, `'rand'`}, optional\n Whether or not to shift the tile boxes by half a region size, to\n prevent the same particles to be chosen every time. If `'rand'`,\n randomly chooses either True or False. Default is False\n\n Returns\n -------\n particle_groups : List\n Each element of particle_groups is an int numpy.ndarray of the\n group of nearby particles. Only contains groups with a nonzero\n number of particles, so the elements don't necessarily correspond\n to a given image region.\n \"\"\"\n imtile = s.oshape.translate(-s.pad)\n bounding_tile = (imtile if bounds is None else Tile(bounds[0], bounds[1]))\n rs = (np.ones(bounding_tile.dim, dtype='int')*region_size if\n np.size(region_size) == 1 else np.array(region_size))\n\n n_translate = np.ceil(bounding_tile.shape.astype('float')/rs).astype('int')\n particle_groups = []\n tile = Tile(left=bounding_tile.l, right=bounding_tile.l + rs)\n if doshift == 'rand':\n doshift = np.random.choice([True, False])\n if doshift:\n shift = rs // 2\n n_translate += 1\n else:\n shift = 0\n deltas = np.meshgrid(*[np.arange(i) for i in n_translate])\n positions = s.obj_get_positions()\n if bounds is None:\n # FIXME this (deliberately) masks a problem where optimization\n # places particles outside the image. However, it ensures that\n # all particles are in at least one group when `bounds is None`,\n # which is the use case within opt. The 1e-3 is to ensure that\n # they are inside the box and not on the edge.\n positions = np.clip(positions, imtile.l+1e-3, imtile.r-1e-3)\n groups = list(map(lambda *args: find_particles_in_tile(positions,\n tile.translate( np.array(args) * rs - shift)), *[d.ravel()\n for d in deltas]))\n\n for i in range(len(groups)-1, -1, -1):\n if groups[i].size == 0:\n groups.pop(i)\n assert _check_groups(s, groups)\n return groups", "response": "Separates particles into convenient groups for optimization."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_groups(s, groups):\n ans = []\n for g in groups:\n ans.extend(g)\n if np.unique(ans).size != np.size(ans):\n return False\n elif np.unique(ans).size != s.obj_get_positions().shape[0]:\n return False\n else:\n return (np.arange(s.obj_get_radii().size) == np.sort(ans)).all()", "response": "Ensures that all particles in exactly 1 group are included in exactly 1 group"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the size of the region size of the particle group.", "response": "def calc_particle_group_region_size(s, region_size=40, max_mem=1e9, **kwargs):\n \"\"\"\n Finds the biggest region size for LM particle optimization with a\n given memory constraint.\n\n Input Parameters\n ----------------\n s : :class:`peri.states.ImageState`\n The state with the particles\n region_size : Int or 3-element list-like of ints, optional.\n The initial guess for the region size. Default is 40\n max_mem : Numeric, optional\n The maximum memory for the optimizer to take. Default is 1e9\n\n Other Parameters\n ----------------\n bounds: 2-element list-like of 3-element lists.\n The sub-region of the image over which to look for particles.\n bounds[0]: The lower-left corner of the image region.\n bounds[1]: The upper-right corner of the image region.\n Default (None -> ([0,0,0], s.oshape.shape)) is a box of the entire\n image size, i.e. the default places every particle in the image\n somewhere in the groups.\n Returns\n -------\n region_size : numpy.ndarray of ints of the region size.\n \"\"\"\n region_size = np.array(region_size).astype('int')\n\n def calc_mem_usage(region_size):\n rs = np.array(region_size)\n particle_groups = separate_particles_into_groups(s, region_size=\n rs.tolist(), **kwargs)\n # The actual mem usage is the max of the memory usage of all the\n # particle groups. However this is too slow. So instead we use the\n # max of the memory of the biggest 5 particle groups:\n numpart = [np.size(g) for g in particle_groups]\n biggroups = [particle_groups[i] for i in np.argsort(numpart)[-5:]]\n def get_tile_jsize(group):\n nms = s.param_particle(group)\n tile = s.get_update_io_tiles(nms, s.get_values(nms))[2]\n return tile.shape.prod() * len(nms)\n mems = [8*get_tile_jsize(g) for g in biggroups] # 8 for bytes/float64\n return np.max(mems)\n\n im_shape = s.oshape.shape\n if calc_mem_usage(region_size) > max_mem:\n while ((calc_mem_usage(region_size) > max_mem) and\n np.any(region_size > 2)):\n region_size = np.clip(region_size-1, 2, im_shape)\n else:\n while ((calc_mem_usage(region_size) < max_mem) and\n np.any(region_size < im_shape)):\n region_size = np.clip(region_size+1, 2, im_shape)\n region_size -= 1 #need to be < memory, so we undo 1 iteration\n\n return region_size"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new state with the same state and a tile that is in the unpadded image.", "response": "def get_residuals_update_tile(st, padded_tile):\n \"\"\"\n Translates a tile in the padded image to the unpadded image.\n\n Given a state and a tile that corresponds to the padded image, returns\n a tile that corresponds to the the corresponding pixels of the difference\n image\n\n Parameters\n ----------\n st : :class:`peri.states.State`\n The state\n padded_tile : :class:`peri.util.Tile`\n The tile in the padded image.\n\n Returns\n -------\n :class:`peri.util.Tile`\n The tile corresponding to padded_tile in the unpadded image.\n \"\"\"\n inner_tile = st.ishape.intersection([st.ishape, padded_tile])\n return inner_tile.translate(-st.pad)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_best_step(err_vals):\n if np.all(np.isnan(err_vals)):\n raise ValueError('All err_vals are nans!')\n return np.nanargmin(err_vals)", "response": "Returns the index of the lowest of the passed values. Catches nans etc."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun Levenberg-Marquardt optimization on a state. Convenience wrapper for LMGlobals. Same keyword args, but the defaults have been set to useful values for optimizing globals. See LMGlobals and LMEngine for documentation. See Also -------- do_levmarq_particles : Levenberg-Marquardt optimization of a specified set of particles. do_levmarq_all_particle_groups : Levenberg-Marquardt optimization of all the particles in the state. LMGlobals : Optimizer object; the workhorse of do_levmarq. LMEngine : Engine superclass for all the optimizers.", "response": "def do_levmarq(s, param_names, damping=0.1, decrease_damp_factor=10.,\n run_length=6, eig_update=True, collect_stats=False, rz_order=0,\n run_type=2, **kwargs):\n \"\"\"\n Runs Levenberg-Marquardt optimization on a state.\n\n Convenience wrapper for LMGlobals. Same keyword args, but the defaults\n have been set to useful values for optimizing globals.\n See LMGlobals and LMEngine for documentation.\n\n See Also\n --------\n do_levmarq_particles : Levenberg-Marquardt optimization of a\n specified set of particles.\n\n do_levmarq_all_particle_groups : Levenberg-Marquardt optimization\n of all the particles in the state.\n\n LMGlobals : Optimizer object; the workhorse of do_levmarq.\n\n LMEngine : Engine superclass for all the optimizers.\n \"\"\"\n if rz_order > 0:\n aug = AugmentedState(s, param_names, rz_order=rz_order)\n lm = LMAugmentedState(aug, damping=damping, run_length=run_length,\n decrease_damp_factor=decrease_damp_factor, eig_update=\n eig_update, **kwargs)\n else:\n lm = LMGlobals(s, param_names, damping=damping, run_length=run_length,\n decrease_damp_factor=decrease_damp_factor, eig_update=\n eig_update, **kwargs)\n if run_type == 2:\n lm.do_run_2()\n elif run_type == 1:\n lm.do_run_1()\n else:\n raise ValueError('run_type=1,2 only')\n if collect_stats:\n return lm.get_termination_stats()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef do_levmarq_particles(s, particles, damping=1.0, decrease_damp_factor=10.,\n run_length=4, collect_stats=False, max_iter=2, **kwargs):\n \"\"\"\n Levenberg-Marquardt optimization on a set of particles.\n\n Convenience wrapper for LMParticles. Same keyword args, but the\n defaults have been set to useful values for optimizing particles.\n See LMParticles and LMEngine for documentation.\n\n See Also\n --------\n do_levmarq_all_particle_groups : Levenberg-Marquardt optimization\n of all the particles in the state.\n\n do_levmarq : Levenberg-Marquardt optimization of the entire state;\n useful for optimizing global parameters.\n\n LMParticles : Optimizer object; the workhorse of do_levmarq_particles.\n\n LMEngine : Engine superclass for all the optimizers.\n \"\"\"\n lp = LMParticles(s, particles, damping=damping, run_length=run_length,\n decrease_damp_factor=decrease_damp_factor, max_iter=max_iter,\n **kwargs)\n lp.do_run_2()\n if collect_stats:\n return lp.get_termination_stats()", "response": "Wrapper for Levenberg - Marquardt optimization on a set of particles."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_levmarq_all_particle_groups(s, region_size=40, max_iter=2, damping=1.0,\n decrease_damp_factor=10., run_length=4, collect_stats=False, **kwargs):\n \"\"\"\n Levenberg-Marquardt optimization for every particle in the state.\n\n Convenience wrapper for LMParticleGroupCollection. Same keyword args,\n but I've set the defaults to what I've found to be useful values for\n optimizing particles. See LMParticleGroupCollection for documentation.\n\n See Also\n --------\n do_levmarq_particles : Levenberg-Marquardt optimization of a\n specified set of particles.\n\n do_levmarq : Levenberg-Marquardt optimization of the entire state;\n useful for optimizing global parameters.\n\n LMParticleGroupCollection : The workhorse of do_levmarq.\n\n LMEngine : Engine superclass for all the optimizers.\n \"\"\"\n lp = LMParticleGroupCollection(s, region_size=region_size, damping=damping,\n run_length=run_length, decrease_damp_factor=decrease_damp_factor,\n get_cos=collect_stats, max_iter=max_iter, **kwargs)\n lp.do_run_2()\n if collect_stats:\n return lp.stats", "response": "This function is used to run all the particle groups of a state."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef burn(s, n_loop=6, collect_stats=False, desc='', rz_order=0, fractol=1e-4,\n errtol=1e-2, mode='burn', max_mem=1e9, include_rad=True,\n do_line_min='default', partial_log=False, dowarn=True):\n \"\"\"\n Optimizes all the parameters of a state.\n\n Burns a state through calling LMParticleGroupCollection and LMGlobals/\n LMAugmentedState.\n\n Parameters\n ----------\n s : :class:`peri.states.ImageState`\n The state to optimize\n n_loop : Int, optional\n The number of times to loop over in the optimizer. Default is 6.\n collect_stats : Bool, optional\n Whether or not to collect information on the optimizer's\n performance. Default is False.\n desc : string, optional\n Description to append to the states.save() call every loop.\n Set to None to avoid saving. Default is '', which selects\n one of 'burning', 'polishing', 'doing_positions'\n rz_order: Int, optional\n Set to an int > 0 to optimize with an augmented state (R(z) as\n a global parameter) vs. with the normal global parameters;\n rz_order is the order of the polynomial approximate for R(z).\n Default is 0 (no augmented state).\n fractol : Float, optional\n Fractional change in error at which to terminate. Default 1e-4\n errtol : Float, optional\n Absolute change in error at which to terminate. Default 1e-2\n mode : {'burn', 'do-particles', or 'polish'}, optional\n What mode to optimize with.\n * 'burn' : Your state is far from the minimum.\n * 'do-particles' : Positions far from minimum, globals well-fit.\n * 'polish' : The state is close to the minimum.\n 'burn' is the default. Only `polish` will get to the global\n minimum.\n max_mem : Numeric, optional\n The maximum amount of memory allowed for the optimizers' J's,\n for both particles & globals. Default is 1e9, i.e. 1GB per\n optimizer.\n do_line_min : Bool or 'default', optional\n Set to True to do an additional, third optimization per loop\n which optimizes along the subspace spanned by the last 3 steps\n of the burn()'s trajectory. In principle this should signifi-\n cantly speed up the convergence; in practice it sometimes does,\n sometimes doesn't. Default is 'default', which picks by mode:\n * 'burn' : False\n * 'do-particles' : False\n * 'polish' : True\n dowarn : Bool, optional\n Whether to log a warning if termination results from finishing\n loops rather than from convergence. Default is True.\n\n Returns\n -------\n dictionary\n Dictionary of convergence information. Contains whether the\n optimization has converged (key ``'converged'``), the values of the\n state after each loop (key ``'all_loop_values'``).\n The values of the state's parameters after each part of the\n loop: globals, particles, linemin. If ``collect_stats`` is set,\n then also contains lists of termination dicts from globals,\n particles, and line minimization (keys ``'global_stats'``,\n ``'particle_stats'``, and ``'line_stats``').\n\n Notes\n -----\n Proceeds by alternating between one Levenberg-Marquardt step\n optimizing the globals, one optimizing the particles, and repeating\n until termination.\n\n In addition, if `do_line_min` is True, at the end of each loop\n step an additional optimization is tried along the subspaced spanned\n by the steps taken during the last 3 loops. Ideally, this changes the\n convergence from linear to quadratic, but it doesn't always do much.\n\n Each of the 3 options proceed by optimizing as follows:\n * burn : lm.do_run_2(), lp.do_run_2(). No psf, 2 loops on lm.\n * do-particles : lp.do_run_2(), scales for ilm, bkg's\n * polish : lm.do_run_2(), lp.do_run_2(). Everything, 1 loop each.\n where lm is a globals LMGlobals instance, and lp a\n LMParticleGroupCollection instance.\n \"\"\"\n # It would be nice if some of these magic #'s (region size,\n # num_eig_dirs, etc) were calculated in a good way. FIXME\n mode = mode.lower()\n if mode not in {'burn', 'do-particles', 'polish'}:\n raise ValueError('mode must be one of burn, do-particles, polish')\n\n #1. Setting Defaults\n if desc is '':\n desc = mode + 'ing' if mode != 'do-particles' else 'doing-particles'\n\n eig_update = (mode != 'do-particles')\n glbl_run_length = 3 if mode == 'do-particles' else 6\n glbl_mx_itr = 2 if mode == 'burn' else 1\n use_accel = (mode == 'burn')\n rz_order = int(rz_order)\n if do_line_min == 'default':\n # do_line_min = (mode == 'polish')\n # temporary fix until we solve the particles-leaving-image issue:\n do_line_min = False\n\n if mode == 'do-particles':\n # FIXME explicit params\n # We pick some parameters for an overall illumination scale:\n glbl_nms = ['ilm-scale', 'ilm-xy-0-0', 'bkg-xy-0-0', 'offset']\n # And now, since we have explicit parameters, we check that they\n # are actually in the state:\n glbl_nms = [nm for nm in glbl_nms if nm in s.params]\n else:\n if mode == 'polish':\n remove_params = None\n else:\n # FIXME explicit params\n remove_params = s.get('psf').params\n if ('zscale' not in remove_params) and ('zscale' in s.params):\n remove_params.append('zscale')\n glbl_nms = name_globals(s, remove_params=remove_params)\n\n all_lp_stats = []\n all_lm_stats = []\n all_line_stats = []\n all_loop_values = []\n\n _delta_vals = [] # storing the directions we've moved along for line min\n #2. Optimize\n CLOG.info('Start of loop %d:\\t%f' % (0, s.error))\n for a in range(n_loop):\n start_err = s.error\n start_params = np.copy(s.state[s.params])\n #2a. Globals\n # glbl_dmp = 0.3 if a == 0 else 3e-2\n ####FIXME we damp degenerate but convenient spaces in the ilm, bkg\n ####manually, but we should do it more betterer.\n BAD_DAMP = 1e7\n BAD_LIST = [['ilm-scale', BAD_DAMP], ['ilm-off', BAD_DAMP], ['ilm-z-0',\n BAD_DAMP], ['bkg-z-0', BAD_DAMP]]\n ####\n glbl_dmp = vectorize_damping(glbl_nms + ['rz']*rz_order, damping=1.0,\n increase_list=[['psf-', 3e1]] + BAD_LIST)\n if a != 0 or mode != 'do-particles':\n if partial_log:\n log.set_level('debug')\n gstats = do_levmarq(s, glbl_nms, max_iter=glbl_mx_itr, run_length=\n glbl_run_length, eig_update=eig_update, num_eig_dirs=10,\n eig_update_frequency=3, rz_order=rz_order, damping=\n glbl_dmp, decrease_damp_factor=10., use_accel=use_accel,\n collect_stats=collect_stats, fractol=0.1*fractol,\n max_mem=max_mem)\n if partial_log:\n log.set_level('info')\n all_lm_stats.append(gstats)\n if desc is not None:\n states.save(s, desc=desc)\n CLOG.info('Globals, loop {}:\\t{}'.format(a, s.error))\n all_loop_values.append(s.values)\n\n #2b. Particles\n prtl_dmp = 1.0 if a==0 else 1e-2\n #For now, I'm calculating the region size. This might be a bad idea\n #because 1 bad particle can spoil the whole group.\n pstats = do_levmarq_all_particle_groups(s, region_size=40, max_iter=1,\n do_calc_size=True, run_length=4, eig_update=False,\n damping=prtl_dmp, fractol=0.1*fractol, collect_stats=\n collect_stats, max_mem=max_mem, include_rad=include_rad)\n all_lp_stats.append(pstats)\n if desc is not None:\n states.save(s, desc=desc)\n CLOG.info('Particles, loop {}:\\t{}'.format(a, s.error))\n gc.collect()\n all_loop_values.append(s.values)\n\n #2c. Line min?\n end_params = np.copy(s.state[s.params])\n _delta_vals.append(start_params - end_params)\n if do_line_min:\n all_line_stats.append(do_levmarq_n_directions(s, _delta_vals[-3:],\n collect_stats=collect_stats))\n if desc is not None:\n states.save(s, desc=desc)\n CLOG.info('Line min., loop {}:\\t{}'.format(a, s.error))\n all_loop_values.append(s.values)\n\n #2d. terminate?\n new_err = s.error\n derr = start_err - new_err\n dobreak = (derr/new_err < fractol) or (derr < errtol)\n if dobreak:\n break\n\n if dowarn and (not dobreak):\n CLOG.warn('burn() did not converge; consider re-running')\n\n d = {'converged':dobreak, 'all_loop_values':all_loop_values}\n if collect_stats:\n d.update({'global_stats':all_lm_stats, 'particle_stats':all_lp_stats,\n 'line_stats':all_line_stats})\n return d", "response": "Burns a state through the LMParticleGroupCollection and LMAugmentedState LMGlobalGlobal and LMGlobalGlobalGlobalGlobalGlobalGlobal and LMParticleGroupCollection and LMAugmentedState and LMAugmentedState LMAugmentedState. LMState. LMA"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncrawls slowly to the minimum-cost state. Blocks the global parameters into small enough sections such that each can be optimized separately while including all the pixels (i.e. no decimation). Optimizes the globals, then the psf separately if desired, then particles, then a line minimization along the step direction to speed up convergence. Parameters ---------- s : :class:`peri.states.ImageState` The state to optimize desc : string, optional Description to append to the states.save() call every loop. Set to `None` to avoid saving. Default is `'finish'`. n_loop : Int, optional The number of times to loop over in the optimizer. Default is 4. max_mem : Numeric, optional The maximum amount of memory allowed for the optimizers' J's, for both particles & globals. Default is 1e9. separate_psf : Bool, optional If True, does the psf optimization separately from the rest of the globals, since the psf has a more tortuous fit landscape. Default is True. fractol : Float, optional Fractional change in error at which to terminate. Default 1e-4 errtol : Float, optional Absolute change in error at which to terminate. Default 1e-2 dowarn : Bool, optional Whether to log a warning if termination results from finishing loops rather than from convergence. Default is True. Returns ------- dictionary Information about the optimization. Has two keys: ``'converged'``, a Bool which of whether optimization stopped due to convergence (True) or due to max number of iterations (False), and ``'loop_values'``, a [n_loop+1, N] ``numpy.ndarray`` of the state's values, at the start of optimization and at the end of each loop, before the line minimization.", "response": "def finish(s, desc='finish', n_loop=4, max_mem=1e9, separate_psf=True,\n fractol=1e-7, errtol=1e-3, dowarn=True):\n \"\"\"\n Crawls slowly to the minimum-cost state.\n\n Blocks the global parameters into small enough sections such that each\n can be optimized separately while including all the pixels (i.e. no\n decimation). Optimizes the globals, then the psf separately if desired,\n then particles, then a line minimization along the step direction to\n speed up convergence.\n\n Parameters\n ----------\n s : :class:`peri.states.ImageState`\n The state to optimize\n desc : string, optional\n Description to append to the states.save() call every loop.\n Set to `None` to avoid saving. Default is `'finish'`.\n n_loop : Int, optional\n The number of times to loop over in the optimizer. Default is 4.\n max_mem : Numeric, optional\n The maximum amount of memory allowed for the optimizers' J's,\n for both particles & globals. Default is 1e9.\n separate_psf : Bool, optional\n If True, does the psf optimization separately from the rest of\n the globals, since the psf has a more tortuous fit landscape.\n Default is True.\n fractol : Float, optional\n Fractional change in error at which to terminate. Default 1e-4\n errtol : Float, optional\n Absolute change in error at which to terminate. Default 1e-2\n dowarn : Bool, optional\n Whether to log a warning if termination results from finishing\n loops rather than from convergence. Default is True.\n\n Returns\n -------\n dictionary\n Information about the optimization. Has two keys: ``'converged'``,\n a Bool which of whether optimization stopped due to convergence\n (True) or due to max number of iterations (False), and\n ``'loop_values'``, a [n_loop+1, N] ``numpy.ndarray`` of the\n state's values, at the start of optimization and at the end of\n each loop, before the line minimization.\n \"\"\"\n values = [np.copy(s.state[s.params])]\n remove_params = s.get('psf').params if separate_psf else None\n # FIXME explicit params\n global_params = name_globals(s, remove_params=remove_params)\n #FIXME this could be done much better, since much of the globals such\n #as the ilm are local. Could be done with sparse matrices and/or taking\n #nearby globals in a group and using the update tile only as the slicer,\n #rather than the full residuals.\n gs = np.floor(max_mem / s.residuals.nbytes).astype('int')\n groups = [global_params[a:a+gs] for a in range(0, len(global_params), gs)]\n CLOG.info('Start ``finish``:\\t{}'.format(s.error))\n for a in range(n_loop):\n start_err = s.error\n #1. Min globals:\n for g in groups:\n do_levmarq(s, g, damping=0.1, decrease_damp_factor=20.,\n max_iter=1, max_mem=max_mem, eig_update=False)\n if separate_psf:\n do_levmarq(s, remove_params, max_mem=max_mem, max_iter=4,\n eig_update=False)\n CLOG.info('Globals, loop {}:\\t{}'.format(a, s.error))\n if desc is not None:\n states.save(s, desc=desc)\n #2. Min particles\n do_levmarq_all_particle_groups(s, max_iter=1, max_mem=max_mem)\n CLOG.info('Particles, loop {}:\\t{}'.format(a, s.error))\n if desc is not None:\n states.save(s, desc=desc)\n #3. Append vals, line min:\n values.append(np.copy(s.state[s.params]))\n # dv = (np.array(values[1:]) - np.array(values[0]))[-3:]\n # do_levmarq_n_directions(s, dv, damping=1e-2, max_iter=2, errtol=3e-4)\n # CLOG.info('Line min., loop {}:\\t{}'.format(a, s.error))\n # if desc is not None:\n # states.save(s, desc=desc)\n #4. terminate?\n new_err = s.error\n derr = start_err - new_err\n dobreak = (derr/new_err < fractol) or (derr < errtol)\n if dobreak:\n break\n\n if dowarn and (not dobreak):\n CLOG.warn('finish() did not converge; consider re-running')\n return {'converged':dobreak, 'loop_values':np.array(values)}"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfit a new component to an old component and returns a new object.", "response": "def fit_comp(new_comp, old_comp, **kwargs):\n \"\"\"\n Fits a new component to an old component\n\n Calls do_levmarq to match the .get() fields of the two objects. The\n parameters of new_comp are modified in place.\n\n Parameters\n ----------\n new_comp : :class:`peri.comps.comp`\n The new object, whose parameters to update to fit the field of\n `old_comp`. Must have a .get() attribute which returns an ndarray\n old_comp : peri.comp\n The old ilm to match to.\n\n Other Parameters\n ----------------\n Any keyword arguments to be passed to the optimizer LMGlobals\n through do_levmarq.\n\n See Also\n --------\n do_levmarq : Levenberg-Marquardt minimization using a random subset\n of the image pixels.\n \"\"\"\n #resetting the category to ilm:\n new_cat = new_comp.category\n new_comp.category = 'ilm'\n fake_s = states.ImageState(Image(old_comp.get().copy()), [new_comp], pad=0,\n mdl=mdl.SmoothFieldModel())\n do_levmarq(fake_s, new_comp.params, **kwargs)\n new_comp.category = new_cat"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nresetting all the internal state of the object to its initial state.", "response": "def reset(self, new_damping=None):\n \"\"\"\n Keeps all user supplied options the same, but resets counters etc.\n \"\"\"\n self._num_iter = 0\n self._inner_run_counter = 0\n self._J_update_counter = self.update_J_frequency\n self._fresh_JTJ = False\n self._has_run = False\n if new_damping is not None:\n self.damping = np.array(new_damping).astype('float')\n self._set_err_paramvals()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef do_run_1(self):\n while not self.check_terminate():\n self._has_run = True\n self._run1()\n self._num_iter += 1; self._inner_run_counter += 1", "response": "Run the LM run until the run is complete."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef do_run_2(self):\n while not self.check_terminate():\n self._has_run = True\n self._run2()\n self._num_iter += 1", "response": "Runs 2 steps until the process terminate."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_LM_updates(self, grad, do_correct_damping=True, subblock=None):\n if subblock is not None:\n if (subblock.sum() == 0) or (subblock.size == 0):\n CLOG.fatal('Empty subblock in find_LM_updates')\n raise ValueError('Empty sub-block')\n j = self.J[subblock]\n JTJ = np.dot(j, j.T)\n damped_JTJ = self._calc_damped_jtj(JTJ, subblock=subblock)\n grad = grad[subblock] #select the subblock of the grad\n else:\n damped_JTJ = self._calc_damped_jtj(self.JTJ, subblock=subblock)\n\n delta = self._calc_lm_step(damped_JTJ, grad, subblock=subblock)\n\n if self.use_accel:\n accel_correction = self.calc_accel_correction(damped_JTJ, delta)\n nrm_d0 = np.sqrt(np.sum(delta**2))\n nrm_corr = np.sqrt(np.sum(accel_correction**2))\n CLOG.debug('|correction| / |LM step|\\t%e' % (nrm_corr/nrm_d0))\n if nrm_corr/nrm_d0 < self.max_accel_correction:\n delta += accel_correction\n elif do_correct_damping:\n CLOG.debug('Untrustworthy step! Increasing damping...')\n self.increase_damping()\n damped_JTJ = self._calc_damped_jtj(self.JTJ, subblock=subblock)\n delta = self._calc_lm_step(damped_JTJ, grad, subblock=subblock)\n\n if np.any(np.isnan(delta)):\n CLOG.fatal('Calculated steps have nans!?')\n raise FloatingPointError('Calculated steps have nans!?')\n return delta", "response": "Calculates the LM updates for the current state of the object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _calc_lm_step(self, damped_JTJ, grad, subblock=None):\n delta0, res, rank, s = np.linalg.lstsq(damped_JTJ, -0.5*grad,\n rcond=self.min_eigval)\n if self._fresh_JTJ:\n CLOG.debug('%d degenerate of %d total directions' % (\n delta0.size-rank, delta0.size))\n if subblock is not None:\n delta = np.zeros(self.J.shape[0])\n delta[subblock] = delta0\n else:\n delta = delta0.copy()\n return delta", "response": "Calculates a Levenberg - Marquard step w / o acceleration"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_param_vals(self, new_vals, incremental=False):\n self._last_vals = self.param_vals.copy()\n if incremental:\n self.param_vals += new_vals\n else:\n self.param_vals = new_vals.copy()\n #And we've updated, so JTJ is no longer valid:\n self._fresh_JTJ = False", "response": "Updates the current set of parameter values and previous values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_expected_error(self, delta_params='calc'):\n grad = self.calc_grad()\n if list(delta_params) in [list('calc'), list('perfect')]:\n jtj = (self.JTJ if delta_params == 'perfect' else\n self._calc_damped_jtj(self.JTJ))\n delta_params = self._calc_lm_step(jtj, self.calc_grad())\n #If the model were linear, then the cost would be quadratic,\n #with Hessian 2*`self.JTJ` and gradient `grad`\n expected_error = (self.error + np.dot(grad, delta_params) +\n np.dot(np.dot(self.JTJ, delta_params), delta_params))\n return expected_error", "response": "Calculates the expected error after an update."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the cosine of the residuals with the model.", "response": "def calc_model_cosine(self, decimate=None, mode='err'):\n \"\"\"\n Calculates the cosine of the residuals with the model.\n\n Parameters\n ----------\n decimate : Int or None, optional\n Decimate the residuals by `decimate` pixels. If None, no\n decimation is used. Valid only with mode='svd'. Default\n is None\n mode : {'svd', 'err'}\n Which mode to use; see Notes section. Default is 'err'.\n\n Returns\n -------\n abs_cos : numpy.float64\n The absolute value of the model cosine.\n\n Notes\n -----\n The model cosine is defined in terms of the geometric view of\n curve-fitting, as a model manifold embedded in a high-dimensional\n space. The model cosine is the cosine of the residuals vector\n with its projection on the tangent space: :math:`cos(phi) = |P^T r|/|r|`\n where :math:`P^T` is the projection operator onto the model manifold\n and :math:`r` the residuals. This can be calculated two ways: By\n calculating the projection operator P directly with SVD (mode=`svd`),\n or by using the expected error if the model were linear to calculate\n a model sine first (mode=`err`). Since the SVD of a large matrix is\n slow, mode=`err` is faster.\n\n `decimate` allows for every nth pixel only to be counted in the\n SVD matrix of J for speed. While this is n x faster, it is\n considerably less accurate, so the default is no decimation.\n \"\"\"\n if mode == 'svd':\n slicer = slice(0, None, decimate)\n\n #1. Calculate projection term\n u, sig, v = np.linalg.svd(self.J[:,slicer], full_matrices=False) #slow part\n # p = np.dot(v.T, v) - memory error, so term-by-term\n r = self.calc_residuals()[slicer]\n abs_r = np.sqrt((r*r).sum())\n\n v_r = np.dot(v,r/abs_r)\n projected = np.dot(v.T, v_r)\n\n abs_cos = np.sqrt((projected*projected).sum())\n elif mode == 'err':\n expected_error = self.find_expected_error(delta_params='perfect')\n model_sine_2 = expected_error / self.error #error = distance^2\n abs_cos = np.sqrt(1 - model_sine_2)\n else:\n raise ValueError('mode must be one of `svd`, `err`')\n return abs_cos"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a dict of termination statistics for the current J.", "response": "def get_termination_stats(self, get_cos=True):\n \"\"\"\n Returns a dict of termination statistics\n\n Parameters\n ----------\n get_cos : Bool, optional\n Whether or not to calcualte the cosine of the residuals\n with the tangent plane of the model using the current J.\n The calculation may take some time. Default is True\n\n Returns\n -------\n dict\n Has keys\n delta_vals : The last change in parameter values.\n delta_err : The last change in the error.\n exp_err : The expected (last) change in the error.\n frac_err : The fractional change in the error.\n num_iter : The number of iterations completed.\n error : The current error.\n \"\"\"\n delta_vals = self._last_vals - self.param_vals\n delta_err = self._last_error - self.error\n frac_err = delta_err / self.error\n to_return = {'delta_vals':delta_vals, 'delta_err':delta_err,\n 'num_iter':1*self._num_iter, 'frac_err':frac_err,\n 'error':self.error, 'exp_err':self._exp_err}\n if get_cos:\n model_cosine = self.calc_model_cosine()\n to_return.update({'model_cosine':model_cosine})\n return to_return"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks if the algorithm has found a satisfactory minimum", "response": "def check_completion(self):\n \"\"\"\n Returns a Bool of whether the algorithm has found a satisfactory minimum\n \"\"\"\n terminate = False\n term_dict = self.get_termination_stats(get_cos=self.costol is not None)\n terminate |= np.all(np.abs(term_dict['delta_vals']) < self.paramtol)\n terminate |= (term_dict['delta_err'] < self.errtol)\n terminate |= (term_dict['exp_err'] < self.exptol)\n terminate |= (term_dict['frac_err'] < self.fractol)\n if self.costol is not None:\n terminate |= (curcos < term_dict['model_cosine'])\n\n return terminate"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_terminate(self):\n if not self._has_run:\n return False\n else:\n #1-3. errtol, paramtol, model cosine low enough?\n terminate = self.check_completion()\n\n #4. too many iterations??\n terminate |= (self._num_iter >= self.max_iter)\n return terminate", "response": "Checks whether a satisfactory minimum has been found and whether to terminate."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check_update_J(self):\n self._J_update_counter += 1\n update = self._J_update_counter >= self.update_J_frequency\n return update & (not self._fresh_JTJ)", "response": "Checks if the full J should be updated."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_J(self):\n self.calc_J()\n # np.dot(j, j.T) is slightly faster but 2x as much mem\n step = np.ceil(1e-2 * self.J.shape[1]).astype('int') # 1% more mem...\n self.JTJ = low_mem_sq(self.J, step=step)\n #copies still, since J is not C -ordered but a slice of j_e...\n #doing self.J.copy() works but takes 2x as much ram..\n self._fresh_JTJ = True\n self._J_update_counter = 0\n if np.any(np.isnan(self.JTJ)):\n raise FloatingPointError('J, JTJ have nans.')\n #Update self._exp_err\n self._exp_err = self.error - self.find_expected_error(delta_params='perfect')", "response": "Updates J TJ and internal counters."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef calc_grad(self):\n residuals = self.calc_residuals()\n return 2*np.dot(self.J, residuals)", "response": "Calculates the gradient of the cost w. r. t. the parameters."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the internal J matrix with the new values.", "response": "def _rank_1_J_update(self, direction, values):\n \"\"\"\n Does J += np.outer(direction, new_values - old_values) without\n using lots of memory\n \"\"\"\n vals_to_sub = np.dot(direction, self.J)\n delta_vals = values - vals_to_sub\n for a in range(direction.size):\n self.J[a] += direction[a] * delta_vals"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_Broyden_J(self):\n CLOG.debug('Broyden update.')\n delta_vals = self.param_vals - self._last_vals\n delta_residuals = self.calc_residuals() - self._last_residuals\n nrm = np.sqrt(np.dot(delta_vals, delta_vals))\n direction = delta_vals / nrm\n vals = delta_residuals / nrm\n self._rank_1_J_update(direction, vals)\n self.JTJ = np.dot(self.J, self.J.T)", "response": "Execute a Broyden update of J"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute an eigen update of J", "response": "def update_eig_J(self):\n \"\"\"Execute an eigen update of J\"\"\"\n CLOG.debug('Eigen update.')\n vls, vcs = np.linalg.eigh(self.JTJ)\n res0 = self.calc_residuals()\n for a in range(min([self.num_eig_dirs, vls.size])):\n #1. Finding stiff directions\n stif_dir = vcs[-(a+1)] #already normalized\n\n #2. Evaluating derivative along that direction, we'll use dl=5e-4:\n dl = self.eig_dl #1e-5\n _ = self.update_function(self.param_vals + dl*stif_dir)\n res1 = self.calc_residuals()\n\n #3. Updating\n grad_stif = (res1-res0)/dl\n self._rank_1_J_update(stif_dir, grad_stif)\n\n self.JTJ = np.dot(self.J, self.J.T)\n #Putting the parameters back:\n _ = self.update_function(self.param_vals)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef calc_accel_correction(self, damped_JTJ, delta0):\n #Get the derivative:\n _ = self.update_function(self.param_vals)\n rm0 = self.calc_residuals().copy()\n _ = self.update_function(self.param_vals + delta0)\n rm1 = self.calc_residuals().copy()\n _ = self.update_function(self.param_vals - delta0)\n rm2 = self.calc_residuals().copy()\n der2 = (rm2 + rm1 - 2*rm0)\n\n corr, res, rank, s = np.linalg.lstsq(damped_JTJ, np.dot(self.J, der2),\n rcond=self.min_eigval)\n corr *= -0.5\n return corr", "response": "Calculates the acceleration correction for a given set of geodesic acceleration in the LM step."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the J matrix with the values of the parameters in the boolean taxonomy.", "response": "def update_select_J(self, blk):\n \"\"\"\n Updates J only for certain parameters, described by the boolean\n mask `blk`.\n \"\"\"\n p0 = self.param_vals.copy()\n self.update_function(p0) #in case things are not put back...\n r0 = self.calc_residuals().copy()\n dl = np.zeros(p0.size, dtype='float')\n blk_J = []\n for i in np.nonzero(blk)[0]:\n dl *= 0; dl[i] = self.eig_dl\n self.update_function(p0 + dl)\n r1 = self.calc_residuals().copy()\n blk_J.append((r1-r0)/self.eig_dl)\n self.J[blk] = np.array(blk_J)\n self.update_function(p0)\n #Then we also need to update JTJ:\n self.JTJ = np.dot(self.J, self.J.T)\n if np.any(np.isnan(self.J)) or np.any(np.isnan(self.JTJ)):\n raise FloatingPointError('J, JTJ have nans.')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting error and param_vals for the exception.", "response": "def _set_err_paramvals(self):\n \"\"\"\n Must update:\n self.error, self._last_error, self.param_vals, self._last_vals\n \"\"\"\n # self.param_vals = p0 #sloppy...\n self._last_vals = self.param_vals.copy()\n self.error = self.update_function(self.param_vals)\n self._last_error = (1 + 2*self.fractol) * self.error"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef calc_J(self):\n del self.J\n self.J = np.zeros([self.param_vals.size, self.data.size])\n dp = np.zeros_like(self.param_vals)\n f0 = self.model.copy()\n for a in range(self.param_vals.size):\n dp *= 0\n dp[a] = self.dl[a]\n f1 = self.func(self.param_vals + dp, *self.func_args, **self.func_kwargs)\n grad_func = (f1 - f0) / dp[a]\n #J = grad(residuals) = -grad(model)\n self.J[a] = -grad_func", "response": "Updates self. J and returns nothing"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntaking an array param_vals updates function returns the new error", "response": "def update_function(self, param_vals):\n \"\"\"Takes an array param_vals, updates function, returns the new error\"\"\"\n self.model = self.func(param_vals, *self.func_args, **self.func_kwargs)\n d = self.calc_residuals()\n return np.dot(d.flat, d.flat)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_function(self, param_vals):\n self.opt_obj.update_function(param_vals)\n return self.opt_obj.get_error()", "response": "Updates the opt_obj returns new error."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_function(self, param_vals):\n dp = np.zeros(self.p0.size)\n for a in range(param_vals.size):\n dp += param_vals[a] * self.directions[a]\n self.state.update(self.state.params, self.p0 + dp)\n self.param_vals[:] = param_vals\n return None", "response": "Updates with param_vals = distance from self. p0 along self. direction."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef calc_J(self):\n r0 = self.state.residuals.copy().ravel()\n dl = np.zeros(self.param_vals.size)\n p0 = self.param_vals.copy()\n J = []\n for a in range(self.param_vals.size):\n dl *= 0\n dl[a] += self.dl\n self.update_function(p0 + dl)\n r1 = self.state.residuals.copy().ravel()\n J.append( (r1-r0)/self.dl)\n self.update_function(p0)\n return np.array(J)", "response": "Calculates the J along the direction."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_select_J(self, blk):\n self.update_function(self.param_vals)\n params = np.array(self.param_names)[blk].tolist()\n blk_J = -self.state.gradmodel(params=params, inds=self._inds, flat=False)\n self.J[blk] = blk_J\n #Then we also need to update JTJ:\n self.JTJ = np.dot(self.J, self.J.T)\n if np.any(np.isnan(self.J)) or np.any(np.isnan(self.JTJ)):\n raise FloatingPointError('J, JTJ have nans.')", "response": "Updates J only for certain parameters described by the boolean\n mask blk."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinding the expected error after an update.", "response": "def find_expected_error(self, delta_params='calc', adjust=True):\n \"\"\"\n Returns the error expected after an update if the model were linear.\n\n Parameters\n ----------\n delta_params : {numpy.ndarray, 'calc', or 'perfect'}, optional\n The relative change in parameters. If 'calc', uses update\n calculated from the current damping, J, etc; if 'perfect',\n uses the update calculated with zero damping.\n\n Returns\n -------\n numpy.float64\n The expected error after the update with `delta_params`\n \"\"\"\n expected_error = super(LMGlobals, self).find_expected_error(\n delta_params=delta_params)\n if adjust:\n #adjust for num_pix\n derr = (expected_error - self.error) * (self.state.residuals.size /\n float(self.num_pix))\n expected_error = self.error + derr\n return expected_error"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef calc_model_cosine(self, decimate=None, mode='err'):\n #we calculate the model cosine only in the data space of the\n #sampled indices\n if mode == 'err':\n expected_error = self.find_expected_error(delta_params='perfect',\n adjust=False)\n derr = self.error - expected_error\n residuals_err = lambda r: np.dot(r,r).sum()\n current_partial_error = residuals_err(self.calc_residuals())\n expected_partial_error = current_partial_error - derr\n model_sine_2 = expected_partial_error / current_partial_error\n abs_cos = np.sqrt(1 - model_sine_2)\n else:\n #superclass is fine\n abs_cos = super(self.__class__, self).calc_model_cosine(decimate=\n decimate, mode=mode)\n return abs_cos", "response": "Calculates the model cosine of the residuals with the model."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the gradient of the cost w. r. t. the parameters.", "response": "def calc_grad(self):\n \"\"\"The gradient of the cost w.r.t. the parameters.\"\"\"\n if self._fresh_JTJ:\n return self._graderr\n else:\n residuals = self.calc_residuals()\n return 2*np.dot(self.J, residuals)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreset the particle groups and optionally the region size and damping of the particle groups.", "response": "def reset(self, new_region_size=None, do_calc_size=True, new_damping=None,\n new_max_mem=None):\n \"\"\"\n Resets the particle groups and optionally the region size and damping.\n\n Parameters\n ----------\n new_region_size : : Int or 3-element list-like of ints, optional\n The region size for sub-blocking particles. Default is 40\n do_calc_size : Bool, optional\n If True, calculates the region size internally based on\n the maximum allowed memory. Default is True\n new_damping : Float or None, optional\n The new damping of the optimizer. Set to None to leave\n as the default for LMParticles. Default is None.\n new_max_mem : Numeric, optional\n The maximum allowed memory for J to occupy. Default is 1e9\n \"\"\"\n if new_region_size is not None:\n self.region_size = new_region_size\n if new_max_mem != None:\n self.max_mem = new_max_mem\n if do_calc_size:\n self.region_size = calc_particle_group_region_size(self.state,\n region_size=self.region_size, max_mem=self.max_mem)\n self.stats = []\n self.particle_groups = separate_particles_into_groups(self.state,\n self.region_size, doshift='rand')\n if new_damping is not None:\n self._kwargs.update({'damping':new_damping})\n if self.save_J:\n if len(self.particle_groups) > 90:\n CLOG.warn('Attempting to create many open files. Consider increasing max_mem and/or region_size to avoid crashes.')\n self._tempfiles = []\n self._has_saved_J = []\n for a in range(len(self.particle_groups)):\n #TemporaryFile is automatically deleted\n for _ in ['j','tile']:\n self._tempfiles.append(tempfile.TemporaryFile(dir=os.getcwd()))\n self._has_saved_J.append(False)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _do_run(self, mode='1'):\n for a in range(len(self.particle_groups)):\n group = self.particle_groups[a]\n lp = LMParticles(self.state, group, **self._kwargs)\n if mode == 'internal':\n lp.J, lp.JTJ, lp._dif_tile = self._load_j_diftile(a)\n\n if mode == '1':\n lp.do_run_1()\n if mode == '2':\n lp.do_run_2()\n if mode == 'internal':\n lp.do_internal_run()\n\n self.stats.append(lp.get_termination_stats(get_cos=self.get_cos))\n if self.save_J and (mode != 'internal'):\n self._dump_j_diftile(a, lp.J, lp._dif_tile)\n self._has_saved_J[a] = True", "response": "workhorse for the self. do_run_xx methods."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling self. _do_internal_run for each group of particles.", "response": "def do_internal_run(self):\n \"\"\"Calls LMParticles.do_internal_run for each group of particles.\"\"\"\n if not self.save_J:\n raise RuntimeError('self.save_J=True required for do_internal_run()')\n if not np.all(self._has_saved_J):\n raise RuntimeError('J, JTJ have not been pre-computed. Call do_run_1 or do_run_2')\n self._do_run(mode='internal')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reset(self):\n inds = list(range(self.state.obj_get_positions().shape[0]))\n self._rad_nms = self.state.param_particle_rad(inds)\n self._pos_nms = self.state.param_particle_pos(inds)\n self._initial_rad = np.copy(self.state.state[self._rad_nms])\n self._initial_pos = np.copy(self.state.state[self._pos_nms]).reshape((-1,3))\n self.param_vals[self.rscale_mask] = 0", "response": "Resets the initial radii used for updating the particles."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrighting now legval z", "response": "def _poly(self, z):\n \"\"\"Right now legval(z)\"\"\"\n shp = self.state.oshape.shape\n zmax = float(shp[0])\n zmin = 0.0\n zmid = zmax * 0.5\n\n coeffs = self.param_vals[self.rscale_mask].copy()\n if coeffs.size == 0:\n ans = 0*z\n else:\n ans = np.polynomial.legendre.legval((z-zmid)/zmid,\n self.param_vals[self.rscale_mask])\n return ans"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update(self, param_vals):\n self.update_rscl_x_params(param_vals[self.rscale_mask])\n self.state.update(self.param_names, param_vals[self.globals_mask])\n self.param_vals[:] = param_vals.copy()\n if np.any(np.isnan(self.state.residuals)):\n raise FloatingPointError('state update caused nans in residuals')", "response": "Updates all the parameters of the state + rscale x"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reset(self, **kwargs):\n self.aug_state.reset()\n super(LMAugmentedState, self).reset(**kwargs)", "response": "Resets the aug_state and the LMEngine"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of objects with a numbers of shares a link has had using Buffer.", "response": "def get_shares(self):\n '''\n Returns an object with a the numbers of shares a link has had using\n Buffer.\n\n www will be stripped, but other subdomains will not.\n '''\n\n self.shares = self.api.get(url=PATHS['GET_SHARES'] % self.url)['shares']\n\n return self.shares"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntake a sample from a field given flat indices or a shaped slice", "response": "def sample(field, inds=None, slicer=None, flat=True):\n \"\"\"\n Take a sample from a field given flat indices or a shaped slice\n\n Parameters\n -----------\n inds : list of indices\n One dimensional (raveled) indices to return from the field\n\n slicer : slice object\n A shaped (3D) slicer that returns a section of image\n\n flat : boolean\n Whether to flatten the sampled item before returning\n \"\"\"\n if inds is not None:\n out = field.ravel()[inds]\n elif slicer is not None:\n out = field[slicer].ravel()\n else:\n out = field\n\n if flat:\n return out.ravel()\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(state, filename=None, desc='', extra=None):\n if isinstance(state.image, util.RawImage):\n desc = desc or 'save'\n filename = filename or state.image.filename + '-peri-' + desc + '.pkl'\n else:\n if not filename:\n raise AttributeError(\"Must provide filename since RawImage is not used\")\n\n if extra is None:\n save = state\n else:\n save = [state] + extra\n\n if os.path.exists(filename):\n ff = \"{}-tmp-for-copy\".format(filename)\n\n if os.path.exists(ff):\n os.remove(ff)\n\n os.rename(filename, ff)\n\n pickle.dump(save, open(filename, 'wb'), protocol=2)", "response": "Save the current state of the state object to a file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(filename):\n path, name = os.path.split(filename)\n path = path or '.'\n\n with util.indir(path):\n return pickle.load(open(name, 'rb'))", "response": "Load the state from a file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef error(self):\n r = self.residuals.ravel()\n return np.dot(r,r)", "response": "Sum of the squared errors"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef loglikelihood(self):\n sig = self.hyper_parameters.get_values('sigma')\n err = self.error\n N = np.size(self.data)\n return -0.5*err/sig**2 - np.log(np.sqrt(2*np.pi)*sig)*N", "response": "Loglikelihood calculated by the model error"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupdate a single parameter or group of parameters params with values.", "response": "def update(self, params, values):\n \"\"\"\n Update a single parameter or group of parameters ``params``\n with ``values``.\n\n Parameters\n ----------\n params : string or list of strings\n Parameter names which to update\n\n value : number or list of numbers\n Values of those parameters which to update\n \"\"\"\n return super(State, self).update(params, values)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nperforming a parameter update and keep track of the change on the state.", "response": "def push_update(self, params, values):\n \"\"\"\n Perform a parameter update and keep track of the change on the state.\n Same call structure as :func:`peri.states.States.update`\n \"\"\"\n curr = self.get_values(params)\n self.stack.append((params, curr))\n self.update(params, values)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\npopping the last update from the stack push by the PeriState. update method.", "response": "def pop_update(self):\n \"\"\"\n Pop the last update from the stack push by\n :func:`peri.states.States.push_update` by undoing the chnage last\n performed.\n \"\"\"\n params, values = self.stack.pop()\n self.update(params, values)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _hess_two_param(self, funct, p0, p1, dl=2e-5, rts=False, **kwargs):\n vals0 = self.get_values(p0)\n vals1 = self.get_values(p1)\n\n f00 = funct(**kwargs)\n\n self.update(p0, vals0+dl)\n f10 = funct(**kwargs)\n\n self.update(p1, vals1+dl)\n f11 = funct(**kwargs)\n\n self.update(p0, vals0)\n f01 = funct(**kwargs)\n\n if rts:\n self.update(p0, vals0)\n self.update(p1, vals1)\n return (f11 - f10 - f01 + f00) / (dl**2)", "response": "Hessian of funct wrt two parameters p0 and p1."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _jtj(self, funct, params=None, dl=2e-5, rts=False, **kwargs):\n grad = self._grad(funct=funct, params=params, dl=dl, rts=rts, **kwargs)\n return np.dot(grad, grad.T)", "response": "get the JTJ of a function wrt to parmaeters params"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_funcs(self):\n # create essentially lambda functions, but with a nice signature\n def m(inds=None, slicer=None, flat=True):\n return sample(self.model, inds=inds, slicer=slicer, flat=flat).copy()\n\n def r(inds=None, slicer=None, flat=True):\n return sample(self.residuals, inds=inds, slicer=slicer, flat=flat).copy()\n\n def l():\n return self.loglikelihood\n\n def r_e(**kwargs):\n \"\"\"sliced etc residuals, with state.error appended on\"\"\"\n return r(**kwargs), np.copy(self.error)\n\n def m_e(**kwargs):\n \"\"\"sliced etc residuals, with state.error appended on\"\"\"\n return m(**kwargs), np.copy(self.error)\n\n # set the member functions using partial\n self.fisherinformation = partial(self._jtj, funct=m)\n self.gradloglikelihood = partial(self._grad, funct=l)\n self.hessloglikelihood = partial(self._hess, funct=l)\n self.gradmodel = partial(self._grad, funct=m)\n self.hessmodel = partial(self._hess, funct=m)\n self.JTJ = partial(self._jtj, funct=r)\n self.J = partial(self._grad, funct=r)\n self.J_e = partial(self._grad, funct=r_e, nout=2)\n self.gradmodel_e = partial(self._grad, funct=m_e, nout=2)\n\n # add the appropriate documentation to the following functions\n self.fisherinformation.__doc__ = _graddoc + _sampledoc\n self.gradloglikelihood.__doc__ = _graddoc\n self.hessloglikelihood.__doc__ = _graddoc\n self.gradmodel.__doc__ = _graddoc + _sampledoc\n self.hessmodel.__doc__ = _graddoc + _sampledoc\n self.JTJ.__doc__ = _graddoc + _sampledoc\n self.J.__doc__ = _graddoc + _sampledoc\n\n # add documentation to the private functions as well. this is done\n # slightly differently, hence the function call\n self._dograddoc(self._grad_one_param)\n self._dograddoc(self._hess_two_param)\n self._dograddoc(self._grad)\n self._dograddoc(self._hess)\n\n # the state object is a workaround so that other interfaces still\n # work. this should probably be removed in the long run\n class _Statewrap(object):\n def __init__(self, obj):\n self.obj = obj\n def __getitem__(self, d=None):\n if d is None:\n d = self.obj.params\n return util.delistify(self.obj.get_values(d), d)\n\n self.state = _Statewrap(self)", "response": "Builds the gradients and hessian functions based on the properties of the state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating the diagonal elements of the minimum covariance of the model containing the model with respect to parameters params.", "response": "def crb(self, params=None, *args, **kwargs):\n \"\"\"\n Calculate the diagonal elements of the minimum covariance of the model\n with respect to parameters params. ``*args`` and ``**kwargs`` go to\n ``fisherinformation``.\n \"\"\"\n fish = self.fisherinformation(params=params, *args, **kwargs)\n return np.sqrt(np.diag(np.linalg.inv(fish))) * self.sigma"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the model of the image.", "response": "def set_model(self, mdl):\n \"\"\"\n Setup the image model formation equation and corresponding objects into\n their various objects. `mdl` is a `peri.models.Model` object\n \"\"\"\n self.mdl = mdl\n self.mdl.check_inputs(self.comps)\n\n for c in self.comps:\n setattr(self, '_comp_'+c.category, c)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the current comparison ( real ) image", "response": "def set_image(self, image):\n \"\"\"\n Update the current comparison (real) image\n \"\"\"\n if isinstance(image, np.ndarray):\n image = util.Image(image)\n\n if isinstance(image, util.NullImage):\n self.model_as_data = True\n else:\n self.model_as_data = False\n\n self.image = image\n self._data = self.image.get_padded_image(self.pad)\n\n # set up various slicers and Tiles associated with the image and pad\n self.oshape = util.Tile(self._data.shape)\n self.ishape = self.oshape.pad(-self.pad)\n self.inner = self.ishape.slicer\n\n for c in self.comps:\n c.set_shape(self.oshape, self.ishape)\n\n self._model = np.zeros(self._data.shape, dtype=np.float64)\n self._residuals = np.zeros(self._data.shape, dtype=np.float64)\n self.calculate_model()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nswitching out the data for the model s recreation of the data.", "response": "def model_to_data(self, sigma=0.0):\n \"\"\" Switch out the data for the model's recreation of the data. \"\"\"\n im = self.model.copy()\n im += sigma*np.random.randn(*im.shape)\n self.set_image(util.NullImage(image=im))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_update_io_tiles(self, params, values):\n # get the affected area of the model image\n otile = self.get_update_tile(params, values)\n if otile is None:\n return [None]*3\n ptile = self.get_padding_size(otile) or util.Tile(0, dim=otile.dim)\n\n otile = util.Tile.intersection(otile, self.oshape)\n\n if (otile.shape <= 0).any():\n raise UpdateError(\"update triggered invalid tile size\")\n\n if (ptile.shape < 0).any() or (ptile.shape > self.oshape.shape).any():\n raise UpdateError(\"update triggered invalid padding tile size\")\n\n # now remove the part of the tile that is outside the image and pad the\n # interior part with that overhang. reflect the necessary padding back\n # into the image itself for the outer slice which we will call outer\n outer = otile.pad((ptile.shape+1)//2)\n inner, outer = outer.reflect_overhang(self.oshape)\n iotile = inner.translate(-outer.l)\n\n outer = util.Tile.intersection(outer, self.oshape)\n inner = util.Tile.intersection(inner, self.oshape)\n return outer, inner, iotile", "response": "Get the tile that needs to be updated. Inputs are the parameters and values. Returns the tile that needs to be updated."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, params, values):\n # FIXME needs to update priors\n comps = self.affected_components(params)\n\n if len(comps) == 0:\n return False\n\n # get the affected area of the model image\n otile, itile, iotile = self.get_update_io_tiles(params, values)\n\n if otile is None:\n return False\n\n # have all components update their tiles\n self.set_tile(otile)\n\n oldmodel = self._model[itile.slicer].copy()\n\n # here we diverge depending if there is only one component update\n # (so that we may calculate a variation / difference image) or if many\n # parameters are being update (should just update the whole model).\n if len(comps) == 1 and self.mdl.get_difference_model(comps[0].category):\n comp = comps[0]\n model0 = copy.deepcopy(comp.get())\n super(ImageState, self).update(params, values)\n model1 = copy.deepcopy(comp.get())\n\n diff = model1 - model0\n diff = self.mdl.evaluate(\n self.comps, 'get', diffmap={comp.category: diff}\n )\n\n if isinstance(model0, (float, int)):\n self._model[itile.slicer] += diff\n else:\n self._model[itile.slicer] += diff[iotile.slicer]\n else:\n super(ImageState, self).update(params, values)\n\n # allow the model to be evaluated using our components\n diff = self.mdl.evaluate(self.comps, 'get')\n self._model[itile.slicer] = diff[iotile.slicer]\n\n newmodel = self._model[itile.slicer].copy()\n\n # use the model image update to modify other class variables which\n # are hard to compute globally for small local updates\n self.update_from_model_change(oldmodel, newmodel, itile)\n return True", "response": "Update the state based on a set of parameters and values."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn component by category name", "response": "def get(self, name):\n \"\"\" Return component by category name \"\"\"\n for c in self.comps:\n if c.category == name:\n return c\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _calc_loglikelihood(self, model=None, tile=None):\n if model is None:\n res = self.residuals\n else:\n res = model - self._data[tile.slicer]\n\n sig, isig = self.sigma, 1.0/self.sigma\n nlogs = -np.log(np.sqrt(2*np.pi)*sig)*res.size\n return -0.5*isig*isig*np.dot(res.flat, res.flat) + nlogs", "response": "Calculates the log - likelihood of the current state of the object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating the internal variables of the internal variables based on the model changes.", "response": "def update_from_model_change(self, oldmodel, newmodel, tile):\n \"\"\"\n Update various internal variables from a model update from oldmodel to\n newmodel for the tile `tile`\n \"\"\"\n self._loglikelihood -= self._calc_loglikelihood(oldmodel, tile=tile)\n self._loglikelihood += self._calc_loglikelihood(newmodel, tile=tile)\n self._residuals[tile.slicer] = self._data[tile.slicer] - newmodel"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the memory usage level of the state.", "response": "def set_mem_level(self, mem_level='hi'):\n \"\"\"\n Sets the memory usage level of the state.\n\n Parameters\n ----------\n mem_level : string\n Can be set to one of:\n * hi : all mem's are np.float64\n * med-hi : image, platonic are float32, rest are float64\n * med : all mem's are float32\n * med-lo : image, platonic are float16, rest float32\n * lo : all are float16, which is bad for accuracy.\n\n Notes\n -----\n Right now the PSF is not affected by the mem-level changes, which is\n OK for mem but it means that self._model, self._residuals are always\n float64, which can be a chunk of mem.\n \"\"\"\n #A little thing to parse strings for convenience:\n key = ''.join([c if c in 'mlh' else '' for c in mem_level])\n if key not in ['h','mh','m','ml','m', 'l']:\n raise ValueError('mem_level must be one of hi, med-hi, med, med-lo, lo.')\n mem_levels = { 'h': [np.float64, np.float64],\n 'mh': [np.float64, np.float32],\n 'm': [np.float32, np.float32],\n 'ml': [np.float32, np.float16],\n 'l': [np.float16, np.float16]\n }\n hi_lvl, lo_lvl = mem_levels[key]\n cat_lvls = {'obj':lo_lvl,\n 'ilm':hi_lvl,\n 'bkg':lo_lvl\n } #no psf...\n\n self.image.float_precision = hi_lvl\n self.image.image = self.image.image.astype(lo_lvl)\n self.set_image(self.image)\n\n for cat in cat_lvls.keys():\n obj = self.get(cat)\n #check if it's a component collection\n if hasattr(obj, 'comps'):\n for c in obj.comps:\n c.float_precision = lo_lvl\n else:\n obj.float_precision = lo_lvl\n self._model = self._model.astype(hi_lvl)\n self._residuals = self._model.astype(hi_lvl)\n self.reset()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_img():\n # 1. particles + coverslip\n rad = 0.5 * np.random.randn(POS.shape[0]) + 4.5 # 4.5 +- 0.5 px particles\n part = objs.PlatonicSpheresCollection(POS, rad, zscale=0.89)\n slab = objs.Slab(zpos=4.92, angles=(-4.7e-3, -7.3e-4))\n objects = comp.ComponentCollection([part, slab], category='obj')\n\n # 2. psf, ilm\n p = exactpsf.FixedSSChebLinePSF(kfki=1.07, zslab=-29.3, alpha=1.17,\n n2n1=0.98, sigkf=-0.33, zscale=0.89, laser_wavelength=0.45)\n i = ilms.BarnesStreakLegPoly2P1D(npts=(16,10,8,4), zorder=8)\n b = ilms.LegendrePoly2P1D(order=(7,2,2), category='bkg')\n off = comp.GlobalScalar(name='offset', value=-2.11)\n mdl = models.ConfocalImageModel()\n st = states.ImageState(util.NullImage(shape=[48,64,64]),\n [objects, p, i, b, off], mdl=mdl, model_as_data=True)\n b.update(b.params, BKGVALS)\n i.update(i.params, ILMVALS)\n im = st.model + np.random.randn(*st.model.shape) * 0.03\n return util.Image(im)", "response": "Creates an image which is similar to the image in the tutorial"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the values of a list or single parameter.", "response": "def get_values(self, params):\n \"\"\"\n Get the value of a list or single parameter.\n\n Parameters\n ----------\n params : string, list of string\n name of parameters which to retrieve\n \"\"\"\n return util.delistify(\n [self.param_dict[p] for p in util.listify(params)], params\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_values(self, params, values):\n for p, v in zip(util.listify(params), util.listify(values)):\n self.param_dict[p] = v", "response": "Directly set the values corresponding to certain parameters."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the overall shape of the image.", "response": "def set_shape(self, shape, inner):\n \"\"\"\n Set the overall shape of the calculation area. The total shape of that\n the calculation can possibly occupy, in pixels. The second, inner, is\n the region of interest within the image.\n \"\"\"\n if self.shape != shape or self.inner != inner:\n self.shape = shape\n self.inner = inner\n self.initialize()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nnotify parent of a parameter change", "response": "def trigger_update(self, params, values):\n \"\"\" Notify parent of a parameter change \"\"\"\n if self._parent:\n self._parent.trigger_update(params, values)\n else:\n self.update(params, values)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef split_params(self, params, values=None):\n pc, vc = [], []\n\n returnvalues = values is not None\n if values is None:\n values = [0]*len(util.listify(params))\n\n for c in self.comps:\n tp, tv = [], []\n for p,v in zip(util.listify(params), util.listify(values)):\n if not p in self.lmap:\n raise NotAParameterError(\"%r does not belong to %r\" % (p, self))\n\n if c in self.pmap[p]:\n tp.append(p)\n tv.append(v)\n\n pc.append(tp)\n vc.append(tv)\n\n if returnvalues:\n return pc, vc\n return pc", "response": "Split params values into groups that correspond to the ordering in\n self. comps."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncombine the fields from all components", "response": "def get(self):\n \"\"\" Combine the fields from all components \"\"\"\n fields = [c.get() for c in self.comps]\n return self.field_reduce_func(fields)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the shape of the component.", "response": "def set_shape(self, shape, inner):\n \"\"\" Set the shape for all components \"\"\"\n for c in self.comps:\n c.set_shape(shape, inner)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sync_params(self):\n def _normalize(comps, param):\n vals = [c.get_values(param) for c in comps]\n diff = any([vals[i] != vals[i+1] for i in range(len(vals)-1)])\n\n if diff:\n for c in comps:\n c.set_values(param, vals[0])\n\n for param, comps in iteritems(self.lmap):\n if isinstance(comps, list) and len(comps) > 1:\n _normalize(comps, param)", "response": "Ensure that shared parameters are the same value everywhere"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setup_passthroughs(self):\n self._nopickle = []\n\n for c in self.comps:\n # take all member functions that start with 'param_'\n funcs = inspect.getmembers(c, predicate=inspect.ismethod)\n for func in funcs:\n if func[0].startswith('param_'):\n setattr(self, func[0], func[1])\n self._nopickle.append(func[0])\n\n # add everything from exports\n funcs = c.exports()\n for func in funcs:\n newname = c.category + '_' + func.__func__.__name__\n setattr(self, newname, func)\n self._nopickle.append(newname)", "response": "Set up the attributes that are used by the object to be used by the object."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the path to the configuration file for the current project.", "response": "def get_conf_filename():\n \"\"\"\n The configuration file either lives in ~/.peri.json or is specified on the\n command line via the environment variables PERI_CONF_FILE\n \"\"\"\n default = os.path.join(os.path.expanduser(\"~\"), \".peri.json\")\n return os.environ.get('PERI_CONF_FILE', default)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_environment():\n out = {}\n for k,v in iteritems(os.environ):\n if transform(k) in default_conf:\n out[transform(k)] = v\n return out", "response": "Read all environment variables to see if they contain PERI"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload the configuration file with the priority", "response": "def load_conf():\n \"\"\"\n Load the configuration with the priority:\n 1. environment variables\n 2. configuration file\n 3. defaults here (default_conf)\n \"\"\"\n try:\n conf = copy.copy(default_conf)\n conf.update(json.load(open(get_conf_filename())))\n conf.update(read_environment())\n return conf\n except IOError as e:\n create_default_conf()\n return load_conf()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nuse for breadcrumb dynamic_list_constructor. Returns the group name.", "response": "def get_group_name(id_group):\n \"\"\"Used for breadcrumb dynamic_list_constructor.\"\"\"\n group = Group.query.get(id_group)\n if group is not None:\n return group.name"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef index():\n page = request.args.get('page', 1, type=int)\n per_page = request.args.get('per_page', 5, type=int)\n q = request.args.get('q', '')\n\n groups = Group.query_by_user(current_user, eager=True)\n if q:\n groups = Group.search(groups, q)\n groups = groups.paginate(page, per_page=per_page)\n\n requests = Membership.query_requests(current_user).count()\n invitations = Membership.query_invitations(current_user).count()\n\n return render_template(\n 'invenio_groups/index.html',\n groups=groups,\n requests=requests,\n invitations=invitations,\n page=page,\n per_page=per_page,\n q=q\n )", "response": "List all user memberships."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef requests():\n page = request.args.get('page', 1, type=int)\n per_page = request.args.get('per_page', 5, type=int)\n memberships = Membership.query_requests(current_user, eager=True).all()\n\n return render_template(\n 'invenio_groups/pending.html',\n memberships=memberships,\n requests=True,\n page=page,\n per_page=per_page,\n )", "response": "List all pending memberships listed only for group admins."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef invitations():\n page = request.args.get('page', 1, type=int)\n per_page = request.args.get('per_page', 5, type=int)\n memberships = Membership.query_invitations(current_user, eager=True).all()\n\n return render_template(\n 'invenio_groups/pending.html',\n memberships=memberships,\n page=page,\n per_page=per_page,\n )", "response": "List all pending memberships."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlist user group members.", "response": "def members(group_id):\n \"\"\"List user group members.\"\"\"\n page = request.args.get('page', 1, type=int)\n per_page = request.args.get('per_page', 5, type=int)\n q = request.args.get('q', '')\n s = request.args.get('s', '')\n\n group = Group.query.get_or_404(group_id)\n if group.can_see_members(current_user):\n members = Membership.query_by_group(group_id, with_invitations=True)\n if q:\n members = Membership.search(members, q)\n if s:\n members = Membership.order(members, Membership.state, s)\n members = members.paginate(page, per_page=per_page)\n\n return render_template(\n \"invenio_groups/members.html\",\n group=group,\n members=members,\n page=page,\n per_page=per_page,\n q=q,\n s=s,\n )\n\n flash(\n _(\n 'You are not allowed to see members of this group %(group_name)s.',\n group_name=group.name\n ),\n 'error'\n )\n return redirect(url_for('.index'))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove(group_id, user_id):\n group = Group.query.get_or_404(group_id)\n user = User.query.get_or_404(user_id)\n\n if group.can_edit(current_user):\n try:\n group.remove_member(user)\n except Exception as e:\n flash(str(e), \"error\")\n return redirect(urlparse(request.referrer).path)\n\n flash(_('User %(user_email)s was removed from %(group_name)s group.',\n user_email=user.email, group_name=group.name), 'success')\n return redirect(urlparse(request.referrer).path)\n\n flash(\n _(\n 'You cannot delete users of the group %(group_name)s',\n group_name=group.name\n ),\n 'error'\n )\n return redirect(url_for('.index'))", "response": "Remove a user from a group."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding ( invite ) new member.", "response": "def new_member(group_id):\n \"\"\"Add (invite) new member.\"\"\"\n group = Group.query.get_or_404(group_id)\n\n if group.can_invite_others(current_user):\n form = NewMemberForm()\n\n if form.validate_on_submit():\n emails = filter(None, form.data['emails'].splitlines())\n group.invite_by_emails(emails)\n flash(_('Requests sent!'), 'success')\n return redirect(url_for('.members', group_id=group.id))\n\n return render_template(\n \"invenio_groups/new_member.html\",\n group=group,\n form=form\n )\n\n flash(\n _(\n 'You cannot invite users or yourself (i.e. join) to the group '\n '%(group_name)s',\n group_name=group.name\n ),\n 'error'\n )\n return redirect(url_for('.index'))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a sequence of initial featuring of spheres in an image.", "response": "def locate_spheres(image, feature_rad, dofilter=False, order=(3 ,3, 3),\n trim_edge=True, **kwargs):\n \"\"\"\n Get an initial featuring of sphere positions in an image.\n\n Parameters\n -----------\n image : :class:`peri.util.Image` object\n Image object which defines the image file as well as the region.\n\n feature_rad : float\n Radius of objects to find, in pixels. This is a featuring radius\n and not a real radius, so a better value is frequently smaller\n than the real radius (half the actual radius is good). If ``use_tp``\n is True, then the twice ``feature_rad`` is passed as trackpy's\n ``diameter`` keyword.\n\n dofilter : boolean, optional\n Whether to remove the background before featuring. Doing so can\n often greatly increase the success of initial featuring and\n decrease later optimization time. Filtering functions by fitting\n the image to a low-order polynomial and featuring the residuals.\n In doing so, this will change the mean intensity of the featured\n image and hence the good value of ``minmass`` will change when\n ``dofilter`` is True. Default is False.\n\n order : 3-element tuple, optional\n If `dofilter`, the 2+1D Leg Poly approximation to the background\n illumination field. Default is (3,3,3).\n\n Other Parameters\n ----------------\n invert : boolean, optional\n Whether to invert the image for featuring. Set to True if the\n image is dark particles on a bright background. Default is True\n minmass : Float or None, optional\n The minimum mass/masscut of a particle. Default is None, which\n calculates internally.\n use_tp : Bool, optional\n Whether or not to use trackpy. Default is False, since trackpy\n cuts out particles at the edge.\n\n Returns\n --------\n positions : np.ndarray [N,3]\n Positions of the particles in order (z,y,x) in image pixel units.\n\n Notes\n -----\n Optionally filters the image by fitting the image I(x,y,z) to a\n polynomial, then subtracts this fitted intensity variation and uses\n centroid methods to find the particles.\n \"\"\"\n # We just want a smoothed field model of the image so that the residuals\n # are simply the particles without other complications\n m = models.SmoothFieldModel()\n I = ilms.LegendrePoly2P1D(order=order, constval=image.get_image().mean())\n s = states.ImageState(image, [I], pad=0, mdl=m)\n if dofilter:\n opt.do_levmarq(s, s.params)\n pos = addsub.feature_guess(s, feature_rad, trim_edge=trim_edge, **kwargs)[0]\n return pos"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_initial_featuring(statemaker, feature_rad, actual_rad=None,\n im_name=None, tile=None, invert=True, desc='', use_full_path=False,\n featuring_params={}, statemaker_kwargs={}, **kwargs):\n \"\"\"\n Completely optimizes a state from an image of roughly monodisperse\n particles.\n\n The user can interactively select the image. The state is periodically\n saved during optimization, with different filename for different stages\n of the optimization.\n\n Parameters\n ----------\n statemaker : Function\n A statemaker function. Given arguments `im` (a\n :class:`~peri.util.Image`), `pos` (numpy.ndarray), `rad` (ndarray),\n and any additional `statemaker_kwargs`, must return a\n :class:`~peri.states.ImageState`. There is an example function in\n scripts/statemaker_example.py\n feature_rad : Int, odd\n The particle radius for featuring, as passed to locate_spheres.\n actual_rad : Float, optional\n The actual radius of the particles. Default is feature_rad\n im_name : string, optional\n The file name of the image to load. If not set, it is selected\n interactively through Tk.\n tile : :class:`peri.util.Tile`, optional\n The tile of the raw image to be analyzed. Default is None, the\n entire image.\n invert : Bool, optional\n Whether to invert the image for featuring, as passed to trackpy.\n Default is True.\n desc : String, optional\n A description to be inserted in saved state. The save name will\n be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''\n use_full_path : Bool, optional\n Set to True to use the full path name for the image. Default\n is False.\n featuring_params : Dict, optional\n kwargs-like dict of any additional keyword arguments to pass to\n ``get_initial_featuring``, such as ``'use_tp'`` or ``'minmass'``.\n Default is ``{}``.\n statemaker_kwargs : Dict, optional\n kwargs-like dict of any additional keyword arguments to pass to\n the statemaker function. Default is ``{}``.\n\n Other Parameters\n ----------------\n max_mem : Numeric\n The maximum additional memory to use for the optimizers, as\n passed to optimize.burn. Default is 1e9.\n min_rad : Float, optional\n The minimum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius smaller than this are identified\n as fake and removed. Default is 0.5 * actual_rad.\n max_rad : Float, optional\n The maximum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius larger than this are identified\n as fake and removed. Default is 1.5 * actual_rad, however you\n may find better results if you make this more stringent.\n rz_order : int, optional\n If nonzero, the order of an additional augmented rscl(z)\n parameter for optimization. Default is 0; i.e. no rscl(z)\n optimization.\n zscale : Float, optional\n The zscale of the image. Default is 1.0\n\n Returns\n -------\n s : :class:`peri.states.ImageState`\n The optimized state.\n\n See Also\n --------\n feature_from_pos_rad : Using a previous state's globals and\n user-provided positions and radii as an initial guess,\n completely optimizes a state.\n\n get_particle_featuring : Using a previous state's globals and\n positions as an initial guess, completely optimizes a state.\n\n translate_featuring : Use a previous state's globals and\n centroids methods for an initial particle guess, completely\n optimizes a state.\n\n Notes\n -----\n Proceeds by centroid-featuring the image for an initial guess of\n particle positions, then optimizing the globals + positions until\n termination as called in _optimize_from_centroid.\n The ``Other Parameters`` are passed to _optimize_from_centroid.\n \"\"\"\n if actual_rad is None:\n actual_rad = feature_rad\n\n _, im_name = _pick_state_im_name('', im_name, use_full_path=use_full_path)\n im = util.RawImage(im_name, tile=tile)\n\n pos = locate_spheres(im, feature_rad, invert=invert, **featuring_params)\n if np.size(pos) == 0:\n msg = 'No particles found. Try using a smaller `feature_rad`.'\n raise ValueError(msg)\n\n rad = np.ones(pos.shape[0], dtype='float') * actual_rad\n s = statemaker(im, pos, rad, **statemaker_kwargs)\n RLOG.info('State Created.')\n if desc is not None:\n states.save(s, desc=desc+'initial')\n optimize_from_initial(s, invert=invert, desc=desc, **kwargs)\n return s", "response": "This function is used to get the initial state from an image of roughly monodisperse\n particles."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a state from an image and an initial guess of the particle positions and radii.", "response": "def feature_from_pos_rad(statemaker, pos, rad, im_name=None, tile=None,\n desc='', use_full_path=False, statemaker_kwargs={}, **kwargs):\n \"\"\"\n Gets a completely-optimized state from an image and an initial guess of\n particle positions and radii.\n\n The state is periodically saved during optimization, with different\n filename for different stages of the optimization. The user can select\n the image.\n\n Parameters\n ----------\n statemaker : Function\n A statemaker function. Given arguments `im` (a\n :class:`~peri.util.Image`), `pos` (numpy.ndarray), `rad` (ndarray),\n and any additional `statemaker_kwargs`, must return a\n :class:`~peri.states.ImageState`. There is an example function in\n scripts/statemaker_example.py\n pos : [N,3] element numpy.ndarray.\n The initial guess for the N particle positions.\n rad : N element numpy.ndarray.\n The initial guess for the N particle radii.\n im_name : string or None, optional\n The filename of the image to feature. Default is None, in which\n the user selects the image.\n tile : :class:`peri.util.Tile`, optional\n A tile of the sub-region of the image to feature. Default is\n None, i.e. entire image.\n desc : String, optional\n A description to be inserted in saved state. The save name will\n be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''\n use_full_path : Bool, optional\n Set to True to use the full path name for the image. Default\n is False.\n statemaker_kwargs : Dict, optional\n kwargs-like dict of any additional keyword arguments to pass to\n the statemaker function. Default is ``{}``.\n\n Other Parameters\n ----------------\n max_mem : Numeric\n The maximum additional memory to use for the optimizers, as\n passed to optimize.burn. Default is 1e9.\n min_rad : Float, optional\n The minimum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius smaller than this are identified\n as fake and removed. Default is 0.5 * actual_rad.\n max_rad : Float, optional\n The maximum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius larger than this are identified\n as fake and removed. Default is 1.5 * actual_rad, however you\n may find better results if you make this more stringent.\n invert : {'guess', True, False}\n Whether to invert the image for featuring, as passed to\n addsubtract.add_subtract. Default is to guess from the\n current state's particle positions.\n rz_order : int, optional\n If nonzero, the order of an additional augmented rscl(z)\n parameter for optimization. Default is 0; i.e. no rscl(z)\n optimization.\n zscale : Float, optional\n The zscale of the image. Default is 1.0\n\n Returns\n -------\n s : :class:`peri.states.ImageState`\n The optimized state.\n\n See Also\n --------\n get_initial_featuring : Features an image from scratch, using\n centroid methods as initial particle locations.\n\n get_particle_featuring : Using a previous state's globals and\n positions as an initial guess, completely optimizes a state.\n\n translate_featuring : Use a previous state's globals and\n centroids methods for an initial particle guess, completely\n optimizes a state.\n\n Notes\n -----\n The ``Other Parameters`` are passed to _optimize_from_centroid.\n Proceeds by centroid-featuring the image for an initial guess of\n particle positions, then optimizing the globals + positions until\n termination as called in _optimize_from_centroid.\n \"\"\"\n if np.size(pos) == 0:\n raise ValueError('`pos` is an empty array.')\n elif np.shape(pos)[1] != 3:\n raise ValueError('`pos` must be an [N,3] element numpy.ndarray.')\n _, im_name = _pick_state_im_name('', im_name, use_full_path=use_full_path)\n im = util.RawImage(im_name, tile=tile)\n s = statemaker(im, pos, rad, **statemaker_kwargs)\n RLOG.info('State Created.')\n if desc is not None:\n states.save(s, desc=desc+'initial')\n optimize_from_initial(s, desc=desc, **kwargs)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef optimize_from_initial(s, max_mem=1e9, invert='guess', desc='', rz_order=3,\n min_rad=None, max_rad=None):\n \"\"\"\n Optimizes a state from an initial set of positions and radii, without\n any known microscope parameters.\n\n Parameters\n ----------\n s : :class:`peri.states.ImageState`\n The state to optimize. It is modified internally and returned.\n max_mem : Numeric, optional\n The maximum memory for the optimizer to use. Default is 1e9 (bytes)\n invert : Bool or `'guess'`, optional\n Set to True if the image is dark particles on a bright\n background, False otherwise. Used for add-subtract. The\n default is to guess from the state's current particles.\n desc : String, optional\n An additional description to infix for periodic saving along the\n way. Default is the null string ``''``.\n rz_order : int, optional\n ``rz_order`` as passed to opt.burn. Default is 3\n min_rad : Float or None, optional\n The minimum radius to identify a particles as bad, as passed to\n add-subtract. Default is None, which picks half the median radii.\n If your sample is not monodisperse you should pick a different\n value.\n max_rad : Float or None, optional\n The maximum radius to identify a particles as bad, as passed to\n add-subtract. Default is None, which picks 1.5x the median radii.\n If your sample is not monodisperse you should pick a different\n value.\n\n Returns\n -------\n s : :class:`peri.states.ImageState`\n The optimized state, which is the same as the input ``s`` but\n modified in-place.\n \"\"\"\n RLOG.info('Initial burn:')\n if desc is not None:\n desc_burn = desc + 'initial-burn'\n desc_polish = desc + 'addsub-polish'\n else:\n desc_burn, desc_polish = [None] * 2\n opt.burn(s, mode='burn', n_loop=3, fractol=0.1, desc=desc_burn,\n max_mem=max_mem, include_rad=False, dowarn=False)\n opt.burn(s, mode='burn', n_loop=3, fractol=0.1, desc=desc_burn,\n max_mem=max_mem, include_rad=True, dowarn=False)\n\n RLOG.info('Start add-subtract')\n rad = s.obj_get_radii()\n if min_rad is None:\n min_rad = 0.5 * np.median(rad)\n if max_rad is None:\n max_rad = 1.5 * np.median(rad)\n addsub.add_subtract(s, tries=30, min_rad=min_rad, max_rad=max_rad,\n invert=invert)\n if desc is not None:\n states.save(s, desc=desc + 'initial-addsub')\n\n RLOG.info('Final polish:')\n d = opt.burn(s, mode='polish', n_loop=8, fractol=3e-4, desc=desc_polish,\n max_mem=max_mem, rz_order=rz_order, dowarn=False)\n if not d['converged']:\n RLOG.warn('Optimization did not converge; consider re-running')\n return s", "response": "Optimize a state from an initial set of positions and radii."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef translate_featuring(state_name=None, im_name=None, use_full_path=False,\n **kwargs):\n \"\"\"\n Translates one optimized state into another image where the particles\n have moved by a small amount (~1 particle radius).\n\n Returns a completely-optimized state. The user can interactively\n selects the initial state and the second raw image. The state is\n periodically saved during optimization, with different filename for\n different stages of the optimization.\n\n Parameters\n ----------\n state_name : String or None, optional\n The name of the initially-optimized state. Default is None,\n which prompts the user to select the name interactively\n through a Tk window.\n im_name : String or None, optional\n The name of the new image to optimize. Default is None,\n which prompts the user to select the name interactively\n through a Tk window.\n use_full_path : Bool, optional\n Set to True to use the full path of the state instead of\n partial path names (e.g. /full/path/name/state.pkl vs\n state.pkl). Default is False\n\n Other Parameters\n ----------------\n max_mem : Numeric\n The maximum additional memory to use for the optimizers, as\n passed to optimize.burn. Default is 1e9.\n desc : String, optional\n A description to be inserted in saved state. The save name will\n be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''\n min_rad : Float, optional\n The minimum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius smaller than this are identified\n as fake and removed. Default is 0.5 * actual_rad.\n max_rad : Float, optional\n The maximum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius larger than this are identified\n as fake and removed. Default is 1.5 * actual_rad, however you\n may find better results if you make this more stringent.\n invert : {True, False, 'guess'}\n Whether to invert the image for featuring, as passed to\n addsubtract.add_subtract. Default is to guess from the\n state's current particles.\n rz_order : int, optional\n If nonzero, the order of an additional augmented rscl(z)\n parameter for optimization. Default is 0; i.e. no rscl(z)\n optimization.\n do_polish : Bool, optional\n Set to False to only optimize the particles and add-subtract.\n Default is True, which then runs a polish afterwards.\n\n Returns\n -------\n s : :class:`peri.states.ImageState`\n The optimized state.\n\n See Also\n --------\n get_initial_featuring : Features an image from scratch, using\n centroid methods as initial particle locations.\n\n feature_from_pos_rad : Using a previous state's globals and\n user-provided positions and radii as an initial guess,\n completely optimizes a state.\n\n get_particle_featuring : Using a previous state's globals and\n positions as an initial guess, completely optimizes a state.\n\n Notes\n -----\n The ``Other Parameters`` are passed to _translate_particles.\n Proceeds by:\n 1. Optimize particle positions only.\n 2. Optimize particle positions and radii only.\n 3. Add-subtract missing and bad particles.\n 4. If polish, optimize the illumination, background, and particles.\n 5. If polish, optimize everything.\n \"\"\"\n state_name, im_name = _pick_state_im_name(\n state_name, im_name, use_full_path=use_full_path)\n\n s = states.load(state_name)\n im = util.RawImage(im_name, tile=s.image.tile)\n\n s.set_image(im)\n _translate_particles(s, **kwargs)\n return s", "response": "Translate a single state into another state."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_particles_featuring(feature_rad, state_name=None, im_name=None,\n use_full_path=False, actual_rad=None, invert=True, featuring_params={},\n **kwargs):\n \"\"\"\n Combines centroid featuring with the globals from a previous state.\n\n Runs trackpy.locate on an image, sets the globals from a previous state,\n calls _translate_particles\n\n Parameters\n ----------\n feature_rad : Int, odd\n The particle radius for featuring, as passed to locate_spheres.\n\n state_name : String or None, optional\n The name of the initially-optimized state. Default is None,\n which prompts the user to select the name interactively\n through a Tk window.\n im_name : String or None, optional\n The name of the new image to optimize. Default is None,\n which prompts the user to select the name interactively\n through a Tk window.\n use_full_path : Bool, optional\n Set to True to use the full path of the state instead of\n partial path names (e.g. /full/path/name/state.pkl vs\n state.pkl). Default is False\n actual_rad : Float or None, optional\n The initial guess for the particle radii. Default is the median\n of the previous state.\n invert : Bool\n Whether to invert the image for featuring, as passed to\n addsubtract.add_subtract and locate_spheres. Set to False\n if the image is bright particles on a dark background.\n Default is True (dark particles on bright background).\n featuring_params : Dict, optional\n kwargs-like dict of any additional keyword arguments to pass to\n ``get_initial_featuring``, such as ``'use_tp'`` or ``'minmass'``.\n Default is ``{}``.\n\n\n Other Parameters\n ----------------\n max_mem : Numeric\n The maximum additional memory to use for the optimizers, as\n passed to optimize.burn. Default is 1e9.\n desc : String, optional\n A description to be inserted in saved state. The save name will\n be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''\n min_rad : Float, optional\n The minimum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius smaller than this are identified\n as fake and removed. Default is 0.5 * actual_rad.\n max_rad : Float, optional\n The maximum particle radius, as passed to addsubtract.add_subtract.\n Particles with a fitted radius larger than this are identified\n as fake and removed. Default is 1.5 * actual_rad, however you\n may find better results if you make this more stringent.\n rz_order : int, optional\n If nonzero, the order of an additional augmented rscl(z)\n parameter for optimization. Default is 0; i.e. no rscl(z)\n optimization.\n do_polish : Bool, optional\n Set to False to only optimize the particles and add-subtract.\n Default is True, which then runs a polish afterwards.\n\n Returns\n -------\n s : :class:`peri.states.ImageState`\n The optimized state.\n\n See Also\n --------\n get_initial_featuring : Features an image from scratch, using\n centroid methods as initial particle locations.\n\n feature_from_pos_rad : Using a previous state's globals and\n user-provided positions and radii as an initial guess,\n completely optimizes a state.\n\n translate_featuring : Use a previous state's globals and\n centroids methods for an initial particle guess, completely\n optimizes a state.\n\n Notes\n -----\n The ``Other Parameters`` are passed to _translate_particles.\n Proceeds by:\n 1. Find a guess of the particle positions through centroid methods.\n 2. Optimize particle positions only.\n 3. Optimize particle positions and radii only.\n 4. Add-subtract missing and bad particles.\n 5. If polish, optimize the illumination, background, and particles.\n 6. If polish, optimize everything.\n \"\"\"\n state_name, im_name = _pick_state_im_name(\n state_name, im_name, use_full_path=use_full_path)\n s = states.load(state_name)\n\n if actual_rad is None:\n actual_rad = np.median(s.obj_get_radii())\n im = util.RawImage(im_name, tile=s.image.tile)\n pos = locate_spheres(im, feature_rad, invert=invert, **featuring_params)\n _ = s.obj_remove_particle(np.arange(s.obj_get_radii().size))\n s.obj_add_particle(pos, np.ones(pos.shape[0])*actual_rad)\n\n s.set_image(im)\n _translate_particles(s, invert=invert, **kwargs)\n return s", "response": "Returns a new state with the given particle radius and state name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _pick_state_im_name(state_name, im_name, use_full_path=False):\n initial_dir = os.getcwd()\n if (state_name is None) or (im_name is None):\n wid = tk.Tk()\n wid.withdraw()\n if state_name is None:\n state_name = tkfd.askopenfilename(\n initialdir=initial_dir, title='Select pre-featured state')\n os.chdir(os.path.dirname(state_name))\n\n if im_name is None:\n im_name = tkfd.askopenfilename(\n initialdir=initial_dir, title='Select new image')\n\n if (not use_full_path) and (os.path.dirname(im_name) != ''):\n im_path = os.path.dirname(im_name)\n os.chdir(im_path)\n im_name = os.path.basename(im_name)\n else:\n os.chdir(initial_dir)\n return state_name, im_name", "response": "Picks the state and image name from the Tk file system."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntranslating a set of particles into a single set of states.", "response": "def _translate_particles(s, max_mem=1e9, desc='', min_rad='calc',\n max_rad='calc', invert='guess', rz_order=0, do_polish=True):\n \"\"\"\n Workhorse for translating particles. See get_particles_featuring for docs.\n \"\"\"\n if desc is not None:\n desc_trans = desc + 'translate-particles'\n desc_burn = desc + 'addsub_burn'\n desc_polish = desc + 'addsub_polish'\n else:\n desc_trans, desc_burn, desc_polish = [None]*3\n RLOG.info('Translate Particles:')\n opt.burn(s, mode='do-particles', n_loop=4, fractol=0.1, desc=desc_trans,\n max_mem=max_mem, include_rad=False, dowarn=False)\n opt.burn(s, mode='do-particles', n_loop=4, fractol=0.05, desc=desc_trans,\n max_mem=max_mem, include_rad=True, dowarn=False)\n\n RLOG.info('Start add-subtract')\n addsub.add_subtract(s, tries=30, min_rad=min_rad, max_rad=max_rad,\n invert=invert)\n if desc is not None:\n states.save(s, desc=desc + 'translate-addsub')\n\n if do_polish:\n RLOG.info('Final Burn:')\n opt.burn(s, mode='burn', n_loop=3, fractol=3e-4, desc=desc_burn,\n max_mem=max_mem, rz_order=rz_order,dowarn=False)\n RLOG.info('Final Polish:')\n d = opt.burn(s, mode='polish', n_loop=4, fractol=3e-4, desc=desc_polish,\n max_mem=max_mem, rz_order=rz_order, dowarn=False)\n if not d['converged']:\n RLOG.warn('Optimization did not converge; consider re-running')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef link_zscale(st):\n # FIXME should be made more generic to other parameters and categories\n psf = st.get('psf')\n psf.param_dict['zscale'] = psf.param_dict['psf-zscale']\n psf.params[psf.params.index('psf-zscale')] = 'zscale'\n psf.global_zscale = True\n psf.param_dict.pop('psf-zscale')\n st.trigger_parameter_change()\n st.reset()", "response": "Links the state st psf zscale with the global zscale"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef finish_state(st, desc='finish-state', invert='guess'):\n for minmass in [None, 0]:\n for _ in range(3):\n npart, poses = addsub.add_subtract_locally(st, region_depth=7,\n minmass=minmass, invert=invert)\n if npart == 0:\n break\n opt.finish(st, n_loop=1, separate_psf=True, desc=desc, dowarn=False)\n opt.burn(st, mode='polish', desc=desc, n_loop=2, dowarn=False)\n d = opt.finish(st, desc=desc, n_loop=4, dowarn=False)\n if not d['converged']:\n RLOG.warn('Optimization did not converge; consider re-running')", "response": "Finishes the state of the best - possible state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\noptimize a particle by computing the gradients and hessian of the particle at the given index.", "response": "def optimize_particle(state, index, method='gn', doradius=True):\n \"\"\"\n Methods available are\n gn : Gauss-Newton with JTJ (recommended)\n nr : Newton-Rhaphson with hessian\n\n if doradius, also optimize the radius.\n \"\"\"\n blocks = state.param_particle(index)\n\n if not doradius:\n blocks = blocks[:-1]\n\n g = state.gradloglikelihood(blocks=blocks)\n if method == 'gn':\n h = state.jtj(blocks=blocks)\n if method == 'nr':\n h = state.hessloglikelihood(blocks=blocks)\n step = np.linalg.solve(h, g)\n\n h = np.zeros_like(g)\n for i in range(len(g)):\n state.update(blocks[i], state.state[blocks[i]] - step[i])\n return g,h"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new image state with an initial centroid of the image.", "response": "def makestate(im, pos, rad, slab=None, mem_level='hi'):\n \"\"\"\n Workhorse for creating & optimizing states with an initial centroid\n guess.\n\n This is an example function that works for a particular microscope. For\n your own microscope, you'll need to change particulars such as the psf\n type and the orders of the background and illumination.\n\n Parameters\n ----------\n im : :class:`~peri.util.RawImage`\n A RawImage of the data.\n pos : [N,3] element numpy.ndarray.\n The initial guess for the N particle positions.\n rad : N element numpy.ndarray.\n The initial guess for the N particle radii.\n\n slab : :class:`peri.comp.objs.Slab` or None, optional\n If not None, a slab corresponding to that in the image. Default\n is None.\n mem_level : {'lo', 'med-lo', 'med', 'med-hi', 'hi'}, optional\n A valid memory level for the state to control the memory overhead\n at the expense of accuracy. Default is `'hi'`\n\n Returns\n -------\n :class:`~peri.states.ImageState`\n An ImageState with a linked z-scale, a ConfocalImageModel, and\n all the necessary components with orders at which are useful for\n my particular test case.\n \"\"\"\n if slab is not None:\n o = comp.ComponentCollection(\n [\n objs.PlatonicSpheresCollection(pos, rad, zscale=zscale),\n slab\n ],\n category='obj'\n )\n else:\n o = objs.PlatonicSpheresCollection(pos, rad, zscale=zscale)\n\n p = exactpsf.FixedSSChebLinePSF()\n npts, iorder = _calc_ilm_order(im.get_image().shape)\n i = ilms.BarnesStreakLegPoly2P1D(npts=npts, zorder=iorder)\n b = ilms.LegendrePoly2P1D(order=(9 ,3, 5), category='bkg')\n c = comp.GlobalScalar('offset', 0.0)\n s = states.ImageState(im, [o, i, b, c, p])\n runner.link_zscale(s)\n if mem_level != 'hi':\n s.set_mem_level(mem_level)\n\n opt.do_levmarq(s, ['ilm-scale'], max_iter=1, run_length=6, max_mem=1e4)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _calc_ilm_order(imshape):\n zorder = int(imshape[0] / 6.25) + 1\n l_npts = int(imshape[1] / 42.5)+1\n npts = ()\n for a in range(l_npts):\n if a < 5:\n npts += (int(imshape[2] * [59, 39, 29, 19, 14][a]/512.) + 1,)\n else:\n npts += (int(imshape[2] * 11/512.) + 1,)\n return npts, zorder", "response": "Calculates an ilm order based on the shape of an image."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _check_for_inception(self, root_dict):\n '''\n Used to check if there is a dict in a dict\n '''\n\n for key in root_dict:\n if isinstance(root_dict[key], dict):\n root_dict[key] = ResponseObject(root_dict[key])\n\n return root_dict", "response": "Checks if the dict contains any inceptions"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating random parameters for this ILM that mimic experiments .", "response": "def randomize_parameters(self, ptp=0.2, fourier=False, vmin=None, vmax=None):\n \"\"\"\n Create random parameters for this ILM that mimic experiments\n as closely as possible without real assumptions.\n \"\"\"\n if vmin is not None and vmax is not None:\n ptp = vmax - vmin\n elif vmax is not None and vmin is None:\n vmin = vmax - ptp\n elif vmin is not None and vmax is None:\n vmax = vmin + ptp\n else:\n vmax = 1.0\n vmin = vmax - ptp\n\n self.set_values(self.category+'-scale', 1.0)\n self.set_values(self.category+'-off', 0.0)\n\n for k, v in iteritems(self.poly_params):\n norm = (self.zorder + 1.0)*2\n self.set_values(k, ptp*(np.random.rand() - 0.5) / norm)\n\n for i, p in enumerate(self.barnes_params):\n N = len(p)\n if fourier:\n t = ((np.random.rand(N)-0.5) + 1.j*(np.random.rand(N)-0.5))/(np.arange(N)+1)\n q = np.real(np.fft.ifftn(t)) / (i+1)\n else:\n t = ptp*np.sqrt(N)*(np.random.rand(N)-0.5)\n q = np.cumsum(t) / (i+1)\n\n q = ptp * q / q.ptp() / len(self.barnes_params)\n q -= q.mean()\n self.set_values(p, q)\n\n self._norm_stat = [ptp, vmin]\n\n if self.shape:\n self.initialize()\n\n if self._parent:\n param = self.category+'-scale'\n self.trigger_update(param, self.get_values(param))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a barnes interpolant and calculates its values", "response": "def _barnes(self, pos):\n \"\"\"Creates a barnes interpolant & calculates its values\"\"\"\n b_in = self.b_in\n dist = lambda x: np.sqrt(np.dot(x,x))\n #we take a filter size as the max distance between the grids along\n #x or y:\n sz = self.npts[1]\n coeffs = self.get_values(self.barnes_params)\n\n b = BarnesInterpolationND(\n b_in, coeffs, filter_size=self.filtsize, damp=0.9, iterations=3,\n clip=self.local_updates, clipsize=self.barnes_clip_size,\n blocksize=100 # FIXME magic blocksize\n )\n return b(pos)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the details of the posting schedules associated with a social media profile.", "response": "def schedules(self):\n '''\n Returns details of the posting schedules associated with a social media\n profile.\n '''\n\n url = PATHS['GET_SCHEDULES'] % self.id\n\n self.__schedules = self.api.get(url=url)\n\n return self.__schedules"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef schedules(self, schedules):\n '''\n Set the posting schedules for the specified social media profile.\n '''\n\n url = PATHS['UPDATE_SCHEDULES'] % self.id\n\n data_format = \"schedules[0][%s][]=%s&\"\n post_data = \"\"\n\n for format_type, values in schedules.iteritems():\n for value in values:\n post_data += data_format % (format_type, value)\n\n self.api.post(url=url, data=post_data)", "response": "Updates the posting schedules for the specified social media profile."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the moments of the probability distribution p with vector v", "response": "def moment(p, v, order=1):\n \"\"\" Calculates the moments of the probability distribution p with vector v \"\"\"\n if order == 1:\n return (v*p).sum()\n elif order == 2:\n return np.sqrt( ((v**2)*p).sum() - (v*p).sum()**2 )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the 3D PSF at a particular z pixel height.", "response": "def psf_slice(self, zint, size=11, zoffset=0., getextent=False):\n \"\"\"\n Calculates the 3D psf at a particular z pixel height\n\n Parameters\n ----------\n zint : float\n z pixel height in image coordinates , converted to 1/k by the\n function using the slab position as well\n\n size : int, list, tuple\n The size over which to calculate the psf, can be 1 or 3 elements\n for the different axes in image pixel coordinates\n\n zoffset : float\n Offset in pixel units to use in the calculation of the psf\n\n cutval : float\n If not None, the psf will be cut along a curve corresponding to\n p(r) == 0 with exponential damping exp(-d^4)\n\n getextent : boolean\n If True, also return the extent of the psf in pixels for example\n to get the support size. Can only be used with cutval.\n \"\"\"\n # calculate the current pixel value in 1/k, making sure we are above the slab\n zint = max(self._p2k(self._tz(zint)), 0)\n offset = np.array([zoffset*(zint>0), 0, 0])\n scale = [self.param_dict[self.zscale], 1.0, 1.0]\n\n # create the coordinate vectors for where to actually calculate the\n tile = util.Tile(left=0, size=size, centered=True)\n vecs = tile.coords(form='flat')\n vecs = [self._p2k(s*i+o) for i,s,o in zip(vecs, scale, offset)]\n\n psf = self.psffunc(*vecs[::-1], zint=zint, **self.pack_args()).T\n vec = tile.coords(form='meshed')\n\n # create a smoothly varying point spread function by cutting off the psf\n # at a certain value and smoothly taking it to zero\n if self.cutoffval is not None and not self.cutbyval:\n # find the edges of the PSF\n edge = psf > psf.max() * self.cutoffval\n dd = nd.morphology.distance_transform_edt(~edge)\n\n # calculate the new PSF and normalize it to the new support\n psf = psf * np.exp(-dd**4)\n psf /= psf.sum()\n\n if getextent:\n # the size is determined by the edge plus a 2 pad for the\n # exponential damping to zero at the edge\n size = np.array([\n (vec*edge).min(axis=(1,2,3))-2,\n (vec*edge).max(axis=(1,2,3))+2,\n ]).T\n return psf, vec, size\n return psf, vec\n\n # perform a cut by value instead\n if self.cutoffval is not None and self.cutbyval:\n cutval = self.cutoffval * psf.max()\n\n dd = (psf - cutval) / cutval\n dd[dd > 0] = 0.\n\n # calculate the new PSF and normalize it to the new support\n psf = psf * np.exp(-(dd / self.cutfallrate)**4)\n psf /= psf.sum()\n\n # let the small values determine the edges\n edge = psf > cutval * self.cutedgeval\n if getextent:\n # the size is determined by the edge plus a 2 pad for the\n # exponential damping to zero at the edge\n size = np.array([\n (vec*edge).min(axis=(1,2,3))-2,\n (vec*edge).max(axis=(1,2,3))+2,\n ]).T\n return psf, vec, size\n return psf, vec\n\n return psf, vec"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting from pixel to k_incoming units", "response": "def _p2k(self, v):\n \"\"\" Convert from pixel to 1/k_incoming (laser_wavelength/(2\\pi)) units \"\"\"\n return 2*np.pi*self.pxsize*v/self.param_dict['psf-laser-wavelength']"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntransforming z to real - space coordinates from tile coordinates", "response": "def _tz(self, z):\n \"\"\" Transform z to real-space coordinates from tile coordinates \"\"\"\n return (z-self.param_dict['psf-zslab'])*self.param_dict[self.zscale]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmeasuring the size of the psf in each direction a particular z", "response": "def measure_size_drift(self, z, size=31, zoffset=0.):\n \"\"\" Returns the 'size' of the psf in each direction a particular z (px) \"\"\"\n drift = 0.0\n for i in range(self.measurement_iterations):\n psf, vec = self.psf_slice(z, size=size, zoffset=zoffset+drift)\n psf = psf / psf.sum()\n\n drift += moment(psf, vec[0], order=1)\n psize = [moment(psf, j, order=2) for j in vec]\n return np.array(psize), drift"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncharacterizes the current set of parameters", "response": "def characterize_psf(self):\n \"\"\" Get support size and drift polynomial for current set of params \"\"\"\n # there may be an issue with the support and characterization--\n # it might be best to do the characterization with the same support\n # as the calculated psf.\n l,u = max(self.zrange[0], self.param_dict['psf-zslab']), self.zrange[1]\n\n size_l, drift_l = self.measure_size_drift(l)\n size_u, drift_u = self.measure_size_drift(u)\n\n # must be odd for now or have a better system for getting the center\n self.support = util.oddify(2*self.support_factor*size_u.astype('int'))\n self.drift_poly = np.polyfit([l, u], [drift_l, drift_u], 1)\n\n if self.cutoffval is not None:\n psf, vec, size_l = self.psf_slice(l, size=51, zoffset=drift_l, getextent=True)\n psf, vec, size_u = self.psf_slice(u, size=51, zoffset=drift_u, getextent=True)\n\n ss = [np.abs(i).sum(axis=-1) for i in [size_l, size_u]]\n self.support = util.oddify(util.amax(*ss))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _kpad(self, field, finalshape, zpad=False, norm=True):\n currshape = np.array(field.shape)\n\n if any(finalshape < currshape):\n raise IndexError(\"PSF tile size is less than minimum support size\")\n\n d = finalshape - currshape\n\n # fix off-by-one issues when going odd to even tile sizes\n o = d % 2\n d = np.floor_divide(d, 2)\n\n if not zpad:\n o[0] = 0\n\n axes = None\n pad = tuple((d[i]+o[i],d[i]) for i in [0,1,2])\n rpsf = np.pad(field, pad, mode='constant', constant_values=0)\n rpsf = np.fft.ifftshift(rpsf, axes=axes)\n kpsf = fft.rfftn(rpsf, **fftkwargs)\n\n if norm:\n kpsf /= kpsf[0,0,0]\n return kpsf", "response": "This function will take a field and pad it with zeros until it has size finalshape."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\npacking the parameters into the form necessary for the integration routines above.", "response": "def pack_args(self):\n \"\"\"\n Pack the parameters into the form necessary for the integration\n routines above. For example, packs for calculate_linescan_psf\n \"\"\"\n mapper = {\n 'psf-kfki': 'kfki',\n 'psf-alpha': 'alpha',\n 'psf-n2n1': 'n2n1',\n 'psf-sigkf': 'sigkf',\n 'psf-sph6-ab': 'sph6_ab',\n 'psf-laser-wavelength': 'laser_wavelength',\n 'psf-pinhole-width': 'pinhole_width'\n }\n bads = [self.zscale, 'psf-zslab']\n\n d = {}\n for k,v in iteritems(mapper):\n if k in self.param_dict:\n d[v] = self.param_dict[k]\n\n d.update({\n 'polar_angle': self.polar_angle,\n 'normalize': self.normalize,\n 'include_K3_det':self.use_J1\n })\n\n if self.polychromatic:\n d.update({'nkpts': self.nkpts})\n d.update({'k_dist': self.k_dist})\n\n if self.do_pinhole:\n d.update({'nlpts': self.num_line_pts})\n\n d.update({'use_laggauss': True})\n return d"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef psffunc(self, *args, **kwargs):\n if self.polychromatic:\n func = psfcalc.calculate_polychrome_linescan_psf\n else:\n func = psfcalc.calculate_linescan_psf\n return func(*args, **kwargs)", "response": "Calculates a linescan psf"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef psffunc(self, x, y, z, **kwargs):\n #do_pinhole?? FIXME\n if self.polychromatic:\n func = psfcalc.calculate_polychrome_pinhole_psf\n else:\n func = psfcalc.calculate_pinhole_psf\n x0, y0 = [psfcalc.vec_to_halfvec(v) for v in [x,y]]\n vls = psfcalc.wrap_and_calc_psf(x0, y0, z, func, **kwargs)\n return vls / vls.sum()", "response": "Calculates a pinhole psf"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncharacterize the psf of the current set of parameters.", "response": "def characterize_psf(self):\n \"\"\" Get support size and drift polynomial for current set of params \"\"\"\n l,u = max(self.zrange[0], self.param_dict['psf-zslab']), self.zrange[1]\n\n size_l, drift_l = self.measure_size_drift(l, size=self.support)\n size_u, drift_u = self.measure_size_drift(u, size=self.support)\n\n self.drift_poly = np.polyfit([l, u], [drift_l, drift_u], 1)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _req(self, url, method='GET', **kw):\n '''Make request and convert JSON response to python objects'''\n send = requests.post if method == 'POST' else requests.get\n try:\n r = send(\n url,\n headers=self._token_header(),\n timeout=self.settings['timeout'],\n **kw)\n except requests.exceptions.Timeout:\n raise ApiError('Request timed out (%s seconds)' % self.settings['timeout'])\n try:\n json = r.json()\n except ValueError:\n raise ApiError('Received not JSON response from API')\n if json.get('status') != 'ok':\n raise ApiError('API error: received unexpected json from API: %s' % json)\n return json", "response": "Make request and convert JSON response to python objects"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_active_bets(self, project_id=None):\n '''Returns all active bets'''\n url = urljoin(\n self.settings['bets_url'],\n 'bets?state=fresh,active,accept_end&page=1&page_size=100')\n\n if project_id is not None:\n url += '&kava_project_id={}'.format(project_id)\n\n bets = []\n has_next_page = True\n while has_next_page:\n res = self._req(url)\n bets.extend(res['bets']['results'])\n url = res['bets'].get('next')\n has_next_page = bool(url)\n\n return bets", "response": "Returns all active bets"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget all bets in a given type and state.", "response": "def get_bets(self, type=None, order_by=None, state=None, project_id=None,\n page=None, page_size=None):\n \"\"\"Return bets with given filters and ordering.\n\n :param type: return bets only with this type.\n Use None to include all (default).\n :param order_by: '-last_stake' or 'last_stake' to sort by stake's\n created date or None for default ordering.\n :param state: one of 'active', 'closed', 'all' (default 'active').\n :param project_id: return bets associated with given project id in kava\n :param page: default 1.\n :param page_site: page size (default 100).\n \"\"\"\n if page is None:\n page = 1\n if page_size is None:\n page_size = 100\n if state == 'all':\n _states = [] # all states == no filter\n elif state == 'closed':\n _states = self.CLOSED_STATES\n else:\n _states = self.ACTIVE_STATES\n\n url = urljoin(\n self.settings['bets_url'],\n 'bets?page={}&page_size={}'.format(page, page_size))\n url += '&state={}'.format(','.join(_states))\n if type is not None:\n url += '&type={}'.format(type)\n if order_by in ['-last_stake', 'last_stake']:\n url += '&order_by={}'.format(order_by)\n if project_id is not None:\n url += '&kava_project_id={}'.format(project_id)\n\n res = self._req(url)\n return res['bets']['results']"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn slug of a project that given bet is associated with", "response": "def get_project_slug(self, bet):\n '''Return slug of a project that given bet is associated with\n or None if bet is not associated with any project.\n '''\n if bet.get('form_params'):\n params = json.loads(bet['form_params'])\n return params.get('project')\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef subscribe(self, event, bet_ids):\n '''Subscribe to event for given bet ids.'''\n if not self._subscriptions.get(event):\n self._subscriptions[event] = set()\n self._subscriptions[event] = self._subscriptions[event].union(bet_ids)", "response": "Subscribe to event for given bet ids."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef preview(context):\n\n config = context.obj\n\n pelican(config, '--verbose', '--ignore-cache')\n\n server_proc = None\n os.chdir(config['OUTPUT_DIR'])\n try:\n try:\n command = 'python -m http.server ' + str(PORT)\n server_proc = run(command, bg=True)\n\n time.sleep(3)\n click.launch('http://localhost:8000')\n\n time.sleep(5)\n pelican(config, '--autoreload')\n except Exception:\n if server_proc is not None:\n server_proc.kill()\n raise\n except KeyboardInterrupt:\n abort(context)", "response": "Opens local preview of your blog website"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_collection_endpoint(cls):\n return cls.Meta.collection_endpoint if cls.Meta.collection_endpoint is not None else cls.__name__.lower() + \"s/\"", "response": "Get the relative path to the API resource collection\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send(self, url, http_method, **client_args):\n return self.client.send(url, http_method, **client_args)", "response": "Send the request to the API"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, resource_id):\n response = self.send(self.get_resource_endpoint(resource_id), \"get\")\n\n try:\n resource = self.resource_class(self.client)\n except (ValueError, TypeError):\n return None\n else:\n resource.update_from_dict(self.client.get_response_data(response, self.Meta.parse_json))\n return resource", "response": "Get one single resource from the API\n endpoint."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef filter(self, **search_args):\n search_args = search_args or {}\n raw_resources = []\n\n for url, paginator_params in self.paginator.get_urls(self.get_collection_endpoint()):\n search_args.update(paginator_params)\n response = self.paginator.process_response(self.send(url, \"get\", params=search_args))\n raw_resources += self.client.get_response_data(response, self.Meta.parse_json)[self.json_collection_attribute] if self.json_collection_attribute is not None else self.client.get_response_data(response, self.Meta.parse_json)\n\n resources = []\n\n for raw_resource in raw_resources:\n try:\n resource = self.resource_class(self.client)\n except (ValueError, TypeError):\n continue\n else:\n resource.update_from_dict(raw_resource)\n resources.append(resource)\n\n return resources", "response": "Get a filtered list of resources by name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create(self, **kwargs):\n resource = self.resource_class(self.client)\n resource.update_from_dict(kwargs)\n resource.save(force_create=True)\n\n return resource", "response": "Create a resource on the server\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend a request to the base url with the given http_method and optional kargs.", "response": "def send(self, relative_path, http_method, **requests_args):\n \"\"\"\n Subclasses must implement this method, that will be used to send API requests with proper auth\n :param relative_path: URL path relative to self.base_url\n :param http_method: HTTP method\n :param requests_args: kargs to be sent to requests\n :return:\n \"\"\"\n url = urljoin(self.base_url, relative_path)\n\n return self.session.request(http_method, url, **requests_args)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_response_data(self, response, parse_json=True):\n if response.status_code in (requests.codes.ok, requests.codes.created):\n if parse_json:\n return response.json()\n return response.content\n elif response.status_code == requests.codes.bad_request:\n response_json = response.json()\n raise BadRequestException(response_json.get(\"error\", False) or response_json.get(\"errors\",\n _(\"Bad Request: {text}\").format(text=response.text)))\n elif response.status_code == requests.codes.not_found:\n raise NotFoundException(_(\"Resource not found: {url}\").format(url=response.url))\n elif response.status_code == requests.codes.internal_server_error:\n raise ServerErrorException(_(\"Internal server error\"))\n elif response.status_code in (requests.codes.unauthorized, requests.codes.forbidden):\n raise AuthErrorException(_(\"Access denied\"))\n elif response.status_code == requests.codes.too_many_requests:\n raise RateLimitException(_(response.text))\n else:\n raise ServerErrorException(_(\"Unknown error occurred\"))", "response": "Get response data or throw an appropiate exception\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send(self, relative_path, http_method, **requests_args):\n if http_method != \"get\":\n warnings.warn(_(\"You are using methods other than get with no authentication!!!\"))\n\n return super(NoAuthClient, self).send(relative_path, http_method, **requests_args)", "response": "Make a unauthorized request to the base_url"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting a new article to the content directory", "response": "def write(context):\n \"\"\"Starts a new article\"\"\"\n\n config = context.obj\n\n title = click.prompt('Title')\n author = click.prompt('Author', default=config.get('DEFAULT_AUTHOR'))\n\n slug = slugify(title)\n creation_date = datetime.now()\n basename = '{:%Y-%m-%d}_{}.md'.format(creation_date, slug)\n meta = (\n ('Title', title),\n ('Date', '{:%Y-%m-%d %H:%M}:00'.format(creation_date)),\n ('Modified', '{:%Y-%m-%d %H:%M}:00'.format(creation_date)),\n ('Author', author),\n )\n\n file_content = ''\n for key, value in meta:\n file_content += '{}: {}\\n'.format(key, value)\n file_content += '\\n\\n'\n file_content += 'Text...\\n\\n'\n file_content += '![image description]({filename}/images/my-photo.jpg)\\n\\n'\n file_content += 'Text...\\n\\n'\n\n os.makedirs(config['CONTENT_DIR'], exist_ok=True)\n path = os.path.join(config['CONTENT_DIR'], basename)\n with click.open_file(path, 'w') as f:\n f.write(file_content)\n\n click.echo(path)\n click.launch(path)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlook for errors in source code of your blog", "response": "def lint(context):\n \"\"\"Looks for errors in source code of your blog\"\"\"\n\n config = context.obj\n try:\n run('flake8 {dir} --exclude={exclude}'.format(\n dir=config['CWD'],\n exclude=','.join(EXCLUDE),\n ))\n except SubprocessError:\n context.exit(1)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the real value class of the resource.", "response": "def set_real_value_class(self):\n \"\"\"\n value_class is initially a string with the import path to the resource class, but we need to get the actual class before doing any work\n\n We do not expect the actual clas to be in value_class since the beginning to avoid nasty import egg-before-chicken errors\n \"\"\"\n if self.value_class is not None and isinstance(self.value_class, str):\n module_name, dot, class_name = self.value_class.rpartition(\".\")\n module = __import__(module_name, fromlist=[class_name])\n self.value_class = getattr(module, class_name)\n self._initialized = True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving changes and sends them to GitHub", "response": "def publish(context):\n \"\"\"Saves changes and sends them to GitHub\"\"\"\n\n header('Recording changes...')\n run('git add -A')\n\n header('Displaying changes...')\n run('git -c color.status=always status')\n\n if not click.confirm('\\nContinue publishing'):\n run('git reset HEAD --')\n abort(context)\n\n header('Saving changes...')\n try:\n run('git commit -m \"{message}\"'.format(\n message='Publishing {}'.format(choose_commit_emoji())\n ), capture=True)\n except subprocess.CalledProcessError as e:\n if 'nothing to commit' not in e.stdout:\n raise\n else:\n click.echo('Nothing to commit.')\n\n header('Pushing to GitHub...')\n branch = get_branch()\n run('git push origin {branch}:{branch}'.format(branch=branch))\n\n pr_link = get_pr_link(branch)\n if pr_link:\n click.launch(pr_link)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deploy(context):\n\n config = context.obj\n\n header('Generating HTML...')\n pelican(config, '--verbose', production=True)\n\n header('Removing unnecessary output...')\n unnecessary_paths = [\n 'author', 'category', 'tag', 'feeds', 'tags.html',\n 'authors.html', 'categories.html', 'archives.html',\n ]\n for path in unnecessary_paths:\n remove_path(os.path.join(config['OUTPUT_DIR'], path))\n\n if os.environ.get('TRAVIS'): # Travis CI\n header('Setting up Git...')\n run(\n 'git config user.name ' +\n run('git show --format=\"%cN\" -s', capture=True)\n )\n run(\n 'git config user.email ' +\n run('git show --format=\"%cE\" -s', capture=True)\n )\n\n github_token = os.environ.get('GITHUB_TOKEN')\n repo_slug = os.environ.get('TRAVIS_REPO_SLUG')\n origin = 'https://{}@github.com/{}.git'.format(github_token, repo_slug)\n run('git remote set-url origin ' + origin)\n\n header('Rewriting gh-pages branch...')\n run('ghp-import -m \"{message}\" {dir}'.format(\n message='Deploying {}'.format(choose_commit_emoji()),\n dir=config['OUTPUT_DIR'],\n ))\n\n header('Pushing to GitHub...')\n run('git push origin gh-pages:gh-pages --force')", "response": "Deploy the blog website to the GitHub repository"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the given number as a string with a sign in front of it ie. + - otherwise.", "response": "def signed_number(number, precision=2):\n \"\"\"\n Return the given number as a string with a sign in front of it, ie. `+` if the number is positive, `-` otherwise.\n \"\"\"\n prefix = '' if number <= 0 else '+'\n number_str = '{}{:.{precision}f}'.format(prefix, number, precision=precision)\n\n return number_str"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nshow Zebra balance. Like the hours balance, vacation left, etc.", "response": "def balance(ctx):\n \"\"\"\n Show Zebra balance.\n\n Like the hours balance, vacation left, etc.\n \"\"\"\n backend = plugins_registry.get_backends_by_class(ZebraBackend)[0]\n\n timesheet_collection = get_timesheet_collection_for_context(ctx, None)\n hours_to_be_pushed = timesheet_collection.get_hours(pushed=False, ignored=False, unmapped=False)\n\n today = datetime.date.today()\n user_info = backend.get_user_info()\n timesheets = backend.get_timesheets(get_first_dow(today), get_last_dow(today))\n total_duration = sum([float(timesheet['time']) for timesheet in timesheets])\n\n vacation = hours_to_days(user_info['vacation']['difference'])\n vacation_balance = '{} days, {:.2f} hours'.format(*vacation)\n\n hours_balance = user_info['hours']['hours']['balance']\n\n click.echo(\"Hours balance: {}\".format(signed_number(hours_balance)))\n click.echo(\"Hours balance after push: {}\".format(signed_number(hours_balance + hours_to_be_pushed)))\n click.echo(\"Hours done this week: {:.2f}\".format(total_duration))\n click.echo(\"Vacation left: {}\".format(vacation_balance))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_zebra_params(params):\n def to_zebra_value(value):\n transform_funcs = {\n bool: lambda v: 'true' if v else 'false',\n }\n\n return transform_funcs.get(type(value), lambda v: v)(value)\n\n return {param: to_zebra_value(value) for param, value in params.items()}", "response": "Transforms the given params dict to values that are understood by Zebra"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nshows all messages in the messages key of the given dict.", "response": "def show_response_messages(response_json):\n \"\"\"\n Show all messages in the `messages` key of the given dict.\n \"\"\"\n message_type_kwargs = {\n 'warning': {'fg': 'yellow'},\n 'error': {'fg': 'red'},\n }\n for message in response_json.get('messages', []):\n click.secho(message['text'], **message_type_kwargs.get(message['type'], {}))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the mapping for an alias.", "response": "def update_alias_mapping(settings, alias, new_mapping):\n \"\"\"\n Override `alias` mapping in the user configuration file with the given `new_mapping`, which should be a tuple with\n 2 or 3 elements (in the form `(project_id, activity_id, role_id)`).\n \"\"\"\n mapping = aliases_database[alias]\n new_mapping = Mapping(mapping=new_mapping, backend=mapping.backend)\n aliases_database[alias] = new_mapping\n settings.add_alias(alias, new_mapping)\n settings.write_config()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef photos(context, path):\n\n config = context.obj\n\n header('Looking for the latest article...')\n article_filename = find_last_article(config['CONTENT_DIR'])\n if not article_filename:\n return click.secho('No articles.', fg='red')\n click.echo(os.path.basename(article_filename))\n\n header('Looking for images...')\n images = list(sorted(find_images(path)))\n if not images:\n return click.secho('Found no images.', fg='red')\n\n for filename in images:\n click.secho(filename, fg='green')\n\n if not click.confirm('\\nAdd these images to the latest article'):\n abort(config)\n\n url_prefix = os.path.join('{filename}', IMAGES_PATH)\n images_dir = os.path.join(config['CONTENT_DIR'], IMAGES_PATH)\n os.makedirs(images_dir, exist_ok=True)\n\n header('Processing images...')\n urls = []\n for filename in images:\n image_basename = os.path.basename(filename).replace(' ', '-').lower()\n urls.append(os.path.join(url_prefix, image_basename))\n image_filename = os.path.join(images_dir, image_basename)\n print(filename, image_filename)\n import_image(filename, image_filename)\n\n content = '\\n'\n for url in urls:\n url = url.replace('\\\\', '/')\n content += '\\n![image description]({})\\n'.format(url)\n\n header('Adding to article: {}'.format(article_filename))\n with click.open_file(article_filename, 'a') as f:\n f.write(content)\n click.launch(article_filename)", "response": "Adds images to the latest article"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_node(self, string_key):\n pos = self.get_node_pos(string_key)\n if pos is None:\n return None\n return self.ring[self._sorted_keys[pos]]", "response": "Given a string key returns the corresponding node in the hash ring."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_node_pos(self, string_key):\n if not self.ring:\n return None\n\n key = self.gen_key(string_key)\n\n nodes = self._sorted_keys\n pos = bisect(nodes, key)\n\n if pos == len(nodes):\n return 0\n else:\n return pos", "response": "Given a string key returns the node in the hash ring along with its position in the ring."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a string key it returns the nodes as a generator that can hold the key.", "response": "def iterate_nodes(self, string_key, distinct=True):\n \"\"\"Given a string key it returns the nodes as a generator that can hold the key.\n\n The generator iterates one time through the ring\n starting at the correct position.\n\n if `distinct` is set, then the nodes returned will be unique,\n i.e. no virtual copies will be returned.\n \"\"\"\n if not self.ring:\n yield None, None\n\n returned_values = set()\n\n def distinct_filter(value):\n if str(value) not in returned_values:\n returned_values.add(str(value))\n return value\n\n pos = self.get_node_pos(string_key)\n for key in self._sorted_keys[pos:]:\n val = distinct_filter(self.ring[key])\n if val:\n yield val\n\n for i, key in enumerate(self._sorted_keys):\n if i < pos:\n val = distinct_filter(self.ring[key])\n if val:\n yield val"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a string key it returns a long value", "response": "def gen_key(self, key):\n \"\"\"Given a string key it returns a long value,\n this long value represents a place on the hash ring.\n\n md5 is currently used because it mixes well.\n \"\"\"\n b_key = self._hash_digest(key)\n return self._hash_val(b_key, lambda x: x)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_number_of_app_ports(app):\n mode = _get_networking_mode(app)\n ports_list = None\n if mode == 'host':\n ports_list = _get_port_definitions(app)\n elif mode == 'container/bridge':\n ports_list = _get_port_definitions(app)\n if ports_list is None:\n ports_list = _get_container_port_mappings(app)\n elif mode == 'container':\n ports_list = _get_ip_address_discovery_ports(app)\n # Marathon 1.5+: the ipAddress field is missing -> ports_list is None\n # Marathon <1.5: the ipAddress field can be present, but ports_list can\n # still be empty while the container port mapping is not :-/\n if not ports_list:\n ports_list = _get_container_port_mappings(app)\n else:\n raise RuntimeError(\n \"Unknown Marathon networking mode '{}'\".format(mode))\n\n return len(ports_list)", "response": "Get the number of ports for the given app JSON."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_networking_mode(app):\n # Marathon 1.5+: there is a `networks` field\n networks = app.get('networks')\n if networks:\n # Modes cannot be mixed, so assigning the last mode is fine\n return networks[-1].get('mode', 'container')\n\n # Older Marathon: determine equivalent network mode\n container = app.get('container')\n if container is not None and 'docker' in container:\n docker_network = container['docker'].get('network')\n if docker_network == 'USER':\n return 'container'\n elif docker_network == 'BRIDGE':\n return 'container/bridge'\n\n return 'container' if _is_legacy_ip_per_task(app) else 'host'", "response": "Get the Marathon networking mode for the app."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_container_port_mappings(app):\n container = app['container']\n\n # Marathon 1.5+: container.portMappings field\n port_mappings = container.get('portMappings')\n\n # Older Marathon: container.docker.portMappings field\n if port_mappings is None and 'docker' in container:\n port_mappings = container['docker'].get('portMappings')\n\n return port_mappings", "response": "Get the portMappings field for the app container."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a list of PEM objects sort them into the private key leaf certificate and list of CA certificates.", "response": "def sort_pem_objects(pem_objects):\n \"\"\"\n Given a list of pem objects, sort the objects into the private key, leaf\n certificate, and list of CA certificates in the trust chain. This function\n assumes that the list of pem objects will contain exactly one private key\n and exactly one leaf certificate and that only key and certificate type\n objects are provided.\n \"\"\"\n keys, certs, ca_certs = [], [], []\n for pem_object in pem_objects:\n if isinstance(pem_object, pem.Key):\n keys.append(pem_object)\n else:\n # This assumes all pem objects provided are either of type pem.Key\n # or pem.Certificate. Technically, there are CSR and CRL types, but\n # we should never be passed those.\n if _is_ca(pem_object):\n ca_certs.append(pem_object)\n else:\n certs.append(pem_object)\n\n [key], [cert] = keys, certs\n return key, cert, ca_certs"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a non - None response from Vault key value store convert the key - value pairs into PEM objects.", "response": "def _cert_data_to_pem_objects(cert_data):\n \"\"\"\n Given a non-None response from the Vault key/value store, convert the\n key/values into a list of PEM objects.\n \"\"\"\n pem_objects = []\n for key in ['privkey', 'cert', 'chain']:\n pem_objects.extend(pem.parse(cert_data[key].encode('utf-8')))\n\n return pem_objects"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef raise_for_not_ok_status(response):\n if response.code != OK:\n raise HTTPError('Non-200 response code (%s) for url: %s' % (\n response.code, uridecode(response.request.absoluteURI)))\n\n return response", "response": "Raises an exception if the response is not 200."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _sse_content_with_protocol(response, handler, **sse_kwargs):\n protocol = SseProtocol(handler, **sse_kwargs)\n finished = protocol.when_finished()\n\n response.deliverBody(protocol)\n\n return finished, protocol", "response": "This function is used to send the Sse content to the response."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _request(self, failure, endpoints, *args, **kwargs):\n # We've run out of endpoints, fail\n if not endpoints:\n return failure\n\n endpoint = endpoints.pop(0)\n d = super(MarathonClient, self).request(*args, url=endpoint, **kwargs)\n\n # If something goes wrong, call ourselves again with the remaining\n # endpoints\n d.addErrback(self._request, endpoints, *args, **kwargs)\n return d", "response": "A wrapper for the base class method _request."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nperforming a GET request and get the contents of the JSON response. Marathon's JSON responses tend to contain an object with a single key which points to the actual data of the response. For example /v2/apps returns something like {\"apps\": [ {\"app1\"}, {\"app2\"} ]}. We're interested in the contents of \"apps\". This method will raise an error if: * There is an error response code * The field with the given name cannot be found", "response": "def get_json_field(self, field, **kwargs):\n \"\"\"\n Perform a GET request and get the contents of the JSON response.\n\n Marathon's JSON responses tend to contain an object with a single key\n which points to the actual data of the response. For example /v2/apps\n returns something like {\"apps\": [ {\"app1\"}, {\"app2\"} ]}. We're\n interested in the contents of \"apps\".\n\n This method will raise an error if:\n * There is an error response code\n * The field with the given name cannot be found\n \"\"\"\n d = self.request(\n 'GET', headers={'Accept': 'application/json'}, **kwargs)\n d.addCallback(raise_for_status)\n d.addCallback(raise_for_header, 'Content-Type', 'application/json')\n d.addCallback(json_content)\n d.addCallback(self._get_json_field, field)\n return d"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_json_field(self, response_json, field_name):\n if field_name not in response_json:\n raise KeyError('Unable to get value for \"%s\" from Marathon '\n 'response: \"%s\"' % (\n field_name, json.dumps(response_json),))\n\n return response_json[field_name]", "response": "Get a JSON field from the response."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_events(self, callbacks):\n d = self.request(\n 'GET', path='/v2/events', unbuffered=True,\n # The event_type parameter was added in Marathon 1.3.7. It can be\n # used to specify which event types we are interested in. On older\n # versions of Marathon it is ignored, and we ignore events we're\n # not interested in anyway.\n params={'event_type': sorted(callbacks.keys())},\n headers={\n 'Accept': 'text/event-stream',\n 'Cache-Control': 'no-store'\n })\n\n def handler(event, data):\n callback = callbacks.get(event)\n # Deserialize JSON if a callback is present\n if callback is not None:\n callback(json.loads(data))\n\n return d.addCallback(\n sse_content, handler, reactor=self._reactor, **self._sse_kwargs)", "response": "Get events from Marathon."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse(\n self,\n value: str,\n type_: typing.Type[typing.Any] = str,\n subtype: typing.Type[typing.Any] = str,\n ) -> typing.Any:\n \"\"\"\n Parse value from string.\n\n Convert :code:`value` to\n\n .. code-block:: python\n\n >>> parser = Config()\n >>> parser.parse('12345', type_=int)\n <<< 12345\n >>>\n >>> parser.parse('1,2,3,4', type_=list, subtype=int)\n <<< [1, 2, 3, 4]\n\n :param value: string\n :param type\\\\_: the type to return\n :param subtype: subtype for iterator types\n :return: the parsed config value\n\n \"\"\"\n if type_ is bool:\n return type_(value.lower() in self.TRUE_STRINGS)\n\n try:\n if isinstance(type_, type) and issubclass(\n type_, (list, tuple, set, frozenset)\n ):\n return type_(\n self.parse(v.strip(\" \"), subtype)\n for v in value.split(\",\")\n if value.strip(\" \")\n )\n\n return type_(value)\n except ValueError as e:\n raise ConfigError(*e.args)", "response": "Parse a string value from a config file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse a value from an environment variable.", "response": "def get(\n self,\n key: str,\n default: typing.Any = UNSET,\n type_: typing.Type[typing.Any] = str,\n subtype: typing.Type[typing.Any] = str,\n mapper: typing.Optional[typing.Callable[[object], object]] = None,\n ) -> typing.Any:\n \"\"\"\n Parse a value from an environment variable.\n\n .. code-block:: python\n\n >>> os.environ['FOO']\n <<< '12345'\n >>>\n >>> os.environ['BAR']\n <<< '1,2,3,4'\n >>>\n >>> 'BAZ' in os.environ\n <<< False\n >>>\n >>> parser = Config()\n >>> parser.get('FOO', type_=int)\n <<< 12345\n >>>\n >>> parser.get('BAR', type_=list, subtype=int)\n <<< [1, 2, 3, 4]\n >>>\n >>> parser.get('BAZ', default='abc123')\n <<< 'abc123'\n >>>\n >>> parser.get('FOO', type_=int, mapper=lambda x: x*10)\n <<< 123450\n\n :param key: the key to look up the value under\n :param default: default value to return when when no value is present\n :param type\\\\_: the type to return\n :param subtype: subtype for iterator types\n :param mapper: a function to post-process the value with\n :return: the parsed config value\n\n \"\"\"\n value = self.environ.get(key, UNSET)\n\n if value is UNSET and default is UNSET:\n raise ConfigError(\"Unknown environment variable: {0}\".format(key))\n\n if value is UNSET:\n value = default\n else:\n value = self.parse(typing.cast(str, value), type_, subtype)\n\n if mapper:\n value = mapper(value)\n\n return value"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nperforms a request to a specific endpoint. Raise an error if the status is not OK.", "response": "def _request(self, endpoint, *args, **kwargs):\n \"\"\"\n Perform a request to a specific endpoint. Raise an error if the status\n code indicates a client or server error.\n \"\"\"\n kwargs['url'] = endpoint\n return (super(MarathonLbClient, self).request(*args, **kwargs)\n .addCallback(raise_for_status))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _check_request_results(self, results):\n responses = []\n failed_endpoints = []\n for index, result_tuple in enumerate(results):\n success, result = result_tuple\n if success:\n responses.append(result)\n else:\n endpoint = self.endpoints[index]\n self.log.failure(\n 'Failed to make a request to a marathon-lb instance: '\n '{endpoint}', result, LogLevel.error, endpoint=endpoint)\n responses.append(None)\n failed_endpoints.append(endpoint)\n\n if len(failed_endpoints) == len(self.endpoints):\n raise RuntimeError(\n 'Failed to make a request to all marathon-lb instances')\n\n if failed_endpoints:\n self.log.error(\n 'Failed to make a request to {x}/{y} marathon-lb instances: '\n '{endpoints}', x=len(failed_endpoints), y=len(self.endpoints),\n endpoints=failed_endpoints)\n\n return responses", "response": "Check the result of each request that we made and return a list of responses with a None value for any requests that failed."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a client key if one does not exist already.", "response": "def maybe_key(pem_path):\n \"\"\"\n Set up a client key if one does not exist already.\n\n https://gist.github.com/glyph/27867a478bb71d8b6046fbfb176e1a33#file-local-certs-py-L32-L50\n\n :type pem_path: twisted.python.filepath.FilePath\n :param pem_path:\n The path to the certificate directory to use.\n :rtype: twisted.internet.defer.Deferred\n \"\"\"\n acme_key_file = pem_path.child(u'client.key')\n if acme_key_file.exists():\n key = _load_pem_private_key_bytes(acme_key_file.getContent())\n else:\n key = generate_private_key(u'rsa')\n acme_key_file.setContent(_dump_pem_private_key_bytes(key))\n return succeed(JWKRSA(key=key))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a client key in Vault if one does not exist already.", "response": "def maybe_key_vault(client, mount_path):\n \"\"\"\n Set up a client key in Vault if one does not exist already.\n\n :param client:\n The Vault API client to use.\n :param mount_path:\n The Vault key/value mount path to use.\n :rtype: twisted.internet.defer.Deferred\n \"\"\"\n d = client.read_kv2('client_key', mount_path=mount_path)\n\n def get_or_create_key(client_key):\n if client_key is not None:\n key_data = client_key['data']['data']\n key = _load_pem_private_key_bytes(key_data['key'].encode('utf-8'))\n return JWKRSA(key=key)\n else:\n key = generate_private_key(u'rsa')\n key_data = {\n 'key': _dump_pem_private_key_bytes(key).decode('utf-8')\n }\n d = client.create_or_update_kv2(\n 'client_key', key_data, mount_path=mount_path)\n\n return d.addCallback(lambda _result: JWKRSA(key=key))\n\n return d.addCallback(get_or_create_key)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_txacme_client_creator(key, reactor, url, alg=RS256):\n # Creating an Agent without specifying a pool gives us the default pool\n # which is non-persistent.\n jws_client = JWSClient(HTTPClient(agent=Agent(reactor)), key, alg)\n\n return partial(txacme_Client.from_url, reactor, url, key, alg, jws_client)", "response": "Create a creator for txacme clients."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a wildcard certificate valid for otope 10 years.", "response": "def generate_wildcard_pem_bytes():\n \"\"\"\n Generate a wildcard (subject name '*') self-signed certificate valid for\n 10 years.\n\n https://cryptography.io/en/latest/x509/tutorial/#creating-a-self-signed-certificate\n\n :return: Bytes representation of the PEM certificate data\n \"\"\"\n key = generate_private_key(u'rsa')\n name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, u'*')])\n cert = (\n x509.CertificateBuilder()\n .issuer_name(name)\n .subject_name(name)\n .not_valid_before(datetime.today() - timedelta(days=1))\n .not_valid_after(datetime.now() + timedelta(days=3650))\n .serial_number(int(uuid.uuid4()))\n .public_key(key.public_key())\n .sign(\n private_key=key,\n algorithm=hashes.SHA256(),\n backend=default_backend())\n )\n\n return b''.join((\n key.private_bytes(\n encoding=serialization.Encoding.PEM,\n format=serialization.PrivateFormat.TraditionalOpenSSL,\n encryption_algorithm=serialization.NoEncryption()),\n cert.public_bytes(serialization.Encoding.PEM)\n ))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_env(cls, reactor=None, env=os.environ):\n address = env.get('VAULT_ADDR', 'https://127.0.0.1:8200')\n # This seems to be what the Vault CLI defaults to\n token = env.get('VAULT_TOKEN', 'TEST')\n\n ca_cert = env.get('VAULT_CACERT')\n tls_server_name = env.get('VAULT_TLS_SERVER_NAME')\n client_cert = env.get('VAULT_CLIENT_CERT')\n client_key = env.get('VAULT_CLIENT_KEY')\n cf = ClientPolicyForHTTPS.from_pem_files(\n caKey=ca_cert, privateKey=client_key, certKey=client_cert,\n tls_server_name=tls_server_name\n )\n client, reactor = default_client(reactor, contextFactory=cf)\n\n return cls(address, token, client=client, reactor=reactor)", "response": "Create a new client with configuration from the environment variables."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading data from Vault. Returns the JSON - decoded response.", "response": "def read(self, path, **params):\n \"\"\"\n Read data from Vault. Returns the JSON-decoded response.\n \"\"\"\n d = self.request('GET', '/v1/' + path, params=params)\n return d.addCallback(self._handle_response)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef write(self, path, **data):\n d = self.request('PUT', '/v1/' + path, json=data)\n return d.addCallback(self._handle_response, check_cas=True)", "response": "Write data to Vault. Returns the JSON - decoded response."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads some data from a key value version 2 secret engine.", "response": "def read_kv2(self, path, version=None, mount_path='secret'):\n \"\"\"\n Read some data from a key/value version 2 secret engine.\n \"\"\"\n params = {}\n if version is not None:\n params['version'] = version\n\n read_path = '{}/data/{}'.format(mount_path, path)\n return self.read(read_path, **params)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_or_update_kv2(self, path, data, cas=None, mount_path='secret'):\n params = {\n 'options': {},\n 'data': data\n }\n if cas is not None:\n params['options']['cas'] = cas\n\n write_path = '{}/data/{}'.format(mount_path, path)\n return self.write(write_path, **params)", "response": "Create or update a key - value entry in a key - value version 2 secret engine."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a single value for the given key out of the given set of headers.", "response": "def get_single_header(headers, key):\n \"\"\"\n Get a single value for the given key out of the given set of headers.\n\n :param twisted.web.http_headers.Headers headers:\n The set of headers in which to look for the header value\n :param str key:\n The header key\n \"\"\"\n raw_headers = headers.getRawHeaders(key)\n if raw_headers is None:\n return None\n\n # Take the final header as the authorative\n header, _ = cgi.parse_header(raw_headers[-1])\n return header"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nraising an exception if the response did not succeed.", "response": "def raise_for_status(response):\n \"\"\"\n Raises a `requests.exceptions.HTTPError` if the response did not succeed.\n Adapted from the Requests library:\n https://github.com/kennethreitz/requests/blob/v2.8.1/requests/models.py#L825-L837\n \"\"\"\n http_error_msg = ''\n\n if 400 <= response.code < 500:\n http_error_msg = '%s Client Error for url: %s' % (\n response.code, uridecode(response.request.absoluteURI))\n\n elif 500 <= response.code < 600:\n http_error_msg = '%s Server Error for url: %s' % (\n response.code, uridecode(response.request.absoluteURI))\n\n if http_error_msg:\n raise HTTPError(http_error_msg, response=response)\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _compose_url(self, url, kwargs):\n if url is None:\n url = self.url\n\n if url is None:\n raise ValueError(\n 'url not provided and this client has no url attribute')\n\n split_result = urisplit(url)\n userinfo = split_result.userinfo\n\n # Build up the kwargs to pass to uricompose\n compose_kwargs = {}\n for key in ['scheme', 'host', 'port', 'path', 'fragment']:\n if key in kwargs:\n compose_kwargs[key] = kwargs.pop(key)\n else:\n compose_kwargs[key] = getattr(split_result, key)\n\n if 'params' in kwargs:\n compose_kwargs['query'] = kwargs.pop('params')\n else:\n compose_kwargs['query'] = split_result.query\n\n # Take the userinfo out of the URL and pass as 'auth' to treq so it can\n # be used for HTTP basic auth headers\n if 'auth' not in kwargs and userinfo is not None:\n # treq expects a 2-tuple (username, password)\n kwargs['auth'] = tuple(userinfo.split(':', 2))\n\n return uricompose(**compose_kwargs)", "response": "Compose a URL with the given kwargs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nperforming a request to the specified URL.", "response": "def request(self, method, url=None, **kwargs):\n \"\"\"\n Perform a request.\n\n :param: method:\n The HTTP method to use (example is `GET`).\n :param: url:\n The URL to use. The default value is the URL this client was\n created with (`self.url`) (example is `http://localhost:8080`)\n :param: kwargs:\n Any other parameters that will be passed to `treq.request`, for\n example headers. Or any URL parameters to override, for example\n path, query or fragment.\n \"\"\"\n url = self._compose_url(url, kwargs)\n\n kwargs.setdefault('timeout', self._timeout)\n\n d = self._client.request(method, url, reactor=self._reactor, **kwargs)\n\n d.addCallback(self._log_request_response, method, url, kwargs)\n d.addErrback(self._log_request_error, url)\n\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef listen(self, reactor, endpoint_description):\n endpoint = serverFromString(reactor, endpoint_description)\n return endpoint.listen(Site(self.app.resource()))", "response": "Start the server listening for requests on the given host and port."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef health(self, request):\n if self.health_handler is None:\n return self._no_health_handler(request)\n\n health = self.health_handler()\n response_code = OK if health.healthy else SERVICE_UNAVAILABLE\n request.setResponseCode(response_code)\n write_request_json(request, health.json_message)", "response": "Listens to incoming health checks from Marathon on / health."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_listen_addr(listen_addr):\n if ':' not in listen_addr:\n raise ValueError(\n \"'%s' does not have the correct form for a listen address: \"\n '[ipaddress]:port' % (listen_addr,))\n host, port = listen_addr.rsplit(':', 1)\n\n # Validate the host\n if host == '':\n protocol = 'tcp'\n interface = None\n else:\n if host.startswith('[') and host.endswith(']'): # IPv6 wrapped in []\n host = host[1:-1]\n ip_address = ipaddress.ip_address(_to_unicode(host))\n protocol = 'tcp6' if ip_address.version == 6 else 'tcp'\n interface = str(ip_address)\n\n # Validate the port\n if not port.isdigit() or int(port) < 1 or int(port) > 65535:\n raise ValueError(\n \"'%s' does not appear to be a valid port number\" % (port,))\n\n args = [protocol, port]\n kwargs = {'interface': interface} if interface is not None else {}\n\n return _create_tx_endpoints_string(args, kwargs)", "response": "Parses a listen address into a Twisted\n endpoint description string for use with with\n ."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_marathon_acme(\n client_creator, cert_store, acme_email, allow_multiple_certs,\n marathon_addrs, marathon_timeout, sse_timeout, mlb_addrs, group,\n reactor):\n \"\"\"\n Create a marathon-acme instance.\n\n :param client_creator:\n The txacme client creator function.\n :param cert_store:\n The txacme certificate store instance.\n :param acme_email:\n Email address to use when registering with the ACME service.\n :param allow_multiple_certs:\n Whether to allow multiple certificates per app port.\n :param marathon_addr:\n Address for the Marathon instance to find app domains that require\n certificates.\n :param marathon_timeout:\n Amount of time in seconds to wait for response headers to be received\n from Marathon.\n :param sse_timeout:\n Amount of time in seconds to wait for some event data to be received\n from Marathon.\n :param mlb_addrs:\n List of addresses for marathon-lb instances to reload when a new\n certificate is issued.\n :param group:\n The marathon-lb group (``HAPROXY_GROUP``) to consider when finding\n app domains.\n :param reactor: The reactor to use.\n \"\"\"\n marathon_client = MarathonClient(marathon_addrs, timeout=marathon_timeout,\n sse_kwargs={'timeout': sse_timeout},\n reactor=reactor)\n marathon_lb_client = MarathonLbClient(mlb_addrs, reactor=reactor)\n\n return MarathonAcme(\n marathon_client,\n group,\n cert_store,\n marathon_lb_client,\n client_creator,\n reactor,\n acme_email,\n allow_multiple_certs\n )", "response": "Create a Marathon - acme instance."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialise the storage directory with the certificates directory and the default wildcard self - signed certificate for HAProxy.", "response": "def init_storage_dir(storage_dir):\n \"\"\"\n Initialise the storage directory with the certificates directory and a\n default wildcard self-signed certificate for HAProxy.\n\n :return: the storage path and certs path\n \"\"\"\n storage_path = FilePath(storage_dir)\n\n # Create the default wildcard certificate if it doesn't already exist\n default_cert_path = storage_path.child('default.pem')\n if not default_cert_path.exists():\n default_cert_path.setContent(generate_wildcard_pem_bytes())\n\n # Create a directory for unmanaged certs. We don't touch this again, but it\n # needs to be there and it makes sense to create it at the same time as\n # everything else.\n unmanaged_certs_path = storage_path.child('unmanaged-certs')\n if not unmanaged_certs_path.exists():\n unmanaged_certs_path.createDirectory()\n\n # Store certificates in a directory inside the storage directory, so\n # HAProxy will read just the certificates there.\n certs_path = storage_path.child('certs')\n if not certs_path.exists():\n certs_path.createDirectory()\n\n return storage_path, certs_path"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialise the logging by adding an observer to the global log publisher.", "response": "def init_logging(log_level):\n \"\"\"\n Initialise the logging by adding an observer to the global log publisher.\n\n :param str log_level: The minimum log level to log messages for.\n \"\"\"\n log_level_filter = LogLevelFilterPredicate(\n LogLevel.levelWithName(log_level))\n log_level_filter.setLogLevelForNamespace(\n 'twisted.web.client._HTTP11ClientFactory', LogLevel.warn)\n log_observer = FilteringLogObserver(\n textFileLogObserver(sys.stdout), [log_level_filter])\n globalLogPublisher.addObserver(log_observer)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parse_field_value(line):\n if line.startswith(':'):\n # Ignore the line\n return None, None\n\n if ':' not in line:\n # Treat the entire line as the field, use empty string as value\n return line, ''\n\n # Else field is before the ':' and value is after\n field, value = line.split(':', 1)\n\n # If value starts with a space, remove it.\n value = value[1:] if value.startswith(' ') else value\n\n return field, value", "response": "Parse the field and value from a line."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _abortConnection(self):\n transport = self.transport\n if isinstance(transport, TransportProxyProducer):\n transport = transport._producer\n\n if hasattr(transport, 'abortConnection'):\n transport.abortConnection()\n else:\n self.log.error(\n 'Transport {} has no abortConnection method'.format(transport))", "response": "Aborts the connection to the specified object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncall by the protocol class when data is received.", "response": "def dataReceived(self, data):\n \"\"\"\n Translates bytes into lines, and calls lineReceived.\n\n Copied from ``twisted.protocols.basic.LineOnlyReceiver`` but using\n str.splitlines() to split on ``\\r\\n``, ``\\n``, and ``\\r``.\n \"\"\"\n self.resetTimeout()\n lines = (self._buffer + data).splitlines()\n\n # str.splitlines() doesn't split the string after a trailing newline\n # character so we must check if there is a trailing newline and, if so,\n # clear the buffer as the line is \"complete\". Else, the line is\n # incomplete and we keep the last line in the buffer.\n if data.endswith(b'\\n') or data.endswith(b'\\r'):\n self._buffer = b''\n else:\n self._buffer = lines.pop(-1)\n\n for line in lines:\n if self.transport.disconnecting:\n # this is necessary because the transport may be told to lose\n # the connection by a line within a larger packet, and it is\n # important to disregard all the lines in that packet following\n # the one that told it to close.\n return\n if len(line) > self._max_length:\n self.lineLengthExceeded(line)\n return\n else:\n self.lineReceived(line)\n if len(self._buffer) > self._max_length:\n self.lineLengthExceeded(self._buffer)\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nhandles the field and value pair.", "response": "def _handle_field_value(self, field, value):\n \"\"\" Handle the field, value pair. \"\"\"\n if field == 'event':\n self._event = value\n elif field == 'data':\n self._data_lines.append(value)\n elif field == 'id':\n # Not implemented\n pass\n elif field == 'retry':\n # Not implemented\n pass"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _dispatch_event(self):\n data = self._prepare_data()\n if data is not None:\n self._handler(self._event, data)\n\n self._reset_event_data()", "response": "Dispatch the event to the handler."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlisten for events from Marathon.", "response": "def listen_events(self, reconnects=0):\n \"\"\"\n Start listening for events from Marathon, running a sync when we first\n successfully subscribe and triggering a sync on API request events.\n \"\"\"\n self.log.info('Listening for events from Marathon...')\n self._attached = False\n\n def on_finished(result, reconnects):\n # If the callback fires then the HTTP request to the event stream\n # went fine, but the persistent connection for the SSE stream was\n # dropped. Just reconnect for now- if we can't actually connect\n # then the errback will fire rather.\n self.log.warn('Connection lost listening for events, '\n 'reconnecting... ({reconnects} so far)',\n reconnects=reconnects)\n reconnects += 1\n return self.listen_events(reconnects)\n\n def log_failure(failure):\n self.log.failure('Failed to listen for events', failure)\n return failure\n\n return self.marathon_client.get_events({\n 'event_stream_attached': self._sync_on_event_stream_attached,\n 'api_post_event': self._sync_on_api_post_event\n }).addCallbacks(on_finished, log_failure, callbackArgs=[reconnects])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sync(self):\n self.log.info('Starting a sync...')\n\n def log_success(result):\n self.log.info('Sync completed successfully')\n return result\n\n def log_failure(failure):\n self.log.failure('Sync failed', failure, LogLevel.error)\n return failure\n\n return (self.marathon_client.get_apps()\n .addCallback(self._apps_acme_domains)\n .addCallback(self._filter_new_domains)\n .addCallback(self._issue_certs)\n .addCallbacks(log_success, log_failure))", "response": "Fetch the list of apps from Marathon and issue certificates for any domains that require a certificate."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _issue_cert(self, domain):\n def errback(failure):\n # Don't fail on some of the errors we could get from the ACME\n # server, rather just log an error so that we can continue with\n # other domains.\n failure.trap(txacme_ServerError)\n acme_error = failure.value.message\n\n if acme_error.code in ['rateLimited', 'serverInternal',\n 'connection', 'unknownHost']:\n # TODO: Fire off an error to Sentry or something?\n self.log.error(\n 'Error ({code}) issuing certificate for \"{domain}\": '\n '{detail}', code=acme_error.code, domain=domain,\n detail=acme_error.detail)\n else:\n # There are more error codes but if they happen then something\n # serious has gone wrong-- carry on error-ing.\n return failure\n\n d = self.txacme_service.issue_cert(domain)\n return d.addErrback(errback)", "response": "Issue a certificate for the given domain."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef warn_if_detached(func):\n @wraps(func)\n def wrapped(this, *args, **kwargs):\n # Check for _detached in __dict__ instead of using hasattr\n # to avoid infinite loop in __getattr__\n if '_detached' in this.__dict__ and this._detached:\n warnings.warn('here')\n return func(this, *args, **kwargs)\n return wrapped", "response": "Decorator to warn if self / cls is detached."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nensuring that self/cls contains a Storage backend.", "response": "def has_storage(func):\n \"\"\" Ensure that self/cls contains a Storage backend. \"\"\"\n @wraps(func)\n def wrapped(*args, **kwargs):\n me = args[0]\n if not hasattr(me, '_storage') or \\\n not me._storage:\n raise exceptions.ImproperConfigurationError(\n 'No storage backend attached to schema <{0}>.'\n .format(me._name.upper())\n )\n return func(*args, **kwargs)\n return wrapped"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rm_fwd_refs(obj):\n for stack, key in obj._backrefs_flat:\n\n # Unpack stack\n backref_key, parent_schema_name, parent_field_name = stack\n\n # Get parent info\n parent_schema = obj._collections[parent_schema_name]\n parent_key_store = parent_schema._pk_to_storage(key)\n parent_object = parent_schema.load(parent_key_store)\n if parent_object is None:\n continue\n\n # Remove forward references\n if parent_object._fields[parent_field_name]._list:\n getattr(parent_object, parent_field_name).remove(obj)\n else:\n parent_field_object = parent_object._fields[parent_field_name]\n setattr(parent_object, parent_field_name, parent_field_object._gen_default())\n\n # Save\n parent_object.save()", "response": "When removing an object other objects with forward references to the current object should remove those references. This function identifies objects\n with forward references to the current object then removes those references."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rm_back_refs(obj):\n for ref in _collect_refs(obj):\n ref['value']._remove_backref(\n ref['field_instance']._backref_field_name,\n obj,\n ref['field_name'],\n strict=False\n )", "response": "When removing an object with foreign fields back - references from other objects to the current object should be deleted."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nensuring that all forward references on the provided object have the appropriate backreferences.", "response": "def ensure_backrefs(obj, fields=None):\n \"\"\"Ensure that all forward references on the provided object have the\n appropriate backreferences.\n\n :param StoredObject obj: Database record\n :param list fields: Optional list of field names to check\n\n \"\"\"\n for ref in _collect_refs(obj, fields):\n updated = ref['value']._update_backref(\n ref['field_instance']._backref_field_name,\n obj,\n ref['field_name'],\n )\n if updated:\n logging.debug('Updated reference {}:{}:{}:{}:{}'.format(\n obj._name, obj._primary_key, ref['field_name'],\n ref['value']._name, ref['value']._primary_key,\n ))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _remove_by_pk(self, key, flush=True):\n try:\n del self.store[key]\n except Exception as error:\n pass\n if flush:\n self.flush()", "response": "Retrieve value from store."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize the bot plugins and everything.", "response": "def start(self):\n \"\"\"Initializes the bot, plugins, and everything.\"\"\"\n self.bot_start_time = datetime.now()\n self.webserver = Webserver(self.config['webserver']['host'], self.config['webserver']['port'])\n self.plugins.load()\n self.plugins.load_state()\n self._find_event_handlers()\n self.sc = ThreadedSlackClient(self.config['slack_token'])\n\n self.always_send_dm = ['_unauthorized_']\n if 'always_send_dm' in self.config:\n self.always_send_dm.extend(map(lambda x: '!' + x, self.config['always_send_dm']))\n\n # Rocket is very noisy at debug\n logging.getLogger('Rocket.Errors.ThreadPool').setLevel(logging.INFO)\n\n self.is_setup = True\n if self.test_mode:\n self.metrics['startup_time'] = (datetime.now() - self.bot_start_time).total_seconds() * 1000.0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun the main loop of the slack RTM API.", "response": "def run(self, start=True):\n \"\"\"\n Connects to slack and enters the main loop.\n\n * start - If True, rtm.start API is used. Else rtm.connect API is used\n\n For more info, refer to\n https://python-slackclient.readthedocs.io/en/latest/real_time_messaging.html#rtm-start-vs-rtm-connect\n \"\"\"\n # Fail out if setup wasn't run\n if not self.is_setup:\n raise NotSetupError\n\n # Start the web server\n self.webserver.start()\n\n first_connect = True\n\n try:\n while self.runnable:\n if self.reconnect_needed:\n if not self.sc.rtm_connect(with_team_state=start):\n return False\n self.reconnect_needed = False\n if first_connect:\n first_connect = False\n self.plugins.connect()\n\n # Get all waiting events - this always returns a list\n try:\n events = self.sc.rtm_read()\n except AttributeError:\n self.log.exception('Something has failed in the slack rtm library. This is fatal.')\n self.runnable = False\n events = []\n except:\n self.log.exception('Unhandled exception in rtm_read()')\n self.reconnect_needed = True\n events = []\n for e in events:\n try:\n self._handle_event(e)\n except KeyboardInterrupt:\n # Gracefully shutdown\n self.runnable = False\n except:\n self.log.exception('Unhandled exception in event handler')\n sleep(0.1)\n except KeyboardInterrupt:\n # On ctrl-c, just exit\n pass\n except:\n self.log.exception('Unhandled exception')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop(self):\n if self.webserver is not None:\n self.webserver.stop()\n if not self.test_mode:\n self.plugins.save_state()", "response": "Does cleanup of bot and plugins."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a message to the specified channel.", "response": "def send_message(self, channel, text, thread=None, reply_broadcast=None):\n \"\"\"\n Sends a message to the specified channel\n\n * channel - The channel to send to. This can be a SlackChannel object, a channel id, or a channel name\n (without the #)\n * text - String to send\n * thread - reply to the thread. See https://api.slack.com/docs/message-threading#threads_party\n * reply_broadcast - Set to true to indicate your reply is germane to all members of a channel\n \"\"\"\n # This doesn't want the # in the channel name\n if isinstance(channel, SlackRoomIMBase):\n channel = channel.id\n self.log.debug(\"Trying to send to %s: %s\", channel, text)\n self.sc.rtm_send_message(channel, text, thread=thread, reply_broadcast=reply_broadcast)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending a message to a user as an IM", "response": "def send_im(self, user, text):\n \"\"\"\n Sends a message to a user as an IM\n\n * user - The user to send to. This can be a SlackUser object, a user id, or the username (without the @)\n * text - String to send\n \"\"\"\n if isinstance(user, SlackUser):\n user = user.id\n channelid = self._find_im_channel(user)\n else:\n channelid = user.id\n self.send_message(channelid, text)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if a token is correct.", "response": "def token_is_correct(self, token):\n \"\"\"\n \u041f\u043e\u0434\u0445\u043e\u0434\u0438\u0442 \u043b\u0438 \u0442\u043e\u043a\u0435\u043d, \u0434\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u0438 \u0442\u0435\u043a\u0441\u0442\u0430.\n \u0414\u043e\u043f\u0443\u0441\u043a\u0430\u044e\u0442\u0441\u044f \u0440\u0443\u0441\u0441\u043a\u0438\u0435 \u0441\u043b\u043e\u0432\u0430, \u0437\u043d\u0430\u043a\u0438 \u043f\u0440\u0435\u043f\u0438\u043d\u0430\u043d\u0438\u044f \u0438 \u0441\u0438\u043c\u0432\u043e\u043b\u044b \u043d\u0430\u0447\u0430\u043b\u0430 \u0438 \u043a\u043e\u043d\u0446\u0430.\n \"\"\"\n if self.is_rus_word(token):\n return True\n elif self.ONLY_MARKS.search(token):\n return True\n elif self.END_TOKENS.search(token):\n return True\n elif token in \"$^\":\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of optimal variants.", "response": "def get_optimal_variant(self, variants, start_words, **kwargs):\n \"\"\"\n \u0412\u043e\u0437\u0432\u0440\u0430\u0449\u0430\u0435\u0442 \u043e\u043f\u0442\u0438\u043c\u0430\u043b\u044c\u043d\u044b\u0439 \u0432\u0430\u0440\u0438\u0430\u043d\u0442, \u0438\u0437 \u0432\u044b\u0431\u043e\u0440\u043a\u0438.\n \"\"\"\n\n if not start_words:\n return (choice(variants), {})\n\n _variants = []\n _weights = []\n for tok in frozenset(variants):\n if not self.token_is_correct(tok):\n continue\n weight = variants.count(tok)\n for word in start_words:\n for token in self.ONLY_WORDS.finditer(word.strip().lower()):\n if token.group() == tok:\n weight <<= 1\n _variants.append(tok)\n _weights.append(weight)\n\n if not _variants:\n return (choice(variants), {})\n\n return (choices(_variants, weights=_weights, k=1)[0], {})"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstarts generation of the user s succesfully.", "response": "def start_generation(self, *start_words, **kwargs):\n \"\"\"\n \u0413\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u0442 \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0435\u043d\u0438\u0435.\n :start_words: \u041f\u043e\u043f\u044b\u0442\u0430\u0442\u044c\u0441\u044f \u043d\u0430\u0447\u0430\u0442\u044c \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0435\u043d\u0438\u0435 \u0441 \u044d\u0442\u0438\u0445 \u0441\u043b\u043e\u0432.\n \"\"\"\n out_text = \"\"\n _need_capialize = True\n for token in self._get_generate_tokens(*start_words, **kwargs):\n if token in \"$^\":\n _need_capialize = True\n continue\n if self.ONLY_WORDS.search(token):\n out_text += \" \"\n if _need_capialize:\n _need_capialize = False\n token = token.title()\n out_text += token\n\n return out_text.strip()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_start_array(self, *start_words, **kwargs):\n if not self.start_arrays:\n raise MarkovTextExcept(\"\u041d\u0435 \u0441 \u0447\u0435\u0433\u043e \u043d\u0430\u0447\u0438\u043d\u0430\u0442\u044c \u0433\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044e.\")\n if not start_words:\n return choice(self.start_arrays)\n\n _variants = []\n _weights = []\n for tokens in self.start_arrays:\n weight = 0b1\n for word in start_words:\n word = word.strip().lower()\n for token in self.ONLY_WORDS.finditer(word):\n if token.group() in tokens:\n weight <<= 1\n if weight > 0b1:\n _variants.append(tokens)\n _weights.append(weight)\n\n if not _variants:\n return choice(self.start_arrays)\n\n return choices(_variants, weights=_weights, k=1)[0]", "response": "returns a list of start_words"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_base(self):\n self.base_dict = {}\n _start_arrays = []\n for tokens, word in self.chain_generator():\n self.base_dict.setdefault(tokens, []).append(word)\n if tokens[0] == \"^\": # \u041f\u0435\u0440\u0432\u044b\u0435 \u043a\u043b\u044e\u0447\u0438, \u0434\u043b\u044f \u043d\u0430\u0447\u0430\u043b\u0430 \u0433\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u0438.\n _start_arrays.append(tokens)\n\n self.start_arrays = tuple(\n frozenset(self.get_corrected_arrays(_start_arrays))\n )", "response": "\u041c\u0435\u0442\u043e\u0434 \u0441\u043e\u0437\u0434\u0430\u0451\u0442 \u0431\u0430\u0437\u043e\u0432\u044b\u0439 \u0441\u043b\u043e\u0432\u0430\u0440\u044c, \u043d\u0430 \u043e\u0441\u043d\u043e\u0432\u0435 \u043c\u0430\u0441\u0441\u0438\u0432\u0430 \u0442\u043e\u043a\u0435\u043d\u043e\u0432.\n \u0412\u044b\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u0438\u0437 \u043c\u0435\u0442\u043e\u0434\u0430 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the vocabulary of the user.", "response": "def set_vocabulary(self, peer_id, from_dialogue=None, update=False):\n \"\"\"\n \u041f\u043e\u043b\u0443\u0447\u0430\u0435\u0442 \u0432\u043e\u043a\u0430\u0431\u0443\u043b\u0430\u0440 \u0438\u0437 \u0444\u0443\u043d\u043a\u0446\u0438\u0438 get_vocabulary \u0438 \u0434\u0435\u043b\u0430\u0435\u0442 \u0435\u0433\u043e \u0430\u043a\u0442\u0438\u0432\u043d\u044b\u043c.\n \"\"\"\n\n self.tokens_array = self.get_vocabulary(\n peer_id,\n from_dialogue,\n update\n )\n self.create_base()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a dump of the current state of the current object.", "response": "def create_dump(self, name=None):\n \"\"\"\n \u0421\u043e\u0445\u0440\u0430\u043d\u044f\u0435\u0442 \u0442\u0435\u043a\u0443\u0449\u0443\u044e \u0431\u0430\u0437\u0443 \u043d\u0430 \u0436\u0451\u0441\u0442\u043a\u0438\u0439 \u0434\u0438\u0441\u043a.\n :name: \u0418\u043c\u044f \u0444\u0430\u0439\u043b\u0430, \u0431\u0435\u0437 \u0440\u0430\u0441\u0448\u0438\u0440\u0435\u043d\u0438\u044f.\n \"\"\"\n name = name or \"vocabularDump\"\n backup_dump_file = os_join(\n self.temp_folder,\n \"{0}.backup\".format(name)\n )\n dump_file = os_join(\n self.temp_folder,\n \"{0}.json\".format(name)\n )\n with open(backup_dump_file, \"w\", encoding=\"utf-8\") as js_file:\n json.dump(self.tokens_array, js_file, ensure_ascii=False)\n copy2(backup_dump_file, dump_file)\n remove(backup_dump_file)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_dump(self, name=None):\n name = name or \"vocabularDump\"\n dump_file = os_join(\n self.temp_folder,\n \"{0}.json\".format(name)\n )\n if not isfile(dump_file):\n raise MarkovTextExcept(\"\u0424\u0430\u0439\u043b {0!r} \u043d\u0435 \u043d\u0430\u0439\u0434\u0435\u043d.\".format(dump_file))\n with open(dump_file, \"rb\") as js_file:\n self.tokens_array = tuple(json.load(js_file))\n self.create_base()", "response": "Load a dump file and create a base object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_vocabulary(self, target, user=None, update=False):\n if not self.vk_object:\n raise MarkovTextExcept(\"\u041e\u0431\u044a\u0435\u043a\u0442 \u0431\u043e\u0442\u0430 \u043d\u0435 \u0437\u0430\u0434\u0430\u043d.\")\n\n json_name = \"{0}{1}_{2}\".format(\n target.__class__.__name__,\n target.id,\n user.id\n )\n json_file = os_join(self.temp_folder, \"{0}.json\".format(json_name))\n\n if not update:\n result = self.vocabulars.get(json_name, None)\n if result:\n return result\n elif isfile(json_file):\n with open(json_file, \"rb\") as js_file:\n result = self.vocabulars[json_name] = tuple(\n json.load(js_file)\n )\n return result\n\n _tokens_array = tuple(self.__parse_from_vk_dialogue(target, user))\n\n backup_file = \"{0}.backup\".format(splitext(json_file)[0])\n with open(backup_file, \"w\", encoding=\"utf-8\") as js_file:\n json.dump(_tokens_array, js_file, ensure_ascii=False)\n\n copy2(backup_file, json_file)\n remove(backup_file)\n self.vocabulars[json_name] = _tokens_array\n return _tokens_array", "response": "Returns a list of the vocabulary of the user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nupdate the internal list with the given data.", "response": "def update(self, data, fromfile=True):\n \"\"\"\n \u041f\u0440\u0438\u043d\u0438\u043c\u0430\u0435\u0442 \u0442\u0435\u043a\u0441\u0442, \u0438\u043b\u0438 \u043f\u0443\u0442\u044c \u043a \u0444\u0430\u0439\u043b\u0443 \u0438 \u043e\u0431\u043d\u043e\u0432\u043b\u044f\u0435\u0442 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0431\u0430\u0437\u0443.\n \"\"\"\n func = (self._parse_from_file if fromfile else self._parse_from_text)\n new_data = tuple(func(data))\n if new_data:\n self.tokens_array += new_data\n self.create_base()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_from_text(self, text):\n if not isinstance(text, str):\n raise MarkovTextExcept(\"\u041f\u0435\u0440\u0435\u0434\u0430\u043d \u043d\u0435 \u0442\u0435\u043a\u0441\u0442.\")\n text = text.strip().lower()\n need_start_token = True\n token = \"$\" # \u041d\u0430 \u0441\u043b\u0443\u0447\u0430\u0439, \u0435\u0441\u043b\u0438 \u043f\u0435\u0440\u0435\u0434\u0430\u043d\u043d\u0430\u044f \u0441\u0442\u0440\u043e\u043a\u0430 \u043f\u0443\u0441\u0442\u0430.\n for token in self.WORD_OR_MARKS.finditer(text):\n token = token.group()\n if need_start_token:\n need_start_token = False\n yield \"^\"\n yield token\n if self.END_TOKENS.search(token):\n need_start_token = True\n yield \"$\"\n if token != \"$\":\n yield \"$\"", "response": "Parse a text string into a list of strings."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_from_file(self, file_path):\n file_path = abspath(file_path)\n if not isfile(file_path):\n raise MarkovTextExcept(\"\u041f\u0435\u0440\u0435\u0434\u0430\u043d \u043d\u0435 \u0444\u0430\u0439\u043b.\")\n with open(file_path, \"rb\") as txt_file:\n for line in txt_file:\n text = line.decode(\"utf-8\", \"ignore\").strip()\n if not text:\n continue\n yield from self._parse_from_text(text)", "response": "A generator that yields MarkovItem objects from a file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntake a SlackEvent parses it for a command and runs it against the registered plugin s command queue and returns the command and the command and the command itself.", "response": "def push(self, message):\n \"\"\"\n Takes a SlackEvent, parses it for a command, and runs against registered plugin\n \"\"\"\n if self._ignore_event(message):\n return None, None\n args = self._parse_message(message)\n self.log.debug(\"Searching for command using chunks: %s\", args)\n cmd, msg_args = self._find_longest_prefix_command(args)\n if cmd is not None:\n if message.user is None:\n self.log.debug(\"Discarded message with no originating user: %s\", message)\n return None, None\n sender = message.user.username\n if message.channel is not None:\n sender = \"#%s/%s\" % (message.channel.name, sender)\n self.log.info(\"Received from %s: %s, args %s\", sender, cmd, msg_args)\n f = self._get_command(cmd, message.user)\n if f:\n if self._is_channel_ignored(f, message.channel):\n self.log.info(\"Channel %s is ignored, discarding command %s\", message.channel, cmd)\n return '_ignored_', \"\"\n return cmd, f.execute(message, msg_args)\n return '_unauthorized_', \"Sorry, you are not authorized to run %s\" % cmd\n return None, None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _ignore_event(self, message):\n if hasattr(message, 'subtype') and message.subtype in self.ignored_events:\n return True\n return False", "response": "Check if the message is not truly an event and ignore it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef register_plugin(self, plugin):\n self.log.info(\"Registering plugin %s\", type(plugin).__name__)\n self._register_commands(plugin)\n plugin.on_load()", "response": "Registers a plugin and commands with the dispatcher for push"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef acl_show(self, msg, args):\n name = args[0] if len(args) > 0 else None\n if name is None:\n return \"%s: The following ACLs are defined: %s\" % (msg.user, ', '.join(self._acl.keys()))\n\n if name not in self._acl:\n return \"Sorry, couldn't find an acl named '%s'\" % name\n\n return '\\n'.join([\n \"%s: ACL '%s' is defined as follows:\" % (msg.user, name),\n \"allow: %s\" % ', '.join(self._acl[name]['allow']),\n \"deny: %s\" % ', '.join(self._acl[name]['deny'])\n ])", "response": "Show current allow and deny blocks for the given acl."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a user to the given acl allow block.", "response": "def add_user_to_allow(self, name, user):\n \"\"\"Add a user to the given acl allow block.\"\"\"\n\n # Clear user from both allow and deny before adding\n if not self.remove_user_from_acl(name, user):\n return False\n\n if name not in self._acl:\n return False\n\n self._acl[name]['allow'].append(user)\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_user_from_acl(self, name, user):\n if name not in self._acl:\n return False\n if user in self._acl[name]['allow']:\n self._acl[name]['allow'].remove(user)\n if user in self._acl[name]['deny']:\n self._acl[name]['deny'].remove(user)\n return True", "response": "Remove a user from the given acl."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new acl.", "response": "def create_acl(self, name):\n \"\"\"Create a new acl.\"\"\"\n if name in self._acl:\n return False\n\n self._acl[name] = {\n 'allow': [],\n 'deny': []\n }\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mongo(daemon=False, port=20771):\n '''Run the mongod process.\n '''\n cmd = \"mongod --port {0}\".format(port)\n if daemon:\n cmd += \" --fork\"\n run(cmd)", "response": "Run the mongod process."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef proxy_factory(BaseSchema, label, ProxiedClass, get_key):\n def local():\n key = get_key()\n try:\n return proxies[BaseSchema][label][key]\n except KeyError:\n proxies[BaseSchema][label][key] = ProxiedClass()\n return proxies[BaseSchema][label][key]\n return LocalProxy(local)", "response": "Create a proxy to a class instance stored in proxies."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef with_proxies(proxy_map, get_key):\n def wrapper(cls):\n for label, ProxiedClass in six.iteritems(proxy_map):\n proxy = proxy_factory(cls, label, ProxiedClass, get_key)\n setattr(cls, label, proxy)\n return cls\n return wrapper", "response": "Class decorator factory that adds proxies to target class."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a value to a primary key.", "response": "def _to_primary_key(self, value):\n \"\"\"\n Return primary key; if value is StoredObject, verify\n that it is loaded.\n\n \"\"\"\n if value is None:\n return None\n if isinstance(value, self.base_class):\n if not value._is_loaded:\n raise exceptions.DatabaseError('Record must be loaded.')\n return value._primary_key\n\n return self.base_class._to_primary_key(value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nassigning to a nested dictionary.", "response": "def set_nested(data, value, *keys):\n \"\"\"Assign to a nested dictionary.\n\n :param dict data: Dictionary to mutate\n :param value: Value to set\n :param list *keys: List of nested keys\n\n >>> data = {}\n >>> set_nested(data, 'hi', 'k0', 'k1', 'k2')\n >>> data\n {'k0': {'k1': {'k2': 'hi'}}}\n\n \"\"\"\n if len(keys) == 1:\n data[keys[0]] = value\n else:\n if keys[0] not in data:\n data[keys[0]] = {}\n set_nested(data[keys[0]], value, *keys[1:])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves user by username", "response": "def get_by_username(self, username):\n \"\"\"Retrieve user by username\"\"\"\n res = filter(lambda x: x.username == username, self.users.values())\n if len(res) > 0:\n return res[0]\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set(self, user):\n\n self.log.info(\"Loading user information for %s/%s\", user.id, user.username)\n self.load_user_info(user)\n self.log.info(\"Loading user rights for %s/%s\", user.id, user.username)\n self.load_user_rights(user)\n self.log.info(\"Added user: %s/%s\", user.id, user.username)\n self._add_user_to_cache(user)\n return user", "response": "Adds a SlackUser object to the user manager"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting permissions on user object", "response": "def load_user_rights(self, user):\n \"\"\"Sets permissions on user object\"\"\"\n if user.username in self.admins:\n user.is_admin = True\n elif not hasattr(user, 'is_admin'):\n user.is_admin = False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef send_message(self, channel, text):\n if isinstance(channel, SlackIM) or isinstance(channel, SlackUser):\n self._bot.send_im(channel, text)\n elif isinstance(channel, SlackRoom):\n self._bot.send_message(channel, text)\n elif isinstance(channel, basestring):\n if channel[0] == '@':\n self._bot.send_im(channel[1:], text)\n elif channel[0] == '#':\n self._bot.send_message(channel[1:], text)\n else:\n self._bot.send_message(channel, text)\n else:\n self._bot.send_message(channel, text)", "response": "Send a message to the specified channel."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nschedule a function to be called after some period of time.", "response": "def start_timer(self, duration, func, *args):\n \"\"\"\n Schedules a function to be called after some period of time.\n\n * duration - time in seconds to wait before firing\n * func - function to be called\n * args - arguments to pass to the function\n \"\"\"\n t = threading.Timer(duration, self._timer_callback, (func, args))\n self._timer_callbacks[func] = t\n t.start()\n self.log.info(\"Scheduled call to %s in %ds\", func.__name__, duration)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop_timer(self, func):\n if func in self._timer_callbacks:\n t = self._timer_callbacks[func]\n t.cancel()\n del self._timer_callbacks[func]", "response": "Stops a timer if it has not fired yet"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting translate result in a better format", "response": "def print_res(data):\n \"\"\" Print translate result in a better format\n Args:\n data(str): result\n \"\"\"\n print('===================================')\n main_part = data['data']\n print(main_part['word_name'])\n symbols = main_part['symbols'][0]\n print(\"\u7f8e\u5f0f\u97f3\u6807\uff1a[\" + symbols['ph_am'] + \"]\")\n print(\"\u82f1\u5f0f\u97f3\u6807\uff1a[\" + symbols['ph_en'] + \"]\")\n print('-----------------------------------')\n parts = symbols['parts']\n for part in parts:\n print(part['part'])\n for mean in part['means']:\n print(\" \", mean)\n print('===================================')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_underlying_data(self, instance):\n self._touch(instance)\n return self.data.get(instance, None)", "response": "Return data from raw data store rather than overridden\n __get__ methods. Should NOT be overwritten."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncasts value to its frozen counterpart.", "response": "def freeze(value):\n \"\"\" Cast value to its frozen counterpart. \"\"\"\n if isinstance(value, list):\n return FrozenList(*value)\n if isinstance(value, dict):\n return FrozenDict(**value)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef help(self, msg, args):\n output = []\n if len(args) == 0:\n commands = sorted(self._bot.dispatcher.commands.items(), key=itemgetter(0))\n commands = filter(lambda x: x[1].is_subcmd is False, commands)\n # Filter commands if auth is enabled, hide_admin_commands is enabled, and user is not admin\n if self._should_filter_help_commands(msg.user):\n commands = filter(lambda x: x[1].admin_only is False, commands)\n for name, cmd in commands:\n output.append(self._get_short_help_for_command(name))\n else:\n name = '!' + args[0]\n output = [self._get_help_for_command(name)]\n return '\\n'.join(output)", "response": "Displays help for each command"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving the current state of the plugin to the backend.", "response": "def save(self, msg, args):\n \"\"\"Causes the bot to write its current state to backend.\"\"\"\n self.send_message(msg.channel, \"Saving current state...\")\n self._bot.plugins.save_state()\n self.send_message(msg.channel, \"Done.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef shutdown(self, msg, args):\n self.log.info(\"Received shutdown from %s\", msg.user.username)\n self._bot.runnable = False\n return \"Shutting down...\"", "response": "Called when a shutdown message is received."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprints information about the user and bot version.", "response": "def whoami(self, msg, args):\n \"\"\"Prints information about the user and bot version.\"\"\"\n output = [\"Hello %s\" % msg.user]\n if hasattr(self._bot.dispatcher, 'auth_manager') and msg.user.is_admin is True:\n output.append(\"You are a *bot admin*.\")\n output.append(\"Bot version: %s-%s\" % (self._bot.version, self._bot.commit))\n return '\\n'.join(output)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nattempt to ignore all messages from the specified channel.", "response": "def sleep(self, channel):\n \"\"\"Causes the bot to ignore all messages from the channel.\n\n Usage:\n !sleep [channel name] - ignore the specified channel (or current if none specified)\n \"\"\"\n self.log.info('Sleeping in %s', channel)\n self._bot.dispatcher.ignore(channel)\n self.send_message(channel, 'Good night')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef wake(self, channel):\n self.log.info('Waking up in %s', channel)\n self._bot.dispatcher.unignore(channel)\n self.send_message(channel, 'Hello, how may I be of service?')", "response": "Wake up the channel."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _arg_name(self, name, types, prefix=\"--\"):\n if 'type:a10_nullable' in types:\n return self._arg_name(name, types['type:a10_nullable'], prefix)\n\n if 'type:a10_list' in types:\n return self._arg_name(name, types['type:a10_list'], prefix)\n\n if 'type:a10_reference' in types:\n if name.endswith('_id'):\n name = name[:-3]\n\n \"\"\"--shish-kabob it\"\"\"\n return prefix + name.replace('_', '-')", "response": "Return the argument name for the user."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _sort_by(key):\n\n @staticmethod\n def sort_by(p_list, reverse=False):\n return sorted(\n p_list,\n key=lambda p: getattr(p, key),\n reverse=reverse,\n )\n\n return sort_by", "response": "High order function for sort methods."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nselecting path by criterion.", "response": "def select(self, filters=all_true, recursive=True):\n \"\"\"Select path by criterion.\n\n :param filters: a lambda function that take a `pathlib.Path` as input,\n boolean as a output.\n :param recursive: include files in subfolder or not.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u6839\u636efilters\u4e2d\u5b9a\u4e49\u7684\u6761\u4ef6\u9009\u62e9\u8def\u5f84\u3002\n \"\"\"\n self.assert_is_dir_and_exists()\n\n if recursive:\n for p in self.glob(\"**/*\"):\n if filters(p):\n yield p\n else:\n for p in self.iterdir():\n if filters(p):\n yield p"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nselects file path by criterion.", "response": "def select_file(self, filters=all_true, recursive=True):\n \"\"\"Select file path by criterion.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u6839\u636efilters\u4e2d\u5b9a\u4e49\u7684\u6761\u4ef6\u9009\u62e9\u6587\u4ef6\u3002\n \"\"\"\n for p in self.select(filters, recursive):\n if p.is_file():\n yield p"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef select_dir(self, filters=all_true, recursive=True):\n for p in self.select(filters, recursive):\n if p.is_dir():\n yield p", "response": "Select all directories in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncounts how many files in this directory. Including file in sub folder.", "response": "def n_file(self):\n \"\"\"\n Count how many files in this directory. Including file in sub folder.\n \"\"\"\n self.assert_is_dir_and_exists()\n n = 0\n for _ in self.select_file(recursive=True):\n n += 1\n return n"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncounts how many folders in this directory. Including folder in sub folder.", "response": "def n_dir(self):\n \"\"\"\n Count how many folders in this directory. Including folder in sub folder.\n \"\"\"\n self.assert_is_dir_and_exists()\n n = 0\n for _ in self.select_dir(recursive=True):\n n += 1\n return n"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncounting how many subfiles in this directory.", "response": "def n_subfile(self):\n \"\"\"\n Count how many files in this directory (doesn't include files in\n sub folders).\n \"\"\"\n self.assert_is_dir_and_exists()\n n = 0\n for _ in self.select_file(recursive=False):\n n += 1\n return n"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef n_subdir(self):\n self.assert_is_dir_and_exists()\n n = 0\n for _ in self.select_dir(recursive=False):\n n += 1\n return n", "response": "Count how many subfolders in this directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef select_by_ext(self, ext, recursive=True):\n ext = [ext.strip().lower() for ext in ensure_list(ext)]\n\n def filters(p): return p.suffix.lower() in ext\n\n return self.select_file(filters, recursive)", "response": "Select file path by extension."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef select_by_pattern_in_fname(self,\n pattern,\n recursive=True,\n case_sensitive=False):\n \"\"\"\n Select file path by text pattern in file name.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9009\u62e9\u6587\u4ef6\u540d\u4e2d\u5305\u542b\u6307\u5b9a\u5b50\u5b57\u7b26\u4e32\u7684\u6587\u4ef6\u3002\n \"\"\"\n if case_sensitive:\n def filters(p):\n return pattern in p.fname\n else:\n pattern = pattern.lower()\n\n def filters(p):\n return pattern in p.fname.lower()\n\n return self.select_file(filters, recursive)", "response": "Select file path by text pattern in file name."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nselecting file path by text pattern in absolute path.", "response": "def select_by_pattern_in_abspath(self,\n pattern,\n recursive=True,\n case_sensitive=False):\n \"\"\"\n Select file path by text pattern in absolute path.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9009\u62e9\u7edd\u5bf9\u8def\u5f84\u4e2d\u5305\u542b\u6307\u5b9a\u5b50\u5b57\u7b26\u4e32\u7684\u6587\u4ef6\u3002\n \"\"\"\n if case_sensitive:\n def filters(p):\n return pattern in p.abspath\n else:\n pattern = pattern.lower()\n\n def filters(p):\n return pattern in p.abspath.lower()\n\n return self.select_file(filters, recursive)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef select_by_size(self, min_size=0, max_size=1 << 40, recursive=True):\n\n def filters(p): return min_size <= p.size <= max_size\n\n return self.select_file(filters, recursive)", "response": "Select file path by size."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nselecting file path by modify time.", "response": "def select_by_mtime(self, min_time=0, max_time=ts_2100,\n recursive=True):\n \"\"\"\n Select file path by modify time.\n\n :param min_time: lower bound timestamp\n :param max_time: upper bound timestamp\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9009\u62e9\u6240\u6709 :attr:`pathlib_mate.pathlib2.Path.mtime` \u5728\u4e00\u5b9a\u8303\u56f4\u5185\u7684\u6587\u4ef6\u3002\n \"\"\"\n\n def filters(p): return min_time <= p.mtime <= max_time\n\n return self.select_file(filters, recursive)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nselecting file path by access time.", "response": "def select_by_atime(self, min_time=0, max_time=ts_2100, recursive=True):\n \"\"\"\n Select file path by access time.\n\n :param min_time: lower bound timestamp\n :param max_time: upper bound timestamp\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9009\u62e9\u6240\u6709 :attr:`pathlib_mate.pathlib2.Path.atime` \u5728\u4e00\u5b9a\u8303\u56f4\u5185\u7684\u6587\u4ef6\u3002\n \"\"\"\n\n def filters(p): return min_time <= p.atime <= max_time\n\n return self.select_file(filters, recursive)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nselect file path by create time.", "response": "def select_by_ctime(self, min_time=0, max_time=ts_2100,\n recursive=True):\n \"\"\"\n Select file path by create time.\n\n :param min_time: lower bound timestamp\n :param max_time: upper bound timestamp\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9009\u62e9\u6240\u6709 :attr:`pathlib_mate.pathlib2.Path.ctime` \u5728\u4e00\u5b9a\u8303\u56f4\u5185\u7684\u6587\u4ef6\u3002\n \"\"\"\n\n def filters(p): return min_time <= p.ctime <= max_time\n\n return self.select_file(filters, recursive)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dirsize(self):\n total = 0\n for p in self.select_file(recursive=True):\n try:\n total += p.size\n except: # pragma: no cover\n print(\"Unable to get file size of: %s\" % p)\n return total", "response": "Return total file size of the file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef make_zip_archive(self,\n dst=None,\n filters=all_true,\n compress=True,\n overwrite=False,\n makedirs=False,\n verbose=False): # pragma: no cover\n \"\"\"\n Make a zip archive.\n\n :param dst: output file path. if not given, will be automatically assigned.\n :param filters: custom path filter. By default it allows any file.\n :param compress: compress or not.\n :param overwrite: overwrite exists or not.\n :param verbose: display log or not.\n :return:\n \"\"\"\n self.assert_exists()\n\n if dst is None:\n dst = self._auto_zip_archive_dst()\n else:\n dst = self.change(new_abspath=dst)\n\n if not dst.basename.lower().endswith(\".zip\"):\n raise ValueError(\"zip archive name has to be endswith '.zip'!\")\n\n if dst.exists():\n if not overwrite:\n raise IOError(\"'%s' already exists!\" % dst)\n\n if compress:\n compression = ZIP_DEFLATED\n else:\n compression = ZIP_STORED\n\n if not dst.parent.exists():\n if makedirs:\n os.makedirs(dst.parent.abspath)\n\n if verbose:\n msg = \"Making zip archive for '%s' ...\" % self\n print(msg)\n\n current_dir = os.getcwd()\n\n if self.is_dir():\n total_size = 0\n selected = list()\n for p in self.glob(\"**/*\"):\n if filters(p):\n selected.append(p)\n total_size += p.size\n\n if verbose:\n msg = \"Got {} files, total size is {}, compressing ...\".format(\n len(selected), repr_data_size(total_size),\n )\n print(msg)\n\n with ZipFile(dst.abspath, \"w\", compression) as f:\n os.chdir(self.abspath)\n for p in selected:\n relpath = p.relative_to(self).__str__()\n f.write(relpath)\n\n elif self.is_file():\n with ZipFile(dst.abspath, \"w\", compression) as f:\n os.chdir(self.parent.abspath)\n f.write(self.basename)\n\n os.chdir(current_dir)\n\n if verbose:\n msg = \"Complete! Archive size is {}.\".format(dst.size_in_text)\n print(msg)", "response": "Makes a zip archive of the current directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef backup(self,\n dst=None,\n ignore=None,\n ignore_ext=None,\n ignore_pattern=None,\n ignore_size_smaller_than=None,\n ignore_size_larger_than=None,\n case_sensitive=False): # pragma: no cover\n \"\"\"\n Create a compressed zip archive backup for a directory.\n\n :param dst: the output file path.\n :param ignore: file or directory defined in this list will be ignored.\n :param ignore_ext: file with extensions defined in this list will be ignored.\n :param ignore_pattern: any file or directory that contains this pattern\n will be ignored.\n :param ignore_size_smaller_than: any file size smaller than this\n will be ignored.\n :param ignore_size_larger_than: any file size larger than this\n will be ignored.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u4e3a\u4e00\u4e2a\u76ee\u5f55\u521b\u5efa\u4e00\u4e2a\u5907\u4efd\u538b\u7f29\u5305\u3002\u53ef\u4ee5\u901a\u8fc7\u8fc7\u6ee4\u5668\u9009\u62e9\u4f60\u8981\u5907\u4efd\u7684\u6587\u4ef6\u3002\n \"\"\"\n\n def preprocess_arg(arg): # pragma: no cover\n if arg is None:\n return []\n\n if isinstance(arg, (tuple, list)):\n return list(arg)\n else:\n return [arg, ]\n\n self.assert_is_dir_and_exists()\n\n ignore = preprocess_arg(ignore)\n for i in ignore:\n if i.startswith(\"/\") or i.startswith(\"\\\\\"):\n raise ValueError\n\n ignore_ext = preprocess_arg(ignore_ext)\n for ext in ignore_ext:\n if not ext.startswith(\".\"):\n raise ValueError\n\n ignore_pattern = preprocess_arg(ignore_pattern)\n\n if case_sensitive:\n pass\n else:\n ignore = [i.lower() for i in ignore]\n ignore_ext = [i.lower() for i in ignore_ext]\n ignore_pattern = [i.lower() for i in ignore_pattern]\n\n def filters(p):\n relpath = p.relative_to(self).abspath\n if not case_sensitive:\n relpath = relpath.lower()\n\n # ignore\n for i in ignore:\n if relpath.startswith(i):\n return False\n\n # ignore_ext\n if case_sensitive:\n ext = p.ext\n else:\n ext = p.ext.lower()\n\n if ext in ignore_ext:\n return False\n\n # ignore_pattern\n for pattern in ignore_pattern:\n if pattern in relpath:\n return False\n\n # ignore_size_smaller_than\n if ignore_size_smaller_than:\n if p.size < ignore_size_smaller_than:\n return False\n\n # ignore_size_larger_than\n if ignore_size_larger_than:\n if p.size > ignore_size_larger_than:\n return False\n\n return True\n\n self.make_zip_archive(\n dst=dst, filters=filters, compress=True, overwrite=False, verbose=True,\n )", "response": "Create a compressed zip archive for a directory."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef acquire_lock(func):\n @wraps(func)\n def wrapper(self, *args, **kwargs):\n with self.locker as r:\n # get the result\n acquired, code, _ = r\n if acquired:\n try:\n r = func(self, *args, **kwargs)\n except Exception as err:\n e = str(err)\n else:\n e = None\n else:\n warnings.warn(\"code %s. Unable to aquire the lock when calling '%s'. You may try again!\"%(code,func.__name__) )\n e = None\n r = None\n # raise error after exiting with statement and releasing the lock!\n if e is not None:\n traceback.print_stack()\n raise Exception(e)\n return r\n return wrapper", "response": "Decorate methods when locking repository is required."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sync_required(func):\n @wraps(func)\n def wrapper(self, *args, **kwargs):\n if not self._keepSynchronized:\n r = func(self, *args, **kwargs)\n else:\n state = self._load_state()\n #print(\"-----------> \",state, self.state)\n if state is None:\n r = func(self, *args, **kwargs)\n elif state == self.state:\n r = func(self, *args, **kwargs)\n else:\n warnings.warn(\"Repository at '%s' is out of date. Need to load it again to avoid conflict.\"%self.path)\n r = None\n return r\n return wrapper", "response": "Decorate methods when synchronizing repository is required."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_list_representation(self):\n if self.__path is None:\n return []\n repr = [ self.__path+\":[\"+','.join(list(dict.__getitem__(self, 'files')))+']' ]\n # walk directories\n for directory in sorted(list(self.walk_directories_relative_path())):\n directoryRepr = os.path.normpath(directory)\n # get directory info\n dirInfoDict, errorMessage = self.get_directory_info(directory)\n assert dirInfoDict is not None, errorMessage\n directoryRepr += \":[\"+','.join( list(dict.__getitem__(dirInfoDict, 'files')))+']'\n repr.append(directoryRepr)\n return repr", "response": "Gets a list representation of the Repository content in a list of directories ( files ) format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef walk_files_relative_path(self, relativePath=\"\"):\n def walk_files(directory, relativePath):\n directories = dict.__getitem__(directory, 'directories')\n files = dict.__getitem__(directory, 'files')\n for f in sorted(files):\n yield os.path.join(relativePath, f)\n for k in sorted(dict.keys(directories)):\n path = os.path.join(relativePath, k)\n dir = directories.__getitem__(k)\n for e in walk_files(dir, path):\n yield e\n dir, errorMessage = self.get_directory_info(relativePath)\n assert dir is not None, errorMessage\n return walk_files(dir, relativePath='')", "response": "Walk the repository and yield all found files relative path joined with file name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef walk_files_info(self, relativePath=\"\"):\n def walk_files(directory, relativePath):\n directories = dict.__getitem__(directory, 'directories')\n files = dict.__getitem__(directory, 'files')\n for fname in sorted(files):\n info = dict.__getitem__(files,fname)\n yield os.path.join(relativePath, fname), info\n for k in sorted(dict.keys(directories)):\n path = os.path.join(relativePath, k)\n dir = dict.__getitem__(directories, k)\n for e in walk_files(dir, path):\n yield e\n dir, errorMessage = self.get_directory_info(relativePath)\n assert dir is not None, errorMessage\n return walk_files(dir, relativePath='')", "response": "Walk the repository and yield tuples as the following:\\n\n relative path to relativePath joined with file name file info dict."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef walk_directories_relative_path(self, relativePath=\"\"):\n def walk_directories(directory, relativePath):\n directories = dict.__getitem__(directory, 'directories')\n dirNames = dict.keys(directories)\n for d in sorted(dirNames):\n yield os.path.join(relativePath, d)\n for k in sorted(dict.keys(directories)):\n path = os.path.join(relativePath, k)\n dir = dict.__getitem__(directories, k)\n for e in walk_directories(dir, path):\n yield e\n dir, errorMessage = self.get_directory_info(relativePath)\n assert dir is not None, errorMessage\n return walk_directories(dir, relativePath='')", "response": "Walks the repository and yield all found directories relative path\nCOOKIE"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef walk_directories_info(self, relativePath=\"\"):\n def walk_directories(directory, relativePath):\n directories = dict.__getitem__(directory, 'directories')\n for fname in sorted(directories):\n info = dict.__getitem__(directories,fname)\n yield os.path.join(relativePath, fname), info\n for k in sorted(dict.keys(directories)):\n path = os.path.join(relativePath, k)\n dir = dict.__getitem__(directories, k)\n for e in walk_directories(dir, path):\n yield e\n dir, errorMessage = self.get_directory_info(relativePath)\n assert dir is not None, errorMessage\n return walk_directories(dir, relativePath='')", "response": "Walk repository and yield all found directories relative path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef walk_directory_directories_relative_path(self, relativePath=\"\"):\n # get directory info dict\n errorMessage = \"\"\n relativePath = os.path.normpath(relativePath)\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n for dname in dict.__getitem__(dirInfoDict, \"directories\"):\n yield os.path.join(relativePath, dname)", "response": "Walk a certain directory in repository and yield all found directories relative path."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwalks a certain directory and yield tuples as the following:\\ n relative path joined with directory name file info dict.", "response": "def walk_directory_directories_info(self, relativePath=\"\"):\n \"\"\"\n Walk a certain directory in repository and yield tuples as the following:\\n\n (relative path joined with directory name, file info dict).\n\n :parameters:\n #. relativePath (str): The relative path of the directory.\n \"\"\"\n # get directory info dict\n relativePath = os.path.normpath(relativePath)\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n for fname in dict.__getitem__(dirInfoDict, \"directories\"):\n yield os.path.join(relativePath, fname), dict.__getitem__(dirInfoDict, \"directories\")[fname]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef synchronize(self, verbose=False):\n if self.__path is None:\n return\n # walk directories\n for dirPath in sorted(list(self.walk_directories_relative_path())):\n realPath = os.path.join(self.__path, dirPath)\n # if directory exist\n if os.path.isdir(realPath):\n continue\n if verbose: warnings.warn(\"%s directory is missing\"%realPath)\n # loop to get dirInfoDict\n keys = dirPath.split(os.sep)\n dirInfoDict = self\n for idx in range(len(keys)-1):\n dirs = dict.get(dirInfoDict, 'directories', None)\n if dirs is None: break\n dirInfoDict = dict.get(dirs, keys[idx], None)\n if dirInfoDict is None: break\n # remove dirInfoDict directory if existing\n if dirInfoDict is not None:\n dirs = dict.get(dirInfoDict, 'directories', None)\n if dirs is not None:\n dict.pop( dirs, keys[-1], None )\n # walk files\n for filePath in sorted(list(self.walk_files_relative_path())):\n realPath = os.path.join(self.__path, filePath)\n # if file exists\n if os.path.isfile( realPath ):\n continue\n if verbose: warnings.warn(\"%s file is missing\"%realPath)\n # loop to get dirInfoDict\n keys = filePath.split(os.sep)\n dirInfoDict = self\n for idx in range(len(keys)-1):\n dirs = dict.get(dirInfoDict, 'directories', None)\n if dirs is None: break\n dirInfoDict = dict.get(dirs, keys[idx], None)\n if dirInfoDict is None: break\n # remove dirInfoDict file if existing\n if dirInfoDict is not None:\n files = dict.get(dirInfoDict, 'files', None)\n if files is not None:\n dict.pop( files, keys[-1], None )", "response": "Synchronizes the Repository information with the directory."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads the current instance of the current repository from a directory path and updates the current instance with the loaded data.", "response": "def load_repository(self, path):\n \"\"\"\n Load repository from a directory path and update the current instance.\n\n :Parameters:\n #. path (string): The path of the directory from where to load the repository.\n If '.' or an empty string is passed, the current working directory will be used.\n\n :Returns:\n #. repository (pyrep.Repository): returns self repository with loaded data.\n \"\"\"\n # try to open\n if path.strip() in ('','.'):\n path = os.getcwd()\n repoPath = os.path.realpath( os.path.expanduser(path) )\n if not self.is_repository(repoPath):\n raise Exception(\"no repository found in '%s'\"%str(repoPath))\n # get pyrepinfo path\n repoInfoPath = os.path.join(repoPath, \".pyrepinfo\")\n try:\n fd = open(repoInfoPath, 'rb')\n except Exception as e:\n raise Exception(\"unable to open repository file(%s)\"%e)\n # before doing anything try to lock repository\n # can't decorate with @acquire_lock because this will point to old repository\n # path or to current working directory which might not be the path anyways\n L = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(repoPath, \".pyreplock\"))\n acquired, code = L.acquire_lock()\n # check if acquired.\n if not acquired:\n warnings.warn(\"code %s. Unable to aquire the lock when calling 'load_repository'. You may try again!\"%(code,) )\n return\n try:\n # unpickle file\n try:\n repo = pickle.load( fd )\n except Exception as e:\n fd.close()\n raise Exception(\"unable to pickle load repository (%s)\"%e)\n finally:\n fd.close()\n # check if it's a PyrepInfo instance\n if not isinstance(repo, Repository):\n raise Exception(\".pyrepinfo in '%s' is not a repository instance.\"%s)\n else:\n # update info path\n self.__reset_repository()\n self.__update_repository(repo)\n self.__path = repoPath\n # set timestamp\n self.__state = self._get_or_create_state()\n except Exception as e:\n L.release_lock()\n raise Exception(e)\n finally:\n L.release_lock()\n # set loaded repo locker path to L because repository have been moved to another directory\n self.__locker = L\n # return\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a repository in a directory. This method insures the creation of the directory in the system if it is missing.\\n **N.B. This method erases existing pyrep repository in the path but not the repository files.** :Parameters: #. path (string): The real absolute path where to create the Repository. If '.' or an empty string is passed, the current working directory will be used. #. info (None, object): Any information that can identify the repository. #. verbose (boolean): Whether to be warn and informed about any abnormalities.", "response": "def create_repository(self, path, info=None, verbose=True):\n \"\"\"\n create a repository in a directory.\n This method insures the creation of the directory in the system if it is missing.\\n\n\n **N.B. This method erases existing pyrep repository in the path but not the repository files.**\n\n :Parameters:\n #. path (string): The real absolute path where to create the Repository.\n If '.' or an empty string is passed, the current working directory will be used.\n #. info (None, object): Any information that can identify the repository.\n #. verbose (boolean): Whether to be warn and informed about any abnormalities.\n \"\"\"\n try:\n info = copy.deepcopy( info )\n except:\n raise Exception(\"Repository info must be a copyable python object.\")\n # get real path\n if path.strip() in ('','.'):\n path = os.getcwd()\n realPath = os.path.realpath( os.path.expanduser(path) )\n # create directory if not existing\n if not os.path.isdir(realPath):\n os.makedirs(realPath)\n self.__path = realPath\n self.__info = info\n # reset if replace is set to True\n if self.is_repository(realPath):\n if verbose:\n warnings.warn(\"A pyrep Repository already exists in the given path '%s' and therefore it has been erased and replaced by a fresh repository.\"%path)\n # reset repository\n self.__reset_repository()\n # update locker because normally this is done in __update_repository method\n lp = '.pyreplock'\n if self.__path is not None:\n lp = os.path.join(self.__path,lp)\n self.__locker.set_lock_path(lp)\n self.__locker.set_lock_pass(str(uuid.uuid1()))\n # save repository\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a repository at given real path or load any existing one.", "response": "def get_repository(self, path, info=None, verbose=True):\n \"\"\"\n Create a repository at given real path or load any existing one.\n This method insures the creation of the directory in the system if it is missing.\\n\n Unlike create_repository, this method doesn't erase any existing repository\n in the path but loads it instead.\n\n **N.B. On some systems and some paths, creating a directory may requires root permissions.**\n\n :Parameters:\n #. path (string): The real absolute path where to create the Repository.\n If '.' or an empty string is passed, the current working directory will be used.\n #. info (None, object): Any information that can identify the repository.\n #. verbose (boolean): Whether to be warn and informed about any abnormalities.\n \"\"\"\n # get real path\n if path.strip() in ('','.'):\n path = os.getcwd()\n realPath = os.path.realpath( os.path.expanduser(path) )\n # create directory if not existing\n if not os.path.isdir(realPath):\n os.makedirs(realPath)\n # create Repository\n if not self.is_repository(realPath):\n self.create_repository(realPath, info=info, verbose=verbose)\n else:\n self.load_repository(realPath)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remove_repository(self, path=None, relatedFiles=False, relatedFolders=False, verbose=True):\n if path is not None:\n realPath = os.path.realpath( os.path.expanduser(path) )\n else:\n realPath = self.__path\n if realPath is None:\n if verbose: warnings.warn('path is None and current Repository is not initialized!')\n return\n if not self.is_repository(realPath):\n if verbose: warnings.warn(\"No repository found in '%s'!\"%realPath)\n return\n # check for security\n if realPath == os.path.realpath('/..') :\n if verbose: warnings.warn('You are about to wipe out your system !!! action aboarded')\n return\n # get repo\n if path is not None:\n repo = Repository()\n repo.load_repository(realPath)\n else:\n repo = self\n # delete files\n if relatedFiles:\n for relativePath in repo.walk_files_relative_path():\n realPath = os.path.join(repo.path, relativePath)\n if not os.path.isfile(realPath):\n continue\n if not os.path.exists(realPath):\n continue\n os.remove( realPath )\n # delete directories\n if relatedFolders:\n for relativePath in reversed(list(repo.walk_directories_relative_path())):\n realPath = os.path.join(repo.path, relativePath)\n # protect from wiping out the system\n if not os.path.isdir(realPath):\n continue\n if not os.path.exists(realPath):\n continue\n if not len(os.listdir(realPath)):\n os.rmdir( realPath )\n # delete repository\n os.remove( os.path.join(repo.path, \".pyrepinfo\" ) )\n for fname in (\".pyrepstate\", \".pyreplock\"):\n p = os.path.join(repo.path, fname )\n if os.path.exists( p ):\n os.remove( p )\n # remove main directory if empty\n if os.path.isdir(repo.path):\n if not len(os.listdir(repo.path)):\n os.rmdir( repo.path )\n # reset repository\n repo.__reset_repository()", "response": "Removes a repository from the system."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save(self):\n # open file\n repoInfoPath = os.path.join(self.__path, \".pyrepinfo\")\n try:\n fdinfo = open(repoInfoPath, 'wb')\n except Exception as e:\n raise Exception(\"unable to open repository info for saving (%s)\"%e)\n # save repository\n try:\n pickle.dump( self, fdinfo, protocol=2 )\n except Exception as e:\n fdinfo.flush()\n os.fsync(fdinfo.fileno())\n fdinfo.close()\n raise Exception( \"Unable to save repository info (%s)\"%e )\n finally:\n fdinfo.flush()\n os.fsync(fdinfo.fileno())\n fdinfo.close()\n # save timestamp\n repoTimePath = os.path.join(self.__path, \".pyrepstate\")\n try:\n self.__state = (\"%.6f\"%time.time()).encode()\n with open(repoTimePath, 'wb') as fdtime:\n fdtime.write( self.__state )\n fdtime.flush()\n os.fsync(fdtime.fileno())\n except Exception as e:\n raise Exception(\"unable to open repository time stamp for saving (%s)\"%e)", "response": "Save the current state of the object to disk."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a tar file package of all the files and directories in the repository.", "response": "def create_package(self, path=None, name=None, mode=None):\n \"\"\"\n Create a tar file package of all the repository files and directories.\n Only files and directories that are stored in the repository info\n are stored in the package tar file.\n\n **N.B. On some systems packaging requires root permissions.**\n\n :Parameters:\n #. path (None, string): The real absolute path where to create the package.\n If None, it will be created in the same directory as the repository\n If '.' or an empty string is passed, the current working directory will be used.\n #. name (None, string): The name to give to the package file\n If None, the package directory name will be used with the appropriate extension added.\n #. mode (None, string): The writing mode of the tarfile.\n If None, automatically the best compression mode will be chose.\n Available modes are ('w', 'w:', 'w:gz', 'w:bz2')\n \"\"\"\n # check mode\n assert mode in (None, 'w', 'w:', 'w:gz', 'w:bz2'), 'unkown archive mode %s'%str(mode)\n if mode is None:\n mode = 'w:bz2'\n mode = 'w:'\n # get root\n if path is None:\n root = os.path.split(self.__path)[0]\n elif path.strip() in ('','.'):\n root = os.getcwd()\n else:\n root = os.path.realpath( os.path.expanduser(path) )\n assert os.path.isdir(root), 'absolute path %s is not a valid directory'%path\n # get name\n if name is None:\n ext = mode.split(\":\")\n if len(ext) == 2:\n if len(ext[1]):\n ext = \".\"+ext[1]\n else:\n ext = '.tar'\n else:\n ext = '.tar'\n name = os.path.split(self.__path)[1]+ext\n # save repository\n self.save()\n # create tar file\n tarfilePath = os.path.join(root, name)\n try:\n tarHandler = tarfile.TarFile.open(tarfilePath, mode=mode)\n except Exception as e:\n raise Exception(\"Unable to create package (%s)\"%e)\n # walk directory and create empty directories\n for directory in sorted(list(self.walk_directories_relative_path())):\n t = tarfile.TarInfo( directory )\n t.type = tarfile.DIRTYPE\n tarHandler.addfile(t)\n # walk files and add to tar\n for file in self.walk_files_relative_path():\n tarHandler.add(os.path.join(self.__path,file), arcname=file)\n # save repository .pyrepinfo\n tarHandler.add(os.path.join(self.__path,\".pyrepinfo\"), arcname=\".pyrepinfo\")\n # close tar file\n tarHandler.close()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if there is a repository in path.", "response": "def is_repository(self, path):\n \"\"\"\n Check if there is a Repository in path.\n\n :Parameters:\n #. path (string): The real path of the directory where to check if there is a repository.\n\n :Returns:\n #. result (boolean): Whether its a repository or not.\n \"\"\"\n realPath = os.path.realpath( os.path.expanduser(path) )\n if not os.path.isdir(realPath):\n return False\n if \".pyrepinfo\" not in os.listdir(realPath):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_directory_info(self, relativePath):\n relativePath = os.path.normpath(relativePath)\n # if root directory\n if relativePath in ('','.'):\n return self, \"\"\n currentDir = self.__path\n dirInfoDict = self\n for dir in relativePath.split(os.sep):\n dirInfoDict = dict.__getitem__(dirInfoDict, \"directories\")\n currentDir = os.path.join(currentDir, dir)\n # check if path exists\n if not os.path.exists(currentDir):\n return None, \"directory '%s' is not found\"%currentDir\n val = dirInfoDict.get(dir, None)\n # check if directory is registered in repository\n if val is None:\n return None, \"directory '%s' is not registered in PyrepInfo\"%currentDir\n dirInfoDict = val\n return dirInfoDict, \"\"", "response": "Returns the directory information dictionary and the error message if any error has occurred."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_parent_directory_info(self, relativePath):\n relativePath = os.path.normpath(relativePath)\n # if root directory\n if relativePath in ('','.'):\n return self, \"relativePath is empty pointing to the repostitory itself.\"\n # split path\n parentDirPath, _ = os.path.split(relativePath)\n # get parent directory info\n return self.get_directory_info(parentDirPath)", "response": "Get the parent directory info of a file or directory from the Repository."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget file information dict from the repository given its relative path and name.", "response": "def get_file_info(self, relativePath, name=None):\n \"\"\"\n get file information dict from the repository given its relative path and name.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of the directory where the file is.\n #. name (string): The file name.\n If None is given, name will be split from relativePath.\n\n :Returns:\n #. info (None, dictionary): The file information dictionary.\n If None, it means an error has occurred.\n #. errorMessage (string): The error message if any error occurred.\n \"\"\"\n # normalize relative path and name\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n assert name != '.pyrepinfo', \"'.pyrepinfo' can't be a file name.\"\n if name is None:\n assert len(relativePath), \"name must be given when relative path is given as empty string or as a simple dot '.'\"\n relativePath,name = os.path.split(relativePath)\n # initialize message\n errorMessage = \"\"\n # get directory info\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n if dirInfoDict is None:\n return None, errorMessage\n # get file info\n fileInfo = dict.__getitem__(dirInfoDict, \"files\").get(name, None)\n if fileInfo is None:\n errorMessage = \"file %s does not exist in relative path '%s'\"%(name, relativePath)\n return fileInfo, errorMessage"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_file_info_by_id(self, id):\n for path, info in self.walk_files_info():\n if info['id']==id:\n return path, info\n # none was found\n return None, None", "response": "Given an id get the corresponding file info dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_file_relative_path_by_id(self, id):\n for path, info in self.walk_files_info():\n if info['id']==id:\n return path\n # none was found\n return None", "response": "Given an id get the corresponding file info relative path joined with file name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_file_relative_path_by_name(self, name, skip=0):\n if skip is None:\n paths = []\n else:\n paths = None\n for path, info in self.walk_files_info():\n _, n = os.path.split(path)\n if n==name:\n if skip is None:\n paths.append(path)\n elif skip>0:\n skip -= 1\n else:\n paths = path\n break\n return paths", "response": "Get file relative path given the file name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_directory(self, relativePath, info=None):\n path = os.path.normpath(relativePath)\n # create directories\n currentDir = self.path\n currentDict = self\n if path in (\"\",\".\"):\n return currentDict\n save = False\n for dir in path.split(os.sep):\n dirPath = os.path.join(currentDir, dir)\n # create directory\n if not os.path.exists(dirPath):\n os.mkdir(dirPath)\n # create dictionary key\n currentDict = dict.__getitem__(currentDict, \"directories\")\n if currentDict.get(dir, None) is None:\n save = True\n currentDict[dir] = {\"directories\":{}, \"files\":{},\n \"timestamp\":datetime.utcnow(),\n \"id\":str(uuid.uuid1()),\n \"info\": info} # INFO MUST BE SET ONLY FOR THE LAST DIRECTORY\n\n currentDict = currentDict[dir]\n currentDir = dirPath\n # save repository\n if save:\n self.save()\n # return currentDict\n return currentDict", "response": "Adds a directory in the repository and creates its\n attribute in the Repository with utc timestamp."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving a directory from the repository.", "response": "def remove_directory(self, relativePath, removeFromSystem=False):\n \"\"\"\n Remove directory from repository.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of the directory to remove from the repository.\n #. removeFromSystem (boolean): Whether to also remove directory and all files from the system.\\n\n Only files saved in the repository will be removed and empty left directories.\n \"\"\"\n # get parent directory info\n relativePath = os.path.normpath(relativePath)\n parentDirInfoDict, errorMessage = self.get_parent_directory_info(relativePath)\n assert parentDirInfoDict is not None, errorMessage\n # split path\n path, name = os.path.split(relativePath)\n if dict.__getitem__(parentDirInfoDict, 'directories').get(name, None) is None:\n raise Exception(\"'%s' is not a registered directory in repository relative path '%s'\"%(name, path))\n # remove from system\n if removeFromSystem:\n # remove files\n for rp in self.walk_files_relative_path(relativePath=relativePath):\n ap = os.path.join(self.__path, relativePath, rp)\n if not os.path.isfile(ap):\n continue\n if not os.path.exists(ap):\n continue\n if os.path.isfile(ap):\n os.remove( ap )\n # remove directories\n for rp in self.walk_directories_relative_path(relativePath=relativePath):\n ap = os.path.join(self.__path, relativePath, rp)\n if not os.path.isdir(ap):\n continue\n if not os.path.exists(ap):\n continue\n if not len(os.listdir(ap)):\n os.rmdir(ap)\n # pop directory from repo\n dict.__getitem__(parentDirInfoDict, 'directories').pop(name, None)\n ap = os.path.join(self.__path, relativePath)\n if not os.path.isdir(ap):\n if not len(os.listdir(ap)):\n os.rmdir(ap)\n # save repository\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef move_directory(self, relativePath, relativeDestination, replace=False, verbose=True):\n # normalize path\n relativePath = os.path.normpath(relativePath)\n relativeDestination = os.path.normpath(relativeDestination)\n # get files and directories\n filesInfo = list( self.walk_files_info(relativePath=relativePath) )\n dirsPath = list( self.walk_directories_relative_path(relativePath=relativePath) )\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n # remove directory info only\n self.remove_directory(relativePath=relativePath, removeFromSystem=False)\n # create new relative path\n self.add_directory(relativeDestination)\n # move files\n for RP, info in filesInfo:\n source = os.path.join(self.__path, relativePath, RP)\n destination = os.path.join(self.__path, relativeDestination, RP)\n # add directory\n newDirRP, fileName = os.path.split(os.path.join(relativeDestination, RP))\n dirInfoDict = self.add_directory( newDirRP )\n # move file\n if os.path.isfile(destination):\n if replace:\n os.remove(destination)\n if verbose:\n warnings.warn(\"file '%s' is copied replacing existing one in destination '%s'.\"%(fileName, newDirRP))\n else:\n if verbose:\n warnings.warn(\"file '%s' is not copied because the same file exists in destination '%s'.\"%(fileName,destination))\n continue\n os.rename(source, destination)\n # set file information\n dict.__getitem__(dirInfoDict, \"files\")[fileName] = info\n # save repository\n self.save()", "response": "Move a directory from one place to another."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rename_file(self, relativePath, name, newName, replace=False, verbose=True):\n # normalize path\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n # check directory in repository\n assert name in dict.__getitem__(dirInfoDict, \"files\"), \"file '%s' is not found in repository relative path '%s'\"%(name, relativePath)\n # get real path\n realPath = os.path.join(self.__path, relativePath, name)\n assert os.path.isfile(realPath), \"file '%s' is not found in system\"%realPath\n # assert directory new name doesn't exist in repository\n assert newName not in dict.__getitem__(dirInfoDict, \"files\"), \"file '%s' already exists in repository relative path '%s'\"%(newName, relativePath)\n # check new directory in system\n newRealPath = os.path.join(self.__path, relativePath, newName)\n if os.path.isfile( newRealPath ):\n if replace:\n os.remove(newRealPath)\n if verbose:\n warnings.warn( \"file '%s' already exists found in system, it is now replaced by '%s' because 'replace' flag is True.\"%(newRealPath,realPath) )\n else:\n raise Exception( \"file '%s' already exists in system but not registered in repository.\"%newRealPath )\n # rename file\n os.rename(realPath, newRealPath)\n dict.__setitem__( dict.__getitem__(dirInfoDict, \"files\"),\n newName,\n dict.__getitem__(dirInfoDict, \"files\").pop(name) )\n # save repository\n self.save()", "response": "Rename a file in the repository. It insures renaming the file in the system."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_file(self, relativePath, name=None, removeFromSystem=False):\n # get relative path normalized\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n assert name != '.pyrepinfo', \"'.pyrepinfo' is not allowed as file name in main repository directory\"\n assert name != '.pyrepstate', \"'.pyrepstate' is not allowed as file name in main repository directory\"\n assert name != '.pyreplock', \"'.pyreplock' is not allowed as file name in main repository directory\"\n if name is None:\n assert len(relativePath), \"name must be given when relative path is given as empty string or as a simple dot '.'\"\n relativePath, name = os.path.split(relativePath)\n # get file info dict\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n # check directory in repository\n assert name in dict.__getitem__(dirInfoDict, \"files\"), \"file '%s' is not found in repository relative path '%s'\"%(name, relativePath)\n # remove file from repo\n dict.__getitem__(dirInfoDict, \"files\").pop(name)\n # remove file from system\n if removeFromSystem:\n ap = os.path.join(self.__path, relativePath, name )\n if os.path.isfile(ap):\n os.remove( ap )\n # save repository\n self.save()", "response": "Removes a file from the repository."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndump an exisitng system file into the repository with utc timestamp.", "response": "def dump_copy(self, path, relativePath, name=None,\n description=None,\n replace=False, verbose=False):\n \"\"\"\n Copy an exisitng system file to the repository.\n attribute in the Repository with utc timestamp.\n\n :Parameters:\n #. path (str): The full path of the file to copy into the repository.\n #. relativePath (str): The relative to the repository path of the directory where the file should be dumped.\n If relativePath does not exist, it will be created automatically.\n #. name (string): The file name.\n If None is given, name will be split from path.\n #. description (None, string, pickable object): Any random description about the file.\n #. replace (boolean): Whether to replace any existing file with the same name if existing.\n #. verbose (boolean): Whether to be warn and informed about any abnormalities.\n \"\"\"\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n if name is None:\n _,name = os.path.split(path)\n # ensure directory added\n self.add_directory(relativePath)\n # ger real path\n realPath = os.path.join(self.__path, relativePath)\n # get directory info dict\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n if name in dict.__getitem__(dirInfoDict, \"files\"):\n if not replace:\n if verbose:\n warnings.warn(\"a file with the name '%s' is already defined in repository dictionary info. Set replace flag to True if you want to replace the existing file\"%(name))\n return\n # convert dump and pull methods to strings\n dump = \"raise Exception(\\\"dump is ambiguous for copied file '$FILE_PATH' \\\")\"\n pull = \"raise Exception(\\\"pull is ambiguous for copied file '$FILE_PATH' \\\")\"\n # dump file\n try:\n shutil.copyfile(path, os.path.join(realPath,name))\n except Exception as e:\n if verbose:\n warnings.warn(e)\n return\n # set info\n klass = None\n # save the new file to the repository\n dict.__getitem__(dirInfoDict, \"files\")[name] = {\"dump\":dump,\n \"pull\":pull,\n \"timestamp\":datetime.utcnow(),\n \"id\":str(uuid.uuid1()),\n \"class\": klass,\n \"description\":description}\n # save repository\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dump_file(self, value, relativePath, name=None,\n description=None, klass=None,\n dump=None, pull=None,\n replace=False, ACID=None, verbose=False):\n \"\"\"\n Dump a file using its value to the system and creates its\n attribute in the Repository with utc timestamp.\n\n :Parameters:\n #. value (object): The value of a file to dump and add to the repository. It is any python object or file.\n #. relativePath (str): The relative to the repository path of the directory where the file should be dumped.\n If relativePath does not exist, it will be created automatically.\n #. name (string): The file name.\n If None is given, name will be split from relativePath.\n #. description (None, string, pickable object): Any random description about the file.\n #. klass (None, class): The dumped object class. If None is given\n klass will be automatically set to the following value.__class__\n #. dump (None, string): The dumping method.\n If None it will be set automatically to pickle and therefore the object must be pickleable.\n If a string is given, the string should include all the necessary imports\n and a '$FILE_PATH' that replaces the absolute file path when the dumping will be performed.\\n\n e.g. \"import numpy as np; np.savetxt(fname='$FILE_PATH', X=value, fmt='%.6e')\"\n #. pull (None, string): The pulling method.\n If None it will be set automatically to pickle and therefore the object must be pickleable.\n If a string is given, the string should include all the necessary imports,\n a '$FILE_PATH' that replaces the absolute file path when the dumping will be performed\n and finally a PULLED_DATA variable.\\n\n e.g \"import numpy as np; PULLED_DATA=np.loadtxt(fname='$FILE_PATH')\"\n #. replace (boolean): Whether to replace any existing file with the same name if existing.\n #. ACID (None, boolean): Whether to ensure the ACID (Atomicity, Consistency, Isolation, Durability)\n properties of the repository upon dumping a file. This is ensured by dumping the file in\n a temporary path first and then moving it to the desired path.\n If None is given, repository ACID property will be used.\n #. verbose (boolean): Whether to be warn and informed about any abnormalities.\n \"\"\"\n # check ACID\n if ACID is None:\n ACID = self.__ACID\n assert isinstance(ACID, bool), \"ACID must be boolean\"\n # check name and path\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n assert name != '.pyrepinfo', \"'.pyrepinfo' is not allowed as file name in main repository directory\"\n assert name != '.pyrepstate', \"'.pyrepstate' is not allowed as file name in main repository directory\"\n assert name != '.pyreplock', \"'.pyreplock' is not allowed as file name in main repository directory\"\n if name is None:\n assert len(relativePath), \"name must be given when relative path is given as empty string or as a simple dot '.'\"\n relativePath,name = os.path.split(relativePath)\n # ensure directory added\n self.add_directory(relativePath)\n # get real path\n realPath = os.path.join(self.__path, relativePath)\n # get directory info dict\n dirInfoDict, errorMessage = self.get_directory_info(relativePath)\n assert dirInfoDict is not None, errorMessage\n if name in dict.__getitem__(dirInfoDict, \"files\"):\n if not replace:\n if verbose:\n warnings.warn(\"a file with the name '%s' is already defined in repository dictionary info. Set replace flag to True if you want to replace the existing file\"%(name))\n return\n # convert dump and pull methods to strings\n if dump is None:\n dump=DEFAULT_DUMP\n if pull is None:\n pull=DEFAULT_PULL\n # get savePath\n if ACID:\n savePath = os.path.join(tempfile.gettempdir(), str(uuid.uuid1()))\n else:\n savePath = os.path.join(realPath,name)\n # dump file\n try:\n exec( dump.replace(\"$FILE_PATH\", str(savePath)) )\n except Exception as e:\n message = \"unable to dump the file (%s)\"%e\n if 'pickle.dump(' in dump:\n message += '\\nmore info: %s'%str(get_pickling_errors(value))\n raise Exception( message )\n # copy if ACID\n if ACID:\n try:\n shutil.copyfile(savePath, os.path.join(realPath,name))\n except Exception as e:\n os.remove(savePath)\n if verbose:\n warnings.warn(e)\n return\n os.remove(savePath)\n # set info\n if klass is None and value is not None:\n klass = value.__class__\n if klass is not None:\n assert inspect.isclass(klass), \"klass must be a class definition\"\n # MUST TRY PICLKING KLASS TEMPORARILY FIRST\n # save the new file to the repository\n dict.__getitem__(dirInfoDict, \"files\")[name] = {\"dump\":dump,\n \"pull\":pull,\n \"timestamp\":datetime.utcnow(),\n \"id\":str(uuid.uuid1()),\n \"class\": klass,\n \"description\":description}\n # save repository\n self.save()", "response": "Dump a file into the repository."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the value and the utc timestamp of a file in the repository.", "response": "def update_file(self, value, relativePath, name=None,\n description=False, klass=False,\n dump=False, pull=False,\n ACID=None, verbose=False):\n \"\"\"\n Update the value and the utc timestamp of a file that is already in the Repository.\\n\n If file is not registered in repository, and error will be thrown.\\n\n If file is missing in the system, it will be regenerated as dump method is called.\n\n :Parameters:\n #. value (object): The value of the file to update. It is any python object or a file.\n #. relativePath (str): The relative to the repository path of the directory where the file should be dumped.\n #. name (None, string): The file name.\n If None is given, name will be split from relativePath.\n #. description (False, string, pickable object): Any random description about the file.\n If False is given, the description info won't be updated,\n otherwise it will be update to what description argument value is.\n #. klass (False, class): The dumped object class. If False is given,\n the class info won't be updated, otherwise it will be update to what klass argument value is.\n #. dump (False, string): The new dump method. If False is given, the old one will be used.\n #. pull (False, string): The new pull method. If False is given, the old one will be used.\n #. ACID (boolean): Whether to ensure the ACID (Atomicity, Consistency, Isolation, Durability)\n properties of the repository upon dumping a file. This is ensured by dumping the file in\n a temporary path first and then moving it to the desired path.\n If None is given, repository ACID property will be used.\n #. verbose (boolean): Whether to be warn and informed about any abnormalities.\n \"\"\"\n # check ACID\n if ACID is None:\n ACID = self.__ACID\n assert isinstance(ACID, bool), \"ACID must be boolean\"\n # get relative path normalized\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n assert name != '.pyrepinfo', \"'.pyrepinfo' is not allowed as file name in main repository directory\"\n assert name != '.pyrepstate', \"'.pyrepstate' is not allowed as file name in main repository directory\"\n assert name != '.pyreplock', \"'.pyreplock' is not allowed as file name in main repository directory\"\n if name is None:\n assert len(relativePath), \"name must be given when relative path is given as empty string or as a simple dot '.'\"\n relativePath,name = os.path.split(relativePath)\n # get file info dict\n fileInfoDict, errorMessage = self.get_file_info(relativePath, name)\n assert fileInfoDict is not None, errorMessage\n # get real path\n realPath = os.path.join(self.__path, relativePath)\n # check if file exists\n if verbose:\n if not os.path.isfile( os.path.join(realPath, name) ):\n warnings.warn(\"file '%s' is in repository but does not exist in the system. It is therefore being recreated.\"%os.path.join(realPath, name))\n # convert dump and pull methods to strings\n if not dump:\n dump = fileInfoDict[\"dump\"]\n if not pull:\n pull = fileInfoDict[\"pull\"]\n # get savePath\n if ACID:\n savePath = os.path.join(tempfile.gettempdir(), name)\n else:\n savePath = os.path.join(realPath,name)\n # dump file\n try:\n exec( dump.replace(\"$FILE_PATH\", str(savePath)) )\n except Exception as e:\n message = \"unable to dump the file (%s)\"%e\n if 'pickle.dump(' in dump:\n message += '\\nmore info: %s'%str(get_pickling_errors(value))\n raise Exception( message )\n # copy if ACID\n if ACID:\n try:\n shutil.copyfile(savePath, os.path.join(realPath,name))\n except Exception as e:\n os.remove(savePath)\n if verbose:\n warnings.warn(e)\n return\n os.remove(savePath)\n # update timestamp\n fileInfoDict[\"timestamp\"] = datetime.utcnow()\n if description is not False:\n fileInfoDict[\"description\"] = description\n if klass is not False:\n assert inspect.isclass(klass), \"klass must be a class definition\"\n fileInfoDict[\"class\"] = klass\n # save repository\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pull_file(self, relativePath, name=None, pull=None, update=True):\n # get relative path normalized\n relativePath = os.path.normpath(relativePath)\n if relativePath == '.':\n relativePath = ''\n assert name != '.pyrepinfo', \"pulling '.pyrepinfo' from main repository directory is not allowed.\"\n assert name != '.pyrepstate', \"pulling '.pyrepstate' from main repository directory is not allowed.\"\n assert name != '.pyreplock', \"pulling '.pyreplock' from main repository directory is not allowed.\"\n if name is None:\n assert len(relativePath), \"name must be given when relative path is given as empty string or as a simple dot '.'\"\n relativePath,name = os.path.split(relativePath)\n # get file info\n fileInfo, errorMessage = self.get_file_info(relativePath, name)\n assert fileInfo is not None, errorMessage\n # get absolute path\n realPath = os.path.join(self.__path, relativePath)\n assert os.path.exists(realPath), \"relative path '%s'within repository '%s' does not exist\"%(relativePath, self.__path)\n # file path\n fileAbsPath = os.path.join(realPath, name)\n assert os.path.isfile(fileAbsPath), \"file '%s' does not exist in absolute path '%s'\"%(name,realPath)\n if pull is None:\n pull = fileInfo[\"pull\"]\n # try to pull file\n try:\n namespace = {}\n namespace.update( globals() )\n exec( pull.replace(\"$FILE_PATH\", str(os.path.join(realPath,name)) ), namespace )\n except Exception as e:\n m = pull.replace(\"$FILE_PATH\", str(os.path.join(realPath,name)) )\n raise Exception( \"unable to pull data using '%s' from file (%s)\"%(m,e) )\n # update\n if update:\n fileInfo[\"pull\"] = pull\n # return data\n return namespace['PULLED_DATA']", "response": "Pulls a file from the repository."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ensure_str(value):\n if isinstance(value, six.string_types):\n return value\n else:\n return six.text_type(value)", "response": "Ensure value is string."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ensure_list(path_or_path_list):\n if isinstance(path_or_path_list, (tuple, list, set)):\n return [ensure_str(path) for path in path_or_path_list]\n else:\n return [ensure_str(path_or_path_list), ]", "response": "Ensures that the input argument is a list of paths."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stream(self, report):\n with self.ClientSession() as session:\n lines = []\n for job in report['traces']:\n key = '%s:%s' % (self.name, job)\n for minute in report['traces'][job]:\n for k, v in report['traces'][job][minute].items():\n lines.append('# TYPE %s_%s gauge' % (key, k))\n lines.append('%s_%s %0.2f' % (key, k, v))\n\n # Empty is required at the end of the payload\n lines.append(\"\")\n data = \"\\n\".join(lines)\n logger.info(data)\n yield from session.post(self.url, data=bytes(data.encode('utf-8')))", "response": "Stream reports to application logs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stream(self, report):\n\n payload = {\n \"agent\": {\n \"host\": report['instance']['hostname'],\n \"version\": \"1.0.0\"\n },\n \"components\": [\n {\n \"name\": self.name,\n \"guid\": \"com.darwinmonroy.aiometrics\",\n \"duration\": 60,\n \"metrics\": {\n 'Component/{}'.format(key): {\n \"total\": metric['count'] * metric['avg'],\n \"count\": metric['count'],\n \"min\": metric['min'],\n \"max\": metric['max'],\n \"sum_of_squares\": metric['min']**2 + metric['max']**2,\n } for key, metric in report['traces'].items()\n }\n }\n ]\n }\n\n with self.ClientSession() as session:\n\n try:\n r = yield from session.post(\n 'https://platform-api.newrelic.com/platform/v1/metrics',\n data=json.dumps(payload),\n headers=(\n ('X-License-Key', self.license_key),\n ('Content-Type', 'application/json'),\n ('Accept', 'application/json'),\n )\n )\n r.close()\n except Exception as e:\n # Any exception should affect the execution of the main\n # program, so we must explicitly silence any error caused by\n # by the streaming of metrics\n # TODO: consider the implementation of a retry logic\n logger.exception(e)", "response": "Stream reports to application logs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds per minute stats for each key", "response": "def stats(cls, traces):\n \"\"\"Build per minute stats for each key\"\"\"\n\n data = {}\n stats = {}\n # Group traces by key and minute\n for trace in traces:\n key = trace['key']\n if key not in data:\n data[key] = []\n stats[key] = {}\n\n data[key].append(trace['total_time'])\n cls._traces.pop(trace['id'])\n\n for key in data:\n times = data[key]\n stats[key] = dict(\n count=len(times),\n max=max(times),\n min=min(times),\n avg=sum(times)/len(times)\n )\n\n return stats"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef no_error_extract_data_from_file(self, file_path: str) -> Iterable[DataSourceType]:\n try:\n return self.extract_data_from_file(file_path)\n except Exception as e:\n logging.warning(e)\n return []", "response": "Proxy for extract_data_from_file that suppresses any errors and instead just returning an empty list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading all of the data from the files in directory location.", "response": "def _load_all_in_directory(self) -> Dict[str, Iterable[DataSourceType]]:\n \"\"\"\n Loads all of the data from the files in directory location.\n :return: a origin map of all the loaded data\n \"\"\"\n origin_mapped_data = dict() # type: Dict[str, Iterable[DataSourceType]]\n for file_path in glob.iglob(\"%s/**/*\" % self._directory_location, recursive=True):\n if self.is_data_file(file_path):\n origin_mapped_data[file_path] = self.no_error_extract_data_from_file(file_path)\n return origin_mapped_data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _extract_data_from_origin_map(origin_mapped_data: Dict[str, Iterable[DataSourceType]]) \\\n -> Iterable[DataSourceType]:\n \"\"\"\n Extracts the data from a data origin map.\n :param origin_mapped_data: a map containing the origin of the data as the key string and the data as the value\n :return: the data contained within the map\n \"\"\"\n data = []\n for _, data_item in origin_mapped_data.items():\n data.extend(data_item)\n return data", "response": "Extracts the data from an origin map."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nstarts monitoring data kept in files in the predefined directory.", "response": "def start(self):\n \"\"\"\n Monitors data kept in files in the predefined directory in a new thread.\n\n Note: Due to the underlying library, it may take a few milliseconds after this method is started for changes to\n start to being noticed.\n \"\"\"\n with self._status_lock:\n if self._running:\n raise RuntimeError(\"Already running\")\n self._running = True\n\n # Cannot re-use Observer after stopped\n self._observer = Observer()\n self._observer.schedule(self._event_handler, self._directory_location, recursive=True)\n self._observer.start()\n\n # Load all in directory afterwards to ensure no undetected changes between loading all and observing\n self._origin_mapped_data = self._load_all_in_directory()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nstops monitoring the predefined directory.", "response": "def stop(self):\n \"\"\"\n Stops monitoring the predefined directory.\n \"\"\"\n with self._status_lock:\n if self._running:\n assert self._observer is not None\n self._observer.stop()\n self._running = False\n self._origin_mapped_data = dict()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncall when a file in the monitored directory has been created.", "response": "def _on_file_created(self, event: FileSystemEvent):\n \"\"\"\n Called when a file in the monitored directory has been created.\n :param event: the file system event\n \"\"\"\n if not event.is_directory and self.is_data_file(event.src_path):\n assert event.src_path not in self._origin_mapped_data\n self._origin_mapped_data[event.src_path] = self.no_error_extract_data_from_file(event.src_path)\n self.notify_listeners(FileSystemChange.CREATE)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall when a file in the monitored directory has been modified.", "response": "def _on_file_modified(self, event: FileSystemEvent):\n \"\"\"\n Called when a file in the monitored directory has been modified.\n :param event: the file system event\n \"\"\"\n if not event.is_directory and self.is_data_file(event.src_path):\n assert event.src_path in self._origin_mapped_data\n self._origin_mapped_data[event.src_path] = self.no_error_extract_data_from_file(event.src_path)\n self.notify_listeners(FileSystemChange.MODIFY)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _on_file_deleted(self, event: FileSystemEvent):\n if not event.is_directory and self.is_data_file(event.src_path):\n assert event.src_path in self._origin_mapped_data\n del(self._origin_mapped_data[event.src_path])\n self.notify_listeners(FileSystemChange.DELETE)", "response": "Called when a file in the monitored directory has been deleted."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _on_file_moved(self, event: FileSystemMovedEvent):\n if not event.is_directory and self.is_data_file(event.src_path):\n delete_event = FileSystemEvent(event.src_path)\n delete_event.event_type = EVENT_TYPE_DELETED\n self._on_file_deleted(delete_event)\n\n create_event = FileSystemEvent(event.dest_path)\n create_event.event_type = EVENT_TYPE_CREATED\n self._on_file_created(create_event)", "response": "Called when a file in the monitored directory has been moved."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tear_down(self):\n while len(self._temp_directories) > 0:\n directory = self._temp_directories.pop()\n shutil.rmtree(directory, ignore_errors=True)\n while len(self._temp_files) > 0:\n file = self._temp_files.pop()\n try:\n os.remove(file)\n except OSError:\n pass", "response": "Tear down all temp files and directories."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_temp_directory(self, **mkdtemp_kwargs) -> str:\n kwargs = {**self.default_mkdtemp_kwargs, **mkdtemp_kwargs}\n location = tempfile.mkdtemp(**kwargs)\n self._temp_directories.add(location)\n return location", "response": "Creates a temp directory."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_temp_file(self, **mkstemp_kwargs) -> Tuple[int, str]:\n kwargs = {**self.default_mkstemp_kwargs, **mkstemp_kwargs}\n handle, location = tempfile.mkstemp(**kwargs)\n self._temp_files.add(location)\n return handle, location", "response": "Creates a temp file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef change(self,\n new_abspath=None,\n new_dirpath=None,\n new_dirname=None,\n new_basename=None,\n new_fname=None,\n new_ext=None):\n \"\"\"\n Return a new :class:`pathlib_mate.pathlib2.Path` object with updated information.\n \"\"\"\n if new_abspath is not None:\n p = self.__class__(new_abspath)\n return p\n\n if (new_dirpath is None) and (new_dirname is not None):\n new_dirpath = os.path.join(self.parent.dirpath, new_dirname)\n\n elif (new_dirpath is not None) and (new_dirname is None):\n new_dirpath = new_dirpath\n\n elif (new_dirpath is None) and (new_dirname is None):\n new_dirpath = self.dirpath\n\n elif (new_dirpath is not None) and (new_dirname is not None):\n raise ValueError(\"Cannot having both new_dirpath and new_dirname!\")\n\n if new_basename is None:\n if new_fname is None:\n new_fname = self.fname\n if new_ext is None:\n new_ext = self.ext\n new_basename = new_fname + new_ext\n else:\n if new_fname is not None or new_ext is not None:\n raise ValueError(\"Cannot having both new_basename, \"\n \"new_fname, new_ext!\")\n\n return self.__class__(new_dirpath, new_basename)", "response": "Change the contents of the current pathlib_mate. pathlib2. Path object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntesting whether a file target is not exists or allow overwrite.", "response": "def is_not_exist_or_allow_overwrite(self, overwrite=False):\n \"\"\"\n Test whether a file target is not exists or it exists but allow\n overwrite.\n \"\"\"\n if self.exists() and overwrite is False:\n return False\n else: # pragma: no cover\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmove the path to another location.", "response": "def moveto(self,\n new_abspath=None,\n new_dirpath=None,\n new_dirname=None,\n new_basename=None,\n new_fname=None,\n new_ext=None,\n overwrite=False,\n makedirs=False):\n \"\"\"\n An advanced :meth:`pathlib_mate.pathlib2.Path.rename` method provide ability to rename by\n each components of a path. A new ``Path`` instance will returns.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u9ad8\u7ea7\u91cd\u547d\u540d\u51fd\u6570, \u5141\u8bb8\u7528\u4e8e\u6839\u636e\u8def\u5f84\u7684\u5404\u4e2a\u7ec4\u6210\u90e8\u5206\u8fdb\u884c\u91cd\u547d\u540d\u3002\u4f46\u548cos.rename\n \u65b9\u6cd5\u4e00\u6837, \u9700\u8981\u4fdd\u8bc1\u6bcd\u6587\u4ef6\u5939\u5b58\u5728\u3002\n \"\"\"\n self.assert_exists()\n\n p = self.change(\n new_abspath=new_abspath,\n new_dirpath=new_dirpath,\n new_dirname=new_dirname,\n new_basename=new_basename,\n new_fname=new_fname,\n new_ext=new_ext,\n )\n\n if p.is_not_exist_or_allow_overwrite(overwrite=overwrite):\n # \u5982\u679c\u4e24\u4e2a\u8def\u5f84\u4e0d\u540c, \u624d\u8fdb\u884cmove\n if self.abspath != p.abspath:\n if makedirs:\n parent = p.parent\n if not parent.exists():\n os.makedirs(parent.abspath)\n self.rename(p)\n return p"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncopies this file to another place.", "response": "def copyto(self,\n new_abspath=None,\n new_dirpath=None,\n new_dirname=None,\n new_basename=None,\n new_fname=None,\n new_ext=None,\n overwrite=False,\n makedirs=False):\n \"\"\"\n Copy this file to other place.\n \"\"\"\n self.assert_exists()\n\n p = self.change(\n new_abspath=new_abspath,\n new_dirpath=new_dirpath,\n new_dirname=new_dirname,\n new_basename=new_basename,\n new_fname=new_fname,\n new_ext=new_ext,\n )\n\n if p.is_not_exist_or_allow_overwrite(overwrite=overwrite):\n # \u5982\u679c\u4e24\u4e2a\u8def\u5f84\u4e0d\u540c, \u624d\u8fdb\u884ccopy\n if self.abspath != p.abspath:\n try:\n shutil.copy(self.abspath, p.abspath)\n except IOError as e:\n if makedirs:\n os.makedirs(p.parent.abspath)\n shutil.copy(self.abspath, p.abspath)\n else:\n raise e\n return p"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _create_client(base_url: str, tls: TLSConfig=False) -> Optional[APIClient]:\n try:\n client = APIClient(base_url=base_url, tls=tls, version=\"auto\")\n return client if client.ping() else None\n except:\n return None", "response": "Creates a Docker client with the given details."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a Docker client.", "response": "def create_client() -> APIClient:\n \"\"\"\n Clients a Docker client.\n\n Will raise a `ConnectionError` if the Docker daemon is not accessible.\n :return: the Docker client\n \"\"\"\n global _client\n client = _client()\n if client is None:\n # First try looking at the environment variables for specification of the daemon's location\n docker_environment = kwargs_from_env(assert_hostname=False)\n if \"base_url\" in docker_environment:\n client = _create_client(docker_environment.get(\"base_url\"), docker_environment.get(\"tls\"))\n if client is None:\n raise ConnectionError(\n \"Could not connect to the Docker daemon specified by the `DOCKER_X` environment variables: %s\"\n % docker_environment)\n else:\n logging.info(\"Connected to Docker daemon specified by the environment variables\")\n else:\n # Let's see if the Docker daemon is accessible via the UNIX socket\n client = _create_client(\"unix://var/run/docker.sock\")\n if client is not None:\n logging.info(\"Connected to Docker daemon running on UNIX socket\")\n else:\n raise ConnectionError(\n \"Cannot connect to Docker - is the Docker daemon running? `$DOCKER_HOST` should be set or the \"\n \"daemon should be accessible via the standard UNIX socket.\")\n _client = weakref.ref(client)\n assert isinstance(client, APIClient)\n return client"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_dump_method(dump, protocol=-1):\n if dump is None:\n dump = 'pickle'\n if dump.startswith('pickle'):\n if dump == 'pickle':\n proto = protocol\n else:\n proto = dump.strip('pickle')\n try:\n proto = int(proto)\n assert proto>=-1\n except:\n raise Exception(\"protocol must be an integer >=-1\")\n code = \"\"\"\ntry:\n import cPickle as pickle\nexcept:\n import pickle\nwith open('$FILE_PATH', 'wb') as fd:\n pickle.dump( value, fd, protocol=%i )\n fd.flush()\n os.fsync(fd.fileno())\n\"\"\"%proto\n elif dump.startswith('dill'):\n if dump == 'dill':\n proto = 2\n else:\n proto = dump.strip('dill')\n try:\n proto = int(proto)\n assert proto>=-1\n except:\n raise Exception(\"protocol must be an integer >=-1\")\n code = \"\"\"\nimport dill\nwith open('$FILE_PATH', 'wb') as fd:\n dill.dump( value, fd, protocol=%i )\n fd.flush()\n os.fsync(fd.fileno())\n\"\"\"%proto\n elif dump == 'json':\n code = \"\"\"\nimport json\nwith open('$FILE_PATH', 'wb') as fd:\n json.dump( value,fd, ensure_ascii=True, indent=4 )\n fd.flush()\n os.fsync(fd.fileno())\n\"\"\"\n elif dump == 'numpy':\n code = \"\"\"\nimport numpy\nwith open('$FILE_PATH', 'wb') as fd:\n numpy.save(file=fd, arr=value)\n fd.flush()\n os.fsync(fd.fileno())\n\"\"\"\n elif dump == 'numpy_text':\n code = \"\"\"\nimport numpy\nnumpy.savetxt(fname='$FILE_PATH', X=value, fmt='%.6e')\n\"\"\"\n else:\n assert isinstance(dump, basestring), \"dump must be None or a string\"\n assert '$FILE_PATH' in dump, \"string dump code must inlcude '$FILE_PATH'\"\n code = dump\n # return\n return code", "response": "Get dump function code string"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget pull function code string", "response": "def get_pull_method(pull):\n \"\"\"Get pull function code string\"\"\"\n if pull is None or pull.startswith('pickle'):\n code = \"\"\"\nimport os\ntry:\n import cPickle as pickle\nexcept:\n import pickle\nwith open('$FILE_PATH', 'rb') as fd:\n PULLED_DATA = pickle.load( fd )\n\"\"\"\n elif pull.startswith('dill'):\n code = \"\"\"\nimport dill\nwith open('$FILE_PATH', 'rb') as fd:\n PULLED_DATA = dill.load( fd )\n\"\"\"\n elif pull == 'json':\n code = \"\"\"\nimport json\nwith open('$FILE_PATH', 'rb') as fd:\n PULLED_DATA = json.load(fd)\n\"\"\"\n elif pull == 'numpy':\n code = \"\"\"\nimport numpy\nwith open('$FILE_PATH', 'rb') as fd:\n PULLED_DATA=numpy.load(file=fd)\n\n\"\"\"\n elif pull == 'numpy_text':\n code = \"\"\"\nimport numpy\nwith open('$FILE_PATH', 'rb') as fd:\n PULLED_DATA=numpy.loadtxt(fname=fd)\n\"\"\"\n else:\n assert isinstance(pull, basestring), \"pull must be None or a string\"\n assert 'PULLED_DATA' in pull, \"string pull code must inlcude 'PULLED_DATA'\"\n assert '$FILE_PATH' in pull, \"string pull code must inlcude '$FILE_PATH'\"\n code = pull\n # return\n return code"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef path_required(func):\n @wraps(func)\n def wrapper(self, *args, **kwargs):\n if self.path is None:\n warnings.warn('Must load (Repository.load_repository) or initialize (Repository.create_repository) the repository first !')\n return\n return func(self, *args, **kwargs)\n return wrapper", "response": "Decorate methods when repository path is required."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncleaning repository given before and after states", "response": "def __clean_before_after(self, stateBefore, stateAfter, keepNoneEmptyDirectory=True):\n \"\"\"clean repository given before and after states\"\"\"\n # prepare after for faster search\n errors = []\n afterDict = {}\n [afterDict.setdefault(list(aitem)[0],[]).append(aitem) for aitem in stateAfter]\n # loop before\n for bitem in reversed(stateBefore):\n relaPath = list(bitem)[0]\n basename = os.path.basename(relaPath)\n btype = bitem[relaPath]['type']\n alist = afterDict.get(relaPath, [])\n aitem = [a for a in alist if a[relaPath]['type']==btype]\n if len(aitem)>1:\n errors.append(\"Multiple '%s' of type '%s' where found in '%s', this should never had happened. Please report issue\"%(basename,btype,relaPath))\n continue\n if not len(aitem):\n removeDirs = []\n removeFiles = []\n if btype == 'dir':\n if not len(relaPath):\n errors.append(\"Removing main repository directory is not allowed\")\n continue\n removeDirs.append(os.path.join(self.__path,relaPath))\n removeFiles.append(os.path.join(self.__path,relaPath,self.__dirInfo))\n removeFiles.append(os.path.join(self.__path,relaPath,self.__dirLock))\n elif btype == 'file':\n removeFiles.append(os.path.join(self.__path,relaPath))\n removeFiles.append(os.path.join(self.__path,relaPath,self.__fileInfo%basename))\n removeFiles.append(os.path.join(self.__path,relaPath,self.__fileLock%basename))\n else:\n ### MUST VERIFY THAT ONCE pyrepobjectdir IS IMPLEMENTED\n removeDirs.append(os.path.join(self.__path,relaPath))\n removeFiles.append(os.path.join(self.__path,relaPath,self.__fileInfo%basename))\n # remove files\n for fpath in removeFiles:\n if os.path.isfile(fpath):\n try:\n os.remove(fpath)\n except Exception as err:\n errors.append(\"Unable to clean file '%s' (%s)\"%(fpath, str(err)))\n # remove directories\n for dpath in removeDirs:\n if os.path.isdir(dpath):\n if keepNoneEmptyDirectory or not len(os.listdir(dpath)):\n try:\n shutil.rmtree(dpath)\n except Exception as err:\n errors.append(\"Unable to clean directory '%s' (%s)\"%(fpath, str(err)))\n # return result and errors list\n return len(errors)==0, errors"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting repository descriptive stats", "response": "def get_stats(self):\n \"\"\"\n Get repository descriptive stats\n\n :Returns:\n #. numberOfDirectories (integer): Number of diretories in repository\n #. numberOfFiles (integer): Number of files in repository\n \"\"\"\n if self.__path is None:\n return 0,0\n nfiles = 0\n ndirs = 0\n for fdict in self.get_repository_state():\n fdname = list(fdict)[0]\n if fdname == '':\n continue\n if fdict[fdname].get('pyrepfileinfo', False):\n nfiles += 1\n elif fdict[fdname].get('pyrepdirinfo', False):\n ndirs += 1\n else:\n raise Exception('Not sure what to do next. Please report issue')\n return ndirs,nfiles"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreset the internal state of the object.", "response": "def reset(self):\n \"\"\"Reset repository instance.\n \"\"\"\n self.__path = None\n self.__repo = {'repository_unique_name': str(uuid.uuid1()),\n 'create_utctime': time.time(),\n 'last_update_utctime': None,\n 'pyrep_version': str(__version__),\n 'repository_information': '',\n 'walk_repo': []}"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_repository(self, path):\n if path.strip() in ('','.'):\n path = os.getcwd()\n repoPath = os.path.realpath( os.path.expanduser(path) )\n if os.path.isfile( os.path.join(repoPath,self.__repoFile) ):\n return True\n else:\n try:\n from .OldRepository import Repository\n REP = Repository()\n result = REP.is_repository(repoPath)\n except:\n return False\n else:\n if result:\n warnings.warn(\"This is an old repository version 2.x.y! Make sure to start using repositories 3.x.y \")\n return result", "response": "Check if there is a repository in path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load_repository(self, path, verbose=True, ntrials=3):\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n repo = None\n for _trial in range(ntrials):\n try:\n self.__load_repository(path=path, verbose=True)\n except Exception as err1:\n try:\n from .OldRepository import Repository\n REP = Repository(path)\n except Exception as err2:\n #traceback.print_exc()\n error = \"Unable to load repository using neiher new style (%s) nor old style (%s)\"%(err1, err2)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n repo = REP\n break\n else:\n error = None\n repo = self\n break\n # check and return\n assert error is None, error\n return repo", "response": "Load a repository from a directory path and update the current instance with the loaded data."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new repository in a directory.", "response": "def create_repository(self, path, info=None, description=None, replace=True, allowNoneEmpty=True, raiseError=True):\n \"\"\"\n create a repository in a directory. This method insures the creation of\n the directory in the system if it is missing.\\n\n\n **N.B. If replace is True and existing repository is found in path, create_repository erases all existing files and directories in path.**\n\n :Parameters:\n #. path (string): The real absolute path where to create the Repository.\n If '.' or an empty string is passed, the current working directory will be used.\n #. description (None, str): Repository main directory information.\n #. info (None, object): Repository information. It can\n be None or any pickle writable type of data.\n #. replace (boolean): Whether to replace existing repository.\n #. allowNoneEmpty (boolean): Allow creating repository in none-empty\n directory.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n\n :Returns:\n #. success (boolean): Whether creating repository was successful\n #. message (None, str): Any returned message.\n \"\"\"\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(allowNoneEmpty, bool), \"allowNoneEmpty must be boolean\"\n assert isinstance(replace, bool), \"replace must be boolean\"\n assert isinstance(path, basestring), \"path must be string\"\n if info is None:\n info = ''\n try:\n pickle.dumps(info)\n except Exception as err:\n raise Exception(\"info must be None or any pickle writable type of data (%s)\"%str(err))\n #assert isinstance(info, basestring), \"info must be None or a string\"\n if description is None:\n description = ''\n assert isinstance(description, basestring), \"description must be None or a string\"\n # get real path\n if path.strip() in ('','.'):\n path = os.getcwd()\n realPath = os.path.realpath( os.path.expanduser(path) )\n # reset if replace is set to True\n message = []\n if self.is_repository(realPath):\n if not replace:\n message.append(\"A pyrep Repository already exists in the given path '%s' set replace to True if you need to proceed.\"%path)\n return False, message\n else:\n message.append(\"Old existing pyrep repository existing in the given path '%s' has been replaced.\"%path)\n try:\n for _df in os.listdir(realPath):\n _p = os.path.join(realPath, _df)\n if os.path.isdir(_p):\n shutil.rmtree( _p )\n else:\n os.remove(_p)\n except Exception as err:\n message.append(\"Unable to clean remove repository before create (%s)\"%(str(err)))\n return False, '\\n'.join(message)\n if not os.path.isdir(realPath):\n os.makedirs(realPath)\n elif len(os.listdir(realPath)) and not allowNoneEmpty:\n return False, \"Not allowed to create repository in a non empty directory\"\n # reset repository\n oldRepo = self.__repo\n self.reset()\n self.__path = realPath.rstrip(os.sep)\n self.__repo['repository_information'] = info\n # save repository\n saved = self.save(description=description)\n if not saved:\n self.__repo = oldRepo\n message.append(\"Absolute path and directories might be created but no pyrep Repository is created.\")\n return False, '\\n'.join(message)\n # return\n return True, '\\n'.join(message)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_repository(self, path=None, removeEmptyDirs=True):\n assert isinstance(removeEmptyDirs, bool), \"removeEmptyDirs must be boolean\"\n if path is not None:\n if path != self.__path:\n repo = Repository()\n repo.load_repository(path)\n else:\n repo = self\n else:\n repo = self\n assert repo.path is not None, \"path is not given and repository is not initialized\"\n # remove repo files and directories\n for fdict in reversed(repo.get_repository_state()):\n relaPath = list(fdict)[0]\n realPath = os.path.join(repo.path, relaPath)\n path, name = os.path.split(realPath)\n if fdict[relaPath]['type'] == 'file':\n if os.path.isfile(realPath):\n os.remove(realPath)\n if os.path.isfile(os.path.join(repo.path,path,self.__fileInfo%name)):\n os.remove(os.path.join(repo.path,path,self.__fileInfo%name))\n if os.path.isfile(os.path.join(repo.path,path,self.__fileLock%name)):\n os.remove(os.path.join(repo.path,path,self.__fileLock%name))\n if os.path.isfile(os.path.join(repo.path,path,self.__fileClass%name)):\n os.remove(os.path.join(repo.path,path,self.__fileClass%name))\n elif fdict[relaPath]['type'] == 'dir':\n if os.path.isfile(os.path.join(realPath,self.__dirInfo)):\n os.remove(os.path.join(realPath,self.__dirInfo))\n if os.path.isfile(os.path.join(realPath,self.__dirLock)):\n os.remove(os.path.join(realPath,self.__dirLock))\n if not len(os.listdir(realPath)) and removeEmptyDirs:\n shutil.rmtree( realPath )\n # remove repo information file\n if os.path.isfile(os.path.join(repo.path,self.__repoFile)):\n os.remove(os.path.join(repo.path,self.__repoFile))\n if os.path.isfile(os.path.join(repo.path,self.__repoLock)):\n os.remove(os.path.join(repo.path,self.__repoLock))", "response": "Removes all files and directories in the repository at the given path along with all repository tracked files and directories."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsave the current repository. pyreprepo file to disk and create a new lock.", "response": "def save(self, description=None, raiseError=True, ntrials=3):\n \"\"\"\n Save repository '.pyreprepo' to disk and create (if missing) or\n update (if description is not None) '.pyrepdirinfo'.\n\n :Parameters:\n #. description (None, str): Repository main directory information.\n If given will be replaced.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (bool): Whether saving was successful.\n #. error (None, string): Fail to save repository message in case\n saving is not successful. If success is True, error will be None.\n \"\"\"\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # get description\n if description is not None:\n assert isinstance(description, basestring), \"description must be None or a string\"\n dirInfoPath = os.path.join(self.__path, self.__dirInfo)\n if description is None and not os.path.isfile(dirInfoPath):\n description = ''\n # create and acquire lock\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n # check if acquired.\n m = \"code %s. Unable to aquire the lock when calling 'save'. You may try again!\"%(code,)\n if not acquired:\n assert not raiseError, Exception(m)\n return False, m\n # save repository\n for _trial in range(ntrials):\n try:\n # open file\n repoInfoPath = os.path.join(self.__path, self.__repoFile)\n error = None\n self.__save_dirinfo(description=description, dirInfoPath=dirInfoPath)\n # load and update repository info if existing\n if os.path.isfile(repoInfoPath):\n with open(repoInfoPath, 'rb') as fd:\n repo = self.__load_repository_pickle_file(os.path.join(self.__path, self.__repoFile))\n self.__repo['walk_repo'] = repo['walk_repo']\n # create repository\n with open(repoInfoPath, 'wb') as fd:\n self.__repo[\"last_update_utctime\"] = time.time()\n pickle.dump( self.__repo,fd, protocol=self._DEFAULT_PICKLE_PROTOCOL )\n fd.flush()\n os.fsync(fd.fileno())\n except Exception as err:\n error = \"Unable to save repository (%s)\"%err\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n break\n # release lock\n LR.release_lock()\n # return\n assert error is None or not raiseError, error\n return error is None, error"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_name_allowed(self, path):\n assert isinstance(path, basestring), \"given path must be a string\"\n name = os.path.basename(path)\n if not len(name):\n return False, \"empty name is not allowed\"\n # exact match\n for em in [self.__repoLock,self.__repoFile,self.__dirInfo,self.__dirLock]:\n if name == em:\n return False, \"name '%s' is reserved for pyrep internal usage\"%em\n # pattern match\n for pm in [self.__fileInfo,self.__fileLock]:#,self.__objectDir]:\n if name == pm or (name.endswith(pm[3:]) and name.startswith('.')):\n return False, \"name pattern '%s' is not allowed as result may be reserved for pyrep internal usage\"%pm\n # name is ok\n return True, None", "response": "Checks whether the given path is allowed for the given file or a directory."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_repo_relative_path(self, path, split=False):\n path = os.path.normpath(path)\n if path == '.':\n path = ''\n path = path.split(self.__path)[-1].strip(os.sep)\n if split:\n return path.split(os.sep)\n else:\n return path", "response": "Given a path return relative path to diretory"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list representation of the repository state along with useful AttributeNames information.", "response": "def get_repository_state(self, relaPath=None):\n \"\"\"\n Get a list representation of repository state along with useful\n information. List state is ordered relativeley to directories level\n\n :Parameters:\n #. relaPath (None, str): relative directory path from where to\n start. If None all repository representation is returned.\n\n :Returns:\n #. state (list): List representation of the repository.\n List items are all dictionaries. Every dictionary has a single\n key which is the file or the directory name and the value is a\n dictionary of information including:\n\n * 'type': the type of the tracked whether it's file, dir, or objectdir\n * 'exists': whether file or directory actually exists on disk\n * 'pyrepfileinfo': In case of a file or an objectdir whether .%s_pyrepfileinfo exists\n * 'pyrepdirinfo': In case of a directory whether .pyrepdirinfo exists\n \"\"\"\n state = []\n def _walk_dir(relaPath, dirList):\n dirDict = {'type':'dir',\n 'exists':os.path.isdir(os.path.join(self.__path,relaPath)),\n 'pyrepdirinfo':os.path.isfile(os.path.join(self.__path,relaPath,self.__dirInfo)),\n }\n state.append({relaPath:dirDict})\n # loop files and dirobjects\n for fname in sorted([f for f in dirList if isinstance(f, basestring)]):\n relaFilePath = os.path.join(relaPath,fname)\n realFilePath = os.path.join(self.__path,relaFilePath)\n #if os.path.isdir(realFilePath) and df.startswith('.') and df.endswith(self.__objectDir[3:]):\n # fileDict = {'type':'objectdir',\n # 'exists':True,\n # 'pyrepfileinfo':os.path.isfile(os.path.join(self.__path,relaPath,self.__fileInfo%fname)),\n # }\n #else:\n # fileDict = {'type':'file',\n # 'exists':os.path.isfile(realFilePath),\n # 'pyrepfileinfo':os.path.isfile(os.path.join(self.__path,relaPath,self.__fileInfo%fname)),\n # }\n fileDict = {'type':'file',\n 'exists':os.path.isfile(realFilePath),\n 'pyrepfileinfo':os.path.isfile(os.path.join(self.__path,relaPath,self.__fileInfo%fname)),\n }\n state.append({relaFilePath:fileDict})\n # loop directories\n #for ddict in sorted([d for d in dirList if isinstance(d, dict) and len(d)], key=lambda k: list(k)[0]):\n for ddict in sorted([d for d in dirList if isinstance(d, dict)], key=lambda k: list(k)[0]):\n dirname = list(ddict)[0]\n _walk_dir(relaPath=os.path.join(relaPath,dirname), dirList=ddict[dirname])\n # call recursive _walk_dir\n if relaPath is None:\n _walk_dir(relaPath='', dirList=self.__repo['walk_repo'])\n else:\n assert isinstance(relaPath, basestring), \"relaPath must be None or a str\"\n relaPath = self.to_repo_relative_path(path=relaPath, split=False)\n spath = relaPath.split(os.sep)\n dirList = self.__repo['walk_repo']\n while len(spath):\n dirname = spath.pop(0)\n dList = [d for d in dirList if isinstance(d, dict)]\n if not len(dList):\n dirList = None\n break\n cDict = [d for d in dList if dirname in d]\n if not len(cDict):\n dirList = None\n break\n dirList = cDict[0][dirname]\n if dirList is not None:\n _walk_dir(relaPath=relaPath, dirList=dirList)\n # return state list\n return state"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_file_info(self, relativePath):\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n fileName = os.path.basename(relativePath)\n isRepoFile,fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n if not isRepoFile:\n return None, \"file is not a registered repository file.\"\n if not infoOnDisk:\n return None, \"file is a registered repository file but info file missing\"\n fileInfoPath = os.path.join(self.__path,os.path.dirname(relativePath),self.__fileInfo%fileName)\n try:\n with open(fileInfoPath, 'rb') as fd:\n info = pickle.load(fd)\n except Exception as err:\n return None, \"Unable to read file info from disk (%s)\"%str(err)\n return info, ''", "response": "Get file information dictionary from the repository given its relative path."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_repository_file(self, relativePath):\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n if relativePath == '':\n return False, False, False, False\n relaDir, name = os.path.split(relativePath)\n fileOnDisk = os.path.isfile(os.path.join(self.__path, relativePath))\n infoOnDisk = os.path.isfile(os.path.join(self.__path,os.path.dirname(relativePath),self.__fileInfo%name))\n classOnDisk = os.path.isfile(os.path.join(self.__path,os.path.dirname(relativePath),self.__fileClass%name))\n cDir = self.__repo['walk_repo']\n if len(relaDir):\n for dirname in relaDir.split(os.sep):\n dList = [d for d in cDir if isinstance(d, dict)]\n if not len(dList):\n cDir = None\n break\n cDict = [d for d in dList if dirname in d]\n if not len(cDict):\n cDir = None\n break\n cDir = cDict[0][dirname]\n if cDir is None:\n return False, fileOnDisk, infoOnDisk, classOnDisk\n #if name not in cDir:\n if str(name) not in [str(i) for i in cDir]:\n return False, fileOnDisk, infoOnDisk, classOnDisk\n # this is a repository registered file. check whether all is on disk\n return True, fileOnDisk, infoOnDisk, classOnDisk", "response": "Check whether a given relative path is a repository file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef walk_files_path(self, relativePath=\"\", fullPath=False, recursive=False):\n assert isinstance(fullPath, bool), \"fullPath must be boolean\"\n assert isinstance(recursive, bool), \"recursive must be boolean\"\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n dirList = self.__get_repository_directory(relativePath=relativePath)\n assert dirList is not None, \"given relative path '%s' is not a repository directory\"%relativePath\n # walk recursive function\n def _walk(rpath, dlist,recursive):\n # walk files\n for fname in dlist:\n if isinstance(fname, basestring):\n if fullPath:\n yield os.path.join(self.__path, rpath, fname)\n else:\n yield os.path.join(rpath, fname)\n if recursive:\n for ddict in dlist:\n if isinstance(ddict, dict):\n dname = list(ddict)[0]\n for p in _walk(rpath=os.path.join(rpath,dname), dlist=ddict[dname],recursive=recursive):\n yield p\n # walk all files\n return _walk(rpath=relativePath, dlist=dirList, recursive=recursive)", "response": "Walk the files directory and yield the file relative or full path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwalking the repository relative path and yield tuple of two items where first item is file relative path second item is file info.", "response": "def walk_files_info(self, relativePath=\"\", fullPath=False, recursive=False):\n \"\"\"\n Walk the repository relative path and yield tuple of two items where\n first item is file relative/full path and second item is file info.\n If file info is not found on disk, second item will be None.\n\n :parameters:\n #. relativePath (string): The relative path from which start the walk.\n #. fullPath (boolean): Whether to return full or relative path.\n #. recursive (boolean): Whether walk all directories files recursively\n \"\"\"\n assert isinstance(fullPath, bool), \"fullPath must be boolean\"\n assert isinstance(recursive, bool), \"recursive must be boolean\"\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n for relaPath in self.walk_files_path(relativePath=relativePath, fullPath=False, recursive=recursive):\n fpath, fname = os.path.split(relaPath)\n fileInfoPath = os.path.join(self.__path,fpath,self.__fileInfo%fname)\n if os.path.isfile(fileInfoPath):\n with open(fileInfoPath, 'rb') as fd:\n info = pickle.load(fd)\n else:\n info = None\n if fullPath:\n yield (os.path.join(self.__path, relaPath), info)\n else:\n yield (relaPath, info)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef walk_directories_info(self, relativePath=\"\", fullPath=False, recursive=False):\n assert isinstance(fullPath, bool), \"fullPath must be boolean\"\n assert isinstance(recursive, bool), \"recursive must be boolean\"\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n # walk directories\n for dpath in self.walk_directories_path(relativePath=relativePath, fullPath=False, recursive=recursive):\n dirInfoPath = os.path.join(self.__path,dpath,self.__dirInfo)\n if os.path.isfile(dirInfoPath):\n with open(dirInfoPath, 'rb') as fd:\n info = pickle.load(fd)\n else:\n info = None\n if fullPath:\n yield (os.path.join(self.__path, dpath), info)\n else:\n yield (dpath, info)", "response": "Walk the repository relative path and yield tuple of two items where first item is directory relative path second item is directory file info."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a tar file package of all the files and directories in the repository.", "response": "def create_package(self, path=None, name=None, mode=None):\n \"\"\"\n Create a tar file package of all the repository files and directories.\n Only files and directories that are tracked in the repository\n are stored in the package tar file.\n\n **N.B. On some systems packaging requires root permissions.**\n\n :Parameters:\n #. path (None, string): The real absolute path where to create the\n package. If None, it will be created in the same directory as\n the repository. If '.' or an empty string is passed, the current\n working directory will be used.\n #. name (None, string): The name to give to the package file\n If None, the package directory name will be used with the\n appropriate extension added.\n #. mode (None, string): The writing mode of the tarfile.\n If None, automatically the best compression mode will be chose.\n Available modes are ('w', 'w:', 'w:gz', 'w:bz2')\n \"\"\"\n # check mode\n assert mode in (None, 'w', 'w:', 'w:gz', 'w:bz2'), 'unkown archive mode %s'%str(mode)\n if mode is None:\n #mode = 'w:bz2'\n mode = 'w:'\n # get root\n if path is None:\n root = os.path.split(self.__path)[0]\n elif path.strip() in ('','.'):\n root = os.getcwd()\n else:\n root = os.path.realpath( os.path.expanduser(path) )\n assert os.path.isdir(root), 'absolute path %s is not a valid directory'%path\n # get name\n if name is None:\n ext = mode.split(\":\")\n if len(ext) == 2:\n if len(ext[1]):\n ext = \".\"+ext[1]\n else:\n ext = '.tar'\n else:\n ext = '.tar'\n name = os.path.split(self.__path)[1]+ext\n # create tar file\n tarfilePath = os.path.join(root, name)\n try:\n tarHandler = tarfile.TarFile.open(tarfilePath, mode=mode)\n except Exception as e:\n raise Exception(\"Unable to create package (%s)\"%e)\n # walk directory and create empty directories\n for dpath in sorted(list(self.walk_directories_path(recursive=True))):\n t = tarfile.TarInfo( dpath )\n t.type = tarfile.DIRTYPE\n tarHandler.addfile(t)\n tarHandler.add(os.path.join(self.__path,dpath,self.__dirInfo), arcname=self.__dirInfo)\n # walk files and add to tar\n for fpath in self.walk_files_path(recursive=True):\n relaPath, fname = os.path.split(fpath)\n tarHandler.add(os.path.join(self.__path,fpath), arcname=fname)\n tarHandler.add(os.path.join(self.__path,relaPath,self.__fileInfo%fname), arcname=self.__fileInfo%fname)\n tarHandler.add(os.path.join(self.__path,relaPath,self.__fileClass%fname), arcname=self.__fileClass%fname)\n # save repository .pyrepinfo\n tarHandler.add(os.path.join(self.__path,self.__repoFile), arcname=\".pyrepinfo\")\n # close tar file\n tarHandler.close()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_directory(self, relativePath, description=None, clean=False,\n raiseError=True, ntrials=3):\n \"\"\"\n Add a directory in the repository and creates its attribute in the\n Repository with utc timestamp. It insures adding all the missing\n directories in the path.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path to\n where directory must be added.\n #. description (None, string): Any random description about the\n added directory.\n #. clean (boolean): Whether to remove existing non repository\n tracked files and folders in all created directory chain tree.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether adding the directory was successful.\n #. message (None, string): Reason why directory was not added or\n random information.\n \"\"\"\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(relativePath, basestring), \"relativePath must be a string\"\n if description is not None:\n assert isinstance(description, basestring), \"description must be None or a string\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # normalise path\n path = self.to_repo_relative_path(path=relativePath, split=False)\n # whether to replace\n if self.is_repository_directory(path):\n return True, \"Directory is already tracked in repository\"\n # check whether name is allowed\n allowed, reason = self.is_name_allowed(path)\n if not allowed:\n if raiseError:\n raise Exception(reason)\n return False, reason\n # lock repository and get __repo updated from disk\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n if not acquired:\n m = \"code %s. Unable to aquire the lock to add directory. You may try again!\"%(code,)\n if raiseError:\n raise Exception(m)\n return False,m\n # load repository info\n for _trial in range(ntrials):\n try:\n repo = self.__load_repository_pickle_file(os.path.join(self.__path, self.__repoFile))\n self.__repo['walk_repo'] = repo['walk_repo']\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n if error is not None:\n _ = LR.release_lock()\n assert not raiseError, Exception(error)\n return False, error\n # create directories\n error = None\n posList = self.__repo['walk_repo']\n dirPath = self.__path\n spath = path.split(os.sep)\n for idx, name in enumerate(spath):\n # create and acquire lock.\n LD = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(dirPath, self.__dirLock))\n acquired, code = LD.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock when adding '%s'. All prior directories were added. You may try again, to finish adding directory\"%(code,dirPath)\n break\n # add to directory\n for _trial in range(ntrials):\n try:\n dirPath = os.path.join(dirPath, name)\n riPath = os.path.join(dirPath, self.__dirInfo)\n dList = [d for d in posList if isinstance(d, dict)]\n dList = [d for d in dList if name in d]\n # clean directory\n if not len(dList) and clean and os.path.exists(dirPath):\n try:\n shutil.rmtree( dirPath, ignore_errors=True )\n except Exception as err:\n error = \"Unable to clean directory '%s' (%s)\"%(dirPath, err)\n break\n # create directory\n if not os.path.exists(dirPath):\n try:\n os.mkdir(dirPath)\n except Exception as err:\n error = \"Unable to create directory '%s' (%s)\"%(dirPath, err)\n break\n # create and dump dirinfo\n self.__save_dirinfo(description=[None, description][idx==len(spath)-1],\n dirInfoPath=riPath, create=True)\n # update directory list\n if not len(dList):\n rsd = {name:[]}\n posList.append(rsd)\n posList = rsd[name]\n else:\n assert len(dList) == 1, \"Same directory name dict is found twice. This should'n have happened. Report issue\"\n posList = dList[0][name]\n except Exception as err:\n LD.release_lock()\n error = \"Unable to create directory '%s' info file (%s)\"%(dirPath, str(err))\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n LD.release_lock()\n break\n if error is not None:\n break\n # save __repo\n if error is None:\n try:\n _, error = self.__save_repository_pickle_file(lockFirst=False, raiseError=False)\n except Exception as err:\n error = str(err)\n pass\n try:\n LD.release_lock()\n except:\n pass\n try:\n LR.release_lock()\n except:\n pass\n # check and return\n assert error is None or not raiseError, error\n return error is None, error", "response": "This function adds a directory to the repository and creates its attribute in the repository with utc timestamp."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves a directory from the repository.", "response": "def remove_directory(self, relativePath, clean=False, raiseError=True, ntrials=3):\n \"\"\"\n Remove directory from repository tracking.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of the\n directory to remove from the repository.\n #. clean (boolean): Whether to os remove directory. If False only\n tracked files will be removed along with left empty directories.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether removing the directory was successful.\n #. reason (None, string): Reason why directory was not removed.\n \"\"\"\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(clean, bool), \"clean must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # normalise path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n parentPath, dirName = os.path.split(relativePath)\n # check if this is main repository directory\n if relativePath == '':\n return False, \"Removing main repository directory is not allowed\"\n # check if this is a repository directory\n if not self.is_repository_directory(relativePath):\n return False, \"Given relative path '%s' is not a repository path\"%relativePath\n # check if directory actually exists on disk\n realPath = os.path.join(self.__path,relativePath)\n if not os.path.isdir(realPath):\n error = \"Repository relative directory '%s' seems to be missing. call maintain_repository to fix all issues\"\n assert not raiseError, error\n return False, error\n # get and acquire lock\n LD = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path,parentPath,self.__dirLock))\n acquired, code = LD.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock when adding '%s'. All prior directories were added. You may try again, to finish adding directory\"%(code,realPath)\n assert not raiseError, error\n return False, error\n # lock repository and get __repo updated from disk\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n if not acquired:\n LD.release_lock()\n m = \"code %s. Unable to aquire the repository lock. You may try again!\"%(code,)\n assert raiseError, Exception(m)\n return False,m\n # remove directory\n for _trial in range(ntrials):\n error = None\n try:\n dirList = self.__get_repository_parent_directory(relativePath=relativePath)\n assert dirList is not None, \"Given relative path '%s' is not a repository directory\"%(relativePath,)\n stateBefore = self.get_repository_state(relaPath=parentPath)\n _files = [f for f in dirList if isinstance(f, basestring)]\n _dirs = [d for d in dirList if isinstance(d, dict)]\n _dirs = [d for d in dirList if dirName not in d]\n _ = [dirList.pop(0) for _ in range(len(dirList))]\n dirList.extend(_files)\n dirList.extend(_dirs)\n if clean:\n shutil.rmtree(realPath)\n else:\n stateAfter = self.get_repository_state(relaPath=parentPath)\n success, errors = self.__clean_before_after(stateBefore=stateBefore, stateAfter=stateAfter, keepNoneEmptyDirectory=True)\n assert success, \"\\n\".join(errors)\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n break\n # return\n if error is None:\n _, error = self.__save_repository_pickle_file(lockFirst=False, raiseError=False)\n LD.release_lock()\n LR.release_lock()\n # check and return\n assert error is None or not raiseError, \"Unable to remove directory after %i trials '%s' (%s)\"%(relativePath, ntrials, error,)\n return error is None, error"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rename_directory(self, relativePath, newName, raiseError=True, ntrials=3):\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n parentPath, dirName = os.path.split(relativePath)\n if relativePath == '':\n error = \"Renaming main repository directory is not allowed\"\n assert not raiseError, error\n return False, error\n realPath = os.path.join(self.__path,relativePath)\n newRealPath = os.path.join(os.path.dirname(realPath), newName)\n if os.path.isdir(newRealPath):\n error = \"New directory path '%s' already exist\"%(newRealPath,)\n assert not raiseError, error\n return False, error\n # get directory parent list\n LD = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path,parentPath, self.__dirLock))\n acquired, code = LD.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire repository lock when renaming '%s'. All prior directories were added. You may try again, to finish adding the directory\"%(code,dirPath)\n assert not raiseError, error\n return False, error\n error = None\n\n # lock repository and get __repo updated from disk\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n if not acquired:\n LD.release_lock()\n m = \"Code %s. Unable to aquire directory lock when renaming '%s'. All prior directories were added. You may try again, to finish adding the directory\"%(code,dirPath)\n assert raiseError, Exception(m)\n return False,m\n # load repository info\n for _trial in range(ntrials):\n try:\n repo = self.__load_repository_pickle_file(os.path.join(self.__path, self.__repoFile))\n self.__repo['walk_repo'] = repo['walk_repo']\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n if error is not None:\n LD.release_lock()\n LR.release_lock()\n assert not raiseError, Exception(error)\n return False, error\n # rename directory\n for _trial in range(ntrials):\n error = None\n try:\n dirList = self.__get_repository_parent_directory(relativePath=relativePath)\n assert dirList is not None, \"Given relative path '%s' is not a repository directory\"%(relativePath,)\n # change dirName in dirList\n _dirDict = [nd for nd in dirList if isinstance(nd,dict)]\n _dirDict = [nd for nd in _dirDict if dirName in nd]\n assert len(_dirDict) == 1, \"This should not have happened. Directory not found in repository. Please report issue\"\n # rename directory\n os.rename(realPath, newRealPath)\n # update dirList\n _dirDict[0][newName] = _dirDict[0][dirName]\n _dirDict[0].pop(dirName)\n # update and dump dirinfo\n self.__save_dirinfo(description=None, dirInfoPath=parentPath, create=False)\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n if error is None:\n _, error = self.__save_repository_pickle_file(lockFirst=False, raiseError=False)\n LR.release_lock()\n LD.release_lock()\n # check and return\n assert error is None or not raiseError, \"Unable to rename directory '%s' to '%s' after %i trials (%s)\"%(relativePath, newName, ntrials, error,)\n return error is None, error", "response": "This function renames a directory in the system."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncopy a directory in the repository.", "response": "def copy_directory(self, relativePath, newRelativePath,\n overwrite=False, raiseError=True, ntrials=3):\n \"\"\"\n Copy a directory in the repository. New directory must not exist.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of\n the directory to be copied.\n #. newRelativePath (string): The new directory relative path.\n #. overwrite (boolean): Whether to overwrite existing but not tracked\n directory in repository.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether renaming the directory was successful.\n #. message (None, string): Some explanatory message or error reason\n why directory was not renamed.\n \"\"\"\n #from distutils.dir_util import copy_tree\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(overwrite, bool), \"overwrite must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n if relativePath == '':\n m = \"Copying to repository main directory is not possible\"\n assert not raiseError, m\n return False, m\n realPath = os.path.join(self.__path,relativePath)\n parentRealPath, dirName = os.path.split(realPath)\n parentRelativePath = os.path.dirname(relativePath)\n if not self.is_repository_directory(relativePath):\n m = \"Directory '%s' is not a tracked repository directory\"%(relativePath)\n assert not raiseError, m\n return False, m\n newRelativePath = self.to_repo_relative_path(path=newRelativePath, split=False)\n newRealPath = os.path.join(self.__path,newRelativePath)\n newParentRealPath, newDirName = os.path.split(newRealPath)\n newParentRelativePath = os.path.dirname(newRelativePath)\n if realPath == newRealPath:\n m = \"Copying to the same directory is not possible\"\n assert not raiseError, m\n return False, m\n if self.is_repository_directory(newRelativePath):\n m = \"Directory '%s' is a tracked repository directory\"%(newRelativePath)\n assert not raiseError, m\n return False, m\n if os.path.isdir(newRealPath):\n if overwrite:\n try:\n shutil.rmtree(newRealPath)\n except Exception as err:\n assert not raiseError, str(err)\n return False, str(err)\n else:\n error = \"New directory path '%s' already exist on disk. Set overwrite to True\"%(newRealPath,)\n assert not raiseError, error\n return False, error\n # add directory\n try:\n success, reason = self.add_directory(newParentRelativePath, raiseError=False, ntrials=ntrials)\n except Exception as err:\n reason = \"Unable to add directory (%s)\"%(str(err))\n success = False\n if not success:\n assert not raiseError, reason\n return False, reason\n # lock repository and get __repo updated from disk\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n if not acquired:\n m = \"code %s. Unable to aquire the repository lock. You may try again!\"%(code,)\n assert raiseError, Exception(m)\n return False,m\n try:\n repo = self.__load_repository_pickle_file(os.path.join(self.__path, self.__repoFile))\n self.__repo['walk_repo'] = repo['walk_repo']\n except Exception as err:\n LR.release_lock()\n assert not raiseError, Exception(str(err))\n return False,m\n # create locks\n L0 = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(parentRealPath, self.__dirLock))\n acquired, code = L0.acquire_lock()\n if not acquired:\n LR.release_lock()\n error = \"Code %s. Unable to aquire the lock when adding '%s'. All prior directories were added. You may try again, to finish adding directory\"%(code,dirPath)\n assert not raiseError, error\n return False, error\n L1 = None\n if parentRealPath != newParentRealPath:\n L1 = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(newParentRealPath, self.__dirLock))\n acquired, code = L1.acquire_lock()\n if not acquired:\n L0.release_lock()\n LR.release_lock()\n error = \"Code %s. Unable to aquire the lock when adding '%s'. All prior directories were added. You may try again, to finish adding directory\"%(code,dirPath)\n assert not raiseError, error\n return False, error\n # get directory parent list\n error = None\n for _trial in range(ntrials):\n try:\n # make sure again because sometimes, when multiple processes are working on the same repo things can happen in between\n assert self.is_repository_directory(relativePath), \"Directory '%s' is not anymore a tracked repository directory\"%(relativePath)\n assert not self.is_repository_directory(newRelativePath), \"Directory '%s' has become a tracked repository directory\"%(relativePath)\n dirList = self.__get_repository_parent_directory(relativePath=relativePath)\n assert dirList is not None, \"Given relative path '%s' is not a repository directory\"%(relativePath,)\n newDirList = self.__get_repository_parent_directory(relativePath=newRelativePath)\n assert newDirList is not None, \"Given new relative path '%s' parent directory is not a repository directory\"%(newRelativePath,)\n # change dirName in dirList\n _dirDict = [nd for nd in dirList if isinstance(nd,dict)]\n _dirDict = [nd for nd in _dirDict if dirName in nd]\n assert len(_dirDict) == 1, \"This should not have happened. Directory not found in repository. Please report issue\"\n _newDirDict = [nd for nd in newDirList if isinstance(nd,dict)]\n _newDirDict = [nd for nd in _newDirDict if newDirName in nd]\n assert len(_newDirDict) == 0, \"This should not have happened. New directory is found in repository. Please report issue\"\n # try to copy directory\n _newDirDict = copy.deepcopy(_dirDict[0])\n if dirName != newDirName:\n _newDirDict[newDirName] = _newDirDict.pop(dirName)\n copy_tree(realPath, newRealPath)\n # update newDirList\n newDirList.append(_newDirDict)\n # update and dump dirinfo\n self.__save_dirinfo(description=None, dirInfoPath=newParentRelativePath, create=False)\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n if error is None:\n _, error = self.__save_repository_pickle_file(lockFirst=False, raiseError=False)\n LR.release_lock()\n L0.release_lock()\n if L1 is not None:\n L1.release_lock()\n # check and return\n assert error is None or not raiseError, \"Unable to copy directory '%s' to '%s' after %i trials (%s)\"%(relativePath, newRelativePath, ntrials, error,)\n return error is None, error"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndumping a file into the repository.", "response": "def dump_file(self, value, relativePath,\n description=None,\n dump=None, pull=None,\n replace=False, raiseError=True, ntrials=3):\n \"\"\"\n Dump a file using its value to the system and creates its\n attribute in the Repository with utc timestamp.\n\n :Parameters:\n #. value (object): The value of a file to dump and add to the\n repository. It is any python object or file.\n #. relativePath (str): The relative to the repository path to where\n to dump the file.\n #. description (None, string): Any description about the file.\n #. dump (None, string): The dumping method.\n If None it will be set automatically to pickle and therefore the\n object must be pickleable. If a string is given, it can be a\n keyword ('json','pickle','dill') or a string compileable code to\n dump the data. The string code must include all the necessary\n imports and a '$FILE_PATH' that replaces the absolute file path\n when the dumping will be performed.\\n\n e.g. \"import numpy as np; np.savetxt(fname='$FILE_PATH', X=value, fmt='%.6e')\"\n #. pull (None, string): The pulling method. If None it will be set\n automatically to pickle and therefore the object must be\n pickleable. If a string is given, it can be a keyword\n ('json','pickle','dill') or a string compileable code to pull\n the data. The string code must include all the necessary imports,\n a '$FILE_PATH' that replaces the absolute file path when the\n dumping will be performed and finally a PULLED_DATA variable.\\n\n e.g \"import numpy as np; PULLED_DATA=np.loadtxt(fname='$FILE_PATH')\"\n #. replace (boolean): Whether to replace any existing file.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether renaming the directory was successful.\n #. message (None, string): Some explanatory message or error reason\n why directory was not dumped.\n \"\"\"\n # check arguments\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(replace, bool), \"replace must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n if description is None:\n description = ''\n assert isinstance(description, basestring), \"description must be None or a string\"\n # convert dump and pull methods to strings\n if pull is None and dump is not None:\n if dump.startswith('pickle') or dump.startswith('dill') or dump.startswith('numpy') or dump =='json':\n pull = dump\n dump = get_dump_method(dump, protocol=self._DEFAULT_PICKLE_PROTOCOL)\n pull = get_pull_method(pull)\n # check name and path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n savePath = os.path.join(self.__path,relativePath)\n fPath, fName = os.path.split(savePath)\n # check if name is allowed\n success, reason = self.is_name_allowed(savePath)\n if not success:\n assert not raiseError, reason\n return False, reason\n # ensure directory added\n try:\n success, reason = self.add_directory(fPath, raiseError=False, ntrials=ntrials)\n except Exception as err:\n reason = \"Unable to add directory (%s)\"%(str(err))\n success = False\n if not success:\n assert not raiseError, reason\n return False, reason\n # lock repository\n LR = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(self.__path, self.__repoLock))\n acquired, code = LR.acquire_lock()\n if not acquired:\n m = \"code %s. Unable to aquire the repository lock. You may try again!\"%(code,)\n assert raiseError, Exception(m)\n return False,m\n # lock file\n LF = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(fPath,self.__fileLock%fName))\n acquired, code = LF.acquire_lock()\n if not acquired:\n LR.release_lock()\n error = \"Code %s. Unable to aquire the lock when adding '%s'\"%(code,relativePath)\n assert not raiseError, error\n return False, error\n # load repository info\n for _trial in range(ntrials):\n try:\n repo = self.__load_repository_pickle_file(os.path.join(self.__path, self.__repoFile))\n self.__repo['walk_repo'] = repo['walk_repo']\n except Exception as err:\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n if error is not None:\n LR.release_lock()\n LF.release_lock()\n assert not raiseError, Exception(error)\n return False, error\n # dump file\n for _trial in range(ntrials):\n error = None\n try:\n isRepoFile, fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n if isRepoFile:\n assert replace, \"file is a registered repository file. set replace to True to replace\"\n fileInfoPath = os.path.join(self.__path,os.path.dirname(relativePath),self.__fileInfo%fName)\n if isRepoFile and fileOnDisk:\n with open(fileInfoPath, 'rb') as fd:\n info = pickle.load(fd)\n assert info['repository_unique_name'] == self.__repo['repository_unique_name'], \"it seems that file was created by another repository\"\n info['last_update_utctime'] = time.time()\n else:\n info = {'repository_unique_name':self.__repo['repository_unique_name']}\n info['create_utctime'] = info['last_update_utctime'] = time.time()\n info['dump'] = dump\n info['pull'] = pull\n info['description'] = description\n # get parent directory list if file is new and not being replaced\n if not isRepoFile:\n dirList = self.__get_repository_directory(fPath)\n # dump file\n #exec( dump.replace(\"$FILE_PATH\", str(savePath)) )\n my_exec( dump.replace(\"$FILE_PATH\", str(savePath)), locals=locals(), globals=globals(), description='dump' )\n # update info\n with open(fileInfoPath, 'wb') as fd:\n pickle.dump( info,fd, protocol=self._DEFAULT_PICKLE_PROTOCOL)\n fd.flush()\n os.fsync(fd.fileno())\n # update class file\n fileClassPath = os.path.join(self.__path,os.path.dirname(relativePath),self.__fileClass%fName)\n with open(fileClassPath, 'wb') as fd:\n if value is None:\n klass = None\n else:\n klass = value.__class__\n pickle.dump(klass , fd, protocol=self._DEFAULT_PICKLE_PROTOCOL )\n fd.flush()\n os.fsync(fd.fileno())\n # add to repo if file is new and not being replaced\n if not isRepoFile:\n dirList.append(fName)\n except Exception as err:\n error = \"unable to dump the file (%s)\"%(str(err),)\n try:\n if 'pickle.dump(' in dump:\n mi = get_pickling_errors(value)\n if mi is not None:\n error += '\\nmore info: %s'%str(mi)\n except:\n pass\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n error = None\n break\n # save repository\n if error is None:\n _, error = self.__save_repository_pickle_file(lockFirst=False, raiseError=False)\n # release locks\n LR.release_lock()\n LF.release_lock()\n assert not raiseError or error is None, \"unable to dump file '%s' after %i trials (%s)\"%(relativePath, ntrials, error,)\n return success, error"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates the value of a file in the repository.", "response": "def update_file(self, value, relativePath, description=False,\n dump=False, pull=False, raiseError=True, ntrials=3):\n \"\"\"\n Update the value of a file that is already in the Repository.\\n\n If file is not registered in repository, and error will be thrown.\\n\n If file is missing in the system, it will be regenerated as dump method\n is called.\n Unlike dump_file, update_file won't block the whole repository but only\n the file being updated.\n\n :Parameters:\n #. value (object): The value of a file to update.\n #. relativePath (str): The relative to the repository path of the\n file to be updated.\n #. description (False, string): Any random description about the file.\n If False is given, the description info won't be updated,\n otherwise it will be update to what description argument value is.\n #. dump (False, string): The new dump method. If False is given,\n the old one will be used.\n #. pull (False, string): The new pull method. If False is given,\n the old one will be used.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether renaming the directory was successful.\n #. message (None, string): Some explanatory message or error reason\n why directory was not updated.\n \"\"\"\n # check arguments\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert description is False or description is None or isinstance(description, basestring), \"description must be False, None or a string\"\n assert dump is False or dump is None or isinstance(dump, basestring), \"dump must be False, None or a string\"\n assert pull is False or pull is None or isinstance(pull, basestring), \"pull must be False, None or a string\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # get name and path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n savePath = os.path.join(self.__path,relativePath)\n fPath, fName = os.path.split(savePath)\n # get locker\n LF = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(fPath,self.__fileLock%fName))\n acquired, code = LF.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock to update '%s'\"%(code,relativePath)\n assert not raiseError, error\n return False, error\n # update file\n for _trial in range(ntrials):\n message = []\n updated = False\n try:\n # check file in repository\n isRepoFile, fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n assert isRepoFile, \"file '%s' is not registered in repository, no update can be performed.\"%(relativePath,)\n # get file info\n if not fileOnDisk:\n assert description is not False, \"file '%s' is found on disk, description must be provided\"%(relativePath,)\n assert dump is not False, \"file '%s' is found on disk, dump must be provided\"%(relativePath,)\n assert pull is not False, \"file '%s' is found on disk, pull must be provided\"%(relativePath,)\n info = {}\n info['repository_unique_name'] = self.__repo['repository_unique_name']\n info['create_utctime'] = info['last_update_utctime'] = time.time()\n else:\n with open(os.path.join(fPath,self.__fileInfo%fName), 'rb') as fd:\n info = pickle.load(fd)\n info['last_update_utctime'] = time.time()\n if not fileOnDisk:\n message.append(\"file %s is registered in repository but it was found on disk prior to updating\"%relativePath)\n if not infoOnDisk:\n message.append(\"%s is not found on disk prior to updating\"%self.__fileInfo%fName)\n if not classOnDisk:\n message.append(\"%s is not found on disk prior to updating\"%self.__fileClass%fName)\n # get dump and pull\n if (description is False) or (dump is False) or (pull is False):\n if description is False:\n description = info['description']\n elif description is None:\n description = ''\n if dump is False:\n dump = info['dump']\n elif dump is None:\n dump = get_dump_method(dump, protocol=self._DEFAULT_PICKLE_PROTOCOL)\n if pull is False:\n pull = info['pull']\n elif pull is None:\n pull = get_pull_method(pull)\n # update dump, pull and description\n info['dump'] = dump\n info['pull'] = pull\n info['description'] = description\n # dump file\n my_exec( dump.replace(\"$FILE_PATH\", str(savePath)), locals=locals(), globals=globals(), description='update' )\n #exec( dump.replace(\"$FILE_PATH\", str(savePath)) )\n # remove file if exists\n _path = os.path.join(fPath,self.__fileInfo%fName)\n # update info\n with open(_path, 'wb') as fd:\n pickle.dump( info,fd, protocol=self._DEFAULT_PICKLE_PROTOCOL )\n fd.flush()\n os.fsync(fd.fileno())\n # update class file\n fileClassPath = os.path.join(self.__path,os.path.dirname(relativePath),self.__fileClass%fName)\n with open(fileClassPath, 'wb') as fd:\n if value is None:\n klass = None\n else:\n klass = value.__class__\n pickle.dump(klass , fd, protocol=self._DEFAULT_PICKLE_PROTOCOL )\n fd.flush()\n os.fsync(fd.fileno())\n except Exception as err:\n message.append(str(err))\n updated = False\n try:\n if 'pickle.dump(' in dump:\n mi = get_pickling_errors(value)\n if mi is not None:\n message.append('more info: %s'%str(mi))\n except:\n pass\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], '\\n'.join(message)))\n else:\n updated = True\n break\n # release lock\n LF.release_lock()\n # check and return\n assert updated or not raiseError, \"Unable to update file '%s' (%s)\"%(relativePath, '\\n'.join(message),)\n return updated, '\\n'.join(message)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npull a file from the repository.", "response": "def pull_file(self, relativePath, pull=None, update=True, ntrials=3):\n \"\"\"\n Pull a file's data from the Repository.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path from\n where to pull the file.\n #. pull (None, string): The pulling method.\n If None, the pull method saved in the file info will be used.\n If a string is given, the string should include all the necessary\n imports, a '$FILE_PATH' that replaces the absolute file path when\n the dumping will be performed and finally a PULLED_DATA variable.\n e.g \"import numpy as np; PULLED_DATA=np.loadtxt(fname='$FILE_PATH')\"\n #. update (boolean): If pull is not None, Whether to update the pull\n method stored in the file info by the given pull method.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. data (object): The pulled data from the file.\n \"\"\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # check name and path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n realPath = os.path.join(self.__path,relativePath)\n fPath, fName = os.path.split(realPath)\n # check whether it's a repository file\n isRepoFile,fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n if not isRepoFile:\n fileOnDisk = [\"\",\". File itself is found on disk\"][fileOnDisk]\n infoOnDisk = [\"\",\". %s is found on disk\"%self.__fileInfo%fName][infoOnDisk]\n classOnDisk = [\"\",\". %s is found on disk\"%self.__fileClass%fName][classOnDisk]\n assert False, \"File '%s' is not a repository file%s%s%s\"%(relativePath,fileOnDisk,infoOnDisk,classOnDisk)\n assert fileOnDisk, \"File '%s' is registered in repository but the file itself was not found on disk\"%(relativePath,)\n if not infoOnDisk:\n if pull is not None:\n warnings.warn(\"'%s' was not found on disk but pull method is given\"%(self.__fileInfo%fName))\n else:\n raise Exception(\"File '%s' is registered in repository but the '%s' was not found on disk and pull method is not specified\"%(relativePath,(self.__fileInfo%fName)))\n # lock repository\n LF = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(fPath,self.__fileLock%fName))\n acquired, code = LF.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock when adding '%s'\"%(code,relativePath)\n return False, error\n # pull file\n for _trial in range(ntrials):\n error = None\n try:\n # get pull method\n if pull is not None:\n pull = get_pull_method(pull)\n else:\n with open(os.path.join(fPath,self.__fileInfo%fName), 'rb') as fd:\n info = pickle.load(fd)\n pull = info['pull']\n # try to pull file\n #namespace = {}\n #namespace.update( globals() )\n #exec( pull.replace(\"$FILE_PATH\", str(realPath) ), namespace )\n my_exec( pull.replace(\"$FILE_PATH\", str(realPath) ), locals=locals(), globals=globals(), description='pull' )\n except Exception as err:\n LF.release_lock()\n m = str(pull).replace(\"$FILE_PATH\", str(realPath) )\n error = \"Unable to pull data using '%s' from file (%s)\"%(m,err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n break\n LF.release_lock()\n assert error is None, \"After %i trials, %s\"%(ntrials, error)\n # return data\n return locals()['PULLED_DATA']"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rename_file(self, relativePath, newRelativePath,\n force=False, raiseError=True, ntrials=3):\n \"\"\"\n Rename a file in the repository. It insures renaming the file in the system.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of\n the file that needst to be renamed.\n #. newRelativePath (string): The new relative to the repository path\n of where to move and rename the file.\n #. force (boolean): Whether to force renaming even when another\n repository file exists. In this case old repository file\n will be removed from the repository and the system as well.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n\n :Returns:\n #. success (boolean): Whether renaming the file was successful.\n #. message (None, string): Some explanatory message or error reason\n why directory was not updated.\n \"\"\"\n assert isinstance(raiseError, bool), \"raiseError must be boolean\"\n assert isinstance(force, bool), \"force must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # check old name and path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n realPath = os.path.join(self.__path,relativePath)\n fPath, fName = os.path.split(realPath)\n # check new name and path\n newRelativePath = self.to_repo_relative_path(path=newRelativePath, split=False)\n newRealPath = os.path.join(self.__path,newRelativePath)\n nfPath, nfName = os.path.split(newRealPath)\n # lock old file\n LO = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(fPath,self.__fileLock%fName))\n acquired, code = LO.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock for old file '%s'\"%(code,relativePath)\n assert not raiseError, error\n return False, error\n # add directory\n try:\n success, reason = self.add_directory(nfPath, raiseError=False, ntrials=ntrials)\n except Exception as err:\n reason = \"Unable to add directory (%s)\"%(str(err))\n success = False\n if not success:\n LO.release_lock()\n assert not raiseError, reason\n return False, reason\n # create new file lock\n LN = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(nfPath,self.__fileLock%nfName))\n acquired, code = LN.acquire_lock()\n if not acquired:\n LO.release_lock()\n error = \"Code %s. Unable to aquire the lock for new file path '%s'\"%(code,newRelativePath)\n assert not raiseError, error\n return False, error\n # rename file\n for _trial in range(ntrials):\n renamed = False\n error = None\n try:\n # check whether it's a repository file\n isRepoFile,fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n assert isRepoFile, \"file '%s' is not a repository file\"%(relativePath,)\n assert fileOnDisk, \"file '%s' is found on disk\"%(relativePath,)\n assert infoOnDisk, \"%s is found on disk\"%self.__fileInfo%fName\n assert classOnDisk, \"%s is found on disk\"%self.__fileClass%fName\n # get new file path\n nisRepoFile,nfileOnDisk,ninfoOnDisk,nclassOnDisk = self.is_repository_file(newRelativePath)\n assert not nisRepoFile or force, \"New file path is a registered repository file, set force to True to proceed regardless\"\n # get parent directories list\n oDirList = self.__get_repository_directory(fPath)\n nDirList = self.__get_repository_directory(nfPath)\n # remove new file and all repository files from disk\n if os.path.isfile(newRealPath):\n os.remove(newRealPath)\n if os.path.isfile(os.path.join(nfPath,self.__fileInfo%nfName)):\n os.remove(os.path.join(nfPath,self.__fileInfo%nfName))\n if os.path.isfile(os.path.join(nfPath,self.__fileClass%nfName)):\n os.remove(os.path.join(nfPath,self.__fileClass%nfName))\n # move old file to new path\n os.rename(realPath, newRealPath)\n os.rename(os.path.join(fPath,self.__fileInfo%fName), os.path.join(nfPath,self.__fileInfo%nfName))\n os.rename(os.path.join(fPath,self.__fileClass%fName), os.path.join(nfPath,self.__fileClass%nfName))\n # update list\n findex = oDirList.index(fName)\n oDirList.pop(findex)\n # update new list\n if nfName not in nDirList:\n nDirList.append(nfName)\n except Exception as err:\n renamed = False\n error = str(err)\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], str(error)))\n else:\n renamed = True\n break\n # release locks\n LO.release_lock()\n LN.release_lock()\n # always clean old file lock\n try:\n if os.path.isfile(os.path.join(fPath,self.__fileLock%fName)):\n os.remove(os.path.join(fPath,self.__fileLock%fName))\n except:\n pass\n # return\n assert renamed or not raiseError, \"Unable to rename file '%s' to '%s' after %i trials (%s)\"%(relativePath, newRelativePath, ntrials, error,)\n #assert renamed or not raiseError, '\\n'.join(message)\n return renamed, error", "response": "This function rename a file in the repository."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove a file from the repository.", "response": "def remove_file(self, relativePath, removeFromSystem=False,\n raiseError=True, ntrials=3):\n \"\"\"\n Remove file from repository.\n\n :Parameters:\n #. relativePath (string): The relative to the repository path of the\n file to remove.\n #. removeFromSystem (boolean): Whether to remove file from disk as\n well.\n #. raiseError (boolean): Whether to raise encountered error instead\n of returning failure.\n #. ntrials (int): After aquiring all locks, ntrials is the maximum\n number of trials allowed before failing.\n In rare cases, when multiple processes\n are accessing the same repository components, different processes\n can alter repository components between successive lock releases\n of some other process. Bigger number of trials lowers the\n likelyhood of failure due to multiple processes same time\n alteration.\n \"\"\"\n assert isinstance(raiseError, bool), \"removeFromSystem must be boolean\"\n assert isinstance(removeFromSystem, bool), \"removeFromSystem must be boolean\"\n assert isinstance(ntrials, int), \"ntrials must be integer\"\n assert ntrials>0, \"ntrials must be >0\"\n # check name and path\n relativePath = self.to_repo_relative_path(path=relativePath, split=False)\n realPath = os.path.join(self.__path,relativePath)\n fPath, fName = os.path.split(realPath)\n # lock repository\n LF = Locker(filePath=None, lockPass=str(uuid.uuid1()), lockPath=os.path.join(fPath,self.__fileLock%fName))\n acquired, code = LF.acquire_lock()\n if not acquired:\n error = \"Code %s. Unable to aquire the lock when adding '%s'\"%(code,relativePath)\n assert not raiseError, error\n return False, error\n # remove file\n for _trial in range(ntrials):\n removed = False\n message = []\n try:\n # check whether it's a repository file\n isRepoFile,fileOnDisk, infoOnDisk, classOnDisk = self.is_repository_file(relativePath)\n if not isRepoFile:\n message(\"File '%s' is not a repository file\"%(relativePath,))\n if fileOnDisk:\n message.append(\"File itself is found on disk\")\n if infoOnDisk:\n message.append(\"%s is found on disk\"%self.__fileInfo%fName)\n if classOnDisk:\n message.append(\"%s is found on disk\"%self.__fileClass%fName)\n else:\n dirList = self.__get_repository_directory(fPath)\n findex = dirList.index(fName)\n dirList.pop(findex)\n if os.path.isfile(realPath):\n os.remove(realPath)\n if os.path.isfile(os.path.join(fPath,self.__fileInfo%fName)):\n os.remove(os.path.join(fPath,self.__fileInfo%fName))\n if os.path.isfile(os.path.join(fPath,self.__fileClass%fName)):\n os.remove(os.path.join(fPath,self.__fileClass%fName))\n except Exception as err:\n removed = False\n message.append(str(err))\n if self.DEBUG_PRINT_FAILED_TRIALS: print(\"Trial %i failed in Repository.%s (%s). Set Repository.DEBUG_PRINT_FAILED_TRIALS to False to mute\"%(_trial, inspect.stack()[1][3], '\\n'.join(message)))\n else:\n removed = True\n break\n # release lock\n LF.release_lock()\n # always clean\n try:\n if os.path.isfile(os.path.join(fPath,self.__fileLock%fName)):\n os.remove(os.path.join(fPath,self.__fileLock%fName))\n except:\n pass\n # check and return\n assert removed or not raiseError, \"Unable to remove file '%s' after %i trials (%s)\"%(relativePath, ntrials, '\\n'.join(message),)\n return removed, '\\n'.join(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rename(self, key: Any, new_key: Any):\n if new_key == key:\n return\n\n required_locks = [self._key_locks[key], self._key_locks[new_key]]\n ordered_required_locks = sorted(required_locks, key=lambda x: id(x))\n for lock in ordered_required_locks:\n lock.acquire()\n\n try:\n if key not in self._data:\n raise KeyError(\"Attribute to rename \\\"%s\\\" does not exist\" % key)\n self._data[new_key] = self[key]\n del self._data[key]\n finally:\n for lock in required_locks:\n lock.release()", "response": "Renames an item in this collection as a transaction."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_text_fingerprint(text, hash_meth, encoding=\"utf-8\"): # pragma: no cover\n m = hash_meth()\n m.update(text.encode(encoding))\n return m.hexdigest()", "response": "Get a fingerprint of a piece of text."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn md5 hash value of a file", "response": "def md5file(abspath, nbytes=0, chunk_size=DEFAULT_CHUNK_SIZE):\n \"\"\"\n Return md5 hash value of a piece of a file\n\n Estimate processing time on:\n\n :param abspath: the absolute path to the file\n :param nbytes: only has first N bytes of the file. if 0 or None,\n hash all file\n\n CPU = i7-4600U 2.10GHz - 2.70GHz, RAM = 8.00 GB\n 1 second can process 0.25GB data\n\n - 0.59G - 2.43 sec\n - 1.3G - 5.68 sec\n - 1.9G - 7.72 sec\n - 2.5G - 10.32 sec\n - 3.9G - 16.0 sec\n \"\"\"\n return get_file_fingerprint(abspath, hashlib.md5, nbytes=nbytes, chunk_size=chunk_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sha256file(abspath, nbytes=0, chunk_size=DEFAULT_CHUNK_SIZE):\n return get_file_fingerprint(abspath, hashlib.sha256, nbytes=nbytes, chunk_size=chunk_size)", "response": "Return the sha256 hash value of a file"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sha512file(abspath, nbytes=0, chunk_size=DEFAULT_CHUNK_SIZE):\n return get_file_fingerprint(abspath, hashlib.sha512, nbytes=nbytes, chunk_size=chunk_size)", "response": "Return the sha512 hash value of a file"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef register(registerable: Any):\n listenable = registration_event_listenable_map[type(registerable)]\n event = RegistrationEvent(registerable, RegistrationEvent.Type.REGISTERED)\n listenable.notify_listeners(event)", "response": "Registers an object notifying any listeners that may be interested in it."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unregister(registerable: Any):\n listenable = registration_event_listenable_map[type(registerable)]\n event = RegistrationEvent(registerable, RegistrationEvent.Type.UNREGISTERED)\n listenable.notify_listeners(event)", "response": "Unregisters an object from the sequence of registered objects."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _load_module(path: str):\n spec = spec_from_file_location(os.path.basename(path), path)\n module = module_from_spec(spec)\n spec.loader.exec_module(module)", "response": "Dynamically loads the python module at the given path."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if the object is empty.", "response": "def is_empty(self, strict=True):\n \"\"\"\n - If it's a file, check if it is a empty file. (0 bytes content)\n - If it's a directory, check if there's no file and dir in it.\n But if ``strict = False``, then only check if there's no file in it.\n\n :param strict: only useful when it is a directory. if True, only\n return True if this dir has no dir and file. if False, return True\n if it doesn't have any file.\n \"\"\"\n if self.exists():\n if self.is_file():\n return self.size == 0\n elif self.is_dir():\n if strict:\n return len(list(self.select(recursive=True))) == 0\n else: # pragma: no cover\n return len(list(self.select_file(recursive=True))) == 0\n else: # pragma: no cover\n msg = \"'%s' is not either file or directory! (maybe simlink)\" % self\n raise EnvironmentError(msg)\n else:\n raise EnvironmentError(\"'%s' not exists!\" % self)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef auto_complete_choices(self, case_sensitive=False):\n self_basename = self.basename\n self_basename_lower = self.basename.lower()\n if case_sensitive: # pragma: no cover\n def match(basename):\n return basename.startswith(self_basename)\n else:\n def match(basename):\n return basename.lower().startswith(self_basename_lower)\n\n choices = list()\n if self.is_dir():\n choices.append(self)\n for p in self.sort_by_abspath(self.select(recursive=False)):\n choices.append(p)\n else:\n p_parent = self.parent\n if p_parent.is_dir():\n for p in self.sort_by_abspath(p_parent.select(recursive=False)):\n if match(p.basename):\n choices.append(p)\n else: # pragma: no cover\n raise ValueError(\"'%s' directory does not exist!\" % p_parent)\n return choices", "response": "A command line auto complete similar behavior. Find all items with same\n prefix of this one."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef print_big_dir(self, top_n=5):\n self.assert_is_dir_and_exists()\n\n size_table = sorted(\n [(p, p.dirsize) for p in self.select_dir(recursive=False)],\n key=lambda x: x[1],\n reverse=True,\n )\n for p, size in size_table[:top_n]:\n print(\"{:<9} {:<9}\".format(repr_data_size(size), p.abspath))", "response": "Print top_n big dir in this dir."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef print_big_file(self, top_n=5):\n self.assert_is_dir_and_exists()\n\n size_table = sorted(\n [(p, p.size) for p in self.select_file(recursive=True)],\n key=lambda x: x[1],\n reverse=True,\n )\n for p, size in size_table[:top_n]:\n print(\"{:<9} {:<9}\".format(repr_data_size(size), p.abspath))", "response": "Print the top n big files in this dir."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef print_big_dir_and_big_file(self, top_n=5):\n self.assert_is_dir_and_exists()\n\n size_table1 = sorted(\n [(p, p.dirsize) for p in self.select_dir(recursive=False)],\n key=lambda x: x[1],\n reverse=True,\n )\n for p1, size1 in size_table1[:top_n]:\n print(\"{:<9} {:<9}\".format(repr_data_size(size1), p1.abspath))\n size_table2 = sorted(\n [(p, p.size) for p in p1.select_file(recursive=True)],\n key=lambda x: x[1],\n reverse=True,\n )\n for p2, size2 in size_table2[:top_n]:\n print(\" {:<9} {:<9}\".format(\n repr_data_size(size2), p2.abspath))", "response": "Print top_n big dir and big file in each dir."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a dict of stat information for all files directories and total size.", "response": "def file_stat_for_all(self, filters=all_true): # pragma: no cover\n \"\"\"\n Find out how many files, directories and total size (Include file in\n it's sub-folder) it has for each folder and sub-folder.\n\n :returns: stat, a dict like ``{\"directory path\": {\n \"file\": number of files, \"dir\": number of directories,\n \"size\": total size in bytes}}``\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u8fd4\u56de\u4e00\u4e2a\u76ee\u5f55\u4e2d\u7684\u6bcf\u4e2a\u5b50\u76ee\u5f55\u7684, \u6587\u4ef6, \u6587\u4ef6\u5939, \u5927\u5c0f\u7684\u7edf\u8ba1\u6570\u636e\u3002\n \"\"\"\n self.assert_is_dir_and_exists()\n\n from collections import OrderedDict\n\n stat = OrderedDict()\n stat[self.abspath] = {\"file\": 0, \"dir\": 0, \"size\": 0}\n\n for p in self.select(filters=filters, recursive=True):\n if p.is_file():\n size = p.size\n while 1:\n parent = p.parent\n\n stat[parent.abspath][\"file\"] += 1\n stat[parent.abspath][\"size\"] += size\n\n if parent.abspath == self.abspath:\n break\n\n p = parent\n\n elif p.is_dir():\n stat[p.abspath] = {\"file\": 0, \"dir\": 0, \"size\": 0}\n\n while 1:\n parent = p.parent\n stat[parent.abspath][\"dir\"] += 1\n\n if parent.abspath == self.abspath:\n break\n\n p = parent\n\n return stat"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding out how many files directorys and total size.", "response": "def file_stat(self, filters=all_true):\n \"\"\"Find out how many files, directorys and total size (Include file in\n it's sub-folder).\n\n :returns: stat, a dict like ``{\"file\": number of files,\n \"dir\": number of directorys, \"size\": total size in bytes}``\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u8fd4\u56de\u4e00\u4e2a\u76ee\u5f55\u4e2d\u7684\u6587\u4ef6, \u6587\u4ef6\u5939, \u5927\u5c0f\u7684\u7edf\u8ba1\u6570\u636e\u3002\n \"\"\"\n self.assert_is_dir_and_exists()\n\n stat = {\"file\": 0, \"dir\": 0, \"size\": 0}\n\n for p in self.select(filters=filters, recursive=True):\n if p.is_file():\n stat[\"file\"] += 1\n stat[\"size\"] += p.size\n elif p.is_dir():\n stat[\"dir\"] += 1\n\n return stat"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mirror_to(self, dst): # pragma: no cover\n self.assert_is_dir_and_exists()\n\n src = self.abspath\n dst = os.path.abspath(dst)\n if os.path.exists(dst): # pragma: no cover\n raise Exception(\"distination already exist!\")\n\n folder_to_create = list()\n file_to_create = list()\n\n for current_folder, _, file_list in os.walk(self.abspath):\n current_folder = current_folder.replace(src, dst)\n try:\n os.mkdir(current_folder)\n except: # pragma: no cover\n pass\n for basename in file_list:\n abspath = os.path.join(current_folder, basename)\n with open(abspath, \"wb\") as _:\n pass", "response": "Mirrors the contents of this directory to another directory."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting every. py file as main script.", "response": "def execute_pyfile(self, py_exe=None): # pragma: no cover\n \"\"\"\n Execute every ``.py`` file as main script.\n\n :param py_exe: str, python command or python executable path.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u5c06\u76ee\u5f55\u4e0b\u7684\u6240\u6709Python\u6587\u4ef6\u4f5c\u4e3a\u4e3b\u811a\u672c\u7528\u5f53\u524d\u89e3\u91ca\u5668\u8fd0\u884c\u3002\n \"\"\"\n import subprocess\n\n self.assert_is_dir_and_exists()\n\n if py_exe is None:\n if six.PY2:\n py_exe = \"python2\"\n elif six.PY3:\n py_exe = \"python3\"\n\n for p in self.select_by_ext(\".py\"):\n subprocess.Popen('%s \"%s\"' % (py_exe, p.abspath))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntrailing white space at end of each line for every. py file.", "response": "def trail_space(self, filters=lambda p: p.ext == \".py\"): # pragma: no cover\n \"\"\"\n Trail white space at end of each line for every ``.py`` file.\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u5c06\u76ee\u5f55\u4e0b\u7684\u6240\u6709\u88ab\u9009\u62e9\u7684\u6587\u4ef6\u4e2d\u884c\u672b\u7684\u7a7a\u683c\u5220\u9664\u3002\n \"\"\"\n self.assert_is_dir_and_exists()\n\n for p in self.select_file(filters):\n try:\n with open(p.abspath, \"rb\") as f:\n lines = list()\n for line in f:\n lines.append(line.decode(\"utf-8\").rstrip())\n\n with open(p.abspath, \"wb\") as f:\n f.write(\"\\n\".join(lines).encode(\"utf-8\"))\n\n except Exception as e: # pragma: no cover\n raise e"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef autopep8(self, **kwargs): # pragma: no cover\n self.assert_is_dir_and_exists()\n\n for p in self.select_by_ext(\".py\"):\n with open(p.abspath, \"rb\") as f:\n code = f.read().decode(\"utf-8\")\n\n formatted_code = autopep8.fix_code(code, **kwargs)\n\n with open(p.abspath, \"wb\") as f:\n f.write(formatted_code.encode(\"utf-8\"))", "response": "Auto convert your python code in a directory to pep8 styled code."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nnotifying all event listeners that the data has been passed to the event listeners.", "response": "def notify_listeners(self, data: Optional[_ListenableDataType]=_NO_DATA_MARKER):\n \"\"\"\n Notify event listeners, passing them the given data (if any).\n :param data: the data to pass to the event listeners\n \"\"\"\n for listener in self._listeners:\n if data is not Listenable._NO_DATA_MARKER:\n listener(data)\n else:\n listener()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the size of the file in bytes.", "response": "def size(self):\n \"\"\"\n File size in bytes.\n \"\"\"\n try:\n return self._stat.st_size\n except: # pragma: no cover\n self._stat = self.stat()\n return self.size"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget most recent modify time in timestamp.", "response": "def mtime(self):\n \"\"\"\n Get most recent modify time in timestamp.\n \"\"\"\n try:\n return self._stat.st_mtime\n except: # pragma: no cover\n self._stat = self.stat()\n return self.mtime"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting most recent access time in timestamp.", "response": "def atime(self):\n \"\"\"\n Get most recent access time in timestamp.\n \"\"\"\n try:\n return self._stat.st_atime\n except: # pragma: no cover\n self._stat = self.stat()\n return self.atime"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ctime(self):\n try:\n return self._stat.st_ctime\n except: # pragma: no cover\n self._stat = self.stat()\n return self.ctime", "response": "Get most recent create time in timestamp."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning current terminal size or a default if detect fails.", "response": "def get_terminal_size(default_cols=80, default_rows=25):\n \"\"\"Return current terminal size (cols, rows) or a default if detect fails.\n\n This snippet comes from color ls by Chuck Blake:\n http://pdos.csail.mit.edu/~cblake/cls/cls.py\n\n \"\"\"\n def ioctl_GWINSZ(fd):\n \"\"\"Get (cols, rows) from a putative fd to a tty.\"\"\"\n try: \n rows_cols = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234')) \n return tuple(reversed(rows_cols))\n except:\n return None\n # Try std in/out/err...\n cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)\n # ...or ctty...\n if not cr:\n try:\n fd = os.open(os.ctermid(), os.O_RDONLY)\n cr = ioctl_GWINSZ(fd)\n os.close(fd)\n except:\n # ...or fall back to defaults\n cr = (int(os.environ.get('COLUMNS', default_cols)), \n int(os.environ.get('LINES', default_rows)))\n return cr"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_metainfo(scriptfile,\n keywords=['author', 'contact', 'copyright', 'download', 'git', 'subversion', 'version', 'website'],\n special={},\n first_line_pattern=r'^(?P.+)(\\s+v(?P\\S+))?',\n keyword_pattern_template=r'^\\s*%(pretty)s:\\s*(?P<%(keyword)s>\\S.+?)\\s*$',\n prettify = lambda kw: kw.capitalize().replace('_', ' ')):\n \"\"\"Dumb helper for pulling metainfo from a script __doc__ string.\n \n Returns a metainfo dict with command, description, progname and the given \n keywords (if present). \n \n This function will only make minimal efforts to succeed. If you need \n anything else: roll your own.\n \n The docstring needs to be multiline and the closing quotes need to be first \n on a line, optionally preceeded by whitespace. \n \n The first non-whitespace line is re.search'ed using first_line_pattern, \n default e.g (version optional, contains no whitespace): PROGNAME [vVERSION]\n \n The next non-whitespace, non-keyword line is expected to be the program \n description.\n\n The following lines are re.search'ed against a keyword:pattern dict which\n is constructed using \n keyword_pattern % dict(pretty=prettify(keyword), keyword=keyword)\n Default prettify is keyword.capitalize().replace('_', ' '). Example,\n for the keyword \"licence\" will match the following line:\n License: The MIT license.\n and set the license metainfo to \"The MIT license.\".\n \n Any keyword:pattern pairs that need special treatment can be supplied with \n special.\n \"\"\"\n patterns = dict((kw, re.compile(keyword_pattern_template % dict(pretty=prettify(kw), keyword=kw))) for kw in keywords)\n patterns.update(special)\n metainfo = dict()\n \n if scriptfile[-4:] in ['.pyc', '.pyo']: \n scriptfile = scriptfile[:-1]\n script = open(scriptfile)\n closer = ''\n for line in script:\n line = line.strip()\n if not line or line.startswith('#'):\n continue\n if line[:3] in ('\"\"\"', \"'''\"):\n closer = line[:3] \n break\n raise ValueError('file contains no docstring')\n if not line:\n for line in script:\n line = line.strip()\n if line:\n break\n g = re.search(first_line_pattern, line[3:]).groupdict()\n metainfo['progname'] = g['progname']\n if g['version']:\n metainfo['version'] = g['version']\n for line in script:\n if line.strip().startswith(closer):\n break\n for keyword, pattern in patterns.items():\n m = pattern.search(line)\n if m:\n metainfo[keyword] = m.group(keyword)\n break\n if line.strip() and not 'description' in metainfo:\n metainfo['description'] = line.strip()\n return metainfo", "response": "This function takes a script file and returns a dictionary with command description progname and keywords."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlisting options that have not been used to format other values in their sections.", "response": "def unusedoptions(self, sections):\n \"\"\"Lists options that have not been used to format other values in \n their sections. \n \n Good for finding out if the user has misspelled any of the options.\n \"\"\"\n unused = set([])\n for section in _list(sections):\n if not self.has_section(section):\n continue\n options = self.options(section)\n raw_values = [self.get(section, option, raw=True) for option in options]\n for option in options:\n formatter = \"%(\" + option + \")s\"\n for raw_value in raw_values:\n if formatter in raw_value:\n break\n else:\n unused.add(option) \n return list(unused)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconsume and process arguments and store the result in self. value.", "response": "def parse(self, argv, usedname, location):\n \"\"\"Consume and process arguments and store the result.\n ARGS:\n argv :\n The argument list to parse.\n usedname :\n The string used by the user to invoke the option.\n location :\n A user friendly sring describing where the parser got this\n data from.\n\n \"\"\"\n try:\n value = self.format.parse(argv)\n except formats.BadNumberOfArguments, e:\n raise BadNumberOfArguments(usedname, e.required, e.supplied)\n except formats.BadArgument, e:\n raise BadArgument(usedname, e.argument, e.message)\n if self.recurring:\n self.value.append(value)\n else:\n self.value = value\n self.location = location"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing a string lexically and stores the result in the internal object.", "response": "def parsestr(self, argsstr, usedname, location):\n \"\"\"Parse a string lexically and store the result.\n ARGS:\n argsstr :\n The string to parse.\n usedname :\n The string used by the user to invoke the option.\n location :\n A user friendly sring describing where the parser got this\n data from.\n\n \"\"\"\n try:\n value = self.format.parsestr(argsstr)\n except formats.BadNumberOfArguments, e:\n raise BadNumberOfArguments(usedname, e.required, e.supplied)\n except formats.BadArgument, e:\n raise BadArgument(usedname, e.argument, e.message)\n if self.recurring:\n self.value.append(value)\n else:\n self.value = value\n self.location = location"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse(self, argv):\n if not argv and self.optional:\n self.value = [] if self.recurring else None\n return\n try:\n value = self.format.parse(argv)\n if not self.recurring:\n self.value = value\n return\n self.value = [value]\n while argv:\n self.value.append(self.format.parse(argv))\n except formats.BadNumberOfArguments, e:\n raise BadNumberOfArguments(self.displayname, e.required, e.given)\n except formats.BadArgument, e:\n raise BadArgument(self.displayname, e.argument, e.details)", "response": "Consume and process arguments and store the result in self. value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef keys(self):\n return self.options.keys() + [p.name for p in self.positional_args]", "response": "List names of options and positional arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef values(self):\n return self.options.values() + [p.value for p in self.positional_args]", "response": "List values of options and positional arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef items(self):\n return [(p.name, p.value) for p in self.options.values() + self.positional_args]", "response": "List values of options and positional arguments."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getparam(self, key):\n try:\n return self.options[key]\n except:\n pass\n for posarg in self.positional_args:\n if posarg.name == key:\n return posarg\n try:\n return self.abbreviations[key[1:]]\n except:\n raise KeyError('no such option or positional argument')", "response": "Get option or positional argument by name index or abbreviation."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd an Option object to the user interface.", "response": "def _add_option(self, option):\n \"\"\"Add an Option object to the user interface.\"\"\"\n if option.name in self.options:\n raise ValueError('name already in use')\n if option.abbreviation in self.abbreviations:\n raise ValueError('abbreviation already in use')\n if option.name in [arg.name for arg in self.positional_args]:\n raise ValueError('name already in use by a positional argument')\n self.options[option.name] = option\n if option.abbreviation:\n self.abbreviations[option.abbreviation] = option\n self.option_order.append(option.name)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _add_positional_argument(self, posarg):\n if self.positional_args:\n if self.positional_args[-1].recurring:\n raise ValueError(\"recurring positional arguments must be last\")\n if self.positional_args[-1].optional and not posarg.optional:\n raise ValueError(\"required positional arguments must precede optional ones\")\n self.positional_args.append(posarg)", "response": "Append a positional argument to the user interface."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read_docs(self, docsfiles):\n updates = DocParser()\n for docsfile in _list(docsfiles):\n if os.path.isfile(docsfile):\n updates.parse(docsfile)\n self.docs.update((k, _docs(updates[k], self.docvars)) for k in self.docs if updates.blocks[k])\n for name, text in updates['parameters'].items():\n if name in self:\n self.getparam(name).docs = text[0] % self.docvars\n elif name not in self.ignore:\n raise ValueError(\"parameter %r does not exist\" % name)", "response": "Read program documentation from a list of paths to potential docsfiles."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing configfiles. INDRA API.", "response": "def parse_files(self, files=None, sections=None):\n \"\"\"Parse configfiles. \n files , or None:\n What files to parse. None means use self.configfiles. New\n values override old ones. A string value will be interpreted\n as a list of one item.\n sections , or None:\n Which sections to parse from the files. None means use\n self.sections. A string value will be interpreted as a list\n of one item. The [DEFAULT]\n section is always read, and it is read as if its contents were\n copied to the beginning of all other sections.\n \n \"\"\"\n files = _list(files, self.configfiles)\n sections = _list(sections, self.sections)\n for file in files:\n parser = StrictConfigParser()\n parser.read(file)\n for section in sections:\n if not parser.has_section(section):\n continue\n for unused in parser.unusedoptions(section):\n if unused not in self.options and unused not in self.ignore: \n templ = \"The option %r in section [%s] of file %s does not exist.\"\n raise InvalidOption(unused, message=templ % (unused, section, file))\n for name in parser.options(section): \n if name in self.options:\n if self.options[name].reserved:\n templ = \"The option %s in section [%s] of file %s is reserved for command line use.\"\n raise ReservedOptionError(message=templ % (unused, section, file))\n value = parser.get(section, name)\n self.options[name].parsestr(value, name, '%s [%s]' % (file, section))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_options(self, argv, location):\n observed = []\n while argv:\n if argv[0].startswith('--'):\n name = argv.pop(0)[2:]\n # '--' means end of options.\n if not name:\n break\n if name not in self.options:\n raise InvalidOption(name)\n option = self.options[name]\n if not option.recurring:\n if option in observed:\n raise OptionRecurrenceError(name)\n observed.append(option)\n option.parse(argv, name, location)\n elif argv[0].startswith('-'):\n # A single - is not an abbreviation block, but the first positional arg.\n if argv[0] == '-':\n break\n block = argv.pop(0)[1:]\n # Abbrevs for options that take values go last in the block.\n for abbreviation in block[:-1]:\n if self.abbreviations[abbreviation].nargs != 0:\n raise BadAbbreviationBlock(abbreviation, block, \"options that require value arguments must be last in abbreviation blocks\")\n # Parse individual options.\n for abbreviation in block:\n option = self.abbreviations[abbreviation]\n if not option.recurring:\n if option in observed:\n raise OptionRecurrenceError(option.name)\n observed.append(option)\n option.parse(argv, '-' + abbreviation, location)\n # only arguments that start with -- or - can be Options.\n else:\n break", "response": "Parse the options part of an argument list."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_positional_arguments(self, argv):\n for posarg in self.positional_args:\n posarg.parse(argv)\n if argv:\n if None in [p.nargs for p in self.positional_args]:\n msg = '%s too many argument%s given'\n plural_s = len(argv) > 1 and 's' or ''\n raise BadNumberOfArguments(message=msg % (len(argv), plural_s))\n msg = 'This program accepts exactly %s positional arguments (%s given).'\n required = len([p.nargs for p in self.positional_args])\n raise BadNumberOfArguments(message=msg % (required, required + len(argv)))", "response": "Parse the positional arguments part of an argument list. Will be altered."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing command line arguments.", "response": "def parse_argv(self, argv=None, location='Command line.'):\n \"\"\"Parse command line arguments.\n \n args or None:\n The argument list to parse. None means use a copy of sys.argv. argv[0] is\n ignored.\n location = '' :\n A user friendly string describing where the parser got this\n data from. '' means use \"Command line.\" if args == None, and\n \"Builtin default.\" otherwise.\n \n \"\"\"\n if argv is None:\n argv = list(sys.argv)\n argv.pop(0)\n self._parse_options(argv, location)\n self._parse_positional_arguments(argv)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning user friendly help on program options.", "response": "def optionhelp(self, indent=0, maxindent=25, width=79):\n \"\"\"Return user friendly help on program options.\"\"\"\n def makelabels(option):\n labels = '%*s--%s' % (indent, ' ', option.name)\n if option.abbreviation:\n labels += ', -' + option.abbreviation\n return labels + ': '\n docs = []\n helpindent = _autoindent([makelabels(o) for o in self.options.values()], indent, maxindent)\n for name in self.option_order:\n option = self.options[name]\n labels = makelabels(option)\n helpstring = \"%s(%s). %s\" % (option.formatname, option.strvalue, option.docs)\n wrapped = self._wrap_labelled(labels, helpstring, helpindent, width)\n docs.extend(wrapped)\n return '\\n'.join(docs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef posarghelp(self, indent=0, maxindent=25, width=79):\n docs = []\n makelabel = lambda posarg: ' ' * indent + posarg.displayname + ': '\n helpindent = _autoindent([makelabel(p) for p in self.positional_args], indent, maxindent)\n for posarg in self.positional_args:\n label = makelabel(posarg)\n text = posarg.formatname + '. ' + posarg.docs\n wrapped = self._wrap_labelled(label, text, helpindent, width)\n docs.extend(wrapped)\n return '\\n'.join(docs)", "response": "Return user friendly help on positional arguments in the program."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef format_usage(self, usage=None):\n if usage is None:\n usage = self.docs['usage']\n if usage is not None:\n return usage[0] % self.docvars\n usage = self.docvars['command']\n if self.basic_option_names.get('help'):\n usage += ' [--%s]' % self.basic_option_names.get('help')\n if self.options:\n usage += ' '\n optional = 0\n for posarg in self.positional_args:\n usage += ' '\n if posarg.optional:\n usage += \"[\" \n optional += 1\n usage += posarg.displayname\n if posarg.recurring:\n usage += ' [%s2 [...]]' % posarg.displayname\n usage += ']' * optional\n return usage", "response": "Return a formatted usage string."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _wraptext(self, text, indent=0, width=0):\n return '\\n'.join(self._wrap(par, indent, width) for par in text)", "response": "Wrap text with the given indent and width."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning brief help containing Title and usage instructions.", "response": "def shorthelp(self, width=0):\n \"\"\"Return brief help containing Title and usage instructions.\n ARGS:\n width = 0 :\n Maximum allowed page width. 0 means use default from\n self.iMaxHelpWidth.\n\n \"\"\"\n out = []\n out.append(self._wrap(self.docs['title'], width=width))\n if self.docs['description']:\n out.append(self._wrap(self.docs['description'], indent=2, width=width))\n out.append('')\n out.append(self._wrapusage(width=width))\n out.append('')\n return '\\n'.join(out)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef strsettings(self, indent=0, maxindent=25, width=0):\n out = []\n makelabel = lambda name: ' ' * indent + name + ': '\n settingsindent = _autoindent([makelabel(s) for s in self.options], indent, maxindent)\n for name in self.option_order:\n option = self.options[name]\n label = makelabel(name)\n settingshelp = \"%s(%s): %s\" % (option.formatname, option.strvalue, option.location)\n wrapped = self._wrap_labelled(label, settingshelp, settingsindent, width)\n out.extend(wrapped)\n return '\\n'.join(out)", "response": "Return a string representation of the options for the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef settingshelp(self, width=0):\n out = []\n out.append(self._wrap(self.docs['title'], width=width))\n if self.docs['description']:\n out.append(self._wrap(self.docs['description'], indent=2, width=width))\n out.append('')\n out.append('SETTINGS:')\n out.append(self.strsettings(indent=2, width=width))\n out.append('')\n return '\\n'.join(out)", "response": "Return a summary of program options their values and origins."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef launch(self,\n argv=None,\n showusageonnoargs=False,\n width=0,\n helphint=\"Use with --help for more information.\\n\",\n debug_parser=False):\n \"\"\"Do the usual stuff to initiallize the program.\n \n Read config files and parse arguments, and if the user has used any \n of the help/version/settings options, display help and exit.\n \n If debug_parser is false, don't catch ParseErrors and exit with user\n friendly help. Crash with traceback instead.\n \n configfiles is a list of config files. None means use self.configfiles.\n \n sections is a list of configfile section names to read. None means use\n self.sections.\n \n argv is a list of arguments to parse. Will be modified. None means use\n copy of sys.argv. argv[0] is ignored.\n \n If showusageonnoargs is true, show usage and exit if the user didn't \n give any args. Should be False if there are no required PositionalArgs.\n \n width is the maximum allowed page width. 0 means use self.width.\n \n helphint is a string that hints on how to get more help which is \n displayed at the end of usage help messages. \n \"\"\"\n if showusageonnoargs and len(argv) == 1:\n print self.shorthelp(width=width)\n if helphint:\n print self._wrap(helphint, indent=2, width=width)\n sys.exit(0)\n parsing_error = None\n try:\n self.parse_files()\n self.parse_argv(argv)\n except ParseError, parsing_error:\n if debug_parser:\n raise\n for optiontype in ['help', 'longhelp', 'settings', 'version']:\n name = self.basic_option_names.get(optiontype)\n if name and self[name]:\n methodname = optiontype.rstrip('help') + 'help'\n print getattr(self, methodname)(width)\n sys.exit()\n if parsing_error:\n self.graceful_exit(parsing_error, width)", "response": "Do the usual stuff to initiallize the program.\n \n Read config files and parse arguments, and if the user has used any \n of the help/version/settings options, display help and exit.\n \n If debug_parser is false, don't catch ParseErrors and exit with user\n friendly help. Crash with traceback instead.\n \n configfiles is a list of config files. None means use self.configfiles.\n \n sections is a list of configfile section names to read. None means use\n self.sections.\n \n argv is a list of arguments to parse. Will be modified. None means use\n copy of sys.argv. argv[0] is ignored.\n \n If showusageonnoargs is true, show usage and exit if the user didn't \n give any args. Should be False if there are no required PositionalArgs.\n \n width is the maximum allowed page width. 0 means use self.width.\n \n helphint is a string that hints on how to get more help which is \n displayed at the end of usage help messages."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing text blocks from a file.", "response": "def parse(self, file):\n \"\"\"Parse text blocks from a file.\"\"\"\n if isinstance(file, basestring):\n file = open(file)\n line_number = 0\n label = None\n block = self.untagged\n for line in file:\n line_number += 1\n line = line.rstrip('\\n')\n if self.tabsize > 0:\n line = line.replace('\\t', ' ' * self.tabsize)\n if self.decommenter:\n line = self.decommenter.decomment(line)\n if line is None:\n continue\n tag = line.split(':', 1)[0].strip()\n # Still in the same block?\n if tag not in self.names:\n if block is None:\n if line and not line.isspace():\n raise ParseError(file.name, line, \"garbage before first block: %r\" % line)\n continue\n block.addline(line)\n continue\n # Open a new block.\n name = self.names[tag]\n label = line.split(':',1)[1].strip()\n if name in self.labelled_classes:\n if not label:\n raise ParseError(file.name, line, \"missing label for %r block\" % name)\n block = self.blocks[name].setdefault(label, self.labelled_classes[name]())\n else:\n if label:\n msg = \"label %r present for unlabelled block %r\" % (label, name)\n raise ParseError(file.name, line_number, msg)\n block = self.blocks[name]\n block.startblock()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_format(format):\n if isinstance(format, BaseFormat):\n return format\n if isinstance(format, basestring):\n for name, formatclass in globals().items():\n if name.lower() == format.lower():\n if not issubclass(formatclass, BaseFormat):\n raise ValueError('%s is not the name of a format class' % format)\n return formatclass()\n try:\n return format()\n except:\n raise ValueError('no such format')", "response": "Get a format object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parsestr(self, argstr):\n argv = shlex.split(argstr, comments=True)\n if len(argv) != self.nargs:\n raise BadNumberOfArguments(self.nargs, len(argv))\n return self.parse(argv)", "response": "Parse arguments found in settings files."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_argument(self, arg):\n lookup = self.casesensitive and arg or arg.lower()\n if lookup in self.special:\n return self.special[lookup]\n try:\n return self.to_python(arg, *self.args, **self.kw)\n except Exception, e:\n raise BadArgument(arg, str(e))", "response": "Parse a single argument."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npopping parse and return the first self. nargs items from args. Raise BadNumberOfArguments or BadArgument on errors.", "response": "def parse(self, argv):\n \"\"\"Pop, parse and return the first self.nargs items from args.\n\n if self.nargs > 1 a list of parsed values will be returned.\n \n Raise BadNumberOfArguments or BadArgument on errors.\n \n NOTE: argv may be modified in place by this method.\n \"\"\"\n if len(argv) < self.nargs:\n raise BadNumberOfArguments(self.nargs, len(argv))\n if self.nargs == 1:\n return self.parse_argument(argv.pop(0))\n return [self.parse_argument(argv.pop(0)) for tmp in range(self.nargs)]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a user - friendly representation of a value.", "response": "def present(self, value):\n \"\"\"Return a user-friendly representation of a value.\n \n Lookup value in self.specials, or call .to_literal() if absent.\n \"\"\"\n for k, v in self.special.items():\n if v == value:\n return k\n return self.to_literal(value, *self.args, **self.kw)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse the arguments found in settings files.", "response": "def parsestr(self, argstr):\n \"\"\"Parse arguments found in settings files.\n \n Use the values in self.true for True in settings files, or those in \n self.false for False, case insensitive.\n \"\"\"\n argv = shlex.split(argstr, comments=True)\n if len(argv) != 1:\n raise BadNumberOfArguments(1, len(argv))\n arg = argv[0]\n lower = arg.lower()\n if lower in self.true:\n return True\n if lower in self.false:\n return False\n raise BadArgument(arg, \"Allowed values are \" + self.allowed + '.')"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npops parse and return the first arg from argv.", "response": "def parse(self, argv):\n \"\"\"Pop, parse and return the first arg from argv.\n \n The arg will be .split() based on self.separator and the (optionally \n stripped) items will be parsed by self.format and returned as a list. \n\n Raise BadNumberOfArguments or BadArgument on errors.\n \n NOTE: args will be modified.\n \"\"\"\n if not argv:\n raise BadNumberOfArguments(1, 0)\n argument = argv.pop(0)\n lookup = self.casesensitive and argument or argument.lower()\n if lookup in self.special:\n return self.special[lookup]\n argv = [(self.strip and s.strip() or s) for s in argument.split(self.separator)]\n values = []\n while argv:\n values.append(self.format.parse(argv))\n return values"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a user - friendly representation of a value.", "response": "def present(self, value):\n \"\"\"Return a user-friendly representation of a value.\n \n Lookup value in self.specials, or call .to_literal() if absent.\n \"\"\"\n for k, v in self.special.items():\n if v == value:\n return k\n return self.separator.join(self.format.present(v) for v in value)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the separator that preceding format i or an empty string for i == 0.", "response": "def get_separator(self, i):\n \"\"\"Return the separator that preceding format i, or '' for i == 0.\"\"\"\n return i and self.separator[min(i - 1, len(self.separator) - 1)] or ''"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse(self, argv):\n if not argv:\n raise BadNumberOfArguments(1, 0)\n remainder = argv.pop(0)\n lookup = self.casesensitive and remainder or remainder.lower()\n if lookup in self.special:\n return self.special[lookup]\n values = []\n for i, format in enumerate(self.format[:-1]):\n print i, self.get_separator(i)\n try:\n arg, remainder = remainder.split(self.get_separator(i + 1), 1)\n except:\n raise BadArgument(remainder, 'does not contain required separator ' + repr(self.get_separator(i + 1)))\n if self.strip:\n arg = arg.strip()\n values.append(format.parse([arg]))\n if self.strip:\n remainder = remainder.strip()\n values.append(format.parse([remainder]))\n return values", "response": "Pop parse and return the first arg from argv."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef present(self, value):\n for k, v in self.special.items():\n if v == value:\n return k\n return ''.join(self.get_separator(i) + self.format[i].present(v) for i, v in enumerate(value))", "response": "Return a user - friendly representation of a value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef authorize_url(self):\n auth_url = OAUTH_ROOT + '/authorize'\n params = {\n 'client_id': self.client_id,\n 'redirect_uri': self.redirect_uri,\n }\n return \"{}?{}\".format(auth_url, urlencode(params))", "response": "Return a URL to redirect the user to for OAuth authentication."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexchange the authorization code for an access token.", "response": "def exchange_token(self, code):\n \"\"\"\n Exchange the authorization code for an access token.\n \"\"\"\n access_token_url = OAUTH_ROOT + '/access_token'\n params = {\n 'client_id': self.client_id,\n 'client_secret': self.client_secret,\n 'redirect_uri': self.redirect_uri,\n 'code': code,\n }\n resp = requests.get(access_token_url, params=params)\n if not resp.ok:\n raise MixcloudOauthError(\"Could not get access token.\")\n return resp.json()['access_token']"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a PORT that will be available on the machine.", "response": "def get_open_port() -> int:\n \"\"\"\n Gets a PORT that will (probably) be available on the machine.\n It is possible that in-between the time in which the open PORT of found and when it is used, another process may\n bind to it instead.\n :return: the (probably) available PORT\n \"\"\"\n free_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n free_socket.bind((\"\", 0))\n free_socket.listen(1)\n port = free_socket.getsockname()[1]\n free_socket.close()\n return port"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef release(self):\n self._lock.release()\n\n with self._stat_lock:\n self._locked = False\n self._last_released = datetime.now()", "response": "Removes the lock and resets the last acquired time."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nhandle a dict that might contain a wrapped state for a custom type.", "response": "def default_decoder(self, obj):\n \"\"\"Handle a dict that might contain a wrapped state for a custom type.\"\"\"\n typename, marshalled_state = self.unwrap_callback(obj)\n if typename is None:\n return obj\n\n try:\n cls, unmarshaller = self.serializer.unmarshallers[typename]\n except KeyError:\n raise LookupError('no unmarshaller found for type \"{}\"'.format(typename)) from None\n\n if cls is not None:\n instance = cls.__new__(cls)\n unmarshaller(instance, marshalled_state)\n return instance\n else:\n return unmarshaller(marshalled_state)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwraps the marshalled state in a dictionary.", "response": "def wrap_state_dict(self, typename: str, state) -> Dict[str, Any]:\n \"\"\"\n Wrap the marshalled state in a dictionary.\n\n The returned dictionary has two keys, corresponding to the ``type_key`` and ``state_key``\n options. The former holds the type name and the latter holds the marshalled state.\n\n :param typename: registered name of the custom type\n :param state: the marshalled state of the object\n :return: an object serializable by the serializer\n\n \"\"\"\n return {self.type_key: typename, self.state_key: state}"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef unwrap_state_dict(self, obj: Dict[str, Any]) -> Union[Tuple[str, Any], Tuple[None, None]]:\n if len(obj) == 2:\n typename = obj.get(self.type_key)\n state = obj.get(self.state_key)\n if typename is not None:\n return typename, state\n\n return None, None", "response": "Unwraps a marshalled state dictionary into a tuple of the type and state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nenabling HTTP access to a dataset.", "response": "def publish(quiet, dataset_uri):\n \"\"\"Enable HTTP access to a dataset.\n\n This only works on datasets in some systems. For example, datasets stored\n in AWS S3 object storage and Microsoft Azure Storage can be published as\n datasets accessible over HTTP. A published dataset is world readable.\n \"\"\"\n access_uri = http_publish(dataset_uri)\n if not quiet:\n click.secho(\"Dataset accessible at \", nl=False, fg=\"green\")\n click.secho(access_uri)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef register_custom_type(\n self, cls: type, marshaller: Optional[Callable[[Any], Any]] = default_marshaller,\n unmarshaller: Union[Callable[[Any, Any], None],\n Callable[[Any], Any], None] = default_unmarshaller, *,\n typename: str = None, wrap_state: bool = True) -> None:\n \"\"\"\n Register a marshaller and/or unmarshaller for the given class.\n\n The state object returned by the marshaller and passed to the unmarshaller can be any\n serializable type. Usually a dictionary mapping of attribute names to values is used.\n\n .. warning:: Registering marshallers/unmarshallers for any custom type will override any\n serializer specific encoding/decoding hooks (respectively) already in place!\n\n :param cls: the class to register\n :param marshaller: a callable that takes the object to be marshalled as the argument and\n returns a state object\n :param unmarshaller: a callable that either:\n\n * takes an uninitialized instance of ``cls`` and its state object as arguments and\n restores the state of the object\n * takes a state object and returns a new instance of ``cls``\n :param typename: a unique identifier for the type (defaults to the ``module:varname``\n reference to the class)\n :param wrap_state: ``True`` to wrap the marshalled state before serialization so that it\n can be recognized later for unmarshalling, ``False`` to serialize it as is\n\n \"\"\"\n assert check_argument_types()\n typename = typename or qualified_name(cls)\n\n if marshaller:\n self.marshallers[cls] = typename, marshaller, wrap_state\n self.custom_type_codec.register_object_encoder_hook(self)\n\n if unmarshaller and self.custom_type_codec is not None:\n target_cls = cls # type: Optional[type]\n if len(signature(unmarshaller).parameters) == 1:\n target_cls = None\n\n self.unmarshallers[typename] = target_cls, unmarshaller\n self.custom_type_codec.register_object_decoder_hook(self)", "response": "Registers a custom type for the given class."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the descriptive metadata interactively.", "response": "def _prompt_for_values(d):\n \"\"\"Update the descriptive metadata interactively.\n\n Uses values entered by the user. Note that the function keeps recursing\n whenever a value is another ``CommentedMap`` or a ``list``. The\n function works as passing dictionaries and lists into a function edits\n the values in place.\n \"\"\"\n for key, value in d.items():\n if isinstance(value, CommentedMap):\n _prompt_for_values(value)\n elif isinstance(value, list):\n for item in value:\n _prompt_for_values(item)\n else:\n typ = type(value)\n\n if isinstance(value, ScalarFloat): # Deal with ruamel.yaml floats.\n typ = float\n\n new_value = click.prompt(key, type=typ, default=value)\n d[key] = new_value\n return d"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create(quiet, name, base_uri, symlink_path):\n _validate_name(name)\n\n admin_metadata = dtoolcore.generate_admin_metadata(name)\n parsed_base_uri = dtoolcore.utils.generous_parse_uri(base_uri)\n\n if parsed_base_uri.scheme == \"symlink\":\n if symlink_path is None:\n raise click.UsageError(\"Need to specify symlink path using the -s/--symlink-path option\") # NOQA\n\n if symlink_path:\n base_uri = dtoolcore.utils.sanitise_uri(\n \"symlink:\" + parsed_base_uri.path\n )\n parsed_base_uri = dtoolcore.utils.generous_parse_uri(base_uri)\n\n # Create the dataset.\n proto_dataset = dtoolcore.generate_proto_dataset(\n admin_metadata=admin_metadata,\n base_uri=dtoolcore.utils.urlunparse(parsed_base_uri),\n config_path=CONFIG_PATH)\n\n # If we are creating a symlink dataset we need to set the symlink_path\n # attribute on the storage broker.\n if symlink_path:\n symlink_abspath = os.path.abspath(symlink_path)\n proto_dataset._storage_broker.symlink_path = symlink_abspath\n try:\n proto_dataset.create()\n except dtoolcore.storagebroker.StorageBrokerOSError as err:\n raise click.UsageError(str(err))\n\n proto_dataset.put_readme(\"\")\n\n if quiet:\n click.secho(proto_dataset.uri)\n else:\n # Give the user some feedback and hints on what to do next.\n click.secho(\"Created proto dataset \", nl=False, fg=\"green\")\n click.secho(proto_dataset.uri)\n click.secho(\"Next steps: \")\n\n step = 1\n\n if parsed_base_uri.scheme != \"symlink\":\n click.secho(\"{}. Add raw data, eg:\".format(step))\n click.secho(\n \" dtool add item my_file.txt {}\".format(proto_dataset.uri),\n fg=\"cyan\")\n\n if parsed_base_uri.scheme == \"file\":\n # Find the abspath of the data directory for user feedback.\n data_path = proto_dataset._storage_broker._data_abspath\n click.secho(\" Or use your system commands, e.g: \")\n click.secho(\n \" mv my_data_directory {}/\".format(data_path),\n fg=\"cyan\"\n )\n step = step + 1\n\n click.secho(\"{}. Add descriptive metadata, e.g: \".format(step))\n click.secho(\n \" dtool readme interactive {}\".format(proto_dataset.uri),\n fg=\"cyan\")\n step = step + 1\n\n click.secho(\n \"{}. Convert the proto dataset into a dataset: \".format(step)\n )\n click.secho(\" dtool freeze {}\".format(proto_dataset.uri), fg=\"cyan\")", "response": "Create a new proto dataset."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef name(dataset_uri, new_name):\n if new_name != \"\":\n _validate_name(new_name)\n\n try:\n dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n except dtoolcore.DtoolCoreTypeError:\n dataset = dtoolcore.DataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n\n dataset.update_name(new_name)\n\n admin_metadata = dtoolcore._admin_metadata_from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n click.secho(admin_metadata[\"name\"])", "response": "Update the name of the dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprompts to populate the readme.", "response": "def interactive(proto_dataset_uri):\n \"\"\"Interactive prompting to populate the readme.\"\"\"\n proto_dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=proto_dataset_uri,\n config_path=CONFIG_PATH)\n\n # Create an CommentedMap representation of the yaml readme template.\n readme_template = _get_readme_template()\n yaml = YAML()\n yaml.explicit_start = True\n yaml.indent(mapping=2, sequence=4, offset=2)\n descriptive_metadata = yaml.load(readme_template)\n\n descriptive_metadata = _prompt_for_values(descriptive_metadata)\n\n # Write out the descriptive metadata to the readme file.\n stream = StringIO()\n\n yaml.dump(descriptive_metadata, stream)\n\n proto_dataset.put_readme(stream.getvalue())\n\n click.secho(\"Updated readme \", fg=\"green\")\n click.secho(\"To edit the readme using your default editor:\")\n click.secho(\n \"dtool readme edit {}\".format(proto_dataset_uri),\n fg=\"cyan\")"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef edit(dataset_uri):\n try:\n dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n except dtoolcore.DtoolCoreTypeError:\n dataset = dtoolcore.DataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n readme_content = dataset.get_readme_content()\n\n try:\n # Python2 compatibility.\n readme_content = unicode(readme_content, \"utf-8\")\n except NameError:\n pass\n\n edited_content = click.edit(readme_content)\n if edited_content is not None:\n _validate_and_put_readme(dataset, edited_content)\n click.secho(\"Updated readme \", nl=False, fg=\"green\")\n else:\n click.secho(\"Did not update readme \", nl=False, fg=\"red\")\n click.secho(dataset_uri)", "response": "Default editor updating of readme content."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef show(dataset_uri):\n try:\n dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n except dtoolcore.DtoolCoreTypeError:\n dataset = dtoolcore.DataSet.from_uri(\n uri=dataset_uri,\n config_path=CONFIG_PATH\n )\n readme_content = dataset.get_readme_content()\n click.secho(readme_content)", "response": "Show the descriptive metadata in the readme."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write(proto_dataset_uri, input):\n proto_dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=proto_dataset_uri\n )\n _validate_and_put_readme(proto_dataset, input.read())", "response": "Write the content of a file or stdin to the README."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a file to the proto dataset.", "response": "def item(proto_dataset_uri, input_file, relpath_in_dataset):\n \"\"\"Add a file to the proto dataset.\"\"\"\n proto_dataset = dtoolcore.ProtoDataSet.from_uri(\n proto_dataset_uri,\n config_path=CONFIG_PATH)\n if relpath_in_dataset == \"\":\n relpath_in_dataset = os.path.basename(input_file)\n proto_dataset.put_item(input_file, relpath_in_dataset)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd metadata to a file in the proto dataset.", "response": "def metadata(proto_dataset_uri, relpath_in_dataset, key, value):\n \"\"\"Add metadata to a file in the proto dataset.\"\"\"\n proto_dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=proto_dataset_uri,\n config_path=CONFIG_PATH)\n proto_dataset.add_item_metadata(\n handle=relpath_in_dataset,\n key=key,\n value=value)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef freeze(proto_dataset_uri):\n proto_dataset = dtoolcore.ProtoDataSet.from_uri(\n uri=proto_dataset_uri,\n config_path=CONFIG_PATH\n )\n\n num_items = len(list(proto_dataset._identifiers()))\n max_files_limit = int(dtoolcore.utils.get_config_value(\n \"DTOOL_MAX_FILES_LIMIT\",\n CONFIG_PATH,\n 10000\n ))\n assert isinstance(max_files_limit, int)\n if num_items > max_files_limit:\n click.secho(\n \"Too many items ({} > {}) in proto dataset\".format(\n num_items,\n max_files_limit\n ),\n fg=\"red\"\n )\n click.secho(\"1. Consider splitting the dataset into smaller datasets\")\n click.secho(\"2. Consider packaging small files using tar\")\n click.secho(\"3. Increase the limit using the DTOOL_MAX_FILES_LIMIT\")\n click.secho(\" environment variable\")\n sys.exit(2)\n\n handles = [h for h in proto_dataset._storage_broker.iter_item_handles()]\n for h in handles:\n if not valid_handle(h):\n click.secho(\n \"Invalid item name: {}\".format(h),\n fg=\"red\"\n )\n click.secho(\"1. Consider renaming the item\")\n click.secho(\"2. Consider removing the item\")\n sys.exit(3)\n\n with click.progressbar(length=len(list(proto_dataset._identifiers())),\n label=\"Generating manifest\") as progressbar:\n try:\n proto_dataset.freeze(progressbar=progressbar)\n except dtoolcore.storagebroker.DiskStorageBrokerValidationWarning as e:\n click.secho(\"\")\n click.secho(str(e), fg=\"red\", nl=False)\n sys.exit(4)\n\n click.secho(\"Dataset frozen \", nl=False, fg=\"green\")\n click.secho(proto_dataset_uri)", "response": "Convert a proto dataset into a dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef copy(resume, quiet, dataset_uri, dest_base_uri):\n click.secho(\n \"The ``dtool copy`` command is deprecated\",\n fg=\"red\",\n err=True\n )\n click.secho(\n \"Use ``dtool cp`` instead\",\n fg=\"red\",\n err=True\n )\n _copy(resume, quiet, dataset_uri, dest_base_uri)", "response": "DEPRECATED - Copy a dataset to a different location."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cp(resume, quiet, dataset_uri, dest_base_uri):\n _copy(resume, quiet, dataset_uri, dest_base_uri)", "response": "Copy a dataset to a different location."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compress(obj, level=6, return_type=\"bytes\"):\n if isinstance(obj, binary_type):\n b = zlib.compress(obj, level)\n elif isinstance(obj, string_types):\n b = zlib.compress(obj.encode(\"utf-8\"), level)\n else:\n b = zlib.compress(pickle.dumps(obj, protocol=2), level)\n\n if return_type == \"bytes\":\n return b\n elif return_type == \"str\":\n return base64.b64encode(b).decode(\"utf-8\")\n else:\n raise ValueError(\"'return_type' has to be one of 'bytes', 'str'!\")", "response": "Compress a single object to bytes or base64."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _parsems(value):\n if \".\" not in value:\n return int(value), 0\n else:\n i, f = value.split(\".\")\n return int(i), int(f.ljust(6, \"0\")[:6])", "response": "Parse a I [. F ] seconds value into ( seconds microseconds )."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_probable_year_index(self, tokens):\n for index, token in enumerate(self):\n potential_year_tokens = _ymd.find_potential_year_tokens(\n token, tokens)\n if len(potential_year_tokens) == 1 and len(potential_year_tokens[0]) > 2:\n return index", "response": "Find the index of the first occurrence of a token in the list that is not longer than 100 year."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing the date or time string into a naive datetime object.", "response": "def parse(self, timestr, default=None, ignoretz=False, tzinfos=None, **kwargs):\n \"\"\"\n Parse the date/time string into a :class:`datetime.datetime` object.\n\n :param timestr:\n Any date/time string using the supported formats.\n\n :param default:\n The default datetime object, if this is a datetime object and not\n ``None``, elements specified in ``timestr`` replace elements in the\n default object.\n\n :param ignoretz:\n If set ``True``, time zones in parsed strings are ignored and a\n naive :class:`datetime.datetime` object is returned.\n\n :param tzinfos:\n Additional time zone names / aliases which may be present in the\n string. This argument maps time zone names (and optionally offsets\n from those time zones) to time zones. This parameter can be a\n dictionary with timezone aliases mapping time zone names to time\n zones or a function taking two parameters (``tzname`` and\n ``tzoffset``) and returning a time zone.\n\n The timezones to which the names are mapped can be an integer\n offset from UTC in minutes or a :class:`tzinfo` object.\n\n .. doctest::\n :options: +NORMALIZE_WHITESPACE\n\n >>> from dateutil.parser import parse\n >>> from dateutil.tz import gettz\n >>> tzinfos = {\"BRST\": -10800, \"CST\": gettz(\"America/Chicago\")}\n >>> parse(\"2012-01-19 17:21:00 BRST\", tzinfos=tzinfos)\n datetime.datetime(2012, 1, 19, 17, 21, tzinfo=tzoffset(u'BRST', -10800))\n >>> parse(\"2012-01-19 17:21:00 CST\", tzinfos=tzinfos)\n datetime.datetime(2012, 1, 19, 17, 21,\n tzinfo=tzfile('/usr/share/zoneinfo/America/Chicago'))\n\n This parameter is ignored if ``ignoretz`` is set.\n\n :param **kwargs:\n Keyword arguments as passed to ``_parse()``.\n\n :return:\n Returns a :class:`datetime.datetime` object or, if the\n ``fuzzy_with_tokens`` option is ``True``, returns a tuple, the\n first element being a :class:`datetime.datetime` object, the second\n a tuple containing the fuzzy tokens.\n\n :raises ValueError:\n Raised for invalid or unknown string format, if the provided\n :class:`tzinfo` is not in a valid format, or if an invalid date\n would be created.\n\n :raises TypeError:\n Raised for non-string or character stream input.\n\n :raises OverflowError:\n Raised if the parsed date exceeds the largest valid C integer on\n your system.\n \"\"\"\n\n if default is None:\n default = datetime.datetime.now().replace(hour=0, minute=0,\n second=0, microsecond=0)\n\n res, skipped_tokens = self._parse(timestr, **kwargs)\n\n if res is None:\n raise ValueError(\"Unknown string format\")\n\n if len(res) == 0:\n raise ValueError(\"String does not contain a date.\")\n\n repl = {}\n for attr in (\"year\", \"month\", \"day\", \"hour\",\n \"minute\", \"second\", \"microsecond\"):\n value = getattr(res, attr)\n if value is not None:\n repl[attr] = value\n\n if 'day' not in repl:\n # If the default day exceeds the last day of the month, fall back to\n # the end of the month.\n cyear = default.year if res.year is None else res.year\n cmonth = default.month if res.month is None else res.month\n cday = default.day if res.day is None else res.day\n\n if cday > monthrange(cyear, cmonth)[1]:\n repl['day'] = monthrange(cyear, cmonth)[1]\n\n ret = default.replace(**repl)\n\n if res.weekday is not None and not res.day:\n ret = ret + relativedelta.relativedelta(weekday=res.weekday)\n\n if not ignoretz:\n if (isinstance(tzinfos, collections.Callable) or\n tzinfos and res.tzname in tzinfos):\n\n if isinstance(tzinfos, collections.Callable):\n tzdata = tzinfos(res.tzname, res.tzoffset)\n else:\n tzdata = tzinfos.get(res.tzname)\n\n if isinstance(tzdata, datetime.tzinfo):\n tzinfo = tzdata\n elif isinstance(tzdata, text_type):\n tzinfo = tz.tzstr(tzdata)\n elif isinstance(tzdata, integer_types):\n tzinfo = tz.tzoffset(res.tzname, tzdata)\n else:\n raise ValueError(\"Offset must be tzinfo subclass, \"\n \"tz string, or int offset.\")\n ret = ret.replace(tzinfo=tzinfo)\n elif res.tzname and res.tzname in time.tzname:\n ret = ret.replace(tzinfo=tz.tzlocal())\n elif res.tzoffset == 0:\n ret = ret.replace(tzinfo=tz.tzutc())\n elif res.tzoffset:\n ret = ret.replace(tzinfo=tz.tzoffset(res.tzname, res.tzoffset))\n\n if kwargs.get('fuzzy_with_tokens', False):\n return ret, skipped_tokens\n else:\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tzname_in_python2(namefunc):\n def adjust_encoding(*args, **kwargs):\n name = namefunc(*args, **kwargs)\n if name is not None and not PY3:\n name = name.encode()\n\n return name\n\n return adjust_encoding", "response": "Change unicode output into bytestrings in Python 2 and Python 3 changed in Python 3."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the wall time of a given datetime is ambiguous in thisCOOKIE zone.", "response": "def is_ambiguous(self, dt):\n \"\"\"\n Whether or not the \"wall time\" of a given datetime is ambiguous in this\n zone.\n\n :param dt:\n A :py:class:`datetime.datetime`, naive or time zone aware.\n\n\n :return:\n Returns ``True`` if ambiguous, ``False`` otherwise.\n\n .. versionadded:: 2.6.0\n \"\"\"\n\n dt = dt.replace(tzinfo=self)\n\n wall_0 = enfold(dt, fold=0)\n wall_1 = enfold(dt, fold=1)\n\n same_offset = wall_0.utcoffset() == wall_1.utcoffset()\n same_dt = wall_0.replace(tzinfo=None) == wall_1.replace(tzinfo=None)\n\n return same_dt and not same_offset"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _fold_status(self, dt_utc, dt_wall):\n if self.is_ambiguous(dt_wall):\n delta_wall = dt_wall - dt_utc\n _fold = int(delta_wall == (dt_utc.utcoffset() - dt_utc.dst()))\n else:\n _fold = 0\n\n return _fold", "response": "Determines the fold status of a given UTC datetime."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a timezone - aware datetime in a given timezone calculates a new object containing the unambiguous entry - set entry for that datetime.", "response": "def _fromutc(self, dt):\n \"\"\"\n Given a timezone-aware datetime in a given timezone, calculates a\n timezone-aware datetime in a new timezone.\n\n Since this is the one time that we *know* we have an unambiguous\n datetime object, we take this opportunity to determine whether the\n datetime is ambiguous and in a \"fold\" state (e.g. if it's the first\n occurence, chronologically, of the ambiguous datetime).\n\n :param dt:\n A timezone-aware :class:`datetime.datetime` object.\n \"\"\"\n\n # Re-implement the algorithm from Python's datetime.py\n dtoff = dt.utcoffset()\n if dtoff is None:\n raise ValueError(\"fromutc() requires a non-None utcoffset() \"\n \"result\")\n\n # The original datetime.py code assumes that `dst()` defaults to\n # zero during ambiguous times. PEP 495 inverts this presumption, so\n # for pre-PEP 495 versions of python, we need to tweak the algorithm.\n dtdst = dt.dst()\n if dtdst is None:\n raise ValueError(\"fromutc() requires a non-None dst() result\")\n delta = dtoff - dtdst\n\n dt += delta\n # Set fold=1 so we can default to being in the fold for\n # ambiguous dates.\n dtdst = enfold(dt, fold=1).dst()\n if dtdst is None:\n raise ValueError(\"fromutc(): dt.dst gave inconsistent \"\n \"results; cannot convert\")\n return dt + dtdst"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating a from a timezone - aware datetime in a given timezone.", "response": "def fromutc(self, dt):\n \"\"\"\n Given a timezone-aware datetime in a given timezone, calculates a\n timezone-aware datetime in a new timezone.\n\n Since this is the one time that we *know* we have an unambiguous\n datetime object, we take this opportunity to determine whether the\n datetime is ambiguous and in a \"fold\" state (e.g. if it's the first\n occurance, chronologically, of the ambiguous datetime).\n\n :param dt:\n A timezone-aware :class:`datetime.datetime` object.\n \"\"\"\n dt_wall = self._fromutc(dt)\n\n # Calculate the fold status given the two datetimes.\n _fold = self._fold_status(dt, dt_wall)\n\n # Set the default fold value for ambiguous dates\n return enfold(dt_wall, fold=_fold)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fromutc(self, dt):\n if not isinstance(dt, datetime):\n raise TypeError(\"fromutc() requires a datetime argument\")\n\n if dt.tzinfo is not self:\n raise ValueError(\"dt.tzinfo is not self\")\n\n # Get transitions - if there are none, fixed offset\n transitions = self.transitions(dt.year)\n if transitions is None:\n return dt + self.utcoffset(dt)\n\n # Get the transition times in UTC\n dston, dstoff = transitions\n\n dston -= self._std_offset\n dstoff -= self._std_offset\n\n utc_transitions = (dston, dstoff)\n dt_utc = dt.replace(tzinfo=None)\n\n isdst = self._naive_isdst(dt_utc, utc_transitions)\n\n if isdst:\n dt_wall = dt + self._dst_offset\n else:\n dt_wall = dt + self._std_offset\n\n _fold = int(not isdst and self.is_ambiguous(dt_wall))\n\n return enfold(dt_wall, fold=_fold)", "response": "Given a datetime in UTC return a new time object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if the wall time of a given datetime is ambiguous in thisCOOKIE zone.", "response": "def is_ambiguous(self, dt):\n \"\"\"\n Whether or not the \"wall time\" of a given datetime is ambiguous in this\n zone.\n\n :param dt:\n A :py:class:`datetime.datetime`, naive or time zone aware.\n\n\n :return:\n Returns ``True`` if ambiguous, ``False`` otherwise.\n\n .. versionadded:: 2.6.0\n \"\"\"\n if not self.hasdst:\n return False\n\n start, end = self.transitions(dt.year)\n\n dt = dt.replace(tzinfo=None)\n return (end <= dt < end + self._dst_base_offset)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef strip_comment_line_with_symbol(line, start):\n parts = line.split(start)\n counts = [len(findall(r'(?:^|[^\"\\\\]|(?:\\\\\\\\|\\\\\")+)(\")', part))\n for part in parts]\n total = 0\n for nr, count in enumerate(counts):\n total += count\n if total % 2 == 0:\n return start.join(parts[:nr + 1]).rstrip()\n else: # pragma: no cover\n return line.rstrip()", "response": "Strip comments from line string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nstrip comments from json string.", "response": "def strip_comments(string, comment_symbols=frozenset(('#', '//'))):\n \"\"\"Strip comments from json string.\n\n :param string: A string containing json with comments started by comment_symbols.\n :param comment_symbols: Iterable of symbols that start a line comment (default # or //).\n :return: The string with the comments removed.\n \"\"\"\n lines = string.splitlines()\n for k in range(len(lines)):\n for symbol in comment_symbols:\n lines[k] = strip_comment_line_with_symbol(lines[k], start=symbol)\n return '\\n'.join(lines)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrebuilding the internal timezone info in dateutil / zoneinfo * tar * file", "response": "def rebuild(filename, tag=None, format=\"gz\", zonegroups=[], metadata=None):\n \"\"\"Rebuild the internal timezone info in dateutil/zoneinfo/zoneinfo*tar*\n\n filename is the timezone tarball from ftp.iana.org/tz.\n\n \"\"\"\n tmpdir = tempfile.mkdtemp()\n zonedir = os.path.join(tmpdir, \"zoneinfo\")\n moduledir = os.path.dirname(__file__)\n try:\n with tar_open(filename) as tf:\n for name in zonegroups:\n tf.extract(name, tmpdir)\n filepaths = [os.path.join(tmpdir, n) for n in zonegroups]\n try:\n check_call([\"zic\", \"-d\", zonedir] + filepaths)\n except OSError as e:\n _print_on_nosuchfile(e)\n raise\n # write metadata file\n with open(os.path.join(zonedir, METADATA_FN), 'w') as f:\n json.dump(metadata, f, indent=4, sort_keys=True)\n target = os.path.join(moduledir, ZONEFILENAME)\n with tar_open(target, \"w:%s\" % format) as tf:\n for entry in os.listdir(zonedir):\n entrypath = os.path.join(zonedir, entry)\n tf.add(entrypath, entry)\n finally:\n shutil.rmtree(tmpdir)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npicking the nth weekday of the given dayofweek.", "response": "def picknthweekday(year, month, dayofweek, hour, minute, whichweek):\n \"\"\" dayofweek == 0 means Sunday, whichweek 5 means last instance \"\"\"\n first = datetime.datetime(year, month, 1, hour, minute)\n\n # This will work if dayofweek is ISO weekday (1-7) or Microsoft-style (0-6),\n # Because 7 % 7 = 0\n weekdayone = first.replace(day=((dayofweek - first.isoweekday()) % 7) + 1)\n wd = weekdayone + ((whichweek - 1) * ONEWEEK)\n if (wd.month != month):\n wd -= ONEWEEK\n\n return wd"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef valuestodict(key):\n dout = {}\n size = winreg.QueryInfoKey(key)[1]\n tz_res = None\n\n for i in range(size):\n key_name, value, dtype = winreg.EnumValue(key, i)\n if dtype == winreg.REG_DWORD or dtype == winreg.REG_DWORD_LITTLE_ENDIAN:\n # If it's a DWORD (32-bit integer), it's stored as unsigned - convert\n # that to a proper signed integer\n if value & (1 << 31):\n value = value - (1 << 32)\n elif dtype == winreg.REG_SZ:\n # If it's a reference to the tzres DLL, load the actual string\n if value.startswith('@tzres'):\n tz_res = tz_res or tzres()\n value = tz_res.name_from_string(value)\n\n value = value.rstrip('\\x00') # Remove trailing nulls\n\n dout[key_name] = value\n\n return dout", "response": "Convert a registry key s values to a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_name(self, offset):\n resource = self.p_wchar()\n lpBuffer = ctypes.cast(ctypes.byref(resource), wintypes.LPWSTR)\n nchar = self.LoadStringW(self._tzres._handle, offset, lpBuffer, 0)\n return resource[:nchar]", "response": "Load a timezone name from a DLL offset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef name_from_string(self, tzname_str):\n if not tzname_str.startswith('@'):\n return tzname_str\n\n name_splt = tzname_str.split(',-')\n try:\n offset = int(name_splt[1])\n except:\n raise ValueError(\"Malformed timezone string.\")\n\n return self.load_name(offset)", "response": "Parse a string into a localized time zone name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef transitions(self, year):\n\n if not self.hasdst:\n return None\n\n dston = picknthweekday(year, self._dstmonth, self._dstdayofweek,\n self._dsthour, self._dstminute,\n self._dstweeknumber)\n\n dstoff = picknthweekday(year, self._stdmonth, self._stddayofweek,\n self._stdhour, self._stdminute,\n self._stdweeknumber)\n\n # Ambiguous dates default to the STD side\n dstoff -= self._dst_base_offset\n\n return dston, dstoff", "response": "Returns the DST on and off transition times expressed\n always on the standard time side."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_zonefile_instance(new_instance=False):\n if new_instance:\n zif = None\n else:\n zif = getattr(get_zonefile_instance, '_cached_instance', None)\n\n if zif is None:\n zif = ZoneInfoFile(getzoneinfofile_stream())\n\n get_zonefile_instance._cached_instance = zif\n\n return zif", "response": "Returns a new instance of the ZoneInfoFile class."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a time zone object for the given name.", "response": "def gettz(name):\n \"\"\"\n This retrieves a time zone from the local zoneinfo tarball that is packaged\n with dateutil.\n\n :param name:\n An IANA-style time zone name, as found in the zoneinfo file.\n\n :return:\n Returns a :class:`dateutil.tz.tzfile` time zone object.\n\n .. warning::\n It is generally inadvisable to use this function, and it is only\n provided for API compatibility with earlier versions. This is *not*\n equivalent to ``dateutil.tz.gettz()``, which selects an appropriate\n time zone based on the inputs, favoring system zoneinfo. This is ONLY\n for accessing the dateutil-specific zoneinfo (which may be out of\n date compared to the system zoneinfo).\n\n .. deprecated:: 2.6\n If you need to use a specific zoneinfofile over the system zoneinfo,\n instantiate a :class:`dateutil.zoneinfo.ZoneInfoFile` object and call\n :func:`dateutil.zoneinfo.ZoneInfoFile.get(name)` instead.\n\n Use :func:`get_zonefile_instance` to retrieve an instance of the\n dateutil-provided zoneinfo.\n \"\"\"\n warnings.warn(\"zoneinfo.gettz() will be removed in future versions, \"\n \"to use the dateutil-provided zoneinfo files, instantiate a \"\n \"ZoneInfoFile object and use ZoneInfoFile.zones.get() \"\n \"instead. See the documentation for details.\",\n DeprecationWarning)\n\n if len(_CLASS_ZONE_INSTANCE) == 0:\n _CLASS_ZONE_INSTANCE.append(ZoneInfoFile(getzoneinfofile_stream()))\n return _CLASS_ZONE_INSTANCE[0].zones.get(name)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef gettz_db_metadata():\n warnings.warn(\"zoneinfo.gettz_db_metadata() will be removed in future \"\n \"versions, to use the dateutil-provided zoneinfo files, \"\n \"ZoneInfoFile object and query the 'metadata' attribute \"\n \"instead. See the documentation for details.\",\n DeprecationWarning)\n\n if len(_CLASS_ZONE_INSTANCE) == 0:\n _CLASS_ZONE_INSTANCE.append(ZoneInfoFile(getzoneinfofile_stream()))\n return _CLASS_ZONE_INSTANCE[0].metadata", "response": "Get the zonefile metadata from the database."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the configuration for the given JID based on XMPP_HTTP_UPLOAD_ACCESS.", "response": "def get_config(jid):\n \"\"\"Get the configuration for the given JID based on XMPP_HTTP_UPLOAD_ACCESS.\n\n If the JID does not match any rule, ``False`` is returned.\n \"\"\"\n\n acls = getattr(settings, 'XMPP_HTTP_UPLOAD_ACCESS', (('.*', False), ))\n\n for regex, config in acls:\n if isinstance(regex, six.string_types):\n regex = [regex]\n\n for subex in regex:\n if re.search(subex, jid):\n return config\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving a datetime and a time zone determine whether or not a given datetime is in a gap.", "response": "def datetime_exists(dt, tz=None):\n \"\"\"\n Given a datetime and a time zone, determine whether or not a given datetime\n would fall in a gap.\n\n :param dt:\n A :class:`datetime.datetime` (whose time zone will be ignored if ``tz``\n is provided.)\n\n :param tz:\n A :class:`datetime.tzinfo` with support for the ``fold`` attribute. If\n ``None`` or not provided, the datetime's own time zone will be used.\n\n :return:\n Returns a boolean value whether or not the \"wall time\" exists in ``tz``.\n \"\"\"\n if tz is None:\n if dt.tzinfo is None:\n raise ValueError('Datetime is naive and no time zone provided.')\n tz = dt.tzinfo\n\n dt = dt.replace(tzinfo=None)\n\n # This is essentially a test of whether or not the datetime can survive\n # a round trip to UTC.\n dt_rt = dt.replace(tzinfo=tz).astimezone(tzutc()).astimezone(tz)\n dt_rt = dt_rt.replace(tzinfo=None)\n\n return dt == dt_rt"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef datetime_ambiguous(dt, tz=None):\n if tz is None:\n if dt.tzinfo is None:\n raise ValueError('Datetime is naive and no time zone provided.')\n\n tz = dt.tzinfo\n\n # If a time zone defines its own \"is_ambiguous\" function, we'll use that.\n is_ambiguous_fn = getattr(tz, 'is_ambiguous', None)\n if is_ambiguous_fn is not None:\n try:\n return tz.is_ambiguous(dt)\n except:\n pass\n\n # If it doesn't come out and tell us it's ambiguous, we'll just check if\n # the fold attribute has any effect on this particular date and time.\n dt = dt.replace(tzinfo=tz)\n wall_0 = enfold(dt, fold=0)\n wall_1 = enfold(dt, fold=1)\n\n same_offset = wall_0.utcoffset() == wall_1.utcoffset()\n same_dst = wall_0.dst() == wall_1.dst()\n\n return not (same_offset and same_dst)", "response": "Given a datetime and a time zone determine whether or not the wall time of the given datetime is ambiguous."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_ambiguous(self, dt):\n naive_dst = self._naive_is_dst(dt)\n return (not naive_dst and\n (naive_dst != self._naive_is_dst(dt - self._dst_saved)))", "response": "Returns True if the wall time of a given datetime is ambiguous in thisCOOKIE zone."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _set_tzdata(self, tzobj):\n # Copy the relevant attributes over as private attributes\n for attr in _tzfile.attrs:\n setattr(self, '_' + attr, getattr(tzobj, attr))", "response": "Set the time zone data of this object from a _tzfile object"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fromutc(self, dt):\n # These isinstance checks are in datetime.tzinfo, so we'll preserve\n # them, even if we don't care about duck typing.\n if not isinstance(dt, datetime.datetime):\n raise TypeError(\"fromutc() requires a datetime argument\")\n\n if dt.tzinfo is not self:\n raise ValueError(\"dt.tzinfo is not self\")\n\n # First treat UTC as wall time and get the transition we're in.\n idx = self._find_last_transition(dt, in_utc=True)\n tti = self._get_ttinfo(idx)\n\n dt_out = dt + datetime.timedelta(seconds=tti.offset)\n\n fold = self.is_ambiguous(dt_out, idx=idx)\n\n return enfold(dt_out, fold=int(fold))", "response": "Returns a new object representing the object in the time zone from the given datetime."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_ambiguous(self, dt, idx=None):\n if idx is None:\n idx = self._find_last_transition(dt)\n\n # Calculate the difference in offsets from current to previous\n timestamp = _datetime_to_timestamp(dt)\n tti = self._get_ttinfo(idx)\n\n if idx is None or idx <= 0:\n return False\n\n od = self._get_ttinfo(idx - 1).offset - tti.offset\n tt = self._trans_list[idx] # Transition time\n\n return timestamp < tt + od", "response": "Returns True if the wall time of a given datetime is ambiguous in thisCOOKIE zone."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef transitions(self, year):\n if not self.hasdst:\n return None\n\n base_year = datetime.datetime(year, 1, 1)\n\n start = base_year + self._start_delta\n end = base_year + self._end_delta\n\n return (start, end)", "response": "Returns the DST on and off transition times expressed\n always on the standard time side."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get(self, tzid=None):\n if tzid is None:\n if len(self._vtz) == 0:\n raise ValueError(\"no timezones defined\")\n elif len(self._vtz) > 1:\n raise ValueError(\"more than one timezone available\")\n tzid = next(iter(self._vtz))\n\n return self._vtz.get(tzid)", "response": "Retrieves a time zone object by its tzid."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef normalized(self):\n # Cascade remainders down (rounding each to roughly nearest\n # microsecond)\n days = int(self.days)\n\n hours_f = round(self.hours + 24 * (self.days - days), 11)\n hours = int(hours_f)\n\n minutes_f = round(self.minutes + 60 * (hours_f - hours), 10)\n minutes = int(minutes_f)\n\n seconds_f = round(self.seconds + 60 * (minutes_f - minutes), 8)\n seconds = int(seconds_f)\n\n microseconds = round(self.microseconds + 1e6 * (seconds_f - seconds))\n\n # Constructor carries overflow back up with call to _fix()\n return self.__class__(years=self.years, months=self.months,\n days=days, hours=hours, minutes=minutes,\n seconds=seconds, microseconds=microseconds,\n leapdays=self.leapdays, year=self.year,\n month=self.month, day=self.day,\n weekday=self.weekday, hour=self.hour,\n minute=self.minute, second=self.second,\n microsecond=self.microsecond)", "response": "Return a version of this object represented entirely using integer\n values for the relative attributes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_algorithm(alg: str) -> Callable:\n if alg not in algorithms:\n raise ValueError('Invalid algorithm: {:s}'.format(alg))\n return algorithms[alg]", "response": "Returns a function that returns the requested algorithm."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _hash(secret: bytes, data: bytes, alg: str) -> bytes:\n algorithm = get_algorithm(alg)\n return hmac \\\n .new(secret, msg=data, digestmod=algorithm) \\\n .digest()", "response": "Create a new HMAC hash."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencode the payload and header in a new token.", "response": "def encode(secret: Union[str, bytes], payload: dict = None,\n alg: str = default_alg, header: dict = None) -> str:\n \"\"\"\n :param secret: The secret used to encode the token.\n :type secret: Union[str, bytes]\n :param payload: The payload to be encoded in the token.\n :type payload: dict\n :param alg: The algorithm used to hash the token.\n :type alg: str\n :param header: The header to be encoded in the token.\n :type header: dict\n :return: A new token\n :rtype: str\n \"\"\"\n secret = util.to_bytes(secret)\n\n payload = payload or {}\n header = header or {}\n\n header_json = util.to_bytes(json.dumps(header))\n header_b64 = util.b64_encode(header_json)\n payload_json = util.to_bytes(json.dumps(payload))\n payload_b64 = util.b64_encode(payload_json)\n\n pre_signature = util.join(header_b64, payload_b64)\n signature = _hash(secret, pre_signature, alg)\n signature_b64 = util.b64_encode(signature)\n\n token = util.join(pre_signature, signature_b64)\n return util.from_bytes(token)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndecoding the given token and validates the signature.", "response": "def decode(secret: Union[str, bytes], token: Union[str, bytes],\n alg: str = default_alg) -> Tuple[dict, dict]:\n \"\"\"\n Decodes the given token's header and payload and validates the signature.\n\n :param secret: The secret used to decode the token. Must match the\n secret used when creating the token.\n :type secret: Union[str, bytes]\n :param token: The token to decode.\n :type token: Union[str, bytes]\n :param alg: The algorithm used to decode the token. Must match the\n algorithm used when creating the token.\n :type alg: str\n :return: The decoded header and payload.\n :rtype: Tuple[dict, dict]\n \"\"\"\n secret = util.to_bytes(secret)\n token = util.to_bytes(token)\n pre_signature, signature_segment = token.rsplit(b'.', 1)\n header_b64, payload_b64 = pre_signature.split(b'.')\n try:\n header_json = util.b64_decode(header_b64)\n header = json.loads(util.from_bytes(header_json))\n except (json.decoder.JSONDecodeError, UnicodeDecodeError, ValueError):\n raise InvalidHeaderError('Invalid header')\n try:\n payload_json = util.b64_decode(payload_b64)\n payload = json.loads(util.from_bytes(payload_json))\n except (json.decoder.JSONDecodeError, UnicodeDecodeError, ValueError):\n raise InvalidPayloadError('Invalid payload')\n\n if not isinstance(header, dict):\n raise InvalidHeaderError('Invalid header: {}'.format(header))\n if not isinstance(payload, dict):\n raise InvalidPayloadError('Invalid payload: {}'.format(payload))\n\n signature = util.b64_decode(signature_segment)\n calculated_signature = _hash(secret, pre_signature, alg)\n\n if not compare_signature(signature, calculated_signature):\n raise InvalidSignatureError('Invalid signature')\n return header, payload"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncompares the given signature and returns True if the signature matches.", "response": "def compare_signature(expected: Union[str, bytes],\n actual: Union[str, bytes]) -> bool:\n \"\"\"\n Compares the given signatures.\n\n :param expected: The expected signature.\n :type expected: Union[str, bytes]\n :param actual: The actual signature.\n :type actual: Union[str, bytes]\n :return: Do the signatures match?\n :rtype: bool\n \"\"\"\n expected = util.to_bytes(expected)\n actual = util.to_bytes(actual)\n return hmac.compare_digest(expected, actual)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompare the given tokens. :param expected: The expected token. :type expected: Union[str, bytes] :param actual: The actual token. :type actual: Union[str, bytes] :return: Do the tokens match? :rtype: bool", "response": "def compare_token(expected: Union[str, bytes],\n actual: Union[str, bytes]) -> bool:\n \"\"\"\n Compares the given tokens.\n\n :param expected: The expected token.\n :type expected: Union[str, bytes]\n :param actual: The actual token.\n :type actual: Union[str, bytes]\n :return: Do the tokens match?\n :rtype: bool\n \"\"\"\n expected = util.to_bytes(expected)\n actual = util.to_bytes(actual)\n _, expected_sig_seg = expected.rsplit(b'.', 1)\n _, actual_sig_seg = actual.rsplit(b'.', 1)\n expected_sig = util.b64_decode(expected_sig_seg)\n actual_sig = util.b64_decode(actual_sig_seg)\n return compare_signature(expected_sig, actual_sig)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the token header.", "response": "def header(self) -> dict:\n \"\"\"\n :return: Token header.\n :rtype: dict\n \"\"\"\n header = {}\n if isinstance(self._header, dict):\n header = self._header.copy()\n header.update(self._header)\n header.update({\n 'type': 'JWT',\n 'alg': self.alg\n })\n return header"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if the token is valid.", "response": "def valid(self, time: int = None) -> bool:\n \"\"\"\n Is the token valid? This method only checks the timestamps within the\n token and compares them against the current time if none is provided.\n\n :param time: The timestamp to validate against\n :type time: Union[int, None]\n :return: The validity of the token.\n :rtype: bool\n \"\"\"\n if time is None:\n epoch = datetime(1970, 1, 1, 0, 0, 0)\n now = datetime.utcnow()\n time = int((now - epoch).total_seconds())\n if isinstance(self.valid_from, int) and time < self.valid_from:\n return False\n if isinstance(self.valid_to, int) and time > self.valid_to:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _pop_claims_from_payload(self):\n claims_in_payload = [k for k in self.payload.keys() if\n k in registered_claims.values()]\n for name in claims_in_payload:\n self.registered_claims[name] = self.payload.pop(name)", "response": "Pop claims from the payload and update the self. registered_claims property."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef encode(self) -> str:\n payload = {}\n payload.update(self.registered_claims)\n payload.update(self.payload)\n return encode(self.secret, payload, self.alg, self.header)", "response": "Encode the data held in the class into a token."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decode(secret: Union[str, bytes], token: Union[str, bytes],\n alg: str = default_alg) -> 'Jwt':\n \"\"\"\n Decodes the given token into an instance of `Jwt`.\n\n :param secret: The secret used to decode the token. Must match the\n secret used when creating the token.\n :type secret: Union[str, bytes]\n :param token: The token to decode.\n :type token: Union[str, bytes]\n :param alg: The algorithm used to decode the token. Must match the\n algorithm used when creating the token.\n :type alg: str\n :return: The decoded token.\n :rtype: `Jwt`\n \"\"\"\n header, payload = decode(secret, token, alg)\n return Jwt(secret, payload, alg, header)", "response": "Decodes the given token into an instance of Jwt."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompare this Jwt with another Jwt.", "response": "def compare(self, jwt: 'Jwt', compare_dates: bool = False) -> bool:\n \"\"\"\n Compare against another `Jwt`.\n\n :param jwt: The token to compare against.\n :type jwt: Jwt\n :param compare_dates: Should the comparision take dates into account?\n :type compare_dates: bool\n :return: Are the two Jwt's the same?\n :rtype: bool\n \"\"\"\n if self.secret != jwt.secret:\n return False\n if self.payload != jwt.payload:\n return False\n if self.alg != jwt.alg:\n return False\n if self.header != jwt.header:\n return False\n expected_claims = self.registered_claims\n actual_claims = jwt.registered_claims\n if not compare_dates:\n strip = ['exp', 'nbf', 'iat']\n expected_claims = {k: {v if k not in strip else None} for k, v in\n expected_claims.items()}\n actual_claims = {k: {v if k not in strip else None} for k, v in\n actual_claims.items()}\n if expected_claims != actual_claims:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbase 64 encode data.", "response": "def b64_encode(data: bytes) -> bytes:\n \"\"\"\n :param data: Data the encode.\n :type data: bytes\n :return: Base 64 encoded data with padding removed.\n :rtype: bytes\n \"\"\"\n encoded = urlsafe_b64encode(data)\n return encoded.replace(b'=', b'')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef b64_decode(data: bytes) -> bytes:\n missing_padding = len(data) % 4\n if missing_padding != 0:\n data += b'=' * (4 - missing_padding)\n return urlsafe_b64decode(data)", "response": "Decode base 64 encoded data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert data to bytes.", "response": "def to_bytes(data: Union[str, bytes]) -> bytes:\n \"\"\"\n :param data: Data to convert to bytes.\n :type data: Union[str, bytes]\n :return: `data` encoded to UTF8.\n :rtype: bytes\n \"\"\"\n if isinstance(data, bytes):\n return data\n return data.encode('utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndecoding a UTF8 byte string into a base64 - encoded version of the object.", "response": "def from_bytes(data: Union[str, bytes]) -> str:\n \"\"\"\n :param data: A UTF8 byte string.\n :type data: Union[str, bytes]\n :return: `data` decoded from UTF8.\n :rtype: str\n \"\"\"\n if isinstance(data, str):\n return data\n return str(data, 'utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef camelize_classname(base, tablename, table):\n \"Produce a 'camelized' class name, e.g. \"\n \"'words_and_underscores' -> 'WordsAndUnderscores'\"\n return str(tablename[0].upper() +\n re.sub(r'_([a-z])', lambda m: m.group(1).upper(),\n tablename[1:]))", "response": "Produce a camelized class name e. g. words_and_underscores -> words_and_underscores"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nproduce an uncamelized pluralized class name e. g.", "response": "def pluralize_collection(base, local_cls, referred_cls, constraint):\n \"Produce an 'uncamelized', 'pluralized' class name, e.g. \"\n \"'SomeTerm' -> 'some_terms'\"\n referred_name = referred_cls.__name__\n uncamelized = re.sub(r'[A-Z]',\n lambda m: \"_%s\" % m.group(0).lower(),\n referred_name)[1:]\n pluralized = _pluralizer.plural(uncamelized)\n return pluralized"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_compressed_json_file(abspath):\n abspath = abspath.lower()\n fname, ext = os.path.splitext(abspath)\n if ext in [\".json\", \".js\"]:\n is_compressed = False\n elif ext == \".gz\":\n is_compressed = True\n else:\n raise ValueError(\n \"'%s' is not a valid json file. \"\n \"extension has to be '.json' or '.js' for uncompressed, '.gz' \"\n \"for compressed.\" % abspath)\n return is_compressed", "response": "Test a file is a valid json file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndumps a set into a dictionary.", "response": "def dump_set(self, obj, class_name=set_class_name):\n \"\"\"\n ``set`` dumper.\n \"\"\"\n return {\"$\" + class_name: [self._json_convert(item) for item in obj]}"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dump_deque(self, obj, class_name=\"collections.deque\"):\n return {\"$\" + class_name: [self._json_convert(item) for item in obj]}", "response": "Dump a deque into a JSON - compatible dict."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndumping an OrderedDict into a dictionary.", "response": "def dump_OrderedDict(self, obj, class_name=\"collections.OrderedDict\"):\n \"\"\"\n ``collections.OrderedDict`` dumper.\n \"\"\"\n return {\n \"$\" + class_name: [\n (key, self._json_convert(value)) for key, value in iteritems(obj)\n ]\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dump_nparray(self, obj, class_name=numpy_ndarray_class_name):\n return {\"$\" + class_name: self._json_convert(obj.tolist())}", "response": "Dump a numpy. ndarray into a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _invalidates_cache(f):\n\n def inner_func(self, *args, **kwargs):\n rv = f(self, *args, **kwargs)\n self._invalidate_cache()\n return rv\n\n return inner_func", "response": "Decorator for rruleset methods which may invalidate the cached length."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef before(self, dt, inc=False):\n if self._cache_complete:\n gen = self._cache\n else:\n gen = self\n last = None\n if inc:\n for i in gen:\n if i > dt:\n break\n last = i\n else:\n for i in gen:\n if i >= dt:\n break\n last = i\n return last", "response": "Returns the last recurrence before the given datetime instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef after(self, dt, inc=False):\n if self._cache_complete:\n gen = self._cache\n else:\n gen = self\n if inc:\n for i in gen:\n if i >= dt:\n return i\n else:\n for i in gen:\n if i > dt:\n return i\n return None", "response": "Returns the first recurrence after the given datetime instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef xafter(self, dt, count=None, inc=False):\n\n if self._cache_complete:\n gen = self._cache\n else:\n gen = self\n\n # Select the comparison function\n if inc:\n def comp(dc, dtc): return dc >= dtc\n else:\n def comp(dc, dtc): return dc > dtc\n\n # Generate dates\n n = 0\n for d in gen:\n if comp(d, dt):\n if count is not None:\n n += 1\n if n > count:\n break\n\n yield d", "response": "Generator which yields up to count recurrences after the given datetime instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef replace(self, **kwargs):\n new_kwargs = {\"interval\": self._interval,\n \"count\": self._count,\n \"dtstart\": self._dtstart,\n \"freq\": self._freq,\n \"until\": self._until,\n \"wkst\": self._wkst,\n \"cache\": False if self._cache is None else True}\n new_kwargs.update(self._original_rule)\n new_kwargs.update(kwargs)\n return rrule(**new_kwargs)", "response": "Return a new rrule with same attributes except for those attributes given new\n values by whichever keyword arguments are specified."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconstructs a set of all possible values for a given frequency range.", "response": "def __construct_byset(self, start, byxxx, base):\n \"\"\"\n If a `BYXXX` sequence is passed to the constructor at the same level as\n `FREQ` (e.g. `FREQ=HOURLY,BYHOUR={2,4,7},INTERVAL=3`), there are some\n specifications which cannot be reached given some starting conditions.\n\n This occurs whenever the interval is not coprime with the base of a\n given unit and the difference between the starting position and the\n ending position is not coprime with the greatest common denominator\n between the interval and the base. For example, with a FREQ of hourly\n starting at 17:00 and an interval of 4, the only valid values for\n BYHOUR would be {21, 1, 5, 9, 13, 17}, because 4 and 24 are not\n coprime.\n\n :param start:\n Specifies the starting position.\n :param byxxx:\n An iterable containing the list of allowed values.\n :param base:\n The largest allowable value for the specified frequency (e.g.\n 24 hours, 60 minutes).\n\n This does not preserve the type of the iterable, returning a set, since\n the values should be unique and the order is irrelevant, this will\n speed up later lookups.\n\n In the event of an empty set, raises a :exception:`ValueError`, as this\n results in an empty rrule.\n \"\"\"\n\n cset = set()\n\n # Support a single byxxx value.\n if isinstance(byxxx, integer_types):\n byxxx = (byxxx, )\n\n for num in byxxx:\n i_gcd = gcd(self._interval, base)\n # Use divmod rather than % because we need to wrap negative nums.\n if i_gcd == 1 or divmod(num - start, i_gcd)[1] == 0:\n cset.add(num)\n\n if len(cset) == 0:\n raise ValueError(\"Invalid rrule byxxx generates an empty set.\")\n\n return cset"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __mod_distance(self, value, byxxx, base):\n accumulator = 0\n for ii in range(1, base + 1):\n # Using divmod() over % to account for negative intervals\n div, value = divmod(value + self._interval, base)\n accumulator += div\n if value in byxxx:\n return (accumulator, value)", "response": "Calculates the distance between the value and the next specified value in byxxx."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _handle_BYWEEKDAY(self, rrkwargs, name, value, **kwargs):\n l = []\n for wday in value.split(','):\n if '(' in wday:\n # If it's of the form TH(+1), etc.\n splt = wday.split('(')\n w = splt[0]\n n = int(splt[1][:-1])\n elif len(wday):\n # If it's of the form +1MO\n for i in range(len(wday)):\n if wday[i] not in '+-0123456789':\n break\n n = wday[:i] or None\n w = wday[i:]\n if n:\n n = int(n)\n else:\n raise ValueError(\"Invalid (empty) BYDAY specification.\")\n\n l.append(weekdays[self._weekday_map[w]](n))\n rrkwargs[\"byweekday\"] = l", "response": "Handle BYDAY keyword arguments."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating what percentage a given number is of another, e.g. 50 is 50% of 100.", "response": "def run_get_percentage():\n \"\"\"\n Calculate what percentage a given number is of another,\n e.g. 50 is 50% of 100.\n \"\"\"\n description = run_get_percentage.__doc__\n parser = argparse.ArgumentParser(\n prog='get_percentage',\n description=description,\n epilog=\"Example use: get_percentage 25 100\",\n )\n parser.add_argument(\n 'a', help='Integer or floating point number that is a percent of another number'\n )\n parser.add_argument(\n 'b',\n help='Integer or floating point number of which the first number is a percent',\n )\n args = parser.parse_args()\n print(sm.get_percentage(float(args.a), float(args.b)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun the excel_to_html function from the command - line.", "response": "def run_excel_to_html():\n \"\"\"\n Run the excel_to_html function from the\n command-line.\n\n Args:\n -p path to file\n -s name of the sheet to convert\n -css classes to apply\n -m attempt to combine merged cells\n -c caption for accessibility\n -su summary for accessibility\n -d details for accessibility\n\n Example use:\n\n excel_to_html -p myfile.xlsx -s SheetName -css diablo-python -m true\n \"\"\"\n # Capture commandline arguments. prog='' argument must\n # match the command name in setup.py entry_points\n parser = argparse.ArgumentParser(prog='excel_to_html')\n parser.add_argument('-p', nargs='?', help='Path to an excel file for conversion.')\n parser.add_argument(\n '-s',\n nargs='?',\n help='The name of a sheet in our excel file. Defaults to \"Sheet1\".',\n )\n parser.add_argument(\n '-css', nargs='?', help='Space separated css classes to append to the table.'\n )\n parser.add_argument(\n '-m', action='store_true', help='Merge, attempt to combine merged cells.'\n )\n parser.add_argument(\n '-c', nargs='?', help='Caption for creating an accessible table.'\n )\n parser.add_argument(\n '-d',\n nargs='?',\n help='Two strings separated by a | character. The first string \\\n is for the html \"summary\" attribute and the second string is for the html \"details\" attribute. \\\n both values must be provided and nothing more.',\n )\n parser.add_argument(\n '-r', action='store_true', help='Row headers. Does the table have row headers?'\n )\n\n args = parser.parse_args()\n inputs = {\n 'p': args.p,\n 's': args.s,\n 'css': args.css,\n 'm': args.m,\n 'c': args.c,\n 'd': args.d,\n 'r': args.r,\n }\n\n p = inputs['p']\n s = inputs['s'] if inputs['s'] else 'Sheet1'\n css = inputs['css'] if inputs['css'] else ''\n m = inputs['m'] if inputs['m'] else False\n c = inputs['c'] if inputs['c'] else ''\n d = inputs['d'].split('|') if inputs['d'] else []\n r = inputs['r'] if inputs['r'] else False\n\n html = fp.excel_to_html(\n p, sheetname=s, css_classes=css, caption=c, details=d, row_headers=r, merge=m\n )\n\n print(html)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the return string for a language that s supported by python.", "response": "def get_built_in(self, language, level, data):\n \"\"\"\n Gets the return string for a language that's supported by python.\n Used in cases when python provides support for the conversion.\n \n Args:\n language: string the langage to return for.\n\n level: integer, the indentation level.\n\n data: python data structure being converted (list of tuples)\n\n Returns:\n None, updates self.data_structure \n \"\"\"\n # Language is python\n pp = pprint.PrettyPrinter(indent=level)\n\n lookup = {'python' : pp.pformat(data),\n 'json' : str(json.dumps(data, sort_keys=True, indent=level, separators=(',', ': ')))}\n\n self.data_structure = lookup[language]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the inner template for the given language and type.", "response": "def get_inner_template(self, language, template_type, indentation, key, val):\n \"\"\"\n Gets the requested template for the given language.\n\n Args:\n language: string, the language of the template to look for.\n\n template_type: string, 'iterable' or 'singular'. \n An iterable template is needed when the value is an iterable\n and needs more unpacking, e.g. list, tuple. A singular template \n is needed when unpacking is complete and the value is singular, \n e.g. string, int, float.\n\n indentation: int, the indentation level.\n \n key: multiple types, the array key.\n\n val: multiple types, the array values\n\n Returns:\n string, template formatting for arrays by language.\n \"\"\"\n #Language specific inner templates\n inner_templates = {'php' : {\n 'iterable' : '%s%s => array \\n%s( \\n%s%s),\\n' % (indentation, key, indentation, val, indentation),\n 'singular' : '%s%s => %s, \\n' % (indentation, key, val) },\n 'javascript' : {\n 'iterable' : '%s%s : {\\n%s\\n%s},\\n' % (indentation, key, val, indentation),\n 'singular' : '%s%s: %s,\\n' % (indentation, key, val)},\n 'ocaml' : { \n 'iterable' : '%s[| (%s, (\\n%s\\n%s))|] ;;\\n' % (indentation, key, val, indentation),\n 'singular' : '%s(%s, %s);\\n' % (indentation, key, val)}}\n\n return inner_templates[language][template_type]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntranslating a string representing a php array into a data structure in the specified language.", "response": "def translate_array(self, string, language, level=3, retdata=False):\n \"\"\"Unserializes a serialized php array and prints it to\n the console as a data structure in the specified language.\n Used to translate or convert a php array into a data structure \n in another language. Currently supports, PHP, Python, Javascript,\n and JSON. \n\n Args:\n string: a string of serialized php\n \n language: a string representing the desired output \n format for the array.\n\n level: integer, indentation level in spaces. \n Defaults to 3.\n\n retdata: boolean, the method will return the string\n in addition to printing it if set to True. Defaults \n to false.\n\n Returns:\n None but prints a string to the console if retdata is \n False, otherwise returns a string.\n \"\"\"\n language = language.lower()\n assert self.is_built_in(language) or language in self.outer_templates, \\\n \"Sorry, \" + language + \" is not a supported language.\"\n\n # Serialized data converted to a python data structure (list of tuples)\n data = phpserialize.loads(bytes(string, 'utf-8'), array_hook=list, decode_strings=True)\n\n # If language conversion is supported by python avoid recursion entirely\n # and use a built in library\n if self.is_built_in(language):\n self.get_built_in(language, level, data) \n print(self)\n return self.data_structure if retdata else None\n\n # The language is not supported. Use recursion to build a data structure.\n def loop_print(iterable, level=3):\n \"\"\"\n Loops over a python representation of a php array \n (list of tuples) and constructs a representation in another language.\n Translates a php array into another structure.\n\n Args:\n iterable: list or tuple to unpack.\n\n level: integer, number of spaces to use for indentation\n \"\"\"\n retval = ''\n indentation = ' ' * level\n\n # Base case - variable is not an iterable\n if not self.is_iterable(iterable) or isinstance(iterable, str):\n non_iterable = str(iterable)\n return str(non_iterable)\n \n # Recursive case\n for item in iterable:\n # If item is a tuple it should be a key, value pair\n if isinstance(item, tuple) and len(item) == 2:\n # Get the key value pair\n key = item[0]\n val = loop_print(item[1], level=level+3)\n \n # Translate special values\n val = self.translate_val(language, val) if language in self.lang_specific_values \\\n and val in self.lang_specific_values[language] else val\n \n # Convert keys to their properly formatted strings\n # Integers are not quoted as array keys\n key = str(key) if isinstance(key, int) else '\\'' + str(key) + '\\''\n\n # The first item is a key and the second item is an iterable, boolean\n needs_unpacking = hasattr(item[0],'__iter__') == False \\\n and hasattr(item[1],'__iter__') == True \n\n # The second item is an iterable\n if needs_unpacking:\n retval += self.get_inner_template(language, 'iterable', indentation, key, val)\n # The second item is not an iterable\n else:\n # Convert values to their properly formatted strings\n # Integers and booleans are not quoted as array values\n val = str(val) if val.isdigit() or val in self.lang_specific_values[language].values() else '\\'' + str(val) + '\\''\n\n retval += self.get_inner_template(language, 'singular', indentation, key, val) \n\n return retval\n \n # Execute the recursive call in language specific wrapper template\n self.data_structure = self.outer_templates[language] % (loop_print(data))\n print(self)\n return self.data_structure if retdata else None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _validate(config):\n for mandatory_key in _mandatory_keys:\n if mandatory_key not in config:\n raise KeyError(mandatory_key)\n for key in config.keys():\n if key not in _mandatory_keys and key not in _optional_keys:\n raise SyntaxError(key)\n if not isinstance(config[key], _default_config[key].__class__):\n raise ValueError(key)", "response": "Validate the config dictionary for the key sequence."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _create_default_config():\n config_path = _get_config_path()\n with open(config_path, 'w+') as f:\n yaml.dump(_default_config, f, default_flow_style=False)", "response": "Writes the default configuration to the appropriate place."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reusable(func):\n sig = signature(func)\n origin = func\n while hasattr(origin, '__wrapped__'):\n origin = origin.__wrapped__\n return type(\n origin.__name__,\n (ReusableGenerator, ),\n dict([\n ('__doc__', origin.__doc__),\n ('__module__', origin.__module__),\n ('__signature__', sig),\n ('__wrapped__', staticmethod(func)),\n ] + [\n (name, property(compose(itemgetter(name),\n attrgetter('_bound_args.arguments'))))\n for name in sig.parameters\n ] + ([\n ('__qualname__', origin.__qualname__),\n ] if sys.version_info > (3, ) else [])))", "response": "Create a reusable class from a generator function."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending an item into a generator expecting a final return value", "response": "def sendreturn(gen, value):\n \"\"\"Send an item into a generator expecting a final return value\n\n Parameters\n ----------\n gen: ~typing.Generator[T_yield, T_send, T_return]\n the generator to send the value to\n value: T_send\n the value to send\n\n Raises\n ------\n RuntimeError\n if the generator did not return as expected\n\n Returns\n -------\n T_return\n the generator's return value\n \"\"\"\n try:\n gen.send(value)\n except StopIteration as e:\n return stopiter_value(e)\n else:\n raise RuntimeError('generator did not return as expected')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef imap_send(func, gen):\n gen = iter(gen)\n assert _is_just_started(gen)\n yielder = yield_from(gen)\n for item in yielder:\n with yielder:\n yielder.send(func((yield item)))\n return_(yielder.result)", "response": "Apply a function to all send values of a generator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new generator by relaying yield and send interactions through another generator.", "response": "def irelay(gen, thru):\n \"\"\"Create a new generator by relaying yield/send interactions\n through another generator\n\n Parameters\n ----------\n gen: Generable[T_yield, T_send, T_return]\n the original generator\n thru: ~typing.Callable[[T_yield], ~typing.Generator]\n the generator callable through which each interaction is relayed\n\n Returns\n -------\n ~typing.Generator\n the relayed generator\n \"\"\"\n gen = iter(gen)\n assert _is_just_started(gen)\n\n yielder = yield_from(gen)\n for item in yielder:\n with yielder:\n\n subgen = thru(item)\n subyielder = yield_from(subgen)\n for subitem in subyielder:\n with subyielder:\n subyielder.send((yield subitem))\n\n yielder.send(subyielder.result)\n\n return_(yielder.result)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if all command dependencies refers to and existing command. If not raises an error. ProjectfileError will be raised with the problematic dependency and it s command.", "response": "def _data_integrity_check(data):\r\n \"\"\"Checks if all command dependencies refers to and existing command. If not, a ProjectfileError\r\n will be raised with the problematic dependency and it's command.\r\n\r\n :param data: parsed raw data set.\r\n :return: None\r\n \"\"\"\r\n deps = []\r\n for command in data['commands']:\r\n if 'dependencies' in data['commands'][command]:\r\n for d in data['commands'][command]['dependencies']:\r\n deps.append({\r\n 'd': d,\r\n 'c': command\r\n })\r\n for d in deps:\r\n if d['d'] not in data['commands']:\r\n raise error.ProjectfileError({\r\n 'error': error.PROJECTFILE_INVALID_DEPENDENCY.format(d['d'], d['c'])\r\n })"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npopulate any database related fields that have _getters to populate them with", "response": "def _link_rels(obj, fields=None, save=False, overwrite=False):\n \"\"\"Populate any database related fields (ForeignKeyField, OneToOneField) that have `_get`ters to populate them with\"\"\"\n if not fields:\n meta = obj._meta\n fields = [f.name for f in meta.fields if hasattr(f, 'do_related_class') and not f.primary_key and hasattr(meta, '_get_' + f.name) and hasattr(meta, '_' + f.name)]\n for field in fields:\n # skip fields if they contain non-null data and `overwrite` option wasn't set\n if not overwrite and not isinstance(getattr(obj, field, None), NoneType):\n # print 'skipping %s which already has a value of %s' % (field, getattr(obj, field, None))\n continue\n if hasattr(obj, field):\n setattr(obj, field, getattr(obj, '_' + field, None))\n if save:\n obj.save()\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates any database fields that have _getters to populate them with", "response": "def _update(obj, fields=None, save=False, overwrite=False):\n \"\"\"Update/populate any database fields that have `_get`ters to populate them with, regardless of whether they are data fields or related fields\"\"\"\n if not fields:\n meta = obj._meta\n fields = [f.name for f in meta.fields if not f.primary_key and hasattr(meta, '_get_' + f.name) and hasattr(meta, '_' + f.name)]\n # print fields\n fields_updated = []\n for field in fields:\n # skip fields if they contain non-null data and `overwrite` option wasn't set\n if not overwrite and not getattr(obj, field, None) == None:\n # print 'skipping %s which already has a value of %s' % (field, getattr(obj, field, None))\n continue\n # print field\n if hasattr(obj, field):\n # print field, getattr(obj, '_' + field, None)\n setattr(obj, field, getattr(obj, '_' + field, None))\n if getattr(obj, field, None) != None:\n fields_updated += [field]\n if save:\n obj.save()\n return fields_updated"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprint the traceback and invokes the ipython debugger on any exception in the order they were passed.", "response": "def bug_info(exc_type, exc_value, exc_trace):\n \"\"\"Prints the traceback and invokes the ipython debugger on any exception\n\n Only invokes ipydb if you are outside ipython or python interactive session.\n So scripts must be called from OS shell in order for exceptions to ipy-shell-out.\n\n Dependencies:\n Needs `pip install ipdb`\n\n Arguments:\n exc_type (type): The exception type/class (e.g. RuntimeError)\n exc_value (Exception): The exception instance (e.g. the error message passed to the Exception constructor)\n exc_trace (Traceback): The traceback instance\n \n References:\n http://stackoverflow.com/a/242531/623735\n\n Example Usage:\n $ python -c 'from pug import debug;x=[];x[0]'\n Traceback (most recent call last):\n File \"\", line 1, in \n IndexError: list index out of range\n\n > (1)()\n\n ipdb> x\n []\n ipdb> locals()\n {'__builtins__': , '__package__': None, 'x': [], 'debug': , '__name__': '__main__', '__doc__': None}\n ipdb> \n \"\"\"\n if hasattr(sys, 'ps1') or not sys.stderr.isatty():\n # We are in interactive mode or don't have a tty-like device, so we call the default hook\n sys.__excepthook__(exc_type, exc_value, exc_trace)\n else:\n # Need to import non-built-ins here, so if dependencies haven't been installed, both tracebacks will print\n # (e.g. the ImportError and the Exception that got you here)\n import ipdb\n # We are NOT in interactive mode, print the exception\n traceback.print_exception(exc_type, exc_value, exc_trace)\n print\n # Start the debugger in post-mortem mode.\n ipdb.post_mortem(exc_trace)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncopying a file from its location on the web to a designated place on the local machine.", "response": "def copy_web_file_to_local(file_path, target_path):\n \"\"\"Copies a file from its location on the web to a designated \n place on the local machine.\n\n Args:\n file_path: Complete url of the file to copy, string (e.g. http://fool.com/input.css).\n\n target_path: Path and name of file on the local machine, string. (e.g. /directory/output.css)\n\n Returns:\n None.\n\n \"\"\"\n response = urllib.request.urlopen(file_path)\n f = open(target_path, 'w')\n f.write(response.read()) \n f.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_line_count(fname):\n i = 0\n with open(fname) as f:\n for i, l in enumerate(f):\n pass\n return i + 1", "response": "Counts the number of lines in a file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef indent_css(f, output):\n line_count = get_line_count(f)\n f = open(f, 'r+')\n output = open(output, 'r+')\n for line in range(line_count):\n string = f.readline().rstrip()\n if len(string) > 0:\n if string[-1] == \";\":\n output.write(\" \" + string + \"\\n\")\n else:\n output.write(string + \"\\n\")\n output.close()\n f.close()", "response": "Indentes css that has not been indented and saves it to a new file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_newlines(f, output, char):\n line_count = get_line_count(f)\n f = open(f, 'r+')\n output = open(output, 'r+')\n for line in range(line_count):\n string = f.readline()\n string = re.sub(char, char + '\\n', string)\n output.write(string)", "response": "Adds line breaks after every occurance of a given character in a file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a space before a character.", "response": "def add_whitespace_before(char, input_file, output_file):\n \"\"\"Adds a space before a character if there's isn't one already.\n \n Args:\n char: string, character that needs a space before it.\n\n input_file: string, path to file to parse.\n\n output_file: string, path to destination file.\n \n Returns:\n None.\n \"\"\"\n line_count = get_line_count(input_file)\n input_file = open(input_file, 'r')\n output_file = open(output_file, 'r+')\n for line in range(line_count):\n string = input_file.readline()\n # If there's not already a space before the character, add one\n if re.search(r'[a-zA-Z0-9]' + char, string) != None:\n string = re.sub(char, ' ' + char, string)\n output_file.write(string)\n input_file.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreformats poorly written css.", "response": "def reformat_css(input_file, output_file):\n \"\"\"Reformats poorly written css. This function does not validate or fix errors in the code.\n It only gives code the proper indentation. \n\n Args:\n input_file: string, path to the input file.\n\n output_file: string, path to where the reformatted css should be saved. If the target file\n doesn't exist, a new file is created.\n\n Returns:\n None.\n \"\"\"\n # Number of lines in the file.\n line_count = get_line_count(input_file)\n\n # Open source and target files.\n f = open(input_file, 'r+')\n output = open(output_file, 'w')\n\n # Loop over every line in the file.\n for line in range(line_count):\n # Eliminate whitespace at the beginning and end of lines.\n string = f.readline().strip()\n # New lines after { \n string = re.sub('\\{', '{\\n', string)\n # New lines after ; \n string = re.sub('; ', ';', string)\n string = re.sub(';', ';\\n', string)\n # Eliminate whitespace before comments\n string = re.sub('} /*', '}/*', string)\n # New lines after } \n string = re.sub('\\}', '}\\n', string)\n # New lines at the end of comments\n string = re.sub('\\*/', '*/\\n', string)\n # Write to the output file.\n output.write(string)\n\n # Close the files.\n output.close()\n f.close()\n\n # Indent the css.\n indent_css(output_file, output_file)\n\n # Make sure there's a space before every {\n add_whitespace_before(\"{\", output_file, output_file)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_int(string):\n try:\n a = float(string)\n b = int(a)\n except ValueError:\n return False\n else:\n return a == b", "response": "Checks if a string is an integer."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef total_hours(input_files):\n hours = 0 \n # Look for singular and plural forms of the word\n # and allow typos.\n allow = set(['hours', 'hour', 'huors', 'huor'])\n for input_file in input_files:\n doc = open(input_file, 'r')\n for line in doc:\n line = line.rstrip()\n data = line.split(' ')\n if (len(data) == 2) and (is_numeric(data[0])) and (data[1].lower() in allow):\n hours += float(data[0])\n doc.close()\n return hours", "response": "Returns the total number of hours spent on a given project."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean_strings(iterable):\n retval = []\n for val in iterable:\n try:\n retval.append(val.strip())\n except(AttributeError):\n retval.append(val)\n return retval", "response": "Takes a list of strings and clear whitespace \n on each one."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert an Excel spreadsheet to an html table.", "response": "def excel_to_html(path, sheetname='Sheet1', css_classes='', \\\n caption='', details=[], row_headers=False, merge=False):\n \"\"\"\n Convert an excel spreadsheet to an html table.\n This function supports the conversion of merged \n cells. It can be used in code or run from the \n command-line. If passed the correct arguments\n it can generate fully accessible html.\n\n Args:\n path: string, path to the spreadsheet.\n\n sheetname: string, name of the sheet\n to convert. \n\n css_classes: string, space separated\n classnames to append to the table.\n\n caption: string, a short heading-like \n description of the table.\n\n details: list of strings, where the first\n item in the list is a string for the html \n summary element and the second item is\n a string for the details element. The \n summary should be very short, e.g. \"Help\",\n where as the details element should be a \n long description regarding the purpose or \n how to navigate the table.\n\n row_headers: boolean, defaults to False.\n Does the table have row headers? If set\n to True, the first element in each row\n will be a element \n instead of a element.\n \n merge: boolean, whether or not to \n combine cells that were merged in the \n spreadsheet.\n\n Returns:\n string, html table \n \"\"\"\n\n def get_data_on_merged_cells():\n \"\"\"\n Build a datastructure with data \n on merged cells.\n \"\"\"\n # Use this to build support for merged columns and rows???? \n merged_cells = xls.book.sheet_by_name(sheetname).merged_cells\n ds = {}\n for crange in merged_cells:\n rlo, rhi, clo, chi = crange\n for rowx in range(rlo, rhi):\n for colx in range(clo, chi):\n # Cell (rlo, clo) (the top left one) will carry the data and \n # formatting info. The remainder will be recorded as blank cells, \n # but a renderer will apply the formatting info for the top left \n # cell (e.g. border, pattern) to all cells in the range.\n #print(str(rlo) + ' ' + str(clo))\n #print(str(rowx) + ' ' + str(colx))\n parent_cell = (rlo,clo)\n child_cell = (rowx,colx)\n if not parent_cell in ds:\n # Return data structure is a dictionary with numeric tuples \n # as keys. Each tuple holds the x, y coordinates of the cell.\n # The dictionary holds two values:\n # 1. A list with two numbers which represent the x/y count \n # starting at 1 for the current cell.\n # 2. A set describing which direction the cells are merged.\n ds[parent_cell] = [[1,1], set([])]\n else:\n if parent_cell != child_cell and child_cell[0] == parent_cell[0]:\n ds[parent_cell][0][0] += 1\n ds[parent_cell][1].add('right')\n elif parent_cell != child_cell and child_cell[0] > parent_cell[0]:\n if child_cell[1] == parent_cell[1]:\n ds[parent_cell][0][1] += 1\n ds[parent_cell][1].add('down')\n else:\n raise RuntimeError('Something went wrong')\n return ds\n\n\n def mark_cells_going_right(cell, curr_cell, merged_cells):\n \"\"\"\n Add a \"colspan\" attribute and mark empty table \n columns for deletion if they are part of a \n merged cell going right.\n\n Args:\n cell: BeautifulSoup element tag object \n representation of the current cell.\n\n curr_cell: tuple, numeric representation \n of the current cell.\n\n merged_cells: dictionary of of data about \n merged cells.\n \"\"\"\n #if curr_cell in merged_cells and merged_cells[curr_cell][1] == set(['right']):\n try:\n xcount = merged_cells[curr_cell][0][0]\n if xcount > 1: # No colspans on 1\n cell['colspan'] = xcount\n col_count = xcount - 1 \n while col_count > 0:\n cell = cell.find_next_sibling()\n cell['class'] = 'delete'\n col_count -= 1\n except:\n pass\n\n def mark_cells_going_down(cell, curr_cell, merged_cells):\n \"\"\"\n Add a \"rowspan\" attribute and mark empty table \n columns for deletion if they are part of a \n merged cell going down.\n\n Args:\n cell: BeautifulSoup element tag object \n representation of the current cell.\n\n curr_cell: tuple, numeric representation \n of the current cell.\n\n merged_cells: dictionary of of data about \n merged cells.\n \"\"\"\n if curr_cell in merged_cells and merged_cells[curr_cell][1] == set(['down']):\n ycount = merged_cells[curr_cell][0][1]\n cell['rowspan'] = ycount\n row_count = ycount\n for child_row in cell.parent.find_next_siblings(limit=row_count - 1):\n i = 0\n for child in child_row.find_all('td'):\n if i == curr_cell[1]:\n child['class'] = 'delete'\n i += 1\n\n def mark_cells_going_down_and_right(cell, curr_cell, merged_cells):\n \"\"\"\n Add \"rowspan\" and \"colspan\" attributes and mark \n empty columns for deletion if they are part of a \n merged cell going down and to the right diagonally.\n\n Args:\n cell: BeautifulSoup element tag object \n representation of the current cell.\n\n curr_cell: tuple, numeric representation \n of the current cell.\n\n merged_cells: dictionary of of data about \n merged cells.\n \"\"\"\n if curr_cell in merged_cells and \\\n ('down' in merged_cells[curr_cell][1] and \\\n 'right' in merged_cells[curr_cell][1]):\n xcount = merged_cells[curr_cell][0][0]\n ycount = merged_cells[curr_cell][0][1]\n row_count = ycount\n col_count = xcount\n mark_cells_going_right(cell, curr_cell, merged_cells)\n \n flag = False\n for child_row in [cell.parent] + cell.parent.find_all_next('tr', limit=row_count - 1):\n i = 0\n for child in child_row.find_all('td'):\n if i == curr_cell[1]:\n mark_cells_going_right(child, curr_cell, merged_cells)\n if not flag:\n child['colspan'] = col_count\n child['rowspan'] = row_count\n flag = True\n else:\n child['class'] = 'delete'\n i += 1\n\n\n def is_empty_th(string):\n \"\"\"\n Detects if a table cell is left\n empty (is a merged cell).\n\n Args:\n string: string\n \"\"\"\n if string[:8] == 'Unnamed:':\n data = string.split(' ')\n if is_numeric(data[1]):\n return True\n return False\n\n\n def mark_header_cells(html):\n \"\"\"\n Mark header cells for deletion if they \n need to be merged. Also, add colspan\n and scope attributes.\n\n Args: \n html: string\n \"\"\"\n th = html.find_all('th')\n for header in th:\n txt = header.string\n if not is_empty_th(txt):\n header['scope'] = 'col'\n count = 1\n for sibling in header.find_next_siblings():\n if is_empty_th(sibling.string):\n count += 1\n sibling['class'] = 'delete'\n else:\n break\n if count > 1:\n header['colspan'] = count\n header['scope'] = 'colgroup'\n\n\n def create_caption(html, caption):\n \"\"\"\n Create a caption element for an \n accessible table and append it\n to the right part of the tree.\n \n Args:\n html: string\n\n caption: string\n \"\"\"\n ctag = html.new_tag('caption')\n ctag.insert(0, caption)\n html.table.insert(0, ctag)\n\n\n def create_summary_and_details(html, details):\n \"\"\"\n Create a summary and details element\n for an accessible table and insert \n it into the right part of the tree.\n\n Args:\n html: string\n\n details: string\n \"\"\"\n if len(details) != 2:\n msg = 'The \"details\" argument should be a list with two items. ' \\\n + 'The first item should be a string for the html summary ' \\\n + 'and the second should be a long description for the details ' \\\n + 'element. Both of those must be included and nothing else.'\n raise RuntimeError(msg)\n\n summary = details[0]\n details = details[1]\n\n if not caption:\n create_caption(html, caption)\n\n dtag = html.new_tag('details')\n stag = html.new_tag('summary')\n ptag = html.new_tag('p')\n stag.insert(0, summary)\n ptag.insert(0, details)\n dtag.insert(0, stag)\n dtag.append(ptag) \n html.table.caption.insert(1, dtag) \n\n\n def format_properly(html):\n \"\"\"\n Fix bad formatting from beautifulsoup.\n\n Args:\n html: string of html representing \n a table.\n \"\"\"\n return html.replace('\\n ', '').replace('\\n ', \\\n '').replace('\\n ', '').replace('\\n ', \\\n '').replace('\\n

', '

')\n\n\n def add_row_headers(html):\n \"\"\"\n Convert s to s if row_headers\n is set to True.\n\n Args:\n html: string, table.\n \"\"\"\n for row in html.tbody.find_all('tr'):\n spans_rows = 'rowspan' in row.td.attrs\n spans_columns = 'colspan' in row.td.attrs\n new_tag = html.new_tag('th')\n new_tag['scope'] = 'row'\n new_tag.string = row.td.string\n if spans_rows:\n new_tag['rowspan'] = row.td.attrs['rowspan']\n new_tag['scope'] = 'rowgroup'\n if spans_columns:\n new_tag['colspan'] = row.td.attrs['colspan']\n row.td.replace_with(new_tag)\n\n\n def beautify(html):\n \"\"\"\n Beautify the html from pandas.\n\n Args:\n html: table markup from pandas.\n \"\"\"\n table = html.find('table')\n first_tr = table.find('tr')\n del table['border']\n del first_tr['style']\n\n return format_properly(html.prettify(formatter='minimal'))\n\n\n def parse_html(html, caption, details):\n \"\"\"\n Use BeautifulSoup to correct the \n html for merged columns and rows.\n What could possibly go wrong?\n\n Args:\n html: string\n\n caption: string\n\n details: list of strings lenght of two\n\n Returns:\n string, modified html\n \"\"\"\n new_html = BeautifulSoup(html, 'html.parser')\n if merge:\n row_num = 1\n # e.g. {(4, 3): [1, 'right'], (2, 1): [1, 'down']}\n merged_cells = get_data_on_merged_cells()\n rows = new_html.find('table').find('tbody').find_all('tr')\n for row in rows:\n cell_num = 0 # Why are we off by 1? Maybe because we set index to False in to_html?\n cells = row.find_all('td')\n for cell in cells:\n #cell['class'] = str(row_num) + ' ' + str(cell_num) # DEBUG\n curr_cell = (row_num, cell_num)\n\n # Mark merged cells for deletion\n mark_cells_going_right(cell, curr_cell, merged_cells) \n mark_cells_going_down(cell, curr_cell, merged_cells)\n mark_cells_going_down_and_right(cell, curr_cell, merged_cells)\n \n cell_num += 1\n row_num += 1\n\n # Mark header cells for deletion\n mark_header_cells(new_html)\n\n # Delete all the renegade cells at once\n destroy = new_html.find_all(attrs={'class' : 'delete' })\n for item in destroy:\n item.extract()\n\n # Convert s to s if needed.\n if row_headers:\n add_row_headers(new_html)\n\n # Add caption if applicable\n if caption:\n create_caption(new_html, caption)\n\n # Add summary and details if possible\n if details:\n create_summary_and_details(new_html, details)\n \n return beautify(new_html)\n\n # Set options for pandas and load the excel file\n pd.options.display.max_colwidth = -1\n xls = pd.ExcelFile(path)\n\n # Parse the sheet you're interested in, results in a Dataframe\n df = xls.parse(sheetname)\n\n # Convert the dataframe to html\n panda_html = df.to_html(classes=css_classes, index=False, na_rep='')\n \n # Parse the panda html to merge cells and beautify the markup \n return parse_html(panda_html, caption, details)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the future value of the current value of the aracartor for a given number of years.", "response": "def future_value(present_value, annual_rate, periods_per_year, years):\n \"\"\"\n Calculates the future value of money invested at an anual interest rate,\n x times per year, for a given number of years.\n\n Args:\n present_value: int or float, the current value of the money (principal).\n\n annual_rate: float 0 to 1 e.g., .5 = 50%), the interest rate paid out.\n\n periods_per_year: int, the number of times money is invested per year.\n\n years: int, the number of years invested.\n\n Returns:\n Float, the future value of the money invested with compound interest.\n \"\"\"\n\n # The nominal interest rate per period (rate) is how much interest you earn during a\n # particular length of time, before accounting for compounding. This is typically\n # expressed as a percentage.\n rate_per_period = annual_rate / float(periods_per_year)\n\n # How many periods in the future the calculation is for.\n periods = periods_per_year * years\n\n return present_value * (1 + rate_per_period) ** periods"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef triangle_area(point1, point2, point3):\n\n \"\"\"Lengths of the three sides of the triangle\"\"\"\n a = point_distance(point1, point2)\n b = point_distance(point1, point3)\n c = point_distance(point2, point3)\n\n \"\"\"Where s is the semiperimeter\"\"\"\n s = (a + b + c) / 2.0\n\n \"\"\"Return the area of the triangle (using Heron's formula)\"\"\"\n return math.sqrt(s * (s - a) * (s - b) * (s - c))", "response": "Calculates the area of a triangle."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef regular_polygon_area(number_of_sides, length_of_sides):\n return (0.25 * number_of_sides * length_of_sides ** 2) / math.tan(\n math.pi / number_of_sides\n )", "response": "Calculates the area of a regular polygon"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the median of a list of integers or floating point numbers.", "response": "def median(data):\n \"\"\"\n Calculates the median of a list of integers or floating point numbers.\n\n Args:\n data: A list of integers or floating point numbers\n\n Returns:\n Sorts the list numerically and returns the middle number if the list has an odd number\n of items. If the list contains an even number of items the mean of the two middle numbers\n is returned.\n \"\"\"\n ordered = sorted(data)\n length = len(ordered)\n if length % 2 == 0:\n return (\n ordered[math.floor(length / 2) - 1] + ordered[math.floor(length / 2)]\n ) / 2.0\n\n elif length % 2 != 0:\n return ordered[math.floor(length / 2)]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the average or mean of a list of numbers or a Decimal object.", "response": "def average(numbers, numtype='float'):\n \"\"\"\n Calculates the average or mean of a list of numbers\n\n Args:\n numbers: a list of integers or floating point numbers.\n\n numtype: string, 'decimal' or 'float'; the type of number to return.\n\n Returns:\n The average (mean) of the numbers as a floating point number\n or a Decimal object.\n\n Requires:\n The math module\n \"\"\"\n if type == 'decimal':\n return Decimal(sum(numbers)) / len(numbers)\n else:\n return float(sum(numbers)) / len(numbers)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef variance(numbers, type='population'):\n mean = average(numbers)\n variance = 0\n for number in numbers:\n variance += (mean - number) ** 2\n\n if type == 'population':\n return variance / len(numbers)\n else:\n return variance / (len(numbers) - 1)", "response": "Calculates the population or sample variance of a list of numbers."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_percentage(a, b, i=False, r=False):\n # Round to the second decimal\n if i is False and r is True:\n percentage = round(100.0 * (float(a) / b), 2)\n\n # Round to the nearest whole number\n elif (i is True and r is True) or (i is True and r is False):\n percentage = int(round(100 * (float(a) / b)))\n\n # A rounded number and an integer were requested\n if r is False:\n warnings.warn(\n \"If integer is set to True and Round is set to False, you will still get a rounded number if you pass floating point numbers as arguments.\"\n )\n\n # A precise unrounded decimal\n else:\n percentage = 100.0 * (float(a) / b)\n\n return percentage", "response": "Returns the percentage of one number over another."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the net take - home pay including employer retirement savings", "response": "def take_home_pay(gross_pay, employer_match, taxes_and_fees, numtype='float'):\n \"\"\"\n Calculate net take-home pay including employer retirement savings match\n using the formula laid out by Mr. Money Mustache:\n http://www.mrmoneymustache.com/2015/01/26/calculating-net-worth/\n\n Args:\n gross_pay: float or int, gross monthly pay.\n\n employer_match: float or int, the 401(k) match from your employer.\n\n taxes_and_fees: list, taxes and fees that are deducted from your paycheck.\n\n numtype: string, 'decimal' or 'float'; the type of number to return.\n\n Returns:\n your monthly take-home pay.\n \"\"\"\n if numtype == 'decimal':\n return (Decimal(gross_pay) + Decimal(employer_match)) - Decimal(\n sum(taxes_and_fees)\n )\n else:\n return (float(gross_pay) + float(employer_match)) - sum(taxes_and_fees)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef savings_rate(take_home_pay, spending, numtype='float'):\n\n if numtype == 'decimal':\n try:\n return (\n (Decimal(take_home_pay) - Decimal(spending)) / (Decimal(take_home_pay))\n ) * Decimal(100.0)\n # Leave InvalidOperation for backwards compatibility\n except (InvalidOperation, DivisionByZero):\n return Decimal(0.0)\n else:\n try:\n return (\n (float(take_home_pay) - float(spending)) / (float(take_home_pay))\n ) * 100.0\n except (ZeroDivisionError):\n return 0.0", "response": "Calculate monthly savings rate for a given monthly take - home pay and spending."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_variable(relpath, keyword='__version__'):\n for line in open(os.path.join(os.path.dirname(__file__), relpath), encoding='cp437'):\n if keyword in line:\n if '\"' in line:\n return line.split('\"')[1]\n elif \"'\" in line:\n return line.split(\"'\")[1]", "response": "Read the variable from a python file without importing it \n from gist. github. com."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a datetime string to a unix timestamp", "response": "def datetime_str_to_timestamp(datetime_str):\n '''\n '2018-01-01 00:00:00' (str) --> 1514736000\n\n :param str datetime_str: datetime string\n :return: unix timestamp (int) or None\n :rtype: int or None\n '''\n\n try:\n dtf = DTFormat()\n struct_time = time.strptime(datetime_str, dtf.datetime_format)\n return time.mktime(struct_time)\n except:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets datetime string from datetime object", "response": "def get_datetime_string(datetime_obj):\n '''\n Get datetime string from datetime object\n\n :param datetime datetime_obj: datetime object\n :return: datetime string\n :rtype: str\n '''\n\n if isinstance(datetime_obj, datetime):\n dft = DTFormat()\n return datetime_obj.strftime(dft.datetime_format)\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef timestamp_to_datetime(timestamp):\n '''\n 1514736000 --> datetime object\n\n :param int timestamp: unix timestamp (int)\n :return: datetime object or None\n :rtype: datetime or None\n '''\n\n if isinstance(timestamp, (int, float, str)):\n try:\n timestamp = float(timestamp)\n if timestamp.is_integer():\n timestamp = int(timestamp)\n except:\n return None\n\n temp = str(timestamp).split('.')[0]\n if len(temp) == 13:\n timestamp = timestamp / 1000.0\n\n if len(temp) < 10:\n return None\n\n else:\n return None\n\n return datetime.fromtimestamp(timestamp)", "response": "converts unix timestamp to datetime object"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef attrdict(prev, attr_names):\n if isinstance(attr_names, dict):\n for obj in prev:\n attr_values = dict()\n for name in attr_names.keys():\n if hasattr(obj, name):\n attr_values[name] = getattr(obj, name)\n else:\n attr_values[name] = attr_names[name]\n yield attr_values\n else:\n for obj in prev:\n attr_values = dict()\n for name in attr_names:\n if hasattr(obj, name):\n attr_values[name] = getattr(obj, name)\n yield attr_values", "response": "This function returns a generator that yields the attribute values of the object in the object that is in the pipe."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flatten(prev, depth=sys.maxsize):\n def inner_flatten(iterable, curr_level, max_levels):\n for i in iterable:\n if hasattr(i, '__iter__') and curr_level < max_levels:\n for j in inner_flatten(i, curr_level + 1, max_levels):\n yield j\n else:\n yield i\n\n for d in prev:\n if hasattr(d, '__iter__') and depth > 0:\n for inner_d in inner_flatten(d, 1, depth):\n yield inner_d\n else:\n yield d", "response": "This function flattens the given pipe into a single nested item."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvaluing pipe extract value from previous pipe.", "response": "def values(prev, *keys, **kw):\n \"\"\"values pipe extract value from previous pipe.\n\n If previous pipe send a dictionary to values pipe, keys should contains\n the key of dictionary which you want to get. If previous pipe send list or\n tuple,\n\n :param prev: The previous iterator of pipe.\n :type prev: Pipe\n :returns: generator\n \"\"\"\n d = next(prev)\n if isinstance(d, dict):\n yield [d[k] for k in keys if k in d]\n for d in prev:\n yield [d[k] for k in keys if k in d]\n else:\n yield [d[i] for i in keys if 0 <= i < len(d)]\n for d in prev:\n yield [d[i] for i in keys if 0 <= i < len(d)]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\npack pipe takes n elements from previous generator and yield one list to next. :param prev: The previous iterator of pipe. :type prev: Pipe :param rest: Set True to allow to output the rest part of last elements. :type prev: boolean :param padding: Specify the padding element for the rest part of last elements. :type prev: boolean :returns: generator :Example: >>> result([1,2,3,4,5,6,7] | pack(3)) [[1, 2, 3], [4, 5, 6]] >>> result([1,2,3,4,5,6,7] | pack(3, rest=True)) [[1, 2, 3], [4, 5, 6], [7,]] >>> result([1,2,3,4,5,6,7] | pack(3, padding=None)) [[1, 2, 3], [4, 5, 6], [7, None, None]]", "response": "def pack(prev, n, rest=False, **kw):\n \"\"\"pack pipe takes n elements from previous generator and yield one\n list to next.\n\n :param prev: The previous iterator of pipe.\n :type prev: Pipe\n :param rest: Set True to allow to output the rest part of last elements.\n :type prev: boolean\n :param padding: Specify the padding element for the rest part of last elements.\n :type prev: boolean\n :returns: generator\n\n :Example:\n >>> result([1,2,3,4,5,6,7] | pack(3))\n [[1, 2, 3], [4, 5, 6]]\n\n >>> result([1,2,3,4,5,6,7] | pack(3, rest=True))\n [[1, 2, 3], [4, 5, 6], [7,]]\n\n >>> result([1,2,3,4,5,6,7] | pack(3, padding=None))\n [[1, 2, 3], [4, 5, 6], [7, None, None]]\n \"\"\"\n\n if 'padding' in kw:\n use_padding = True\n padding = kw['padding']\n else:\n use_padding = False\n padding = None\n\n items = []\n for i, data in enumerate(prev, 1):\n items.append(data)\n if (i % n) == 0:\n yield items\n items = []\n if len(items) != 0 and rest:\n if use_padding:\n items.extend([padding, ] * (n - (i % n)))\n yield items"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef grep(prev, pattern, *args, **kw):\n inv = False if 'inv' not in kw else kw.pop('inv')\n pattern_obj = re.compile(pattern, *args, **kw)\n\n for data in prev:\n if bool(inv) ^ bool(pattern_obj.match(data)):\n yield data", "response": "The grep function for the given pattern in the given pipe."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef match(prev, pattern, *args, **kw):\n to = 'to' in kw and kw.pop('to')\n pattern_obj = re.compile(pattern, *args, **kw)\n\n if to is dict:\n for data in prev:\n match = pattern_obj.match(data)\n if match is not None:\n yield match.groupdict()\n elif to is tuple:\n for data in prev:\n match = pattern_obj.match(data)\n if match is not None:\n yield match.groups()\n elif to is list:\n for data in prev:\n match = pattern_obj.match(data)\n if match is not None:\n yield list(match.groups())\n else:\n for data in prev:\n match = pattern_obj.match(data)\n if match is not None:\n yield match", "response": "The pattern is a regular expression which can be used to filter the data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef resplit(prev, pattern, *args, **kw):\n maxsplit = 0 if 'maxsplit' not in kw else kw.pop('maxsplit')\n pattern_obj = re.compile(pattern, *args, **kw)\n for s in prev:\n yield pattern_obj.split(s, maxsplit=maxsplit)", "response": "The resplit pipe split previous string by regular expression."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sub(prev, pattern, repl, *args, **kw):\n count = 0 if 'count' not in kw else kw.pop('count')\n pattern_obj = re.compile(pattern, *args, **kw)\n for s in prev:\n yield pattern_obj.sub(repl, s, count=count)", "response": "A wrapper around re. sub method."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nyields all data that matches pattern in the previous generator of natalatalatalels", "response": "def wildcard(prev, pattern, *args, **kw):\n \"\"\"wildcard pipe greps data passed from previous generator\n according to given regular expression.\n\n :param prev: The previous iterator of pipe.\n :type prev: Pipe\n :param pattern: The wildcard string which used to filter data.\n :type pattern: str|unicode|re pattern object\n :param inv: If true, invert the match condition.\n :type inv: boolean\n :returns: generator\n \"\"\"\n import fnmatch\n\n inv = 'inv' in kw and kw.pop('inv')\n pattern_obj = re.compile(fnmatch.translate(pattern), *args, **kw)\n\n if not inv:\n for data in prev:\n if pattern_obj.match(data):\n yield data\n else:\n for data in prev:\n if not pattern_obj.match(data):\n yield data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stdout(prev, endl='\\n', thru=False):\n for i in prev:\n sys.stdout.write(str(i) + endl)\n if thru:\n yield i", "response": "This pipe writes data from previous iterator to stdout."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes a shell command.", "response": "def sh(prev, *args, **kw):\n \"\"\"sh pipe execute shell command specified by args. If previous pipe exists,\n read data from it and write it to stdin of shell process. The stdout of\n shell process will be passed to next pipe object line by line.\n\n A optional keyword argument 'trim' can pass a function into sh pipe. It is\n used to trim the output from shell process. The default trim function is\n str.rstrip. Therefore, any space characters in tail of\n shell process output line will be removed.\n\n For example:\n\n py_files = result(sh('ls') | strip | wildcard('*.py'))\n\n :param prev: The previous iterator of pipe.\n :type prev: Pipe\n :param args: The command line arguments. It will be joined by space character.\n :type args: list of string.\n :param kw: arguments for subprocess.Popen.\n :type kw: dictionary of options.\n :returns: generator\n \"\"\"\n endl = '\\n' if 'endl' not in kw else kw.pop('endl')\n trim = None if 'trim' not in kw else kw.pop('trim')\n if trim is None:\n trim = bytes.rstrip if is_py3 else str.rstrip\n\n cmdline = ' '.join(args)\n if not cmdline:\n if prev is not None:\n for i in prev:\n yield i\n else:\n while True:\n yield None\n\n process = subprocess.Popen(cmdline, shell=True,\n stdin=subprocess.PIPE, stdout=subprocess.PIPE,\n **kw)\n if prev is not None:\n stdin_buffer = StringIO()\n for i in prev:\n stdin_buffer.write(i)\n if endl:\n stdin_buffer.write(endl)\n if is_py3:\n process.stdin.write(stdin_buffer.getvalue().encode('utf-8'))\n else:\n process.stdin.write(stdin_buffer.getvalue())\n process.stdin.flush()\n process.stdin.close()\n stdin_buffer.close()\n\n for line in process.stdout:\n yield trim(line)\n\n process.wait()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef join(prev, sep, *args, **kw):\n '''alias of str.join'''\n yield sep.join(prev, *args, **kw)", "response": "alias of str. join"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\naliases of string. Template. substitute", "response": "def substitute(prev, *args, **kw):\n '''alias of string.Template.substitute'''\n template_obj = string.Template(*args, **kw)\n for data in prev:\n yield template_obj.substitute(data)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\naliases of string. Template. safe_substitute", "response": "def safe_substitute(prev, *args, **kw):\n '''alias of string.Template.safe_substitute'''\n template_obj = string.Template(*args, **kw)\n for data in prev:\n yield template_obj.safe_substitute(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_str(prev, encoding=None):\n first = next(prev)\n if isinstance(first, str):\n if encoding is None:\n yield first\n for s in prev:\n yield s\n else:\n yield first.encode(encoding)\n for s in prev:\n yield s.encode(encoding)\n else:\n if encoding is None:\n encoding = sys.stdout.encoding or 'utf-8'\n yield first.decode(encoding)\n for s in prev:\n yield s.decode(encoding)", "response": "Convert data from previous pipe with specified encoding."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef register_default_types():\n register_type(type, pipe.map)\n register_type(types.FunctionType, pipe.map)\n register_type(types.MethodType, pipe.map)\n register_type(tuple, seq)\n register_type(list, seq)\n register_type(types.GeneratorType, seq)\n register_type(string_type, sh)\n register_type(unicode_type, sh)\n register_type(file_type, fileobj)\n\n if is_py3:\n register_type(range, seq)\n register_type(map, seq)", "response": "Regiser all default type - to - pipe convertors."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert Paginator instance to dict", "response": "def get_dict(self):\n '''\n Convert Paginator instance to dict\n\n :return: Paging data\n :rtype: dict\n '''\n\n return dict(\n current_page=self.current_page,\n total_page_count=self.total_page_count,\n items=self.items,\n total_item_count=self.total_item_count,\n page_size=self.page_size\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write_log(log_directory, log_filename, header, logline, debug,\n require_latest, latest_directory, latest_filename):\n \"\"\"This function logs a line of data to both a 'log' file, and a 'latest'\n file The 'latest' file is optional, and is sent to this function as a\n boolean value via the variable 'require_latest'.\n So the 2 log directories and filenames are:\n a. (REQUIRED): log_directory + log_filename\n b. (OPTIONAL): latest_directory + latest_filename\n\n The 'latest' directory and filename is provided so as to have a consistent\n file of the latest events This is usually the latest day of events.\n The way this function works with the 'latest' log_dir is as follows:\n a. It checks for the existance of log_directory + log_filename\n b. If (a) doesn't exist, then any 'latest' file is removed and a new one created\n c. If (a) already exists, logs are written to any existing 'latest' file\n If one doesn't exist, it will be created\n\n For both the 'log' and 'latest' files, a header line will be written if a new\n file is created Please note that a header must start with the '#' symbol, so\n the Ardexa agent can interpret this line as a header , and will not send it to\n the cloud\n \"\"\"\n create_new_file = False\n\n # Make sure the logging directory exists. The following will create all the necessary subdirs,\n # if the subdirs exist in part or in full\n if not os.path.exists(log_directory):\n os.makedirs(log_directory)\n full_path_log = os.path.join(log_directory, log_filename)\n if debug > 1:\n print(\"Full path of log directory: \", full_path_log)\n # If the file doesn't exist, annotate that a new 'latest' file is to be created\n # and that a header is to be created\n if not os.path.isfile(full_path_log):\n if debug > 1:\n print(\"Log file doesn't exist: \", full_path_log)\n create_new_file = True\n\n # Repeat for the 'latest', if it doesn't exist\n if require_latest:\n if not os.path.exists(latest_directory):\n os.makedirs(latest_directory)\n full_path_latest = os.path.join(latest_directory, latest_filename)\n if debug > 1:\n print(\"Full path of latest directory: \", full_path_latest)\n # If the 'create_new_file' tag is set AND the file exists, then remove it\n if create_new_file and os.path.isfile(full_path_latest):\n # then remove the file\n os.remove(full_path_latest)\n\n # Now create both (or open both) and write to them\n if debug > 1:\n print(\"##########################################\")\n print(\"Writing the line to\", full_path_latest)\n print(logline)\n print(\"##########################################\")\n\n # Write the logline to the log file\n output_file = open(full_path_log, \"a\")\n if create_new_file:\n output_file.write(header)\n output_file.write(logline)\n output_file.close()\n\n # And write it to the 'latest' if required\n if require_latest:\n write_latest = open(full_path_latest, \"a\")\n if create_new_file:\n write_latest.write(header)\n write_latest.write(logline)\n write_latest.close()", "response": "This function writes a line of data to a log file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_pidfile(pidfile, debug):\n # Check PID exists and see if the PID is running\n if os.path.isfile(pidfile):\n pidfile_handle = open(pidfile, 'r')\n # try and read the PID file. If no luck, remove it\n try:\n pid = int(pidfile_handle.read())\n pidfile_handle.close()\n if check_pid(pid, debug):\n return True\n except:\n pass\n\n # PID is not active, remove the PID file\n os.unlink(pidfile)\n\n # Create a PID file, to ensure this is script is only run once (at a time)\n pid = str(os.getpid())\n open(pidfile, 'w').write(pid)\n return False", "response": "Check that a process is not running more than once using PIDFILE"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_pid(pid, debug):\n try:\n # A Kill of 0 is to check if the PID is active. It won't kill the process\n os.kill(pid, 0)\n if debug > 1:\n print(\"Script has a PIDFILE where the process is still running\")\n return True\n except OSError:\n if debug > 1:\n print(\"Script does not appear to be running\")\n return False", "response": "This function will check whether a process is currently running."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert_words_to_uint(high_word, low_word):\n try:\n low_num = int(low_word)\n # low_word might arrive as a signed number. Convert to unsigned\n if low_num < 0:\n low_num = abs(low_num) + 2**15\n number = (int(high_word) << 16) | low_num\n return number, True\n except:\n return 0, False", "response": "Convert two words to a floating point number"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts two words to a floating point number", "response": "def convert_words_to_float(high_word, low_word):\n \"\"\"Convert two words to a floating point\"\"\"\n number, retval = convert_words_to_uint(high_word, low_word)\n if not retval:\n return 0.0, False\n\n try:\n packed_float = struct.pack('>l', number)\n return struct.unpack('>f', packed_float)[0], True\n except:\n return 0.0, False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disown(debug):\n # Get the current PID\n pid = os.getpid()\n cgroup_file = \"/proc/\" + str(pid) + \"/cgroup\"\n try:\n infile = open(cgroup_file, \"r\")\n except IOError:\n print(\"Could not open cgroup file: \", cgroup_file)\n return False\n\n # Read each line\n for line in infile:\n # Check if the line contains \"ardexa.service\"\n if line.find(\"ardexa.service\") == -1:\n continue\n\n # if the lines contains \"name=\", replace it with nothing\n line = line.replace(\"name=\", \"\")\n # Split the line by commas\n items_list = line.split(':')\n accounts = items_list[1]\n dir_str = accounts + \"/ardexa.disown\"\n # If accounts is empty, continue\n if not accounts:\n continue\n\n # Create the dir and all subdirs\n full_dir = \"/sys/fs/cgroup/\" + dir_str\n if not os.path.exists(full_dir):\n os.makedirs(full_dir)\n if debug >= 1:\n print(\"Making directory: \", full_dir)\n else:\n if debug >= 1:\n print(\"Directory already exists: \", full_dir)\n\n # Add the PID to the file\n full_path = full_dir + \"/cgroup.procs\"\n prog_list = [\"echo\", str(pid), \">\", full_path]\n run_program(prog_list, debug, True)\n\n # If this item contains a comma, then separate it, and reverse\n # some OSes will need cpuacct,cpu reversed to actually work\n if accounts.find(\",\") != -1:\n acct_list = accounts.split(',')\n accounts = acct_list[1] + \",\" + acct_list[0]\n dir_str = accounts + \"/ardexa.disown\"\n # Create the dir and all subdirs. But it may not work. So use a TRY\n full_dir = \"/sys/fs/cgroup/\" + dir_str\n try:\n if not os.path.exists(full_dir):\n os.makedirs(full_dir)\n except:\n continue\n\n # Add the PID to the file\n full_path = full_dir + \"/cgroup.procs\"\n prog_list = [\"echo\", str(pid), \">\", full_path]\n run_program(prog_list, debug, True)\n\n infile.close()\n\n # For debug purposes only\n if debug >= 1:\n prog_list = [\"cat\", cgroup_file]\n run_program(prog_list, debug, False)\n\n # If there are any \"ardexa.service\" in the proc file. If so, exit with error\n prog_list = [\"grep\", \"-q\", \"ardexa.service\", cgroup_file]\n if run_program(prog_list, debug, False):\n # There are entries still left in the file\n return False\n\n return True", "response": "This function will disown the Ardexa service and all of its sub - directories and the Ardexa service."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run_program(prog_list, debug, shell):\n try:\n if not shell:\n process = Popen(prog_list, stdout=PIPE, stderr=PIPE)\n stdout, stderr = process.communicate()\n retcode = process.returncode\n if debug >= 1:\n print(\"Program : \", \" \".join(prog_list))\n print(\"Return Code: \", retcode)\n print(\"Stdout: \", stdout)\n print(\"Stderr: \", stderr)\n return bool(retcode)\n else:\n command = \" \".join(prog_list)\n os.system(command)\n return True\n except:\n return False", "response": "Run a program and check return code"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_address_list(addrs):\n for addr in addrs.split(','):\n elem = addr.split('-')\n if len(elem) == 1: # a number\n yield int(elem[0])\n elif len(elem) == 2: # a range inclusive\n start, end = list(map(int, elem))\n for i in range(start, end+1):\n yield i\n else: # more than one hyphen\n raise ValueError('format error in %s' % addr)", "response": "Yield each integer from a complex range string like 1 - 9 12 15 - 20 23"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndoes url - encode resource ids", "response": "def _encode_ids(*args):\n \"\"\"\n Do url-encode resource ids\n \"\"\"\n\n ids = []\n for v in args:\n if isinstance(v, basestring):\n qv = v.encode('utf-8') if isinstance(v, unicode) else v\n ids.append(urllib.quote(qv))\n else:\n qv = str(v)\n ids.append(urllib.quote(qv))\n\n return ';'.join(ids)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef random_string(length):\n '''\n Generate random string with parameter length.\n Example:\n\n >>> from eggit.egg_string import random_string\n >>> random_string(8)\n 'q4f2eaT4'\n >>>\n\n '''\n\n str_list = [random.choice(string.digits + string.ascii_letters) for i in range(length)]\n return ''.join(str_list)", "response": "Generate random string with parameter length."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_item_creator(item_type):\n if item_type not in Pipe.pipe_item_types:\n for registered_type in Pipe.pipe_item_types:\n if issubclass(item_type, registered_type):\n return Pipe.pipe_item_types[registered_type]\n return None\n else:\n return Pipe.pipe_item_types[item_type]", "response": "Get item creator function according registered item type."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef clone(self):\n new_object = copy.copy(self)\n if new_object.next:\n new_object.next = new_object.next.clone()\n return new_object", "response": "Self - cloning. All its next Pipe objects are cloned too."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nappending next object to the end of the pipe.", "response": "def append(self, next):\n \"\"\"Append next object to pipe tail.\n\n :param next: The Pipe object to be appended to tail.\n :type next: Pipe object.\n \"\"\"\n next.chained = True\n if self.next:\n self.next.append(next)\n else:\n self.next = next"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef iter(self, prev=None):\n\n if self.next:\n generator = self.next.iter(self.func(prev, *self.args, **self.kw))\n else:\n generator = self.func(prev, *self.args, **self.kw)\n return generator", "response": "Return an iterator object which yields the data from the next pipe."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reduce(func):\n def wrapper(prev, *argv, **kw):\n accum_value = None if 'init' not in kw else kw.pop('init')\n if prev is None:\n raise TypeError('A reducer must have input.')\n for i in prev:\n accum_value = func(accum_value, i, *argv, **kw)\n yield accum_value\n return Pipe(wrapper)", "response": "Wrap a function to reduce the data from a list of entries."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stopper(func):\n def wrapper(prev, *argv, **kw):\n if prev is None:\n raise TypeError('A stopper must have input.')\n for i in prev:\n if func(i, *argv, **kw):\n break\n yield i\n return Pipe(wrapper)", "response": "Wrap a conditoinal function to a pipe object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a dictionary of network name to active status bools.", "response": "def _list_networks():\n \"\"\"Return a dictionary of network name to active status bools.\n\n Sample virsh net-list output::\n\n Name State Autostart\n -----------------------------------------\n default active yes\n juju-test inactive no\n foobar inactive no\n\n Parsing the above would return::\n {\"default\": True, \"juju-test\": False, \"foobar\": False}\n\n See: http://goo.gl/kXwfC\n \"\"\"\n output = core.run(\"virsh net-list --all\")\n networks = {}\n\n # Take the header off and normalize whitespace.\n net_lines = [n.strip() for n in output.splitlines()[2:]]\n for line in net_lines:\n if not line:\n continue\n name, state, auto = line.split()\n networks[name] = state == \"active\"\n return networks"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef flush(self, line):\n # TODO -- maybe use echo?\n sys.stdout.write(line)\n sys.stdout.flush()", "response": "flush the line to stdout"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun the passed in arguments and returns an iterator on the output of the running command", "response": "def execute(self, arg_str='', **kwargs):\n \"\"\"runs the passed in arguments and returns an iterator on the output of\n running command\"\"\"\n cmd = \"{} {} {}\".format(self.cmd_prefix, self.script, arg_str)\n expected_ret_code = kwargs.pop('code', 0)\n\n # any kwargs with all capital letters should be considered environment\n # variables\n environ = self.environ\n for k in list(kwargs.keys()):\n if k.isupper():\n environ[k] = kwargs.pop(k)\n\n # we will allow overriding of these values\n kwargs.setdefault(\"stderr\", subprocess.STDOUT)\n\n # we will not allow these to be overridden via kwargs\n kwargs[\"shell\"] = True\n kwargs[\"stdout\"] = subprocess.PIPE\n kwargs[\"cwd\"] = self.cwd\n kwargs[\"env\"] = environ\n\n process = None\n self.buf = deque(maxlen=self.bufsize)\n\n try:\n process = subprocess.Popen(\n cmd,\n **kwargs\n )\n\n # another round of links\n # http://stackoverflow.com/a/17413045/5006 (what I used)\n # http://stackoverflow.com/questions/2715847/\n for line in iter(process.stdout.readline, b\"\"):\n line = line.decode(self.encoding)\n self.buf.append(line.rstrip())\n yield line\n\n process.wait()\n if process.returncode != expected_ret_code:\n if process.returncode > 0:\n raise RuntimeError(\"{} returned {} with output: {}\".format(\n cmd,\n process.returncode,\n self.output\n ))\n\n except subprocess.CalledProcessError as e:\n if e.returncode != expected_ret_code:\n raise RuntimeError(\"{} returned {} with output: {}\".format(\n cmd,\n e.returncode,\n self.output\n ))\n\n finally:\n if process:\n process.stdout.close()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds a basic 035 subfield with basic information from the OAI - PMH request.", "response": "def get_request_subfields(root):\n \"\"\"Build a basic 035 subfield with basic information from the OAI-PMH request.\n\n :param root: ElementTree root node\n\n :return: list of subfield tuples [(..),(..)]\n \"\"\"\n request = root.find('request')\n responsedate = root.find('responseDate')\n\n subs = [(\"9\", request.text),\n (\"h\", responsedate.text),\n (\"m\", request.attrib[\"metadataPrefix\"])]\n return subs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef strip_xml_namespace(root):\n try:\n root.tag = root.tag.split('}')[1]\n except IndexError:\n pass\n\n for element in root.getchildren():\n strip_xml_namespace(element)", "response": "This function removes out namespace data from an ElementTree."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef element_tree_collection_to_records(tree):\n from .bibrecord import create_record\n\n records = []\n collection = tree.getroot()\n for record_element in collection.getchildren():\n marcxml = ET.tostring(record_element, encoding=\"utf-8\")\n record, status, errors = create_record(marcxml)\n if errors:\n print(str(status))\n records.append(record)\n return records", "response": "Take an ElementTree and converts the nodes into BibRecord records."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntaking an ElementTree object and converts the nodes into BibRecord records.", "response": "def element_tree_oai_records(tree, header_subs=None):\n \"\"\"Take an ElementTree and converts the nodes into BibRecord records.\n\n This expects a clean OAI response with the tree root as ListRecords\n or GetRecord and record as the subtag like so:\n \n \n
\n \n
\n \n \n \n \n \n \n ... \n \n\n :param tree: ElementTree object corresponding to GetRecord node from\n OAI request\n :param header_subs: OAI header subfields, if any\n\n :yield: (record, is_deleted) A tuple, with first a BibRecord found and\n second a boolean value saying if this is a deleted record or not.\n \"\"\"\n from .bibrecord import record_add_field, create_record\n\n if not header_subs:\n header_subs = []\n # Make it a tuple, this information should not be changed\n header_subs = tuple(header_subs)\n\n oai_records = tree.getroot()\n for record_element in oai_records.getchildren():\n header = record_element.find('header')\n\n # Add to OAI subfield\n datestamp = header.find('datestamp')\n identifier = header.find('identifier')\n identifier = identifier.text\n\n # The record's subfield is based on header information\n subs = list(header_subs)\n subs.append((\"a\", identifier))\n subs.append((\"d\", datestamp.text))\n\n if \"status\" in header.attrib and header.attrib[\"status\"] == \"deleted\":\n # Record was deleted - create delete record\n deleted_record = {}\n record_add_field(deleted_record, \"037\", subfields=subs)\n yield deleted_record, True\n else:\n marc_root = record_element.find('metadata').find('record')\n marcxml = ET.tostring(marc_root, encoding=\"utf-8\")\n record, status, errors = create_record(marcxml)\n if status == 1:\n # Add OAI request information\n record_add_field(record, \"035\", subfields=subs)\n yield record, False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstarting a server instance.", "response": "def run(app=None,\n server='wsgiref',\n host='127.0.0.1',\n port=8080,\n interval=1,\n reloader=False,\n quiet=False,\n plugins=None,\n debug=None, **kargs):\n \"\"\" Start a server instance. This method blocks until the server terminates.\n\n :param app: WSGI application or target string supported by\n :func:`load_app`. (default: :func:`default_app`)\n :param server: Server adapter to use. See :data:`server_names` keys\n for valid names or pass a :class:`ServerAdapter` subclass.\n (default: `wsgiref`)\n :param host: Server address to bind to. Pass ``0.0.0.0`` to listens on\n all interfaces including the external one. (default: 127.0.0.1)\n :param port: Server port to bind to. Values below 1024 require root\n privileges. (default: 8080)\n :param reloader: Start auto-reloading server? (default: False)\n :param interval: Auto-reloader interval in seconds (default: 1)\n :param quiet: Suppress output to stdout and stderr? (default: False)\n :param options: Options passed to the server adapter.\n \"\"\"\n if NORUN: return\n if reloader and not os.environ.get('BOTTLE_CHILD'):\n import subprocess\n lockfile = None\n try:\n fd, lockfile = tempfile.mkstemp(prefix='bottle.', suffix='.lock')\n os.close(fd) # We only need this file to exist. We never write to it\n while os.path.exists(lockfile):\n args = [sys.executable] + sys.argv\n environ = os.environ.copy()\n environ['BOTTLE_CHILD'] = 'true'\n environ['BOTTLE_LOCKFILE'] = lockfile\n p = subprocess.Popen(args, env=environ)\n while p.poll() is None: # Busy wait...\n os.utime(lockfile, None) # I am alive!\n time.sleep(interval)\n if p.poll() != 3:\n if os.path.exists(lockfile): os.unlink(lockfile)\n sys.exit(p.poll())\n except KeyboardInterrupt:\n pass\n finally:\n if os.path.exists(lockfile):\n os.unlink(lockfile)\n return\n\n try:\n if debug is not None: _debug(debug)\n app = app or default_app()\n if isinstance(app, basestring):\n app = load_app(app)\n if not callable(app):\n raise ValueError(\"Application is not callable: %r\" % app)\n\n for plugin in plugins or []:\n if isinstance(plugin, basestring):\n plugin = load(plugin)\n app.install(plugin)\n\n if server in server_names:\n server = server_names.get(server)\n if isinstance(server, basestring):\n server = load(server)\n if isinstance(server, type):\n server = server(host=host, port=port, **kargs)\n if not isinstance(server, ServerAdapter):\n raise ValueError(\"Unknown or unsupported server: %r\" % server)\n\n server.quiet = server.quiet or quiet\n if not server.quiet:\n _stderr(\"Bottle v%s server starting up (using %s)...\\n\" %\n (__version__, repr(server)))\n _stderr(\"Listening on http://%s:%d/\\n\" %\n (server.host, server.port))\n _stderr(\"Hit Ctrl-C to quit.\\n\\n\")\n\n if reloader:\n lockfile = os.environ.get('BOTTLE_LOCKFILE')\n bgcheck = FileCheckerThread(lockfile, interval)\n with bgcheck:\n server.run(app)\n if bgcheck.status == 'reload':\n sys.exit(3)\n else:\n server.run(app)\n except KeyboardInterrupt:\n pass\n except (SystemExit, MemoryError):\n raise\n except:\n if not reloader: raise\n if not getattr(server, 'quiet', quiet):\n print_exc()\n time.sleep(interval)\n sys.exit(3)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading values from a dictionary structure.", "response": "def load_dict(self, source, namespace=''):\n \"\"\" Load values from a dictionary structure. Nesting can be used to\n represent namespaces.\n\n >>> c = ConfigDict()\n >>> c.load_dict({'some': {'namespace': {'key': 'value'} } })\n {'some.namespace.key': 'value'}\n \"\"\"\n for key, value in source.items():\n if isinstance(key, str):\n nskey = (namespace + '.' + key).strip('.')\n if isinstance(value, dict):\n self.load_dict(value, namespace=nskey)\n else:\n self[nskey] = value\n else:\n raise TypeError('Key has type %r (not a string)' % type(key))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef json(request, *args, **kwargs):\n # coerce to dictionary\n params = dict(request.GET.items())\n \n callback = params.pop('callback', None)\n url = params.pop('url', None)\n \n if not url:\n return HttpResponseBadRequest('Required parameter missing: URL')\n \n try:\n provider = oembed.site.provider_for_url(url)\n if not provider.provides:\n raise OEmbedMissingEndpoint()\n except OEmbedMissingEndpoint:\n raise Http404('No provider found for %s' % url)\n \n query = dict([(smart_str(k), smart_str(v)) for k, v in params.items() if v])\n \n try:\n resource = oembed.site.embed(url, **query)\n except OEmbedException, e:\n raise Http404('Error embedding %s: %s' % (url, str(e)))\n\n response = HttpResponse(mimetype='application/json')\n json = resource.json\n \n if callback:\n response.write('%s(%s)' % (defaultfilters.force_escape(callback), json))\n else:\n response.write(json)\n \n return response", "response": "Return a JSON representation of the site s metadata."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextracting and return oembed content for given urls", "response": "def consume_json(request):\n \"\"\"\n Extract and return oembed content for given urls.\n\n Required GET params:\n urls - list of urls to consume\n\n Optional GET params:\n width - maxwidth attribute for oembed content\n height - maxheight attribute for oembed content\n template_dir - template_dir to use when rendering oembed\n\n Returns:\n list of dictionaries with oembed metadata and renderings, json encoded\n \"\"\"\n client = OEmbedConsumer()\n \n urls = request.GET.getlist('urls')\n width = request.GET.get('width')\n height = request.GET.get('height')\n template_dir = request.GET.get('template_dir')\n\n output = {}\n ctx = RequestContext(request)\n\n for url in urls:\n try:\n provider = oembed.site.provider_for_url(url)\n except OEmbedMissingEndpoint:\n oembeds = None\n rendered = None\n else:\n oembeds = url\n rendered = client.parse_text(url, width, height, context=ctx, template_dir=template_dir)\n\n output[url] = {\n 'oembeds': oembeds,\n 'rendered': rendered,\n }\n\n return HttpResponse(simplejson.dumps(output), mimetype='application/json')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef oembed_schema(request):\n current_domain = Site.objects.get_current().domain\n url_schemes = [] # a list of dictionaries for all the urls we can match\n endpoint = reverse('oembed_json') # the public endpoint for our oembeds\n providers = oembed.site.get_providers()\n\n for provider in providers:\n # first make sure this provider class is exposed at the public endpoint\n if not provider.provides:\n continue\n \n match = None\n if isinstance(provider, DjangoProvider):\n # django providers define their regex_list by using urlreversing\n url_pattern = resolver.reverse_dict.get(provider._meta.named_view)\n\n # this regex replacement is set to be non-greedy, which results\n # in things like /news/*/*/*/*/ -- this is more explicit\n if url_pattern:\n regex = re.sub(r'%\\(.+?\\)s', '*', url_pattern[0][0][0])\n match = 'http://%s/%s' % (current_domain, regex)\n elif isinstance(provider, HTTPProvider):\n match = provider.url_scheme\n else:\n match = provider.regex\n\n if match:\n url_schemes.append({\n 'type': provider.resource_type,\n 'matches': match,\n 'endpoint': endpoint\n })\n \n url_schemes.sort(key=lambda item: item['matches'])\n \n response = HttpResponse(mimetype='application/json')\n response.write(simplejson.dumps(url_schemes))\n return response", "response": "A basic schema for OEmbed - able content."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_meta(*meta_file_parts, meta_key):\n meta_file = read(*meta_file_parts)\n meta_match = re.search(r\"^__{}__ = ['\\\"]([^'\\\"]*)['\\\"]\".format(meta_key),\n meta_file, re.M)\n if meta_match:\n return meta_match.group(1)\n raise RuntimeError(\"Unable to find __{}__ string.\".format(meta_key))", "response": "Extract __*meta__ from meta_file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef main(path):\n '''scan path directory and any subdirectories for valid captain scripts'''\n basepath = os.path.abspath(os.path.expanduser(str(path)))\n\n echo.h2(\"Available scripts in {}\".format(basepath))\n echo.br()\n for root_dir, dirs, files in os.walk(basepath, topdown=True):\n for f in fnmatch.filter(files, '*.py'):\n try:\n filepath = os.path.join(root_dir, f)\n\n # super edge case, this makes sure the python script won't start\n # an interactive console session which would cause the session\n # to start and not allow the for loop to complete\n with open(filepath, encoding=\"UTF-8\") as fp:\n body = fp.read()\n is_console = \"InteractiveConsole\" in body\n is_console = is_console or \"code\" in body\n is_console = is_console and \"interact(\" in body\n if is_console:\n continue\n\n s = captain.Script(filepath)\n if s.can_run_from_cli():\n rel_filepath = s.call_path(basepath)\n p = s.parser\n\n echo.h3(rel_filepath)\n\n desc = p.description\n if desc:\n echo.indent(desc, indent=(\" \" * 4))\n\n subcommands = s.subcommands\n if subcommands:\n echo.br()\n echo.indent(\"Subcommands:\", indent=(\" \" * 4))\n for sc in subcommands.keys():\n echo.indent(sc, indent=(\" \" * 6))\n\n echo.br()\n\n except captain.ParseError:\n pass\n\n except Exception as e:\n #echo.exception(e)\n #echo.err(\"Failed to parse {} because {}\", f, e.message)\n echo.err(\"Failed to parse {}\", f)\n echo.verbose(e.message)\n echo.br()", "response": "scan path directory and any subdirectories for valid captain scripts"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn Decimal of the tax rate for given info.", "response": "def get_rate(self, zipcode, city=None, state=None, multiple_rates=False):\n \"\"\"\n Finds sales tax for given info.\n Returns Decimal of the tax rate, e.g. 8.750.\n \"\"\"\n data = self.make_request_data(zipcode, city, state)\n\n r = requests.get(self.url, params=data)\n resp = r.json()\n\n return self.process_response(resp, multiple_rates)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_request_data(self, zipcode, city, state):\n data = {'key': self.api_key,\n 'postalcode': str(zipcode),\n 'city': city,\n 'state': state\n }\n data = ZipTaxClient._clean_request_data(data)\n return data", "response": "Make the request params given location data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprocesses the response from the ZipTax API and return the tax rates", "response": "def process_response(self, resp, multiple_rates):\n \"\"\" Get the tax rate from the ZipTax response \"\"\"\n self._check_for_exceptions(resp, multiple_rates)\n\n rates = {}\n for result in resp['results']:\n rate = ZipTaxClient._cast_tax_rate(result['taxSales'])\n rates[result['geoCity']] = rate\n if not multiple_rates:\n return rates[list(rates.keys())[0]]\n return rates"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if there are exceptions that should be raised", "response": "def _check_for_exceptions(self, resp, multiple_rates):\n \"\"\" Check if there are exceptions that should be raised \"\"\"\n if resp['rCode'] != 100:\n raise exceptions.get_exception_for_code(resp['rCode'])(resp)\n\n results = resp['results']\n if len(results) == 0:\n raise exceptions.ZipTaxNoResults('No results found')\n if len(results) > 1 and not multiple_rates:\n # It's fine if all the taxes are the same\n rates = [result['taxSales'] for result in results]\n if len(set(rates)) != 1:\n raise exceptions.ZipTaxMultipleResults('Multiple results found but requested only one')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_all_text(node):\n if node.nodeType == node.TEXT_NODE:\n return node.data\n else:\n text_string = \"\"\n for child_node in node.childNodes:\n text_string += get_all_text(child_node)\n return text_string", "response": "Recursively extract all text from node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nextract a package in a new temporary directory.", "response": "def _extract_packages(self):\n \"\"\"\n Extract a package in a new temporary directory.\n \"\"\"\n self.path_unpacked = mkdtemp(prefix=\"scoap3_package_\",\n dir=CFG_TMPSHAREDDIR)\n for path in self.retrieved_packages_unpacked:\n scoap3utils_extract_package(path, self.path_unpacked, self.logger)\n\n return self.path_unpacked"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef register(self, provider_class):\n if not issubclass(provider_class, BaseProvider):\n raise TypeError('%s is not a subclass of BaseProvider' % provider_class.__name__)\n \n if provider_class in self._registered_providers:\n raise AlreadyRegistered('%s is already registered' % provider_class.__name__)\n \n if issubclass(provider_class, DjangoProvider):\n # set up signal handler for cache invalidation\n signals.post_save.connect(\n self.invalidate_stored_oembeds,\n sender=provider_class._meta.model\n )\n \n # don't build the regex yet - if not all urlconfs have been loaded\n # and processed at this point, the DjangoProvider instances will fail\n # when attempting to reverse urlpatterns that haven't been created.\n # Rather, the regex-list will be populated once, on-demand.\n self._registered_providers.append(provider_class)\n \n # flag for re-population\n self.invalidate_providers()", "response": "Registers a provider with the site."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef unregister(self, provider_class):\n if not issubclass(provider_class, BaseProvider):\n raise TypeError('%s must be a subclass of BaseProvider' % provider_class.__name__)\n \n if provider_class not in self._registered_providers:\n raise NotRegistered('%s is not registered' % provider_class.__name__)\n \n self._registered_providers.remove(provider_class)\n \n # flag for repopulation\n self.invalidate_providers()", "response": "Unregisters a provider from the site."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef populate(self):\n self._registry = {}\n \n for provider_class in self._registered_providers:\n instance = provider_class()\n self._registry[instance] = instance.regex\n \n for stored_provider in StoredProvider.objects.active():\n self._registry[stored_provider] = stored_provider.regex\n \n self._populated = True", "response": "Populate the internal registry with the regexes for each\n provider instance"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the right provider for a URL", "response": "def provider_for_url(self, url):\n \"\"\"\n Find the right provider for a URL\n \"\"\"\n for provider, regex in self.get_registry().items():\n if re.match(regex, url) is not None:\n return provider\n \n raise OEmbedMissingEndpoint('No endpoint matches URL: %s' % url)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef embed(self, url, **kwargs):\n try:\n # first figure out the provider\n provider = self.provider_for_url(url)\n except OEmbedMissingEndpoint:\n raise\n else:\n try:\n # check the database for a cached response, because of certain\n # race conditions that exist with get_or_create(), do a filter\n # lookup and just grab the first item\n stored_match = StoredOEmbed.objects.filter(\n match=url, \n maxwidth=kwargs.get('maxwidth', None), \n maxheight=kwargs.get('maxheight', None),\n date_expires__gte=datetime.datetime.now())[0]\n return OEmbedResource.create_json(stored_match.response_json)\n except IndexError:\n # query the endpoint and cache response in db\n # prevent None from being passed in as a GET param\n params = dict([(k, v) for k, v in kwargs.items() if v])\n \n # request an oembed resource for the url\n resource = provider.request_resource(url, **params)\n \n try:\n cache_age = int(resource.cache_age)\n if cache_age < MIN_OEMBED_TTL:\n cache_age = MIN_OEMBED_TTL\n except:\n cache_age = DEFAULT_OEMBED_TTL\n \n date_expires = datetime.datetime.now() + datetime.timedelta(seconds=cache_age)\n \n stored_oembed, created = StoredOEmbed.objects.get_or_create(\n match=url,\n maxwidth=kwargs.get('maxwidth', None),\n maxheight=kwargs.get('maxheight', None))\n \n stored_oembed.response_json = resource.json\n stored_oembed.resource_type = resource.type\n stored_oembed.date_expires = date_expires\n \n if resource.content_object:\n stored_oembed.content_object = resource.content_object\n \n stored_oembed.save()\n return resource", "response": "Get a resource from the database and return it."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads up StoredProviders from url if it is an oembed scheme", "response": "def autodiscover(self, url):\n \"\"\"\n Load up StoredProviders from url if it is an oembed scheme\n \"\"\"\n headers, response = fetch_url(url)\n if headers['content-type'].split(';')[0] in ('application/json', 'text/javascript'):\n provider_data = json.loads(response)\n return self.store_providers(provider_data)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstoring all providers in the database.", "response": "def store_providers(self, provider_data):\n \"\"\"\n Iterate over the returned json and try to sort out any new providers\n \"\"\"\n if not hasattr(provider_data, '__iter__'):\n raise OEmbedException('Autodiscovered response not iterable')\n \n provider_pks = []\n \n for provider in provider_data:\n if 'endpoint' not in provider or \\\n 'matches' not in provider:\n continue\n \n resource_type = provider.get('type')\n if resource_type not in RESOURCE_TYPES:\n continue\n \n stored_provider, created = StoredProvider.objects.get_or_create(\n wildcard_regex=provider['matches']\n )\n \n if created:\n stored_provider.endpoint_url = relative_to_full( \n provider['endpoint'],\n provider['matches']\n )\n stored_provider.resource_type = resource_type\n stored_provider.save()\n \n provider_pks.append(stored_provider.pk)\n \n return StoredProvider.objects.filter(pk__in=provider_pks)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrequests an OEmbedResource for a given url.", "response": "def request_resource(self, url, **kwargs):\n \"\"\"\n Request an OEmbedResource for a given url. Some valid keyword args:\n - format\n - maxwidth\n - maxheight\n \"\"\"\n params = kwargs\n \n params['url'] = url\n params['format'] = 'json'\n \n if '?' in self.endpoint_url:\n url_with_qs = '%s&%s' % (self.endpoint_url.rstrip('&'), urlencode(params))\n else:\n url_with_qs = \"%s?%s\" % (self.endpoint_url, urlencode(params))\n \n headers, raw_response = self._fetch(url_with_qs)\n resource = self.convert_to_resource(headers, raw_response, params)\n \n return resource"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _image_field(self):\n for field in self.model._meta.fields:\n if isinstance(field, ImageField):\n return field.name", "response": "Try to automatically detect an image field"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _date_field(self):\n for field in self.model._meta.fields:\n if isinstance(field, (DateTimeField, DateField)):\n return field.name", "response": "Try to automatically detect an image field\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds a regex that matches the object_id of the object.", "response": "def _build_regex(self):\n \"\"\"\n Performs a reverse lookup on a named view and generates\n a list of regexes that match that object. It generates\n regexes with the domain name included, using sites provided\n by the get_sites() method.\n \n >>> regex = provider.regex\n >>> regex.pattern\n 'http://(www2.kusports.com|www2.ljworld.com|www.lawrence.com)/photos/(?P\\\\d{4})/(?P\\\\w{3})/(?P\\\\d{1,2})/(?P\\\\d+)/$'\n \"\"\"\n # get the regexes from the urlconf\n url_patterns = resolver.reverse_dict.get(self._meta.named_view)\n \n try:\n regex = url_patterns[1]\n except TypeError:\n raise OEmbedException('Error looking up %s' % self._meta.named_view)\n \n # get a list of normalized domains\n cleaned_sites = self.get_cleaned_sites()\n \n site_regexes = []\n \n for site in self.get_sites():\n site_regexes.append(cleaned_sites[site.pk][0])\n \n # join the sites together with the regex 'or'\n sites = '|'.join(site_regexes)\n \n # create URL-matching regexes for sites\n regex = re.compile('(%s)/%s' % (sites, regex))\n \n return regex"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a URL for any of our sites try and match it to one and return the domain & name of the match. If no match is found return the current. nacute_name.", "response": "def provider_from_url(self, url):\n \"\"\"\n Given a URL for any of our sites, try and match it to one, returning\n the domain & name of the match. If no match is found, return current.\n \n Returns a tuple of domain, site name -- used to determine 'provider'\n \"\"\"\n domain = get_domain(url)\n site_tuples = self.get_cleaned_sites().values()\n for domain_re, name, normalized_domain in site_tuples:\n if re.match(domain_re, domain):\n return normalized_domain, name\n site = Site.objects.get_current()\n return site.domain, site.name"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract the named parameters from a url regex.", "response": "def get_params(self, url):\n \"\"\"\n Extract the named parameters from a url regex. If the url regex does not contain\n named parameters, they will be keyed _0, _1, ...\n \n * Named parameters\n Regex:\n /photos/^(?P\\d{4})/(?P\\w{3})/(?P\\d{1,2})/(?P\\d+)/\n \n URL:\n http://www2.ljworld.com/photos/2009/oct/11/12345/\n \n Return Value:\n {u'day': '11', u'month': 'oct', u'object_id': '12345', u'year': '2009'}\n \n * Unnamed parameters\n Regex:\n /blah/([\\w-]+)/(\\d+)/\n \n URL:\n http://www.example.com/blah/hello/123/\n \n Return Value:\n {u'_0': 'hello', u'_1': '123'}\n \"\"\"\n match = re.match(self.regex, url)\n if match is not None:\n params = match.groupdict()\n if not params:\n params = {}\n for i, group in enumerate(match.groups()[1:]):\n params['_%s' % i] = group\n return params\n \n raise OEmbedException('No regex matched the url %s' % (url))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget an object from the database", "response": "def get_object(self, url):\n \"\"\"\n Fields to match is a mapping of url params to fields, so for\n the photos example above, it would be:\n \n fields_to_match = { 'object_id': 'id' }\n \n This procedure parses out named params from a URL and then uses\n the fields_to_match dictionary to generate a query.\n \"\"\"\n params = self.get_params(url)\n query = {}\n for key, value in self._meta.fields_to_match.iteritems():\n try:\n query[value] = params[key]\n except KeyError:\n raise OEmbedException('%s was not found in the urlpattern parameters. Valid names are: %s' % (key, ', '.join(params.keys())))\n \n try:\n obj = self.get_queryset().get(**query)\n except self._meta.model.DoesNotExist:\n raise OEmbedException('Requested object not found')\n \n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates the html attribute of an oembed resource using a template.", "response": "def render_html(self, obj, context=None):\n \"\"\"\n Generate the 'html' attribute of an oembed resource using a template.\n Sort of a corollary to the parser's render_oembed method. By default,\n the current mapping will be passed in as the context.\n \n OEmbed templates are stored in:\n \n oembed/provider/[app_label]_[model].html\n \n -- or --\n \n oembed/provider/media_video.html\n \"\"\" \n provided_context = context or Context()\n context = RequestContext(mock_request())\n context.update(provided_context)\n \n context.push()\n context[self._meta.context_varname] = obj\n rendered = render_to_string(self._meta.template_name, context)\n context.pop()\n return rendered"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmapping an attribute to the object.", "response": "def map_attr(self, mapping, attr, obj):\n \"\"\"\n A kind of cheesy method that allows for callables or attributes to\n be used interchangably\n \"\"\"\n if attr not in mapping and hasattr(self, attr):\n if not callable(getattr(self, attr)):\n mapping[attr] = getattr(self, attr)\n else:\n mapping[attr] = getattr(self, attr)(obj)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an ImageFileField instance", "response": "def get_image(self, obj):\n \"\"\"\n Return an ImageFileField instance\n \"\"\"\n if self._meta.image_field:\n return getattr(obj, self._meta.image_field)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef map_to_dictionary(self, url, obj, **kwargs):\n maxwidth = kwargs.get('maxwidth', None)\n maxheight = kwargs.get('maxheight', None)\n \n provider_url, provider_name = self.provider_from_url(url)\n \n mapping = {\n 'version': '1.0',\n 'url': url,\n 'provider_name': provider_name,\n 'provider_url': provider_url,\n 'type': self.resource_type\n }\n \n # a hook\n self.preprocess(obj, mapping, **kwargs)\n \n # resize image if we have a photo, otherwise use the given maximums\n if self.resource_type == 'photo' and self.get_image(obj):\n self.resize_photo(obj, mapping, maxwidth, maxheight)\n elif self.resource_type in ('video', 'rich', 'photo'):\n width, height = size_to_nearest(\n maxwidth,\n maxheight,\n self._meta.valid_sizes,\n self._meta.force_fit\n )\n mapping.update(width=width, height=height)\n \n # create a thumbnail\n if self.get_image(obj):\n self.thumbnail(obj, mapping)\n \n # map attributes to the mapping dictionary. if the attribute is\n # a callable, it must have an argument signature of\n # (self, obj)\n for attr in ('title', 'author_name', 'author_url', 'html'):\n self.map_attr(mapping, attr, obj)\n \n # fix any urls\n if 'url' in mapping:\n mapping['url'] = relative_to_full(mapping['url'], url)\n \n if 'thumbnail_url' in mapping:\n mapping['thumbnail_url'] = relative_to_full(mapping['thumbnail_url'], url)\n \n if 'html' not in mapping and mapping['type'] in ('video', 'rich'):\n mapping['html'] = self.render_html(obj, context=Context(mapping))\n \n # a hook\n self.postprocess(obj, mapping, **kwargs)\n \n return mapping", "response": "Map the object to a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrequests an OEmbedResource for a given url.", "response": "def request_resource(self, url, **kwargs):\n \"\"\"\n Request an OEmbedResource for a given url. Some valid keyword args:\n - format\n - maxwidth\n - maxheight\n \"\"\"\n obj = self.get_object(url)\n \n mapping = self.map_to_dictionary(url, obj, **kwargs)\n \n resource = OEmbedResource.create(mapping)\n resource.content_object = obj\n \n return resource"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_object(self, url, month_format='%b', day_format='%d'):\n params = self.get_params(url)\n try:\n year = params[self._meta.year_part]\n month = params[self._meta.month_part]\n day = params[self._meta.day_part]\n except KeyError:\n try:\n # named lookups failed, so try to get the date using the first\n # three parameters\n year, month, day = params['_0'], params['_1'], params['_2']\n except KeyError:\n raise OEmbedException('Error extracting date from url parameters')\n \n try:\n tt = time.strptime('%s-%s-%s' % (year, month, day),\n '%s-%s-%s' % ('%Y', month_format, day_format))\n date = datetime.date(*tt[:3])\n except ValueError:\n raise OEmbedException('Error parsing date from: %s' % url)\n\n # apply the date-specific lookups\n if isinstance(self._meta.model._meta.get_field(self._meta.date_field), DateTimeField):\n min_date = datetime.datetime.combine(date, datetime.time.min)\n max_date = datetime.datetime.combine(date, datetime.time.max)\n query = {'%s__range' % self._meta.date_field: (min_date, max_date)}\n else:\n query = {self._meta.date_field: date}\n \n # apply the regular search lookups\n for key, value in self._meta.fields_to_match.iteritems():\n try:\n query[value] = params[key]\n except KeyError:\n raise OEmbedException('%s was not found in the urlpattern parameters. Valid names are: %s' % (key, ', '.join(params.keys())))\n \n try:\n obj = self.get_queryset().get(**query)\n except self._meta.model.DoesNotExist:\n raise OEmbedException('Requested object not found')\n \n return obj", "response": "Parses the date from a url and uses it in the query."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_record(self):\n self.recid = self.get_recid()\n self.remove_controlfields()\n self.update_system_numbers()\n self.add_systemnumber(\"Inspire\", recid=self.recid)\n self.add_control_number(\"003\", \"SzGeCERN\")\n self.update_collections()\n self.update_languages()\n self.update_reportnumbers()\n self.update_authors()\n self.update_journals()\n self.update_subject_categories(\"INSPIRE\", \"SzGeCERN\", \"categories_cds\")\n self.update_pagenumber()\n self.update_notes()\n self.update_experiments()\n self.update_isbn()\n self.update_dois()\n self.update_links_and_ffts()\n self.update_date()\n self.update_date_year()\n self.update_hidden_notes()\n self.update_oai_info()\n self.update_cnum()\n self.update_conference_info()\n\n self.fields_list = [\n \"909\", \"541\", \"961\",\n \"970\", \"690\", \"695\",\n \"981\",\n ]\n self.strip_fields()\n\n if \"ANNOUNCEMENT\" in self.collections:\n self.update_conference_111()\n self.update_conference_links()\n record_add_field(self.record, \"690\", ind1=\"C\", subfields=[(\"a\", \"CONFERENCE\")])\n\n if \"THESIS\" in self.collections:\n self.update_thesis_information()\n self.update_thesis_supervisors()\n\n if \"PROCEEDINGS\" in self.collections:\n # Special proceeding syntax\n self.update_title_to_proceeding()\n self.update_author_to_proceeding()\n record_add_field(self.record, \"690\", ind1=\"C\", subfields=[(\"a\", \"CONFERENCE\")])\n\n # 690 tags\n if self.tag_as_cern:\n record_add_field(self.record, \"690\", ind1=\"C\", subfields=[(\"a\", \"CERN\")])\n\n return self.record", "response": "Override the base method to get the record."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_oai_info(self):\n for field in record_get_field_instances(self.record, '909', ind1=\"C\", ind2=\"O\"):\n new_subs = []\n for tag, value in field[0]:\n if tag == \"o\":\n new_subs.append((\"a\", value))\n else:\n new_subs.append((tag, value))\n if value in [\"CERN\", \"CDS\", \"ForCDS\"]:\n self.tag_as_cern = True\n record_add_field(self.record, '024', ind1=\"8\", subfields=new_subs)\n record_delete_fields(self.record, '909')", "response": "Update the 909 OAI info to 035."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_cnum(self):\n if \"ConferencePaper\" not in self.collections:\n cnums = record_get_field_values(self.record, '773', code=\"w\")\n for cnum in cnums:\n cnum_subs = [\n (\"9\", \"INSPIRE-CNUM\"),\n (\"a\", cnum)\n ]\n record_add_field(self.record, \"035\", subfields=cnum_subs)", "response": "Check if we shall add cnum in 035."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_hidden_notes(self):\n if not self.tag_as_cern:\n notes = record_get_field_instances(self.record,\n tag=\"595\")\n for field in notes:\n for dummy, value in field[0]:\n if value == \"CDS\":\n self.tag_as_cern = True\n record_delete_fields(self.record, tag=\"595\")", "response": "Remove hidden notes and tag a CERN if detected."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_system_numbers(self):\n scn_035_fields = record_get_field_instances(self.record, '035')\n new_fields = []\n for field in scn_035_fields:\n subs = field_get_subfields(field)\n if '9' in subs:\n if subs['9'][0].lower() == \"cds\" and subs.get('a'):\n self.add_control_number(\"001\", subs.get('a')[0])\n if subs['9'][0].lower() in [\"inspire\", \"spirestex\", \"inspiretex\", \"desy\", \"cds\"]:\n continue\n new_fields.append(field_get_subfield_instances(field))\n record_delete_fields(self.record, tag=\"035\")\n for field in new_fields:\n record_add_field(self.record, tag=\"035\", subfields=field)", "response": "Update the system numbers of the 3035 records."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the list of collections that this record belongs to.", "response": "def update_collections(self):\n \"\"\"Try to determine which collections this record should belong to.\"\"\"\n for value in record_get_field_values(self.record, '980', code='a'):\n if 'NOTE' in value.upper():\n self.collections.add('NOTE')\n if 'THESIS' in value.upper():\n self.collections.add('THESIS')\n\n if 'PUBLISHED' in value.upper():\n self.collections.add('ARTICLE')\n\n if 'CONFERENCES' in value.upper():\n self.collections.add('ANNOUNCEMENT')\n\n if 'PROCEEDINGS' in value.upper():\n self.collections.add('PROCEEDINGS')\n elif 'CONFERENCEPAPER' in value.upper() and \\\n \"ConferencePaper\" not in self.collections:\n self.collections.add('ConferencePaper')\n if self.is_published() and \"ARTICLE\" not in self.collections:\n self.collections.add('ARTICLE')\n else:\n self.collections.add('PREPRINT')\n\n if \"HIDDEN\" in value.upper():\n self.hidden = True\n\n # Clear out any existing ones.\n record_delete_fields(self.record, \"980\")\n\n if not self.collections:\n self.collections.add('PREPRINT')\n\n for collection in self.collections:\n record_add_field(self.record,\n tag='980',\n subfields=[('a', collection)])\n if collection in self.collection_base:\n subs = [('a', self.collection_base[collection])]\n record_add_field(self.record,\n tag='960',\n subfields=subs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove INSPIRE specific notes.", "response": "def update_notes(self):\n \"\"\"Remove INSPIRE specific notes.\"\"\"\n fields = record_get_field_instances(self.record, '500')\n for field in fields:\n subs = field_get_subfields(field)\n for sub in subs.get('a', []):\n sub = sub.strip() # remove any spaces before/after\n if sub.startswith(\"*\") and sub.endswith(\"*\"):\n record_delete_field(self.record, tag=\"500\",\n field_position_global=field[4])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmove title info from 245 to 111 proceeding style.", "response": "def update_title_to_proceeding(self):\n \"\"\"Move title info from 245 to 111 proceeding style.\"\"\"\n titles = record_get_field_instances(self.record,\n tag=\"245\")\n for title in titles:\n subs = field_get_subfields(title)\n new_subs = []\n if \"a\" in subs:\n new_subs.append((\"a\", subs['a'][0]))\n if \"b\" in subs:\n new_subs.append((\"c\", subs['b'][0]))\n record_add_field(self.record,\n tag=\"111\",\n subfields=new_subs)\n record_delete_fields(self.record, tag=\"245\")\n record_delete_fields(self.record, tag=\"246\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_experiments(self):\n # 693 Remove if 'not applicable'\n for field in record_get_field_instances(self.record, '693'):\n subs = field_get_subfields(field)\n acc_experiment = subs.get(\"e\", [])\n if not acc_experiment:\n acc_experiment = subs.get(\"a\", [])\n if not acc_experiment:\n continue\n experiment = acc_experiment[-1]\n\n # Handle special case of leading experiments numbers NA-050 -> NA 50\n e_suffix = \"\"\n if \"-NA-\" in experiment or \\\n \"-RD-\" in experiment or \\\n \"-WA-\" in experiment:\n splitted_experiment = experiment.split(\"-\")\n e_suffix = \"-\".join(splitted_experiment[2:])\n if e_suffix.startswith(\"0\"):\n e_suffix = e_suffix[1:]\n experiment = \"-\".join(splitted_experiment[:2]) # only CERN-NA\n\n translated_experiment = self.get_config_item(experiment,\n \"experiments\")\n if not translated_experiment:\n continue\n new_subs = []\n if \"---\" in translated_experiment:\n experiment_a, experiment_e = translated_experiment.split(\"---\")\n new_subs.append((\"a\", experiment_a.replace(\"-\", \" \")))\n else:\n experiment_e = translated_experiment\n new_subs.append((\"e\", experiment_e.replace(\"-\", \" \") + e_suffix))\n record_delete_field(self.record, tag=\"693\",\n field_position_global=field[4])\n record_add_field(self.record, \"693\", subfields=new_subs)", "response": "Update the experiments of the CERN record."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_authors(self):\n author_names = record_get_field_instances(self.record, '100')\n author_names.extend(record_get_field_instances(self.record, '700'))\n for field in author_names:\n subs = field_get_subfields(field)\n for idx, (key, value) in enumerate(field[0]):\n if key == 'a':\n field[0][idx] = ('a', value.replace(\".\", \" \").strip())\n elif key == 'v':\n del field[0][idx]\n if subs.get(\"u\", None) == \"CERN\":\n self.tag_as_cern = True", "response": "Update the author names of the archive."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_isbn(self):\n isbns = record_get_field_instances(self.record, '020')\n for field in isbns:\n for idx, (key, value) in enumerate(field[0]):\n if key == 'a':\n field[0][idx] = ('a', value.replace(\"-\", \"\").strip())", "response": "Remove dashes from ISBN."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_dois(self):\n dois = record_get_field_instances(self.record, '024', ind1=\"7\")\n all_dois = {}\n for field in dois:\n subs = field_get_subfield_instances(field)\n subs_dict = dict(subs)\n if subs_dict.get('a'):\n if subs_dict['a'] in all_dois:\n record_delete_field(self.record, tag='024', ind1='7', field_position_global=field[4])\n continue\n all_dois[subs_dict['a']] = field", "response": "Update all BibMatch DOIs."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_journals(self):\n for field in record_get_field_instances(self.record, '773'):\n subs = field_get_subfield_instances(field)\n new_subs = []\n volume_letter = \"\"\n journal_name = \"\"\n for idx, (key, value) in enumerate(subs):\n if key == 'p':\n journal_name = self.get_config_item(value, \"journals\", allow_substring=False)\n # Make sure journal names have the form (dot)(space) (I know it's horrible)\n journal_name = journal_name.replace('. ', '.').replace('.', '. ').replace('. ,', '.,').strip()\n elif key == 'v':\n volume_letter = value\n else:\n new_subs.append((key, value))\n\n if not journal_name == \"PoS\":\n # Special handling of journal name and volumes, except PoS\n letter = return_letters_from_string(volume_letter)\n if letter:\n journal_name = \"{0} {1}\".format(journal_name, letter)\n volume_letter = volume_letter.strip(letter)\n\n if journal_name:\n new_subs.append((\"p\", journal_name))\n if volume_letter:\n new_subs.append((\"v\", volume_letter))\n record_delete_field(self.record, tag=\"773\",\n field_position_global=field[4])\n record_add_field(self.record, \"773\", subfields=new_subs)", "response": "Update the journals field in the record."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_thesis_supervisors(self):\n for field in record_get_field_instances(self.record, '701'):\n subs = list(field[0])\n subs.append((\"e\", \"dir.\"))\n record_add_field(self.record, '700', subfields=subs)\n record_delete_fields(self.record, '701')", "response": "700 -> 701 Thesis supervisors."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates the page number of the records.", "response": "def update_pagenumber(self):\n \"\"\"300 page number.\"\"\"\n pages = record_get_field_instances(self.record, '300')\n for field in pages:\n for idx, (key, value) in enumerate(field[0]):\n if key == 'a':\n field[0][idx] = ('a', \"{0} p\".format(value))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks if the record has already been published.", "response": "def is_published(self):\n \"\"\"Check fields 980 and 773 to see if the record has already been published.\n\n :return: True is published, else False\n \"\"\"\n field773 = record_get_field_instances(self.record, '773')\n for f773 in field773:\n if 'c' in field_get_subfields(f773):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupdate links and FFTs for the current ISO.", "response": "def update_links_and_ffts(self):\n \"\"\"FFT (856) Dealing with files.\"\"\"\n for field in record_get_field_instances(self.record,\n tag='856',\n ind1='4'):\n subs = field_get_subfields(field)\n newsubs = []\n url = subs.get(\"u\", [])\n\n if not url:\n record_delete_field(self.record, '856', ind1='4',\n field_position_global=field[4])\n continue\n url = url[0]\n if \"inspirehep.net/record\" in url and url.endswith(\"pdf\"):\n # We have an FFT from INSPIRE\n newsubs.append(('a', url))\n description = subs.get(\"y\", [])\n if description:\n newsubs.append(('d', description[0]))\n if newsubs:\n record_add_field(self.record, 'FFT', subfields=newsubs)\n record_delete_field(self.record, '856', ind1='4',\n field_position_global=field[4])\n else:\n # Remove $w\n for idx, (key, value) in enumerate(field[0]):\n if key == 'w':\n del field[0][idx]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update_languages(self):\n language_fields = record_get_field_instances(self.record, '041')\n language = \"eng\"\n record_delete_fields(self.record, \"041\")\n for field in language_fields:\n subs = field_get_subfields(field)\n if 'a' in subs:\n language = self.get_config_item(subs['a'][0], \"languages\")\n break\n new_subs = [('a', language)]\n record_add_field(self.record, \"041\", subfields=new_subs)", "response": "Update the languages field in the record."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pathjoin(*args, **kwargs):\n log.debug('pathjoin: %r' % list(args))\n\n def _pathjoin(*args, **kwargs):\n len_ = len(args) - 1\n if len_ < 0:\n raise Exception('no args specified')\n elif len_ == 0:\n if not isinstance(args, basestring):\n if hasattr(args, '__iter__'):\n _args = args\n _args\n args = args[0]\n for i, arg in enumerate(args):\n if not i:\n yield arg.rstrip('/')\n elif i == len_:\n yield arg.lstrip('/')\n else:\n yield arg.strip('/')\n joined_path = u'/'.join(_pathjoin(*args))\n return sanitize_path(joined_path)", "response": "Join the paths of a single object with the key - value pairs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate the HTML directory listing for the given directory.", "response": "def generate_dirlist_html(FS, filepath):\n \"\"\"\n Generate directory listing HTML\n\n Arguments:\n FS (FS): filesystem object to read files from\n filepath (str): path to generate directory listings for\n\n Keyword Arguments:\n list_dir (callable: list[str]): list file names in a directory\n isdir (callable: bool): os.path.isdir\n\n Yields:\n str: lines of an HTML table\n \"\"\"\n yield ''\n if filepath == '/':\n filepath = ''\n for name in FS.listdir(filepath):\n full_path = pathjoin(filepath, name)\n if FS.isdir(full_path):\n full_path = full_path + '/'\n yield u''.format(\n cgi.escape(full_path)) # TODO XXX\n yield '
{0}
'"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if files are not being uploaded to server.", "response": "def check_pkgs_integrity(filelist, logger, ftp_connector,\n timeout=120, sleep_time=10):\n \"\"\"\n Checks if files are not being uploaded to server.\n @timeout - time after which the script will register an error.\n \"\"\"\n ref_1 = []\n ref_2 = []\n i = 1\n print >> sys.stdout, \"\\nChecking packages integrity.\"\n for filename in filelist:\n # ref_1.append(self.get_remote_file_size(filename))\n get_remote_file_size(ftp_connector, filename, ref_1)\n print >> sys.stdout, \"\\nGoing to sleep for %i sec.\" % (sleep_time,)\n time.sleep(sleep_time)\n\n while sleep_time*i < timeout:\n for filename in filelist:\n # ref_2.append(self.get_remote_file_size(filename))\n get_remote_file_size(ftp_connector, filename, ref_2)\n if ref_1 == ref_2:\n print >> sys.stdout, \"\\nIntegrity OK:)\"\n logger.info(\"Packages integrity OK.\")\n break\n else:\n print >> sys.stdout, \"\\nWaiting %d time for itegrity...\" % (i,)\n logger.info(\"\\nWaiting %d time for itegrity...\" % (i,))\n i += 1\n ref_1, ref_2 = ref_2, []\n time.sleep(sleep_time)\n else:\n not_finished_files = []\n for count, val1 in enumerate(ref_1):\n if val1 != ref_2[count]:\n not_finished_files.append(filelist[count])\n\n print >> sys.stdout, \"\\nOMG, OMG something wrong with integrity.\"\n logger.error(\"Integrity check faild for files %s\"\n % (not_finished_files,))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nnormalize a query string.", "response": "def normalize_query(query_string, terms=TERMS, norm_space=NORM_SPACE):\n \"\"\"\n Example:\n >>> normalize_query(' some random words \"with quotes \" and spaces')\n ['some', 'random', 'words', 'with quotes', 'and', 'spaces']\n \"\"\"\n return [\n norm_space(' ', (t[0] or t[1]).strip()) for t in terms(query_string)]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef collapse_initials(name):\n if len(name.split()) > 1:\n name = re.sub(r'([A-Z]\\.) +(?=[A-Z]\\.)', r'\\1', name)\n return name", "response": "Removes the space between initials.\n eg T. A. --> T. A."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fix_name_capitalization(lastname, givennames):\n lastnames = lastname.split()\n if len(lastnames) == 1:\n if '-' in lastname:\n names = lastname.split('-')\n names = map(lambda a: a[0] + a[1:].lower(), names)\n lastname = '-'.join(names)\n else:\n lastname = lastname[0] + lastname[1:].lower()\n else:\n names = []\n for name in lastnames:\n if re.search(r'[A-Z]\\.', name):\n names.append(name)\n else:\n names.append(name[0] + name[1:].lower())\n lastname = ' '.join(names)\n lastname = collapse_initials(lastname)\n names = []\n for name in givennames:\n if re.search(r'[A-Z]\\.', name):\n names.append(name)\n else:\n names.append(name[0] + name[1:].lower())\n givennames = ' '.join(names)\n return lastname, givennames", "response": "Converts capital letters to lower keeps first letter capital."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nscan a block of text and extracts oembed data on any urls and returns a list of dictionaries containing the oembed data.", "response": "def extract_oembeds(self, text, maxwidth=None, maxheight=None, resource_type=None):\n \"\"\"\n Scans a block of text and extracts oembed data on any urls,\n returning it in a list of dictionaries\n \"\"\"\n parser = text_parser()\n urls = parser.extract_urls(text)\n return self.handle_extracted_urls(urls, maxwidth, maxheight, resource_type)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef strip(self, text, *args, **kwargs):\n if OEMBED_DEFAULT_PARSE_HTML:\n extracted = self.extract_oembeds_html(text, *args, **kwargs)\n else:\n extracted = self.extract_oembeds(text, *args, **kwargs)\n \n matches = [r['original_url'] for r in extracted]\n match_handler = lambda m: m.group() not in matches and m.group() or ''\n \n return re.sub(URL_RE, match_handler, text)", "response": "Strip the OEMBED tags from the given text."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef autodiscover():\n import imp\n from django.conf import settings\n \n for app in settings.INSTALLED_APPS:\n try:\n app_path = __import__(app, {}, {}, [app.split('.')[-1]]).__path__\n except AttributeError:\n continue\n \n try:\n imp.find_module('oembed_providers', app_path)\n except ImportError:\n continue\n \n __import__(\"%s.oembed_providers\" % app)", "response": "Autodiscover the provider index."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npass in a list of options, promt the user to select one, and return the selected option or None", "response": "def select(options=None):\n \"\"\" pass in a list of options, promt the user to select one, and return the selected option or None \"\"\"\n if not options:\n return None\n width = len(str(len(options)))\n for x,option in enumerate(options):\n sys.stdout.write('{:{width}}) {}\\n'.format(x+1,option, width=width))\n\n sys.stdout.write('{:>{width}} '.format('#?', width=width+1))\n sys.stdout.flush()\n if sys.stdin.isatty():\n # regular prompt\n try:\n response = raw_input().strip()\n except (EOFError, KeyboardInterrupt):\n # handle ctrl-d, ctrl-c\n response = ''\n else:\n # try connecting to current tty, when using pipes\n sys.stdin = open(\"/dev/tty\")\n try:\n response = ''\n while True:\n response += sys.stdin.read(1)\n if response.endswith('\\n'):\n break\n except (EOFError, KeyboardInterrupt):\n sys.stdout.flush()\n pass\n try:\n response = int(response) - 1\n except ValueError:\n return None\n if response < 0 or response >= len(options):\n return None\n return options[response]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntransforms the argparse arguments from Namespace to dict and then to Bunch Therefore it is not necessary to access the arguments using the dict syntax The settings can be called like regular vars on the settings object", "response": "def main():\n argparser = ArgumentParser()\n\n subparsers = argparser.add_subparsers(dest='selected_subparser')\n\n all_parser = subparsers.add_parser('all')\n elsevier_parser = subparsers.add_parser('elsevier')\n oxford_parser = subparsers.add_parser('oxford')\n springer_parser = subparsers.add_parser('springer')\n\n all_parser.add_argument('--update-credentials', action='store_true')\n\n elsevier_parser.add_argument('--run-locally', action='store_true')\n elsevier_parser.add_argument('--package-name')\n elsevier_parser.add_argument('--path')\n elsevier_parser.add_argument('--CONSYN', action='store_true')\n elsevier_parser.add_argument('--update-credentials', action='store_true')\n elsevier_parser.add_argument('--extract-nations', action='store_true')\n\n oxford_parser.add_argument('--dont-empty-ftp', action='store_true')\n oxford_parser.add_argument('--package-name')\n oxford_parser.add_argument('--path')\n oxford_parser.add_argument('--update-credentials', action='store_true')\n oxford_parser.add_argument('--extract-nations', action='store_true')\n\n springer_parser.add_argument('--package-name')\n springer_parser.add_argument('--path')\n springer_parser.add_argument('--update-credentials', action='store_true')\n springer_parser.add_argument('--extract-nations', action='store_true')\n\n '''\n Transforms the argparse arguments from Namespace to dict and then to Bunch\n Therefore it is not necessary to access the arguments using the dict syntax\n The settings can be called like regular vars on the settings object\n '''\n\n settings = Bunch(vars(argparser.parse_args()))\n\n call_package(settings)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread a dom xml element in oaidc format and returns the bibrecord object", "response": "def get_record(self, record):\n \"\"\" Reads a dom xml element in oaidc format and\n returns the bibrecord object \"\"\"\n self.document = record\n rec = create_record()\n language = self._get_language()\n if language and language != 'en':\n record_add_field(rec, '041', subfields=[('a', language)])\n publisher = self._get_publisher()\n date = self._get_date()\n if publisher and date:\n record_add_field(rec, '260', subfields=[('b', publisher),\n ('c', date)])\n elif publisher:\n record_add_field(rec, '260', subfields=[('b', publisher)])\n elif date:\n record_add_field(rec, '260', subfields=[('c', date)])\n title = self._get_title()\n if title:\n record_add_field(rec, '245', subfields=[('a', title)])\n record_copyright = self._get_copyright()\n if record_copyright:\n record_add_field(rec, '540', subfields=[('a', record_copyright)])\n subject = self._get_subject()\n if subject:\n record_add_field(rec, '650', ind1='1', ind2='7', subfields=[('a', subject),\n ('2', 'PoS')])\n authors = self._get_authors()\n first_author = True\n for author in authors:\n subfields = [('a', author[0])]\n for affiliation in author[1]:\n subfields.append(('v', affiliation))\n if first_author:\n record_add_field(rec, '100', subfields=subfields)\n first_author = False\n else:\n record_add_field(rec, '700', subfields=subfields)\n identifier = self.get_identifier()\n conference = identifier.split(':')[2]\n conference = conference.split('/')[0]\n contribution = identifier.split(':')[2]\n contribution = contribution.split('/')[1]\n record_add_field(rec, '773', subfields=[('p', 'PoS'),\n ('v', conference.replace(' ', '')),\n ('c', contribution),\n ('y', date[:4])])\n record_add_field(rec, '980', subfields=[('a', 'ConferencePaper')])\n record_add_field(rec, '980', subfields=[('a', 'HEP')])\n return rec"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndisplays a progress that can update in place", "response": "def progress(length, **kwargs):\n \"\"\"display a progress that can update in place\n\n example -- \n total_length = 1000\n with echo.progress(total_length) as p:\n for x in range(total_length):\n # do something crazy\n p.update(x)\n\n length -- int -- the total size of what you will be updating progress on\n \"\"\"\n quiet = False\n progress_class = kwargs.pop(\"progress_class\", Progress)\n kwargs[\"write_method\"] = istdout.info\n kwargs[\"width\"] = kwargs.get(\"width\", globals()[\"WIDTH\"])\n kwargs[\"length\"] = length\n pbar = progress_class(**kwargs)\n pbar.update(0)\n yield pbar\n pbar.update(length)\n br()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef err(format_msg, *args, **kwargs):\n '''print format_msg to stderr'''\n exc_info = kwargs.pop(\"exc_info\", False)\n stderr.warning(str(format_msg).format(*args, **kwargs), exc_info=exc_info)", "response": "print format_msg to stderr"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprint format_msg to stdout taking into account -- quiet setting", "response": "def out(format_msg=\"\", *args, **kwargs):\n '''print format_msg to stdout, taking into account --quiet setting'''\n logmethod = kwargs.get(\"logmethod\", stdout.info)\n\n if format_msg != \"\":\n if Prefix.has():\n if isinstance(format_msg, basestring):\n format_msg = Prefix.get() + format_msg\n else:\n format_msg = Prefix.get() + str(format_msg)\n\n if isinstance(format_msg, basestring):\n if args or kwargs:\n s = format_msg.format(*args, **kwargs)\n else:\n s = format_msg\n logmethod(s)\n# width = globals()[\"width\"]\n# s = textwrap.fill(s, width=width)\n# stdout.info(s)\n\n else:\n logmethod(str(format_msg))\n\n else:\n logmethod(\"\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef verbose(format_msg=\"\", *args, **kwargs):\n '''print format_msg to stdout, taking into account --verbose flag'''\n kwargs[\"logmethod\"] = stdout.debug\n out(format_msg, *args, **kwargs)", "response": "print format_msg to stdout taking into account verbose flag"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprints a banner for the available language files", "response": "def banner(*lines, **kwargs):\n \"\"\"prints a banner\n\n sep -- string -- the character that will be on the line on the top and bottom\n and before any of the lines, defaults to *\n count -- integer -- the line width, defaults to 80\n \"\"\"\n sep = kwargs.get(\"sep\", \"*\")\n count = kwargs.get(\"width\", globals()[\"WIDTH\"])\n\n out(sep * count)\n if lines:\n out(sep)\n\n for line in lines:\n out(\"{} {}\".format(sep, line))\n\n out(sep)\n out(sep * count)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef table(*columns, **kwargs):\n ret = []\n prefix = kwargs.get('prefix', '')\n buf_count = kwargs.get('buf_count', 2)\n if len(columns) == 1:\n columns = list(columns[0])\n else:\n # without the list the zip iterator gets spent, I'm sure I can make this\n # better\n columns = list(zip(*columns))\n\n headers = kwargs.get(\"headers\", [])\n if headers:\n columns.insert(0, headers)\n\n # we have to go through all the rows and calculate the length of each\n # column of each row\n widths = kwargs.get(\"widths\", [])\n row_counts = Counter()\n for i in range(len(widths)):\n row_counts[i] = int(widths[i])\n\n width = int(kwargs.get(\"width\", 0))\n for row in columns:\n for i, c in enumerate(row):\n if isinstance(c, basestring):\n cl = len(c)\n else:\n cl = len(str(c))\n if cl > row_counts[i]:\n row_counts[i] = cl\n\n width = int(kwargs.get(\"width\", 0))\n if width:\n for i in row_counts:\n if row_counts[i] < width:\n row_counts[i] = width\n\n # actually go through and format each row\n def colstr(c):\n if isinstance(c, basestring): return c\n return str(c)\n\n def rowstr(row, prefix, row_counts):\n row_format = prefix\n cols = list(map(colstr, row))\n for i in range(len(row_counts)):\n c = cols[i]\n # build the format string for each row, we use the row_counts found\n # above to decide how much padding each column should get\n # https://stackoverflow.com/a/9536084/5006\n if re.match(r\"^\\d+(?:\\.\\d+)?$\", c):\n if i == 0:\n row_format += \"{:>\" + str(row_counts[i]) + \"}\"\n else:\n row_format += \"{:>\" + str(row_counts[i] + buf_count) + \"}\"\n else:\n row_format += \"{:<\" + str(row_counts[i] + buf_count) + \"}\"\n\n return row_format.format(*cols)\n\n for row in columns:\n ret.append(rowstr(row, prefix, row_counts))\n\n out(os.linesep.join(ret))", "response": "format columned data so we can easily print it out on a console"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef prompt(question, choices=None):\n\n if not re.match(\"\\s$\", question):\n question = \"{}: \".format(question)\n\n while True:\n if sys.version_info[0] > 2:\n answer = input(question)\n\n else:\n answer = raw_input(question)\n\n if not choices or answer in choices:\n break\n\n return answer", "response": "echo a prompt to the user and wait for an answer"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_records(self, url):\n page = urllib2.urlopen(url)\n pages = [BeautifulSoup(page)]\n #content spread over several pages?\n numpag = pages[0].body.findAll('span', attrs={'class': 'number-of-pages'})\n if len(numpag) > 0:\n if re.search('^\\d+$', numpag[0].string):\n for i in range(int(numpag[0].string)-1):\n page = urllib2.urlopen('%s/page/%i' % (url, i+2))\n pages.append(BeautifulSoup(page))\n else:\n print(\"number of pages %s not an integer\" % (numpag[0].string))\n impl = getDOMImplementation()\n doc = impl.createDocument(None, \"collection\", None)\n links = []\n for page in pages:\n links += page.body.findAll('p', attrs={'class': 'title'})\n links += page.body.findAll('h3', attrs={'class': 'title'})\n for link in links:\n record = self._get_record(link)\n doc.firstChild.appendChild(record)\n return doc.toprettyxml()", "response": "Returns the records listed in the webpage given as\n parameter as a xml String."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef inject_quiet(levels):\n loggers = list(Logger.manager.loggerDict.items())\n loggers.append((\"root\", getLogger()))\n level_filter = LevelFilter(levels)\n\n for logger_name, logger in loggers:\n for handler in getattr(logger, \"handlers\", []):\n handler.addFilter(level_filter)", "response": "injects a level filter to all loggers and handlers"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconnects to the specified ftp server and returns the connector.", "response": "def connect(self):\n \"\"\"Logs into the specified ftp server and returns connector.\"\"\"\n for tried_connection_count in range(CFG_FTP_CONNECTION_ATTEMPTS):\n try:\n self.ftp = FtpHandler(self.config.OXFORD.URL,\n self.config.OXFORD.LOGIN,\n self.config.OXFORD.PASSWORD)\n self.logger.debug((\"Successful connection to the \"\n \"Oxford University Press server\"))\n return\n except socket_timeout_exception as err:\n self.logger.error(('Failed to connect %d of %d times. '\n 'Will sleep for %d seconds and try again.')\n % (tried_connection_count+1,\n CFG_FTP_CONNECTION_ATTEMPTS,\n CFG_FTP_TIMEOUT_SLEEP_DURATION))\n time.sleep(CFG_FTP_TIMEOUT_SLEEP_DURATION)\n except Exception as err:\n self.logger.error(('Failed to connect to the Oxford '\n 'University Press server. %s') % (err,))\n break\n\n raise LoginException(err)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _extract_packages(self):\n if not hasattr(self, \"retrieved_packages_unpacked\"):\n self.retrieved_packages_unpacked = [self.package_name]\n for path in self.retrieved_packages_unpacked:\n package_name = basename(path)\n self.path_unpacked = join(CFG_UNPACKED_FILES,\n package_name.split('.')[0])\n self.logger.debug(\"Extracting package: %s\"\n % (path.split(\"/\")[-1],))\n try:\n if \"_archival_pdf\" in self.path_unpacked:\n self.path_unpacked = (self.path_unpacked\n .rstrip(\"_archival_pdf\"))\n ZipFile(path).extractall(join(self.path_unpacked,\n \"archival_pdfs\"))\n else:\n ZipFile(path).extractall(self.path_unpacked)\n #TarFile.open(path).extractall(self.path_unpacked)\n except Exception:\n register_exception(alert_admin=True,\n prefix=\"OUP error extracting package.\")\n self.logger.error(\"Error extraction package file: %s\"\n % (path,))\n\n if hasattr(self, \"path_unpacked\"):\n return self.path_unpacked", "response": "Extracts a package in a new directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _crawl_oxford_and_find_main_xml(self):\n self.found_articles = []\n\n def visit(arg, dirname, names):\n files = [filename for filename in names if \".xml\" in filename]\n if files:\n try:\n for f in files:\n self.found_articles.append(join(dirname, f))\n\n except Exception as err:\n register_exception()\n print(\"ERROR: can't normalize %s: %s\" % (dirname, err),\n file=sys.stderr)\n\n if hasattr(self, 'path_unpacked'):\n walk(self.path_unpacked, visit, None)\n elif self.path:\n walk(self.path, visit, None)\n else:\n self.logger.info(\"Nothing to do.\")", "response": "Crawl the path and find main. xml in the current directory."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfetches the current instance s data from the NuHeat API and populate the instance s attributes.", "response": "def get_data(self):\n \"\"\"\n Fetch/refresh the current instance's data from the NuHeat API\n \"\"\"\n params = {\n \"serialnumber\": self.serial_number\n }\n data = self._session.request(config.THERMOSTAT_URL, params=params)\n\n self._data = data\n\n self.heating = data.get(\"Heating\")\n self.online = data.get(\"Online\")\n self.room = data.get(\"Room\")\n self.serial_number = data.get(\"SerialNumber\")\n self.temperature = data.get(\"Temperature\")\n self.min_temperature = data.get(\"MinTemp\")\n self.max_temperature = data.get(\"MaxTemp\")\n self.target_temperature = data.get(\"SetPointTemp\")\n self._schedule_mode = data.get(\"ScheduleMode\")"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the thermostat mode", "response": "def schedule_mode(self, mode):\n \"\"\"\n Set the thermostat mode\n\n :param mode: The desired mode integer value.\n Auto = 1\n Temporary hold = 2\n Permanent hold = 3\n \"\"\"\n modes = [config.SCHEDULE_RUN, config.SCHEDULE_TEMPORARY_HOLD, config.SCHEDULE_HOLD]\n if mode not in modes:\n raise Exception(\"Invalid mode. Please use one of: {}\".format(modes))\n\n self.set_data({\"ScheduleMode\": mode})"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_target_fahrenheit(self, fahrenheit, mode=config.SCHEDULE_HOLD):\n temperature = fahrenheit_to_nuheat(fahrenheit)\n self.set_target_temperature(temperature, mode)", "response": "Set the target temperature to the desired fahrenheit."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_target_celsius(self, celsius, mode=config.SCHEDULE_HOLD):\n temperature = celsius_to_nuheat(celsius)\n self.set_target_temperature(temperature, mode)", "response": "Set the target temperature to the desired celsius."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating the target temperature on the NuHeat API.", "response": "def set_target_temperature(self, temperature, mode=config.SCHEDULE_HOLD):\n \"\"\"\n Updates the target temperature on the NuHeat API\n\n :param temperature: The desired temperature in NuHeat format\n :param permanent: Permanently hold the temperature. If set to False, the schedule will\n resume at the next programmed event\n \"\"\"\n if temperature < self.min_temperature:\n temperature = self.min_temperature\n\n if temperature > self.max_temperature:\n temperature = self.max_temperature\n\n modes = [config.SCHEDULE_TEMPORARY_HOLD, config.SCHEDULE_HOLD]\n if mode not in modes:\n raise Exception(\"Invalid mode. Please use one of: {}\".format(modes))\n\n self.set_data({\n \"SetPointTemp\": temperature,\n \"ScheduleMode\": mode\n })"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_data(self, post_data):\n params = {\n \"serialnumber\": self.serial_number\n }\n self._session.request(config.THERMOSTAT_URL, method=\"POST\", data=post_data, params=params)", "response": "Update the current instance s data on the NuHeat API."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_config(filename=None, section_option_dict={}):\n\n config = ConfigParser()\n config.read(filename)\n\n working_dict = _prepare_working_dict(config, section_option_dict)\n\n tmp_dict = {}\n\n for section, options in working_dict.iteritems():\n tmp_dict[section] = {}\n for option in options:\n tmp_dict[section][option] = config.get(section, option)\n\n return Bunch(tmp_dict)", "response": "This function loads the config file and returns a Bunch object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef authenticate(self):\n if self._session_id:\n _LOGGER.debug(\"Using existing NuHeat session\")\n return\n\n _LOGGER.debug(\"Creating NuHeat session\")\n post_data = {\n \"Email\": self.username,\n \"Password\": self.password,\n \"application\": \"0\"\n }\n data = self.request(config.AUTH_URL, method=\"POST\", data=post_data)\n session_id = data.get(\"SessionId\")\n if not session_id:\n raise Exception(\"Authentication error\")\n\n self._session_id = session_id", "response": "Authenticate against the NuHeat API."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes a request to the NuHeat API.", "response": "def request(self, url, method=\"GET\", data=None, params=None, retry=True):\n \"\"\"\n Make a request to the NuHeat API\n\n :param url: The URL to request\n :param method: The type of request to make (GET, POST)\n :param data: Data to be sent along with POST requests\n :param params: Querystring parameters\n :param retry: Attempt to re-authenticate and retry request if necessary\n \"\"\"\n headers = config.REQUEST_HEADERS\n\n if params and self._session_id:\n params['sessionid'] = self._session_id\n\n if method == \"GET\":\n response = requests.get(url, headers=headers, params=params)\n elif method == \"POST\":\n response = requests.post(url, headers=headers, params=params, data=data)\n\n # Handle expired sessions\n if response.status_code == 401 and retry:\n _LOGGER.warn(\"NuHeat APIrequest unauthorized [401]. Try to re-authenticate.\")\n self._session_id = None\n self.authenticate()\n return self.request(url, method=method, data=data, params=params, retry=False)\n\n response.raise_for_status()\n try:\n return response.json()\n except ValueError:\n # No JSON object\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef handle_starttag(self, tag, attrs):\n if tag in self.mathml_elements:\n final_attr = \"\"\n for key, value in attrs:\n final_attr += ' {0}=\"{1}\"'.format(key, value)\n self.fed.append(\"<{0}{1}>\".format(tag, final_attr))", "response": "Return representation of html start tag and attributes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning representation of html end tag.", "response": "def handle_endtag(self, tag):\n \"\"\"Return representation of html end tag.\"\"\"\n if tag in self.mathml_elements:\n self.fed.append(\"\".format(tag))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef html_to_text(cls, html):\n s = cls()\n s.feed(html)\n unescaped_data = s.unescape(s.get_data())\n return escape_for_xml(unescaped_data, tags_to_keep=s.mathml_elements)", "response": "Return stripped HTML keeping only MathML."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_instance(self):\n ret = False\n val = self.callback\n if self.is_class(): return False\n\n ret = not inspect.isfunction(val) and not inspect.ismethod(val)\n# if is_py2:\n# ret = isinstance(val, types.InstanceType) or hasattr(val, '__dict__') \\\n# and not (hasattr(val, 'func_name') or hasattr(val, 'im_func'))\n# \n# else:\n# ret = not inspect.isfunction(val) and not inspect.ismethod(val)\n\n return ret", "response": "return True if callback is an instance of a class"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if callback is a vanilla plain jane function", "response": "def is_function(self):\n \"\"\"return True if callback is a vanilla plain jane function\"\"\"\n if self.is_instance() or self.is_class(): return False\n return isinstance(self.callback, (Callable, classmethod))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge_kwargs(self, kwargs):\n if kwargs:\n self.parser_kwargs.update(kwargs)\n\n #self.parser_kwargs['dest'] = self.name\n self.parser_kwargs.setdefault('dest', self.name)\n\n # special handling of any passed in values\n if 'default' in kwargs:\n # NOTE -- this doesn't use .set_default() because that is meant to\n # parse from the function definition so it actually has different syntax\n # than what the .set_default() method does. eg, @arg(\"--foo\", default=[1, 2]) means\n # that the default value should be an array with 1 and 2 in it, where main(foo=[1, 2])\n # means foo should be constrained to choices=[1, 2]\n self.parser_kwargs[\"default\"] = kwargs[\"default\"]\n self.parser_kwargs[\"required\"] = False\n\n elif 'action' in kwargs:\n if kwargs['action'] in set(['store_false', 'store_true']):\n self.parser_kwargs['required'] = False\n\n elif kwargs['action'] in set(['version']):\n self.parser_kwargs.pop('required', False)\n\n else:\n self.parser_kwargs.setdefault(\"required\", True)", "response": "merge kwargs into any kwargs that were automatically generated from the main function\n introspection"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmerge the args and kwargs of this instance into the instance", "response": "def merge_from_list(self, list_args):\n \"\"\"find any matching parser_args from list_args and merge them into this\n instance\n\n list_args -- list -- an array of (args, kwargs) tuples\n \"\"\"\n def xs(name, parser_args, list_args):\n \"\"\"build the generator of matching list_args\"\"\"\n for args, kwargs in list_args:\n if len(set(args) & parser_args) > 0:\n yield args, kwargs\n\n else:\n if 'dest' in kwargs:\n if kwargs['dest'] == name:\n yield args, kwargs\n\n for args, kwargs in xs(self.name, self.parser_args, list_args):\n self.merge_args(args)\n self.merge_kwargs(kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _fill_text(self, text, width, indent):\n lines = []\n for line in text.splitlines(False):\n if line:\n # https://docs.python.org/2/library/textwrap.html\n lines.extend(textwrap.wrap(\n line.strip(),\n width,\n initial_indent=indent,\n subsequent_indent=indent\n ))\n\n else:\n lines.append(line)\n\n text = \"\\n\".join(lines)\n return text", "response": "Overridden to not get rid of newlines\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_callback_args(self, raw_args):\n args = []\n arg_info = self.arg_info\n kwargs = dict(arg_info['optional'])\n\n parsed_args = []\n unknown_args = getattr(self, \"unknown_args\", False)\n if unknown_args:\n parsed_args, parsed_unknown_args = self.parse_known_args(raw_args)\n\n # TODO -- can this be moved to UnknownParser?\n\n # **kwargs have to be in --key=val form\n # http://stackoverflow.com/a/12807809/5006\n d = defaultdict(list)\n for k, v in ((k.lstrip('-'), v) for k,v in (a.split('=') for a in parsed_unknown_args)):\n d[k].append(v)\n\n for k in (k for k in d if len(d[k])==1):\n d[k] = d[k][0]\n\n kwargs.update(d)\n\n else:\n parsed_args = self.parse_args(raw_args)\n\n # http://parezcoydigo.wordpress.com/2012/08/04/from-argparse-to-dictionary-in-python-2-7/\n kwargs.update(vars(parsed_args))\n\n # because of how args works, we need to make sure the kwargs are put in correct\n # order to be passed to the function, otherwise our real *args won't make it\n # to the *args variable\n for k in arg_info['order']:\n args.append(kwargs.pop(k))\n\n # now that we have the correct order, tack the real *args on the end so they\n # get correctly placed into the function's *args variable\n if arg_info['args']:\n args.extend(kwargs.pop(arg_info['args']))\n\n return args, kwargs", "response": "This method parses all the arguments from the callback function and returns a list of the arguments that are passed to the callback function."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds all the arguments that are required by the parent parser", "response": "def find_args(self):\n \"\"\"when a new parser is created this is the method that is called from its\n __init__ method to find all the arguments\"\"\"\n arg_info = self.arg_info\n main = self.callback\n cbi = CallbackInspect(main)\n all_arg_names = set()\n decorator_args = cbi.args\n args, args_name, kwargs_name, args_defaults = cbi.argspec\n\n arg_info['order'] = args\n default_offset = len(args) - len(args_defaults)\n #pout.v(args, args_name, kwargs_name, args_defaults, default_offset)\n #pout.v(args, decorator_args)\n\n # build a list of potential *args, basically, if an arg_name matches exactly\n # then it is an *arg and we shouldn't mess with it in the function\n comp_args = set()\n for da in decorator_args:\n comp_args.update(da[0])\n\n for i, arg_name in enumerate(args):\n if arg_name in comp_args: continue\n\n a = ScriptKwarg(arg_name)\n\n # set the default if it is available\n default_i = i - default_offset\n if default_i >= 0:\n na = args_defaults[default_i]\n a.set_default(na)\n\n a.merge_from_list(decorator_args)\n\n if a.required:\n arg_info['required'].append(a.name)\n\n else:\n arg_info['optional'][a.name] = a.default\n\n #pout.v(a.parser_args, a.parser_kwargs)\n all_arg_names |= a.parser_args\n\n # if the callback arg is just a value, respect the parent parser's config\n if \"default\" not in a.parser_kwargs \\\n and \"action\" not in a.parser_kwargs \\\n and \"choices\" not in a.parser_kwargs:\n keys = self._option_string_actions.keys()\n found_arg = False\n for pa in a.parser_args:\n if pa in keys:\n found_arg = True\n break\n\n if not found_arg:\n self.add_argument(*a.parser_args, **a.parser_kwargs)\n\n else:\n # we want to override parent parser\n self.add_argument(*a.parser_args, **a.parser_kwargs)\n\n self.unknown_args = False\n if self.add_help:\n if args_name:\n a = ScriptArg(args_name, nargs='*')\n a.merge_from_list(decorator_args)\n all_arg_names |= a.parser_args\n self.add_argument(*a.parser_args, **a.parser_kwargs)\n arg_info['args'] = args_name\n\n if kwargs_name:\n self.unknown_args = True\n arg_info['kwargs'] = kwargs_name\n\n # pick up any stragglers\n for da, dkw in decorator_args:\n if da[0] not in all_arg_names:\n arg_name = da[0]\n if arg_name.startswith(\"-\"):\n a = ScriptKwarg(*da)\n else:\n a = ScriptArg(*da)\n\n a.merge_kwargs(dkw)\n self.add_argument(*a.parser_args, **a.parser_kwargs)\n\n self.arg_info = arg_info"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_user_agent(component=None):\n packageinfo = pkg_resources.require(\"harvestingkit\")[0]\n useragent = \"{0}/{1}\".format(packageinfo.project_name, packageinfo.version)\n if component is not None:\n useragent += \" {0}\".format(component)\n return useragent", "response": "create string suitable for HTTP User - Agent header"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef record_add_field(rec, tag, ind1='', ind2='', subfields=[],\n controlfield_value=''):\n \"\"\"Add a MARCXML datafield as a new child to a XML document.\"\"\"\n if controlfield_value:\n doc = etree.Element(\"controlfield\",\n attrib={\n \"tag\": tag,\n })\n doc.text = unicode(controlfield_value)\n else:\n doc = etree.Element(\"datafield\",\n attrib={\n \"tag\": tag,\n \"ind1\": ind1,\n \"ind2\": ind2,\n })\n for code, value in subfields:\n field = etree.SubElement(doc, \"subfield\", attrib={\"code\": code})\n field.text = value\n rec.append(doc)\n return rec", "response": "Add a MARCXML datafield as a new child to a XML document."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngive a document return XML prettified.", "response": "def record_xml_output(rec, pretty=True):\n \"\"\"Given a document, return XML prettified.\"\"\"\n from .html_utils import MathMLParser\n ret = etree.tostring(rec, xml_declaration=False)\n\n # Special MathML handling\n ret = re.sub(\"(<)(([\\/]?{0}))\".format(\"|[\\/]?\".join(MathMLParser.mathml_elements)), '<\\g<2>', ret)\n ret = re.sub(\">\", '>', ret)\n if pretty:\n # We are doing our own prettyfication as etree pretty_print is too insane.\n ret = ret.replace('', ' \\n')\n ret = re.sub(r'', r' \\n', ret)\n ret = ret.replace('', '\\n')\n ret = ret.replace('', 'record>\\n')\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntransforms & and < to XML valid & ; and < ; and < ; to XML valid & ; and < ; and < ;", "response": "def escape_for_xml(data, tags_to_keep=None):\n \"\"\"Transform & and < to XML valid & and <.\n\n Pass a list of tags as string to enable replacement of\n '<' globally but keep any XML tags in the list.\n \"\"\"\n data = re.sub(\"&\", \"&\", data)\n if tags_to_keep:\n data = re.sub(r\"(<)(?![\\/]?({0})\\b)\".format(\"|\".join(tags_to_keep)), '<', data)\n else:\n data = re.sub(\"<\", \"<\", data)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef collapse_initials(name):\n if len(name.split(\".\")) > 1:\n name = re.sub(r'([A-Z]\\.)[\\s\\-]+(?=[A-Z]\\.)', r'\\1', name)\n return name", "response": "Collapse initials in a name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fix_journal_name(journal, knowledge_base):\n if not journal:\n return '', ''\n if not knowledge_base:\n return journal, ''\n if len(journal) < 2:\n return journal, ''\n volume = ''\n if (journal[-1] <= 'Z' and journal[-1] >= 'A') \\\n and (journal[-2] == '.' or journal[-2] == ' '):\n volume += journal[-1]\n journal = journal[:-1]\n journal = journal.strip()\n\n if journal.upper() in knowledge_base:\n journal = knowledge_base[journal.upper()].strip()\n elif journal in knowledge_base:\n journal = knowledge_base[journal].strip()\n elif '.' in journal:\n journalnodots = journal.replace('. ', ' ')\n journalnodots = journalnodots.replace('.', ' ').strip().upper()\n if journalnodots in knowledge_base:\n journal = knowledge_base[journalnodots].strip()\n\n journal = journal.replace('. ', '.')\n return journal, volume", "response": "Convert journal name to Inspire s short form."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_nations_field(authors_subfields):\n from .config import NATIONS_DEFAULT_MAP\n result = []\n for field in authors_subfields:\n if field[0] == 'v':\n values = [x.replace('.', '') for x in field[1].split(', ')]\n possible_affs = filter(lambda x: x is not None,\n map(NATIONS_DEFAULT_MAP.get, values))\n if 'CERN' in possible_affs and 'Switzerland' in possible_affs:\n # Don't use remove in case of multiple Switzerlands\n possible_affs = [x for x in possible_affs\n if x != 'Switzerland']\n\n result.extend(possible_affs)\n\n result = sorted(list(set(result)))\n\n if result:\n authors_subfields.extend([('w', res) for res in result])\n else:\n authors_subfields.append(('w', 'HUMAN CHECK'))", "response": "Add correct nations field according to mapping in NATIONS_DEFAULT_MAP."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfix bad Unicode special dashes in string.", "response": "def fix_dashes(string):\n \"\"\"Fix bad Unicode special dashes in string.\"\"\"\n string = string.replace(u'\\u05BE', '-')\n string = string.replace(u'\\u1806', '-')\n string = string.replace(u'\\u2E3A', '-')\n string = string.replace(u'\\u2E3B', '-')\n string = unidecode(string)\n return re.sub(r'--+', '-', string)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntries to capitalize properly a title string.", "response": "def fix_title_capitalization(title):\n \"\"\"Try to capitalize properly a title string.\"\"\"\n if re.search(\"[A-Z]\", title) and re.search(\"[a-z]\", title):\n return title\n word_list = re.split(' +', title)\n final = [word_list[0].capitalize()]\n for word in word_list[1:]:\n if word.upper() in COMMON_ACRONYMS:\n final.append(word.upper())\n elif len(word) > 3:\n final.append(word.capitalize())\n else:\n final.append(word.lower())\n return \" \".join(final)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert_html_subscripts_to_latex(text):\n text = re.sub(\"(.*?)\", r\"$_{\\1}$\", text)\n text = re.sub(\"(.*?)\", r\"$^{\\1}$\", text)\n return text", "response": "Convert some HTML subscripts to latex equivalents."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndownloading a file from a URL.", "response": "def download_file(from_url, to_filename=None,\n chunk_size=1024 * 8, retry_count=3):\n \"\"\"Download URL to a file.\"\"\"\n if not to_filename:\n to_filename = get_temporary_file()\n\n session = requests.Session()\n adapter = requests.adapters.HTTPAdapter(max_retries=retry_count)\n session.mount(from_url, adapter)\n response = session.get(from_url, stream=True)\n with open(to_filename, 'wb') as fd:\n for chunk in response.iter_content(chunk_size):\n fd.write(chunk)\n return to_filename"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_shell_command(commands, **kwargs):\n p = subprocess.Popen(commands,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n **kwargs)\n output, error = p.communicate()\n return p.returncode, output, error", "response": "Run a shell command."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a logger object.", "response": "def create_logger(name,\n filename=None,\n logging_level=logging.DEBUG):\n \"\"\"Create a logger object.\"\"\"\n logger = logging.getLogger(name)\n formatter = logging.Formatter(('%(asctime)s - %(name)s - '\n '%(levelname)-8s - %(message)s'))\n\n if filename:\n fh = logging.FileHandler(filename=filename)\n fh.setFormatter(formatter)\n logger.addHandler(fh)\n\n ch = logging.StreamHandler()\n ch.setFormatter(formatter)\n logger.addHandler(ch)\n\n logger.setLevel(logging_level)\n\n return logger"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nunzip a file from a given filepath to a temporary directory.", "response": "def unzip(zipped_file, output_directory=None,\n prefix=\"harvestingkit_unzip_\", suffix=\"\"):\n \"\"\"Uncompress a zipped file from given filepath to an (optional) location.\n\n If no location is given, a temporary folder will be generated inside\n CFG_TMPDIR, prefixed with \"apsharvest_unzip_\".\n \"\"\"\n if not output_directory:\n # We create a temporary directory to extract our stuff in\n try:\n output_directory = mkdtemp(suffix=suffix,\n prefix=prefix)\n except Exception, e:\n try:\n os.removedirs(output_directory)\n except TypeError:\n pass\n raise e\n return _do_unzip(zipped_file, output_directory)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nperforms the actual uncompression.", "response": "def _do_unzip(zipped_file, output_directory):\n \"\"\"Perform the actual uncompression.\"\"\"\n z = zipfile.ZipFile(zipped_file)\n for path in z.namelist():\n relative_path = os.path.join(output_directory, path)\n dirname, dummy = os.path.split(relative_path)\n try:\n if relative_path.endswith(os.sep) and not os.path.exists(dirname):\n os.makedirs(relative_path)\n elif not os.path.exists(relative_path):\n dirname = os.path.join(output_directory, os.path.dirname(path))\n if os.path.dirname(path) and not os.path.exists(dirname):\n os.makedirs(dirname)\n fd = open(relative_path, \"w\")\n fd.write(z.read(path))\n fd.close()\n except IOError, e:\n raise e\n return output_directory"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlocate all files matching supplied filename pattern recursively.", "response": "def locate(pattern, root=os.curdir):\n \"\"\"Locate all files matching supplied filename pattern recursively.\"\"\"\n for path, dummy, files in os.walk(os.path.abspath(root)):\n for filename in fnmatch.filter(files, pattern):\n yield os.path.join(path, filename)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npunctuate author names properly.", "response": "def punctuate_authorname(an):\n \"\"\"Punctuate author names properly.\n\n Expects input in the form 'Bloggs, J K' and will return 'Bloggs, J. K.'.\n \"\"\"\n name = an.strip()\n parts = [x for x in name.split(',') if x != '']\n ret_str = ''\n for idx, part in enumerate(parts):\n subparts = part.strip().split(' ')\n for sidx, substr in enumerate(subparts):\n ret_str += substr\n if len(substr) == 1:\n ret_str += '.'\n if sidx < (len(subparts) - 1):\n ret_str += ' '\n if idx < (len(parts) - 1):\n ret_str += ', '\n return ret_str.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef convert_date_to_iso(value):\n date_formats = [\"%d %b %Y\", \"%Y/%m/%d\"]\n for dformat in date_formats:\n try:\n date = datetime.strptime(value, dformat)\n return date.strftime(\"%Y-%m-%d\")\n except ValueError:\n pass\n return value", "response": "Convert a date - value to the ISO date standard."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_date_from_iso_to_human(value):\n try:\n year, month, day = value.split(\"-\")\n except ValueError:\n # Not separated by \"-\". Space?\n try:\n year, month, day = value.split(\" \")\n except ValueError:\n # What gives? OK, lets just return as is\n return value\n\n try:\n date_object = datetime(int(year), int(month), int(day))\n except TypeError:\n return value\n return date_object.strftime(\"%d %b %Y\")", "response": "Convert a date - value to the ISO date standard for humans."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert list of images to PNG format.", "response": "def convert_images(image_list):\n \"\"\"Convert list of images to PNG format.\n\n @param: image_list ([string, string, ...]): the list of image files\n extracted from the tarball in step 1\n\n @return: image_list ([str, str, ...]): The list of image files when all\n have been converted to PNG format.\n \"\"\"\n png_output_contains = 'PNG image'\n ret_list = []\n for image_file in image_list:\n if os.path.isdir(image_file):\n continue\n\n dummy1, cmd_out, dummy2 = run_shell_command('file %s', (image_file,))\n if cmd_out.find(png_output_contains) > -1:\n ret_list.append(image_file)\n else:\n # we're just going to assume that ImageMagick can convert all\n # the image types that we may be faced with\n # for sure it can do EPS->PNG and JPG->PNG and PS->PNG\n # and PSTEX->PNG\n converted_image_file = get_converted_image_name(image_file)\n cmd_list = ['convert', image_file, converted_image_file]\n dummy1, cmd_out, cmd_err = run_shell_command(cmd_list)\n if cmd_err == '':\n ret_list.append(converted_image_file)\n else:\n raise Exception(cmd_err)\n return ret_list"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a safe and closed filepath.", "response": "def get_temporary_file(prefix=\"tmp_\",\n suffix=\"\",\n directory=None):\n \"\"\"Generate a safe and closed filepath.\"\"\"\n try:\n file_fd, filepath = mkstemp(prefix=prefix,\n suffix=suffix,\n dir=directory)\n os.close(file_fd)\n except IOError, e:\n try:\n os.remove(filepath)\n except Exception:\n pass\n raise e\n return filepath"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting letters from string only.", "response": "def return_letters_from_string(text):\n \"\"\"Get letters from string only.\"\"\"\n out = \"\"\n for letter in text:\n if letter.isalpha():\n out += letter\n return out"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if license is compatible with Open Access", "response": "def license_is_oa(license):\n \"\"\"Return True if license is compatible with Open Access\"\"\"\n for oal in OA_LICENSES:\n if re.search(oal, license):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nextracts a package in a new temporary directory.", "response": "def _extract_package(self):\n \"\"\"\n Extract a package in a new temporary directory.\n \"\"\"\n self.path = mkdtemp(prefix=\"scoap3_package_\", dir=CFG_TMPSHAREDDIR)\n self.logger.debug(\"Extracting package: %s\" % (self.package_name,))\n scoap3utils_extract_package(self.package_name, self.path, self.logger)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _crawl_elsevier_and_find_main_xml(self):\n self.found_articles = []\n if not self.path and not self.package_name:\n for doc in self.conn.found_articles:\n dirname = doc['xml'].rstrip('/main.xml')\n try:\n self._normalize_article_dir_with_dtd(dirname)\n self.found_articles.append(dirname)\n except Exception as err:\n register_exception()\n print(\"ERROR: can't normalize %s: %s\" % (dirname, err))\n else:\n def visit(dummy, dirname, names):\n if \"main.xml\" in names and \"main.pdf\" in names:\n try:\n self._normalize_article_dir_with_dtd(dirname)\n self.found_articles.append(dirname)\n except Exception as err:\n register_exception()\n print(\"ERROR: can't normalize %s: %s\" % (dirname, err))\n walk(self.path, visit, None)", "response": "Crawl the elsevier and find main. xml in the current directory."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncrawl the elsevier and find the issue. xml file.", "response": "def _crawl_elsevier_and_find_issue_xml(self):\n \"\"\"\n Information about the current volume, issue, etc. is available\n in a file called issue.xml that is available in a higher directory.\n \"\"\"\n self._found_issues = []\n if not self.path and not self.package_name:\n for issue in self.conn._get_issues():\n dirname = issue.rstrip('/issue.xml')\n try:\n self._normalize_issue_dir_with_dtd(dirname)\n self._found_issues.append(dirname)\n except Exception as err:\n register_exception()\n print(\"ERROR: can't normalize %s: %s\" % (dirname, err))\n else:\n def visit(dummy, dirname, names):\n if \"issue.xml\" in names:\n try:\n self._normalize_issue_dir_with_dtd(dirname)\n self._found_issues.append(dirname)\n except Exception as err:\n register_exception()\n print(\"ERROR: can't normalize %s: %s\"\n % (dirname, err))\n walk(self.path, visit, None)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _normalize_article_dir_with_dtd(self, path):\n if exists(join(path, 'resolved_main.xml')):\n return\n main_xml_content = open(join(path, 'main.xml')).read()\n arts = ['art501.dtd','art510.dtd','art520.dtd','art540.dtd']\n tmp_extracted = 0\n for art in arts:\n if art in main_xml_content:\n self._extract_correct_dtd_package(art.split('.')[0], path)\n tmp_extracted = 1\n\n if not tmp_extracted:\n message = \"It looks like the path \" + path\n message += \"does not contain an art501, art510, art520 or art540 in main.xml file\"\n self.logger.error(message)\n raise ValueError(message)\n command = [\"xmllint\", \"--format\", \"--loaddtd\",\n join(path, 'main.xml'),\n \"--output\", join(path, 'resolved_main.xml')]\n dummy, dummy, cmd_err = run_shell_command(command)\n if cmd_err:\n message = \"Error in cleaning %s: %s\" % (\n join(path, 'main.xml'), cmd_err)\n self.logger.error(message)\n raise ValueError(message)", "response": "This function checks if the directory contains a local DTD and if so checks if the main. xml file contains a valid DTD."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the best effort start_date.", "response": "def get_publication_date(self, xml_doc):\n \"\"\"Return the best effort start_date.\"\"\"\n start_date = get_value_in_tag(xml_doc, \"prism:coverDate\")\n if not start_date:\n start_date = get_value_in_tag(xml_doc, \"prism:coverDisplayDate\")\n if not start_date:\n start_date = get_value_in_tag(xml_doc, 'oa:openAccessEffective')\n if start_date:\n start_date = datetime.datetime.strptime(\n start_date, \"%Y-%m-%dT%H:%M:%SZ\"\n )\n return start_date.strftime(\"%Y-%m-%d\")\n import dateutil.parser\n #dateutil.parser.parse cant process dates like April-June 2016\n start_date = re.sub('([A-Z][a-z]+)[\\s\\-][A-Z][a-z]+ (\\d{4})', \n r'\\1 \\2', start_date)\n try:\n date = dateutil.parser.parse(start_date)\n except ValueError:\n return ''\n # Special case where we ignore the deduced day form dateutil\n # in case it was not given in the first place.\n if len(start_date.split(\" \")) == 3:\n return date.strftime(\"%Y-%m-%d\")\n else:\n return date.strftime(\"%Y-%m\")\n else:\n if len(start_date) is 8:\n start_date = time.strftime(\n '%Y-%m-%d', time.strptime(start_date, '%Y%m%d'))\n elif len(start_date) is 6:\n start_date = time.strftime(\n '%Y-%m', time.strptime(start_date, '%Y%m'))\n return start_date"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a record to MARCXML format.", "response": "def get_record(self, path=None, no_pdf=False,\n test=False, refextract_callback=None):\n \"\"\"Convert a record to MARCXML format.\n\n :param path: path to a record.\n :type path: string\n :param test: flag to determine if it is a test call.\n :type test: bool\n :param refextract_callback: callback to be used to extract\n unstructured references. It should\n return a marcxml formated string\n of the reference.\n :type refextract_callback: callable\n\n :returns: marcxml formated string.\n \"\"\"\n xml_doc = self.get_article(path)\n rec = create_record()\n title = self.get_title(xml_doc)\n if title:\n record_add_field(rec, '245', subfields=[('a', title)])\n (journal, dummy, volume, issue, first_page, last_page, year,\n start_date, doi) = self.get_publication_information(xml_doc, path)\n if not journal:\n journal = self.get_article_journal(xml_doc)\n if start_date:\n record_add_field(rec, '260', subfields=[('c', start_date),\n ('t', 'published')])\n else:\n record_add_field(\n rec, '260', subfields=[('c', time.strftime('%Y-%m-%d'))])\n if doi:\n record_add_field(rec, '024', ind1='7', subfields=[('a', doi),\n ('2', 'DOI')])\n license, license_url = self.get_license(xml_doc)\n if license and license_url:\n record_add_field(rec, '540', subfields=[('a', license),\n ('u', license_url)])\n elif license_url:\n record_add_field(rec, '540', subfields=[('u', license_url)])\n self.logger.info(\"Creating record: %s %s\" % (path, doi))\n authors = self.get_authors(xml_doc)\n first_author = True\n for author in authors:\n author_name = (author['surname'], author.get(\n 'given_name') or author.get('initials'))\n subfields = [('a', '%s, %s' % author_name)]\n if 'orcid' in author:\n subfields.append(('j', author['orcid']))\n if 'affiliation' in author:\n for aff in author[\"affiliation\"]:\n subfields.append(('v', aff))\n\n if self.extract_nations:\n add_nations_field(subfields)\n\n if author.get('email'):\n subfields.append(('m', author['email']))\n if first_author:\n record_add_field(rec, '100', subfields=subfields)\n first_author = False\n else:\n record_add_field(rec, '700', subfields=subfields)\n\n abstract = self.get_abstract(xml_doc)\n if abstract:\n record_add_field(rec, '520', subfields=[('a', abstract),\n ('9', 'Elsevier')])\n record_copyright = self.get_copyright(xml_doc)\n if record_copyright:\n record_add_field(rec, '542', subfields=[('f', record_copyright)])\n keywords = self.get_keywords(xml_doc)\n if self.CONSYN:\n for tag in xml_doc.getElementsByTagName('ce:collaboration'):\n collaboration = get_value_in_tag(tag, 'ce:text')\n if collaboration:\n record_add_field(rec, '710',\n subfields=[('g', collaboration)])\n\n # We add subjects also as author keywords\n subjects = xml_doc.getElementsByTagName('dct:subject')\n for subject in subjects:\n for listitem in subject.getElementsByTagName('rdf:li'):\n keyword = xml_to_text(listitem)\n if keyword not in keywords:\n keywords.append(keyword)\n for keyword in keywords:\n record_add_field(rec, '653', ind1='1',\n subfields=[('a', keyword),\n ('9', 'author')])\n journal, dummy = fix_journal_name(journal.strip(),\n self.journal_mappings)\n subfields = []\n doctype = self.get_doctype(xml_doc)\n try:\n page_count = int(last_page) - int(first_page) + 1\n record_add_field(rec, '300',\n subfields=[('a', str(page_count))])\n except ValueError: # do nothing\n pass\n if doctype == 'err':\n subfields.append(('m', 'Erratum'))\n elif doctype == 'add':\n subfields.append(('m', 'Addendum'))\n elif doctype == 'pub':\n subfields.append(('m', 'Publisher Note'))\n elif doctype == 'rev':\n record_add_field(rec, '980', subfields=[('a', 'Review')])\n if journal:\n subfields.append(('p', journal))\n if first_page and last_page:\n subfields.append(('c', '%s-%s' %\n (first_page, last_page)))\n elif first_page:\n subfields.append(('c', first_page))\n if volume:\n subfields.append(('v', volume))\n if year:\n subfields.append(('y', year))\n record_add_field(rec, '773', subfields=subfields)\n if not test:\n if license:\n url = 'http://www.sciencedirect.com/science/article/pii/'\\\n + path.split('/')[-1][:-4]\n record_add_field(rec, '856', ind1='4',\n subfields=[('u', url),\n ('y', 'Elsevier server')])\n record_add_field(rec, 'FFT', subfields=[('a', path),\n ('t', 'INSPIRE-PUBLIC'),\n ('d', 'Fulltext')])\n else:\n record_add_field(rec, 'FFT', subfields=[('a', path),\n ('t', 'Elsevier'),\n ('o', 'HIDDEN')])\n record_add_field(rec, '980', subfields=[('a', 'HEP')])\n record_add_field(rec, '980', subfields=[('a', 'Citeable')])\n record_add_field(rec, '980', subfields=[('a', 'Published')])\n self._add_references(xml_doc, rec, refextract_callback)\n else:\n licence = 'http://creativecommons.org/licenses/by/3.0/'\n record_add_field(rec,\n '540',\n subfields=[('a', 'CC-BY-3.0'), ('u', licence)])\n if keywords:\n for keyword in keywords:\n record_add_field(\n rec, '653', ind1='1', subfields=[('a', keyword),\n ('9', 'author')])\n\n pages = ''\n if first_page and last_page:\n pages = '{0}-{1}'.format(first_page, last_page)\n elif first_page:\n pages = first_page\n\n subfields = filter(lambda x: x[1] and x[1] != '-', [('p', journal),\n ('v', volume),\n ('n', issue),\n ('c', pages),\n ('y', year)])\n\n record_add_field(rec, '773', subfields=subfields)\n if not no_pdf:\n from invenio.search_engine import perform_request_search\n query = '0247_a:\"%s\" AND NOT 980:DELETED\"' % (doi,)\n prev_version = perform_request_search(p=query)\n\n old_pdf = False\n\n if prev_version:\n from invenio.bibdocfile import BibRecDocs\n prev_rec = BibRecDocs(prev_version[0])\n try:\n pdf_path = prev_rec.get_bibdoc('main')\n pdf_path = pdf_path.get_file(\n \".pdf;pdfa\", exact_docformat=True)\n pdf_path = pdf_path.fullpath\n old_pdf = True\n record_add_field(rec, 'FFT',\n subfields=[('a', pdf_path),\n ('n', 'main'),\n ('f', '.pdf;pdfa')])\n message = ('Leaving previously delivered PDF/A for: '\n + doi)\n self.logger.info(message)\n except:\n pass\n try:\n if exists(join(path, 'main_a-2b.pdf')):\n pdf_path = join(path, 'main_a-2b.pdf')\n record_add_field(rec, 'FFT',\n subfields=[('a', pdf_path),\n ('n', 'main'),\n ('f', '.pdf;pdfa')])\n self.logger.debug('Adding PDF/A to record: %s'\n % (doi,))\n elif exists(join(path, 'main.pdf')):\n pdf_path = join(path, 'main.pdf')\n record_add_field(rec, 'FFT', subfields=[('a', pdf_path)])\n else:\n if not old_pdf:\n message = \"Record \" + doi\n message += \" doesn't contain PDF file.\"\n self.logger.warning(message)\n raise MissingFFTError(message)\n except MissingFFTError:\n message = \"Elsevier paper: %s is missing PDF.\" % (doi,)\n register_exception(alert_admin=True, prefix=message)\n version = self.get_elsevier_version(find_package_name(path))\n record_add_field(rec, '583', subfields=[('l', version)])\n xml_path = join(path, 'main.xml')\n record_add_field(rec, 'FFT', subfields=[('a', xml_path)])\n record_add_field(rec, '980', subfields=[('a', 'SCOAP3'),\n ('b', 'Elsevier')])\n try:\n return record_xml_output(rec)\n except UnicodeDecodeError:\n message = \"Found a bad char in the file for the article \" + doi\n sys.stderr.write(message)\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nextract oembed resources from a block of text.", "response": "def extract_oembeds(text, args=None):\n \"\"\"\n Extract oembed resources from a block of text. Returns a list\n of dictionaries.\n\n Max width & height can be specified:\n {% for embed in block_of_text|extract_oembeds:\"400x300\" %}\n\n Resource type can be specified:\n {% for photo_embed in block_of_text|extract_oembeds:\"photo\" %}\n\n Or both:\n {% for embed in block_of_text|extract_oembeds:\"400x300xphoto\" %}\n \"\"\"\n resource_type = width = height = None\n if args:\n dimensions = args.lower().split('x')\n if len(dimensions) in (3, 1):\n resource_type = dimensions.pop()\n\n if len(dimensions) == 2:\n width, height = map(lambda x: int(x), dimensions)\n\n client = OEmbedConsumer()\n return client.extract(text, width, height, resource_type)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef strip_oembeds(text, args=None):\n resource_type = width = height = None\n if args:\n dimensions = args.lower().split('x')\n if len(dimensions) in (3, 1):\n resource_type = dimensions.pop()\n\n if len(dimensions) == 2:\n width, height = map(lambda x: int(x), dimensions)\n \n client = OEmbedConsumer()\n return mark_safe(client.strip(text, width, height, resource_type))", "response": "Strip all the embeds from a block of text."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef do_oembed(parser, token):\n args = token.split_contents()\n template_dir = None\n var_name = None\n if len(args) > 2:\n if len(args) == 3 and args[1] == 'in':\n template_dir = args[2]\n elif len(args) == 3 and args[1] == 'as':\n var_name = args[2]\n elif len(args) == 4 and args[2] == 'in':\n template_dir = args[3]\n elif len(args) == 4 and args[2] == 'as':\n var_name = args[3]\n elif len(args) == 6 and args[4] == 'as':\n template_dir = args[3]\n var_name = args[5]\n else:\n raise template.TemplateSyntaxError(\"OEmbed either takes a single \" \\\n \"(optional) argument: WIDTHxHEIGHT, where WIDTH and HEIGHT \" \\\n \"are positive integers, and or an optional 'in \" \\\n \" \\\"template_dir\\\"' argument set.\")\n if template_dir:\n if not (template_dir[0] == template_dir[-1] and template_dir[0] in ('\"', \"'\")):\n raise template.TemplateSyntaxError(\"template_dir must be quoted\")\n template_dir = template_dir[1:-1]\n\n if len(args) >= 2 and 'x' in args[1]:\n width, height = args[1].lower().split('x')\n if not width and height:\n raise template.TemplateSyntaxError(\"OEmbed's optional WIDTHxHEIGH\" \\\n \"T argument requires WIDTH and HEIGHT to be positive integers.\")\n else:\n width, height = None, None\n nodelist = parser.parse(('endoembed',))\n parser.delete_first_token()\n return OEmbedNode(nodelist, width, height, template_dir, var_name)", "response": "A function which parses the token and replaces any links between the two nodes with OEmbed - provided objects."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef do_autodiscover(parser, token):\n args = token.split_contents()\n if len(args) != 2:\n raise template.TemplateSyntaxError('%s takes an object as its parameter.' % args[0])\n else:\n obj = args[1]\n return OEmbedAutodiscoverNode(obj)", "response": "Generates a < ; link> ; tag with oembed autodiscovery bits for an object."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a link tag with oembed url scheme bits.", "response": "def do_url_scheme(parser, token):\n \"\"\"\n Generates a <link> tag with oembed autodiscovery bits.\n\n {% oembed_url_scheme %}\n \"\"\"\n args = token.split_contents()\n if len(args) != 1:\n raise template.TemplateSyntaxError('%s takes no parameters.' % args[0])\n return OEmbedURLSchemeNode()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parser(self):\n module = self.module\n\n subcommands = self.subcommands\n if subcommands:\n module_desc = inspect.getdoc(module)\n parser = Parser(description=module_desc, module=module)\n subparsers = parser.add_subparsers()\n\n for sc_name, callback in subcommands.items():\n sc_name = sc_name.replace(\"_\", \"-\")\n cb_desc = inspect.getdoc(callback)\n sc_parser = subparsers.add_parser(\n sc_name,\n callback=callback,\n help=cb_desc\n )\n\n else:\n parser = Parser(callback=self.callbacks[self.function_name], module=module)\n\n return parser", "response": "return the parser for the current name"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef module(self):\n # we have to guard this value because:\n # https://thingspython.wordpress.com/2010/09/27/another-super-wrinkle-raising-typeerror/\n if not hasattr(self, '_module'):\n if \"__main__\" in sys.modules:\n mod = sys.modules[\"__main__\"]\n path = self.normalize_path(mod.__file__)\n if os.path.splitext(path) == os.path.splitext(self.path):\n self._module = mod\n\n else:\n # http://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path\n self._module = imp.load_source('captain_script', self.path)\n #self._module = imp.load_source(self.module_name, self.path)\n\n return self._module", "response": "load the module so we can actually run the script s function"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef body(self):\n if not hasattr(self, '_body'):\n self._body = inspect.getsource(self.module)\n return self._body", "response": "get the contents of the script"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing and import the script and then run the script s main function", "response": "def run(self, raw_args):\n \"\"\"parse and import the script, and then run the script's main function\"\"\"\n parser = self.parser\n args, kwargs = parser.parse_callback_args(raw_args)\n\n callback = kwargs.pop(\"main_callback\")\n if parser.has_injected_quiet():\n levels = kwargs.pop(\"quiet_inject\", \"\")\n logging.inject_quiet(levels)\n\n try:\n ret_code = callback(*args, **kwargs)\n ret_code = int(ret_code) if ret_code else 0\n\n except ArgError as e:\n # https://hg.python.org/cpython/file/2.7/Lib/argparse.py#l2374\n echo.err(\"{}: error: {}\", parser.prog, str(e))\n ret_code = 2\n\n return ret_code"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef call_path(self, basepath):\n rel_filepath = self.path\n if basepath:\n rel_filepath = os.path.relpath(self.path, basepath)\n\n basename = self.name\n if basename in set(['__init__.py', '__main__.py']):\n rel_filepath = os.path.dirname(rel_filepath)\n\n return rel_filepath", "response": "return that path to be able to execute this script from the passed in\n basepath"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse(self):\n if self.parsed: return\n\n self.callbacks = {}\n\n # search for main and any main_* callable objects\n regex = re.compile(\"^{}_?\".format(self.function_name), flags=re.I)\n mains = set()\n body = self.body\n ast_tree = ast.parse(self.body, self.path)\n for n in ast_tree.body:\n if hasattr(n, 'name'):\n if regex.match(n.name):\n mains.add(n.name)\n\n if hasattr(n, 'value'):\n ns = n.value\n if hasattr(ns, 'id'):\n if regex.match(ns.id):\n mains.add(ns.id)\n\n if hasattr(n, 'targets'):\n ns = n.targets[0]\n if hasattr(ns, 'id'):\n if regex.match(ns.id):\n mains.add(ns.id)\n\n if hasattr(n, 'names'):\n for ns in n.names:\n if hasattr(ns, 'name'):\n if regex.match(ns.name):\n mains.add(ns.name)\n\n if getattr(ns, 'asname', None):\n if regex.match(ns.asname):\n mains.add(ns.asname)\n\n if len(mains) > 0:\n module = self.module\n for function_name in mains:\n cb = getattr(module, function_name, None)\n if cb and callable(cb):\n self.callbacks[function_name] = cb\n\n else:\n raise ParseError(\"no main function found\")\n\n self.parsed = True\n return len(self.callbacks) > 0", "response": "parse the script and set the parser and argument info"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef can_run_from_cli(self):\n ret = False\n ast_tree = ast.parse(self.body, self.path)\n calls = self._find_calls(ast_tree, __name__, \"exit\")\n for call in calls:\n if re.search(\"{}\\(\".format(re.escape(call)), self.body):\n ret = True\n break\n\n return ret", "response": "return True if this script can be run from the command line"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nregisters a field with the FieldRegistry and creating a post - save signal for the model.", "response": "def register_field(cls, field):\n \"\"\"\n Handles registering the fields with the FieldRegistry and creating a \n post-save signal for the model.\n \"\"\"\n FieldRegistry.add_field(cls, field)\n \n signals.post_save.connect(handle_save_embeds, sender=cls,\n dispatch_uid='%s.%s.%s' % \\\n (cls._meta.app_label, cls._meta.module_name, field.name))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef render_oembed(self, oembed_resource, original_url, template_dir=None,\n context=None):\n \"\"\"\n Render the oembed resource and return as a string.\n \n Template directory will always fall back to 'oembed/[type].html', but\n a custom template dir can be passed in using the kwargs.\n \n Templates are given two context variables:\n - response: an OEmbedResource\n - original_url: the url that was passed to the consumer\n \"\"\"\n provided_context = context or Context()\n context = RequestContext(context.get(\"request\") or mock_request())\n context.update(provided_context)\n \n # templates are named for the resources they display, i.e. video.html\n template_name = '%s.html' % oembed_resource.type\n \n # set up template finder to fall back to the link template\n templates = [os.path.join('oembed', template_name), 'oembed/link.html']\n \n # if there's a custom template dir, look there first\n if template_dir:\n templates.insert(0, os.path.join('oembed', template_dir, template_name))\n \n template = select_template(templates)\n \n context.push()\n context['response'] = oembed_resource\n context['original_url'] = original_url\n rendered = template.render(context)\n context.pop()\n \n return rendered.strip()", "response": "Render the oembed resource and return as a string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing a block of text and returns a dictionary of data.", "response": "def parse(self, text, maxwidth=None, maxheight=None, template_dir=None,\n context=None, urlize_all_links=CONSUMER_URLIZE_ALL):\n \"\"\"\n Scans a block of text, replacing anything matching a provider pattern\n with an OEmbed html snippet, if possible.\n \n Templates should be stored at oembed/{format}.html, so for example:\n \n oembed/video.html\n \n An optional template_dir can be provided, allowing for\n \n oembed/[template_dir]/video.html\n \n These templates are passed a context variable, ``response``, which is\n an OEmbedResource, as well as the ``original_url``\n \"\"\"\n context = context or Context()\n context['maxwidth'] = maxwidth\n context['maxheight'] = maxheight\n\n try:\n text = unicode(text)\n except UnicodeDecodeError:\n text = unicode(text.decode('utf-8'))\n \n return self.parse_data(text, maxwidth, maxheight, template_dir,\n context, urlize_all_links)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef size_to_nearest(width=None, height=None, allowed_sizes=OEMBED_ALLOWED_SIZES,\n force_fit=False):\n \"\"\"\n Generate some dimensions for resizing an object. This function DOES NOT handle\n scaling, it simply calculates maximums. These values should then be passed to\n the resize() method which will scale it and return the scaled width & height.\n \"\"\"\n minwidth, minheight = min(allowed_sizes)\n maxwidth, maxheight = max(allowed_sizes)\n\n if not width and not height:\n return maxwidth, maxheight\n\n if width:\n width = int(width) > minwidth and int(width) or minwidth\n elif force_fit:\n width = maxwidth\n\n if height:\n height = int(height) > minheight and int(height) or minheight\n elif force_fit:\n height = maxheight\n\n for size in sorted(allowed_sizes):\n if width and not height:\n if width >= size[0]:\n maxwidth = size[0]\n if force_fit:\n maxheight = size[1]\n else:\n break\n elif height and not width:\n if height >= size[1]:\n maxheight = size[1]\n if force_fit:\n maxwidth = size[0]\n else:\n break\n else:\n if force_fit:\n if (width >= size[0]) and (height >= size[1]):\n maxwidth, maxheight = size\n else:\n break\n else:\n if width >= size[0]:\n maxwidth = size[0]\n if height >= size[1]:\n maxheight = size[1]\n return maxwidth, maxheight", "response": "This function calculates the width and height of a node in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfetching a URL and return the headers and raw data.", "response": "def fetch_url(url, method='GET', user_agent='django-oembed', timeout=SOCKET_TIMEOUT):\n \"\"\"\n Fetch response headers and data from a URL, raising a generic exception\n for any kind of failure.\n \"\"\"\n sock = httplib2.Http(timeout=timeout)\n request_headers = {\n 'User-Agent': user_agent,\n 'Accept-Encoding': 'gzip'}\n try:\n headers, raw = sock.request(url, headers=request_headers, method=method)\n except:\n raise OEmbedHTTPException('Error fetching %s' % url)\n return headers, raw"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a url which may or may not be a relative url to a full url", "response": "def relative_to_full(url, example_url):\n \"\"\"\n Given a url which may or may not be a relative url, convert it to a full\n url path given another full url as an example\n \"\"\"\n if re.match('https?:\\/\\/', url):\n return url\n domain = get_domain(example_url)\n if domain:\n return '%s%s' % (domain, url)\n return url"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mock_request():\n current_site = Site.objects.get_current()\n request = HttpRequest()\n request.META['SERVER_NAME'] = current_site.domain\n return request", "response": "Generate a fake request object to allow oEmbeds to use context processors."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cleaned_sites():\n mappings = {}\n for site in Site.objects.all():\n # match the site domain, breaking it into several pieces\n match = re.match(r'(https?://)?(www[^\\.]*\\.)?([^/]+)', site.domain)\n \n if match is not None:\n http, www, domain = match.groups()\n \n # if the protocol is specified, use it, otherwise accept 80/443\n http_re = http or r'https?:\\/\\/'\n \n # whether or not there's a www (or www2 :x) allow it in the match\n www_re = r'(?:www[^\\.]*\\.)?'\n \n # build a regex of the permissive http re, the www re, and the domain\n domain_re = http_re + www_re + domain\n \n # now build a pretty string representation of the domain\n http = http or r'http://'\n www = www or ''\n normalized = http + www + domain\n \n mappings[site.pk] = (domain_re, site.name, normalized)\n return mappings", "response": "Create a list of tuples mapping domains from the sites table to their respective site names."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the Marc xml of the files in xaml_jp directory and returns a string with the marcxml formated version of the file.", "response": "def get_record(self, fileName, ref_extract_callback=None):\n \"\"\"\n Gets the Marc xml of the files in xaml_jp directory\n\n :param fileName: the name of the file to parse.\n :type fileName: string\n :param refextract_callback: callback to be used to extract\n unstructured references. It should\n return a marcxml formated string\n of the reference.\n :type refextract_callback: callable\n\n :returns: a string with the marc xml version of the file.\n \"\"\"\n self.document = parse(fileName)\n article_type = self._get_article_type()\n if article_type not in ['research-article',\n 'introduction',\n 'letter']:\n return ''\n rec = create_record()\n title, subtitle, notes = self._get_title()\n subfields = []\n if subtitle:\n subfields.append(('b', subtitle))\n if title:\n subfields.append(('a', title))\n record_add_field(rec, '245', subfields=subfields)\n subjects = self.document.getElementsByTagName('kwd')\n subjects = map(xml_to_text, subjects)\n for note_id in notes:\n note = self._get_note(note_id)\n if note:\n record_add_field(rec, '500', subfields=[('a', note)])\n for subject in subjects:\n record_add_field(rec, '650', ind1='1', ind2='7',\n subfields=[('2', 'EDPSciences'),\n ('a', subject)])\n keywords = self._get_keywords()\n for keyword in keywords:\n record_add_field(rec, '653', ind1='1', subfields=[('a', keyword),\n ('9', 'author')])\n journal, volume, issue, year, date, doi, page,\\\n fpage, lpage = self._get_publication_information()\n astronomy_journals = ['EAS Publ.Ser.', 'Astron.Astrophys.']\n if journal in astronomy_journals:\n record_add_field(rec, '650', ind1='1', ind2='7',\n subfields=[('2', 'INSPIRE'),\n ('a', 'Astrophysics')])\n if date:\n record_add_field(rec, '260', subfields=[('c', date),\n ('t', 'published')])\n if doi:\n record_add_field(rec, '024', ind1='7', subfields=[('a', doi),\n ('2', 'DOI')])\n abstract = self._get_abstract()\n abstract = self._format_abstract(abstract)\n if abstract:\n record_add_field(rec, '520', subfields=[('a', abstract),\n ('9', 'EDPSciences')])\n license, license_type, license_url = self._get_license()\n subfields = []\n if license:\n subfields.append(('a', license))\n if license_url:\n subfields.append(('u', license_url))\n if subfields:\n record_add_field(rec, '540', subfields=subfields)\n if license_type == 'open-access':\n self._attach_fulltext(rec, doi)\n number_of_pages = self._get_page_count()\n if number_of_pages:\n record_add_field(rec, '300', subfields=[('a', number_of_pages)])\n c_holder, c_year, c_statement = self._get_copyright()\n if c_holder and c_year:\n record_add_field(rec, '542', subfields=[('d', c_holder),\n ('g', c_year),\n ('e', 'Article')])\n elif c_statement:\n record_add_field(rec, '542', subfields=[('f', c_statement),\n ('e', 'Article')])\n subfields = []\n if journal:\n subfields.append(('p', journal))\n if issue:\n subfields.append(('n', issue))\n if volume:\n subfields.append(('v', volume))\n if fpage and lpage:\n subfields.append(('c', '%s-%s' % (fpage,\n lpage)))\n elif page:\n subfields.append(('c', page))\n if year:\n subfields.append(('y', year))\n record_add_field(rec, '773', subfields=subfields)\n record_add_field(rec, '980', subfields=[('a', 'HEP')])\n conference = ''\n for tag in self.document.getElementsByTagName('conference'):\n conference = xml_to_text(tag)\n if conference:\n record_add_field(rec, '980', subfields=[('a', 'ConferencePaper')])\n record_add_field(rec, '500', subfields=[('a', conference)])\n self._add_references(rec, ref_extract_callback)\n self._add_authors(rec)\n try:\n return record_xml_output(rec)\n except UnicodeDecodeError:\n message = \"Found a bad char in the file for the article \" + doi\n sys.stderr.write(message)\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the Marc xml of the files in xaml_rich directory.", "response": "def get_record_rich(self, filename, ref_extract_callback=None):\n \"\"\"\n Gets the Marc xml of the files in xaml_rich directory\n\n :param fileName: the name of the file to parse.\n :type fileName: string\n\n :returns: a string with the marc xml version of the file.\n \"\"\"\n self.document = parse(filename)\n rec = create_record()\n articles = self.document.getElementsByTagName('ArticleID')\n for article in articles:\n article_type = article.getAttribute('Type')\n if not article_type == 'Article':\n return ''\n doi = get_value_in_tag(self.document, 'DOI')\n date = ''\n for tag in self.document.getElementsByTagName('Accepted'):\n year = get_value_in_tag(tag, 'Year')\n month = get_value_in_tag(tag, 'Month').zfill(2)\n day = get_value_in_tag(tag, 'Day').zfill(2)\n date = \"%s-%s-%s\" % (year, month, day)\n if not date:\n for tag in self.document.getElementsByTagName('OnlineDate'):\n year = get_value_in_tag(tag, 'Year')\n month = get_value_in_tag(tag, 'Month').zfill(2)\n day = get_value_in_tag(tag, 'Day').zfill(2)\n date = \"%s-%s-%s\" % (year, month, day)\n first_page = get_value_in_tag(article, 'FirstPage')\n last_page = get_value_in_tag(article, 'LastPage')\n subjects = article.getElementsByTagName('Keyword')\n subjects = map(xml_to_text, subjects)\n subject = ', '.join(subjects)\n copyright_statement = get_value_in_tag(article, 'Copyright')\n journal = get_value_in_tag(self.document, 'JournalTitle')\n journal, volume = fix_journal_name(journal, self.journal_mappings)\n issues = self.document.getElementsByTagName('IssueID')\n for issue in issues:\n volume += get_value_in_tag(issue, 'Volume')\n year = get_value_in_tag(issue, 'Year')\n title = get_value_in_tag(self.document, 'Title')\n authors = self.document.getElementsByTagName('Author')\n affiliations = self.document.getElementsByTagName('Affiliation')\n\n def affiliation_pair(a):\n return a.getAttribute('ID'), get_value_in_tag(\n a, 'UnstructuredAffiliation'\n )\n\n affiliations = map(affiliation_pair, affiliations)\n affiliations = dict(affiliations)\n\n def author_pair(a):\n surname = get_value_in_tag(a, 'LastName')\n first_name = get_value_in_tag(a, 'FirstName')\n middle_name = get_value_in_tag(a, 'MiddleName')\n if middle_name:\n name = '%s, %s %s' % (surname, first_name, middle_name)\n else:\n name = '%s, %s' % (surname, first_name)\n try:\n affid = a.getElementsByTagName(\n 'AffiliationID'\n )[0].getAttribute('Label')\n affiliation = affiliations[affid]\n except IndexError:\n affiliation = ''\n except KeyError:\n affiliation = ''\n return name, affiliation\n\n authors = map(author_pair, authors)\n abstract = get_value_in_tag(self.document, 'Abstract')\n references = self.document.getElementsByTagName('Bibliomixed')\n\n for reference in references:\n subfields = []\n label = reference.getAttribute('N')\n if label:\n subfields.append(('o', label))\n bibliosets = reference.getElementsByTagName('Biblioset')\n for tag in bibliosets:\n ref_year = get_value_in_tag(tag, 'Date')\n ref_journal = get_value_in_tag(tag, 'JournalShortTitle')\n ref_journal, ref_volume = fix_journal_name(\n ref_journal, self.journal_mappings\n )\n ref_volume += get_value_in_tag(tag, 'Volume')\n ref_page = get_value_in_tag(tag, 'ArtPageNums')\n if ref_year:\n subfields.append(('y', ref_year))\n if ref_journal and ref_volume and ref_page:\n subfields.append(('s', '%s,%s,%s' % (ref_journal,\n ref_volume,\n ref_page)))\n reference.removeChild(tag)\n text_ref = xml_to_text(reference)\n if ref_extract_callback:\n ref_xml = ref_extract_callback(text_ref)\n dom = parseString(ref_xml)\n fields = dom.getElementsByTagName(\"datafield\")[0]\n fields = fields.getElementsByTagName(\"subfield\")\n if fields:\n subfields.append(('9', 'refextract'))\n for field in fields:\n data = field.firstChild.data\n code = field.getAttribute(\"code\")\n if code == 'm' and bibliosets:\n continue\n else:\n subfields.append((code, data))\n else:\n subfields.append(('m', text_ref))\n if subfields:\n record_add_field(rec, '999', ind1='C', ind2='5',\n subfields=subfields)\n\n if title:\n record_add_field(rec, '245', subfields=[('a', title)])\n if date:\n record_add_field(rec, '260', subfields=[('c', date),\n ('t', 'published')])\n if doi:\n record_add_field(rec, '024', ind1='7', subfields=[('a', doi),\n ('2', 'DOI')])\n if abstract:\n record_add_field(rec, '520', subfields=[('a', abstract),\n ('9', 'EDPSciences')])\n first_author = True\n for author in authors:\n if first_author:\n subfields = [('a', author[0])]\n if author[1]:\n subfields.append(('v', author[1]))\n record_add_field(rec, '100', subfields=subfields)\n first_author = False\n else:\n subfields = [('a', author[0])]\n if author[1]:\n subfields.append(('v', author[1]))\n record_add_field(rec, '700', subfields=subfields)\n subfields = []\n if journal and volume and first_page:\n subfields.append(('s', \"%s,%s,%s\" % (journal,\n volume,\n first_page)))\n if first_page and last_page:\n try:\n nuber_of_pages = int(last_page) - int(first_page)\n record_add_field(rec, '300',\n subfields=[('a', str(nuber_of_pages))])\n except ValueError:\n pass\n subfields.append(('c', '%s-%s' % (first_page,\n last_page)))\n if year:\n subfields.append(('y', year))\n record_add_field(rec, '773', subfields=subfields)\n record_add_field(rec, '980', subfields=[('a', 'HEP')])\n if copyright_statement:\n record_add_field(rec, '542',\n subfields=[('f', copyright_statement)])\n if subject:\n record_add_field(rec, '650', ind1='1', ind2='7',\n subfields=[('2', 'EDPSciences'),\n ('a', subject)])\n try:\n return record_xml_output(rec)\n except UnicodeDecodeError:\n message = \"Found a bad char in the file for the article \" + doi\n sys.stderr.write(message)\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\noverride the base get_record method to add additional fields to the record.", "response": "def get_record(self):\n \"\"\"Override the base get_record.\"\"\"\n self.update_system_numbers()\n self.add_systemnumber(\"CDS\")\n self.fields_list = [\n \"024\", \"041\", \"035\", \"037\", \"088\", \"100\",\n \"110\", \"111\", \"242\", \"245\", \"246\", \"260\",\n \"269\", \"300\", \"502\", \"650\", \"653\", \"693\",\n \"700\", \"710\", \"773\", \"856\", \"520\", \"500\",\n \"980\"\n ]\n self.keep_only_fields()\n\n self.determine_collections()\n self.add_cms_link()\n self.update_languages()\n self.update_reportnumbers()\n self.update_date()\n self.update_pagenumber()\n self.update_authors()\n self.update_subject_categories(\"SzGeCERN\", \"INSPIRE\", \"categories_inspire\")\n self.update_keywords()\n self.update_experiments()\n self.update_collaboration()\n self.update_journals()\n self.update_links_and_ffts()\n\n if 'THESIS' in self.collections:\n self.update_thesis_supervisors()\n self.update_thesis_information()\n\n if 'NOTE' in self.collections:\n self.add_notes()\n\n for collection in self.collections:\n record_add_field(self.record,\n tag='980',\n subfields=[('a', collection)])\n self.remove_controlfields()\n return self.record"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntrying to determine which collections this record belongs to.", "response": "def determine_collections(self):\n \"\"\"Try to determine which collections this record should belong to.\"\"\"\n for value in record_get_field_values(self.record, '980', code='a'):\n if 'NOTE' in value.upper():\n self.collections.add('NOTE')\n if 'THESIS' in value.upper():\n self.collections.add('THESIS')\n if 'CONFERENCEPAPER' in value.upper():\n self.collections.add('ConferencePaper')\n if \"HIDDEN\" in value.upper():\n self.hidden = True\n\n if self.is_published():\n self.collections.add(\"PUBLISHED\")\n self.collections.add(\"CITEABLE\")\n\n if 'NOTE' not in self.collections:\n from itertools import product\n # TODO: Move this to a KB\n kb = ['ATLAS-CONF-', 'CMS-PAS-', 'ATL-', 'CMS-DP-',\n 'ALICE-INT-', 'LHCb-PUB-']\n values = record_get_field_values(self.record, \"088\", code='a')\n for val, rep in product(values, kb):\n if val.startswith(rep):\n self.collections.add('NOTE')\n break\n\n # 980 Arxiv tag\n if record_get_field_values(self.record, '035',\n filter_subfield_code=\"a\",\n filter_subfield_value=\"arXiv\"):\n self.collections.add(\"arXiv\")\n\n # 980 HEP && CORE\n self.collections.add('HEP')\n self.collections.add('CORE')\n\n # 980 Conference Note\n if 'ConferencePaper' not in self.collections:\n for value in record_get_field_values(self.record,\n tag='962',\n code='n'):\n if value[-2:].isdigit():\n self.collections.add('ConferencePaper')\n break\n # Clear out any existing ones.\n record_delete_fields(self.record, \"980\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if the record has already been published.", "response": "def is_published(self):\n \"\"\"Check fields 980 and 773 to see if the record has already been published.\n\n :return: True is published, else False\n \"\"\"\n field980 = record_get_field_instances(self.record, '980')\n field773 = record_get_field_instances(self.record, '773')\n for f980 in field980:\n if 'a' in field_get_subfields(f980):\n for f773 in field773:\n if 'p' in field_get_subfields(f773):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_system_numbers(self):\n scn_035_fields = record_get_field_instances(self.record, '035')\n forbidden_values = [\"cercer\",\n \"inspire\",\n \"xx\",\n \"cern annual report\",\n \"cmscms\",\n \"wai01\"]\n for field in scn_035_fields:\n subs = field_get_subfields(field)\n if '9' in subs:\n if 'a' not in subs:\n continue\n for sub in subs['9']:\n if sub.lower() in forbidden_values:\n break\n else:\n # No forbidden values (We did not \"break\")\n suffixes = [s.lower() for s in subs['9']]\n if 'spires' in suffixes:\n new_subs = [('a', 'SPIRES-%s' % subs['a'][0])]\n record_add_field(\n self.record, '970', subfields=new_subs)\n continue\n if 'a' in subs:\n for sub in subs['a']:\n if sub.lower() in forbidden_values:\n record_delete_field(self.record, tag=\"035\",\n field_position_global=field[4])", "response": "Update the system numbers of the 3035 records."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_pagenumber(self):\n for field in record_get_field_instances(self.record, '300'):\n for idx, (key, value) in enumerate(field[0]):\n if key == 'a':\n if \"mult.\" not in value and value != \" p\":\n field[0][idx] = ('a', re.sub(r'[^\\d-]+', '', value))\n else:\n record_delete_field(self.record, '300',\n field_position_global=field[4])\n break", "response": "Update the page number of the resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_authors(self):\n author_names = record_get_field_instances(self.record, '100')\n author_names.extend(record_get_field_instances(self.record, '700'))\n for field in author_names:\n subs = field_get_subfields(field)\n if 'i' not in subs or 'XX' in subs['i']:\n if 'j' not in subs or 'YY' in subs['j']:\n for idx, (key, value) in enumerate(field[0]):\n if key == 'a':\n field[0][idx] = ('a', punctuate_authorname(value))", "response": "Update the author names of the record."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_thesis_information(self):\n fields_501 = record_get_field_instances(self.record, '502')\n for idx, field in enumerate(fields_501):\n new_subs = []\n for key, value in field[0]:\n if key == 'a':\n new_subs.append(('b', value))\n elif key == 'b':\n new_subs.append(('c', value))\n elif key == 'c':\n new_subs.append(('d', value))\n else:\n new_subs.append((key, value))\n fields_501[idx] = field_swap_subfields(field, new_subs)", "response": "501 degree info - move subfields."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates 653 Free Keywords.", "response": "def update_keywords(self):\n \"\"\"653 Free Keywords.\"\"\"\n for field in record_get_field_instances(self.record, '653', ind1='1'):\n subs = field_get_subfields(field)\n new_subs = []\n if 'a' in subs:\n for val in subs['a']:\n new_subs.extend([('9', 'author'), ('a', val)])\n new_field = create_field(subfields=new_subs, ind1='1')\n record_replace_field(\n self.record, '653', new_field, field_position_global=field[4])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_experiments(self):\n # 693 Remove if 'not applicable'\n for field in record_get_field_instances(self.record, '693'):\n subs = field_get_subfields(field)\n all_subs = subs.get('a', []) + subs.get('e', [])\n if 'not applicable' in [x.lower() for x in all_subs]:\n record_delete_field(self.record, '693',\n field_position_global=field[4])\n new_subs = []\n experiment_a = \"\"\n experiment_e = \"\"\n for (key, value) in subs.iteritems():\n if key == 'a':\n experiment_a = value[0]\n new_subs.append((key, value[0]))\n elif key == 'e':\n experiment_e = value[0]\n experiment = \"%s---%s\" % (experiment_a.replace(\" \", \"-\"),\n experiment_e)\n translated_experiments = self.get_config_item(experiment,\n \"experiments\")\n new_subs.append((\"e\", translated_experiments))\n record_delete_field(self.record, tag=\"693\",\n field_position_global=field[4])\n record_add_field(self.record, \"693\", subfields=new_subs)", "response": "Update the experiments of the user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_collaboration(self):\n for field in record_get_field_instances(self.record, '710'):\n subs = field_get_subfield_instances(field)\n for idx, (key, value) in enumerate(subs[:]):\n if key == '5':\n subs.pop(idx)\n elif value.startswith('CERN. Geneva'):\n subs.pop(idx)\n if len(subs) == 0:\n record_delete_field(self.record,\n tag='710',\n field_position_global=field[4])", "response": "Update the collaboration of the 710 record."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_journals(self):\n for field in record_get_field_instances(self.record, '773'):\n subs = field_get_subfield_instances(field)\n new_subs = []\n for idx, (key, value) in enumerate(subs):\n if key == 'p':\n journal_name = self.get_config_item(value, \"journals\", allow_substring=False)\n journal_name = journal_name.replace('. ', '.').strip()\n new_subs.append((key, journal_name))\n else:\n new_subs.append((key, value))\n record_delete_field(self.record, tag=\"773\",\n field_position_global=field[4])\n record_add_field(self.record, \"773\", subfields=new_subs)", "response": "Update the journals field of the resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_links_and_ffts(self):\n figure_counter = 0\n for field in record_get_field_instances(self.record,\n tag='856',\n ind1='4'):\n subs = field_get_subfields(field)\n\n newsubs = []\n remove = False\n\n if 'z' in subs:\n is_figure = [s for s in subs['z'] if \"figure\" in s.lower()]\n if is_figure and 'u' in subs:\n is_subformat = [\n s for s in subs['u'] if \"subformat\" in s.lower()]\n if not is_subformat:\n url = subs['u'][0]\n if url.endswith(\".pdf\"):\n # We try to convert\n fd, local_url = mkstemp(suffix=os.path.basename(url))\n os.close(fd)\n self.logger.info(\n \"Downloading %s into %s\" % (url, local_url))\n plotfile = \"\"\n try:\n plotfile = download_file(url=url,\n download_to_file=local_url)\n except Exception as e:\n self.logger.exception(e)\n remove = True\n if plotfile:\n converted = convert_images([plotfile])\n if converted:\n url = converted.pop()\n msg = \"Successfully converted %s to %s\" \\\n % (local_url, url)\n self.logger.info(msg)\n else:\n msg = \"Conversion failed on %s\" \\\n % (local_url,)\n self.logger.error(msg)\n url = None\n remove = True\n if url:\n newsubs.append(('a', url))\n newsubs.append(('t', 'Plot'))\n figure_counter += 1\n if 'y' in subs:\n newsubs.append(\n ('d', \"%05d %s\" % (figure_counter, subs['y'][0])))\n newsubs.append(('n', subs['y'][0]))\n else:\n # Get basename without extension.\n name = os.path.basename(\n os.path.splitext(subs['u'][0])[0])\n newsubs.append(\n ('d', \"%05d %s\" % (figure_counter, name)))\n newsubs.append(('n', name))\n\n if not newsubs and 'u' in subs:\n is_fulltext = [s for s in subs['u'] if \".pdf\" in s]\n if is_fulltext:\n newsubs = [('t', 'INSPIRE-PUBLIC'), ('a', subs['u'][0])]\n\n if not newsubs and 'u' in subs:\n remove = True\n is_zipfile = [s for s in subs['u'] if \".zip\" in s]\n if is_zipfile:\n url = is_zipfile[0]\n\n local_url = os.path.join(self.get_local_folder(), os.path.basename(url))\n self.logger.info(\"Downloading %s into %s\" %\n (url, local_url))\n zipped_archive = \"\"\n try:\n zipped_archive = download_file(url=is_zipfile[0],\n download_to_file=local_url)\n except Exception as e:\n self.logger.exception(e)\n remove = True\n if zipped_archive:\n unzipped_archive = unzip(zipped_archive)\n list_of_pngs = locate(\"*.png\", unzipped_archive)\n for png in list_of_pngs:\n if \"_vti_\" in png or \"__MACOSX\" in png:\n continue\n figure_counter += 1\n plotsubs = []\n plotsubs.append(('a', png))\n caption = '%05d %s' % (\n figure_counter, os.path.basename(png))\n plotsubs.append(('d', caption))\n plotsubs.append(('t', 'Plot'))\n record_add_field(\n self.record, 'FFT', subfields=plotsubs)\n\n if not remove and not newsubs and 'u' in subs:\n urls = ('http://cdsweb.cern.ch', 'http://cms.cern.ch',\n 'http://cmsdoc.cern.ch', 'http://documents.cern.ch',\n 'http://preprints.cern.ch', 'http://cds.cern.ch')\n for val in subs['u']:\n if any(url in val for url in urls):\n remove = True\n break\n if val.endswith('ps.gz'):\n remove = True\n\n if newsubs:\n record_add_field(self.record, 'FFT', subfields=newsubs)\n remove = True\n\n if remove:\n record_delete_field(self.record, '856', ind1='4',\n field_position_global=field[4])", "response": "Update links and ffts for the record."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _extract_packages(self):\n self.path_unpacked = []\n if not hasattr(self, \"retrieved_packages_unpacked\"):\n self.retrieved_packages_unpacked = [self.package_name]\n for path in self.retrieved_packages_unpacked:\n self.logger.debug(\"Extracting package: %s\" % (path,))\n\n p_name = 'EPJC' if 'EPJC' in path else 'JHEP'\n p_message = 'scoap3_package_%s_%s_' % (p_name, datetime.now())\n\n self.path_unpacked.append(mkdtemp(prefix=p_message,\n dir=CFG_TMPSHAREDDIR))\n\n try:\n ZipFile(path).extractall(self.path_unpacked[-1])\n except Exception:\n register_exception(alert_admin=True,\n prefix=\"Springer error extracting package.\")\n self.logger.error(\"Error extraction package file: %s\"\n % (path,))\n\n return self.path_unpacked", "response": "Extracts a package in a new directory."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _crawl_springer_and_find_main_xml(self):\n self.found_articles = []\n\n def visit(arg, dirname, names):\n files = [filename for filename in names if \"nlm.xml\" in filename]\n if not files:\n files = [filename for filename in names\n if \".xml.scoap\" in filename]\n if files:\n try:\n # self._normalize_article_dir_with_dtd(dirname)\n self.found_articles.append(dirname)\n except Exception as err:\n register_exception()\n print \"ERROR: can't normalize %s: %s\" % (dirname, err)\n\n if hasattr(self, 'path_unpacked'):\n for path in self.path_unpacked:\n walk(path, visit, None)\n elif self.path:\n walk(self.path, visit, None)\n else:\n self.logger.info(\"Nothing to do.\")", "response": "Crawl the springer and find main. xml in a given directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _normalize_article_dir_with_dtd(self, path):\n files = [filename for filename in listdir(path)\n if \"nlm.xml\" in filename]\n if not files:\n files = [filename for filename in listdir(path)\n if \".xml.scoap\" in filename]\n if exists(join(path, 'resolved_main.xml')):\n return\n\n if 'JATS-archivearticle1.dtd' in open(join(path, files[0])).read():\n path_normalized = mkdtemp(prefix=\"scoap3_normalized_jats_\",\n dir=CFG_TMPSHAREDDIR)\n ZipFile(CFG_SPRINGER_JATS_PATH).extractall(path_normalized)\n elif 'A++V2.4.dtd' in open(join(path, files[0])).read():\n path_normalized = mkdtemp(prefix=\"scoap3_normalized_app_\",\n dir=CFG_TMPSHAREDDIR)\n ZipFile(CFG_SPRINGER_AV24_PATH).extractall(path_normalized)\n else:\n error_msg = (\"It looks like the path %s does not contain an \"\n \"JATS-archivearticle1.dtd nor A++V2.4.dtd XML file.\")\n self.logger.error(error_msg % path)\n raise ValueError(error_msg % path)\n print \"Normalizing %s\" % (files[0],)\n (cmd_exit_code,\n cmd_out,\n cmd_err) = run_shell_command((\"xmllint --format \"\n \"--loaddtd %s --output %s\"),\n (join(path, files[0]),\n join(path_normalized,\n 'resolved_main.xml')))\n if cmd_err:\n error_msg = \"Error in cleaning %s: %s\"\n self.logger.error(error_msg % (join(path, 'issue.xml'), cmd_err))\n raise ValueError(error_msg % (join(path, 'main.xml'), cmd_err))\n self.articles_normalized.append(path_normalized)", "response": "This function will take the path of the JATS archive article and check if it contains a DTD and if it does then install it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_all_filters(pelican):\n pelican.env.filters.update({'datetime': filters.datetime})\n pelican.env.filters.update({'article_date': filters.article_date})\n pelican.env.filters.update({'breaking_spaces': filters.breaking_spaces})\n pelican.env.filters.update({'titlecase': filters.titlecase})", "response": "Add all filters to Pelican."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_field(subfields=None, ind1=' ', ind2=' ', controlfield_value='',\n global_position=-1):\n \"\"\"\n Return a field created with the provided elements.\n\n Global position is set arbitrary to -1.\n \"\"\"\n if subfields is None:\n subfields = []\n\n ind1, ind2 = _wash_indicators(ind1, ind2)\n field = (subfields, ind1, ind2, controlfield_value, global_position)\n _check_field_validity(field)\n return field", "response": "Create a new field with the provided elements."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_records(marcxml, verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,\n correct=CFG_BIBRECORD_DEFAULT_CORRECT, parser='',\n keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):\n \"\"\"\n Create a list of records from the marcxml description.\n\n :returns: a list of objects initiated by the function create_record().\n Please see that function's docstring.\n \"\"\"\n # Use the DOTALL flag to include newlines.\n regex = re.compile('.*?', re.DOTALL)\n record_xmls = regex.findall(marcxml)\n\n return [create_record(record_xml, verbose=verbose, correct=correct,\n parser=parser, keep_singletons=keep_singletons)\n for record_xml in record_xmls]", "response": "Create a list of records from the marcxml description."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a record object from the marcxml description.", "response": "def create_record(marcxml=None, verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,\n correct=CFG_BIBRECORD_DEFAULT_CORRECT, parser='',\n sort_fields_by_indicators=False,\n keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):\n \"\"\"Create a record object from the marcxml description.\n\n Uses the lxml parser.\n\n The returned object is a tuple (record, status_code, list_of_errors),\n where status_code is 0 when there are errors, 1 when no errors.\n\n The return record structure is as follows::\n\n Record := {tag : [Field]}\n Field := (Subfields, ind1, ind2, value)\n Subfields := [(code, value)]\n\n .. code-block:: none\n\n .--------.\n | record |\n '---+----'\n |\n .------------------------+------------------------------------.\n |record['001'] |record['909'] |record['520'] |\n | | | |\n [list of fields] [list of fields] [list of fields] ...\n | | |\n | .--------+--+-----------. |\n | | | | |\n |[0] |[0] |[1] ... |[0]\n .----+------. .-----+-----. .--+--------. .---+-------.\n | Field 001 | | Field 909 | | Field 909 | | Field 520 |\n '-----------' '-----+-----' '--+--------' '---+-------'\n | | | |\n ... | ... ...\n |\n .----------+-+--------+------------.\n | | | |\n |[0] |[1] |[2] |\n [list of subfields] 'C' '4' ...\n |\n .----+---------------+------------------------+\n | | |\n ('a', 'value') | ('a', 'value for another a')\n ('b', 'value for subfield b')\n\n :param marcxml: an XML string representation of the record to create\n :param verbose: the level of verbosity: 0 (silent), 1-2 (warnings),\n 3(strict:stop when errors)\n :param correct: 1 to enable correction of marcxml syntax. Else 0.\n :return: a tuple (record, status_code, list_of_errors), where status\n code is 0 where there are errors, 1 when no errors\n \"\"\"\n if marcxml is None:\n return {}\n try:\n rec = _create_record_lxml(marcxml, verbose, correct,\n keep_singletons=keep_singletons)\n except InvenioBibRecordParserError as ex1:\n return (None, 0, str(ex1))\n\n if sort_fields_by_indicators:\n _record_sort_by_indicators(rec)\n\n errs = []\n if correct:\n # Correct the structure of the record.\n errs = _correct_record(rec)\n\n return (rec, int(not errs), errs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef filter_field_instances(field_instances, filter_subcode, filter_value,\n filter_mode='e'):\n \"\"\"Filter the given field.\n\n Filters given field and returns only that field instances that contain\n filter_subcode with given filter_value. As an input for search function\n accepts output from record_get_field_instances function. Function can be\n run in three modes:\n\n - 'e' - looking for exact match in subfield value\n - 's' - looking for substring in subfield value\n - 'r' - looking for regular expression in subfield value\n\n Example:\n\n record_filter_field(record_get_field_instances(rec, '999', '%', '%'),\n 'y', '2001')\n\n In this case filter_subcode is 'y' and filter_value is '2001'.\n\n :param field_instances: output from record_get_field_instances\n :param filter_subcode: name of the subfield\n :type filter_subcode: string\n :param filter_value: value of the subfield\n :type filter_value: string\n :param filter_mode: 'e','s' or 'r'\n \"\"\"\n matched = []\n if filter_mode == 'e':\n to_match = (filter_subcode, filter_value)\n for instance in field_instances:\n if to_match in instance[0]:\n matched.append(instance)\n elif filter_mode == 's':\n for instance in field_instances:\n for subfield in instance[0]:\n if subfield[0] == filter_subcode and \\\n subfield[1].find(filter_value) > -1:\n matched.append(instance)\n break\n elif filter_mode == 'r':\n reg_exp = re.compile(filter_value)\n for instance in field_instances:\n for subfield in instance[0]:\n if subfield[0] == filter_subcode and \\\n reg_exp.match(subfield[1]) is not None:\n matched.append(instance)\n break\n return matched", "response": "Filter the given field and returns only that field instances that contain filter_subcode with given filter_value."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a record where all the duplicate fields have been removed.", "response": "def record_drop_duplicate_fields(record):\n \"\"\"\n Return a record where all the duplicate fields have been removed.\n\n Fields are considered identical considering also the order of their\n subfields.\n \"\"\"\n out = {}\n position = 0\n tags = sorted(record.keys())\n for tag in tags:\n fields = record[tag]\n out[tag] = []\n current_fields = set()\n for full_field in fields:\n field = (tuple(full_field[0]),) + full_field[1:4]\n if field not in current_fields:\n current_fields.add(field)\n position += 1\n out[tag].append(full_field[:4] + (position,))\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if records are identical.", "response": "def records_identical(rec1, rec2, skip_005=True, ignore_field_order=False,\n ignore_subfield_order=False,\n ignore_duplicate_subfields=False,\n ignore_duplicate_controlfields=False):\n \"\"\"\n Return True if rec1 is identical to rec2.\n\n It does so regardless of a difference in the 005 tag (i.e. the timestamp).\n \"\"\"\n rec1_keys = set(rec1.keys())\n rec2_keys = set(rec2.keys())\n if skip_005:\n rec1_keys.discard(\"005\")\n rec2_keys.discard(\"005\")\n if rec1_keys != rec2_keys:\n return False\n for key in rec1_keys:\n if ignore_duplicate_controlfields and key.startswith('00'):\n if set(field[3] for field in rec1[key]) != \\\n set(field[3] for field in rec2[key]):\n return False\n continue\n\n rec1_fields = rec1[key]\n rec2_fields = rec2[key]\n if len(rec1_fields) != len(rec2_fields):\n # They already differs in length...\n return False\n if ignore_field_order:\n # We sort the fields, first by indicators and then by anything else\n rec1_fields = sorted(\n rec1_fields,\n key=lambda elem: (elem[1], elem[2], elem[3], elem[0]))\n rec2_fields = sorted(\n rec2_fields,\n key=lambda elem: (elem[1], elem[2], elem[3], elem[0]))\n else:\n # We sort the fields, first by indicators, then by global position\n # and then by anything else\n rec1_fields = sorted(\n rec1_fields,\n key=lambda elem: (elem[1], elem[2], elem[4], elem[3], elem[0]))\n rec2_fields = sorted(\n rec2_fields,\n key=lambda elem: (elem[1], elem[2], elem[4], elem[3], elem[0]))\n for field1, field2 in zip(rec1_fields, rec2_fields):\n if ignore_duplicate_subfields:\n if field1[1:4] != field2[1:4] or \\\n set(field1[0]) != set(field2[0]):\n return False\n elif ignore_subfield_order:\n if field1[1:4] != field2[1:4] or \\\n sorted(field1[0]) != sorted(field2[0]):\n return False\n elif field1[:4] != field2[:4]:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of field instances for the specified tag and indications.", "response": "def record_get_field_instances(rec, tag=\"\", ind1=\" \", ind2=\" \"):\n \"\"\"\n Return the list of field instances for the specified tag and indications.\n\n Return empty list if not found.\n If tag is empty string, returns all fields\n\n Parameters (tag, ind1, ind2) can contain wildcard %.\n\n :param rec: a record structure as returned by create_record()\n :param tag: a 3 characters long string\n :param ind1: a 1 character long string\n :param ind2: a 1 character long string\n :param code: a 1 character long string\n :return: a list of field tuples (Subfields, ind1, ind2, value,\n field_position_global) where subfields is list of (code, value)\n \"\"\"\n if not rec:\n return []\n if not tag:\n return rec.items()\n else:\n out = []\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n if '%' in tag:\n # Wildcard in tag. Check all possible\n for field_tag in rec:\n if _tag_matches_pattern(field_tag, tag):\n for possible_field_instance in rec[field_tag]:\n if (ind1 in ('%', possible_field_instance[1]) and\n ind2 in ('%', possible_field_instance[2])):\n out.append(possible_field_instance)\n else:\n # Completely defined tag. Use dict\n for possible_field_instance in rec.get(tag, []):\n if (ind1 in ('%', possible_field_instance[1]) and\n ind2 in ('%', possible_field_instance[2])):\n out.append(possible_field_instance)\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a new field into the record.", "response": "def record_add_field(rec, tag, ind1=' ', ind2=' ', controlfield_value='',\n subfields=None, field_position_global=None,\n field_position_local=None):\n \"\"\"\n Add a new field into the record.\n\n If field_position_global or field_position_local is specified then\n this method will insert the new field at the desired position.\n Otherwise a global field position will be computed in order to\n insert the field at the best position (first we try to keep the\n order of the tags and then we insert the field at the end of the\n fields with the same tag).\n\n If both field_position_global and field_position_local are present,\n then field_position_local takes precedence.\n\n :param rec: the record data structure\n :param tag: the tag of the field to be added\n :param ind1: the first indicator\n :param ind2: the second indicator\n :param controlfield_value: the value of the controlfield\n :param subfields: the subfields (a list of tuples (code, value))\n :param field_position_global: the global field position (record wise)\n :param field_position_local: the local field position (tag wise)\n :return: the global field position of the newly inserted field or -1 if the\n operation failed\n \"\"\"\n error = _validate_record_field_positions_global(rec)\n if error:\n # FIXME one should write a message here\n pass\n\n # Clean the parameters.\n if subfields is None:\n subfields = []\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n if controlfield_value and (ind1 != ' ' or ind2 != ' ' or subfields):\n return -1\n\n # Detect field number to be used for insertion:\n # Dictionaries for uniqueness.\n tag_field_positions_global = {}.fromkeys([field[4]\n for field in rec.get(tag, [])])\n all_field_positions_global = {}.fromkeys([field[4]\n for fields in rec.values()\n for field in fields])\n\n if field_position_global is None and field_position_local is None:\n # Let's determine the global field position of the new field.\n if tag in rec:\n try:\n field_position_global = max([field[4] for field in rec[tag]]) \\\n + 1\n except IndexError:\n if tag_field_positions_global:\n field_position_global = max(tag_field_positions_global) + 1\n elif all_field_positions_global:\n field_position_global = max(all_field_positions_global) + 1\n else:\n field_position_global = 1\n else:\n if tag in ('FMT', 'FFT', 'BDR', 'BDM'):\n # Add the new tag to the end of the record.\n if tag_field_positions_global:\n field_position_global = max(tag_field_positions_global) + 1\n elif all_field_positions_global:\n field_position_global = max(all_field_positions_global) + 1\n else:\n field_position_global = 1\n else:\n # Insert the tag in an ordered way by selecting the\n # right global field position.\n immediate_lower_tag = '000'\n for rec_tag in rec:\n if (tag not in ('FMT', 'FFT', 'BDR', 'BDM') and\n rec[rec_tag] and\n immediate_lower_tag < rec_tag < tag):\n immediate_lower_tag = rec_tag\n\n if immediate_lower_tag == '000':\n field_position_global = 1\n else:\n field_position_global = rec[immediate_lower_tag][-1][4] + 1\n\n field_position_local = len(rec.get(tag, []))\n _shift_field_positions_global(rec, field_position_global, 1)\n elif field_position_local is not None:\n if tag in rec:\n if field_position_local >= len(rec[tag]):\n field_position_global = rec[tag][-1][4] + 1\n else:\n field_position_global = rec[tag][field_position_local][4]\n _shift_field_positions_global(rec, field_position_global, 1)\n else:\n if all_field_positions_global:\n field_position_global = max(all_field_positions_global) + 1\n else:\n # Empty record.\n field_position_global = 1\n elif field_position_global is not None:\n # If the user chose an existing global field position, shift all the\n # global field positions greater than the input global field position.\n if tag not in rec:\n if all_field_positions_global:\n field_position_global = max(all_field_positions_global) + 1\n else:\n field_position_global = 1\n field_position_local = 0\n elif field_position_global < min(tag_field_positions_global):\n field_position_global = min(tag_field_positions_global)\n _shift_field_positions_global(rec, min(tag_field_positions_global),\n 1)\n field_position_local = 0\n elif field_position_global > max(tag_field_positions_global):\n field_position_global = max(tag_field_positions_global) + 1\n _shift_field_positions_global(rec,\n max(tag_field_positions_global) + 1,\n 1)\n field_position_local = len(rec.get(tag, []))\n else:\n if field_position_global in tag_field_positions_global:\n _shift_field_positions_global(rec, field_position_global, 1)\n\n field_position_local = 0\n for position, field in enumerate(rec[tag]):\n if field[4] == field_position_global + 1:\n field_position_local = position\n\n # Create the new field.\n newfield = (subfields, ind1, ind2, str(controlfield_value),\n field_position_global)\n rec.setdefault(tag, []).insert(field_position_local, newfield)\n\n # Return new field number:\n return field_position_global"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_delete_field(rec, tag, ind1=' ', ind2=' ',\n field_position_global=None, field_position_local=None):\n \"\"\"\n Delete the field with the given position.\n\n If global field position is specified, deletes the field with the\n corresponding global field position.\n If field_position_local is specified, deletes the field with the\n corresponding local field position and tag.\n Else deletes all the fields matching tag and optionally ind1 and\n ind2.\n\n If both field_position_global and field_position_local are present,\n then field_position_local takes precedence.\n\n :param rec: the record data structure\n :param tag: the tag of the field to be deleted\n :param ind1: the first indicator of the field to be deleted\n :param ind2: the second indicator of the field to be deleted\n :param field_position_global: the global field position (record wise)\n :param field_position_local: the local field position (tag wise)\n :return: the list of deleted fields\n \"\"\"\n error = _validate_record_field_positions_global(rec)\n if error:\n # FIXME one should write a message here.\n pass\n\n if tag not in rec:\n return False\n\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n deleted = []\n newfields = []\n\n if field_position_global is None and field_position_local is None:\n # Remove all fields with tag 'tag'.\n for field in rec[tag]:\n if field[1] != ind1 or field[2] != ind2:\n newfields.append(field)\n else:\n deleted.append(field)\n rec[tag] = newfields\n elif field_position_global is not None:\n # Remove the field with 'field_position_global'.\n for field in rec[tag]:\n if (field[1] != ind1 and field[2] != ind2 or\n field[4] != field_position_global):\n newfields.append(field)\n else:\n deleted.append(field)\n rec[tag] = newfields\n elif field_position_local is not None:\n # Remove the field with 'field_position_local'.\n try:\n del rec[tag][field_position_local]\n except IndexError:\n return []\n\n if not rec[tag]:\n # Tag is now empty, remove it.\n del rec[tag]\n\n return deleted", "response": "Delete a field from the record with the given position."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes all fields defined with MARC tag from record rec.", "response": "def record_delete_fields(rec, tag, field_positions_local=None):\n \"\"\"\n Delete all/some fields defined with MARC tag 'tag' from record 'rec'.\n\n :param rec: a record structure.\n :type rec: tuple\n :param tag: three letter field.\n :type tag: string\n :param field_position_local: if set, it is the list of local positions\n within all the fields with the specified tag, that should be deleted.\n If not set all the fields with the specified tag will be deleted.\n :type field_position_local: sequence\n :return: the list of deleted fields.\n :rtype: list\n :note: the record is modified in place.\n \"\"\"\n if tag not in rec:\n return []\n\n new_fields, deleted_fields = [], []\n\n for position, field in enumerate(rec.get(tag, [])):\n if field_positions_local is None or position in field_positions_local:\n deleted_fields.append(field)\n else:\n new_fields.append(field)\n\n if new_fields:\n rec[tag] = new_fields\n else:\n del rec[tag]\n\n return deleted_fields"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd the fields into the record at the required position.", "response": "def record_add_fields(rec, tag, fields, field_position_local=None,\n field_position_global=None):\n \"\"\"\n Add the fields into the record at the required position.\n\n The position is specified by the tag and the field_position_local in the\n list of fields.\n\n :param rec: a record structure\n :param tag: the tag of the fields to be moved\n :param field_position_local: the field_position_local to which the field\n will be inserted. If not specified, appends\n the fields to the tag.\n :param a: list of fields to be added\n :return: -1 if the operation failed, or the field_position_local if it was\n successful\n \"\"\"\n if field_position_local is None and field_position_global is None:\n for field in fields:\n record_add_field(\n rec, tag, ind1=field[1],\n ind2=field[2], subfields=field[0],\n controlfield_value=field[3])\n else:\n fields.reverse()\n for field in fields:\n record_add_field(\n rec, tag, ind1=field[1], ind2=field[2],\n subfields=field[0], controlfield_value=field[3],\n field_position_local=field_position_local,\n field_position_global=field_position_global)\n\n return field_position_local"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_move_fields(rec, tag, field_positions_local,\n field_position_local=None):\n \"\"\"\n Move some fields to the position specified by 'field_position_local'.\n\n :param rec: a record structure as returned by create_record()\n :param tag: the tag of the fields to be moved\n :param field_positions_local: the positions of the fields to move\n :param field_position_local: insert the field before that\n field_position_local. If unspecified, appends\n the fields :return: the field_position_local\n is the operation was successful\n \"\"\"\n fields = record_delete_fields(\n rec, tag,\n field_positions_local=field_positions_local)\n return record_add_fields(\n rec, tag, fields,\n field_position_local=field_position_local)", "response": "Move some fields to the position specified by field_position_local."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndelete all subfields with subfield_code in the record.", "response": "def record_delete_subfield(rec, tag, subfield_code, ind1=' ', ind2=' '):\n \"\"\"Delete all subfields with subfield_code in the record.\"\"\"\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n for field in rec.get(tag, []):\n if field[1] == ind1 and field[2] == ind2:\n field[0][:] = [subfield for subfield in field[0]\n if subfield_code != subfield[0]]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the matching field in a record.", "response": "def record_get_field(rec, tag, field_position_global=None,\n field_position_local=None):\n \"\"\"\n Return the the matching field.\n\n One has to enter either a global field position or a local field position.\n\n :return: a list of subfield tuples (subfield code, value).\n :rtype: list\n \"\"\"\n if field_position_global is None and field_position_local is None:\n raise InvenioBibRecordFieldError(\n \"A field position is required to \"\n \"complete this operation.\")\n elif field_position_global is not None and \\\n field_position_local is not None:\n raise InvenioBibRecordFieldError(\n \"Only one field position is required \"\n \"to complete this operation.\")\n elif field_position_global:\n if tag not in rec:\n raise InvenioBibRecordFieldError(\"No tag '%s' in record.\" % tag)\n\n for field in rec[tag]:\n if field[4] == field_position_global:\n return field\n raise InvenioBibRecordFieldError(\n \"No field has the tag '%s' and the \"\n \"global field position '%d'.\" % (tag, field_position_global))\n else:\n try:\n return rec[tag][field_position_local]\n except KeyError:\n raise InvenioBibRecordFieldError(\"No tag '%s' in record.\" % tag)\n except IndexError:\n raise InvenioBibRecordFieldError(\n \"No field has the tag '%s' and \"\n \"the local field position '%d'.\" % (tag, field_position_local))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreplace a field with a new field.", "response": "def record_replace_field(rec, tag, new_field, field_position_global=None,\n field_position_local=None):\n \"\"\"Replace a field with a new field.\"\"\"\n if field_position_global is None and field_position_local is None:\n raise InvenioBibRecordFieldError(\n \"A field position is required to \"\n \"complete this operation.\")\n elif field_position_global is not None and \\\n field_position_local is not None:\n raise InvenioBibRecordFieldError(\n \"Only one field position is required \"\n \"to complete this operation.\")\n elif field_position_global:\n if tag not in rec:\n raise InvenioBibRecordFieldError(\"No tag '%s' in record.\" % tag)\n\n replaced = False\n for position, field in enumerate(rec[tag]):\n if field[4] == field_position_global:\n rec[tag][position] = new_field\n replaced = True\n\n if not replaced:\n raise InvenioBibRecordFieldError(\n \"No field has the tag '%s' and \"\n \"the global field position '%d'.\" %\n (tag, field_position_global))\n else:\n try:\n rec[tag][field_position_local] = new_field\n except KeyError:\n raise InvenioBibRecordFieldError(\"No tag '%s' in record.\" % tag)\n except IndexError:\n raise InvenioBibRecordFieldError(\n \"No field has the tag '%s' and \"\n \"the local field position '%d'.\" % (tag, field_position_local))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the subfield of the matching field. One has to enter either a global field position or a local field position.", "response": "def record_get_subfields(rec, tag, field_position_global=None,\n field_position_local=None):\n \"\"\"\n Return the subfield of the matching field.\n\n One has to enter either a global field position or a local field position.\n\n :return: a list of subfield tuples (subfield code, value).\n :rtype: list\n \"\"\"\n field = record_get_field(\n rec, tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n return field[0]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndelete a subfield from the record.", "response": "def record_delete_subfield_from(rec, tag, subfield_position,\n field_position_global=None,\n field_position_local=None):\n \"\"\"\n Delete subfield from position specified.\n\n Specify the subfield by tag, field number and subfield position.\n \"\"\"\n subfields = record_get_subfields(\n rec, tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n try:\n del subfields[subfield_position]\n except IndexError:\n raise InvenioBibRecordFieldError(\n \"The record does not contain the subfield \"\n \"'%(subfieldIndex)s' inside the field (local: \"\n \"'%(fieldIndexLocal)s, global: '%(fieldIndexGlobal)s' ) of tag \"\n \"'%(tag)s'.\" %\n {\"subfieldIndex\": subfield_position,\n \"fieldIndexLocal\": str(field_position_local),\n \"fieldIndexGlobal\": str(field_position_global),\n \"tag\": tag})\n if not subfields:\n if field_position_global is not None:\n for position, field in enumerate(rec[tag]):\n if field[4] == field_position_global:\n del rec[tag][position]\n else:\n del rec[tag][field_position_local]\n\n if not rec[tag]:\n del rec[tag]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a subfield into a record.", "response": "def record_add_subfield_into(rec, tag, subfield_code, value,\n subfield_position=None,\n field_position_global=None,\n field_position_local=None):\n \"\"\"Add subfield into specified position.\n\n Specify the subfield by tag, field number and optionally by subfield\n position.\n \"\"\"\n subfields = record_get_subfields(\n rec, tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n if subfield_position is None:\n subfields.append((subfield_code, value))\n else:\n subfields.insert(subfield_position, (subfield_code, value))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmodifying a controlfield at position specified by tag and field number.", "response": "def record_modify_controlfield(rec, tag, controlfield_value,\n field_position_global=None,\n field_position_local=None):\n \"\"\"Modify controlfield at position specified by tag and field number.\"\"\"\n field = record_get_field(\n rec, tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n new_field = (field[0], field[1], field[2], controlfield_value, field[4])\n\n record_replace_field(\n rec, tag, new_field,\n field_position_global=field_position_global,\n field_position_local=field_position_local)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef record_modify_subfield(rec, tag, subfield_code, value, subfield_position,\n field_position_global=None,\n field_position_local=None):\n \"\"\"Modify subfield at specified position.\n\n Specify the subfield by tag, field number and subfield position.\n \"\"\"\n subfields = record_get_subfields(\n rec, tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n try:\n subfields[subfield_position] = (subfield_code, value)\n except IndexError:\n raise InvenioBibRecordFieldError(\n \"There is no subfield with position '%d'.\" % subfield_position)", "response": "Modify a subfield in a record at specified position."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef record_move_subfield(rec, tag, subfield_position, new_subfield_position,\n field_position_global=None,\n field_position_local=None):\n \"\"\"Move subfield at specified position.\n\n Sspecify the subfield by tag, field number and subfield position to new\n subfield position.\n \"\"\"\n subfields = record_get_subfields(\n rec,\n tag,\n field_position_global=field_position_global,\n field_position_local=field_position_local)\n\n try:\n subfield = subfields.pop(subfield_position)\n subfields.insert(new_subfield_position, subfield)\n except IndexError:\n raise InvenioBibRecordFieldError(\n \"There is no subfield with position '%d'.\" % subfield_position)", "response": "Move a subfield at specified position."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the value of a field in a record.", "response": "def record_get_field_value(rec, tag, ind1=\" \", ind2=\" \", code=\"\"):\n \"\"\"Return first (string) value that matches specified field of the record.\n\n Returns empty string if not found.\n\n Parameters (tag, ind1, ind2, code) can contain wildcard %.\n\n Difference between wildcard % and empty '':\n\n - Empty char specifies that we are not interested in a field which\n has one of the indicator(s)/subfield specified.\n\n - Wildcard specifies that we are interested in getting the value\n of the field whatever the indicator(s)/subfield is.\n\n For e.g. consider the following record in MARC::\n\n 100C5 $$a val1\n 555AB $$a val2\n 555AB val3\n 555 $$a val4\n 555A val5\n\n .. doctest::\n\n >>> record_get_field_value(record, '555', 'A', '', '')\n \"val5\"\n >>> record_get_field_value(record, '555', 'A', '%', '')\n \"val3\"\n >>> record_get_field_value(record, '555', 'A', '%', '%')\n \"val2\"\n >>> record_get_field_value(record, '555', 'A', 'B', '')\n \"val3\"\n >>> record_get_field_value(record, '555', '', 'B', 'a')\n \"\"\n >>> record_get_field_value(record, '555', '', '', 'a')\n \"val4\"\n >>> record_get_field_value(record, '555', '', '', '')\n \"\"\n >>> record_get_field_value(record, '%%%', '%', '%', '%')\n \"val1\"\n\n :param rec: a record structure as returned by create_record()\n :param tag: a 3 characters long string\n :param ind1: a 1 character long string\n :param ind2: a 1 character long string\n :param code: a 1 character long string\n :return: string value (empty if nothing found)\n \"\"\"\n # Note: the code is quite redundant for speed reasons (avoid calling\n # functions or doing tests inside loops)\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n if '%' in tag:\n # Wild card in tag. Must find all corresponding fields\n if code == '':\n # Code not specified.\n for field_tag, fields in rec.items():\n if _tag_matches_pattern(field_tag, tag):\n for field in fields:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n # Return matching field value if not empty\n if field[3]:\n return field[3]\n elif code == '%':\n # Code is wildcard. Take first subfield of first matching field\n for field_tag, fields in rec.items():\n if _tag_matches_pattern(field_tag, tag):\n for field in fields:\n if (ind1 in ('%', field[1]) and\n ind2 in ('%', field[2]) and field[0]):\n return field[0][0][1]\n else:\n # Code is specified. Take corresponding one\n for field_tag, fields in rec.items():\n if _tag_matches_pattern(field_tag, tag):\n for field in fields:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n for subfield in field[0]:\n if subfield[0] == code:\n return subfield[1]\n\n else:\n # Tag is completely specified. Use tag as dict key\n if tag in rec:\n if code == '':\n # Code not specified.\n for field in rec[tag]:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n # Return matching field value if not empty\n # or return \"\" empty if not exist.\n if field[3]:\n return field[3]\n\n elif code == '%':\n # Code is wildcard. Take first subfield of first matching field\n for field in rec[tag]:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]) and\\\n field[0]:\n return field[0][0][1]\n else:\n # Code is specified. Take corresponding one\n for field in rec[tag]:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n for subfield in field[0]:\n if subfield[0] == code:\n return subfield[1]\n # Nothing was found\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef record_get_field_values(rec, tag, ind1=\" \", ind2=\" \", code=\"\",\n filter_subfield_code=\"\",\n filter_subfield_value=\"\",\n filter_subfield_mode=\"e\"):\n \"\"\"Return the list of values for the specified field of the record.\n\n List can be filtered. Use filter_subfield_code\n and filter_subfield_value to search\n only in fields that have these values inside them as a subfield.\n filter_subfield_mode can have 3 different values:\n 'e' for exact search\n 's' for substring search\n 'r' for regexp search\n\n Returns empty list if nothing was found.\n\n Parameters (tag, ind1, ind2, code) can contain wildcard %.\n\n :param rec: a record structure as returned by create_record()\n :param tag: a 3 characters long string\n :param ind1: a 1 character long string\n :param ind2: a 1 character long string\n :param code: a 1 character long string\n :return: a list of strings\n \"\"\"\n tmp = []\n\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n if filter_subfield_code and filter_subfield_mode == \"r\":\n reg_exp = re.compile(filter_subfield_value)\n\n tags = []\n if '%' in tag:\n # Wild card in tag. Must find all corresponding tags and fields\n tags = [k for k in rec if _tag_matches_pattern(k, tag)]\n elif rec and tag in rec:\n tags = [tag]\n\n if code == '':\n # Code not specified. Consider field value (without subfields)\n for tag in tags:\n for field in rec[tag]:\n if (ind1 in ('%', field[1]) and ind2 in ('%', field[2]) and\n field[3]):\n tmp.append(field[3])\n elif code == '%':\n # Code is wildcard. Consider all subfields\n for tag in tags:\n for field in rec[tag]:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n if filter_subfield_code:\n if filter_subfield_mode == \"e\":\n subfield_to_match = (filter_subfield_code,\n filter_subfield_value)\n if subfield_to_match in field[0]:\n for subfield in field[0]:\n tmp.append(subfield[1])\n elif filter_subfield_mode == \"s\":\n if (dict(field[0]).get(filter_subfield_code, '')) \\\n .find(filter_subfield_value) > -1:\n for subfield in field[0]:\n tmp.append(subfield[1])\n elif filter_subfield_mode == \"r\":\n if reg_exp.match(dict(field[0])\n .get(filter_subfield_code, '')):\n for subfield in field[0]:\n tmp.append(subfield[1])\n else:\n for subfield in field[0]:\n tmp.append(subfield[1])\n else:\n # Code is specified. Consider all corresponding subfields\n for tag in tags:\n for field in rec[tag]:\n if ind1 in ('%', field[1]) and ind2 in ('%', field[2]):\n if filter_subfield_code:\n if filter_subfield_mode == \"e\":\n subfield_to_match = (filter_subfield_code,\n filter_subfield_value)\n if subfield_to_match in field[0]:\n for subfield in field[0]:\n if subfield[0] == code:\n tmp.append(subfield[1])\n elif filter_subfield_mode == \"s\":\n if (dict(field[0]).get(filter_subfield_code, '')) \\\n .find(filter_subfield_value) > -1:\n for subfield in field[0]:\n if subfield[0] == code:\n tmp.append(subfield[1])\n elif filter_subfield_mode == \"r\":\n if reg_exp.match(dict(field[0])\n .get(filter_subfield_code, '')):\n for subfield in field[0]:\n if subfield[0] == code:\n tmp.append(subfield[1])\n else:\n for subfield in field[0]:\n if subfield[0] == code:\n tmp.append(subfield[1])\n\n # If tmp was not set, nothing was found\n return tmp", "response": "Return the list of values for the specified field of the record."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate the XML for the record rec.", "response": "def record_xml_output(rec, tags=None, order_fn=None):\n \"\"\"Generate the XML for record 'rec'.\n\n :param rec: record\n :param tags: list of tags to be printed\n :return: string\n \"\"\"\n if tags is None:\n tags = []\n if isinstance(tags, str):\n tags = [tags]\n if tags and '001' not in tags:\n # Add the missing controlfield.\n tags.append('001')\n\n marcxml = ['']\n\n # Add the tag 'tag' to each field in rec[tag]\n fields = []\n if rec is not None:\n for tag in rec:\n if not tags or tag in tags:\n for field in rec[tag]:\n fields.append((tag, field))\n if order_fn is None:\n record_order_fields(fields)\n else:\n record_order_fields(fields, order_fn)\n for field in fields:\n marcxml.append(field_xml_output(field[1], field[0]))\n marcxml.append('')\n return '\\n'.join(marcxml)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef field_xml_output(field, tag):\n marcxml = []\n if field[3]:\n marcxml.append(' %s' %\n (tag, MathMLParser.html_to_text(field[3])))\n else:\n marcxml.append(' ' %\n (tag, field[1], field[2]))\n marcxml += [_subfield_xml_output(subfield) for subfield in field[0]]\n marcxml.append(' ')\n return '\\n'.join(marcxml)", "response": "Generate the XML for field and returns it as a string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef record_extract_dois(record):\n record_dois = []\n tag = \"024\"\n ind1 = \"7\"\n ind2 = \"_\"\n subfield_source_code = \"2\"\n subfield_value_code = \"a\"\n identifiers_fields = record_get_field_instances(record, tag, ind1, ind2)\n for identifer_field in identifiers_fields:\n if 'doi' in [val.lower() for val in\n field_get_subfield_values(identifer_field,\n subfield_source_code)]:\n record_dois.extend(\n field_get_subfield_values(\n identifer_field,\n subfield_value_code))\n return record_dois", "response": "Return the DOIs of the record."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprinting a record. :param format: 1 XML, 2 HTML (not implemented) :param tags: list of tags to be printed", "response": "def print_rec(rec, format=1, tags=None):\n \"\"\"\n Print a record.\n\n :param format: 1 XML, 2 HTML (not implemented)\n :param tags: list of tags to be printed\n \"\"\"\n if tags is None:\n tags = []\n if format == 1:\n text = record_xml_output(rec, tags)\n else:\n return ''\n\n return text"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef print_recs(listofrec, format=1, tags=None):\n if tags is None:\n tags = []\n text = \"\"\n\n if type(listofrec).__name__ != 'list':\n return \"\"\n else:\n for rec in listofrec:\n text = \"%s\\n%s\" % (text, print_rec(rec, format, tags))\n return text", "response": "Print a list of records."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the global and local positions of the first occurrence of the field in the record.", "response": "def record_find_field(rec, tag, field, strict=False):\n \"\"\"\n Return the global and local positions of the first occurrence of the field.\n\n :param rec: A record dictionary structure\n :type rec: dictionary\n :param tag: The tag of the field to search for\n :type tag: string\n :param field: A field tuple as returned by create_field()\n :type field: tuple\n :param strict: A boolean describing the search method. If strict\n is False, then the order of the subfields doesn't\n matter. Default search method is strict.\n :type strict: boolean\n :return: A tuple of (global_position, local_position) or a\n tuple (None, None) if the field is not present.\n :rtype: tuple\n :raise InvenioBibRecordFieldError: If the provided field is invalid.\n \"\"\"\n try:\n _check_field_validity(field)\n except InvenioBibRecordFieldError:\n raise\n\n for local_position, field1 in enumerate(rec.get(tag, [])):\n if _compare_fields(field, field1, strict):\n return (field1[4], local_position)\n\n return (None, None)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_match_subfields(rec, tag, ind1=\" \", ind2=\" \", sub_key=None,\n sub_value='', sub_key2=None, sub_value2='',\n case_sensitive=True):\n \"\"\"\n Find subfield instances in a particular field.\n\n It tests values in 1 of 3 possible ways:\n - Does a subfield code exist? (ie does 773__a exist?)\n - Does a subfield have a particular value? (ie 773__a == 'PhysX')\n - Do a pair of subfields have particular values?\n (ie 035__2 == 'CDS' and 035__a == '123456')\n\n Parameters:\n * rec - dictionary: a bibrecord structure\n * tag - string: the tag of the field (ie '773')\n * ind1, ind2 - char: a single characters for the MARC indicators\n * sub_key - char: subfield key to find\n * sub_value - string: subfield value of that key\n * sub_key2 - char: key of subfield to compare against\n * sub_value2 - string: expected value of second subfield\n * case_sensitive - bool: be case sensitive when matching values\n\n :return: false if no match found, else provides the field position (int)\n \"\"\"\n if sub_key is None:\n raise TypeError(\"None object passed for parameter sub_key.\")\n\n if sub_key2 is not None and sub_value2 is '':\n raise TypeError(\"Parameter sub_key2 defined but sub_value2 is None, \"\n + \"function requires a value for comparrison.\")\n ind1, ind2 = _wash_indicators(ind1, ind2)\n\n if not case_sensitive:\n sub_value = sub_value.lower()\n sub_value2 = sub_value2.lower()\n\n for field in record_get_field_instances(rec, tag, ind1, ind2):\n subfields = dict(field_get_subfield_instances(field))\n if not case_sensitive:\n for k, v in subfields.iteritems():\n subfields[k] = v.lower()\n\n if sub_key in subfields:\n if sub_value is '':\n return field[4]\n else:\n if sub_value == subfields[sub_key]:\n if sub_key2 is None:\n return field[4]\n else:\n if sub_key2 in subfields:\n if sub_value2 == subfields[sub_key2]:\n return field[4]\n return False", "response": "Find all subfields in a particular field in a particular record."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef record_strip_empty_volatile_subfields(rec):\n for tag in rec.keys():\n for field in rec[tag]:\n field[0][:] = [subfield for subfield in field[0]\n if subfield[1][:9] != \"VOLATILE:\"]", "response": "Remove empty volatile subfields from the record."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_make_all_subfields_volatile(rec):\n for tag in rec.keys():\n for field_position, field in enumerate(rec[tag]):\n for subfield_position, subfield in enumerate(field[0]):\n if subfield[1][:9] != \"VOLATILE:\":\n record_modify_subfield(rec, tag, subfield[0], \"VOLATILE:\" + subfield[1],\n subfield_position, field_position_local=field_position)", "response": "Turn all subfields to volatile"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef record_strip_empty_fields(rec, tag=None):\n # Check whole record\n if tag is None:\n tags = rec.keys()\n for tag in tags:\n record_strip_empty_fields(rec, tag)\n\n # Check specific tag of the record\n elif tag in rec:\n # in case of a controlfield\n if tag[:2] == '00':\n if len(rec[tag]) == 0 or not rec[tag][0][3]:\n del rec[tag]\n\n #in case of a normal field\n else:\n fields = []\n for field in rec[tag]:\n subfields = []\n for subfield in field[0]:\n # check if the subfield has been given a value\n if subfield[1]:\n # Always strip values\n subfield = (subfield[0], subfield[1].strip())\n subfields.append(subfield)\n if len(subfields) > 0:\n new_field = create_field(subfields, field[1], field[2],\n field[3])\n fields.append(new_field)\n if len(fields) > 0:\n rec[tag] = fields\n else:\n del rec[tag]", "response": "Removes empty fields from the record."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nremoves all non - empty controlfields from the record.", "response": "def record_strip_controlfields(rec):\n \"\"\"\n Remove all non-empty controlfields from the record.\n\n :param rec: A record dictionary structure\n :type rec: dictionary\n \"\"\"\n for tag in rec.keys():\n if tag[:2] == '00' and rec[tag][0][3]:\n del rec[tag]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\norder subfields from a record alphabetically based on subfield code.", "response": "def record_order_subfields(rec, tag=None):\n \"\"\"\n Order subfields from a record alphabetically based on subfield code.\n\n If 'tag' is not None, only a specific tag of the record will be reordered,\n otherwise the whole record.\n\n :param rec: bibrecord\n :type rec: bibrec\n :param tag: tag where the subfields will be ordered\n :type tag: str\n \"\"\"\n if rec is None:\n return rec\n if tag is None:\n tags = rec.keys()\n for tag in tags:\n record_order_subfields(rec, tag)\n elif tag in rec:\n for i in xrange(len(rec[tag])):\n field = rec[tag][i]\n # Order subfields alphabetically by subfield code\n ordered_subfields = sorted(field[0],\n key=lambda subfield: subfield[0])\n rec[tag][i] = (ordered_subfields, field[1], field[2], field[3],\n field[4])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef field_get_subfields(field):\n pairs = {}\n for key, value in field[0]:\n if key in pairs and pairs[key] != value:\n pairs[key].append(value)\n else:\n pairs[key] = [value]\n return pairs", "response": "Given a field returns a dictionary with codes as keys and a list of values as values"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _compare_fields(field1, field2, strict=True):\n if strict:\n # Return a simple equal test on the field minus the position.\n return field1[:4] == field2[:4]\n else:\n if field1[1:4] != field2[1:4]:\n # Different indicators or controlfield value.\n return False\n else:\n # Compare subfields in a loose way.\n return set(field1[0]) == set(field2[0])", "response": "Compare 2 fields.\n\n If strict is True, then the order of the subfield will be taken care of, if\n not then the order of the subfields doesn't matter.\n\n :return: True if the field are equivalent, False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if a field is well - formed.", "response": "def _check_field_validity(field):\n \"\"\"\n Check if a field is well-formed.\n\n :param field: A field tuple as returned by create_field()\n :type field: tuple\n :raise InvenioBibRecordFieldError: If the field is invalid.\n \"\"\"\n if type(field) not in (list, tuple):\n raise InvenioBibRecordFieldError(\n \"Field of type '%s' should be either \"\n \"a list or a tuple.\" % type(field))\n\n if len(field) != 5:\n raise InvenioBibRecordFieldError(\n \"Field of length '%d' should have 5 \"\n \"elements.\" % len(field))\n\n if type(field[0]) not in (list, tuple):\n raise InvenioBibRecordFieldError(\n \"Subfields of type '%s' should be \"\n \"either a list or a tuple.\" % type(field[0]))\n\n if type(field[1]) is not str:\n raise InvenioBibRecordFieldError(\n \"Indicator 1 of type '%s' should be \"\n \"a string.\" % type(field[1]))\n\n if type(field[2]) is not str:\n raise InvenioBibRecordFieldError(\n \"Indicator 2 of type '%s' should be \"\n \"a string.\" % type(field[2]))\n\n if type(field[3]) is not str:\n raise InvenioBibRecordFieldError(\n \"Controlfield value of type '%s' \"\n \"should be a string.\" % type(field[3]))\n\n if type(field[4]) is not int:\n raise InvenioBibRecordFieldError(\n \"Global position of type '%s' should \"\n \"be an int.\" % type(field[4]))\n\n for subfield in field[0]:\n if (type(subfield) not in (list, tuple) or\n len(subfield) != 2 or type(subfield[0]) is not str or\n type(subfield[1]) is not str):\n raise InvenioBibRecordFieldError(\n \"Subfields are malformed. \"\n \"Should a list of tuples of 2 strings.\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nshifting all global field positions with global field positions higher or equal to start from the value delta.", "response": "def _shift_field_positions_global(record, start, delta=1):\n \"\"\"\n Shift all global field positions.\n\n Shift all global field positions with global field positions\n higher or equal to 'start' from the value 'delta'.\n \"\"\"\n if not delta:\n return\n\n for tag, fields in record.items():\n newfields = []\n for field in fields:\n if field[4] < start:\n newfields.append(field)\n else:\n # Increment the global field position by delta.\n newfields.append(tuple(list(field[:4]) + [field[4] + delta]))\n record[tag] = newfields"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if MARC tag matches a pattern.", "response": "def _tag_matches_pattern(tag, pattern):\n \"\"\"Return true if MARC 'tag' matches a 'pattern'.\n\n 'pattern' is plain text, with % as wildcard\n\n Both parameters must be 3 characters long strings.\n\n .. doctest::\n\n >>> _tag_matches_pattern(\"909\", \"909\")\n True\n >>> _tag_matches_pattern(\"909\", \"9%9\")\n True\n >>> _tag_matches_pattern(\"909\", \"9%8\")\n False\n\n :param tag: a 3 characters long string\n :param pattern: a 3 characters long string\n :return: False or True\n \"\"\"\n for char1, char2 in zip(tag, pattern):\n if char2 not in ('%', char1):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks if the global field positions in the record are valid.", "response": "def _validate_record_field_positions_global(record):\n \"\"\"\n Check if the global field positions in the record are valid.\n\n I.e., no duplicate global field positions and local field positions in the\n list of fields are ascending.\n\n :param record: the record data structure\n :return: the first error found as a string or None if no error was found\n \"\"\"\n all_fields = []\n for tag, fields in record.items():\n previous_field_position_global = -1\n for field in fields:\n if field[4] < previous_field_position_global:\n return (\"Non ascending global field positions in tag '%s'.\" %\n tag)\n previous_field_position_global = field[4]\n if field[4] in all_fields:\n return (\"Duplicate global field position '%d' in tag '%s'\" %\n (field[4], tag))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsort the fields inside the record by indicators.", "response": "def _record_sort_by_indicators(record):\n \"\"\"Sort the fields inside the record by indicators.\"\"\"\n for tag, fields in record.items():\n record[tag] = _fields_sort_by_indicators(fields)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fields_sort_by_indicators(fields):\n field_dict = {}\n field_positions_global = []\n for field in fields:\n field_dict.setdefault(field[1:3], []).append(field)\n field_positions_global.append(field[4])\n\n indicators = field_dict.keys()\n indicators.sort()\n\n field_list = []\n for indicator in indicators:\n for field in field_dict[indicator]:\n field_list.append(field[:4] + (field_positions_global.pop(0),))\n\n return field_list", "response": "Sort a set of fields by their indicators."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _create_record_lxml(marcxml,\n verbose=CFG_BIBRECORD_DEFAULT_VERBOSE_LEVEL,\n correct=CFG_BIBRECORD_DEFAULT_CORRECT,\n keep_singletons=CFG_BIBRECORD_KEEP_SINGLETONS):\n \"\"\"\n Create a record object using the LXML parser.\n\n If correct == 1, then perform DTD validation\n If correct == 0, then do not perform DTD validation\n\n If verbose == 0, the parser will not give warnings.\n If 1 <= verbose <= 3, the parser will not give errors, but will warn\n the user about possible mistakes (implement me!)\n If verbose > 3 then the parser will be strict and will stop in case of\n well-formedness errors or DTD errors.\n\n \"\"\"\n parser = etree.XMLParser(dtd_validation=correct,\n recover=(verbose <= 3))\n if correct:\n marcxml = '\\n' \\\n '\\n%s\\n' % (marcxml,)\n try:\n tree = etree.parse(StringIO(marcxml), parser)\n # parser errors are located in parser.error_log\n # if 1 <= verbose <=3 then show them to the user?\n # if verbose == 0 then continue\n # if verbose >3 then an exception will be thrown\n except Exception as e:\n raise InvenioBibRecordParserError(str(e))\n\n record = {}\n field_position_global = 0\n\n controlfield_iterator = tree.iter(tag='{*}controlfield')\n for controlfield in controlfield_iterator:\n tag = controlfield.attrib.get('tag', '!').encode(\"UTF-8\")\n ind1 = ' '\n ind2 = ' '\n text = controlfield.text\n if text is None:\n text = ''\n else:\n text = text.encode(\"UTF-8\")\n subfields = []\n if text or keep_singletons:\n field_position_global += 1\n record.setdefault(tag, []).append((subfields, ind1, ind2, text,\n field_position_global))\n\n datafield_iterator = tree.iter(tag='{*}datafield')\n for datafield in datafield_iterator:\n tag = datafield.attrib.get('tag', '!').encode(\"UTF-8\")\n ind1 = datafield.attrib.get('ind1', '!').encode(\"UTF-8\")\n ind2 = datafield.attrib.get('ind2', '!').encode(\"UTF-8\")\n if ind1 in ('', '_'):\n ind1 = ' '\n if ind2 in ('', '_'):\n ind2 = ' '\n subfields = []\n subfield_iterator = datafield.iter(tag='{*}subfield')\n for subfield in subfield_iterator:\n code = subfield.attrib.get('code', '!').encode(\"UTF-8\")\n text = subfield.text\n if text is None:\n text = ''\n else:\n text = text.encode(\"UTF-8\")\n if text or keep_singletons:\n subfields.append((code, text))\n if subfields or keep_singletons:\n text = ''\n field_position_global += 1\n record.setdefault(tag, []).append((subfields, ind1, ind2, text,\n field_position_global))\n\n return record", "response": "Create a record object from a MARCXML string."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_children_by_tag_name(node, name):\n try:\n return [child for child in node.childNodes if child.nodeName == name]\n except TypeError:\n return []", "response": "Retrieve all children from node with name name."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\niterates through all the children of a node and return a string containing the values from all the text - nodes recursively.", "response": "def _get_children_as_string(node):\n \"\"\"Iterate through all the children of a node.\n\n Returns one string containing the values from all the text-nodes\n recursively.\n \"\"\"\n out = []\n if node:\n for child in node:\n if child.nodeType == child.TEXT_NODE:\n out.append(child.data)\n else:\n out.append(_get_children_as_string(child.childNodes))\n return ''.join(out)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _correct_record(record):\n errors = []\n\n for tag in record.keys():\n upper_bound = '999'\n n = len(tag)\n\n if n > 3:\n i = n - 3\n while i > 0:\n upper_bound = '%s%s' % ('0', upper_bound)\n i -= 1\n\n # Missing tag. Replace it with dummy tag '000'.\n if tag == '!':\n errors.append((1, '(field number(s): ' +\n str([f[4] for f in record[tag]]) + ')'))\n record['000'] = record.pop(tag)\n tag = '000'\n elif not ('001' <= tag <= upper_bound or\n tag in ('FMT', 'FFT', 'BDR', 'BDM')):\n errors.append(2)\n record['000'] = record.pop(tag)\n tag = '000'\n\n fields = []\n for field in record[tag]:\n # Datafield without any subfield.\n if field[0] == [] and field[3] == '':\n errors.append((8, '(field number: ' + str(field[4]) + ')'))\n\n subfields = []\n for subfield in field[0]:\n if subfield[0] == '!':\n errors.append((3, '(field number: ' + str(field[4]) + ')'))\n newsub = ('', subfield[1])\n else:\n newsub = subfield\n subfields.append(newsub)\n\n if field[1] == '!':\n errors.append((4, '(field number: ' + str(field[4]) + ')'))\n ind1 = \" \"\n else:\n ind1 = field[1]\n\n if field[2] == '!':\n errors.append((5, '(field number: ' + str(field[4]) + ')'))\n ind2 = \" \"\n else:\n ind2 = field[2]\n\n fields.append((subfields, ind1, ind2, field[3], field[4]))\n\n record[tag] = fields\n\n return errors", "response": "Check and correct the structure of the record."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a warning message of code.", "response": "def _warning(code):\n \"\"\"\n Return a warning message of code 'code'.\n\n If code = (cd, str) it returns the warning message of code 'cd' and appends\n str at the end\n \"\"\"\n if isinstance(code, str):\n return code\n\n message = ''\n if isinstance(code, tuple):\n if isinstance(code[0], str):\n message = code[1]\n code = code[0]\n return CFG_BIBRECORD_WARNING_MSGS.get(code, '') + message"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompare two lists of resources using given comparison function.", "response": "def _compare_lists(list1, list2, custom_cmp):\n \"\"\"Compare twolists using given comparing function.\n\n :param list1: first list to compare\n :param list2: second list to compare\n :param custom_cmp: a function taking two arguments (element of\n list 1, element of list 2) and\n :return: True or False depending if the values are the same\n \"\"\"\n if len(list1) != len(list2):\n return False\n for element1, element2 in zip(list1, list2):\n if not custom_cmp(element1, element2):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse(self, path_to_xml=None):\n if not path_to_xml:\n if not self.path:\n self.logger.error(\"No path defined!\")\n return\n path_to_xml = self.path\n root = self._clean_xml(path_to_xml)\n\n # See first of this XML is clean or OAI request\n if root.tag.lower() == 'collection':\n tree = ET.ElementTree(root)\n self.records = element_tree_collection_to_records(tree)\n elif root.tag.lower() == 'record':\n new_root = ET.Element('collection')\n new_root.append(root)\n tree = ET.ElementTree(new_root)\n self.records = element_tree_collection_to_records(tree)\n else:\n # We have an OAI request\n header_subs = get_request_subfields(root)\n records = root.find('ListRecords')\n if records is None:\n records = root.find('GetRecord')\n if records is None:\n raise ValueError(\"Cannot find ListRecords or GetRecord!\")\n\n tree = ET.ElementTree(records)\n for record, is_deleted in element_tree_oai_records(tree, header_subs):\n if is_deleted:\n # It was OAI deleted. Create special record\n self.deleted_records.append(\n self.create_deleted_record(record)\n )\n else:\n self.records.append(record)", "response": "Parse an XML document and clean any namespaces."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncleaning MARCXML harvested from OAI.", "response": "def _clean_xml(self, path_to_xml):\n \"\"\"Clean MARCXML harvested from OAI.\n\n Allows the xml to be used with BibUpload or BibRecord.\n\n :param xml: either XML as a string or path to an XML file\n\n :return: ElementTree of clean data\n \"\"\"\n try:\n if os.path.isfile(path_to_xml):\n tree = ET.parse(path_to_xml)\n root = tree.getroot()\n else:\n root = ET.fromstring(path_to_xml)\n except Exception, e:\n self.logger.error(\"Could not read OAI XML, aborting filter!\")\n raise e\n strip_xml_namespace(root)\n return root"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate the record deletion if deleted form OAI - PMH.", "response": "def create_deleted_record(self, record):\n \"\"\"Generate the record deletion if deleted form OAI-PMH.\"\"\"\n identifier = record_get_field_value(record,\n tag=\"037\",\n code=\"a\")\n recid = identifier.split(\":\")[-1]\n try:\n source = identifier.split(\":\")[1]\n except IndexError:\n source = \"Unknown\"\n record_add_field(record, \"035\",\n subfields=[(\"9\", source), (\"a\", recid)])\n record_add_field(record, \"980\",\n subfields=[(\"c\", \"DELETED\")])\n return record"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _login(self, session, get_request=False):\n req = session.post(self._login_url, data=self._logindata)\n if _LOGIN_ERROR_STRING in req.text or \\\n req.status_code == 403 or \\\n req.url == _LOGIN_URL:\n err_mess = \"YesssSMS: login failed, username or password wrong\"\n\n if _LOGIN_LOCKED_MESS in req.text:\n err_mess += \", page says: \" + _LOGIN_LOCKED_MESS_ENG\n self._suspended = True\n raise self.AccountSuspendedError(err_mess)\n raise self.LoginError(err_mess)\n\n self._suspended = False # login worked\n\n return (session, req) if get_request else session", "response": "Return a session for yesss. at."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef login_data_valid(self):\n login_working = False\n try:\n with self._login(requests.Session()) as sess:\n sess.get(self._logout_url)\n except self.LoginError:\n pass\n else:\n login_working = True\n return login_working", "response": "Check if working login data is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_date(self, filename):\n try:\n self.document = parse(filename)\n return self._get_date()\n except DateNotFoundException:\n print(\"Date problem found in {0}\".format(filename))\n return datetime.datetime.strftime(datetime.datetime.now(),\n \"%Y-%m-%d\")", "response": "Return the date of the article in file."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns this articles s collection.", "response": "def get_collection(self, journal):\n \"\"\"Return this articles' collection.\"\"\"\n conference = ''\n for tag in self.document.getElementsByTagName('conference'):\n conference = xml_to_text(tag)\n if conference or journal == \"International Journal of Modern Physics: Conference Series\":\n return [('a', 'HEP'), ('a', 'ConferencePaper')]\n elif self._get_article_type() == \"review-article\":\n return [('a', 'HEP'), ('a', 'Review')]\n else:\n return [('a', 'HEP'), ('a', 'Published')]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the MARCXML of the files in xaml_jp directory.", "response": "def get_record(self, filename, ref_extract_callback=None):\n \"\"\"Get the MARCXML of the files in xaml_jp directory.\n\n :param filename: the name of the file to parse.\n :type filename: string\n :param refextract_callback: callback to be used to extract\n unstructured references. It should\n return a marcxml formated string\n of the reference.\n :type refextract_callback: callable\n\n :returns: a string with the marc xml version of the file.\n \"\"\"\n self.document = parse(filename)\n\n article_type = self._get_article_type()\n if article_type not in ['research-article',\n 'corrected-article',\n 'original-article',\n 'introduction',\n 'letter',\n 'correction',\n 'addendum',\n 'review-article',\n 'rapid-communications']:\n return \"\"\n\n rec = create_record()\n title, subtitle, notes = self._get_title()\n subfields = []\n if subtitle:\n subfields.append(('b', subtitle))\n if title:\n title = fix_title_capitalization(title)\n subfields.append(('a', title))\n record_add_field(rec, '245', subfields=subfields)\n for note_id in notes:\n note = self._get_note(note_id)\n if note:\n record_add_field(rec, '500', subfields=[('a', note)])\n keywords = self._get_keywords()\n for keyword in keywords:\n record_add_field(rec, '653', ind1='1', subfields=[('a', keyword),\n ('9', 'author')])\n journal, volume, issue, year, date, doi, page,\\\n fpage, lpage = self._get_publication_information()\n if date:\n record_add_field(rec, '260', subfields=[('c', date),\n ('t', 'published')])\n if doi:\n record_add_field(rec, '024', ind1='7', subfields=[('a', doi),\n ('2', 'DOI')])\n abstract = self._get_abstract()\n if abstract:\n abstract = convert_html_subscripts_to_latex(abstract)\n record_add_field(rec, '520', subfields=[('a', abstract),\n ('9', 'World Scientific')])\n license, license_type, license_url = self._get_license()\n subfields = []\n if license:\n subfields.append(('a', license))\n if license_url:\n subfields.append(('u', license_url))\n if subfields:\n record_add_field(rec, '540', subfields=subfields)\n if license_type == 'open-access':\n self._attach_fulltext(rec, doi)\n number_of_pages = self._get_page_count()\n if number_of_pages:\n record_add_field(rec, '300', subfields=[('a', number_of_pages)])\n c_holder, c_year, c_statement = self._get_copyright()\n if c_holder and c_year:\n record_add_field(rec, '542', subfields=[('d', c_holder),\n ('g', c_year),\n ('e', 'Article')])\n elif c_statement:\n record_add_field(rec, '542', subfields=[('f', c_statement),\n ('e', 'Article')])\n subfields = []\n if journal:\n subfields.append(('p', journal))\n if issue:\n subfields.append(('n', issue))\n if volume:\n subfields.append(('v', volume))\n if fpage and lpage:\n subfields.append(('c', '%s-%s' % (fpage,\n lpage)))\n elif page:\n subfields.append(('c', page))\n if year:\n subfields.append(('y', year))\n if article_type == 'correction':\n subfields.append(('m', 'Erratum'))\n elif article_type == 'addendum':\n subfields.append(('m', 'Addendum'))\n record_add_field(rec, '773', subfields=subfields)\n\n collections = self.get_collection(journal)\n for collection in collections:\n record_add_field(rec, '980', subfields=[collection])\n\n self._add_authors(rec)\n if article_type in ['correction',\n 'addendum']:\n related_article = self._get_related_article()\n if related_article:\n record_add_field(rec, '024', ind1='7', subfields=[('a', related_article),\n ('2', 'DOI')])\n try:\n return record_xml_output(rec)\n except UnicodeDecodeError:\n message = \"Found a bad char in the file for the article \" + doi\n sys.stderr.write(message)\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting the list of bibrecs into one MARCXML.", "response": "def convert_all(cls, records):\n \"\"\"Convert the list of bibrecs into one MARCXML.\n\n >>> from harvestingkit.bibrecord import BibRecordPackage\n >>> from harvestingkit.inspire_cds_package import Inspire2CDS\n >>> bibrecs = BibRecordPackage(\"inspire.xml\")\n >>> bibrecs.parse()\n >>> xml = Inspire2CDS.convert_all(bibrecs.get_records())\n\n :param records: list of BibRecord dicts\n :type records: list\n\n :returns: MARCXML as string\n \"\"\"\n out = [\"\"]\n for rec in records:\n conversion = cls(rec)\n out.append(conversion.convert())\n out.append(\"\")\n return \"\\n\".join(out)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_source(cls, source):\n bibrecs = BibRecordPackage(source)\n bibrecs.parse()\n for bibrec in bibrecs.get_records():\n yield cls(bibrec)", "response": "Yields single conversion objects from a MARCXML file or string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the opposite mapping by searching the imported KB.", "response": "def get_config_item(cls, key, kb_name, allow_substring=True):\n \"\"\"Return the opposite mapping by searching the imported KB.\"\"\"\n config_dict = cls.kbs.get(kb_name, None)\n if config_dict:\n if key in config_dict:\n return config_dict[key]\n elif allow_substring:\n res = [v for k, v in config_dict.items() if key in k]\n if res:\n return res[0]\n return key"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_config(from_key, to_key):\n from .mappings import mappings\n kbs = {}\n for key, values in mappings['config'].iteritems():\n parse_dict = {}\n for mapping in values:\n # {'inspire': 'Norwegian', 'cds': 'nno'}\n # -> {\"Norwegian\": \"nno\"}\n parse_dict[mapping[from_key]] = mapping[to_key]\n kbs[key] = parse_dict\n return kbs", "response": "Load configuration from config.\nMimeType Meant to run only once per system process as\nMimeType class variable in subclasses."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef match(self, query=None, **kwargs):\n from invenio.search_engine import perform_request_search\n if not query:\n # We use default setup\n recid = self.record[\"001\"][0][3]\n return perform_request_search(p=\"035:%s\" % (recid,),\n of=\"id\")\n else:\n if \"recid\" not in kwargs:\n kwargs[\"recid\"] = self.record[\"001\"][0][3]\n return perform_request_search(p=query % kwargs,\n of=\"id\")", "response": "Try to match the current record to the database."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef keep_only_fields(self):\n for tag in self.record.keys():\n if tag not in self.fields_list:\n record_delete_fields(self.record, tag)", "response": "Keep only fields listed in field_list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nclearing any fields listed in field_list.", "response": "def strip_fields(self):\n \"\"\"Clear any fields listed in field_list.\"\"\"\n for tag in self.record.keys():\n if tag in self.fields_list:\n record_delete_fields(self.record, tag)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding 035 number from 001 recid with given source.", "response": "def add_systemnumber(self, source, recid=None):\n \"\"\"Add 035 number from 001 recid with given source.\"\"\"\n if not recid:\n recid = self.get_recid()\n if not self.hidden and recid:\n record_add_field(\n self.record,\n tag='035',\n subfields=[('9', source), ('a', recid)]\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a control - number 00x for given tag with value.", "response": "def add_control_number(self, tag, value):\n \"\"\"Add a control-number 00x for given tag with value.\"\"\"\n record_add_field(self.record,\n tag,\n controlfield_value=value)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_reference(self, ref):\n label = get_value_in_tag(ref, 'label')\n label = re.sub('\\D', '', label)\n for innerref in ref.getElementsByTagName('mixed-citation'):\n ref_type = innerref.getAttribute('publication-type')\n institution = get_value_in_tag(innerref, 'institution')\n report_no = ''\n for tag in innerref.getElementsByTagName('pub-id'):\n if tag.getAttribute('pub-id-type') == 'other':\n if tag.hasChildNodes():\n report_no = get_all_text(tag)\n doi = ''\n for tag in innerref.getElementsByTagName('pub-id'):\n if tag.getAttribute('pub-id-type') == 'doi':\n doi = xml_to_text(tag)\n collaboration = get_value_in_tag(innerref, 'collab')\n authors = []\n person_groups = innerref.getElementsByTagName('person-group')\n for author_group in person_groups:\n if author_group.getAttribute('person-group-type') == 'author':\n for author in author_group.getElementsByTagName('string-name'):\n if author.hasChildNodes():\n authors.append(get_all_text(author))\n editors = []\n for editor_group in person_groups:\n if editor_group.getAttribute('person-group-type') == 'editor':\n for editor in editor_group.getElementsByTagName('string-name'):\n if editor.hasChildNodes():\n editors.append(get_all_text(editor))\n journal = get_value_in_tag(innerref, 'source')\n journal, volume = fix_journal_name(journal, self.journal_mappings)\n volume += get_value_in_tag(innerref, 'volume')\n if journal == 'J.High Energy Phys.' or journal == 'JHEP':\n issue = get_value_in_tag(innerref, 'issue')\n volume = volume[2:] + issue\n journal = 'JHEP'\n page = get_value_in_tag(innerref, 'page-range')\n year = get_value_in_tag(innerref, 'year')\n external_link = get_value_in_tag(innerref, 'ext-link')\n arxiv = ''\n for tag in innerref.getElementsByTagName('pub-id'):\n if tag.getAttribute('pub-id-type') == 'arxiv':\n if tag.hasChildNodes():\n arxiv = get_all_text(tag)\n arxiv = format_arxiv_id(arxiv)\n publisher = get_value_in_tag(innerref, 'publisher-name')\n publisher_location = get_value_in_tag(innerref, 'publisher-loc')\n if publisher_location:\n publisher = publisher_location + ': ' + publisher\n unstructured_text = []\n for child in innerref.childNodes:\n if child.nodeType == child.TEXT_NODE:\n text = child.nodeValue.strip()\n text = re.sub(r'[\\[\\]\\(\\.;\\)]', '', text).strip()\n if text.startswith(','):\n text = text[1:].strip()\n if text.endswith('Report No'):\n text = institution + \" \" + text\n institution = ''\n text = text.strip()\n elif text.endswith(' ed'):\n text += '.'\n elif text.endswith('PhD thesis,'):\n if institution:\n text += ' ' + institution\n institution = ''\n else:\n text = text[:-1]\n elif text.startswith('Seminar,'):\n article_title = get_value_in_tag(innerref, 'article-title')\n text = institution + \" Seminar, \\\"\" + article_title + \"\\\"\"\n institution = ''\n elif text == u'\\u201d':\n text = ''\n ignore_text = ['in', 'pp', 'edited by']\n if text.startswith('Vol'):\n temp = re.sub(r'\\D', '', text)\n if temp:\n volume += temp\n elif len(text) > 1 and text not in ignore_text\\\n and not (text.isdigit() or text[:-1].isdigit()):\n unstructured_text.append(text)\n if unstructured_text:\n unstructured_text = \" \".join(unstructured_text)\n if ref_type == 'book':\n if volume and not volume.lower().startswith('vol'):\n volume = 'Vol ' + volume\n if volume and page:\n volume = volume + ', pp ' + page\n yield ref_type, doi, authors, collaboration, journal, volume, page, year,\\\n label, arxiv, publisher, institution, unstructured_text, external_link,\\\n report_no, editors", "response": "Retrieve the data for a reference."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _add_references(self, rec):\n for ref in self.document.getElementsByTagName('ref'):\n for ref_type, doi, authors, collaboration, journal, volume, page, year,\\\n label, arxiv, publisher, institution, unstructured_text,\\\n external_link, report_no, editors in self._get_reference(ref):\n subfields = []\n if doi:\n subfields.append(('a', doi))\n for author in authors:\n subfields.append(('h', author))\n for editor in editors:\n subfields.append(('e', editor))\n if year:\n subfields.append(('y', year))\n if unstructured_text:\n if page:\n subfields.append(('m', unstructured_text + ', ' + page))\n else:\n subfields.append(('m', unstructured_text))\n if collaboration:\n subfields.append(('c', collaboration))\n if institution:\n subfields.append(('m', institution))\n if publisher:\n subfields.append(('p', publisher))\n if arxiv:\n subfields.append(('r', arxiv))\n if report_no:\n subfields.append(('r', report_no))\n if external_link:\n subfields.append(('u', external_link))\n if label:\n subfields.append(('o', label))\n if ref_type == 'book':\n if journal:\n subfields.append(('t', journal))\n if volume:\n subfields.append(('m', volume))\n elif page and not unstructured_text:\n subfields.append(('m', page))\n else:\n if volume and page:\n subfields.append(('s', journal + \",\" + volume + \",\" + page))\n elif journal:\n subfields.append(('t', journal))\n if ref_type:\n subfields.append(('d', ref_type))\n if not subfields:\n #misc-type references\n try:\n r = ref.getElementsByTagName('mixed-citation')[0]\n text = xml_to_text(r)\n label = text.split()[0]\n text = \" \".join(text.split()[1:])\n subfields.append(('s', text))\n record_add_field(rec, '999', ind1='C', ind2='5', subfields=subfields)\n except IndexError:\n #references without 'mixed-citation' tag\n try:\n r = ref.getElementsByTagName('note')[0]\n subfields.append(('s', xml_to_text(r)))\n record_add_field(rec, '999', ind1='C', ind2='5', subfields=subfields)\n except IndexError:\n #references without 'note' tag\n subfields.append(('s', xml_to_text(ref)))\n record_add_field(rec, '999', ind1='C', ind2='5', subfields=subfields)\n else:\n record_add_field(rec, '999', ind1='C', ind2='5', subfields=subfields)", "response": "Adds the references to the record."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_record(self, xml_file):\n self.document = parse(xml_file)\n\n if get_value_in_tag(self.document, \"meta\"):\n raise ApsPackageXMLError(\"The XML format of %s is not correct\"\n % (xml_file,))\n page_count = self._get_page_count()\n rec = create_record()\n if page_count:\n record_add_field(rec, '300', subfields=[('a', page_count)])\n pacscodes = self._get_pacscodes()\n for pacscode in pacscodes:\n record_add_field(rec, '084', subfields=[('2', 'PACS'),\n ('a', pacscode)])\n subject = self._get_subject()\n if subject:\n record_add_field(rec, '650', ind1='1', ind2='7', subfields=[('2', 'APS'),\n ('a', subject)])\n keywords = self._get_keywords()\n if keywords:\n record_add_field(rec, '653', ind1='1', subfields=[('a', ', '.join(keywords)),\n ('9', 'author')])\n title, subtitle, _ = self._get_title()\n subfields = []\n if subtitle:\n subfields.append(('b', subtitle))\n if title:\n subfields.append(('a', title))\n record_add_field(rec, '245', subfields=subfields)\n journal, volume, issue, year, start_date, doi,\\\n article_id, _, _ = self._get_publication_information()\n if start_date:\n record_add_field(rec, '260', subfields=[('c', start_date),\n ('t', 'published')])\n if doi:\n record_add_field(rec, '024', ind1='7', subfields=[('a', doi),\n ('2', 'DOI')])\n abstract = self._get_abstract()\n if abstract:\n record_add_field(rec, '520', subfields=[('a', abstract),\n ('9', 'APS')])\n license, license_type, license_url = self._get_license()\n subfields = []\n if license:\n subfields.append(('a', license))\n if license_url:\n subfields.append(('u', license_url))\n if subfields:\n record_add_field(rec, '540', subfields=subfields)\n c_holder, c_year, c_statement = self._get_copyright()\n c_holder, c_year, c_statement = self._get_copyright()\n if c_holder and c_year:\n record_add_field(rec, '542', subfields=[('d', c_holder),\n ('g', c_year),\n ('e', 'Article')])\n elif c_statement:\n record_add_field(rec, '542', subfields=[('f', c_statement),\n ('e', 'Article')])\n record_add_field(rec, '773', subfields=[('p', journal),\n ('v', volume),\n ('n', issue),\n ('y', year),\n ('c', article_id)])\n record_add_field(rec, '980', subfields=[('a', 'HEP')])\n record_add_field(rec, '980', subfields=[('a', 'Citeable')])\n record_add_field(rec, '980', subfields=[('a', 'Published')])\n self._add_authors(rec)\n self._add_references(rec)\n try:\n return record_xml_output(rec)\n except UnicodeDecodeError:\n sys.stderr.write(\"\"\"Found a bad char in the file\n for the article \"\"\" + doi)\n return \"\"", "response": "Reads a xml file in JATS format and returns a record in marc format."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef connect(self):\n self._ftp.connect()\n self._ftp.login(user=self._username, passwd=self._passwd)", "response": "Connects and logsins to the server."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndownloads all the files in the folder and all the files in the folder in the target_folder.", "response": "def download_folder(self, folder='', target_folder=''):\n \"\"\" Downloads a whole folder from the server.\n FtpHandler.download_folder() will download all the files\n from the server in the working directory.\n\n :param folder: the absolute path for the folder on the server.\n\n :type folder: string\n :param target_folder: absolute or relative path for the\n destination folder default is the\n working directory.\n :type target_folder: string\n \"\"\"\n files, folders = self.ls(folder)\n for fl in files:\n self.download(join(folder, fl), target_folder)\n for fld in folders:\n self.download_folder(join(folder, fld), target_folder)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef download(self, source_file, target_folder=''):\n current_folder = self._ftp.pwd()\n\n if not target_folder.startswith('/'): # relative path\n target_folder = join(getcwd(), target_folder)\n\n folder = os.path.dirname(source_file)\n self.cd(folder)\n\n if folder.startswith(\"/\"):\n folder = folder[1:]\n\n destination_folder = join(target_folder, folder)\n if not os.path.exists(destination_folder):\n print(\"Creating folder\", destination_folder)\n os.makedirs(destination_folder)\n\n source_file = os.path.basename(source_file)\n destination = join(destination_folder, source_file)\n try:\n with open(destination, 'wb') as result:\n self._ftp.retrbinary('RETR %s' % (source_file,),\n result.write)\n except error_perm as e: # source_file is a folder\n print(e)\n remove(join(target_folder, source_file))\n raise\n self._ftp.cwd(current_folder)", "response": "Downloads a file from the FTP server to a target folder."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchanging the working directory on the server.", "response": "def cd(self, folder):\n \"\"\" Changes the working directory on the server.\n\n :param folder: the desired directory.\n :type folder: string\n \"\"\"\n if folder.startswith('/'):\n self._ftp.cwd(folder)\n else:\n for subfolder in folder.split('/'):\n if subfolder:\n self._ftp.cwd(subfolder)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlist the files and folders of a specific directory.", "response": "def ls(self, folder=''):\n \"\"\" Lists the files and folders of a specific directory\n default is the current working directory.\n\n :param folder: the folder to be listed.\n :type folder: string\n\n :returns: a tuple with the list of files in the folder\n and the list of subfolders in the folder.\n \"\"\"\n current_folder = self._ftp.pwd()\n self.cd(folder)\n contents = []\n self._ftp.retrlines('LIST', lambda a: contents.append(a))\n files = filter(lambda a: a.split()[0].startswith('-'), contents)\n folders = filter(lambda a: a.split()[0].startswith('d'), contents)\n files = map(lambda a: ' '.join(a.split()[8:]), files)\n folders = map(lambda a: ' '.join(a.split()[8:]), folders)\n self._ftp.cwd(current_folder)\n return files, folders"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dir(self, folder='', prefix=''):\n files, folders = self.ls(folder)\n result = files\n inner = []\n for fld in folders:\n try:\n inner += self.dir(folder + '/' + fld, prefix + fld + '/')\n except:\n pass\n result += inner\n if prefix:\n result = map(lambda a: prefix + a, result)\n return result", "response": "Lists all the files on the folder given as parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a folder in the server", "response": "def mkdir(self, folder):\n \"\"\" Creates a folder in the server\n\n :param folder: the folder to be created.\n :type folder: string\n \"\"\"\n current_folder = self._ftp.pwd()\n #creates the necessary folders on\n #the server if they don't exist\n folders = folder.split('/')\n for fld in folders:\n try:\n self.cd(fld)\n except error_perm: # folder does not exist\n self._ftp.mkd(fld)\n self.cd(fld)\n self.cd(current_folder)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndelete a file from the server.", "response": "def rm(self, filename):\n \"\"\" Delete a file from the server.\n\n :param filename: the file to be deleted.\n :type filename: string\n \"\"\"\n try:\n self._ftp.delete(filename)\n except error_perm: # target is either a directory\n # either it does not exist\n try:\n current_folder = self._ftp.pwd()\n self.cd(filename)\n except error_perm:\n print('550 Delete operation failed %s '\n 'does not exist!' % (filename,))\n else:\n self.cd(current_folder)\n print('550 Delete operation failed %s '\n 'is a folder. Use rmdir function '\n 'to delete it.' % (filename,))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rmdir(self, foldername):\n current_folder = self._ftp.pwd()\n try:\n self.cd(foldername)\n except error_perm:\n print('550 Delete operation failed folder %s '\n 'does not exist!' % (foldername,))\n else:\n self.cd(current_folder)\n try:\n self._ftp.rmd(foldername)\n except error_perm: # folder not empty\n self.cd(foldername)\n contents = self.ls()\n #delete the files\n map(self._ftp.delete, contents[0])\n #delete the subfolders\n map(self.rmdir, contents[1])\n self.cd(current_folder)\n self._ftp.rmd(foldername)", "response": "Delete a folder from the server."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_filesize(self, filename):\n result = []\n\n def dir_callback(val):\n result.append(val.split()[4])\n\n self._ftp.dir(filename, dir_callback)\n return result[0]", "response": "Returns the filesize of a file on the server."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef upload(self, filename, location=''):\n current_folder = self._ftp.pwd()\n self.mkdir(location)\n self.cd(location)\n fl = open(filename, 'rb')\n filename = filename.split('/')[-1]\n self._ftp.storbinary('STOR %s' % filename, fl)\n fl.close()\n self.cd(current_folder)", "response": "Uploads a file to the server to the desired location."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a block of text and returns a dictionary of user - defined data.", "response": "def parse_data(self, text, maxwidth, maxheight, template_dir, context,\n urlize_all_links):\n \"\"\"\n Parses a block of text indiscriminately\n \"\"\"\n # create a dictionary of user urls -> rendered responses\n replacements = {}\n user_urls = set(re.findall(URL_RE, text))\n \n for user_url in user_urls:\n try:\n resource = oembed.site.embed(user_url, maxwidth=maxwidth, maxheight=maxheight)\n except OEmbedException:\n if urlize_all_links:\n replacements[user_url] = '%(LINK)s' % {'LINK': user_url}\n else:\n context['minwidth'] = min(maxwidth, resource.width)\n context['minheight'] = min(maxheight, resource.height)\n \n replacement = self.render_oembed(\n resource, \n user_url, \n template_dir=template_dir, \n context=context\n )\n replacements[user_url] = replacement.strip()\n \n # go through the text recording URLs that can be replaced\n # taking note of their start & end indexes\n user_urls = re.finditer(URL_RE, text)\n matches = []\n for match in user_urls:\n if match.group() in replacements:\n matches.append([match.start(), match.end(), match.group()])\n \n # replace the URLs in order, offsetting the indices each go\n for indx, (start, end, user_url) in enumerate(matches):\n replacement = replacements[user_url]\n difference = len(replacement) - len(user_url)\n \n # insert the replacement between two slices of text surrounding the\n # original url\n text = text[:start] + replacement + text[end:]\n \n # iterate through the rest of the matches offsetting their indices\n # based on the difference between replacement/original\n for j in xrange(indx + 1, len(matches)):\n matches[j][0] += difference\n matches[j][1] += difference\n return mark_safe(text)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_data(self, text, maxwidth, maxheight, template_dir, context, \n urlize_all_links):\n \"\"\"\n Parses a block of text rendering links that occur on their own line\n normally but rendering inline links using a special template dir\n \"\"\"\n block_parser = TextBlockParser()\n \n lines = text.splitlines()\n parsed = []\n \n for line in lines:\n if STANDALONE_URL_RE.match(line):\n user_url = line.strip()\n try:\n resource = oembed.site.embed(user_url, maxwidth=maxwidth, maxheight=maxheight)\n context['minwidth'] = min(maxwidth, resource.width)\n context['minheight'] = min(maxheight, resource.height)\n except OEmbedException:\n if urlize_all_links:\n line = '%(LINK)s' % {'LINK': user_url}\n else:\n context['minwidth'] = min(maxwidth, resource.width)\n context['minheight'] = min(maxheight, resource.height)\n \n line = self.render_oembed(\n resource, \n user_url, \n template_dir=template_dir, \n context=context)\n else:\n line = block_parser.parse(line, maxwidth, maxheight, 'inline',\n context, urlize_all_links)\n \n parsed.append(line)\n \n return mark_safe('\\n'.join(parsed))", "response": "Parses a block of text rendering links that occur on their own line using a special template dir"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef login(email=None, password=None, api_key=None, application='Default',\n url=None, verify_ssl_certificate=True):\n \"\"\"\n Do the legwork of logging into the Midas Server instance, storing the API\n key and token.\n\n :param email: (optional) Email address to login with. If not set, the\n console will be prompted.\n :type email: None | string\n :param password: (optional) User password to login with. If not set and no\n 'api_key' is set, the console will be prompted.\n :type password: None | string\n :param api_key: (optional) API key to login with. If not set, password\n login with be used.\n :type api_key: None | string\n :param application: (optional) Application name to be used with 'api_key'.\n :type application: string\n :param url: (optional) URL address of the Midas Server instance to login\n to. If not set, the console will be prompted.\n :type url: None | string\n :param verify_ssl_certificate: (optional) If True, the SSL certificate will\n be verified\n :type verify_ssl_certificate: bool\n :returns: API token.\n :rtype: string\n \"\"\"\n try:\n input_ = raw_input\n except NameError:\n input_ = input\n\n if url is None:\n url = input_('Server URL: ')\n url = url.rstrip('/')\n if session.communicator is None:\n session.communicator = Communicator(url)\n else:\n session.communicator.url = url\n\n session.communicator.verify_ssl_certificate = verify_ssl_certificate\n\n if email is None:\n email = input_('Email: ')\n session.email = email\n\n if api_key is None:\n if password is None:\n password = getpass.getpass()\n session.api_key = session.communicator.get_default_api_key(\n session.email, password)\n session.application = 'Default'\n else:\n session.api_key = api_key\n session.application = application\n\n return renew_token()", "response": "Login to the Midas Server instance."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrenew or get a token to use for transactions with the Midas Server instance.", "response": "def renew_token():\n \"\"\"\n Renew or get a token to use for transactions with the Midas Server\n instance.\n\n :returns: API token.\n :rtype: string\n \"\"\"\n session.token = session.communicator.login_with_api_key(\n session.email, session.api_key, application=session.application)\n if len(session.token) < 10: # HACK to check for mfa being enabled\n one_time_pass = getpass.getpass('One-Time Password: ')\n session.token = session.communicator.mfa_otp_login(\n session.token, one_time_pass)\n return session.token"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _create_or_reuse_item(local_file, parent_folder_id, reuse_existing=False):\n local_item_name = os.path.basename(local_file)\n item_id = None\n if reuse_existing:\n # check by name to see if the item already exists in the folder\n children = session.communicator.folder_children(\n session.token, parent_folder_id)\n items = children['items']\n\n for item in items:\n if item['name'] == local_item_name:\n item_id = item['item_id']\n break\n\n if item_id is None:\n # create the item for the subdir\n new_item = session.communicator.create_item(\n session.token, local_item_name, parent_folder_id)\n item_id = new_item['item_id']\n\n return item_id", "response": "Create an item from the local file in the Midas Server folder corresponding to the parent folder id."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating or reuse a new MIDI site site", "response": "def _create_or_reuse_folder(local_folder, parent_folder_id,\n reuse_existing=False):\n \"\"\"\n Create a folder from the local file in the midas folder corresponding to\n the parent folder id.\n\n :param local_folder: full path to a directory on the local file system\n :type local_folder: string\n :param parent_folder_id: id of parent folder on the Midas Server instance,\n where the folder will be added\n :type parent_folder_id: int | long\n :param reuse_existing: (optional) whether to accept an existing folder of\n the same name in the same location, or create a new one instead\n :type reuse_existing: bool\n \"\"\"\n local_folder_name = os.path.basename(local_folder)\n folder_id = None\n if reuse_existing:\n # check by name to see if the folder already exists in the folder\n children = session.communicator.folder_children(\n session.token, parent_folder_id)\n folders = children['folders']\n\n for folder in folders:\n if folder['name'] == local_folder_name:\n folder_id = folder['folder_id']\n break\n\n if folder_id is None:\n # create the item for the subdir\n new_folder = session.communicator.create_folder(session.token,\n local_folder_name,\n parent_folder_id)\n folder_id = new_folder['folder_id']\n\n return folder_id"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _streaming_file_md5(file_path):\n md5 = hashlib.md5()\n with open(file_path, 'rb') as f:\n # iter needs an empty byte string for the returned iterator to halt at\n # EOF\n for chunk in iter(lambda: f.read(128 * md5.block_size), b''):\n md5.update(chunk)\n return md5.hexdigest()", "response": "Create and return a hex checksum using the MD5 sum of the passed in file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new bitstream in the given item.", "response": "def _create_bitstream(file_path, local_file, item_id, log_ind=None):\n \"\"\"\n Create a bitstream in the given item.\n\n :param file_path: full path to the local file\n :type file_path: string\n :param local_file: name of the local file\n :type local_file: string\n :param log_ind: (optional) any additional message to log upon creation of\n the bitstream\n :type log_ind: None | string\n \"\"\"\n checksum = _streaming_file_md5(file_path)\n upload_token = session.communicator.generate_upload_token(\n session.token, item_id, local_file, checksum)\n\n if upload_token != '':\n log_trace = 'Uploading bitstream from {0}'.format(file_path)\n # only need to perform the upload if we haven't uploaded before\n # in this cae, the upload token would not be empty\n session.communicator.perform_upload(\n upload_token, local_file, filepath=file_path, itemid=item_id)\n else:\n log_trace = 'Adding a bitstream link in this item to an existing ' \\\n 'bitstream from {0}'.format(file_path)\n\n if log_ind is not None:\n log_trace += log_ind\n print(log_trace)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfunction for doing an upload of a file as an item. This should be a building block for user-level functions. :param local_file: name of local file to upload :type local_file: string :param parent_folder_id: id of parent folder on the Midas Server instance, where the item will be added :type parent_folder_id: int | long :param file_path: full path to the file :type file_path: string :param reuse_existing: (optional) whether to accept an existing item of the same name in the same location, or create a new one instead :type reuse_existing: bool", "response": "def _upload_as_item(local_file, parent_folder_id, file_path,\n reuse_existing=False):\n \"\"\"\n Function for doing an upload of a file as an item. This should be a\n building block for user-level functions.\n\n :param local_file: name of local file to upload\n :type local_file: string\n :param parent_folder_id: id of parent folder on the Midas Server instance,\n where the item will be added\n :type parent_folder_id: int | long\n :param file_path: full path to the file\n :type file_path: string\n :param reuse_existing: (optional) whether to accept an existing item of the\n same name in the same location, or create a new one instead\n :type reuse_existing: bool\n \"\"\"\n current_item_id = _create_or_reuse_item(local_file, parent_folder_id,\n reuse_existing)\n _create_bitstream(file_path, local_file, current_item_id)\n for callback in session.item_upload_callbacks:\n callback(session.communicator, session.token, current_item_id)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction for creating a remote folder and returning the id.", "response": "def _create_folder(local_folder, parent_folder_id):\n \"\"\"\n Function for creating a remote folder and returning the id. This should be\n a building block for user-level functions.\n\n :param local_folder: full path to a local folder\n :type local_folder: string\n :param parent_folder_id: id of parent folder on the Midas Server instance,\n where the new folder will be added\n :type parent_folder_id: int | long\n :returns: id of the remote folder that was created\n :rtype: int | long\n \"\"\"\n new_folder = session.communicator.create_folder(\n session.token, os.path.basename(local_folder), parent_folder_id)\n return new_folder['folder_id']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _upload_folder_recursive(local_folder,\n parent_folder_id,\n leaf_folders_as_items=False,\n reuse_existing=False):\n \"\"\"\n Function to recursively upload a folder and all of its descendants.\n\n :param local_folder: full path to local folder to be uploaded\n :type local_folder: string\n :param parent_folder_id: id of parent folder on the Midas Server instance,\n where the new folder will be added\n :type parent_folder_id: int | long\n :param leaf_folders_as_items: (optional) whether leaf folders should have\n all files uploaded as single items\n :type leaf_folders_as_items: bool\n :param reuse_existing: (optional) whether to accept an existing item of the\n same name in the same location, or create a new one instead\n :type reuse_existing: bool\n \"\"\"\n if leaf_folders_as_items and _has_only_files(local_folder):\n print('Creating item from {0}'.format(local_folder))\n _upload_folder_as_item(local_folder, parent_folder_id, reuse_existing)\n return\n else:\n # do not need to check if folder exists, if it does, an attempt to\n # create it will just return the existing id\n print('Creating folder from {0}'.format(local_folder))\n new_folder_id = _create_or_reuse_folder(local_folder, parent_folder_id,\n reuse_existing)\n\n for entry in sorted(os.listdir(local_folder)):\n full_entry = os.path.join(local_folder, entry)\n if os.path.islink(full_entry):\n # os.walk skips symlinks by default\n continue\n elif os.path.isdir(full_entry):\n _upload_folder_recursive(full_entry,\n new_folder_id,\n leaf_folders_as_items,\n reuse_existing)\n else:\n print('Uploading item from {0}'.format(full_entry))\n _upload_as_item(entry,\n new_folder_id,\n full_entry,\n reuse_existing)", "response": "Function to recursively upload a folder and all its descendants."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if a folder contains only files. This will be False if the folder contains any subdirectories.", "response": "def _has_only_files(local_folder):\n \"\"\"\n Return whether a folder contains only files. This will be False if the\n folder contains any subdirectories.\n\n :param local_folder: full path to the local folder\n :type local_folder: string\n :returns: True if the folder contains only files\n :rtype: bool\n \"\"\"\n return not any(os.path.isdir(os.path.join(local_folder, entry))\n for entry in os.listdir(local_folder))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupload a folder as a new item.", "response": "def _upload_folder_as_item(local_folder, parent_folder_id,\n reuse_existing=False):\n \"\"\"\n Upload a folder as a new item. Take a folder and use its base name as the\n name of a new item. Then, upload its containing files into the new item as\n bitstreams.\n\n :param local_folder: The path to the folder to be uploaded\n :type local_folder: string\n :param parent_folder_id: The id of the destination folder for the new item.\n :type parent_folder_id: int | long\n :param reuse_existing: (optional) whether to accept an existing item of the\n same name in the same location, or create a new one instead\n :type reuse_existing: bool\n \"\"\"\n item_id = _create_or_reuse_item(local_folder, parent_folder_id,\n reuse_existing)\n\n subdir_contents = sorted(os.listdir(local_folder))\n # for each file in the subdir, add it to the item\n filecount = len(subdir_contents)\n for (ind, current_file) in enumerate(subdir_contents):\n file_path = os.path.join(local_folder, current_file)\n log_ind = '({0} of {1})'.format(ind + 1, filecount)\n _create_bitstream(file_path, current_file, item_id, log_ind)\n\n for callback in session.item_upload_callbacks:\n callback(session.communicator, session.token, item_id)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef upload(file_pattern, destination='Private', leaf_folders_as_items=False,\n reuse_existing=False):\n \"\"\"\n Upload a pattern of files. This will recursively walk down every tree in\n the file pattern to create a hierarchy on the server. As of right now, this\n places the file into the currently logged in user's home directory.\n\n :param file_pattern: a glob type pattern for files\n :type file_pattern: string\n :param destination: (optional) name of the midas destination folder,\n defaults to Private\n :type destination: string\n :param leaf_folders_as_items: (optional) whether leaf folders should have\n all files uploaded as single items\n :type leaf_folders_as_items: bool\n :param reuse_existing: (optional) whether to accept an existing item of the\n same name in the same location, or create a new one instead\n :type reuse_existing: bool\n \"\"\"\n session.token = verify_credentials()\n\n # Logic for finding the proper folder to place the files in.\n parent_folder_id = None\n user_folders = session.communicator.list_user_folders(session.token)\n if destination.startswith('/'):\n parent_folder_id = _find_resource_id_from_path(destination)\n else:\n for cur_folder in user_folders:\n if cur_folder['name'] == destination:\n parent_folder_id = cur_folder['folder_id']\n if parent_folder_id is None:\n print('Unable to locate specified destination. Defaulting to {0}.'\n .format(user_folders[0]['name']))\n parent_folder_id = user_folders[0]['folder_id']\n\n for current_file in glob.iglob(file_pattern):\n current_file = os.path.normpath(current_file)\n if os.path.isfile(current_file):\n print('Uploading item from {0}'.format(current_file))\n _upload_as_item(os.path.basename(current_file),\n parent_folder_id,\n current_file,\n reuse_existing)\n else:\n _upload_folder_recursive(current_file,\n parent_folder_id,\n leaf_folders_as_items,\n reuse_existing)", "response": "Upload a pattern of files to the midas server."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndescend a path to return a folder id starting from the given folder id.", "response": "def _descend_folder_for_id(parsed_path, folder_id):\n \"\"\"\n Descend a path to return a folder id starting from the given folder id.\n\n :param parsed_path: a list of folders from top to bottom of a hierarchy\n :type parsed_path: list[string]\n :param folder_id: The id of the folder from which to start the descent\n :type folder_id: int | long\n :returns: The id of the found folder or -1\n :rtype: int | long\n \"\"\"\n if len(parsed_path) == 0:\n return folder_id\n\n session.token = verify_credentials()\n\n base_folder = session.communicator.folder_get(session.token,\n folder_id)\n cur_folder_id = -1\n for path_part in parsed_path:\n cur_folder_id = base_folder['folder_id']\n cur_children = session.communicator.folder_children(\n session.token, cur_folder_id)\n for inner_folder in cur_children['folders']:\n if inner_folder['name'] == path_part:\n base_folder = session.communicator.folder_get(\n session.token, inner_folder['folder_id'])\n cur_folder_id = base_folder['folder_id']\n break\n else:\n return -1\n return cur_folder_id"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsearching the folder for an item or folder matching the name.", "response": "def _search_folder_for_item_or_folder(name, folder_id):\n \"\"\"\n Find an item or folder matching the name. A folder will be found first if\n both are present.\n\n :param name: The name of the resource\n :type name: string\n :param folder_id: The folder to search within\n :type folder_id: int | long\n :returns: A tuple indicating whether the resource is an item an the id of\n said resource. i.e. (True, item_id) or (False, folder_id). Note that in\n the event that we do not find a result return (False, -1)\n :rtype: (bool, int | long)\n \"\"\"\n session.token = verify_credentials()\n\n children = session.communicator.folder_children(session.token, folder_id)\n for folder in children['folders']:\n if folder['name'] == name:\n return False, folder['folder_id'] # Found a folder\n for item in children['items']:\n if item['name'] == name:\n return True, item['item_id'] # Found an item\n return False, -1"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds a resource id from a virtual path on the server.", "response": "def _find_resource_id_from_path(path):\n \"\"\"\n Get a folder id from a path on the server.\n\n Warning: This is NOT efficient at all.\n\n The schema for this path is:\n path := \"/users//\" | \"/communities/\" , {/}\n name := , \"_\" , \n\n :param path: The virtual path on the server.\n :type path: string\n :returns: a tuple indicating True or False about whether the resource is an\n item and id of the resource i.e. (True, item_id) or (False, folder_id)\n :rtype: (bool, int | long)\n \"\"\"\n session.token = verify_credentials()\n\n parsed_path = path.split('/')\n if parsed_path[-1] == '':\n parsed_path.pop()\n if path.startswith('/users/'):\n parsed_path.pop(0) # remove '' before /\n parsed_path.pop(0) # remove 'users'\n name = parsed_path.pop(0) # remove '_'\n firstname, lastname = name.split('_')\n end = parsed_path.pop()\n user = session.communicator.get_user_by_name(firstname, lastname)\n leaf_folder_id = _descend_folder_for_id(parsed_path, user['folder_id'])\n return _search_folder_for_item_or_folder(end, leaf_folder_id)\n elif path.startswith('/communities/'):\n print(parsed_path)\n parsed_path.pop(0) # remove '' before /\n parsed_path.pop(0) # remove 'communities'\n community_name = parsed_path.pop(0) # remove ''\n end = parsed_path.pop()\n community = session.communicator.get_community_by_name(community_name)\n leaf_folder_id = _descend_folder_for_id(parsed_path,\n community['folder_id'])\n return _search_folder_for_item_or_folder(end, leaf_folder_id)\n else:\n return False, -1"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndownload a folder to the specified path along with any children. :param folder_id: The id of the target folder :type folder_id: int | long :param path: (optional) the location to download the folder :type path: string", "response": "def _download_folder_recursive(folder_id, path='.'):\n \"\"\"\n Download a folder to the specified path along with any children.\n\n :param folder_id: The id of the target folder\n :type folder_id: int | long\n :param path: (optional) the location to download the folder\n :type path: string\n \"\"\"\n session.token = verify_credentials()\n\n cur_folder = session.communicator.folder_get(session.token, folder_id)\n # Replace any '/' in the folder name.\n folder_path = os.path.join(path, cur_folder['name'].replace('/', '_'))\n print('Creating folder at {0}'.format(folder_path))\n try:\n os.mkdir(folder_path)\n except OSError as e:\n if e.errno == errno.EEXIST and session.allow_existing_download_paths:\n pass\n else:\n raise\n cur_children = session.communicator.folder_children(\n session.token, folder_id)\n for item in cur_children['items']:\n _download_item(item['item_id'], folder_path, item=item)\n for folder in cur_children['folders']:\n _download_folder_recursive(folder['folder_id'], folder_path)\n for callback in session.folder_download_callbacks:\n callback(session.communicator, session.token, cur_folder, folder_path)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndownload the requested item to the specified path.", "response": "def _download_item(item_id, path='.', item=None):\n \"\"\"\n Download the requested item to the specified path.\n\n :param item_id: The id of the item to be downloaded\n :type item_id: int | long\n :param path: (optional) the location to download the item\n :type path: string\n :param item: The dict of item info\n :type item: dict | None\n \"\"\"\n session.token = verify_credentials()\n\n filename, content_iter = session.communicator.download_item(\n item_id, session.token)\n item_path = os.path.join(path, filename)\n print('Creating file at {0}'.format(item_path))\n out_file = open(item_path, 'wb')\n for block in content_iter:\n out_file.write(block)\n out_file.close()\n for callback in session.item_download_callbacks:\n if not item:\n item = session.communicator.item_get(session.token, item_id)\n callback(session.communicator, session.token, item, item_path)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndownloading a file or item from the Midas Server instance.", "response": "def download(server_path, local_path='.'):\n \"\"\"\n Recursively download a file or item from the Midas Server instance.\n\n :param server_path: The location on the server to find the resource to\n download\n :type server_path: string\n :param local_path: The location on the client to store the downloaded data\n :type local_path: string\n \"\"\"\n session.token = verify_credentials()\n\n is_item, resource_id = _find_resource_id_from_path(server_path)\n if resource_id == -1:\n print('Unable to locate {0}'.format(server_path))\n else:\n if is_item:\n _download_item(resource_id, local_path)\n else:\n _download_folder_recursive(resource_id, local_path)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef request(self, method, parameters=None, file_payload=None):\n method_url = self.full_url + method\n response = None\n\n try:\n if file_payload:\n response = requests.put(method_url,\n data=file_payload.read(),\n params=parameters,\n allow_redirects=True,\n verify=self._verify_ssl_certificate,\n auth=self.auth)\n else:\n response = requests.post(method_url,\n params=parameters,\n allow_redirects=True,\n verify=self._verify_ssl_certificate,\n auth=self.auth)\n\n except requests.exceptions.SSLError:\n exception = pydas.exceptions.SSLVerificationFailed(\n 'Request failed with an SSL verification error')\n exception.method = method\n exception.request = response.request\n raise exception\n\n except requests.exceptions.ConnectionError:\n exception = pydas.exceptions.RequestError(\n 'Request failed with a connection error')\n exception.method = method\n if response is not None:\n exception.request = response.request\n raise exception\n\n status_code = response.status_code\n\n try:\n response.raise_for_status()\n\n except requests.exceptions.HTTPError:\n error_code = None\n message = 'Request failed with HTTP status code {0}'.format(\n status_code)\n\n try:\n content = response.json()\n\n if 'code' in content:\n error_code = int(content['code'])\n message = 'Request failed with HTTP status code {0}, ' \\\n 'Midas Server error code {1}, and response content ' \\\n '{2}'.format(status_code, error_code, response.content)\n\n except ValueError:\n pass\n\n exception = pydas.exceptions \\\n .get_exception_from_status_and_error_codes(status_code,\n error_code,\n message)\n exception.code = error_code\n exception.method = method\n exception.response = response\n raise exception\n\n try:\n content = response.json()\n\n except ValueError:\n exception = pydas.exceptions.ParseError(\n 'Request failed with HTTP status code {0} and response '\n 'content {1}'.format(status_code, response.content))\n exception.method = method\n exception.response = response\n raise exception\n\n if 'stat' not in content:\n exception = pydas.exceptions.ParseError(\n 'Request failed with HTTP status code {0} and response '\n 'content {1}'.format(status_code, response.content))\n exception.method = method\n raise exception\n\n if content['stat'] != 'ok':\n if 'code' in content:\n error_code = int(content['code'])\n message = 'Request failed with HTTP status code {0}, Midas ' \\\n 'Server error code {1}, and response content {2}' \\\n .format(status_code, error_code, response.content)\n else:\n error_code = None\n message = 'Request failed with HTTP status code {0} and ' \\\n 'response content {1}'.format(status_code,\n response.content)\n\n exception = pydas.exceptions \\\n .get_exception_from_status_and_error_codes(status_code,\n error_code,\n message)\n exception.method = method\n exception.response = response\n raise exception\n\n if 'data' not in content:\n exception = pydas.exceptions.ParseError(\n 'Request failed with HTTP status code {0} and response '\n 'content {1}'.format(status_code, response.content))\n exception.method = method\n exception.response = response\n raise exception\n\n return content['data']", "response": "This method performs the actual HTTP request to the desired API."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef login_with_api_key(self, email, api_key, application='Default'):\n parameters = dict()\n parameters['email'] = BaseDriver.email = email # Cache email\n parameters['apikey'] = BaseDriver.apikey = api_key # Cache API key\n parameters['appname'] = application\n response = self.request('midas.login', parameters)\n if 'token' in response: # normal case\n return response['token']\n if 'mfa_token_id': # case with multi-factor authentication\n return response['mfa_token_id']", "response": "Login and get a token."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlists the user s folders in the users home area.", "response": "def list_user_folders(self, token):\n \"\"\"\n List the folders in the users home area.\n\n :param token: A valid token for the user in question.\n :type token: string\n :returns: List of dictionaries containing folder information.\n :rtype: list[dict]\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n response = self.request('midas.user.folders', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_default_api_key(self, email, password):\n parameters = dict()\n parameters['email'] = email\n parameters['password'] = password\n response = self.request('midas.user.apikey.default', parameters)\n return response['apikey']", "response": "Get the default API key for a user."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list_users(self, limit=20):\n parameters = dict()\n parameters['limit'] = limit\n response = self.request('midas.user.list', parameters)\n return response", "response": "List the public users in the system."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting a user by the first and last name of that user.", "response": "def get_user_by_name(self, firstname, lastname):\n \"\"\"\n Get a user by the first and last name of that user.\n\n :param firstname: The first name of the user.\n :type firstname: string\n :param lastname: The last name of the user.\n :type lastname: string\n :returns: The user requested.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['firstname'] = firstname\n parameters['lastname'] = lastname\n response = self.request('midas.user.get', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a user by the first and last name of that user.", "response": "def get_user_by_id(self, user_id):\n \"\"\"\n Get a user by the first and last name of that user.\n\n :param user_id: The id of the desired user.\n :type user_id: int | long\n :returns: The user requested.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['user_id'] = user_id\n response = self.request('midas.user.get', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a user by email.", "response": "def get_user_by_email(self, email):\n \"\"\"\n Get a user by the email of that user.\n\n :param email: The email of the desired user.\n :type email: string\n :returns: The user requested.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['email'] = email\n response = self.request('midas.user.get', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new community or update an existing one using the uuid.", "response": "def create_community(self, token, name, **kwargs):\n \"\"\"\n Create a new community or update an existing one using the uuid.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param name: The community name.\n :type name: string\n :param description: (optional) The community description.\n :type description: string\n :param uuid: (optional) uuid of the community. If none is passed, will\n generate one.\n :type uuid: string\n :param privacy: (optional) Default 'Public', possible values\n [Public|Private].\n :type privacy: string\n :param can_join: (optional) Default 'Everyone', possible values\n [Everyone|Invitation].\n :type can_join: string\n :returns: The community dao that was created.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['name'] = name\n optional_keys = ['description', 'uuid', 'privacy', 'can_join']\n for key in optional_keys:\n if key in kwargs:\n if key == 'can_join':\n parameters['canjoin'] = kwargs[key]\n continue\n parameters[key] = kwargs[key]\n response = self.request('midas.community.create', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a community based on its name.", "response": "def get_community_by_name(self, name, token=None):\n \"\"\"\n Get a community based on its name.\n\n :param name: The name of the target community.\n :type name: string\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: The requested community.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['name'] = name\n if token:\n parameters['token'] = token\n response = self.request('midas.community.get', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_community_by_id(self, community_id, token=None):\n parameters = dict()\n parameters['id'] = community_id\n if token:\n parameters['token'] = token\n response = self.request('midas.community.get', parameters)\n return response", "response": "Get a community based on its id."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the non - recursive children of the passed in community.", "response": "def get_community_children(self, community_id, token=None):\n \"\"\"\n Get the non-recursive children of the passed in community_id.\n\n :param community_id: The id of the requested community.\n :type community_id: int | long\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: List of the folders in the community.\n :rtype: dict[string, list]\n \"\"\"\n parameters = dict()\n parameters['id'] = community_id\n if token:\n parameters['token'] = token\n response = self.request('midas.community.children', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlist all communities visible to a user.", "response": "def list_communities(self, token=None):\n \"\"\"\n List all communities visible to a user.\n\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: The list of communities.\n :rtype: list[dict]\n \"\"\"\n parameters = dict()\n if token:\n parameters['token'] = token\n response = self.request('midas.community.list', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef folder_get(self, token, folder_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = folder_id\n response = self.request('midas.folder.get', parameters)\n return response", "response": "Get the attributes of the specified folder."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef folder_children(self, token, folder_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = folder_id\n response = self.request('midas.folder.children', parameters)\n return response", "response": "Get the non - recursive children of the passed in folder_id."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef delete_folder(self, token, folder_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = folder_id\n response = self.request('midas.folder.delete', parameters)\n return response", "response": "Delete the folder with the passed in id."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmoves a folder to another folder.", "response": "def move_folder(self, token, folder_id, dest_folder_id):\n \"\"\"\n Move a folder to the destination folder.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param folder_id: The id of the folder to be moved.\n :type folder_id: int | long\n :param dest_folder_id: The id of destination (new parent) folder.\n :type dest_folder_id: int | long\n :returns: Dictionary containing the details of the moved folder.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = folder_id\n parameters['dstfolderid'] = dest_folder_id\n response = self.request('midas.folder.move', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate an item in the server.", "response": "def create_item(self, token, name, parent_id, **kwargs):\n \"\"\"\n Create an item to the server.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param name: The name of the item to be created.\n :type name: string\n :param parent_id: The id of the destination folder.\n :type parent_id: int | long\n :param description: (optional) The description text of the item.\n :type description: string\n :param uuid: (optional) The UUID for the item. It will be generated if\n not given.\n :type uuid: string\n :param privacy: (optional) The privacy state of the item\n ('Public' or 'Private').\n :type privacy: string\n :returns: Dictionary containing the details of the created item.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['name'] = name\n parameters['parentid'] = parent_id\n optional_keys = ['description', 'uuid', 'privacy']\n for key in optional_keys:\n if key in kwargs:\n parameters[key] = kwargs[key]\n response = self.request('midas.item.create', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the attributes of the specified item.", "response": "def item_get(self, token, item_id):\n \"\"\"\n Get the attributes of the specified item.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: The id of the requested item.\n :type item_id: int | string\n :returns: Dictionary of the item attributes.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = item_id\n response = self.request('midas.item.get', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndownload an item from the server to disk.", "response": "def download_item(self, item_id, token=None, revision=None):\n \"\"\"\n Download an item to disk.\n\n :param item_id: The id of the item to be downloaded.\n :type item_id: int | long\n :param token: (optional) The authentication token of the user\n requesting the download.\n :type token: None | string\n :param revision: (optional) The revision of the item to download, this\n defaults to HEAD.\n :type revision: None | int | long\n :returns: A tuple of the filename and the content iterator.\n :rtype: (string, unknown)\n \"\"\"\n parameters = dict()\n parameters['id'] = item_id\n if token:\n parameters['token'] = token\n if revision:\n parameters['revision'] = revision\n method_url = self.full_url + 'midas.item.download'\n request = requests.get(method_url,\n params=parameters,\n stream=True,\n verify=self._verify_ssl_certificate)\n filename = request.headers['content-disposition'][21:].strip('\"')\n return filename, request.iter_content(chunk_size=10 * 1024)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_item(self, token, item_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = item_id\n response = self.request('midas.item.delete', parameters)\n return response", "response": "Delete the item with the passed in item_id."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_item_metadata(self, item_id, token=None, revision=None):\n parameters = dict()\n parameters['id'] = item_id\n if token:\n parameters['token'] = token\n if revision:\n parameters['revision'] = revision\n response = self.request('midas.item.getmetadata', parameters)\n return response", "response": "Get the metadata associated with an item."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the metadata associated with an item.", "response": "def set_item_metadata(self, token, item_id, element, value,\n qualifier=None):\n \"\"\"\n Set the metadata associated with an item.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: The id of the item for which metadata will be set.\n :type item_id: int | long\n :param element: The metadata element name.\n :type element: string\n :param value: The metadata value for the field.\n :type value: string\n :param qualifier: (optional) The metadata qualifier. Defaults to empty\n string.\n :type qualifier: None | string\n :returns: None.\n :rtype: None\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['itemId'] = item_id\n parameters['element'] = element\n parameters['value'] = value\n if qualifier:\n parameters['qualifier'] = qualifier\n response = self.request('midas.item.setmetadata', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef share_item(self, token, item_id, dest_folder_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = item_id\n parameters['dstfolderid'] = dest_folder_id\n response = self.request('midas.item.share', parameters)\n return response", "response": "Share an item to a destination folder."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef move_item(self, token, item_id, src_folder_id, dest_folder_id):\n parameters = dict()\n parameters['token'] = token\n parameters['id'] = item_id\n parameters['srcfolderid'] = src_folder_id\n parameters['dstfolderid'] = dest_folder_id\n response = self.request('midas.item.move', parameters)\n return response", "response": "Move an item from one source folder to another."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch for items with the given name.", "response": "def search_item_by_name(self, name, token=None):\n \"\"\"\n Return all items.\n\n :param name: The name of the item to search by.\n :type name: string\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: A list of all items with the given name.\n :rtype: list[dict]\n \"\"\"\n parameters = dict()\n parameters['name'] = name\n if token:\n parameters['token'] = token\n response = self.request('midas.item.searchbyname', parameters)\n return response['items']"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsearch for items with a given name and parent folder id.", "response": "def search_item_by_name_and_folder(self, name, folder_id, token=None):\n \"\"\"\n Return all items with a given name and parent folder id.\n\n :param name: The name of the item to search by.\n :type name: string\n :param folder_id: The id of the parent folder to search by.\n :type folder_id: int | long\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: A list of all items with the given name and parent folder id.\n :rtype: list[dict]\n \"\"\"\n parameters = dict()\n parameters['name'] = name\n parameters['folderId'] = folder_id\n if token:\n parameters['token'] = token\n response = self.request('midas.item.searchbynameandfolder', parameters)\n return response['items']"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsearches for items with a given name and parent folder name.", "response": "def search_item_by_name_and_folder_name(self, name, folder_name,\n token=None):\n \"\"\"\n Return all items with a given name and parent folder name.\n\n :param name: The name of the item to search by.\n :type name: string\n :param folder_name: The name of the parent folder to search by.\n :type folder_name: string\n :param token: (optional) A valid token for the user in question.\n :type token: None | string\n :returns: A list of all items with the given name and parent folder\n name.\n :rtype: list[dict]\n \"\"\"\n parameters = dict()\n parameters['name'] = name\n parameters['folderName'] = folder_name\n if token:\n parameters['token'] = token\n response = self.request('midas.item.searchbynameandfoldername',\n parameters)\n return response['items']"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a link in the MIDI database.", "response": "def create_link(self, token, folder_id, url, **kwargs):\n \"\"\"\n Create a link bitstream.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param folder_id: The id of the folder in which to create a new item\n that will contain the link. The new item will have the same name as\n the URL unless an item name is supplied.\n :type folder_id: int | long\n :param url: The URL of the link you will create, will be used as the\n name of the bitstream and of the item unless an item name is\n supplied.\n :type url: string\n :param item_name: (optional) The name of the newly created item, if\n not supplied, the item will have the same name as the URL.\n :type item_name: string\n :param length: (optional) The length in bytes of the file to which the\n link points.\n :type length: int | long\n :param checksum: (optional) The MD5 checksum of the file to which the\n link points.\n :type checksum: string\n :returns: The item information of the item created.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['folderid'] = folder_id\n parameters['url'] = url\n optional_keys = ['item_name', 'length', 'checksum']\n for key in optional_keys:\n if key in kwargs:\n if key == 'item_name':\n parameters['itemname'] = kwargs[key]\n continue\n parameters[key] = kwargs[key]\n response = self.request('midas.link.create', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a token for the upload of a file.", "response": "def generate_upload_token(self, token, item_id, filename, checksum=None):\n \"\"\"\n Generate a token to use for upload.\n\n Midas Server uses a individual token for each upload. The token\n corresponds to the file specified and that file only. Passing the MD5\n checksum allows the server to determine if the file is already in the\n asset store.\n\n If :param:`checksum` is passed and the token returned is blank, the\n server already has this file and there is no need to follow this\n call with a call to `perform_upload`, as the passed in file will have\n been added as a bitstream to the item's latest revision, creating a\n new revision if one doesn't exist.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: The id of the item in which to upload the file as a\n bitstream.\n :type item_id: int | long\n :param filename: The name of the file to generate the upload token for.\n :type filename: string\n :param checksum: (optional) The checksum of the file to upload.\n :type checksum: None | string\n :returns: String of the upload token.\n :rtype: string\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['itemid'] = item_id\n parameters['filename'] = filename\n if checksum is not None:\n parameters['checksum'] = checksum\n response = self.request('midas.upload.generatetoken', parameters)\n return response['token']"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef perform_upload(self, upload_token, filename, **kwargs):\n parameters = dict()\n parameters['uploadtoken'] = upload_token\n parameters['filename'] = filename\n\n try:\n create_additional_revision = kwargs['create_additional_revision']\n except KeyError:\n create_additional_revision = False\n\n if not create_additional_revision:\n parameters['revision'] = 'head'\n optional_keys = ['mode', 'folderid', 'item_id', 'itemid', 'revision']\n for key in optional_keys:\n if key in kwargs:\n if key == 'item_id':\n parameters['itemid'] = kwargs[key]\n continue\n if key == 'folder_id':\n parameters['folderid'] = kwargs[key]\n continue\n parameters[key] = kwargs[key]\n\n # We may want a different name than path\n file_payload = open(kwargs.get('filepath', filename), 'rb')\n # Arcane getting of the file size using fstat. More details can be\n # found in the python library docs\n parameters['length'] = os.fstat(file_payload.fileno()).st_size\n\n response = self.request('midas.upload.perform', parameters,\n file_payload)\n return response", "response": "This method uploads a file into a given item in the public folder."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch the resources associated to a given query.", "response": "def search(self, search, token=None):\n \"\"\"\n Get the resources corresponding to a given query.\n\n :param search: The search criterion.\n :type search: string\n :param token: (optional) The credentials to use when searching.\n :type token: None | string\n :returns: Dictionary containing the search result. Notable is the\n dictionary item 'results', which is a list of item details.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['search'] = search\n if token:\n parameters['token'] = token\n response = self.request('midas.resource.search', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_condor_dag(self, token, batchmaketaskid, dagfilename,\n dagmanoutfilename):\n \"\"\"\n Add a Condor DAG to the given Batchmake task.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param batchmaketaskid: id of the Batchmake task for this DAG\n :type batchmaketaskid: int | long\n :param dagfilename: Filename of the DAG file\n :type dagfilename: string\n :param dagmanoutfilename: Filename of the DAG processing output\n :type dagmanoutfilename: string\n :returns: The created Condor DAG DAO\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['batchmaketaskid'] = batchmaketaskid\n parameters['dagfilename'] = dagfilename\n parameters['outfilename'] = dagmanoutfilename\n response = self.request('midas.batchmake.add.condor.dag', parameters)\n return response", "response": "Adds a Condor DAG to the given Batchmake task."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a Condor DAG job to the Condor DAG associated with this Batchmake task :param token: A valid token for the user in question. :type token: string :param batchmaketaskid: id of the Batchmake task for this DAG :type batchmaketaskid: int | long :param jobdefinitionfilename: Filename of the definition file for the job :type jobdefinitionfilename: string :param outputfilename: Filename of the output file for the job :type outputfilename: string :param errorfilename: Filename of the error file for the job :type errorfilename: string :param logfilename: Filename of the log file for the job :type logfilename: string :param postfilename: Filename of the post script log file for the job :type postfilename: string :return: The created Condor job DAO. :rtype: dict", "response": "def add_condor_job(self, token, batchmaketaskid, jobdefinitionfilename,\n outputfilename, errorfilename, logfilename,\n postfilename):\n \"\"\"\n Add a Condor DAG job to the Condor DAG associated with this\n Batchmake task\n\n :param token: A valid token for the user in question.\n :type token: string\n :param batchmaketaskid: id of the Batchmake task for this DAG\n :type batchmaketaskid: int | long\n :param jobdefinitionfilename: Filename of the definition file for the\n job\n :type jobdefinitionfilename: string\n :param outputfilename: Filename of the output file for the job\n :type outputfilename: string\n :param errorfilename: Filename of the error file for the job\n :type errorfilename: string\n :param logfilename: Filename of the log file for the job\n :type logfilename: string\n :param postfilename: Filename of the post script log file for the job\n :type postfilename: string\n :return: The created Condor job DAO.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['batchmaketaskid'] = batchmaketaskid\n parameters['jobdefinitionfilename'] = jobdefinitionfilename\n parameters['outputfilename'] = outputfilename\n parameters['errorfilename'] = errorfilename\n parameters['logfilename'] = logfilename\n parameters['postfilename'] = postfilename\n response = self.request('midas.batchmake.add.condor.job', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts DICOM metadata from the given item", "response": "def extract_dicommetadata(self, token, item_id):\n \"\"\"\n Extract DICOM metadata from the given item\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: id of the item to be extracted\n :type item_id: int | long\n :return: the item revision DAO\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['item'] = item_id\n response = self.request('midas.dicomextractor.extract', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlogging in to get the real token using the temporary token and otp.", "response": "def mfa_otp_login(self, temp_token, one_time_pass):\n \"\"\"\n Log in to get the real token using the temporary token and otp.\n\n :param temp_token: The temporary token or id returned from normal login\n :type temp_token: string\n :param one_time_pass: The one-time pass to be sent to the underlying\n multi-factor engine.\n :type one_time_pass: string\n :returns: A standard token for interacting with the web api.\n :rtype: string\n \"\"\"\n parameters = dict()\n parameters['mfaTokenId'] = temp_token\n parameters['otp'] = one_time_pass\n response = self.request('midas.mfa.otp.login', parameters)\n return response['token']"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_big_thumbnail(self, token, bitstream_id, item_id, width=575):\n parameters = dict()\n parameters['token'] = token\n parameters['bitstreamId'] = bitstream_id\n parameters['itemId'] = item_id\n parameters['width'] = width\n response = self.request('midas.thumbnailcreator.create.big.thumbnail',\n parameters)\n return response", "response": "Create a big thumbnail for the given token and bitstream with the given width."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a 100x100 small thumbnail for the given item. It is used for preview purpose and displayed in the 'preview' and 'thumbnails' sidebar sections. :param token: A valid token for the user in question. :type token: string :param item_id: The item on which to set the thumbnail. :type item_id: int | long :returns: The item object (with the new thumbnail id) and the path where the newly created thumbnail is stored. :rtype: dict", "response": "def create_small_thumbnail(self, token, item_id):\n \"\"\"\n Create a 100x100 small thumbnail for the given item. It is used for\n preview purpose and displayed in the 'preview' and 'thumbnails'\n sidebar sections.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: The item on which to set the thumbnail.\n :type item_id: int | long\n :returns: The item object (with the new thumbnail id) and the path\n where the newly created thumbnail is stored.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['itemId'] = item_id\n response = self.request(\n 'midas.thumbnailcreator.create.small.thumbnail', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef solr_advanced_search(self, query, token=None, limit=20):\n parameters = dict()\n parameters['query'] = query\n parameters['limit'] = limit\n if token:\n parameters['token'] = token\n response = self.request('midas.solr.search.advanced', parameters)\n return response", "response": "Search the item metadata using Apache Solr."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef associate_item_with_scalar_data(self, token, item_id, scalar_id,\n label):\n \"\"\"\n Associate a result item with a particular scalar value.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param item_id: The id of the item to associate with the scalar.\n :type item_id: int | long\n :param scalar_id: Scalar id with which to associate the item.\n :type scalar_id: int | long\n :param label: The label describing the nature of the association.\n :type label: string\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['scalarIds'] = scalar_id\n parameters['itemId'] = item_id\n parameters['label'] = label\n self.request('midas.tracker.item.associate', parameters)", "response": "Associate a result item with a particular scalar value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_submission(self, token, **kwargs):\n parameters = {}\n parameters['token'] = token\n optional_keys = ['uuid', 'name']\n for key in optional_keys:\n if key in kwargs:\n parameters[key] = kwargs[key]\n return self.request('midas.tracker.submission.create', parameters)", "response": "Associate a result item with a particular scalar value."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd scalar data point to the current configuration item.", "response": "def add_scalar_data(self, token, community_id, producer_display_name,\n metric_name, producer_revision, submit_time, value,\n **kwargs):\n \"\"\"\n Create a new scalar data point.\n\n :param token: A valid token for the user in question.\n :type token: string\n :param community_id: The id of the community that owns the producer.\n :type community_id: int | long\n :param producer_display_name: The display name of the producer.\n :type producer_display_name: string\n :param metric_name: The metric name that identifies which trend this\n point belongs to.\n :type metric_name: string\n :param producer_revision: The repository revision of the producer that\n produced this value.\n :type producer_revision: int | long | string\n :param submit_time: The submit timestamp. Must be parsable with PHP\n strtotime().\n :type submit_time: string\n :param value: The value of the scalar.\n :type value: float\n :param config_item_id: (optional) If this value pertains to a specific\n configuration item, pass its id here.\n :type config_item_id: int | long\n :param test_dataset_id: (optional) If this value pertains to a\n specific test dataset, pass its id here.\n :type test_dataset_id: int | long\n :param truth_dataset_id: (optional) If this value pertains to a\n specific ground truth dataset, pass its id here.\n :type truth_dataset_id: int | long\n :param silent: (optional) If true, do not perform threshold-based email\n notifications for this scalar.\n :type silent: bool\n :param unofficial: (optional) If true, creates an unofficial scalar\n visible only to the user performing the submission.\n :type unofficial: bool\n :param build_results_url: (optional) A URL for linking to build results\n for this submission.\n :type build_results_url: string\n :param branch: (optional) The branch name in the source repository for\n this submission.\n :type branch: string\n :param submission_id: (optional) The id of the submission.\n :type submission_id: int | long\n :param submission_uuid: (optional) The uuid of the submission. If one\n does not exist, it will be created.\n :type submission_uuid: string\n :type branch: string\n :param params: (optional) Any key/value pairs that should be displayed\n with this scalar result.\n :type params: dict\n :param extra_urls: (optional) Other URL's that should be displayed with\n with this scalar result. Each element of the list should be a dict\n with the following keys: label, text, href\n :type extra_urls: list[dict]\n :param unit: (optional) The unit of the scalar value.\n :type unit: string\n :param reproduction_command: (optional) The command to reproduce this\n scalar.\n :type reproduction_command: string\n :returns: The scalar object that was created.\n :rtype: dict\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['communityId'] = community_id\n parameters['producerDisplayName'] = producer_display_name\n parameters['metricName'] = metric_name\n parameters['producerRevision'] = producer_revision\n parameters['submitTime'] = submit_time\n parameters['value'] = value\n optional_keys = [\n 'config_item_id', 'test_dataset_id', 'truth_dataset_id', 'silent',\n 'unofficial', 'build_results_url', 'branch', 'extra_urls',\n 'params', 'submission_id', 'submission_uuid', 'unit',\n 'reproduction_command'\n ]\n for key in optional_keys:\n if key in kwargs:\n if key == 'config_item_id':\n parameters['configItemId'] = kwargs[key]\n elif key == 'test_dataset_id':\n parameters['testDatasetId'] = kwargs[key]\n elif key == 'truth_dataset_id':\n parameters['truthDatasetId'] = kwargs[key]\n elif key == 'build_results_url':\n parameters['buildResultsUrl'] = kwargs[key]\n elif key == 'extra_urls':\n parameters['extraUrls'] = json.dumps(kwargs[key])\n elif key == 'params':\n parameters[key] = json.dumps(kwargs[key])\n elif key == 'silent':\n if kwargs[key]:\n parameters[key] = kwargs[key]\n elif key == 'unofficial':\n if kwargs[key]:\n parameters[key] = kwargs[key]\n elif key == 'submission_id':\n parameters['submissionId'] = kwargs[key]\n elif key == 'submission_uuid':\n parameters['submissionUuid'] = kwargs[key]\n elif key == 'unit':\n parameters['unit'] = kwargs[key]\n elif key == 'reproduction_command':\n parameters['reproductionCommand'] = kwargs[key]\n else:\n parameters[key] = kwargs[key]\n response = self.request('midas.tracker.scalar.add', parameters)\n return response"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef upload_json_results(self, token, filepath, community_id,\n producer_display_name, metric_name,\n producer_revision, submit_time, **kwargs):\n \"\"\"\n Upload a JSON file containing numeric scoring results to be added as\n scalars. File is parsed and then deleted from the server.\n\n :param token: A valid token for the user in question.\n :param filepath: The path to the JSON file.\n :param community_id: The id of the community that owns the producer.\n :param producer_display_name: The display name of the producer.\n :param producer_revision: The repository revision of the producer\n that produced this value.\n :param submit_time: The submit timestamp. Must be parsable with PHP\n strtotime().\n :param config_item_id: (optional) If this value pertains to a specific\n configuration item, pass its id here.\n :param test_dataset_id: (optional) If this value pertains to a\n specific test dataset, pass its id here.\n :param truth_dataset_id: (optional) If this value pertains to a\n specific ground truth dataset, pass its id here.\n :param parent_keys: (optional) Semicolon-separated list of parent keys\n to look for numeric results under. Use '.' to denote nesting, like\n in normal javascript syntax.\n :param silent: (optional) If true, do not perform threshold-based email\n notifications for this scalar.\n :param unofficial: (optional) If true, creates an unofficial scalar\n visible only to the user performing the submission.\n :param build_results_url: (optional) A URL for linking to build results\n for this submission.\n :param branch: (optional) The branch name in the source repository for\n this submission.\n :param params: (optional) Any key/value pairs that should be displayed\n with this scalar result.\n :type params: dict\n :param extra_urls: (optional) Other URL's that should be displayed with\n with this scalar result. Each element of the list should be a dict\n with the following keys: label, text, href\n :type extra_urls: list of dicts\n :returns: The list of scalars that were created.\n \"\"\"\n parameters = dict()\n parameters['token'] = token\n parameters['communityId'] = community_id\n parameters['producerDisplayName'] = producer_display_name\n parameters['metricName'] = metric_name\n parameters['producerRevision'] = producer_revision\n parameters['submitTime'] = submit_time\n optional_keys = [\n 'config_item_id', 'test_dataset_id', 'truth_dataset_id', 'silent',\n 'unofficial', 'build_results_url', 'branch', 'extra_urls',\n 'params']\n for key in optional_keys:\n if key in kwargs:\n if key == 'config_item_id':\n parameters['configItemId'] = kwargs[key]\n elif key == 'test_dataset_id':\n parameters['testDatasetId'] = kwargs[key]\n elif key == 'truth_dataset_id':\n parameters['truthDatasetId'] = kwargs[key]\n elif key == 'parent_keys':\n parameters['parentKeys'] = kwargs[key]\n elif key == 'build_results_url':\n parameters['buildResultsUrl'] = kwargs[key]\n elif key == 'extra_urls':\n parameters['extraUrls'] = json.dumps(kwargs[key])\n elif key == 'params':\n parameters[key] = json.dumps(kwargs[key])\n elif key == 'silent':\n if kwargs[key]:\n parameters[key] = kwargs[key]\n elif key == 'unofficial':\n if kwargs[key]:\n parameters[key] = kwargs[key]\n else:\n parameters[key] = kwargs[key]\n file_payload = open(filepath, 'rb')\n response = self.request('midas.tracker.results.upload.json',\n parameters, file_payload)\n return response", "response": "Uploads a JSON file containing numeric scoring results to the server."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nobtains particular version of the doc at key.", "response": "def __get_rev(self, key, version, **kwa):\n '''Obtain particular version of the doc at key.'''\n if '_doc' in kwa:\n doc = kwa['_doc']\n else:\n if type(version) is int:\n if version == 0:\n order = pymongo.ASCENDING\n elif version == -1:\n order = pymongo.DESCENDING\n doc = self._collection.find_one({'k': key}, sort=[['d', order]])\n elif type(version) is datetime:\n ver = self.__round_time(version)\n doc = self._collection.find_one({'k': key, 'd': ver})\n\n if doc is None:\n raise KeyError('Supplied key `{0}` or version `{1}` does not exist'\n .format(key, str(version)))\n\n coded_val = doc['v']\n return pickle.loads(coded_val)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __round_time(self, dt):\n round_to = self._resolution.total_seconds()\n seconds = (dt - dt.min).seconds\n rounding = (seconds + round_to / 2) // round_to * round_to\n return dt + timedelta(0, rounding - seconds, -dt.microsecond)", "response": "Round a datetime object to a multiple of a timedelta\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind a hash value for the linear combination of invocation methods.", "response": "def _hashkey(self, method, url, **kwa):\n '''Find a hash value for the linear combination of invocation methods.\n '''\n to_hash = ''.join([str(method), str(url),\n str(kwa.get('data', '')),\n str(kwa.get('params', ''))\n ])\n return hashlib.md5(to_hash.encode()).hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconnecting to a Siemens S7 PLC and sets up the internal ID of the current Merker entry.", "response": "def setup(self, address, rack=0, slot=1, port=102):\n \"\"\"Connects to a Siemens S7 PLC.\n\n Connects to a Siemens S7 using the Snap7 library.\n See [the snap7 documentation](http://snap7.sourceforge.net/) for\n supported models and more details.\n\n It's not currently possible to query the device for available pins,\n so `available_pins()` returns an empty list. Instead, you should use\n `map_pin()` to map to a Merker, Input or Output in the PLC. The\n internal id you should use is a string following this format:\n '[DMQI][XBWD][0-9]+.?[0-9]*' where:\n\n * [DMQI]: D for DB, M for Merker, Q for Output, I for Input\n * [XBWD]: X for bit, B for byte, W for word, D for dword\n * [0-9]+: Address of the resource\n * [0-9]*: Bit of the address (type X only, ignored in others)\n\n For example: 'IB100' will read a byte from an input at address 100 and\n 'MX50.2' will read/write bit 2 of the Merker at address 50. It's not\n allowed to write to inputs (I), but you can read/write Outpus, DBs and\n Merkers. If it's disallowed by the PLC, an exception will be thrown by\n python-snap7 library.\n\n For this library to work, it might be needed to change some settings\n in the PLC itself. See\n [the snap7 documentation](http://snap7.sourceforge.net/) for more\n information. You also need to put the PLC in RUN mode. Not however that\n having a Ladder program downloaded, running and modifying variables\n will probably interfere with inputs and outputs, so put it in RUN mode,\n but preferably without a downloaded program.\n\n @arg address IP address of the module.\n @arg rack rack where the module is installed.\n @arg slot slot in the rack where the module is installed.\n @arg port port the PLC is listenning to.\n\n @throw RuntimeError if something went wrong\n @throw any exception thrown by `snap7`'s methods.\n \"\"\"\n rack = int(rack)\n slot = int(slot)\n port = int(port)\n address = str(address)\n self._client = snap7.client.Client()\n self._client.connect(address, rack, slot, port)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef setup(self, port):\n port = str(port)\n # timeout is used by all I/O operations\n self._serial = serial.Serial(port, 115200, timeout=2)\n time.sleep(2) # time to Arduino reset\n\n if not self._serial.is_open:\n raise RuntimeError('Could not connect to Arduino')\n\n self._serial.write(b'\\x01')\n\n if self._serial.read() != b'\\x06':\n raise RuntimeError('Could not connect to Arduino')\n\n ps = [p for p in self.available_pins() if p['digital']['output']]\n for pin in ps:\n self._set_pin_direction(pin['id'], ahio.Direction.Output)", "response": "Connects to an Arduino UNO on serial port port."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef block_resource_fitnesses(self, block: block.Block):\n\n # Short-circuit! My algorithm is terrible, so it doesn't work well for the edge case where\n # the block has no requirements\n if not block.resources:\n return {n: 1 for n in self.config.nodes.keys()}\n\n node_fitnesses = {}\n\n for resource in block.resources:\n resource_fitnesses = self.resource_fitnesses(resource)\n\n if not resource_fitnesses:\n raise UnassignableBlock(block.name)\n\n max_fit = max(resource_fitnesses.values())\n min_fit = min(resource_fitnesses.values())\n\n for node, fitness in resource_fitnesses.items():\n if node not in node_fitnesses:\n node_fitnesses[node] = {}\n\n if not fitness:\n # Since we're rescaling, 0 is now an OK value...\n # We will check for `is False` after this\n node_fitnesses[node][resource.describe()] = False\n else:\n if max_fit - min_fit:\n node_fitnesses[node][resource.describe()] = (fitness - min_fit) / (max_fit - min_fit)\n else:\n # All the values are the same, default to 1\n node_fitnesses[node][resource.describe()] = 1.0\n\n res = {}\n\n for node, res_fits in node_fitnesses.items():\n fit_sum = 0\n for res_desc, fit in res_fits.items():\n if fit is False:\n fit_sum = False\n break\n\n fit_sum += fit\n\n if fit_sum is False:\n # Skip this node entirely\n res[node] = False\n continue\n\n res[node] = fit_sum\n\n return res", "response": "Returns a map of nodename to average fitness value for this block."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of available drivers names.", "response": "def available_drivers():\n \"\"\"Returns a list of available drivers names.\n \"\"\"\n global __modules\n global __available\n\n if type(__modules) is not list:\n __modules = list(__modules)\n\n if not __available:\n __available = [d.ahioDriverInfo.NAME\n for d in __modules\n if d.ahioDriverInfo.AVAILABLE]\n\n return __available"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntry to load the driver named name.", "response": "def __load_driver(name):\n \"\"\"Tries to load the driver named @arg name.\n\n A driver is considered valid if it has a ahioDriverInfo object. It should\n however implement all APIs described in `ahio.abstract_driver`, as they'll\n be needed to use the driver.\n\n @returns the driver package, or False if it failed.\n \"\"\"\n global __count\n try:\n dname = os.path.basename(name).replace('.py', '')\n mod_name = 'ahio.drivers.%s%d' % (dname, __count)\n loader = importlib.machinery.SourceFileLoader(mod_name, name)\n driver = loader.load_module()\n __count += 1\n return driver if hasattr(driver, 'ahioDriverInfo') else False\n except Exception:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __locate_driver_named(name):\n global __modules\n\n if type(__modules) is not list:\n __modules = list(__modules)\n\n ms = [d for d in __modules if d.ahioDriverInfo.NAME == name]\n if not ms:\n return None\n return ms[0]", "response": "Searches the modules list for a driver named name. Returns the first module found or None if no such module can be found."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmaps a pin number to a physical device pin.", "response": "def map_pin(self, abstract_pin_id, physical_pin_id):\n \"\"\"Maps a pin number to a physical device pin.\n\n To make it easy to change drivers without having to refactor a lot of\n code, this library does not use the names set by the driver to identify\n a pin. This function will map a number, that will be used by other\n functions, to a physical pin represented by the drivers pin id. That\n way, if you need to use another pin or change the underlying driver\n completly, you only need to redo the mapping.\n\n If you're developing a driver, keep in mind that your driver will not\n know about this. The other functions will translate the mapped pin to\n your id before calling your function.\n\n @arg abstract_pin_id the id that will identify this pin in the\n other function calls. You can choose what you want.\n\n @arg physical_pin_id the id returned in the driver.\n See `AbstractDriver.available_pins`. Setting it to None removes the\n mapping.\n \"\"\"\n if physical_pin_id:\n self._pin_mapping[abstract_pin_id] = physical_pin_id\n else:\n self._pin_mapping.pop(abstract_pin_id, None)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_pin_interpolation(self,\n pin,\n read_min,\n read_max,\n write_min,\n write_max):\n \"\"\"Interpolates input and output values for `pin`.\n\n Changes the output and input of `AbstractDriver.read` and\n `AbstractDriver.write` functions to use a value in range\n (`read_min`, `read_max`) or (`write_min`, `write_max`) instead of the\n values returned by `available_pins` (analog only). The conversion is\n done using linear interpolation. If `read_min`, `read_max`, `write_min`\n and `write_max` are all None or don't form valid pairs (like, read_min\n has a value but read_max is None), the pin is deregistered. If you pass\n a pair but leave the other with None values, only one direction is\n registered.\n\n @arg pin pin id you've set using `AbstractDriver.map_pin`\n @arg read_min the min value for the linear interpolation of\n `AbstractDriver.read`.\n @arg read_max the max value for the linear interpolation of\n `AbstractDriver.read`.\n @arg write_min the min value for the linear interpolation of\n `AbstractDriver.write`.\n @arg write_max the max value for the linear interpolation of\n `AbstractDriver.write`.\n \"\"\"\n if type(pin) is list:\n # I don't like breaking calls in multiple lines\n args = (read_min, read_max, write_min, write_max)\n for p in pin:\n self.set_pin_interpolation(p, *args)\n return\n\n valid_read = (read_min is not None and read_max is not None)\n valid_write = (write_min is not None and write_max is not None)\n\n if not valid_read and not valid_write:\n self._pin_lin.pop(pin, None)\n return\n\n pin_id = self._pin_mapping.get(pin, None)\n pins = [pin for pin in self.available_pins() if pin_id == pin['id']]\n read = pins[0]['analog']['read_range']\n write = pins[0]['analog']['write_range']\n valid_read = valid_read and read\n valid_write = valid_write and write\n self._pin_lin[pin] = {\n 'read': (*read, read_min, read_max) if valid_read else None,\n 'write': (write_min, write_max, *write) if valid_write else None\n }", "response": "Interpolates input and output values for a given pin."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the direction of the specified pin.", "response": "def set_pin_direction(self, pin, direction):\n \"\"\"Sets pin `pin` to `direction`.\n\n The pin should support the requested mode. Calling this function\n on a unmapped pin does nothing. Calling it with a unsupported direction\n throws RuntimeError.\n\n If you're developing a driver, you should implement\n _set_pin_direction(self, pin, direction) where `pin` will be one of\n your internal IDs. If a pin is set to OUTPUT, put it on LOW state.\n\n @arg pin pin id you've set using `AbstractDriver.map_pin`\n @arg mode a value from `AbstractDriver.Direction`\n\n @throw KeyError if pin isn't mapped.\n @throw RuntimeError if direction is not supported by pin.\n \"\"\"\n if type(pin) is list:\n for p in pin:\n self.set_pin_direction(p, direction)\n return\n\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id and type(direction) is ahio.Direction:\n self._set_pin_direction(pin_id, direction)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the ahio. Direction that this pin was set to.", "response": "def pin_direction(self, pin):\n \"\"\"Gets the `ahio.Direction` this pin was set to.\n\n If you're developing a driver, implement _pin_direction(self, pin)\n\n @arg pin the pin you want to see the mode\n @returns the `ahio.Direction` the pin is set to\n\n @throw KeyError if pin isn't mapped.\n \"\"\"\n if type(pin) is list:\n return [self.pin_direction(p) for p in pin]\n\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n return self._pin_direction(pin_id)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_pin_type(self, pin, ptype):\n if type(pin) is list:\n for p in pin:\n self.set_pin_type(p, ptype)\n return\n\n pin_id = self._pin_mapping.get(pin, None)\n if type(ptype) is not ahio.PortType:\n raise KeyError('ptype must be of type ahio.PortType')\n elif pin_id:\n self._set_pin_type(pin_id, ptype)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)", "response": "Sets the type of the specified pin to ptype."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pin_type(self, pin):\n if type(pin) is list:\n return [self.pin_type(p) for p in pin]\n\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n return self._pin_type(pin_id)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)", "response": "Returns the type of the specified pin."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write(self, pin, value, pwm=False):\n if type(pin) is list:\n for p in pin:\n self.write(p, value, pwm)\n return\n\n if pwm and type(value) is not int and type(value) is not float:\n raise TypeError('pwm is set, but value is not a float or int')\n\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n lpin = self._pin_lin.get(pin, None)\n if lpin and type(lpin['write']) is tuple:\n write_range = lpin['write']\n value = self._linear_interpolation(value, *write_range)\n self._write(pin_id, value, pwm)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)", "response": "Sets the output to the given value."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads value from pin.", "response": "def read(self, pin):\n \"\"\"Reads value from pin `pin`.\n\n Returns the value read from pin `pin`. If it's an analog pin, returns\n a number in analog.input_range. If it's digital, returns\n `ahio.LogicValue`.\n\n If you're developing a driver, implement _read(self, pin)\n\n @arg pin the pin to read from\n @returns the value read from the pin\n\n @throw KeyError if pin isn't mapped.\n \"\"\"\n if type(pin) is list:\n return [self.read(p) for p in pin]\n\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n value = self._read(pin_id)\n lpin = self._pin_lin.get(pin, None)\n if lpin and type(lpin['read']) is tuple:\n read_range = lpin['read']\n value = self._linear_interpolation(value, *read_range)\n return value\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the analog reference to the specified value.", "response": "def set_analog_reference(self, reference, pin=None):\n \"\"\"Sets the analog reference to `reference`\n\n If the driver supports per pin reference setting, set pin to the\n desired reference. If not, passing None means set to all, which is the\n default in most hardware. If only per pin reference is supported and\n pin is None, raise RuntimeError.\n\n If you're developing a driver, implement\n _set_analog_reference(self, reference, pin). Raise RuntimeError if pin\n was set but is not supported by the platform.\n\n @arg reference the value that describes the analog reference. See\n `AbstractDriver.analog_references`\n @arg pin if the the driver supports it, the pin that will use\n `reference` as reference. None for all.\n\n @throw RuntimeError if pin is None on a per pin only hardware, or if\n it's a valid pin on a global only analog reference hardware.\n @throw KeyError if pin isn't mapped.\n \"\"\"\n if pin is None:\n self._set_analog_reference(reference, None)\n else:\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n self._set_analog_reference(reference, pin_id)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the analog reference for the specified pin.", "response": "def analog_reference(self, pin=None):\n \"\"\"Returns the analog reference.\n\n If the driver supports per pin analog reference setting, returns the\n reference for pin `pin`. If pin is None, returns the global analog\n reference. If only per pin reference is supported and pin is None,\n raise RuntimeError.\n\n If you're developing a driver, implement _analog_reference(self, pin)\n\n @arg pin if the the driver supports it, the pin that will use\n `reference` as reference. None for all.\n\n @returns the reference used for pin\n\n @throw RuntimeError if pin is None on a per pin only hardware, or if\n it's a valid pin on a global only analog reference hardware.\n @throw KeyError if pin isn't mapped.\n \"\"\"\n if pin is None:\n return self._analog_reference(None)\n else:\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n return self._analog_reference(pin_id)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_pwm_frequency(self, frequency, pin=None):\n if pin is None:\n self._set_pwm_frequency(frequency, None)\n else:\n pin_id = self._pin_mapping.get(pin, None)\n if pin_id:\n self._set_pwm_frequency(frequency, pin_id)\n else:\n raise KeyError('Requested pin is not mapped: %s' % pin)", "response": "Sets the frequency of the PWM in Hz."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef notify(self, message, title=\"\", **kwargs):\n logger.info(\n \"pushover notify title:{0} message:{1}\".format(title, message))\n try:\n data = {\n 'token': self._token,\n 'user': self._user,\n 'title': title,\n 'message': message,\n }\n data.update(kwargs)\n payload = []\n for (k, v) in data.items():\n if isinstance(v, str_type):\n payload.append((k, v.encode(\"utf-8\")))\n else:\n payload.append((k, v))\n headers = {\"Content-type\": \"application/x-www-form-urlencoded\",}\n conn = HTTPSConnection(\"api.pushover.net\")\n params = urlencode(payload)\n conn.request(\"POST\", \"/1/messages.json\", params, headers)\n rsp = conn.getresponse()\n if rsp.status != 200:\n raise PushoverException(\"pushover:{0}\".format(rsp.status))\n conn.close()\n except Exception as e:\n raise PushoverException(\"exception:{0!r}\".format(e))", "response": "Send a pushover notification to the user."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef SIRode(y0, time, beta, gamma):\n \n Xsim = rk4(SIR_D, y0, time, args=(beta,gamma,))\n Xsim = Xsim.transpose()\n return Xsim", "response": "Integrate a very basic deterministic SIR system."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the URL of the server.", "response": "def url(self):\n \"\"\"\n Return the URL of the server.\n\n :returns: URL of the server\n :rtype: string\n \"\"\"\n if len(self.drivers) > 0:\n return self.drivers[0].url\n else:\n return self._url"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef guess_array_memory_usage( bam_readers, dtype, use_strand=False ):\n ARRAY_COUNT = 5\n if not isinstance( bam_readers, list ):\n bam_readers = [ bam_readers ]\n if isinstance( dtype, basestring ):\n dtype = NUMPY_DTYPES.get( dtype, None )\n use_strand = use_strand + 1 #if false, factor of 1, if true, factor of 2\n dtypes = guess_numpy_dtypes_from_idxstats( bam_readers, default=None, force_dtype=False )\n if not [ dt for dt in dtypes if dt is not None ]:\n #found no info from idx\n dtypes = guess_numpy_dtypes_from_idxstats( bam_readers, default=dtype or numpy.uint64, force_dtype=True )\n elif dtype:\n dtypes = [ dtype if dt else None for dt in dtypes ]\n read_groups = []\n no_read_group = False\n for bam in bam_readers:\n rgs = bam.get_read_groups()\n if rgs:\n for rg in rgs:\n if rg not in read_groups:\n read_groups.append( rg )\n else:\n no_read_group = True\n read_groups = len( read_groups ) + no_read_group\n max_ref_size = 0\n array_byte_overhead = sys.getsizeof( numpy.zeros( ( 0 ), dtype=numpy.uint64 ) )\n array_count = ARRAY_COUNT * use_strand * read_groups\n for bam in bam_readers:\n for i, ( name, length ) in enumerate( bam.get_references() ):\n if dtypes[i] is not None:\n max_ref_size = max( max_ref_size, ( length + length * dtypes[i]().nbytes * array_count + ( array_byte_overhead * ( array_count + 1 ) ) ) )\n return max_ref_size", "response": "Returns an estimate for the maximum amount of memory to be consumed by numpy arrays."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndetect the version of the Raspberry Pi.", "response": "def pi_version():\n \"\"\"Detect the version of the Raspberry Pi. Returns either 1, 2 or\n None depending on if it's a Raspberry Pi 1 (model A, B, A+, B+),\n Raspberry Pi 2 (model B+), or not a Raspberry Pi.\n https://github.com/adafruit/Adafruit_Python_GPIO/blob/master/Adafruit_GPIO/Platform.py\n \"\"\"\n if not os.path.isfile('/proc/cpuinfo'):\n return None\n # Check /proc/cpuinfo for the Hardware field value.\n # 2708 is pi 1\n # 2709 is pi 2\n # Anything else is not a pi.\n with open('/proc/cpuinfo', 'r') as infile:\n cpuinfo = infile.read()\n # Match a line like 'Hardware : BCM2709'\n match = re.search('^Hardware\\s+:\\s+(\\w+)$', cpuinfo,\n flags=re.MULTILINE | re.IGNORECASE)\n if not match:\n # Couldn't find the hardware, assume it isn't a pi.\n return None\n if match.group(1) == 'BCM2708':\n # Pi 1\n return 1\n elif match.group(1) == 'BCM2709':\n # Pi 2\n return 2\n elif match.group(1) == 'BCM2835':\n # Pi 3\n return 3\n else:\n # Something else, not a pi.\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef system(command, input=None):\n logger.debug(\"Executing command: %s\", command)\n p = subprocess.Popen(command,\n shell=True,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n close_fds=MUST_CLOSE_FDS)\n stdoutdata, stderrdata = p.communicate(input=input)\n result = stdoutdata + stderrdata\n if p.returncode:\n logger.error(\"Something went wrong when executing '%s'\",\n command)\n logger.error(\"Returncode: %s\", p.returncode)\n logger.error(\"Output:\")\n logger.error(result)\n sys.exit(1)\n logger.info(result)", "response": "Execute a command and return the output of the command."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates coverage reports and open them in the browser.", "response": "def main():\n \"\"\"Create coverage reports and open them in the browser.\"\"\"\n usage = \"Usage: %prog PATH_TO_PACKAGE\"\n parser = optparse.OptionParser(usage=usage)\n parser.add_option(\n \"-v\", \"--verbose\",\n action=\"store_true\", dest=\"verbose\", default=False,\n help=\"Show debug output\")\n parser.add_option(\n \"-d\", \"--output-dir\",\n action=\"store\", type=\"string\", dest=\"output_dir\",\n default='',\n help=\"\")\n parser.add_option(\n \"-t\", \"--test-args\",\n action=\"store\", type=\"string\", dest=\"test_args\",\n default='',\n help=(\"Pass argument on to bin/test. Quote the argument, \" +\n \"for instance \\\"-t '-m somemodule'\\\".\"))\n (options, args) = parser.parse_args()\n if options.verbose:\n log_level = logging.DEBUG\n else:\n log_level = logging.INFO\n logging.basicConfig(level=log_level,\n format=\"%(levelname)s: %(message)s\")\n\n curdir = os.getcwd()\n testbinary = os.path.join(curdir, 'bin', 'test')\n if not os.path.exists(testbinary):\n raise RuntimeError(\"Test command doesn't exist: %s\" % testbinary)\n\n coveragebinary = os.path.join(curdir, 'bin', 'coverage')\n if not os.path.exists(coveragebinary):\n logger.debug(\"Trying globally installed coverage command.\")\n coveragebinary = 'coverage'\n\n logger.info(\"Running tests in coverage mode (can take a long time)\")\n parts = [coveragebinary, 'run', testbinary]\n if options.test_args:\n parts.append(options.test_args)\n system(\" \".join(parts))\n logger.debug(\"Creating coverage reports...\")\n if options.output_dir:\n coverage_dir = options.output_dir\n open_in_browser = False\n else:\n coverage_dir = 'htmlcov' # The default\n open_in_browser = True\n system(\"%s html --directory=%s\" % (coveragebinary, coverage_dir))\n logger.info(\"Wrote coverage files to %s\", coverage_dir)\n if open_in_browser:\n index_file = os.path.abspath(\n os.path.join(coverage_dir, 'index.html'))\n logger.debug(\"About to open %s in your webbrowser.\", index_file)\n webbrowser.open('file://' + index_file)\n logger.info(\"Opened reports in your browser.\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstarting a Modbus client and connect to it", "response": "def setup(\n self,\n configuration=\"ModbusSerialClient(method='rtu',port='/dev/cu.usbmodem14101',baudrate=9600)\"\n ):\n \"\"\"Start a Modbus server.\n\n The following classes are available with their respective named\n parameters:\n \n ModbusTcpClient\n host: The host to connect to (default 127.0.0.1)\n port: The modbus port to connect to (default 502)\n source_address: The source address tuple to bind to (default ('', 0))\n timeout: The timeout to use for this socket (default Defaults.Timeout)\n\n ModbusUdpClient\n host: The host to connect to (default 127.0.0.1)\n port: The modbus port to connect to (default 502)\n timeout: The timeout to use for this socket (default None)\n\n ModbusSerialClient\n method: The method to use for connection (asii, rtu, binary)\n port: The serial port to attach to\n stopbits: The number of stop bits to use (default 1)\n bytesize: The bytesize of the serial messages (default 8 bits)\n parity: Which kind of parity to use (default None)\n baudrate: The baud rate to use for the serial device\n timeout: The timeout between serial requests (default 3s)\n\n When configuring the ports, the following convention should be\n respected:\n \n portname: C1:13 -> Coil on device 1, address 13\n\n The letters can be:\n\n C = Coil\n I = Input\n R = Register\n H = Holding\n\n @arg configuration a string that instantiates one of those classes.\n\n @throw RuntimeError can't connect to Arduino\n \"\"\"\n from pymodbus3.client.sync import ModbusSerialClient, ModbusUdpClient, ModbusTcpClient\n self._client = eval(configuration)\n self._client.connect()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconnect to a TCP server and connects to it.", "response": "def setup(self, address, port):\n \"\"\"Connects to server at `address`:`port`.\n\n Connects to a TCP server listening at `address`:`port` that implements\n the protocol described in the file \"Generic TCP I:O Protocol.md\"\n\n @arg address IP or address to connect to.\n @arg port port to connect to.\n\n @throw RuntimeError if connection was successiful but protocol isn't\n supported.\n @throw any exception thrown by `socket.socket`'s methods.\n \"\"\"\n address = str(address)\n port = int(port)\n self._socket = socket.socket()\n self._socket.connect((address, port))\n self._socket.send(b'HELLO 1.0\\n')\n with self._socket.makefile() as f:\n if f.readline().strip() != 'OK':\n raise RuntimeError('Protocol not supported')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning an exception given a status code and error code.", "response": "def get_exception_from_status_and_error_codes(status_code, error_code, value):\n \"\"\"\n Return an exception given status and error codes.\n\n :param status_code: HTTP status code.\n :type status_code: None | int\n :param error_code: Midas Server error code.\n :type error_code: None | int\n :param value: Message to display.\n :type value: string\n :returns: Exception.\n :rtype : pydas.exceptions.ResponseError\n \"\"\"\n if status_code == requests.codes.bad_request:\n exception = BadRequest(value)\n elif status_code == requests.codes.unauthorized:\n exception = Unauthorized(value)\n elif status_code == requests.codes.forbidden:\n exception = Unauthorized(value)\n elif status_code in [requests.codes.not_found, requests.codes.gone]:\n exception = NotFound(value)\n elif status_code == requests.codes.method_not_allowed:\n exception = MethodNotAllowed(value)\n elif status_code >= requests.codes.bad_request:\n exception = HTTPError(value)\n else:\n exception = ResponseError(value)\n\n if error_code == -100: # MIDAS_INTERNAL_ERROR\n exception = InternalError(value)\n elif error_code == -101: # MIDAS_INVALID_TOKEN\n exception = InvalidToken(value)\n elif error_code == -105: # MIDAS_UPLOAD_FAILED\n exception = UploadFailed(value)\n elif error_code == -140: # MIDAS_UPLOAD_TOKEN_GENERATION_FAILED\n exception = UploadTokenGenerationFailed(value)\n elif error_code == -141: # MIDAS_INVALID_UPLOAD_TOKEN\n exception = InvalidUploadToken(value)\n elif error_code == -150: # MIDAS_INVALID_PARAMETER\n exception = InvalidParameter(value)\n elif error_code == -151: # MIDAS_INVALID_POLICY\n exception = InvalidPolicy(value)\n\n return exception"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef analog_read(self, pin):\n with self.data_lock:\n data = self._command_handler.analog_response_table[pin][self._command_handler.RESPONSE_TABLE_PIN_DATA_VALUE]\n return data", "response": "This method returns the last value received for the specified pin."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef analog_write(self, pin, value):\n\n if self._command_handler.ANALOG_MESSAGE + pin < 0xf0:\n command = [self._command_handler.ANALOG_MESSAGE + pin, value & 0x7f, (value >> 7) & 0x7f]\n self._command_handler.send_command(command)\n else:\n self.extended_analog(pin, value)", "response": "This method writes an analog value to the specified pin."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef close(self):\n\n self._command_handler.system_reset()\n self._command_handler.stop()\n self.transport.stop()\n self.transport.close()\n\n if self.verbose:\n print(\"PyMata close(): Calling sys.exit(0): Hope to see you soon!\")\n\n sys.exit(0)", "response": "This method closes the transport and the command handler and closes the transport and the transport and the command handler."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving the last digital data value received for the specified pin. NOTE: This command will return values for digital, pwm, etc, pin types :param pin: Selected pin :return: The last value entered into the digital response table.", "response": "def digital_read(self, pin):\n \"\"\"\n Retrieve the last digital data value received for the specified pin.\n NOTE: This command will return values for digital, pwm, etc, pin types\n\n :param pin: Selected pin\n\n :return: The last value entered into the digital response table.\n \"\"\"\n with self.data_lock:\n data = \\\n self._command_handler.digital_response_table[pin][self._command_handler.RESPONSE_TABLE_PIN_DATA_VALUE]\n return data"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the specified pin to the specified value. :param pin: pin number :param value: pin value :return: No return value", "response": "def digital_write(self, pin, value):\n \"\"\"\n Set the specified pin to the specified value.\n\n :param pin: pin number\n\n :param value: pin value\n\n :return: No return value\n \"\"\"\n # The command value is not a fixed value, but needs to be calculated using the\n # pin's port number\n #\n #\n port = pin // 8\n\n calculated_command = self._command_handler.DIGITAL_MESSAGE + port\n mask = 1 << (pin % 8)\n # Calculate the value for the pin's position in the port mask\n if value == 1:\n self.digital_output_port_pins[port] |= mask\n\n else:\n self.digital_output_port_pins[port] &= ~mask\n\n # Assemble the command\n command = (calculated_command, self.digital_output_port_pins[port] & 0x7f,\n (self.digital_output_port_pins[port] >> 7) & 0x7f)\n\n self._command_handler.send_command(command)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disable_analog_reporting(self, pin):\n command = [self._command_handler.REPORT_ANALOG + pin, self.REPORTING_DISABLE]\n self._command_handler.send_command(command)", "response": "Disables analog reporting for a single analog pin."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef disable_digital_reporting(self, pin):\n port = pin // 8\n command = [self._command_handler.REPORT_DIGITAL + port, self.REPORTING_DISABLE]\n self._command_handler.send_command(command)", "response": "Disables digital reporting for the specified pin."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef enable_analog_reporting(self, pin):\n command = [self._command_handler.REPORT_ANALOG + pin, self.REPORTING_ENABLE]\n self._command_handler.send_command(command)", "response": "Enables reporting for an analog pin."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef enable_digital_reporting(self, pin):\n port = pin // 8\n command = [self._command_handler.REPORT_DIGITAL + port, self.REPORTING_ENABLE]\n self._command_handler.send_command(command)", "response": "Enables digital reporting on for all 8 bits in the port."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef encoder_config(self, pin_a, pin_b, cb=None):\n data = [pin_a, pin_b]\n self._command_handler.digital_response_table[pin_a][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.ENCODER\n self._command_handler.digital_response_table[pin_a][self._command_handler.RESPONSE_TABLE_CALLBACK] = cb\n self.enable_digital_reporting(pin_a)\n\n self._command_handler.digital_response_table[pin_b][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.ENCODER\n self._command_handler.digital_response_table[pin_b][self._command_handler.RESPONSE_TABLE_CALLBACK] = cb\n self.enable_digital_reporting(pin_b)\n\n self._command_handler.send_sysex(self._command_handler.ENCODER_CONFIG, data)", "response": "This command enables the rotary encoder and willEncodeEncode enable encoder reporting."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_stepper_version(self, timeout=20):\n # get current time\n start_time = time.time()\n\n # wait for up to 20 seconds for a successful capability query to occur\n\n while self._command_handler.stepper_library_version <= 0:\n if time.time() - start_time > timeout:\n if self.verbose is True:\n print(\"Stepper Library Version Request timed-out. \"\n \"Did you send a stepper_request_library_version command?\")\n return\n else:\n pass\n return self._command_handler.stepper_library_version", "response": "Get the stepper library version number."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nnotes THIS METHOD MUST BE CALLED BEFORE ANY I2C REQUEST IS MADE This method initializes Firmata for I2c operations. It allows setting of a read time delay amount, and to optionally track the pins as I2C in the appropriate response table. To track pins: Set the pin_type to ANALOG or DIGITAL and provide the pin numbers. If using ANALOG, pin numbers use the analog number, for example A4: use 4. :param read_delay_time: an optional parameter, default is 0 :param pin_type: ANALOG or DIGITAL to select response table type to track pin numbers :param clk_pin: pin number (see comment above). :param data_pin: pin number (see comment above). :return: No Return Value", "response": "def i2c_config(self, read_delay_time=0, pin_type=None, clk_pin=0, data_pin=0):\n \"\"\"\n NOTE: THIS METHOD MUST BE CALLED BEFORE ANY I2C REQUEST IS MADE\n This method initializes Firmata for I2c operations.\n It allows setting of a read time delay amount, and to optionally track\n the pins as I2C in the appropriate response table.\n To track pins: Set the pin_type to ANALOG or DIGITAL and provide the pin numbers.\n If using ANALOG, pin numbers use the analog number, for example A4: use 4.\n\n :param read_delay_time: an optional parameter, default is 0\n\n :param pin_type: ANALOG or DIGITAL to select response table type to track pin numbers\n\n :param clk_pin: pin number (see comment above).\n\n :param data_pin: pin number (see comment above).\n\n :return: No Return Value\n \"\"\"\n data = [read_delay_time & 0x7f, (read_delay_time >> 7) & 0x7f]\n self._command_handler.send_sysex(self._command_handler.I2C_CONFIG, data)\n\n # If pin type is set, set pin mode in appropriate response table for these pins\n if pin_type:\n if pin_type == self.DIGITAL:\n self._command_handler.digital_response_table[clk_pin][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.I2C\n self._command_handler.digital_response_table[data_pin][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.I2C\n else:\n self._command_handler.analog_response_table[clk_pin][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.I2C\n self._command_handler.analog_response_table[data_pin][self._command_handler.RESPONSE_TABLE_MODE] \\\n = self.I2C"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef i2c_read(self, address, register, number_of_bytes, read_type, cb=None):\n data = [address, read_type, register & 0x7f, (register >> 7) & 0x7f,\n number_of_bytes & 0x7f, (number_of_bytes >> 7) & 0x7f]\n\n # add or update entry in i2c_map for reply\n self._command_handler.i2c_map[address] = [cb, None]\n\n self._command_handler.send_sysex(self._command_handler.I2C_REQUEST, data)", "response": "This method requests a read from an i2c device."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrite data to an i2c device.", "response": "def i2c_write(self, address, *args):\n \"\"\"\n Write data to an i2c device.\n\n :param address: i2c device address\n\n :param args: A variable number of bytes to be sent to the device\n \"\"\"\n data = [address, self.I2C_WRITE]\n for item in args:\n data.append(item & 0x7f)\n data.append((item >> 7) & 0x7f)\n self._command_handler.send_sysex(self._command_handler.I2C_REQUEST, data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef i2c_get_read_data(self, address):\n if address in self._command_handler.i2c_map:\n map_entry = self._command_handler.i2c_map[address]\n return map_entry[1]", "response": "This method retrieves the raw data read from the i2c device."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pin_state_query(self, pin):\n self._command_handler.send_sysex(self._command_handler.PIN_STATE_QUERY, [pin])", "response": "This method issues a pin state query command."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef reset(self):\n # set all output pins to a value of 0\n for pin in range(0, self._command_handler.total_pins_discovered):\n if self._command_handler.digital_response_table[self._command_handler.RESPONSE_TABLE_MODE] \\\n == self.PWM:\n self.analog_write(pin, 0)\n elif self._command_handler.digital_response_table[self._command_handler.RESPONSE_TABLE_MODE] \\\n == self.SERVO:\n self.analog_write(pin, 0)\n elif self._command_handler.digital_response_table[self._command_handler.RESPONSE_TABLE_MODE] \\\n == self.TONE:\n data = [self.TONE_NO_TONE, pin]\n self._command_handler.send_sysex(self._command_handler.TONE_PLAY, data)\n else:\n self.digital_write(pin, 0)\n self._command_handler.system_reset()", "response": "This command sends a reset message to the Arduino. The response tables will be reinitialized if necessary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_analog_latch(self, pin, threshold_type, threshold_value, cb=None):\n if self.ANALOG_LATCH_GT <= threshold_type <= self.ANALOG_LATCH_LTE:\n if 0 <= threshold_value <= 1023:\n self._command_handler.set_analog_latch(pin, threshold_type, threshold_value, cb)\n return True\n else:\n return False", "response": "This method is used to set an analog pin for its data to be latched and saved in the latching table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_digital_latch(self, pin, threshold_type, cb=None):\n if 0 <= threshold_type <= 1:\n self._command_handler.set_digital_latch(pin, threshold_type, cb)\n return True\n else:\n return False", "response": "This method is used to set a digital pin for its data to be latched and saved in the latching table."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconfiguring a servo pin.", "response": "def servo_config(self, pin, min_pulse=544, max_pulse=2400):\n \"\"\"\n Configure a pin as a servo pin. Set pulse min, max in ms.\n\n :param pin: Servo Pin.\n\n :param min_pulse: Min pulse width in ms.\n\n :param max_pulse: Max pulse width in ms.\n\n :return: No return value\n \"\"\"\n self.set_pin_mode(pin, self.SERVO, self.OUTPUT)\n command = [pin, min_pulse & 0x7f, (min_pulse >> 7) & 0x7f,\n max_pulse & 0x7f, (max_pulse >> 7) & 0x7f]\n\n self._command_handler.send_sysex(self._command_handler.SERVO_CONFIG, command)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sonar_config(self, trigger_pin, echo_pin, cb=None, ping_interval=50, max_distance=200):\n if max_distance > 200:\n max_distance = 200\n max_distance_lsb = max_distance & 0x7f\n max_distance_msb = (max_distance >> 7) & 0x7f\n data = [trigger_pin, echo_pin, ping_interval, max_distance_lsb, max_distance_msb]\n self.set_pin_mode(trigger_pin, self.SONAR, self.INPUT)\n self.set_pin_mode(echo_pin, self.SONAR, self.INPUT)\n # update the ping data map for this pin\n if len(self._command_handler.active_sonar_map) > 6:\n if self.verbose:\n print(\"sonar_config: maximum number of devices assigned - ignoring request\")\n return\n else:\n with self.data_lock:\n\n # self._command_handler.active_sonar_map[trigger_pin] = self.IGNORE\n self._command_handler.active_sonar_map[trigger_pin] = [cb, [self.IGNORE]]\n self._command_handler.send_sysex(self._command_handler.SONAR_CONFIG, data)", "response": "Configure the pins ping_interval and max_distance for a HC - SR04 type device."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stepper_config(self, steps_per_revolution, stepper_pins):\n data = [self.STEPPER_CONFIGURE, steps_per_revolution & 0x7f, (steps_per_revolution >> 7) & 0x7f]\n for pin in range(len(stepper_pins)):\n data.append(stepper_pins[pin])\n self._command_handler.send_sysex(self._command_handler.STEPPER_DATA, data)", "response": "Configure stepper motor prior to operation."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stepper_step(self, motor_speed, number_of_steps):\n if number_of_steps > 0:\n direction = 1\n else:\n direction = 0\n abs_number_of_steps = abs(number_of_steps)\n data = [self.STEPPER_STEP, motor_speed & 0x7f, (motor_speed >> 7) & 0x7f, (motor_speed >> 14) & 0x7f,\n abs_number_of_steps & 0x7f, (abs_number_of_steps >> 7) & 0x7f, direction]\n self._command_handler.send_sysex(self._command_handler.STEPPER_DATA, data)", "response": "Move a stepper motor for the specified number of steps at the specified speed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrequesting the stepper library version from the Arduino.", "response": "def stepper_request_library_version(self):\n \"\"\"\n Request the stepper library version from the Arduino.\n To retrieve the version after this command is called, call\n get_stepper_version\n \"\"\"\n data = [self.STEPPER_LIBRARY_VERSION]\n self._command_handler.send_sysex(self._command_handler.STEPPER_DATA, data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef open(self, verbose):\n # open a serial port\n if verbose:\n print('\\nOpening Arduino Serial port %s ' % self.port_id)\n\n try:\n\n # in case the port is already open, let's close it and then\n # reopen it\n self.arduino.close()\n time.sleep(1)\n self.arduino.open()\n time.sleep(1)\n return self.arduino\n\n except Exception:\n # opened failed - will report back to caller\n raise", "response": "Open the Arduino serial port using the configuration data\nAttributeNames returns a reference to this instance"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write(self, data):\n if sys.version_info[0] < 3:\n self.arduino.write(data)\n else:\n self.arduino.write(bytes([ord(data)]))", "response": "write the data to the serial port\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_blink_rate(self, b):\n if b > 3:\n b = 0 # turn off if not sure\n self.firmata.i2c_write(self.board_address,\n (self.HT16K33_BLINK_CMD | self.HT16K33_BLINK_DISPLAYON | (b << 1)))", "response": "Set the user s blink rate"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_brightness(self, brightness):\n if brightness > 15:\n brightness = 15\n brightness |= 0xE0\n self.brightness = brightness\n self.firmata.i2c_write(0x70, brightness)", "response": "Set the brightness level for the entire display."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_bit_map(self, shape, color):\n for row in range(0, 8):\n data = shape[row]\n # shift data into buffer\n bit_mask = 0x80\n for column in range(0, 8):\n if data & bit_mask:\n self.set_pixel(row, column, color, True)\n bit_mask >>= 1\n self.output_entire_buffer()", "response": "Populate the bit map with the supplied shape and color."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef output_entire_buffer(self):\n green = 0\n red = 0\n\n for row in range(0, 8):\n for col in range(0, 8):\n if self.display_buffer[row][col] == self.LED_GREEN:\n green |= 1 << col\n elif self.display_buffer[row][col] == self.LED_RED:\n red |= 1 << col\n elif self.display_buffer[row][col] == self.LED_YELLOW:\n green |= 1 << col\n red |= 1 << col\n elif self.display_buffer[row][col] == self.LED_OFF:\n green &= ~(1 << col)\n red &= ~(1 << col)\n\n self.firmata.i2c_write(0x70, row * 2, 0, green)\n self.firmata.i2c_write(0x70, row * 2 + 1, 0, red)", "response": "Write the entire buffer to the display"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nclears all LEDs in display buffer.", "response": "def clear_display_buffer(self):\n \"\"\"\n Set all led's to off.\n \"\"\"\n for row in range(0, 8):\n self.firmata.i2c_write(0x70, row * 2, 0, 0)\n self.firmata.i2c_write(0x70, (row * 2) + 1, 0, 0)\n\n for column in range(0, 8):\n self.display_buffer[row][column] = 0"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef report_version(self, data):\n self.firmata_version.append(data[0]) # add major\n self.firmata_version.append(data[1])", "response": "This method processes the report version message and adds it to the list of available version information."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_digital_latch(self, pin, threshold_type, cb):\n with self.pymata.data_lock:\n self.digital_latch_table[pin] = [self.LATCH_ARMED, threshold_type, 0, 0, cb]", "response": "This method is used to set the latching for a digital pin."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_analog_latch_data(self, pin):\n with self.pymata.data_lock:\n pin_data = self.analog_latch_table[pin]\n current_latch_data = [pin,\n pin_data[self.LATCH_STATE],\n pin_data[self.ANALOG_LATCHED_DATA],\n pin_data[self.ANALOG_TIME_STAMP],\n pin_data[self.ANALOG_LATCH_CALLBACK]]\n # if this is latched data, clear the latch table entry for this pin\n if pin_data[self.LATCH_STATE] == self.LATCH_LATCHED:\n self.analog_latch_table[pin] = [0, 0, 0, 0, 0, None]\n return current_latch_data", "response": "This method reads the analog latch table for the specified pin and returns a list containing the current state latched_data and time_stamp."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_digital_latch_data(self, pin):\n with self.pymata.data_lock:\n pin_data = self.digital_latch_table[pin]\n current_latch_data = [pin,\n pin_data[self.LATCH_STATE],\n pin_data[self.DIGITAL_LATCHED_DATA],\n pin_data[self.DIGITAL_TIME_STAMP],\n pin_data[self.DIGITAL_LATCH_CALLBACK]]\n if pin_data[self.LATCH_STATE] == self.LATCH_LATCHED:\n self.digital_latch_table[pin] = [0, 0, 0, 0, None]\n return current_latch_data", "response": "This method reads the digital latch table for the specified pin and returns a list containing the current latch state latched data and time stamps."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef analog_message(self, data):\n with self.pymata.data_lock:\n # hold on to the previous value\n previous_value = \\\n self.analog_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_PIN_DATA_VALUE]\n self.analog_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_PIN_DATA_VALUE] \\\n = (data[self.MSB] << 7) + data[self.LSB]\n pin = data[0]\n pin_response_data_data = self.analog_response_table[pin]\n value = pin_response_data_data[self.RESPONSE_TABLE_PIN_DATA_VALUE]\n # check to see if there is a callback function attached to this pin\n callback = self.analog_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_CALLBACK]\n # send the pin mode, pin number, and current data value\n if callback is not None:\n if value != previous_value:\n # has the value changed since the last report\n callback([self.pymata.ANALOG, pin, value])\n\n # check if data is to be latched\n # get the analog latching table entry for this pin\n latching_entry = self.analog_latch_table[pin]\n if latching_entry[self.LATCH_STATE] == self.LATCH_ARMED:\n # Has the latching criteria been met\n if latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.ANALOG_LATCH_GT:\n if value > latching_entry[self.ANALOG_LATCH_DATA_TARGET]:\n if latching_entry[self.ANALOG_LATCH_CALLBACK] is not None:\n self.analog_latch_table[pin] = [0, 0, 0, 0, 0, None]\n latching_entry[self.ANALOG_LATCH_CALLBACK]([self.pymata.ANALOG | self.pymata.LATCH_MODE,\n pin, value, time.time()])\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.ANALOG_LATCHED_DATA] = value\n # time stamp it\n updated_latch_entry[self.ANALOG_TIME_STAMP] = time.time()\n self.analog_latch_table[pin] = updated_latch_entry\n else:\n pass # haven't hit target\n elif latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.ANALOG_LATCH_GTE:\n if value >= latching_entry[self.ANALOG_LATCH_DATA_TARGET]:\n if latching_entry[self.ANALOG_LATCH_CALLBACK] is not None:\n self.analog_latch_table[pin] = [0, 0, 0, 0, 0, None]\n latching_entry[self.ANALOG_LATCH_CALLBACK]([self.pymata.ANALOG | self.pymata.LATCH_MODE,\n pin, value, time.time()])\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.ANALOG_LATCHED_DATA] = value\n # time stamp it\n updated_latch_entry[self.ANALOG_TIME_STAMP] = time.time()\n self.analog_latch_table[pin] = updated_latch_entry\n else:\n pass # haven't hit target:\n elif latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.ANALOG_LATCH_LT:\n if value < latching_entry[self.ANALOG_LATCH_DATA_TARGET]:\n if latching_entry[self.ANALOG_LATCH_CALLBACK] is not None:\n latching_entry[self.ANALOG_LATCH_CALLBACK]([self.pymata.ANALOG | self.pymata.LATCH_MODE,\n pin, value, time.time()])\n self.analog_latch_table[pin] = [0, 0, 0, 0, 0, None]\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.ANALOG_LATCHED_DATA] = value\n # time stamp it\n updated_latch_entry[self.ANALOG_TIME_STAMP] = time.time()\n self.analog_latch_table[pin] = updated_latch_entry\n else:\n pass # haven't hit target:\n elif latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.ANALOG_LATCH_LTE:\n if value <= latching_entry[self.ANALOG_LATCH_DATA_TARGET]:\n if latching_entry[self.ANALOG_LATCH_CALLBACK] is not None:\n latching_entry[self.ANALOG_LATCH_CALLBACK]([self.pymata.ANALOG | self.pymata.LATCH_MODE,\n pin, value, time.time()])\n self.analog_latch_table[pin] = [0, 0, 0, 0, 0, None]\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.ANALOG_LATCHED_DATA] = value\n # time stamp it\n updated_latch_entry[self.ANALOG_TIME_STAMP] = time.time()\n self.analog_latch_table[pin] = updated_latch_entry\n else:\n pass # haven't hit target:\n else:\n pass", "response": "This method handles the incoming analog data message. It stores the data value for the specified pin in the response table and sends the callback function to the user if the callback function is not provided by the user."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef digital_message(self, data):\n port = data[0]\n port_data = (data[self.MSB] << 7) + data[self.LSB]\n\n # set all the pins for this reporting port\n # get the first pin number for this report\n pin = port * 8\n for pin in range(pin, min(pin + 8, self.total_pins_discovered)):\n # shift through all the bit positions and set the digital response table\n with self.pymata.data_lock:\n # look at the previously stored value for this pin\n prev_data = self.digital_response_table[pin][self.RESPONSE_TABLE_PIN_DATA_VALUE]\n # get the current value\n self.digital_response_table[pin][self.RESPONSE_TABLE_PIN_DATA_VALUE] = port_data & 0x01\n # if the values differ and callback is enabled for the pin, then send out the callback\n if prev_data != port_data & 0x01:\n callback = self.digital_response_table[pin][self.RESPONSE_TABLE_CALLBACK]\n if callback:\n callback([self.pymata.DIGITAL, pin,\n self.digital_response_table[pin][self.RESPONSE_TABLE_PIN_DATA_VALUE]])\n\n # determine if the latch data table needs to be updated for each pin\n latching_entry = self.digital_latch_table[pin]\n if latching_entry[self.LATCH_STATE] == self.LATCH_ARMED:\n if latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.DIGITAL_LATCH_LOW:\n if (port_data & 0x01) == 0:\n if latching_entry[self.DIGITAL_LATCH_CALLBACK] is not None:\n self.digital_latch_table[pin] = [0, 0, 0, 0, None]\n latching_entry[self.DIGITAL_LATCH_CALLBACK](\n [self.pymata.OUTPUT | self.pymata.LATCH_MODE,\n pin, 0, time.time()])\n\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.DIGITAL_LATCHED_DATA] = self.DIGITAL_LATCH_LOW\n # time stamp it\n updated_latch_entry[self.DIGITAL_TIME_STAMP] = time.time()\n else:\n pass\n elif latching_entry[self.LATCHED_THRESHOLD_TYPE] == self.DIGITAL_LATCH_HIGH:\n if port_data & 0x01:\n if latching_entry[self.DIGITAL_LATCH_CALLBACK] is not None:\n self.digital_latch_table[pin] = [0, 0, 0, 0, None]\n latching_entry[self.DIGITAL_LATCH_CALLBACK](\n [self.pymata.OUTPUT | self.pymata.LATCH_MODE,\n pin, 1, time.time()])\n else:\n updated_latch_entry = latching_entry\n updated_latch_entry[self.LATCH_STATE] = self.LATCH_LATCHED\n updated_latch_entry[self.DIGITAL_LATCHED_DATA] = self.DIGITAL_LATCH_HIGH\n # time stamp it\n updated_latch_entry[self.DIGITAL_TIME_STAMP] = time.time()\n else:\n pass\n else:\n pass\n\n # get the next data bit\n port_data >>= 1", "response": "This method handles the incoming digital message. It stores the data values in the response table and then sends out the callback and latch data table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encoder_data(self, data):\n prev_val = self.digital_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_PIN_DATA_VALUE]\n val = int((data[self.MSB] << 7) + data[self.LSB])\n # set value so that it shows positive and negative values\n if val > 8192:\n val -= 16384\n pin = data[0]\n with self.pymata.data_lock:\n self.digital_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_PIN_DATA_VALUE] = val\n if prev_val != val:\n callback = self.digital_response_table[pin][self.RESPONSE_TABLE_CALLBACK]\n if callback is not None:\n callback([self.pymata.ENCODER, pin,\n self.digital_response_table[pin][self.RESPONSE_TABLE_PIN_DATA_VALUE]])", "response": "This method handles incoming encoder data message and stores the value in the response table."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sonar_data(self, data):\n val = int((data[self.MSB] << 7) + data[self.LSB])\n pin_number = data[0]\n with self.pymata.data_lock:\n sonar_pin_entry = self.active_sonar_map[pin_number]\n # also write it into the digital response table\n self.digital_response_table[data[self.RESPONSE_TABLE_MODE]][self.RESPONSE_TABLE_PIN_DATA_VALUE] = val\n # send data through callback if there is a callback function for the pin\n if sonar_pin_entry[0] is not None:\n # check if value changed since last reading\n if sonar_pin_entry[1] != val:\n self.active_sonar_map[pin_number][0]([self.pymata.SONAR, pin_number, val])\n # update the data in the table with latest value\n sonar_pin_entry[1] = val\n self.active_sonar_map[pin_number] = sonar_pin_entry", "response": "This method handles incoming sonar data message and stores the data in the response table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send_sysex(self, sysex_command, sysex_data=None):\n if not sysex_data:\n sysex_data = []\n\n # convert the message command and data to characters\n sysex_message = chr(self.START_SYSEX)\n sysex_message += chr(sysex_command)\n if len(sysex_data):\n for d in sysex_data:\n sysex_message += chr(d)\n sysex_message += chr(self.END_SYSEX)\n\n for data in sysex_message:\n self.pymata.transport.write(data)", "response": "This method sends a Sysex command to Firmata with any accompanying data."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef system_reset(self):\n data = chr(self.SYSTEM_RESET)\n self.pymata.transport.write(data)\n\n # response table re-initialization\n # for each pin set the mode to input and the last read data value to zero\n with self.pymata.data_lock:\n # remove all old entries from existing tables\n for _ in range(len(self.digital_response_table)):\n self.digital_response_table.pop()\n\n for _ in range(len(self.analog_response_table)):\n self.analog_response_table.pop()\n\n # reinitialize tables\n for pin in range(0, self.total_pins_discovered):\n response_entry = [self.pymata.INPUT, 0, None]\n self.digital_response_table.append(response_entry)\n\n for pin in range(0, self.number_of_analog_pins_discovered):\n response_entry = [self.pymata.INPUT, 0, None]\n self.analog_response_table.append(response_entry)", "response": "Send the system reset command to the Arduino."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run(self):\n # To add a command to the command dispatch table, append here.\n self.command_dispatch.update({self.REPORT_VERSION: [self.report_version, 2]})\n self.command_dispatch.update({self.REPORT_FIRMWARE: [self.report_firmware, 1]})\n self.command_dispatch.update({self.ANALOG_MESSAGE: [self.analog_message, 2]})\n self.command_dispatch.update({self.DIGITAL_MESSAGE: [self.digital_message, 2]})\n self.command_dispatch.update({self.ENCODER_DATA: [self.encoder_data, 3]})\n self.command_dispatch.update({self.SONAR_DATA: [self.sonar_data, 3]})\n self.command_dispatch.update({self.STRING_DATA: [self._string_data, 2]})\n self.command_dispatch.update({self.I2C_REPLY: [self.i2c_reply, 2]})\n self.command_dispatch.update({self.CAPABILITY_RESPONSE: [self.capability_response, 2]})\n self.command_dispatch.update({self.PIN_STATE_RESPONSE: [self.pin_state_response, 2]})\n self.command_dispatch.update({self.ANALOG_MAPPING_RESPONSE: [self.analog_mapping_response, 2]})\n self.command_dispatch.update({self.STEPPER_DATA: [self.stepper_version_response, 2]})\n\n while not self.is_stopped():\n if len(self.pymata.command_deque):\n # get next byte from the deque and process it\n data = self.pymata.command_deque.popleft()\n\n # this list will be populated with the received data for the command\n command_data = []\n\n # process sysex commands\n if data == self.START_SYSEX:\n # next char is the actual sysex command\n # wait until we can get data from the deque\n while len(self.pymata.command_deque) == 0:\n pass\n sysex_command = self.pymata.command_deque.popleft()\n # retrieve the associated command_dispatch entry for this command\n dispatch_entry = self.command_dispatch.get(sysex_command)\n\n # get a \"pointer\" to the method that will process this command\n method = dispatch_entry[0]\n\n # now get the rest of the data excluding the END_SYSEX byte\n end_of_sysex = False\n while not end_of_sysex:\n # wait for more data to arrive\n while len(self.pymata.command_deque) == 0:\n pass\n data = self.pymata.command_deque.popleft()\n if data != self.END_SYSEX:\n command_data.append(data)\n else:\n end_of_sysex = True\n\n # invoke the method to process the command\n method(command_data)\n # go to the beginning of the loop to process the next command\n continue\n\n # is this a command byte in the range of 0x80-0xff - these are the non-sysex messages\n\n elif 0x80 <= data <= 0xff:\n # look up the method for the command in the command dispatch table\n # for the digital reporting the command value is modified with port number\n # the handler needs the port to properly process, so decode that from the command and\n # place in command_data\n if 0x90 <= data <= 0x9f:\n port = data & 0xf\n command_data.append(port)\n data = 0x90\n # the pin number for analog data is embedded in the command so, decode it\n elif 0xe0 <= data <= 0xef:\n pin = data & 0xf\n command_data.append(pin)\n data = 0xe0\n else:\n pass\n\n dispatch_entry = self.command_dispatch.get(data)\n\n # this calls the method retrieved from the dispatch table\n method = dispatch_entry[0]\n\n # get the number of parameters that this command provides\n num_args = dispatch_entry[1]\n\n # look at the number of args that the selected method requires\n # now get that number of bytes to pass to the called method\n for i in range(num_args):\n while len(self.pymata.command_deque) == 0:\n pass\n data = self.pymata.command_deque.popleft()\n command_data.append(data)\n # go execute the command with the argument list\n method(command_data)\n\n # go to the beginning of the loop to process the next command\n continue\n else:\n time.sleep(.1)", "response": "This method starts the thread that processes all the messages coming from Firmata and processes them."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef retrieve_url(self, url):\n\n try:\n r = requests.get(url)\n except requests.ConnectionError:\n raise exceptions.RetrieveError('Connection fail')\n\n if r.status_code >= 400:\n raise exceptions.RetrieveError('Connected, but status code is %s' % (r.status_code))\n\n real_url = r.url\n content = r.content\n\n try:\n content_type = r.headers['Content-Type']\n except KeyError:\n content_type, encoding = mimetypes.guess_type(real_url, strict=False)\n\n self.response = r\n\n return content_type.lower(), content", "response": "Use requests to fetch remote content and return content type and encoding"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_html(self, html):\n\n soup = BeautifulSoup(html, self.parser)\n\n title_tag = soup.find('title')\n self.result.title = title_tag.string if title_tag else None\n\n self.soup = soup\n\n return soup", "response": "Use BeautifulSoup to parse HTML and XML\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncombines finder_image_urls and extender_image_urls and return a list of all the image urls that are not already in the list.", "response": "def image_urls(self):\n \"\"\"\n Combine finder_image_urls and extender_image_urls,\n remove duplicate but keep order\n \"\"\"\n\n all_image_urls = self.finder_image_urls[:]\n for image_url in self.extender_image_urls:\n if image_url not in all_image_urls:\n all_image_urls.append(image_url)\n\n return all_image_urls"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ggpht_s1600_extender(pipeline_index,\n finder_image_urls,\n extender_image_urls=[],\n *args, **kwargs):\n \"\"\"\n Example:\n http://lh4.ggpht.com/-fFi-qJRuxeY/UjwHSOTHGOI/AAAAAAAArgE/SWTMT-hXzB4/s640/Celeber-ru-Emma-Watson-Net-A-Porter-The-Edit-Magazine-Photoshoot-2013-01.jpg\n to\n http://lh4.ggpht.com/-fFi-qJRuxeY/UjwHSOTHGOI/AAAAAAAArgE/SWTMT-hXzB4/s1600/Celeber-ru-Emma-Watson-Net-A-Porter-The-Edit-Magazine-Photoshoot-2013-01.jpg\n \"\"\"\n\n now_extender_image_urls = []\n\n search_re = re.compile(r'/s\\d+/', re.IGNORECASE)\n\n for image_url in finder_image_urls:\n if 'ggpht.com/' in image_url.lower():\n if search_re.search(image_url):\n extender_image_url = search_re.sub('/s1600/', image_url)\n now_extender_image_urls.append(extender_image_url)\n\n output = {}\n output['extender_image_urls'] = extender_image_urls + now_extender_image_urls\n\n return output", "response": "This function is used to extend the GGH to 1600."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef background_image_finder(pipeline_index,\n soup,\n finder_image_urls=[],\n *args, **kwargs):\n \"\"\"\n Find image URL in background-image\n\n Example:\n
\n to\n http://distilleryimage10.ak.instagram.com/bde04558a43b11e28e5d22000a1f979a_7.jpg\n \"\"\"\n\n now_finder_image_urls = []\n\n for tag in soup.find_all(style=True):\n style_string = tag['style']\n if 'background-image' in style_string.lower():\n style = cssutils.parseStyle(style_string)\n background_image = style.getProperty('background-image')\n if background_image:\n for property_value in background_image.propertyValue:\n background_image_url = str(property_value.value)\n if background_image_url:\n if (background_image_url not in finder_image_urls) and \\\n (background_image_url not in now_finder_image_urls):\n now_finder_image_urls.append(background_image_url)\n\n output = {}\n output['finder_image_urls'] = finder_image_urls + now_finder_image_urls\n\n return output", "response": "This function will find images in the background - image - finder page."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind image URL in 's src attribute", "response": "def img_src_finder(pipeline_index,\n soup,\n finder_image_urls=[],\n *args, **kwargs):\n \"\"\"\n Find image URL in 's src attribute\n \"\"\"\n\n now_finder_image_urls = []\n\n for img in soup.find_all('img'):\n src = img.get('src', None)\n if src:\n src = str(src)\n if (src not in finder_image_urls) and \\\n (src not in now_finder_image_urls):\n now_finder_image_urls.append(src)\n\n output = {}\n output['finder_image_urls'] = finder_image_urls + now_finder_image_urls\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef a_href_finder(pipeline_index,\n soup,\n finder_image_urls=[],\n *args, **kwargs):\n \"\"\"\n Find image URL in 's href attribute\n \"\"\"\n\n now_finder_image_urls = []\n\n for a in soup.find_all('a'):\n href = a.get('href', None)\n if href:\n href = str(href)\n if filter(href.lower().endswith, ('.jpg', '.jpeg', '.gif', '.png')):\n if (href not in finder_image_urls) and \\\n (href not in now_finder_image_urls):\n now_finder_image_urls.append(href)\n\n output = {}\n output['finder_image_urls'] = finder_image_urls + now_finder_image_urls\n\n return output", "response": "Find image URL in a link s href attribute"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the node name where the name would land to", "response": "def _getnodenamefor(self, name):\n \"Return the node name where the ``name`` would land to\"\n return 'node_' + str(\n (abs(binascii.crc32(b(name)) & 0xffffffff) % self.no_servers) + 1)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef getnodefor(self, name):\n \"Return the node where the ``name`` would land to\"\n node = self._getnodenamefor(name)\n return {node: self.cluster['nodes'][node]}", "response": "Return the node where the name would land to"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the encoding idletime or refcount about the key", "response": "def object(self, infotype, key):\n \"Return the encoding, idletime, or refcount about the key\"\n redisent = self.redises[self._getnodenamefor(key) + '_slave']\n return getattr(redisent, 'object')(infotype, key)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _rc_brpoplpush(self, src, dst, timeout=0):\n rpop = self.brpop(src, timeout)\n if rpop is not None:\n self.lpush(dst, rpop[1])\n return rpop[1]\n return None", "response": "Internal helper for _rc_brpoplpush."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _rc_sdiff(self, src, *args):\n args = list_or_args(src, args)\n src_set = self.smembers(args.pop(0))\n if src_set is not set([]):\n for key in args:\n src_set.difference_update(self.smembers(key))\n return src_set", "response": "Returns the members of the set resulting from the difference between the first set and all the successive sets."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstoring the difference of sets ``src``, ``args`` into a new set named ``dest``. Returns the number of keys in the new set.", "response": "def _rc_sdiffstore(self, dst, src, *args):\n \"\"\"\n Store the difference of sets ``src``, ``args`` into a new\n set named ``dest``. Returns the number of keys in the new set.\n \"\"\"\n args = list_or_args(src, args)\n result = self.sdiff(*args)\n if result is not set([]):\n return self.sadd(dst, *list(result))\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _rc_sinter(self, src, *args):\n args = list_or_args(src, args)\n src_set = self.smembers(args.pop(0))\n if src_set is not set([]):\n for key in args:\n src_set.intersection_update(self.smembers(key))\n return src_set", "response": "Returns the members of the set resulting from the difference between the first set and all the successive sets."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstore the difference of sets ``src``, ``args`` into a new set named ``dest``. Returns the number of keys in the new set.", "response": "def _rc_sinterstore(self, dst, src, *args):\n \"\"\"\n Store the difference of sets ``src``, ``args`` into a new\n set named ``dest``. Returns the number of keys in the new set.\n \"\"\"\n args = list_or_args(src, args)\n result = self.sinter(*args)\n if result is not set([]):\n return self.sadd(dst, *list(result))\n return 0"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmoves value from set src to set dst not atomic", "response": "def _rc_smove(self, src, dst, value):\n \"\"\"\n Move ``value`` from set ``src`` to set ``dst``\n not atomic\n \"\"\"\n if self.type(src) != b(\"set\"):\n return self.smove(src + \"{\" + src + \"}\", dst, value)\n if self.type(dst) != b(\"set\"):\n return self.smove(dst + \"{\" + dst + \"}\", src, value)\n if self.srem(src, value):\n return 1 if self.sadd(dst, value) else 0\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _rc_sunion(self, src, *args):\n args = list_or_args(src, args)\n src_set = self.smembers(args.pop(0))\n if src_set is not set([]):\n for key in args:\n src_set.update(self.smembers(key))\n return src_set", "response": "Returns the members of the set resulting from the union between the first set and all the successive sets."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _rc_sunionstore(self, dst, src, *args):\n args = list_or_args(src, args)\n result = self.sunion(*args)\n if result is not set([]):\n return self.sadd(dst, *list(result))\n return 0", "response": "Internal sunion store function."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _rc_mset(self, mapping):\n \"Sets each key in the ``mapping`` dict to its corresponding value\"\n result = True\n for k, v in iteritems(mapping):\n result = result and self.set(k, v)\n return result", "response": "Sets each key in the mapping dict to its corresponding value"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets each key in the mapping dict to its corresponding value. Returns True if successful False otherwise.", "response": "def _rc_msetnx(self, mapping):\n \"\"\"\n Sets each key in the ``mapping`` dict to its corresponding value if\n none of the keys are already set\n \"\"\"\n for k in iterkeys(mapping):\n if self.exists(k):\n return False\n\n return self._rc_mset(mapping)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of values ordered identically to ``*args``", "response": "def _rc_mget(self, keys, *args):\n \"\"\"\n Returns a list of values ordered identically to ``*args``\n \"\"\"\n args = list_or_args(keys, args)\n result = []\n for key in args:\n result.append(self.get(key))\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrenames key src to dst.", "response": "def _rc_rename(self, src, dst):\n \"\"\"\n Rename key ``src`` to ``dst``\n \"\"\"\n if src == dst:\n return self.rename(src + \"{\" + src + \"}\", src)\n if not self.exists(src):\n return self.rename(src + \"{\" + src + \"}\", src)\n\n self.delete(dst)\n ktype = self.type(src)\n kttl = self.ttl(src)\n\n if ktype == b('none'):\n return False\n\n if ktype == b('string'):\n self.set(dst, self.get(src))\n elif ktype == b('hash'):\n self.hmset(dst, self.hgetall(src))\n elif ktype == b('list'):\n for k in self.lrange(src, 0, -1):\n self.rpush(dst, k)\n elif ktype == b('set'):\n for k in self.smembers(src):\n self.sadd(dst, k)\n elif ktype == b('zset'):\n for k, v in self.zrange(src, 0, -1, withscores=True):\n self.zadd(dst, v, k)\n\n # Handle keys with an expire time set\n kttl = -1 if kttl is None or kttl < 0 else int(kttl)\n if kttl != -1:\n self.expire(dst, kttl)\n\n return self.delete(src)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrename key src to dst doesn t already exist", "response": "def _rc_renamenx(self, src, dst):\n \"Rename key ``src`` to ``dst`` if ``dst`` doesn't already exist\"\n if self.exists(dst):\n return False\n\n return self._rc_rename(src, dst)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of keys matching pattern", "response": "def _rc_keys(self, pattern='*'):\n \"Returns a list of keys matching ``pattern``\"\n\n result = []\n for alias, redisent in iteritems(self.redises):\n if alias.find('_slave') == -1:\n continue\n\n result.extend(redisent.keys(pattern))\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the number of keys in the current database", "response": "def _rc_dbsize(self):\n \"Returns the number of keys in the current database\"\n\n result = 0\n for alias, redisent in iteritems(self.redises):\n if alias.find('_slave') == -1:\n continue\n\n result += redisent.dbsize()\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prepare(self):\n\n # Create a collection for the attributes and elements of\n # this instance.\n attributes, elements = OrderedDict(), []\n\n # Initialize the namespace map.\n nsmap = dict([self.meta.namespace])\n\n # Iterate through all declared items.\n for name, item in self._items.items():\n if isinstance(item, Attribute):\n # Prepare the item as an attribute.\n attributes[name] = item.prepare(self)\n\n elif isinstance(item, Element):\n # Update the nsmap.\n nsmap.update([item.namespace])\n\n # Prepare the item as an element.\n elements.append(item)\n\n # Return the collected attributes and elements\n return attributes, elements, nsmap", "response": "Prepare the date in the instance state for serialization."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sign(xml, stream, password=None):\n\n # Import xmlsec here to delay initializing the C library in\n # case we don't need it.\n import xmlsec\n\n # Resolve the SAML/2.0 element in question.\n from saml.schema.base import _element_registry\n element = _element_registry.get(xml.tag)\n\n # Create a signature template for RSA-SHA1 enveloped signature.\n signature_node = xmlsec.template.create(\n xml,\n xmlsec.Transform.EXCL_C14N,\n xmlsec.Transform.RSA_SHA1)\n\n # Add the node to the document.\n xml.insert(element.meta.signature_index, signature_node)\n\n # Add the node to the signature template.\n ref = xmlsec.template.add_reference(\n signature_node, xmlsec.Transform.SHA1)\n\n # Add the enveloped transform descriptor.\n xmlsec.template.add_transform(ref, xmlsec.Transform.ENVELOPED)\n\n # Create a digital signature context (no key manager is needed).\n ctx = xmlsec.SignatureContext()\n\n # Load private key.\n key = xmlsec.Key.from_memory(stream, xmlsec.KeyFormat.PEM, password)\n\n # Set the key on the context.\n ctx.key = key\n\n # Sign the template.\n ctx.sign(signature_node)", "response": "Sign an XML document with the given private key."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef verify(xml, stream):\n # Import xmlsec here to delay initializing the C library in\n # case we don't need it.\n import xmlsec\n\n # Find the node.\n signature_node = xmlsec.tree.find_node(xml, xmlsec.Node.SIGNATURE)\n if signature_node is None:\n # No `signature` node found; we cannot verify\n return False\n\n # Create a digital signature context (no key manager is needed).\n ctx = xmlsec.SignatureContext()\n\n # Register and \n ctx.register_id(xml)\n for assertion in xml.xpath(\"//*[local-name()='Assertion']\"):\n ctx.register_id(assertion)\n\n # Load the public key.\n key = None\n for fmt in [\n xmlsec.KeyFormat.PEM,\n xmlsec.KeyFormat.CERT_PEM]:\n stream.seek(0)\n try:\n key = xmlsec.Key.from_memory(stream, fmt)\n break\n except ValueError: \n # xmlsec now throws when it can't load the key\n pass\n\n # Set the key on the context.\n ctx.key = key\n\n # Verify the signature.\n try:\n ctx.verify(signature_node)\n\n return True\n\n except Exception:\n return False", "response": "Verify the signaure of an XML document with the given certificate."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds number of photos to each gallery.", "response": "def get_queryset(self, request):\n \"\"\"\n Add number of photos to each gallery.\n \"\"\"\n qs = super(GalleryAdmin, self).get_queryset(request)\n return qs.annotate(photo_count=Count('photos'))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save_model(self, request, obj, form, change):\n obj.author = request.user\n obj.save()", "response": "Save the object to the database."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_byteranges(cls, environ):\n r = []\n s = environ.get(cls.header_range, '').replace(' ','').lower()\n if s:\n l = s.split('=')\n if len(l) == 2:\n unit, vals = tuple(l)\n if unit == 'bytes' and vals:\n gen_rng = ( tuple(rng.split('-')) for rng in vals.split(',') if '-' in rng )\n for start, end in gen_rng:\n if start or end:\n r.append( (int(start) if start else None, int(end) if end else None) )\n return r", "response": "Outputs a list of tuples with ranges or the empty list\n Outputs a list of tuples with ranges or the empty list\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert_ranges(cls, ranges, length):\n result = []\n for start, end in ranges:\n if end is None:\n result.append( (start, length-1) )\n elif start is None:\n s = length - end\n result.append( (0 if s < 0 else s, length-1) )\n else:\n result.append( (start, end if end < length else length-1) )\n return result", "response": "Converts a list of byte ranges to a list of valid byte ranges."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsorting and removes overlaps", "response": "def condense_ranges(cls, ranges):\n \"\"\"Sorts and removes overlaps\"\"\"\n result = []\n if ranges:\n ranges.sort(key=lambda tup: tup[0])\n result.append(ranges[0])\n for i in range(1, len(ranges)):\n if result[-1][1] + 1 >= ranges[i][0]:\n result[-1] = (result[-1][0], max(result[-1][1], ranges[i][1]))\n else:\n result.append(ranges[i])\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrendering the selected social widget.", "response": "def social_widget_render(parser, token):\n \"\"\" Renders the selected social widget. You can specify optional settings\n that will be passed to widget template.\n\n Sample usage:\n {% social_widget_render widget_template ke1=val1 key2=val2 %}\n\n For example to render Twitter follow button you can use code like this:\n {% social_widget_render 'twitter/follow_button.html' username=\"ev\" %}\n \"\"\"\n bits = token.split_contents()\n tag_name = bits[0]\n\n if len(bits) < 2:\n raise TemplateSyntaxError(\"'%s' takes at least one argument\" %\n tag_name)\n args = []\n kwargs = {}\n\n bits = bits[1:]\n\n if len(bits):\n for bit in bits:\n match = kwarg_re.match(bit)\n if not match:\n raise TemplateSyntaxError(\"Malformed arguments to %s tag\" %\n tag_name)\n name, value = match.groups()\n\n if name:\n # Replacing hyphens with underscores because\n # variable names cannot contain hyphens.\n name = name.replace('-', '_')\n kwargs[name] = parser.compile_filter(value)\n else:\n args.append(parser.compile_filter(value))\n\n return SocialWidgetNode(args, kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmultiplies the data matrix or vector multiplied by the given multiplier.", "response": "def multiply(self, multiplier, axis=None):\n \"\"\"\n In-place multiplication\n\n :param multiplier: A matrix or vector to be multiplied\n :param axis: The dim along which 'multiplier' is multiplied\n :return: Nothing (as it performs in-place operations)\n \"\"\"\n if self.finalized:\n if multiplier.ndim == 1:\n if axis == 0: # multiplier is np.array of length |haplotypes|\n raise NotImplementedError('The method is not yet implemented for the axis.')\n elif axis == 1: # multiplier is np.array of length |loci|\n sz = len(multiplier)\n multiplier_mat = lil_matrix((sz, sz))\n multiplier_mat.setdiag(multiplier)\n for hid in xrange(self.shape[1]):\n self.data[hid] = self.data[hid] * multiplier_mat\n elif axis == 2: # multiplier is np.array of length |reads|\n for hid in xrange(self.shape[1]):\n self.data[hid].data *= multiplier[self.data[hid].indices]\n else:\n raise RuntimeError('The axis should be 0, 1, or 2.')\n elif multiplier.ndim == 2:\n if axis == 0: # multiplier is sp.sparse matrix of shape |reads| x |haplotypes|\n for hid in xrange(self.shape[1]):\n self.data[hid].data *= multiplier[self.data[hid].indices, hid]\n elif axis == 1: # multiplier is sp.sparse matrix of shape |reads| x |loci|\n for hid in xrange(self.shape[1]):\n self.data[hid] = self.data[hid].multiply(multiplier)\n elif axis == 2: # multiplier is np.matrix of shape |haplotypes| x |loci|\n for hid in xrange(self.shape[1]):\n multiplier_vec = multiplier[hid, :]\n multiplier_vec = multiplier_vec.ravel()\n self.data[hid].data *= multiplier_vec.repeat(np.diff(self.data[hid].indptr))\n else:\n raise RuntimeError('The axis should be 0, 1, or 2.')\n elif isinstance(multiplier, Sparse3DMatrix): # multiplier is Sparse3DMatrix object\n for hid in xrange(self.shape[1]):\n self.data[hid] = self.data[hid].multiply(multiplier.data[hid])\n else:\n raise RuntimeError('The multiplier should be 1, 2 dimensional numpy array or a Sparse3DMatrix object.')\n else:\n raise RuntimeError('The original matrix must be finalized.')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef prepare(self, pseudocount=0.0, lenfile=None, read_length=100):\n if self.probability.num_groups > 0:\n self.grp_conv_mat = lil_matrix((self.probability.num_loci, self.probability.num_groups))\n for i in xrange(self.probability.num_groups):\n self.grp_conv_mat[self.probability.groups[i], i] = 1.0\n self.grp_conv_mat = self.grp_conv_mat.tocsc()\n self.t2t_mat = eye(self.probability.num_loci, self.probability.num_loci)\n self.t2t_mat = self.t2t_mat.tolil()\n for tid_list in self.probability.groups:\n for ii in xrange(len(tid_list)):\n for jj in xrange(ii):\n i = tid_list[ii]\n j = tid_list[jj]\n self.t2t_mat[i, j] = 1\n self.t2t_mat[j, i] = 1\n self.t2t_mat = self.t2t_mat.tocsc()\n if lenfile is not None:\n hid = dict(zip(self.probability.hname, np.arange(len(self.probability.hname))))\n self.target_lengths = np.zeros((self.probability.num_loci, self.probability.num_haplotypes))\n if self.probability.num_haplotypes > 1:\n with open(lenfile) as fh:\n for curline in fh:\n item = curline.rstrip().split(\"\\t\")\n locus, hap = item[0].split(\"_\")\n self.target_lengths[self.probability.lid[locus], hid[hap]] = max(float(item[1]) - read_length + 1.0, 1.0)\n elif self.probability.num_haplotypes > 0:\n with open(lenfile) as fh:\n for curline in fh:\n item = curline.rstrip().split(\"\\t\")\n self.target_lengths[self.probability.lid[item[0]], 0] = max(float(item[1]) - read_length + 1.0, 1.0)\n else:\n raise RuntimeError('There is something wrong with your emase-format alignment file.')\n self.target_lengths = self.target_lengths.transpose()\n #self.target_lengths = self.target_lengths.transpose() / read_length # lengths in terms of read counts\n if not np.all(self.target_lengths > 0.0):\n raise RuntimeError('There exist transcripts missing length information.')\n self.probability.normalize_reads(axis=APM.Axis.READ) # Initialize alignment probability matrix\n self.allelic_expression = self.probability.sum(axis=APM.Axis.READ)\n if self.target_lengths is not None: # allelic_expression will be at depth-level\n self.allelic_expression = np.divide(self.allelic_expression, self.target_lengths)\n if pseudocount > 0.0: # pseudocount is at depth-level\n orig_allelic_expression_sum = self.allelic_expression.sum()\n nzloci = np.nonzero(self.allelic_expression)[1]\n self.allelic_expression[:, nzloci] += pseudocount\n self.allelic_expression *= (orig_allelic_expression_sum / self.allelic_expression.sum())", "response": "Prepares the probability of the allele specificity estimation based on the alignment profile."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nresetting the probability matrix and the alignment probability matrix to the original value.", "response": "def reset(self, pseudocount=0.0):\n \"\"\"\n Initializes the probability of read origin according to the alignment profile\n\n :param pseudocount: Uniform prior for allele specificity estimation\n :return: Nothing (as it performs an in-place operations)\n \"\"\"\n self.probability.reset()\n self.probability.normalize_reads(axis=APM.Axis.READ) # Initialize alignment probability matrix\n self.allelic_expression = self.probability.sum(axis=APM.Axis.READ)\n if self.target_lengths is not None: # allelic_expression will be at depth-level\n self.allelic_expression = np.divide(self.allelic_expression, self.target_lengths)\n if pseudocount > 0.0: # pseudocount is at depth-level\n orig_allelic_expression_sum = self.allelic_expression.sum()\n nzloci = np.nonzero(self.allelic_expression)[1]\n self.allelic_expression[:, nzloci] += pseudocount\n self.allelic_expression *= (orig_allelic_expression_sum / self.allelic_expression.sum())"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the probability of read origin at read level.", "response": "def update_probability_at_read_level(self, model=3):\n \"\"\"\n Updates the probability of read origin at read level\n\n :param model: Normalization model (1: Gene->Allele->Isoform, 2: Gene->Isoform->Allele, 3: Gene->Isoform*Allele, 4: Gene*Isoform*Allele)\n :return: Nothing (as it performs in-place operations)\n \"\"\"\n self.probability.reset() # reset to alignment incidence matrix\n if model == 1:\n self.probability.multiply(self.allelic_expression, axis=APM.Axis.READ)\n self.probability.normalize_reads(axis=APM.Axis.HAPLOGROUP, grouping_mat=self.t2t_mat)\n haplogroup_sum_mat = self.allelic_expression * self.t2t_mat\n self.probability.multiply(haplogroup_sum_mat, axis=APM.Axis.READ)\n self.probability.normalize_reads(axis=APM.Axis.GROUP, grouping_mat=self.t2t_mat)\n self.probability.multiply(haplogroup_sum_mat.sum(axis=0), axis=APM.Axis.HAPLOTYPE)\n self.probability.normalize_reads(axis=APM.Axis.READ)\n elif model == 2:\n self.probability.multiply(self.allelic_expression, axis=APM.Axis.READ)\n self.probability.normalize_reads(axis=APM.Axis.LOCUS)\n self.probability.multiply(self.allelic_expression.sum(axis=0), axis=APM.Axis.HAPLOTYPE)\n self.probability.normalize_reads(axis=APM.Axis.GROUP, grouping_mat=self.t2t_mat)\n self.probability.multiply((self.allelic_expression * self.t2t_mat).sum(axis=0), axis=APM.Axis.HAPLOTYPE)\n self.probability.normalize_reads(axis=APM.Axis.READ)\n elif model == 3:\n self.probability.multiply(self.allelic_expression, axis=APM.Axis.READ)\n self.probability.normalize_reads(axis=APM.Axis.GROUP, grouping_mat=self.t2t_mat)\n self.probability.multiply((self.allelic_expression * self.t2t_mat).sum(axis=0), axis=APM.Axis.HAPLOTYPE)\n self.probability.normalize_reads(axis=APM.Axis.READ)\n elif model == 4:\n self.probability.multiply(self.allelic_expression, axis=APM.Axis.READ)\n self.probability.normalize_reads(axis=APM.Axis.READ)\n else:\n raise RuntimeError('The read normalization model should be 1, 2, 3, or 4.')"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the probability at read level and then re - estimate the allelic specific expression.", "response": "def update_allelic_expression(self, model=3):\n \"\"\"\n A single EM step: Update probability at read level and then re-estimate allelic specific expression\n\n :param model: Normalization model (1: Gene->Allele->Isoform, 2: Gene->Isoform->Allele, 3: Gene->Isoform*Allele, 4: Gene*Isoform*Allele)\n :return: Nothing (as it performs in-place operations)\n \"\"\"\n self.update_probability_at_read_level(model)\n self.allelic_expression = self.probability.sum(axis=APM.Axis.READ)\n if self.target_lengths is not None:\n self.allelic_expression = np.divide(self.allelic_expression, self.target_lengths)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self, model, tol=0.001, max_iters=999, verbose=True):\n orig_err_states = np.seterr(all='raise')\n np.seterr(under='ignore')\n if verbose:\n print\n print \"Iter No Time (hh:mm:ss) Total change (TPM) \"\n print \"------- --------------- ----------------------\"\n num_iters = 0\n err_sum = 1000000.0\n time0 = time.time()\n target_err = 1000000.0 * tol\n while err_sum > target_err and num_iters < max_iters:\n prev_isoform_expression = self.get_allelic_expression().sum(axis=0)\n prev_isoform_expression *= (1000000.0 / prev_isoform_expression.sum())\n self.update_allelic_expression(model=model)\n curr_isoform_expression = self.get_allelic_expression().sum(axis=0)\n curr_isoform_expression *= (1000000.0 / curr_isoform_expression.sum())\n err = np.abs(curr_isoform_expression - prev_isoform_expression)\n err_sum = err.sum()\n num_iters += 1\n if verbose:\n time1 = time.time()\n delmin, s = divmod(int(time1 - time0), 60)\n h, m = divmod(delmin, 60)\n print \" %5d %4d:%02d:%02d %9.1f / 1000000\" % (num_iters, h, m, s, err_sum)", "response": "Runs EM iterations on the current state of the hierarchy."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef report_read_counts(self, filename, grp_wise=False, reorder='as-is', notes=None):\n expected_read_counts = self.probability.sum(axis=APM.Axis.READ)\n if grp_wise:\n lname = self.probability.gname\n expected_read_counts = expected_read_counts * self.grp_conv_mat\n else:\n lname = self.probability.lname\n total_read_counts = expected_read_counts.sum(axis=0)\n if reorder == 'decreasing':\n report_order = np.argsort(total_read_counts.flatten())\n report_order = report_order[::-1]\n elif reorder == 'increasing':\n report_order = np.argsort(total_read_counts.flatten())\n elif reorder == 'as-is':\n report_order = np.arange(len(lname)) # report in the original locus order\n cntdata = np.vstack((expected_read_counts, total_read_counts))\n fhout = open(filename, 'w')\n fhout.write(\"locus\\t\" + \"\\t\".join(self.probability.hname) + \"\\ttotal\")\n if notes is not None:\n fhout.write(\"\\tnotes\")\n fhout.write(\"\\n\")\n for locus_id in report_order:\n lname_cur = lname[locus_id]\n fhout.write(\"\\t\".join([lname_cur] + map(str, cntdata[:, locus_id].ravel())))\n if notes is not None:\n fhout.write(\"\\t%s\" % notes[lname_cur])\n fhout.write(\"\\n\")\n fhout.close()", "response": "Writes the expected read counts to a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting the expected depths of the current object to a file.", "response": "def report_depths(self, filename, tpm=True, grp_wise=False, reorder='as-is', notes=None):\n \"\"\"\n Exports expected depths\n\n :param filename: File name for output\n :param grp_wise: whether the report is at isoform level or gene level\n :param reorder: whether the report should be either 'decreasing' or 'increasing' order or just 'as-is'\n :return: Nothing but the method writes a file\n \"\"\"\n if grp_wise:\n lname = self.probability.gname\n depths = self.allelic_expression * self.grp_conv_mat\n else:\n lname = self.probability.lname\n depths = self.allelic_expression\n if tpm:\n depths *= (1000000.0 / depths.sum())\n total_depths = depths.sum(axis=0)\n if reorder == 'decreasing':\n report_order = np.argsort(total_depths.flatten())\n report_order = report_order[::-1]\n elif reorder == 'increasing':\n report_order = np.argsort(total_depths.flatten())\n elif reorder == 'as-is':\n report_order = np.arange(len(lname)) # report in the original locus order\n cntdata = np.vstack((depths, total_depths))\n fhout = open(filename, 'w')\n fhout.write(\"locus\\t\" + \"\\t\".join(self.probability.hname) + \"\\ttotal\")\n if notes is not None:\n fhout.write(\"\\tnotes\")\n fhout.write(\"\\n\")\n for locus_id in report_order:\n lname_cur = lname[locus_id]\n fhout.write(\"\\t\".join([lname_cur] + map(str, cntdata[:, locus_id].ravel())))\n if notes is not None:\n fhout.write(\"\\t%s\" % notes[lname_cur])\n fhout.write(\"\\n\")\n fhout.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite the posterior probability of read origin to file.", "response": "def export_posterior_probability(self, filename, title=\"Posterior Probability\"):\n \"\"\"\n Writes the posterior probability of read origin\n\n :param filename: File name for output\n :param title: The title of the posterior probability matrix\n :return: Nothing but the method writes a file in EMASE format (PyTables)\n \"\"\"\n self.probability.save(h5file=filename, title=title)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a copy of the original matrix with the same number of loci and group information.", "response": "def bundle(self, reset=False, shallow=False): # Copies the original matrix (Use lots of memory)\r\n \"\"\"\r\n Returns ``AlignmentPropertyMatrix`` object in which loci are bundled using grouping information.\r\n\r\n :param reset: whether to reset the values at the loci\r\n :param shallow: whether to copy all the meta data\r\n \"\"\"\r\n if self.finalized:\r\n # if self.num_groups > 0:\r\n if self.groups is not None and self.gname is not None:\r\n grp_conv_mat = lil_matrix((self.num_loci, self.num_groups))\r\n for i in xrange(self.num_groups):\r\n grp_conv_mat[self.groups[i], i] = 1.0\r\n grp_align = Sparse3DMatrix.__mul__(self, grp_conv_mat) # The core of the bundling\r\n grp_align.num_loci = self.num_groups\r\n grp_align.num_haplotypes = self.num_haplotypes\r\n grp_align.num_reads = self.num_reads\r\n grp_align.shape = (grp_align.num_loci, grp_align.num_haplotypes, grp_align.num_reads)\r\n if not shallow:\r\n grp_align.lname = copy.copy(self.gname)\r\n grp_align.hname = self.hname\r\n grp_align.rname = copy.copy(self.rname)\r\n grp_align.lid = dict(zip(grp_align.lname, np.arange(grp_align.num_loci)))\r\n grp_align.rid = copy.copy(self.rid)\r\n if reset:\r\n grp_align.reset()\r\n return grp_align\r\n else:\r\n raise RuntimeError('No group information is available for bundling.')\r\n else:\r\n raise RuntimeError('The matrix is not finalized.')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef normalize_reads(self, axis, grouping_mat=None):\r\n if self.finalized:\r\n if axis == self.Axis.LOCUS: # Locus-wise normalization on each read\r\n normalizer = self.sum(axis=self.Axis.HAPLOTYPE) # Sparse matrix of |reads| x |loci|\r\n normalizer.eliminate_zeros()\r\n for hid in xrange(self.num_haplotypes):\r\n self.data[hid].eliminate_zeros() # Trying to avoid numerical problem (inf or nan)\r\n self.data[hid] = np.divide(self.data[hid], normalizer) # element-wise division\r\n elif axis == self.Axis.HAPLOTYPE: # haplotype-wise normalization on each read\r\n for hid in xrange(self.num_haplotypes):\r\n normalizer = self.data[hid].sum(axis=self.Axis.HAPLOTYPE) # 1-dim Sparse matrix of |reads| x 1\r\n normalizer = normalizer.A.flatten()\r\n self.data[hid].data /= normalizer[self.data[hid].indices]\r\n elif axis == self.Axis.READ: # normalization each read as a whole\r\n sum_mat = self.sum(axis=self.Axis.LOCUS)\r\n normalizer = sum_mat.sum(axis=self.Axis.HAPLOTYPE)\r\n normalizer = normalizer.ravel()\r\n for hid in xrange(self.num_haplotypes):\r\n self.data[hid].data /= normalizer[self.data[hid].indices]\r\n elif axis == self.Axis.GROUP: # group-wise normalization on each read\r\n if grouping_mat is None:\r\n raise RuntimeError('Group information matrix is missing.')\r\n normalizer = self.sum(axis=self.Axis.HAPLOTYPE) * grouping_mat\r\n for hid in xrange(self.num_haplotypes):\r\n self.data[hid].eliminate_zeros() # Trying to avoid numerical problem (inf or nan)\r\n self.data[hid] = np.divide(self.data[hid], normalizer)\r\n elif axis == self.Axis.HAPLOGROUP: # haplotype-wise & group-wise normalization on each read\r\n if grouping_mat is None:\r\n raise RuntimeError('Group information matrix is missing.')\r\n for hid in xrange(self.num_haplotypes): # normalizer is different hap-by-hap\r\n normalizer = self.data[hid] * grouping_mat # Sparse matrix of |reads| x |loci|\r\n self.data[hid].eliminate_zeros() # Trying to avoid numerical problem (inf or nan)\r\n self.data[hid] = np.divide(self.data[hid], normalizer)\r\n else:\r\n raise RuntimeError('The axis should be 0, 1, 2, or 3.')\r\n else:\r\n raise RuntimeError('The original matrix must be finalized.')", "response": "This method normalizes the reads in the given axis."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pull_alignments_from(self, reads_to_use, shallow=False):\r\n new_alnmat = self.copy(shallow=shallow)\r\n for hid in xrange(self.num_haplotypes):\r\n hdata = new_alnmat.data[hid]\r\n hdata.data *= reads_to_use[hdata.indices]\r\n hdata.eliminate_zeros()\r\n if new_alnmat.count is not None:\r\n new_alnmat.count[np.logical_not(reads_to_use)] = 0\r\n return new_alnmat", "response": "Returns a new AlignmentPropertyMatrix object with only the alignments that are in the given set of reads."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new AlignmentPropertyMatrix object that contains unique reads in the specified region.", "response": "def get_unique_reads(self, ignore_haplotype=False, shallow=False):\r\n \"\"\"\r\n Pull out alignments of uniquely-aligning reads\r\n\r\n :param ignore_haplotype: whether to regard allelic multiread as uniquely-aligning read\r\n :param shallow: whether to copy sparse 3D matrix only or not\r\n :return: a new AlignmentPropertyMatrix object that particular reads are\r\n \"\"\"\r\n if self.finalized:\r\n if ignore_haplotype:\r\n summat = self.sum(axis=self.Axis.HAPLOTYPE)\r\n nnz_per_read = np.diff(summat.tocsr().indptr)\r\n unique_reads = np.logical_and(nnz_per_read > 0, nnz_per_read < 2)\r\n else: # allelic multireads should be removed\r\n alncnt_per_read = self.sum(axis=self.Axis.LOCUS).sum(axis=self.Axis.HAPLOTYPE)\r\n unique_reads = np.logical_and(alncnt_per_read > 0, alncnt_per_read < 2)\r\n return self.pull_alignments_from(unique_reads, shallow=shallow)\r\n else:\r\n raise RuntimeError('The matrix is not finalized.')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef print_read(self, rid):\r\n if self.rname is not None:\r\n print self.rname[rid]\r\n print '--'\r\n r = self.get_read_data(rid)\r\n aligned_loci = np.unique(r.nonzero()[1])\r\n for locus in aligned_loci:\r\n nzvec = r[:, locus].todense().transpose()[0].A.flatten()\r\n if self.lname is not None:\r\n print self.lname[locus],\r\n else:\r\n print locus,\r\n print nzvec", "response": "Print the contents of a read."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_standard_form(self, data):\n if self.synonym_map is None:\n return data\n from indic_transliteration import sanscript\n return sanscript.transliterate(data=sanscript.transliterate(_from=self.name, _to=sanscript.DEVANAGARI, data=data), _from=sanscript.DEVANAGARI, _to=self.name)", "response": "This method transliterate the given data into a standard representation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _roman(data, scheme_map, **kw):\n vowels = scheme_map.vowels\n marks = scheme_map.marks\n virama = scheme_map.virama\n consonants = scheme_map.consonants\n non_marks_viraama = scheme_map.non_marks_viraama\n max_key_length_from_scheme = scheme_map.max_key_length_from_scheme\n to_roman = scheme_map.to_scheme.is_roman\n\n togglers = kw.pop('togglers', set())\n suspend_on = kw.pop('suspend_on', set())\n suspend_off = kw.pop('suspend_off', set())\n if kw:\n raise TypeError('Unexpected keyword argument %s' % list(kw.keys())[0])\n\n buf = []\n i = 0\n had_consonant = found = False\n len_data = len(data)\n append = buf.append\n\n # If true, don't transliterate. The toggle token is discarded.\n toggled = False\n # If true, don't transliterate. The suspend token is retained.\n # `suspended` overrides `toggled`.\n suspended = False\n\n while i <= len_data:\n # The longest token in the source scheme has length `max_key_length_from_scheme`. Iterate\n # over `data` while taking `max_key_length_from_scheme` characters at a time. If we don`t\n # find the character group in our scheme map, lop off a character and\n # try again.\n #\n # If we've finished reading through `data`, then `token` will be empty\n # and the loop below will be skipped.\n token = data[i:i + max_key_length_from_scheme]\n\n while token:\n if token in togglers:\n toggled = not toggled\n i += 2 # skip over the token\n found = True # force the token to fill up again\n break\n\n if token in suspend_on:\n suspended = True\n elif token in suspend_off:\n suspended = False\n\n if toggled or suspended:\n token = token[:-1]\n continue\n\n # Catch the pattern CV, where C is a consonant and V is a vowel.\n # V should be rendered as a vowel mark, a.k.a. a \"dependent\"\n # vowel. But due to the nature of Brahmic scripts, 'a' is implicit\n # and has no vowel mark. If we see 'a', add nothing.\n if had_consonant and token in vowels:\n mark = marks.get(token, '')\n if mark:\n append(mark)\n elif to_roman:\n append(vowels[token])\n found = True\n\n # Catch any non_marks_viraama character, including consonants, punctuation,\n # and regular vowels. Due to the implicit 'a', we must explicitly\n # end any lingering consonants before we can handle the current\n # token.\n elif token in non_marks_viraama:\n if had_consonant:\n append(virama[''])\n append(non_marks_viraama[token])\n found = True\n\n if found:\n had_consonant = token in consonants\n i += len(token)\n break\n else:\n token = token[:-1]\n\n # We've exhausted the token; this must be some other character. Due to\n # the implicit 'a', we must explicitly end any lingering consonants\n # before we can handle the current token.\n if not found:\n if had_consonant:\n append(virama[''])\n if i < len_data:\n append(data[i])\n had_consonant = False\n i += 1\n\n found = False\n\n return ''.join(buf)", "response": "Transliterate the given data with the given scheme_map."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving some devanAgarI sanskrit text, this function produces a \"key\" so that 1] The key should be the same for different observed orthographical forms of the same text. For example: :: - \"dharmma\" vs \"dharma\" - \"rAmaM gacChati\" vs \"rAma~N gacChati\" vs \"rAma~N gacChati\" - \"kurvan eva\" vs \"kurvanneva\" 2] The key should be different for different for different texts. - \"stamba\" vs \"stambha\" This function attempts to succeed at [1] and [2] almostall the time. Longer the text, probability of failing at [2] decreases, while probability of failing at [1] increases (albeit very slightly). Sources of orthographically divergent forms: - Phonetically sensible grammar rules - Neglect of sandhi while writing - Punctuation, spaces, avagraha-s. - Regional-language-influenced mistakes (La instead of la.) Some example applications of this function: - Create a database of quotes or words with minimal duplication. - Search a database of quotes or words while being robust to optional forms. Also see equivalent function in the scala indic-transliteration package.", "response": "def get_approx_deduplicating_key(text, encoding_scheme=sanscript.DEVANAGARI):\n \"\"\"\n Given some devanAgarI sanskrit text, this function produces a \"key\" so\n that\n \n 1] The key should be the same for different observed orthographical\n forms of the same text. For example:\n \n ::\n \n - \"dharmma\" vs \"dharma\"\n - \"rAmaM gacChati\" vs \"rAma~N gacChati\" vs \"rAma~N gacChati\"\n - \"kurvan eva\" vs \"kurvanneva\"\n \n 2] The key should be different for different for different texts.\n \n - \"stamba\" vs \"stambha\"\n \n This function attempts to succeed at [1] and [2] almostall the time.\n Longer the text, probability of failing at [2] decreases, while\n probability of failing at [1] increases (albeit very slightly).\n \n Sources of orthographically divergent forms:\n \n - Phonetically sensible grammar rules\n - Neglect of sandhi while writing\n - Punctuation, spaces, avagraha-s.\n - Regional-language-influenced mistakes (La instead of la.)\n \n Some example applications of this function:\n \n - Create a database of quotes or words with minimal duplication.\n - Search a database of quotes or words while being robust to optional\n forms.\n \n Also see equivalent function in the scala indic-transliteration package.\n \"\"\"\n if encoding_scheme == sanscript.DEVANAGARI:\n key = text\n key = regex.sub(\"\\\\P{IsDevanagari}\", \"\", key)\n # Remove spaces\n key = regex.sub(\"\\\\s\", \"\", key)\n # Remove punctuations\n key = regex.sub(\"\\\\p{P}\", \"\", key)\n # Remove digits, abbreviation sign, svara-s, avagraha\n key = regex.sub(\"[\u0966-\u096f\u0964\u0965\u0970\u093d]|[\u0951-\u0954]\", \"\", key)\n # Collapse semi-vowel-anunAsika-s \u0938\u0902\u0932\u0917\u094d\u0928\u092e\u094d \u0938\u0932\u094d\u0901\u0932\u0917\u094d\u0928\u092e\u094d into m\n key = regex.sub(\"[\u092f\u0930\u0932]\u094d\u0901\", \"\u092e\u094d\", key)\n # Collapse all panchama-s into m\n key = regex.sub(\"[\u0919\u091e\u0923\u0928]\", \"\u092e\", key)\n # Collapse anusvAra into m\n key = regex.sub(\"\u0901|\u0902\", \"\u092e\u094d\", key)\n key = regex.sub(\"\u0950\", \"\u0913\u092e\u094d\", key)\n key = regex.sub(\"[\u0933\u0934]\", \"\u0932\", key)\n # Deal with optional forms where consonants are duplicated - like dharmma\n # Details in https://docs.google.com/spreadsheets/d/1GP8Ps_hmgCGLZPWKIVBCfQB9ZmPQOaCwTrH9OybaWaQ/edit#gid=21\n key = regex.sub(\"([\u0915-\u0939\u0958-\u095f])\u094d\\\\1+\", \"\\\\1\", key)\n key = regex.sub(\"[\u0915\u0917]\u094d\u0916\u094d\", \"\u0916\u094d\", key)\n key = regex.sub(\"[\u0915\u0917]\u094d\u0918\u094d\", \"\u0918\u094d\", key)\n key = regex.sub(\"\u091a\u094d\u091b\u094d\", \"\u091b\u094d\", key)\n key = regex.sub(\"\u091c\u094d\u091d\u094d\", \"\u091d\u094d\", key)\n key = regex.sub(\"\u0924\u094d\u0925\u094d\", \"\u0925\u094d\", key)\n key = regex.sub(\"\u0926\u094d\u0927\u094d\", \"\u0927\u094d\", key)\n key = regex.sub(\"\u0921\u094d\u0922\u094d\", \"\u0922\u094d\", key)\n key = regex.sub(\"\u092a\u094d\u092b\u094d\", \"\u092b\u094d\", key)\n key = regex.sub(\"\u092c\u094d\u092d\u094d\", \"\u092d\u094d\", key)\n return key\n else:\n logging.warning(\"got script {} for '{}'\".format(encoding_scheme, text))\n return regex.sub(\"\\\\s\", \"\", text)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntransliterate the given data with the given scheme_map.", "response": "def _brahmic(data, scheme_map, **kw):\n \"\"\"Transliterate `data` with the given `scheme_map`. This function is used\n when the source scheme is a Brahmic scheme.\n\n :param data: the data to transliterate\n :param scheme_map: a dict that maps between characters in the old scheme\n and characters in the new scheme\n \"\"\"\n if scheme_map.from_scheme.name == northern.GURMUKHI:\n data = northern.GurmukhiScheme.replace_tippi(text=data) \n marks = scheme_map.marks\n virama = scheme_map.virama\n consonants = scheme_map.consonants\n non_marks_viraama = scheme_map.non_marks_viraama\n to_roman = scheme_map.to_scheme.is_roman\n max_key_length_from_scheme = scheme_map.max_key_length_from_scheme\n\n buf = []\n i = 0\n to_roman_had_consonant = found = False\n append = buf.append\n # logging.debug(pprint.pformat(scheme_map.consonants))\n\n # We dont just translate each brAhmic character one after another in order to prefer concise transliterations when possible - for example \u091c\u094d\u091e -> jn in optitrans rather than j~n.\n while i <= len(data):\n # The longest token in the source scheme has length `max_key_length_from_scheme`. Iterate\n # over `data` while taking `max_key_length_from_scheme` characters at a time. If we don`t\n # find the character group in our scheme map, lop off a character and\n # try again.\n #\n # If we've finished reading through `data`, then `token` will be empty\n # and the loop below will be skipped.\n token = data[i:i + max_key_length_from_scheme]\n\n while token:\n if len(token) == 1:\n if token in marks:\n append(marks[token])\n found = True\n elif token in virama:\n append(virama[token])\n found = True\n else:\n if to_roman_had_consonant:\n append('a')\n append(non_marks_viraama.get(token, token))\n found = True\n else:\n if token in non_marks_viraama:\n if to_roman_had_consonant:\n append('a')\n append(non_marks_viraama.get(token))\n found = True\n\n if found:\n to_roman_had_consonant = to_roman and token in consonants\n i += len(token)\n break \n else:\n token = token[:-1]\n\n # Continuing the outer while loop.\n # We've exhausted the token; this must be some other character. Due to\n # the implicit 'a', we must explicitly end any lingering consonants\n # before we can handle the current token.\n if not found:\n if to_roman_had_consonant:\n append(next(iter(virama.values())))\n if i < len(data):\n append(data[i])\n to_roman_had_consonant = False\n i += 1\n\n found = False\n\n if to_roman_had_consonant:\n append('a')\n return ''.join(buf)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntransliterate data with the given parameters.", "response": "def transliterate(data, _from=None, _to=None, scheme_map=None, **kw):\n \"\"\"Transliterate `data` with the given parameters::\n\n output = transliterate('idam adbhutam', HK, DEVANAGARI)\n\n Each time the function is called, a new :class:`SchemeMap` is created\n to map the input scheme to the output scheme. This operation is fast\n enough for most use cases. But for higher performance, you can pass a\n pre-computed :class:`SchemeMap` instead::\n\n scheme_map = SchemeMap(SCHEMES[HK], SCHEMES[DEVANAGARI])\n output = transliterate('idam adbhutam', scheme_map=scheme_map)\n\n :param data: the data to transliterate\n :param scheme_map: the :class:`SchemeMap` to use. If specified, ignore\n `_from` and `_to`. If unspecified, create a\n :class:`SchemeMap` from `_from` to `_to`.\n \"\"\"\n if scheme_map is None:\n scheme_map = _get_scheme_map(_from, _to)\n options = {\n 'togglers': {'##'},\n 'suspend_on': set('<'),\n 'suspend_off': set('>')\n }\n options.update(kw)\n\n from indic_transliteration.sanscript.brahmic_mapper import _brahmic\n from indic_transliteration.sanscript.roman_mapper import _roman\n func = _roman if scheme_map.from_scheme.is_roman else _brahmic\n return func(data, scheme_map, **options)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef detect(text):\n if sys.version_info < (3, 0):\n # Verify encoding\n try:\n text = text.decode('utf-8')\n except UnicodeError:\n pass\n\n # Brahmic schemes are all within a specific range of code points.\n for L in text:\n code = ord(L)\n if code >= BRAHMIC_FIRST_CODE_POINT:\n for name, start_code in BLOCKS:\n if start_code <= code <= BRAHMIC_LAST_CODE_POINT:\n return name\n\n # Romanizations\n if Regex.IAST_OR_KOLKATA_ONLY.search(text):\n if Regex.KOLKATA_ONLY.search(text):\n return Scheme.Kolkata\n else:\n return Scheme.IAST\n\n if Regex.ITRANS_ONLY.search(text):\n return Scheme.ITRANS\n\n if Regex.SLP1_ONLY.search(text):\n return Scheme.SLP1\n\n if Regex.VELTHUIS_ONLY.search(text):\n return Scheme.Velthuis\n\n if Regex.ITRANS_OR_VELTHUIS_ONLY.search(text):\n return Scheme.ITRANS\n\n return Scheme.HK", "response": "Detects the transliteration scheme of the input."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntransliterate data with the given parameters.", "response": "def transliterate(data, _from=None, _to=None, scheme_map=None, **kw):\r\n \"\"\"Transliterate `data` with the given parameters::\r\n\r\n output = transliterate('idam adbhutam', HK, DEVANAGARI)\r\n\r\n Each time the function is called, a new :class:`SchemeMap` is created\r\n to map the input scheme to the output scheme. This operation is fast\r\n enough for most use cases. But for higher performance, you can pass a\r\n pre-computed :class:`SchemeMap` instead::\r\n\r\n scheme_map = SchemeMap(SCHEMES[HK], SCHEMES[DEVANAGARI])\r\n output = transliterate('idam adbhutam', scheme_map=scheme_map)\r\n\r\n :param data: the data to transliterate\r\n :param _from: the name of a source scheme\r\n :param _to: the name of a destination scheme\r\n :param scheme_map: the :class:`SchemeMap` to use. If specified, ignore\r\n `_from` and `_to`. If unspecified, create a\r\n :class:`SchemeMap` from `_from` to `_to`.\r\n \"\"\"\r\n if scheme_map is None:\r\n from_scheme = SCHEMES[_from]\r\n to_scheme = SCHEMES[_to]\r\n scheme_map = sanscript.SchemeMap(from_scheme, to_scheme)\r\n return sanscript.transliterate(data=data, scheme_map=scheme_map)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _setup():\r\n s = str.split\r\n if sys.version_info < (3, 0):\r\n # noinspection PyUnresolvedReferences\r\n s = unicode.split\r\n\r\n def pop_all(some_dict, some_list):\r\n for scheme in some_list:\r\n some_dict.pop(scheme)\r\n global SCHEMES\r\n SCHEMES = copy.deepcopy(sanscript.SCHEMES)\r\n pop_all(SCHEMES, [sanscript.ORIYA, sanscript.BENGALI, sanscript.GUJARATI])\r\n SCHEMES[HK].update({\r\n 'vowels': s(\"\"\"a A i I u U R RR lR lRR E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'marks': s(\"\"\"A i I u U R RR lR lRR E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'consonants': sanscript.SCHEMES[HK]['consonants'] + s(\"\"\"n2 r2 zh\"\"\")\r\n })\r\n SCHEMES[ITRANS].update({\r\n 'vowels': s(\"\"\"a A i I u U R RR LLi LLI E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'marks': s(\"\"\"A i I u U R RR LLi LLI E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'consonants': sanscript.SCHEMES[ITRANS]['consonants'] + s(\"\"\"n2 r2 zh\"\"\")\r\n })\r\n pop_all(SCHEMES[ITRANS].synonym_map, s(\"\"\"e o\"\"\"))\r\n SCHEMES[OPTITRANS].update({\r\n 'vowels': s(\"\"\"a A i I u U R RR LLi LLI E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'marks': s(\"\"\"A i I u U R RR LLi LLI E ai O au\"\"\") + s(\"\"\"e o\"\"\"),\r\n 'consonants': sanscript.SCHEMES[OPTITRANS]['consonants'] + s(\"\"\"n2 r2 zh\"\"\")\r\n })\r\n pop_all(SCHEMES[OPTITRANS].synonym_map, s(\"\"\"e o\"\"\"))", "response": "Add a variety of default schemes."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert an array of integers to utf8 string", "response": "def to_utf8(y):\n \"\"\"\n converts an array of integers to utf8 string\n \"\"\"\n\n out = []\n\n for x in y:\n\n if x < 0x080:\n out.append(x)\n elif x < 0x0800:\n out.append((x >> 6) | 0xC0)\n out.append((x & 0x3F) | 0x80)\n elif x < 0x10000:\n out.append((x >> 12) | 0xE0)\n out.append(((x >> 6) & 0x3F) | 0x80)\n out.append((x & 0x3F) | 0x80)\n else:\n out.append((x >> 18) | 0xF0)\n out.append((x >> 12) & 0x3F)\n out.append(((x >> 6) & 0x3F) | 0x80)\n out.append((x & 0x3F) | 0x80)\n\n return ''.join(map(chr, out))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the value of delta to reflect the current codepage s script", "response": "def set_script(self, i):\n \"\"\"\n set the value of delta to reflect the current codepage\n \n \"\"\"\n\n if i in range(1, 10):\n n = i - 1\n else:\n raise IllegalInput(\"Invalid Value for ATR %s\" % (hex(i)))\n\n if n > -1: # n = -1 is the default script ..\n self.curr_script = n\n self.delta = n * DELTA\n \n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef main(argv=None):\n print (transliterate('jaya gaNeza! zrIrAmajayam', 'harvardkyoto', 'devanagari'))\n if argv is None:\n argv = sys.argv\n try: \n text, inFormat, outFormat = argv[1:4]\n except ValueError:\n print (main.__doc__)\n return 2\n inFormat = inFormat.upper()\n outFormat = outFormat.upper()\n # try assuming \"text\" is a filename\n try:\n f = open(text)\n except IOError:\n # it wasn't, so it must be the actual text\n print (transliterate(text, inFormat, outFormat))\n return 0\n else:\n i = 1\n for text in f.readlines():\n if len(text) > 0 and not text.startswith('#'):\n print (transliterate(text, inFormat, outFormat).strip('\\n'))\n i = i + 1\n f.close()\n return 0", "response": "This function is the main function for transliterator. py."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntransliterate a text into the target format.", "response": "def _transliterate(self, text, outFormat):\n \"\"\" Transliterate a devanagari text into the target format.\n \n Transliterating a character to or from Devanagari is not a simple \n lookup: it depends on the preceding and following characters.\n \"\"\"\n def getResult(): \n if curMatch.isspace():\n result.append(curMatch)\n return\n if prevMatch in self:\n prev = self[prevMatch]\n else:\n prev = None\n if nextMatch in self:\n next = self[nextMatch]\n else:\n next = None\n try:\n equiv = outFormat._equivalent(self[curMatch], \n prev, #self.get(prevMatch, None), \n next, #self.get(nextMatch, None),\n self._implicitA)\n except KeyError:\n equiv = _unrecognised(curMatch)\n for e in equiv:\n result.append(e)\n \n def incr(c):\n if self._longestEntry == 1:\n return 1\n return len(c)\n \n \n result = []\n text = self._preprocess(text)\n i = 0\n prevMatch = None\n nextMatch = None\n curMatch = self._getNextChar(text, i)\n i = i + len(curMatch)\n while i < len(text):\n nextMatch = self._getNextChar(text, i)\n getResult()\n i = i + len(nextMatch)\n prevMatch = curMatch\n curMatch = nextMatch\n nextMatch = None\n getResult() \n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _equivalent(self, char, prev, next, implicitA):\n result = []\n if char.isVowel == False:\n result.append(char.chr)\n if char.isConsonant \\\n and ((next is not None and next.isConsonant) \\\n or next is None): \n result.append(DevanagariCharacter._VIRAMA)\n else:\n if prev is None or prev.isConsonant == False:\n result.append(char.chr)\n else:\n if char._dependentVowel is not None:\n result.append(char._dependentVowel)\n return result", "response": "Translate a Latin character equivalent to Devanagari."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_devanagari(self, data):\n from indic_transliteration import sanscript\n return sanscript.transliterate(data=data, _from=sanscript.DEVANAGARI, _to=self.name)", "response": "A convenience method that transliterate a string from the DEVANAGARI to the name of the object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading and generate num top - level rules from the specified grammar file and write the result to the specified output stream.", "response": "def generate(grammar=None, num=1, output=sys.stdout, max_recursion=10, seed=None):\n \"\"\"Load and generate ``num`` number of top-level rules from the specified grammar.\n\n :param list grammar: The grammar file to load and generate data from\n :param int num: The number of times to generate data\n :param output: The output destination (an open, writable stream-type object. default=``sys.stdout``)\n :param int max_recursion: The maximum reference-recursion when generating data (default=``10``)\n :param int seed: The seed to initialize the PRNG with. If None, will not initialize it.\n \"\"\"\n if seed is not None:\n gramfuzz.rand.seed(seed)\n\n fuzzer = gramfuzz.GramFuzzer()\n fuzzer.load_grammar(grammar)\n\n cat_group = os.path.basename(grammar).replace(\".py\", \"\")\n\n results = fuzzer.gen(cat_group=cat_group, num=num, max_recursion=max_recursion)\n for res in results:\n output.write(res)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds the quote instance of the object.", "response": "def build(self, pre=None, shortest=False):\n \"\"\"Build the ``Quote`` instance\n\n :param list pre: The prerequisites list\n :param bool shortest: Whether or not the shortest reference-chain (most minimal) version of the field should be generated.\n \"\"\"\n res = super(Q, self).build(pre, shortest=shortest)\n\n if self.escape:\n return repr(res)\n elif self.html_js_escape:\n return (\"'\" + res.encode(\"string_escape\").replace(\"<\", \"\\\\x3c\").replace(\">\", \"\\\\x3e\") + \"'\")\n else:\n return \"\".join([self.quote, res, self.quote])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmakes the list of verbs into present participles", "response": "def make_present_participles(verbs):\n \"\"\"Make the list of verbs into present participles\n\n E.g.:\n\n empower -> empowering\n drive -> driving\n \"\"\"\n res = []\n for verb in verbs:\n parts = verb.split()\n if parts[0].endswith(\"e\"):\n parts[0] = parts[0][:-1] + \"ing\"\n else:\n parts[0] = parts[0] + \"ing\"\n res.append(\" \".join(parts))\n return res"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clear_sent_messages(self, offset=None):\n if offset is None:\n offset = getattr(settings, 'MAILQUEUE_CLEAR_OFFSET', defaults.MAILQUEUE_CLEAR_OFFSET)\n\n if type(offset) is int:\n offset = datetime.timedelta(hours=offset)\n\n delete_before = timezone.now() - offset\n self.filter(sent=True, last_attempt__lte=delete_before).delete()", "response": "Deletes all sent MailerMessage records"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns string with unicode values", "response": "def unicodevalues_asstring(values):\n \"\"\" Return string with unicodenames (unless that is disabled) \"\"\"\n if not os.environ.get('DISABLE_UNAMES'):\n return map(lambda x: '%s' % format(x).strip(), values)\n return map(lambda x: u'U+%04x %s' % (x, unichr(x)), sorted(values))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads the includes of an encoding Namelist file.", "response": "def _loadNamelistIncludes(item, unique_glyphs, cache):\n \"\"\"Load the includes of an encoding Namelist files.\n\n This is an implementation detail of readNamelist.\n \"\"\"\n includes = item[\"includes\"] = []\n charset = item[\"charset\"] = set() | item[\"ownCharset\"]\n\n noCharcode = item[\"noCharcode\"] = set() | item[\"ownNoCharcode\"]\n\n dirname = os.path.dirname(item[\"fileName\"])\n for include in item[\"header\"][\"includes\"]:\n includeFile = os.path.join(dirname, include)\n try:\n includedItem = readNamelist(includeFile, unique_glyphs, cache)\n except NamelistRecursionError:\n continue\n if includedItem in includes:\n continue\n includes.append(includedItem)\n charset |= includedItem[\"charset\"]\n noCharcode |= includedItem[\"ownNoCharcode\"]\n return item"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __readNamelist(cache, filename, unique_glyphs):\n if filename in cache:\n item = cache[filename]\n else:\n cps, header, noncodes = parseNamelist(filename)\n item = {\n \"fileName\": filename\n , \"ownCharset\": cps\n , \"header\": header\n , \"ownNoCharcode\": noncodes\n , \"includes\": None # placeholder\n , \"charset\": None # placeholder\n , \"noCharcode\": None\n }\n cache[filename] = item\n\n if unique_glyphs or item[\"charset\"] is not None:\n return item\n\n # full-charset/includes are requested and not cached yet\n _loadNamelistIncludes(item, unique_glyphs, cache)\n return item", "response": "Return a dict with the data of an encoding Namelist file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndetecting infinite recursion and prevent it. This is an implementation detail of readNamelist. Raises NamelistRecursionError if namFilename is in the process of being included", "response": "def _readNamelist(currentlyIncluding, cache, namFilename, unique_glyphs):\n \"\"\" Detect infinite recursion and prevent it.\n\n This is an implementation detail of readNamelist.\n\n Raises NamelistRecursionError if namFilename is in the process of being included\n \"\"\"\n # normalize\n filename = os.path.abspath(os.path.normcase(namFilename))\n if filename in currentlyIncluding:\n raise NamelistRecursionError(filename)\n currentlyIncluding.add(filename)\n try:\n result = __readNamelist(cache, filename, unique_glyphs)\n finally:\n currentlyIncluding.remove(filename)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef readNamelist(namFilename, unique_glyphs=False, cache=None):\n currentlyIncluding = set()\n if not cache:\n cache = {}\n return _readNamelist(currentlyIncluding, cache, namFilename, unique_glyphs)", "response": "Reads a Namelist file and returns a dict of the contents."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef codepointsInNamelist(namFilename, unique_glyphs=False, cache=None):\n key = 'charset' if not unique_glyphs else 'ownCharset'\n\n internals_dir = os.path.dirname(os.path.abspath(__file__))\n target = os.path.join(internals_dir, namFilename)\n result = readNamelist(target, unique_glyphs, cache)\n return result[key]", "response": "Returns the set of codepoints contained in a given Namelist file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_orthographies(self, _library=library):\n ''' Returns list of CharsetInfo about supported orthographies '''\n results = []\n for charset in _library.charsets:\n if self._charsets:\n cn = getattr(charset, 'common_name', False)\n abbr = getattr(charset, 'abbreviation', False)\n nn = getattr(charset, 'short_name', False)\n naive = getattr(charset, 'native_name', False)\n\n if cn and cn.lower() in self._charsets:\n results.append(charset)\n\n elif nn and nn.lower() in self._charsets:\n results.append(charset)\n\n elif naive and naive.lower() in self._charsets:\n results.append(charset)\n\n elif abbr and abbr.lower() in self._charsets:\n results.append(charset)\n else:\n results.append(charset)\n\n for result in results:\n yield CharsetInfo(self, result)", "response": "Returns list of CharsetInfo about supported orthographies"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_codepoints():\n # response = requests.get(EXTENSIS_LANG_XML)\n # if response.status_code != 200:\n # return []\n\n path = get_from_cache('languages.xml', EXTENSIS_LANG_XML)\n\n try:\n xml_content = open(path, 'r').read()\n except IOError:\n logging.error('Could not read languages.xml from cache')\n xml_content = ''\n\n content = re.sub('', '', xml_content)\n doc = etree.fromstring(content.lstrip('`'))\n\n return doc.findall('.//scanning-codepoints')", "response": "Return all XML in received XML"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn list of unicode codepoints for the given codepoint.", "response": "def get_unicodes(codepoint):\n \"\"\" Return list of unicodes for \"\"\"\n result = re.sub('\\s', '', codepoint.text)\n return Extension.convert_to_list_of_unicodes(result)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nhandle the OAuth2 flow.", "response": "def handler(self,):\n \"\"\"* get request token if OAuth1\n * Get user authorization\n * Get access token\n \"\"\"\n\n if self.oauth_version == 'oauth1':\n request_token, request_token_secret = self.oauth.get_request_token(params={'oauth_callback': self.callback_uri})\n logger.debug(\"REQUEST_TOKEN = {0}\\n REQUEST_TOKEN_SECRET = {1}\\n\".format(request_token, request_token_secret))\n authorize_url = self.oauth.get_authorize_url(request_token)\n else:\n authorize_url = self.oauth.get_authorize_url(client_secret=self.consumer_secret, redirect_uri=self.callback_uri, response_type='code')\n\n logger.debug(\"AUTHORISATION URL : {0}\".format(authorize_url))\n # Open authorize_url\n webbrowser.open(authorize_url)\n self.verifier = input(\"Enter verifier : \")\n\n self.token_time = time.time()\n \n credentials = {'token_time': self.token_time}\n \n if self.oauth_version == 'oauth1':\n raw_access = self.oauth.get_raw_access_token(request_token, request_token_secret, params={\"oauth_verifier\": self.verifier})\n parsed_access = parse_utf8_qsl(raw_access.content)\n self.access_token = parsed_access['oauth_token']\n self.access_token_secret = parsed_access['oauth_token_secret']\n self.session_handle = parsed_access['oauth_session_handle']\n self.guid = parsed_access['xoauth_yahoo_guid']\n \n # Updating credentials \n credentials.update({\n 'access_token': self.access_token,\n 'access_token_secret': self.access_token_secret,\n 'session_handle': self.session_handle,\n 'guid': self.guid\n })\n else:\n # Building headers \n headers = self.generate_oauth2_headers()\n # Getting access token\n raw_access = self.oauth.get_raw_access_token(data={\"code\": self.verifier, 'redirect_uri': self.callback_uri,'grant_type':'authorization_code'}, headers=headers)\n #parsed_access = parse_utf8_qsl(raw_access.content.decode('utf-8'))\n credentials.update(self.oauth2_access_parser(raw_access))\n \n return credentials"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_oauth2_headers(self):\n encoded_credentials = base64.b64encode(('{0}:{1}'.format(self.consumer_key,self.consumer_secret)).encode('utf-8'))\n headers={\n 'Authorization':'Basic {0}'.format(encoded_credentials.decode('utf-8')),\n 'Content-Type': 'application/x-www-form-urlencoded'\n }\n\n return headers", "response": "Generates header for oauth2"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef oauth2_access_parser(self, raw_access):\n parsed_access = json.loads(raw_access.content.decode('utf-8'))\n self.access_token = parsed_access['access_token']\n self.token_type = parsed_access['token_type']\n self.refresh_token = parsed_access['refresh_token']\n self.guid = parsed_access['xoauth_yahoo_guid']\n\n credentials = {\n 'access_token': self.access_token,\n 'token_type': self.token_type,\n 'refresh_token': self.refresh_token,\n 'guid': self.guid\n }\n \n return credentials", "response": "Parse the oauth2 access token and return a dictionary of credentials."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef refresh_access_token(self,):\n logger.debug(\"REFRESHING TOKEN\")\n self.token_time = time.time()\n credentials = {\n 'token_time': self.token_time\n }\n\n if self.oauth_version == 'oauth1':\n self.access_token, self.access_token_secret = self.oauth.get_access_token(self.access_token, self.access_token_secret, params={\"oauth_session_handle\": self.session_handle})\n credentials.update({\n 'access_token': self.access_token,\n 'access_token_secret': self.access_token_secret,\n 'session_handle': self.session_handle,\n 'token_time': self.token_time\n })\n else:\n headers = self.generate_oauth2_headers()\n\n raw_access = self.oauth.get_raw_access_token(data={\"refresh_token\": self.refresh_token, 'redirect_uri': self.callback_uri,'grant_type':'refresh_token'}, headers=headers)\n credentials.update(self.oauth2_access_parser(raw_access)) \n\n return credentials", "response": "Refresh the access token and return a dictionary with the new token and the new token_time."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking the validity of the token", "response": "def token_is_valid(self,):\n \"\"\"Check the validity of the token :3600s\n \"\"\"\n elapsed_time = time.time() - self.token_time\n logger.debug(\"ELAPSED TIME : {0}\".format(elapsed_time))\n if elapsed_time > 3540: # 1 minute before it expires\n logger.debug(\"TOKEN HAS EXPIRED\")\n return False\n\n logger.debug(\"TOKEN IS STILL VALID\")\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling right function according to file extension", "response": "def get_data(filename):\n \"\"\"Calls right function according to file extension\n \"\"\"\n name, ext = get_file_extension(filename)\n func = json_get_data if ext == '.json' else yaml_get_data\n return func(filename)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef write_data(data, filename):\n name, ext = get_file_extension(filename)\n func = json_write_data if ext == '.json' else yaml_write_data\n return func(data, filename)", "response": "Call right func to save data according to file extension\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef json_write_data(json_data, filename):\n with open(filename, 'w') as fp:\n json.dump(json_data, fp, indent=4, sort_keys=True, ensure_ascii=False)\n return True\n return False", "response": "Write json data into a file\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef json_get_data(filename):\n with open(filename) as fp:\n json_data = json.load(fp)\n return json_data\n\n return False", "response": "Get data from json file\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef yaml_get_data(filename):\n with open(filename, 'rb') as fd:\n yaml_data = yaml.load(fd)\n return yaml_data\n return False", "response": "Get data from. yml file\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting data into a. yml file", "response": "def yaml_write_data(yaml_data, filename):\n \"\"\"Write data into a .yml file\n \"\"\"\n with open(filename, 'w') as fd:\n yaml.dump(yaml_data, fd, default_flow_style=False)\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfit the class to the raw pairwise distances.", "response": "def fit(self, X, y=None):\n '''\n If scale_by_median, find :attr:`median_`; otherwise, do nothing.\n\n Parameters\n ----------\n X : array\n The raw pairwise distances.\n '''\n\n X = check_array(X)\n if self.scale_by_median:\n self.median_ = np.median(X[np.triu_indices_from(X, k=1)],\n overwrite_input=True)\n elif hasattr(self, 'median_'):\n del self.median_\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform(self, X):\n '''\n Turns distances into RBF values.\n\n Parameters\n ----------\n X : array\n The raw pairwise distances.\n\n Returns\n -------\n X_rbf : array of same shape as X\n The distances in X passed through the RBF kernel.\n '''\n X = check_array(X)\n X_rbf = np.empty_like(X) if self.copy else X\n\n X_in = X\n if not self.squared:\n np.power(X_in, 2, out=X_rbf)\n X_in = X_rbf\n\n if self.scale_by_median:\n scale = self.median_ if self.squared else self.median_ ** 2\n gamma = self.gamma * scale\n else:\n gamma = self.gamma\n np.multiply(X_in, -gamma, out=X_rbf)\n\n np.exp(X_rbf, out=X_rbf)\n return X_rbf", "response": "Turns distances into RBF values."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, X, y=None):\n '''\n Learn the linear transformation to clipped eigenvalues.\n\n Note that if min_eig isn't zero and any of the original eigenvalues\n were exactly zero, this will leave those eigenvalues as zero.\n\n Parameters\n ----------\n X : array, shape [n, n]\n The *symmetric* input similarities. If X is asymmetric, it will be\n treated as if it were symmetric based on its lower-triangular part.\n '''\n n = X.shape[0]\n if X.shape != (n, n):\n raise TypeError(\"Input must be a square matrix.\")\n\n # TODO: only get negative eigs somehow?\n memory = get_memory(self.memory)\n vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(\n X, overwrite_a=not self.copy)\n vals = vals.reshape(-1, 1)\n\n if self.min_eig == 0:\n inner = vals > self.min_eig\n else:\n with np.errstate(divide='ignore'):\n inner = np.where(vals >= self.min_eig, 1,\n np.where(vals == 0, 0, self.min_eig / vals))\n\n self.clip_ = np.dot(vecs, inner * vecs.T)\n return self", "response": "Fit the Linear transformation to clipped eigenvalues."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fit(self, X, y=None):\n '''\n Learn the linear transformation to flipped eigenvalues.\n\n Parameters\n ----------\n X : array, shape [n, n]\n The *symmetric* input similarities. If X is asymmetric, it will be\n treated as if it were symmetric based on its lower-triangular part.\n '''\n n = X.shape[0]\n if X.shape != (n, n):\n raise TypeError(\"Input must be a square matrix.\")\n\n # TODO: only get negative eigs somehow?\n memory = get_memory(self.memory)\n vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(\n X, overwrite_a=not self.copy)\n vals = vals[:, None]\n\n self.flip_ = np.dot(vecs, np.sign(vals) * vecs.T)\n return self", "response": "Fits the classical entry point to the input similarities."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transform(self, X):\n '''\n Transforms X according to the linear transformation corresponding to\n flipping the input eigenvalues.\n\n Parameters\n ----------\n X : array, shape [n_test, n]\n The test similarities to training points.\n\n Returns\n -------\n Xt : array, shape [n_test, n]\n The transformed test similarites to training points.\n '''\n n = self.flip_.shape[0]\n if X.ndim != 2 or X.shape[1] != n:\n msg = \"X should have {} columns, the number of samples at fit time\"\n raise TypeError(msg.format(self.flip_.shape[0]))\n return np.dot(X, self.flip_)", "response": "Transforms X according to the linear transformation corresponding to\n flipping the input eigenvalues."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nflipping the negative eigenvalues of X and transforms the training similarities to the original training similarities.", "response": "def fit_transform(self, X, y=None):\n '''\n Flips the negative eigenvalues of X.\n\n Parameters\n ----------\n X : array, shape [n, n]\n The *symmetric* input similarities. If X is asymmetric, it will be\n treated as if it were symmetric based on its lower-triangular part.\n\n Returns\n -------\n Xt : array, shape [n, n]\n The transformed training similarities.\n '''\n n = X.shape[0]\n if X.shape != (n, n):\n raise TypeError(\"Input must be a square matrix.\")\n\n memory = get_memory(self.memory)\n discard_X = not self.copy and self.negatives_likely\n vals, vecs = memory.cache(scipy.linalg.eigh, ignore=['overwrite_a'])(\n X, overwrite_a=discard_X)\n vals = vals[:, None]\n\n self.clip_ = np.dot(vecs, np.sign(vals) * vecs.T)\n\n if discard_X or vals[0, 0] < 0:\n del X\n np.abs(vals, out=vals)\n X = np.dot(vecs, vals * vecs.T)\n del vals, vecs\n\n # should be symmetric, but make sure because floats\n X = Symmetrize(copy=False).fit_transform(X)\n return X"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fit(self, X, y=None):\n '''\n Learn the transformation to shifted eigenvalues. Only depends\n on the input dimension.\n\n Parameters\n ----------\n X : array, shape [n, n]\n The *symmetric* input similarities.\n '''\n n = X.shape[0]\n if X.shape != (n, n):\n raise TypeError(\"Input must be a square matrix.\")\n\n self.train_ = X\n\n memory = get_memory(self.memory)\n lo, = memory.cache(scipy.linalg.eigvalsh)(X, eigvals=(0, 0))\n self.shift_ = max(self.min_eig - lo, 0)\n\n return self", "response": "Learn the transformation to shifted eigenvalues. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension. Only dependsOnTheInputDimension on the input dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transform(self, X):\n '''\n Transforms X according to the linear transformation corresponding to\n shifting the input eigenvalues to all be at least ``self.min_eig``.\n\n Parameters\n ----------\n X : array, shape [n_test, n]\n The test similarities to training points.\n\n Returns\n -------\n Xt : array, shape [n_test, n]\n The transformed test similarites to training points. Only different\n from X if X is the training data.\n '''\n n = self.train_.shape[0]\n if X.ndim != 2 or X.shape[1] != n:\n msg = \"X should have {} columns, the number of samples at fit time\"\n raise TypeError(msg.format(n))\n\n if self.copy:\n X = X.copy()\n\n if self.shift_ != 0 and X is self.train_ or (\n X.shape == self.train_.shape and np.allclose(X, self.train_)):\n X[xrange(n), xrange(n)] += self.shift_\n return X", "response": "Transforms X according to the linear transformation corresponding to\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit(self, X, y=None):\n '''\n Picks the elements of the basis to use for the given data.\n\n Only depends on the dimension of X. If it's more convenient, you can\n pass a single integer for X, which is the dimension to use.\n\n Parameters\n ----------\n X : an integer, a :class:`Features` instance, or a list of bag features\n The input data, or just its dimension, since only the dimension is\n needed here.\n '''\n if is_integer(X):\n dim = X\n else:\n X = as_features(X)\n dim = X.dim\n M = self.smoothness\n\n # figure out the smooth-enough elements of our basis\n inds = np.mgrid[(slice(M + 1),) * dim].reshape(dim, (M + 1) ** dim).T\n self.inds_ = inds[(inds ** 2).sum(axis=1) <= M ** 2]\n return self", "response": "Fits the object to the given data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntransforming a list of bag features into a list of projection series series representation.", "response": "def transform(self, X):\n '''\n Transform a list of bag features into its projection series\n representation.\n\n Parameters\n ----------\n X : :class:`skl_groups.features.Features` or list of bag feature arrays\n New data to transform. The data should all lie in [0, 1];\n use :class:`skl_groups.preprocessing.BagMinMaxScaler` if not.\n\n Returns\n -------\n X_new : integer array, shape ``[len(X), dim_]``\n X transformed into the new space.\n '''\n self._check_fitted()\n M = self.smoothness\n dim = self.dim_\n inds = self.inds_\n do_check = self.do_bounds_check\n\n X = as_features(X)\n if X.dim != dim:\n msg = \"model fit for dimension {} but got dim {}\"\n raise ValueError(msg.format(dim, X.dim))\n\n Xt = np.empty((len(X), self.inds_.shape[0]))\n Xt.fill(np.nan)\n\n if self.basis == 'cosine': # TODO: put this in a C extension?\n coefs = (np.pi * np.arange(M + 1))[..., :]\n for i, bag in enumerate(X):\n if do_check:\n if np.min(bag) < 0 or np.max(bag) > 1:\n raise ValueError(\"Bag {} not in [0, 1]\".format(i))\n\n # apply each phi func to each dataset point: n x dim x M\n phi = coefs * bag[..., np.newaxis]\n np.cos(phi, out=phi)\n phi[:, :, 1:] *= np.sqrt(2)\n\n # B is the evaluation of each tensor-prodded basis func\n # at each point: n x inds.shape[0]\n B = reduce(op.mul, (phi[:, i, inds[:, i]] for i in xrange(dim)))\n\n Xt[i, :] = np.mean(B, axis=0)\n else:\n raise ValueError(\"unknown basis '{}'\".format(self.basis))\n\n return Xt"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_version(self): \n if (self.name is not None and self.version is not None\n and self.version.startswith(\":versiontools:\")):\n return (self.__get_live_version() or self.__get_frozen_version()\n or self.__fail_to_get_any_version())\n else:\n return self.__base.get_version(self)", "response": "Get the version of the current version of the object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a live version string using versiontools. Version. from_expression", "response": "def __get_live_version(self):\n \"\"\"\n Get a live version string using versiontools\n \"\"\"\n try:\n import versiontools\n except ImportError:\n return None\n else:\n return str(versiontools.Version.from_expression(self.name))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfit the transformer on the stacked points.", "response": "def fit(self, X, y=None, **params):\n '''\n Fit the transformer on the stacked points.\n\n Parameters\n ----------\n X : :class:`Features` or list of arrays of shape ``[n_samples[i], n_features]``\n Training set. If a Features object, it will be stacked.\n\n any other keyword argument :\n Passed on as keyword arguments to the transformer's ``fit()``.\n '''\n X = as_features(X, stack=True)\n self.transformer.fit(X.stacked_features, y, **params)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transform(self, X, **params):\n '''\n Transform the stacked points.\n\n Parameters\n ----------\n X : :class:`Features` or list of bag feature arrays\n New data to transform.\n\n any other keyword argument :\n Passed on as keyword arguments to the transformer's ``transform()``.\n\n Returns\n -------\n X_new : :class:`Features`\n Transformed features.\n '''\n X = as_features(X, stack=True)\n X_new = self.transformer.transform(X.stacked_features, **params)\n return self._gather_outputs(X, X_new)", "response": "Transform the stacked points X."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit_transform(self, X, y=None, **params):\n '''\n Fit and transform the stacked points.\n\n Parameters\n ----------\n X : :class:`Features` or list of bag feature arrays\n Data to train on and transform.\n\n any other keyword argument :\n Passed on as keyword arguments to the transformer's ``transform()``.\n\n Returns\n -------\n X_new : :class:`Features`\n Transformed features.\n '''\n X = as_features(X, stack=True)\n X_new = self.transformer.fit_transform(X.stacked_features, y, **params)\n return self._gather_outputs(X, X_new)", "response": "Fit and transform the stacked points."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntransforming data back to its original space i. e. return an input X_original whose transform would be X.", "response": "def inverse_transform(self, X, **params):\n '''\n Transform data back to its original space, i.e., return an input\n X_original whose transform would (maybe approximately) be X.\n\n Parameters\n ----------\n X : :class:`Features` or list of bag feature arrays\n Data to train on and transform.\n\n any other keyword argument :\n Passed on as keyword arguments to the transformer's \n ``inverse_transform()``.\n\n Returns\n -------\n X_original : :class:`Features`\n '''\n X = as_features(X, stack=True)\n Xo = self.transformer.inverse_transform(X.stacked_features, **params)\n return self._gather_outputs(X, Xo)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit(self, X, y=None):\n X = check_array(X, copy=self.copy,\n dtype=[np.float64, np.float32, np.float16, np.float128])\n\n feature_range = self.feature_range\n if feature_range[0] >= feature_range[1]:\n raise ValueError(\"Minimum of desired feature range must be smaller\"\n \" than maximum. Got %s.\" % str(feature_range))\n if self.fit_feature_range is not None:\n fit_feature_range = self.fit_feature_range\n if fit_feature_range[0] >= fit_feature_range[1]:\n raise ValueError(\"Minimum of desired (fit) feature range must \"\n \"be smaller than maximum. Got %s.\"\n % str(feature_range))\n if (fit_feature_range[0] < feature_range[0] or\n fit_feature_range[1] > feature_range[1]):\n raise ValueError(\"fit_feature_range must be a subset of \"\n \"feature_range. Got %s, fit %s.\"\n % (str(feature_range),\n str(fit_feature_range)))\n feature_range = fit_feature_range\n\n data_min = np.min(X, axis=0)\n data_range = np.max(X, axis=0) - data_min\n # Do not scale constant features\n data_range[data_range == 0.0] = 1.0\n self.scale_ = (feature_range[1] - feature_range[0]) / data_range\n self.min_ = feature_range[0] - data_min * self.scale_\n self.data_range = data_range\n self.data_min = data_min\n return self", "response": "Fit the minimum and maximum of the current object to the data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nscale features of X according to feature_range.", "response": "def transform(self, X):\n \"\"\"Scaling features of X according to feature_range.\n\n Parameters\n ----------\n X : array-like with shape [n_samples, n_features]\n Input data that will be transformed.\n \"\"\"\n X = check_array(X, copy=self.copy)\n X *= self.scale_\n X += self.min_\n if self.truncate:\n np.maximum(self.feature_range[0], X, out=X)\n np.minimum(self.feature_range[1], X, out=X)\n return X"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nundoing the scaling of X according to feature_range.", "response": "def inverse_transform(self, X):\n \"\"\"Undo the scaling of X according to feature_range.\n\n Note that if truncate is true, any truncated points will not\n be restored exactly.\n\n Parameters\n ----------\n X : array-like with shape [n_samples, n_features]\n Input data that will be transformed.\n \"\"\"\n X = check_array(X, copy=self.copy)\n X -= self.min_\n X /= self.scale_\n return X"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfits the KMeans model on a training set.", "response": "def fit(self, X, y=None):\n '''\n Choose the codewords based on a training set.\n\n Parameters\n ----------\n X : :class:`skl_groups.features.Features` or list of arrays of shape ``[n_samples[i], n_features]``\n Training set. If a Features object, it will be stacked.\n '''\n self.kmeans_fit_ = copy(self.kmeans)\n X = as_features(X, stack=True)\n self.kmeans_fit_.fit(X.stacked_features) \n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transform(self, X):\n '''\n Transform a list of bag features into its bag-of-words representation.\n\n Parameters\n ----------\n X : :class:`skl_groups.features.Features` or list of bag feature arrays\n New data to transform.\n\n Returns\n -------\n X_new : integer array, shape [len(X), kmeans.n_clusters]\n X transformed into the new space.\n '''\n self._check_fitted()\n X = as_features(X, stack=True)\n assignments = self.kmeans_fit_.predict(X.stacked_features)\n return self._group_assignments(X, assignments)", "response": "Transform a list of bag features into its bag - of - words representation."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute clustering and transform a list of bag features into its bag - of - words representation. Like calling fit and then transform ( X ).", "response": "def fit_transform(self, X):\n '''\n Compute clustering and transform a list of bag features into its\n bag-of-words representation. Like calling fit(X) and then transform(X),\n but more efficient.\n\n Parameters\n ----------\n X : :class:`skl_groups.features.Features` or list of bag feature arrays\n New data to transform.\n\n Returns\n -------\n X_new : integer array, shape [len(X), kmeans.n_clusters]\n X transformed into the new space.\n '''\n X = as_features(X, stack=True)\n self.kmeans_fit_ = copy(self.kmeans)\n assignments = self.kmeans_fit_.fit_predict(X.stacked_features) \n return self._group_assignments(X, assignments)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks whether the array is either integral or boolean.", "response": "def is_categorical_type(ary):\n \"Checks whether the array is either integral or boolean.\"\n ary = np.asanyarray(ary)\n return is_integer_type(ary) or ary.dtype.kind == 'b'"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns argument as an integer array converting floats if convertable.", "response": "def as_integer_type(ary):\n '''\n Returns argument as an integer array, converting floats if convertable.\n Raises ValueError if it's a float array with nonintegral values.\n '''\n ary = np.asanyarray(ary)\n if is_integer_type(ary):\n return ary\n rounded = np.rint(ary)\n if np.any(rounded != ary):\n raise ValueError(\"argument array must contain only integers\")\n return rounded.astype(int)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndisplay the progress logs for a given module.", "response": "def show_progress(name, **kwargs):\n '''\n Sets up a :class:`ProgressBarHandler` to handle progess logs for\n a given module.\n\n Parameters\n ----------\n name : string\n The module name of the progress logger to use. For example,\n :class:`skl_groups.divergences.KNNDivergenceEstimator`\n uses ``'skl_groups.divergences.knn.progress'``.\n\n * : anything\n Other keyword arguments are passed to the :class:`ProgressBarHandler`.\n '''\n logger = logging.getLogger(name)\n logger.setLevel(logging.INFO)\n logger.addHandler(ProgressBarHandler(**kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsignal the start of the process.", "response": "def start(self, total):\n '''\n Signal the start of the process.\n\n Parameters\n ----------\n total : int\n The total number of steps in the process, or None if unknown.\n '''\n self.logger.info(json.dumps(['START', self.name, total]))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _build_indices(X, flann_args):\n \"Builds FLANN indices for each bag.\"\n # TODO: should probably multithread this\n logger.info(\"Building indices...\")\n indices = [None] * len(X)\n for i, bag in enumerate(plog(X, name=\"index building\")):\n indices[i] = idx = FLANNIndex(**flann_args)\n idx.build_index(bag)\n return indices", "response": "Builds FLANN indices for each bag."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets within - bag distances for each bag.", "response": "def _get_rhos(X, indices, Ks, max_K, save_all_Ks, min_dist):\n \"Gets within-bag distances for each bag.\"\n logger.info(\"Getting within-bag distances...\")\n\n if max_K >= X.n_pts.min():\n msg = \"asked for K = {}, but there's a bag with only {} points\"\n raise ValueError(msg.format(max_K, X.n_pts.min()))\n\n # need to throw away the closest neighbor, which will always be self\n # thus K=1 corresponds to column 1 in the result array\n which_Ks = slice(1, None) if save_all_Ks else Ks\n\n indices = plog(indices, name=\"within-bag distances\")\n rhos = [None] * len(X)\n for i, (idx, bag) in enumerate(zip(indices, X)):\n r = np.sqrt(idx.nn_index(bag, max_K + 1)[1][:, which_Ks])\n np.maximum(min_dist, r, out=r)\n rhos[i] = r\n return rhos"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef linear(Ks, dim, num_q, rhos, nus):\n r'''\n Estimates the linear inner product \\int p q between two distributions,\n based on kNN distances.\n '''\n return _get_linear(Ks, dim)(num_q, rhos, nus)", "response": "r Estimates the linear inner product between two distributions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef renyi(alphas, Ks, dim, required, min_val=np.spacing(1),\n clamp=True, to_self=False):\n r'''\n Estimate the Renyi-alpha divergence between distributions, based on kNN\n distances: 1/(\\alpha-1) \\log \\int p^alpha q^(1-\\alpha)\n\n If the inner integral is less than min_val (default ``np.spacing(1)``),\n uses the log of min_val instead.\n\n If clamp (the default), enforces that the estimates are nonnegative by\n replacing any negative estimates with 0.\n\n Returns an array of shape (num_alphas, num_Ks).\n '''\n alphas = np.reshape(alphas, (-1, 1))\n est = required\n\n est = np.maximum(est, min_val) # TODO: can we modify in-place?\n np.log(est, out=est)\n est /= alphas - 1\n if clamp:\n np.maximum(est, 0, out=est)\n return est", "response": "r Estimates the Renyi - alpha divergence between distributions based on kNN alphabetical distances."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef tsallis(alphas, Ks, dim, required, clamp=True, to_self=False):\n r'''\n Estimate the Tsallis-alpha divergence between distributions, based on kNN\n distances: (\\int p^alpha q^(1-\\alpha) - 1) / (\\alpha - 1)\n\n If clamp (the default), enforces the estimate is nonnegative.\n\n Returns an array of shape (num_alphas, num_Ks).\n '''\n alphas = np.reshape(alphas, (-1, 1))\n alpha_est = required\n\n est = alpha_est - 1\n est /= alphas - 1\n if clamp:\n np.maximum(est, 0, out=est)\n return est", "response": "r Estimates the Tsallis - alpha divergence between distributions based on kNN - alpha distance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef quadratic(Ks, dim, rhos, required=None):\n r'''\n Estimates \\int p^2 based on kNN distances.\n\n In here because it's used in the l2 distance, above.\n\n Returns array of shape (num_Ks,).\n '''\n # Estimated with alpha=1, beta=0:\n # B_{k,d,1,0} is the same as B_{k,d,0,1} in linear()\n # and the full estimator is\n # B / (n - 1) * mean(rho ^ -dim)\n N = rhos.shape[0]\n Ks = np.asarray(Ks)\n Bs = (Ks - 1) / np.pi ** (dim / 2) * gamma(dim / 2 + 1) # shape (num_Ks,)\n est = Bs / (N - 1) * np.mean(rhos ** (-dim), axis=0)\n return est", "response": "r Estimates \\ int p^2 based on kNN distances rhos."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef jensen_shannon(Ks, dim, X_rhos, Y_rhos, required,\n clamp=True, to_self=False):\n r'''\n Estimate the difference between the Shannon entropy of an equally-weighted\n mixture between X and Y and the mixture of the Shannon entropies:\n\n JS(X, Y) = H[ (X + Y) / 2 ] - (H[X] + H[Y]) / 2\n\n We use a special case of the Hino-Murata weighted information estimator with\n a fixed M = n \\alpha, about equivalent to the K-nearest-neighbor approach\n used for the other estimators:\n\n Hideitsu Hino and Noboru Murata (2013).\n Information estimators for weighted observations. Neural Networks.\n http://linkinghub.elsevier.com/retrieve/pii/S0893608013001676\n\n\n The estimator for JS(X, Y) is:\n\n log volume of the unit ball - log M + log(n + m - 1) + digamma(M)\n + 1/2 mean_X( d * log radius of largest ball in X+Y around X_i\n with no more than M/(n+m-1) weight\n where X points have weight 1 / (2 n - 1)\n and Y points have weight n / (m (2 n - 1))\n - digamma(# of neighbors in that ball) )\n + 1/2 mean_Y( d * log radius of largest ball in X+Y around Y_i\n with no more than M/(n+m-1) weight\n where X points have weight m / (n (2 m - 1))\n and Y points have weight 1 / (2 m - 1)\n - digamma(# of neighbors in that ball) )\n\n - 1/2 (log volume of the unit ball - log M + log(n - 1) + digamma(M))\n - 1/2 mean_X( d * log radius of the largest ball in X around X_i\n with no more than M/(n-1) weight\n where X points have weight 1 / (n - 1))\n - digamma(# of neighbors in that ball) )\n\n - 1/2 (log volume of the unit ball - log M + log(m - 1) + digamma(M))\n - 1/2 mean_Y( d * log radius of the largest ball in Y around Y_i\n with no more than M/(n-1) weight\n where X points have weight 1 / (m - 1))\n - digamma(# of neighbors in that ball) )\n\n =\n\n log(n + m - 1) + digamma(M)\n + 1/2 mean_X( d * log radius of largest ball in X+Y around X_i\n with no more than M/(n+m-1) weight\n where X points have weight 1 / (2 n - 1)\n and Y points have weight n / (m (2 n - 1))\n - digamma(# of neighbors in that ball) )\n + 1/2 mean_Y( d * log radius of largest ball in X+Y around Y_i\n with no more than M/(n+m-1) weight\n where X points have weight m / (n (2 m - 1))\n and Y points have weight 1 / (2 m - 1)\n - digamma(# of neighbors in that ball) )\n - 1/2 [log(n-1) + mean_X( d * log rho_M(X_i) )]\n - 1/2 [log(m-1) + mean_Y( d * log rho_M(Y_i) )]\n '''\n\n X_ns = np.array([rho.shape[0] for rho in X_rhos])\n Y_ns = np.array([rho.shape[0] for rho in Y_rhos])\n n_X = X_ns.size\n n_Y = Y_ns.size\n\n # cores[0, k, i, j, 0] is mean_X(d * ... - psi(...)) for X[i], Y[j], M=Ks[k]\n # cores[0, k, i, j, 1] is mean_Y(d * ... - psi(...)) for X[i], Y[j], M=Ks[k]\n cores = required\n assert cores.shape == (1, Ks.size, n_X, n_Y, 2)\n\n # X_bits[k, i] is log(n-1) + mean_X( d * log rho_M(X_i) ) for X[i], M=Ks[k]\n X_bits = np.empty((Ks.size, n_X), dtype=np.float32)\n for i, rho in enumerate(X_rhos):\n X_bits[:, i] = dim * np.mean(np.log(rho), axis=0)\n X_bits += np.log(X_ns - 1)[np.newaxis, :]\n\n # Y_bits[k, j] is log(n-1) + mean_Y( d * log rho_M(Y_i) ) for Y[j], M=Ks[k]\n Y_bits = np.empty((Ks.size, n_Y), dtype=np.float32)\n for j, rho in enumerate(Y_rhos):\n Y_bits[:, j] = dim * np.mean(np.log(rho), axis=0)\n Y_bits += np.log(Y_ns - 1)[np.newaxis, :]\n\n est = cores.sum(axis=4)\n est -= X_bits.reshape(1, Ks.size, n_X, 1)\n est -= Y_bits.reshape(1, Ks.size, 1, n_Y)\n est /= 2\n est += np.log(-1 + X_ns[None, None, :, None] + Y_ns[None, None, None, :])\n est += psi(Ks)[None, :, None, None]\n\n # diagonal is zero\n if to_self:\n est[:, :, xrange(n_X), xrange(n_Y)] = 0\n\n if clamp: # know that 0 <= JS <= ln(2)\n np.maximum(0, est, out=est)\n np.minimum(np.log(2), est, out=est)\n return est[:, :, :, :, None]", "response": "r Estimates the difference between the Shannon entropy between X and Y and the mixture of the Shannon entropies between X and Y."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef topological_sort(deps):\n '''\n Topologically sort a DAG, represented by a dict of child => set of parents.\n The dependency dict is destroyed during operation.\n\n Uses the Kahn algorithm: http://en.wikipedia.org/wiki/Topological_sorting\n Not a particularly good implementation, but we're just running it on tiny\n graphs.\n '''\n order = []\n available = set()\n\n def _move_available():\n to_delete = []\n for n, parents in iteritems(deps):\n if not parents:\n available.add(n)\n to_delete.append(n)\n for n in to_delete:\n del deps[n]\n\n _move_available()\n while available:\n n = available.pop()\n order.append(n)\n for parents in itervalues(deps):\n parents.discard(n)\n _move_available()\n\n if available:\n raise ValueError(\"dependency cycle found\")\n return order", "response": "Topologically sort a DAG by a dict of child => set of parents."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse_specs(specs, Ks):\n '''\n Set up the different functions we need to call.\n\n Returns:\n - a dict mapping base estimator functions to _FuncInfo objects.\n If the function needs_alpha, then the alphas attribute is an array\n of alpha values and pos is a corresponding array of indices.\n Otherwise, alphas is None and pos is a list containing a single index.\n Indices are >= 0 if they correspond to something in a spec,\n and negative if they're just used for a meta estimator but not\n directly requested.\n - an OrderedDict mapping functions to _MetaFuncInfo objects.\n alphas and pos are like for _FuncInfo; deps is a list of indices\n which should be passed to the estimator. Note that these might be\n other meta functions; this list is guaranteed to be in an order\n such that all dependencies are resolved before calling that function.\n If no such order is possible, raise ValueError.\n - the number of meta-only results\n\n # TODO: update doctests for _parse_specs\n\n >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9'])\n ({:\n _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3])},\n OrderedDict([\n (,\n _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),\n (,\n _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))\n ]), 3)\n\n >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9', 'l2'])\n ({:\n _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3]),\n : _FuncInfo(alphas=None, pos=[-4])\n }, OrderedDict([\n (,\n _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),\n (,\n _MetaFuncInfo(alphas=None, pos=[3], deps=[-4])),\n (,\n _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))\n ]), 4)\n\n >>> _parse_specs(['renyi:.8', 'hellinger', 'renyi:.9', 'l2', 'linear'])\n ({:\n _FuncInfo(alphas=[0.8, 0.5, 0.9], pos=[-1, -2, -3]),\n : _FuncInfo(alphas=None, pos=[4])\n }, OrderedDict([\n (,\n _MetaFuncInfo(alphas=None, pos=[1], deps=[array(-2)])),\n (,\n _MetaFuncInfo(alphas=None, pos=[3], deps=[4])),\n (,\n _MetaFuncInfo(alphas=[0.8, 0.9], pos=[0, 2], deps=[-1, -3]))\n ]), 3)\n '''\n funcs = {}\n metas = {}\n meta_deps = defaultdict(set)\n\n def add_func(func, alpha=None, pos=None):\n needs_alpha = getattr(func, 'needs_alpha', False)\n is_meta = hasattr(func, 'needs_results')\n\n d = metas if is_meta else funcs\n if func not in d:\n if needs_alpha:\n args = {'alphas': [alpha], 'pos': [pos]}\n else:\n args = {'alphas': None, 'pos': [pos]}\n\n if not is_meta:\n d[func] = _FuncInfo(**args)\n else:\n d[func] = _MetaFuncInfo(deps=[], **args)\n for req in func.needs_results:\n if callable(req.alpha):\n req_alpha = req.alpha(alpha)\n else:\n req_alpha = req.alpha\n add_func(req.func, alpha=req_alpha)\n meta_deps[func].add(req.func)\n meta_deps[req.func] # make sure required func is in there\n\n else:\n # already have an entry for the func\n # need to give it this pos, if it's not None\n # and also make sure that the alpha is present\n info = d[func]\n if not needs_alpha:\n if pos is not None:\n if info.pos != [None]:\n msg = \"{} passed more than once\"\n raise ValueError(msg.format(func_name))\n\n info.pos[0] = pos\n else: # needs alpha\n try:\n idx = info.alphas.index(alpha)\n except ValueError:\n # this is a new alpha value we haven't seen yet\n info.alphas.append(alpha)\n info.pos.append(pos)\n if is_meta:\n for req in func.needs_results:\n if callable(req.alpha):\n req_alpha = req.alpha(alpha)\n else:\n req_alpha = req.alpha\n add_func(req.func, alpha=req_alpha)\n else:\n # repeated alpha value\n if pos is not None:\n if info.pos[idx] is not None:\n msg = \"{} with alpha {} passed more than once\"\n raise ValueError(msg.format(func_name, alpha))\n info.pos[idx] = pos\n\n # add functions for each spec\n for i, spec in enumerate(specs):\n func_name, alpha = (spec.split(':', 1) + [None])[:2]\n if alpha is not None:\n alpha = float(alpha)\n\n try:\n func = func_mapping[func_name]\n except KeyError:\n msg = \"'{}' is not a known function type\"\n raise ValueError(msg.format(func_name))\n\n needs_alpha = getattr(func, 'needs_alpha', False)\n if needs_alpha and alpha is None:\n msg = \"{} needs alpha but not passed in spec '{}'\"\n raise ValueError(msg.format(func_name, spec))\n elif not needs_alpha and alpha is not None:\n msg = \"{} doesn't need alpha but is passed in spec '{}'\"\n raise ValueError(msg.format(func_name, spec))\n\n add_func(func, alpha, i)\n\n # number things that are dependencies only\n meta_counter = itertools.count(-1, step=-1)\n for info in itertools.chain(itervalues(funcs), itervalues(metas)):\n for i, pos in enumerate(info.pos):\n if pos is None:\n info.pos[i] = next(meta_counter)\n\n # fill in the dependencies for metas\n for func, info in iteritems(metas):\n deps = info.deps\n assert deps == []\n\n for req in func.needs_results:\n f = req.func\n req_info = (metas if hasattr(f, 'needs_results') else funcs)[f]\n if req.alpha is not None:\n if callable(req.alpha):\n req_alpha = req.alpha(info.alphas)\n else:\n req_alpha = req.alpha\n\n find_alpha = np.vectorize(req_info.alphas.index, otypes=[int])\n pos = np.asarray(req_info.pos)[find_alpha(req_alpha)]\n if np.isscalar(pos):\n deps.append(pos[()])\n else:\n deps.extend(pos)\n else:\n pos, = req_info.pos\n deps.append(pos)\n\n # topological sort of metas\n meta_order = topological_sort(meta_deps)\n metas_ordered = OrderedDict(\n (f, metas[f]) for f in meta_order if hasattr(f, 'needs_results'))\n\n return funcs, metas_ordered, -next(meta_counter) - 1", "response": "Parse the spec and return a new base estimator that can be used to generate the new ones."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _flann_args(self, X=None):\n \"The dictionary of arguments to give to FLANN.\"\n args = {'cores': self._n_jobs}\n if self.flann_algorithm == 'auto':\n if X is None or X.dim > 5:\n args['algorithm'] = 'linear'\n else:\n args['algorithm'] = 'kdtree_single'\n else:\n args['algorithm'] = self.flann_algorithm\n if self.flann_args:\n args.update(self.flann_args)\n\n # check that arguments are correct\n try:\n FLANNParameters().update(args)\n except AttributeError as e:\n msg = \"flann_args contains an invalid argument:\\n {}\"\n raise TypeError(msg.format(e))\n\n return args", "response": "The dictionary of arguments to give to FLANN."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X, y=None, get_rhos=False):\n '''\n Sets up for divergence estimation \"from\" new data \"to\" X.\n Builds FLANN indices for each bag, and maybe gets within-bag distances.\n\n Parameters\n ----------\n X : list of arrays or :class:`skl_groups.features.Features`\n The bags to search \"to\".\n\n get_rhos : boolean, optional, default False\n Compute within-bag distances :attr:`rhos_`. These are only needed\n for some divergence functions or if do_sym is passed, and they'll\n be computed (and saved) during :meth:`transform` if they're not\n computed here.\n\n If you're using Jensen-Shannon divergence, a higher max_K may\n be needed once it sees the number of points in the transformed bags,\n so the computation here might be wasted.\n '''\n self.features_ = X = as_features(X, stack=True, bare=True)\n\n # if we're using a function that needs to pick its K vals itself,\n # then we need to set max_K here. when we transform(), might have to\n # re-do this :|\n Ks = self._get_Ks()\n _, _, _, max_K, save_all_Ks, _ = _choose_funcs(\n self.div_funcs, Ks, X.dim, X.n_pts, None, self.version)\n\n if max_K >= X.n_pts.min():\n msg = \"asked for K = {}, but there's a bag with only {} points\"\n raise ValueError(msg.format(max_K, X.n_pts.min()))\n\n memory = self.memory\n if isinstance(memory, string_types):\n memory = Memory(cachedir=memory, verbose=0)\n\n self.indices_ = id = memory.cache(_build_indices)(X, self._flann_args())\n if get_rhos:\n self.rhos_ = _get_rhos(X, id, Ks, max_K, save_all_Ks, self.min_dist)\n elif hasattr(self, 'rhos_'):\n del self.rhos_\n return self", "response": "Fits the divergence estimator to the new data X."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef transform(self, X):\n r'''\n Computes the divergences from X to :attr:`features_`.\n\n Parameters\n ----------\n X : list of bag feature arrays or :class:`skl_groups.features.Features`\n The bags to search \"from\".\n\n Returns\n -------\n divs : array of shape ``[len(div_funcs), len(Ks), len(X), len(features_)] + ([2] if do_sym else [])``\n The divergences from X to :attr:`features_`.\n ``divs[d, k, i, j]`` is the ``div_funcs[d]`` divergence\n from ``X[i]`` to ``fetaures_[j]`` using a K of ``Ks[k]``.\n If ``do_sym``, ``divs[d, k, i, j, 0]`` is\n :math:`D_{d,k}( X_i \\| \\texttt{features_}_j)` and\n ``divs[d, k, i, j, 1]`` is :math:`D_{d,k}(\\texttt{features_}_j \\| X_i)`.\n '''\n X = as_features(X, stack=True, bare=True)\n Y = self.features_\n\n Ks = np.asarray(self.Ks)\n\n if X.dim != Y.dim:\n msg = \"incompatible dimensions: fit with {}, transform with {}\"\n raise ValueError(msg.format(Y.dim, X.dim))\n\n memory = self.memory\n if isinstance(memory, string_types):\n memory = Memory(cachedir=memory, verbose=0)\n\n # ignore Y_indices to avoid slow pickling of them\n # NOTE: if the indices are approximate, then might not get the same\n # results!\n est = memory.cache(_est_divs, ignore=['n_jobs', 'Y_indices', 'Y_rhos'])\n output, self.rhos_ = est(\n X, Y, self.indices_, getattr(self, 'rhos_', None),\n self.div_funcs, Ks,\n self.do_sym, self.clamp, self.version, self.min_dist,\n self._flann_args(), self._n_jobs)\n return output", "response": "r Returns the divergences from X to Y."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a version of X as a features object.", "response": "def as_features(X, stack=False, bare=False):\n '''\n Returns a version of X as a :class:`Features` object.\n\n Parameters\n ----------\n stack : boolean, default False\n Make a stacked version of X. Note that if X is a features object,\n this will stack it in-place, since that's usually what you want.\n (If not, just use the :class:`Features` constructor instead.)\n\n bare : boolean, default False\n Return a bare version of X (no metadata).\n\n Returns\n -------\n feats : :class:`Features`\n A version of X. If X is already a :class:`Features` object, the original\n X may be returned, depending on the arguments.\n '''\n if isinstance(X, Features):\n if stack:\n X.make_stacked()\n return X.bare() if bare else X\n return Features(X, stack=stack, bare=bare)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncopy the Feature object. Makes a copy of the features array.", "response": "def copy(self, stack=False, copy_meta=False, memo=None):\n '''\n Copies the Feature object. Makes a copy of the features array.\n\n Parameters\n ----------\n stack : boolean, optional, default False\n Whether to stack the copy if this one is unstacked.\n\n copy_meta : boolean, optional, default False\n Also copy the metadata. If False, metadata in both points to the\n same object.\n '''\n if self.stacked:\n fs = deepcopy(self.stacked_features, memo)\n n_pts = self.n_pts.copy()\n elif stack:\n fs = np.vstack(self.features)\n n_pts = self.n_pts.copy()\n else:\n fs = deepcopy(self.features, memo)\n n_pts = None\n\n meta = deepcopy(self.meta, memo) if copy_meta else self.meta\n return Features(fs, n_pts, copy=False, **meta)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a Features object with no metadata ; points to the same features.", "response": "def bare(self):\n \"Make a Features object with no metadata; points to the same features.\"\n if not self.meta:\n return self\n elif self.stacked:\n return Features(self.stacked_features, self.n_pts, copy=False)\n else:\n return Features(self.features, copy=False)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef kl(Ks, dim, num_q, rhos, nus, clamp=True):\n r'''\n Estimate the KL divergence between distributions:\n \\int p(x) \\log (p(x) / q(x))\n using the kNN-based estimator (5) of\n Qing Wang, Sanjeev R Kulkarni, and Sergio Verdu (2009).\n Divergence Estimation for Multidimensional Densities Via\n k-Nearest-Neighbor Distances.\n IEEE Transactions on Information Theory.\n http://www.ee.princeton.edu/~verdu/reprints/WanKulVer.May2009.pdf\n which is:\n d * 1/n \\sum \\log (nu_k(i) / rho_k(i)) + log(m / (n - 1))\n\n If clamp, enforces KL >= 0.\n\n Returns an array of shape (num_Ks,).\n '''\n est = dim * np.mean(np.log(nus) - np.log(rhos), axis=0)\n est += np.log(num_q / (rhos.shape[0] - 1))\n if clamp:\n np.maximum(est, 0, out=est)\n return est", "response": "r Estimates the KL divergence between a set of multidimensional Densities and a set of kernels."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfitting the kernel to the data X.", "response": "def fit(self, X, y=None):\n '''\n Specify the data to which kernel values should be computed.\n\n Parameters\n ----------\n X : list of arrays or :class:`skl_groups.features.Features`\n The bags to compute \"to\".\n '''\n self.features_ = as_features(X, stack=True, bare=True)\n # TODO: could precompute things like squared norms if kernel == \"rbf\".\n # Probably should add support to sklearn instead of hacking it here.\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing kernels from X to Y.", "response": "def transform(self, X):\n '''\n Compute kernels from X to :attr:`features_`.\n\n Parameters\n ----------\n X : list of arrays or :class:`skl_groups.features.Features`\n The bags to compute \"from\". Must have same dimension as\n :attr:`features_`.\n\n Returns\n -------\n K : array of shape ``[len(X), len(features_)]``\n The kernel evaluations from X to :attr:`features_`.\n '''\n\n X = as_features(X, stack=True, bare=True)\n Y = self.features_\n\n if X.dim != Y.dim:\n raise ValueError(\"MMK transform got dimension {} but had {} at fit\"\n .format(X.dim, Y.dim))\n\n pointwise = pairwise_kernels(X.stacked_features, Y.stacked_features,\n metric=self.kernel,\n filter_params=True,\n **self._get_kernel_params())\n\n # TODO: is there a way to do this without a Python loop?\n K = np.empty((len(X), len(Y)))\n for i in range(len(X)):\n for j in range(len(Y)):\n K[i, j] = pointwise[X._boundaries[i]:X._boundaries[i+1],\n Y._boundaries[j]:Y._boundaries[j+1]].mean()\n\n return K"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transform(self, X):\n '''\n Transform a list of bag features into a matrix of its mean features.\n\n Parameters\n ----------\n X : :class:`skl_groups.features.Features` or list of bag feature arrays\n Data to transform.\n\n Returns\n -------\n X_new : array, shape ``[len(X), X.dim]``\n X transformed into its means.\n '''\n X = as_features(X)\n return np.vstack([np.mean(bag, axis=0) for bag in X])", "response": "Transform a list of bag features into a matrix of its mean features."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef from_shypo(cls, xml, encoding='utf-8'):\n score = float(xml.get('SCORE'))\n words = [Word.from_whypo(w_xml, encoding) for w_xml in xml.findall('WHYPO') if w_xml.get('WORD') not in ['', '']]\n return cls(words, score)", "response": "Constructor from xml element"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_whypo(cls, xml, encoding='utf-8'):\n word = unicode(xml.get('WORD'), encoding)\n confidence = float(xml.get('CM'))\n return cls(word, confidence)", "response": "Constructor from xml element *WHYPO*"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(self):\n logger.info(u'Started listening')\n while not self._stop:\n xml = self._readxml()\n\n # Exit on invalid XML\n if xml is None:\n break\n\n # Raw xml only\n if not self.modelize:\n logger.info(u'Raw xml: %s' % xml)\n self.results.put(xml)\n continue\n\n # Model objects + raw xml as fallback\n if xml.tag == 'RECOGOUT':\n sentence = Sentence.from_shypo(xml.find('SHYPO'), self.encoding)\n logger.info(u'Modelized recognition: %r' % sentence)\n self.results.put(sentence)\n else:\n logger.info(u'Unmodelized xml: %s' % xml)\n self.results.put(xml)\n\n logger.info(u'Stopped listening')", "response": "Read XML and store results in self. results."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef connect(self):\n try:\n logger.info(u'Connecting %s:%d' % (self.host, self.port))\n self.sock.connect((self.host, self.port))\n except socket.error:\n raise ConnectionError()\n self.state = CONNECTED", "response": "Connect to the server and update the internal state of the object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef disconnect(self):\n logger.info(u'Disconnecting')\n self.sock.shutdown(socket.SHUT_RDWR)\n self.sock.close()\n self.state = DISCONNECTED", "response": "Disconnect from the server"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send(self, command, timeout=5):\n logger.info(u'Sending %s' % command)\n _, writable, __ = select.select([], [self.sock], [], timeout)\n if not writable:\n raise SendTimeoutError()\n writable[0].sendall(command + '\\n')", "response": "Send a command to the server."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _readline(self):\n line = ''\n while 1:\n readable, _, __ = select.select([self.sock], [], [], 0.5)\n if self._stop:\n break\n if not readable:\n continue\n data = readable[0].recv(1)\n if data == '\\n':\n break\n line += unicode(data, self.encoding)\n return line", "response": "Read a line from the server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreading a block from the server. Lines are read until a character. is found.", "response": "def _readblock(self):\n \"\"\"Read a block from the server. Lines are read until a character ``.`` is found\n\n :return: the read block\n :rtype: string\n\n \"\"\"\n block = ''\n while not self._stop:\n line = self._readline()\n if line == '.':\n break\n block += line\n return block"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _readxml(self):\n block = re.sub(r'<(/?)s>', r'<\\1s>', self._readblock())\n try:\n xml = XML(block)\n except ParseError:\n xml = None\n return xml", "response": "Read a block and return the result as XML\n\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cli(id):\n ch = Analyse(id)\n ch.full_analysis()\n click.echo(\n 'Created: %s. Modified: %s. Deleted: %s' % (ch.create, ch.modify, ch.delete)\n )\n if ch.is_suspect:\n click.echo('The changeset {} is suspect! Reasons: {}'.format(\n id,\n ', '.join(ch.suspicion_reasons)\n ))\n else:\n click.echo('The changeset %s is not suspect!' % id)", "response": "Analyse an OpenStreetMap changeset."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_user_details(user_id):\n reasons = []\n try:\n url = OSM_USERS_API.format(user_id=requests.compat.quote(user_id))\n user_request = requests.get(url)\n if user_request.status_code == 200:\n user_data = user_request.content\n xml_data = ET.fromstring(user_data).getchildren()[0].getchildren()\n changesets = [i for i in xml_data if i.tag == 'changesets'][0]\n blocks = [i for i in xml_data if i.tag == 'blocks'][0]\n if int(changesets.get('count')) <= 5:\n reasons.append('New mapper')\n elif int(changesets.get('count')) <= 30:\n url = MAPBOX_USERS_API.format(\n user_id=requests.compat.quote(user_id)\n )\n user_request = requests.get(url)\n if user_request.status_code == 200:\n mapping_days = int(\n user_request.json().get('extra').get('mapping_days')\n )\n if mapping_days <= 5:\n reasons.append('New mapper')\n if int(blocks.getchildren()[0].get('count')) > 1:\n reasons.append('User has multiple blocks')\n except Exception as e:\n message = 'Could not verify user of the changeset: {}, {}'\n print(message.format(user_id, str(e)))\n return reasons", "response": "Get information about number of changesets blocks and mapping days of a user."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef changeset_info(changeset):\n keys = [tag.attrib.get('k') for tag in changeset.getchildren()]\n keys += ['id', 'user', 'uid', 'bbox', 'created_at']\n values = [tag.attrib.get('v') for tag in changeset.getchildren()]\n values += [\n changeset.get('id'), changeset.get('user'), changeset.get('uid'),\n get_bounds(changeset), changeset.get('created_at')\n ]\n\n return dict(zip(keys, values))", "response": "Return a dictionary with id user user_id bounds date of creation\n and all the tags of the changeset."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_changeset(changeset):\n url = 'https://www.openstreetmap.org/api/0.6/changeset/{}/download'.format(\n changeset\n )\n return ET.fromstring(requests.get(url).content)", "response": "Get the changeset using the OSM API and return the content as a XML\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the metadata of a changeset using the OSM API and return it as a XML .", "response": "def get_metadata(changeset):\n \"\"\"Get the metadata of a changeset using the OSM API and return it as a XML\n ElementTree.\n\n Args:\n changeset: the id of the changeset.\n \"\"\"\n url = 'https://www.openstreetmap.org/api/0.6/changeset/{}'.format(changeset)\n return ET.fromstring(requests.get(url).content).getchildren()[0]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the bounds of the changeset and return it as a Polygon object.", "response": "def get_bounds(changeset):\n \"\"\"Get the bounds of the changeset and return it as a Polygon object. If\n the changeset has not coordinates (case of the changesets that deal only\n with relations), it returns an empty Polygon.\n\n Args:\n changeset: the XML string of the changeset.\n \"\"\"\n try:\n return Polygon([\n (float(changeset.get('min_lon')), float(changeset.get('min_lat'))),\n (float(changeset.get('max_lon')), float(changeset.get('min_lat'))),\n (float(changeset.get('max_lon')), float(changeset.get('max_lat'))),\n (float(changeset.get('min_lon')), float(changeset.get('max_lat'))),\n (float(changeset.get('min_lon')), float(changeset.get('min_lat'))),\n ])\n except TypeError:\n return Polygon()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_words(text, suspect_words, excluded_words=[]):\n text = text.lower()\n suspect_found = [i for i in re.finditer(make_regex(suspect_words), text)]\n if len(excluded_words) > 0:\n excluded_found = [i for i in re.finditer(make_regex(excluded_words), text)]\n if len(suspect_found) > len(excluded_found):\n return True\n else:\n return False\n else:\n if len(suspect_found) > 0:\n return True\n else:\n return False", "response": "Check if a text has some of the suspect words or if it starts with a non - whitelisted word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndownloading the replication changeset file or read it directly from the changeset file system.", "response": "def read_file(self, changeset_file):\n \"\"\"Download the replication changeset file or read it directly from the\n filesystem (to test purposes).\n \"\"\"\n if isfile(changeset_file):\n self.filename = changeset_file\n else:\n self.path = mkdtemp()\n self.filename = join(self.path, basename(changeset_file))\n download(changeset_file, self.path)\n\n self.xml = ET.fromstring(gzip.open(self.filename).read())\n\n # delete folder created to download the file\n if not isfile(changeset_file):\n rmtree(self.path)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_area(self, geojson):\n geojson = json.load(open(geojson, 'r'))\n self.area = Polygon(geojson['features'][0]['geometry']['coordinates'][0])", "response": "Read the first feature from the geojson and return it as a Polygon\n object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfiltering the changesets that intersect with the geojson geometry.", "response": "def filter(self):\n \"\"\"Filter the changesets that intersects with the geojson geometry.\"\"\"\n self.content = [\n ch\n for ch in self.xml.getchildren()\n if get_bounds(ch).intersects(self.area)\n ]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the fields of this object based on the changeset.", "response": "def set_fields(self, changeset):\n \"\"\"Set the fields of this class with the metadata of the analysed\n changeset.\n \"\"\"\n self.id = int(changeset.get('id'))\n self.user = changeset.get('user')\n self.uid = changeset.get('uid')\n self.editor = changeset.get('created_by', None)\n self.review_requested = changeset.get('review_requested', False)\n self.host = changeset.get('host', 'Not reported')\n self.bbox = changeset.get('bbox').wkt\n self.comment = changeset.get('comment', 'Not reported')\n self.source = changeset.get('source', 'Not reported')\n self.imagery_used = changeset.get('imagery_used', 'Not reported')\n self.date = datetime.strptime(\n changeset.get('created_at'),\n '%Y-%m-%dT%H:%M:%SZ'\n )\n self.suspicion_reasons = []\n self.is_suspect = False\n self.powerfull_editor = False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef label_suspicious(self, reason):\n self.suspicion_reasons.append(reason)\n self.is_suspect = True", "response": "Add suspicion reason and set the suspect flag."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef full_analysis(self):\n self.count()\n self.verify_words()\n self.verify_user()\n\n if self.review_requested == 'yes':\n self.label_suspicious('Review requested')", "response": "Execute the count and verify_words methods."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nverify if the changeset was made by a user that was blocked more than once.", "response": "def verify_user(self):\n \"\"\"Verify if the changeset was made by a inexperienced mapper (anyone\n with less than 5 edits) or by a user that was blocked more than once.\n \"\"\"\n user_reasons = get_user_details(self.uid)\n [self.label_suspicious(reason) for reason in user_reasons]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef verify_words(self):\n if self.comment:\n if find_words(self.comment, self.suspect_words, self.excluded_words):\n self.label_suspicious('suspect_word')\n\n if self.source:\n for word in self.illegal_sources:\n if word in self.source.lower():\n self.label_suspicious('suspect_word')\n break\n\n if self.imagery_used:\n for word in self.illegal_sources:\n if word in self.imagery_used.lower():\n self.label_suspicious('suspect_word')\n break\n\n self.suspicion_reasons = list(set(self.suspicion_reasons))", "response": "Verify the fields source imagery_used and comment of the changeset\n for some suspect words."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef verify_editor(self):\n powerful_editors = [\n 'josm', 'level0', 'merkaartor', 'qgis', 'arcgis', 'upload.py',\n 'osmapi', 'Services_OpenStreetMap'\n ]\n if self.editor is not None:\n for editor in powerful_editors:\n if editor in self.editor.lower():\n self.powerfull_editor = True\n break\n\n if 'iD' in self.editor:\n trusted_hosts = [\n 'www.openstreetmap.org/id',\n 'www.openstreetmap.org/edit',\n 'improveosm.org',\n 'strava.github.io/iD',\n 'preview.ideditor.com/release',\n 'preview.ideditor.com/master',\n 'hey.mapbox.com/iD-internal',\n 'projets.pavie.info/id-indoor',\n 'maps.mapcat.com/edit',\n 'id.softek.ir'\n ]\n if self.host.split('://')[-1].strip('/') not in trusted_hosts:\n self.label_suspicious('Unknown iD instance')\n else:\n self.powerfull_editor = True\n self.label_suspicious('Software editor was not declared')", "response": "Verify if the software used in the changeset is a powerfull_editor."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef count(self):\n xml = get_changeset(self.id)\n actions = [action.tag for action in xml.getchildren()]\n self.create = actions.count('create')\n self.modify = actions.count('modify')\n self.delete = actions.count('delete')\n self.verify_editor()\n\n try:\n if (self.create / len(actions) > self.percentage and\n self.create > self.create_threshold and\n (self.powerfull_editor or self.create > self.top_threshold)):\n self.label_suspicious('possible import')\n elif (self.modify / len(actions) > self.percentage and\n self.modify > self.modify_threshold):\n self.label_suspicious('mass modification')\n elif ((self.delete / len(actions) > self.percentage and\n self.delete > self.delete_threshold) or\n self.delete > self.top_threshold):\n self.label_suspicious('mass deletion')\n except ZeroDivisionError:\n print('It seems this changeset was redacted')", "response": "Count the number of elements created modified and deleted by the\n archive."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nunwrapping a stream from a playlist URI.", "response": "def _unwrap_stream(uri, timeout, scanner, requests_session):\n \"\"\"\n Get a stream URI from a playlist URI, ``uri``.\n Unwraps nested playlists until something that's not a playlist is found or\n the ``timeout`` is reached.\n \"\"\"\n\n original_uri = uri\n seen_uris = set()\n deadline = time.time() + timeout\n\n while time.time() < deadline:\n if uri in seen_uris:\n logger.info(\n 'Unwrapping stream from URI (%s) failed: '\n 'playlist referenced itself', uri)\n return None\n else:\n seen_uris.add(uri)\n\n logger.debug('Unwrapping stream from URI: %s', uri)\n\n try:\n scan_timeout = deadline - time.time()\n if scan_timeout < 0:\n logger.info(\n 'Unwrapping stream from URI (%s) failed: '\n 'timed out in %sms', uri, timeout)\n return None\n scan_result = scanner.scan(uri, timeout=scan_timeout)\n except exceptions.ScannerError as exc:\n logger.debug('GStreamer failed scanning URI (%s): %s', uri, exc)\n scan_result = None\n\n if scan_result is not None and not (\n scan_result.mime.startswith('text/') or\n scan_result.mime.startswith('application/')):\n logger.debug(\n 'Unwrapped potential %s stream: %s', scan_result.mime, uri)\n return uri\n\n download_timeout = deadline - time.time()\n if download_timeout < 0:\n logger.info(\n 'Unwrapping stream from URI (%s) failed: timed out in %sms',\n uri, timeout)\n return None\n content = http.download(\n requests_session, uri, timeout=download_timeout)\n\n if content is None:\n logger.info(\n 'Unwrapping stream from URI (%s) failed: '\n 'error downloading URI %s', original_uri, uri)\n return None\n\n uris = playlists.parse(content)\n if not uris:\n logger.debug(\n 'Failed parsing URI (%s) as playlist; found potential stream.',\n uri)\n return uri\n\n # TODO Test streams and return first that seems to be playable\n logger.debug(\n 'Parsed playlist (%s) and found new URI: %s', uri, uris[0])\n uri = uris[0]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef serve(self, sock, request_handler, error_handler, debug=False,\n request_timeout=60, ssl=None, request_max_size=None,\n reuse_port=False, loop=None, protocol=HttpProtocol,\n backlog=100, **kwargs):\n \"\"\"Start asynchronous HTTP Server on an individual process.\n\n :param request_handler: Sanic request handler with middleware\n :param error_handler: Sanic error handler with middleware\n :param debug: enables debug output (slows server)\n :param request_timeout: time in seconds\n :param ssl: SSLContext\n :param sock: Socket for the server to accept connections from\n :param request_max_size: size in bytes, `None` for no limit\n :param reuse_port: `True` for multiple workers\n :param loop: asyncio compatible event loop\n :param protocol: subclass of asyncio protocol class\n :return: Nothing\n \"\"\"\n if debug:\n loop.set_debug(debug)\n\n server = partial(\n protocol,\n loop=loop,\n connections=self.connections,\n signal=self.signal,\n request_handler=request_handler,\n error_handler=error_handler,\n request_timeout=request_timeout,\n request_max_size=request_max_size,\n )\n\n server_coroutine = loop.create_server(\n server,\n host=None,\n port=None,\n ssl=ssl,\n reuse_port=reuse_port,\n sock=sock,\n backlog=backlog\n )\n # Instead of pulling time at the end of every request,\n # pull it once per minute\n loop.call_soon(partial(update_current_time, loop))\n return server_coroutine", "response": "Start asynchronous HTTP Server on an individual process."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef spawn(self, generations):\n egg_donors = [god for god in self.gods.values() if god.chromosomes == 'XX']\n sperm_donors = [god for god in self.gods.values() if god.chromosomes == 'XY']\n\n for i in range(generations):\n print(\"\\nGENERATION %d\\n\" % (i+1))\n gen_xx = []\n gen_xy = []\n\n for egg_donor in egg_donors:\n sperm_donor = random.choice(sperm_donors)\n brood = self.breed(egg_donor, sperm_donor)\n\n for child in brood:\n if child.divinity > human:\n # divine offspring join the Pantheon\n self.add_god(child)\n if child.chromosomes == 'XX':\n gen_xx.append(child)\n else:\n gen_xy.append(child)\n\n # elder gods leave the breeding pool\n egg_donors = [ed for ed in egg_donors if ed.generation > (i-2)]\n sperm_donors = [sd for sd in sperm_donors if sd.generation > (i-3)]\n\n # mature offspring join the breeding pool\n egg_donors += gen_xx\n sperm_donors += gen_xy", "response": "Grow this Pantheon by multiplying Gods."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn words from tokens that are most closely related to word.", "response": "def get_matches(word, tokens, limit, offset=0):\n \"\"\"Return words from that are most closely related to .\"\"\"\n return closest(tokens, word_vec(word), limit, offset)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cosine(vec1, vec2):\n if norm(vec1) > 0 and norm(vec2) > 0:\n return dot(vec1, vec2) / (norm(vec1) * norm(vec2))\n else:\n return 0.0", "response": "Compare vectors. Borrowed from A. Parish."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the limit words from tokens whose vectors most closely resemble the search_vec. Skip the first limit results.", "response": "def closest(tokens, search_vec, limit, offset=0):\n \"\"\"Return the words from whose vectors most closely\n resemble the search_vec. Skip the first results.\n \"\"\"\n return sorted(tokens,\n key=lambda x: cosine(search_vec, word_vec(x)),\n reverse=True)[offset:offset+limit]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a json file for each txt file in the / data directory.", "response": "def tokenize_texts():\n \"\"\"Generate a json file for each txt file in the /data/corpora directory.\"\"\"\n text_files = [fname for fname in os.listdir(corpora_dir) \\\n if fname.split('.')[1] == 'txt']\n\n for text_fname in text_files:\n json_fname = text_fname.split('.')[0] + '.json'\n if os.path.isfile(corpora_dir + json_fname):\n continue # already tokenized\n\n print(\"Tokenizing \" + text_fname)\n text = open(corpora_dir + text_fname).read()\n words = nltk.word_tokenize(text)\n with open(corpora_dir + json_fname, 'w') as outjson:\n json.dump(words, outjson)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new directory named dir_. Create a new file within it with sources. json.", "response": "def make_tokens_dir(dir_, sources):\n \"\"\"Create a new directory named . Create a new file within it called\n sources.json. The input is a list of names of tokenized texts.\n Write into sources.json.\n \"\"\"\n os.mkdir(tokens_dir + dir_)\n for source in sources:\n if not os.path.isfile(corpora_dir + source):\n print('Invalid source: ' + source)\n return\n\n with open(tokens_dir + dir_ + '/sources.json', 'w') as outjson:\n json.dump(sources, outjson)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_tokens_list(dir_, filters):\n with open(tokens_dir + dir_ + '/sources.json', 'r') as injson:\n data = json.load(injson)\n sources = [corpora_dir + fname for fname in data]\n\n with open('data/skipwords.txt', 'r') as f:\n skipwords = [line.rstrip() for line in f]\n\n tokens_list = []\n for fname in sources:\n print(\"Incorporating tokens from \" + fname)\n with open(fname, 'r') as injson:\n data = json.load(injson)\n words = [w.lower() for w in data if not w == '']\n filtered = [w for w,p in nltk.pos_tag(words) if p in filters]\n sanitized = [w for w in filtered if not w in skipwords]\n tokens_list += sanitized\n\n tokens_list = list(set(tokens_list)) # unique\n target = tokens_dir + dir_ + '/' + '-'.join(filters) + '.json'\n with open(target, 'w') as outjson:\n json.dump(tokens_list, outjson)", "response": "Find sources. json in dir_. It contains a list of tokenized texts. For each tokenized text listed in sources. json read its tokens and filter them and add them to the list."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_name_lists(ethnicity=None):\n if not ethnicity: ethnicity = random.choice(get_ethnicities())\n print(\"Loading names from: \" + ethnicity)\n filename = names_dir + ethnicity + '.json'\n\n try:\n with open(filename, 'r') as injson:\n data = json.load(injson)\n except:\n return 'Unable to read from file: ' + filename\n else:\n names = [ tuple(name.split(',')) for name in data ]\n random.shuffle(names)\n\n global female_names\n female_names = [name for name,gender,*desc in names if gender == 'girl']\n\n global male_names\n male_names = [name for name,gender,*desc in names if gender == 'boy']\n\n global nb_names\n nb_names = [name for name,gender,*desc in names if gender == 'boygirl']", "response": "Set three globally available lists of names."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_chromosomes(self, chromosomes=None):\n if chromosomes and chromosomes in valid_chromosomes:\n self.chromosomes = chromosomes\n else:\n self.chromosomes = random.choice([XX, XY])", "response": "This method sets the chromosomes of the log entry."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_inherited_traits(self, egg_donor, sperm_donor):\n if type(egg_donor) == str:\n self.reproduce_asexually(egg_donor, sperm_donor)\n else:\n self.reproduce_sexually(egg_donor, sperm_donor)", "response": "Accept either strings or Gods as inputs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nproduce two gametes an egg and a sperm from the input strings. Combine them to produce a genome a la sexual reproduction.", "response": "def reproduce_asexually(self, egg_word, sperm_word):\n \"\"\"Produce two gametes, an egg and a sperm, from the input strings.\n Combine them to produce a genome a la sexual reproduction.\n \"\"\"\n egg = self.generate_gamete(egg_word)\n sperm = self.generate_gamete(sperm_word)\n\n self.genome = list(set(egg + sperm)) # Eliminate duplicates\n self.generation = 1\n self.divinity = god"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reproduce_sexually(self, egg_donor, sperm_donor):\n egg_word = random.choice(egg_donor.genome)\n egg = self.generate_gamete(egg_word)\n sperm_word = random.choice(sperm_donor.genome)\n sperm = self.generate_gamete(sperm_word)\n\n self.genome = list(set(egg + sperm)) # Eliminate duplicates\n self.parents = [egg_donor.name, sperm_donor.name]\n self.generation = max(egg_donor.generation, sperm_donor.generation) + 1\n sum_ = egg_donor.divinity + sperm_donor.divinity\n self.divinity = int(npchoice(divinities, 1, p=p_divinity[sum_])[0])", "response": "Combine two gametes an egg and a sperm and assign divinity to the parents."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npicking a random name from the lists loaded with the model.", "response": "def set_name(self):\n \"\"\"Pick a random name from the lists loaded with the model. For Gods that\n identify as neither M nor F, the model attempts to retrieve an androgynous\n name. Note: not all of the scraped name lists contain androgynous names.\n \"\"\"\n if not self.gender: self.set_gender()\n\n name = ''\n if self.gender == female:\n name = names.female_names.pop()\n elif self.gender == male:\n name = names.male_names.pop()\n else:\n try:\n name = names.nb_names.pop()\n except:\n # No androgynous names available\n name = names.male_names.pop()\n\n self.name = name"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndivining an appropriate epithet for this God.", "response": "def set_epithet(self):\n \"\"\"Divine an appropriate epithet for this God. (See what I did there?)\"\"\"\n if self.divinity == human:\n obsession = random.choice(self.genome)\n if self.gender == female:\n self.epithet = 'ordinary woman'\n elif self.gender == male:\n self.epithet = 'ordinary man'\n else:\n self.epithet = 'ordinary human being'\n self.epithet += ' who loves ' + obsession\n return # Return early. The rest of the function deals with gods.\n\n if self.gender == female:\n title = 'Goddess'\n elif self.gender == male:\n title = 'God'\n else:\n title = 'Divine Being'\n if self.divinity == demi_god:\n title = 'Semi-' + title if self.gender == non_binary else 'Demi-' + title\n\n num_domains = npchoice([1,2,3,4], 1, p=[0.05, 0.35, 0.55, 0.05])[0]\n\n if num_domains == 1:\n template = '%s of %s'\n if num_domains == 2:\n template = '%s of %s and %s'\n elif num_domains == 3:\n template = '%s of %s, %s, and %s' # Oxford comma, the most divine punctuation.\n elif num_domains == 4:\n template = '%s of %s, %s, %s, and %s'\n\n self.domains = [d.title() for d in random.sample(self.genome, num_domains)]\n\n # Put it all together\n self.epithet = template % (title, *self.domains)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_gamete(self, egg_or_sperm_word):\n p_rate_of_mutation = [0.9, 0.1]\n should_use_mutant_pool = (npchoice([0,1], 1, p=p_rate_of_mutation)[0] == 1)\n if should_use_mutant_pool:\n pool = tokens.secondary_tokens\n else:\n pool = tokens.primary_tokens\n\n return get_matches(egg_or_sperm_word, pool, 23)", "response": "Generate a list of chromosomes from the given egg or SPerm word."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprints parents name and epithets.", "response": "def print_parents(self):\n \"\"\"Print parents' names and epithets.\"\"\"\n if self.gender == female:\n title = 'Daughter'\n elif self.gender == male:\n title = 'Son'\n else:\n title = 'Child'\n\n p1 = self.parents[0]\n p2 = self.parents[1]\n\n template = '%s of %s, the %s, and %s, the %s.'\n\n print(template % (title, p1.name, p1.epithet, p2.name, p2.epithet))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef instance(self, counter=None, pipeline_counter=None):\n pipeline_counter = pipeline_counter or self.pipeline_counter\n pipeline_instance = None\n\n if not pipeline_counter:\n pipeline_instance = self.server.pipeline(self.pipeline_name).instance()\n self.pipeline_counter = int(pipeline_instance['counter'])\n\n if not counter:\n if pipeline_instance is None:\n pipeline_instance = (\n self.server\n .pipeline(self.pipeline_name)\n .instance(pipeline_counter)\n )\n\n for stages in pipeline_instance['stages']:\n if stages['name'] == self.stage_name:\n return self.instance(\n counter=int(stages['counter']),\n pipeline_counter=pipeline_counter\n )\n\n return self._get('/instance/{pipeline_counter:d}/{counter:d}'\n .format(pipeline_counter=pipeline_counter, counter=counter))", "response": "Returns all the information regarding a specific stage run\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_json(self):\n return (self.content_type.startswith('application/json') or\n re.match(r'application/vnd.go.cd.v(\\d+)\\+json', self.content_type))", "response": "Returns True if content_type is json"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the payload of the message.", "response": "def payload(self):\n \"\"\"\n Returns:\n `str` when not json.\n `dict` when json.\n \"\"\"\n if self.is_json:\n if not self._body_parsed:\n if hasattr(self._body, 'decode'):\n body = self._body.decode('utf-8')\n else:\n body = self._body\n\n self._body_parsed = json.loads(body)\n\n return self._body_parsed\n else:\n return self._body"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nperforming a HTTP request to the Go server.", "response": "def request(self, path, data=None, headers=None, method=None):\n \"\"\"Performs a HTTP request to the Go server\n\n Args:\n path (str): The full path on the Go server to request.\n This includes any query string attributes.\n data (str, dict, bool, optional): If any data is present this\n request will become a POST request.\n headers (dict, optional): Headers to set for this particular\n request\n\n Raises:\n HTTPError: when the HTTP request fails.\n\n Returns:\n file like object: The response from a\n :func:`urllib2.urlopen` call\n \"\"\"\n if isinstance(data, str):\n data = data.encode('utf-8')\n response = urlopen(self._request(path, data=data, headers=headers, method=method))\n self._set_session_cookie(response)\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_logged_in_session(self, response=None):\n if not response:\n response = self.get('go/api/pipelines.xml')\n\n self._set_session_cookie(response)\n\n if not self._session_id:\n raise AuthenticationFailed('No session id extracted from request.')\n\n response = self.get('go/pipelines')\n match = re.search(\n r'name=\"authenticity_token\".+?value=\"([^\"]+)',\n response.read().decode('utf-8')\n )\n if match:\n self._authenticity_token = match.group(1)\n else:\n raise AuthenticationFailed('Authenticity token not found on page')", "response": "Add the logged in session to the request object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn an instance of a Stage class.", "response": "def stage(self, pipeline_name, stage_name, pipeline_counter=None):\n \"\"\"Returns an instance of :class:`Stage`\n\n Args:\n pipeline_name (str): Name of the pipeline the stage belongs to\n stage_name (str): Name of the stage to act on\n pipeline_counter (int): The pipeline instance the stage is for.\n\n Returns:\n Stage: an instantiated :class:`Stage`.\n \"\"\"\n return Stage(self, pipeline_name, stage_name, pipeline_counter=pipeline_counter)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef flatten(d):\n\n if not isinstance(d, dict):\n return [[d]]\n\n returned = []\n for key, value in d.items():\n # Each key, value is treated as a row.\n nested = flatten(value)\n for nest in nested:\n current_row = [key]\n current_row.extend(nest)\n returned.append(current_row)\n\n return returned", "response": "Return a dict as a list of lists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn all the information regarding a specific pipeline instance.", "response": "def instance(self, counter=None):\n \"\"\"Returns all the information regarding a specific pipeline run\n\n See the `Go pipeline instance documentation`__ for examples.\n\n .. __: http://api.go.cd/current/#get-pipeline-instance\n\n Args:\n counter (int): The pipeline instance to fetch.\n If falsey returns the latest pipeline instance from :meth:`history`.\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n \"\"\"\n if not counter:\n history = self.history()\n if not history:\n return history\n else:\n return Response._from_json(history['pipelines'][0])\n\n return self._get('/instance/{counter:d}'.format(counter=counter))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef schedule(self, variables=None, secure_variables=None, materials=None,\n return_new_instance=False, backoff_time=1.0):\n \"\"\"Schedule a pipeline run\n\n Aliased as :meth:`run`, :meth:`schedule`, and :meth:`trigger`.\n\n Args:\n variables (dict, optional): Variables to set/override\n secure_variables (dict, optional): Secure variables to set/override\n materials (dict, optional): Material revisions to be used for\n this pipeline run. The exact format for this is a bit iffy,\n have a look at the official\n `Go pipeline scheduling documentation`__ or inspect a call\n from triggering manually in the UI.\n return_new_instance (bool): Returns a :meth:`history` compatible\n response for the newly scheduled instance. This is primarily so\n users easily can get the new instance number. **Note:** This is done\n in a very naive way, it just checks that the instance number is\n higher than before the pipeline was triggered.\n backoff_time (float): How long between each check for\n :arg:`return_new_instance`.\n\n .. __: http://api.go.cd/current/#scheduling-pipelines\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n \"\"\"\n scheduling_args = dict(\n variables=variables,\n secure_variables=secure_variables,\n material_fingerprint=materials,\n headers={\"Confirm\": True},\n )\n\n scheduling_args = dict((k, v) for k, v in scheduling_args.items() if v is not None)\n\n # TODO: Replace this with whatever is the official way as soon as gocd#990 is fixed.\n # https://github.com/gocd/gocd/issues/990\n if return_new_instance:\n pipelines = self.history()['pipelines']\n if len(pipelines) == 0:\n last_run = None\n else:\n last_run = pipelines[0]['counter']\n response = self._post('/schedule', ok_status=202, **scheduling_args)\n if not response:\n return response\n\n max_tries = 10\n while max_tries > 0:\n current = self.instance()\n if not last_run and current:\n return current\n elif last_run and current['counter'] > last_run:\n return current\n else:\n time.sleep(backoff_time)\n max_tries -= 1\n\n # I can't come up with a scenario in testing where this would happen, but it seems\n # better than returning None.\n return response\n else:\n return self._post('/schedule', ok_status=202, **scheduling_args)", "response": "Schedules a new pipeline run."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef artifact(self, counter, stage, job, stage_counter=1):\n return Artifact(self.server, self.name, counter, stage, job, stage_counter)", "response": "Helper to instantiate an artifact object for a given pipeline."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nyield the output and metadata from all jobs in the pipeline", "response": "def console_output(self, instance=None):\n \"\"\"Yields the output and metadata from all jobs in the pipeline\n\n Args:\n instance: The result of a :meth:`instance` call, if not supplied\n the latest of the pipeline will be used.\n\n Yields:\n tuple: (metadata (dict), output (str)).\n\n metadata contains:\n - pipeline\n - pipeline_counter\n - stage\n - stage_counter\n - job\n - job_result\n \"\"\"\n if instance is None:\n instance = self.instance()\n\n for stage in instance['stages']:\n for job in stage['jobs']:\n if job['result'] not in self.final_results:\n continue\n\n artifact = self.artifact(\n instance['counter'],\n stage['name'],\n job['name'],\n stage['counter']\n )\n output = artifact.get('cruise-output/console.log')\n\n yield (\n {\n 'pipeline': self.name,\n 'pipeline_counter': instance['counter'],\n 'stage': stage['name'],\n 'stage_counter': stage['counter'],\n 'job': job['name'],\n 'job_result': job['result'],\n },\n output.body\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef stage(self, name, pipeline_counter=None):\n return Stage(\n self.server,\n pipeline_name=self.name,\n stage_name=name,\n pipeline_counter=pipeline_counter,\n )", "response": "Helper to instantiate a : class : gocd. api. stage. Stage object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate template config for specified template name.", "response": "def edit(self, config, etag):\n \"\"\"Update template config for specified template name.\n\n .. __: https://api.go.cd/current/#edit-template-config\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n \"\"\"\n\n data = self._json_encode(config)\n headers = self._default_headers()\n\n if etag is not None:\n headers[\"If-Match\"] = etag\n\n return self._request(self.name,\n ok_status=None,\n data=data,\n headers=headers,\n method=\"PUT\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new template config for the given template name.", "response": "def create(self, config):\n \"\"\"Create template config for specified template name.\n\n .. __: https://api.go.cd/current/#create-template-config\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n \"\"\"\n\n assert config[\"name\"] == self.name, \"Given config is not for this template\"\n\n data = self._json_encode(config)\n headers = self._default_headers()\n\n return self._request(\"\",\n ok_status=None,\n data=data,\n headers=headers)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndelete the template config for the specified template name.", "response": "def delete(self):\n \"\"\"Delete template config for specified template name.\n\n .. __: https://api.go.cd/current/#delete-a-template\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n \"\"\"\n\n headers = self._default_headers()\n\n return self._request(self.name,\n ok_status=None,\n data=None,\n headers=headers,\n method=\"DELETE\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pipelines(self):\n if not self.response:\n return set()\n elif self._pipelines is None and self.response:\n self._pipelines = set()\n for group in self.response.payload:\n for pipeline in group['pipelines']:\n self._pipelines.add(pipeline['name'])\n\n return self._pipelines", "response": "Returns a set of all the pipelines available in the last response\nAttributeNames"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget an artifact directory by its path.", "response": "def get_directory(self, path_to_directory, timeout=30, backoff=0.4, max_wait=4):\n \"\"\"Gets an artifact directory by its path.\n\n See the `Go artifact directory documentation`__ for example responses.\n\n .. __: http://api.go.cd/current/#get-artifact-directory\n\n .. note::\n Getting a directory relies on Go creating a zip file of the\n directory in question. Because of this Go will zip the file in\n the background and return a 202 Accepted response. It's then up\n to the client to check again later and get the final file.\n\n To work with normal assumptions this :meth:`get_directory` will\n retry itself up to ``timeout`` seconds to get a 200 response to\n return. At that point it will then return the response as is, no\n matter whether it's still 202 or 200. The retry is done with an\n exponential backoff with a max value between retries. See the\n ``backoff`` and ``max_wait`` variables.\n\n If you want to handle the retry logic yourself then use :meth:`get`\n and add '.zip' as a suffix on the directory.\n\n Args:\n path_to_directory (str): The path to the directory to get.\n It can be nested eg ``target/dist.zip``\n timeout (int): How many seconds we will wait in total for a\n successful response from Go when we're receiving 202\n backoff (float): The initial value used for backoff, raises\n exponentially until it reaches ``max_wait``\n max_wait (int): The max time between retries\n\n Returns:\n Response: :class:`gocd.api.response.Response` object\n A successful response is a zip-file.\n \"\"\"\n response = None\n started_at = None\n time_elapsed = 0\n\n i = 0\n while time_elapsed < timeout:\n response = self._get('{0}.zip'.format(path_to_directory))\n\n if response:\n break\n else:\n if started_at is None:\n started_at = time.time()\n\n time.sleep(min(backoff * (2 ** i), max_wait))\n i += 1\n time_elapsed = time.time() - started_at\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ask(self):\n with self.mutex:\n if not len(self.queue): \n raise Empty\n utcnow = dt.datetime.utcnow()\n if self.queue[0][0] <= utcnow:\n self.ready.notify()\n return 0\n return (self.queue[0][0] - utcnow).total_seconds()", "response": "Returns the wait time in seconds required to retrieve the next item in the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef p_transition(p):\n p[3] = None if p[3] == 'NULL' else p[3]\n if p[4] == 'error':\n p[0] = MarionetteTransition(p[1], p[2], p[3], 0, True)\n else:\n p[0] = MarionetteTransition(p[1], p[2], p[3], p[4], False)", "response": "transition : START_KWD KEY NULL_KWD FLOAT\n transition : KEY KEY NULL_KWD FLOAT\n transition : KEY END_KWD NULL_KWD FLOAT\n transition : START_KWD KEY KEY FLOAT\n transition : KEY KEY KEY FLOAT\n transition : KEY END_KWD KEY FLOAT\n transition : START_KWD KEY NULL_KWD INTEGER\n transition : KEY KEY NULL_KWD INTEGER\n transition : KEY END_KWD NULL_KWD INTEGER\n transition : START_KWD KEY KEY INTEGER\n transition : KEY KEY KEY INTEGER\n transition : KEY END_KWD KEY INTEGER\n transition : START_KWD KEY NULL_KWD KEY\n transition : KEY KEY NULL_KWD KEY\n transition : KEY END_KWD NULL_KWD KEY\n transition : START_KWD KEY KEY KEY\n transition : KEY KEY KEY KEY\n transition : KEY END_KWD KEY KEY"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse the action line.", "response": "def p_action(p):\n \"\"\"\n action : CLIENT_KWD KEY DOT KEY LPAREN args RPAREN\n action : SERVER_KWD KEY DOT KEY LPAREN args RPAREN\n action : CLIENT_KWD KEY DOT KEY LPAREN args RPAREN IF_KWD REGEX_MATCH_INCOMING_KWD LPAREN p_string_arg RPAREN\n action : SERVER_KWD KEY DOT KEY LPAREN args RPAREN IF_KWD REGEX_MATCH_INCOMING_KWD LPAREN p_string_arg RPAREN\n \"\"\"\n if len(p)==8:\n p[0] = [p[1], p[2], p[4], p[6], None]\n elif len(p)==13:\n p[0] = [p[1], p[2], p[4], p[6], p[11]]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef app_class():\n try:\n pkg_resources.get_distribution('invenio-files-rest')\n from invenio_files_rest.app import Flask as FlaskBase\n except pkg_resources.DistributionNotFound:\n from flask import Flask as FlaskBase\n\n # Add Host header validation via APP_ALLOWED_HOSTS configuration variable.\n class Request(TrustedHostsMixin, FlaskBase.request_class):\n pass\n\n class Flask(FlaskBase):\n request_class = Request\n\n return Flask", "response": "Create Flask application class."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef init_app(self, app, **kwargs):\n # Init the configuration\n self.init_config(app)\n # Enable Rate limiter\n self.limiter = Limiter(app, key_func=get_ipaddr)\n # Enable secure HTTP headers\n if app.config['APP_ENABLE_SECURE_HEADERS']:\n self.talisman = Talisman(\n app, **app.config.get('APP_DEFAULT_SECURE_HEADERS', {})\n )\n # Enable PING view\n if app.config['APP_HEALTH_BLUEPRINT_ENABLED']:\n blueprint = Blueprint('invenio_app_ping', __name__)\n\n @blueprint.route('/ping')\n def ping():\n \"\"\"Load balancer ping view.\"\"\"\n return 'OK'\n\n ping.talisman_view_options = {'force_https': False}\n\n app.register_blueprint(blueprint)\n\n requestid_header = app.config.get('APP_REQUESTID_HEADER')\n if requestid_header:\n @app.before_request\n def set_request_id():\n \"\"\"Extracts a request id from an HTTP header.\"\"\"\n request_id = request.headers.get(requestid_header)\n if request_id:\n # Capped at 200 to protect against malicious clients\n # sending very large headers.\n g.request_id = request_id[:200]\n\n # If installed register the Flask-DebugToolbar extension\n try:\n from flask_debugtoolbar import DebugToolbarExtension\n app.extensions['flask-debugtoolbar'] = DebugToolbarExtension(app)\n except ImportError:\n app.logger.debug('Flask-DebugToolbar extension not installed.')\n\n # Register self\n app.extensions['invenio-app'] = self", "response": "Initialize the application object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninitializing configuration. :param app: An instance of :class:`~flask.Flask`.", "response": "def init_config(self, app):\n \"\"\"Initialize configuration.\n\n :param app: An instance of :class:`~flask.Flask`.\n \"\"\"\n config_apps = ['APP_', 'RATELIMIT_']\n flask_talisman_debug_mode = [\"'unsafe-inline'\"]\n for k in dir(config):\n if any([k.startswith(prefix) for prefix in config_apps]):\n app.config.setdefault(k, getattr(config, k))\n\n if app.config['DEBUG']:\n app.config.setdefault('APP_DEFAULT_SECURE_HEADERS', {})\n headers = app.config['APP_DEFAULT_SECURE_HEADERS']\n # ensure `content_security_policy` is not set to {}\n if headers.get('content_security_policy') != {}:\n headers.setdefault('content_security_policy', {})\n csp = headers['content_security_policy']\n # ensure `default-src` is not set to []\n if csp.get('default-src') != []:\n csp.setdefault('default-src', [])\n # add default `content_security_policy` value when debug\n csp['default-src'] += flask_talisman_debug_mode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef remove_leading(needle, haystack):\n if haystack[:len(needle)] == needle:\n return haystack[len(needle):]\n return haystack", "response": "Remove leading needle from haystack."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove trailing needle from haystack.", "response": "def remove_trailing(needle, haystack):\n \"\"\"Remove trailing needle string (if exists).\n\n >>> remove_trailing('Test', 'ThisAndThatTest')\n 'ThisAndThat'\n >>> remove_trailing('Test', 'ArbitraryName')\n 'ArbitraryName'\n \"\"\"\n if haystack[-len(needle):] == needle:\n return haystack[:-len(needle)]\n return haystack"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef camel2word(string):\n def wordize(match):\n return ' ' + match.group(1).lower()\n\n return string[0] + re.sub(r'([A-Z])', wordize, string[1:])", "response": "Covert name from CamelCase to Normal case."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef complete_english(string):\n for x, y in [(\"dont\", \"don't\"),\n (\"doesnt\", \"doesn't\"),\n (\"wont\", \"won't\"),\n (\"wasnt\", \"wasn't\")]:\n string = string.replace(x, y)\n return string", "response": "Complete a string by removing all non - English characters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nformatting a time in seconds.", "response": "def format_seconds(self, n_seconds):\n \"\"\"Format a time in seconds.\"\"\"\n func = self.ok\n if n_seconds >= 60:\n n_minutes, n_seconds = divmod(n_seconds, 60)\n return \"%s minutes %s seconds\" % (\n func(\"%d\" % n_minutes),\n func(\"%.3f\" % n_seconds))\n else:\n return \"%s seconds\" % (\n func(\"%.3f\" % n_seconds))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ppdict(dict_to_print, br='\\n', html=False, key_align='l', sort_keys=True,\n key_preffix='', key_suffix='', value_prefix='', value_suffix='', left_margin=3, indent=2):\n \"\"\"Indent representation of a dict\"\"\"\n if dict_to_print:\n if sort_keys:\n dic = dict_to_print.copy()\n keys = list(dic.keys())\n keys.sort()\n dict_to_print = OrderedDict()\n for k in keys:\n dict_to_print[k] = dic[k]\n\n tmp = ['{']\n ks = [type(x) == str and \"'%s'\" % x or x for x in dict_to_print.keys()]\n vs = [type(x) == str and \"'%s'\" % x or x for x in dict_to_print.values()]\n max_key_len = max([len(str(x)) for x in ks])\n\n for i in range(len(ks)):\n k = {1: str(ks[i]).ljust(max_key_len),\n key_align == 'r': str(ks[i]).rjust(max_key_len)}[1]\n\n v = vs[i]\n tmp.append(' ' * indent + '{}{}{}:{}{}{},'.format(key_preffix, k, key_suffix,\n value_prefix, v, value_suffix))\n\n tmp[-1] = tmp[-1][:-1] # remove the ',' in the last item\n tmp.append('}')\n\n if left_margin:\n tmp = [' ' * left_margin + x for x in tmp]\n\n if html:\n return '{}'.format(br.join(tmp).replace(' ', ' '))\n else:\n return br.join(tmp)\n else:\n return '{}'", "response": "Pretty print a dict of items"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nasserting that result == expected.", "response": "def eq_(result, expected, msg=None):\n \"\"\"\n Shadow of the Nose builtin which presents easier to read multiline output.\n \"\"\"\n params = {'expected': expected, 'result': result}\n aka = \"\"\"\n\n--------------------------------- aka -----------------------------------------\n\nExpected:\n%(expected)r\n\nGot:\n%(result)r\n\"\"\" % params\n default_msg = \"\"\"\nExpected:\n%(expected)s\n\nGot:\n%(result)s\n\"\"\" % params\n if (\n (repr(result) != six.text_type(result)) or\n (repr(expected) != six.text_type(expected))\n ):\n default_msg += aka\n assertion_msg = msg or default_msg\n # This assert will bubble up to Nose's failure handling, which at some\n # point calls explicit str() - which will UnicodeDecodeError on any non\n # ASCII text.\n # To work around this, we make sure Unicode strings become bytestrings\n # beforehand, with explicit encode.\n if isinstance(assertion_msg, six.text_type):\n assertion_msg = assertion_msg.encode('utf-8')\n assert result == expected, assertion_msg"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that haystack contains needle.", "response": "def _assert_contains(haystack, needle, invert, escape=False):\n \"\"\"\n Test for existence of ``needle`` regex within ``haystack``.\n\n Say ``escape`` to escape the ``needle`` if you aren't really using the\n regex feature & have special characters in it.\n \"\"\"\n myneedle = re.escape(needle) if escape else needle\n matched = re.search(myneedle, haystack, re.M)\n if (invert and matched) or (not invert and not matched):\n raise AssertionError(\"'%s' %sfound in '%s'\" % (\n needle,\n \"\" if invert else \"not \",\n haystack\n ))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind the. splunk_logger config file in the current directory or in the user s home and parse it.", "response": "def parse_config_file():\n \"\"\"\n Find the .splunk_logger config file in the current directory, or in the\n user's home and parse it. The one in the current directory has precedence.\n \n :return: A tuple with:\n - project_id\n - access_token\n \"\"\"\n for filename in ('.splunk_logger', os.path.expanduser('~/.splunk_logger')):\n\n project_id, access_token, api_domain = _parse_config_file_impl(filename)\n\n if project_id is not None\\\n and access_token is not None\\\n and api_domain is not None:\n return project_id, access_token, api_domain\n\n else:\n return None, None, None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse_config_file_impl(filename):\n try:\n doc = yaml.load(file(filename).read())\n \n project_id = doc[\"credentials\"][\"project_id\"]\n access_token = doc[\"credentials\"][\"access_token\"]\n api_domain = doc[\"credentials\"][\"api_domain\"]\n \n return project_id, access_token, api_domain\n except:\n return None, None, None", "response": "Parse the config file into a tuple with the project_id access_token api_domain"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dem_url_dia(dt_day='2015-06-22'):\n\n def _url_tipo_dato(str_dia, k):\n url = SERVER + '/archives/{}/download_json?locale=es'.format(D_TIPOS_REQ_DEM[k])\n if type(str_dia) is str:\n return url + '&date=' + str_dia\n else:\n return url + '&date=' + str_dia.date().isoformat()\n\n urls = [_url_tipo_dato(dt_day, k) for k in D_TIPOS_REQ_DEM.keys()]\n return urls", "response": "Obtiene las urls de los datos de demanda energ\u00e9tica de un d\u00eda concreto."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dem_procesa_datos_dia(key_day, response):\n dfs_import, df_import, dfs_maxmin, hay_errores = [], None, [], 0\n for r in response:\n tipo_datos, data = _extract_func_json_data(r)\n if tipo_datos is not None:\n if ('IND_MaxMin' in tipo_datos) and data:\n df_import = _import_daily_max_min(data)\n dfs_maxmin.append(df_import)\n elif data:\n df_import = _import_json_ts_data(data)\n dfs_import.append(df_import)\n if tipo_datos is None or df_import is None:\n hay_errores += 1\n if hay_errores == 4:\n # No hay nada, salida temprana sin retry:\n print_redb('** No hay datos para el d\u00eda {}!'.format(key_day))\n return None, -2\n else: # if hay_errores < 3:\n # TODO formar datos incompletos!! (max-min con NaN's, etc.)\n data_import = {}\n if dfs_import:\n data_import[KEYS_DATA_DEM[0]] = dfs_import[0].join(dfs_import[1])\n if len(dfs_maxmin) == 2:\n data_import[KEYS_DATA_DEM[1]] = dfs_maxmin[0].join(dfs_maxmin[1])\n elif dfs_maxmin:\n data_import[KEYS_DATA_DEM[1]] = dfs_maxmin[0]\n if not data_import:\n print_err('D\u00cdA: {} -> # ERRORES: {}'.format(key_day, hay_errores))\n return None, -2\n return data_import, 0", "response": "Procesa los datos descargados en JSON."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflag all inner classes of obj as well as any child classes of obj.", "response": "def flag_inner_classes(obj):\n \"\"\"\n Mutates any attributes on ``obj`` which are classes, with link to ``obj``.\n\n Adds a convenience accessor which instantiates ``obj`` and then calls its\n ``setup`` method.\n\n Recurses on those objects as well.\n \"\"\"\n for tup in class_members(obj):\n tup[1]._parent = obj\n tup[1]._parent_inst = None\n tup[1].__getattr__ = my_getattr\n flag_inner_classes(tup[1])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef autohide(obj):\n # Members on obj\n for name, item in six.iteritems(vars(obj)):\n if callable(item) and name in ('setup', 'teardown'):\n item = hide(item)\n # Recurse into class members\n for name, subclass in class_members(obj):\n autohide(subclass)", "response": "Auto hide setup and teardown methods recursively."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreplacing sys.std(out|err) with a wrapper during execution, restored after. In addition, a new combined-streams output (another wrapper) will appear at ``sys.stdall``. This stream will resemble what a user sees at a terminal, i.e. both out/err streams intermingled.", "response": "def trap(func):\n \"\"\"\n Replace sys.std(out|err) with a wrapper during execution, restored after.\n\n In addition, a new combined-streams output (another wrapper) will appear at\n ``sys.stdall``. This stream will resemble what a user sees at a terminal,\n i.e. both out/err streams intermingled.\n \"\"\"\n @wraps(func)\n def wrapper(*args, **kwargs):\n # Use another CarbonCopy even though we're not cc'ing; for our \"write\n # bytes, return strings on py3\" behavior. Meh.\n sys.stdall = CarbonCopy()\n my_stdout, sys.stdout = sys.stdout, CarbonCopy(cc=sys.stdall)\n my_stderr, sys.stderr = sys.stderr, CarbonCopy(cc=sys.stdall)\n try:\n return func(*args, **kwargs)\n finally:\n sys.stdout = my_stdout\n sys.stderr = my_stderr\n del sys.stdall\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pvpc_url_dia(dt_day):\n if type(dt_day) is str:\n return SERVER + '/archives/70/download_json?locale=es' + '&date=' + dt_day\n else:\n return SERVER + '/archives/70/download_json?locale=es' + '&date=' + dt_day.date().isoformat()", "response": "Obtiene la url de descarga de los datos de PVPC de un d\u00eda concreto."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pvpc_calc_tcu_cp_feu_d(df, verbose=True, convert_kwh=True):\n if 'TCU' + TARIFAS[0] not in df.columns:\n # Pasa de \u20ac/MWh a \u20ac/kWh:\n if convert_kwh:\n cols_mwh = [c + t for c in COLS_PVPC for t in TARIFAS if c != 'COF']\n df[cols_mwh] = df[cols_mwh].applymap(lambda x: x / 1000.)\n # Obtiene columnas TCU, CP, precio d\u00eda\n gb_t = df.groupby(lambda x: TARIFAS[np.argmax([t in x for t in TARIFAS])], axis=1)\n for k, g in gb_t:\n if verbose:\n print('TARIFA {}'.format(k))\n print(g.head())\n\n # C\u00e1lculo de TCU\n df['TCU{}'.format(k)] = g[k] - g['TEU{}'.format(k)]\n\n # C\u00e1lculo de CP\n # cols_cp = [c + k for c in ['FOS', 'FOM', 'INT', 'PCAP', 'PMH', 'SAH']]\n cols_cp = [c + k for c in COLS_PVPC if c not in ['', 'COF', 'TEU']]\n df['CP{}'.format(k)] = g[cols_cp].sum(axis=1)\n\n # C\u00e1lculo de PERD --> No es posible as\u00ed, ya que los valores base ya vienen con PERD\n # dfs_pvpc[k]['PERD{}'.format(k)] = dfs_pvpc[k]['TCU{}'.format(k)] / dfs_pvpc[k]['CP{}'.format(k)]\n # dfs_pvpc[k]['PERD{}'.format(k)] = dfs_pvpc[k]['INT{}'.format(k)] / 1.92\n\n # C\u00e1lculo de FEU diario\n cols_k = ['TEU' + k, 'TCU' + k, 'COF' + k]\n g = df[cols_k].groupby('TEU' + k)\n pr = g.apply(lambda x: x['TCU' + k].dot(x['COF' + k]) / x['COF' + k].sum())\n pr.name = 'PD_' + k\n df = df.join(pr, on='TEU' + k, rsuffix='_r')\n df['PD_' + k] += df['TEU' + k]\n return df", "response": "Calculate TCU CP FEU D."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pvpc_procesa_datos_dia(_, response, verbose=True):\n try:\n d_data = response['PVPC']\n df = _process_json_pvpc_hourly_data(pd.DataFrame(d_data))\n return df, 0\n except Exception as e:\n if verbose:\n print('ERROR leyendo informaci\u00f3n de web: {}'.format(e))\n return None, -2", "response": "Procesa la informaci\u00f3n JSON descargada y forma el dataframe de los datos de un d\u00eda."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pvpc_data_dia(str_dia, str_dia_fin=None):\n params = {'date_fmt': DATE_FMT, 'usar_multithread': False,\n 'func_procesa_data_dia': pvpc_procesa_datos_dia, 'func_url_data_dia': pvpc_url_dia,\n 'data_extra_request': {'json_req': True, 'headers': HEADERS}}\n if str_dia_fin is not None:\n params['usar_multithread'] = True\n data, hay_errores, str_import = get_data_en_intervalo(str_dia, str_dia_fin, **params)\n else:\n data, hay_errores, str_import = get_data_en_intervalo(str_dia, str_dia, **params)\n if not hay_errores:\n return data\n else:\n return str_import", "response": "Obtiene datos de PVPC en un d\u00eda concreto o un intervalo accediendo directamente a la web."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompress the log message in order to send less bytes to the wire.", "response": "def _compress(self, input_str):\n \"\"\"\n Compress the log message in order to send less bytes to the wire.\n \"\"\"\n compressed_bits = cStringIO.StringIO()\n \n f = gzip.GzipFile(fileobj=compressed_bits, mode='wb')\n f.write(input_str)\n f.close()\n \n return compressed_bits.getvalue()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets data coefficientsicientes de perfilado para 2017.", "response": "def get_data_coeficientes_perfilado_2017(force_download=False):\n \"\"\"Extrae la informaci\u00f3n de las dos hojas del Excel proporcionado por REE\n con los perfiles iniciales para 2017.\n :param force_download: Descarga el fichero 'raw' del servidor, en vez de acudir a la copia local.\n :return: perfiles_2017, coefs_alpha_beta_gamma\n :rtype: tuple\n \"\"\"\n path_perfs = os.path.join(STORAGE_DIR, 'perfiles_consumo_2017.h5')\n\n if force_download or not os.path.exists(path_perfs):\n # Coeficientes de perfilado y demanda de referencia (1\u00aa hoja)\n cols_sheet1 = ['Mes', 'D\u00eda', 'Hora',\n 'Pa,0m,d,h', 'Pb,0m,d,h', 'Pc,0m,d,h', 'Pd,0m,d,h', 'Demanda de Referencia 2017 (MW)']\n perfs_2017 = pd.read_excel(URL_PERFILES_2017, header=None, skiprows=[0, 1], names=cols_sheet1)\n perfs_2017['ts'] = pd.DatetimeIndex(start='2017-01-01', freq='H', tz=TZ, end='2017-12-31 23:59')\n perfs_2017 = perfs_2017.set_index('ts').drop(['Mes', 'D\u00eda', 'Hora'], axis=1)\n\n # Coefs Alfa, Beta, Gamma (2\u00aa hoja):\n coefs_alpha_beta_gamma = pd.read_excel(URL_PERFILES_2017, sheetname=1)\n print('Escribiendo perfiles 2017 en disco, en {}'.format(path_perfs))\n with pd.HDFStore(path_perfs, 'w') as st:\n st.put('coefs', coefs_alpha_beta_gamma)\n st.put('perfiles', perfs_2017)\n print('HDFStore de tama\u00f1o {:.3f} KB'.format(os.path.getsize(path_perfs) / 1000))\n else:\n with pd.HDFStore(path_perfs, 'r') as st:\n coefs_alpha_beta_gamma = st['coefs']\n perfs_2017 = st['perfiles']\n return perfs_2017, coefs_alpha_beta_gamma"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_data_perfiles_estimados_2017(force_download=False):\n global DATA_PERFILES_2017\n if (DATA_PERFILES_2017 is None) or force_download:\n perf_demref_2017, _ = get_data_coeficientes_perfilado_2017(force_download=force_download)\n # Conversi\u00f3n de formato de dataframe de perfiles 2017 a finales (para uniformizar):\n cols_usar = ['Pa,0m,d,h', 'Pb,0m,d,h', 'Pc,0m,d,h', 'Pd,0m,d,h']\n perfs_2017 = perf_demref_2017[cols_usar].copy()\n perfs_2017.columns = ['COEF. PERFIL {}'.format(p) for p in 'ABCD']\n DATA_PERFILES_2017 = perfs_2017\n return perfs_2017\n return DATA_PERFILES_2017", "response": "Get the data for the 2017 - level perfiles"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd class to good list.", "response": "def registerGoodClass(self, class_):\n \"\"\"\n Internal bookkeeping to handle nested classes\n \"\"\"\n # Class itself added to \"good\" list\n self._valid_classes.append(class_)\n # Recurse into any inner classes\n for name, cls in class_members(class_):\n if self.isValidClass(cls):\n self.registerGoodClass(cls)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef isValidClass(self, class_):\n module = inspect.getmodule(class_)\n valid = (\n module in self._valid_modules\n or (\n hasattr(module, '__file__')\n and module.__file__ in self._valid_named_modules\n )\n )\n return valid and not private(class_)", "response": "Returns True if the class is valid."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef last_entry(self, data_revisar=None, key_revisar=None):\n if data_revisar is None and key_revisar is None:\n data_revisar = self.data[self.masterkey][pd.notnull(self.data[self.masterkey]['dem'])]\n super(DatosREE, self).printif('\u00daltimos valores de generaci\u00f3n y demanda:', 'info')\n super(DatosREE, self).printif(data_revisar.tail(), 'info')\n return super(DatosREE, self).last_entry(data_revisar, 'dem')\n else:\n return super(DatosREE, self).last_entry(data_revisar, key_revisar)", "response": "Return the last entry in the table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef integridad_data(self, data_integr=None, key=None):\n if data_integr is None and key is None and all(k in self.data.keys() for k in KEYS_DATA_DEM):\n assert(self.data[KEYS_DATA_DEM[0]].index.freq == FREQ_DAT_DEM\n and self.data[KEYS_DATA_DEM[0]].index.tz == self.TZ)\n if self.data[KEYS_DATA_DEM[1]] is not None:\n assert(self.data[KEYS_DATA_DEM[1]].index.freq == 'D')\n super(DatosREE, self).integridad_data(data_integr, key)", "response": "Integra los datos de la base de datos y creciente archivo."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sanitize_path(path):\n\n if path == '/': # Nothing to do, just return\n return path\n\n if path[:1] != '/':\n raise InvalidPath('The path must start with a slash')\n\n # Deduplicate slashes in path\n path = re.sub(r'/+', '/', path)\n\n # Strip trailing slashes and return\n return path.rstrip('/')", "response": "Performs sanitation of the path after validating\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _validate_schema(obj):\n\n if obj is not None and not isinstance(obj, Schema):\n raise IncompatibleSchema('Schema must be of type {0}'.format(Schema))\n\n return obj", "response": "Ensures that the passed schema instance is of a compatible type\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\njourneying route decorator Enables simple serialization, deserialization and validation of Flask routes with the help of Marshmallow. :param bp: :class:`flask.Blueprint` object :param args: args to pass along to `Blueprint.route` :param kwargs: - :strict_slashes: Enable / disable strict slashes (default False) - :validate: Enable / disable body/query validation (default True) - :_query: Unmarshal Query string into this schema - :_body: Unmarshal JSON body into this schema - :marshal_with: Serialize the output with this schema :raises: - ValidationError if the query parameters or JSON body fails validation", "response": "def route(bp, *args, **kwargs):\n \"\"\"Journey route decorator\n\n Enables simple serialization, deserialization and validation of Flask routes with the help of Marshmallow.\n\n :param bp: :class:`flask.Blueprint` object\n :param args: args to pass along to `Blueprint.route`\n :param kwargs:\n - :strict_slashes: Enable / disable strict slashes (default False)\n - :validate: Enable / disable body/query validation (default True)\n - :_query: Unmarshal Query string into this schema\n - :_body: Unmarshal JSON body into this schema\n - :marshal_with: Serialize the output with this schema\n :raises:\n - ValidationError if the query parameters or JSON body fails validation\n \"\"\"\n\n kwargs['strict_slashes'] = kwargs.pop('strict_slashes', False)\n body = _validate_schema(kwargs.pop('_body', None))\n query = _validate_schema(kwargs.pop('_query', None))\n output = _validate_schema(kwargs.pop('marshal_with', None))\n validate = kwargs.pop('validate', True)\n\n def decorator(f):\n @bp.route(*args, **kwargs)\n @wraps(f)\n def wrapper(*inner_args, **inner_kwargs):\n \"\"\"If a schema (_body and/or _query) was supplied to the route decorator, the deserialized\n :class`marshmallow.Schema` object is injected into the decorated function's kwargs.\"\"\"\n\n try:\n if query is not None:\n query.strict = validate\n url = furl(request.url)\n inner_kwargs['_query'] = query.load(data=url.args)\n\n if body is not None:\n body.strict = validate\n json_data = request.get_json()\n\n if json_data is None:\n # Set json_data to empty dict if body is empty, so it gets picked up by the validator\n json_data = {}\n\n inner_kwargs['_body'] = body.load(data=json_data)\n\n except ValidationError as err:\n return jsonify(err.messages), 422\n\n if output:\n data = output.dump(f(*inner_args, **inner_kwargs))\n return jsonify(data[0])\n\n return f(*inner_args, **inner_kwargs)\n\n return f\n\n return decorator"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef attach_bp(self, bp, description=''):\n\n if not isinstance(bp, Blueprint):\n raise InvalidBlueprint('Blueprints attached to the bundle must be of type {0}'.format(Blueprint))\n\n self.blueprints.append((bp, description))", "response": "Attaches a flask. Blueprint to the bundle\n\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the DottedRule that results from moving the dot.", "response": "def move_dot(self):\n \"\"\"Returns the DottedRule that results from moving the dot.\"\"\"\n return self.__class__(self.production, self.pos + 1, self.lookahead)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the intermediate FIRST set using symbols.", "response": "def first(self, symbols):\n \"\"\"Computes the intermediate FIRST set using symbols.\"\"\"\n ret = set()\n\n if EPSILON in symbols:\n return set([EPSILON])\n\n for symbol in symbols:\n ret |= self._first[symbol] - set([EPSILON])\n if EPSILON not in self._first[symbol]:\n break\n else:\n ret.add(EPSILON)\n\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _compute_first(self):\n for terminal in self.terminals:\n self._first[terminal].add(terminal)\n self._first[END_OF_INPUT].add(END_OF_INPUT)\n\n while True:\n changed = False\n\n for nonterminal, productions in self.nonterminals.items():\n for production in productions:\n new_first = self.first(production.rhs)\n if new_first - self._first[nonterminal]:\n self._first[nonterminal] |= new_first\n changed = True\n\n if not changed:\n break", "response": "Computes the FIRST set for every symbol in the grammar."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _compute_follow(self):\n self._follow[self.start_symbol].add(END_OF_INPUT)\n\n while True:\n changed = False\n\n for nonterminal, productions in self.nonterminals.items():\n for production in productions:\n for i, symbol in enumerate(production.rhs):\n if symbol not in self.nonterminals:\n continue\n\n first = self.first(production.rhs[i + 1:])\n new_follow = first - set([EPSILON])\n if EPSILON in first or i == (len(production.rhs) - 1):\n new_follow |= self._follow[nonterminal]\n\n if new_follow - self._follow[symbol]:\n self._follow[symbol] |= new_follow\n changed = True\n\n if not changed:\n break", "response": "Computes the FOLLOW set for every non - terminal in the grammar."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef initial_closure(self):\n first_rule = DottedRule(self.start, 0, END_OF_INPUT)\n return self.closure([first_rule])", "response": "Computes the initial closure using the START_foo production."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the next closure for the given rules based on the symbol we ve just seen.", "response": "def goto(self, rules, symbol):\n \"\"\"Computes the next closure for rules based on the symbol we got.\n\n Args:\n rules - an iterable of DottedRules\n symbol - a string denoting the symbol we've just seen\n\n Returns: frozenset of DottedRules\n \"\"\"\n return self.closure(\n {rule.move_dot() for rule in rules\n if not rule.at_end and rule.rhs[rule.pos] == symbol},\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef closure(self, rules):\n closure = set()\n\n todo = set(rules)\n while todo:\n rule = todo.pop()\n closure.add(rule)\n\n # If the dot is at the end, there's no need to process it.\n if rule.at_end:\n continue\n\n symbol = rule.rhs[rule.pos]\n for production in self.nonterminals[symbol]:\n for first in self.first(rule.rest):\n if EPSILON in production.rhs:\n # Move immediately to the end if the production\n # goes to epsilon\n new_rule = DottedRule(production, 1, first)\n else:\n new_rule = DottedRule(production, 0, first)\n\n if new_rule not in closure:\n todo.add(new_rule)\n\n return frozenset(closure)", "response": "Fills out the entire closure based on some initial dotted rules."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef closures(self):\n initial = self.initial_closure()\n closures = collections.OrderedDict()\n goto = collections.defaultdict(dict)\n\n todo = set([initial])\n while todo:\n closure = todo.pop()\n closures[closure] = closure\n\n symbols = {rule.rhs[rule.pos] for rule in closure\n if not rule.at_end}\n for symbol in symbols:\n next_closure = self.goto(closure, symbol)\n\n if next_closure in closures or next_closure in todo:\n next_closure = (closures.get(next_closure)\n or todo.get(next_closure))\n else:\n closures[next_closure] = next_closure\n todo.add(next_closure)\n\n goto[closure][symbol] = next_closure\n\n return initial, closures, goto", "response": "Computes all LR1 closures for the grammar."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes the extension for the given app.", "response": "def init_app(self, app):\n \"\"\"Initializes Journey extension\n\n :param app: App passed from constructor or directly to init_app\n :raises:\n - NoBundlesAttached if no bundles has been attached attached\n\n \"\"\"\n\n if len(self._attached_bundles) == 0:\n raise NoBundlesAttached(\"At least one bundle must be attached before initializing Journey\")\n\n for bundle in self._attached_bundles:\n processed_bundle = {\n 'path': bundle.path,\n 'description': bundle.description,\n 'blueprints': []\n }\n\n for (bp, description) in bundle.blueprints:\n # Register the BP\n blueprint = self._register_blueprint(app, bp, bundle.path,\n self.get_bp_path(bp), description)\n\n # Finally, attach the blueprints to its parent\n processed_bundle['blueprints'].append(blueprint)\n\n self._registered_bundles.append(processed_bundle)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a tuple containing endpoint path and allowed methods for each route", "response": "def routes_simple(self):\n \"\"\"Returns simple info about registered blueprints\n\n :return: Tuple containing endpoint, path and allowed methods for each route\n \"\"\"\n\n routes = []\n\n for bundle in self._registered_bundles:\n bundle_path = bundle['path']\n for blueprint in bundle['blueprints']:\n bp_path = blueprint['path']\n for child in blueprint['routes']:\n routes.append(\n (\n child['endpoint'],\n bundle_path + bp_path + child['path'],\n child['methods']\n )\n )\n\n return routes"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _bundle_exists(self, path):\n\n for attached_bundle in self._attached_bundles:\n if path == attached_bundle.path:\n return True\n\n return False", "response": "Checks if a bundle exists at the provided path"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef attach_bundle(self, bundle):\n\n if not isinstance(bundle, BlueprintBundle):\n raise IncompatibleBundle('BlueprintBundle object passed to attach_bundle must be of type {0}'\n .format(BlueprintBundle))\n elif len(bundle.blueprints) == 0:\n raise MissingBlueprints(\"Bundles must contain at least one flask.Blueprint\")\n elif self._bundle_exists(bundle.path):\n raise ConflictingPath(\"Duplicate bundle path {0}\".format(bundle.path))\n elif self._journey_path == bundle.path == '/':\n raise ConflictingPath(\"Bundle path and Journey path cannot both be {0}\".format(bundle.path))\n\n self._attached_bundles.append(bundle)", "response": "Attaches a bundle object to the journey."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _register_blueprint(self, app, bp, bundle_path, child_path, description):\n\n base_path = sanitize_path(self._journey_path + bundle_path + child_path)\n\n app.register_blueprint(bp, url_prefix=base_path)\n\n return {\n 'name': bp.name,\n 'path': child_path,\n 'import_name': bp.import_name,\n 'description': description,\n 'routes': self.get_blueprint_routes(app, base_path)\n }", "response": "Register and return info about the blueprint"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn detailed information about registered blueprint routes matching the base_path", "response": "def get_blueprint_routes(app, base_path):\n \"\"\"Returns detailed information about registered blueprint routes matching the `BlueprintBundle` path\n\n :param app: App instance to obtain rules from\n :param base_path: Base path to return detailed route info for\n :return: List of route detail dicts\n \"\"\"\n\n routes = []\n\n for child in app.url_map.iter_rules():\n if child.rule.startswith(base_path):\n relative_path = child.rule[len(base_path):]\n routes.append({\n 'path': relative_path,\n 'endpoint': child.endpoint,\n 'methods': list(child.methods)\n })\n\n return routes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the precedence of a terminal and production.", "response": "def compute_precedence(terminals, productions, precedence_levels):\n \"\"\"Computes the precedence of terminal and production.\n\n The precedence of a terminal is it's level in the PRECEDENCE tuple. For\n a production, the precedence is the right-most terminal (if it exists).\n The default precedence is DEFAULT_PREC - (LEFT, 0).\n\n Returns:\n precedence - dict[terminal | production] = (assoc, level)\n\n \"\"\"\n precedence = collections.OrderedDict()\n\n for terminal in terminals:\n precedence[terminal] = DEFAULT_PREC\n\n level_precs = range(len(precedence_levels), 0, -1)\n for i, level in zip(level_precs, precedence_levels):\n assoc = level[0]\n for symbol in level[1:]:\n precedence[symbol] = (assoc, i)\n\n for production, prec_symbol in productions:\n if prec_symbol is None:\n prod_terminals = [symbol for symbol in production.rhs\n if symbol in terminals] or [None]\n precedence[production] = precedence.get(prod_terminals[-1],\n DEFAULT_PREC)\n else:\n precedence[production] = precedence.get(prec_symbol,\n DEFAULT_PREC)\n\n return precedence"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_tables(grammar, precedence):\n ACTION = {}\n GOTO = {}\n\n labels = {}\n\n def get_label(closure):\n if closure not in labels:\n labels[closure] = len(labels)\n return labels[closure]\n\n def resolve_shift_reduce(lookahead, s_action, r_action):\n s_assoc, s_level = precedence[lookahead]\n r_assoc, r_level = precedence[r_action[1]]\n\n if s_level < r_level:\n return r_action\n elif s_level == r_level and r_assoc == LEFT:\n return r_action\n else:\n return s_action\n\n initial, closures, goto = grammar.closures()\n for closure in closures:\n label = get_label(closure)\n\n for rule in closure:\n new_action, lookahead = None, rule.lookahead\n\n if not rule.at_end:\n symbol = rule.rhs[rule.pos]\n is_terminal = symbol in grammar.terminals\n has_goto = symbol in goto[closure]\n if is_terminal and has_goto:\n next_state = get_label(goto[closure][symbol])\n new_action, lookahead = ('shift', next_state), symbol\n elif rule.production == grammar.start and rule.at_end:\n new_action = ('accept',)\n elif rule.at_end:\n new_action = ('reduce', rule.production)\n\n if new_action is None:\n continue\n\n prev_action = ACTION.get((label, lookahead))\n if prev_action is None or prev_action == new_action:\n ACTION[label, lookahead] = new_action\n else:\n types = (prev_action[0], new_action[0])\n if types == ('shift', 'reduce'):\n chosen = resolve_shift_reduce(lookahead,\n prev_action,\n new_action)\n elif types == ('reduce', 'shift'):\n chosen = resolve_shift_reduce(lookahead,\n new_action,\n prev_action)\n else:\n raise TableConflictError(prev_action, new_action)\n\n ACTION[label, lookahead] = chosen\n\n for symbol in grammar.nonterminals:\n if symbol in goto[closure]:\n GOTO[label, symbol] = get_label(goto[closure][symbol])\n\n return get_label(initial), ACTION, GOTO", "response": "Generates the ACTION and GOTO tables for the grammar."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef expr(s):\n if isinstance(s, Expr): return s\n if isnumber(s): return Expr(s)\n ## Replace the alternative spellings of operators with canonical spellings\n s = s.replace('==>', '>>').replace('<==', '<<')\n s = s.replace('<=>', '%').replace('=/=', '^')\n ## Replace a symbol or number, such as 'P' with 'Expr(\"P\")'\n s = re.sub(r'([a-zA-Z0-9_.]+)', r'Expr(\"\\1\")', s)\n ## Now eval the string. (A security hole; do not use with an adversary.)\n return eval(s, {'Expr':Expr})", "response": "Create an Expr representing a logic expression."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a set of the variables in expression s.", "response": "def variables(s):\n \"\"\"Return a set of the variables in expression s.\n >>> ppset(variables(F(x, A, y)))\n set([x, y])\n >>> ppset(variables(F(G(x), z)))\n set([x, z])\n >>> ppset(variables(expr('F(x, x) & G(x, y) & H(y, z) & R(A, z, z)')))\n set([x, y, z])\n \"\"\"\n result = set([])\n def walk(s):\n if is_variable(s):\n result.add(s)\n else:\n for arg in s.args:\n walk(arg)\n walk(s)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning True if exprs s is of the form A B... C == > D where all literals are positive.", "response": "def is_definite_clause(s):\n \"\"\"returns True for exprs s of the form A & B & ... & C ==> D,\n where all literals are positive. In clause form, this is\n ~A | ~B | ... | ~C | D, where exactly one clause is positive.\n >>> is_definite_clause(expr('Farmer(Mac)'))\n True\n >>> is_definite_clause(expr('~Farmer(Mac)'))\n False\n >>> is_definite_clause(expr('(Farmer(f) & Rabbit(r)) ==> Hates(f, r)'))\n True\n >>> is_definite_clause(expr('(Farmer(f) & ~Rabbit(r)) ==> Hates(f, r)'))\n False\n >>> is_definite_clause(expr('(Farmer(f) | Rabbit(r)) ==> Hates(f, r)'))\n False\n \"\"\"\n if is_symbol(s.op):\n return True\n elif s.op == '>>':\n antecedent, consequent = s.args\n return (is_symbol(consequent.op)\n and every(lambda arg: is_symbol(arg.op), conjuncts(antecedent)))\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the antecedents and the consequent of a definite clause.", "response": "def parse_definite_clause(s):\n \"Return the antecedents and the consequent of a definite clause.\"\n assert is_definite_clause(s)\n if is_symbol(s.op):\n return [], s\n else:\n antecedent, consequent = s.args\n return conjuncts(antecedent), consequent"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndoing kb entail the sentence alpha? Use truth tables. For propositional kb s and sentences.", "response": "def tt_entails(kb, alpha):\n \"\"\"Does kb entail the sentence alpha? Use truth tables. For propositional\n kb's and sentences. [Fig. 7.10]\n >>> tt_entails(expr('P & Q'), expr('Q'))\n True\n \"\"\"\n assert not variables(alpha)\n return tt_check_all(kb, alpha, prop_symbols(kb & alpha), {})"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tt_check_all(kb, alpha, symbols, model):\n \"Auxiliary routine to implement tt_entails.\"\n if not symbols:\n if pl_true(kb, model):\n result = pl_true(alpha, model)\n assert result in (True, False)\n return result\n else:\n return True\n else:\n P, rest = symbols[0], symbols[1:]\n return (tt_check_all(kb, alpha, rest, extend(model, P, True)) and\n tt_check_all(kb, alpha, rest, extend(model, P, False)))", "response": "Auxiliary routine to implement tt_entails."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef prop_symbols(x):\n \"Return a list of all propositional symbols in x.\"\n if not isinstance(x, Expr):\n return []\n elif is_prop_symbol(x.op):\n return [x]\n else:\n return list(set(symbol for arg in x.args\n for symbol in prop_symbols(arg)))", "response": "Return a list of all propositional symbols in x."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if the propositional logic expression is true in the model and False if it is false.", "response": "def pl_true(exp, model={}):\n \"\"\"Return True if the propositional logic expression is true in the model,\n and False if it is false. If the model does not specify the value for\n every proposition, this may return None to indicate 'not obvious';\n this may happen even when the expression is tautological.\"\"\"\n op, args = exp.op, exp.args\n if exp == TRUE:\n return True\n elif exp == FALSE:\n return False\n elif is_prop_symbol(op):\n return model.get(exp)\n elif op == '~':\n p = pl_true(args[0], model)\n if p is None: return None\n else: return not p\n elif op == '|':\n result = False\n for arg in args:\n p = pl_true(arg, model)\n if p is True: return True\n if p is None: result = None\n return result\n elif op == '&':\n result = True\n for arg in args:\n p = pl_true(arg, model)\n if p is False: return False\n if p is None: result = None\n return result\n p, q = args\n if op == '>>':\n return pl_true(~p | q, model)\n elif op == '<<':\n return pl_true(p | ~q, model)\n pt = pl_true(p, model)\n if pt is None: return None\n qt = pl_true(q, model)\n if qt is None: return None\n if op == '<=>':\n return pt == qt\n elif op == '^':\n return pt != qt\n else:\n raise ValueError, \"illegal operator in logic expression\" + str(exp)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a propositional logical sentence s to conjunctive normal form.", "response": "def to_cnf(s):\n \"\"\"Convert a propositional logical sentence s to conjunctive normal form.\n That is, to the form ((A | ~B | ...) & (B | C | ...) & ...) [p. 253]\n >>> to_cnf(\"~(B|C)\")\n (~B & ~C)\n >>> to_cnf(\"B <=> (P1|P2)\")\n ((~P1 | B) & (~P2 | B) & (P1 | P2 | ~B))\n >>> to_cnf(\"a | (b & c) | d\")\n ((b | a | d) & (c | a | d))\n >>> to_cnf(\"A & (B | (D & E))\")\n (A & (D | B) & (E | B))\n >>> to_cnf(\"A | (B | (C | (D & E)))\")\n ((D | A | B | C) & (E | A | B | C))\n \"\"\"\n if isinstance(s, str): s = expr(s)\n s = eliminate_implications(s) # Steps 1, 2 from p. 253\n s = move_not_inwards(s) # Step 3\n return distribute_and_over_or(s)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an Expr that is equivalent to s but has only & and | and ~ as logical operators.", "response": "def eliminate_implications(s):\n \"\"\"Change >>, <<, and <=> into &, |, and ~. That is, return an Expr\n that is equivalent to s, but has only &, |, and ~ as logical operators.\n >>> eliminate_implications(A >> (~B << C))\n ((~B | ~C) | ~A)\n >>> eliminate_implications(A ^ B)\n ((A & ~B) | (~A & B))\n \"\"\"\n if not s.args or is_symbol(s.op): return s ## (Atoms are unchanged.)\n args = map(eliminate_implications, s.args)\n a, b = args[0], args[-1]\n if s.op == '>>':\n return (b | ~a)\n elif s.op == '<<':\n return (a | ~b)\n elif s.op == '<=>':\n return (a | ~b) & (b | ~a)\n elif s.op == '^':\n assert len(args) == 2 ## TODO: relax this restriction\n return (a & ~b) | (~a & b)\n else:\n assert s.op in ('&', '|', '~')\n return Expr(s.op, *args)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrewrites sentence s by moving negation sign inward.", "response": "def move_not_inwards(s):\n \"\"\"Rewrite sentence s by moving negation sign inward.\n >>> move_not_inwards(~(A | B))\n (~A & ~B)\n >>> move_not_inwards(~(A & B))\n (~A | ~B)\n >>> move_not_inwards(~(~(A | ~B) | ~~C))\n ((A | ~B) & ~C)\n \"\"\"\n if s.op == '~':\n NOT = lambda b: move_not_inwards(~b)\n a = s.args[0]\n if a.op == '~': return move_not_inwards(a.args[0]) # ~~A ==> A\n if a.op =='&': return associate('|', map(NOT, a.args))\n if a.op =='|': return associate('&', map(NOT, a.args))\n return s\n elif is_symbol(s.op) or not s.args:\n return s\n else:\n return Expr(s.op, *map(move_not_inwards, s.args))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef distribute_and_over_or(s):\n if s.op == '|':\n s = associate('|', s.args)\n if s.op != '|':\n return distribute_and_over_or(s)\n if len(s.args) == 0:\n return FALSE\n if len(s.args) == 1:\n return distribute_and_over_or(s.args[0])\n conj = find_if((lambda d: d.op == '&'), s.args)\n if not conj:\n return s\n others = [a for a in s.args if a is not conj]\n rest = associate('|', others)\n return associate('&', [distribute_and_over_or(c|rest)\n for c in conj.args])\n elif s.op == '&':\n return associate('&', map(distribute_and_over_or, s.args))\n else:\n return s", "response": "Given a sentence s consisting of conjunctions and disjunctions\n of literals return an equivalent sentence in CNF."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef associate(op, args):\n args = dissociate(op, args)\n if len(args) == 0:\n return _op_identity[op]\n elif len(args) == 1:\n return args[0]\n else:\n return Expr(op, *args)", "response": "Given an associative op return an expression with the same\n meaning as Expr ( op * args"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dissociate(op, args):\n result = []\n def collect(subargs):\n for arg in subargs:\n if arg.op == op: collect(arg.args)\n else: result.append(arg)\n collect(args)\n return result", "response": "Given an associative op return a flattened list result such that Expr ( op args ) means the same as Expr ( op args )."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pl_resolution(KB, alpha):\n \"Propositional-logic resolution: say if alpha follows from KB. [Fig. 7.12]\"\n clauses = KB.clauses + conjuncts(to_cnf(~alpha))\n new = set()\n while True:\n n = len(clauses)\n pairs = [(clauses[i], clauses[j])\n for i in range(n) for j in range(i+1, n)]\n for (ci, cj) in pairs:\n resolvents = pl_resolve(ci, cj)\n if FALSE in resolvents: return True\n new = new.union(set(resolvents))\n if new.issubset(set(clauses)): return False\n for c in new:\n if c not in clauses: clauses.append(c)", "response": "Propositional - logic resolution : say if alpha follows from KB. [ Fig. 7. 12 ]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all clauses that can be obtained by resolving clauses ci and cj.", "response": "def pl_resolve(ci, cj):\n \"\"\"Return all clauses that can be obtained by resolving clauses ci and cj.\n >>> for res in pl_resolve(to_cnf(A|B|C), to_cnf(~B|~C|F)):\n ... ppset(disjuncts(res))\n set([A, C, F, ~C])\n set([A, B, F, ~B])\n \"\"\"\n clauses = []\n for di in disjuncts(ci):\n for dj in disjuncts(cj):\n if di == ~dj or ~di == dj:\n dnew = unique(removeall(di, disjuncts(ci)) +\n removeall(dj, disjuncts(cj)))\n clauses.append(associate('|', dnew))\n return clauses"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pl_fc_entails(KB, q):\n count = dict([(c, len(conjuncts(c.args[0]))) for c in KB.clauses\n if c.op == '>>'])\n inferred = DefaultDict(False)\n agenda = [s for s in KB.clauses if is_prop_symbol(s.op)]\n while agenda:\n p = agenda.pop()\n if p == q: return True\n if not inferred[p]:\n inferred[p] = True\n for c in KB.clauses_with_premise(p):\n count[c] -= 1\n if count[c] == 0:\n agenda.append(c.args[1])\n return False", "response": "Return True if a PropDefiniteKB entails symbol q."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dpll_satisfiable(s):\n clauses = conjuncts(to_cnf(s))\n symbols = prop_symbols(s)\n return dpll(clauses, symbols, {})", "response": "Check satisfiability of a propositional sentence."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nseeing if the clauses are true in a partial model.", "response": "def dpll(clauses, symbols, model):\n \"See if the clauses are true in a partial model.\"\n unknown_clauses = [] ## clauses with an unknown truth value\n for c in clauses:\n val = pl_true(c, model)\n if val == False:\n return False\n if val != True:\n unknown_clauses.append(c)\n if not unknown_clauses:\n return model\n P, value = find_pure_symbol(symbols, unknown_clauses)\n if P:\n return dpll(clauses, removeall(P, symbols), extend(model, P, value))\n P, value = find_unit_clause(clauses, model)\n if P:\n return dpll(clauses, removeall(P, symbols), extend(model, P, value))\n P, symbols = symbols[0], symbols[1:]\n return (dpll(clauses, symbols, extend(model, P, True)) or\n dpll(clauses, symbols, extend(model, P, False)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_pure_symbol(symbols, clauses):\n for s in symbols:\n found_pos, found_neg = False, False\n for c in clauses:\n if not found_pos and s in disjuncts(c): found_pos = True\n if not found_neg and ~s in disjuncts(c): found_neg = True\n if found_pos != found_neg: return s, found_pos\n return None, None", "response": "Find a symbol and its value if it appears only as a positive literal\n or only as a negative literal\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_unit_clause(clauses, model):\n for clause in clauses:\n P, value = unit_clause_assign(clause, model)\n if P: return P, value\n return None, None", "response": "Find a forced assignment if possible from a clause with only 1\n variable not bound in the model."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a single variable and value pair that makes clause true in the model if possible.", "response": "def unit_clause_assign(clause, model):\n \"\"\"Return a single variable/value pair that makes clause true in\n the model, if possible.\n >>> unit_clause_assign(A|B|C, {A:True})\n (None, None)\n >>> unit_clause_assign(B|~C, {A:True})\n (None, None)\n >>> unit_clause_assign(~A|~B, {A:True})\n (B, False)\n \"\"\"\n P, value = None, None\n for literal in disjuncts(clause):\n sym, positive = inspect_literal(literal)\n if sym in model:\n if model[sym] == positive:\n return None, None # clause already True\n elif P:\n return None, None # more than 1 unbound variable\n else:\n P, value = sym, positive\n return P, value"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef SAT_plan(init, transition, goal, t_max, SAT_solver=dpll_satisfiable):\n \"[Fig. 7.22]\"\n for t in range(t_max):\n cnf = translate_to_SAT(init, transition, goal, t)\n model = SAT_solver(cnf)\n if model is not False:\n return extract_solution(model)\n return None", "response": "Given a set of initial states and a set of goal states return the SAT plan."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nunifies expressions x y with substitution s ; return a substitution that would make x y equal or None if x and y can not be unified.", "response": "def unify(x, y, s):\n \"\"\"Unify expressions x,y with substitution s; return a substitution that\n would make x,y equal, or None if x,y can not unify. x and y can be\n variables (e.g. Expr('x')), constants, lists, or Exprs. [Fig. 9.1]\n >>> ppsubst(unify(x + y, y + C, {}))\n {x: y, y: C}\n \"\"\"\n if s is None:\n return None\n elif x == y:\n return s\n elif is_variable(x):\n return unify_var(x, y, s)\n elif is_variable(y):\n return unify_var(y, x, s)\n elif isinstance(x, Expr) and isinstance(y, Expr):\n return unify(x.args, y.args, unify(x.op, y.op, s))\n elif isinstance(x, str) or isinstance(y, str):\n return None\n elif issequence(x) and issequence(y) and len(x) == len(y):\n if not x: return s\n return unify(x[1:], y[1:], unify(x[0], y[0], s))\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef occur_check(var, x, s):\n if var == x:\n return True\n elif is_variable(x) and x in s:\n return occur_check(var, s[x], s)\n elif isinstance(x, Expr):\n return (occur_check(var, x.op, s) or\n occur_check(var, x.args, s))\n elif isinstance(x, (list, tuple)):\n return some(lambda element: occur_check(var, element, s), x)\n else:\n return False", "response": "Return true if variable var occurs anywhere in x\n or in subst ( s x )."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncopy the substitution s and extend it by setting var to val", "response": "def extend(s, var, val):\n \"\"\"Copy the substitution s and extend it by setting var to val;\n return copy.\n >>> ppsubst(extend({x: 1}, y, 2))\n {x: 1, y: 2}\n \"\"\"\n s2 = s.copy()\n s2[var] = val\n return s2"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef subst(s, x):\n if isinstance(x, list):\n return [subst(s, xi) for xi in x]\n elif isinstance(x, tuple):\n return tuple([subst(s, xi) for xi in x])\n elif not isinstance(x, Expr):\n return x\n elif is_var_symbol(x.op):\n return s.get(x, x)\n else:\n return Expr(x.op, *[subst(s, arg) for arg in x.args])", "response": "Substitute the substitution s into the expression x."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fol_fc_ask(KB, alpha):\n while True:\n new = {}\n for r in KB.clauses:\n ps, q = parse_definite_clause(standardize_variables(r))\n raise NotImplementedError", "response": "Inefficient forward chaining for first - order logic. [ Fig. 9. 3 ]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a new expression that contains all the variables in the sentence.", "response": "def standardize_variables(sentence, dic=None):\n \"\"\"Replace all the variables in sentence with new variables.\n >>> e = expr('F(a, b, c) & G(c, A, 23)')\n >>> len(variables(standardize_variables(e)))\n 3\n >>> variables(e).intersection(variables(standardize_variables(e)))\n set([])\n >>> is_variable(standardize_variables(expr('x')))\n True\n \"\"\"\n if dic is None: dic = {}\n if not isinstance(sentence, Expr):\n return sentence\n elif is_var_symbol(sentence.op):\n if sentence in dic:\n return dic[sentence]\n else:\n v = Expr('v_%d' % standardize_variables.counter.next())\n dic[sentence] = v\n return v\n else:\n return Expr(sentence.op,\n *[standardize_variables(a, dic) for a in sentence.args])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the symbolic derivative of dy and dx as an Expr.", "response": "def diff(y, x):\n \"\"\"Return the symbolic derivative, dy/dx, as an Expr.\n However, you probably want to simplify the results with simp.\n >>> diff(x * x, x)\n ((x * 1) + (x * 1))\n >>> simp(diff(x * x, x))\n (2 * x)\n \"\"\"\n if y == x: return ONE\n elif not y.args: return ZERO\n else:\n u, op, v = y.args[0], y.op, y.args[-1]\n if op == '+': return diff(u, x) + diff(v, x)\n elif op == '-' and len(args) == 1: return -diff(u, x)\n elif op == '-': return diff(u, x) - diff(v, x)\n elif op == '*': return u * diff(v, x) + v * diff(u, x)\n elif op == '/': return (v*diff(u, x) - u*diff(v, x)) / (v * v)\n elif op == '**' and isnumber(x.op):\n return (v * u ** (v - 1) * diff(u, x))\n elif op == '**': return (v * u ** (v - 1) * diff(u, x)\n + u ** v * Expr('log')(u) * diff(v, x))\n elif op == 'log': return diff(u, x) / u\n else: raise ValueError(\"Unknown op: %s in diff(%s, %s)\" % (op, y, x))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pretty_dict(d):\n return '{%s}' % ', '.join('%r: %r' % (k, v)\n for k, v in sorted(d.items(), key=repr))", "response": "Return dictionary d s repr but with the items sorted."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef retract(self, sentence):\n \"Remove the sentence's clauses from the KB.\"\n for c in conjuncts(to_cnf(sentence)):\n if c in self.clauses:\n self.clauses.remove(c)", "response": "Remove the sentence s clauses from the KB."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of the clauses in KB that have p in their premise.", "response": "def clauses_with_premise(self, p):\n \"\"\"Return a list of the clauses in KB that have p in their premise.\n This could be cached away for O(1) speed, but we'll recompute it.\"\"\"\n return [c for c in self.clauses\n if c.op == '>>' and p in conjuncts(c.args[0])]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef refresh(self):\n # `values_list('name', 'value')` doesn't work because `value` is not a\n # setting (base class) field, it's a setting value (subclass) field. So\n # we have to get real instances.\n args = [(obj.name, obj.value) for obj in self.queryset.all()]\n super(SettingDict, self).update(args)\n self.empty_cache = False", "response": "Updates the cache with setting values from the database."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef minimax_decision(state, game):\n\n player = game.to_move(state)\n\n def max_value(state):\n if game.terminal_test(state):\n return game.utility(state, player)\n v = -infinity\n for a in game.actions(state):\n v = max(v, min_value(game.result(state, a)))\n return v\n\n def min_value(state):\n if game.terminal_test(state):\n return game.utility(state, player)\n v = infinity\n for a in game.actions(state):\n v = min(v, max_value(game.result(state, a)))\n return v\n\n # Body of minimax_decision:\n return argmax(game.actions(state),\n lambda a: min_value(game.result(state, a)))", "response": "Given a state in a game calculate the best move by searching all the way to the terminal states. [ Fig. 5. 3 ]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef alphabeta_search(state, game, d=4, cutoff_test=None, eval_fn=None):\n\n player = game.to_move(state)\n\n def max_value(state, alpha, beta, depth):\n if cutoff_test(state, depth):\n return eval_fn(state)\n v = -infinity\n for a in game.actions(state):\n v = max(v, min_value(game.result(state, a),\n alpha, beta, depth+1))\n if v >= beta:\n return v\n alpha = max(alpha, v)\n return v\n\n def min_value(state, alpha, beta, depth):\n if cutoff_test(state, depth):\n return eval_fn(state)\n v = infinity\n for a in game.actions(state):\n v = min(v, max_value(game.result(state, a),\n alpha, beta, depth+1))\n if v <= alpha:\n return v\n beta = min(beta, v)\n return v\n\n # Body of alphabeta_search starts here:\n # The default test cuts off at depth d or at a terminal state\n cutoff_test = (cutoff_test or\n (lambda state,depth: depth>d or game.terminal_test(state)))\n eval_fn = eval_fn or (lambda state: game.utility(state, player))\n return argmax(game.actions(state),\n lambda a: min_value(game.result(state, a),\n -infinity, infinity, 0))", "response": "Search game to determine best action ; use alpha - beta pruning."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef play_game(game, *players):\n state = game.initial\n while True:\n for player in players:\n move = player(game, state)\n state = game.result(state, move)\n if game.terminal_test(state):\n return game.utility(state, game.to_move(game.initial))", "response": "Play an n - person move - alternateating game."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the value to player ; 1 for win - 1 for loss 0 otherwise.", "response": "def utility(self, state, player):\n \"Return the value to player; 1 for win, -1 for loss, 0 otherwise.\"\n return if_(player == 'X', state.utility, -state.utility)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compute_utility(self, board, move, player):\n \"If X wins with this move, return 1; if O return -1; else return 0.\"\n if (self.k_in_row(board, move, player, (0, 1)) or\n self.k_in_row(board, move, player, (1, 0)) or\n self.k_in_row(board, move, player, (1, -1)) or\n self.k_in_row(board, move, player, (1, 1))):\n return if_(player == 'X', +1, -1)\n else:\n return 0", "response": "If X wins with this move return 1 ; else return - 1."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef k_in_row(self, board, move, player, (delta_x, delta_y)):\n \"Return true if there is a line through move on board for player.\"\n x, y = move\n n = 0 # n is number of moves in row\n while board.get((x, y)) == player:\n n += 1\n x, y = x + delta_x, y + delta_y\n x, y = move\n while board.get((x, y)) == player:\n n += 1\n x, y = x - delta_x, y - delta_y\n n -= 1 # Because we counted move itself twice\n return n >= self.k", "response": "Return true if there is a line through move on board for player."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(x, **entries):\n if isinstance(x, dict):\n x.update(entries)\n else:\n x.__dict__.update(entries)\n return x", "response": "Update a dict or object with slots according to entries dict.\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a copy of seq with all occurences of item removed.", "response": "def removeall(item, seq):\n \"\"\"Return a copy of seq (or string) with all occurences of item removed.\n >>> removeall(3, [1, 2, 3, 3, 2, 1, 3])\n [1, 2, 2, 1]\n >>> removeall(4, [1, 2, 3])\n [1, 2, 3]\n \"\"\"\n if isinstance(seq, str):\n return seq.replace(item, '')\n else:\n return [x for x in seq if x != item]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncounting the number of elements of seq for which the predicate is true.", "response": "def count_if(predicate, seq):\n \"\"\"Count the number of elements of seq for which the predicate is true.\n >>> count_if(callable, [42, None, max, min])\n 2\n \"\"\"\n f = lambda count, x: count + (not not predicate(x))\n return reduce(f, seq, 0)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if some element of seq satisfies predicate.", "response": "def some(predicate, seq):\n \"\"\"If some element x of seq satisfies predicate(x), return predicate(x).\n >>> some(callable, [min, 3])\n 1\n >>> some(callable, [2, 3])\n 0\n \"\"\"\n for x in seq:\n px = predicate(x)\n if px: return px\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef argmin(seq, fn):\n best = seq[0]; best_score = fn(best)\n for x in seq:\n x_score = fn(x)\n if x_score < best_score:\n best, best_score = x, x_score\n return best", "response": "Return an element with lowest fn ( seq [ i ] ) score ; tie goes to first one."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef argmin_list(seq, fn):\n best_score, best = fn(seq[0]), []\n for x in seq:\n x_score = fn(x)\n if x_score < best_score:\n best, best_score = [x], x_score\n elif x_score == best_score:\n best.append(x)\n return best", "response": "Return a list of elements of seq with the lowest fn ( seq [ i ] scores."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef argmin_random_tie(seq, fn):\n best_score = fn(seq[0]); n = 0\n for x in seq:\n x_score = fn(x)\n if x_score < best_score:\n best, best_score = x, x_score; n = 1\n elif x_score == best_score:\n n += 1\n if random.randrange(n) == 0:\n best = x\n return best", "response": "Return an element with lowest fn ( seq [ i ] score ; break ties at random."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef histogram(values, mode=0, bin_function=None):\n if bin_function: values = map(bin_function, values)\n bins = {}\n for val in values:\n bins[val] = bins.get(val, 0) + 1\n if mode:\n return sorted(bins.items(), key=lambda x: (x[1],x[0]), reverse=True)\n else:\n return sorted(bins.items())", "response": "Return a list of pairs summarizing the input values."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the median value of the given list of values.", "response": "def median(values):\n \"\"\"Return the middle value, when the values are sorted.\n If there are an odd number of elements, try to average the middle two.\n If they can't be averaged (e.g. they are strings), choose one at random.\n >>> median([10, 100, 11])\n 11\n >>> median([1, 2, 3, 4])\n 2.5\n \"\"\"\n n = len(values)\n values = sorted(values)\n if n % 2 == 1:\n return values[n/2]\n else:\n middle2 = values[(n/2)-1:(n/2)+1]\n try:\n return mean(middle2)\n except TypeError:\n return random.choice(middle2)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the sum of the element - wise product of vectors x and y.", "response": "def dotproduct(X, Y):\n \"\"\"Return the sum of the element-wise product of vectors x and y.\n >>> dotproduct([1, 2, 3], [1000, 100, 10])\n 1230\n \"\"\"\n return sum([x * y for x, y in zip(X, Y)])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npicks n samples from seq at random with replacement with the corresponding probability of each element in proportion to its corresponding weight.", "response": "def weighted_sample_with_replacement(seq, weights, n):\n \"\"\"Pick n samples from seq at random, with replacement, with the\n probability of each element in proportion to its corresponding\n weight.\"\"\"\n sample = weighted_sampler(seq, weights)\n return [sample() for s in range(n)]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef weighted_sampler(seq, weights):\n \"Return a random-sample function that picks from seq weighted by weights.\"\n totals = []\n for w in weights:\n totals.append(w + totals[-1] if totals else w)\n return lambda: seq[bisect.bisect(totals, random.uniform(0, totals[-1]))]", "response": "Return a random - sample function that picks from seq weighted by weights."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef num_or_str(x):\n if isnumber(x): return x\n try:\n return int(x)\n except ValueError:\n try:\n return float(x)\n except ValueError:\n return str(x).strip()", "response": "Convert to a number if possible or strip it."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmultiplies each number by a constant such that the sum is 1. 0", "response": "def normalize(numbers):\n \"\"\"Multiply each number by a constant such that the sum is 1.0\n >>> normalize([1,2,1])\n [0.25, 0.5, 0.25]\n \"\"\"\n total = float(sum(numbers))\n return [n / total for n in numbers]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef distance((ax, ay), (bx, by)):\n \"The distance between two (x, y) points.\"\n return math.hypot((ax - bx), (ay - by))", "response": "The distance between two ( x y ) points."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef vector_clip(vector, lowest, highest):\n return type(vector)(map(clip, vector, lowest, highest))", "response": "Return vector except if any element is less than the corresponding\n value of lowest or more than the corresponding\n value of highest."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef printf(format, *args):\n sys.stdout.write(str(format) % args)\n return if_(args, lambda: args[-1], lambda: format)", "response": "Format args with the first argument as format string and write.\n Return the last arg or format itself."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlike C ++ and Java s if_.", "response": "def if_(test, result, alternative):\n \"\"\"Like C++ and Java's (test ? result : alternative), except\n both result and alternative are always evaluated. However, if\n either evaluates to a function, it is applied to the empty arglist,\n so you can delay execution by putting it in a lambda.\n >>> if_(2 + 2 == 4, 'ok', lambda: expensive_computation())\n 'ok'\n \"\"\"\n if test:\n if callable(result): return result()\n return result\n else:\n if callable(alternative): return alternative()\n return alternative"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntry to find some reasonable name for the object.", "response": "def name(object):\n \"Try to find some reasonable name for the object.\"\n return (getattr(object, 'name', 0) or getattr(object, '__name__', 0)\n or getattr(getattr(object, '__class__', 0), '__name__', 0)\n or str(object))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef print_table(table, header=None, sep=' ', numfmt='%g'):\n justs = [if_(isnumber(x), 'rjust', 'ljust') for x in table[0]]\n if header:\n table = [header] + table\n table = [[if_(isnumber(x), lambda: numfmt % x, lambda: x) for x in row]\n for row in table]\n maxlen = lambda seq: max(map(len, seq))\n sizes = map(maxlen, zip(*[map(str, row) for row in table]))\n for row in table:\n print sep.join(getattr(str(x), j)(size)\n for (j, size, x) in zip(justs, sizes, row))", "response": "Print a list of lists as a table so that columns line up nicely."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nopening a file based at the AIMA root directory.", "response": "def AIMAFile(components, mode='r'):\n \"Open a file based at the AIMA root directory.\"\n import utils\n dir = os.path.dirname(utils.__file__)\n return open(apply(os.path.join, [dir] + components), mode)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_csv(input, delim=','):\n lines = [line for line in input.splitlines() if line.strip()]\n return [map(num_or_str, line.split(delim)) for line in lines]", "response": "r Convert a CSV string into a list of lists."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef PluralityLearner(dataset):\n most_popular = mode([e[dataset.target] for e in dataset.examples])\n def predict(example):\n \"Always return same result: the most popular from the training set.\"\n return most_popular\n return predict", "response": "A very dumb algorithm that always pick the most popular result from the training set."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef NearestNeighborLearner(dataset, k=1):\n \"k-NearestNeighbor: the k nearest neighbors vote.\"\n def predict(example):\n \"Find the k closest, and have them vote for the best.\"\n best = heapq.nsmallest(k, ((dataset.distance(e, example), e)\n for e in dataset.examples))\n return mode(e[dataset.target] for (d, e) in best)\n return predict", "response": "k - NearestNeighbor : the k nearest neighbors vote."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef information_content(values):\n \"Number of bits to represent the probability distribution in values.\"\n probabilities = normalize(removeall(0, values))\n return sum(-p * log2(p) for p in probabilities)", "response": "Number of bits to represent the probability distribution in values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef DecisionListLearner(dataset):\n\n def decision_list_learning(examples):\n if not examples:\n return [(True, False)]\n t, o, examples_t = find_examples(examples)\n if not t:\n raise Failure\n return [(t, o)] + decision_list_learning(examples - examples_t)\n\n def find_examples(examples):\n \"\"\"Find a set of examples that all have the same outcome under\n some test. Return a tuple of the test, outcome, and examples.\"\"\"\n unimplemented()\n\n def passes(example, test):\n \"Does the example pass the test?\"\n unimplemented()\n\n def predict(example):\n \"Predict the outcome for the first passing test.\"\n for test, outcome in predict.decision_list:\n if passes(example, test):\n return outcome\n predict.decision_list = decision_list_learning(set(dataset.examples))\n\n return predict", "response": "A decision list learner that returns a list of the examples that are in the dataset."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef NeuralNetLearner(dataset, sizes):\n\n activations = map(lambda n: [0.0 for i in range(n)], sizes)\n weights = []\n\n def predict(example):\n unimplemented()\n\n return predict", "response": "Layered feed - forward network."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef EnsembleLearner(learners):\n def train(dataset):\n predictors = [learner(dataset) for learner in learners]\n def predict(example):\n return mode(predictor(example) for predictor in predictors)\n return predict\n return train", "response": "Given a list of learning algorithms have them vote."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef AdaBoost(L, K):\n def train(dataset):\n examples, target = dataset.examples, dataset.target\n N = len(examples)\n epsilon = 1./(2*N)\n w = [1./N] * N\n h, z = [], []\n for k in range(K):\n h_k = L(dataset, w)\n h.append(h_k)\n error = sum(weight for example, weight in zip(examples, w)\n if example[target] != h_k(example))\n # Avoid divide-by-0 from either 0% or 100% error rates:\n error = clip(error, epsilon, 1-epsilon)\n for j, example in enumerate(examples):\n if example[target] == h_k(example):\n w[j] *= error / (1. - error)\n w = normalize(w)\n z.append(math.log((1. - error) / error))\n return WeightedMajority(h, z)\n return train", "response": "AdaBoost is a function that returns a weighted majority model for each of the K elements."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a predictor that takes a weighted vote.", "response": "def WeightedMajority(predictors, weights):\n \"Return a predictor that takes a weighted vote.\"\n def predict(example):\n return weighted_mode((predictor(example) for predictor in predictors),\n weights)\n return predict"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef weighted_mode(values, weights):\n totals = defaultdict(int)\n for v, w in zip(values, weights):\n totals[v] += w\n return max(totals.keys(), key=totals.get)", "response": "Return the value with the greatest total weight."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a new unweighted learner that takes just an unweighted dataset and returns a new unweighted learner that takes also a weight for each example.", "response": "def WeightedLearner(unweighted_learner):\n \"\"\"Given a learner that takes just an unweighted dataset, return\n one that takes also a weight for each example. [p. 749 footnote 14]\"\"\"\n def train(dataset, weights):\n return unweighted_learner(replicated_dataset(dataset, weights))\n return train"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncopy dataset replicating each example in proportion to its weight.", "response": "def replicated_dataset(dataset, weights, n=None):\n \"Copy dataset, replicating each example in proportion to its weight.\"\n n = n or len(dataset.examples)\n result = copy.copy(dataset)\n result.examples = weighted_replicate(dataset.examples, weights, n)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning n selections from seq with the count of each element of seq proportional to the corresponding weight.", "response": "def weighted_replicate(seq, weights, n):\n \"\"\"Return n selections from seq, with the count of each element of\n seq proportional to the corresponding weight (filling in fractions\n randomly).\n >>> weighted_replicate('ABC', [1,2,1], 4)\n ['A', 'B', 'B', 'C']\"\"\"\n assert len(seq) == len(weights)\n weights = normalize(weights)\n wholes = [int(w*n) for w in weights]\n fractions = [(w*n) % 1 for w in weights]\n return (flatten([x] * nx for x, nx in zip(seq, wholes))\n + weighted_sample_with_replacement(seq, fractions, n - sum(wholes)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cross_validation(learner, dataset, k=10, trials=1):\n if k is None:\n k = len(dataset.examples)\n if trials > 1:\n return mean([cross_validation(learner, dataset, k, trials=1)\n for t in range(trials)])\n else:\n n = len(dataset.examples)\n random.shuffle(dataset.examples)\n return mean([train_and_test(learner, dataset, i*(n/k), (i+1)*(n/k))\n for i in range(k)])", "response": "Do k - fold cross_validate and return their mean."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nleave one out cross - validation over the dataset.", "response": "def leave1out(learner, dataset):\n \"Leave one out cross-validation over the dataset.\"\n return cross_validation(learner, dataset, k=len(dataset.examples))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef SyntheticRestaurant(n=20):\n \"Generate a DataSet with n examples.\"\n def gen():\n example = map(random.choice, restaurant.values)\n example[restaurant.target] = Fig[18,2](example)\n return example\n return RestaurantDataSet([gen() for i in range(n)])", "response": "Generate a DataSet with n examples."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a DataSet with n k - bit examples of the majority problem.", "response": "def Majority(k, n):\n \"\"\"Return a DataSet with n k-bit examples of the majority problem:\n k random bits followed by a 1 if more than half the bits are 1, else 0.\"\"\"\n examples = []\n for i in range(n):\n bits = [random.choice([0, 1]) for i in range(k)]\n bits.append(int(sum(bits) > k/2))\n examples.append(bits)\n return DataSet(name=\"majority\", examples=examples)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compare(algorithms=[PluralityLearner, NaiveBayesLearner,\n NearestNeighborLearner, DecisionTreeLearner],\n datasets=[iris, orings, zoo, restaurant, SyntheticRestaurant(20),\n Majority(7, 100), Parity(7, 100), Xor(100)],\n k=10, trials=1):\n \"\"\"Compare various learners on various datasets using cross-validation.\n Print results as a table.\"\"\"\n print_table([[a.__name__.replace('Learner','')] +\n [cross_validation(a, d, k, trials) for d in datasets]\n for a in algorithms],\n header=[''] + [d.name[0:7] for d in datasets], numfmt='%.2f')", "response": "Compare various learners on various datasets using cross - validation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setproblem(self, target, inputs=None, exclude=()):\n self.target = self.attrnum(target)\n exclude = map(self.attrnum, exclude)\n if inputs:\n self.inputs = removeall(self.target, inputs)\n else:\n self.inputs = [a for a in self.attrs\n if a != self.target and a not in exclude]\n if not self.values:\n self.values = map(unique, zip(*self.examples))\n self.check_me()", "response": "Set or change the target and inputs."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_me(self):\n \"Check that my fields make sense.\"\n assert len(self.attrnames) == len(self.attrs)\n assert self.target in self.attrs\n assert self.target not in self.inputs\n assert set(self.inputs).issubset(set(self.attrs))\n map(self.check_example, self.examples)", "response": "Check that my fields make sense."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds an example to the list of examples checking it first.", "response": "def add_example(self, example):\n \"Add an example to the list of examples, checking it first.\"\n self.check_example(example)\n self.examples.append(example)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef check_example(self, example):\n \"Raise ValueError if example has any invalid values.\"\n if self.values:\n for a in self.attrs:\n if example[a] not in self.values[a]:\n raise ValueError('Bad value %s for attribute %s in %s' %\n (example[a], self.attrnames[a], example))", "response": "Raise ValueError if example has any invalid values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef attrnum(self, attr):\n \"Returns the number used for attr, which can be a name, or -n .. n-1.\"\n if attr < 0:\n return len(self.attrs) + attr\n elif isinstance(attr, str):\n return self.attrnames.index(attr)\n else:\n return attr", "response": "Returns the number used for attr which can be a name or - n.. n - 1."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a copy of example with non - input attributes replaced by None.", "response": "def sanitize(self, example):\n \"Return a copy of example, with non-input attributes replaced by None.\"\n return [attr_i if i in self.inputs else None\n for i, attr_i in enumerate(example)]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add(self, o):\n \"Add an observation o to the distribution.\"\n self.smooth_for(o)\n self.dictionary[o] += 1\n self.n_obs += 1\n self.sampler = None", "response": "Add an observation o to the distribution."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nincludes o among the possible observations whether or not it s observed yet.", "response": "def smooth_for(self, o):\n \"\"\"Include o among the possible observations, whether or not\n it's been observed yet.\"\"\"\n if o not in self.dictionary:\n self.dictionary[o] = self.default\n self.n_obs += self.default\n self.sampler = None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns ( count obs ) tuples for the n most frequent observations.", "response": "def top(self, n):\n \"Return (count, obs) tuples for the n most frequent observations.\"\n return heapq.nlargest(n, [(v, k) for (k, v) in self.dictionary.items()])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sample(self):\n \"Return a random sample from the distribution.\"\n if self.sampler is None:\n self.sampler = weighted_sampler(self.dictionary.keys(),\n self.dictionary.values())\n return self.sampler()", "response": "Return a random sample from the distribution."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef AC3(csp, queue=None, removals=None):\n if queue is None:\n queue = [(Xi, Xk) for Xi in csp.vars for Xk in csp.neighbors[Xi]]\n csp.support_pruning()\n while queue:\n (Xi, Xj) = queue.pop()\n if revise(csp, Xi, Xj, removals):\n if not csp.curr_domains[Xi]:\n return False\n for Xk in csp.neighbors[Xi]:\n if Xk != Xi:\n queue.append((Xk, Xi))\n return True", "response": "Fig. 6. 3. 1. 3. 1. 2. 2."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef revise(csp, Xi, Xj, removals):\n \"Return true if we remove a value.\"\n revised = False\n for x in csp.curr_domains[Xi][:]:\n # If Xi=x conflicts with Xj=y for every possible y, eliminate Xi=x\n if every(lambda y: not csp.constraints(Xi, x, Xj, y),\n csp.curr_domains[Xj]):\n csp.prune(Xi, x, removals)\n revised = True\n return revised", "response": "Return true if we remove a value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef mrv(assignment, csp):\n \"Minimum-remaining-values heuristic.\"\n return argmin_random_tie(\n [v for v in csp.vars if v not in assignment],\n lambda var: num_legal_values(csp, var, assignment))", "response": "Minimum - remaining - values heuristic."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lcv(var, assignment, csp):\n \"Least-constraining-values heuristic.\"\n return sorted(csp.choices(var),\n key=lambda val: csp.nconflicts(var, val, assignment))", "response": "Least - contraining - values heuristic."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef forward_checking(csp, var, value, assignment, removals):\n \"Prune neighbor values inconsistent with var=value.\"\n for B in csp.neighbors[var]:\n if B not in assignment:\n for b in csp.curr_domains[B][:]:\n if not csp.constraints(var, value, B, b):\n csp.prune(B, b, removals)\n if not csp.curr_domains[B]:\n return False\n return True", "response": "Prune neighbor values inconsistent with var = value."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef min_conflicts(csp, max_steps=100000):\n # Generate a complete assignment for all vars (probably with conflicts)\n csp.current = current = {}\n for var in csp.vars:\n val = min_conflicts_value(csp, var, current)\n csp.assign(var, val, current)\n # Now repeatedly choose a random conflicted variable and change it\n for i in range(max_steps):\n conflicted = csp.conflicted_vars(current)\n if not conflicted:\n return current\n var = random.choice(conflicted)\n val = min_conflicts_value(csp, var, current)\n csp.assign(var, val, current)\n return None", "response": "Solve a CSP by stochastic hillclimbing on the number of conflicts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the value that will give var the least number of conflicts.", "response": "def min_conflicts_value(csp, var, current):\n \"\"\"Return the value that will give var the least number of conflicts.\n If there is a tie, choose at random.\"\"\"\n return argmin_random_tie(csp.domains[var],\n lambda val: csp.nconflicts(var, val, current))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef MapColoringCSP(colors, neighbors):\n if isinstance(neighbors, str):\n neighbors = parse_neighbors(neighbors)\n return CSP(neighbors.keys(), UniversalDict(colors), neighbors,\n different_values_constraint)", "response": "Make a CSP for the problem of coloring a map with different colors."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a string of the form X Y Z ; Y Z ; Z ; X Y Z ; Y Z ; Z ; X Y Z ; Y Z ; X Y Z ; Y Z Z.", "response": "def parse_neighbors(neighbors, vars=[]):\n \"\"\"Convert a string of the form 'X: Y Z; Y: Z' into a dict mapping\n regions to neighbors. The syntax is a region name followed by a ':'\n followed by zero or more region names, followed by ';', repeated for\n each region name. If you say 'X: Y' you don't need 'Y: X'.\n >>> parse_neighbors('X: Y Z; Y: Z')\n {'Y': ['X', 'Z'], 'X': ['Y', 'Z'], 'Z': ['X', 'Y']}\n \"\"\"\n dict = DefaultDict([])\n for var in vars:\n dict[var] = []\n specs = [spec.split(':') for spec in neighbors.split(';')]\n for (A, Aneighbors) in specs:\n A = A.strip()\n dict.setdefault(A, [])\n for B in Aneighbors.split():\n dict[A].append(B)\n dict[B].append(A)\n return dict"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef queen_constraint(A, a, B, b):\n return A == B or (a != b and A + a != B + b and A - a != B - b)", "response": "Constraint is satisfied if A and B are not the same variable and are not in the same row down diagonal or up diagonal."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Zebra():\n \"Return an instance of the Zebra Puzzle.\"\n Colors = 'Red Yellow Blue Green Ivory'.split()\n Pets = 'Dog Fox Snails Horse Zebra'.split()\n Drinks = 'OJ Tea Coffee Milk Water'.split()\n Countries = 'Englishman Spaniard Norwegian Ukranian Japanese'.split()\n Smokes = 'Kools Chesterfields Winston LuckyStrike Parliaments'.split()\n vars = Colors + Pets + Drinks + Countries + Smokes\n domains = {}\n for var in vars:\n domains[var] = range(1, 6)\n domains['Norwegian'] = [1]\n domains['Milk'] = [3]\n neighbors = parse_neighbors(\"\"\"Englishman: Red;\n Spaniard: Dog; Kools: Yellow; Chesterfields: Fox;\n Norwegian: Blue; Winston: Snails; LuckyStrike: OJ;\n Ukranian: Tea; Japanese: Parliaments; Kools: Horse;\n Coffee: Green; Green: Ivory\"\"\", vars)\n for type in [Colors, Pets, Drinks, Countries, Smokes]:\n for A in type:\n for B in type:\n if A != B:\n if B not in neighbors[A]: neighbors[A].append(B)\n if A not in neighbors[B]: neighbors[B].append(A)\n def zebra_constraint(A, a, B, b, recurse=0):\n same = (a == b)\n next_to = abs(a - b) == 1\n if A == 'Englishman' and B == 'Red': return same\n if A == 'Spaniard' and B == 'Dog': return same\n if A == 'Chesterfields' and B == 'Fox': return next_to\n if A == 'Norwegian' and B == 'Blue': return next_to\n if A == 'Kools' and B == 'Yellow': return same\n if A == 'Winston' and B == 'Snails': return same\n if A == 'LuckyStrike' and B == 'OJ': return same\n if A == 'Ukranian' and B == 'Tea': return same\n if A == 'Japanese' and B == 'Parliaments': return same\n if A == 'Kools' and B == 'Horse': return next_to\n if A == 'Coffee' and B == 'Green': return same\n if A == 'Green' and B == 'Ivory': return (a - 1) == b\n if recurse == 0: return zebra_constraint(B, b, A, a, 1)\n if ((A in Colors and B in Colors) or\n (A in Pets and B in Pets) or\n (A in Drinks and B in Drinks) or\n (A in Countries and B in Countries) or\n (A in Smokes and B in Smokes)): return not same\n raise 'error'\n return CSP(vars, domains, neighbors, zebra_constraint)", "response": "Return an instance of the Zebra Puzzle."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef assign(self, var, val, assignment):\n \"Add {var: val} to assignment; Discard the old value if any.\"\n assignment[var] = val\n self.nassigns += 1", "response": "Add { var : val to assignment ; Discard the old value if any."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the number of conflicts var = val has with other variables.", "response": "def nconflicts(self, var, val, assignment):\n \"Return the number of conflicts var=val has with other variables.\"\n # Subclasses may implement this more efficiently\n def conflict(var2):\n return (var2 in assignment\n and not self.constraints(var, val, var2, assignment[var2]))\n return count_if(conflict, self.neighbors[var])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef actions(self, state):\n if len(state) == len(self.vars):\n return []\n else:\n assignment = dict(state)\n var = find_if(lambda v: v not in assignment, self.vars)\n return [(var, val) for val in self.domains[var]\n if self.nconflicts(var, val, assignment) == 0]", "response": "Return a list of applicable actions for nonconflicting\n assignments to an unassigned variable."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes sure we can prune values from domains.", "response": "def support_pruning(self):\n \"\"\"Make sure we can prune values from domains. (We want to pay\n for this only if we use it.)\"\"\"\n if self.curr_domains is None:\n self.curr_domains = dict((v, list(self.domains[v]))\n for v in self.vars)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef suppose(self, var, value):\n \"Start accumulating inferences from assuming var=value.\"\n self.support_pruning()\n removals = [(var, a) for a in self.curr_domains[var] if a != value]\n self.curr_domains[var] = [value]\n return removals", "response": "Start accumulating inferences from assuming var = value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef prune(self, var, value, removals):\n \"Rule out var=value.\"\n self.curr_domains[var].remove(value)\n if removals is not None: removals.append((var, value))", "response": "Rule out var = value."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the partial assignment implied by the current inferences.", "response": "def infer_assignment(self):\n \"Return the partial assignment implied by the current inferences.\"\n self.support_pruning()\n return dict((v, self.curr_domains[v][0])\n for v in self.vars if 1 == len(self.curr_domains[v]))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nundoes a supposition and all inferences from it.", "response": "def restore(self, removals):\n \"Undo a supposition and all inferences from it.\"\n for B, b in removals:\n self.curr_domains[B].append(b)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of variables in current assignment that are in conflict", "response": "def conflicted_vars(self, current):\n \"Return a list of variables in current assignment that are in conflict\"\n return [var for var in self.vars\n if self.nconflicts(var, current[var], current) > 0]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nassigns var and keep track of conflicts.", "response": "def assign(self, var, val, assignment):\n \"Assign var, and keep track of conflicts.\"\n oldval = assignment.get(var, None)\n if val != oldval:\n if oldval is not None: # Remove old val if there was one\n self.record_conflict(assignment, var, oldval, -1)\n self.record_conflict(assignment, var, val, +1)\n CSP.assign(self, var, val, assignment)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef unassign(self, var, assignment):\n \"Remove var from assignment (if it is there) and track conflicts.\"\n if var in assignment:\n self.record_conflict(assignment, var, assignment[var], -1)\n CSP.unassign(self, var, assignment)", "response": "Remove var from assignment ( if it is there and track conflicts."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_conflict(self, assignment, var, val, delta):\n \"Record conflicts caused by addition or deletion of a Queen.\"\n n = len(self.vars)\n self.rows[val] += delta\n self.downs[var + val] += delta\n self.ups[var - val + n - 1] += delta", "response": "Record conflicts caused by addition or deletion of a Queen."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprint the queens and the nconflicts values ( for debugging.", "response": "def display(self, assignment):\n \"Print the queens and the nconflicts values (for debugging).\"\n n = len(self.vars)\n for val in range(n):\n for var in range(n):\n if assignment.get(var,'') == val: ch = 'Q'\n elif (var+val) % 2 == 0: ch = '.'\n else: ch = '-'\n print ch,\n print ' ',\n for var in range(n):\n if assignment.get(var,'') == val: ch = '*'\n else: ch = ' '\n print str(self.nconflicts(var, val, assignment))+ch,\n print"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind the best segmentation of the string of characters given the UnigramTextModel P.", "response": "def viterbi_segment(text, P):\n \"\"\"Find the best segmentation of the string of characters, given the\n UnigramTextModel P.\"\"\"\n # best[i] = best probability for text[0:i]\n # words[i] = best word ending at position i\n n = len(text)\n words = [''] + list(text)\n best = [1.0] + [0.0] * n\n ## Fill in the vectors best, words via dynamic programming\n for i in range(n+1):\n for j in range(0, i):\n w = text[j:i]\n if P[w] * best[i - len(w)] >= best[i]:\n best[i] = P[w] * best[i - len(w)]\n words[i] = w\n ## Now recover the sequence of best words\n sequence = []; i = len(words)-1\n while i > 0:\n sequence[0:0] = [words[i]]\n i = i - len(words[i])\n ## Return sequence of best words and overall probability\n return sequence, best[-1]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef encode(plaintext, code):\n \"Encodes text, using a code which is a permutation of the alphabet.\"\n from string import maketrans\n trans = maketrans(alphabet + alphabet.upper(), code + code.upper())\n return plaintext.translate(trans)", "response": "Encodes text using a code which is a permutation of the alphabet."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncount 1 for P w1... wn and for P wn - 1... wn - 1", "response": "def add(self, ngram):\n \"\"\"Count 1 for P[(w1, ..., wn)] and for P(wn | (w1, ..., wn-1)\"\"\"\n CountingProbDist.add(self, ngram)\n self.cond_prob[ngram[:-1]].add(ngram[-1])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd each of the tuple words [ i : i + n using a sliding window.", "response": "def add_sequence(self, words):\n \"\"\"Add each of the tuple words[i:i+n], using a sliding window.\n Prefix some copies of the empty word, '', to make the start work.\"\"\"\n n = self.n\n words = ['',] * (n-1) + words\n for i in range(len(words)-n):\n self.add(tuple(words[i:i+n]))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds up a random sample of text nwords words long using the conditional probability given the n - 1 preceding words.", "response": "def samples(self, nwords):\n \"\"\"Build up a random sample of text nwords words long, using\n the conditional probability given the n-1 preceding words.\"\"\"\n n = self.n\n nminus1gram = ('',) * (n-1)\n output = []\n for i in range(nwords):\n if nminus1gram not in self.cond_prob:\n nminus1gram = ('',) * (n-1) # Cannot continue, so restart.\n wn = self.cond_prob[nminus1gram].sample()\n output.append(wn)\n nminus1gram = nminus1gram[1:] + (wn,)\n return ' '.join(output)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nindexing a whole collection of files.", "response": "def index_collection(self, filenames):\n \"Index a whole collection of files.\"\n for filename in filenames:\n self.index_document(open(filename).read(), filename)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef index_document(self, text, url):\n \"Index the text of a document.\"\n ## For now, use first line for title\n title = text[:text.index('\\n')].strip()\n docwords = words(text)\n docid = len(self.documents)\n self.documents.append(Document(title, url, len(docwords)))\n for word in docwords:\n if word not in self.stopwords:\n self.index[word][docid] += 1", "response": "Index the text of a document."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef query(self, query_text, n=10):\n if query_text.startswith(\"learn:\"):\n doctext = os.popen(query_text[len(\"learn:\"):], 'r').read()\n self.index_document(doctext, query_text)\n return []\n qwords = [w for w in words(query_text) if w not in self.stopwords]\n shortest = argmin(qwords, lambda w: len(self.index[w]))\n docs = self.index[shortest]\n results = [(sum([self.score(w, d) for w in qwords]), d) for d in docs]\n results.sort(); results.reverse()\n return results[:n]", "response": "Query the database for the best matches."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute a score for this word on this docid.", "response": "def score(self, word, docid):\n \"Compute a score for this word on this docid.\"\n ## There are many options; here we take a very simple approach\n return (math.log(1 + self.index[word][docid])\n / math.log(1 + self.documents[docid].nwords))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef present(self, results):\n \"Present the results as a list.\"\n for (score, d) in results:\n doc = self.documents[d]\n print (\"%5.2f|%25s | %s\"\n % (100 * score, doc.url, doc.title[:45].expandtabs()))", "response": "Present the results as a list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget results for the query and present them.", "response": "def present_results(self, query_text, n=10):\n \"Get results for the query and present them.\"\n self.present(self.query(query_text, n))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef score(self, plaintext):\n \"Return a score for text based on how common letters pairs are.\"\n s = 1.0\n for bi in bigrams(plaintext):\n s = s * self.P2[bi]\n return s", "response": "Return a score for text based on how common letters pairs are."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch for a decoding of the ciphertext.", "response": "def decode(self, ciphertext):\n \"Search for a decoding of the ciphertext.\"\n self.ciphertext = ciphertext\n problem = PermutationDecoderProblem(decoder=self)\n return search.best_first_tree_search(\n problem, lambda node: self.score(node.state))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef score(self, code):\n text = permutation_decode(self.ciphertext, code)\n logP = (sum([log(self.Pwords[word]) for word in words(text)]) +\n sum([log(self.P1[c]) for c in text]) +\n sum([log(self.P2[b]) for b in bigrams(text)]))\n return exp(logP)", "response": "Score the code with the unigram and bigram words."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_value(self, context, default):\n if default is None:\n settings = self.setting_model.objects.as_dict()\n else:\n settings = self.setting_model.objects.as_dict(default=default)\n return settings", "response": "Returns a dictionary representation of the object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_value(self, context, name, default):\n settings = self.setting_model.objects.filter(name=name)\n if default is None:\n settings = settings.as_dict()\n else:\n settings = settings.as_dict(default=default)\n value = settings[name]\n return value", "response": "Returns the value of the named setting."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the value of the named setting.", "response": "def render_tag(self, context, name, nodelist):\n \"\"\"\n Returns the value of the named setting.\n \"\"\"\n # Use `try` and `except` instead of `setdefault()` so we can skip\n # rendering the nodelist when the setting already exists.\n settings = self.setting_model.objects.filter(name=name).as_dict()\n try:\n value = settings[name]\n except KeyError:\n value = settings[name] = nodelist.render(context)\n return value"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef value_iteration(mdp, epsilon=0.001):\n \"Solving an MDP by value iteration. [Fig. 17.4]\"\n U1 = dict([(s, 0) for s in mdp.states])\n R, T, gamma = mdp.R, mdp.T, mdp.gamma\n while True:\n U = U1.copy()\n delta = 0\n for s in mdp.states:\n U1[s] = R(s) + gamma * max([sum([p * U[s1] for (p, s1) in T(s, a)])\n for a in mdp.actions(s)])\n delta = max(delta, abs(U1[s] - U[s]))\n if delta < epsilon * (1 - gamma) / gamma:\n return U", "response": "Solving an MDP by value iteration. [ Fig. 17. 4 ]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives an MDP and a utility function U determine the best policy for the given MDP and U.", "response": "def best_policy(mdp, U):\n \"\"\"Given an MDP and a utility function U, determine the best policy,\n as a mapping from state to action. (Equation 17.4)\"\"\"\n pi = {}\n for s in mdp.states:\n pi[s] = argmax(mdp.actions(s), lambda a:expected_utility(a, s, U, mdp))\n return pi"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef expected_utility(a, s, U, mdp):\n \"The expected utility of doing a in state s, according to the MDP and U.\"\n return sum([p * U[s1] for (p, s1) in mdp.T(s, a)])", "response": "The expected utility of doing a in state s according to the MDP and U."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsolving an MDP by policy iteration [ Fig. 17. 7 ]", "response": "def policy_iteration(mdp):\n \"Solve an MDP by policy iteration [Fig. 17.7]\"\n U = dict([(s, 0) for s in mdp.states])\n pi = dict([(s, random.choice(mdp.actions(s))) for s in mdp.states])\n while True:\n U = policy_evaluation(pi, U, mdp)\n unchanged = True\n for s in mdp.states:\n a = argmax(mdp.actions(s), lambda a: expected_utility(a,s,U,mdp))\n if a != pi[s]:\n pi[s] = a\n unchanged = False\n if unchanged:\n return pi"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef policy_evaluation(pi, U, mdp, k=20):\n R, T, gamma = mdp.R, mdp.T, mdp.gamma\n for i in range(k):\n for s in mdp.states:\n U[s] = R(s) + gamma * sum([p * U[s1] for (p, s1) in T(s, pi[s])])\n return U", "response": "Return an updated utility mapping U from each state in the MDP to its\n utility using an approximation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef go(self, state, direction):\n \"Return the state that results from going in this direction.\"\n state1 = vector_add(state, direction)\n return if_(state1 in self.states, state1, state)", "response": "Return the state that results from going in this direction."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_grid(self, mapping):\n return list(reversed([[mapping.get((x,y), None)\n for x in range(self.cols)]\n for y in range(self.rows)]))", "response": "Convert a mapping from x y to v into a list of grids."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_dict(self, default=None):\n settings = SettingDict(queryset=self, default=default)\n return settings", "response": "Returns a dictionary representation of this queryset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate and returns an object of the appropriate type for value.", "response": "def create(self, name, value):\n \"\"\"\n Creates and returns an object of the appropriate type for ``value``.\n \"\"\"\n if value is None:\n raise ValueError('Setting value cannot be `None`.')\n model = Setting.get_model_for_value(value)\n # Call `create()` method on the super class to avoid recursion.\n obj = super(SettingQuerySet, model.objects.all()) \\\n .create(name=name, value=value)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the model that is compatible with the type of value.", "response": "def get_model_for_value(cls, value):\n \"\"\"\n Iterates through setting value subclasses, returning one that is\n compatible with the type of ``value``. Calls ``is_compatible()`` on\n each subclass.\n \"\"\"\n for related_object in get_all_related_objects(cls._meta):\n model = getattr(related_object, 'related_model', related_object.model)\n if issubclass(model, cls):\n if model.is_compatible(value):\n return model\n raise ValueError(\n 'No compatible `SettingValueModel` subclass for %r' % value)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_compatible(cls, value):\n if not hasattr(cls, 'value_type'):\n raise NotImplementedError(\n 'You must define a `value_type` attribute or override the '\n '`is_compatible()` method on `SettingValueModel` subclasses.')\n return isinstance(value, cls.value_type)", "response": "Checks if value is an instance of cls. value_type."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsearching through the successors of a problem to find a goal.", "response": "def tree_search(problem, frontier):\n \"\"\"Search through the successors of a problem to find a goal.\n The argument frontier should be an empty queue.\n Don't worry about repeated paths to a state. [Fig. 3.7]\"\"\"\n frontier.append(Node(problem.initial))\n while frontier:\n node = frontier.pop()\n if problem.goal_test(node.state):\n return node\n frontier.extend(node.expand(problem))\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsearch through the successors of a problem to find a goal.", "response": "def graph_search(problem, frontier):\n \"\"\"Search through the successors of a problem to find a goal.\n The argument frontier should be an empty queue.\n If two paths reach a state, only use the first one. [Fig. 3.7]\"\"\"\n frontier.append(Node(problem.initial))\n explored = set()\n while frontier:\n node = frontier.pop()\n if problem.goal_test(node.state):\n return node\n explored.add(node.state)\n frontier.extend(child for child in node.expand(problem)\n if child.state not in explored\n and child not in frontier)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsearches the nodes with the lowest f scores first.", "response": "def best_first_graph_search(problem, f):\n \"\"\"Search the nodes with the lowest f scores first.\n You specify the function f(node) that you want to minimize; for example,\n if f is a heuristic estimate to the goal, then we have greedy best\n first search; if f is node.depth then we have breadth-first search.\n There is a subtlety: the line \"f = memoize(f, 'f')\" means that the f\n values will be cached on the nodes as they are computed. So after doing\n a best first search you can examine the f values of the path returned.\"\"\"\n f = memoize(f, 'f')\n node = Node(problem.initial)\n if problem.goal_test(node.state):\n return node\n frontier = PriorityQueue(min, f)\n frontier.append(node)\n explored = set()\n while frontier:\n node = frontier.pop()\n if problem.goal_test(node.state):\n return node\n explored.add(node.state)\n for child in node.expand(problem):\n if child.state not in explored and child not in frontier:\n frontier.append(child)\n elif child in frontier:\n incumbent = frontier[child]\n if f(child) < f(incumbent):\n del frontier[incumbent]\n frontier.append(child)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef recursive_best_first_search(problem, h=None):\n \"[Fig. 3.26]\"\n h = memoize(h or problem.h, 'h')\n\n def RBFS(problem, node, flimit):\n if problem.goal_test(node.state):\n return node, 0 # (The second value is immaterial)\n successors = node.expand(problem)\n if len(successors) == 0:\n return None, infinity\n for s in successors:\n s.f = max(s.path_cost + h(s), node.f)\n while True:\n successors.sort(lambda x,y: cmp(x.f, y.f)) # Order by lowest f value\n best = successors[0]\n if best.f > flimit:\n return None, best.f\n if len(successors) > 1:\n alternative = successors[1].f\n else:\n alternative = infinity\n result, best.f = RBFS(problem, best, min(flimit, alternative))\n if result is not None:\n return result, best.f\n\n node = Node(problem.initial)\n node.f = h(node)\n result, bestf = RBFS(problem, node, infinity)\n return result", "response": "Recursive function to find the best first node in the tree."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef exp_schedule(k=20, lam=0.005, limit=100):\n \"One possible schedule function for simulated annealing\"\n return lambda t: if_(t < limit, k * math.exp(-lam * t), 0)", "response": "One possible schedule function for simulated annealing"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling genetic_algorithm on the appropriate parts of a problem. This is a convenience function that uses the fitness_fn to compute the fitness of the problem.", "response": "def genetic_search(problem, fitness_fn, ngen=1000, pmut=0.1, n=20):\n \"\"\"Call genetic_algorithm on the appropriate parts of a problem.\n This requires the problem to have states that can mate and mutate,\n plus a value method that scores states.\"\"\"\n s = problem.initial_state\n states = [problem.result(s, a) for a in problem.actions(s)]\n random.shuffle(states)\n return genetic_algorithm(states[:n], problem.value, ngen, pmut)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef genetic_algorithm(population, fitness_fn, ngen=1000, pmut=0.1):\n \"[Fig. 4.8]\"\n for i in range(ngen):\n new_population = []\n for i in len(population):\n fitnesses = map(fitness_fn, population)\n p1, p2 = weighted_sample_with_replacement(population, fitnesses, 2)\n child = p1.mate(p2)\n if random.uniform(0, 1) < pmut:\n child.mutate()\n new_population.append(child)\n population = new_population\n return argmax(population, fitness_fn)", "response": "Generate a random set of all the objects in the given population."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconstruct a random graph with the specified nodes and random links.", "response": "def RandomGraph(nodes=range(10), min_links=2, width=400, height=300,\n curvature=lambda: random.uniform(1.1, 1.5)):\n \"\"\"Construct a random graph, with the specified nodes, and random links.\n The nodes are laid out randomly on a (width x height) rectangle.\n Then each node is connected to the min_links nearest neighbors.\n Because inverse links are added, some nodes will have more connections.\n The distance between nodes is the hypotenuse times curvature(),\n where curvature() defaults to a random number between 1.1 and 1.5.\"\"\"\n g = UndirectedGraph()\n g.locations = {}\n ## Build the cities\n for node in nodes:\n g.locations[node] = (random.randrange(width), random.randrange(height))\n ## Build roads from each city to at least min_links nearest neighbors.\n for i in range(min_links):\n for node in nodes:\n if len(g.get(node)) < min_links:\n here = g.locations[node]\n def distance_to_node(n):\n if n is node or g.get(node,n): return infinity\n return distance(g.locations[n], here)\n neighbor = argmin(nodes, distance_to_node)\n d = distance(g.locations[neighbor], here) * curvature()\n g.connect(node, neighbor, int(d))\n return g"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef random_boggle(n=4):\n cubes = [cubes16[i % 16] for i in range(n*n)]\n random.shuffle(cubes)\n return map(random.choice, cubes)", "response": "Return a random Boggle board of size n x n."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprinting the board in a 2 - d array.", "response": "def print_boggle(board):\n \"Print the board in a 2-d array.\"\n n2 = len(board); n = exact_sqrt(n2)\n for i in range(n2):\n if i % n == 0 and i > 0: print\n if board[i] == 'Q': print 'Qu',\n else: print str(board[i]) + ' ',\n print"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of lists where the i - th element is the list of indexes for the neighbors of square i.", "response": "def boggle_neighbors(n2, cache={}):\n \"\"\"Return a list of lists, where the i-th element is the list of indexes\n for the neighbors of square i.\"\"\"\n if cache.get(n2):\n return cache.get(n2)\n n = exact_sqrt(n2)\n neighbors = [None] * n2\n for i in range(n2):\n neighbors[i] = []\n on_top = i < n\n on_bottom = i >= n2 - n\n on_left = i % n == 0\n on_right = (i+1) % n == 0\n if not on_top:\n neighbors[i].append(i - n)\n if not on_left: neighbors[i].append(i - n - 1)\n if not on_right: neighbors[i].append(i - n + 1)\n if not on_bottom:\n neighbors[i].append(i + n)\n if not on_left: neighbors[i].append(i + n - 1)\n if not on_right: neighbors[i].append(i + n + 1)\n if not on_left: neighbors[i].append(i - 1)\n if not on_right: neighbors[i].append(i + 1)\n cache[n2] = neighbors\n return neighbors"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef exact_sqrt(n2):\n \"If n2 is a perfect square, return its square root, else raise error.\"\n n = int(math.sqrt(n2))\n assert n * n == n2\n return n", "response": "If n2 is a perfect square return its square root else raise error."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsolves inverse Boggle by hill - climbing", "response": "def boggle_hill_climbing(board=None, ntimes=100, verbose=True):\n \"\"\"Solve inverse Boggle by hill-climbing: find a high-scoring board by\n starting with a random one and changing it.\"\"\"\n finder = BoggleFinder()\n if board is None:\n board = random_boggle()\n best = len(finder.set_board(board))\n for _ in range(ntimes):\n i, oldc = mutate_boggle(board)\n new = len(finder.set_board(board))\n if new > best:\n best = new\n if verbose: print best, _, board\n else:\n board[i] = oldc ## Change back\n if verbose:\n print_boggle(board)\n return board, best"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compare_graph_searchers():\n compare_searchers(problems=[GraphProblem('A', 'B', romania),\n GraphProblem('O', 'N', romania),\n GraphProblem('Q', 'WA', australia)],\n header=['Searcher', 'Romania(A, B)', 'Romania(O, N)', 'Australia'])", "response": "Prints a table of results like this"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlist the nodes reachable in one step from this node.", "response": "def expand(self, problem):\n \"List the nodes reachable in one step from this node.\"\n return [self.child_node(problem, action)\n for action in problem.actions(self.state)]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef path(self):\n \"Return a list of nodes forming the path from the root to this node.\"\n node, path_back = self, []\n while node:\n path_back.append(node)\n node = node.parent\n return list(reversed(path_back))", "response": "Return a list of nodes forming the path from the root to this node."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef mate(self, other):\n \"Return a new individual crossing self and other.\"\n c = random.randrange(len(self.genes))\n return self.__class__(self.genes[:c] + other.genes[c:])", "response": "Return a new individual crossing self and other."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_undirected(self):\n \"Make a digraph into an undirected graph by adding symmetric edges.\"\n for a in self.dict.keys():\n for (b, distance) in self.dict[a].items():\n self.connect1(b, a, distance)", "response": "Make a digraph into an undirected graph by adding symmetric edges."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef connect(self, A, B, distance=1):\n self.connect1(A, B, distance)\n if not self.directed: self.connect1(B, A, distance)", "response": "Add a link from A and B of given distance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef connect1(self, A, B, distance):\n \"Add a link from A to B of given distance, in one direction only.\"\n self.dict.setdefault(A,{})[B] = distance", "response": "Add a link from A to B of given distance in one direction only."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a link distance or a dict of { node : distance } entries.", "response": "def get(self, a, b=None):\n \"\"\"Return a link distance or a dict of {node: distance} entries.\n .get(a,b) returns the distance or None;\n .get(a) returns a dict of {node: distance} entries, possibly {}.\"\"\"\n links = self.dict.setdefault(a, {})\n if b is None: return links\n else: return links.get(b)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef actions(self, state):\n \"In the leftmost empty column, try all non-conflicting rows.\"\n if state[-1] is not None:\n return [] # All columns filled; no successors\n else:\n col = state.index(None)\n return [row for row in range(self.N)\n if not self.conflicted(state, row, col)]", "response": "In the leftmost empty column try all non - conflicting rows."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nplace the next queen at the given row.", "response": "def result(self, state, row):\n \"Place the next queen at the given row.\"\n col = state.index(None)\n new = state[:]\n new[col] = row\n return new"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef conflicted(self, state, row, col):\n \"Would placing a queen at (row, col) conflict with anything?\"\n return any(self.conflict(row, col, state[c], c)\n for c in range(col))", "response": "Would placing a queen at ( row col conflict with anything?"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef conflict(self, row1, col1, row2, col2):\n \"Would putting two queens in (row1, col1) and (row2, col2) conflict?\"\n return (row1 == row2 ## same row\n or col1 == col2 ## same column\n or row1-col1 == row2-col2 ## same \\ diagonal\n or row1+col1 == row2+col2)", "response": "Would put two queens in row1 col1 and row2 col2 conflict?"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsee if prefix is in dictionary as a full word or as a prefix. Returns two values first is the lowest i such that prefix is in the Wordlist False iff prefix itself is in the Wordlist.", "response": "def lookup(self, prefix, lo=0, hi=None):\n \"\"\"See if prefix is in dictionary, as a full word or as a prefix.\n Return two values: the first is the lowest i such that\n words[i].startswith(prefix), or is None; the second is\n True iff prefix itself is in the Wordlist.\"\"\"\n words = self.words\n if hi is None: hi = len(words)\n i = bisect.bisect_left(words, prefix, lo, hi)\n if i < len(words) and words[i].startswith(prefix):\n return i, (words[i] == prefix)\n else:\n return None, False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_board(self, board=None):\n \"Set the board, and find all the words in it.\"\n if board is None:\n board = random_boggle()\n self.board = board\n self.neighbors = boggle_neighbors(len(board))\n self.found = {}\n for i in range(len(board)):\n lo, hi = self.wordlist.bounds[board[i]]\n self.find(lo, hi, i, [], '')\n return self", "response": "Set the board and find all the words in it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nlook in square i find the words that continue the prefix and return the i - th entry in visited.", "response": "def find(self, lo, hi, i, visited, prefix):\n \"\"\"Looking in square i, find the words that continue the prefix,\n considering the entries in self.wordlist.words[lo:hi], and not\n revisiting the squares in visited.\"\"\"\n if i in visited:\n return\n wordpos, is_word = self.wordlist.lookup(prefix, lo, hi)\n if wordpos is not None:\n if is_word:\n self.found[prefix] = True\n visited.append(i)\n c = self.board[i]\n if c == 'Q': c = 'QU'\n prefix += c\n for j in self.neighbors[i]:\n self.find(wordpos, hi, j, visited, prefix)\n visited.pop()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef score(self):\n \"The total score for the words found, according to the rules.\"\n return sum([self.scores[len(w)] for w in self.words()])", "response": "The total score for the words found according to the rules."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef TraceAgent(agent):\n old_program = agent.program\n def new_program(percept):\n action = old_program(percept)\n print '%s perceives %s and does %s' % (agent, percept, action)\n return action\n agent.program = new_program\n return agent", "response": "Wrap the agent s program to print its input and output."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef TableDrivenAgentProgram(table):\n percepts = []\n def program(percept):\n percepts.append(percept)\n action = table.get(tuple(percepts))\n return action\n return program", "response": "This agent provides a program that selects an action based on the percept sequence."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ModelBasedReflexAgentProgram(rules, update_state):\n \"This agent takes action based on the percept and state. [Fig. 2.12]\"\n def program(percept):\n program.state = update_state(program.state, program.action, percept)\n rule = rule_match(program.state, rules)\n action = rule.action\n return action\n program.state = program.action = None\n return program", "response": "This agent takes action based on the percept and state. [ Fig. 2. 12 ]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ModelBasedVacuumAgent():\n \"An agent that keeps track of what locations are clean or dirty.\"\n model = {loc_A: None, loc_B: None}\n def program((location, status)):\n \"Same as ReflexVacuumAgent, except if everything is clean, do NoOp.\"\n model[location] = status ## Update the model here\n if model[loc_A] == model[loc_B] == 'Clean': return 'NoOp'\n elif status == 'Dirty': return 'Suck'\n elif location == loc_A: return 'Right'\n elif location == loc_B: return 'Left'\n return Agent(program)", "response": "An agent that keeps track of what locations are clean or dirty."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compare_agents(EnvFactory, AgentFactories, n=10, steps=1000):\n envs = [EnvFactory() for i in range(n)]\n return [(A, test_agent(A, steps, copy.deepcopy(envs)))\n for A in AgentFactories]", "response": "See how well each of several agents do in n instances of an environment."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef step(self):\n if not self.is_done():\n actions = [agent.program(self.percept(agent))\n for agent in self.agents]\n for (agent, action) in zip(self.agents, actions):\n self.execute_action(agent, action)\n self.exogenous_change()", "response": "Run the environment for one time step."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run(self, steps=1000):\n \"Run the Environment for given number of time steps.\"\n for step in range(steps):\n if self.is_done(): return\n self.step()", "response": "Run the Environment for given number of time steps."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef list_things_at(self, location, tclass=Thing):\n \"Return all things exactly at a given location.\"\n return [thing for thing in self.things\n if thing.location == location and isinstance(thing, tclass)]", "response": "Return all things exactly at a given location."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd a thing to the environment setting its location. For convenience add a thing to the environment setting its performance.", "response": "def add_thing(self, thing, location=None):\n \"\"\"Add a thing to the environment, setting its location. For\n convenience, if thing is an agent program we make a new agent\n for it. (Shouldn't need to override this.\"\"\"\n if not isinstance(thing, Thing):\n thing = Agent(thing)\n assert thing not in self.things, \"Don't add the same thing twice\"\n thing.location = location or self.default_location(thing)\n self.things.append(thing)\n if isinstance(thing, Agent):\n thing.performance = 0\n self.agents.append(thing)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nremoves a thing from the environment.", "response": "def delete_thing(self, thing):\n \"\"\"Remove a thing from the environment.\"\"\"\n try:\n self.things.remove(thing)\n except ValueError, e:\n print e\n print \" in Environment delete_thing\"\n print \" Thing to be removed: %s at %s\" % (thing, thing.location)\n print \" from list: %s\" % [(thing, thing.location)\n for thing in self.things]\n if thing in self.agents:\n self.agents.remove(thing)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef things_near(self, location, radius=None):\n \"Return all things within radius of location.\"\n if radius is None: radius = self.perceptible_distance\n radius2 = radius * radius\n return [thing for thing in self.things\n if distance2(location, thing.location) <= radius2]", "response": "Return all things within radius of location."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmoving a thing to a new location.", "response": "def move_to(self, thing, destination):\n \"Move a thing to a new location.\"\n thing.bump = self.some_things_at(destination, Obstacle)\n if not thing.bump:\n thing.location = destination\n for o in self.observers:\n o.thing_moved(thing)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nputting walls around the entire perimeter of the grid.", "response": "def add_walls(self):\n \"Put walls around the entire perimeter of the grid.\"\n for x in range(self.width):\n self.add_thing(Wall(), (x, 0))\n self.add_thing(Wall(), (x, self.height-1))\n for y in range(self.height):\n self.add_thing(Wall(), (0, y))\n self.add_thing(Wall(), (self.width-1, y))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef percept(self, agent):\n status = if_(self.some_things_at(agent.location, Dirt),\n 'Dirty', 'Clean')\n bump = if_(agent.bump, 'Bump', 'None')\n return (status, bump)", "response": "The percept is a tuple of ('Dirty Clean Bump None."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nchange agent s location and status ; track performance.", "response": "def execute_action(self, agent, action):\n \"\"\"Change agent's location and/or location's status; track performance.\n Score 10 for each dirt cleaned; -1 for each move.\"\"\"\n if action == 'Right':\n agent.location = loc_B\n agent.performance -= 1\n elif action == 'Left':\n agent.location = loc_A\n agent.performance -= 1\n elif action == 'Suck':\n if self.status[agent.location] == 'Dirty':\n agent.performance += 10\n self.status[agent.location] = 'Clean'"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a dictionary mapping symbols to alternative sequences.", "response": "def Rules(**rules):\n \"\"\"Create a dictionary mapping symbols to alternative sequences.\n >>> Rules(A = \"B C | D E\")\n {'A': [['B', 'C'], ['D', 'E']]}\n \"\"\"\n for (lhs, rhs) in rules.items():\n rules[lhs] = [alt.strip().split() for alt in rhs.split('|')]\n return rules"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Lexicon(**rules):\n for (lhs, rhs) in rules.items():\n rules[lhs] = [word.strip() for word in rhs.split('|')]\n return rules", "response": "Create a dictionary mapping symbols to alternative words."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef generate_random(grammar=E_, s='S'):\n import random\n\n def rewrite(tokens, into):\n for token in tokens:\n if token in grammar.rules:\n rewrite(random.choice(grammar.rules[token]), into)\n elif token in grammar.lexicon:\n into.append(random.choice(grammar.lexicon[token]))\n else:\n into.append(token)\n return into\n\n return ' '.join(rewrite(s.split(), []))", "response": "Generate a random entry in a grammar."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parses(self, words, S='S'):\n if isinstance(words, str):\n words = words.split()\n self.parse(words, S)\n # Return all the parses that span the whole input\n # 'span the whole input' => begin at 0, end at len(words)\n return [[i, j, S, found, []]\n for (i, j, lhs, found, expects) in self.chart[len(words)]\n # assert j == len(words)\n if i == 0 and lhs == S and expects == []]", "response": "Return a list of parses ; words can be a list or string."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse(self, words, S='S'):\n self.chart = [[] for i in range(len(words)+1)]\n self.add_edge([0, 0, 'S_', [], [S]])\n for i in range(len(words)):\n self.scanner(i, words[i])\n return self.chart", "response": "Parse a list of words ; according to the grammar.\n Leave results in the chart."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd edge to chart and see if it extends or predicts another edge.", "response": "def add_edge(self, edge):\n \"Add edge to chart, and see if it extends or predicts another edge.\"\n start, end, lhs, found, expects = edge\n if edge not in self.chart[end]:\n self.chart[end].append(edge)\n if self.trace:\n print '%10s: added %s' % (caller(2), edge)\n if not expects:\n self.extender(edge)\n else:\n self.predictor(edge)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef scanner(self, j, word):\n \"For each edge expecting a word of this category here, extend the edge.\"\n for (i, j, A, alpha, Bb) in self.chart[j]:\n if Bb and self.grammar.isa(word, Bb[0]):\n self.add_edge([i, j+1, A, alpha + [(Bb[0], word)], Bb[1:]])", "response": "For each edge expecting a word of this category here extend the edge."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef predictor(self, (i, j, A, alpha, Bb)):\n \"Add to chart any rules for B that could help extend this edge.\"\n B = Bb[0]\n if B in self.grammar.rules:\n for rhs in self.grammar.rewrites_for(B):\n self.add_edge([j, j, B, [], rhs])", "response": "Add to chart any rules for B that could help extend this edge."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef extender(self, edge):\n \"See what edges can be extended by this edge.\"\n (j, k, B, _, _) = edge\n for (i, j, A, alpha, B1b) in self.chart[j]:\n if B1b and B == B1b[0]:\n self.add_edge([i, k, A, alpha + [edge], B1b[1:]])", "response": "See what edges can be extended by this edge."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a SettingDict object to the context as . Automatically creates non - existent settings with an empty value.", "response": "def settings(request):\n \"\"\"\n Adds a ``SettingDict`` object for the ``Setting`` model to the context as\n ``SETTINGS``. Automatically creates non-existent settings with an empty\n string as the default value.\n \"\"\"\n settings = Setting.objects.all().as_dict(default='')\n context = {\n 'SETTINGS': settings,\n }\n return context"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_child_models(self):\n child_models = []\n # Loop through all models with FKs back to `base_model`.\n for related_object in get_all_related_objects(self.base_model._meta):\n # Django 1.8 deprecated `get_all_related_objects()`. We're still\n # using it for now with the documented work-around for\n # compatibility with Django <=1.7.\n model = getattr(\n related_object, 'related_model', related_object.model)\n # Only consider `base_model` subclasses.\n if issubclass(model, self.base_model):\n class SettingValueAdmin(self.base_admin_class):\n pass\n child_models.append((model, SettingValueAdmin))\n return child_models", "response": "Returns a list of tuples for base_model and base_admin_class subclasses."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef event_values(event, vars):\n if isinstance(event, tuple) and len(event) == len(vars):\n return event\n else:\n return tuple([event[var] for var in vars])", "response": "Return a tuple of the values of variables vars in event."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a probability distribution over the values of the variable X given the observations e in the JointProbDist P.", "response": "def enumerate_joint_ask(X, e, P):\n \"\"\"Return a probability distribution over the values of the variable X,\n given the {var:val} observations e, in the JointProbDist P. [Section 13.3]\n >>> P = JointProbDist(['X', 'Y'])\n >>> P[0,0] = 0.25; P[0,1] = 0.5; P[1,1] = P[2,1] = 0.125\n >>> enumerate_joint_ask('X', dict(Y=1), P).show_approx()\n '0: 0.667, 1: 0.167, 2: 0.167'\n \"\"\"\n assert X not in e, \"Query variable must be distinct from evidence\"\n Q = ProbDist(X) # probability distribution for X, initially empty\n Y = [v for v in P.variables if v != X and v not in e] # hidden vars.\n for xi in P.values(X):\n Q[xi] = enumerate_joint(Y, extend(e, X, xi), P)\n return Q.normalize()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef enumerate_joint(vars, e, P):\n if not vars:\n return P[e]\n Y, rest = vars[0], vars[1:]\n return sum([enumerate_joint(rest, extend(e, Y, y), P)\n for y in P.values(Y)])", "response": "Return the sum of those entries in P consistent with e."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef enumeration_ask(X, e, bn):\n assert X not in e, \"Query variable must be distinct from evidence\"\n Q = ProbDist(X)\n for xi in bn.variable_values(X):\n Q[xi] = enumerate_all(bn.vars, extend(e, X, xi), bn)\n return Q.normalize()", "response": "Return the conditional probability distribution of variable X given evidence e from BayesNet bn."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the sum of those entries in P consistent with e where P is the joint distribution represented by bn.", "response": "def enumerate_all(vars, e, bn):\n \"\"\"Return the sum of those entries in P(vars | e{others})\n consistent with e, where P is the joint distribution represented\n by bn, and e{others} means e restricted to bn's other variables\n (the ones other than vars). Parents must precede children in vars.\"\"\"\n if not vars:\n return 1.0\n Y, rest = vars[0], vars[1:]\n Ynode = bn.variable_node(Y)\n if Y in e:\n return Ynode.p(e[Y], e) * enumerate_all(rest, e, bn)\n else:\n return sum(Ynode.p(y, e) * enumerate_all(rest, extend(e, Y, y), bn)\n for y in bn.variable_values(Y))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef elimination_ask(X, e, bn):\n assert X not in e, \"Query variable must be distinct from evidence\"\n factors = []\n for var in reversed(bn.vars):\n factors.append(make_factor(var, e, bn))\n if is_hidden(var, X, e):\n factors = sum_out(var, factors, bn)\n return pointwise_product(factors, bn).normalize()", "response": "Compute bn s P by variable elimination."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the factor for var in bn s joint distribution given e. That is the factor for var in bn s full joint distribution given e. is the pointwise product of these factors for bn s variables.", "response": "def make_factor(var, e, bn):\n \"\"\"Return the factor for var in bn's joint distribution given e.\n That is, bn's full joint distribution, projected to accord with e,\n is the pointwise product of these factors for bn's variables.\"\"\"\n node = bn.variable_node(var)\n vars = [X for X in [var] + node.parents if X not in e]\n cpt = dict((event_values(e1, vars), node.p(e1[var], e1))\n for e1 in all_events(vars, bn, e))\n return Factor(vars, cpt)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\neliminates var from all factors by summing over its values.", "response": "def sum_out(var, factors, bn):\n \"Eliminate var from all factors by summing over its values.\"\n result, var_factors = [], []\n for f in factors:\n (var_factors if var in f.vars else result).append(f)\n result.append(pointwise_product(var_factors, bn).sum_out(var, bn))\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nyields every way of extending e with values for all vars.", "response": "def all_events(vars, bn, e):\n \"Yield every way of extending e with values for all vars.\"\n if not vars:\n yield e\n else:\n X, rest = vars[0], vars[1:]\n for e1 in all_events(rest, bn, e):\n for x in bn.variable_values(X):\n yield extend(e1, X, x)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef prior_sample(bn):\n event = {}\n for node in bn.nodes:\n event[node.variable] = node.sample(event)\n return event", "response": "Randomly sample from bn s full joint distribution."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rejection_sampling(X, e, bn, N):\n counts = dict((x, 0) for x in bn.variable_values(X)) # bold N in Fig. 14.14\n for j in xrange(N):\n sample = prior_sample(bn) # boldface x in Fig. 14.14\n if consistent_with(sample, e):\n counts[sample[X]] += 1\n return ProbDist(X, counts)", "response": "Estimate the probability distribution of variable X given an evidence e in BayesNet bn using N samples."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nam event consistent with the given evidence?", "response": "def consistent_with(event, evidence):\n \"Is event consistent with the given evidence?\"\n return every(lambda (k, v): evidence.get(k, v) == v,\n event.items())"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef likelihood_weighting(X, e, bn, N):\n W = dict((x, 0) for x in bn.variable_values(X))\n for j in xrange(N):\n sample, weight = weighted_sample(bn, e) # boldface x, w in Fig. 14.15\n W[sample[X]] += weight\n return ProbDist(X, W)", "response": "Estimate the probability distribution of variable X given\n evidence e in BayesNet bn."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsampling an event from bn that s consistent with the evidence e ; return the event and its weight the likelihood that the event accords to the evidence e.", "response": "def weighted_sample(bn, e):\n \"\"\"Sample an event from bn that's consistent with the evidence e;\n return the event and its weight, the likelihood that the event\n accords to the evidence.\"\"\"\n w = 1\n event = dict(e) # boldface x in Fig. 14.15\n for node in bn.nodes:\n Xi = node.variable\n if Xi in e:\n w *= node.p(e[Xi], event)\n else:\n event[Xi] = node.sample(event)\n return event, w"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a sample from P where mb denotes that the variables in the Markov blanket of X take their values from event e. The Markov blanket of X is X and e are assigned a value to each. The Markov blanket of X is X and e are assigned a value to each. The Markov blanket of X is the same as the Markov blanket of X.", "response": "def markov_blanket_sample(X, e, bn):\n \"\"\"Return a sample from P(X | mb) where mb denotes that the\n variables in the Markov blanket of X take their values from event\n e (which must assign a value to each). The Markov blanket of X is\n X's parents, children, and children's parents.\"\"\"\n Xnode = bn.variable_node(X)\n Q = ProbDist(X)\n for xi in bn.variable_values(X):\n ei = extend(e, X, xi)\n # [Equation 14.12:]\n Q[xi] = Xnode.p(xi, e) * product(Yj.p(ei[Yj.variable], ei)\n for Yj in Xnode.children)\n return probability(Q.normalize()[True])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef normalize(self):\n total = float(sum(self.prob.values()))\n if not (1.0-epsilon < total < 1.0+epsilon):\n for val in self.prob:\n self.prob[val] /= total\n return self", "response": "Make sure the probabilities of all values sum to 1. Returns the normalized distribution."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nshow the probabilities rounded and sorted by key for the sake of portable doctests.", "response": "def show_approx(self, numfmt='%.3g'):\n \"\"\"Show the probabilities rounded and sorted by key, for the\n sake of portable doctests.\"\"\"\n return ', '.join([('%s: ' + numfmt) % (v, p)\n for (v, p) in sorted(self.prob.items())])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add(self, node_spec):\n node = BayesNode(*node_spec)\n assert node.variable not in self.vars\n assert every(lambda parent: parent in self.vars, node.parents)\n self.nodes.append(node)\n self.vars.append(node.variable)\n for parent in node.parents:\n self.variable_node(parent).children.append(node)", "response": "Add a node to the net."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the node for the variable named var.", "response": "def variable_node(self, var):\n \"\"\"Return the node for the variable named var.\n >>> burglary.variable_node('Burglary').variable\n 'Burglary'\"\"\"\n for n in self.nodes:\n if n.variable == var:\n return n\n raise Exception(\"No such variable: %s\" % var)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the conditional probability of the logarithmic probability of the event.", "response": "def p(self, value, event):\n \"\"\"Return the conditional probability\n P(X=value | parents=parent_values), where parent_values\n are the values of parents in event. (event must assign each\n parent a value.)\n >>> bn = BayesNode('X', 'Burglary', {T: 0.2, F: 0.625})\n >>> bn.p(False, {'Burglary': False, 'Earthquake': True})\n 0.375\"\"\"\n assert isinstance(value, bool)\n ptrue = self.cpt[event_values(event, self.parents)]\n return if_(value, ptrue, 1 - ptrue)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pointwise_product(self, other, bn):\n \"Multiply two factors, combining their variables.\"\n vars = list(set(self.vars) | set(other.vars))\n cpt = dict((event_values(e, vars), self.p(e) * other.p(e))\n for e in all_events(vars, bn, {}))\n return Factor(vars, cpt)", "response": "Multiply two factors combining their variables."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmakes a factor eliminating var by summing over its values.", "response": "def sum_out(self, var, bn):\n \"Make a factor eliminating var by summing over its values.\"\n vars = [X for X in self.vars if X != var]\n cpt = dict((event_values(e, vars),\n sum(self.p(extend(e, var, val))\n for val in bn.variable_values(var)))\n for e in all_events(vars, bn, {}))\n return Factor(vars, cpt)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef normalize(self):\n \"Return my probabilities; must be down to one variable.\"\n assert len(self.vars) == 1\n return ProbDist(self.vars[0],\n dict((k, v) for ((k,), v) in self.cpt.items()))", "response": "Return my probabilities ; must be down to one variable."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef next_chunk_boundaries(self, buf, prepend_bytes=0):\n return (boundary for boundary, _ in self.next_chunk_boundaries_levels(buf, prepend_bytes))", "response": "Computes the next chunk boundaries within buf."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the next chunk boundaries within buf.", "response": "def next_chunk_boundaries_levels(self, buf, prepend_bytes=0):\n \"\"\"Computes the next chunk boundaries within `buf`.\n\n Similar to :meth:`.next_chunk_boundaries`, but information about which chunker led to a respective boundary is\n included in the returned value.\n\n Args:\n buf (bytes): The message that is to be chunked.\n prepend_bytes (Optional[int]): Optional number of zero bytes that should be input to the chunking algorithm\n before `buf`.\n\n Returns:\n list: List of tuples (boundary, level), where boundary is a boundary position relative to `buf` and level is\n the index of the chunker (i.e., the index of its chunk size specified during instantiation) that yielded\n the boundary.\n\n If multiple chunkers yield the same boundary, it is returned only once, along with the highest matching\n chunker index.\n \"\"\"\n boundaries = {}\n for level_index, chunker in enumerate(self._chunkers):\n boundaries.update(\n dict([(boundary, level_index) for boundary in chunker.next_chunk_boundaries(buf, prepend_bytes)]))\n return sorted(boundaries.items())"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a chunker performing content - defined chunking using Rabin Karp s rolling hash scheme with the specified chunk size.", "response": "def create_chunker(self, chunk_size):\n \"\"\"Create a chunker performing content-defined chunking (CDC) using Rabin Karp's rolling hash scheme with a\n specific, expected chunk size.\n\n Args:\n chunk_size (int): (Expected) target chunk size.\n\n Returns:\n BaseChunker: A chunker object.\n \"\"\"\n rolling_hash = _rabinkarprh.RabinKarpHash(self.window_size, self._seed)\n rolling_hash.set_threshold(1.0 / chunk_size)\n return RabinKarpCDC._Chunker(rolling_hash)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a multi - level chunker performing content - defined chunking using Rabin Karp s rolling hash scheme.", "response": "def create_multilevel_chunker(self, chunk_sizes):\n \"\"\"Create a multi-level chunker performing content-defined chunking (CDC) using Rabin Karp's rolling hash scheme\n with different specific, expected chunk sizes.\n\n Args:\n chunk_sizes (list): List of (expected) target chunk sizes.\n\n Warning:\n For performance reasons, behavior is only defined if chunk sizes are passed in order, i.e., from\n lowest to highest value.\n\n Returns:\n BaseMultiLevelChunker: A multi-level chunker object.\n \"\"\"\n rolling_hash = _rabinkarprh.RabinKarpMultiThresholdHash(self.window_size, self._seed,\n [1.0 / chunk_size for chunk_size in chunk_sizes])\n return RabinKarpCDC._MultiLevelChunker(rolling_hash)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef brightness(level=100, group=0):\n if level not in range(0,101):\n raise Exception(\"Brightness must be value between 0 and 100\")\n b = int(floor(level / 4.0) + 2) #lights want values 2-27\n return (COMMANDS['ON'][group], Command(0x4E, b))", "response": "Returns a command that sets the brightness of the given level."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstripping all whitespace from a minidom XML node and its children Arcs.", "response": "def strip_minidom_whitespace(node):\n \"\"\"Strips all whitespace from a minidom XML node and its children\n\n This operation is made in-place.\"\"\"\n for child in node.childNodes:\n if child.nodeType == Node.TEXT_NODE:\n if child.nodeValue:\n child.nodeValue = child.nodeValue.strip()\n elif child.nodeType == Node.ELEMENT_NODE:\n strip_minidom_whitespace(child)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a tuple of commands that can be used to set the brightness of a single light.", "response": "def brightness(level=100, group=0):\n \"\"\" Assumes level is out of 100 \"\"\"\n if level not in range(0,101):\n raise Exception(\"Brightness must be value between 0 and 100\")\n b = int(floor(level / 10.0)) #lights have 10 levels of brightness\n commands = list(darkest(group))\n for i in range(0, b):\n commands.append(COMMANDS['BRIGHTER'])\n return tuple(commands)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef warmness(level=100, group=0):\n if level not in range(0,101):\n raise Exception(\"Warmness must be value between 0 and 100\")\n b = int(floor(level / 10.0)) #lights have 10 levels of warmness\n commands = list(coolest(group))\n for i in range(0, b):\n commands.append(COMMANDS['WARMER'])\n return tuple(commands)", "response": "Returns a tuple of warmness commands."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef color_from_hls(hue, light, sat):\n if light > 0.95: #too bright, let's just switch to white\n return 256\n elif light < 0.05: #too dark, let's shut it off\n return -1\n else:\n hue = (-hue + 1 + 2.0/3.0) % 1 # invert and translate by 2/3\n return int(floor(hue * 256))", "response": "Converts a HSL color to proper HSL color"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntakes a standard rgb color and converts it to a proper hue value", "response": "def color_from_rgb(red, green, blue):\n \"\"\" Takes your standard rgb color \n and converts it to a proper hue value \"\"\"\n \n r = min(red, 255)\n g = min(green, 255)\n b = min(blue, 255)\n if r > 1 or g > 1 or b > 1:\n r = r / 255.0\n g = g / 255.0\n b = b / 255.0\n\n return color_from_hls(*rgb_to_hls(r,g,b))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntakes an HTML hex code and converts it to a proper hue value", "response": "def color_from_hex(value):\n \"\"\" Takes an HTML hex code\n and converts it to a proper hue value \"\"\"\n if \"#\" in value:\n value = value[1:]\n \n try:\n unhexed = bytes.fromhex(value)\n except:\n unhexed = binascii.unhexlify(value) # Fallback for 2.7 compatibility\n return color_from_rgb(*struct.unpack('BBB',unhexed))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef wait(self, sec=0.1):\n sec = max(sec, 0)\n reps = int(floor(sec / 0.1))\n commands = []\n for i in range(0, reps):\n commands.append(Command(0x00, wait=True))\n return tuple(commands)", "response": "Wait for x seconds\n each wait command is 100ms"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef tictoc(name='tictoc'):\n t = time.time()\n yield\n logg.info('%s Elapsed: %s secs' % (name, time.time() - t))", "response": "A generator that yields the sequence of tokens that are not in the tictoc."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef split_rand(data_or_size, ratio, seed):\n if not isinstance(data_or_size, int):\n sz = len(data_or_size)\n data = np.asarray(data_or_size)\n else:\n sz = data_or_size\n data = np.arange(sz)\n if not ratio:\n return data, []\n\n i = np.zeros(sz, dtype='bool')\n lattersz = int(sz * ratio)\n i[:lattersz] = True\n\n with np_seed(seed):\n np.random.shuffle(i)\n\n return data[~i], data[i]", "response": "split_rand - splits the data into two lists of size ratio"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns train, valid [,test] testset if p_testset :param n: :param k: :param sz: :param p_testset: :param seed: :return:", "response": "def kfolds(n, k, sz, p_testset=None, seed=7238):\n \"\"\"\n return train, valid [,test]\n testset if p_testset\n :param n:\n :param k:\n :param sz:\n :param p_testset:\n :param seed:\n :return:\n \"\"\"\n trains, tests = split_rand(sz, p_testset, seed)\n\n ntrain = len(trains)\n\n # np.random.seed(seed)\n with np_seed(seed):\n np.random.shuffle(trains)\n if n == k:\n # no split\n train, valid = trains, trains\n else:\n foldsz = ntrain // k\n\n itrain = np.arange(ntrain) // foldsz != n\n train = trains[itrain]\n valid = trains[~itrain]\n\n if not p_testset:\n return train, valid\n else:\n return train, valid, tests"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef queryString_required_ClassVersion(strList):\n\tdef _dec(function):\n\t\t@wraps(function)\n\t\tdef _wrap(classInstance, request, *args, **kwargs):\n\t\t\tfor i in strList:\n\t\t\t\tif i not in request.GET:\n\t\t\t\t\traise Http404(\"api does not exist\")\n\t\t\treturn function(classInstance, request, *args, **kwargs)\n\t\treturn _wrap\n\treturn _dec", "response": "This is a decorator that checks whether the given queryString key is valid or not."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn json from querying Web Api", "response": "def getJsonFromApi(view, request):\n\t\"\"\"Return json from querying Web Api\n\n\t\tArgs:\n\t\t\tview: django view function.\n\t\t\trequest: http request object got from django.\n\t\t\t\t\n\t\tReturns: json format dictionary\n\t\t\"\"\"\n\tjsonText = view(request)\n\tjsonText = json.loads(jsonText.content.decode('utf-8'))\n\treturn jsonText"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef progress(iter, **kwargs):\n if isinstance(iter, int):\n iter = xrange(iter)\n if hasattr(iter, '__len__') or 'target' in kwargs:\n cls = Progress\n else:\n cls = ProgressBase\n\n return cls(iter, **kwargs)", "response": "\ud504\ub85c\uadf8\ub798\uc2a4 bar\n for i in progress(10):\n print i\n\n for i in progress(iter):\n print i"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef threaded(f, *args, **kwargs):\n if args or kwargs:\n return Threaded(f, *args, **kwargs)\n\n @wraps(f)\n def wrapped(*wargs, **wkwargs):\n return Threaded(f, *wargs, **wkwargs)\n\n return wrapped", "response": "decorator for threaded functions"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a new ODict containing only the keys that are in self and other.", "response": "def intersect(self, other):\n \"\"\"\n self\uc640 other \ud0a4\uac00 \ub3d9\uc77c\ud55c \uc544\uc774\ud15c\uc758 dictobj\n :type other: dict\n :rtype: dictobj:\n \"\"\"\n return ODict((k, self[k]) for k in self if k in other)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_dict(dic):\n return ODict((k, ODict.convert_ifdic(v)) for k, v in dic.items())", "response": "Convert a recursive dict to dictobj"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting a grid of images on the nhwc alphabetical order.", "response": "def imshow_grid(images, grid=None, showfun=None, **opt):\n \"\"\"\n :param images: nhwc\n :return:\n \"\"\"\n # assert images.ndim == 4 or list\n showfun = showfun or plt.imshow\n count = len(images)\n grid = grid or grid_recommend(count, sorted(images[0].shape[:2]))\n\n res = []\n for i, img in enumerate(images):\n # grid row first index\n plt.subplot2grid(grid, (i % grid[0], i // grid[0]))\n res.append(showfun(img.squeeze(), **opt))\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef plt_range(*args, **kwargs):\n wait = kwargs.pop('wait', True)\n if not wait:\n # no interactive just pass range\n for i in progress(range(*args)):\n yield i\n return\n\n class _holder(object):\n pass\n hold = _holder()\n hold.i = 0\n hold.done = False\n\n def press(event):\n # import sys\n # sys.stdout.flush()\n hold.i += -1 if event.key == 'left' else 1\n hold.i = 0 if hold.i < 0 else hold.i\n\n def onclose(event):\n hold.done = True\n\n fig = kwargs.pop('fig', None)\n figsize = kwargs.pop('figsize', None)\n if fig is None:\n fig = plt.gcf()\n if figsize:\n fig.set_size_inches(figsize)\n elif isinstance(fig, (int, str)):\n if figsize:\n fig = plt.figure(fig, figsize=figsize)\n else:\n fig = plt.figure(fig)\n elif isinstance(fig, plt.Figure):\n if figsize:\n fig.set_size_inches(figsize)\n else:\n raise ValueError\n\n onkey_fig(press, fig)\n onclose_fig(onclose, fig)\n\n ranges = range(*args)\n l = len(ranges)\n\n while hold.i < l:\n print('hold.i', ranges[hold.i])\n yield ranges[hold.i] # yield first without keypress\n before = hold.i\n while before == hold.i:\n while not fig.waitforbuttonpress(0.01):\n if hold.done:\n return\n while fig.waitforbuttonpress(0.1):\n if hold.done:\n return", "response": "A generator function that yields a range of the available key sequences."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plot_pause(timeout=None, msg=''):\n\n if timeout is not None:\n print(msg or 'Press key for continue in time {}'.format(timeout))\n plt.waitforbuttonpress(timeout=timeout)\n return True\n\n print(msg or 'Press key for continue')\n while not plt.waitforbuttonpress(timeout=0.01):\n if not plt.get_fignums():\n return False\n return len(plt.get_fignums()) != 0", "response": "todo : add some example\n :param timeout: wait time. if None, blocking\n :param msg:\n :return:"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flat_images(images, grid=None, bfill=1.0, bsz=(1, 1)):\n if images.ndim == 4 and images.shape[-1] == 1:\n images = images.squeeze(axis=-1)\n\n grid = grid or grid_recommend(len(images), sorted(images[0].shape[:2]))\n if not isinstance(bsz, (tuple, list)):\n bsz = (bsz, bsz)\n\n # np.empty()\n imshape = list(images.shape)\n imshape[0] = grid[0] * grid[1]\n imshape[1] += bsz[0]\n imshape[2] += bsz[1]\n\n # data = np.empty((grid[0] * grid[1], imshape[1], imshape[2]), dtype=images.dtype)\n data = np.empty(imshape, dtype=images.dtype)\n\n data.fill(bfill)\n bslice0 = slice(0, -bsz[0]) if bsz[0] else slice(None, None)\n bslice1 = slice(0, -bsz[1]) if bsz[1] else slice(None, None)\n\n data[:len(images), bslice0, bslice1] = images\n\n imshape = list(grid) + imshape[1:] # [grid[0], grid[1], H, W, [Channel]]\n data = data.reshape(imshape)\n if len(imshape) == 5:\n data = data.transpose(0, 2, 1, 3, 4)\n imshape = [imshape[0]*imshape[2], imshape[1]*imshape[3], imshape[4]]\n else: # len == 4\n data = data.transpose(0, 2, 1, 3)\n imshape = [imshape[0]*imshape[2], imshape[1]*imshape[3]]\n data = data.reshape(imshape)\n\n # remove last margin\n data = data[bslice0, bslice1]\n\n return data", "response": "converts a batch image to a flat image with margin inserted\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef imbox(xy, w, h, angle=0.0, **kwargs):\n from matplotlib.patches import Rectangle\n return imbound(Rectangle, xy, w, h, angle, **kwargs)", "response": "Draw a bounding box on the n - dimensional area of a series."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef imbound(clspatch, *args, **kwargs):\n # todo : add example\n\n c = kwargs.pop('color', kwargs.get('edgecolor', None))\n kwargs.update(facecolor='none', edgecolor=c)\n return impatch(clspatch, *args, **kwargs)", "response": "imbalance the given clspatch"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef imslic(img, n_segments=100, aspect=None):\n from skimage.segmentation import (slic, mark_boundaries)\n from skimage.morphology import (dilation)\n\n if img.ndim == 2 or img.ndim == 3 and img.shape[-1] == 1:\n imz = np.stack([img, img, img], 2)\n else:\n imz = img\n\n slics = slic(imz, n_segments=n_segments)\n\n boundaries = mark_boundaries(imz, slics)\n return plt.imshow(boundaries, aspect=aspect)", "response": "Function to plot a slic image."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef put(xy, *args):\n cmd = [TermCursor.save, TermCursor.move(*xy), ''.join(args), TermCursor.restore]\n write(''.join(cmd))", "response": "put text on screen\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getpassword(prompt=\"Password: \"):\n\n fd = sys.stdin.fileno()\n old = termios.tcgetattr(fd)\n new = termios.tcgetattr(fd)\n new[3] &= ~termios.ECHO # lflags\n try:\n termios.tcsetattr(fd, termios.TCSADRAIN, new)\n passwd = raw_input(prompt)\n finally:\n termios.tcsetattr(fd, termios.TCSADRAIN, old)\n return passwd", "response": "get user input without echo\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getch():\n try:\n termios.tcsetattr(_fd, termios.TCSANOW, _new_settings)\n ch = sys.stdin.read(1)\n finally:\n termios.tcsetattr(_fd, termios.TCSADRAIN, _old_settings)\n return ch", "response": "get character from the current keyboard"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getlogger(pkg='', handler=None):\n from .caller import caller\n\n if not pkg:\n m = caller.modulename()\n s = m.split('.', 1)\n if len(s) > 1:\n pkg = s[0]\n\n if haslogger(pkg):\n return logging.getLogger(pkg)\n else:\n # local\n logger = logging.getLogger(pkg)\n logger.addHandler(handler or default_handler)\n logger.setLevel(logging.DEBUG)\n return logger", "response": "get logger by package name"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nformat a log record", "response": "def format(self, record):\n \"\"\"tweaked from source of base\"\"\"\n try:\n record.message = record.getMessage()\n except TypeError:\n # if error during msg = msg % self.args\n if record.args:\n if isinstance(record.args, collections.Mapping):\n record.message = record.msg.format(**record.args)\n else:\n record.message = record.msg.format(record.args)\n self._fmt = self.getfmt(record.levelname)\n if self.usesTime():\n record.asctime = self.formatTime(record, self.datefmt)\n\n s = self._fmt.format(**record.__dict__)\n\n if record.exc_info:\n # Cache the traceback text to avoid converting it multiple times\n # (it's constant anyway)\n if not record.exc_text:\n record.exc_text = self.formatException(record.exc_info)\n if record.exc_text:\n if s[-1:] != '\\n':\n s += '\\n'\n try:\n s = s + record.exc_text\n except UnicodeError:\n s = s + record.exc_text.decode(sys.getfilesystemencoding(), 'replace')\n return s"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the process owner of a process id .", "response": "def getProcessOwner(pid):\n '''\n getProcessOwner - Get the process owner of a pid\n\n @param pid - process id\n\n @return - None if process not found or can't be determined. Otherwise, a dict: \n {\n uid - Owner UID\n name - Owner name, or None if one cannot be determined\n }\n '''\n try:\n ownerUid = os.stat('/proc/' + str(pid)).st_uid\n except:\n return None\n \n try:\n ownerName = pwd.getpwuid(ownerUid).pw_name\n except:\n ownerName = None\n\n return {\n 'uid' : ownerUid,\n 'name' : ownerName\n }"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getProcessOwnerStr(pid):\n '''\n getProcessOwner - Get Process owner of a pid as a string instead of components (#getProcessOwner)\n\n @return - Returns username if it can be determined, otherwise uid, otherwise \"unknown\"\n '''\n ownerInfo = getProcessOwner(pid)\n if ownerInfo:\n if ownerInfo['name']:\n owner = ownerInfo['name']\n else:\n owner = str(ownerInfo['uid'])\n else:\n owner = 'unknown'\n\n return owner", "response": "Get the Process owner of a pid as a string instead of components (#getProcessOwner )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a commandline of a given process ID.", "response": "def getProcessCommandLineStr(pid):\n '''\n getProcessCommandLineStr - Gets a the commandline (program + arguments) of a given pid\n\n @param pid - Process ID\n\n @return - None if process not found or can't be determined. Otherwise a string of commandline.\n\n @note Caution, args may have spaces in them, and you cannot surmise from this method. If you care (like trying to replay a command), use getProcessCommandLineList instead\n '''\n try:\n with open('/proc/%d/cmdline' %(int(pid),), 'r') as f:\n cmdline = f.read()\n return cmdline.replace('\\x00', ' ')\n except:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getProcessCommandLineList(pid):\n '''\n getProcessCommandLineList - Gets the commandline (program + argumentS) of a given pid as a list.\n\n @param pid - Process ID\n\n @return - None if process not found or can't be determined. Otherwise a list representing argv. First argument is process name, remainder are arguments.\n\n @note - Use this if you care about whether a process had a space in the commands\n '''\n try:\n with open('/proc/%d/cmdline' %(int(pid),), 'r') as f:\n cmdline = f.read()\n\n return cmdline.split('\\x00')\n except:\n return None", "response": "Gets the commandline of a given process ID."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsearches a given pid s cwd for a given pattern and returns a dictionary of the process information.", "response": "def scanProcessForCwd(pid, searchPortion, isExactMatch=False):\n '''\n scanProcessForCwd - Searches a given pid's cwd for a given pattern\n\n @param pid - A running process ID on this system\n @param searchPortion - Any portion of directory to search\n @param isExactMatch Default False - If match should be exact, otherwise a partial match is performed.\n\n @return - If result is found, the following dict is returned. If no match found on the given pid, or pid is not found running, None is returned.\n {\n 'searchPortion' : The passed search pattern\n 'pid' : The passed pid (as an integer)\n 'owner' : String of process owner, or uid if no mapping can be found, or \"unknown\" if neither could be determined.\n 'cmdline' : Commandline string\n 'cwd' : The exact cwd of matched process\n }\n '''\n try: \n try:\n pid = int(pid)\n except ValueError as e:\n sys.stderr.write('Expected an integer, got %s for pid.\\n' %(str(type(pid)),))\n raise e\n \n\n cwd = getProcessCwd(pid)\n if not cwd:\n return None\n\n isMatch = False\n if isExactMatch is True:\n if searchPortion == cwd:\n isMatch = True\n else:\n if searchPortion.endswith('/') and searchPortion[:-1] == cwd:\n isMatch = True\n else:\n if searchPortion in cwd:\n isMatch = True\n else:\n if searchPortion.endswith('/') and searchPortion[:-1] in cwd:\n isMatch = True\n\n if not isMatch:\n return None\n\n cmdline = getProcessCommandLineStr(pid)\n owner = getProcessOwnerStr(pid)\n\n return {\n 'searchPortion' : searchPortion,\n 'pid' : pid,\n 'owner' : owner,\n 'cmdline' : cmdline,\n 'cwd' : cwd,\n }\n except OSError:\n return None\n except IOError:\n return None\n except FileNotFoundError:\n return None\n except PermissionError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef scanAllProcessesForCwd(searchPortion, isExactMatch=False):\n '''\n scanAllProcessesForCwd - Scans all processes on the system for a given search pattern.\n\n @param searchPortion - Any portion of directory to search\n @param isExactMatch Default False - If match should be exact, otherwise a partial match is performed.\n\n @return - - A dictionary of pid -> cwdResults for each pid that matched the search pattern. For format of \"cwdResults\", @see scanProcessForCwd\n '''\n \n pids = getAllRunningPids()\n\n cwdResults = [scanProcessForCwd(pid, searchPortion, isExactMatch) for pid in pids]\n ret = {}\n for i in range(len(pids)):\n if cwdResults[i] is not None:\n ret[pids[i]] = cwdResults[i]\n\n return ret", "response": "This function scans all processes on the system for a given search pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef enum(name, *members, **withvalue):\n if len(members) == 1:\n if isinstance(members[0], str):\n members = members[0].split()\n elif isinstance(members[0], (list, tuple)):\n members = members[0]\n\n dic = {v: v for v in members}\n dic.update(withvalue)\n\n return type(name, (Enum,), dic)", "response": "create an enum object"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef connect(db='', **kwargs):\n arg = db_config(db)\n arg.update(kwargs)\n return MySQLdb.connect(**arg)", "response": "db \uc811\uc18d \uacf5\ud1b5 \uc778\uc790\ub4e4 \ucc44\uc6cc\uc11c \uc811\uc18d, schema\ub9cc \ub123\uc73c\uba74 \ub429\ub2c8\ub2e4.\n db connection \uac1d\uccb4 \ubc18\ud658\uc774\uc9c0\ub9cc\n with \ubb38\uacfc \uac19\uc774 \uc4f0\uc774\uba74 cursor\uc784\uc5d0 \uc8fc\uc758 (MySQLdb\uc758 \uad6c\ud604\uc774 \uadf8\ub807\uc2b5\ub2c8\ub2e4.)\n ex1)\n import snipy.database as db\n conn = db.connect('my_db')\n cursor = conn.cursor()\n\n ex2)\n import snipy.database as db\n with db.connect('my_db') as cursor:\n cursor.execute(query)\n\n :param db: str: db schema\n :param kwargs: \ucd94\uac00 \uc811\uc18d \uc815\ubcf4\n :return: connection or cursor"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _cursor_exit(cursor, exc_type, exc_value, traceback):\n if exc_type is not None:\n print(exc_value, traceback)\n cursor.connection.close()", "response": "Close the connection to the database."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching \uc774\ubca4\ud2b8 \uc2a4\ud2b8\ub9c1 \uc2a4\ud2b8", "response": "def fetch(query, args=None, **kwargs):\n \"\"\"\n for record in fetch(query, args, **configs):\n print record\n :param args:\n :param db: str: db \uc2a4\ud0a4\ub9c8\n :param query: \ucffc\ub9ac \uc2a4\ud2b8\ub9c1\n :param kwargs: db connection \ucd94\uac00 \uc778\uc790. \ubcf4\ud1b5 \uc0dd\ub7b5\n :return: iterator\n \"\"\"\n cur = execute(kwargs.pop('db', ''), query, args, **kwargs)\n\n for r in cur:\n yield r\n\n cur.connection.close()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nformat insert query :param table: str :param fields: list[str] :param field_count: int :return: str", "response": "def get_insert_query(table, fields=None, field_count=None):\n \"\"\"\n format insert query\n :param table: str\n :param fields: list[str]\n :param field_count: int\n :return: str\n \"\"\"\n if fields:\n q = 'insert into %s ({0}) values ({1});' % table\n l = len(fields)\n q = q.format(','.join(fields), ','.join(['%s'] * l))\n elif field_count:\n q = 'insert into %s values ({0});' % table\n q = q.format(','.join(['%s'] * field_count))\n else:\n raise ValueError('fields or field_count need')\n\n return q"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninsert a new record in the specified table.", "response": "def insert(cursor, table, *args, **field_values):\n \"\"\"\n db\uc5d0 \ub808\ucf54\ub4dc \uc9d1\uc5b4\ub123\uae30\n ex)\n cursor.insert(table, v1, v2,...)\n ex)\n cursor.insert(table, id=v1, word=v2, commit=True)\n :param commit:\n :param cursor:\n :param table:\n :param args:\n :param field_values:\n :return:\n \"\"\"\n commit = field_values.pop('commit', True)\n\n q, a = None, None\n if args is not None and len(args) > 0:\n q = get_insert_query(table, field_count=len(args))\n a = args\n elif len(field_values) > 0:\n q = get_insert_query(table, fields=field_values.keys())\n a = field_values.values()\n else:\n raise ValueError('need table, record...')\n\n cursor.execute(q, args=a)\n if commit:\n cursor.connection.commit()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninserting or update \uc774\ub9ac \ubc1c\uc2e4\ud589", "response": "def insert_or_update(cursor, table, commit=True, **field_values):\n \"\"\"\n db update \ucffc\ub9ac \ube4c\ub529 \ubc0f \uc2e4\ud589, \ub2e8, commit\uc740\n :param cursor: \ucee4\uc11c\n :type cursor: Cursor\n :param table: \ud14c\uc774\ube14\uc774\ub984\n :type table: str\n :param commit: \ucee4\ubc0b \uc5ec\ubd80\n :type commit: bool\n :param field_values: insert \ub610\ub294 \uc5c5\ub370\uc774\ud2b8 \ud560 \ud544\ub4dc \ubc0f \uac12 dict pairs\n :type field_values:dict\n :return:\n \"\"\"\n\n q = \"\"\"INSERT INTO %s ({0}) \\nVALUES ({1}) \\nON DUPLICATE KEY UPDATE {2} \"\"\" % table\n l = len(field_values)\n\n fields = field_values.keys()\n field = ','.join(fields)\n value = ','.join(['%s'] * l)\n kv = ','.join(['{}=%s'.format(f) for f in fields])\n\n q = q.format(field, value, kv)\n args = field_values.values() * 2\n\n cursor.execute(q, args=args)\n if commit:\n cursor.connection.commit()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tojson(o):\n try:\n return json.encode(o)\n except json.EncodeError:\n pass\n try:\n return o.tojson()\n except AttributeError as e:\n pass\n\n t = type(o)\n if isinstance(o, list):\n return '[%s]' % ', '.join([tojson(e) for e in o])\n elif isinstance(o, dict):\n d = ['%s:%s' % (k, tojson(v)) for k, v in o.iteritems()]\n return '{%s}' % ', '.join(d)\n elif isinstance(o, set):\n d = ['%s:%s' % (tojson(e)) for e in o]\n return '{%s}' % ', '.join(d)\n elif isinstance(o, np.ndarray):\n return numpy_to_json(o)\n else:\n raise ValueError('error, failed encoding type(%s) to json' % t)", "response": "Convert object to json string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef named(typename, *fieldnames, **defaults):\n if len(fieldnames) == 1:\n if isinstance(fieldnames[0], str):\n fieldnames = tuple(fieldnames[0].replace(',', ' ').split())\n elif isinstance(fieldnames[0], (list, tuple)):\n fieldnames = fieldnames[0]\n\n # set default of defaults\n default_of_defaults = defaults.pop('default', None)\n\n dfields = tuple(f for f in defaults if f not in fieldnames)\n\n T = collections.namedtuple(typename, fieldnames + dfields)\n T.__new__.__defaults__ = (default_of_defaults,) * len(T._fields)\n prototype = T(**defaults)\n T.__new__.__defaults__ = tuple(prototype)\n\n # make picklable\n globals()[typename] = T\n\n return T", "response": "named tuple with default values"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef np_seed(seed):\n if seed is not None:\n state = np.random.get_state()\n np.random.seed(seed)\n yield\n np.random.set_state(state)\n\n else:\n yield", "response": "numpy random seed context\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef connect(self):\n try:\n self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n self._socket.settimeout(TIMEOUT_SECONDS)\n self._socket.connect((self._ip, self._port))\n _LOGGER.debug(\"Successfully created Hub at %s:%s :)\", self._ip,\n self._port)\n except socket.error as error:\n _LOGGER.error(\"Error creating Hub: %s :(\", error)\n self._socket.close()", "response": "Create and connect to socket for TCP communication with hub."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send_command(self, command):\n # use lock to make TCP send/receive thread safe\n with self._lock:\n try:\n self._socket.send(command.encode(\"utf8\"))\n result = self.receive()\n # hub may send \"status\"/\"new\" messages that should be ignored\n while result.startswith(\"S\") or result.startswith(\"NEW\"):\n _LOGGER.debug(\"!Got response: %s\", result)\n result = self.receive()\n _LOGGER.debug(\"Received: %s\", result)\n return result\n except socket.error as error:\n _LOGGER.error(\"Error sending command: %s\", error)\n # try re-connecting socket\n self.connect()\n return \"\"", "response": "Send command to hub and return response."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef receive(self):\n try:\n buffer = self._socket.recv(BUFFER_SIZE)\n except socket.timeout as error:\n # Something is wrong, assume it's offline temporarily\n _LOGGER.error(\"Error receiving: %s\", error)\n # self._socket.close()\n return \"\"\n\n # Read until a newline or timeout\n buffering = True\n response = ''\n while buffering:\n if '\\n' in buffer.decode(\"utf8\"):\n response = buffer.decode(\"utf8\").split('\\n')[0]\n buffering = False\n else:\n try:\n more = self._socket.recv(BUFFER_SIZE)\n except socket.timeout:\n more = None\n if not more:\n buffering = False\n response = buffer.decode(\"utf8\")\n else:\n buffer += more\n return response", "response": "Receive TCP response, looping to get whole thing or timeout."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget current light data as dictionary with light zids as keys.", "response": "def get_data(self):\n \"\"\"Get current light data as dictionary with light zids as keys.\"\"\"\n response = self.send_command(GET_LIGHTS_COMMAND)\n _LOGGER.debug(\"get_data response: %s\", repr(response))\n if not response:\n _LOGGER.debug(\"Empty response: %s\", response)\n return {}\n response = response.strip()\n # Check string before splitting (avoid IndexError if malformed)\n if not (response.startswith(\"GLB\") and response.endswith(\";\")):\n _LOGGER.debug(\"Invalid response: %s\", repr(response))\n return {}\n\n # deconstruct response string into light data. Example data:\n # GLB 143E,1,1,25,255,255,255,0,0;287B,1,1,22,255,255,255,0,0;\\r\\n\n response = response[4:-3] # strip start (GLB) and end (;\\r\\n)\n light_strings = response.split(';')\n light_data_by_id = {}\n for light_string in light_strings:\n values = light_string.split(',')\n try:\n light_data_by_id[values[0]] = [int(values[2]), int(values[4]),\n int(values[5]), int(values[6]),\n int(values[7])]\n except ValueError as error:\n _LOGGER.error(\"Error %s: %s (%s)\", error, values, response)\n except IndexError as error:\n _LOGGER.error(\"Error %s: %s (%s)\", error, values, response)\n return light_data_by_id"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget current light data set and return as list of Bulb objects.", "response": "def get_lights(self):\n \"\"\"Get current light data, set and return as list of Bulb objects.\"\"\"\n # Throttle updates. Use cached data if within UPDATE_INTERVAL_SECONDS\n now = datetime.datetime.now()\n if (now - self._last_updated) < datetime.timedelta(\n seconds=UPDATE_INTERVAL_SECONDS):\n # _LOGGER.debug(\"Using cached light data\")\n return self._bulbs\n else:\n self._last_updated = now\n\n light_data = self.get_data()\n _LOGGER.debug(\"got: %s\", light_data)\n if not light_data:\n return []\n\n if self._bulbs:\n # Bulbs already created, just update values\n for bulb in self._bulbs:\n # use the values for the bulb with the correct ID\n try:\n values = light_data[bulb.zid]\n bulb._online, bulb._red, bulb._green, bulb._blue, \\\n bulb._level = values\n except KeyError:\n pass\n else:\n for light_id in light_data:\n self._bulbs.append(Bulb(self, light_id, *light_data[light_id]))\n # return a list of Bulb objects\n return self._bulbs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the color property as list of R G B each 0 - 255.", "response": "def rgb_color(self):\n \"\"\"Return the color property as list of [R, G, B], each 0-255.\"\"\"\n self.update()\n return [self._red, self._green, self._blue]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nturns bulb on ( full brightness ).", "response": "def turn_on(self):\n \"\"\"Turn bulb on (full brightness).\"\"\"\n command = \"C {},,,,100,\\r\\n\".format(self._zid)\n response = self._hub.send_command(command)\n _LOGGER.debug(\"Turn on %s: %s\", repr(command), response)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the brightness of the bulb.", "response": "def set_brightness(self, brightness):\n \"\"\"Set brightness of bulb.\"\"\"\n command = \"C {},,,,{},\\r\\n\".format(self._zid, brightness)\n response = self._hub.send_command(command)\n _LOGGER.debug(\"Set brightness %s: %s\", repr(command), response)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset color and brightness of bulb.", "response": "def set_all(self, red, green, blue, brightness):\n \"\"\"Set color and brightness of bulb.\"\"\"\n command = \"C {},{},{},{},{},\\r\\n\".format(self._zid, red, green, blue,\n brightness)\n response = self._hub.send_command(command)\n _LOGGER.debug(\"Set all %s: %s\", repr(command), response)\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates light objects to their current values.", "response": "def update(self):\n \"\"\"Update light objects to their current values.\"\"\"\n bulbs = self._hub.get_lights()\n if not bulbs:\n _LOGGER.debug(\"%s is offline, send command failed\", self._zid)\n self._online = False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef functional(ifunctional):\n\n @wraps(ifunctional)\n def wrapper(fn, *args, **kw):\n\n fn = ifunctional(fn)\n if args or kw:\n return fn(*args, **kw)\n else:\n return fn\n\n return wrapper", "response": "Decorator for functional functions."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tuple_args(fn):\n\n @wraps(fn)\n def wrapped(*args, **kwargs):\n\n if len(args) == 1:\n if isinstance(args[0], tuple):\n return fn(args[0], **kwargs)\n elif isinstance(args[0], list):\n return fn(tuple(args[0]), **kwargs)\n return fn(args, **kwargs)\n\n return wrapped", "response": "Function to wrap a function to return a tuple of arguments."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef optional_str(deco):\n\n @wraps(deco)\n def dispatcher(*args, **kwargs):\n\n # when only function arg\n if not kwargs and len(args) == 1 and not isinstance(args[0], str) \\\n and args[0] is not None:\n decorator = deco()\n return decorator(args[0])\n else:\n # decorator with args\n decorator = deco(*args, **kwargs)\n return decorator\n\n return dispatcher", "response": "A function decorator that can be used to decorate a string in a sequence."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef patchproperty(*cls, **kwargs):\n\n def _patch(fun):\n m = kwargs.pop('property', None) or fun.__name__\n p = property(fun)\n for c in cls:\n setattr(c, m, p)\n\n def wrap(fun):\n _patch(fun)\n return fun\n\n return wrap", "response": "A decorator to patch the property of the class cls with the new value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_interrupt(handler, reraise=False):\n\n def _handler(sig, frame):\n handler.signal = (sig, frame)\n handler._reraise = handler()\n\n handler._reraise = False\n handler.signal = None\n oldhandler = signal.getsignal(signal.SIGINT)\n signal.signal(signal.SIGINT, _handler)\n\n yield handler\n\n signal.signal(signal.SIGINT, oldhandler)\n if (reraise or handler._reraise) and handler.signal:\n oldhandler(*handler.signal)", "response": "context manager for handling keyboardinterrupt on_interrupt"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if main thread is alive False otherwise", "response": "def is_main_alive():\n \"\"\"\n is \uba54\uc778 \uc4f0\ub808\ub4dc alive?\n :rtype: bool\n \"\"\"\n\n for t in threading.enumerate():\n if t.name == 'MainThread':\n return t.is_alive()\n print('MainThread not found')\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef action(self, item):\n fun, args, kwargs = item\n return fun(*args, **kwargs)", "response": "for overriding\n :param item:\n :return:"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef push_job(self, fun, *args, **kwargs):\n assert callable(fun)\n return self.put((fun, args, kwargs), block=True)", "response": "Pushes a job to the cache if possible."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef put_job(self, fun, *args, **kwargs):\n if not args and not kwargs and isinstance(fun, (tuple, list)):\n # ex) q.put_job([fun, args, kwargs])\n fun, args, kwargs = fun\n\n assert callable(fun)\n return self.put((fun, args, kwargs), block=False)", "response": "Put a job if possible non - blocking"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_flag(*args, **kwargs):\n if len(args) == 1 and isinstance(args[0], (list, tuple)):\n for a in args[0]:\n flag.add_flag(*a)\n elif args:\n flag.add_flag(*args, **kwargs)\n else:\n for f, v in kwargs.items():\n flag.add_flag(f, v)", "response": "Adds a flag to the current node."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning the program with an optional main function and argv list.", "response": "def run(main=None, argv=None, **flags):\n \"\"\"\n :param main: main or sys.modules['__main__'].main\n :param argv: argument list used in argument parse\n :param flags: flags to define with defaults\n :return:\n \"\"\"\n \"\"\"Runs the program with an optional 'main' function and 'argv' list.\"\"\"\n import sys as _sys\n import inspect\n main = main or _sys.modules['__main__'].main\n\n if main.__doc__:\n docstring = main.__doc__.split(':param')[0]\n _parser.usage = 'from docstring \\n {}'.format(docstring) # add_help\n\n # if not flags:\n try:\n a = inspect.getfullargspec(main)\n except AttributeError:\n a = inspect.getargspec(main) # namedtuple(args, varargs, keywords, defaults)\n if a.defaults:\n kwargs = dict(zip(reversed(a.args), reversed(a.defaults)))\n add_flag(**kwargs)\n else:\n kwargs = dict()\n\n # add to command argument\n if a.defaults is None:\n nargs = len(a.args)\n else:\n nargs = len(a.args) - len(a.defaults)\n # if nargs > 0:\n posargs = a.args[:nargs]\n flag.add_args(posargs)\n add_flag(**flags)\n\n # Extract the args from the optional `argv` list.\n args = argv[1:] if argv else None\n\n # Parse the known flags from that list, or from the command\n # line otherwise.\n unparsed, kw = flag._parse_flags_kw(args=args)\n\n d = flag.__dict__['__flags']\n args = [d[k] for k in posargs]\n args += unparsed\n\n kwargs.update({k: d[k] for k in kwargs.keys()})\n kwargs.update(kw)\n\n # Call the main function, passing through any arguments,\n # with parsed flags as kwwargs\n # to the final program.\n _sys.exit(main(*args, **kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads lines from a textfile", "response": "def readlines(filepath):\n \"\"\"\n read lines from a textfile\n :param filepath:\n :return: list[line]\n \"\"\"\n with open(filepath, 'rt') as f:\n lines = f.readlines()\n lines = map(str.strip, lines)\n lines = [l for l in lines if l]\n return lines"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread file as is", "response": "def readtxt(filepath):\n \"\"\" read file as is\"\"\"\n with open(filepath, 'rt') as f:\n lines = f.readlines()\n return ''.join(lines)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef savefile(obj, filepath, compress=True):\n try:\n import cPickle as pickle\n except Exception:\n import pickle\n import joblib\n\n # \uc77c\ub2e8 \uc784\uc2dc \ud30c\uc77c\uc5d0 \uc800\uc7a5.\n tmpfile = filepath + '.tmp'\n mkdir_if_not(tmpfile)\n if compress:\n joblib.dump(obj, tmpfile, compress=3, cache_size=100, protocol=pickle.HIGHEST_PROTOCOL)\n else:\n joblib.dump(obj, tmpfile, compress=0)\n\n os.rename(tmpfile, filepath)\n\n return obj", "response": "Save object to file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef loadfile(filepath, mmap_mode=None):\n import joblib\n\n try:\n return joblib.load(filepath, mmap_mode=mmap_mode)\n except IOError:\n return None", "response": "Load a single node from a file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_or_run(filepath, fun, *args, **kwargs):\n force = kwargs.pop('force', False)\n compress = kwargs.pop('compress', True)\n\n if not filepath.startswith('/') or not filepath.startswith('~'):\n filepath = os.path.join('/tmp/snipy/load_or_run/', filepath)\n\n if not force and os.path.exists(filepath):\n # \uc800\uc7a5\ub418\uc5b4 \uc788\ub294 \uac83 \ub85c\ub529\n mmap_mode = 'r+' if not compress else None\n return loadfile(filepath, mmap_mode=mmap_mode)\n\n res = fun(*args, **kwargs)\n savefile(res, filepath, compress=compress)\n\n return res", "response": "Load file or run"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fnmatches(fname, patterns, matchfun):\n import fnmatch\n matchfun = matchfun or fnmatch.fnmatch\n for p in patterns:\n yield matchfun(fname, p)", "response": "matches?\n :param fname: file name\n :type fname: str\n :param patterns: list of filename pattern. see fnmatch.fnamtch\n :type patterns: [str]\n :rtype: generator of bool"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef listdir(p, match='*', exclude='', listtype='file', matchfun=None):\n if listtype == 'file':\n gen = listfile(p)\n elif listtype == 'filepath':\n gen = listfilepath(p)\n elif listtype == 'dir':\n gen = listfolder(p)\n elif listtype == 'dirpath':\n gen = listfolderpath(p)\n else: # list file or folder\n gen = (entry.name for entry in scandir.scandir(p))\n\n return filter_pattern(gen, match, exclude, matchfun)", "response": "list file or folder for this path"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef listfile(p):\n try:\n for entry in scandir.scandir(p):\n if entry.is_file():\n yield entry.name\n except OSError:\n return", "response": "generator of list files in the path"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef findfolder(toppath, match='*', exclude=''):\n pred = _pred_pattern(match, exclude)\n\n return (p for p in walkfolder(toppath, pred))", "response": "recursive function to find folder path from toppath."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwalking folder if pred(foldername) is True :type toppath: str :type pred: function(str) => bool", "response": "def walkfolder(toppath, pred):\n \"\"\"\n walk folder if pred(foldername) is True\n :type toppath: str\n :type pred: function(str) => bool\n \"\"\"\n for entry in scandir.scandir(toppath):\n if not entry.is_dir() or not pred(entry.name):\n continue\n yield entry.path\n for p in walkfolder(entry.path, pred):\n yield p"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a temporary folder", "response": "def tempfolder(prefix=''):\n \"\"\"\uc784\uc2dc \ud3f4\ub354\ub97c \ub9cc\ub4e4\uc5b4\uc11c \ub9ac\ud134\"\"\"\n import uuid\n\n p = prefix + str(uuid.uuid4())\n d = tempdir()\n tmpd = os.path.join(d, p)\n return mkdir_if_not(tmpd, ispath=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn image size (height, width) :param fname: :return:", "response": "def imsize(fname):\n \"\"\"\n return image size (height, width)\n :param fname:\n :return:\n \"\"\"\n from PIL import Image\n im = Image.open(fname)\n return im.size[1], im.size[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef intersect(self, other):\n return DictObj({k: self[k] for k in self if k in other})", "response": "Return dictobj with keys that are in self and other."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_dict(dic):\n return DictObj({k: DictObj.convert_ifdic(v) for k, v in dic.items()})", "response": "Convert a recursive dict to dictobj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclean up after ourselves removing files created by create .", "response": "def _clean_up(paths):\n \"\"\"\n Clean up after ourselves, removing created files.\n @param {[String]} A list of file paths specifying the files we've created\n during run. Will all be deleted.\n @return {None}\n \"\"\"\n print('Cleaning up')\n # Iterate over the given paths, unlinking them\n for path in paths:\n print('Removing %s' % path)\n os.unlink(path)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an index file in the given location supplying known lists of image files and subdirectories.", "response": "def _create_index_file(\n root_dir, location, image_files, dirs, force_no_processing=False):\n \"\"\"\n Create an index file in the given location, supplying known lists of\n present image files and subdirectories.\n @param {String} root_dir - The root directory of the entire crawl. Used to\n ascertain whether the given location is the top level.\n @param {String} location - The current directory of the crawl. The index\n file will be created here.\n @param {[String]} image_files - A list of image file names in the location.\n These will be displayed in the index file's gallery.\n @param {[String]} dirs - The subdirectories of the location directory.\n These will be displayed as links further down the file structure.\n @param {Boolean=False} force_no_processing - If True, do not attempt to\n actually process thumbnails, PIL images or anything. Simply index\n tags with original file src attributes.\n @return {String} The full path (location plus filename) of the newly\n created index file. Intended for usage cleaning up created files.\n \"\"\"\n # Put together HTML as a list of the lines we'll want to include\n # Issue #2 exists to do this better than HTML in-code\n header_text = \\\n 'imageMe: ' + location + ' [' + str(len(image_files)) + ' image(s)]'\n html = [\n '',\n '',\n ' ',\n ' imageMe'\n ' ',\n ' ',\n ' ',\n '
',\n '

' + header_text + '

'\n ]\n # Populate the present subdirectories - this includes '..' unless we're at\n # the top level\n directories = []\n if root_dir != location:\n directories = ['..']\n directories += dirs\n if len(directories) > 0:\n html.append('
')\n # For each subdirectory, include a link to its index file\n for directory in directories:\n link = directory + '/' + INDEX_FILE_NAME\n html += [\n '

',\n ' ' + directory + '',\n '

'\n ]\n # Populate the image gallery table\n # Counter to cycle down through table rows\n table_row_count = 1\n html += ['
', '']\n # For each image file, potentially create a new and create a new ')\n img_src = _get_thumbnail_src_from_file(\n location, image_file, force_no_processing\n )\n link_target = _get_image_link_target_from_file(\n location, image_file, force_no_processing\n )\n html += [\n ' '\n ]\n if table_row_count == IMAGES_PER_ROW:\n table_row_count = 0\n html.append('')\n table_row_count += 1\n html += ['', '
\n for image_file in image_files:\n if table_row_count == 1:\n html.append('
',\n ' ',\n ' ',\n ' ',\n '
']\n html += [\n '
',\n ' ',\n ''\n ]\n # Actually create the file, now we've put together the HTML content\n index_file_path = _get_index_file_path(location)\n print('Creating index file %s' % index_file_path)\n index_file = open(index_file_path, 'w')\n index_file.write('\\n'.join(html))\n index_file.close()\n # Return the path for cleaning up later\n return index_file_path"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _create_index_files(root_dir, force_no_processing=False):\n # Initialise list of created file paths to build up as we make them\n created_files = []\n # Walk the root dir downwards, creating index files as we go\n for here, dirs, files in os.walk(root_dir):\n print('Processing %s' % here)\n # Sort the subdirectories by name\n dirs = sorted(dirs)\n # Get image files - all files in the directory matching IMAGE_FILE_REGEX\n image_files = [f for f in files if re.match(IMAGE_FILE_REGEX, f)]\n # Sort the image files by name\n image_files = sorted(image_files)\n # Create this directory's index file and add its name to the created\n # files list\n created_files.append(\n _create_index_file(\n root_dir, here, image_files, dirs, force_no_processing\n )\n )\n # Return the list of created files\n return created_files", "response": "Create the index files in each node in the directory root_dir and return the list of created files."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets an instance of PIL. Image from the given file.", "response": "def _get_image_from_file(dir_path, image_file):\n \"\"\"\n Get an instance of PIL.Image from the given file.\n @param {String} dir_path - The directory containing the image file\n @param {String} image_file - The filename of the image file within dir_path\n @return {PIL.Image} An instance of the image file as a PIL Image, or None\n if the functionality is not available. This could be because PIL is not\n present, or because it can't process the given file type.\n \"\"\"\n # Save ourselves the effort if PIL is not present, and return None now\n if not PIL_ENABLED:\n return None\n # Put together full path\n path = os.path.join(dir_path, image_file)\n # Try to read the image\n img = None\n try:\n img = Image.open(path)\n except IOError as exptn:\n print('Error loading image file %s: %s' % (path, exptn))\n # Return image or None\n return img"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the value to be used as the href for links from thumbnail images.", "response": "def _get_image_link_target_from_file(dir_path, image_file, force_no_processing=False):\n \"\"\"\n Get the value to be used as the href for links from thumbnail images. For\n most image formats this will simply be the image file name itself. However,\n some image formats (tif) are not natively displayable by many browsers and\n therefore we must link to image data in another format.\n @param {String} dir_path - The directory containing the image file\n @param {String} image_file - The filename of the image file within dir_path\n @param {Boolean=False} force_no_processing - If True, do not attempt to\n actually process a thumbnail, PIL image or anything. Simply return the\n image filename as src.\n @return {String} The href to use.\n \"\"\"\n # If we've specified to force no processing, just return the image filename\n if force_no_processing:\n return image_file\n # First try to get an image\n img = _get_image_from_file(dir_path, image_file)\n # If format is directly displayable in-browser, just return the filename\n # Else, we need to return a full-sized chunk of displayable image data\n if img.format.lower() in ['tif', 'tiff']:\n return _get_image_src_from_file(\n dir_path, image_file, force_no_processing\n )\n return image_file"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_image_src_from_file(dir_path, image_file, force_no_processing=False):\n # If we've specified to force no processing, just return the image filename\n if force_no_processing:\n if image_file.endswith('tif') or image_file.endswith('tiff'):\n return UNSUPPORTED_IMAGE_TYPE_DATA\n return image_file\n # First try to get an image\n img = _get_image_from_file(dir_path, image_file)\n return _get_src_from_image(img, image_file)", "response": "Get the base - 64 encoded data for the given image file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget base - 64 encoded data as a string for the given image. Fallback to return fallback_image_file", "response": "def _get_src_from_image(img, fallback_image_file):\n \"\"\"\n Get base-64 encoded data as a string for the given image. Fallback to return\n fallback_image_file if cannot get the image data or img is None.\n @param {Image} img - The PIL Image to get src data for\n @param {String} fallback_image_file - The filename of the image file,\n to be used when image data capture fails\n @return {String} The base-64 encoded image data string, or path to the file\n itself if not supported.\n \"\"\"\n # If the image is None, then we can't process, so we should return the\n # path to the file itself\n if img is None:\n return fallback_image_file\n # Target format should be the same as the original image format, unless it's\n # a TIF/TIFF, which can't be displayed by most browsers; we convert these\n # to jpeg\n target_format = img.format\n if target_format.lower() in ['tif', 'tiff']:\n target_format = 'JPEG'\n # If we have an actual Image, great - put together the base64 image string\n try:\n bytesio = io.BytesIO()\n img.save(bytesio, target_format)\n byte_value = bytesio.getvalue()\n b64 = base64.b64encode(byte_value)\n return 'data:image/%s;base64,%s' % (target_format.lower(), b64)\n except IOError as exptn:\n print('IOError while saving image bytes: %s' % exptn)\n return fallback_image_file"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_thumbnail_image_from_file(dir_path, image_file):\n # Get image\n img = _get_image_from_file(dir_path, image_file)\n # If it's not supported, exit now\n if img is None:\n return None\n if img.format.lower() == 'gif':\n return None\n # Get image dimensions\n img_width, img_height = img.size\n # We need to perform a resize - first, work out the scale ratio to take the\n # image width to THUMBNAIL_WIDTH (THUMBNAIL_WIDTH:img_width ratio)\n scale_ratio = THUMBNAIL_WIDTH / float(img_width)\n # Work out target image height based on the scale ratio\n target_height = int(scale_ratio * img_height)\n # Perform the resize\n try:\n img.thumbnail((THUMBNAIL_WIDTH, target_height), resample=RESAMPLE)\n except IOError as exptn:\n print('WARNING: IOError when thumbnailing %s/%s: %s' % (\n dir_path, image_file, exptn\n ))\n return None\n # Return the resized image\n return img", "response": "Get a PIL. Image from the given image file which has been scaled down to\n THUMBNAIL_WIDTH wide."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_thumbnail_src_from_file(dir_path, image_file, force_no_processing=False):\n # If we've specified to force no processing, just return the image filename\n if force_no_processing:\n if image_file.endswith('tif') or image_file.endswith('tiff'):\n return UNSUPPORTED_IMAGE_TYPE_DATA\n return image_file\n # First try to get a thumbnail image\n img = _get_thumbnail_image_from_file(dir_path, image_file)\n return _get_src_from_image(img, image_file)", "response": "Get the base - 64 encoded data for the given image file s thumbnail and the image filename as src."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _run_server():\n # Get the port to run on\n port = _get_server_port()\n # Configure allow_reuse_address to make re-runs of the script less painful -\n # if this is not True then waiting for the address to be freed after the\n # last run can block a subsequent run\n SocketServer.TCPServer.allow_reuse_address = True\n # Create the server instance\n server = SocketServer.TCPServer(\n ('', port),\n SimpleHTTPServer.SimpleHTTPRequestHandler\n )\n # Print out before actually running the server (cheeky / optimistic, however\n # you want to look at it)\n print('Your images are at http://127.0.0.1:%d/%s' % (\n port,\n INDEX_FILE_NAME\n ))\n # Try to run the server\n try:\n # Run it - this call blocks until the server is killed\n server.serve_forever()\n except KeyboardInterrupt:\n # This is the expected way of the server being killed, since imageMe is\n # intended for ad-hoc running from command line\n print('User interrupted, stopping')\n except Exception as exptn:\n # Catch everything else - this will handle shutdowns via other signals\n # and faults actually starting the server in the first place\n print(exptn)\n print('Unhandled exception in server, stopping')", "response": "Runs the image server. This is blocking. Will handle user KeyboardInterrupt and other exceptions appropriately and return control once the server is stopped."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating indexes and run server from the given directory downwards.", "response": "def serve_dir(dir_path):\n \"\"\"\n Generate indexes and run server from the given directory downwards.\n @param {String} dir_path - The directory path (absolute, or relative to CWD)\n @return {None}\n \"\"\"\n # Create index files, and store the list of their paths for cleanup later\n # This time, force no processing - this gives us a fast first-pass in terms\n # of page generation, but potentially slow serving for large image files\n print('Performing first pass index file generation')\n created_files = _create_index_files(dir_path, True)\n if (PIL_ENABLED):\n # If PIL is enabled, we'd like to process the HTML indexes to include\n # generated thumbnails - this slows down generation so we don't do it\n # first time around, but now we're serving it's good to do in the\n # background\n print('Performing PIL-enchanced optimised index file generation in background')\n background_indexer = BackgroundIndexFileGenerator(dir_path)\n background_indexer.run()\n # Run the server in the current location - this blocks until it's stopped\n _run_server()\n # Clean up the index files created earlier so we don't make a mess of\n # the image directories\n _clean_up(created_files)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting caller s modulename", "response": "def modulename(cls, depth=1):\n \"\"\"\n get caller's __name__\n \"\"\"\n depth += cls.extra_depth\n frame = sys._getframe(depth)\n return frame.f_globals['__name__']"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deco_optional(decorator):\n\n @functools.wraps(decorator)\n def dispatcher(*args, **kwargs):\n one_arg = len(args) == 1 and not kwargs\n if one_arg and inspect.isfunction(args[0]):\n decor_obj = decorator()\n return decor_obj(args[0])\n else:\n return decorator(*args, **kwargs)\n\n return dispatcher", "response": "Decorator to be used as a decorator for optional arguments."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef optional(deco):\n @functools.wraps(deco)\n def dispatcher(*args, **kwargs):\n decorator = deco(**kwargs)\n if args:\n assert len(args) == 1\n return decorator(args[0])\n else:\n return decorator\n return dispatcher", "response": "A decorator that returns a function that takes a single argument and returns the first argument."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbinds a function to a sequence of arguments", "response": "def bindargs(fun, *argsbind, **kwbind):\n \"\"\"\n _ = bind.placeholder # unbound placeholder (arg)\n f = bind(fun, _, _, arg3, kw=kw1, kw2=kw2), f(arg1, arg2)\n :param fun:\n :param argsbind:\n :param kwbind:\n :return:\n \"\"\"\n\n assert argsbind\n argsb = list(argsbind)\n iargs = [i for i in range(len(argsbind)) if argsbind[i] is bind.placeholder]\n # iargs = [a is bind.placeholder for a in argsbind]\n\n @functools.wraps(fun)\n def wrapped(*args, **kwargs):\n kws = kwbind.copy()\n\n args_this = [a for a in argsb]\n for i, arg in zip(iargs, args):\n args_this[i] = arg\n args_this.extend(args[len(iargs):])\n\n # kwargs.update(kwbind)\n kws.update(kwargs)\n # return fun(*argsb, **kws)\n return fun(*args_this, **kws)\n\n return wrapped"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef default(fun, **kwdefault):\n\n @functools.wraps(fun)\n def wrapped(*args, **kwargs):\n merge = wrapped.default.copy()\n merge.update(kwargs)\n return fun(*args, **merge)\n\n wrapped.default = kwdefault\n\n return wrapped", "response": "change default value for function\n ex"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef setup_once(initfn):\n def wrap(method):\n\n finit = initfn.__name__\n fnname = method.__name__\n\n @functools.wraps(method)\n def wrapped(self, *args, **kwargs):\n\n @functools.wraps(method)\n def aftersetup(*a, **kw):\n return method(self, *a, **kw)\n\n setupfn = getattr(self, finit)\n setupfn(*args, **kwargs)\n\n res = method(self, *args, **kwargs)\n setattr(self, fnname, aftersetup)\n return res\n return wrapped\n\n return wrap", "response": "A decorator that makes sure that the initfn is called only once."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rand_crop(sz, *imagez):\n\n def _rand_crop(*imgz):\n imsz = imgz[0].shape[:2]\n\n assert imsz[0] >= sz[0] and imsz[1] >= sz[1]\n\n si = np.random.randint(imsz[0] - sz[0]) if imsz[0] > sz[0] else 0\n sj = np.random.randint(imsz[1] - sz[1]) if imsz[1] > sz[1] else 0\n\n slicei = slice(si, si+sz[0])\n slicej = slice(sj, sj+sz[1])\n\n outs = tuple(img[slicei, slicej] for img in imgz)\n return tuple_or_not(*outs)\n\n return _rand_crop(*imagez) if imagez else _rand_crop", "response": "random crop\n # assume imagez has same size (H, W)\n # assume sz is less or equal than size of image\n :param sz: cropped image sz\n :param imagez: imagez\n :return: rand cropped image pairs or function bound to sz"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef blend_discrete(images, depthmask, depth=None):\n imshape = images.shape\n depth = depth or images.shape[3]\n blend = np.empty(shape=(imshape[0], imshape[1], imshape[2]))\n for d in range(depth):\n imask = (depthmask == d)\n channel = images[..., d]\n blend[imask] = channel[imask]\n return np.expand_dims(blend, axis=-1)", "response": "Blend discrete images into one image at a time."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvectoring parameters :param size: :param vz: :param vscale: :param voctave: :param vpersistence: :param vlacunarity: :return:", "response": "def snoise2dvec(size, *params, **kwargs): #, vlacunarity):\n \"\"\"\n vector parameters\n :param size:\n :param vz:\n :param vscale:\n :param voctave:\n :param vpersistence:\n :param vlacunarity:\n :return:\n \"\"\"\n data = (snoise2d(size, *p, **kwargs) for p in zip(*params)) # , vlacunarity))\n return np.stack(data, 0)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef snoise2d(size, z=0.0, scale=0.05, octaves=1, persistence=0.25, lacunarity=2.0):\n import noise\n data = np.empty(size, dtype='float32')\n for y in range(size[0]):\n for x in range(size[1]):\n v = noise.snoise3(x * scale, y * scale, z,\n octaves=octaves, persistence=persistence, lacunarity=lacunarity)\n data[x, y] = v\n data = data * 0.5 + 0.5\n if __debug__:\n assert data.min() >= 0. and data.max() <= 1.0\n return data", "response": "snoise2d - 2D function for the given size z - value as like a seed\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef snoise2dz(size, z, scale=0.05, octaves=1, persistence=0.25, lacunarity=2.0):\n import noise\n z_l = len(z)\n\n data = np.empty((z_l, size[0], size[1]), dtype='float32')\n for iz in range(z_l):\n zvalue = z[iz]\n for y in range(size[0]):\n for x in range(size[1]):\n v = noise.snoise3(x * scale, y * scale, zvalue,\n octaves=octaves, persistence=persistence, lacunarity=lacunarity)\n data[iz, y, x] = v\n data = data * 0.5 + 0.5\n if __debug__:\n assert data.min() >= 0. and data.max() <= 1.0\n return data", "response": "snoise2dz - 2D z - > numpy array"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rand_brightness(imagez, scale=1.0, randfun=rand.normal(0., .1), clamp=(0., 1.)):\n l, h = clamp\n r = randfun((imagez[0].shape[0], 1, 1, 1)) * scale\n\n def apply(im):\n im += r\n im[im < l] = l\n im[im > h] = h\n return im\n\n return tuple(map(apply, imagez))", "response": "Randomly select a set of images from the imagez array."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef elastic_transform(im, alpha=0.5, sigma=0.2, affine_sigma=1.):\n # fixme : not implemented for multi channel !\n import cv2\n\n islist = isinstance(im, (tuple, list))\n ima = im[0] if islist else im\n\n # image shape\n shape = ima.shape\n shape_size = shape[:2]\n\n # Random affine transform\n center_square = np.float32(shape_size) // 2\n square_size = min(shape_size) // 3\n pts1 = np.float32([center_square + square_size,\n [center_square[0] + square_size, center_square[1] - square_size],\n center_square - square_size])\n pts2 = pts1 + np.random.uniform(-affine_sigma, affine_sigma, size=pts1.shape).astype(np.float32)\n M = cv2.getAffineTransform(pts1, pts2)\n if islist:\n res = []\n for i, ima in enumerate(im):\n if i == 0:\n res.append(cv2.warpAffine(ima, M, shape_size[::-1], borderMode=cv2.BORDER_REFLECT_101))\n else:\n res.append(cv2.warpAffine(ima, M, shape_size[::-1]))\n im = res\n else:\n ima = cv2.warpAffine(ima, M, shape_size[::-1], borderMode=cv2.BORDER_REFLECT_101)\n # ima = cv2.warpAffine(ima, M, shape_size[::-1])\n\n # fast gaussian filter\n blur_size = int(4 * sigma) | 1\n dx = cv2.GaussianBlur((np.random.rand(*shape) * 2 - 1), ksize=(blur_size, blur_size), sigmaX=sigma) * alpha\n dy = cv2.GaussianBlur((np.random.rand(*shape) * 2 - 1), ksize=(blur_size, blur_size), sigmaX=sigma) * alpha\n\n # remap\n x, y = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))\n map_x, map_y = (y + dy).astype('float32'), (x + dx).astype('float32')\n\n def remap(data):\n r = cv2.remap(data, map_y, map_x, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REFLECT_101)\n return r[..., np.newaxis]\n\n if islist:\n return tuple([remap(ima) for ima in im])\n else:\n return remap(ima)", "response": "This function transforms an image into an elastic deformation of images."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrotating and crop the image with the given centerij and angle", "response": "def rotate_crop(centerij, sz, angle, img=None, mode='constant', **kwargs):\n \"\"\"\n rotate and crop\n if no img, then return crop function\n :param centerij:\n :param sz:\n :param angle:\n :param img: [h,w,d]\n :param mode: padding option\n :return: cropped image or function\n \"\"\"\n # crop enough size ( 2 * sqrt(sum(sz^2) )\n # rotate\n from skimage import transform\n sz = np.array(sz)\n crop_half = int(np.ceil(np.sqrt(np.square(sz).sum())))\n\n if centerij[0] >= crop_half or centerij[1] >= crop_half:\n raise NotImplementedError\n\n slicei = slice(centerij[0] - crop_half, centerij[0] + crop_half)\n slicej = slice(centerij[1] - crop_half, centerij[1] + crop_half)\n\n # slicei = (centerij[0] - crop_half, centerij[0] + crop_half)\n # slicej = (centerij[1] - crop_half, centerij[1] + crop_half)\n\n # def _pad_if_need(im):\n # imshape = im.shape\n # pad_need = slicei[0] < 0 or slicej[0] < 0 or slice\n # padwidth = [(slicei[0], np.maximum(0, slicei[1] - imshape[0])),\n # (slicej[0], np.maximum(0, slicej[1] - imshape[1]))]\n\n def _rotate_cropcenter(im):\n enoughcrop = im[slicei, slicej]\n\n rotated = transform.rotate(enoughcrop, angle, resize=False, preserve_range=True, mode=mode, **kwargs)\n return cropcenter(sz, rotated)\n if img is not None:\n return _rotate_cropcenter(img)\n\n return _rotate_cropcenter"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef crop(img, center, sz, mode='constant'):\n center = np.array(center)\n sz = np.array(sz)\n istart = (center - sz / 2.).astype('int32')\n iend = istart + sz\n imsz = img.shape[:2]\n if np.any(istart < 0) or np.any(iend > imsz):\n # padding\n padwidth = [(np.minimum(0, istart[0]), np.maximum(0, iend[0]-imsz[0])),\n (np.minimum(0, istart[1]), np.maximum(0, iend[1]-imsz[1]))]\n padwidth += [(0, 0)] * (len(img.shape) - 2)\n img = np.pad(img, padwidth, mode=mode)\n istart = (np.maximum(0, istart[0]), np.maximum(0, istart[1]))\n\n return img[istart[0]:istart[0]+sz[0], istart[1]:istart[1]+sz[1]]\n\n return img[istart[0]:iend[0], istart[1]:iend[1]]", "response": "crop sz from ij as center"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a function that crops the image with the given size.", "response": "def cropcenter(sz, img=None):\n \"\"\"\n if no img, then return crop function\n :param sz:\n :param img:\n :return:\n \"\"\"\n l = len(sz)\n sz = np.array(sz)\n\n def wrapped(im):\n imsz = np.array(im.shape)\n s = (imsz[:l] - sz) / 2 # start index\n to = s + sz # end index\n\n # img[s[0]:to[0], ... s[end]:to[end], ...]\n slices = [slice(s, e) for s, e in zip(s, to)]\n\n return im[slices]\n\n if img is not None:\n return wrapped(img)\n\n return wrapped"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npads img if need to guarantee minumum size", "response": "def pad_if_need(sz_atleast, img, mode='constant'):\n # fixme : function or ....\n \"\"\"\n pad img if need to guarantee minumum size\n :param sz_atleast: [H,W] at least\n :param img: image np.array [H,W, ...]\n :param mode: str, padding mode\n :return: padded image or asis if enought size\n \"\"\"\n # sz_atleast = np.asarray(sz_atleast)\n imsz = img.shape[:2] # assume img [H,W, ...]\n padneed = np.asarray((sz_atleast[0] - imsz[0], sz_atleast[1] - imsz[1]))\n if np.any(padneed > 0):\n # need padding\n padding = np.zeros((img.ndim, 2), dtype='int16')\n padneed = np.maximum(padneed, 0)\n padding[:2, 0] = padneed/2\n padding[:2, 1] = padneed - padneed/2\n img = np.pad(img, padding, mode=mode)\n\n return img"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nguesses the package path for the given object.", "response": "def guess_package_path(searchfrom):\n \"\"\"\n package path. return None if failed to guess\n \"\"\"\n from snipy.io import fileutil\n\n current = searchfrom + '/'\n init_found = False\n pack_found = False\n while not init_found and current != '/':\n current = os.path.dirname(current)\n initfile = os.path.join(current, '__init__.py')\n init_found = os.path.exists(initfile)\n\n if not init_found:\n # search for breadth\n searchfrom = dirname(searchfrom)\n\n for folder in fileutil.listfolder(searchfrom):\n current = os.path.join(searchfrom, folder)\n initfile = os.path.join(current, '__init__.py')\n init_found = os.path.exists(initfile)\n if init_found:\n break\n\n while init_found:\n current = os.path.dirname(current)\n initfile = os.path.join(current, '__init__.py')\n init_found = os.path.exists(initfile)\n pack_found = not init_found\n\n return current if pack_found else None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_package_path(searchfrom):\n\n current = searchfrom + '/'\n init_found = False\n pack_found = False\n while not init_found and current != '/':\n current = os.path.dirname(current)\n initfile = os.path.join(current, '__init__.py')\n init_found = os.path.exists(initfile)\n\n while init_found:\n current = os.path.dirname(current)\n initfile = os.path.join(current, '__init__.py')\n init_found = os.path.exists(initfile)\n pack_found = not init_found\n\n return current if pack_found else None", "response": "Find the package path of the\n ."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nappending sys path to this_package. py", "response": "def append_this_package_path(depth=1):\n \"\"\"\n this_package.py \uc5d0\uc11c \uc0ac\uc6a9\n import snipy.this_package\n \"\"\"\n from .caller import caller\n\n logg.debug('caller module %s', caller.modulename(depth + 1))\n c = caller.abspath(depth + 1)\n logg.debug('caller path %s', c)\n\n p = guess_package_path(dirname(c))\n if p:\n logg.debug('appending sys path %s', p)\n append_sys_path(p)\n else:\n # do some logging\n logg.debug('failed to guess package path for: %s', c)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef flows(args):\n\n def flow_if_not(fun):\n # t = type(fun)\n if isinstance(fun, iterator):\n return fun\n elif isinstance(fun, type) and 'itertools' in str(fun.__class__):\n return fun\n else:\n try:\n return flow(fun)\n except AttributeError:\n # generator object has no attribute '__module__'\n return fun\n\n return FlowList(map(flow_if_not, args))", "response": "return a FlowList of the passed in flows"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds example :param size: :param iterable: :param rest: :return:", "response": "def ibatch(size, iterable=None, rest=False):\n \"\"\"\n add example\n :param size:\n :param iterable:\n :param rest:\n :return:\n \"\"\"\n\n @iterflow\n def exact_size(it):\n it = iter(it)\n while True:\n yield [it.next() for _ in xrange(size)]\n\n @iterflow\n def at_most(it):\n it = iter(it)\n while True:\n data = []\n for _ in xrange(size):\n try:\n data.append(it.next())\n except StopIteration:\n if data:\n yield data\n raise StopIteration\n yield data\n\n ibatchit = at_most if rest else exact_size\n\n return ibatchit if iterable is None else ibatchit(iterable)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef batchzip(size, iterable=None, rest=False):\n fn = ibatch(size, rest=rest) >> zipflow\n\n return fn if iterable is None else fn(iterable)", "response": "batchzip - returns a sequence of size size"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntaking a size in batches and returns a sequence of n - tuples", "response": "def batchstack(size, iterable=None, rest=False):\n \"\"\"\n todo : add example\n :param size:\n :param iterable:\n :param rest:\n :return:\n \"\"\"\n\n def stack(data):\n import numpy as np\n return map(np.vstack, data)\n\n fn = batchzip(size, rest=rest) >> flow(stack)\n\n return fn if iterable is None else fn(iterable)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef shuffle(qsize=1024, iterable=None):\n\n @iterflow\n def shuffleit(it):\n from random import randrange\n q = []\n\n for i, d in enumerate(it):\n q.insert(randrange(0, len(q) + 1), d)\n if i < qsize:\n continue\n yield q.pop(randrange(0, len(q)))\n\n while q:\n yield q.pop(randrange(0, len(q)))\n\n return shuffleit if iterable is None else shuffleit(iterable)", "response": "Returns an iterator that shuffle the given iterable."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a permutation into a permutation matrix.", "response": "def to_permutation_matrix(matches):\n \"\"\"Converts a permutation into a permutation matrix.\n\n `matches` is a dictionary whose keys are vertices and whose values are\n partners. For each vertex ``u`` and ``v``, entry (``u``, ``v``) in the\n returned matrix will be a ``1`` if and only if ``matches[u] == v``.\n\n Pre-condition: `matches` must be a permutation on an initial subset of the\n natural numbers.\n\n Returns a permutation matrix as a square NumPy array.\n\n \"\"\"\n n = len(matches)\n P = np.zeros((n, n))\n # This is a cleverer way of doing\n #\n # for (u, v) in matches.items():\n # P[u, v] = 1\n #\n P[list(zip(*(matches.items())))] = 1\n return P"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the adjacency matrix of a bipartite graph whose biadjacency matrix is A.", "response": "def to_bipartite_matrix(A):\n \"\"\"Returns the adjacency matrix of a bipartite graph whose biadjacency\n matrix is `A`.\n\n `A` must be a NumPy array.\n\n If `A` has **m** rows and **n** columns, then the returned matrix has **m +\n n** rows and columns.\n\n \"\"\"\n m, n = A.shape\n return four_blocks(zeros(m, m), A, A.T, zeros(n, n))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_pattern_matrix(D):\n result = np.zeros_like(D)\n # This is a cleverer way of doing\n #\n # for (u, v) in zip(*(D.nonzero())):\n # result[u, v] = 1\n #\n result[D.nonzero()] = 1\n return result", "response": "Returns the Boolean matrix in the same shape as D with ones exactly\n where there are nonzero entries in D."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Birkhoff - von Neumann decomposition of the given doubly - stochastic matrix D.", "response": "def birkhoff_von_neumann_decomposition(D):\n \"\"\"Returns the Birkhoff--von Neumann decomposition of the doubly\n stochastic matrix `D`.\n\n The input `D` must be a square NumPy array representing a doubly\n stochastic matrix (that is, a matrix whose entries are nonnegative\n reals and whose row sums and column sums are all 1). Each doubly\n stochastic matrix is a convex combination of at most ``n ** 2``\n permutation matrices, where ``n`` is the dimension of the input\n array.\n\n The returned value is a list of pairs whose length is at most ``n **\n 2``. In each pair, the first element is a real number in the interval **(0,\n 1]** and the second element is a NumPy array representing a permutation\n matrix. This represents the doubly stochastic matrix as a convex\n combination of the permutation matrices.\n\n The input matrix may also be a scalar multiple of a doubly\n stochastic matrix, in which case the row sums and column sums must\n each be *c*, for some positive real number *c*. This may be useful\n in avoiding precision issues: given a doubly stochastic matrix that\n will have many entries close to one, multiply it by a large positive\n integer. The returned permutation matrices will be the same\n regardless of whether the given matrix is a doubly stochastic matrix\n or a scalar multiple of a doubly stochastic matrix, but in the\n latter case, the coefficients will all be scaled by the appropriate\n scalar multiple, and their sum will be that scalar instead of one.\n\n For example::\n\n >>> import numpy as np\n >>> from birkhoff import birkhoff_von_neumann_decomposition as decomp\n >>> D = np.ones((2, 2))\n >>> zipped_pairs = decomp(D)\n >>> coefficients, permutations = zip(*zipped_pairs)\n >>> coefficients\n (1.0, 1.0)\n >>> permutations[0]\n array([[ 1., 0.],\n [ 0., 1.]])\n >>> permutations[1]\n array([[ 0., 1.],\n [ 1., 0.]])\n >>> zipped_pairs = decomp(D / 2) # halve each value in the matrix\n >>> coefficients, permutations = zip(*zipped_pairs)\n >>> coefficients # will be half as large as before\n (0.5, 0.5)\n >>> permutations[0] # will be the same as before\n array([[ 1., 0.],\n [ 0., 1.]])\n >>> permutations[1]\n array([[ 0., 1.],\n [ 1., 0.]])\n\n The returned list of pairs is given in the order computed by the algorithm\n (so in particular they are not sorted in any way).\n\n \"\"\"\n m, n = D.shape\n if m != n:\n raise ValueError('Input matrix must be square ({} x {})'.format(m, n))\n indices = list(itertools.product(range(m), range(n)))\n # These two lists will store the result as we build it up each iteration.\n coefficients = []\n permutations = []\n # Create a copy of D so that we don't modify it directly. Cast the\n # entries of the matrix to floating point numbers, regardless of\n # whether they were integers.\n S = D.astype('float')\n while not np.all(S == 0):\n # Create an undirected graph whose adjacency matrix contains a 1\n # exactly where the matrix S has a nonzero entry.\n W = to_pattern_matrix(S)\n # Construct the bipartite graph whose left and right vertices both\n # represent the vertex set of the pattern graph (whose adjacency matrix\n # is ``W``).\n X = to_bipartite_matrix(W)\n # Convert the matrix of a bipartite graph into a NetworkX graph object.\n G = from_numpy_matrix(X)\n # Compute a perfect matching for this graph. The dictionary `M` has one\n # entry for each matched vertex (in both the left and the right vertex\n # sets), and the corresponding value is its partner.\n #\n # The bipartite maximum matching algorithm requires specifying\n # the left set of nodes in the bipartite graph. By construction,\n # the left set of nodes is {0, ..., n - 1} and the right set is\n # {n, ..., 2n - 1}; see `to_bipartite_matrix()`.\n left_nodes = range(n)\n M = maximum_matching(G, left_nodes)\n # However, since we have both a left vertex set and a right vertex set,\n # each representing the original vertex set of the pattern graph\n # (``W``), we need to convert any vertex greater than ``n`` to its\n # original vertex number. To do this,\n #\n # - ignore any keys greater than ``n``, since they are already\n # covered by earlier key/value pairs,\n # - ensure that all values are less than ``n``.\n #\n M = {u: v % n for u, v in M.items() if u < n}\n # Convert that perfect matching to a permutation matrix.\n P = to_permutation_matrix(M)\n # Get the smallest entry of S corresponding to the 1 entries in the\n # permutation matrix.\n q = min(S[i, j] for (i, j) in indices if P[i, j] == 1)\n # Store the coefficient and the permutation matrix for later.\n coefficients.append(q)\n permutations.append(P)\n # Subtract P scaled by q. After this subtraction, S has a zero entry\n # where the value q used to live.\n S -= q * P\n # PRECISION ISSUE: There seems to be a problem with floating point\n # precision here, so we need to round down to 0 any entry that is very\n # small.\n S[np.abs(S) < TOLERANCE] = 0.0\n return list(zip(coefficients, permutations))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the result of incrementing version.", "response": "def bump_version(version, which=None):\n \"\"\"Returns the result of incrementing `version`.\n\n If `which` is not specified, the \"patch\" part of the version number will be\n incremented. If `which` is specified, it must be ``'major'``, ``'minor'``,\n or ``'patch'``. If it is one of these three strings, the corresponding part\n of the version number will be incremented instead of the patch number.\n\n Returns a string representing the next version number.\n\n Example::\n\n >>> bump_version('2.7.1')\n '2.7.2'\n >>> bump_version('2.7.1', 'minor')\n '2.8.0'\n >>> bump_version('2.7.1', 'major')\n '3.0.0'\n\n \"\"\"\n try:\n parts = [int(n) for n in version.split('.')]\n except ValueError:\n fail('Current version is not numeric')\n if len(parts) != 3:\n fail('Current version is not semantic versioning')\n # Determine where to increment the version number\n PARTS = {'major': 0, 'minor': 1, 'patch': 2}\n index = PARTS[which] if which in PARTS else 2\n # Increment the version number at that index and set the subsequent parts\n # to 0.\n before, middle, after = parts[:index], parts[index], parts[index + 1:]\n middle += 1\n return '.'.join(str(n) for n in before + [middle] + after)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_version(filename, pattern):\n with open(filename) as f:\n match = re.search(r\"^(\\s*%s\\s*=\\s*')(.+?)(')(?sm)\" % pattern, f.read())\n if match:\n before, version, after = match.groups()\n return version\n fail('Could not find {} in {}'.format(pattern, filename))", "response": "Gets the current version from the specified file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fail(message=None, exit_status=None):\n print('Error:', message, file=sys.stderr)\n sys.exit(exit_status or 1)", "response": "Prints the specified message and exits the program with the specified exit status."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntags the current version.", "response": "def git_tag(tag):\n \"\"\"Tags the current version.\"\"\"\n print('Tagging \"{}\"'.format(tag))\n msg = '\"Released version {}\"'.format(tag)\n Popen(['git', 'tag', '-s', '-m', msg, tag]).wait()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninitializing with templates path parameters templates_path str the position of templates directory global_dataSynchronize with global data", "response": "def initialize(self, templates_path, global_data):\n \"\"\"initialize with templates' path\n parameters\n templates_path str the position of templates directory\n global_data dict globa data can be got in any templates\"\"\"\n self.env = Environment(loader=FileSystemLoader(templates_path))\n self.env.trim_blocks = True\n self.global_data = global_data"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nrenders data with template return html unicodes.", "response": "def render(self, template, **data):\n \"\"\"Render data with template, return html unicodes.\n parameters\n template str the template's filename\n data dict the data to render\n \"\"\"\n # make a copy and update the copy\n dct = self.global_data.copy()\n dct.update(data)\n\n try:\n html = self.env.get_template(template).render(**dct)\n except TemplateNotFound:\n raise JinjaTemplateNotFound\n return html"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrender data with template and then write to path", "response": "def render_to(self, path, template, **data):\n \"\"\"Render data with template and then write to path\"\"\"\n html = self.render(template, **data)\n with open(path, 'w') as f:\n f.write(html.encode(charset))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef replace_relative_url_to_absolute(self, content):\n p = os.path.join(os.getcwd(), './src', '../')\n return content.replace('../', p)", "response": "Replace.. / leaded url with absolute uri."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_dataframe(self):\n assert self.dataframe is not None, (\n \"'%s' should either include a `dataframe` attribute, \"\n \"or override the `get_dataframe()` method.\"\n % self.__class__.__name__\n )\n\n dataframe = self.dataframe\n return dataframe", "response": "Get the DataFrame for this view."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef index_row(self, dataframe):\n return dataframe.loc[self.kwargs[self.lookup_url_kwarg]].to_frame().T", "response": "Index the row based on the request parameters."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_object(self):\n dataframe = self.filter_dataframe(self.get_dataframe())\n\n assert self.lookup_url_kwarg in self.kwargs, (\n 'Expected view %s to be called with a URL keyword argument '\n 'named \"%s\". Fix your URL conf, or set the `.lookup_field` '\n 'attribute on the view correctly.' %\n (self.__class__.__name__, self.lookup_url_kwarg)\n )\n\n try:\n obj = self.index_row(dataframe)\n except (IndexError, KeyError, ValueError):\n raise Http404\n\n # May raise a permission denied\n self.check_object_permissions(self.request, obj)\n\n return obj", "response": "Returns the row of the view is displaying."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the paginator instance associated with the view or None if the view is not yet initialized.", "response": "def paginator(self):\n \"\"\"\n The paginator instance associated with the view, or `None`.\n \"\"\"\n if not hasattr(self, '_paginator'):\n if self.pagination_class is None:\n self._paginator = None\n else:\n self._paginator = self.pagination_class()\n return self._paginator"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef paginate_dataframe(self, dataframe):\n if self.paginator is None:\n return None\n return self.paginator.paginate_dataframe(dataframe, self.request, view=self)", "response": "Returns a single page of results."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse(self):\n\n if exists(self.filepath):\n content = open(self.filepath).read().decode(charset)\n else:\n content = \"\"\n\n try:\n config = toml.loads(content)\n except toml.TomlSyntaxError:\n raise ConfigSyntaxError\n\n return config", "response": "parse config return a dict"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef chunks(lst, number):\n lst_len = len(lst)\n\n for i in xrange(0, lst_len, number):\n yield lst[i: i+number]", "response": "A generator that yields a list of size number equal size parts."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates nested dict a with another dict b", "response": "def update_nested_dict(a, b):\n \"\"\"\n update nested dict `a` with another dict b.\n usage::\n\n >>> a = {'x' : { 'y': 1}}\n >>> b = {'x' : {'z':2, 'y':3}, 'w': 4}\n >>> update_nested_dict(a,b)\n {'x': {'y': 3, 'z': 2}, 'w': 4}\n\n \"\"\"\n for k, v in b.iteritems():\n if isinstance(v, dict):\n d = a.setdefault(k, {})\n update_nested_dict(d, v)\n else:\n a[k] = v\n return a"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef render_to(path, template, **data):\n try:\n renderer.render_to(path, template, **data)\n except JinjaTemplateNotFound as e:\n logger.error(e.__doc__ + ', Template: %r' % template)\n sys.exit(e.exit_code)", "response": "shortcut to render data with template and then write to path"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses ascii post source return dict", "response": "def parse(self, source):\n \"\"\"Parse ascii post source, return dict\"\"\"\n\n rt, title, title_pic, markdown = libparser.parse(source)\n\n if rt == -1:\n raise SeparatorNotFound\n elif rt == -2:\n raise PostTitleNotFound\n\n # change to unicode\n title, title_pic, markdown = map(to_unicode, (title, title_pic,\n markdown))\n\n # render to html\n html = self.markdown.render(markdown)\n summary = self.markdown.render(markdown[:200])\n\n return {\n 'title': title,\n 'markdown': markdown,\n 'html': html,\n 'summary': summary,\n 'title_pic': title_pic\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing post source files name to datetime object", "response": "def parse_filename(self, filepath):\n \"\"\"parse post source files name to datetime object\"\"\"\n name = os.path.basename(filepath)[:-src_ext_len]\n try:\n dt = datetime.strptime(name, \"%Y-%m-%d-%H-%M\")\n except ValueError:\n raise PostNameInvalid\n return {'name': name, 'datetime': dt, 'filepath': filepath}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_server(self, port):\n\n try:\n self.server = MultiThreadedHTTPServer(('0.0.0.0', port), Handler)\n except socket.error, e: # failed to bind port\n logger.error(str(e))\n sys.exit(1)\n\n logger.info(\"HTTP serve at http://0.0.0.0:%d (ctrl-c to stop) ...\"\n % port)\n\n try:\n self.server.serve_forever()\n except KeyboardInterrupt:\n logger.info(\"^C received, shutting down server\")\n self.shutdown_server()", "response": "run a server binding to port"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget source files update time", "response": "def get_files_stat(self):\n \"\"\"get source files' update time\"\"\"\n\n if not exists(Post.src_dir):\n logger.error(SourceDirectoryNotFound.__doc__)\n sys.exit(SourceDirectoryNotFound.exit_code)\n\n paths = []\n\n for fn in ls(Post.src_dir):\n if fn.endswith(src_ext):\n paths.append(join(Post.src_dir, fn))\n\n # config.toml\n if exists(config.filepath):\n paths.append(config.filepath)\n\n # files: a dict\n files = dict((p, stat(p).st_mtime) for p in paths)\n return files"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef watch_files(self):\n\n try:\n while 1:\n sleep(1) # check every 1s\n\n try:\n files_stat = self.get_files_stat()\n except SystemExit:\n logger.error(\"Error occurred, server shut down\")\n self.shutdown_server()\n\n if self.files_stat != files_stat:\n logger.info(\"Changes detected, start rebuilding..\")\n\n try:\n generator.re_generate()\n global _root\n _root = generator.root\n except SystemExit: # catch sys.exit, it means fatal error\n logger.error(\"Error occurred, server shut down\")\n self.shutdown_server()\n\n self.files_stat = files_stat # update files' stat\n except KeyboardInterrupt:\n # I dont know why, but this exception won't be catched\n # because absolutly each KeyboardInterrupt is catched by\n # the server thread, which will terminate this thread the same time\n logger.info(\"^C received, shutting down watcher\")\n self.shutdown_watcher()", "response": "watch files for changes rebuild blog. this thread will quit if the main process ends"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing a single post from src.", "response": "def parse(src):\n \"\"\"Note: src should be ascii string\"\"\"\n rt = libparser.parse(byref(post), src)\n return (\n rt,\n string_at(post.title, post.tsz),\n string_at(post.tpic, post.tpsz),\n post.body\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeploys new blog to current directory", "response": "def deploy_blog():\n \"\"\"Deploy new blog to current directory\"\"\"\n logger.info(deploy_blog.__doc__)\n # `rsync -aqu path/to/res/* .`\n call(\n 'rsync -aqu ' + join(dirname(__file__), 'res', '*') + ' .',\n shell=True)\n logger.success('Done')\n logger.info('Please edit config.toml to meet your needs')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntouch a new post in src_dir", "response": "def new_post():\n \"\"\"Touch a new post in src/\"\"\"\n logger.info(new_post.__doc__)\n # make the new post's filename\n now = datetime.datetime.now()\n now_s = now.strftime('%Y-%m-%d-%H-%M')\n filepath = join(Post.src_dir, now_s + src_ext)\n # check if `src/` exists\n if not exists(Post.src_dir):\n logger.error(SourceDirectoryNotFound.__doc__)\n sys.exit(SourceDirectoryNotFound.exit_code)\n # write sample content to new post\n content = (\n 'Title\\n'\n 'Title Picture URL\\n'\n '---\\n'\n 'Markdown content ..'\n )\n f = open(filepath, 'w')\n f.write(content)\n f.close()\n logger.success('New post created: %s' % filepath)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clean():\n logger.info(clean.__doc__)\n paths = ['post', 'page', 'index.html']\n call(['rm', '-rf'] + paths)\n logger.success('Done')", "response": "Clean htmls rux built"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef resolve_blocks(template, context):\n '''\n Return a BlockContext instance of all the {% block %} tags in the template.\n\n If template is a string, it will be resolved through get_template\n '''\n try:\n blocks = context.render_context[BLOCK_CONTEXT_KEY]\n except KeyError:\n blocks = context.render_context[BLOCK_CONTEXT_KEY] = BlockContext()\n\n # If it's just the name, resolve into template\n if isinstance(template, six.string_types):\n template = get_template(template)\n\n # For Django 1.8 compatibility\n template = getattr(template, 'template', template)\n\n # Add this templates blocks as the first\n local_blocks = {\n block.name: block\n for block in template.nodelist.get_nodes_by_type(BlockNode)\n }\n blocks.add_blocks(local_blocks)\n\n # Do we extend a parent template?\n extends = template.nodelist.get_nodes_by_type(ExtendsNode)\n if extends:\n # Can only have one extends in a template\n extends_node = extends[0]\n\n # Get the parent, and recurse\n parent_template = extends_node.get_parent(context)\n resolve_blocks(parent_template, context)\n\n return blocks", "response": "Resolves all the blocks in the template."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_widget_name(widget):\n '''\n Parse a alias:block_name string into separate parts.\n '''\n try:\n alias, block_name = widget.split(':', 1)\n except ValueError:\n raise template.TemplateSyntaxError('widget name must be \"alias:block_name\" - %s' % widget)\n\n return alias, block_name", "response": "Parse a widget name into a alias and block name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind the first matching block in the current block_context", "response": "def find_block(context, *names):\n '''\n Find the first matching block in the current block_context\n '''\n block_set = context.render_context[BLOCK_CONTEXT_KEY]\n for name in names:\n block = block_set.get_block(name)\n if block is not None:\n return block\n\n raise template.TemplateSyntaxError('No widget found for: %r' % (names,))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_widgets(context, **kwargs):\n '''\n Load a series of widget libraries.\n '''\n _soft = kwargs.pop('_soft', False)\n\n try:\n widgets = context.render_context[WIDGET_CONTEXT_KEY]\n except KeyError:\n widgets = context.render_context[WIDGET_CONTEXT_KEY] = {}\n\n for alias, template_name in kwargs.items():\n if _soft and alias in widgets:\n continue\n\n with context.render_context.push({BLOCK_CONTEXT_KEY: BlockContext()}):\n blocks = resolve_blocks(template_name, context)\n widgets[alias] = blocks\n\n return ''", "response": "Load a series of widget libraries."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef auto_widget(field):\n '''Return a list of widget names for the provided field.'''\n # Auto-detect\n info = {\n 'widget': field.field.widget.__class__.__name__,\n 'field': field.field.__class__.__name__,\n 'name': field.name,\n }\n\n return [\n fmt.format(**info)\n for fmt in (\n '{field}_{widget}_{name}',\n '{field}_{name}',\n '{widget}_{name}',\n '{field}_{widget}',\n '{name}',\n '{widget}',\n '{field}',\n )\n ]", "response": "Return a list of widget names for the provided field."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nallowing reuse of a block within a template.", "response": "def reuse(context, block_list, **kwargs):\n '''\n Allow reuse of a block within a template.\n\n {% reuse '_myblock' foo=bar %}\n\n If passed a list of block names, will use the first that matches:\n\n {% reuse list_of_block_names .... %}\n '''\n try:\n block_context = context.render_context[BLOCK_CONTEXT_KEY]\n except KeyError:\n block_context = BlockContext()\n\n if not isinstance(block_list, (list, tuple)):\n block_list = [block_list]\n\n for block in block_list:\n block = block_context.get_block(block)\n if block:\n break\n else:\n return ''\n\n with context.push(kwargs):\n return block.render(context)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef display(self):\n if not self.is_group():\n return self._display\n return ((force_text(k), v) for k, v in self._display)", "response": "Return a list of key - value pairs that can be used to display the current value of the user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlisting the items in the cache with the given key.", "response": "def _list_key(self, key):\n \"\"\"\n boilerplate\n \"\"\"\n ret = []\n for msg_json in self.client.lrange(key, 0, -1):\n ret.append(self._fromJSON(msg_json))\n return ret"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_message(self, level, msg_text, extra_tags='', date=None, url=None):\n if not date:\n now = timezone.now()\n else:\n now = date\n r = now.isoformat()\n if now.microsecond:\n r = r[:23] + r[26:]\n if r.endswith('+00:00'):\n r = r[:-6] + 'Z'\n\n fingerprint = r + msg_text\n\n msg_id = hashlib.sha256(fingerprint.encode('ascii', 'ignore')).hexdigest()\n return Message(id=msg_id, message=msg_text, level=level, tags=extra_tags, date=r, url=url)", "response": "Create a new message object."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsends a message to a list of users without passing through django. contrib. messages", "response": "def add_message_for(users, level, message_text, extra_tags='', date=None, url=None, fail_silently=False):\n \"\"\"\n Send a message to a list of users without passing through `django.contrib.messages`\n\n :param users: an iterable containing the recipients of the messages\n :param level: message level\n :param message_text: the string containing the message\n :param extra_tags: like the Django api, a string containing extra tags for the message\n :param date: a date, different than the default timezone.now\n :param url: an optional url\n :param fail_silently: not used at the moment\n \"\"\"\n BackendClass = stored_messages_settings.STORAGE_BACKEND\n backend = BackendClass()\n m = backend.create_message(level, message_text, extra_tags, date, url)\n backend.archive_store(users, m)\n backend.inbox_store(users, m)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending a message to all users aka broadcast.", "response": "def broadcast_message(level, message_text, extra_tags='', date=None, url=None, fail_silently=False):\n \"\"\"\n Send a message to all users aka broadcast.\n\n :param level: message level\n :param message_text: the string containing the message\n :param extra_tags: like the Django api, a string containing extra tags for the message\n :param date: a date, different than the default timezone.now\n :param url: an optional url\n :param fail_silently: not used at the moment\n \"\"\"\n from django.contrib.auth import get_user_model\n users = get_user_model().objects.all()\n add_message_for(users, level, message_text, extra_tags=extra_tags, date=date, url=url, fail_silently=fail_silently)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mark_read(user, message):\n BackendClass = stored_messages_settings.STORAGE_BACKEND\n backend = BackendClass()\n backend.inbox_delete(user, message)", "response": "Mark a message as read for a user."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmarks all message instances for a user as read.", "response": "def mark_all_read(user):\n \"\"\"\n Mark all message instances for a user as read.\n\n :param user: user instance for the recipient\n \"\"\"\n BackendClass = stored_messages_settings.STORAGE_BACKEND\n backend = BackendClass()\n backend.inbox_purge(user)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmarking all messages as read", "response": "def mark_all_read(request):\n \"\"\"\n Mark all messages as read (i.e. delete from inbox) for current logged in user\n \"\"\"\n from .settings import stored_messages_settings\n backend = stored_messages_settings.STORAGE_BACKEND()\n backend.inbox_purge(request.user)\n return Response({\"message\": \"All messages read\"})"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmark the message as read", "response": "def read(self, request, pk=None):\n \"\"\"\n Mark the message as read (i.e. delete from inbox)\n \"\"\"\n from .settings import stored_messages_settings\n backend = stored_messages_settings.STORAGE_BACKEND()\n\n try:\n backend.inbox_delete(request.user, pk)\n except MessageDoesNotExist as e:\n return Response(e.message, status='404')\n\n return Response({'status': 'message marked as read'})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of unread stored messages for the current user", "response": "def stored_messages_list(context, num_elements=10):\n \"\"\"\n Renders a list of unread stored messages for the current user\n \"\"\"\n if \"user\" in context:\n user = context[\"user\"]\n if user.is_authenticated():\n qs = Inbox.objects.select_related(\"message\").filter(user=user)\n return {\n \"messages\": qs[:num_elements],\n \"count\": qs.count(),\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef stored_messages_count(context):\n if \"user\" in context:\n user = context[\"user\"]\n if user.is_authenticated():\n return Inbox.objects.select_related(\"message\").filter(user=user).count()", "response": "Returns a list of unread stored messages for the current user"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stored_messages_archive(context, num_elements=10):\n if \"user\" in context:\n user = context[\"user\"]\n if user.is_authenticated():\n qs = MessageArchive.objects.select_related(\"message\").filter(user=user)\n return {\n \"messages\": qs[:num_elements],\n \"count\": qs.count(),\n }", "response": "Returns a list of archived messages for the current user"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get(self, *args, **kwargs):\n messages, all_retrieved = super(StorageMixin, self)._get(*args, **kwargs)\n if self.user.is_authenticated():\n inbox_messages = self.backend.inbox_list(self.user)\n else:\n inbox_messages = []\n\n return messages + inbox_messages, all_retrieved", "response": "Retrieve unread messages for current user both from the inbox and\n from other storages"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add(self, level, message, extra_tags=''):\n if not message:\n return\n # Check that the message level is not less than the recording level.\n level = int(level)\n if level < self.level:\n return\n # Check if the message doesn't have a level that needs to be persisted\n if level not in stored_messages_settings.STORE_LEVELS or self.user.is_anonymous():\n return super(StorageMixin, self).add(level, message, extra_tags)\n\n self.added_new = True\n m = self.backend.create_message(level, message, extra_tags)\n self.backend.archive_store([self.user], m)\n self._queued_messages.append(m)", "response": "Add a new message to the database."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstoring the given messages in the database.", "response": "def _store(self, messages, response, *args, **kwargs):\n \"\"\"\n persistent messages are already in the database inside the 'archive',\n so we can say they're already \"stored\".\n Here we put them in the inbox, or remove from the inbox in case the\n messages were iterated.\n\n messages contains only new msgs if self.used==True\n else contains both new and unread messages\n \"\"\"\n contrib_messages = []\n if self.user.is_authenticated():\n if not messages:\n # erase inbox\n self.backend.inbox_purge(self.user)\n else:\n for m in messages:\n try:\n self.backend.inbox_store([self.user], m)\n except MessageTypeNotSupported:\n contrib_messages.append(m)\n\n super(StorageMixin, self)._store(contrib_messages, response, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _prepare_messages(self, messages):\n for message in messages:\n if not self.backend.can_handle(message):\n message._prepare()", "response": "Prepare a list of messages for storage\n ."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize a base logger", "response": "def init(base_level=DEFAULT_BASE_LOGGING_LEVEL,\n verbose_level=DEFAULT_VERBOSE_LOGGING_LEVEL,\n logging_config=None):\n \"\"\"initializes a base logger\n\n you can use this to init a logger in any of your files.\n this will use config.py's LOGGER param and logging.dictConfig to configure\n the logger for you.\n\n :param int|logging.LEVEL base_level: desired base logging level\n :param int|logging.LEVEL verbose_level: desired verbose logging level\n :param dict logging_dict: dictConfig based configuration.\n used to override the default configuration from config.py\n :rtype: `python logger`\n \"\"\"\n if logging_config is None:\n logging_config = {}\n logging_config = logging_config or LOGGER\n # TODO: (IMPRV) only perform file related actions if file handler is\n # TODO: (IMPRV) defined.\n\n log_file = LOGGER['handlers']['file']['filename']\n log_dir = os.path.dirname(os.path.expanduser(log_file))\n if os.path.isfile(log_dir):\n sys.exit('file {0} exists - log directory cannot be created '\n 'there. please remove the file and try again.'\n .format(log_dir))\n try:\n if not os.path.exists(log_dir) and not len(log_dir) == 0:\n os.makedirs(log_dir)\n dictconfig.dictConfig(logging_config)\n lgr = logging.getLogger('user')\n lgr.setLevel(base_level)\n return lgr\n except ValueError as e:\n sys.exit('could not initialize logger.'\n ' verify your logger config'\n ' and permissions to write to {0} ({1})'\n .format(log_file, e))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndefaulting converter for the cfg:// protocol.", "response": "def cfg_convert(self, value):\n \"\"\"Default converter for the cfg:// protocol.\"\"\"\n rest = value\n m = self.WORD_PATTERN.match(rest)\n if m is None:\n raise ValueError(\"Unable to convert %r\" % value)\n else:\n rest = rest[m.end():]\n d = self.config[m.groups()[0]]\n # print d, rest\n while rest:\n m = self.DOT_PATTERN.match(rest)\n if m:\n d = d[m.groups()[0]]\n else:\n m = self.INDEX_PATTERN.match(rest)\n if m:\n idx = m.groups()[0]\n if not self.DIGIT_PATTERN.match(idx):\n d = d[idx]\n else:\n try:\n n = int(idx)\n d = d[n]\n except TypeError:\n d = d[idx]\n if m:\n rest = rest[m.end():]\n else:\n raise ValueError('Unable to convert '\n '%r at %r' % (value, rest))\n # rest should be empty\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef configure_custom(self, config):\n c = config.pop('()')\n if not hasattr(c, '__call__') and \\\n hasattr(types, 'ClassType') and isinstance(c, types.ClassType):\n c = self.resolve(c)\n props = config.pop('.', None)\n # Check for valid identifiers\n kwargs = dict((k, config[k]) for k in config if valid_ident(k))\n result = c(**kwargs)\n if props:\n for name, value in props.items():\n setattr(result, name, value)\n return result", "response": "Configure an object with a user - supplied factory."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the global verbosity level for console and the jocker_lgr logger.", "response": "def _set_global_verbosity_level(is_verbose_output=False):\n \"\"\"sets the global verbosity level for console and the jocker_lgr logger.\n\n :param bool is_verbose_output: should be output be verbose\n \"\"\"\n global verbose_output\n # TODO: (IMPRV) only raise exceptions in verbose mode\n verbose_output = is_verbose_output\n if verbose_output:\n jocker_lgr.setLevel(logging.DEBUG)\n else:\n jocker_lgr.setLevel(logging.INFO)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _import_config(config_file):\n # get config file path\n jocker_lgr.debug('config file is: {0}'.format(config_file))\n # append to path for importing\n try:\n jocker_lgr.debug('importing config...')\n with open(config_file, 'r') as c:\n return yaml.safe_load(c.read())\n except IOError as ex:\n jocker_lgr.error(str(ex))\n raise RuntimeError('cannot access config file')\n except yaml.parser.ParserError as ex:\n jocker_lgr.error('invalid yaml file: {0}'.format(ex))\n raise RuntimeError('invalid yaml file')", "response": "returns a configuration object from a config file"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a Dockerfile and builds an image and pushes it to DockerHub.", "response": "def execute(varsfile, templatefile, outputfile=None, configfile=None,\n dryrun=False, build=False, push=False, verbose=False):\n \"\"\"generates a Dockerfile, builds an image and pushes it to DockerHub\n\n A `Dockerfile` will be generated by Jinja2 according to the `varsfile`\n imported. If build is true, an image will be generated from the\n `outputfile` which is the generated Dockerfile and committed to the\n image:tag string supplied to `build`.\n If push is true, a build will be triggered and the produced image\n will be pushed to DockerHub upon completion.\n\n :param string varsfile: path to file with variables.\n :param string templatefile: path to template file to use.\n :param string outputfile: path to output Dockerfile.\n :param string configfile: path to yaml file with docker-py config.\n :param bool dryrun: mock run.\n :param build: False or the image:tag to build to.\n :param push: False or the image:tag to build to. (triggers build)\n :param bool verbose: verbose output.\n \"\"\"\n if dryrun and (build or push):\n jocker_lgr.error('dryrun requested, cannot build.')\n sys.exit(100)\n\n _set_global_verbosity_level(verbose)\n j = Jocker(varsfile, templatefile, outputfile, configfile, dryrun,\n build, push)\n formatted_text = j.generate()\n if dryrun:\n g = j.dryrun(formatted_text)\n if build or push:\n j.build_image()\n if push:\n j.push_image()\n if dryrun:\n return g"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing the output of a dumb push process", "response": "def _parse_dumb_push_output(self, string):\n \"\"\"since the push process outputs a single unicode string consisting of\n multiple JSON formatted \"status\" lines, we need to parse it so that it\n can be read as multiple strings.\n\n This will receive the string as an input, count curly braces and ignore\n any newlines. When the curly braces stack is 0, it will append the\n entire string it has read up until then to a list and so forth.\n\n :param string: the string to parse\n :rtype: list of JSON's\n \"\"\"\n stack = 0\n json_list = []\n tmp_json = ''\n for char in string:\n if not char == '\\r' and not char == '\\n':\n tmp_json += char\n if char == '{':\n stack += 1\n elif char == '}':\n stack -= 1\n if stack == 0:\n if not len(tmp_json) == 0:\n json_list.append(tmp_json)\n tmp_json = ''\n return json_list"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nuploading an image file to Imgur", "response": "def upload_gif(gif):\n \"\"\"Uploads an image file to Imgur\"\"\"\n\n client_id = os.environ.get('IMGUR_API_ID')\n client_secret = os.environ.get('IMGUR_API_SECRET')\n\n if client_id is None or client_secret is None:\n click.echo('Cannot upload - could not find IMGUR_API_ID or IMGUR_API_SECRET environment variables')\n return\n\n client = ImgurClient(client_id, client_secret)\n\n click.echo('Uploading file {}'.format(click.format_filename(gif)))\n\n response = client.upload_from_path(gif)\n\n click.echo('File uploaded - see your gif at {}'.format(response['link']))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the IP address is in dotted decimal notation.", "response": "def is_dot(ip):\n \"\"\"Return true if the IP address is in dotted decimal notation.\"\"\"\n octets = str(ip).split('.')\n if len(octets) != 4:\n return False\n for i in octets:\n try:\n val = int(i)\n except ValueError:\n return False\n if val > 255 or val < 0:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_bin(ip):\n try:\n ip = str(ip)\n if len(ip) != 32:\n return False\n dec = int(ip, 2)\n except (TypeError, ValueError):\n return False\n if dec > 4294967295 or dec < 0:\n return False\n return True", "response": "Return True if the IP address is in binary notation."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the IP address is in octal notation.", "response": "def is_oct(ip):\n \"\"\"Return true if the IP address is in octal notation.\"\"\"\n try:\n dec = int(str(ip), 8)\n except (TypeError, ValueError):\n return False\n if dec > 0o37777777777 or dec < 0:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if the IP address is in decimal notation.", "response": "def is_dec(ip):\n \"\"\"Return true if the IP address is in decimal notation.\"\"\"\n try:\n dec = int(str(ip))\n except ValueError:\n return False\n if dec > 4294967295 or dec < 0:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _check_nm(nm, notation):\n # Convert to decimal, and check if it's in the list of valid netmasks.\n _NM_CHECK_FUNCT = {\n NM_DOT: _dot_to_dec,\n NM_HEX: _hex_to_dec,\n NM_BIN: _bin_to_dec,\n NM_OCT: _oct_to_dec,\n NM_DEC: _dec_to_dec_long}\n try:\n dec = _NM_CHECK_FUNCT[notation](nm, check=True)\n except ValueError:\n return False\n if dec in _NETMASKS_VALUES:\n return True\n return False", "response": "Function internally used to check if the given netmask\n is of the specified notation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_bits_nm(nm):\n try:\n bits = int(str(nm))\n except ValueError:\n return False\n if bits > 32 or bits < 0:\n return False\n return True", "response": "Return true if the netmask is in bits notatation."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if the netmask is in wildcard bits notatation.", "response": "def is_wildcard_nm(nm):\n \"\"\"Return true if the netmask is in wildcard bits notatation.\"\"\"\n try:\n dec = 0xFFFFFFFF - _dot_to_dec(nm, check=True)\n except ValueError:\n return False\n if dec in _NETMASKS_VALUES:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _dot_to_dec(ip, check=True):\n if check and not is_dot(ip):\n raise ValueError('_dot_to_dec: invalid IP: \"%s\"' % ip)\n octets = str(ip).split('.')\n dec = 0\n dec |= int(octets[0]) << 24\n dec |= int(octets[1]) << 16\n dec |= int(octets[2]) << 8\n dec |= int(octets[3])\n return dec", "response": "Dotted decimal notation to decimal conversion."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _dec_to_dot(ip):\n first = int((ip >> 24) & 255)\n second = int((ip >> 16) & 255)\n third = int((ip >> 8) & 255)\n fourth = int(ip & 255)\n return '%d.%d.%d.%d' % (first, second, third, fourth)", "response": "Decimal to dotted decimal notation conversion."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _BYTES_TO_BITS():\n the_table = 256*[None]\n bits_per_byte = list(range(7, -1, -1))\n for n in range(256):\n l = n\n bits = 8*[None]\n for i in bits_per_byte:\n bits[i] = '01'[n & 1]\n n >>= 1\n the_table[l] = ''.join(bits)\n return the_table", "response": "Generate a table to convert a whole byte to binary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _dec_to_bin(ip):\n bits = []\n while ip:\n bits.append(_BYTES_TO_BITS[ip & 255])\n ip >>= 8\n bits.reverse()\n return ''.join(bits) or 32*'0'", "response": "Decimal to binary conversion."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _wildcard_to_dec(nm, check=False):\n if check and not is_wildcard_nm(nm):\n raise ValueError('_wildcard_to_dec: invalid netmask: \"%s\"' % nm)\n return 0xFFFFFFFF - _dot_to_dec(nm, check=False)", "response": "Wildcard bits to decimal conversion."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _is_notation(ip, notation, _isnm):\n notation_orig = notation\n notation = _get_notation(notation)\n if notation not in _CHECK_FUNCT_KEYS:\n raise ValueError('_is_notation: unkown notation: \"%s\"' % notation_orig)\n return _CHECK_FUNCT[notation][_isnm](ip)", "response": "Internally used to check if an IP is in the given notation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _detect(ip, _isnm):\n ip = str(ip)\n if len(ip) > 1:\n if ip[0:2] == '0x':\n if _CHECK_FUNCT[IP_HEX][_isnm](ip):\n return IP_HEX\n elif ip[0] == '0':\n if _CHECK_FUNCT[IP_OCT][_isnm](ip):\n return IP_OCT\n if _CHECK_FUNCT[IP_DOT][_isnm](ip):\n return IP_DOT\n elif _isnm and _CHECK_FUNCT[NM_BITS][_isnm](ip):\n return NM_BITS\n elif _CHECK_FUNCT[IP_DEC][_isnm](ip):\n return IP_DEC\n elif _isnm and _CHECK_FUNCT[NM_WILDCARD][_isnm](ip):\n return NM_WILDCARD\n elif _CHECK_FUNCT[IP_BIN][_isnm](ip):\n return IP_BIN\n return IP_UNKNOWN", "response": "Function internally used to detect the notation of the\n given IP or netmask."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef convert(ip, notation=IP_DOT, inotation=IP_UNKNOWN, check=True):\n return _convert(ip, notation, inotation, _check=check, _isnm=False)", "response": "Convert a given IP address to another notation."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a netmask to another notation.", "response": "def convert_nm(nm, notation=IP_DOT, inotation=IP_UNKNOWN, check=True):\n \"\"\"Convert a netmask to another notation.\"\"\"\n return _convert(nm, notation, inotation, _check=check, _isnm=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the IP address and netmask.", "response": "def set(self, ip, notation=IP_UNKNOWN):\n \"\"\"Set the IP address/netmask.\"\"\"\n self._ip_dec = int(_convert(ip, notation=IP_DEC, inotation=notation,\n _check=True, _isnm=self._isnm))\n self._ip = _convert(self._ip_dec, notation=IP_DOT, inotation=IP_DEC,\n _check=False, _isnm=self._isnm)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the hexadecimal notation of the address or netmask.", "response": "def get_hex(self):\n \"\"\"Return the hexadecimal notation of the address/netmask.\"\"\"\n return _convert(self._ip_dec, notation=IP_HEX,\n inotation=IP_DEC, _check=False, _isnm=self._isnm)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the binary notation of the address and netmask.", "response": "def get_bin(self):\n \"\"\"Return the binary notation of the address/netmask.\"\"\"\n return _convert(self._ip_dec, notation=IP_BIN,\n inotation=IP_DEC, _check=False, _isnm=self._isnm)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the octal notation of the address or netmask.", "response": "def get_oct(self):\n \"\"\"Return the octal notation of the address/netmask.\"\"\"\n return _convert(self._ip_dec, notation=IP_OCT,\n inotation=IP_DEC, _check=False, _isnm=self._isnm)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprepares the item to be compared with this address / netmask.", "response": "def _cmp_prepare(self, other):\n \"\"\"Prepare the item to be compared with this address/netmask.\"\"\"\n if isinstance(other, self.__class__):\n return other._ip_dec\n elif isinstance(other, int):\n # NOTE: this hides the fact that \"other\" can be a non valid IP/nm.\n return other\n return self.__class__(other)._ip_dec"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsumming two IP addresses.", "response": "def _add(self, other):\n \"\"\"Sum two IP addresses.\"\"\"\n if isinstance(other, self.__class__):\n sum_ = self._ip_dec + other._ip_dec\n elif isinstance(other, int):\n sum_ = self._ip_dec + other\n else:\n other = self.__class__(other)\n sum_ = self._ip_dec + other._ip_dec\n return sum_"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _sub(self, other):\n if isinstance(other, self.__class__):\n sub = self._ip_dec - other._ip_dec\n if isinstance(other, int):\n sub = self._ip_dec - other\n else:\n other = self.__class__(other)\n sub = self._ip_dec - other._ip_dec\n return sub", "response": "Subtract two IP addresses."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the bits notation of the netmask.", "response": "def get_bits(self):\n \"\"\"Return the bits notation of the netmask.\"\"\"\n return _convert(self._ip, notation=NM_BITS,\n inotation=IP_DOT, _check=False, _isnm=self._isnm)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_wildcard(self):\n return _convert(self._ip, notation=NM_WILDCARD,\n inotation=IP_DOT, _check=False, _isnm=self._isnm)", "response": "Return the wildcard bits notation of the netmask."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set(self, ip, netmask=None):\n if isinstance(ip, str) and netmask is None:\n ipnm = ip.split('/')\n if len(ipnm) != 2:\n raise ValueError('set: invalid CIDR: \"%s\"' % ip)\n ip = ipnm[0]\n netmask = ipnm[1]\n if isinstance(ip, IPv4Address):\n self._ip = ip\n else:\n self._ip = IPv4Address(ip)\n if isinstance(netmask, IPv4NetMask):\n self._nm = netmask\n else:\n self._nm = IPv4NetMask(netmask)\n ipl = int(self._ip)\n nml = int(self._nm)\n base_add = ipl & nml\n self._ip_num = 0xFFFFFFFF - 1 - nml\n # NOTE: quite a mess.\n # This's here to handle /32 (-1) and /31 (0) netmasks.\n if self._ip_num in (-1, 0):\n if self._ip_num == -1:\n self._ip_num = 1\n else:\n self._ip_num = 2\n self._net_ip = None\n self._bc_ip = None\n self._first_ip_dec = base_add\n self._first_ip = IPv4Address(self._first_ip_dec, notation=IP_DEC)\n if self._ip_num == 1:\n last_ip_dec = self._first_ip_dec\n else:\n last_ip_dec = self._first_ip_dec + 1\n self._last_ip = IPv4Address(last_ip_dec, notation=IP_DEC)\n return\n self._net_ip = IPv4Address(base_add, notation=IP_DEC)\n self._bc_ip = IPv4Address(base_add + self._ip_num + 1, notation=IP_DEC)\n self._first_ip_dec = base_add + 1\n self._first_ip = IPv4Address(self._first_ip_dec, notation=IP_DEC)\n self._last_ip = IPv4Address(base_add + self._ip_num, notation=IP_DEC)", "response": "Set the IP address and netmask."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nchange the current IP.", "response": "def set_ip(self, ip):\n \"\"\"Change the current IP.\"\"\"\n self.set(ip=ip, netmask=self._nm)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_netmask(self, netmask):\n self.set(ip=self._ip, netmask=netmask)", "response": "Change the netmask of the current log entry."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning true if the given address in amongst the usable addresses and is contained in this one.", "response": "def is_valid_ip(self, ip):\n \"\"\"Return true if the given address in amongst the usable addresses,\n or if the given CIDR is contained in this one.\"\"\"\n if not isinstance(ip, (IPv4Address, CIDR)):\n if str(ip).find('/') == -1:\n ip = IPv4Address(ip)\n else:\n # Support for CIDR strings/objects, an idea of Nicola Novello.\n ip = CIDR(ip)\n if isinstance(ip, IPv4Address):\n if ip < self._first_ip or ip > self._last_ip:\n return False\n elif isinstance(ip, CIDR):\n # NOTE: manage /31 networks; 127.0.0.1/31 is considered to\n # be included in 127.0.0.1/8.\n if ip._nm._ip_dec == 0xFFFFFFFE \\\n and self._nm._ip_dec != 0xFFFFFFFE:\n compare_to_first = self._net_ip._ip_dec\n compare_to_last = self._bc_ip._ip_dec\n else:\n compare_to_first = self._first_ip._ip_dec\n compare_to_last = self._last_ip._ip_dec\n if ip._first_ip._ip_dec < compare_to_first or \\\n ip._last_ip._ip_dec > compare_to_last:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nuploading a file to S3 possibly using the multi - part uploader", "response": "async def upload_file(self, bucket, file, uploadpath=None, key=None,\n ContentType=None, **kw):\n \"\"\"Upload a file to S3 possibly using the multi-part uploader\n Return the key uploaded\n \"\"\"\n is_filename = False\n\n if hasattr(file, 'read'):\n if hasattr(file, 'seek'):\n file.seek(0)\n file = file.read()\n size = len(file)\n elif key:\n size = len(file)\n else:\n is_filename = True\n size = os.stat(file).st_size\n key = os.path.basename(file)\n\n assert key, 'key not available'\n\n if not ContentType:\n ContentType, _ = mimetypes.guess_type(key)\n\n if uploadpath:\n if not uploadpath.endswith('/'):\n uploadpath = '%s/' % uploadpath\n key = '%s%s' % (uploadpath, key)\n\n params = dict(Bucket=bucket, Key=key)\n\n if not ContentType:\n ContentType = 'application/octet-stream'\n\n params['ContentType'] = ContentType\n\n if size > MULTI_PART_SIZE and is_filename:\n resp = await _multipart(self, file, params)\n elif is_filename:\n with open(file, 'rb') as fp:\n params['Body'] = fp.read()\n resp = await self.put_object(**params)\n else:\n params['Body'] = file\n resp = await self.put_object(**params)\n if 'Key' not in resp:\n resp['Key'] = key\n if 'Bucket' not in resp:\n resp['Bucket'] = bucket\n return resp"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\nasync def copy_storage_object(self, source_bucket, source_key,\n bucket, key):\n \"\"\"Copy a file from one bucket into another\n \"\"\"\n info = await self.head_object(Bucket=source_bucket, Key=source_key)\n size = info['ContentLength']\n\n if size > MULTI_PART_SIZE:\n result = await _multipart_copy(self, source_bucket, source_key,\n bucket, key, size)\n else:\n result = await self.copy_object(\n Bucket=bucket, Key=key,\n CopySource=_source_string(source_bucket, source_key)\n )\n return result", "response": "Copy a file from one bucket into another."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def _upload_file(self, full_path):\n rel_path = os.path.relpath(full_path, self.folder)\n key = s3_key(os.path.join(self.key, rel_path))\n ct = self.content_types.get(key.split('.')[-1])\n with open(full_path, 'rb') as fp:\n file = fp.read()\n try:\n await self.botocore.upload_file(self.bucket, file, key=key,\n ContentType=ct)\n except Exception as exc:\n LOGGER.error('Could not upload \"%s\": %s', key, exc)\n self.failures[key] = self.all.pop(full_path)\n return\n size = self.all.pop(full_path)\n self.success[key] = size\n self.total_size += size\n percentage = 100*(1 - len(self.all)/self.total_files)\n message = '{0:.0f}% completed - uploaded \"{1}\" - {2}'.format(\n percentage, key, convert_bytes(size))\n LOGGER.info(message)", "response": "Coroutine for uploading a single file to S3."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a paginator for an operation.", "response": "def get_paginator(self, operation_name):\n \"\"\"Create a paginator for an operation.\n :type operation_name: string\n :param operation_name: The operation name. This is the same name\n as the method name on the client. For example, if the\n method name is ``create_foo``, and you'd normally invoke the\n operation as ``client.create_foo(**kwargs)``, if the\n ``create_foo`` operation can be paginated, you can use the\n call ``client.get_paginator(\"create_foo\")``.\n :raise OperationNotPageableError: Raised if the operation is not\n pageable. You can use the ``client.can_paginate`` method to\n check if an operation is pageable.\n :rtype: L{botocore.paginate.Paginator}\n :return: A paginator object.\n \"\"\"\n if not self.can_paginate(operation_name):\n raise OperationNotPageableError(operation_name=operation_name)\n else:\n actual_operation_name = self._PY_TO_OP_NAME[operation_name]\n # substitute iterator with async one\n Paginator.PAGE_ITERATOR_CLS = AsyncPageIterator\n paginator = Paginator(\n getattr(self, operation_name),\n self._cache['page_config'][actual_operation_name])\n return paginator"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntrigger an event on this channel", "response": "async def trigger(self, event, data=None, socket_id=None):\n '''Trigger an ``event`` on this channel\n '''\n json_data = json.dumps(data, cls=self.pusher.encoder)\n query_string = self.signed_query(event, json_data, socket_id)\n signed_path = \"%s?%s\" % (self.path, query_string)\n pusher = self.pusher\n absolute_url = pusher.get_absolute_path(signed_path)\n response = await pusher.http.post(\n absolute_url, data=json_data,\n headers=[('Content-Type', 'application/json')])\n response.raise_for_status()\n return response.status_code == 202"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def connect(self):\n '''Connect to a Pusher websocket\n '''\n if not self._consumer:\n waiter = self._waiter = asyncio.Future()\n try:\n address = self._websocket_host()\n self.logger.info('Connect to %s', address)\n self._consumer = await self.http.get(address)\n if self._consumer.status_code != 101:\n raise PusherError(\"Could not connect to websocket\")\n except Exception as exc:\n waiter.set_exception(exc)\n raise\n else:\n await waiter\n return self._consumer", "response": "Connect to a Pusher websocket"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nhandle incoming messages from the pusher.", "response": "def on_message(self, websocket, message):\n '''Handle websocket incoming messages\n '''\n waiter = self._waiter\n self._waiter = None\n encoded = json.loads(message)\n event = encoded.get('event')\n channel = encoded.get('channel')\n data = json.loads(encoded.get('data'))\n try:\n if event == PUSHER_ERROR:\n raise PusherError(data['message'], data['code'])\n elif event == PUSHER_CONNECTION:\n self.socket_id = data.get('socket_id')\n self.logger.info('Succesfully connected on socket %s',\n self.socket_id)\n waiter.set_result(self.socket_id)\n elif event == PUSHER_SUBSCRIBED:\n self.logger.info('Succesfully subscribed to %s',\n encoded.get('channel'))\n elif channel:\n self[channel]._event(event, data)\n except Exception as exc:\n if waiter:\n waiter.set_exception(exc)\n else:\n self.logger.exception('pusher error')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef urlsafe_nopadding_b64decode(data):\n '''URL safe Base64 decode without padding (=)'''\n\n padding = len(data) % 4\n if padding != 0:\n padding = 4 - padding\n padding = '=' * padding\n data = data + padding\n return urlsafe_b64decode(data)", "response": "URL safe Base64 decode without padding"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndecoding a limited set of HTML entities.", "response": "def decode_html_entities(html):\n \"\"\"\n Decodes a limited set of HTML entities.\n \"\"\"\n if not html:\n return html\n\n for entity, char in six.iteritems(html_entity_map):\n html = html.replace(entity, char)\n\n return html"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_signature_passphrases(self, signature_passphrases):\n '''Set signature passphrases'''\n self.signature_passphrases = self._update_dict(signature_passphrases,\n {}, replace_data=True)", "response": "Set the signature passphrases for this resource."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the set of encryption passphrases for this resource.", "response": "def set_encryption_passphrases(self, encryption_passphrases):\n '''Set encryption passphrases'''\n self.encryption_passphrases = self._update_dict(encryption_passphrases,\n {}, replace_data=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_algorithms(self, signature=None, encryption=None,\n serialization=None, compression=None):\n '''Set algorithms used for sealing. Defaults can not be overridden.'''\n\n self.signature_algorithms = \\\n self._update_dict(signature, self.DEFAULT_SIGNATURE)\n\n self.encryption_algorithms = \\\n self._update_dict(encryption, self.DEFAULT_ENCRYPTION)\n\n self.serialization_algorithms = \\\n self._update_dict(serialization, self.DEFAULT_SERIALIZATION)\n\n self.compression_algorithms = \\\n self._update_dict(compression, self.DEFAULT_COMPRESSION)", "response": "Sets the algorithms used for sealing. Defaults can not be overridden."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets algorithms used for sealing", "response": "def get_algorithms(self):\n '''Get algorithms used for sealing'''\n\n return {\n 'signature': self.signature_algorithms,\n 'encryption': self.encryption_algorithms,\n 'serialization': self.serialization_algorithms,\n 'compression': self.compression_algorithms,\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef verify_signature(self, data):\n '''Verify sealed data signature'''\n\n data = self._remove_magic(data)\n data = urlsafe_nopadding_b64decode(data)\n options = self._read_header(data)\n data = self._add_magic(data)\n self._unsign_data(data, options)", "response": "Verify sealed data signature"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nencode data with specific algorithm", "response": "def _encode(self, data, algorithm, key=None):\n '''Encode data with specific algorithm'''\n\n if algorithm['type'] == 'hmac':\n return data + self._hmac_generate(data, algorithm, key)\n elif algorithm['type'] == 'aes':\n return self._aes_encrypt(data, algorithm, key)\n elif algorithm['type'] == 'no-serialization':\n return data\n elif algorithm['type'] == 'json':\n return json.dumps(data)\n elif algorithm['type'] == 'no-compression':\n return data\n elif algorithm['type'] == 'gzip':\n return self._zlib_compress(data, algorithm)\n else:\n raise Exception('Algorithm not supported: %s' % algorithm['type'])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _decode(self, data, algorithm, key=None):\n '''Decode data with specific algorithm'''\n\n if algorithm['type'] == 'hmac':\n verify_signature = data[-algorithm['hash_size']:]\n data = data[:-algorithm['hash_size']]\n signature = self._hmac_generate(data, algorithm, key)\n if not const_equal(verify_signature, signature):\n raise Exception('Invalid signature')\n return data\n elif algorithm['type'] == 'aes':\n return self._aes_decrypt(data, algorithm, key)\n elif algorithm['type'] == 'no-serialization':\n return data\n elif algorithm['type'] == 'json':\n return json.loads(data)\n elif algorithm['type'] == 'no-compression':\n return data\n elif algorithm['type'] == 'gzip':\n return self._zlib_decompress(data, algorithm)\n else:\n raise Exception('Algorithm not supported: %s' % algorithm['type'])", "response": "Decode data with specific algorithm"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _sign_data(self, data, options):\n '''Add signature to data'''\n\n if options['signature_algorithm_id'] not in self.signature_algorithms:\n raise Exception('Unknown signature algorithm id: %d'\n % options['signature_algorithm_id'])\n\n signature_algorithm = \\\n self.signature_algorithms[options['signature_algorithm_id']]\n\n algorithm = self._get_algorithm_info(signature_algorithm)\n\n key_salt = get_random_bytes(algorithm['salt_size'])\n key = self._generate_key(options['signature_passphrase_id'],\n self.signature_passphrases, key_salt, algorithm)\n\n data = self._encode(data, algorithm, key)\n\n return data + key_salt", "response": "Add signature to data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _unsign_data(self, data, options):\n '''Verify and remove signature'''\n\n if options['signature_algorithm_id'] not in self.signature_algorithms:\n raise Exception('Unknown signature algorithm id: %d'\n % options['signature_algorithm_id'])\n\n signature_algorithm = \\\n self.signature_algorithms[options['signature_algorithm_id']]\n\n algorithm = self._get_algorithm_info(signature_algorithm)\n\n key_salt = ''\n if algorithm['salt_size']:\n key_salt = data[-algorithm['salt_size']:]\n data = data[:-algorithm['salt_size']]\n\n key = self._generate_key(options['signature_passphrase_id'],\n self.signature_passphrases, key_salt, algorithm)\n\n data = self._decode(data, algorithm, key)\n\n return data", "response": "Verify and remove signature"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _encrypt_data(self, data, options):\n '''Encrypt data'''\n\n if options['encryption_algorithm_id'] not in self.encryption_algorithms:\n raise Exception('Unknown encryption algorithm id: %d'\n % options['encryption_algorithm_id'])\n\n encryption_algorithm = \\\n self.encryption_algorithms[options['encryption_algorithm_id']]\n\n algorithm = self._get_algorithm_info(encryption_algorithm)\n\n key_salt = get_random_bytes(algorithm['salt_size'])\n key = self._generate_key(options['encryption_passphrase_id'],\n self.encryption_passphrases, key_salt, algorithm)\n\n data = self._encode(data, algorithm, key)\n\n return data + key_salt", "response": "Encrypt data with the key and passphrase"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _remove_magic(self, data):\n '''Verify and remove magic'''\n\n if not self.magic:\n return data\n\n magic_size = len(self.magic)\n magic = data[:magic_size]\n if magic != self.magic:\n raise Exception('Invalid magic')\n data = data[magic_size:]\n\n return data", "response": "Verify and remove magic from data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _add_header(self, data, options):\n '''Add header to data'''\n\n # pylint: disable=W0142\n\n version_info = self._get_version_info(options['version'])\n\n flags = options['flags']\n\n header_flags = dict(\n (i, str(int(j))) for i, j in options['flags'].iteritems())\n header_flags = ''.join(version_info['flags'](**header_flags))\n header_flags = int(header_flags, 2)\n options['flags'] = header_flags\n\n header = version_info['header']\n header = header(**options)\n header = pack(version_info['header_format'], *header)\n\n if 'timestamp' in flags and flags['timestamp']:\n timestamp = long(time())\n timestamp = pack(version_info['timestamp_format'], timestamp)\n header = header + timestamp\n\n return header + data", "response": "Add header to data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _read_header(self, data):\n '''Read header from data'''\n\n # pylint: disable=W0212\n\n version = self._read_version(data)\n version_info = self._get_version_info(version)\n header_data = data[:version_info['header_size']]\n header = version_info['header']\n header = header._make(\n unpack(version_info['header_format'], header_data))\n header = dict(header._asdict())\n\n flags = list(\"{0:0>8b}\".format(header['flags']))\n flags = dict(version_info['flags']._make(flags)._asdict())\n flags = dict((i, bool(int(j))) for i, j in flags.iteritems())\n header['flags'] = flags\n\n timestamp = None\n if flags['timestamp']:\n ts_start = version_info['header_size']\n ts_end = ts_start + version_info['timestamp_size']\n timestamp_data = data[ts_start:ts_end]\n timestamp = unpack(\n version_info['timestamp_format'], timestamp_data)[0]\n header['info'] = {'timestamp': timestamp}\n\n return header", "response": "Read the header from data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves the header from the data", "response": "def _remove_header(self, data, options):\n '''Remove header from data'''\n\n version_info = self._get_version_info(options['version'])\n header_size = version_info['header_size']\n\n if options['flags']['timestamp']:\n header_size += version_info['timestamp_size']\n\n data = data[header_size:]\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_version(self, data):\n '''Read header version from data'''\n\n version = ord(data[0])\n if version not in self.VERSIONS:\n raise Exception('Version not defined: %d' % version)\n return version", "response": "Read the version from data"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _generate_key(pass_id, passphrases, salt, algorithm):\n '''Generate and return PBKDF2 key'''\n\n if pass_id not in passphrases:\n raise Exception('Passphrase not defined for id: %d' % pass_id)\n\n passphrase = passphrases[pass_id]\n\n if len(passphrase) < 32:\n raise Exception('Passphrase less than 32 characters long')\n\n digestmod = EncryptedPickle._get_hashlib(algorithm['pbkdf2_algorithm'])\n\n encoder = PBKDF2(passphrase, salt,\n iterations=algorithm['pbkdf2_iterations'],\n digestmodule=digestmod)\n\n return encoder.read(algorithm['key_size'])", "response": "Generate and return PBKDF2 key"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _update_dict(data, default_data, replace_data=False):\n '''Update algorithm definition type dictionaries'''\n\n if not data:\n data = default_data.copy()\n return data\n\n if not isinstance(data, dict):\n raise TypeError('Value not dict type')\n if len(data) > 255:\n raise ValueError('More than 255 values defined')\n for i in data.keys():\n if not isinstance(i, int):\n raise TypeError('Index not int type')\n if i < 0 or i > 255:\n raise ValueError('Index value out of range')\n\n if not replace_data:\n data.update(default_data)\n\n return data", "response": "Update algorithm definition type dictionaries"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_hashlib(digestmode):\n '''Generate HMAC hash'''\n if digestmode == 'sha1':\n return SHA\n if digestmode == 'sha256':\n return SHA256\n elif digestmode == 'sha384':\n return SHA384\n elif digestmode == 'sha512':\n return SHA512\n else:\n raise Exception('digestmode not supported: %s'\n % digestmode)", "response": "Generate HMAC hash from digestmode"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getTableOfContents(self):\n\n self.directory_size = self.getDirectorySize()\n if self.directory_size > 65536:\n self.directory_size += 2\n self.requestContentDirectory()\n\n\n # and find the offset from start of file where it can be found\n directory_start = unpack(\"i\", self.raw_bytes[self.directory_end + 16: self.directory_end + 20])[0]\n\n # find the data in the raw_bytes\n self.raw_bytes = self.raw_bytes\n current_start = directory_start - self.start\n filestart = 0\n compressedsize = 0\n tableOfContents = []\n\n try:\n while True:\n # get file name size (n), extra len (m) and comm len (k)\n zip_n = unpack(\"H\", self.raw_bytes[current_start + 28: current_start + 28 + 2])[0]\n zip_m = unpack(\"H\", self.raw_bytes[current_start + 30: current_start + 30 + 2])[0]\n zip_k = unpack(\"H\", self.raw_bytes[current_start + 32: current_start + 32 + 2])[0]\n\n filename = self.raw_bytes[current_start + 46: current_start + 46 + zip_n]\n\n # check if this is the index file\n filestart = unpack(\"I\", self.raw_bytes[current_start + 42: current_start + 42 + 4])[0]\n compressedsize = unpack(\"I\", self.raw_bytes[current_start + 20: current_start + 20 + 4])[0]\n uncompressedsize = unpack(\"I\", self.raw_bytes[current_start + 24: current_start + 24 + 4])[0]\n tableItem = {\n 'filename': filename,\n 'compressedsize': compressedsize,\n 'uncompressedsize': uncompressedsize,\n 'filestart': filestart\n }\n tableOfContents.append(tableItem)\n\n # not this file, move along\n current_start = current_start + 46 + zip_n + zip_m + zip_k\n except:\n pass\n\n self.tableOfContents = tableOfContents\n return tableOfContents", "response": "This function returns the contents of the internal tableOfContents list. It will raise an exception if the server does not support ranged requests."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_postcard(self, npix=300, shape=(1070, 1132), buffer_size=15):\n \n source = self.kic\n client = kplr.API()\n targ = client.target(source)\n \n channel = [targ.params['Channel_0'], targ.params['Channel_1'], \n targ.params['Channel_2'], targ.params['Channel_3']]\n\n col = [targ.params['Column_0'], targ.params['Column_1'], \n targ.params['Column_2'], targ.params['Column_3']] \n\n row = [targ.params['Row_0'], targ.params['Row_1'], \n targ.params['Row_2'], targ.params['Row_3']] \n \n\n if None in row:\n raise ValueError('Star not on detector all quarters!')\n\n if None in col:\n raise ValueError('Star not on detector all quarters!')\n \n center = np.array([npix/2, npix/2])\n \n \n # If star close to edge, shift frame so that we have the full npix by npix\n # In this case, postcard will not be centered on target star\n if (np.min(col) < npix/2):\n jump = npix/2 - np.min(col) + buffer_size\n col += jump\n center[1] -= jump\n \n if (np.min(row) < npix/2):\n jump = npix/2 - np.min(row) + buffer_size\n row += jump\n center[0] -= jump\n \n if (np.max(row) > shape[0] - npix/2):\n jump = shape[0]-npix/2 - np.max(row) - buffer_size\n row += jump\n center[0] -= jump\n \n if (np.max(col) > shape[1] - npix/2):\n jump = shape[1]-npix/2 - np.max(col) - buffer_size\n col += jump\n center[1] -= jump\n \n fin_arr = np.zeros((len(self.times), npix, npix))\n\n for icount, iname in enumerate(self.obs_filenames): \n a = fits.open(self.ffi_dir+iname)\n\n \n quarter = a[0].header['quarter']\n\n if int(quarter) == 0:\n season = 3\n else:\n season = (int(quarter) - 2) % 4\n\n #season_arr[icount] = season\n img = a[channel[season]].data\n img -= np.median(img)\n \n \n ymin = int(max([int(row[season])-npix/2,0]))\n ymax = int(min([int(row[season])+npix/2,img.shape[0]]))\n xmin = int(max([int(col[season])-npix/2,0]))\n xmax = int(min([int(col[season])+npix/2,img.shape[1]]))\n\n pimg = img[ymin:ymax,xmin:xmax]\n fin_arr[icount,:,:] = pimg\n \n self.postcard = fin_arr\n self.integrated_postcard = np.sum(self.postcard, axis=0)\n self.center = center", "response": "This function will make a postcard region around the target star."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_other_sources(self, edge_lim = 0.015, min_val = 5000, \n ntargets = 250, extend_region_size=3, remove_excess=4,\n plot_flag = False, plot_window=15):\n \"\"\"\n Identify apertures for all sources on the postcard, both for the \n target and potential reference stars\n \n Args: \n edge_lim: The initial limit for the creation of apertures. The aperture will be a region of\n contiguous pixels with flux values larger than the product of ``edge_lim`` and the brightest\n pixel value for this star, as long as that product is larger than ``min_val`` (default: ``0.015``)\n min_val: Threshold for the minimum flux value in the ``integrated_postcard`` for a pixel to be included \n in the default apertures (default: ``5000``)\n ntargets: The maximum number of potential reference stars to be included in the analysis (default: ``250``)\n extend_region_size: After the initial apertures are generated, they will be optionally extended an\n additional number of pixels following this flag. Safe practice for reasonable apertures is to \n leave ``min_val`` at a value well above the noise and then extend apertures via this flag until \n they are of suitable size (default: ``3``)\n remove_excess: Stars with apertures that touch will be combined into a single aperture. \n This is done by iterating through the starlist; this flag represents the number of times the\n list will be iterated through to delete redundant apertures (default: ``4``)\n plot_flag: If true, a series of diagnostic plots will appear while this function runs to observe\n apertures for the target star and other stars.\n (default: ``False``)\n plot_window: If ``plot_flag`` is ``True``, the size of the region to be plotted around the target star\n to show the drawn aperture for visualization purposes only (default: ``15``)\n \n \"\"\"\n j,i = self.center\n \n region = self.integrated_postcard + 0.0\n if plot_flag == True:\n ff = plt.imshow(self.integrated_postcard, interpolation='nearest', cmap='gray', \n vmax = np.percentile(region, 99.6))\n plt.colorbar(ff)\n plt.show()\n targets = np.zeros_like(self.integrated_postcard)\n sizeimg = np.shape(targets)[0]\n \n jj = j + 0\n ii = i + 0\n \n edge = edge_lim\n \n lim = max(min_val, self.integrated_postcard[int(j), int(i)]*edge)\n \n maxpt = np.percentile(self.integrated_postcard, 94)\n \n bin_img = (region > lim)\n lab_img, n_features = label(bin_img)\n \n key_targ = (lab_img == (lab_img[int(j), int(i)]))\n tot = np.sum(key_targ)\n \n targets[key_targ] = 1\n region[key_targ] = 0.0\n \n\n \n lim = np.zeros(ntargets)\n for peaks in range(1,ntargets):\n k = np.argmax(region)\n j,i = np.unravel_index(k, region.shape)\n lim[peaks] = max(maxpt, edge*region[j,i])\n\n\n bin_img = (region >= lim[peaks])\n lab_img, n_features = label(bin_img)\n\n key_targ = (lab_img == (lab_img[j,i]))\n targets[key_targ] = peaks + 1\n region[key_targ] = 0.0\n \n lab_img, n_features = label(targets)\n for i in range(1, ntargets+1):\n for j in range(extend_region_size):\n border= mh.labeled.border(targets, 0, i)\n\n targets[border*(region < (10)*lim[peaks])] = i\n for i in range(2, ntargets+1):\n for j in range(2, ntargets+1):\n if i != j:\n border = mh.labeled.border(targets, i, j)\n if np.sum(border) != 0:\n targets[targets == j] = i\n \n targets = mh.labeled.remove_bordering(targets)\n for k in range(remove_excess):\n for i in range(ntargets):\n if np.sum(self.integrated_postcard[targets == i]) < 0.01:\n targets[targets > i] -= 1\n\n self.targets = targets\n \n if plot_flag == True:\n plt.imshow(self.targets, interpolation='nearest')\n plt.show()\n\n plt.imshow(((targets == 1)*self.integrated_postcard + (targets == 1)*100000)\n [jj-plot_window:jj+plot_window,ii-plot_window:ii+plot_window], \n interpolation='nearest', cmap='gray', vmax=np.percentile(self.integrated_postcard, 99.6))\n plt.show()\n\n plt.imshow((np.ceil(targets/100.0)*self.integrated_postcard+np.ceil(targets/500.0)*3500000), \n interpolation='nearest', cmap='gray', vmax=np.percentile(self.integrated_postcard, 99.99))\n plt.show()", "response": "Find other sources on the postcard."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndoes photometry and estimates uncertainties by calculating the scatter around a linear fit to the data.", "response": "def do_photometry(self):\n \"\"\"\n Does photometry and estimates uncertainties by calculating the scatter around a linear fit to the data\n in each orientation. This function is called by other functions and generally the user will not need\n to interact with it directly.\n \"\"\"\n \n std_f = np.zeros(4)\n data_save = np.zeros_like(self.postcard)\n self.obs_flux = np.zeros_like(self.reference_flux)\n\n\n for i in range(4):\n g = np.where(self.qs == i)[0]\n wh = np.where(self.times[g] > 54947)\n\n data_save[g] = np.roll(self.postcard[g], int(self.roll_best[i,0]), axis=1)\n data_save[g] = np.roll(data_save[g], int(self.roll_best[i,1]), axis=2)\n\n self.target_flux_pixels = data_save[:,self.targets == 1]\n self.target_flux = np.sum(self.target_flux_pixels, axis=1)\n \n self.obs_flux[g] = self.target_flux[g] / self.reference_flux[g]\n self.obs_flux[g] /= np.median(self.obs_flux[g[wh]])\n \n fitline = np.polyfit(self.times[g][wh], self.obs_flux[g][wh], 1)\n std_f[i] = np.max([np.std(self.obs_flux[g][wh]/(fitline[0]*self.times[g][wh]+fitline[1])), 0.001])\n \n self.flux_uncert = std_f"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_panel(self, img):\n plt.figure(figsize=(14,6))\n ax = plt.gca()\n fig = plt.gcf()\n plt.subplot(122)\n \n \n data_save = np.zeros_like(self.postcard)\n self.roll_best = np.zeros((4,2))\n \n for i in range(4):\n g = np.where(self.qs == i)[0]\n wh = np.where(self.times[g] > 54947)\n\n self.roll_best[i] = self.do_rolltest(g, wh)\n \n self.do_photometry()\n for i in range(4):\n g = np.where(self.qs == i)[0]\n plt.errorbar(self.times[g], self.obs_flux[g], yerr=self.flux_uncert[i], fmt=fmt[i])\n \n plt.xlabel('Time', fontsize=20)\n plt.ylabel('Relative Flux', fontsize=20)\n \n \n plt.subplot(121)\n implot = plt.imshow(img, interpolation='nearest', cmap='gray', vmin=98000*52, vmax=104000*52)\n cid = fig.canvas.mpl_connect('button_press_event', self.onclick)\n \n plt.show(block=True)", "response": "Generates the figure shown in adjust_aperture for visualization purposes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef adjust_aperture(self, image_region=15, ignore_bright=0):\n self.ignore_bright = ignore_bright\n self.calc_fluxes()\n \n self.coordsx = []\n self.coordsy = []\n \n jj, ii = self.center\n jj, ii = int(jj), int(ii) # Indices must be integer\n plt.ion()\n img = np.sum(((self.targets == 1)*self.postcard + (self.targets == 1)*100000)\n [:,jj-image_region:jj+image_region,ii-image_region:ii+image_region], axis=0)\n self.generate_panel(img)\n \n while len(self.coordsx) != 0:\n for i in range(len(self.coordsx)):\n if self.targets[self.coordsy[i]+jj-image_region,self.coordsx[i]+ii-image_region] != 1:\n self.targets[self.coordsy[i]+jj-image_region,self.coordsx[i]+ii-image_region] = 1\n elif self.targets[self.coordsy[i]+jj-image_region,self.coordsx[i]+ii-image_region] == 1:\n self.targets[self.coordsy[i]+jj-image_region,self.coordsx[i]+ii-image_region] = 0\n if self.coordsy[i] == 0:\n thiscol = np.where(self.targets[:,self.coordsx[i]+ii-image_region] == 1)\n self.targets[thiscol,self.coordsx[i]+ii-image_region] = 0\n if self.coordsx[i] == 0:\n thiscol = np.where(self.targets[self.coordsy[i]+jj-image_region,:] == 1)\n self.targets[self.coordsy[i]+jj-image_region, thiscol] = 0\n\n\n self.coordsx = []\n self.coordsy = []\n img = np.sum(((self.targets == 1)*self.postcard + \n (self.targets == 1)*100000)[:,jj-image_region:jj+image_region,ii-image_region:ii+image_region],\n axis=0)\n self.generate_panel(img)", "response": "This function updates the aperture and the light curve in the aperture panel."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the normalized photometry for the target aperture.", "response": "def data_for_target(self, do_roll=True, ignore_bright=0):\n \"\"\"\n Determine the normalized photometry, accounting for effects shared by reference stars. Does not provide\n the opportunity to adjust the aperture\n \n Args: \n image_region: If ``True`` allow the aperture to be shifted up to one pixel in both the x and y\n directions to account for differential velocity aberration (default: ``True``)\n ignore_bright: The number of brightest stars to be ignored in the determination of the flux from \n reference stars. If there is reason to believe (for example) that saturated stars may behave\n differently than the target star, they can be avoided with this flag (default: ``0``)\n \n \"\"\"\n self.ignore_bright = ignore_bright\n self.calc_fluxes()\n self.roll_best = np.zeros((4,2))\n\n \n if do_roll == True:\n for i in range(4):\n g = np.where(self.qs == i)[0]\n wh = np.where(self.times[g] > 54947)\n\n self.roll_best[i] = self.do_rolltest(g, wh)\n \n self.do_photometry()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the total flux in each object in each epoch and returns a numpy array of the fluxes in each object.", "response": "def calc_fluxes(self, min_flux = 5000, outlier_iterations=5,\n max_outlier_obs=4, outlier_limit=1.7):\n \"\"\"\n Determine the suitable reference stars, and then the total flux in those stars and \n in the target star in each epoch\n \n Args: \n min_flux: The size of the region around the target star to be plotted. Images will be a square \n with side length ``image_region`` (default: ``5000``)\n outlier_iterations: The number of iterations to remove outliers from the reference star sample\n (stars at epochs with more than ``max_outlier_obs`` observations more than ``outlier_limit`` standard\n deviations from the median value for all stars after normalization) (default: ``5``)\n max_outlier_obs: The maximum number of epochs at which a star is allowed to be more than ``outlier_limit``\n standard deviations from the median value for all stars before it is removed as a suitable\n reference star (default: ``4``)\n outlier_limit: The level of deviation (measured in standard deviations) which a target is allowed\n to be discrepant from the median. If it is this discrepant at more than ``max_outlier_obs``\n epochs, it is removed from consideration (default: ``1.7``)\n \n \"\"\"\n \n jj, ii = self.center\n \n\n \n numer = np.zeros(len(self.times))\n denom = np.zeros(len(self.times))\n factr = np.zeros(len(self.times))\n \n numer_pix = self.postcard[:,self.targets == 1]\n numer = np.sum(numer_pix, axis=1)\n \n tar_vals = np.zeros((len(self.times), int(np.max(self.targets)+1-2-self.ignore_bright)))\n \n for i in range(2+self.ignore_bright,int(np.max(self.targets)+1)):\n tval = np.sum(self.postcard[:,self.targets == i], axis=1)\n #denom += tval/np.median(tval)\n tar_vals[:,i-2-self.ignore_bright] = tval #/ np.median(tval)\n \n for i in range(len(self.obs_filenames)):\n if np.max(tar_vals[i]) < min_flux:\n tar_vals[self.qs == self.qs[i]] = 0.0\n\n all_tar = np.zeros((len(self.times), int(np.max(self.targets)-self.ignore_bright)))\n all_tar[:,0] = numer\n all_tar[:,1:] = tar_vals\n \n self.photometry_array = all_tar\n \n for i in range(len(tar_vals[0])):\n for j in range(4):\n g = np.where(self.qs == j)[0] \n tar_vals[g,i] /= (np.median(tar_vals[g,i])+1e-15)\n \n\n tar_vals_old = tar_vals + 0.0\n \n for i in range(outlier_iterations):\n nonzeros = np.where(tar_vals[0,:] != 0)[0]\n med = np.median(tar_vals[:,nonzeros], axis=1)\n std = np.std(tar_vals[:,nonzeros], axis=1)\n\n if np.sum(tar_vals) != 0.0:\n tar_vals_old = tar_vals + 0.0\n\n for k in range(len(tar_vals[0])):\n h = np.where((np.abs(med-tar_vals[:,k])/std) > outlier_limit)[0]\n if len(h) >= max_outlier_obs:\n tar_vals[:,k] = 0\n\n if np.sum(tar_vals) == 0.0:\n tar_vals = tar_vals_old + 0.0\n\n denom = np.sum(tar_vals, axis=1)\n self.target_flux_pixels = numer_pix\n self.reference_flux = denom"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the centroid positions for the target star at all epochs.", "response": "def calc_centroids(self):\n \"\"\"\n Identify the centroid positions for the target star at all epochs. Useful for verifying that there is\n no correlation between flux and position, as might be expected for high proper motion stars.\n \"\"\"\n self.cm = np.zeros((len(self.postcard), 2))\n for i in range(len(self.postcard)):\n target = self.postcard[i]\n target[self.targets != 1] = 0.0\n self.cm[i] = center_of_mass(target)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndefining the spotsignal for each single cadence object.", "response": "def define_spotsignal(self):\n \"\"\"\n Identify the \"expected\" flux value at the time of each observation based on the \n Kepler long-cadence data, to ensure variations observed are not the effects of a single\n large starspot. Only works if the target star was targeted for long or short cadence\n observations during the primary mission.\n \"\"\"\n client = kplr.API()\n star = client.star(self.kic)\n\n lcs = star.get_light_curves(short_cadence=False)\n time, flux, ferr, qual = [], [], [], []\n for lc in lcs:\n with lc.open() as f:\n hdu_data = f[1].data\n time.append(hdu_data[\"time\"])\n flux.append(hdu_data[\"pdcsap_flux\"])\n ferr.append(hdu_data[\"pdcsap_flux_err\"])\n qual.append(hdu_data[\"sap_quality\"])\n tout = np.array([])\n fout = np.array([])\n eout = np.array([])\n for i in range(len(flux)):\n t = time[i][qual[i] == 0]\n f = flux[i][qual[i] == 0]\n e = ferr[i][qual[i] == 0]\n\n t = t[np.isfinite(f)]\n e = e[np.isfinite(f)]\n f = f[np.isfinite(f)]\n\n e /= np.median(f)\n f /= np.median(f)\n tout = np.append(tout, t[50:]+54833)\n fout = np.append(fout, f[50:])\n eout = np.append(eout, e[50:])\n\n self.spot_signal = np.zeros(52)\n\n for i in range(len(self.times)):\n if self.times[i] < 55000:\n self.spot_signal[i] = 1.0\n else:\n self.spot_signal[i] = fout[np.abs(self.times[i] - tout) == np.min(np.abs(self.times[i] - tout))]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef model_uncert(self):\n Y = self.photometry_array.T\n Y /= np.median(Y, axis=1)[:, None]\n C = np.median(Y, axis=0)\n \n nstars, nobs = np.shape(Y)\n \n Z = np.empty((nstars, 4))\n \n qs = self.qs.astype(int)\n \n for s in range(4):\n Z[:, s] = np.median((Y / C)[:, qs == s], axis=1)\n\n resid2 = (Y - Z[:, qs] * C)**2\n z = Z[:, qs]\n trend = z * C[None, :]\n \n lnS = np.log(np.nanmedian(resid2, axis=0))\n jitter = np.log(0.1*np.nanmedian(np.abs(np.diff(Y, axis=1))))\n\n cal_ferr = np.sqrt(np.exp(2*(jitter/trend))+z**2*np.exp(lnS)[None, :])\n \n self.modeled_uncert = cal_ferr\n self.target_uncert = cal_ferr[0]", "response": "Estimate the photometric uncertainties on each data point following Equation A. 2 of The Paper."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nappending line to internal list.", "response": "def _print(self, line=''):\n \"\"\"Append line to internal list. \n Uses self.tabs to format indents.\n \n Keyword arguments:\n line -- line to append\n \"\"\"\n self.lines.append('{}{}'.format('\\t'*self.tabs , line))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndumping single enum type.", "response": "def _dump_enum(self, e, top=''):\n \"\"\"Dump single enum type.\n \n Keyword arguments:\n top -- top namespace\n \"\"\"\n self._print()\n self._print('enum {} {{'.format(e.name))\n self.defines.append('{}.{}'.format(top,e.name))\n \n self.tabs+=1\n for v in e.value:\n self._print('{} = {};'.format(v.name, v.number))\n self.tabs-=1\n self._print('}')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndump a single field.", "response": "def _dump_field(self, fd):\n \"\"\"Dump single field.\n \"\"\"\n v = {}\n v['label'] = Pbd.LABELS[fd.label]\n v['type'] = fd.type_name if len(fd.type_name) > 0 else Pbd.TYPES[fd.type]\n v['name'] = fd.name\n v['number'] = fd.number\n v['default'] = '[default = {}]'.format(fd.default_value) if len(fd.default_value) > 0 else ''\n \n f = '{label} {type} {name} = {number} {default};'.format(**v)\n f = ' '.join(f.split())\n self._print(f)\n \n if len(fd.type_name) > 0:\n self.uses.append(fd.type_name)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _dump_message(self, m, top=''):\n self._print()\n self._print('message {} {{'.format(m.name))\n self.defines.append('{}.{}'.format(top, m.name))\n self.tabs+=1\n \n for f in m.field:\n self._dump_field(f)\n \n for e in m.enum_type:\n self._dump_enum(e, top='{}.{}'.format(top, m.name))\n \n for n in m.nested_type:\n self._dump_message(n, top='{}.{}'.format(top, m.name))\n \n self.tabs-=1\n self._print('}')", "response": "Dump a single message type."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _walk(self, fd):\n top = '.{}'.format(fd.package) if len(fd.package) > 0 else ''\n \n for e in fd.enum_type: self._dump_enum(e, top)\n for m in fd.message_type: self. _dump_message(m, top)", "response": "Walk and dump the class metadata."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisassembling serialized protocol buffers file.", "response": "def disassemble(self):\n \"\"\"Disassemble serialized protocol buffers file.\n \"\"\"\n ser_pb = open(self.input_file, 'rb').read() # Read serialized pb file\n \n fd = FileDescriptorProto()\n fd.ParseFromString(ser_pb)\n self.name = fd.name\n \n self._print('// Reversed by pbd (https://github.com/rsc-dev/pbd)')\n self._print('syntax = \"proto2\";')\n self._print('')\n \n if len(fd.package) > 0:\n self._print('package {};'.format(fd.package))\n self.package = fd.package\n else:\n self._print('// Package not defined')\n \n self._walk(fd)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndumps the proto file to given directory.", "response": "def dump(self, out_dir='.'):\n \"\"\"Dump proto file to given directory.\n \n Keyword arguments:\n out_dir -- dump directory. Default='.'\n \"\"\"\n uri = out_dir + os.sep + self.name\n with open(uri, 'w') as fh:\n fh.write('\\n'.join(self.lines))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds all missing imports in list of Pbd instances.", "response": "def find_imports(self, pbds):\n \"\"\"Find all missing imports in list of Pbd instances.\n \"\"\"\n # List of types used, but not defined\n imports = list(set(self.uses).difference(set(self.defines)))\n \n # Clumpsy, but enought for now \n for imp in imports:\n for p in pbds:\n if imp in p.defines:\n self.imports.append(p.name)\n break\n \n self.imports = list(set(self.imports))\n \n for import_file in self.imports:\n self.lines.insert(2, 'import \"{}\";'.format(import_file))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a dict from a fasta file and the number of sequences as the second return value.", "response": "def fasta_file_to_dict(fasta_file, id=True, header=False, seq=False):\n \"\"\"Returns a dict from a fasta file and the number of sequences as the second return value.\n fasta_file can be a string path or a file object.\n The key of fasta_dict can be set using the keyword arguments and\n results in a combination of id, header, sequence, in that order. joined with '||'. (default: id)\n Duplicate keys are checked and a warning is logged if found.\n The value of fasta_dict is a python dict with 3 keys: header, id and seq\n\n Changelog:\n 2014/11/17:\n * Added support for url escaped id\n \"\"\"\n fasta_file_f = fasta_file\n if isinstance(fasta_file, str):\n fasta_file_f = open(fasta_file, 'rb')\n\n fasta_dict = OrderedDict()\n keys = ['id', 'header', 'seq']\n flags = dict([('id', id), ('header', header), ('seq', seq)])\n entry = dict([('id', ''), ('header', ''), ('seq', '')])\n count = 0\n line_num = 0\n\n for line in fasta_file_f:\n line = line.strip()\n if line and line[0] == '>':\n count += 1\n key = '||'.join([entry[i] for i in keys if flags[i]])\n if key: # key != ''\n if key in fasta_dict: # check for duplicate key\n logger.warning('%s : Line %d : Duplicate %s [%s] : ID = [%s].', fasta_file_f.name, line_num, '||'.join([i for i in keys if flags[i]]), key[:25] + (key[25:] and '..'), entry['id'])\n entry['seq'] = ''.join(entry['seq'])\n fasta_dict[key] = entry\n # check for url escaped id\n if id:\n unescaped_id = unquote(entry['id'])\n if id != unescaped_id:\n key = '||'.join([unescaped_id] + [entry[i] for i in keys if i != 'id' and flags[i]])\n entry['unescaped_id'] = unescaped_id\n fasta_dict[key] = entry\n entry = dict()\n entry['header'] = line\n entry['id'] = line.split()[0][1:]\n entry['seq'] = []\n else:\n entry['seq'].append(line.upper())\n line_num += 1\n\n if isinstance(fasta_file, str):\n fasta_file_f.close()\n\n key = '||'.join([entry[i] for i in keys if flags[i]])\n if key: # key != ''\n if key in fasta_dict:\n logger.warning('%s : Line %d : Duplicate %s [%s] : ID = [%s].', fasta_file_f.name, line_num, '||'.join([i for i in keys if flags[i]]), key[:25] + (key[25:] and '..'), entry['id'])\n entry['seq'] = ''.join(entry['seq'])\n fasta_dict[key] = entry\n # check for url escaped id\n if id:\n unescaped_id = unquote(entry['id'])\n if id != unescaped_id:\n key = '||'.join([unescaped_id] + [entry[i] for i in keys if i != 'id' and flags[i]])\n entry['unescaped_id'] = unescaped_id\n fasta_dict[key] = entry\n\n return fasta_dict, count"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fasta_dict_to_file(fasta_dict, fasta_file, line_char_limit=None):\n fasta_fp = fasta_file\n if isinstance(fasta_file, str):\n fasta_fp = open(fasta_file, 'wb')\n\n for key in fasta_dict:\n seq = fasta_dict[key]['seq']\n if line_char_limit:\n seq = '\\n'.join([seq[i:i+line_char_limit] for i in range(0, len(seq), line_char_limit)])\n fasta_fp.write(u'{0:s}\\n{1:s}\\n'.format(fasta_dict[key]['header'], seq))", "response": "Write a dictionary to a fasta file"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck whether child features are within the coordinate boundaries of parent features.", "response": "def check_parent_boundary(self):\n \"\"\"\n checks whether child features are within the coordinate boundaries of parent features\n\n :return:\n \"\"\"\n for line in self.lines:\n for parent_feature in line['parents']:\n ok = False\n for parent_line in parent_feature:\n if parent_line['start'] <= line['start'] and line['end'] <= parent_line['end']:\n ok = True\n break\n if not ok:\n self.add_line_error(line, {'message': 'This feature is not contained within the feature boundaries of parent: {0:s}: {1:s}'.format(\n parent_feature[0]['attributes']['ID'],\n ','.join(['({0:s}, {1:d}, {2:d})'.format(line['seqid'], line['start'], line['end']) for line in parent_feature])\n ), 'error_type': 'BOUNDS', 'location': 'parent_boundary'})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if the phase of the current node is correct", "response": "def check_phase(self):\n \"\"\"\n 1. get a list of CDS with the same parent\n 2. sort according to strand\n 3. calculate and validate phase\n \"\"\"\n plus_minus = set(['+', '-'])\n for k, g in groupby(sorted([line for line in self.lines if line['line_type'] == 'feature' and line['type'] == 'CDS' and 'Parent' in line['attributes']], key=lambda x: x['attributes']['Parent']), key=lambda x: x['attributes']['Parent']):\n cds_list = list(g)\n strand_set = list(set([line['strand'] for line in cds_list]))\n if len(strand_set) != 1:\n for line in cds_list:\n self.add_line_error(line, {'message': 'Inconsistent CDS strand with parent: {0:s}'.format(k), 'error_type': 'STRAND'})\n continue\n if len(cds_list) == 1:\n if cds_list[0]['phase'] != 0:\n self.add_line_error(cds_list[0], {'message': 'Wrong phase {0:d}, should be {1:d}'.format(cds_list[0]['phase'], 0), 'error_type': 'PHASE'})\n continue\n strand = strand_set[0]\n if strand not in plus_minus:\n # don't process unknown strands\n continue\n if strand == '-':\n # sort end descending\n sorted_cds_list = sorted(cds_list, key=lambda x: x['end'], reverse=True)\n else:\n sorted_cds_list = sorted(cds_list, key=lambda x: x['start'])\n phase = 0\n for line in sorted_cds_list:\n if line['phase'] != phase:\n self.add_line_error(line, {'message': 'Wrong phase {0:d}, should be {1:d}'.format(line['phase'], phase), 'error_type': 'PHASE'})\n phase = (3 - ((line['end'] - line['start'] + 1 - phase) % 3)) % 3"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_reference(self, sequence_region=False, fasta_embedded=False, fasta_external=False, check_bounds=True, check_n=True, allowed_num_of_n=0, feature_types=('CDS',)):\n # collect lines with errors in this set\n error_lines = set()\n # check if we have a parsed gff3\n if not self.lines:\n self.logger.debug('.parse(gff_file) before calling .check_bounds()')\n return error_lines\n # setup default line_types\n check_n_feature_types = set(feature_types)\n if len(check_n_feature_types) == 0:\n check_n_feature_types.add('CDS')\n # compile regex\n n_segments_finditer = re.compile(r'[Nn]+').finditer\n # check_all_sources mode\n check_all_sources = True\n if sequence_region or fasta_embedded or fasta_external:\n check_all_sources = False\n # get a list of line_data with valid start and end coordinates and unescape the seqid\n start_end_error_locations = set(('start', 'end', 'start,end'))\n valid_line_data_seqid = [(line_data, unquote(line_data['seqid'])) for line_data in self.lines if line_data['line_type'] == 'feature' and line_data['seqid'] != '.' and (not line_data['line_errors'] or not [error_info for error_info in line_data['line_errors'] if 'location' in error_info and error_info['location'] in start_end_error_locations])]\n checked_at_least_one_source = False\n # check directive\n # don't use any directives with errors\n valid_sequence_regions = dict([(unquote(line_data['seqid']), line_data) for line_data in self.lines if line_data['directive'] == '##sequence-region' and not line_data['line_errors']])\n unresolved_seqid = set()\n if (check_all_sources or sequence_region) and valid_sequence_regions:\n checked_at_least_one_source = True\n for line_data, seqid in valid_line_data_seqid:\n if seqid not in valid_sequence_regions and seqid not in unresolved_seqid:\n unresolved_seqid.add(seqid)\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': u'Seqid not found in any ##sequence-region: {0:s}'.format(\n seqid), 'error_type': 'BOUNDS', 'location': 'sequence_region'})\n continue\n if line_data['start'] < valid_sequence_regions[seqid]['start']:\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'Start is less than the ##sequence-region start: %d' % valid_sequence_regions[seqid]['start'], 'error_type': 'BOUNDS', 'location': 'sequence_region'})\n if line_data['end'] > valid_sequence_regions[seqid]['end']:\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'End is greater than the ##sequence-region end: %d' % valid_sequence_regions[seqid]['end'], 'error_type': 'BOUNDS', 'location': 'sequence_region'})\n elif sequence_region:\n self.logger.debug('##sequence-region not found in GFF3')\n # check fasta_embedded\n unresolved_seqid = set()\n if (check_all_sources or fasta_embedded) and self.fasta_embedded:\n checked_at_least_one_source = True\n for line_data, seqid in valid_line_data_seqid:\n if seqid not in self.fasta_embedded and seqid not in unresolved_seqid:\n unresolved_seqid.add(seqid)\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'Seqid not found in the embedded ##FASTA: %s' % seqid, 'error_type': 'BOUNDS', 'location': 'fasta_embedded'})\n continue\n # check bounds\n if line_data['end'] > len(self.fasta_embedded[seqid]['seq']):\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'End is greater than the embedded ##FASTA sequence length: %d' % len(self.fasta_embedded[seqid]['seq']), 'error_type': 'BOUNDS', 'location': 'fasta_embedded'})\n # check n\n if check_n and line_data['type'] in check_n_feature_types:\n \"\"\"\n >>> timeit(\"a.lower().count('n')\", \"import re; a = ('ASDKADSJHFIUDNNNNNNNnnnnSHFD'*50)\")\n 5.540903252684302\n >>> timeit(\"a.count('n'); a.count('N')\", \"import re; a = ('ASDKADSJHFIUDNNNNNNNnnnnSHFD'*50)\")\n 2.3504867946058425\n >>> timeit(\"re.findall('[Nn]+', a)\", \"import re; a = ('ASDKADSJHFIUDNNNNNNNnnnnSHFD'*50)\")\n 30.60731204915959\n \"\"\"\n n_count = self.fasta_embedded[seqid]['seq'].count('N', line_data['start'] - 1, line_data['end']) + self.fasta_embedded[seqid]['seq'].count('n', line_data['start'] - 1, line_data['end'])\n if n_count > allowed_num_of_n:\n # get detailed segments info\n n_segments = [(m.start(), m.end() - m.start()) for m in n_segments_finditer(self.fasta_embedded[seqid]['seq'], line_data['start'] - 1, line_data['end'])]\n n_segments_str = ['(%d, %d)' % (m[0], m[1]) for m in n_segments]\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'Found %d Ns in %s feature of length %d using the embedded ##FASTA, consists of %d segment (start, length): %s' % (n_count, line_data['type'], line_data['end'] - line_data['start'], len(n_segments), ', '.join(n_segments_str)), 'error_type': 'N_COUNT', 'n_segments': n_segments, 'location': 'fasta_embedded'})\n elif fasta_embedded:\n self.logger.debug('Embedded ##FASTA not found in GFF3')\n # check fasta_external\n unresolved_seqid = set()\n if (check_all_sources or fasta_external) and self.fasta_external:\n checked_at_least_one_source = True\n for line_data, seqid in valid_line_data_seqid:\n if seqid not in self.fasta_external and seqid not in unresolved_seqid:\n unresolved_seqid.add(seqid)\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'Seqid not found in the external FASTA file: %s' % seqid, 'error_type': 'BOUNDS', 'location': 'fasta_external'})\n continue\n # check bounds\n if line_data['end'] > len(self.fasta_external[seqid]['seq']):\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'End is greater than the external FASTA sequence length: %d' % len(self.fasta_external[seqid]['seq']), 'error_type': 'BOUNDS', 'location': 'fasta_external'})\n # check n\n if check_n and line_data['type'] in check_n_feature_types:\n n_count = self.fasta_external[seqid]['seq'].count('N', line_data['start'] - 1, line_data['end']) + self.fasta_external[seqid]['seq'].count('n', line_data['start'] - 1, line_data['end'])\n if n_count > allowed_num_of_n:\n # get detailed segments info\n n_segments = [(m.start(), m.end() - m.start()) for m in n_segments_finditer(self.fasta_external[seqid]['seq'], line_data['start'] - 1, line_data['end'])]\n n_segments_str = ['(%d, %d)' % (m[0], m[1]) for m in n_segments]\n error_lines.add(line_data['line_index'])\n self.add_line_error(line_data, {'message': 'Found %d Ns in %s feature of length %d using the external FASTA, consists of %d segment (start, length): %s' % (n_count, line_data['type'], line_data['end'] - line_data['start'], len(n_segments), ', '.join(n_segments_str)), 'error_type': 'N_COUNT', 'n_segments': n_segments, 'location': 'fasta_external'})\n elif fasta_external:\n self.logger.debug('External FASTA file not given')\n if check_all_sources and not checked_at_least_one_source:\n self.logger.debug('Unable to perform bounds check, requires at least one of the following sources: ##sequence-region, embedded ##FASTA, or external FASTA file')\n return error_lines", "response": "Checks the validity of the reference sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse(self, gff_file, strict=False):\n valid_strand = set(('+', '-', '.', '?'))\n valid_phase = set((0, 1, 2))\n multi_value_attributes = set(('Parent', 'Alias', 'Note', 'Dbxref', 'Ontology_term'))\n valid_attribute_target_strand = set(('+', '-', ''))\n reserved_attributes = set(('ID', 'Name', 'Alias', 'Parent', 'Target', 'Gap', 'Derives_from', 'Note', 'Dbxref', 'Ontology_term', 'Is_circular'))\n\n # illegal character check\n # Literal use of tab, newline, carriage return, the percent (%) sign, and control characters must be encoded using RFC 3986 Percent-Encoding; no other characters may be encoded.\n # control characters: \\x00-\\x1f\\x7f this includes tab(\\x09), newline(\\x0a), carriage return(\\x0d)\n # seqid may contain any characters, but must escape any characters not in the set [a-zA-Z0-9.:^*$@!+_?-|]\n # URL escaping rules are used for tags or values containing the following characters: \",=;\".\n #>>> timeit(\"unescaped_seqid('Un.7589')\", \"import re; unescaped_seqid = re.compile(r'[^a-zA-Z0-9.:^*$@!+_?|%-]|%(?![0-9a-fA-F]{2})').search\")\n #0.4128372745785036\n #>>> timeit(\"unescaped_seqid2('Un.7589')\", \"import re; unescaped_seqid2 = re.compile(r'^([a-zA-Z0-9.:^*$@!+_?|-]|%[0-9a-fA-F]{2})+$').search\")\n #0.9012313532265175\n unescaped_seqid = re.compile(r'[^a-zA-Z0-9.:^*$@!+_?|%-]|%(?![0-9a-fA-F]{2})').search\n unescaped_field = re.compile(r'[\\x00-\\x1f\\x7f]|%(?![0-9a-fA-F]{2})').search\n\n gff_fp = gff_file\n if isinstance(gff_file, str):\n gff_fp = open(gff_file, 'r')\n\n lines = []\n current_line_num = 1 # line numbers start at 1\n features = defaultdict(list)\n # key = the unresolved id, value = a list of line_data(dict)\n unresolved_parents = defaultdict(list)\n\n for line_raw in gff_fp:\n line_data = {\n 'line_index': current_line_num - 1,\n 'line_raw': line_raw,\n 'line_status': 'normal',\n 'parents': [],\n 'children': [],\n 'line_type': '',\n 'directive': '',\n 'line_errors': [],\n 'type': '',\n }\n line_strip = line_raw.strip()\n if line_strip != line_raw[:len(line_strip)]:\n self.add_line_error(line_data, {'message': 'White chars not allowed at the start of a line', 'error_type': 'FORMAT', 'location': ''})\n if current_line_num == 1 and not line_strip.startswith('##gff-version'):\n self.add_line_error(line_data, {'message': '\"##gff-version\" missing from the first line', 'error_type': 'FORMAT', 'location': ''})\n if len(line_strip) == 0:\n line_data['line_type'] = 'blank'\n continue\n if line_strip.startswith('##'):\n line_data['line_type'] = 'directive'\n if line_strip.startswith('##sequence-region'):\n # ##sequence-region seqid start end\n # This element is optional, but strongly encouraged because it allows parsers to perform bounds checking on features.\n # only one ##sequence-region directive may be given for any given seqid\n # all features on that landmark feature (having that seqid) must be contained within the range defined by that ##sequence-region diretive. An exception to this rule is allowed when a landmark feature is marked with the Is_circular attribute.\n line_data['directive'] = '##sequence-region'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 3:\n self.add_line_error(line_data, {'message': 'Expecting 3 fields, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['seqid'] = tokens[0]\n # check for duplicate ##sequence-region seqid\n if [True for d in lines if ('directive' in d and d['directive'] == '##sequence-region' and 'seqid' in d and d['seqid'] == line_data['seqid'])]:\n self.add_line_error(line_data, {'message': '##sequence-region seqid: \"%s\" may only appear once' % line_data['seqid'], 'error_type': 'FORMAT', 'location': ''})\n try:\n all_good = True\n try:\n line_data['start'] = int(tokens[1])\n if line_data['start'] < 1:\n self.add_line_error(line_data, {'message': 'Start is not a valid 1-based integer coordinate: \"%s\"' % tokens[1], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n all_good = False\n self.add_line_error(line_data, {'message': 'Start is not a valid integer: \"%s\"' % tokens[1], 'error_type': 'FORMAT', 'location': ''})\n line_data['start'] = tokens[1]\n try:\n line_data['end'] = int(tokens[2])\n if line_data['end'] < 1:\n self.add_line_error(line_data, {'message': 'End is not a valid 1-based integer coordinate: \"%s\"' % tokens[2], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n all_good = False\n self.add_line_error(line_data, {'message': 'End is not a valid integer: \"%s\"' % tokens[2], 'error_type': 'FORMAT', 'location': ''})\n line_data['start'] = tokens[2]\n # if all_good then both start and end are int, so we can check if start is not less than or equal to end\n if all_good and line_data['start'] > line_data['end']:\n self.add_line_error(line_data, {'message': 'Start is not less than or equal to end', 'error_type': 'FORMAT', 'location': ''})\n except IndexError:\n pass\n elif line_strip.startswith('##gff-version'):\n # The GFF version, always 3 in this specification must be present, must be the topmost line of the file and may only appear once in the file.\n line_data['directive'] = '##gff-version'\n # check if it appeared before\n if [True for d in lines if ('directive' in d and d['directive'] == '##gff-version')]:\n self.add_line_error(line_data, {'message': '##gff-version missing from the first line', 'error_type': 'FORMAT', 'location': ''})\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 1:\n self.add_line_error(line_data, {'message': 'Expecting 1 field, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n try:\n line_data['version'] = int(tokens[0])\n if line_data['version'] != 3:\n self.add_line_error(line_data, {'message': 'Version is not \"3\": \"%s\"' % tokens[0], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n self.add_line_error(line_data, {'message': 'Version is not a valid integer: \"%s\"' % tokens[0], 'error_type': 'FORMAT', 'location': ''})\n line_data['version'] = tokens[0]\n elif line_strip.startswith('###'):\n # This directive (three # signs in a row) indicates that all forward references to feature IDs that have been seen to this point have been resolved.\n line_data['directive'] = '###'\n elif line_strip.startswith('##FASTA'):\n # This notation indicates that the annotation portion of the file is at an end and that the\n # remainder of the file contains one or more sequences (nucleotide or protein) in FASTA format.\n line_data['directive'] = '##FASTA'\n self.logger.info('Reading embedded ##FASTA sequence')\n self.fasta_embedded, count = fasta_file_to_dict(gff_fp)\n self.logger.info('%d sequences read' % len(self.fasta_embedded))\n elif line_strip.startswith('##feature-ontology'):\n # ##feature-ontology URI\n # This directive indicates that the GFF3 file uses the ontology of feature types located at the indicated URI or URL.\n line_data['directive'] = '##feature-ontology'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 1:\n self.add_line_error(line_data, {'message': 'Expecting 1 field, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['URI'] = tokens[0]\n elif line_strip.startswith('##attribute-ontology'):\n # ##attribute-ontology URI\n # This directive indicates that the GFF3 uses the ontology of attribute names located at the indicated URI or URL.\n line_data['directive'] = '##attribute-ontology'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 1:\n self.add_line_error(line_data, {'message': 'Expecting 1 field, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['URI'] = tokens[0]\n elif line_strip.startswith('##source-ontology'):\n # ##source-ontology URI\n # This directive indicates that the GFF3 uses the ontology of source names located at the indicated URI or URL.\n line_data['directive'] = '##source-ontology'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 1:\n self.add_line_error(line_data, {'message': 'Expecting 1 field, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['URI'] = tokens[0]\n elif line_strip.startswith('##species'):\n # ##species NCBI_Taxonomy_URI\n # This directive indicates the species that the annotations apply to.\n line_data['directive'] = '##species'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 1:\n self.add_line_error(line_data, {'message': 'Expecting 1 field, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['NCBI_Taxonomy_URI'] = tokens[0]\n elif line_strip.startswith('##genome-build'):\n # ##genome-build source buildName\n # The genome assembly build name used for the coordinates given in the file.\n line_data['directive'] = '##genome-build'\n tokens = list(line_strip.split()[1:])\n if len(tokens) != 2:\n self.add_line_error(line_data, {'message': 'Expecting 2 fields, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n if len(tokens) > 0:\n line_data['source'] = tokens[0]\n try:\n line_data['buildName'] = tokens[1]\n except IndexError:\n pass\n else:\n self.add_line_error(line_data, {'message': 'Unknown directive', 'error_type': 'FORMAT', 'location': ''})\n tokens = list(line_strip.split())\n line_data['directive'] = tokens[0]\n elif line_strip.startswith('#'):\n line_data['line_type'] = 'comment'\n else:\n # line_type may be a feature or unknown\n line_data['line_type'] = 'feature'\n tokens = list(map(str.strip, line_raw.split('\\t')))\n if len(tokens) != 9:\n self.add_line_error(line_data, {'message': 'Features should contain 9 fields, got %d: %s' % (len(tokens) - 1, repr(tokens[1:])), 'error_type': 'FORMAT', 'location': ''})\n for i, t in enumerate(tokens):\n if not t:\n self.add_line_error(line_data, {'message': 'Empty field: %d, must have a \".\"' % (i + 1), 'error_type': 'FORMAT', 'location': ''})\n try:\n line_data['seqid'] = tokens[0]\n if unescaped_seqid(tokens[0]):\n self.add_line_error(line_data, {'message': 'Seqid must escape any characters not in the set [a-zA-Z0-9.:^*$@!+_?-|]: \"%s\"' % tokens[0], 'error_type': 'FORMAT', 'location': ''})\n line_data['source'] = tokens[1]\n if unescaped_field(tokens[1]):\n self.add_line_error(line_data, {'message': 'Source must escape the percent (%%) sign and any control characters: \"%s\"' % tokens[1], 'error_type': 'FORMAT', 'location': ''})\n line_data['type'] = tokens[2]\n if unescaped_field(tokens[2]):\n self.add_line_error(line_data, {'message': 'Type must escape the percent (%%) sign and any control characters: \"%s\"' % tokens[2], 'error_type': 'FORMAT', 'location': ''})\n all_good = True\n try:\n line_data['start'] = int(tokens[3])\n if line_data['start'] < 1:\n self.add_line_error(line_data, {'message': 'Start is not a valid 1-based integer coordinate: \"%s\"' % tokens[3], 'error_type': 'FORMAT', 'location': 'start'})\n except ValueError:\n all_good = False\n line_data['start'] = tokens[3]\n if line_data['start'] != '.':\n self.add_line_error(line_data, {'message': 'Start is not a valid integer: \"%s\"' % line_data['start'], 'error_type': 'FORMAT', 'location': 'start'})\n try:\n line_data['end'] = int(tokens[4])\n if line_data['end'] < 1:\n self.add_line_error(line_data, {'message': 'End is not a valid 1-based integer coordinate: \"%s\"' % tokens[4], 'error_type': 'FORMAT', 'location': 'end'})\n except ValueError:\n all_good = False\n line_data['end'] = tokens[4]\n if line_data['end'] != '.':\n self.add_line_error(line_data, {'message': 'End is not a valid integer: \"%s\"' % line_data['end'], 'error_type': 'FORMAT', 'location': 'end'})\n # if all_good then both start and end are int, so we can check if start is not less than or equal to end\n if all_good and line_data['start'] > line_data['end']:\n self.add_line_error(line_data, {'message': 'Start is not less than or equal to end', 'error_type': 'FORMAT', 'location': 'start,end'})\n try:\n line_data['score'] = float(tokens[5])\n except ValueError:\n line_data['score'] = tokens[5]\n if line_data['score'] != '.':\n self.add_line_error(line_data, {'message': 'Score is not a valid floating point number: \"%s\"' % line_data['score'], 'error_type': 'FORMAT', 'location': ''})\n line_data['strand'] = tokens[6]\n if line_data['strand'] not in valid_strand: # set(['+', '-', '.', '?'])\n self.add_line_error(line_data, {'message': 'Strand has illegal characters: \"%s\"' % tokens[6], 'error_type': 'FORMAT', 'location': ''})\n try:\n line_data['phase'] = int(tokens[7])\n if line_data['phase'] not in valid_phase: # set([0, 1, 2])\n self.add_line_error(line_data, {'message': 'Phase is not 0, 1, or 2: \"%s\"' % tokens[7], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n line_data['phase'] = tokens[7]\n if line_data['phase'] != '.':\n self.add_line_error(line_data, {'message': 'Phase is not a valid integer: \"%s\"' % line_data['phase'], 'error_type': 'FORMAT', 'location': ''})\n elif line_data['type'] == 'CDS':\n self.add_line_error(line_data, {'message': 'Phase is required for all CDS features', 'error_type': 'FORMAT', 'location': ''})\n # parse attributes, ex: ID=exon00003;Parent=mRNA00001,mRNA00003;Name=EXON.1\n # URL escaping rules are used for tags or values containing the following characters: \",=;\". Spaces are allowed in this field, but tabs must be replaced with the %09 URL escape.\n # Note that attribute names are case sensitive. \"Parent\" is not the same as \"parent\".\n # All attributes that begin with an uppercase letter are reserved for later use. Attributes that begin with a lowercase letter can be used freely by applications.\n if unescaped_field(tokens[8]):\n self.add_line_error(line_data, {'message': 'Attributes must escape the percent (%) sign and any control characters', 'error_type': 'FORMAT', 'location': ''})\n attribute_tokens = tuple(tuple(t for t in a.split('=')) for a in tokens[8].split(';') if a)\n line_data['attributes'] = {}\n if len(attribute_tokens) == 1 and len(attribute_tokens[0]) == 1 and attribute_tokens[0][0] == '.':\n pass # no attributes\n else:\n for a in attribute_tokens:\n if len(a) != 2:\n self.add_line_error(line_data, {'message': 'Attributes must contain one and only one equal (=) sign: \"%s\"' % ('='.join(a)), 'error_type': 'FORMAT', 'location': ''})\n try:\n tag, value = a\n except ValueError:\n tag, value = a[0], ''\n if not tag:\n self.add_line_error(line_data, {'message': 'Empty attribute tag: \"%s\"' % '='.join(a), 'error_type': 'FORMAT', 'location': ''})\n if not value.strip():\n self.add_line_error(line_data, {'message': 'Empty attribute value: \"%s\"' % '='.join(a), 'error_type': 'FORMAT', 'location': ''}, log_level=logging.WARNING)\n if tag in line_data['attributes']:\n self.add_line_error(line_data, {'message': 'Found multiple attribute tags: \"%s\"' % tag, 'error_type': 'FORMAT', 'location': ''})\n if tag in multi_value_attributes: # set(['Parent', 'Alias', 'Note', 'Dbxref', 'Ontology_term'])\n if value.find(', ') >= 0:\n self.add_line_error(line_data, {'message': 'Found \", \" in %s attribute, possible unescaped \",\": \"%s\"' % (tag, value), 'error_type': 'FORMAT', 'location': ''}, log_level=logging.WARNING)\n # In addition to Parent, the Alias, Note, Dbxref and Ontology_term attributes can have multiple values.\n if tag in line_data['attributes']: # if this tag has been seen before\n if tag == 'Note': # don't check for duplicate notes\n line_data['attributes'][tag].extend(value.split(','))\n else: # only add non duplicate values\n line_data['attributes'][tag].extend([s for s in value.split(',') if s not in line_data['attributes'][tag]])\n else:\n line_data['attributes'][tag] = value.split(',')\n # check for duplicate values\n if tag != 'Note' and len(line_data['attributes'][tag]) != len(set(line_data['attributes'][tag])):\n count_values = [(len(list(group)), key) for key, group in groupby(sorted(line_data['attributes'][tag]))]\n self.add_line_error(line_data, {'message': '%s attribute has identical values (count, value): %s' % (tag, ', '.join(['(%d, %s)' % (c, v) for c, v in count_values if c > 1])), 'error_type': 'FORMAT', 'location': ''})\n # remove duplicate\n line_data['attributes'][tag] = list(set(line_data['attributes'][tag]))\n\n if tag == 'Parent':\n for feature_id in line_data['attributes']['Parent']:\n try:\n line_data['parents'].append(features[feature_id])\n for ld in features[feature_id]:\n # no need to check if line_data in ld['children'], because it is impossible, each ld maps to only one feature_id, so the ld we get are all different\n ld['children'].append(line_data)\n except KeyError: # features[id]\n self.add_line_error(line_data, {'message': '%s attribute has unresolved forward reference: %s' % (tag, feature_id), 'error_type': 'FORMAT', 'location': ''})\n unresolved_parents[feature_id].append(line_data)\n elif tag == 'Target':\n if value.find(',') >= 0:\n self.add_line_error(line_data, {'message': 'Value of %s attribute contains unescaped \",\": \"%s\"' % (tag, value), 'error_type': 'FORMAT', 'location': ''})\n target_tokens = value.split(' ')\n if len(target_tokens) < 3 or len(target_tokens) > 4:\n self.add_line_error(line_data, {'message': 'Target attribute should have 3 or 4 values, got %d: %s' % (len(target_tokens), repr(tokens)), 'error_type': 'FORMAT', 'location': ''})\n line_data['attributes'][tag] = {}\n try:\n line_data['attributes'][tag]['target_id'] = target_tokens[0]\n all_good = True\n try:\n line_data['attributes'][tag]['start'] = int(target_tokens[1])\n if line_data['attributes'][tag]['start'] < 1:\n self.add_line_error(line_data, {'message': 'Start value of Target attribute is not a valid 1-based integer coordinate: \"%s\"' % target_tokens[1], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n all_good = False\n line_data['attributes'][tag]['start'] = target_tokens[1]\n self.add_line_error(line_data, {'message': 'Start value of Target attribute is not a valid integer: \"%s\"' % line_data['attributes'][tag]['start'], 'error_type': 'FORMAT', 'location': ''})\n try:\n line_data['attributes'][tag]['end'] = int(target_tokens[2])\n if line_data['attributes'][tag]['end'] < 1:\n self.add_line_error(line_data, {'message': 'End value of Target attribute is not a valid 1-based integer coordinate: \"%s\"' % target_tokens[2], 'error_type': 'FORMAT', 'location': ''})\n except ValueError:\n all_good = False\n line_data['attributes'][tag]['end'] = target_tokens[2]\n self.add_line_error(line_data, {'message': 'End value of Target attribute is not a valid integer: \"%s\"' % line_data['attributes'][tag]['end'], 'error_type': 'FORMAT', 'location': ''})\n # if all_good then both start and end are int, so we can check if start is not less than or equal to end\n if all_good and line_data['attributes'][tag]['start'] > line_data['attributes'][tag]['end']:\n self.add_line_error(line_data, {'message': 'Start is not less than or equal to end', 'error_type': 'FORMAT', 'location': ''})\n line_data['attributes'][tag]['strand'] = target_tokens[3]\n if line_data['attributes'][tag]['strand'] not in valid_attribute_target_strand: # set(['+', '-', ''])\n self.add_line_error(line_data, {'message': 'Strand value of Target attribute has illegal characters: \"%s\"' % line_data['attributes'][tag]['strand'], 'error_type': 'FORMAT', 'location': ''})\n except IndexError:\n pass\n else:\n if value.find(',') >= 0:\n self.add_line_error(line_data, {'message': 'Value of %s attribute contains unescaped \",\": \"%s\"' % (tag, value), 'error_type': 'FORMAT', 'location': ''})\n line_data['attributes'][tag] = value\n if tag == 'Is_circular' and value != 'true':\n self.add_line_error(line_data, {'message': 'Value of Is_circular attribute is not \"true\": \"%s\"' % value, 'error_type': 'FORMAT', 'location': ''})\n elif tag[:1].isupper() and tag not in reserved_attributes: # {'ID', 'Name', 'Alias', 'Parent', 'Target', 'Gap', 'Derives_from', 'Note', 'Dbxref', 'Ontology_term', 'Is_circular'}\n self.add_line_error(line_data, {'message': 'Unknown reserved (uppercase) attribute: \"%s\"' % tag, 'error_type': 'FORMAT', 'location': ''})\n elif tag == 'ID':\n # check for duplicate ID in non-adjacent lines\n if value in features and lines[-1]['attributes'][tag] != value:\n self.add_line_error(line_data, {'message': 'Duplicate ID: \"%s\" in non-adjacent lines: %s' % (value, ','.join([str(f['line_index'] + 1) for f in features[value]])), 'error_type': 'FORMAT', 'location': ''}, log_level=logging.WARNING)\n features[value].append(line_data)\n except IndexError:\n pass\n current_line_num += 1\n lines.append(line_data)\n\n if isinstance(gff_file, str):\n gff_fp.close()\n\n # global look up of unresolved parents\n for feature_id in unresolved_parents:\n if feature_id in features:\n for line in unresolved_parents[feature_id]:\n self.add_line_error(line, {'message': 'Unresolved forward reference: \"%s\", found defined in lines: %s' % (feature_id, ','.join([str(ld['line_index'] + 1) for ld in features[feature_id]])), 'error_type': 'FORMAT', 'location': ''})\n\n self.lines = lines\n self.features = features\n return 1", "response": "Parse the GFF file into a set of data structures."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn list of line_data with line_index as first element", "response": "def descendants(self, line_data):\n \"\"\"\n BFS graph algorithm\n :param line_data: line_data(dict) with line_data['line_index'] or line_index(int)\n :return: list of line_data(dict)\n \"\"\"\n # get start node\n try:\n start = line_data['line_index']\n except TypeError:\n start = self.lines[line_data]['line_index']\n visited_set, visited_list, queue = set(), [], [start]\n while queue:\n node = queue.pop(0)\n if node not in visited_set:\n visited_set.add(node)\n visited_list.append(self.lines[node])\n queue.extend([ld['line_index'] for ld in self.lines[node]['children'] if ld['line_index'] not in visited_set])\n return visited_list[1:]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef adopt(self, old_parent, new_parent):\n try: # assume line_data(dict)\n old_id = old_parent['attributes']['ID']\n except TypeError:\n try: # assume line_index(int)\n old_id = self.lines[old_parent]['attributes']['ID']\n except TypeError: # assume feature_id(str)\n old_id = old_parent\n old_feature = self.features[old_id]\n old_indexes = [ld['line_index'] for ld in old_feature]\n try: # assume line_data(dict)\n new_id = new_parent['attributes']['ID']\n except TypeError:\n try: # assume line_index(int)\n new_id = self.lines[new_parent]['attributes']['ID']\n except TypeError: # assume feature_id(str)\n new_id = new_parent\n new_feature = self.features[new_id]\n new_indexes = [ld['line_index'] for ld in new_feature]\n # build a list of children to be moved\n # add the child to the new parent's children list if its not already there\n # update the child's parent list and parent attribute\n # finally remove the old parent's children list\n children = old_feature[0]['children']\n new_parent_children_set = set([ld['line_index'] for ld in new_feature[0]['children']])\n for child in children:\n if child['line_index'] not in new_parent_children_set:\n new_parent_children_set.add(child['line_index'])\n for new_ld in new_feature:\n new_ld['children'].append(child)\n child['parents'].append(new_feature)\n child['attributes']['Parent'].append(new_id)\n # remove multiple, list.remove() only removes 1\n child['parents'] = [f for f in child['parents'] if f[0]['attributes']['ID'] != old_id]\n child['attributes']['Parent'] = [d for d in child['attributes']['Parent'] if d != old_id]\n for old_ld in old_feature:\n old_ld['children'] = []\n return children", "response": "Transfer children from old_parent to new_parent."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove(self, line_data, root_type=None):\n roots = [ld for ld in self.ancestors(line_data) if (root_type and ld['line_type'] == root_type) or (not root_type and not ld['parents'])] or [line_data]\n for root in roots:\n root['line_status'] = 'removed'\n root_descendants = self.descendants(root)\n for root_descendant in root_descendants:\n root_descendant['line_status'] = 'removed'\n root_ancestors = self.ancestors(root) # BFS, so we will process closer ancestors first\n for root_ancestor in root_ancestors:\n if len([ld for ld in root_ancestor['children'] if ld['line_status'] != 'removed']) == 0: # if all children of a root_ancestor is removed\n # remove this root_ancestor\n root_ancestor['line_status'] = 'removed'", "response": "Removes a line_data from the data structure."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sequence(self, line_data, child_type=None, reference=None):\n # get start node\n reference = reference or self.fasta_external or self.fasta_embedded\n if not reference:\n raise Exception('External or embedded fasta reference needed')\n try:\n line_index = line_data['line_index']\n except TypeError:\n line_index = self.lines[line_data]['line_index']\n ld = self.lines[line_index]\n if ld['type'] != 'feature':\n return None\n seq = reference[ld['seqid']][ld['start']-1:ld['end']]\n if ld['strand'] == '-':\n seq = complement(seq[::-1])\n return seq", "response": "Get the sequence of line_data according to the columns seqid start end strand and child_type"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef abfIDfromFname(fname):\n fname=os.path.abspath(fname)\n basename=os.path.basename(fname)\n return os.path.splitext(basename)[0]", "response": "given a filename return the ABFs ID string."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndetermine the protocol used to record an ABF file", "response": "def abfProtocol(fname):\n \"\"\"Determine the protocol used to record an ABF file\"\"\"\n f=open(fname,'rb')\n raw=f.read(30*1000) #it should be in the first 30k of the file\n f.close()\n raw=raw.decode(\"utf-8\",\"ignore\")\n raw=raw.split(\"Clampex\")[1].split(\".pro\")[0]\n protocol = os.path.basename(raw) # the whole protocol filename\n protocolID = protocol.split(\" \")[0] # just the first number\n return protocolID"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving the bytestring ABF header make and launch HTML.", "response": "def headerHTML(header,fname):\n \"\"\"given the bytestring ABF header, make and launch HTML.\"\"\"\n html=\"\"\n html+=\"

%s

\"%(fname)\n html+=pprint.pformat(header, indent=1)\n html=html.replace(\"\\n\",'
').replace(\" \",\" \")\n html=html.replace(r\"\\x00\",\"\")\n html+=\"\"\n print(\"saving header file:\",fname)\n f=open(fname,'w')\n f.write(html)\n f.close()\n webbrowser.open(fname)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setsweep(self, sweep=0, channel=0):\n try:\n sweep=int(sweep)\n except:\n self.log.error(\"trying to set sweep to [%s]\",sweep)\n return\n if sweep<0:\n sweep=self.sweeps-1-sweep # if negative, start from the end\n sweep=max(0,min(sweep,self.sweeps-1)) # correct for out of range sweeps\n if 'sweep' in dir(self) and self.sweep == sweep and self.derivative is False:\n self.log.debug(\"sweep %d already set\",sweep)\n return\n #self.log.debug(\"loading sweep %d (Ch%d)\",sweep,channel)\n self.channels=self.ABFblock.segments[sweep].size[\"analogsignals\"]\n if self.channels>1 and sweep==0:\n self.log.info(\"WARNING: multichannel not yet supported!\") #TODO:\n self.trace = self.ABFblock.segments[sweep].analogsignals[channel]\n self.sweep=sweep # currently selected sweep\n self.channel=channel # currently selected channel\n\n # sweep information\n self.rate = int(self.trace.sampling_rate) # Hz\n self.period = float(1/self.rate) # seconds (inverse of sample rate)\n self.pointsPerSec = int(self.rate) # for easy access\n self.pointsPerMs = int(self.rate/1000.0) # for easy access\n self.sweepSize = len(self.trace) # number of data points per sweep\n self.sweepInterval = self.trace.duration.magnitude # sweep interval (seconds)\n self.sweepLength = float(self.trace.t_stop-self.trace.t_start) # in seconds\n self.length = self.sweepLength*self.sweeps # length (sec) of total recording\n self.lengthMinutes = self.length/60.0 # length (minutes) of total recording\n\n if str(self.trace.dimensionality) == 'pA':\n self.units,self.units2=\"pA\",\"clamp current (pA)\"\n self.unitsD,self.unitsD2=\"pA/ms\",\"current velocity (pA/ms)\"\n self.protoUnits,self.protoUnits2=\"mV\",\"command voltage (mV)\"\n elif str(self.trace.dimensionality) == 'mV':\n self.units,self.units2=\"mV\",\"membrane potential (mV)\"\n self.unitsD,self.unitsD2=\"V/s\",\"potential velocity (V/s)\"\n self.protoUnits,self.protoUnits2=\"pA\",\"command current (pA)\"\n else:\n self.units,self.units2=\"?\",\"unknown units\"\n self.unitsD,self.unitsD2=\"?\",\"unknown units\"\n\n # sweep data\n self.sweepY = self.trace.magnitude # sweep data (mV or pA)\n self.sweepT = self.trace.times.magnitude # actual sweep times (sec)\n self.sweepStart = float(self.trace.t_start) # time start of sweep (sec)\n self.sweepX2 = self.sweepT-self.trace.t_start.magnitude # sweeps overlap\n self.sweepX = self.sweepX2+sweep*self.sweepInterval # assume no gaps\n if self.derivative:\n self.log.debug(\"taking derivative\")\n #self.sweepD=np.diff(self.sweepY) # take derivative\n self.sweepD=self.sweepY[1:]-self.sweepY[:-1] # better?\n self.sweepD=np.insert(self.sweepD,0,self.sweepD[0]) # add a point\n self.sweepD/=(self.period*1000) # correct for sample rate\n else:\n self.sweepD=[0] # derivative is forced to be empty\n\n # generate the protocol too\n self.generate_protocol()", "response": "set the sweep and channel of an ABF. Both start at 0."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\niterate over every sweep", "response": "def setsweeps(self):\n \"\"\"iterate over every sweep\"\"\"\n for sweep in range(self.sweeps):\n self.setsweep(sweep)\n yield self.sweep"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread the header and populate self with information about comments", "response": "def comments_load(self):\n \"\"\"read the header and populate self with information about comments\"\"\"\n self.comment_times,self.comment_sweeps,self.comment_tags=[],[],[]\n self.comments=0 # will be >0 if comments exist\n self.comment_text=\"\"\n\n try:\n # this used to work\n self.comment_tags = list(self.ABFblock.segments[0].eventarrays[0].annotations['comments'])\n self.comment_times = list(self.ABFblock.segments[0].eventarrays[0].times/self.trace.itemsize)\n self.comment_sweeps = list(self.comment_times)\n except:\n # now this notation seems to work\n for events in self.ABFblock.segments[0].events: # this should only happen once actually\n self.comment_tags = events.annotations['comments'].tolist()\n self.comment_times = np.array(events.times.magnitude/self.trace.itemsize)\n self.comment_sweeps = self.comment_times/self.sweepInterval\n\n for i,c in enumerate(self.comment_tags):\n self.comment_tags[i]=c.decode(\"utf-8\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating the protocol for the current sweep.", "response": "def generate_protocol(self):\n \"\"\"\n Recreate the command stimulus (protocol) for the current sweep.\n It's not stored point by point (that's a waste of time and memory!)\n Instead it's stored as a few (x,y) points which can be easily graphed.\n\n TODO: THIS\n for segment in abf.ABFreader.read_protocol():\n for analogsignal in segment.analogsignals:\n print(analogsignal)\n plt.plot(analogsignal)\n plt.show()\n plt.close('all')\n\n \"\"\"\n # TODO: elegantly read the protocol like this:\n #abf.ABFreader.read_protocol()[0].analogsignals()[sigNum]\n\n\n\n # TODO: right now this works only for the first channel\n\n # correct for weird recording/protocol misalignment\n #what is magic here? 64-bit data points? #1,000,000/64 = 15625 btw\n self.offsetX = int(self.sweepSize/64)\n\n # if there's not a header, get out of here!\n if not len(self.header['dictEpochInfoPerDAC']):\n self.log.debug(\"no protocol defined, so I'll make one\")\n self.protoX,self.protoY=[0,self.sweepX[-1]],[self.holding,self.holding]\n self.protoSeqX,self.protoSeqY=[0],[self.holding]\n return\n\n # load our protocol from the header\n proto=self.header['dictEpochInfoPerDAC'][self.channel]\n\n # prepare our (x,y) pair arrays\n self.protoX,self.protoY=[] ,[]\n\n # assume our zero time point is the \"holding\" value\n self.protoX.append(0)\n self.protoY.append(self.holding) #TODO: what is this???\n\n # now add x,y points for each change in the protocol\n for step in proto:\n dX = proto[step]['lEpochInitDuration']\n Y = proto[step]['fEpochInitLevel']+proto[step]['fEpochLevelInc']*self.sweep\n # we have a new Y value, so add it to the last time point\n self.protoX.append(self.protoX[-1])\n self.protoY.append(Y)\n # now add the same Y point after \"dX\" amount of time\n self.protoX.append(self.protoX[-1]+dX)\n self.protoY.append(Y)\n # TODO: detect ramps and warn what's up\n\n # The last point is probably holding current\n finalVal=self.holding #regular holding\n # although if it's set to \"use last value\", maybe that should be the last one\n if self.header['listDACInfo'][0]['nInterEpisodeLevel']:\n finalVal=self.protoY[-1]\n\n # add the shift to the final value to the list\n self.protoX.append(self.protoX[-1])\n self.protoY.append(finalVal)\n # and again as the very last time point\n self.protoX.append(self.sweepSize)\n self.protoY.append(finalVal)\n\n # update the sequence of protocols now (eliminate duplicate entries)\n for i in range(1,len(self.protoX)-1): #correct for weird ABF offset issue.\n self.protoX[i]=self.protoX[i]+self.offsetX\n self.protoSeqY=[self.protoY[0]]\n self.protoSeqX=[self.protoX[0]]\n for i in range(1,len(self.protoY)):\n if not self.protoY[i]==self.protoY[i-1]:\n self.protoSeqY.append(self.protoY[i])\n self.protoSeqX.append(self.protoX[i])\n if self.protoY[0]!=self.protoY[1]:\n self.protoY.insert(1,self.protoY[0])\n self.protoX.insert(1,self.protoX[1])\n self.protoY.insert(1,self.protoY[0])\n self.protoX.insert(1,self.protoX[0]+self.offsetX/2)\n self.protoSeqY.append(finalVal)\n self.protoSeqX.append(self.sweepSize)\n\n # convert lists to numpy arrays and do any final conversions\n self.protoX=np.array(self.protoX)/self.pointsPerSec\n self.protoY=np.array(self.protoY)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the protocol of the current sweep", "response": "def get_protocol(self,sweep):\n \"\"\"\n given a sweep, return the protocol as [Xs,Ys].\n This is good for plotting/recreating the protocol trace.\n There may be duplicate numbers.\n \"\"\"\n self.setsweep(sweep)\n return list(self.protoX),list(self.protoY)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the protocol sequence for a given sweep", "response": "def get_protocol_sequence(self,sweep):\n \"\"\"\n given a sweep, return the protocol as condensed sequence.\n This is better for comparing similarities and determining steps.\n There should be no duplicate numbers.\n \"\"\"\n self.setsweep(sweep)\n return list(self.protoSeqX),list(self.protoSeqY)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget start and stop times of an epoch", "response": "def epochTimes(self,nEpoch=2):\n \"\"\"\n alternative to the existing abf protocol stuff\n return the start/stop time of an epoch.\n Epoch start at zero.\n A=0, B=1, C=2, D=3, ...\n \"\"\"\n times=[]\n durations=[]\n for epoch in self.header['dictEpochInfoPerDAC'][self.channel].values():\n print(epoch['lEpochInitDuration']/self.pointsPerSec)\n times.append(sum(durations))\n durations.append(epoch['lEpochInitDuration']/self.pointsPerSec)\n times.append(sum(durations))\n times=np.array(times)+self.offsetX/self.pointsPerSec # WHY?!?\n if nEpoch:\n return times[nEpoch],times[nEpoch+1]\n else:\n return times"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef average(self,t1=0,t2=None,setsweep=False):\n if setsweep:\n self.setsweep(setsweep)\n if t2 is None or t2>self.sweepLength:\n t2=self.sweepLength\n self.log.debug(\"resetting t2 to [%f]\",t2)\n t1=max(t1,0)\n if t1>t2:\n self.log.error(\"t1 cannot be larger than t2\")\n return False\n I1,I2=int(t1*self.pointsPerSec),int(t2*self.pointsPerSec)\n if I1==I2:\n return np.nan\n return np.average(self.sweepY[I1:I2])", "response": "return the average of part of the current sweep."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef averageSweep(self,sweepFirst=0,sweepLast=None):\n if sweepLast is None:\n sweepLast=self.sweeps-1\n nSweeps=sweepLast-sweepFirst+1\n runningSum=np.zeros(len(self.sweepY))\n self.log.debug(\"averaging sweep %d to %d\",sweepFirst,sweepLast)\n for sweep in np.arange(nSweeps)+sweepFirst:\n self.setsweep(sweep)\n runningSum+=self.sweepY.flatten()\n average=runningSum/nSweeps\n #TODO: standard deviation?\n return average", "response": "Return an average sweep of multiple sweeps."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates kernel based on this ABF info.", "response": "def kernel_gaussian(self, sizeMS, sigmaMS=None, forwardOnly=False):\n \"\"\"create kernel based on this ABF info.\"\"\"\n sigmaMS=sizeMS/10 if sigmaMS is None else sigmaMS\n size,sigma=sizeMS*self.pointsPerMs,sigmaMS*self.pointsPerMs\n self.kernel=swhlab.common.kernel_gaussian(size,sigma,forwardOnly)\n return self.kernel"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sweepYfiltered(self):\n assert self.kernel is not None\n return swhlab.common.convolve(self.sweepY,self.kernel)", "response": "Returns the filtered sweepY of the current sweep."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef phasicNet(self,biggestEvent=50,m1=.5,m2=None):\n\n # determine marks (between which we will analyze)\n m1=0 if m1 is None else self.pointsPerSec*m1\n m2=-1 if m2 is None else self.pointsPerSec*m2\n\n # acquire the baseline-subtracted sweep\n Y=self.sweepYsmartbase()[int(m1):int(m2)]\n\n # create the histogram\n nBins=1000\n hist,bins=np.histogram(Y,bins=nBins,range=[-biggestEvent,biggestEvent],density=True)\n histSmooth=swhlab.common.lowpass(hist)\n\n # normalize height to 1\n #TODO: should it be normalized first or not?\n #YES if reporting the ratio of the up/down area, NO if reporting the up-down difference\n #hist,histSmooth=hist/max(histSmooth),histSmooth/max(histSmooth)\n\n # center the peak at 0 pA\n peakI=np.where(histSmooth==max(histSmooth))[0][0]\n hist=np.roll(hist,int(nBins/2-peakI))\n histSmooth=np.roll(histSmooth,int(nBins/2-peakI))\n\n # calculate our mirrored difference\n downward,upward=np.split(histSmooth,2)\n downward=downward[::-1]\n diff=np.sum(upward-downward)\n\n # convert our \"pA/time\" to \"pA/sec\"\n diff=diff/(len(Y)/self.pointsPerSec)\n\n return diff", "response": "Calculates the phasic difference between positive and negative phasic events\n Returns the phasic difference value of the current sweep."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef output_touch(self):\n if not os.path.exists(self.outFolder):\n self.log.debug(\"creating %s\",self.outFolder)\n os.mkdir(self.outFolder)", "response": "ensure the. whlab folder exists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngive a list of dicts return just the dicts.", "response": "def dictFlat(l):\n \"\"\"Given a list of list of dicts, return just the dicts.\"\"\"\n if type(l) is dict:\n return [l]\n if \"numpy\" in str(type(l)):\n return l\n dicts=[]\n for item in l:\n if type(item)==dict:\n dicts.append(item)\n elif type(item)==list:\n for item2 in item:\n dicts.append(item2)\n return dicts"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef listCount(l):\n for i in range(len(l)):\n l[i]=len(l[i])\n return l", "response": "returns len ( each item in a list as a list"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all key from a list of dicts.", "response": "def dictVals(l,key):\n \"\"\"Return all 'key' from a list of dicts. (or list of list of dicts)\"\"\"\n dicts=dictFlat(l)\n vals=np.empty(len(dicts))*np.nan\n for i in range(len(dicts)):\n if key in dicts[i]:\n vals[i]=dicts[i][key]\n return vals"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dictAvg(listOfDicts,key,stdErr=False):\n vals=dictVals(listOfDicts,key)\n if len(vals) and np.any(vals):\n av=np.nanmean(vals)\n er=np.nanstd(vals)\n if stdErr:\n er=er/np.sqrt(np.count_nonzero(~np.isnan(er)))\n else:\n av,er=np.nan,np.nan\n return av,er", "response": "Given a list of dicts d return AV and SD."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of dicts with fake data.", "response": "def dummyListOfDicts(size=100):\n \"\"\"\n returns a list (of the given size) of dicts with fake data.\n some dictionary keys are missing for some of the items.\n \"\"\"\n titles=\"ahp,halfwidth,peak,expT,expI,sweep\".split(\",\")\n ld=[] #list of dicts\n for i in range(size):\n d={}\n for t in titles:\n if int(np.random.random(1)*100)>5: #5% of values are missing\n d[t]=float(np.random.random(1)*100) #random number 0-100\n if t==\"sweep\" and \"sweep\" in d.keys():\n d[t]=int(d[t])\n ld.append(d)\n return ld"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef matrixValues(matrix,key):\n assert key in matrix.dtype.names\n col=matrix.dtype.names.index(key)\n values=np.empty(len(matrix))*np.nan\n for i in range(len(matrix)):\n values[i]=matrix[i][col]\n return values", "response": "given a key return a list of values from the matrix with that key."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef matrixToDicts(data):\n\n # 1D array\n if \"float\" in str(type(data[0])):\n d={}\n for x in range(len(data)):\n d[data.dtype.names[x]]=data[x]\n return d\n\n # 2D array\n l=[]\n for y in range(len(data)):\n d={}\n for x in range(len(data[y])):\n d[data.dtype.names[x]]=data[y][x]\n l.append(d)\n return l", "response": "given a recarray return it as a list of dicts."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a list of dicts to a structured array.", "response": "def matrixfromDicts(dicts):\n \"\"\"\n Give a list of dicts (or list of list of dicts) return a structured array.\n Headings will be sorted in alphabetical order.\n \"\"\"\n if 'numpy' in str(type(dicts)):\n return dicts #already an array?\n names=set([])\n dicts=dictFlat(dicts)\n for item in dicts:\n names=names.union(list(item.keys()))\n names=sorted(list(names))\n data=np.empty((len(dicts),len(names)),dtype=float)*np.nan\n for y in range(len(dicts)):\n for key in dicts[y].keys():\n for x in range(len(names)):\n if names[x] in dicts[y]:\n data[y,x]=dicts[y][names[x]]\n if len(dicts):\n data=np.core.records.fromarrays(data.transpose(),names=names)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a list into a list of HTML tables.", "response": "def htmlListToTR(l,trClass=None,tdClass=None,td1Class=None):\n \"\"\"\n turns a list into a something\n call this when generating HTML tables dynamically.\n \"\"\"\n html=\"\"\n for item in l:\n if 'array' in str(type(item)):\n item=item[0] #TODO: why is this needed\n html+=\"%s\"%item\n html+=\"\"\n if trClass:\n html=html.replace(\"\",''%trClass)\n if td1Class:\n html=html.replace(\"\",''%td1Class,1)\n if tdClass:\n html=html.replace(\"\",''%tdClass)\n\n\n return html"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef html_temp_launch(html):\n fname = tempfile.gettempdir()+\"/swhlab/temp.html\"\n with open(fname,'w') as f:\n f.write(html)\n webbrowser.open(fname)", "response": "given text make it a temporary HTML file and launch it"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nshowing everything we can about an object s projects and methods.", "response": "def checkOut(thing,html=True):\n \"\"\"show everything we can about an object's projects and methods.\"\"\"\n msg=\"\"\n for name in sorted(dir(thing)):\n if not \"__\" in name:\n msg+=\"%s\\n\"%name\n try:\n msg+=\" ^-VALUE: %s\\n\"%getattr(thing,name)()\n except:\n pass\n if html:\n html=''+msg+''\n html=html.replace(\" \",\" \").replace(\"\\n\",\"
\")\n fname = tempfile.gettempdir()+\"/swhlab/checkout.html\"\n with open(fname,'w') as f:\n f.write(html)\n webbrowser.open(fname)\n print(msg.replace('','').replace('',''))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a 2d numpy array into an Origin worksheet.", "response": "def matrixToWks(data,names=None,units=None,bookName=None,sheetName=\" \",xCol=None):\n \"\"\"\n Put 2d numpy data into an Origin worksheet.\n If bookname and sheetname are given try to load data into that book/sheet.\n If the book/sheet doesn't exist, create it.\n \"\"\"\n if type(data) is list:\n data=matrixfromDicts(data)\n if not names:\n names=[\"\"]*len(data[0])\n if data.dtype.names:\n names=list(data.dtype.names)\n if not units:\n units=[\"\"]*len(data[0])\n for i in range(len(units)):\n if names[i] in UNITS.keys():\n units[i]=UNITS[names[i]]\n if 'recarray' in str(type(data)): #make it a regular array\n data=data.view(float).reshape(data.shape + (-1,))\n if xCol and xCol in names:\n xCol=names.index(xCol)\n names.insert(0,names[xCol])\n units.insert(0,units[xCol])\n data=np.insert(data,0,data[:,xCol],1)\n\n if not bookName:\n bookName=\"tempBook\"\n if not sheetName:\n sheetName=\"temp-\"+str(time.clock())[-5:]\n\n try:\n import PyOrigin\n PyOrigin.LT_execute(\"\") #try to crash a non-orign environment\n except:\n print(\" -- PyOrigin not running, making HTML output.\")\n matrixToHTML(data,names,units,bookName,sheetName,xCol)\n return\n\n nrows,ncols=len(data),len(data[0])\n if 'recarray' in str(type(data)): #convert structured array to regular array\n data=np.array(data.view(),dtype=float).reshape((nrows,ncols))\n data=np.transpose(data) #transpose it\n\n PyOrigin.LT_execute(\"activateSheet(%s, %s)\"%(bookName,sheetName))\n wks=PyOrigin.ActiveLayer()\n while wks.GetColCount() < ncols:\n wks.InsertCol(wks.GetColCount(),'')\n for i in range(ncols):\n col=wks.Columns(i)\n col.SetLongName(names[i])\n col.SetUnits(units[i])\n wks.SetData(data,0,0)\n PyOrigin.LT_execute(\"FixNanBug\")\n PyOrigin.LT_execute(\"ABFPathToMetaData\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nputting 2d numpy data into a temporary HTML file.", "response": "def matrixToHTML(data,names=None,units=None,bookName=None,sheetName=None,xCol=None):\n \"\"\"Put 2d numpy data into a temporary HTML file.\"\"\"\n if not names:\n names=[\"\"]*len(data[0])\n if data.dtype.names:\n names=list(data.dtype.names)\n if not units:\n units=[\"\"]*len(data[0])\n for i in range(len(units)):\n if names[i] in UNITS.keys():\n units[i]=UNITS[names[i]]\n if 'recarray' in str(type(data)): #make it a regular array\n data=data.view(float).reshape(data.shape + (-1,))\n if xCol and xCol in names:\n xCol=names.index(xCol)\n names.insert(0,names[xCol])\n units.insert(0,units[xCol])\n data=np.insert(data,0,data[:,xCol],1)\n\n htmlFname = tempfile.gettempdir()+\"/swhlab/WKS-%s.%s.html\"%(bookName,sheetName)\n html=\"\"\"\n \n \"\"\"\n html+=\"

FauxRigin

\"\n if bookName or sheetName:\n html+='%s / %s

'%(bookName,sheetName)\n html+=\"\"\n #cols=list(range(len(names)))\n colNames=['']\n for i in range(len(units)):\n label=\"%s (%d)\"%(chr(i+ord('A')),i)\n colNames.append(label)\n html+=htmlListToTR(colNames,'labelCol','labelCol')\n html+=htmlListToTR(['Long Name']+list(names),'name',td1Class='labelRow')\n html+=htmlListToTR(['Units']+list(units),'units',td1Class='labelRow')\n cutOff=False\n for y in range(len(data)):\n html+=htmlListToTR([y+1]+list(data[y]),trClass='data%d'%(y%2),td1Class='labelRow')\n if y>=200:\n cutOff=True\n break\n html+=\"
\"\n html=html.replace(\">nan<\",\">--<\")\n html=html.replace(\">None<\",\"><\")\n if cutOff:\n html+=\"

... showing only %d of %d rows ...

\"%(y,len(data))\n html+=\"\"\n with open(htmlFname,'w') as f:\n f.write(html)\n webbrowser.open(htmlFname)\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn an XML object", "response": "def XMLtoPython(xmlStr=r\"C:\\Apps\\pythonModules\\GSTemp.xml\"):\n \"\"\"\n given a string or a path to an XML file, return an XML object.\n \"\"\"\n #TODO: this absolute file path crazy stuff needs to stop!\n if os.path.exists(xmlStr):\n with open(xmlStr) as f:\n xmlStr=f.read()\n print(xmlStr)\n print(\"DONE\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fit_exp(y,graphToo=False):\n x=np.arange(len(y))\n try:\n params, cv = scipy.optimize.curve_fit(algo_exp, x, y, p0=(1,1e-6,1))\n except:\n print(\" !! curve fit failed (%.02f points)\"%len(x))\n return np.nan,np.nan,np.nan,np.nan #can't fit so little data!\n m,t,b=params\n tau=1/t\n if graphToo:\n pylab.figure(figsize=(6,4))\n pylab.grid()\n pylab.title(\"FIT ASSESSMENT\")\n pylab.plot(x,y,'o',mfc='none',ms=10)\n pylab.plot(x,algo_exp(x,m,t,b),'b-',lw=2)\n pylab.show()\n return m,t,b,tau", "response": "Exponential fit. Returns [ multiplier t offset time constant"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\naligns a numpy array of XY columns into a single array of XYYY columns.", "response": "def numpyAlignXY(data):\n \"\"\"\n given a numpy array (XYXYXY columns), return it aligned.\n data returned will be XYYY. NANs may be returned.\n \"\"\"\n print(data)\n Xs=data.flatten()[::2] # get all X values\n Xs=Xs[~np.isnan(Xs)] # remove nans\n Xs=sorted(list(set(Xs))) # eliminate duplicates then sort it\n aligned=np.empty((len(Xs),int(len(data[0])/2+1)))*np.nan\n aligned[:,0]=Xs\n for col in range(0,len(data[0]),2):\n for row in range(len(data)):\n X=data[row,col]\n Y=data[row,col+1]\n if np.isnan(X) or np.isnan(Y):\n continue\n aligned[Xs.index(X),int(col/2+1)]=Y\n return aligned"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter_gaussian(Ys,sigma,plotToo=False):\n timeA=time.time()\n window=scipy.signal.gaussian(len(Ys),sigma)\n window/=sum(window)\n Ys2=np.convolve(Ys,window,'same')\n print(\"LEN:\",len(Ys2),len(Ys))\n timeB=time.time()\n print(\"convolution took %.03f ms\"%((timeB-timeA)*1000))\n if len(Ys2)!=len(Ys):\n print(\"?!?!?!? convolution point size mismatch\")\n if plotToo:\n pylab.plot(Ys,label='original',alpha=.2)\n pylab.plot(Ys2,'b-',label='smooth')\n pylab.legend()\n pylab.show()\n return Ys2", "response": "simple gaussian convolution. Returns same number of points as gotten."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef where_cross(data,threshold):\n Is=np.where(data>threshold)[0]\n Is=np.concatenate(([0],Is))\n Ds=Is[:-1]-Is[1:]+1\n return Is[np.where(Ds)[0]+1]", "response": "return a list of Is where the data first crosses threshold."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of dicts as a 2d matrix.", "response": "def originFormat_listOfDicts(l):\n \"\"\"Return [{},{},{}] as a 2d matrix.\"\"\"\n titles=[]\n for d in l:\n for k in d.keys():\n if not k in titles:\n titles.append(k)\n titles.sort()\n data=np.empty((len(l),len(titles)))*np.nan\n for y in range(len(l)):\n for x in range(len(titles)):\n if titles[x] in l[y].keys():\n data[y][x]=l[y][titles[x]]\n return titles,data"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef originFormat(thing):\n if type(thing) is list and type(thing[0]) is dict:\n return originFormat_listOfDicts(thing)\n if type(thing) is list and type(thing[0]) is list:\n return originFormat_listOfDicts(dictFlat(thing))\n else:\n print(\" !! I don't know how to format this object!\")\n print(thing)", "response": "Try to format anything as a 2D matrix with column names."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pickle_save(thing,fname):\n pickle.dump(thing, open(fname,\"wb\"),pickle.HIGHEST_PROTOCOL)\n return thing", "response": "save something to a pickle file"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a dictionary to a pretty formatted string.", "response": "def msgDict(d,matching=None,sep1=\"=\",sep2=\"\\n\",sort=True,cantEndWith=None):\n \"\"\"convert a dictionary to a pretty formatted string.\"\"\"\n msg=\"\"\n if \"record\" in str(type(d)):\n keys=d.dtype.names\n else:\n keys=d.keys()\n if sort:\n keys=sorted(keys)\n for key in keys:\n if key[0]==\"_\":\n continue\n if matching:\n if not key in matching:\n continue\n if cantEndWith and key[-len(cantEndWith)]==cantEndWith:\n continue\n if 'float' in str(type(d[key])):\n s=\"%.02f\"%d[key]\n else:\n s=str(d[key])\n if \"object\" in s:\n s=''\n msg+=key+sep1+s+sep2\n return msg.strip()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a groups file return a dict of groups.", "response": "def groupsFromKey(keyFile='./key.txt'):\n \"\"\"\n given a groups file, return a dict of groups.\n Example:\n ### GROUP: TR\n 16602083\n 16608059\n ### GROUP: TU\n 16504000\n 16507011\n \"\"\"\n groups={}\n thisGroup=\"?\"\n with open(keyFile) as f:\n raw=f.read().split(\"\\n\")\n for line in raw:\n line=line.strip()\n if len(line)<3:\n continue\n if \"### GROUP\" in line:\n thisGroup=line.split(\": \")[1]\n groups[thisGroup]=[]\n else:\n groups[thisGroup]=groups[thisGroup]+[line]\n return groups"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an abf of the FIRST of every type of thing", "response": "def findRelevantData(fileList,abfs):\n \"\"\"return an abf of the *FIRST* of every type of thing.\"\"\"\n relevant=[]\n things={}\n for abf in abfs:\n for fname in fileList:\n if abf in fname and not fname in relevant:\n relevant.append(fname)\n for item in sorted(relevant):\n thing = os.path.basename(item)\n if \".png\" in thing:\n continue\n if not \"_\" in thing:\n continue\n thing=thing.split(\"_\")[-1].split(\".\")[0]\n if not thing in things.keys(): #prevent overwriting\n things[thing]=item\n return things"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndetermining the comment cooked in the protocol", "response": "def determineProtocol(fname):\n \"\"\"determine the comment cooked in the protocol.\"\"\"\n f=open(fname,'rb')\n raw=f.read(5000) #it should be in the first 5k of the file\n f.close()\n protoComment=\"unknown\"\n if b\"SWHLab4[\" in raw:\n protoComment=raw.split(b\"SWHLab4[\")[1].split(b\"]\",1)[0]\n elif b\"SWH[\" in raw:\n protoComment=raw.split(b\"SWH[\")[1].split(b\"]\",1)[0]\n else:\n protoComment=\"?\"\n if not type(protoComment) is str:\n protoComment=protoComment.decode(\"utf-8\")\n return protoComment"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting silly C:\\\\names\\\\like\\\\this. txt to c:/names", "response": "def forwardSlash(listOfFiles):\n \"\"\"convert silly C:\\\\names\\\\like\\\\this.txt to c:/names/like/this.txt\"\"\"\n for i,fname in enumerate(listOfFiles):\n listOfFiles[i]=fname.replace(\"\\\\\",\"/\")\n return listOfFiles"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef scanABFfolder(abfFolder):\n assert os.path.isdir(abfFolder)\n filesABF=forwardSlash(sorted(glob.glob(abfFolder+\"/*.*\")))\n filesSWH=[]\n if os.path.exists(abfFolder+\"/swhlab4/\"):\n filesSWH=forwardSlash(sorted(glob.glob(abfFolder+\"/swhlab4/*.*\")))\n groups=getABFgroups(filesABF)\n return filesABF,filesSWH,groups", "response": "Scan an ABF folder and return a list of files SWHLab files and groups."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getParent(abfFname):\n child=os.path.abspath(abfFname)\n files=sorted(glob.glob(os.path.dirname(child)+\"/*.*\"))\n parentID=abfFname #its own parent\n for fname in files:\n if fname.endswith(\".abf\") and fname.replace(\".abf\",\".TIF\") in files:\n parentID=os.path.basename(fname).replace(\".abf\",\"\")\n if os.path.basename(child) in fname:\n break\n return parentID", "response": "given an ABF file name return the ABF of its parent."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives an ABF and the groups dict return the ID of its parent.", "response": "def getParent2(abfFname,groups):\n \"\"\"given an ABF and the groups dict, return the ID of its parent.\"\"\"\n if \".abf\" in abfFname:\n abfFname=os.path.basename(abfFname).replace(\".abf\",\"\")\n for parentID in groups.keys():\n if abfFname in groups[parentID]:\n return parentID\n return abfFname"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the notes for an ABF", "response": "def getNotesForABF(abfFile):\n \"\"\"given an ABF, find the parent, return that line of experiments.txt\"\"\"\n parent=getParent(abfFile)\n parent=os.path.basename(parent).replace(\".abf\",\"\")\n expFile=os.path.dirname(abfFile)+\"/experiment.txt\"\n if not os.path.exists(expFile):\n return \"no experiment file\"\n with open(expFile) as f:\n raw=f.readlines()\n for line in raw:\n if line[0]=='~':\n line=line[1:].strip()\n if line.startswith(parent):\n while \"\\t\\t\" in line:\n line=line.replace(\"\\t\\t\",\"\\t\")\n line=line.replace(\"\\t\",\"\\n\")\n return line\n return \"experiment.txt found, but didn't contain %s\"%parent"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getABFgroups(files):\n children=[]\n groups={}\n for fname in sorted(files):\n if fname.endswith(\".abf\"):\n if fname.replace(\".abf\",\".TIF\") in files: #TODO: cap sensitive\n if len(children):\n groups[children[0]]=children\n children=[os.path.basename(fname)[:-4]]\n else:\n children.append(os.path.basename(fname)[:-4])\n groups[children[0]]=children\n #print(\" -- found %d groups of %d ABFs\"%(len(groups),len(files)))\n return groups", "response": "Given a list of files return a dict of ID - > children IDs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a list of files return a dict [ ID = > list of files.", "response": "def getIDfileDict(files):\n \"\"\"\n given a list of files, return a dict[ID]=[files].\n This is made to assign children files to parent ABF IDs.\n \"\"\"\n d={}\n orphans=[]\n for fname in files:\n if fname.endswith(\".abf\"):\n d[os.path.basename(fname)[:-4]]=[]\n for fname in files:\n if fname.endswith(\".html\") or fname.endswith(\".txt\"):\n continue #don't try to assign to an ABF\n if len(os.path.basename(fname).split(\".\")[0])>=8:\n ID = os.path.basename(fname)[:8] #ABF ID (first several chars)\n else:\n ID = os.path.basename(fname).split(\".\")[0] #short filename, just not extension\n if ID in d.keys():\n d[ID]=d[ID]+[fname]\n else:\n orphans.append(os.path.basename(fname))\n #print(\" ?? orphan file\",ID,os.path.basename(fname))\n if orphans:\n print(\" ?? found %d orphan files\"%len(orphans))\n return d"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a path or list of files return ABF IDs.", "response": "def getIDsFromFiles(files):\n \"\"\"given a path or list of files, return ABF IDs.\"\"\"\n if type(files) is str:\n files=glob.glob(files+\"/*.*\")\n IDs=[]\n for fname in files:\n if fname[-4:].lower()=='.abf':\n ext=fname.split('.')[-1]\n IDs.append(os.path.basename(fname).replace('.'+ext,''))\n return sorted(IDs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninspects an ABF object or filename and returns a tuple of tuples.", "response": "def inspectABF(abf=exampleABF,saveToo=False,justPlot=False):\n \"\"\"May be given an ABF object or filename.\"\"\"\n pylab.close('all')\n print(\" ~~ inspectABF()\")\n if type(abf) is str:\n abf=swhlab.ABF(abf)\n swhlab.plot.new(abf,forceNewFigure=True)\n if abf.sweepInterval*abf.sweeps<60*5: #shorter than 5 minutes\n pylab.subplot(211)\n pylab.title(\"%s [%s]\"%(abf.ID,abf.protoComment))\n swhlab.plot.sweep(abf,'all')\n pylab.subplot(212)\n swhlab.plot.sweep(abf,'all',continuous=True)\n swhlab.plot.comments(abf)\n else:\n print(\" -- plotting as long recording\")\n swhlab.plot.sweep(abf,'all',continuous=True,minutes=True)\n swhlab.plot.comments(abf,minutes=True)\n pylab.title(\"%s [%s]\"%(abf.ID,abf.protoComment))\n swhlab.plot.annotate(abf)\n if justPlot:\n return\n if saveToo:\n path=os.path.split(abf.fname)[0]\n basename=os.path.basename(abf.fname)\n pylab.savefig(os.path.join(path,\"_\"+basename.replace(\".abf\",\".png\")))\n pylab.show()\n return"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ftp_login(folder=None):\n pwDir=os.path.realpath(__file__)\n for i in range(3):\n pwDir=os.path.dirname(pwDir)\n pwFile = os.path.join(pwDir,\"passwd.txt\")\n print(\" -- looking for login information in:\\n [%s]\"%pwFile)\n try:\n with open(pwFile) as f:\n lines=f.readlines()\n username=lines[0].strip()\n password=lines[1].strip()\n print(\" -- found a valid username/password\")\n except:\n print(\" -- password lookup FAILED.\")\n username=TK_askPassword(\"FTP LOGIN\",\"enter FTP username\")\n password=TK_askPassword(\"FTP LOGIN\",\"enter password for %s\"%username)\n if not username or not password:\n print(\" !! failed getting login info. aborting FTP effort.\")\n return\n print(\" username:\",username)\n print(\" password:\",\"*\"*(len(password)))\n print(\" -- logging in to FTP ...\")\n try:\n ftp = ftplib.FTP(\"swharden.com\")\n ftp.login(username, password)\n if folder:\n ftp.cwd(folder)\n return ftp\n except:\n print(\" !! login failure !!\")\n return False", "response": "return an \"FTP\" object after logging in."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nuploading all files from localFolder into the current FTP folder.", "response": "def ftp_folder_match(ftp,localFolder,deleteStuff=True):\n \"\"\"upload everything from localFolder into the current FTP folder.\"\"\"\n for fname in glob.glob(localFolder+\"/*.*\"):\n ftp_upload(ftp,fname)\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef version_upload(fname,username=\"nibjb\"):\n print(\"popping up pasword window...\")\n password=TK_askPassword(\"FTP LOGIN\",\"enter password for %s\"%username)\n if not password:\n return\n print(\"username:\",username)\n print(\"password:\",\"*\"*(len(password)))\n print(\"connecting...\")\n ftp = ftplib.FTP(\"swharden.com\")\n ftp.login(username, password)\n print(\"successful login!\")\n ftp.cwd(\"/software/swhlab/versions\") #IMMEDIATELY GO HERE!!!\n print(\"uploading\",os.path.basename(fname))\n ftp.storbinary(\"STOR \" + os.path.basename(fname), open(fname, \"rb\"), 1024) #for binary files\n print(\"disconnecting...\")\n ftp.quit()", "response": "Only scott should do this. Upload new version to site."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef TK_askPassword(title=\"input\",msg=\"type here:\"):\n root = tkinter.Tk()\n root.withdraw() #hide tk window\n root.attributes(\"-topmost\", True) #always on top\n root.lift() #bring to top\n value=tkinter.simpledialog.askstring(title,msg)\n root.destroy()\n return value", "response": "use the GUI to ask for a string."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef TK_message(title,msg):\n root = tkinter.Tk()\n root.withdraw() #hide tk window\n root.attributes(\"-topmost\", True) #always on top\n root.lift() #bring to top\n tkinter.messagebox.showwarning(title, msg)\n root.destroy()", "response": "use the GUI to pop up a message."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nuse the GUI to ask YES or NO.", "response": "def TK_ask(title,msg):\n \"\"\"use the GUI to ask YES or NO.\"\"\"\n root = tkinter.Tk()\n root.attributes(\"-topmost\", True) #always on top\n root.withdraw() #hide tk window\n result=tkinter.messagebox.askyesno(title,msg)\n root.destroy()\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert weird TIF files into web - friendly versions.", "response": "def image_convert(fname,saveAs=True,showToo=False):\n \"\"\"\n Convert weird TIF files into web-friendly versions.\n Auto contrast is applied (saturating lower and upper 0.1%).\n make saveAs True to save as .TIF.png\n make saveAs False and it won't save at all\n make saveAs \"someFile.jpg\" to save it as a different path/format\n \"\"\"\n\n # load the image\n #im = Image.open(fname) #PIL can't handle 12-bit TIFs well\n im=scipy.ndimage.imread(fname) #scipy does better with it\n im=np.array(im,dtype=float) # now it's a numpy array\n\n # do all image enhancement here\n cutoffLow=np.percentile(im,.01)\n cutoffHigh=np.percentile(im,99.99)\n im[np.where(imcutoffHigh)]=cutoffHigh\n\n # IMAGE FORMATTING\n im-=np.min(im) #auto contrast\n im/=np.max(im) #normalize\n im*=255 #stretch contrast (8-bit)\n im = Image.fromarray(im)\n\n # IMAGE DRAWING\n msg=\"Filename: %s\\n\"%os.path.basename(fname)\n timestamp = datetime.datetime.fromtimestamp(os.path.getctime(fname))\n msg+=\"Created: %s\\n\"%timestamp.strftime('%Y-%m-%d %H:%M:%S')\n d = ImageDraw.Draw(im)\n fnt = ImageFont.truetype(\"arial.ttf\", 20)\n d.text((6,6),msg,font=fnt,fill=0)\n d.text((4,4),msg,font=fnt,fill=255)\n\n if showToo:\n im.show()\n if saveAs is False:\n return\n if saveAs is True:\n saveAs=fname+\".png\"\n im.convert('RGB').save(saveAs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef processArgs():\n if len(sys.argv)<2:\n print(\"\\n\\nERROR:\")\n print(\"this script requires arguments!\")\n print('try \"python command.py info\"')\n return\n if sys.argv[1]=='info':\n print(\"import paths:\\n \",\"\\n \".join(sys.path))\n print()\n print(\"python version:\",sys.version)\n print(\"SWHLab path:\",__file__)\n print(\"SWHLab version:\",swhlab.__version__)\n return\n if sys.argv[1]=='glanceFolder':\n abfFolder=swhlab.common.gui_getFolder()\n if not abfFolder or not os.path.isdir(abfFolder):\n print(\"bad path\")\n return\n fnames=sorted(glob.glob(abfFolder+\"/*.abf\"))\n outFolder=tempfile.gettempdir()+\"/swhlab/\"\n if os.path.exists(outFolder):\n shutil.rmtree(outFolder)\n os.mkdir(outFolder)\n outFile=outFolder+\"/index.html\"\n out=''\n out+='

%s

'%abfFolder\n for i,fname in enumerate(fnames):\n print(\"\\n\\n### PROCESSING %d of %d\"%(i,len(fnames)))\n saveAs=os.path.join(os.path.dirname(outFolder),os.path.basename(fname))+\".png\"\n out+='


%s
'%os.path.abspath(fname)\n out+='
'%(saveAs,saveAs)\n swhlab.analysis.glance.processAbf(fname,saveAs)\n out+=''\n with open(outFile,'w') as f:\n f.write(out)\n webbrowser.open_new_tab(outFile)\n return\n\n\n print(\"\\n\\nERROR:\\nI'm not sure how to process these arguments!\")\n print(sys.argv)", "response": "check out the arguments and figure out what to do."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndetect the APs and SAPs in the given sweeps.", "response": "def detect(abf,sweep=None,threshold_upslope=50,dT=.1,saveToo=True):\n \"\"\"\n An AP will be detected by a upslope that exceeds 50V/s. Analyzed too.\n if type(sweep) is int, graph int(sweep)\n if sweep==None, process all sweeps sweep.\n \"\"\"\n if type(sweep) is int:\n sweeps=[sweep]\n else:\n sweeps=list(range(abf.sweeps))\n timeStart=time.clock()\n abf.APs=[None]*abf.sweeps\n abf.SAP=[None]*abf.sweeps\n for sweep in sweeps:\n abf.setSweep(sweep)\n Y=abf.dataY\n dI = int(dT/1000*abf.rate) #dI is dT/rate\n dY = (Y[dI:]-Y[:-dI])*(abf.rate/1000/dI) #now in V/S\n Is = cm.where_cross(dY,threshold_upslope) #found every putative AP (I units)\n abf.APs[sweep]=[]\n for i in range(len(Is)): #for each putative AP\n try:\n AP=analyzeAP(Y,dY,Is[i],abf.rate) #try to characterize it\n if AP:\n AP[\"sweep\"]=sweep\n AP[\"expI\"]=sweep*abf.sweepInterval*abf.rate*+AP[\"sweepI\"]\n AP[\"expT\"]=sweep*abf.sweepInterval+AP[\"sweepT\"]\n AP[\"freq\"]=np.nan #default\n if len(abf.APs[sweep]):\n AP[\"freq\"]=1/(AP[\"expT\"]-abf.APs[sweep][-1][\"expT\"])\n if AP[\"freq\"] is np.nan or AP[\"freq\"]<500: #at 500Hz, assume you have a duplicate AP\n abf.APs[sweep].append(AP)\n except:\n print(\" -- AP %d of %d excluded from analysis...\"%(i+1,len(Is)))\n #print(\"!!! AP CRASH !!!\")\n #print(traceback.format_exc())\n analyzeAPgroup(abf) #now that APs are known, get grouping stats\n abf.APs=cm.matrixfromDicts(abf.APs)\n abf.SAP=cm.matrixfromDicts(abf.SAP)\n print(\" -- analyzed %d APs in %.02f ms\"%(len(cm.dictFlat(abf.APs)),(time.clock()-timeStart)*1000))\n if saveToo:\n abf.saveThing(abf.APs,\"APs\")\n abf.saveThing(abf.SAP,\"SAP\")"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nanalyzes the current sweep and calculate things like accomodation.", "response": "def analyzeAPgroup(abf=exampleABF,T1=None,T2=None,plotToo=False):\n \"\"\"\n On the current (setSweep()) sweep, calculate things like accomodation.\n Only call directly just for demonstrating how it works by making a graph.\n Or call this if you want really custom T1 and T2 (multiple per sweep)\n This is called by default with default T1 and T2.\n Manually call it again for custom.\n \"\"\"\n if T1 is None or T2 is None:\n if len(abf.protoSeqX)>2:\n T1=abf.protoSeqX[1]/abf.rate\n T2=abf.protoSeqX[2]/abf.rate\n else:\n T1=0\n T2=abf.sweepLength\n s={} #sweep dictionary to contain our stas\n s[\"sweep\"]=abf.currentSweep\n s[\"commandI\"]=abf.protoSeqY[1]\n\n APs=[]\n\n for key in ['freqAvg','freqBin']:\n s[key]=0\n\n for AP in abf.APs[abf.currentSweep]:\n if T11: #some measurements require multiple APs, like accomodation\n s[\"centerBinTime\"]=np.average(apTimes)-T1 #average time of APs in the bin\n s[\"centerBinFrac\"]=s[\"centerBinTime\"]/(T2-T1)*100 #fractional average of APs in bin (steady = .5)\n s[\"centerTime\"]=np.average(apTimes)-APs[0][\"sweepT\"] #time of average AP WRT first AP (not bin)\n s[\"centerFrac\"]=s[\"centerTime\"]/(APs[-1][\"sweepT\"]-APs[0][\"sweepT\"])*100 #WRT first/last AP\n s[\"msToFirst\"]=(APs[0][\"sweepT\"]-T1)*1000 #ms to first AP (from T1)\n s[\"freqFirst1\"]=APs[1]['freq'] #inst frequency of first AP\n s[\"freqFirst5\"]=cm.dictAvg(APs[1:6],'freq')[0] #inst frequency of first AP\n s[\"freqLast\"]=APs[-1]['freq'] #inst frequency of last AP\n s[\"freqAvg\"]=cm.dictAvg(APs,'freq')[0] #average inst frequency of all aps\n s[\"freqBin\"]=len(APs)/(T2-T1) #frequency of APs in the bin (T1-T2)\n s[\"freqSteady25\"]=cm.dictAvg(APs[-int(len(APs)*.25):],'freq')[0] # average freq of the last 25% of APs\n s[\"accom1Avg\"]=s[\"freqFirst1\"]/s[\"freqAvg\"] #accomodation (first II / average)\n s[\"accom1Steady25\"]=s[\"freqFirst1\"]/s[\"freqSteady25\"] #accomodation (first II / steady state)\n s[\"accom5Avg\"]=s[\"freqFirst5\"]/s[\"freqAvg\"] #accomodation from average 5 first\n s[\"accom5Steady25\"]=s[\"freqFirst5\"]/s[\"freqSteady25\"] #accomodation from average 5 first\n s[\"freqCV\"]=cm.dictAvg(APs,'freq')[1]/cm.dictAvg(APs,'freq')[0] #coefficient of variation (Hz)\n s[\"T1\"]=T1\n s[\"T2\"]=T2\n abf.SAP[abf.currentSweep]=s"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if the A - Group is populated.", "response": "def check_AP_group(abf=exampleABF,sweep=0):\n \"\"\"\n after running detect() and abf.SAP is populated, this checks it.\n \"\"\"\n abf.setSweep(sweep)\n swhlab.plot.new(abf,title=\"sweep %d (%d pA)\"%(abf.currentSweep,abf.protoSeqY[1]))\n swhlab.plot.sweep(abf)\n SAP=cm.matrixToDicts(abf.SAP[sweep])\n if \"T1\" in SAP.keys():\n T1=SAP[\"T1\"]\n T2=SAP[\"T2\"]\n pylab.axvspan(T1/abf.rate,T2/abf.rate,color='r',alpha=.1)\n else:\n T1=0\n T2=abf.sweepLength\n swhlab.plot.annotate(abf)\n pylab.tight_layout()\n pylab.subplots_adjust(right=0.6)\n pylab.annotate(cm.msgDict(SAP),(.71,.95),ha='left',va='top',\n weight='bold',family='monospace',\n xycoords='figure fraction',size=12,color='g')\n pylab.axis([T1-.05,T2+.05,None,None])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a sweep and a time point, return the AP array for that AP. APs will be centered in time by their maximum upslope.", "response": "def analyzeAP(Y,dY,I,rate,verbose=False):\n \"\"\"\n given a sweep and a time point, return the AP array for that AP.\n APs will be centered in time by their maximum upslope.\n \"\"\"\n Ims = int(rate/1000) #Is per MS\n IsToLook=5*Ims #TODO: clarify this, ms until downslope is over\n upslope=np.max(dY[I:I+IsToLook]) #maximum rise velocity\n upslopeI=np.where(dY[I:I+IsToLook]==upslope)[0][0]+I\n I=upslopeI #center sweep at the upslope\n downslope=np.min(dY[I:I+IsToLook]) #maximum fall velocity\n downslopeI=np.where(dY[I:I+IsToLook]==downslope)[0][0]+I\n peak=np.max(Y[I:I+IsToLook]) #find peak value (mV)\n peakI=np.where(Y[I:I+IsToLook]==peak)[0][0]+I #find peak I\n thresholdI=I-np.where(dY[I:I+IsToLook:--1]<10)[0] #detect <10V/S\n if not len(thresholdI):\n return False\n thresholdI=thresholdI[0]\n threshold=Y[thresholdI] # mV where >10mV/S\n height=peak-threshold # height (mV) from threshold to peak\n halfwidthPoint=np.average((threshold,peak))\n halfwidth=np.where(Y[I-IsToLook:I+IsToLook]>halfwidthPoint)[0]\n if not len(halfwidth):\n return False #doesn't look like a real AP\n halfwidthI1=halfwidth[0]+I-IsToLook\n halfwidthI2=halfwidth[-1]+I-IsToLook\n if Y[halfwidthI1-1]>halfwidthPoint or Y[halfwidthI2+1]>halfwidthPoint:\n return False #doesn't look like a real AP\n halfwidth=len(halfwidth)/rate*1000 #now in MS\n riseTime=(peakI-thresholdI)*1000/rate # time (ms) from threshold to peak\n\n IsToLook=100*Ims #TODO: max prediction until AHP reaches nadir\n AHPchunk=np.diff(Y[downslopeI:downslopeI+IsToLook]) #first inflection\n AHPI=np.where(AHPchunk>0)[0]\n if len(AHPI)==0:\n AHPI=np.nan\n else:\n AHPI=AHPI[0]+downslopeI\n AHPchunk=Y[AHPI:AHPI+IsToLook]\n if max(AHPchunk)>threshold: #if another AP is coming, cut it out\n AHPchunk=AHPchunk[:np.where(AHPchunk>threshold)[0][0]]\n if len(AHPchunk):\n AHP=np.nanmin(AHPchunk)\n AHPI=np.where(AHPchunk==AHP)[0][0]+AHPI\n AHPheight=threshold-AHP # AHP magnitude from threshold (mV)\n IsToLook=500*Ims #TODO: max prediction until AHP reaches threshold\n AHPreturn=np.average((AHP,threshold)) #half of threshold\n AHPreturnI=np.where(Y[AHPI:AHPI+IsToLook]>AHPreturn)[0]\n if len(AHPreturnI): #not having a clean decay won't cause AP to crash\n AHPreturnI=AHPreturnI[0]+AHPI\n AHPrisetime=(AHPreturnI-AHPI)*2/rate*1000 #predicted return time (ms)\n AHPupslope=AHPheight/AHPrisetime #mV/ms = V/S\n AHPreturnFullI=(AHPreturnI-AHPI)*2+AHPI\n else: #make them nan so you can do averages later\n AHPreturnI,AHPrisetime,AHPupslope=np.nan,np.nan,np.nan\n downslope=np.nan\n\n #fasttime (10V/S to 10V/S) #TODO:\n #dpp (deriv peak to peak) #TODO:\n\n sweepI,sweepT=I,I/rate # clean up variable names\n del IsToLook,I, Y, dY, Ims, AHPchunk, verbose #delete what we don't need\n return locals()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nplot for an eyeball check of AP detection in the given sweep.", "response": "def check_sweep(abf,sweep=None,dT=.1):\n \"\"\"Plotting for an eyeball check of AP detection in the given sweep.\"\"\"\n if abf.APs is None:\n APs=[]\n else:\n APs=cm.matrixToDicts(abf.APs)\n\n if sweep is None or len(sweep)==0: #find the first sweep with >5APs in it\n for sweepNum in range(abf.sweeps):\n foundInThisSweep=0\n for AP in APs:\n if AP[\"sweep\"]==sweepNum:\n foundInThisSweep+=1\n if foundInThisSweep>=5:\n break\n sweep=sweepNum\n abf.setSweep(sweep)\n Y=abf.dataY\n\n dI = int(dT/1000*abf.rate) #dI is dT/rate\n dY = (Y[dI:]-Y[:-dI])*(abf.rate/1000/dI) #now in V/S\n\n pylab.figure(figsize=(12,6))\n ax=pylab.subplot(211)\n pylab.title(\"sweep %d\"%abf.currentSweep)\n pylab.ylabel(\"membrane potential (mV)\")\n pylab.plot(Y,'-',alpha=.8)\n for AP in APs:\n if not AP[\"sweep\"]==sweep:\n continue\n pylab.axvline(AP[\"sweepI\"],alpha=.2,color='r')\n pylab.plot(AP[\"peakI\"],AP[\"peak\"],'.',alpha=.5,ms=20,color='r')\n pylab.plot(AP[\"thresholdI\"],AP[\"threshold\"],'.',alpha=.5,ms=20,color='c')\n pylab.plot([AP[\"AHPI\"],AP[\"AHPreturnI\"]],\n [AP[\"AHP\"],AP[\"AHPreturn\"]],\n '-',alpha=.2,ms=20,color='b',lw=7)\n pylab.plot([AP[\"halfwidthI1\"],AP[\"halfwidthI2\"]],\n [AP[\"halfwidthPoint\"],AP[\"halfwidthPoint\"]],\n '-',lw=5,alpha=.5,color='g')\n\n pylab.subplot(212,sharex=ax)\n pylab.ylabel(\"velocity (V/S)\")\n pylab.xlabel(\"data points (%.02f kHz)\"%(abf.rate/1000))\n pylab.plot(dY,'-',alpha=.8)\n pylab.margins(0,.1)\n for AP in APs:\n if not AP[\"sweep\"]==sweep:\n continue\n pylab.axvline(AP[\"sweepI\"],alpha=.2,color='r')\n pylab.plot(AP[\"upslopeI\"],AP[\"upslope\"],'.',alpha=.5,ms=20,color='g')\n pylab.plot(AP[\"downslopeI\"],AP[\"downslope\"],'.',alpha=.5,ms=20,color='g')\n pylab.axis([APs[0][\"sweepI\"]-1000,APs[-1][\"sweepI\"]+1000,None,None])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn list of time points of all AP events in experiment", "response": "def get_AP_timepoints(abf):\n \"\"\"return list of time points (sec) of all AP events in experiment.\"\"\"\n col=abf.APs.dtype.names.index(\"expT\")\n timePoints=[]\n for i in range(len(abf.APs)):\n timePoints.append(abf.APs[i][col])\n return timePoints"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if APSD data is in the correct format.", "response": "def check_AP_raw(abf,n=10):\n \"\"\"X\"\"\"\n timePoints=get_AP_timepoints(abf)[:n] #first 10\n if len(timePoints)==0:\n return\n swhlab.plot.new(abf,True,title=\"AP shape (n=%d)\"%n,xlabel=\"ms\")\n Ys=abf.get_data_around(timePoints,padding=.2)\n Xs=(np.arange(len(Ys[0]))-len(Ys[0])/2)*1000/abf.rate\n for i in range(1,len(Ys)):\n pylab.plot(Xs,Ys[i],alpha=.2,color='b')\n pylab.plot(Xs,Ys[0],alpha=.4,color='r',lw=2)\n pylab.margins(0,.1)\n msg=cm.msgDict(cm.dictFlat(abf.APs)[0],cantEndWith=\"I\")\n pylab.subplots_adjust(right=0.7)\n pylab.annotate(msg,(.71,.95),ha='left',va='top',\n xycoords='figure fraction',family='monospace',size=10)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking the derivative of the AP.", "response": "def check_AP_deriv(abf,n=10):\n \"\"\"X\"\"\"\n timePoints=get_AP_timepoints(abf)[:10] #first 10\n if len(timePoints)==0:\n return\n swhlab.plot.new(abf,True,title=\"AP velocity (n=%d)\"%n,xlabel=\"ms\",ylabel=\"V/S\")\n pylab.axhline(-50,color='r',lw=2,ls=\"--\",alpha=.2)\n pylab.axhline(-100,color='r',lw=2,ls=\"--\",alpha=.2)\n Ys=abf.get_data_around(timePoints,msDeriv=.1,padding=.005)\n Xs=(np.arange(len(Ys[0]))-len(Ys[0])/2)*1000/abf.rate\n for i in range(1,len(Ys)):\n pylab.plot(Xs,Ys[i],alpha=.2,color='b')\n pylab.plot(Xs,Ys[0],alpha=.4,color='r',lw=2)\n pylab.margins(0,.1)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_AP_phase(abf,n=10):\n timePoints=get_AP_timepoints(abf)[:10] #first 10\n if len(timePoints)==0:\n return\n swhlab.plot.new(abf,True,title=\"AP phase (n=%d)\"%n,xlabel=\"mV\",ylabel=\"V/S\")\n Ys=abf.get_data_around(timePoints,msDeriv=.1,padding=.005)\n Xs=abf.get_data_around(timePoints,padding=.005)\n for i in range(1,len(Ys)):\n pylab.plot(Xs[i],Ys[i],alpha=.2,color='b')\n pylab.plot(Xs[0],Ys[0],alpha=.4,color='r',lw=1)\n pylab.margins(.1,.1)", "response": "Check the APC phase of the APC"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nproviding all stats on the first AP.", "response": "def stats_first(abf):\n \"\"\"provide all stats on the first AP.\"\"\"\n msg=\"\"\n for sweep in range(abf.sweeps):\n for AP in abf.APs[sweep]:\n for key in sorted(AP.keys()):\n if key[-1] is \"I\" or key[-2:] in [\"I1\",\"I2\"]:\n continue\n msg+=\"%s = %s\\n\"%(key,AP[key])\n return msg"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning Xs Ys and sweeps for every AP found", "response": "def get_values(abf,key=\"freq\",continuous=False):\n \"\"\"returns Xs, Ys (the key), and sweep #s for every AP found.\"\"\"\n Xs,Ys,Ss=[],[],[]\n for sweep in range(abf.sweeps):\n for AP in cm.matrixToDicts(abf.APs):\n if not AP[\"sweep\"]==sweep:\n continue\n Ys.append(AP[key])\n Ss.append(AP[\"sweep\"])\n if continuous:\n Xs.append(AP[\"expT\"])\n else:\n Xs.append(AP[\"sweepT\"])\n\n return np.array(Xs),np.array(Ys),np.array(Ss)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getAvgBySweep(abf,feature,T0=None,T1=None):\n if T1 is None:\n T1=abf.sweepLength\n if T0 is None:\n T0=0\n data = [np.empty((0))]*abf.sweeps\n for AP in cm.dictFlat(cm.matrixToDicts(abf.APs)):\n if T01 and np.any(data[sweep]):\n data[sweep]=np.nanmean(data[sweep])\n elif len(data[sweep])==1:\n data[sweep]=data[sweep][0]\n else:\n data[sweep]=np.nan\n return data", "response": "return average of a feature divided by sweep."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef readLog(fname=\"workdays.csv\",onlyAfter=datetime.datetime(year=2017,month=1,day=1)):\r\n with open(fname) as f:\r\n raw=f.read().split(\"\\n\")\r\n efforts=[] #date,nickname\r\n for line in raw[1:]:\r\n line=line.strip().split(\",\")\r\n date=datetime.datetime.strptime(line[0], \"%Y-%m-%d\")\r\n if onlyAfter and date0:\n pylab.grid(None)\n\n # figure which sweeps to plot\n if sweep is None:\n sweeps=[ABF.currentSweep]\n if not ABF.currentSweep:\n sweeps=[0]\n elif sweep==\"all\":\n sweeps=range(0,ABF.sweeps)\n elif type(sweep) in [int,float]:\n sweeps=[int(sweep)]\n elif type(sweep) is list:\n sweeps=sweep\n else:\n print(\"DONT KNOW WHAT TO DO WITH THIS SWEEPS!!!\\n\",type(sweep),sweep)\n\n #figure out offsets:\n if continuous:\n offsetX=ABF.sweepInterval\n\n # determine the colors to use\n colors=[color]*len(sweeps) #detault to blue\n if rainbow and len(sweeps)>1:\n for i in range(len(sweeps)):\n colors[i]=ABF.colormap[i]\n if alpha is None and len(sweeps)==1:\n alpha=1\n if rainbow and alpha is None:\n alpha=.5\n\n # correct for alpha\n if alpha is None:\n alpha=1\n\n # conversion to minutes?\n if minutes == False:\n minutes=1\n else:\n minutes=60\n pylab.xlabel(\"minutes\")\n\n ABF.decimateMethod=decimate\n # do the plotting of each sweep\n for i in range(len(sweeps)):\n ABF.setSweep(sweeps[i])\n if protocol:\n pylab.plot((np.array(ABF.protoX)/ABF.rate+offsetX*i)/minutes,\n ABF.protoY+offsetY*i,\n alpha=alpha,color=colors[i])\n else:\n pylab.plot((ABF.dataX+offsetX*i)/minutes,\n ABF.dataY+offsetY*i,alpha=alpha,color=colors[i])\n ABF.decimateMethod=None\n pylab.margins(0,.02)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef annotate(abf):\n msg=\"SWHLab %s \"%str(swhlab.VERSION)\n msg+=\"ID:%s \"%abf.ID\n msg+=\"CH:%d \"%abf.channel\n msg+=\"PROTOCOL:%s \"%abf.protoComment\n msg+=\"COMMAND: %d%s \"%(abf.holding,abf.units)\n msg+=\"GENERATED:%s \"%'{0:%Y-%m-%d %H:%M:%S}'.format(datetime.datetime.now())\n pylab.annotate(msg,(.001,.001),xycoords='figure fraction',ha='left',\n va='bottom',color='#999999',family='monospace',size=8,\n weight='bold')\n if abf.nADC>1:\n msg=\"Ch %d/%d\"%(abf.channel+1,abf.nADC)\n pylab.annotate(msg,(.01,.99),xycoords='figure fraction',ha='left',\n va='top',color='#FF0000',family='monospace',size=12,\n weight='bold')", "response": "stamp the bottom with file info."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef new(ABF,forceNewFigure=False,title=None,xlabel=None,ylabel=None):\n if len(pylab.get_fignums()) and forceNewFigure==False:\n #print(\"adding to existing figure\")\n return\n pylab.figure(figsize=(8,6))\n pylab.grid(alpha=.5)\n pylab.title(ABF.ID)\n pylab.ylabel(ABF.units)\n pylab.xlabel(\"seconds\")\n if xlabel:\n pylab.xlabel(xlabel)\n if ylabel:\n pylab.ylabel(ylabel)\n if title:\n pylab.title(title)\n annotate(ABF)", "response": "make a new matplotlib figure with default dims and DPI."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef save(abf,fname=None,tag=None,width=700,close=True,facecolor='w',\n resize=True):\n \"\"\"\n Save the pylab figure somewhere.\n If fname==False, show it instead.\n Height force > dpi force\n if a tag is given instead of a filename, save it alongside the ABF\n \"\"\"\n if len(pylab.gca().get_lines())==0:\n print(\"can't save, no figure!\")\n return\n if resize:\n pylab.tight_layout()\n pylab.subplots_adjust(bottom=.1)\n annotate(abf)\n if tag:\n fname = abf.outpath+abf.ID+\"_\"+tag+\".png\"\n inchesX,inchesY = pylab.gcf().get_size_inches()\n dpi=width/inchesX\n if fname:\n if not os.path.exists(abf.outpath):\n os.mkdir(abf.outpath)\n print(\" <- saving [%s] at %d DPI (%dx%d)\"%(os.path.basename(fname),dpi,inchesX*dpi,inchesY*dpi))\n pylab.savefig(fname,dpi=dpi,facecolor=facecolor)\n else:\n pylab.show()\n if close:\n pylab.close()", "response": "Save the current ABF to a file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntries to load a custom swhlab module from a local path.", "response": "def tryLoadingFrom(tryPath,moduleName='swhlab'):\n \"\"\"if the module is in this path, load it from the local folder.\"\"\"\n if not 'site-packages' in swhlab.__file__:\n print(\"loaded custom swhlab module from\",\n os.path.dirname(swhlab.__file__))\n return # no need to warn if it's already outside.\n while len(tryPath)>5:\n sp=tryPath+\"/swhlab/\" # imaginary swhlab module path\n if os.path.isdir(sp) and os.path.exists(sp+\"/__init__.py\"):\n if not os.path.dirname(tryPath) in sys.path:\n sys.path.insert(0,os.path.dirname(tryPath))\n print(\"#\"*80)\n print(\"# WARNING: using site-packages swhlab module\")\n print(\"#\"*80)\n tryPath=os.path.dirname(tryPath)\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update(self, tids, info):\n outputs_dir = os.path.join(info['root_directory'], 'streams')\n pattern = '%s_*_tid_*{tid}.o.{tid}*' % info['batch_name']\n flist = os.listdir(outputs_dir)\n try:\n outputs = []\n for tid in tids:\n matches = fnmatch.filter(flist, pattern.format(tid=tid))\n if len(matches) != 1:\n self.warning(\"No unique output file for tid %d\" % tid)\n contents = open(os.path.join(outputs_dir, matches[0]),'r').read()\n outputs.append(self.output_extractor(contents))\n\n self._next_val = self._update_state(outputs)\n self.trace.append((outputs, self._next_val))\n except:\n self.warning(\"Cannot load required output files. Cannot continue.\")\n self._next_val = StopIteration", "response": "This method updates the state of the iterator."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nshows the current set of available key - values.", "response": "def show(self):\n \"\"\"\n When dynamic, not all argument values may be available.\n \"\"\"\n copied = self.copy()\n enumerated = [el for el in enumerate(copied)]\n for (group_ind, specs) in enumerated:\n if len(enumerated) > 1: print(\"Group %d\" % group_ind)\n ordering = self.constant_keys + self.varying_keys\n # Ordered nicely by varying_keys definition.\n spec_lines = [', '.join(['%s=%s' % (k, s[k]) for k in ordering]) for s in specs]\n print('\\n'.join(['%d: %s' % (i,l) for (i,l) in enumerate(spec_lines)]))\n\n print('Remaining arguments not available for %s' % self.__class__.__name__)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _trace_summary(self):\n for (i, (val, args)) in enumerate(self.trace):\n if args is StopIteration:\n info = \"Terminated\"\n else:\n pprint = ','.join('{' + ','.join('%s=%r' % (k,v)\n for (k,v) in arg.items()) + '}' for arg in args)\n info = (\"exploring arguments [%s]\" % pprint )\n\n if i == 0: print(\"Step %d: Initially %s.\" % (i, info))\n else: print(\"Step %d: %s after receiving input(s) %s.\" % (i, info.capitalize(), val))", "response": "Summarizes the trace of the values used to update the DynamicArgs\n and the arguments subsequently returned."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef proto_0111(theABF):\n abf=ABF(theABF)\n abf.log.info(\"analyzing as an IC ramp\")\n\n # AP detection\n ap=AP(abf)\n ap.detect()\n\n # also calculate derivative for each sweep\n abf.derivative=True\n\n # create the multi-plot figure\n plt.figure(figsize=(SQUARESIZE,SQUARESIZE))\n ax1=plt.subplot(221)\n plt.ylabel(abf.units2)\n ax2=plt.subplot(222,sharey=ax1)\n ax3=plt.subplot(223)\n plt.ylabel(abf.unitsD2)\n ax4=plt.subplot(224,sharey=ax3)\n\n # put data in each subplot\n for sweep in range(abf.sweeps):\n abf.setsweep(sweep)\n ax1.plot(abf.sweepX,abf.sweepY,color='b',lw=.25)\n ax2.plot(abf.sweepX,abf.sweepY,color='b')\n ax3.plot(abf.sweepX,abf.sweepD,color='r',lw=.25)\n ax4.plot(abf.sweepX,abf.sweepD,color='r')\n\n # modify axis\n for ax in [ax1,ax2,ax3,ax4]: # everything\n ax.margins(0,.1)\n ax.grid(alpha=.5)\n for ax in [ax3,ax4]: # only derivative APs\n ax.axhline(-100,color='r',alpha=.5,ls=\"--\",lw=2)\n for ax in [ax2,ax4]: # only zoomed in APs\n ax.get_yaxis().set_visible(False)\n if len(ap.APs):\n firstAP=ap.APs[0][\"T\"]\n ax2.axis([firstAP-.25,firstAP+.25,None,None])\n ax4.axis([firstAP-.01,firstAP+.01,None,None])\n\n # show message from first AP\n if len(ap.APs):\n firstAP=ap.APs[0]\n msg=\"\\n\".join([\"%s = %s\"%(x,str(firstAP[x])) for x in sorted(firstAP.keys()) if not \"I\" in x[-2:]])\n plt.subplot(221)\n plt.gca().text(0.02, 0.98, msg, transform= plt.gca().transAxes, fontsize=10, verticalalignment='top', family='monospace')\n\n # save it\n plt.tight_layout()\n frameAndSave(abf,\"AP shape\")\n plt.close('all')", "response": "protocol: IC ramp for AP shape analysis"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef proto_gain(theABF,stepSize=25,startAt=-100):\n abf=ABF(theABF)\n abf.log.info(\"analyzing as an IC ramp\")\n plot=ABFplot(abf)\n plot.kwargs[\"lw\"]=.5\n plot.title=\"\"\n currents=np.arange(abf.sweeps)*stepSize-startAt\n\n # AP detection\n ap=AP(abf)\n ap.detect_time1=.1\n ap.detect_time2=.7\n ap.detect()\n\n # stacked plot\n plt.figure(figsize=(SQUARESIZE,SQUARESIZE))\n\n ax1=plt.subplot(221)\n plot.figure_sweeps()\n\n ax2=plt.subplot(222)\n ax2.get_yaxis().set_visible(False)\n plot.figure_sweeps(offsetY=150)\n\n # add vertical marks to graphs:\n for ax in [ax1,ax2]:\n for limit in [ap.detect_time1,ap.detect_time2]:\n ax.axvline(limit,color='r',ls='--',alpha=.5,lw=2)\n\n # make stacked gain function\n ax4=plt.subplot(223)\n plt.ylabel(\"frequency (Hz)\")\n plt.ylabel(\"seconds\")\n plt.grid(alpha=.5)\n freqs=ap.get_bySweep(\"freqs\")\n times=ap.get_bySweep(\"times\")\n for i in range(abf.sweeps):\n if len(freqs[i]):\n plt.plot(times[i][:-1],freqs[i],'-',alpha=.5,lw=2,\n color=plot.getColor(i/abf.sweeps))\n\n # make gain function graph\n ax4=plt.subplot(224)\n ax4.grid(alpha=.5)\n plt.plot(currents,ap.get_bySweep(\"median\"),'b.-',label=\"median\")\n plt.plot(currents,ap.get_bySweep(\"firsts\"),'g.-',label=\"first\")\n plt.xlabel(\"applied current (pA)\")\n plt.legend(loc=2,fontsize=10)\n plt.axhline(40,color='r',alpha=.5,ls=\"--\",lw=2)\n plt.margins(.02,.1)\n\n # save it\n plt.tight_layout()\n frameAndSave(abf,\"AP Gain %d_%d\"%(startAt,stepSize))\n plt.close('all')\n\n # make a second figure that just shows every sweep up to the first AP\n plt.figure(figsize=(SQUARESIZE,SQUARESIZE))\n plt.grid(alpha=.5)\n plt.ylabel(\"Membrane Potential (mV)\")\n plt.xlabel(\"Time (seconds)\")\n for sweep in abf.setsweeps():\n plt.plot(abf.sweepX2,abf.sweepY,color='b',alpha=.5)\n if np.max(abf.sweepY>0):\n break\n plt.tight_layout()\n plt.margins(0,.1)\n\n plt.axis([0,1,None,None])\n plt.title(\"%d pA Steps from Rest\"%stepSize)\n frameAndSave(abf,\"voltage response fromRest\",closeWhenDone=False)\n plt.axis([1.5,2.5,None,None])\n plt.title(\"%d pA Steps from %d pA\"%(stepSize,startAt))\n frameAndSave(abf,\"voltage response hyperpol\",closeWhenDone=False)\n plt.close('all')", "response": "protocol: gain function of some sort. step size and start at are pA."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprotocolling : membrane test", "response": "def proto_0201(theABF):\n \"\"\"protocol: membrane test.\"\"\"\n abf=ABF(theABF)\n abf.log.info(\"analyzing as a membrane test\")\n plot=ABFplot(abf)\n plot.figure_height,plot.figure_width=SQUARESIZE/2,SQUARESIZE/2\n plot.figure_sweeps()\n\n # save it\n plt.tight_layout()\n frameAndSave(abf,\"membrane test\")\n plt.close('all')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprotocols 2. 0. 1. 2. 1", "response": "def proto_0202(theABF):\n \"\"\"protocol: MTIV.\"\"\"\n abf=ABF(theABF)\n abf.log.info(\"analyzing as MTIV\")\n plot=ABFplot(abf)\n plot.figure_height,plot.figure_width=SQUARESIZE,SQUARESIZE\n plot.title=\"\"\n plot.kwargs[\"alpha\"]=.6\n plot.figure_sweeps()\n\n # frame to uppwer/lower bounds, ignoring peaks from capacitive transients\n abf.setsweep(0)\n plt.axis([None,None,abf.average(.9,1)-100,None])\n abf.setsweep(-1)\n plt.axis([None,None,None,abf.average(.9,1)+100])\n\n # save it\n plt.tight_layout()\n frameAndSave(abf,\"MTIV\")\n plt.close('all')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef proto_0203(theABF):\n abf=ABF(theABF)\n abf.log.info(\"analyzing as a fast IV\")\n plot=ABFplot(abf)\n plot.title=\"\"\n m1,m2=.7,1\n plt.figure(figsize=(SQUARESIZE,SQUARESIZE/2))\n\n plt.subplot(121)\n plot.figure_sweeps()\n plt.axvspan(m1,m2,color='r',ec=None,alpha=.1)\n\n plt.subplot(122)\n plt.grid(alpha=.5)\n Xs=np.arange(abf.sweeps)*5-110\n Ys=[]\n for sweep in range(abf.sweeps):\n abf.setsweep(sweep)\n Ys.append(abf.average(m1,m2))\n plt.plot(Xs,Ys,'.-',ms=10)\n plt.axvline(-70,color='r',ls='--',lw=2,alpha=.5)\n plt.axhline(0,color='r',ls='--',lw=2,alpha=.5)\n plt.margins(.1,.1)\n plt.xlabel(\"membrane potential (mV)\")\n\n # save it\n plt.tight_layout()\n frameAndSave(abf,\"fast IV\")\n plt.close('all')", "response": "protocol : vast IV"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef proto_0303(theABF):\n\n abf=ABF(theABF)\n abf.log.info(\"analyzing as a halorhodopsin (2s pulse)\")\n\n # show average voltage\n proto_avgRange(theABF,0.2,1.2)\n plt.close('all')\n\n # show stacked sweeps\n plt.figure(figsize=(8,8))\n for sweep in abf.setsweeps():\n color='b'\n if sweep in np.array(abf.comment_sweeps,dtype=int):\n color='r'\n plt.plot(abf.sweepX2,abf.sweepY+100*sweep,color=color,alpha=.5)\n plt.margins(0,.01)\n plt.tight_layout()\n frameAndSave(abf,\"IC ramps\")\n plt.close('all')\n\n # do AP event detection\n ap=AP(abf)\n ap.detect_time1=2.3\n ap.detect_time2=8.3\n ap.detect()\n apCount=[]\n apSweepTimes=[]\n\n for sweepNumber,times in enumerate(ap.get_bySweep(\"times\")):\n apCount.append(len(times))\n if len(times):\n apSweepTimes.append(times[0])\n else:\n apSweepTimes.append(0)\n\n # plot AP frequency vs time\n plt.figure(figsize=(8,8))\n\n ax1=plt.subplot(211)\n plt.grid(alpha=.4,ls='--')\n plt.plot(np.arange(len(apCount))*abf.sweepLength/60,apCount,'.-',ms=15)\n comment_lines(abf)\n plt.ylabel(\"AP Count\")\n\n plt.subplot(212,sharex=ax1)\n plt.grid(alpha=.4,ls='--')\n plt.plot(np.arange(len(apCount))*abf.sweepLength/60,apSweepTimes,'.-',ms=15)\n comment_lines(abf)\n plt.ylabel(\"First AP Time (s)\")\n plt.xlabel(\"Experiment Duration (minutes)\")\n plt.tight_layout()\n frameAndSave(abf,\"IC ramp freq\")\n plt.close('all')", "response": "protocol: repeated IC ramps."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprotocolling for repeated IC steps.", "response": "def proto_0304(theABF):\n \"\"\"protocol: repeated IC steps.\"\"\"\n\n abf=ABF(theABF)\n abf.log.info(\"analyzing as repeated current-clamp step\")\n\n # prepare for AP analysis\n ap=AP(abf)\n\n # calculate rest potential\n avgVoltagePerSweep = [];\n times = []\n for sweep in abf.setsweeps():\n avgVoltagePerSweep.append(abf.average(0,3))\n times.append(abf.sweepStart/60)\n\n # detect only step APs\n M1,M2=3.15,4.15\n ap.detect_time1, ap.detect_time2 = M1,M2\n ap.detect()\n apsPerSweepCos=[len(x) for x in ap.get_bySweep()]\n\n # detect all APs\n M1,M2=0,10\n ap.detect_time1, ap.detect_time2 = M1,M2\n ap.detect()\n apsPerSweepRamp=[len(x) for x in ap.get_bySweep()]\n\n # make the plot of APs and stuff\n plt.figure(figsize=(8,8))\n\n plt.subplot(311)\n plt.grid(ls='--',alpha=.5)\n plt.plot(times,avgVoltagePerSweep,'.-')\n plt.ylabel(\"Rest Potential (mV)\")\n comment_lines(abf)\n\n plt.subplot(312)\n plt.grid(ls='--',alpha=.5)\n plt.plot(times,apsPerSweepCos,'.-')\n plt.ylabel(\"APs in Step (#)\")\n comment_lines(abf)\n\n plt.subplot(313)\n plt.grid(ls='--',alpha=.5)\n plt.plot(times,apsPerSweepRamp,'.-')\n plt.ylabel(\"APs in Sweep (#)\")\n comment_lines(abf)\n\n plt.tight_layout()\n\n frameAndSave(abf,\"cos ramp\")\n plt.close('all')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef proto_avgRange(theABF,m1=None,m2=None):\n\n abf=ABF(theABF)\n abf.log.info(\"analyzing as a fast IV\")\n if m1 is None:\n m1=abf.sweepLength\n if m2 is None:\n m2=abf.sweepLength\n\n I1=int(abf.pointsPerSec*m1)\n I2=int(abf.pointsPerSec*m2)\n\n Ts=np.arange(abf.sweeps)*abf.sweepInterval\n Yav=np.empty(abf.sweeps)*np.nan # average\n Ysd=np.empty(abf.sweeps)*np.nan # standard deviation\n #Yar=np.empty(abf.sweeps)*np.nan # area\n\n for sweep in abf.setsweeps():\n Yav[sweep]=np.average(abf.sweepY[I1:I2])\n Ysd[sweep]=np.std(abf.sweepY[I1:I2])\n #Yar[sweep]=np.sum(abf.sweepY[I1:I2])/(I2*I1)-Yav[sweep]\n\n plot=ABFplot(abf)\n plt.figure(figsize=(SQUARESIZE*2,SQUARESIZE/2))\n\n plt.subplot(131)\n plot.title=\"first sweep\"\n plot.figure_sweep(0)\n plt.title(\"First Sweep\\n(shaded measurement range)\")\n plt.axvspan(m1,m2,color='r',ec=None,alpha=.1)\n\n plt.subplot(132)\n plt.grid(alpha=.5)\n for i,t in enumerate(abf.comment_times):\n plt.axvline(t/60,color='r',alpha=.5,lw=2,ls='--')\n plt.plot(Ts/60,Yav,'.',alpha=.75)\n plt.title(\"Range Average\\nTAGS: %s\"%(\", \".join(abf.comment_tags)))\n plt.ylabel(abf.units2)\n plt.xlabel(\"minutes\")\n plt.margins(0,.1)\n\n plt.subplot(133)\n plt.grid(alpha=.5)\n for i,t in enumerate(abf.comment_times):\n plt.axvline(t/60,color='r',alpha=.5,lw=2,ls='--')\n plt.plot(Ts/60,Ysd,'.',alpha=.5,color='g',ms=15,mew=0)\n #plt.fill_between(Ts/60,Ysd*0,Ysd,lw=0,alpha=.5,color='g')\n plt.title(\"Range Standard Deviation\\nTAGS: %s\"%(\", \".join(abf.comment_tags)))\n plt.ylabel(abf.units2)\n plt.xlabel(\"minutes\")\n plt.margins(0,.1)\n plt.axis([None,None,0,np.percentile(Ysd,99)*1.25])\n\n plt.tight_layout()\n frameAndSave(abf,\"sweep vs average\",\"experiment\")\n plt.close('all')", "response": "experiment: generic VC time course experiment"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef analyze(fname=False,save=True,show=None):\n if fname and os.path.exists(fname.replace(\".abf\",\".rst\")):\n print(\"SKIPPING DUE TO RST FILE\")\n return\n swhlab.plotting.core.IMAGE_SAVE=save\n if show is None:\n if cm.isIpython():\n swhlab.plotting.core.IMAGE_SHOW=True\n else:\n swhlab.plotting.core.IMAGE_SHOW=False\n #swhlab.plotting.core.IMAGE_SHOW=show\n abf=ABF(fname) # ensure it's a class\n print(\">>>>> PROTOCOL >>>>>\",abf.protocomment)\n runFunction=\"proto_unknown\"\n if \"proto_\"+abf.protocomment in globals():\n runFunction=\"proto_\"+abf.protocomment\n abf.log.debug(\"running %s()\"%(runFunction))\n plt.close('all') # get ready\n globals()[runFunction](abf) # run that function\n try:\n globals()[runFunction](abf) # run that function\n except:\n abf.log.error(\"EXCEPTION DURING PROTOCOL FUNCTION\")\n abf.log.error(sys.exc_info()[0])\n return \"ERROR\"\n plt.close('all') # clean up\n return \"SUCCESS\"", "response": "try to analyze a file or ABF object"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef processFolder(abfFolder):\n if not type(abfFolder) is str or not len(abfFolder)>3:\n return\n files=sorted(glob.glob(abfFolder+\"/*.abf\"))\n for i,fname in enumerate(files):\n print(\"\\n\\n\\n### PROCESSING {} of {}:\".format(i,len(files)),os.path.basename(fname))\n processAbf(fname,show=False)\n plt.show()\n return", "response": "call processAbf for every ABF in a folder"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef processAbf(abfFname,saveAs=False,dpi=100,show=True):\n if not type(abfFname) is str or not len(abfFname)>3:\n return\n abf=swhlab.ABF(abfFname)\n plot=swhlab.plotting.ABFplot(abf)\n plot.figure_height=6\n plot.figure_width=10\n plot.subplot=False\n plot.figure(True)\n if abf.get_protocol_sequence(0)==abf.get_protocol_sequence(1) or abf.sweeps<2:\n # same protocol every time\n if abf.lengthMinutes<2:\n # short (probably a memtest or tau)\n ax1=plt.subplot(211)\n plot.figure_sweeps()\n plt.title(\"{} ({} sweeps)\".format(abf.ID,abf.sweeps))\n plt.gca().get_xaxis().set_visible(False)\n plt.subplot(212,sharex=ax1)\n plot.figure_protocol()\n plt.title(\"\")\n else:\n # long (probably a drug experiment)\n plot.figure_chronological()\n else:\n # protocol changes every sweep\n plots=[211,212] # assume we want 2 images\n if abf.units=='mV': # maybe it's something with APs?\n ap=swhlab.AP(abf) # go ahead and do AP detection\n ap.detect() # try to detect APs\n if len(ap.APs): # if we found some\n plots=[221,223,222,224] # get ready for 4 images\n ax1=plt.subplot(plots[0])\n plot.figure_sweeps()\n plt.title(\"{} ({} sweeps)\".format(abf.ID,abf.sweeps))\n plt.gca().get_xaxis().set_visible(False)\n plt.subplot(plots[1],sharex=ax1)\n plot.figure_protocols()\n plt.title(\"protocol\")\n if len(plots)>2:\n # assume we want to look at the first AP\n ax2=plt.subplot(plots[2])\n plot.rainbow=False\n plot.kwargs[\"color\"]='b'\n plot.figure_chronological()\n plt.gca().get_xaxis().set_visible(False)\n plt.title(\"first AP magnitude\")\n # velocity plot\n plt.subplot(plots[3],sharex=ax2)\n plot.abf.derivative=True\n plot.rainbow=False\n plot.traceColor='r'\n plot.figure_chronological()\n plt.axis([ap.APs[0][\"T\"]-.05,ap.APs[0][\"T\"]+.05,None,None])\n plt.title(\"first AP velocity\")\n\n if saveAs:\n print(\"saving\",os.path.abspath(saveAs))\n plt.savefig(os.path.abspath(saveAs),dpi=dpi)\n return\n if show:\n plot.show()", "response": "Process an ABF and return a tuple of the ID and the image."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef frameAndSave(abf,tag=\"\",dataType=\"plot\",saveAsFname=False,closeWhenDone=True):\n print(\"closeWhenDone\",closeWhenDone)\n plt.tight_layout()\n plt.subplots_adjust(top=.93,bottom =.07)\n plt.annotate(tag,(.01,.99),xycoords='figure fraction',ha='left',va='top',family='monospace',size=10,alpha=.5)\n msgBot=\"%s [%s]\"%(abf.ID,abf.protocomment)\n plt.annotate(msgBot,(.01,.01),xycoords='figure fraction',ha='left',va='bottom',family='monospace',size=10,alpha=.5)\n fname=tag.lower().replace(\" \",'_')+\".jpg\"\n fname=dataType+\"_\"+fname\n plt.tight_layout()\n if IMAGE_SAVE:\n abf.log.info(\"saving [%s]\",fname)\n try:\n if saveAsFname:\n saveAs=os.path.abspath(saveAsFname)\n else:\n saveAs=os.path.abspath(abf.outPre+fname)\n if not os.path.exists(abf.outFolder):\n os.mkdir(abf.outFolder)\n plt.savefig(saveAs)\n except Exception as E:\n abf.log.error(\"saving [%s] failed! 'pip install pillow'?\",fname)\n print(E)\n if IMAGE_SHOW==True:\n if closeWhenDone==False:\n print(\"NOT SHOWING (because closeWhenDone==True and showing would mess things up)\")\n else:\n abf.log.info(\"showing [%s]\",fname)\n plt.show()\n if closeWhenDone:\n print(\"closing figure\")\n plt.close('all')", "response": "frame the current matplotlib plot with ABF info and optionally save it."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef figure(self,forceNew=False):\n if plt._pylab_helpers.Gcf.get_num_fig_managers()>0 and forceNew is False:\n self.log.debug(\"figure already seen, not creating one.\")\n return\n\n if self.subplot:\n self.log.debug(\"subplot mode enabled, not creating new figure\")\n else:\n self.log.debug(\"creating new figure\")\n plt.figure(figsize=(self.figure_width,self.figure_height))", "response": "make sure a figure is ready."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsaving the existing figure. does not close it.", "response": "def save(self,callit=\"misc\",closeToo=True,fullpath=False):\n \"\"\"save the existing figure. does not close it.\"\"\"\n if fullpath is False:\n fname=self.abf.outPre+\"plot_\"+callit+\".jpg\"\n else:\n fname=callit\n if not os.path.exists(os.path.dirname(fname)):\n os.mkdir(os.path.dirname(fname))\n plt.savefig(fname)\n self.log.info(\"saved [%s]\",os.path.basename(fname))\n if closeToo:\n plt.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd comments to an existing plot.", "response": "def comments(self,minutes=False):\n \"\"\"\n Add comment lines/text to an existing plot. Defaults to seconds.\n Call after a plot has been made, and after margins have been set.\n \"\"\"\n if self.comments==0:\n return\n self.log.debug(\"adding comments to plot\")\n for i,t in enumerate(self.abf.comment_times):\n if minutes:\n t/=60.0\n plt.axvline(t,color='r',ls=':')\n X1,X2,Y1,Y2=plt.axis()\n Y2=Y2-abs(Y2-Y1)*.02\n plt.text(t,Y2,self.abf.comment_tags[i],color='r',rotation='vertical',\n ha='right',va='top',weight='bold',alpha=.5,size=8,)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a chronological plot of every sweep of an ABF file", "response": "def figure_chronological(self):\n \"\"\"plot every sweep of an ABF file (with comments).\"\"\"\n self.log.debug(\"creating chronological plot\")\n self.figure()\n for sweep in range(self.abf.sweeps):\n self.abf.setsweep(sweep)\n self.setColorBySweep()\n if self.abf.derivative:\n plt.plot(self.abf.sweepX,self.abf.sweepD,**self.kwargs)\n else:\n plt.plot(self.abf.sweepX,self.abf.sweepY,**self.kwargs)\n self.comments()\n self.decorate()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef figure_sweeps(self, offsetX=0, offsetY=0):\n self.log.debug(\"creating overlayed sweeps plot\")\n self.figure()\n for sweep in range(self.abf.sweeps):\n self.abf.setsweep(sweep)\n self.setColorBySweep()\n plt.plot(self.abf.sweepX2+sweep*offsetX,\n self.abf.sweepY+sweep*offsetY,\n **self.kwargs)\n if offsetX:\n self.marginX=.05\n self.decorate()", "response": "This function creates the overlayed sweeps plot for each sweep of an ABF file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef figure_protocol(self):\n self.log.debug(\"creating overlayed protocols plot\")\n self.figure()\n plt.plot(self.abf.protoX,self.abf.protoY,color='r')\n self.marginX=0\n self.decorate(protocol=True)", "response": "create the overlayed sweep protocol plot"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates overlayed protocols plot", "response": "def figure_protocols(self):\n \"\"\"plot the protocol of all sweeps.\"\"\"\n self.log.debug(\"creating overlayed protocols plot\")\n self.figure()\n for sweep in range(self.abf.sweeps):\n self.abf.setsweep(sweep)\n plt.plot(self.abf.protoX,self.abf.protoY,color='r')\n self.marginX=0\n self.decorate(protocol=True)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fread(f,byteLocation,structFormat=None,nBytes=1):\n f.seek(byteLocation)\n if structFormat:\n val = struct.unpack(structFormat, f.read(struct.calcsize(structFormat)))\n val = val[0] if len(val)==1 else list(val)\n return val\n else:\n return f.read(nBytes)", "response": "Reads a certain number of bytes from a file object f."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread the header of an ABF file into memory.", "response": "def abf_read_header(fname, saveHeader=True):\n \"\"\"\n Practice pulling data straight out of an ABF's binary header. Support only ABF2 (ClampEx an ClampFit 10).\n Use only native python libraries. Strive for simplicity and readability (to promote language portability).\n This was made by Scott Harden after a line-by-line analysis of axonrawio.py from the neo io library.\n Unlike NeoIO's format, I'm going to try to prevent nested dictionaries to keep things simple.\n \"\"\"\n \n ### THESE OBJETS WILL BE FULL WHEN THIS FUNCTION IS COMPLETE\n header={} # information about the file format\n sections={} # contains byte positions (and block sizes) for header information\n strings=[] # a list of strings stored in the ABF header (protocol file, abf comment, and channel labels / units)\n protocol = {} # info about the ABF recording\n tags=[] # timed comments made during the abf as a list of lists [pos,comment,tagtype,voice]\n adcs=[] # ADC info as a list of dicts\n dacs=[] # DAC info as a list of dicts\n digitalOutputs=[] # one 0b00000000 code per epoch\n config={} # a concise collection of ABF info I think is useful. I intend only to return this object.\n \n ### READ THE FIRST PART OF THE FILE INTO MEMORY\n # TODO: figure out the most memory-effecient way to do this\n f=open(fname,'rb')\n config[\"abf_filename\"]=os.path.abspath(fname) # full path to the abf file on disk\n config[\"abf_ID\"]=os.path.basename(fname)[:-4] # abf filename without the \".abf\"\n \n ### DECODE HEADER - this tells basic information about the file format\n for key, byte_location, fmt in headerDescriptionV2:\n header[key]=fread(f,byte_location,fmt)\n header['fFileSignature']=header['fFileSignature'].decode()\n \n ### DECODE SECTIONS - sections are where (byte location) in this file different data is stored\n for sectionNumber, sectionName in enumerate(sectionNames):\n uBlockIndex, uBytes, llNumEntries = fread(f,76+sectionNumber*16,\"IIl\")\n sections[sectionName] = [uBlockIndex,uBytes,llNumEntries]\n config[\"abf_sweep_start_time\"]=header['uFileStartTimeMS']/1000/1000\n \n ### DECODE STRINGS - figure out where (byte location) strings are in this file\n # There are 20 strings. Protocol path, ABF comment, then alternating channel name and units.\n byte_location = sections['StringsSection'][0]*BLOCKSIZE\n string_size = sections['StringsSection'][1]\n strings_data = fread(f,byte_location,structFormat=None,nBytes=string_size)\n for key in [b'AXENGN', b'clampex', b'Clampex', b'CLAMPEX', b'axoscope']:\n if key in strings_data:\n for line in strings_data.split(key)[1].split(b'\\x00')[1:-1]:\n strings.append(line.decode())\n config[\"abf_protocol_file\"]=strings[0]\n config[\"abf_comment\"]=strings[1]\n config[\"abf_channels\"]=strings[2::2]\n config[\"abf_units\"]=strings[3::2]\n break\n \n ### DECODE ADC INFO - a list of dictionaries, one per ADC employed\n for ADCsection in range(sections['ADCSection'][2]):\n thisADC={}\n byte_location=sections['ADCSection'][0]*BLOCKSIZE+sections['ADCSection'][1]*ADCsection\n for key, fmt in ADCInfoDescription:\n thisADC[key]=fread(f,byte_location,fmt)\n byte_location+=struct.calcsize(fmt)\n adcs.append(thisADC)\n \n ### PROTOCOL - info about the nature of the recording\n byte_location=sections['ProtocolSection'][0]*BLOCKSIZE\n for key, fmt in protocolInfoDescription:\n protocol[key]=fread(f,byte_location,fmt)\n byte_location+=struct.calcsize(fmt)\n protocol.pop('sUnused', None) # we don't need this\n \n ### TAGS (COMMMENTS) - those with a timestamp/comment (not those embedded in the protocol)\n #TODO: not sure what the tagtime units actually are. Byte positions? Block number?\n byte_location=sections['TagSection'][0]*BLOCKSIZE\n for i in range(sections['TagSection'][2]):\n thisTag=[]\n for key, fmt in TagInfoDescription:\n val=fread(f,byte_location,fmt)\n if type(val) is bytes:\n val=val.decode().strip()\n thisTag.append(val)\n byte_location+=struct.calcsize(fmt)\n tags.append(thisTag)\n \n ### DAC SECTIONS \n for dacNumber in range(sections['DACSection'][2]):\n thisDAC={}\n byte_location=sections['DACSection'][0]*BLOCKSIZE+sections['DACSection'][1]*dacNumber \n for key, fmt in DACInfoDescription:\n thisDAC[key]=fread(f,byte_location,fmt)\n byte_location+=struct.calcsize(fmt)\n thisDAC.pop('sUnused', None) # we don't need this\n if thisDAC['nWaveformEnable']==0: continue # don't record unused DACs\n dacs.append(thisDAC)\n \n ### EPOCHS PER DAC - this is what appear on the waveform tab\n epochs=[]\n for epochNumber in range(sections['EpochPerDACSection'][2]):\n thisEpoch={}\n byte_location=sections['EpochPerDACSection'][0]*BLOCKSIZE+sections['EpochPerDACSection'][1]*epochNumber\n for key, fmt in EpochInfoPerDACDescription:\n thisEpoch[key]=fread(f,byte_location,fmt)\n byte_location+=struct.calcsize(fmt)\n thisEpoch.pop('sUnused', None) # we don't need this\n epochs.append(thisEpoch)\n \n ### DIGITAL OUTPUTS - this is where digital outputs are stored. Returns binary string (7-0)\n # this does not exist in Neo IO. It was hacked-in by Scott Harden (github: swharden) to capture digital outputs.\n # let's just add the digital output string to the epochs array we already have started.\n byte_location=sections['EpochSection'][0]*BLOCKSIZE\n for epochNumber in range(sections['EpochSection'][0]):\n if epochNumber>=len(epochs):\n break # don't bother looking up unused epochs\n thisEpoch=epochs[epochNumber]\n for key, fmt in EpochSectionDescription:\n val=fread(f,byte_location,fmt)\n if key=='nEpochDigitalOutput':\n val=format(val, 'b').rjust(8,'0') # convert to a binary string (7->0)\n thisEpoch[key]=val\n byte_location+=struct.calcsize(fmt)\n thisEpoch.pop('sUnused', None) # we don't need this\n epochs[epochNumber]=thisEpoch\n \n ### WE ARE DONE READING THE FILE\n f.close()\n \n ### EXTRA CONFIG - cherry-pick just what I find useful and arrange it in a simple dictionary\n config[\"abfVersion\"]=float(\"\".join([str(x) for x in header['fFileVersionNumber']]))/100 # now a float\n config['signalNames']=config['abf_channels'][:len(adcs)] # don't need more than the number of channels\n config['signalUnits']=config['abf_units'][:len(adcs)] # don't need more than the number of channels\n config['comments']=[x[:2] for x in tags]\n config['nSweeps']=sections['SynchArraySection'][2]\n \n # Determine the recording date from the header \n YY = int(header['uFileStartDate'] / 10000)\n MM = int((header['uFileStartDate'] - YY * 10000) / 100)\n DD = int(header['uFileStartDate'] - YY * 10000 - MM * 100)\n hh = int(header['uFileStartTimeMS'] / 1000. / 3600.)\n mm = int((header['uFileStartTimeMS'] / 1000. - hh * 3600) / 60)\n ss = header['uFileStartTimeMS'] / 1000. - hh * 3600 - mm * 60\n ms = int((ss%1)*1e6)\n ss = int(ss)\n config['abf_datetime'] = datetime.datetime(YY, MM, DD, hh, mm, ss, ms)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates and save a two column frames HTML file.", "response": "def frames(fname=None,menuWidth=200,launch=False):\n \"\"\"create and save a two column frames HTML file.\"\"\"\n html=\"\"\"\n \n \n \n \"\"\"%(menuWidth)\n with open(fname,'w') as f:\n f.write(html)\n if launch:\n webbrowser.open(fname)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving HTML to disk", "response": "def save(html,fname=None,launch=False):\n \"\"\"wrap HTML in a top and bottom (with css) and save to disk.\"\"\"\n html=html_top+html+html_bot\n html=html.replace(\"~GENAT~\",swhlab.common.datetimeToString())\n if fname is None:\n fname = tempfile.gettempdir()+\"/temp.html\"\n launch=True\n fname=os.path.abspath(fname)\n with open(fname,'w') as f:\n f.write(html)\n\n global stylesheetSaved\n stylesheetPath=os.path.join(os.path.dirname(fname),\"style.css\")\n if not os.path.exists(stylesheetPath) or stylesheetSaved is False:\n with open(stylesheetPath,'w') as f:\n f.write(stylesheet)\n stylesheetSaved=True\n if launch:\n webbrowser.open(fname)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrenaming all files in a tree with the given character.", "response": "def clampfit_rename(path,char):\n \"\"\"\n Given ABFs and TIFs formatted long style, rename each of them to prefix their number with a different number.\n\n Example: 2017_10_11_0011.abf\n Becomes: 2017_10_11_?011.abf\n where ? can be any character.\n \"\"\"\n assert len(char)==1 and type(char)==str, \"replacement character must be a single character\"\n assert os.path.exists(path), \"path doesn't exist\"\n files = sorted(os.listdir(path))\n files = [x for x in files if len(x)>18 and x[4]+x[7]+x[10]=='___']\n for fname in files:\n fname2 = list(fname)\n fname2[11]=char\n fname2=\"\".join(fname2)\n\n if fname==fname2:\n print(fname, \"==\", fname2)\n else:\n print(fname, \"->\", fname2)\n# fname=os.path.join(path,fname)\n# fname2=os.path.join(path,fname2)\n# if not os.path.exists(fname2):\n# os.rename(fname,fname2)\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a 1d gassuan array of a given size and sigma.", "response": "def kernel_gaussian(size=100, sigma=None, forwardOnly=False):\n \"\"\"\n return a 1d gassuan array of a given size and sigma.\n If sigma isn't given, it will be 1/10 of the size, which is usually good.\n \"\"\"\n if sigma is None:sigma=size/10\n points=np.exp(-np.power(np.arange(size)-size/2,2)/(2*np.power(sigma,2)))\n if forwardOnly:\n points[:int(len(points)/2)]=0\n return points/sum(points)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving a list of files return a dict organized by extension.", "response": "def filesByExtension(fnames):\n \"\"\"given a list of files, return a dict organized by extension.\"\"\"\n byExt={\"abf\":[],\"jpg\":[],\"tif\":[]} # prime it with empties\n for fname in fnames:\n ext = os.path.splitext(fname)[1].replace(\".\",'').lower()\n if not ext in byExt.keys():\n byExt[ext]=[]\n byExt[ext]=byExt[ext]+[fname]\n return byExt"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef findCells(fnames):\n IDs=[]\n filesByExt = filesByExtension(fnames)\n for abfFname in filesByExt['abf']:\n ID=os.path.splitext(abfFname)[0]\n for picFname in filesByExt['jpg']+filesByExt['tif']:\n if picFname.startswith(ID):\n IDs.append(ID)\n break\n return smartSort(IDs)", "response": "Given a list of files return a list of cells by their ID."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filesByCell(fnames,cells):\n byCell={}\n fnames=smartSort(fnames)\n days = list(set([elem[:5] for elem in fnames if elem.endswith(\".abf\")])) # so pythonic!\n for day in smartSort(days):\n parent=None\n for i,fname in enumerate([elem for elem in fnames if elem.startswith(day) and elem.endswith(\".abf\")]):\n ID=os.path.splitext(fname)[0]\n if len([x for x in fnames if x.startswith(ID)])-1:\n parent=ID\n if not parent in byCell:\n byCell[parent]=[]\n byCell[parent]=byCell[parent]+[fname]\n return byCell", "response": "given files and cells return a dict of files grouped by cell."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npopulates class properties relating to files in the folder.", "response": "def folderScan(self,abfFolder=None):\n \"\"\"populate class properties relating to files in the folder.\"\"\"\n if abfFolder is None and 'abfFolder' in dir(self):\n abfFolder=self.abfFolder\n else:\n self.abfFolder=abfFolder\n self.abfFolder=os.path.abspath(self.abfFolder)\n self.log.info(\"scanning [%s]\",self.abfFolder)\n if not os.path.exists(self.abfFolder):\n self.log.error(\"path doesn't exist: [%s]\",abfFolder)\n return\n self.abfFolder2=os.path.abspath(self.abfFolder+\"/swhlab/\")\n if not os.path.exists(self.abfFolder2):\n self.log.error(\"./swhlab/ doesn't exist. creating it...\") \n os.mkdir(self.abfFolder2)\n self.fnames=os.listdir(self.abfFolder)\n self.fnames2=os.listdir(self.abfFolder2)\n self.log.debug(\"./ has %d files\",len(self.fnames))\n self.log.debug(\"./swhlab/ has %d files\",len(self.fnames2))\n self.fnamesByExt = filesByExtension(self.fnames)\n if not \"abf\" in self.fnamesByExt.keys():\n self.log.error(\"no ABF files found\")\n self.log.debug(\"found %d ABFs\",len(self.fnamesByExt[\"abf\"]))\n \n self.cells=findCells(self.fnames) # list of cells by their ID\n self.log.debug(\"found %d cells\"%len(self.cells))\n \n self.fnamesByCell = filesByCell(self.fnames,self.cells) # only ABFs\n self.log.debug(\"grouped cells by number of source files: %s\"%\\\n str([len(self.fnamesByCell[elem]) for elem in self.fnamesByCell]))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates list of cells with links and links. keep this simple.", "response": "def html_index(self,launch=False,showChildren=False):\n \"\"\"\n generate list of cells with links. keep this simple.\n automatically generates splash page and regnerates frames.\n \"\"\"\n self.makePics() # ensure all pics are converted\n # generate menu\n html='./%s/
'%os.path.basename(self.abfFolder)\n for ID in smartSort(self.fnamesByCell.keys()):\n link=''\n if ID+\".html\" in self.fnames2:\n link='href=\"%s.html\" target=\"content\"'%ID\n html+=('%s
'%(link,ID)) # show the parent ABF (ID)\n if showChildren:\n for fname in self.fnamesByCell[ID]:\n thisID=os.path.splitext(fname)[0]\n files2=[x for x in self.fnames2 if x.startswith(thisID) and not x.endswith(\".html\")]\n html+='%s'%thisID # show the child ABF\n if len(files2):\n html+=' (%s)'%len(files2) # show number of supporting files\n html+='
'\n html+=\"
\"\n style.save(html,self.abfFolder2+\"/index_menu.html\")\n self.html_index_splash() # make splash page\n style.frames(self.abfFolder2+\"/index.html\",launch=launch)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate HTML for a single given cell.", "response": "def html_single_basic(self,ID):\n \"\"\"\n generate ./swhlab/xxyxxzzz.html for a single given abf.\n Input can be an ABF file path of ABF ID.\n \"\"\"\n if not ID in self.cells:\n self.log.error(\"ID [%s] not seen!\",ID)\n return\n htmlFname=os.path.abspath(self.abfFolder2+\"/\"+ID+\".html\")\n html=\"

Data for ID %s

\"%ID\n npics=0\n for childID in [os.path.splitext(x)[0] for x in self.fnamesByCell[ID]]:\n pics=[x for x in self.fnames2 if x.startswith(childID) and os.path.splitext(x)[1].lower() in [\".png\",\".jpg\"]]\n html+=\"%s
\"%(os.path.abspath(self.abfFolder+'/'+childID+\".abf\"))\n for i,pic in enumerate(pics):\n html+=''%(pic,pic)\n npics+=1\n html+=\"


\"\n style.save(html,htmlFname)\n self.log.info(\"created %s containing %d pictures\",htmlFname,npics)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a data view for every ABF in the project folder.", "response": "def html_singleAll(self,template=\"basic\"):\n \"\"\"generate a data view for every ABF in the project folder.\"\"\"\n for fname in smartSort(self.cells):\n if template==\"fixed\":\n self.html_single_fixed(fname)\n else:\n self.html_single_basic(fname)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef makePics(self):\n rescanNeeded=False\n for fname in smartSort(self.fnames):\n if fname in self.fnames2:\n continue\n ext=os.path.splitext(fname)[1].lower()\n if ext in [\".jpg\",\".png\"]:\n if not fname in self.abfFolder2:\n self.log.debug(\"copying %s\",fname)\n shutil.copy(os.path.join(self.abfFolder,fname),os.path.join(self.abfFolder2,fname))\n rescanNeeded=True\n if ext in [\".tif\",\".tiff\"]:\n if not fname+\".jpg\" in self.fnames2:\n self.log.debug(\"converting %s\",fname)\n swhlab.swh_image.TIF_to_jpg(os.path.join(self.abfFolder,fname),saveAs=os.path.join(self.abfFolder2,fname+\".jpg\"))\n rescanNeeded=True\n if rescanNeeded:\n self.log.debug(\"new pics, so a rescan is needed...\")\n self.log.debug(\"REBUILDING ALL RECOMMENDED!!!!!!!!!!!\")\n self.folderScan()", "response": "convert every. image we find to a. swhlab image"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses to determine gain function.", "response": "def proto_01_12_steps025(abf=exampleABF):\n \"\"\"IC steps. Use to determine gain function.\"\"\"\n swhlab.ap.detect(abf)\n standard_groupingForInj(abf,200)\n\n for feature in ['freq','downslope']:\n swhlab.ap.plot_values(abf,feature,continuous=False) #plot AP info\n swhlab.plot.save(abf,tag='A_'+feature)\n\n swhlab.plot.gain(abf) #easy way to do a gain function!\n swhlab.plot.save(abf,tag='05-gain')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef proto_02_03_IVfast(abf=exampleABF):\n av1,sd1=swhlab.plot.IV(abf,.6,.9,True)\n swhlab.plot.save(abf,tag='iv1')\n Xs=abf.clampValues(.6) #generate IV clamp values\n abf.saveThing([Xs,av1],'iv')", "response": "Generate fast sweeps 1 step per sweep for clean IV without fast currents."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrepeating membrane tests likely with drug added. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs. Maybe IPSCs.", "response": "def proto_04_01_MTmon70s2(abf=exampleABF):\n \"\"\"repeated membrane tests, likely with drug added. Maybe IPSCs.\"\"\"\n standard_inspect(abf)\n swhlab.memtest.memtest(abf)\n swhlab.memtest.checkSweep(abf)\n swhlab.plot.save(abf,tag='check',resize=False)\n swhlab.memtest.plot_standard4(abf)\n swhlab.plot.save(abf,tag='memtests')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef proto_VC_50_MT_IV(abf=exampleABF):\n swhlab.memtest.memtest(abf) #do membrane test on every sweep\n swhlab.memtest.checkSweep(abf) #see all MT values\n swhlab.plot.save(abf,tag='02-check',resize=False)\n\n av1,sd1=swhlab.plot.IV(abf,1.2,1.4,True,'b')\n swhlab.plot.save(abf,tag='iv')\n Xs=abf.clampValues(1.2) #generate IV clamp values\n abf.saveThing([Xs,av1],'01_iv')", "response": "combination of membrane test and IV steps."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nincreasing ramps in (? pA steps.", "response": "def proto_IC_ramp_gain(abf=exampleABF):\n \"\"\"increasing ramps in (?) pA steps.\"\"\"\n standard_inspect(abf)\n swhlab.ap.detect(abf)\n\n swhlab.ap.check_AP_raw(abf) #show overlayed first few APs\n swhlab.plot.save(abf,tag=\"01-raw\",resize=False)\n swhlab.ap.check_AP_deriv(abf) #show overlayed first few APs\n swhlab.plot.save(abf,tag=\"02-deriv\")\n swhlab.ap.check_AP_phase(abf) #show overlayed first few APs\n swhlab.plot.save(abf,tag=\"03-phase\")\n\n swhlab.ap.plot_values(abf,'freq',continuous=True) #plot AP info\n pylab.subplot(211)\n pylab.axhline(40,color='r',lw=2,ls=\"--\",alpha=.2)\n swhlab.plot.save(abf,tag='04-freq')\n\n swhlab.ap.plot_values(abf,'downslope',continuous=True) #plot AP info\n pylab.subplot(211)\n pylab.axhline(-100,color='r',lw=2,ls=\"--\",alpha=.2)\n swhlab.plot.save(abf,tag='04-downslope')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nindex images in folder.", "response": "def indexImages(folder,fname=\"index.html\"):\n \"\"\"OBSOLETE WAY TO INDEX A FOLDER.\"\"\" #TODO: REMOVE\n html=\"\"\n for item in glob.glob(folder+\"/*.*\"):\n if item.split(\".\")[-1] in ['jpg','png']:\n html+=\"

%s

\"%os.path.basename(item)\n html+=''%os.path.basename(item)\n html+='
'*10\n html+=\"\"\n f=open(folder+\"/\"+fname,'w')\n f.write(html)\n f.close\n print(\"indexed:\")\n print(\" \",os.path.abspath(folder+\"/\"+fname))\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef save(self, *args, **kwargs):\n current_activable_value = getattr(self, self.ACTIVATABLE_FIELD_NAME)\n is_active_changed = self.id is None or self.__original_activatable_value != current_activable_value\n self.__original_activatable_value = current_activable_value\n\n ret_val = super(BaseActivatableModel, self).save(*args, **kwargs)\n\n # Emit the signals for when the is_active flag is changed\n if is_active_changed:\n model_activations_changed.send(self.__class__, instance_ids=[self.id], is_active=current_activable_value)\n if self.activatable_field_updated:\n model_activations_updated.send(self.__class__, instance_ids=[self.id], is_active=current_activable_value)\n\n return ret_val", "response": "A custom save method that handles figuring out when something is activated or deactivated."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndeleting an activatable object.", "response": "def delete(self, force=False, **kwargs):\n \"\"\"\n It is impossible to delete an activatable model unless force is True. This function instead sets it to inactive.\n \"\"\"\n if force:\n return super(BaseActivatableModel, self).delete(**kwargs)\n else:\n setattr(self, self.ACTIVATABLE_FIELD_NAME, False)\n return self.save(update_fields=[self.ACTIVATABLE_FIELD_NAME])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites to file_handle if supplied othewise print output", "response": "def show(self, args, file_handle=None, **kwargs):\n \"Write to file_handle if supplied, othewise print output\"\n full_string = ''\n info = {'root_directory': '',\n 'batch_name': '',\n 'batch_tag': '',\n 'batch_description': '',\n 'launcher': '',\n 'timestamp_format': '',\n 'timestamp': tuple(time.localtime()),\n 'varying_keys': args.varying_keys,\n 'constant_keys': args.constant_keys,\n 'constant_items': args.constant_items}\n\n quoted_cmds = [ subprocess.list2cmdline(\n [el for el in self(self._formatter(s),'',info)])\n for s in args.specs]\n\n cmd_lines = ['%d: %s\\n' % (i, qcmds) for (i,qcmds)\n in enumerate(quoted_cmds)]\n full_string += ''.join(cmd_lines)\n if file_handle:\n file_handle.write(full_string)\n file_handle.flush()\n else:\n print(full_string)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update(self):\n launches = []\n for path in os.listdir(self.output_dir):\n full_path = os.path.join(self.output_dir, path)\n if os.path.isdir(full_path):\n launches.append(self._get_launch_info(full_path))\n self.launches = sorted(launches)", "response": "Update the launch information for the current user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the root directory name given the timestamp.", "response": "def get_root_directory(self, timestamp=None):\n \"\"\"\n A helper method that supplies the root directory name given a\n timestamp.\n \"\"\"\n if timestamp is None: timestamp = self.timestamp\n if self.timestamp_format is not None:\n root_name = (time.strftime(self.timestamp_format, timestamp)\n + '-' + self.batch_name)\n else:\n root_name = self.batch_name\n\n path = os.path.join(self.output_directory,\n *(self.subdir+[root_name]))\n return os.path.abspath(path)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _append_log(self, specs):\n self._spec_log += specs # This should be removed\n log_path = os.path.join(self.root_directory, (\"%s.log\" % self.batch_name))\n core.Log.write_log(log_path, [spec for (_, spec) in specs], allow_append=True)", "response": "Append the log to the log file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _record_info(self, setup_info=None):\n info_path = os.path.join(self.root_directory, ('%s.info' % self.batch_name))\n\n if setup_info is None:\n try:\n with open(info_path, 'r') as info_file:\n setup_info = json.load(info_file)\n except:\n setup_info = {}\n\n setup_info.update({'end_time' : tuple(time.localtime())})\n else:\n setup_info.update({\n 'end_time' : None,\n 'metadata' : self.metadata\n })\n\n with open(info_path, 'w') as info_file:\n json.dump(setup_info, info_file, sort_keys=True, indent=4)", "response": "Record the current state of the cluster."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints a succinct summary of the Launcher configuration.", "response": "def summary(self):\n \"\"\"\n A succinct summary of the Launcher configuration. Unlike the\n repr, a summary does not have to be complete but must supply\n key information relevant to the user.\n \"\"\"\n print(\"Type: %s\" % self.__class__.__name__)\n print(\"Batch Name: %r\" % self.batch_name)\n if self.tag:\n print(\"Tag: %s\" % self.tag)\n print(\"Root directory: %r\" % self.get_root_directory())\n print(\"Maximum concurrency: %s\" % self.max_concurrency)\n if self.description:\n print(\"Description: %s\" % self.description)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrun qdel command to remove all remaining jobs using the job_timestamp of the current job. Returns exit - code of qdel.", "response": "def qdel_batch(self):\n \"\"\"\n Runs qdel command to remove all remaining queued jobs using\n the * pattern . Necessary when StopIteration is\n raised with scheduled jobs left on the queue.\n Returns exit-code of qdel.\n \"\"\"\n p = subprocess.Popen(['qdel', '%s_%s*' % (self.batch_name,\n self.job_timestamp)],\n stdout=subprocess.PIPE)\n (stdout, stderr) = p.communicate()\n return p.poll()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _launch_process_group(self, process_commands, streams_path):\n processes = []\n for cmd, tid in process_commands:\n job_timestamp = time.strftime('%H%M%S')\n basename = \"%s_%s_tid_%d\" % (self.batch_name, job_timestamp, tid)\n stdout_path = os.path.join(streams_path, \"%s.o.%d\" % (basename, tid))\n stderr_path = os.path.join(streams_path, \"%s.e.%d\" % (basename, tid))\n process = { 'tid' : tid,\n 'cmd' : cmd,\n 'stdout' : stdout_path,\n 'stderr' : stderr_path }\n processes.append(process)\n\n # To make the JSON filename unique per group, we use the last tid in\n # this group.\n json_path = os.path.join(self.root_directory, self.json_name % (tid))\n with open(json_path, 'w') as json_file:\n json.dump(processes, json_file, sort_keys=True, indent=4)\n\n p = subprocess.Popen([self.script_path, json_path, self.batch_name,\n str(len(processes)), str(self.max_concurrency)])\n if p.wait() != 0:\n raise EnvironmentError(\"Script command exit with code: %d\" % p.poll())", "response": "Launches the process group."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cross_check_launchers(self, launchers):\n if len(launchers) == 0: raise Exception('Empty launcher list')\n timestamps = [launcher.timestamp for launcher in launchers]\n\n if not all(timestamps[0] == tstamp for tstamp in timestamps):\n raise Exception(\"Launcher timestamps not all equal. \"\n \"Consider setting timestamp explicitly.\")\n\n root_directories = []\n for launcher in launchers:\n command = launcher.command\n args = launcher.args\n command.verify(args)\n root_directory = launcher.get_root_directory()\n if os.path.isdir(root_directory):\n raise Exception(\"Root directory already exists: %r\" % root_directory)\n if root_directory in root_directories:\n raise Exception(\"Each launcher requires a unique root directory\")\n root_directories.append(root_directory)", "response": "Cross - checks the launchers and returns a list of root directories that are needed for the launcher."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _launch_all(self, launchers):\n for launcher in launchers:\n print(\"== Launching %s ==\" % launcher.batch_name)\n launcher()\n return True", "response": "Launches all available launchers."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _review_all(self, launchers):\n # Run review of launch args if necessary\n if self.launch_args is not None:\n proceed = self.review_args(self.launch_args,\n show_repr=True,\n heading='Meta Arguments')\n if not proceed: return False\n\n reviewers = [self.review_args,\n self.review_command,\n self.review_launcher]\n\n for (count, launcher) in enumerate(launchers):\n\n # Run reviews for all launchers if desired...\n if not all(reviewer(launcher) for reviewer in reviewers):\n print(\"\\n == Aborting launch ==\")\n return False\n # But allow the user to skip these extra reviews\n if len(launchers)!= 1 and count < len(launchers)-1:\n skip_remaining = self.input_options(['Y', 'n','quit'],\n '\\nSkip remaining reviews?', default='y')\n\n if skip_remaining == 'y': break\n elif skip_remaining == 'quit': return False\n\n if self.input_options(['y','N'], 'Execute?', default='n') != 'y':\n return False\n else:\n return self._launch_all(launchers)", "response": "Runs the review process for all the launchers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef input_options(self, options, prompt='Select option', default=None):\n check_options = [x.lower() for x in options]\n while True:\n response = input('%s [%s]: ' % (prompt, ', '.join(options))).lower()\n if response in check_options: return response.strip()\n elif response == '' and default is not None:\n return default.lower().strip()", "response": "Helper to prompt user for input on the commandline."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsave the data to a file.", "response": "def save(self, filename, metadata={}, **data):\n \"\"\"\n The implementation in the base class simply checks there is no\n clash between the metadata and data keys.\n \"\"\"\n intersection = set(metadata.keys()) & set(data.keys())\n if intersection:\n msg = 'Key(s) overlap between data and metadata: %s'\n raise Exception(msg % ','.join(intersection))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the full path for saving the file.", "response": "def _savepath(self, filename):\n \"\"\"\n Returns the full path for saving the file, adding an extension\n and making the filename unique as necessary.\n \"\"\"\n (basename, ext) = os.path.splitext(filename)\n basename = basename if (ext in self.extensions) else filename\n ext = ext if (ext in self.extensions) else self.extensions[0]\n savepath = os.path.abspath(os.path.join(self.directory,\n '%s%s' % (basename, ext)))\n return (tempfile.mkstemp(ext, basename + \"_\", self.directory)[1]\n if self.hash_suffix else savepath)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a boolean indicating whether the filename has an anonacus related extension for this class.", "response": "def file_supported(cls, filename):\n \"\"\"\n Returns a boolean indicating whether the filename has an\n appropriate extension for this class.\n \"\"\"\n if not isinstance(filename, str):\n return False\n (_, ext) = os.path.splitext(filename)\n if ext not in cls.extensions:\n return False\n else:\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving the data in the specified file.", "response": "def save(self, filename, imdata, **data):\n \"\"\"\n Data may be either a PIL Image object or a Numpy array.\n \"\"\"\n if isinstance(imdata, numpy.ndarray):\n imdata = Image.fromarray(numpy.uint8(imdata))\n elif isinstance(imdata, Image.Image):\n imdata.save(self._savepath(filename))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the timestamp of the file modified", "response": "def fileModifiedTimestamp(fname):\r\n \"\"\"return \"YYYY-MM-DD\" when the file was modified.\"\"\"\r\n modifiedTime=os.path.getmtime(fname)\r\n stamp=time.strftime('%Y-%m-%d', time.localtime(modifiedTime))\r\n return stamp"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the results file into a dict of active folders with days as keys.", "response": "def loadResults(resultsFile):\r\n \"\"\"returns a dict of active folders with days as keys.\"\"\"\r\n with open(resultsFile) as f:\r\n raw=f.read().split(\"\\n\")\r\n foldersByDay={}\r\n for line in raw:\r\n folder=line.split('\"')[1]+\"\\\\\"\r\n line=[]+line.split('\"')[2].split(\", \")\r\n for day in line[1:]:\r\n if not day in foldersByDay:\r\n foldersByDay[day]=[]\r\n foldersByDay[day]=foldersByDay[day]+[folder]\r\n nActiveDays=len(foldersByDay)\r\n dayFirst=sorted(foldersByDay.keys())[0]\r\n dayLast=sorted(foldersByDay.keys())[-1]\r\n dayFirst=datetime.datetime.strptime(dayFirst, \"%Y-%m-%d\" )\r\n dayLast=datetime.datetime.strptime(dayLast, \"%Y-%m-%d\" )\r\n nDays = (dayLast - dayFirst).days + 1\r\n emptyDays=0\r\n for deltaDays in range(nDays):\r\n day=dayFirst+datetime.timedelta(days=deltaDays)\r\n stamp=datetime.datetime.strftime(day, \"%Y-%m-%d\" )\r\n if not stamp in foldersByDay:\r\n foldersByDay[stamp]=[]\r\n emptyDays+=1\r\n percActive=nActiveDays/nDays*100\r\n print(\"%d of %d days were active (%.02f%%)\"%(nActiveDays,nDays,percActive))\r\n return foldersByDay"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef HTML_results(resultsFile):\r\n foldersByDay=loadResults(resultsFile)\r\n\r\n # optionally skip dates before a certain date\r\n# for day in sorted(list(foldersByDay.keys())):\r\n# if time.strptime(day,\"%Y-%m-%d\")Active Folder Report (updated TIMESTAMP)\"\r\n html+=\"
  • When a file is created (or modified) its parent folder is marked active for that day.\"\r\n html+=\"
  • This page reports all folders which were active in the last several years. \"\r\n html+=\"
  • A single folder can be active for more than one date.\"\r\n html=html.replace(\"TIMESTAMP\",(time.strftime('%Y-%m-%d', time.localtime())))\r\n html+=\"
    \"*5\r\n\r\n\r\n # create menu at the top of the page\r\n html+=\"
    Active Folder Dates
    \"\r\n html+=\"\"\r\n lastMonth=\"\"\r\n lastYear=\"\"\r\n for day in sorted(list(foldersByDay.keys())):\r\n month=day[:7]\r\n year=day[:4]\r\n if year!=lastYear:\r\n html+=\"

    %s \"%year\r\n lastYear=year\r\n if month!=lastMonth:\r\n html+=\"
    %s: \"%month\r\n lastMonth=month\r\n html+=\"%s, \"%(day,day[8:])\r\n html+=\"
    \"*5\r\n html=html.replace(\",
    \",\"
    \")\r\n html+=\"    \"\r\n\r\n # create the full list of folders organized by active date\r\n html+=\"
    Active Folders
    \"\r\n for day in sorted(list(foldersByDay.keys())):\r\n dt=datetime.datetime.strptime(day, \"%Y-%m-%d\" )\r\n classPrefix=\"weekday\"\r\n if int(dt.weekday())>4:\r\n classPrefix=\"weekend\"\r\n html+=\"\"%(day,day)\r\n title=\"%s (%s)\"%(day,DAYSOFWEEK[dt.weekday()])\r\n html+=\"
    %s
    \"%(classPrefix,title)\r\n html+=\"
    \"%(classPrefix)\r\n # define folders to skip\r\n for folder in foldersByDay[day]:\r\n if \"\\\\References\\\\\" in folder:\r\n continue\r\n if \"\\\\MIP\\\\\" in folder:\r\n continue\r\n if \"LineScan-\" and \"\\\\analysis\\\\\" in folder:\r\n continue\r\n if \"trakem2\" in folder:\r\n continue\r\n if \"SWHlab-\" in folder:\r\n continue\r\n if \"\\\\swhlab\" in folder:\r\n continue\r\n html+=\"%s
    \"%folder\r\n html+=\"
    \"\r\n fnameSave=resultsFile+\".html\"\r\n html=html.replace(\"D:\\\\X_Drive\\\\\",\"X:\\\\\")\r\n with open(fnameSave,'w') as f:\r\n f.write(HTML_TEMPLATE.replace(\"\",\"\"+html))\r\n print(\"saved\",fnameSave)", "response": "generates HTML report of active folders and days."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving some data Y break it into chunks and return just the quiet ones.", "response": "def quietParts(data,percentile=10):\n \"\"\"\n Given some data (Y) break it into chunks and return just the quiet ones.\n Returns data where the variance for its chunk size is below the given percentile.\n CHUNK_POINTS should be adjusted so it's about 10ms of data.\n \"\"\"\n nChunks=int(len(Y)/CHUNK_POINTS)\n chunks=np.reshape(Y[:nChunks*CHUNK_POINTS],(nChunks,CHUNK_POINTS))\n variances=np.var(chunks,axis=1)\n percentiles=np.empty(len(variances))\n for i,variance in enumerate(variances):\n percentiles[i]=sorted(variances).index(variance)/len(variances)*100\n selected=chunks[np.where(percentiles<=percentile)[0]].flatten()\n return selected"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef ndist(data,Xs):\n sigma=np.sqrt(np.var(data))\n center=np.average(data)\n curve=mlab.normpdf(Xs,center,sigma)\n curve*=len(data)*HIST_RESOLUTION\n return curve", "response": "return the normal\n distribution curve as a Y point at each of those Xs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef abfinfo(self,printToo=False,returnDict=False):\n info=\"\\n### ABF INFO ###\\n\"\n d={}\n for thingName in sorted(dir(self)):\n if thingName in ['cm','evIs','colormap','dataX','dataY',\n 'protoX','protoY']:\n continue\n if \"_\" in thingName:\n continue\n thing=getattr(self,thingName)\n if type(thing) is list and len(thing)>5:\n continue\n thingType=str(type(thing)).split(\"'\")[1]\n if \"method\" in thingType or \"neo.\" in thingType:\n continue\n if thingName in [\"header\",\"MT\"]:\n continue\n info+=\"%s <%s> %s\\n\"%(thingName,thingType,thing)\n d[thingName]=thing\n if printToo:\n print()\n for line in info.split(\"\\n\"):\n if len(line)<3:\n continue\n print(\" \",line)\n print()\n if returnDict:\n return d\n return info", "response": "show basic info about ABF class variables."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef headerHTML(self,fname=None):\n if fname is None:\n fname = self.fname.replace(\".abf\",\"_header.html\")\n html=\"\"\n html+=\"

    abfinfo() for %s.abf

    \"%self.ID\n html+=self.abfinfo().replace(\"<\",\"<\").replace(\">\",\">\").replace(\"\\n\",\"
    \")\n html+=\"

    Header for %s.abf

    \"%self.ID\n html+=pprint.pformat(self.header, indent=1)\n html=html.replace(\"\\n\",'
    ').replace(\" \",\" \")\n html=html.replace(r\"\\x00\",\"\")\n html+=\"    \"\n print(\"WRITING HEADER TO:\")\n print(fname)\n f=open(fname,'w')\n f.write(html)\n f.close()", "response": "read the ABF header and save it HTML formatted."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate colormap for all sweeps", "response": "def generate_colormap(self,colormap=None,reverse=False):\n \"\"\"use 1 colormap for the whole abf. You can change it!.\"\"\"\n if colormap is None:\n colormap = pylab.cm.Dark2\n self.cm=colormap\n self.colormap=[]\n for i in range(self.sweeps): #TODO: make this the only colormap\n self.colormap.append(colormap(i/self.sweeps))\n if reverse:\n self.colormap.reverse()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setSweep(self,sweep=0,force=False):\n if sweep is None or sweep is False:\n sweep=0\n if sweep<0:\n sweep=self.sweeps-sweep #-1 means last sweep\n if sweep<0: #still!\n sweep=0 #first sweep\n if sweep>(self.sweeps-1):\n print(\" !! there aren't %d sweeps. Reverting to last (%d) sweep.\"%(sweep,self.sweeps-1))\n sweep=self.sweeps-1\n sweep=int(sweep)\n try:\n if self.currentSweep==sweep and force==False:\n return\n self.currentSweep=sweep\n self.dataY = self.block.segments[sweep].analogsignals[self.channel]\n self.dataY = np.array(self.dataY)\n\n B1,B2=self.baseline\n if B1==None:\n B1=0\n else:\n B1=B1*self.rate\n if B2==None:\n B2==self.sweepSize\n else:\n B2=B2*self.rate\n\n self.dataY-=np.average(self.dataY[self.baseline[0]*self.rate:self.baseline[1]*self.rate])\n self.sweep_genXs()\n self.sweep_decimate()\n self.generate_protocol(sweep=sweep)\n self.dataStart = self.sweepInterval*self.currentSweep\n except Exception:\n print(\"#\"*400,\"\\n\",traceback.format_exc(),'\\n',\"#\"*400)\n return self.dataX,self.dataY", "response": "Load X and Y data for a particular sweep."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate sweepX to match sweepY", "response": "def sweep_genXs(self):\n \"\"\"generate sweepX (in seconds) to match sweepY\"\"\"\n if self.decimateMethod:\n self.dataX=np.arange(len(self.dataY))/self.rate\n self.dataX*=self.decimateBy\n return\n if self.dataX is None or len(self.dataX)!=len(self.dataY):\n self.dataX=np.arange(len(self.dataY))/self.rate"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndecimate data using one of the following methods: 'avg','max','min','fast' They're self explainatory. 'fast' just plucks the n'th data point.", "response": "def sweep_decimate(self):\n \"\"\"\n decimate data using one of the following methods:\n 'avg','max','min','fast'\n They're self explainatory. 'fast' just plucks the n'th data point.\n \"\"\"\n if len(self.dataY)1:\n self.setSweep(0)\n P0=str(self.protoX)+str(self.protoY)\n self.setSweep(1)\n P1=str(self.protoX)+str(self.protoY)\n if not P0==P1:\n command=\"steps\"\n tags=\"notag\"\n if len(self.commentSweeps):\n tags=\"drugs\"\n ch=\"1ch\"\n if self.nADC>1:\n ch=\"2ch\"\n guess=\"-\".join([clamp,command,tags,ch])\n return guess"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving an array of sweeps, return X,Y,Err average. This returns *SWEEPS* of data, not just 1 data point.", "response": "def average_sweep(self,T1=0,T2=None,sweeps=None,stdErr=False):\n \"\"\"\n given an array of sweeps, return X,Y,Err average.\n This returns *SWEEPS* of data, not just 1 data point.\n \"\"\"\n T1=T1*self.rate\n if T2 is None:\n T2 = self.sweepSize-1\n else:\n T2 = T2*self.rate\n if sweeps is None:\n sweeps = range(self.sweeps)\n Ys=np.empty((len(sweeps),(T2-T1)))\n for i in range(len(sweeps)):\n self.setSweep(sweeps[i])\n Ys[i]=self.dataY[T1:T2]\n Av = np.average(Ys,0)\n Es = np.std(Ys,0)\n Xs = self.dataX[T1:T2]\n if stdErr: #otherwise return stdev\n Es = Es/np.sqrt(len(sweeps))\n return Xs,Av,Es"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a numpy array of average data for every sweep", "response": "def average_data(self,ranges=[[None,None]],percentile=None):\n \"\"\"\n given a list of ranges, return single point averages for every sweep.\n Units are in seconds. Expects something like:\n ranges=[[1,2],[4,5],[7,7.5]]\n None values will be replaced with maximum/minimum bounds.\n For baseline subtraction, make a range baseline then sub it youtself.\n returns datas[iSweep][iRange][AVorSD]\n if a percentile is given, return that percentile rather than average.\n percentile=50 is the median, but requires sorting, and is slower.\n \"\"\"\n ranges=copy.deepcopy(ranges) #TODO: make this cleaner. Why needed?\n # clean up ranges, make them indexes\n for i in range(len(ranges)):\n if ranges[i][0] is None:\n ranges[i][0] = 0\n else:\n ranges[i][0] = int(ranges[i][0]*self.rate)\n if ranges[i][1] is None:\n ranges[i][1] = -1\n else:\n ranges[i][1] = int(ranges[i][1]*self.rate)\n\n # do the math\n datas=np.empty((self.sweeps,len(ranges),2)) #[sweep][range]=[Av,Er]\n for iSweep in range(self.sweeps):\n self.setSweep(iSweep)\n for iRange in range(len(ranges)):\n I1=ranges[iRange][0]\n I2=ranges[iRange][1]\n if percentile:\n datas[iSweep][iRange][0]=np.percentile(self.dataY[I1:I2],percentile)\n else:\n datas[iSweep][iRange][0]=np.average(self.dataY[I1:I2])\n datas[iSweep][iRange][1]=np.std(self.dataY[I1:I2])\n return datas"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter_gaussian(self,sigmaMs=100,applyFiltered=False,applyBaseline=False):\n if sigmaMs==0:\n return self.dataY\n filtered=cm.filter_gaussian(self.dataY,sigmaMs)\n if applyBaseline:\n self.dataY=self.dataY-filtered\n elif applyFiltered:\n self.dataY=filtered\n else:\n return filtered", "response": "RETURNS filtered trace. Desn t filter it in place."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving any object as a file", "response": "def saveThing(self,thing,fname,overwrite=True,ext=\".pkl\"):\n \"\"\"save any object as /swhlab4/ID_[fname].pkl\"\"\"\n if not os.path.exists(os.path.dirname(self.outpre)):\n os.mkdir(os.path.dirname(self.outpre))\n if ext and not ext in fname:\n fname+=ext\n fname=self.outpre+fname\n if overwrite is False:\n if os.path.exists(fname):\n print(\" o- not overwriting [%s]\"%os.path.basename(fname))\n return\n time1=cm.timethis()\n pickle.dump(thing, open(fname,\"wb\"),pickle.HIGHEST_PROTOCOL)\n print(\" <- saving [%s] %s (%.01f kB) took %.02f ms\"%(\\\n os.path.basename(fname),str(type(thing)),\n sys.getsizeof(pickle.dumps(thing, -1))/1e3,\n cm.timethis(time1)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading any object from a file", "response": "def loadThing(self,fname,ext=\".pkl\"):\n \"\"\"save any object from /swhlab4/ID_[fname].pkl\"\"\"\n if ext and not ext in fname:\n fname+=ext\n fname=self.outpre+fname\n time1=cm.timethis()\n thing = pickle.load(open(fname,\"rb\"))\n print(\" -> loading [%s] (%.01f kB) took %.02f ms\"%(\\\n os.path.basename(fname),\n sys.getsizeof(pickle.dumps(thing, -1))/1e3,\n cm.timethis(time1)))\n return thing"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes all the files in the system", "response": "def deleteStuff(self,ext=\"*\",spareInfo=True,spare=[\"_info.pkl\"]):\n \"\"\"delete /swhlab4/ID_*\"\"\"\n print(\" -- deleting /swhlab4/\"+ext)\n for fname in sorted(glob.glob(self.outpre+ext)):\n reallyDelete=True\n for item in spare:\n if item in fname:\n reallyDelete=False\n if reallyDelete:\n os.remove(fname)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_activatable_models():\n for model in get_activatable_models():\n # Verify the activatable model has an activatable boolean field\n activatable_field = next((\n f for f in model._meta.fields\n if f.__class__ == models.BooleanField and f.name == model.ACTIVATABLE_FIELD_NAME\n ), None)\n if activatable_field is None:\n raise ValidationError((\n 'Model {0} is an activatable model. It must define an activatable BooleanField that '\n 'has a field name of model.ACTIVATABLE_FIELD_NAME (which defaults to is_active)'.format(model)\n ))\n\n # Ensure all foreign keys and onetoone fields will not result in cascade deletions if not cascade deletable\n if not model.ALLOW_CASCADE_DELETE:\n for field in model._meta.fields:\n if field.__class__ in (models.ForeignKey, models.OneToOneField):\n if field.remote_field.on_delete == models.CASCADE:\n raise ValidationError((\n 'Model {0} is an activatable model. All ForeignKey and OneToOneFields '\n 'must set on_delete methods to something other than CASCADE (the default). '\n 'If you want to explicitely allow cascade deletes, then you must set the '\n 'ALLOW_CASCADE_DELETE=True class variable on your model.'\n ).format(model))", "response": "Validates that the ActivatableModel s fields are not None and that all foreign keys and OneToOneFields that will be deleted will not cause cascading deletions to occur."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_table(args, vdims=[]):\n \"Helper function to convet an Args object to a HoloViews Table\"\n if not Table:\n return \"HoloViews Table not available\"\n kdims = [dim for dim in args.constant_keys + args.varying_keys\n if dim not in vdims]\n items = [tuple([spec[k] for k in kdims+vdims])\n for spec in args.specs]\n return Table(items, kdims=kdims, vdims=vdims)", "response": "Helper function to convet an Args object to a HoloViews Table"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pprint_args(self, pos_args, keyword_args, infix_operator=None, extra_params={}):\n if infix_operator and not (len(pos_args)==2 and keyword_args==[]):\n raise Exception('Infix format requires exactly two'\n ' positional arguments and no keywords')\n (kwargs,_,_,_) = self._pprint_args\n self._pprint_args = (keyword_args + kwargs, pos_args, infix_operator, extra_params)", "response": "Method to define the positional arguments and keyword order\n for pretty printing."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _pprint(self, cycle=False, flat=False, annotate=False, onlychanged=True, level=1, tab = ' '):\n (kwargs, pos_args, infix_operator, extra_params) = self._pprint_args\n (br, indent) = ('' if flat else '\\n', '' if flat else tab * level)\n prettify = lambda x: isinstance(x, PrettyPrinted) and not flat\n pretty = lambda x: x._pprint(flat=flat, level=level+1) if prettify(x) else repr(x)\n\n params = dict(self.get_param_values())\n show_lexsort = getattr(self, '_lexorder', None) is not None\n modified = [k for (k,v) in self.get_param_values(onlychanged=onlychanged)]\n pkwargs = [(k, params[k]) for k in kwargs if (k in modified)] + list(extra_params.items())\n arg_list = [(k,params[k]) for k in pos_args] + pkwargs\n\n lines = []\n if annotate: # Optional annotating comment\n len_ckeys, len_vkeys = len(self.constant_keys), len(self.varying_keys)\n info_triple = (len(self),\n ', %d constant key(s)' % len_ckeys if len_ckeys else '',\n ', %d varying key(s)' % len_vkeys if len_vkeys else '')\n annotation = '# == %d items%s%s ==\\n' % info_triple\n lines = [annotation]\n\n if show_lexsort: lines.append('(')\n if cycle:\n lines.append('%s(...)' % self.__class__.__name__)\n elif infix_operator:\n level = level - 1\n triple = (pretty(params[pos_args[0]]), infix_operator, pretty(params[pos_args[1]]))\n lines.append('%s %s %s' % triple)\n else:\n lines.append('%s(' % self.__class__.__name__)\n for (k,v) in arg_list:\n lines.append('%s%s=%s' % (br+indent, k, pretty(v)))\n lines.append(',')\n lines = lines[:-1] +[br+(tab*(level-1))+')'] # Remove trailing comma\n\n if show_lexsort:\n lines.append(').lexsort(%s)' % ', '.join(repr(el) for el in self._lexorder))\n return ''.join(lines)", "response": "Pretty printer that prints all the modified keywords and the modified keywords and optionally annotates the top of the repr with a comment."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nformat the elements of an argument set appropriately", "response": "def spec_formatter(cls, spec):\n \" Formats the elements of an argument set appropriately\"\n return type(spec)((k, str(v)) for (k,v) in spec.items())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _collect_by_key(self,specs):\n # Collect (key, value) tuples as list of lists, flatten with chain\n allkeys = itertools.chain.from_iterable(\n [[(k, run[k]) for k in run] for run in specs])\n collection = defaultdict(list)\n for (k,v) in allkeys: collection[k].append(v)\n return collection", "response": "Collect the keys of the log entries in the specs dictionary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Cartesian product of the two sets of specifications.", "response": "def _cartesian_product(self, first_specs, second_specs):\n \"\"\"\n Takes the Cartesian product of the specifications. Result will\n contain N specifications where N = len(first_specs) *\n len(second_specs) and keys are merged.\n Example: [{'a':1},{'b':2}] * [{'c':3},{'d':4}] =\n [{'a':1,'c':3},{'a':1,'d':4},{'b':2,'c':3},{'b':2,'d':4}]\n \"\"\"\n return [ dict(zip(\n list(s1.keys()) + list(s2.keys()),\n list(s1.values()) + list(s2.values())\n ))\n for s1 in first_specs for s2 in second_specs ]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef summary(self):\n print(\"Items: %s\" % len(self))\n varying_keys = ', '.join('%r' % k for k in self.varying_keys)\n print(\"Varying Keys: %s\" % varying_keys)\n items = ', '.join(['%s=%r' % (k,v)\n for (k,v) in self.constant_items])\n if self.constant_items:\n print(\"Constant Items: %s\" % items)", "response": "A succinct summary of the argument specifier."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _build_specs(self, specs, kwargs, fp_precision):\n if specs is None:\n overrides = param.ParamOverrides(self, kwargs,\n allow_extra_keywords=True)\n extra_kwargs = overrides.extra_keywords()\n kwargs = dict([(k,v) for (k,v) in kwargs.items()\n if k not in extra_kwargs])\n rounded_specs = list(self.round_floats([extra_kwargs],\n fp_precision))\n\n if extra_kwargs=={}: return [], kwargs, True\n else: return rounded_specs, kwargs, False\n\n return list(self.round_floats(specs, fp_precision)), kwargs, True", "response": "Builds the list of specs and kwargs for the resource."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns unique items in a sequence.", "response": "def _unique(self, sequence, idfun=repr):\n \"\"\"\n Note: repr() must be implemented properly on all objects. This\n is implicitly assumed by Lancet when Python objects need to be\n formatted to string representation.\n \"\"\"\n seen = {}\n return [seen.setdefault(idfun(e),e) for e in sequence\n if idfun(e) not in seen]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprint the available argument values in the human - readable format.", "response": "def show(self, exclude=[]):\n \"\"\"\n Convenience method to inspect the available argument values in\n human-readable format. The ordering of keys is determined by\n how quickly they vary.\n\n The exclude list allows specific keys to be excluded for\n readability (e.g. to hide long, absolute filenames).\n \"\"\"\n ordering = self.constant_keys + self.varying_keys\n spec_lines = [', '.join(['%s=%s' % (k, s[k]) for k in ordering\n if (k in s) and (k not in exclude)])\n for s in self.specs]\n print('\\n'.join(['%d: %s' % (i,l) for (i,l) in enumerate(spec_lines)]))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef lexsort(self, *order):\n if order == []:\n raise Exception(\"Please specify the keys for sorting, use\"\n \"'+' prefix for ascending,\"\n \"'-' for descending.)\")\n\n if not set(el[1:] for el in order).issubset(set(self.varying_keys)):\n raise Exception(\"Key(s) specified not in the set of varying keys.\")\n\n sorted_args = copy.deepcopy(self)\n specs_param = sorted_args.params('specs')\n specs_param.constant = False\n sorted_args.specs = self._lexsorted_specs(order)\n specs_param.constant = True\n sorted_args._lexorder = order\n return sorted_args", "response": "Returns a new instance of the command with the same arguments as this one but with the same arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _lexsorted_specs(self, order):\n specs = self.specs[:]\n if not all(el[0] in ['+', '-'] for el in order):\n raise Exception(\"Please specify the keys for sorting, use\"\n \"'+' prefix for ascending,\"\n \"'-' for descending.)\")\n\n sort_cycles = [(el[1:], True if el[0]=='+' else False)\n for el in reversed(order)\n if el[1:] in self.varying_keys]\n\n for (key, ascending) in sort_cycles:\n specs = sorted(specs, key=lambda s: s.get(key, None),\n reverse=(not ascending))\n return specs", "response": "Return a list of the keys that are sorted in the given order."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef extract_log(log_path, dict_type=dict):\n log_path = (log_path if os.path.isfile(log_path)\n else os.path.join(os.getcwd(), log_path))\n with open(log_path,'r') as log:\n splits = (line.split() for line in log)\n uzipped = ((int(split[0]), json.loads(\" \".join(split[1:]))) for split in splits)\n szipped = [(i, dict((str(k),v) for (k,v) in d.items())) for (i,d) in uzipped]\n return dict_type(szipped)", "response": "Extracts the log file generated by a launcher and returns the dictionary with tid keys and specification values."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_log(log_path, data, allow_append=True):\n append = os.path.isfile(log_path)\n islist = isinstance(data, list)\n\n if append and not allow_append:\n raise Exception('Appending has been disabled'\n ' and file %s exists' % log_path)\n\n if not (islist or isinstance(data, Args)):\n raise Exception('Can only write Args objects or dictionary'\n ' lists to log file.')\n\n specs = data if islist else data.specs\n if not all(isinstance(el,dict) for el in specs):\n raise Exception('List elements must be dictionaries.')\n\n log_file = open(log_path, 'r+') if append else open(log_path, 'w')\n start = int(log_file.readlines()[-1].split()[0])+1 if append else 0\n ascending_indices = range(start, start+len(data))\n\n log_str = '\\n'.join(['%d %s' % (tid, json.dumps(el))\n for (tid, el) in zip(ascending_indices,specs)])\n log_file.write(\"\\n\"+log_str if append else log_str)\n log_file.close()", "response": "Writes the supplied data to the log file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading all the files in a given directory selecting only files with the given extension if specified.", "response": "def directory(cls, directory, root=None, extension=None, **kwargs):\n \"\"\"\n Load all the files in a given directory selecting only files\n with the given extension if specified. The given kwargs are\n passed through to the normal constructor.\n \"\"\"\n root = os.getcwd() if root is None else root\n suffix = '' if extension is None else '.' + extension.rsplit('.')[-1]\n pattern = directory + os.sep + '*' + suffix\n key = os.path.join(root, directory,'*').rsplit(os.sep)[-2]\n format_parse = list(string.Formatter().parse(key))\n if not all([el is None for el in zip(*format_parse)[1]]):\n raise Exception('Directory cannot contain format field specifications')\n return cls(key, pattern, root, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fields(self):\n parse = list(string.Formatter().parse(self.pattern))\n return [f for f in zip(*parse)[1] if f is not None]", "response": "Return the fields specified in the pattern using Python s\n formatting mini - language."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload the files that match the given pattern.", "response": "def _load_expansion(self, key, root, pattern):\n \"\"\"\n Loads the files that match the given pattern.\n \"\"\"\n path_pattern = os.path.join(root, pattern)\n expanded_paths = self._expand_pattern(path_pattern)\n\n specs=[]\n for (path, tags) in expanded_paths:\n filelist = [os.path.join(path,f) for f in os.listdir(path)] if os.path.isdir(path) else [path]\n for filepath in filelist:\n specs.append(dict(tags,**{key:os.path.abspath(filepath)}))\n\n return sorted(specs, key=lambda s: s[key])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _expand_pattern(self, pattern):\n (globpattern, regexp, fields, types) = self._decompose_pattern(pattern)\n filelist = glob.glob(globpattern)\n expansion = []\n\n for fname in filelist:\n if fields == []:\n expansion.append((fname, {}))\n continue\n match = re.match(regexp, fname)\n if match is None: continue\n match_items = match.groupdict().items()\n tags = dict((k,types.get(k, str)(v)) for (k,v) in match_items)\n expansion.append((fname, tags))\n\n return expansion", "response": "Given a pattern expands the pattern into a list of absolute paths and returns a list of dicts where the keys are the absolute paths of the files and the values are the tags of the files."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a path pattern with format declaration generates a four - tuple", "response": "def _decompose_pattern(self, pattern):\n \"\"\"\n Given a path pattern with format declaration, generates a\n four-tuple (glob_pattern, regexp pattern, fields, type map)\n \"\"\"\n sep = '~lancet~sep~'\n float_codes = ['e','E','f', 'F','g', 'G', 'n']\n typecodes = dict([(k,float) for k in float_codes]\n + [('b',bin), ('d',int), ('o',oct), ('x',hex)])\n parse = list(string.Formatter().parse(pattern))\n text, fields, codes, _ = zip(*parse)\n\n # Finding the field types from format string\n types = []\n for (field, code) in zip(fields, codes):\n if code in ['', None]: continue\n constructor = typecodes.get(code[-1], None)\n if constructor: types += [(field, constructor)]\n\n stars = ['' if not f else '*' for f in fields]\n globpat = ''.join(text+star for (text,star) in zip(text,stars))\n\n refields = ['' if not f else sep+('(?P<%s>.*?)'% f)+sep for f in fields]\n parts = ''.join(text+group for (text,group) in zip(text, refields)).split(sep)\n for i in range(0, len(parts), 2): parts[i] = re.escape(parts[i])\n\n regexp_pattern = ''.join(parts).replace('\\\\*','.*')\n fields = list(f for f in fields if f)\n return globpat, regexp_pattern , fields, dict(types)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_pattern(cls, pattern, filetype=None, key='filename', root=None, ignore=[]):\n filepattern = FilePattern(key, pattern, root=root)\n if FileInfo.filetype and filetype is None:\n filetype = FileInfo.filetype\n elif filetype is None:\n raise Exception(\"The filetype argument must be supplied unless \"\n \"an appropriate default has been specified as \"\n \"FileInfo.filetype\")\n return FileInfo(filepattern, key, filetype, ignore=ignore)", "response": "Convenience method to chain a pattern into a FileInfo instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(self, val, **kwargs):\n if Table and isinstance(val, Table):\n return self.load_table(val, **kwargs)\n elif DataFrame and isinstance(val, DataFrame):\n return self.load_dframe(val, **kwargs)\n else:\n raise Exception(\"Type %s not a DataFrame or Table.\" % type(val))", "response": "Load the contents into the supplied pandas dataframe or HoloViews Table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_table(self, table):\n items, data_keys = [], None\n for key, filename in table.items():\n data_dict = self.filetype.data(filename[0])\n current_keys = tuple(sorted(data_dict.keys()))\n values = [data_dict[k] for k in current_keys]\n if data_keys is None:\n data_keys = current_keys\n elif data_keys != current_keys:\n raise Exception(\"Data keys are inconsistent\")\n items.append((key, values))\n\n return Table(items, kdims=table.kdims, vdims=data_keys)", "response": "Load the contents of the file into the supplied Table using the specified key and filetype."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_dframe(self, dframe):\n filename_series = dframe[self.key]\n loaded_data = filename_series.map(self.filetype.data)\n keys = [list(el.keys()) for el in loaded_data.values]\n for key in set().union(*keys):\n key_exists = key in dframe.columns\n if key_exists:\n self.warning(\"Appending '_data' suffix to data key %r to avoid\"\n \"overwriting existing metadata with the same name.\" % key)\n suffix = '_data' if key_exists else ''\n dframe[key+suffix] = loaded_data.map(lambda x: x.get(key, np.nan))\n return dframe", "response": "Load the file contents into the supplied dataframe using the specified key and filetype."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _info(self, source, key, filetype, ignore):\n specs, mdata = [], {}\n mdata_clashes = set()\n\n for spec in source.specs:\n if key not in spec:\n raise Exception(\"Key %r not available in 'source'.\" % key)\n\n mdata = dict((k,v) for (k,v) in filetype.metadata(spec[key]).items()\n if k not in ignore)\n mdata_spec = {}\n mdata_spec.update(spec)\n mdata_spec.update(mdata)\n specs.append(mdata_spec)\n mdata_clashes = mdata_clashes | (set(spec.keys()) & set(mdata.keys()))\n # Metadata clashes can be avoided by using the ignore list.\n if mdata_clashes:\n self.warning(\"Loaded metadata keys overriding source keys.\")\n return specs", "response": "Generates the union of the source. specs and the metadata dicts loaded by the filetype object."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npushes new data into the buffer. Resume looping if paused.", "response": "async def _push(self, *args, **kwargs):\n \"\"\"Push new data into the buffer. Resume looping if paused.\"\"\"\n self._data.append((args, kwargs))\n if self._future is not None:\n\n future, self._future = self._future, None\n future.set_result(True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef newVersion():\n version=None\n fname='../swhlab/version.py'\n with open(fname) as f:\n raw=f.read().split(\"\\n\")\n for i,line in enumerate(raw):\n if line.startswith(\"__counter__\"):\n if version is None:\n version = int(line.split(\"=\")[1])\n raw[i]=\"__counter__=%d\"%(version+1)\n with open(fname,'w') as f:\n f.write(\"\\n\".join(raw))\n print(\"upgraded from version %03d to %03d\"%(version,version+1))", "response": "increments version counter in swhlab version. py"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a plot of one area of interest of a single sweep.", "response": "def figureStimulus(abf,sweeps=[0]):\n \"\"\"\n Create a plot of one area of interest of a single sweep.\n \"\"\"\n\n stimuli=[2.31250, 2.35270]\n for sweep in sweeps:\n abf.setsweep(sweep)\n for stimulus in stimuli:\n S1=int(abf.pointsPerSec*stimulus)\n S2=int(abf.pointsPerSec*(stimulus+0.001)) # 1ms of blanking\n abf.sweepY[S1:S2]=np.nan # blank out the stimulus area\n I1=int(abf.pointsPerSec*2.2) # time point (sec) to start\n I2=int(abf.pointsPerSec*2.6) # time point (sec) to end\n baseline=np.average(abf.sweepY[int(abf.pointsPerSec*2.0):int(abf.pointsPerSec*2.2)])\n Ys=lowPassFilter(abf.sweepY[I1:I2])-baseline\n Xs=abf.sweepX2[I1:I1+len(Ys)].flatten()\n plt.plot(Xs,Ys,alpha=.5,lw=2)\n return"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef analyzeSingle(abfFname):\n assert os.path.exists(abfFname) and abfFname.endswith(\".abf\")\n ABFfolder,ABFfname=os.path.split(abfFname)\n abfID=os.path.splitext(ABFfname)[0]\n IN=INDEX(ABFfolder)\n IN.analyzeABF(abfID)\n IN.scan()\n IN.html_single_basic([abfID],overwrite=True)\n IN.html_single_plot([abfID],overwrite=True)\n IN.scan()\n IN.html_index()\n\n return", "response": "Reanalyze data for a single ABF. Also remakes child and parent html."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef scan(self):\n t1=cm.timeit()\n self.files1=cm.list_to_lowercase(sorted(os.listdir(self.folder1)))\n self.files2=cm.list_to_lowercase(sorted(os.listdir(self.folder2)))\n self.files1abf=[x for x in self.files1 if x.endswith(\".abf\")]\n self.files1abf=cm.list_to_lowercase(cm.abfSort(self.files1abf))\n self.IDs=[x[:-4] for x in self.files1abf]\n self.log.debug(\"folder1 has %d files\",len(self.files1))\n self.log.debug(\"folder1 has %d abfs\",len(self.files1abf))\n self.log.debug(\"folder2 has %d files\",len(self.files2))\n self.log.debug(\"scanning folders took %s\",cm.timeit(t1))", "response": "Scan the folder1 and folder2 into files1 and files2."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrun this to turn all files in folder1 TIFs and JPGs into folder2 data.", "response": "def convertImages(self):\n \"\"\"\n run this to turn all folder1 TIFs and JPGs into folder2 data.\n TIFs will be treated as micrographs and converted to JPG with enhanced\n contrast. JPGs will simply be copied over.\n \"\"\"\n\n # copy over JPGs (and such)\n exts=['.jpg','.png']\n for fname in [x for x in self.files1 if cm.ext(x) in exts]:\n ID=\"UNKNOWN\"\n if len(fname)>8 and fname[:8] in self.IDs:\n ID=fname[:8]\n fname2=ID+\"_jpg_\"+fname\n if not fname2 in self.files2:\n self.log.info(\"copying over [%s]\"%fname2)\n shutil.copy(os.path.join(self.folder1,fname),os.path.join(self.folder2,fname2))\n if not fname[:8]+\".abf\" in self.files1:\n self.log.error(\"orphan image: %s\",fname)\n\n # convert TIFs (and such) to JPGs\n exts=['.tif','.tiff']\n for fname in [x for x in self.files1 if cm.ext(x) in exts]:\n ID=\"UNKNOWN\"\n if len(fname)>8 and fname[:8] in self.IDs:\n ID=fname[:8]\n fname2=ID+\"_tif_\"+fname+\".jpg\"\n if not fname2 in self.files2:\n self.log.info(\"converting micrograph [%s]\"%fname2)\n imaging.TIF_to_jpg(os.path.join(self.folder1,fname),saveAs=os.path.join(self.folder2,fname2))\n if not fname[:8]+\".abf\" in self.files1:\n self.log.error(\"orphan image: %s\",fname)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nanalyze every unanalyzed ABF in the folder", "response": "def analyzeAll(self):\n \"\"\"analyze every unanalyzed ABF in the folder.\"\"\"\n searchableData=str(self.files2)\n self.log.debug(\"considering analysis for %d ABFs\",len(self.IDs))\n for ID in self.IDs:\n if not ID+\"_\" in searchableData:\n self.log.debug(\"%s needs analysis\",ID)\n try:\n self.analyzeABF(ID)\n except:\n print(\"EXCEPTION! \"*100)\n else:\n self.log.debug(\"%s has existing analysis, not overwriting\",ID)\n self.log.debug(\"verified analysis of %d ABFs\",len(self.IDs))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef analyzeABF(self,ID):\n for fname in self.files2:\n if fname.startswith(ID+\"_data_\"):\n self.log.debug(\"deleting [%s]\",fname)\n os.remove(os.path.join(self.folder2,fname))\n self.log.info(\"analyzing (with overwrite) [%s]\",ID)\n protocols.analyze(os.path.join(self.folder1,ID+\".abf\"))", "response": "Analye a single ABF and index it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef htmlFor(self,fname):\n if os.path.splitext(fname)[1].lower() in ['.jpg','.png']:\n html=''%(fname,fname)\n if \"_tif_\" in fname:\n html=html.replace('Experimental Data:\"\n elif category=='plot':\n html+=\"

    Intrinsic Properties:

    \"\n elif category=='tif':\n html+=\"

    Micrographs:

    \"\n elif category=='other':\n html+=\"

    Additional Files:

    \"\n else:\n html+=\"

    ????:

    \"\n #html+=\"
    \"\n #html+='
    '*3\n for fname in filesByType[category]:\n html+=self.htmlFor(fname)\n html+='
    '*3\n print(\"creating\",saveAs,'...')\n style.save(html,saveAs,launch=launch)", "response": "Generate a generic flat file html for a single ABF."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates HTML of just intrinsic properties for a single file.", "response": "def html_single_plot(self,abfID,launch=False,overwrite=False):\n \"\"\"create ID_plot.html of just intrinsic properties.\"\"\"\n if type(abfID) is str:\n abfID=[abfID]\n for thisABFid in cm.abfSort(abfID):\n parentID=cm.parent(self.groups,thisABFid)\n saveAs=os.path.abspath(\"%s/%s_plot.html\"%(self.folder2,parentID))\n if overwrite is False and os.path.basename(saveAs) in self.files2:\n continue\n filesByType=cm.filesByType(self.groupFiles[parentID])\n html=\"\"\n html+='
    '\n html+='intrinsic properties for: %s
    '%parentID\n html+='%s'%os.path.abspath(self.folder1+\"/\"+parentID+\".abf\")\n html+='
    '\n for fname in filesByType['plot']:\n html+=self.htmlFor(fname)\n print(\"creating\",saveAs,'...')\n style.save(html,saveAs,launch=launch)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lowpass(data,filterSize=None):\n if filterSize is None:\n filterSize=len(data)/10\n kernel=kernel_gaussian(size=filterSize)\n data=convolve(data,kernel) # do the convolution with padded edges\n return data", "response": "minimal complexity low - pass filtering."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convolve(signal,kernel):\n pad=np.ones(len(kernel)/2)\n signal=np.concatenate((pad*signal[0],signal,pad*signal[-1]))\n signal=np.convolve(signal,kernel,mode='same')\n signal=signal[len(pad):-len(pad)]\n return signal", "response": "This function applies a kernel to a signal and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef timeit(timer=None):\n if timer is None:\n return time.time()\n else:\n took=time.time()-timer\n if took<1:\n return \"%.02f ms\"%(took*1000.0)\n elif took<60:\n return \"%.02f s\"%(took)\n else:\n return \"%.02f min\"%(took/60.0)", "response": "simple timer. returns a time object or a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmove the value to the front and return it", "response": "def list_move_to_front(l,value='other'):\n \"\"\"if the value is in the list, move it to the front and return it.\"\"\"\n l=list(l)\n if value in l:\n l.remove(value)\n l.insert(0,value)\n return l"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef list_move_to_back(l,value='other'):\n l=list(l)\n if value in l:\n l.remove(value)\n l.append(value)\n return l", "response": "move the value to the back and return it"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a list and a list of items to be first return the list in the same order except that it begins with each of the first items.", "response": "def list_order_by(l,firstItems):\n \"\"\"given a list and a list of items to be first, return the list in the\n same order except that it begins with each of the first items.\"\"\"\n l=list(l)\n for item in firstItems[::-1]: #backwards\n if item in l:\n l.remove(item)\n l.insert(0,item)\n return l"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a list of goofy ABF names return it sorted intelligently.", "response": "def abfSort(IDs):\n \"\"\"\n given a list of goofy ABF names, return it sorted intelligently.\n This places things like 16o01001 after 16901001.\n \"\"\"\n IDs=list(IDs)\n monO=[]\n monN=[]\n monD=[]\n good=[]\n for ID in IDs:\n if ID is None:\n continue\n if 'o' in ID:\n monO.append(ID)\n elif 'n' in ID:\n monN.append(ID)\n elif 'd' in ID:\n monD.append(ID)\n else:\n good.append(ID)\n return sorted(good)+sorted(monO)+sorted(monN)+sorted(monD)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef abfGroups(abfFolder):\n\n # prepare the list of files, filenames, and IDs\n files=False\n if type(abfFolder) is str and os.path.isdir(abfFolder):\n files=abfSort(os.listdir(abfFolder))\n elif type(abfFolder) is list:\n files=abfSort(abfFolder)\n assert type(files) is list\n files=list_to_lowercase(files)\n\n # group every filename in a different list, and determine parents\n abfs, IDs, others, parents, days = [],[],[],[],[]\n for fname in files:\n if fname.endswith(\".abf\"):\n abfs.append(fname)\n IDs.append(fname[:-4])\n days.append(fname[:5])\n else:\n others.append(fname)\n for ID in IDs:\n for fname in others:\n if fname.startswith(ID):\n parents.append(ID)\n parents=abfSort(set(parents)) # allow only one copy each\n days=abfSort(set(days)) # allow only one copy each\n\n # match up children with parents, respecting daily orphans.\n groups={}\n for day in days:\n parent=None\n for fname in [x for x in abfs if x.startswith(day)]:\n ID=fname[:-4]\n if ID in parents:\n parent=ID\n if not parent in groups.keys():\n groups[parent]=[]\n groups[parent].extend([ID])\n return groups", "response": "Given a folder path or list of files return a dict of parent IDs and children IDs."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef abfGroupFiles(groups,folder):\n assert os.path.exists(folder)\n files=os.listdir(folder)\n group2={}\n for parent in groups.keys():\n if not parent in group2.keys():\n group2[parent]=[]\n for ID in groups[parent]:\n for fname in [x.lower() for x in files if ID in x.lower()]:\n group2[parent].extend([fname])\n return group2", "response": "This function takes a dictionary of groups and a folder and returns a dictionary of all files in that folder."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parent(groups,ID):\n if ID in groups.keys():\n return ID # already a parent\n if not ID in groups.keys():\n for actualParent in groups.keys():\n if ID in groups[actualParent]:\n return actualParent # found the actual parent\n return None", "response": "given a groups dictionary and an ID return its actual parent ID."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef filesByType(fileList):\n features=[\"plot\",\"tif\",\"data\",\"other\",\"experiment\"]\n files={}\n for feature in features:\n files[feature]=[]\n for fname in fileList:\n other=True\n for feature in features:\n if \"_\"+feature+\"_\" in fname:\n files[feature].extend([fname])\n other=False\n if other:\n files['other'].extend([fname])\n return files", "response": "return a list of files sorted by type"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef userFolder():\n #path=os.path.abspath(tempfile.gettempdir()+\"/swhlab/\")\n #don't use tempdir! it will get deleted easily.\n path=os.path.expanduser(\"~\")+\"/.swhlab/\" # works on windows or linux\n # for me, path=r\"C:\\Users\\swharden\\.swhlab\"\n if not os.path.exists(path):\n print(\"creating\",path)\n os.mkdir(path)\n return os.path.abspath(path)", "response": "return the semi - temporary user folder"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef abfFname_Load():\n fname=userFolder()+\"/abfFname.ini\"\n if os.path.exists(fname):\n abfFname=open(fname).read().strip()\n if os.path.exists(abfFname) or abfFname.endswith(\"_._\"):\n return abfFname\n return os.path.abspath(os.sep)", "response": "return the path of the last loaded ABF"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the path of the last loaded ABF.", "response": "def abfFname_Save(abfFname):\n \"\"\"return the path of the last loaded ABF.\"\"\"\n fname=userFolder()+\"/abfFname.ini\"\n with open(fname,'w') as f:\n f.write(os.path.abspath(abfFname))\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gui_getFile():\n import tkinter as tk\n from tkinter import filedialog\n root = tk.Tk() # this is natively supported by python\n root.withdraw() # hide main window\n root.wm_attributes('-topmost', 1) # always on top\n fname = filedialog.askopenfilename(title = \"select ABF file\",\n filetypes=[('ABF Files', '.abf')],\n initialdir=os.path.dirname(abfFname_Load()))\n if fname.endswith(\".abf\"):\n abfFname_Save(fname)\n return fname\n else:\n print(\"didn't select an ABF!\")\n return None", "response": "Launch an ABF file selection file dialog."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef gui_getFolder():\n import tkinter as tk\n from tkinter import filedialog\n root = tk.Tk() # this is natively supported by python\n root.withdraw() # hide main window\n root.wm_attributes('-topmost', 1) # always on top\n fname = filedialog.askdirectory(title = \"select folder of ABFs\",\n initialdir=os.path.dirname(abfFname_Load()))\n if len(fname)>3:\n abfFname_Save(fname+\"/_._\")\n return fname\n else:\n print(\"didn't select an ABF!\")\n return None", "response": "Launch a folder selection dialog."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def _try_catch_coro(emitter, event, listener, coro):\n try:\n\n await coro\n\n except Exception as exc:\n\n if event == emitter.LISTENER_ERROR_EVENT:\n\n raise\n\n emitter.emit(emitter.LISTENER_ERROR_EVENT, event, listener, exc)", "response": "Coroutine wrapper to catch errors after async scheduling."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _check_limit(self, event):\n if self.count(event) > self.max_listeners:\n\n warnings.warn(\n 'Too many listeners for event {}'.format(event),\n ResourceWarning,\n )", "response": "Check if the listener limit is hit and warn if needed."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbinding a listener to an event.", "response": "def add_listener(self, event, listener):\n \"\"\"Bind a listener to a particular event.\n\n Args:\n event (str): The name of the event to listen for. This may be any\n string value.\n listener (def or async def): The callback to execute when the event\n fires. This may be a sync or async function.\n \"\"\"\n self.emit('new_listener', event, listener)\n self._listeners[event].append(listener)\n self._check_limit(event)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a listener that is only called once.", "response": "def once(self, event, listener):\n \"\"\"Add a listener that is only called once.\"\"\"\n self.emit('new_listener', event, listener)\n self._once[event].append(listener)\n self._check_limit(event)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove a listener from the emitter.", "response": "def remove_listener(self, event, listener):\n \"\"\"Remove a listener from the emitter.\n\n Args:\n event (str): The event name on which the listener is bound.\n listener: A reference to the same object given to add_listener.\n\n Returns:\n bool: True if a listener was removed else False.\n\n This method only removes one listener at a time. If a listener is\n attached multiple times then this method must be called repeatedly.\n Additionally, this method removes listeners first from the those\n registered with 'on' or 'add_listener'. If none are found it continue\n to remove afterwards from those added with 'once'.\n \"\"\"\n with contextlib.suppress(ValueError):\n\n self._listeners[event].remove(listener)\n return True\n\n with contextlib.suppress(ValueError):\n\n self._once[event].remove(listener)\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves all listeners or those of the specified event.", "response": "def remove_all_listeners(self, event=None):\n \"\"\"Remove all listeners, or those of the specified *event*.\n\n It's not a good idea to remove listeners that were added elsewhere in\n the code, especially when it's on an emitter that you didn't create\n (e.g. sockets or file streams).\n \"\"\"\n if event is None:\n self._listeners = collections.defaultdict(list)\n self._once = collections.defaultdict(list)\n else:\n del self._listeners[event]\n del self._once[event]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _dispatch_coroutine(self, event, listener, *args, **kwargs):\n try:\n\n coro = listener(*args, **kwargs)\n\n except Exception as exc:\n\n if event == self.LISTENER_ERROR_EVENT:\n\n raise\n\n return self.emit(self.LISTENER_ERROR_EVENT, event, listener, exc)\n\n asyncio.ensure_future(\n _try_catch_coro(self, event, listener, coro),\n loop=self._loop,\n )", "response": "Dispatches a coroutine for execution."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndispatch a sync function.", "response": "def _dispatch_function(self, event, listener, *args, **kwargs):\n \"\"\"Execute a sync function.\n\n Args:\n event (str): The name of the event that triggered this call.\n listener (def): The def that needs to be executed.\n *args: Any number of positional arguments.\n **kwargs: Any number of keyword arguments.\n\n The values of *args and **kwargs are passed, unaltered, to the def\n when exceuting. If there is an exception executing the def, such as the\n wrong number of arguments, the emitter's error event is triggered. If\n the triggering event _is_ the emitter's error event then the exception\n is reraised. The reraised exception may show in debug mode for the\n event loop but is otherwise silently dropped.\n \"\"\"\n try:\n\n return listener(*args, **kwargs)\n\n except Exception as exc:\n\n if event == self.LISTENER_ERROR_EVENT:\n\n raise\n\n return self.emit(self.LISTENER_ERROR_EVENT, event, listener, exc)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _dispatch(self, event, listener, *args, **kwargs):\n if (\n asyncio.iscoroutinefunction(listener) or\n isinstance(listener, functools.partial) and\n asyncio.iscoroutinefunction(listener.func)\n ):\n\n return self._dispatch_coroutine(event, listener, *args, **kwargs)\n\n return self._dispatch_function(event, listener, *args, **kwargs)", "response": "Dispatch an event to a listener."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef emit(self, event, *args, **kwargs):\n listeners = self._listeners[event]\n listeners = itertools.chain(listeners, self._once[event])\n self._once[event] = []\n for listener in listeners:\n\n self._loop.call_soon(\n functools.partial(\n self._dispatch,\n event,\n listener,\n *args,\n **kwargs,\n )\n )\n\n return self", "response": "Calls each listener for the given event with the given arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the number of listeners for the event.", "response": "def count(self, event):\n \"\"\"Get the number of listeners for the event.\n\n Args:\n event (str): The event for which to count all listeners.\n\n The resulting count is a combination of listeners added using\n 'on'/'add_listener' and 'once'.\n \"\"\"\n return len(self._listeners[event]) + len(self._once[event])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert each TIF to PNG. Return filenames of new PNGs.", "response": "def genPNGs(folder,files=None):\n \"\"\"Convert each TIF to PNG. Return filenames of new PNGs.\"\"\"\n if files is None:\n files=glob.glob(folder+\"/*.*\")\n new=[]\n for fname in files:\n ext=os.path.basename(fname).split(\".\")[-1].lower()\n if ext in ['tif','tiff']:\n if not os.path.exists(fname+\".png\"):\n print(\" -- converting %s to PNG...\"%os.path.basename(fname))\n cm.image_convert(fname)\n new.append(fname) #fancy burn-in of image data\n else:\n pass\n #print(\" -- already converted %s to PNG...\"%os.path.basename(fname))\n return new"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating text to go inside HTML page for single ABF page.", "response": "def htmlABFcontent(ID,group,d):\n \"\"\"generate text to go inside for single ABF page.\"\"\"\n html=\"\"\n files=[]\n for abfID in group:\n files.extend(d[abfID])\n files=sorted(files)\n\n #start with root images\n html+=\"
    \"\n for fname in files:\n if \".png\" in fname.lower() and not \"swhlab4\" in fname:\n fname=\"../\"+os.path.basename(fname)\n html+=' '%(fname,fname)\n\n #progress to /swhlab4/ images\n html+=\"
    \"\n #ABFinfo\n lastID=''\n for fname in sorted(files):\n if not \"swhlab4\" in fname:\n continue\n ID=os.path.basename(fname).split(\"_\")[0]\n if not ID==lastID:\n lastID=ID\n html+=\"

    %s

    \"%os.path.basename(fname).split(\"_\")[0]\n if \".png\" in fname.lower():\n fname=os.path.basename(fname)\n html+=' '%(fname,fname)\n continue\n\n html+=\"
    \"\n for fname in files:\n if not \"swhlab4\" in fname:\n continue\n if \".pkl\" in fname:\n callit=os.path.basename(fname)\n thing=cm.getPkl(fname)\n if \"_APs.pkl\" in fname:\n callit+=\" (first AP)\"\n thing=cm.dictFlat(thing)\n if len(thing):\n thing=thing[0]\n elif \"_MTs.pkl\" in fname:\n if type(thing) == dict:\n callit+=\" (from AVG of all sweeps)\"\n else:\n callit+=\" (first sweep)\"\n thing=thing[0]\n elif \"_SAP.pkl\" in fname:\n continue #don't plot those, too complicated\n elif \"_info.pkl\" in fname or \"_iv.pkl\" in fname:\n pass #no trouble, go for it\n else:\n print(\" ?? not sure how to index [%s]\"%os.path.basename(fname))\n continue\n if type(thing) is dict:\n thing=cm.msgDict(thing)\n if type(thing) is list:\n out=''\n for item in thing:\n out+=str(item)+\"\\n\"\n thing=out\n thing=str(thing) #lol stringthing\n thing=\"### %s ###\\n\"%os.path.basename(fname)+thing\n\n # putting it in a textbox is obnoxious. put it in the source instead.\n #html+='

    '%(str(thing).count(\"\\n\")+5,thing)\n\n html+=\"(view source for %s) \"%(os.path.basename(fname),thing)\n\n return html"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a static html for that abf", "response": "def htmlABF(ID,group,d,folder,overwrite=False):\n \"\"\"given an ID and the dict of files, generate a static html for that abf.\"\"\"\n fname=folder+\"/swhlab4/%s_index.html\"%ID\n if overwrite is False and os.path.exists(fname):\n return\n html=TEMPLATES['abf']\n html=html.replace(\"~ID~\",ID)\n html=html.replace(\"~CONTENT~\",htmlABFcontent(ID,group,d))\n print(\" <- writing [%s]\"%os.path.basename(fname))\n with open(fname,'w') as f:\n f.write(html)\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread experiment. txt and return a dict with [ firstOfNewExp color star comments", "response": "def expMenu(groups,folder):\n \"\"\"read experiment.txt and return a dict with [firstOfNewExp, color, star, comments].\"\"\"\n ### GENERATE THE MENU DATA BASED ON EXPERIMENT FILE\n orphans = sorted(list(groups.keys()))\n menu=[]\n if os.path.exists(folder+'/experiment.txt'):\n with open(folder+'/experiment.txt') as f:\n raw=f.read()\n else:\n raw=\"\"\n for line in raw.split(\"\\n\"):\n item={}\n if len(line)==0:\n continue\n if line.startswith(\"~\"):\n line=line[1:].split(\" \",2)\n item[\"ID\"]=line[0]\n item[\"symbol\"]=''\n if len(line)>1:\n item[\"color\"]=line[1]\n else:\n item[\"color\"]=\"white\"\n if len(line)>2 and len(line[2]):\n item[\"comment\"]=line[2]\n if item[\"comment\"][0]==\"*\":\n item[\"symbol\"]='*'\n else:\n item[\"comment\"]=''\n if item[\"ID\"] in orphans:\n orphans.remove(item[\"ID\"])\n elif line.startswith(\"###\"):\n line=line[3:].strip().split(\" \",1)\n item[\"title\"]=line[0]\n item[\"comment\"]=''\n if len(line)>1:\n if line[1].startswith(\"- \"):\n line[1]=line[1][2:]\n item[\"comment\"]=line[1]\n else:\n item[\"unknown\"]=line\n menu.append(item)\n menu.append({\"title\":\"orphans\",\"comment\":\"\"})\n for ophan in orphans:\n menu.append({\"orphan\":ophan,\"ID\":ophan,\"color\":'',\"symbol\":'',\"comment\":''})\n return menu"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef genIndex(folder,forceIDs=[]):\n if not os.path.exists(folder+\"/swhlab4/\"):\n print(\" !! cannot index if no /swhlab4/\")\n return\n timestart=cm.timethis()\n files=glob.glob(folder+\"/*.*\") #ABF folder\n files.extend(glob.glob(folder+\"/swhlab4/*.*\"))\n print(\" -- indexing glob took %.02f ms\"%(cm.timethis(timestart)*1000))\n files.extend(genPNGs(folder,files))\n files=sorted(files)\n timestart=cm.timethis()\n d=cm.getIDfileDict(files) #TODO: this is really slow\n print(\" -- filedict length:\",len(d))\n print(\" -- generating ID dict took %.02f ms\"%(cm.timethis(timestart)*1000))\n groups=cm.getABFgroups(files)\n print(\" -- groups length:\",len(groups))\n for ID in sorted(list(groups.keys())):\n overwrite=False\n for abfID in groups[ID]:\n if abfID in forceIDs:\n overwrite=True\n try:\n htmlABF(ID,groups[ID],d,folder,overwrite)\n except:\n print(\"~~ HTML GENERATION FAILED!!!\")\n menu=expMenu(groups,folder)\n makeSplash(menu,folder)\n makeMenu(menu,folder)\n htmlFrames(d,folder)\n makeMenu(menu,folder)\n makeSplash(menu,folder)", "response": "generates an index of the given folder"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndrawing the phase plot of the current sweep.", "response": "def drawPhasePlot(abf,m1=0,m2=None):\n \"\"\"\n Given an ABF object (SWHLab), draw its phase plot of the current sweep.\n m1 and m2 are optional marks (in seconds) for plotting only a range of data.\n Assume a matplotlib figure is already open and just draw on top if it.\n \"\"\"\n\n if not m2:\n m2 = abf.sweepLength\n\n cm = plt.get_cmap('CMRmap')\n #cm = plt.get_cmap('CMRmap_r')\n #cm = plt.get_cmap('spectral')\n #cm = plt.get_cmap('winter')\n\n # prepare Xs, Ys, and dYs\n Y = abf.sweepY\n Y = Y[int(abf.pointsPerSec*m1):int(abf.pointsPerSec*m2)]\n dY = (Y[1:]-Y[:-1])*abf.rate/1000.0 # mV/ms\n dY = np.append(dY,dY[-1])\n Xs = np.arange(len(dY))/abf.pointsPerSec\n Xs = Xs + Xs[-1]*abf.sweep\n\n # plot the voltage\n plt.subplot(131)\n plt.grid(alpha=.5)\n plt.plot(Xs,Y,lw=.5,color=cm(abf.sweep/abf.sweeps))\n plt.title(\"membrane voltage\")\n plt.ylabel(\"V (mV)\")\n plt.xlabel(\"time (sec)\")\n plt.margins(0,.1)\n\n # plot the first derivative of the voltage\n plt.subplot(132)\n plt.grid(alpha=.5)\n plt.plot(Xs,dY,lw=.5,color=cm(abf.sweep/abf.sweeps))\n plt.title(\"voltage velocity\")\n plt.ylabel(\"dV (mV/ms)\")\n plt.xlabel(\"time (sec)\")\n plt.margins(0,.1)\n\n # make the phase plot\n plt.subplot(133)\n plt.grid(alpha=.5)\n plt.plot(Y,dY,alpha=.5,lw=.5,color=cm(abf.sweep/abf.sweeps))\n plt.title(\"phase plot\")\n plt.ylabel(\"dV (mV/ms)\")\n plt.xlabel(\"V (mV)\")\n plt.margins(.1,.1)\n\n # tighten up the figure\n plt.tight_layout()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef TIF_to_jpg(fnameTiff, overwrite=False, saveAs=\"\"):\n \n if saveAs == \"\":\n saveAs=fnameTiff+\".jpg\"\n\n if overwrite is False and os.path.exists(saveAs):\n print(\"file exists, not overwriting...\")\n return\n\n # load the image\n img=pylab.imread(fnameTiff)\n img=img/np.max(img) # now the data is from 0 to 1\n\n # determine the old histogram\n hist1,bins1=np.histogram(img.ravel(),bins=256, range=(0,1))\n #pylab.plot(bins[:-1],hist)\n\n # detect darkfield by average:\n if np.average(img)<.2:\n vmin=None\n vmax=None\n msg=\" | FLU\"\n while np.average(img)<.5:\n img=np.sqrt(img)\n msg+=\"^(.5)\"\n else:\n msg=\" | DIC\"\n percentile=.005\n vmin=np.percentile(img.ravel(),percentile)\n vmax=np.percentile(img.ravel(),100-percentile)\n\n\n # determine the new histogram\n hist2,bins2=np.histogram(img.ravel(),bins=256, range=(0,1))\n\n # plot it with resizing magic\n fig=pylab.figure(facecolor='r')\n fig.gca().imshow(img,cmap=pylab.gray(),vmin=vmin,vmax=vmax)\n pylab.subplots_adjust(top=1, bottom=0, right=1, left=0, hspace=0, wspace=0)\n pylab.gca().xaxis.set_major_locator(pylab.NullLocator())\n pylab.gca().yaxis.set_major_locator(pylab.NullLocator())\n pylab.axis('off')\n\n # resize it to the original size\n fig.set_size_inches(img.shape[1]/100, img.shape[0]/100)\n\n # add text\n msg=\"%s | %s\"%(os.path.basename(fnameTiff),\n datetime.datetime.fromtimestamp(os.path.getmtime(fnameTiff)))+msg\n center=10\n pylab.text(center,center,\"%s\"%(msg),va=\"top\",color='w',size='small',\n family='monospace',weight='bold',\n bbox=dict(facecolor='k', alpha=.5))\n\n # add scale bar\n scaleWidthPx=False\n if \"f10\" in fnameTiff:\n scaleWidthPx,scaleBarText=39,\"25 um\"\n if \"f20\" in fnameTiff:\n scaleWidthPx,scaleBarText=31,\"10 um\"\n if scaleWidthPx:\n scaleBarPadding=10\n x2,y2=img.shape[1]-scaleBarPadding,img.shape[0]-scaleBarPadding\n x1,y1=x2-scaleWidthPx,y2\n for offset,color,alpha in [[2,'k',.5],[0,'w',1]]:\n pylab.plot([x1+offset,x2+offset],[y1+offset,y2+offset],'-',\n color=color,lw=4,alpha=alpha)\n pylab.text((x1+x2)/2+offset,y1-5+offset,scaleBarText,color=color,\n ha=\"center\",weight=\"bold\",alpha=alpha,\n size=\"small\",va=\"bottom\",family=\"monospace\")\n\n # add histogram\n #pylab.plot(img.shape[1]-bins1[:-1][::-1]*200,-hist1/max(hist1)*100+110,color='g')\n #pylab.plot(img.shape[1]-bins2[:-1][::-1]*200,-hist2/max(hist2)*100+110,color='b')\n #pylab.show()\n\n # save it\n pylab.savefig(saveAs,dpi=100)\n\n # clean up\n pylab.close()", "response": "This function takes a TIF file and creates a pretty labeled JPG."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef TIF_to_jpg_all(path):\n for fname in sorted(glob.glob(path+\"/*.tif\")):\n print(fname)\n TIF_to_jpg(fname)", "response": "run TIF_to_jpg on every TIF of a folder"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nanalyzing the sweep and return the EPSCs and IPSCs.", "response": "def analyzeSweep(abf,sweep,m1=None,m2=None,plotToo=False):\n \"\"\"\n m1 and m2, if given, are in seconds.\n returns [# EPSCs, # IPSCs]\n \"\"\"\n abf.setsweep(sweep)\n if m1 is None: m1=0\n else: m1=m1*abf.pointsPerSec\n if m2 is None: m2=-1\n else: m2=m2*abf.pointsPerSec\n\n # obtain X and Y\n Yorig=abf.sweepY[int(m1):int(m2)]\n X=np.arange(len(Yorig))/abf.pointsPerSec\n\n Ylpf=linear_gaussian(Yorig,sigmaSize=abf.pointsPerMs*300,forwardOnly=False)\n Yflat=Yorig-Ylpf\n\n EPSCs,IPSCs=[],[]\n\n if plotToo:\n plt.figure(figsize=(15,6))\n ax1=plt.subplot(211)\n plt.title(\"%s sweep %d\"%(abf.ID,sweep))\n plt.grid()\n plt.plot(X,Yorig,alpha=.5)\n plt.plot(X,Ylpf,'k',alpha=.5,lw=2)\n plt.margins(0,.2)\n\n plt.subplot(212,sharex=ax1)\n plt.title(\"gaussian baseline subtraction\")\n plt.grid()\n plt.plot(X,Yflat,alpha=.5)\n plt.axhline(0,color='k',lw=2,alpha=.5)\n\n plt.tight_layout()\n plt.show()\n\n # TEST GAUSS\n hist, bin_edges = np.histogram(Yflat, density=True, bins=200)\n peakPa=bin_edges[np.where(hist==max(hist))[0][0]+1]\n\n if plotToo:\n plt.figure()\n plt.grid()\n plt.plot(bin_edges[1:],hist,alpha=.5)\n plt.axvline(0,color='k')\n plt.axvline(peakPa,color='r',ls='--',lw=2,alpha=.5)\n plt.semilogy()\n plt.title(\"sweep data distribution\")\n plt.ylabel(\"power\")\n plt.xlabel(\"pA deviation\")\n plt.show()\n\n return peakPa"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convert(fname,saveAs=True,showToo=False):\n\n # load the image\n #im = Image.open(fname) #PIL can't handle 12-bit TIFs well\n im=ndimage.imread(fname) #scipy does better with it\n im=np.array(im,dtype=float) # now it's a numpy array\n\n # do all image enhancement here\n cutoffLow=np.percentile(im,.01)\n cutoffHigh=np.percentile(im,99.99)\n im[np.where(imcutoffHigh)]=cutoffHigh\n\n # IMAGE FORMATTING\n im-=np.min(im) #auto contrast\n im/=np.max(im) #normalize\n im*=255 #stretch contrast (8-bit)\n im = Image.fromarray(im)\n\n # IMAGE DRAWING\n msg=\"%s\\n\"%os.path.basename(fname)\n msg+=\"%s\\n\"%cm.epochToString(os.path.getmtime(fname))\n d = ImageDraw.Draw(im)\n fnt = ImageFont.truetype(\"arial.ttf\", 20)\n d.text((6,6),msg,font=fnt,fill=0)\n d.text((4,4),msg,font=fnt,fill=255)\n\n if showToo:\n im.show()\n if saveAs is False:\n return\n if saveAs is True:\n saveAs=fname+\".png\"\n im.convert('RGB').save(saveAs)\n return saveAs", "response": "Convert weird TIF files into web - friendly versions."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots X and Y data then shade its background by variance.", "response": "def plot_shaded_data(X,Y,variances,varianceX):\n \"\"\"plot X and Y data, then shade its background by variance.\"\"\"\n plt.plot(X,Y,color='k',lw=2)\n nChunks=int(len(Y)/CHUNK_POINTS)\n for i in range(0,100,PERCENT_STEP):\n varLimitLow=np.percentile(variances,i)\n varLimitHigh=np.percentile(variances,i+PERCENT_STEP)\n varianceIsAboveMin=np.where(variances>=varLimitLow)[0]\n varianceIsBelowMax=np.where(variances<=varLimitHigh)[0]\n varianceIsRange=[chunkNumber for chunkNumber in range(nChunks) \\\n if chunkNumber in varianceIsAboveMin \\\n and chunkNumber in varianceIsBelowMax]\n for chunkNumber in varianceIsRange:\n t1=chunkNumber*CHUNK_POINTS/POINTS_PER_SEC\n t2=t1+CHUNK_POINTS/POINTS_PER_SEC\n plt.axvspan(t1,t2,alpha=.3,color=COLORMAP(i/100),lw=0)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating some fancy graphs to show color - coded variances.", "response": "def show_variances(Y,variances,varianceX,logScale=False):\n \"\"\"create some fancy graphs to show color-coded variances.\"\"\"\n \n plt.figure(1,figsize=(10,7))\n plt.figure(2,figsize=(10,7))\n varSorted=sorted(variances)\n \n plt.figure(1)\n plt.subplot(211)\n plt.grid()\n plt.title(\"chronological variance\")\n plt.ylabel(\"original data\")\n plot_shaded_data(X,Y,variances,varianceX)\n plt.margins(0,.1) \n plt.subplot(212)\n plt.ylabel(\"variance (pA) (log%s)\"%str(logScale))\n plt.xlabel(\"time in sweep (sec)\")\n plt.plot(varianceX,variances,'k-',lw=2)\n \n plt.figure(2)\n plt.ylabel(\"variance (pA) (log%s)\"%str(logScale))\n plt.xlabel(\"chunk number\")\n plt.title(\"sorted variance\")\n plt.plot(varSorted,'k-',lw=2)\n \n for i in range(0,100,PERCENT_STEP):\n varLimitLow=np.percentile(variances,i)\n varLimitHigh=np.percentile(variances,i+PERCENT_STEP)\n label=\"%2d-%d percentile\"%(i,i++PERCENT_STEP)\n color=COLORMAP(i/100)\n print(\"%s: variance = %.02f - %.02f\"%(label,varLimitLow,varLimitHigh))\n plt.figure(1)\n plt.axhspan(varLimitLow,varLimitHigh,alpha=.5,lw=0,color=color,label=label)\n plt.figure(2)\n chunkLow=np.where(varSorted>=varLimitLow)[0][0]\n chunkHigh=np.where(varSorted>=varLimitHigh)[0][0]\n plt.axvspan(chunkLow,chunkHigh,alpha=.5,lw=0,color=color,label=label)\n \n for fignum in [1,2]:\n plt.figure(fignum)\n if logScale:\n plt.semilogy()\n plt.margins(0,0)\n plt.grid()\n if fignum is 2:\n plt.legend(fontsize=10,loc='upper left',shadow=True)\n plt.tight_layout()\n plt.savefig('2016-12-15-variance-%d-log%s.png'%(fignum,str(logScale)))\n plt.show()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if event detection occured. Checks if APs flag is set and runs detection on all sweeps.", "response": "def ensureDetection(self):\n \"\"\"\n run this before analysis. Checks if event detection occured.\n If not, runs AP detection on all sweeps.\n \"\"\"\n if self.APs==False:\n self.log.debug(\"analysis attempted before event detection...\")\n self.detect()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef detect(self):\n self.log.info(\"initializing AP detection on all sweeps...\")\n t1=cm.timeit()\n for sweep in range(self.abf.sweeps):\n self.detectSweep(sweep)\n self.log.info(\"AP analysis of %d sweeps found %d APs (completed in %s)\",\n self.abf.sweeps,len(self.APs),cm.timeit(t1))", "response": "runs AP detection on every sweep."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nperforms AP detection on current sweep.", "response": "def detectSweep(self,sweep=0):\n \"\"\"perform AP detection on current sweep.\"\"\"\n\n if self.APs is False: # indicates detection never happened\n self.APs=[] # now indicates detection occured\n\n # delete every AP from this sweep from the existing array\n for i,ap in enumerate(self.APs):\n if ap[\"sweep\"]==sweep:\n self.APs[i]=None\n if self.APs.count(None):\n self.log.debug(\"deleting %d existing APs from memory\",self.APs.count(None))\n while None in self.APs:\n self.APs.remove(None)\n self.log.debug(\"initiating AP detection (%d already in memory)\",len(self.APs))\n\n self.abf.derivative=True\n self.abf.setsweep(sweep)\n\n # detect potential AP (Is) by a dV/dT threshold crossing\n Is = cm.where_cross(self.abf.sweepD,self.detect_over)\n self.log.debug(\"initial AP detection: %d APs\"%len(Is))\n\n # eliminate APs where dV/dT doesn't cross below -10 V/S within 2 ms\n for i,I in enumerate(Is):\n if np.min(self.abf.sweepD[I:I+2*self.abf.pointsPerMs])>-10:\n Is[i]=0\n Is=Is[np.nonzero(Is)]\n self.log.debug(\"after lower threshold checking: %d APs\"%len(Is))\n\n # walk 1ms backwards and find point of +10 V/S threshold crossing\n for i,I in enumerate(Is):\n stepBack=0\n while(self.abf.sweepD[I-stepBack])>10 and stepBack/self.abf.pointsPerMs<1: #2ms max\n stepBack+=1\n Is[i]-=stepBack\n\n # analyze each AP\n sweepAPs=[]\n for i,I in enumerate(Is):\n try:\n timeInSweep=I/self.abf.pointsPerSec\n if timeInSweepself.detect_time2:\n continue # skip because it's not within the marks\n ap={} # create the AP entry\n ap[\"sweep\"]=sweep # number of the sweep containing this AP\n ap[\"I\"]=I # index sweep point of start of AP (10 mV/ms threshold crossing)\n ap[\"Tsweep\"]=I/self.abf.pointsPerSec # time in the sweep of index crossing (sec)\n ap[\"T\"]=ap[\"Tsweep\"]+self.abf.sweepInterval*sweep # time in the experiment\n ap[\"Vthreshold\"]=self.abf.sweepY[I] # threshold at rate of -10mV/ms\n\n # determine how many points from the start dV/dt goes below -10 (from a 5ms chunk)\n chunk=self.abf.sweepD[I:I+5*self.abf.pointsPerMs] # give it 5ms to cross once\n I_toNegTen=np.where(chunk<-10)[0][0]\n chunk=self.abf.sweepD[I+I_toNegTen:I+I_toNegTen+10*self.abf.pointsPerMs] # give it 30ms to cross back\n if not max(chunk)>-10:\n self.log.debug(\"skipping unreal AP at T=%f\"%ap[\"T\"])\n self.log.error(\"^^^ can you confirm this is legit?\")\n continue # probably a pre-AP \"bump\" to be ignored\n I_recover=np.where(chunk>-10)[0][0]+I_toNegTen+I # point where trace returns to above -10 V/S\n ap[\"dVfastIs\"]=[I,I_recover] # span of the fast component of the dV/dt trace\n ap[\"dVfastMS\"]=(I_recover-I)/self.abf.pointsPerMs # time (in ms) of this fast AP component\n\n # determine derivative min/max from a 2ms chunk which we expect to capture the fast AP\n chunk=self.abf.sweepD[ap[\"dVfastIs\"][0]:ap[\"dVfastIs\"][1]]\n ap[\"dVmax\"]=np.max(chunk)\n ap[\"dVmaxI\"]=np.where(chunk==ap[\"dVmax\"])[0][0]+I\n ap[\"dVmin\"]=np.min(chunk)\n ap[\"dVminI\"]=np.where(chunk==ap[\"dVmin\"])[0][0]+I\n if ap[\"dVmax\"]<10 or ap[\"dVmin\"]>-10:\n self.log.debug(\"throwing out AP with low dV/dt to be an AP\")\n self.log.error(\"^^^ can you confirm this is legit?\")\n continue\n\n # before determining AP shape stats, see where trace recovers to threshold\n chunkSize=self.abf.pointsPerMs*10 #AP shape may be 10ms\n if len(Is)-1>i and Is[i+1]<(I+chunkSize): # if slow AP runs into next AP\n chunkSize=Is[i+1]-I # chop it down\n if chunkSize<(self.abf.pointsPerMs*2):\n continue # next AP is so soon, it's >500 Hz. Can't be real.\n ap[\"VslowIs\"]=[I,I+chunkSize] # time range of slow AP dynamics\n chunk=self.abf.sweepY[I:I+chunkSize]\n\n # determine AP peak and minimum\n ap[\"Vmax\"]=np.max(chunk)\n ap[\"VmaxI\"]=np.where(chunk==ap[\"Vmax\"])[0][0]+I\n chunkForMin=np.copy(chunk) # so we can destroy it\n chunkForMin[:ap[\"VmaxI\"]-I]=np.inf # minimum won't be before peak now\n ap[\"Vmin\"]=np.min(chunkForMin) # supposedly the minimum is the AHP\n ap[\"VminI\"]=np.where(chunkForMin==ap[\"Vmin\"])[0][0]+I\n if ap[\"VminI\"] @\" syntax is a Parsley extension\n # construct CSS selector and append attribute to XPath expression\n m = self.REGEX_ENDING_ATTRIBUTE.match(selection)\n if m:\n # the selector should be a regular CSS selector\n cssxpath = css_to_xpath(m.group(\"expr\"))\n\n # if \"|\" is used for namespace prefix reference,\n # convert it to XPath prefix syntax\n attribute = m.group(\"attr\").replace('|', ':')\n\n cssxpath = \"%s/%s\" % (cssxpath, attribute)\n else:\n cssxpath = css_to_xpath(selection)\n\n selector = lxml.etree.XPath(\n cssxpath,\n namespaces = self.namespaces,\n extensions = self.extensions,\n smart_strings=(self.SMART_STRINGS\n or self._test_smart_strings_needed(selection)),\n )\n\n except tuple(self.CSSSELECT_SYNTAXERROR_EXCEPTIONS) as syntax_error:\n if self.DEBUG:\n print(repr(syntax_error), selection)\n print(\"Try interpreting as XPath selector\")\n try:\n selector = lxml.etree.XPath(selection,\n namespaces = self.namespaces,\n extensions = self.extensions,\n smart_strings=(self.SMART_STRINGS\n or self._test_smart_strings_needed(selection)),\n )\n\n except lxml.etree.XPathSyntaxError as syntax_error:\n syntax_error.msg += \": %s\" % selection\n raise syntax_error\n\n except Exception as e:\n if self.DEBUG:\n print(repr(e), selection)\n raise\n\n # for exception when trying to convert @ syntax\n except lxml.etree.XPathSyntaxError as syntax_error:\n syntax_error.msg += \": %s\" % selection\n raise syntax_error\n\n except Exception as e:\n if self.DEBUG:\n print(repr(e), selection)\n raise\n\n # wrap it/cache it\n self._selector_cache[selection] = Selector(selector)\n return self._selector_cache[selection]", "response": "Creates a new CSS selector and XPath expression from a CSS selector."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\njoining the real - time messaging service.", "response": "async def join_rtm(self, filters=None):\n \"\"\"Join the real-time messaging service.\n\n Arguments:\n filters (:py:class:`dict`, optional): Dictionary mapping\n message filters to the functions they should dispatch to.\n Use a :py:class:`collections.OrderedDict` if precedence is\n important; only one filter, the first match, will be\n applied to each message.\n\n \"\"\"\n if filters is None:\n filters = [cls(self) for cls in self.MESSAGE_FILTERS]\n url = await self._get_socket_url()\n logger.debug('Connecting to %r', url)\n async with ws_connect(url) as socket:\n first_msg = await socket.receive()\n self._validate_first_message(first_msg)\n self.socket = socket\n async for message in socket:\n if message.tp == MsgType.text:\n await self.handle_message(message, filters)\n elif message.tp in (MsgType.closed, MsgType.error):\n if not socket.closed:\n await socket.close()\n self.socket = None\n break\n logger.info('Left real-time messaging.')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def handle_message(self, message, filters):\n data = self._unpack_message(message)\n logger.debug(data)\n if data.get('type') == 'error':\n raise SlackApiError(\n data.get('error', {}).get('msg', str(data))\n )\n elif self.message_is_to_me(data):\n text = data['text'][len(self.address_as):].strip()\n if text == 'help':\n return self._respond(\n channel=data['channel'],\n text=self._instruction_list(filters),\n )\n elif text == 'version':\n return self._respond(\n channel=data['channel'],\n text=self.VERSION,\n )\n for _filter in filters:\n if _filter.matches(data):\n logger.debug('Response triggered')\n async for response in _filter:\n self._respond(channel=data['channel'], text=response)", "response": "Handle an incoming\n message appropriately."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new Bot instance from the API token.", "response": "async def from_api_token(cls, token=None, api_cls=SlackBotApi):\n \"\"\"Create a new instance from the API token.\n\n Arguments:\n token (:py:class:`str`, optional): The bot's API token\n (defaults to ``None``, which means looking in the\n environment).\n api_cls (:py:class:`type`, optional): The class to create\n as the ``api`` argument for API access (defaults to\n :py:class:`aslack.slack_api.SlackBotApi`).\n\n Returns:\n :py:class:`SlackBot`: The new instance.\n\n \"\"\"\n api = api_cls.from_env() if token is None else api_cls(api_token=token)\n data = await api.execute_method(cls.API_AUTH_ENDPOINT)\n return cls(data['user_id'], data['user'], api)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nformat an outgoing message for transmission.", "response": "def _format_message(self, channel, text):\n \"\"\"Format an outgoing message for transmission.\n\n Note:\n Adds the message type (``'message'``) and incremental ID.\n\n Arguments:\n channel (:py:class:`str`): The channel to send to.\n text (:py:class:`str`): The message text to send.\n\n Returns:\n :py:class:`str`: The JSON string of the message.\n\n \"\"\"\n payload = {'type': 'message', 'id': next(self._msg_ids)}\n payload.update(channel=channel, text=text)\n return json.dumps(payload)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def _get_socket_url(self):\n data = await self.api.execute_method(\n self.RTM_START_ENDPOINT,\n simple_latest=True,\n no_unreads=True,\n )\n return data['url']", "response": "Get the WebSocket URL for the RTM session."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _instruction_list(self, filters):\n return '\\n\\n'.join([\n self.INSTRUCTIONS.strip(),\n '*Supported methods:*',\n 'If you send \"@{}: help\" to me I reply with these '\n 'instructions.'.format(self.user),\n 'If you send \"@{}: version\" to me I reply with my current '\n 'version.'.format(self.user),\n ] + [filter.description() for filter in filters])", "response": "Generates the list of instructions for a specific class and filters."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nresponding to a message on the current socket.", "response": "def _respond(self, channel, text):\n \"\"\"Respond to a message on the current socket.\n\n Args:\n channel (:py:class:`str`): The channel to send to.\n text (:py:class:`str`): The message text to send.\n\n \"\"\"\n result = self._format_message(channel, text)\n if result is not None:\n logger.info(\n 'Sending message: %r',\n truncate(result, max_len=50),\n )\n self.socket.send_str(result)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvalidating the first message.", "response": "def _validate_first_message(cls, msg):\n \"\"\"Check the first message matches the expected handshake.\n\n Note:\n The handshake is provided as :py:attr:`RTM_HANDSHAKE`.\n\n Arguments:\n msg (:py:class:`aiohttp.Message`): The message to validate.\n\n Raises:\n :py:class:`SlackApiError`: If the data doesn't match the\n expected handshake.\n\n \"\"\"\n data = cls._unpack_message(msg)\n logger.debug(data)\n if data != cls.RTM_HANDSHAKE:\n raise SlackApiError('Unexpected response: {!r}'.format(data))\n logger.info('Joined real-time messaging.')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfinding first existing executable file in a list of options", "response": "def find_first_existing_executable(exe_list):\n \"\"\"\n Accepts list of [('executable_file_path', 'options')],\n Returns first working executable_file_path\n \"\"\"\n for filepath, opts in exe_list:\n try:\n proc = subprocess.Popen([filepath, opts],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n proc.communicate()\n except OSError:\n pass\n else:\n return filepath"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_app_locations():\n return [os.path.dirname(os.path.normpath(import_module(app_name).__file__))\n for app_name in PROJECT_APPS]", "response": "Returns list of paths to tested apps"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_tasks():\n task_classes = []\n for task_path in TASKS:\n try:\n module, classname = task_path.rsplit('.', 1)\n except ValueError:\n raise ImproperlyConfigured('%s isn\\'t a task module' % task_path)\n try:\n mod = import_module(module)\n except ImportError as e:\n raise ImproperlyConfigured('Error importing task %s: \"%s\"'\n % (module, e))\n try:\n task_class = getattr(mod, classname)\n except AttributeError:\n raise ImproperlyConfigured('Task module \"%s\" does not define a '\n '\"%s\" class' % (module, classname))\n task_classes.append(task_class)\n return task_classes", "response": "Get the imported task classes for each task that will be run"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_task_options():\n options = ()\n\n task_classes = get_tasks()\n for cls in task_classes:\n options += cls.option_list\n\n return options", "response": "Get the options for each task that will be run"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites the data from the db to a CLDF dataset according to the metadata in self. dataset.", "response": "def to_cldf(self, dest, mdname='cldf-metadata.json'):\n \"\"\"\n Write the data from the db to a CLDF dataset according to the metadata in `self.dataset`.\n\n :param dest:\n :param mdname:\n :return: path of the metadata file\n \"\"\"\n dest = Path(dest)\n if not dest.exists():\n dest.mkdir()\n\n data = self.read()\n\n if data[self.source_table_name]:\n sources = Sources()\n for src in data[self.source_table_name]:\n sources.add(Source(\n src['genre'],\n src['id'],\n **{k: v for k, v in src.items() if k not in ['id', 'genre']}))\n sources.write(dest / self.dataset.properties.get('dc:source', 'sources.bib'))\n\n for table_type, items in data.items():\n try:\n table = self.dataset[table_type]\n table.common_props['dc:extent'] = table.write(\n [self.retranslate(table, item) for item in items],\n base=dest)\n except KeyError:\n assert table_type == self.source_table_name, table_type\n return self.dataset.write_metadata(dest / mdname)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints basic stats for CLDF dataset", "response": "def stats(args):\n \"\"\"\n cldf stats \n\n Print basic stats for CLDF dataset , where may be the path to\n - a CLDF metadata file\n - a CLDF core data file\n \"\"\"\n ds = _get_dataset(args)\n print(ds)\n md = Table('key', 'value')\n md.extend(ds.properties.items())\n print(md.render(condensed=False, tablefmt=None))\n print()\n t = Table('Path', 'Type', 'Rows')\n for p, type_, r in ds.stats():\n t.append([p, type_, r])\n print(t.render(condensed=False, tablefmt=None))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef createdb(args):\n if len(args.args) < 2:\n raise ParserError('not enough arguments')\n ds = _get_dataset(args)\n db = Database(ds, fname=args.args[1])\n db.write_from_tg()\n args.log.info('{0} loaded in {1}'.format(ds, db.fname))", "response": "cldf createdb \n\n Load CLDF dataset into a SQLite DB, where may be the path to\n - a CLDF metadata file\n - a CLDF core data file"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_jsonfile(cls, fp, selector_handler=None, strict=False, debug=False):\n\n return cls._from_jsonlines(fp,\n selector_handler=selector_handler, strict=strict, debug=debug)", "response": "Create a Parselet instance from a file containing a JSON object containing the Parsley script as a JSON object containing the Parsley script s ID and the optional selector handler."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a Parselet instance from a file containing a YAML object containing the Parsley script as a YAML object.", "response": "def from_yamlfile(cls, fp, selector_handler=None, strict=False, debug=False):\n \"\"\"\n Create a Parselet instance from a file containing\n the Parsley script as a YAML object\n\n >>> import parslepy\n >>> with open('parselet.yml') as fp:\n ... parslepy.Parselet.from_yamlfile(fp)\n ...\n \n\n :param file fp: an open file-like pointer containing the Parsley script\n :rtype: :class:`.Parselet`\n\n Other arguments: same as for :class:`.Parselet` contructor\n \"\"\"\n\n return cls.from_yamlstring(fp.read(), selector_handler=selector_handler, strict=strict, debug=debug)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_yamlstring(cls, s, selector_handler=None, strict=False, debug=False):\n\n import yaml\n return cls(yaml.load(s), selector_handler=selector_handler, strict=strict, debug=debug)", "response": "Create a Parselet instance from a YAML string containing a Parsley script as YAML string containing a Parsley language."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a Parselet instance from a JSON string containing a Parsley script as a JSON string.", "response": "def from_jsonstring(cls, s, selector_handler=None, strict=False, debug=False):\n \"\"\"\n Create a Parselet instance from s (str) containing\n the Parsley script as JSON\n\n >>> import parslepy\n >>> parsley_string = '{ \"title\": \"h1\", \"link\": \"a @href\"}'\n >>> p = parslepy.Parselet.from_jsonstring(parsley_string)\n >>> type(p)\n \n >>>\n\n :param string s: a Parsley script as a JSON string\n :rtype: :class:`.Parselet`\n\n Other arguments: same as for :class:`.Parselet` contructor\n \"\"\"\n\n return cls._from_jsonlines(s.split(\"\\n\"),\n selector_handler=selector_handler, strict=strict, debug=debug)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _from_jsonlines(cls, lines, selector_handler=None, strict=False, debug=False):\n\n return cls(json.loads(\n \"\\n\".join([l for l in lines if not cls.REGEX_COMMENT_LINE.match(l)])\n ), selector_handler=selector_handler, strict=strict, debug=debug)", "response": "Create a new object from a list of lines."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse(self, fp, parser=None, context=None):\n\n if parser is None:\n parser = lxml.etree.HTMLParser()\n doc = lxml.etree.parse(fp, parser=parser).getroot()\n return self.extract(doc, context=context)", "response": "Parse an HTML or XML document and return the extracted object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_fromstring(self, s, parser=None, context=None):\n if parser is None:\n parser = lxml.etree.HTMLParser()\n doc = lxml.etree.fromstring(s, parser=parser)\n return self.extract(doc, context=context)", "response": "Parse an HTML or XML document and return the extacted object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncompiles the Parsley tree starting from the root node", "response": "def compile(self):\n \"\"\"\n Build the abstract Parsley tree starting from the root node\n (recursive)\n \"\"\"\n if not isinstance(self.parselet, dict):\n raise ValueError(\"Parselet must be a dict of some sort. Or use .from_jsonstring(), \" \\\n \".from_jsonfile(), .from_yamlstring(), or .from_yamlfile()\")\n self.parselet_tree = self._compile(self.parselet)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds the abstract Parsley extraction tree.", "response": "def _compile(self, parselet_node, level=0):\n \"\"\"\n Build part of the abstract Parsley extraction tree\n\n Arguments:\n parselet_node (dict) -- part of the Parsley tree to compile\n (can be the root dict/node)\n level (int) -- current recursion depth (used for debug)\n \"\"\"\n\n if self.DEBUG:\n debug_offset = \"\".join([\" \" for x in range(level)])\n\n if self.DEBUG:\n print(debug_offset, \"%s::compile(%s)\" % (\n self.__class__.__name__, parselet_node))\n\n if isinstance(parselet_node, dict):\n parselet_tree = ParsleyNode()\n for k, v in list(parselet_node.items()):\n\n # we parse the key raw elements but without much\n # interpretation (which is done by the SelectorHandler)\n try:\n m = self.REGEX_PARSELET_KEY.match(k)\n if not m:\n if self.DEBUG:\n print(debug_offset, \"could not parse key\", k)\n raise InvalidKeySyntax(k)\n except:\n raise InvalidKeySyntax(\"Key %s is not valid\" % k)\n\n key = m.group('key')\n # by default, fields are required\n key_required = True\n operator = m.group('operator')\n if operator == '?':\n key_required = False\n # FIXME: \"!\" operator not supported (complete array)\n scope = m.group('scope')\n\n # example: get list of H3 tags\n # { \"titles\": [\"h3\"] }\n # FIXME: should we support multiple selectors in list?\n # e.g. { \"titles\": [\"h1\", \"h2\", \"h3\", \"h4\"] }\n if isinstance(v, (list, tuple)):\n v = v[0]\n iterate = True\n else:\n iterate = False\n\n # keys in the abstract Parsley trees are of type `ParsleyContext`\n try:\n parsley_context = ParsleyContext(\n key,\n operator=operator,\n required=key_required,\n scope=self.selector_handler.make(scope) if scope else None,\n iterate=iterate)\n except SyntaxError:\n if self.DEBUG:\n print(\"Invalid scope:\", k, scope)\n raise\n\n if self.DEBUG:\n print(debug_offset, \"current context:\", parsley_context)\n\n # go deeper in the Parsley tree...\n try:\n child_tree = self._compile(v, level=level+1)\n except SyntaxError:\n if self.DEBUG:\n print(\"Invalid value: \", v)\n raise\n except:\n raise\n\n if self.DEBUG:\n print(debug_offset, \"child tree:\", child_tree)\n\n parselet_tree[parsley_context] = child_tree\n\n return parselet_tree\n\n # a string leaf should match some kind of selector,\n # let the selector handler deal with it\n elif isstr(parselet_node):\n return self.selector_handler.make(parselet_node)\n else:\n raise ValueError(\n \"Unsupported type(%s) for Parselet node <%s>\" % (\n type(parselet_node), parselet_node))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef extract(self, document, context=None):\n if context:\n self.selector_handler.context = context\n return self._extract(self.parselet_tree, document)", "response": "Extract the values from a lxml - parsed document into a Python dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _extract(self, parselet_node, document, level=0):\n\n if self.DEBUG:\n debug_offset = \"\".join([\" \" for x in range(level)])\n\n # we must go deeper in the Parsley tree\n if isinstance(parselet_node, ParsleyNode):\n\n # default output\n output = {}\n\n # process all children\n for ctx, v in list(parselet_node.items()):\n if self.DEBUG:\n print(debug_offset, \"context:\", ctx, v)\n extracted=None\n try:\n # scoped-extraction:\n # extraction should be done deeper in the document tree\n if ctx.scope:\n extracted = []\n selected = self.selector_handler.select(document, ctx.scope)\n if selected:\n for i, elem in enumerate(selected, start=1):\n parse_result = self._extract(v, elem, level=level+1)\n\n if isinstance(parse_result, (list, tuple)):\n extracted.extend(parse_result)\n else:\n extracted.append(parse_result)\n\n # if we're not in an array,\n # we only care about the first iteration\n if not ctx.iterate:\n break\n\n if self.DEBUG:\n print(debug_offset,\n \"parsed %d elements in scope (%s)\" % (i, ctx.scope))\n\n # local extraction\n else:\n extracted = self._extract(v, document, level=level+1)\n\n except NonMatchingNonOptionalKey as e:\n if self.DEBUG:\n print(debug_offset, str(e))\n if not ctx.required or not self.STRICT_MODE:\n output[ctx.key] = {}\n else:\n raise\n except Exception as e:\n if self.DEBUG:\n print(str(e))\n raise\n\n # replace empty-list result when not looping by empty dict\n if ( isinstance(extracted, list)\n and not extracted\n and not ctx.iterate):\n extracted = {}\n\n # keep only the first element if we're not in an array\n if self.KEEP_ONLY_FIRST_ELEMENT_IF_LIST:\n try:\n if ( isinstance(extracted, list)\n and extracted\n and not ctx.iterate):\n\n if self.DEBUG:\n print(debug_offset, \"keep only 1st element\")\n extracted = extracted[0]\n\n except Exception as e:\n if self.DEBUG:\n print(str(e))\n print(debug_offset, \"error getting first element\")\n\n # extraction for a required key gave nothing\n if ( self.STRICT_MODE\n and ctx.required\n and extracted is None):\n raise NonMatchingNonOptionalKey(\n 'key \"%s\" is required but yield nothing\\nCurrent path: %s/(%s)\\n' % (\n ctx.key,\n document.getroottree().getpath(document),v\n )\n )\n\n # special key to extract a selector-defined level deeper\n # but still output at same level\n # this can be useful for breaking up long selectors\n # or when you need to mix XPath and CSS selectors\n # e.g.\n # {\n # \"something(#content div.main)\": {\n # \"--(.//div[re:test(@class, 'style\\d{3,6}')])\": {\n # \"title\": \"h1\",\n # \"subtitle\": \"h2\"\n # }\n # }\n # }\n #\n if ctx.key == self.SPECIAL_LEVEL_KEY:\n if isinstance(extracted, dict):\n output.update(extracted)\n elif isinstance(extracted, list):\n if extracted:\n raise RuntimeError(\n \"could not merge non-empty list at higher level\")\n else:\n #empty list, dont bother?\n pass\n else:\n # required keys are handled above\n if extracted is not None:\n output[ctx.key] = extracted\n else:\n # do not add this optional key/value pair in the output\n pass\n\n return output\n\n # a leaf/Selector node\n elif isinstance(parselet_node, Selector):\n return self.selector_handler.extract(document, parselet_node)\n\n else:\n # FIXME: can this happen?\n # if selector handler returned None at compile time,\n # probably yes\n pass", "response": "Extract values at this document node level using the parselet_node instructions."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef auto_constraints(self, component=None):\n if not component:\n for table in self.tables:\n self.auto_constraints(table)\n return\n\n if not component.tableSchema.primaryKey:\n idcol = component.get_column(term_uri('id'))\n if idcol:\n component.tableSchema.primaryKey = [idcol.name]\n\n self._auto_foreign_keys(component)\n\n try:\n table_type = self.get_tabletype(component)\n except ValueError:\n # New component is not a known CLDF term, so cannot add components\n # automatically. TODO: We might me able to infer some based on\n # `xxxReference` column properties?\n return\n\n # auto-add foreign keys targetting the new component:\n for table in self.tables:\n self._auto_foreign_keys(table, component=component, table_type=table_type)", "response": "Auto - adds foreign key constraints to the specified component."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef url_builder(self, endpoint, *, root=None, params=None, url_params=None):\n if root is None:\n root = self.ROOT\n scheme, netloc, path, _, _ = urlsplit(root)\n return urlunsplit((\n scheme,\n netloc,\n urljoin(path, endpoint),\n urlencode(url_params or {}),\n '',\n )).format(**params or {})", "response": "Create a URL for the specified endpoint."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nraising an appropriate error for a given response.", "response": "def raise_for_status(response):\n \"\"\"Raise an appropriate error for a given response.\n\n Arguments:\n response (:py:class:`aiohttp.ClientResponse`): The API response.\n\n Raises:\n :py:class:`aiohttp.web_exceptions.HTTPException`: The appropriate\n error for the response's status.\n\n \"\"\"\n for err_name in web_exceptions.__all__:\n err = getattr(web_exceptions, err_name)\n if err.status_code == response.status:\n payload = dict(\n headers=response.headers,\n reason=response.reason,\n )\n if issubclass(err, web_exceptions._HTTPMove): # pylint: disable=protected-access\n raise err(response.headers['Location'], **payload)\n raise err(**payload)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef truncate(text, max_len=350, end='...'):\n if len(text) <= max_len:\n return text\n return text[:max_len].rsplit(' ', maxsplit=1)[0] + end", "response": "Truncate the supplied text for display."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a source or a list of source entries to the list.", "response": "def add(self, *entries):\n \"\"\"\n Add a source, either specified by glottolog reference id, or as bibtex record.\n \"\"\"\n for entry in entries:\n if isinstance(entry, string_types):\n self._add_entries(database.parse_string(entry, bib_format='bibtex'))\n else:\n self._add_entries(entry)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_cache_key(user_or_username, size, prefix):\n if isinstance(user_or_username, get_user_model()):\n user_or_username = user_or_username.username\n return '%s_%s_%s' % (prefix, user_or_username, size)", "response": "Returns a cache key that can be used to cache the user_or_username and size."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cache_result(func):\n def cache_set(key, value):\n cache.set(key, value, AVATAR_CACHE_TIMEOUT)\n return value\n\n def cached_func(user, size):\n prefix = func.__name__\n cached_funcs.add(prefix)\n key = get_cache_key(user, size, prefix=prefix)\n return cache.get(key) or cache_set(key, func(user, size))\n return cached_func", "response": "Decorator to cache the result of a function that take a user and size value."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction to be called when saving or changing an user s avatars.", "response": "def invalidate_cache(user, size=None):\n \"\"\"\n Function to be called when saving or changing an user's avatars.\n \"\"\"\n sizes = set(AUTO_GENERATE_AVATAR_SIZES)\n if size is not None:\n sizes.add(size)\n for prefix in cached_funcs:\n for size in sizes:\n cache.delete(get_cache_key(user, size, prefix))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a field object instance for a given PrefProxy object.", "response": "def get_field_for_proxy(pref_proxy):\n \"\"\"Returns a field object instance for a given PrefProxy object.\n\n :param PrefProxy pref_proxy:\n\n :rtype: models.Field\n\n \"\"\"\n field = {\n bool: models.BooleanField,\n int: models.IntegerField,\n float: models.FloatField,\n datetime: models.DateTimeField,\n\n }.get(type(pref_proxy.default), models.TextField)()\n\n update_field_from_proxy(field, pref_proxy)\n\n return field"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_field_from_proxy(field_obj, pref_proxy):\n attr_names = ('verbose_name', 'help_text', 'default')\n\n for attr_name in attr_names:\n setattr(field_obj, attr_name, getattr(pref_proxy, attr_name))", "response": "Updates a field object with data from a PrefProxy object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_pref_model_class(app, prefs, get_prefs_func):\n\n module = '%s.%s' % (app, PREFS_MODULE_NAME)\n\n model_dict = {\n '_prefs_app': app,\n '_get_prefs': staticmethod(get_prefs_func),\n '__module__': module,\n 'Meta': type('Meta', (models.options.Options,), {\n 'verbose_name': _('Preference'),\n 'verbose_name_plural': _('Preferences'),\n 'app_label': app,\n 'managed': False,\n })\n }\n\n for field_name, val_proxy in prefs.items():\n model_dict[field_name] = val_proxy.field\n\n model = type('Preferences', (models.Model,), model_dict)\n\n def fake_save_base(self, *args, **kwargs):\n\n updated_prefs = {\n f.name: getattr(self, f.name) for f in self._meta.fields if not isinstance(f, models.fields.AutoField)\n }\n\n app_prefs = self._get_prefs(self._prefs_app)\n\n for pref in app_prefs.keys():\n if pref in updated_prefs:\n app_prefs[pref].db_value = updated_prefs[pref]\n\n self.pk = self._prefs_app # Make Django 1.7 happy.\n prefs_save.send(sender=self, app=self._prefs_app, updated_prefs=updated_prefs)\n\n return True\n\n model.save_base = fake_save_base\n\n return model", "response": "Returns a preferences model class dynamically crated for a given app."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a dictionary of locals from a given frame.", "response": "def get_frame_locals(stepback=0):\n \"\"\"Returns locals dictionary from a given frame.\n\n :param int stepback:\n\n :rtype: dict\n\n \"\"\"\n with Frame(stepback=stepback) as frame:\n locals_dict = frame.f_locals\n\n return locals_dict"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef traverse_local_prefs(stepback=0):\n locals_dict = get_frame_locals(stepback+1)\n for k in locals_dict:\n if not k.startswith('_') and k.upper() == k:\n yield k, locals_dict", "response": "Generator to traverse through preferences available in local frame."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nimports preferences modules from packages and project root.", "response": "def import_prefs():\n \"\"\"Imports preferences modules from packages (apps) and project root.\"\"\"\n \n # settings.py locals if autodiscover_siteprefs() is in urls.py\n settings_locals = get_frame_locals(3)\n\n if 'self' not in settings_locals: # If not SiteprefsConfig.ready()\n # Try to import project-wide prefs.\n\n project_package = settings_locals['__package__'] # Expected project layout introduced in Django 1.4\n if not project_package:\n # Fallback to old layout.\n project_package = os.path.split(os.path.dirname(settings_locals['__file__']))[-1]\n\n import_module(project_package, PREFS_MODULE_NAME)\n\n import_project_modules(PREFS_MODULE_NAME)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting file details in the current directory", "response": "def print_file_info():\n \"\"\"Prints file details in the current directory\"\"\"\n tpl = TableLogger(columns='file,created,modified,size')\n for f in os.listdir('.'):\n size = os.stat(f).st_size\n date_created = datetime.fromtimestamp(os.path.getctime(f))\n date_modified = datetime.fromtimestamp(os.path.getmtime(f))\n tpl(f, date_created, date_modified, size)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _bind_args(sig, param_matchers, args, kwargs):\n '''\n Attempt to bind the args to the type signature. First try to just bind\n to the signature, then ensure that all arguments match the parameter\n types.\n '''\n #Bind to signature. May throw its own TypeError\n bound = sig.bind(*args, **kwargs)\n\n if not all(param_matcher(bound.arguments[param_name])\n for param_name, param_matcher in param_matchers):\n raise TypeError\n\n return bound", "response": "Bind the args to the type signature."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a function which returns true if the argument of the a function matches the given annotation.", "response": "def _make_param_matcher(annotation, kind=None):\n '''\n For a given annotation, return a function which, when called on a\n function argument, returns true if that argument matches the annotation.\n If the annotation is a type, it calls isinstance; if it's a callable,\n it calls it on the object; otherwise, it performs a value comparison.\n If the parameter is variadic (*args) and the annotation is a type, the\n matcher will attempt to match each of the arguments in args\n '''\n if isinstance(annotation, type) or (\n isinstance(annotation, tuple) and\n all(isinstance(a, type) for a in annotation)):\n if kind is Parameter.VAR_POSITIONAL:\n return (lambda args: all(isinstance(x, annotation) for x in args))\n else:\n return (lambda x: isinstance(x, annotation))\n elif callable(annotation):\n return annotation\n else:\n return (lambda x: x == annotation)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _make_all_matchers(cls, parameters):\n '''\n For every parameter, create a matcher if the parameter has an\n annotation.\n '''\n for name, param in parameters:\n annotation = param.annotation\n if annotation is not Parameter.empty:\n yield name, cls._make_param_matcher(annotation, param.kind)", "response": "Create a matcher for all the parameters in the list."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a dispatch pair for func - a function that takes args kwargs attempts to bind those args to the type signature of func raises a TypeError", "response": "def _make_dispatch(cls, func):\n '''\n Create a dispatch pair for func- a tuple of (bind_args, func), where\n bind_args is a function that, when called with (args, kwargs), attempts\n to bind those args to the type signature of func, or else raise a\n TypeError\n '''\n sig = signature(func)\n matchers = tuple(cls._make_all_matchers(sig.parameters.items()))\n return (partial(cls._bind_args, sig, matchers), func)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _make_wrapper(self, func):\n '''\n Makes a wrapper function that executes a dispatch call for func. The\n wrapper has the dispatch and dispatch_first attributes, so that\n additional overloads can be added to the group.\n '''\n\n #TODO: consider using a class to make attribute forwarding easier.\n #TODO: consider using simply another DispatchGroup, with self.callees\n # assigned by reference to the original callees.\n @wraps(func)\n def executor(*args, **kwargs):\n return self.execute(args, kwargs)\n executor.dispatch = self.dispatch\n executor.dispatch_first = self.dispatch_first\n executor.func = func\n executor.lookup = self.lookup\n return executor", "response": "Make a wrapper function that executes a dispatch call for func."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dispatch(self, func):\n '''\n Adds the decorated function to this dispatch.\n '''\n self.callees.append(self._make_dispatch(func))\n return self._make_wrapper(func)", "response": "Adds the decorated function to this dispatch."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds the decorated function to this dispatch at the FRONT of the order.", "response": "def dispatch_first(self, func):\n '''\n Adds the decorated function to this dispatch, at the FRONT of the order.\n Useful for allowing third parties to add overloaded functionality\n to be executed before default functionality.\n '''\n self.callees.appendleft(self._make_dispatch(func))\n return self._make_wrapper(func)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndispatching a call to the first function whose type signature matches the arguemts.", "response": "def execute(self, args, kwargs):\n '''\n Dispatch a call. Call the first function whose type signature matches\n the arguemts.\n '''\n return self.lookup_explicit(args, kwargs)(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef setup_formatters(self, *args):\n formatters = []\n col_offset = 0\n # initialize formatters for row-id, timestamp and time-diff columns\n if self.rownum:\n formatters.append(fmt.RowNumberFormatter.setup(0))\n col_offset += 1\n if self.timestamp:\n formatters.append(fmt.DatetimeFormatter.setup(\n datetime.datetime.now(),\n fmt='{:%Y-%m-%d %H:%M:%S.%f}'.format,\n col_width=26))\n col_offset += 1\n if self.time_diff:\n formatters.append(fmt.TimeDeltaFormatter.setup(0))\n col_offset += 1\n\n # initialize formatters for user-defined columns\n for coli, value in enumerate(args):\n fmt_class = type2fmt.get(type(value), fmt.GenericFormatter)\n kwargs = {}\n\n # set column width\n if self.default_colwidth is not None:\n kwargs['col_width'] = self.default_colwidth\n if coli in self.column_widths:\n kwargs['col_width'] = self.column_widths[coli]\n elif self.columns and self.columns[coli + col_offset] in self.column_widths:\n kwargs['col_width'] = self.column_widths[self.columns[coli + col_offset]]\n\n # set formatter function\n if fmt_class == fmt.FloatFormatter and self.float_format is not None:\n kwargs['fmt'] = self.float_format\n if coli in self.column_formatters:\n kwargs['fmt'] = self.column_formatters[coli]\n elif self.columns and self.columns[coli + col_offset] in self.column_formatters:\n kwargs['fmt'] = self.column_formatters[self.columns[coli + col_offset]]\n\n formatter = fmt_class.setup(value, **kwargs)\n formatters.append(formatter)\n\n self.formatters = formatters", "response": "Setup formatters by observing the first row."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef setup(self, *args):\n self.setup_formatters(*args)\n if self.columns:\n self.print_header()\n elif self.border and not self.csv:\n self.print_line(self.make_horizontal_border())", "response": "Do preparations before printing the first row\n \noment."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting row values into a csv line", "response": "def csv_format(self, row):\n \"\"\"Converts row values into a csv line\n \n Args:\n row: a list of row cells as unicode\n Returns:\n csv_line (unicode)\n \"\"\"\n if PY2:\n buf = io.BytesIO()\n csvwriter = csv.writer(buf)\n csvwriter.writerow([c.strip().encode(self.encoding) for c in row])\n csv_line = buf.getvalue().decode(self.encoding).rstrip()\n else:\n buf = io.StringIO()\n csvwriter = csv.writer(buf)\n csvwriter.writerow([c.strip() for c in row])\n csv_line = buf.getvalue().rstrip()\n return csv_line"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convertShpToExtend(pathToShp):\n \n driver = ogr.GetDriverByName('ESRI Shapefile')\n dataset = driver.Open(pathToShp)\n if dataset is not None:\n # from Layer\n layer = dataset.GetLayer()\n spatialRef = layer.GetSpatialRef()\n # from Geometry\n feature = layer.GetNextFeature()\n geom = feature.GetGeometryRef()\n spatialRef = geom.GetSpatialReference()\n \n #WGS84\n outSpatialRef = osr.SpatialReference()\n outSpatialRef.ImportFromEPSG(4326)\n\n coordTrans = osr.CoordinateTransformation(spatialRef, outSpatialRef)\n\n env = geom.GetEnvelope()\n\n pointMAX = ogr.Geometry(ogr.wkbPoint)\n pointMAX.AddPoint(env[1], env[3])\n pointMAX.Transform(coordTrans)\n \n pointMIN = ogr.Geometry(ogr.wkbPoint)\n pointMIN.AddPoint(env[0], env[2])\n pointMIN.Transform(coordTrans)\n\n\n return [pointMAX.GetPoint()[1],pointMIN.GetPoint()[0],pointMIN.GetPoint()[1],pointMAX.GetPoint()[0]]\n else:\n exit(\" shapefile not found. Please verify your path to the shapefile\")", "response": "converts a shapefile to an extend shapefile"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a request for a single GFS object.", "response": "def create_request_gfs(dateStart,dateEnd,stepList,levelList,grid,extent,paramList,typeData):\n \"\"\"\n Genere la structure de requete pour le t\u00e9l\u00e9chargement de donn\u00e9es GFS\n \n INPUTS:\\n\n -date : au format annee-mois-jour\\n\n -heure : au format heure:minute:seconde\\n\n -coord : une liste des coordonnees au format [N,W,S,E]\\n\n -dim_grille : taille de la grille en degree \\n\n \"\"\"\n \n URLlist=[]\n \n #Control datetype\n listForcastSurface=['GUST','HINDEX','PRES','HGT','TMP','WEASD','SNOD','CPOFP','WILT','FLDCP','SUNSD','LFTX','CAPE','CIN','4LFTX','HPBL','LAND']\n if (0 not in [int(x) for x in stepList]):\n listForcastSurface=listForcastSurface+['PEVPR','CPRAT','PRATE','APCP','ACPCP','WATR','CSNOW','CICEP','CFPER','CRAIN','LHTFL','SHTFL','SHTFL','GFLUX','UFLX','VFLX','U-GWD','V-GWD','DSWRF','DLWRF','ULWRF','USWRF','ALBDO']\n listAnalyseSurface=['HGT','PRES','LFTX','CAPE','CIN','4LFTX']\n \n if typeData == 'analyse' and all([x in listAnalyseSurface for x in paramList]):\n typeData= 'analyse'\n validChoice = None\n prbParameters = None\n else:\n if all([x in listForcastSurface for x in paramList]) and typeData != 'cycleforecast':\n if typeData=='analyse':\n typeData= 'forecast'\n validChoice = typeData\n else:\n validChoice = None\n indexParameters=[i for i, elem in enumerate([x in listAnalyseSurface for x in paramList], 1) if not elem]\n prbParameters=[]\n for i in indexParameters:\n prbParameters.append(paramList[i-1])\n else:\n if typeData != 'cycleforecast':\n typeData= 'cycleforecast'\n validChoice = typeData\n else:\n validChoice = None\n indexParameters=[i for i, elem in enumerate([x in listAnalyseSurface for x in paramList], 1) if not elem]\n prbParameters=[]\n for i in indexParameters:\n prbParameters.append(paramList[i-1])\n \n #Control si date/timeList disponible\n today=date.today()\n lastData = today - timedelta(days=14)\n if dateStart < lastData or dateEnd > today : \n exit('date are not in 14 days range from today' )\n else:\n #Pour chaque jour souhait\u00e9\n nbDays=(dateEnd-dateStart).days+1\n for i in range(0,nbDays):\n #on crontrole pour les timeList\n if dateStart + timedelta(days=i) == today:\n maxT=datetime.now().hour-5\n timeListCorr=[ x for x in stepList if x y\n prefs = get_prefs()\n\n if app_name not in prefs:\n prefs[app_name] = OrderedDict()\n\n prefs[app_name][local_name.lower()] = proxy\n\n # Replace original pref variable with a proxy.\n locals_dict[local_name] = proxy\n proxies.append(proxy)\n\n return proxies"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nregistering dynamically created preferences models for Admin interface.", "response": "def register_admin_models(admin_site):\n \"\"\"Registers dynamically created preferences models for Admin interface.\n\n :param admin.AdminSite admin_site: AdminSite object.\n\n \"\"\"\n global __MODELS_REGISTRY\n\n prefs = get_prefs()\n\n for app_label, prefs_items in prefs.items():\n\n model_class = get_pref_model_class(app_label, prefs_items, get_app_prefs)\n\n if model_class is not None:\n __MODELS_REGISTRY[app_label] = model_class\n admin_site.register(model_class, get_pref_model_admin_class(prefs_items))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef autodiscover_siteprefs(admin_site=None):\n if admin_site is None:\n admin_site = admin.site\n\n # Do not discover anything if called from manage.py (e.g. executing commands from cli).\n if 'manage' not in sys.argv[0] or (len(sys.argv) > 1 and sys.argv[1] in MANAGE_SAFE_COMMANDS):\n import_prefs()\n Preference.read_prefs(get_prefs())\n register_admin_models(admin_site)", "response": "Automatically discovers and registers all preferences available in all apps."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef patch_locals(depth=2):\n for name, locals_dict in traverse_local_prefs(depth):\n locals_dict[name] = PatchedLocal(name, locals_dict[name])\n\n get_frame_locals(depth)[__PATCHED_LOCALS_SENTINEL] = True", "response": "Temporarily replaces all module variables\n considered preferences with PatchedLocal objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unpatch_locals(depth=3):\n for name, locals_dict in traverse_local_prefs(depth):\n if isinstance(locals_dict[name], PatchedLocal):\n locals_dict[name] = locals_dict[name].val\n\n del get_frame_locals(depth)[__PATCHED_LOCALS_SENTINEL]", "response": "Restores the original values of module variables\n considered preferences if they are still PatchedLocal\n and PrefProxy."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef proxy_settings_module(depth=3):\n proxies = []\n\n modules = sys.modules\n module_name = get_frame_locals(depth)['__name__']\n\n module_real = modules[module_name]\n\n for name, locals_dict in traverse_local_prefs(depth):\n\n value = locals_dict[name]\n\n if isinstance(value, PrefProxy):\n proxies.append(name)\n\n new_module = type(module_name, (ModuleType, ModuleProxy), {})(module_name) # ModuleProxy\n new_module.bind(module_real, proxies)\n\n modules[module_name] = new_module", "response": "Replaces a settings module with a Module proxy to intercept\n an access to settings."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nregistering preferences that should be handled by the site.", "response": "def register_prefs(*args, **kwargs):\n \"\"\"Registers preferences that should be handled by siteprefs.\n\n Expects preferences as *args.\n\n Use keyword arguments to batch apply params supported by\n ``PrefProxy`` to all preferences not constructed by ``pref`` and ``pref_group``.\n\n Batch kwargs:\n\n :param str|unicode help_text: Field help text.\n\n :param bool static: Leave this preference static (do not store in DB).\n\n :param bool readonly: Make this field read only.\n\n :param bool swap_settings_module: Whether to automatically replace settings module\n with a special ``ProxyModule`` object to access dynamic values of settings\n transparently (so not to bother with calling ``.value`` of ``PrefProxy`` object).\n\n \"\"\"\n swap_settings_module = bool(kwargs.get('swap_settings_module', True))\n\n if __PATCHED_LOCALS_SENTINEL not in get_frame_locals(2):\n raise SitePrefsException('Please call `patch_locals()` right before the `register_prefs()`.')\n\n bind_proxy(args, **kwargs)\n\n unpatch_locals()\n\n swap_settings_module and proxy_settings_module()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pref_group(title, prefs, help_text='', static=True, readonly=False):\n bind_proxy(prefs, title, help_text=help_text, static=static, readonly=readonly)\n\n for proxy in prefs: # For preferences already marked by pref().\n if isinstance(proxy, PrefProxy):\n proxy.category = title", "response": "Marks preferences group.\n\n :param str|unicode title: Group title\n\n :param list|tuple prefs: Preferences to group.\n\n :param str|unicode help_text: Field help text.\n\n :param bool static: Leave this preference static (do not store in DB).\n\n :param bool readonly: Make this field read only."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pref(preference, field=None, verbose_name=None, help_text='', static=True, readonly=False):\n try:\n bound = bind_proxy(\n (preference,),\n field=field,\n verbose_name=verbose_name,\n help_text=help_text,\n static=static,\n readonly=readonly,\n )\n return bound[0]\n\n except IndexError:\n return", "response": "Marks a preference variable."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bind(self, module, prefs):\n self._module = module\n self._prefs = set(prefs)", "response": "Bind the current module to the given list of preference names."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef generate_versionwarning_data_json(app, config=None, **kwargs):\n\n # In Sphinx >= 1.8 we use ``config-initied`` signal which comes with the\n # ``config`` object and in Sphinx < 1.8 we use ``builder-initied`` signal\n # that doesn't have the ``config`` object and we take it from the ``app``\n config = config or kwargs.pop('config', None)\n if config is None:\n config = app.config\n\n if config.versionwarning_project_version in config.versionwarning_messages:\n custom = True\n message = config.versionwarning_messages.get(config.versionwarning_project_version)\n else:\n custom = False\n message = config.versionwarning_default_message\n\n banner_html = config.versionwarning_banner_html.format(\n id_div=config.versionwarning_banner_id_div,\n banner_title=config.versionwarning_banner_title,\n message=message.format(\n **{config.versionwarning_message_placeholder: ''},\n ),\n admonition_type=config.versionwarning_admonition_type,\n )\n\n data = json.dumps({\n 'meta': {\n 'api_url': config.versionwarning_api_url,\n },\n 'banner': {\n 'html': banner_html,\n 'id_div': config.versionwarning_banner_id_div,\n 'body_selector': config.versionwarning_body_selector,\n 'custom': custom,\n },\n 'project': {\n 'slug': config.versionwarning_project_slug,\n },\n 'version': {\n 'slug': config.versionwarning_project_version,\n },\n }, indent=4)\n\n data_path = os.path.join(STATIC_PATH, 'data')\n if not os.path.exists(data_path):\n os.mkdir(data_path)\n\n with open(os.path.join(data_path, JSON_DATA_FILENAME), 'w') as f:\n f.write(data)\n\n # Add the path where ``versionwarning-data.json`` file and\n # ``versionwarning.js`` are saved\n config.html_static_path.append(STATIC_PATH)", "response": "Generate the versionwarning - data. json file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives objective functions a number of dimensions and parameter range Parameters ---------- param_scales : (int, int) Scale (std. dev.) for choosing each parameter xstar : array_like Optimal parameters", "response": "def objective(param_scales=(1, 1), xstar=None, seed=None):\n \"\"\"Gives objective functions a number of dimensions and parameter range\n\n Parameters\n ----------\n param_scales : (int, int)\n Scale (std. dev.) for choosing each parameter\n\n xstar : array_like\n Optimal parameters\n \"\"\"\n ndim = len(param_scales)\n\n def decorator(func):\n\n @wraps(func)\n def wrapper(theta):\n return func(theta)\n\n def param_init():\n np.random.seed(seed)\n return np.random.randn(ndim,) * np.array(param_scales)\n\n wrapper.ndim = ndim\n wrapper.param_init = param_init\n wrapper.xstar = xstar\n\n return wrapper\n\n return decorator"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef doublewell(theta):\n k0, k1, depth = 0.01, 100, 0.5\n shallow = 0.5 * k0 * theta ** 2 + depth\n deep = 0.5 * k1 * theta ** 2\n obj = float(np.minimum(shallow, deep))\n grad = np.where(deep < shallow, k1 * theta, k0 * theta)\n return obj, grad", "response": "Pointwise minimum of two quadratic bowls"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef camel(theta):\n x, y = theta\n obj = 2 * x ** 2 - 1.05 * x ** 4 + x ** 6 / 6 + x * y + y ** 2\n grad = np.array([\n 4 * x - 4.2 * x ** 3 + x ** 5 + y,\n x + 2 * y\n ])\n return obj, grad", "response": "Three - hump camel function"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bohachevsky1(theta):\n x, y = theta\n obj = x ** 2 + 2 * y ** 2 - 0.3 * np.cos(3 * np.pi * x) - 0.4 * np.cos(4 * np.pi * y) + 0.7\n grad = np.array([\n 2 * x + 0.3 * np.sin(3 * np.pi * x) * 3 * np.pi,\n 4 * y + 0.4 * np.sin(4 * np.pi * y) * 4 * np.pi,\n ])\n return obj, grad", "response": "One of the Bohachevsky functions"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_all_buckets(self, *args, **kwargs):\n if kwargs.pop('force', None):\n buckets = super(S3Connection, self).get_all_buckets(*args, **kwargs)\n\n for bucket in buckets:\n mimicdb.backend.sadd(tpl.connection, bucket.name)\n\n return buckets\n\n return [Bucket(self, bucket) for bucket in mimicdb.backend.smembers(tpl.connection)]", "response": "Return a list of buckets in MimicDB."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_bucket(self, bucket_name, validate=True, headers=None, force=None):\n if force:\n bucket = super(S3Connection, self).get_bucket(bucket_name, validate, headers)\n mimicdb.backend.sadd(tpl.connection, bucket.name)\n return bucket\n\n if mimicdb.backend.sismember(tpl.connection, bucket_name):\n return Bucket(self, bucket_name)\n else:\n if validate:\n raise S3ResponseError(404, 'NoSuchBucket')\n else:\n return Bucket(self, bucket_name)", "response": "Return a bucket from MimicDB if it exists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd the bucket to MimicDB after successful creation.", "response": "def create_bucket(self, *args, **kwargs):\n \"\"\"Add the bucket to MimicDB after successful creation.\n \"\"\"\n bucket = super(S3Connection, self).create_bucket(*args, **kwargs)\n\n if bucket:\n mimicdb.backend.sadd(tpl.connection, bucket.name)\n\n return bucket"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef delete_bucket(self, *args, **kwargs):\n super(S3Connection, self).delete_bucket(*args, **kwargs)\n\n bucket = kwargs.get('bucket_name', args[0] if args else None)\n\n if bucket:\n mimicdb.backend.srem(tpl.connection, bucket)", "response": "Delete the bucket on S3 before removing it from MimicDB."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the key from MimicDB.", "response": "def get_key(self, *args, **kwargs):\n \"\"\"Return the key from MimicDB.\n\n :param boolean force: If true, API call is forced to S3\n \"\"\"\n if kwargs.pop('force', None):\n headers = kwargs.get('headers', {})\n headers['force'] = True\n kwargs['headers'] = headers\n\n return super(Bucket, self).get_key(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_key_internal(self, *args, **kwargs):\n if args[1] is not None and 'force' in args[1]:\n key, res = super(Bucket, self)._get_key_internal(*args, **kwargs)\n\n if key:\n mimicdb.backend.sadd(tpl.bucket % self.name, key.name)\n mimicdb.backend.hmset(tpl.key % (self.name, key.name),\n dict(size=key.size,\n md5=key.etag.strip('\"')))\n return key, res\n\n key = None\n\n if mimicdb.backend.sismember(tpl.bucket % self.name, args[0]):\n key = Key(self)\n key.name = args[0]\n\n return key, None", "response": "Internal method that checks the S3 for the key and saves the metadata and key to the bucket set."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_all_keys(self, *args, **kwargs):\n if kwargs.pop('force', None):\n headers = kwargs.get('headers', args[0] if len(args) else None) or dict()\n headers['force'] = True\n kwargs['headers'] = headers\n\n return super(Bucket, self).get_all_keys(*args, **kwargs)", "response": "Return a list of keys from MimicDB."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete_keys(self, *args, **kwargs):\n ikeys = iter(kwargs.get('keys', args[0] if args else []))\n\n while True:\n try:\n key = ikeys.next()\n except StopIteration:\n break\n\n if isinstance(key, basestring):\n mimicdb.backend.srem(tpl.bucket % self.name, key)\n mimicdb.backend.delete(tpl.key % (self.name, key))\n elif isinstance(key, BotoKey) or isinstance(key, Key):\n mimicdb.backend.srem(tpl.bucket % self.name, key.name)\n mimicdb.backend.delete(tpl.key % (self.name, key.name))\n\n return super(Bucket, self).delete_keys(*args, **kwargs)", "response": "Remove each key or key name in an iterable from the bucket set."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove key from bucket set.", "response": "def _delete_key_internal(self, *args, **kwargs):\n \"\"\"Remove key name from bucket set.\n \"\"\"\n mimicdb.backend.srem(tpl.bucket % self.name, args[0])\n mimicdb.backend.delete(tpl.key % (self.name, args[0]))\n\n return super(Bucket, self)._delete_key_internal(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef list(self, *args, **kwargs):\n if kwargs.pop('force', None):\n headers = kwargs.get('headers', args[4] if len(args) > 4 else None) or dict()\n headers['force'] = True\n kwargs['headers'] = headers\n\n for key in super(Bucket, self).list(*args, **kwargs):\n yield key\n\n else:\n prefix = kwargs.get('prefix', args[0] if args else '')\n\n for key in mimicdb.backend.smembers(tpl.bucket % self.name):\n if key.startswith(prefix):\n k = Key(self, key)\n\n meta = mimicdb.backend.hgetall(tpl.key % (self.name, key))\n\n if meta:\n k._load_meta(meta['size'], meta['md5'])\n\n yield k", "response": "Return an iterable of keys from MimicDB."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_all(self, *args, **kwargs):\n headers = kwargs.get('headers', args[2] if len(args) > 2 else None) or dict()\n\n if 'force' in headers:\n keys = super(Bucket, self)._get_all(*args, **kwargs)\n\n for key in keys:\n mimicdb.backend.sadd(tpl.bucket % self.name, key.name)\n mimicdb.backend.hmset(tpl.key % (self.name, key.name), dict(size=key.size, md5=key.etag.strip('\"')))\n\n key.name = key.name\n\n return keys\n\n prefix = kwargs.get('prefix', '')\n\n return list(self.list(prefix=prefix))", "response": "Get all keys from S3 or MimicDB."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sync(self):\n for key in mimicdb.backend.smembers(tpl.bucket % self.name):\n mimicdb.backend.delete(tpl.key % (self.name, key))\n\n mimicdb.backend.delete(tpl.bucket % self.name)\n mimicdb.backend.sadd(tpl.connection, self.name)\n\n for key in self.list(force=True):\n mimicdb.backend.sadd(tpl.bucket % self.name, key.name)\n mimicdb.backend.hmset(tpl.key % (self.name, key.name), dict(size=key.size, md5=key.etag.strip('\"')))", "response": "Sync a bucket.\n\n Force all API calls to S3 and populate the database with the current state of S3."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the sparse probability of a single node.", "response": "def sparse(x, rho, penalty):\n \"\"\"\n Proximal operator for the l1-norm: soft thresholding\n\n Parameters\n ----------\n penalty : float\n Strength or weight on the l1-norm\n \"\"\"\n\n lmbda = penalty / rho\n return (x - lmbda) * (x >= lmbda) + (x + lmbda) * (x <= -lmbda)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lbfgs(x, rho, f_df, maxiter=20):\n\n def f_df_augmented(theta):\n f, df = f_df(theta)\n obj = f + (rho / 2.) * np.linalg.norm(theta - x) ** 2\n grad = df + rho * (theta - x)\n return obj, grad\n\n res = scipy_minimize(f_df_augmented, x, jac=True, method='L-BFGS-B',\n options={'maxiter': maxiter, 'disp': False})\n\n return res.x", "response": "Minimize the proximal operator of a given objective using L - BFGS - B method."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef smooth(x, rho, penalty, axis=0, newshape=None):\n\n orig_shape = x.shape\n\n if newshape is not None:\n x = x.reshape(newshape)\n\n # Apply Laplacian smoothing (l2 norm on the parameters multiplied by\n # the laplacian)\n n = x.shape[axis]\n lap_op = spdiags([(2 + rho / penalty) * np.ones(n),\n -1 * np.ones(n), -1 * np.ones(n)],\n [0, -1, 1], n, n, format='csc')\n\n A = penalty * lap_op\n b = rho * np.rollaxis(x, axis, 0)\n return np.rollaxis(spsolve(A, b), axis, 0).reshape(orig_shape)", "response": "Applies a smoothing operator along one dimension to a matrix x."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\napply a proximal operator to the columns of a matrix", "response": "def columns(x, rho, proxop):\n \"\"\"Applies a proximal operator to the columns of a matrix\"\"\"\n\n xnext = np.zeros_like(x)\n\n for ix in range(x.shape[1]):\n xnext[:, ix] = proxop(x[:, ix], rho)\n\n return xnext"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fantope(x, rho, dim, tol=1e-4):\n\n U, V = np.linalg.eigh(x)\n\n minval, maxval = np.maximum(U.min(), 0), np.maximum(U.max(), 20 * dim)\n\n while True:\n\n theta = 0.5 * (maxval + minval)\n thr_eigvals = np.minimum(np.maximum((U - theta), 0), 1)\n constraint = np.sum(thr_eigvals)\n\n if np.abs(constraint - dim) <= tol:\n break\n\n elif constraint < dim:\n maxval = theta\n\n elif constraint > dim:\n minval = theta\n\n else:\n break\n\n return np.linalg.multi_dot((V, np.diag(thr_eigvals), V.T))", "response": "Fantope projection and selection of Aracite Aracite."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nturn a coroutine into a gradient based optimizer.", "response": "def gradient_optimizer(coro):\n \"\"\"Turns a coroutine into a gradient based optimizer.\"\"\"\n\n class GradientOptimizer(Optimizer):\n\n @wraps(coro)\n def __init__(self, *args, **kwargs):\n self.algorithm = coro(*args, **kwargs)\n self.algorithm.send(None)\n self.operators = []\n\n def set_transform(self, func):\n self.transform = compose(destruct, func, self.restruct)\n\n def minimize(self, f_df, x0, display=sys.stdout, maxiter=1e3):\n\n self.display = display\n self.theta = x0\n\n # setup\n xk = self.algorithm.send(destruct(x0).copy())\n store = defaultdict(list)\n runtimes = []\n if len(self.operators) == 0:\n self.operators = [proxops.identity()]\n\n # setup\n obj, grad = wrap(f_df, x0)\n transform = compose(destruct, *reversed(self.operators), self.restruct)\n\n self.optional_print(tp.header(['Iteration', 'Objective', '||Grad||', 'Runtime']))\n try:\n for k in count():\n\n # setup\n tstart = perf_counter()\n f = obj(xk)\n df = grad(xk)\n xk = transform(self.algorithm.send(df))\n runtimes.append(perf_counter() - tstart)\n store['f'].append(f)\n\n # Update display\n self.optional_print(tp.row([k,\n f,\n np.linalg.norm(destruct(df)),\n tp.humantime(runtimes[-1])]))\n\n if k >= maxiter:\n break\n\n except KeyboardInterrupt:\n pass\n\n self.optional_print(tp.bottom(4))\n\n # cleanup\n self.optional_print(u'\\u279b Final objective: {}'.format(store['f'][-1]))\n self.optional_print(u'\\u279b Total runtime: {}'.format(tp.humantime(sum(runtimes))))\n self.optional_print(u'\\u279b Per iteration runtime: {} +/- {}'.format(\n tp.humantime(np.mean(runtimes)),\n tp.humantime(np.std(runtimes)),\n ))\n\n # result\n return OptimizeResult({\n 'x': self.restruct(xk),\n 'f': f,\n 'df': self.restruct(df),\n 'k': k,\n 'obj': np.array(store['f']),\n })\n\n return GradientOptimizer"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add(self, operator, *args):\n\n if isinstance(operator, str):\n op = getattr(proxops, operator)(*args)\n elif isinstance(operator, proxops.ProximalOperatorBaseClass):\n op = operator\n else:\n raise ValueError(\"operator must be a string or a subclass of ProximalOperator\")\n\n self.operators.append(op)\n return self", "response": "Adds a proximal operator to the list of operators"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload the key attributes from the cache.", "response": "def _load_meta(self, size, md5):\n \"\"\"Set key attributes to retrived metadata. Might be extended in the\n future to support more attributes.\n \"\"\"\n if not hasattr(self, 'local_hashes'):\n self.local_hashes = {}\n\n self.size = int(size)\n\n if (re.match('^[a-fA-F0-9]{32}$', md5)):\n self.md5 = md5"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef name(self, value):\n self._name = value\n\n if value:\n meta = mimicdb.backend.hgetall(tpl.key % (self.bucket.name, value))\n\n if meta:\n mimicdb.backend.sadd(tpl.bucket % self.bucket.name, value)\n self._load_meta(meta['size'], meta['md5'])", "response": "Set the name of the key."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _send_file_internal(self, *args, **kwargs):\n super(Key, self)._send_file_internal(*args, **kwargs)\n\n mimicdb.backend.sadd(tpl.bucket % self.bucket.name, self.name)\n mimicdb.backend.hmset(tpl.key % (self.bucket.name, self.name),\n dict(size=self.size, md5=self.md5))", "response": "Called internally for any type of upload."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef wrap(f_df, xref, size=1):\n memoized_f_df = lrucache(lambda x: f_df(restruct(x, xref)), size)\n objective = compose(first, memoized_f_df)\n gradient = compose(destruct, second, memoized_f_df)\n return objective, gradient", "response": "Wraps a function f_df into a memoized objective + gradient function and splits it into two functions that return just the objective and gradient respectively."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndecorates a function with the given docstring Parameters ---------- docstr : string", "response": "def docstring(docstr):\n \"\"\"\n Decorates a function with the given docstring\n\n Parameters\n ----------\n docstr : string\n \"\"\"\n def decorator(func):\n @wraps(func)\n def wrapper(*args, **kwargs):\n return func(*args, **kwargs)\n wrapper.__doc__ = docstr\n return wrapper\n return decorator"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks the numerical gradient of a function and outputs a table of the correct and incorrect gradients.", "response": "def check_grad(f_df, xref, stepsize=1e-6, tol=1e-6, width=15, style='round', out=sys.stdout):\n \"\"\"\n Compares the numerical gradient to the analytic gradient\n\n Parameters\n ----------\n f_df : function\n The analytic objective and gradient function to check\n\n x0 : array_like\n Parameter values to check the gradient at\n\n stepsize : float, optional\n Stepsize for the numerical gradient. Too big and this will poorly estimate the gradient.\n Too small and you will run into precision issues (default: 1e-6)\n\n tol : float, optional\n Tolerance to use when coloring correct/incorrect gradients (default: 1e-5)\n\n width : int, optional\n Width of the table columns (default: 15)\n\n style : string, optional\n Style of the printed table, see tableprint for a list of styles (default: 'round')\n \"\"\"\n CORRECT = u'\\x1b[32m\\N{CHECK MARK}\\x1b[0m'\n INCORRECT = u'\\x1b[31m\\N{BALLOT X}\\x1b[0m'\n\n obj, grad = wrap(f_df, xref, size=0)\n x0 = destruct(xref)\n df = grad(x0)\n\n # header\n out.write(tp.header([\"Numerical\", \"Analytic\", \"Error\"], width=width, style=style) + \"\\n\")\n out.flush()\n\n # helper function to parse a number\n def parse_error(number):\n\n # colors\n failure = \"\\033[91m\"\n passing = \"\\033[92m\"\n warning = \"\\033[93m\"\n end = \"\\033[0m\"\n base = \"{}{:0.3e}{}\"\n\n # correct\n if error < 0.1 * tol:\n return base.format(passing, error, end)\n\n # warning\n elif error < tol:\n return base.format(warning, error, end)\n\n # failure\n else:\n return base.format(failure, error, end)\n\n # check each dimension\n num_errors = 0\n for j in range(x0.size):\n\n # take a small step in one dimension\n dx = np.zeros(x0.size)\n dx[j] = stepsize\n\n # compute the centered difference formula\n df_approx = (obj(x0 + dx) - obj(x0 - dx)) / (2 * stepsize)\n df_analytic = df[j]\n\n # absolute error\n abs_error = np.linalg.norm(df_approx - df_analytic)\n\n # relative error\n error = abs_error if np.allclose(abs_error, 0) else abs_error / \\\n (np.linalg.norm(df_analytic) + np.linalg.norm(df_approx))\n\n num_errors += error >= tol\n errstr = CORRECT if error < tol else INCORRECT\n out.write(tp.row([df_approx, df_analytic, parse_error(error) + ' ' + errstr],\n width=width, style=style) + \"\\n\")\n out.flush()\n\n out.write(tp.bottom(3, width=width, style=style) + \"\\n\")\n return num_errors"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nevaluate the files identified for checksum.", "response": "def evaluate(self, repo, spec, args):\n \"\"\"\n Evaluate the files identified for checksum.\n \"\"\"\n\n status = []\n\n # Do we have to any thing at all? \n if len(spec['files']) == 0: \n return status \n\n with cd(repo.rootdir):\n \n rules = None \n if 'rules-files' in spec and len(spec['rules-files']) > 0: \n rulesfiles = spec['rules-files']\n rules = {} \n for f in rulesfiles: \n d = json.loads(open(f).read())\n rules.update(d)\n elif 'rules' in spec: \n rules = {\n 'inline': spec['rules'] \n }\n \n if rules is None or len(rules) == 0:\n print(\"Regression quality validation has been enabled but no rules file has been specified\")\n print(\"Example: { 'min-r2': 0.25 }. Put this either in file or in dgit.json\")\n raise InvalidParameters(\"Regression quality checking rules missing\")\n\n files = dict([(f, open(f).read()) for f in spec['files']])\n\n for r in rules:\n if 'min-r2' not in rules[r]:\n continue\n minr2 = float(rules[r]['min-r2'])\n for f in files:\n match = re.search(r\"R-squared:\\s+(\\d.\\d+)\", files[f])\n if match is None:\n status.append({\n 'target': f,\n 'validator': self.name,\n 'description': self.description,\n 'rules': r,\n 'status': \"ERROR\",\n 'message': \"Invalid model output\"\n })\n else:\n r2 = match.group(1)\n r2 = float(r2)\n if r2 > minr2:\n status.append({\n 'target': f,\n 'validator': self.name,\n 'description': self.description,\n 'rules': r,\n 'status': \"OK\",\n 'message': \"Acceptable R2\"\n })\n else:\n status.append({\n 'target': f,\n 'validator': self.name,\n 'description': self.description,\n 'rules': r,\n 'status': \"ERROR\",\n 'message': \"R2 is too low\"\n })\n\n return status"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nevaluates the integrity of the datapackage. json", "response": "def evaluate(self, repo, spec, args):\n \"\"\"\n Check the integrity of the datapackage.json\n \"\"\"\n\n status = []\n with cd(repo.rootdir):\n files = spec.get('files', ['*'])\n resource_files = repo.find_matching_files(files)\n files = glob2.glob(\"**/*\")\n disk_files = [f for f in files if os.path.isfile(f) and f != \"datapackage.json\"]\n\n allfiles = list(set(resource_files + disk_files))\n allfiles.sort()\n\n for f in allfiles:\n if f in resource_files and f in disk_files:\n r = repo.get_resource(f)\n coded_sha256 = r['sha256']\n computed_sha256 = compute_sha256(f)\n if computed_sha256 != coded_sha256:\n status.append({\n 'target': f,\n 'rules': \"\",\n 'validator': self.name,\n 'description': self.description,\n 'status': 'ERROR',\n 'message': \"Mismatch in checksum on disk and in datapackage.json\"\n })\n else:\n status.append({\n 'target': f,\n 'rules': \"\",\n 'validator': self.name,\n 'description': self.description,\n 'status': 'OK',\n 'message': \"\"\n })\n elif f in resource_files:\n status.append({\n 'target': f,\n 'rules': \"\",\n 'validator': self.name,\n 'description': self.description,\n 'status': 'ERROR',\n 'message': \"In datapackage.json but not in repo\"\n })\n else:\n status.append({\n 'target': f,\n 'rules': \"\",\n 'validator': self.name,\n 'description': self.description,\n 'status': 'ERROR',\n 'message': \"In repo but not in datapackage.json\"\n })\n\n\n return status"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads the file into a table set.", "response": "def read_file(self, filename): \n \"\"\"\n Guess the filetype and read the file into row sets\n \"\"\"\n #print(\"Reading file\", filename)\n\n try:\n fh = open(filename, 'rb')\n table_set = any_tableset(fh) # guess the type...\n except:\n #traceback.print_exc()\n # Cannot find the schema.\n table_set = None\n \n return table_set"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_schema(self, filename):\n table_set = self.read_file(filename)\n \n # Have I been able to read the filename\n if table_set is None: \n return [] \n\n # Get the first table as rowset\n row_set = table_set.tables[0]\n\n offset, headers = headers_guess(row_set.sample)\n row_set.register_processor(headers_processor(headers))\n row_set.register_processor(offset_processor(offset + 1))\n types = type_guess(row_set.sample, strict=True)\n\n # Get a sample as well..\n sample = next(row_set.sample)\n\n clean = lambda v: str(v) if not isinstance(v, str) else v \n schema = []\n for i, h in enumerate(headers):\n schema.append([h,\n str(types[i]),\n clean(sample[i].value)])\n\n return schema", "response": "Guess schema using messytables\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef int2fin_reference(n):\n checksum = 10 - (sum([int(c) * i for c, i in zip(str(n)[::-1], it.cycle((7, 3, 1)))]) % 10)\n if checksum == 10:\n checksum = 0\n return \"%s%s\" % (n, checksum)", "response": "Calculates a checksum for a Finnish national reference number"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef iso_reference_str2int(n):\n n = n.upper()\n numbers = []\n for c in n:\n iso_reference_valid_char(c)\n if c in ISO_REFERENCE_VALID_NUMERIC:\n numbers.append(c)\n else:\n numbers.append(str(iso_reference_char2int(c)))\n return int(''.join(numbers))", "response": "Creates the huge number from an ISO alphanumeric ISO reference string"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates ISO reference number", "response": "def iso_reference_isvalid(ref):\n \"\"\"Validates ISO reference number\"\"\"\n ref = str(ref)\n cs_source = ref[4:] + ref[:4]\n return (iso_reference_str2int(cs_source) % 97) == 1"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef barcode(iban, reference, amount, due=None):\n\n iban = iban.replace(' ', '')\n reference = reference.replace(' ', '')\n\n if reference.startswith('RF'):\n version = 5\n else:\n version = 4\n\n if version == 5:\n reference = reference[2:] # test RF and add 00 where needed\n if len(reference) < 23:\n reference = reference[:2] + (\"0\" * (23 - len(reference))) + reference[2:]\n elif version == 4:\n reference = reference.zfill(20)\n\n if not iban.startswith('FI'):\n raise BarcodeException('Barcodes can be printed only for IBANs starting with FI')\n\n iban = iban[2:]\n amount = \"%08d\" % (amount.quantize(Decimal('.01')).shift(2).to_integral_value())\n if len(amount) != 8:\n raise BarcodeException(\"Barcode payment amount must be less than 1000000.00\")\n\n if due:\n due = due.strftime(\"%y%m%d\")\n else:\n due = \"000000\"\n\n if version == 4:\n barcode = \"%s%s%s000%s%s\" % (version, iban, amount, reference, due)\n elif version == 5:\n barcode = \"%s%s%s%s%s\" % (version, iban, amount, reference, due)\n\n return barcode", "response": "Calculates virtual barcode for given IBAN account number and ISO reference"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_file_normal(f, targetdir, generator,script, source):\n\n basename = os.path.basename(f)\n if targetdir != \".\":\n relativepath = os.path.join(targetdir, basename)\n else:\n relativepath = basename\n\n relpath = os.path.relpath(f, os.getcwd())\n filetype = 'data'\n if script:\n filetype = 'script'\n if generator:\n filetype = 'generator'\n\n update = OrderedDict([\n ('type', filetype),\n ('generator', generator),\n ('relativepath', relativepath),\n ('content', \"\"),\n ('source', source),\n ('localfullpath', f),\n ('localrelativepath', relpath)\n ])\n\n update = annotate_record(update)\n\n return (basename, update)", "response": "Add a normal file including its source"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef extract_files(filename, includes):\n\n # Load the execution strace log\n lines = open(filename).readlines()\n\n # Extract only open files - whether for read or write. You often\n # want to capture the json/ini configuration file as well\n files = {}\n lines = [l.strip() for l in lines if 'open(' in l]\n for l in lines:\n\n # Check both these formats...\n # 20826 open(\"/usr/lib/locale/locale-archive\", O_RDONLY|O_CLOEXEC) = 3\n #[28940] access(b'/etc/ld.so.nohwcap', F_OK) = -2 (No such file or directory)\n matchedfile = re.search('open\\([b][\"\\'](.+?)[\"\\']', l)\n if matchedfile is None:\n matchedfile = re.search('open\\(\"(.+?)\\\"', l)\n\n if matchedfile is None:\n continue\n\n matchedfile = matchedfile.group(1)\n\n if os.path.exists(matchedfile) and os.path.isfile(matchedfile):\n\n #print(\"Looking at \", matchedfile)\n\n # Check what action is being performed on these\n action = 'input' if 'O_RDONLY' in l else 'output'\n\n matchedfile = os.path.relpath(matchedfile, \".\")\n #print(\"Matched file's relative path\", matchedfile)\n\n for i in includes:\n if fnmatch.fnmatch(matchedfile, i):\n # Exclude python libraries\n if 'site-packages' in matchedfile:\n continue\n if matchedfile not in files:\n files[matchedfile] = [action]\n else:\n if action not in files[matchedfile]:\n files[matchedfile].append(action)\n\n # A single file may be opened and closed multiple times\n\n if len(files) == 0:\n print(\"No input or output files found that match pattern\")\n return []\n\n print('We captured files that matched the pattern you specified.')\n print('Please select files to keep (press ENTER)')\n\n # Let the user have the final say on which files must be included.\n filenames = list(files.keys())\n filenames.sort()\n with tempfile.NamedTemporaryFile(suffix=\".tmp\") as temp:\n temp.write(yaml.dump(filenames, default_flow_style=False).encode('utf-8'))\n temp.flush()\n EDITOR = os.environ.get('EDITOR','/usr/bin/vi')\n subprocess.call(\"%s %s\" %(EDITOR,temp.name), shell=True)\n temp.seek(0)\n data = temp.read()\n selected = yaml.load(data)\n\n print(\"You selected\", len(selected), \"file(s)\")\n if len(selected) == 0:\n return []\n\n # Get the action corresponding to the selected files\n filenames = [f for f in filenames if f in selected]\n\n # Now we know the list of files. Where should they go?\n print('Please select target locations for the various directories we found')\n print('Please make sure you do not delete any rows or edit the keys.')\n input('(press ENTER)')\n prefixes = {}\n for f in filenames:\n dirname = os.path.dirname(f)\n if dirname == \"\":\n dirname = \".\"\n prefixes[dirname] = dirname\n\n while True:\n with tempfile.NamedTemporaryFile(suffix=\".tmp\") as temp:\n temp.write(yaml.dump(prefixes, default_flow_style=False).encode('utf-8'))\n temp.flush()\n EDITOR = os.environ.get('EDITOR','/usr/bin/vi')\n subprocess.call(\"%s %s\" %(EDITOR,temp.name), shell=True)\n temp.seek(0)\n data = temp.read()\n try:\n revised = yaml.load(data)\n except Exception as e:\n revised = {}\n\n #print(list(revised.keys()))\n #print(list(prefixes.keys()))\n\n if set(list(revised.keys())) == set(list(prefixes.keys())):\n prefixes = revised\n break\n else:\n print(\"Could not process edited file. Either some rows are missing or entry has YAML syntax errors\")\n input(\"Press ENTER to continue\")\n\n # Add the root directory back\n if \".\" in prefixes:\n prefixes[\"\"] = prefixes[\".\"]\n\n result = []\n ts = datetime.now().isoformat()\n for f in filenames:\n relativepath = prefixes[os.path.dirname(f)]\n if relativepath == \".\":\n relativepath = os.path.basename(f)\n else:\n relativepath = os.path.join(relativepath, os.path.basename(f))\n\n result.append(OrderedDict([\n ('relativepath', relativepath),\n ('type', 'run-output'),\n ('actions', files[f]),\n ('mimetypes', mimetypes.guess_type(f)[0]),\n ('content', open(f).read(512)),\n ('sha256', compute_sha256(f)),\n ('ts', ts),\n ('localrelativepath', os.path.relpath(f, \".\")),\n ('localfullpath', os.path.abspath(f)),\n ]))\n\n print(json.dumps(result, indent=4))\n return result", "response": "Extract the files that are being added based on the includes"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run_executable(repo, args, includes):\n\n # Get platform information\n mgr = plugins_get_mgr()\n repomgr = mgr.get(what='instrumentation', name='platform')\n platform_metadata = repomgr.get_metadata()\n\n print(\"Obtaining Commit Information\")\n (executable, commiturl) = \\\n find_executable_commitpath(repo, args)\n\n # Create a local directory\n tmpdir = tempfile.mkdtemp()\n\n # Construct the strace command\n print(\"Running the command\")\n strace_filename = os.path.join(tmpdir,'strace.out.txt')\n cmd = [\"strace.py\", \"-f\", \"-o\", strace_filename,\n \"-s\", \"1024\", \"-q\", \"--\"] + args\n\n # Run the command\n p = subprocess.Popen(cmd,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n out, err = p.communicate()\n\n # Capture the stdout/stderr\n stdout = os.path.join(tmpdir, 'stdout.log.txt')\n with open(stdout, 'w') as fd:\n fd.write(out.decode('utf-8'))\n\n stderr = os.path.join(tmpdir, 'stderr.log.txt')\n with open(stderr, 'w') as fd:\n fd.write(err.decode('utf-8'))\n\n\n # Check the strace output\n files = extract_files(strace_filename, includes)\n\n\n # Now insert the execution metadata\n execution_metadata = {\n 'likelyexecutable': executable,\n 'commitpath': commiturl,\n 'args': args,\n }\n execution_metadata.update(platform_metadata)\n\n for i in range(len(files)):\n files[i]['execution_metadata'] = execution_metadata\n\n return files", "response": "Run the executable and capture the input and output..."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add(repo, args, targetdir,\n execute=False, generator=False,\n includes=[], script=False,\n source=None):\n \"\"\"\n Add files to the repository by explicitly specifying them or by\n specifying a pattern over files accessed during execution of an\n executable.\n\n Parameters\n ----------\n\n repo: Repository\n\n args: files or command line\n (a) If simply adding files, then the list of files that must\n be added (including any additional arguments to be passed to\n git\n (b) If files to be added are an output of a command line, then\n args is the command lined\n targetdir: Target directory to store the files\n execute: Args are not files to be added but scripts that must be run.\n includes: patterns used to select files to\n script: Is this a script?\n generator: Is this a generator\n source: Link to the original source of the data\n\n \"\"\"\n\n # Gather the files...\n if not execute:\n files = add_files(args=args,\n targetdir=targetdir,\n source=source,\n script=script,\n generator=generator)\n else:\n files = run_executable(repo, args, includes)\n\n if files is None or len(files) == 0:\n return repo\n\n\n # Update the repo package but with only those that have changed.\n\n filtered_files = []\n package = repo.package\n for h in files:\n found = False\n for i, r in enumerate(package['resources']):\n if h['relativepath'] == r['relativepath']:\n found = True\n if h['sha256'] == r['sha256']:\n change = False\n for attr in ['source']:\n if h[attr] != r[attr]:\n r[attr] = h[attr]\n change = True\n if change:\n filtered_files.append(h)\n continue\n else:\n filtered_files.append(h)\n package['resources'][i] = h\n break\n if not found:\n filtered_files.append(h)\n package['resources'].append(h)\n\n if len(filtered_files) == 0:\n return 0\n\n # Copy the files\n repo.manager.add_files(repo, filtered_files)\n\n # Write to disk...\n rootdir = repo.rootdir\n with cd(rootdir):\n datapath = \"datapackage.json\"\n with open(datapath, 'w') as fd:\n fd.write(json.dumps(package, indent=4))\n\n return len(filtered_files)", "response": "Add files to the repository."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_matching_files(self, includes):\n\n if len(includes) == 0: \n return [] \n\n files = [f['relativepath'] for f in self.package['resources']]\n includes = r'|'.join([fnmatch.translate(x) for x in includes])\n\n # Match both the file name as well the path..\n files = [f for f in files if re.match(includes, os.path.basename(f))] + \\\n [f for f in files if re.match(includes, f)]\n files = list(set(files))\n\n return files", "response": "Find files that match patterns"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun a specific command using the manager", "response": "def run(self, cmd, *args):\n \"\"\"\n Run a specific command using the manager\n \"\"\"\n if self.manager is None:\n raise Exception(\"Fatal internal error: Missing repository manager\")\n if cmd not in dir(self.manager):\n raise Exception(\"Fatal internal error: Invalid command {} being run\".format(cmd))\n func = getattr(self.manager, cmd)\n repo = self\n return func(repo, *args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting metadata for a given file", "response": "def get_resource(self, p):\n \"\"\"\n Get metadata for a given file\n \"\"\"\n for r in self.package['resources']:\n if r['relativepath'] == p:\n r['localfullpath'] = os.path.join(self.rootdir, p)\n return r\n\n raise Exception(\"Invalid path\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef lookup(self, username=None, reponame=None, key=None):\n if key is None:\n key = self.key(username, reponame)\n if key not in self.repos:\n raise UnknownRepository()\n\n return self.repos[key]", "response": "Lookup a specific user in the registry."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rootdir(self, username, reponame, create=True):\n path = os.path.join(self.workspace,\n 'datasets',\n username,\n reponame)\n if create:\n try:\n os.makedirs(path)\n except:\n pass\n\n return path", "response": "Return the root directory for the repo."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a repository to the internal lookup table...", "response": "def add(self, repo):\n \"\"\"\n Add repo to the internal lookup table...\n \"\"\"\n key = self.key(repo.username, repo.reponame)\n repo.key = key\n self.repos[key] = repo\n return key"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlist all repos in the", "response": "def list_repos(remote=False):\n \"\"\"\n List repos\n\n Parameters\n ----------\n\n remote: Flag\n \"\"\"\n mgr = plugins_get_mgr()\n\n if not remote:\n repomgr = mgr.get(what='repomanager', name='git')\n repos = repomgr.get_repo_list()\n repos.sort()\n return repos\n else:\n raise Exception(\"Not supported yet\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning a shell command within the repo s rootdir", "response": "def shellcmd(repo, args):\n \"\"\"\n Run a shell command within the repo's context\n\n Parameters\n ----------\n\n repo: Repository object\n args: Shell command\n \"\"\"\n with cd(repo.rootdir):\n result = run(args)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if the datapackage exists...", "response": "def datapackage_exists(repo):\n \"\"\"\n Check if the datapackage exists...\n \"\"\"\n datapath = os.path.join(repo.rootdir, \"datapackage.json\")\n return os.path.exists(datapath)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes files in the specified object.", "response": "def delete(repo, args=[]):\n \"\"\"\n Delete files\n\n Parameters\n ----------\n\n repo: Repository object\n args: Arguments to git command\n \"\"\"\n\n # Remove the files \n result = generic_repo_cmd(repo, 'delete', args)\n if result['status'] != 'success': \n return status \n\n with cd(repo.rootdir): \n \n package = repo.package \n resources = package['resources'] \n\n cleaned_resources = []\n for r in resources: \n relativepath = r['relativepath'] \n sha256 = r['sha256'] \n if relativepath not in ['', None]: \n if not os.path.exists(relativepath): \n # This file does not exist on disk. \n print(\"Skipping\", relativepath) \n continue \n cleaned_resources.append(r) \n \n package['resources'] = cleaned_resources \n repo.package = package \n \n with open('datapackage.json', 'w') as fd: \n fd.write(json.dumps(repo.package, indent=4))\n\n return {\n 'status': 'success',\n 'message': ''\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbootstrapping the datapackage file for a repository.", "response": "def bootstrap_datapackage(repo, force=False,\n options=None, noinput=False):\n \"\"\"\n Create the datapackage file..\n \"\"\"\n\n print(\"Bootstrapping datapackage\")\n\n # get the directory\n tsprefix = datetime.now().date().isoformat()\n\n # Initial data package json\n package = OrderedDict([\n ('title', ''),\n ('description', ''),\n ('username', repo.username),\n ('reponame', repo.reponame),\n ('name', str(repo)),\n ('title', \"\"),\n ('description', \"\"),\n ('keywords', []),\n ('resources', []),\n ('creator', getpass.getuser()),\n ('createdat', datetime.now().isoformat()),\n ('remote-url', repo.remoteurl)\n ])\n\n if options is not None:\n package['title'] = options['title']\n package['description'] = options['description']\n else:\n if noinput:\n raise IncompleteParameters(\"Option field with title and description\")\n\n for var in ['title', 'description']:\n value = ''\n while value in ['',None]:\n value = input('Your Repo ' + var.title() + \": \")\n if len(value) == 0:\n print(\"{} cannot be empty. Please re-enter.\".format(var.title()))\n\n package[var] = value\n\n\n # Now store the package...\n (handle, filename) = tempfile.mkstemp()\n with open(filename, 'w') as fd:\n fd.write(json.dumps(package, indent=4))\n\n repo.package = package\n\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef init(username, reponame, setup,\n force=False, options=None,\n noinput=False):\n \"\"\"\n Initialize an empty repository with datapackage.json\n\n Parameters\n ----------\n\n username: Name of the user\n reponame: Name of the repo\n setup: Specify the 'configuration' (git only, git+s3 backend etc)\n force: Force creation of the files\n options: Dictionary with content of dgit.json, if available.\n noinput: Automatic operation with no human interaction\n \"\"\"\n\n mgr = plugins_get_mgr()\n repomgr = mgr.get(what='repomanager', name='git')\n\n backendmgr = None\n if setup == 'git+s3':\n backendmgr = mgr.get(what='backend', name='s3')\n\n repo = repomgr.init(username, reponame, force, backendmgr)\n\n # Now bootstrap the datapackage.json metadata file and copy it in...\n\n # Insert a gitignore with .dgit directory in the repo. This\n # directory will be used to store partial results\n (handle, gitignore) = tempfile.mkstemp()\n with open(gitignore, 'w') as fd:\n fd.write(\".dgit\")\n\n # Try to bootstrap. If you cant, cleanup and return\n try:\n filename = bootstrap_datapackage(repo, force, options, noinput)\n except Exception as e:\n repomgr.drop(repo,[])\n os.unlink(gitignore)\n raise e\n\n repo.run('add_files',\n [\n {\n 'relativepath': 'datapackage.json',\n 'localfullpath': filename,\n },\n {\n 'relativepath': '.gitignore',\n 'localfullpath': gitignore,\n },\n ])\n\n\n # Cleanup temp files\n os.unlink(filename)\n os.unlink(gitignore)\n\n args = ['-a', '-m', 'Bootstrapped the repo']\n repo.run('commit', args)\n \n return repo", "response": "Initialize an empty repository with datapackage. json and return it"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncloning a URL and return a new repository.", "response": "def clone(url):\n \"\"\"\n Clone a URL. Examples include:\n\n - git@github.com:pingali/dgit.git\n - https://github.com:pingali/dgit.git\n - s3://mybucket/git/pingali/dgit.git\n\n Parameters\n ----------\n\n url: URL of the repo\n\n \"\"\"\n backend = None\n backendmgr = None\n if url.startswith('s3'):\n backendtype = 's3'\n elif url.startswith(\"http\") or url.startswith(\"git\"):\n backendtype = 'git'\n else:\n backendtype = None\n\n mgr = plugins_get_mgr()\n repomgr = mgr.get(what='repomanager', name='git')\n backendmgr = mgr.get(what='backend', name=backendtype)\n\n # print(\"Testing {} with backend {}\".format(url, backendmgr))\n if backendmgr is not None and not backendmgr.url_is_valid(url):\n raise InvalidParameters(\"Invalid URL\")\n\n key = repomgr.clone(url, backendmgr)\n\n # Insert a datapackage if it doesnt already exist...\n repo = repomgr.lookup(key=key)\n if not datapackage_exists(repo):\n filename = bootstrap_datapackage(repo)\n repo.run('add_files',\n [\n {\n 'relativepath': 'datapackage.json',\n 'localfullpath': filename,\n },\n ])\n os.unlink(filename)\n args = ['-a', '-m', 'Bootstrapped the repo']\n repo.run('commit', args)\n\n return repo"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef annotate_metadata_data(repo, task, patterns=[\"*\"], size=0):\n\n mgr = plugins_get_mgr() \n keys = mgr.search('representation')['representation']\n representations = [mgr.get_by_key('representation', k) for k in keys]\n\n matching_files = repo.find_matching_files(patterns)\n package = repo.package\n rootdir = repo.rootdir\n files = package['resources']\n for f in files:\n relativepath = f['relativepath']\n if relativepath in matching_files:\n path = os.path.join(rootdir, relativepath)\n if task == 'preview':\n print(\"Adding preview for \", relativepath)\n f['content'] = open(path).read()[:size]\n elif task == 'schema':\n for r in representations: \n if r.can_process(path): \n print(\"Adding schema for \", path)\n f['schema'] = r.get_schema(path)\n break", "response": "Update metadata with the content of the files that match the patterns"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds the code to the metadata for the files in files.", "response": "def annotate_metadata_code(repo, files):\n \"\"\"\n Update metadata with the commit information\n \"\"\"\n\n package = repo.package\n package['code'] = []\n for p in files:\n matching_files = glob2.glob(\"**/{}\".format(p))\n for f in matching_files:\n absf = os.path.abspath(f)\n print(\"Add commit data for {}\".format(f))\n package['code'].append(OrderedDict([\n ('script', f),\n ('permalink', repo.manager.permalink(repo, absf)),\n ('mimetypes', mimetypes.guess_type(absf)[0]),\n ('sha256', compute_sha256(absf))\n ]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating metadata with the actions in the history of actions", "response": "def annotate_metadata_action(repo):\n \"\"\"\n Update metadata with the action history \n \"\"\"\n package = repo.package \n\n print(\"Including history of actions\")\n with cd(repo.rootdir): \n filename = \".dgit/log.json\" \n if os.path.exists(filename): \n history = open(filename).readlines() \n actions = []\n for a in history: \n try: \n a = json.loads(a)\n for x in ['code']: \n if x not in a or a[x] == None: \n a[x] = \"...\"\n actions.append(a)\n except:\n pass \n package['actions'] = actions"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef annotate_metadata_platform(repo):\n\n print(\"Added platform information\")\n package = repo.package\n mgr = plugins_get_mgr()\n repomgr = mgr.get(what='instrumentation', name='platform')\n package['platform'] = repomgr.get_metadata()", "response": "Add platform information to the metadata host information"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef annotate_metadata_dependencies(repo):\n\n options = repo.options\n\n if 'dependencies' not in options:\n print(\"No dependencies\")\n return []\n\n repos = []\n dependent_repos = options['dependencies']\n for d in dependent_repos:\n if \"/\" not in d:\n print(\"Invalid dependency specification\")\n (username, reponame) = d.split(\"/\")\n try:\n repos.append(repo.manager.lookup(username, reponame))\n except:\n print(\"Repository does not exist. Please create one\", d)\n\n package = repo.package\n package['dependencies'] = []\n for r in repos:\n package['dependencies'].append({\n 'username': r.username,\n 'reponame': r.reponame,\n })", "response": "Annotate metadata dependencies of the dependent repo with information from the dependent repo s\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef post(repo, args=[]):\n\n mgr = plugins_get_mgr()\n keys = mgr.search(what='metadata')\n keys = keys['metadata']\n\n if len(keys) == 0:\n return\n\n # Incorporate pipeline information...\n if 'pipeline' in repo.options:\n for name, details in repo.options['pipeline'].items():\n patterns = details['files']\n matching_files = repo.find_matching_files(patterns)\n matching_files.sort()\n details['files'] = matching_files\n for i, f in enumerate(matching_files):\n r = repo.get_resource(f)\n if 'pipeline' not in r:\n r['pipeline'] = []\n r['pipeline'].append(name + \" [Step {}]\".format(i))\n\n if 'metadata-management' in repo.options:\n\n print(\"Collecting all the required metadata to post\")\n metadata = repo.options['metadata-management']\n\n # Add data repo history\n if 'include-data-history' in metadata and metadata['include-data-history']:\n repo.package['history'] = get_history(repo.rootdir)\n\n # Add action history \n if 'include-action-history' in metadata and metadata['include-action-history']:\n annotate_metadata_action(repo) \n\n # Add data repo history\n if 'include-preview' in metadata:\n annotate_metadata_data(repo,\n task='preview',\n patterns=metadata['include-preview']['files'],\n size=metadata['include-preview']['length'])\n\n if (('include-schema' in metadata) and metadata['include-schema']):\n annotate_metadata_data(repo, task='schema')\n\n if 'include-code-history' in metadata:\n annotate_metadata_code(repo, files=metadata['include-code-history'])\n\n if 'include-platform' in metadata:\n annotate_metadata_platform(repo)\n\n if 'include-validation' in metadata:\n annotate_metadata_validation(repo)\n\n if 'include-dependencies' in metadata:\n annotate_metadata_dependencies(repo)\n\n history = repo.package.get('history',None)\n if (('include-tab-diffs' in metadata) and\n metadata['include-tab-diffs'] and\n history is not None):\n annotate_metadata_diffs(repo)\n\n # Insert options as well\n repo.package['config'] = repo.options\n\n try:\n for k in keys:\n # print(\"Key\", k)\n metadatamgr = mgr.get_by_key('metadata', k)\n url = metadatamgr.url\n o = urlparse(url)\n print(\"Posting to \", o.netloc)\n response = metadatamgr.post(repo)\n if isinstance(response, str):\n print(\"Error while posting:\", response)\n elif response.status_code in [400]:\n content = response.json()\n print(\"Error while posting:\")\n for k in content:\n print(\" \", k,\"- \", \",\".join(content[k]))\n except NetworkError as e:\n print(\"Unable to reach metadata server!\")\n except NetworkInvalidConfiguration as e:\n print(\"Invalid network configuration in the INI file\")\n print(e.message)\n except Exception as e:\n print(\"Could not post. Unknown error\")\n print(e)", "response": "Post to metadata server"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nshowing details of available plugins", "response": "def plugins_show(what=None, name=None, version=None, details=False):\n \"\"\"\n Show details of available plugins\n\n Parameters\n ----------\n what: Class of plugins e.g., backend\n name: Name of the plugin e.g., s3\n version: Version of the plugin\n details: Show details be shown?\n\n \"\"\"\n global pluginmgr\n return pluginmgr.show(what, name, version, details)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef discover_all_plugins(self):\n for v in pkg_resources.iter_entry_points('dgit.plugins'):\n m = v.load()\n m.setup(self)", "response": "Load all plugins from dgit extension\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nregistering a plugin with the registry", "response": "def register(self, what, obj):\n \"\"\"\n Registering a plugin\n\n Params\n ------\n what: Nature of the plugin (backend, instrumentation, repo)\n obj: Instance of the plugin\n \"\"\"\n # print(\"Registering pattern\", name, pattern)\n name = obj.name\n version = obj.version\n enable = obj.enable\n if enable == 'n':\n return\n\n key = Key(name, version)\n self.plugins[what][key] = obj"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef search(self, what, name=None, version=None):\n filtered = {}\n\n # The search may for a scan (what is None) or\n if what is None:\n whats = list(self.plugins.keys())\n elif what is not None:\n if what not in self.plugins:\n raise Exception(\"Unknown class of plugins\")\n whats = [what]\n for what in whats:\n if what not in filtered:\n filtered[what] = []\n for key in self.plugins[what].keys():\n (k_name, k_version) = key\n if name is not None and k_name != name:\n continue\n if version is not None and k_version != version:\n continue\n if self.plugins[what][key].enable == 'n':\n continue\n filtered[what].append(key)\n\n # print(filtered)\n return filtered", "response": "Search for a plugin in the cache."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngather configuration requirements of all plugins", "response": "def gather_configs(self):\n \"\"\"\n Gather configuration requirements of all plugins\n \"\"\"\n configs = []\n for what in self.order:\n for key in self.plugins[what]:\n mgr = self.plugins[what][key]\n c = mgr.config(what='get')\n if c is not None:\n c.update({\n 'description': mgr.description\n })\n # print(\"Gathering configuration from \", c)\n configs.append(c)\n return configs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupdate all configuration requirements of all plugins", "response": "def update_configs(self, config):\n \"\"\"\n Gather configuration requirements of all plugins\n \"\"\"\n for what in self.plugins: # backend, repo etc.\n for key in self.plugins[what]: # s3, filesystem etc.\n # print(\"Updating configuration of\", what, key)\n self.plugins[what][key].config(what='set', params=config)\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self):\n run_once = True\n while (run_once or self._threaded) and self.end is False:\n self.service_tx_queue()\n self.parse_messages()\n\n run_once = False\n\n if self._threaded:\n time.sleep(self._timeout)\n\n if self._threaded:\n logger.info('bootloader thread complete')", "response": "This method is run in a separate thread."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef instantiate(repo, validator_name=None, filename=None, rulesfiles=None):\n\n default_validators = repo.options.get('validator', {})\n\n validators = {}\n if validator_name is not None:\n # Handle the case validator is specified..\n if validator_name in default_validators:\n validators = {\n validator_name : default_validators[validator_name]\n }\n else:\n validators = {\n validator_name : {\n 'files': [],\n 'rules': {},\n 'rules-files': []\n }\n }\n else:\n validators = default_validators\n\n #=========================================\n # Insert the file names\n #=========================================\n if filename is not None:\n matching_files = repo.find_matching_files([filename])\n if len(matching_files) == 0:\n print(\"Filename could not be found\", filename)\n raise Exception(\"Invalid filename pattern\")\n for v in validators:\n validators[v]['files'] = matching_files\n else:\n # Instantiate the files from the patterns specified\n for v in validators:\n if 'files' not in validators[v]:\n validators[v]['files'] = []\n elif len(validators[v]['files']) > 0:\n matching_files = repo.find_matching_files(validators[v]['files'])\n validators[v]['files'] = matching_files\n\n #=========================================\n # Insert the rules files..\n #=========================================\n if rulesfiles is not None:\n # Command lines...\n matching_files = repo.find_matching_files([rulesfiles])\n if len(matching_files) == 0:\n print(\"Could not find matching rules files ({}) for {}\".format(rulesfiles,v))\n raise Exception(\"Invalid rules\")\n for v in validators:\n validators[v]['rules-files'] = matching_files\n else:\n # Instantiate the files from the patterns specified\n for v in validators:\n if 'rules-files' not in validators[v]:\n validators[v]['rules-files'] = []\n else:\n rulesfiles = validators[v]['rules-files']\n matching_files = repo.find_matching_files(rulesfiles)\n validators[v]['rules-files'] = matching_files\n\n return validators", "response": "Instantiate the validation specification"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate(repo, \n validator_name=None, \n filename=None, \n rulesfiles=None,\n args=[]):\n \"\"\"\n Validate the content of the files for consistency. Validators can\n look as deeply as needed into the files. dgit treats them all as\n black boxes.\n\n Parameters\n ----------\n\n repo: Repository object\n validator_name: Name of validator, if any. If none, then all validators specified in dgit.json will be included.\n filename: Pattern that specifies files that must be processed by the validators selected. If none, then the default specification in dgit.json is used.\n rules: Pattern specifying the files that have rules that validators will use\n show: Print the validation results on the terminal\n\n Returns\n -------\n\n status: A list of dictionaries, each with target file processed, rules file applied, status of the validation and any error message.\n \"\"\"\n\n mgr = plugins_get_mgr()\n\n # Expand the specification. Now we have full file paths\n validator_specs = instantiate(repo, validator_name, filename, rulesfiles)\n\n # Run the validators with rules files...\n allresults = []\n for v in validator_specs:\n\n keys = mgr.search(what='validator',name=v)['validator']\n for k in keys:\n validator = mgr.get_by_key('validator', k)\n result = validator.evaluate(repo, \n validator_specs[v],\n args)\n allresults.extend(result)\n\n return allresults", "response": "Validate the content of the files."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if a URL exists and if the file system path exists...", "response": "def url_is_valid(self, url):\n \"\"\"\n Check if a URL exists\n \"\"\"\n # Check if the file system path exists...\n if url.startswith(\"file://\"):\n url = url.replace(\"file://\",\"\")\n\n return os.path.exists(url)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npost to the metadata server", "response": "def post(self, repo):\n \"\"\"\n Post to the metadata server\n\n Parameters\n ----------\n\n repo\n \"\"\"\n\n datapackage = repo.package\n\n url = self.url\n token = self.token\n headers = {\n 'Authorization': 'Token {}'.format(token),\n 'Content-Type': 'application/json'\n }\n\n try:\n r = requests.post(url,\n data = json.dumps(datapackage),\n headers=headers)\n\n return r\n except Exception as e: \n #print(e)\n #traceback.print_exc()\n raise NetworkError()\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nimports and returns module class from path. to. module. Class argument", "response": "def get_module_class(class_path):\n \"\"\"\n imports and returns module class from ``path.to.module.Class``\n argument\n \"\"\"\n mod_name, cls_name = class_path.rsplit('.', 1)\n\n try:\n mod = import_module(mod_name)\n except ImportError as ex:\n raise EvoStreamException('Error importing module %s: '\n '\"%s\"' % (mod_name, ex))\n\n return getattr(mod, cls_name)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds all executable files in the current directory.", "response": "def find_executable_files():\n \"\"\"\n Find max 5 executables that are responsible for this repo.\n \"\"\"\n files = glob.glob(\"*\") + glob.glob(\"*/*\") + glob.glob('*/*/*')\n files = filter(lambda f: os.path.isfile(f), files)\n executable = stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH\n final = []\n for filename in files:\n if os.path.isfile(filename):\n st = os.stat(filename)\n mode = st.st_mode\n if mode & executable:\n final.append(filename)\n if len(final) > 5:\n break\n return final"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize a new empty tree from a git - specific configuration file.", "response": "def auto_init(autofile, force_init=False):\n \"\"\"\n Initialize a repo-specific configuration file to execute dgit\n\n Parameters\n ----------\n\n autofile: Repo-specific configuration file (dgit.json)\n force_init: Flag to force to re-initialization of the configuration file\n\n \"\"\"\n\n if os.path.exists(autofile) and not force_init:\n try:\n autooptions = json.loads(open(autofile).read())\n return autooptions\n except:\n print(\"Error in dgit.json configuration file\")\n traceback.print_exc()\n raise Exception(\"Invalid configuration file\")\n\n config = get_config()\n pluginmgr = plugins_get_mgr()\n\n print(\"Repo configuration file missing or corrupted. Creating one\")\n print(\"Let us know a few details about your data repository\")\n\n # Get the username\n username = getpass.getuser()\n revised = input(\"Please specify username [{}]\".format(username))\n if revised not in [\"\", None]:\n username = revised\n\n # Get the reponame\n thisdir = os.path.abspath(os.getcwd())\n reponame = os.path.basename(thisdir)\n revised = input(\"Please specify repo name [{}]\".format(reponame))\n if revised not in [\"\", None]:\n reponame = revised\n\n # Get the default backend URL\n keys = pluginmgr.search('backend')\n keys = keys['backend']\n keys = [k for k in keys if k[0] != \"local\"]\n remoteurl = \"\"\n backend = None\n if len(keys) > 0:\n backend = pluginmgr.get_by_key('backend', keys[0])\n candidate = backend.url(username, reponame)\n revised = input(\"Please specify remote URL [{}]\".format(candidate))\n if revised in [\"\", None]:\n remoteurl = candidate\n else:\n remoteurl = revised\n\n # Get title...\n title = \"\"\n while title == \"\":\n title = input(\"One line summary of your repo:\")\n if title == \"\":\n print(\"The repo requires a one line summary\")\n else:\n break\n\n # Get description\n description = \"\"\n while description == \"\":\n description = input(\"Detailed description:\")\n if description == \"\":\n print(\"The repo requires some text as well\")\n else:\n break\n\n autooptions = OrderedDict([\n (\"username\", username),\n (\"reponame\", reponame),\n (\"remoteurl\", remoteurl),\n (\"title\", title),\n (\"description\", description),\n (\"working-directory\", \".\"),\n ('track' ,OrderedDict([\n ('includes', ['*.csv', '*.tsv', '*.txt','*.json', '*.xls', '*.xlsx', \"*.sql\", \"*.hql\"]),\n ('excludes', ['.git', '.svn', os.path.basename(autofile)]),\n ])),\n ('auto-push', False),\n ('pipeline' ,OrderedDict([])),\n ('import' ,OrderedDict([\n ('directory-mapping' ,OrderedDict([\n ('.', '')\n ]))\n ])),\n ('dependencies' ,OrderedDict([]))\n ])\n\n # Gather options from each of the enabled plugins\n for p in ['validator', 'transformer']:\n keys = pluginmgr.search(p)\n keys = keys[p]\n options = OrderedDict()\n for k in keys:\n if k.name in options:\n continue\n mgr = pluginmgr.get_by_key(p, k)\n options[k.name] = mgr.autooptions()\n autooptions[p] = options\n\n keys = pluginmgr.search('metadata')\n keys = keys['metadata']\n if len(keys) > 0:\n\n # => Select domains that be included.\n servers = []\n for k in keys:\n server = pluginmgr.get_by_key('metadata', k)\n server = server.url.split(\"/\")[2]\n servers.append(server)\n\n # Specify what should be included. Some of these should go ino\n # the metadata modules\n\n autooptions.update(OrderedDict([\n ('metadata-management', OrderedDict([\n ('servers', servers),\n ('include-code-history', find_executable_files()),\n ('include-preview', OrderedDict([\n ('length', 512),\n ('files', ['*.txt', '*.csv', '*.tsv'])\n ])),\n ('include-data-history', True),\n ('include-action-history', True),\n ('include-validation', True),\n ('include-dependencies', True),\n ('include-schema', True),\n ('include-tab-diffs', True),\n ('include-platform', True),\n ]))]))\n\n with open(autofile, 'w') as fd:\n fd.write(json.dumps(autooptions, indent=4))\n\n print(\"\")\n print(\"Updated dataset specific config file: {}\".format(autofile))\n print(\"Please edit it and rerun dgit auto.\")\n print(\"Tip: Consider committing dgit.json to the code repository.\")\n\n #if platform.system() == \"Linux\":\n # subprocess.call([\"xdg-open\", autofile])\n\n sys.exit()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef auto_get_repo(autooptions, debug=False):\n\n # plugin manager\n pluginmgr = plugins_get_mgr()\n\n # get the repo manager\n repomgr = pluginmgr.get(what='repomanager', name='git')\n\n repo = None\n\n try:\n if debug:\n print(\"Looking repo\")\n repo = repomgr.lookup(username=autooptions['username'],\n reponame=autooptions['reponame'])\n except:\n # Clone the repo\n try:\n print(\"Checking and cloning if the dataset exists on backend\")\n url = autooptions['remoteurl']\n if debug:\n print(\"Doesnt exist. trying to clone: {}\".format(url))\n common_clone(url)\n repo = repomgr.lookup(username=autooptions['username'],\n reponame=autooptions['reponame'])\n if debug:\n print(\"Cloning successful\")\n except:\n # traceback.print_exc()\n yes = input(\"Repo doesnt exist. Should I create one? [yN]\")\n if yes == 'y':\n setup = \"git\"\n if autooptions['remoteurl'].startswith('s3://'):\n setup = 'git+s3'\n repo = common_init(username=autooptions['username'],\n reponame=autooptions['reponame'],\n setup=setup,\n force=True,\n options=autooptions)\n\n if debug:\n print(\"Successfully inited repo\")\n else:\n raise Exception(\"Cannot load repo\")\n\n repo.options = autooptions\n\n return repo", "response": "Automatically get the repo from the autooptions dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the files to commit", "response": "def get_files_to_commit(autooptions):\n \"\"\"\n Look through the local directory to pick up files to check\n \"\"\"\n workingdir = autooptions['working-directory']\n includes = autooptions['track']['includes']\n excludes = autooptions['track']['excludes']\n\n # transform glob patterns to regular expressions\n # print(\"Includes \", includes) \n includes = r'|'.join([fnmatch.translate(x) for x in includes])\n excludes = r'|'.join([fnmatch.translate(x) for x in excludes]) or r'$.'\n\n matched_files = []\n for root, dirs, files in os.walk(workingdir):\n\n # print(\"Looking at \", files)\n\n # exclude dirs\n # dirs[:] = [os.path.join(root, d) for d in dirs]\n dirs[:] = [d for d in dirs if not re.match(excludes, d)]\n\n # exclude/include files\n files = [f for f in files if not re.match(excludes, f)]\n #print(\"Files after excludes\", files)\n #print(includes) \n files = [f for f in files if re.match(includes, f)]\n #print(\"Files after includes\", files) \n files = [os.path.join(root, f) for f in files]\n\n matched_files.extend(files)\n\n return matched_files"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef auto_add(repo, autooptions, files):\n # Get the mappings and keys.\n mapping = { \".\": \"\" }\n if (('import' in autooptions) and\n ('directory-mapping' in autooptions['import'])):\n mapping = autooptions['import']['directory-mapping']\n\n # Apply the longest prefix first...\n keys = mapping.keys()\n keys = sorted(keys, key=lambda k: len(k), reverse=True)\n\n count = 0\n params = []\n for f in files:\n\n # Find the destination\n relativepath = f\n for k in keys:\n v = mapping[k]\n if f.startswith(k + \"/\"):\n #print(\"Replacing \", k)\n relativepath = f.replace(k + \"/\", v)\n break\n\n # Now add to repository\n count += files_add(repo=repo,\n args=[f],\n targetdir=os.path.dirname(relativepath))\n\n return count", "response": "Add the files to the specified repository."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pull_stream(self, uri, **kwargs):\n return self.protocol.execute('pullStream', uri=uri, **kwargs)", "response": "This method attempts to pull a stream from an external source."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef push_stream(self, uri, **kwargs):\n return self.protocol.execute('pushStream', uri=uri, **kwargs)", "response": "Push a local stream to an external destination."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_hls_stream(self, localStreamNames, targetFolder, **kwargs):\n return self.protocol.execute('createhlsstream',\n localStreamNames=localStreamNames,\n targetFolder=targetFolder, **kwargs)", "response": "Create an existing H. 288 stream from a list of local stream names and a target folder."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_hds_stream(self, localStreamNames, targetFolder, **kwargs):\n return self.protocol.execute('createhdsstream',\n localStreamNames=localStreamNames,\n targetFolder=targetFolder, **kwargs)", "response": "Create an HDS stream out of an existing H. 288 or AAC stream."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a Microsoft Smooth Streaming object from an existing H. 288 stream.", "response": "def create_mss_stream(self, localStreamNames, targetFolder, **kwargs):\n \"\"\"\n Create a Microsoft Smooth Stream (MSS) out of an existing H.264/AAC\n stream. Smooth Streaming was developed by Microsoft to compete with\n other adaptive streaming technologies.\n\n :param localStreamNames: The stream(s) that will be used as the input.\n This is a comma-delimited list of active stream names (local\n stream names)\n :type localStreamNames: str\n\n :param targetFolder: The folder where all the manifest and fragment\n files will be stored. This folder must be accessible by the MSS\n clients. It is usually in the web-root of the server.\n :type targetFolder: str\n\n :param bandwidths: The corresponding bandwidths for each stream listed\n in `localStreamNames`. Again, this can be a comma-delimited list.\n :type bandwidths: int or str\n\n :param groupName: The name assigned to the MSS stream or group. If the\n `localStreamNames` parameter contains only one entry and groupName\n is not specified, groupName will have the value of the input\n stream name.\n :type groupName: str\n\n :param playlistType: Either `appending` or `rolling`\n :type playlistType: str\n\n :param playlistLength: The number of fragments before the server\n starts to overwrite the older fragments. Used only when\n `playlistType` is `rolling`. Ignored otherwise.\n :type playlistLength: int\n\n :param manifestName: The manifest file name.\n :type manifestName: str\n\n :param chunkLength: The length (in seconds) of fragments to be made.\n :type chunkLength: int\n\n :param chunkOnIDR: If 1 (true), chunking is performed ONLY on IDR.\n Otherwise, chunking is performed whenever chunk length is\n achieved.\n :type chunkOnIDR: int\n\n :param keepAlive: If 1 (true), the EMS will attempt to reconnect to\n the stream source if the connection is severed.\n :type keepAlive: int\n\n :param overwriteDestination: If 1 (true), it will allow overwrite of\n destination files.\n :type overwriteDestination: int\n\n :param staleRetentionCount: How many old files are kept besides the\n ones present in the current version of the playlist. Only\n applicable for rolling playlists.\n :type staleRetentionCount: int\n\n :param cleanupDestination: If 1 (true), all manifest and fragment\n files in the target folder will be removed before MSS creation is\n started.\n :type cleanupDestination: int\n\n :param ismType: Either ismc for serving content to client or isml for\n serving content to smooth server.\n :type ismType: int\n\n :param isLive: If true, creates a live MSS stream, otherwise set to 0\n for VOD.\n :type isLive: int\n\n :param publishingPoint: This parameter is needed when `ismType=isml`,\n it is the REST URI where the mss contents will be ingested.\n :type publishingPoint: str\n\n :param ingestMode: Either `single` for a non looping ingest or `loop`\n for looping an ingest.\n :type ingestMode: str\n\n :link: http://docs.evostream.com/ems_api_definition/createmssstream\n \"\"\"\n return self.protocol.execute('createmssstream',\n localStreamNames=localStreamNames,\n targetFolder=targetFolder, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_dash_stream(self, localStreamNames, targetFolder, **kwargs):\n return self.protocol.execute('createdashstream',\n localStreamNames=localStreamNames,\n targetFolder=targetFolder, **kwargs)", "response": "Creates a DASH stream for the given local stream names."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef record(self, localStreamName, pathToFile, **kwargs):\n return self.protocol.execute('record',\n localStreamName=localStreamName,\n pathToFile=pathToFile, **kwargs)", "response": "This method records any inbound stream."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchanging the compression characteristics of an audio and/or video stream. Allows you to change the resolution of a source stream, change the bitrate of a stream, change a VP8 or MPEG2 stream into H.264 and much more. Allow users to create overlays on the final stream as well as crop streams. :param source: Can be a URI or a local stream name from EMS. :type source: str :param destinations: The target URI(s) or stream name(s) of the transcoded stream. If only a name is given, it will be pushed back to the EMS. :type destinations: str :param targetStreamNames: The name of the stream(s) at destination(s). If not specified, and a full URI is provided to destinations, name will have a time stamped value. :type targetStreamNames: str :param groupName: The group name assigned to this process. If not specified, groupName will have a random value. :type groupName: str :param videoBitrates: Target output video bitrate(s) (in bits/s, append `k` to value for kbits/s). Accepts the value `copy` to copy the input bitrate. An empty value passed would mean no video. :type videoBitrates: str :param videoSizes: Target output video size(s) in wxh (width x height) format. IE: 240x480. :type videoSizes: str :param videoAdvancedParamsProfiles: Name of video profile template that will be used. :type videoAdvancedParamsProfiles: str :param audioBitrates: Target output audio bitrate(s) (in bits/s, append `k` to value for kbits/s). Accepts the value `copy` to copy the input bitrate. An empty value passed would mean no audio. :type audioBitrates: str :param audioChannelsCounts: Target output audio channel(s) count(s). Valid values are 1 (mono), 2 (stereo), and so on. Actual supported channel count is dependent on the number of input audio channels. :type audioChannelsCounts: str :param audioFrequencies: Target output audio frequency(ies) (in Hz, append `k` to value for kHz). :type audioFrequencies: str :param audioAdvancedParamsProfiles: Name of audio profile template that will be used. :type audioAdvancedParamsProfiles: str :param overlays: Location of the overlay source(s) to be used. These are transparent images (normally in PNG format) that have the same or smaller size than the video. Image is placed at the top-left position of the video. :type overlays: str :param croppings: Target video cropping position(s) and size(s) in `left : top : width : height` format (e.g. 0:0:200:100. Positions are optional (200:100 for a centered cropping of 200 width and 100 height in pixels). Values are limited to the actual size of the video. :type croppings: str :param keepAlive: If keepAlive is set to 1, the server will restart transcoding if it was previously activated. :type keepAlive: int :param commandFlags: Other commands to the transcode process that are not supported by the baseline transcode command. :type commandFlags: str :link: http://docs.evostream.com/ems_api_definition/transcode", "response": "def transcode(self, source, destinations, **kwargs):\n \"\"\"\n Changes the compression characteristics of an audio and/or video\n stream. Allows you to change the resolution of a source stream, change\n the bitrate of a stream, change a VP8 or MPEG2 stream into H.264 and\n much more. Allow users to create overlays on the final stream as well\n as crop streams.\n\n :param source: Can be a URI or a local stream name from EMS.\n :type source: str\n\n :param destinations: The target URI(s) or stream name(s) of the\n transcoded stream. If only a name is given, it will be pushed\n back to the EMS.\n :type destinations: str\n\n :param targetStreamNames: The name of the stream(s) at destination(s).\n If not specified, and a full URI is provided to destinations,\n name will have a time stamped value.\n :type targetStreamNames: str\n\n :param groupName: The group name assigned to this process. If not\n specified, groupName will have a random value.\n :type groupName: str\n\n :param videoBitrates: Target output video bitrate(s) (in bits/s,\n append `k` to value for kbits/s). Accepts the value `copy` to\n copy the input bitrate. An empty value passed would mean no video.\n :type videoBitrates: str\n\n :param videoSizes: Target output video size(s) in wxh (width x height)\n format. IE: 240x480.\n :type videoSizes: str\n\n :param videoAdvancedParamsProfiles: Name of video profile template\n that will be used.\n :type videoAdvancedParamsProfiles: str\n\n :param audioBitrates: Target output audio bitrate(s) (in bits/s,\n append `k` to value for kbits/s). Accepts the value `copy` to\n copy the input bitrate. An empty value passed would mean no audio.\n :type audioBitrates: str\n\n :param audioChannelsCounts: Target output audio channel(s) count(s).\n Valid values are 1 (mono), 2 (stereo), and so on. Actual supported\n channel count is dependent on the number of input audio channels.\n :type audioChannelsCounts: str\n\n :param audioFrequencies: Target output audio frequency(ies) (in Hz,\n append `k` to value for kHz).\n :type audioFrequencies: str\n\n :param audioAdvancedParamsProfiles: Name of audio profile template\n that will be used.\n :type audioAdvancedParamsProfiles: str\n\n :param overlays: Location of the overlay source(s) to be used. These\n are transparent images (normally in PNG format) that have the same\n or smaller size than the video. Image is placed at the top-left\n position of the video.\n :type overlays: str\n\n :param croppings: Target video cropping position(s) and size(s) in\n `left : top : width : height` format (e.g. 0:0:200:100. Positions\n are optional (200:100 for a centered cropping of 200 width and 100\n height in pixels). Values are limited to the actual size of the\n video.\n :type croppings: str\n\n :param keepAlive: If keepAlive is set to 1, the server will restart\n transcoding if it was previously activated.\n :type keepAlive: int\n\n :param commandFlags: Other commands to the transcode process that are\n not supported by the baseline transcode command.\n :type commandFlags: str\n\n :link: http://docs.evostream.com/ems_api_definition/transcode\n \"\"\"\n return self.protocol.execute('transcode', source=source,\n destinations=destinations, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds an alias to an existing stream.", "response": "def add_stream_alias(self, localStreamName, aliasName, **kwargs):\n \"\"\"\n Allows you to create secondary name(s) for internal streams. Once an\n alias is created the localstreamname cannot be used to request\n playback of that stream. Once an alias is used (requested by a client)\n the alias is removed. Aliases are designed to be used to protect/hide\n your source streams.\n\n :param localStreamName: The original stream name\n :type localStreamName: str\n\n :param aliasName: The alias alternative to the localStreamName\n :type aliasName: str\n\n :param expirePeriod: The expiration period for this alias. Negative\n values will be treated as one-shot but no longer than the absolute\n positive value in seconds, 0 means it will not expire, positive\n values mean the alias can be used multiple times but expires after\n this many seconds. The default is -600 (one-shot, 10 mins)\n :type expirePeriod: int\n\n :link: http://docs.evostream.com/ems_api_definition/addstreamalias\n \"\"\"\n return self.protocol.execute('addStreamAlias',\n localStreamName=localStreamName,\n aliasName=aliasName, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_group_name_alias(self, groupName, aliasName):\n return self.protocol.execute('addGroupNameAlias', groupName=groupName,\n aliasName=aliasName)", "response": "Adds an alias to a group name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating an RTMP ingest point which mandates that streams pushed into the privateStreamName and the publicStreamName.", "response": "def create_ingest_point(self, privateStreamName, publicStreamName):\n \"\"\"\n Creates an RTMP ingest point, which mandates that streams pushed into\n the EMS have a target stream name which matches one Ingest Point\n privateStreamName.\n\n :param privateStreamName: The name that RTMP Target Stream Names must\n match.\n :type privateStreamName: str\n\n :param publicStreamName: The name that is used to access the stream\n pushed to the privateStreamName. The publicStreamName becomes the\n streams localStreamName.\n :type publicStreamName: str\n\n :link: http://docs.evostream.com/ems_api_definition/createingestpoint\n \"\"\"\n return self.protocol.execute('createIngestPoint',\n privateStreamName=privateStreamName,\n publicStreamName=publicStreamName)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start_web_rtc(self, ersip, ersport, roomId):\n return self.protocol.execute('startwebrtc', ersip=ersip, ersport=ersport,\n roomId=roomId)", "response": "Starts a WebRTC signalling client to an EMS."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmaterializing queries other content within the repository.", "response": "def transform(repo,\n name=None,\n filename=None,\n force=False,\n args=[]):\n \"\"\"\n Materialize queries/other content within the repo.\n\n Parameters\n ----------\n\n repo: Repository object\n name: Name of transformer, if any. If none, then all transformers specified in dgit.json will be included.\n filename: Pattern that specifies files that must be processed by the generators selected. If none, then the default specification in dgit.json is used.\n\n \"\"\"\n mgr = plugins_get_mgr()\n\n # Expand the specification. Now we have full file paths\n specs = instantiate(repo, name, filename)\n\n # Run the validators with rules files...\n allresults = []\n for s in specs:\n keys = mgr.search(what='transformer',name=s)['transformer']\n for k in keys:\n t = mgr.get_by_key('transformer', k)\n result = t.evaluate(repo,\n specs[s],\n force,\n args)\n allresults.extend(result)\n\n return allresults"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun a generic command within the repo.", "response": "def _run_generic_command(self, repo, cmd):\n \"\"\"\n Run a generic command within the repo. Assumes that you are\n in the repo's root directory\n \"\"\"\n \n result = None\n with cd(repo.rootdir):\n # Dont use sh. It is not collecting the stdout of all\n # child processes.\n output = self._run(cmd)\n try:\n result = {\n 'cmd': cmd,\n 'status': 'success',\n 'message': output,\n }\n except Exception as e:\n result = {\n 'cmd': cmd,\n 'status': 'error',\n 'message': str(e)\n }\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ninitializing a Git repo and return a new one.", "response": "def init(self, username, reponame, force, backend=None):\n \"\"\"\n Initialize a Git repo\n\n Parameters\n ----------\n\n username, reponame : Repo name is tuple (name, reponame)\n force: force initialization of the repo even if exists\n backend: backend that must be used for this (e.g. s3)\n \"\"\"\n key = self.key(username, reponame)\n\n # In local filesystem-based server, add a repo\n server_repodir = self.server_rootdir(username,\n reponame,\n create=False)\n\n # Force cleanup if needed\n if os.path.exists(server_repodir) and not force:\n raise RepositoryExists()\n\n if os.path.exists(server_repodir):\n shutil.rmtree(server_repodir)\n os.makedirs(server_repodir)\n\n # Initialize the repo\n with cd(server_repodir):\n git.init(\".\", \"--bare\")\n\n if backend is not None:\n backend.init_repo(server_repodir)\n\n # Now clone the filesystem-based repo\n repodir = self.rootdir(username, reponame, create=False)\n\n # Prepare it if needed\n if os.path.exists(repodir) and not force:\n raise Exception(\"Local repo already exists\")\n if os.path.exists(repodir):\n shutil.rmtree(repodir)\n os.makedirs(repodir)\n\n # Now clone...\n with cd(os.path.dirname(repodir)):\n git.clone(server_repodir, '--no-hardlinks')\n\n url = server_repodir\n if backend is not None:\n url = backend.url(username, reponame)\n\n repo = Repo(username, reponame)\n repo.manager = self\n repo.remoteurl = url\n repo.rootdir = self.rootdir(username, reponame)\n\n self.add(repo)\n return repo"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clone(self, url, backend=None):\n\n\n # s3://bucket/git/username/repo.git\n username = self.username\n reponame = url.split(\"/\")[-1] # with git\n reponame = reponame.replace(\".git\",\"\")\n\n key = (username, reponame)\n\n # In local filesystem-based server, add a repo\n server_repodir = self.server_rootdir(username,\n reponame,\n create=False)\n\n rootdir = self.rootdir(username, reponame, create=False)\n\n\n if backend is None:\n # Backend is standard git repo (https://, git@...)\n with cd(os.path.dirname(rootdir)):\n self._run(['clone', '--no-hardlinks', url])\n else:\n # Backend is s3\n # Sync if needed.\n if not os.path.exists(server_repodir):\n # s3 -> .dgit/git/pingali/hello.git -> .dgit/datasets/pingali/hello\n backend.clone_repo(url, server_repodir)\n\n # After sync clone,\n with cd(os.path.dirname(rootdir)):\n self._run(['clone', '--no-hardlinks', server_repodir])\n\n\n # Insert the notes push\n if True:\n configfile = os.path.join(rootdir, '.git', 'config')\n content = open(configfile).read()\n original = \"fetch = +refs/heads/*:refs/remotes/origin/*\"\n replacement =\"\"\"fetch = +refs/heads/*:refs/remotes/origin/*\\n fetch = +refs/notes/*:refs/notes/*\"\"\"\n if \"notes\" not in content:\n content = content.replace(original, replacement)\n with open(configfile, 'w') as fd:\n fd.write(content)\n\n # Pull the notes if any as well..\n with cd(rootdir):\n self._run(['pull','origin'])\n\n # Insert the object into the internal table we maintain...\n r = Repo(username, reponame)\n r.rootdir = rootdir\n r.remoteurl = url\n r.manager = self\n\n package = os.path.join(r.rootdir, 'datapackage.json')\n packagedata = open(package).read()\n r.package = json.JSONDecoder(object_pairs_hook=collections.OrderedDict).decode(packagedata)\n\n return self.add(r)", "response": "Clone a URL and create a new version of the new version of the new version."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting files from the repo", "response": "def delete(self, repo, args=[]):\n \"\"\"\n Delete files from the repo\n \"\"\"\n\n result = None\n with cd(repo.rootdir):\n try:\n cmd = ['rm'] + list(args)\n result = {\n 'status': 'success',\n 'message': self._run(cmd)\n }\n except Exception as e:\n result = {\n 'status': 'error',\n 'message': str(e)\n }\n\n # print(result)\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndrops the repo and all of its data.", "response": "def drop(self, repo, args=[]):\n \"\"\"\n Cleanup the repo\n \"\"\"\n\n # Clean up the rootdir\n rootdir = repo.rootdir\n if os.path.exists(rootdir):\n print(\"Cleaning repo directory: {}\".format(rootdir))\n shutil.rmtree(rootdir)\n\n # Cleanup the local version of the repo (this could be on\n # the server etc.\n server_repodir = self.server_rootdir_from_repo(repo,\n create=False)\n if os.path.exists(server_repodir):\n print(\"Cleaning data from local git 'server': {}\".format(server_repodir))\n shutil.rmtree(server_repodir)\n\n super(GitRepoManager, self).drop(repo)\n\n return {\n 'status': 'success',\n 'message': \"successful cleanup\"\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef permalink(self, repo, path):\n\n if not os.path.exists(path): \n # print(\"Path does not exist\", path)\n return (None, None) \n\n # Get this directory\n cwd = os.getcwd()\n\n # Find the root of the repo and cd into that directory..\n if os.path.isfile(path):\n os.chdir(os.path.dirname(path))\n\n rootdir = self._run([\"rev-parse\", \"--show-toplevel\"])\n if \"fatal\" in rootdir:\n # print(\"fatal\", rootdir)\n return (None, None) \n\n os.chdir(rootdir)\n # print(\"Rootdir = \", rootdir)\n\n # Now find relative path\n relpath = os.path.relpath(path, rootdir)\n # print(\"relpath = \", relpath)\n\n # Get the last commit for this file\n #3764cc2600b221ac7d7497de3d0dbcb4cffa2914\n sha1 = self._run([\"log\", \"-n\", \"1\", \"--format=format:%H\", relpath])\n # print(\"sha1 = \", sha1)\n\n # Get the repo URL\n #git@gitlab.com:pingali/simple-regression.git\n #https://gitlab.com/kanban_demo/test_project.git\n remoteurl = self._run([\"config\", \"--get\", \"remote.origin.url\"])\n # print(\"remoteurl = \", remoteurl)\n\n # Go back to the original directory...\n os.chdir(cwd)\n\n # Now match it against two possible formats of the remote url\n # Examples\n #https://help.github.com/articles/getting-permanent-links-to-files/\n #https://github.com/github/hubot/blob/ed25584f5ac2520a6c28547ffd0961c7abd7ea49/README.md\n #https://gitlab.com/pingali/simple-regression/blob/3764cc2600b221ac7d7497de3d0dbcb4cffa2914/model.py\n #https://github.com/pingali/dgit/blob/ff91b5d04b2978cad0bf9b006d1b0a16d18a778e/README.rst\n #https://gitlab.com/kanban_demo/test_project/blob/b004677c23b3a31eb7b5588a5194857b2c8b2b95/README.md\n\n m = re.search('^git@([^:\\/]+):([^/]+)/([^/]+)', remoteurl)\n if m is None:\n m = re.search('^https://([^:/]+)/([^/]+)/([^/]+)', remoteurl)\n if m is not None:\n domain = m.group(1)\n username = m.group(2)\n project = m.group(3)\n if project.endswith(\".git\"):\n project = project[:-4]\n permalink = \"https://{}/{}/{}/blob/{}/{}\".format(domain, username, project,\n sha1, relpath)\n # print(\"permalink = \", permalink)\n return (relpath, permalink)\n else:\n return (None, None)", "response": "Get the permalink to command that generated the dataset"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_files(self, repo, files):\n rootdir = repo.rootdir\n for f in files:\n relativepath = f['relativepath']\n sourcepath = f['localfullpath']\n if sourcepath is None:\n # This can happen if the relative path is a URL\n continue #\n # Prepare the target path\n targetpath = os.path.join(rootdir, relativepath)\n try:\n os.makedirs(os.path.dirname(targetpath))\n except:\n pass\n # print(sourcepath,\" => \", targetpath)\n print(\"Updating: {}\".format(relativepath))\n shutil.copyfile(sourcepath, targetpath)\n with cd(repo.rootdir):\n self._run(['add', relativepath])", "response": "Add files to the repo."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate the base set of attributes invoice has and needs", "response": "def _init_empty(self):\n \"\"\"Creates the base set of attributes invoice has/needs\"\"\"\n self._jsondata = {\n \"code\": None,\n \"currency\": \"EUR\",\n \"subject\": \"\",\n \"due_date\": (datetime.datetime.now().date() + datetime.timedelta(days=14)).isoformat(),\n \"issue_date\": datetime.datetime.now().date().isoformat(),\n \"number\": None,\n \"type\": \"outbound\",\n \"receiver\": {\n \"name\": \"\",\n \"email\": \"\",\n \"street\": \"\",\n \"city\": \"\",\n \"postcode\": \"\",\n \"country\": \"\"\n },\n \"items\": [],\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend the invoice to the recipient Holvi .", "response": "def send(self, send_email=True):\n \"\"\"Marks the invoice as sent in Holvi\n\n If send_email is False then the invoice is *not* automatically emailed to the recipient\n and your must take care of sending the invoice yourself.\n \"\"\"\n url = str(self.api.base_url + '{code}/status/').format(code=self.code) # six.u messes this up\n payload = {\n 'mark_as_sent': True,\n 'send_email': send_email,\n }\n stat = self.api.connection.make_put(url, payload)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting our Python object to JSON acceptable to Holvi API", "response": "def to_holvi_dict(self):\n \"\"\"Convert our Python object to JSON acceptable to Holvi API\"\"\"\n self._jsondata[\"items\"] = []\n for item in self.items:\n self._jsondata[\"items\"].append(item.to_holvi_dict())\n self._jsondata[\"issue_date\"] = self.issue_date.isoformat()\n self._jsondata[\"due_date\"] = self.due_date.isoformat()\n self._jsondata[\"receiver\"] = self.receiver.to_holvi_dict()\n return {k: v for (k, v) in self._jsondata.items() if k in self._valid_keys}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save(self):\n if not self.items:\n raise HolviError(\"No items\")\n if not self.subject:\n raise HolviError(\"No subject\")\n send_json = self.to_holvi_dict()\n if self.code:\n url = str(self.api.base_url + '{code}/').format(code=self.code)\n if not self.code:\n send_patch = {k: v for (k, v) in send_json.items() if k in self._patch_valid_keys}\n send_patch[\"items\"] = []\n for item in self.items:\n send_patch[\"items\"].append(item.to_holvi_dict(True))\n stat = self.api.connection.make_patch(url, send_patch)\n else:\n stat = self.api.connection.make_put(url, send_json)\n return Invoice(self.api, stat)\n else:\n url = str(self.api.base_url)\n stat = self.api.connection.make_post(url, send_json)\n return Invoice(self.api, stat)", "response": "Saves this invoice to Holvi returns the created or updated invoice"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the plugin source for the given module or the given module.", "response": "def get_plugin_source(module=None, stacklevel=None):\n \"\"\"Returns the :class:`PluginSource` for the current module or the given\n module. The module can be provided by name (in which case an import\n will be attempted) or as a module object.\n\n If no plugin source can be discovered, the return value from this method\n is `None`.\n\n This function can be very useful if additional data has been attached\n to the plugin source. For instance this could allow plugins to get\n access to a back reference to the application that created them.\n\n :param module: optionally the module to locate the plugin source of.\n :param stacklevel: defines how many levels up the module should search\n for before it discovers the plugin frame. The\n default is 0. This can be useful for writing wrappers\n around this function.\n \"\"\"\n if module is None:\n frm = sys._getframe((stacklevel or 0) + 1)\n name = frm.f_globals['__name__']\n glob = frm.f_globals\n elif isinstance(module, string_types):\n frm = sys._getframe(1)\n name = module\n glob = __import__(module, frm.f_globals,\n frm.f_locals, ['__dict__']).__dict__\n else:\n name = module.__name__\n glob = module.__dict__\n return _discover_space(name, glob)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef list_plugins(self):\n rv = []\n for _, modname, ispkg in pkgutil.iter_modules(self.mod.__path__):\n rv.append(modname)\n return sorted(rv)", "response": "Returns a sorted list of all plugins that are available in this\n plugin source."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open_resource(self, plugin, filename):\n mod = self.load_plugin(plugin)\n fn = getattr(mod, '__file__', None)\n if fn is not None:\n if fn.endswith(('.pyc', '.pyo')):\n fn = fn[:-1]\n if os.path.isfile(fn):\n return open(os.path.join(os.path.dirname(fn), filename), 'rb')\n buf = pkgutil.get_data(self.mod.__name__ + '.' + plugin, filename)\n if buf is None:\n raise IOError(errno.ENOEXITS, 'Could not find resource')\n return NativeBytesIO(buf)", "response": "This function locates a resource inside the plugin and returns a byte stream to the contents of that resource."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef api_call_action(func): \n def _inner(*args, **kwargs):\n return func(*args, **kwargs)\n _inner.__name__ = func.__name__\n _inner.__doc__ = func.__doc__\n return _inner", "response": "Wrapper for the API call action."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntransmit a series of bytes to the local memory.", "response": "def tx(self, message):\n \"\"\"\n Transmit a series of bytes\n :param message: a list of bytes to send\n :return: None\n \"\"\"\n message = message if isinstance(message, list) else [message]\n\n length = len(message)\n length_high_byte = (length & 0xff00) >> 8\n length_low_byte = length & 0x00ff\n\n message_with_length = [length_low_byte, length_high_byte] + message\n\n sum1, sum2 = self._fletcher16_checksum(message_with_length)\n message_with_length.append(sum1)\n message_with_length.append(sum2)\n\n message = [self._START_OF_FRAME]\n\n for b in message_with_length:\n if b in [self._START_OF_FRAME, self._END_OF_FRAME, self._ESC]:\n message.append(self._ESC)\n message.append(b ^ self._ESC_XOR)\n else:\n message.append(b)\n\n message.append(self._END_OF_FRAME)\n\n self._port.write(message)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rx(self):\n if not self._threaded:\n self.run()\n\n try:\n return tuple(self._messages.pop(0))\n except IndexError:\n return None", "response": "Receive a series of bytes that have been verified\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses the incoming data and determines if it is valid.", "response": "def _parse_raw_data(self):\n \"\"\"\n Parses the incoming data and determines if it is valid. Valid\n data gets placed into self._messages\n :return: None\n \"\"\"\n if self._START_OF_FRAME in self._raw and self._END_OF_FRAME in self._raw:\n\n while self._raw[0] != self._START_OF_FRAME and len(self._raw) > 0:\n self._raw.pop(0)\n\n if self._raw[0] == self._START_OF_FRAME:\n self._raw.pop(0)\n\n eof_index = self._raw.index(self._END_OF_FRAME)\n raw_message = self._raw[:eof_index]\n self._raw = self._raw[eof_index:]\n\n logger.debug('raw message: {}'.format(raw_message))\n\n message = self._remove_esc_chars(raw_message)\n logger.debug('message with checksum: {}'.format(message))\n\n expected_checksum = (message[-1] << 8) | message[-2]\n logger.debug('checksum: {}'.format(expected_checksum))\n\n message = message[:-2] # checksum bytes\n logger.debug('message: {}'.format(message))\n\n sum1, sum2 = self._fletcher16_checksum(message)\n calculated_checksum = (sum2 << 8) | sum1\n\n if expected_checksum == calculated_checksum:\n message = message[2:] # remove length\n logger.debug('valid message received: {}'.format(message))\n self._messages.append(message)\n else:\n logger.warning('invalid message received: {}, discarding'.format(message))\n logger.debug('expected checksum: {}, calculated checksum: {}'.format(expected_checksum, calculated_checksum))\n\n # remove any extra bytes at the beginning\n try:\n while self._raw[0] != self._START_OF_FRAME and len(self._raw) > 0:\n self._raw.pop(0)\n except IndexError:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fletcher16_checksum(self, data):\n sum1 = 0\n sum2 = 0\n\n for i, b in enumerate(data):\n sum1 += b\n sum1 &= 0xff # Results wrapped at 16 bits\n sum2 += sum1\n sum2 &= 0xff\n\n logger.debug('sum1: {} sum2: {}'.format(sum1, sum2))\n\n return sum1, sum2", "response": "Calculates a fletcher16 checksum for the list of bytes that comprise the message\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove any escape characters from the message that have the escaped characters appropriately un - escaped.", "response": "def _remove_esc_chars(self, raw_message):\n \"\"\"\n Removes any escape characters from the message\n :param raw_message: a list of bytes containing the un-processed data\n :return: a message that has the escaped characters appropriately un-escaped\n \"\"\"\n message = []\n escape_next = False\n for c in raw_message:\n if escape_next:\n message.append(c ^ self._ESC_XOR)\n escape_next = False\n else:\n if c == self._ESC:\n escape_next = True\n else:\n message.append(c)\n\n return message"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading the serial data into the self. _raw buffer and parses the data into self. _raw buffer.", "response": "def run(self):\n \"\"\"\n Receives the serial data into the self._raw buffer\n :return:\n \"\"\"\n run_once = True\n while run_once or self._threaded:\n waiting = self._port.in_waiting\n if waiting > 0:\n temp = [int(c) for c in self._port.read(waiting)]\n self._raw += temp\n\n self._parse_raw_data()\n run_once = False\n\n if self._threaded:\n time.sleep(self._timeout)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving this order to Holvi returns a tuple with the order itself and checkout_uri", "response": "def save(self):\n \"\"\"Saves this order to Holvi, returns a tuple with the order itself and checkout_uri\"\"\"\n if self.code:\n raise HolviError(\"Orders cannot be updated\")\n send_json = self.to_holvi_dict()\n send_json.update({\n 'pool': self.api.connection.pool\n })\n url = six.u(self.api.base_url + \"order/\")\n stat = self.api.connection.make_post(url, send_json)\n code = stat[\"details_uri\"].split(\"/\")[-2] # Maybe slightly ugly but I don't want to basically reimplement all but uri formation of the api method\n return (stat[\"checkout_uri\"], self.api.get_order(code))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef untokenize(tokens):\n text = ''\n previous_line = ''\n last_row = 0\n last_column = -1\n last_non_whitespace_token_type = None\n\n for (token_type, token_string, start, end, line) in tokens:\n if TOKENIZE_HAS_ENCODING and token_type == tokenize.ENCODING:\n continue\n\n (start_row, start_column) = start\n (end_row, end_column) = end\n\n # Preserve escaped newlines.\n if (\n last_non_whitespace_token_type != tokenize.COMMENT and\n start_row > last_row and\n previous_line.endswith(('\\\\\\n', '\\\\\\r\\n', '\\\\\\r'))\n ):\n text += previous_line[len(previous_line.rstrip(' \\t\\n\\r\\\\')):]\n\n # Preserve spacing.\n if start_row > last_row:\n last_column = 0\n if start_column > last_column:\n text += line[last_column:start_column]\n\n text += token_string\n\n previous_line = line\n\n last_row = end_row\n last_column = end_column\n\n if token_type not in WHITESPACE_TOKENS:\n last_non_whitespace_token_type = token_type\n\n return text", "response": "Return source code based on tokens."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef init(globalvars=None, show=False):\n global config\n\n profileini = getprofileini()\n if os.path.exists(profileini):\n config = configparser.ConfigParser()\n config.read(profileini)\n mgr = plugins_get_mgr()\n mgr.update_configs(config)\n\n if show:\n for source in config:\n print(\"[%s] :\" %(source))\n for k in config[source]:\n print(\" %s : %s\" % (k, config[source][k]))\n\n else:\n print(\"Profile does not exist. So creating one\")\n if not show:\n update(globalvars)\n\n print(\"Complete init\")", "response": "Load the INI\n and update the global variables."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the profile and profile ini file with the given global variables.", "response": "def update(globalvars):\n \"\"\"\n Update the profile\n \"\"\"\n global config\n\n profileini = getprofileini()\n config = configparser.ConfigParser()\n config.read(profileini)\n defaults = {}\n\n if globalvars is not None:\n defaults = {a[0]: a[1] for a in globalvars }\n\n # Generic variables to be captured...\n generic_configs = [{\n 'name': 'User',\n 'nature': 'generic',\n 'description': \"General information\",\n 'variables': ['user.email', 'user.name',\n 'user.fullname'],\n 'defaults': {\n 'user.email': {\n 'value': defaults.get('user.email',''),\n 'description': \"Email address\",\n 'validator': EmailValidator()\n },\n 'user.fullname': {\n 'value': defaults.get('user.fullname',''),\n 'description': \"Full Name\",\n 'validator': NonEmptyValidator()\n },\n 'user.name': {\n 'value': defaults.get('user.name', getpass.getuser()),\n 'description': \"Name\",\n 'validator': NonEmptyValidator()\n },\n }\n }]\n\n # Gather configuration requirements from all plugins\n mgr = plugins_get_mgr()\n extra_configs = mgr.gather_configs()\n allconfigs = generic_configs + extra_configs\n\n # Read the existing config and update the defaults\n for c in allconfigs:\n name = c['name']\n for v in c['variables']:\n try:\n c['defaults'][v]['value'] = config[name][v]\n except:\n continue\n\n for c in allconfigs:\n\n print(\"\")\n print(c['description'])\n print(\"==================\")\n if len(c['variables']) == 0:\n print(\"Nothing to do. Enabled by default\")\n continue\n\n name = c['name']\n config[name] = {}\n config[name]['nature'] = c['nature']\n for v in c['variables']:\n\n # defaults\n value = ''\n description = v + \" \"\n helptext = \"\"\n validator = None\n\n # Look up pre-set values\n if v in c['defaults']:\n value = c['defaults'][v].get('value','')\n helptext = c['defaults'][v].get(\"description\",\"\")\n validator = c['defaults'][v].get('validator',None)\n if helptext != \"\":\n description += \"(\" + helptext + \")\"\n\n # Get user input..\n while True:\n choice = input_with_default(description, value)\n if validator is not None:\n if validator.is_valid(choice):\n break\n else:\n print(\"Invalid input. Expected input is {}\".format(validator.message))\n else:\n break\n\n config[name][v] = choice\n\n if v == 'enable' and choice == 'n': \n break \n\n with open(profileini, 'w') as fd:\n config.write(fd)\n\n print(\"Updated profile file:\", config)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating the post - receive hook file and insert hook into the repo", "response": "def init_repo(self, gitdir):\n \"\"\"\n Insert hook into the repo\n \"\"\"\n\n hooksdir = os.path.join(gitdir, 'hooks')\n content = postreceive_template % {\n 'client': self.client,\n 'bucket': self.bucket,\n 's3cfg': self.s3cfg,\n 'prefix': self.prefix\n }\n\n postrecv_filename =os.path.join(hooksdir, 'post-receive')\n with open(postrecv_filename,'w') as fd:\n fd.write(content)\n\n self.make_hook_executable(postrecv_filename)\n print(\"Wrote to\", postrecv_filename)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntrying the library to compute the sha256 hash of the file.", "response": "def compute_sha256(filename):\n \"\"\"\n Try the library. If it doesnt work, use the command line..\n \"\"\"\n try:\n h = sha256()\n fd = open(filename, 'rb')\n while True:\n buf = fd.read(0x1000000)\n if buf in [None, \"\"]:\n break\n h.update(buf.encode('utf-8'))\n fd.close()\n return h.hexdigest()\n except:\n output = run([\"sha256sum\", \"-b\", filename])\n return output.split(\" \")[0]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(cmd):\n cmd = [pipes.quote(c) for c in cmd]\n cmd = \" \".join(cmd)\n cmd += \"; exit 0\"\n # print(\"Running {} in {}\".format(cmd, os.getcwd()))\n try:\n output = subprocess.check_output(cmd,\n stderr=subprocess.STDOUT,\n shell=True)\n except subprocess.CalledProcessError as e:\n output = e.output\n\n output = output.decode('utf-8')\n output = output.strip()\n return output", "response": "Run a shell command\nTaxonomy"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef log_repo_action(func): \n \n def _inner(*args, **kwargs):\n result = func(*args, **kwargs) \n log_action(func, result, *args, **kwargs)\n return result \n \n _inner.__name__ = func.__name__\n _inner.__doc__ = func.__doc__\n return _inner", "response": "Decorator to log all repo actions to. dgit / log. json \n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_tree(gitdir=\".\"):\n\n cmd = [\"git\", \"log\", \"--all\", \"--branches\", '--pretty=format:{ \"commit\": \"%H\", \"abbreviated_commit\": \"%h\", \"tree\": \"%T\", \"abbreviated_tree\": \"%t\", \"parent\": \"%P\", \"abbreviated_parent\": \"%p\", \"refs\": \"%d\", \"encoding\": \"%e\", \"subject\": \"%s\", \"sanitized_subject_line\": \"%f\", \"commit_notes\": \"\", \"author\": { \"name\": \"%aN\", \"email\": \"%aE\", \"date\": \"%ai\" }, \"commiter\": { \"name\": \"%cN\", \"email\": \"%cE\", \"date\": \"%ci\" }},']\n\n output = run(cmd)\n lines = output.split(\"\\n\")\n\n content = \"\"\n history = []\n for l in lines:\n try:\n revisedcontent = content + l\n if revisedcontent.count('\"') % 2 == 0:\n j = json.loads(revisedcontent[:-1])\n if \"Notes added by\" in j['subject']:\n content = \"\"\n continue\n history.append(j)\n content = \"\"\n else:\n content = revisedcontent\n except Exception as e:\n print(\"Error while parsing record\")\n print(revisedcontent)\n content = \"\"\n\n # Order by time. First commit first...\n history.reverse()\n\n #\n changes = get_change()\n\n for i in range(len(history)):\n abbrev_commit = history[i]['abbreviated_commit']\n if abbrev_commit not in changes:\n raise Exception(\"Missing changes for \" + abbrev_commit)\n\n history[i]['changes'] = changes[abbrev_commit]['changes']\n\n\n return history", "response": "Get the commit history for a given dataset"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the diffs for all the files in the current language", "response": "def get_diffs(history):\n \"\"\"\n Look at files and compute the diffs intelligently\n \"\"\"\n\n # First get all possible representations\n mgr = plugins_get_mgr() \n keys = mgr.search('representation')['representation']\n representations = [mgr.get_by_key('representation', k) for k in keys]\n\n for i in range(len(history)):\n if i+1 > len(history) - 1:\n continue\n\n prev = history[i]\n curr = history[i+1]\n\n #print(prev['subject'], \"==>\", curr['subject'])\n #print(curr['changes'])\n for c in curr['changes']:\n \n path = c['path']\n\n # Skip the metadata file\n if c['path'].endswith('datapackage.json'): \n continue \n\n # Find a handler for this kind of file...\n handler = None \n for r in representations: \n if r.can_process(path): \n handler = r \n break \n \n if handler is None: \n continue \n\n # print(path, \"being handled by\", handler)\n\n v1_hex = prev['commit']\n v2_hex = curr['commit']\n\n temp1 = tempfile.mkdtemp(prefix=\"dgit-diff-\") \n \n try: \n for h in [v1_hex, v2_hex]: \n filename = '{}/{}/checkout.tar'.format(temp1, h)\n try:\n os.makedirs(os.path.dirname(filename))\n except:\n pass \n extractcmd = ['git', 'archive', '-o', filename, h, path]\n output = run(extractcmd)\n if 'fatal' in output: \n raise Exception(\"File not present in commit\") \n with cd(os.path.dirname(filename)): \n cmd = ['tar', 'xvf', 'checkout.tar']\n output = run(cmd) \n if 'fatal' in output: \n print(\"Cleaning up - fatal 1\", temp1)\n shutil.rmtree(temp1)\n continue \n\n # Check to make sure that \n path1 = os.path.join(temp1, v1_hex, path) \n path2 = os.path.join(temp1, v2_hex, path) \n if not os.path.exists(path1) or not os.path.exists(path2): \n # print(\"One of the two output files is missing\") \n shutil.rmtree(temp1)\n continue \n\n #print(path1, path2) \n\n # Now call the handler\n diff = handler.get_diff(path1, path2)\n\n # print(\"Inserting diff\", diff)\n c['diff'] = diff\n\n except Exception as e: \n #traceback.print_exc() \n #print(\"Cleaning up - Exception \", temp1)\n shutil.rmtree(temp1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nchange the current directory of the log entry.", "response": "def chdir(self, new_pwd, relative=True):\n \"\"\"\n Parameters\n ----------\n new_pwd: str,\n Directory to change to\n relative: bool, default True\n If True then the given directory is treated as relative to the\n current directory\n \"\"\"\n if new_pwd and self.pwd and relative:\n new_pwd = os.path.join(self.pwd, new_pwd)\n self.pwd = new_pwd"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef exec_command(self, cmd):\n if self.pwd is not None:\n cmd = 'cd %s ; %s' % (self.pwd, cmd)\n if self.interactive:\n print(cmd)\n return self.con.exec_command(cmd)", "response": "Execute a command and return the result."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nexecutes command and wait for it to finish.", "response": "def wait(self, cmd, raise_on_error=True):\n \"\"\"\n Execute command and wait for it to finish. Proceed with caution because\n if you run a command that causes a prompt this will hang\n \"\"\"\n _, stdout, stderr = self.exec_command(cmd)\n stdout.channel.recv_exit_status()\n output = stdout.read()\n if self.interactive:\n print(output)\n errors = stderr.read()\n if self.interactive:\n print(errors)\n if errors and raise_on_error:\n raise ValueError(errors)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apt(self, package_names, raise_on_error=False):\n if isinstance(package_names, basestring):\n package_names = [package_names]\n cmd = \"apt-get install -y %s\" % (' '.join(package_names))\n return self.wait(cmd, raise_on_error=raise_on_error)", "response": "Install specified packages using apt - get. Waits for command to finish."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninstall specified python packages using pip - U option added Waits for command to finish.", "response": "def pip(self, package_names, raise_on_error=True):\n \"\"\"\n Install specified python packages using pip. -U option added\n Waits for command to finish.\n\n Parameters\n ----------\n package_names: list-like of str\n raise_on_error: bool, default True\n If True then raise ValueError if stderr is not empty\n \"\"\"\n if isinstance(package_names, basestring):\n package_names = [package_names]\n cmd = \"pip install -U %s\" % (' '.join(package_names))\n return self.wait(cmd, raise_on_error=raise_on_error)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pip_r(self, requirements, raise_on_error=True):\n cmd = \"pip install -r %s\" % requirements\n return self.wait(cmd, raise_on_error=raise_on_error)", "response": "Install all requirements contained in the given file path"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate fiji - macros for stitching all channels and z - stacks for a well.", "response": "def stitch_macro(path, output_folder=None):\n \"\"\"Create fiji-macros for stitching all channels and z-stacks for a well.\n\n Parameters\n ----------\n path : string\n Well path.\n output_folder : string\n Folder to store images. If not given well path is used.\n\n Returns\n -------\n output_files, macros : tuple\n Tuple with filenames and macros for stitched well.\n \"\"\"\n output_folder = output_folder or path\n debug('stitching ' + path + ' to ' + output_folder)\n\n fields = glob(_pattern(path, _field))\n\n # assume we have rectangle of fields\n xs = [attribute(field, 'X') for field in fields]\n ys = [attribute(field, 'Y') for field in fields]\n x_min, x_max = min(xs), max(xs)\n y_min, y_max = min(ys), max(ys)\n fields_column = len(set(xs))\n fields_row = len(set(ys))\n\n # assume all fields are the same\n # and get properties from images in first field\n images = glob(_pattern(fields[0], _image))\n\n # assume attributes are the same on all images\n attr = attributes(images[0])\n\n # find all channels and z-stacks\n channels = []\n z_stacks = []\n for image in images:\n channel = attribute_as_str(image, 'C')\n if channel not in channels:\n channels.append(channel)\n\n z = attribute_as_str(image, 'Z')\n if z not in z_stacks:\n z_stacks.append(z)\n\n debug('channels ' + str(channels))\n debug('z-stacks ' + str(z_stacks))\n\n # create macro\n _, extension = os.path.splitext(images[-1])\n if extension == '.tif':\n # assume .ome.tif\n extension = '.ome.tif'\n macros = []\n output_files = []\n for Z in z_stacks:\n for C in channels:\n filenames = os.path.join(\n\n _field + '--X{xx}--Y{yy}',\n _image + '--L' + attr.L +\n '--S' + attr.S +\n '--U' + attr.U +\n '--V' + attr.V +\n '--J' + attr.J +\n '--E' + attr.E +\n '--O' + attr.O +\n '--X{xx}--Y{yy}' +\n '--T' + attr.T +\n '--Z' + Z +\n '--C' + C +\n extension)\n debug('filenames ' + filenames)\n\n cur_attr = attributes(filenames)._asdict()\n f = 'stitched--U{U}--V{V}--C{C}--Z{Z}.png'.format(**cur_attr)\n\n output = os.path.join(output_folder, f)\n debug('output ' + output)\n output_files.append(output)\n if os.path.isfile(output):\n # file already exists\n print('leicaexperiment stitched file already'\n ' exists {}'.format(output))\n continue\n macros.append(fijibin.macro.stitch(path, filenames,\n fields_column, fields_row,\n output_filename=output,\n x_start=x_min, y_start=y_min))\n\n return (output_files, macros)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compress(images, delete_tif=False, folder=None):\n if type(images) == str:\n # only one image\n return [compress_blocking(images, delete_tif, folder)]\n\n filenames = copy(images) # as images property will change when looping\n\n\n return Parallel(n_jobs=_pools)(delayed(compress_blocking)\n (image=image, delete_tif=delete_tif, folder=folder)\n for image in filenames)", "response": "Lossless compression. Save images as PNG and TIFF tags to json."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef compress_blocking(image, delete_tif=False, folder=None, force=False):\n\n debug('compressing {}'.format(image))\n try:\n new_filename, extension = os.path.splitext(image)\n # remove last occurrence of .ome\n new_filename = new_filename.rsplit('.ome', 1)[0]\n\n # if compressed file should be put in specified folder\n if folder:\n basename = os.path.basename(new_filename)\n new_filename = os.path.join(folder, basename + '.png')\n else:\n new_filename = new_filename + '.png'\n\n # check if png exists\n if os.path.isfile(new_filename) and not force:\n compressed_images.append(new_filename)\n msg = \"Aborting compress, PNG already\" \\\n \" exists: {}\".format(new_filename)\n raise AssertionError(msg)\n if extension != '.tif':\n msg = \"Aborting compress, not a TIFF: {}\".format(image)\n raise AssertionError(msg)\n\n # open image, load and close file pointer\n img = Image.open(image)\n fptr = img.fp # keep file pointer, for closing\n img.load() # load img-data before switching mode, also closes fp\n\n # get tags and save them as json\n tags = img.tag.as_dict()\n with open(new_filename[:-4] + '.json', 'w') as f:\n if img.mode == 'P':\n # keep palette\n tags['palette'] = img.getpalette()\n json.dump(tags, f)\n\n # check if image is palette-mode\n if img.mode == 'P':\n # switch to luminance to keep data intact\n debug('palette-mode switched to luminance')\n img.mode = 'L'\n if img.mode == 'I;16':\n # https://github.com/python-pillow/Pillow/issues/1099\n img = img.convert(mode='I')\n\n # compress/save\n debug('saving to {}'.format(new_filename))\n img.save(new_filename)\n\n fptr.close() # windows bug Pillow\n if delete_tif:\n os.remove(image)\n\n except (IOError, AssertionError) as e:\n # print error - continue\n print('leicaexperiment {}'.format(e))\n return ''\n\n return new_filename", "response": "Compresses a single image and saves it as json."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef decompress(images, delete_png=False, delete_json=False, folder=None):\n if type(images) == str:\n # only one image\n return decompress([images])\n\n filenames = copy(images) # as images property will change when looping\n\n decompressed_images = []\n for orig_filename in filenames:\n debug('decompressing {}'.format(orig_filename))\n try:\n filename, extension = os.path.splitext(orig_filename)\n\n # if decompressed file should be put in specified folder\n if folder:\n basename = os.path.basename(filename)\n new_filename = os.path.join(folder, basename + '.ome.tif')\n else:\n new_filename = filename + '.ome.tif'\n\n # check if tif exists\n if os.path.isfile(new_filename):\n decompressed_images.append(new_filename)\n msg = \"Aborting decompress, TIFF already exists:\" \\\n \" {}\".format(orig_filename)\n raise AssertionError(msg)\n if extension != '.png':\n msg = \"Aborting decompress, not a \" \\\n \"PNG: {}\".format(orig_filename)\n raise AssertionError(msg)\n\n # open image, load and close file pointer\n img = Image.open(orig_filename)\n img.load() # load img-data before switching mode, also closes fp\n\n # get tags from json\n info = {}\n with open(filename + '.json', 'r') as f:\n tags = json.load(f)\n # convert dictionary to original types (lost in json conversion)\n for tag,val in tags.items():\n if tag == 'palette':\n # hack hack\n continue\n if type(val) == list:\n val = tuple(val)\n if type(val[0]) == list:\n # list of list\n val = tuple(tuple(x) for x in val)\n info[int(tag)] = val\n\n # check for color map\n if 'palette' in tags:\n img.putpalette(tags['palette'])\n\n # save as tif\n debug('saving to {}'.format(new_filename))\n img.save(new_filename, tiffinfo=info)\n decompressed_images.append(new_filename)\n\n if delete_png:\n os.remove(orig_filename)\n if delete_json:\n os.remove(filename + '.json')\n\n except (IOError, AssertionError) as e:\n # print error - continue\n print('leicaexperiment {}'.format(e))\n\n return decompressed_images", "response": "Reverse compression from tif to png and save them in original format\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the two numbers found behind the given name in path.", "response": "def attribute(path, name):\n \"\"\"Returns the two numbers found behind --[A-Z] in path. If several matches\n are found, the last one is returned.\n\n Parameters\n ----------\n path : string\n String with path of file/folder to get attribute from.\n name : string\n Name of attribute to get. Should be A-Z or a-z (implicit converted to\n uppercase).\n\n Returns\n -------\n integer\n Returns number found in path behind --name as an integer.\n \"\"\"\n matches = re.findall('--' + name.upper() + '([0-9]{2})', path)\n if matches:\n return int(matches[-1])\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the two digit number found behind name in path.", "response": "def attribute_as_str(path, name):\n \"\"\"Returns the two numbers found behind --[A-Z] in path. If several matches\n are found, the last one is returned.\n\n Parameters\n ----------\n path : string\n String with path of file/folder to get attribute from.\n name : string\n Name of attribute to get. Should be A-Z or a-z (implicit converted to\n uppercase).\n\n Returns\n -------\n string\n Returns two digit number found in path behind --name.\n \"\"\"\n matches = re.findall('--' + name.upper() + '([0-9]{2})', path)\n if matches:\n return matches[-1]\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting attributes from path based on format - X - X01.tif. Returns namedtuple with upper case attributes equal to what found in path and lower case attributes as int.", "response": "def attributes(path):\n \"\"\"Get attributes from path based on format --[A-Z]. Returns namedtuple\n with upper case attributes equal to what found in path (string) and lower\n case as int. If path holds several occurrences of same character, only the\n last one is kept.\n\n >>> attrs = attributes('/folder/file--X00-X01.tif')\n >>> print(attrs)\n namedtuple('attributes', 'X x')('01', 1)\n >>> print(attrs.x)\n 1\n\n Parameters\n ----------\n path : string\n\n Returns\n -------\n collections.namedtuple\n \"\"\"\n # number of charcters set to numbers have changed in LAS AF X !!\n matches = re.findall('--([A-Z]{1})([0-9]{2,4})', path)\n\n keys = []\n values = []\n for k,v in matches:\n if k in keys:\n # keep only last key\n i = keys.index(k)\n del keys[i]\n del values[i]\n keys.append(k)\n values.append(v)\n\n lower_keys = [k.lower() for k in keys]\n int_values= [int(v) for v in values]\n\n attributes = namedtuple('attributes', keys + lower_keys)\n\n return attributes(*values + int_values)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a globbing pattern for names and extension.", "response": "def _pattern(*names, **kwargs):\n \"\"\"Returns globbing pattern for name1/name2/../lastname + '--*' or\n name1/name2/../lastname + extension if parameter `extension` it set.\n\n Parameters\n ----------\n names : strings\n Which path to join. Example: _pattern('path', 'to', 'experiment') will\n return `path/to/experiment--*`.\n extension : string\n If other extension then --* is wanted.\n Example: _pattern('path', 'to', 'image', extension='*.png') will return\n `path/to/image*.png`.\n\n Returns\n -------\n string\n Joined glob pattern string.\n \"\"\"\n if 'extension' not in kwargs:\n kwargs['extension'] = '--*'\n return os.path.join(*names) + kwargs['extension']"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _set_path(self, path):\n \"Set self.path, self.dirname and self.basename.\"\n import os.path\n self.path = os.path.abspath(path)\n self.dirname = os.path.dirname(path)\n self.basename = os.path.basename(path)", "response": "Set self. path self. dirname and self. basename."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef images(self):\n \"List of paths to images.\"\n tifs = _pattern(self._image_path, extension='tif')\n pngs = _pattern(self._image_path, extension='png')\n imgs = []\n imgs.extend(glob(tifs))\n imgs.extend(glob(pngs))\n return imgs", "response": "List of paths to images."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef well_rows(self, well_row, well_column):\n return list(set([attribute(img, 'u') for img in self.images]))", "response": "Returns a list of integers representing the set of images that are in the well row."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget path of specified image.", "response": "def image(self, well_row, well_column, field_row, field_column):\n \"\"\"Get path of specified image.\n\n Parameters\n ----------\n well_row : int\n Starts at 0. Same as --U in files.\n well_column : int\n Starts at 0. Same as --V in files.\n field_row : int\n Starts at 0. Same as --Y in files.\n field_column : int\n Starts at 0. Same as --X in files.\n\n Returns\n -------\n string\n Path to image or empty string if image is not found.\n \"\"\"\n return next((i for i in self.images\n if attribute(i, 'u') == well_column and\n attribute(i, 'v') == well_row and\n attribute(i, 'x') == field_column and\n attribute(i, 'y') == field_row), '')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef well_images(self, well_row, well_column):\n return list(i for i in self.images\n if attribute(i, 'u') == well_column and\n attribute(i, 'v') == well_row)", "response": "Get list of paths to images in specified well."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of ints where each element is an attribute of the x attribute of the image that is found in the well.", "response": "def field_columns(self, well_row, well_column):\n \"\"\"Field columns for given well. Equivalent to --X in files.\n\n Parameters\n ----------\n well_row : int\n Starts at 0. Same as --V in files.\n well_column : int\n Starts at 0. Same as --U in files.\n\n Returns\n -------\n list of ints\n Columns found for specified well.\n \"\"\"\n imgs = self.well_images(well_row, well_column)\n return list(set([attribute(img, 'x') for img in imgs]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stitch(self, folder=None):\n debug('stitching ' + self.__str__())\n if not folder:\n folder = self.path\n\n # create list of macros and files\n macros = []\n files = []\n for well in self.wells:\n f,m = stitch_macro(well, folder)\n macros.extend(m)\n files.extend(f)\n\n chopped_arguments = zip(chop(macros, _pools), chop(files, _pools))\n chopped_filenames = Parallel(n_jobs=_pools)(delayed(fijibin.macro.run)\n (macro=arg[0], output_files=arg[1])\n for arg in chopped_arguments)\n\n # flatten\n return [f for list_ in chopped_filenames for f in list_]", "response": "Stitches all wells in experiment with ImageJ."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compress(self, delete_tif=False, folder=None):\n return compress(self.images, delete_tif, folder)", "response": "Lossless compress all images in experiment to PNG."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef field_metadata(self, well_row=0, well_column=0,\n field_row=0, field_column=0):\n \"\"\"Get OME-XML metadata of given field.\n\n Parameters\n ----------\n well_row : int\n Y well coordinate. Same as --V in files.\n well_column : int\n X well coordinate. Same as --U in files.\n field_row : int\n Y field coordinate. Same as --Y in files.\n field_column : int\n X field coordinate. Same as --X in files.\n\n Returns\n -------\n lxml.objectify.ObjectifiedElement\n lxml object of OME-XML found in slide/chamber/field/metadata.\n \"\"\"\n def condition(path):\n attrs = attributes(path)\n return (attrs.u == well_column and attrs.v == well_row\n and attrs.x == field_column and attrs.y == field_row)\n\n field = [f for f in self.fields if condition(f)]\n\n if field:\n field = field[0]\n filename = _pattern(field, 'metadata',\n _image, extension='*.ome.xml')\n filename = glob(filename)[0] # resolve, assume found\n return objectify.parse(filename).getroot()", "response": "Get OME - XML metadata of given field."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a list of stitch coordinates for the given well.", "response": "def stitch_coordinates(self, well_row=0, well_column=0):\n \"\"\"Get a list of stitch coordinates for the given well.\n\n Parameters\n ----------\n well_row : int\n Y well coordinate. Same as --V in files.\n well_column : int\n X well coordinate. Same as --U in files.\n\n Returns\n -------\n (xs, ys, attr) : tuples with float and collections.OrderedDict\n Tuple of x's, y's and attributes.\n \"\"\"\n well = [w for w in self.wells\n if attribute(w, 'u') == well_column and\n attribute(w, 'v') == well_row]\n\n if len(well) == 1:\n well = well[0]\n tile = os.path.join(well, 'TileConfiguration.registered.txt')\n\n with open(tile) as f:\n data = [x.strip()\n for l in f.readlines()\n if l[0:7] == 'image--'\n for x in l.split(';')] # flat list\n coordinates = (ast.literal_eval(x) for x in data[2::3])\n # flatten\n coordinates = sum(coordinates, ())\n attr = tuple(attributes(x) for x in data[0::3])\n return coordinates[0::2], coordinates[1::2], attr\n\n else:\n print('leicaexperiment stitch_coordinates'\n '({}, {}) Well not found'.format(well_row, well_column))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new droplet in a specific region.", "response": "def create(self, name, region, size, image, ssh_keys=None,\n backups=None, ipv6=None, private_networking=None, wait=True):\n \"\"\"\n Create a new droplet\n\n Parameters\n ----------\n name: str\n Name of new droplet\n region: str\n slug for region (e.g., sfo1, nyc1)\n size: str\n slug for droplet size (e.g., 512mb, 1024mb)\n image: int or str\n image id (e.g., 12352) or slug (e.g., 'ubuntu-14-04-x64')\n ssh_keys: list, optional\n default SSH keys to be added on creation\n this is highly recommended for ssh access\n backups: bool, optional\n whether automated backups should be enabled for the Droplet.\n Automated backups can only be enabled when the Droplet is created.\n ipv6: bool, optional\n whether IPv6 is enabled on the Droplet\n private_networking: bool, optional\n whether private networking is enabled for the Droplet. Private\n networking is currently only available in certain regions\n wait: bool, default True\n if True then block until creation is complete\n \"\"\"\n if ssh_keys and not isinstance(ssh_keys, (list, tuple)):\n raise TypeError(\"ssh_keys must be a list\")\n resp = self.post(name=name, region=region, size=size, image=image,\n ssh_keys=ssh_keys,\n private_networking=private_networking,\n backups=backups, ipv6=ipv6)\n droplet = self.get(resp[self.singular]['id'])\n if wait:\n droplet.wait()\n # HACK sometimes the IP address doesn't return correctly\n droplet = self.get(resp[self.singular]['id'])\n return droplet"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves a droplet by id", "response": "def get(self, id):\n \"\"\"\n Retrieve a droplet by id\n\n Parameters\n ----------\n id: int\n droplet id\n\n Returns\n -------\n droplet: DropletActions\n \"\"\"\n info = self._get_droplet_info(id)\n return DropletActions(self.api, self, **info)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nretrieving a droplet by name", "response": "def by_name(self, name):\n \"\"\"\n Retrieve a droplet by name (return first if duplicated)\n\n Parameters\n ----------\n name: str\n droplet name\n\n Returns\n -------\n droplet: DropletActions\n \"\"\"\n for d in self.list():\n if d['name'] == name:\n return self.get(d['id'])\n raise KeyError(\"Could not find droplet with name %s\" % name)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef resize(self, size, wait=True):\n return self._action('resize', size=size, wait=wait)", "response": "Change the size of this droplet"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef restore(self, image, wait=True):\n return self._action('restore', image=image, wait=wait)", "response": "Restores the droplet with given image id"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rebuild(self, image, wait=True):\n return self._action('rebuild', image=image, wait=wait)", "response": "Rebuild this droplet with given image id"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rename(self, name, wait=True):\n return self._action('rename', name=name, wait=wait)", "response": "Change the name of the droplet\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef change_kernel(self, kernel_id, wait=True):\n return self._action('change_kernel', kernel=kernel_id, wait=wait)", "response": "Change the kernel of this droplet"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef take_snapshot(self, name, wait=True):\n return self._action('snapshot', name=name, wait=wait)", "response": "Take a snapshot of the current droplet"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndelete this droplet Parameters ---------- wait: bool, default True Whether to block until the pending action is completed", "response": "def delete(self, wait=True):\n \"\"\"\n Delete this droplet\n\n Parameters\n ----------\n wait: bool, default True\n Whether to block until the pending action is completed\n \"\"\"\n resp = self.parent.delete(self.id)\n if wait:\n self.wait()\n return resp"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwaiting for all actions to complete on a droplet", "response": "def wait(self):\n \"\"\"\n wait for all actions to complete on a droplet\n \"\"\"\n interval_seconds = 5\n while True:\n actions = self.actions()\n slept = False\n for a in actions:\n if a['status'] == 'in-progress':\n # n.b. gevent will monkey patch\n time.sleep(interval_seconds)\n slept = True\n break\n if not slept:\n break"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ip_address(self):\n ip = None\n for eth in self.networks['v4']:\n if eth['type'] == 'public':\n ip = eth['ip_address']\n break\n if ip is None:\n raise ValueError(\"No public IP found\")\n return ip", "response": "Returns public IP address"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the private IP address of the current node.", "response": "def private_ip(self):\n \"\"\"\n Private ip_address\n \"\"\"\n ip = None\n for eth in self.networks['v4']:\n if eth['type'] == 'private':\n ip = eth['ip_address']\n break\n if ip is None:\n raise ValueError(\"No private IP found\")\n return ip"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef connect(self, interactive=False):\n from poseidon.ssh import SSHClient\n rs = SSHClient(self.ip_address, interactive=interactive)\n return rs", "response": "Connect to the droplet s local cache."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending a request to the REST API", "response": "def send_request(self, kind, resource, url_components, **kwargs):\n \"\"\"\n Send a request to the REST API\n\n Parameters\n ----------\n kind: str, {get, delete, put, post, head}\n resource: str\n url_components: list or tuple to be appended to the request URL\n\n Notes\n -----\n kwargs contain request parameters to be sent as request data\n \"\"\"\n url = self.format_request_url(resource, *url_components)\n meth = getattr(requests, kind)\n headers = self.get_request_headers()\n req_data = self.format_parameters(**kwargs)\n response = meth(url, headers=headers, data=req_data)\n data = self.get_response(response)\n if response.status_code >= 300:\n msg = data.pop('message', 'API request returned error')\n raise APIError(msg, response.status_code, **data)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format_parameters(self, **kwargs):\n req_data = {}\n for k, v in kwargs.items():\n if isinstance(v, (list, tuple)):\n k = k + '[]'\n req_data[k] = v\n return req_data", "response": "Formats the parameters of the get_resource_types call."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating request url for resource", "response": "def format_request_url(self, resource, *args):\n \"\"\"create request url for resource\"\"\"\n return '/'.join((self.api_url, self.api_version, resource) +\n tuple(str(x) for x in args))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a request to the API.", "response": "def send_request(self, kind, url_components, **kwargs):\n \"\"\"\n Send a request for this resource to the API\n\n Parameters\n ----------\n kind: str, {'get', 'delete', 'put', 'post', 'head'}\n \"\"\"\n return self.api.send_request(kind, self.resource_path, url_components,\n **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending list request for all members of a collection", "response": "def list(self, url_components=()):\n \"\"\"\n Send list request for all members of a collection\n \"\"\"\n resp = self.get(url_components)\n return resp.get(self.result_key, [])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, id, **kwargs):\n return (super(MutableCollection, self).get((id,), **kwargs)\n .get(self.singular, None))", "response": "Get single unit of collection"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntransfer this image to given region Returns the unique ID of the image in the given region.", "response": "def transfer(self, region):\n \"\"\"\n Transfer this image to given region\n\n Parameters\n ----------\n region: str\n region slug to transfer to (e.g., sfo1, nyc1)\n \"\"\"\n action = self.post(type='transfer', region=region)['action']\n return self.parent.get(action['resource_id'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a single image", "response": "def get(self, id):\n \"\"\"id or slug\"\"\"\n info = super(Images, self).get(id)\n return ImageActions(self.api, parent=self, **info)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update(self, id, name):\n return super(Keys, self).update(id, name=name)", "response": "Update the id or fingerprint of a key."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a new domain and returns the new domain record.", "response": "def create(self, name, ip_address):\n \"\"\"\n Creates a new domain\n\n Parameters\n ----------\n name: str\n new domain name\n ip_address: str\n IP address for the new domain\n \"\"\"\n return (self.post(name=name, ip_address=ip_address)\n .get(self.singular, None))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of all domain records for the given domain name.", "response": "def records(self, name):\n \"\"\"\n Get a list of all domain records for the given domain name\n\n Parameters\n ----------\n name: str\n domain name\n \"\"\"\n if self.get(name):\n return DomainRecords(self.api, name)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchanges the name of this domain record", "response": "def rename(self, id, name):\n \"\"\"\n Change the name of this domain record\n\n Parameters\n ----------\n id: int\n domain record id\n name: str\n new name of record\n \"\"\"\n return super(DomainRecords, self).update(id, name=name)[self.singular]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create(self, type, name=None, data=None, priority=None,\n port=None, weight=None):\n \"\"\"\n Parameters\n ----------\n type: str\n {A, AAAA, CNAME, MX, TXT, SRV, NS}\n name: str\n Name of the record\n data: object, type-dependent\n type == 'A' : IPv4 address\n type == 'AAAA' : IPv6 address\n type == 'CNAME' : destination host name\n type == 'MX' : mail host name\n type == 'TXT' : txt contents\n type == 'SRV' : target host name to direct requests for the service\n type == 'NS' : name server that is authoritative for the domain\n priority:\n port:\n weight:\n \"\"\"\n if type == 'A' and name is None:\n name = self.domain\n return self.post(type=type, name=name, data=data, priority=priority,\n port=port, weight=weight)[self.singular]", "response": "Creates a new record in the domain."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, id, **kwargs):\n return super(DomainRecords, self).get(id, **kwargs)", "response": "Retrieve a single domain record given the id."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef logon(self, username, password):\n if self._token:\n self.logoff()\n try:\n response = self.__makerequest(\n 'logon', email=username, password=password)\n except FogBugzAPIError:\n e = sys.exc_info()[1]\n raise FogBugzLogonError(e)\n\n self._token = response.token.string\n if type(self._token) == CData:\n self._token = self._token.encode('utf-8')", "response": "Logs the user on to FogBugz."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __encode_multipart_formdata(self, fields, files):\n BOUNDARY = _make_boundary()\n\n if len(files) > 0:\n fields['nFileCount'] = str(len(files))\n\n crlf = '\\r\\n'\n buf = BytesIO()\n\n for k, v in fields.items():\n if DEBUG:\n print(\"field: %s: %s\" % (repr(k), repr(v)))\n lines = [\n '--' + BOUNDARY,\n 'Content-disposition: form-data; name=\"%s\"' % k,\n '',\n str(v),\n '',\n ]\n buf.write(crlf.join(lines).encode('utf-8'))\n\n n = 0\n for f, h in files.items():\n n += 1\n lines = [\n '--' + BOUNDARY,\n 'Content-disposition: form-data; name=\"File%d\"; '\n 'filename=\"%s\"' % (n, f),\n '',\n ]\n buf.write(crlf.join(lines).encode('utf-8'))\n lines = [\n 'Content-type: application/octet-stream',\n '',\n '',\n ]\n buf.write(crlf.join(lines).encode('utf-8'))\n buf.write(h.read())\n buf.write(crlf.encode('utf-8'))\n\n buf.write(('--' + BOUNDARY + '--' + crlf).encode('utf-8'))\n content_type = \"multipart/form-data; boundary=%s\" % BOUNDARY\n return content_type, buf.getvalue()", "response": "Encodes the multipart form data for the current locale."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef chop(list_, n):\n \"Chop list_ into n chunks. Returns a list.\"\n # could look into itertools also, might be implemented there\n size = len(list_)\n each = size // n\n if each == 0:\n return [list_]\n chopped = []\n for i in range(n):\n start = i * each\n end = (i+1) * each\n if i == (n - 1):\n # make sure we get all items, let last worker do a litte more\n end = size\n chopped.append(list_[start:end])\n return chopped", "response": "Chop list_ into n chunks. Returns a list."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntakes a snapshot of a droplet", "response": "def take_snapshot(droplet, name):\n \"\"\"\n Take a snapshot of a droplet\n\n Parameters\n ----------\n name: str\n name for snapshot\n \"\"\"\n print \"powering off\"\n droplet.power_off()\n droplet.wait() # wait for pending actions to complete\n print \"taking snapshot\"\n droplet.take_snapshot(name)\n droplet.wait()\n snapshots = droplet.snapshots()\n print \"Current snapshots\"\n print snapshots"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nretrieving the allowed operations for this request.", "response": "def allowed_operations(self):\n \"\"\"Retrieves the allowed operations for this request.\"\"\"\n if self.slug is not None:\n return self.meta.detail_allowed_operations\n\n return self.meta.list_allowed_operations"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef assert_operations(self, *args):\n if not set(args).issubset(self.allowed_operations):\n raise http.exceptions.Forbidden()", "response": "Assets if the requested operations are allowed in this context."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfills the response object from the passed data.", "response": "def make_response(self, data=None):\n \"\"\"Fills the response object from the passed data.\"\"\"\n if data is not None:\n # Prepare the data for transmission.\n data = self.prepare(data)\n\n # Encode the data using a desired encoder.\n self.response.write(data, serialize=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprocessing a GET request.", "response": "def get(self, request, response):\n \"\"\"Processes a `GET` request.\"\"\"\n # Ensure we're allowed to read the resource.\n self.assert_operations('read')\n\n # Delegate to `read` to retrieve the items.\n items = self.read()\n\n # if self.slug is not None and not items:\n # # Requested a specific resource but nothing is returned.\n\n # # Attempt to resolve by changing what we understand as\n # # a slug to a path.\n # self.path = self.path + self.slug if self.path else self.slug\n # self.slug = None\n\n # # Attempt to retreive the resource again.\n # items = self.read()\n\n # Ensure that if we have a slug and still no items that a 404\n # is rasied appropriately.\n if not items:\n raise http.exceptions.NotFound()\n\n if (isinstance(items, Iterable)\n and not isinstance(items, six.string_types)) and items:\n # Paginate over the collection.\n items = pagination.paginate(self.request, self.response, items)\n\n # Build the response object.\n self.make_response(items)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef post(self, request, response):\n if self.slug is not None:\n # Don't know what to do an item access.\n raise http.exceptions.NotImplemented()\n\n # Ensure we're allowed to create a resource.\n self.assert_operations('create')\n\n # Deserialize and clean the incoming object.\n data = self._clean(None, self.request.read(deserialize=True))\n\n # Delegate to `create` to create the item.\n item = self.create(data)\n\n # Build the response object.\n self.response.status = http.client.CREATED\n self.make_response(item)", "response": "Processes a POST request."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef put(self, request, response):\n if self.slug is None:\n # Mass-PUT is not implemented.\n raise http.exceptions.NotImplemented()\n\n # Check if the resource exists.\n target = self.read()\n\n # Deserialize and clean the incoming object.\n data = self._clean(target, self.request.read(deserialize=True))\n\n if target is not None:\n # Ensure we're allowed to update the resource.\n self.assert_operations('update')\n\n try:\n # Delegate to `update` to create the item.\n self.update(target, data)\n\n except AttributeError:\n # No read method defined.\n raise http.exceptions.NotImplemented()\n\n # Build the response object.\n self.make_response(target)\n\n else:\n # Ensure we're allowed to create the resource.\n self.assert_operations('create')\n\n # Delegate to `create` to create the item.\n target = self.create(data)\n\n # Build the response object.\n self.response.status = http.client.CREATED\n self.make_response(target)", "response": "Processes a PUT request."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprocess a DELETE request.", "response": "def delete(self, request, response):\n \"\"\"Processes a `DELETE` request.\"\"\"\n if self.slug is None:\n # Mass-DELETE is not implemented.\n raise http.exceptions.NotImplemented()\n\n # Ensure we're allowed to destroy a resource.\n self.assert_operations('destroy')\n\n # Delegate to `destroy` to destroy the item.\n self.destroy()\n\n # Build the response object.\n self.response.status = http.client.NO_CONTENT\n self.make_response()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef link(self, request, response):\n from armet.resources.managed.request import read\n\n if self.slug is None:\n # Mass-LINK is not implemented.\n raise http.exceptions.NotImplemented()\n\n # Get the current target.\n target = self.read()\n\n # Collect all the passed link headers.\n links = self._parse_link_headers(request['Link'])\n\n # Pull targets for each represented link.\n for link in links:\n # Delegate to a connector.\n self.relate(target, read(self, link['uri']))\n\n # Build the response object.\n self.response.status = http.client.NO_CONTENT\n self.make_response()", "response": "Processes a LINK request."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a base Django project and returns its ID", "response": "def create_project(self):\n '''\n Creates a base Django project\n '''\n if os.path.exists(self._py):\n prj_dir = os.path.join(self._app_dir, self._project_name)\n if os.path.exists(prj_dir):\n if self._force:\n logging.warn('Removing existing project')\n shutil.rmtree(prj_dir)\n else:\n logging.warn('Found existing project; not creating (use --force to overwrite)')\n return\n logging.info('Creating project')\n p = subprocess.Popen('cd {0} ; {1} startproject {2} > /dev/null'.format(self._app_dir, self._ve_dir + os.sep + self._project_name + \\\n os.sep + 'bin' + os.sep + 'django-admin.py', self._project_name), \\\n shell=True)\n os.waitpid(p.pid, 0)\n else:\n logging.error('Unable to find Python interpreter in virtualenv')\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses the querystring into a normalized form.", "response": "def parse(text, encoding='utf8'):\n \"\"\"Parse the querystring into a normalized form.\"\"\"\n\n # Decode the text if we got bytes.\n if isinstance(text, six.binary_type):\n text = text.decode(encoding)\n\n return Query(text, split_segments(text))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn objects representing segments and combinators.", "response": "def split_segments(text, closing_paren=False):\n \"\"\"Return objects representing segments.\"\"\"\n buf = StringIO()\n\n # The segments we're building, and the combinators used to combine them.\n # Note that after this is complete, this should be true:\n # len(segments) == len(combinators) + 1\n # Thus we can understand the relationship between segments and combinators\n # like so:\n # s1 (c1) s2 (c2) s3 (c3) where sN are segments and cN are combination\n # functions.\n # TODO: Figure out exactly where the querystring died and post cool\n # error messages about it.\n segments = []\n combinators = []\n\n # A flag dictating if the last character we processed was a group.\n # This is used to determine if the next character (being a combinator)\n # is allowed to\n last_group = False\n\n # The recursive nature of this function relies on keeping track of the\n # state of iteration. This iterator will be passed down to recursed calls.\n iterator = iter(text)\n\n # Detection for exclamation points. only matters for this situation:\n # foo=bar&!(bar=baz)\n last_negation = False\n\n for character in iterator:\n if character in COMBINATORS:\n\n if last_negation:\n buf.write(constants.OPERATOR_NEGATION)\n\n # The string representation of our segment.\n val = buf.getvalue()\n reset_stringio(buf)\n\n if not last_group and not len(val):\n raise ValueError('Unexpected %s.' % character)\n\n # When a group happens, the previous value is empty.\n if len(val):\n segments.append(parse_segment(val))\n\n combinators.append(COMBINATORS[character])\n\n elif character == constants.GROUP_BEGIN:\n # Recursively go into the next group.\n\n if buf.tell():\n raise ValueError('Unexpected %s' % character)\n\n seg = split_segments(iterator, True)\n\n if last_negation:\n seg = UnarySegmentCombinator(seg)\n\n segments.append(seg)\n\n # Flag that the last entry was a grouping, so that we don't panic\n # when the next character is a logical combinator\n last_group = True\n continue\n\n elif character == constants.GROUP_END:\n # Build the segment for anything remaining, and then combine\n # all the segments.\n val = buf.getvalue()\n\n # Check for unbalanced parens or an empty thing: foo=bar&();bar=baz\n if not buf.tell() or not closing_paren:\n raise ValueError('Unexpected %s' % character)\n\n segments.append(parse_segment(val))\n return combine(segments, combinators)\n\n elif character == constants.OPERATOR_NEGATION and not buf.tell():\n last_negation = True\n continue\n\n else:\n if last_negation:\n buf.write(constants.OPERATOR_NEGATION)\n if last_group:\n raise ValueError('Unexpected %s' % character)\n buf.write(character)\n\n last_negation = False\n last_group = False\n else:\n # Check and see if the iterator exited early (unbalanced parens)\n if closing_paren:\n raise ValueError('Expected %s.' % constants.GROUP_END)\n\n if not last_group:\n # Add the final segment.\n segments.append(parse_segment(buf.getvalue()))\n\n # Everything completed normally, combine all the segments into one\n # and return them.\n return combine(segments, combinators)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntake a key of type foo : bar and returns either the key and the the directive or None.", "response": "def parse_directive(key):\n \"\"\"\n Takes a key of type (foo:bar) and returns either the key and the\n directive, or the key and None (for no directive.)\n \"\"\"\n if constants.DIRECTIVE in key:\n return key.split(constants.DIRECTIVE, 1)\n else:\n return key, None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_segment(text):\n \"we expect foo=bar\"\n\n if not len(text):\n return NoopQuerySegment()\n\n q = QuerySegment()\n\n # First we need to split the segment into key/value pairs. This is done\n # by attempting to split the sequence for each equality comparison. Then\n # discard any that did not split properly. Then chose the smallest key\n # (greedily chose the first comparator we encounter in the string)\n # followed by the smallest value (greedily chose the largest comparator\n # possible.)\n\n # translate into [('=', 'foo=bar')]\n equalities = zip(constants.OPERATOR_EQUALITIES, itertools.repeat(text))\n # Translate into [('=', ['foo', 'bar'])]\n equalities = map(lambda x: (x[0], x[1].split(x[0], 1)), equalities)\n # Remove unsplit entries and translate into [('=': ['foo', 'bar'])]\n # Note that the result from this stage is iterated over twice.\n equalities = list(filter(lambda x: len(x[1]) > 1, equalities))\n # Get the smallest key and use the length of that to remove other items\n key_len = len(min((x[1][0] for x in equalities), key=len))\n equalities = filter(lambda x: len(x[1][0]) == key_len, equalities)\n\n # Get the smallest value length. thus we have the earliest key and the\n # smallest value.\n op, (key, value) = min(equalities, key=lambda x: len(x[1][1]))\n\n key, directive = parse_directive(key)\n if directive:\n op = constants.OPERATOR_EQUALITY_FALLBACK\n q.directive = directive\n\n # Process negation. This comes in both foo.not= and foo!= forms.\n path = key.split(constants.SEP_PATH)\n last = path[-1]\n\n # Check for !=\n if last.endswith(constants.OPERATOR_NEGATION):\n last = last[:-1]\n q.negated = not q.negated\n\n # Check for foo.not=\n if last == constants.PATH_NEGATION:\n path.pop(-1)\n q.negated = not q.negated\n\n q.values = value.split(constants.SEP_VALUE)\n\n # Check for suffixed operators (foo.gte=bar). Prioritize suffixed\n # entries over actual equality checks.\n if path[-1] in constants.OPERATOR_SUFFIXES:\n\n # The case where foo.gte<=bar, which obviously makes no sense.\n if op not in constants.OPERATOR_FALLBACK:\n raise ValueError(\n 'Both path-style operator and equality style operator '\n 'provided. Please provide only a single style operator.')\n\n q.operator = constants.OPERATOR_SUFFIX_MAP[path[-1]]\n path.pop(-1)\n else:\n q.operator = constants.OPERATOR_EQUALITY_MAP[op]\n\n if not len(path):\n raise ValueError('No attribute navigation path provided.')\n\n q.path = path\n\n return q", "response": "we expect foo = bar"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the value of this attribute for the passed object.", "response": "def set(self, target, value):\n \"\"\"Set the value of this attribute for the passed object.\n \"\"\"\n\n if not self._set:\n return\n\n if self.path is None:\n # There is no path defined on this resource.\n # We can do no magic to set the value.\n self.set = lambda *a: None\n return None\n\n if self._segments[target.__class__]:\n # Attempt to resolve access to this attribute.\n self.get(target)\n\n if self._segments[target.__class__]:\n # Attribute is not fully resolved; an interim segment is null.\n return\n\n # Resolve access to the parent object.\n # For a single-segment path this will effectively be a no-op.\n parent_getter = compose(*self._getters[target.__class__][:-1])\n target = parent_getter(target)\n\n # Make the setter.\n func = self._make_setter(self.path.split('.')[-1], target.__class__)\n\n # Apply the setter now.\n func(target, value)\n\n # Replace this function with the constructed setter.\n def setter(target, value):\n func(parent_getter(target), value)\n\n self.set = setter"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse the given byte - range - specifier into a sequence of tuples.", "response": "def parse(specifiers):\n \"\"\"\n Consumes set specifiers as text and forms a generator to retrieve\n the requested ranges.\n\n @param[in] specifiers\n Expected syntax is from the byte-range-specifier ABNF found in the\n [RFC 2616]; eg. 15-17,151,-16,26-278,15\n\n @returns\n Consecutive tuples that describe the requested range; eg. (1, 72) or\n (1, 1) [read as 1 to 72 or 1 to 1].\n \"\"\"\n specifiers = \"\".join(specifiers.split())\n for specifier in specifiers.split(','):\n if len(specifier) == 0:\n raise ValueError(\"Range: Invalid syntax; missing specifier.\")\n\n count = specifier.count('-')\n if (count and specifier[0] == '-') or not count:\n # Single specifier; return as a tuple to itself.\n yield int(specifier), int(specifier)\n continue\n\n specifier = list(map(int, specifier.split('-')))\n if len(specifier) == 2:\n # Range specifier; return as a tuple.\n if specifier[0] < 0 or specifier[1] < 0:\n # Negative indexing is not supported in range specifiers\n # as stated in the HTTP/1.1 Range header specification.\n raise ValueError(\n \"Range: Invalid syntax; negative indexing \"\n \"not supported in a range specifier.\")\n\n if specifier[1] < specifier[0]:\n # Range must be for at least one item.\n raise ValueError(\n \"Range: Invalid syntax; stop is less than start.\")\n\n # Return them as a immutable tuple.\n yield tuple(specifier)\n continue\n\n # Something weird happened.\n raise ValueError(\"Range: Invalid syntax.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npaginates an iterable during a request.", "response": "def paginate(request, response, items):\n \"\"\"Paginate an iterable during a request.\n\n Magically splicling an iterable in our supported ORMs allows LIMIT and\n OFFSET queries. We should probably delegate this to the ORM or something\n in the future.\n \"\"\"\n # TODO: support dynamic rangewords and page lengths\n # TODO: support multi-part range requests\n\n # Get the header\n header = request.headers.get('Range')\n if not header:\n # No range header; move along.\n return items\n\n # do some validation\n prefix = RANGE_SPECIFIER + '='\n if not header.find(prefix) == 0:\n # This is not using a range specifier that we understand\n raise exceptions.RequestedRangeNotSatisfiable()\n else:\n # Chop the prefix off the header and parse it\n ranges = parse(header[len(prefix):])\n\n ranges = list(ranges)\n if len(ranges) > 1:\n raise exceptions.RequestedRangeNotSatisfiable(\n 'Multiple ranges in a single request is not yet supported.')\n start, end = ranges[0]\n\n # Make sure the length is not higher than the total number allowed.\n max_length = request.resource.count(items)\n end = min(end, max_length)\n\n response.status = client.PARTIAL_CONTENT\n response.headers['Content-Range'] = '%d-%d/%d' % (start, end, max_length)\n response.headers['Accept-Ranges'] = RANGE_SPECIFIER\n\n # Splice and return the items.\n items = items[start:end + 1]\n return items"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef indexesOptional(f):\n stack = inspect.stack()\n _NO_INDEX_CHECK_NEEDED.add('%s.%s.%s' % (f.__module__, stack[1][3], f.__name__))\n del stack\n return f", "response": "Decorate test methods with this if you don t require strict index checking"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(self, deserialize=False, format=None):\n\n if deserialize:\n data, _ = self.deserialize(format=format)\n return data\n\n content = self._read()\n\n if not content:\n return ''\n\n if type(content) is six.binary_type:\n content = content.decode(self.encoding)\n\n return content", "response": "Read and return the request data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the active resource configuration with the passed keyword arguments.", "response": "def use(**kwargs):\n \"\"\"\n Updates the active resource configuration to the passed\n keyword arguments.\n\n Invoking this method without passing arguments will just return the\n active resource configuration.\n\n @returns\n The previous configuration.\n \"\"\"\n config = dict(use.config)\n use.config.update(kwargs)\n return config"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef try_delegation(method):\n '''This decorator wraps descriptor methods with a new method that tries\n to delegate to a function of the same name defined on the owner instance\n for convenience for dispatcher clients.\n '''\n @functools.wraps(method)\n def delegator(self, *args, **kwargs):\n if self.try_delegation:\n # Try to dispatch to the instance's implementation.\n inst = getattr(self, 'inst', None)\n if inst is not None:\n method_name = (self.delegator_prefix or '') + method.__name__\n func = getattr(inst, method_name, None)\n if func is not None:\n return func(*args, **kwargs)\n\n # Otherwise run the decorated func.\n return method(self, *args, **kwargs)\n\n return delegator", "response": "Decorator that tries to delegate to a function of the same name defined on the owner instance\n archive."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef register(self, method, args, kwargs):\n '''Given a single decorated handler function,\n prepare, append desired data to self.registry.\n '''\n invoc = self.dump_invoc(*args, **kwargs)\n self.registry.append((invoc, method.__name__))", "response": "Given a single decorated handler function prepare append desired data to self. registry.\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nyield the methods that this input dispatches to.", "response": "def gen_methods(self, *args, **kwargs):\n '''Find all method names this input dispatches to. This method\n can accept *args, **kwargs, but it's the gen_dispatch method's\n job of passing specific args to handler methods.\n '''\n dispatched = False\n for invoc, methodname in self.registry:\n args, kwargs = self.loads(invoc)\n yield getattr(self.inst, methodname), args, kwargs\n dispatched = True\n\n if dispatched:\n return\n\n # Try the generic handler.\n generic_handler = getattr(self.inst, 'generic_handler', None)\n if generic_handler is not None:\n yield generic_handler, args, kwargs\n\n # Give up.\n msg = 'No method was found for %r on %r.'\n raise self.DispatchError(msg % ((args, kwargs), self.inst))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinding the first method this input dispatches to.", "response": "def get_method(self, *args, **kwargs):\n '''Find the first method this input dispatches to.\n '''\n for method in self.gen_methods(*args, **kwargs):\n return method\n msg = 'No method was found for %r on %r.'\n raise self.DispatchError(msg % ((args, kwargs), self.inst))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dispatch(self, *args, **kwargs):\n '''Find and evaluate/return the first method this input dispatches to.\n '''\n for result in self.gen_dispatch(*args, **kwargs):\n return result", "response": "Find and evaluate and return the first method this input dispatches to.\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef gen_dispatch(self, *args, **kwargs):\n '''Find and evaluate/yield every method this input dispatches to.\n '''\n dispatched = False\n for method_data in self.gen_methods(*args, **kwargs):\n dispatched = True\n\n result = self.apply_handler(method_data, *args, **kwargs)\n yield result\n # return self.yield_from_handler(result)\n if dispatched:\n return\n msg = 'No method was found for %r on %r.'\n raise self.DispatchError(msg % ((args, kwargs), self.inst))", "response": "Find and evaluate and yield every method this input dispatches to.\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gen_method_keys(self, *args, **kwargs):\n '''Given a node, return the string to use in computing the\n matching visitor methodname. Can also be a generator of strings.\n '''\n token = args[0]\n for mro_type in type(token).__mro__[:-1]:\n name = mro_type.__name__\n yield name", "response": "Given a node return the string to use in computing the\n matching visitor methodname. Can also be a generator of strings."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gen_methods(self, *args, **kwargs):\n '''Find all method names this input dispatches to.\n '''\n token = args[0]\n inst = self.inst\n prefix = self._method_prefix\n for method_key in self.gen_method_keys(*args, **kwargs):\n method = getattr(inst, prefix + method_key, None)\n if method is not None:\n yield method\n\n # Fall back to built-in types, then types, then collections.\n typename = type(token).__name__\n yield from self.check_basetype(\n token, typename, self.builtins.get(typename))\n\n for basetype_name in self.interp_types:\n yield from self.check_basetype(\n token, basetype_name, getattr(self.types, basetype_name, None))\n\n for basetype_name in self.abc_types:\n yield from self.check_basetype(\n token, basetype_name, getattr(self.collections, basetype_name, None))\n\n # Try the generic handler.\n yield from self.gen_generic()", "response": "Generate all methods that this input dispatches to."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalls the dispatched function optionally with other data stored during. register and. prepare.", "response": "def apply_handler(self, method_data, *args, **kwargs):\n '''Call the dispatched function, optionally with other data\n stored/created during .register and .prepare. Assume the arguments\n passed in by the dispathcer are the only ones required.\n '''\n if isinstance(method_data, tuple):\n len_method = len(method_data)\n method = method_data[0]\n if 1 < len_method:\n args = method_data[1]\n if 2 < len_method:\n kwargs = method_data[2]\n else:\n method = method_data\n return method(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse a string containing a set of key - value pairs into a new object.", "response": "def parse(cls, s, required=False):\n \"\"\"\n Parse string to create an instance\n\n :param str s: String with requirement to parse\n :param bool required: Is this requirement required to be fulfilled? If not, then it is a filter.\n \"\"\"\n req = pkg_resources.Requirement.parse(s)\n return cls(req, required=required)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add(self, requirements, required=None):\n if isinstance(requirements, RequirementsManager):\n requirements = list(requirements)\n elif not isinstance(requirements, list):\n requirements = [requirements]\n\n for req in requirements:\n name = req.project_name\n\n if not isinstance(req, BumpRequirement):\n req = BumpRequirement(req, required=required)\n elif required is not None:\n req.required = required\n\n add = True\n\n if name in self.requirements:\n for existing_req in self.requirements[name]:\n if req == existing_req:\n add = False\n break\n\n # Need to replace existing as the new req will be used to bump next, and req.required could be\n # updated.\n replace = False\n\n # Two pins: Use highest pinned version\n if (req.specs and req.specs[0][0] == '==' and existing_req.specs and\n existing_req.specs[0][0] == '=='):\n if pkg_resources.parse_version(req.specs[0][1]) < pkg_resources.parse_version(\n existing_req.specs[0][1]):\n req.requirement = existing_req.requirement\n replace = True\n\n # Replace Any\n if not (req.specs and existing_req.specs):\n if existing_req.specs:\n req.requirement = existing_req.requirement\n replace = True\n\n if replace:\n req.required |= existing_req.required\n if existing_req.required_by and not req.required_by:\n req.required_by = existing_req.required_by\n self.requirements[name].remove(existing_req)\n break\n\n if add:\n self.requirements[name].append(req)", "response": "Adds the given list of requirements to the managed version."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check(self, context, version=None):\n req_str = None\n\n self.checked.append((context, version))\n\n if isinstance(context, str) and not version:\n context = BumpRequirement.parse(context)\n\n if isinstance(context, Bump):\n name = context.name\n if context.new_version and context.new_version[0] == '==':\n version = context.new_version[1]\n else:\n req_str = str(context)\n\n elif isinstance(context, (pkg_resources.Requirement, BumpRequirement)):\n name = context.project_name\n if context.specs and context.specs[0][0] == '==':\n version = context.specs[0][1]\n else:\n req_str = str(context)\n\n else:\n name = context\n\n if name in self:\n self.matched_name = True\n\n for req in self[name]:\n if req.required and (version and version in req or req_str == str(req)):\n req.required = False\n return True\n\n return False", "response": "Check if any of the requirements are met by name or version."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if a requirement is already satisfied by what was previously checked", "response": "def satisfied_by_checked(self, req):\n \"\"\"\n Check if requirement is already satisfied by what was previously checked\n\n :param Requirement req: Requirement to check\n \"\"\"\n req_man = RequirementsManager([req])\n\n return any(req_man.check(*checked) for checked in self.checked)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an instance from a pkg_resources. Requirement instance", "response": "def from_requirement(cls, req, changes=None):\n \"\"\" Create an instance from :class:`pkg_resources.Requirement` instance \"\"\"\n return cls(req.project_name, req.specs and ''.join(req.specs[0]) or '', changes=changes)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_requirement(self):\n if self.new_version:\n return pkg_resources.Requirement.parse(self.name + ''.join(self.new_version))\n else:\n return pkg_resources.Requirement.parse(self.name)", "response": "Convert to a pkg_resources. Requirement instance"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding new requirements that must be fulfilled for this bump to occur", "response": "def require(self, req):\n \"\"\" Add new requirements that must be fulfilled for this bump to occur \"\"\"\n reqs = req if isinstance(req, list) else [req]\n\n for req in reqs:\n if not isinstance(req, BumpRequirement):\n req = BumpRequirement(req)\n req.required = True\n req.required_by = self\n self.requirements.append(req)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef requirements_for_changes(self, changes):\n requirements = []\n reqs_set = set()\n\n if isinstance(changes, str):\n changes = changes.split('\\n')\n\n if not changes or changes[0].startswith('-'):\n return requirements\n\n for line in changes:\n line = line.strip(' -+*')\n\n if not line:\n continue\n\n match = IS_REQUIREMENTS_RE2.search(line) # or IS_REQUIREMENTS_RE.match(line)\n if match:\n for match in REQUIREMENTS_RE.findall(match.group(1)):\n if match[1]:\n version = '==' + match[2] if match[1].startswith(' to ') else match[1]\n req_str = match[0] + version\n else:\n req_str = match[0]\n\n if req_str not in reqs_set:\n reqs_set.add(req_str)\n try:\n requirements.append(pkg_resources.Requirement.parse(req_str))\n except Exception as e:\n log.warn('Could not parse requirement \"%s\" from changes: %s', req_str, e)\n\n return requirements", "response": "Parse changes for requirements"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef package_changes(self, name, current_version, new_version):\n if pkg_resources.parse_version(current_version) > pkg_resources.parse_version(new_version):\n downgrade_sign = '- '\n (current_version, new_version) = (new_version, current_version)\n else:\n downgrade_sign = None\n\n changes = self._package_changes(name, current_version, new_version)\n\n if changes and downgrade_sign:\n changes = [downgrade_sign + c for c in changes]\n\n return changes", "response": "Returns a list of changes for a given package name from current_version to new_version in descending order."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _bump(self, existing_req=None, bump_reqs=None):\n if existing_req or bump_reqs and any(r.required for r in bump_reqs):\n name = existing_req and existing_req.project_name or bump_reqs[0].project_name\n\n log.info('Checking %s', name)\n\n bump = current_version = new_version = None\n\n if bump_reqs:\n # BLR: Pin with Many bump requirements\n if self.should_pin() and (len(bump_reqs) > 1 or bump_reqs[0] and\n bump_reqs[0].specs and bump_reqs[0].specs[0][0] != '=='):\n log.debug('Bump to latest within requirements: %s', bump_reqs)\n\n new_version = self.latest_version_for_requirements(bump_reqs)\n current_version = (existing_req and existing_req.specs and existing_req.specs[0][0] == '==' and\n existing_req.specs[0][1])\n\n if current_version == new_version:\n return None\n\n bump = Bump(name, ('==', new_version))\n\n elif len(bump_reqs) > 1:\n raise BumpAccident('Not sure which requirement to use for %s: %s' % (\n name, ', '.join(str(r) for r in bump_reqs)))\n\n # BR: Pin with One bump requirement or Filter with One or Many bump requirements or\n # Bump to Any required.\n elif bump_reqs[0].specs or not (existing_req or self.should_pin() or bump_reqs[0].specs):\n log.debug('Bump to requirement: %s', bump_reqs)\n\n latest_version = self.latest_version_for_requirements(bump_reqs)\n\n new_version = (bump_reqs[0].specs and bump_reqs[0].specs[0][0] == '==' and\n bump_reqs[0].specs[0][1] or latest_version)\n current_version = (existing_req and existing_req.specs and existing_req.specs[0][0] == '==' and\n existing_req.specs[0][1])\n\n if current_version == new_version:\n return None\n\n if len(bump_reqs[0].specs) > 1:\n version = (','.join(s[0] + s[1] for s in bump_reqs[0].specs),)\n elif bump_reqs[0].specs:\n version = bump_reqs[0].specs[0]\n else:\n version = None\n bump = Bump(name, version)\n\n # BL: Pin to Latest\n if not bump and (existing_req and existing_req.specs and existing_req.specs[0][0] == '==' or\n self.should_pin() and not existing_req):\n log.debug('Bump to latest: %s', bump_reqs or name)\n\n current_version = existing_req and existing_req.specs[0][1]\n new_version = self.latest_package_version(name)\n\n if current_version == new_version:\n return None\n\n if not new_version:\n raise BumpAccident('No published version found for %s' % name)\n\n bump = Bump(name, ('==', new_version))\n\n if bump and current_version and new_version and self.detail:\n changes = self.package_changes(bump.name, current_version, new_version)\n bump.changes.extend(changes)\n if self.should_pin():\n bump.require(self.requirements_for_changes(changes))\n\n if bump:\n log.debug('Bumped %s', bump)\n\n if bump.requirements:\n log.info('Changes in %s require: %s',\n bump.name, ', '.join(sorted(str(r) for r in bump.requirements)))\n\n return bump if str(bump) != str(existing_req) else None", "response": "Bump an existing requirement to the desired version if any."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbumps dependencies using given requirements manager.", "response": "def bump(self, bump_reqs=None, **kwargs):\n \"\"\"\n Bump dependencies using given requirements.\n\n :param RequirementsManager bump_reqs: Bump requirements manager\n :param dict kwargs: Additional args from argparse. Some bumpers accept user options, and some not.\n :return: List of :class:`Bump` changes made.\n \"\"\"\n\n bumps = {}\n\n for existing_req in sorted(self.requirements(), key=lambda r: r.project_name):\n if bump_reqs and existing_req.project_name not in bump_reqs:\n continue\n\n bump_reqs.check(existing_req)\n\n try:\n bump = self._bump(existing_req, bump_reqs.get(existing_req.project_name))\n\n if bump:\n bumps[bump.name] = bump\n bump_reqs.check(bump)\n\n except Exception as e:\n if bump_reqs and bump_reqs.get(existing_req.project_name) and all(\n r.required_by is None for r in bump_reqs.get(existing_req.project_name)):\n raise\n else:\n log.warn(e)\n\n for reqs in bump_reqs.required_requirements().values():\n name = reqs[0].project_name\n if name not in bumps and self.should_add(name):\n try:\n bump = self._bump(None, reqs)\n\n if bump:\n bumps[bump.name] = bump\n bump_reqs.check(bump)\n\n except Exception as e:\n if all(r.required_by is None for r in reqs):\n raise\n else:\n log.warn(e)\n\n self.bumps.update(bumps.values())\n\n return bumps.values()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrestore content in target file to be before any changes", "response": "def reverse(self):\n \"\"\" Restore content in target file to be before any changes \"\"\"\n if self._original_target_content:\n with open(self.target, 'w') as fp:\n fp.write(self._original_target_content)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nserialize the object into an acceptable format for transmission.", "response": "def serialize(self, data=None):\n \"\"\"\n Transforms the object into an acceptable format for transmission.\n\n @throws ValueError\n To indicate this serializer does not support the encoding of the\n specified object.\n \"\"\"\n if data is not None and self.response is not None:\n # Set the content type.\n self.response['Content-Type'] = self.media_types[0]\n\n # Write the encoded and prepared data to the response.\n self.response.write(data)\n\n # Return the serialized data.\n # This has normally been transformed by a base class.\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cons(collection, value):\n if isinstance(value, collections.Mapping):\n if collection is None:\n collection = {}\n collection.update(**value)\n\n elif isinstance(value, six.string_types):\n if collection is None:\n collection = []\n collection.append(value)\n\n elif isinstance(value, collections.Iterable):\n if collection is None:\n collection = []\n collection.extend(value)\n\n else:\n if collection is None:\n collection = []\n collection.append(value)\n\n return collection", "response": "Extends a collection with a value."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmerges a named option collection.", "response": "def _merge(options, name, bases, default=None):\n \"\"\"Merges a named option collection.\"\"\"\n result = None\n for base in bases:\n if base is None:\n continue\n\n value = getattr(base, name, None)\n if value is None:\n continue\n\n result = utils.cons(result, value)\n\n value = options.get(name)\n if value is not None:\n result = utils.cons(result, value)\n\n return result or default"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse a string into a list of pkg_resources. Requirement instances.", "response": "def parse_requirements(requirements, in_file=None):\n \"\"\"\n Parse string requirements into list of :class:`pkg_resources.Requirement` instances\n\n :param str requirements: Requirements text to parse\n :param str in_file: File the requirements came from\n :return: List of requirements\n :raises ValueError: if failed to parse\n \"\"\"\n try:\n return list(pkg_resources.parse_requirements(requirements))\n except Exception as e:\n in_file = ' in %s' % in_file if in_file else ''\n raise ValueError('{} {}'.format(e, in_file))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets all package info for given package", "response": "def package_info(cls, package):\n \"\"\" All package info for given package \"\"\"\n\n if package not in cls.package_info_cache:\n package_json_url = 'https://pypi.python.org/pypi/%s/json' % package\n\n try:\n logging.getLogger('requests').setLevel(logging.WARN)\n response = requests.get(package_json_url)\n response.raise_for_status()\n\n cls.package_info_cache[package] = simplejson.loads(response.text)\n\n except Exception as e:\n log.debug('Could not get package info from %s: %s', package_json_url, e)\n cls.package_info_cache[package] = None\n\n return cls.package_info_cache[package]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef all_package_versions(package):\n info = PyPI.package_info(package)\n return info and sorted(info['releases'].keys(), key=lambda x: x.split(), reverse=True) or []", "response": "Return a list of all versions of a package."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninsert a value at the passed index in the named header.", "response": "def insert(self, name, index, value):\n \"\"\"Insert a value at the passed index in the named header.\"\"\"\n return self._sequence[name].insert(index, value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nflushing and close the response stream.", "response": "def close(self):\n \"\"\"Flush and close the stream.\n\n This is called automatically by the base resource on resources\n unless the resource is operating asynchronously; in that case,\n this method MUST be called in order to signal the end of the request.\n If not the request will simply hang as it is waiting for some\n thread to tell it to return to the client.\n \"\"\"\n\n # Ensure we're not closed.\n self.require_not_closed()\n\n if not self.streaming or self.asynchronous:\n # We're not streaming, auto-write content-length if not\n # already set.\n if 'Content-Length' not in self.headers:\n self.headers['Content-Length'] = self.tell()\n\n # Flush out the current buffer.\n self.flush()\n\n # We're done with the response; inform the HTTP connector\n # to close the response stream.\n self._closed = True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write(self, chunk, serialize=False, format=None):\n\n # Ensure we're not closed.\n self.require_not_closed()\n\n if chunk is None:\n # There is nothing here.\n return\n\n if serialize or format is not None:\n # Forward to the serializer to serialize the chunk\n # before it gets written to the response.\n self.serialize(chunk, format=format)\n return # `serialize` invokes write(...)\n\n if type(chunk) is six.binary_type:\n # Update the stream length.\n self._length += len(chunk)\n\n # If passed a byte string, we hope the user encoded it properly.\n self._stream.write(chunk)\n\n elif isinstance(chunk, six.string_types):\n encoding = self.encoding\n if encoding is not None:\n # If passed a string, we can encode it for the user.\n chunk = chunk.encode(encoding)\n\n else:\n # Bail; we don't have an encoding.\n raise exceptions.InvalidOperation(\n 'Attempting to write textual data without an encoding.')\n\n # Update the stream length.\n self._length += len(chunk)\n\n # Write the encoded data into the byte stream.\n self._stream.write(chunk)\n\n elif isinstance(chunk, collections.Iterable):\n # If passed some kind of iterator, attempt to recurse into\n # oblivion.\n for section in chunk:\n self.write(section)\n\n else:\n # Bail; we have no idea what to do with this.\n raise exceptions.InvalidOperation(\n 'Attempting to write something not recognized.')", "response": "Writes the given chunk to the output buffer."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef serialize(self, data, format=None):\n return self._resource.serialize(data, response=self, format=format)", "response": "Serializes the data into this response using a serializer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef flush(self):\n\n # Ensure we're not closed.\n self.require_not_closed()\n\n # Pull out the accumulated chunk.\n chunk = self._stream.getvalue()\n self._stream.truncate(0)\n self._stream.seek(0)\n\n # Append the chunk to the body.\n self.body = chunk if (self._body is None) else (self._body + chunk)\n\n if self.asynchronous:\n # We are now streaming because we're asynchronous.\n self.streaming = True", "response": "Flush the write buffers of the stream."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send(self, *args, **kwargs):\n self.write(*args, **kwargs)\n self.flush()", "response": "Writes the passed chunk and flushes it to the client."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef end(self, *args, **kwargs):\n self.send(*args, **kwargs)\n self.close()", "response": "Closes the connection and sends the passed chunk to the client and closes the connection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninsert a value at the passed index in the named header.", "response": "def insert(self, name, index, value):\n \"\"\"Insert a value at the passed index in the named header.\"\"\"\n return self.headers.insert(index, value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a base Flask project and a base Flask project stubs it with the specified name.", "response": "def create_project(self):\n \"\"\"\n Creates a base Flask project\n\n \"\"\"\n if os.path.exists(self._py):\n prj_dir = os.path.join(self._app_dir, self._project_name)\n if os.path.exists(prj_dir):\n if self._force:\n logging.warn('Removing existing project')\n shutil.rmtree(prj_dir)\n else:\n logging.warn('Found existing project; not creating (use --force to overwrite)')\n return\n logging.info('Creating project')\n os.makedirs(prj_dir)\n # create the flask project stub\n app = \"\"\"#!/usr/bin/env python\\n\"\"\"\\\n \"\"\"from flask import Flask\\n\"\"\"\\\n \"\"\"app = Flask(__name__)\\n\\n\"\"\"\\\n \"\"\"@app.route(\\\"/\\\")\\n\"\"\"\\\n \"\"\"def hello():\\n\"\"\"\\\n \"\"\" return \\\"Hello from Flask...\\\"\\n\\n\"\"\"\\\n \"\"\"if __name__==\\\"__main__\\\":\\n\"\"\"\\\n \"\"\" app.run()\\n\\n\"\"\"\n with open(os.path.join(prj_dir, 'app.py'), 'w') as f:\n f.write(app)\n else:\n logging.error('Unable to find Python interpreter in virtualenv')\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef capture_stdout():\n stdout = sys.stdout\n try:\n capture_out = StringIO()\n sys.stdout = capture_out\n yield capture_out\n finally:\n sys.stdout = stdout", "response": "Context Manager that redirects STDOUT to a StringIO object and then restores the stdout."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef capture_stderr():\n stderr = sys.stderr\n try:\n capture_out = StringIO()\n sys.stderr = capture_out\n yield capture_out\n finally:\n sys.stderr = stderr", "response": "Context manager that redirects STDERR to a StringIO object and then restores it."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create(self, a, b, c):\n self.options = CreateOptions(\n str_to_cppstr(a), str_to_cppstr(b), str_to_cppstr(c))\n return True", "response": "Create an option object used to start the manager"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef addOptionBool(self, name, value):\n return self.options.AddOptionBool(str_to_cppstr(name), value)", "response": "Add a boolean option."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd an integer option.", "response": "def addOptionInt(self, name, value):\n \"\"\"\n .. _addOptionInt:\n\n Add an integer option.\n\n :param name: The name of the option.\n :type name: str\n :param value: The value of the option.\n :type value: boolean\n :return: The result of the operation.\n :rtype: bool\n\n :see: addOption_, addOptionBool_, addOptionString_\n\n \"\"\"\n return self.options.AddOptionInt(str_to_cppstr(name), value)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef addOptionString(self, name, value, append=False):\n return self.options.AddOptionString(\n str_to_cppstr(name), str_to_cppstr(value), append)", "response": ".. _addOptionString : Add a string option."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef addOption(self, name, value):\n if name not in PyOptionList:\n return False\n if PyOptionList[name]['type'] == \"String\":\n return self.addOptionString(name, value)\n elif PyOptionList[name]['type'] == \"Bool\":\n return self.addOptionBool(name, value)\n elif PyOptionList[name]['type'] == \"Int\":\n return self.addOptionInt(name, value)\n return False", "response": "Add an option to the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getOption(self, name):\n if name not in PyOptionList:\n return None\n if PyOptionList[name]['type'] == \"String\":\n return self.getOptionAsString(name)\n elif PyOptionList[name]['type'] == \"Bool\":\n return self.getOptionAsBool(name)\n elif PyOptionList[name]['type'] == \"Int\":\n return self.getOptionAsInt(name)\n return False", "response": "Get the value of an option."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef urls(cls):\n return urls.patterns('', urls.url(\n r'^{}(?:$|(?P[/:(.].*))'.format(cls.meta.name),\n cls.view,\n name='armet-api-{}'.format(cls.meta.name),\n kwargs={'resource': cls.meta.name}))", "response": "Builds the URL configuration for this resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dump(obj, fp, startindex=1, separator=DEFAULT, index_separator=DEFAULT):\n '''Dump an object in req format to the fp given.\n\n :param Mapping obj: The object to serialize. Must have a keys method.\n :param fp: A writable that can accept all the types given.\n :param separator: The separator between key and value. Defaults to u'|' or b'|', depending on the types.\n :param index_separator: The separator between key and index. Defaults to u'_' or b'_', depending on the types.\n '''\n\n if startindex < 0:\n raise ValueError('startindex must be non-negative, but was {}'.format(startindex))\n\n try:\n firstkey = next(iter(obj.keys()))\n except StopIteration:\n return\n\n if isinstance(firstkey, six.text_type):\n converter = six.u\n else:\n converter = six.b\n\n default_separator = converter('|')\n default_index_separator = converter('_')\n newline = converter('\\n')\n\n if separator is DEFAULT:\n separator = default_separator\n if index_separator is DEFAULT:\n index_separator = default_index_separator\n\n for key, value in six.iteritems(obj):\n if isinstance(value, (list, tuple, set)):\n for index, item in enumerate(value, start=startindex):\n fp.write(key)\n fp.write(index_separator)\n fp.write(converter(str(index)))\n fp.write(separator)\n fp.write(item)\n fp.write(newline)\n else:\n fp.write(key)\n fp.write(separator)\n fp.write(value)\n fp.write(newline)", "response": "Dump an object in req format to the fp given."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndumping an object in req format to a string.", "response": "def dumps(obj, startindex=1, separator=DEFAULT, index_separator=DEFAULT):\n '''Dump an object in req format to a string.\n\n :param Mapping obj: The object to serialize. Must have a keys method.\n :param separator: The separator between key and value. Defaults to u'|' or b'|', depending on the types.\n :param index_separator: The separator between key and index. Defaults to u'_' or b'_', depending on the types.\n '''\n\n try:\n firstkey = next(iter(obj.keys()))\n except StopIteration:\n return str()\n\n if isinstance(firstkey, six.text_type):\n io = StringIO()\n else:\n io = BytesIO()\n\n dump(\n obj=obj,\n fp=io,\n startindex=startindex,\n separator=separator,\n index_separator=index_separator,\n )\n return io.getvalue()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load(fp, separator=DEFAULT, index_separator=DEFAULT, cls=dict, list_cls=list):\n '''Load an object from the file pointer.\n\n :param fp: A readable filehandle.\n :param separator: The separator between key and value. Defaults to u'|' or b'|', depending on the types.\n :param index_separator: The separator between key and index. Defaults to u'_' or b'_', depending on the types.\n :param cls: A callable that returns a Mapping that is filled with pairs. The most common alternate option would be OrderedDict.\n :param list_cls: A callable that takes an iterable and returns a sequence.\n '''\n\n converter = None\n\n output = cls()\n arraykeys = set()\n\n for line in fp:\n if converter is None:\n if isinstance(line, six.text_type):\n converter = six.u\n else:\n converter = six.b\n default_separator = converter('|')\n default_index_separator = converter('_')\n newline = converter('\\n')\n\n if separator is DEFAULT:\n separator = default_separator\n if index_separator is DEFAULT:\n index_separator = default_index_separator\n\n key, value = line.strip().split(separator, 1)\n\n keyparts = key.split(index_separator)\n\n try:\n index = int(keyparts[-1])\n endwithint = True\n except ValueError:\n endwithint = False\n\n # We do everything in-place to ensure that we maintain order when using\n # an OrderedDict.\n if len(keyparts) > 1 and endwithint:\n # If this is an array key\n basekey = key.rsplit(index_separator, 1)[0]\n if basekey not in arraykeys:\n arraykeys.add(basekey)\n\n if basekey in output:\n # If key already exists as non-array, fix it\n if not isinstance(output[basekey], dict):\n output[basekey] = {-1: output[basekey]}\n else:\n output[basekey] = {}\n\n output[basekey][index] = value\n\n else:\n if key in output and isinstance(output[key], dict):\n output[key][-1] = value\n else:\n output[key] = value\n\n # Convert array keys\n for key in arraykeys:\n output[key] = list_cls(pair[1] for pair in sorted(six.iteritems(output[key])))\n\n return output", "response": "Load an object from the file pointer."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef loads(s, separator=DEFAULT, index_separator=DEFAULT, cls=dict, list_cls=list):\n '''Loads an object from a string.\n\n :param s: An object to parse\n :type s: bytes or str\n :param separator: The separator between key and value. Defaults to u'|' or b'|', depending on the types.\n :param index_separator: The separator between key and index. Defaults to u'_' or b'_', depending on the types.\n :param cls: A callable that returns a Mapping that is filled with pairs. The most common alternate option would be OrderedDict.\n :param list_cls: A callable that takes an iterable and returns a sequence.\n '''\n\n if isinstance(s, six.text_type):\n io = StringIO(s)\n else:\n io = BytesIO(s)\n\n return load(\n fp=io,\n separator=separator,\n index_separator=index_separator,\n cls=cls,\n list_cls=list_cls,\n )", "response": "Loads an object from a string."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef bump():\n\n parser = argparse.ArgumentParser(description=bump.__doc__)\n parser.add_argument('names', nargs='*', help=\"\"\"\n Only bump dependencies that match the name.\n Name can be a product group name defined in workspace.cfg.\n To bump to a specific version instead of latest, append version to name\n (i.e. requests==1.2.3 or 'requests>=1.2.3'). When > or < is used, be sure to quote.\"\"\")\n parser.add_argument('--add', '--require', action='store_true',\n help='Add the `names` to the requirements file if they don\\'t exist.')\n parser.add_argument('--file', help='Requirement file to bump. Defaults to requirements.txt and pinned.txt')\n parser.add_argument('--force', action='store_true',\n help='Force a bump even when certain bump requirements are not met.')\n parser.add_argument('-d', '--detail', '--dependencies', action='store_true',\n help='If available, show detailed changes. '\n 'For pinned.txt, pin parsed dependency requirements from changes')\n parser.add_argument('-n', '--dry-run', action='store_true', help='Perform a dry run without making changes')\n parser.add_argument('--debug', action='store_true', help='Turn on debug mode')\n\n args = parser.parse_args()\n targets = [args.file] if args.file else ['requirements.txt', 'pinned.txt']\n\n level = logging.DEBUG if args.debug else logging.INFO\n logging.basicConfig(level=level, format='[%(levelname)s] %(message)s')\n\n try:\n bumper = BumperDriver(targets, full_throttle=args.force, detail=args.detail, test_drive=args.dry_run)\n bumper.bump(args.names, required=args.add, show_detail=args.detail)\n except Exception as e:\n if args.debug:\n raise\n else:\n log.error(e)\n sys.exit(1)", "response": "Bump the dependencies of a single product group."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nbump the dependency requirements for the current set of targets.", "response": "def bump(self, filter_requirements, required=False, show_summary=True, show_detail=False, **kwargs):\n \"\"\"\n Bump dependency requirements using filter.\n\n :param list filter_requirements: List of dependency filter requirements.\n :param bool required: Require the filter_requirements to be met (by adding if possible).\n :param bool show_summary: Show summary for each bump made.\n :param bool show_detail: Show detail for each bump made if available.\n :return: Tuple with two elements: Dict of target file to bump message, List of :class:`Bump`\n :raise BumpAccident: for any bump errors\n \"\"\"\n found_targets = [target for target in self.targets if os.path.exists(target)]\n\n if not found_targets:\n raise BumpAccident('None of the requirement file(s) were found: %s' % ', '.join(self.targets))\n\n bump_reqs = RequirementsManager()\n\n if filter_requirements:\n requirements = parse_requirements(filter_requirements)\n bump_reqs.add(requirements, required=required)\n\n try:\n\n for target in found_targets:\n log.debug('Target: %s', target)\n\n target_bumpers = []\n target_bump_reqs = RequirementsManager(bump_reqs)\n loops = 0\n\n while True:\n\n # Insurance to ensure that we don't get stuck forever.\n loops += 1\n if loops > 5:\n log.debug('Too many transitive bump loops. Bailing out.')\n break\n\n if not target_bumpers:\n target_bumpers = [model(target, detail=self.detail, test_drive=self.test_drive)\n for model in self.bumper_models if model.likes(target)]\n\n if not target_bumpers:\n log.debug('No bumpers found that can bump %s. Defaulting to %s', target, self.default_model)\n target_bumpers = [self.default_model(target, detail=self.detail,\n test_drive=self.test_drive)]\n\n self.bumpers.extend(target_bumpers)\n\n new_target_bump_reqs = RequirementsManager()\n\n for bumper in target_bumpers:\n target_bumps = bumper.bump(target_bump_reqs)\n self.bumps.update(dict((b.name, b) for b in target_bumps))\n\n for bump in target_bumps:\n for new_req in bump.requirements:\n if not (bump_reqs.satisfied_by_checked(new_req) or\n target_bump_reqs.satisfied_by_checked(new_req)):\n new_target_bump_reqs.add(new_req)\n\n bump_reqs.matched_name |= target_bump_reqs.matched_name\n bump_reqs.checked.extend(target_bump_reqs.checked)\n\n if new_target_bump_reqs:\n bump_reqs.add(new_target_bump_reqs)\n\n target_bump_reqs = RequirementsManager(list(\n r for r in new_target_bump_reqs if r.project_name not in self.bumps))\n\n if not target_bump_reqs:\n break\n\n if not self.bumpers:\n raise BumpAccident('No bumpers found for %s' % ', '.join(found_targets))\n\n if bump_reqs and not bump_reqs.matched_name:\n raise BumpAccident('None of the provided filter names were found in %s' % ', '.join(found_targets))\n\n if self.bumps:\n for bump in self.bumps.values():\n bump_reqs.check(bump)\n\n for reqs in bump_reqs.required_requirements().values():\n for req in reqs:\n if not self.full_throttle:\n use_force = 'Use --force to ignore / force the bump' if req.required_by else ''\n raise BumpAccident('Requirement \"%s\" could not be met so '\n 'bump can not proceed. %s' % (req, use_force))\n\n if self.test_drive:\n log.info(\"Changes that would be made:\\n\")\n\n messages = {}\n\n for bumper in self.bumpers:\n if bumper.bumps:\n if not self.test_drive:\n bumper.update_requirements()\n\n if self.test_drive or show_summary:\n msg = bumper.bump_message(self.test_drive or show_detail)\n\n if self.test_drive:\n print(msg)\n else:\n rewords = [('Bump ', 'Bumped '), ('Pin ', 'Pinned '),\n ('Require ', 'Updated requirements: ')]\n for word, new_word in rewords:\n if msg.startswith(word):\n msg = msg.replace(word, new_word, 1)\n break\n log.info(msg)\n\n messages[bumper.target] = bumper.bump_message(True)\n\n return messages, self.bumps\n\n else:\n log.info('No need to bump. Everything is up to date!')\n return {}, []\n\n except Exception:\n if not self.test_drive and self.bumps:\n map(lambda b: b.reverse(), self.bumpers)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexpand the list of targets by looking for - r in targets.", "response": "def _expand_targets(self, targets, base_dir=None):\n \"\"\" Expand targets by looking for '-r' in targets. \"\"\"\n all_targets = []\n\n for target in targets:\n target_dirs = [p for p in [base_dir, os.path.dirname(target)] if p]\n target_dir = target_dirs and os.path.join(*target_dirs) or ''\n target = os.path.basename(target)\n target_path = os.path.join(target_dir, target)\n\n if os.path.exists(target_path):\n all_targets.append(target_path)\n\n with open(target_path) as fp:\n for line in fp:\n if line.startswith('-r '):\n _, new_target = line.split(' ', 1)\n all_targets.extend(self._expand_targets([new_target.strip()], base_dir=target_dir))\n\n return all_targets"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the Nginx config for the project", "response": "def get_nginx_config(self):\n \"\"\"\n Gets the Nginx config for the project\n\n \"\"\"\n if os.path.exists(self._nginx_config):\n return open(self._nginx_config, 'r').read()\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_directories(self):\n self.log.debug('Checking directories')\n if not os.path.exists(self._ve_dir):\n os.makedirs(self._ve_dir)\n if not os.path.exists(self._app_dir):\n os.makedirs(self._app_dir)\n if not os.path.exists(self._conf_dir):\n os.makedirs(self._conf_dir)\n if not os.path.exists(self._var_dir):\n os.makedirs(self._var_dir)\n if not os.path.exists(self._log_dir):\n os.makedirs(self._log_dir)\n if not os.path.exists(self._script_dir):\n os.makedirs(self._script_dir)\n\n # copy uswgi_params for nginx\n uwsgi_params = '/etc/nginx/uwsgi_params'\n if os.path.exists(uwsgi_params):\n shutil.copy(uwsgi_params, self._conf_dir)\n else:\n logging.warning('Unable to find Nginx uwsgi_params. You must manually copy this to {0}.'.format(self._conf_dir))\n\n # copy mime.types for nginx\n mime_types = '/etc/nginx/mime.types'\n if os.path.exists(mime_types):\n shutil.copy(mime_types, self._conf_dir)\n self._include_mimetypes = True\n else:\n logging.warn('Unable to find mime.types for Nginx. You must manually copy this to {0}.'.format(self._conf_dir))", "response": "Creates base directories for app virtualenv and nginx and returns a dict of base directories."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating the virtualenv for the project", "response": "def create_virtualenv(self):\n \"\"\"\n Creates the virtualenv for the project\n \n \"\"\"\n if check_command('virtualenv'):\n ve_dir = os.path.join(self._ve_dir, self._project_name)\n if os.path.exists(ve_dir):\n if self._force:\n logging.warn('Removing existing virtualenv')\n shutil.rmtree(ve_dir)\n else:\n logging.warn('Found existing virtualenv; not creating (use --force to overwrite)')\n return\n logging.info('Creating virtualenv')\n p = subprocess.Popen('virtualenv --no-site-packages {0} > /dev/null'.format(ve_dir), shell=True)\n os.waitpid(p.pid, 0)\n # install modules\n for m in self._modules:\n self.log.info('Installing module {0}'.format(m))\n p = subprocess.Popen('{0} install {1} > /dev/null'.format(os.path.join(self._ve_dir, \\\n self._project_name) + os.sep + 'bin' + os.sep + 'pip', m), shell=True)\n os.waitpid(p.pid, 0)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate the Nginx configuration for the project.", "response": "def create_nginx_config(self):\n \"\"\"\n Creates the Nginx configuration for the project\n\n \"\"\"\n cfg = '# nginx config for {0}\\n'.format(self._project_name)\n if not self._shared_hosting:\n # user\n if self._user:\n cfg += 'user {0};\\n'.format(self._user)\n # misc nginx config\n cfg += 'worker_processes 1;\\nerror_log {0}-errors.log;\\n\\\npid {1}_ nginx.pid;\\n\\n'.format(os.path.join(self._log_dir, \\\n self._project_name), os.path.join(self._var_dir, self._project_name))\n cfg += 'events {\\n\\tworker_connections 32;\\n}\\n\\n'\n # http section\n cfg += 'http {\\n'\n if self._include_mimetypes:\n cfg += '\\tinclude mime.types;\\n'\n cfg += '\\tdefault_type application/octet-stream;\\n'\n cfg += '\\tclient_max_body_size 1G;\\n'\n cfg += '\\tproxy_max_temp_file_size 0;\\n'\n cfg += '\\tproxy_buffering off;\\n'\n cfg += '\\taccess_log {0}-access.log;\\n'.format(os.path.join \\\n (self._log_dir, self._project_name))\n cfg += '\\tsendfile on;\\n'\n cfg += '\\tkeepalive_timeout 65;\\n'\n # server section\n cfg += '\\tserver {\\n'\n cfg += '\\t\\tlisten 0.0.0.0:{0};\\n'.format(self._port)\n if self._server_name:\n cfg += '\\t\\tserver_name {0};\\n'.format(self._server_name)\n # location section\n cfg += '\\t\\tlocation / {\\n'\n cfg += '\\t\\t\\tuwsgi_pass unix:///{0}.sock;\\n'.format(\\\n os.path.join(self._var_dir, self._project_name))\n cfg += '\\t\\t\\tinclude uwsgi_params;\\n'\n cfg += '\\t\\t}\\n\\n'\n # end location\n # error page templates\n cfg += '\\t\\terror_page 500 502 503 504 /50x.html;\\n'\n cfg += '\\t\\tlocation = /50x.html {\\n'\n cfg += '\\t\\t\\troot html;\\n'\n # end error page section\n cfg += '\\t\\t}\\n'\n # end server section\n cfg += '\\t}\\n'\n if not self._shared_hosting:\n # end http section\n cfg += '}\\n'\n\n # create conf\n f = open(self._nginx_config, 'w')\n f.write(cfg)\n f.close()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate the start and stop scripts to start and stop the application", "response": "def create_manage_scripts(self):\n \"\"\"\n Creates scripts to start and stop the application\n\n \"\"\"\n # create start script\n start = '# start script for {0}\\n\\n'.format(self._project_name)\n # start uwsgi\n start += 'echo \\'Starting uWSGI...\\'\\n'\n start += 'sh {0}.uwsgi\\n'.format(os.path.join(self._conf_dir, self._project_name))\n start += 'sleep 1\\n'\n # start nginx\n start += 'echo \\'Starting Nginx...\\'\\n'\n start += 'nginx -c {0}_nginx.conf\\n'.format(os.path.join(self._conf_dir, self._project_name))\n start += 'sleep 1\\n'\n start += 'echo \\'{0} started\\'\\n\\n'.format(self._project_name)\n\n # stop script\n stop = '# stop script for {0}\\n\\n'.format(self._project_name)\n # stop nginx\n stop += 'if [ -e {0}_nginx.pid ]; then nginx -c {1}_nginx.conf -s stop ; fi\\n'.format(os.path.join(self._var_dir, self._project_name), os.path.join(self._conf_dir, self._project_name))\n # stop uwsgi\n stop += 'if [ -e {0}_uwsgi.pid ]; then kill -9 `cat {0}_uwsgi.pid` ; rm {0}_uwsgi.pid 2>&1 > /dev/null ; fi\\n'.format(os.path.join(self._var_dir, self._project_name))\n stop += 'echo \\'{0} stopped\\'\\n'.format(self._project_name)\n\n # write scripts\n start_file = '{0}_start.sh'.format(os.path.join(self._script_dir, self._project_name))\n stop_file = '{0}_stop.sh'.format(os.path.join(self._script_dir, self._project_name))\n f = open(start_file, 'w')\n f.write(start)\n f.close()\n f = open(stop_file, 'w')\n f.write(stop)\n f.close()\n # make executable\n os.chmod(start_file, 0754)\n os.chmod(stop_file, 0754)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate the full project and virtual environment.", "response": "def create(self):\n \"\"\"\n Creates the full project\n\n \"\"\"\n # create virtualenv\n self.create_virtualenv()\n # create project\n self.create_project()\n # generate uwsgi script\n self.create_uwsgi_script()\n # generate nginx config\n self.create_nginx_config()\n # generate management scripts\n self.create_manage_scripts()\n logging.info('** Make sure to set proper permissions for the webserver user account on the var and log directories in the project root')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dasherize(value):\n value = value.strip()\n value = re.sub(r'([A-Z])', r'-\\1', value)\n value = re.sub(r'[-_\\s]+', r'-', value)\n value = re.sub(r'^-', r'', value)\n value = value.lower()\n return value", "response": "Dasherizes the passed value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nredirect to the canonical URI for this resource.", "response": "def redirect(cls, request, response):\n \"\"\"Redirect to the canonical URI for this resource.\"\"\"\n if cls.meta.legacy_redirect:\n if request.method in ('GET', 'HEAD',):\n # A SAFE request is allowed to redirect using a 301\n response.status = http.client.MOVED_PERMANENTLY\n\n else:\n # All other requests must use a 307\n response.status = http.client.TEMPORARY_REDIRECT\n\n else:\n # Modern redirects are allowed. Let's have some fun.\n # Hopefully you're client supports this.\n # The RFC explicitly discourages UserAgent sniffing.\n response.status = http.client.PERMANENT_REDIRECT\n\n # Terminate the connection.\n response.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nhandle the resource creation and delegation cycle of the request and response.", "response": "def view(cls, request, response):\n \"\"\"\n Entry-point of the request / response cycle; Handles resource creation\n and delegation.\n\n @param[in] requset\n The HTTP request object; containing accessors for information\n about the request.\n\n @param[in] response\n The HTTP response object; contains accessors for modifying\n the information that will be sent to the client.\n \"\"\"\n # Determine if we need to redirect.\n test = cls.meta.trailing_slash\n if test ^ request.path.endswith('/'):\n # Construct a new URL by removing or adding the trailing slash.\n path = request.path + '/' if test else request.path[:-1]\n response['Location'] = '{}://{}{}{}{}'.format(\n request.protocol.lower(),\n request.host,\n request.mount_point,\n path,\n '?' + request.query if request.query else '')\n\n # Redirect to the version with the correct trailing slash.\n return cls.redirect(request, response)\n\n try:\n # Instantiate the resource.\n obj = cls(request, response)\n\n # Bind the request and response objects to the constructed\n # resource.\n request.bind(obj)\n response.bind(obj)\n\n # Bind the request object to the resource.\n # This is used to facilitate the serializer and deserializer.\n obj._request = request\n\n # Initiate the dispatch cycle and handle its result on\n # synchronous requests.\n result = obj.dispatch(request, response)\n\n if not response.asynchronous:\n # There is several things that dispatch is allowed to return.\n if (isinstance(result, collections.Iterable) and\n not isinstance(result, six.string_types)):\n # Return the stream generator.\n return cls.stream(response, result)\n\n else:\n # Leave it up to the response to throw or write whatever\n # we got back.\n response.end(result)\n if response.body:\n # Return the body if there was any set.\n return response.body\n\n except http.exceptions.BaseHTTPException as e:\n # Something that we can handle and return properly happened.\n # Set response properties from the exception.\n response.status = e.status\n response.headers.update(e.headers)\n\n if e.content:\n # Write the exception body if present and close\n # the response.\n # TODO: Use the plain-text encoder.\n response.send(e.content, serialize=True, format='json')\n\n # Terminate the connection and return the body.\n response.close()\n if response.body:\n return response.body\n\n except Exception:\n # Something unexpected happened.\n # Log error message to the logger.\n logger.exception('Internal server error')\n\n # Write a debug message for the client.\n if not response.streaming and not response.closed:\n response.status = http.client.INTERNAL_SERVER_ERROR\n response.headers.clear()\n response.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses out parameters and separates them out of the path.", "response": "def parse(cls, path):\n \"\"\"Parses out parameters and separates them out of the path.\n\n This uses one of the many defined patterns on the options class. But,\n it defaults to a no-op if there are no defined patterns.\n \"\"\"\n # Iterate through the available patterns.\n for resource, pattern in cls.meta.patterns:\n # Attempt to match the path.\n match = re.match(pattern, path)\n if match is not None:\n # Found something.\n return resource, match.groupdict(), match.string[match.end():]\n\n # No patterns at all; return unsuccessful.\n return None if not cls.meta.patterns else False"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntraverses down the path and determines the accessed resource.", "response": "def traverse(cls, request, params=None):\n \"\"\"Traverses down the path and determines the accessed resource.\n\n This makes use of the patterns array to implement simple traversal.\n This defaults to a no-op if there are no defined patterns.\n \"\"\"\n # Attempt to parse the path using a pattern.\n result = cls.parse(request.path)\n if result is None:\n # No parsing was requested; no-op.\n return cls, {}\n\n elif not result:\n # Parsing failed; raise 404.\n raise http.exceptions.NotFound()\n\n # Partition out the result.\n resource, data, rest = result\n\n if params:\n # Append params to data.\n data.update(params)\n\n if resource is None:\n # No traversal; return parameters.\n return cls, data\n\n # Modify the path appropriately.\n if data.get('path') is not None:\n request.path = data.pop('path')\n\n elif rest is not None:\n request.path = rest\n\n # Send us through traversal again.\n result = resource.traverse(request, params=data)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstream the response to the client.", "response": "def stream(cls, response, sequence):\n \"\"\"\n Helper method used in conjunction with the view handler to\n stream responses to the client.\n \"\"\"\n # Construct the iterator and run the sequence once in order\n # to capture any headers and status codes set.\n iterator = iter(sequence)\n data = {'chunk': next(iterator)}\n response.streaming = True\n\n def streamer():\n # Iterate through the iterator and yield its content\n while True:\n if response.asynchronous:\n # Yield our current chunk.\n yield data['chunk']\n\n else:\n # Write the chunk to the response\n response.send(data['chunk'])\n\n # Yield its body\n yield response.body\n\n # Unset the body.\n response.body = None\n\n try:\n # Get the next chunk.\n data['chunk'] = next(iterator)\n\n except StopIteration:\n # Get out of the loop.\n break\n\n if not response.asynchronous:\n # Close the response.\n response.close()\n\n # Return the streaming function.\n return streamer()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nserialize the data using a determined serializer.", "response": "def serialize(self, data, response=None, request=None, format=None):\n \"\"\"Serializes the data using a determined serializer.\n\n @param[in] data\n The data to be serialized.\n\n @param[in] response\n The response object to serialize the data to.\n If this method is invoked as an instance method, the response\n object can be omitted and it will be taken from the instance.\n\n @param[in] request\n The request object to pull information from; normally used to\n determine the serialization format (when `format` is not provided).\n May be used by some serializers as well to pull additional headers.\n If this method is invoked as an instance method, the request\n object can be omitted and it will be taken from the instance.\n\n @param[in] format\n A specific format to serialize in; if provided, no detection is\n done. If not provided, the accept header (as well as the URL\n extension) is looked at to determine an appropriate serializer.\n\n @returns\n A tuple of the serialized text and an instance of the\n serializer used.\n \"\"\"\n if isinstance(self, Resource):\n if not request:\n # Ensure we have a response object.\n request = self._request\n\n Serializer = None\n if format:\n # An explicit format was given; do not attempt to auto-detect\n # a serializer.\n Serializer = self.meta.serializers[format]\n\n if not Serializer:\n # Determine an appropriate serializer to use by\n # introspecting the request object and looking at the `Accept`\n # header.\n media_ranges = (request.get('Accept') or '*/*').strip()\n if not media_ranges:\n # Default the media ranges to */*\n media_ranges = '*/*'\n\n if media_ranges != '*/*':\n # Parse the media ranges and determine the serializer\n # that is the closest match.\n media_types = six.iterkeys(self._serializer_map)\n media_type = mimeparse.best_match(media_types, media_ranges)\n if media_type:\n format = self._serializer_map[media_type]\n Serializer = self.meta.serializers[format]\n\n else:\n # Client indicated no preference; use the default.\n default = self.meta.default_serializer\n Serializer = self.meta.serializers[default]\n\n if Serializer:\n try:\n # Attempt to serialize the data using the determined\n # serializer.\n serializer = Serializer(request, response)\n return serializer.serialize(data), serializer\n\n except ValueError:\n # Failed to serialize the data.\n pass\n\n # Either failed to determine a serializer or failed to serialize\n # the data; construct a list of available and valid encoders.\n available = {}\n for name in self.meta.allowed_serializers:\n Serializer = self.meta.serializers[name]\n instance = Serializer(request, None)\n if instance.can_serialize(data):\n available[name] = Serializer.media_types[0]\n\n # Raise a Not Acceptable exception.\n raise http.exceptions.NotAcceptable(available)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprocessing a cross - domain request.", "response": "def _process_cross_domain_request(cls, request, response):\n \"\"\"Facilitate Cross-Origin Requests (CORs).\n \"\"\"\n\n # Step 1\n # Check for Origin header.\n origin = request.get('Origin')\n if not origin:\n return\n\n # Step 2\n # Check if the origin is in the list of allowed origins.\n if not (origin in cls.meta.http_allowed_origins or\n '*' == cls.meta.http_allowed_origins):\n return\n\n # Step 3\n # Try to parse the Request-Method header if it exists.\n method = request.get('Access-Control-Request-Method')\n if method and method not in cls.meta.http_allowed_methods:\n return\n\n # Step 4\n # Try to parse the Request-Header header if it exists.\n headers = request.get('Access-Control-Request-Headers', ())\n if headers:\n headers = [h.strip() for h in headers.split(',')]\n\n # Step 5\n # Check if the headers are allowed on this resource.\n allowed_headers = [h.lower() for h in cls.meta.http_allowed_headers]\n if any(h.lower() not in allowed_headers for h in headers):\n return\n\n # Step 6\n # Always add the origin.\n response['Access-Control-Allow-Origin'] = origin\n\n # TODO: Check if we can provide credentials.\n response['Access-Control-Allow-Credentials'] = 'true'\n\n # Step 7\n # TODO: Optionally add Max-Age header.\n\n # Step 8\n # Add the allowed methods.\n allowed_methods = ', '.join(cls.meta.http_allowed_methods)\n response['Access-Control-Allow-Methods'] = allowed_methods\n\n # Step 9\n # Add any allowed headers.\n allowed_headers = ', '.join(cls.meta.http_allowed_headers)\n if allowed_headers:\n response['Access-Control-Allow-Headers'] = allowed_headers\n\n # Step 10\n # Add any exposed headers.\n exposed_headers = ', '.join(cls.meta.http_exposed_headers)\n if exposed_headers:\n response['Access-Control-Expose-Headers'] = exposed_headers"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef require_authentication(self, request):\n request.user = user = None\n\n if request.method == 'OPTIONS':\n # Authentication should not be checked on an OPTIONS request.\n return\n\n for auth in self.meta.authentication:\n user = auth.authenticate(request)\n if user is False:\n # Authentication protocol failed to authenticate;\n # pass the baton.\n continue\n\n if user is None and not auth.allow_anonymous:\n # Authentication protocol determined the user is\n # unauthenticated.\n auth.unauthenticated()\n\n # Authentication protocol determined the user is indeed\n # authenticated (or not); Store the user for later reference.\n request.user = user\n return\n\n if not user and not auth.allow_anonymous:\n # No authenticated user found and protocol doesn't allow\n # anonymous users.\n auth.unauthenticated()", "response": "Ensure we are authenticated."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef require_accessibility(self, user, method):\n if method == 'OPTIONS':\n # Authorization should not be checked on an OPTIONS request.\n return\n\n authz = self.meta.authorization\n if not authz.is_accessible(user, method, self):\n # User is not authorized; raise an appropriate message.\n authz.unaccessible()", "response": "Ensure we are allowed to access this resource."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nensuring that we re allowed to use this HTTP method.", "response": "def require_http_allowed_method(cls, request):\n \"\"\"Ensure that we're allowed to use this HTTP method.\"\"\"\n allowed = cls.meta.http_allowed_methods\n if request.method not in allowed:\n # The specified method is not allowed for the resource\n # identified by the request URI.\n # RFC 2616 \u00a7 10.4.6 \u2014 405 Method Not Allowed\n raise http.exceptions.MethodNotAllowed(allowed)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess every request. Directs control flow to the appropriate HTTP method.", "response": "def route(self, request, response):\n \"\"\"Processes every request.\n\n Directs control flow to the appropriate HTTP/1.1 method.\n \"\"\"\n # Ensure that we're allowed to use this HTTP method.\n self.require_http_allowed_method(request)\n\n # Retrieve the function corresponding to this HTTP method.\n function = getattr(self, request.method.lower(), None)\n if function is None:\n # Server is not capable of supporting it.\n raise http.exceptions.NotImplemented()\n\n # Delegate to the determined function to process the request.\n return function(request, response)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef options(self, request, response):\n # Gather a list available HTTP/1.1 methods for this URI.\n response['Allowed'] = ', '.join(self.meta.http_allowed_methods)\n\n # All CORS handling is done for every HTTP/1.1 method.\n # No more handling is neccesary; set the response to 200 and return.\n response.status = http.client.OK", "response": "Process an OPTIONS request."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwraps the decorated function in a lightweight resource.", "response": "def resource(**kwargs):\n \"\"\"Wraps the decorated function in a lightweight resource.\"\"\"\n def inner(function):\n name = kwargs.pop('name', None)\n if name is None:\n name = utils.dasherize(function.__name__)\n\n methods = kwargs.pop('methods', None)\n if isinstance(methods, six.string_types):\n # Tuple-ify the method if we got just a string.\n methods = methods,\n\n # Construct a handler.\n handler = (function, methods)\n\n if name not in _resources:\n # Initiate the handlers list.\n _handlers[name] = []\n\n # Construct a light-weight resource using the passed kwargs\n # as the arguments for the meta.\n from armet import resources\n kwargs['name'] = name\n\n class LightweightResource(resources.Resource):\n Meta = type(str('Meta'), (), kwargs)\n\n def route(self, request, response):\n for handler, methods in _handlers[name]:\n if methods is None or request.method in methods:\n return handler(request, response)\n\n resources.Resource.route(self)\n\n # Construct and add this resource.\n _resources[name] = LightweightResource\n\n # Add this to the handlers.\n _handlers[name].append(handler)\n\n # Return the resource.\n return _resources[name]\n\n # Return the inner method.\n return inner"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef threewise(iterable):\n a, b, c = itertools.tee(iterable,3)\n def prepend(val, l):\n yield val\n for i in l: yield i\n def postpend(val, l):\n for i in l: yield i\n yield val\n next(c,None)\n\n for _xa, _xb, _xc in six.moves.zip(prepend(None,a), b, postpend(None,c)):\n yield (_xa, _xb, _xc)", "response": "A generator that yields the sequence of unique identifiers in the given iterable."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lines2less(lines):\n lines = iter(lines) #cast list to iterator\n\n #print output to stdout if small, otherwise to less\n has_term = True\n terminal_cols = 100\n try:\n terminal_cols = terminal_size()\n except:\n #getting terminal info failed -- maybe it's a\n #weird situation like running through cron\n has_term = False\n\n MAX_CAT_ROWS = 20 #if there are <= this many rows then print to screen\n\n first_rows = list(itertools.islice(lines,0,MAX_CAT_ROWS))\n wide = any(len(l) > terminal_cols for l in first_rows)\n\n use_less = False\n if has_term and (wide or len(first_rows) == MAX_CAT_ROWS):\n use_less = True\n\n lines = itertools.chain(first_rows, lines)\n lines = six.moves.map(lambda x: x + '\\n', lines)\n\n if use_less:\n lesspager(lines)\n else:\n for l in lines:\n sys.stdout.write(l)", "response": "prints the lines to stdout if the output is small + narrow\n otherwise print the lines to screen if the output is small + narrow\n otherwise print the lines to lesspager"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lesspager(lines):\n cmd = \"less -S\"\n if sys.version_info[0] >= 3:\n \"\"\"Page through text by feeding it to another program.\"\"\"\n import subprocess\n proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)\n try:\n with io.TextIOWrapper(proc.stdin, errors='backslashreplace') as pipe:\n try:\n for l in lines:\n pipe.write(l)\n except KeyboardInterrupt:\n # We've hereby abandoned whatever text hasn't been written,\n # but the pager is still in control of the terminal.\n pass\n except OSError:\n pass # Ignore broken pipes caused by quitting the pager program.\n while True:\n try:\n proc.wait()\n break\n except KeyboardInterrupt:\n # Ignore ctl-c like the pager itself does. Otherwise the pager is\n # left running and the terminal is in raw mode and unusable.\n pass\n\n else:\n proc = os.popen(cmd, 'w')\n try:\n for l in lines:\n proc.write(l)\n except IOError:\n proc.close()\n sys.exit()", "response": "Page through text by streaming writes to a less process"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef argmax(l,f=None):\n if f:\n l = [f(i) for i in l]\n return max(enumerate(l), key=lambda x:x[1])[0]", "response": "Returns the maximum value of a list of items."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nrenders to cookie strings.", "response": "def render_to_string(self):\n \"\"\"Render to cookie strings.\n \"\"\"\n values = ''\n for key, value in self.items():\n values += '{}={};'.format(key, value)\n return values"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating self with cookie_string.", "response": "def from_cookie_string(self, cookie_string):\n \"\"\"update self with cookie_string.\n \"\"\"\n for key_value in cookie_string.split(';'):\n if '=' in key_value:\n key, value = key_value.split('=', 1)\n else:\n key = key_value\n strip_key = key.strip()\n if strip_key and strip_key.lower() not in COOKIE_ATTRIBUTE_NAMES:\n self[strip_key] = value.strip()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _add_method(self, effect, verb, resource, conditions):\n if verb != '*' and not hasattr(HttpVerb, verb):\n raise NameError('Invalid HTTP verb ' + verb +\n '. Allowed verbs in HttpVerb class')\n resource_pattern = re.compile(self.path_regex)\n if not resource_pattern.match(resource):\n raise NameError('Invalid resource path: ' + resource +\n '. Path should match ' + self.path_regex)\n\n if resource[:1] == '/':\n resource = resource[1:]\n\n resource_arn = ('arn:aws:execute-api:' +\n self.region + ':' +\n self.aws_account_id + ':' +\n self.rest_api_id + '/' +\n self.stage + '/' +\n verb + '/' +\n resource)\n\n if effect.lower() == 'allow':\n self.allowMethods.append({\n 'resource_arn': resource_arn,\n 'conditions': conditions\n })\n elif effect.lower() == 'deny':\n self.denyMethods.append({\n 'resource_arn': resource_arn,\n 'conditions': conditions\n })", "response": "Adds a method to the internal list of allowed or denied methods."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_effect_statement(self, effect, methods):\n statements = []\n\n if len(methods) > 0:\n statement = self._get_empty_statement(effect)\n\n for method in methods:\n if (method['conditions'] is None or\n len(method['conditions']) == 0):\n statement['Resource'].append(method['resource_arn'])\n else:\n cond_statement = self._get_empty_statement(effect)\n cond_statement['Resource'].append(method['resource_arn'])\n cond_statement['Condition'] = method['conditions']\n statements.append(cond_statement)\n statements.append(statement)\n\n return statements", "response": "This function loops over an array of objects containing\n a resourceArn and conditions statement and generates the array of statements that are used to create the policy."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd an API Gateway method with a list of conditions to the allowed methods list.", "response": "def allow_method_with_conditions(self, verb, resource, conditions):\n \"\"\"\n Adds an API Gateway method (Http verb + Resource path) to the\n list of allowed methods and includes a condition for the policy\n statement. More on AWS policy conditions here:\n http://docs.aws.amazon.com/IAM/latest/UserGuide/\n reference_policies_elements.html#Condition\n \"\"\"\n self._add_method('Allow', verb, resource, conditions)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef deny_method_with_conditions(self, verb, resource, conditions):\n self._add_method('Deny', verb, resource, conditions)", "response": "Adds an API Gateway method to the list of denied methods and includes a condition for the policy\n statement."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef build(self):\n if ((self.allowMethods is None or len(self.allowMethods) == 0) and\n (self.denyMethods is None or len(self.denyMethods) == 0)):\n raise NameError('No statements defined for the policy')\n\n policy = {\n 'principalId': self.principal_id,\n 'policyDocument': {\n 'Version': self.version,\n 'Statement': []\n }\n }\n\n policy['policyDocument']['Statement'].extend(\n self._get_effect_statement('Allow', self.allowMethods))\n policy['policyDocument']['Statement'].extend(\n self._get_effect_statement('Deny', self.denyMethods))\n\n return policy", "response": "Builds the policy document based on the internal lists of allow and deny conditions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deref(self, data):\n\n # We have to make a deepcopy here to create a proper JSON\n # compatible object, otherwise `json.dumps` fails when it\n # hits jsonref.JsonRef objects.\n deref = copy.deepcopy(jsonref.JsonRef.replace_refs(data))\n\n # Write out JSON version because we might want this.\n self.write_template(deref, filename='swagger.json')\n\n return deref", "response": "Convert to JSON version and write out the JSON version."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_pre_requirements(pre_requirements):\n pre_requirements = set(pre_requirements or [])\n pre_requirements.add('virtualenv')\n\n for requirement in pre_requirements:\n if not which(requirement):\n print_error('Requirement {0!r} is not found in system'.\n format(requirement))\n return False\n\n return True", "response": "Check all necessary system requirements to exist."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef config_to_args(config):\n result = []\n\n for key, value in iteritems(config):\n if value is False:\n continue\n\n key = '--{0}'.format(key.replace('_', '-'))\n\n if isinstance(value, (list, set, tuple)):\n for item in value:\n result.extend((key, smart_str(item)))\n elif value is not True:\n result.extend((key, smart_str(value)))\n else:\n result.append(key)\n\n return tuple(result)", "response": "Convert config dict to list of arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates virtual environment. :param env: Virtual environment name. :param args: Pass given arguments to ``virtualenv`` script. :param recerate: Recreate virtual environment? By default: False :param ignore_activated: Ignore already activated virtual environment and create new one. By default: False :param quiet: Do not output messages into terminal. By default: False", "response": "def create_env(env, args, recreate=False, ignore_activated=False, quiet=False):\n \"\"\"Create virtual environment.\n\n :param env: Virtual environment name.\n :param args: Pass given arguments to ``virtualenv`` script.\n :param recerate: Recreate virtual environment? By default: False\n :param ignore_activated:\n Ignore already activated virtual environment and create new one. By\n default: False\n :param quiet: Do not output messages into terminal. By default: False\n \"\"\"\n cmd = None\n result = True\n\n inside_env = hasattr(sys, 'real_prefix') or os.environ.get('VIRTUAL_ENV')\n env_exists = os.path.isdir(env)\n\n if not quiet:\n print_message('== Step 1. Create virtual environment ==')\n\n if (\n recreate or (not inside_env and not env_exists)\n ) or (\n ignore_activated and not env_exists\n ):\n cmd = ('virtualenv', ) + args + (env, )\n\n if not cmd and not quiet:\n if inside_env:\n message = 'Working inside of virtual environment, done...'\n else:\n message = 'Virtual environment {0!r} already created, done...'\n print_message(message.format(env))\n\n if cmd:\n with disable_error_handler():\n result = not run_cmd(cmd, echo=not quiet)\n\n if not quiet:\n print_message()\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef install(env, requirements, args, ignore_activated=False,\n install_dev_requirements=False, quiet=False):\n \"\"\"Install library or project into virtual environment.\n\n :param env: Use given virtual environment name.\n :param requirements: Use given requirements file for pip.\n :param args: Pass given arguments to pip script.\n :param ignore_activated:\n Do not run pip inside already activated virtual environment. By\n default: False\n :param install_dev_requirements:\n When enabled install prefixed or suffixed dev requirements after\n original installation process completed. By default: False\n :param quiet: Do not output message to terminal. By default: False\n \"\"\"\n if os.path.isfile(requirements):\n args += ('-r', requirements)\n label = 'project'\n else:\n args += ('-U', '-e', '.')\n label = 'library'\n\n # Attempt to install development requirements\n if install_dev_requirements:\n dev_requirements = None\n dirname = os.path.dirname(requirements)\n basename, ext = os.path.splitext(os.path.basename(requirements))\n\n # Possible dev requirements files:\n #\n # * -dev.\n # * dev-.\n # * _dev.\n # * dev_.\n # * dev.\n # * dev.\n #\n # Where is basename of given requirements file to use\n # and is its extension.\n for delimiter in ('-', '_', ''):\n filename = os.path.join(\n dirname, ''.join((basename, delimiter, 'dev', ext))\n )\n if os.path.isfile(filename):\n dev_requirements = filename\n break\n\n filename = os.path.join(\n dirname, ''.join(('dev', delimiter, basename, ext))\n )\n if os.path.isfile(filename):\n dev_requirements = filename\n break\n\n # If at least one dev requirements file found, install dev requirements\n if dev_requirements:\n args += ('-r', dev_requirements)\n\n if not quiet:\n print_message('== Step 2. Install {0} =='.format(label))\n\n result = not pip_cmd(env,\n ('install', ) + args,\n ignore_activated,\n echo=not quiet)\n\n if not quiet:\n print_message()\n\n return result", "response": "Install library or project into virtual environment."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef iteritems(data, **kwargs):\n return iter(data.items(**kwargs)) if IS_PY3 else data.iteritems(**kwargs)", "response": "Iterate over dict items."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\niterating over dict keys.", "response": "def iterkeys(data, **kwargs):\n \"\"\"Iterate over dict keys.\"\"\"\n return iter(data.keys(**kwargs)) if IS_PY3 else data.iterkeys(**kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses command line arguments and return a parser instance.", "response": "def parse_args(args):\n \"\"\"\n Parse args from command line by creating argument parser instance and\n process it.\n\n :param args: Command line arguments list.\n \"\"\"\n from argparse import ArgumentParser\n\n description = ('Bootstrap Python projects and libraries with virtualenv '\n 'and pip.')\n parser = ArgumentParser(description=description)\n parser.add_argument('--version', action='version', version=__version__)\n\n parser.add_argument(\n '-c', '--config', default=DEFAULT_CONFIG,\n help='Path to config file. By default: {0}'.format(DEFAULT_CONFIG)\n )\n parser.add_argument(\n '-p', '--pre-requirements', default=[], nargs='+',\n help='List of pre-requirements to check, separated by space.'\n )\n parser.add_argument(\n '-e', '--env',\n help='Virtual environment name. By default: {0}'.\n format(CONFIG[__script__]['env'])\n )\n parser.add_argument(\n '-r', '--requirements',\n help='Path to requirements file. By default: {0}'.\n format(CONFIG[__script__]['requirements'])\n )\n parser.add_argument(\n '-d', '--install-dev-requirements', action='store_true', default=None,\n help='Install prefixed or suffixed \"dev\" requirements after '\n 'installation of original requirements file or library completed '\n 'without errors.'\n )\n parser.add_argument(\n '-C', '--hook', help='Execute this hook after bootstrap process.'\n )\n parser.add_argument(\n '--ignore-activated', action='store_true', default=None,\n help='Ignore pre-activated virtualenv, like on Travis CI.'\n )\n parser.add_argument(\n '--recreate', action='store_true', default=None,\n help='Recreate virtualenv on every run.'\n )\n parser.add_argument(\n '-q', '--quiet', action='store_true', default=None,\n help='Minimize output, show only error messages.'\n )\n\n return parser.parse_args(args)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pip_cmd(env, cmd, ignore_activated=False, **kwargs):\n cmd = tuple(cmd)\n dirname = safe_path(env)\n\n if not ignore_activated:\n activated_env = os.environ.get('VIRTUAL_ENV')\n\n if hasattr(sys, 'real_prefix'):\n dirname = sys.prefix\n elif activated_env:\n dirname = activated_env\n\n pip_path = os.path.join(dirname, 'Scripts' if IS_WINDOWS else 'bin', 'pip')\n\n if kwargs.pop('return_path', False):\n return pip_path\n\n if not os.path.isfile(pip_path):\n raise OSError('No pip found at {0!r}'.format(pip_path))\n\n # Disable pip version check in tests\n if BOOTSTRAPPER_TEST_KEY in os.environ and cmd[0] == 'install':\n cmd = list(cmd)\n cmd.insert(1, '--disable-pip-version-check')\n cmd = tuple(cmd)\n\n with disable_error_handler():\n return run_cmd((pip_path, ) + cmd, **kwargs)", "response": "r Run pip command in given virtual environment."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef prepare_args(config, bootstrap):\n config = copy.deepcopy(config)\n environ = dict(copy.deepcopy(os.environ))\n\n data = {'env': bootstrap['env'],\n 'pip': pip_cmd(bootstrap['env'], '', return_path=True),\n 'requirements': bootstrap['requirements']}\n environ.update(data)\n\n if isinstance(config, string_types):\n return config.format(**environ)\n\n for key, value in iteritems(config):\n if not isinstance(value, string_types):\n continue\n config[key] = value.format(**environ)\n\n return config_to_args(config)", "response": "Convert config dict to command line args line."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprinting error message to stderr using ANSI - colors.", "response": "def print_error(message, wrap=True):\n \"\"\"Print error message to stderr, using ANSI-colors.\n\n :param message: Message to print\n :param wrap:\n Wrap message into ``ERROR: . Exit...`` template. By default:\n True\n \"\"\"\n if wrap:\n message = 'ERROR: {0}. Exit...'.format(message.rstrip('.'))\n\n colorizer = (_color_wrap(colorama.Fore.RED)\n if colorama\n else lambda message: message)\n return print(colorizer(message), file=sys.stderr)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef print_message(message=None):\n kwargs = {'stdout': sys.stdout,\n 'stderr': sys.stderr,\n 'shell': True}\n return subprocess.call('echo \"{0}\"'.format(message or ''), **kwargs)", "response": "Print message via subprocess. call function."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread and parse a config file.", "response": "def read_config(filename, args):\n \"\"\"\n Read and parse configuration file. By default, ``filename`` is relative\n path to current work directory.\n\n If no config file found, default ``CONFIG`` would be used.\n\n :param filename: Read config from given filename.\n :param args: Parsed command line arguments.\n \"\"\"\n # Initial vars\n config = defaultdict(dict)\n splitter = operator.methodcaller('split', ' ')\n\n converters = {\n __script__: {\n 'env': safe_path,\n 'pre_requirements': splitter,\n },\n 'pip': {\n 'allow_external': splitter,\n 'allow_unverified': splitter,\n }\n }\n default = copy.deepcopy(CONFIG)\n sections = set(iterkeys(default))\n\n # Append download-cache for old pip versions\n if int(getattr(pip, '__version__', '1.x').split('.')[0]) < 6:\n default['pip']['download_cache'] = safe_path(os.path.expanduser(\n os.path.join('~', '.{0}'.format(__script__), 'pip-cache')\n ))\n\n # Expand user and environ vars in config filename\n is_default = filename == DEFAULT_CONFIG\n filename = os.path.expandvars(os.path.expanduser(filename))\n\n # Read config if it exists on disk\n if not is_default and not os.path.isfile(filename):\n print_error('Config file does not exist at {0!r}'.format(filename))\n return None\n\n parser = ConfigParser()\n\n try:\n parser.read(filename)\n except ConfigParserError:\n print_error('Cannot parse config file at {0!r}'.format(filename))\n return None\n\n # Apply config for each possible section\n for section in sections:\n if not parser.has_section(section):\n continue\n\n items = parser.items(section)\n\n # Make auto convert here for integers and boolean values\n for key, value in items:\n try:\n value = int(value)\n except (TypeError, ValueError):\n try:\n value = bool(strtobool(value))\n except ValueError:\n pass\n\n if section in converters and key in converters[section]:\n value = converters[section][key](value)\n\n config[section][key] = value\n\n # Update config with default values if necessary\n for section, data in iteritems(default):\n if section not in config:\n config[section] = data\n else:\n for key, value in iteritems(data):\n config[section].setdefault(key, value)\n\n # Update bootstrap config from parsed args\n keys = set((\n 'env', 'hook', 'install_dev_requirements', 'ignore_activated',\n 'pre_requirements', 'quiet', 'recreate', 'requirements'\n ))\n\n for key in keys:\n value = getattr(args, key)\n config[__script__].setdefault(key, value)\n\n if key == 'pre_requirements' and not value:\n continue\n\n if value is not None:\n config[__script__][key] = value\n\n return config"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_cmd(cmd, echo=False, fail_silently=False, **kwargs):\n out, err = None, None\n\n if echo:\n cmd_str = cmd if isinstance(cmd, string_types) else ' '.join(cmd)\n kwargs['stdout'], kwargs['stderr'] = sys.stdout, sys.stderr\n print_message('$ {0}'.format(cmd_str))\n else:\n out, err = get_temp_streams()\n kwargs['stdout'], kwargs['stderr'] = out, err\n\n try:\n retcode = subprocess.call(cmd, **kwargs)\n except subprocess.CalledProcessError as err:\n if fail_silently:\n return False\n print_error(str(err) if IS_PY3 else unicode(err)) # noqa\n finally:\n if out:\n out.close()\n if err:\n err.close()\n\n if retcode and echo and not fail_silently:\n print_error('Command {0!r} returned non-zero exit status {1}'.\n format(cmd_str, retcode))\n\n return retcode", "response": "r Run given command with subprocess. call."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun post - bootstrap hook if any.", "response": "def run_hook(hook, config, quiet=False):\n \"\"\"Run post-bootstrap hook if any.\n\n :param hook: Hook to run.\n :param config: Configuration dict.\n :param quiet: Do not output messages to STDOUT/STDERR. By default: False\n \"\"\"\n if not hook:\n return True\n\n if not quiet:\n print_message('== Step 3. Run post-bootstrap hook ==')\n\n result = not run_cmd(prepare_args(hook, config),\n echo=not quiet,\n fail_silently=True,\n shell=True)\n\n if not quiet:\n print_message()\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_traceback(err):\n # Store logs to ~/.bootstrapper directory\n dirname = safe_path(os.path.expanduser(\n os.path.join('~', '.{0}'.format(__script__))\n ))\n\n # But ensure that directory exists\n if not os.path.isdir(dirname):\n os.mkdir(dirname)\n\n # Now we ready to put traceback to log file\n filename = os.path.join(dirname, '{0}.log'.format(__script__))\n\n with open(filename, 'a+') as handler:\n traceback.print_exc(file=handler)\n\n # And show colorized message\n message = ('User aborted workflow'\n if isinstance(err, KeyboardInterrupt)\n else 'Unexpected error catched')\n print_error(message)\n print_error('Full log stored to {0}'.format(filename), False)\n\n return True", "response": "Save error traceback to bootstrapper log file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef smart_str(value, encoding='utf-8', errors='strict'):\n if not IS_PY3 and isinstance(value, unicode): # noqa\n return value.encode(encoding, errors)\n return str(value)", "response": "Convert Python object to string."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncopy srcfile in destdir with extension basename + get_extension", "response": "def copy_w_ext(srcfile, destdir, basename):\n \"\"\" Copy `srcfile` in `destdir` with name `basename + get_extension(srcfile)`.\n Add pluses to the destination path basename if a file with the same name already\n exists in `destdir`.\n\n Parameters\n ----------\n srcfile: str\n\n destdir: str\n\n basename:str\n\n Returns\n -------\n dstpath: str\n \"\"\"\n\n ext = get_extension(op.basename(srcfile))\n\n dstpath = op.join(destdir, basename + ext)\n\n return copy_w_plus(srcfile, dstpath)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef copy_w_plus(src, dst):\n dst_ext = get_extension(dst)\n dst_pre = remove_ext (dst)\n\n while op.exists(dst_pre + dst_ext):\n dst_pre += '+'\n\n shutil.copy(src, dst_pre + dst_ext)\n\n return dst_pre + dst_ext", "response": "Copy file from src path to dst path."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_abspath(folderpath):\n if not op.exists(folderpath):\n raise FolderNotFound(folderpath)\n\n return op.abspath(folderpath)", "response": "Returns the absolute path of folderpath."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the extension of the file.", "response": "def get_extension(filepath, check_if_exists=False, allowed_exts=ALLOWED_EXTS):\n \"\"\"Return the extension of fpath.\n\n Parameters\n ----------\n fpath: string\n File name or path\n\n check_if_exists: bool\n\n allowed_exts: dict\n Dictionary of strings, where the key if the last part of a complex ('.' separated) extension\n and the value is the previous part.\n For example: for the '.nii.gz' extension I would have a dict as {'.gz': ['.nii',]}\n\n Returns\n -------\n str\n The extension of the file name or path\n \"\"\"\n if check_if_exists:\n if not op.exists(filepath):\n raise IOError('File not found: ' + filepath)\n\n rest, ext = op.splitext(filepath)\n if ext in allowed_exts:\n alloweds = allowed_exts[ext]\n _, ext2 = op.splitext(rest)\n if ext2 in alloweds:\n ext = ext2 + ext\n\n return ext"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds the extension ext to fpath if it doesn t already exist.", "response": "def add_extension_if_needed(filepath, ext, check_if_exists=False):\n \"\"\"Add the extension ext to fpath if it doesn't have it.\n\n Parameters\n ----------\n filepath: str\n File name or path\n\n ext: str\n File extension\n\n check_if_exists: bool\n\n Returns\n -------\n File name or path with extension added, if needed.\n \"\"\"\n if not filepath.endswith(ext):\n filepath += ext\n\n if check_if_exists:\n if not op.exists(filepath):\n raise IOError('File not found: ' + filepath)\n\n return filepath"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses a file with a list of subjects.", "response": "def parse_subjects_list(filepath, datadir='', split=':', labelsf=None):\n \"\"\"Parses a file with a list of: :.\n\n Parameters\n ----------\n filepath: str\n Path to file with a list of: :.\n Where ':' can be any split character\n\n datadir: str\n String to be path prefix of each line of the fname content,\n only in case the lines are relative file paths.\n\n split: str\n Split character for each line\n\n labelsf: str\n Path to file with a list of the labels if it is not included in\n fname. It will overwrite the labels from fname.\n\n Returns\n -------\n [labels, subjs] where labels is a list of labels and subjs a list of\n filepaths\n \"\"\"\n labels = []\n subjs = []\n\n if datadir:\n datadir += op.sep\n\n with open(filepath, 'r') as f:\n for s in f:\n line = s.strip().split(split)\n if len(line) == 2:\n labels.append(np.float(line[1]))\n subjf = line[0].strip()\n else:\n subjf = line.strip()\n\n if not op.isabs(subjf):\n subjs.append(datadir + subjf)\n else:\n subjs.append(subjf)\n\n\n if labelsf is not None:\n labels = np.loadtxt(labelsf)\n\n return [labels, subjs]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a file where each line is a subject_file.", "response": "def create_subjects_file(filelist, labels, output_file, split=':'):\n \"\"\"Creates a file where each line is :.\n\n Parameters\n ----------\n filelist: list of str\n List of filepaths\n\n labels: list of int, str or labels that can be transformed with str()\n List of labels\n\n output_file: str\n Output file path\n\n split: str\n Split character for each line\n\n \"\"\"\n if len(filelist) != len(labels):\n raise ValueError('Expected `filelist` and `labels` to have the same length.'\n 'Got {} and {}.'.format(len(filelist), len(labels)))\n\n lines = []\n for i, subj in enumerate(filelist):\n lab = labels[i]\n line = subj + split + str(lab)\n lines.append(line)\n\n lines = np.array(lines)\n np.savetxt(output_file, lines, fmt='%s')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef join_path_to_filelist(path, filelist):\n return [op.join(path, str(item)) for item in filelist]", "response": "Joins path to each line in filelist"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndeletes all files in filelist in folder", "response": "def remove_all(filelist, folder=''):\n \"\"\"Deletes all files in filelist\n\n Parameters\n ----------\n filelist: list of str\n List of the file paths to be removed\n\n folder: str\n Path to be used as common directory for all file paths in filelist\n \"\"\"\n if not folder:\n for f in filelist:\n os.remove(f)\n else:\n for f in filelist:\n os.remove(op.join(folder, f))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_folder_subpath(path, folder_depth):\n if path[0] == op.sep:\n folder_depth += 1\n\n return op.sep.join(path.split(op.sep)[0:folder_depth])", "response": "Returns a folder path of path with depth given by folder_dept"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a TemporaryDirectory object for the specified prefix and basepath.", "response": "def get_temp_dir(prefix=None, basepath=None):\n \"\"\"\n Uses tempfile to create a TemporaryDirectory using\n the default arguments.\n The folder is created using tempfile.mkdtemp() function.\n\n Parameters\n ----------\n prefix: str\n Name prefix for the temporary folder.\n\n basepath: str\n Directory where the new folder must be created.\n The default directory is chosen from a platform-dependent\n list, but the user of the application can control the\n directory location by setting the TMPDIR, TEMP or TMP\n environment variables.\n\n Returns\n -------\n folder object\n \"\"\"\n if basepath is None:\n return tempfile.TemporaryDirectory(dir=basepath)\n else:\n return tempfile.TemporaryDirectory(prefix=prefix, dir=basepath)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the length of the file using the wc GNU command.", "response": "def ux_file_len(filepath):\n \"\"\"Returns the length of the file using the 'wc' GNU command\n\n Parameters\n ----------\n filepath: str\n\n Returns\n -------\n float\n \"\"\"\n p = subprocess.Popen(['wc', '-l', filepath], stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n result, err = p.communicate()\n\n if p.returncode != 0:\n raise IOError(err)\n\n l = result.strip()\n l = int(l.split()[0])\n return l"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmerges two dictionaries. Values that evaluate to true take priority over falsy values. `dict_1` takes priority over `dict_2`.", "response": "def merge(dict_1, dict_2):\n \"\"\"Merge two dictionaries.\n\n Values that evaluate to true take priority over falsy values.\n `dict_1` takes priority over `dict_2`.\n\n \"\"\"\n return dict((str(key), dict_1.get(key) or dict_2.get(key))\n for key in set(dict_2) | set(dict_1))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a system path if it exists.", "response": "def get_sys_path(rcpath, app_name, section_name=None):\n \"\"\"Return a folder path if it exists.\n\n First will check if it is an existing system path, if it is, will return it\n expanded and absoluted.\n\n If this fails will look for the rcpath variable in the app_name rcfiles or\n exclusively within the given section_name, if given.\n\n Parameters\n ----------\n rcpath: str\n Existing folder path or variable name in app_name rcfile with an\n existing one.\n\n section_name: str\n Name of a section in the app_name rcfile to look exclusively there for\n variable names.\n\n app_name: str\n Name of the application to look for rcfile configuration files.\n\n Returns\n -------\n sys_path: str\n A expanded absolute file or folder path if the path exists.\n\n Raises\n ------\n IOError if the proposed sys_path does not exist.\n \"\"\"\n # first check if it is an existing path\n if op.exists(rcpath):\n return op.realpath(op.expanduser(rcpath))\n\n # look for the rcfile\n try:\n settings = rcfile(app_name, section_name)\n except:\n raise\n\n # look for the variable within the rcfile configutarions\n try:\n sys_path = op.expanduser(settings[rcpath])\n except KeyError:\n raise IOError('Could not find an existing variable with name {0} in'\n ' section {1} of {2}rc config setup. Maybe it is a '\n ' folder that could not be found.'.format(rcpath,\n section_name,\n app_name))\n # found the variable, now check if it is an existing path\n else:\n if not op.exists(sys_path):\n raise IOError('Could not find the path {3} indicated by the '\n 'variable {0} in section {1} of {2}rc config '\n 'setup.'.format(rcpath, section_name, app_name,\n sys_path))\n\n # expand the path and return\n return op.realpath(op.expanduser(sys_path))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread environment variables and config files and return them merged with the config files and the command line arguments.", "response": "def rcfile(appname, section=None, args={}, strip_dashes=True):\n \"\"\"Read environment variables and config files and return them merged with\n predefined list of arguments.\n\n Parameters\n ----------\n appname: str\n Application name, used for config files and environment variable\n names.\n\n section: str\n Name of the section to be read. If this is not set: appname.\n\n args:\n arguments from command line (optparse, docopt, etc).\n\n strip_dashes: bool\n Strip dashes prefixing key names from args dict.\n\n Returns\n --------\n dict\n containing the merged variables of environment variables, config\n files and args.\n\n Raises\n ------\n IOError\n In case the return value is empty.\n\n Notes\n -----\n Environment variables are read if they start with appname in uppercase\n with underscore, for example:\n\n TEST_VAR=1\n\n Config files compatible with ConfigParser are read and the section name\n appname is read, example:\n\n [appname]\n var=1\n\n We can also have host-dependent configuration values, which have\n priority over the default appname values.\n\n [appname]\n var=1\n\n [appname:mylinux]\n var=3\n\n\n For boolean flags do not try to use: 'True' or 'False',\n 'on' or 'off',\n '1' or '0'.\n Unless you are willing to parse this values by yourself.\n We recommend commenting the variables out with '#' if you want to set a\n flag to False and check if it is in the rcfile cfg dict, i.e.:\n\n flag_value = 'flag_variable' in cfg\n\n\n Files are read from: /etc/appname/config,\n /etc/appfilerc,\n ~/.config/appname/config,\n ~/.config/appname,\n ~/.appname/config,\n ~/.appnamerc,\n appnamerc,\n .appnamerc,\n appnamerc file found in 'path' folder variable in args,\n .appnamerc file found in 'path' folder variable in args,\n file provided by 'config' variable in args.\n\n Example\n -------\n args = rcfile(__name__, docopt(__doc__, version=__version__))\n \"\"\"\n if strip_dashes:\n for k in args.keys():\n args[k.lstrip('-')] = args.pop(k)\n\n environ = get_environment(appname)\n\n if section is None:\n section = appname\n\n config = get_config(appname,\n section,\n args.get('config', ''),\n args.get('path', ''))\n config = merge(merge(args, config), environ)\n\n if not config:\n raise IOError('Could not find any rcfile for application '\n '{}.'.format(appname))\n\n return config"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the dictionary containing the rcfile section configuration and variable values.", "response": "def get_rcfile_section(app_name, section_name):\n \"\"\" Return the dictionary containing the rcfile section configuration\n variables.\n\n Parameters\n ----------\n section_name: str\n Name of the section in the rcfiles.\n\n app_name: str\n Name of the application to look for its rcfiles.\n\n Returns\n -------\n settings: dict\n Dict with variable values\n \"\"\"\n try:\n settings = rcfile(app_name, section_name)\n except IOError:\n raise\n except:\n raise KeyError('Error looking for section {} in {} '\n ' rcfiles.'.format(section_name, app_name))\n else:\n return settings"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_rcfile_variable_value(var_name, app_name, section_name=None):\n cfg = get_rcfile_section(app_name, section_name)\n\n if var_name in cfg:\n raise KeyError('Option {} not found in {} '\n 'section.'.format(var_name, section_name))\n\n return cfg[var_name]", "response": "Returns the value of the variable in the section_name section of the rc file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds the first section in the rcfiles where the first var_name is found in the app_name rcfiles.", "response": "def find_in_sections(var_name, app_name):\n \"\"\" Return the section and the value of the variable where the first\n var_name is found in the app_name rcfiles.\n\n Parameters\n ----------\n var_name: str\n Name of the variable to be searched for.\n\n app_name: str\n Name of the application to look for its rcfiles.\n\n Returns\n -------\n section_name: str\n Name of the section in the rcfiles where var_name was first found.\n\n var_value: str\n The value of the first variable with given var_name.\n \"\"\"\n sections = get_sections(app_name)\n\n if not sections:\n raise ValueError('No sections found in {} rcfiles.'.format(app_name))\n\n for s in sections:\n try:\n var_value = get_rcfile_variable_value(var_name, section_name=s,\n app_name=app_name)\n except:\n pass\n else:\n return s, var_value\n\n raise KeyError('No variable {} has been found in {} '\n 'rcfiles.'.format(var_name, app_name))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef filter_list(lst, pattern):\n if is_fnmatch_regex(pattern) and not is_regex(pattern):\n #use fnmatch\n log.info('Using fnmatch for {0}'.format(pattern))\n filst = fnmatch.filter(lst, pattern)\n\n else:\n #use re\n log.info('Using regex match for {0}'.format(pattern))\n filst = match_list(lst, pattern)\n\n if filst:\n filst.sort()\n\n return filst", "response": "Filters the list using pattern."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving a nested dictionary adict and a path string returns its childen just below the path.", "response": "def get_subdict(adict, path, sep=os.sep):\n \"\"\"\n Given a nested dictionary adict.\n This returns its childen just below the path.\n The path is a string composed of adict keys separated by sep.\n\n :param adict: nested dict\n\n :param path: str\n\n :param sep: str\n\n :return: dict or list or leaf of treemap\n\n \"\"\"\n return reduce(adict.__class__.get, [p for p in op.split(sep) if p], adict)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a nested dictionary this returns all its leave elements in a list.", "response": "def get_dict_leaves(data):\n \"\"\"\n Given a nested dictionary, this returns all its leave elements in a list.\n\n :param adict:\n\n :return: list\n \"\"\"\n result = []\n if isinstance(data, dict):\n for item in data.values():\n result.extend(get_dict_leaves(item))\n elif isinstance(data, list):\n result.extend(data)\n else:\n result.append(data)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_possible_paths(base_path, path_regex):\n if not path_regex:\n return []\n\n if len(path_regex) < 1:\n return []\n\n if path_regex[0] == os.sep:\n path_regex = path_regex[1:]\n\n rest_files = ''\n if os.sep in path_regex:\n #split by os.sep\n node_names = path_regex.partition(os.sep)\n first_node = node_names[0]\n rest_nodes = node_names[2]\n\n folder_names = filter_list(os.listdir(base_path), first_node)\n\n for nom in folder_names:\n new_base = op.join(base_path, nom)\n if op.isdir(new_base):\n rest_files = get_possible_paths(new_base, rest_nodes)\n else:\n rest_files = filter_list(os.listdir(base_path), path_regex)\n\n files = []\n if rest_files:\n files = [op.join(base_path, f) for f in rest_files]\n\n return files", "response": "Returns a list of all possible paths within base_path."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _import_config(filepath):\n if not op.isfile(filepath):\n raise IOError('Data config file not found. '\n 'Got: {0}'.format(filepath))\n\n cfg = import_pyfile(filepath)\n\n if not hasattr(cfg, 'root_path'):\n raise KeyError('Config file root_path key not found.')\n\n if not hasattr(cfg, 'filetree'):\n raise KeyError('Config file filetree key not found.')\n\n return cfg.root_path, cfg.filetree", "response": "Imports filetree and root_path variable values from the filepath."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove the nodes that match the pattern.", "response": "def remove_nodes(self, pattern, adict):\n \"\"\"\n Remove the nodes that match the pattern.\n \"\"\"\n mydict = self._filetree if adict is None else adict\n\n if isinstance(mydict, dict):\n for nom in mydict.keys():\n if isinstance(mydict[nom], dict):\n matchs = filter_list(mydict[nom], pattern)\n for nom in matchs:\n mydict = self.remove_nodes(pattern, mydict[nom])\n mydict.pop(nom)\n else:\n mydict[nom] = filter_list(mydict[nom], pattern)\n else:\n matchs = set(filter_list(mydict, pattern))\n mydict = set(mydict) - matchs\n\n return mydict"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the number of nodes that match the pattern.", "response": "def count_node_match(self, pattern, adict=None):\n \"\"\"\n Return the number of nodes that match the pattern.\n\n :param pattern:\n\n :param adict:\n :return: int\n \"\"\"\n mydict = self._filetree if adict is None else adict\n\n k = 0\n if isinstance(mydict, dict):\n names = mydict.keys()\n k += len(filter_list(names, pattern))\n for nom in names:\n k += self.count_node_match(pattern, mydict[nom])\n else:\n k = len(filter_list(mydict, pattern))\n\n return k"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef as_float_array(X, copy=True, force_all_finite=True):\n if isinstance(X, np.matrix) or (not isinstance(X, np.ndarray)\n and not sp.issparse(X)):\n return check_array(X, ['csr', 'csc', 'coo'], dtype=np.float64,\n copy=copy, force_all_finite=force_all_finite,\n ensure_2d=False)\n elif sp.issparse(X) and X.dtype in [np.float32, np.float64]:\n return X.copy() if copy else X\n elif X.dtype in [np.float32, np.float64]: # is numpy array\n return X.copy('F' if X.flags['F_CONTIGUOUS'] else 'C') if copy else X\n else:\n return X.astype(np.float32 if X.dtype == np.int32 else np.float64)", "response": "Converts an array - like to an array of floats"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn number of samples in array - like x.", "response": "def _num_samples(x):\n \"\"\"Return number of samples in array-like x.\"\"\"\n if not hasattr(x, '__len__') and not hasattr(x, 'shape'):\n if hasattr(x, '__array__'):\n x = np.asarray(x)\n else:\n raise TypeError(\"Expected sequence or array-like, got %r\" % x)\n return x.shape[0] if hasattr(x, 'shape') else len(x)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck that all arrays have consistent length.", "response": "def check_consistent_length(*arrays):\n \"\"\"Check that all arrays have consistent first dimensions.\n\n Checks whether all objects in arrays have the same shape or length.\n\n Parameters\n ----------\n arrays : list or tuple of input objects.\n Objects that will be checked for consistent length.\n \"\"\"\n\n uniques = np.unique([_num_samples(X) for X in arrays if X is not None])\n if len(uniques) > 1:\n raise ValueError(\"Found arrays with inconsistent numbers of samples: %s\"\n % str(uniques))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef indexable(*iterables):\n result = []\n for X in iterables:\n if sp.issparse(X):\n result.append(X.tocsr())\n elif hasattr(X, \"__getitem__\") or hasattr(X, \"iloc\"):\n result.append(X)\n elif X is None:\n result.append(X)\n else:\n result.append(np.array(X))\n check_consistent_length(*result)\n return result", "response": "Make arrays indexable for cross - validation."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninputting validation on an array.", "response": "def check_array(array, accept_sparse=None, dtype=None, order=None, copy=False,\n force_all_finite=True, ensure_2d=True, allow_nd=False):\n \"\"\"Input validation on an array, list, sparse matrix or similar.\n\n By default, the input is converted to an at least 2nd numpy array.\n\n Parameters\n ----------\n array : object\n Input object to check / convert.\n\n accept_sparse : string, list of string or None (default=None)\n String[s] representing allowed sparse matrix formats, such as 'csc',\n 'csr', etc. None means that sparse matrix input will raise an error.\n If the input is sparse but not in the allowed format, it will be\n converted to the first listed format.\n\n order : 'F', 'C' or None (default=None)\n Whether an array will be forced to be fortran or c-style.\n\n copy : boolean (default=False)\n Whether a forced copy will be triggered. If copy=False, a copy might\n be triggered by a conversion.\n\n force_all_finite : boolean (default=True)\n Whether to raise an error on np.inf and np.nan in X.\n\n ensure_2d : boolean (default=True)\n Whether to make X at least 2d.\n\n allow_nd : boolean (default=False)\n Whether to allow X.ndim > 2.\n\n Returns\n -------\n X_converted : object\n The converted and validated X.\n \"\"\"\n if isinstance(accept_sparse, str):\n accept_sparse = [accept_sparse]\n\n if sp.issparse(array):\n array = _ensure_sparse_format(array, accept_sparse, dtype, order,\n copy, force_all_finite)\n else:\n if ensure_2d:\n array = np.atleast_2d(array)\n array = np.array(array, dtype=dtype, order=order, copy=copy)\n if not allow_nd and array.ndim >= 3:\n raise ValueError(\"Found array with dim %d. Expected <= 2\" %\n array.ndim)\n if force_all_finite:\n _assert_all_finite(array)\n\n return array"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_X_y(X, y, accept_sparse=None, dtype=None, order=None, copy=False,\n force_all_finite=True, ensure_2d=True, allow_nd=False,\n multi_output=False):\n \"\"\"Input validation for standard estimators.\n\n Checks X and y for consistent length, enforces X 2d and y 1d.\n Standard input checks are only applied to y. For multi-label y,\n set multi_ouput=True to allow 2d and sparse y.\n\n Parameters\n ----------\n X : nd-array, list or sparse matrix\n Input data.\n\n y : nd-array, list or sparse matrix\n Labels.\n\n accept_sparse : string, list of string or None (default=None)\n String[s] representing allowed sparse matrix formats, such as 'csc',\n 'csr', etc. None means that sparse matrix input will raise an error.\n If the input is sparse but not in the allowed format, it will be\n converted to the first listed format.\n\n order : 'F', 'C' or None (default=None)\n Whether an array will be forced to be fortran or c-style.\n\n copy : boolean (default=False)\n Whether a forced copy will be triggered. If copy=False, a copy might\n be triggered by a conversion.\n\n force_all_finite : boolean (default=True)\n Whether to raise an error on np.inf and np.nan in X.\n\n ensure_2d : boolean (default=True)\n Whether to make X at least 2d.\n\n allow_nd : boolean (default=False)\n Whether to allow X.ndim > 2.\n\n Returns\n -------\n X_converted : object\n The converted and validated X.\n \"\"\"\n X = check_array(X, accept_sparse, dtype, order, copy, force_all_finite,\n ensure_2d, allow_nd)\n if multi_output:\n y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False)\n else:\n y = column_or_1d(y, warn=True)\n _assert_all_finite(y)\n\n check_consistent_length(X, y)\n\n return X, y", "response": "Input validation for standard estimators."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef column_or_1d(y, warn=False):\n shape = np.shape(y)\n if len(shape) == 1:\n return np.ravel(y)\n if len(shape) == 2 and shape[1] == 1:\n if warn:\n warnings.warn(\"A column-vector y was passed when a 1d array was\"\n \" expected. Please change the shape of y to \"\n \"(n_samples, ), for example using ravel().\",\n DataConversionWarning, stacklevel=2)\n return np.ravel(y)\n\n raise ValueError(\"bad input shape {0}\".format(shape))", "response": "Ravel column or 1d numpy array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef warn_if_not_float(X, estimator='This algorithm'):\n if not isinstance(estimator, str):\n estimator = estimator.__class__.__name__\n if X.dtype.kind != 'f':\n warnings.warn(\"%s assumes floating point values as input, \"\n \"got %s\" % (estimator, X.dtype))\n return True\n return False", "response": "Warning utility function to check that data type is floating point."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts an arbitrary array to numpy. ndarray.", "response": "def as_ndarray(arr, copy=False, dtype=None, order='K'):\n \"\"\"Convert an arbitrary array to numpy.ndarray.\n\n In the case of a memmap array, a copy is automatically made to break the\n link with the underlying file (whatever the value of the \"copy\" keyword).\n\n The purpose of this function is mainly to get rid of memmap objects, but\n it can be used for other purposes. In particular, combining copying and\n casting can lead to performance improvements in some cases, by avoiding\n unnecessary copies.\n\n If not specified, input array order is preserved, in all cases, even when\n a copy is requested.\n\n Caveat: this function does not copy during bool to/from 1-byte dtype\n conversions. This can lead to some surprising results in some rare cases.\n Example:\n\n a = numpy.asarray([0, 1, 2], dtype=numpy.int8)\n b = as_ndarray(a, dtype=bool) # array([False, True, True], dtype=bool)\n c = as_ndarray(b, dtype=numpy.int8) # array([0, 1, 2], dtype=numpy.int8)\n\n The usually expected result for the last line would be array([0, 1, 1])\n because True evaluates to 1. Since there is no copy made here, the original\n array is recovered.\n\n Parameters\n ----------\n arr: array-like\n input array. Any value accepted by numpy.asarray is valid.\n\n copy: bool\n if True, force a copy of the array. Always True when arr is a memmap.\n\n dtype: any numpy dtype\n dtype of the returned array. Performing copy and type conversion at the\n same time can in some cases avoid an additional copy.\n\n order: string\n gives the order of the returned array.\n Valid values are: \"C\", \"F\", \"A\", \"K\", None.\n default is \"K\". See ndarray.copy() for more information.\n\n Returns\n -------\n ret: np.ndarray\n Numpy array containing the same data as arr, always of class\n numpy.ndarray, and with no link to any underlying file.\n \"\"\"\n if order not in ('C', 'F', 'A', 'K', None):\n raise ValueError(\"Invalid value for 'order': {}\".format(str(order)))\n\n if isinstance(arr, np.memmap):\n if dtype is None:\n if order in ('K', 'A', None):\n ret = np.array(np.asarray(arr), copy=True)\n else:\n ret = np.array(np.asarray(arr), copy=True, order=order)\n else:\n if order in ('K', 'A', None):\n # always copy (even when dtype does not change)\n ret = np.asarray(arr).astype(dtype)\n else:\n # load data from disk without changing order\n # Changing order while reading through a memmap is incredibly\n # inefficient.\n ret = _asarray(np.array(arr, copy=True), dtype=dtype, order=order)\n\n elif isinstance(arr, np.ndarray):\n ret = _asarray(arr, dtype=dtype, order=order)\n # In the present cas, np.may_share_memory result is always reliable.\n if np.may_share_memory(ret, arr) and copy:\n # order-preserving copy\n ret = ret.T.copy().T if ret.flags['F_CONTIGUOUS'] else ret.copy()\n\n elif isinstance(arr, (list, tuple)):\n if order in (\"A\", \"K\"):\n ret = np.asarray(arr, dtype=dtype)\n else:\n ret = np.asarray(arr, dtype=dtype, order=order)\n\n else:\n raise ValueError(\"Type not handled: {}\".format(arr.__class__))\n\n return ret"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall FSL tools to apply transformations to a given atlas to a functional image. Given the transformation matrices. Parameters ---------- atlas_filepath: str Path to the 3D atlas volume file. anatbrain_filepath: str Path to the anatomical brain volume file (skull-stripped and registered to the same space as the atlas, e.g., MNI). meanfunc_filepath: str Path to the average functional image to be used as reference in the last applywarp step. atlas2anat_nonlin_xfm_filepath: str Path to the atlas to anatomical brain linear transformation .mat file. If you have the inverse transformation, i.e., anatomical brain to atlas, set is_atlas2anat_inverted to True. is_atlas2anat_inverted: bool If False will have to calculate the inverse atlas2anat transformation to apply the transformations. This step will be performed with FSL invwarp. anat2func_lin_xfm_filepath: str Path to the anatomical to functional .mat linear transformation file. atlasinanat_out_filepath: str Path to output file which will contain the 3D atlas in the subject anatomical space. atlasinfunc_out_filepath: str Path to output file which will contain the 3D atlas in the subject functional space. verbose: bool If verbose will show DEBUG log info. rewrite: bool If True will re-run all the commands overwriting any existing file. Otherwise will check if each file exists and if it does won't run the command. parallel: bool If True will launch the commands using ${FSLDIR}/fsl_sub to use the cluster infrastructure you have setup with FSL (SGE or HTCondor).", "response": "def xfm_atlas_to_functional(atlas_filepath, anatbrain_filepath, meanfunc_filepath,\n atlas2anat_nonlin_xfm_filepath, is_atlas2anat_inverted,\n anat2func_lin_xfm_filepath,\n atlasinanat_out_filepath, atlasinfunc_out_filepath,\n interp='nn', rewrite=True, parallel=False):\n \"\"\"Call FSL tools to apply transformations to a given atlas to a functional image.\n Given the transformation matrices.\n\n Parameters\n ----------\n atlas_filepath: str\n Path to the 3D atlas volume file.\n\n anatbrain_filepath: str\n Path to the anatomical brain volume file (skull-stripped and registered to the same space as the atlas,\n e.g., MNI).\n\n meanfunc_filepath: str\n Path to the average functional image to be used as reference in the last applywarp step.\n\n atlas2anat_nonlin_xfm_filepath: str\n Path to the atlas to anatomical brain linear transformation .mat file.\n If you have the inverse transformation, i.e., anatomical brain to atlas, set is_atlas2anat_inverted to True.\n\n is_atlas2anat_inverted: bool\n If False will have to calculate the inverse atlas2anat transformation to apply the transformations.\n This step will be performed with FSL invwarp.\n\n anat2func_lin_xfm_filepath: str\n Path to the anatomical to functional .mat linear transformation file.\n\n atlasinanat_out_filepath: str\n Path to output file which will contain the 3D atlas in the subject anatomical space.\n\n atlasinfunc_out_filepath: str\n Path to output file which will contain the 3D atlas in the subject functional space.\n\n verbose: bool\n If verbose will show DEBUG log info.\n\n rewrite: bool\n If True will re-run all the commands overwriting any existing file. Otherwise will check if\n each file exists and if it does won't run the command.\n\n parallel: bool\n If True will launch the commands using ${FSLDIR}/fsl_sub to use the cluster infrastructure you have setup\n with FSL (SGE or HTCondor).\n \"\"\"\n if is_atlas2anat_inverted:\n # I already have the inverted fields I need\n anat_to_mni_nl_inv = atlas2anat_nonlin_xfm_filepath\n else:\n # I am creating the inverted fields then...need output file path:\n output_dir = op.abspath (op.dirname(atlasinanat_out_filepath))\n ext = get_extension(atlas2anat_nonlin_xfm_filepath)\n anat_to_mni_nl_inv = op.join(output_dir, remove_ext(op.basename(atlas2anat_nonlin_xfm_filepath)) + '_inv' + ext)\n\n # setup the commands to be called\n invwarp_cmd = op.join('${FSLDIR}', 'bin', 'invwarp')\n applywarp_cmd = op.join('${FSLDIR}', 'bin', 'applywarp')\n fslsub_cmd = op.join('${FSLDIR}', 'bin', 'fsl_sub')\n\n # add fsl_sub before the commands\n if parallel:\n invwarp_cmd = fslsub_cmd + ' ' + invwarp_cmd\n applywarp_cmd = fslsub_cmd + ' ' + applywarp_cmd\n\n # create the inverse fields\n if rewrite or (not is_atlas2anat_inverted and not op.exists(anat_to_mni_nl_inv)):\n log.debug('Creating {}.\\n'.format(anat_to_mni_nl_inv))\n cmd = invwarp_cmd + ' '\n cmd += '-w {} '.format(atlas2anat_nonlin_xfm_filepath)\n cmd += '-o {} '.format(anat_to_mni_nl_inv)\n cmd += '-r {} '.format(anatbrain_filepath)\n log.debug('Running {}'.format(cmd))\n check_call(cmd)\n\n # transform the atlas to anatomical space\n if rewrite or not op.exists(atlasinanat_out_filepath):\n log.debug('Creating {}.\\n'.format(atlasinanat_out_filepath))\n cmd = applywarp_cmd + ' '\n cmd += '--in={} '.format(atlas_filepath)\n cmd += '--ref={} '.format(anatbrain_filepath)\n cmd += '--warp={} '.format(anat_to_mni_nl_inv)\n cmd += '--interp={} '.format(interp)\n cmd += '--out={} '.format(atlasinanat_out_filepath)\n log.debug('Running {}'.format(cmd))\n check_call(cmd)\n\n # transform the atlas to functional space\n if rewrite or not op.exists(atlasinfunc_out_filepath):\n log.debug('Creating {}.\\n'.format(atlasinfunc_out_filepath))\n cmd = applywarp_cmd + ' '\n cmd += '--in={} '.format(atlasinanat_out_filepath)\n cmd += '--ref={} '.format(meanfunc_filepath)\n cmd += '--premat={} '.format(anat2func_lin_xfm_filepath)\n cmd += '--interp={} '.format(interp)\n cmd += '--out={} '.format(atlasinfunc_out_filepath)\n log.debug('Running {}'.format(cmd))\n check_call(cmd)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fwhm2sigma(fwhm):\n fwhm = np.asarray(fwhm)\n return fwhm / np.sqrt(8 * np.log(2))", "response": "Convert a FWHM value to sigma in a Gaussian kernel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a sigma in a Gaussian kernel to a FWHM value.", "response": "def sigma2fwhm(sigma):\n \"\"\"Convert a sigma in a Gaussian kernel to a FWHM value.\n\n Parameters\n ----------\n sigma: float or numpy.array\n sigma value or values\n\n Returns\n -------\n fwhm: float or numpy.array\n fwhm values corresponding to `sigma` values\n \"\"\"\n sigma = np.asarray(sigma)\n return np.sqrt(8 * np.log(2)) * sigma"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsmoothing images with a Gaussian filter.", "response": "def _smooth_data_array(arr, affine, fwhm, copy=True):\n \"\"\"Smooth images with a a Gaussian filter.\n\n Apply a Gaussian filter along the three first dimensions of arr.\n\n Parameters\n ----------\n arr: numpy.ndarray\n 3D or 4D array, with image number as last dimension.\n\n affine: numpy.ndarray\n Image affine transformation matrix for image.\n\n fwhm: scalar, numpy.ndarray\n Smoothing kernel size, as Full-Width at Half Maximum (FWHM) in millimeters.\n If a scalar is given, kernel width is identical on all three directions.\n A numpy.ndarray must have 3 elements, giving the FWHM along each axis.\n\n copy: bool\n if True, will make a copy of the input array. Otherwise will directly smooth the input array.\n\n Returns\n -------\n smooth_arr: numpy.ndarray\n \"\"\"\n\n if arr.dtype.kind == 'i':\n if arr.dtype == np.int64:\n arr = arr.astype(np.float64)\n else:\n arr = arr.astype(np.float32)\n if copy:\n arr = arr.copy()\n\n # Zeroe possible NaNs and Inf in the image.\n arr[np.logical_not(np.isfinite(arr))] = 0\n\n try:\n # Keep the 3D part of the affine.\n affine = affine[:3, :3]\n\n # Convert from FWHM in mm to a sigma.\n fwhm_sigma_ratio = np.sqrt(8 * np.log(2))\n vox_size = np.sqrt(np.sum(affine ** 2, axis=0))\n sigma = fwhm / (fwhm_sigma_ratio * vox_size)\n for n, s in enumerate(sigma):\n ndimage.gaussian_filter1d(arr, s, output=arr, axis=n)\n except:\n raise ValueError('Error smoothing the array.')\n else:\n return arr"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsmooth images using a Gaussian filter along the three first dimensions of each image in images.", "response": "def smooth_imgs(images, fwhm):\n \"\"\"Smooth images using a Gaussian filter.\n\n Apply a Gaussian filter along the three first dimensions of each image in images.\n In all cases, non-finite values in input are zeroed.\n\n Parameters\n ----------\n imgs: str or img-like object or iterable of img-like objects\n See boyle.nifti.read.read_img\n Image(s) to smooth.\n\n fwhm: scalar or numpy.ndarray\n Smoothing kernel size, as Full-Width at Half Maximum (FWHM) in millimeters.\n If a scalar is given, kernel width is identical on all three directions.\n A numpy.ndarray must have 3 elements, giving the FWHM along each axis.\n\n Returns\n -------\n smooth_imgs: nibabel.Nifti1Image or list of.\n Smooth input image/s.\n \"\"\"\n if fwhm <= 0:\n return images\n\n if not isinstance(images, string_types) and hasattr(images, '__iter__'):\n only_one = False\n else:\n only_one = True\n images = [images]\n\n result = []\n for img in images:\n img = check_img(img)\n affine = img.get_affine()\n smooth = _smooth_data_array(img.get_data(), affine, fwhm=fwhm, copy=True)\n result.append(nib.Nifti1Image(smooth, affine))\n\n if only_one:\n return result[0]\n else:\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _smooth_array(arr, affine, fwhm=None, ensure_finite=True, copy=True, **kwargs):\n\n if arr.dtype.kind == 'i':\n if arr.dtype == np.int64:\n arr = arr.astype(np.float64)\n else:\n # We don't need crazy precision\n arr = arr.astype(np.float32)\n if copy:\n arr = arr.copy()\n\n if ensure_finite:\n # SPM tends to put NaNs in the data outside the brain\n arr[np.logical_not(np.isfinite(arr))] = 0\n\n if fwhm == 'fast':\n arr = _fast_smooth_array(arr)\n elif fwhm is not None:\n # Keep only the scale part.\n affine = affine[:3, :3]\n\n # Convert from a FWHM to a sigma:\n fwhm_over_sigma_ratio = np.sqrt(8 * np.log(2))\n vox_size = np.sqrt(np.sum(affine ** 2, axis=0))\n sigma = fwhm / (fwhm_over_sigma_ratio * vox_size)\n for n, s in enumerate(sigma):\n ndimage.gaussian_filter1d(arr, s, output=arr, axis=n, **kwargs)\n\n return arr", "response": "Smooth images by applying a Gaussian filter along the three first dimensions of arr."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef smooth_img(imgs, fwhm, **kwargs):\n\n # Use hasattr() instead of isinstance to workaround a Python 2.6/2.7 bug\n # See http://bugs.python.org/issue7624\n if hasattr(imgs, \"__iter__\") \\\n and not isinstance(imgs, string_types):\n single_img = False\n else:\n single_img = True\n imgs = [imgs]\n\n ret = []\n for img in imgs:\n img = check_niimg(img)\n affine = img.get_affine()\n filtered = _smooth_array(img.get_data(), affine, fwhm=fwhm,\n ensure_finite=True, copy=True, **kwargs)\n ret.append(new_img_like(img, filtered, affine, copy_header=True))\n\n if single_img:\n return ret[0]\n else:\n return ret", "response": "Smooth images by applying a Gaussian filter along the three first dimensions of arr."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef signed_session(self, session=None):\n\n if session:\n session = super(ClientCertAuthentication, self).signed_session(session)\n else:\n session = super(ClientCertAuthentication, self).signed_session()\n\n if self.cert is not None:\n session.cert = self.cert\n if self.ca_cert is not None:\n session.verify = self.ca_cert\n if self.no_verify:\n session.verify = False\n\n return session", "response": "Create requests. Session object with any required auth headers\n applied."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef signed_session(self, session=None):\n\n from sfctl.config import (aad_metadata, aad_cache)\n\n if session:\n session = super(AdalAuthentication, self).signed_session(session)\n else:\n session = super(AdalAuthentication, self).signed_session()\n\n if self.no_verify:\n session.verify = False\n\n authority_uri, cluster_id, client_id = aad_metadata()\n existing_token, existing_cache = aad_cache()\n context = adal.AuthenticationContext(authority_uri,\n cache=existing_cache)\n new_token = context.acquire_token(cluster_id,\n existing_token['userId'], client_id)\n header = \"{} {}\".format(\"Bearer\", new_token['accessToken'])\n session.headers['Authorization'] = header\n return session", "response": "Create requests. Session object with AAD auth headers"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef voxspace_to_mmspace(img):\n shape, affine = img.shape[:3], img.affine\n coords = np.array(np.meshgrid(*(range(i) for i in shape), indexing='ij'))\n coords = np.rollaxis(coords, 0, len(shape) + 1)\n mm_coords = nib.affines.apply_affine(affine, coords)\n\n return mm_coords", "response": "Return a grid with coordinates in 3D physical space for img."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a Voxel coordinates to a Triplet with real 3D world coordinates.", "response": "def voxcoord_to_mm(cm, i, j, k):\n '''\n Parameters\n ----------\n cm: nipy.core.reference.coordinate_map.CoordinateMap\n\n i, j, k: floats\n Voxel coordinates\n\n Returns\n -------\n Triplet with real 3D world coordinates\n\n '''\n try:\n mm = cm([i, j, k])\n except Exception as exc:\n raise Exception('Error on converting coordinates.') from exc\n else:\n return mm"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a coordinate map to a 3D voxel coordinates.", "response": "def mm_to_voxcoord(cm, x, y, z):\n '''\n Parameters\n ----------\n cm: nipy.core.reference.coordinate_map.CoordinateMap\n\n x, y, z: floats\n Physical coordinates\n\n Returns\n -------\n Triplet with 3D voxel coordinates\n '''\n try:\n vox = cm.inverse()([x, y, z])\n except Exception as exc:\n raise Exception('Error on converting coordinates') from exc\n else:\n return vox"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a 3D CoordinateMap from img.", "response": "def get_3D_coordmap(img):\n '''\n Gets a 3D CoordinateMap from img.\n\n Parameters\n ----------\n img: nib.Nifti1Image or nipy Image\n\n Returns\n -------\n nipy.core.reference.coordinate_map.CoordinateMap\n '''\n if isinstance(img, nib.Nifti1Image):\n img = nifti2nipy(img)\n\n if img.ndim == 4:\n from nipy.core.reference.coordinate_map import drop_io_dim\n cm = drop_io_dim(img.coordmap, 3)\n else:\n cm = img.coordmap\n\n return cm"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the header and affine matrix from a Nifti file.", "response": "def get_img_info(image):\n \"\"\"Return the header and affine matrix from a Nifti file.\n\n Parameters\n ----------\n image: img-like object or str\n Can either be:\n - a file path to a Nifti image\n - any object with get_data() and get_affine() methods, e.g., nibabel.Nifti1Image.\n If niimg is a string, consider it as a path to Nifti image and\n call nibabel.load on it. If it is an object, check if get_data()\n and get_affine() methods are present, raise TypeError otherwise.\n\n Returns\n -------\n hdr, aff\n \"\"\"\n try:\n img = check_img(image)\n except Exception as exc:\n raise Exception('Error reading file {0}.'.format(repr_imgs(image))) from exc\n else:\n return img.get_header(), img.get_affine()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the voxel matrix of the Nifti image.", "response": "def get_img_data(image, copy=True):\n \"\"\"Return the voxel matrix of the Nifti file.\n If safe_mode will make a copy of the img before returning the data, so the input image is not modified.\n\n Parameters\n ----------\n image: img-like object or str\n Can either be:\n - a file path to a Nifti image\n - any object with get_data() and get_affine() methods, e.g., nibabel.Nifti1Image.\n If niimg is a string, consider it as a path to Nifti image and\n call nibabel.load on it. If it is an object, check if get_data()\n and get_affine() methods are present, raise TypeError otherwise.\n\n copy: bool\n If safe_mode will make a copy of the img before returning the data, so the input image is not modified.\n\n Returns\n -------\n array_like\n \"\"\"\n try:\n img = check_img(image)\n if copy:\n return get_data(img)\n else:\n return img.get_data()\n except Exception as exc:\n raise Exception('Error when reading file {0}.'.format(repr_imgs(image))) from exc"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_nipy_img(nii_file):\n # delayed import because could not install nipy on Python 3 on OSX\n import nipy\n\n if not os.path.exists(nii_file):\n raise FileNotFound(nii_file)\n\n try:\n return nipy.load_image(nii_file)\n except Exception as exc:\n raise Exception('Reading file {0}.'.format(repr_imgs(nii_file))) from exc", "response": "Read a Nifti file and return as nipy. Image"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef niftilist_to_array(img_filelist, outdtype=None):\n try:\n first_img = img_filelist[0]\n vol = get_img_data(first_img)\n except IndexError as ie:\n raise Exception('Error getting the first item of img_filelis: {}'.format(repr_imgs(img_filelist[0]))) from ie\n\n if not outdtype:\n outdtype = vol.dtype\n\n outmat = np.zeros((len(img_filelist), np.prod(vol.shape)), dtype=outdtype)\n\n try:\n for i, img_file in enumerate(img_filelist):\n vol = get_img_data(img_file)\n outmat[i, :] = vol.flatten()\n except Exception as exc:\n raise Exception('Error on reading file {0}.'.format(img_file)) from exc\n\n return outmat, vol.shape", "response": "This function takes a list of absolute paths to nifti files and creates a Numpy array containing the data of the volumes."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _crop_img_to(image, slices, copy=True):\n\n img = check_img(image)\n data = img.get_data()\n affine = img.get_affine()\n\n cropped_data = data[slices]\n if copy:\n cropped_data = cropped_data.copy()\n\n linear_part = affine[:3, :3]\n old_origin = affine[:3, 3]\n new_origin_voxel = np.array([s.start for s in slices])\n new_origin = old_origin + linear_part.dot(new_origin_voxel)\n\n new_affine = np.eye(4)\n new_affine[:3, :3] = linear_part\n new_affine[:3, 3] = new_origin\n\n new_img = nib.Nifti1Image(cropped_data, new_affine)\n\n return new_img", "response": "Crop an image to a smaller size and modify the affine accordingly."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef crop_img(image, rtol=1e-8, copy=True):\n\n img = check_img(image)\n data = img.get_data()\n infinity_norm = max(-data.min(), data.max())\n passes_threshold = np.logical_or(data < -rtol * infinity_norm,\n data > rtol * infinity_norm)\n\n if data.ndim == 4:\n passes_threshold = np.any(passes_threshold, axis=-1)\n\n coords = np.array(np.where(passes_threshold))\n start = coords.min(axis=1)\n end = coords.max(axis=1) + 1\n\n # pad with one voxel to avoid resampling problems\n start = np.maximum(start - 1, 0)\n end = np.minimum(end + 1, data.shape[:3])\n\n slices = [slice(s, e) for s, e in zip(start, end)]\n\n return _crop_img_to(img, slices, copy=copy)", "response": "Crop an image as much as possible and return a new image with the same size."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new image of the same type as the reference image.", "response": "def new_img_like(ref_niimg, data, affine=None, copy_header=False):\n \"\"\"Create a new image of the same class as the reference image\n\n Parameters\n ----------\n ref_niimg: image\n Reference image. The new image will be of the same type.\n\n data: numpy array\n Data to be stored in the image\n\n affine: 4x4 numpy array, optional\n Transformation matrix\n\n copy_header: boolean, optional\n Indicated if the header of the reference image should be used to\n create the new image\n\n Returns\n -------\n new_img: image\n A loaded image with the same type (and header) as the reference image.\n \"\"\"\n # Hand-written loading code to avoid too much memory consumption\n if not (hasattr(ref_niimg, 'get_data')\n and hasattr(ref_niimg,'get_affine')):\n if isinstance(ref_niimg, _basestring):\n ref_niimg = nib.load(ref_niimg)\n elif operator.isSequenceType(ref_niimg):\n ref_niimg = nib.load(ref_niimg[0])\n else:\n raise TypeError(('The reference image should be a niimg, %r '\n 'was passed') % ref_niimg )\n\n if affine is None:\n affine = ref_niimg.get_affine()\n if data.dtype == bool:\n default_dtype = np.int8\n if (LooseVersion(nib.__version__) >= LooseVersion('1.2.0') and\n isinstance(ref_niimg, nib.freesurfer.mghformat.MGHImage)):\n default_dtype = np.uint8\n data = as_ndarray(data, dtype=default_dtype)\n header = None\n if copy_header:\n header = copy.copy(ref_niimg.get_header())\n header['scl_slope'] = 0.\n header['scl_inter'] = 0.\n header['glmax'] = 0.\n header['cal_max'] = np.max(data) if data.size > 0 else 0.\n header['cal_max'] = np.min(data) if data.size > 0 else 0.\n return ref_niimg.__class__(data, affine, header=header)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsave variables to HDF5 file.", "response": "def save_variables_to_hdf5(file_path, variables, mode='w', h5path='/'):\n \"\"\"\n Parameters\n ----------\n file_path: str\n\n variables: dict\n Dictionary with objects. Object name -> object\n\n mode: str\n HDF5 file access mode\n See h5py documentation for details.\n r Readonly, file must exist\n r+ Read/write, file must exist\n w Create file, truncate if exists\n w- Create file, fail if exists\n a Read/write if exists, create otherwise (default)\n\n Notes\n -----\n It is recommended to use numpy arrays as objects.\n List or tuples of strings won't work, convert them into numpy.arrays before.\n \"\"\"\n if not isinstance(variables, dict):\n raise ValueError('Expected `variables` to be a dict, got a {}.'.format(type(variables)))\n\n if not variables:\n raise ValueError('Expected `variables` to be a non-empty dict.')\n\n h5file = h5py.File(file_path, mode=mode)\n h5group = h5file.require_group(h5path)\n\n for vn in variables:\n data = variables[vn]\n\n # fix for string numpy arrays\n if hasattr(data, 'dtype') and (data.dtype.type is np.string_ or data.dtype.type is np.unicode_):\n dt = h5py.special_dtype(vlen=str)\n data = data.astype(dt)\n\n if isinstance(data, dict):\n for key in data:\n #h5group.create_dataset(str(key))\n #import ipdb\n #ipdb.set_trace()\n h5group[str(key)] = data[key]\n\n elif isinstance(data, list):\n for idx, item in enumerate(data):\n #h5group.create_dataset(str(idx))\n h5group[str(idx)] = item\n else:\n h5group[vn] = data\n\n h5file.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_h5file(file_path, mode='r'):\n if not op.exists(file_path):\n raise IOError('Could not find file {}.'.format(file_path))\n\n try:\n h5file = h5py.File(file_path, mode=mode)\n except:\n raise\n else:\n return h5file", "response": "Returns the h5py. File given its file path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef extract_datasets(h5file, h5path='/'):\n if isinstance(h5file, str):\n _h5file = h5py.File(h5file, mode='r')\n else:\n _h5file = h5file\n\n _datasets = get_datasets(_h5file, h5path)\n datasets = OrderedDict()\n try:\n for ds in _datasets:\n datasets[ds.name.split('/')[-1]] = ds[:]\n except:\n raise RuntimeError('Error reading datasets in {}/{}.'.format(_h5file.filename, h5path))\n finally:\n if isinstance(h5file, str):\n _h5file.close()\n\n return datasets", "response": "Return all datasets contained in file_path in h5file in an OrderedDict."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_node_names(h5file, h5path='/', node_type=h5py.Dataset):\n if isinstance(h5file, str):\n _h5file = get_h5file(h5file, mode='r')\n else:\n _h5file = h5file\n\n if not h5path.startswith('/'):\n h5path = '/' + h5path\n\n names = []\n try:\n h5group = _h5file.require_group(h5path)\n\n for node in _hdf5_walk(h5group, node_type=node_type):\n names.append(node.name)\n except:\n raise RuntimeError('Error getting node names from {}/{}.'.format(_h5file.filename, h5path))\n finally:\n if isinstance(h5file, str):\n _h5file.close()\n\n return names", "response": "Get the node names within the HDF5 file h5path."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mask(self, image):\n if image is None:\n self._mask = None\n\n try:\n mask = load_mask(image)\n except Exception as exc:\n raise Exception('Could not load mask image {}.'.format(image)) from exc\n else:\n self._mask = mask", "response": "setter\n - Sets the mask attribute of the object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if two images are compatible with the current set of items.", "response": "def check_compatibility(self, one_img, another_img=None):\n \"\"\"\n Parameters\n ----------\n one_img: str or img-like object.\n See NeuroImage constructor docstring.\n\n anoter_img: str or img-like object.\n See NeuroImage constructor docstring.\n If None will use the first image of self.images, if there is any.\n\n Raises\n ------\n NiftiFilesNotCompatible\n If one_img and another_img aren't compatible.\n\n ValueError\n If another_img is None and there are no other images in this set.\n \"\"\"\n if another_img is None:\n if len(self.items) > 0:\n another_img = self.items[0]\n else:\n raise ValueError('self.items is empty, need an image to compare '\n 'with {}'.format(repr_imgs(one_img)))\n\n try:\n if self.all_compatible:\n check_img_compatibility(one_img, another_img)\n if self.mask is not None:\n check_img_compatibility(one_img, self.mask, only_check_3d=True)\n except:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the labels of the related object.", "response": "def set_labels(self, labels):\n \"\"\"\n Parameters\n ----------\n labels: list of int or str\n This list will be checked to have the same size as\n\n Raises\n ------\n ValueError\n if len(labels) != self.n_subjs\n \"\"\"\n if not isinstance(labels, string_types) and len(labels) != self.n_subjs:\n raise ValueError('The number of given labels ({}) is not the same '\n 'as the number of subjects ({}).'.format(len(labels), self.n_subjs))\n\n self.labels = labels"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _load_images_and_labels(self, images, labels=None):\n if not isinstance(images, (list, tuple)):\n raise ValueError('Expected an iterable (list or tuple) of strings or img-like objects. '\n 'Got a {}.'.format(type(images)))\n\n if not len(images) > 0:\n raise ValueError('Expected an iterable (list or tuple) of strings or img-like objects '\n 'of size higher than 0. Got {} items.'.format(len(images)))\n\n if labels is not None and len(labels) != len(images):\n raise ValueError('Expected the same length for image set ({}) and '\n 'labels list ({}).'.format(len(images), len(labels)))\n\n first_file = images[0]\n if first_file:\n first_img = NeuroImage(first_file)\n else:\n raise('Error reading image {}.'.format(repr_imgs(first_file)))\n\n for idx, image in enumerate(images):\n try:\n img = NeuroImage(image)\n self.check_compatibility(img, first_img)\n except:\n log.exception('Error reading image {}.'.format(repr_imgs(image)))\n raise\n else:\n self.items.append(img)\n\n self.set_labels(labels)", "response": "Read the images and labels into self. items and set the labels."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns numpy. ndarray with the masked or flatten image data and the relevant information.", "response": "def to_matrix(self, smooth_fwhm=0, outdtype=None):\n \"\"\"Return numpy.ndarray with the masked or flatten image data and\n the relevant information (mask indices and volume shape).\n\n Parameters\n ----------\n smooth__fwhm: int\n Integer indicating the size of the FWHM Gaussian smoothing kernel\n to smooth the subject volumes before creating the data matrix\n\n outdtype: dtype\n Type of the elements of the array, if None will obtain the dtype from\n the first nifti file.\n\n Returns\n -------\n outmat, mask_indices, vol_shape\n\n outmat: Numpy array with shape N x prod(vol.shape)\n containing the N files as flat vectors.\n\n mask_indices: matrix with indices of the voxels in the mask\n\n vol_shape: Tuple with shape of the volumes, for reshaping.\n \"\"\"\n if not self.all_compatible:\n raise ValueError(\"`self.all_compatible` must be True in order to use this function.\")\n\n if not outdtype:\n outdtype = self.items[0].dtype\n\n # extract some info from the mask\n n_voxels = None\n mask_indices = None\n mask_shape = self.items[0].shape[:3]\n if self.has_mask:\n mask_arr = self.mask.get_data()\n mask_indices = np.nonzero(mask_arr)\n mask_shape = self.mask.shape\n n_voxels = np.count_nonzero(mask_arr)\n\n # if the mask is empty will use the whole image\n if n_voxels is None:\n log.debug('Non-zero voxels have not been found in mask {}'.format(self.mask))\n n_voxels = np.prod(mask_shape)\n mask_indices = None\n\n # get the shape of the flattened subject data\n ndims = self.items[0].ndim\n if ndims == 3:\n subj_flat_shape = (n_voxels, )\n elif ndims == 4:\n subj_flat_shape = (n_voxels, self.items[0].shape[3])\n else:\n raise NotImplementedError('The subject images have {} dimensions. '\n 'Still have not implemented t_matrix for this shape.'.format(ndims))\n\n # create and fill the big matrix\n outmat = np.zeros((self.n_subjs, ) + subj_flat_shape, dtype=outdtype)\n try:\n for i, image in enumerate(self.items):\n if smooth_fwhm > 0:\n image.fwhm = smooth_fwhm\n\n if self.has_mask:\n image.set_mask(self.mask)\n\n outmat[i, :], _, _ = image.mask_and_flatten()\n image.clear_data()\n\n except Exception as exc:\n raise Exception('Error flattening file {0}'.format(image)) from exc\n else:\n return outmat, mask_indices, mask_shape"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_file(self, output_file, smooth_fwhm=0, outdtype=None):\n outmat, mask_indices, mask_shape = self.to_matrix(smooth_fwhm, outdtype)\n\n exporter = ExportData()\n content = {'data': outmat,\n 'labels': self.labels,\n 'mask_indices': mask_indices,\n 'mask_shape': mask_shape, }\n\n if self.others:\n content.update(self.others)\n\n log.debug('Creating content in file {}.'.format(output_file))\n try:\n exporter.save_variables(output_file, content)\n except Exception as exc:\n raise Exception('Error saving variables to file {}.'.format(output_file)) from exc", "response": "Save the Numpy array created from to_matrix function to the output_file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _init_subj_data(self, subj_files):\n try:\n if isinstance(subj_files, list):\n self.from_list(subj_files)\n\n elif isinstance(subj_files, dict):\n self.from_dict(subj_files)\n else:\n raise ValueError('Could not recognize subj_files argument variable type.')\n except Exception as exc:\n raise Exception('Cannot read subj_files input argument.') from exc", "response": "Initialize the object from a list of subj_files."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _load_image(file_path):\n if not os.path.exists(file_path):\n raise FileNotFound(file_path)\n\n try:\n nii_img = load_nipy_img(file_path)\n nii_img.file_path = file_path\n return nii_img\n except Exception as exc:\n raise Exception('Reading file {0}.'.format(file_path)) from exc", "response": "Loads a nipy. Image from a file and returns a nipy. Image object with a file_path member."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsmoothing the image using LinearFilter", "response": "def _smooth_img(nii_img, smooth_fwhm):\n \"\"\"\n Parameters\n ----------\n nii_img: nipy.Image\n\n smooth_fwhm: float\n\n Returns\n -------\n smoothed nipy.Image\n \"\"\"\n # delayed import because could not install nipy on Python 3 on OSX\n from nipy.algorithms.kernel_smooth import LinearFilter\n\n if smooth_fwhm <= 0:\n return nii_img\n\n filter = LinearFilter(nii_img.coordmap, nii_img.shape)\n return filter.smooth(nii_img)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads items and labels from a dictionary of str files.", "response": "def from_dict(self, subj_files):\n \"\"\"\n Parameters\n ----------\n subj_files: dict of str\n file_path -> int/str\n \"\"\"\n for group_label in subj_files:\n try:\n group_files = subj_files[group_label]\n self.items.extend([self._load_image(get_abspath(imgf)) for imgf in group_files])\n\n self.labels.extend([group_label]*len(group_files))\n\n except Exception as exc:\n raise Exception('Error while reading files from '\n 'group {0}.'.format(group_label)) from exc"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_list(self, subj_files):\n for sf in subj_files:\n try:\n nii_img = self._load_image(get_abspath(sf))\n self.items.append(nii_img)\n except Exception as exc:\n raise Exception('Error while reading file {0}.'.format(sf)) from exc", "response": "Reads a list of file paths and adds them to the internal list of items."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_labels(self, subj_labels):\n if len(subj_labels) != self.n_subjs:\n raise ValueError('The number of given labels is not the same as the number of subjects.')\n\n self.labels = subj_labels", "response": "Sets the labels of the items in the log file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a Numpy array with the data and return the relevant information.", "response": "def to_matrix(self, smooth_fwhm=0, outdtype=None):\n \"\"\"Create a Numpy array with the data and return the relevant information (mask indices and volume shape).\n\n Parameters\n ----------\n smooth_fwhm: int\n Integer indicating the size of the FWHM Gaussian smoothing kernel\n to smooth the subject volumes before creating the data matrix\n\n outdtype: dtype\n Type of the elements of the array, if None will obtain the dtype from\n the first nifti file.\n\n Returns\n -------\n outmat, mask_indices, vol_shape\n\n outmat: Numpy array with shape N x prod(vol.shape)\n containing the N files as flat vectors.\n\n mask_indices: matrix with indices of the voxels in the mask\n\n vol_shape: Tuple with shape of the volumes, for reshaping.\n \"\"\"\n\n vol = self.items[0].get_data()\n if not outdtype:\n outdtype = vol.dtype\n\n n_voxels = None\n mask_indices = None\n mask_shape = self.items[0].shape\n\n if self.has_mask:\n mask_arr = get_img_data(self.mask_file)\n mask_indices = np.where(mask_arr > 0)\n mask_shape = mask_arr.shape\n n_voxels = np.count_nonzero(mask_arr)\n\n if n_voxels is None:\n log.debug('Non-zero voxels have not been found in mask {}'.format(self.mask_file))\n n_voxels = np.prod(vol.shape)\n\n outmat = np.zeros((self.n_subjs, n_voxels), dtype=outdtype)\n try:\n for i, nipy_img in enumerate(self.items):\n vol = self._smooth_img(nipy_img, smooth_fwhm).get_data()\n if self.has_mask is not None:\n outmat[i, :] = vol[mask_indices]\n else:\n outmat[i, :] = vol.flatten()\n except Exception as exc:\n raise Exception('Error when flattening file {0}'.format(nipy_img.file_path)) from exc\n else:\n return outmat, mask_indices, mask_shape"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting msg to stderr and exits with return code", "response": "def die(msg, code=-1):\n \"\"\"Writes msg to stderr and exits with return code\"\"\"\n sys.stderr.write(msg + \"\\n\")\n sys.exit(code)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_call(cmd_args):\n p = subprocess.Popen(cmd_args, stdout=subprocess.PIPE)\n (output, err) = p.communicate()\n return output", "response": "Calls the command in a list of command line arguments and returns the output of the command."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef call_command(cmd_name, args_strings):\n if not op.isabs(cmd_name):\n cmd_fullpath = which(cmd_name)\n else:\n cmd_fullpath = cmd_name\n\n try:\n cmd_line = [cmd_fullpath] + args_strings\n\n log.info('Calling: {}.'.format(cmd_line))\n retval = subprocess.check_call(cmd_line)\n except CalledProcessError as ce:\n log.exception(\"Error calling command {} with arguments: \"\n \"{} \\n With return code: {}\".format(cmd_name, args_strings,\n ce.returncode))\n raise\n else:\n return retval", "response": "Call a command with arguments and returns its return value."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsubmit cmd to HTCondor and calls subprocess. call.", "response": "def condor_call(cmd, shell=True):\n \"\"\"\n Tries to submit cmd to HTCondor, if it does not succeed, it will\n be called with subprocess.call.\n\n Parameters\n ----------\n cmd: string\n Command to be submitted\n\n Returns\n -------\n \"\"\"\n log.info(cmd)\n ret = condor_submit(cmd)\n if ret != 0:\n subprocess.call(cmd, shell=shell)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef condor_submit(cmd):\n is_running = subprocess.call('condor_status', shell=True) == 0\n if not is_running:\n raise CalledProcessError('HTCondor is not running.')\n\n sub_cmd = 'condor_qsub -shell n -b y -r y -N ' \\\n + cmd.split()[0] + ' -m n'\n\n log.info('Calling: ' + sub_cmd)\n\n return subprocess.call(sub_cmd + ' ' + cmd, shell=True)", "response": "Submits cmd to HTCondor queue\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncleaning previously built package artifacts.", "response": "def clean(ctx):\n \"\"\"Clean previously built package artifacts.\n \"\"\"\n ctx.run(f'python setup.py clean')\n dist = ROOT.joinpath('dist')\n print(f'removing {dist}')\n shutil.rmtree(str(dist))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef upload(ctx, repo):\n artifacts = ' '.join(\n shlex.quote(str(n))\n for n in ROOT.joinpath('dist').glob('pipfile[-_]cli-*')\n )\n ctx.run(f'twine upload --repository=\"{repo}\" {artifacts}')", "response": "Upload the package to an index server."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_command_table(self, args): #pylint: disable=too-many-statements\n\n # Need an empty client for the select and upload operations\n with CommandSuperGroup(__name__, self,\n 'rcctl.custom_cluster#{}') as super_group:\n with super_group.group('cluster') as group:\n group.command('select', 'select')\n\n with CommandSuperGroup(__name__, self, 'rcctl.custom_reliablecollections#{}',\n client_factory=client_create) as super_group: \n with super_group.group('dictionary') as group:\n group.command('query', 'query_reliabledictionary')\n group.command('execute', 'execute_reliabledictionary')\n group.command('schema', 'get_reliabledictionary_schema')\n group.command('list', 'get_reliabledictionary_list')\n group.command('type-schema', 'get_reliabledictionary_type_schema')\n\n with ArgumentsContext(self, 'dictionary') as ac:\n ac.argument('application_name', options_list=['--application-name', '-a'])\n ac.argument('service_name', options_list=['--service-name', '-s'])\n ac.argument('dictionary_name', options_list=['--dictionary-name', '-d'])\n ac.argument('output_file', options_list=['--output-file', '-out'])\n ac.argument('input_file', options_list=['--input-file', '-in'])\n ac.argument('query_string', options_list=['--query-string', '-q'])\n ac.argument('type_name', options_list=['--type-name', '-t'])\n \n return OrderedDict(self.command_table)", "response": "Load all Service Fabric commands and return a dictionary of all commands that can be executed."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef open_volume_file(filepath):\n # check if the file exists\n if not op.exists(filepath):\n raise IOError('Could not find file {}.'.format(filepath))\n\n # define helper functions\n def open_nifti_file(filepath):\n return NiftiImage(filepath)\n\n def open_mhd_file(filepath):\n return MedicalImage(filepath)\n vol_data, hdr_data = load_raw_data_with_mhd(filepath)\n # TODO: convert vol_data and hdr_data into MedicalImage\n return vol_data, hdr_data\n\n def open_mha_file(filepath):\n raise NotImplementedError('This function has not been implemented yet.')\n\n # generic loader function\n def _load_file(filepath, loader):\n return loader(filepath)\n\n # file_extension -> file loader function\n filext_loader = {\n 'nii': open_nifti_file,\n 'mhd': open_mhd_file,\n 'mha': open_mha_file,\n }\n\n # get extension of the `filepath`\n ext = get_extension(filepath)\n\n # find the loader from `ext`\n loader = None\n for e in filext_loader:\n if ext in e:\n loader = filext_loader[e]\n\n if loader is None:\n raise ValueError('Could not find a loader for file {}.'.format(filepath))\n\n return _load_file(filepath, loader)", "response": "Open a nifti image and medical image file and return the data and header data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck that the input image is a proper img.", "response": "def _check_medimg(image, make_it_3d=True):\n \"\"\"Check that image is a proper img. Turn filenames into objects.\n\n Parameters\n ----------\n image: img-like object or str\n Can either be:\n - a file path to a medical image file, e.g. NifTI, .mhd/raw, .mha\n - any object with get_data() method and affine & header attributes, e.g., nibabel.Nifti1Image.\n - a Numpy array, which will be wrapped by a nibabel.Nifti2Image class with an `eye` affine.\n If niimg is a string, consider it as a path to Nifti image and\n call nibabel.load on it. If it is an object, check if get_data()\n and get_affine() methods are present, raise TypeError otherwise.\n\n make_it_3d: boolean, optional\n If True, check if the image is a 3D image and raise an error if not.\n\n Returns\n -------\n result: nifti-like\n result can be nibabel.Nifti1Image or the input, as-is. It is guaranteed\n that the returned object has get_data() and get_affine() methods.\n \"\"\"\n if isinstance(image, string_types):\n # a filename, load it\n img = open_volume_file(image)\n\n if make_it_3d:\n img = _make_it_3d(img)\n\n return img\n\n elif isinstance(image, np.array):\n return nib.Nifti2Image(image, affine=np.eye(image.ndim + 1))\n\n elif isinstance(image, nib.Nifti1Image) or is_img(image):\n return image\n\n else:\n raise TypeError('Data given cannot be converted to a medical image'\n ' image: this object -\"{}\"- does not have'\n ' get_data or get_affine methods'.format(type(image)))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch for dicoms in folders and save file paths into self. dicom_paths set.", "response": "def _store_dicom_paths(self, folders):\n \"\"\"Search for dicoms in folders and save file paths into\n self.dicom_paths set.\n\n :param folders: str or list of str\n \"\"\"\n if isinstance(folders, str):\n folders = [folders]\n\n for folder in folders:\n\n if not os.path.exists(folder):\n raise FolderNotFound(folder)\n\n self.items.extend(list(find_all_dicom_files(folder)))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_set(self, fileset, check_if_dicoms=True):\n if check_if_dicoms:\n self.items = []\n for f in fileset:\n if is_dicom_file(f):\n self.items.append(f)\n else:\n self.items = fileset", "response": "Overwrites self. items with the given set of files."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update(self, dicomset):\n if not isinstance(dicomset, DicomFileSet):\n raise ValueError('Given dicomset is not a DicomFileSet.')\n\n self.items = list(set(self.items).update(dicomset))", "response": "Update this set with the union of itself and dicomset."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncopies all files within this set to the output_folder.", "response": "def copy_files_to_other_folder(self, output_folder, rename_files=True,\n mkdir=True, verbose=False):\n \"\"\"\n Copies all files within this set to the output_folder\n\n Parameters\n ----------\n output_folder: str\n Path of the destination folder of the files\n\n rename_files: bool\n Whether or not rename the files to a sequential format\n\n mkdir: bool\n Whether to make the folder if it does not exist\n\n verbose: bool\n Whether to print to stdout the files that are beind copied\n \"\"\"\n import shutil\n\n if not os.path.exists(output_folder):\n os.mkdir(output_folder)\n\n if not rename_files:\n for dcmf in self.items:\n outf = os.path.join(output_folder, os.path.basename(dcmf))\n if verbose:\n print('{} -> {}'.format(dcmf, outf))\n shutil.copyfile(dcmf, outf)\n else:\n n_pad = len(self.items)+2\n for idx, dcmf in enumerate(self.items):\n outf = '{number:0{width}d}.dcm'.format(width=n_pad, number=idx)\n outf = os.path.join(output_folder, outf)\n if verbose:\n print('{} -> {}'.format(dcmf, outf))\n shutil.copyfile(dcmf, outf)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a lambda function to read DICOM files.", "response": "def get_dcm_reader(store_metadata=True, header_fields=None):\n \"\"\"\n Creates a lambda function to read DICOM files.\n If store_store_metadata is False, will only return the file path.\n Else if you give header_fields, will return only the set of of\n header_fields within a DicomFile object or the whole DICOM file if\n None.\n\n :return: function\n This function has only one parameter: file_path\n \"\"\"\n if not store_metadata:\n return lambda fpath: fpath\n\n if header_fields is None:\n build_dcm = lambda fpath: DicomFile(fpath)\n else:\n dicom_header = namedtuple('DicomHeader', header_fields)\n build_dcm = lambda fpath: dicom_header._make(DicomFile(fpath).get_attributes(header_fields))\n\n return build_dcm"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef scrape_all_files(self):\n try:\n for dcmf in self.items:\n yield self.read_dcm(dcmf)\n except IOError as ioe:\n raise IOError('Error reading DICOM file: {}.'.format(dcmf)) from ioe", "response": "Generator that yields one by one the return value for self. read_dcm\n for each DICOM file within this set"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_unique_field_values(dcm_file_list, field_name):\n field_values = set()\n\n for dcm in dcm_file_list:\n field_values.add(str(DicomFile(dcm).get_attributes(field_name)))\n\n return field_values", "response": "Returns a set of unique field values from a list of DICOM files"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef find_all_dicom_files(root_path):\n dicoms = set()\n\n try:\n for fpath in get_all_files(root_path):\n if is_dicom_file(fpath):\n dicoms.add(fpath)\n except IOError as ioe:\n raise IOError('Error reading file {0}.'.format(fpath)) from ioe\n\n return dicoms", "response": "Returns a list of the dicom files within root_path"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if the file exists and returns True if it exists and False otherwise.", "response": "def is_dicom_file(filepath):\n \"\"\"\n Tries to read the file using dicom.read_file,\n if the file exists and dicom.read_file does not raise\n and Exception returns True. False otherwise.\n\n :param filepath: str\n Path to DICOM file\n\n :return: bool\n \"\"\"\n if not os.path.exists(filepath):\n raise IOError('File {} not found.'.format(filepath))\n\n filename = os.path.basename(filepath)\n if filename == 'DICOMDIR':\n return False\n\n try:\n _ = dicom.read_file(filepath)\n except Exception as exc:\n log.debug('Checking if {0} was a DICOM, but returned '\n 'False.'.format(filepath))\n return False\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef group_dicom_files(dicom_paths, hdr_field='PatientID'):\n dicom_groups = defaultdict(list)\n try:\n for dcm in dicom_paths:\n hdr = dicom.read_file(dcm)\n group_key = getattr(hdr, hdr_field)\n dicom_groups[group_key].append(dcm)\n except KeyError as ke:\n raise KeyError('Error reading field {} from file {}.'.format(hdr_field, dcm)) from ke\n\n return dicom_groups", "response": "Group in a dictionary all the DICOM files in dicom_paths\n separated by hdr_field."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndecompress all DICOM files in input_dir and return a list of DICOM files in output_dir.", "response": "def decompress(input_dir, dcm_pattern='*.dcm'):\n \"\"\" Decompress all *.dcm files recursively found in DICOM_DIR.\n This uses 'gdcmconv --raw'.\n It works when 'dcm2nii' shows the `Unsupported Transfer Syntax` error. This error is\n usually caused by lack of JPEG2000 support in dcm2nii compilation.\n\n Read more:\n http://www.nitrc.org/plugins/mwiki/index.php/dcm2nii:MainPage#Transfer_Syntaxes_and_Compressed_Images\n\n Parameters\n ----------\n input_dir: str\n Folder path\n\n dcm_patther: str\n Pattern of the DICOM file names in `input_dir`.\n\n Notes\n -----\n The *.dcm files in `input_folder` will be overwritten.\n \"\"\"\n dcmfiles = sorted(recursive_glob(input_dir, dcm_pattern))\n for dcm in dcmfiles:\n cmd = 'gdcmconv --raw -i \"{0}\" -o \"{0}\"'.format(dcm)\n log.debug('Calling {}.'.format(cmd))\n subprocess.check_call(cmd, shell=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_attributes(self, attributes, default=''):\n if isinstance(attributes, str):\n attributes = [attributes]\n\n attrs = [getattr(self, attr, default) for attr in attributes]\n\n if len(attrs) == 1:\n return attrs[0]\n\n return tuple(attrs)", "response": "Returns the values of the attributes values from this DicomFile."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a new object containing all images in the given axis.", "response": "def merge_images(images, axis='t'):\n \"\"\" Concatenate `images` in the direction determined in `axis`.\n\n Parameters\n ----------\n images: list of str or img-like object.\n See NeuroImage constructor docstring.\n\n axis: str\n 't' : concatenate images in time\n 'x' : concatenate images in the x direction\n 'y' : concatenate images in the y direction\n 'z' : concatenate images in the z direction\n\n Returns\n -------\n merged: img-like object\n \"\"\"\n # check if images is not empty\n if not images:\n return None\n\n # the given axis name to axis idx\n axis_dim = {'x': 0,\n 'y': 1,\n 'z': 2,\n 't': 3,\n }\n\n # check if the given axis name is valid\n if axis not in axis_dim:\n raise ValueError('Expected `axis` to be one of ({}), got {}.'.format(set(axis_dim.keys()), axis))\n\n # check if all images are compatible with each other\n img1 = images[0]\n for img in images:\n check_img_compatibility(img1, img)\n\n # read the data of all the given images\n # TODO: optimize memory consumption by merging one by one.\n image_data = []\n for img in images:\n image_data.append(check_img(img).get_data())\n\n # if the work_axis is bigger than the number of axis of the images,\n # create a new axis for the images\n work_axis = axis_dim[axis]\n ndim = image_data[0].ndim\n if ndim - 1 < work_axis:\n image_data = [np.expand_dims(img, axis=work_axis) for img in image_data]\n\n # concatenate and return\n return np.concatenate(image_data, axis=work_axis)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef nifti_out(f):\n @wraps(f)\n def wrapped(*args, **kwargs):\n r = f(*args, **kwargs)\n\n img = read_img(args[0])\n return nib.Nifti1Image(r, affine=img.get_affine(), header=img.header)\n\n return wrapped", "response": "Takes a function whose first argument is an img processes its\n data and returns a numpy array. This decorator takes a numpy array containing the image data and returns a nibabel. Nifti1Image."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef thr_img(img, thr=2., mode='+'):\n\n vol = read_img(img).get_data()\n\n if mode == '+':\n mask = vol > thr\n elif mode == '+-' or mode == '-+':\n mask = np.abs(vol) > thr\n elif mode == '-':\n mask = vol < -thr\n else:\n raise ValueError(\"Expected `mode` to be one of ('+', '+-', '-+', '-'), \"\n \"got {}.\".format(mode))\n\n return vol * mask", "response": "This function returns a new image that is threshold with value thr."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef div_img(img1, div2):\n if is_img(div2):\n return img1.get_data()/div2.get_data()\n elif isinstance(div2, (float, int)):\n return img1.get_data()/div2\n else:\n raise NotImplementedError('Cannot divide {}({}) by '\n '{}({})'.format(type(img1),\n img1,\n type(div2),\n div2))", "response": "Pixelwise division or divide by a number"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef apply_mask(img, mask):\n from .mask import apply_mask\n\n vol, _ = apply_mask(img, mask)\n return vector_to_volume(vol, read_img(mask).get_data().astype(bool))", "response": "Return the image with the given mask applied."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn an image with the binarised version of the data of img.", "response": "def abs_img(img):\n \"\"\" Return an image with the binarised version of the data of `img`.\"\"\"\n bool_img = np.abs(read_img(img).get_data())\n return bool_img.astype(int)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a z - scored version of icc.", "response": "def icc_img_to_zscore(icc, center_image=False):\n \"\"\" Return a z-scored version of `icc`.\n This function is based on GIFT `icatb_convertImageToZScores` function.\n \"\"\"\n vol = read_img(icc).get_data()\n\n v2 = vol[vol != 0]\n if center_image:\n v2 = detrend(v2, axis=0)\n\n vstd = np.linalg.norm(v2, ord=2) / np.sqrt(np.prod(v2.shape) - 1)\n\n eps = np.finfo(vstd.dtype).eps\n vol /= (eps + vstd)\n\n return vol"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the thresholded z - scored icc.", "response": "def spatial_map(icc, thr, mode='+'):\n \"\"\" Return the thresholded z-scored `icc`. \"\"\"\n return thr_img(icc_img_to_zscore(icc), thr=thr, mode=mode).get_data()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter_icc(icc, mask=None, thr=2, zscore=True, mode=\"+\"):\n if zscore:\n icc_filt = thr_img(icc_img_to_zscore(icc), thr=thr, mode=mode)\n else:\n icc_filt = thr_img(icc, thr=thr, mode=mode)\n\n if mask is not None:\n icc_filt = apply_mask(icc_filt, mask)\n\n return icc_filt", "response": "Threshold then mask an IC correlation map."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef check_mhd_img(image, make_it_3d=False):\n if isinstance(image, string_types):\n # a filename, load it\n if not op.exists(image):\n raise FileNotFound(image)\n\n ext = get_extension(image).lower()\n if not 'mhd' in ext:\n warnings.warn('Expecting a filepath with `.mhd` extension, got {}.'.format(image))\n\n img, hdr = load_raw_data_with_mhd(image)\n if make_it_3d:\n img = _make_it_3d(img)\n\n return img\n\n elif is_img(image):\n return image\n\n else:\n raise TypeError('Data given cannot be converted to a nifti'\n ' image: this object -\"{}\"- does not have'\n ' get_data or get_affine methods'.format(type(image)))", "response": "Checks that the given image is a proper img."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nenforce that img is a 3D img - like object raise a TypeError.", "response": "def _make_it_3d(img):\n \"\"\"Enforce that img is a 3D img-like object, if it is not, raise a TypeError.\n i.e., remove dimensions of size 1.\n\n Parameters\n ----------\n img: numpy.ndarray\n Image data array\n\n Returns\n -------\n 3D numpy ndarray object\n \"\"\"\n shape = img.shape\n\n if len(shape) == 3:\n return img\n\n elif len(shape) == 4 and shape[3] == 1:\n # \"squeeze\" the image.\n return img[:, :, :, 0]\n\n else:\n raise TypeError('A 3D image is expected, but an image with a shape of {} was given.'.format(shape))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite the content of the meta_dict into filename.", "response": "def write_meta_header(filename, meta_dict):\n \"\"\" Write the content of the `meta_dict` into `filename`.\n\n Parameters\n ----------\n filename: str\n Path to the output file\n\n meta_dict: dict\n Dictionary with the fields of the metadata .mhd file\n \"\"\"\n header = ''\n # do not use tags = meta_dict.keys() because the order of tags matters\n for tag in MHD_TAGS:\n if tag in meta_dict.keys():\n header += '{} = {}\\n'.format(tag, meta_dict[tag])\n\n with open(filename, 'w') as f:\n f.write(header)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting the data into a raw format file.", "response": "def dump_raw_data(filename, data):\n \"\"\" Write the data into a raw format file. Big endian is always used.\n\n Parameters\n ----------\n filename: str\n Path to the output file\n\n data: numpy.ndarray\n n-dimensional image data array.\n \"\"\"\n if data.ndim == 3:\n # Begin 3D fix\n data = data.reshape([data.shape[0], data.shape[1]*data.shape[2]])\n # End 3D fix\n\n a = array.array('f')\n for o in data:\n a.fromlist(list(o.flatten()))\n\n # if is_little_endian():\n # a.byteswap()\n\n with open(filename, 'wb') as rawf:\n a.tofile(rawf)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write_mhd_file(filename, data, shape=None, meta_dict=None):\n # check its extension\n ext = get_extension(filename)\n fname = op.basename(filename)\n if ext != '.mhd' or ext != '.raw':\n mhd_filename = fname + '.mhd'\n raw_filename = fname + '.raw'\n elif ext == '.mhd':\n mhd_filename = fname\n raw_filename = remove_ext(fname) + '.raw'\n elif ext == '.raw':\n mhd_filename = remove_ext(fname) + '.mhd'\n raw_filename = fname\n else:\n raise ValueError('`filename` extension {} from {} is not recognised. '\n 'Expected .mhd or .raw.'.format(ext, filename))\n\n # default values\n if meta_dict is None:\n meta_dict = {}\n\n if shape is None:\n shape = data.shape\n\n # prepare the default header\n meta_dict['ObjectType'] = meta_dict.get('ObjectType', 'Image')\n meta_dict['BinaryData'] = meta_dict.get('BinaryData', 'True' )\n meta_dict['BinaryDataByteOrderMSB'] = meta_dict.get('BinaryDataByteOrderMSB', 'False')\n meta_dict['ElementType'] = meta_dict.get('ElementType', NUMPY_TO_MHD_TYPE[data.dtype.type])\n meta_dict['NDims'] = meta_dict.get('NDims', str(len(shape)))\n meta_dict['DimSize'] = meta_dict.get('DimSize', ' '.join([str(i) for i in shape]))\n meta_dict['ElementDataFile'] = meta_dict.get('ElementDataFile', raw_filename)\n\n # target files\n mhd_filename = op.join(op.dirname(filename), mhd_filename)\n raw_filename = op.join(op.dirname(filename), raw_filename)\n\n # write the header\n write_meta_header(mhd_filename, meta_dict)\n\n # write the data\n dump_raw_data(raw_filename, data)\n\n return mhd_filename, raw_filename", "response": "Write the data and meta_dict to a. mhd file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncopy the mhd and raw files to dst.", "response": "def copy_mhd_and_raw(src, dst):\n \"\"\"Copy .mhd and .raw files to dst.\n\n If dst is a folder, won't change the file, but if dst is another filepath,\n will modify the ElementDataFile field in the .mhd to point to the\n new renamed .raw file.\n\n Parameters\n ----------\n src: str\n Path to the .mhd file to be copied\n\n dst: str\n Path to the destination of the .mhd and .raw files.\n If a new file name is given, the extension will be ignored.\n\n Returns\n -------\n dst: str\n \"\"\"\n # check if src exists\n if not op.exists(src):\n raise IOError('Could not find file {}.'.format(src))\n\n # check its extension\n ext = get_extension(src)\n if ext != '.mhd':\n msg = 'The src file path must be a .mhd file. Given: {}.'.format(src)\n raise ValueError(msg)\n\n # get the raw file for this src mhd file\n meta_src = _read_meta_header(src)\n\n # get the source raw file\n src_raw = meta_src['ElementDataFile']\n if not op.isabs(src_raw):\n src_raw = op.join(op.dirname(src), src_raw)\n\n # check if dst is dir\n if op.isdir(dst):\n # copy the mhd and raw file to its destiny\n shutil.copyfile(src, dst)\n shutil.copyfile(src_raw, dst)\n return dst\n\n # build raw file dst file name\n dst_raw = op.join(op.dirname(dst), remove_ext(op.basename(dst))) + '.raw'\n\n # add extension to the dst path\n if get_extension(dst) != '.mhd':\n dst += '.mhd'\n\n # copy the mhd and raw file to its destiny\n log.debug('cp: {} -> {}'.format(src, dst))\n log.debug('cp: {} -> {}'.format(src_raw, dst_raw))\n shutil.copyfile(src, dst)\n shutil.copyfile(src_raw, dst_raw)\n\n # check if src file name is different than dst file name\n # if not the same file name, change the content of the ElementDataFile field\n if op.basename(dst) != op.basename(src):\n log.debug('modify {}: ElementDataFile: {} -> {}'.format(dst, src_raw,\n op.basename(dst_raw)))\n meta_dst = _read_meta_header(dst)\n meta_dst['ElementDataFile'] = op.basename(dst_raw)\n write_meta_header(dst, meta_dst)\n\n return dst"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sav_to_pandas_rpy2(input_file):\n import pandas.rpy.common as com\n\n w = com.robj.r('foreign::read.spss(\"%s\", to.data.frame=TRUE)' % input_file)\n return com.convert_robj(w)", "response": "SPSS. sav files to Pandas DataFrame through Rpy2\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading the SPSS. sav files to Pandas DataFrame through savreader module", "response": "def sav_to_pandas_savreader(input_file):\n \"\"\"\n SPSS .sav files to Pandas DataFrame through savreader module\n\n :param input_file: string\n\n :return:\n \"\"\"\n from savReaderWriter import SavReader\n lines = []\n with SavReader(input_file, returnHeader=True) as reader:\n header = next(reader)\n for line in reader:\n lines.append(line)\n\n return pd.DataFrame(data=lines, columns=header)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save_variables(filename, variables):\n ext = get_extension(filename).lower()\n out_exts = {'.pyshelf', '.shelf', '.mat', '.hdf5', '.h5'}\n\n output_file = filename\n if not ext in out_exts:\n output_file = add_extension_if_needed(filename, '.pyshelf')\n ext = get_extension(filename)\n\n if ext == '.pyshelf' or ext == '.shelf':\n save_variables_to_shelve(output_file, variables)\n\n elif ext == '.mat':\n save_variables_to_mat(output_file, variables)\n\n elif ext == '.hdf5' or ext == '.h5':\n from .hdf5 import save_variables_to_hdf5\n save_variables_to_hdf5(output_file, variables)\n\n else:\n raise ValueError('Filename extension {0} not accepted.'.format(ext))", "response": "Save given variables in a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving the variables in the specified file.", "response": "def save_varlist(filename, varnames, varlist):\n \"\"\"\n Valid extensions '.pyshelf', '.mat', '.hdf5' or '.h5'\n\n @param filename: string\n\n @param varnames: list of strings\n Names of the variables\n\n @param varlist: list of objects\n The objects to be saved\n \"\"\"\n variables = {}\n for i, vn in enumerate(varnames):\n variables[vn] = varlist[i]\n\n ExportData.save_variables(filename, variables)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef drain_rois(img):\n img_data = get_img_data(img)\n\n out = np.zeros(img_data.shape, dtype=img_data.dtype)\n\n krn_dim = [3] * img_data.ndim\n kernel = np.ones(krn_dim, dtype=int)\n\n vals = np.unique(img_data)\n vals = vals[vals != 0]\n\n for i in vals:\n roi = img_data == i\n hits = scn.binary_hit_or_miss(roi, kernel)\n roi[hits] = 0\n out[roi > 0] = i\n\n return out", "response": "Return a similar volume with the ROIs emptied."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pick_rois(rois_img, roi_values, bg_val=0):\n img = read_img(rois_img)\n img_data = img.get_data()\n\n if bg_val == 0:\n out = np.zeros(img_data.shape, dtype=img_data.dtype)\n else:\n out = np.ones(img_data.shape, dtype=img_data.dtype) * bg_val\n\n for r in roi_values:\n out[img_data == r] = r\n\n return nib.Nifti2Image(out, affine=img.affine, header=img.header)", "response": "Return the rois_img only with the ROI values from roi_values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef largest_connected_component(volume):\n # We use asarray to be able to work with masked arrays.\n volume = np.asarray(volume)\n labels, num_labels = scn.label(volume)\n if not num_labels:\n raise ValueError('No non-zero values: no connected components found.')\n\n if num_labels == 1:\n return volume.astype(np.bool)\n\n label_count = np.bincount(labels.ravel().astype(np.int))\n # discard the 0 label\n label_count[0] = 0\n return labels == label_count.argmax()", "response": "Return the largest connected component of a 3D array."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef large_clusters_mask(volume, min_cluster_size):\n labels, num_labels = scn.label(volume)\n\n labels_to_keep = set([i for i in range(num_labels)\n if np.sum(labels == i) >= min_cluster_size])\n\n clusters_mask = np.zeros_like(volume, dtype=int)\n for l in range(num_labels):\n if l in labels_to_keep:\n clusters_mask[labels == l] = 1\n\n return clusters_mask", "response": "Return as mask for volume that includes only areas where the connected components have a size bigger than min_cluster_size."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a mask from a list of ROIs in a file list.", "response": "def create_rois_mask(roislist, filelist):\n \"\"\"Look for the files in filelist containing the names in roislist, these files will be opened, binarised\n and merged in one mask.\n\n Parameters\n ----------\n roislist: list of strings\n Names of the ROIs, which will have to be in the names of the files in filelist.\n\n filelist: list of strings\n List of paths to the volume files containing the ROIs.\n\n Returns\n -------\n numpy.ndarray\n Mask volume\n \"\"\"\n roifiles = []\n\n for roi in roislist:\n try:\n roi_file = search_list(roi, filelist)[0]\n except Exception as exc:\n raise Exception('Error creating list of roi files. \\n {}'.format(str(exc)))\n else:\n roifiles.append(roi_file)\n\n return binarise(roifiles)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_unique_nonzeros(arr):\n rois = np.unique(arr)\n rois = rois[np.nonzero(rois)]\n rois.sort()\n\n return rois", "response": "Returns a sorted list of the non - zero unique values of arr."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the center of mass for each ROI in the given volume.", "response": "def get_rois_centers_of_mass(vol):\n \"\"\"Get the center of mass for each ROI in the given volume.\n\n Parameters\n ----------\n vol: numpy ndarray\n Volume with different values for each ROI.\n\n Returns\n -------\n OrderedDict\n Each entry in the dict has the ROI value as key and the center_of_mass coordinate as value.\n \"\"\"\n from scipy.ndimage.measurements import center_of_mass\n\n roisvals = np.unique(vol)\n roisvals = roisvals[roisvals != 0]\n\n rois_centers = OrderedDict()\n for r in roisvals:\n rois_centers[r] = center_of_mass(vol, vol, r)\n\n return rois_centers"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npartition the timeseries in tsvol according to the ROIs in roivol. If a mask is given, will use it to exclude any voxel outside of it. The outdict indicates whether you want a dictionary for each set of timeseries keyed by the ROI value or a list of timeseries sets. If True and roi_img is not None will return an OrderedDict, if False or roi_img or roi_list is None will return a list. Background value is assumed to be 0 and won't be used here. Parameters ---------- image: img-like object or str 4D timeseries volume roi_img: img-like object or str 3D volume defining different ROIs. mask_img: img-like object or str 3D mask volume zeroe: bool If true will remove the null timeseries voxels. roi_values: list of ROI values (int?) List of the values of the ROIs to indicate the order and which ROIs will be processed. outdict: bool If True will return an OrderedDict of timeseries sets, otherwise a list. Returns ------- timeseries: list or OrderedDict A dict with the timeseries as items and keys as the ROIs voxel values or a list where each element is the timeseries set ordered by the sorted values in roi_img or by the roi_values argument.", "response": "def partition_timeseries(image, roi_img, mask_img=None, zeroe=True, roi_values=None, outdict=False):\n \"\"\"Partition the timeseries in tsvol according to the ROIs in roivol.\n If a mask is given, will use it to exclude any voxel outside of it.\n\n The outdict indicates whether you want a dictionary for each set of timeseries keyed by the ROI value\n or a list of timeseries sets. If True and roi_img is not None will return an OrderedDict, if False\n or roi_img or roi_list is None will return a list.\n\n Background value is assumed to be 0 and won't be used here.\n\n Parameters\n ----------\n image: img-like object or str\n 4D timeseries volume\n\n roi_img: img-like object or str\n 3D volume defining different ROIs.\n\n mask_img: img-like object or str\n 3D mask volume\n\n zeroe: bool\n If true will remove the null timeseries voxels.\n\n roi_values: list of ROI values (int?)\n List of the values of the ROIs to indicate the\n order and which ROIs will be processed.\n\n outdict: bool\n If True will return an OrderedDict of timeseries sets, otherwise a list.\n\n Returns\n -------\n timeseries: list or OrderedDict\n A dict with the timeseries as items and keys as the ROIs voxel values or\n a list where each element is the timeseries set ordered by the sorted values in roi_img or by the roi_values\n argument.\n\n \"\"\"\n img = read_img(image)\n rois = read_img(roi_img)\n\n # check if roi_img and image are compatible\n check_img_compatibility(img, rois, only_check_3d=True)\n\n # check if rois has all roi_values\n roi_data = rois.get_data()\n if roi_values is not None:\n for rv in roi_values:\n if not np.any(roi_data == rv):\n raise ValueError('Could not find value {} in rois_img {}.'.format(rv, repr_imgs(roi_img)))\n else:\n roi_values = get_unique_nonzeros(roi_data)\n\n # check if mask and image are compatible\n if mask_img is None:\n mask_data = None\n else:\n mask = load_mask(mask_img)\n check_img_compatibility(img, mask, only_check_3d=True)\n mask_data = mask.get_data()\n\n # choose function to call\n if outdict:\n extract_data = _extract_timeseries_dict\n else:\n extract_data = _extract_timeseries_list\n\n # extract data and return it\n try:\n return extract_data(img.get_data(), rois.get_data(), mask_data,\n roi_values=roi_values, zeroe=zeroe)\n except:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nextracts the values in `datavol` that are in the ROI with value `roivalue` in `roivol`. The ROI can be masked by `maskvol`. Parameters ---------- datavol: numpy.ndarray 4D timeseries volume or a 3D volume to be partitioned roivol: numpy.ndarray 3D ROIs volume roivalue: int or float A value from roivol that represents the ROI to be used for extraction. maskvol: numpy.ndarray 3D mask volume zeroe: bool If true will remove the null timeseries voxels. Only applied to timeseries (4D) data. Returns ------- values: np.array An array of the values in the indicated ROI. A 2D matrix if `datavol` is 4D or a 1D vector if `datavol` is 3D.", "response": "def _partition_data(datavol, roivol, roivalue, maskvol=None, zeroe=True):\n \"\"\" Extracts the values in `datavol` that are in the ROI with value `roivalue` in `roivol`.\n The ROI can be masked by `maskvol`.\n\n Parameters\n ----------\n datavol: numpy.ndarray\n 4D timeseries volume or a 3D volume to be partitioned\n\n roivol: numpy.ndarray\n 3D ROIs volume\n\n roivalue: int or float\n A value from roivol that represents the ROI to be used for extraction.\n\n maskvol: numpy.ndarray\n 3D mask volume\n\n zeroe: bool\n If true will remove the null timeseries voxels. Only applied to timeseries (4D) data.\n\n Returns\n -------\n values: np.array\n An array of the values in the indicated ROI.\n A 2D matrix if `datavol` is 4D or a 1D vector if `datavol` is 3D.\n \"\"\"\n if maskvol is not None:\n # get all masked time series within this roi r\n indices = (roivol == roivalue) * (maskvol > 0)\n else:\n # get all time series within this roi r\n indices = roivol == roivalue\n\n if datavol.ndim == 4:\n ts = datavol[indices, :]\n else:\n ts = datavol[indices]\n\n # remove zeroed time series\n if zeroe:\n if datavol.ndim == 4:\n ts = ts[ts.sum(axis=1) != 0, :]\n\n return ts"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npartitions the timeseries in tsvol according to the ROIs in maskvol.", "response": "def _extract_timeseries_dict(tsvol, roivol, maskvol=None, roi_values=None, zeroe=True):\n \"\"\"Partition the timeseries in tsvol according to the ROIs in roivol.\n If a mask is given, will use it to exclude any voxel outside of it.\n\n Parameters\n ----------\n tsvol: numpy.ndarray\n 4D timeseries volume or a 3D volume to be partitioned\n\n roivol: numpy.ndarray\n 3D ROIs volume\n\n maskvol: numpy.ndarray\n 3D mask volume\n\n zeroe: bool\n If true will remove the null timeseries voxels.\n\n roi_values: list of ROI values (int?)\n List of the values of the ROIs to indicate the\n order and which ROIs will be processed.\n\n Returns\n -------\n ts_dict: OrderedDict\n A dict with the timeseries as items and keys as the ROIs voxel values.\n \"\"\"\n _check_for_partition(tsvol, roivol, maskvol)\n\n # get unique values of the atlas\n if roi_values is None:\n roi_values = get_unique_nonzeros(roivol)\n\n ts_dict = OrderedDict()\n for r in roi_values:\n ts = _partition_data(tsvol, roivol, r, maskvol, zeroe)\n\n if len(ts) == 0:\n ts = np.zeros(tsvol.shape[-1])\n\n ts_dict[r] = ts\n\n return ts_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _extract_timeseries_list(tsvol, roivol, maskvol=None, roi_values=None, zeroe=True):\n _check_for_partition(tsvol, roivol, maskvol)\n\n if roi_values is None:\n roi_values = get_unique_nonzeros(roivol)\n\n ts_list = []\n for r in roi_values:\n ts = _partition_data(tsvol, roivol, r, maskvol, zeroe)\n\n if len(ts) == 0:\n ts = np.zeros(tsvol.shape[-1])\n\n ts_list.append(ts)\n\n return ts_list", "response": "Partition the timeseries in tsvol according to the ROIs in maskvol."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_3D_from_4D(image, vol_idx=0):\n img = check_img(image)\n hdr, aff = get_img_info(img)\n\n if len(img.shape) != 4:\n raise AttributeError('Volume in {} does not have 4 dimensions.'.format(repr_imgs(img)))\n\n if not 0 <= vol_idx < img.shape[3]:\n raise IndexError('IndexError: 4th dimension in volume {} has {} volumes, '\n 'not {}.'.format(repr_imgs(img), img.shape[3], vol_idx))\n\n img_data = img.get_data()\n new_vol = img_data[:, :, :, vol_idx].copy()\n\n hdr.set_data_shape(hdr.get_data_shape()[:3])\n\n return new_vol, hdr, aff", "response": "Pick one 3D volume from a 4D nifti image file."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the HDF5 file for the current object.", "response": "def create_hdf_file(self):\n \"\"\"\n :return: h5py DataSet\n \"\"\"\n mode = 'w'\n if not self._overwrite and os.path.exists(self._fname):\n mode = 'a'\n\n self._hdf_file = h5py.File(self._fname, mode)\n\n if self._hdf_basepath == '/':\n self._group = self._hdf_file['/']\n else:\n self._group = self._hdf_file.create_group(self._hdf_basepath)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a h5py dataset given its registered name.", "response": "def get_dataset(self, ds_name, mode='r'):\n \"\"\"\n Returns a h5py dataset given its registered name.\n\n :param ds_name: string\n Name of the dataset to be returned.\n\n :return:\n \"\"\"\n if ds_name in self._datasets:\n return self._datasets[ds_name]\n else:\n return self.create_empty_dataset(ds_name)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate an empty dataset with unknown size.", "response": "def create_empty_dataset(self, ds_name, dtype=np.float32):\n \"\"\"\n Creates a Dataset with unknown size.\n Resize it before using.\n\n :param ds_name: string\n\n :param dtype: dtype\n Datatype of the dataset\n\n :return: h5py DataSet\n \"\"\"\n if ds_name in self._datasets:\n return self._datasets[ds_name]\n\n ds = self._group.create_dataset(ds_name, (1, 1), maxshape=None,\n dtype=dtype)\n self._datasets[ds_name] = ds\n\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_dataset(self, ds_name, data, attrs=None, dtype=None):\n if ds_name in self._datasets:\n ds = self._datasets[ds_name]\n if ds.dtype != data.dtype:\n warnings.warn('Dataset and data dtype are different!')\n\n else:\n if dtype is None:\n dtype = data.dtype\n\n ds = self._group.create_dataset(ds_name, data.shape,\n dtype=dtype)\n\n if attrs is not None:\n for key in attrs:\n setattr(ds.attrs, key, attrs[key])\n\n ds.read_direct(data)\n self._datasets[ds_name] = ds\n\n return ds", "response": "Creates a new dataset in the HDF file and registers it as a new dataset in the HDF file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self, ds_name, data, dtype=None):\n return self.create_dataset(ds_name, data, dtype)", "response": "Save data to a new dataset."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fill_missing_values(df, range_values, fill_value=0, fill_method=None):\n idx_colnames = df.index.names\n\n idx_colranges = [range_values[x] for x in idx_colnames]\n\n fullindex = pd.Index([p for p in product(*idx_colranges)],\n name=tuple(idx_colnames))\n\n fulldf = df.reindex(index=fullindex, fill_value=fill_value,\n method=fill_method)\n\n fulldf.index.names = idx_colnames\n\n return fulldf, idx_colranges", "response": "This function will fill missing values in the index column of df with the given values."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving pandas object or group of Numpy ndarrays stored in file", "response": "def get(self, key):\n \"\"\"\n Retrieve pandas object or group of Numpy ndarrays\n stored in file\n\n Parameters\n ----------\n key : object\n\n Returns\n -------\n obj : type of object stored in file\n \"\"\"\n node = self.get_node(key)\n if node is None:\n raise KeyError('No object named %s in the file' % key)\n\n if hasattr(node, 'attrs'):\n if 'pandas_type' in node.attrs:\n return self._read_group(node)\n\n return self._read_array(node)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef put(self, key, value, attrs=None, format=None, append=False, **kwargs):\n if not isinstance(value, np.ndarray):\n super(NumpyHDFStore, self).put(key, value, format, append, **kwargs)\n else:\n group = self.get_node(key)\n\n # remove the node if we are not appending\n if group is not None and not append:\n self._handle.removeNode(group, recursive=True)\n group = None\n\n if group is None:\n paths = key.split('/')\n\n # recursively create the groups\n path = '/'\n for p in paths:\n if not len(p):\n continue\n new_path = path\n if not path.endswith('/'):\n new_path += '/'\n new_path += p\n group = self.get_node(new_path)\n if group is None:\n group = self._handle.createGroup(path, p)\n path = new_path\n\n ds_name = kwargs.get('ds_name', self._array_dsname)\n\n ds = self._handle.createArray(group, ds_name, value)\n if attrs is not None:\n for key in attrs:\n setattr(ds.attrs, key, attrs[key])\n\n self._handle.flush()\n\n return ds", "response": "Put a value into the HDFStore."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _push_dfblock(self, key, df, ds_name, range_values):\n #create numpy array and put into hdf_file\n vals_colranges = [range_values[x] for x in df.index.names]\n nu_shape = [len(x) for x in vals_colranges]\n\n return self.put(key, np.reshape(df.values, tuple(nu_shape)),\n attrs={'axes': df.index.names},\n ds_name=ds_name, append=True)", "response": "Push a new block into the hdf_file\n table"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef put_df_as_ndarray(self, key, df, range_values, loop_multiindex=False,\n unstack=False, fill_value=0, fill_method=None):\n \"\"\"Returns a PyTables HDF Array from df in the shape given by its index columns range values.\n\n :param key: string object\n\n :param df: pandas DataFrame\n\n :param range_values: dict or array-like\n Must contain for each index column of df an entry with all the values\n within the range of the column.\n\n :param loop_multiindex: bool\n Will loop through the first index in a multiindex dataframe, extract a\n dataframe only for one value, complete and fill the missing values and\n store in the HDF.\n If this is True, it will not use unstack.\n This is as fast as unstacking.\n\n :param unstack: bool\n Unstack means that this will use the first index name to\n unfold the DataFrame, and will create a group with as many datasets\n as valus has this first index.\n Use this if you think the filled dataframe won't fit in your RAM memory.\n If set to False, this will transform the dataframe in memory first\n and only then save it.\n\n :param fill_value: scalar or 'nearest', default 0\n Value to use for missing values. Defaults to 0, but can be any\n \"compatible\" value, e.g., NaN.\n The 'nearest' mode will fill the missing value with the nearest value in\n the column.\n\n :param fill_method: {'backfill', 'bfill', 'pad', 'ffill', None}, default None\n Method to use for filling holes in reindexed DataFrame\n 'pad' / 'ffill': propagate last valid observation forward to next valid\n 'backfill' / 'bfill': use NEXT valid observation to fill gap\n\n :return: PyTables data node\n \"\"\"\n idx_colnames = df.index.names\n #idx_colranges = [range_values[x] for x in idx_colnames]\n\n #dataset group name if not given\n if key is None:\n key = idx_colnames[0]\n\n if loop_multiindex:\n idx_values = df.index.get_level_values(0).unique()\n\n for idx in idx_values:\n vals, _ = self._fill_missing_values(df.xs((idx,), level=idx_colnames[0]),\n range_values,\n fill_value=fill_value,\n fill_method=fill_method)\n\n ds_name = str(idx) + '_' + '_'.join(vals.columns)\n\n self._push_dfblock(key, vals, ds_name, range_values)\n\n return self._handle.get_node('/' + str(key))\n\n #separate the dataframe into blocks, only with the first index\n else:\n if unstack:\n df = df.unstack(idx_colnames[0])\n for idx in df:\n vals, _ = self._fill_missing_values(df[idx], range_values,\n fill_value=fill_value,\n fill_method=fill_method)\n vals = np.nan_to_num(vals)\n\n ds_name = '_'.join([str(x) for x in vals.name])\n\n self._push_dfblock(key, vals, ds_name, range_values)\n\n return self._handle.get_node('/' + str(key))\n\n #not separate the data\n vals, _ = self._fill_missing_values(df, range_values,\n fill_value=fill_value,\n fill_method=fill_method)\n\n ds_name = self._array_dsname\n\n return self._push_dfblock(key, vals, ds_name, range_values)", "response": "Returns a PyTables HDF Array from a pandas DataFrame"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the data in the image.", "response": "def get_data(self, safe_copy=False):\n \"\"\"Get the data in the image.\n If save_copy is True, will perform a deep copy of the data and return it.\n\n Parameters\n ----------\n smoothed: (optional) bool\n If True and self._smooth_fwhm > 0 will smooth the data before masking.\n\n masked: (optional) bool\n If True and self.has_mask will return the masked data, the plain data otherwise.\n\n safe_copy: (optional) bool\n\n Returns\n -------\n np.ndarray\n \"\"\"\n if safe_copy:\n data = get_data(self.img)\n else:\n data = self.img.get_data(caching=self._caching)\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset a smoothing Gaussian kernel given its FWHM in mm.", "response": "def smooth_fwhm(self, fwhm):\n \"\"\" Set a smoothing Gaussian kernel given its FWHM in mm. \"\"\"\n if fwhm != self._smooth_fwhm:\n self._is_data_smooth = False\n self._smooth_fwhm = fwhm"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the data in the image.", "response": "def get_data(self, smoothed=True, masked=True, safe_copy=False):\n \"\"\"Get the data in the image.\n If save_copy is True, will perform a deep copy of the data and return it.\n\n Parameters\n ----------\n smoothed: (optional) bool\n If True and self._smooth_fwhm > 0 will smooth the data before masking.\n\n masked: (optional) bool\n If True and self.has_mask will return the masked data, the plain data otherwise.\n\n safe_copy: (optional) bool\n\n Returns\n -------\n np.ndarray\n \"\"\"\n if not safe_copy and smoothed == self._is_data_smooth and masked == self._is_data_masked:\n if self.has_data_loaded() and self._caching == 'fill':\n return self.get_data()\n\n if safe_copy:\n data = get_data(self.img)\n else:\n data = self.img.get_data(caching=self._caching)\n\n is_smoothed = False\n if smoothed and self._smooth_fwhm > 0:\n try:\n data = _smooth_data_array(data, self.get_affine(), self._smooth_fwhm, copy=False)\n except ValueError as ve:\n raise ValueError('Error smoothing image {} with a {}mm FWHM '\n 'kernel.'.format(self.img, self._smooth_fwhm)) from ve\n else:\n is_smoothed = True\n\n is_data_masked = False\n if masked and self.has_mask():\n try:\n data = self.unmask(self._mask_data(data)[0])\n except:\n raise\n else:\n is_data_masked = True\n\n if not safe_copy:\n self._is_data_masked = is_data_masked\n self._is_data_smooth = is_smoothed\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef apply_mask(self, mask_img):\n self.set_mask(mask_img)\n return self.get_data(masked=True, smoothed=True, safe_copy=True)", "response": "Set the mask and get_masked_data methods."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_mask(self, mask_img):\n mask = load_mask(mask_img, allow_empty=True)\n check_img_compatibility(self.img, mask, only_check_3d=True) # this will raise an exception if something is wrong\n self.mask = mask", "response": "Sets a mask img to this."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _mask_data(self, data):\n self._check_for_mask()\n\n msk_data = self.mask.get_data()\n if self.ndim == 3:\n return data[msk_data], np.where(msk_data)\n elif self.ndim == 4:\n return _apply_mask_to_4d_data(data, self.mask)\n else:\n raise ValueError('Cannot mask {} with {} dimensions using mask {}.'.format(self, self.ndim, self.mask))", "response": "Return the data masked with self. mask"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef apply_smoothing(self, smooth_fwhm):\n if smooth_fwhm <= 0:\n return\n\n old_smooth_fwhm = self._smooth_fwhm\n self._smooth_fwhm = smooth_fwhm\n try:\n data = self.get_data(smoothed=True, masked=True, safe_copy=True)\n except ValueError as ve:\n self._smooth_fwhm = old_smooth_fwhm\n raise\n else:\n self._smooth_fwhm = smooth_fwhm\n return data", "response": "Returns the data in the new memory with the given fwhm."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a vector of the masked data.", "response": "def mask_and_flatten(self):\n \"\"\"Return a vector of the masked data.\n\n Returns\n -------\n np.ndarray, tuple of indices (np.ndarray), tuple of the mask shape\n \"\"\"\n self._check_for_mask()\n\n return self.get_data(smoothed=True, masked=True, safe_copy=False)[self.get_mask_indices()],\\\n self.get_mask_indices(), self.mask.shape"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nunmasking the image using the mask", "response": "def unmask(self, arr):\n \"\"\"Use self.mask to reshape arr and self.img to get an affine and header to create\n a new self.img using the data in arr.\n If self.has_mask() is False, will return the same arr.\n \"\"\"\n self._check_for_mask()\n\n if 1 > arr.ndim > 2:\n raise ValueError('The given array has {} dimensions while my mask has {}. '\n 'Masked data must be 1D or 2D array. '.format(arr.ndim,\n len(self.mask.shape)))\n\n if arr.ndim == 2:\n return matrix_to_4dvolume(arr, self.mask.get_data())\n elif arr.ndim == 1:\n return vector_to_volume(arr, self.mask.get_data())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving this object instance in outpath.", "response": "def to_file(self, outpath):\n \"\"\"Save this object instance in outpath.\n\n Parameters\n ----------\n outpath: str\n Output file path\n \"\"\"\n if not self.has_mask() and not self.is_smoothed():\n save_niigz(outpath, self.img)\n else:\n save_niigz(outpath, self.get_data(masked=True, smoothed=True),\n self.get_header(), self.get_affine())"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _read_meta_header(filename):\n fileIN = open(filename, 'r')\n line = fileIN.readline()\n\n meta_dict = {}\n tag_flag = [False]*len(MHD_TAGS)\n while line:\n tags = str.split(line, '=')\n # print tags[0]\n for i in range(len(MHD_TAGS)):\n tag = MHD_TAGS[i]\n if (str.strip(tags[0]) == tag) and (not tag_flag[i]):\n # print tags[1]\n meta_dict[tag] = str.strip(tags[1])\n tag_flag[i] = True\n line = fileIN.readline()\n # comment\n fileIN.close()\n return meta_dict", "response": "Read the. mhd header file and return a dictionary of meta data."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the n - dimensional image data array from a. mhd file.", "response": "def load_raw_data_with_mhd(filename):\n \"\"\"Return a dictionary of meta data from meta header file.\n\n Parameters\n ----------\n filename: str\n Path to a .mhd file\n\n Returns\n -------\n data: numpy.ndarray\n n-dimensional image data array.\n\n meta_dict: dict\n A dictionary with the .mhd header content.\n \"\"\"\n meta_dict = _read_meta_header(filename)\n dim = int(meta_dict['NDims'])\n\n assert (meta_dict['ElementType'] in MHD_TO_NUMPY_TYPE)\n\n arr = [int(i) for i in meta_dict['DimSize'].split()]\n volume = reduce(lambda x, y: x*y, arr[0:dim-1], 1)\n\n pwd = op.dirname(filename)\n raw_file = meta_dict['ElementDataFile']\n data_file = op.join(pwd, raw_file)\n\n ndtype = MHD_TO_NUMPY_TYPE[meta_dict['ElementType']]\n arrtype = NDARRAY_TO_ARRAY_TYPE[ndtype]\n\n with open(data_file, 'rb') as fid:\n binvalues = array.array(arrtype)\n binvalues.fromfile(fid, volume*arr[dim-1])\n\n data = np.array (binvalues, ndtype)\n data = np.reshape(data, (arr[dim-1], volume))\n\n if dim >= 3:\n # Begin 3D fix\n dimensions = [int(i) for i in meta_dict['DimSize'].split()]\n # dimensions.reverse() ??\n data = data.reshape(dimensions)\n # End 3D fix\n\n return data, meta_dict"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_3D_from_4D(filename, vol_idx=0):\n def remove_4th_element_from_hdr_string(hdr, fieldname):\n if fieldname in hdr:\n hdr[fieldname] = ' '.join(hdr[fieldname].split()[:3])\n\n vol, hdr = load_raw_data_with_mhd(filename)\n\n if vol.ndim != 4:\n raise ValueError('Volume in {} does not have 4 dimensions.'.format(op.join(op.dirname(filename),\n hdr['ElementDataFile'])))\n\n if not 0 <= vol_idx < vol.shape[3]:\n raise IndexError('IndexError: 4th dimension in volume {} has {} volumes, not {}.'.format(filename,\n vol.shape[3], vol_idx))\n\n new_vol = vol[:, :, :, vol_idx].copy()\n\n hdr['NDims'] = 3\n remove_4th_element_from_hdr_string(hdr, 'ElementSpacing')\n remove_4th_element_from_hdr_string(hdr, 'DimSize')\n\n return new_vol, hdr", "response": "Return a 3D volume from a 4D nifti image file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new object that caches the output of a function func at the specified level.", "response": "def cache(func, memory, func_memory_level=None, memory_level=None,\n **kwargs):\n \"\"\" Return a joblib.Memory object.\n\n The memory_level determines the level above which the wrapped\n function output is cached. By specifying a numeric value for\n this level, the user can to control the amount of cache memory\n used. This function will cache the function call or not\n depending on the cache level.\n\n Parameters\n ----------\n func: function\n The function which output is to be cached.\n\n memory: instance of joblib.Memory or string\n Used to cache the function call.\n\n func_memory_level: int, optional\n The memory_level from which caching must be enabled for the wrapped\n function.\n\n memory_level: int, optional\n The memory_level used to determine if function call must\n be cached or not (if user_memory_level is equal of greater than\n func_memory_level the function is cached)\n\n kwargs: keyword arguments\n The keyword arguments passed to memory.cache\n\n Returns\n -------\n mem: joblib.MemorizedFunc\n object that wraps the function func. This object may be\n a no-op, if the requested level is lower than the value given\n to _cache()). For consistency, a joblib.Memory object is always\n returned.\n \"\"\"\n verbose = kwargs.get('verbose', 0)\n\n # memory_level and func_memory_level must be both None or both integers.\n memory_levels = [memory_level, func_memory_level]\n both_params_integers = all(isinstance(lvl, int) for lvl in memory_levels)\n both_params_none = all(lvl is None for lvl in memory_levels)\n\n if not (both_params_integers or both_params_none):\n raise ValueError('Reference and user memory levels must be both None '\n 'or both integers.')\n\n if memory is not None and (func_memory_level is None or\n memory_level >= func_memory_level):\n if isinstance(memory, _basestring):\n memory = Memory(cachedir=memory, verbose=verbose)\n if not isinstance(memory, MEMORY_CLASSES):\n raise TypeError(\"'memory' argument must be a string or a \"\n \"joblib.Memory object. \"\n \"%s %s was given.\" % (memory, type(memory)))\n if (memory.cachedir is None and memory_level is not None\n and memory_level > 1):\n warnings.warn(\"Caching has been enabled (memory_level = %d) \"\n \"but no Memory object or path has been provided\"\n \" (parameter memory). Caching deactivated for \"\n \"function %s.\" %\n (memory_level, func.__name__),\n stacklevel=2)\n else:\n memory = Memory(cachedir=None, verbose=verbose)\n return _safe_cache(memory, func, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _cache(self, func, func_memory_level=1, **kwargs):\n\n verbose = getattr(self, 'verbose', 0)\n\n # Creates attributes if they don't exist\n # This is to make creating them in __init__() optional.\n if not hasattr(self, \"memory_level\"):\n self.memory_level = 0\n if not hasattr(self, \"memory\"):\n self.memory = Memory(cachedir=None, verbose=verbose)\n if isinstance(self.memory, _basestring):\n self.memory = Memory(cachedir=self.memory, verbose=verbose)\n\n # If cache level is 0 but a memory object has been provided, set\n # memory_level to 1 with a warning.\n if self.memory_level == 0:\n if (isinstance(self.memory, _basestring)\n or self.memory.cachedir is not None):\n warnings.warn(\"memory_level is currently set to 0 but \"\n \"a Memory object has been provided. \"\n \"Setting memory_level to 1.\")\n self.memory_level = 1\n\n return cache(func, self.memory, func_memory_level=func_memory_level,\n memory_level=self.memory_level, **kwargs)", "response": "Wrapper for the cache function."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save_niigz(filepath, vol, header=None, affine=None):\n # delayed import because could not install nipy on Python 3 on OSX\n we_have_nipy = False\n try:\n import nipy.core.image as niim\n from nipy import save_image\n except:\n pass\n else:\n we_have_nipy = True\n\n if isinstance(vol, np.ndarray):\n log.debug('Saving numpy nifti file: {}.'.format(filepath))\n ni = nib.Nifti1Image(vol, affine, header)\n nib.save(ni, filepath)\n\n elif isinstance(vol, nib.Nifti1Image):\n log.debug('Saving nibabel nifti file: {}.'.format(filepath))\n nib.save(vol, filepath)\n\n elif we_have_nipy and isinstance(vol, niim.Image):\n log.debug('Saving nipy nifti file: {}.'.format(filepath))\n save_image(vol, filepath)\n\n #elif isinstance(vol, NeuroImage):\n # log.debug('Saving boyle.NeuroImage nifti file: {}.'.format(filepath))\n # nib.save(vol.img, filepath)\n\n else:\n raise ValueError('Could not recognise input vol filetype. Got: {}.'.format(repr_imgs(vol)))", "response": "Saves a volume into a Nifti file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef spatialimg_to_hdfgroup(h5group, spatial_img):\n try:\n h5group['data'] = spatial_img.get_data()\n h5group['affine'] = spatial_img.get_affine()\n\n if hasattr(h5group, 'get_extra'):\n h5group['extra'] = spatial_img.get_extra()\n\n hdr = spatial_img.get_header()\n for k in list(hdr.keys()):\n h5group['data'].attrs[k] = hdr[k]\n\n except ValueError as ve:\n raise Exception('Error creating group ' + h5group.name) from ve", "response": "Saves a Nifti1Image into an HDF5 group."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef spatialimg_to_hdfpath(file_path, spatial_img, h5path=None, append=True):\n if h5path is None:\n h5path = '/img'\n\n mode = 'w'\n if os.path.exists(file_path):\n if append:\n mode = 'a'\n\n with h5py.File(file_path, mode) as f:\n try:\n h5img = f.create_group(h5path)\n spatialimg_to_hdfgroup(h5img, spatial_img)\n\n except ValueError as ve:\n raise Exception('Error creating group ' + h5path) from ve", "response": "Saves a Nifti1Image into an HDF5 file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef hdfpath_to_nifti1image(file_path, h5path):\n with h5py.File(file_path, 'r') as f:\n return hdfgroup_to_nifti1image(f[h5path])", "response": "Returns a nibabel Nifti1Image from a HDF5 file path"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a nibabel Nifti1Image from a HDF5 group datasets", "response": "def hdfgroup_to_nifti1image(h5group):\n \"\"\"Returns a nibabel Nifti1Image from a HDF5 group datasets\n\n Parameters\n ----------\n h5group: h5py.Group\n HDF5 group\n\n Returns\n -------\n nibabel Nifti1Image\n \"\"\"\n try:\n data = h5group['data'][:]\n affine = h5group['affine'][:]\n\n extra = None\n if 'extra' in h5group:\n extra = h5group['extra'][:]\n\n header = get_nifti1hdr_from_h5attrs(h5group['data'].attrs)\n\n img = nib.Nifti1Image(data, affine, header=header, extra=extra)\n\n return img\n\n except KeyError as ke:\n raise Exception('Could not read Nifti1Image datasets from ' + h5group.name) from ke"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntransform an H5py Attributes set to a dict.", "response": "def get_nifti1hdr_from_h5attrs(h5attrs):\n \"\"\"Transforms an H5py Attributes set to a dict.\n Converts unicode string keys into standard strings\n and each value into a numpy array.\n\n Parameters\n ----------\n h5attrs: H5py Attributes\n\n Returns\n --------\n dict\n \"\"\"\n hdr = nib.Nifti1Header()\n for k in list(h5attrs.keys()):\n hdr[str(k)] = np.array(h5attrs[k])\n\n return hdr"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns in a list all images found under h5group.", "response": "def all_childnodes_to_nifti1img(h5group):\n \"\"\"Returns in a list all images found under h5group.\n\n Parameters\n ----------\n h5group: h5py.Group\n HDF group\n\n Returns\n -------\n list of nifti1Image\n \"\"\"\n child_nodes = []\n def append_parent_if_dataset(name, obj):\n if isinstance(obj, h5py.Dataset):\n if name.split('/')[-1] == 'data':\n child_nodes.append(obj.parent)\n\n vols = []\n h5group.visititems(append_parent_if_dataset)\n for c in child_nodes:\n vols.append(hdfgroup_to_nifti1image(c))\n\n return vols"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninserting all given nifti files from file_list into one dataset in fname.", "response": "def insert_volumes_in_one_dataset(file_path, h5path, file_list, newshape=None,\n concat_axis=0, dtype=None, append=True):\n \"\"\"Inserts all given nifti files from file_list into one dataset in fname.\n This will not check if the dimensionality of all files match.\n\n Parameters\n ----------\n file_path: string\n HDF5 file path\n\n h5path: string\n\n file_list: list of strings\n\n newshape: tuple or lambda function\n If None, it will not reshape the images.\n If a lambda function, this lambda will receive only the shape array.\n e.g., newshape = lambda x: (np.prod(x[0:3]), x[3])\n If a tuple, it will try to reshape all the images with the same shape.\n It must work for all the images in file_list.\n\n concat_axis: int\n Axis of concatenation after reshaping\n\n dtype: data type\n Dataset data type\n If not set, will use the type of the first file.\n\n append: bool\n\n Raises\n ------\n ValueError if concat_axis is bigger than data dimensionality.\n\n Note\n ----\n For now, this only works if the dataset ends up being a 2D matrix.\n I haven't tested for multi-dimensionality concatenations.\n \"\"\"\n\n def isalambda(v):\n return isinstance(v, type(lambda: None)) and v.__name__ == ''\n\n mode = 'w'\n if os.path.exists(file_path):\n if append:\n mode = 'a'\n\n #loading the metadata into spatialimages\n imgs = [nib.load(vol) for vol in file_list]\n\n #getting the shapes of all volumes\n shapes = [np.array(img.get_shape()) for img in imgs]\n\n #getting the reshaped shapes\n if newshape is not None:\n if isalambda(newshape):\n nushapes = np.array([newshape(shape) for shape in shapes])\n else:\n nushapes = np.array([shape for shape in shapes])\n\n #checking if concat_axis is available in this new shapes\n for nushape in nushapes:\n assert(len(nushape) - 1 < concat_axis)\n\n #calculate the shape of the new dataset\n n_dims = nushapes.shape[1]\n ds_shape = np.zeros(n_dims, dtype=np.int)\n for a in list(range(n_dims)):\n if a == concat_axis:\n ds_shape[a] = np.sum(nushapes[:, concat_axis])\n else:\n ds_shape[a] = np.max(nushapes[:, a])\n\n #get the type of the new dataset\n #dtypes = [img.get_data_dtype() for img in imgs]\n if dtype is None:\n dtype = imgs[0].get_data_dtype()\n\n with h5py.File(file_path, mode) as f:\n try:\n ic = 0\n h5grp = f.create_group(os.path.dirname(h5path))\n h5ds = h5grp.create_dataset(os.path.basename(h5path),\n ds_shape, dtype)\n for img in imgs:\n\n #get the shape of the current image\n nushape = nushapes[ic, :]\n\n def append_to_dataset(h5ds, idx, data, concat_axis):\n \"\"\"\n @param h5ds: H5py DataSet\n @param idx: int\n @param data: ndarray\n @param concat_axis: int\n @return:\n \"\"\"\n shape = data.shape\n ndims = len(shape)\n\n if ndims == 1:\n if concat_axis == 0:\n h5ds[idx] = data\n\n elif ndims == 2:\n if concat_axis == 0:\n h5ds[idx ] = data\n elif concat_axis == 1:\n h5ds[idx ] = data\n\n elif ndims == 3:\n if concat_axis == 0:\n h5ds[idx ] = data\n elif concat_axis == 1:\n h5ds[idx ] = data\n elif concat_axis == 2:\n h5ds[idx ] = data\n\n #appending the reshaped image into the dataset\n append_to_dataset(h5ds, ic,\n np.reshape(img.get_data(), tuple(nushape)),\n concat_axis)\n\n ic += 1\n\n except ValueError as ve:\n raise Exception('Error creating group {} in hdf file {}'.format(h5path, file_path)) from ve"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef treefall(iterable):\n num_elems = len(iterable)\n for i in range(num_elems, -1, -1):\n for c in combinations(iterable, i):\n yield c", "response": "Generator that yields all possible combinations of the elements of iterable and its subsets."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_reliabledictionary_list(client, application_name, service_name):\n cluster = Cluster.from_sfclient(client)\n service = cluster.get_application(application_name).get_service(service_name)\n for dictionary in service.get_dictionaries():\n print(dictionary.name)", "response": "List existing reliable dictionaries and respective schema for given application and service."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nquery Schema information for existing reliable dictionaries.", "response": "def get_reliabledictionary_schema(client, application_name, service_name, dictionary_name, output_file=None):\n \"\"\"Query Schema information for existing reliable dictionaries.\n\n Query Schema information existing reliable dictionaries for given application and service.\n\n :param application_name: Name of the application.\n :type application_name: str\n :param service_name: Name of the service.\n :type service_name: str\n :param dictionary: Name of the reliable dictionary.\n :type dictionary: str\n :param output_file: Optional file to save the schema.\n \"\"\"\n cluster = Cluster.from_sfclient(client)\n dictionary = cluster.get_application(application_name).get_service(service_name).get_dictionary(dictionary_name)\n \n result = json.dumps(dictionary.get_information(), indent=4)\n \n if (output_file == None):\n output_file = \"{}-{}-{}-schema-output.json\".format(application_name, service_name, dictionary_name)\n \n with open(output_file, \"w\") as output:\n output.write(result)\n print('Printed schema information to: ' + output_file)\n print(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef query_reliabledictionary(client, application_name, service_name, dictionary_name, query_string, partition_key=None, partition_id=None, output_file=None):\n cluster = Cluster.from_sfclient(client)\n dictionary = cluster.get_application(application_name).get_service(service_name).get_dictionary(dictionary_name)\n \n \n start = time.time()\n if (partition_id != None):\n result = dictionary.query(query_string, PartitionLookup.ID, partition_id)\n elif (partition_key != None):\n result = dictionary.query(query_string, PartitionLookup.KEY, partition_key)\n else:\n result = dictionary.query(query_string)\n \n if type(result) is str:\n print(result)\n return\n else:\n result = json.dumps(result.get(\"value\"), indent=4)\n \n print(\"Query took \" + str(time.time() - start) + \" seconds\")\n \n if (output_file == None):\n output_file = \"{}-{}-{}-query-output.json\".format(application_name, service_name, dictionary_name)\n \n with open(output_file, \"w\") as output:\n output.write(result)\n print()\n print('Printed output to: ' + output_file)\n print(result)", "response": "Query existing reliable dictionary for given application and service."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes create update delete operations on existing reliable dictionaries.", "response": "def execute_reliabledictionary(client, application_name, service_name, input_file):\n \"\"\"Execute create, update, delete operations on existing reliable dictionaries.\n\n carry out create, update and delete operations on existing reliable dictionaries for given application and service.\n\n :param application_name: Name of the application.\n :type application_name: str\n :param service_name: Name of the service.\n :type service_name: str\n :param output_file: input file with list of json to provide the operation information for reliable dictionaries.\n \"\"\"\n\n cluster = Cluster.from_sfclient(client)\n service = cluster.get_application(application_name).get_service(service_name)\n\n # call get service with headers and params\n with open(input_file) as json_file:\n json_data = json.load(json_file)\n service.execute(json_data)\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef select_arg_verify(endpoint, cert, key, pem, ca, aad, no_verify): #pylint: disable=invalid-name,too-many-arguments\n\n if not (endpoint.lower().startswith('http')\n or endpoint.lower().startswith('https')):\n raise CLIError('Endpoint must be HTTP or HTTPS')\n\n usage = ('Valid syntax : --endpoint [ [ --key --cert | --pem | --aad] '\n '[ --ca | --no-verify ] ]')\n\n if ca and not (pem or all([key, cert])):\n raise CLIError(usage)\n\n if no_verify and not (pem or all([key, cert]) or aad):\n raise CLIError(usage)\n\n if no_verify and ca:\n raise CLIError(usage)\n\n if any([cert, key]) and not all([cert, key]):\n raise CLIError(usage)\n\n if aad and any([pem, cert, key]):\n raise CLIError(usage)\n\n if pem and any([cert, key]):\n raise CLIError(usage)", "response": "Verify arguments for select command"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef select(endpoint, cert=None, key=None, pem=None, ca=None, #pylint: disable=invalid-name, too-many-arguments\n aad=False, no_verify=False):\n #pylint: disable-msg=too-many-locals\n \"\"\"\n Connects to a Service Fabric cluster endpoint.\n If connecting to secure cluster specify an absolute path to a cert (.crt)\n and key file (.key) or a single file with both (.pem). Do not specify both.\n Optionally, if connecting to a secure cluster, specify also an absolute\n path to a CA bundle file or directory of trusted CA certs.\n :param str endpoint: Cluster endpoint URL, including port and HTTP or HTTPS\n prefix\n :param str cert: Absolute path to a client certificate file\n :param str key: Absolute path to client certificate key file\n :param str pem: Absolute path to client certificate, as a .pem file\n :param str ca: Absolute path to CA certs directory to treat as valid\n or CA bundle\n file\n :param bool aad: Use Azure Active Directory for authentication\n :param bool no_verify: Disable verification for certificates when using\n HTTPS, note: this is an insecure option and should not be used for\n production environments\n \"\"\"\n from sfctl.config import (set_ca_cert, set_auth, set_aad_cache,\n set_cluster_endpoint,\n set_no_verify)\n from msrest import ServiceClient, Configuration\n from sfctl.auth import ClientCertAuthentication, AdalAuthentication\n\n select_arg_verify(endpoint, cert, key, pem, ca, aad, no_verify)\n\n if aad:\n new_token, new_cache = get_aad_token(endpoint, no_verify)\n set_aad_cache(new_token, new_cache)\n rest_client = ServiceClient(\n AdalAuthentication(no_verify),\n Configuration(endpoint)\n )\n\n # Make sure basic GET request succeeds\n rest_client.send(rest_client.get('/')).raise_for_status()\n else:\n client_cert = None\n if pem:\n client_cert = pem\n elif cert:\n client_cert = (cert, key)\n\n rest_client = ServiceClient(\n ClientCertAuthentication(client_cert, ca, no_verify),\n Configuration(endpoint)\n )\n\n # Make sure basic GET request succeeds\n rest_client.send(rest_client.get('/')).raise_for_status()\n\n set_cluster_endpoint(endpoint)\n set_no_verify(no_verify)\n set_ca_cert(ca)\n set_auth(pem, cert, key, aad)", "response": "Select a new node from a Service Fabric cluster."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _openpyxl_read_xl(xl_path: str):\n try:\n wb = load_workbook(filename=xl_path, read_only=True)\n except:\n raise\n else:\n return wb", "response": "Use openpyxl to read an Excel file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _check_xl_path(xl_path: str):\n xl_path = op.abspath(op.expanduser(xl_path))\n\n if not op.isfile(xl_path):\n raise IOError(\"Could not find file in {}.\".format(xl_path))\n\n return xl_path, _use_openpyxl_or_xlrf(xl_path)", "response": "Check if the given path is a valid Excel file and return the expanded absolute path of xl_path if exists and openpyxl if it is not."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef read_xl(xl_path: str):\n xl_path, choice = _check_xl_path(xl_path)\n reader = XL_READERS[choice]\n\n return reader(xl_path)", "response": "Return the workbook from the Excel file in xl_path."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_sheet_list(xl_path: str) -> List:\n wb = read_xl(xl_path)\n\n if hasattr(wb, 'sheetnames'):\n return wb.sheetnames\n else:\n return wb.sheet_names()", "response": "Return a list with the name of the sheets in\n the Excel file in xl_path."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef concat_sheets(xl_path: str, sheetnames=None, add_tab_names=False):\n xl_path, choice = _check_xl_path(xl_path)\n\n if sheetnames is None:\n sheetnames = get_sheet_list(xl_path)\n\n sheets = pd.read_excel(xl_path, sheetname=sheetnames)\n\n if add_tab_names:\n for tab in sheets:\n sheets[tab]['Tab'] = [tab] * len(sheets[tab])\n\n return pd.concat([sheets[tab] for tab in sheets])", "response": "Returns a pandas DataFrame with the concated\n content of the sheetnames from the Excel file inxl_path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nraising an AttributeError if df does not have a column named as an item of the list of strings col_names.", "response": "def _check_cols(df, col_names):\n \"\"\" Raise an AttributeError if `df` does not have a column named as an item of\n the list of strings `col_names`.\n \"\"\"\n for col in col_names:\n if not hasattr(df, col):\n raise AttributeError(\"DataFrame does not have a '{}' column, got {}.\".format(col,\n df.columns))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef col_values(df, col_name):\n _check_cols(df, [col_name])\n\n if 'O' in df[col_name] or pd.np.issubdtype(df[col_name].dtype, str): # if the column is of strings\n return [nom.lower() for nom in df[pd.notnull(df)][col_name] if not pd.isnull(nom)]\n else:\n return [nom for nom in df[pd.notnull(df)][col_name] if not pd.isnull(nom)]", "response": "Return a list of not null values from the col_name column of df."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a DataFrame with the duplicated values of the column col_name in df.", "response": "def duplicated_rows(df, col_name):\n \"\"\" Return a DataFrame with the duplicated values of the column `col_name`\n in `df`.\"\"\"\n _check_cols(df, [col_name])\n\n dups = df[pd.notnull(df[col_name]) & df.duplicated(subset=[col_name])]\n return dups"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the duplicated items in values", "response": "def duplicated(values: Sequence):\n \"\"\" Return the duplicated items in `values`\"\"\"\n vals = pd.Series(values)\n return vals[vals.duplicated()]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef timestamp_with_tzinfo(dt):\n utc = tzutc()\n\n if dt.tzinfo:\n dt = dt.astimezone(utc).replace(tzinfo=None)\n return dt.isoformat() + 'Z'", "response": "Serialize a date or time value into an ISO8601 text representation with a timezone adjusted to UTC timezone."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert all values in data to string all values in data.", "response": "def _to_string(data):\n \"\"\" Convert to string all values in `data`.\n\n Parameters\n ----------\n data: dict[str]->object\n\n Returns\n -------\n string_data: dict[str]->str\n \"\"\"\n sdata = data.copy()\n for k, v in data.items():\n if isinstance(v, datetime):\n sdata[k] = timestamp_to_date_str(v)\n\n elif not isinstance(v, (string_types, float, int)):\n sdata[k] = str(v)\n\n return sdata"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninsert data into table ensuring that data has unique values in unique_fields.", "response": "def insert_unique(table, data, unique_fields=None, *, raise_if_found=False):\n \"\"\"Insert `data` into `table` ensuring that data has unique values\n in `table` for the fields listed in `unique_fields`.\n\n If `raise_if_found` is True, will raise an NotUniqueItemError if\n another item with the same `unique_fields` values are found\n previously in `table`.\n If False, will return the `eid` from the item found.\n\n Parameters\n ----------\n table: tinydb.Table\n\n data: dict\n\n unique_fields: list of str\n Name of fields (keys) from `data` which are going to be used to build\n a sample to look for exactly the same values in the database.\n If None, will use every key in `data`.\n\n raise_if_found: bool\n\n Returns\n -------\n eid: int\n Id of the object inserted or the one found with same `unique_fields`.\n\n Raises\n ------\n MoreThanOneItemError\n Raise even with `raise_with_found` == False if it finds more than one item\n with the same values as the sample.\n\n NotUniqueItemError\n If `raise_if_found` is True and an item with the same `unique_fields`\n values from `data` is found in `table`.\n \"\"\"\n item = find_unique(table, data, unique_fields)\n if item is not None:\n if raise_if_found:\n raise NotUniqueItemError('Not expected to find an item with the same '\n 'values for {}. Inserting {} got {} in eid {}.'.format(unique_fields,\n data,\n table.get(eid=item),\n item))\n else:\n return item\n\n return table.insert(data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef search_sample(table, sample):\n query = _query_sample(sample=sample, operators='__eq__')\n\n return table.search(query)", "response": "Search for items in table that have the same field sub - set values as in sample."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef search_unique(table, sample, unique_fields=None):\n if unique_fields is None:\n unique_fields = list(sample.keys())\n\n query = _query_data(sample, field_names=unique_fields, operators='__eq__')\n items = table.search(query)\n\n if len(items) == 1:\n return items[0]\n\n if len(items) == 0:\n return None\n\n raise MoreThanOneItemError('Expected to find zero or one items, but found '\n '{} items.'.format(len(items)))", "response": "Search for items in table that have the same field sub - set values as in sample."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsearches in table an item with the value of the unique_fields in the sample.", "response": "def find_unique(table, sample, unique_fields=None):\n \"\"\"Search in `table` an item with the value of the `unique_fields` in the `sample` sample.\n Check if the the obtained result is unique. If nothing is found will return an empty list,\n if there is more than one item found, will raise an IndexError.\n\n Parameters\n ----------\n table: tinydb.table\n\n sample: dict\n Sample data\n\n unique_fields: list of str\n Name of fields (keys) from `data` which are going to be used to build\n a sample to look for exactly the same values in the database.\n If None, will use every key in `data`.\n\n Returns\n -------\n eid: int\n Id of the object found with same `unique_fields`.\n None if none is found.\n\n Raises\n ------\n MoreThanOneItemError\n If more than one example is found.\n \"\"\"\n res = search_unique(table, sample, unique_fields)\n if res is not None:\n return res.eid\n else:\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a TinyDB query that looks for items that have each field in sample with a value compared with the correspondent operation in operators.", "response": "def _query_sample(sample, operators='__eq__'):\n \"\"\"Create a TinyDB query that looks for items that have each field in `sample` with a value\n compared with the correspondent operation in `operators`.\n\n Parameters\n ----------\n sample: dict\n The sample data\n\n operators: str or list of str\n A list of comparison operations for each field value in `sample`.\n If this is a str, will use the same operator for all `sample` fields.\n If you want different operators for each field, remember to use an OrderedDict for `sample`.\n Check TinyDB.Query class for possible choices.\n\n Returns\n -------\n query: tinydb.database.Query\n \"\"\"\n if isinstance(operators, str):\n operators = [operators] * len(sample)\n\n if len(sample) != len(operators):\n raise ValueError('Expected `operators` to be a string or a list with the same'\n ' length as `field_names` ({}), got {}.'.format(len(sample),\n operators))\n\n queries = []\n for i, fn in enumerate(sample):\n fv = sample[fn]\n op = operators[i]\n queries.append(_build_query(field_name=fn,\n field_value=fv,\n operator=op))\n\n return _concat_queries(queries, operators='__and__')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _query_data(data, field_names=None, operators='__eq__'):\n if field_names is None:\n field_names = list(data.keys())\n\n if isinstance(field_names, str):\n field_names = [field_names]\n\n # using OrderedDict by default, in case operators has different operators for each field.\n sample = OrderedDict([(fn, data[fn]) for fn in field_names])\n return _query_sample(sample, operators=operators)", "response": "Create a tinyDB Query object that looks for items that confirms the correspondent operator\n from operators for each field_names field values from data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _concat_queries(queries, operators='__and__'):\n # checks first\n if not queries:\n raise ValueError('Expected some `queries`, got {}.'.format(queries))\n\n if len(queries) == 1:\n return queries[0]\n\n if isinstance(operators, str):\n operators = [operators] * (len(queries) - 1)\n\n if len(queries) - 1 != len(operators):\n raise ValueError('Expected `operators` to be a string or a list with the same'\n ' length as `field_names` ({}), got {}.'.format(len(queries),\n operators))\n\n # recursively build the query\n first, rest, end = queries[0], queries[1:-1], queries[-1:][0]\n bigop = getattr(first, operators[0])\n for i, q in enumerate(rest):\n bigop = getattr(bigop(q), operators[i])\n\n return bigop(end)", "response": "Create a tinyDB Query object that is the concatenation of each query in queries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nbuild a tinyDB Query object with the format that can be used to query the database.", "response": "def _build_query(field_name, field_value, operator='__eq__'):\n \"\"\"Create a tinyDB Query object with the format:\n (where(`field_name`) `operator` `field_value`)\n\n Parameters\n ----------\n field_name: str\n The name of the field to be queried.\n\n field_value:\n The value of the field\n\n operator: str\n The comparison operator.\n Check TinyDB.Query class for possible choices.\n\n Returns\n -------\n query: tinydb.database.Query\n \"\"\"\n qelem = where(field_name)\n\n if not hasattr(qelem, operator):\n raise NotImplementedError('Operator `{}` not found in query object.'.format(operator))\n else:\n query = getattr(qelem, operator)\n\n return query(field_value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsearching the table with the specified object ID.", "response": "def search_by_eid(self, table_name, eid):\n \"\"\"Return the element in `table_name` with Object ID `eid`.\n If None is found will raise a KeyError exception.\n\n Parameters\n ----------\n table_name: str\n The name of the table to look in.\n\n eid: int\n The Object ID of the element to look for.\n\n Returns\n -------\n elem: tinydb.database.Element\n\n Raises\n ------\n KeyError\n If the element with ID `eid` is not found.\n \"\"\"\n elem = self.table(table_name).get(eid=eid)\n if elem is None:\n raise KeyError('Could not find {} with eid {}.'.format(table_name, eid))\n\n return elem"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef insert_unique(self, table_name, data, unique_fields=None, *, raise_if_found=False):\n return insert_unique(table=self.table(table_name),\n data=_to_string(data),\n unique_fields=unique_fields,\n raise_if_found=raise_if_found)", "response": "Insert data into table_name ensuring that data has unique values in unique_fields."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch in table an item with the value of the unique_fields in the data sample.", "response": "def search_unique(self, table_name, sample, unique_fields=None):\n \"\"\" Search in `table` an item with the value of the `unique_fields` in the `data` sample.\n Check if the the obtained result is unique. If nothing is found will return an empty list,\n if there is more than one item found, will raise an IndexError.\n\n Parameters\n ----------\n table_name: str\n\n sample: dict\n Sample data\n\n unique_fields: list of str\n Name of fields (keys) from `data` which are going to be used to build\n a sample to look for exactly the same values in the database.\n If None, will use every key in `data`.\n\n Returns\n -------\n eid: int\n Id of the object found with same `unique_fields`.\n None if none is found.\n\n Raises\n ------\n MoreThanOneItemError\n If more than one example is found.\n \"\"\"\n return search_unique(table=self.table(table_name),\n sample=sample,\n unique_fields=unique_fields)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsearch for items in table that have the same field sub - set values as in sample.", "response": "def search_sample(self, table_name, sample):\n \"\"\"Search for items in `table` that have the same field sub-set values as in `sample`.\n\n Parameters\n ----------\n table_name: str\n\n sample: dict\n Sample data\n\n Returns\n -------\n search_result: list of dict\n List of the items found. The list is empty if no item is found.\n \"\"\"\n return search_sample(table=self.table(table_name),\n sample=sample)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_unique(self, table_name, sample, unique_fields=None):\n try:\n eid = find_unique(self.table(table_name),\n sample=sample,\n unique_fields=unique_fields)\n except:\n return False\n else:\n return eid is not None", "response": "Return True if an item with the value of unique_fields from table_name is unique in the table with table_name False otherwise."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_unique(self, table_name, fields, data, cond=None, unique_fields=None,\n *, raise_if_not_found=False):\n \"\"\"Update the unique matching element to have a given set of fields.\n\n Parameters\n ----------\n table_name: str\n\n fields: dict or function[dict -> None]\n new data/values to insert into the unique element\n or a method that will update the elements.\n\n data: dict\n Sample data for query\n\n cond: tinydb.Query\n which elements to update\n\n unique_fields: list of str\n\n raise_if_not_found: bool\n Will raise an exception if the element is not found for update.\n\n Returns\n -------\n eid: int\n The eid of the updated element if found, None otherwise.\n \"\"\"\n eid = find_unique(self.table(table_name), data, unique_fields)\n\n if eid is None:\n if raise_if_not_found:\n msg = 'Could not find {} with {}'.format(table_name, data)\n if cond is not None:\n msg += ' where {}.'.format(cond)\n raise IndexError(msg)\n\n else:\n self.table(table_name).update(_to_string(fields), cond=cond, eids=[eid])\n\n return eid", "response": "Update the unique matching element with a given set of fields."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the number of items that match the sample field values in table table_name", "response": "def count(self, table_name, sample):\n \"\"\"Return the number of items that match the `sample` field values\n in table `table_name`.\n Check function search_sample for more details.\n \"\"\"\n return len(list(search_sample(table=self.table(table_name),\n sample=sample)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if the object contains get_data and get_affine methods and returns a boolean if so returns True if the object contains get_data and get_affine methods and False otherwise.", "response": "def is_img(obj):\n \"\"\" Check for get_data and get_affine method in an object\n\n Parameters\n ----------\n obj: any object\n Tested object\n\n Returns\n -------\n is_img: boolean\n True if get_data and get_affine methods are present and callable,\n False otherwise.\n \"\"\"\n try:\n get_data = getattr(obj, 'get_data')\n get_affine = getattr(obj, 'get_affine')\n\n return isinstance(get_data, collections.Callable) and \\\n isinstance(get_affine, collections.Callable)\n except AttributeError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the data in the image without having a side effect on the Nifti1Image object", "response": "def get_data(img):\n \"\"\"Get the data in the image without having a side effect on the Nifti1Image object\n\n Parameters\n ----------\n img: Nifti1Image\n\n Returns\n -------\n np.ndarray\n \"\"\"\n if hasattr(img, '_data_cache') and img._data_cache is None:\n # Copy locally the nifti_image to avoid the side effect of data\n # loading\n img = copy.deepcopy(img)\n # force garbage collector\n gc.collect()\n return img.get_data()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_shape(img):\n if hasattr(img, 'shape'):\n shape = img.shape\n else:\n shape = img.get_data().shape\n return shape", "response": "Return the shape of the\n image"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_valid_coordinate(img, i, j, k):\n imgx, imgy, imgz = get_shape(img)\n return (i >= 0 and i < imgx) and \\\n (j >= 0 and j < imgy) and \\\n (k >= 0 and k < imgz)", "response": "Return True if the given image coordinate values are within the img boundaries."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if one_img and another_img are compatible with Nifti1Image.", "response": "def check_img_compatibility(one_img, another_img, only_check_3d=False):\n \"\"\"Return true if one_img and another_img have the same shape.\n False otherwise.\n If both are nibabel.Nifti1Image will also check for affine matrices.\n\n Parameters\n ----------\n one_img: nibabel.Nifti1Image or np.ndarray\n\n another_img: nibabel.Nifti1Image or np.ndarray\n\n only_check_3d: bool\n If True will check only the 3D part of the affine matrices when they have more dimensions.\n\n Raises\n ------\n NiftiFilesNotCompatible\n \"\"\"\n nd_to_check = None\n if only_check_3d:\n nd_to_check = 3\n\n if hasattr(one_img, 'shape') and hasattr(another_img, 'shape'):\n if not have_same_shape(one_img, another_img, nd_to_check=nd_to_check):\n msg = 'Shape of the first image: \\n{}\\n is different from second one: \\n{}'.format(one_img.shape,\n another_img.shape)\n raise NiftiFilesNotCompatible(repr_imgs(one_img), repr_imgs(another_img), message=msg)\n\n if hasattr(one_img, 'get_affine') and hasattr(another_img, 'get_affine'):\n if not have_same_affine(one_img, another_img, only_check_3d=only_check_3d):\n msg = 'Affine matrix of the first image: \\n{}\\n is different ' \\\n 'from second one:\\n{}'.format(one_img.get_affine(), another_img.get_affine())\n raise NiftiFilesNotCompatible(repr_imgs(one_img), repr_imgs(another_img), message=msg)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef have_same_affine(one_img, another_img, only_check_3d=False):\n img1 = check_img(one_img)\n img2 = check_img(another_img)\n\n ndim1 = len(img1.shape)\n ndim2 = len(img2.shape)\n\n if ndim1 < 3:\n raise ValueError('Image {} has only {} dimensions, at least 3 dimensions is expected.'.format(repr_imgs(img1), ndim1))\n\n if ndim2 < 3:\n raise ValueError('Image {} has only {} dimensions, at least 3 dimensions is expected.'.format(repr_imgs(img2), ndim1))\n\n affine1 = img1.get_affine()\n affine2 = img2.get_affine()\n if only_check_3d:\n affine1 = affine1[:3, :3]\n affine2 = affine2[:3, :3]\n\n try:\n return np.allclose(affine1, affine2)\n except ValueError:\n return False\n except:\n raise", "response": "Return True if the affine matrix of one_img is close to the affine matrix of another_img False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _make_it_3d(img):\n shape = get_shape(img)\n if len(shape) == 3:\n return img\n\n elif (len(shape) == 4 and shape[3] == 1):\n # \"squeeze\" the image.\n try:\n data = get_data(img)\n affine = img.get_affine()\n img = nib.Nifti1Image(data[:, :, :, 0], affine)\n except Exception as exc:\n raise Exception(\"Error making image '{}' a 3D volume file.\".format(img)) from exc\n else:\n return img\n else:\n raise TypeError(\"A 3D image is expected, but an image with a shape of {} was given.\".format(shape))", "response": "Enforce that img is a 3D img - like object raise a TypeError."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_img(image, make_it_3d=False):\n if isinstance(image, string_types):\n # a filename, load it\n if not op.exists(image):\n raise FileNotFound(image)\n\n try:\n img = nib.load(image)\n if make_it_3d:\n img = _make_it_3d(img)\n except Exception as exc:\n raise Exception('Error loading image file {}.'.format(image)) from exc\n else:\n return img\n\n elif isinstance(image, nib.Nifti1Image) or is_img(image):\n return image\n\n else:\n raise TypeError('Data given cannot be converted to a nifti'\n ' image: this object -\"{}\"- does not have'\n ' get_data or get_affine methods'.format(type(image)))", "response": "Checks that an image is a proper img."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprint of img or imgs", "response": "def repr_imgs(imgs):\n \"\"\"Printing of img or imgs\"\"\"\n if isinstance(imgs, string_types):\n return imgs\n\n if isinstance(imgs, collections.Iterable):\n return '[{}]'.format(', '.join(repr_imgs(img) for img in imgs))\n\n # try get_filename\n try:\n filename = imgs.get_filename()\n if filename is not None:\n img_str = \"{}('{}')\".format(imgs.__class__.__name__, filename)\n else:\n img_str = \"{}(shape={}, affine={})\".format(imgs.__class__.__name__,\n repr(get_shape(imgs)),\n repr(imgs.get_affine()))\n except Exception as exc:\n log.error('Error reading attributes from img.get_filename()')\n return repr(imgs)\n else:\n return img_str"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef have_same_shape(array1, array2, nd_to_check=None):\n shape1 = array1.shape\n shape2 = array2.shape\n if nd_to_check is not None:\n if len(shape1) < nd_to_check:\n msg = 'Number of dimensions to check {} is out of bounds for the shape of the first image: \\n{}\\n.'.format(shape1)\n raise ValueError(msg)\n elif len(shape2) < nd_to_check:\n msg = 'Number of dimensions to check {} is out of bounds for the shape of the second image: \\n{}\\n.'.format(shape2)\n raise ValueError(msg)\n\n shape1 = shape1[:nd_to_check]\n shape2 = shape2[:nd_to_check]\n\n return shape1 == shape2", "response": "Returns True if array1 and array2 have the same shapes False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if both images have the same geometry.", "response": "def have_same_geometry(fname1, fname2):\n \"\"\"\n @param fname1: string\n File path of an image\n\n @param fname2: string\n File path of an image\n\n @return: bool\n True if both have the same geometry\n \"\"\"\n img1shape = nib.load(fname1).get_shape()\n img2shape = nib.load(fname2).get_shape()\n return have_same_shape(img1shape, img2shape)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if both images have the same spatial geometry.", "response": "def have_same_spatial_geometry(fname1, fname2):\n \"\"\"\n @param fname1: string\n File path of an image\n\n @param fname2: string\n File path of an image\n\n @return: bool\n True if both have the same geometry\n \"\"\"\n img1shape = nib.load(fname1).get_shape()\n img2shape = nib.load(fname2).get_shape()\n return img1shape[:3] == img2shape[:3]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dir_match(regex, wd=os.curdir):\n ls = os.listdir(wd)\n\n filt = re.compile(regex).match\n return filter_list(ls, filt)", "response": "Create a list of regex matches that result from the match_regex\n of all files within wd."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning absolute paths of folders that match the regex within folder_path and all its children folders. Note: The regex matching is done using the match function of the re module. Parameters ---------- folder_path: string regex: string Returns ------- A list of strings.", "response": "def recursive_dir_match(folder_path, regex=''):\n \"\"\"\n Returns absolute paths of folders that match the regex within folder_path and\n all its children folders.\n\n Note: The regex matching is done using the match function\n of the re module.\n\n Parameters\n ----------\n folder_path: string\n\n regex: string\n\n Returns\n -------\n A list of strings.\n \"\"\"\n outlist = []\n for root, dirs, files in os.walk(folder_path):\n outlist.extend([op.join(root, f) for f in dirs\n if re.match(regex, f)])\n\n return outlist"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of files that match the search_regex within file_dir.", "response": "def get_file_list(file_dir, regex=''):\n \"\"\"\n Creates a list of files that match the search_regex within file_dir.\n The list of files will have file_dir as path prefix.\n\n Parameters\n ----------\n @param file_dir:\n\n @param search_regex:\n\n Returns:\n --------\n List of paths to files that match the search_regex\n \"\"\"\n file_list = os.listdir(file_dir)\n file_list.sort()\n\n if regex:\n file_list = search_list(file_list, regex)\n\n file_list = [op.join(file_dir, fname) for fname in file_list]\n\n return file_list"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef recursive_find_search(folder_path, regex=''):\n outlist = []\n for root, dirs, files in os.walk(folder_path):\n outlist.extend([op.join(root, f) for f in files\n if re.search(regex, f)])\n\n return outlist", "response": "Recursive function to find files that match the regex within folder_path and return a list of strings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef iter_recursive_find(folder_path, *regex):\n for root, dirs, files in os.walk(folder_path):\n if len(files) > 0:\n outlist = []\n for f in files:\n for reg in regex:\n if re.search(reg, f):\n outlist.append(op.join(root, f))\n if len(outlist) == len(regex):\n yield outlist", "response": "Returns absolute paths of files that match the regexs within folder_path and all its children folders."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nuses glob to find all files or folders that match the regex starting from the base_directory.", "response": "def recursive_glob(base_directory, regex=''):\n \"\"\"\n Uses glob to find all files or folders that match the regex\n starting from the base_directory.\n\n Parameters\n ----------\n base_directory: str\n\n regex: str\n\n Returns\n -------\n files: list\n\n \"\"\"\n files = glob(op.join(base_directory, regex))\n for path, dirlist, filelist in os.walk(base_directory):\n for dir_name in dirlist:\n files.extend(glob(op.join(path, dir_name, regex)))\n\n return files"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the path to the latest modified file in input_dir.", "response": "def get_last_file(input_dir, glob_pattern='*', key=op.getctime, reverse=True):\n \"\"\" Return the path to the latest file in `input_dir`.\n The `key` argument defines which information to use for sorting\n the list of files, could be:\n - creation date: os.path.getctime,\n - modification date: os.path.getmtime,\n etc.\n\n Parameters\n ----------\n input_dir: str\n Path to the folder where to perform the `glob`.\n\n glob_pattern: str\n `glob` Pattern to filter the files in `input_dir`.\n\n key: str\n Sorting key function\n\n reverse: bool\n Set to True if you want the sorting to be in decreasing order,\n False otherwise.\n\n Returns\n -------\n latest_filepath: str\n Path to the latest modified file in `input_dir`.\n \"\"\"\n files = glob(op.join(input_dir, glob_pattern))\n files.sort(key=key, reverse=reverse)\n return files[0]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomposing an error message with keyword arguments.", "response": "def compose_err_msg(msg, **kwargs):\n \"\"\"Append key-value pairs to msg, for display.\n\n Parameters\n ----------\n msg: string\n arbitrary message\n kwargs: dict\n arbitrary dictionary\n\n Returns\n -------\n updated_msg: string\n msg, with \"key: value\" appended. Only string values are appended.\n\n Example\n -------\n >>> compose_err_msg('Error message with arguments...', arg_num=123, \\\n arg_str='filename.nii', arg_bool=True)\n 'Error message with arguments...\\\\narg_str: filename.nii'\n >>>\n \"\"\"\n updated_msg = msg\n for k, v in sorted(kwargs.items()):\n if isinstance(v, _basestring): # print only str-like arguments\n updated_msg += \"\\n\" + k + \": \" + v\n\n return updated_msg"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef group_dicom_files(dicom_file_paths, header_fields):\n dist = SimpleDicomFileDistance(field_weights=header_fields)\n\n path_list = dicom_file_paths.copy()\n\n path_groups = DefaultOrderedDict(DicomFileSet)\n\n while len(path_list) > 0:\n file_path1 = path_list.pop()\n file_subgroup = [file_path1]\n\n dist.set_dicom_file1(file_path1)\n j = len(path_list)-1\n while j >= 0:\n file_path2 = path_list[j]\n dist.set_dicom_file2(file_path2)\n\n if dist.transform():\n file_subgroup.append(file_path2)\n path_list.pop(j)\n\n j -= 1\n path_groups[file_path1].from_set(file_subgroup, check_if_dicoms=False)\n\n return path_groups", "response": "Returns a list of lists of DICOM file absolute paths and returns a list of lists of DICOM file paths that have the same header fields."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef copy_groups_to_folder(dicom_groups, folder_path, groupby_field_name):\n if dicom_groups is None or not dicom_groups:\n raise ValueError('Expected a boyle.dicom.sets.DicomFileSet.')\n if not os.path.exists(folder_path):\n os.makedirs(folder_path, exist_ok=False)\n\n for dcmg in dicom_groups:\n if groupby_field_name is not None and len(groupby_field_name) > 0:\n dfile = DicomFile(dcmg)\n dir_name = ''\n for att in groupby_field_name:\n dir_name = os.path.join(dir_name, dfile.get_attributes(att))\n dir_name = str(dir_name)\n else:\n dir_name = os.path.basename(dcmg)\n\n group_folder = os.path.join(folder_path, dir_name)\n os.makedirs(group_folder, exist_ok=False)\n\n log.debug('Copying files to {}.'.format(group_folder))\n\n import shutil\n dcm_files = dicom_groups[dcmg]\n\n for srcf in dcm_files:\n destf = os.path.join(group_folder, os.path.basename(srcf))\n while os.path.exists(destf):\n destf += '+'\n shutil.copy2(srcf, destf)", "response": "Copy the DICOM files in the group_name list to the folder_path."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the Levenshtein distance between all files in dicom_files using an anonymized Levenshtein measure between all field names in field_weights and atabase.", "response": "def calculate_file_distances(dicom_files, field_weights=None,\n dist_method_cls=None, **kwargs):\n \"\"\"\n Calculates the DicomFileDistance between all files in dicom_files, using an\n weighted Levenshtein measure between all field names in field_weights and\n their corresponding weights.\n\n Parameters\n ----------\n dicom_files: iterable of str\n Dicom file paths\n\n field_weights: dict of str to float\n A dict with header field names to float scalar values, that\n indicate a distance measure ratio for the levenshtein distance\n averaging of all the header field names in it. e.g., {'PatientID': 1}\n\n dist_method_cls: DicomFileDistance class\n Distance method object to compare the files.\n If None, the default DicomFileDistance method using Levenshtein\n distance between the field_wieghts will be used.\n\n kwargs: DicomFileDistance instantiation named arguments\n Apart from the field_weitghts argument.\n\n Returns\n -------\n file_dists: np.ndarray or scipy.sparse.lil_matrix of shape NxN\n Levenshtein distances between each of the N items in dicom_files.\n \"\"\"\n if dist_method_cls is None:\n dist_method = LevenshteinDicomFileDistance(field_weights)\n else:\n try:\n dist_method = dist_method_cls(field_weights=field_weights, **kwargs)\n except:\n log.exception('Could not instantiate {} object with field_weights '\n 'and {}'.format(dist_method_cls, kwargs))\n\n dist_dtype = np.float16\n n_files = len(dicom_files)\n\n try:\n file_dists = np.zeros((n_files, n_files), dtype=dist_dtype)\n except MemoryError as mee:\n import scipy.sparse\n file_dists = scipy.sparse.lil_matrix((n_files, n_files),\n dtype=dist_dtype)\n\n for idxi in range(n_files):\n dist_method.set_dicom_file1(dicom_files[idxi])\n\n for idxj in range(idxi+1, n_files):\n dist_method.set_dicom_file2(dicom_files[idxj])\n\n if idxi != idxj:\n file_dists[idxi, idxj] = dist_method.transform()\n\n return file_dists"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfit the set of keys to the set of dictionaries.", "response": "def fit(self, dcm_file1, dcm_file2):\n \"\"\"\n Parameters\n ----------\n dcm_file1: str (path to file) or DicomFile or namedtuple\n\n dcm_file2: str (path to file) or DicomFile or namedtuple\n \"\"\"\n self.set_dicom_file1(dcm_file1)\n self.set_dicom_file2(dcm_file2)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform(self):\n if self.dcmf1 is None or self.dcmf2 is None:\n return np.inf\n\n for field_name in self.field_weights:\n if (str(getattr(self.dcmf1, field_name, ''))\n != str(getattr(self.dcmf2, field_name, ''))):\n return False\n\n return True", "response": "Returns True if all the field values in self. dcmf1 and self. dcmf2 are the same False otherwise."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef levenshtein_analysis(self, field_weights=None):\n if field_weights is None:\n if not isinstance(self.field_weights, dict):\n raise ValueError('Expected a dict for `field_weights` parameter, '\n 'got {}'.format(type(self.field_weights)))\n\n key_dicoms = list(self.dicom_groups.keys())\n file_dists = calculate_file_distances(key_dicoms, field_weights, self._dist_method_cls)\n return file_dists", "response": "Calculates the levenshtein distance averaging of all the header fields in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dist_percentile_threshold(dist_matrix, perc_thr=0.05, k=1):\n triu_idx = np.triu_indices(dist_matrix.shape[0], k=k)\n upper = np.zeros_like(dist_matrix)\n upper[triu_idx] = dist_matrix[triu_idx] < np.percentile(dist_matrix[triu_idx], perc_thr)\n return upper", "response": "Thresholds a distance matrix and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of 2 - tuples with pairs of dicom groups that are in the same folder within given depth.", "response": "def get_groups_in_same_folder(self, folder_depth=3):\n \"\"\"\n Returns a list of 2-tuples with pairs of dicom groups that\n are in the same folder within given depth.\n\n Parameters\n ----------\n folder_depth: int\n Path depth to check for folder equality.\n\n Returns\n -------\n list of tuples of str\n \"\"\"\n group_pairs = []\n key_dicoms = list(self.dicom_groups.keys())\n idx = len(key_dicoms)\n while idx > 0:\n group1 = key_dicoms.pop()\n dir_group1 = get_folder_subpath(group1, folder_depth)\n for group in key_dicoms:\n if group.startswith(dir_group1):\n group_pairs.append((group1, group))\n idx -= 1\n\n return group_pairs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nplots the distances between files in the file.", "response": "def plot_file_distances(dist_matrix):\n \"\"\"\n Plots dist_matrix\n\n Parameters\n ----------\n dist_matrix: np.ndarray\n \"\"\"\n import matplotlib.pyplot as plt\n fig = plt.figure()\n ax = fig.add_subplot(111)\n\n ax.matshow(dist_matrix, interpolation='nearest',\n cmap=plt.cm.get_cmap('PuBu'))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nextend the DICOM groups dictionary with the lists of items in indices.", "response": "def merge_groups(self, indices):\n \"\"\"Extend the lists within the DICOM groups dictionary.\n The indices will indicate which list have to be extended by which\n other list.\n\n Parameters\n ----------\n indices: list or tuple of 2 iterables of int, bot having the same len\n The indices of the lists that have to be merged, both iterables\n items will be read pair by pair, the first is the index to the\n list that will be extended with the list of the second index.\n The indices can be constructed with Numpy e.g.,\n indices = np.where(square_matrix)\n \"\"\"\n try:\n merged = merge_dict_of_lists(self.dicom_groups, indices,\n pop_later=True, copy=True)\n self.dicom_groups = merged\n except IndexError:\n raise IndexError('Index out of range to merge DICOM groups.')"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmoves the file groups into a folder.", "response": "def move_to_folder(self, folder_path, groupby_field_name=None):\n \"\"\"Copy the file groups to folder_path. Each group will be copied into\n a subfolder with named given by groupby_field.\n\n Parameters\n ----------\n folder_path: str\n Path to where copy the DICOM files.\n\n groupby_field_name: str\n DICOM field name. Will get the value of this field to name the group\n folder. If empty or None will use the basename of the group key file.\n \"\"\"\n try:\n copy_groups_to_folder(self.dicom_groups, folder_path, groupby_field_name)\n except IOError as ioe:\n raise IOError('Error moving dicom groups to {}.'.format(folder_path)) from ioe"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_unique_field_values_per_group(self, field_name,\n field_to_use_as_key=None):\n \"\"\"Return a dictionary where the key is the group key file path and\n the values are sets of unique values of the field name of all DICOM\n files in the group.\n\n Parameters\n ----------\n field_name: str\n Name of the field to read from all files\n\n field_to_use_as_key: str\n Name of the field to get the value and use as key.\n If None, will use the same key as the dicom_groups.\n\n Returns\n -------\n Dict of sets\n \"\"\"\n unique_vals = DefaultOrderedDict(set)\n for dcmg in self.dicom_groups:\n for f in self.dicom_groups[dcmg]:\n field_val = DicomFile(f).get_attributes(field_name)\n key_val = dcmg\n if field_to_use_as_key is not None:\n try:\n key_val = str(DicomFile(dcmg).get_attributes(field_to_use_as_key))\n except KeyError as ke:\n raise KeyError('Error getting field {} from '\n 'file {}'.format(field_to_use_as_key,\n dcmg)) from ke\n unique_vals[key_val].add(field_val)\n\n return unique_vals", "response": "Get the unique values of a field in a group."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_config_value(name, fallback=None):\n\n cli_config = CLIConfig(SF_CLI_CONFIG_DIR, SF_CLI_ENV_VAR_PREFIX)\n\n return cli_config.get('servicefabric', name, fallback)", "response": "Gets a config value by name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if a config value is set to a valid bool value.", "response": "def get_config_bool(name):\n \"\"\"Checks if a config value is set to a valid bool value.\"\"\"\n\n cli_config = CLIConfig(SF_CLI_CONFIG_DIR, SF_CLI_ENV_VAR_PREFIX)\n return cli_config.getboolean('servicefabric', name, False)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets a config by name to a value.", "response": "def set_config_value(name, value):\n \"\"\"Set a config by name to a value.\"\"\"\n\n cli_config = CLIConfig(SF_CLI_CONFIG_DIR, SF_CLI_ENV_VAR_PREFIX)\n cli_config.set_value('servicefabric', name, value)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the path to certificate related files either a single file path or a tuple.", "response": "def cert_info():\n \"\"\"Path to certificate related files, either a single file path or a\n tuple. In the case of no security, returns None.\"\"\"\n\n sec_type = security_type()\n if sec_type == 'pem':\n return get_config_value('pem_path', fallback=None)\n if sec_type == 'cert':\n cert_path = get_config_value('cert_path', fallback=None)\n key_path = get_config_value('key_path', fallback=None)\n return cert_path, key_path\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_aad_cache(token, cache):\n set_config_value('aad_token', jsonpickle.encode(token))\n set_config_value('aad_cache', jsonpickle.encode(cache))", "response": "Set AAD token cache."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the authentication settings for the current application.", "response": "def set_auth(pem=None, cert=None, key=None, aad=False):\n \"\"\"Set certificate usage paths\"\"\"\n\n if any([cert, key]) and pem:\n raise ValueError('Cannot specify both pem and cert or key')\n\n if any([cert, key]) and not all([cert, key]):\n raise ValueError('Must specify both cert and key')\n\n if pem:\n set_config_value('security', 'pem')\n set_config_value('pem_path', pem)\n elif cert or key:\n set_config_value('security', 'cert')\n set_config_value('cert_path', cert)\n set_config_value('key_path', key)\n elif aad:\n set_config_value('security', 'aad')\n else:\n set_config_value('security', 'none')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef filter_objlist(olist, fieldname, fieldval):\n return [x for x in olist if getattr(x, fieldname) == fieldval]", "response": "Returns a list with of the objects in olist that have a fieldname valued as fieldval"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter_list(lst, filt):\n return [m for s in lst for m in (filt(s),) if m]", "response": "Returns a list of items that pass the filter function filter\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of strings that match the pattern", "response": "def match_list(lst, pattern, group_names=[]):\n \"\"\"\n Parameters\n ----------\n lst: list of str\n\n regex: string\n\n group_names: list of strings\n See re.MatchObject group docstring\n\n Returns\n -------\n list of strings\n Filtered list, with the strings that match the pattern\n \"\"\"\n filtfn = re.compile(pattern).match\n filtlst = filter_list(lst, filtfn)\n if not group_names:\n return [m.string for m in filtlst]\n else:\n return [m.group(group_names) for m in filtlst]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef append_to_keys(adict, preffix):\n return {preffix + str(key): (value if isinstance(value, dict) else value)\n for key, value in list(adict.items())}", "response": "Returns a new dict with the preffix appended to the keys of the input dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_valid_regex(string):\n try:\n re.compile(string)\n is_valid = True\n except re.error:\n is_valid = False\n return is_valid", "response": "Checks whether the given regular expression is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_regex(string):\n is_regex = False\n regex_chars = ['\\\\', '(', '+', '^', '$']\n for c in regex_chars:\n if string.find(c) > -1:\n return is_valid_regex(string)\n return is_regex", "response": "Returns True if the given string is a regular expression False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_fnmatch_regex(string):\n is_regex = False\n regex_chars = ['!', '*', '$']\n for c in regex_chars:\n if string.find(c) > -1:\n return True\n return is_regex", "response": "Returns True if the given string is considered a fnmatch - style regular expression False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef where_is(strings, pattern, n=1, lookup_func=re.match):\n count = 0\n for idx, item in enumerate(strings):\n if lookup_func(pattern, item):\n count += 1\n if count == n:\n return idx\n return -1", "response": "Returns the index of the nth match found of pattern in strings"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_config(output_directory):\n if not op.isdir(output_directory):\n os.makedirs(output_directory)\n\n config_file = op.join(output_directory, \"config.ini\")\n open_file = open(config_file, \"w\")\n open_file.write(\"[BOOL]\\nManualNIfTIConv=0\\n\")\n open_file.close()\n return config_file", "response": "Generate a dcm2nii configuration file that disable the interactive\n mode."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd slice duration and acquisition times to the nifit1 file in nii_file.", "response": "def add_meta_to_nii(nii_file, dicom_file, dcm_tags=''):\n \"\"\" Add slice duration and acquisition times to the headers of the nifit1 files in `nii_file`.\n It will add the repetition time of the DICOM file (field: {0x0018, 0x0080, DS, Repetition Time})\n to the NifTI file as well as any other tag in `dcm_tags`.\n All selected DICOM tags values are set in the `descrip` nifti header field.\n Note that this will modify the header content of `nii_file`.\n\n Parameters\n ----------\n nii_files: str\n Path to the NifTI file to modify.\n\n dicom_file: str\n Paths to the DICOM file from where to get the meta data.\n\n dcm_tags: list of str\n List of tags from the DICOM file to read and store in the nifti file.\n \"\"\"\n # Load a dicom image\n dcmimage = dicom.read_file(dicom_file)\n\n # Load the nifti1 image\n image = nibabel.load(nii_file)\n\n # Check the we have a nifti1 format image\n if not isinstance(image, nibabel.nifti1.Nifti1Image):\n raise Exception(\n \"Only Nifti1 image are supported not '{0}'.\".format(\n type(image)))\n\n # check if dcm_tags is one string, if yes put it in a list:\n if isinstance(dcm_tags, str):\n dcm_tags = [dcm_tags]\n\n # Fill the nifti1 header\n header = image.get_header()\n\n # slice_duration: Time for 1 slice\n repetition_time = float(dcmimage[(\"0x0018\", \"0x0080\")].value)\n header.set_dim_info(slice=2)\n nb_slices = header.get_n_slices()\n # Force round to 0 digit after coma. If more, nibabel completes to\n # 6 digits with random numbers...\n slice_duration = round(repetition_time / nb_slices, 0)\n header.set_slice_duration(slice_duration)\n\n # add free dicom fields\n if dcm_tags:\n content = [\"{0}={1}\".format(name, dcmimage[tag].value)\n for name, tag in dcm_tags]\n free_field = numpy.array(\";\".join(content),\n dtype=header[\"descrip\"].dtype)\n image.get_header()[\"descrip\"] = free_field\n\n # Update the image header\n image.update_header()\n\n # Save the filled image\n nibabel.save(image, nii_file)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting all DICOM files within `work_dir` into one or more NifTi files by calling dcm2nii on this folder. Parameters ---------- work_dir: str Path to the folder that contain the DICOM files arguments: str String containing all the flag arguments for `dcm2nii` CLI. Returns ------- sys_code: int dcm2nii execution return code", "response": "def call_dcm2nii(work_dir, arguments=''):\n \"\"\"Converts all DICOM files within `work_dir` into one or more\n NifTi files by calling dcm2nii on this folder.\n\n Parameters\n ----------\n work_dir: str\n Path to the folder that contain the DICOM files\n\n arguments: str\n String containing all the flag arguments for `dcm2nii` CLI.\n\n Returns\n -------\n sys_code: int\n dcm2nii execution return code\n \"\"\"\n if not op.exists(work_dir):\n raise IOError('Folder {} not found.'.format(work_dir))\n\n cmd_line = 'dcm2nii {0} \"{1}\"'.format(arguments, work_dir)\n log.info(cmd_line)\n return subprocess.check_call(cmd_line, shell=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting DICOM files inside input_dir to Nifti and save the Nifti file in output_dir with a filename prefix.", "response": "def convert_dcm2nii(input_dir, output_dir, filename):\n \"\"\" Call MRICron's `dcm2nii` to convert the DICOM files inside `input_dir`\n to Nifti and save the Nifti file in `output_dir` with a `filename` prefix.\n\n Parameters\n ----------\n input_dir: str\n Path to the folder that contains the DICOM files\n\n output_dir: str\n Path to the folder where to save the NifTI file\n\n filename: str\n Output file basename\n\n Returns\n -------\n filepaths: list of str\n List of file paths created in `output_dir`.\n \"\"\"\n # a few checks before doing the job\n if not op.exists(input_dir):\n raise IOError('Expected an existing folder in {}.'.format(input_dir))\n\n if not op.exists(output_dir):\n raise IOError('Expected an existing output folder in {}.'.format(output_dir))\n\n # create a temporary folder for dcm2nii export\n tmpdir = tempfile.TemporaryDirectory(prefix='dcm2nii_')\n\n # call dcm2nii\n arguments = '-o \"{}\" -i y'.format(tmpdir.name)\n try:\n call_out = call_dcm2nii(input_dir, arguments)\n except:\n raise\n else:\n log.info('Converted \"{}\" to nifti.'.format(input_dir))\n\n # get the filenames of the files that dcm2nii produced\n filenames = glob(op.join(tmpdir.name, '*.nii*'))\n\n # cleanup `filenames`, using only the post-processed (reoriented, cropped, etc.) images by dcm2nii\n cleaned_filenames = remove_dcm2nii_underprocessed(filenames)\n\n # copy files to the output_dir\n filepaths = []\n for srcpath in cleaned_filenames:\n dstpath = op.join(output_dir, filename)\n realpath = copy_w_plus(srcpath, dstpath)\n filepaths.append(realpath)\n\n # copy any other file produced by dcm2nii that is not a NifTI file, e.g., *.bvals, *.bvecs, etc.\n basename = op.basename(remove_ext(srcpath))\n aux_files = set(glob(op.join(tmpdir.name, '{}.*' .format(basename)))) - \\\n set(glob(op.join(tmpdir.name, '{}.nii*'.format(basename))))\n for aux_file in aux_files:\n aux_dstpath = copy_w_ext(aux_file, output_dir, remove_ext(op.basename(realpath)))\n filepaths.append(aux_dstpath)\n\n return filepaths"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a subset of filepaths. Keep only the files that have a basename longer than the others with same suffix.", "response": "def remove_dcm2nii_underprocessed(filepaths):\n \"\"\" Return a subset of `filepaths`. Keep only the files that have a basename longer than the\n others with same suffix.\n This works based on that dcm2nii appends a preffix character for each processing\n step it does automatically in the DICOM to NifTI conversion.\n\n Parameters\n ----------\n filepaths: iterable of str\n\n Returns\n -------\n cleaned_paths: iterable of str\n \"\"\"\n cln_flist = []\n\n # sort them by size\n len_sorted = sorted(filepaths, key=len)\n\n for idx, fpath in enumerate(len_sorted):\n remove = False\n\n # get the basename and the rest of the files\n fname = op.basename(fpath)\n rest = len_sorted[idx+1:]\n\n # check if the basename is in the basename of the rest of the files\n for rest_fpath in rest:\n rest_file = op.basename(rest_fpath)\n if rest_file.endswith(fname):\n remove = True\n break\n\n if not remove:\n cln_flist.append(fpath)\n\n return cln_flist"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntransforming a named tuple into a dictionary", "response": "def dictify(a_named_tuple):\n \"\"\"Transform a named tuple into a dictionary\"\"\"\n return dict((s, getattr(a_named_tuple, s)) for s in a_named_tuple._fields)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef merge_dict_of_lists(adict, indices, pop_later=True, copy=True):\n def check_indices(idxs, x):\n for i in chain(*idxs):\n if i < 0 or i >= x:\n raise IndexError(\"Given indices are out of dict range.\")\n\n check_indices(indices, len(adict))\n\n rdict = adict.copy() if copy else adict\n\n dict_keys = list(rdict.keys())\n for i, j in zip(*indices):\n rdict[dict_keys[i]].extend(rdict[dict_keys[j]])\n\n if pop_later:\n for i, j in zip(*indices):\n rdict.pop(dict_keys[j], '')\n\n return rdict", "response": "Extend the within a dict of lists."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef append_dict_values(list_of_dicts, keys=None):\n if keys is None:\n keys = list(list_of_dicts[0].keys())\n\n dict_of_lists = DefaultOrderedDict(list)\n for d in list_of_dicts:\n for k in keys:\n dict_of_lists[k].append(d[k])\n return dict_of_lists", "response": "Returns a dict of lists from a list of dicts with the same keys and append them to the output dict."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef import_pyfile(filepath, mod_name=None):\n import sys\n if sys.version_info.major == 3:\n import importlib.machinery\n loader = importlib.machinery.SourceFileLoader('', filepath)\n mod = loader.load_module(mod_name)\n else:\n import imp\n mod = imp.load_source(mod_name, filepath)\n\n return mod", "response": "Imports the contents of filepath as a Python module."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncopy the files in the file tree map to destpath.", "response": "def copy(configfile='', destpath='', overwrite=False, sub_node=''):\n \"\"\"Copies the files in the built file tree map\n to despath.\n\n :param configfile: string\n Path to the FileTreeMap config file\n\n :param destpath: string\n Path to the files destination\n\n :param overwrite: bool\n Overwrite files if they already exist.\n\n :param sub_node: string\n Tree map configuration sub path.\n Will copy only the contents within this sub-node\n\n \"\"\"\n log.info('Running {0} {1} {2}'.format(os.path.basename(__file__),\n whoami(),\n locals()))\n\n assert(os.path.isfile(configfile))\n\n if os.path.exists(destpath):\n if os.listdir(destpath):\n raise FolderAlreadyExists('Folder {0} already exists. Please clean '\n 'it or change destpath.'.format(destpath))\n else:\n log.info('Creating folder {0}'.format(destpath))\n path(destpath).makedirs_p()\n\n from boyle.files.file_tree_map import FileTreeMap\n file_map = FileTreeMap()\n\n try:\n file_map.from_config_file(configfile)\n except Exception as e:\n raise FileTreeMapError(str(e))\n\n if sub_node:\n sub_map = file_map.get_node(sub_node)\n if not sub_map:\n raise FileTreeMapError('Could not find sub node '\n '{0}'.format(sub_node))\n\n file_map._filetree = {}\n file_map._filetree[sub_node] = sub_map\n\n try:\n file_map.copy_to(destpath, overwrite=overwrite)\n except Exception as e:\n raise FileTreeMapError(str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns ndarray of references in self. reflst whose self id is None.", "response": "def get_noneid_references(self):\n \"\"\"\n Returns\n -------\n ndarray\n Array of references in self.reflst whose self id is None.\n \"\"\"\n #return [self.reflst[idx] for idx, idval in enumerate(self) if idval is None]\n try:\n nun = np.array(None).astype(self.dtype)\n return np.array(self.reflst)[self == nun]\n except:\n nun = None\n return np.array(self.reflst)[self is None]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting a table of general vs. reference names.", "response": "def _print_general_vs_table(self, idset1, idset2):\n \"\"\"\n :param idset1:\n :param idset2:\n \"\"\"\n ref1name = ''\n set1_hasref = isinstance(idset1, idset_with_reference)\n if set1_hasref:\n ref1arr = np.array(idset1.reflst)\n ref1name = idset1.refname\n\n ref2name = ref1name\n set2_hasref = isinstance(idset2, idset_with_reference)\n if set2_hasref:\n ref2arr = np.array(idset2.reflst)\n ref2name = idset2.refname\n else:\n ref2name = ref1name\n\n #First show a general table\n hdr11 = '{0} > {1}'.format(idset1.name, idset2.name)\n hdr12 = '{0} > {1} {2}'.format(idset1.name, idset2.name, ref2name)\n hdr13 = '{0} < {1}'.format(idset1.name, idset2.name)\n hdr14 = '{0} < {1} {2}'.format(idset1.name, idset2.name, ref1name)\n table = [[hdr11, hdr12, hdr13, hdr14]]\n\n set1 = set(idset1)\n set2 = set(idset2)\n row11 = list(set1 - set2)\n if set1_hasref:\n row12 = [ref1arr[np.where(idset1 == nom)][0] for nom in row11]\n else:\n row12 = ['Not found' for _ in row11]\n\n row13 = list(set2 - set1)\n if set2_hasref:\n row14 = [ref2arr[np.where(idset2 == nom)][0] for nom in row13]\n else:\n row14 = ['Not found' for _ in row13]\n\n tablst = self._tabulate_4_lists(row11, row12, row13, row14)\n table.extend(tablst)\n\n if len(table) > 1:\n print(tabulate(table, headers='firstrow'))\n print('\\n')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _print_foreign_repetition_table(self, idset1, idset2):\n\n assert(isinstance(idset1, idset_with_reference))\n assert(isinstance(idset2, idset))\n\n reps = idset2.get_repetitions()\n if len(reps) < 1:\n return\n\n refs = np.array(idset1.reflst)\n table = [['{0} {1} values of repetitions in {2}'.format(idset1.name,\n idset1.refname,\n idset2.name),\n '']]\n\n for rep in reps:\n if np.any(idset1 == rep):\n matches = refs[np.where(idset1 == rep)]\n myrep = rep\n for m in matches:\n table.append([myrep, m])\n myrep = ''\n\n print(tabulate(table, headers='firstrow'))\n print('\\n')", "response": "Print the foreign repetitions table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef print_compare_idsets_one_ref(self, idset1_name, idset2_name):\n try:\n idset1 = self[idset1_name]\n idset2 = self[idset2_name]\n except KeyError as ke:\n log.error('Error compare_idsets: getting keys {0} and {1}'.format(idset1_name,\n idset2_name))\n import sys, pdb\n pdb.post_mortem(sys.exc_info()[2])\n raise\n\n assert(isinstance(idset1, idset_with_reference))\n assert(isinstance(idset2, idset))\n\n self._print_general_vs_table(idset1, idset2)\n self._print_foreign_repetition_table(idset1, idset2)", "response": "Print the general vs foreign repetition table for two idsets."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the input. sav file into other format.", "response": "def convert_sav(inputfile, outputfile=None, method='rpy2', otype='csv'):\n \"\"\" Transforms the input .sav SPSS file into other format.\n If you don't specify an outputfile, it will use the\n inputfile and change its extension to .csv\n \"\"\"\n assert(os.path.isfile(inputfile))\n assert(method=='rpy2' or method=='savread')\n\n if method == 'rpy2':\n df = sav_to_pandas_rpy2(inputfile)\n elif method == 'savread':\n df = sav_to_pandas_savreader(inputfile)\n\n otype_exts = {'csv': '.csv', \n 'hdf': '.h5', \n 'stata': '.dta',\n 'json': '.json',\n 'pickle': '.pickle',\n 'excel': '.xls',\n 'html': '.html'}\n\n if outputfile is None:\n outputfile = inputfile.replace(path(inputfile).ext, '')\n\n outputfile = add_extension_if_needed(outputfile, otype_exts[otype])\n\n if otype == 'csv':\n df.to_csv(outputfile)\n elif otype == 'hdf':\n df.to_hdf(outputfile, os.path.basename(outputfile))\n elif otype == 'stata':\n df.to_stata(outputfile)\n elif otype == 'json':\n df.to_json(outputfile)\n elif otype == 'pickle':\n df.to_pickle(outputfile)\n elif otype == 'excel':\n df.to_excel(outputfile)\n elif otype == 'html':\n df.to_html(outputfile)\n else:\n df.to_csv(outputfile)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_mask(image, allow_empty=True):\n img = check_img(image, make_it_3d=True)\n values = np.unique(img.get_data())\n\n if len(values) == 1:\n # We accept a single value if it is not 0 (full true mask).\n if values[0] == 0 and not allow_empty:\n raise ValueError('Given mask is invalid because it masks all data')\n\n elif len(values) == 2:\n # If there are 2 different values, one of them must be 0 (background)\n if 0 not in values:\n raise ValueError('Background of the mask must be represented with 0.'\n ' Given mask contains: {}.'.format(values))\n\n elif len(values) != 2:\n # If there are more than 2 values, the mask is invalid\n raise ValueError('Given mask is not made of 2 values: {}. '\n 'Cannot interpret as true or false'.format(values))\n\n return nib.Nifti1Image(as_ndarray(get_img_data(img), dtype=bool), img.get_affine(), img.get_header())", "response": "Load a Nifti mask volume."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_mask_data(image, allow_empty=True):\n mask = load_mask(image, allow_empty=allow_empty)\n return get_img_data(mask), mask.get_affine()", "response": "Loads a Nifti mask volume and returns its data matrix as boolean and affine."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef union_mask(filelist):\n firstimg = check_img(filelist[0])\n mask = np.zeros_like(firstimg.get_data())\n\n # create space for all features and read from subjects\n try:\n for volf in filelist:\n roiimg = check_img(volf)\n check_img_compatibility(firstimg, roiimg)\n mask += get_img_data(roiimg)\n except Exception as exc:\n raise ValueError('Error joining mask {} and {}.'.format(repr_imgs(firstimg), repr_imgs(volf))) from exc\n else:\n return as_ndarray(mask > 0, dtype=bool)", "response": "Returns a binarised mask with the union of the files in filelist."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads a Nifti file nii_file and a mask Nifti file and apply the mask to the nii_file.", "response": "def apply_mask(image, mask_img):\n \"\"\"Read a Nifti file nii_file and a mask Nifti file.\n Returns the voxels in nii_file that are within the mask, the mask indices\n and the mask shape.\n\n Parameters\n ----------\n image: img-like object or boyle.nifti.NeuroImage or str\n Can either be:\n - a file path to a Nifti image\n - any object with get_data() and get_affine() methods, e.g., nibabel.Nifti1Image.\n If niimg is a string, consider it as a path to Nifti image and\n call nibabel.load on it. If it is an object, check if get_data()\n and get_affine() methods are present, raise TypeError otherwise.\n\n mask_img: img-like object or boyle.nifti.NeuroImage or str\n 3D mask array: True where a voxel should be used.\n See img description.\n\n Returns\n -------\n vol[mask_indices], mask_indices\n\n Note\n ----\n nii_file and mask_file must have the same shape.\n\n Raises\n ------\n NiftiFilesNotCompatible, ValueError\n \"\"\"\n img = check_img(image)\n mask = check_img(mask_img)\n check_img_compatibility(img, mask)\n\n vol = img.get_data()\n mask_data, _ = load_mask_data(mask)\n\n return vol[mask_data], mask_data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef apply_mask_4d(image, mask_img): # , smooth_mm=None, remove_nans=True):\n img = check_img(image)\n mask = check_img(mask_img)\n check_img_compatibility(img, mask, only_check_3d=True)\n\n vol = get_data(img)\n series, mask_data = _apply_mask_to_4d_data(vol, mask)\n return series, mask_data", "response": "Read a Nifti file nii_file and a mask Nifti file and apply mask to the session series."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntransform a given vector to a volume.", "response": "def vector_to_volume(arr, mask, order='C'):\n \"\"\"Transform a given vector to a volume. This is a reshape function for\n 3D flattened and maybe masked vectors.\n\n Parameters\n ----------\n arr: np.array\n 1-Dimensional array\n\n mask: numpy.ndarray\n Mask image. Must have 3 dimensions, bool dtype.\n\n Returns\n -------\n np.ndarray\n \"\"\"\n if mask.dtype != np.bool:\n raise ValueError(\"mask must be a boolean array\")\n\n if arr.ndim != 1:\n raise ValueError(\"vector must be a 1-dimensional array\")\n\n if arr.ndim == 2 and any(v == 1 for v in arr.shape):\n log.debug('Got an array of shape {}, flattening for my purposes.'.format(arr.shape))\n arr = arr.flatten()\n\n volume = np.zeros(mask.shape[:3], dtype=arr.dtype, order=order)\n volume[mask] = arr\n return volume"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef matrix_to_4dvolume(arr, mask, order='C'):\n if mask.dtype != np.bool:\n raise ValueError(\"mask must be a boolean array\")\n\n if arr.ndim != 2:\n raise ValueError(\"X must be a 2-dimensional array\")\n\n if mask.sum() != arr.shape[0]:\n # raise an error if the shape of arr is not what expected\n raise ValueError('Expected arr of shape ({}, samples). Got {}.'.format(mask.sum(), arr.shape))\n\n data = np.zeros(mask.shape + (arr.shape[1],), dtype=arr.dtype,\n order=order)\n data[mask, :] = arr\n return data", "response": "Transform a given vector to a 4D volume. This is a reshape function for\n 4D flattened masked matrices where the second dimension of the matrix corresponds to the original 4th dimension."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef niftilist_mask_to_array(img_filelist, mask_file=None, outdtype=None):\n img = check_img(img_filelist[0])\n if not outdtype:\n outdtype = img.dtype\n\n mask_data, _ = load_mask_data(mask_file)\n indices = np.where (mask_data)\n\n mask = check_img(mask_file)\n\n outmat = np.zeros((len(img_filelist), np.count_nonzero(mask_data)),\n dtype=outdtype)\n\n for i, img_item in enumerate(img_filelist):\n img = check_img(img_item)\n if not are_compatible_imgs(img, mask):\n raise NiftiFilesNotCompatible(repr_imgs(img), repr_imgs(mask_file))\n\n vol = get_img_data(img)\n outmat[i, :] = vol[indices]\n\n return outmat, mask_data", "response": "Given a list of absolute paths to nifti files creates a Numpy array containing the masked data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create(_):\n\n endpoint = client_endpoint()\n\n if not endpoint:\n raise CLIError(\"Connection endpoint not found. \"\n \"Before running sfctl commands, connect to a cluster using \"\n \"the 'sfctl cluster select' command.\")\n\n no_verify = no_verify_setting()\n\n if security_type() == 'aad':\n auth = AdalAuthentication(no_verify)\n else:\n cert = cert_info()\n ca_cert = ca_cert_info()\n auth = ClientCertAuthentication(cert, ca_cert, no_verify)\n\n return ServiceFabricClientAPIs(auth, base_url=endpoint)", "response": "Create a client for Service Fabric APIs."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef aggregate(self, clazz, new_col, *args):\n if is_callable(clazz) and not is_none(new_col) and has_elements(*args):\n return self.__do_aggregate(clazz, new_col, *args)", "response": "Aggregate the rows of the DataFrame into a single value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef subset(self, *args):\n cols = {}\n for k in self.colnames:\n if k in args:\n cols[str(k)] = \\\n self.__data_columns[self.colnames.index(k)].values\n return DataFrame(**cols)", "response": "Returns a subset of the DataFrame with only the columns you selected\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef modify(self, clazz, new_col, *args):\n if is_callable(clazz) and not is_none(new_col) and has_elements(*args):\n return self.__do_modify(clazz, new_col, *args)", "response": "Modify some columns i. e. apply a function and add the\notos to the table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef aggregate(*args):\n\n if args and isinstance(args[0], dataframe.DataFrame):\n return args[0].aggregate(args[1], args[2], *args[3:])\n elif not args:\n raise ValueError(\"No arguments provided\")\n else:\n return pipeable.Pipeable(pipeable.PipingMethod.AGGREGATE, *args)", "response": "This method returns a dataframe object that is piped into the base object of the base object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef modify(*args):\n\n if args and isinstance(args[0], dataframe.DataFrame):\n return args[0].modify(args[1], args[2], *args[3:])\n elif not args:\n raise ValueError(\"No arguments provided\")\n else:\n return pipeable.Pipeable(pipeable.PipingMethod.MODIFY, *args)", "response": "A function that takes either a dataframe and a tuple of arguments required for modification."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _escape_char(c, escape_char=ESCAPE_CHAR):\n buf = []\n for byte in c.encode('utf8'):\n buf.append(escape_char)\n buf.append('%X' % _ord(byte))\n return ''.join(buf)", "response": "Escape a single character"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nescapes a string so that it only contains characters in a safe set.", "response": "def escape(to_escape, safe=SAFE, escape_char=ESCAPE_CHAR, allow_collisions=False):\n \"\"\"Escape a string so that it only contains characters in a safe set.\n\n Characters outside the safe list will be escaped with _%x_,\n where %x is the hex value of the character.\n\n If `allow_collisions` is True, occurrences of `escape_char`\n in the input will not be escaped.\n\n In this case, `unescape` cannot be used to reverse the transform\n because occurrences of the escape char in the resulting string are ambiguous.\n Only use this mode when:\n\n 1. collisions cannot occur or do not matter, and\n 2. unescape will never be called.\n\n .. versionadded: 1.0\n allow_collisions argument.\n Prior to 1.0, behavior was the same as allow_collisions=False (default).\n\n \"\"\"\n if isinstance(to_escape, bytes):\n # always work on text\n to_escape = to_escape.decode('utf8')\n\n if not isinstance(safe, set):\n safe = set(safe)\n\n if allow_collisions:\n safe.add(escape_char)\n elif escape_char in safe:\n # escape char can't be in safe list\n safe.remove(escape_char)\n\n chars = []\n for c in to_escape:\n if c in safe:\n chars.append(c)\n else:\n chars.append(_escape_char(c, escape_char))\n return u''.join(chars)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndetermining whether this backend can send a notification to the given user and notice_type.", "response": "def can_send(self, user, notice_type):\n \"\"\"\n Determines whether this backend is allowed to send a notification to\n the given user and notice_type.\n \"\"\"\n from notification.models import NoticeSetting\n return NoticeSetting.for_user(user, notice_type, self.medium_id).send"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a dictionary with the format identifier as the key. The values are a dictionary with the format identifier as the key. The values are a string containing the rendered templates with the given context.", "response": "def get_formatted_messages(self, formats, label, context):\n \"\"\"\n Returns a dictionary with the format identifier as the key. The values are\n are fully rendered templates with the given context.\n \"\"\"\n format_templates = {}\n for fmt in formats:\n # conditionally turn off autoescaping for .txt extensions in format\n if fmt.endswith(\".txt\"):\n context.autoescape = False\n format_templates[fmt] = render_to_string((\n \"notification/%s/%s\" % (label, fmt),\n \"notification/%s\" % fmt), context_instance=context)\n return format_templates"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef copy_attributes(source, destination, ignore_patterns=[]):\n for attr in _wildcard_filter(dir(source), *ignore_patterns):\n setattr(destination, attr, getattr(source, attr))", "response": "Copy the attributes from a source object to a destination object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a DataFrameRow given its index.", "response": "def row(self, idx):\n \"\"\"\n Returns DataFrameRow of the DataFrame given its index.\n\n :param idx: the index of the row in the DataFrame.\n :return: returns a DataFrameRow\n \"\"\"\n return DataFrameRow(idx, [x[idx] for x in self], self.colnames)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the list of all the notice settings for each medium.", "response": "def notice_settings(request):\n \"\"\"\n The notice settings view.\n\n Template: :template:`notification/notice_settings.html`\n\n Context:\n\n notice_types\n A list of all :model:`notification.NoticeType` objects.\n\n notice_settings\n A dictionary containing ``column_headers`` for each ``NOTICE_MEDIA``\n and ``rows`` containing a list of dictionaries: ``notice_type``, a\n :model:`notification.NoticeType` object and ``cells``, a list of\n tuples whose first value is suitable for use in forms and the second\n value is ``True`` or ``False`` depending on a ``request.POST``\n variable called ``form_label``, whose valid value is ``on``.\n \"\"\"\n notice_types = NoticeType.objects.all()\n settings_table = []\n for notice_type in notice_types:\n settings_row = []\n for medium_id, medium_display in NOTICE_MEDIA:\n form_label = \"%s_%s\" % (notice_type.label, medium_id)\n setting = NoticeSetting.for_user(request.user, notice_type, medium_id)\n if request.method == \"POST\":\n if request.POST.get(form_label) == \"on\":\n if not setting.send:\n setting.send = True\n setting.save()\n else:\n if setting.send:\n setting.send = False\n setting.save()\n settings_row.append((form_label, setting.send))\n settings_table.append({\"notice_type\": notice_type, \"cells\": settings_row})\n\n if request.method == \"POST\":\n next_page = request.POST.get(\"next_page\", \".\")\n return HttpResponseRedirect(next_page)\n\n settings = {\n \"column_headers\": [medium_display for medium_id, medium_display in NOTICE_MEDIA],\n \"rows\": settings_table,\n }\n\n return render_to_response(\"notification/notice_settings.html\", {\n \"notice_types\": notice_types,\n \"notice_settings\": settings,\n }, context_instance=RequestContext(request))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef query(self, input = '', params = {}):\n # Get and construct query parameters\n # Default parameters\n payload = {'input': input,\n 'appid': self.appid}\n # Additional parameters (from params), formatted for url\n for key, value in params.items():\n # Check if value is list or tuple type (needs to be comma joined)\n if isinstance(value, (list, tuple)):\n payload[key] = ','.join(value)\n else:\n payload[key] = value\n\n # Catch any issues with connecting to Wolfram Alpha API\n try:\n r = requests.get(\"http://api.wolframalpha.com/v2/query\", params=payload)\n\n # Raise Exception (to be returned as error)\n if r.status_code != 200:\n raise Exception('Invalid response status code: %s' % (r.status_code))\n if r.encoding != 'utf-8':\n raise Exception('Invalid encoding: %s' % (r.encoding))\n\n except Exception, e:\n return Result(error = e)\n\n return Result(xml = r.text)", "response": "Query Wolfram Alpha and return a Result object"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pods(self):\n # Return empty list if xml_tree is not defined (error Result object)\n if not self.xml_tree:\n return []\n\n # Create a Pod object for every pod group in xml\n return [Pod(elem) for elem in self.xml_tree.findall('pod')]", "response": "Return list of all Pod objects in result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a dictionary of available formats corresponding to a list of the values AttributeNames", "response": "def format(self):\n \"\"\"\n Dictionary of available formats, corresponding to a list of the values\n Example: pod.format['plaintext'] will return a list of every plaintext\n content in the pod's subpods\n \"\"\"\n formats = {}\n\n # Iterate through all the tags (formats) in subpods\n # 'state' is a tag but not an acceptable format\n for subpod in self.root.findall('subpod'):\n\n # elem will be a specific format\n for elem in list(subpod):\n\n # skip any subpod state xml groups (not a format)\n if elem.tag == 'state':\n continue\n\n # Content of elem (specific format)\n content = elem.text\n\n # img needs special content packaging\n if elem.tag == 'img':\n content = {'url': elem.get('src'),\n 'alt': elem.get('alt'),\n 'title': elem.get('title'),\n 'width': int(elem.get('width', 0)),\n 'height': int(elem.get('height', 0))}\n\n # Create / append to return dict\n if elem.tag not in formats:\n formats[elem.tag] = [content]\n else:\n formats[elem.tag].append(content)\n\n return formats"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind a node in the tree.", "response": "def find(self, *args):\n \"\"\"\n Find a node in the tree. If the node is not found it is added first and then returned.\n\n :param args: a tuple\n :return: returns the node\n \"\"\"\n curr_node = self.__root\n return self.__traverse(curr_node, 0, *args)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_notification_language(user):\n if getattr(settings, \"NOTIFICATION_LANGUAGE_MODULE\", False):\n try:\n app_label, model_name = settings.NOTIFICATION_LANGUAGE_MODULE.split(\".\")\n model = models.get_model(app_label, model_name)\n # pylint: disable-msg=W0212\n language_model = model._default_manager.get(user__id__exact=user.id)\n if hasattr(language_model, \"language\"):\n return language_model.language\n except (ImportError, ImproperlyConfigured, model.DoesNotExist):\n raise LanguageStoreNotAvailable\n raise LanguageStoreNotAvailable", "response": "Returns the language of the user s notification. Raises LanguageStoreNotAvailable if this site does not use translated\n notifications."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send_now(users, label, extra_context=None, sender=None):\n sent = False\n if extra_context is None:\n extra_context = {}\n\n notice_type = NoticeType.objects.get(label=label)\n\n current_language = get_language()\n\n for user in users:\n # get user language for user from language store defined in\n # NOTIFICATION_LANGUAGE_MODULE setting\n try:\n language = get_notification_language(user)\n except LanguageStoreNotAvailable:\n language = None\n\n if language is not None:\n # activate the user's language\n activate(language)\n\n for backend in NOTIFICATION_BACKENDS.values():\n if backend.can_send(user, notice_type):\n backend.deliver(user, sender, notice_type, extra_context)\n sent = True\n\n # reset environment to original language\n activate(current_language)\n return sent", "response": "Sends a new notice for the given list of users."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nqueues the user notifications in NoticeQueueBatch.", "response": "def queue(users, label, extra_context=None, sender=None):\n \"\"\"\n Queue the notification in NoticeQueueBatch. This allows for large amounts\n of user notifications to be deferred to a seperate process running outside\n the webserver.\n \"\"\"\n if extra_context is None:\n extra_context = {}\n if isinstance(users, QuerySet):\n users = [row[\"pk\"] for row in users.values(\"pk\")]\n else:\n users = [user.pk for user in users]\n notices = []\n for user in users:\n notices.append((user, label, extra_context, sender))\n NoticeQueueBatch(pickled_data=base64.b64encode(pickle.dumps(notices))).save()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites tersoff potential file from dictionary of parameters to string", "response": "def write_tersoff_potential(parameters):\n \"\"\"Write tersoff potential file from parameters to string\n\n Parameters\n ----------\n parameters: dict\n keys are tuple of elements with the values being the parameters length 14\n \"\"\"\n lines = []\n for (e1, e2, e3), params in parameters.items():\n if len(params) != 14:\n raise ValueError('tersoff three body potential expects 14 parameters')\n lines.append(' '.join([e1, e2, e3] + ['{:16.8g}'.format(_) for _ in params]))\n return '\\n'.join(lines)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef subset(self, *args):\n args = list(args)\n args.extend([x for x in\n self.__grouping.grouping_colnames if x not in args])\n return GroupedDataFrame(self.__grouping.ungroup().subset(*args),\n *self.__grouping.grouping_colnames)", "response": "Subset only some of the columns of the DataFrame."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmodifies some columns of a class that extends class Callable and add the result to the table.", "response": "def modify(self, clazz, new_col, *args):\n \"\"\"\n Modify some columns (i.e. apply a function)\n and add the result to the table.\n\n :param clazz: name of a class that extends class Callable\n :type clazz: class\n :param new_col: name of the new column\n :type new_col: str\n :param args: list of column names of the object that\n function should be applied to\n :type args: tuple\n :return: returns a new GroupedDataFrame object with the modified\n values, i.e. the new column of values\n :rtype: GroupedDataFrame\n \"\"\"\n if is_callable(clazz) \\\n and not is_none(new_col) \\\n and has_elements(*args) \\\n and is_disjoint(self.__grouping.grouping_colnames,\n args,\n __DISJOINT_SETS_ERROR__):\n return self.__do_modify(clazz, new_col, *args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\naggregating the rows of each group into a single value.", "response": "def aggregate(self, clazz, new_col, *args):\n \"\"\"\n Aggregate the rows of each group into a single value.\n\n :param clazz: name of a class that extends class Callable\n :type clazz: class\n :param new_col: name of the new column\n :type new_col: str\n :param args: list of column names of the object that\n function should be applied to\n :type args: varargs\n :return: returns a new dataframe object with the aggregated value\n :rtype: DataFrame\n \"\"\"\n if is_callable(clazz) \\\n and not is_none(new_col) \\\n and has_elements(*args) \\\n and is_disjoint(self.__grouping.grouping_colnames,\n args,\n __DISJOINT_SETS_ERROR__):\n return self.__do_aggregate(clazz, new_col, *args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmake LTSV Reader for reading selected labels.", "response": "def reader(ltsvfile, labels=None):\n \"\"\"Make LTSV Reader for reading selected labels.\n\n :param ltsvfile: iterable of lines.\n :param labels: sequence of labels. (optional)\n :return: generator of record in [[label, value], ...] form.\n \"\"\"\n label_pattern = re.compile(r\"^[0-9A-Za-z_.-]+:\")\n\n if labels is not None:\n prefixes = tuple(L + ':' for L in labels\n if label_pattern.match(L + ':'))\n for record in ltsvfile:\n record = record.rstrip('\\r\\n')\n yield [x.split(':', 1) for x in record.split('\\t')\n if x.startswith(prefixes)]\n return\n\n for record in ltsvfile:\n record = record.rstrip('\\r\\n')\n yield [x.split(':', 1) for x in record.split('\\t')\n if label_pattern.match(x)]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmaking LTSV Reader for reading selected labels.", "response": "def DictReader(ltsvfile, labels=None, dict_type=dict):\n \"\"\"Make LTSV Reader for reading selected labels.\n\n :param ltsvfile: iterable of lines.\n :param labels: sequence of labels.\n :return: generator of record in {label: value, ...} form.\n \"\"\"\n for rec in reader(ltsvfile, labels):\n yield dict_type(rec)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if elements of set2 are not in set1.", "response": "def is_disjoint(set1, set2, warn):\n \"\"\"\n Checks if elements of set2 are in set1.\n\n :param set1: a set of values\n :param set2: a set of values\n :param warn: the error message that should be thrown\n when the sets are NOT disjoint\n :return: returns true no elements of set2 are in set1\n \"\"\"\n for elem in set2:\n if elem in set1:\n raise ValueError(warn)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if all elements from set2 are in set1.", "response": "def contains_all(set1, set2, warn):\n \"\"\"\n Checks if all elements from set2 are in set1.\n\n :param set1: a set of values\n :param set2: a set of values\n :param warn: the error message that should be thrown \n when the sets are not containd\n :return: returns true if all values of set2 are in set1\n \"\"\"\n for elem in set2:\n if elem not in set1:\n raise ValueError(warn)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_XML(self):\n marcxml_template = \"\"\"\n$LEADER\n$CONTROL_FIELDS\n$DATA_FIELDS\n\n\"\"\"\n\n oai_template = \"\"\"\n\n\n$LEADER$CONTROL_FIELDS\n$DATA_FIELDS\n\n\n\n\"\"\"\n\n # serialize leader, if it is present and record is marc xml\n leader = self.leader if self.leader is not None else \"\"\n if leader: # print only visible leaders\n leader = \"\" + leader + \"\"\n\n # discard leader for oai\n if self.oai_marc:\n leader = \"\"\n\n # serialize\n xml_template = oai_template if self.oai_marc else marcxml_template\n xml_output = Template(xml_template).substitute(\n LEADER=leader.strip(),\n CONTROL_FIELDS=self._serialize_ctl_fields().strip(),\n DATA_FIELDS=self._serialize_data_fields().strip()\n )\n\n return xml_output", "response": "Serialize object back to XML string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse_string(self, xml):\n if not isinstance(xml, HTMLElement):\n xml = dhtmlparser.parseString(str(xml))\n\n # check if there are any records\n record = xml.find(\"record\")\n if not record:\n raise ValueError(\"There is no in your MARC XML document!\")\n record = record[0]\n\n self.oai_marc = len(record.find(\"oai_marc\")) > 0\n\n # leader is separate only in marc21\n if not self.oai_marc:\n leader = record.find(\"leader\")\n if len(leader) >= 1:\n self.leader = leader[0].getContent()\n\n # parse body in respect of OAI MARC format possibility\n if self.oai_marc:\n self._parse_control_fields(record.find(\"fixfield\"), \"id\")\n self._parse_data_fields(record.find(\"varfield\"), \"id\", \"label\")\n else:\n self._parse_control_fields(record.find(\"controlfield\"), \"tag\")\n self._parse_data_fields(record.find(\"datafield\"), \"tag\", \"code\")\n\n # for backward compatibility of MARC XML with OAI\n if self.oai_marc and \"LDR\" in self.controlfields:\n self.leader = self.controlfields[\"LDR\"]", "response": "Parse MARC XML document to dicts which are contained in\n self. controlfields and self. datafields."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the control fields.", "response": "def _parse_control_fields(self, fields, tag_id=\"tag\"):\n \"\"\"\n Parse control fields.\n\n Args:\n fields (list): list of HTMLElements\n tag_id (str): parameter name, which holds the information, about\n field name this is normally \"tag\", but in case of\n oai_marc \"id\".\n \"\"\"\n for field in fields:\n params = field.params\n\n # skip tags without parameters\n if tag_id not in params:\n continue\n\n self.controlfields[params[tag_id]] = field.getContent().strip()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing data fields. Args: fields (list): of HTMLElements tag_id (str): parameter name, which holds the information, about field name this is normally \"tag\", but in case of oai_marc \"id\" sub_id (str): id of parameter, which holds informations about subfield name this is normally \"code\" but in case of oai_marc \"label\"", "response": "def _parse_data_fields(self, fields, tag_id=\"tag\", sub_id=\"code\"):\n \"\"\"\n Parse data fields.\n\n Args:\n fields (list): of HTMLElements\n tag_id (str): parameter name, which holds the information, about\n field name this is normally \"tag\", but in case of\n oai_marc \"id\"\n sub_id (str): id of parameter, which holds informations about\n subfield name this is normally \"code\" but in case of\n oai_marc \"label\"\n\n \"\"\"\n for field in fields:\n params = field.params\n\n if tag_id not in params:\n continue\n\n # take care of iX/indX (indicator) parameters\n field_repr = OrderedDict([\n [self.i1_name, params.get(self.i1_name, \" \")],\n [self.i2_name, params.get(self.i2_name, \" \")],\n ])\n\n # process all subfields\n for subfield in field.find(\"subfield\"):\n if sub_id not in subfield.params:\n continue\n\n content = MARCSubrecord(\n val=subfield.getContent().strip(),\n i1=field_repr[self.i1_name],\n i2=field_repr[self.i2_name],\n other_subfields=field_repr\n )\n\n # add or append content to list of other contents\n code = subfield.params[sub_id]\n if code in field_repr:\n field_repr[code].append(content)\n else:\n field_repr[code] = [content]\n\n tag = params[tag_id]\n if tag in self.datafields:\n self.datafields[tag].append(field_repr)\n else:\n self.datafields[tag] = [field_repr]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd new control field value with under name into control field dictionary.", "response": "def add_ctl_field(self, name, value):\n \"\"\"\n Add new control field `value` with under `name` into control field\n dictionary :attr:`controlfields`.\n \"\"\"\n if len(name) != 3:\n raise ValueError(\"name parameter have to be exactly 3 chars long!\")\n\n self.controlfields[name] = value"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_data_field(self, name, i1, i2, subfields_dict):\n if i1 not in self.valid_i_chars:\n raise ValueError(\"Invalid i1 parameter '\" + i1 + \"'!\")\n if i2 not in self.valid_i_chars:\n raise ValueError(\"Invalid i2 parameter '\" + i2 + \"'!\")\n\n if len(name) != 3:\n raise ValueError(\n \"`name` parameter have to be exactly 3 chars long!\"\n )\n if not subfields_dict:\n raise ValueError(\n \"`subfields_dict` have to contain something!\"\n )\n if not isinstance(subfields_dict, dict):\n raise ValueError(\n \"`subfields_dict` parameter has to be dict instance!\"\n )\n\n # check local keys, convert strings to MARCSubrecord instances\n subrecords = []\n for key, val in subfields_dict.items():\n if len(key) > 1:\n raise KeyError(\n \"`subfields_dict` can be only one character long!\"\n )\n\n # convert other values to lists\n if not isinstance(val, list):\n val = [val]\n\n subfields = map(\n lambda x: MARCSubrecord(x, i1, i2, None),\n val\n )\n\n subfields_dict[key] = subfields\n subrecords.extend(subfields)\n\n # save i/ind values\n subfields_dict[self.i1_name] = i1\n subfields_dict[self.i2_name] = i2\n\n # append dict, or add new dict into self.datafields\n if name in self.datafields:\n self.datafields[name].append(subfields_dict)\n else:\n self.datafields[name] = [subfields_dict]\n\n # to each subrecord add reference to list of all subfields in this\n # datafield\n other_subfields = self.datafields[name]\n for record in subrecords:\n record.other_subfields = other_subfields", "response": "Add new datafield into the MARC record."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_ctl_field(self, controlfield, alt=None):\n if not alt:\n return self.controlfields[controlfield]\n\n return self.controlfields.get(controlfield, alt)", "response": "Returns the value of a given controlfield."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the data records for a specific data field and subfield.", "response": "def getDataRecords(self, datafield, subfield, throw_exceptions=True):\n \"\"\"\n .. deprecated::\n Use :func:`get_subfields` instead.\n \"\"\"\n return self.get_subfields(\n datafield=datafield,\n subfield=subfield,\n exception=throw_exceptions\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn content of given subfield in datafield.", "response": "def get_subfields(self, datafield, subfield, i1=None, i2=None,\n exception=False):\n \"\"\"\n Return content of given `subfield` in `datafield`.\n\n Args:\n datafield (str): Section name (for example \"001\", \"100\", \"700\").\n subfield (str): Subfield name (for example \"a\", \"1\", etc..).\n i1 (str, default None): Optional i1/ind1 parameter value, which\n will be used for search.\n i2 (str, default None): Optional i2/ind2 parameter value, which\n will be used for search.\n exception (bool): If ``True``, :exc:`~exceptions.KeyError` is\n raised when method couldn't found given `datafield` /\n `subfield`. If ``False``, blank array ``[]`` is returned.\n\n Returns:\n list: of :class:`.MARCSubrecord`.\n\n Raises:\n KeyError: If the subfield or datafield couldn't be found.\n\n Note:\n MARCSubrecord is practically same thing as string, but has defined\n :meth:`.MARCSubrecord.i1` and :attr:`.MARCSubrecord.i2`\n methods.\n\n You may need to be able to get this, because MARC XML depends on\n i/ind parameters from time to time (names of authors for example).\n\n \"\"\"\n if len(datafield) != 3:\n raise ValueError(\n \"`datafield` parameter have to be exactly 3 chars long!\"\n )\n if len(subfield) != 1:\n raise ValueError(\n \"Bad subfield specification - subfield have to be 1 char long!\"\n )\n\n # if datafield not found, return or raise exception\n if datafield not in self.datafields:\n if exception:\n raise KeyError(datafield + \" is not in datafields!\")\n\n return []\n\n # look for subfield defined by `subfield`, `i1` and `i2` parameters\n output = []\n for datafield in self.datafields[datafield]:\n if subfield not in datafield:\n continue\n\n # records are not returned just like plain string, but like\n # MARCSubrecord, because you will need ind1/ind2 values\n for sfield in datafield[subfield]:\n if i1 and sfield.i1 != i1:\n continue\n\n if i2 and sfield.i2 != i2:\n continue\n\n output.append(sfield)\n\n if not output and exception:\n raise KeyError(subfield + \" couldn't be found in subfields!\")\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef timeit(unit='s'):\n def wrapper(func):\n @wraps(func)\n def inner(*args, **kwargs):\n start = time.time()\n _result = func(*args, **kwargs)\n _format(unit, time.time() - start, func.__name__ + '()')\n return _result\n return inner\n return wrapper", "response": "\u6d4b\u8bd5\u51fd\u6570\u8017\u65f6\n\n :param unit: \u65f6\u95f4\u5355\u4f4d\uff0c\u6709 's','m','h' \u53ef\u9009\uff08seconds\uff0cminutes\uff0chours\uff09"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprint the stats to stdout.", "response": "def _print(stats, limit, label):\n \"\"\"\n \u63a7\u5236\u8f93\u51fa\u91cf\n \"\"\"\n print(\"TraceMalloc for {}\".format(label))\n for index, stat in enumerate(stats):\n if index < limit:\n print(stat)\n else:\n break"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef memoryit_block(group_by='lineno', limit=10, label='code block'):\n tracemalloc.start()\n _start = tracemalloc.take_snapshot()\n try:\n yield\n finally:\n _end = tracemalloc.take_snapshot()\n stats = _end.compare_to(_start, group_by)\n _print(stats, limit, label)", "response": "memoryit_block - \u5185\u5b58\u6d88\u8017\u60c5\u51b5 \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block \u53ef\u9009 memoryit_block"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_params(target, param, dof):\n '''Get the given param from each of the DOFs for a joint.'''\n return [target.getParam(getattr(ode, 'Param{}{}'.format(param, s)))\n for s in ['', '2', '3'][:dof]]", "response": "Get the given param from each of the DOFs for a joint."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _set_params(target, param, values, dof):\n '''Set the given param for each of the DOFs for a joint.'''\n if not isinstance(values, (list, tuple, np.ndarray)):\n values = [values] * dof\n assert dof == len(values)\n for s, value in zip(['', '2', '3'][:dof], values):\n target.setParam(getattr(ode, 'Param{}{}'.format(param, s)), value)", "response": "Set the given param for each of the DOFs for a joint."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives an angle and an axis create a quaternion.", "response": "def make_quaternion(theta, *axis):\n '''Given an angle and an axis, create a quaternion.'''\n x, y, z = axis\n r = np.sqrt(x * x + y * y + z * z)\n st = np.sin(theta / 2.)\n ct = np.cos(theta / 2.)\n return [x * st / r, y * st / r, z * st / r, ct]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a set of bodies compute their center of mass in world coordinates.", "response": "def center_of_mass(bodies):\n '''Given a set of bodies, compute their center of mass in world coordinates.\n '''\n x = np.zeros(3.)\n t = 0.\n for b in bodies:\n m = b.mass\n x += b.body_to_world(m.c) * m.mass\n t += m.mass\n return x / t"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef state(self):\n '''The state of this body includes:\n\n - name of the body (str)\n - position (3-tuple)\n - quaternion (4-tuple)\n - linear velocity (3-tuple)\n - angular velocity (3-tuple)\n '''\n return BodyState(self.name,\n tuple(self.position),\n tuple(self.quaternion),\n tuple(self.linear_velocity),\n tuple(self.angular_velocity))", "response": "Returns a BodyState object containing the current state of the object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the state of this body.", "response": "def state(self, state):\n '''Set the state of this body.\n\n Parameters\n ----------\n state : BodyState tuple\n The desired state of the body.\n '''\n assert self.name == state.name, \\\n 'state name \"{}\" != body name \"{}\"'.format(state.name, self.name)\n self.position = state.position\n self.quaternion = state.quaternion\n self.linear_velocity = state.linear_velocity\n self.angular_velocity = state.angular_velocity"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the rotation of this object using a rotation matrix.", "response": "def rotation(self, rotation):\n '''Set the rotation of this body using a rotation matrix.\n\n Parameters\n ----------\n rotation : sequence of 9 floats\n The desired rotation matrix for this body.\n '''\n if isinstance(rotation, np.ndarray):\n rotation = rotation.ravel()\n self.ode_body.setRotation(tuple(rotation))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the kinematic or dynamic attributes for this body.", "response": "def is_kinematic(self, is_kinematic):\n '''Set the kinematic/dynamic attribute for this body.\n\n In pagoda, kinematic bodies have infinite mass and do interact with\n other bodies via collisions.\n\n Parameters\n ----------\n is_kinematic : bool\n If True, this body will be set to kinematic. If False, it will be\n set to dynamic.\n '''\n if is_kinematic:\n self.ode_body.setKinematic()\n else:\n self.ode_body.setDynamic()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a body - relative offset to world coordinates.", "response": "def body_to_world(self, position):\n '''Convert a body-relative offset to world coordinates.\n\n Parameters\n ----------\n position : 3-tuple of float\n A tuple giving body-relative offsets.\n\n Returns\n -------\n position : 3-tuple of float\n A tuple giving the world coordinates of the given offset.\n '''\n return np.array(self.ode_body.getRelPointPos(tuple(position)))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef world_to_body(self, position):\n '''Convert a point in world coordinates to a body-relative offset.\n\n Parameters\n ----------\n position : 3-tuple of float\n A world coordinates position.\n\n Returns\n -------\n offset : 3-tuple of float\n A tuple giving the body-relative offset of the given position.\n '''\n return np.array(self.ode_body.getPosRelPoint(tuple(position)))", "response": "Convert a point in world coordinates to a body - relative offset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef relative_offset_to_world(self, offset):\n '''Convert a relative body offset to world coordinates.\n\n Parameters\n ----------\n offset : 3-tuple of float\n The offset of the desired point, given as a relative fraction of the\n size of this body. For example, offset (0, 0, 0) is the center of\n the body, while (0.5, -0.2, 0.1) describes a point halfway from the\n center towards the maximum x-extent of the body, 20% of the way from\n the center towards the minimum y-extent, and 10% of the way from the\n center towards the maximum z-extent.\n\n Returns\n -------\n position : 3-tuple of float\n A position in world coordinates of the given body offset.\n '''\n return np.array(self.body_to_world(offset * self.dimensions / 2))", "response": "Convert a relative body offset to world coordinates."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a force to the current object.", "response": "def add_force(self, force, relative=False, position=None, relative_position=None):\n '''Add a force to this body.\n\n Parameters\n ----------\n force : 3-tuple of float\n A vector giving the forces along each world or body coordinate axis.\n relative : bool, optional\n If False, the force values are assumed to be given in the world\n coordinate frame. If True, they are assumed to be given in the\n body-relative coordinate frame. Defaults to False.\n position : 3-tuple of float, optional\n If given, apply the force at this location in world coordinates.\n Defaults to the current position of the body.\n relative_position : 3-tuple of float, optional\n If given, apply the force at this relative location on the body. If\n given, this method ignores the ``position`` parameter.\n '''\n b = self.ode_body\n if relative_position is not None:\n op = b.addRelForceAtRelPos if relative else b.addForceAtRelPos\n op(force, relative_position)\n elif position is not None:\n op = b.addRelForceAtPos if relative else b.addForceAtPos\n op(force, position)\n else:\n op = b.addRelForce if relative else b.addForce\n op(force)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a torque to this body.", "response": "def add_torque(self, torque, relative=False):\n '''Add a torque to this body.\n\n Parameters\n ----------\n force : 3-tuple of float\n A vector giving the torque along each world or body coordinate axis.\n relative : bool, optional\n If False, the torque values are assumed to be given in the world\n coordinate frame. If True, they are assumed to be given in the\n body-relative coordinate frame. Defaults to False.\n '''\n op = self.ode_body.addRelTorque if relative else self.ode_body.addTorque\n op(torque)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconnecting this body to another one using a joint.", "response": "def join_to(self, joint, other_body=None, **kwargs):\n '''Connect this body to another one using a joint.\n\n This method creates a joint to fasten this body to the other one. See\n :func:`World.join`.\n\n Parameters\n ----------\n joint : str\n The type of joint to use when connecting these bodies.\n other_body : :class:`Body` or str, optional\n The other body to join with this one. If not given, connects this\n body to the world.\n '''\n self.world.join(joint, self, other_body, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconnecting two sets of bodies to this one and join them together.", "response": "def connect_to(self, joint, other_body, offset=(0, 0, 0), other_offset=(0, 0, 0),\n **kwargs):\n '''Move another body next to this one and join them together.\n\n This method will move the ``other_body`` so that the anchor points for\n the joint coincide. It then creates a joint to fasten the two bodies\n together. See :func:`World.move_next_to` and :func:`World.join`.\n\n Parameters\n ----------\n joint : str\n The type of joint to use when connecting these bodies.\n other_body : :class:`Body` or str\n The other body to join with this one.\n offset : 3-tuple of float, optional\n The body-relative offset where the anchor for the joint should be\n placed. Defaults to (0, 0, 0). See :func:`World.move_next_to` for a\n description of how offsets are specified.\n other_offset : 3-tuple of float, optional\n The offset on the second body where the joint anchor should be\n placed. Defaults to (0, 0, 0). Like ``offset``, this is given as an\n offset relative to the size and shape of ``other_body``.\n '''\n anchor = self.world.move_next_to(self, other_body, offset, other_offset)\n self.world.join(joint, self, other_body, anchor=anchor, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef positions(self):\n '''List of positions for linear degrees of freedom.'''\n return [self.ode_obj.getPosition(i) for i in range(self.LDOF)]", "response": "Returns a list of positions for linear degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlisting of position rates for linear degrees of freedom.", "response": "def position_rates(self):\n '''List of position rates for linear degrees of freedom.'''\n return [self.ode_obj.getPositionRate(i) for i in range(self.LDOF)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef angles(self):\n '''List of angles for rotational degrees of freedom.'''\n return [self.ode_obj.getAngle(i) for i in range(self.ADOF)]", "response": "List of angles for rotational degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef angle_rates(self):\n '''List of angle rates for rotational degrees of freedom.'''\n return [self.ode_obj.getAngleRate(i) for i in range(self.ADOF)]", "response": "List of angle rates for rotational degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nlist of axes for this object s degrees of freedom.", "response": "def axes(self):\n '''List of axes for this object's degrees of freedom.'''\n return [np.array(self.ode_obj.getAxis(i))\n for i in range(self.ADOF or self.LDOF)]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the axes for this object.", "response": "def axes(self, axes):\n '''Set the axes for this object's degrees of freedom.\n\n Parameters\n ----------\n axes : list of axes specifications\n A list of axis values to set. This list must have the same number of\n elements as the degrees of freedom of the underlying ODE object.\n Each element can be\n\n (a) None, which has no effect on the corresponding axis, or\n (b) three floats specifying the axis to set.\n '''\n assert self.ADOF == len(axes) or self.LDOF == len(axes)\n for i, axis in enumerate(axes):\n if axis is not None:\n self.ode_obj.setAxis(i, 0, axis)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lo_stops(self, lo_stops):\n '''Set the lo stop values for this object's degrees of freedom.\n\n Parameters\n ----------\n lo_stops : float or sequence of float\n A lo stop value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom. For rotational\n degrees of freedom, these values must be in radians.\n '''\n _set_params(self.ode_obj, 'LoStop', lo_stops, self.ADOF + self.LDOF)", "response": "Set the lo stop values for this object s degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef hi_stops(self, hi_stops):\n '''Set the hi stop values for this object's degrees of freedom.\n\n Parameters\n ----------\n hi_stops : float or sequence of float\n A hi stop value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom. For rotational\n degrees of freedom, these values must be in radians.\n '''\n _set_params(self.ode_obj, 'HiStop', hi_stops, self.ADOF + self.LDOF)", "response": "Set the hi stop values for this object s degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the target velocities for this object s degrees of freedom.", "response": "def velocities(self, velocities):\n '''Set the target velocities for this object's degrees of freedom.\n\n Parameters\n ----------\n velocities : float or sequence of float\n A target velocity value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom. For rotational\n degrees of freedom, these values must be in radians / second.\n '''\n _set_params(self.ode_obj, 'Vel', velocities, self.ADOF + self.LDOF)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef max_forces(self, max_forces):\n '''Set the maximum forces for this object's degrees of freedom.\n\n Parameters\n ----------\n max_forces : float or sequence of float\n A maximum force value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom.\n '''\n _set_params(self.ode_obj, 'FMax', max_forces, self.ADOF + self.LDOF)", "response": "Set the maximum forces for this object s degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting the ERP values for this object s degrees of freedom.", "response": "def erps(self, erps):\n '''Set the ERP values for this object's degrees of freedom.\n\n Parameters\n ----------\n erps : float or sequence of float\n An ERP value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom.\n '''\n _set_params(self.ode_obj, 'ERP', erps, self.ADOF + self.LDOF)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cfms(self, cfms):\n '''Set the CFM values for this object's degrees of freedom.\n\n Parameters\n ----------\n cfms : float or sequence of float\n A CFM value to set on all degrees of freedom, or a list\n containing one such value for each degree of freedom.\n '''\n _set_params(self.ode_obj, 'CFM', cfms, self.ADOF + self.LDOF)", "response": "Set the CFM values for this object s degrees of freedom."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stop_cfms(self, stop_cfms):\n '''Set the CFM values for this object's DOF limits.\n\n Parameters\n ----------\n stop_cfms : float or sequence of float\n A CFM value to set on all degrees of freedom limits, or a list\n containing one such value for each degree of freedom limit.\n '''\n _set_params(self.ode_obj, 'StopCFM', stop_cfms, self.ADOF + self.LDOF)", "response": "Set the CFM values for this object s DOF limits."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the ERP values for this object s DOF limits.", "response": "def stop_erps(self, stop_erps):\n '''Set the ERP values for this object's DOF limits.\n\n Parameters\n ----------\n stop_erps : float or sequence of float\n An ERP value to set on all degrees of freedom limits, or a list\n containing one such value for each degree of freedom limit.\n '''\n _set_params(self.ode_obj, 'StopERP', stop_erps, self.ADOF + self.LDOF)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the axes for this object.", "response": "def axes(self, axes):\n '''Set the axes for this object's degrees of freedom.\n\n Parameters\n ----------\n axes : list of axis parameters\n A list of axis values to set. This list must have the same number of\n elements as the degrees of freedom of the underlying ODE object.\n Each element can be\n\n (a) None, which has no effect on the corresponding axis, or\n (b) three floats specifying the axis to set, or\n (c) a dictionary with an \"axis\" key specifying the axis to set and\n an optional \"rel\" key (defaults to 0) specifying the relative\n body to set the axis on.\n '''\n assert len(axes) == self.ADOF\n for i, ax in enumerate(axes):\n if ax is None:\n continue\n if not isinstance(ax, dict):\n ax = dict(axis=ax)\n self.ode_obj.setAxis(i, ax.get('rel', 0), ax['axis'])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef axes(self, axes):\n '''Set the linear axis of displacement for this joint.\n\n Parameters\n ----------\n axes : list containing one 3-tuple of floats\n A list of the axes for this joint. For a slider joint, which has one\n degree of freedom, this must contain one 3-tuple specifying the X,\n Y, and Z axis for the joint.\n '''\n self.lmotor.axes = [axes[0]]\n self.ode_obj.setAxis(tuple(axes[0]))", "response": "Set the linear axis of displacement for this joint."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef axes(self, axes):\n '''Set the angular axis of rotation for this joint.\n\n Parameters\n ----------\n axes : list containing one 3-tuple of floats\n A list of the axes for this joint. For a hinge joint, which has one\n degree of freedom, this must contain one 3-tuple specifying the X,\n Y, and Z axis for the joint.\n '''\n self.amotor.axes = [axes[0]]\n self.ode_obj.setAxis(tuple(axes[0]))", "response": "Set the angular axis of rotation for this joint."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_body(self, shape, name=None, **kwargs):\n '''Create a new body.\n\n Parameters\n ----------\n shape : str\n The \"shape\" of the body to be created. This should name a type of\n body object, e.g., \"box\" or \"cap\".\n name : str, optional\n The name to use for this body. If not given, a default name will be\n constructed of the form \"{shape}{# of objects in the world}\".\n\n Returns\n -------\n body : :class:`Body`\n The created body object.\n '''\n shape = shape.lower()\n if name is None:\n for i in range(1 + len(self._bodies)):\n name = '{}{}'.format(shape, i)\n if name not in self._bodies:\n break\n self._bodies[name] = Body.build(shape, name, self, **kwargs)\n return self._bodies[name]", "response": "Create a new body."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a new joint that connects two bodies together.", "response": "def join(self, shape, body_a, body_b=None, name=None, **kwargs):\n '''Create a new joint that connects two bodies together.\n\n Parameters\n ----------\n shape : str\n The \"shape\" of the joint to use for joining together two bodies.\n This should name a type of joint, such as \"ball\" or \"piston\".\n body_a : str or :class:`Body`\n The first body to join together with this joint. If a string is\n given, it will be used as the name of a body to look up in the\n world.\n body_b : str or :class:`Body`, optional\n If given, identifies the second body to join together with\n ``body_a``. If not given, ``body_a`` is joined to the world.\n name : str, optional\n If given, use this name for the created joint. If not given, a name\n will be constructed of the form\n \"{body_a.name}^{shape}^{body_b.name}\".\n\n Returns\n -------\n joint : :class:`Joint`\n The joint object that was created.\n '''\n ba = self.get_body(body_a)\n bb = self.get_body(body_b)\n shape = shape.lower()\n if name is None:\n name = '{}^{}^{}'.format(ba.name, shape, bb.name if bb else '')\n self._joints[name] = Joint.build(\n shape, name, self, body_a=ba, body_b=bb, **kwargs)\n return self._joints[name]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmove one body to near another body.", "response": "def move_next_to(self, body_a, body_b, offset_a, offset_b):\n '''Move one body to be near another one.\n\n After moving, the location described by ``offset_a`` on ``body_a`` will\n be coincident with the location described by ``offset_b`` on ``body_b``.\n\n Parameters\n ----------\n body_a : str or :class:`Body`\n The body to use as a reference for moving the other body. If this is\n a string, it is treated as the name of a body to look up in the\n world.\n body_b : str or :class:`Body`\n The body to move next to ``body_a``. If this is a string, it is\n treated as the name of a body to look up in the world.\n offset_a : 3-tuple of float\n The offset of the anchor point, given as a relative fraction of the\n size of ``body_a``. See :func:`Body.relative_offset_to_world`.\n offset_b : 3-tuple of float\n The offset of the anchor point, given as a relative fraction of the\n size of ``body_b``.\n\n Returns\n -------\n anchor : 3-tuple of float\n The location of the shared point, which is often useful to use as a\n joint anchor.\n '''\n ba = self.get_body(body_a)\n bb = self.get_body(body_b)\n if ba is None:\n return bb.relative_offset_to_world(offset_b)\n if bb is None:\n return ba.relative_offset_to_world(offset_a)\n anchor = ba.relative_offset_to_world(offset_a)\n offset = bb.relative_offset_to_world(offset_b)\n bb.position = bb.position + anchor - offset\n return anchor"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the states of some bodies in the world.", "response": "def set_body_states(self, states):\n '''Set the states of some bodies in the world.\n\n Parameters\n ----------\n states : sequence of states\n A complete state tuple for one or more bodies in the world. See\n :func:`get_body_states`.\n '''\n for state in states:\n self.get_body(state.name).state = state"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef step(self, substeps=2):\n '''Step the world forward by one frame.\n\n Parameters\n ----------\n substeps : int, optional\n Split the step into this many sub-steps. This helps to prevent the\n time delta for an update from being too large.\n '''\n self.frame_no += 1\n dt = self.dt / substeps\n for _ in range(substeps):\n self.ode_contactgroup.empty()\n self.ode_space.collide(None, self.on_collision)\n self.ode_world.step(dt)", "response": "Step the world forward by one frame."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nhandles an otherwise unhandled keypress event.", "response": "def on_key_press(self, key, modifiers, keymap):\n '''Handle an otherwise unhandled keypress event (from a GUI).'''\n if key == keymap.ENTER:\n self.reset()\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef are_connected(self, body_a, body_b):\n '''Determine whether the given bodies are currently connected.\n\n Parameters\n ----------\n body_a : str or :class:`Body`\n One body to test for connectedness. If this is a string, it is\n treated as the name of a body to look up.\n body_b : str or :class:`Body`\n One body to test for connectedness. If this is a string, it is\n treated as the name of a body to look up.\n\n Returns\n -------\n connected : bool\n Return True iff the two bodies are connected.\n '''\n return bool(ode.areConnected(\n self.get_body(body_a).ode_body,\n self.get_body(body_b).ode_body))", "response": "Determine whether the given bodies are currently connected."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef on_collision(self, args, geom_a, geom_b):\n '''Callback function for the collide() method.\n\n Parameters\n ----------\n args : None\n Arguments passed when the callback was registered. Not used.\n geom_a : ODE geometry\n The geometry object of one of the bodies that has collided.\n geom_b : ODE geometry\n The geometry object of one of the bodies that has collided.\n '''\n body_a = geom_a.getBody()\n body_b = geom_b.getBody()\n if ode.areConnected(body_a, body_b) or \\\n (body_a and body_a.isKinematic()) or \\\n (body_b and body_b.isKinematic()):\n return\n for c in ode.collide(geom_a, geom_b):\n c.setBounce(self.elasticity)\n c.setMu(self.friction)\n ode.ContactJoint(self.ode_world, self.ode_contactgroup, c).attach(\n geom_a.getBody(), geom_b.getBody())", "response": "Callback function for the collide() method."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef record_iterator(xml):\n # handle file-like objects\n if hasattr(xml, \"read\"):\n xml = xml.read()\n\n dom = None\n try:\n dom = dhtmlparser.parseString(xml)\n except UnicodeError:\n dom = dhtmlparser.parseString(xml.encode(\"utf-8\"))\n\n for record_xml in dom.findB(\"record\"):\n yield MARCXMLRecord(record_xml)", "response": "Iterate over all record tags in xml."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef profileit(field='cumulative'):\n def wrapper(func):\n @wraps(func)\n def inner(*args, **kwargs):\n pro = Profile()\n pro.runcall(func, *args, **kwargs)\n stats = Stats(pro)\n stats.strip_dirs()\n stats.sort_stats(field)\n print(\"Profile for {}()\".format(func.__name__))\n stats.print_stats()\n stats.print_callers()\n return inner\n return wrapper", "response": "\u6d4b\u8bd5\u51fd\u6570\u8fd0\u884c\u6d88\u8017\u60c5\u51b5\n\n :param field: \u8f93\u51fa\u5185\u5bb9\u6392\u5e8f\u65b9\u5f0f\u3002\n \u53ef\u9009\u53c2\u6570\u4e3a \"stdname\", \"calls\", \"time\", \"cumulative\""} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading and parse a source file.", "response": "def parse(source, world, jointgroup=None, density=1000, color=None):\n '''Load and parse a source file.\n\n Parameters\n ----------\n source : file\n A file-like object that contains text information describing bodies and\n joints to add to the world.\n world : :class:`pagoda.physics.World`\n The world to add objects and joints to.\n jointgroup : ode.JointGroup, optional\n If provided, add all joints from this parse to the given group. The\n default behavior adds joints to the world without an explicit group.\n density : float, optional\n Default density for bodies. This is overridden if the source provides a\n density or mass value for a given body.\n color : tuple of floats, optional\n Default color for bodies from this source. Defaults to None, which does\n not assign a color to parsed bodies.\n '''\n visitor = Visitor(world, jointgroup, density, color)\n visitor.parse(re.sub(r'#.*', ' ', source.read()))\n return visitor"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload and parse an Asf file into a new object.", "response": "def parse_asf(source, world, jointgroup=None, density=1000, color=None):\n '''Load and parse a source file.\n\n Parameters\n ----------\n source : file\n A file-like object that contains text information describing bodies and\n joints to add to the world.\n world : :class:`pagoda.physics.World`\n The world to add objects and joints to.\n jointgroup : ode.JointGroup, optional\n If provided, add all joints from this parse to the given group. The\n default behavior adds joints to the world without an explicit group.\n density : float, optional\n Default density for bodies. This is overridden if the source provides a\n density or mass value for a given body.\n color : tuple of floats, optional\n Default color for bodies from this source. Defaults to None, which does\n not assign a color to parsed bodies.\n '''\n visitor = AsfVisitor(world, jointgroup, density, color)\n visitor.parse(re.sub(r'#.*', ' ', source.read()))\n return visitor"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse an AMC motion capture data file into a series of motion capture frames.", "response": "def parse_amc(source):\n '''Parse an AMC motion capture data file.\n\n Parameters\n ----------\n source : file\n A file-like object that contains AMC motion capture text.\n\n Yields\n ------\n frame : dict\n Yields a series of motion capture frames. Each frame is a dictionary\n that maps a bone name to a list of the DOF configurations for that bone.\n '''\n lines = 0\n frames = 1\n frame = {}\n degrees = False\n for line in source:\n lines += 1\n line = line.split('#')[0].strip()\n if not line:\n continue\n if line.startswith(':'):\n if line.lower().startswith(':deg'):\n degrees = True\n continue\n if line.isdigit():\n if int(line) != frames:\n raise RuntimeError(\n 'frame mismatch on line {}: '\n 'produced {} but file claims {}'.format(lines, frames, line))\n yield frame\n frames += 1\n frame = {}\n continue\n fields = line.split()\n frame[fields[0]] = list(map(float, fields[1:]))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_bodies(self, translate=(0, 1, 0), size=0.1):\n '''Traverse the bone hierarchy and create physics bodies.'''\n stack = [('root', 0, self.root['position'] + translate)]\n while stack:\n name, depth, end = stack.pop()\n\n for child in self.hierarchy.get(name, ()):\n stack.append((child, depth + 1, end + self.bones[child].end))\n\n if name not in self.bones:\n continue\n\n bone = self.bones[name]\n body = self.world.create_body(\n 'box', name=bone.name, density=self.density,\n lengths=(size, size, bone.length))\n body.color = self.color\n\n # move the center of the body to the halfway point between\n # the parent (joint) and child (joint).\n x, y, z = end - bone.direction * bone.length / 2\n\n # swizzle y and z -- asf uses y as up, but we use z as up.\n body.position = x, z, y\n\n # compute an orthonormal (rotation) matrix using the ground and\n # the body. this is mind-bending but seems to work.\n u = bone.direction\n v = np.cross(u, [0, 1, 0])\n l = np.linalg.norm(v)\n if l > 0:\n v /= l\n rot = np.vstack([np.cross(u, v), v, u]).T\n swizzle = [[1, 0, 0], [0, 0, 1], [0, -1, 0]]\n body.rotation = np.dot(swizzle, rot)\n\n self.bodies.append(body)", "response": "Traverse the bone hierarchy and create physics bodies."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_joints(self):\n '''Traverse the bone hierarchy and create physics joints.'''\n stack = ['root']\n while stack:\n parent = stack.pop()\n for child in self.hierarchy.get(parent, ()):\n stack.append(child)\n if parent not in self.bones:\n continue\n bone = self.bones[parent]\n body = [b for b in self.bodies if b.name == parent][0]\n for child in self.hierarchy.get(parent, ()):\n child_bone = self.bones[child]\n child_body = [b for b in self.bodies if b.name == child][0]\n shape = ('', 'hinge', 'universal', 'ball')[len(child_bone.dof)]\n self.joints.append(self.world.join(shape, body, child_body))", "response": "Traverse the bone hierarchy and create physics joints."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndelete the empty directory", "response": "def delete_empty_dir(directory):\n \"\"\"\n \u5220\u9664\u7a7a\u76ee\u5f55\n\n :param directory: \u76ee\u5f55\u8def\u5f84\n \"\"\"\n if os.path.exists(directory):\n if os.path.isdir(directory):\n for d in os.listdir(directory):\n path = os.path.join(directory, d)\n if os.path.isdir(path):\n delete_empty_dir(path)\n\n if not os.listdir(directory):\n os.rmdir(directory)\n print(\"Remove the empty directory: \" + directory)\n else:\n print(\"The directory is not exist!\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse informations about corporations from given MARC field ID subfield and roles.", "response": "def _parse_corporations(self, datafield, subfield, roles=[\"any\"]):\n \"\"\"\n Parse informations about corporations from given field identified\n by `datafield` parameter.\n\n Args:\n datafield (str): MARC field ID (\"``110``\", \"``610``\", etc..)\n subfield (str): MARC subfield ID with name, which is typically\n stored in \"``a``\" subfield.\n roles (str): specify which roles you need. Set to ``[\"any\"]`` for\n any role, ``[\"dst\"]`` for distributors, etc.. For\n details, see\n http://www.loc.gov/marc/relators/relaterm.html\n\n Returns:\n list: :class:`Corporation` objects.\n \"\"\"\n if len(datafield) != 3:\n raise ValueError(\n \"datafield parameter have to be exactly 3 chars long!\"\n )\n if len(subfield) != 1:\n raise ValueError(\n \"Bad subfield specification - subield have to be 3 chars long!\"\n )\n parsed_corporations = []\n for corporation in self.get_subfields(datafield, subfield):\n other_subfields = corporation.other_subfields\n\n # check if corporation have at least one of the roles specified in\n # 'roles' parameter of function\n if \"4\" in other_subfields and roles != [\"any\"]:\n corp_roles = other_subfields[\"4\"] # list of role parameters\n\n relevant = any(map(lambda role: role in roles, corp_roles))\n\n # skip non-relevant corporations\n if not relevant:\n continue\n\n name = \"\"\n place = \"\"\n date = \"\"\n\n name = corporation\n\n if \"c\" in other_subfields:\n place = \",\".join(other_subfields[\"c\"])\n if \"d\" in other_subfields:\n date = \",\".join(other_subfields[\"d\"])\n\n parsed_corporations.append(Corporation(name, place, date))\n\n return parsed_corporations"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses persons from given datafield subfield and roles.", "response": "def _parse_persons(self, datafield, subfield, roles=[\"aut\"]):\n \"\"\"\n Parse persons from given datafield.\n\n Args:\n datafield (str): code of datafield (\"010\", \"730\", etc..)\n subfield (char): code of subfield (\"a\", \"z\", \"4\", etc..)\n role (list of str): set to [\"any\"] for any role, [\"aut\"] for\n authors, etc.. For details see\n http://www.loc.gov/marc/relators/relaterm.html\n\n Main records for persons are: \"100\", \"600\" and \"700\", subrecords \"c\".\n\n Returns:\n list: Person objects.\n \"\"\"\n # parse authors\n parsed_persons = []\n raw_persons = self.get_subfields(datafield, subfield)\n for person in raw_persons:\n # check if person have at least one of the roles specified in\n # 'roles' parameter of function\n other_subfields = person.other_subfields\n if \"4\" in other_subfields and roles != [\"any\"]:\n person_roles = other_subfields[\"4\"] # list of role parameters\n\n relevant = any(map(lambda role: role in roles, person_roles))\n\n # skip non-relevant persons\n if not relevant:\n continue\n\n # result of .strip() is string, so ind1/2 in MARCSubrecord are lost\n ind1 = person.i1\n ind2 = person.i2\n person = person.strip()\n\n name = \"\"\n second_name = \"\"\n surname = \"\"\n title = \"\"\n\n # here it gets nasty - there is lot of options in ind1/ind2\n # parameters\n if ind1 == \"1\" and ind2 == \" \":\n if \",\" in person:\n surname, name = person.split(\",\", 1)\n elif \" \" in person:\n surname, name = person.split(\" \", 1)\n else:\n surname = person\n\n if \"c\" in other_subfields:\n title = \",\".join(other_subfields[\"c\"])\n elif ind1 == \"0\" and ind2 == \" \":\n name = person.strip()\n\n if \"b\" in other_subfields:\n second_name = \",\".join(other_subfields[\"b\"])\n\n if \"c\" in other_subfields:\n surname = \",\".join(other_subfields[\"c\"])\n elif ind1 == \"1\" and ind2 == \"0\" or ind1 == \"0\" and ind2 == \"0\":\n name = person.strip()\n if \"c\" in other_subfields:\n title = \",\".join(other_subfields[\"c\"])\n\n parsed_persons.append(\n Person(\n name.strip(),\n second_name.strip(),\n surname.strip(),\n title.strip()\n )\n )\n\n return parsed_persons"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_subname(self, undefined=\"\"):\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"245\", \"b\")),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns the name of the the\n probe s company."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_price(self, undefined=\"\"):\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"020\", \"c\")),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns the price of the book with currency."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_part(self, undefined=\"\"):\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"245\", \"p\")),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns the name of the part of the book series."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_part_name(self, undefined=\"\"):\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"245\", \"n\")),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns the name of the part of the series."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_publisher(self, undefined=\"\"):\n publishers = set([\n remove_hairs_fn(publisher)\n for publisher in self[\"260b \"] + self[\"264b\"]\n ])\n\n return _undefined_pattern(\n \", \".join(publishers),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns the name of the publisher record."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the date of the publication.", "response": "def get_pub_date(self, undefined=\"\"):\n \"\"\"\n Args:\n undefined (optional): Argument, which will be returned if the\n `pub_date` record is not found.\n\n Returns:\n str: Date of publication (month and year usually) or `undefined` \\\n if `pub_date` is not found.\n \"\"\"\n dates = self[\"260c \"] + self[\"264c\"]\n\n def clean_date(date):\n \"\"\"\n Clean the `date` strings from special characters, but leave\n sequences of numbers followed by -.\n\n So:\n [2015]- -> 2015\n 2015- -> 2015-\n \"\"\"\n out = \"\"\n was_digit = False\n for c in date:\n if c.isdigit() or (c == \"-\" and was_digit) or c == \" \":\n out += c\n\n was_digit = c.isdigit()\n\n return out\n\n # clean all the date strings\n dates = set([\n clean_date(date)\n for date in self[\"260c \"] + self[\"264c\"]\n ])\n\n return _undefined_pattern(\n \", \".join(dates),\n lambda x: x.strip() == \"\",\n undefined\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the public order of the o is the", "response": "def get_pub_order(self, undefined=\"\"):\n \"\"\"\n Args:\n undefined (optional): Argument, which will be returned if the\n `pub_order` record is not found.\n\n Returns:\n str: Information about order in which was the book published or \\\n `undefined` if `pub_order` is not found.\n \"\"\"\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"901\", \"f\")),\n lambda x: x.strip() == \"\",\n undefined\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the name of the pub_place record.", "response": "def get_pub_place(self, undefined=\"\"):\n \"\"\"\n Args:\n undefined (optional): Argument, which will be returned if the\n `pub_place` record is not found.\n\n Returns:\n str: Name of city/country where the book was published or \\\n `undefined` if `pub_place` is not found.\n \"\"\"\n places = set([\n remove_hairs_fn(place)\n for place in self[\"260a \"] + self[\"264a\"]\n ])\n\n return _undefined_pattern(\n \", \".join(places),\n lambda x: x.strip() == \"\",\n undefined\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_format(self, undefined=\"\"):\n return _undefined_pattern(\n \"\".join(self.get_subfields(\"300\", \"c\")),\n lambda x: x.strip() == \"\",\n undefined\n )", "response": "Returns a string that can be used to format the o is not found in the o o"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of Authors represented as : class :. Person objects.", "response": "def get_authors(self):\n \"\"\"\n Returns:\n list: Authors represented as :class:`.Person` objects.\n \"\"\"\n authors = self._parse_persons(\"100\", \"a\")\n authors += self._parse_persons(\"600\", \"a\")\n authors += self._parse_persons(\"700\", \"a\")\n authors += self._parse_persons(\"800\", \"a\")\n\n return authors"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_corporations(self, roles=[\"dst\"]):\n corporations = self._parse_corporations(\"110\", \"a\", roles)\n corporations += self._parse_corporations(\"610\", \"a\", roles)\n corporations += self._parse_corporations(\"710\", \"a\", roles)\n corporations += self._parse_corporations(\"810\", \"a\", roles)\n\n return corporations", "response": "Returns a list of all corporations that you have in the given list of roles."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_ISBNs(self):\n invalid_isbns = set(self.get_invalid_ISBNs())\n\n valid_isbns = [\n self._clean_isbn(isbn)\n for isbn in self[\"020a\"]\n if self._clean_isbn(isbn) not in invalid_isbns\n ]\n\n if valid_isbns:\n return valid_isbns\n\n # this is used sometimes in czech national library\n return [\n self._clean_isbn(isbn)\n for isbn in self[\"901i\"]\n ]", "response": "Get list of VALID ISBN."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_ISSNs(self):\n invalid_issns = set(self.get_invalid_ISSNs())\n\n return [\n self._clean_isbn(issn)\n for issn in self[\"022a\"]\n if self._clean_isbn(issn) not in invalid_issns\n ]", "response": "Get list of VALID ISSNs."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfilters ISBN record binding from ISBN record.", "response": "def _filter_binding(self, binding):\n \"\"\"\n Filter binding from ISBN record. In MARC XML / OAI, the binding\n information is stored in same subrecord as ISBN.\n\n Example:\n ``80-251-0225-4 (bro\u017e.) :`` ->\n ``bro\u017e.``.\n \"\"\"\n binding = binding.strip().split(\" \", 1)[-1] # isolate bind. from ISBN\n binding = remove_hairs_fn(binding) # remove special chars from binding\n\n return binding.split(\":\")[-1].strip()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of URLs defined by the current locale.", "response": "def get_urls(self):\n \"\"\"\n Content of field ``856u42``. Typically URL pointing to producers\n homepage.\n\n Returns:\n list: List of URLs defined by producer.\n \"\"\"\n urls = self.get_subfields(\"856\", \"u\", i1=\"4\", i2=\"2\")\n\n return map(lambda x: x.replace(\"&\", \"&\"), urls)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of internal URLs.", "response": "def get_internal_urls(self):\n \"\"\"\n URL's, which may point to edeposit, aleph, kramerius and so on.\n\n Fields ``856u40``, ``998a`` and ``URLu``.\n\n Returns:\n list: List of internal URLs. \n \"\"\"\n internal_urls = self.get_subfields(\"856\", \"u\", i1=\"4\", i2=\"0\")\n internal_urls.extend(self.get_subfields(\"998\", \"a\"))\n internal_urls.extend(self.get_subfields(\"URL\", \"u\"))\n\n return map(lambda x: x.replace(\"&\", \"&\"), internal_urls)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the PublicationType of the current entry.", "response": "def get_pub_type(self):\n \"\"\"\n Returns:\n PublicationType: :class:`.PublicationType` enum **value**.\n \"\"\"\n INFO_CHAR_INDEX = 6\n SECOND_INFO_CHAR_I = 18\n\n if not len(self.leader) >= INFO_CHAR_INDEX + 1:\n return PublicationType.monographic\n\n if self.controlfields.get(\"FMT\") == \"SE\":\n return PublicationType.continuing\n\n info_char = self.leader[INFO_CHAR_INDEX]\n multipart_n = self.get_subfields(\"245\", \"n\", exception=False)\n multipart_p = self.get_subfields(\"245\", \"p\", exception=False)\n\n if info_char in \"acd\":\n return PublicationType.monographic\n elif info_char in \"bis\":\n return PublicationType.continuing\n elif info_char == \"m\" and (multipart_n or multipart_p):\n return PublicationType.multipart_monograph\n elif info_char == \"m\" and len(self.leader) >= SECOND_INFO_CHAR_I + 1:\n if self.leader[SECOND_INFO_CHAR_I] == \"a\":\n return PublicationType.multipart_monograph\n elif self.leader[SECOND_INFO_CHAR_I] == \" \":\n return PublicationType.single_unit\n\n return PublicationType.monographic"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get(self, item, alt=None):\n try:\n val = self[item]\n except ValueError:\n return alt\n\n return val if val is not None else alt", "response": "Standard dict - like get method."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef as_flat_array(iterables):\n '''Given a sequence of sequences, return a flat numpy array.\n\n Parameters\n ----------\n iterables : sequence of sequence of number\n A sequence of tuples or lists containing numbers. Typically these come\n from something that represents each joint in a skeleton, like angle.\n\n Returns\n -------\n ndarray :\n An array of flattened data from each of the source iterables.\n '''\n arr = []\n for x in iterables:\n arr.extend(x)\n return np.array(arr)", "response": "Given a sequence of iterables return a flat numpy array."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading a skeleton definition from a file.", "response": "def load(self, source, **kwargs):\n '''Load a skeleton definition from a file.\n\n Parameters\n ----------\n source : str or file\n A filename or file-like object that contains text information\n describing a skeleton. See :class:`pagoda.parser.Parser` for more\n information about the format of the text file.\n '''\n if hasattr(source, 'endswith') and source.lower().endswith('.asf'):\n self.load_asf(source, **kwargs)\n else:\n self.load_skel(source, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_skel(self, source, **kwargs):\n '''Load a skeleton definition from a text file.\n\n Parameters\n ----------\n source : str or file\n A filename or file-like object that contains text information\n describing a skeleton. See :class:`pagoda.parser.BodyParser` for\n more information about the format of the text file.\n '''\n logging.info('%s: parsing skeleton configuration', source)\n if hasattr(source, 'read'):\n p = parser.parse(source, self.world, self.jointgroup, **kwargs)\n else:\n with open(source) as handle:\n p = parser.parse(handle, self.world, self.jointgroup, **kwargs)\n self.bodies = p.bodies\n self.joints = p.joints\n self.set_pid_params(kp=0.999 / self.world.dt)", "response": "Load a skeleton definition from a text file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading a skeleton definition from an ASF text file.", "response": "def load_asf(self, source, **kwargs):\n '''Load a skeleton definition from an ASF text file.\n\n Parameters\n ----------\n source : str or file\n A filename or file-like object that contains text information\n describing a skeleton, in ASF format.\n '''\n if hasattr(source, 'read'):\n p = parser.parse_asf(source, self.world, self.jointgroup, **kwargs)\n else:\n with open(source) as handle:\n p = parser.parse_asf(handle, self.world, self.jointgroup, **kwargs)\n self.bodies = p.bodies\n self.joints = p.joints\n self.set_pid_params(kp=0.999 / self.world.dt)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the PID parameters for all joints in the skeleton.", "response": "def set_pid_params(self, *args, **kwargs):\n '''Set PID parameters for all joints in the skeleton.\n\n Parameters for this method are passed directly to the `pid` constructor.\n '''\n for joint in self.joints:\n joint.target_angles = [None] * joint.ADOF\n joint.controllers = [pid(*args, **kwargs) for i in range(joint.ADOF)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a list of all current joint torques in the skeleton.", "response": "def joint_torques(self):\n '''Get a list of all current joint torques in the skeleton.'''\n return as_flat_array(getattr(j, 'amotor', j).feedback[-1][:j.ADOF]\n for j in self.joints)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a list of the indices for a specific joint.", "response": "def indices_for_joint(self, name):\n '''Get a list of the indices for a specific joint.\n\n Parameters\n ----------\n name : str\n The name of the joint to look up.\n\n Returns\n -------\n list of int :\n A list of the index values for quantities related to the named\n joint. Often useful for getting, say, the angles for a specific\n joint in the skeleton.\n '''\n j = 0\n for joint in self.joints:\n if joint.name == name:\n return list(range(j, j + joint.ADOF))\n j += joint.ADOF\n return []"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a list of the indices for a specific body.", "response": "def indices_for_body(self, name, step=3):\n '''Get a list of the indices for a specific body.\n\n Parameters\n ----------\n name : str\n The name of the body to look up.\n step : int, optional\n The number of numbers for each body. Defaults to 3, should be set\n to 4 for body rotation (since quaternions have 4 values).\n\n Returns\n -------\n list of int :\n A list of the index values for quantities related to the named body.\n '''\n for j, body in enumerate(self.bodies):\n if body.name == name:\n return list(range(j * step, (j + 1) * step))\n return []"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the current joint separations for the skeleton.", "response": "def joint_distances(self):\n '''Get the current joint separations for the skeleton.\n\n Returns\n -------\n distances : list of float\n A list expressing the distance between the two joint anchor points,\n for each joint in the skeleton. These quantities describe how\n \"exploded\" the bodies in the skeleton are; a value of 0 indicates\n that the constraints are perfectly satisfied for that joint.\n '''\n return [((np.array(j.anchor) - j.anchor2) ** 2).sum() for j in self.joints]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nenabling the joint motors in this skeleton.", "response": "def enable_motors(self, max_force):\n '''Enable the joint motors in this skeleton.\n\n This method sets the maximum force that can be applied by each joint to\n attain the desired target velocities. It also enables torque feedback\n for all joint motors.\n\n Parameters\n ----------\n max_force : float\n The maximum force that each joint is allowed to apply to attain its\n target velocity.\n '''\n for joint in self.joints:\n amotor = getattr(joint, 'amotor', joint)\n amotor.max_forces = max_force\n if max_force > 0:\n amotor.enable_feedback()\n else:\n amotor.disable_feedback()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmove each joint toward a target angle.", "response": "def set_target_angles(self, angles):\n '''Move each joint toward a target angle.\n\n This method uses a PID controller to set a target angular velocity for\n each degree of freedom in the skeleton, based on the difference between\n the current and the target angle for the respective DOF.\n\n PID parameters are by default set to achieve a tiny bit less than\n complete convergence in one time step, using only the P term (i.e., the\n P coefficient is set to 1 - \\delta, while I and D coefficients are set\n to 0). PID parameters can be updated by calling the `set_pid_params`\n method.\n\n Parameters\n ----------\n angles : list of float\n A list of the target angles for every joint in the skeleton.\n '''\n j = 0\n for joint in self.joints:\n velocities = [\n ctrl(tgt - cur, self.world.dt) for cur, tgt, ctrl in\n zip(joint.angles, angles[j:j+joint.ADOF], joint.controllers)]\n joint.velocities = velocities\n j += joint.ADOF"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_torques(self, torques):\n '''Add torques for each degree of freedom in the skeleton.\n\n Parameters\n ----------\n torques : list of float\n A list of the torques to add to each degree of freedom in the\n skeleton.\n '''\n j = 0\n for joint in self.joints:\n joint.add_torques(\n list(torques[j:j+joint.ADOF]) + [0] * (3 - joint.ADOF))\n j += joint.ADOF", "response": "Add the torques to each degree of freedom in the skeleton."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef labels(self):\n '''Return the names of our marker labels in canonical order.'''\n return sorted(self.channels, key=lambda c: self.channels[c])", "response": "Return the names of our marker labels in canonical order."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load_csv(self, filename, start_frame=10, max_frames=int(1e300)):\n '''Load marker data from a CSV file.\n\n The file will be imported using Pandas, which must be installed to use\n this method. (``pip install pandas``)\n\n The first line of the CSV file will be used for header information. The\n \"time\" column will be used as the index for the data frame. There must\n be columns named 'markerAB-foo-x','markerAB-foo-y','markerAB-foo-z', and\n 'markerAB-foo-c' for marker 'foo' to be included in the model.\n\n Parameters\n ----------\n filename : str\n Name of the CSV file to load.\n '''\n import pandas as pd\n\n compression = None\n if filename.endswith('.gz'):\n compression = 'gzip'\n df = pd.read_csv(filename, compression=compression).set_index('time').fillna(-1)\n\n # make sure the data frame's time index matches our world.\n assert self.world.dt == pd.Series(df.index).diff().mean()\n\n markers = []\n for c in df.columns:\n m = re.match(r'^marker\\d\\d-(.*)-c$', c)\n if m:\n markers.append(m.group(1))\n self.channels = self._map_labels_to_channels(markers)\n\n cols = [c for c in df.columns if re.match(r'^marker\\d\\d-.*-[xyzc]$', c)]\n self.data = df[cols].values.reshape((len(df), len(markers), 4))[start_frame:]\n self.data[:, :, [1, 2]] = self.data[:, :, [2, 1]]\n\n logging.info('%s: loaded marker data %s', filename, self.data.shape)\n self.process_data()\n self.create_bodies()", "response": "Load marker data from a CSV file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_c3d(self, filename, start_frame=0, max_frames=int(1e300)):\n '''Load marker data from a C3D file.\n\n The file will be imported using the c3d module, which must be installed\n to use this method. (``pip install c3d``)\n\n Parameters\n ----------\n filename : str\n Name of the C3D file to load.\n start_frame : int, optional\n Discard the first N frames. Defaults to 0.\n max_frames : int, optional\n Maximum number of frames to load. Defaults to loading all frames.\n '''\n import c3d\n\n with open(filename, 'rb') as handle:\n reader = c3d.Reader(handle)\n\n logging.info('world frame rate %s, marker frame rate %s',\n 1 / self.world.dt, reader.point_rate)\n\n # set up a map from marker label to index in the data stream.\n self.channels = self._map_labels_to_channels([\n s.strip() for s in reader.point_labels])\n\n # read the actual c3d data into a numpy array.\n data = []\n for i, (_, frame, _) in enumerate(reader.read_frames()):\n if i >= start_frame:\n data.append(frame[:, [0, 1, 2, 4]])\n if len(data) > max_frames:\n break\n self.data = np.array(data)\n\n # scale the data to meters -- mm is a very common C3D unit.\n if reader.get('POINT:UNITS').string_value.strip().lower() == 'mm':\n logging.info('scaling point data from mm to m')\n self.data[:, :, :3] /= 1000.\n\n logging.info('%s: loaded marker data %s', filename, self.data.shape)\n self.process_data()\n self.create_bodies()", "response": "Load marker data from a C3D file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_data(self):\n '''Process data to produce velocity and dropout information.'''\n self.visibility = self.data[:, :, 3]\n self.positions = self.data[:, :, :3]\n self.velocities = np.zeros_like(self.positions) + 1000\n for frame_no in range(1, len(self.data) - 1):\n prev = self.data[frame_no - 1]\n next = self.data[frame_no + 1]\n for c in range(self.num_markers):\n if -1 < prev[c, 3] < 100 and -1 < next[c, 3] < 100:\n self.velocities[frame_no, c] = (\n next[c, :3] - prev[c, :3]) / (2 * self.world.dt)\n self.cfms = np.zeros_like(self.visibility) + self.DEFAULT_CFM", "response": "Process data to produce velocity and dropout information."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates physics bodies corresponding to each marker in our data.", "response": "def create_bodies(self):\n '''Create physics bodies corresponding to each marker in our data.'''\n self.bodies = {}\n for label in self.channels:\n body = self.world.create_body(\n 'sphere', name='marker:{}'.format(label), radius=0.02)\n body.is_kinematic = True\n body.color = 0.9, 0.1, 0.1, 0.5\n self.bodies[label] = body"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_attachments(self, source, skeleton):\n '''Load attachment configuration from the given text source.\n\n The attachment configuration file has a simple format. After discarding\n Unix-style comments (any part of a line that starts with the pound (#)\n character), each line in the file is then expected to have the following\n format::\n\n marker-name body-name X Y Z\n\n The marker name must correspond to an existing \"channel\" in our marker\n data. The body name must correspond to a rigid body in the skeleton. The\n X, Y, and Z coordinates specify the body-relative offsets where the\n marker should be attached: 0 corresponds to the center of the body along\n the given axis, while -1 and 1 correspond to the minimal (maximal,\n respectively) extent of the body's bounding box along the corresponding\n dimension.\n\n Parameters\n ----------\n source : str or file-like\n A filename or file-like object that we can use to obtain text\n configuration that describes how markers are attached to skeleton\n bodies.\n\n skeleton : :class:`pagoda.skeleton.Skeleton`\n The skeleton to attach our marker data to.\n '''\n self.targets = {}\n self.offsets = {}\n\n filename = source\n if isinstance(source, str):\n source = open(source)\n else:\n filename = '(file-{})'.format(id(source))\n\n for i, line in enumerate(source):\n tokens = line.split('#')[0].strip().split()\n if not tokens:\n continue\n label = tokens.pop(0)\n if label not in self.channels:\n logging.info('%s:%d: unknown marker %s', filename, i, label)\n continue\n if not tokens:\n continue\n name = tokens.pop(0)\n bodies = [b for b in skeleton.bodies if b.name == name]\n if len(bodies) != 1:\n logging.info('%s:%d: %d skeleton bodies match %s',\n filename, i, len(bodies), name)\n continue\n b = self.targets[label] = bodies[0]\n o = self.offsets[label] = \\\n np.array(list(map(float, tokens))) * b.dimensions / 2\n logging.info('%s <--> %s, offset %s', label, b.name, o)", "response": "Load the attachment configuration from the given text source."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nattach marker bodies to the corresponding skeleton bodies.", "response": "def attach(self, frame_no):\n '''Attach marker bodies to the corresponding skeleton bodies.\n\n Attachments are only made for markers that are not in a dropout state in\n the given frame.\n\n Parameters\n ----------\n frame_no : int\n The frame of data we will use for attaching marker bodies.\n '''\n assert not self.joints\n for label, j in self.channels.items():\n target = self.targets.get(label)\n if target is None:\n continue\n if self.visibility[frame_no, j] < 0:\n continue\n if np.linalg.norm(self.velocities[frame_no, j]) > 10:\n continue\n joint = ode.BallJoint(self.world.ode_world, self.jointgroup)\n joint.attach(self.bodies[label].ode_body, target.ode_body)\n joint.setAnchor1Rel([0, 0, 0])\n joint.setAnchor2Rel(self.offsets[label])\n joint.setParam(ode.ParamCFM, self.cfms[frame_no, j])\n joint.setParam(ode.ParamERP, self.erp)\n joint.name = label\n self.joints[label] = joint\n self._frame_no = frame_no"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef distances(self):\n '''Get a list of the distances between markers and their attachments.\n\n Returns\n -------\n distances : ndarray of shape (num-markers, 3)\n Array of distances for each marker joint in our attachment setup. If\n a marker does not currently have an associated joint (e.g. because\n it is not currently visible) this will contain NaN for that row.\n '''\n distances = []\n for label in self.labels:\n joint = self.joints.get(label)\n distances.append([np.nan, np.nan, np.nan] if joint is None else\n np.array(joint.getAnchor()) - joint.getAnchor2())\n return np.array(distances)", "response": "Get a list of the distances between markers and their attachments."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn an array of the forces exerted by the marker springs.", "response": "def forces(self, dx_tm1=None):\n '''Return an array of the forces exerted by marker springs.\n\n Notes\n -----\n\n The forces exerted by the marker springs can be approximated by::\n\n F = kp * dx\n\n where ``dx`` is the current array of marker distances. An even more\n accurate value is computed by approximating the velocity of the spring\n displacement::\n\n F = kp * dx + kd * (dx - dx_tm1) / dt\n\n where ``dx_tm1`` is an array of distances from the previous time step.\n\n Parameters\n ----------\n dx_tm1 : ndarray\n An array of distances from markers to their attachment targets,\n measured at the previous time step.\n\n Returns\n -------\n F : ndarray\n An array of forces that the markers are exerting on the skeleton.\n '''\n cfm = self.cfms[self._frame_no][:, None]\n kp = self.erp / (cfm * self.world.dt)\n kd = (1 - self.erp) / cfm\n dx = self.distances()\n F = kp * dx\n if dx_tm1 is not None:\n bad = np.isnan(dx) | np.isnan(dx_tm1)\n F[~bad] += (kd * (dx - dx_tm1) / self.world.dt)[~bad]\n return F"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_skeleton(self, filename, pid_params=None):\n '''Create and configure a skeleton in our model.\n\n Parameters\n ----------\n filename : str\n The name of a file containing skeleton configuration data.\n pid_params : dict, optional\n If given, use this dictionary to set the PID controller\n parameters on each joint in the skeleton. See\n :func:`pagoda.skeleton.pid` for more information.\n '''\n self.skeleton = skeleton.Skeleton(self)\n self.skeleton.load(filename, color=(0.3, 0.5, 0.9, 0.8))\n if pid_params:\n self.skeleton.set_pid_params(**pid_params)\n self.skeleton.erp = 0.1\n self.skeleton.cfm = 0", "response": "Create and configure a skeleton in our model."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the marker data and attachment preferences into the model.", "response": "def load_markers(self, filename, attachments, max_frames=1e100):\n '''Load marker data and attachment preferences into the model.\n\n Parameters\n ----------\n filename : str\n The name of a file containing marker data. This currently needs to\n be either a .C3D or a .CSV file. CSV files must adhere to a fairly\n strict column naming convention; see :func:`Markers.load_csv` for\n more information.\n attachments : str\n The name of a text file specifying how markers are attached to\n skeleton bodies.\n max_frames : number, optional\n Only read in this many frames of marker data. By default, the entire\n data file is read into memory.\n\n Returns\n -------\n markers : :class:`Markers`\n Returns a markers object containing loaded marker data as well as\n skeleton attachment configuration.\n '''\n self.markers = Markers(self)\n fn = filename.lower()\n if fn.endswith('.c3d'):\n self.markers.load_c3d(filename, max_frames=max_frames)\n elif fn.endswith('.csv') or fn.endswith('.csv.gz'):\n self.markers.load_csv(filename, max_frames=max_frames)\n else:\n logging.fatal('%s: not sure how to load markers!', filename)\n self.markers.load_attachments(attachments, self.skeleton)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadvance the physics world by one step.", "response": "def step(self, substeps=2):\n '''Advance the physics world by one step.\n\n Typically this is called as part of a :class:`pagoda.viewer.Viewer`, but\n it can also be called manually (or some other stepping mechanism\n entirely can be used).\n '''\n # by default we step by following our loaded marker data.\n self.frame_no += 1\n try:\n next(self.follower)\n except (AttributeError, StopIteration) as err:\n self.reset()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsettle the skeleton to our kinematic data at a specific frame.", "response": "def settle_to_markers(self, frame_no=0, max_distance=0.05, max_iters=300,\n states=None):\n '''Settle the skeleton to our marker data at a specific frame.\n\n Parameters\n ----------\n frame_no : int, optional\n Settle the skeleton to marker data at this frame. Defaults to 0.\n max_distance : float, optional\n The settling process will stop when the mean marker distance falls\n below this threshold. Defaults to 0.1m (10cm). Setting this too\n small prevents the settling process from finishing (it will loop\n indefinitely), and setting it too large prevents the skeleton from\n settling to a stable state near the markers.\n max_iters : int, optional\n Attempt to settle markers for at most this many iterations. Defaults\n to 1000.\n states : list of body states, optional\n If given, set the bodies in our skeleton to these kinematic states\n before starting the settling process.\n '''\n if states is not None:\n self.skeleton.set_body_states(states)\n dist = None\n for _ in range(max_iters):\n for _ in self._step_to_marker_frame(frame_no):\n pass\n dist = np.nanmean(abs(self.markers.distances()))\n logging.info('settling to frame %d: marker distance %.3f', frame_no, dist)\n if dist < max_distance:\n return self.skeleton.get_body_states()\n for b in self.skeleton.bodies:\n b.linear_velocity = 0, 0, 0\n b.angular_velocity = 0, 0, 0\n return states"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef follow_markers(self, start=0, end=1e100, states=None):\n '''Iterate over a set of marker data, dragging its skeleton along.\n\n Parameters\n ----------\n start : int, optional\n Start following marker data after this frame. Defaults to 0.\n end : int, optional\n Stop following marker data after this frame. Defaults to the end of\n the marker data.\n states : list of body states, optional\n If given, set the states of the skeleton bodies to these values\n before starting to follow the marker data.\n '''\n if states is not None:\n self.skeleton.set_body_states(states)\n for frame_no, frame in enumerate(self.markers):\n if frame_no < start:\n continue\n if frame_no >= end:\n break\n for states in self._step_to_marker_frame(frame_no):\n yield states", "response": "Iterate over a set of marker data dragging its skeleton along."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the simulation to a specific frame of marker data.", "response": "def _step_to_marker_frame(self, frame_no, dt=None):\n '''Update the simulator to a specific frame of marker data.\n\n This method returns a generator of body states for the skeleton! This\n generator must be exhausted (e.g., by consuming this call in a for loop)\n for the simulator to work properly.\n\n This process involves the following steps:\n\n - Move the markers to their new location:\n - Detach from the skeleton\n - Update marker locations\n - Reattach to the skeleton\n - Detect ODE collisions\n - Yield the states of the bodies in the skeleton\n - Advance the ODE world one step\n\n Parameters\n ----------\n frame_no : int\n Step to this frame of marker data.\n dt : float, optional\n Step with this time duration. Defaults to ``self.dt``.\n\n Returns\n -------\n states : sequence of state tuples\n A generator of a sequence of one body state for the skeleton. This\n generator must be exhausted for the simulation to work properly.\n '''\n # update the positions and velocities of the markers.\n self.markers.detach()\n self.markers.reposition(frame_no)\n self.markers.attach(frame_no)\n\n # detect collisions.\n self.ode_space.collide(None, self.on_collision)\n\n # record the state of each skeleton body.\n states = self.skeleton.get_body_states()\n self.skeleton.set_body_states(states)\n\n # yield the current simulation state to our caller.\n yield states\n\n # update the ode world.\n self.ode_world.step(dt or self.dt)\n\n # clear out contact joints to prepare for the next frame.\n self.ode_contactgroup.empty()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfollow a set of kinematics and yield the kinematic joint angles for each frame of the skeleton.", "response": "def inverse_kinematics(self, start=0, end=1e100, states=None, max_force=20):\n '''Follow a set of marker data, yielding kinematic joint angles.\n\n Parameters\n ----------\n start : int, optional\n Start following marker data after this frame. Defaults to 0.\n end : int, optional\n Stop following marker data after this frame. Defaults to the end of\n the marker data.\n states : list of body states, optional\n If given, set the states of the skeleton bodies to these values\n before starting to follow the marker data.\n max_force : float, optional\n Allow each degree of freedom in the skeleton to exert at most this\n force when attempting to maintain its equilibrium position. This\n defaults to 20N. Set this value higher to simulate a stiff skeleton\n while following marker data.\n\n Returns\n -------\n angles : sequence of angle frames\n Returns a generator of joint angle data for the skeleton. One set of\n joint angles will be generated for each frame of marker data between\n `start` and `end`.\n '''\n zeros = None\n if max_force > 0:\n self.skeleton.enable_motors(max_force)\n zeros = np.zeros(self.skeleton.num_dofs)\n for _ in self.follow_markers(start, end, states):\n if zeros is not None:\n self.skeleton.set_target_angles(zeros)\n yield self.skeleton.joint_angles"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef inverse_dynamics(self, angles, start=0, end=1e100, states=None, max_force=100):\n '''Follow a set of angle data, yielding dynamic joint torques.\n\n Parameters\n ----------\n angles : ndarray (num-frames x num-dofs)\n Follow angle data provided by this array of angle values.\n start : int, optional\n Start following angle data after this frame. Defaults to the start\n of the angle data.\n end : int, optional\n Stop following angle data after this frame. Defaults to the end of\n the angle data.\n states : list of body states, optional\n If given, set the states of the skeleton bodies to these values\n before starting to follow the marker data.\n max_force : float, optional\n Allow each degree of freedom in the skeleton to exert at most this\n force when attempting to follow the given joint angles. Defaults to\n 100N. Setting this value to be large results in more accurate\n following but can cause oscillations in the PID controllers,\n resulting in noisy torques.\n\n Returns\n -------\n torques : sequence of torque frames\n Returns a generator of joint torque data for the skeleton. One set\n of joint torques will be generated for each frame of angle data\n between `start` and `end`.\n '''\n if states is not None:\n self.skeleton.set_body_states(states)\n\n for frame_no, frame in enumerate(angles):\n if frame_no < start:\n continue\n if frame_no >= end:\n break\n\n self.ode_space.collide(None, self.on_collision)\n\n states = self.skeleton.get_body_states()\n self.skeleton.set_body_states(states)\n\n # joseph's stability fix: step to compute torques, then reset the\n # skeleton to the start of the step, and then step using computed\n # torques. thus any numerical errors between the body states after\n # stepping using angle constraints will be removed, because we\n # will be stepping the model using the computed torques.\n\n self.skeleton.enable_motors(max_force)\n self.skeleton.set_target_angles(angles[frame_no])\n self.ode_world.step(self.dt)\n torques = self.skeleton.joint_torques\n self.skeleton.disable_motors()\n\n self.skeleton.set_body_states(states)\n self.skeleton.add_torques(torques)\n yield torques\n self.ode_world.step(self.dt)\n\n self.ode_contactgroup.empty()", "response": "Follow a set of angles and return a generator of dynamic joint torques."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmoving the body according to a set of torques data.", "response": "def forward_dynamics(self, torques, start=0, states=None):\n '''Move the body according to a set of torque data.'''\n if states is not None:\n self.skeleton.set_body_states(states)\n for frame_no, torque in enumerate(torques):\n if frame_no < start:\n continue\n if frame_no >= end:\n break\n self.ode_space.collide(None, self.on_collision)\n self.skeleton.add_torques(torque)\n self.ode_world.step(self.dt)\n yield\n self.ode_contactgroup.empty()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsort values but put numbers after alphabetically sorted words.", "response": "def resorted(values):\n \"\"\"\n Sort values, but put numbers after alphabetically sorted words.\n\n This function is here to make outputs diff-compatible with Aleph.\n\n Example::\n >>> sorted([\"b\", \"1\", \"a\"])\n ['1', 'a', 'b']\n >>> resorted([\"b\", \"1\", \"a\"])\n ['a', 'b', '1']\n\n Args:\n values (iterable): any iterable object/list/tuple/whatever.\n\n Returns:\n list of sorted values, but with numbers after words\n \"\"\"\n if not values:\n return values\n\n values = sorted(values)\n\n # look for first word\n first_word = next(\n (cnt for cnt, val in enumerate(values)\n if val and not val[0].isdigit()),\n None\n )\n\n # if not found, just return the values\n if first_word is None:\n return values\n\n words = values[first_word:]\n numbers = values[:first_word]\n\n return words + numbers"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndraw all bodies in the world.", "response": "def render(self, dt):\n '''Draw all bodies in the world.'''\n for frame in self._frozen:\n for body in frame:\n self.draw_body(body)\n for body in self.world.bodies:\n self.draw_body(body)\n\n if hasattr(self.world, 'markers'):\n # draw line between anchor1 and anchor2 for marker joints.\n window.glColor4f(0.9, 0.1, 0.1, 0.9)\n window.glLineWidth(3)\n for j in self.world.markers.joints.values():\n window.glBegin(window.GL_LINES)\n window.glVertex3f(*j.getAnchor())\n window.glVertex3f(*j.getAnchor2())\n window.glEnd()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_stream(self, error_callback=None, live=True):\n self.join()\n return Stream(self, error_callback=error_callback, live=live)", "response": "Get a stream that will listen for messages."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_users(self, sort=True):\n self._load()\n if sort:\n self.users.sort(key=operator.itemgetter(\"name\"))\n return self.users", "response": "Get list of users in the room."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef recent(self, message_id=None, limit=None):\n parameters = {}\n if message_id:\n parameters[\"since_message_id\"] = message_id\n if limit:\n parameters[\"limit\"] = limit\n messages = self._connection.get(\"room/%s/recent\" % self.id, key=\"messages\", parameters=parameters)\n if messages:\n messages = [Message(self._campfire, message) for message in messages]\n return messages", "response": "Returns a list of recent messages."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_name(self, name):\n if not self._campfire.get_user().admin:\n return False\n\n result = self._connection.put(\"room/%s\" % self.id, {\"room\": {\"name\": name}})\n if result[\"success\"]:\n self._load()\n return result[\"success\"]", "response": "Set the room name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_topic(self, topic):\n if not topic:\n topic = ''\n result = self._connection.put(\"room/%s\" % self.id, {\"room\": {\"topic\": topic}})\n if result[\"success\"]:\n self._load()\n\n return result[\"success\"]", "response": "Set the room topic."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef speak(self, message):\n campfire = self.get_campfire()\n if not isinstance(message, Message):\n message = Message(campfire, message)\n\n result = self._connection.post(\n \"room/%s/speak\" % self.id,\n {\"message\": message.get_data()},\n parse_data=True,\n key=\"message\"\n )\n\n if result[\"success\"]:\n return Message(campfire, result[\"data\"])\n return result[\"success\"]", "response": "Post a message to the campfire."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transcript(self, for_date=None):\n url = \"room/%s/transcript\" % self.id\n if for_date:\n url = \"%s/%d/%d/%d\" % (url, for_date.year, for_date.month, for_date.day)\n messages = self._connection.get(url, key=\"messages\")\n if messages:\n messages = [Message(self._campfire, message) for message in messages]\n return messages", "response": "Get the most recent messages for a specific date."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a new Upload object and uploads a file to the server.", "response": "def upload(self, path, progress_callback=None, finished_callback=None, error_callback=None):\n \"\"\" Create a new thread to upload a file (thread should be\n then started with start() to perform upload.)\n\n Args:\n path (str): Path to file\n\n Kwargs:\n progress_callback (func): Callback to call as file is uploaded (parameters: current, total)\n finished_callback (func): Callback to call when upload is finished\n error_callback (func): Callback to call when an error occurred (parameters: exception)\n\n Returns:\n :class:`Upload`. Upload thread\n \"\"\"\n return Upload(\n self,\n {\"upload\": path},\n progress_callback = progress_callback,\n finished_callback = finished_callback,\n error_callback = error_callback\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_new_call(group_name, app_name, search_path, filename, require_load,\n version, secure):\n # type: (str, str, Optional[str], str, bool, Optional[str], bool) -> str\n '''\n Build a call to use the new ``get_config`` function from args passed to\n ``Config.__init__``.\n '''\n new_call_kwargs = {\n 'group_name': group_name,\n 'filename': filename\n } # type: Dict[str, Any]\n new_call_lookup_options = {} # type: Dict[str, Any]\n new_call_lookup_options['secure'] = secure\n if search_path:\n new_call_lookup_options['search_path'] = search_path\n if require_load:\n new_call_lookup_options['require_load'] = require_load\n if version:\n new_call_lookup_options['version'] = version\n if new_call_lookup_options:\n new_call_kwargs['lookup_options'] = new_call_lookup_options\n\n output = build_call_str('get_config', (app_name,), new_call_kwargs)\n return output", "response": "Builds a call to use the new get_config function from args passed to\n Config. __init__."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_call_str(prefix, args, kwargs):\n # type: (str, Any, Any) -> str\n '''\n Build a callable Python string for a function call. The output will be\n combined similar to this template::\n\n (, )\n\n Example::\n\n >>> build_call_str('foo', (1, 2), {'a': '10'})\n \"foo(1, 2, a='10')\"\n '''\n kwargs_str = ', '.join(['%s=%r' % (key, value) for key, value in\n kwargs.items()])\n args_str = ', '.join([repr(arg) for arg in args])\n output = [prefix, '(']\n if args:\n output.append(args_str)\n if args and kwargs:\n output.append(', ')\n if kwargs:\n output.append(kwargs_str)\n output.append(')')\n return ''.join(output)", "response": "Builds a callable Python string for a function call."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of paths to XDG directories that are set in the XDG_CONFIG_DIRS environment variable or the appropriate default if none is set.", "response": "def get_xdg_dirs(self):\n # type: () -> List[str]\n \"\"\"\n Returns a list of paths specified by the XDG_CONFIG_DIRS environment\n variable or the appropriate default.\n\n The list is sorted by precedence, with the most important item coming\n *last* (required by the existing config_resolver logic).\n \"\"\"\n config_dirs = getenv('XDG_CONFIG_DIRS', '')\n if config_dirs:\n self._log.debug('XDG_CONFIG_DIRS is set to %r', config_dirs)\n output = []\n for path in reversed(config_dirs.split(':')):\n output.append(join(path, self.group_name, self.app_name))\n return output\n return ['/etc/xdg/%s/%s' % (self.group_name, self.app_name)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the value specified in the XDG_CONFIG_HOME environment variable or the appropriate default value.", "response": "def get_xdg_home(self):\n # type: () -> str\n \"\"\"\n Returns the value specified in the XDG_CONFIG_HOME environment variable\n or the appropriate default.\n \"\"\"\n config_home = getenv('XDG_CONFIG_HOME', '')\n if config_home:\n self._log.debug('XDG_CONFIG_HOME is set to %r', config_home)\n return expanduser(join(config_home, self.group_name, self.app_name))\n return expanduser('~/.config/%s/%s' % (self.group_name, self.app_name))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the filename which is effectively used by the application.", "response": "def _effective_filename(self):\n # type: () -> str\n \"\"\"\n Returns the filename which is effectively used by the application. If\n overridden by an environment variable, it will return that filename.\n \"\"\"\n # same logic for the configuration filename. First, check if we were\n # initialized with a filename...\n config_filename = ''\n if self.filename:\n config_filename = self.filename\n\n # ... next, take the value from the environment\n env_filename = getenv(self.env_filename_name)\n if env_filename:\n self._log.info('Configuration filename was overridden with %r '\n 'by the environment variable %s.',\n env_filename,\n self.env_filename_name)\n config_filename = env_filename\n\n return config_filename"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _effective_path(self):\n # type: () -> List[str]\n \"\"\"\n Returns a list of paths to search for config files in reverse order of\n precedence. In other words: the last path element will override the\n settings from the first one.\n \"\"\"\n # default search path\n path = (['/etc/%s/%s' % (self.group_name, self.app_name)] +\n self.get_xdg_dirs() +\n [expanduser('~/.%s/%s' % (self.group_name, self.app_name)),\n self.get_xdg_home(),\n join(getcwd(), '.{}'.format(self.group_name), self.app_name)])\n\n # If a path was passed directly to this instance, override the path.\n if self.search_path:\n path = self.search_path.split(pathsep)\n\n # Next, consider the environment variables...\n env_path = getenv(self.env_path_name)\n\n if env_path and env_path.startswith('+'):\n # If prefixed with a '+', append the path elements\n additional_paths = env_path[1:].split(pathsep)\n self._log.info('Search path extended with %r by the environment '\n 'variable %s.',\n additional_paths,\n self.env_path_name)\n path.extend(additional_paths)\n elif env_path:\n # Otherwise, override again. This takes absolute precedence.\n self._log.info(\"Configuration search path was overridden with \"\n \"%r by the environment variable %r.\",\n env_path,\n self.env_path_name)\n path = env_path.split(pathsep)\n\n return path", "response": "Returns the effective path for the current config file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_file(self, filename):\n # type: (str) -> bool\n \"\"\"\n Check if ``filename`` can be read. Will return boolean which is True if\n the file can be read, False otherwise.\n \"\"\"\n if not exists(filename):\n return False\n\n # Check if the file is version-compatible with this instance.\n new_config = ConfigResolverBase()\n new_config.read(filename)\n if self.version and not new_config.has_option('meta', 'version'):\n # self.version is set, so we MUST have a version in the file!\n raise NoVersionError(\n \"The config option 'meta.version' is missing in {}. The \"\n \"application expects version {}!\".format(filename,\n self.version))\n elif not self.version and new_config.has_option('meta', 'version'):\n # Automatically \"lock-in\" a version number if one is found.\n # This prevents loading a chain of config files with incompatible\n # version numbers!\n self.version = StrictVersion(new_config.get('meta', 'version'))\n self._log.info('%r contains a version number, but the config '\n 'instance was not created with a version '\n 'restriction. Will set version number to \"%s\" to '\n 'prevent accidents!',\n filename, self.version)\n elif self.version:\n # This instance expected a certain version. We need to check the\n # version in the file and compare.\n file_version = new_config.get('meta', 'version')\n major, minor, _ = StrictVersion(file_version).version\n expected_major, expected_minor, _ = self.version.version\n if expected_major != major:\n self._log.error(\n 'Invalid major version number in %r. Expected %r, got %r!',\n abspath(filename),\n str(self.version),\n file_version)\n return False\n\n if expected_minor != minor:\n self._log.warning(\n 'Mismatching minor version number in %r. '\n 'Expected %r, got %r!',\n abspath(filename),\n str(self.version),\n file_version)\n return True\n return True", "response": "Checks if the file exists and if it can be read. Will return boolean which is True if the file can be read False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\noverride :py:meth:`configparser.ConfigParser.get`. In addition to ``section`` and ``option``, this call takes an optional ``default`` value. This behaviour works in *addition* to the :py:class:`configparser.ConfigParser` default mechanism. Note that a default value from ``ConfigParser`` takes precedence. The reason this additional functionality is added, is because the defaults of :py:class:`configparser.ConfigParser` are not dependent on sections. If you specify a default for the option ``test``, then this value will be returned for both ``section1.test`` and for ``section2.test``. Using the default on the ``get`` call gives you more fine-grained control over this. Also note, that if a default value was used, it will be logged with level ``logging.DEBUG``. :param section: The config file section. :param option: The option name. :param kwargs: These keyword args are passed through to :py:meth:`configparser.ConfigParser.get`.", "response": "def get(self, section, option, **kwargs): # type: ignore\n # type: (str, str, Any) -> Any\n \"\"\"\n Overrides :py:meth:`configparser.ConfigParser.get`.\n\n In addition to ``section`` and ``option``, this call takes an optional\n ``default`` value. This behaviour works in *addition* to the\n :py:class:`configparser.ConfigParser` default mechanism. Note that\n a default value from ``ConfigParser`` takes precedence.\n\n The reason this additional functionality is added, is because the\n defaults of :py:class:`configparser.ConfigParser` are not dependent\n on sections. If you specify a default for the option ``test``, then\n this value will be returned for both ``section1.test`` and for\n ``section2.test``. Using the default on the ``get`` call gives you more\n fine-grained control over this.\n\n Also note, that if a default value was used, it will be logged with\n level ``logging.DEBUG``.\n\n :param section: The config file section.\n :param option: The option name.\n :param kwargs: These keyword args are passed through to\n :py:meth:`configparser.ConfigParser.get`.\n \"\"\"\n if \"default\" in kwargs:\n default = kwargs.pop(\"default\")\n new_kwargs = {'fallback': default}\n new_kwargs.update(kwargs)\n new_call = build_call_str('.get', (section, option), new_kwargs)\n warn('Using the \"default\" argument to Config.get() will no '\n 'longer work in config_resolver 5.0! Version 5 will return '\n 'standard Python ConfigParser instances which use \"fallback\" '\n 'instead of \"default\". Replace your code with \"%s\"' % new_call,\n DeprecationWarning,\n stacklevel=2)\n have_default = True\n else:\n have_default = False\n\n try:\n value = super(Config, self).get(section, option, **kwargs)\n return value\n except (NoSectionError, NoOptionError) as exc:\n if have_default:\n self._log.debug(\"%s: Returning default value %r\", exc, default)\n return default\n else:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load(self, reload=False, require_load=False):\n # type: (bool, bool) -> None\n \"\"\"\n Searches for an appropriate config file. If found, loads the file into\n the current instance. This method can also be used to reload a\n configuration. Note that you may want to set ``reload`` to ``True`` to\n clear the configuration before loading in that case. Without doing\n that, values will remain available even if they have been removed from\n the config files.\n\n :param reload: if set to ``True``, the existing values are cleared\n before reloading.\n :param require_load: If set to ``True`` this will raise a\n :py:exc:`IOError` if no config file has been found\n to load.\n \"\"\"\n\n if reload: # pragma: no cover\n self.config = None\n\n # only load the config if necessary (or explicitly requested)\n if self.config: # pragma: no cover\n self._log.debug('Returning cached config instance. Use '\n '``reload=True`` to avoid caching!')\n return\n\n path = self._effective_path()\n config_filename = self._effective_filename()\n\n # Next, use the resolved path to find the filenames. Keep track of\n # which files we loaded in order to inform the user.\n self._active_path = [join(_, config_filename) for _ in path]\n for dirname in path:\n conf_name = join(dirname, config_filename)\n readable = self.check_file(conf_name)\n if readable:\n action = 'Updating' if self._loaded_files else 'Loading initial'\n self._log.info('%s config from %s', action, conf_name)\n self.read(conf_name)\n if conf_name == expanduser(\"~/.%s/%s/%s\" % (\n self.group_name, self.app_name, self.filename)):\n self._log.warning(\n \"DEPRECATION WARNING: The file \"\n \"'%s/.%s/%s/app.ini' was loaded. The XDG \"\n \"Basedir standard requires this file to be in \"\n \"'%s/.config/%s/%s/app.ini'! This location \"\n \"will no longer be parsed in a future version of \"\n \"config_resolver! You can already (and should) move \"\n \"the file!\", expanduser(\"~\"), self.group_name,\n self.app_name, expanduser(\"~\"), self.group_name,\n self.app_name)\n self._loaded_files.append(conf_name)\n\n if not self._loaded_files and not require_load:\n self._log.warning(\n \"No config file named %s found! Search path was %r\",\n config_filename,\n path)\n elif not self._loaded_files and require_load:\n raise IOError(\"No config file named %s found! Search path \"\n \"was %r\" % (config_filename, path))", "response": "Loads the config file into the current instance."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check_file(self, filename):\n # type: (str) -> bool\n \"\"\"\n Overrides :py:meth:`.Config.check_file`\n \"\"\"\n can_read = super(SecuredConfig, self).check_file(filename)\n if not can_read:\n return False\n\n mode = get_stat(filename).st_mode\n if (mode & stat.S_IRGRP) or (mode & stat.S_IROTH):\n msg = \"File %r is not secure enough. Change it's mode to 600\"\n self._log.warning(msg, filename)\n return False\n return True", "response": "Checks if a file is secure enough."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting up the environment variables for the current Python interpreter.", "response": "def setup_environ(self):\n \"\"\"https://www.python.org/dev/peps/pep-0333/#environ-variables\"\"\"\n # Set up base environment\n env = self.base_environ = {}\n env['SERVER_NAME'] = self.server_name\n env['GATEWAY_INTERFACE'] = 'CGI/1.1'\n env['SERVER_PORT'] = str(self.server_port)\n env['REMOTE_HOST'] = ''\n env['CONTENT_LENGTH'] = ''\n env['SCRIPT_NAME'] = ''"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_environ(self):\n env = self.base_environ.copy()\n env['REQUEST_METHOD'] = self.request_method\n\n if '?' in self.path:\n path, query = self.path.split('?', 1)\n else:\n path, query = self.path, ''\n env['PATH_INFO'] = urllib.parse.unquote(path)\n env['QUERY_STRING'] = query\n\n env['CONTENT_TYPE'] = self.headers.get('Content-Type', '')\n env['CONTENT_LENGTH'] = self.headers.get('Content-Length', '0')\n\n env['SERVER_PROTOCOL'] = self.request_version\n env['REMOTE_ADDR'] = self.client_address[0]\n env['REMOTE_PORT'] = self.client_address[1]\n\n env['wsgi.version'] = (1, 0)\n env['wsgi.url_scheme'] = 'http'\n env['wsgi.input'] = io.BytesIO(self.raw_request)\n env['wsgi.errors'] = sys.stderr\n env['wsgi.multithread'] = False\n env['wsgi.multiprocess'] = True\n env['wsgi.run_once'] = False\n\n for k, v in self.headers.items():\n k = k.replace('-', '_').upper()\n if k in env:\n continue\n env['HTTP_' + k] = v\n return env", "response": "Returns a copy of the environment with all the necessary variables set."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get(self, q=None, page=None):\n # Check cache to exit early if needed\n etag = generate_etag(current_ext.content_version.encode('utf8'))\n self.check_etag(etag, weak=True)\n\n # Build response\n res = jsonify(current_ext.styles)\n res.set_etag(etag)\n\n return res", "response": "Get the current styles."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a connection with given settings.", "response": "def create_from_settings(settings):\n \"\"\" Create a connection with given settings.\n\n Args:\n settings (dict): A dictionary of settings\n\n Returns:\n :class:`Connection`. The connection\n \"\"\"\n return Connection(\n settings[\"url\"], \n settings[\"base_url\"],\n settings[\"user\"],\n settings[\"password\"],\n authorizations = settings[\"authorizations\"],\n debug = settings[\"debug\"]\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete(self, url=None, post_data={}, parse_data=False, key=None, parameters=None):\n return self._fetch(\"DELETE\", url, post_data=post_data, parse_data=parse_data, key=key, parameters=parameters, full_return=True)", "response": "Issue a DELETE request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef post(self, url=None, post_data={}, parse_data=False, key=None, parameters=None, listener=None):\n return self._fetch(\"POST\", url, post_data=post_data, parse_data=parse_data, key=key, parameters=parameters, listener=listener, full_return=True)", "response": "Issue a POST request to the specified URL."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, url=None, parse_data=True, key=None, parameters=None):\n return self._fetch(\"GET\", url, post_data=None, parse_data=parse_data, key=key, parameters=parameters)", "response": "Issue a GET request to the specified URL and return the response."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_headers(self):\n headers = {\n \"User-Agent\": \"kFlame 1.0\"\n }\n\n password_url = self._get_password_url()\n if password_url and password_url in self._settings[\"authorizations\"]:\n headers[\"Authorization\"] = self._settings[\"authorizations\"][password_url]\n\n return headers", "response": "Get headers.\n\n Returns:\n tuple: Headers"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_password_url(self):\n password_url = None\n if self._settings[\"user\"] or self._settings[\"authorization\"]:\n if self._settings[\"url\"]:\n password_url = self._settings[\"url\"]\n elif self._settings[\"base_url\"]:\n password_url = self._settings[\"base_url\"]\n return password_url", "response": "Get URL used for authentication\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse(self, text, key=None):\n try:\n data = json.loads(text)\n except ValueError as e:\n raise ValueError(\"%s: Value: [%s]\" % (e, text))\n\n if data and key:\n if key not in data:\n raise ValueError(\"Invalid response (key %s not found): %s\" % (key, data))\n data = data[key]\n return data", "response": "Parses a response.\n text and returns a parsed value."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef build_twisted_request(self, method, url, extra_headers={}, body_producer=None, full_url=False):\n uri = url if full_url else self._url(url)\n\n raw_headers = self.get_headers()\n if extra_headers:\n raw_headers.update(extra_headers)\n\n headers = http_headers.Headers()\n for header in raw_headers:\n headers.addRawHeader(header, raw_headers[header])\n\n agent = client.Agent(reactor)\n request = agent.request(method, uri, headers, body_producer)\n\n return (reactor, request)", "response": "Builds a twisted request for the twisted broker."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _fetch(self, method, url=None, post_data=None, parse_data=True, key=None, parameters=None, listener=None, full_return=False):\n\n headers = self.get_headers()\n headers[\"Content-Type\"] = \"application/json\"\n\n handlers = []\n debuglevel = int(self._settings[\"debug\"])\n \n handlers.append(urllib2.HTTPHandler(debuglevel=debuglevel))\n if hasattr(httplib, \"HTTPS\"):\n handlers.append(urllib2.HTTPSHandler(debuglevel=debuglevel))\n\n handlers.append(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))\n\n password_url = self._get_password_url()\n if password_url and \"Authorization\" not in headers:\n pwd_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()\n pwd_manager.add_password(None, password_url, self._settings[\"user\"], self._settings[\"password\"])\n handlers.append(HTTPBasicAuthHandler(pwd_manager))\n\n opener = urllib2.build_opener(*handlers)\n\n if post_data is not None:\n post_data = json.dumps(post_data)\n\n uri = self._url(url, parameters)\n request = RESTRequest(uri, method=method, headers=headers)\n if post_data is not None:\n request.add_data(post_data)\n\n response = None\n\n try:\n response = opener.open(request)\n body = response.read()\n if password_url and password_url not in self._settings[\"authorizations\"] and request.has_header(\"Authorization\"):\n self._settings[\"authorizations\"][password_url] = request.get_header(\"Authorization\")\n except urllib2.HTTPError as e:\n if e.code == 401:\n raise AuthenticationError(\"Access denied while trying to access %s\" % uri)\n elif e.code == 404:\n raise ConnectionError(\"URL not found: %s\" % uri)\n else:\n raise\n except urllib2.URLError as e:\n raise ConnectionError(\"Error while fetching from %s: %s\" % (uri, e))\n finally:\n if response:\n response.close()\n\n opener.close()\n\n data = None\n if parse_data:\n if not key:\n key = string.split(url, \"/\")[0]\n\n data = self.parse(body, key)\n\n if full_return:\n info = response.info() if response else None\n status = int(string.split(info[\"status\"])[0]) if (info and \"status\" in info) else None\n\n return {\n \"success\": (status >= 200 and status < 300), \n \"data\": data, \n \"info\": info, \n \"body\": body\n }\n\n return data", "response": "Issue a request to the specified URL and return the response."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _url(self, url=None, parameters=None):\n\n uri = url or self._settings[\"url\"]\n if url and self._settings[\"base_url\"]:\n uri = \"%s/%s\" % (self._settings[\"base_url\"], url)\n uri += \".json\"\n if parameters:\n uri += \"?%s\" % urllib.urlencode(parameters)\n return uri", "response": "Build the destination URL."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntells if this message is a text message.", "response": "def is_text(self):\n \"\"\" Tells if this message is a text message.\n\n Returns:\n bool. Success\n \"\"\"\n return self.type in [\n self._TYPE_PASTE,\n self._TYPE_TEXT,\n self._TYPE_TWEET\n ]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_rooms(self, sort=True):\n rooms = self._connection.get(\"rooms\")\n if sort:\n rooms.sort(key=operator.itemgetter(\"name\"))\n return rooms", "response": "Get the rooms list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a room by name.", "response": "def get_room_by_name(self, name):\n \"\"\" Get a room by name.\n\n Returns:\n :class:`Room`. Room\n\n Raises:\n RoomNotFoundException\n \"\"\"\n rooms = self.get_rooms()\n for room in rooms or []:\n if room[\"name\"] == name:\n return self.get_room(room[\"id\"])\n raise RoomNotFoundException(\"Room %s not found\" % name)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_room(self, id):\n if id not in self._rooms:\n self._rooms[id] = Room(self, id)\n return self._rooms[id]", "response": "Get room.\n\n Returns:\n :class:`Room`. Room"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget user. Returns: :class:`User`. User", "response": "def get_user(self, id = None):\n \"\"\" Get user.\n\n Returns:\n :class:`User`. User\n \"\"\"\n if not id:\n id = self._user.id\n\n if id not in self._users:\n self._users[id] = self._user if id == self._user.id else User(self, id)\n\n return self._users[id]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsearching transcripts. Args: terms (str): Terms for search Returns: array. Messages", "response": "def search(self, terms):\n \"\"\" Search transcripts.\n\n Args:\n terms (str): Terms for search\n\n Returns:\n array. Messages\n \"\"\"\n messages = self._connection.get(\"search/%s\" % urllib.quote_plus(terms), key=\"messages\")\n if messages:\n messages = [Message(self, message) for message in messages]\n return messages"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndump a cookie into a simple cookie.", "response": "def cookie_dump(key, value='', max_age=None, expires=None, path='/',\n domain=None, secure=False, httponly=False):\n \"\"\"\n :rtype: ``Cookie.SimpleCookie``\n \"\"\"\n cookie = SimpleCookie()\n cookie[key] = value\n for attr in ('max_age', 'expires', 'path', 'domain',\n 'secure', 'httponly'):\n attr_key = attr.replace('_', '-')\n attr_value = locals()[attr]\n if attr_value:\n cookie[key][attr_key] = attr_value\n return cookie"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef response_status_string(code):\n mean = HTTP_STATUS_CODES.get(code, 'unknown').upper()\n return '{code} {mean}'.format(code=code, mean=mean)", "response": "Return a string representation of the status code."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cookies(self):\n http_cookie = self.environ.get('HTTP_COOKIE', '')\n _cookies = {\n k: v.value\n for (k, v) in SimpleCookie(http_cookie).items()\n }\n return _cookies", "response": "Request cookies\n\n :rtype: dict"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nattaches an observer function to the current stream.", "response": "def attach(self, observer):\n \"\"\" Attach an observer.\n\n Args:\n observer (func): A function to be called when new messages arrive\n\n Returns:\n :class:`Stream`. Current instance to allow chaining\n \"\"\"\n if not observer in self._observers:\n self._observers.append(observer)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef incoming(self, messages):\n if self._observers:\n campfire = self._room.get_campfire()\n for message in messages:\n for observer in self._observers:\n observer(Message(campfire, message))", "response": "Called when incoming messages arrive."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall by the thread, it runs the process. NEVER call this method directly. Instead call start() to start the thread. To stop, call stop(), and then join()", "response": "def run(self):\n \"\"\" Called by the thread, it runs the process.\n\n NEVER call this method directly. Instead call start() to start the thread.\n\n To stop, call stop(), and then join()\n \"\"\"\n\n if self._live:\n self._use_process = True\n\n self._abort = False\n campfire = self._room.get_campfire()\n\n if self._live:\n process = LiveStreamProcess(campfire.get_connection().get_settings(), self._room.id)\n else:\n process = StreamProcess(campfire.get_connection().get_settings(), self._room.id, pause=self._pause)\n\n if not self._use_process:\n process.set_callback(self.incoming)\n\n if self._use_process:\n queue = Queue()\n process.set_queue(queue)\n process.start()\n if not process.is_alive():\n return\n\n self._streaming = True\n\n while not self._abort:\n if self._use_process:\n if not process.is_alive():\n self._abort = True\n break\n\n try:\n incoming = queue.get_nowait()\n if isinstance(incoming, list):\n self.incoming(incoming)\n elif isinstance(incoming, Exception):\n self._abort = True\n if self._error_callback:\n self._error_callback(incoming, self._room)\n\n except Empty:\n time.sleep(self._pause)\n pass\n else:\n process.fetch()\n time.sleep(self._pause)\n\n self._streaming = False\n if self._use_process and self._abort and not process.is_alive() and self._error_callback:\n self._error_callback(Exception(\"Streaming process was killed\"), self._room)\n\n if self._use_process:\n queue.close()\n if process.is_alive():\n process.stop()\n process.terminate()\n process.join()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run(self):\n if not self._queue:\n raise Exception(\"No queue available to send messages\")\n\n while True:\n self.fetch()\n time.sleep(self._pause)", "response": "Called by the process, it runs it.\n\n NEVER call this method directly. Instead call start() to start the separate process.\n If you don't want to use a second process, then call fetch() directly on this istance.\n\n To stop, call terminate()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef received(self, messages):\n if messages:\n if self._queue:\n self._queue.put_nowait(messages)\n\n if self._callback:\n self._callback(messages)", "response": "Called when new messages arrive."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall by the process, it runs it. NEVER call this method directly. Instead call start() to start the separate process. If you don't want to use a second process, then call fetch() directly on this istance. To stop, call terminate()", "response": "def run(self):\n \"\"\" Called by the process, it runs it.\n\n NEVER call this method directly. Instead call start() to start the separate process.\n If you don't want to use a second process, then call fetch() directly on this istance.\n\n To stop, call terminate()\n \"\"\"\n if not self._queue:\n raise Exception(\"No queue available to send messages\")\n\n factory = LiveStreamFactory(self)\n self._reactor.connectSSL(\"streaming.campfirenow.com\", 443, factory, ssl.ClientContextFactory())\n self._reactor.run()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalls when a connection is made", "response": "def connectionMade(self):\n \"\"\" Called when a connection is made, and used to send out headers \"\"\"\n\n headers = [\n \"GET %s HTTP/1.1\" % (\"/room/%s/live.json\" % self.factory.get_stream().get_room_id())\n ]\n\n connection_headers = self.factory.get_stream().get_connection().get_headers()\n for header in connection_headers:\n headers.append(\"%s: %s\" % (header, connection_headers[header]))\n\n headers.append(\"Host: streaming.campfirenow.com\")\n\n self.transport.write(\"\\r\\n\".join(headers) + \"\\r\\n\\r\\n\")\n self.factory.get_stream().set_protocol(self)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lineReceived(self, line):\n while self._in_header:\n if line:\n self._headers.append(line)\n else:\n http, status, message = self._headers[0].split(\" \", 2)\n status = int(status)\n if status == 200:\n self.factory.get_stream().connected()\n else:\n self.factory.continueTrying = 0\n self.transport.loseConnection()\n self.factory.get_stream().disconnected(RuntimeError(status, message))\n return\n\n self._in_header = False\n break\n else:\n try:\n self._len_expected = int(line, 16)\n self.setRawMode()\n except:\n pass", "response": "Callback issued by twisted when new line arrives."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rawDataReceived(self, data):\n if self._len_expected is not None:\n data, extra = data[:self._len_expected], data[self._len_expected:]\n self._len_expected -= len(data)\n else:\n extra = \"\"\n\n self._buffer += data\n if self._len_expected == 0:\n data = self._buffer.strip()\n if data:\n lines = data.split(\"\\r\")\n for line in lines:\n try:\n message = self.factory.get_stream().get_connection().parse(line)\n if message:\n self.factory.get_stream().received([message])\n except ValueError:\n pass\n\n self._buffer = \"\"\n self._len_expected = None\n self.setLineMode(extra)", "response": "Process data received from the broker."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_func(self, path):\n for url_match, func_pair in self._urls_regex_map.items():\n m = url_match.match(path)\n if m is not None:\n return func_pair.func, func_pair.methods, m.groupdict()\n return None, None, None", "response": "Get the function and methods for a given path."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreplace type to regex.", "response": "def _replace_type_to_regex(cls, match):\n \"\"\" / -> r'(?P\\d+)' \"\"\"\n groupdict = match.groupdict()\n _type = groupdict.get('type')\n type_regex = cls.TYPE_REGEX_MAP.get(_type, '[^/]+')\n name = groupdict.get('name')\n return r'(?P<{name}>{type_regex})'.format(\n name=name, type_regex=type_regex\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef styles(self):\n styles = get_all_styles()\n whitelist = self.app.config.get('CSL_STYLES_WHITELIST')\n if whitelist:\n return {k: v for k, v in styles.items() if k in whitelist}\n return styles", "response": "Get a dictionary of CSL styles."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef init_app(self, app):\n state = _InvenioCSLRESTState(app)\n app.extensions['invenio-csl-rest'] = state\n return state", "response": "Initialize the Flask application."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstart producing a new entry for the specified consumer.", "response": "def startProducing(self, consumer):\n \"\"\" Start producing.\n\n Args:\n consumer: Consumer\n \"\"\"\n self._consumer = consumer\n self._current_deferred = defer.Deferred()\n self._sent = 0\n self._paused = False\n\n if not hasattr(self, \"_chunk_headers\"):\n self._build_chunk_headers()\n\n if self._data:\n block = \"\"\n for field in self._data:\n block += self._chunk_headers[field]\n block += self._data[field]\n block += \"\\r\\n\"\n\n self._send_to_consumer(block)\n\n if self._files:\n self._files_iterator = self._files.iterkeys()\n self._files_sent = 0\n self._files_length = len(self._files)\n self._current_file_path = None\n self._current_file_handle = None\n self._current_file_length = None\n self._current_file_sent = 0\n\n result = self._produce()\n if result:\n return result\n else:\n return defer.succeed(None)\n\n return self._current_deferred"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a block of bytes to the consumer.", "response": "def _send_to_consumer(self, block):\n \"\"\" Send a block of bytes to the consumer.\n\n Args:\n block (str): Block of bytes\n \"\"\"\n self._consumer.write(block)\n self._sent += len(block)\n if self._callback:\n self._callback(self._sent, self.length)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the total length of this request.", "response": "def _length(self):\n \"\"\" Returns total length for this request.\n\n Returns:\n int. Length\n \"\"\"\n self._build_chunk_headers()\n\n length = 0\n\n if self._data:\n for field in self._data:\n length += len(self._chunk_headers[field])\n length += len(self._data[field])\n length += 2\n\n if self._files:\n for field in self._files:\n length += len(self._chunk_headers[field])\n length += self._file_size(field)\n length += 2\n\n length += len(self.boundary)\n length += 6\n\n return length"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbuilding headers for each field.", "response": "def _build_chunk_headers(self):\n \"\"\" Build headers for each field. \"\"\"\n if hasattr(self, \"_chunk_headers\") and self._chunk_headers:\n return\n\n self._chunk_headers = {}\n for field in self._files:\n self._chunk_headers[field] = self._headers(field, True)\n for field in self._data:\n self._chunk_headers[field] = self._headers(field)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the headers of the encoding of this parameter.", "response": "def _headers(self, name, is_file=False):\n \"\"\" Returns the header of the encoding of this parameter.\n \n Args:\n name (str): Field name\n \n Kwargs:\n is_file (bool): If true, this is a file field\n \n Returns:\n array. Headers\n \"\"\"\n value = self._files[name] if is_file else self._data[name]\n _boundary = self.boundary.encode(\"utf-8\") if isinstance(self.boundary, unicode) else urllib.quote_plus(self.boundary)\n \n headers = [\"--%s\" % _boundary]\n\n if is_file:\n disposition = 'form-data; name=\"%s\"; filename=\"%s\"' % (name, os.path.basename(value))\n else:\n disposition = 'form-data; name=\"%s\"' % name\n\n headers.append(\"Content-Disposition: %s\" % disposition)\n\n if is_file:\n file_type = self._file_type(name)\n else:\n file_type = \"text/plain; charset=utf-8\"\n\n headers.append(\"Content-Type: %s\" % file_type)\n\n if is_file:\n headers.append(\"Content-Length: %i\" % self._file_size(name))\n else:\n headers.append(\"Content-Length: %i\" % len(value))\n\n headers.append(\"\")\n headers.append(\"\")\n\n return \"\\r\\n\".join(headers)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _boundary(self):\n boundary = None\n try:\n import uuid\n boundary = uuid.uuid4().hex\n except ImportError:\n import random, sha\n bits = random.getrandbits(160)\n boundary = sha.new(str(bits)).hexdigest()\n return boundary", "response": "Returns a random string to use as the boundary for a message."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _file_type(self, field):\n type = mimetypes.guess_type(self._files[field])[0]\n return type.encode(\"utf-8\") if isinstance(type, unicode) else str(type)", "response": "Returns the file type for given file field."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the size of the file in the given field.", "response": "def _file_size(self, field):\n \"\"\" Returns the file size for given file field.\n\n Args:\n field (str): File field\n\n Returns:\n int. File size\n \"\"\"\n size = 0\n try:\n handle = open(self._files[field], \"r\")\n size = os.fstat(handle.fileno()).st_size\n handle.close()\n except:\n size = 0\n self._file_lengths[field] = size\n return self._file_lengths[field]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _filename(draw, result_type=None):\n # Various ASCII chars have a special meaning for the operating system,\n # so make them more common\n ascii_char = characters(min_codepoint=0x01, max_codepoint=0x7f)\n if os.name == 'nt': # pragma: no cover\n # Windows paths can contain all surrogates and even surrogate pairs\n # if two paths are concatenated. This makes it more likely for them to\n # be generated.\n surrogate = characters(\n min_codepoint=0xD800, max_codepoint=0xDFFF)\n uni_char = characters(min_codepoint=0x1)\n text_strategy = text(\n alphabet=one_of(uni_char, surrogate, ascii_char))\n\n def text_to_bytes(path):\n fs_enc = sys.getfilesystemencoding()\n try:\n return path.encode(fs_enc, 'surrogatepass')\n except UnicodeEncodeError:\n return path.encode(fs_enc, 'replace')\n\n bytes_strategy = text_strategy.map(text_to_bytes)\n else:\n latin_char = characters(min_codepoint=0x01, max_codepoint=0xff)\n bytes_strategy = text(alphabet=one_of(latin_char, ascii_char)).map(\n lambda t: t.encode('latin-1'))\n\n unix_path_text = bytes_strategy.map(\n lambda b: b.decode(\n sys.getfilesystemencoding(),\n 'surrogateescape' if PY3 else 'ignore'))\n\n # Two surrogates generated through surrogateescape can generate\n # a valid utf-8 sequence when encoded and result in a different\n # code point when decoded again. Can happen when two paths get\n # concatenated. Shuffling makes it possible to generate such a case.\n text_strategy = permutations(draw(unix_path_text)).map(u\"\".join)\n\n if result_type is None:\n return draw(one_of(bytes_strategy, text_strategy))\n elif result_type is bytes:\n return draw(bytes_strategy)\n else:\n return draw(text_strategy)", "response": "Generate a path value of type result_type."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _str_to_path(s, result_type):\n\n assert isinstance(s, str)\n if isinstance(s, bytes) and result_type is text_type:\n return s.decode('ascii')\n elif isinstance(s, text_type) and result_type is bytes:\n return s.encode('ascii')\n return s", "response": "Given an ASCII str returns a path of the given type."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate a root component for a path.", "response": "def _path_root(draw, result_type):\n \"\"\"Generates a root component for a path.\"\"\"\n\n # Based on https://en.wikipedia.org/wiki/Path_(computing)\n\n def tp(s=''):\n return _str_to_path(s, result_type)\n\n if os.name != 'nt':\n return tp(os.sep)\n\n sep = sampled_from([os.sep, os.altsep or os.sep]).map(tp)\n name = _filename(result_type)\n char = characters(min_codepoint=ord(\"A\"), max_codepoint=ord(\"z\")).map(\n lambda c: tp(str(c)))\n\n relative = sep\n # [drive_letter]:\\\n drive = builds(lambda *x: tp().join(x), char, just(tp(':')), sep)\n # \\\\?\\[drive_spec]:\\\n extended = builds(\n lambda *x: tp().join(x), sep, sep, just(tp('?')), sep, drive)\n\n network = one_of([\n # \\\\[server]\\[sharename]\\\n builds(lambda *x: tp().join(x), sep, sep, name, sep, name, sep),\n # \\\\?\\[server]\\[sharename]\\\n builds(lambda *x: tp().join(x),\n sep, sep, just(tp('?')), sep, name, sep, name, sep),\n # \\\\?\\UNC\\[server]\\[sharename]\\\n builds(lambda *x: tp().join(x),\n sep, sep, just(tp('?')), sep, just(tp('UNC')), sep, name, sep,\n name, sep),\n # \\\\.\\[physical_device]\\\n builds(lambda *x: tp().join(x),\n sep, sep, just(tp('.')), sep, name, sep),\n ])\n\n final = one_of(relative, drive, extended, network)\n\n return draw(final)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fspaths(draw, allow_pathlike=None):\n has_pathlike = hasattr(os, 'PathLike')\n\n if allow_pathlike is None:\n allow_pathlike = has_pathlike\n if allow_pathlike and not has_pathlike:\n raise InvalidArgument(\n 'allow_pathlike: os.PathLike not supported, use None instead '\n 'to enable it only when available')\n\n result_type = draw(sampled_from([bytes, text_type]))\n\n def tp(s=''):\n return _str_to_path(s, result_type)\n\n special_component = sampled_from([tp(os.curdir), tp(os.pardir)])\n normal_component = _filename(result_type)\n path_component = one_of(normal_component, special_component)\n extension = normal_component.map(lambda f: tp(os.extsep) + f)\n root = _path_root(result_type)\n\n def optional(st):\n return one_of(st, just(result_type()))\n\n sep = sampled_from([os.sep, os.altsep or os.sep]).map(tp)\n path_part = builds(lambda s, l: s.join(l), sep, lists(path_component))\n main_strategy = builds(lambda *x: tp().join(x),\n optional(root), path_part, optional(extension))\n\n if allow_pathlike and hasattr(os, 'fspath'):\n pathlike_strategy = main_strategy.map(lambda p: _PathLike(p))\n main_strategy = one_of(main_strategy, pathlike_strategy)\n\n return draw(main_strategy)", "response": "A strategy which generates filesystem path values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling the extends command", "response": "def handle_extends(self, text):\n \"\"\"replace all blocks in extends with current blocks\"\"\"\n match = self.re_extends.match(text)\n if match:\n extra_text = self.re_extends.sub('', text, count=1)\n blocks = self.get_blocks(extra_text)\n path = os.path.join(self.base_dir, match.group('path'))\n with open(path, encoding='utf-8') as fp:\n return self.replace_blocks_in_extends(fp.read(), blocks)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef flush_buffer(self):\n self.code_builder.add_line('{0}.extend([{1}])',\n self.result_var, ','.join(self.buffered))\n self.buffered = []", "response": "flush all buffered string into code"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nstrips the token from the given text.", "response": "def strip_token(self, text, start, end):\n \"\"\"{{ a }} -> a\"\"\"\n text = text.replace(start, '', 1)\n text = text.replace(end, '', 1)\n return text"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling by the thread, it runs the process. NEVER call this method directly. Instead call start() to start the thread. Before finishing the thread using this thread, call join()", "response": "def run(self):\n \"\"\" Called by the thread, it runs the process.\n\n NEVER call this method directly. Instead call start() to start the thread.\n\n Before finishing the thread using this thread, call join()\n \"\"\"\n queue = Queue()\n process = UploadProcess(self._connection_settings, self._room, queue, self._files)\n if self._data:\n process.add_data(self._data)\n process.start()\n\n if not process.is_alive():\n return\n\n self._uploading = True\n\n done = False\n while not self._abort and not done:\n if not process.is_alive():\n self._abort = True\n break\n\n messages = None\n try:\n data = queue.get()\n if not data:\n done = True\n if self._finished_callback:\n self._finished_callback()\n elif isinstance(data, tuple):\n sent, total = data\n if self._progress_callback:\n self._progress_callback(sent, total)\n else:\n self._abort = True\n if self._error_callback:\n self._error_callback(data, self._room)\n except Empty:\n time.sleep(0.5)\n\n self._uploading = False\n if self._abort and not process.is_alive() and self._error_callback:\n self._error_callback(Exception(\"Upload process was killed\"), self._room)\n\n queue.close()\n if process.is_alive():\n queue.close()\n process.terminate()\n process.join()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_data(self, data):\n if not self._data:\n self._data = {}\n self._data.update(data)", "response": "Add POST data.\n\n Args:\n data (dict): key => value dictionary"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall by the process, it runs it. NEVER call this method directly. Instead call start() to start the separate process. If you don't want to use a second process, then call fetch() directly on this istance. To stop, call terminate()", "response": "def run(self):\n \"\"\" Called by the process, it runs it.\n\n NEVER call this method directly. Instead call start() to start the separate process.\n If you don't want to use a second process, then call fetch() directly on this istance.\n\n To stop, call terminate()\n \"\"\"\n\n producer_deferred = defer.Deferred()\n producer_deferred.addCallback(self._request_finished)\n producer_deferred.addErrback(self._request_error)\n\n receiver_deferred = defer.Deferred()\n receiver_deferred.addCallback(self._response_finished)\n receiver_deferred.addErrback(self._response_error)\n\n self._producer = producer.MultiPartProducer(\n self._files,\n self._data,\n callback = self._request_progress,\n deferred = producer_deferred\n )\n self._receiver = receiver.StringReceiver(receiver_deferred)\n\n headers = {\n 'Content-Type': \"multipart/form-data; boundary=%s\" % self._producer.boundary\n }\n\n self._reactor, request = self._connection.build_twisted_request(\n \"POST\",\n \"room/%s/uploads\" % self._room.id,\n extra_headers = headers,\n body_producer = self._producer\n )\n\n request.addCallback(self._response)\n request.addErrback(self._shutdown)\n\n self._reactor.run()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlogging an error message to the log file.", "response": "def log_error(self, text: str) -> None:\n '''\n Given some error text it will log the text if self.log_errors is True\n\n :param text: Error text to log\n '''\n if self.log_errors:\n with self._log_fp.open('a+') as log_file:\n log_file.write(f'{text}\\n')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_conll(self, texts: List[str], retry_count: int = 0) -> List[str]:\n '''\n Processes the texts using TweeboParse and returns them in CoNLL format.\n\n :param texts: The List of Strings to be processed by TweeboParse.\n :param retry_count: The number of times it has retried for. Default\n 0 does not require setting, main purpose is for\n recursion.\n :return: A list of CoNLL formated strings.\n :raises ServerError: Caused when the server is not running.\n :raises :py:class:`requests.exceptions.HTTPError`: Caused when the\n input texts is not formated correctly e.g. When you give it a\n String not a list of Strings.\n :raises :py:class:`json.JSONDecodeError`: Caused if after self.retries\n attempts to parse the data it cannot decode the data.\n\n :Example:\n\n '''\n post_data = {'texts': texts, 'output_type': 'conll'}\n try:\n response = requests.post(f'http://{self.hostname}:{self.port}',\n json=post_data,\n headers={'Connection': 'close'})\n response.raise_for_status()\n except (requests.exceptions.ConnectionError,\n requests.exceptions.Timeout) as server_error:\n raise ServerError(server_error, self.hostname, self.port)\n except requests.exceptions.HTTPError as http_error:\n raise http_error\n else:\n try:\n return response.json()\n except json.JSONDecodeError as json_exception:\n if retry_count == self.retries:\n self.log_error(response.text)\n raise Exception('Json Decoding error cannot parse this '\n f':\\n{response.text}')\n return self.parse_conll(texts, retry_count + 1)", "response": "Processes the texts using TweeboParse and returns them in CoNLL format."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset entity data to be parsed as datetimes", "response": "def set_data(self, data={}, datetime_fields=[]):\n \"\"\" Set entity data\n\n Args:\n data (dict): Entity data\n datetime_fields (array): Fields that should be parsed as datetimes\n \"\"\"\n if datetime_fields:\n for field in datetime_fields:\n if field in data:\n data[field] = self._parse_datetime(data[field])\n\n super(CampfireEntity, self).set_data(data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse a datetime string from YYYYMMDD HHMMSS + HHMM", "response": "def _parse_datetime(self, value):\n \"\"\" Parses a datetime string from \"YYYY/MM/DD HH:MM:SS +HHMM\" format\n\n Args:\n value (str): String\n\n Returns:\n datetime. Datetime\n \"\"\"\n offset = 0\n pattern = r\"\\s+([+-]{1}\\d+)\\Z\"\n matches = re.search(pattern, value)\n if matches:\n value = re.sub(pattern, '', value)\n offset = datetime.timedelta(hours=int(matches.group(1))/100)\n return datetime.datetime.strptime(value, \"%Y/%m/%d %H:%M:%S\") - offset"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, time):\n \n # .... I feel this stuff could be done a lot better.\n total_acceleration = Vector.null()\n max_jerk = self.max_acceleration\n\n for behavior in self.behaviors:\n acceleration, importance = behavior.update()\n weighted_acceleration = acceleration * importance\n\n \"\"\" \n if max_jerk >= weighted_acceleration.magnitude:\n max_jerk -= weighted_acceleration.magnitude\n total_acceleration += weighted_acceleration\n elif max_jerk > 0 and max_jerk < weighted_acceleration.magnitude:\n total_acceleration += weighted_acceleration.normal * max_jerk\n break\n else:\n break \"\"\"\n total_acceleration += weighted_acceleration\n\n self.acceleration = total_acceleration\n\n # Update position and velocity.\n Sprite.update(self, time)\n\n # Update facing direction.\n if self.velocity.magnitude > 0.0:\n self.facing = self.velocity.normal", "response": "Update the acceleration and position of the current object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update (self):\n delta_velocity = Vector.null()\n target_position = self.target.get_position()\n sprite_position = self.sprite.get_position()\n\n desired_direction = target_position - sprite_position\n\n if 0.0 == self.los or desired_direction.magnitude <= self.los:\n try:\n desired_normal = desired_direction.normal\n except NullVectorError:\n desired_normal = Vector.null()\n desired_velocity = desired_normal * self.sprite.get_max_speed()\n delta_velocity = desired_velocity - self.sprite.get_velocity()\n\n self.last_delta_velocity = delta_velocity\n return delta_velocity, self.power", "response": "Update the current time and velocity."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef on_enter_stage(self):\n with self.world._unlock_temporarily():\n self.forum.connect_everyone(self.world, self.actors)\n\n # 1. Setup the forum.\n\n self.forum.on_start_game()\n\n # 2. Setup the world.\n\n with self.world._unlock_temporarily():\n self.world.on_start_game()\n\n # 3. Setup the actors. Because this is done after the forum and the \n # world have been setup, this signals to the actors that they can \n # send messages and query the game world as usual.\n\n num_players = len(self.actors) - 1\n\n for actor in self.actors:\n actor.on_setup_gui(self.gui)\n\n for actor in self.actors:\n actor.on_start_game(num_players)", "response": "Setup the actors the forum and the world to begin the game."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef on_update_stage(self, dt):\n\n for actor in self.actors:\n actor.on_update_game(dt)\n\n self.forum.on_update_game()\n\n with self.world._unlock_temporarily():\n self.world.on_update_game(dt)\n\n if self.world.has_game_ended():\n self.exit_stage()", "response": "Sequentially update the actors the world and the messaging system."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef on_exit_stage(self):\n\n # 1. Let the forum react to the end of the game. Local forums don't \n # react to this, but remote forums take the opportunity to stop \n # trying to extract tokens from messages.\n\n self.forum.on_finish_game()\n\n # 2. Let the actors react to the end of the game.\n\n for actor in self.actors:\n actor.on_finish_game()\n\n # 3. Let the world react to the end of the game.\n\n with self.world._unlock_temporarily():\n self.world.on_finish_game()", "response": "This method is called when the game is finished."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of all the tokens that are referenced by this message.", "response": "def tokens_referenced(self):\n \"\"\"\n Return a list of all the tokens that are referenced (i.e. contained in) \n this message. Tokens that haven't been assigned an id yet are searched \n recursively for tokens. So this method may return fewer results after \n the message is sent. This information is used by the game engine to \n catch mistakes like forgetting to add a token to the world or keeping a \n stale reference to a token after its been removed.\n \"\"\"\n tokens = set()\n\n # Use the pickle machinery to find all the tokens contained at any \n # level of this message. When an object is being pickled, the Pickler \n # calls its persistent_id() method for each object it encounters. We \n # hijack this method to add every Token we encounter to a list.\n\n # This definitely feels like a hacky abuse of the pickle machinery, but \n # that notwithstanding this should be quite robust and quite fast.\n\n def persistent_id(obj):\n from .tokens import Token\n\n if isinstance(obj, Token):\n tokens.add(obj)\n\n # Recursively descend into tokens that haven't been assigned an \n # id yet, but not into tokens that have.\n\n return obj.id\n\n from pickle import Pickler\n from io import BytesIO\n\n # Use BytesIO to basically ignore the serialized data stream, since we \n # only care about visiting all the objects that would be pickled.\n\n pickler = Pickler(BytesIO())\n pickler.persistent_id = persistent_id\n pickler.dump(self)\n\n return tokens"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wrap_handler(self, handler, context_switcher):\n context_switcher.add_context_in(lambda: LOGGER.addHandler(self.handler))\n context_switcher.add_context_out(lambda: LOGGER.removeHandler(self.handler))", "response": "Enable or disable handler."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nusing when application is starting.", "response": "def init_async(self, loop=None):\n \"\"\"Use when application is starting.\"\"\"\n self._loop = loop or asyncio.get_event_loop()\n self._async_lock = asyncio.Lock(loop=loop)\n\n # FIX: SQLITE in memory database\n if not self.database == ':memory:':\n self._state = ConnectionLocal()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def async_connect(self):\n if self._async_lock is None:\n raise Exception('Error, database not properly initialized before async connection')\n\n async with self._async_lock:\n self.connect(True)\n\n return self._state.conn", "response": "Catch a connection asyncrounosly."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _close(self, conn):\n super(PooledAIODatabase, self)._close(conn)\n for waiter in self._waiters:\n if not waiter.done():\n logger.debug('Release a waiter')\n waiter.set_result(True)\n break", "response": "Release all waiters and set the result to True."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef receive_id_from_server(self):\n for message in self.pipe.receive():\n if isinstance(message, IdFactory):\n self.actor_id_factory = message\n return True\n return False", "response": "This method is called by the game when a new id is received from the server."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes a sync request.", "response": "def execute_sync(self, message):\n \"\"\"\n Respond when the server indicates that the client is out of sync.\n\n The server can request a sync when this client sends a message that \n fails the check() on the server. If the reason for the failure isn't \n very serious, then the server can decide to send it as usual in the \n interest of a smooth gameplay experience. When this happens, the \n server sends out an extra response providing the clients with the\n information they need to resync themselves.\n \"\"\"\n info(\"synchronizing message: {message}\")\n\n # Synchronize the world.\n\n with self.world._unlock_temporarily():\n message._sync(self.world)\n self.world._react_to_sync_response(message)\n\n # Synchronize the tokens.\n\n for actor in self.actors:\n actor._react_to_sync_response(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute the undo action for the given message.", "response": "def execute_undo(self, message):\n \"\"\"\n Manage the response when the server rejects a message.\n\n An undo is when required this client sends a message that the server \n refuses to pass on to the other clients playing the game. When this \n happens, the client must undo the changes that the message made to the \n world before being sent or crash. Note that unlike sync requests, undo \n requests are only reported to the client that sent the offending \n message.\n \"\"\"\n info(\"undoing message: {message}\")\n\n # Roll back changes that the original message made to the world.\n\n with self.world._unlock_temporarily():\n message._undo(self.world)\n self.world._react_to_undo_response(message)\n\n # Give the actors a chance to react to the error. For example, a \n # GUI actor might inform the user that there are connectivity \n # issues and that their last action was countermanded.\n\n for actor in self.actors:\n actor._react_to_undo_response(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _relay_message(self, message):\n info(\"relaying message: {message}\")\n\n if not message.was_sent_by(self._id_factory):\n self.pipe.send(message)\n self.pipe.deliver()", "response": "Relay a message from the server to the client represented \n by this actor."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new DataItem.", "response": "def generate(request):\n \"\"\" Create a new DataItem. \"\"\"\n models.DataItem.create(\n content=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(20))\n )\n return muffin.HTTPFound('/')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the given object is an active token and raises an ApiUsageError if it is not.", "response": "def require_active_token(object):\n \"\"\"\n Raise an ApiUsageError if the given object is not a token that is currently \n participating in the game. To be participating in the game, the given \n token must have an id number and be associated with the world.\n \"\"\"\n require_token(object)\n token = object\n\n if not token.has_id:\n raise ApiUsageError(\"\"\"\\\n token {token} should have an id, but doesn't.\n\n This error usually means that a token was added to the world \n without being assigned an id number. To correct this, make \n sure that you're using a message (i.e. CreateToken) to create \n all of your tokens.\"\"\")\n\n if not token.has_world:\n raise ApiUsageError(\"\"\"\\\n token {token} (id={token.id}) not in world.\n\n You can get this error if you try to remove the same token from \n the world twice. This might happen is you don't get rid of \n every reference to a token after it's removed the first time, \n then later on you try to remove the stale reference.\"\"\")"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_safety_checks(meta, members):\n for member_name, member_value in members.items():\n members[member_name] = meta.add_safety_check(\n member_name, member_value)", "response": "Iterate through each member of the class being created and add a safety check to every member of the class that isn t marked as read - only."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_safety_check(member_name, member_value):\n import functools\n from types import FunctionType\n\n # Bail if the given member is read-only, private, or not a method.\n\n is_method = isinstance(member_value, FunctionType)\n is_read_only = hasattr(member_value, '_kxg_read_only')\n is_private = member_name.startswith('_')\n\n if not is_method or is_read_only or is_private:\n return member_value\n\n def safety_checked_method(self, *args, **kwargs):\n \"\"\"\n Make sure that the token the world is locked before a non-read-only \n method is called.\n \"\"\"\n # Because these checks are pretty magical, I want to be really \n # careful to avoid raising any exceptions other than the check \n # itself (which comes with a very clear error message). Here, that \n # means using getattr() to make sure the world attribute actually \n # exists. For example, there's nothing wrong with the following \n # code, but it does call a safety-checked method before the world \n # attribute is defined:\n #\n # class MyToken(kxg.Token):\n # def __init__(self):\n # self.init_helper()\n # super().__init__()\n\n world = getattr(self, 'world', None)\n if world and world.is_locked():\n nonlocal member_name\n raise ApiUsageError(\"\"\"\\\n attempted unsafe invocation of \n {self.__class__.__name__}.{member_name}().\n\n This error brings attention to situations that might \n cause synchronization issues in multiplayer games. The \n {member_name}() method is not marked as read-only, but \n it was invoked from outside the context of a message. \n This means that if {member_name}() makes any changes to \n the world, those changes will not be propagated. If \n {member_name}() is actually read-only, label it with \n the @kxg.read_only decorator.\"\"\")\n\n # After making that check, call the method as usual.\n\n return member_value(self, *args, **kwargs)\n\n # Preserve any \"forum observer\" decorations that have been placed on \n # the method and restore the method's original name and module strings, \n # to make inspection and debugging a little easier.\n\n functools.update_wrapper(\n safety_checked_method, member_value,\n assigned=functools.WRAPPER_ASSIGNMENTS + (\n '_kxg_subscribe_to_message',\n '_kxg_subscribe_to_sync_response',\n '_kxg_subscribe_to_undo_response',\n )\n )\n return safety_checked_method", "response": "Add a safety check to the given member."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef watch_method(self, method_name, callback):\n\n # Make sure a token method with the given name exists, and complain if \n # nothing is found.\n\n try:\n method = getattr(self, method_name)\n except AttributeError:\n raise ApiUsageError(\"\"\"\\\n {self.__class__.__name__} has no such method \n {method_name}() to watch.\n\n This error usually means that you used the @watch_token \n decorator on a method of a token extension class that \n didn't match the name of any method in the corresponding \n token class. Check for typos.\"\"\")\n\n # Wrap the method in a WatchedMethod object, if that hasn't already \n # been done. This object manages a list of callback method and takes \n # responsibility for calling them after the method itself has been \n # called.\n\n if not isinstance(method, Token.WatchedMethod):\n setattr(self, method_name, Token.WatchedMethod(method))\n method = getattr(self, method_name)\n\n # Add the given callback to the watched method.\n\n method.add_watcher(callback)", "response": "Register a given callback to be called whenever the method with the given name is called."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _remove_from_world(self):\n self.on_remove_from_world()\n self._extensions = {}\n self._disable_forum_observation()\n self._world = None\n self._id = None", "response": "Remove the token from the world."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _unlock_temporarily(self):\n if not self._is_locked:\n yield\n else:\n try:\n self._is_locked = False\n yield\n finally:\n self._is_locked = True", "response": "Unlocks the world for the duration of a with - block."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef scan(xml):\n\n if xml.tag is et.Comment:\n yield {'type': COMMENT, 'text': xml.text}\n return\n\n if xml.tag is et.PI:\n if xml.text:\n yield {'type': PI, 'target': xml.target, 'text': xml.text}\n else:\n yield {'type': PI, 'target': xml.target}\n return\n\n obj = _elt2obj(xml)\n obj['type'] = ENTER\n yield obj\n\n assert type(xml.tag) is str, xml\n if xml.text:\n yield {'type': TEXT, 'text': xml.text}\n\n for c in xml:\n for x in scan(c): yield x\n if c.tail:\n yield {'type': TEXT, 'text': c.tail}\n\n yield {'type': EXIT}", "response": "Converts XML tree to event generator"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef unscan(events, nsmap=None):\n\n root = None\n last_closed_elt = None\n stack = []\n for obj in events:\n\n if obj['type'] == ENTER:\n elt = _obj2elt(obj, nsmap=nsmap)\n if stack:\n stack[-1].append(elt)\n elif root is not None:\n raise RuntimeError('Event stream tried to create second XML tree')\n else:\n root = elt\n stack.append(elt)\n last_closed_elt = None\n\n elif obj['type'] == EXIT:\n last_closed_elt = stack.pop()\n\n elif obj['type'] == COMMENT:\n elt = et.Comment(obj['text'])\n stack[-1].append(elt)\n\n elif obj['type'] == PI:\n elt = et.PI(obj['target'])\n if obj.get('text'):\n elt.text = obj['text']\n stack[-1].append(elt)\n\n elif obj['type'] == TEXT:\n text = obj['text']\n if text:\n if last_closed_elt is None:\n stack[-1].text = (stack[-1].text or '') + text\n else:\n last_closed_elt.tail = (last_closed_elt.tail or '') + text\n else:\n assert False, obj\n\n if root is None:\n raise RuntimeError('Empty XML event stream')\n\n return root", "response": "Converts events stream into lXML tree"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse the file content into events stream", "response": "def parse(filename):\n \"\"\"Parses file content into events stream\"\"\"\n for event, elt in et.iterparse(filename, events= ('start', 'end', 'comment', 'pi'), huge_tree=True):\n if event == 'start':\n obj = _elt2obj(elt)\n obj['type'] = ENTER\n yield obj\n if elt.text:\n yield {'type': TEXT, 'text': elt.text}\n elif event == 'end':\n yield {'type': EXIT}\n if elt.tail:\n yield {'type': TEXT, 'text': elt.tail}\n elt.clear()\n elif event == 'comment':\n yield {'type': COMMENT, 'text': elt.text}\n elif event == 'pi':\n yield {'type': PI, 'text': elt.text}\n else:\n assert False, (event, elt)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nyield all events that are sub - tree", "response": "def subtree(events):\n \"\"\"selects sub-tree events\"\"\"\n stack = 0\n for obj in events:\n if obj['type'] == ENTER:\n stack += 1\n elif obj['type'] == EXIT:\n if stack == 0:\n break\n stack -= 1\n yield obj"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef merge_text(events):\n text = []\n for obj in events:\n if obj['type'] == TEXT:\n text.append(obj['text'])\n else:\n if text:\n yield {'type': TEXT, 'text': ''.join(text)}\n text.clear()\n yield obj\n if text:\n yield {'type': TEXT, 'text': ''.join(text)}", "response": "merges each run of successive text events into one text event"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef with_peer(events):\n\n stack = []\n for obj in events:\n if obj['type'] == ENTER:\n stack.append(obj)\n yield obj, None\n elif obj['type'] == EXIT:\n yield obj, stack.pop()\n else:\n yield obj, None", "response": "yields all events that are ENTER or EXIT"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_date(datetime_date):\n return BusinessDate.from_ymd(datetime_date.year, datetime_date.month, datetime_date.day)", "response": "construct BusinessDate instance from datetime. date instance"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_string(date_str):\n if date_str.count('-'):\n str_format = '%Y-%m-%d'\n elif date_str.count('.'):\n str_format = '%d.%m.%Y'\n elif date_str.count('/'):\n str_format = '%m/%d/%Y'\n elif len(date_str) == 8:\n str_format = '%Y%m%d'\n elif len(date_str) == 4:\n year = ord(date_str[0]) * 256 + ord(date_str[1])\n month = ord(date_str[2])\n day = ord(date_str[3])\n return BusinessDate.from_ymd(year, month, day)\n else:\n msg = \"the date string \" + date_str + \" has not the right format\"\n raise ValueError(msg)\n\n d = datetime.strptime(date_str, str_format)\n\n return BusinessDate.from_ymd(d.year, d.month, d.day)", "response": "constructs a new date object from a string"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconstructing datetime. date instance represented calendar date of BusinessDate instance", "response": "def to_date(self):\n \"\"\"\n construct datetime.date instance represented calendar date of BusinessDate instance\n\n :return datetime.date:\n \"\"\"\n y, m, d = self.to_ymd()\n return date(y, m, d)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking whether the provided date is a business date.", "response": "def is_businessdate(in_date):\n \"\"\"\n checks whether the provided date is a date\n :param BusinessDate, int or float in_date:\n :return bool:\n \"\"\"\n # Note: if the data range has been created from pace_xl, then all the dates are bank dates\n # and here it remains to check the validity.\n # !!! However, if the data has been read from json string via json.load() function\n # it does not recognize that this numbers are bankdates, just considers them as integers\n # therefore, additional check is useful here, first to convert the date if it is integer to BusinessDate,\n # then check the validity.\n # (as the parameter to this method should always be a BusinessDate)\n if not isinstance(in_date, BaseDate):\n try: # to be removed\n in_date = BusinessDate(in_date)\n except:\n return False\n y, m, d, = in_date.to_ymd()\n return is_valid_ymd(y, m, d)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_business_day(self, holiday_obj=None):\n y, m, d = BusinessDate.to_ymd(self)\n if weekday(y, m, d) > FRIDAY:\n return False\n holiday_list = holiday_obj if holiday_obj is not None else DEFAULT_HOLIDAYS\n if self in holiday_list:\n return False\n elif date(y, m, d) in holiday_list:\n return False\n return True", "response": "method to check if a date falls on business day"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a period to the current object", "response": "def add_period(self, p, holiday_obj=None):\n \"\"\"\n addition of a period object\n\n :param BusinessDate d:\n :param p:\n :type p: BusinessPeriod or str\n :param list holiday_obj:\n :return bankdate:\n \"\"\"\n\n if isinstance(p, (list, tuple)):\n return [BusinessDate.add_period(self, pd) for pd in p]\n elif isinstance(p, str):\n period = BusinessPeriod(p)\n else:\n period = p\n\n res = self\n res = BusinessDate.add_months(res, period.months)\n res = BusinessDate.add_years(res, period.years)\n res = BusinessDate.add_days(res, period.days)\n\n if period.businessdays:\n if holiday_obj:\n res = BusinessDate.add_business_days(res, period.businessdays, holiday_obj)\n else:\n res = BusinessDate.add_business_days(res, period.businessdays, period.holiday)\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_business_days(self, days_int, holiday_obj=None):\n\n res = self\n if days_int >= 0:\n count = 0\n while count < days_int:\n res = BusinessDate.add_days(res, 1)\n if BusinessDate.is_business_day(res, holiday_obj):\n count += 1\n else:\n count = 0\n while count > days_int:\n res = BusinessDate.add_days(res, -1)\n if BusinessDate.is_business_day(res, holiday_obj):\n count -= 1\n\n return res", "response": "private method for the addition of business days used in the addition of a BusinessPeriod only\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef diff(self, end_date):\n\n if end_date < self:\n y, m, d = BusinessDate.diff(end_date, self)\n return -y, -m, -d\n y = end_date.year - self.year\n m = end_date.month - self.month\n\n while m < 0:\n y -= 1\n m += 12\n while m > 12:\n y += 1\n m -= 12\n\n s = BusinessDate.add_years(BusinessDate.add_months(self, m), y)\n d = BusinessDate.diff_in_days(s, end_date)\n\n if d < 0:\n m -= 1\n if m < 0:\n y -= 1\n m += 12\n s = BusinessDate.add_years(BusinessDate.add_months(self, m), y)\n\n d = BusinessDate.diff_in_days(s, end_date)\n\n return -int(y), -int(m), -int(d)", "response": "returns a tuple of years months days"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_30_360(self, end):\n start_day = min(self.day, 30)\n end_day = 30 if (start_day == 30 and end.day == 31) else end.day\n return (360 * (end.year - self.year) + 30 * (end.month - self.month) + (end_day - start_day)) / 360.0", "response": "Return the 30 - degree count of the current object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_act_act(self, end):\n # split end-self in year portions\n\n # if the period does not lie within a year split the days in the period as following:\n # restdays of start year / years in between / days in the end year\n # REMARK: following the affore mentioned ISDA Definition the first day of the period is included whereas the\n # last day will be excluded\n # What remains to check now is only whether the start and end year are leap or non-leap years. The quotients\n # can be easily calculated and for the years in between they are always one (365/365 = 1; 366/366 = 1)\n\n if end.year - self.year == 0:\n if BusinessDate.is_leap_year(self.year):\n return BusinessDate.diff_in_days(self, end) / 366.0 # leap year: 366 days\n else:\n # return BusinessDate.diff_in_days(self, end) / 366.0\n return BusinessDate.diff_in_days(self, end) / 365.0 # non-leap year: 365 days\n else:\n rest_year1 = BusinessDate.diff_in_days(self, BusinessDate(\n date(self.year, 12, 31))) + 1 # since the first day counts\n\n rest_year2 = abs(BusinessDate.diff_in_days(end, BusinessDate(\n date(end.year, 1, 1)))) # here the last day is automatically not counted\n\n years_in_between = end.year - self.year - 1\n\n return years_in_between + rest_year1 / (366.0 if is_leap_year(self.year) else 365.0) + rest_year2 / (\n 366.0 if is_leap_year(end.year) else 365.0)", "response": "returns the number of days in the act day of the current object"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the 30E - 360 of the time between this date and the given date.", "response": "def get_30E_360(self, end):\n \"\"\"\n implements the 30E/360 Day Count Convention (4.16(g) 2006 ISDA Definitons)\n \"\"\"\n\n y1, m1, d1 = self.to_ymd()\n # adjust to date immediately following the the last day\n y2, m2, d2 = end.add_days(0).to_ymd()\n\n d1 = min(d1, 30)\n d2 = min(d2, 30)\n\n return (360 * (y2 - y1) + 30 * (m2 - m1) + (d2 - d1)) / 360.0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nimplement the 30E/360 (ISDA) Day Count Convention (4.16(h) 2006 ISDA Definitions) :param end: :return:", "response": "def get_30E_360_ISDA(self, end):\n \"\"\"\n implements the 30E/360 (ISDA) Day Count Convention (4.16(h) 2006 ISDA Definitions)\n :param end:\n :return:\n \"\"\"\n y1, m1, d1 = self.to_ymd()\n # ajdust to date immediately following the last day\n y2, m2, d2 = end.add_days(0).to_ymd()\n\n if (m1 == 2 and d1 >= 28) or d1 == 31:\n d1 = 30\n if (m2 == 2 and d2 >= 28) or d2 == 31:\n d2 = 30\n\n return (360 * (y2 - y1) + 30 * (m2 - m1) + (d2 - d1)) / 360.0"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef adjust_previous(self, holidays_obj=None):\n while not BusinessDate.is_business_day(self, holidays_obj):\n self = BusinessDate.add_days(self, -1)\n return self", "response": "Adjusts the calendar to the previous date."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef adjust_follow(self, holidays_obj=None):\n while not BusinessDate.is_business_day(self, holidays_obj):\n self = BusinessDate.add_days(self, 1)\n return self", "response": "Adjusts the follow date to follow the current date."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef adjust_mod_follow(self, holidays_obj=None):\n month = self.month\n new = BusinessDate.adjust_follow(self, holidays_obj)\n if month != new.month:\n new = BusinessDate.adjust_previous(self, holidays_obj)\n self = new\n return self", "response": "Adjusts the calendar to be modified following."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadjusts the calendar to be the previous date.", "response": "def adjust_mod_previous(self, holidays_obj=None):\n \"\"\"\n ajusts to Business Day Convention \"Modified Preceding\" (not in 2006 ISDA Definitons).\n \"\"\"\n month = self.month\n new = BusinessDate.adjust_previous(self, holidays_obj)\n if month != new.month:\n new = BusinessDate.adjust_follow(self, holidays_obj)\n self = new\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_businessperiod(cls, in_period):\n try: # to be removed\n if str(in_period).upper() == '0D':\n return True\n else:\n p = BusinessPeriod(str(in_period))\n return not (p.days == 0 and p.months == 0 and p.years == 0 and p.businessdays == 0)\n except:\n return False", "response": "Checks if the object in_period is a businessperiod."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef quoted(parser=any_token):\n quote_char = quote()\n value, _ = many_until(parser, partial(one_of, quote_char))\n return build_string(value)", "response": "Parses as much as possible until it encounters a matching closing quote."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the number of days in a given month", "response": "def days_in_month(year, month):\n \"\"\"\n returns number of days for the given year and month\n\n :param int year: calendar year\n :param int month: calendar month\n :return int:\n \"\"\"\n\n eom = _days_per_month[month - 1]\n if is_leap_year(year) and month == 2:\n eom += 1\n\n return eom"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning True if the year month and day are valid YMD.", "response": "def is_valid_ymd(year, month, day):\n \"\"\"\n return True if (year,month, day) can be represented in Excel-notation\n (number of days since 30.12.1899) for calendar days, otherwise False\n\n :param int year: calendar year\n :param int month: calendar month\n :param int day: calendar day\n :return bool:\n \"\"\"\n\n return 1 <= month <= 12 and 1 <= day <= days_in_month(year, month) and year >= 1899"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_excel_to_ymd(excel_int):\n\n int_date = int(floor(excel_int))\n int_date -= 1 if excel_int > 60 else 0\n # jan dingerkus: There are two errors in excels own date <> int conversion.\n # The first is that there exists the 00.01.1900 and the second that there never happend to be a 29.2.1900 since it\n # was no leap year. So there is the int 60 <> 29.2.1900 which has to be jumped over.\n\n year = (int_date - 1) // 365\n rest_days = int_date - 365 * year - (year + 3) // 4 + (year + 99) // 100 - (year + 299) // 400\n year += 1900\n\n while rest_days <= 0:\n year -= 1\n rest_days += days_in_year(year)\n\n month = 1\n if is_leap_year(year) and rest_days == 60:\n month = 2\n day = 29\n else:\n if is_leap_year(year) and rest_days > 60:\n rest_days -= 1\n\n while rest_days > _cum_month_days[month]:\n month += 1\n\n day = rest_days - _cum_month_days[month - 1]\n return year, month, day", "response": "Convert an int into a tuple of year month and day."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_ymd_to_excel(year, month, day):\n if not is_valid_ymd(year, month, day):\n raise ValueError(\"Invalid date {0}.{1}.{2}\".format(year, month, day))\n\n days = _cum_month_days[month - 1] + day\n days += 1 if (is_leap_year(year) and month > 2) else 0\n\n years_distance = year - 1900\n days += years_distance * 365 + \\\n (years_distance + 3) // 4 - (years_distance + 99) // 100 + (years_distance + 299) // 400\n\n # count days since 30.12.1899 (excluding 30.12.1899) (workaround for excel bug)\n days += 1 if (year, month, day) > (1900, 2, 28) else 0\n return days", "response": "Converts a date in YYYYMMDD format into Microsoft Excel representation"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a number of years to a date", "response": "def add_years(d, years_int):\n \"\"\"\n adds number of years to a date\n :param BaseDateFloat d: date to add years to\n :param int years_int: number of years to add\n :return BaseDate: resulting date\n \"\"\"\n\n y, m, d = BaseDate.to_ymd(d)\n if not is_leap_year(years_int) and m == 2:\n d = min(28, d)\n return BaseDateFloat.from_ymd(y + years_int, m, d)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_days(d, days_int):\n n = date(d.year, d.month, d.day) + timedelta(days_int)\n return BaseDateDatetimeDate(n.year, n.month, n.day)", "response": "add_days adds a number of days to a date"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates difference between start and end dates in days", "response": "def diff_in_days(start, end):\n \"\"\"\n calculate difference between given dates in days\n\n :param BaseDateDatetimeDate start: state date\n :param BaseDateDatetimeDate end: end date\n :return float: difference between end date and start date in days\n \"\"\"\n diff = date(end.year, end.month, end.day) - date(start.year, start.month, start.day)\n return float(diff.days)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_days(date_obj, days_int):\n n = from_ymd_to_excel(*date_obj.date) + days_int\n\n return BaseDateTuple(*from_excel_to_ymd(n))", "response": "add_days adds a number of days to a date tuple"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_years(date_obj, years_int):\n y, m, d = BaseDateTuple.to_ymd(date_obj)\n y += years_int\n if not is_leap_year(y) and m == 2:\n d = min(28, d)\n return BaseDateTuple.from_ymd(y, m, d)", "response": "add_years adds a number of years to the base date tuple d"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate difference between start and end dates in days", "response": "def diff_in_days(start, end):\n \"\"\"\n calculate difference between given dates in days\n\n :param BaseDateTuple start: state date\n :param BaseDateTuple end: end date\n :return float: difference between end date and start date in days\n \"\"\"\n\n diff = from_ymd_to_excel(*end.date)-from_ymd_to_excel(*start.date)\n return float(diff)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nregistering connection s middleware and prepare self database.", "response": "def startup(self, app):\n \"\"\"Register connection's middleware and prepare self database.\"\"\"\n self.database.init_async(app.loop)\n if not self.cfg.connection_manual:\n app.middlewares.insert(0, self._middleware)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nregistering a model in self. models.", "response": "def register(self, model):\n \"\"\"Register a model in self.\"\"\"\n self.models[model._meta.table_name] = model\n model._meta.database = self.database\n return model"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def manage(self):\n cm = _ContextManager(self.database)\n if isinstance(self.database.obj, AIODatabase):\n cm.connection = await self.database.async_connect()\n\n else:\n cm.connection = self.database.connect()\n\n return cm", "response": "Manage a database connection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting your migrations here. > Model = migrator.orm['name'] > migrator.sql(sql) > migrator.create_table(Model) > migrator.drop_table(Model, cascade=True) > migrator.add_columns(Model, **fields) > migrator.change_columns(Model, **fields) > migrator.drop_columns(Model, *field_names, cascade=True) > migrator.rename_column(Model, old_field_name, new_field_name) > migrator.rename_table(Model, new_table_name) > migrator.add_index(Model, *col_names, unique=False) > migrator.drop_index(Model, index_name) > migrator.add_not_null(Model, field_name) > migrator.drop_not_null(Model, field_name) > migrator.add_default(Model, field_name, default)", "response": "def migrate(migrator, database, **kwargs):\n \"\"\" Write your migrations here.\n\n > Model = migrator.orm['name']\n\n > migrator.sql(sql)\n > migrator.create_table(Model)\n > migrator.drop_table(Model, cascade=True)\n > migrator.add_columns(Model, **fields)\n > migrator.change_columns(Model, **fields)\n > migrator.drop_columns(Model, *field_names, cascade=True)\n > migrator.rename_column(Model, old_field_name, new_field_name)\n > migrator.rename_table(Model, new_table_name)\n > migrator.add_index(Model, *col_names, unique=False)\n > migrator.drop_index(Model, index_name)\n > migrator.add_not_null(Model, field_name)\n > migrator.drop_not_null(Model, field_name)\n > migrator.add_default(Model, field_name, default)\n\n \"\"\"\n @migrator.create_table\n class DataItem(pw.Model):\n created = pw.DateTimeField(default=dt.datetime.utcnow)\n content = pw.CharField()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef chain(*args):\n def chain_block(*args, **kwargs):\n v = args[0](*args, **kwargs)\n for p in args[1:]:\n v = p(v)\n return v\n return chain_block", "response": "Runs a series of parsers in sequence passing the result of each parser to the next."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef one_of(these):\n ch = peek()\n try:\n if (ch is EndOfFile) or (ch not in these):\n fail(list(these))\n except TypeError:\n if ch != these:\n fail([these])\n next()\n return ch", "response": "Returns the current token if is found in the collection provided. Fails otherwise."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef not_one_of(these):\n ch = peek()\n desc = \"not_one_of\" + repr(these)\n try:\n if (ch is EndOfFile) or (ch in these):\n fail([desc])\n except TypeError:\n if ch != these:\n fail([desc])\n next()\n return ch", "response": "Returns the current token if it is not found in the collection provided."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the current token if it satisfies the guard function provided. Fails if the guard function provided is not a function.", "response": "def satisfies(guard):\n \"\"\"Returns the current token if it satisfies the guard function provided.\n \n Fails otherwise.\n This is the a generalisation of one_of.\n \"\"\"\n i = peek()\n if (i is EndOfFile) or (not guard(i)):\n fail([\"\"])\n next()\n return i"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsucceeds if the given parser cannot consume input", "response": "def not_followed_by(parser):\n \"\"\"Succeeds if the given parser cannot consume input\"\"\"\n @tri\n def not_followed_by_block():\n failed = object()\n result = optional(tri(parser), failed)\n if result != failed:\n fail([\"not \" + _fun_to_str(parser)])\n choice(not_followed_by_block)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\napplying the parser to input zero or more times.", "response": "def many(parser):\n \"\"\"Applies the parser to input zero or more times.\n \n Returns a list of parser results.\n \"\"\"\n results = []\n terminate = object()\n while local_ps.value:\n result = optional(parser, terminate)\n if result == terminate:\n break\n results.append(result)\n return results"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconsuming as many of these as it can until it is encountered.", "response": "def many_until(these, term):\n \"\"\"Consumes as many of these as it can until it term is encountered.\n \n Returns a tuple of the list of these results and the term result \n \"\"\"\n results = []\n while True:\n stop, result = choice(_tag(True, term),\n _tag(False, these))\n if stop:\n return results, result\n else:\n results.append(result)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef many_until1(these, term):\n first = [these()]\n these_results, term_result = many_until(these, term)\n return (first + these_results, term_result)", "response": "Like many_until but must consume at least one of these."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sep1(parser, separator):\n first = [parser()]\n def inner():\n separator()\n return parser()\n return first + many(tri(inner))", "response": "Like sep but must consume at least one of parser.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef string(string):\n found = []\n for c in string:\n found.append(one_of(c))\n return found", "response": "Iterates over a string and returns a list of items that match the input."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef seq(*sequence):\n results = {}\n for p in sequence:\n if callable(p): \n p()\n continue\n k, v = p\n results[k] = v()\n return results", "response": "Runs a series of parsers in sequence optionally storing results in a dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _fill(self, size):\n try:\n for i in range(size):\n self.buffer.append(self.source.next())\n except StopIteration:\n self.buffer.append((EndOfFile, EndOfFile))\n self.len = len(self.buffer)", "response": "fills the internal buffer with the specified size"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef next(self):\n self.index += 1\n t = self.peek()\n if not self.depth:\n self._cut()\n return t", "response": "Advances to and returns the next token or returns EndOfFile"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef current(self):\n if self.index >= self.len:\n self._fill((self.index - self.len) + 1)\n return self.index < self.len and self.buffer[self.index] or (EndOfFile, EndOfFile)", "response": "Returns the current token or position"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrunning a game being developed with the kxg game engine. Usage: {exe_name} sandbox [] [-v...] {exe_name} client [--host HOST] [--port PORT] [-v...] {exe_name} server [] [--host HOST] [--port PORT] [-v...] {exe_name} debug [] [--host HOST] [--port PORT] [-v...] {exe_name} --help Commands: sandbox Play a single-player game with the specified number of AIs. None of the multiplayer machinery will be used. client Launch a client that will try to connect to a server on the given host and port. Once it connects and the game starts, the client will allow you to play the game against any other connected clients. server Launch a server that will manage a game between the given number of human and AI players. The human players must connect using this command's client mode. debug Debug a multiplayer game locally. This command launches a server and the given number of clients all in different processes, and configures the logging system such that the output from each process can be easily distinguished. Arguments: The number of human players that will be playing the game. Only needed by commands that will launch some sort of multiplayer server. The number of AI players that will be playing the game. Only needed by commands that will launch single-player games or multiplayer servers. Options: -x --host HOST [default: {default_host}] The address of the machine running the server. Must be accessible from the machines running the clients. -p --port PORT [default: {default_port}] The port that the server should listen on. Don't specify a value less than 1024 unless the server is running with root permissions. -v --verbose Have the game engine log more information about what it's doing. You can specify this option several times to get more and more information. This command is provided so that you can start writing your game with the least possible amount of boilerplate code. However, the clients and servers provided by this command are not capable of running a production game. Once you have written your game and want to give it a polished set of menus and options, you'll have to write new Stage subclasses encapsulating that logic and you'll have to call those stages yourself by interacting more directly with the Theater class. The online documentation has more information on this process.", "response": "def main(world_cls, referee_cls, gui_cls, gui_actor_cls, ai_actor_cls,\n theater_cls=PygletTheater, default_host=DEFAULT_HOST,\n default_port=DEFAULT_PORT, argv=None):\n \"\"\"\nRun a game being developed with the kxg game engine.\n\nUsage:\n {exe_name} sandbox [] [-v...]\n {exe_name} client [--host HOST] [--port PORT] [-v...]\n {exe_name} server [] [--host HOST] [--port PORT] [-v...] \n {exe_name} debug [] [--host HOST] [--port PORT] [-v...]\n {exe_name} --help\n\nCommands:\n sandbox\n Play a single-player game with the specified number of AIs. None of \n the multiplayer machinery will be used.\n\n client\n Launch a client that will try to connect to a server on the given host \n and port. Once it connects and the game starts, the client will allow \n you to play the game against any other connected clients.\n\n server\n Launch a server that will manage a game between the given number of \n human and AI players. The human players must connect using this \n command's client mode.\n\n debug\n Debug a multiplayer game locally. This command launches a server and \n the given number of clients all in different processes, and configures \n the logging system such that the output from each process can be easily \n distinguished.\n\nArguments:\n \n The number of human players that will be playing the game. Only needed \n by commands that will launch some sort of multiplayer server.\n\n \n The number of AI players that will be playing the game. Only needed by \n commands that will launch single-player games or multiplayer servers.\n\nOptions:\n -x --host HOST [default: {default_host}]\n The address of the machine running the server. Must be accessible from \n the machines running the clients.\n\n -p --port PORT [default: {default_port}]\n The port that the server should listen on. Don't specify a value less \n than 1024 unless the server is running with root permissions.\n\n -v --verbose \n Have the game engine log more information about what it's doing. You \n can specify this option several times to get more and more information.\n\nThis command is provided so that you can start writing your game with the least \npossible amount of boilerplate code. However, the clients and servers provided \nby this command are not capable of running a production game. Once you have \nwritten your game and want to give it a polished set of menus and options, \nyou'll have to write new Stage subclasses encapsulating that logic and you'll \nhave to call those stages yourself by interacting more directly with the \nTheater class. The online documentation has more information on this process.\n \"\"\"\n import sys, os, docopt, nonstdlib\n\n exe_name = os.path.basename(sys.argv[0])\n usage = main.__doc__.format(**locals()).strip()\n args = docopt.docopt(usage, argv or sys.argv[1:])\n num_guis = int(args[''] or 1)\n num_ais = int(args[''] or 0)\n host, port = args['--host'], int(args['--port'])\n\n logging.basicConfig(\n format='%(levelname)s: %(name)s: %(message)s',\n level=nonstdlib.verbosity(args['--verbose']),\n )\n\n # Use the given game objects and command line arguments to play a game!\n\n if args['debug']:\n print(\"\"\"\\\n****************************** KNOWN BUG WARNING ******************************\nIn debug mode, every message produced by the logging system gets printed twice.\nI know vaguely why this is happening, but as of yet I've not been able to fix\nit. In the mean time, don't let this confuse you!\n*******************************************************************************\"\"\")\n game = MultiplayerDebugger(\n world_cls, referee_cls, gui_cls, gui_actor_cls, num_guis,\n ai_actor_cls, num_ais, theater_cls, host, port)\n else:\n game = theater_cls()\n ai_actors = [ai_actor_cls() for i in range(num_ais)]\n\n if args['sandbox']:\n game.gui = gui_cls()\n game.initial_stage = UniplayerGameStage(\n world_cls(), referee_cls(), gui_actor_cls(), ai_actors)\n game.initial_stage.successor = PostgameSplashStage()\n\n if args['client']:\n game.gui = gui_cls()\n game.initial_stage = ClientConnectionStage(\n world_cls(), gui_actor_cls(), host, port)\n\n if args['server']:\n game.initial_stage = ServerConnectionStage(\n world_cls(), referee_cls(), num_guis, ai_actors,\n host, port)\n\n game.play()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _run_supervisor(self):\n import time\n\n still_supervising = lambda: (\n multiprocessing.active_children()\n or not self.log_queue.empty()\n or not self.exception_queue.empty())\n\n try:\n while still_supervising():\n # When a log message is received, make a logger with the same \n # name in this process and use it to re-log the message. It \n # will get handled in this process.\n\n try:\n record = self.log_queue.get_nowait()\n logger = logging.getLogger(record.name)\n logger.handle(record)\n except queue.Empty:\n pass\n\n # When an exception is received, immediately re-raise it.\n\n try:\n exception = self.exception_queue.get_nowait()\n except queue.Empty:\n pass\n else:\n raise exception\n\n # Sleep for a little bit, and make sure that the workers haven't \n # outlived their time limit.\n\n time.sleep(1/self.frame_rate)\n self.elapsed_time += 1/self.frame_rate\n\n if self.time_limit and self.elapsed_time > self.time_limit:\n raise RuntimeError(\"timeout\")\n\n # Make sure the workers don't outlive the supervisor, no matter how the \n # polling loop ended (e.g. normal execution or an exception).\n\n finally:\n for process in multiprocessing.active_children():\n process.terminate()", "response": "This function runs the supervisor process."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef field_type(self):\n if not self.model:\n return 'JSON'\n database = self.model._meta.database\n if isinstance(database, Proxy):\n database = database.obj\n if Json and isinstance(database, PostgresqlDatabase):\n return 'JSON'\n return 'TEXT'", "response": "Return database field type."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef python_value(self, value):\n if self.field_type == 'TEXT' and isinstance(value, str):\n return self.loads(value)\n return value", "response": "Parse value from database."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_fsapi_endpoint(self):\n endpoint = yield from self.__session.get(self.fsapi_device_url, timeout = self.timeout)\n text = yield from endpoint.text(encoding='utf-8')\n doc = objectify.fromstring(text)\n return doc.webfsapi.text", "response": "Parse the fsapi endpoint from the device url."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a session on the frontier silicon device.", "response": "def create_session(self):\n \"\"\"Create a session on the frontier silicon device.\"\"\"\n req_url = '%s/%s' % (self.__webfsapi, 'CREATE_SESSION')\n sid = yield from self.__session.get(req_url, params=dict(pin=self.pin),\n timeout = self.timeout)\n text = yield from sid.text(encoding='utf-8')\n doc = objectify.fromstring(text)\n return doc.sessionId.text"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef call(self, path, extra=None):\n try:\n if not self.__webfsapi:\n self.__webfsapi = yield from self.get_fsapi_endpoint()\n\n if not self.sid:\n self.sid = yield from self.create_session()\n\n if not isinstance(extra, dict):\n extra = dict()\n\n params = dict(pin=self.pin, sid=self.sid)\n params.update(**extra)\n\n req_url = ('%s/%s' % (self.__webfsapi, path))\n result = yield from self.__session.get(req_url, params=params,\n timeout = self.timeout)\n if result.status == 200:\n text = yield from result.text(encoding='utf-8')\n else:\n self.sid = yield from self.create_session()\n params = dict(pin=self.pin, sid=self.sid)\n params.update(**extra)\n result = yield from self.__session.get(req_url, params=params,\n timeout = self.timeout)\n text = yield from result.text(encoding='utf-8')\n\n return objectify.fromstring(text)\n except Exception as e:\n logging.info('AFSAPI Exception: ' +traceback.format_exc())\n\n return None", "response": "Execute a frontier silicon API call."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef handle_text(self, item):\n doc = yield from self.handle_get(item)\n if doc is None:\n return None\n\n return doc.value.c8_array.text or None", "response": "Helper method for fetching a text value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef handle_int(self, item):\n doc = yield from self.handle_get(item)\n if doc is None:\n return None\n\n return int(doc.value.u8.text) or None", "response": "Helper method for fetching an integer value."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling a long item. Result is integer.", "response": "def handle_long(self, item):\n \"\"\"Helper method for fetching a long value. Result is integer.\"\"\"\n doc = yield from self.handle_get(item)\n if doc is None:\n return None\n\n return int(doc.value.u32.text) or None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef handle_list(self, item):\n doc = yield from self.call('LIST_GET_NEXT/'+item+'/-1', dict(\n maxItems=100,\n ))\n\n if doc is None:\n return []\n\n if not doc.status == 'FS_OK':\n return []\n\n ret = list()\n for index, item in enumerate(list(doc.iterchildren('item'))):\n temp = dict(band=index)\n for field in list(item.iterchildren()):\n temp[field.get('name')] = list(field.iterchildren()).pop()\n ret.append(temp)\n\n return ret", "response": "Helper method for fetching a list ( map ) value."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if the device is on.", "response": "def get_power(self):\n \"\"\"Check if the device is on.\"\"\"\n power = (yield from self.handle_int(self.API.get('power')))\n return bool(power)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npowers on or off the device.", "response": "def set_power(self, value=False):\n \"\"\"Power on or off the device.\"\"\"\n power = (yield from self.handle_set(\n self.API.get('power'), int(value)))\n return bool(power)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the modes supported by this device.", "response": "def get_modes(self):\n \"\"\"Get the modes supported by this device.\"\"\"\n if not self.__modes:\n self.__modes = yield from self.handle_list(\n self.API.get('valid_modes'))\n\n return self.__modes"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_mode_list(self):\n self.__modes = yield from self.get_modes()\n return (yield from self.collect_labels(self.__modes))", "response": "Get the label list of the supported modes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_mode(self):\n mode = None\n int_mode = (yield from self.handle_long(self.API.get('mode')))\n modes = yield from self.get_modes()\n for temp_mode in modes:\n if temp_mode['band'] == int_mode:\n mode = temp_mode['label']\n\n return str(mode)", "response": "Get the currently active mode on the device."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the currently active mode on the device.", "response": "def set_mode(self, value):\n \"\"\"Set the currently active mode on the device (DAB, FM, Spotify).\"\"\"\n mode = -1\n modes = yield from self.get_modes()\n for temp_mode in modes:\n if temp_mode['label'] == value:\n mode = temp_mode['band']\n\n return (yield from self.handle_set(self.API.get('mode'), mode))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_volume_steps(self):\n if not self.__volume_steps:\n self.__volume_steps = yield from self.handle_int(\n self.API.get('volume_steps'))\n\n return self.__volume_steps", "response": "Read the maximum volume level of the device."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_mute(self):\n mute = (yield from self.handle_int(self.API.get('mute')))\n return bool(mute)", "response": "Check if the device is muted."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_mute(self, value=False):\n mute = (yield from self.handle_set(self.API.get('mute'), int(value)))\n return bool(mute)", "response": "Mute or unmute the device."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_play_status(self):\n status = yield from self.handle_int(self.API.get('status'))\n return self.PLAY_STATES.get(status)", "response": "Get the play status of the device."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the equaliser modes supported by this device.", "response": "def get_equalisers(self):\n \"\"\"Get the equaliser modes supported by this device.\"\"\"\n if not self.__equalisers:\n self.__equalisers = yield from self.handle_list(\n self.API.get('equalisers'))\n\n return self.__equalisers"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_equaliser_list(self):\n self.__equalisers = yield from self.get_equalisers()\n return (yield from self.collect_labels(self.__equalisers))", "response": "Get the list of the supported modes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_sleep(self, value=False):\n return (yield from self.handle_set(self.API.get('sleep'), int(value)))", "response": "Set device sleep timer."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _set_range(self, start, stop, value, value_len):\n assert stop >= start and value_len >= 0\n range_len = stop - start\n if range_len < value_len:\n self._insert_zeros(stop, stop + value_len - range_len)\n self._copy_to_range(start, value, value_len)\n elif range_len > value_len:\n self._del_range(stop - (range_len - value_len), stop)\n self._copy_to_range(start, value, value_len)\n else:\n self._copy_to_range(start, value, value_len)", "response": "Set the value of the log entry."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses genotype from VCF line data", "response": "def _parse_genotype(self, vcf_fields):\n \"\"\"Parse genotype from VCF line data\"\"\"\n format_col = vcf_fields[8].split(':')\n genome_data = vcf_fields[9].split(':')\n try:\n gt_idx = format_col.index('GT')\n except ValueError:\n return []\n return [int(x) for x in re.split(r'[\\|/]', genome_data[gt_idx]) if\n x != '.']"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef setDefaultIREncoding(encoding):\n\t'''\n\t\tsetDefaultIREncoding - Sets the default encoding used by IndexedRedis.\n\t\t This will be the default encoding used for field data. You can override this on a\n\t\t per-field basis by using an IRField (such as IRUnicodeField or IRRawField)\n\n\t\t@param encoding - An encoding (like utf-8)\n\t'''\n\ttry:\n\t\tb''.decode(encoding)\n\texcept:\n\t\traise ValueError('setDefaultIREncoding was provided an invalid codec. Got (encoding=\"%s\")' %(str(encoding), ))\n\n\tglobal defaultIREncoding\n\tdefaultIREncoding = encoding", "response": "Sets the default encoding used by IndexedRedis."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _getReprProperties(self):\n\t\t'''\n\t\t\t_getReprProperties - Get the properties of this field to display in repr().\n\n\t\t\t\tThese should be in the form of $propertyName=$propertyRepr\n\n\t\t\t\tThe default IRField implementation handles just the \"hashIndex\" property.\n\n\t\t\t\tdefaultValue is part of \"__repr__\" impl. You should just extend this method\n\t\t\t\twith your object's properties instead of rewriting repr.\n\n\t\t'''\n\t\tret = []\n\t\tif getattr(self, 'valueType', None) is not None:\n\t\t\tret.append('valueType=%s' %(self.valueType.__name__, ))\n\t\tif hasattr(self, 'hashIndex'):\n\t\t\tret.append('hashIndex=%s' %(self.hashIndex, ))\n\n\t\treturn ret", "response": "Returns a list of properties to be displayed in the repr."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a copy of this IRField.", "response": "def copy(self):\n\t\t'''\n\t\t\tcopy - Create a copy of this IRField.\n\n\t\t\t Each subclass should implement this, as you'll need to pass in the args to constructor.\n\n\t\t\t@return - Another IRField that has all the same values as this one.\n\t\t'''\n\t\treturn self.__class__(name=self.name, valueType=self.valueType, defaultValue=self.defaultValue, hashIndex=self.hashIndex)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the object associated with this data.", "response": "def getObj(self):\n\t\t'''\n\t\t\tgetObj - Fetch (if not fetched) and return the obj associated with this data.\n\t\t'''\n\t\tif self.obj is None:\n\t\t\tif not self.pk:\n\t\t\t\treturn None\n\t\t\tself.obj = self.foreignModel.objects.get(self.pk)\n\n\t\treturn self.obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the pk of the object.", "response": "def getPk(self):\n\t\t'''\n\t\t\tgetPk - Resolve any absent pk's off the obj's (like if an obj has been saved), and return the pk.\n\t\t'''\n\t\tif not self.pk and self.obj:\n\t\t\tif self.obj._id:\n\t\t\t\tself.pk = self.obj._id\n\n\t\treturn self.pk"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True if any object has unsaved changes.", "response": "def objHasUnsavedChanges(self):\n\t\t'''\n\t\t\tobjHasUnsavedChanges - Check if any object has unsaved changes, cascading.\n\t\t'''\n\t\tif not self.obj:\n\t\t\treturn False\n\n\t\treturn self.obj.hasUnsavedChanges(cascadeObjects=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting PK - @see ForeignLinkData. getPk - @see ForeignLinkData. getPk", "response": "def getPk(self):\n\t\t'''\n\t\t\tgetPk - @see ForeignLinkData.getPk\n\t\t'''\n\t\tif not self.pk or None in self.pk:\n\t\t\tfor i in range( len(self.pk) ):\n\t\t\t\tif self.pk[i]:\n\t\t\t\t\tcontinue\n\n\t\t\t\tif self.obj[i] and self.obj[i]._id:\n\t\t\t\t\tself.pk[i] = self.obj[i]._id\n\n\t\treturn self.pk"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if the object is fetched.", "response": "def isFetched(self):\n\t\t'''\n\t\t\tisFetched - @see ForeignLinkData.isFetched\n\t\t'''\n\t\tif not self.obj:\n\t\t\treturn False\n\n\t\tif not self.pk or None in self.obj:\n\t\t\treturn False\n\t\treturn not bool(self.obj is None)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef objHasUnsavedChanges(self):\n\t\t'''\n\t\t\tobjHasUnsavedChanges - @see ForeignLinkData.objHasUnsavedChanges\n\n\t\t\tTrue if ANY object has unsaved changes.\n\t\t'''\n\t\tif not self.obj:\n\t\t\treturn False\n\n\t\tfor thisObj in self.obj:\n\t\t\tif not thisObj:\n\t\t\t\tcontinue\n\t\t\tif thisObj.hasUnsavedChanges(cascadeObjects=True):\n\t\t\t\treturn True\n\n\t\treturn False", "response": "Returns True if any of the objects in the object has unsaved changes."}