{"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef solve_discrete_lyapunov(A, B, max_it=50, method=\"doubling\"):\n if method == \"doubling\":\n A, B = list(map(np.atleast_2d, [A, B]))\n alpha0 = A\n gamma0 = B\n\n diff = 5\n n_its = 1\n\n while diff > 1e-15:\n\n alpha1 = alpha0.dot(alpha0)\n gamma1 = gamma0 + np.dot(alpha0.dot(gamma0), alpha0.conjugate().T)\n\n diff = np.max(np.abs(gamma1 - gamma0))\n alpha0 = alpha1\n gamma0 = gamma1\n\n n_its += 1\n\n if n_its > max_it:\n msg = \"Exceeded maximum iterations {}, check input matrics\"\n raise ValueError(msg.format(n_its))\n\n elif method == \"bartels-stewart\":\n gamma1 = sp_solve_discrete_lyapunov(A, B)\n\n else:\n msg = \"Check your method input. Should be doubling or bartels-stewart\"\n raise ValueError(msg)\n\n return gamma1", "response": "r Solve the discrete lyapunov problem."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef solve_discrete_riccati(A, B, Q, R, N=None, tolerance=1e-10, max_iter=500,\n method=\"doubling\"):\n \"\"\"\n Solves the discrete-time algebraic Riccati equation\n\n .. math::\n\n X = A'XA - (N + B'XA)'(B'XB + R)^{-1}(N + B'XA) + Q\n\n Computation is via a modified structured doubling algorithm, an\n explanation of which can be found in the reference below, if\n `method=\"doubling\"` (default), and via a QZ decomposition method by\n calling `scipy.linalg.solve_discrete_are` if `method=\"qz\"`.\n\n Parameters\n ----------\n A : array_like(float, ndim=2)\n k x k array.\n B : array_like(float, ndim=2)\n k x n array\n Q : array_like(float, ndim=2)\n k x k, should be symmetric and non-negative definite\n R : array_like(float, ndim=2)\n n x n, should be symmetric and positive definite\n N : array_like(float, ndim=2)\n n x k array\n tolerance : scalar(float), optional(default=1e-10)\n The tolerance level for convergence\n max_iter : scalar(int), optional(default=500)\n The maximum number of iterations allowed\n method : string, optional(default=\"doubling\")\n Describes the solution method to use. If it is \"doubling\" then\n uses the doubling algorithm to solve, if it is \"qz\" then it uses\n `scipy.linalg.solve_discrete_are` (in which case `tolerance` and\n `max_iter` are irrelevant).\n\n Returns\n -------\n X : array_like(float, ndim=2)\n The fixed point of the Riccati equation; a k x k array\n representing the approximate solution\n\n References\n ----------\n Chiang, Chun-Yueh, Hung-Yuan Fan, and Wen-Wei Lin. \"STRUCTURED DOUBLING\n ALGORITHM FOR DISCRETE-TIME ALGEBRAIC RICCATI EQUATIONS WITH SINGULAR\n CONTROL WEIGHTING MATRICES.\" Taiwanese Journal of Mathematics 14, no. 3A\n (2010): pp-935.\n\n \"\"\"\n methods = ['doubling', 'qz']\n if method not in methods:\n msg = \"Check your method input. Should be {} or {}\".format(*methods)\n raise ValueError(msg)\n\n # == Set up == #\n error = tolerance + 1\n fail_msg = \"Convergence failed after {} iterations.\"\n\n # == Make sure that all array_likes are np arrays, two-dimensional == #\n A, B, Q, R = np.atleast_2d(A, B, Q, R)\n n, k = R.shape[0], Q.shape[0]\n I = np.identity(k)\n if N is None:\n N = np.zeros((n, k))\n else:\n N = np.atleast_2d(N)\n\n if method == 'qz':\n X = sp_solve_discrete_are(A, B, Q, R, s=N.T)\n return X\n\n # if method == 'doubling'\n # == Choose optimal value of gamma in R_hat = R + gamma B'B == #\n current_min = np.inf\n candidates = (0.01, 0.1, 0.25, 0.5, 1.0, 2.0, 10.0, 100.0, 10e5)\n BB = dot(B.T, B)\n BTA = dot(B.T, A)\n for gamma in candidates:\n Z = R + gamma * BB\n cn = np.linalg.cond(Z)\n if cn * EPS < 1:\n Q_tilde = - Q + dot(N.T, solve(Z, N + gamma * BTA)) + gamma * I\n G0 = dot(B, solve(Z, B.T))\n A0 = dot(I - gamma * G0, A) - dot(B, solve(Z, N))\n H0 = gamma * dot(A.T, A0) - Q_tilde\n f1 = np.linalg.cond(Z, np.inf)\n f2 = gamma * f1\n f3 = np.linalg.cond(I + dot(G0, H0))\n f_gamma = max(f1, f2, f3)\n if f_gamma < current_min:\n best_gamma = gamma\n current_min = f_gamma\n\n # == If no candidate successful then fail == #\n if current_min == np.inf:\n msg = \"Unable to initialize routine due to ill conditioned arguments\"\n raise ValueError(msg)\n\n gamma = best_gamma\n R_hat = R + gamma * BB\n\n # == Initial conditions == #\n Q_tilde = - Q + dot(N.T, solve(R_hat, N + gamma * BTA)) + gamma * I\n G0 = dot(B, solve(R_hat, B.T))\n A0 = dot(I - gamma * G0, A) - dot(B, solve(R_hat, N))\n H0 = gamma * dot(A.T, A0) - Q_tilde\n i = 1\n\n # == Main loop == #\n while error > tolerance:\n\n if i > max_iter:\n raise ValueError(fail_msg.format(i))\n\n else:\n A1 = dot(A0, solve(I + dot(G0, H0), A0))\n G1 = G0 + dot(dot(A0, G0), solve(I + dot(H0, G0), A0.T))\n H1 = H0 + dot(A0.T, solve(I + dot(H0, G0), dot(H0, A0)))\n\n error = np.max(np.abs(H1 - H0))\n A0 = A1\n G0 = G1\n H0 = H1\n i += 1\n\n return H1 + gamma * I", "response": "Solve the discrete - time algebraic Riccati equation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef next_k_array(a):\n # Logic taken from Algotirhm T in D. Knuth, The Art of Computer\n # Programming, Section 7.2.1.3 \"Generating All Combinations\".\n k = len(a)\n if k == 1 or a[0] + 1 < a[1]:\n a[0] += 1\n return a\n\n a[0] = 0\n i = 1\n x = a[i] + 1\n\n while i < k-1 and x == a[i+1]:\n i += 1\n a[i-1] = i - 1\n x = a[i] + 1\n a[i] = x\n\n return a", "response": "Given an array a of k distinct nonnegative integers return the next k - array in the lexicographic order."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef k_array_rank(a):\n k = len(a)\n idx = int(a[0]) # Convert to Python int\n for i in range(1, k):\n idx += comb(a[i], i+1, exact=True)\n return idx", "response": "Returns the ranking of the k distinct nonnegative integers in the lexicographic ordering of the elements of the elements in the array a."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nnumbs jit version of k_array_rank.", "response": "def k_array_rank_jit(a):\n \"\"\"\n Numba jit version of `k_array_rank`.\n\n Notes\n -----\n An incorrect value will be returned without warning or error if\n overflow occurs during the computation. It is the user's\n responsibility to ensure that the rank of the input array fits\n within the range of possible values of `np.intp`; a sufficient\n condition for it is `scipy.special.comb(a[-1]+1, len(a), exact=True)\n <= np.iinfo(np.intp).max`.\n\n \"\"\"\n k = len(a)\n idx = a[0]\n for i in range(1, k):\n idx += comb_jit(a[i], i+1)\n return idx"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a function that solves the linear equation ax = b directly calling a Numba internal function.", "response": "def _numba_linalg_solve(a, b):\n \"\"\"\n Solve the linear equation ax = b directly calling a Numba internal\n function. The data in `a` and `b` are interpreted in Fortran order,\n and dtype of `a` and `b` must be the same, one of {float32, float64,\n complex64, complex128}. `a` and `b` are modified in place, and the\n solution is stored in `b`. *No error check is made for the inputs.*\n\n Parameters\n ----------\n a : ndarray(ndim=2)\n 2-dimensional ndarray of shape (n, n).\n\n b : ndarray(ndim=1 or 2)\n 1-dimensional ndarray of shape (n,) or 2-dimensional ndarray of\n shape (n, nrhs).\n\n Returns\n -------\n r : scalar(int)\n r = 0 if successful.\n\n Notes\n -----\n From github.com/numba/numba/blob/master/numba/targets/linalg.py\n\n \"\"\"\n numba_xgesv = _LAPACK().numba_xgesv(a.dtype)\n kind = ord(_blas_kinds[a.dtype])\n\n def _numba_linalg_solve_impl(a, b): # pragma: no cover\n n = a.shape[-1]\n if b.ndim == 1:\n nrhs = 1\n else: # b.ndim == 2\n nrhs = b.shape[-1]\n F_INT_nptype = np.int32\n ipiv = np.empty(n, dtype=F_INT_nptype)\n\n r = numba_xgesv(\n kind, # kind\n n, # n\n nrhs, # nhrs\n a.ctypes, # a\n n, # lda\n ipiv.ctypes, # ipiv\n b.ctypes, # b\n n # ldb\n )\n return r\n\n return _numba_linalg_solve_impl"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nnumbing jitted function that computes N choose k. Return `0` if the outcome exceeds the maximum value of `np.intp` or if N < 0, k < 0, or k > N. Parameters ---------- N : scalar(int) k : scalar(int) Returns ------- val : scalar(int)", "response": "def comb_jit(N, k):\n \"\"\"\n Numba jitted function that computes N choose k. Return `0` if the\n outcome exceeds the maximum value of `np.intp` or if N < 0, k < 0,\n or k > N.\n\n Parameters\n ----------\n N : scalar(int)\n\n k : scalar(int)\n\n Returns\n -------\n val : scalar(int)\n\n \"\"\"\n # From scipy.special._comb_int_long\n # github.com/scipy/scipy/blob/v1.0.0/scipy/special/_comb.pyx\n INTP_MAX = np.iinfo(np.intp).max\n if N < 0 or k < 0 or k > N:\n return 0\n if k == 0:\n return 1\n if k == 1:\n return N\n if N == INTP_MAX:\n return 0\n\n M = N + 1\n nterms = min(k, N - k)\n\n val = 1\n\n for j in range(1, nterms + 1):\n # Overflow check\n if val > INTP_MAX // (M - j):\n return 0\n\n val *= M - j\n val //= j\n\n return val"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _integrate_fixed_trajectory(self, h, T, step, relax):\n # initialize the solution using initial condition\n solution = np.hstack((self.t, self.y))\n\n while self.successful():\n\n self.integrate(self.t + h, step, relax)\n current_step = np.hstack((self.t, self.y))\n solution = np.vstack((solution, current_step))\n\n if (h > 0) and (self.t >= T):\n break\n elif (h < 0) and (self.t <= T):\n break\n else:\n continue\n\n return solution", "response": "Generates a solution trajectory of fixed length."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _integrate_variable_trajectory(self, h, g, tol, step, relax):\n # initialize the solution using initial condition\n solution = np.hstack((self.t, self.y))\n\n while self.successful():\n\n self.integrate(self.t + h, step, relax)\n current_step = np.hstack((self.t, self.y))\n solution = np.vstack((solution, current_step))\n\n if g(self.t, self.y, *self.f_params) < tol:\n break\n else:\n continue\n\n return solution", "response": "Generates a solution trajectory of variable length."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _initialize_integrator(self, t0, y0, integrator, **kwargs):\n # set the initial condition\n self.set_initial_value(y0, t0)\n\n # select the integrator\n self.set_integrator(integrator, **kwargs)", "response": "Initializes the integrator prior to integration."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compute_residual(self, traj, ti, k=3, ext=2):\n # B-spline approximations of the solution and its derivative\n soln = self.interpolate(traj, ti, k, 0, ext)\n deriv = self.interpolate(traj, ti, k, 1, ext)\n\n # rhs of ode evaluated along approximate solution\n T = ti.size\n rhs_ode = np.vstack(self.f(ti[i], soln[i, 1:], *self.f_params)\n for i in range(T))\n rhs_ode = np.hstack((ti[:, np.newaxis], rhs_ode))\n\n # should be roughly zero everywhere (if approximation is any good!)\n residual = deriv - rhs_ode\n\n return residual", "response": "r Compute the residual of the knot sequence of the current knot sequence."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the Lorenz curve for a set of income and wealth.", "response": "def lorenz_curve(y):\n \"\"\"\n Calculates the Lorenz Curve, a graphical representation of the distribution of income\n or wealth.\n\n It returns the cumulative share of people (x-axis) and the cumulative share of income earned\n\n Parameters\n ----------\n y : array_like(float or int, ndim=1)\n Array of income/wealth for each individual. Unordered or ordered is fine.\n\n Returns\n -------\n cum_people : array_like(float, ndim=1)\n Cumulative share of people for each person index (i/n)\n cum_income : array_like(float, ndim=1)\n Cumulative share of income for each person index\n\n\n References\n ----------\n .. [1] https://en.wikipedia.org/wiki/Lorenz_curve\n\n Examples\n --------\n >>> a_val, n = 3, 10_000\n >>> y = np.random.pareto(a_val, size=n)\n >>> f_vals, l_vals = lorenz(y)\n\n \"\"\"\n\n n = len(y)\n y = np.sort(y)\n s = np.zeros(n + 1)\n s[1:] = np.cumsum(y)\n cum_people = np.zeros(n + 1)\n cum_income = np.zeros(n + 1)\n for i in range(1, n + 1):\n cum_people[i] = i / n\n cum_income[i] = s[i] / s[n]\n return cum_people, cum_income"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef gini_coefficient(y):\n n = len(y)\n i_sum = np.zeros(n)\n for i in prange(n):\n for j in range(n):\n i_sum[i] += abs(y[i] - y[j])\n return np.sum(i_sum) / (2 * n * np.sum(y))", "response": "r Implements the Gini inequality index\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the internal _q and Q attributes.", "response": "def q(self, val):\n \"\"\"\n Setter method for q.\n\n \"\"\"\n self._q = np.asarray(val)\n self.Q = cumsum(val)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw(self, k=1, random_state=None):\n random_state = check_random_state(random_state)\n\n return self.Q.searchsorted(random_state.uniform(0, 1, size=k),\n side='right')", "response": "Returns k draws from the probability set."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a randomly sampled MarkovChain instance with n states where each state has k states with positive transition probability.", "response": "def random_markov_chain(n, k=None, sparse=False, random_state=None):\n \"\"\"\n Return a randomly sampled MarkovChain instance with n states, where\n each state has k states with positive transition probability.\n\n Parameters\n ----------\n n : scalar(int)\n Number of states.\n\n k : scalar(int), optional(default=None)\n Number of states that may be reached from each state with\n positive probability. Set to n if not specified.\n\n sparse : bool, optional(default=False)\n Whether to store the transition probability matrix in sparse\n matrix form.\n\n random_state : int or np.random.RandomState, optional\n Random seed (integer) or np.random.RandomState instance to set\n the initial state of the random number generator for\n reproducibility. If None, a randomly initialized RandomState is\n used.\n\n Returns\n -------\n mc : MarkovChain\n\n Examples\n --------\n >>> mc = qe.markov.random_markov_chain(3, random_state=1234)\n >>> mc.P\n array([[ 0.19151945, 0.43058932, 0.37789123],\n [ 0.43772774, 0.34763084, 0.21464142],\n [ 0.27259261, 0.5073832 , 0.22002419]])\n >>> mc = qe.markov.random_markov_chain(3, k=2, random_state=1234)\n >>> mc.P\n array([[ 0.19151945, 0.80848055, 0. ],\n [ 0. , 0.62210877, 0.37789123],\n [ 0.56227226, 0. , 0.43772774]])\n\n \"\"\"\n P = random_stochastic_matrix(n, k, sparse, format='csr',\n random_state=random_state)\n mc = MarkovChain(P)\n return mc"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef random_stochastic_matrix(n, k=None, sparse=False, format='csr',\n random_state=None):\n \"\"\"\n Return a randomly sampled n x n stochastic matrix with k nonzero\n entries for each row.\n\n Parameters\n ----------\n n : scalar(int)\n Number of states.\n\n k : scalar(int), optional(default=None)\n Number of nonzero entries in each row of the matrix. Set to n if\n not specified.\n\n sparse : bool, optional(default=False)\n Whether to generate the matrix in sparse matrix form.\n\n format : str, optional(default='csr')\n Sparse matrix format, str in {'bsr', 'csr', 'csc', 'coo', 'lil',\n 'dia', 'dok'}. Relevant only when sparse=True.\n\n random_state : int or np.random.RandomState, optional\n Random seed (integer) or np.random.RandomState instance to set\n the initial state of the random number generator for\n reproducibility. If None, a randomly initialized RandomState is\n used.\n\n Returns\n -------\n P : numpy ndarray or scipy sparse matrix (float, ndim=2)\n Stochastic matrix.\n\n See also\n --------\n random_markov_chain : Return a random MarkovChain instance.\n\n \"\"\"\n P = _random_stochastic_matrix(m=n, n=n, k=k, sparse=sparse, format=format,\n random_state=random_state)\n return P", "response": "Returns a randomly sampled n x n stochastic matrix with k nonzero entries for each row."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a random non - square stochastic matrix.", "response": "def _random_stochastic_matrix(m, n, k=None, sparse=False, format='csr',\n random_state=None):\n \"\"\"\n Generate a \"non-square stochastic matrix\" of shape (m, n), which\n contains as rows m probability vectors of length n with k nonzero\n entries.\n\n For other parameters, see `random_stochastic_matrix`.\n\n \"\"\"\n if k is None:\n k = n\n # m prob vectors of dimension k, shape (m, k)\n probvecs = probvec(m, k, random_state=random_state)\n\n if k == n:\n P = probvecs\n if sparse:\n return scipy.sparse.coo_matrix(P).asformat(format)\n else:\n return P\n\n # if k < n:\n rows = np.repeat(np.arange(m), k)\n cols = \\\n sample_without_replacement(\n n, k, num_trials=m, random_state=random_state\n ).ravel()\n data = probvecs.ravel()\n\n if sparse:\n P = scipy.sparse.coo_matrix((data, (rows, cols)), shape=(m, n))\n return P.asformat(format)\n else:\n P = np.zeros((m, n))\n P[rows, cols] = data\n return P"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef random_discrete_dp(num_states, num_actions, beta=None,\n k=None, scale=1, sparse=False, sa_pair=False,\n random_state=None):\n \"\"\"\n Generate a DiscreteDP randomly. The reward values are drawn from the\n normal distribution with mean 0 and standard deviation `scale`.\n\n Parameters\n ----------\n num_states : scalar(int)\n Number of states.\n\n num_actions : scalar(int)\n Number of actions.\n\n beta : scalar(float), optional(default=None)\n Discount factor. Randomly chosen from [0, 1) if not specified.\n\n k : scalar(int), optional(default=None)\n Number of possible next states for each state-action pair. Equal\n to `num_states` if not specified.\n\n scale : scalar(float), optional(default=1)\n Standard deviation of the normal distribution for the reward\n values.\n\n sparse : bool, optional(default=False)\n Whether to store the transition probability array in sparse\n matrix form.\n\n sa_pair : bool, optional(default=False)\n Whether to represent the data in the state-action pairs\n formulation. (If `sparse=True`, automatically set `True`.)\n\n random_state : int or np.random.RandomState, optional\n Random seed (integer) or np.random.RandomState instance to set\n the initial state of the random number generator for\n reproducibility. If None, a randomly initialized RandomState is\n used.\n\n Returns\n -------\n ddp : DiscreteDP\n An instance of DiscreteDP.\n\n \"\"\"\n if sparse:\n sa_pair = True\n\n # Number of state-action pairs\n L = num_states * num_actions\n\n random_state = check_random_state(random_state)\n R = scale * random_state.randn(L)\n Q = _random_stochastic_matrix(L, num_states, k=k,\n sparse=sparse, format='csr',\n random_state=random_state)\n if beta is None:\n beta = random_state.random_sample()\n\n if sa_pair:\n s_indices, a_indices = sa_indices(num_states, num_actions)\n else:\n s_indices, a_indices = None, None\n R.shape = (num_states, num_actions)\n Q.shape = (num_states, num_actions, num_states)\n\n ddp = DiscreteDP(R, Q, beta, s_indices, a_indices)\n return ddp", "response": "Generates a DiscreteDP from the given state - action pairs and random values."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef robust_rule_simple(self, P_init=None, max_iter=80, tol=1e-8):\n # == Simplify names == #\n A, B, C, Q, R = self.A, self.B, self.C, self.Q, self.R\n beta, theta = self.beta, self.theta\n # == Set up loop == #\n P = np.zeros((self.n, self.n)) if P_init is None else P_init\n iterate, e = 0, tol + 1\n while iterate < max_iter and e > tol:\n F, new_P = self.b_operator(self.d_operator(P))\n e = np.sqrt(np.sum((new_P - P)**2))\n iterate += 1\n P = new_P\n I = np.identity(self.j)\n S1 = P.dot(C)\n S2 = C.T.dot(S1)\n K = inv(theta * I - S2).dot(S1.T).dot(A - B.dot(F))\n\n return F, K, P", "response": "A simple algorithm for computing the robust policy F and the value function P for the current key - value entry. This method is used to compute the robust policy F and the value function P for the current key - value entry."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef F_to_K(self, F, method='doubling'):\n Q2 = self.beta * self.theta\n R2 = - self.R - dot(F.T, dot(self.Q, F))\n A2 = self.A - dot(self.B, F)\n B2 = self.C\n lq = LQ(Q2, R2, A2, B2, beta=self.beta)\n neg_P, neg_K, d = lq.stationary_values(method=method)\n\n return -neg_K, -neg_P", "response": "Compute agent 2 s best cost minimizing response K given a given F."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomputing agent 1 s best value - maximizing response F given a given K.", "response": "def K_to_F(self, K, method='doubling'):\n \"\"\"\n Compute agent 1's best value-maximizing response F, given K.\n\n Parameters\n ----------\n K : array_like(float, ndim=2)\n A j x n array\n method : str, optional(default='doubling')\n Solution method used in solving the associated Riccati\n equation, str in {'doubling', 'qz'}.\n\n Returns\n -------\n F : array_like(float, ndim=2)\n The policy function for a given K\n P : array_like(float, ndim=2)\n The value function for a given K\n\n \"\"\"\n A1 = self.A + dot(self.C, K)\n B1 = self.B\n Q1 = self.Q\n R1 = self.R - self.beta * self.theta * dot(K.T, K)\n lq = LQ(Q1, R1, A1, B1, beta=self.beta)\n P, F, d = lq.stationary_values(method=method)\n\n return F, P"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nevaluate the policy function F and returns the matrix P_F and constant d_F associated with discounted cost F associated with discounted cost F.", "response": "def evaluate_F(self, F):\n \"\"\"\n Given a fixed policy F, with the interpretation :math:`u = -F x`, this\n function computes the matrix :math:`P_F` and constant :math:`d_F`\n associated with discounted cost :math:`J_F(x) = x' P_F x + d_F`\n\n Parameters\n ----------\n F : array_like(float, ndim=2)\n The policy function, a k x n array\n\n Returns\n -------\n P_F : array_like(float, ndim=2)\n Matrix for discounted cost\n d_F : scalar(float)\n Constant for discounted cost\n K_F : array_like(float, ndim=2)\n Worst case policy\n O_F : array_like(float, ndim=2)\n Matrix for discounted entropy\n o_F : scalar(float)\n Constant for discounted entropy\n\n \"\"\"\n # == Simplify names == #\n Q, R, A, B, C = self.Q, self.R, self.A, self.B, self.C\n beta, theta = self.beta, self.theta\n\n # == Solve for policies and costs using agent 2's problem == #\n K_F, P_F = self.F_to_K(F)\n I = np.identity(self.j)\n H = inv(I - C.T.dot(P_F.dot(C)) / theta)\n d_F = log(det(H))\n\n # == Compute O_F and o_F == #\n sig = -1.0 / theta\n AO = sqrt(beta) * (A - dot(B, F) + dot(C, K_F))\n O_F = solve_discrete_lyapunov(AO.T, beta * dot(K_F.T, K_F))\n ho = (trace(H - 1) - d_F) / 2.0\n tr = trace(dot(O_F, C.dot(H.dot(C.T))))\n o_F = (ho + beta * tr) / (1 - beta)\n\n return K_F, P_F, d_F, O_F, o_F"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_input_line(self, line, store_history=True):\n #print \"input='%s'\"%self.input\n stdout = sys.stdout\n splitter = self.IP.input_splitter\n try:\n sys.stdout = self.cout\n splitter.push(line)\n more = splitter.push_accepts_more()\n if not more:\n source_raw = splitter.source_raw_reset()[1]\n self.IP.run_cell(source_raw, store_history=store_history)\n finally:\n sys.stdout = stdout", "response": "process the input line capturing stdout"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef process_image(self, decorator):\n savefig_dir = self.savefig_dir\n source_dir = self.source_dir\n saveargs = decorator.split(' ')\n filename = saveargs[1]\n # insert relative path to image file in source\n outfile = os.path.relpath(os.path.join(savefig_dir,filename),\n source_dir)\n\n imagerows = ['.. image:: %s'%outfile]\n\n for kwarg in saveargs[2:]:\n arg, val = kwarg.split('=')\n arg = arg.strip()\n val = val.strip()\n imagerows.append(' :%s: %s'%(arg, val))\n\n image_file = os.path.basename(outfile) # only return file name\n image_directive = '\\n'.join(imagerows)\n return image_file, image_directive", "response": "process an image and return the filename and the image directive"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef process_input(self, data, input_prompt, lineno):\n decorator, input, rest = data\n image_file = None\n image_directive = None\n #print 'INPUT:', data # dbg\n is_verbatim = decorator=='@verbatim' or self.is_verbatim\n is_doctest = decorator=='@doctest' or self.is_doctest\n is_suppress = decorator=='@suppress' or self.is_suppress\n is_okexcept = decorator=='@okexcept' or self.is_okexcept\n is_savefig = decorator is not None and \\\n decorator.startswith('@savefig')\n\n input_lines = input.split('\\n')\n\n self.datacontent = data\n\n continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2))\n\n if is_savefig:\n image_file, image_directive = self.process_image(decorator)\n\n ret = []\n is_semicolon = False\n store_history = True\n\n for i, line in enumerate(input_lines):\n if line.endswith(';'):\n is_semicolon = True\n if is_semicolon or is_suppress:\n store_history = False\n\n if i==0:\n # process the first input line\n if is_verbatim:\n self.process_input_line('')\n self.IP.execution_count += 1 # increment it anyway\n else:\n # only submit the line in non-verbatim mode\n self.process_input_line(line, store_history=store_history)\n formatted_line = '%s %s'%(input_prompt, line)\n else:\n # process a continuation line\n if not is_verbatim:\n self.process_input_line(line, store_history=store_history)\n\n formatted_line = '%s%s'%(continuation, line)\n\n if not is_suppress:\n ret.append(formatted_line)\n\n if not is_suppress:\n if len(rest.strip()):\n if is_verbatim:\n # the \"rest\" is the standard output of the\n # input, which needs to be added in\n # verbatim mode\n ret.append(rest)\n\n self.cout.seek(0)\n output = self.cout.read()\n if not is_suppress and not is_semicolon:\n ret.append(output.decode('utf-8'))\n\n if not is_okexcept and \"Traceback\" in output:\n sys.stdout.write(output)\n\n self.cout.truncate(0)\n return (ret, input_lines, output, is_doctest, image_file,\n image_directive)", "response": "Process the input block for INPUT token."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprocess data block for OUTPUT token.", "response": "def process_output(self, data, output_prompt,\n input_lines, output, is_doctest, image_file):\n \"\"\"Process data block for OUTPUT token.\"\"\"\n if is_doctest:\n submitted = data.strip()\n found = output\n if found is not None:\n found = found.strip()\n\n # XXX - fperez: in 0.11, 'output' never comes with the prompt\n # in it, just the actual output text. So I think all this code\n # can be nuked...\n\n # the above comment does not appear to be accurate... (minrk)\n\n ind = found.find(output_prompt)\n if ind<0:\n e='output prompt=\"%s\" does not match out line=%s' % \\\n (output_prompt, found)\n raise RuntimeError(e)\n found = found[len(output_prompt):].strip()\n\n if found!=submitted:\n e = ('doctest failure for input_lines=\"%s\" with '\n 'found_output=\"%s\" and submitted output=\"%s\"' %\n (input_lines, found, submitted) )\n raise RuntimeError(e)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_pure_python(self, content):\n output = []\n savefig = False # keep up with this to clear figure\n multiline = False # to handle line continuation\n fmtin = self.promptin\n\n for lineno, line in enumerate(content):\n\n line_stripped = line.strip()\n\n if not len(line):\n output.append(line) # preserve empty lines in output\n continue\n\n # handle decorators\n if line_stripped.startswith('@'):\n output.extend([line])\n if 'savefig' in line:\n savefig = True # and need to clear figure\n continue\n\n # handle comments\n if line_stripped.startswith('#'):\n output.extend([line])\n continue\n\n # deal with multilines\n if not multiline: # not currently on a multiline\n\n if line_stripped.endswith('\\\\'): # now we are\n multiline = True\n cont_len = len(str(lineno)) + 2\n line_to_process = line.strip('\\\\')\n output.extend([u\"%s %s\" % (fmtin%lineno,line)])\n continue\n else: # no we're still not\n line_to_process = line.strip('\\\\')\n else: # we are currently on a multiline\n line_to_process += line.strip('\\\\')\n if line_stripped.endswith('\\\\'): # and we still are\n continuation = '.' * cont_len\n output.extend([(u' %s: '+line_stripped) % continuation])\n continue\n # else go ahead and run this multiline then carry on\n\n # get output of line\n self.process_input_line(unicode(line_to_process.strip()),\n store_history=False)\n out_line = self.cout.getvalue()\n self.clear_cout()\n\n # clear current figure if plotted\n if savefig:\n self.ensure_pyplot()\n self.process_input_line('plt.clf()', store_history=False)\n self.clear_cout()\n savefig = False\n\n # line numbers don't actually matter, they're replaced later\n if not multiline:\n in_line = u\"%s %s\" % (fmtin%lineno,line)\n\n output.extend([in_line])\n else:\n output.extend([(u' %s: '+line_stripped) % continuation])\n multiline = False\n if len(out_line):\n output.extend([out_line])\n output.extend([u''])\n\n return output", "response": "This function processes the pure python code."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process_pure_python2(self, content):\n output = []\n savefig = False # keep up with this to clear figure\n multiline = False # to handle line continuation\n multiline_start = None\n fmtin = self.promptin\n\n ct = 0\n\n # nuke empty lines\n content = [line for line in content if len(line.strip()) > 0]\n\n for lineno, line in enumerate(content):\n\n line_stripped = line.strip()\n if not len(line):\n output.append(line)\n continue\n\n # handle decorators\n if line_stripped.startswith('@'):\n output.extend([line])\n if 'savefig' in line:\n savefig = True # and need to clear figure\n continue\n\n # handle comments\n if line_stripped.startswith('#'):\n output.extend([line])\n continue\n\n continuation = u' %s:'% ''.join(['.']*(len(str(ct))+2))\n if not multiline:\n modified = u\"%s %s\" % (fmtin % ct, line_stripped)\n output.append(modified)\n ct += 1\n try:\n ast.parse(line_stripped)\n output.append(u'')\n except Exception:\n multiline = True\n multiline_start = lineno\n else:\n modified = u'%s %s' % (continuation, line)\n output.append(modified)\n\n try:\n ast.parse('\\n'.join(content[multiline_start:lineno+1]))\n\n if (lineno < len(content) - 1 and\n _count_indent(content[multiline_start]) <\n _count_indent(content[lineno + 1])):\n\n continue\n\n output.extend([continuation, u''])\n multiline = False\n except Exception:\n pass\n\n continue\n\n return output", "response": "This function processes the pure python2 code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef simulate_linear_model(A, x0, v, ts_length):\n A = np.asarray(A)\n n = A.shape[0]\n x = np.empty((n, ts_length))\n x[:, 0] = x0\n for t in range(ts_length-1):\n # x[:, t+1] = A.dot(x[:, t]) + v[:, t]\n for i in range(n):\n x[i, t+1] = v[i, t] # Shock\n for j in range(n):\n x[i, t+1] += A[i, j] * x[j, t] # Dot Product\n return x", "response": "r Simulates a vector linear system of A."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef searchsorted(a, v):\n lo = -1\n hi = len(a)\n while(lo < hi-1):\n m = (lo + hi) // 2\n if v < a[m]:\n hi = m\n else:\n lo = m\n return hi", "response": "Custom version of np. searchsorted."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nusing a jitted version of the maximization routine from SciPy s fminbound method to perform the maximum function call.", "response": "def brent_max(func, a, b, args=(), xtol=1e-5, maxiter=500):\n \"\"\"\n Uses a jitted version of the maximization routine from SciPy's fminbound.\n The algorithm is identical except that it's been switched to maximization\n rather than minimization, and the tests for convergence have been stripped\n out to allow for jit compilation.\n\n Note that the input function `func` must be jitted or the call will fail.\n\n Parameters\n ----------\n func : jitted function\n a : scalar\n Lower bound for search\n b : scalar\n Upper bound for search\n args : tuple, optional\n Extra arguments passed to the objective function.\n maxiter : int, optional\n Maximum number of iterations to perform.\n xtol : float, optional\n Absolute error in solution `xopt` acceptable for convergence.\n\n Returns\n -------\n xf : float\n The maximizer\n fval : float\n The maximum value attained\n info : tuple\n A tuple of the form (status_flag, num_iter). Here status_flag\n indicates whether or not the maximum number of function calls was\n attained. A value of 0 implies that the maximum was not hit.\n The value `num_iter` is the number of function calls.\n\n Example\n -------\n\n ```\n @njit\n def f(x):\n return -(x + 2.0)**2 + 1.0\n\n xf, fval, info = brent_max(f, -2, 2)\n ```\n\n \"\"\"\n if not np.isfinite(a):\n raise ValueError(\"a must be finite.\")\n\n if not np.isfinite(b):\n raise ValueError(\"b must be finite.\")\n\n if not a < b:\n raise ValueError(\"a must be less than b.\")\n\n maxfun = maxiter\n status_flag = 0\n\n sqrt_eps = np.sqrt(2.2e-16)\n golden_mean = 0.5 * (3.0 - np.sqrt(5.0))\n\n fulc = a + golden_mean * (b - a)\n nfc, xf = fulc, fulc\n rat = e = 0.0\n x = xf\n fx = -func(x, *args)\n num = 1\n\n ffulc = fnfc = fx\n xm = 0.5 * (a + b)\n tol1 = sqrt_eps * np.abs(xf) + xtol / 3.0\n tol2 = 2.0 * tol1\n\n while (np.abs(xf - xm) > (tol2 - 0.5 * (b - a))):\n golden = 1\n # Check for parabolic fit\n if np.abs(e) > tol1:\n golden = 0\n r = (xf - nfc) * (fx - ffulc)\n q = (xf - fulc) * (fx - fnfc)\n p = (xf - fulc) * q - (xf - nfc) * r\n q = 2.0 * (q - r)\n if q > 0.0:\n p = -p\n q = np.abs(q)\n r = e\n e = rat\n\n # Check for acceptability of parabola\n if ((np.abs(p) < np.abs(0.5*q*r)) and (p > q*(a - xf)) and\n (p < q * (b - xf))):\n rat = (p + 0.0) / q\n x = xf + rat\n\n if ((x - a) < tol2) or ((b - x) < tol2):\n si = np.sign(xm - xf) + ((xm - xf) == 0)\n rat = tol1 * si\n else: # do a golden section step\n golden = 1\n\n if golden: # Do a golden-section step\n if xf >= xm:\n e = a - xf\n else:\n e = b - xf\n rat = golden_mean*e\n\n if rat == 0:\n si = np.sign(rat) + 1\n else:\n si = np.sign(rat)\n\n x = xf + si * np.maximum(np.abs(rat), tol1)\n fu = -func(x, *args)\n num += 1\n\n if fu <= fx:\n if x >= xf:\n a = xf\n else:\n b = xf\n fulc, ffulc = nfc, fnfc\n nfc, fnfc = xf, fx\n xf, fx = x, fu\n else:\n if x < xf:\n a = x\n else:\n b = x\n if (fu <= fnfc) or (nfc == xf):\n fulc, ffulc = nfc, fnfc\n nfc, fnfc = x, fu\n elif (fu <= ffulc) or (fulc == xf) or (fulc == nfc):\n fulc, ffulc = x, fu\n\n xm = 0.5 * (a + b)\n tol1 = sqrt_eps * np.abs(xf) + xtol / 3.0\n tol2 = 2.0 * tol1\n\n if num >= maxfun:\n status_flag = 1\n break\n\n fval = -fx\n info = status_flag, num\n\n return xf, fval, info"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mean(self):\n \"mean\"\n n, a, b = self.n, self.a, self.b\n return n * a / (a + b)", "response": "get the mean value of the record set"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef skew(self):\n \"skewness\"\n n, a, b = self.n, self.a, self.b\n t1 = (a+b+2*n) * (b - a) / (a+b+2)\n t2 = sqrt((1+a+b) / (n*a*b * (n+a+b)))\n return t1 * t2", "response": "skewness of the current set of entries"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _equilibrium_payoffs_abreu_sannikov(rpg, tol=1e-12, max_iter=500,\n u_init=np.zeros(2)):\n \"\"\"\n Using 'abreu_sannikov' algorithm to compute the set of payoff pairs\n of all pure-strategy subgame-perfect equilibria with public randomization\n for any repeated two-player games with perfect monitoring and\n discounting, following Abreu and Sannikov (2014).\n\n Parameters\n ----------\n rpg : RepeatedGame\n Two player repeated game.\n\n tol : scalar(float), optional(default=1e-12)\n Tolerance for convergence checking.\n\n max_iter : scalar(int), optional(default=500)\n Maximum number of iterations.\n\n u_init : ndarray(float, ndim=1), optional(default=np.zeros(2))\n The initial guess of threat points.\n\n Returns\n -------\n hull : scipy.spatial.ConvexHull\n The convex hull of equilibrium payoff pairs.\n\n References\n ----------\n .. [1] Abreu, Dilip, and Yuliy Sannikov. \"An algorithm for\n two\u2010player repeated games with perfect monitoring.\" Theoretical\n Economics 9.2 (2014): 313-338.\n \"\"\"\n sg, delta = rpg.sg, rpg.delta\n\n if sg.N != 2:\n msg = \"this algorithm only applies to repeated two-player games.\"\n raise NotImplementedError(msg)\n\n best_dev_gains = _best_dev_gains(rpg)\n IC = np.empty(2)\n action_profile_payoff = np.empty(2)\n # auxiliary array for checking if payoff is inside the convex hull\n # first two entries for payoff point, and the last entry is 1.\n extended_payoff = np.ones(3)\n # array to store new points of C in each intersection\n # at most 4 new points will be generated\n new_pts = np.empty((4, 2))\n # array to store the points of W\n # the length of v is limited by |A1|*|A2|*4\n W_new = np.empty((np.prod(sg.nums_actions)*4, 2))\n W_old = np.empty((np.prod(sg.nums_actions)*4, 2))\n # count the new points generated in each iteration\n n_new_pt = 0\n\n # copy the threat points\n u = np.copy(u_init)\n\n # initialization\n payoff_pts = \\\n sg.payoff_profile_array.reshape(np.prod(sg.nums_actions), 2)\n W_new[:np.prod(sg.nums_actions)] = payoff_pts\n n_new_pt = np.prod(sg.nums_actions)\n\n n_iter = 0\n while True:\n W_old[:n_new_pt] = W_new[:n_new_pt]\n n_old_pt = n_new_pt\n hull = ConvexHull(W_old[:n_old_pt])\n\n W_new, n_new_pt = \\\n _R(delta, sg.nums_actions, sg.payoff_arrays,\n best_dev_gains, hull.points, hull.vertices,\n hull.equations, u, IC, action_profile_payoff,\n extended_payoff, new_pts, W_new)\n\n n_iter += 1\n if n_iter >= max_iter:\n break\n\n # check convergence\n if n_new_pt == n_old_pt:\n if np.linalg.norm(W_new[:n_new_pt] - W_old[:n_new_pt]) < tol:\n break\n\n # update threat points\n _update_u(u, W_new[:n_new_pt])\n\n hull = ConvexHull(W_new[:n_new_pt])\n\n return hull", "response": "This function computes the set of payoff pairs for all pure - strategy subgame - perfect equilibria with public randomization and public randomization of the two - player repeated game with perfect monitoring and public randomization of the two - player repeated game with public randomization."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the normalized payoff gains from deviating from the current action to the best response for each player.", "response": "def _best_dev_gains(rpg):\n \"\"\"\n Calculate the normalized payoff gains from deviating from the current\n action to the best response for each player.\n\n Parameters\n ----------\n rpg : RepeatedGame\n Two player repeated game.\n\n Returns\n -------\n best_dev_gains : tuple(ndarray(float, ndim=2))\n The normalized best deviation payoff gain arrays.\n best_dev_gains[i][ai, a-i] is normalized payoff gain\n player i can get if originally players are choosing\n ai and a-i, and player i deviates to the best response action.\n \"\"\"\n sg, delta = rpg.sg, rpg.delta\n\n best_dev_gains = ((1-delta)/delta *\n (np.max(sg.payoff_arrays[i], 0) - sg.payoff_arrays[i])\n for i in range(2))\n\n return tuple(best_dev_gains)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupdate the payoff convex hull by iterating all action pairs. Using the R operator proposed by Abreu and Sannikov 2014. Parameters ---------- delta : scalar(float) The common discount rate at which all players discount the future. nums_actions : tuple(int) Tuple of the numbers of actions, one for each player. payoff_arrays : tuple(ndarray(float, ndim=2)) Tuple of the payoff arrays, one for each player. best_dev_gains : tuple(ndarray(float, ndim=2)) Tuple of the normalized best deviation payoff gain arrays. best_dev_gains[i][ai, a-i] is payoff gain player i can get if originally players are choosing ai and a-i, and player i deviates to the best response action. points : ndarray(float, ndim=2) Coordinates of the points in the W, which construct a feasible payoff convex hull. vertices : ndarray(float, ndim=1) Indices of points forming the vertices of the convex hull, which are in counterclockwise order. equations : ndarray(float, ndim=2) [normal, offset] forming the hyperplane equation of the facet u : ndarray(float, ndim=1) The threat points. IC : ndarray(float, ndim=1) The minimum IC continuation values. action_profile_payoff : ndarray(float, ndim=1) Array of payoff for one action profile. extended_payoff : ndarray(float, ndim=2) The array [payoff0, payoff1, 1] for checking if [payoff0, payoff1] is in the feasible payoff convex hull. new_pts : ndarray(float, ndim=1) The 4 by 2 array for storing the generated potential extreme points of one action profile. One action profile can only generate at most 4 points. W_new : ndarray(float, ndim=2) Array for storing the coordinates of the generated potential extreme points that construct a new feasible payoff convex hull. tol: scalar(float), optional(default=1e-10) The tolerance for checking if two values are equal. Returns ------- W_new : ndarray(float, ndim=2) The coordinates of the generated potential extreme points that construct a new feasible payoff convex hull. n_new_pt : scalar(int) The number of points in W_new that construct the feasible payoff convex hull.", "response": "def _R(delta, nums_actions, payoff_arrays, best_dev_gains, points,\n vertices, equations, u, IC, action_profile_payoff,\n extended_payoff, new_pts, W_new, tol=1e-10):\n \"\"\"\n Updating the payoff convex hull by iterating all action pairs.\n Using the R operator proposed by Abreu and Sannikov 2014.\n\n Parameters\n ----------\n delta : scalar(float)\n The common discount rate at which all players discount\n the future.\n\n nums_actions : tuple(int)\n Tuple of the numbers of actions, one for each player.\n\n payoff_arrays : tuple(ndarray(float, ndim=2))\n Tuple of the payoff arrays, one for each player.\n\n best_dev_gains : tuple(ndarray(float, ndim=2))\n Tuple of the normalized best deviation payoff gain arrays.\n best_dev_gains[i][ai, a-i] is payoff gain player i\n can get if originally players are choosing ai and a-i,\n and player i deviates to the best response action.\n\n points : ndarray(float, ndim=2)\n Coordinates of the points in the W, which construct a\n feasible payoff convex hull.\n\n vertices : ndarray(float, ndim=1)\n Indices of points forming the vertices of the convex hull,\n which are in counterclockwise order.\n\n equations : ndarray(float, ndim=2)\n [normal, offset] forming the hyperplane equation of the facet\n\n u : ndarray(float, ndim=1)\n The threat points.\n\n IC : ndarray(float, ndim=1)\n The minimum IC continuation values.\n\n action_profile_payoff : ndarray(float, ndim=1)\n Array of payoff for one action profile.\n\n extended_payoff : ndarray(float, ndim=2)\n The array [payoff0, payoff1, 1] for checking if\n [payoff0, payoff1] is in the feasible payoff convex hull.\n\n new_pts : ndarray(float, ndim=1)\n The 4 by 2 array for storing the generated potential\n extreme points of one action profile. One action profile\n can only generate at most 4 points.\n\n W_new : ndarray(float, ndim=2)\n Array for storing the coordinates of the generated potential\n extreme points that construct a new feasible payoff convex hull.\n\n tol: scalar(float), optional(default=1e-10)\n The tolerance for checking if two values are equal.\n\n Returns\n -------\n W_new : ndarray(float, ndim=2)\n The coordinates of the generated potential extreme points\n that construct a new feasible payoff convex hull.\n\n n_new_pt : scalar(int)\n The number of points in W_new that construct the feasible\n payoff convex hull.\n \"\"\"\n n_new_pt = 0\n for a0 in range(nums_actions[0]):\n for a1 in range(nums_actions[1]):\n action_profile_payoff[0] = payoff_arrays[0][a0, a1]\n action_profile_payoff[1] = payoff_arrays[1][a1, a0]\n IC[0] = u[0] + best_dev_gains[0][a0, a1]\n IC[1] = u[1] + best_dev_gains[1][a1, a0]\n\n # check if payoff is larger than IC\n if (action_profile_payoff >= IC).all():\n # check if payoff is inside the convex hull\n extended_payoff[:2] = action_profile_payoff\n if (np.dot(equations, extended_payoff) <= tol).all():\n W_new[n_new_pt] = action_profile_payoff\n n_new_pt += 1\n continue\n\n new_pts, n = _find_C(new_pts, points, vertices, equations,\n extended_payoff, IC, tol)\n\n for i in range(n):\n W_new[n_new_pt] = \\\n delta * new_pts[i] + (1-delta) * action_profile_payoff\n n_new_pt += 1\n\n return W_new, n_new_pt"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _find_C(C, points, vertices, equations, extended_payoff, IC, tol):\n n = 0\n weights = np.empty(2)\n # vertices is ordered counterclockwise\n for i in range(len(vertices)-1):\n n = _intersect(C, n, weights, IC,\n points[vertices[i]],\n points[vertices[i+1]], tol)\n\n n = _intersect(C, n, weights, IC,\n points[vertices[-1]],\n points[vertices[0]], tol)\n\n # check the case that IC is an interior point of the convex hull\n extended_payoff[:2] = IC\n if (np.dot(equations, extended_payoff) <= tol).all():\n C[n, :] = IC\n n += 1\n\n return C, n", "response": "This function finds all the intersection points between the current convex hull and the two IC constraints."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind the intersection points of a half-closed simplex (pt0, pt1] and IC constraints. Parameters ---------- C : ndarray(float, ndim=2) The 4 by 2 array for storing the generated points of one action profile. One action profile can only generate at most 4 points. n : scalar(int) The number of intersection points that have been found. weights : ndarray(float, ndim=1) The size 2 array for storing the weights when calculate the intersection point of simplex and IC constraints. IC : ndarray(float, ndim=1) The minimum IC continuation values. pt0 : ndarray(float, ndim=1) Coordinates of the starting point of the simplex. pt1 : ndarray(float, ndim=1) Coordinates of the ending point of the simplex. tol : scalar(float) The tolerance for checking if two values are equal. Returns ------- n : scalar(int) The updated number of found intersection points.", "response": "def _intersect(C, n, weights, IC, pt0, pt1, tol):\n \"\"\"\n Find the intersection points of a half-closed simplex\n (pt0, pt1] and IC constraints.\n\n Parameters\n ----------\n C : ndarray(float, ndim=2)\n The 4 by 2 array for storing the generated points of\n one action profile. One action profile can only\n generate at most 4 points.\n\n n : scalar(int)\n The number of intersection points that have been found.\n\n weights : ndarray(float, ndim=1)\n The size 2 array for storing the weights when calculate\n the intersection point of simplex and IC constraints.\n\n IC : ndarray(float, ndim=1)\n The minimum IC continuation values.\n\n pt0 : ndarray(float, ndim=1)\n Coordinates of the starting point of the simplex.\n\n pt1 : ndarray(float, ndim=1)\n Coordinates of the ending point of the simplex.\n\n tol : scalar(float)\n The tolerance for checking if two values are equal.\n\n Returns\n -------\n n : scalar(int)\n The updated number of found intersection points.\n \"\"\"\n for i in range(2):\n if (abs(pt0[i] - pt1[i]) < tol):\n if (abs(pt1[i] - IC[i]) < tol):\n x = pt1[1-i]\n else:\n continue\n else:\n weights[i] = (pt0[i] - IC[i]) / (pt0[i] - pt1[i])\n # pt0 is not included to avoid duplication\n # weights in (0, 1]\n if (0 < weights[i] <= 1):\n x = (1 - weights[i]) * pt0[1-i] + weights[i] * pt1[1-i]\n else:\n continue\n # x has to be strictly higher than IC[1-j]\n # if it is equal, then it means IC is one of the vertex\n # it will be added to C in below\n if x - IC[1-i] > tol:\n C[n, i] = IC[i]\n C[n, 1-i] = x\n n += 1\n elif x - IC[1-i] > -tol:\n # to avoid duplication when IC is a vertex\n break\n\n return n"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the threat points u by the minimum of feasible payoffs.", "response": "def _update_u(u, W):\n \"\"\"\n Update the threat points if it not feasible in the new W,\n by the minimum of new feasible payoffs.\n\n Parameters\n ----------\n u : ndarray(float, ndim=1)\n The threat points.\n\n W : ndarray(float, ndim=1)\n The points that construct the feasible payoff convex hull.\n\n Returns\n -------\n u : ndarray(float, ndim=1)\n The updated threat points.\n \"\"\"\n for i in range(2):\n W_min = W[:, i].min()\n if u[i] < W_min:\n u[i] = W_min\n\n return u"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef equilibrium_payoffs(self, method=None, options=None):\n if method is None:\n method = 'abreu_sannikov'\n\n if options is None:\n options = {}\n\n if method in ('abreu_sannikov', 'AS'):\n return _equilibrium_payoffs_abreu_sannikov(self, **options)\n else:\n msg = f\"method {method} not supported.\"\n raise NotImplementedError(msg)", "response": "Compute the set of payoff pairs of all pure - strategy subgame - perfect and public randomization for any repeated two - player games\n with perfect monitoring and discounting."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef random_game(nums_actions, random_state=None):\n N = len(nums_actions)\n if N == 0:\n raise ValueError('nums_actions must be non-empty')\n\n random_state = check_random_state(random_state)\n players = [\n Player(random_state.random_sample(nums_actions[i:]+nums_actions[:i]))\n for i in range(N)\n ]\n g = NormalFormGame(players)\n return g", "response": "Returns a random NormalFormGame instance where the payoffs are drawn independently from the uniform distribution on [ 0 1 )."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef covariance_game(nums_actions, rho, random_state=None):\n N = len(nums_actions)\n if N <= 1:\n raise ValueError('length of nums_actions must be at least 2')\n if not (-1 / (N - 1) <= rho <= 1):\n lb = '-1' if N == 2 else '-1/{0}'.format(N-1)\n raise ValueError('rho must be in [{0}, 1]'.format(lb))\n\n mean = np.zeros(N)\n cov = np.empty((N, N))\n cov.fill(rho)\n cov[range(N), range(N)] = 1\n\n random_state = check_random_state(random_state)\n payoff_profile_array = \\\n random_state.multivariate_normal(mean, cov, nums_actions)\n g = NormalFormGame(payoff_profile_array)\n return g", "response": "Returns a random NormalFormGame instance where the payoff profiles are drawn independently from the standard multi - normal with the covariance of any pair of payoffs equal to rho."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef random_pure_actions(nums_actions, random_state=None):\n random_state = check_random_state(random_state)\n action_profile = tuple(\n [random_state.randint(num_actions) for num_actions in nums_actions]\n )\n return action_profile", "response": "Returns a tuple of random pure actions ( integers."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef random_mixed_actions(nums_actions, random_state=None):\n random_state = check_random_state(random_state)\n action_profile = tuple(\n [probvec(1, num_actions, random_state).ravel()\n for num_actions in nums_actions]\n )\n return action_profile", "response": "Returns a tuple of random mixed actions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fetch_nb_dependencies(files, repo=REPO, raw=RAW, branch=BRANCH, folder=FOLDER, overwrite=False, verbose=True):\n import requests\n\n #-Generate Common Data Structure-#\n if type(files) == list:\n files = {\"\" : files}\n\n status = []\n\n #-Obtain each requested file-#\n for directory in files.keys():\n if directory != \"\":\n if verbose: print(\"Parsing directory: %s\"%directory)\n for fl in files[directory]:\n if directory != \"\":\n fl = directory+\"/\"+fl\n #-Check for Local Copy of File (Default Behaviour is to Skip)-#\n if not overwrite:\n if os.path.isfile(fl):\n if verbose: print(\n \"A file named %s already exists in the specified directory ... skipping download.\" % fl)\n status.append(False)\n continue\n else:\n if verbose: print(\"Overwriting file %s ...\"%fl)\n if verbose: print(\"Fetching file: %s\"%fl)\n #-Get file in OS agnostic way using requests-#\n url = \"/\".join([repo, raw, branch, folder, fl])\n r = requests.get(url)\n with open(fl, \"wb\") as fl:\n fl.write(r.content)\n status.append(True)\n\n return status", "response": "This function fetches raw files from QuantEcon. notebooks or other Github repository and returns a list of files in the specified branch."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _results(r):\n x, funcalls, iterations, flag = r\n return results(x, funcalls, iterations, flag == 0)", "response": "r Select from a tuple of root funccalls iterations and flag"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef newton(func, x0, fprime, args=(), tol=1.48e-8, maxiter=50,\n disp=True):\n \"\"\"\n Find a zero from the Newton-Raphson method using the jitted version of\n Scipy's newton for scalars. Note that this does not provide an alternative\n method such as secant. Thus, it is important that `fprime` can be provided.\n\n Note that `func` and `fprime` must be jitted via Numba.\n They are recommended to be `njit` for performance.\n\n Parameters\n ----------\n func : callable and jitted\n The function whose zero is wanted. It must be a function of a\n single variable of the form f(x,a,b,c...), where a,b,c... are extra\n arguments that can be passed in the `args` parameter.\n x0 : float\n An initial estimate of the zero that should be somewhere near the\n actual zero.\n fprime : callable and jitted\n The derivative of the function (when available and convenient).\n args : tuple, optional(default=())\n Extra arguments to be used in the function call.\n tol : float, optional(default=1.48e-8)\n The allowable error of the zero value.\n maxiter : int, optional(default=50)\n Maximum number of iterations.\n disp : bool, optional(default=True)\n If True, raise a RuntimeError if the algorithm didn't converge\n\n Returns\n -------\n results : namedtuple\n A namedtuple containing the following items:\n ::\n\n root - Estimated location where function is zero.\n function_calls - Number of times the function was called.\n iterations - Number of iterations needed to find the root.\n converged - True if the routine converged\n\n \"\"\"\n\n if tol <= 0:\n raise ValueError(\"tol is too small <= 0\")\n if maxiter < 1:\n raise ValueError(\"maxiter must be greater than 0\")\n\n # Convert to float (don't use float(x0); this works also for complex x0)\n p0 = 1.0 * x0\n funcalls = 0\n status = _ECONVERR\n\n # Newton-Raphson method\n for itr in range(maxiter):\n # first evaluate fval\n fval = func(p0, *args)\n funcalls += 1\n # If fval is 0, a root has been found, then terminate\n if fval == 0:\n status = _ECONVERGED\n p = p0\n itr -= 1\n break\n fder = fprime(p0, *args)\n funcalls += 1\n # derivative is zero, not converged\n if fder == 0:\n p = p0\n break\n newton_step = fval / fder\n # Newton step\n p = p0 - newton_step\n if abs(p - p0) < tol:\n status = _ECONVERGED\n break\n p0 = p\n\n if disp and status == _ECONVERR:\n msg = \"Failed to converge\"\n raise RuntimeError(msg)\n\n return _results((p, funcalls, itr + 1, status))", "response": "This function is used to find a zero from the Newton - Raphson method using the jitted version of the Scala algorithm."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef newton_secant(func, x0, args=(), tol=1.48e-8, maxiter=50,\n disp=True):\n \"\"\"\n Find a zero from the secant method using the jitted version of\n Scipy's secant method.\n\n Note that `func` must be jitted via Numba.\n\n Parameters\n ----------\n func : callable and jitted\n The function whose zero is wanted. It must be a function of a\n single variable of the form f(x,a,b,c...), where a,b,c... are extra\n arguments that can be passed in the `args` parameter.\n x0 : float\n An initial estimate of the zero that should be somewhere near the\n actual zero.\n args : tuple, optional(default=())\n Extra arguments to be used in the function call.\n tol : float, optional(default=1.48e-8)\n The allowable error of the zero value.\n maxiter : int, optional(default=50)\n Maximum number of iterations.\n disp : bool, optional(default=True)\n If True, raise a RuntimeError if the algorithm didn't converge.\n\n Returns\n -------\n results : namedtuple\n A namedtuple containing the following items:\n ::\n\n root - Estimated location where function is zero.\n function_calls - Number of times the function was called.\n iterations - Number of iterations needed to find the root.\n converged - True if the routine converged\n \"\"\"\n\n if tol <= 0:\n raise ValueError(\"tol is too small <= 0\")\n if maxiter < 1:\n raise ValueError(\"maxiter must be greater than 0\")\n\n # Convert to float (don't use float(x0); this works also for complex x0)\n p0 = 1.0 * x0\n funcalls = 0\n status = _ECONVERR\n\n # Secant method\n if x0 >= 0:\n p1 = x0 * (1 + 1e-4) + 1e-4\n else:\n p1 = x0 * (1 + 1e-4) - 1e-4\n q0 = func(p0, *args)\n funcalls += 1\n q1 = func(p1, *args)\n funcalls += 1\n for itr in range(maxiter):\n if q1 == q0:\n p = (p1 + p0) / 2.0\n status = _ECONVERGED\n break\n else:\n p = p1 - q1 * (p1 - p0) / (q1 - q0)\n if np.abs(p - p1) < tol:\n status = _ECONVERGED\n break\n p0 = p1\n q0 = q1\n p1 = p\n q1 = func(p1, *args)\n funcalls += 1\n\n if disp and status == _ECONVERR:\n msg = \"Failed to converge\"\n raise RuntimeError(msg)\n\n return _results((p, funcalls, itr + 1, status))", "response": "This function returns a newton version of the current state of the newton s secant method."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding root of a function within an interval adapted from Scipy's bisect. Basic bisection routine to find a zero of the function `f` between the arguments `a` and `b`. `f(a)` and `f(b)` cannot have the same signs. `f` must be jitted via numba. Parameters ---------- f : jitted and callable Python function returning a number. `f` must be continuous. a : number One end of the bracketing interval [a,b]. b : number The other end of the bracketing interval [a,b]. args : tuple, optional(default=()) Extra arguments to be used in the function call. xtol : number, optional(default=2e-12) The computed root ``x0`` will satisfy ``np.allclose(x, x0, atol=xtol, rtol=rtol)``, where ``x`` is the exact root. The parameter must be nonnegative. rtol : number, optional(default=4*np.finfo(float).eps) The computed root ``x0`` will satisfy ``np.allclose(x, x0, atol=xtol, rtol=rtol)``, where ``x`` is the exact root. maxiter : number, optional(default=100) Maximum number of iterations. disp : bool, optional(default=True) If True, raise a RuntimeError if the algorithm didn't converge. Returns ------- results : namedtuple", "response": "def bisect(f, a, b, args=(), xtol=_xtol,\n rtol=_rtol, maxiter=_iter, disp=True):\n \"\"\"\n Find root of a function within an interval adapted from Scipy's bisect.\n\n Basic bisection routine to find a zero of the function `f` between the\n arguments `a` and `b`. `f(a)` and `f(b)` cannot have the same signs.\n\n `f` must be jitted via numba.\n\n Parameters\n ----------\n f : jitted and callable\n Python function returning a number. `f` must be continuous.\n a : number\n One end of the bracketing interval [a,b].\n b : number\n The other end of the bracketing interval [a,b].\n args : tuple, optional(default=())\n Extra arguments to be used in the function call.\n xtol : number, optional(default=2e-12)\n The computed root ``x0`` will satisfy ``np.allclose(x, x0,\n atol=xtol, rtol=rtol)``, where ``x`` is the exact root. The\n parameter must be nonnegative.\n rtol : number, optional(default=4*np.finfo(float).eps)\n The computed root ``x0`` will satisfy ``np.allclose(x, x0,\n atol=xtol, rtol=rtol)``, where ``x`` is the exact root.\n maxiter : number, optional(default=100)\n Maximum number of iterations.\n disp : bool, optional(default=True)\n If True, raise a RuntimeError if the algorithm didn't converge.\n\n Returns\n -------\n results : namedtuple\n\n \"\"\"\n\n if xtol <= 0:\n raise ValueError(\"xtol is too small (<= 0)\")\n\n if maxiter < 1:\n raise ValueError(\"maxiter must be greater than 0\")\n\n # Convert to float\n xa = a * 1.0\n xb = b * 1.0\n\n fa = f(xa, *args)\n fb = f(xb, *args)\n funcalls = 2\n root, status = _bisect_interval(xa, xb, fa, fb)\n\n # Check for sign error and early termination\n if status == _ECONVERGED:\n itr = 0\n else:\n # Perform bisection\n dm = xb - xa\n for itr in range(maxiter):\n dm *= 0.5\n xm = xa + dm\n fm = f(xm, *args)\n funcalls += 1\n\n if fm * fa >= 0:\n xa = xm\n\n if fm == 0 or abs(dm) < xtol + rtol * abs(xm):\n root = xm\n status = _ECONVERGED\n itr += 1\n break\n\n if disp and status == _ECONVERR:\n raise RuntimeError(\"Failed to converge\")\n\n return _results((root, funcalls, itr, status))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef brentq(f, a, b, args=(), xtol=_xtol,\n rtol=_rtol, maxiter=_iter, disp=True):\n \"\"\"\n Find a root of a function in a bracketing interval using Brent's method\n adapted from Scipy's brentq.\n\n Uses the classic Brent's method to find a zero of the function `f` on\n the sign changing interval [a , b].\n\n `f` must be jitted via numba.\n\n Parameters\n ----------\n f : jitted and callable\n Python function returning a number. `f` must be continuous.\n a : number\n One end of the bracketing interval [a,b].\n b : number\n The other end of the bracketing interval [a,b].\n args : tuple, optional(default=())\n Extra arguments to be used in the function call.\n xtol : number, optional(default=2e-12)\n The computed root ``x0`` will satisfy ``np.allclose(x, x0,\n atol=xtol, rtol=rtol)``, where ``x`` is the exact root. The\n parameter must be nonnegative.\n rtol : number, optional(default=4*np.finfo(float).eps)\n The computed root ``x0`` will satisfy ``np.allclose(x, x0,\n atol=xtol, rtol=rtol)``, where ``x`` is the exact root.\n maxiter : number, optional(default=100)\n Maximum number of iterations.\n disp : bool, optional(default=True)\n If True, raise a RuntimeError if the algorithm didn't converge.\n\n Returns\n -------\n results : namedtuple\n\n \"\"\"\n if xtol <= 0:\n raise ValueError(\"xtol is too small (<= 0)\")\n if maxiter < 1:\n raise ValueError(\"maxiter must be greater than 0\")\n\n # Convert to float\n xpre = a * 1.0\n xcur = b * 1.0\n\n fpre = f(xpre, *args)\n fcur = f(xcur, *args)\n funcalls = 2\n\n root, status = _bisect_interval(xpre, xcur, fpre, fcur)\n\n # Check for sign error and early termination\n if status == _ECONVERGED:\n itr = 0\n else:\n # Perform Brent's method\n for itr in range(maxiter):\n\n if fpre * fcur < 0:\n xblk = xpre\n fblk = fpre\n spre = scur = xcur - xpre\n if abs(fblk) < abs(fcur):\n xpre = xcur\n xcur = xblk\n xblk = xpre\n\n fpre = fcur\n fcur = fblk\n fblk = fpre\n\n delta = (xtol + rtol * abs(xcur)) / 2\n sbis = (xblk - xcur) / 2\n\n # Root found\n if fcur == 0 or abs(sbis) < delta:\n status = _ECONVERGED\n root = xcur\n itr += 1\n break\n\n if abs(spre) > delta and abs(fcur) < abs(fpre):\n if xpre == xblk:\n # interpolate\n stry = -fcur * (xcur - xpre) / (fcur - fpre)\n else:\n # extrapolate\n dpre = (fpre - fcur) / (xpre - xcur)\n dblk = (fblk - fcur) / (xblk - xcur)\n stry = -fcur * (fblk * dblk - fpre * dpre) / \\\n (dblk * dpre * (fblk - fpre))\n\n if (2 * abs(stry) < min(abs(spre), 3 * abs(sbis) - delta)):\n # good short step\n spre = scur\n scur = stry\n else:\n # bisect\n spre = sbis\n scur = sbis\n else:\n # bisect\n spre = sbis\n scur = sbis\n\n xpre = xcur\n fpre = fcur\n if (abs(scur) > delta):\n xcur += scur\n else:\n xcur += (delta if sbis > 0 else -delta)\n fcur = f(xcur, *args)\n funcalls += 1\n\n if disp and status == _ECONVERR:\n raise RuntimeError(\"Failed to converge\")\n\n return _results((root, funcalls, itr, status))", "response": "This function returns a root of a function in a bracketing interval using the classic Brent s method."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _compute_fixed_point_ig(T, v, max_iter, verbose, print_skip, is_approx_fp,\n *args, **kwargs):\n \"\"\"\n Implement the imitation game algorithm by McLennan and Tourky (2006)\n for computing an approximate fixed point of `T`.\n\n Parameters\n ----------\n is_approx_fp : callable\n A callable with signature `is_approx_fp(v)` which determines\n whether `v` is an approximate fixed point with a bool return\n value (i.e., True or False)\n\n For the other parameters, see Parameters in compute_fixed_point.\n\n Returns\n -------\n x_new : scalar(float) or ndarray(float)\n Approximate fixed point.\n\n converged : bool\n Whether the routine has converged.\n\n iterate : scalar(int)\n Number of iterations.\n\n \"\"\"\n if verbose == 2:\n start_time = time.time()\n _print_after_skip(print_skip, it=None)\n\n x_new = v\n y_new = T(x_new, *args, **kwargs)\n iterate = 1\n converged = is_approx_fp(x_new)\n\n if converged or iterate >= max_iter:\n if verbose == 2:\n error = np.max(np.abs(y_new - x_new))\n etime = time.time() - start_time\n print_skip = 1\n _print_after_skip(print_skip, iterate, error, etime)\n if verbose >= 1:\n if not converged:\n warnings.warn(_non_convergence_msg, RuntimeWarning)\n elif verbose == 2:\n print(_convergence_msg.format(iterate=iterate))\n return x_new, converged, iterate\n\n if verbose == 2:\n error = np.max(np.abs(y_new - x_new))\n etime = time.time() - start_time\n _print_after_skip(print_skip, iterate, error, etime)\n\n # Length of the arrays to store the computed sequences of x and y.\n # If exceeded, reset to min(max_iter, buff_size*2).\n buff_size = 2**8\n buff_size = min(max_iter, buff_size)\n\n shape = (buff_size,) + np.asarray(x_new).shape\n X, Y = np.empty(shape), np.empty(shape)\n X[0], Y[0] = x_new, y_new\n x_new = Y[0]\n\n tableaux = tuple(np.empty((buff_size, buff_size*2+1)) for i in range(2))\n bases = tuple(np.empty(buff_size, dtype=int) for i in range(2))\n max_piv = 10**6 # Max number of pivoting steps in _lemke_howson_tbl\n\n while True:\n y_new = T(x_new, *args, **kwargs)\n iterate += 1\n converged = is_approx_fp(x_new)\n\n if converged or iterate >= max_iter:\n break\n\n if verbose == 2:\n error = np.max(np.abs(y_new - x_new))\n etime = time.time() - start_time\n _print_after_skip(print_skip, iterate, error, etime)\n\n try:\n X[iterate-1] = x_new\n Y[iterate-1] = y_new\n except IndexError:\n buff_size = min(max_iter, buff_size*2)\n shape = (buff_size,) + X.shape[1:]\n X_tmp, Y_tmp = X, Y\n X, Y = np.empty(shape), np.empty(shape)\n X[:X_tmp.shape[0]], Y[:Y_tmp.shape[0]] = X_tmp, Y_tmp\n X[iterate-1], Y[iterate-1] = x_new, y_new\n\n tableaux = tuple(np.empty((buff_size, buff_size*2+1))\n for i in range(2))\n bases = tuple(np.empty(buff_size, dtype=int) for i in range(2))\n\n m = iterate\n tableaux_curr = tuple(tableau[:m, :2*m+1] for tableau in tableaux)\n bases_curr = tuple(basis[:m] for basis in bases)\n _initialize_tableaux_ig(X[:m], Y[:m], tableaux_curr, bases_curr)\n converged, num_iter = _lemke_howson_tbl(\n tableaux_curr, bases_curr, init_pivot=m-1, max_iter=max_piv\n )\n _, rho = _get_mixed_actions(tableaux_curr, bases_curr)\n\n if Y.ndim <= 2:\n x_new = rho.dot(Y[:m])\n else:\n shape_Y = Y.shape\n Y_2d = Y.reshape(shape_Y[0], np.prod(shape_Y[1:]))\n x_new = rho.dot(Y_2d[:m]).reshape(shape_Y[1:])\n\n if verbose == 2:\n error = np.max(np.abs(y_new - x_new))\n etime = time.time() - start_time\n print_skip = 1\n _print_after_skip(print_skip, iterate, error, etime)\n if verbose >= 1:\n if not converged:\n warnings.warn(_non_convergence_msg, RuntimeWarning)\n elif verbose == 2:\n print(_convergence_msg.format(iterate=iterate))\n\n return x_new, converged, iterate", "response": "This function computes an approximate fixed point of T."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninitialize the tableaux and base arrays in place for the geometric imitation game as defined by McLennan and Tourky 2006.", "response": "def _initialize_tableaux_ig(X, Y, tableaux, bases):\n \"\"\"\n Given sequences `X` and `Y` of ndarrays, initialize the tableau and\n basis arrays in place for the \"geometric\" imitation game as defined\n in McLennan and Tourky (2006), to be passed to `_lemke_howson_tbl`.\n\n Parameters\n ----------\n X, Y : ndarray(float)\n Arrays of the same shape (m, n).\n\n tableaux : tuple(ndarray(float, ndim=2))\n Tuple of two arrays to be used to store the tableaux, of shape\n (2m, 2m). Modified in place.\n\n bases : tuple(ndarray(int, ndim=1))\n Tuple of two arrays to be used to store the bases, of shape\n (m,). Modified in place.\n\n Returns\n -------\n tableaux : tuple(ndarray(float, ndim=2))\n View to `tableaux`.\n\n bases : tuple(ndarray(int, ndim=1))\n View to `bases`.\n\n \"\"\"\n m = X.shape[0]\n min_ = np.zeros(m)\n\n # Mover\n for i in range(m):\n for j in range(2*m):\n if j == i or j == i + m:\n tableaux[0][i, j] = 1\n else:\n tableaux[0][i, j] = 0\n # Right hand side\n tableaux[0][i, 2*m] = 1\n\n # Imitator\n for i in range(m):\n # Slack variables\n for j in range(m):\n if j == i:\n tableaux[1][i, j] = 1\n else:\n tableaux[1][i, j] = 0\n # Payoff variables\n for j in range(m):\n d = X[i] - Y[j]\n tableaux[1][i, m+j] = _square_sum(d) * (-1)\n if tableaux[1][i, m+j] < min_[j]:\n min_[j] = tableaux[1][i, m+j]\n # Right hand side\n tableaux[1][i, 2*m] = 1\n # Shift the payoff values\n for i in range(m):\n for j in range(m):\n tableaux[1][i, m+j] -= min_[j]\n tableaux[1][i, m+j] += 1\n\n for pl, start in enumerate([m, 0]):\n for i in range(m):\n bases[pl][i] = start + i\n\n return tableaux, bases"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_participants(self, namespace, room):\n for sid, active in six.iteritems(self.rooms[namespace][room].copy()):\n yield sid", "response": "Return an iterable with the active participants in a room."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef connect(self, sid, namespace):\n self.enter_room(sid, namespace, None)\n self.enter_room(sid, namespace, sid)", "response": "Register a client connection to a namespace."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nputting the client in the to - be - disconnected list.", "response": "def pre_disconnect(self, sid, namespace):\n \"\"\"Put the client in the to-be-disconnected list.\n\n This allows the client data structures to be present while the\n disconnect handler is invoked, but still recognize the fact that the\n client is soon going away.\n \"\"\"\n if namespace not in self.pending_disconnect:\n self.pending_disconnect[namespace] = []\n self.pending_disconnect[namespace].append(sid)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef disconnect(self, sid, namespace):\n if namespace not in self.rooms:\n return\n rooms = []\n for room_name, room in six.iteritems(self.rooms[namespace].copy()):\n if sid in room:\n rooms.append(room_name)\n for room in rooms:\n self.leave_room(sid, namespace, room)\n if sid in self.callbacks and namespace in self.callbacks[sid]:\n del self.callbacks[sid][namespace]\n if len(self.callbacks[sid]) == 0:\n del self.callbacks[sid]\n if namespace in self.pending_disconnect and \\\n sid in self.pending_disconnect[namespace]:\n self.pending_disconnect[namespace].remove(sid)\n if len(self.pending_disconnect[namespace]) == 0:\n del self.pending_disconnect[namespace]", "response": "Register a client disconnect from a namespace."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a client to a room.", "response": "def enter_room(self, sid, namespace, room):\n \"\"\"Add a client to a room.\"\"\"\n if namespace not in self.rooms:\n self.rooms[namespace] = {}\n if room not in self.rooms[namespace]:\n self.rooms[namespace][room] = {}\n self.rooms[namespace][room][sid] = True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef leave_room(self, sid, namespace, room):\n try:\n del self.rooms[namespace][room][sid]\n if len(self.rooms[namespace][room]) == 0:\n del self.rooms[namespace][room]\n if len(self.rooms[namespace]) == 0:\n del self.rooms[namespace]\n except KeyError:\n pass", "response": "Remove a client from a room."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef close_room(self, room, namespace):\n try:\n for sid in self.get_participants(namespace, room):\n self.leave_room(sid, namespace, room)\n except KeyError:\n pass", "response": "Remove all participants from a room."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_rooms(self, sid, namespace):\n r = []\n try:\n for room_name, room in six.iteritems(self.rooms[namespace]):\n if room_name is not None and sid in room and room[sid]:\n r.append(room_name)\n except KeyError:\n pass\n return r", "response": "Return the rooms a client is in."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef trigger_callback(self, sid, namespace, id, data):\n callback = None\n try:\n callback = self.callbacks[sid][namespace][id]\n except KeyError:\n # if we get an unknown callback we just ignore it\n self._get_logger().warning('Unknown callback received, ignoring.')\n else:\n del self.callbacks[sid][namespace][id]\n if callback is not None:\n callback(*data)", "response": "Invoke an application callback."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _generate_ack_id(self, sid, namespace, callback):\n namespace = namespace or '/'\n if sid not in self.callbacks:\n self.callbacks[sid] = {}\n if namespace not in self.callbacks[sid]:\n self.callbacks[sid][namespace] = {0: itertools.count(1)}\n id = six.next(self.callbacks[sid][namespace][0])\n self.callbacks[sid][namespace][id] = callback\n return id", "response": "Generate a unique identifier for an ACK packet."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the appropriate logger for this instance.", "response": "def _get_logger(self):\n \"\"\"Get the appropriate logger\n\n Prevents uninitialized servers in write-only mode from failing.\n \"\"\"\n\n if self.logger:\n return self.logger\n elif self.server:\n return self.server.logger\n else:\n return default_logger"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nregisters an event handler for the specified event.", "response": "def on(self, event, handler=None, namespace=None):\n \"\"\"Register an event handler.\n\n :param event: The event name. It can be any string. The event names\n ``'connect'``, ``'message'`` and ``'disconnect'`` are\n reserved and should not be used.\n :param handler: The function that should be invoked to handle the\n event. When this parameter is not given, the method\n acts as a decorator for the handler function.\n :param namespace: The Socket.IO namespace for the event. If this\n argument is omitted the handler is associated with\n the default namespace.\n\n Example usage::\n\n # as a decorator:\n @socket_io.on('connect', namespace='/chat')\n def connect_handler(sid, environ):\n print('Connection request')\n if environ['REMOTE_ADDR'] in blacklisted:\n return False # reject\n\n # as a method:\n def message_handler(sid, msg):\n print('Received message: ', msg)\n eio.send(sid, 'response')\n socket_io.on('message', namespace='/chat', message_handler)\n\n The handler function receives the ``sid`` (session ID) for the\n client as first argument. The ``'connect'`` event handler receives the\n WSGI environment as a second argument, and can return ``False`` to\n reject the connection. The ``'message'`` handler and handlers for\n custom event names receive the message payload as a second argument.\n Any values returned from a message handler will be passed to the\n client's acknowledgement callback function if it exists. The\n ``'disconnect'`` handler does not take a second argument.\n \"\"\"\n namespace = namespace or '/'\n\n def set_handler(handler):\n if namespace not in self.handlers:\n self.handlers[namespace] = {}\n self.handlers[namespace][event] = handler\n return handler\n\n if handler is None:\n return set_handler\n set_handler(handler)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nregister a namespace handler object.", "response": "def register_namespace(self, namespace_handler):\n \"\"\"Register a namespace handler object.\n\n :param namespace_handler: An instance of a :class:`Namespace`\n subclass that handles all the event traffic\n for a namespace.\n \"\"\"\n if not isinstance(namespace_handler, namespace.Namespace):\n raise ValueError('Not a namespace instance')\n if self.is_asyncio_based() != namespace_handler.is_asyncio_based():\n raise ValueError('Not a valid namespace class for this server')\n namespace_handler._set_server(self)\n self.namespace_handlers[namespace_handler.namespace] = \\\n namespace_handler"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nenter a room. This function adds the client to a room. The :func:`emit` and :func:`send` functions can optionally broadcast events to all the clients in a room. :param sid: Session ID of the client. :param room: Room name. If the room does not exist it is created. :param namespace: The Socket.IO namespace for the event. If this argument is omitted the default namespace is used.", "response": "def enter_room(self, sid, room, namespace=None):\n \"\"\"Enter a room.\n\n This function adds the client to a room. The :func:`emit` and\n :func:`send` functions can optionally broadcast events to all the\n clients in a room.\n\n :param sid: Session ID of the client.\n :param room: Room name. If the room does not exist it is created.\n :param namespace: The Socket.IO namespace for the event. If this\n argument is omitted the default namespace is used.\n \"\"\"\n namespace = namespace or '/'\n self.logger.info('%s is entering room %s [%s]', sid, room, namespace)\n self.manager.enter_room(sid, namespace, room)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef leave_room(self, sid, room, namespace=None):\n namespace = namespace or '/'\n self.logger.info('%s is leaving room %s [%s]', sid, room, namespace)\n self.manager.leave_room(sid, namespace, room)", "response": "This function removes the client from a room."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rooms(self, sid, namespace=None):\n namespace = namespace or '/'\n return self.manager.get_rooms(sid, namespace)", "response": "Return the rooms a client is in."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_session(self, sid, namespace=None):\n namespace = namespace or '/'\n eio_session = self.eio.get_session(sid)\n return eio_session.setdefault(namespace, {})", "response": "Return the user session for a client."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save_session(self, sid, session, namespace=None):\n namespace = namespace or '/'\n eio_session = self.eio.get_session(sid)\n eio_session[namespace] = session", "response": "Store the user session for a client."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the user session dictionary for a client with context manager syntax.", "response": "def session(self, sid, namespace=None):\n \"\"\"Return the user session for a client with context manager syntax.\n\n :param sid: The session id of the client.\n\n This is a context manager that returns the user session dictionary for\n the client. Any changes that are made to this dictionary inside the\n context manager block are saved back to the session. Example usage::\n\n @sio.on('connect')\n def on_connect(sid, environ):\n username = authenticate_user(environ)\n if not username:\n return False\n with sio.session(sid) as session:\n session['username'] = username\n\n @sio.on('message')\n def on_message(sid, msg):\n with sio.session(sid) as session:\n print('received message from ', session['username'])\n \"\"\"\n class _session_context_manager(object):\n def __init__(self, server, sid, namespace):\n self.server = server\n self.sid = sid\n self.namespace = namespace\n self.session = None\n\n def __enter__(self):\n self.session = self.server.get_session(sid,\n namespace=namespace)\n return self.session\n\n def __exit__(self, *args):\n self.server.save_session(sid, self.session,\n namespace=namespace)\n\n return _session_context_manager(self, sid, namespace)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef start_background_task(self, target, *args, **kwargs):\n return self.eio.start_background_task(target, *args, **kwargs)", "response": "Start a background task using the appropriate async model."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a message to a client.", "response": "def _emit_internal(self, sid, event, data, namespace=None, id=None):\n \"\"\"Send a message to a client.\"\"\"\n if six.PY2 and not self.binary:\n binary = False # pragma: nocover\n else:\n binary = None\n # tuples are expanded to multiple arguments, everything else is sent\n # as a single argument\n if isinstance(data, tuple):\n data = list(data)\n else:\n data = [data]\n self._send_packet(sid, packet.Packet(packet.EVENT, namespace=namespace,\n data=[event] + data, id=id,\n binary=binary))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a Socket. IO packet to a client.", "response": "def _send_packet(self, sid, pkt):\n \"\"\"Send a Socket.IO packet to a client.\"\"\"\n encoded_packet = pkt.encode()\n if isinstance(encoded_packet, list):\n binary = False\n for ep in encoded_packet:\n self.eio.send(sid, ep, binary=binary)\n binary = True\n else:\n self.eio.send(sid, encoded_packet, binary=False)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _handle_connect(self, sid, namespace):\n namespace = namespace or '/'\n self.manager.connect(sid, namespace)\n if self.always_connect:\n self._send_packet(sid, packet.Packet(packet.CONNECT,\n namespace=namespace))\n fail_reason = None\n try:\n success = self._trigger_event('connect', namespace, sid,\n self.environ[sid])\n except exceptions.ConnectionRefusedError as exc:\n fail_reason = exc.error_args\n success = False\n\n if success is False:\n if self.always_connect:\n self.manager.pre_disconnect(sid, namespace)\n self._send_packet(sid, packet.Packet(\n packet.DISCONNECT, data=fail_reason, namespace=namespace))\n self.manager.disconnect(sid, namespace)\n if not self.always_connect:\n self._send_packet(sid, packet.Packet(\n packet.ERROR, data=fail_reason, namespace=namespace))\n if sid in self.environ: # pragma: no cover\n del self.environ[sid]\n return False\n elif not self.always_connect:\n self._send_packet(sid, packet.Packet(packet.CONNECT,\n namespace=namespace))", "response": "Handle a client connection request."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _handle_disconnect(self, sid, namespace):\n namespace = namespace or '/'\n if namespace == '/':\n namespace_list = list(self.manager.get_namespaces())\n else:\n namespace_list = [namespace]\n for n in namespace_list:\n if n != '/' and self.manager.is_connected(sid, n):\n self._trigger_event('disconnect', n, sid)\n self.manager.disconnect(sid, n)\n if namespace == '/' and self.manager.is_connected(sid, namespace):\n self._trigger_event('disconnect', '/', sid)\n self.manager.disconnect(sid, '/')", "response": "Handle a client disconnect."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling an incoming client event.", "response": "def _handle_event(self, sid, namespace, id, data):\n \"\"\"Handle an incoming client event.\"\"\"\n namespace = namespace or '/'\n self.logger.info('received event \"%s\" from %s [%s]', data[0], sid,\n namespace)\n if self.async_handlers:\n self.start_background_task(self._handle_event_internal, self, sid,\n data, namespace, id)\n else:\n self._handle_event_internal(self, sid, data, namespace, id)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ninvoking an application event handler.", "response": "def _trigger_event(self, event, namespace, *args):\n \"\"\"Invoke an application event handler.\"\"\"\n # first see if we have an explicit handler for the event\n if namespace in self.handlers and event in self.handlers[namespace]:\n return self.handlers[namespace][event](*args)\n\n # or else, forward the event to a namespace handler if one exists\n elif namespace in self.namespace_handlers:\n return self.namespace_handlers[namespace].trigger_event(\n event, *args)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _handle_eio_connect(self, sid, environ):\n if not self.manager_initialized:\n self.manager_initialized = True\n self.manager.initialize()\n self.environ[sid] = environ\n return self._handle_connect(sid, '/')", "response": "Handle the Engine. IO connection event."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndispatches Engine. IO messages.", "response": "def _handle_eio_message(self, sid, data):\n \"\"\"Dispatch Engine.IO messages.\"\"\"\n if sid in self._binary_packet:\n pkt = self._binary_packet[sid]\n if pkt.add_attachment(data):\n del self._binary_packet[sid]\n if pkt.packet_type == packet.BINARY_EVENT:\n self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)\n else:\n self._handle_ack(sid, pkt.namespace, pkt.id, pkt.data)\n else:\n pkt = packet.Packet(encoded_packet=data)\n if pkt.packet_type == packet.CONNECT:\n self._handle_connect(sid, pkt.namespace)\n elif pkt.packet_type == packet.DISCONNECT:\n self._handle_disconnect(sid, pkt.namespace)\n elif pkt.packet_type == packet.EVENT:\n self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)\n elif pkt.packet_type == packet.ACK:\n self._handle_ack(sid, pkt.namespace, pkt.id, pkt.data)\n elif pkt.packet_type == packet.BINARY_EVENT or \\\n pkt.packet_type == packet.BINARY_ACK:\n self._binary_packet[sid] = pkt\n elif pkt.packet_type == packet.ERROR:\n raise ValueError('Unexpected ERROR packet.')\n else:\n raise ValueError('Unknown packet type.')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _handle_eio_disconnect(self, sid):\n self._handle_disconnect(sid, '/')\n if sid in self.environ:\n del self.environ[sid]", "response": "Handle Engine. IO disconnect event."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nemit a message to a single client or a room or all the servers that are connected to the namespace.", "response": "async def emit(self, event, data, namespace=None, room=None, skip_sid=None,\n callback=None, **kwargs):\n \"\"\"Emit a message to a single client, a room, or all the clients\n connected to the namespace.\n\n This method takes care or propagating the message to all the servers\n that are connected through the message queue.\n\n The parameters are the same as in :meth:`.Server.emit`.\n\n Note: this method is a coroutine.\n \"\"\"\n if kwargs.get('ignore_queue'):\n return await super().emit(\n event, data, namespace=namespace, room=room, skip_sid=skip_sid,\n callback=callback)\n namespace = namespace or '/'\n if callback is not None:\n if self.server is None:\n raise RuntimeError('Callbacks can only be issued from the '\n 'context of a server.')\n if room is None:\n raise ValueError('Cannot use callback without a room set.')\n id = self._generate_ack_id(room, namespace, callback)\n callback = (room, namespace, id)\n else:\n callback = None\n await self._publish({'method': 'emit', 'event': event, 'data': data,\n 'namespace': namespace, 'room': room,\n 'skip_sid': skip_sid, 'callback': callback,\n 'host_id': self.host_id})"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def connect(self, url, headers={}, transports=None,\n namespaces=None, socketio_path='socket.io'):\n \"\"\"Connect to a Socket.IO server.\n\n :param url: The URL of the Socket.IO server. It can include custom\n query string parameters if required by the server.\n :param headers: A dictionary with custom headers to send with the\n connection request.\n :param transports: The list of allowed transports. Valid transports\n are ``'polling'`` and ``'websocket'``. If not\n given, the polling transport is connected first,\n then an upgrade to websocket is attempted.\n :param namespaces: The list of custom namespaces to connect, in\n addition to the default namespace. If not given,\n the namespace list is obtained from the registered\n event handlers.\n :param socketio_path: The endpoint where the Socket.IO server is\n installed. The default value is appropriate for\n most cases.\n\n Note: this method is a coroutine.\n\n Example usage::\n\n sio = socketio.Client()\n sio.connect('http://localhost:5000')\n \"\"\"\n self.connection_url = url\n self.connection_headers = headers\n self.connection_transports = transports\n self.connection_namespaces = namespaces\n self.socketio_path = socketio_path\n\n if namespaces is None:\n namespaces = set(self.handlers.keys()).union(\n set(self.namespace_handlers.keys()))\n elif isinstance(namespaces, six.string_types):\n namespaces = [namespaces]\n self.connection_namespaces = namespaces\n self.namespaces = [n for n in namespaces if n != '/']\n try:\n await self.eio.connect(url, headers=headers,\n transports=transports,\n engineio_path=socketio_path)\n except engineio.exceptions.ConnectionError as exc:\n six.raise_from(exceptions.ConnectionError(exc.args[0]), None)", "response": "Connect to a Socket. IO server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def wait(self):\n while True:\n await self.eio.wait()\n await self.sleep(1) # give the reconnect task time to start up\n if not self._reconnect_task:\n break\n await self._reconnect_task\n if self.eio.state != 'connected':\n break", "response": "Wait until the connection with the server ends."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nregisters a namespace handler object.", "response": "def register_namespace(self, namespace_handler):\n \"\"\"Register a namespace handler object.\n\n :param namespace_handler: An instance of a :class:`Namespace`\n subclass that handles all the event traffic\n for a namespace.\n \"\"\"\n if not isinstance(namespace_handler, namespace.ClientNamespace):\n raise ValueError('Not a namespace instance')\n if self.is_asyncio_based() != namespace_handler.is_asyncio_based():\n raise ValueError('Not a valid namespace class for this client')\n namespace_handler._set_client(self)\n self.namespace_handlers[namespace_handler.namespace] = \\\n namespace_handler"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef emit(self, event, data=None, namespace=None, callback=None):\n namespace = namespace or '/'\n self.logger.info('Emitting event \"%s\" [%s]', event, namespace)\n if callback is not None:\n id = self._generate_ack_id(namespace, callback)\n else:\n id = None\n if six.PY2 and not self.binary:\n binary = False # pragma: nocover\n else:\n binary = None\n # tuples are expanded to multiple arguments, everything else is sent\n # as a single argument\n if isinstance(data, tuple):\n data = list(data)\n elif data is not None:\n data = [data]\n else:\n data = []\n self._send_packet(packet.Packet(packet.EVENT, namespace=namespace,\n data=[event] + data, id=id,\n binary=binary))", "response": "Emit a custom event to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send(self, data, namespace=None, callback=None):\n self.emit('message', data=data, namespace=namespace,\n callback=callback)", "response": "Send a message to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef call(self, event, data=None, namespace=None, timeout=60):\n callback_event = self.eio.create_event()\n callback_args = []\n\n def event_callback(*args):\n callback_args.append(args)\n callback_event.set()\n\n self.emit(event, data=data, namespace=namespace,\n callback=event_callback)\n if not callback_event.wait(timeout=timeout):\n raise exceptions.TimeoutError()\n return callback_args[0] if len(callback_args[0]) > 1 \\\n else callback_args[0][0] if len(callback_args[0]) == 1 \\\n else None", "response": "Emit a custom event to a client and wait for the response."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisconnecting from the server.", "response": "def disconnect(self):\n \"\"\"Disconnect from the server.\"\"\"\n # here we just request the disconnection\n # later in _handle_eio_disconnect we invoke the disconnect handler\n for n in self.namespaces:\n self._send_packet(packet.Packet(packet.DISCONNECT, namespace=n))\n self._send_packet(packet.Packet(\n packet.DISCONNECT, namespace='/'))\n self.eio.disconnect(abort=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nhandle the Engine. IO disconnection event.", "response": "def _handle_eio_disconnect(self):\n \"\"\"Handle the Engine.IO disconnection event.\"\"\"\n self.logger.info('Engine.IO connection dropped')\n for n in self.namespaces:\n self._trigger_event('disconnect', namespace=n)\n self._trigger_event('disconnect', namespace='/')\n self.callbacks = {}\n self._binary_packet = None\n if self.eio.state == 'connected' and self.reconnection:\n self._reconnect_task = self.start_background_task(\n self._handle_reconnect)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encode(self):\n encoded_packet = six.text_type(self.packet_type)\n if self.packet_type == BINARY_EVENT or self.packet_type == BINARY_ACK:\n data, attachments = self._deconstruct_binary(self.data)\n encoded_packet += six.text_type(len(attachments)) + '-'\n else:\n data = self.data\n attachments = None\n needs_comma = False\n if self.namespace is not None and self.namespace != '/':\n encoded_packet += self.namespace\n needs_comma = True\n if self.id is not None:\n if needs_comma:\n encoded_packet += ','\n needs_comma = False\n encoded_packet += six.text_type(self.id)\n if data is not None:\n if needs_comma:\n encoded_packet += ','\n encoded_packet += self.json.dumps(data, separators=(',', ':'))\n if attachments is not None:\n encoded_packet = [encoded_packet] + attachments\n return encoded_packet", "response": "Encode the packet for transmission."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecode a transmitted package.", "response": "def decode(self, encoded_packet):\n \"\"\"Decode a transmitted package.\n\n The return value indicates how many binary attachment packets are\n necessary to fully decode the packet.\n \"\"\"\n ep = encoded_packet\n try:\n self.packet_type = int(ep[0:1])\n except TypeError:\n self.packet_type = ep\n ep = ''\n self.namespace = None\n self.data = None\n ep = ep[1:]\n dash = ep.find('-')\n attachment_count = 0\n if dash > 0 and ep[0:dash].isdigit():\n attachment_count = int(ep[0:dash])\n ep = ep[dash + 1:]\n if ep and ep[0:1] == '/':\n sep = ep.find(',')\n if sep == -1:\n self.namespace = ep\n ep = ''\n else:\n self.namespace = ep[0:sep]\n ep = ep[sep + 1:]\n q = self.namespace.find('?')\n if q != -1:\n self.namespace = self.namespace[0:q]\n if ep and ep[0].isdigit():\n self.id = 0\n while ep and ep[0].isdigit():\n self.id = self.id * 10 + int(ep[0])\n ep = ep[1:]\n if ep:\n self.data = self.json.loads(ep)\n return attachment_count"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreconstruct a decoded packet using the given list of binary attachments.", "response": "def reconstruct_binary(self, attachments):\n \"\"\"Reconstruct a decoded packet using the given list of binary\n attachments.\n \"\"\"\n self.data = self._reconstruct_binary_internal(self.data,\n self.attachments)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _deconstruct_binary(self, data):\n attachments = []\n data = self._deconstruct_binary_internal(data, attachments)\n return data, attachments", "response": "Extract binary components in the packet."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the data contains binary components.", "response": "def _data_is_binary(self, data):\n \"\"\"Check if the data contains binary components.\"\"\"\n if isinstance(data, six.binary_type):\n return True\n elif isinstance(data, list):\n return functools.reduce(\n lambda a, b: a or b, [self._data_is_binary(item)\n for item in data], False)\n elif isinstance(data, dict):\n return functools.reduce(\n lambda a, b: a or b, [self._data_is_binary(item)\n for item in six.itervalues(data)],\n False)\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def trigger_event(self, event, *args):\n handler_name = 'on_' + event\n if hasattr(self, handler_name):\n handler = getattr(self, handler_name)\n if asyncio.iscoroutinefunction(handler) is True:\n try:\n ret = await handler(*args)\n except asyncio.CancelledError: # pragma: no cover\n ret = None\n else:\n ret = handler(*args)\n return ret", "response": "Dispatch an event to the proper handler method."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncloses a room. The only difference with the :func:`socketio.Server.close_room` method is that when the ``namespace`` argument is not given the namespace associated with the class is used. Note: this method is a coroutine.", "response": "async def close_room(self, room, namespace=None):\n \"\"\"Close a room.\n\n The only difference with the :func:`socketio.Server.close_room` method\n is that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n\n Note: this method is a coroutine.\n \"\"\"\n return await self.server.close_room(\n room, namespace=namespace or self.namespace)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def disconnect(self, sid, namespace=None):\n return await self.server.disconnect(\n sid, namespace=namespace or self.namespace)", "response": "Disconnect a client.\n\n The only difference with the :func:`socketio.Server.disconnect` method\n is that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n\n Note: this method is a coroutine."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def emit(self, event, data=None, namespace=None, callback=None):\n return await self.client.emit(event, data=data,\n namespace=namespace or self.namespace,\n callback=callback)", "response": "Emit a custom event to the server."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a message to the server.", "response": "async def send(self, data, namespace=None, callback=None):\n \"\"\"Send a message to the server.\n\n The only difference with the :func:`socketio.Client.send` method is\n that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n\n Note: this method is a coroutine.\n \"\"\"\n return await self.client.send(data,\n namespace=namespace or self.namespace,\n callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndispatches an event to the proper handler method.", "response": "def trigger_event(self, event, *args):\n \"\"\"Dispatch an event to the proper handler method.\n\n In the most common usage, this method is not overloaded by subclasses,\n as it performs the routing of events to methods. However, this\n method can be overriden if special dispatching rules are needed, or if\n having a single method that catches all events is desired.\n \"\"\"\n handler_name = 'on_' + event\n if hasattr(self, handler_name):\n return getattr(self, handler_name)(*args)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef emit(self, event, data=None, room=None, skip_sid=None, namespace=None,\n callback=None):\n \"\"\"Emit a custom event to one or more connected clients.\n\n The only difference with the :func:`socketio.Server.emit` method is\n that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n \"\"\"\n return self.server.emit(event, data=data, room=room, skip_sid=skip_sid,\n namespace=namespace or self.namespace,\n callback=callback)", "response": "Emit a custom event to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send(self, data, room=None, skip_sid=None, namespace=None,\n callback=None):\n \"\"\"Send a message to one or more connected clients.\n\n The only difference with the :func:`socketio.Server.send` method is\n that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n \"\"\"\n return self.server.send(data, room=room, skip_sid=skip_sid,\n namespace=namespace or self.namespace,\n callback=callback)", "response": "Send a message to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nentering a room. The only difference with the :func:`socketio.Server.enter_room` method is that when the ``namespace`` argument is not given the namespace associated with the class is used.", "response": "def enter_room(self, sid, room, namespace=None):\n \"\"\"Enter a room.\n\n The only difference with the :func:`socketio.Server.enter_room` method\n is that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n \"\"\"\n return self.server.enter_room(sid, room,\n namespace=namespace or self.namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef leave_room(self, sid, room, namespace=None):\n return self.server.leave_room(sid, room,\n namespace=namespace or self.namespace)", "response": "Leave a room.\n\n The only difference with the :func:`socketio.Server.leave_room` method\n is that when the ``namespace`` argument is not given the namespace\n associated with the class is used."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the rooms a client is in.", "response": "def rooms(self, sid, namespace=None):\n \"\"\"Return the rooms a client is in.\n\n The only difference with the :func:`socketio.Server.rooms` method is\n that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n \"\"\"\n return self.server.rooms(sid, namespace=namespace or self.namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_session(self, sid, namespace=None):\n return self.server.get_session(\n sid, namespace=namespace or self.namespace)", "response": "Return the user session for a client."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nstore the user session for a client.", "response": "def save_session(self, sid, session, namespace=None):\n \"\"\"Store the user session for a client.\n\n The only difference with the :func:`socketio.Server.save_session`\n method is that when the ``namespace`` argument is not given the\n namespace associated with the class is used.\n \"\"\"\n return self.server.save_session(\n sid, session, namespace=namespace or self.namespace)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a message to the server.", "response": "def send(self, data, room=None, skip_sid=None, namespace=None,\n callback=None):\n \"\"\"Send a message to the server.\n\n The only difference with the :func:`socketio.Client.send` method is\n that when the ``namespace`` argument is not given the namespace\n associated with the class is used.\n \"\"\"\n return self.client.send(data, namespace=namespace or self.namespace,\n callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nemitting a message to a single client a room or all the clients that are connected to the namespace.", "response": "async def emit(self, event, data, namespace, room=None, skip_sid=None,\n callback=None, **kwargs):\n \"\"\"Emit a message to a single client, a room, or all the clients\n connected to the namespace.\n\n Note: this method is a coroutine.\n \"\"\"\n if namespace not in self.rooms or room not in self.rooms[namespace]:\n return\n tasks = []\n for sid in self.get_participants(namespace, room):\n if sid != skip_sid:\n if callback is not None:\n id = self._generate_ack_id(sid, namespace, callback)\n else:\n id = None\n tasks.append(self.server._emit_internal(sid, event, data,\n namespace, id))\n if tasks == []: # pragma: no cover\n return\n await asyncio.wait(tasks)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninvoking an application callback.", "response": "async def trigger_callback(self, sid, namespace, id, data):\n \"\"\"Invoke an application callback.\n\n Note: this method is a coroutine.\n \"\"\"\n callback = None\n try:\n callback = self.callbacks[sid][namespace][id]\n except KeyError:\n # if we get an unknown callback we just ignore it\n self._get_logger().warning('Unknown callback received, ignoring.')\n else:\n del self.callbacks[sid][namespace][id]\n if callback is not None:\n ret = callback(*data)\n if asyncio.iscoroutine(ret):\n try:\n await ret\n except asyncio.CancelledError: # pragma: no cover\n pass"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def emit(self, event, data=None, room=None, skip_sid=None,\n namespace=None, callback=None, **kwargs):\n \"\"\"Emit a custom event to one or more connected clients.\n\n :param event: The event name. It can be any string. The event names\n ``'connect'``, ``'message'`` and ``'disconnect'`` are\n reserved and should not be used.\n :param data: The data to send to the client or clients. Data can be of\n type ``str``, ``bytes``, ``list`` or ``dict``. If a\n ``list`` or ``dict``, the data will be serialized as JSON.\n :param room: The recipient of the message. This can be set to the\n session ID of a client to address that client's room, or\n to any custom room created by the application, If this\n argument is omitted the event is broadcasted to all\n connected clients.\n :param skip_sid: The session ID of a client to skip when broadcasting\n to a room or to all clients. This can be used to\n prevent a message from being sent to the sender.\n :param namespace: The Socket.IO namespace for the event. If this\n argument is omitted the event is emitted to the\n default namespace.\n :param callback: If given, this function will be called to acknowledge\n the the client has received the message. The arguments\n that will be passed to the function are those provided\n by the client. Callback functions can only be used\n when addressing an individual client.\n :param ignore_queue: Only used when a message queue is configured. If\n set to ``True``, the event is emitted to the\n clients directly, without going through the queue.\n This is more efficient, but only works when a\n single server process is used. It is recommended\n to always leave this parameter with its default\n value of ``False``.\n\n Note: this method is a coroutine.\n \"\"\"\n namespace = namespace or '/'\n self.logger.info('emitting event \"%s\" to %s [%s]', event,\n room or 'all', namespace)\n await self.manager.emit(event, data, namespace, room=room,\n skip_sid=skip_sid, callback=callback,\n **kwargs)", "response": "Emit a custom event to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def send(self, data, room=None, skip_sid=None, namespace=None,\n callback=None, **kwargs):\n \"\"\"Send a message to one or more connected clients.\n\n This function emits an event with the name ``'message'``. Use\n :func:`emit` to issue custom event names.\n\n :param data: The data to send to the client or clients. Data can be of\n type ``str``, ``bytes``, ``list`` or ``dict``. If a\n ``list`` or ``dict``, the data will be serialized as JSON.\n :param room: The recipient of the message. This can be set to the\n session ID of a client to address that client's room, or\n to any custom room created by the application, If this\n argument is omitted the event is broadcasted to all\n connected clients.\n :param skip_sid: The session ID of a client to skip when broadcasting\n to a room or to all clients. This can be used to\n prevent a message from being sent to the sender.\n :param namespace: The Socket.IO namespace for the event. If this\n argument is omitted the event is emitted to the\n default namespace.\n :param callback: If given, this function will be called to acknowledge\n the the client has received the message. The arguments\n that will be passed to the function are those provided\n by the client. Callback functions can only be used\n when addressing an individual client.\n :param ignore_queue: Only used when a message queue is configured. If\n set to ``True``, the event is emitted to the\n clients directly, without going through the queue.\n This is more efficient, but only works when a\n single server process is used. It is recommended\n to always leave this parameter with its default\n value of ``False``.\n\n Note: this method is a coroutine.\n \"\"\"\n await self.emit('message', data=data, room=room, skip_sid=skip_sid,\n namespace=namespace, callback=callback, **kwargs)", "response": "Send a message to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclose a room. This function removes all the clients from the given room. :param room: Room name. :param namespace: The Socket.IO namespace for the event. If this argument is omitted the default namespace is used. Note: this method is a coroutine.", "response": "async def close_room(self, room, namespace=None):\n \"\"\"Close a room.\n\n This function removes all the clients from the given room.\n\n :param room: Room name.\n :param namespace: The Socket.IO namespace for the event. If this\n argument is omitted the default namespace is used.\n\n Note: this method is a coroutine.\n \"\"\"\n namespace = namespace or '/'\n self.logger.info('room %s is closing [%s]', room, namespace)\n await self.manager.close_room(room, namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndisconnects a client. :param sid: Session ID of the client. :param namespace: The Socket.IO namespace to disconnect. If this argument is omitted the default namespace is used. Note: this method is a coroutine.", "response": "async def disconnect(self, sid, namespace=None):\n \"\"\"Disconnect a client.\n\n :param sid: Session ID of the client.\n :param namespace: The Socket.IO namespace to disconnect. If this\n argument is omitted the default namespace is used.\n\n Note: this method is a coroutine.\n \"\"\"\n namespace = namespace or '/'\n if self.manager.is_connected(sid, namespace=namespace):\n self.logger.info('Disconnecting %s [%s]', sid, namespace)\n self.manager.pre_disconnect(sid, namespace=namespace)\n await self._send_packet(sid, packet.Packet(packet.DISCONNECT,\n namespace=namespace))\n await self._trigger_event('disconnect', namespace, sid)\n self.manager.disconnect(sid, namespace=namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nhandles ACK packets from the client.", "response": "async def _handle_ack(self, sid, namespace, id, data):\n \"\"\"Handle ACK packets from the client.\"\"\"\n namespace = namespace or '/'\n self.logger.info('received ack from %s [%s]', sid, namespace)\n await self.manager.trigger_callback(sid, namespace, id, data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninvoking an application event handler.", "response": "async def _trigger_event(self, event, namespace, *args):\n \"\"\"Invoke an application event handler.\"\"\"\n # first see if we have an explicit handler for the event\n if namespace in self.handlers and event in self.handlers[namespace]:\n if asyncio.iscoroutinefunction(self.handlers[namespace][event]) \\\n is True:\n try:\n ret = await self.handlers[namespace][event](*args)\n except asyncio.CancelledError: # pragma: no cover\n ret = None\n else:\n ret = self.handlers[namespace][event](*args)\n return ret\n\n # or else, forward the event to a namepsace handler if one exists\n elif namespace in self.namespace_handlers:\n return await self.namespace_handlers[namespace].trigger_event(\n event, *args)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef should_set_tablename(cls):\n if (\n cls.__dict__.get('__abstract__', False)\n or not any(isinstance(b, DeclarativeMeta) for b in cls.__mro__[1:])\n ):\n return False\n\n for base in cls.__mro__:\n if '__tablename__' not in base.__dict__:\n continue\n\n if isinstance(base.__dict__['__tablename__'], declared_attr):\n return False\n\n return not (\n base is cls\n or base.__dict__.get('__abstract__', False)\n or not isinstance(base, DeclarativeMeta)\n )\n\n return True", "response": "Determine whether the name of the class cls is set."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef login_required(view):\n\n @functools.wraps(view)\n def wrapped_view(**kwargs):\n if g.user is None:\n return redirect(url_for(\"auth.login\"))\n\n return view(**kwargs)\n\n return wrapped_view", "response": "View decorator that redirects anonymous users to the login page."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_logged_in_user():\n user_id = session.get(\"user_id\")\n g.user = User.query.get(user_id) if user_id is not None else None", "response": "Load the logged in user into the g. user attribute."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef register():\n if request.method == \"POST\":\n username = request.form[\"username\"]\n password = request.form[\"password\"]\n error = None\n\n if not username:\n error = \"Username is required.\"\n elif not password:\n error = \"Password is required.\"\n elif db.session.query(\n User.query.filter_by(username=username).exists()\n ).scalar():\n error = f\"User {username} is already registered.\"\n\n if error is None:\n # the name is available, create the user and go to the login page\n db.session.add(User(username=username, password=password))\n db.session.commit()\n return redirect(url_for(\"auth.login\"))\n\n flash(error)\n\n return render_template(\"auth/register.html\")", "response": "Register a new user."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef login():\n if request.method == \"POST\":\n username = request.form[\"username\"]\n password = request.form[\"password\"]\n error = None\n user = User.query.filter_by(username=username).first()\n\n if user is None:\n error = \"Incorrect username.\"\n elif not user.check_password(password):\n error = \"Incorrect password.\"\n\n if error is None:\n # store the user id in a new session and return to the index\n session.clear()\n session[\"user_id\"] = user.id\n return redirect(url_for(\"index\"))\n\n flash(error)\n\n return render_template(\"auth/login.html\")", "response": "Log in a registered user by adding the user id to the session."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef index():\n posts = Post.query.order_by(Post.created.desc()).all()\n return render_template(\"blog/index.html\", posts=posts)", "response": "Show all the posts most recent first."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a post and its author by id.", "response": "def get_post(id, check_author=True):\n \"\"\"Get a post and its author by id.\n\n Checks that the id exists and optionally that the current user is\n the author.\n\n :param id: id of post to get\n :param check_author: require the current user to be the author\n :return: the post with author information\n :raise 404: if a post with the given id doesn't exist\n :raise 403: if the current user isn't the author\n \"\"\"\n post = Post.query.get_or_404(id, f\"Post id {id} doesn't exist.\")\n\n if check_author and post.author != g.user:\n abort(403)\n\n return post"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create():\n if request.method == \"POST\":\n title = request.form[\"title\"]\n body = request.form[\"body\"]\n error = None\n\n if not title:\n error = \"Title is required.\"\n\n if error is not None:\n flash(error)\n else:\n db.session.add(Post(title=title, body=body, author=g.user))\n db.session.commit()\n return redirect(url_for(\"blog.index\"))\n\n return render_template(\"blog/create.html\")", "response": "Create a new post for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(id):\n post = get_post(id)\n\n if request.method == \"POST\":\n title = request.form[\"title\"]\n body = request.form[\"body\"]\n error = None\n\n if not title:\n error = \"Title is required.\"\n\n if error is not None:\n flash(error)\n else:\n post.title = title\n post.body = body\n db.session.commit()\n return redirect(url_for(\"blog.index\"))\n\n return render_template(\"blog/update.html\", post=post)", "response": "Update a post if the current user is the author."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef delete(id):\n post = get_post(id)\n db.session.delete(post)\n db.session.commit()\n return redirect(url_for(\"blog.index\"))", "response": "Delete a post.\n\n Ensures that the post exists and that the logged in user is the\n author of the post."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_version(v):\n parts = v.split(\".\")\n # Pad the list to make sure there is three elements so that we get major, minor, point\n # comparisons that default to \"0\" if not given. I.e. \"1.2\" --> (1, 2, 0)\n parts = (parts + 3 * ['0'])[:3]\n return tuple(int(x) for x in parts)", "response": "Takes a string version and conver it to a tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the engine or connection for a given model or an object.", "response": "def get_bind(self, mapper=None, clause=None):\n \"\"\"Return the engine or connection for a given model or\n table, using the ``__bind_key__`` if it is set.\n \"\"\"\n # mapper is None if someone tries to just get a connection\n if mapper is not None:\n try:\n # SA >= 1.3\n persist_selectable = mapper.persist_selectable\n except AttributeError:\n # SA < 1.3\n persist_selectable = mapper.mapped_table\n\n info = getattr(persist_selectable, 'info', {})\n bind_key = info.get('bind_key')\n if bind_key is not None:\n state = get_state(self.app)\n return state.db.get_engine(self.app, bind=bind_key)\n return SessionBase.get_bind(self, mapper, clause)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef next(self, error_out=False):\n assert self.query is not None, 'a query object is required ' \\\n 'for this method to work'\n return self.query.paginate(self.page + 1, self.per_page, error_out)", "response": "Returns a : class : Pagination object for the next page."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_or_404(self, ident, description=None):\n\n rv = self.get(ident)\n if rv is None:\n abort(404, description=description)\n return rv", "response": "Like get but aborts with 404 if not found instead of returning None."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef first_or_404(self, description=None):\n\n rv = self.first()\n if rv is None:\n abort(404, description=description)\n return rv", "response": "Like first but aborts with 404 if not found instead of returning None."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef paginate(self, page=None, per_page=None, error_out=True, max_per_page=None, count=True):\n\n if request:\n if page is None:\n try:\n page = int(request.args.get('page', 1))\n except (TypeError, ValueError):\n if error_out:\n abort(404)\n\n page = 1\n\n if per_page is None:\n try:\n per_page = int(request.args.get('per_page', 20))\n except (TypeError, ValueError):\n if error_out:\n abort(404)\n\n per_page = 20\n else:\n if page is None:\n page = 1\n\n if per_page is None:\n per_page = 20\n\n if max_per_page is not None:\n per_page = min(per_page, max_per_page)\n\n if page < 1:\n if error_out:\n abort(404)\n else:\n page = 1\n\n if per_page < 0:\n if error_out:\n abort(404)\n else:\n per_page = 20\n\n items = self.limit(per_page).offset((page - 1) * per_page).all()\n\n if not items and page != 1 and error_out:\n abort(404)\n\n # No need to count if we're on the first page and there are fewer\n # items than we expected or if count is disabled.\n\n if not count:\n total = None\n elif page == 1 and len(items) < per_page:\n total = len(items)\n else:\n total = self.order_by(None).count()\n\n return Pagination(self, page, per_page, total, items)", "response": "Returns a pagination object for the items in the current page."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_scoped_session(self, options=None):\n\n if options is None:\n options = {}\n\n scopefunc = options.pop('scopefunc', _app_ctx_stack.__ident_func__)\n options.setdefault('query_cls', self.Query)\n return orm.scoped_session(\n self.create_session(options), scopefunc=scopefunc\n )", "response": "Create a scoped session on the factory from the Flask s create_session method."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates the session factory used by create_scoped_session.", "response": "def create_session(self, options):\n \"\"\"Create the session factory used by :meth:`create_scoped_session`.\n\n The factory **must** return an object that SQLAlchemy recognizes as a session,\n or registering session events may raise an exception.\n\n Valid factories include a :class:`~sqlalchemy.orm.session.Session`\n class or a :class:`~sqlalchemy.orm.session.sessionmaker`.\n\n The default implementation creates a ``sessionmaker`` for :class:`SignallingSession`.\n\n :param options: dict of keyword arguments passed to session class\n \"\"\"\n\n return orm.sessionmaker(class_=SignallingSession, db=self, **options)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a declarative base that all models will inherit from.", "response": "def make_declarative_base(self, model, metadata=None):\n \"\"\"Creates the declarative base that all models will inherit from.\n\n :param model: base model class (or a tuple of base classes) to pass\n to :func:`~sqlalchemy.ext.declarative.declarative_base`. Or a class\n returned from ``declarative_base``, in which case a new base class\n is not created.\n :param metadata: :class:`~sqlalchemy.MetaData` instance to use, or\n none to use SQLAlchemy's default.\n\n .. versionchanged 2.3.0::\n ``model`` can be an existing declarative base in order to support\n complex customization such as changing the metaclass.\n \"\"\"\n if not isinstance(model, DeclarativeMeta):\n model = declarative_base(\n cls=model,\n name='Model',\n metadata=metadata,\n metaclass=DefaultMeta\n )\n\n # if user passed in a declarative base and a metaclass for some reason,\n # make sure the base uses the metaclass\n if metadata is not None and model.metadata is not metadata:\n model.metadata = metadata\n\n if not getattr(model, 'query_class', None):\n model.query_class = self.Query\n\n model.query = _QueryProperty(self)\n return model"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_connector(self, app=None, bind=None):\n return _EngineConnector(self, self.get_app(app), bind)", "response": "Creates the connector for a given state and bind."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_engine(self, app=None, bind=None):\n\n app = self.get_app(app)\n state = get_state(app)\n\n with self._engine_lock:\n connector = state.connectors.get(bind)\n\n if connector is None:\n connector = self.make_connector(app, bind)\n state.connectors[bind] = connector\n\n return connector.get_engine()", "response": "Returns a specific engine."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_app(self, reference_app=None):\n\n if reference_app is not None:\n return reference_app\n\n if current_app:\n return current_app._get_current_object()\n\n if self.app is not None:\n return self.app\n\n raise RuntimeError(\n 'No application found. Either work inside a view function or push'\n ' an application context. See'\n ' http://flask-sqlalchemy.pocoo.org/contexts/.'\n )", "response": "Helper method that returns the application object that is used to access the anova."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of all tables relevant for a bind.", "response": "def get_tables_for_bind(self, bind=None):\n \"\"\"Returns a list of all tables relevant for a bind.\"\"\"\n result = []\n for table in itervalues(self.Model.metadata.tables):\n if table.info.get('bind_key') == bind:\n result.append(table)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a dictionary with a table - > engine mapping.", "response": "def get_binds(self, app=None):\n \"\"\"Returns a dictionary with a table->engine mapping.\n\n This is suitable for use of sessionmaker(binds=db.get_binds(app)).\n \"\"\"\n app = self.get_app(app)\n binds = [None] + list(app.config.get('SQLALCHEMY_BINDS') or ())\n retval = {}\n for bind in binds:\n engine = self.get_engine(app, bind)\n tables = self.get_tables_for_bind(bind)\n retval.update(dict((table, engine) for table in tables))\n return retval"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reflect(self, bind='__all__', app=None):\n self._execute_for_all_tables(app, bind, 'reflect', skip_tables=True)", "response": "Reflects tables from the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating and configure an instance of the Flask application.", "response": "def create_app(test_config=None):\n \"\"\"Create and configure an instance of the Flask application.\"\"\"\n app = Flask(__name__, instance_relative_config=True)\n\n # some deploy systems set the database url in the environ\n db_url = os.environ.get(\"DATABASE_URL\")\n\n if db_url is None:\n # default to a sqlite database in the instance folder\n db_url = \"sqlite:///\" + os.path.join(app.instance_path, \"flaskr.sqlite\")\n # ensure the instance folder exists\n os.makedirs(app.instance_path, exist_ok=True)\n\n app.config.from_mapping(\n # default secret that should be overridden in environ or config\n SECRET_KEY=os.environ.get(\"SECRET_KEY\", \"dev\"),\n SQLALCHEMY_DATABASE_URI=db_url,\n SQLALCHEMY_TRACK_MODIFICATIONS=False,\n )\n\n if test_config is None:\n # load the instance config, if it exists, when not testing\n app.config.from_pyfile(\"config.py\", silent=True)\n else:\n # load the test config if passed in\n app.config.update(test_config)\n\n # initialize Flask-SQLAlchemy and the init-db command\n db.init_app(app)\n app.cli.add_command(init_db_command)\n\n # apply the blueprints to the app\n from flaskr import auth, blog\n\n app.register_blueprint(auth.bp)\n app.register_blueprint(blog.bp)\n\n # make \"index\" point at \"/\", which is handled by \"blog.index\"\n app.add_url_rule(\"/\", endpoint=\"index\")\n\n return app"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef negative_sharpe(\n weights, expected_returns, cov_matrix, gamma=0, risk_free_rate=0.02\n):\n \"\"\"\n Calculate the negative Sharpe ratio of a portfolio\n\n :param weights: asset weights of the portfolio\n :type weights: np.ndarray\n :param expected_returns: expected return of each asset\n :type expected_returns: pd.Series\n :param cov_matrix: the covariance matrix of asset returns\n :type cov_matrix: pd.DataFrame\n :param gamma: L2 regularisation parameter, defaults to 0. Increase if you want more\n non-negligible weights\n :type gamma: float, optional\n :param risk_free_rate: risk-free rate of borrowing/lending, defaults to 0.02\n :type risk_free_rate: float, optional\n :return: negative Sharpe ratio\n :rtype: float\n \"\"\"\n mu = weights.dot(expected_returns)\n sigma = np.sqrt(np.dot(weights, np.dot(cov_matrix, weights.T)))\n L2_reg = gamma * (weights ** 2).sum()\n return -(mu - risk_free_rate) / sigma + L2_reg", "response": "Calculate the negative Sharpe ratio of a single asset on the cluster"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef volatility(weights, cov_matrix, gamma=0):\n L2_reg = gamma * (weights ** 2).sum()\n portfolio_volatility = np.dot(weights.T, np.dot(cov_matrix, weights))\n return portfolio_volatility + L2_reg", "response": "Calculate the volatility of a portfolio."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef negative_cvar(weights, returns, s=10000, beta=0.95, random_state=None):\n np.random.seed(seed=random_state)\n # Calcualte the returns given the weights\n portfolio_returns = (weights * returns).sum(axis=1)\n # Sample from the historical distribution\n dist = scipy.stats.gaussian_kde(portfolio_returns)\n sample = dist.resample(s)\n # Calculate the value at risk\n var = portfolio_returns.quantile(1 - beta)\n # Mean of all losses worse than the value at risk\n return -sample[sample < var].mean()", "response": "Calculate the negative CVaR for a given portfolio."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mean_historical_return(prices, frequency=252):\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n daily_returns = daily_price_returns(prices)\n return daily_returns.mean() * frequency", "response": "Calculate annualised mean historical return for each asset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the exponentially - weighted mean of the historical returns of an asset.", "response": "def ema_historical_return(prices, frequency=252, span=500):\n \"\"\"\n Calculate the exponentially-weighted mean of (daily) historical returns, giving\n higher weight to more recent data.\n\n :param prices: adjusted closing prices of the asset, each row is a date\n and each column is a ticker/id.\n :type prices: pd.DataFrame\n :param frequency: number of time periods in a year, defaults to 252 (the number\n of trading days in a year)\n :type frequency: int, optional\n :param span: the time-span for the EMA, defaults to 500-day EMA.\n :type span: int, optional\n :return: annualised exponentially-weighted mean (daily) return of each asset\n :rtype: pd.Series\n \"\"\"\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n daily_returns = daily_price_returns(prices)\n return daily_returns.ewm(span=span).mean().iloc[-1] * frequency"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef max_sharpe(self, risk_free_rate=0.02):\n if not isinstance(risk_free_rate, (int, float)):\n raise ValueError(\"risk_free_rate should be numeric\")\n\n args = (self.expected_returns, self.cov_matrix, self.gamma, risk_free_rate)\n result = sco.minimize(\n objective_functions.negative_sharpe,\n x0=self.initial_guess,\n args=args,\n method=\"SLSQP\",\n bounds=self.bounds,\n constraints=self.constraints,\n )\n self.weights = result[\"x\"]\n return dict(zip(self.tickers, self.weights))", "response": "Maximise the Sharpe - maximising portfolio."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nminimising volatility. :return: asset weights for the volatility-minimising portfolio :rtype: dict", "response": "def min_volatility(self):\n \"\"\"\n Minimise volatility.\n\n :return: asset weights for the volatility-minimising portfolio\n :rtype: dict\n \"\"\"\n args = (self.cov_matrix, self.gamma)\n result = sco.minimize(\n objective_functions.volatility,\n x0=self.initial_guess,\n args=args,\n method=\"SLSQP\",\n bounds=self.bounds,\n constraints=self.constraints,\n )\n self.weights = result[\"x\"]\n return dict(zip(self.tickers, self.weights))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef custom_objective(self, objective_function, *args):\n result = sco.minimize(\n objective_function,\n x0=self.initial_guess,\n args=args,\n method=\"SLSQP\",\n bounds=self.bounds,\n constraints=self.constraints,\n )\n self.weights = result[\"x\"]\n return dict(zip(self.tickers, self.weights))", "response": "Optimise some objective function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the Sharpe - Maximising portfolio for a given target risk.", "response": "def efficient_risk(self, target_risk, risk_free_rate=0.02, market_neutral=False):\n \"\"\"\n Calculate the Sharpe-maximising portfolio for a given volatility (i.e max return\n for a target risk).\n\n :param target_risk: the desired volatility of the resulting portfolio.\n :type target_risk: float\n :param risk_free_rate: risk-free rate of borrowing/lending, defaults to 0.02\n :type risk_free_rate: float, optional\n :param market_neutral: whether the portfolio should be market neutral (weights sum to zero),\n defaults to False. Requires negative lower weight bound.\n :param market_neutral: bool, optional\n :raises ValueError: if ``target_risk`` is not a positive float\n :raises ValueError: if ``risk_free_rate`` is non-numeric\n :return: asset weights for the efficient risk portfolio\n :rtype: dict\n \"\"\"\n if not isinstance(target_risk, float) or target_risk < 0:\n raise ValueError(\"target_risk should be a positive float\")\n if not isinstance(risk_free_rate, (int, float)):\n raise ValueError(\"risk_free_rate should be numeric\")\n\n args = (self.expected_returns, self.cov_matrix, self.gamma, risk_free_rate)\n target_constraint = {\n \"type\": \"ineq\",\n \"fun\": lambda w: target_risk\n - np.sqrt(objective_functions.volatility(w, self.cov_matrix)),\n }\n # The equality constraint is either \"weights sum to 1\" (default), or\n # \"weights sum to 0\" (market neutral).\n if market_neutral:\n if self.bounds[0][0] is not None and self.bounds[0][0] >= 0:\n warnings.warn(\n \"Market neutrality requires shorting - bounds have been amended\",\n RuntimeWarning,\n )\n self.bounds = self._make_valid_bounds((-1, 1))\n constraints = [\n {\"type\": \"eq\", \"fun\": lambda x: np.sum(x)},\n target_constraint,\n ]\n else:\n constraints = self.constraints + [target_constraint]\n\n result = sco.minimize(\n objective_functions.negative_sharpe,\n x0=self.initial_guess,\n args=args,\n method=\"SLSQP\",\n bounds=self.bounds,\n constraints=constraints,\n )\n self.weights = result[\"x\"]\n return dict(zip(self.tickers, self.weights))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef efficient_return(self, target_return, market_neutral=False):\n if not isinstance(target_return, float) or target_return < 0:\n raise ValueError(\"target_return should be a positive float\")\n\n args = (self.cov_matrix, self.gamma)\n target_constraint = {\n \"type\": \"eq\",\n \"fun\": lambda w: w.dot(self.expected_returns) - target_return,\n }\n # The equality constraint is either \"weights sum to 1\" (default), or\n # \"weights sum to 0\" (market neutral).\n if market_neutral:\n if self.bounds[0][0] is not None and self.bounds[0][0] >= 0:\n warnings.warn(\n \"Market neutrality requires shorting - bounds have been amended\",\n RuntimeWarning,\n )\n self.bounds = self._make_valid_bounds((-1, 1))\n constraints = [\n {\"type\": \"eq\", \"fun\": lambda x: np.sum(x)},\n target_constraint,\n ]\n else:\n constraints = self.constraints + [target_constraint]\n\n result = sco.minimize(\n objective_functions.volatility,\n x0=self.initial_guess,\n args=args,\n method=\"SLSQP\",\n bounds=self.bounds,\n constraints=constraints,\n )\n self.weights = result[\"x\"]\n return dict(zip(self.tickers, self.weights))", "response": "Calculates the asset weights for a given target return."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef portfolio_performance(self, verbose=False, risk_free_rate=0.02):\n return base_optimizer.portfolio_performance(\n self.expected_returns,\n self.cov_matrix,\n self.weights,\n verbose,\n risk_free_rate,\n )", "response": "Calculates the performance of the optimal\n portfolio."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef portfolio_performance(\n expected_returns, cov_matrix, weights, verbose=False, risk_free_rate=0.02\n):\n \"\"\"\n After optimising, calculate (and optionally print) the performance of the optimal\n portfolio. Currently calculates expected return, volatility, and the Sharpe ratio.\n\n :param expected_returns: expected returns for each asset. Set to None if\n optimising for volatility only.\n :type expected_returns: pd.Series, list, np.ndarray\n :param cov_matrix: covariance of returns for each asset\n :type cov_matrix: pd.DataFrame or np.array\n :param weights: weights or assets\n :type weights: list, np.array or dict, optional\n :param verbose: whether performance should be printed, defaults to False\n :type verbose: bool, optional\n :param risk_free_rate: risk-free rate of borrowing/lending, defaults to 0.02\n :type risk_free_rate: float, optional\n :raises ValueError: if weights have not been calcualted yet\n :return: expected return, volatility, Sharpe ratio.\n :rtype: (float, float, float)\n \"\"\"\n if isinstance(weights, dict):\n if isinstance(expected_returns, pd.Series):\n tickers = list(expected_returns.index)\n elif isinstance(cov_matrix, pd.DataFrame):\n tickers = list(cov_matrix.columns)\n else:\n tickers = list(range(len(expected_returns)))\n newweights = np.zeros(len(tickers))\n for i, k in enumerate(tickers):\n if k in weights:\n newweights[i] = weights[k]\n if newweights.sum() == 0:\n raise ValueError(\"Weights add to zero, or ticker names don't match\")\n elif weights is not None:\n newweights = np.asarray(weights)\n else:\n raise ValueError(\"Weights is None\")\n sigma = np.sqrt(objective_functions.volatility(newweights, cov_matrix))\n mu = newweights.dot(expected_returns)\n\n sharpe = -objective_functions.negative_sharpe(\n newweights, expected_returns, cov_matrix, risk_free_rate = risk_free_rate\n )\n if verbose:\n print(\"Expected annual return: {:.1f}%\".format(100 * mu))\n print(\"Annual volatility: {:.1f}%\".format(100 * sigma))\n print(\"Sharpe Ratio: {:.2f}\".format(sharpe))\n return mu, sigma, sharpe", "response": "Calculates the performance of a single node in a cluster portfolio."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncleaning the raw asset weights", "response": "def clean_weights(self, cutoff=1e-4, rounding=5):\n \"\"\"\n Helper method to clean the raw weights, setting any weights whose absolute\n values are below the cutoff to zero, and rounding the rest.\n\n :param cutoff: the lower bound, defaults to 1e-4\n :type cutoff: float, optional\n :param rounding: number of decimal places to round the weights, defaults to 5.\n Set to None if rounding is not desired.\n :type rounding: int, optional\n :return: asset weights\n :rtype: dict\n \"\"\"\n if not isinstance(rounding, int) or rounding < 1:\n raise ValueError(\"rounding must be a positive integer\")\n clean_weights = self.weights.copy()\n clean_weights[np.abs(clean_weights) < cutoff] = 0\n if rounding is not None:\n clean_weights = np.round(clean_weights, rounding)\n return dict(zip(self.tickers, clean_weights))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef portfolio(weights, latest_prices, min_allocation=0.01, total_portfolio_value=10000):\n if not isinstance(weights, dict):\n raise TypeError(\"weights should be a dictionary of {ticker: weight}\")\n if not isinstance(latest_prices, (pd.Series, dict)):\n raise TypeError(\"latest_prices should be a pd.Series\")\n if min_allocation > 0.3:\n raise ValueError(\"min_allocation should be a small float\")\n if total_portfolio_value <= 0:\n raise ValueError(\"total_portfolio_value must be greater than zero\")\n\n # Drop any companies with negligible weights. We use a tuple because order matters.\n nonzero_weights = [(k, v) for k, v in weights.items() if v > min_allocation]\n print(\n \"{} out of {} tickers were removed\".format(\n len(weights) - len(nonzero_weights), len(weights)\n )\n )\n # Sort in descending order of weight\n nonzero_weights.sort(key=lambda x: x[1], reverse=True)\n available_funds = total_portfolio_value\n shares_bought = []\n buy_prices = []\n\n # First round\n for ticker, weight in nonzero_weights:\n price = latest_prices[ticker]\n # Attempt to buy the lower integer number of shares\n n_shares = int(weight * total_portfolio_value / price)\n cost = n_shares * price\n if cost > available_funds:\n # Buy as many as possible\n n_shares = available_funds // price\n if n_shares == 0:\n print(\"Insufficient funds\")\n available_funds -= cost\n shares_bought.append(n_shares)\n buy_prices.append(price)\n\n # Second round\n while available_funds > 0:\n # Calculate the equivalent continuous weights of the shares that\n # have already been bought\n current_weights = np.array(buy_prices) * np.array(shares_bought)\n current_weights /= current_weights.sum()\n ideal_weights = np.array([i[1] for i in nonzero_weights])\n deficit = ideal_weights - current_weights\n\n # Attempt to buy the asset whose current weights deviate the most\n idx = np.argmax(deficit)\n ticker, weight = nonzero_weights[idx]\n price = latest_prices[ticker]\n\n # If we can't afford this asset, search for the next highest deficit that we\n # can purchase.\n counter = 0\n while price > available_funds:\n deficit[idx] = 0 # we can no longer purchase the asset at idx\n idx = np.argmax(deficit) # find the next most deviant asset\n\n # If either of these conditions is met, we break out of both while loops\n # hence the repeated statement below\n if deficit[idx] < 0 or counter == 10:\n break\n\n ticker, weight = nonzero_weights[idx]\n price = latest_prices[ticker]\n counter += 1\n\n if deficit[idx] <= 0 or counter == 10:\n # See https://stackoverflow.com/questions/189645/\n break\n\n # Buy one share at a time\n shares_bought[idx] += 1\n available_funds -= price\n\n print(\"Funds remaining: {:.2f}\".format(available_funds))\n\n num_shares = dict(zip([i[0] for i in nonzero_weights], shares_bought))\n return num_shares, available_funds", "response": "This function takes a continuous weight dictionary and returns a discrete allocation of the number of shares of each asset in the order they are purchased."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts continuous weights to discrete allocation", "response": "def portfolio_lp(\n weights, latest_prices, min_allocation=0.01, total_portfolio_value=10000\n):\n \"\"\"\n For a long only portfolio, convert the continuous weights to a discrete allocation\n using Mixed Integer Linear Programming. This can be thought of as a clever way to round\n the continuous weights to an integer number of shares\n\n :param weights: continuous weights generated from the ``efficient_frontier`` module\n :type weights: dict\n :param latest_prices: the most recent price for each asset\n :type latest_prices: pd.Series or dict\n :param min_allocation: any weights less than this number are considered negligible,\n defaults to 0.01\n :type min_allocation: float, optional\n :param total_portfolio_value: the desired total value of the portfolio, defaults to 10000\n :type total_portfolio_value: int/float, optional\n :raises TypeError: if ``weights`` is not a dict\n :raises TypeError: if ``latest_prices`` isn't a series\n :raises ValueError: if not ``0 < min_allocation < 0.3``\n :return: the number of shares of each ticker that should be purchased, along with the amount\n of funds leftover.\n :rtype: (dict, float)\n \"\"\"\n import pulp\n\n if not isinstance(weights, dict):\n raise TypeError(\"weights should be a dictionary of {ticker: weight}\")\n if not isinstance(latest_prices, (pd.Series, dict)):\n raise TypeError(\"latest_prices should be a pd.Series\")\n if min_allocation > 0.3:\n raise ValueError(\"min_allocation should be a small float\")\n if total_portfolio_value <= 0:\n raise ValueError(\"total_portfolio_value must be greater than zero\")\n\n m = pulp.LpProblem(\"PfAlloc\", pulp.LpMinimize)\n vals = {}\n realvals = {}\n etas = {}\n abss = {}\n remaining = pulp.LpVariable(\"remaining\", 0)\n for k, w in weights.items():\n if w < min_allocation:\n continue\n vals[k] = pulp.LpVariable(\"x_%s\" % k, 0, cat=\"Integer\")\n realvals[k] = latest_prices[k] * vals[k]\n etas[k] = w * total_portfolio_value - realvals[k]\n abss[k] = pulp.LpVariable(\"u_%s\" % k, 0)\n m += etas[k] <= abss[k]\n m += -etas[k] <= abss[k]\n m += remaining == total_portfolio_value - pulp.lpSum(realvals.values())\n m += pulp.lpSum(abss.values()) + remaining\n m.solve()\n results = {k: val.varValue for k, val in vals.items()}\n return results, remaining.varValue"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind the portfolio weights that minimises the CVaR via Monte Carlo sampling from the return distribution.", "response": "def min_cvar(self, s=10000, beta=0.95, random_state=None):\n \"\"\"\n Find the portfolio weights that minimises the CVaR, via\n Monte Carlo sampling from the return distribution.\n\n :param s: number of bootstrap draws, defaults to 10000\n :type s: int, optional\n :param beta: \"significance level\" (i. 1 - q), defaults to 0.95\n :type beta: float, optional\n :param random_state: seed for random sampling, defaults to None\n :type random_state: int, optional\n :return: asset weights for the Sharpe-maximising portfolio\n :rtype: dict\n \"\"\"\n args = (self.returns, s, beta, random_state)\n result = noisyopt.minimizeSPSA(\n objective_functions.negative_cvar,\n args=args,\n bounds=self.bounds,\n x0=self.initial_guess,\n niter=1000,\n paired=False,\n )\n self.weights = self.normalize_weights(result[\"x\"])\n return dict(zip(self.tickers, self.weights))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the annualised sample covariance matrix of the given prices.", "response": "def sample_cov(prices, frequency=252):\n \"\"\"\n Calculate the annualised sample covariance matrix of (daily) asset returns.\n\n :param prices: adjusted closing prices of the asset, each row is a date\n and each column is a ticker/id.\n :type prices: pd.DataFrame\n :param frequency: number of time periods in a year, defaults to 252 (the number\n of trading days in a year)\n :type frequency: int, optional\n :return: annualised sample covariance matrix\n :rtype: pd.DataFrame\n \"\"\"\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n daily_returns = daily_price_returns(prices)\n return daily_returns.cov() * frequency"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef semicovariance(prices, benchmark=0, frequency=252):\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n daily_returns = daily_price_returns(prices)\n drops = np.fmin(daily_returns - benchmark, 0)\n return drops.cov() * frequency", "response": "Estimate the semicovariance matrix given a set of prices."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the exponential covariance between two timeseries of returns.", "response": "def _pair_exp_cov(X, Y, span=180):\n \"\"\"\n Calculate the exponential covariance between two timeseries of returns.\n\n :param X: first time series of returns\n :type X: pd.Series\n :param Y: second time series of returns\n :type Y: pd.Series\n :param span: the span of the exponential weighting function, defaults to 180\n :type span: int, optional\n :return: the exponential covariance between X and Y\n :rtype: float\n \"\"\"\n covariation = (X - X.mean()) * (Y - Y.mean())\n # Exponentially weight the covariation and take the mean\n if span < 10:\n warnings.warn(\"it is recommended to use a higher span, e.g 30 days\")\n return covariation.ewm(span=span).mean()[-1]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef exp_cov(prices, span=180, frequency=252):\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n assets = prices.columns\n daily_returns = daily_price_returns(prices)\n N = len(assets)\n\n # Loop over matrix, filling entries with the pairwise exp cov\n S = np.zeros((N, N))\n for i in range(N):\n for j in range(i, N):\n S[i, j] = S[j, i] = _pair_exp_cov(\n daily_returns.iloc[:, i], daily_returns.iloc[:, j], span\n )\n return pd.DataFrame(S * frequency, columns=assets, index=assets)", "response": "Estimate the exponential covariance matrix for a single asset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef min_cov_determinant(prices, frequency=252, random_state=None):\n if not isinstance(prices, pd.DataFrame):\n warnings.warn(\"prices are not in a dataframe\", RuntimeWarning)\n prices = pd.DataFrame(prices)\n assets = prices.columns\n X = prices.pct_change().dropna(how=\"all\")\n X = np.nan_to_num(X.values)\n raw_cov_array = covariance.fast_mcd(X, random_state=random_state)[1]\n return pd.DataFrame(raw_cov_array, index=assets, columns=assets) * frequency", "response": "Calculate the minimum covariance determinant of a single asset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef shrunk_covariance(self, delta=0.2):\n self.delta = delta\n N = self.S.shape[1]\n # Shrinkage target\n mu = np.trace(self.S) / N\n F = np.identity(N) * mu\n # Shrinkage\n shrunk_cov = delta * F + (1 - delta) * self.S\n return self.format_and_annualise(shrunk_cov)", "response": "Shrink a sample covariance matrix to the identity matrix scaled by the average\n sample variance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ledoit_wolf(self):\n X = np.nan_to_num(self.X.values)\n shrunk_cov, self.delta = covariance.ledoit_wolf(X)\n return self.format_and_annualise(shrunk_cov)", "response": "Calculate the Ledoit - Wolf shrinkage estimate."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef oracle_approximating(self):\n X = np.nan_to_num(self.X.values)\n shrunk_cov, self.delta = covariance.oas(X)\n return self.format_and_annualise(shrunk_cov)", "response": "Calculate the Oracle Approximating Shrinkage estimate\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef shrink(self):\n x = np.nan_to_num(self.X.values)\n\n # de-mean returns\n t, n = np.shape(x)\n meanx = x.mean(axis=0)\n x = x - np.tile(meanx, (t, 1))\n\n # compute sample covariance matrix\n sample = (1.0 / t) * np.dot(x.T, x)\n\n # compute prior\n var = np.diag(sample).reshape(-1, 1)\n sqrtvar = np.sqrt(var)\n _var = np.tile(var, (n,))\n _sqrtvar = np.tile(sqrtvar, (n,))\n r_bar = (np.sum(sample / (_sqrtvar * _sqrtvar.T)) - n) / (n * (n - 1))\n prior = r_bar * (_sqrtvar * _sqrtvar.T)\n prior[np.eye(n) == 1] = var.reshape(-1)\n\n # compute shrinkage parameters and constant\n if self.delta is None:\n\n # what we call pi-hat\n y = x ** 2.0\n phi_mat = np.dot(y.T, y) / t - 2 * np.dot(x.T, x) * sample / t + sample ** 2\n phi = np.sum(phi_mat)\n\n # what we call rho-hat\n term1 = np.dot((x ** 3).T, x) / t\n help_ = np.dot(x.T, x) / t\n help_diag = np.diag(help_)\n term2 = np.tile(help_diag, (n, 1)).T * sample\n term3 = help_ * _var\n term4 = _var * sample\n theta_mat = term1 - term2 - term3 + term4\n theta_mat[np.eye(n) == 1] = np.zeros(n)\n rho = sum(np.diag(phi_mat)) + r_bar * np.sum(\n np.dot((1.0 / sqrtvar), sqrtvar.T) * theta_mat\n )\n\n # what we call gamma-hat\n gamma = np.linalg.norm(sample - prior, \"fro\") ** 2\n\n # compute shrinkage constant\n kappa = (phi - rho) / gamma\n shrinkage = max(0.0, min(1.0, kappa / t))\n self.delta = shrinkage\n else:\n # use specified constant\n shrinkage = self.delta\n\n # compute the estimator\n sigma = shrinkage * prior + (1 - shrinkage) * sample\n return self.format_and_annualise(sigma)", "response": "Calculate the Constant - Correlation covariance matrix."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef shrink(self):\n x = np.nan_to_num(self.X.values)\n\n # de-mean returns\n t, n = np.shape(x)\n meanx = x.mean(axis=0)\n x = x - np.tile(meanx, (t, 1))\n xmkt = x.mean(axis=1).reshape(t, 1)\n\n # compute sample covariance matrix\n sample = np.cov(np.append(x, xmkt, axis=1), rowvar=False) * (t - 1) / t\n covmkt = sample[0:n, n].reshape(n, 1)\n varmkt = sample[n, n]\n sample = sample[:n, :n]\n prior = np.dot(covmkt, covmkt.T) / varmkt\n prior[np.eye(n) == 1] = np.diag(sample)\n\n # compute shrinkage parameters\n if self.delta is None:\n c = np.linalg.norm(sample - prior, \"fro\") ** 2\n y = x ** 2\n p = 1 / t * np.sum(np.dot(y.T, y)) - np.sum(sample ** 2)\n # r is divided into diagonal\n # and off-diagonal terms, and the off-diagonal term\n # is itself divided into smaller terms\n rdiag = 1 / t * np.sum(y ** 2) - sum(np.diag(sample) ** 2)\n z = x * np.tile(xmkt, (n,))\n v1 = 1 / t * np.dot(y.T, z) - np.tile(covmkt, (n,)) * sample\n roff1 = (\n np.sum(v1 * np.tile(covmkt, (n,)).T) / varmkt\n - np.sum(np.diag(v1) * covmkt.T) / varmkt\n )\n v3 = 1 / t * np.dot(z.T, z) - varmkt * sample\n roff3 = (\n np.sum(v3 * np.dot(covmkt, covmkt.T)) / varmkt ** 2\n - np.sum(np.diag(v3).reshape(-1, 1) * covmkt ** 2) / varmkt ** 2\n )\n roff = 2 * roff1 - roff3\n r = rdiag + roff\n\n # compute shrinkage constant\n k = (p - r) / c\n shrinkage = max(0, min(1, k / t))\n self.delta = shrinkage\n else:\n # use specified constant\n shrinkage = self.delta\n\n # compute the estimator\n sigma = shrinkage * prior + (1 - shrinkage) * sample\n return self.format_and_annualise(sigma)", "response": "Calculate the Constant - Correlation covariance matrix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the minimum variance solution", "response": "def min_volatility(self):\n \"\"\"Get the minimum variance solution\"\"\"\n if not self.w:\n self.solve()\n var = []\n for w in self.w:\n a = np.dot(np.dot(w.T, self.cov_matrix), w)\n var.append(a)\n # return min(var)**.5, self.w[var.index(min(var))]\n self.weights = self.w[var.index(min(var))].reshape((self.n_assets,))\n return dict(zip(self.tickers, self.weights))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the max Sharpe ratio portfolio", "response": "def max_sharpe(self):\n \"\"\"Get the max Sharpe ratio portfolio\"\"\"\n if not self.w:\n self.solve()\n # 1) Compute the local max SR portfolio between any two neighbor turning points\n w_sr, sr = [], []\n for i in range(len(self.w) - 1):\n w0 = np.copy(self.w[i])\n w1 = np.copy(self.w[i + 1])\n kargs = {\"minimum\": False, \"args\": (w0, w1)}\n a, b = self.golden_section(self.eval_sr, 0, 1, **kargs)\n w_sr.append(a * w0 + (1 - a) * w1)\n sr.append(b)\n # return max(sr), w_sr[sr.index(max(sr))]\n self.weights = w_sr[sr.index(max(sr))].reshape((self.n_assets,))\n return dict(zip(self.tickers, self.weights))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef efficient_frontier(self, points):\n mu, sigma, weights = [], [], []\n # remove the 1, to avoid duplications\n a = np.linspace(0, 1, points / len(self.w))[:-1]\n b = list(range(len(self.w) - 1))\n for i in b:\n w0, w1 = self.w[i], self.w[i + 1]\n if i == b[-1]:\n # include the 1 in the last iteration\n a = np.linspace(0, 1, points / len(self.w))\n for j in a:\n w = w1 * j + (1 - j) * w0\n weights.append(np.copy(w))\n mu.append(np.dot(w.T, self.mean)[0, 0])\n sigma.append(np.dot(np.dot(w.T, self.cov_matrix), w)[0, 0] ** 0.5)\n return mu, sigma, weights", "response": "Get the efficient frontier for the given set of points."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nemit a SocketIO event to one or more connected clients.", "response": "def emit(event, *args, **kwargs):\n \"\"\"Emit a SocketIO event.\n\n This function emits a SocketIO event to one or more connected clients. A\n JSON blob can be attached to the event as payload. This is a function that\n can only be called from a SocketIO event handler, as in obtains some\n information from the current client context. Example::\n\n @socketio.on('my event')\n def handle_my_custom_event(json):\n emit('my response', {'data': 42})\n\n :param event: The name of the user event to emit.\n :param args: A dictionary with the JSON data to send as payload.\n :param namespace: The namespace under which the message is to be sent.\n Defaults to the namespace used by the originating event.\n A ``'/'`` can be used to explicitly specify the global\n namespace.\n :param callback: Callback function to invoke with the client's\n acknowledgement.\n :param broadcast: ``True`` to send the message to all clients, or ``False``\n to only reply to the sender of the originating event.\n :param room: Send the message to all the users in the given room. If this\n argument is set, then broadcast is implied to be ``True``.\n :param include_self: ``True`` to include the sender when broadcasting or\n addressing a room, or ``False`` to send to everyone\n but the sender.\n :param ignore_queue: Only used when a message queue is configured. If\n set to ``True``, the event is emitted to the\n clients directly, without going through the queue.\n This is more efficient, but only works when a\n single server process is used, or when there is a\n single addresee. It is recommended to always leave\n this parameter with its default value of ``False``.\n \"\"\"\n if 'namespace' in kwargs:\n namespace = kwargs['namespace']\n else:\n namespace = flask.request.namespace\n callback = kwargs.get('callback')\n broadcast = kwargs.get('broadcast')\n room = kwargs.get('room')\n if room is None and not broadcast:\n room = flask.request.sid\n include_self = kwargs.get('include_self', True)\n ignore_queue = kwargs.get('ignore_queue', False)\n\n socketio = flask.current_app.extensions['socketio']\n return socketio.emit(event, *args, namespace=namespace, room=room,\n include_self=include_self, callback=callback,\n ignore_queue=ignore_queue)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending a SocketIO message to one or more connected clients.", "response": "def send(message, **kwargs):\n \"\"\"Send a SocketIO message.\n\n This function sends a simple SocketIO message to one or more connected\n clients. The message can be a string or a JSON blob. This is a simpler\n version of ``emit()``, which should be preferred. This is a function that\n can only be called from a SocketIO event handler.\n\n :param message: The message to send, either a string or a JSON blob.\n :param json: ``True`` if ``message`` is a JSON blob, ``False``\n otherwise.\n :param namespace: The namespace under which the message is to be sent.\n Defaults to the namespace used by the originating event.\n An empty string can be used to use the global namespace.\n :param callback: Callback function to invoke with the client's\n acknowledgement.\n :param broadcast: ``True`` to send the message to all connected clients, or\n ``False`` to only reply to the sender of the originating\n event.\n :param room: Send the message to all the users in the given room.\n :param include_self: ``True`` to include the sender when broadcasting or\n addressing a room, or ``False`` to send to everyone\n but the sender.\n :param ignore_queue: Only used when a message queue is configured. If\n set to ``True``, the event is emitted to the\n clients directly, without going through the queue.\n This is more efficient, but only works when a\n single server process is used, or when there is a\n single addresee. It is recommended to always leave\n this parameter with its default value of ``False``.\n \"\"\"\n json = kwargs.get('json', False)\n if 'namespace' in kwargs:\n namespace = kwargs['namespace']\n else:\n namespace = flask.request.namespace\n callback = kwargs.get('callback')\n broadcast = kwargs.get('broadcast')\n room = kwargs.get('room')\n if room is None and not broadcast:\n room = flask.request.sid\n include_self = kwargs.get('include_self', True)\n ignore_queue = kwargs.get('ignore_queue', False)\n\n socketio = flask.current_app.extensions['socketio']\n return socketio.send(message, json=json, namespace=namespace, room=room,\n include_self=include_self, callback=callback,\n ignore_queue=ignore_queue)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef join_room(room, sid=None, namespace=None):\n socketio = flask.current_app.extensions['socketio']\n sid = sid or flask.request.sid\n namespace = namespace or flask.request.namespace\n socketio.server.enter_room(sid, room, namespace=namespace)", "response": "Join a room.\n\n This function puts the user in a room, under the current namespace. The\n user and the namespace are obtained from the event context. This is a\n function that can only be called from a SocketIO event handler. Example::\n\n @socketio.on('join')\n def on_join(data):\n username = session['username']\n room = data['room']\n join_room(room)\n send(username + ' has entered the room.', room=room)\n\n :param room: The name of the room to join.\n :param sid: The session id of the client. If not provided, the client is\n obtained from the request context.\n :param namespace: The namespace for the room. If not provided, the\n namespace is obtained from the request context."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef leave_room(room, sid=None, namespace=None):\n socketio = flask.current_app.extensions['socketio']\n sid = sid or flask.request.sid\n namespace = namespace or flask.request.namespace\n socketio.server.leave_room(sid, room, namespace=namespace)", "response": "Leave a room.\n\n This function removes the user from a room, under the current namespace.\n The user and the namespace are obtained from the event context. Example::\n\n @socketio.on('leave')\n def on_leave(data):\n username = session['username']\n room = data['room']\n leave_room(room)\n send(username + ' has left the room.', room=room)\n\n :param room: The name of the room to leave.\n :param sid: The session id of the client. If not provided, the client is\n obtained from the request context.\n :param namespace: The namespace for the room. If not provided, the\n namespace is obtained from the request context."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclose a room. This function removes any users that are in the given room and then deletes the room from the server. :param room: The name of the room to close. :param namespace: The namespace for the room. If not provided, the namespace is obtained from the request context.", "response": "def close_room(room, namespace=None):\n \"\"\"Close a room.\n\n This function removes any users that are in the given room and then deletes\n the room from the server.\n\n :param room: The name of the room to close.\n :param namespace: The namespace for the room. If not provided, the\n namespace is obtained from the request context.\n \"\"\"\n socketio = flask.current_app.extensions['socketio']\n namespace = namespace or flask.request.namespace\n socketio.server.close_room(room, namespace=namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rooms(sid=None, namespace=None):\n socketio = flask.current_app.extensions['socketio']\n sid = sid or flask.request.sid\n namespace = namespace or flask.request.namespace\n return socketio.server.rooms(sid, namespace=namespace)", "response": "Return a list of the rooms the client is in."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndisconnecting the client. This function terminates the connection with the client. As a result of this call the client will receive a disconnect event. Example:: @socketio.on('message') def receive_message(msg): if is_banned(session['username']): disconnect() else: # ... :param sid: The session id of the client. If not provided, the client is obtained from the request context. :param namespace: The namespace for the room. If not provided, the namespace is obtained from the request context. :param silent: this option is deprecated.", "response": "def disconnect(sid=None, namespace=None, silent=False):\n \"\"\"Disconnect the client.\n\n This function terminates the connection with the client. As a result of\n this call the client will receive a disconnect event. Example::\n\n @socketio.on('message')\n def receive_message(msg):\n if is_banned(session['username']):\n disconnect()\n else:\n # ...\n\n :param sid: The session id of the client. If not provided, the client is\n obtained from the request context.\n :param namespace: The namespace for the room. If not provided, the\n namespace is obtained from the request context.\n :param silent: this option is deprecated.\n \"\"\"\n socketio = flask.current_app.extensions['socketio']\n sid = sid or flask.request.sid\n namespace = namespace or flask.request.namespace\n return socketio.server.disconnect(sid, namespace=namespace)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef on(self, message, namespace=None):\n namespace = namespace or '/'\n\n def decorator(handler):\n def _handler(sid, *args):\n return self._handle_event(handler, message, namespace, sid,\n *args)\n\n if self.server:\n self.server.on(message, _handler, namespace=namespace)\n else:\n self.handlers.append((message, _handler, namespace))\n return handler\n return decorator", "response": "Decorator to register a SocketIO event handler."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef on_error_default(self, exception_handler):\n if not callable(exception_handler):\n raise ValueError('exception_handler must be callable')\n self.default_exception_handler = exception_handler\n return exception_handler", "response": "Decorator to define a default error handler for SocketIO events."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_event(self, message, handler, namespace=None):\n self.on(message, namespace=namespace)(handler)", "response": "Register a SocketIO event handler."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nemitting a SocketIO event to one or more connected clients.", "response": "def emit(self, event, *args, **kwargs):\n \"\"\"Emit a server generated SocketIO event.\n\n This function emits a SocketIO event to one or more connected clients.\n A JSON blob can be attached to the event as payload. This function can\n be used outside of a SocketIO event context, so it is appropriate to\n use when the server is the originator of an event, outside of any\n client context, such as in a regular HTTP request handler or a\n background task. Example::\n\n @app.route('/ping')\n def ping():\n socketio.emit('ping event', {'data': 42}, namespace='/chat')\n\n :param event: The name of the user event to emit.\n :param args: A dictionary with the JSON data to send as payload.\n :param namespace: The namespace under which the message is to be sent.\n Defaults to the global namespace.\n :param room: Send the message to all the users in the given room. If\n this parameter is not included, the event is sent to\n all connected users.\n :param skip_sid: The session id of a client to ignore when broadcasting\n or addressing a room. This is typically set to the\n originator of the message, so that everyone except\n that client receive the message.\n :param callback: If given, this function will be called to acknowledge\n that the client has received the message. The\n arguments that will be passed to the function are\n those provided by the client. Callback functions can\n only be used when addressing an individual client.\n \"\"\"\n namespace = kwargs.pop('namespace', '/')\n room = kwargs.pop('room', None)\n include_self = kwargs.pop('include_self', True)\n skip_sid = kwargs.pop('skip_sid', None)\n if not include_self and not skip_sid:\n skip_sid = flask.request.sid\n callback = kwargs.pop('callback', None)\n self.server.emit(event, *args, namespace=namespace, room=room,\n skip_sid=skip_sid, callback=callback, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends a message to one or more connected SocketIO clients.", "response": "def send(self, data, json=False, namespace=None, room=None,\n callback=None, include_self=True, skip_sid=None, **kwargs):\n \"\"\"Send a server-generated SocketIO message.\n\n This function sends a simple SocketIO message to one or more connected\n clients. The message can be a string or a JSON blob. This is a simpler\n version of ``emit()``, which should be preferred. This function can be\n used outside of a SocketIO event context, so it is appropriate to use\n when the server is the originator of an event.\n\n :param data: The message to send, either a string or a JSON blob.\n :param json: ``True`` if ``message`` is a JSON blob, ``False``\n otherwise.\n :param namespace: The namespace under which the message is to be sent.\n Defaults to the global namespace.\n :param room: Send the message only to the users in the given room. If\n this parameter is not included, the message is sent to\n all connected users.\n :param skip_sid: The session id of a client to ignore when broadcasting\n or addressing a room. This is typically set to the\n originator of the message, so that everyone except\n that client receive the message.\n :param callback: If given, this function will be called to acknowledge\n that the client has received the message. The\n arguments that will be passed to the function are\n those provided by the client. Callback functions can\n only be used when addressing an individual client.\n \"\"\"\n skip_sid = flask.request.sid if not include_self else skip_sid\n if json:\n self.emit('json', data, namespace=namespace, room=room,\n skip_sid=skip_sid, callback=callback, **kwargs)\n else:\n self.emit('message', data, namespace=namespace, room=room,\n skip_sid=skip_sid, callback=callback, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(self, app, host=None, port=None, **kwargs):\n if host is None:\n host = '127.0.0.1'\n if port is None:\n server_name = app.config['SERVER_NAME']\n if server_name and ':' in server_name:\n port = int(server_name.rsplit(':', 1)[1])\n else:\n port = 5000\n\n debug = kwargs.pop('debug', app.debug)\n log_output = kwargs.pop('log_output', debug)\n use_reloader = kwargs.pop('use_reloader', debug)\n extra_files = kwargs.pop('extra_files', None)\n\n app.debug = debug\n if app.debug and self.server.eio.async_mode != 'threading':\n # put the debug middleware between the SocketIO middleware\n # and the Flask application instance\n #\n # mw1 mw2 mw3 Flask app\n # o ---- o ---- o ---- o\n # /\n # o Flask-SocketIO\n # \\ middleware\n # o\n # Flask-SocketIO WebSocket handler\n #\n # BECOMES\n #\n # dbg-mw mw1 mw2 mw3 Flask app\n # o ---- o ---- o ---- o ---- o\n # /\n # o Flask-SocketIO\n # \\ middleware\n # o\n # Flask-SocketIO WebSocket handler\n #\n self.sockio_mw.wsgi_app = DebuggedApplication(self.sockio_mw.wsgi_app,\n evalex=True)\n\n if self.server.eio.async_mode == 'threading':\n from werkzeug._internal import _log\n _log('warning', 'WebSocket transport not available. Install '\n 'eventlet or gevent and gevent-websocket for '\n 'improved performance.')\n app.run(host=host, port=port, threaded=True,\n use_reloader=use_reloader, **kwargs)\n elif self.server.eio.async_mode == 'eventlet':\n def run_server():\n import eventlet\n import eventlet.wsgi\n import eventlet.green\n addresses = eventlet.green.socket.getaddrinfo(host, port)\n if not addresses:\n raise RuntimeError('Could not resolve host to a valid address')\n eventlet_socket = eventlet.listen(addresses[0][4], addresses[0][0])\n\n # If provided an SSL argument, use an SSL socket\n ssl_args = ['keyfile', 'certfile', 'server_side', 'cert_reqs',\n 'ssl_version', 'ca_certs',\n 'do_handshake_on_connect', 'suppress_ragged_eofs',\n 'ciphers']\n ssl_params = {k: kwargs[k] for k in kwargs if k in ssl_args}\n if len(ssl_params) > 0:\n for k in ssl_params:\n kwargs.pop(k)\n ssl_params['server_side'] = True # Listening requires true\n eventlet_socket = eventlet.wrap_ssl(eventlet_socket,\n **ssl_params)\n\n eventlet.wsgi.server(eventlet_socket, app,\n log_output=log_output, **kwargs)\n\n if use_reloader:\n run_with_reloader(run_server, extra_files=extra_files)\n else:\n run_server()\n elif self.server.eio.async_mode == 'gevent':\n from gevent import pywsgi\n try:\n from geventwebsocket.handler import WebSocketHandler\n websocket = True\n except ImportError:\n websocket = False\n\n log = 'default'\n if not log_output:\n log = None\n if websocket:\n self.wsgi_server = pywsgi.WSGIServer(\n (host, port), app, handler_class=WebSocketHandler,\n log=log, **kwargs)\n else:\n self.wsgi_server = pywsgi.WSGIServer((host, port), app,\n log=log, **kwargs)\n\n if use_reloader:\n # monkey patching is required by the reloader\n from gevent import monkey\n monkey.patch_all()\n\n def run_server():\n self.wsgi_server.serve_forever()\n\n run_with_reloader(run_server, extra_files=extra_files)\n else:\n self.wsgi_server.serve_forever()", "response": "This method runs the SocketIO web server."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nstops a running SocketIO web server.", "response": "def stop(self):\n \"\"\"Stop a running SocketIO web server.\n\n This method must be called from a HTTP or SocketIO handler function.\n \"\"\"\n if self.server.eio.async_mode == 'threading':\n func = flask.request.environ.get('werkzeug.server.shutdown')\n if func:\n func()\n else:\n raise RuntimeError('Cannot stop unknown web server')\n elif self.server.eio.async_mode == 'eventlet':\n raise SystemExit\n elif self.server.eio.async_mode == 'gevent':\n self.wsgi_server.stop()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef start_background_task(self, target, *args, **kwargs):\n return self.server.start_background_task(target, *args, **kwargs)", "response": "Start a background task using the appropriate async model."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndispatch an event to the proper handler method.", "response": "def trigger_event(self, event, *args):\n \"\"\"Dispatch an event to the proper handler method.\n\n In the most common usage, this method is not overloaded by subclasses,\n as it performs the routing of events to methods. However, this\n method can be overriden if special dispatching rules are needed, or if\n having a single method that catches all events is desired.\n \"\"\"\n handler_name = 'on_' + event\n if not hasattr(self, handler_name):\n # there is no handler for this event, so we ignore it\n return\n handler = getattr(self, handler_name)\n return self.socketio._handle_event(handler, event, self.namespace,\n *args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nemit a custom event to one or more connected clients.", "response": "def emit(self, event, data=None, room=None, include_self=True,\n namespace=None, callback=None):\n \"\"\"Emit a custom event to one or more connected clients.\"\"\"\n return self.socketio.emit(event, data, room=room,\n include_self=include_self,\n namespace=namespace or self.namespace,\n callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send(self, data, room=None, include_self=True, namespace=None,\n callback=None):\n \"\"\"Send a message to one or more connected clients.\"\"\"\n return self.socketio.send(data, room=room, include_self=include_self,\n namespace=namespace or self.namespace,\n callback=callback)", "response": "Send a message to one or more connected clients."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef background_thread():\n count = 0\n while True:\n socketio.sleep(10)\n count += 1\n socketio.emit('my_response',\n {'data': 'Server generated event', 'count': count},\n namespace='/test')", "response": "Example of how to send server generated events to clients."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating query status and send email notification to a user", "response": "def on_failure(self, exc, task_id, args, kwargs, einfo):\n \"\"\"Update query status and send email notification to a user\"\"\"\n super().on_failure(exc, task_id, args, kwargs, einfo)\n if isinstance(exc, OperationRunDoesNotExist):\n return\n self._operation_run.on_failure()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef on_success(self, retval, task_id, args, kwargs):\n super().on_success(retval, task_id, args, kwargs)\n self._operation_run.on_success()", "response": "Send email notification and a file if requested to do so by a user"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrecording the event async.", "response": "def record_event(self, event: Event) -> None:\n \"\"\"\n Record the event async.\n \"\"\"\n from polyaxon.celery_api import celery_app\n from polyaxon.settings import EventsCeleryTasks\n\n if not event.ref_id:\n event.ref_id = self.get_ref_id()\n serialized_event = event.serialize(dumps=False,\n include_actor_name=True,\n include_instance_info=True)\n\n celery_app.send_task(EventsCeleryTasks.EVENTS_TRACK, kwargs={'event': serialized_event})\n celery_app.send_task(EventsCeleryTasks.EVENTS_LOG, kwargs={'event': serialized_event})\n celery_app.send_task(EventsCeleryTasks.EVENTS_NOTIFY, kwargs={'event': serialized_event})\n # We include the instance in the serialized event for executor\n serialized_event['instance'] = event.instance\n self.executor.record(event_type=event.event_type, event_data=serialized_event)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the list of event types where use acted on an object.", "response": "def user_write_events(self) -> List[str]:\n \"\"\"Return event types where use acted on an object.\n\n The write events are events with actions:\n * CREATED\n * UPDATED\n * DELETED\n * RESUMED\n * COPIED\n * CLONED\n * STOPPED\n \"\"\"\n return [event_type for event_type, event in self.items if event.get_event_action()\n in event_actions.WRITE_ACTIONS]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef user_view_events(self) -> List[str]:\n return [event_type for event_type, event in self.items if event.get_event_action()\n == event_actions.VIEWED]", "response": "Return event types where use viewed a main object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_new_email(email):\n email = normalize_email(email)\n User = get_user_model() # noqa\n if User.objects.filter(email=email).exists():\n raise ValidationError('The given email address is already registered.')", "response": "Validates a new email address by checking if it is already used by other users."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_project_pod_spec(volume_mounts,\n volumes,\n image,\n command,\n args,\n ports,\n env_vars=None,\n env_from=None,\n container_name=None,\n resources=None,\n node_selector=None,\n affinity=None,\n tolerations=None,\n image_pull_policy=None,\n restart_policy=None,\n service_account_name=None):\n \"\"\"Pod spec to be used to create pods for project: tensorboard, notebooks.\"\"\"\n volume_mounts = to_list(volume_mounts, check_none=True)\n volumes = to_list(volumes, check_none=True)\n\n gpu_volume_mounts, gpu_volumes = get_gpu_volumes_def(resources)\n volume_mounts += gpu_volume_mounts\n volumes += gpu_volumes\n\n ports = [client.V1ContainerPort(container_port=port) for port in ports]\n\n pod_container = get_pod_container(\n volume_mounts=volume_mounts,\n image=image,\n command=command,\n args=args,\n ports=ports,\n env_vars=env_vars,\n env_from=env_from,\n container_name=container_name,\n resources=resources,\n image_pull_policy=image_pull_policy)\n containers = [pod_container]\n\n if service_account_name and not conf.get('K8S_RBAC_ENABLED'):\n service_account_name = None\n\n return client.V1PodSpec(restart_policy=restart_policy,\n security_context=get_security_context(),\n service_account_name=service_account_name,\n containers=containers,\n volumes=volumes,\n node_selector=node_selector,\n affinity=affinity,\n tolerations=tolerations)", "response": "Returns a pod spec for the project."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_object(self):\n if self._object:\n return self._object\n self._object = super().get_object()\n if not self.AUDITOR_EVENT_TYPES:\n return self._object\n method = self.request.method\n event_type = self.AUDITOR_EVENT_TYPES.get(method)\n if method == 'GET' and event_type and is_user(self.request.user):\n auditor.record(event_type=event_type,\n instance=self._object,\n actor_id=self.request.user.id,\n actor_name=self.request.user.username)\n elif method == 'DELETE' and event_type and is_user(self.request.user):\n auditor.record(event_type=event_type,\n instance=self._object,\n actor_id=self.request.user.id,\n actor_name=self.request.user.username)\n return self._object", "response": "We memoize the access to this function in case a second call is made."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef initialize_context(self, request: HttpRequest, *args, **kwargs) -> None:\n for key in self.CONTEXT_KEYS:\n setattr(self, key, kwargs.get(key))\n for key in self.QUERY_CONTEXT_KEYS:\n setattr(self, key, request.query_params.get(key))\n self._initialize_context()\n self.validate_context()", "response": "Initializes the endpoint with the passed context keys based on the passed query parameters and the passed request."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_shm_volumes():\n volumes, volume_mounts = [], []\n shm_volume = client.V1Volume(\n name=constants.SHM_VOLUME,\n empty_dir=client.V1EmptyDirVolumeSource(medium='Memory')\n )\n volumes.append(shm_volume)\n shm_volume_mount = client.V1VolumeMount(name=shm_volume.name, mount_path='/dev/shm')\n volume_mounts.append(shm_volume_mount)\n return volumes, volume_mounts", "response": "Get the list of volumes and volumes_mounts for the current project."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a boolean indicating if the experiment has any running jobs", "response": "def has_running_jobs(self) -> bool:\n \"\"\"\"Return a boolean indicating if the experiment has any running jobs\"\"\"\n return self.jobs.exclude(status__status__in=ExperimentLifeCycle.DONE_STATUS).exists()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the internal header of the request.", "response": "def get_internal_header(request: HttpRequest) -> str:\n \"\"\"\n Return request's 'X_POLYAXON_INTERNAL:' header, as a bytestring.\n \"\"\"\n return get_header(request=request, header_service=conf.get('HEADERS_INTERNAL'))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsignaling handling for sidecars logs.", "response": "def logs_sidecars_experiments(experiment_name: str,\n experiment_uuid: str,\n job_uuid: str,\n log_lines: Optional[Union[str, Iterable[str]]]) -> None:\n \"\"\"Signal handling for sidecars logs.\"\"\"\n handle_experiment_job_log(experiment_name=experiment_name,\n experiment_uuid=experiment_uuid,\n log_lines=log_lines)\n publisher.publish_experiment_job_log(\n log_lines=log_lines,\n experiment_uuid=experiment_uuid,\n experiment_name=experiment_name,\n job_uuid=job_uuid,\n send_task=False\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsignals handling for sidecars logs.", "response": "def logs_sidecars_jobs(job_uuid: str,\n job_name: str,\n log_lines: Optional[Union[str, Iterable[str]]]) -> None:\n \"\"\"Signal handling for sidecars logs.\"\"\"\n handle_job_logs(job_uuid=job_uuid,\n job_name=job_name,\n log_lines=log_lines)\n publisher.publish_job_log(\n log_lines=log_lines,\n job_uuid=job_uuid,\n job_name=job_name,\n send_task=False\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsignal handling for sidecars logs.", "response": "def logs_sidecars_builds(job_uuid: str,\n job_name: str,\n log_lines: Optional[Union[str, Iterable[str]]]) -> None:\n \"\"\"Signal handling for sidecars logs.\"\"\"\n handle_build_job_logs(job_uuid=job_uuid,\n job_name=job_name,\n log_lines=log_lines)\n publisher.publish_build_job_log(\n log_lines=log_lines,\n job_uuid=job_uuid,\n job_name=job_name,\n send_task=False\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if one task ran ( checked ).", "response": "def is_checked(self) -> bool:\n \"\"\"One task ran (checked).\"\"\"\n if not self.redis_key_checked:\n return False\n\n value = self._red.get(self.redis_key_checked)\n if not value:\n return False\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if one task ran and one task has been delayed.", "response": "def is_delayed(self) -> bool:\n \"\"\"One task ran (checked), and one task has been delayed.\"\"\"\n if not self.redis_key_delayed:\n return False\n\n value = self._red.get(self.redis_key_delayed)\n if not value:\n return False\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef connect(self):\n _logger.info('Connecting to %s', self.AMQP_URL)\n while True:\n try:\n return adapters.AsyncioConnection(pika.URLParameters(self.AMQP_URL),\n self.on_connection_open)\n except AMQPConnectionError:\n asyncio.sleep(1)", "response": "This method connects to RabbitMQ returning the connection handle."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef setup_exchange(self, exchange_name):\n _logger.debug('Declaring exchange %s', exchange_name)\n self._channel.exchange_declare(self.on_exchange_declareok,\n exchange_name,\n self.EXCHANGE_TYPE,\n durable=True,\n passive=True)", "response": "Declare an exchange on RabbitMQ by invoking the Exchange. Declare RPC\n command."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndeclare a queue on RabbitMQ by invoking the Queue. Declare RPC command.", "response": "def setup_queue(self, queue_name):\n \"\"\"Setup the queue on RabbitMQ by invoking the Queue.Declare RPC\n command. When it is complete, the on_queue_declareok method will\n be invoked by pika.\n\n :param str|unicode queue_name: The name of the queue to declare.\n \"\"\"\n _logger.debug('Declaring queue %s', queue_name)\n self._channel.queue_declare(self.on_queue_declareok, queue_name)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef on_queue_declareok(self, method_frame):\n _logger.info('Binding %s to %s with %s',\n self.EXCHANGE, self._queue, self._routing_key)\n self._channel.queue_bind(self.on_bindok, self._queue,\n self.EXCHANGE, self._routing_key)", "response": "Invoked by pika when the Queue. Declare RPC call made by pika to bind the queue to the exchange together with the routing key."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninvoking by pika when a Basic. Cancel frame is received.", "response": "def on_consumer_cancelled(self, method_frame):\n \"\"\"Invoked by pika when RabbitMQ sends a Basic.Cancel for a consumer\n receiving messages.\n\n :param pika.frame.Method method_frame: The Basic.Cancel frame\n \"\"\"\n _logger.info('Consumer was cancelled remotely, shutting down: %r', method_frame)\n if self._channel:\n self._channel.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef acknowledge_message(self, delivery_tag):\n _logger.debug('Acknowledging message %s', delivery_tag)\n self._channel.basic_ack(delivery_tag)", "response": "Acknowledge the delivery from RabbitMQ by sending Basic. Ack RPC method for the delivery tag."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef on_message(self, unused_channel, basic_deliver, properties, body):\n _logger.debug('Received message # %s from %s: %s',\n basic_deliver.delivery_tag, properties.app_id, body)\n if self.ws and body:\n self.messages.append(body)\n _logger.debug('out ws : %s', len(self.ws))\n _logger.debug('out messages : %s', len(self.messages))\n self.acknowledge_message(basic_deliver.delivery_tag)", "response": "Invoked by pika when a message is delivered from RabbitMQ."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntells RabbitMQ that you would like to stop consuming.", "response": "def stop_consuming(self):\n \"\"\"Tell RabbitMQ that you would like to stop consuming by sending the\n Basic.Cancel RPC command.\n \"\"\"\n if self._channel:\n _logger.debug('Sending a Basic.Cancel RPC command to RabbitMQ')\n self._channel.basic_cancel(self.on_cancelok, self._consumer_tag)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nopening a new channel with RabbitMQ by issuing the Channel. Open RPC command.", "response": "def open_channel(self):\n \"\"\"Open a new channel with RabbitMQ by issuing the Channel.Open RPC\n command. When RabbitMQ responds that the channel is open, the\n on_channel_open callback will be invoked by pika.\n \"\"\"\n _logger.info('Creating a new channel')\n self._connection.channel(on_open_callback=self.on_channel_open)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(self):\n self._connection = self.connect()\n self._connection.ioloop.start()", "response": "Run the example consumer by connecting to RabbitMQ and then starting the IOLoop to block and allow the SelectConnection to operate."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstopping the connection to RabbitMQ by stopping the consumer and closing the connection.", "response": "def stop(self):\n \"\"\"Cleanly shutdown the connection to RabbitMQ by stopping the consumer\n with RabbitMQ. When RabbitMQ confirms the cancellation, on_cancelok\n will be invoked by pika, which will then closing the channel and\n connection. The IOLoop is started again because this method is invoked\n when CTRL-C is pressed raising a KeyboardInterrupt exception. This\n exception stops the IOLoop which needs to be running for pika to\n communicate with RabbitMQ. All of the commands issued prior to starting\n the IOLoop will be buffered but not processed.\n \"\"\"\n _logger.debug('Stopping')\n self._closing = True\n self.stop_consuming()\n self._connection.ioloop.start()\n _logger.info('Stopped')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrendering both the subject and body templates in the given context.", "response": "def render_mail_template(subject_template, body_template, context):\n \"\"\"\n Renders both the subject and body templates in the given context.\n Returns a tuple (subject, body) of the result.\n \"\"\"\n try:\n subject = strip_spaces(render_to_string(subject_template, context))\n body = render_to_string(body_template, context)\n finally:\n pass\n\n return subject, body"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send_mass_template_mail(subject_template, body_template, recipients, context=None):\n if context:\n subject, body = render_mail_template(subject_template, body_template, context)\n else:\n subject, body = subject_template, body_template\n\n message_tuples = [(subject, body, conf.get('DEFAULT_FROM_EMAIL'), [r]) for r in recipients]\n\n send_mass_mail(message_tuples)", "response": "Sends an email using the given templates and context."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a tar file based on the list of files passed", "response": "def create_tarfile(files: List[str], tar_path: str) -> None:\n \"\"\"Create a tar file based on the list of files passed\"\"\"\n with tarfile.open(tar_path, \"w:gz\") as tar:\n for f in files:\n tar.add(f)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nopen and close the broker channel.", "response": "def check() -> Result:\n \"\"\"Open and close the broker channel.\"\"\"\n try:\n # Context to release connection\n with Connection(conf.get('CELERY_BROKER_URL')) as conn:\n conn.connect()\n except ConnectionRefusedError:\n return Result(message='Service unable to connect, \"Connection was refused\".',\n severity=Result.ERROR)\n\n except AccessRefused:\n return Result(message='Service unable to connect, \"Authentication error\".',\n severity=Result.ERROR)\n\n except IOError:\n return Result(message='Service has an \"IOError\".', severity=Result.ERROR)\n\n except Exception as e:\n return Result(message='Service has an \"{}\" error.'.format(e), severity=Result.ERROR)\n else:\n return Result()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_suggestions(self, iteration_config=None):\n matrix = self.hptuning_config.matrix\n\n suggestions = []\n keys = list(matrix.keys())\n values = [v.to_numpy() for v in matrix.values()]\n for v in itertools.product(*values):\n suggestions.append(dict(zip(keys, v)))\n\n if self.hptuning_config.grid_search:\n n_suggestions = self.hptuning_config.grid_search.n_experiments\n if n_suggestions:\n return suggestions[:n_suggestions]\n return suggestions", "response": "Returns a list of suggestions based on grid search."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind the argmax of the acquisition function.", "response": "def _maximize(self):\n \"\"\" Find argmax of the acquisition function.\"\"\"\n if not self.space.is_observations_valid():\n return None\n y_max = self.space.y.max()\n self.utility_function.gaussian_process.fit(self.space.x, self.space.y)\n return self.utility_function.max_compute(y_max=y_max,\n bounds=self.space.bounds,\n n_warmup=self.n_warmup,\n n_iter=self.n_iter)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dag(self) -> Tuple[Dict, Dict]:\n from pipelines import dags\n\n operations = self.operations.all().prefetch_related('downstream_operations')\n\n def get_downstream(op):\n return op.downstream_operations.values_list('id', flat=True)\n\n return dags.get_dag(operations, get_downstream)", "response": "Construct the DAG of this pipeline based on the operations and their downstream."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the countdown for a celery task retry.", "response": "def get_countdown(self, retries) -> int:\n \"\"\"Calculate the countdown for a celery task retry.\"\"\"\n retry_delay = self.retry_delay\n if self.retry_exponential_backoff:\n return min(\n max(2 ** retries, retry_delay), # Exp. backoff\n self.max_retry_delay # The countdown should be more the max allowed\n )\n return retry_delay"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the params to run the celery task.", "response": "def get_run_params(self) -> Dict:\n \"\"\"Return the params to run the celery task.\"\"\"\n params = {}\n if self.celery_queue:\n params['queue'] = self.celery_queue\n\n if self.timeout:\n params['soft_time_limit'] = self.timeout\n # We set also a hard time limit that will send sig 9\n # This hard time limit should not happened, as it will set inconsistent state\n params['time_limit'] = self.timeout + settings.CELERY_HARD_TIME_LIMIT_DELAY\n\n if self.execute_at:\n params['eta'] = self.execute_at\n\n return params"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef can_transition(self, status_from: str, status_to: str) -> bool:\n if not self.STATUSES.can_transition(status_from=status_from, status_to=status_to):\n _logger.info(\n '`%s` tried to transition from status `%s` to non permitted status `%s`',\n str(self), status_from, status_to)\n return False\n\n return True", "response": "Check if the current instance can be updated."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconstructs the DAG of this pipeline run based on the operation runs and their downstream runs.", "response": "def dag(self) -> Tuple[Dict, Dict]:\n \"\"\"Construct the DAG of this pipeline run\n based on the its operation runs and their downstream.\n \"\"\"\n from pipelines import dags\n\n operation_runs = self.operation_runs.all().prefetch_related('downstream_runs')\n\n def get_downstream(op_run):\n return op_run.downstream_runs.values_list('id', flat=True)\n\n return dags.get_dag(operation_runs, get_downstream)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_concurrency(self) -> bool:\n if not self.operation.concurrency: # No concurrency set\n return True\n\n ops_count = self.operation.runs.filter(\n status__status__in=self.STATUSES.RUNNING_STATUS).count()\n return ops_count < self.operation.concurrency", "response": "Checks the concurrency of the operation run."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the upstream and the trigger rule.", "response": "def check_upstream_trigger(self) -> bool:\n \"\"\"Checks the upstream and the trigger rule.\"\"\"\n if self.operation.trigger_policy == TriggerPolicy.ONE_DONE:\n return self.upstream_runs.filter(\n status__status__in=self.STATUSES.DONE_STATUS).exists()\n if self.operation.trigger_policy == TriggerPolicy.ONE_SUCCEEDED:\n return self.upstream_runs.filter(\n status__status=self.STATUSES.SUCCEEDED).exists()\n if self.operation.trigger_policy == TriggerPolicy.ONE_FAILED:\n return self.upstream_runs.filter(\n status__status=self.STATUSES.FAILED).exists()\n\n statuses = self.upstream_runs.values_list('status__status', flat=True)\n if self.operation.trigger_policy == TriggerPolicy.ALL_DONE:\n return not bool([True for status in statuses\n if status not in self.STATUSES.DONE_STATUS])\n if self.operation.trigger_policy == TriggerPolicy.ALL_SUCCEEDED:\n return not bool([True for status in statuses\n if status != self.STATUSES.SUCCEEDED])\n if self.operation.trigger_policy == TriggerPolicy.ALL_FAILED:\n return not bool([True for status in statuses\n if status not in self.STATUSES.FAILED_STATUS])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nschedules the task: check first if the task can start: 1. we check that the task is still in the CREATED state. 2. we check that the upstream dependency is met. 3. we check that pipeline can start a new task; i.e. we check the concurrency of the pipeline. 4. we check that operation can start a new instance; i.e. we check the concurrency of the operation. -> If all checks pass we schedule the task start it. -> 1. If the operation is not in created status, nothing to do. -> 2. If the upstream dependency check is not met, two use cases need to be validated: * The upstream dependency is not met but could be met in the future, because some ops are still CREATED/SCHEDULED/RUNNING/... in this case nothing need to be done, every time an upstream operation finishes, it will notify all the downstream ops including this one. * The upstream dependency is not met and could not be met at all. In this case we need to mark the task with `UPSTREAM_FAILED`. -> 3. If the pipeline has reached it's concurrency limit, we just delay schedule based on the interval/time delay defined by the user. The pipeline scheduler will keep checking until the task can be scheduled or stopped. -> 4. If the operation has reached it's concurrency limit, Same as above we keep trying based on an interval defined by the user. Returns: boolean: Whether to try to schedule this operation run in the future or not.", "response": "def schedule_start(self) -> bool:\n \"\"\"Schedule the task: check first if the task can start:\n 1. we check that the task is still in the CREATED state.\n 2. we check that the upstream dependency is met.\n 3. we check that pipeline can start a new task;\n i.e. we check the concurrency of the pipeline.\n 4. we check that operation can start a new instance;\n i.e. we check the concurrency of the operation.\n\n -> If all checks pass we schedule the task start it.\n\n -> 1. If the operation is not in created status, nothing to do.\n -> 2. If the upstream dependency check is not met, two use cases need to be validated:\n * The upstream dependency is not met but could be met in the future,\n because some ops are still CREATED/SCHEDULED/RUNNING/...\n in this case nothing need to be done, every time an upstream operation finishes,\n it will notify all the downstream ops including this one.\n * The upstream dependency is not met and could not be met at all.\n In this case we need to mark the task with `UPSTREAM_FAILED`.\n -> 3. If the pipeline has reached it's concurrency limit,\n we just delay schedule based on the interval/time delay defined by the user.\n The pipeline scheduler will keep checking until the task can be scheduled or stopped.\n -> 4. If the operation has reached it's concurrency limit,\n Same as above we keep trying based on an interval defined by the user.\n\n Returns:\n boolean: Whether to try to schedule this operation run in the future or not.\n \"\"\"\n if self.last_status != self.STATUSES.CREATED:\n return False\n\n upstream_trigger_check = self.check_upstream_trigger()\n if not upstream_trigger_check and self.is_upstream_done:\n # This task cannot be scheduled anymore\n self.on_upstream_failed()\n return False\n\n if not self.pipeline_run.check_concurrency():\n return True\n\n if not self.check_concurrency():\n return True\n\n self.on_scheduled()\n self.start()\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nstarts the celery task of this operation.", "response": "def start(self) -> None:\n \"\"\"Start the celery task of this operation.\"\"\"\n kwargs = self.celery_task_context\n # Update we the operation run id\n kwargs['operation_run_id'] = self.id # pylint:disable=unsupported-assignment-operation\n\n async_result = celery_app.send_task(\n self.operation.celery_task,\n kwargs=kwargs,\n **self.operation.get_run_params())\n self.celery_task_id = async_result.id\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the image name and image tag for an experiment", "response": "def get_experiment_image_info(experiment: 'Experiment') -> Tuple[str, str]:\n \"\"\"Return the image name and image tag for an experiment\"\"\"\n project_name = experiment.project.name\n repo_name = project_name\n\n image_name = '{}/{}'.format(conf.get('REGISTRY_URI'), repo_name)\n image_tag = experiment.code_reference.commit\n return image_name, image_tag"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_job_image_info(project: 'Project', job: Any) -> Tuple[str, str]:\n project_name = project.name\n repo_name = project_name\n\n image_name = '{}/{}'.format(conf.get('REGISTRY_URI'), repo_name)\n try:\n last_commit = project.repo.last_commit\n except ValueError:\n raise ValueError('Repo was not found for project `{}`.'.format(project))\n return image_name, last_commit[0]", "response": "Return the image name and image tag for a job"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_notebook_image_info(project: 'Project', job: Any) -> Tuple[str, str]:\n image_name, _ = get_job_image_info(project, job)\n return image_name, LATEST_IMAGE_TAG", "response": "Return the image name and image tag for a job"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate an iteration for the experiment group.", "response": "def create_iteration(self, num_suggestions):\n \"\"\"Create an iteration for the experiment group.\"\"\"\n from db.models.experiment_groups import ExperimentGroupIteration\n\n iteration_config = self.experiment_group.iteration_config\n\n if iteration_config is None:\n iteration = 0\n old_experiment_ids = None\n old_experiments_configs = None\n old_experiments_metrics = None\n else:\n iteration = iteration_config.iteration + 1\n old_experiment_ids = iteration_config.combined_experiment_ids\n old_experiments_configs = iteration_config.combined_experiments_configs\n old_experiments_metrics = iteration_config.combined_experiments_metrics\n\n # Create a new iteration config\n iteration_config = BOIterationConfig(\n iteration=iteration,\n num_suggestions=num_suggestions,\n old_experiment_ids=old_experiment_ids,\n old_experiments_configs=old_experiments_configs,\n old_experiments_metrics=old_experiments_metrics,\n experiment_ids=[],\n experiments_configs=[],\n )\n return ExperimentGroupIteration.objects.create(\n experiment_group=self.experiment_group,\n data=iteration_config.to_dict())"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_iteration(self):\n iteration_config = self.get_iteration_config()\n if not iteration_config:\n return\n experiments_metrics = self.experiment_group.get_experiments_metrics(\n experiment_ids=iteration_config.experiment_ids,\n metric=self.get_metric_name()\n )\n experiments_configs = self.experiment_group.get_experiments_declarations(\n experiment_ids=iteration_config.experiment_ids\n )\n iteration_config.experiments_configs = list(experiments_configs)\n iteration_config.experiments_metrics = [m for m in experiments_metrics if m[1] is not None]\n self._update_config(iteration_config)", "response": "Update the last experiment group s iteration with experiment performance."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_n_config_to_keep(self, n_suggestions, bracket_iteration):\n n_configs = n_suggestions * (self.eta ** -bracket_iteration)\n return int(n_configs / self.eta)", "response": "Return the number of configs to keep and resume."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_n_config_to_keep_for_iteration(self, iteration, bracket_iteration):\n bracket = self.get_bracket(iteration=iteration)\n if bracket_iteration == bracket + 1:\n # End of loop `for bracket_iteration in range(bracket + 1):`\n return 0\n\n n_configs = self.get_n_configs(bracket=bracket)\n return self.get_n_config_to_keep(\n n_suggestions=n_configs, bracket_iteration=bracket_iteration)", "response": "Return the number of configs to keep for an iteration and bracket."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the number of resources for this iteration", "response": "def get_n_resources_for_iteration(self, iteration, bracket_iteration):\n \"\"\"Return the number of iterations to run for this barcket_i\n\n This is just util function around `get_n_resources`\n \"\"\"\n bracket = self.get_bracket(iteration=iteration)\n n_resources = self.get_resources(bracket=bracket)\n return self.get_n_resources(n_resources=n_resources, bracket_iteration=bracket_iteration)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_suggestions(self, iteration_config=None):\n if not iteration_config or not isinstance(iteration_config, HyperbandIterationConfig):\n raise ValueError('Hyperband get suggestions requires an iteration.')\n bracket = self.get_bracket(iteration=iteration_config.iteration)\n n_configs = self.get_n_configs(bracket=bracket)\n n_resources = self.get_n_resources_for_iteration(\n iteration=iteration_config.iteration,\n bracket_iteration=iteration_config.bracket_iteration)\n n_resources = self.hptuning_config.hyperband.resource.cast_value(n_resources)\n suggestion_params = {\n self.hptuning_config.hyperband.resource.name: n_resources\n }\n return get_random_suggestions(matrix=self.hptuning_config.matrix,\n n_suggestions=n_configs,\n suggestion_params=suggestion_params,\n seed=self.hptuning_config.seed)", "response": "Return a list of suggestions based on hyperband."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef should_reschedule(self, iteration, bracket_iteration):\n bracket = self.get_bracket(iteration=iteration)\n if bracket_iteration < bracket:\n # The bracket is still processing\n return False\n\n # We can only reschedule if we can create a new bracket\n return self.get_bracket(iteration=iteration + 1) >= 0", "response": "Return a boolean to indicate if we need to reschedule another iteration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef should_reduce_configs(self, iteration, bracket_iteration):\n n_configs_to_keep = self.get_n_config_to_keep_for_iteration(\n iteration=iteration, bracket_iteration=bracket_iteration)\n return n_configs_to_keep > 0", "response": "Return a boolean to indicate if we need to reschedule another bracket iteration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_init_container(self,\n init_command,\n init_args,\n env_vars,\n context_mounts,\n persistence_outputs,\n persistence_data):\n \"\"\"Pod init container for setting outputs path.\"\"\"\n env_vars = to_list(env_vars, check_none=True)\n if self.original_name is not None and self.cloning_strategy == CloningStrategy.RESUME:\n return []\n if self.original_name is not None and self.cloning_strategy == CloningStrategy.COPY:\n command = InitCommands.COPY\n original_outputs_path = stores.get_experiment_outputs_path(\n persistence=persistence_outputs,\n experiment_name=self.original_name)\n else:\n command = InitCommands.CREATE\n original_outputs_path = None\n\n outputs_path = stores.get_experiment_outputs_path(\n persistence=persistence_outputs,\n experiment_name=self.experiment_name)\n _, outputs_volume_mount = get_pod_outputs_volume(persistence_outputs=persistence_outputs)\n volume_mounts = outputs_volume_mount + to_list(context_mounts, check_none=True)\n init_command = init_command or [\"/bin/sh\", \"-c\"]\n init_args = init_args or to_list(\n get_output_args(command=command,\n outputs_path=outputs_path,\n original_outputs_path=original_outputs_path))\n init_args += to_list(get_auth_context_args(entity='experiment',\n entity_name=self.experiment_name))\n return [\n client.V1Container(\n name=self.init_container_name,\n image=self.init_docker_image,\n image_pull_policy=self.init_docker_image_pull_policy,\n command=init_command,\n args=[''.join(init_args)],\n env=env_vars,\n volume_mounts=volume_mounts)\n ]", "response": "Get the init container for setting outputs path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef path(self) -> str:\n return os.path.join(self.project_path, self.project.name)", "response": "The path of the git repository."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef last_commit(self) -> Tuple:\n from libs.repos import git\n\n return git.get_last_commit(repo_path=self.path)", "response": "Returns a tuple of hash and commit object"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate that we can only authenticate to one S3 and one GCS.", "response": "def validate_stores_secrets_keys(stores_secrets):\n \"\"\"Validates that we can only authenticate to one S3 and one GCS.\"\"\"\n stores = set([])\n for store_secret in stores_secrets:\n if store_secret['store'] in stores:\n raise TensorboardValidation('Received an invalid store configuration.')\n elif store_secret['store'] not in {GCS, S3}:\n raise TensorboardValidation('Received an unsupported store configuration.')\n stores.add(store_secret['store'])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nhandles the case of GCS and S3 and create a volume with secret file.", "response": "def get_stores_secrets_volumes(cls, stores_secrets):\n \"\"\"Handles the case of GCS and S3 and create a volume with secret file.\"\"\"\n volumes = []\n volume_mounts = []\n for store_secret in stores_secrets:\n store = store_secret['store']\n if store in {GCS, S3}:\n secrets_volumes, secrets_volume_mounts = get_volume_from_secret(\n volume_name=cls.STORE_SECRET_VOLUME_NAME.format(store),\n mount_path=cls.STORE_SECRET_KEY_MOUNT_PATH.format(store),\n secret_name=store_secret['persistence_secret'],\n )\n volumes += secrets_volumes\n volume_mounts += secrets_volume_mounts\n\n return volumes, volume_mounts"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate an auth command for S3 and GCS.", "response": "def get_stores_secrets_command_args(cls, stores_secrets):\n \"\"\"Create an auth command for S3 and GCS.\"\"\"\n commands = []\n for store_secret in stores_secrets:\n store = store_secret['store']\n if store == GCS:\n commands.append('export GOOGLE_APPLICATION_CREDENTIALS={}'.format(\n cls.STORE_SECRET_KEY_MOUNT_PATH.format(store) + '/' +\n store_secret['persistence_secret_key']\n ))\n elif store == S3:\n commands.append(\n \"import json; data = json.loads(open('{}').read()); content = []; [content.append('export {}={}'.format(k, data[k])) for k in data]; output = open('{}', 'w'); output.write('\\n'.join(content)); output.close()\".format( # noqa\n cls.STORE_SECRET_KEY_MOUNT_PATH.format(store) + '/' +\n store_secret['persistence_secret_key'],\n cls.STORE_SECRET_KEY_MOUNT_PATH.format('envs3'),\n ))\n commands.append(\"source {}\".format(\n cls.STORE_SECRET_KEY_MOUNT_PATH.format('envs3')))\n\n return commands"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a token to be used for a user account to access the resource.", "response": "def make_token(cls, ephemeral_token: 'RedisEphemeralTokens') -> str:\n \"\"\"\n Returns a token to be used x number of times to allow a user account to access\n certain resource.\n \"\"\"\n value = ephemeral_token.key\n if ephemeral_token.scope:\n value += ''.join(ephemeral_token.scope)\n\n return get_hmac(cls.KEY_SALT + ephemeral_token.salt, value)[::2]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check_token(self, token: 'RedisEphemeralTokens') -> bool:\n if self.get_state() is None: # Token expired\n return False\n\n correct_token = self.make_token(self)\n self.clear()\n return constant_time_compare(correct_token, token)", "response": "Check that a given token is correct for a given scope."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_ordering(self, request, queryset, view):\n params = request.query_params.get(self.ordering_param)\n if params:\n fields = strip_spaces(value=params, sep=',', join=False)\n ordering, annotations = self.remove_invalid_fields(queryset, fields, view, request)\n if ordering:\n return ordering, annotations\n\n # No ordering was included, or all the ordering fields were invalid\n return self.get_default_ordering(view), None", "response": "Returns the ordering for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the init container for setting outputs path.", "response": "def get_init_container(self,\n init_command,\n init_args,\n env_vars,\n context_mounts,\n persistence_outputs,\n persistence_data):\n \"\"\"Pod init container for setting outputs path.\"\"\"\n env_vars = to_list(env_vars, check_none=True)\n volume_mounts = to_list(context_mounts, check_none=True)\n init_command = init_command or [\"/bin/sh\", \"-c\"]\n init_args = to_list(get_auth_context_args(entity='notebook', entity_name=self.job_name))\n return client.V1Container(\n name=self.init_container_name,\n image=self.init_docker_image,\n image_pull_policy=self.init_docker_image_pull_policy,\n command=init_command,\n args=init_args,\n env=env_vars,\n volume_mounts=volume_mounts)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate the activation key which will be emailed to the user.", "response": "def get_activation_key(self, user):\n \"\"\"\n Generate the activation key which will be emailed to the user.\n\n \"\"\"\n return signing.dumps(\n obj=getattr(user, user.USERNAME_FIELD),\n salt=self.key_salt\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send_activation_email(self, user):\n activation_key = self.get_activation_key(user)\n context = self.get_email_context(activation_key)\n context.update({\n 'user': user\n })\n subject = render_to_string(self.email_subject_template,\n context)\n # Force subject to a single line to avoid header-injection\n # issues.\n subject = ''.join(subject.splitlines())\n message = render_to_string(self.email_body_template,\n context)\n user.email_user(subject, message, conf.get('DEFAULT_FROM_EMAIL'))", "response": "Send the activation email to the user."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the inactive user account and wait for validation from superuser", "response": "def create_inactive_user(self, form):\n \"\"\"Create the inactive user account and wait for validation from superuser\"\"\"\n new_user = form.save(commit=False)\n new_user.is_active = False\n new_user.save()\n return new_user"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_key(self, activation_key):\n try:\n username = signing.loads(\n activation_key,\n salt=self.key_salt,\n max_age=conf.get('ACCOUNT_ACTIVATION_DAYS') * 86400\n )\n return username\n # SignatureExpired is a subclass of BadSignature, so this will\n # catch either one.\n except signing.BadSignature:\n return None", "response": "Verify that the activation key is valid and within the activation time window return the username if valid or None if not."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlook up and return the corresponding user account if it exists or None if it doesn t.", "response": "def get_user(self, username):\n \"\"\"\n Given the verified username, look up and return the\n corresponding user account if it exists, or ``None`` if it\n doesn't.\n\n \"\"\"\n User = get_user_model() # noqa\n try:\n user = User.objects.get(**{\n User.USERNAME_FIELD: username,\n 'is_active': False\n })\n return user\n except User.DoesNotExist:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, request, *args, **kwargs):\n activated_user = self.activate(*args, **kwargs)\n if activated_user:\n users_signals.user_activated.send(\n sender=self.__class__,\n user=activated_user,\n request=request\n )\n return redirect(self.success_url)\n return super().get(request, *args, **kwargs)", "response": "The base get method that handles activation logic."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a pod container for the current task.", "response": "def get_pod_container(self,\n volume_mounts,\n persistence_outputs=None,\n persistence_data=None,\n outputs_refs_jobs=None,\n outputs_refs_experiments=None,\n secret_refs=None,\n configmap_refs=None,\n env_vars=None,\n command=None,\n args=None,\n resources=None,\n ports=None,\n ephemeral_token=None):\n \"\"\"Pod job container for task.\"\"\"\n self._pod_container_checks()\n\n # Env vars preparations\n env_vars = to_list(env_vars, check_none=True)\n env_vars += self._get_container_pod_env_vars(\n persistence_outputs=persistence_outputs,\n persistence_data=persistence_data,\n outputs_refs_jobs=outputs_refs_jobs,\n outputs_refs_experiments=outputs_refs_experiments,\n ephemeral_token=ephemeral_token\n )\n env_vars += get_resources_env_vars(resources=resources)\n\n # Env from configmap and secret refs\n env_from = get_pod_env_from(secret_refs=secret_refs, configmap_refs=configmap_refs)\n\n def get_ports():\n _ports = to_list(ports) if ports else []\n return [client.V1ContainerPort(container_port=port) for port in _ports] or None\n\n return client.V1Container(name=self.job_container_name,\n image=self.job_docker_image,\n command=command,\n args=args,\n ports=get_ports(),\n env=env_vars,\n env_from=env_from,\n resources=get_resources(resources),\n volume_mounts=volume_mounts)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the sidecar container for task logs.", "response": "def get_sidecar_container(self, volume_mounts):\n \"\"\"Pod sidecar container for task logs.\"\"\"\n return get_sidecar_container(\n job_container_name=self.job_container_name,\n sidecar_container_name=self.sidecar_container_name,\n sidecar_docker_image=self.sidecar_docker_image,\n sidecar_docker_image_pull_policy=self.sidecar_docker_image_pull_policy,\n namespace=self.namespace,\n sidecar_config=self.sidecar_config,\n sidecar_args=get_sidecar_args(container_id=self.job_container_name,\n app_label=self.app_label),\n internal_health_check_url=self.health_check_url,\n volume_mounts=volume_mounts)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a pod spec for a task.", "response": "def get_task_pod_spec(self,\n volume_mounts,\n volumes,\n resource_name,\n persistence_outputs=None,\n persistence_data=None,\n outputs_refs_jobs=None,\n outputs_refs_experiments=None,\n env_vars=None,\n command=None,\n args=None,\n init_command=None,\n init_args=None,\n init_env_vars=None,\n resources=None,\n ports=None,\n secret_refs=None,\n configmap_refs=None,\n ephemeral_token=None,\n node_selector=None,\n affinity=None,\n tolerations=None,\n sidecar_context_mounts=None,\n init_context_mounts=None,\n restart_policy=None):\n \"\"\"Pod spec to be used to create pods for tasks: master, worker, ps.\"\"\"\n sidecar_context_mounts = to_list(sidecar_context_mounts, check_none=True)\n init_context_mounts = to_list(init_context_mounts, check_none=True)\n volume_mounts = to_list(volume_mounts, check_none=True)\n volumes = to_list(volumes, check_none=True)\n\n gpu_volume_mounts, gpu_volumes = get_gpu_volumes_def(resources)\n volume_mounts += gpu_volume_mounts\n volumes += gpu_volumes\n\n pod_container = self.get_pod_container(volume_mounts=volume_mounts,\n persistence_outputs=persistence_outputs,\n persistence_data=persistence_data,\n outputs_refs_jobs=outputs_refs_jobs,\n outputs_refs_experiments=outputs_refs_experiments,\n secret_refs=secret_refs,\n configmap_refs=configmap_refs,\n resources=resources,\n env_vars=env_vars,\n command=command,\n args=args,\n ports=ports,\n ephemeral_token=ephemeral_token)\n\n containers = [pod_container]\n if self.use_sidecar:\n sidecar_volume_mounts = self.get_sidecar_volume_mounts(\n persistence_outputs=persistence_outputs,\n persistence_data=persistence_data,\n context_mounts=sidecar_context_mounts)\n sidecar_container = self.get_sidecar_container(volume_mounts=sidecar_volume_mounts)\n containers.append(sidecar_container)\n\n init_container = self.get_init_container(init_command=init_command,\n init_args=init_args,\n env_vars=init_env_vars,\n context_mounts=init_context_mounts,\n persistence_outputs=persistence_outputs,\n persistence_data=persistence_data)\n init_containers = to_list(init_container, check_none=True)\n\n node_selector = self._get_node_selector(node_selector=node_selector)\n affinity = self._get_affinity(affinity=affinity)\n tolerations = self._get_tolerations(tolerations=tolerations)\n service_account_name = self._get_service_account_name()\n return client.V1PodSpec(\n security_context=get_security_context() if self.use_security_context else None,\n restart_policy=restart_policy,\n service_account_name=service_account_name,\n init_containers=init_containers,\n containers=containers,\n volumes=volumes,\n node_selector=node_selector,\n tolerations=tolerations,\n affinity=affinity)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _prepare(cls, context: Dict) -> Dict:\n if not context and cls.raise_empty_context:\n raise PolyaxonActionException('{} received invalid payload context.'.format(cls.name))\n\n return context", "response": "This method is used to prepare the payload context."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef copy_experiment(experiment):\n try:\n publisher.publish_experiment_job_log(\n log_lines='Copying outputs from experiment `{}` into experiment `{}`'.format(\n experiment.original_experiment.unique_name, experiment.unique_name\n ),\n experiment_uuid=experiment.uuid.hex,\n experiment_name=experiment.unique_name,\n job_uuid='all',\n )\n stores.copy_experiment_outputs(\n persistence_outputs_from=experiment.original_experiment.persistence_outputs,\n persistence_outputs_to=experiment.persistence_outputs,\n experiment_name_from=experiment.original_experiment.unique_name,\n experiment_name_to=experiment.unique_name)\n\n except OSError:\n publisher.publish_experiment_job_log(\n log_lines='Could not copy the outputs of experiment `{}` into experiment `{}`'.format(\n experiment.original_experiment.unique_name, experiment.unique_name\n ),\n experiment_uuid=experiment.uuid.hex,\n experiment_name=experiment.unique_name,\n job_uuid='all',\n )\n _logger.warning(\n 'Could not copy the outputs of experiment `%s` into experiment `%s`',\n experiment.original_experiment.unique_name, experiment.unique_name)", "response": "Copy the outputs of the given experiment into the given experiment."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef record(self,\n event_type: str,\n event_data: Mapping = None,\n instance: Any = None,\n **kwargs) -> 'Event':\n \"\"\" Validate and record an event.\n\n >>> record('event.action', object_instance)\n \"\"\"\n if not self.is_setup:\n return\n if not self.can_handle(event_type=event_type):\n return\n\n event = self.get_event(event_type=event_type,\n event_data=event_data,\n instance=instance,\n **kwargs)\n self.record_event(event)\n return event", "response": "Validate and record an event."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a new image and log it to the job.", "response": "def generate(job,\n build_path: str,\n from_image: str,\n build_steps: Optional[List[str]] = None,\n env_vars: Optional[List[Tuple[str, str]]] = None,\n nvidia_bin: str = None,\n set_lang_env: bool = True,\n uid: int = None,\n gid: int = None) -> bool:\n \"\"\"Build necessary code for a job to run\"\"\"\n rendered_dockerfile = dockerizer_generate(repo_path=build_path,\n from_image=from_image,\n build_steps=build_steps,\n env_vars=env_vars,\n nvidia_bin=nvidia_bin,\n set_lang_env=set_lang_env,\n uid=uid,\n gid=gid)\n\n if rendered_dockerfile:\n job.log_dockerfile(dockerfile=rendered_dockerfile)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_suggestions(self, iteration_config=None):\n matrix = self.hptuning_config.matrix\n n_suggestions = self.hptuning_config.random_search.n_experiments\n seed = self.hptuning_config.seed\n return get_random_suggestions(matrix=matrix, n_suggestions=n_suggestions, seed=seed)", "response": "Return a list of suggestions based on random search."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting serializer to many if data is passed", "response": "def get_serializer(self, *args, **kwargs):\n \"\"\" if an array is passed, set serializer to many \"\"\"\n if isinstance(kwargs.get('data', {}), list):\n kwargs['many'] = True\n return super().get_serializer(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntasks handling for sidecars logs.", "response": "def logs_handle_experiment_job(experiment_name: str,\n experiment_uuid: str,\n log_lines: Optional[Union[str, Iterable[str]]],\n temp: bool = True) -> None:\n \"\"\"Task handling for sidecars logs.\"\"\"\n handle_experiment_job_log(experiment_name=experiment_name,\n experiment_uuid=experiment_uuid,\n log_lines=log_lines,\n temp=temp)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntask handling for sidecars logs.", "response": "def logs_handle_job(job_uuid: str,\n job_name: str,\n log_lines: Optional[Union[str, Iterable[str]]],\n temp: bool = True) -> None:\n \"\"\"Task handling for sidecars logs.\"\"\"\n handle_job_logs(job_uuid=job_uuid,\n job_name=job_name,\n log_lines=log_lines,\n temp=temp)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef logs_handle_build_job(job_uuid: str,\n job_name: str,\n log_lines: Optional[Union[str, Iterable[str]]],\n temp: bool = True) -> None:\n \"\"\"Task handling for sidecars logs.\"\"\"\n handle_build_job_logs(job_uuid=job_uuid,\n job_name=job_name,\n log_lines=log_lines,\n temp=temp)", "response": "Task handling for sidecars logs."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_iteration(self, num_suggestions=0):\n from db.models.experiment_groups import ExperimentGroupIteration\n\n search_manager = self.experiment_group.search_manager\n iteration_config = self.experiment_group.iteration_config\n\n if iteration_config is None:\n iteration = 0\n bracket_iteration = 0\n else:\n should_reschedule = search_manager.should_reschedule(\n iteration=iteration_config.iteration,\n bracket_iteration=iteration_config.bracket_iteration)\n should_reduce_configs = search_manager.should_reduce_configs(\n iteration=iteration_config.iteration,\n bracket_iteration=iteration_config.bracket_iteration)\n if should_reschedule:\n iteration = iteration_config.iteration + 1\n bracket_iteration = 0\n elif should_reduce_configs:\n iteration = iteration_config.iteration\n bracket_iteration = iteration_config.bracket_iteration + 1\n else:\n raise ValueError(\n 'Hyperband create iteration failed for `{}`, '\n 'could not reschedule ot reduce configs'.format(self.experiment_group.id))\n\n # Create a new iteration config\n iteration_config = HyperbandIterationConfig(\n iteration=iteration,\n num_suggestions=num_suggestions,\n bracket_iteration=bracket_iteration)\n iteration_config.experiment_ids = []\n return ExperimentGroupIteration.objects.create(\n experiment_group=self.experiment_group,\n data=iteration_config.to_dict())", "response": "Create an iteration for the experiment group."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreduce the experiments to restart.", "response": "def get_reduced_configs(self):\n \"\"\"Reduce the experiments to restart.\"\"\"\n iteration_config = self.experiment_group.iteration_config\n if iteration_config is None:\n logger.error(\n 'Experiment group `%s` attempt to update iteration, but has no iteration',\n self.experiment_group.id,\n extra={'stack': True})\n return\n search_manager = self.experiment_group.search_manager\n\n # Get the number of experiments to keep\n n_configs_to_keep = search_manager.get_n_config_to_keep_for_iteration(\n iteration=iteration_config.iteration,\n bracket_iteration=iteration_config.bracket_iteration)\n\n # Get the last group's experiments metrics\n experiments_metrics = self.experiment_group.iteration_config.experiments_metrics\n if experiments_metrics is None:\n raise ExperimentGroupException()\n\n # Order the experiments\n reverse = Optimization.maximize(\n self.experiment_group.hptuning_config.hyperband.metric.optimization)\n experiments_metrics = sorted(experiments_metrics, key=lambda x: x[1], reverse=reverse)\n\n # Keep n experiments\n return [xp[0] for xp in experiments_metrics[:n_configs_to_keep]]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreducing the experiments to restart.", "response": "def reduce_configs(self):\n \"\"\"Reduce the experiments to restart.\"\"\"\n experiment_ids = self.get_reduced_configs()\n experiments = self.experiment_group.experiments.filter(id__in=experiment_ids)\n self.create_iteration()\n iteration_config = self.experiment_group.iteration_config\n hptuning_config = self.experiment_group.hptuning_config\n n_resources = self.experiment_group.search_manager.get_resources_for_iteration(\n iteration=iteration_config.iteration)\n resource_value = self.experiment_group.search_manager.get_n_resources(\n n_resources=n_resources, bracket_iteration=iteration_config.bracket_iteration\n )\n resource_name = hptuning_config.hyperband.resource.name\n resource_value = hptuning_config.hyperband.resource.cast_value(resource_value)\n\n # Check if we need to resume or restart the experiments\n for experiment in experiments:\n declarations = experiment.declarations\n declarations[resource_name] = resource_value\n declarations_spec = {'declarations': declarations}\n specification = experiment.specification.patch(declarations_spec)\n status_message = 'Hyperband iteration: {}, bracket iteration: {}'.format(\n iteration_config.iteration,\n iteration_config.bracket_iteration)\n\n if hptuning_config.hyperband.resume:\n experiment.resume(\n declarations=declarations,\n config=specification.parsed_data,\n message=status_message)\n else:\n experiment.restart(\n experiment_group=self.experiment_group,\n declarations=declarations,\n config=specification.parsed_data)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef all_experiment_groups(self):\n from db.models.experiment_groups import ExperimentGroup\n\n return ExperimentGroup.all.filter(project=self)", "response": "Returns a QuerySet of all experiment groups that are associated with this project."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef all_jobs(self):\n from db.models.jobs import Job\n\n return Job.all.filter(project=self)", "response": "Returns a QuerySet of all jobs in this project."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning all build jobs for this project.", "response": "def all_build_jobs(self):\n \"\"\"\n Similar to build_jobs,\n but uses the default manager to return archived experiments as well.\n \"\"\"\n from db.models.build_jobs import BuildJob\n\n return BuildJob.all.filter(project=self)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef all_notebook_jobs(self):\n from db.models.notebooks import NotebookJob\n\n return NotebookJob.all.filter(project=self)", "response": "Returns a QuerySet of all NotebookJob objects for this project."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef all_tensorboard_jobs(self):\n from db.models.tensorboards import TensorboardJob\n\n return TensorboardJob.all.filter(project=self)", "response": "Returns a QuerySet of all TensorboardJob objects for this project."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef all_experiments(self):\n from db.models.experiments import Experiment\n\n return Experiment.all.filter(experiment_group=self)", "response": "Returns a QuerySet of all experiments in this group."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn all the group experiments.", "response": "def all_group_experiments(self):\n \"\"\"\n Similar to group_experiments,\n but uses the default manager to return archived experiments as well.\n \"\"\"\n from db.models.experiments import Experiment\n\n if self.is_selection:\n return Experiment.all.filter(selections=self)\n return Experiment.all.filter(experiment_group=self)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_dag(nodes, downstream_fn) -> Tuple[Dict, Dict]:\n dag = {}\n node_by_ids = {}\n for node in nodes:\n downstream_ops = downstream_fn(node)\n dag[node.id] = set(downstream_ops)\n node_by_ids[node.id] = node\n\n return dag, node_by_ids", "response": "Return a dag representation of the nodes passed."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a list of all node in the graph with no dependencies.", "response": "def get_independent_nodes(dag):\n \"\"\"Get a list of all node in the graph with no dependencies.\"\"\"\n nodes = set(dag.keys())\n dependent_nodes = set([node for downstream_nodes in dag.values() for node in downstream_nodes])\n return set(nodes - dependent_nodes)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting orphan nodes for given dag.", "response": "def get_orphan_nodes(dag):\n \"\"\"Get orphan nodes for given dag.\"\"\"\n independent_nodes = get_independent_nodes(dag)\n return set([node for node in independent_nodes if not dag[node]])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if the node has any dependencies.", "response": "def has_dependencies(node, dag):\n \"\"\"Checks if the node has dependencies.\"\"\"\n for downstream_nodes in dag.values():\n if node in downstream_nodes:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsort the dag first topologically.", "response": "def sort_topologically(dag):\n \"\"\"Sort the dag breath first topologically.\n\n Only the nodes inside the dag are returned, i.e. the nodes that are also keys.\n\n Returns:\n a topological ordering of the DAG.\n Raises:\n an error if this is not possible (graph is not valid).\n \"\"\"\n dag = copy.deepcopy(dag)\n sorted_nodes = []\n independent_nodes = deque(get_independent_nodes(dag))\n while independent_nodes:\n node = independent_nodes.popleft()\n sorted_nodes.append(node)\n # this alters the dag so that we are sure we are visiting the nodes only once\n downstream_nodes = dag[node]\n while downstream_nodes:\n downstream_node = downstream_nodes.pop(0)\n if downstream_node not in dag:\n continue\n if not has_dependencies(downstream_node, dag):\n independent_nodes.append(downstream_node)\n\n if len(sorted_nodes) != len(dag.keys()):\n raise ValueError('graph is not acyclic')\n return sorted_nodes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse the negation modifier in an operation.", "response": "def parse_negation_operation(operation: str) -> Tuple[bool, str]:\n \"\"\"Parse the negation modifier in an operation.\"\"\"\n _operation = operation.strip()\n if not _operation:\n raise QueryParserException('Operation is not valid: {}'.format(operation))\n negation = False\n if _operation[0] == '~':\n negation = True\n _operation = _operation[1:]\n\n return negation, _operation.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_comparison_operation(operation: str) -> Tuple[Optional[str], str]:\n _operation = operation.strip()\n if not _operation:\n raise QueryParserException('Operation is not valid: {}'.format(operation))\n # Check inclusion comparison\n if _operation[:2] in ('<=', '=<'):\n return '<=', _operation[2:].strip()\n\n if _operation[:2] in ('>=', '=>'):\n return '>=', _operation[2:].strip()\n\n # Non inclusive\n if _operation[:1] in ('>', '<'):\n return _operation[:1], _operation[1:].strip()\n\n return None, _operation", "response": "Parse the comparision operator in an operation."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing a datetime operation.", "response": "def parse_datetime_operation(operation: str) -> 'QueryOpSpec':\n \"\"\"Parse datetime operations.\n\n A datetime operation can one of the following:\n\n * single value: start_date:2014-10-10, start_date:>2014-10-10, start_date:>=2014-10-10\n * negation single value: start_date:~2014-10-10\n * interval: start_date:2010-10-10 10:10 .. 2012-10-10\n * negation interval: start_date:~2010-10-10 10:10 .. 2012-10-10\n\n This parser does not allow `|`\n \"\"\"\n _operation = operation.strip()\n if not _operation:\n raise QueryParserException('Operation is not valid: {}'.format(operation))\n # Check not allowed ops\n if '|' in _operation:\n raise QueryParserException('`|` is not allowed for datetime operations. '\n 'Operation: {}'.format(operation))\n\n # Check negation\n negation, _operation = parse_negation_operation(_operation)\n\n # Check range operator\n if '..' in _operation:\n op = '..'\n params = _operation.split('..')\n params = [param.strip() for param in params if param]\n if len(params) != 2:\n raise QueryParserException('Expression is not valid, ranges requires only 2 params, '\n 'Operation: {}'.format(operation))\n return QueryOpSpec(op, negation, params)\n\n # Check comparison operators\n op, _operation = parse_comparison_operation(_operation)\n if not op:\n # Now the operation must be an equality param param\n op = '='\n\n if not _operation:\n raise QueryParserException('Expression is not valid, it must be formatted as '\n 'name:operation, '\n 'Operation: {}'.format(operation))\n return QueryOpSpec(op, negation, _operation)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_scalar_operation(operation: str) -> 'QueryOpSpec':\n _operation = operation.strip()\n if not _operation:\n raise QueryParserException('Operation is not valid: {}'.format(operation))\n # Check not allowed ops\n if '|' in _operation:\n raise QueryParserException('`|` is not allowed for scalar operations. '\n 'Operation: {}'.format(operation))\n if '..' in _operation:\n raise QueryParserException('`..` is not allowed for scalar operations. '\n 'Operation: {}'.format(operation))\n\n # Check negation\n negation, _operation = parse_negation_operation(_operation)\n\n # Check comparison operators\n op, _operation = parse_comparison_operation(_operation)\n if not op:\n # Now the operation must be an equality param param\n op = '='\n\n # Check that params are scalar (int, float)\n try:\n _operation = int(_operation)\n except (ValueError, TypeError):\n try:\n _operation = float(_operation)\n except (ValueError, TypeError):\n raise QueryParserException('Scalar operation requires int or float params, '\n 'receive {}.'.format(operation))\n return QueryOpSpec(op, negation, _operation)", "response": "Parse a scalar operation."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbases parsing for expressions.", "response": "def parse_expression(expression: str) -> Tuple[str, str]:\n \"\"\"Base parsing for expressions.\n\n Every expression must follow a basic format:\n `name:[modifier|operator]operation[*[operator]operation]`\n\n So this parser just split the expression into: field name, operation.\n \"\"\"\n try:\n _expression = expression.strip()\n name, operation = _expression.split(':')\n name = name.strip()\n operation = operation.strip()\n if not name or not operation:\n raise ValueError\n except (ValueError, AttributeError):\n raise QueryParserException('Expression is not valid, it must be formatted as '\n 'name:operation, '\n 'Expression: {}'.format(expression))\n return name, operation"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef split_query(query: str) -> List[str]:\n try:\n _query = query.strip()\n except (ValueError, AttributeError):\n raise QueryParserException('query is not valid, received instead {}'.format(query))\n\n expressions = _query.split(',')\n expressions = [exp.strip() for exp in expressions if exp.strip()]\n if not expressions:\n raise QueryParserException('Query is not valid: {}'.format(query))\n\n return expressions", "response": "Split a query into different expressions."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tokenize_query(query: str) -> Dict[str, Iterable]:\n expressions = split_query(query)\n name_operation_tuples = [parse_expression(expression) for expression in expressions]\n operation_by_name = defaultdict(list)\n for name, operation in name_operation_tuples:\n operation_by_name[name].append(operation)\n return operation_by_name", "response": "Tokenizes a standard search query in name and operations mapping."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_field(field: str) -> Tuple[str, Optional[str]]:\n _field = field.split('.')\n _field = [f.strip() for f in _field]\n if len(_field) == 1 and _field[0]:\n return _field[0], None\n elif len(_field) == 2 and _field[0] and _field[1]:\n return _field[0], _field[1]\n raise QueryParserException('Query field must be either a single value,'\n 'possibly with single underscores, '\n 'or a prefix double underscore field. '\n 'Received `{}`'.format(field))", "response": "Parses a single field with underscores and return field and suffix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the upstream operations for an operation run.", "response": "def set_op_upstreams(op_run, op):\n \"\"\"Set the upstream operations for operation run.\"\"\"\n # We get a list of all upstream ops or the current op\n upstream_ops = op.upstream_operations.values_list('id', flat=True)\n # We get latest op runs for the upstream_ops\n latest_op_runs = OperationRun.objects.filter(id__in=upstream_ops)\n latest_op_runs = latest_op_runs.annotate(max_date=Max('created_at'))\n latest_op_runs = latest_op_runs.filter(date=F('max_date'))\n # Set the upstream ops\n op_run.set(latest_op_runs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_topological_dag_upstreams(dag, ops, op_runs, runs_by_ops):\n sorted_ops = dags.sort_topologically(dag=dag)\n for op_id in sorted_ops:\n op_run_id = runs_by_ops[op_id]\n op_run = op_runs[op_run_id]\n set_op_upstreams(op_run=op_run, op=ops[op_id])", "response": "Set the upstream runs for the operations in the dag following the topological sort."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_pipeline_run(pipeline, context_by_op):\n pipeline_run = PipelineRun.objects.create(pipeline=pipeline)\n dag, ops = pipeline.dag\n # Go trough the operation and create operation runs and the upstreams\n op_runs = {}\n runs_by_ops = {}\n for op_id in dag.keys():\n op_run = OperationRun.objects.create(\n pipeline_run=pipeline_run,\n operation_id=op_id,\n celery_task_context=context_by_op.get(op_id))\n op_run_id = op_run.id\n op_runs[op_run_id] = op_run\n runs_by_ops[op_id] = op_run_id\n\n # Create operations upstreams\n # We set the upstream for the topologically sorted dag\n set_topological_dag_upstreams(dag=dag, ops=ops, op_runs=op_runs, runs_by_ops=runs_by_ops)", "response": "Create a pipeline run and instance."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef k8s_events_handle_experiment_job_statuses(self: 'celery_app.task', payload: Dict) -> None:\n details = payload['details']\n job_uuid = details['labels']['job_uuid']\n logger.debug('handling events status for job_uuid: %s, status: %s',\n job_uuid, payload['status'])\n\n try:\n job = ExperimentJob.objects.get(uuid=job_uuid)\n except ExperimentJob.DoesNotExist:\n logger.debug('Job uuid`%s` does not exist', job_uuid)\n return\n\n try:\n job.experiment\n except Experiment.DoesNotExist:\n logger.debug('Experiment for job `%s` does not exist anymore', job_uuid)\n return\n\n if job.last_status is None and self.request.retries < 2:\n self.retry(countdown=1)\n\n # Set the new status\n try:\n set_node_scheduling(job, details['node_name'])\n job.set_status(status=payload['status'],\n message=payload['message'],\n created_at=payload.get('created_at'),\n traceback=payload.get('traceback'),\n details=details)\n logger.debug('status %s is set for job %s %s', payload['status'], job_uuid, job.id)\n except IntegrityError:\n # Due to concurrency this could happen, we just retry it\n logger.info('Retry job status %s handling %s', payload['status'], job_uuid)\n self.retry(countdown=Intervals.EXPERIMENTS_SCHEDULER)", "response": "Handle the status of an experiment job."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef k8s_events_handle_job_statuses(self: 'celery_app.task', payload: Dict) -> None:\n details = payload['details']\n job_uuid = details['labels']['job_uuid']\n job_name = details['labels']['job_name']\n project_name = details['labels'].get('project_name')\n logger.debug('handling events status for job %s', job_name)\n\n try:\n job = Job.objects.get(uuid=job_uuid)\n except Job.DoesNotExist:\n logger.debug('Job `%s` does not exist', job_name)\n return\n\n try:\n job.project\n except Project.DoesNotExist:\n logger.debug('Project for job `%s` does not exist', project_name)\n return\n\n # Set the new status\n try:\n set_node_scheduling(job, details['node_name'])\n job.set_status(status=payload['status'],\n message=payload['message'],\n traceback=payload.get('traceback'),\n details=details)\n except IntegrityError:\n # Due to concurrency this could happen, we just retry it\n self.retry(countdown=Intervals.EXPERIMENTS_SCHEDULER)", "response": "Handle the status of a job."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nhandling the status of a plugin job.", "response": "def k8s_events_handle_plugin_job_statuses(self: 'celery_app.task', payload: Dict) -> None:\n \"\"\"Project Plugin jobs statuses\"\"\"\n details = payload['details']\n app = details['labels']['app']\n job_uuid = details['labels']['job_uuid']\n job_name = details['labels']['job_name']\n project_name = details['labels'].get('project_name')\n logger.debug('handling events status for job %s %s', job_name, app)\n\n try:\n if app == conf.get('APP_LABELS_TENSORBOARD'):\n job = TensorboardJob.objects.get(uuid=job_uuid)\n elif app == conf.get('APP_LABELS_NOTEBOOK'):\n job = NotebookJob.objects.get(uuid=job_uuid)\n else:\n logger.info('Plugin job `%s` does not exist', app)\n return\n except (NotebookJob.DoesNotExist, TensorboardJob.DoesNotExist):\n logger.debug('`%s - %s` does not exist', app, job_name)\n return\n\n try:\n job.project\n except Project.DoesNotExist:\n logger.debug('`%s` does not exist anymore', project_name)\n\n # Set the new status\n try:\n set_node_scheduling(job, details['node_name'])\n job.set_status(status=payload['status'],\n message=payload['message'],\n traceback=payload.get('traceback'),\n details=details)\n except IntegrityError:\n # Due to concurrency this could happen, we just retry it\n self.retry(countdown=Intervals.EXPERIMENTS_SCHEDULER)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling the build job statuses", "response": "def k8s_events_handle_build_job_statuses(self: 'celery_app.task', payload: Dict) -> None:\n \"\"\"Project Plugin jobs statuses\"\"\"\n details = payload['details']\n app = details['labels']['app']\n job_uuid = details['labels']['job_uuid']\n job_name = details['labels']['job_name']\n project_name = details['labels'].get('project_name')\n logger.debug('handling events status for build jon %s %s', job_name, app)\n\n try:\n build_job = BuildJob.objects.get(uuid=job_uuid)\n except BuildJob.DoesNotExist:\n logger.info('Build job `%s` does not exist', job_name)\n return\n\n try:\n build_job.project\n except Project.DoesNotExist:\n logger.debug('`%s` does not exist anymore', project_name)\n\n # Set the new status\n try:\n set_node_scheduling(build_job, details['node_name'])\n build_job.set_status(status=payload['status'],\n message=payload['message'],\n traceback=payload.get('traceback'),\n details=details)\n except IntegrityError:\n # Due to concurrency this could happen, we just retry it\n self.retry(countdown=Intervals.EXPERIMENTS_SCHEDULER)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a pod sidecar container.", "response": "def get_sidecar_container(job_container_name,\n sidecar_container_name,\n sidecar_docker_image,\n sidecar_docker_image_pull_policy,\n namespace,\n sidecar_config,\n sidecar_args,\n internal_health_check_url,\n volume_mounts,\n env_vars=None):\n \"\"\"Return a pod sidecar container.\"\"\"\n env_vars = to_list(env_vars) if env_vars else []\n env_vars += get_sidecar_env_vars(namespace=namespace,\n job_container_name=job_container_name,\n internal_health_check_url=internal_health_check_url)\n for k, v in sidecar_config.items():\n env_vars.append(get_env_var(name=k, value=v))\n return client.V1Container(name=sidecar_container_name,\n image=sidecar_docker_image,\n image_pull_policy=sidecar_docker_image_pull_policy,\n command=get_sidecar_command(),\n env=env_vars,\n volume_mounts=volume_mounts,\n args=sidecar_args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nuse custom exception handler for errors.", "response": "def handle_exception(self, exc):\n \"\"\"Use custom exception handler for errors.\"\"\"\n if isinstance(\n exc, (rest_exceptions.NotAuthenticated,\n rest_exceptions.AuthenticationFailed)) and self.HANDLE_UNAUTHENTICATED:\n return HttpResponseRedirect('{}?next={}'.format(\n reverse('users:login'),\n self.request.get_full_path()))\n\n if isinstance(exc, Http404):\n raise Http404()\n\n if isinstance(exc, rest_exceptions.PermissionDenied):\n raise django_exceptions.PermissionDenied()\n\n return super().handle_exception(exc)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_resources(resources): # pylint:disable=too-many-branches\n limits = {}\n requests = {}\n if resources is None:\n return None\n if resources.cpu:\n if resources.cpu.limits:\n limits['cpu'] = resources.cpu.limits\n if resources.cpu.requests:\n requests['cpu'] = resources.cpu.requests\n\n if resources.memory:\n if resources.memory.limits:\n limits['memory'] = '{}Mi'.format(resources.memory.limits)\n if resources.memory.requests:\n requests['memory'] = '{}Mi'.format(resources.memory.requests)\n\n if resources.gpu:\n if resources.gpu.limits:\n limits[conf.get('K8S_GPU_RESOURCE_KEY')] = resources.gpu.limits\n if resources.gpu.requests:\n requests[conf.get('K8S_GPU_RESOURCE_KEY')] = resources.gpu.requests\n\n if resources.tpu:\n if resources.tpu.limits:\n limits[conf.get('K8S_TPU_RESOURCE_KEY')] = resources.tpu.limits\n if resources.tpu.requests:\n requests[conf.get('K8S_TPU_RESOURCE_KEY')] = resources.tpu.requests\n return client.V1ResourceRequirements(limits=limits or None, requests=requests or None)", "response": "Create resources requirements.\n\n Args:\n resources: `PodResourcesConfig`\n\n Return:\n `V1ResourceRequirements`"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef has_project_permissions(user: 'User', project: 'Project', request_method: str) -> bool:\n # Superusers and the creator is allowed to do everything\n if user.is_staff or user.is_superuser or project.user == user:\n return True\n\n # Other user\n return request_method in permissions.SAFE_METHODS and project.is_public", "response": "Returns True if the user has permission to do project."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes all group outputs and logs.", "response": "def experiment_group_pre_delete(sender, **kwargs):\n \"\"\"Delete all group outputs.\"\"\"\n instance = kwargs['instance']\n\n if instance.is_selection:\n return\n\n # Delete outputs and logs\n celery_app.send_task(\n SchedulerCeleryTasks.STORES_SCHEDULE_OUTPUTS_DELETION,\n kwargs={\n 'persistence': instance.persistence_outputs,\n 'subpath': instance.subpath,\n },\n countdown=conf.get('GLOBAL_COUNTDOWN'))\n celery_app.send_task(\n SchedulerCeleryTasks.STORES_SCHEDULE_LOGS_DELETION,\n kwargs={\n 'persistence': instance.persistence_logs,\n 'subpath': instance.subpath,\n },\n countdown=conf.get('GLOBAL_COUNTDOWN'))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndeleting all group outputs.", "response": "def experiment_group_post_delete(sender, **kwargs):\n \"\"\"Delete all group outputs.\"\"\"\n instance = kwargs['instance']\n auditor.record(event_type=EXPERIMENT_GROUP_DELETED, instance=instance)\n remove_bookmarks(object_id=instance.id, content_type='experimentgroup')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_asset_url(module: str, path: str) -> str:\n return '{}/{}/{}'.format(\n settings.STATIC_URL.rstrip('/'),\n module,\n path.lstrip('/'),\n )", "response": "Return a static asset URL located within Polyaxon s static files."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the init container for setting outputs path.", "response": "def get_init_container(self,\n init_command,\n init_args,\n env_vars,\n context_mounts,\n persistence_outputs,\n persistence_data):\n \"\"\"Pod init container for setting outputs path.\"\"\"\n env_vars = to_list(env_vars, check_none=True)\n outputs_path = stores.get_job_outputs_path(\n persistence=persistence_outputs,\n job_name=self.job_name)\n _, outputs_volume_mount = get_pod_outputs_volume(persistence_outputs=persistence_outputs)\n volume_mounts = outputs_volume_mount + to_list(context_mounts, check_none=True)\n init_command = init_command or [\"/bin/sh\", \"-c\"]\n init_args = init_args or to_list(\n get_output_args(command=InitCommands.CREATE,\n outputs_path=outputs_path))\n init_args += to_list(get_auth_context_args(entity='job',\n entity_name=self.job_name))\n return client.V1Container(\n name=self.init_container_name,\n image=self.init_docker_image,\n image_pull_policy=self.init_docker_image_pull_policy,\n command=init_command,\n args=[''.join(init_args)],\n env=env_vars,\n volume_mounts=volume_mounts)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_init_container(self,\n init_command,\n init_args,\n env_vars,\n context_mounts,\n persistence_outputs,\n persistence_data):\n \"\"\"Pod init container for setting outputs path.\"\"\"\n env_vars = to_list(env_vars, check_none=True)\n env_vars += [\n get_env_var(name=constants.CONFIG_MAP_JOB_INFO_KEY_NAME,\n value=json.dumps(self.labels)),\n ]\n return client.V1Container(\n name=self.init_container_name,\n image=self.init_docker_image,\n image_pull_policy=self.init_docker_image_pull_policy,\n command=init_command,\n env=env_vars,\n args=init_args,\n volume_mounts=context_mounts)", "response": "Get the init container for setting outputs path."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates an iteration for the experiment group.", "response": "def create_iteration(self, num_suggestions):\n \"\"\"Create an iteration for the experiment group (works for grid and random).\"\"\"\n from db.models.experiment_groups import ExperimentGroupIteration\n\n iteration_config = BaseIterationConfig(iteration=0,\n num_suggestions=num_suggestions,\n experiment_ids=[])\n\n return ExperimentGroupIteration.objects.create(\n experiment_group=self.experiment_group,\n data=iteration_config.to_dict())"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_iteration(self):\n iteration_config = self.get_iteration_config()\n if not iteration_config:\n return\n experiments_metrics = self.experiment_group.get_experiments_metrics(\n experiment_ids=iteration_config.experiment_ids,\n metric=self.get_metric_name()\n )\n iteration_config.experiments_metrics = [m for m in experiments_metrics if m[1] is not None]\n self._update_config(iteration_config)", "response": "Update the last experiment group s iteration with experiment performance."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates iteration s num_suggestions.", "response": "def update_iteration_num_suggestions(self, num_suggestions):\n \"\"\"Update iteration's num_suggestions.\"\"\"\n iteration_config = self.experiment_group.iteration_config\n\n iteration_config.num_suggestions = num_suggestions\n self._update_config(iteration_config)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef validate_config(self, config):\n # config is optional\n if not config:\n return config\n\n spec = validate_experiment_spec_config(config)\n\n if spec.is_experiment:\n # Resume normal creation\n return config\n\n # Raise an error to tell the user to use experiment creation instead\n raise ValidationError('Current experiment creation could not be performed.\\n'\n 'The reason is that the specification sent correspond '\n 'to a `{}`.\\n'.format(spec.kind))", "response": "Validate the config and return the config if it is valid."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_event_action(cls) -> Optional[str]:\n if not cls.actor:\n return None\n return event_context.get_event_action(cls.event_type)", "response": "Return the second part of the event_type\n e. g. event. deleted"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_build_job(user, project, config, code_reference, configmap_refs=None, secret_refs=None):\n build_job, rebuild = BuildJob.create(\n user=user,\n project=project,\n config=config,\n code_reference=code_reference,\n configmap_refs=configmap_refs,\n secret_refs=secret_refs)\n\n if build_job.succeeded and not rebuild:\n return build_job, True, False\n\n if build_job.is_done:\n build_job, _ = BuildJob.create(\n user=user,\n project=project,\n config=config,\n code_reference=code_reference,\n configmap_refs=configmap_refs,\n secret_refs=secret_refs,\n nocache=True)\n\n if not build_job.is_running:\n # We need to build the image first\n auditor.record(event_type=BUILD_JOB_STARTED_TRIGGERED,\n instance=build_job,\n actor_id=user.id,\n actor_name=user.username)\n build_status = start_dockerizer(build_job=build_job)\n else:\n build_status = True\n\n return build_job, False, build_status", "response": "Create a build job based on the params."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef scan(stream, Loader=Loader):\n loader = Loader(stream)\n try:\n while loader.check_token():\n yield loader.get_token()\n finally:\n loader.dispose()", "response": "Scan a YAML stream and produce scanning tokens."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse(stream, Loader=Loader):\n loader = Loader(stream)\n try:\n while loader.check_event():\n yield loader.get_event()\n finally:\n loader.dispose()", "response": "Parse a YAML stream and produce parsing events."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the first YAML document in a stream and produce a representation tree.", "response": "def compose(stream, Loader=Loader):\n \"\"\"\n Parse the first YAML document in a stream\n and produce the corresponding representation tree.\n \"\"\"\n loader = Loader(stream)\n try:\n return loader.get_single_node()\n finally:\n loader.dispose()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compose_all(stream, Loader=Loader):\n loader = Loader(stream)\n try:\n while loader.check_node():\n yield loader.get_node()\n finally:\n loader.dispose()", "response": "Parse all YAML documents in a stream and produce corresponding representation trees."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load(stream, Loader=None):\n if Loader is None:\n load_warning('load')\n Loader = FullLoader\n\n loader = Loader(stream)\n try:\n return loader.get_single_data()\n finally:\n loader.dispose()", "response": "Parse the first YAML document in a stream and produce a Python object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_all(stream, Loader=None):\n if Loader is None:\n load_warning('load_all')\n Loader = FullLoader\n\n loader = Loader(stream)\n try:\n while loader.check_data():\n yield loader.get_data()\n finally:\n loader.dispose()", "response": "Parse all YAML documents in a stream and produce corresponding Python objects."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef emit(events, stream=None, Dumper=Dumper,\n canonical=None, indent=None, width=None,\n allow_unicode=None, line_break=None):\n \"\"\"\n Emit YAML parsing events into a stream.\n If stream is None, return the produced string instead.\n \"\"\"\n getvalue = None\n if stream is None:\n from StringIO import StringIO\n stream = StringIO()\n getvalue = stream.getvalue\n dumper = Dumper(stream, canonical=canonical, indent=indent, width=width,\n allow_unicode=allow_unicode, line_break=line_break)\n try:\n for event in events:\n dumper.emit(event)\n finally:\n dumper.dispose()\n if getvalue:\n return getvalue()", "response": "Emits YAML parsing events into a stream."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nserializing a representation tree into a YAML stream.", "response": "def serialize(node, stream=None, Dumper=Dumper, **kwds):\n \"\"\"\n Serialize a representation tree into a YAML stream.\n If stream is None, return the produced string instead.\n \"\"\"\n return serialize_all([node], stream, Dumper=Dumper, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nserialize a Python object into a YAML stream.", "response": "def dump(data, stream=None, Dumper=Dumper, **kwds):\n \"\"\"\n Serialize a Python object into a YAML stream.\n If stream is None, return the produced string instead.\n \"\"\"\n return dump_all([data], stream, Dumper=Dumper, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nserialize a sequence of Python objects into a YAML stream.", "response": "def safe_dump_all(documents, stream=None, **kwds):\n \"\"\"\n Serialize a sequence of Python objects into a YAML stream.\n Produce only basic YAML tags.\n If stream is None, return the produced string instead.\n \"\"\"\n return dump_all(documents, stream, Dumper=SafeDumper, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nserializing a Python object into a YAML stream.", "response": "def safe_dump(data, stream=None, **kwds):\n \"\"\"\n Serialize a Python object into a YAML stream.\n Produce only basic YAML tags.\n If stream is None, return the produced string instead.\n \"\"\"\n return dump_all([data], stream, Dumper=SafeDumper, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds an implicit scalar detector.", "response": "def add_implicit_resolver(tag, regexp, first=None,\n Loader=Loader, Dumper=Dumper):\n \"\"\"\n Add an implicit scalar detector.\n If an implicit scalar value matches the given regexp,\n the corresponding tag is assigned to the scalar.\n first is a sequence of possible initial characters or None.\n \"\"\"\n Loader.add_implicit_resolver(tag, regexp, first)\n Dumper.add_implicit_resolver(tag, regexp, first)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd a path based resolver for the given tag.", "response": "def add_path_resolver(tag, path, kind=None, Loader=Loader, Dumper=Dumper):\n \"\"\"\n Add a path based resolver for the given tag.\n A path is a list of keys that forms a path\n to a node in the representation tree.\n Keys can be string values, integers, or None.\n \"\"\"\n Loader.add_path_resolver(tag, path, kind)\n Dumper.add_path_resolver(tag, path, kind)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a Python object to a representation node.", "response": "def to_yaml(cls, dumper, data):\n \"\"\"\n Convert a Python object to a representation node.\n \"\"\"\n return dumper.represent_yaml_object(cls.yaml_tag, data, cls,\n flow_style=cls.yaml_flow_style)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef serialize_all(nodes, stream=None, Dumper=Dumper,\n canonical=None, indent=None, width=None,\n allow_unicode=None, line_break=None,\n encoding=None, explicit_start=None, explicit_end=None,\n version=None, tags=None):\n \"\"\"\n Serialize a sequence of representation trees into a YAML stream.\n If stream is None, return the produced string instead.\n \"\"\"\n getvalue = None\n if stream is None:\n if encoding is None:\n stream = io.StringIO()\n else:\n stream = io.BytesIO()\n getvalue = stream.getvalue\n dumper = Dumper(stream, canonical=canonical, indent=indent, width=width,\n allow_unicode=allow_unicode, line_break=line_break,\n encoding=encoding, version=version, tags=tags,\n explicit_start=explicit_start, explicit_end=explicit_end)\n try:\n dumper.open()\n for node in nodes:\n dumper.serialize(node)\n dumper.close()\n finally:\n dumper.dispose()\n if getvalue:\n return getvalue()", "response": "Serialize a sequence of representation trees into a YAML stream."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef something(TokenClass):\n def callback(lexer, match, context):\n text = match.group()\n if not text:\n return\n yield match.start(), TokenClass, text\n context.pos = match.end()\n return callback", "response": "Return a callback that returns a list of tokens."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_indent(TokenClass, implicit=False):\n def callback(lexer, match, context):\n text = match.group()\n if context.indent < context.next_indent:\n context.indent_stack.append(context.indent)\n context.indent = context.next_indent\n if not implicit:\n context.next_indent += len(text)\n yield match.start(), TokenClass, text\n context.pos = match.end()\n return callback", "response": "Set the previously saved indentation level."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_block_scalar_empty_line(IndentTokenClass, ContentTokenClass):\n def callback(lexer, match, context):\n text = match.group()\n if (context.block_scalar_indent is None or\n len(text) <= context.block_scalar_indent):\n if text:\n yield match.start(), IndentTokenClass, text\n else:\n indentation = text[:context.block_scalar_indent]\n content = text[context.block_scalar_indent:]\n yield match.start(), IndentTokenClass, indentation\n yield (match.start()+context.block_scalar_indent,\n ContentTokenClass, content)\n context.pos = match.end()\n return callback", "response": "Return a callback that yields a match for an empty line in a block scalar."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess indentation spaces in a plain scalar.", "response": "def parse_plain_scalar_indent(TokenClass):\n \"\"\"Process indentation spaces in a plain scalar.\"\"\"\n def callback(lexer, match, context):\n text = match.group()\n if len(text) <= context.indent:\n context.stack.pop()\n context.stack.pop()\n return\n if text:\n yield match.start(), TokenClass, text\n context.pos = match.end()\n return callback"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process_tokens(words, normalize_plurals=True):\n # words can be either a list of unigrams or bigrams\n # d is a dict of dicts.\n # Keys of d are word.lower(). Values are dicts\n # counting frequency of each capitalization\n d = defaultdict(dict)\n for word in words:\n word_lower = word.lower()\n # get dict of cases for word_lower\n case_dict = d[word_lower]\n # increase this case\n case_dict[word] = case_dict.get(word, 0) + 1\n if normalize_plurals:\n # merge plurals into the singular count (simple cases only)\n merged_plurals = {}\n for key in list(d.keys()):\n if key.endswith('s') and not key.endswith(\"ss\"):\n key_singular = key[:-1]\n if key_singular in d:\n dict_plural = d[key]\n dict_singular = d[key_singular]\n for word, count in dict_plural.items():\n singular = word[:-1]\n dict_singular[singular] = (\n dict_singular.get(singular, 0) + count)\n merged_plurals[key] = key_singular\n del d[key]\n fused_cases = {}\n standard_cases = {}\n item1 = itemgetter(1)\n for word_lower, case_dict in d.items():\n # Get the most popular case.\n first = max(case_dict.items(), key=item1)[0]\n fused_cases[first] = sum(case_dict.values())\n standard_cases[word_lower] = first\n if normalize_plurals:\n # add plurals to fused cases:\n for plural, singular in merged_plurals.items():\n standard_cases[plural] = standard_cases[singular.lower()]\n return fused_cases, standard_cases", "response": "Given a list of words and a list of words return a new version of the most common case."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a single_color_func associated with the word", "response": "def get_color_func(self, word):\n \"\"\"Returns a single_color_func associated with the word\"\"\"\n try:\n color_func = next(\n color_func for (color_func, words) in self.color_func_to_words\n if word in words)\n except StopIteration:\n color_func = self.default_color_func\n\n return color_func"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef random_color_func(word=None, font_size=None, position=None,\n orientation=None, font_path=None, random_state=None):\n \"\"\"Random hue color generation.\n\n Default coloring method. This just picks a random hue with value 80% and\n lumination 50%.\n\n Parameters\n ----------\n word, font_size, position, orientation : ignored.\n\n random_state : random.Random object or None, (default=None)\n If a random object is given, this is used for generating random\n numbers.\n\n \"\"\"\n if random_state is None:\n random_state = Random()\n return \"hsl(%d, 80%%, 50%%)\" % random_state.randint(0, 255)", "response": "Random hue color generation.\n\n Default coloring method. This just picks a random hue with value 80% and\n lumination 50%.\n\n Parameters\n ----------\n word, font_size, position, orientation : ignored.\n\n random_state : random.Random object or None, (default=None)\n If a random object is given, this is used for generating random\n numbers."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_single_color_func(color):\n old_r, old_g, old_b = ImageColor.getrgb(color)\n rgb_max = 255.\n h, s, v = colorsys.rgb_to_hsv(old_r / rgb_max, old_g / rgb_max,\n old_b / rgb_max)\n\n def single_color_func(word=None, font_size=None, position=None,\n orientation=None, font_path=None, random_state=None):\n \"\"\"Random color generation.\n\n Additional coloring method. It picks a random value with hue and\n saturation based on the color given to the generating function.\n\n Parameters\n ----------\n word, font_size, position, orientation : ignored.\n\n random_state : random.Random object or None, (default=None)\n If a random object is given, this is used for generating random\n numbers.\n\n \"\"\"\n if random_state is None:\n random_state = Random()\n r, g, b = colorsys.hsv_to_rgb(h, s, random_state.uniform(0.2, 1))\n return 'rgb({:.0f}, {:.0f}, {:.0f})'.format(r * rgb_max, g * rgb_max,\n b * rgb_max)\n return single_color_func", "response": "Create a color function which returns a single hue and saturation with."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_from_frequencies(self, frequencies, max_font_size=None): # noqa: C901\n # make sure frequencies are sorted and normalized\n frequencies = sorted(frequencies.items(), key=itemgetter(1), reverse=True)\n if len(frequencies) <= 0:\n raise ValueError(\"We need at least 1 word to plot a word cloud, \"\n \"got %d.\" % len(frequencies))\n frequencies = frequencies[:self.max_words]\n\n # largest entry will be 1\n max_frequency = float(frequencies[0][1])\n\n frequencies = [(word, freq / max_frequency)\n for word, freq in frequencies]\n\n if self.random_state is not None:\n random_state = self.random_state\n else:\n random_state = Random()\n\n if self.mask is not None:\n boolean_mask = self._get_bolean_mask(self.mask)\n width = self.mask.shape[1]\n height = self.mask.shape[0]\n else:\n boolean_mask = None\n height, width = self.height, self.width\n occupancy = IntegralOccupancyMap(height, width, boolean_mask)\n\n # create image\n img_grey = Image.new(\"L\", (width, height))\n draw = ImageDraw.Draw(img_grey)\n img_array = np.asarray(img_grey)\n font_sizes, positions, orientations, colors = [], [], [], []\n\n last_freq = 1.\n\n if max_font_size is None:\n # if not provided use default font_size\n max_font_size = self.max_font_size\n\n if max_font_size is None:\n # figure out a good font size by trying to draw with\n # just the first two words\n if len(frequencies) == 1:\n # we only have one word. We make it big!\n font_size = self.height\n else:\n self.generate_from_frequencies(dict(frequencies[:2]),\n max_font_size=self.height)\n # find font sizes\n sizes = [x[1] for x in self.layout_]\n try:\n font_size = int(2 * sizes[0] * sizes[1]\n / (sizes[0] + sizes[1]))\n # quick fix for if self.layout_ contains less than 2 values\n # on very small images it can be empty\n except IndexError:\n try:\n font_size = sizes[0]\n except IndexError:\n raise ValueError(\n \"Couldn't find space to draw. Either the Canvas size\"\n \" is too small or too much of the image is masked \"\n \"out.\")\n else:\n font_size = max_font_size\n\n # we set self.words_ here because we called generate_from_frequencies\n # above... hurray for good design?\n self.words_ = dict(frequencies)\n\n if self.repeat and len(frequencies) < self.max_words:\n # pad frequencies with repeating words.\n times_extend = int(np.ceil(self.max_words / len(frequencies))) - 1\n # get smallest frequency\n frequencies_org = list(frequencies)\n downweight = frequencies[-1][1]\n for i in range(times_extend):\n frequencies.extend([(word, freq * downweight ** (i + 1))\n for word, freq in frequencies_org])\n\n # start drawing grey image\n for word, freq in frequencies:\n # select the font size\n rs = self.relative_scaling\n if rs != 0:\n font_size = int(round((rs * (freq / float(last_freq))\n + (1 - rs)) * font_size))\n if random_state.random() < self.prefer_horizontal:\n orientation = None\n else:\n orientation = Image.ROTATE_90\n tried_other_orientation = False\n while True:\n # try to find a position\n font = ImageFont.truetype(self.font_path, font_size)\n # transpose font optionally\n transposed_font = ImageFont.TransposedFont(\n font, orientation=orientation)\n # get size of resulting text\n box_size = draw.textsize(word, font=transposed_font)\n # find possible places using integral image:\n result = occupancy.sample_position(box_size[1] + self.margin,\n box_size[0] + self.margin,\n random_state)\n if result is not None or font_size < self.min_font_size:\n # either we found a place or font-size went too small\n break\n # if we didn't find a place, make font smaller\n # but first try to rotate!\n if not tried_other_orientation and self.prefer_horizontal < 1:\n orientation = (Image.ROTATE_90 if orientation is None else\n Image.ROTATE_90)\n tried_other_orientation = True\n else:\n font_size -= self.font_step\n orientation = None\n\n if font_size < self.min_font_size:\n # we were unable to draw any more\n break\n\n x, y = np.array(result) + self.margin // 2\n # actually draw the text\n draw.text((y, x), word, fill=\"white\", font=transposed_font)\n positions.append((x, y))\n orientations.append(orientation)\n font_sizes.append(font_size)\n colors.append(self.color_func(word, font_size=font_size,\n position=(x, y),\n orientation=orientation,\n random_state=random_state,\n font_path=self.font_path))\n # recompute integral image\n if self.mask is None:\n img_array = np.asarray(img_grey)\n else:\n img_array = np.asarray(img_grey) + boolean_mask\n # recompute bottom right\n # the order of the cumsum's is important for speed ?!\n occupancy.update(img_array, x, y)\n last_freq = freq\n\n self.layout_ = list(zip(frequencies, font_sizes, positions,\n orientations, colors))\n return self", "response": "Generate a word cloud from words and associated frequencies."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsplit a long text into words eliminates the stopwords and returns a list of lists of word counts.", "response": "def process_text(self, text):\n \"\"\"Splits a long text into words, eliminates the stopwords.\n\n Parameters\n ----------\n text : string\n The text to be processed.\n\n Returns\n -------\n words : dict (string, int)\n Word tokens with associated frequency.\n\n ..versionchanged:: 1.2.2\n Changed return type from list of tuples to dict.\n\n Notes\n -----\n There are better ways to do word tokenization, but I don't want to\n include all those things.\n \"\"\"\n\n stopwords = set([i.lower() for i in self.stopwords])\n\n flags = (re.UNICODE if sys.version < '3' and type(text) is unicode # noqa: F821\n else 0)\n regexp = self.regexp if self.regexp is not None else r\"\\w[\\w']+\"\n\n words = re.findall(regexp, text, flags)\n # remove stopwords\n words = [word for word in words if word.lower() not in stopwords]\n # remove 's\n words = [word[:-2] if word.lower().endswith(\"'s\") else word\n for word in words]\n # remove numbers\n if not self.include_numbers:\n words = [word for word in words if not word.isdigit()]\n # remove short words\n if self.min_word_length:\n words = [word for word in words if len(word) >= self.min_word_length]\n\n if self.collocations:\n word_counts = unigrams_and_bigrams(words, self.normalize_plurals)\n else:\n word_counts, _ = process_tokens(words, self.normalize_plurals)\n\n return word_counts"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates wordcloud from text.", "response": "def generate_from_text(self, text):\n \"\"\"Generate wordcloud from text.\n\n The input \"text\" is expected to be a natural text. If you pass a sorted\n list of words, words will appear in your output twice. To remove this\n duplication, set ``collocations=False``.\n\n Calls process_text and generate_from_frequencies.\n\n ..versionchanged:: 1.2.2\n Argument of generate_from_frequencies() is not return of\n process_text() any more.\n\n Returns\n -------\n self\n \"\"\"\n words = self.process_text(text)\n self.generate_from_frequencies(words)\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexports to image file.", "response": "def to_file(self, filename):\n \"\"\"Export to image file.\n\n Parameters\n ----------\n filename : string\n Location to write to.\n\n Returns\n -------\n self\n \"\"\"\n\n img = self.to_image()\n img.save(filename, optimize=True)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_bolean_mask(self, mask):\n if mask.dtype.kind == 'f':\n warnings.warn(\"mask image should be unsigned byte between 0\"\n \" and 255. Got a float array\")\n if mask.ndim == 2:\n boolean_mask = mask == 255\n elif mask.ndim == 3:\n # if all channels are white, mask out\n boolean_mask = np.all(mask[:, :, :3] == 255, axis=-1)\n else:\n raise ValueError(\"Got mask of invalid shape: %s\" % str(mask.shape))\n return boolean_mask", "response": "Cast to two dimensional boolean mask."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing a contour on a pillow image.", "response": "def _draw_contour(self, img):\n \"\"\"Draw mask contour on a pillow image.\"\"\"\n if self.mask is None or self.contour_width == 0:\n return img\n\n mask = self._get_bolean_mask(self.mask) * 255\n contour = Image.fromarray(mask.astype(np.uint8))\n contour = contour.resize(img.size)\n contour = contour.filter(ImageFilter.FIND_EDGES)\n contour = np.array(contour)\n\n # make sure borders are not drawn before changing width\n contour[[0, -1], :] = 0\n contour[:, [0, -1]] = 0\n\n # use gaussian to change width, divide by 10 to give more resolution\n radius = self.contour_width / 10\n contour = Image.fromarray(contour)\n contour = contour.filter(ImageFilter.GaussianBlur(radius=radius))\n contour = np.array(contour) > 0\n contour = np.dstack((contour, contour, contour))\n\n # color the contour\n ret = np.array(img) * np.invert(contour)\n if self.contour_color != 'black':\n color = Image.new(img.mode, img.size, self.contour_color)\n ret += np.array(color) * contour\n\n return Image.fromarray(ret)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking the status of a given object and stores it in the service check.", "response": "def check_election_status(self, config):\n \"\"\"\n Retrieves the leader-election annotation from a given object, and\n submits metrics and a service check.\n\n An integration warning is sent if the object is not retrievable,\n or no record is found. Monitors on the service-check should have\n no-data alerts enabled to account for this.\n\n The config objet requires the following fields:\n namespace (prefix for the metrics and check)\n record_kind (endpoints or configmap)\n record_name\n record_namespace\n tags (optional)\n\n It reads the following agent configuration:\n kubernetes_kubeconfig_path: defaut is to use in-cluster config\n \"\"\"\n try:\n record = self._get_record(\n config.get(\"record_kind\", \"\"), config.get(\"record_name\", \"\"), config.get(\"record_namespace\", \"\")\n )\n self._report_status(config, record)\n except Exception as e:\n self.warning(\"Cannot retrieve leader election record {}: {}\".format(config.get(\"record_name\", \"\"), e))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _partition(entity, sep):\n parts = entity.split(sep, 1)\n if len(parts) == 2:\n return parts[0], sep, parts[1]\n else:\n return entity, '', ''", "response": "Python 2. 4 doesn t have a partition method so we have to use str. partition."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_userinfo(userinfo):\n if '@' in userinfo or userinfo.count(':') > 1:\n if PY3:\n quote_fn = \"urllib.parse.quote_plus\"\n else:\n quote_fn = \"urllib.quote_plus\"\n raise InvalidURI(\"Username and password must be escaped according to \"\n \"RFC 3986, use %s().\" % quote_fn)\n user, _, passwd = _partition(userinfo, \":\")\n # No password is expected with GSSAPI authentication.\n if not user:\n raise InvalidURI(\"The empty string is not valid username.\")\n return unquote_plus(user), unquote_plus(passwd)", "response": "Validates the format of user information in a MongoDB URI."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_ipv6_literal_host(entity, default_port):\n if entity.find(']') == -1:\n raise ValueError(\"an IPv6 address literal must be \"\n \"enclosed in '[' and ']' according \"\n \"to RFC 2732.\")\n i = entity.find(']:')\n if i == -1:\n return entity[1:-1], default_port\n return entity[1: i], entity[i + 2:]", "response": "Validates an IPv6 literal host string. Returns a 2 - tuple of IPv6 literal followed by port where the port is default_port."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates a string containing a host and port and returns a 2 - tuple of host followed by port where port is default_port.", "response": "def parse_host(entity, default_port=DEFAULT_PORT):\n \"\"\"Validates a host string\n\n Returns a 2-tuple of host followed by port where port is default_port\n if it wasn't specified in the string.\n\n :Parameters:\n - `entity`: A host or host:port string where host could be a\n hostname or IP address.\n - `default_port`: The port number to use when one wasn't\n specified in entity.\n \"\"\"\n host = entity\n port = default_port\n if entity[0] == '[':\n host, port = parse_ipv6_literal_host(entity, default_port)\n elif entity.endswith(\".sock\"):\n return entity, default_port\n elif entity.find(':') != -1:\n if entity.count(':') > 1:\n raise ValueError(\"Reserved characters such as ':' must be \"\n \"escaped according RFC 2396. An IPv6 \"\n \"address literal must be enclosed in '[' \"\n \"and ']' according to RFC 2732.\")\n host, port = host.split(':', 1)\n if isinstance(port, string_type):\n if not port.isdigit() or int(port) > 65535 or int(port) <= 0:\n raise ValueError(\"Port must be an integer between 0 and 65535: %s\"\n % (port,))\n port = int(port)\n\n # Normalize hostname to lowercase, since DNS is case-insensitive:\n # http://tools.ietf.org/html/rfc4343\n # This prevents useless rediscovery if \"foo.com\" is in the seed list but\n # \"FOO.com\" is in the ismaster response.\n return host.lower(), port"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef split_hosts(hosts, default_port=DEFAULT_PORT):\n nodes = []\n for entity in hosts.split(','):\n if not entity:\n raise ConfigurationError(\"Empty host \"\n \"(or extra comma in host list).\")\n port = default_port\n # Unix socket entities don't have ports\n if entity.endswith('.sock'):\n port = None\n nodes.append(parse_host(entity, port))\n return nodes", "response": "Takes a string of the form host1. port host2. port... and splits it into host1. port and host2. port. default_port is used when no port is specified."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_uri(uri, default_port=DEFAULT_PORT, validate=True, warn=False):\n if not uri.startswith(SCHEME):\n raise InvalidURI(\"Invalid URI scheme: URI \"\n \"must begin with '%s'\" % (SCHEME,))\n\n scheme_free = uri[SCHEME_LEN:]\n\n if not scheme_free:\n raise InvalidURI(\"Must provide at least one hostname or IP.\")\n\n user = None\n passwd = None\n dbase = None\n collection = None\n options = {}\n\n host_part, _, path_part = _partition(scheme_free, '/')\n if not host_part:\n host_part = path_part\n path_part = \"\"\n\n if not path_part and '?' in host_part:\n raise InvalidURI(\"A '/' is required between \"\n \"the host list and any options.\")\n\n if '@' in host_part:\n userinfo, _, hosts = _rpartition(host_part, '@')\n user, passwd = parse_userinfo(userinfo)\n else:\n hosts = host_part\n\n if '/' in hosts:\n raise InvalidURI(\"Any '/' in a unix domain socket must be\"\n \" percent-encoded: %s\" % host_part)\n\n hosts = unquote_plus(hosts)\n nodes = split_hosts(hosts, default_port=default_port)\n\n if path_part:\n if path_part[0] == '?':\n opts = unquote_plus(path_part[1:])\n else:\n dbase, _, opts = map(unquote_plus, _partition(path_part, '?'))\n if '.' in dbase:\n dbase, collection = dbase.split('.', 1)\n\n if _BAD_DB_CHARS.search(dbase):\n raise InvalidURI('Bad database name \"%s\"' % dbase)\n\n if opts:\n options = split_options(opts, validate, warn)\n\n if dbase is not None:\n dbase = unquote_plus(dbase)\n if collection is not None:\n collection = unquote_plus(collection)\n\n return {\n 'nodelist': nodes,\n 'username': user,\n 'password': passwd,\n 'database': dbase,\n 'collection': collection,\n 'options': options\n }", "response": "Parse and validate a MongoDB URI."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\napply max_staleness in seconds to a Selection with a known primary.", "response": "def _with_primary(max_staleness, selection):\n \"\"\"Apply max_staleness, in seconds, to a Selection with a known primary.\"\"\"\n primary = selection.primary\n sds = []\n\n for s in selection.server_descriptions:\n if s.server_type == SERVER_TYPE.RSSecondary:\n # See max-staleness.rst for explanation of this formula.\n staleness = (\n (s.last_update_time - s.last_write_date) -\n (primary.last_update_time - primary.last_write_date) +\n selection.heartbeat_frequency)\n\n if staleness <= max_staleness:\n sds.append(s)\n else:\n sds.append(s)\n\n return selection.with_server_descriptions(sds)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _no_primary(max_staleness, selection):\n # Secondary that's replicated the most recent writes.\n smax = selection.secondary_with_max_last_write_date()\n if not smax:\n # No secondaries and no primary, short-circuit out of here.\n return selection.with_server_descriptions([])\n\n sds = []\n\n for s in selection.server_descriptions:\n if s.server_type == SERVER_TYPE.RSSecondary:\n # See max-staleness.rst for explanation of this formula.\n staleness = (smax.last_write_date -\n s.last_write_date +\n selection.heartbeat_frequency)\n\n if staleness <= max_staleness:\n sds.append(s)\n else:\n sds.append(s)\n\n return selection.with_server_descriptions(sds)", "response": "Apply max_staleness in seconds to a Selection with no known primary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef select(max_staleness, selection):\n if max_staleness == -1:\n return selection\n\n # Server Selection Spec: If the TopologyType is ReplicaSetWithPrimary or\n # ReplicaSetNoPrimary, a client MUST raise an error if maxStaleness <\n # heartbeatFrequency + IDLE_WRITE_PERIOD, or if maxStaleness < 90.\n _validate_max_staleness(max_staleness, selection.heartbeat_frequency)\n\n if selection.primary:\n return _with_primary(max_staleness, selection)\n else:\n return _no_primary(max_staleness, selection)", "response": "Apply max_staleness in seconds to a Selection."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates an event describing the replication state of a mongo node", "response": "def create_event(self, state, server, agentConfig):\n \"\"\"Create an event with a message describing the replication\n state of a mongo node\"\"\"\n\n def get_state_description(state):\n if state == 0:\n return 'Starting Up'\n elif state == 1:\n return 'Primary'\n elif state == 2:\n return 'Secondary'\n elif state == 3:\n return 'Recovering'\n elif state == 4:\n return 'Fatal'\n elif state == 5:\n return 'Starting up (initial sync)'\n elif state == 6:\n return 'Unknown'\n elif state == 7:\n return 'Arbiter'\n elif state == 8:\n return 'Down'\n elif state == 9:\n return 'Rollback'\n\n status = get_state_description(state)\n msg_title = \"%s is %s\" % (server, status)\n msg = \"TokuMX %s just reported as %s\" % (server, status)\n\n self.event(\n {\n 'timestamp': int(time.time()),\n 'event_type': 'tokumx',\n 'msg_title': msg_title,\n 'msg_text': msg,\n 'host': self.hostname,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind all processes with the given name and search string.", "response": "def find_pids(self, name, search_string, exact_match, ignore_ad=True):\n \"\"\"\n Create a set of pids of selected processes.\n Search for search_string\n \"\"\"\n if not self.should_refresh_pid_cache(name):\n return self.pid_cache[name]\n\n ad_error_logger = self.log.debug\n if not ignore_ad:\n ad_error_logger = self.log.error\n\n refresh_ad_cache = self.should_refresh_ad_cache(name)\n\n matching_pids = set()\n\n for proc in psutil.process_iter():\n # Skip access denied processes\n if not refresh_ad_cache and proc.pid in self.ad_cache:\n continue\n\n found = False\n for string in search_string:\n try:\n # FIXME 8.x: All has been deprecated\n # from the doc, should be removed\n if string == 'All':\n found = True\n if exact_match:\n if os.name == 'nt':\n if proc.name().lower() == string.lower():\n found = True\n else:\n if proc.name() == string:\n found = True\n\n else:\n cmdline = proc.cmdline()\n if os.name == 'nt':\n lstring = string.lower()\n if re.search(lstring, ' '.join(cmdline).lower()):\n found = True\n else:\n if re.search(string, ' '.join(cmdline)):\n found = True\n except psutil.NoSuchProcess:\n self.log.warning('Process disappeared while scanning')\n except psutil.AccessDenied as e:\n ad_error_logger('Access denied to process with PID {}'.format(proc.pid))\n ad_error_logger('Error: {}'.format(e))\n if refresh_ad_cache:\n self.ad_cache.add(proc.pid)\n if not ignore_ad:\n raise\n else:\n if refresh_ad_cache:\n self.ad_cache.discard(proc.pid)\n if found:\n matching_pids.add(proc.pid)\n break\n\n self.pid_cache[name] = matching_pids\n self.last_pid_cache_ts[name] = time.time()\n if refresh_ad_cache:\n self.last_ad_cache_ts[name] = time.time()\n return matching_pids"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef psutil_wrapper(self, process, method, accessors, try_sudo, *args, **kwargs):\n\n if accessors is None:\n result = None\n else:\n result = {}\n\n # Ban certain method that we know fail\n if method == 'num_fds' and not Platform.is_unix():\n return result\n elif method == 'num_handles' and not Platform.is_win32():\n return result\n\n try:\n res = getattr(process, method)(*args, **kwargs)\n if accessors is None:\n result = res\n else:\n for acc in accessors:\n try:\n result[acc] = getattr(res, acc)\n except AttributeError:\n self.log.debug(\"psutil.{}().{} attribute does not exist\".format(method, acc))\n except (NotImplementedError, AttributeError):\n self.log.debug(\"psutil method {} not implemented\".format(method))\n except psutil.AccessDenied:\n self.log.debug(\"psutil was denied access for method {}\".format(method))\n if method == 'num_fds' and Platform.is_unix() and try_sudo:\n try:\n # It is up the agent's packager to grant\n # corresponding sudo policy on unix platforms\n ls_args = ['sudo', 'ls', '/proc/{}/fd/'.format(process.pid)]\n process_ls = subprocess.check_output(ls_args)\n result = len(process_ls.splitlines())\n\n except subprocess.CalledProcessError as e:\n self.log.exception(\n \"trying to retrieve {} with sudo failed with return code {}\".format(method, e.returncode)\n )\n except Exception:\n self.log.exception(\"trying to retrieve {} with sudo also failed\".format(method))\n except psutil.NoSuchProcess:\n self.warning(\"Process {} disappeared while scanning\".format(process.pid))\n\n return result", "response": "A wrapper that returns the result of calling the method with the given arguments and the given kwargs."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreport a service check for each process in search_string.", "response": "def _process_service_check(self, name, nb_procs, bounds, tags):\n \"\"\"\n Report a service check, for each process in search_string.\n Report as OK if the process is in the warning thresholds\n CRITICAL out of the critical thresholds\n WARNING out of the warning thresholds\n \"\"\"\n # FIXME 8.x remove the `process:name` tag\n service_check_tags = tags + [\"process:{}\".format(name)]\n status = AgentCheck.OK\n status_str = {AgentCheck.OK: \"OK\", AgentCheck.WARNING: \"WARNING\", AgentCheck.CRITICAL: \"CRITICAL\"}\n\n if not bounds and nb_procs < 1:\n status = AgentCheck.CRITICAL\n elif bounds:\n warning = bounds.get('warning', [1, float('inf')])\n critical = bounds.get('critical', [1, float('inf')])\n\n if warning[1] < nb_procs or nb_procs < warning[0]:\n status = AgentCheck.WARNING\n if critical[1] < nb_procs or nb_procs < critical[0]:\n status = AgentCheck.CRITICAL\n\n self.service_check(\n \"process.up\",\n status,\n tags=service_check_tags,\n message=\"PROCS {}: {} processes found for {}\".format(status_str[status], nb_procs, name),\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _filter_by_user(self, user, pids):\n filtered_pids = set()\n for pid in pids:\n try:\n proc = psutil.Process(pid)\n if proc.username() == user:\n self.log.debug(\"Collecting pid {} belonging to {}\".format(pid, user))\n filtered_pids.add(pid)\n else:\n self.log.debug(\"Discarding pid {} not belonging to {}\".format(pid, user))\n except psutil.NoSuchProcess:\n pass\n\n return filtered_pids", "response": "Filter pids by user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _msg_text(self):\n msg_text = u\"\"\n\n if self._format:\n msg_text_fields = [\"%%%\\n```\"]\n\n for event_property in self._format:\n property_value = self.event.get(event_property)\n if property_value is None:\n self.log.warning(u\"Unrecognized `%s` event property.\", event_property)\n continue\n msg_text_fields.append(\n u\"{property_name}: {property_value}\".format(\n property_name=event_property, property_value=property_value\n )\n )\n\n msg_text_fields.append(\"```\\n%%%\")\n\n msg_text = u\"\\n\".join(msg_text_fields)\n else:\n # Override when verbosity\n if self.event.get('Message'):\n msg_text = u\"{message}\\n\".format(message=self.event['Message'])\n elif self.event.get('InsertionStrings'):\n msg_text = u\"\\n\".join([i_str for i_str in self.event['InsertionStrings'] if i_str.strip()])\n\n if self.notify_list:\n msg_text += u\"\\n{notify_list}\".format(notify_list=' '.join([\" @\" + n for n in self.notify_list]))\n\n return msg_text", "response": "Generates the event s body to send to Datadog."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_after(self, ts):\n ''' Compare this event's timestamp to a give timestamp. '''\n if self.timestamp >= int(calendar.timegm(ts.timetuple())):\n return True\n return False", "response": "Compare this event s timestamp to a give timestamp."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a wmi formatted timestamp into an epoch.", "response": "def _wmi_to_ts(self, wmi_ts):\n ''' Convert a wmi formatted timestamp into an epoch.\n '''\n year, month, day, hour, minute, second, microsecond, tz = to_time(wmi_ts)\n tz_delta = timedelta(minutes=int(tz))\n if '+' in wmi_ts:\n tz_delta = -tz_delta\n\n dt = (\n datetime(year=year, month=month, day=day, hour=hour, minute=minute, second=second, microsecond=microsecond)\n + tz_delta\n )\n return int(calendar.timegm(dt.timetuple()))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _tags(self, tags, event_code):\n ''' Inject additional tags into the list already supplied to LogEvent.\n '''\n tags_list = []\n if tags is not None:\n tags_list += list(tags)\n tags_list.append(\"event_id:{event_id}\".format(event_id=event_code))\n return tags_list", "response": "Inject additional tags into the list already supplied to LogEvent.\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nkill a cursor. Raises TypeError if cursor_id is not an instance of (int, long). :Parameters: - `cursor_id`: cursor id to close - `address`: the cursor's server's (host, port) pair .. versionchanged:: 3.0 Now requires an `address` argument.", "response": "def close(self, cursor_id, address):\n \"\"\"Kill a cursor.\n\n Raises TypeError if cursor_id is not an instance of (int, long).\n\n :Parameters:\n - `cursor_id`: cursor id to close\n - `address`: the cursor's server's (host, port) pair\n\n .. versionchanged:: 3.0\n Now requires an `address` argument.\n \"\"\"\n if not isinstance(cursor_id, integer_types):\n raise TypeError(\"cursor_id must be an integer\")\n\n self.__client().kill_cursors([cursor_id], address)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the health status of a specific resource.", "response": "def _get_health_status(self, url, ssl_params, timeout):\n \"\"\"\n Don't send the \"can connect\" service check if we have troubles getting\n the health status\n \"\"\"\n try:\n r = self._perform_request(url, \"/health\", ssl_params, timeout)\n # we don't use get() here so we can report a KeyError\n return r.json()[self.HEALTH_KEY]\n except Exception as e:\n self.log.debug(\"Can't determine health status: {}\".format(e))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stop(check, env):\n all_checks = check == 'all'\n checks = get_configured_checks() if all_checks else [check]\n\n if all_checks:\n env_indent = DEFAULT_INDENT\n status_indent = DEFAULT_INDENT * 2\n else:\n env_indent = None\n status_indent = DEFAULT_INDENT\n\n for check in checks:\n if all_checks:\n envs = get_configured_envs(check)\n if envs:\n echo_success('{}:'.format(check))\n else:\n envs = [env] if env else get_configured_envs(check)\n\n for env in envs:\n echo_info('{}:'.format(env), indent=env_indent)\n environment = create_interface(check, env)\n\n echo_waiting('Stopping the Agent... ', nl=False, indent=status_indent)\n environment.stop_agent()\n echo_success('success!')\n\n echo_waiting('Removing configuration files... ', nl=False, indent=status_indent)\n environment.remove_config()\n echo_success('success!')\n\n echo_waiting('Stopping the environment... ', nl=False, indent=status_indent)\n _, _, error = stop_environment(check, env, metadata=environment.metadata)\n if error:\n echo_failure('failed!')\n abort(error)\n echo_success('success!')", "response": "Stop the check and environment."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprocesses both the istio_mesh instance and process_mixer instance associated with this instance", "response": "def check(self, instance):\n \"\"\"\n Process both the istio_mesh instance and process_mixer instance associated with this instance\n \"\"\"\n\n # Get the config for the istio_mesh instance\n istio_mesh_endpoint = instance.get('istio_mesh_endpoint')\n istio_mesh_config = self.config_map[istio_mesh_endpoint]\n\n # Process istio_mesh\n self.process(istio_mesh_config)\n\n # Get the config for the process_mixer instance\n process_mixer_endpoint = instance.get('mixer_endpoint')\n process_mixer_config = self.config_map[process_mixer_endpoint]\n\n # Process process_mixer\n self.process(process_mixer_config)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngeneralizes each Istio instance into two OpenMetricsBaseCheck instances", "response": "def create_generic_instances(self, instances):\n \"\"\"\n Generalize each (single) Istio instance into two OpenMetricsBaseCheck instances\n \"\"\"\n generic_instances = []\n\n for instance in instances:\n istio_mesh_instance = self._create_istio_mesh_instance(instance)\n process_mixer_instance = self._create_process_mixer_instance(instance)\n\n generic_instances.extend([istio_mesh_instance, process_mixer_instance])\n\n return generic_instances"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate the istio mesh scraper from the dict and return it if it exists otherwise create it and add it to the dict", "response": "def _create_istio_mesh_instance(self, instance):\n \"\"\"\n Grab the istio mesh scraper from the dict and return it if it exists,\n otherwise create the scraper and add it to the dict\n \"\"\"\n endpoint = instance.get('istio_mesh_endpoint')\n\n if endpoint is None:\n raise CheckException(\"Unable to find istio_mesh_endpoint in config file.\")\n\n istio_mesh_instance = deepcopy(instance)\n istio_mesh_instance.update(\n {\n 'namespace': self.MESH_NAMESPACE,\n 'prometheus_url': endpoint,\n 'label_to_hostname': endpoint,\n 'metrics': [\n {\n # These metrics support Istio 1.0\n 'istio_requests_total': 'request.count',\n 'istio_request_duration_seconds': 'request.duration',\n 'istio_request_bytes': 'request.size',\n 'istio_response_bytes': 'response.size',\n # These metrics support Istio 0.8\n 'istio_request_count': 'request.count',\n 'istio_request_duration': 'request.duration',\n 'istio_request_size': 'request.size',\n 'istio_response_size': 'response.size',\n }\n ],\n # Defaults that were set when istio was based on PrometheusCheck\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n\n return istio_mesh_instance"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _create_process_mixer_instance(self, instance):\n endpoint = instance.get('mixer_endpoint')\n if endpoint is None:\n raise CheckException(\"Unable to find mixer_endpoint in config file.\")\n\n process_mixer_instance = deepcopy(instance)\n process_mixer_instance.update(\n {\n 'namespace': self.MIXER_NAMESPACE,\n 'prometheus_url': endpoint,\n 'metrics': [\n {\n 'go_gc_duration_seconds': 'go.gc_duration_seconds',\n 'go_goroutines': 'go.goroutines',\n 'go_info': 'go.info',\n 'go_memstats_alloc_bytes': 'go.memstats.alloc_bytes',\n 'go_memstats_alloc_bytes_total': 'go.memstats.alloc_bytes_total',\n 'go_memstats_buck_hash_sys_bytes': 'go.memstats.buck_hash_sys_bytes',\n 'go_memstats_frees_total': 'go.memstats.frees_total',\n 'go_memstats_gc_cpu_fraction': 'go.memstats.gc_cpu_fraction',\n 'go_memstats_gc_sys_bytes': 'go.memstats.gc_sys_bytes',\n 'go_memstats_heap_alloc_bytes': 'go.memstats.heap_alloc_bytes',\n 'go_memstats_heap_idle_bytes': 'go.memstats.heap_idle_bytes',\n 'go_memstats_heap_inuse_bytes': 'go.memstats.heap_inuse_bytes',\n 'go_memstats_heap_objects': 'go.memstats.heap_objects',\n 'go_memstats_heap_released_bytes': 'go.memstats.heap_released_bytes',\n 'go_memstats_heap_sys_bytes': 'go.memstats.heap_sys_bytes',\n 'go_memstats_last_gc_time_seconds': 'go.memstats.last_gc_time_seconds',\n 'go_memstats_lookups_total': 'go.memstats.lookups_total',\n 'go_memstats_mallocs_total': 'go.memstats.mallocs_total',\n 'go_memstats_mcache_inuse_bytes': 'go.memstats.mcache_inuse_bytes',\n 'go_memstats_mcache_sys_bytes': 'go.memstats.mcache_sys_bytes',\n 'go_memstats_mspan_inuse_bytes': 'go.memstats.mspan_inuse_bytes',\n 'go_memstats_mspan_sys_bytes': 'go.memstats.mspan_sys_bytes',\n 'go_memstats_next_gc_bytes': 'go.memstats.next_gc_bytes',\n 'go_memstats_other_sys_bytes': 'go.memstats.other_sys_bytes',\n 'go_memstats_stack_inuse_bytes': 'go.memstats.stack_inuse_bytes',\n 'go_memstats_stack_sys_bytes': 'go.memstats.stack_sys_bytes',\n 'go_memstats_sys_bytes': 'go.memstats.sys_bytes',\n 'go_threads': 'go.threads',\n 'grpc_server_handled_total': 'grpc.server.handled_total',\n 'grpc_server_handling_seconds': 'grpc.server.handling_seconds',\n 'grpc_server_msg_received_total': 'grpc.server.msg_received_total',\n 'grpc_server_msg_sent_total': 'grpc.server.msg_sent_total',\n 'grpc_server_started_total': 'grpc.server.started_total',\n 'mixer_adapter_dispatch_count': 'adapter.dispatch_count',\n 'mixer_adapter_dispatch_duration': 'adapter.dispatch_duration',\n 'mixer_adapter_old_dispatch_count': 'adapter.old_dispatch_count',\n 'mixer_adapter_old_dispatch_duration': 'adapter.old_dispatch_duration',\n 'mixer_config_resolve_actions': 'config.resolve_actions',\n 'mixer_config_resolve_count': 'config.resolve_count',\n 'mixer_config_resolve_duration': 'config.resolve_duration',\n 'mixer_config_resolve_rules': 'config.resolve_rules',\n 'process_cpu_seconds_total': 'process.cpu_seconds_total',\n 'process_max_fds': 'process.max_fds',\n 'process_open_fds': 'process.open_fds',\n 'process_resident_memory_bytes': 'process.resident_memory_bytes',\n 'process_start_time_seconds': 'process.start_time_seconds',\n 'process_virtual_memory_bytes': 'process.virtual_memory_bytes',\n }\n ],\n # Defaults that were set when istio was based on PrometheusCheck\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n\n return process_mixer_instance", "response": "Create the process_mixer_instance from the dict and return it if it exists otherwise create the scraper and add it to the dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the apps from Marathon and return the response.", "response": "def get_apps_json(self, url, timeout, auth, acs_url, ssl_verify, tags, group):\n \"\"\"\n The dictionary containing the apps is cached during collection and reset\n at every `check()` call.\n \"\"\"\n if self.apps_response is not None:\n return self.apps_response\n\n # Marathon apps\n if group is None:\n # embed=apps.counts is not a required parameter but will be in the future:\n # http://mesosphere.github.io/marathon/1.4/docs/rest-api.html#get-v2apps\n marathon_path = urljoin(url, \"v2/apps?embed=apps.counts\")\n else:\n marathon_path = urljoin(\n url, \"v2/groups/{}?embed=group.groups\".format(group) + \"&embed=group.apps&embed=group.apps.counts\"\n )\n\n self.apps_response = self.get_json(marathon_path, timeout, auth, acs_url, ssl_verify, tags)\n return self.apps_response"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nensures queue. count is reported as zero for apps with queued instances.", "response": "def ensure_queue_count(\n self, queued, url, timeout, auth, acs_url, ssl_verify, tags=None, label_tags=None, group=None\n ):\n \"\"\"\n Ensure `marathon.queue.count` is reported as zero for apps without queued instances.\n \"\"\"\n metric_name = '{}.count'.format(self.QUEUE_PREFIX)\n\n apps_response = self.get_apps_json(url, timeout, auth, acs_url, ssl_verify, tags, group)\n\n for app in apps_response['apps']:\n if app['id'] not in queued:\n q_tags = self.get_app_tags(app, tags, label_tags)\n self.gauge(metric_name, 0, tags=q_tags)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef py3(check):\n\n if check in get_valid_checks():\n path_to_module = os.path.dirname(get_version_file(check))\n else:\n path_to_module = check\n\n if not os.path.exists(path_to_module):\n abort(u\"{} does not exist.\".format(path_to_module))\n\n echo_info(u\"Validating python3 compatibility of {}...\".format(check))\n results = validate_py3(path_to_module)\n\n if results:\n echo_failure(u\"Incompatibilities were found for {}:\".format(check))\n current_path = None\n for problem in sorted(results, key=itemgetter(\"path\")):\n # validate_py3 returns an array a dicts like\n # {\n # \"message\": \"Line 23, Column 8: Calling a dict.iter*() method\",\n # \"file\": \"/path/to/file.py\",\n # }\n path = problem[\"path\"]\n if current_path is None or path != current_path:\n echo_info(u\"File {}:\".format(path))\n echo_failure(\" {}\".format(problem[\"message\"]))\n current_path = path\n abort()\n else:\n echo_success(u\"{} is compatible with python3\".format(check))", "response": "Verify if a custom check can run on python 3."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_instance(instance, default_ca_certs=None):\n method = instance.get('method', 'get')\n data = instance.get('data', {})\n tags = instance.get('tags', [])\n ntlm_domain = instance.get('ntlm_domain')\n username = instance.get('username')\n password = instance.get('password')\n client_cert = instance.get('client_cert')\n client_key = instance.get('client_key')\n http_response_status_code = str(instance.get('http_response_status_code', DEFAULT_EXPECTED_CODE))\n timeout = int(instance.get('timeout', 10))\n config_headers = instance.get('headers', {})\n default_headers = is_affirmative(instance.get(\"include_default_headers\", True))\n if default_headers:\n headers = agent_headers({})\n else:\n headers = {}\n headers.update(config_headers)\n url = instance.get('url')\n if url is not None:\n url = ensure_unicode(url)\n content_match = instance.get('content_match')\n if content_match is not None:\n content_match = ensure_unicode(content_match)\n reverse_content_match = is_affirmative(instance.get('reverse_content_match', False))\n response_time = is_affirmative(instance.get('collect_response_time', True))\n if not url:\n raise ConfigurationError(\"Bad configuration. You must specify a url\")\n if not url.startswith(\"http\"):\n raise ConfigurationError(\"The url {} must start with the scheme http or https\".format(url))\n include_content = is_affirmative(instance.get('include_content', False))\n disable_ssl_validation = is_affirmative(instance.get('disable_ssl_validation', True))\n ssl_expire = is_affirmative(instance.get('check_certificate_expiration', True))\n instance_ca_certs = instance.get('ca_certs', default_ca_certs)\n weakcipher = is_affirmative(instance.get('weakciphers', False))\n ignore_ssl_warning = is_affirmative(instance.get('ignore_ssl_warning', False))\n check_hostname = is_affirmative(instance.get('check_hostname', True))\n skip_proxy = is_affirmative(instance.get('skip_proxy', instance.get('no_proxy', False)))\n allow_redirects = is_affirmative(instance.get('allow_redirects', True))\n stream = is_affirmative(instance.get('stream', False))\n\n return Config(\n url,\n ntlm_domain,\n username,\n password,\n client_cert,\n client_key,\n method,\n data,\n http_response_status_code,\n timeout,\n include_content,\n headers,\n response_time,\n content_match,\n reverse_content_match,\n tags,\n disable_ssl_validation,\n ssl_expire,\n instance_ca_certs,\n weakcipher,\n check_hostname,\n ignore_ssl_warning,\n skip_proxy,\n allow_redirects,\n stream,\n )", "response": "Create a config object from a dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the Python code in the version module so that it contains the new value.", "response": "def update_version_module(check_name, old_ver, new_ver):\n \"\"\"\n Change the Python code in the __about__.py module so that `__version__`\n contains the new value.\n \"\"\"\n version_file = get_version_file(check_name)\n contents = read_file(version_file)\n\n contents = contents.replace(old_ver, new_ver)\n write_file(version_file, contents)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_agent_requirement_line(check, version):\n package_name = get_package_name(check)\n\n # no manifest\n if check in ('datadog_checks_base', 'datadog_checks_downloader'):\n return '{}=={}'.format(package_name, version)\n\n m = load_manifest(check)\n platforms = sorted(m.get('supported_os', []))\n\n # all platforms\n if platforms == ALL_PLATFORMS:\n return '{}=={}'.format(package_name, version)\n # one specific platform\n elif len(platforms) == 1:\n return \"{}=={}; sys_platform == '{}'\".format(package_name, version, PLATFORMS_TO_PY.get(platforms[0]))\n elif platforms:\n if 'windows' not in platforms:\n return \"{}=={}; sys_platform != 'win32'\".format(package_name, version)\n elif 'mac_os' not in platforms:\n return \"{}=={}; sys_platform != 'darwin'\".format(package_name, version)\n elif 'linux' not in platforms:\n return \"{}=={}; sys_platform != 'linux2'\".format(package_name, version)\n\n raise ManifestError(\"Can't parse the `supported_os` list for the check {}: {}\".format(check, platforms))", "response": "Returns a text line to be used in a requirements. txt file to install a check\n pinned to a specific version."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_agent_requirements(req_file, check, newline):\n package_name = get_package_name(check)\n lines = read_file_lines(req_file)\n\n for i, line in enumerate(lines):\n current_package_name = line.split('==')[0]\n\n if current_package_name == package_name:\n lines[i] = '{}\\n'.format(newline)\n break\n\n write_file_lines(req_file, sorted(lines))", "response": "Update the requirements line for the given check"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _walk(top):\n dirs = []\n nondirs = []\n\n try:\n scandir_iter = scandir(top)\n except OSError:\n return\n\n # Avoid repeated global lookups.\n get_next = next\n\n while True:\n try:\n entry = get_next(scandir_iter)\n except StopIteration:\n break\n except OSError:\n return\n\n try:\n is_dir = entry.is_dir()\n except OSError:\n is_dir = False\n\n if is_dir:\n dirs.append(entry)\n else:\n nondirs.append(entry)\n\n yield top, dirs, nondirs\n\n for dir_entry in dirs:\n for entry in walk(dir_entry.path):\n yield entry", "response": "Modified version of os. scandir that returns https://docs. python. org / 3. html#os. DirEntry for files\n and os. stat calls."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _yarn_cluster_metrics(self, rm_address, instance, addl_tags):\n metrics_json = self._rest_request_to_json(rm_address, instance, YARN_CLUSTER_METRICS_PATH, addl_tags)\n\n if metrics_json:\n\n yarn_metrics = metrics_json[YARN_CLUSTER_METRICS_ELEMENT]\n\n if yarn_metrics is not None:\n self._set_yarn_metrics_from_json(addl_tags, yarn_metrics, YARN_CLUSTER_METRICS)", "response": "Get metrics related to YARN cluster"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _yarn_app_metrics(self, rm_address, instance, app_tags, addl_tags):\n metrics_json = self._rest_request_to_json(\n rm_address, instance, YARN_APPS_PATH, addl_tags, states=YARN_APPLICATION_STATES\n )\n\n if metrics_json and metrics_json['apps'] is not None and metrics_json['apps']['app'] is not None:\n\n for app_json in metrics_json['apps']['app']:\n\n tags = []\n for dd_tag, yarn_key in iteritems(app_tags):\n try:\n val = app_json[yarn_key]\n if val:\n tags.append('{tag}:{value}'.format(tag=dd_tag, value=val))\n except KeyError:\n self.log.error(\"Invalid value {} for application_tag\".format(yarn_key))\n\n tags.extend(addl_tags)\n\n self._set_yarn_metrics_from_json(tags, app_json, YARN_APP_METRICS)", "response": "Get metrics for running applications"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _yarn_node_metrics(self, rm_address, instance, addl_tags):\n metrics_json = self._rest_request_to_json(rm_address, instance, YARN_NODES_PATH, addl_tags)\n\n if metrics_json and metrics_json['nodes'] is not None and metrics_json['nodes']['node'] is not None:\n\n for node_json in metrics_json['nodes']['node']:\n node_id = node_json['id']\n\n tags = ['node_id:{}'.format(str(node_id))]\n tags.extend(addl_tags)\n\n self._set_yarn_metrics_from_json(tags, node_json, YARN_NODE_METRICS)", "response": "Get metrics related to YARN nodes"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _yarn_scheduler_metrics(self, rm_address, instance, addl_tags, queue_blacklist):\n metrics_json = self._rest_request_to_json(rm_address, instance, YARN_SCHEDULER_PATH, addl_tags)\n\n try:\n metrics_json = metrics_json['scheduler']['schedulerInfo']\n\n if metrics_json['type'] == 'capacityScheduler':\n self._yarn_capacity_scheduler_metrics(metrics_json, addl_tags, queue_blacklist)\n\n except KeyError:\n pass", "response": "Get metrics from YARN scheduler"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting metrics from YARN capacity scheduler if it s type is capacityScheduler", "response": "def _yarn_capacity_scheduler_metrics(self, metrics_json, addl_tags, queue_blacklist):\n \"\"\"\n Get metrics from YARN scheduler if it's type is capacityScheduler\n \"\"\"\n tags = ['queue_name:{}'.format(metrics_json['queueName'])]\n tags.extend(addl_tags)\n\n self._set_yarn_metrics_from_json(tags, metrics_json, YARN_ROOT_QUEUE_METRICS)\n\n if metrics_json['queues'] is not None and metrics_json['queues']['queue'] is not None:\n\n queues_count = 0\n for queue_json in metrics_json['queues']['queue']:\n queue_name = queue_json['queueName']\n\n if queue_name in queue_blacklist:\n self.log.debug('Queue \"{}\" is blacklisted. Ignoring it'.format(queue_name))\n continue\n\n queues_count += 1\n if queues_count > MAX_DETAILED_QUEUES:\n self.warning(\n \"Found more than 100 queues, will only send metrics on first 100 queues. \"\n \"Please filter the queues with the check's `queue_blacklist` parameter\"\n )\n break\n\n tags = ['queue_name:{}'.format(str(queue_name))]\n tags.extend(addl_tags)\n\n self._set_yarn_metrics_from_json(tags, queue_json, YARN_QUEUE_METRICS)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _set_yarn_metrics_from_json(self, tags, metrics_json, yarn_metrics):\n for dict_path, metric in iteritems(yarn_metrics):\n metric_name, metric_type = metric\n\n metric_value = self._get_value_from_json(dict_path, metrics_json)\n\n if metric_value is not None:\n self._set_metric(metric_name, metric_type, metric_value, tags)", "response": "Parse the JSON response and set the metrics"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a value from a dictionary under N keys represented as str ( key1. key2... keyN", "response": "def _get_value_from_json(self, dict_path, metrics_json):\n \"\"\"\n Get a value from a dictionary under N keys, represented as str(\"key1.key2...key{n}\")\n \"\"\"\n for key in dict_path.split('.'):\n if key in metrics_json:\n metrics_json = metrics_json.get(key)\n else:\n return None\n return metrics_json"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets a metric in the cache.", "response": "def _set_metric(self, metric_name, metric_type, value, tags=None, device_name=None):\n \"\"\"\n Set a metric\n \"\"\"\n if metric_type == GAUGE:\n self.gauge(metric_name, value, tags=tags, device_name=device_name)\n elif metric_type == INCREMENT:\n self.increment(metric_name, value, tags=tags, device_name=device_name)\n else:\n self.log.error('Metric type \"{}\" unknown'.format(metric_type))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _rest_request_to_json(self, url, instance, object_path, tags, *args, **kwargs):\n service_check_tags = ['url:{}'.format(self._get_url_base(url))] + tags\n service_check_tags = list(set(service_check_tags))\n\n if object_path:\n url = self._join_url_dir(url, object_path)\n\n # Add args to the url\n if args:\n for directory in args:\n url = self._join_url_dir(url, directory)\n\n auth = None\n\n # Authenticate our connection to JMX endpoint if required\n kerberos = instance.get('kerberos')\n username = instance.get('username')\n password = instance.get('password')\n if username is not None and password is not None:\n auth = (username, password)\n elif kerberos is not None:\n if kerberos not in KERBEROS_STRATEGIES:\n raise Exception('Invalid Kerberos strategy `{}`'.format(kerberos))\n\n auth = requests_kerberos.HTTPKerberosAuth(\n mutual_authentication=KERBEROS_STRATEGIES[kerberos],\n delegate=is_affirmative(instance.get('kerberos_delegate', False)),\n force_preemptive=is_affirmative(instance.get('kerberos_force_initiate', False)),\n hostname_override=instance.get('kerberos_hostname'),\n principal=instance.get('kerberos_principal'),\n )\n\n ssl_verify = is_affirmative(instance.get('ssl_verify', True))\n\n old_keytab_path = None\n if 'kerberos_keytab' in instance:\n old_keytab_path = os.getenv('KRB5_CLIENT_KTNAME')\n os.environ['KRB5_CLIENT_KTNAME'] = instance['kerberos_keytab']\n\n self.log.debug('Attempting to connect to \"{}\"'.format(url))\n\n # Add kwargs as arguments\n if kwargs:\n query = '&'.join(['{}={}'.format(key, value) for key, value in iteritems(kwargs)])\n url = urljoin(url, '?' + query)\n\n try:\n response = requests.get(url, auth=auth, verify=ssl_verify, timeout=self.default_integration_http_timeout)\n response.raise_for_status()\n response_json = response.json()\n\n except Timeout as e:\n self.service_check(\n SERVICE_CHECK_NAME,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"Request timeout: {}, {}\".format(url, e),\n )\n raise\n\n except (HTTPError, InvalidURL, ConnectionError, SSLError) as e:\n self.service_check(\n SERVICE_CHECK_NAME,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"Request failed: {}, {}\".format(url, e),\n )\n raise\n\n except ValueError as e:\n self.service_check(SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=str(e))\n raise\n\n else:\n self.service_check(\n SERVICE_CHECK_NAME,\n AgentCheck.OK,\n tags=service_check_tags,\n message=\"Connection to {} was successful\".format(url),\n )\n\n return response_json\n\n finally:\n if old_keytab_path is not None:\n os.environ['KRB5_CLIENT_KTNAME'] = old_keytab_path", "response": "Helper function to make a REST request to the given URL and return the JSON response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\njoin a URL with multiple directories", "response": "def _join_url_dir(self, url, *args):\n \"\"\"\n Join a URL with multiple directories\n \"\"\"\n for path in args:\n url = url.rstrip('/') + '/'\n url = urljoin(url, path.lstrip('/'))\n\n return url"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_url_base(self, url):\n s = urlsplit(url)\n return urlunsplit([s.scheme, s.netloc, '', '', ''])", "response": "Return the base of a URL"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_service_port(self, instance):\n host = instance.get('host', DEFAULT_HOST)\n port = instance.get('port', DEFAULT_PORT)\n # default port is the name of the service but lookup would fail\n # if the /etc/services file is missing. In that case, fallback to numeric\n try:\n socket.getaddrinfo(host, port)\n except socket.gaierror:\n port = DEFAULT_PORT_NUM\n\n return port", "response": "Get the service port from the instance dict."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _create_cadvisor_prometheus_instance(self, instance):\n cadvisor_instance = deepcopy(instance)\n cadvisor_instance.update(\n {\n 'namespace': self.NAMESPACE,\n # We need to specify a prometheus_url so the base class can use it as the key for our config_map,\n # we specify a dummy url that will be replaced in the `check()` function. We append it with \"cadvisor\"\n # so the key is different than the kubelet scraper.\n 'prometheus_url': instance.get('cadvisor_metrics_endpoint', 'dummy_url/cadvisor'),\n 'ignore_metrics': [\n 'container_cpu_cfs_periods_total',\n 'container_fs_inodes_free',\n 'container_fs_inodes_total',\n 'container_fs_io_current',\n 'container_fs_io_time_seconds_total',\n 'container_fs_io_time_weighted_seconds_total',\n 'container_fs_read_seconds_total',\n 'container_fs_reads_merged_total',\n 'container_fs_reads_total',\n 'container_fs_sector_reads_total',\n 'container_fs_sector_writes_total',\n 'container_fs_write_seconds_total',\n 'container_fs_writes_merged_total',\n 'container_fs_writes_total',\n 'container_last_seen',\n 'container_start_time_seconds',\n 'container_spec_memory_swap_limit_bytes',\n 'container_scrape_error',\n ],\n # Defaults that were set when CadvisorPrometheusScraper was based on PrometheusScraper\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n return cadvisor_instance", "response": "Create a copy of the instance and set default values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _is_container_metric(labels):\n for lbl in CONTAINER_LABELS:\n if lbl == 'container_name':\n if lbl in labels:\n if labels[lbl] == '' or labels[lbl] == 'POD':\n return False\n if lbl not in labels:\n return False\n return True", "response": "Return whether a metric is about a container or not."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn whether a metric is about a pod or not.", "response": "def _is_pod_metric(labels):\n \"\"\"\n Return whether a metric is about a pod or not.\n It can be about containers, pods, or higher levels in the cgroup hierarchy\n and we don't want to report on that.\n :param metric\n :return bool\n \"\"\"\n if 'container_name' in labels:\n if labels['container_name'] == 'POD':\n return True\n # containerd does not report container_name=\"POD\"\n elif labels['container_name'] == '' and labels.get('pod_name', False):\n return True\n # container_cpu_usage_seconds_total has an id label that is a cgroup path\n # eg: /kubepods/burstable/pod531c80d9-9fc4-11e7-ba8b-42010af002bb\n # FIXME: this was needed because of a bug:\n # https://github.com/kubernetes/kubernetes/pull/51473\n # starting from k8s 1.8 we can remove this\n if 'id' in labels:\n if labels['id'].split('/')[-1].startswith('pod'):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_container_id(self, labels):\n namespace = CadvisorPrometheusScraperMixin._get_container_label(labels, \"namespace\")\n pod_name = CadvisorPrometheusScraperMixin._get_container_label(labels, \"pod_name\")\n container_name = CadvisorPrometheusScraperMixin._get_container_label(labels, \"container_name\")\n return self.pod_list_utils.get_cid_by_name_tuple((namespace, pod_name, container_name))", "response": "This method gets the container id from the podlist using the metrics labels"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the labels indicate a container metric and extract the container id from them.", "response": "def _get_entity_id_if_container_metric(self, labels):\n \"\"\"\n Checks the labels indicate a container metric,\n then extract the container id from them.\n\n :param labels\n :return str or None\n \"\"\"\n if CadvisorPrometheusScraperMixin._is_container_metric(labels):\n pod = self._get_pod_by_metric_label(labels)\n if is_static_pending_pod(pod):\n # If the pod is static, ContainerStatus is unavailable.\n # Return the pod UID so that we can collect metrics from it later on.\n return self._get_pod_uid(labels)\n return self._get_container_id(labels)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_pod_uid(self, labels):\n namespace = CadvisorPrometheusScraperMixin._get_container_label(labels, \"namespace\")\n pod_name = CadvisorPrometheusScraperMixin._get_container_label(labels, \"pod_name\")\n return self.pod_list_utils.get_uid_by_name_tuple((namespace, pod_name))", "response": "Get the pod uid from the pod_name and namespace"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if the pod is on host Network Return False if the pod is not in the pod list", "response": "def _is_pod_host_networked(self, pod_uid):\n \"\"\"\n Return if the pod is on host Network\n Return False if the Pod isn't in the pod list\n :param pod_uid: str\n :return: bool\n \"\"\"\n for pod in self.pod_list['items']:\n if pod.get('metadata', {}).get('uid', '') == pod_uid:\n return pod.get('spec', {}).get('hostNetwork', False)\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a pod by metric label", "response": "def _get_pod_by_metric_label(self, labels):\n \"\"\"\n :param labels: metric labels: iterable\n :return:\n \"\"\"\n pod_uid = self._get_pod_uid(labels)\n return get_pod_by_uid(pod_uid, self.pod_list)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _sum_values_by_context(metric, uid_from_labels):\n seen = {}\n for sample in metric.samples:\n uid = uid_from_labels(sample[OpenMetricsBaseCheck.SAMPLE_LABELS])\n if not uid:\n # TODO\n # metric.Clear() # Ignore this metric message\n continue\n # Sum the counter value accross all contexts\n if uid not in seen:\n seen[uid] = sample\n else:\n # We have to create a new tuple\n seen[uid] = (\n seen[uid][OpenMetricsBaseCheck.SAMPLE_NAME],\n seen[uid][OpenMetricsBaseCheck.SAMPLE_LABELS],\n seen[uid][OpenMetricsBaseCheck.SAMPLE_VALUE] + sample[OpenMetricsBaseCheck.SAMPLE_VALUE],\n )\n # TODO\n # metric.Clear() # Ignore this metric message\n return seen", "response": "Sums the values of all the contexts in a metric."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _process_container_metric(self, type, metric_name, metric, scraper_config):\n if metric.type not in METRIC_TYPES:\n self.log.error(\"Metric type %s unsupported for metric %s\" % (metric.type, metric.name))\n return\n\n samples = self._sum_values_by_context(metric, self._get_entity_id_if_container_metric)\n for c_id, sample in iteritems(samples):\n pod_uid = self._get_pod_uid(sample[self.SAMPLE_LABELS])\n if self.pod_list_utils.is_excluded(c_id, pod_uid):\n continue\n\n tags = tagger.tag(c_id, tagger.HIGH)\n tags += scraper_config['custom_tags']\n\n # FIXME we are forced to do that because the Kubelet PodList isn't updated\n # for static pods, see https://github.com/kubernetes/kubernetes/pull/59948\n pod = self._get_pod_by_metric_label(sample[self.SAMPLE_LABELS])\n if pod is not None and is_static_pending_pod(pod):\n tags += tagger.tag('kubernetes_pod://%s' % pod[\"metadata\"][\"uid\"], tagger.HIGH)\n tags += self._get_kube_container_name(sample[self.SAMPLE_LABELS])\n tags = list(set(tags))\n\n val = sample[self.SAMPLE_VALUE]\n\n if \"rate\" == type:\n self.rate(metric_name, val, tags)\n elif \"gauge\" == type:\n self.gauge(metric_name, val, tags)", "response": "Processes a metric about a container."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprocesses a metric about a pod and reports it as a rate.", "response": "def _process_pod_rate(self, metric_name, metric, scraper_config):\n \"\"\"\n Takes a simple metric about a pod, reports it as a rate.\n If several series are found for a given pod, values are summed before submission.\n \"\"\"\n if metric.type not in METRIC_TYPES:\n self.log.error(\"Metric type %s unsupported for metric %s\" % (metric.type, metric.name))\n return\n\n samples = self._sum_values_by_context(metric, self._get_pod_uid_if_pod_metric)\n for pod_uid, sample in iteritems(samples):\n if '.network.' in metric_name and self._is_pod_host_networked(pod_uid):\n continue\n tags = tagger.tag('kubernetes_pod://%s' % pod_uid, tagger.HIGH)\n tags += scraper_config['custom_tags']\n val = sample[self.SAMPLE_VALUE]\n self.rate(metric_name, val, tags)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _process_usage_metric(self, m_name, metric, cache, scraper_config):\n # track containers that still exist in the cache\n seen_keys = {k: False for k in cache}\n\n samples = self._sum_values_by_context(metric, self._get_entity_id_if_container_metric)\n for c_id, sample in iteritems(samples):\n c_name = self._get_container_label(sample[self.SAMPLE_LABELS], 'name')\n if not c_name:\n continue\n pod_uid = self._get_pod_uid(sample[self.SAMPLE_LABELS])\n if self.pod_list_utils.is_excluded(c_id, pod_uid):\n continue\n\n tags = tagger.tag(c_id, tagger.HIGH)\n tags += scraper_config['custom_tags']\n\n # FIXME we are forced to do that because the Kubelet PodList isn't updated\n # for static pods, see https://github.com/kubernetes/kubernetes/pull/59948\n pod = self._get_pod_by_metric_label(sample[self.SAMPLE_LABELS])\n if pod is not None and is_static_pending_pod(pod):\n tags += tagger.tag('kubernetes_pod://%s' % pod[\"metadata\"][\"uid\"], tagger.HIGH)\n tags += self._get_kube_container_name(sample[self.SAMPLE_LABELS])\n tags = list(set(tags))\n\n val = sample[self.SAMPLE_VALUE]\n cache[c_name] = (val, tags)\n seen_keys[c_name] = True\n self.gauge(m_name, val, tags)\n\n # purge the cache\n for k, seen in iteritems(seen_keys):\n if not seen:\n del cache[k]", "response": "Processes a usage metric."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess a limit metric.", "response": "def _process_limit_metric(self, m_name, metric, cache, scraper_config, pct_m_name=None):\n \"\"\"\n Reports limit metrics if m_name is not an empty string,\n and optionally checks in the given cache if there's a usage\n for each sample in the metric and reports the usage_pct\n \"\"\"\n samples = self._sum_values_by_context(metric, self._get_entity_id_if_container_metric)\n for c_id, sample in iteritems(samples):\n limit = sample[self.SAMPLE_VALUE]\n pod_uid = self._get_pod_uid(sample[self.SAMPLE_LABELS])\n if self.pod_list_utils.is_excluded(c_id, pod_uid):\n continue\n\n tags = tagger.tag(c_id, tagger.HIGH)\n tags += scraper_config['custom_tags']\n\n if m_name:\n self.gauge(m_name, limit, tags)\n\n if pct_m_name and limit > 0:\n c_name = self._get_container_label(sample[self.SAMPLE_LABELS], 'name')\n if not c_name:\n continue\n usage, tags = cache.get(c_name, (None, None))\n if usage:\n self.gauge(pct_m_name, float(usage / float(limit)), tags)\n else:\n self.log.debug(\n \"No corresponding usage found for metric %s and \"\n \"container %s, skipping usage_pct for now.\" % (pct_m_name, c_name)\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nnumbering of bytes that are consumed by the container on this filesystem.", "response": "def container_fs_usage_bytes(self, metric, scraper_config):\n \"\"\"\n Number of bytes that are consumed by the container on this filesystem.\n \"\"\"\n metric_name = scraper_config['namespace'] + '.filesystem.usage'\n if metric.type not in METRIC_TYPES:\n self.log.error(\"Metric type %s unsupported for metric %s\" % (metric.type, metric.name))\n return\n self._process_usage_metric(metric_name, metric, self.fs_usage_bytes, scraper_config)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef container_fs_limit_bytes(self, metric, scraper_config):\n pct_m_name = scraper_config['namespace'] + '.filesystem.usage_pct'\n if metric.type not in METRIC_TYPES:\n self.log.error(\"Metric type %s unsupported for metric %s\" % (metric.type, metric.name))\n return\n self._process_limit_metric('', metric, self.fs_usage_bytes, scraper_config, pct_m_name)", "response": "This method is used by container_fs_usage_bytes to report the number of bytes that can be consumed by the container on this filesystem."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _report_replica_set_state(self, state, clean_server_name, replset_name):\n last_state = self._last_state_by_server.get(clean_server_name, -1)\n self._last_state_by_server[clean_server_name] = state\n if last_state != state and last_state != -1:\n return self.create_event(last_state, state, clean_server_name, replset_name)", "response": "Report the state of a replica set."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef hostname_for_event(self, clean_server_name):\n uri = urlsplit(clean_server_name)\n if '@' in uri.netloc:\n hostname = uri.netloc.split('@')[1].split(':')[0]\n else:\n hostname = uri.netloc.split(':')[0]\n if hostname == 'localhost':\n hostname = self.hostname\n return hostname", "response": "Return a reasonable hostname for a replset membership event to mention."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_event(self, last_state, state, clean_server_name, replset_name):\n\n status = self.get_state_description(state)\n short_status = self.get_state_name(state)\n last_short_status = self.get_state_name(last_state)\n hostname = self.hostname_for_event(clean_server_name)\n msg_title = \"%s is %s for %s\" % (hostname, short_status, replset_name)\n msg = \"MongoDB %s (%s) just reported as %s (%s) for %s; it was %s before.\"\n msg = msg % (hostname, clean_server_name, status, short_status, replset_name, last_short_status)\n\n self.event(\n {\n 'timestamp': int(time.time()),\n 'source_type_name': self.SOURCE_TYPE_NAME,\n 'msg_title': msg_title,\n 'msg_text': msg,\n 'host': hostname,\n 'tags': [\n 'action:mongo_replset_member_status_change',\n 'member_status:' + short_status,\n 'previous_member_status:' + last_short_status,\n 'replset:' + replset_name,\n ],\n }\n )", "response": "Create an event describing the replication\n state of a mongo node."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _build_metric_list_to_collect(self, additional_metrics):\n metrics_to_collect = {}\n\n # Defaut metrics\n for default_metrics in itervalues(self.DEFAULT_METRICS):\n metrics_to_collect.update(default_metrics)\n\n # Additional metrics metrics\n for option in additional_metrics:\n additional_metrics = self.AVAILABLE_METRICS.get(option)\n if not additional_metrics:\n if option in self.DEFAULT_METRICS:\n self.log.warning(\n u\"`%s` option is deprecated. The corresponding metrics are collected by default.\", option\n )\n else:\n self.log.warning(\n u\"Failed to extend the list of metrics to collect: unrecognized `%s` option\", option\n )\n continue\n\n self.log.debug(u\"Adding `%s` corresponding metrics to the list of metrics to collect.\", option)\n metrics_to_collect.update(additional_metrics)\n\n return metrics_to_collect", "response": "Build the metric list to collect based on the instance preferences."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_metrics_to_collect(self, instance_key, additional_metrics):\n if instance_key not in self.metrics_to_collect_by_instance:\n self.metrics_to_collect_by_instance[instance_key] = self._build_metric_list_to_collect(additional_metrics)\n return self.metrics_to_collect_by_instance[instance_key]", "response": "Return the list of metrics to collect."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _resolve_metric(self, original_metric_name, metrics_to_collect, prefix=\"\"):\n\n submit_method = (\n metrics_to_collect[original_metric_name][0]\n if isinstance(metrics_to_collect[original_metric_name], tuple)\n else metrics_to_collect[original_metric_name]\n )\n metric_name = (\n metrics_to_collect[original_metric_name][1]\n if isinstance(metrics_to_collect[original_metric_name], tuple)\n else original_metric_name\n )\n\n return submit_method, self._normalize(metric_name, submit_method, prefix)", "response": "Resolve the metric name to use."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _normalize(self, metric_name, submit_method, prefix):\n metric_prefix = \"mongodb.\" if not prefix else \"mongodb.{0}.\".format(prefix)\n metric_suffix = \"ps\" if submit_method == RATE else \"\"\n\n # Replace case-sensitive metric name characters\n for pattern, repl in iteritems(self.CASE_SENSITIVE_METRIC_NAME_SUFFIXES):\n metric_name = re.compile(pattern).sub(repl, metric_name)\n\n # Normalize, and wrap\n return u\"{metric_prefix}{normalized_metric_name}{metric_suffix}\".format(\n normalized_metric_name=self.normalize(metric_name.lower()),\n metric_prefix=metric_prefix,\n metric_suffix=metric_suffix,\n )", "response": "Normalize the metric name and wrap it in a new one."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _authenticate(self, database, username, password, use_x509, server_name, service_check_tags):\n authenticated = False\n try:\n # X.509\n if use_x509:\n self.log.debug(u\"Authenticate `%s` to `%s` using `MONGODB-X509` mechanism\", username, database)\n authenticated = database.authenticate(username, mechanism='MONGODB-X509')\n\n # Username & password\n else:\n authenticated = database.authenticate(username, password)\n\n except pymongo.errors.PyMongoError as e:\n self.log.error(u\"Authentication failed due to invalid credentials or configuration issues. %s\", e)\n\n if not authenticated:\n message = \"Mongo: cannot connect with config %s\" % server_name\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=message)\n raise Exception(message)\n\n return authenticated", "response": "Authenticate to the database."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses a MongoDB - formatted URI and returns parsed elements and a sanitized URI.", "response": "def _parse_uri(cls, server, sanitize_username=False):\n \"\"\"\n Parses a MongoDB-formatted URI (e.g. mongodb://user:pass@server/db) and returns parsed elements\n and a sanitized URI.\n \"\"\"\n parsed = pymongo.uri_parser.parse_uri(server)\n\n username = parsed.get('username')\n password = parsed.get('password')\n db_name = parsed.get('database')\n nodelist = parsed.get('nodelist')\n auth_source = parsed.get('options', {}).get('authsource')\n\n # Remove password (and optionally username) from sanitized server URI.\n # To ensure that the `replace` works well, we first need to url-decode the raw server string\n # since the password parsed by pymongo is url-decoded\n decoded_server = unquote_plus(server)\n clean_server_name = decoded_server.replace(password, \"*\" * 5) if password else decoded_server\n\n if sanitize_username and username:\n username_pattern = u\"{}[@:]\".format(re.escape(username))\n clean_server_name = re.sub(username_pattern, \"\", clean_server_name)\n\n return username, password, db_name, nodelist, clean_server_name, auth_source"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _collect_indexes_stats(self, instance, db, tags):\n for coll_name in instance.get('collections', []):\n try:\n for stats in db[coll_name].aggregate([{\"$indexStats\": {}}], cursor={}):\n idx_tags = tags + [\n \"name:{0}\".format(stats.get('name', 'unknown')),\n \"collection:{0}\".format(coll_name),\n ]\n val = int(stats.get('accesses', {}).get('ops', 0))\n self.gauge('mongodb.collection.indexes.accesses.ops', val, idx_tags)\n except Exception as e:\n self.log.error(\"Could not fetch indexes stats for collection %s: %s\", coll_name, e)", "response": "Collect indexes stats for all collections in the configuration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check(self, instance):\n\n def total_seconds(td):\n \"\"\"\n Returns total seconds of a timedelta in a way that's safe for\n Python < 2.7\n \"\"\"\n if hasattr(td, 'total_seconds'):\n return td.total_seconds()\n else:\n return (lag.microseconds + (lag.seconds + lag.days * 24 * 3600) * 10 ** 6) / 10.0 ** 6\n\n if 'server' not in instance:\n raise Exception(\"Missing 'server' in mongo config\")\n\n # x.509 authentication\n ssl_params = {\n 'ssl': instance.get('ssl', None),\n 'ssl_keyfile': instance.get('ssl_keyfile', None),\n 'ssl_certfile': instance.get('ssl_certfile', None),\n 'ssl_cert_reqs': instance.get('ssl_cert_reqs', None),\n 'ssl_ca_certs': instance.get('ssl_ca_certs', None),\n }\n\n for key, param in list(iteritems(ssl_params)):\n if param is None:\n del ssl_params[key]\n\n server = instance['server']\n username, password, db_name, nodelist, clean_server_name, auth_source = self._parse_uri(\n server, sanitize_username=bool(ssl_params)\n )\n\n additional_metrics = instance.get('additional_metrics', [])\n\n # Get the list of metrics to collect\n collect_tcmalloc_metrics = 'tcmalloc' in additional_metrics\n metrics_to_collect = self._get_metrics_to_collect(server, additional_metrics)\n\n # Tagging\n tags = instance.get('tags', [])\n # ...de-dupe tags to avoid a memory leak\n tags = list(set(tags))\n\n if not db_name:\n self.log.info('No MongoDB database found in URI. Defaulting to admin.')\n db_name = 'admin'\n\n service_check_tags = [\"db:%s\" % db_name]\n service_check_tags.extend(tags)\n\n # ...add the `server` tag to the metrics' tags only\n # (it's added in the backend for service checks)\n tags.append('server:%s' % clean_server_name)\n\n if nodelist:\n host = nodelist[0][0]\n port = nodelist[0][1]\n service_check_tags = service_check_tags + [\"host:%s\" % host, \"port:%s\" % port]\n\n timeout = float(instance.get('timeout', DEFAULT_TIMEOUT)) * 1000\n try:\n cli = pymongo.mongo_client.MongoClient(\n server,\n socketTimeoutMS=timeout,\n connectTimeoutMS=timeout,\n serverSelectionTimeoutMS=timeout,\n read_preference=pymongo.ReadPreference.PRIMARY_PREFERRED,\n **ssl_params\n )\n # some commands can only go against the admin DB\n admindb = cli['admin']\n db = cli[db_name]\n except Exception:\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags)\n raise\n\n # Authenticate\n do_auth = True\n use_x509 = ssl_params and not password\n\n if not username:\n self.log.debug(u\"A username is required to authenticate to `%s`\", server)\n do_auth = False\n\n if do_auth:\n if auth_source:\n msg = \"authSource was specified in the the server URL: using '%s' as the authentication database\"\n self.log.info(msg, auth_source)\n self._authenticate(\n cli[auth_source], username, password, use_x509, clean_server_name, service_check_tags\n )\n else:\n self._authenticate(db, username, password, use_x509, clean_server_name, service_check_tags)\n\n try:\n status = db.command('serverStatus', tcmalloc=collect_tcmalloc_metrics)\n except Exception:\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags)\n raise\n else:\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK, tags=service_check_tags)\n\n if status['ok'] == 0:\n raise Exception(status['errmsg'].__str__())\n\n ops = db.current_op()\n status['fsyncLocked'] = 1 if ops.get('fsyncLock') else 0\n\n status['stats'] = db.command('dbstats')\n dbstats = {db_name: {'stats': status['stats']}}\n\n # Handle replica data, if any\n # See\n # http://www.mongodb.org/display/DOCS/Replica+Set+Commands#ReplicaSetCommands-replSetGetStatus # noqa\n if is_affirmative(instance.get('replica_check', True)):\n try:\n data = {}\n\n replSet = admindb.command('replSetGetStatus')\n if replSet:\n primary = None\n current = None\n\n # need a new connection to deal with replica sets\n setname = replSet.get('set')\n cli_rs = pymongo.mongo_client.MongoClient(\n server,\n socketTimeoutMS=timeout,\n connectTimeoutMS=timeout,\n serverSelectionTimeoutMS=timeout,\n replicaset=setname,\n read_preference=pymongo.ReadPreference.NEAREST,\n **ssl_params\n )\n\n if do_auth:\n if auth_source:\n self._authenticate(\n cli_rs[auth_source], username, password, use_x509, server, service_check_tags\n )\n else:\n self._authenticate(\n cli_rs[db_name], username, password, use_x509, server, service_check_tags\n )\n\n # Replication set information\n replset_name = replSet['set']\n replset_state = self.get_state_name(replSet['myState']).lower()\n\n tags.extend([u\"replset_name:{0}\".format(replset_name), u\"replset_state:{0}\".format(replset_state)])\n\n # Find nodes: master and current node (ourself)\n for member in replSet.get('members'):\n if member.get('self'):\n current = member\n if int(member.get('state')) == 1:\n primary = member\n\n # Compute a lag time\n if current is not None and primary is not None:\n if 'optimeDate' in primary and 'optimeDate' in current:\n lag = primary['optimeDate'] - current['optimeDate']\n data['replicationLag'] = total_seconds(lag)\n\n if current is not None:\n data['health'] = current['health']\n\n data['state'] = replSet['myState']\n\n if current is not None:\n total = 0.0\n cfg = cli_rs['local']['system.replset'].find_one()\n for member in cfg.get('members'):\n total += member.get('votes', 1)\n if member['_id'] == current['_id']:\n data['votes'] = member.get('votes', 1)\n data['voteFraction'] = data['votes'] / total\n\n status['replSet'] = data\n\n # Submit events\n self._report_replica_set_state(data['state'], clean_server_name, replset_name)\n\n except Exception as e:\n if \"OperationFailure\" in repr(e) and (\n \"not running with --replSet\" in str(e) or \"replSetGetStatus\" in str(e)\n ):\n pass\n else:\n raise e\n\n # If these keys exist, remove them for now as they cannot be serialized\n try:\n status['backgroundFlushing'].pop('last_finished')\n except KeyError:\n pass\n try:\n status.pop('localTime')\n except KeyError:\n pass\n\n dbnames = cli.database_names()\n self.gauge('mongodb.dbs', len(dbnames), tags=tags)\n\n for db_n in dbnames:\n db_aux = cli[db_n]\n dbstats[db_n] = {'stats': db_aux.command('dbstats')}\n\n # Go through the metrics and save the values\n for metric_name in metrics_to_collect:\n # each metric is of the form: x.y.z with z optional\n # and can be found at status[x][y][z]\n value = status\n\n if metric_name.startswith('stats'):\n continue\n else:\n try:\n for c in metric_name.split(\".\"):\n value = value[c]\n except KeyError:\n continue\n\n # value is now status[x][y][z]\n if not isinstance(value, (int, long, float)):\n raise TypeError(\n u\"{0} value is a {1}, it should be an int, a float or a long instead.\".format(\n metric_name, type(value)\n )\n )\n\n # Submit the metric\n submit_method, metric_name_alias = self._resolve_metric(metric_name, metrics_to_collect)\n submit_method(self, metric_name_alias, value, tags=tags)\n\n for st, value in iteritems(dbstats):\n for metric_name in metrics_to_collect:\n if not metric_name.startswith('stats.'):\n continue\n\n try:\n val = value['stats'][metric_name.split('.')[1]]\n except KeyError:\n continue\n\n # value is now status[x][y][z]\n if not isinstance(val, (int, long, float)):\n raise TypeError(\n u\"{0} value is a {1}, it should be an int, a float or a long instead.\".format(\n metric_name, type(val)\n )\n )\n\n # Submit the metric\n metrics_tags = tags + [\n u\"cluster:db:{0}\".format(st), # FIXME 6.0 - keep for backward compatibility\n u\"db:{0}\".format(st),\n ]\n\n submit_method, metric_name_alias = self._resolve_metric(metric_name, metrics_to_collect)\n submit_method(self, metric_name_alias, val, tags=metrics_tags)\n\n if is_affirmative(instance.get('collections_indexes_stats')):\n mongo_version = cli.server_info().get('version', '0.0')\n if LooseVersion(mongo_version) >= LooseVersion(\"3.2\"):\n self._collect_indexes_stats(instance, db, tags)\n else:\n msg = \"'collections_indexes_stats' is only available starting from mongo 3.2: your mongo version is %s\"\n self.log.error(msg, mongo_version)\n\n # Report the usage metrics for dbs/collections\n if 'top' in additional_metrics:\n try:\n dbtop = admindb.command('top')\n for ns, ns_metrics in iteritems(dbtop['totals']):\n if \".\" not in ns:\n continue\n\n # configure tags for db name and collection name\n dbname, collname = ns.split(\".\", 1)\n ns_tags = tags + [\"db:%s\" % dbname, \"collection:%s\" % collname]\n\n # iterate over DBTOP metrics\n for m in self.TOP_METRICS:\n # each metric is of the form: x.y.z with z optional\n # and can be found at ns_metrics[x][y][z]\n value = ns_metrics\n try:\n for c in m.split(\".\"):\n value = value[c]\n except Exception:\n continue\n\n # value is now status[x][y][z]\n if not isinstance(value, (int, long, float)):\n raise TypeError(\n u\"{0} value is a {1}, it should be an int, a float or a long instead.\".format(\n m, type(value)\n )\n )\n\n # Submit the metric\n submit_method, metric_name_alias = self._resolve_metric(m, metrics_to_collect, prefix=\"usage\")\n submit_method(self, metric_name_alias, value, tags=ns_tags)\n # Keep old incorrect metric\n if metric_name_alias.endswith('countps'):\n GAUGE(self, metric_name_alias[:-2], value, tags=ns_tags)\n except Exception as e:\n self.log.warning('Failed to record `top` metrics %s' % str(e))\n\n if 'local' in dbnames: # it might not be if we are connectiing through mongos\n # Fetch information analogous to Mongo's db.getReplicationInfo()\n localdb = cli['local']\n\n oplog_data = {}\n\n for ol_collection_name in (\"oplog.rs\", \"oplog.$main\"):\n ol_options = localdb[ol_collection_name].options()\n if ol_options:\n break\n\n if ol_options:\n try:\n oplog_data['logSizeMB'] = round_value(ol_options['size'] / 2.0 ** 20, 2)\n\n oplog = localdb[ol_collection_name]\n\n oplog_data['usedSizeMB'] = round_value(\n localdb.command(\"collstats\", ol_collection_name)['size'] / 2.0 ** 20, 2\n )\n\n op_asc_cursor = oplog.find({\"ts\": {\"$exists\": 1}}).sort(\"$natural\", pymongo.ASCENDING).limit(1)\n op_dsc_cursor = oplog.find({\"ts\": {\"$exists\": 1}}).sort(\"$natural\", pymongo.DESCENDING).limit(1)\n\n try:\n first_timestamp = op_asc_cursor[0]['ts'].as_datetime()\n last_timestamp = op_dsc_cursor[0]['ts'].as_datetime()\n oplog_data['timeDiff'] = total_seconds(last_timestamp - first_timestamp)\n except (IndexError, KeyError):\n # if the oplog collection doesn't have any entries\n # if an object in the collection doesn't have a ts value, we ignore it\n pass\n except KeyError:\n # encountered an error trying to access options.size for the oplog collection\n self.log.warning(u\"Failed to record `ReplicationInfo` metrics.\")\n\n for m, value in iteritems(oplog_data):\n submit_method, metric_name_alias = self._resolve_metric('oplog.%s' % m, metrics_to_collect)\n submit_method(self, metric_name_alias, value, tags=tags)\n\n else:\n self.log.debug('\"local\" database not in dbnames. Not collecting ReplicationInfo metrics')\n\n # get collection level stats\n try:\n # Ensure that you're on the right db\n db = cli[db_name]\n # grab the collections from the configutation\n coll_names = instance.get('collections', [])\n # loop through the collections\n for coll_name in coll_names:\n # grab the stats from the collection\n stats = db.command(\"collstats\", coll_name)\n # loop through the metrics\n for m in self.collection_metrics_names:\n coll_tags = tags + [\"db:%s\" % db_name, \"collection:%s\" % coll_name]\n value = stats.get(m, None)\n if not value:\n continue\n\n # if it's the index sizes, then it's a dict.\n if m == 'indexSizes':\n submit_method, metric_name_alias = self._resolve_metric(\n 'collection.%s' % m, self.COLLECTION_METRICS\n )\n # loop through the indexes\n for idx, val in iteritems(value):\n # we tag the index\n idx_tags = coll_tags + [\"index:%s\" % idx]\n submit_method(self, metric_name_alias, val, tags=idx_tags)\n else:\n submit_method, metric_name_alias = self._resolve_metric(\n 'collection.%s' % m, self.COLLECTION_METRICS\n )\n submit_method(self, metric_name_alias, value, tags=coll_tags)\n except Exception as e:\n self.log.warning(u\"Failed to record `collection` metrics.\")\n self.log.exception(e)", "response": "Check the status of a specific instance of a specific version of the database."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_normal_connection(config):\n if config.username and config.password:\n log.debug(\"connecting with username and password\")\n queue_manager = pymqi.connect(\n config.queue_manager_name, config.channel, config.host_and_port, config.username, config.password\n )\n else:\n log.debug(\"connecting without a username and password\")\n queue_manager = pymqi.connect(config.queue_manager_name, config.channel, config.host_and_port)\n\n return queue_manager", "response": "Get the connection to the normal queue manager."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the connection with SSL", "response": "def get_ssl_connection(config):\n \"\"\"\n Get the connection with SSL\n \"\"\"\n cd = pymqi.CD()\n cd.ChannelName = config.channel\n cd.ConnectionName = config.host_and_port\n cd.ChannelType = pymqi.CMQC.MQCHT_CLNTCONN\n cd.TransportType = pymqi.CMQC.MQXPT_TCP\n cd.SSLCipherSpec = config.ssl_cipher_spec\n\n sco = pymqi.SCO()\n sco.KeyRepository = config.ssl_key_repository_location\n\n queue_manager = pymqi.QueueManager(None)\n queue_manager.connect_with_options(config.queue_manager_name, cd, sco)\n\n return queue_manager"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef check_run(check, env, rate, times, pause, delay, log_level, as_json, break_point):\n envs = get_configured_envs(check)\n if not envs:\n echo_failure('No active environments found for `{}`.'.format(check))\n echo_info('See what is available to start via `ddev env ls {}`.'.format(check))\n abort()\n\n if not env:\n if len(envs) > 1:\n echo_failure('Multiple active environments found for `{}`, please specify one.'.format(check))\n echo_info('See what is active via `ddev env ls`.')\n abort()\n\n env = envs[0]\n\n if env not in envs:\n echo_failure('`{}` is not an active environment.'.format(env))\n echo_info('See what is active via `ddev env ls`.')\n abort()\n\n environment = create_interface(check, env)\n\n environment.run_check(\n rate=rate, times=times, pause=pause, delay=delay, log_level=log_level, as_json=as_json, break_point=break_point\n )\n echo_success('Note: ', nl=False)\n echo_info('If some metrics are missing, you may want to try again with the -r / --rate flag.')", "response": "Run an Agent check."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse user identity out of init_config", "response": "def get_user_identity(cls, instance_config):\n \"\"\"\n Parse user identity out of init_config\n\n To guarantee a uniquely identifiable user, expects\n {\"user\": {\"name\": \"my_username\", \"password\": \"my_password\",\n \"domain\": {\"id\": \"my_domain_id\"}\n }\n }\n \"\"\"\n user = instance_config.get('user')\n\n if not (\n user and user.get('name') and user.get('password') and user.get(\"domain\") and user.get(\"domain\").get(\"id\")\n ):\n raise IncompleteIdentity()\n\n identity = {\"methods\": ['password'], \"password\": {\"user\": user}}\n return identity"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_auth_scope(cls, instance_config):\n auth_scope = instance_config.get('auth_scope')\n if not auth_scope:\n return None\n\n if not auth_scope.get('project'):\n raise IncompleteAuthScope()\n\n if auth_scope['project'].get('name'):\n # We need to add a domain scope to avoid name clashes. Search for one. If not raise IncompleteAuthScope\n if not auth_scope['project'].get('domain', {}).get('id'):\n raise IncompleteAuthScope()\n else:\n # Assume a unique project id has been given\n if not auth_scope['project'].get('id'):\n raise IncompleteAuthScope()\n\n return auth_scope", "response": "Parse authorization scope out of init_config and return it"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing the JSON response from the Neutron Identity API and return the endpoint that is available for the neutron service", "response": "def get_neutron_endpoint(cls, json_resp):\n \"\"\"\n Parse the service catalog returned by the Identity API for an endpoint matching the Neutron service\n Sends a CRITICAL service check when none are found registered in the Catalog\n \"\"\"\n catalog = json_resp.get('token', {}).get('catalog', [])\n match = 'neutron'\n\n neutron_endpoint = None\n for entry in catalog:\n if entry['name'] == match or 'Networking' in entry['name']:\n valid_endpoints = {}\n for ep in entry['endpoints']:\n interface = ep.get('interface', '')\n if interface in ['public', 'internal']:\n valid_endpoints[interface] = ep['url']\n\n if valid_endpoints:\n # Favor public endpoints over internal\n neutron_endpoint = valid_endpoints.get(\"public\", valid_endpoints.get(\"internal\"))\n break\n else:\n raise MissingNeutronEndpoint()\n\n return neutron_endpoint"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the Nova service catalog and return the endpoint that is available for the requested version", "response": "def get_nova_endpoint(cls, json_resp, nova_api_version=None):\n \"\"\"\n Parse the service catalog returned by the Identity API for an endpoint matching\n the Nova service with the requested version\n Sends a CRITICAL service check when no viable candidates are found in the Catalog\n \"\"\"\n nova_version = nova_api_version or DEFAULT_NOVA_API_VERSION\n catalog = json_resp.get('token', {}).get('catalog', [])\n\n nova_match = 'novav21' if nova_version == V21_NOVA_API_VERSION else 'nova'\n\n for entry in catalog:\n if entry['name'] == nova_match or 'Compute' in entry['name']:\n # Collect any endpoints on the public or internal interface\n valid_endpoints = {}\n for ep in entry['endpoints']:\n interface = ep.get('interface', '')\n if interface in ['public', 'internal']:\n valid_endpoints[interface] = ep['url']\n\n if valid_endpoints:\n # Favor public endpoints over internal\n nova_endpoint = valid_endpoints.get(\"public\", valid_endpoints.get(\"internal\"))\n return nova_endpoint\n else:\n raise MissingNovaEndpoint()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _make_request_with_auth_fallback(self, url, headers=None, params=None):\n self.log.debug(\"Request URL and Params: %s, %s\", url, params)\n try:\n resp = requests.get(\n url,\n headers=headers,\n verify=self._ssl_verify,\n params=params,\n timeout=DEFAULT_API_REQUEST_TIMEOUT,\n proxies=self.proxy_config,\n )\n resp.raise_for_status()\n except requests.exceptions.HTTPError as e:\n self.log.debug(\"Error contacting openstack endpoint: %s\", e)\n if resp.status_code == 401:\n self.log.info('Need to reauthenticate before next check')\n\n # Delete the scope, we'll populate a new one on the next run for this instance\n self.delete_current_scope()\n elif resp.status_code == 409:\n raise InstancePowerOffFailure()\n elif resp.status_code == 404:\n raise e\n else:\n raise\n\n return resp.json()", "response": "This function is a generic request handler for OpenStack API requests Raises specialized Exceptions for commonly encountered error codes"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_network_stats(self, tags):\n\n # FIXME: (aaditya) Check all networks defaults to true\n # until we can reliably assign agents to networks to monitor\n if is_affirmative(self.init_config.get('check_all_networks', True)):\n all_network_ids = set(self.get_all_network_ids())\n\n # Filter out excluded networks\n network_ids = [\n network_id\n for network_id in all_network_ids\n if not any([re.match(exclude_id, network_id) for exclude_id in self.exclude_network_id_rules])\n ]\n else:\n network_ids = self.init_config.get('network_ids', [])\n\n if not network_ids:\n self.warning(\n \"Your check is not configured to monitor any networks.\\n\"\n + \"Please list `network_ids` under your init_config\"\n )\n\n for nid in network_ids:\n self.get_stats_for_single_network(nid, tags)", "response": "Collect stats for all reachable networks and add agents to the monitor."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the uptime string into a dictionary of load average and uptime_sec.", "response": "def _parse_uptime_string(self, uptime):\n \"\"\" Parse u' 16:53:48 up 1 day, 21:34, 3 users, load average: 0.04, 0.14, 0.19\\n' \"\"\"\n uptime = uptime.strip()\n load_averages = uptime[uptime.find('load average:') :].split(':')[1].split(',')\n uptime_sec = uptime.split(',')[0]\n\n return {'loads': list(map(float, load_averages)), 'uptime_sec': uptime_sec}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ensure_auth_scope(self, instance):\n instance_scope = None\n\n custom_tags = instance.get('tags', [])\n if custom_tags is None:\n custom_tags = []\n try:\n instance_scope = self.get_scope_for_instance(instance)\n except KeyError:\n\n # We're missing a project scope for this instance\n # Let's populate it now\n try:\n if 'auth_scope' in instance:\n instance_scope = OpenStackProjectScope.from_config(self.init_config, instance, self.proxy_config)\n else:\n instance_scope = OpenStackUnscoped.from_config(self.init_config, instance, self.proxy_config)\n\n self.service_check(\n self.IDENTITY_API_SC,\n AgentCheck.OK,\n tags=[\"server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n except KeystoneUnreachable as e:\n self.warning(\n \"The agent could not contact the specified identity server at %s . \\\n Are you sure it is up at that address?\"\n % self.init_config.get(\"keystone_server_url\")\n )\n self.log.debug(\"Problem grabbing auth token: %s\", e)\n self.service_check(\n self.IDENTITY_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n\n # If Keystone is down/unreachable, we default the\n # Nova and Neutron APIs to UNKNOWN since we cannot access the service catalog\n self.service_check(\n self.NETWORK_API_SC,\n AgentCheck.UNKNOWN,\n tags=[\"keystone_server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n self.service_check(\n self.COMPUTE_API_SC,\n AgentCheck.UNKNOWN,\n tags=[\"keystone_server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n\n except MissingNovaEndpoint as e:\n self.warning(\"The agent could not find a compatible Nova endpoint in your service catalog!\")\n self.log.debug(\"Failed to get nova endpoint for response catalog: %s\", e)\n self.service_check(\n self.COMPUTE_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n\n except MissingNeutronEndpoint:\n self.warning(\"The agent could not find a compatible Neutron endpoint in your service catalog!\")\n self.service_check(\n self.NETWORK_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server:%s\" % self.init_config.get(\"keystone_server_url\")] + custom_tags,\n )\n else:\n self.set_scope_for_instance(instance, instance_scope)\n\n return instance_scope", "response": "Ensures that the auth token for this instance is valid and returns it"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the local hypervisor running on this host and assumes a 1 - 1 between host and hypervisor", "response": "def get_local_hypervisor(self):\n \"\"\"\n Returns the hypervisor running on this host, and assumes a 1-1 between host and hypervisor\n \"\"\"\n # Look up hypervisors available filtered by my hostname\n host = self.get_my_hostname()\n hyp = self.get_all_hypervisor_ids(filter_by_host=host)\n if hyp:\n return hyp[0]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_all_projects(self, scope):\n url = \"{0}/{1}/{2}\".format(self.keystone_server_url, DEFAULT_KEYSTONE_API_VERSION, \"projects\")\n headers = {'X-Auth-Token': scope.auth_token}\n try:\n r = self._make_request_with_auth_fallback(url, headers)\n return r['projects']\n\n except Exception as e:\n self.warning('Unable to get projects: {0}'.format(str(e)))\n raise e\n\n return None", "response": "Returns all projects in the domain"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the project that this check is scoped to", "response": "def get_scoped_project(self, project_auth_scope):\n \"\"\"\n Returns the project that this instance of the check is scoped to\n \"\"\"\n\n filter_params = {}\n url = \"{0}/{1}/{2}\".format(self.keystone_server_url, DEFAULT_KEYSTONE_API_VERSION, \"projects\")\n if project_auth_scope.tenant_id:\n if project_auth_scope.project_name:\n return {\"id\": project_auth_scope.tenant_id, \"name\": project_auth_scope.project_name}\n\n url = \"{}/{}\".format(url, project_auth_scope.tenant_id)\n else:\n filter_params = {\"name\": project_auth_scope.project_name, \"domain_id\": project_auth_scope.domain_id}\n\n headers = {'X-Auth-Token': project_auth_scope.auth_token}\n\n try:\n project_details = self._make_request_with_auth_fallback(url, headers, params=filter_params)\n if filter_params:\n assert len(project_details[\"projects\"]) == 1, \"Non-unique project credentials\"\n\n # Set the tenant_id so we won't have to fetch it next time\n project_auth_scope.tenant_id = project_details[\"projects\"][0].get(\"id\")\n return project_details[\"projects\"][0]\n else:\n project_auth_scope.project_name = project_details[\"project\"][\"name\"]\n return project_details[\"project\"]\n\n except Exception as e:\n self.warning('Unable to get the project details: {0}'.format(str(e)))\n raise e\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a best guess for the hostname registered with OpenStack for this host.", "response": "def get_my_hostname(self, split_hostname_on_first_period=False):\n \"\"\"\n Returns a best guess for the hostname registered with OpenStack for this host\n \"\"\"\n\n hostname = self.init_config.get(\"os_host\") or self.hostname\n if split_hostname_on_first_period:\n hostname = hostname.split('.')[0]\n\n return hostname"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_external_host_tags(self):\n self.log.debug(\"Collecting external_host_tags now\")\n external_host_tags = []\n for k, v in iteritems(self.external_host_tags):\n external_host_tags.append((k, {SOURCE_TYPE: v}))\n\n self.log.debug(\"Sending external_host_tags: %s\", external_host_tags)\n return external_host_tags", "response": "Returns a list of pairs of hostname and list_of_tags"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun the Docker check for one instance.", "response": "def check(self, instance):\n \"\"\"Run the Docker check for one instance.\"\"\"\n if not self.init_success:\n # Initialization can fail if cgroups are not ready or docker daemon is down. So we retry if needed\n # https://github.com/DataDog/dd-agent/issues/1896\n self.init()\n\n try:\n if self.docker_util.client is None:\n message = \"Unable to connect to Docker daemon\"\n self.service_check(SERVICE_CHECK_NAME, AgentCheck.CRITICAL,\n message=message, tags=self.custom_tags)\n return\n except Exception as ex:\n self.service_check(SERVICE_CHECK_NAME, AgentCheck.CRITICAL,\n message=str(ex), tags=self.custom_tags)\n return\n\n if not self.init_success:\n # Initialization failed, will try later\n return\n\n try:\n # Report image metrics\n if self.collect_image_stats:\n self._count_and_weigh_images()\n\n if Platform.is_k8s():\n self.kube_pod_tags = {}\n if self.kubeutil:\n try:\n self.kube_pod_tags = self.kubeutil.get_kube_pod_tags()\n except Exception as e:\n self.log.warning('Could not retrieve kubernetes labels: %s' % str(e))\n\n # containers running with custom cgroups?\n custom_cgroups = _is_affirmative(instance.get('custom_cgroups', False))\n\n # Get the list of containers and the index of their names\n health_service_checks = True if self.whitelist_patterns else False\n containers_by_id = self._get_and_count_containers(custom_cgroups, health_service_checks)\n containers_by_id = self._crawl_container_pids(containers_by_id, custom_cgroups)\n\n # Send events from Docker API\n if self.collect_events or self._service_discovery or not self._disable_net_metrics or self.collect_exit_codes:\n self._process_events(containers_by_id)\n\n # Report performance container metrics (cpu, mem, net, io)\n self._report_performance_metrics(containers_by_id)\n\n if self.collect_container_size:\n self._report_container_size(containers_by_id)\n\n if self.collect_container_count:\n self._report_container_count(containers_by_id)\n\n if self.collect_volume_count:\n self._report_volume_count()\n\n # Collect disk stats from Docker info command\n if self.collect_disk_stats:\n self._report_disk_stats()\n\n if health_service_checks:\n self._send_container_healthcheck_sc(containers_by_id)\n except:\n self.log.exception(\"Docker_daemon check failed\")\n self.warning(\"Check failed. Will retry at next iteration\")\n\n if self.capped_metrics:\n self.filter_capped_metrics()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_and_count_containers(self, custom_cgroups=False, healthchecks=False):\n\n # Querying the size of containers is slow, we don't do it at each run\n must_query_size = self.collect_container_size and self._latest_size_query == 0\n self._latest_size_query = (self._latest_size_query + 1) % SIZE_REFRESH_RATE\n\n running_containers_count = Counter()\n all_containers_count = Counter()\n\n try:\n containers = self.docker_util.client.containers(all=True, size=must_query_size)\n except Exception as e:\n message = \"Unable to list Docker containers: {0}\".format(e)\n self.service_check(SERVICE_CHECK_NAME, AgentCheck.CRITICAL,\n message=message, tags=self.custom_tags)\n raise Exception(message)\n\n else:\n self.service_check(SERVICE_CHECK_NAME, AgentCheck.OK, tags=self.custom_tags)\n\n # Create a set of filtered containers based on the exclude/include rules\n # and cache these rules in docker_util\n self._filter_containers(containers)\n\n containers_by_id = {}\n\n for container in containers:\n container_name = DockerUtil.container_name_extractor(container)[0]\n\n container_status_tags = self._get_tags(container, CONTAINER)\n\n all_containers_count[tuple(sorted(container_status_tags))] += 1\n if self._is_container_running(container):\n running_containers_count[tuple(sorted(container_status_tags))] += 1\n\n # Check if the container is included/excluded via its tags\n if self._is_container_excluded(container):\n self.log.debug(\"Container {0} is excluded\".format(container_name))\n continue\n\n containers_by_id[container['Id']] = container\n\n # grab pid via API if custom cgroups - otherwise we won't find process when\n # crawling for pids.\n if custom_cgroups or healthchecks:\n try:\n inspect_dict = self.docker_util.client.inspect_container(container_name)\n container['_pid'] = inspect_dict['State']['Pid']\n container['health'] = inspect_dict['State'].get('Health', {})\n except Exception as e:\n self.log.debug(\"Unable to inspect Docker container: %s\", e)\n\n total_count = 0\n # TODO: deprecate these 2, they should be replaced by _report_container_count\n for tags, count in running_containers_count.iteritems():\n total_count += count\n self.gauge(\"docker.containers.running\", count, tags=list(tags))\n self.gauge(\"docker.containers.running.total\", total_count, tags=self.custom_tags)\n\n total_count = 0\n for tags, count in all_containers_count.iteritems():\n stopped_count = count - running_containers_count[tags]\n total_count += stopped_count\n self.gauge(\"docker.containers.stopped\", stopped_count, tags=list(tags))\n self.gauge(\"docker.containers.stopped.total\", total_count, tags=self.custom_tags)\n\n return containers_by_id", "response": "List all the containers from the API filter and count them."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_tags(self, entity=None, tag_type=None):\n # Start with custom tags\n tags = list(self.custom_tags)\n\n # Collect pod names as tags on kubernetes\n if Platform.is_k8s() and KubeUtil.POD_NAME_LABEL not in self.collect_labels_as_tags:\n self.collect_labels_as_tags.append(KubeUtil.POD_NAME_LABEL)\n self.collect_labels_as_tags.append(KubeUtil.CONTAINER_NAME_LABEL)\n\n # Collect container names as tags on rancher\n if Platform.is_rancher():\n if RANCHER_CONTAINER_NAME not in self.collect_labels_as_tags:\n self.collect_labels_as_tags.append(RANCHER_CONTAINER_NAME)\n if RANCHER_SVC_NAME not in self.collect_labels_as_tags:\n self.collect_labels_as_tags.append(RANCHER_SVC_NAME)\n if RANCHER_STACK_NAME not in self.collect_labels_as_tags:\n self.collect_labels_as_tags.append(RANCHER_STACK_NAME)\n\n if entity is not None:\n pod_name = None\n namespace = None\n # Get labels as tags\n labels = entity.get(\"Labels\")\n if labels is not None:\n for k in self.collect_labels_as_tags:\n if k in labels:\n v = labels[k]\n if k == KubeUtil.POD_NAME_LABEL and Platform.is_k8s():\n pod_name = v\n k = \"pod_name\"\n if \"-\" in pod_name:\n replication_controller = \"-\".join(pod_name.split(\"-\")[:-1])\n if \"/\" in replication_controller: # k8s <= 1.1\n namespace, replication_controller = replication_controller.split(\"/\", 1)\n\n elif KubeUtil.NAMESPACE_LABEL in labels: # k8s >= 1.2\n namespace = labels[KubeUtil.NAMESPACE_LABEL]\n\n tags.append(\"kube_namespace:%s\" % namespace)\n tags.append(\"kube_replication_controller:%s\" % replication_controller)\n tags.append(\"pod_name:%s\" % pod_name)\n\n elif k == KubeUtil.CONTAINER_NAME_LABEL and Platform.is_k8s():\n if v:\n tags.append(\"kube_container_name:%s\" % v)\n elif k == SWARM_SVC_LABEL and Platform.is_swarm():\n if v:\n tags.append(\"swarm_service:%s\" % v)\n elif k == RANCHER_CONTAINER_NAME and Platform.is_rancher():\n if v:\n tags.append('rancher_container:%s' % v)\n elif k == RANCHER_SVC_NAME and Platform.is_rancher():\n if v:\n tags.append('rancher_service:%s' % v)\n elif k == RANCHER_STACK_NAME and Platform.is_rancher():\n if v:\n tags.append('rancher_stack:%s' % v)\n\n elif not v:\n tags.append(k)\n\n else:\n tags.append(\"%s:%s\" % (k, v))\n\n if k == KubeUtil.POD_NAME_LABEL and Platform.is_k8s() and k not in labels:\n tags.append(\"pod_name:no_pod\")\n\n # Get entity specific tags\n if tag_type is not None:\n tag_names = self.tag_names[tag_type]\n for tag_name in tag_names:\n tag_value = self._extract_tag_value(entity, tag_name)\n if tag_value is not None:\n for t in tag_value:\n tags.append('%s:%s' % (tag_name, str(t).strip()))\n\n # Add kube labels and creator/service tags\n if Platform.is_k8s() and namespace and pod_name:\n kube_tags = self.kube_pod_tags.get(\"{0}/{1}\".format(namespace, pod_name))\n if kube_tags:\n tags.extend(list(kube_tags))\n\n if self.metadata_collector.has_detected():\n orch_tags = self.metadata_collector.get_container_tags(co=entity)\n tags.extend(orch_tags)\n\n return tags", "response": "Generate the tags for a given entity according to a list of tag names."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _extract_tag_value(self, entity, tag_name):\n if tag_name not in TAG_EXTRACTORS:\n self.warning(\"{0} isn't a supported tag\".format(tag_name))\n return\n\n # Check for already extracted tags\n if \"_tag_values\" not in entity:\n entity[\"_tag_values\"] = {}\n\n if tag_name not in entity[\"_tag_values\"]:\n entity[\"_tag_values\"][tag_name] = TAG_EXTRACTORS[tag_name](entity)\n\n return entity[\"_tag_values\"][tag_name]", "response": "Extract the value of a tag from the API result."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if a container is excluded according to the filter rules.", "response": "def _is_container_excluded(self, container):\n \"\"\"Check if a container is excluded according to the filter rules.\n\n Requires _filter_containers to run first.\n \"\"\"\n container_name = DockerUtil.container_name_extractor(container)[0]\n return container_name in self._filtered_containers"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _send_container_healthcheck_sc(self, containers_by_id):\n for container in containers_by_id.itervalues():\n healthcheck_tags = self._get_tags(container, HEALTHCHECK)\n match = False\n for tag in healthcheck_tags:\n for rule in self.whitelist_patterns:\n if re.match(rule, tag):\n match = True\n\n self._submit_healthcheck_sc(container)\n break\n\n if match:\n break", "response": "Send health service checks for containers."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _report_container_count(self, containers_by_id):\n m_func = FUNC_MAP[GAUGE][self.use_histogram]\n\n per_state_count = defaultdict(int)\n\n filterlambda = lambda ctr: not self._is_container_excluded(ctr)\n containers = list(filter(filterlambda, containers_by_id.values()))\n\n for ctr in containers:\n per_state_count[ctr.get('State', '')] += 1\n\n for state in per_state_count:\n if state:\n m_func(self, 'docker.container.count', per_state_count[state], tags=['container_state:%s' % state.lower()])", "response": "Report the number of containers in each state."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreports volume count per state ( attached or not", "response": "def _report_volume_count(self):\n \"\"\"Report volume count per state (dangling or not)\"\"\"\n m_func = FUNC_MAP[GAUGE][self.use_histogram]\n\n attached_volumes = self.docker_util.client.volumes(filters={'dangling': False})\n dangling_volumes = self.docker_util.client.volumes(filters={'dangling': True})\n attached_count = len(attached_volumes.get('Volumes', []) or [])\n dangling_count = len(dangling_volumes.get('Volumes', []) or [])\n m_func(self, 'docker.volume.count', attached_count, tags=['volume_state:attached'])\n m_func(self, 'docker.volume.count', dangling_count, tags=['volume_state:dangling'])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _report_net_metrics(self, container, tags):\n if self._disable_net_metrics:\n self.log.debug(\"Network metrics are disabled. Skipping\")\n return\n\n proc_net_file = os.path.join(container['_proc_root'], 'net/dev')\n\n try:\n if container['Id'] in self.network_mappings:\n networks = self.network_mappings[container['Id']]\n else:\n networks = self.docker_util.get_container_network_mapping(container)\n if not networks:\n networks = {'eth0': 'bridge'}\n self.network_mappings[container['Id']] = networks\n except Exception as e:\n # Revert to previous behaviour if the method is missing or failing\n # Debug message will only appear once per container, then the cache is used\n self.log.debug(\"Failed to build docker network mapping, using failsafe. Exception: {0}\".format(e))\n networks = {'eth0': 'bridge'}\n self.network_mappings[container['Id']] = networks\n\n try:\n with open(proc_net_file, 'r') as fp:\n lines = fp.readlines()\n \"\"\"Two first lines are headers:\n Inter-| Receive | Transmit\n face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed\n \"\"\"\n for l in lines[2:]:\n cols = l.split(':', 1)\n interface_name = str(cols[0]).strip()\n if interface_name in networks:\n net_tags = tags + ['docker_network:'+networks[interface_name]]\n x = cols[1].split()\n m_func = FUNC_MAP[RATE][self.use_histogram]\n m_func(self, \"docker.net.bytes_rcvd\", long(x[0]), net_tags)\n m_func(self, \"docker.net.bytes_sent\", long(x[8]), net_tags)\n\n except IOError as e:\n # It is possible that the container got stopped between the API call and now\n self.log.debug(\"Cannot read network interface file, container likely raced to finish : {0}\".format(e))", "response": "Report the network metrics for the container."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_events(self):\n events, changed_container_ids = self.docker_util.get_events()\n if not self._disable_net_metrics:\n self._invalidate_network_mapping_cache(events)\n if changed_container_ids and self._service_discovery:\n get_sd_backend(self.agentConfig).update_checks(changed_container_ids)\n if changed_container_ids:\n self.metadata_collector.invalidate_cache(events)\n return events", "response": "Get the list of events."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate the actual event to submit from a list of similar docker events", "response": "def _create_dd_event(self, events, image, c_tags, priority='Normal'):\n \"\"\"Create the actual event to submit from a list of similar docker events\"\"\"\n if not events:\n return\n\n max_timestamp = 0\n status = defaultdict(int)\n status_change = []\n\n for ev, c_name in events:\n max_timestamp = max(max_timestamp, int(ev['time']))\n status[ev['status']] += 1\n status_change.append([c_name, ev['status']])\n\n status_text = \", \".join([\"%d %s\" % (count, st) for st, count in status.iteritems()])\n msg_title = \"%s %s on %s\" % (image, status_text, self.hostname)\n msg_body = (\n \"%%%\\n\"\n \"{image_name} {status} on {hostname}\\n\"\n \"```\\n{status_changes}\\n```\\n\"\n \"%%%\"\n ).format(\n image_name=image,\n status=status_text,\n hostname=self.hostname,\n status_changes=\"\\n\".join(\n [\"%s \\t%s\" % (change[1].upper(), change[0]) for change in status_change])\n )\n\n if any(error in status_text for error in ERROR_ALERT_TYPE):\n alert_type = \"error\"\n else:\n alert_type = None\n\n return {\n 'timestamp': max_timestamp,\n 'host': self.hostname,\n 'event_type': EVENT_TYPE,\n 'msg_title': msg_title,\n 'msg_text': msg_body,\n 'source_type_name': EVENT_TYPE,\n 'event_object': 'docker:%s' % image,\n 'tags': list(c_tags),\n 'alert_type': alert_type,\n 'priority': priority\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreporting metrics about the disk usage", "response": "def _report_disk_stats(self):\n \"\"\"Report metrics about the volume space usage\"\"\"\n stats = {\n 'docker.data.used': None,\n 'docker.data.total': None,\n 'docker.data.free': None,\n 'docker.metadata.used': None,\n 'docker.metadata.total': None,\n 'docker.metadata.free': None\n # these two are calculated by _calc_percent_disk_stats\n # 'docker.data.percent': None,\n # 'docker.metadata.percent': None\n }\n info = self.docker_util.client.info()\n driver_status = info.get('DriverStatus', [])\n if not driver_status:\n self.log.warning('Disk metrics collection is enabled but docker info did not'\n ' report any. Your storage driver might not support them, skipping.')\n return\n for metric in driver_status:\n # only consider metrics about disk space\n if len(metric) == 2 and 'Space' in metric[0]:\n # identify Data and Metadata metrics\n mtype = 'data'\n if 'Metadata' in metric[0]:\n mtype = 'metadata'\n\n if 'Used' in metric[0]:\n stats['docker.{0}.used'.format(mtype)] = metric[1]\n elif 'Space Total' in metric[0]:\n stats['docker.{0}.total'.format(mtype)] = metric[1]\n elif 'Space Available' in metric[0]:\n stats['docker.{0}.free'.format(mtype)] = metric[1]\n stats = self._format_disk_metrics(stats)\n stats.update(self._calc_percent_disk_stats(stats))\n tags = self._get_tags()\n for name, val in stats.iteritems():\n if val is not None:\n self.gauge(name, val, tags)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncasts the disk stats to float and convert them to bytes", "response": "def _format_disk_metrics(self, metrics):\n \"\"\"Cast the disk stats to float and convert them to bytes\"\"\"\n for name, raw_val in metrics.iteritems():\n if raw_val:\n match = DISK_STATS_RE.search(raw_val)\n if match is None or len(match.groups()) != 2:\n self.log.warning('Can\\'t parse value %s for disk metric %s. Dropping it.' % (raw_val, name))\n metrics[name] = None\n val, unit = match.groups()\n # by default some are uppercased others lowercased. That's error prone.\n unit = unit.lower()\n try:\n val = int(float(val) * UNIT_MAP[unit])\n metrics[name] = val\n except KeyError:\n self.log.error('Unrecognized unit %s for disk metric %s. Dropping it.' % (unit, name))\n metrics[name] = None\n return metrics"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate a percentage of used disk space for data and metadata", "response": "def _calc_percent_disk_stats(self, stats):\n \"\"\"Calculate a percentage of used disk space for data and metadata\"\"\"\n mtypes = ['data', 'metadata']\n percs = {}\n for mtype in mtypes:\n used = stats.get('docker.{0}.used'.format(mtype))\n total = stats.get('docker.{0}.total'.format(mtype))\n free = stats.get('docker.{0}.free'.format(mtype))\n if used and total and free and ceil(total) < free + used:\n self.log.debug('used, free, and total disk metrics may be wrong, '\n 'used: %s, free: %s, total: %s',\n used, free, total)\n total = used + free\n try:\n if isinstance(used, int):\n percs['docker.{0}.percent'.format(mtype)] = round(100 * float(used) / float(total), 2)\n elif isinstance(free, int):\n percs['docker.{0}.percent'.format(mtype)] = round(100 * (1.0 - (float(free) / float(total))), 2)\n except ZeroDivisionError:\n self.log.error('docker.{0}.total is 0, calculating docker.{1}.percent'\n ' is not possible.'.format(mtype, mtype))\n return percs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfinds a specific cgroup file containing metrics to extract.", "response": "def _get_cgroup_from_proc(self, cgroup, pid, filename):\n \"\"\"Find a specific cgroup file, containing metrics to extract.\"\"\"\n params = {\n \"file\": filename,\n }\n return DockerUtil.find_cgroup_from_proc(self._mountpoints, pid, cgroup, self.docker_util._docker_root) % (params)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse a cgroup pseudo file for key - value pairs.", "response": "def _parse_cgroup_file(self, stat_file):\n \"\"\"Parse a cgroup pseudo file for key/values.\"\"\"\n self.log.debug(\"Opening cgroup file: %s\" % stat_file)\n try:\n with open(stat_file, 'r') as fp:\n if 'blkio' in stat_file:\n return self._parse_blkio_metrics(fp.read().splitlines())\n elif 'cpuacct.usage' in stat_file:\n return dict({'usage': str(int(fp.read())/10000000)})\n elif 'memory.soft_limit_in_bytes' in stat_file:\n value = int(fp.read())\n # do not report kernel max default value (uint64 * 4096)\n # see https://github.com/torvalds/linux/blob/5b36577109be007a6ecf4b65b54cbc9118463c2b/mm/memcontrol.c#L2844-L2845\n # 2 ** 60 is kept for consistency of other cgroups metrics\n if value < 2 ** 60:\n return dict({'softlimit': value})\n elif 'cpu.shares' in stat_file:\n value = int(fp.read())\n return {'shares': value}\n else:\n return dict(map(lambda x: x.split(' ', 1), fp.read().splitlines()))\n except IOError:\n # It is possible that the container got stopped between the API call and now.\n # Some files can also be missing (like cpu.stat) and that's fine.\n self.log.debug(\"Can't open %s. Its metrics will be missing.\" % stat_file)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the blkio metrics.", "response": "def _parse_blkio_metrics(self, stats):\n \"\"\"Parse the blkio metrics.\"\"\"\n metrics = {\n 'io_read': 0,\n 'io_write': 0,\n }\n for line in stats:\n if 'Read' in line:\n metrics['io_read'] += int(line.split()[2])\n if 'Write' in line:\n metrics['io_write'] += int(line.split()[2])\n return metrics"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncrawl proc to find container PIDs and add them to containers_by_id.", "response": "def _crawl_container_pids(self, container_dict, custom_cgroups=False):\n \"\"\"Crawl `/proc` to find container PIDs and add them to `containers_by_id`.\"\"\"\n proc_path = os.path.join(self.docker_util._docker_root, 'proc')\n pid_dirs = [_dir for _dir in os.listdir(proc_path) if _dir.isdigit()]\n\n if len(pid_dirs) == 0:\n self.warning(\"Unable to find any pid directory in {0}. \"\n \"If you are running the agent in a container, make sure to \"\n 'share the volume properly: \"/proc:/host/proc:ro\". '\n \"See https://github.com/DataDog/docker-dd-agent/blob/master/README.md for more information. \"\n \"Network metrics will be missing\".format(proc_path))\n self._disable_net_metrics = True\n return container_dict\n\n self._disable_net_metrics = False\n\n for folder in pid_dirs:\n try:\n path = os.path.join(proc_path, folder, 'cgroup')\n with open(path, 'r') as f:\n content = [line.strip().split(':') for line in f.readlines()]\n\n selinux_policy = ''\n path = os.path.join(proc_path, folder, 'attr', 'current')\n if os.path.exists(path):\n with open(path, 'r') as f:\n selinux_policy = f.readlines()[0]\n except IOError, e:\n # Issue #2074\n self.log.debug(\"Cannot read %s, process likely raced to finish : %s\", path, e)\n except Exception as e:\n self.warning(\"Cannot read %s : %s\" % (path, str(e)))\n continue\n\n try:\n for line in content:\n if self._is_container_cgroup(line, selinux_policy):\n cpuacct = line[2]\n break\n else:\n continue\n\n matches = re.findall(CONTAINER_ID_RE, cpuacct)\n if matches:\n container_id = matches[-1]\n if container_id not in container_dict:\n self.log.debug(\n \"Container %s not in container_dict, it's likely excluded\", container_id\n )\n continue\n container_dict[container_id]['_pid'] = folder\n container_dict[container_id]['_proc_root'] = os.path.join(proc_path, folder)\n elif custom_cgroups: # if we match by pid that should be enough (?) - O(n) ugh!\n for _, container in container_dict.iteritems():\n if container.get('_pid') == int(folder):\n container['_proc_root'] = os.path.join(proc_path, folder)\n break\n\n except Exception, e:\n self.warning(\"Cannot parse %s content: %s\" % (path, str(e)))\n continue\n return container_dict"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_projects(self):\n url = urljoin(self.keystone_endpoint, \"{}/{}\".format(DEFAULT_KEYSTONE_API_VERSION, \"projects\"))\n try:\n r = self._make_request(url, self.headers)\n return r.get('projects', [])\n\n except Exception as e:\n self.logger.warning('Unable to get projects: {}'.format(e))\n raise e", "response": "Returns all projects in the domain"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_user_identity(user):\n if not (\n user and user.get('name') and user.get('password') and user.get(\"domain\") and user.get(\"domain\").get(\"id\")\n ):\n raise IncompleteIdentity()\n\n identity = {\"methods\": ['password'], \"password\": {\"user\": user}}\n return identity", "response": "Parse user identity out of init_config and return it as dict"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the JSON response from the Neutron Identity API and return the endpoint that is valid", "response": "def _get_neutron_endpoint(cls, json_resp):\n \"\"\"\n Parse the service catalog returned by the Identity API for an endpoint matching the Neutron service\n Sends a CRITICAL service check when none are found registered in the Catalog\n \"\"\"\n valid_endpoint = cls._get_valid_endpoint(json_resp, 'neutron', 'network')\n if valid_endpoint:\n return valid_endpoint\n raise MissingNeutronEndpoint()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_nova_endpoint(cls, json_resp):\n valid_endpoint = cls._get_valid_endpoint(json_resp, 'nova', 'compute')\n if valid_endpoint:\n return valid_endpoint\n raise MissingNovaEndpoint()", "response": "Parse the Nova service catalog response and return the endpoint that matches the requested version"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the response from the Identity API to find the endpoint that is valid for the requested version of the service", "response": "def _get_valid_endpoint(resp, name, entry_type):\n \"\"\"\n Parse the service catalog returned by the Identity API for an endpoint matching\n the Nova service with the requested version\n Sends a CRITICAL service check when no viable candidates are found in the Catalog\n \"\"\"\n catalog = resp.get('token', {}).get('catalog', [])\n for entry in catalog:\n if (\n entry.get('name')\n and entry.get('type')\n and entry.get('name') == name\n and entry.get('type') == entry_type\n ):\n # Collect any endpoints on the public or internal interface\n valid_endpoints = {}\n for ep in entry.get('endpoints'):\n interface = ep.get('interface', '')\n if interface in ['public', 'internal']:\n valid_endpoints[interface] = ep.get('url')\n\n if valid_endpoints:\n # Favor public endpoints over internal\n return valid_endpoints.get('public', valid_endpoints.get('internal'))\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dep():\n failed = False\n catalog, errors = make_catalog()\n\n # Check unpinned\n if errors:\n for error in errors:\n echo_failure(error)\n failed = True\n\n # Check uniqueness\n have_multiple_versions = set()\n have_multiple_markers = set()\n for package in catalog.packages:\n versions = catalog.get_package_versions(package)\n if len(versions) > 1:\n if package.name in have_multiple_versions:\n # don't print the error multiple times\n continue\n\n failed = True\n have_multiple_versions.add(package.name)\n display_multiple_attributes(versions, 'Multiple versions found for package `{}`:'.format(package.name))\n\n markers = catalog.get_package_markers(package)\n if len(markers) > 1:\n if package.name in have_multiple_markers:\n # don't print the error multiple times\n continue\n\n failed = True\n have_multiple_markers.add(package.name)\n display_multiple_attributes(markers, 'Multiple markers found for package `{}`:'.format(package))\n\n # Check embedded env compatibility\n agent_req_file = get_agent_requirements()\n embedded_deps = {p.name: p for p in read_packages(agent_req_file)}\n for check_name in sorted(os.listdir(get_root())):\n for package in catalog.get_check_packages(check_name):\n if package.name not in embedded_deps:\n failed = True\n echo_failure(\n 'Dependency `{}` for check `{}` missing from the embedded environment'.format(\n package.name, check_name\n )\n )\n elif embedded_deps[package.name] != package:\n failed = True\n echo_failure(\n 'Dependency `{}` mismatch for check `{}` in the embedded environment'.format(\n package.name, check_name\n )\n )\n echo_info(' have: {}'.format(embedded_deps[package.name]))\n echo_info(' want: {}'.format(package))\n\n if failed:\n abort()", "response": "This command checks if all the dependencies of the base check are compatible."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nshowing metric info from a Prometheus endpoint.", "response": "def info(endpoint):\n \"\"\"Show metric info from a Prometheus endpoint.\n\n \\b\n Example:\n $ ddev meta prom info :8080/_status/vars\n \"\"\"\n endpoint = sanitize_endpoint(endpoint)\n\n metrics = parse_metrics(endpoint)\n num_metrics = len(metrics)\n num_gauge = 0\n num_counter = 0\n num_histogram = 0\n\n for data in itervalues(metrics):\n metric_type = data.get('type')\n\n if metric_type == 'gauge':\n num_gauge += 1\n elif metric_type == 'counter':\n num_counter += 1\n elif metric_type == 'histogram':\n num_histogram += 1\n\n if num_metrics:\n echo_success('Number of metrics: {}'.format(num_metrics))\n else:\n echo_warning('No metrics!')\n return\n\n if num_gauge:\n echo_info('Type `gauge`: {}'.format(num_gauge))\n\n if num_counter:\n echo_info('Type `counter`: {}'.format(num_counter))\n\n if num_histogram:\n echo_info('Type `histogram`: {}'.format(num_histogram))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _add_to_bulk(self, bulkobj):\n bulkobj.add_delete(self._filter, 0, collation=self._collation)", "response": "Add this operation to the _Bulk instance bulkobj."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _add_to_bulk(self, bulkobj):\n bulkobj.add_replace(self._filter, self._doc, self._upsert,\n collation=self._collation)", "response": "Add this operation to the _Bulk instance bulkobj."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd this operation to the _Bulk instance bulkobj.", "response": "def _add_to_bulk(self, bulkobj):\n \"\"\"Add this operation to the _Bulk instance `bulkobj`.\"\"\"\n bulkobj.add_update(self._filter, self._doc, False, self._upsert,\n collation=self._collation)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload a private key and the corresponding certificate.", "response": "def load_cert_chain(self, certfile, keyfile=None):\n \"\"\"Load a private key and the corresponding certificate. The certfile\n string must be the path to a single file in PEM format containing the\n certificate as well as any number of CA certificates needed to\n establish the certificate's authenticity. The keyfile string, if\n present, must point to a file containing the private key. Otherwise\n the private key will be taken from certfile as well.\n \"\"\"\n self._certfile = certfile\n self._keyfile = keyfile"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef wrap_socket(self, sock, server_side=False,\n do_handshake_on_connect=True,\n suppress_ragged_eofs=True, dummy=None):\n \"\"\"Wrap an existing Python socket sock and return an ssl.SSLSocket\n object.\n \"\"\"\n return ssl.wrap_socket(sock, keyfile=self._keyfile,\n certfile=self._certfile,\n server_side=server_side,\n cert_reqs=self._verify_mode,\n ssl_version=self._protocol,\n ca_certs=self._cafile,\n do_handshake_on_connect=do_handshake_on_connect,\n suppress_ragged_eofs=suppress_ragged_eofs)", "response": "Wrap an existing Python socket sock and return an ssl. SSLSocket object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse the Prometheus metric family from the binary buffer buf.", "response": "def parse_metric_family(buf):\n \"\"\"\n Parse the binary buffer in input, searching for Prometheus messages\n of type MetricFamily [0] delimited by a varint32 [1].\n\n [0] https://github.com/prometheus/client_model/blob/086fe7ca28bde6cec2acd5223423c1475a362858/metrics.proto#L76-%20%20L81 # noqa: E501\n [1] https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractMessageLite#writeDelimitedTo(java.io.OutputStream) # noqa: E501\n \"\"\"\n n = 0\n while n < len(buf):\n msg_len, new_pos = _DecodeVarint32(buf, n)\n n = new_pos\n msg_buf = buf[n : n + msg_len]\n n += msg_len\n\n message = metrics_pb2.MetricFamily()\n message.ParseFromString(msg_buf)\n yield message"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef multidict(ordered_pairs):\n # read all values into lists\n d = defaultdict(list)\n for k, v in ordered_pairs:\n d[k].append(v)\n # unpack lists that have only 1 item\n dict_copy = deepcopy(d)\n for k, v in iteritems(dict_copy):\n if len(v) == 1:\n d[k] = v[0]\n return dict(d)", "response": "Convert duplicate keys values to lists."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndump the packages in the catalog in a requirements file", "response": "def write_packages(self, reqs_file):\n \"\"\"\n Dump the packages in the catalog in a requirements file\n \"\"\"\n write_file_lines(reqs_file, ('{}\\n'.format(package) for package in self.packages))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding a Package to the catalog for the given check_name.", "response": "def add_package(self, check_name, package):\n \"\"\"\n Add a Package to the catalog for the given check\n \"\"\"\n self._package_set.add(package)\n package_data = self._packages[package.name]\n self._checks_deps[check_name].append(package)\n\n # Versions\n if package.version:\n versions = package_data['versions']\n versions[package.version].append(check_name)\n\n # Marker section\n if package.marker:\n markers = package_data['markers']\n markers[package.marker].append(check_name)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _update_container_metrics(self, instance, subcontainer, kube_labels):\n tags = list(instance.get('tags', [])) # add support for custom tags\n\n if len(subcontainer.get('aliases', [])) >= 1:\n # The first alias seems to always match the docker container name\n container_name = subcontainer['aliases'][0]\n else:\n self.log.debug(\"Subcontainer doesn't have a name, skipping.\")\n return\n\n tags.append('container_name:%s' % container_name)\n\n container_image = self.kubeutil.image_name_resolver(subcontainer['spec'].get('image'))\n if container_image:\n tags.append('container_image:%s' % container_image)\n\n split = container_image.split(\":\")\n if len(split) > 2:\n # if the repo is in the image name and has the form 'docker.clearbit:5000'\n # the split will be like [repo_url, repo_port/image_name, image_tag]. Let's avoid that\n split = [':'.join(split[:-1]), split[-1]]\n\n tags.append('image_name:%s' % split[0])\n if len(split) == 2:\n tags.append('image_tag:%s' % split[1])\n\n try:\n cont_labels = subcontainer['spec']['labels']\n except KeyError:\n self.log.debug(\"Subcontainer, doesn't have any labels\")\n cont_labels = {}\n\n # Collect pod names, namespaces, rc...\n if KubeUtil.NAMESPACE_LABEL in cont_labels and KubeUtil.POD_NAME_LABEL in cont_labels:\n # Kubernetes >= 1.2\n tags += self._get_post_1_2_tags(cont_labels, subcontainer, kube_labels)\n\n elif KubeUtil.POD_NAME_LABEL in cont_labels:\n # Kubernetes <= 1.1\n tags += self._get_pre_1_2_tags(cont_labels, subcontainer, kube_labels)\n\n else:\n # Those are containers that are not part of a pod.\n # They are top aggregate views and don't have the previous metadata.\n tags.append(\"pod_name:no_pod\")\n\n # if the container should be filtered we return its tags without publishing its metrics\n is_filtered = self.kubeutil.are_tags_filtered(tags)\n if is_filtered:\n self._filtered_containers.add(subcontainer['id'])\n return tags\n\n stats = subcontainer['stats'][-1] # take the latest\n self._publish_raw_metrics(NAMESPACE, stats, tags)\n\n if subcontainer.get(\"spec\", {}).get(\"has_filesystem\") and stats.get('filesystem', []) != []:\n fs = stats['filesystem'][-1]\n if fs['capacity'] > 0:\n fs_utilization = float(fs['usage'])/float(fs['capacity'])\n self.publish_gauge(self, NAMESPACE + '.filesystem.usage_pct', fs_utilization, tags)\n else:\n self.log.debug(\"Filesystem capacity is 0: cannot report usage metrics.\")\n\n if subcontainer.get(\"spec\", {}).get(\"has_network\"):\n net = stats['network']\n self.publish_rate(self, NAMESPACE + '.network_errors',\n sum(float(net[x]) for x in NET_ERRORS),\n tags)\n\n return tags", "response": "Update the tags for a container and handle filtering on tags"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating the metrics for running pods on this node.", "response": "def _update_pods_metrics(self, instance, pods):\n \"\"\"\n Reports the number of running pods on this node, tagged by service and creator\n\n We go though all the pods, extract tags then count them by tag list, sorted and\n serialized in a pipe-separated string (it is an illegar character for tags)\n \"\"\"\n tags_map = defaultdict(int)\n for pod in pods['items']:\n pod_meta = pod.get('metadata', {})\n pod_tags = self.kubeutil.get_pod_creator_tags(pod_meta, legacy_rep_controller_tag=True)\n services = self.kubeutil.match_services_for_pod(pod_meta)\n if isinstance(services, list):\n for service in services:\n pod_tags.append('kube_service:%s' % service)\n if 'namespace' in pod_meta:\n pod_tags.append('kube_namespace:%s' % pod_meta['namespace'])\n\n tags_map[frozenset(pod_tags)] += 1\n\n commmon_tags = instance.get('tags', [])\n for pod_tags, pod_count in tags_map.iteritems():\n tags = list(pod_tags)\n tags.extend(commmon_tags)\n self.publish_gauge(self, NAMESPACE + '.pods.running', pod_count, tags)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the list of kube events and send ddog events", "response": "def _update_kube_events(self, instance, pods_list, event_items):\n \"\"\"\n Process kube events and send ddog events\n The namespace filtering is done here instead of KubeEventRetriever\n to avoid interfering with service discovery\n \"\"\"\n node_ip, node_name = self.kubeutil.get_node_info()\n self.log.debug('Processing events on {} [{}]'.format(node_name, node_ip))\n\n k8s_namespaces = instance.get('namespaces', DEFAULT_NAMESPACES)\n if not isinstance(k8s_namespaces, list):\n self.log.warning('Configuration key \"namespaces\" is not a list: fallback to the default value')\n k8s_namespaces = DEFAULT_NAMESPACES\n\n # handle old config value\n if 'namespace' in instance and instance.get('namespace') not in (None, 'default'):\n self.log.warning('''The 'namespace' parameter is deprecated and will stop being supported starting '''\n '''from 5.13. Please use 'namespaces' and/or 'namespace_name_regexp' instead.''')\n k8s_namespaces.append(instance.get('namespace'))\n\n if self.k8s_namespace_regexp:\n namespaces_endpoint = '{}/namespaces'.format(self.kubeutil.kubernetes_api_url)\n self.log.debug('Kubernetes API endpoint to query namespaces: %s' % namespaces_endpoint)\n\n namespaces = self.kubeutil.retrieve_json_auth(namespaces_endpoint).json()\n for namespace in namespaces.get('items', []):\n name = namespace.get('metadata', {}).get('name', None)\n if name and self.k8s_namespace_regexp.match(name):\n k8s_namespaces.append(name)\n\n k8s_namespaces = set(k8s_namespaces)\n\n for event in event_items:\n event_ts = calendar.timegm(time.strptime(event.get('lastTimestamp'), '%Y-%m-%dT%H:%M:%SZ'))\n involved_obj = event.get('involvedObject', {})\n\n # filter events by white listed namespaces (empty namespace belong to the 'default' one)\n if involved_obj.get('namespace', 'default') not in k8s_namespaces:\n continue\n\n tags = self.kubeutil.extract_event_tags(event)\n tags.extend(instance.get('tags', []))\n\n title = '{} {} on {}'.format(involved_obj.get('name'), event.get('reason'), node_name)\n message = event.get('message')\n source = event.get('source')\n k8s_event_type = event.get('type')\n alert_type = K8S_ALERT_MAP.get(k8s_event_type, 'info')\n\n if source:\n message += '\\nSource: {} {}\\n'.format(source.get('component', ''), source.get('host', ''))\n msg_body = \"%%%\\n{}\\n```\\n{}\\n```\\n%%%\".format(title, message)\n dd_event = {\n 'timestamp': event_ts,\n 'host': node_ip,\n 'event_type': EVENT_TYPE,\n 'msg_title': title,\n 'msg_text': msg_body,\n 'source_type_name': EVENT_TYPE,\n 'alert_type': alert_type,\n 'event_object': 'kubernetes:{}'.format(involved_obj.get('name')),\n 'tags': tags,\n }\n self.event(dd_event)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef refresh_leader_status(self, instance):\n if not self.leader_candidate:\n return\n\n leader_status = self.kubeutil.is_leader\n self.kubeutil.refresh_leader()\n\n # nothing changed, no-op\n if leader_status == self.kubeutil.is_leader:\n return\n # else, reset the event collection config\n else:\n self.log.info(\"Leader status changed, updating event collection config...\")\n self._configure_event_collection(instance)", "response": "Updates the leader status of the current instance."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading the RRD file and return the number of metrics that were found.", "response": "def _read_rrd(self, rrd_path, hostname, device_name, tags):\n ''' Main metric fetching method '''\n metric_count = 0\n\n try:\n info = self._get_rrd_info(rrd_path)\n except Exception:\n # Unable to read RRD file, ignore it\n self.log.exception(\"Unable to read RRD file at %s\" % rrd_path)\n return metric_count\n\n # Find the consolidation functions for the RRD metrics\n c_funcs = set([v for k, v in info.items() if k.endswith('.cf')])\n\n for c in list(c_funcs):\n last_ts_key = '%s.%s' % (rrd_path, c)\n if last_ts_key not in self.last_ts:\n self.last_ts[last_ts_key] = int(time.time())\n continue\n\n start = self.last_ts[last_ts_key]\n last_ts = start\n\n try:\n fetched = self._get_rrd_fetch(rrd_path, c, start)\n except rrdtool.error:\n # Start time was out of range, skip this RRD\n self.log.warn(\"Time %s out of range for %s\" % (rrd_path, start))\n return metric_count\n\n # Extract the data\n (start_ts, end_ts, interval) = fetched[0]\n metric_names = fetched[1]\n points = fetched[2]\n for k, m_name in enumerate(metric_names):\n m_name = self._format_metric_name(m_name, c)\n for i, p in enumerate(points):\n ts = start_ts + (i * interval)\n\n if p[k] is None:\n continue\n\n # Save this metric as a gauge\n val = self._transform_metric(m_name, p[k])\n self.gauge(m_name, val, hostname=hostname, device_name=device_name, tags=tags)\n metric_count += 1\n last_ts = ts + interval\n\n # Update the last timestamp based on the last valid metric\n self.last_ts[last_ts_key] = last_ts\n return metric_count"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching metadata about each RRD in this Cacti DB returning a list of tuples of hostname device_name rrd_path.", "response": "def _fetch_rrd_meta(self, connection, rrd_path_root, whitelist, field_names, tags):\n ''' Fetch metadata about each RRD in this Cacti DB, returning a list of\n tuples of (hostname, device_name, rrd_path)\n '''\n\n def _in_whitelist(rrd):\n path = rrd.replace('/', '')\n for p in whitelist:\n if fnmatch(path, p):\n return True\n return False\n\n c = connection.cursor()\n\n and_parameters = \" OR \".join([\"hsc.field_name = '%s'\" % field_name for field_name in field_names])\n\n # Check for the existence of the `host_snmp_cache` table\n rrd_query = \"\"\"\n SELECT\n h.hostname as hostname,\n hsc.field_value as device_name,\n dt.data_source_path as rrd_path\n FROM data_local dl\n JOIN host h on dl.host_id = h.id\n JOIN data_template_data dt on dt.local_data_id = dl.id\n LEFT JOIN host_snmp_cache hsc on h.id = hsc.host_id\n AND dl.snmp_index = hsc.snmp_index\n WHERE dt.data_source_path IS NOT NULL\n AND dt.data_source_path != ''\n AND ({} OR hsc.field_name is NULL) \"\"\".format(\n and_parameters\n )\n\n c.execute(rrd_query)\n res = []\n for hostname, device_name, rrd_path in c.fetchall():\n if not whitelist or _in_whitelist(rrd_path):\n if hostname in ('localhost', '127.0.0.1'):\n hostname = self.hostname\n rrd_path = rrd_path.replace('', rrd_path_root)\n device_name = device_name or None\n res.append((hostname, device_name, rrd_path))\n\n # Collect stats\n num_hosts = len(set([r[0] for r in res]))\n self.gauge('cacti.rrd.count', len(res), tags=tags)\n self.gauge('cacti.hosts.count', num_hosts, tags=tags)\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nformats a cacti metric name into a Datadog - friendly name", "response": "def _format_metric_name(self, m_name, cfunc):\n ''' Format a cacti metric name into a Datadog-friendly name '''\n try:\n aggr = CFUNC_TO_AGGR[cfunc]\n except KeyError:\n aggr = cfunc.lower()\n\n try:\n m_name = CACTI_TO_DD[m_name]\n if aggr != 'avg':\n m_name += '.{}'.format(aggr)\n return m_name\n except KeyError:\n return \"cacti.{}.{}\".format(m_name.lower(), aggr)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calculate_perf_100nsec_timer(previous, current, property_name):\n n0 = previous[property_name]\n n1 = current[property_name]\n d0 = previous[\"Timestamp_Sys100NS\"]\n d1 = current[\"Timestamp_Sys100NS\"]\n\n if n0 is None or n1 is None:\n return\n\n return (n1 - n0) / (d1 - d0) * 100", "response": "Calculates the percentage of elapsed time between two previous and current objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the bulk count of a given property in a given previous object.", "response": "def calculate_perf_counter_bulk_count(previous, current, property_name):\n \"\"\"\n PERF_COUNTER_BULK_COUNT\n\n https://technet.microsoft.com/en-us/library/cc757486(v=ws.10).aspx\n \"\"\"\n n0 = previous[property_name]\n n1 = current[property_name]\n d0 = previous[\"Timestamp_Sys100NS\"]\n d1 = current[\"Timestamp_Sys100NS\"]\n f = current[\"Frequency_Sys100NS\"]\n\n if n0 is None or n1 is None:\n return\n\n return (n1 - n0) / ((d1 - d0) / f)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef calculate_perf_counter_100ns_queuelen_type(previous, current, property_name):\n n0 = previous[property_name]\n n1 = current[property_name]\n d0 = previous[\"Timestamp_Sys100NS\"]\n d1 = current[\"Timestamp_Sys100NS\"]\n if n0 is None or n1 is None:\n return\n\n return (n1 - n0) / (d1 - d0)", "response": "Calculates the 100ns queuelen type of the queue."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _server_known(cls, host, port):\n with PostgreSql._known_servers_lock:\n return (host, port) in PostgreSql._known_servers", "response": "Return whether the hostname and port combination is already seen."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the host and port combination for this server.", "response": "def _set_server_known(cls, host, port):\n \"\"\"\n Store the host/port combination for this server\n \"\"\"\n with PostgreSql._known_servers_lock:\n PostgreSql._known_servers.add((host, port))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the list of metrics for the current check instance.", "response": "def _get_instance_metrics(self, key, db, database_size_metrics, collect_default_db):\n \"\"\"\n Add NEWER_92_METRICS to the default set of COMMON_METRICS when server\n version is 9.2 or later.\n\n Store the list of metrics in the check instance to avoid rebuilding it at\n every collection cycle.\n\n In case we have multiple instances pointing to the same postgres server\n monitoring different databases, we want to collect server metrics\n only once. See https://github.com/DataDog/dd-agent/issues/1211\n \"\"\"\n metrics = self.instance_metrics.get(key)\n\n if metrics is None:\n host, port, dbname = key\n # check whether we already collected server-wide metrics for this\n # postgres instance\n if self._server_known(host, port):\n # explicitly set instance metrics for this key to an empty list\n # so we don't get here more than once\n self.instance_metrics[key] = []\n self.log.debug(\n \"Not collecting instance metrics for key: {} as \"\n \"they are already collected by another instance\".format(key)\n )\n return None\n self._set_server_known(host, port)\n\n # select the right set of metrics to collect depending on postgres version\n if self._is_9_2_or_above(key, db):\n self.instance_metrics[key] = dict(self.COMMON_METRICS, **self.NEWER_92_METRICS)\n else:\n self.instance_metrics[key] = dict(self.COMMON_METRICS)\n\n # add size metrics if needed\n if database_size_metrics:\n self.instance_metrics[key].update(self.DATABASE_SIZE_METRICS)\n\n metrics = self.instance_metrics.get(key)\n\n # this will happen when the current key contains a postgres server that\n # we already know, let's avoid to collect duplicates\n if not metrics:\n return None\n\n res = {\n 'descriptors': [('psd.datname', 'db')],\n 'metrics': metrics,\n 'query': \"SELECT psd.datname, %s \"\n \"FROM pg_stat_database psd \"\n \"JOIN pg_database pd ON psd.datname = pd.datname \"\n \"WHERE psd.datname not ilike 'template%%' \"\n \" AND psd.datname not ilike 'rdsadmin' \"\n \" AND psd.datname not ilike 'azure_maintenance' \",\n 'relation': False,\n }\n\n if not collect_default_db:\n res[\"query\"] += \" AND psd.datname not ilike 'postgres'\"\n\n return res"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the bgw metrics for the given key.", "response": "def _get_bgw_metrics(self, key, db):\n \"\"\"Use either COMMON_BGW_METRICS or COMMON_BGW_METRICS + NEWER_92_BGW_METRICS\n depending on the postgres version.\n Uses a dictionnary to save the result for each instance\n \"\"\"\n # Extended 9.2+ metrics if needed\n metrics = self.bgw_metrics.get(key)\n\n if metrics is None:\n # Hack to make sure that if we have multiple instances that connect to\n # the same host, port, we don't collect metrics twice\n # as it will result in https://github.com/DataDog/dd-agent/issues/1211\n sub_key = key[:2]\n if sub_key in self.db_bgw_metrics:\n self.bgw_metrics[key] = None\n self.log.debug(\n \"Not collecting bgw metrics for key: {0} as \"\n \"they are already collected by another instance\".format(key)\n )\n return None\n\n self.db_bgw_metrics.append(sub_key)\n\n self.bgw_metrics[key] = dict(self.COMMON_BGW_METRICS)\n if self._is_9_1_or_above(key, db):\n self.bgw_metrics[key].update(self.NEWER_91_BGW_METRICS)\n if self._is_9_2_or_above(key, db):\n self.bgw_metrics[key].update(self.NEWER_92_BGW_METRICS)\n\n metrics = self.bgw_metrics.get(key)\n\n if not metrics:\n return None\n\n return {'descriptors': [], 'metrics': metrics, 'query': \"select %s FROM pg_stat_bgwriter\", 'relation': False}"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_archiver_metrics(self, key, db):\n # While there's only one set for now, prepare for future additions to\n # the table, mirroring _get_bgw_metrics()\n metrics = self.archiver_metrics.get(key)\n\n if self._is_9_4_or_above(key, db) and metrics is None:\n # Collect from only one instance. See _get_bgw_metrics() for details on why.\n sub_key = key[:2]\n if sub_key in self.db_archiver_metrics:\n self.archiver_metrics[key] = None\n self.log.debug(\n \"Not collecting archiver metrics for key: {0} as \"\n \"they are already collected by another instance\".format(key)\n )\n return None\n\n self.db_archiver_metrics.append(sub_key)\n\n self.archiver_metrics[key] = dict(self.COMMON_ARCHIVER_METRICS)\n metrics = self.archiver_metrics.get(key)\n\n if not metrics:\n return None\n\n return {'descriptors': [], 'metrics': metrics, 'query': \"select %s FROM pg_stat_archiver\", 'relation': False}", "response": "Get the archiver metrics for the given key."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_replication_metrics(self, key, db):\n metrics = self.replication_metrics.get(key)\n if self._is_10_or_above(key, db) and metrics is None:\n self.replication_metrics[key] = dict(self.REPLICATION_METRICS_10)\n metrics = self.replication_metrics.get(key)\n elif self._is_9_1_or_above(key, db) and metrics is None:\n self.replication_metrics[key] = dict(self.REPLICATION_METRICS_9_1)\n if self._is_9_2_or_above(key, db):\n self.replication_metrics[key].update(self.REPLICATION_METRICS_9_2)\n metrics = self.replication_metrics.get(key)\n return metrics", "response": "Get the replication metrics for the given key."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_activity_metrics(self, key, db):\n metrics_data = self.activity_metrics.get(key)\n\n if metrics_data is None:\n query = self.ACTIVITY_QUERY_10 if self._is_10_or_above(key, db) else self.ACTIVITY_QUERY_LT_10\n metrics_query = None\n if self._is_9_6_or_above(key, db):\n metrics_query = self.ACTIVITY_METRICS_9_6\n elif self._is_9_2_or_above(key, db):\n metrics_query = self.ACTIVITY_METRICS_9_2\n elif self._is_8_3_or_above(key, db):\n metrics_query = self.ACTIVITY_METRICS_8_3\n else:\n metrics_query = self.ACTIVITY_METRICS_LT_8_3\n\n metrics = {k: v for k, v in zip(metrics_query, self.ACTIVITY_DD_METRICS)}\n self.activity_metrics[key] = (metrics, query)\n else:\n metrics, query = metrics_data\n\n return {'descriptors': [('datname', 'db')], 'metrics': metrics, 'query': query, 'relation': False}", "response": "Get the activity metrics for the specified instance."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nbuild a dictionary from relations configuration while maintaining compatibility", "response": "def _build_relations_config(self, yamlconfig):\n \"\"\"Builds a dictionary from relations configuration while maintaining compatibility\n \"\"\"\n config = {}\n for element in yamlconfig:\n if isinstance(element, str):\n config[element] = {'relation_name': element, 'schemas': []}\n elif isinstance(element, dict):\n if 'relation_name' not in element or 'schemas' not in element:\n self.log.warning(\"Unknown element format for relation element %s\", element)\n continue\n if not isinstance(element['schemas'], list):\n self.log.warning(\"Expected a list of schemas for %s\", element)\n continue\n name = element['relation_name']\n config[name] = {'relation_name': name, 'schemas': element['schemas']}\n else:\n self.log.warning('Unhandled relations config type: {}'.format(element))\n return config"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nqueries pg_stat_* for various metrics.", "response": "def _collect_stats(\n self,\n key,\n db,\n instance_tags,\n relations,\n custom_metrics,\n collect_function_metrics,\n collect_count_metrics,\n collect_activity_metrics,\n collect_database_size_metrics,\n collect_default_db,\n interface_error,\n programming_error,\n ):\n \"\"\"Query pg_stat_* for various metrics\n If relations is not an empty list, gather per-relation metrics\n on top of that.\n If custom_metrics is not an empty list, gather custom metrics defined in postgres.yaml\n \"\"\"\n\n db_instance_metrics = self._get_instance_metrics(key, db, collect_database_size_metrics, collect_default_db)\n bgw_instance_metrics = self._get_bgw_metrics(key, db)\n archiver_instance_metrics = self._get_archiver_metrics(key, db)\n\n metric_scope = [self.CONNECTION_METRICS, self.LOCK_METRICS]\n\n if collect_function_metrics:\n metric_scope.append(self.FUNCTION_METRICS)\n if collect_count_metrics:\n metric_scope.append(self.COUNT_METRICS)\n\n # Do we need relation-specific metrics?\n relations_config = {}\n if relations:\n metric_scope += [self.REL_METRICS, self.IDX_METRICS, self.SIZE_METRICS, self.STATIO_METRICS]\n relations_config = self._build_relations_config(relations)\n\n replication_metrics = self._get_replication_metrics(key, db)\n if replication_metrics is not None:\n # FIXME: constants shouldn't be modified\n self.REPLICATION_METRICS['metrics'] = replication_metrics\n metric_scope.append(self.REPLICATION_METRICS)\n\n try:\n cursor = db.cursor()\n results_len = self._query_scope(\n cursor, db_instance_metrics, key, db, instance_tags, False, programming_error, relations_config\n )\n if results_len is not None:\n self.gauge(\n \"postgresql.db.count\", results_len, tags=[t for t in instance_tags if not t.startswith(\"db:\")]\n )\n\n self._query_scope(\n cursor, bgw_instance_metrics, key, db, instance_tags, False, programming_error, relations_config\n )\n self._query_scope(\n cursor, archiver_instance_metrics, key, db, instance_tags, False, programming_error, relations_config\n )\n\n if collect_activity_metrics:\n activity_metrics = self._get_activity_metrics(key, db)\n self._query_scope(\n cursor, activity_metrics, key, db, instance_tags, False, programming_error, relations_config\n )\n\n for scope in list(metric_scope) + custom_metrics:\n self._query_scope(\n cursor, scope, key, db, instance_tags, scope in custom_metrics, programming_error, relations_config\n )\n\n cursor.close()\n except (interface_error, socket.error) as e:\n self.log.error(\"Connection error: %s\" % str(e))\n raise ShouldRestartException"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_connection(self, key, host, port, user, password, dbname, ssl, connect_fct, tags, use_cached=True):\n if key in self.dbs and use_cached:\n return self.dbs[key]\n\n elif host != \"\" and user != \"\":\n try:\n if host == 'localhost' and password == '':\n # Use ident method\n connection = connect_fct(\"user=%s dbname=%s\" % (user, dbname))\n elif port != '':\n connection = connect_fct(\n host=host, port=port, user=user, password=password, database=dbname, ssl=ssl\n )\n elif host.startswith('/'):\n # If the hostname starts with /, it's probably a path\n # to a UNIX socket. This is similar behaviour to psql\n connection = connect_fct(unix_sock=host, user=user, password=password, database=dbname)\n else:\n connection = connect_fct(host=host, user=user, password=password, database=dbname, ssl=ssl)\n self.dbs[key] = connection\n return connection\n except Exception as e:\n message = u'Error establishing postgres connection: %s' % (str(e))\n service_check_tags = self._get_service_check_tags(host, port, tags)\n self.service_check(\n self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=message\n )\n raise\n else:\n if not host:\n raise ConfigurationError('Please specify a Postgres host to connect to.')\n elif not user:\n raise ConfigurationError('Please specify a user to connect to Postgres as.')", "response": "Get and memoize connections to instances"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_custom_queries(self, db, tags, custom_queries, programming_error):\n for custom_query in custom_queries:\n metric_prefix = custom_query.get('metric_prefix')\n if not metric_prefix:\n self.log.error(\"custom query field `metric_prefix` is required\")\n continue\n metric_prefix = metric_prefix.rstrip('.')\n\n query = custom_query.get('query')\n if not query:\n self.log.error(\"custom query field `query` is required for metric_prefix `{}`\".format(metric_prefix))\n continue\n\n columns = custom_query.get('columns')\n if not columns:\n self.log.error(\"custom query field `columns` is required for metric_prefix `{}`\".format(metric_prefix))\n continue\n\n cursor = db.cursor()\n with closing(cursor) as cursor:\n try:\n self.log.debug(\"Running query: {}\".format(query))\n cursor.execute(query)\n except programming_error as e:\n self.log.error(\"Error executing query for metric_prefix {}: {}\".format(metric_prefix, str(e)))\n db.rollback()\n continue\n\n for row in cursor:\n if not row:\n self.log.debug(\n \"query result for metric_prefix {}: returned an empty result\".format(metric_prefix)\n )\n continue\n\n if len(columns) != len(row):\n self.log.error(\n \"query result for metric_prefix {}: expected {} columns, got {}\".format(\n metric_prefix, len(columns), len(row)\n )\n )\n continue\n\n metric_info = []\n query_tags = custom_query.get('tags', [])\n query_tags.extend(tags)\n\n for column, value in zip(columns, row):\n # Columns can be ignored via configuration.\n if not column:\n continue\n\n name = column.get('name')\n if not name:\n self.log.error(\n \"column field `name` is required for metric_prefix `{}`\".format(metric_prefix)\n )\n break\n\n column_type = column.get('type')\n if not column_type:\n self.log.error(\n \"column field `type` is required for column `{}` \"\n \"of metric_prefix `{}`\".format(name, metric_prefix)\n )\n break\n\n if column_type == 'tag':\n query_tags.append('{}:{}'.format(name, value))\n else:\n if not hasattr(self, column_type):\n self.log.error(\n \"invalid submission method `{}` for column `{}` of \"\n \"metric_prefix `{}`\".format(column_type, name, metric_prefix)\n )\n break\n try:\n metric_info.append(('{}.{}'.format(metric_prefix, name), float(value), column_type))\n except (ValueError, TypeError):\n self.log.error(\n \"non-numeric value `{}` for metric column `{}` of \"\n \"metric_prefix `{}`\".format(value, name, metric_prefix)\n )\n break\n\n # Only submit metrics if there were absolutely no errors - all or nothing.\n else:\n for info in metric_info:\n metric, value, method = info\n getattr(self, method)(metric, value, tags=query_tags)", "response": "Execute each query and parse the result for metrics"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_stats(self, stats, prefix, metric_categories, nested_tags, tags, recursion_level=0):\n for child in stats:\n if child.tag in metrics.METRIC_VALUE_FIELDS:\n self.submit_metrics(child, prefix, tags)\n elif child.tag in metrics.CATEGORY_FIELDS:\n recursion_tags = tags + [\"{}:{}\".format(nested_tags.get(prefix)[recursion_level], child.get('name'))]\n self.process_stats(child, prefix, metric_categories, nested_tags, recursion_tags, recursion_level + 1)", "response": "This function processes the XML stats and submits the metrics to the tags list."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts duration 'dur' to microseconds.", "response": "def _to_micros(dur):\n \"\"\"Convert duration 'dur' to microseconds.\"\"\"\n if hasattr(dur, 'total_seconds'):\n return int(dur.total_seconds() * 10e5)\n # Python 2.6\n return dur.microseconds + (dur.seconds + dur.days * 24 * 3600) * 1000000"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nregisters a global event listener.", "response": "def register(listener):\n \"\"\"Register a global event listener.\n\n :Parameters:\n - `listener`: A subclasses of :class:`CommandListener`,\n :class:`ServerHeartbeatListener`, :class:`ServerListener`, or\n :class:`TopologyListener`.\n \"\"\"\n if not isinstance(listener, _EventListener):\n raise TypeError(\"Listeners for %s must be either a \"\n \"CommandListener, ServerHeartbeatListener, \"\n \"ServerListener, or TopologyListener.\" % (listener,))\n if isinstance(listener, CommandListener):\n _LISTENERS.command_listeners.append(listener)\n if isinstance(listener, ServerHeartbeatListener):\n _LISTENERS.server_heartbeat_listeners.append(listener)\n if isinstance(listener, ServerListener):\n _LISTENERS.server_listeners.append(listener)\n if isinstance(listener, TopologyListener):\n _LISTENERS.topology_listeners.append(listener)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlisting of registered event listeners.", "response": "def event_listeners(self):\n \"\"\"List of registered event listeners.\"\"\"\n return (self.__command_listeners[:],\n self.__server_heartbeat_listeners[:],\n self.__server_listeners[:],\n self.__topology_listeners[:])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\npublishing a CommandStartedEvent to all command listeners.", "response": "def publish_command_start(self, command, database_name,\n request_id, connection_id, op_id=None):\n \"\"\"Publish a CommandStartedEvent to all command listeners.\n\n :Parameters:\n - `command`: The command document.\n - `database_name`: The name of the database this command was run\n against.\n - `request_id`: The request id for this operation.\n - `connection_id`: The address (host, port) of the server this\n command was sent to.\n - `op_id`: The (optional) operation id for this operation.\n \"\"\"\n if op_id is None:\n op_id = request_id\n event = CommandStartedEvent(\n command, database_name, request_id, connection_id, op_id)\n for subscriber in self.__command_listeners:\n try:\n subscriber.started(event)\n except Exception:\n _handle_exception()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npublishes a CommandSucceededEvent to all command listeners.", "response": "def publish_command_success(self, duration, reply, command_name,\n request_id, connection_id, op_id=None):\n \"\"\"Publish a CommandSucceededEvent to all command listeners.\n\n :Parameters:\n - `duration`: The command duration as a datetime.timedelta.\n - `reply`: The server reply document.\n - `command_name`: The command name.\n - `request_id`: The request id for this operation.\n - `connection_id`: The address (host, port) of the server this\n command was sent to.\n - `op_id`: The (optional) operation id for this operation.\n \"\"\"\n if op_id is None:\n op_id = request_id\n event = CommandSucceededEvent(\n duration, reply, command_name, request_id, connection_id, op_id)\n for subscriber in self.__command_listeners:\n try:\n subscriber.succeeded(event)\n except Exception:\n _handle_exception()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef publish_command_failure(self, duration, failure, command_name,\n request_id, connection_id, op_id=None):\n \"\"\"Publish a CommandFailedEvent to all command listeners.\n\n :Parameters:\n - `duration`: The command duration as a datetime.timedelta.\n - `failure`: The server reply document or failure description\n document.\n - `command_name`: The command name.\n - `request_id`: The request id for this operation.\n - `connection_id`: The address (host, port) of the server this\n command was sent to.\n - `op_id`: The (optional) operation id for this operation.\n \"\"\"\n if op_id is None:\n op_id = request_id\n event = CommandFailedEvent(\n duration, failure, command_name, request_id, connection_id, op_id)\n for subscriber in self.__command_listeners:\n try:\n subscriber.failed(event)\n except Exception:\n _handle_exception()", "response": "Publish a CommandFailedEvent to all command listeners."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef publish_server_heartbeat_started(self, connection_id):\n event = ServerHeartbeatStartedEvent(connection_id)\n for subscriber in self.__server_heartbeat_listeners:\n try:\n subscriber.started(event)\n except Exception:\n _handle_exception()", "response": "Publish a ServerHeartbeatStartedEvent to all server heartbeat listeners."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef publish_server_heartbeat_succeeded(self, connection_id, duration,\n reply):\n \"\"\"Publish a ServerHeartbeatSucceededEvent to all server heartbeat\n listeners.\n\n :Parameters:\n - `connection_id`: The address (host/port pair) of the connection.\n - `duration`: The execution time of the event in the highest possible\n resolution for the platform.\n - `reply`: The command reply.\n \"\"\"\n event = ServerHeartbeatSucceededEvent(duration, reply, connection_id)\n for subscriber in self.__server_heartbeat_listeners:\n try:\n subscriber.succeeded(event)\n except Exception:\n _handle_exception()", "response": "Publish a ServerHeartbeatSucceededEvent to all server heartbeat listeners."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef publish_server_heartbeat_failed(self, connection_id, duration, reply):\n event = ServerHeartbeatFailedEvent(duration, reply, connection_id)\n for subscriber in self.__server_heartbeat_listeners:\n try:\n subscriber.failed(event)\n except Exception:\n _handle_exception()", "response": "Publish a ServerHeartbeatFailedEvent to all server heartbeat listeners."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef publish_server_opened(self, server_address, topology_id):\n event = ServerOpeningEvent(server_address, topology_id)\n for subscriber in self.__server_listeners:\n try:\n subscriber.opened(event)\n except Exception:\n _handle_exception()", "response": "Publish a ServerOpeningEvent to all server listeners."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npublish a ServerClosedEvent to all server listeners.", "response": "def publish_server_closed(self, server_address, topology_id):\n \"\"\"Publish a ServerClosedEvent to all server listeners.\n\n :Parameters:\n - `server_address`: The address (host/port pair) of the server.\n - `topology_id`: A unique identifier for the topology this server\n is a part of.\n \"\"\"\n event = ServerClosedEvent(server_address, topology_id)\n for subscriber in self.__server_listeners:\n try:\n subscriber.closed(event)\n except Exception:\n _handle_exception()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef publish_server_description_changed(self, previous_description,\n new_description, server_address,\n topology_id):\n \"\"\"Publish a ServerDescriptionChangedEvent to all server listeners.\n\n :Parameters:\n - `previous_description`: The previous server description.\n - `server_address`: The address (host/port pair) of the server.\n - `new_description`: The new server description.\n - `topology_id`: A unique identifier for the topology this server\n is a part of.\n \"\"\"\n event = ServerDescriptionChangedEvent(previous_description,\n new_description, server_address,\n topology_id)\n for subscriber in self.__server_listeners:\n try:\n subscriber.description_changed(event)\n except Exception:\n _handle_exception()", "response": "Publish a ServerDescriptionChangedEvent to all server listeners."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef publish_topology_opened(self, topology_id):\n event = TopologyOpenedEvent(topology_id)\n for subscriber in self.__topology_listeners:\n try:\n subscriber.opened(event)\n except Exception:\n _handle_exception()", "response": "Publish a TopologyOpenedEvent to all topology listeners."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef publish_topology_closed(self, topology_id):\n event = TopologyClosedEvent(topology_id)\n for subscriber in self.__topology_listeners:\n try:\n subscriber.closed(event)\n except Exception:\n _handle_exception()", "response": "Publish a TopologyClosedEvent to all topology listeners."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npublish a TopologyDescriptionChangedEvent to all topology listeners.", "response": "def publish_topology_description_changed(self, previous_description,\n new_description, topology_id):\n \"\"\"Publish a TopologyDescriptionChangedEvent to all topology listeners.\n\n :Parameters:\n - `previous_description`: The previous topology description.\n - `new_description`: The new topology description.\n - `topology_id`: A unique identifier for the topology this server\n is a part of.\n \"\"\"\n event = TopologyDescriptionChangedEvent(previous_description,\n new_description, topology_id)\n for subscriber in self.__topology_listeners:\n try:\n subscriber.description_changed(event)\n except Exception:\n _handle_exception()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget statistics on the postqueue.", "response": "def _get_postqueue_stats(self, postfix_config_dir, tags):\n\n # get some intersting configuratin values from postconf\n pc_output, _, _ = get_subprocess_output(['postconf', 'mail_version'], self.log, False)\n postfix_version = pc_output.strip('\\n').split('=')[1].strip()\n pc_output, _, _ = get_subprocess_output(['postconf', 'authorized_mailq_users'], self.log, False)\n authorized_mailq_users = pc_output.strip('\\n').split('=')[1].strip()\n\n self.log.debug('authorized_mailq_users : {}'.format(authorized_mailq_users))\n\n output, _, _ = get_subprocess_output(['postqueue', '-c', postfix_config_dir, '-p'], self.log, False)\n\n active_count = 0\n hold_count = 0\n deferred_count = 0\n\n # postque -p sample output\n '''\n root@postfix:/opt/datadog-agent/agent/checks.d# postqueue -p\n ----Queue ID----- --Size-- ---Arrival Time---- --Sender/Recipient------\n 3xWyLP6Nmfz23fk 367 Tue Aug 15 16:17:33 root@postfix.devnull.home\n (deferred transport)\n alice@crypto.io\n\n 3xWyD86NwZz23ff! 358 Tue Aug 15 16:12:08 root@postfix.devnull.home\n (deferred transport)\n bob@crypto.io\n\n -- 1 Kbytes in 2 Requests.\n '''\n\n for line in output.splitlines():\n if '*' in line:\n active_count += 1\n continue\n if '!' in line:\n hold_count += 1\n continue\n if line[0:1].isdigit():\n deferred_count += 1\n\n self.log.debug('Postfix Version: %s' % postfix_version)\n\n self.gauge(\n 'postfix.queue.size', active_count, tags=tags + ['queue:active', 'instance:{}'.format(postfix_config_dir)]\n )\n self.gauge(\n 'postfix.queue.size', hold_count, tags=tags + ['queue:hold', 'instance:{}'.format(postfix_config_dir)]\n )\n self.gauge(\n 'postfix.queue.size',\n deferred_count,\n tags=tags + ['queue:deferred', 'instance:{}'.format(postfix_config_dir)],\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef integrations(since, to, write, force):\n agent_tags = get_agent_tags(since, to)\n # get the list of integrations shipped with the agent from the requirements file\n req_file_name = os.path.basename(get_agent_release_requirements())\n\n integrations_contents = StringIO()\n for tag in agent_tags:\n integrations_contents.write('## Datadog Agent version {}\\n\\n'.format(tag))\n # Requirements for current tag\n file_contents = git_show_file(req_file_name, tag)\n for name, ver in iteritems(parse_agent_req_file(file_contents)):\n integrations_contents.write('* {}: {}\\n'.format(name, ver))\n integrations_contents.write('\\n')\n\n # save the changelog on disk if --write was passed\n if write:\n dest = get_agent_integrations_file()\n # don't overwrite an existing file\n if os.path.exists(dest) and not force:\n msg = \"Output file {} already exists, run the command again with --force to overwrite\"\n abort(msg.format(dest))\n\n write_file(dest, integrations_contents.getvalue())\n else:\n echo_info(integrations_contents.getvalue())", "response": "Generates a markdown file containing the list of integrations shipped with the Agent release."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _collect_stats(self, db, instance_tags):\n\n metric_scope = [self.STATS_METRICS, self.POOLS_METRICS, self.DATABASES_METRICS]\n\n try:\n with db.cursor(cursor_factory=pgextras.DictCursor) as cursor:\n for scope in metric_scope:\n descriptors = scope['descriptors']\n metrics = scope['metrics']\n query = scope['query']\n\n try:\n self.log.debug(\"Running query: %s\", query)\n cursor.execute(query)\n\n rows = cursor.fetchall()\n\n except pg.Error:\n self.log.exception(\"Not all metrics may be available\")\n\n else:\n for row in rows:\n self.log.debug(\"Processing row: %r\", row)\n\n # Skip the \"pgbouncer\" database\n if row['database'] == self.DB_NAME:\n continue\n\n tags = list(instance_tags)\n tags += [\"%s:%s\" % (tag, row[column]) for (column, tag) in descriptors if column in row]\n for (column, (name, reporter)) in metrics:\n if column in row:\n reporter(self, name, row[column], tags)\n\n if not rows:\n self.log.warning(\"No results were found for query: %s\", query)\n\n except pg.Error:\n self.log.exception(\"Connection error\")\n\n raise ShouldRestartException", "response": "Query pgbouncer for various metrics and collect stats"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_connect_kwargs(self, host, port, user, password, database_url):\n if database_url:\n return {'dsn': database_url}\n\n if not host:\n raise CheckException(\"Please specify a PgBouncer host to connect to.\")\n\n if not user:\n raise CheckException(\"Please specify a user to connect to PgBouncer as.\")\n\n if host in ('localhost', '127.0.0.1') and password == '':\n # Use ident method\n return {'dsn': \"user={} dbname={}\".format(user, self.DB_NAME)}\n\n if port:\n return {'host': host, 'user': user, 'password': password, 'database': self.DB_NAME, 'port': port}\n\n return {'host': host, 'user': user, 'password': password, 'database': self.DB_NAME}", "response": "Get the params to pass to psycopg2. connect"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_connection(self, key, host='', port='', user='', password='', database_url='', tags=None, use_cached=True):\n \"Get and memoize connections to instances\"\n if key in self.dbs and use_cached:\n return self.dbs[key]\n try:\n connect_kwargs = self._get_connect_kwargs(\n host=host, port=port, user=user, password=password, database_url=database_url\n )\n\n connection = pg.connect(**connect_kwargs)\n connection.set_isolation_level(pg.extensions.ISOLATION_LEVEL_AUTOCOMMIT)\n\n # re-raise the CheckExceptions raised by _get_connect_kwargs()\n except CheckException:\n raise\n\n except Exception:\n redacted_url = self._get_redacted_dsn(host, port, user, database_url)\n message = u'Cannot establish connection to {}'.format(redacted_url)\n\n self.service_check(\n self.SERVICE_CHECK_NAME,\n AgentCheck.CRITICAL,\n tags=self._get_service_checks_tags(host, port, database_url, tags),\n message=message,\n )\n raise\n\n self.dbs[key] = connection\n return connection", "response": "Get and memoize connections to instances"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef seconds_until_renew(self):\n delta = self.renew_time - datetime.now(self.renew_time.tzinfo)\n return delta.total_seconds()", "response": "Returns the number of seconds until the current time is set to the set s renew time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate and return an error document.", "response": "def _make_error(index, code, errmsg, operation):\n \"\"\"Create and return an error document.\n \"\"\"\n return {\n _UINDEX: index,\n _UCODE: code,\n _UERRMSG: errmsg,\n _UOP: operation\n }"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _merge_legacy(run, full_result, result, index):\n affected = result.get('n', 0)\n\n errmsg = result.get(\"errmsg\", result.get(\"err\", \"\"))\n if errmsg:\n # wtimeout is not considered a hard failure in\n # MongoDB 2.6 so don't treat it like one here.\n if result.get(\"wtimeout\"):\n error_doc = {'errmsg': errmsg, 'code': _WRITE_CONCERN_ERROR}\n full_result['writeConcernErrors'].append(error_doc)\n else:\n code = result.get(\"code\", _UNKNOWN_ERROR)\n error = _make_error(run.index(index), code, errmsg, run.ops[index])\n if \"errInfo\" in result:\n error[\"errInfo\"] = result[\"errInfo\"]\n full_result[\"writeErrors\"].append(error)\n return\n if run.op_type == _INSERT:\n full_result['nInserted'] += 1\n elif run.op_type == _UPDATE:\n if \"upserted\" in result:\n doc = {_UINDEX: run.index(index), _UID: result[\"upserted\"]}\n full_result[\"upserted\"].append(doc)\n full_result['nUpserted'] += affected\n # Versions of MongoDB before 2.6 don't return the _id for an\n # upsert if _id is not an ObjectId.\n elif result.get(\"updatedExisting\") is False and affected == 1:\n op = run.ops[index]\n # If _id is in both the update document *and* the query spec\n # the update document _id takes precedence.\n _id = op['u'].get('_id', op['q'].get('_id'))\n doc = {_UINDEX: run.index(index), _UID: _id}\n full_result[\"upserted\"].append(doc)\n full_result['nUpserted'] += affected\n else:\n full_result['nMatched'] += affected\n\n elif run.op_type == _DELETE:\n full_result['nRemoved'] += affected", "response": "Merge a result from a legacy opcode into the full result."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _merge_command(run, full_result, results):\n for offset, result in results:\n\n affected = result.get(\"n\", 0)\n\n if run.op_type == _INSERT:\n full_result[\"nInserted\"] += affected\n\n elif run.op_type == _DELETE:\n full_result[\"nRemoved\"] += affected\n\n elif run.op_type == _UPDATE:\n upserted = result.get(\"upserted\")\n if upserted:\n if isinstance(upserted, list):\n n_upserted = len(upserted)\n for doc in upserted:\n doc[\"index\"] = run.index(doc[\"index\"] + offset)\n full_result[\"upserted\"].extend(upserted)\n else:\n n_upserted = 1\n index = run.index(offset)\n doc = {_UINDEX: index, _UID: upserted}\n full_result[\"upserted\"].append(doc)\n full_result[\"nUpserted\"] += n_upserted\n full_result[\"nMatched\"] += (affected - n_upserted)\n else:\n full_result[\"nMatched\"] += affected\n n_modified = result.get(\"nModified\")\n # SERVER-13001 - in a mixed sharded cluster a call to\n # update could return nModified (>= 2.6) or not (<= 2.4).\n # If any call does not return nModified we can't report\n # a valid final count so omit the field completely.\n if n_modified is not None and \"nModified\" in full_result:\n full_result[\"nModified\"] += n_modified\n else:\n full_result.pop(\"nModified\", None)\n\n write_errors = result.get(\"writeErrors\")\n if write_errors:\n for doc in write_errors:\n # Leave the server response intact for APM.\n replacement = doc.copy()\n idx = doc[\"index\"] + offset\n replacement[\"index\"] = run.index(idx)\n # Add the failed operation to the error document.\n replacement[_UOP] = run.ops[idx]\n full_result[\"writeErrors\"].append(replacement)\n\n wc_error = result.get(\"writeConcernError\")\n if wc_error:\n full_result[\"writeConcernErrors\"].append(wc_error)", "response": "Merge a group of results from write commands into the full result."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd an operation to this Run instance.", "response": "def add(self, original_index, operation):\n \"\"\"Add an operation to this Run instance.\n\n :Parameters:\n - `original_index`: The original index of this operation\n within a larger bulk operation.\n - `operation`: The operation document.\n \"\"\"\n self.index_map.append(original_index)\n self.ops.append(operation)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds an insert document to the list of ops.", "response": "def add_insert(self, document):\n \"\"\"Add an insert document to the list of ops.\n \"\"\"\n validate_is_document_type(\"document\", document)\n # Generate ObjectId client side.\n if not (isinstance(document, RawBSONDocument) or '_id' in document):\n document['_id'] = ObjectId()\n self.ops.append((_INSERT, document))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_replace(self, selector, replacement, upsert=False,\n collation=None):\n \"\"\"Create a replace document and add it to the list of ops.\n \"\"\"\n validate_ok_for_replace(replacement)\n cmd = SON([('q', selector), ('u', replacement),\n ('multi', False), ('upsert', upsert)])\n collation = validate_collation_or_none(collation)\n if collation is not None:\n self.uses_collation = True\n cmd['collation'] = collation\n self.ops.append((_UPDATE, cmd))", "response": "Create a replace document and add it to the list of ops."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a delete document and add it to the list of ops.", "response": "def add_delete(self, selector, limit, collation=None):\n \"\"\"Create a delete document and add it to the list of ops.\n \"\"\"\n cmd = SON([('q', selector), ('limit', limit)])\n collation = validate_collation_or_none(collation)\n if collation is not None:\n self.uses_collation = True\n cmd['collation'] = collation\n self.ops.append((_DELETE, cmd))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gen_ordered(self):\n run = None\n for idx, (op_type, operation) in enumerate(self.ops):\n if run is None:\n run = _Run(op_type)\n elif run.op_type != op_type:\n yield run\n run = _Run(op_type)\n run.add(idx, operation)\n yield run", "response": "Generate batches of operations in the order provided."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates batches of operations in arbitrary order.", "response": "def gen_unordered(self):\n \"\"\"Generate batches of operations, batched by type of\n operation, in arbitrary order.\n \"\"\"\n operations = [_Run(_INSERT), _Run(_UPDATE), _Run(_DELETE)]\n for idx, (op_type, operation) in enumerate(self.ops):\n operations[op_type].add(idx, operation)\n\n for run in operations:\n if run.ops:\n yield run"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef execute_command(self, sock_info, generator, write_concern):\n # nModified is only reported for write commands, not legacy ops.\n full_result = {\n \"writeErrors\": [],\n \"writeConcernErrors\": [],\n \"nInserted\": 0,\n \"nUpserted\": 0,\n \"nMatched\": 0,\n \"nModified\": 0,\n \"nRemoved\": 0,\n \"upserted\": [],\n }\n op_id = _randint()\n db_name = self.collection.database.name\n listeners = self.collection.database.client._event_listeners\n\n for run in generator:\n cmd = SON([(_COMMANDS[run.op_type], self.collection.name),\n ('ordered', self.ordered)])\n if write_concern.document:\n cmd['writeConcern'] = write_concern.document\n if self.bypass_doc_val and sock_info.max_wire_version >= 4:\n cmd['bypassDocumentValidation'] = True\n\n bwc = _BulkWriteContext(db_name, cmd, sock_info, op_id, listeners)\n results = _do_batched_write_command(\n self.namespace, run.op_type, cmd,\n run.ops, True, self.collection.codec_options, bwc)\n\n _merge_command(run, full_result, results)\n # We're supposed to continue if errors are\n # at the write concern level (e.g. wtimeout)\n if self.ordered and full_result['writeErrors']:\n break\n\n if full_result[\"writeErrors\"] or full_result[\"writeConcernErrors\"]:\n if full_result['writeErrors']:\n full_result['writeErrors'].sort(\n key=lambda error: error['index'])\n raise BulkWriteError(full_result)\n return full_result", "response": "Execute a list of bulk write commands."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef execute_no_results(self, sock_info, generator):\n # Cannot have both unacknowledged write and bypass document validation.\n if self.bypass_doc_val and sock_info.max_wire_version >= 4:\n raise OperationFailure(\"Cannot set bypass_document_validation with\"\n \" unacknowledged write concern\")\n coll = self.collection\n # If ordered is True we have to send GLE or use write\n # commands so we can abort on the first error.\n write_concern = WriteConcern(w=int(self.ordered))\n op_id = _randint()\n\n for run in generator:\n try:\n if run.op_type == _INSERT:\n coll._insert(\n sock_info,\n run.ops,\n self.ordered,\n write_concern=write_concern,\n op_id=op_id,\n bypass_doc_val=self.bypass_doc_val)\n elif run.op_type == _UPDATE:\n for operation in run.ops:\n doc = operation['u']\n check_keys = True\n if doc and next(iter(doc)).startswith('$'):\n check_keys = False\n coll._update(\n sock_info,\n operation['q'],\n doc,\n operation['upsert'],\n check_keys,\n operation['multi'],\n write_concern=write_concern,\n op_id=op_id,\n ordered=self.ordered,\n bypass_doc_val=self.bypass_doc_val)\n else:\n for operation in run.ops:\n coll._delete(sock_info,\n operation['q'],\n not operation['limit'],\n write_concern,\n op_id,\n self.ordered)\n except OperationFailure:\n if self.ordered:\n break", "response": "Execute all operations returning no results."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef execute_legacy(self, sock_info, generator, write_concern):\n coll = self.collection\n full_result = {\n \"writeErrors\": [],\n \"writeConcernErrors\": [],\n \"nInserted\": 0,\n \"nUpserted\": 0,\n \"nMatched\": 0,\n \"nRemoved\": 0,\n \"upserted\": [],\n }\n op_id = _randint()\n stop = False\n for run in generator:\n for idx, operation in enumerate(run.ops):\n try:\n # To do per-operation reporting we have to do ops one\n # at a time. That means the performance of bulk insert\n # will be slower here than calling Collection.insert()\n if run.op_type == _INSERT:\n coll._insert(sock_info,\n operation,\n self.ordered,\n write_concern=write_concern,\n op_id=op_id)\n result = {}\n elif run.op_type == _UPDATE:\n doc = operation['u']\n check_keys = True\n if doc and next(iter(doc)).startswith('$'):\n check_keys = False\n result = coll._update(sock_info,\n operation['q'],\n doc,\n operation['upsert'],\n check_keys,\n operation['multi'],\n write_concern=write_concern,\n op_id=op_id,\n ordered=self.ordered)\n else:\n result = coll._delete(sock_info,\n operation['q'],\n not operation['limit'],\n write_concern,\n op_id,\n self.ordered)\n _merge_legacy(run, full_result, result, idx)\n except DocumentTooLarge as exc:\n # MongoDB 2.6 uses error code 2 for \"too large\".\n error = _make_error(\n run.index(idx), _BAD_VALUE, str(exc), operation)\n full_result['writeErrors'].append(error)\n if self.ordered:\n stop = True\n break\n except OperationFailure as exc:\n if not exc.details:\n # Some error not related to the write operation\n # (e.g. kerberos failure). Re-raise immediately.\n raise\n _merge_legacy(run, full_result, exc.details, idx)\n # We're supposed to continue if errors are\n # at the write concern level (e.g. wtimeout)\n if self.ordered and full_result[\"writeErrors\"]:\n stop = True\n break\n if stop:\n break\n\n if full_result[\"writeErrors\"] or full_result['writeConcernErrors']:\n if full_result['writeErrors']:\n full_result['writeErrors'].sort(\n key=lambda error: error['index'])\n raise BulkWriteError(full_result)\n return full_result", "response": "Execute legacy wire protocol ops."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef execute(self, write_concern):\n if not self.ops:\n raise InvalidOperation('No operations to execute')\n if self.executed:\n raise InvalidOperation('Bulk operations can '\n 'only be executed once.')\n self.executed = True\n write_concern = (WriteConcern(**write_concern) if\n write_concern else self.collection.write_concern)\n\n if self.ordered:\n generator = self.gen_ordered()\n else:\n generator = self.gen_unordered()\n\n client = self.collection.database.client\n with client._socket_for_writes() as sock_info:\n if sock_info.max_wire_version < 5 and self.uses_collation:\n raise ConfigurationError(\n 'Must be connected to MongoDB 3.4+ to use a collation.')\n if not write_concern.acknowledged:\n if self.uses_collation:\n raise ConfigurationError(\n 'Collation is unsupported for unacknowledged writes.')\n self.execute_no_results(sock_info, generator)\n elif sock_info.max_wire_version > 1:\n return self.execute_command(sock_info, generator, write_concern)\n else:\n return self.execute_legacy(sock_info, generator, write_concern)", "response": "Execute the bulk operation."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_one(self, update):\n self.__bulk.add_update(self.__selector,\n update, multi=False, upsert=True,\n collation=self.__collation)", "response": "Update one document matching the selector."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreplacing one entire document matching the selector criteria.", "response": "def replace_one(self, replacement):\n \"\"\"Replace one entire document matching the selector criteria.\n\n :Parameters:\n - `replacement` (dict): the replacement document\n \"\"\"\n self.__bulk.add_replace(self.__selector, replacement, upsert=True,\n collation=self.__collation)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nremove a single document matching the selector criteria.", "response": "def remove_one(self):\n \"\"\"Remove a single document matching the selector criteria.\n \"\"\"\n self.__bulk.add_delete(self.__selector, _DELETE_ONE,\n collation=self.__collation)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef remove(self):\n self.__bulk.add_delete(self.__selector, _DELETE_ALL,\n collation=self.__collation)", "response": "Remove all documents matching the selector criteria."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a new BulkWriteOperation instance that will be executed by the specified selector.", "response": "def find(self, selector, collation=None):\n \"\"\"Specify selection criteria for bulk operations.\n\n :Parameters:\n - `selector` (dict): the selection criteria for update\n and remove operations.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`. This option is only\n supported on MongoDB 3.4 and above.\n\n :Returns:\n - A :class:`BulkWriteOperation` instance, used to add\n update and remove operations to this bulk operation.\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n\n \"\"\"\n validate_is_mapping(\"selector\", selector)\n return BulkWriteOperation(selector, self.__bulk, collation)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes all provided operations.", "response": "def execute(self, write_concern=None):\n \"\"\"Execute all provided operations.\n\n :Parameters:\n - write_concern (optional): the write concern for this bulk\n execution.\n \"\"\"\n if write_concern is not None:\n validate_is_mapping(\"write_concern\", write_concern)\n return self.__bulk.execute(write_concern)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef select_servers(self,\n selector,\n server_selection_timeout=None,\n address=None):\n \"\"\"Return a list of Servers matching selector, or time out.\n\n :Parameters:\n - `selector`: function that takes a list of Servers and returns\n a subset of them.\n - `server_selection_timeout` (optional): maximum seconds to wait.\n If not provided, the default value common.SERVER_SELECTION_TIMEOUT\n is used.\n - `address`: optional server address to select.\n\n Calls self.open() if needed.\n\n Raises exc:`ServerSelectionTimeoutError` after\n `server_selection_timeout` if no matching servers are found.\n \"\"\"\n if server_selection_timeout is None:\n server_timeout = self._settings.server_selection_timeout\n else:\n server_timeout = server_selection_timeout\n\n with self._lock:\n self._description.check_compatible()\n\n now = _time()\n end_time = now + server_timeout\n server_descriptions = self._description.apply_selector(\n selector, address)\n\n while not server_descriptions:\n # No suitable servers.\n if server_timeout == 0 or now > end_time:\n raise ServerSelectionTimeoutError(\n self._error_message(selector))\n\n self._ensure_opened()\n self._request_check_all()\n\n # Release the lock and wait for the topology description to\n # change, or for a timeout. We won't miss any changes that\n # came after our most recent apply_selector call, since we've\n # held the lock until now.\n self._condition.wait(common.MIN_HEARTBEAT_INTERVAL)\n self._description.check_compatible()\n now = _time()\n server_descriptions = self._description.apply_selector(\n selector, address)\n\n return [self.get_server_by_address(sd.address)\n for sd in server_descriptions]", "response": "Returns a list of Servers matching selector or time out."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlike select_servers but choose a random server if several match.", "response": "def select_server(self,\n selector,\n server_selection_timeout=None,\n address=None):\n \"\"\"Like select_servers, but choose a random server if several match.\"\"\"\n return random.choice(self.select_servers(selector,\n server_selection_timeout,\n address))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprocesses a new ServerDescription after an ismaster call completes.", "response": "def on_change(self, server_description):\n \"\"\"Process a new ServerDescription after an ismaster call completes.\"\"\"\n # We do no I/O holding the lock.\n with self._lock:\n # Any monitored server was definitely in the topology description\n # once. Check if it's still in the description or if some state-\n # change removed it. E.g., we got a host list from the primary\n # that didn't include this server.\n if self._description.has_server(server_description.address):\n td_old = self._description\n if self._publish_server:\n old_server_description = td_old._server_descriptions[\n server_description.address]\n self._events.put((\n self._listeners.publish_server_description_changed,\n (old_server_description, server_description,\n server_description.address, self._topology_id)))\n\n self._description = updated_topology_description(\n self._description, server_description)\n\n self._update_servers()\n\n if self._publish_tp:\n self._events.put((\n self._listeners.publish_topology_description_changed,\n (td_old, self._description, self._topology_id)))\n\n # Wake waiters in select_servers().\n self._condition.notify_all()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn primary s address or None.", "response": "def get_primary(self):\n \"\"\"Return primary's address or None.\"\"\"\n # Implemented here in Topology instead of MongoClient, so it can lock.\n with self._lock:\n topology_type = self._description.topology_type\n if topology_type != TOPOLOGY_TYPE.ReplicaSetWithPrimary:\n return None\n\n return writable_server_selector(self._new_selection())[0].address"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_replica_set_members(self, selector):\n # Implemented here in Topology instead of MongoClient, so it can lock.\n with self._lock:\n topology_type = self._description.topology_type\n if topology_type not in (TOPOLOGY_TYPE.ReplicaSetWithPrimary,\n TOPOLOGY_TYPE.ReplicaSetNoPrimary):\n return set()\n\n return set([sd.address for sd in selector(self._new_selection())])", "response": "Return set of replica set member addresses."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwakes all monitors wait for at least one to check its server.", "response": "def request_check_all(self, wait_time=5):\n \"\"\"Wake all monitors, wait for at least one to check its server.\"\"\"\n with self._lock:\n self._request_check_all()\n self._condition.wait(wait_time)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nresets the server and request check.", "response": "def reset_server_and_request_check(self, address):\n \"\"\"Clear our pool for a server, mark it Unknown, and check it soon.\"\"\"\n with self._lock:\n self._reset_server(address)\n self._request_check(address)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nclearing pools and terminate monitors. Topology reopens on demand.", "response": "def close(self):\n \"\"\"Clear pools and terminate monitors. Topology reopens on demand.\"\"\"\n with self._lock:\n for server in self._servers.values():\n server.close()\n\n # Mark all servers Unknown.\n self._description = self._description.reset()\n self._update_servers()\n self._opened = False\n\n # Publish only after releasing the lock.\n if self._publish_tp:\n self._events.put((self._listeners.publish_topology_closed,\n (self._topology_id,)))\n if self._publish_server or self._publish_tp:\n self.__events_executor.close()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _reset_server(self, address):\n server = self._servers.get(address)\n\n # \"server\" is None if another thread removed it from the topology.\n if server:\n server.reset()\n\n # Mark this server Unknown.\n self._description = self._description.reset_server(address)\n self._update_servers()", "response": "Clear our pool for a server and mark it Unknown."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwaking one monitor. Hold the lock when calling this.", "response": "def _request_check(self, address):\n \"\"\"Wake one monitor. Hold the lock when calling this.\"\"\"\n server = self._servers.get(address)\n\n # \"server\" is None if another thread removed it from the topology.\n if server:\n server.request_check()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _error_message(self, selector):\n is_replica_set = self._description.topology_type in (\n TOPOLOGY_TYPE.ReplicaSetWithPrimary,\n TOPOLOGY_TYPE.ReplicaSetNoPrimary)\n\n if is_replica_set:\n server_plural = 'replica set members'\n elif self._description.topology_type == TOPOLOGY_TYPE.Sharded:\n server_plural = 'mongoses'\n else:\n server_plural = 'servers'\n\n if self._description.known_servers:\n # We've connected, but no servers match the selector.\n if selector is writable_server_selector:\n if is_replica_set:\n return 'No primary available for writes'\n else:\n return 'No %s available for writes' % server_plural\n else:\n return 'No %s match selector \"%s\"' % (server_plural, selector)\n else:\n addresses = list(self._description.server_descriptions())\n servers = list(self._description.server_descriptions().values())\n if not servers:\n if is_replica_set:\n # We removed all servers because of the wrong setName?\n return 'No %s available for replica set name \"%s\"' % (\n server_plural, self._settings.replica_set_name)\n else:\n return 'No %s available' % server_plural\n\n # 1 or more servers, all Unknown. Are they unknown for one reason?\n error = servers[0].error\n same = all(server.error == error for server in servers[1:])\n if same:\n if error is None:\n # We're still discovering.\n return 'No %s found yet' % server_plural\n\n if (is_replica_set and not\n set(addresses).intersection(self._seed_addresses)):\n # We replaced our seeds with new hosts but can't reach any.\n return (\n 'Could not reach any servers in %s. Replica set is'\n ' configured with internal hostnames or IPs?' %\n addresses)\n\n return str(error)\n else:\n return ','.join(str(server.error) for server in servers\n if server.error)", "response": "Format an error message for the server selection fails."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an empty instance if it doesn t exist.", "response": "def init_instance(self, key):\n \"\"\"\n Create an empty instance if it doesn't exist.\n If the instance already exists, this is a noop.\n \"\"\"\n with self._mor_lock:\n if key not in self._mor:\n self._mor[key] = {}"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nstores a Mor object in the cache with the given name.", "response": "def set_mor(self, key, name, mor):\n \"\"\"\n Store a Mor object in the cache with the given name.\n If the key is not in the cache, raises a KeyError.\n \"\"\"\n with self._mor_lock:\n self._mor[key][name] = mor\n self._mor[key][name]['creation_time'] = time.time()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_mor(self, key, name):\n with self._mor_lock:\n mors = self._mor[key]\n try:\n return mors[name]\n except KeyError:\n raise MorNotFoundError(\"Mor object '{}' is not in the cache.\".format(name))", "response": "Returns the Mor object identified by name for the given instance key."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the list of metric identifiers for the given instance key and Mor object name.", "response": "def set_metrics(self, key, name, metrics):\n \"\"\"\n Store a list of metric identifiers for the given instance key and Mor\n object name.\n If the key is not in the cache, raises a KeyError.\n If the Mor object is not in the cache, raises a MorNotFoundError\n \"\"\"\n with self._mor_lock:\n mor = self._mor[key].get(name)\n if mor is None:\n raise MorNotFoundError(\"Mor object '{}' is not in the cache.\".format(name))\n mor['metrics'] = metrics"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves all the items in the cache for the given key that are older than ttl seconds.", "response": "def purge(self, key, ttl):\n \"\"\"\n Remove all the items in the cache for the given key that are older than\n ttl seconds.\n If the key is not in the cache, raises a KeyError.\n \"\"\"\n mors_to_purge = []\n now = time.time()\n with self._mor_lock:\n # Don't change the dict during iteration!\n # First collect the names of the Mors to remove...\n for name, mor in iteritems(self._mor[key]):\n age = now - mor['creation_time']\n if age > ttl:\n mors_to_purge.append(name)\n\n # ...then actually remove the Mors from the cache.\n for name in mors_to_purge:\n del self._mor[key][name]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntakes a metric formatted by Envoy and splits it into a unique taxonomy and returns the unique taxonomy name a list of tags and a unique taxonomy name. Returns the unique taxonomy name a list of tags and a unique taxonomy name.", "response": "def parse_metric(metric, metric_mapping=METRIC_TREE):\n \"\"\"Takes a metric formatted by Envoy and splits it into a unique\n metric name. Returns the unique metric name, a list of tags, and\n the name of the submission method.\n\n Example:\n 'listener.0.0.0.0_80.downstream_cx_total' ->\n ('listener.downstream_cx_total', ['address:0.0.0.0_80'], 'count')\n \"\"\"\n metric_parts = []\n tag_names = []\n tag_values = []\n tag_builder = []\n unknown_tags = []\n num_tags = 0\n minimum_tag_length = 0\n\n for metric_part in metric.split('.'):\n if metric_part in metric_mapping and num_tags >= minimum_tag_length:\n # Rebuild any built up tags whenever we encounter a known metric part.\n if tag_builder:\n for tags in metric_mapping['|_tags_|']:\n if num_tags >= len(tags):\n break\n\n constructed_tags = construct_tags(tag_builder, len(tags))\n\n if tags:\n tag_names.extend(tags)\n tag_values.extend(constructed_tags)\n else:\n unknown_tags.extend(constructed_tags)\n\n num_tags = 0\n\n metric_parts.append(metric_part)\n metric_mapping = metric_mapping[metric_part]\n minimum_tag_length = len(metric_mapping['|_tags_|'][-1])\n else:\n tag_builder.append(metric_part)\n num_tags += 1\n\n metric = '.'.join(metric_parts)\n if metric not in METRICS:\n raise UnknownMetric\n\n # Rebuild any trailing tags\n if tag_builder:\n for tags in metric_mapping['|_tags_|']:\n if num_tags >= len(tags):\n break\n\n constructed_tags = construct_tags(tag_builder, len(tags))\n\n if tags:\n tag_names.extend(tags)\n tag_values.extend(constructed_tags)\n else:\n unknown_tags.extend(constructed_tags)\n\n if unknown_tags:\n raise UnknownTags('{}'.format('|||'.join(unknown_tags)))\n\n tags = ['{}:{}'.format(tag_name, tag_value) for tag_name, tag_value in zip(tag_names, tag_values)]\n\n return METRIC_PREFIX + metric, tags, METRICS[metric]['method']"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\niterate over histogram data yielding metric - value pairs.", "response": "def parse_histogram(metric, histogram):\n \"\"\"Iterates over histogram data, yielding metric-value pairs.\"\"\"\n for match in HISTOGRAM.finditer(histogram):\n percentile, value = match.groups()\n value = float(value)\n\n if not isnan(value):\n try:\n yield metric + PERCENTILE_SUFFIX[percentile], value\n\n # In case Envoy adds more\n except KeyError:\n yield '{}.{}percentile'.format(metric, percentile[1:].replace('.', '_')), value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nputting data in GridFS as a new file.", "response": "def put(self, data, **kwargs):\n \"\"\"Put data in GridFS as a new file.\n\n Equivalent to doing::\n\n try:\n f = new_file(**kwargs)\n f.write(data)\n finally:\n f.close()\n\n `data` can be either an instance of :class:`str` (:class:`bytes`\n in python 3) or a file-like object providing a :meth:`read` method.\n If an `encoding` keyword argument is passed, `data` can also be a\n :class:`unicode` (:class:`str` in python 3) instance, which will\n be encoded as `encoding` before being written. Any keyword arguments\n will be passed through to the created file - see\n :meth:`~gridfs.grid_file.GridIn` for possible arguments. Returns the\n ``\"_id\"`` of the created file.\n\n If the ``\"_id\"`` of the file is manually specified, it must\n not already exist in GridFS. Otherwise\n :class:`~gridfs.errors.FileExists` is raised.\n\n :Parameters:\n - `data`: data to be written as a file.\n - `**kwargs` (optional): keyword arguments for file creation\n\n .. versionchanged:: 3.0\n w=0 writes to GridFS are now prohibited.\n \"\"\"\n grid_file = GridIn(self.__collection, **kwargs)\n try:\n grid_file.write(data)\n finally:\n grid_file.close()\n\n return grid_file._id"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef delete(self, file_id):\n self.__files.delete_one({\"_id\": file_id})\n self.__chunks.delete_many({\"files_id\": file_id})", "response": "Delete a file from GridFS by _id."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef find_one(self, filter=None, *args, **kwargs):\n if filter is not None and not isinstance(filter, Mapping):\n filter = {\"_id\": filter}\n\n for f in self.find(filter, *args, **kwargs):\n return f\n\n return None", "response": "Returns a single file from gridfs."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if a file exists in this instance of GridFS.", "response": "def exists(self, document_or_id=None, **kwargs):\n \"\"\"Check if a file exists in this instance of :class:`GridFS`.\n\n The file to check for can be specified by the value of its\n ``_id`` key, or by passing in a query document. A query\n document can be passed in as dictionary, or by using keyword\n arguments. Thus, the following three calls are equivalent:\n\n >>> fs.exists(file_id)\n >>> fs.exists({\"_id\": file_id})\n >>> fs.exists(_id=file_id)\n\n As are the following two calls:\n\n >>> fs.exists({\"filename\": \"mike.txt\"})\n >>> fs.exists(filename=\"mike.txt\")\n\n And the following two:\n\n >>> fs.exists({\"foo\": {\"$gt\": 12}})\n >>> fs.exists(foo={\"$gt\": 12})\n\n Returns ``True`` if a matching file exists, ``False``\n otherwise. Calls to :meth:`exists` will not automatically\n create appropriate indexes; application developers should be\n sure to create indexes if needed and as appropriate.\n\n :Parameters:\n - `document_or_id` (optional): query document, or _id of the\n document to check for\n - `**kwargs` (optional): keyword arguments are used as a\n query document, if they're present.\n \"\"\"\n if kwargs:\n return self.__files.find_one(kwargs, [\"_id\"]) is not None\n return self.__files.find_one(document_or_id, [\"_id\"]) is not None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open_upload_stream(self, filename, chunk_size_bytes=None,\n metadata=None):\n \"\"\"Opens a Stream that the application can write the contents of the\n file to.\n\n The user must specify the filename, and can choose to add any\n additional information in the metadata field of the file document or\n modify the chunk size.\n For example::\n\n my_db = MongoClient().test\n fs = GridFSBucket(my_db)\n grid_in, file_id = fs.open_upload_stream(\n \"test_file\", chunk_size_bytes=4,\n metadata={\"contentType\": \"text/plain\"})\n grid_in.write(\"data I want to store!\")\n grid_in.close() # uploaded on close\n\n Returns an instance of :class:`~gridfs.grid_file.GridIn`.\n\n Raises :exc:`~gridfs.errors.NoFile` if no such version of\n that file exists.\n Raises :exc:`~ValueError` if `filename` is not a string.\n\n :Parameters:\n - `filename`: The name of the file to upload.\n - `chunk_size_bytes` (options): The number of bytes per chunk of this\n file. Defaults to the chunk_size_bytes in :class:`GridFSBucket`.\n - `metadata` (optional): User data for the 'metadata' field of the\n files collection document. If not provided the metadata field will\n be omitted from the files collection document.\n \"\"\"\n validate_string(\"filename\", filename)\n\n opts = {\"filename\": filename,\n \"chunk_size\": (chunk_size_bytes if chunk_size_bytes\n is not None else self._chunk_size_bytes)}\n if metadata is not None:\n opts[\"metadata\"] = metadata\n\n return GridIn(self._collection, **opts)", "response": "Opens a Stream that the application can write the contents of the file to."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nopens a Stream that the application can write the contents of the file to.", "response": "def open_upload_stream_with_id(\n self, file_id, filename, chunk_size_bytes=None, metadata=None):\n \"\"\"Opens a Stream that the application can write the contents of the\n file to.\n\n The user must specify the file id and filename, and can choose to add\n any additional information in the metadata field of the file document\n or modify the chunk size.\n For example::\n\n my_db = MongoClient().test\n fs = GridFSBucket(my_db)\n grid_in, file_id = fs.open_upload_stream(\n ObjectId(),\n \"test_file\",\n chunk_size_bytes=4,\n metadata={\"contentType\": \"text/plain\"})\n grid_in.write(\"data I want to store!\")\n grid_in.close() # uploaded on close\n\n Returns an instance of :class:`~gridfs.grid_file.GridIn`.\n\n Raises :exc:`~gridfs.errors.NoFile` if no such version of\n that file exists.\n Raises :exc:`~ValueError` if `filename` is not a string.\n\n :Parameters:\n - `file_id`: The id to use for this file. The id must not have\n already been used for another file.\n - `filename`: The name of the file to upload.\n - `chunk_size_bytes` (options): The number of bytes per chunk of this\n file. Defaults to the chunk_size_bytes in :class:`GridFSBucket`.\n - `metadata` (optional): User data for the 'metadata' field of the\n files collection document. If not provided the metadata field will\n be omitted from the files collection document.\n \"\"\"\n validate_string(\"filename\", filename)\n\n opts = {\"_id\": file_id,\n \"filename\": filename,\n \"chunk_size\": (chunk_size_bytes if chunk_size_bytes\n is not None else self._chunk_size_bytes)}\n if metadata is not None:\n opts[\"metadata\"] = metadata\n\n return GridIn(self._collection, **opts)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef upload_from_stream_with_id(self, file_id, filename, source,\n chunk_size_bytes=None, metadata=None):\n \"\"\"Uploads a user file to a GridFS bucket with a custom file id.\n\n Reads the contents of the user file from `source` and uploads\n it to the file `filename`. Source can be a string or file-like object.\n For example::\n\n my_db = MongoClient().test\n fs = GridFSBucket(my_db)\n file_id = fs.upload_from_stream(\n ObjectId(),\n \"test_file\",\n \"data I want to store!\",\n chunk_size_bytes=4,\n metadata={\"contentType\": \"text/plain\"})\n\n Raises :exc:`~gridfs.errors.NoFile` if no such version of\n that file exists.\n Raises :exc:`~ValueError` if `filename` is not a string.\n\n :Parameters:\n - `file_id`: The id to use for this file. The id must not have\n already been used for another file.\n - `filename`: The name of the file to upload.\n - `source`: The source stream of the content to be uploaded. Must be\n a file-like object that implements :meth:`read` or a string.\n - `chunk_size_bytes` (options): The number of bytes per chunk of this\n file. Defaults to the chunk_size_bytes of :class:`GridFSBucket`.\n - `metadata` (optional): User data for the 'metadata' field of the\n files collection document. If not provided the metadata field will\n be omitted from the files collection document.\n \"\"\"\n with self.open_upload_stream_with_id(\n file_id, filename, chunk_size_bytes, metadata) as gin:\n gin.write(source)", "response": "Uploads a user file from a stream to a GridFS bucket with a custom file id."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open_download_stream(self, file_id):\n gout = GridOut(self._collection, file_id)\n\n # Raise NoFile now, instead of on first attribute access.\n gout._ensure_file()\n return gout", "response": "Opens a Stream from which the application can read the contents of\n the stored file specified by file_id."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef download_to_stream(self, file_id, destination):\n gout = self.open_download_stream(file_id)\n for chunk in gout:\n destination.write(chunk)", "response": "Downloads the contents of the stored file specified by file_id and writes the contents to destination."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef delete(self, file_id):\n res = self._files.delete_one({\"_id\": file_id})\n self._chunks.delete_many({\"files_id\": file_id})\n if not res.deleted_count:\n raise NoFile(\n \"no file could be deleted because none matched %s\" % file_id)", "response": "Given an file_id delete this stored file s files collection document\n and associated chunks from a GridFS bucket."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nopen a Stream from which the application can read the contents of the specified file.", "response": "def open_download_stream_by_name(self, filename, revision=-1):\n \"\"\"Opens a Stream from which the application can read the contents of\n `filename` and optional `revision`.\n\n For example::\n\n my_db = MongoClient().test\n fs = GridFSBucket(my_db)\n grid_out = fs.open_download_stream_by_name(\"test_file\")\n contents = grid_out.read()\n\n Returns an instance of :class:`~gridfs.grid_file.GridOut`.\n\n Raises :exc:`~gridfs.errors.NoFile` if no such version of\n that file exists.\n\n Raises :exc:`~ValueError` filename is not a string.\n\n :Parameters:\n - `filename`: The name of the file to read from.\n - `revision` (optional): Which revision (documents with the same\n filename and different uploadDate) of the file to retrieve.\n Defaults to -1 (the most recent revision).\n\n :Note: Revision numbers are defined as follows:\n\n - 0 = the original stored file\n - 1 = the first revision\n - 2 = the second revision\n - etc...\n - -2 = the second most recent revision\n - -1 = the most recent revision\n \"\"\"\n validate_string(\"filename\", filename)\n\n query = {\"filename\": filename}\n\n cursor = self._files.find(query)\n if revision < 0:\n skip = abs(revision) - 1\n cursor.limit(-1).skip(skip).sort(\"uploadDate\", DESCENDING)\n else:\n cursor.limit(-1).skip(revision).sort(\"uploadDate\", ASCENDING)\n try:\n grid_file = next(cursor)\n return GridOut(self._collection, file_document=grid_file)\n except StopIteration:\n raise NoFile(\n \"no version %d for filename %r\" % (revision, filename))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlisting active or available environments.", "response": "def ls(checks):\n \"\"\"List active or available environments.\"\"\"\n if checks:\n checks = sorted(get_testable_checks() & set(checks))\n\n for check in checks:\n envs = get_available_tox_envs(check, e2e_only=True)\n\n if envs:\n echo_success('{}:'.format(check))\n for env in envs:\n echo_info(env, indent=True)\n\n else:\n for check in get_configured_checks():\n envs = get_configured_envs(check)\n\n if envs:\n echo_success('{}:'.format(check))\n for env in envs:\n echo_info(env, indent=True)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fetch_url_data(self, url, username, password, verify, custom_headers):\n ''' Hit a given http url and return the stats lines '''\n # Try to fetch data from the stats URL\n\n auth = (username, password)\n url = \"%s%s\" % (url, STATS_URL)\n custom_headers.update(headers(self.agentConfig))\n\n self.log.debug(\"Fetching haproxy stats from url: %s\" % url)\n\n response = requests.get(\n url, auth=auth, headers=custom_headers, verify=verify, timeout=self.default_integration_http_timeout\n )\n response.raise_for_status()\n\n # it only needs additional decoding in py3, so skip it if it's py2\n if PY2:\n return response.content.splitlines()\n else:\n content = response.content\n\n # If the content is a string, it can't be decoded again\n # But if it's bytes, it can be decoded.\n # So, check if it has the decode method\n decode_fn = getattr(content, \"decode\", None)\n if callable(decode_fn):\n content = content.decode('utf-8')\n\n return content.splitlines()", "response": "Fetch stats from a given http url and return the stats lines"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfetches the stats lines from a given stats socket and return the stats lines", "response": "def _fetch_socket_data(self, parsed_url):\n ''' Hit a given stats socket and return the stats lines '''\n\n self.log.debug(\"Fetching haproxy stats from socket: %s\" % parsed_url.geturl())\n\n if parsed_url.scheme == 'tcp':\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n splitted_loc = parsed_url.netloc.split(':')\n host = splitted_loc[0]\n port = int(splitted_loc[1])\n sock.connect((host, port))\n else:\n sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n sock.connect(parsed_url.path)\n sock.send(b\"show stat\\r\\n\")\n\n response = \"\"\n output = sock.recv(BUFSIZE)\n while output:\n response += output.decode(\"ASCII\")\n output = sock.recv(BUFSIZE)\n\n sock.close()\n\n return response.splitlines()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess the data for the current entry - set of known hosts.", "response": "def _process_data(\n self,\n data,\n collect_aggregates_only,\n process_events,\n url=None,\n collect_status_metrics=False,\n collect_status_metrics_by_host=False,\n tag_service_check_by_host=False,\n services_incl_filter=None,\n services_excl_filter=None,\n collate_status_tags_per_host=False,\n count_status_by_service=True,\n custom_tags=None,\n tags_regex=None,\n active_tag=None,\n enable_service_check=False,\n ):\n ''' Main data-processing loop. For each piece of useful data, we'll\n either save a metric, save an event or both. '''\n\n # Split the first line into an index of fields\n # The line looks like (broken up onto multiple lines)\n # \"# pxname,svname,qcur,qmax,scur,smax,slim,\n # stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,\n # wredis,status,weight,act,bck,chkfail,chkdown,lastchg,\n # downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,\n # type,rate,rate_lim,rate_max,\"\n fields = []\n for f in data[0].split(','):\n if f:\n f = f.replace('# ', '')\n fields.append(f.strip())\n\n self.hosts_statuses = defaultdict(int)\n\n back_or_front = None\n\n # Sanitize CSV, handle line breaks\n data = self._sanitize_lines(data)\n custom_tags = [] if custom_tags is None else custom_tags\n active_tag = [] if active_tag is None else active_tag\n\n # First initialize here so that it is defined whether or not we enter the for loop\n line_tags = list(custom_tags)\n\n # Skip the first line, go backwards to set back_or_front\n for line in data[:0:-1]:\n if not line.strip():\n continue\n\n # Store each line's values in a dictionary\n data_dict = self._line_to_dict(fields, line)\n\n if self._is_aggregate(data_dict):\n back_or_front = data_dict['svname']\n\n self._update_data_dict(data_dict, back_or_front)\n\n self._update_hosts_statuses_if_needed(\n collect_status_metrics, collect_status_metrics_by_host, data_dict, self.hosts_statuses\n )\n\n # Clone the list to avoid extending the original\n # which would carry over previous iteration tags\n line_tags = list(custom_tags)\n\n regex_tags = self._tag_from_regex(tags_regex, data_dict['pxname'])\n if regex_tags:\n line_tags.extend(regex_tags)\n\n if self._should_process(data_dict, collect_aggregates_only):\n # update status\n # Send the list of data to the metric and event callbacks\n self._process_metrics(\n data_dict,\n url,\n services_incl_filter=services_incl_filter,\n services_excl_filter=services_excl_filter,\n custom_tags=line_tags,\n active_tag=active_tag,\n )\n if process_events:\n self._process_event(\n data_dict,\n url,\n services_incl_filter=services_incl_filter,\n services_excl_filter=services_excl_filter,\n custom_tags=line_tags,\n )\n if enable_service_check:\n self._process_service_check(\n data_dict,\n url,\n tag_by_host=tag_service_check_by_host,\n services_incl_filter=services_incl_filter,\n services_excl_filter=services_excl_filter,\n custom_tags=line_tags,\n )\n\n if collect_status_metrics:\n self._process_status_metric(\n self.hosts_statuses,\n collect_status_metrics_by_host,\n services_incl_filter=services_incl_filter,\n services_excl_filter=services_excl_filter,\n collate_status_tags_per_host=collate_status_tags_per_host,\n count_status_by_service=count_status_by_service,\n custom_tags=line_tags,\n active_tag=active_tag,\n )\n\n self._process_backend_hosts_metric(\n self.hosts_statuses,\n services_incl_filter=services_incl_filter,\n services_excl_filter=services_excl_filter,\n custom_tags=line_tags,\n active_tag=active_tag,\n )\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates the data_dict with the current values of the relevant keys.", "response": "def _update_data_dict(self, data_dict, back_or_front):\n \"\"\"\n Adds spct if relevant, adds service\n \"\"\"\n data_dict['back_or_front'] = back_or_front\n # The percentage of used sessions based on 'scur' and 'slim'\n if 'slim' in data_dict and 'scur' in data_dict:\n try:\n data_dict['spct'] = (data_dict['scur'] / data_dict['slim']) * 100\n except (TypeError, ZeroDivisionError):\n pass"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _should_process(self, data_dict, collect_aggregates_only):\n if collect_aggregates_only:\n if self._is_aggregate(data_dict):\n return True\n return False\n elif data_dict['svname'] == Services.BACKEND:\n return False\n return True", "response": "Returns True if we should process the data_dict."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _tag_from_regex(self, tags_regex, service_name):\n if not tags_regex or not service_name:\n return []\n\n match = re.compile(tags_regex).match(service_name)\n\n if not match:\n return []\n\n # match.groupdict() returns tags dictionary in the form of {'name': 'value'}\n # convert it to Datadog tag LIST: ['name:value']\n return [\"%s:%s\" % (name, value) for name, value in iteritems(match.groupdict())]", "response": "Create a list of Datadog tags from a regex"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nnormalizes the HAProxy status as one of the statuses defined in ALL_STATUSES and returns the status as - is in a tag - friendly format.", "response": "def _normalize_status(status):\n \"\"\"\n Try to normalize the HAProxy status as one of the statuses defined in `ALL_STATUSES`,\n if it can't be matched return the status as-is in a tag-friendly format\n ex: 'UP 1/2' -> 'up'\n 'no check' -> 'no_check'\n \"\"\"\n formatted_status = status.lower().replace(\" \", \"_\")\n for normalized_status in Services.ALL_STATUSES:\n if formatted_status.startswith(normalized_status):\n return normalized_status\n return formatted_status"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprocesses metrics from a single line of the CSV file.", "response": "def _process_metrics(\n self, data, url, services_incl_filter=None, services_excl_filter=None, custom_tags=None, active_tag=None\n ):\n \"\"\"\n Data is a dictionary related to one host\n (one line) extracted from the csv.\n It should look like:\n {'pxname':'dogweb', 'svname':'i-4562165', 'scur':'42', ...}\n \"\"\"\n hostname = data['svname']\n service_name = data['pxname']\n back_or_front = data['back_or_front']\n custom_tags = [] if custom_tags is None else custom_tags\n active_tag = [] if active_tag is None else active_tag\n tags = [\"type:%s\" % back_or_front, \"instance_url:%s\" % url, \"service:%s\" % service_name]\n tags.extend(custom_tags)\n tags.extend(active_tag)\n\n if self._is_service_excl_filtered(service_name, services_incl_filter, services_excl_filter):\n return\n\n if back_or_front == Services.BACKEND:\n tags.append('backend:%s' % hostname)\n if data.get('addr'):\n tags.append('server_address:{}'.format(data.get('addr')))\n\n for key, value in data.items():\n if HAProxy.METRICS.get(key):\n suffix = HAProxy.METRICS[key][1]\n name = \"haproxy.%s.%s\" % (back_or_front.lower(), suffix)\n try:\n if HAProxy.METRICS[key][0] == 'rate':\n self.rate(name, float(value), tags=tags)\n else:\n self.gauge(name, float(value), tags=tags)\n except ValueError:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess an event from the server side.", "response": "def _process_event(self, data, url, services_incl_filter=None, services_excl_filter=None, custom_tags=None):\n '''\n Main event processing loop. An event will be created for a service\n status change.\n Service checks on the server side can be used to provide the same functionality\n '''\n hostname = data['svname']\n service_name = data['pxname']\n key = \"%s:%s\" % (hostname, service_name)\n status = self.host_status[url][key]\n custom_tags = [] if custom_tags is None else custom_tags\n\n if self._is_service_excl_filtered(service_name, services_incl_filter, services_excl_filter):\n return\n\n data_status = data['status']\n if status is None:\n self.host_status[url][key] = data_status\n return\n\n if status != data_status and data_status in ('up', 'down'):\n # If the status of a host has changed, we trigger an event\n try:\n lastchg = int(data['lastchg'])\n except Exception:\n lastchg = 0\n\n # Create the event object\n ev = self._create_event(\n data_status, hostname, lastchg, service_name, data['back_or_front'], custom_tags=custom_tags\n )\n self.event(ev)\n\n # Store this host status so we can check against it later\n self.host_status[url][key] = data_status"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess a single service check.", "response": "def _process_service_check(\n self, data, url, tag_by_host=False, services_incl_filter=None, services_excl_filter=None, custom_tags=None\n ):\n ''' Report a service check, tagged by the service and the backend.\n Statuses are defined in `STATUS_TO_SERVICE_CHECK` mapping.\n '''\n custom_tags = [] if custom_tags is None else custom_tags\n service_name = data['pxname']\n status = data['status']\n haproxy_hostname = to_string(self.hostname)\n check_hostname = haproxy_hostname if tag_by_host else ''\n\n if self._is_service_excl_filtered(service_name, services_incl_filter, services_excl_filter):\n return\n\n if status in Services.STATUS_TO_SERVICE_CHECK:\n service_check_tags = [\"service:%s\" % service_name]\n service_check_tags.extend(custom_tags)\n hostname = data['svname']\n if data['back_or_front'] == Services.BACKEND:\n service_check_tags.append('backend:%s' % hostname)\n\n status = Services.STATUS_TO_SERVICE_CHECK[status]\n message = \"%s reported %s:%s %s\" % (haproxy_hostname, service_name, hostname, status)\n self.service_check(\n self.SERVICE_CHECK_NAME, status, message=message, hostname=check_hostname, tags=service_check_tags\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef apply_single_tag_set(tag_set, selection):\n def tags_match(server_tags):\n for key, value in tag_set.items():\n if key not in server_tags or server_tags[key] != value:\n return False\n\n return True\n\n return selection.with_server_descriptions(\n [s for s in selection.server_descriptions if tags_match(s.tags)])", "response": "Applies a single tag set to a server selection."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef apply_tag_sets(tag_sets, selection):\n for tag_set in tag_sets:\n with_tag_set = apply_single_tag_set(tag_set, selection)\n if with_tag_set:\n return with_tag_set\n\n return selection.with_server_descriptions([])", "response": "Applies a list of tag sets to a list of servers."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate metadata. csv files and returns a list of all the available metadata files.", "response": "def metadata(check):\n \"\"\"Validates metadata.csv files\n\n If `check` is specified, only the check will be validated,\n otherwise all metadata files in the repo will be.\n \"\"\"\n metric_sources = get_metric_sources()\n\n if check:\n if check not in metric_sources:\n abort('Metadata file `{}` does not exist.'.format(get_metadata_file(check)))\n metric_sources = [check]\n else:\n metric_sources = sorted(metric_sources)\n\n errors = False\n\n for current_check in metric_sources:\n if current_check.startswith('datadog_checks_'):\n continue\n\n # get any manifest info needed for validation\n manifest = load_manifest(current_check)\n try:\n metric_prefix = manifest['metric_prefix'].rstrip('.')\n except KeyError:\n metric_prefix = None\n\n metadata_file = get_metadata_file(current_check)\n\n # To make logging less verbose, common errors are counted for current check\n metric_prefix_count = defaultdict(int)\n empty_count = defaultdict(int)\n empty_warning_count = defaultdict(int)\n duplicate_set = set()\n metric_prefix_error_shown = False\n\n # Python 2 csv module does not support unicode\n with open(metadata_file, 'rb' if PY2 else 'r', encoding=None if PY2 else 'utf-8') as f:\n reader = csv.DictReader(f, delimiter=',')\n\n # Read header\n if PY2:\n reader._fieldnames = [key.decode('utf-8') for key in reader.fieldnames]\n else:\n reader._fieldnames = reader.fieldnames\n\n for row in reader:\n\n # Number of rows is correct. Since metric is first in the list, should be safe to access\n if len(row) != len(ALL_HEADERS):\n errors = True\n echo_failure('{}: {} Has the wrong amount of columns'.format(current_check, row['metric_name']))\n continue\n\n if PY2:\n for key, value in iteritems(row):\n if value is not None:\n row[key] = value.decode('utf-8')\n\n # all headers exist, no invalid headers\n all_keys = set(row)\n if all_keys != ALL_HEADERS:\n invalid_headers = all_keys.difference(ALL_HEADERS)\n if invalid_headers:\n errors = True\n echo_failure('{}: Invalid column {}'.format(current_check, invalid_headers))\n\n missing_headers = ALL_HEADERS.difference(all_keys)\n if missing_headers:\n errors = True\n echo_failure('{}: Missing columns {}'.format(current_check, missing_headers))\n\n continue\n\n # duplicate metric_name\n if row['metric_name'] and row['metric_name'] not in duplicate_set:\n duplicate_set.add(row['metric_name'])\n else:\n errors = True\n echo_failure('{}: `{}` is a duplicate metric_name'.format(current_check, row['metric_name']))\n\n # metric_name header\n if metric_prefix:\n if not row['metric_name'].startswith(metric_prefix):\n prefix = row['metric_name'].split('.')[0]\n metric_prefix_count[prefix] += 1\n else:\n errors = True\n if not metric_prefix_error_shown and current_check not in PROVIDER_INTEGRATIONS:\n metric_prefix_error_shown = True\n echo_failure('{}: metric_prefix does not exist in manifest'.format(current_check))\n\n # metric_type header\n if row['metric_type'] and row['metric_type'] not in VALID_METRIC_TYPE:\n errors = True\n echo_failure('{}: `{}` is an invalid metric_type.'.format(current_check, row['metric_type']))\n\n # unit_name header\n if row['unit_name'] and row['unit_name'] not in VALID_UNIT_NAMES:\n errors = True\n echo_failure('{}: `{}` is an invalid unit_name.'.format(current_check, row['unit_name']))\n\n # orientation header\n if row['orientation'] and row['orientation'] not in VALID_ORIENTATION:\n errors = True\n echo_failure('{}: `{}` is an invalid orientation.'.format(current_check, row['orientation']))\n\n # empty required fields\n for header in REQUIRED_HEADERS:\n if not row[header]:\n empty_count[header] += 1\n\n # empty description field, description is recommended\n if not row['description']:\n empty_warning_count['description'] += 1\n # exceeds max allowed length of description\n elif len(row['description']) > MAX_DESCRIPTION_LENGTH:\n errors = True\n echo_failure(\n '{}: `{}` exceeds the max length: {} for descriptions.'.format(\n current_check, row['metric_name'], MAX_DESCRIPTION_LENGTH\n )\n )\n\n for header, count in iteritems(empty_count):\n errors = True\n echo_failure('{}: {} is empty in {} rows.'.format(current_check, header, count))\n\n for header, count in iteritems(empty_warning_count):\n echo_warning('{}: {} is empty in {} rows.'.format(current_check, header, count))\n\n for prefix, count in iteritems(metric_prefix_count):\n # Don't spam this warning when we're validating everything\n if check:\n echo_warning(\n '{}: `{}` appears {} time(s) and does not match metric_prefix '\n 'defined in the manifest.'.format(current_check, prefix, count)\n )\n\n if errors:\n abort()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _build_credentials_tuple(mech, source, user, passwd, extra):\n user = _unicode(user) if user is not None else None\n password = passwd if passwd is None else _unicode(passwd)\n if mech == 'GSSAPI':\n properties = extra.get('authmechanismproperties', {})\n service_name = properties.get('SERVICE_NAME', 'mongodb')\n canonicalize = properties.get('CANONICALIZE_HOST_NAME', False)\n service_realm = properties.get('SERVICE_REALM')\n props = GSSAPIProperties(service_name=service_name,\n canonicalize_host_name=canonicalize,\n service_realm=service_realm)\n # Source is always $external.\n return MongoCredential(mech, '$external', user, password, props)\n elif mech == 'MONGODB-X509':\n # user can be None.\n return MongoCredential(mech, '$external', user, None, None)\n else:\n if passwd is None:\n raise ConfigurationError(\"A password is required.\")\n return MongoCredential(mech, source, user, password, None)", "response": "Build and return a mechanism specific credentials tuple."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_scram_response(response):\n return dict(item.split(b\"=\", 1) for item in response.split(b\",\"))", "response": "Split a scram response into key value pairs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nauthenticate using SCRAM - SHA - 1.", "response": "def _authenticate_scram_sha1(credentials, sock_info):\n \"\"\"Authenticate using SCRAM-SHA-1.\"\"\"\n username = credentials.username\n password = credentials.password\n source = credentials.source\n\n # Make local\n _hmac = hmac.HMAC\n _sha1 = sha1\n\n user = username.encode(\"utf-8\").replace(b\"=\", b\"=3D\").replace(b\",\", b\"=2C\")\n nonce = standard_b64encode(\n ((\"%s\" % (SystemRandom().random(),))[2:]).encode(\"utf-8\"))\n first_bare = b\"n=\" + user + b\",r=\" + nonce\n\n cmd = SON([('saslStart', 1),\n ('mechanism', 'SCRAM-SHA-1'),\n ('payload', Binary(b\"n,,\" + first_bare)),\n ('autoAuthorize', 1)])\n res = sock_info.command(source, cmd)\n\n server_first = res['payload']\n parsed = _parse_scram_response(server_first)\n iterations = int(parsed[b'i'])\n salt = parsed[b's']\n rnonce = parsed[b'r']\n if not rnonce.startswith(nonce):\n raise OperationFailure(\"Server returned an invalid nonce.\")\n\n without_proof = b\"c=biws,r=\" + rnonce\n salted_pass = _hi(_password_digest(username, password).encode(\"utf-8\"),\n standard_b64decode(salt),\n iterations)\n client_key = _hmac(salted_pass, b\"Client Key\", _sha1).digest()\n stored_key = _sha1(client_key).digest()\n auth_msg = b\",\".join((first_bare, server_first, without_proof))\n client_sig = _hmac(stored_key, auth_msg, _sha1).digest()\n client_proof = b\"p=\" + standard_b64encode(_xor(client_key, client_sig))\n client_final = b\",\".join((without_proof, client_proof))\n\n server_key = _hmac(salted_pass, b\"Server Key\", _sha1).digest()\n server_sig = standard_b64encode(\n _hmac(server_key, auth_msg, _sha1).digest())\n\n cmd = SON([('saslContinue', 1),\n ('conversationId', res['conversationId']),\n ('payload', Binary(client_final))])\n res = sock_info.command(source, cmd)\n\n parsed = _parse_scram_response(res['payload'])\n if not compare_digest(parsed[b'v'], server_sig):\n raise OperationFailure(\"Server returned an invalid signature.\")\n\n # Depending on how it's configured, Cyrus SASL (which the server uses)\n # requires a third empty challenge.\n if not res['done']:\n cmd = SON([('saslContinue', 1),\n ('conversationId', res['conversationId']),\n ('payload', Binary(b''))])\n res = sock_info.command(source, cmd)\n if not res['done']:\n raise OperationFailure('SASL conversation failed to complete.')"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nauthenticates using SASL PLAIN ( RFC 4616 )", "response": "def _authenticate_plain(credentials, sock_info):\n \"\"\"Authenticate using SASL PLAIN (RFC 4616)\n \"\"\"\n source = credentials.source\n username = credentials.username\n password = credentials.password\n payload = ('\\x00%s\\x00%s' % (username, password)).encode('utf-8')\n cmd = SON([('saslStart', 1),\n ('mechanism', 'PLAIN'),\n ('payload', Binary(payload)),\n ('autoAuthorize', 1)])\n sock_info.command(source, cmd)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nauthenticating using CRAM - MD5 ( RFC 2195 section 7. 1. 1.", "response": "def _authenticate_cram_md5(credentials, sock_info):\n \"\"\"Authenticate using CRAM-MD5 (RFC 2195)\n \"\"\"\n source = credentials.source\n username = credentials.username\n password = credentials.password\n # The password used as the mac key is the\n # same as what we use for MONGODB-CR\n passwd = _password_digest(username, password)\n cmd = SON([('saslStart', 1),\n ('mechanism', 'CRAM-MD5'),\n ('payload', Binary(b'')),\n ('autoAuthorize', 1)])\n response = sock_info.command(source, cmd)\n # MD5 as implicit default digest for digestmod is deprecated\n # in python 3.4\n mac = hmac.HMAC(key=passwd.encode('utf-8'), digestmod=md5)\n mac.update(response['payload'])\n challenge = username.encode('utf-8') + b' ' + b(mac.hexdigest())\n cmd = SON([('saslContinue', 1),\n ('conversationId', response['conversationId']),\n ('payload', Binary(challenge))])\n sock_info.command(source, cmd)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _authenticate_x509(credentials, sock_info):\n query = SON([('authenticate', 1),\n ('mechanism', 'MONGODB-X509')])\n if credentials.username is not None:\n query['user'] = credentials.username\n elif sock_info.max_wire_version < 5:\n raise ConfigurationError(\n \"A username is required for MONGODB-X509 authentication \"\n \"when connected to MongoDB versions older than 3.4.\")\n sock_info.command('$external', query)", "response": "Authenticate using MONGODB - X509."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nauthenticate using MONGODB - CR.", "response": "def _authenticate_mongo_cr(credentials, sock_info):\n \"\"\"Authenticate using MONGODB-CR.\n \"\"\"\n source = credentials.source\n username = credentials.username\n password = credentials.password\n # Get a nonce\n response = sock_info.command(source, {'getnonce': 1})\n nonce = response['nonce']\n key = _auth_key(nonce, username, password)\n\n # Actually authenticate\n query = SON([('authenticate', 1),\n ('user', username),\n ('nonce', nonce),\n ('key', key)])\n sock_info.command(source, query)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_metric_family(self, response, scraper_config):\n input_gen = response.iter_lines(chunk_size=self.REQUESTS_CHUNK_SIZE, decode_unicode=True)\n if scraper_config['_text_filter_blacklist']:\n input_gen = self._text_filter_input(input_gen, scraper_config)\n\n for metric in text_fd_to_metric_families(input_gen):\n metric.type = scraper_config['type_overrides'].get(metric.name, metric.type)\n if metric.type not in self.METRIC_TYPES:\n continue\n metric.name = self._remove_metric_prefix(metric.name, scraper_config)\n yield metric", "response": "Parse the MetricFamily from a valid requests. Response object to provide a MetricFamily object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfiltering out the text input line by line to avoid parsing and processing metrics.", "response": "def _text_filter_input(self, input_gen, scraper_config):\n \"\"\"\n Filters out the text input line by line to avoid parsing and processing\n metrics we know we don't want to process. This only works on `text/plain`\n payloads, and is an INTERNAL FEATURE implemented for the kubelet check\n :param input_get: line generator\n :output: generator of filtered lines\n \"\"\"\n for line in input_gen:\n for item in scraper_config['_text_filter_blacklist']:\n if item in line:\n break\n else:\n # No blacklist matches, passing the line through\n yield line"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef scrape_metrics(self, scraper_config):\n response = self.poll(scraper_config)\n try:\n # no dry run if no label joins\n if not scraper_config['label_joins']:\n scraper_config['_dry_run'] = False\n elif not scraper_config['_watched_labels']:\n # build the _watched_labels set\n for val in itervalues(scraper_config['label_joins']):\n scraper_config['_watched_labels'].add(val['label_to_match'])\n\n for metric in self.parse_metric_family(response, scraper_config):\n yield metric\n\n # Set dry run off\n scraper_config['_dry_run'] = False\n # Garbage collect unused mapping and reset active labels\n for metric, mapping in list(iteritems(scraper_config['_label_mapping'])):\n for key in list(mapping):\n if key not in scraper_config['_active_label_mapping'][metric]:\n del scraper_config['_label_mapping'][metric][key]\n scraper_config['_active_label_mapping'] = {}\n finally:\n response.close()", "response": "Scrape the metrics from prometheus and return the metrics as a generator."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprocesses the metrics from Prometheus and pushes them as gauges", "response": "def process(self, scraper_config, metric_transformers=None):\n \"\"\"\n Polls the data from prometheus and pushes them as gauges\n `endpoint` is the metrics endpoint to use to poll metrics from Prometheus\n\n Note that if the instance has a 'tags' attribute, it will be pushed\n automatically as additional custom tags and added to the metrics\n \"\"\"\n for metric in self.scrape_metrics(scraper_config):\n self.process_metric(metric, scraper_config, metric_transformers=metric_transformers)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_metric(self, metric, scraper_config, metric_transformers=None):\n # If targeted metric, store labels\n self._store_labels(metric, scraper_config)\n\n if metric.name in scraper_config['ignore_metrics']:\n return # Ignore the metric\n\n # Filter metric to see if we can enrich with joined labels\n self._join_labels(metric, scraper_config)\n\n if scraper_config['_dry_run']:\n return\n\n try:\n self.submit_openmetric(scraper_config['metrics_mapper'][metric.name], metric, scraper_config)\n except KeyError:\n if metric_transformers is not None:\n if metric.name in metric_transformers:\n try:\n # Get the transformer function for this specific metric\n transformer = metric_transformers[metric.name]\n transformer(metric, scraper_config)\n except Exception as err:\n self.log.warning(\"Error handling metric: {} - error: {}\".format(metric.name, err))\n else:\n self.log.debug(\n \"Unable to handle metric: {0} - error: \"\n \"No handler function named '{0}' defined\".format(metric.name)\n )\n else:\n # build the wildcard list if first pass\n if scraper_config['_metrics_wildcards'] is None:\n scraper_config['_metrics_wildcards'] = [x for x in scraper_config['metrics_mapper'] if '*' in x]\n\n # try matching wildcard (generic check)\n for wildcard in scraper_config['_metrics_wildcards']:\n if fnmatchcase(metric.name, wildcard):\n self.submit_openmetric(metric.name, metric, scraper_config)", "response": "Process a metric according to the metrics_mapper and the metric_transformers dictionary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npolling the prometheus server for new metrics and return a valid response.", "response": "def poll(self, scraper_config, headers=None):\n \"\"\"\n Custom headers can be added to the default headers.\n\n Returns a valid requests.Response, raise requests.HTTPError if the status code of the requests.Response\n isn't valid - see response.raise_for_status()\n\n The caller needs to close the requests.Response\n\n :param endpoint: string url endpoint\n :param headers: extra headers\n :return: requests.Response\n \"\"\"\n endpoint = scraper_config.get('prometheus_url')\n\n # Should we send a service check for when we make a request\n health_service_check = scraper_config['health_service_check']\n service_check_name = '{}{}'.format(scraper_config['namespace'], '.prometheus.health')\n service_check_tags = ['endpoint:{}'.format(endpoint)]\n service_check_tags.extend(scraper_config['custom_tags'])\n\n try:\n response = self.send_request(endpoint, scraper_config, headers)\n except requests.exceptions.SSLError:\n self.log.error(\"Invalid SSL settings for requesting {} endpoint\".format(endpoint))\n raise\n except IOError:\n if health_service_check:\n self.service_check(service_check_name, AgentCheck.CRITICAL, tags=service_check_tags)\n raise\n try:\n response.raise_for_status()\n if health_service_check:\n self.service_check(service_check_name, AgentCheck.OK, tags=service_check_tags)\n return response\n except requests.HTTPError:\n response.close()\n if health_service_check:\n self.service_check(service_check_name, AgentCheck.CRITICAL, tags=service_check_tags)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef submit_openmetric(self, metric_name, metric, scraper_config, hostname=None):\n if metric.type in [\"gauge\", \"counter\", \"rate\"]:\n metric_name_with_namespace = '{}.{}'.format(scraper_config['namespace'], metric_name)\n for sample in metric.samples:\n val = sample[self.SAMPLE_VALUE]\n if not self._is_value_valid(val):\n self.log.debug(\"Metric value is not supported for metric {}\".format(sample[self.SAMPLE_NAME]))\n continue\n custom_hostname = self._get_hostname(hostname, sample, scraper_config)\n # Determine the tags to send\n tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname=custom_hostname)\n if metric.type == \"counter\" and scraper_config['send_monotonic_counter']:\n self.monotonic_count(metric_name_with_namespace, val, tags=tags, hostname=custom_hostname)\n elif metric.type == \"rate\":\n self.rate(metric_name_with_namespace, val, tags=tags, hostname=custom_hostname)\n else:\n self.gauge(metric_name_with_namespace, val, tags=tags, hostname=custom_hostname)\n elif metric.type == \"histogram\":\n self._submit_gauges_from_histogram(metric_name, metric, scraper_config)\n elif metric.type == \"summary\":\n self._submit_gauges_from_summary(metric_name, metric, scraper_config)\n else:\n self.log.error(\"Metric type {} unsupported for metric {}.\".format(metric.type, metric_name))", "response": "Submit an OpenMetric to Datadog."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_hostname(self, hostname, sample, scraper_config):\n if (\n hostname is None\n and scraper_config['label_to_hostname'] is not None\n and scraper_config['label_to_hostname'] in sample[self.SAMPLE_LABELS]\n ):\n hostname = sample[self.SAMPLE_LABELS][scraper_config['label_to_hostname']]\n suffix = scraper_config['label_to_hostname_suffix']\n if suffix is not None:\n hostname += suffix\n\n return hostname", "response": "Get the hostname from the label_to_hostname setting."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _submit_gauges_from_histogram(self, metric_name, metric, scraper_config, hostname=None):\n for sample in metric.samples:\n val = sample[self.SAMPLE_VALUE]\n if not self._is_value_valid(val):\n self.log.debug(\"Metric value is not supported for metric {}\".format(sample[self.SAMPLE_NAME]))\n continue\n custom_hostname = self._get_hostname(hostname, sample, scraper_config)\n if sample[self.SAMPLE_NAME].endswith(\"_sum\"):\n tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname)\n self.gauge(\n \"{}.{}.sum\".format(scraper_config['namespace'], metric_name),\n val,\n tags=tags,\n hostname=custom_hostname,\n )\n elif sample[self.SAMPLE_NAME].endswith(\"_count\"):\n tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname)\n self.gauge(\n \"{}.{}.count\".format(scraper_config['namespace'], metric_name),\n val,\n tags=tags,\n hostname=custom_hostname,\n )\n elif (\n scraper_config['send_histograms_buckets']\n and sample[self.SAMPLE_NAME].endswith(\"_bucket\")\n and \"Inf\" not in sample[self.SAMPLE_LABELS][\"le\"]\n ):\n sample[self.SAMPLE_LABELS][\"le\"] = float(sample[self.SAMPLE_LABELS][\"le\"])\n tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname)\n self.gauge(\n \"{}.{}.count\".format(scraper_config['namespace'], metric_name),\n val,\n tags=tags,\n hostname=custom_hostname,\n )", "response": "Extracts metrics from a prometheus histogram and sends them as gauges."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_stat(self, buf):\n metrics = []\n buf.seek(0)\n\n # Check the version line to make sure we parse the rest of the\n # body correctly. Particularly, the Connections val was added in\n # >= 3.4.4.\n start_line = buf.readline()\n match = self.version_pattern.search(start_line)\n if match is None:\n return (None, None, \"inactive\", None)\n raise Exception(\"Could not parse version from stat command output: %s\" % start_line)\n else:\n version = match.group()\n has_connections_val = LooseVersion(version) > LooseVersion(\"3.4.4\")\n\n # Clients:\n buf.readline() # skip the Clients: header\n connections = 0\n client_line = buf.readline().strip()\n if client_line:\n connections += 1\n while client_line:\n client_line = buf.readline().strip()\n if client_line:\n connections += 1\n\n # Latency min/avg/max: -10/0/20007\n _, value = buf.readline().split(':')\n l_min, l_avg, l_max = [int(v) for v in value.strip().split('/')]\n metrics.append(ZKMetric('zookeeper.latency.min', l_min))\n metrics.append(ZKMetric('zookeeper.latency.avg', l_avg))\n metrics.append(ZKMetric('zookeeper.latency.max', l_max))\n\n # Received: 101032173\n _, value = buf.readline().split(':')\n # Fixme: This metric name is wrong. It should be removed in a major version of the agent\n # See https://github.com/DataDog/integrations-core/issues/816\n metrics.append(ZKMetric('zookeeper.bytes_received', long(value.strip())))\n metrics.append(ZKMetric('zookeeper.packets.received', long(value.strip()), \"rate\"))\n\n # Sent: 1324\n _, value = buf.readline().split(':')\n # Fixme: This metric name is wrong. It should be removed in a major version of the agent\n # See https://github.com/DataDog/integrations-core/issues/816\n metrics.append(ZKMetric('zookeeper.bytes_sent', long(value.strip())))\n metrics.append(ZKMetric('zookeeper.packets.sent', long(value.strip()), \"rate\"))\n\n if has_connections_val:\n # Connections: 1\n _, value = buf.readline().split(':')\n metrics.append(ZKMetric('zookeeper.connections', int(value.strip())))\n else:\n # If the zk version doesnt explicitly give the Connections val,\n # use the value we computed from the client list.\n metrics.append(ZKMetric('zookeeper.connections', connections))\n\n # Outstanding: 0\n _, value = buf.readline().split(':')\n metrics.append(ZKMetric('zookeeper.outstanding_requests', long(value.strip())))\n\n # Zxid: 0x1034799c7\n _, value = buf.readline().split(':')\n # Parse as a 64 bit hex int\n zxid = long(value.strip(), 16)\n # convert to bytes\n zxid_bytes = struct.pack('>q', zxid)\n # the higher order 4 bytes is the epoch\n (zxid_epoch,) = struct.unpack('>i', zxid_bytes[0:4])\n # the lower order 4 bytes is the count\n (zxid_count,) = struct.unpack('>i', zxid_bytes[4:8])\n\n metrics.append(ZKMetric('zookeeper.zxid.epoch', zxid_epoch))\n metrics.append(ZKMetric('zookeeper.zxid.count', zxid_count))\n\n # Mode: leader\n _, value = buf.readline().split(':')\n mode = value.strip().lower()\n tags = [u'mode:' + mode]\n\n # Node count: 487\n _, value = buf.readline().split(':')\n metrics.append(ZKMetric('zookeeper.nodes', long(value.strip())))\n\n return metrics, tags, mode, version", "response": "Parse the stat command output of the zookeeper stat command."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_mntr(self, buf):\n buf.seek(0)\n first = buf.readline() # First is version string or error\n if first == 'This ZooKeeper instance is not currently serving requests':\n return (None, 'inactive')\n\n metrics = []\n mode = 'inactive'\n\n for line in buf:\n try:\n key, value = line.split()\n\n if key == \"zk_server_state\":\n mode = value.lower()\n continue\n\n metric_name = self._normalize_metric_label(key)\n metric_type = \"rate\" if key in self._MNTR_RATES else \"gauge\"\n metric_value = int(value)\n metrics.append(ZKMetric(metric_name, metric_value, metric_type))\n\n except ValueError:\n self.log.warning(u\"Cannot format `mntr` value. key={key}, value{value}\".format(key=key, value=value))\n continue\n except Exception:\n self.log.exception(\n u\"Unexpected exception occurred while parsing `mntr` command content:\\n{buf}\".format(buf=buf)\n )\n\n return (metrics, mode)", "response": "Parse the mntr command s content."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _osd_pct_used(self, health):\n # Full string looks like: osd.2 is full at 95%\n # Near full string: osd.1 is near full at 94%\n pct = re.compile(r'\\d+%').findall(health)\n osd = re.compile(r'osd.\\d+').findall(health)\n if len(pct) > 0 and len(osd) > 0:\n return osd[0], int(pct[0][:-1])\n else:\n return None, None", "response": "Take a single health check string return the OSD name and percentage used"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a son manipulator to this database.", "response": "def add_son_manipulator(self, manipulator):\n \"\"\"Add a new son manipulator to this database.\n\n **DEPRECATED** - `add_son_manipulator` is deprecated.\n\n .. versionchanged:: 3.0\n Deprecated add_son_manipulator.\n \"\"\"\n warnings.warn(\"add_son_manipulator is deprecated\",\n DeprecationWarning, stacklevel=2)\n base = SONManipulator()\n def method_overwritten(instance, method):\n \"\"\"Test if this method has been overridden.\"\"\"\n return (getattr(\n instance, method).__func__ != getattr(base, method).__func__)\n\n if manipulator.will_copy():\n if method_overwritten(manipulator, \"transform_incoming\"):\n self.__incoming_copying_manipulators.insert(0, manipulator)\n if method_overwritten(manipulator, \"transform_outgoing\"):\n self.__outgoing_copying_manipulators.insert(0, manipulator)\n else:\n if method_overwritten(manipulator, \"transform_incoming\"):\n self.__incoming_manipulators.insert(0, manipulator)\n if method_overwritten(manipulator, \"transform_outgoing\"):\n self.__outgoing_manipulators.insert(0, manipulator)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of all incoming SON manipulators.", "response": "def incoming_manipulators(self):\n \"\"\"**DEPRECATED**: All incoming SON manipulators.\n\n .. versionchanged:: 3.5\n Deprecated.\n\n .. versionadded:: 2.0\n \"\"\"\n warnings.warn(\"Database.incoming_manipulators() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n return [manipulator.__class__.__name__\n for manipulator in self.__incoming_manipulators]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of all incoming SON copying manipulators.", "response": "def incoming_copying_manipulators(self):\n \"\"\"**DEPRECATED**: All incoming SON copying manipulators.\n\n .. versionchanged:: 3.5\n Deprecated.\n\n .. versionadded:: 2.0\n \"\"\"\n warnings.warn(\"Database.incoming_copying_manipulators() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n return [manipulator.__class__.__name__\n for manipulator in self.__incoming_copying_manipulators]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef outgoing_manipulators(self):\n warnings.warn(\"Database.outgoing_manipulators() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n return [manipulator.__class__.__name__\n for manipulator in self.__outgoing_manipulators]", "response": "Return a list of all outgoing SON manipulators."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of all outgoing SON copying manipulators.", "response": "def outgoing_copying_manipulators(self):\n \"\"\"**DEPRECATED**: All outgoing SON copying manipulators.\n\n .. versionchanged:: 3.5\n Deprecated.\n\n .. versionadded:: 2.0\n \"\"\"\n warnings.warn(\"Database.outgoing_copying_manipulators() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n return [manipulator.__class__.__name__\n for manipulator in self.__outgoing_copying_manipulators]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a new collection with the given name and options.", "response": "def get_collection(self, name, codec_options=None, read_preference=None,\n write_concern=None, read_concern=None):\n \"\"\"Get a :class:`~pymongo.collection.Collection` with the given name\n and options.\n\n Useful for creating a :class:`~pymongo.collection.Collection` with\n different codec options, read preference, and/or write concern from\n this :class:`Database`.\n\n >>> db.read_preference\n Primary()\n >>> coll1 = db.test\n >>> coll1.read_preference\n Primary()\n >>> from pymongo import ReadPreference\n >>> coll2 = db.get_collection(\n ... 'test', read_preference=ReadPreference.SECONDARY)\n >>> coll2.read_preference\n Secondary(tag_sets=None)\n\n :Parameters:\n - `name`: The name of the collection - a string.\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`. If ``None`` (the\n default) the :attr:`codec_options` of this :class:`Database` is\n used.\n - `read_preference` (optional): The read preference to use. If\n ``None`` (the default) the :attr:`read_preference` of this\n :class:`Database` is used. See :mod:`~pymongo.read_preferences`\n for options.\n - `write_concern` (optional): An instance of\n :class:`~pymongo.write_concern.WriteConcern`. If ``None`` (the\n default) the :attr:`write_concern` of this :class:`Database` is\n used.\n - `read_concern` (optional): An instance of\n :class:`~pymongo.read_concern.ReadConcern`. If ``None`` (the\n default) the :attr:`read_concern` of this :class:`Database` is\n used.\n \"\"\"\n return Collection(\n self, name, False, codec_options, read_preference,\n write_concern, read_concern)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a Collection instance with the default settings.", "response": "def _collection_default_options(self, name, **kargs):\n \"\"\"Get a Collection instance with the default settings.\"\"\"\n wc = (self.write_concern\n if self.write_concern.acknowledged else WriteConcern())\n return self.get_collection(\n name, codec_options=DEFAULT_CODEC_OPTIONS,\n read_preference=ReadPreference.PRIMARY,\n write_concern=wc)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_collection(self, name, codec_options=None,\n read_preference=None, write_concern=None,\n read_concern=None, **kwargs):\n \"\"\"Create a new :class:`~pymongo.collection.Collection` in this\n database.\n\n Normally collection creation is automatic. This method should\n only be used to specify options on\n creation. :class:`~pymongo.errors.CollectionInvalid` will be\n raised if the collection already exists.\n\n Options should be passed as keyword arguments to this method. Supported\n options vary with MongoDB release. Some examples include:\n\n - \"size\": desired initial size for the collection (in\n bytes). For capped collections this size is the max\n size of the collection.\n - \"capped\": if True, this is a capped collection\n - \"max\": maximum number of objects if capped (optional)\n\n See the MongoDB documentation for a full list of supported options by\n server version.\n\n :Parameters:\n - `name`: the name of the collection to create\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`. If ``None`` (the\n default) the :attr:`codec_options` of this :class:`Database` is\n used.\n - `read_preference` (optional): The read preference to use. If\n ``None`` (the default) the :attr:`read_preference` of this\n :class:`Database` is used.\n - `write_concern` (optional): An instance of\n :class:`~pymongo.write_concern.WriteConcern`. If ``None`` (the\n default) the :attr:`write_concern` of this :class:`Database` is\n used.\n - `read_concern` (optional): An instance of\n :class:`~pymongo.read_concern.ReadConcern`. If ``None`` (the\n default) the :attr:`read_concern` of this :class:`Database` is\n used.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`.\n - `**kwargs` (optional): additional keyword arguments will\n be passed as options for the create collection command\n\n .. versionchanged:: 3.4\n Added the collation option.\n\n .. versionchanged:: 3.0\n Added the codec_options, read_preference, and write_concern options.\n\n .. versionchanged:: 2.2\n Removed deprecated argument: options\n \"\"\"\n if name in self.collection_names():\n raise CollectionInvalid(\"collection %s already exists\" % name)\n\n return Collection(self, name, True, codec_options,\n read_preference, write_concern,\n read_concern, **kwargs)", "response": "Creates a new collection in this database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _apply_incoming_manipulators(self, son, collection):\n for manipulator in self.__incoming_manipulators:\n son = manipulator.transform_incoming(son, collection)\n return son", "response": "Apply incoming manipulators to son."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _apply_incoming_copying_manipulators(self, son, collection):\n for manipulator in self.__incoming_copying_manipulators:\n son = manipulator.transform_incoming(son, collection)\n return son", "response": "Apply incoming copying manipulators to son."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _fix_incoming(self, son, collection):\n son = self._apply_incoming_manipulators(son, collection)\n son = self._apply_incoming_copying_manipulators(son, collection)\n return son", "response": "Apply manipulators to an incoming SON object before it gets stored."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _fix_outgoing(self, son, collection):\n for manipulator in reversed(self.__outgoing_manipulators):\n son = manipulator.transform_outgoing(son, collection)\n for manipulator in reversed(self.__outgoing_copying_manipulators):\n son = manipulator.transform_outgoing(son, collection)\n return son", "response": "Apply manipulators to a SON object as it comes out of the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef command(self, command, value=1, check=True,\n allowable_errors=None, read_preference=ReadPreference.PRIMARY,\n codec_options=DEFAULT_CODEC_OPTIONS, **kwargs):\n \"\"\"Issue a MongoDB command.\n\n Send command `command` to the database and return the\n response. If `command` is an instance of :class:`basestring`\n (:class:`str` in python 3) then the command {`command`: `value`}\n will be sent. Otherwise, `command` must be an instance of\n :class:`dict` and will be sent as is.\n\n Any additional keyword arguments will be added to the final\n command document before it is sent.\n\n For example, a command like ``{buildinfo: 1}`` can be sent\n using:\n\n >>> db.command(\"buildinfo\")\n\n For a command where the value matters, like ``{collstats:\n collection_name}`` we can do:\n\n >>> db.command(\"collstats\", collection_name)\n\n For commands that take additional arguments we can use\n kwargs. So ``{filemd5: object_id, root: file_root}`` becomes:\n\n >>> db.command(\"filemd5\", object_id, root=file_root)\n\n :Parameters:\n - `command`: document representing the command to be issued,\n or the name of the command (for simple commands only).\n\n .. note:: the order of keys in the `command` document is\n significant (the \"verb\" must come first), so commands\n which require multiple keys (e.g. `findandmodify`)\n should use an instance of :class:`~bson.son.SON` or\n a string and kwargs instead of a Python `dict`.\n\n - `value` (optional): value to use for the command verb when\n `command` is passed as a string\n - `check` (optional): check the response for errors, raising\n :class:`~pymongo.errors.OperationFailure` if there are any\n - `allowable_errors`: if `check` is ``True``, error messages\n in this list will be ignored by error-checking\n - `read_preference`: The read preference for this operation.\n See :mod:`~pymongo.read_preferences` for options.\n - `codec_options`: A :class:`~bson.codec_options.CodecOptions`\n instance.\n - `**kwargs` (optional): additional keyword arguments will\n be added to the command document before it is sent\n\n .. note:: :meth:`command` does **not** obey :attr:`read_preference`\n or :attr:`codec_options`. You must use the `read_preference` and\n `codec_options` parameters instead.\n\n .. versionchanged:: 3.0\n Removed the `as_class`, `fields`, `uuid_subtype`, `tag_sets`,\n and `secondary_acceptable_latency_ms` option.\n Removed `compile_re` option: PyMongo now always represents BSON\n regular expressions as :class:`~bson.regex.Regex` objects. Use\n :meth:`~bson.regex.Regex.try_compile` to attempt to convert from a\n BSON regular expression to a Python regular expression object.\n Added the `codec_options` parameter.\n\n .. versionchanged:: 2.7\n Added `compile_re` option. If set to False, PyMongo represented BSON\n regular expressions as :class:`~bson.regex.Regex` objects instead of\n attempting to compile BSON regular expressions as Python native\n regular expressions, thus preventing errors for some incompatible\n patterns, see `PYTHON-500`_.\n\n .. versionchanged:: 2.3\n Added `tag_sets` and `secondary_acceptable_latency_ms` options.\n .. versionchanged:: 2.2\n Added support for `as_class` - the class you want to use for\n the resulting documents\n\n .. _PYTHON-500: https://jira.mongodb.org/browse/PYTHON-500\n\n .. mongodoc:: commands\n \"\"\"\n client = self.__client\n with client._socket_for_reads(read_preference) as (sock_info, slave_ok):\n return self._command(sock_info, command, slave_ok, value,\n check, allowable_errors, read_preference,\n codec_options, **kwargs)", "response": "Issue a MongoDB command to the database and return the response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a list of all the collection names in this database.", "response": "def collection_names(self, include_system_collections=True):\n \"\"\"Get a list of all the collection names in this database.\n\n :Parameters:\n - `include_system_collections` (optional): if ``False`` list\n will not include system collections (e.g ``system.indexes``)\n \"\"\"\n with self.__client._socket_for_reads(\n ReadPreference.PRIMARY) as (sock_info, slave_okay):\n\n wire_version = sock_info.max_wire_version\n results = self._list_collections(sock_info, slave_okay)\n\n # Iterating the cursor to completion may require a socket for getmore.\n # Ensure we do that outside the \"with\" block so we don't require more\n # than one socket at a time.\n names = [result[\"name\"] for result in results]\n if wire_version <= 2:\n # MongoDB 2.4 and older return index namespaces and collection\n # namespaces prefixed with the database name.\n names = [n[len(self.__name) + 1:] for n in names\n if n.startswith(self.__name + \".\") and \"$\" not in n]\n\n if not include_system_collections:\n names = [name for name in names if not name.startswith(\"system.\")]\n return names"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef drop_collection(self, name_or_collection):\n name = name_or_collection\n if isinstance(name, Collection):\n name = name.name\n\n if not isinstance(name, string_type):\n raise TypeError(\"name_or_collection must be an \"\n \"instance of %s\" % (string_type.__name__,))\n\n self.__client._purge_index(self.__name, name)\n\n with self.__client._socket_for_reads(\n ReadPreference.PRIMARY) as (sock_info, slave_ok):\n return self._command(\n sock_info, 'drop', slave_ok, _unicode(name),\n allowable_errors=['ns not found'],\n write_concern=self.write_concern,\n parse_write_concern_error=True)", "response": "Drop a collection.\n\n :Parameters:\n - `name_or_collection`: the name of a collection to drop or the\n collection object itself\n\n .. note:: The :attr:`~pymongo.database.Database.write_concern` of\n this database is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this database's write concern automatically to this operation\n when connected to MongoDB >= 3.4."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef current_op(self, include_all=False):\n cmd = SON([(\"currentOp\", 1), (\"$all\", include_all)])\n with self.__client._socket_for_writes() as sock_info:\n if sock_info.max_wire_version >= 4:\n return sock_info.command(\"admin\", cmd)\n else:\n spec = {\"$all\": True} if include_all else {}\n x = helpers._first_batch(sock_info, \"admin\", \"$cmd.sys.inprog\",\n spec, -1, True, self.codec_options,\n ReadPreference.PRIMARY, cmd, self.client._event_listeners)\n return x.get('data', [None])[0]", "response": "Get information on currently running idle operations."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the error message from the last operation.", "response": "def error(self):\n \"\"\"**DEPRECATED**: Get the error if one occurred on the last operation.\n\n This method is obsolete: all MongoDB write operations (insert, update,\n remove, and so on) use the write concern ``w=1`` and report their\n errors by default.\n\n .. versionchanged:: 2.8\n Deprecated.\n \"\"\"\n warnings.warn(\"Database.error() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n error = self.command(\"getlasterror\")\n error_msg = error.get(\"err\", \"\")\n if error_msg is None:\n return None\n if error_msg.startswith(\"not master\"):\n # Reset primary server and request check, if another thread isn't\n # doing so already.\n primary = self.__client.primary\n if primary:\n self.__client._reset_server_and_request_check(primary)\n return error"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the most recent error on this database.", "response": "def previous_error(self):\n \"\"\"**DEPRECATED**: Get the most recent error on this database.\n\n This method is obsolete: all MongoDB write operations (insert, update,\n remove, and so on) use the write concern ``w=1`` and report their\n errors by default.\n\n Only returns errors that have occurred since the last call to\n :meth:`reset_error_history`. Returns None if no such errors have\n occurred.\n\n .. versionchanged:: 2.8\n Deprecated.\n \"\"\"\n warnings.warn(\"previous_error() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n error = self.command(\"getpreverror\")\n if error.get(\"err\", 0) is None:\n return None\n return error"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nuse a command to create or update a user.", "response": "def _create_or_update_user(\n self, create, name, password, read_only, **kwargs):\n \"\"\"Use a command to create (if create=True) or modify a user.\n \"\"\"\n opts = {}\n if read_only or (create and \"roles\" not in kwargs):\n warnings.warn(\"Creating a user with the read_only option \"\n \"or without roles is deprecated in MongoDB \"\n \">= 2.6\", DeprecationWarning)\n\n opts[\"roles\"] = [self._default_role(read_only)]\n\n elif read_only:\n warnings.warn(\"The read_only option is deprecated in MongoDB \"\n \">= 2.6, use 'roles' instead\", DeprecationWarning)\n\n if password is not None:\n # We always salt and hash client side.\n if \"digestPassword\" in kwargs:\n raise ConfigurationError(\"The digestPassword option is not \"\n \"supported via add_user. Please use \"\n \"db.command('createUser', ...) \"\n \"instead for this option.\")\n opts[\"pwd\"] = auth._password_digest(name, password)\n opts[\"digestPassword\"] = False\n\n # Don't send {} as writeConcern.\n if self.write_concern.acknowledged and self.write_concern.document:\n opts[\"writeConcern\"] = self.write_concern.document\n opts.update(kwargs)\n\n if create:\n command_name = \"createUser\"\n else:\n command_name = \"updateUser\"\n\n self.command(command_name, name, **opts)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a user to the system. users collection.", "response": "def _legacy_add_user(self, name, password, read_only, **kwargs):\n \"\"\"Uses v1 system to add users, i.e. saving to system.users.\n \"\"\"\n # Use a Collection with the default codec_options.\n system_users = self._collection_default_options('system.users')\n user = system_users.find_one({\"user\": name}) or {\"user\": name}\n if password is not None:\n user[\"pwd\"] = auth._password_digest(name, password)\n if read_only is not None:\n user[\"readOnly\"] = read_only\n user.update(kwargs)\n\n # We don't care what the _id is, only that it has one\n # for the replace_one call below.\n user.setdefault(\"_id\", ObjectId())\n try:\n system_users.replace_one({\"_id\": user[\"_id\"]}, user, True)\n except OperationFailure as exc:\n # First admin user add fails gle in MongoDB >= 2.1.2\n # See SERVER-4225 for more information.\n if 'login' in str(exc):\n pass\n # First admin user add fails gle from mongos 2.0.x\n # and 2.2.x.\n elif (exc.details and\n 'getlasterror' in exc.details.get('note', '')):\n pass\n else:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_user(self, name, password=None, read_only=None, **kwargs):\n if not isinstance(name, string_type):\n raise TypeError(\"name must be an \"\n \"instance of %s\" % (string_type.__name__,))\n if password is not None:\n if not isinstance(password, string_type):\n raise TypeError(\"password must be an \"\n \"instance of %s\" % (string_type.__name__,))\n if len(password) == 0:\n raise ValueError(\"password can't be empty\")\n if read_only is not None:\n read_only = common.validate_boolean('read_only', read_only)\n if 'roles' in kwargs:\n raise ConfigurationError(\"Can not use \"\n \"read_only and roles together\")\n\n try:\n uinfo = self.command(\"usersInfo\", name)\n # Create the user if not found in uinfo, otherwise update one.\n self._create_or_update_user(\n (not uinfo[\"users\"]), name, password, read_only, **kwargs)\n except OperationFailure as exc:\n # MongoDB >= 2.5.3 requires the use of commands to manage\n # users.\n if exc.code in common.COMMAND_NOT_FOUND_CODES:\n self._legacy_add_user(name, password, read_only, **kwargs)\n return\n # Unauthorized. Attempt to create the user in case of\n # localhost exception.\n elif exc.code == 13:\n self._create_or_update_user(\n True, name, password, read_only, **kwargs)\n else:\n raise", "response": "Create a new user with the specified password and read only permissions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef remove_user(self, name):\n try:\n cmd = SON([(\"dropUser\", name)])\n # Don't send {} as writeConcern.\n if self.write_concern.acknowledged and self.write_concern.document:\n cmd[\"writeConcern\"] = self.write_concern.document\n self.command(cmd)\n except OperationFailure as exc:\n # See comment in add_user try / except above.\n if exc.code in common.COMMAND_NOT_FOUND_CODES:\n coll = self._collection_default_options('system.users')\n coll.delete_one({\"user\": name})\n return\n raise", "response": "Removes a user from this database."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef authenticate(self, name=None, password=None,\n source=None, mechanism='DEFAULT', **kwargs):\n \"\"\"**DEPRECATED**: Authenticate to use this database.\n\n Authentication lasts for the life of the underlying client\n instance, or until :meth:`logout` is called.\n\n Raises :class:`TypeError` if (required) `name`, (optional) `password`,\n or (optional) `source` is not an instance of :class:`basestring`\n (:class:`str` in python 3).\n\n .. note::\n - This method authenticates the current connection, and\n will also cause all new :class:`~socket.socket` connections\n in the underlying client instance to be authenticated automatically.\n\n - Authenticating more than once on the same database with different\n credentials is not supported. You must call :meth:`logout` before\n authenticating with new credentials.\n\n - When sharing a client instance between multiple threads, all\n threads will share the authentication. If you need different\n authentication profiles for different purposes you must use\n distinct client instances.\n\n :Parameters:\n - `name`: the name of the user to authenticate. Optional when\n `mechanism` is MONGODB-X509 and the MongoDB server version is\n >= 3.4.\n - `password` (optional): the password of the user to authenticate.\n Not used with GSSAPI or MONGODB-X509 authentication.\n - `source` (optional): the database to authenticate on. If not\n specified the current database is used.\n - `mechanism` (optional): See\n :data:`~pymongo.auth.MECHANISMS` for options.\n By default, use SCRAM-SHA-1 with MongoDB 3.0 and later,\n MONGODB-CR (MongoDB Challenge Response protocol) for older servers.\n - `authMechanismProperties` (optional): Used to specify\n authentication mechanism specific options. To specify the service\n name for GSSAPI authentication pass\n authMechanismProperties='SERVICE_NAME:'\n\n .. versionchanged:: 3.5\n Deprecated. Authenticating multiple users conflicts with support for\n logical sessions in MongoDB 3.6. To authenticate as multiple users,\n create multiple instances of MongoClient.\n\n .. versionadded:: 2.8\n Use SCRAM-SHA-1 with MongoDB 3.0 and later.\n\n .. versionchanged:: 2.5\n Added the `source` and `mechanism` parameters. :meth:`authenticate`\n now raises a subclass of :class:`~pymongo.errors.PyMongoError` if\n authentication fails due to invalid credentials or configuration\n issues.\n\n .. mongodoc:: authenticate\n \"\"\"\n if name is not None and not isinstance(name, string_type):\n raise TypeError(\"name must be an \"\n \"instance of %s\" % (string_type.__name__,))\n if password is not None and not isinstance(password, string_type):\n raise TypeError(\"password must be an \"\n \"instance of %s\" % (string_type.__name__,))\n if source is not None and not isinstance(source, string_type):\n raise TypeError(\"source must be an \"\n \"instance of %s\" % (string_type.__name__,))\n common.validate_auth_mechanism('mechanism', mechanism)\n\n validated_options = {}\n for option, value in iteritems(kwargs):\n normalized, val = common.validate_auth_option(option, value)\n validated_options[normalized] = val\n\n credentials = auth._build_credentials_tuple(\n mechanism,\n source or self.name,\n name,\n password,\n validated_options)\n\n self.client._cache_credentials(\n self.name,\n credentials,\n connect=True)\n\n return True", "response": "Authenticate to use this database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef logout(self):\n warnings.warn(\"Database.logout() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n # Sockets will be deauthenticated as they are used.\n self.client._purge_credentials(self.name)", "response": "Deauthorize use of this database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nevaluating a JavaScript expression in MongoDB.", "response": "def eval(self, code, *args):\n \"\"\"**DEPRECATED**: Evaluate a JavaScript expression in MongoDB.\n\n :Parameters:\n - `code`: string representation of JavaScript code to be\n evaluated\n - `args` (optional): additional positional arguments are\n passed to the `code` being evaluated\n\n .. warning:: the eval command is deprecated in MongoDB 3.0 and\n will be removed in a future server version.\n \"\"\"\n warnings.warn(\"Database.eval() is deprecated\",\n DeprecationWarning, stacklevel=2)\n\n if not isinstance(code, Code):\n code = Code(code)\n\n result = self.command(\"$eval\", code, args=args)\n return result.get(\"retval\", None)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a list of the names of the functions stored in this database.", "response": "def list(self):\n \"\"\"Get a list of the names of the functions stored in this database.\"\"\"\n return [x[\"_id\"] for x in self._db.system.js.find(projection=[\"_id\"])]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmoving _id to the front if it s there.", "response": "def transform_incoming(self, son, collection):\n \"\"\"Move _id to the front if it's there.\n \"\"\"\n if not \"_id\" in son:\n return son\n transformed = SON({\"_id\": son[\"_id\"]})\n transformed.update(son)\n return transformed"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransforms incoming Son to DBRefs.", "response": "def transform_incoming(self, son, collection):\n \"\"\"Replace embedded documents with DBRefs.\n \"\"\"\n\n def transform_value(value):\n if isinstance(value, collections.MutableMapping):\n if \"_id\" in value and \"_ns\" in value:\n return DBRef(value[\"_ns\"], transform_value(value[\"_id\"]))\n else:\n return transform_dict(SON(value))\n elif isinstance(value, list):\n return [transform_value(v) for v in value]\n return value\n\n def transform_dict(object):\n for (key, value) in object.items():\n object[key] = transform_value(value)\n return object\n\n return transform_dict(SON(son))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_command(command, capture=None, check=False, encoding='utf-8', shell=False, env=None):\n if shell == 'detect':\n shell = NEED_SHELL\n\n if isinstance(command, string_types) and not ON_WINDOWS:\n command = shlex.split(command)\n\n if capture:\n if capture is True or capture == 'both':\n stdout, stderr = TemporaryFile, TemporaryFile\n elif capture in ('stdout', 'out'):\n stdout, stderr = TemporaryFile, mock_context_manager\n elif capture in ('stderr', 'err'):\n stdout, stderr = mock_context_manager, TemporaryFile\n else:\n raise ValueError('Unknown capture method `{}`'.format(capture))\n else:\n stdout, stderr = mock_context_manager, mock_context_manager\n\n with stdout() as stdout, stderr() as stderr:\n process = Popen(command, stdout=stdout, stderr=stderr, shell=shell, env=env)\n process.wait()\n\n if stdout is None:\n stdout = ''\n else:\n stdout.seek(0)\n stdout = stdout.read().decode(encoding)\n\n if stderr is None:\n stderr = ''\n else:\n stderr.seek(0)\n stderr = stderr.read().decode(encoding)\n\n if check and process.returncode != 0:\n raise SubprocessError(\n 'Command: {}\\n' 'Exit code: {}\\n' 'Captured Output: {}'.format(command, process.returncode, stdout + stderr)\n )\n\n return SubprocessResult(stdout, stderr, process.returncode)", "response": "Runs a command in a specific way."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _hdfs_namenode_metrics(self, beans, metrics, tags):\n bean = next(iter(beans))\n bean_name = bean.get('name')\n\n if bean_name != bean_name:\n raise Exception(\"Unexpected bean name {}\".format(bean_name))\n\n for metric, (metric_name, metric_type) in iteritems(metrics):\n metric_value = bean.get(metric)\n\n if metric_value is not None:\n self._set_metric(metric_name, metric_type, metric_value, tags)\n\n if 'CapacityUsed' in bean and 'CapacityTotal' in bean:\n self._set_metric(\n 'hdfs.namenode.capacity_in_use',\n self.GAUGE,\n float(bean['CapacityUsed']) / float(bean['CapacityTotal']),\n tags,\n )", "response": "Get HDFS namenode metrics from JMX bean"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _set_metric(self, metric_name, metric_type, value, tags=None):\n if metric_type == self.GAUGE:\n self.gauge(metric_name, value, tags=tags)\n else:\n self.log.error('Metric type \"{}\" unknown'.format(metric_type))", "response": "Set a metric in the cache."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nperform a REST request to the given URL and return the JSON response.", "response": "def _rest_request_to_json(self, instance, url, object_path, query_params, tags=None):\n \"\"\"\n Query the given URL and return the JSON response\n \"\"\"\n if object_path:\n url = self._join_url_dir(url, object_path)\n\n # Add query_params as arguments\n if query_params:\n query = '&'.join(['{}={}'.format(key, value) for key, value in iteritems(query_params)])\n url = urljoin(url, '?' + query)\n\n auth = None\n\n # Authenticate our connection to JMX endpoint if required\n kerberos = instance.get('kerberos')\n username = instance.get('username')\n password = instance.get('password')\n if username is not None and password is not None:\n auth = (username, password)\n elif kerberos is not None:\n if kerberos not in KERBEROS_STRATEGIES:\n raise Exception('Invalid Kerberos strategy `{}`'.format(kerberos))\n\n auth = requests_kerberos.HTTPKerberosAuth(\n mutual_authentication=KERBEROS_STRATEGIES[kerberos],\n delegate=is_affirmative(instance.get('kerberos_delegate', False)),\n force_preemptive=is_affirmative(instance.get('kerberos_force_initiate', False)),\n hostname_override=instance.get('kerberos_hostname'),\n principal=instance.get('kerberos_principal'),\n )\n\n disable_ssl_validation = instance.get('disable_ssl_validation', False)\n\n old_keytab_path = None\n if 'kerberos_keytab' in instance:\n old_keytab_path = os.getenv('KRB5_CLIENT_KTNAME')\n os.environ['KRB5_CLIENT_KTNAME'] = instance['kerberos_keytab']\n\n self.log.debug('Attempting to connect to \"{}\"'.format(url))\n\n try:\n response = requests.get(\n url, auth=auth, timeout=self.default_integration_http_timeout, verify=not disable_ssl_validation\n )\n response.raise_for_status()\n response_json = response.json()\n\n except Timeout as e:\n self.service_check(\n self.JMX_SERVICE_CHECK, AgentCheck.CRITICAL, tags=tags, message=\"Request timeout: {}, {}\".format(url, e)\n )\n raise\n\n except (HTTPError, InvalidURL, ConnectionError) as e:\n self.service_check(\n self.JMX_SERVICE_CHECK, AgentCheck.CRITICAL, tags=tags, message=\"Request failed: {}, {}\".format(url, e)\n )\n raise\n\n except JSONDecodeError as e:\n self.service_check(\n self.JMX_SERVICE_CHECK,\n AgentCheck.CRITICAL,\n tags=tags,\n message='JSON Parse failed: {}, {}'.format(url, e),\n )\n raise\n\n except ValueError as e:\n self.service_check(self.JMX_SERVICE_CHECK, AgentCheck.CRITICAL, tags=tags, message=str(e))\n raise\n\n else:\n return response_json\n\n finally:\n if old_keytab_path is not None:\n os.environ['KRB5_CLIENT_KTNAME'] = old_keytab_path"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _join_url_dir(cls, url, *args):\n\n for path in args:\n url = url.rstrip('/') + '/'\n url = urljoin(url, path.lstrip('/'))\n\n return url", "response": "Join a URL with multiple directories"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds readPreference to spec when appropriate.", "response": "def _maybe_add_read_preference(spec, read_preference):\n \"\"\"Add $readPreference to spec when appropriate.\"\"\"\n mode = read_preference.mode\n tag_sets = read_preference.tag_sets\n max_staleness = read_preference.max_staleness\n # Only add $readPreference if it's something other than primary to avoid\n # problems with mongos versions that don't support read preferences. Also,\n # for maximum backwards compatibility, don't add $readPreference for\n # secondaryPreferred unless tags or maxStalenessSeconds are in use (setting\n # the slaveOkay bit has the same effect).\n if mode and (\n mode != ReadPreference.SECONDARY_PREFERRED.mode\n or tag_sets != [{}]\n or max_staleness != -1):\n\n if \"$query\" not in spec:\n spec = SON([(\"$query\", spec)])\n spec[\"$readPreference\"] = read_preference.document\n return spec"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _convert_write_result(operation, command, result):\n\n # Based on _merge_legacy from bulk.py\n affected = result.get(\"n\", 0)\n res = {\"ok\": 1, \"n\": affected}\n errmsg = result.get(\"errmsg\", result.get(\"err\", \"\"))\n if errmsg:\n # The write was successful on at least the primary so don't return.\n if result.get(\"wtimeout\"):\n res[\"writeConcernError\"] = {\"errmsg\": errmsg,\n \"code\": 64,\n \"errInfo\": {\"wtimeout\": True}}\n else:\n # The write failed.\n error = {\"index\": 0,\n \"code\": result.get(\"code\", 8),\n \"errmsg\": errmsg}\n if \"errInfo\" in result:\n error[\"errInfo\"] = result[\"errInfo\"]\n res[\"writeErrors\"] = [error]\n return res\n if operation == \"insert\":\n # GLE result for insert is always 0 in most MongoDB versions.\n res[\"n\"] = len(command['documents'])\n elif operation == \"update\":\n if \"upserted\" in result:\n res[\"upserted\"] = [{\"index\": 0, \"_id\": result[\"upserted\"]}]\n # Versions of MongoDB before 2.6 don't return the _id for an\n # upsert if _id is not an ObjectId.\n elif result.get(\"updatedExisting\") is False and affected == 1:\n # If _id is in both the update document *and* the query spec\n # the update document _id takes precedence.\n update = command['updates'][0]\n _id = update[\"u\"].get(\"_id\", update[\"q\"].get(\"_id\"))\n res[\"upserted\"] = [{\"index\": 0, \"_id\": _id}]\n return res", "response": "Convert a legacy write result to write commmand format."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _gen_explain_command(\n coll, spec, projection, skip, limit, batch_size,\n options, read_concern):\n \"\"\"Generate an explain command document.\"\"\"\n cmd = _gen_find_command(\n coll, spec, projection, skip, limit, batch_size, options)\n if read_concern.level:\n return SON([('explain', cmd), ('readConcern', read_concern.document)])\n return SON([('explain', cmd)])", "response": "Generate an explain command document."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _gen_find_command(coll, spec, projection, skip, limit, batch_size,\n options, read_concern=DEFAULT_READ_CONCERN,\n collation=None):\n \"\"\"Generate a find command document.\"\"\"\n cmd = SON([('find', coll)])\n if '$query' in spec:\n cmd.update([(_MODIFIERS[key], val) if key in _MODIFIERS else (key, val)\n for key, val in spec.items()])\n if '$explain' in cmd:\n cmd.pop('$explain')\n if '$readPreference' in cmd:\n cmd.pop('$readPreference')\n else:\n cmd['filter'] = spec\n\n if projection:\n cmd['projection'] = projection\n if skip:\n cmd['skip'] = skip\n if limit:\n cmd['limit'] = abs(limit)\n if limit < 0:\n cmd['singleBatch'] = True\n if batch_size:\n cmd['batchSize'] = batch_size\n if read_concern.level:\n cmd['readConcern'] = read_concern.document\n if collation:\n cmd['collation'] = collation\n\n if options:\n cmd.update([(opt, True)\n for opt, val in _OPTIONS.items()\n if options & val])\n return cmd", "response": "Generate a find command document."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _gen_get_more_command(cursor_id, coll, batch_size, max_await_time_ms):\n cmd = SON([('getMore', cursor_id),\n ('collection', coll)])\n if batch_size:\n cmd['batchSize'] = batch_size\n if max_await_time_ms is not None:\n cmd['maxTimeMS'] = max_await_time_ms\n return cmd", "response": "Generate a getMore command document."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the last error from the last error in a namespace", "response": "def __last_error(namespace, args):\n \"\"\"Data to send to do a lastError.\n \"\"\"\n cmd = SON([(\"getlasterror\", 1)])\n cmd.update(args)\n splitns = namespace.split('.', 1)\n return query(0, splitns[0] + '.$cmd', 0, -1, cmd,\n None, DEFAULT_CODEC_OPTIONS)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntakes message data and adds a message header based on the operation.", "response": "def __pack_message(operation, data):\n \"\"\"Takes message data and adds a message header based on the operation.\n\n Returns the resultant message string.\n \"\"\"\n request_id = _randint()\n message = struct.pack(\"= max_cmd_size\n enough_documents = (idx >= max_write_batch_size)\n if enough_data or enough_documents:\n if not idx:\n write_op = \"insert\" if operation == _INSERT else None\n _raise_document_too_large(\n write_op, len(value), max_bson_size)\n result = send_message()\n results.append((idx_offset, result))\n if ordered and \"writeErrors\" in result:\n return results\n\n # Truncate back to the start of list elements\n buf.seek(list_start + 4)\n buf.truncate()\n idx_offset += idx\n idx = 0\n key = b'0'\n to_send = []\n buf.write(_BSONOBJ)\n buf.write(key)\n buf.write(_ZERO_8)\n buf.write(value)\n to_send.append(doc)\n idx += 1\n\n if not has_docs:\n raise InvalidOperation(\"cannot do an empty bulk write\")\n\n results.append((idx_offset, send_message()))\n return results", "response": "Execute a batch of insert update or delete commands."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a command document for this query.", "response": "def as_command(self):\n \"\"\"Return a find command document for this query.\n\n Should be called *after* get_message.\n \"\"\"\n if '$explain' in self.spec:\n self.name = 'explain'\n return _gen_explain_command(\n self.coll, self.spec, self.fields, self.ntoskip,\n self.limit, self.batch_size, self.flags,\n self.read_concern), self.db\n return _gen_find_command(\n self.coll, self.spec, self.fields, self.ntoskip, self.limit,\n self.batch_size, self.flags, self.read_concern,\n self.collation), self.db"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a message from the database.", "response": "def get_message(self, set_slave_ok, is_mongos, use_cmd=False):\n \"\"\"Get a query message, possibly setting the slaveOk bit.\"\"\"\n if set_slave_ok:\n # Set the slaveOk bit.\n flags = self.flags | 4\n else:\n flags = self.flags\n\n ns = _UJOIN % (self.db, self.coll)\n spec = self.spec\n\n if use_cmd:\n ns = _UJOIN % (self.db, \"$cmd\")\n spec = self.as_command()[0]\n ntoreturn = -1 # All DB commands return 1 document\n else:\n # OP_QUERY treats ntoreturn of -1 and 1 the same, return\n # one document and close the cursor. We have to use 2 for\n # batch size if 1 is specified.\n ntoreturn = self.batch_size == 1 and 2 or self.batch_size\n if self.limit:\n if ntoreturn:\n ntoreturn = min(self.limit, ntoreturn)\n else:\n ntoreturn = self.limit\n\n if is_mongos:\n spec = _maybe_add_read_preference(spec,\n self.read_preference)\n\n return query(flags, ns, self.ntoskip, ntoreturn,\n spec, self.fields, self.codec_options)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef as_command(self):\n return _gen_get_more_command(self.cursor_id, self.coll,\n self.ntoreturn,\n self.max_await_time_ms), self.db", "response": "Return a getMore command document for this query."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a getmore message.", "response": "def get_message(self, dummy0, dummy1, use_cmd=False):\n \"\"\"Get a getmore message.\"\"\"\n\n ns = _UJOIN % (self.db, self.coll)\n\n if use_cmd:\n ns = _UJOIN % (self.db, \"$cmd\")\n spec = self.as_command()[0]\n\n return query(0, ns, 0, -1, spec, None, self.codec_options)\n\n return get_more(ns, self.ntoreturn, self.cursor_id)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef legacy_write(self, request_id, msg, max_doc_size, acknowledged, docs):\n if self.publish:\n duration = datetime.datetime.now() - self.start_time\n cmd = self._start(request_id, docs)\n start = datetime.datetime.now()\n try:\n result = self.sock_info.legacy_write(\n request_id, msg, max_doc_size, acknowledged)\n if self.publish:\n duration = (datetime.datetime.now() - start) + duration\n if result is not None:\n reply = _convert_write_result(self.name, cmd, result)\n else:\n # Comply with APM spec.\n reply = {'ok': 1}\n self._succeed(request_id, reply, duration)\n except OperationFailure as exc:\n if self.publish:\n duration = (datetime.datetime.now() - start) + duration\n self._fail(\n request_id,\n _convert_write_result(\n self.name, cmd, exc.details),\n duration)\n raise\n finally:\n self.start_time = datetime.datetime.now()\n return result", "response": "A proxy for SocketInfo. legacy_write that handles event publishing."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _machine_bytes():\n machine_hash = hashlib.md5()\n if PY3:\n # gethostname() returns a unicode string in python 3.x\n # while update() requires a byte string.\n machine_hash.update(socket.gethostname().encode())\n else:\n # Calling encode() here will fail with non-ascii hostnames\n machine_hash.update(socket.gethostname())\n return machine_hash.digest()[0:3]", "response": "Get the machine portion of an ObjectId."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a dummy ObjectId instance with a specific generation time.", "response": "def from_datetime(cls, generation_time):\n \"\"\"Create a dummy ObjectId instance with a specific generation time.\n\n This method is useful for doing range queries on a field\n containing :class:`ObjectId` instances.\n\n .. warning::\n It is not safe to insert a document containing an ObjectId\n generated using this method. This method deliberately\n eliminates the uniqueness guarantee that ObjectIds\n generally provide. ObjectIds generated with this method\n should be used exclusively in queries.\n\n `generation_time` will be converted to UTC. Naive datetime\n instances will be treated as though they already contain UTC.\n\n An example using this helper to get documents where ``\"_id\"``\n was generated before January 1, 2010 would be:\n\n >>> gen_time = datetime.datetime(2010, 1, 1)\n >>> dummy_id = ObjectId.from_datetime(gen_time)\n >>> result = collection.find({\"_id\": {\"$lt\": dummy_id}})\n\n :Parameters:\n - `generation_time`: :class:`~datetime.datetime` to be used\n as the generation time for the resulting ObjectId.\n \"\"\"\n if generation_time.utcoffset() is not None:\n generation_time = generation_time - generation_time.utcoffset()\n timestamp = calendar.timegm(generation_time.timetuple())\n oid = struct.pack(\n \">i\", int(timestamp)) + b\"\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"\n return cls(oid)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_valid(cls, oid):\n if not oid:\n return False\n\n try:\n ObjectId(oid)\n return True\n except (InvalidId, TypeError):\n return False", "response": "Checks if a oid string is valid or not."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __generate(self):\n\n # 4 bytes current time\n oid = struct.pack(\">i\", int(time.time()))\n\n # 3 bytes machine\n oid += ObjectId._machine_bytes\n\n # 2 bytes pid\n oid += struct.pack(\">H\", os.getpid() % 0xFFFF)\n\n # 3 bytes inc\n with ObjectId._inc_lock:\n oid += struct.pack(\">i\", ObjectId._inc)[1:4]\n ObjectId._inc = (ObjectId._inc + 1) % 0xFFFFFF\n\n self.__id = oid", "response": "Generate a new value for this ObjectId."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __validate(self, oid):\n if isinstance(oid, ObjectId):\n self.__id = oid.binary\n # bytes or unicode in python 2, str in python 3\n elif isinstance(oid, string_type):\n if len(oid) == 24:\n try:\n self.__id = bytes_from_hex(oid)\n except (TypeError, ValueError):\n _raise_invalid_id(oid)\n else:\n _raise_invalid_id(oid)\n else:\n raise TypeError(\"id must be an instance of (bytes, %s, ObjectId), \"\n \"not %s\" % (text_type.__name__, type(oid)))", "response": "Validate and use the given id for this ObjectId."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_master_address(self, instance):\n\n master_address = instance.get(MASTER_ADDRESS)\n if master_address is None:\n master_address = instance.get(DEPRECATED_MASTER_ADDRESS)\n\n if master_address:\n self.log.warning(\n 'The use of `%s` is deprecated. Please use `%s` instead.'\n % (DEPRECATED_MASTER_ADDRESS, MASTER_ADDRESS)\n )\n else:\n raise Exception('URL for `%s` must be specified in the instance configuration' % MASTER_ADDRESS)\n\n return master_address", "response": "Get the master address from the instance configuration"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the request address build with proxy if necessary", "response": "def _get_request_url(self, instance, url):\n \"\"\"\n Get the request address, build with proxy if necessary\n \"\"\"\n parsed = urlparse(url)\n\n _url = url\n if not (parsed.netloc and parsed.scheme) and is_affirmative(instance.get('spark_proxy_enabled', False)):\n master_address = self._get_master_address(instance)\n _url = urljoin(master_address, parsed.path)\n\n self.log.debug('Request URL returned: %s', _url)\n return _url"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_running_apps(self, instance, requests_config):\n tags = instance.get('tags', [])\n if tags is None:\n tags = []\n master_address = self._get_master_address(instance)\n # Get the cluster name from the instance configuration\n cluster_name = instance.get('cluster_name')\n if cluster_name is None:\n raise Exception('The cluster_name must be specified in the instance configuration')\n tags.append('cluster_name:%s' % cluster_name)\n tags = list(set(tags))\n # Determine the cluster mode\n cluster_mode = instance.get(SPARK_CLUSTER_MODE)\n if cluster_mode is None:\n self.log.warning(\n 'The value for `spark_cluster_mode` was not set in the configuration. '\n 'Defaulting to \"%s\"' % SPARK_YARN_MODE\n )\n cluster_mode = SPARK_YARN_MODE\n\n if cluster_mode == SPARK_STANDALONE_MODE:\n # check for PRE-20\n pre20 = is_affirmative(instance.get(SPARK_PRE_20_MODE, False))\n return self._standalone_init(master_address, pre20, requests_config, tags)\n\n elif cluster_mode == SPARK_MESOS_MODE:\n running_apps = self._mesos_init(instance, master_address, requests_config, tags)\n return self._get_spark_app_ids(running_apps, requests_config, tags)\n\n elif cluster_mode == SPARK_YARN_MODE:\n running_apps = self._yarn_init(master_address, requests_config, tags)\n return self._get_spark_app_ids(running_apps, requests_config, tags)\n\n else:\n raise Exception('Invalid setting for %s. Received %s.' % (SPARK_CLUSTER_MODE, cluster_mode))", "response": "Get the running apps from the instance."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialize the standalone application list.", "response": "def _standalone_init(self, spark_master_address, pre_20_mode, requests_config, tags):\n \"\"\"\n Return a dictionary of {app_id: (app_name, tracking_url)} for the running Spark applications\n \"\"\"\n metrics_json = self._rest_request_to_json(\n spark_master_address, SPARK_MASTER_STATE_PATH, SPARK_STANDALONE_SERVICE_CHECK, requests_config, tags\n )\n\n running_apps = {}\n\n if metrics_json.get('activeapps'):\n for app in metrics_json['activeapps']:\n app_id = app.get('id')\n app_name = app.get('name')\n\n # Parse through the HTML to grab the application driver's link\n try:\n app_url = self._get_standalone_app_url(app_id, spark_master_address, requests_config, tags)\n\n if app_id and app_name and app_url:\n if pre_20_mode:\n self.log.debug('Getting application list in pre-20 mode')\n applist = self._rest_request_to_json(\n app_url, SPARK_APPS_PATH, SPARK_STANDALONE_SERVICE_CHECK, requests_config, tags\n )\n for appl in applist:\n aid = appl.get('id')\n aname = appl.get('name')\n running_apps[aid] = (aname, app_url)\n else:\n running_apps[app_id] = (app_name, app_url)\n except Exception:\n # it's possible for the requests to fail if the job\n # completed since we got the list of apps. Just continue\n pass\n\n # Report success after gathering metrics from Spark master\n self.service_check(\n SPARK_STANDALONE_SERVICE_CHECK,\n AgentCheck.OK,\n tags=['url:%s' % spark_master_address] + tags,\n message='Connection to Spark master \"%s\" was successful' % spark_master_address,\n )\n self.log.info(\"Returning running apps %s\" % running_apps)\n return running_apps"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a dictionary of {app_id: (app_name, tracking_url)} for running Spark applications.", "response": "def _mesos_init(self, instance, master_address, requests_config, tags):\n \"\"\"\n Return a dictionary of {app_id: (app_name, tracking_url)} for running Spark applications.\n \"\"\"\n running_apps = {}\n\n metrics_json = self._rest_request_to_json(\n master_address, MESOS_MASTER_APP_PATH, MESOS_SERVICE_CHECK, requests_config, tags\n )\n\n if metrics_json.get('frameworks'):\n for app_json in metrics_json.get('frameworks'):\n app_id = app_json.get('id')\n tracking_url = app_json.get('webui_url')\n app_name = app_json.get('name')\n\n if app_id and tracking_url and app_name:\n spark_ports = instance.get('spark_ui_ports')\n if spark_ports is None:\n # No filtering by port, just return all the frameworks\n running_apps[app_id] = (app_name, tracking_url)\n else:\n # Only return the frameworks running on the correct port\n tracking_url_port = urlparse(tracking_url).port\n if tracking_url_port in spark_ports:\n running_apps[app_id] = (app_name, tracking_url)\n\n # Report success after gathering all metrics from ResourceManager\n self.service_check(\n MESOS_SERVICE_CHECK,\n AgentCheck.OK,\n tags=['url:%s' % master_address] + tags,\n message='Connection to ResourceManager \"%s\" was successful' % master_address,\n )\n\n return running_apps"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _yarn_init(self, rm_address, requests_config, tags):\n running_apps = self._yarn_get_running_spark_apps(rm_address, requests_config, tags)\n\n # Report success after gathering all metrics from ResourceManager\n self.service_check(\n YARN_SERVICE_CHECK,\n AgentCheck.OK,\n tags=['url:%s' % rm_address] + tags,\n message='Connection to ResourceManager \"%s\" was successful' % rm_address,\n )\n\n return running_apps", "response": "Initialize Spark applications and return a dictionary of Spark application IDs and tracking URLs."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the standalone application URL from the Spark master.", "response": "def _get_standalone_app_url(self, app_id, spark_master_address, requests_config, tags):\n \"\"\"\n Return the application URL from the app info page on the Spark master.\n Due to a bug, we need to parse the HTML manually because we cannot\n fetch JSON data from HTTP interface.\n \"\"\"\n app_page = self._rest_request(\n spark_master_address,\n SPARK_MASTER_APP_PATH,\n SPARK_STANDALONE_SERVICE_CHECK,\n requests_config,\n tags,\n appId=app_id,\n )\n\n dom = BeautifulSoup(app_page.text, 'html.parser')\n app_detail_ui_links = dom.find_all('a', string='Application Detail UI')\n\n if app_detail_ui_links and len(app_detail_ui_links) == 1:\n return app_detail_ui_links[0].attrs['href']"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a dictionary of running Spark applications.", "response": "def _yarn_get_running_spark_apps(self, rm_address, requests_config, tags):\n \"\"\"\n Return a dictionary of {app_id: (app_name, tracking_url)} for running Spark applications.\n\n The `app_id` returned is that of the YARN application. This will eventually be mapped into\n a Spark application ID.\n \"\"\"\n metrics_json = self._rest_request_to_json(\n rm_address,\n YARN_APPS_PATH,\n YARN_SERVICE_CHECK,\n requests_config,\n tags,\n states=APPLICATION_STATES,\n applicationTypes=YARN_APPLICATION_TYPES,\n )\n\n running_apps = {}\n\n if metrics_json.get('apps'):\n if metrics_json['apps'].get('app') is not None:\n\n for app_json in metrics_json['apps']['app']:\n app_id = app_json.get('id')\n tracking_url = app_json.get('trackingUrl')\n app_name = app_json.get('name')\n\n if app_id and tracking_url and app_name:\n running_apps[app_id] = (app_name, tracking_url)\n\n return running_apps"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a dictionary of Spark application IDs and tracking URLs for each Spark application.", "response": "def _get_spark_app_ids(self, running_apps, requests_config, tags):\n \"\"\"\n Traverses the Spark application master in YARN to get a Spark application ID.\n\n Return a dictionary of {app_id: (app_name, tracking_url)} for Spark applications\n \"\"\"\n spark_apps = {}\n for app_id, (app_name, tracking_url) in iteritems(running_apps):\n response = self._rest_request_to_json(\n tracking_url, SPARK_APPS_PATH, SPARK_SERVICE_CHECK, requests_config, tags\n )\n\n for app in response:\n app_id = app.get('id')\n app_name = app.get('name')\n\n if app_id and app_name:\n spark_apps[app_id] = (app_name, tracking_url)\n\n return spark_apps"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget metrics for each Spark job.", "response": "def _spark_job_metrics(self, instance, running_apps, addl_tags, requests_config):\n \"\"\"\n Get metrics for each Spark job.\n \"\"\"\n for app_id, (app_name, tracking_url) in iteritems(running_apps):\n\n base_url = self._get_request_url(instance, tracking_url)\n response = self._rest_request_to_json(\n base_url, SPARK_APPS_PATH, SPARK_SERVICE_CHECK, requests_config, addl_tags, app_id, 'jobs'\n )\n\n for job in response:\n\n status = job.get('status')\n\n tags = ['app_name:%s' % str(app_name)]\n tags.extend(addl_tags)\n tags.append('status:%s' % str(status).lower())\n\n self._set_metrics_from_json(tags, job, SPARK_JOB_METRICS)\n self._set_metric('spark.job.count', COUNT, 1, tags)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting metrics for each Spark executor.", "response": "def _spark_executor_metrics(self, instance, running_apps, addl_tags, requests_config):\n \"\"\"\n Get metrics for each Spark executor.\n \"\"\"\n for app_id, (app_name, tracking_url) in iteritems(running_apps):\n\n base_url = self._get_request_url(instance, tracking_url)\n response = self._rest_request_to_json(\n base_url, SPARK_APPS_PATH, SPARK_SERVICE_CHECK, requests_config, addl_tags, app_id, 'executors'\n )\n\n tags = ['app_name:%s' % str(app_name)]\n tags.extend(addl_tags)\n\n for executor in response:\n if executor.get('id') == 'driver':\n self._set_metrics_from_json(tags, executor, SPARK_DRIVER_METRICS)\n else:\n self._set_metrics_from_json(tags, executor, SPARK_EXECUTOR_METRICS)\n\n if len(response):\n self._set_metric('spark.executor.count', COUNT, len(response), tags)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets metrics for each Spark RDD.", "response": "def _spark_rdd_metrics(self, instance, running_apps, addl_tags, requests_config):\n \"\"\"\n Get metrics for each Spark RDD.\n \"\"\"\n for app_id, (app_name, tracking_url) in iteritems(running_apps):\n\n base_url = self._get_request_url(instance, tracking_url)\n response = self._rest_request_to_json(\n base_url, SPARK_APPS_PATH, SPARK_SERVICE_CHECK, requests_config, addl_tags, app_id, 'storage/rdd'\n )\n\n tags = ['app_name:%s' % str(app_name)]\n tags.extend(addl_tags)\n\n for rdd in response:\n self._set_metrics_from_json(tags, rdd, SPARK_RDD_METRICS)\n\n if len(response):\n self._set_metric('spark.rdd.count', COUNT, len(response), tags)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting metrics for each application streaming statistics.", "response": "def _spark_streaming_statistics_metrics(self, instance, running_apps, addl_tags, requests_config):\n \"\"\"\n Get metrics for each application streaming statistics.\n \"\"\"\n for app_id, (app_name, tracking_url) in iteritems(running_apps):\n try:\n base_url = self._get_request_url(instance, tracking_url)\n response = self._rest_request_to_json(\n base_url,\n SPARK_APPS_PATH,\n SPARK_SERVICE_CHECK,\n requests_config,\n addl_tags,\n app_id,\n 'streaming/statistics',\n )\n self.log.debug('streaming/statistics: %s', response)\n tags = ['app_name:%s' % str(app_name)]\n tags.extend(addl_tags)\n\n # NOTE: response is a dict\n self._set_metrics_from_json(tags, response, SPARK_STREAMING_STATISTICS_METRICS)\n except HTTPError as e:\n # NOTE: If api call returns response 404\n # then it means that the application is not a streaming application, we should skip metric submission\n if e.response.status_code != 404:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _set_metric(self, metric_name, metric_type, value, tags=None):\n if tags is None:\n tags = []\n if metric_type == GAUGE:\n self.gauge(metric_name, value, tags=tags)\n elif metric_type == COUNT:\n self.count(metric_name, value, tags=tags)\n elif metric_type == MONOTONIC_COUNT:\n self.monotonic_count(metric_name, value, tags=tags)\n else:\n self.log.error('Metric type \"{}\" unknown'.format(metric_type))", "response": "Set a metric in the log."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform a request to Spark and return the response.", "response": "def _rest_request(self, url, object_path, service_name, requests_config, tags, *args, **kwargs):\n \"\"\"\n Query the given URL and return the response\n \"\"\"\n service_check_tags = ['url:%s' % self._get_url_base(url)] + tags\n\n if object_path:\n url = self._join_url_dir(url, object_path)\n\n # Load SSL configuration, if available.\n # ssl_verify can be a bool or a string\n # (http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification)\n if isinstance(requests_config.ssl_verify, bool) or isinstance(requests_config.ssl_verify, str):\n verify = requests_config.ssl_verify\n else:\n verify = None\n if requests_config.ssl_cert and requests_config.ssl_key:\n cert = (requests_config.ssl_cert, requests_config.ssl_key)\n elif requests_config.ssl_cert:\n cert = requests_config.ssl_cert\n else:\n cert = None\n\n # Add args to the url\n if args:\n for directory in args:\n url = self._join_url_dir(url, directory)\n\n # Add kwargs as arguments\n if kwargs:\n query = '&'.join(['{0}={1}'.format(key, value) for key, value in iteritems(kwargs)])\n url = urljoin(url, '?' + query)\n\n old_keytab_path = None\n if requests_config.kerberos_keytab:\n old_keytab_path = os.getenv('KRB5_CLIENT_KTNAME')\n os.environ['KRB5_CLIENT_KTNAME'] = requests_config.kerberos_keytab\n\n try:\n self.log.debug('Spark check URL: %s' % url)\n response = requests.get(url, auth=requests_config.auth, verify=verify, cert=cert)\n\n response.raise_for_status()\n\n except Timeout as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message='Request timeout: {0}, {1}'.format(url, e),\n )\n raise\n\n except (HTTPError, InvalidURL, ConnectionError) as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message='Request failed: {0}, {1}'.format(url, e),\n )\n raise\n\n except ValueError as e:\n self.service_check(service_name, AgentCheck.CRITICAL, tags=service_check_tags, message=str(e))\n raise\n\n else:\n return response\n\n finally:\n if old_keytab_path is not None:\n os.environ['KRB5_CLIENT_KTNAME'] = old_keytab_path"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nqueries the given URL and return the JSON response", "response": "def _rest_request_to_json(self, address, object_path, service_name, requests_config, tags, *args, **kwargs):\n \"\"\"\n Query the given URL and return the JSON response\n \"\"\"\n response = self._rest_request(address, object_path, service_name, requests_config, tags, *args, **kwargs)\n\n try:\n response_json = response.json()\n\n except JSONDecodeError as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=['url:%s' % self._get_url_base(address)] + tags,\n message='JSON Parse failed: {0}'.format(e),\n )\n raise\n\n return response_json"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the base of a URL.", "response": "def _get_url_base(cls, url):\n \"\"\"\n Return the base of a URL\n \"\"\"\n s = urlsplit(url)\n return urlunsplit([s.scheme, s.netloc, '', '', ''])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_event_tags_from_dn(dn):\n tags = []\n node = get_node_from_dn(dn)\n if node:\n tags.append(\"node:\" + node)\n app = get_app_from_dn(dn)\n if app:\n tags.append(\"app:\" + app)\n bd = get_bd_from_dn(dn)\n if bd:\n tags.append(\"bd:\" + bd)\n cep = get_cep_from_dn(dn)\n if cep:\n tags.append(\"mac:\" + cep)\n ip = get_ip_from_dn(dn)\n if ip:\n tags.append(\"ip:\" + ip)\n epg = get_epg_from_dn(dn)\n if epg:\n tags.append(\"epg:\" + epg)\n return tags", "response": "Returns a list of event tags from a DN."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if a metric value is zero", "response": "def check_metric_can_be_zero(metric_name, metric_value, json_attributes):\n \"\"\"\n When a counter is reset, don't send a zero because it will look bad on the graphs\n This checks if the zero makes sense or not\n \"\"\"\n if \"last\" in metric_name.lower():\n return True\n if not metric_value:\n return False\n try:\n if metric_value == 0 or metric_value == \"0\" or metric_value == \"0.000000\" or float(metric_value) == 0.0:\n if not json_attributes or not json_attributes.get('cnt'):\n return False\n if json_attributes.get('cnt') == \"0\" or json_attributes.get('cnt') == 0:\n return False\n except ValueError:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create(ctx, name, integration_type, location, non_interactive, quiet, dry_run):\n repo_choice = ctx.obj['repo_choice']\n root = resolve_path(location) if location else get_root()\n path_sep = os.path.sep\n\n integration_dir = os.path.join(root, normalize_package_name(name))\n if os.path.exists(integration_dir):\n abort('Path `{}` already exists!'.format(integration_dir))\n\n template_fields = {}\n if repo_choice != 'core' and not non_interactive and not dry_run:\n template_fields['author'] = click.prompt('Your name')\n template_fields['email'] = click.prompt('Your email')\n template_fields['email_packages'] = template_fields['email']\n click.echo()\n\n config = construct_template_fields(name, repo_choice, **template_fields)\n\n files = create_template_files(integration_type, root, config, read=not dry_run)\n file_paths = [file.file_path.replace('{}{}'.format(root, path_sep), '', 1) for file in files]\n\n path_tree = tree()\n for file_path in file_paths:\n branch = path_tree\n\n for part in file_path.split(path_sep):\n branch = branch[part]\n\n if dry_run:\n if quiet:\n echo_info('Will create `{}`'.format(integration_dir))\n else:\n echo_info('Will create in `{}`:'.format(root))\n display_path_tree(path_tree)\n return\n\n for file in files:\n file.write()\n\n if quiet:\n echo_info('Created `{}`'.format(integration_dir))\n else:\n echo_info('Created in `{}`:'.format(root))\n display_path_tree(path_tree)", "response": "Create a new scaffolding for a new integration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef freeze(o):\n if isinstance(o, (tuple, list)):\n return tuple(sorted(freeze(e) for e in o))\n\n if isinstance(o, dict):\n return tuple(sorted((k, freeze(v)) for k, v in iteritems(o)))\n\n if isinstance(o, (set, frozenset)):\n return tuple(sorted(freeze(e) for e in o))\n\n return o", "response": "Freezes any mutable object including dictionaries and lists for hashing."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a decimal. Decimal to BID.", "response": "def _decimal_to_128(value):\n \"\"\"Converts a decimal.Decimal to BID (high bits, low bits).\n\n :Parameters:\n - `value`: An instance of decimal.Decimal\n \"\"\"\n with decimal.localcontext(_DEC128_CTX) as ctx:\n value = ctx.create_decimal(value)\n\n if value.is_infinite():\n return _NINF if value.is_signed() else _PINF\n\n sign, digits, exponent = value.as_tuple()\n\n if value.is_nan():\n if digits:\n raise ValueError(\"NaN with debug payload is not supported\")\n if value.is_snan():\n return _NSNAN if value.is_signed() else _PSNAN\n return _NNAN if value.is_signed() else _PNAN\n\n significand = int(\"\".join([str(digit) for digit in digits]))\n bit_length = _bit_length(significand)\n\n high = 0\n low = 0\n for i in range(min(64, bit_length)):\n if significand & (1 << i):\n low |= 1 << i\n\n for i in range(64, bit_length):\n if significand & (1 << i):\n high |= 1 << (i - 64)\n\n biased_exponent = exponent + _EXPONENT_BIAS\n\n if high >> 49 == 1:\n high = high & 0x7fffffffffff\n high |= _EXPONENT_MASK\n high |= (biased_exponent & 0x3fff) << 47\n else:\n high |= biased_exponent << 49\n\n if sign:\n high |= _SIGN\n\n return high, low"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns an instance of decimal. Decimal for thisCOOKIE.", "response": "def to_decimal(self):\n \"\"\"Returns an instance of :class:`decimal.Decimal` for this\n :class:`Decimal128`.\n \"\"\"\n high = self.__high\n low = self.__low\n sign = 1 if (high & _SIGN) else 0\n\n if (high & _SNAN) == _SNAN:\n return decimal.Decimal((sign, (), 'N'))\n elif (high & _NAN) == _NAN:\n return decimal.Decimal((sign, (), 'n'))\n elif (high & _INF) == _INF:\n return decimal.Decimal((sign, (), 'F'))\n\n if (high & _EXPONENT_MASK) == _EXPONENT_MASK:\n exponent = ((high & 0x1fffe00000000000) >> 47) - _EXPONENT_BIAS\n return decimal.Decimal((sign, (0,), exponent))\n else:\n exponent = ((high & 0x7fff800000000000) >> 49) - _EXPONENT_BIAS\n\n arr = bytearray(15)\n mask = 0x00000000000000ff\n for i in range(14, 6, -1):\n arr[i] = (low & mask) >> ((14 - i) << 3)\n mask = mask << 8\n\n mask = 0x00000000000000ff\n for i in range(6, 0, -1):\n arr[i] = (high & mask) >> ((6 - i) << 3)\n mask = mask << 8\n\n mask = 0x0001000000000000\n arr[0] = (high & mask) >> 48\n\n # Have to convert bytearray to bytes for python 2.6.\n # cdecimal only accepts a tuple for digits.\n digits = tuple(\n int(digit) for digit in str(_from_bytes(bytes(arr), 'big')))\n\n with decimal.localcontext(_DEC128_CTX) as ctx:\n return ctx.create_decimal((sign, digits, exponent))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating an instance of Decimal128 from binary integer value.", "response": "def from_bid(cls, value):\n \"\"\"Create an instance of :class:`Decimal128` from Binary Integer\n Decimal string.\n\n :Parameters:\n - `value`: 16 byte string (128-bit IEEE 754-2008 decimal floating\n point in Binary Integer Decimal (BID) format).\n \"\"\"\n if not isinstance(value, bytes):\n raise TypeError(\"value must be an instance of bytes\")\n if len(value) != 16:\n raise ValueError(\"value must be exactly 16 bytes\")\n return cls((_UNPACK_64(value[8:])[0], _UNPACK_64(value[:8])[0]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _compile_tag_re(self):\n queue_tag_list = []\n for regex_str, tags in iteritems(self._queue_tag_re):\n try:\n queue_tag_list.append([re.compile(regex_str), [t.strip() for t in tags.split(',')]])\n except TypeError:\n log.warning('{} is not a valid regular expression and will be ignored'.format(regex_str))\n return queue_tag_list", "response": "Compile regex strings from queue_tag_re option and return list of compiled regex / tag pairs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndetermining the server type from an ismaster response.", "response": "def _get_server_type(doc):\n \"\"\"Determine the server type from an ismaster response.\"\"\"\n if not doc.get('ok'):\n return SERVER_TYPE.Unknown\n\n if doc.get('isreplicaset'):\n return SERVER_TYPE.RSGhost\n elif doc.get('setName'):\n if doc.get('hidden'):\n return SERVER_TYPE.RSOther\n elif doc.get('ismaster'):\n return SERVER_TYPE.RSPrimary\n elif doc.get('secondary'):\n return SERVER_TYPE.RSSecondary\n elif doc.get('arbiterOnly'):\n return SERVER_TYPE.RSArbiter\n else:\n return SERVER_TYPE.RSOther\n elif doc.get('msg') == 'isdbgrid':\n return SERVER_TYPE.Mongos\n else:\n return SERVER_TYPE.Standalone"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef all_hosts(self):\n return set(imap(common.clean_node, itertools.chain(\n self._doc.get('hosts', []),\n self._doc.get('passives', []),\n self._doc.get('arbiters', []))))", "response": "List of hosts passives and arbiters known to this server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _create_gitlab_runner_prometheus_instance(self, instance, init_config):\n # Mapping from Prometheus metrics names to Datadog ones\n # For now it's a 1:1 mapping\n allowed_metrics = init_config.get('allowed_metrics')\n if allowed_metrics is None:\n raise CheckException(\"At least one metric must be whitelisted in `allowed_metrics`.\")\n\n gitlab_runner_instance = deepcopy(instance)\n\n # gitlab_runner uses 'prometheus_endpoint' and not 'prometheus_url', so we have to rename the key\n gitlab_runner_instance['prometheus_url'] = instance.get('prometheus_endpoint', None)\n\n gitlab_runner_instance.update(\n {\n 'namespace': 'gitlab_runner',\n 'metrics': allowed_metrics,\n # Defaults that were set when gitlab_runner was based on PrometheusCheck\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n\n return gitlab_runner_instance", "response": "Create a gitlab_runner instance for the Prometheus check."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_jmx_data(self, instance, jmx_address, tags):\n response = self._rest_request_to_json(\n instance, jmx_address, self.JMX_PATH, {'qry': self.HDFS_DATANODE_BEAN_NAME}, tags=tags\n )\n beans = response.get('beans', [])\n return beans", "response": "Get namenode beans data from JMX endpoint"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprocess HDFS Datanode metrics from given beans and tags", "response": "def _hdfs_datanode_metrics(self, beans, tags):\n \"\"\"\n Process HDFS Datanode metrics from given beans\n \"\"\"\n # Only get the first bean\n bean = next(iter(beans))\n bean_name = bean.get('name')\n\n self.log.debug(\"Bean name retrieved: {}\".format(bean_name))\n\n for metric, (metric_name, metric_type) in iteritems(self.HDFS_METRICS):\n metric_value = bean.get(metric)\n if metric_value is not None:\n self._set_metric(metric_name, metric_type, metric_value, tags)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef requirements(ctx):\n echo_info('Freezing check releases')\n checks = get_valid_checks()\n checks.remove('datadog_checks_dev')\n\n entries = []\n for check in checks:\n if check in AGENT_V5_ONLY:\n echo_info('Check `{}` is only shipped with Agent 5, skipping'.format(check))\n continue\n\n try:\n version = get_version_string(check)\n entries.append('{}\\n'.format(get_agent_requirement_line(check, version)))\n except Exception as e:\n echo_failure('Error generating line: {}'.format(e))\n continue\n\n lines = sorted(entries)\n\n req_file = get_agent_release_requirements()\n write_file_lines(req_file, lines)\n echo_success('Successfully wrote to `{}`!'.format(req_file))", "response": "Write the requirements - agent - release. txt file at the root of the repo\n ."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _nest_payload(self, keys, payload):\n if len(keys) == 0:\n return payload\n\n return {keys[0]: self._nest_payload(keys[1:], payload)}", "response": "Nest a payload in a dict under the keys contained in keys"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of tags from integrations - core representing an Agent release sorted by recent first.", "response": "def get_agent_tags(since, to):\n \"\"\"\n Return a list of tags from integrations-core representing an Agent release,\n sorted by more recent first.\n \"\"\"\n agent_tags = sorted(parse_version_info(t) for t in git_tag_list(r'^\\d+\\.\\d+\\.\\d+$'))\n\n # default value for `to` is the latest tag\n if to:\n to = parse_version_info(to)\n else:\n to = agent_tags[-1]\n\n since = parse_version_info(since)\n\n # filter out versions according to the interval [since, to]\n agent_tags = [t for t in agent_tags if since <= t <= to]\n\n # reverse so we have descendant order\n return [str(t) for t in reversed(agent_tags)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef run(ctx, args):\n if not args:\n click.echo(ctx.get_help())\n return\n\n with chdir(get_root()):\n result = run_command(args)\n\n ctx.exit(result.code)", "response": "Run commands in the proper repo."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nnormalize the server URL.", "response": "def _normalize_server_url(self, server):\n \"\"\"\n Check if the server URL starts with a HTTP or HTTPS scheme, fall back to http if not present\n \"\"\"\n server = server if server.startswith((\"http://\", \"https://\")) else \"http://{}\".format(server)\n return server"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_es_version(self, config):\n try:\n data = self._get_data(config.url, config, send_sc=False)\n # pre-release versions of elasticearch are suffixed with -rcX etc..\n # peel that off so that the map below doesn't error out\n version = data['version']['number'].split('-')[0]\n version = [int(p) for p in version.split('.')[0:3]]\n except AuthenticationError:\n raise\n except Exception as e:\n self.warning(\"Error while trying to get Elasticsearch version from %s %s\" % (config.url, str(e)))\n version = [1, 0, 0]\n\n self.service_metadata('version', version)\n self.log.debug(\"Elasticsearch version is %s\" % version)\n return version", "response": "Get the running version of elasticsearch."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\njoin base url with url.", "response": "def _join_url(self, base, url, admin_forwarder=False):\n \"\"\"\n overrides `urlparse.urljoin` since it removes base url path\n https://docs.python.org/2/library/urlparse.html#urlparse.urljoin\n \"\"\"\n if admin_forwarder:\n return base + url\n else:\n return urljoin(base, url)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_urls(self, version, cluster_stats):\n pshard_stats_url = \"/_stats\"\n health_url = \"/_cluster/health\"\n\n if version >= [0, 90, 10]:\n pending_tasks_url = \"/_cluster/pending_tasks\"\n stats_url = \"/_nodes/stats\" if cluster_stats else \"/_nodes/_local/stats\"\n if version < [5, 0, 0]:\n # version 5 errors out if the `all` parameter is set\n stats_url += \"?all=true\"\n else:\n # legacy\n pending_tasks_url = None\n stats_url = \"/_cluster/nodes/stats?all=true\" if cluster_stats else \"/_cluster/nodes/_local/stats?all=true\"\n\n return health_url, stats_url, pshard_stats_url, pending_tasks_url", "response": "Compute the URLs we need to hit depending on the running ES version."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_data(self, url, config, send_sc=True):\n # Load basic authentication configuration, if available.\n if config.username and config.password:\n auth = (config.username, config.password)\n else:\n auth = None\n\n # Load SSL configuration, if available.\n # ssl_verify can be a bool or a string\n # (http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification)\n if isinstance(config.ssl_verify, bool) or isinstance(config.ssl_verify, str):\n verify = config.ssl_verify\n else:\n verify = None\n\n if config.ssl_cert:\n if config.ssl_key:\n cert = (config.ssl_cert, config.ssl_key)\n else:\n cert = config.ssl_cert\n else:\n cert = None\n\n resp = None\n try:\n resp = requests.get(\n url, timeout=config.timeout, headers=headers(self.agentConfig), auth=auth, verify=verify, cert=cert\n )\n resp.raise_for_status()\n except Exception as e:\n # this means we've hit a particular kind of auth error that means the config is broken\n if resp and resp.status_code == 400:\n raise AuthenticationError(\"The ElasticSearch credentials are incorrect\")\n\n if send_sc:\n self.service_check(\n self.SERVICE_CHECK_CONNECT_NAME,\n AgentCheck.CRITICAL,\n message=\"Error {} when hitting {}\".format(e, url),\n tags=config.service_check_tags,\n )\n raise\n\n self.log.debug(\"request to url {} returned: {}\".format(url, resp))\n\n return resp.json()", "response": "Get data from an ElasticSearch URL."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess a metric from the data dictionary.", "response": "def _process_metric(self, data, metric, xtype, path, xform=None, tags=None, hostname=None):\n \"\"\"\n data: dictionary containing all the stats\n metric: datadog metric\n path: corresponding path in data, flattened, e.g. thread_pool.bulk.queue\n xform: a lambda to apply to the numerical value\n \"\"\"\n value = data\n\n # Traverse the nested dictionaries\n for key in path.split('.'):\n if value is not None:\n value = value.get(key)\n else:\n break\n\n if value is not None:\n if xform:\n value = xform(value)\n if xtype == \"gauge\":\n self.gauge(metric, value, tags=tags, hostname=hostname)\n else:\n self.rate(metric, value, tags=tags, hostname=hostname)\n else:\n self.log.debug(\"Metric not found: %s -> %s\", path, metric)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nresets the state and uid set.", "response": "def reset(self):\n \"\"\"\n Resets state and uid set. To be called asap to free memory\n \"\"\"\n self.reached_limit = False\n self.count = 0\n self.seen.clear()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_reached(self, uid=None):\n if self.reached_limit:\n return True\n\n if uid:\n if uid in self.seen:\n return False\n self.count += 1\n self.seen.add(uid)\n else:\n self.count += 1\n\n if self.count > self.limit:\n if self.warning:\n self.warning(\n \"Check {} exceeded limit of {} {}, ignoring next ones\".format(\n self.check_name, self.limit, self.name\n )\n )\n self.reached_limit = True\n return True\n return False", "response": "This method is called when the count of the object is exceeded the limit."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the highwater offsets for each topic and partition.", "response": "def _get_broker_offsets(self, instance, topics):\n \"\"\"\n Fetch highwater offsets for each topic/partition from Kafka cluster.\n\n Do this for all partitions in the cluster because even if it has no\n consumers, we may want to measure whether producers are successfully\n producing. No need to limit this for performance because fetching broker\n offsets from Kafka is a relatively inexpensive operation.\n\n Sends one OffsetRequest per broker to get offsets for all partitions\n where that broker is the leader:\n https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI(AKAListOffset)\n\n Can we cleanup connections on agent restart?\n Brokers before 0.9 - accumulate stale connections on restarts.\n In 0.9 Kafka added connections.max.idle.ms\n https://issues.apache.org/jira/browse/KAFKA-1282\n \"\"\"\n\n # Connect to Kafka\n highwater_offsets = {}\n topic_partitions_without_a_leader = []\n topics_to_fetch = defaultdict(set)\n cli = self._get_kafka_client(instance)\n\n for topic, partitions in iteritems(topics):\n # if no partitions are provided\n # we're falling back to all available partitions (?)\n if len(partitions) == 0:\n partitions = cli.cluster.available_partitions_for_topic(topic)\n topics_to_fetch[topic].update(partitions)\n\n leader_tp = defaultdict(lambda: defaultdict(set))\n for topic, partitions in iteritems(topics_to_fetch):\n for partition in partitions:\n partition_leader = cli.cluster.leader_for_partition(TopicPartition(topic, partition))\n if partition_leader is not None and partition_leader >= 0:\n leader_tp[partition_leader][topic].add(partition)\n\n max_offsets = 1\n for node_id, tps in iteritems(leader_tp):\n # Construct the OffsetRequest\n request = OffsetRequest[0](\n replica_id=-1,\n topics=[\n (topic, [(partition, OffsetResetStrategy.LATEST, max_offsets) for partition in partitions])\n for topic, partitions in iteritems(tps)\n ],\n )\n\n response = self._make_blocking_req(cli, request, node_id=node_id)\n offsets, unled = self._process_highwater_offsets(response)\n highwater_offsets.update(offsets)\n topic_partitions_without_a_leader.extend(unled)\n\n return highwater_offsets, list(set(topic_partitions_without_a_leader))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_zk_path_children(self, zk_conn, zk_path, name_for_error):\n children = []\n try:\n children = zk_conn.get_children(zk_path)\n except NoNodeError:\n self.log.info('No zookeeper node at %s', zk_path)\n except Exception:\n self.log.exception('Could not read %s from %s', name_for_error, zk_path)\n return children", "response": "Fetch child nodes for a given Zookeeper path."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfetch consumer group offsets from Zookeeper.", "response": "def _get_zk_consumer_offsets(self, zk_hosts_ports, consumer_groups=None, zk_prefix=''):\n \"\"\"\n Fetch Consumer Group offsets from Zookeeper.\n\n Also fetch consumer_groups, topics, and partitions if not\n already specified in consumer_groups.\n\n :param dict consumer_groups: The consumer groups, topics, and partitions\n that you want to fetch offsets for. If consumer_groups is None, will\n fetch offsets for all consumer_groups. For examples of what this\n dict can look like, see _validate_explicit_consumer_groups().\n \"\"\"\n zk_consumer_offsets = {}\n\n # Construct the Zookeeper path pattern\n # /consumers/[groupId]/offsets/[topic]/[partitionId]\n zk_path_consumer = zk_prefix + '/consumers/'\n zk_path_topic_tmpl = zk_path_consumer + '{group}/offsets/'\n zk_path_partition_tmpl = zk_path_topic_tmpl + '{topic}/'\n\n zk_conn = KazooClient(zk_hosts_ports, timeout=self._zk_timeout)\n zk_conn.start()\n try:\n if consumer_groups is None:\n # If consumer groups aren't specified, fetch them from ZK\n consumer_groups = {\n consumer_group: None\n for consumer_group in self._get_zk_path_children(zk_conn, zk_path_consumer, 'consumer groups')\n }\n\n for consumer_group, topics in iteritems(consumer_groups):\n if topics is None:\n # If topics are't specified, fetch them from ZK\n zk_path_topics = zk_path_topic_tmpl.format(group=consumer_group)\n topics = {topic: None for topic in self._get_zk_path_children(zk_conn, zk_path_topics, 'topics')}\n consumer_groups[consumer_group] = topics\n\n for topic, partitions in iteritems(topics):\n if partitions is not None:\n partitions = set(partitions) # defend against bad user input\n else:\n # If partitions aren't specified, fetch them from ZK\n zk_path_partitions = zk_path_partition_tmpl.format(group=consumer_group, topic=topic)\n # Zookeeper returns the partition IDs as strings because\n # they are extracted from the node path\n partitions = [\n int(x) for x in self._get_zk_path_children(zk_conn, zk_path_partitions, 'partitions')\n ]\n consumer_groups[consumer_group][topic] = partitions\n\n # Fetch consumer offsets for each partition from ZK\n for partition in partitions:\n zk_path = (zk_path_partition_tmpl + '{partition}/').format(\n group=consumer_group, topic=topic, partition=partition\n )\n try:\n consumer_offset = int(zk_conn.get(zk_path)[0])\n key = (consumer_group, topic, partition)\n zk_consumer_offsets[key] = consumer_offset\n except NoNodeError:\n self.log.info('No zookeeper node at %s', zk_path)\n except Exception:\n self.log.exception('Could not read consumer offset from %s', zk_path)\n finally:\n try:\n zk_conn.stop()\n zk_conn.close()\n except Exception:\n self.log.exception('Error cleaning up Zookeeper connection')\n return zk_consumer_offsets, consumer_groups"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_kafka_consumer_offsets(self, instance, consumer_groups):\n consumer_offsets = {}\n topics = defaultdict(set)\n\n cli = self._get_kafka_client(instance)\n\n for consumer_group, topic_partitions in iteritems(consumer_groups):\n try:\n coordinator_id = self._get_group_coordinator(cli, consumer_group)\n if coordinator_id:\n offsets = self._get_consumer_offsets(cli, consumer_group, topic_partitions, coordinator_id)\n else:\n offsets = self._get_consumer_offsets(cli, consumer_group, topic_partitions)\n self.log.info(\"unable to find group coordinator for %s\", consumer_group)\n\n for (topic, partition), offset in iteritems(offsets):\n topics[topic].update([partition])\n key = (consumer_group, topic, partition)\n consumer_offsets[key] = offset\n except Exception:\n self.log.exception('Could not read consumer offsets from kafka.')\n\n return consumer_offsets, topics", "response": "retrieve consumer offsets via the new consumer api"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _validate_explicit_consumer_groups(cls, val):\n assert isinstance(val, dict)\n for consumer_group, topics in iteritems(val):\n assert isinstance(consumer_group, string_types)\n # topics are optional\n assert isinstance(topics, dict) or topics is None\n if topics is not None:\n for topic, partitions in iteritems(topics):\n assert isinstance(topic, string_types)\n # partitions are optional\n assert isinstance(partitions, (list, tuple)) or partitions is None\n if partitions is not None:\n for partition in partitions:\n assert isinstance(partition, int)", "response": "Validate any explicitly specified consumer groups."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nemits an event to the Datadog Event Stream.", "response": "def _send_event(self, title, text, tags, event_type, aggregation_key, severity='info'):\n \"\"\"Emit an event to the Datadog Event Stream.\"\"\"\n event_dict = {\n 'timestamp': int(time()),\n 'source_type_name': self.SOURCE_TYPE_NAME,\n 'msg_title': title,\n 'event_type': event_type,\n 'alert_type': severity,\n 'msg_text': text,\n 'tags': tags,\n 'aggregation_key': aggregation_key,\n }\n self.event(event_dict)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef display_package_changes(pre_packages, post_packages, indent=''):\n # use package name to determine what's changed\n pre_package_names = {p.name: p for p in pre_packages}\n post_package_names = {p.name: p for p in post_packages}\n\n added = set(post_package_names.keys()) - set(pre_package_names.keys())\n removed = set(pre_package_names.keys()) - set(post_package_names.keys())\n changed_maybe = set(pre_package_names.keys()) & set(post_package_names.keys())\n\n changed = []\n for package_name in sorted(changed_maybe):\n if pre_package_names[package_name] != post_package_names[package_name]:\n changed.append((pre_package_names[package_name], post_package_names[package_name]))\n\n if not (added or removed or changed):\n echo_info('{}No changes'.format(indent))\n\n if added:\n echo_success('{}Added packages:'.format(indent))\n for package_name in sorted(added):\n echo_info('{} {}'.format(indent, post_package_names[package_name]))\n\n if removed:\n echo_failure('{}Removed packages:'.format(indent))\n for package_name in sorted(removed):\n echo_info('{} {}'.format(indent, pre_package_names[package_name]))\n\n if changed:\n echo_warning('{}Changed packages:'.format(indent))\n for pre, post in changed:\n echo_info('{} {} -> {}'.format(indent, pre, post))", "response": "Print packages that have been added removed or changed."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nresolving transient dependencies for any number of checks.", "response": "def resolve(checks, lazy, quiet):\n \"\"\"Resolve transient dependencies for any number of checks.\n If you want to do this en masse, put `all`.\n \"\"\"\n root = get_root()\n if 'all' in checks:\n checks = os.listdir(root)\n\n for check_name in sorted(checks):\n pinned_reqs_file = os.path.join(root, check_name, 'requirements.in')\n resolved_reqs_file = os.path.join(root, check_name, 'requirements.txt')\n\n if os.path.isfile(pinned_reqs_file):\n if not quiet:\n echo_info('Check `{}`:'.format(check_name))\n\n if not quiet:\n echo_waiting(' Resolving dependencies...')\n\n pre_packages = read_packages(resolved_reqs_file)\n result = resolve_requirements(pinned_reqs_file, resolved_reqs_file, lazy=lazy)\n if result.code:\n abort(result.stdout + result.stderr)\n\n if not quiet:\n post_packages = read_packages(resolved_reqs_file)\n display_package_changes(pre_packages, post_packages, indent=' ')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pin(package, version, checks, marker, resolving, lazy, quiet):\n root = get_root()\n package_name = package.lower()\n version = version.lower()\n\n for check_name in sorted(os.listdir(root)):\n pinned_reqs_file = os.path.join(root, check_name, 'requirements.in')\n resolved_reqs_file = os.path.join(root, check_name, 'requirements.txt')\n\n if os.path.isfile(pinned_reqs_file):\n pinned_packages = {package.name: package for package in read_packages(pinned_reqs_file)}\n if package not in pinned_packages and check_name not in checks:\n continue\n\n if resolving:\n pre_packages = list(read_packages(resolved_reqs_file))\n else:\n pre_packages = list(itervalues(pinned_packages))\n\n if not quiet:\n echo_info('Check `{}`:'.format(check_name))\n\n if version == 'none':\n del pinned_packages[package_name]\n else:\n pinned_packages[package_name] = Package(package_name, version, marker)\n\n package_list = sorted(itervalues(pinned_packages))\n write_file_lines(pinned_reqs_file, ('{}\\n'.format(package) for package in package_list))\n\n if not quiet:\n echo_waiting(' Resolving dependencies...')\n\n if resolving:\n result = resolve_requirements(pinned_reqs_file, resolved_reqs_file, lazy=lazy)\n if result.code:\n abort(result.stdout + result.stderr)\n\n if not quiet:\n post_packages = read_packages(resolved_reqs_file if resolving else pinned_reqs_file)\n display_package_changes(pre_packages, post_packages, indent=' ')", "response": "Pin a dependency for all checks that require it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncombine all dependencies for the Agent s static environment.", "response": "def freeze():\n \"\"\"Combine all dependencies for the Agent's static environment.\"\"\"\n echo_waiting('Verifying collected packages...')\n catalog, errors = make_catalog()\n if errors:\n for error in errors:\n echo_failure(error)\n abort()\n\n static_file = get_agent_requirements()\n\n echo_info('Static file: {}'.format(static_file))\n\n pre_packages = list(read_packages(static_file))\n\n catalog.write_packages(static_file)\n\n post_packages = list(read_packages(static_file))\n display_package_changes(pre_packages, post_packages)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrestarts an Agent to detect environment changes.", "response": "def reload_env(check, env):\n \"\"\"Restart an Agent to detect environment changes.\"\"\"\n envs = get_configured_envs(check)\n if not envs:\n echo_failure('No active environments found for `{}`.'.format(check))\n echo_info('See what is available to start via `ddev env ls {}`.'.format(check))\n abort()\n\n if not env:\n if len(envs) > 1:\n echo_failure('Multiple active environments found for `{}`, please specify one.'.format(check))\n echo_info('See what is active via `ddev env ls`.')\n abort()\n\n env = envs[0]\n\n if env not in envs:\n echo_failure('`{}` is not an active environment.'.format(env))\n echo_info('See what is active via `ddev env ls`.')\n abort()\n\n environment = create_interface(check, env)\n\n result = environment.restart_agent()\n\n if result.code:\n abort(result.stdout + result.stderr, code=result.code)\n else:\n echo_success('Successfully reloaded environment `{}`!'.format(env))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _create_gitlab_prometheus_instance(self, instance, init_config):\n # Mapping from Prometheus metrics names to Datadog ones\n # For now it's a 1:1 mapping\n allowed_metrics = init_config.get('allowed_metrics')\n if allowed_metrics is None:\n raise CheckException(\"At least one metric must be whitelisted in `allowed_metrics`.\")\n\n gitlab_instance = deepcopy(instance)\n # gitlab uses 'prometheus_endpoint' and not 'prometheus_url', so we have to rename the key\n gitlab_instance['prometheus_url'] = instance.get('prometheus_endpoint')\n\n gitlab_instance.update(\n {\n 'namespace': 'gitlab',\n 'metrics': allowed_metrics,\n # Defaults that were set when gitlab was based on PrometheusCheck\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n\n return gitlab_instance", "response": "Create a gitlab instance based on the instance configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsubmit a metric as a gauge", "response": "def _submit_gauge(self, metric_name, val, metric, custom_tags=None, hostname=None):\n \"\"\"\n Submit a metric as a gauge, additional tags provided will be added to\n the ones from the label provided via the metrics object.\n\n `custom_tags` is an array of 'tag:value' that will be added to the\n metric when sending the gauge to Datadog.\n \"\"\"\n _tags = self._metric_tags(metric_name, val, metric, custom_tags, hostname)\n self.check.gauge('{}.{}'.format(self.NAMESPACE, metric_name), val, _tags, hostname=hostname)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate the tag sets for a MongoReplicaSetClient.", "response": "def _validate_tag_sets(tag_sets):\n \"\"\"Validate tag sets for a MongoReplicaSetClient.\n \"\"\"\n if tag_sets is None:\n return tag_sets\n\n if not isinstance(tag_sets, list):\n raise TypeError((\n \"Tag sets %r invalid, must be a list\") % (tag_sets,))\n if len(tag_sets) == 0:\n raise ValueError((\n \"Tag sets %r invalid, must be None or contain at least one set of\"\n \" tags\") % (tag_sets,))\n\n for tags in tag_sets:\n if not isinstance(tags, Mapping):\n raise TypeError(\n \"Tag set %r invalid, must be an instance of dict, \"\n \"bson.son.SON or other type that inherits from \"\n \"collection.Mapping\" % (tags,))\n\n return tag_sets"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreading preference as a document.", "response": "def document(self):\n \"\"\"Read preference as a document.\n \"\"\"\n doc = {'mode': self.__mongos_mode}\n if self.__tag_sets not in (None, [{}]):\n doc['tags'] = self.__tag_sets\n if self.__max_staleness != -1:\n doc['maxStalenessSeconds'] = self.__max_staleness\n return doc"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates default configuration files.", "response": "def config(check):\n \"\"\"Validate default configuration files.\"\"\"\n if check:\n checks = [check]\n else:\n checks = sorted(get_valid_checks())\n\n files_failed = {}\n files_warned = {}\n num_files = 0\n\n echo_waiting('Validating default configuration files...')\n for check in checks:\n check_display_queue = []\n\n config_files = get_config_files(check)\n for config_file in config_files:\n num_files += 1\n file_display_queue = []\n file_name = basepath(config_file)\n\n try:\n config_data = yaml.safe_load(read_file(config_file))\n except Exception as e:\n files_failed[config_file] = True\n\n # We must convert to text here to free Exception object before it goes out of scope\n error = str(e)\n\n check_display_queue.append(lambda: echo_info('{}:'.format(file_name), indent=True))\n check_display_queue.append(lambda: echo_failure('Invalid YAML -', indent=FILE_INDENT))\n check_display_queue.append(lambda: echo_info(error, indent=FILE_INDENT * 2))\n continue\n\n # Verify there is an `instances` section\n if 'instances' not in config_data:\n files_failed[config_file] = True\n file_display_queue.append(lambda: echo_failure('Missing `instances` section', indent=FILE_INDENT))\n\n # Verify there is a default instance\n else:\n instances = config_data['instances']\n if check not in IGNORE_DEFAULT_INSTANCE and not isinstance(instances, list):\n files_failed[config_file] = True\n file_display_queue.append(lambda: echo_failure('No default instance', indent=FILE_INDENT))\n\n if file_display_queue:\n check_display_queue.append(lambda: echo_info('{}:'.format(file_name), indent=True))\n check_display_queue.extend(file_display_queue)\n\n if check_display_queue:\n echo_success('{}:'.format(check))\n for display in check_display_queue:\n display()\n\n files_failed = len(files_failed)\n files_warned = len(files_warned)\n files_passed = num_files - (files_failed + files_warned)\n\n if files_failed or files_warned:\n click.echo()\n\n if files_failed:\n echo_failure('Files with errors: {}'.format(files_failed))\n\n if files_warned:\n echo_warning('Files with warnings: {}'.format(files_warned))\n\n if files_passed:\n if files_failed or files_warned:\n echo_success('Files valid: {}'.format(files_passed))\n else:\n echo_success('All {} configuration files are valid!'.format(num_files))\n\n if files_failed:\n abort()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets stats from the queue manager", "response": "def queue_manager_stats(self, queue_manager, tags):\n \"\"\"\n Get stats from the queue manager\n \"\"\"\n for mname, pymqi_value in iteritems(metrics.queue_manager_metrics()):\n try:\n m = queue_manager.inquire(pymqi_value)\n\n mname = '{}.queue_manager.{}'.format(self.METRIC_PREFIX, mname)\n self.gauge(mname, m, tags=tags)\n self.service_check(self.QUEUE_MANAGER_SERVICE_CHECK, AgentCheck.OK, tags)\n except pymqi.Error as e:\n self.warning(\"Error getting queue manager stats: {}\".format(e))\n self.service_check(self.QUEUE_MANAGER_SERVICE_CHECK, AgentCheck.CRITICAL, tags)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngrabbing stats from queues 69", "response": "def queue_stats(self, queue, tags):\n \"\"\"\n Grab stats from queues\n \"\"\"\n for mname, pymqi_value in iteritems(metrics.queue_metrics()):\n try:\n mname = '{}.queue.{}'.format(self.METRIC_PREFIX, mname)\n m = queue.inquire(pymqi_value)\n self.gauge(mname, m, tags=tags)\n except pymqi.Error as e:\n self.warning(\"Error getting queue stats for {}: {}\".format(queue, e))\n\n for mname, func in iteritems(metrics.queue_metrics_functions()):\n try:\n mname = '{}.queue.{}'.format(self.METRIC_PREFIX, mname)\n m = func(queue)\n self.gauge(mname, m, tags=tags)\n except pymqi.Error as e:\n self.warning(\"Error getting queue stats for {}: {}\".format(queue, e))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_version_string(check_name):\n version = VERSION.search(read_version_file(check_name))\n if version:\n return version.group(1)", "response": "Get the version string for the given check."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading the manifest file into a dictionary", "response": "def load_manifest(check_name):\n \"\"\"\n Load the manifest file into a dictionary\n \"\"\"\n manifest_path = os.path.join(get_root(), check_name, 'manifest.json')\n if file_exists(manifest_path):\n return json.loads(read_file(manifest_path))\n return {}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse the requirements file contents into a dictionary mapping check - package - name - > pinned_version.", "response": "def parse_agent_req_file(contents):\n \"\"\"\n Returns a dictionary mapping {check-package-name --> pinned_version} from the\n given file contents. We can assume lines are in the form:\n\n datadog-active-directory==1.1.1; sys_platform == 'win32'\n\n \"\"\"\n catalog = OrderedDict()\n for line in contents.splitlines():\n toks = line.split('==', 1)\n if len(toks) != 2 or not toks[0] or not toks[1]:\n # if we get here, the requirements file is garbled but let's stay\n # resilient\n continue\n\n name, other = toks\n version = other.split(';')\n catalog[name] = version[0]\n\n return catalog"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncollecting the dictionary of the current state of the object.", "response": "def _collect_dict(self, metric_type, field_metric_map, query, db, tags):\n \"\"\"\n Query status and get a dictionary back.\n Extract each field out of the dictionary\n and stuff it in the corresponding metric.\n\n query: show status...\n field_metric_map: {\"Seconds_behind_master\": \"mysqlSecondsBehindMaster\"}\n \"\"\"\n try:\n with closing(db.cursor()) as cursor:\n cursor.execute(query)\n result = cursor.fetchone()\n if result is not None:\n for field, metric in list(iteritems(field_metric_map)):\n # Find the column name in the cursor description to identify the column index\n # http://www.python.org/dev/peps/pep-0249/\n # cursor.description is a tuple of (column_name, ..., ...)\n try:\n col_idx = [d[0].lower() for d in cursor.description].index(field.lower())\n self.log.debug(\"Collecting metric: %s\" % metric)\n if result[col_idx] is not None:\n self.log.debug(\"Collecting done, value %s\" % result[col_idx])\n if metric_type == GAUGE:\n self.gauge(metric, float(result[col_idx]), tags=tags)\n elif metric_type == RATE:\n self.rate(metric, float(result[col_idx]), tags=tags)\n else:\n self.gauge(metric, float(result[col_idx]), tags=tags)\n else:\n self.log.debug(\"Received value is None for index %d\" % col_idx)\n except ValueError:\n self.log.exception(\"Cannot find %s in the columns %s\" % (field, cursor.description))\n except Exception:\n self.warning(\"Error while running %s\\n%s\" % (query, traceback.format_exc()))\n self.log.exception(\"Error while running %s\" % query)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the pid_file variable of MySQL.", "response": "def _get_pid_file_variable(self, db):\n \"\"\"\n Get the `pid_file` variable\n \"\"\"\n pid_file = None\n try:\n with closing(db.cursor()) as cursor:\n cursor.execute(\"SHOW VARIABLES LIKE 'pid_file'\")\n pid_file = cursor.fetchone()[1]\n except Exception:\n self.warning(\"Error while fetching pid_file variable of MySQL.\")\n\n return pid_file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_slave_status(self, db, above_560, nonblocking):\n try:\n with closing(db.cursor()) as cursor:\n if above_560 and nonblocking:\n # Query `performance_schema.threads` instead of `\n # information_schema.processlist` to avoid mutex impact on performance.\n cursor.execute(\"SELECT THREAD_ID, NAME FROM performance_schema.threads WHERE NAME LIKE '%worker'\")\n else:\n cursor.execute(\"SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE COMMAND LIKE '%Binlog dump%'\")\n slave_results = cursor.fetchall()\n slaves = 0\n for _ in slave_results:\n slaves += 1\n\n return {'Slaves_connected': slaves}\n\n except (pymysql.err.InternalError, pymysql.err.OperationalError) as e:\n self.warning(\"Privileges error accessing the process tables (must grant PROCESS): %s\" % str(e))\n return {}", "response": "Retrieve the slaves status using the process tables."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nparses the general counters from the init_config and return a dictionary for each job counter", "response": "def _parse_general_counters(self, init_config):\n \"\"\"\n Return a dictionary for each job counter\n {\n counter_group_name: [\n counter_name\n ]\n }\n }\n \"\"\"\n job_counter = {}\n\n if init_config.get('general_counters'):\n\n # Parse the custom metrics\n for counter_group in init_config['general_counters']:\n counter_group_name = counter_group.get('counter_group_name')\n counters = counter_group.get('counters')\n\n if not counter_group_name:\n raise Exception('\"general_counters\" must contain a valid \"counter_group_name\"')\n\n if not counters:\n raise Exception('\"general_counters\" must contain a list of \"counters\"')\n\n # Add the counter_group to the job_counters if it doesn't already exist\n if counter_group_name not in job_counter:\n job_counter[counter_group_name] = []\n\n for counter in counters:\n counter_name = counter.get('counter_name')\n\n if not counter_name:\n raise Exception('At least one \"counter_name\" should be specified in the list of \"counters\"')\n\n job_counter[counter_group_name].append(counter_name)\n\n return job_counter"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the job specific counters.", "response": "def _parse_job_specific_counters(self, init_config):\n \"\"\"\n Return a dictionary for each job counter\n {\n job_name: {\n counter_group_name: [\n counter_name\n ]\n }\n }\n }\n \"\"\"\n job_counter = {}\n\n if init_config.get('job_specific_counters'):\n\n # Parse the custom metrics\n for job in init_config['job_specific_counters']:\n job_name = job.get('job_name')\n metrics = job.get('metrics')\n\n if not job_name:\n raise Exception('Counter metrics must have a \"job_name\"')\n\n if not metrics:\n raise Exception(\"Jobs specified in counter metrics must contain at least one metric\")\n\n # Add the job to the custom metrics if it doesn't already exist\n if job_name not in job_counter:\n job_counter[job_name] = {}\n\n for metric in metrics:\n counter_group_name = metric.get('counter_group_name')\n counters = metric.get('counters')\n\n if not counter_group_name:\n raise Exception('Each counter metric must contain a valid \"counter_group_name\"')\n\n if not counters:\n raise Exception('Each counter metric must contain a list of \"counters\"')\n\n # Add the counter group name if it doesn't exist for the current job\n if counter_group_name not in job_counter[job_name]:\n job_counter[job_name][counter_group_name] = []\n\n for counter in counters:\n counter_name = counter.get('counter_name')\n\n if not counter_name:\n raise Exception('At least one \"counter_name\" should be specified in the list of \"counters\"')\n\n job_counter[job_name][counter_group_name].append(counter_name)\n\n return job_counter"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_running_app_ids(self, rm_address, auth, ssl_verify):\n metrics_json = self._rest_request_to_json(\n rm_address,\n auth,\n ssl_verify,\n self.YARN_APPS_PATH,\n self.YARN_SERVICE_CHECK,\n states=self.YARN_APPLICATION_STATES,\n applicationTypes=self.YARN_APPLICATION_TYPES,\n )\n\n running_apps = {}\n\n if metrics_json.get('apps'):\n if metrics_json['apps'].get('app') is not None:\n\n for app_json in metrics_json['apps']['app']:\n app_id = app_json.get('id')\n tracking_url = app_json.get('trackingUrl')\n app_name = app_json.get('name')\n\n if app_id and tracking_url and app_name:\n running_apps[app_id] = (app_name, tracking_url)\n\n return running_apps", "response": "Get the running MapReduce applications."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _mapreduce_job_metrics(self, running_apps, auth, ssl_verify, addl_tags):\n running_jobs = {}\n\n for app_name, tracking_url in itervalues(running_apps):\n\n metrics_json = self._rest_request_to_json(\n tracking_url, auth, ssl_verify, self.MAPREDUCE_JOBS_PATH, self.MAPREDUCE_SERVICE_CHECK\n )\n\n if metrics_json.get('jobs'):\n if metrics_json['jobs'].get('job'):\n\n for job_json in metrics_json['jobs']['job']:\n job_id = job_json.get('id')\n job_name = job_json.get('name')\n user_name = job_json.get('user')\n\n if job_id and job_name and user_name:\n\n # Build the structure to hold the information for each job ID\n running_jobs[str(job_id)] = {\n 'job_name': str(job_name),\n 'app_name': str(app_name),\n 'user_name': str(user_name),\n 'tracking_url': self._join_url_dir(tracking_url, self.MAPREDUCE_JOBS_PATH, job_id),\n }\n\n tags = [\n 'app_name:' + str(app_name),\n 'user_name:' + str(user_name),\n 'job_name:' + str(job_name),\n ]\n\n tags.extend(addl_tags)\n\n self._set_metrics_from_json(job_json, self.MAPREDUCE_JOB_METRICS, tags)\n\n return running_jobs", "response": "Get metrics for each MapReduce job."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _mapreduce_job_counters_metrics(self, running_jobs, auth, ssl_verify, addl_tags):\n for job_metrics in itervalues(running_jobs):\n job_name = job_metrics['job_name']\n\n # Check if the job_name exist in the custom metrics\n if self.general_counters or (job_name in self.job_specific_counters):\n job_specific_metrics = self.job_specific_counters.get(job_name)\n\n metrics_json = self._rest_request_to_json(\n job_metrics['tracking_url'],\n auth,\n ssl_verify,\n 'counters',\n self.MAPREDUCE_SERVICE_CHECK,\n tags=addl_tags,\n )\n\n if metrics_json.get('jobCounters'):\n if metrics_json['jobCounters'].get('counterGroup'):\n\n # Cycle through all the counter groups for this job\n for counter_group in metrics_json['jobCounters']['counterGroup']:\n group_name = counter_group.get('counterGroupName')\n\n if group_name:\n counter_metrics = set([])\n\n # Add any counters in the job specific metrics\n if job_specific_metrics and group_name in job_specific_metrics:\n counter_metrics = counter_metrics.union(job_specific_metrics[group_name])\n\n # Add any counters in the general metrics\n if group_name in self.general_counters:\n counter_metrics = counter_metrics.union(self.general_counters[group_name])\n\n if counter_metrics:\n # Cycle through all the counters in this counter group\n if counter_group.get('counter'):\n for counter in counter_group['counter']:\n counter_name = counter.get('name')\n\n # Check if the counter name is in the custom metrics for this group name\n if counter_name and counter_name in counter_metrics:\n tags = [\n 'app_name:' + job_metrics.get('app_name'),\n 'user_name:' + job_metrics.get('user_name'),\n 'job_name:' + job_name,\n 'counter_name:' + str(counter_name).lower(),\n ]\n\n tags.extend(addl_tags)\n\n self._set_metrics_from_json(\n counter, self.MAPREDUCE_JOB_COUNTER_METRICS, tags\n )", "response": "Get the custom metrics specified for each counter group and add them to the running jobs list"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting metrics for each MapReduce task", "response": "def _mapreduce_task_metrics(self, running_jobs, auth, ssl_verify, addl_tags):\n \"\"\"\n Get metrics for each MapReduce task\n Return a dictionary of {task_id: 'tracking_url'} for each MapReduce task\n \"\"\"\n for job_stats in itervalues(running_jobs):\n\n metrics_json = self._rest_request_to_json(\n job_stats['tracking_url'], auth, ssl_verify, 'tasks', self.MAPREDUCE_SERVICE_CHECK, tags=addl_tags\n )\n\n if metrics_json.get('tasks'):\n if metrics_json['tasks'].get('task'):\n\n for task in metrics_json['tasks']['task']:\n task_type = task.get('type')\n\n if task_type:\n tags = [\n 'app_name:' + job_stats['app_name'],\n 'user_name:' + job_stats['user_name'],\n 'job_name:' + job_stats['job_name'],\n 'task_type:' + str(task_type).lower(),\n ]\n\n tags.extend(addl_tags)\n\n if task_type == 'MAP':\n self._set_metrics_from_json(task, self.MAPREDUCE_MAP_TASK_METRICS, tags)\n\n elif task_type == 'REDUCE':\n self._set_metrics_from_json(task, self.MAPREDUCE_REDUCE_TASK_METRICS, tags)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse the JSON response and set the metrics", "response": "def _set_metrics_from_json(self, metrics_json, metrics, tags):\n \"\"\"\n Parse the JSON response and set the metrics\n \"\"\"\n for status, (metric_name, metric_type) in iteritems(metrics):\n metric_status = metrics_json.get(status)\n\n if metric_status is not None:\n self._set_metric(metric_name, metric_type, metric_status, tags=tags)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets a metric in the cache.", "response": "def _set_metric(self, metric_name, metric_type, value, tags=None, device_name=None):\n \"\"\"\n Set a metric\n \"\"\"\n if metric_type == self.HISTOGRAM:\n self.histogram(metric_name, value, tags=tags, device_name=device_name)\n elif metric_type == self.INCREMENT:\n self.increment(metric_name, value, tags=tags, device_name=device_name)\n else:\n self.log.error('Metric type \"{}\" unknown'.format(metric_type))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _rest_request_to_json(self, address, auth, ssl_verify, object_path, service_name, tags=None, *args, **kwargs):\n response_json = None\n tags = [] if tags is None else tags\n\n service_check_tags = ['url:{}'.format(self._get_url_base(address))] + tags\n\n url = address\n\n if object_path:\n url = self._join_url_dir(url, object_path)\n\n # Add args to the url\n if args:\n for directory in args:\n url = self._join_url_dir(url, directory)\n\n self.log.debug('Attempting to connect to \"{}\"'.format(url))\n\n # Add kwargs as arguments\n if kwargs:\n query = '&'.join(['{}={}'.format(key, value) for key, value in iteritems(kwargs)])\n url = urljoin(url, '?' + query)\n\n try:\n response = requests.get(url, auth=auth, verify=ssl_verify, timeout=self.default_integration_http_timeout)\n response.raise_for_status()\n response_json = response.json()\n\n except Timeout as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"Request timeout: {}, {}\".format(url, e),\n )\n raise\n\n except (HTTPError, InvalidURL, ConnectionError) as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"Request failed: {}, {}\".format(url, e),\n )\n raise\n\n except JSONDecodeError as e:\n self.service_check(\n service_name,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"JSON Parse failed: {}, {}\".format(url, e),\n )\n raise\n\n except ValueError as e:\n self.service_check(service_name, AgentCheck.CRITICAL, tags=service_check_tags, message=str(e))\n raise\n\n return response_json", "response": "Perform a REST request to the given URL and return the JSON response."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _psutil_config_to_stats(self, instance):\n process_metrics = instance.get('process_metrics', self.init_config.get('process_metrics', None))\n if not process_metrics:\n self.log.error('No metrics configured for AgentMetrics check!')\n return {}\n\n methods, metric_types = zip(\n *[(p['name'], p.get('type', GAUGE))\n for p in process_metrics if _is_affirmative(p.get('active'))]\n )\n\n names_to_metric_types = {}\n for i, m in enumerate(methods):\n names_to_metric_types[AgentMetrics._get_statistic_name_from_method(m)] = metric_types[i]\n\n stats = AgentMetrics._collect_internal_stats(methods)\n return stats, names_to_metric_types", "response": "Reads the init_config for the current process_metrics and returns the stats and the internal stats"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nregisters sample metrics from psutil.", "response": "def _register_psutil_metrics(self, stats, names_to_metric_types, tags=None):\n \"\"\"\n Saves sample metrics from psutil\n\n :param stats: a dictionary that looks like:\n {\n 'memory_info': OrderedDict([('rss', 24395776), ('vms', 144666624)]),\n 'io_counters': OrderedDict([('read_count', 4536),\n ('write_count', 100),\n ('read_bytes', 0),\n ('write_bytes', 61440)])\n ...\n }\n\n This creates a metric like `datadog.agent.collector.{key_1}.{key_2}` where key_1 is a top-level\n key in `stats`, and key_2 is a nested key.\n E.g. datadog.agent.collector.memory_info.rss\n \"\"\"\n if tags is None:\n tags = []\n base_metric = 'datadog.agent.collector.{0}.{1}'\n # TODO: May have to call self.normalize(metric_name) to get a compliant name\n for k, v in stats.iteritems():\n metric_type = names_to_metric_types[k]\n if isinstance(v, dict):\n for _k, _v in v.iteritems():\n full_metric_name = base_metric.format(k, _k)\n self._send_single_metric(full_metric_name, _v, metric_type, tags)\n else:\n full_metric_name = 'datadog.agent.collector.{0}'.format(k)\n self._send_single_metric(full_metric_name, v, metric_type, tags)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_container_ip(container_id_or_name):\n command = [\n 'docker',\n 'inspect',\n '-f',\n '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}',\n container_id_or_name,\n ]\n\n return run_command(command, capture='out', check=True).stdout.strip()", "response": "Get a Docker container s IP address from its id or name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef docker_run(\n compose_file=None,\n build=False,\n service_name=None,\n up=None,\n down=None,\n sleep=None,\n endpoints=None,\n log_patterns=None,\n conditions=None,\n env_vars=None,\n wrapper=None,\n):\n \"\"\"This utility provides a convenient way to safely set up and tear down Docker environments.\n\n :param compose_file: A path to a Docker compose file. A custom tear\n down is not required when using this.\n :type compose_file: ``str``\n :param build: Whether or not to build images for when ``compose_file`` is provided.\n :type build: ``bool``\n :param service_name: Optional name for when ``compose_file`` is provided.\n :type service_name: ``str``\n :param up: A custom setup callable.\n :type up: ``callable``\n :param down: A custom tear down callable. This is required when using a custom setup.\n :type down: ``callable``\n :param sleep: Number of seconds to wait before yielding.\n :type sleep: ``float``\n :param endpoints: Endpoints to verify access for before yielding. Shorthand for adding\n ``conditions.CheckEndpoints(endpoints)`` to the ``conditions`` argument.\n :type endpoints: ``list`` of ``str``, or a single ``str``\n :param log_patterns: Patterns to find in Docker logs before yielding. This is only available\n when ``compose_file`` is provided. Shorthand for adding\n ``conditions.CheckDockerLogs(compose_file, log_patterns)`` to the ``conditions`` argument.\n :type log_patterns: ``list`` of (``str`` or ``re.Pattern``)\n :param conditions: A list of callable objects that will be executed before yielding to check for errors.\n :type conditions: ``callable``\n :param env_vars: A dictionary to update ``os.environ`` with during execution.\n :type env_vars: ``dict``\n :param wrapper: A context manager to use during execution.\n \"\"\"\n if compose_file and up:\n raise TypeError('You must select either a compose file or a custom setup callable, not both.')\n\n if compose_file is not None:\n if not isinstance(compose_file, string_types):\n raise TypeError('The path to the compose file is not a string: {}'.format(repr(compose_file)))\n\n set_up = ComposeFileUp(compose_file, build=build, service_name=service_name)\n if down is not None:\n tear_down = down\n else:\n tear_down = ComposeFileDown(compose_file)\n else:\n set_up = up\n tear_down = down\n\n docker_conditions = []\n\n if log_patterns is not None:\n if compose_file is None:\n raise ValueError(\n 'The `log_patterns` convenience is unavailable when using '\n 'a custom setup. Please use a custom condition instead.'\n )\n docker_conditions.append(CheckDockerLogs(compose_file, log_patterns))\n\n if conditions is not None:\n docker_conditions.extend(conditions)\n\n with environment_run(\n up=set_up,\n down=tear_down,\n sleep=sleep,\n endpoints=endpoints,\n conditions=docker_conditions,\n env_vars=env_vars,\n wrapper=wrapper,\n ) as result:\n yield result", "response": "This utility is used to set up and tear down Docker environments."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the SON document representation of this DBRef.", "response": "def as_doc(self):\n \"\"\"Get the SON document representation of this DBRef.\n\n Generally not needed by application developers\n \"\"\"\n doc = SON([(\"$ref\", self.collection),\n (\"$id\", self.id)])\n if self.database is not None:\n doc[\"$db\"] = self.database\n doc.update(self.__kwargs)\n return doc"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef clear(self):\n with self._lock:\n self._config = {\n CacheConfig.Morlist: {'last': defaultdict(float), 'intl': {}},\n CacheConfig.Metadata: {'last': defaultdict(float), 'intl': {}},\n }", "response": "Reset the config object to its initial state"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the last value for a given key in the configuration.", "response": "def get_last(self, type_, key):\n \"\"\"\n Notice: this will return the defaultdict default value also for keys\n that are not in the configuration, this is a tradeoff to keep the code simple.\n \"\"\"\n self._check_type(type_)\n with self._lock:\n return self._config[type_]['last'][key]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a list of possible paths containing certificates", "response": "def _get_ca_certs_paths():\n \"\"\"\n Get a list of possible paths containing certificates\n\n Check is installed via pip to:\n * Windows: embedded/lib/site-packages/datadog_checks/http_check\n * Linux: embedded/lib/python2.7/site-packages/datadog_checks/http_check\n\n Certificate is installed to:\n * embedded/ssl/certs/cacert.pem\n\n walk up to `embedded`, and back down to ssl/certs to find the certificate file\n \"\"\"\n ca_certs = []\n\n embedded_root = os.path.dirname(os.path.abspath(__file__))\n for _ in range(10):\n if os.path.basename(embedded_root) == 'embedded':\n ca_certs.append(os.path.join(embedded_root, 'ssl', 'certs', 'cacert.pem'))\n break\n embedded_root = os.path.dirname(embedded_root)\n else:\n raise OSError(\n 'Unable to locate `embedded` directory. Please specify ca_certs in your http yaml configuration file.'\n )\n\n try:\n import tornado\n except ImportError:\n # if `tornado` is not present, simply ignore its certificates\n pass\n else:\n ca_certs.append(os.path.join(os.path.dirname(tornado.__file__), 'ca-certificates.crt'))\n\n ca_certs.append('/etc/ssl/certs/ca-certificates.crt')\n\n return ca_certs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates logo files. Specifying no check will validate all logos.", "response": "def logos(check):\n\n \"\"\"Validate logo files. Specifying no check will validate all logos\"\"\"\n\n valid_checks = get_valid_integrations()\n if check:\n if check in valid_checks:\n checks = [check]\n else:\n echo_info('{} is not an integration.'.format(check))\n return\n else:\n checks = sorted(valid_checks)\n\n blacklisted_integrations_msg = ''\n count_successful = 0\n error_checks = set()\n\n for check in checks:\n errors = dict()\n display_name = load_manifest(check).get('display_name', check)\n if check in NOT_TILES:\n blacklisted_integrations_msg += '{} does not currently have an integration tile.\\n'.format(display_name)\n continue\n\n path_to_check_logos = os.path.join(get_root(), check, 'logos')\n\n for logo, required_size in REQUIRED_IMAGES.items():\n logo_file_name = os.path.join(path_to_check_logos, logo)\n if not os.path.isfile(logo_file_name):\n errors[logo] = ' {} is missing for {}'.format(logo, display_name)\n else:\n size = get_resolution(logo_file_name)\n if size != required_size:\n errors[logo] = ' {} has improper resolution: {}. Should be {}'.format(logo, size, required_size)\n\n if errors:\n echo_waiting('{}:'.format(display_name))\n echo_failure('\\n'.join(errors.values()))\n error_checks.add(check)\n else:\n count_successful += 1\n\n blacklisted_integrations_msg = blacklisted_integrations_msg.rstrip()\n\n if error_checks:\n echo_success(blacklisted_integrations_msg)\n abort()\n elif len(checks) == 1:\n if blacklisted_integrations_msg:\n echo_success(blacklisted_integrations_msg)\n else:\n echo_success('Congrats, all {} logos are valid!'.format(display_name))\n else:\n echo_success(\n 'Congrats, all {} checks\\' logo files are valid! {} checks were blacklisted and skipped.'.format(\n count_successful, len(NOT_TILES)\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_varnishstat(self, output, varnishstat_format, tags=None):\n tags = tags or []\n # FIXME: this check is processing an unbounded amount of data\n # we should explicitly list the metrics we want to get from the check\n if varnishstat_format == \"xml\":\n p = xml.parsers.expat.ParserCreate()\n p.StartElementHandler = self._start_element\n p.EndElementHandler = lambda name: self._end_element(name, tags)\n p.CharacterDataHandler = self._char_data\n self._reset()\n p.Parse(output, True)\n elif varnishstat_format == \"json\":\n json_output = json.loads(output)\n for name, metric in iteritems(json_output):\n if not isinstance(metric, dict): # skip 'timestamp' field\n continue\n\n if name.startswith(\"MAIN.\"):\n name = name.split('.', 1)[1]\n value = metric.get(\"value\", 0)\n\n if metric[\"flag\"] in (\"a\", \"c\"):\n self.rate(self.normalize(name, prefix=\"varnish\"), long(value), tags=tags)\n elif metric[\"flag\"] in (\"g\", \"i\"):\n self.gauge(self.normalize(name, prefix=\"varnish\"), long(value), tags=tags)\n if 'n_purges' in self.normalize(name, prefix=\"varnish\"):\n self.rate('varnish.n_purgesps', long(value), tags=tags)\n elif varnishstat_format == \"text\":\n for line in output.split(\"\\n\"):\n self.log.debug(\"Parsing varnish results: %s\" % line)\n fields = line.split()\n if len(fields) < 3:\n break\n name, gauge_val, rate_val = fields[0], fields[1], fields[2]\n metric_name = self.normalize(name, prefix=\"varnish\")\n\n # Now figure out which value to pick\n if rate_val.lower() in (\"nan\", \".\"):\n # col 2 matters\n self.log.debug(\"Varnish (gauge) %s %d\" % (metric_name, int(gauge_val)))\n self.gauge(metric_name, int(gauge_val), tags=tags)\n if 'n_purges' in metric_name:\n self.rate('varnish.n_purgesps', float(gauge_val), tags=tags)\n else:\n # col 3 has a rate (since restart)\n self.log.debug(\"Varnish (rate) %s %d\" % (metric_name, int(gauge_val)))\n self.rate(metric_name, float(gauge_val), tags=tags)", "response": "Parse the output of the varnishstat command and add it to the internal object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing out service checks from varnishadm output.", "response": "def _parse_varnishadm(self, output, tags):\n \"\"\" Parse out service checks from varnishadm.\n\n Example output:\n\n Backend b0 is Sick\n Current states good: 2 threshold: 3 window: 5\n Average responsetime of good probes: 0.000000\n Oldest Newest\n ================================================================\n -------------------------------------------------------------444 Good IPv4\n -------------------------------------------------------------XXX Good Xmit\n -------------------------------------------------------------RRR Good Recv\n ----------------------------------------------------------HHH--- Happy\n Backend b1 is Sick\n Current states good: 2 threshold: 3 window: 5\n Average responsetime of good probes: 0.000000\n Oldest Newest\n ================================================================\n ----------------------------------------------------------HHH--- Happy\n\n Example output (new output format):\n\n Backend name Admin Probe\n boot.default probe Healthy (no probe)\n boot.backend2 probe Healthy 4/4\n Current states good: 4 threshold: 3 window: 4\n Average response time of good probes: 0.002504\n Oldest ================================================== Newest\n --------------------------------------------------------------44 Good IPv4\n --------------------------------------------------------------XX Good Xmit\n --------------------------------------------------------------RR Good Recv\n ------------------------------------------------------------HHHH Happy\n\n \"\"\"\n # Process status by backend.\n backends_by_status = defaultdict(list)\n for line in output.split(\"\\n\"):\n backend, status, message = None, None, None\n # split string and remove all empty fields\n tokens = filter(None, line.strip().split(' '))\n tokens = [t for t in tokens]\n if len(tokens):\n if tokens == ['Backend', 'name', 'Admin', 'Probe']:\n # skip the column headers that exist in new output format\n continue\n # parse new output format\n # the backend name will include the vcl name\n # so split on first . to remove prefix\n elif len(tokens) >= 4 and tokens[1] in ['healthy', 'sick']:\n # If the backend health was overriden, lets grab the\n # overriden value instead of the probed health\n backend = tokens[0].split('.', 1)[-1]\n status = tokens[1].lower()\n elif len(tokens) >= 4 and tokens[1] == 'probe':\n backend = tokens[0].split('.', 1)[-1]\n status = tokens[2].lower()\n # Parse older Varnish backend output\n elif tokens[0] == 'Backend':\n backend = tokens[1]\n status = tokens[-1].lower()\n\n if tokens[0] == 'Current' and backend is not None:\n try:\n message = ' '.join(tokens[2:]).strip()\n except Exception:\n # If we can't parse a message still send a status.\n self.log.exception('Error when parsing message from varnishadm')\n message = ''\n\n if backend is not None:\n backends_by_status[status].append((backend, message))\n\n for status, backends in iteritems(backends_by_status):\n check_status = BackendStatus.to_check_status(status)\n for backend, message in backends:\n service_checks_tags = ['backend:%s' % backend] + tags\n self.service_check(self.SERVICE_CHECK_NAME, check_status, tags=service_checks_tags, message=message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_metric_family(self, response):\n if 'application/vnd.google.protobuf' in response.headers['Content-Type']:\n n = 0\n buf = response.content\n while n < len(buf):\n msg_len, new_pos = _DecodeVarint32(buf, n)\n n = new_pos\n msg_buf = buf[n : n + msg_len]\n n += msg_len\n\n message = metrics_pb2.MetricFamily()\n message.ParseFromString(msg_buf)\n message.name = self.remove_metric_prefix(message.name)\n\n # Lookup type overrides:\n if self.type_overrides and message.name in self.type_overrides:\n new_type = self.type_overrides[message.name]\n if new_type in self.METRIC_TYPES:\n message.type = self.METRIC_TYPES.index(new_type)\n else:\n self.log.debug(\"type override %s for %s is not a valid type name\" % (new_type, message.name))\n yield message\n\n elif 'text/plain' in response.headers['Content-Type']:\n input_gen = response.iter_lines(chunk_size=self.REQUESTS_CHUNK_SIZE, decode_unicode=True)\n if self._text_filter_blacklist:\n input_gen = self._text_filter_input(input_gen)\n\n messages = defaultdict(list) # map with the name of the element (before the labels)\n # and the list of occurrences with labels and values\n\n obj_map = {} # map of the types of each metrics\n obj_help = {} # help for the metrics\n for metric in text_fd_to_metric_families(input_gen):\n metric.name = self.remove_metric_prefix(metric.name)\n metric_name = \"%s_bucket\" % metric.name if metric.type == \"histogram\" else metric.name\n metric_type = self.type_overrides.get(metric_name, metric.type)\n if metric_type == \"untyped\" or metric_type not in self.METRIC_TYPES:\n continue\n\n for sample in metric.samples:\n if (sample[0].endswith(\"_sum\") or sample[0].endswith(\"_count\")) and metric_type in [\n \"histogram\",\n \"summary\",\n ]:\n messages[sample[0]].append({\"labels\": sample[1], 'value': sample[2]})\n else:\n messages[metric_name].append({\"labels\": sample[1], 'value': sample[2]})\n\n obj_map[metric.name] = metric_type\n obj_help[metric.name] = metric.documentation\n\n for _m in obj_map:\n if _m in messages or (obj_map[_m] == 'histogram' and ('{}_bucket'.format(_m) in messages)):\n yield self._extract_metric_from_map(_m, messages, obj_map, obj_help)\n else:\n raise UnknownFormatError('Unsupported content-type provided: {}'.format(response.headers['Content-Type']))", "response": "Parse the MetricFamily from a valid requests. Response object to provide a MetricFamily object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfilters out the text input line by line to avoid parsing and processing metrics.", "response": "def _text_filter_input(self, input_gen):\n \"\"\"\n Filters out the text input line by line to avoid parsing and processing\n metrics we know we don't want to process. This only works on `text/plain`\n payloads, and is an INTERNAL FEATURE implemented for the kubelet check\n :param input_get: line generator\n :output: generator of filtered lines\n \"\"\"\n for line in input_gen:\n for item in self._text_filter_blacklist:\n if item in line:\n break\n else:\n # No blacklist matches, passing the line through\n yield line"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the value of the metric_name matched by the labels in the _metric dictionary.", "response": "def get_metric_value_by_labels(messages, _metric, _m, metric_suffix):\n \"\"\"\n :param messages: dictionary as metric_name: {labels: {}, value: 10}\n :param _metric: dictionary as {labels: {le: '0.001', 'custom': 'value'}}\n :param _m: str as metric name\n :param metric_suffix: str must be in (count or sum)\n :return: value of the metric_name matched by the labels\n \"\"\"\n metric_name = '{}_{}'.format(_m, metric_suffix)\n expected_labels = set(\n [(k, v) for k, v in iteritems(_metric[\"labels\"]) if k not in PrometheusScraperMixin.UNWANTED_LABELS]\n )\n for elt in messages[metric_name]:\n current_labels = set(\n [(k, v) for k, v in iteritems(elt[\"labels\"]) if k not in PrometheusScraperMixin.UNWANTED_LABELS]\n )\n # As we have two hashable objects we can compare them without any side effects\n if current_labels == expected_labels:\n return float(elt[\"value\"])\n\n raise AttributeError(\"cannot find expected labels for metric %s with suffix %s\" % (metric_name, metric_suffix))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _extract_metric_from_map(self, _m, messages, obj_map, obj_help):\n _obj = metrics_pb2.MetricFamily()\n _obj.name = _m\n _obj.type = self.METRIC_TYPES.index(obj_map[_m])\n if _m in obj_help:\n _obj.help = obj_help[_m]\n # trick for histograms\n _newlbl = _m\n if obj_map[_m] == 'histogram':\n _newlbl = '{}_bucket'.format(_m)\n # Loop through the array of metrics ({labels, value}) built earlier\n for _metric in messages[_newlbl]:\n # in the case of quantiles and buckets, they need to be grouped by labels\n if obj_map[_m] in ['summary', 'histogram'] and len(_obj.metric) > 0:\n _label_exists = False\n _metric_minus = {k: v for k, v in list(iteritems(_metric['labels'])) if k not in ['quantile', 'le']}\n _metric_idx = 0\n for mls in _obj.metric:\n _tmp_lbl = {idx.name: idx.value for idx in mls.label}\n if _metric_minus == _tmp_lbl:\n _label_exists = True\n break\n _metric_idx = _metric_idx + 1\n if _label_exists:\n _g = _obj.metric[_metric_idx]\n else:\n _g = _obj.metric.add()\n else:\n _g = _obj.metric.add()\n if obj_map[_m] == 'counter':\n _g.counter.value = float(_metric['value'])\n elif obj_map[_m] == 'gauge':\n _g.gauge.value = float(_metric['value'])\n elif obj_map[_m] == 'summary':\n if '{}_count'.format(_m) in messages:\n _g.summary.sample_count = long(self.get_metric_value_by_labels(messages, _metric, _m, 'count'))\n if '{}_sum'.format(_m) in messages:\n _g.summary.sample_sum = self.get_metric_value_by_labels(messages, _metric, _m, 'sum')\n # TODO: see what can be done with the untyped metrics\n elif obj_map[_m] == 'histogram':\n if '{}_count'.format(_m) in messages:\n _g.histogram.sample_count = long(self.get_metric_value_by_labels(messages, _metric, _m, 'count'))\n if '{}_sum'.format(_m) in messages:\n _g.histogram.sample_sum = self.get_metric_value_by_labels(messages, _metric, _m, 'sum')\n # last_metric = len(_obj.metric) - 1\n # if last_metric >= 0:\n for lbl in _metric['labels']:\n # In the string format, the quantiles are in the labels\n if lbl == 'quantile':\n # _q = _obj.metric[last_metric].summary.quantile.add()\n _q = _g.summary.quantile.add()\n _q.quantile = float(_metric['labels'][lbl])\n _q.value = float(_metric['value'])\n # The upper_bounds are stored as \"le\" labels on string format\n elif obj_map[_m] == 'histogram' and lbl == 'le':\n # _q = _obj.metric[last_metric].histogram.bucket.add()\n _q = _g.histogram.bucket.add()\n _q.upper_bound = float(_metric['labels'][lbl])\n _q.cumulative_count = long(float(_metric['value']))\n else:\n # labels deduplication\n is_in_labels = False\n for _existing_lbl in _g.label:\n if lbl == _existing_lbl.name:\n is_in_labels = True\n if not is_in_labels:\n _l = _g.label.add()\n _l.name = lbl\n _l.value = _metric['labels'][lbl]\n return _obj", "response": "Extracts the metric from the given object map."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef scrape_metrics(self, endpoint):\n response = self.poll(endpoint)\n try:\n # no dry run if no label joins\n if not self.label_joins:\n self._dry_run = False\n elif not self._watched_labels:\n # build the _watched_labels set\n for val in itervalues(self.label_joins):\n self._watched_labels.add(val['label_to_match'])\n\n for metric in self.parse_metric_family(response):\n yield metric\n\n # Set dry run off\n self._dry_run = False\n # Garbage collect unused mapping and reset active labels\n for metric, mapping in list(iteritems(self._label_mapping)):\n for key in list(mapping):\n if key not in self._active_label_mapping[metric]:\n del self._label_mapping[metric][key]\n self._active_label_mapping = {}\n finally:\n response.close()", "response": "Scrape the metrics from prometheus and return the metrics as a generator."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprocesses the metrics from Prometheus and pushes them as gauges", "response": "def process(self, endpoint, **kwargs):\n \"\"\"\n Polls the data from prometheus and pushes them as gauges\n `endpoint` is the metrics endpoint to use to poll metrics from Prometheus\n\n Note that if the instance has a 'tags' attribute, it will be pushed\n automatically as additional custom tags and added to the metrics\n \"\"\"\n instance = kwargs.get('instance')\n if instance:\n kwargs['custom_tags'] = instance.get('tags', [])\n\n for metric in self.scrape_metrics(endpoint):\n self.process_metric(metric, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_metric(self, message, **kwargs):\n\n # If targeted metric, store labels\n self.store_labels(message)\n\n if message.name in self.ignore_metrics:\n return # Ignore the metric\n\n # Filter metric to see if we can enrich with joined labels\n self.join_labels(message)\n\n send_histograms_buckets = kwargs.get('send_histograms_buckets', True)\n send_monotonic_counter = kwargs.get('send_monotonic_counter', False)\n custom_tags = kwargs.get('custom_tags')\n ignore_unmapped = kwargs.get('ignore_unmapped', False)\n\n try:\n if not self._dry_run:\n try:\n self._submit(\n self.metrics_mapper[message.name],\n message,\n send_histograms_buckets,\n send_monotonic_counter,\n custom_tags,\n )\n except KeyError:\n if not ignore_unmapped:\n # call magic method (non-generic check)\n handler = getattr(self, message.name) # Lookup will throw AttributeError if not found\n try:\n handler(message, **kwargs)\n except Exception as err:\n self.log.warning(\"Error handling metric: {} - error: {}\".format(message.name, err))\n else:\n # build the wildcard list if first pass\n if self._metrics_wildcards is None:\n self._metrics_wildcards = [x for x in self.metrics_mapper.keys() if '*' in x]\n # try matching wildcard (generic check)\n for wildcard in self._metrics_wildcards:\n if fnmatchcase(message.name, wildcard):\n self._submit(\n message.name, message, send_histograms_buckets, send_monotonic_counter, custom_tags\n )\n\n except AttributeError as err:\n self.log.debug(\"Unable to handle metric: {} - error: {}\".format(message.name, err))", "response": "Process a metric message according to the above flow."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npoll the metrics from the prometheus metrics endpoint provided.", "response": "def poll(self, endpoint, pFormat=PrometheusFormat.PROTOBUF, headers=None):\n \"\"\"\n Polls the metrics from the prometheus metrics endpoint provided.\n Defaults to the protobuf format, but can use the formats specified by\n the PrometheusFormat class.\n Custom headers can be added to the default headers.\n\n Returns a valid requests.Response, raise requests.HTTPError if the status code of the requests.Response\n isn't valid - see response.raise_for_status()\n\n The caller needs to close the requests.Response\n\n :param endpoint: string url endpoint\n :param pFormat: the preferred format defined in PrometheusFormat\n :param headers: extra headers\n :return: requests.Response\n \"\"\"\n if headers is None:\n headers = {}\n if 'accept-encoding' not in headers:\n headers['accept-encoding'] = 'gzip'\n if pFormat == PrometheusFormat.PROTOBUF:\n headers['accept'] = (\n 'application/vnd.google.protobuf; ' 'proto=io.prometheus.client.MetricFamily; ' 'encoding=delimited'\n )\n headers.update(self.extra_headers)\n cert = None\n if isinstance(self.ssl_cert, string_types):\n cert = self.ssl_cert\n if isinstance(self.ssl_private_key, string_types):\n cert = (self.ssl_cert, self.ssl_private_key)\n verify = True\n if isinstance(self.ssl_ca_cert, string_types):\n verify = self.ssl_ca_cert\n elif self.ssl_ca_cert is False:\n disable_warnings(InsecureRequestWarning)\n verify = False\n try:\n response = requests.get(\n endpoint, headers=headers, stream=False, timeout=self.prometheus_timeout, cert=cert, verify=verify\n )\n except requests.exceptions.SSLError:\n self.log.error(\"Invalid SSL settings for requesting {} endpoint\".format(endpoint))\n raise\n except IOError:\n if self.health_service_check:\n self._submit_service_check(\n \"{}{}\".format(self.NAMESPACE, \".prometheus.health\"),\n AgentCheck.CRITICAL,\n tags=[\"endpoint:\" + endpoint],\n )\n raise\n try:\n response.raise_for_status()\n if self.health_service_check:\n self._submit_service_check(\n \"{}{}\".format(self.NAMESPACE, \".prometheus.health\"), AgentCheck.OK, tags=[\"endpoint:\" + endpoint]\n )\n return response\n except requests.HTTPError:\n response.close()\n if self.health_service_check:\n self._submit_service_check(\n \"{}{}\".format(self.NAMESPACE, \".prometheus.health\"),\n AgentCheck.CRITICAL,\n tags=[\"endpoint:\" + endpoint],\n )\n raise"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsubmit a message to Datadog.", "response": "def _submit(\n self,\n metric_name,\n message,\n send_histograms_buckets=True,\n send_monotonic_counter=False,\n custom_tags=None,\n hostname=None,\n ):\n \"\"\"\n For each metric in the message, report it as a gauge with all labels as tags\n except if a labels dict is passed, in which case keys are label names we'll extract\n and corresponding values are tag names we'll use (eg: {'node': 'node'}).\n\n Histograms generate a set of values instead of a unique metric.\n send_histograms_buckets is used to specify if yes or no you want to\n send the buckets as tagged values when dealing with histograms.\n\n `custom_tags` is an array of 'tag:value' that will be added to the\n metric when sending the gauge to Datadog.\n \"\"\"\n if message.type < len(self.METRIC_TYPES):\n for metric in message.metric:\n custom_hostname = self._get_hostname(hostname, metric)\n if message.type == 0:\n val = getattr(metric, self.METRIC_TYPES[message.type]).value\n if self._is_value_valid(val):\n if send_monotonic_counter:\n self._submit_monotonic_count(metric_name, val, metric, custom_tags, custom_hostname)\n else:\n self._submit_gauge(metric_name, val, metric, custom_tags, custom_hostname)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.\".format(metric_name))\n elif message.type == 4:\n self._submit_gauges_from_histogram(\n metric_name, metric, send_histograms_buckets, custom_tags, custom_hostname\n )\n elif message.type == 2:\n self._submit_gauges_from_summary(metric_name, metric, custom_tags, custom_hostname)\n else:\n val = getattr(metric, self.METRIC_TYPES[message.type]).value\n if self._is_value_valid(val):\n if message.name in self.rate_metrics:\n self._submit_rate(metric_name, val, metric, custom_tags, custom_hostname)\n else:\n self._submit_gauge(metric_name, val, metric, custom_tags, custom_hostname)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.\".format(metric_name))\n\n else:\n self.log.error(\"Metric type {} unsupported for metric {}.\".format(message.type, message.name))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_hostname(self, hostname, metric):\n if hostname is None and self.label_to_hostname is not None:\n for label in metric.label:\n if label.name == self.label_to_hostname:\n return label.value + self.label_to_hostname_suffix\n\n return hostname", "response": "Get the hostname from the label_to_hostname setting."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _submit_gauges_from_summary(self, name, metric, custom_tags=None, hostname=None):\n if custom_tags is None:\n custom_tags = []\n # summaries do not have a value attribute\n val = getattr(metric, self.METRIC_TYPES[2]).sample_count\n if self._is_value_valid(val):\n self._submit_gauge(\"{}.count\".format(name), val, metric, custom_tags)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.count.\".format(name))\n val = getattr(metric, self.METRIC_TYPES[2]).sample_sum\n if self._is_value_valid(val):\n self._submit_gauge(\"{}.sum\".format(name), val, metric, custom_tags)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.sum.\".format(name))\n for quantile in getattr(metric, self.METRIC_TYPES[2]).quantile:\n val = quantile.value\n limit = quantile.quantile\n if self._is_value_valid(val):\n self._submit_gauge(\n \"{}.quantile\".format(name),\n val,\n metric,\n custom_tags=custom_tags + [\"quantile:{}\".format(limit)],\n hostname=hostname,\n )\n else:\n self.log.debug(\"Metric value is not supported for metric {}.quantile.\".format(name))", "response": "Extracts metrics from a prometheus summary metric and sends them as gauges."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _submit_gauges_from_histogram(\n self, name, metric, send_histograms_buckets=True, custom_tags=None, hostname=None\n ):\n \"\"\"\n Extracts metrics from a prometheus histogram and sends them as gauges\n \"\"\"\n if custom_tags is None:\n custom_tags = []\n # histograms do not have a value attribute\n val = getattr(metric, self.METRIC_TYPES[4]).sample_count\n if self._is_value_valid(val):\n self._submit_gauge(\"{}.count\".format(name), val, metric, custom_tags)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.count.\".format(name))\n val = getattr(metric, self.METRIC_TYPES[4]).sample_sum\n if self._is_value_valid(val):\n self._submit_gauge(\"{}.sum\".format(name), val, metric, custom_tags)\n else:\n self.log.debug(\"Metric value is not supported for metric {}.sum.\".format(name))\n if send_histograms_buckets:\n for bucket in getattr(metric, self.METRIC_TYPES[4]).bucket:\n val = bucket.cumulative_count\n limit = bucket.upper_bound\n if self._is_value_valid(val):\n self._submit_gauge(\n \"{}.count\".format(name),\n val,\n metric,\n custom_tags=custom_tags + [\"upper_bound:{}\".format(limit)],\n hostname=hostname,\n )\n else:\n self.log.debug(\"Metric value is not supported for metric {}.count.\".format(name))", "response": "Extracts metrics from a prometheus histogram and sends them as gauges."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_auth_info(config=None):\n gh_config = (config or {}).get('github', {})\n user = gh_config.get('user') or os.getenv('DD_GITHUB_USER')\n token = gh_config.get('token') or os.getenv('DD_GITHUB_TOKEN')\n if user and token:\n return user, token", "response": "Get user and token from GitHub config"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_changelog_types(pr_payload):\n changelog_labels = []\n for name in get_pr_labels(pr_payload):\n if name.startswith(CHANGELOG_LABEL_PREFIX):\n # only add the name, e.g. for `changelog/Added` it's just `Added`\n changelog_labels.append(name.split(CHANGELOG_LABEL_PREFIX)[1])\n\n return changelog_labels", "response": "Fetch the labels from the PR and process them."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_pr(pr_num, config=None, repo=DEFAULT_REPO, raw=False):\n response = requests.get(PR_ENDPOINT.format(repo, pr_num), auth=get_auth_info(config))\n\n if raw:\n return response\n else:\n response.raise_for_status()\n return response.json()", "response": "Get the payload for the given PR number."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_pr_numbers(git_log_lines):\n prs = []\n for line in git_log_lines:\n pr_number = parse_pr_number(line)\n if pr_number:\n prs.append(pr_number)\n return prs", "response": "Parse PR numbers from commit messages."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef open(self):\n with self._lock:\n if self._thread_will_exit:\n # If the background thread has read self._stopped as True\n # there is a chance that it has not yet exited. The call to\n # join should not block indefinitely because there is no\n # other work done outside the while loop in self._run.\n try:\n self._thread.join()\n except ReferenceError:\n # Thread terminated.\n pass\n self._thread_will_exit = False\n self._stopped = False\n started = False\n try:\n started = self._thread and self._thread.is_alive()\n except ReferenceError:\n # Thread terminated.\n pass\n\n if not started:\n thread = threading.Thread(target=self._run, name=self._name)\n thread.daemon = True\n self._thread = weakref.proxy(thread)\n _register_executor(self)\n thread.start()", "response": "Start. Multiple calls have no effect."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconfiguring the tox environment.", "response": "def tox_configure(config):\n \"\"\"\n For more info, see: https://tox.readthedocs.io/en/latest/plugins.html\n For an example, see: https://github.com/tox-dev/tox-travis\n \"\"\"\n sections = config._cfg.sections\n\n # Cache these\n make_envconfig = None\n reader = None\n\n # Default to false so:\n # 1. we don't affect other projects using tox\n # 2. check migrations can happen gradually\n if str(sections.get('testenv', {}).get(STYLE_FLAG, 'false')).lower() == 'true':\n # Disable flake8 since we already include that\n config.envlist[:] = [env for env in config.envlist if not env.endswith('flake8')]\n\n make_envconfig = get_make_envconfig(make_envconfig)\n reader = get_reader(reader, config)\n\n add_style_checker(config, sections, make_envconfig, reader)\n add_style_formatter(config, sections, make_envconfig, reader)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _is_collect_cx_state_runnable(self, proc_location):\n if self._collect_cx_state is False:\n return False\n\n if proc_location != \"/proc\":\n self.warning(\"Cannot collect connection state: currently with a custom /proc path: %s\" % proc_location)\n return False\n\n return True", "response": "Determine if collect_connection_state is set and can effectively run."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _check_linux(self, instance):\n proc_location = self.agentConfig.get('procfs_path', '/proc').rstrip('/')\n custom_tags = instance.get('tags', [])\n\n if Platform.is_containerized() and proc_location != \"/proc\":\n proc_location = \"%s/1\" % proc_location\n\n if self._is_collect_cx_state_runnable(proc_location):\n try:\n self.log.debug(\"Using `ss` to collect connection state\")\n # Try using `ss` for increased performance over `netstat`\n for ip_version in ['4', '6']:\n for protocol in ['tcp', 'udp']:\n # Call `ss` for each IP version because there's no built-in way of distinguishing\n # between the IP versions in the output\n # Also calls `ss` for each protocol, because on some systems (e.g. Ubuntu 14.04), there is a\n # bug that print `tcp` even if it's `udp`\n output, _, _ = get_subprocess_output(\n [\"ss\", \"-n\", \"-{0}\".format(protocol[0]), \"-a\", \"-{0}\".format(ip_version)], self.log\n )\n lines = output.splitlines()\n\n # State Recv-Q Send-Q Local Address:Port Peer Address:Port\n # UNCONN 0 0 127.0.0.1:8125 *:*\n # ESTAB 0 0 127.0.0.1:37036 127.0.0.1:8125\n # UNCONN 0 0 fe80::a00:27ff:fe1c:3c4:123 :::*\n # TIME-WAIT 0 0 90.56.111.177:56867 46.105.75.4:143\n # LISTEN 0 0 ::ffff:127.0.0.1:33217 ::ffff:127.0.0.1:7199\n # ESTAB 0 0 ::ffff:127.0.0.1:58975 ::ffff:127.0.0.1:2181\n\n metrics = self._parse_linux_cx_state(\n lines[1:], self.tcp_states['ss'], 0, protocol=protocol, ip_version=ip_version\n )\n # Only send the metrics which match the loop iteration's ip version\n for stat, metric in iteritems(self.cx_state_gauge):\n if stat[0].endswith(ip_version) and stat[0].startswith(protocol):\n self.gauge(metric, metrics.get(metric), tags=custom_tags)\n\n except OSError:\n self.log.info(\"`ss` not found: using `netstat` as a fallback\")\n output, _, _ = get_subprocess_output([\"netstat\", \"-n\", \"-u\", \"-t\", \"-a\"], self.log)\n lines = output.splitlines()\n # Active Internet connections (w/o servers)\n # Proto Recv-Q Send-Q Local Address Foreign Address State\n # tcp 0 0 46.105.75.4:80 79.220.227.193:2032 SYN_RECV\n # tcp 0 0 46.105.75.4:143 90.56.111.177:56867 ESTABLISHED\n # tcp 0 0 46.105.75.4:50468 107.20.207.175:443 TIME_WAIT\n # tcp6 0 0 46.105.75.4:80 93.15.237.188:58038 FIN_WAIT2\n # tcp6 0 0 46.105.75.4:80 79.220.227.193:2029 ESTABLISHED\n # udp 0 0 0.0.0.0:123 0.0.0.0:*\n # udp6 0 0 :::41458 :::*\n\n metrics = self._parse_linux_cx_state(lines[2:], self.tcp_states['netstat'], 5)\n for metric, value in iteritems(metrics):\n self.gauge(metric, value, tags=custom_tags)\n except SubprocessOutputEmptyError:\n self.log.exception(\"Error collecting connection stats.\")\n\n proc_dev_path = \"{}/net/dev\".format(proc_location)\n with open(proc_dev_path, 'r') as proc:\n lines = proc.readlines()\n # Inter-| Receive | Transmit\n # face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed # noqa: E501\n # lo:45890956 112797 0 0 0 0 0 0 45890956 112797 0 0 0 0 0 0 # noqa: E501\n # eth0:631947052 1042233 0 19 0 184 0 1206 1208625538 1320529 0 0 0 0 0 0 # noqa: E501\n # eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # noqa: E501\n for l in lines[2:]:\n cols = l.split(':', 1)\n x = cols[1].split()\n # Filter inactive interfaces\n if self._parse_value(x[0]) or self._parse_value(x[8]):\n iface = cols[0].strip()\n metrics = {\n 'bytes_rcvd': self._parse_value(x[0]),\n 'bytes_sent': self._parse_value(x[8]),\n 'packets_in.count': self._parse_value(x[1]),\n 'packets_in.error': self._parse_value(x[2]) + self._parse_value(x[3]),\n 'packets_out.count': self._parse_value(x[9]),\n 'packets_out.error': self._parse_value(x[10]) + self._parse_value(x[11]),\n }\n self._submit_devicemetrics(iface, metrics, custom_tags)\n\n netstat_data = {}\n for f in ['netstat', 'snmp']:\n proc_data_path = \"{}/net/{}\".format(proc_location, f)\n try:\n with open(proc_data_path, 'r') as netstat:\n while True:\n n_header = netstat.readline()\n if not n_header:\n break # No more? Abort!\n n_data = netstat.readline()\n\n h_parts = n_header.strip().split(' ')\n h_values = n_data.strip().split(' ')\n ns_category = h_parts[0][:-1]\n netstat_data[ns_category] = {}\n # Turn the data into a dictionary\n for idx, hpart in enumerate(h_parts[1:]):\n netstat_data[ns_category][hpart] = h_values[idx + 1]\n except IOError:\n # On Openshift, /proc/net/snmp is only readable by root\n self.log.debug(\"Unable to read %s.\", proc_data_path)\n\n nstat_metrics_names = {\n 'Tcp': {\n 'RetransSegs': 'system.net.tcp.retrans_segs',\n 'InSegs': 'system.net.tcp.in_segs',\n 'OutSegs': 'system.net.tcp.out_segs',\n },\n 'TcpExt': {\n 'ListenOverflows': 'system.net.tcp.listen_overflows',\n 'ListenDrops': 'system.net.tcp.listen_drops',\n 'TCPBacklogDrop': 'system.net.tcp.backlog_drops',\n 'TCPRetransFail': 'system.net.tcp.failed_retransmits',\n },\n 'Udp': {\n 'InDatagrams': 'system.net.udp.in_datagrams',\n 'NoPorts': 'system.net.udp.no_ports',\n 'InErrors': 'system.net.udp.in_errors',\n 'OutDatagrams': 'system.net.udp.out_datagrams',\n 'RcvbufErrors': 'system.net.udp.rcv_buf_errors',\n 'SndbufErrors': 'system.net.udp.snd_buf_errors',\n 'InCsumErrors': 'system.net.udp.in_csum_errors',\n },\n }\n\n # Skip the first line, as it's junk\n for k in nstat_metrics_names:\n for met in nstat_metrics_names[k]:\n if met in netstat_data.get(k, {}):\n self._submit_netmetric(\n nstat_metrics_names[k][met], self._parse_value(netstat_data[k][met]), tags=custom_tags\n )\n\n # Get the conntrack -S information\n conntrack_path = instance.get('conntrack_path')\n if conntrack_path is not None:\n self._add_conntrack_stats_metrics(conntrack_path, custom_tags)\n\n # Get the rest of the metric by reading the files. Metrics available since kernel 3.6\n conntrack_files_location = os.path.join(proc_location, 'sys', 'net', 'netfilter')\n # By default, only max and count are reported. However if the blacklist is set,\n # the whitelist is loosing its default value\n blacklisted_files = instance.get('blacklist_conntrack_metrics')\n whitelisted_files = instance.get('whitelist_conntrack_metrics')\n if blacklisted_files is None and whitelisted_files is None:\n whitelisted_files = ['max', 'count']\n\n available_files = []\n\n # Get the metrics to read\n try:\n for metric_file in os.listdir(conntrack_files_location):\n if (\n os.path.isfile(os.path.join(conntrack_files_location, metric_file))\n and 'nf_conntrack_' in metric_file\n ):\n available_files.append(metric_file[len('nf_conntrack_') :])\n except Exception as e:\n self.log.debug(\"Unable to list the files in {}. {}\".format(conntrack_files_location, e))\n\n filtered_available_files = pattern_filter(\n available_files, whitelist=whitelisted_files, blacklist=blacklisted_files\n )\n\n for metric_name in filtered_available_files:\n metric_file_location = os.path.join(conntrack_files_location, 'nf_conntrack_{}'.format(metric_name))\n try:\n with open(metric_file_location, 'r') as conntrack_file:\n # Checking it's an integer\n try:\n value = int(conntrack_file.read().rstrip())\n self.gauge('system.net.conntrack.{}'.format(metric_name), value, tags=custom_tags)\n except ValueError:\n self.log.debug(\"{} is not an integer\".format(metric_name))\n except IOError as e:\n self.log.debug(\"Unable to read {}, skipping {}.\".format(metric_file_location, e))", "response": "Private function to check if Linux is running."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the output of conntrack - S and add the parsed metrics to the self. metrics", "response": "def _add_conntrack_stats_metrics(self, conntrack_path, tags):\n \"\"\"\n Parse the output of conntrack -S\n Add the parsed metrics\n \"\"\"\n try:\n output, _, _ = get_subprocess_output([\"sudo\", conntrack_path, \"-S\"], self.log)\n # conntrack -S sample:\n # cpu=0 found=27644 invalid=19060 ignore=485633411 insert=0 insert_failed=1 \\\n # drop=1 early_drop=0 error=0 search_restart=39936711\n # cpu=1 found=21960 invalid=17288 ignore=475938848 insert=0 insert_failed=1 \\\n # drop=1 early_drop=0 error=0 search_restart=36983181\n\n lines = output.splitlines()\n\n for line in lines:\n cols = line.split()\n cpu_num = cols[0].split('=')[-1]\n cpu_tag = ['cpu:{}'.format(cpu_num)]\n cols = cols[1:]\n\n for cell in cols:\n metric, value = cell.split('=')\n self.monotonic_count('system.net.conntrack.{}'.format(metric), int(value), tags=tags + cpu_tag)\n except SubprocessOutputEmptyError:\n self.log.debug(\"Couldn't use {} to get conntrack stats\".format(conntrack_path))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _parse_linux_cx_state(self, lines, tcp_states, state_col, protocol=None, ip_version=None):\n metrics = {}\n for _, val in iteritems(self.cx_state_gauge):\n metrics[val] = 0\n for l in lines:\n cols = l.split()\n if cols[0].startswith('tcp') or protocol == 'tcp':\n proto = \"tcp{0}\".format(ip_version) if ip_version else (\"tcp4\", \"tcp6\")[cols[0] == \"tcp6\"]\n if cols[state_col] in tcp_states:\n metric = self.cx_state_gauge[proto, tcp_states[cols[state_col]]]\n metrics[metric] += 1\n elif cols[0].startswith('udp') or protocol == 'udp':\n proto = \"udp{0}\".format(ip_version) if ip_version else (\"udp4\", \"udp6\")[cols[0] == \"udp6\"]\n metric = self.cx_state_gauge[proto, 'connections']\n metrics[metric] += 1\n\n return metrics", "response": "Parse the output of the Linux CX state command."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the Solaris network stat output.", "response": "def _parse_solaris_netstat(self, netstat_output):\n \"\"\"\n Return a mapping of network metrics by interface. For example:\n { interface:\n {'bytes_sent': 0,\n 'bytes_rcvd': 0,\n 'bytes_rcvd': 0,\n ...\n }\n }\n \"\"\"\n # Here's an example of the netstat output:\n #\n # link:0:net0:brdcstrcv 527336\n # link:0:net0:brdcstxmt 1595\n # link:0:net0:class net\n # link:0:net0:collisions 0\n # link:0:net0:crtime 16359935.2637943\n # link:0:net0:ierrors 0\n # link:0:net0:ifspeed 10000000000\n # link:0:net0:ipackets 682834\n # link:0:net0:ipackets64 682834\n # link:0:net0:link_duplex 0\n # link:0:net0:link_state 1\n # link:0:net0:multircv 0\n # link:0:net0:multixmt 1595\n # link:0:net0:norcvbuf 0\n # link:0:net0:noxmtbuf 0\n # link:0:net0:obytes 12820668\n # link:0:net0:obytes64 12820668\n # link:0:net0:oerrors 0\n # link:0:net0:opackets 105445\n # link:0:net0:opackets64 105445\n # link:0:net0:rbytes 113983614\n # link:0:net0:rbytes64 113983614\n # link:0:net0:snaptime 16834735.1607669\n # link:0:net0:unknowns 0\n # link:0:net0:zonename 53aa9b7e-48ba-4152-a52b-a6368c3d9e7c\n # link:0:net1:brdcstrcv 4947620\n # link:0:net1:brdcstxmt 1594\n # link:0:net1:class net\n # link:0:net1:collisions 0\n # link:0:net1:crtime 16359935.2839167\n # link:0:net1:ierrors 0\n # link:0:net1:ifspeed 10000000000\n # link:0:net1:ipackets 4947620\n # link:0:net1:ipackets64 4947620\n # link:0:net1:link_duplex 0\n # link:0:net1:link_state 1\n # link:0:net1:multircv 0\n # link:0:net1:multixmt 1594\n # link:0:net1:norcvbuf 0\n # link:0:net1:noxmtbuf 0\n # link:0:net1:obytes 73324\n # link:0:net1:obytes64 73324\n # link:0:net1:oerrors 0\n # link:0:net1:opackets 1594\n # link:0:net1:opackets64 1594\n # link:0:net1:rbytes 304384894\n # link:0:net1:rbytes64 304384894\n # link:0:net1:snaptime 16834735.1613302\n # link:0:net1:unknowns 0\n # link:0:net1:zonename 53aa9b7e-48ba-4152-a52b-a6368c3d9e7c\n\n # A mapping of solaris names -> datadog names\n metric_by_solaris_name = {\n 'rbytes64': 'bytes_rcvd',\n 'obytes64': 'bytes_sent',\n 'ipackets64': 'packets_in.count',\n 'ierrors': 'packets_in.error',\n 'opackets64': 'packets_out.count',\n 'oerrors': 'packets_out.error',\n }\n\n lines = [l for l in netstat_output.splitlines() if len(l) > 0]\n\n metrics_by_interface = {}\n\n for l in lines:\n # Parse the metric & interface.\n cols = l.split()\n link, n, iface, name = cols[0].split(\":\")\n assert link == \"link\"\n\n # Get the datadog metric name.\n ddname = metric_by_solaris_name.get(name, None)\n if ddname is None:\n continue\n\n # Add it to this interface's list of metrics.\n metrics = metrics_by_interface.get(iface, {})\n metrics[ddname] = self._parse_value(cols[1])\n metrics_by_interface[iface] = metrics\n\n return metrics_by_interface"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _check_psutil(self, instance):\n custom_tags = instance.get('tags', [])\n if self._collect_cx_state:\n self._cx_state_psutil(tags=custom_tags)\n\n self._cx_counters_psutil(tags=custom_tags)", "response": "Gather metrics about connections states and interfaces counters using psutil facilities."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cx_state_psutil(self, tags=None):\n metrics = defaultdict(int)\n tags = [] if tags is None else tags\n for conn in psutil.net_connections():\n protocol = self._parse_protocol_psutil(conn)\n status = self.tcp_states['psutil'].get(conn.status)\n metric = self.cx_state_gauge.get((protocol, status))\n if metric is None:\n self.log.warning('Metric not found for: %s,%s', protocol, status)\n else:\n metrics[metric] += 1\n\n for metric, value in iteritems(metrics):\n self.gauge(metric, value, tags=tags)", "response": "Collect metrics about connections state using psutil"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cx_counters_psutil(self, tags=None):\n tags = [] if tags is None else tags\n for iface, counters in iteritems(psutil.net_io_counters(pernic=True)):\n metrics = {\n 'bytes_rcvd': counters.bytes_recv,\n 'bytes_sent': counters.bytes_sent,\n 'packets_in.count': counters.packets_recv,\n 'packets_in.error': counters.errin,\n 'packets_out.count': counters.packets_sent,\n 'packets_out.error': counters.errout,\n }\n self._submit_devicemetrics(iface, metrics, tags)", "response": "Collect metrics about interfaces counters using psutil."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a string describing the protocol for the given connection", "response": "def _parse_protocol_psutil(self, conn):\n \"\"\"\n Returns a string describing the protocol for the given connection\n in the form `tcp4`, 'udp4` as in `self.cx_state_gauge`\n \"\"\"\n protocol = self.PSUTIL_TYPE_MAPPING.get(conn.type, '')\n family = self.PSUTIL_FAMILY_MAPPING.get(conn.family, '')\n return '{}{}'.format(protocol, family)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprocess the work unit or wait for sentinel to exit", "response": "def run(self):\n \"\"\"Process the work unit, or wait for sentinel to exit\"\"\"\n while True:\n self.running = True\n workunit = self._workq.get()\n if is_sentinel(workunit):\n # Got sentinel\n break\n\n # Run the job / sequence\n workunit.process()\n self.running = False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef imap(self, func, iterable, chunksize=1):\n collector = OrderedResultCollector(as_iterator=True)\n self._create_sequences(func, iterable, chunksize, collector)\n return iter(collector)", "response": "A generator function that returns the result of the function func over the given iterable."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef imap_unordered(self, func, iterable, chunksize=1):\n collector = UnorderedResultCollector()\n self._create_sequences(func, iterable, chunksize, collector)\n return iter(collector)", "response": "The same as imap except that the order of the results\n is arbitrary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef apply_async(self, func, args=(), kwds=dict(), callback=None):\n assert not self._closed # No lock here. We assume it's atomic...\n apply_result = ApplyResult(callback=callback)\n job = Job(func, args, kwds, apply_result)\n self._workq.put(job)\n return apply_result", "response": "A variant of the apply method which returns an ApplyResult object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef map_async(self, func, iterable, chunksize=None, callback=None):\n apply_result = ApplyResult(callback=callback)\n collector = OrderedResultCollector(apply_result, as_iterator=False)\n self._create_sequences(func, iterable, chunksize, collector)\n return apply_result", "response": "A variant of the map method which returns a ApplyResult object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef terminate(self):\n self.close()\n\n # Clearing the job queue\n try:\n while 1:\n self._workq.get_nowait()\n except queue.Empty:\n pass\n\n # Send one sentinel for each worker thread: each thread will die\n # eventually, leaving the next sentinel for the next thread\n for _ in self._workers:\n self._workq.put(SENTINEL)", "response": "Stops the worker processes immediately without completing any outstanding work."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _create_sequences(self, func, iterable, chunksize, collector=None):\n assert not self._closed # No lock here. We assume it's atomic...\n sequences = []\n results = []\n it_ = iter(iterable)\n exit_loop = False\n while not exit_loop:\n seq = []\n for _ in range(chunksize or 1):\n try:\n arg = next(it_)\n except StopIteration:\n exit_loop = True\n break\n apply_result = ApplyResult(collector)\n job = Job(func, (arg,), {}, apply_result)\n seq.append(job)\n results.append(apply_result)\n sequences.append(JobSequence(seq))\n\n for seq in sequences:\n self._workq.put(seq)\n\n return sequences", "response": "Create the work unit objects to process and push them onto the work queue."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess the result of the function.", "response": "def process(self):\n \"\"\"\n Call the function with the args/kwds and tell the ApplyResult\n that its result is ready. Correctly handles the exceptions\n happening during the execution of the function\n \"\"\"\n try:\n result = self._func(*self._args, **self._kwds)\n except:\n self._result._set_exception()\n else:\n self._result._set_value(result)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the result when it arrives within timeout seconds.", "response": "def get(self, timeout=None):\n \"\"\"\n Returns the result when it arrives. If timeout is not None and\n the result does not arrive within timeout seconds then\n TimeoutError is raised. If the remote call raised an exception\n then that exception will be reraised by get().\n \"\"\"\n if not self.wait(timeout):\n raise TimeoutError(\"Result not available within %fs\" % timeout)\n if self._success:\n return self._data\n raise self._data[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef wait(self, timeout=None):\n self._event.wait(timeout)\n return self._event.isSet()", "response": "Waits until the result is available or until timeout is reached."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _set_value(self, value):\n assert not self.ready()\n self._data = value\n self._success = True\n self._event.set()\n if self._collector is not None:\n self._collector.notify_ready(self)\n if self._callback is not None:\n try:\n self._callback(value)\n except:\n traceback.print_exc()", "response": "Set the value of the result of this object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _set_exception(self):\n assert not self.ready()\n self._data = sys.exc_info()\n self._success = False\n self._event.set()\n if self._collector is not None:\n self._collector.notify_ready(self)", "response": "Sets the exception of the object to be raised by the job."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the values of the resultset at the given index", "response": "def _get_result(self, idx, timeout=None):\n \"\"\"Called by the CollectorIterator object to retrieve the\n result's values one after another, in the order the results have\n become available.\n \\param idx The index of the result we want, wrt collector's order\n \\param timeout integer telling how long to wait (in seconds)\n for the result at index idx to be available, or None (wait\n forever)\n \"\"\"\n self._cond.acquire()\n try:\n if idx >= self._expected:\n raise IndexError\n elif idx < len(self._collection):\n return self._collection[idx]\n elif idx != len(self._collection):\n # Violation of the sequence protocol\n raise IndexError()\n else:\n self._cond.wait(timeout=timeout)\n try:\n return self._collection[idx]\n except IndexError:\n # Still not added !\n raise TimeoutError(\"Timeout while waiting for results\")\n finally:\n self._cond.release()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef notify_ready(self, apply_result):\n first_item = False\n self._cond.acquire()\n try:\n self._collection.append(apply_result)\n first_item = len(self._collection) == 1\n\n self._cond.notifyAll()\n finally:\n self._cond.release()\n\n if first_item and self._to_notify is not None:\n self._to_notify._set_value(iter(self))", "response": "Called by the ApplyResult object telling us that the job s result\n is ready."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the values one after another", "response": "def _get_result(self, idx, timeout=None):\n \"\"\"Called by the CollectorIterator object to retrieve the\n result's values one after another (order defined by the\n implementation)\n \\param idx The index of the result we want, wrt collector's order\n \\param timeout integer telling how long to wait (in seconds)\n for the result at index idx to be available, or None (wait\n forever)\n \"\"\"\n res = self._results[idx]\n res.wait(timeout)\n return res"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef notify_ready(self, apply_result):\n got_first = False\n got_last = False\n self._lock.acquire()\n try:\n assert self._remaining > 0\n got_first = len(self._results) == self._remaining\n self._remaining -= 1\n got_last = self._remaining == 0\n finally:\n self._lock.release()\n\n if self._to_notify is not None:\n if self._as_iterator and got_first:\n self._to_notify._set_value(iter(self))\n elif not self._as_iterator and got_last:\n try:\n lst = [r.get(0) for r in self._results]\n except:\n self._to_notify._set_exception()\n else:\n self._to_notify._set_value(lst)", "response": "Called by the ApplyResult object telling us that the job s result\n is ready."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting a command over a socket.", "response": "def command(sock, dbname, spec, slave_ok, is_mongos,\n read_preference, codec_options, check=True,\n allowable_errors=None, address=None,\n check_keys=False, listeners=None, max_bson_size=None,\n read_concern=DEFAULT_READ_CONCERN,\n parse_write_concern_error=False,\n collation=None):\n \"\"\"Execute a command over the socket, or raise socket.error.\n\n :Parameters:\n - `sock`: a raw socket instance\n - `dbname`: name of the database on which to run the command\n - `spec`: a command document as a dict, SON, or mapping object\n - `slave_ok`: whether to set the SlaveOkay wire protocol bit\n - `is_mongos`: are we connected to a mongos?\n - `read_preference`: a read preference\n - `codec_options`: a CodecOptions instance\n - `check`: raise OperationFailure if there are errors\n - `allowable_errors`: errors to ignore if `check` is True\n - `address`: the (host, port) of `sock`\n - `check_keys`: if True, check `spec` for invalid keys\n - `listeners`: An instance of :class:`~pymongo.monitoring.EventListeners`\n - `max_bson_size`: The maximum encoded bson size for this server\n - `read_concern`: The read concern for this command.\n - `parse_write_concern_error`: Whether to parse the ``writeConcernError``\n field in the command response.\n - `collation`: The collation for this command.\n\n \"\"\"\n name = next(iter(spec))\n ns = dbname + '.$cmd'\n flags = 4 if slave_ok else 0\n # Publish the original command document.\n orig = spec\n if is_mongos:\n spec = message._maybe_add_read_preference(spec, read_preference)\n if read_concern.level:\n spec['readConcern'] = read_concern.document\n if collation is not None:\n spec['collation'] = collation\n\n publish = listeners is not None and listeners.enabled_for_commands\n if publish:\n start = datetime.datetime.now()\n\n request_id, msg, size = message.query(flags, ns, 0, -1, spec,\n None, codec_options, check_keys)\n\n if (max_bson_size is not None\n and size > max_bson_size + message._COMMAND_OVERHEAD):\n message._raise_document_too_large(\n name, size, max_bson_size + message._COMMAND_OVERHEAD)\n\n if publish:\n encoding_duration = datetime.datetime.now() - start\n listeners.publish_command_start(orig, dbname, request_id, address)\n start = datetime.datetime.now()\n\n try:\n sock.sendall(msg)\n response = receive_message(sock, 1, request_id)\n unpacked = helpers._unpack_response(\n response, codec_options=codec_options)\n\n response_doc = unpacked['data'][0]\n if check:\n helpers._check_command_response(\n response_doc, None, allowable_errors,\n parse_write_concern_error=parse_write_concern_error)\n except Exception as exc:\n if publish:\n duration = (datetime.datetime.now() - start) + encoding_duration\n if isinstance(exc, (NotMasterError, OperationFailure)):\n failure = exc.details\n else:\n failure = message._convert_exception(exc)\n listeners.publish_command_failure(\n duration, failure, name, request_id, address)\n raise\n if publish:\n duration = (datetime.datetime.now() - start) + encoding_duration\n listeners.publish_command_success(\n duration, response_doc, name, request_id, address)\n return response_doc"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreceives a BSON message from the socket.", "response": "def receive_message(\n sock, operation, request_id, max_message_size=MAX_MESSAGE_SIZE):\n \"\"\"Receive a raw BSON message or raise socket.error.\"\"\"\n header = _receive_data_on_socket(sock, 16)\n length = _UNPACK_INT(header[:4])[0]\n\n actual_op = _UNPACK_INT(header[12:])[0]\n if operation != actual_op:\n raise ProtocolError(\"Got opcode %r but expected \"\n \"%r\" % (actual_op, operation))\n # No request_id for exhaust cursor \"getMore\".\n if request_id is not None:\n response_id = _UNPACK_INT(header[8:12])[0]\n if request_id != response_id:\n raise ProtocolError(\"Got response id %r but expected \"\n \"%r\" % (response_id, request_id))\n if length <= 16:\n raise ProtocolError(\"Message length (%r) not longer than standard \"\n \"message header size (16)\" % (length,))\n if length > max_message_size:\n raise ProtocolError(\"Message length (%r) is larger than server max \"\n \"message size (%r)\" % (length, max_message_size))\n\n return _receive_data_on_socket(sock, length - 16)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a connection to the server and return the socket object.", "response": "def create_connection(self):\n \"\"\"See: https://github.com/python/cpython/blob/40ee9a3640d702bce127e9877c82a99ce817f0d1/Lib/socket.py#L691\"\"\"\n err = None\n try:\n for res in socket.getaddrinfo(self._server, self._port, 0, self._sock_type):\n af, socktype, proto, canonname, sa = res\n sock = None\n try:\n sock = socket.socket(af, socktype, proto)\n sock.settimeout(self._timeout)\n sock.connect(sa)\n # Break explicitly a reference cycle\n err = None\n return sock\n\n except socket.error as _:\n err = _\n if sock is not None:\n sock.close()\n\n if err is not None:\n raise err\n else:\n raise socket.error('No valid addresses found, try checking your IPv6 connectivity')\n except socket.gaierror as e:\n err_code, message = e.args\n if err_code == socket.EAI_NODATA or err_code == socket.EAI_NONAME:\n raise socket.error('Unable to resolve host, check your DNS: {}'.format(message))\n\n raise"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dumps(obj, *args, **kwargs):\n json_options = kwargs.pop(\"json_options\", DEFAULT_JSON_OPTIONS)\n return json.dumps(_json_convert(obj, json_options), *args, **kwargs)", "response": "Wrapper function that wraps json. dumps."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef loads(s, *args, **kwargs):\n json_options = kwargs.pop(\"json_options\", DEFAULT_JSON_OPTIONS)\n if _HAS_OBJECT_PAIRS_HOOK:\n kwargs[\"object_pairs_hook\"] = lambda pairs: object_pairs_hook(\n pairs, json_options)\n else:\n kwargs[\"object_hook\"] = lambda obj: object_hook(obj, json_options)\n return json.loads(s, *args, **kwargs)", "response": "A wrapper function that wraps json. loads."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndecoding a JSON datetime to python datetime. datetime.", "response": "def _parse_canonical_datetime(doc, json_options):\n \"\"\"Decode a JSON datetime to python datetime.datetime.\"\"\"\n dtm = doc[\"$date\"]\n if len(doc) != 1:\n raise TypeError('Bad $date, extra field(s): %s' % (doc,))\n # mongoexport 2.6 and newer\n if isinstance(dtm, string_type):\n # Parse offset\n if dtm[-1] == 'Z':\n dt = dtm[:-1]\n offset = 'Z'\n elif dtm[-3] == ':':\n # (+|-)HH:MM\n dt = dtm[:-6]\n offset = dtm[-6:]\n elif dtm[-5] in ('+', '-'):\n # (+|-)HHMM\n dt = dtm[:-5]\n offset = dtm[-5:]\n elif dtm[-3] in ('+', '-'):\n # (+|-)HH\n dt = dtm[:-3]\n offset = dtm[-3:]\n else:\n dt = dtm\n offset = ''\n\n # Parse the optional factional seconds portion.\n dot_index = dt.rfind('.')\n microsecond = 0\n if dot_index != -1:\n microsecond = int(float(dt[dot_index:]) * 1000000)\n dt = dt[:dot_index]\n\n aware = datetime.datetime.strptime(\n dt, \"%Y-%m-%dT%H:%M:%S\").replace(microsecond=microsecond,\n tzinfo=utc)\n\n if offset and offset != 'Z':\n if len(offset) == 6:\n hours, minutes = offset[1:].split(':')\n secs = (int(hours) * 3600 + int(minutes) * 60)\n elif len(offset) == 5:\n secs = (int(offset[1:3]) * 3600 + int(offset[3:]) * 60)\n elif len(offset) == 3:\n secs = int(offset[1:3]) * 3600\n if offset[0] == \"-\":\n secs *= -1\n aware = aware - datetime.timedelta(seconds=secs)\n\n if json_options.tz_aware:\n if json_options.tzinfo:\n aware = aware.astimezone(json_options.tzinfo)\n return aware\n else:\n return aware.replace(tzinfo=None)\n return bson._millis_to_datetime(int(dtm), json_options)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecodes a JSON symbol to Python string.", "response": "def _parse_canonical_symbol(doc):\n \"\"\"Decode a JSON symbol to Python string.\"\"\"\n symbol = doc['$symbol']\n if len(doc) != 1:\n raise TypeError('Bad $symbol, extra field(s): %s' % (doc,))\n return text_type(symbol)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _parse_canonical_code(doc):\n for key in doc:\n if key not in ('$code', '$scope'):\n raise TypeError('Bad $code, extra field(s): %s' % (doc,))\n return Code(doc['$code'], scope=doc.get('$scope'))", "response": "Decode a JSON code to bson. code. Code."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndecoding a JSON regex to bson. regex. Regex.", "response": "def _parse_canonical_regex(doc):\n \"\"\"Decode a JSON regex to bson.regex.Regex.\"\"\"\n regex = doc['$regularExpression']\n if len(doc) != 1:\n raise TypeError('Bad $regularExpression, extra field(s): %s' % (doc,))\n if len(regex) != 2:\n raise TypeError('Bad $regularExpression must include only \"pattern\"'\n 'and \"options\" components: %s' % (doc,))\n return Regex(regex['pattern'], regex['options'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecode a JSON DBRef to bson. dbref. DBRef.", "response": "def _parse_canonical_dbref(doc):\n \"\"\"Decode a JSON DBRef to bson.dbref.DBRef.\"\"\"\n for key in doc:\n if key.startswith('$') and key not in _DBREF_KEYS:\n # Other keys start with $, so dct cannot be parsed as a DBRef.\n return doc\n return DBRef(doc.pop('$ref'), doc.pop('$id'),\n database=doc.pop('$db', None), **doc)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_canonical_dbpointer(doc):\n dbref = doc['$dbPointer']\n if len(doc) != 1:\n raise TypeError('Bad $dbPointer, extra field(s): %s' % (doc,))\n if isinstance(dbref, DBRef):\n dbref_doc = dbref.as_doc()\n # DBPointer must not contain $db in its value.\n if dbref.database is not None:\n raise TypeError(\n 'Bad $dbPointer, extra field $db: %s' % (dbref_doc,))\n if not isinstance(dbref.id, ObjectId):\n raise TypeError(\n 'Bad $dbPointer, $id must be an ObjectId: %s' % (dbref_doc,))\n if len(dbref_doc) != 2:\n raise TypeError(\n 'Bad $dbPointer, extra field(s) in DBRef: %s' % (dbref_doc,))\n return dbref\n else:\n raise TypeError('Bad $dbPointer, expected a DBRef: %s' % (doc,))", "response": "Decode a JSON ( deprecated ) DBPointer to bson. dbref. DBRef."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_canonical_int32(doc):\n i_str = doc['$numberInt']\n if len(doc) != 1:\n raise TypeError('Bad $numberInt, extra field(s): %s' % (doc,))\n if not isinstance(i_str, string_type):\n raise TypeError('$numberInt must be string: %s' % (doc,))\n return int(i_str)", "response": "Decode a JSON int32 to python int."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndecode a JSON int64 to bson. int64. Int64.", "response": "def _parse_canonical_int64(doc):\n \"\"\"Decode a JSON int64 to bson.int64.Int64.\"\"\"\n l_str = doc['$numberLong']\n if len(doc) != 1:\n raise TypeError('Bad $numberLong, extra field(s): %s' % (doc,))\n return Int64(l_str)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndecode a JSON double to python float.", "response": "def _parse_canonical_double(doc):\n \"\"\"Decode a JSON double to python float.\"\"\"\n d_str = doc['$numberDouble']\n if len(doc) != 1:\n raise TypeError('Bad $numberDouble, extra field(s): %s' % (doc,))\n if not isinstance(d_str, string_type):\n raise TypeError('$numberDouble must be string: %s' % (doc,))\n return float(d_str)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_canonical_decimal128(doc):\n d_str = doc['$numberDecimal']\n if len(doc) != 1:\n raise TypeError('Bad $numberDecimal, extra field(s): %s' % (doc,))\n if not isinstance(d_str, string_type):\n raise TypeError('$numberDecimal must be string: %s' % (doc,))\n return Decimal128(d_str)", "response": "Decode a JSON decimal128 to bson. decimal128. Decimal128."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecode a JSON MinKey to bson. min_key. MinKey.", "response": "def _parse_canonical_minkey(doc):\n \"\"\"Decode a JSON MinKey to bson.min_key.MinKey.\"\"\"\n if doc['$minKey'] is not 1:\n raise TypeError('$minKey value must be 1: %s' % (doc,))\n if len(doc) != 1:\n raise TypeError('Bad $minKey, extra field(s): %s' % (doc,))\n return MinKey()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse_canonical_maxkey(doc):\n if doc['$maxKey'] is not 1:\n raise TypeError('$maxKey value must be 1: %s', (doc,))\n if len(doc) != 1:\n raise TypeError('Bad $minKey, extra field(s): %s' % (doc,))\n return MaxKey()", "response": "Decode a JSON MaxKey to bson. max_key. MaxKey."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses read preference options.", "response": "def _parse_read_preference(options):\n \"\"\"Parse read preference options.\"\"\"\n if 'read_preference' in options:\n return options['read_preference']\n\n name = options.get('readpreference', 'primary')\n mode = read_pref_mode_from_name(name)\n tags = options.get('readpreferencetags')\n max_staleness = options.get('maxstalenessseconds', -1)\n return make_read_preference(mode, tags, max_staleness)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing connection pool options.", "response": "def _parse_pool_options(options):\n \"\"\"Parse connection pool options.\"\"\"\n max_pool_size = options.get('maxpoolsize', common.MAX_POOL_SIZE)\n min_pool_size = options.get('minpoolsize', common.MIN_POOL_SIZE)\n max_idle_time_ms = options.get('maxidletimems', common.MAX_IDLE_TIME_MS)\n if max_pool_size is not None and min_pool_size > max_pool_size:\n raise ValueError(\"minPoolSize must be smaller or equal to maxPoolSize\")\n connect_timeout = options.get('connecttimeoutms', common.CONNECT_TIMEOUT)\n socket_keepalive = options.get('socketkeepalive', True)\n socket_timeout = options.get('sockettimeoutms')\n wait_queue_timeout = options.get('waitqueuetimeoutms')\n wait_queue_multiple = options.get('waitqueuemultiple')\n event_listeners = options.get('event_listeners')\n appname = options.get('appname')\n ssl_context, ssl_match_hostname = _parse_ssl_options(options)\n return PoolOptions(max_pool_size,\n min_pool_size,\n max_idle_time_ms,\n connect_timeout, socket_timeout,\n wait_queue_timeout, wait_queue_multiple,\n ssl_context, ssl_match_hostname, socket_keepalive,\n _EventListeners(event_listeners),\n appname)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate all service_checks. json files.", "response": "def service_checks():\n \"\"\"Validate all `service_checks.json` files.\"\"\"\n root = get_root()\n echo_info(\"Validating all service_checks.json files...\")\n failed_checks = 0\n ok_checks = 0\n for check_name in sorted(os.listdir(root)):\n service_checks_file = os.path.join(root, check_name, 'service_checks.json')\n\n if file_exists(service_checks_file):\n file_failed = False\n display_queue = []\n\n try:\n decoded = json.loads(read_file(service_checks_file).strip(), object_pairs_hook=OrderedDict)\n except JSONDecodeError as e:\n failed_checks += 1\n echo_info(\"{}/service_checks.json... \".format(check_name), nl=False)\n echo_failure(\"FAILED\")\n echo_failure(' invalid json: {}'.format(e))\n continue\n\n unique_names = set()\n unique_checks = set()\n for service_check in decoded:\n # attributes are valid\n attrs = set(service_check)\n for attr in sorted(attrs - REQUIRED_ATTRIBUTES):\n file_failed = True\n display_queue.append((echo_failure, ' Attribute `{}` is invalid'.format(attr)))\n for attr in sorted(REQUIRED_ATTRIBUTES - attrs):\n file_failed = True\n display_queue.append((echo_failure, ' Attribute `{}` is required'.format(attr)))\n\n # agent_version\n agent_version = service_check.get('agent_version')\n version_parts = parse_version_parts(agent_version)\n if len(version_parts) != 3:\n file_failed = True\n\n if not agent_version:\n output = ' required non-null string: agent_version'\n else:\n output = ' invalid `agent_version`: {}'.format(agent_version)\n\n display_queue.append((echo_failure, output))\n\n # check\n check = service_check.get('check')\n if not check or not isinstance(check, string_types):\n file_failed = True\n display_queue.append((echo_failure, ' required non-null string: check'))\n else:\n if check in unique_checks:\n file_failed = True\n display_queue.append((echo_failure, ' {} is not a unique check'.format(check)))\n else:\n unique_checks.add(check)\n\n # description\n description = service_check.get('description')\n if not description or not isinstance(description, string_types):\n file_failed = True\n display_queue.append((echo_failure, ' required non-null string: description'))\n\n # groups\n groups = service_check.get('groups')\n if groups is None or not isinstance(groups, list):\n file_failed = True\n display_queue.append((echo_failure, ' required list: groups'))\n\n # integration\n integration = service_check.get('integration')\n if integration is None or not isinstance(integration, string_types):\n file_failed = True\n display_queue.append((echo_failure, ' required non-null string: integration'))\n\n # name\n name = service_check.get('name')\n if not name or not isinstance(name, string_types):\n file_failed = True\n display_queue.append((echo_failure, ' required non-null string: name'))\n else:\n if name in unique_names:\n file_failed = True\n display_queue.append((echo_failure, ' {} is not a unique name'.format(name)))\n else:\n unique_names.add(name)\n\n # statuses\n statuses = service_check.get('statuses')\n if not statuses or not isinstance(statuses, list):\n file_failed = True\n display_queue.append((echo_failure, ' required non empty list: statuses'))\n\n if file_failed:\n failed_checks += 1\n # Display detailed info if file invalid\n echo_info(\"{}/service_checks.json... \".format(check_name), nl=False)\n echo_failure(\"FAILED\")\n for display_func, message in display_queue:\n display_func(message)\n else:\n ok_checks += 1\n\n if ok_checks:\n echo_success(\"{} valid files\".format(ok_checks))\n if failed_checks:\n echo_failure(\"{} invalid files\".format(failed_checks))\n abort()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a WMI time string into a 9 - tuple.", "response": "def to_time(wmi_time):\n \"\"\"Convenience wrapper to take a WMI datetime string of the form\n yyyymmddHHMMSS.mmmmmm+UUU and return a 9-tuple containing the\n individual elements, or None where string contains placeholder\n stars.\n\n :param wmi_time: The WMI datetime string in `yyyymmddHHMMSS.mmmmmm+UUU` format\n\n :returns: A 9-tuple of (year, month, day, hours, minutes, seconds, microseconds, timezone)\n \"\"\"\n\n def int_or_none(s, start, end):\n try:\n return int(s[start:end])\n except ValueError:\n return None\n\n year = int_or_none(wmi_time, 0, 4)\n month = int_or_none(wmi_time, 4, 6)\n day = int_or_none(wmi_time, 6, 8)\n hours = int_or_none(wmi_time, 8, 10)\n minutes = int_or_none(wmi_time, 10, 12)\n seconds = int_or_none(wmi_time, 12, 14)\n microseconds = int_or_none(wmi_time, 15, 21)\n timezone = wmi_time[22:]\n\n if timezone == \"***\":\n timezone = None\n\n return year, month, day, hours, minutes, seconds, microseconds, timezone"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _format_tag_query(self, sampler, wmi_obj, tag_query):\n try:\n link_source_property = int(wmi_obj[tag_query[0]])\n target_class = tag_query[1]\n link_target_class_property = tag_query[2]\n target_property = tag_query[3]\n except IndexError:\n self.log.error(\n u\"Wrong `tag_queries` parameter format. \" \"Please refer to the configuration file for more information.\"\n )\n raise\n except TypeError:\n self.log.error(\n u\"Incorrect 'link source property' in `tag_queries` parameter:\"\n \" `{wmi_property}` is not a property of `{wmi_class}`\".format(\n wmi_property=tag_query[0], wmi_class=sampler.class_name\n )\n )\n raise\n\n return (target_class, target_property, [{link_target_class_property: link_source_property}])", "response": "Format `tag_query` or raise on incorrect parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndesigns a query based on the given WMIObject to extract a tag.", "response": "def _get_tag_query_tag(self, sampler, wmi_obj, tag_query):\n \"\"\"\n Design a query based on the given WMIObject to extract a tag.\n\n Returns: tag or TagQueryUniquenessFailure exception.\n \"\"\"\n self.log.debug(\n u\"`tag_queries` parameter found.\"\n \" wmi_object={wmi_obj} - query={tag_query}\".format(wmi_obj=wmi_obj, tag_query=tag_query)\n )\n\n # Extract query information\n target_class, target_property, filters = self._format_tag_query(sampler, wmi_obj, tag_query)\n\n # Create a specific sampler\n tag_query_sampler = WMISampler(self.log, target_class, [target_property], filters=filters, **sampler.connection)\n\n tag_query_sampler.sample()\n\n # Extract tag\n self._raise_on_invalid_tag_query_result(tag_query_sampler, wmi_obj, tag_query)\n\n link_value = str(tag_query_sampler[0][target_property]).lower()\n\n tag = \"{tag_name}:{tag_value}\".format(tag_name=target_property.lower(), tag_value=\"_\".join(link_value.split()))\n\n self.log.debug(u\"Extracted `tag_queries` tag: '{tag}'\".format(tag=tag))\n return tag"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextracting and tag metrics from the WMI sampler.", "response": "def _extract_metrics(self, wmi_sampler, tag_by, tag_queries, constant_tags):\n \"\"\"\n Extract and tag metrics from the WMISampler.\n\n Raise when multiple WMIObject were returned by the sampler with no `tag_by` specified.\n\n Returns: List of WMIMetric\n ```\n [\n WMIMetric(\"freemegabytes\", 19742, [\"name:_total\"]),\n WMIMetric(\"avgdiskbytesperwrite\", 1536, [\"name:c:\"]),\n ]\n ```\n \"\"\"\n if len(wmi_sampler) > 1 and not tag_by:\n raise MissingTagBy(\n u\"WMI query returned multiple rows but no `tag_by` value was given.\"\n \" class={wmi_class} - properties={wmi_properties} - filters={filters}\".format(\n wmi_class=wmi_sampler.class_name,\n wmi_properties=wmi_sampler.property_names,\n filters=wmi_sampler.filters,\n )\n )\n\n metrics = []\n tag_by = tag_by.lower()\n\n for wmi_obj in wmi_sampler:\n tags = list(constant_tags) if constant_tags else []\n\n # Tag with `tag_queries` parameter\n for query in tag_queries:\n try:\n tags.append(self._get_tag_query_tag(wmi_sampler, wmi_obj, query))\n except TagQueryUniquenessFailure:\n continue\n\n for wmi_property, wmi_value in iteritems(wmi_obj):\n # skips any property not in arguments since SWbemServices.ExecQuery will return key prop properties\n # https://msdn.microsoft.com/en-us/library/aa393866(v=vs.85).aspx\n if wmi_property.lower() not in (s.lower() for s in wmi_sampler.property_names):\n continue\n # Tag with `tag_by` parameter\n if wmi_property == tag_by:\n tag_value = str(wmi_value).lower()\n if tag_queries and tag_value.find(\"#\") > 0:\n tag_value = tag_value[: tag_value.find(\"#\")]\n\n tags.append(\"{name}:{value}\".format(name=tag_by, value=tag_value))\n continue\n\n # No metric extraction on 'Name' property\n if wmi_property == 'name':\n continue\n\n try:\n metrics.append(WMIMetric(wmi_property, float(wmi_value), tags))\n except ValueError:\n self.log.warning(\n u\"When extracting metrics with WMI, found a non digit value\"\n \" for property '{0}'.\".format(wmi_property)\n )\n continue\n except TypeError:\n self.log.warning(\n u\"When extracting metrics with WMI, found a missing property\" \" '{0}'\".format(wmi_property)\n )\n continue\n return metrics"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsubmits the given metrics to the user.", "response": "def _submit_metrics(self, metrics, metric_name_and_type_by_property):\n \"\"\"\n Resolve metric names and types and submit it.\n \"\"\"\n for metric in metrics:\n if (\n metric.name not in metric_name_and_type_by_property\n and metric.name.lower() not in metric_name_and_type_by_property\n ):\n # Only report the metrics that were specified in the configration\n # Ignore added properties like 'Timestamp_Sys100NS', `Frequency_Sys100NS`, etc ...\n continue\n\n if metric_name_and_type_by_property.get(metric.name):\n metric_name, metric_type = metric_name_and_type_by_property[metric.name]\n elif metric_name_and_type_by_property.get(metric.name.lower()):\n metric_name, metric_type = metric_name_and_type_by_property[metric.name.lower()]\n else:\n continue\n\n try:\n func = getattr(self, metric_type.lower())\n except AttributeError:\n raise Exception(u\"Invalid metric type: {0}\".format(metric_type))\n\n func(metric_name, metric.value, metric.tags)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn an index key for a given instance. Useful for caching.", "response": "def _get_instance_key(self, host, namespace, wmi_class, other=None):\n \"\"\"\n Return an index key for a given instance. Useful for caching.\n \"\"\"\n if other:\n return \"{host}:{namespace}:{wmi_class}-{other}\".format(\n host=host, namespace=namespace, wmi_class=wmi_class, other=other\n )\n\n return \"{host}:{namespace}:{wmi_class}\".format(host=host, namespace=namespace, wmi_class=wmi_class)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a WMISampler instance for the given instance_key.", "response": "def _get_wmi_sampler(self, instance_key, wmi_class, properties, tag_by=\"\", **kwargs):\n \"\"\"\n Create and cache a WMISampler for the given (class, properties)\n \"\"\"\n properties = list(properties) + [tag_by] if tag_by else list(properties)\n\n if instance_key not in self.wmi_samplers:\n wmi_sampler = WMISampler(self.log, wmi_class, properties, **kwargs)\n self.wmi_samplers[instance_key] = wmi_sampler\n\n return self.wmi_samplers[instance_key]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget WMI properties for a given instance.", "response": "def _get_wmi_properties(self, instance_key, metrics, tag_queries):\n \"\"\"\n Create and cache a (metric name, metric type) by WMI property map and a property list.\n \"\"\"\n if instance_key not in self.wmi_props:\n metric_name_by_property = dict(\n (wmi_property.lower(), (metric_name, metric_type)) for wmi_property, metric_name, metric_type in metrics\n )\n properties = map(lambda x: x[0], metrics + tag_queries)\n self.wmi_props[instance_key] = (metric_name_by_property, properties)\n\n return self.wmi_props[instance_key]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsubmits a metric as a rate", "response": "def _submit_rate(self, metric_name, val, metric, custom_tags=None, hostname=None):\n \"\"\"\n Submit a metric as a rate, additional tags provided will be added to\n the ones from the label provided via the metrics object.\n\n `custom_tags` is an array of 'tag:value' that will be added to the\n metric when sending the rate to Datadog.\n \"\"\"\n _tags = self._metric_tags(metric_name, val, metric, custom_tags, hostname)\n self.rate('{}.{}'.format(self.NAMESPACE, metric_name), val, _tags, hostname=hostname)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsubmits a metric as a monotonic count.", "response": "def _submit_monotonic_count(self, metric_name, val, metric, custom_tags=None, hostname=None):\n \"\"\"\n Submit a metric as a monotonic count, additional tags provided will be added to\n the ones from the label provided via the metrics object.\n\n `custom_tags` is an array of 'tag:value' that will be added to the\n metric when sending the monotonic count to Datadog.\n \"\"\"\n\n _tags = self._metric_tags(metric_name, val, metric, custom_tags, hostname)\n self.monotonic_count('{}.{}'.format(self.NAMESPACE, metric_name), val, _tags, hostname=hostname)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_expvar_data(self, data, tags, metrics, max_metrics, namespace):\n '''\n Report all the metrics based on the configuration in instance\n If a metric is not well configured or is not present in the payload,\n continue processing metrics but log the information to the info page\n '''\n count = 0\n for metric in metrics:\n path = metric.get(PATH)\n metric_type = metric.get(TYPE, DEFAULT_TYPE)\n metric_tags = list(metric.get(TAGS, []))\n metric_tags += tags\n alias = metric.get(ALIAS)\n\n if not path:\n self.warning(\"Metric %s has no path\" % metric)\n continue\n\n if metric_type not in SUPPORTED_TYPES:\n self.warning(\"Metric type %s not supported for this check\" % metric_type)\n continue\n\n keys = path.split(\"/\")\n values = self.deep_get(data, keys)\n\n if len(values) == 0:\n self.warning(\"No results matching path %s\" % path)\n continue\n\n tag_by_path = alias is not None\n\n for traversed_path, value in values:\n actual_path = \".\".join(traversed_path)\n path_tag = [\"path:%s\" % actual_path] if tag_by_path else []\n\n metric_name = alias or self.normalize(actual_path, namespace, fix_case=True)\n\n try:\n float(value)\n except ValueError:\n self.log.warning(\"Unreportable value for path %s: %s\" % (path, value))\n continue\n\n if count >= max_metrics:\n self.warning(\n \"Reporting more metrics than the allowed maximum. \"\n \"Please contact support@datadoghq.com for more information.\"\n )\n return\n\n SUPPORTED_TYPES[metric_type](self, metric_name, value, metric_tags + path_tag)\n count += 1", "response": "Parse the expvar data and return the corresponding instance of ExpVar class."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deep_get(self, content, keys, traversed_path=None):\n '''\n Allow to retrieve content nested inside a several layers deep dict/list\n\n Examples: -content: {\n \"key1\": {\n \"key2\" : [\n {\n \"name\" : \"object1\",\n \"value\" : 42\n },\n {\n \"name\" : \"object2\",\n \"value\" : 72\n }\n ]\n }\n }\n -keys: [\"key1\", \"key2\", \"1\", \"value\"]\n would return:\n [([\"key1\", \"key2\", \"1\", \"value\"], 72)]\n -keys: [\"key1\", \"key2\", \"1\", \"*\"]\n would return:\n [([\"key1\", \"key2\", \"1\", \"value\"], 72), ([\"key1\", \"key2\", \"1\", \"name\"], \"object2\")]\n -keys: [\"key1\", \"key2\", \"*\", \"value\"]\n would return:\n [([\"key1\", \"key2\", \"1\", \"value\"], 72), ([\"key1\", \"key2\", \"0\", \"value\"], 42)]\n '''\n\n if traversed_path is None:\n traversed_path = []\n\n if keys == []:\n return [(traversed_path, content)]\n\n key = keys[0]\n regex = \"\".join([\"^\", key, \"$\"])\n try:\n key_rex = re.compile(regex)\n except Exception:\n self.warning(\"Cannot compile regex: %s\" % regex)\n return []\n\n results = []\n for new_key, new_content in self.items(content):\n if key_rex.match(new_key):\n results.extend(self.deep_get(new_content, keys[1:], traversed_path + [str(new_key)]))\n return results", "response": "This method recursively gets the content of a deep dict."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsplits a host string into host and port pair.", "response": "def partition_node(node):\n \"\"\"Split a host:port string into (host, int(port)) pair.\"\"\"\n host = node\n port = 27017\n idx = node.rfind(':')\n if idx != -1:\n host, port = node[:idx], int(node[idx + 1:])\n if host.startswith('['):\n host = host[1:-1]\n return host, port"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_boolean_or_string(option, value):\n if isinstance(value, string_type):\n if value not in ('true', 'false'):\n raise ValueError(\"The value of %s must be \"\n \"'true' or 'false'\" % (option,))\n return value == 'true'\n return validate_boolean(option, value)", "response": "Validates that value is True False True or false."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate_positive_integer(option, value):\n val = validate_integer(option, value)\n if val <= 0:\n raise ValueError(\"The value of %s must be \"\n \"a positive integer\" % (option,))\n return val", "response": "Validate that value is a positive integer which does not include 0."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate_non_negative_integer(option, value):\n val = validate_integer(option, value)\n if val < 0:\n raise ValueError(\"The value of %s must be \"\n \"a non negative integer\" % (option,))\n return val", "response": "Validate that value is a positive integer or 0."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_readable(option, value):\n if value is None:\n return value\n # First make sure its a string py3.3 open(True, 'r') succeeds\n # Used in ssl cert checking due to poor ssl module error reporting\n value = validate_string(option, value)\n open(value, 'r').close()\n return value", "response": "Validates that value is file - like and readable."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nvalidate that value is an instance of basestring for Python 2 or str for Python 3.", "response": "def validate_string(option, value):\n \"\"\"Validates that 'value' is an instance of `basestring` for Python 2\n or `str` for Python 3.\n \"\"\"\n if isinstance(value, string_type):\n return value\n raise TypeError(\"Wrong type for %s, value must be \"\n \"an instance of %s\" % (option, string_type.__name__))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_positive_float(option, value):\n errmsg = \"%s must be an integer or float\" % (option,)\n try:\n value = float(value)\n except ValueError:\n raise ValueError(errmsg)\n except TypeError:\n raise TypeError(errmsg)\n\n # float('inf') doesn't work in 2.4 or 2.5 on Windows, so just cap floats at\n # one billion - this is a reasonable approximation for infinity\n if not 0 < value < 1e9:\n raise ValueError(\"%s must be greater than 0 and \"\n \"less than one billion\" % (option,))\n return value", "response": "Validates that value is a float or can be converted to one and is\n positive."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate a timeout specified in milliseconds returning a value in floating point seconds for the case where None is an error 0 is a ConfigurationError.", "response": "def validate_timeout_or_zero(option, value):\n \"\"\"Validates a timeout specified in milliseconds returning\n a value in floating point seconds for the case where None is an error\n and 0 is valid. Setting the timeout to nothing in the URI string is a\n config error.\n \"\"\"\n if value is None:\n raise ConfigurationError(\"%s cannot be None\" % (option, ))\n if value == 0 or value == \"0\":\n return 0\n return validate_positive_float(option, value) / 1000.0"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_auth_mechanism(option, value):\n # CRAM-MD5 is for server testing only. Undocumented,\n # unsupported, may be removed at any time. You have\n # been warned.\n if value not in MECHANISMS and value != 'CRAM-MD5':\n raise ValueError(\"%s must be in %s\" % (option, tuple(MECHANISMS)))\n return value", "response": "Validate the authMechanism URI option."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvalidating the uuid representation option selected in the URI.", "response": "def validate_uuid_representation(dummy, value):\n \"\"\"Validate the uuid representation option selected in the URI.\n \"\"\"\n try:\n return _UUID_REPRESENTATIONS[value]\n except KeyError:\n raise ValueError(\"%s is an invalid UUID representation. \"\n \"Must be one of \"\n \"%s\" % (value, tuple(_UUID_REPRESENTATIONS)))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_read_preference_tags(name, value):\n if not isinstance(value, list):\n value = [value]\n\n tag_sets = []\n for tag_set in value:\n if tag_set == '':\n tag_sets.append({})\n continue\n try:\n tag_sets.append(dict([tag.split(\":\")\n for tag in tag_set.split(\",\")]))\n except Exception:\n raise ValueError(\"%r not a valid \"\n \"value for %s\" % (tag_set, name))\n return tag_sets", "response": "Parse readPreferenceTags if passed as a client kwarg."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_document_class(option, value):\n if not issubclass(value, (collections.MutableMapping, RawBSONDocument)):\n raise TypeError(\"%s must be dict, bson.son.SON, \"\n \"bson.raw_bson.RawBSONDocument, or a \"\n \"sublass of collections.MutableMapping\" % (option,))\n return value", "response": "Validate the document_class option."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef validate_is_document_type(option, value):\n if not isinstance(value, (collections.MutableMapping, RawBSONDocument)):\n raise TypeError(\"%s must be an instance of dict, bson.son.SON, \"\n \"bson.raw_bson.RawBSONDocument, or \"\n \"a type that inherits from \"\n \"collections.MutableMapping\" % (option,))", "response": "Validate the type of method arguments that expect a MongoDB document."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates the appname option.", "response": "def validate_appname_or_none(option, value):\n \"\"\"Validate the appname option.\"\"\"\n if value is None:\n return value\n validate_string(option, value)\n # We need length in bytes, so encode utf8 first.\n if len(value.encode('utf-8')) > 128:\n raise ValueError(\"%s must be <= 128 bytes\" % (option,))\n return value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate a replacement document.", "response": "def validate_ok_for_replace(replacement):\n \"\"\"Validate a replacement document.\"\"\"\n validate_is_mapping(\"replacement\", replacement)\n # Replacement can be {}\n if replacement and not isinstance(replacement, RawBSONDocument):\n first = next(iter(replacement))\n if first.startswith('$'):\n raise ValueError('replacement can not include $ operators')"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nvalidate an update document.", "response": "def validate_ok_for_update(update):\n \"\"\"Validate an update document.\"\"\"\n validate_is_mapping(\"update\", update)\n # Update can not be {}\n if not update:\n raise ValueError('update only works with $ operators')\n first = next(iter(update))\n if not first.startswith('$'):\n raise ValueError('update only works with $ operators')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_unicode_decode_error_handler(dummy, value):\n if value not in _UNICODE_DECODE_ERROR_HANDLERS:\n raise ValueError(\"%s is an invalid Unicode decode error handler. \"\n \"Must be one of \"\n \"%s\" % (value, tuple(_UNICODE_DECODE_ERROR_HANDLERS)))\n return value", "response": "Validate Unicode decode error handler option of CodecOptions."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates the tzinfo option", "response": "def validate_tzinfo(dummy, value):\n \"\"\"Validate the tzinfo option\n \"\"\"\n if value is not None and not isinstance(value, datetime.tzinfo):\n raise TypeError(\"%s must be an instance of datetime.tzinfo\" % value)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_auth_option(option, value):\n lower, value = validate(option, value)\n if lower not in _AUTH_OPTIONS:\n raise ConfigurationError('Unknown '\n 'authentication option: %s' % (option,))\n return lower, value", "response": "Validate optional authentication parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nvalidates each entry in options and raise a warning if it is not valid.", "response": "def get_validated_options(options, warn=True):\n \"\"\"Validate each entry in options and raise a warning if it is not valid.\n Returns a copy of options with invalid entries removed\n \"\"\"\n validated_options = {}\n for opt, value in iteritems(options):\n lower = opt.lower()\n try:\n validator = URI_VALIDATORS.get(lower, raise_config_error)\n value = validator(opt, value)\n except (ValueError, ConfigurationError) as exc:\n if warn:\n warnings.warn(str(exc))\n else:\n raise\n else:\n validated_options[lower] = value\n return validated_options"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse_dict_string(self, string, key, default):\n try:\n for item in string.split(','):\n k, v = item.rsplit('=', 1)\n if k == key:\n try:\n return int(v)\n except ValueError:\n return v\n return default\n except Exception:\n self.log.exception(\"Cannot parse dictionary string: %s\" % string)\n return default", "response": "Take from a more recent redis. py parse_info"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _check_key_lengths(self, conn, instance, tags):\n key_list = instance.get('keys')\n\n if key_list is None:\n return\n\n if not isinstance(key_list, list) or len(key_list) == 0:\n self.warning(\"keys in redis configuration is either not a list or empty\")\n return\n\n # get all the available databases\n databases = list(conn.info('keyspace'))\n if not databases:\n self.warning(\"Redis database is empty\")\n return\n\n # convert to integer the output of `keyspace`, from `db0` to `0`\n # and store items in a set\n databases = [int(dbstring[2:]) for dbstring in databases]\n\n # user might have configured the instance to target one specific db\n if 'db' in instance:\n db = instance['db']\n if db not in databases:\n self.warning(\"Cannot find database {}\".format(instance['db']))\n return\n databases = [db]\n\n # maps a key to the total length across databases\n lengths_overall = defaultdict(int)\n\n # don't overwrite the configured instance, use a copy\n tmp_instance = deepcopy(instance)\n\n for db in databases:\n lengths = defaultdict(lambda: defaultdict(int))\n tmp_instance['db'] = db\n db_conn = self._get_conn(tmp_instance)\n\n for key_pattern in key_list:\n if re.search(r\"(? 0:\n self.gauge(\n 'redis.key.length',\n total[\"length\"],\n tags=tags\n + ['key:{}'.format(key), 'key_type:{}'.format(total[\"key_type\"]), 'redis_db:db{}'.format(db)],\n )\n\n # Warn if a key is missing from the entire redis instance.\n # Send 0 if the key is missing/empty from the entire redis instance.\n for key, total in iteritems(lengths_overall):\n if total == 0 and instance.get(\"warn_on_missing_keys\", True):\n self.gauge('redis.key.length', total, tags=tags + ['key:{}'.format(key)])\n self.warning(\"{0} key not found in redis\".format(key))", "response": "Compute the length of the configured keys across all the databases and store the result in a set of keys that are configured for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves length and entries from Redis SLOWLOG", "response": "def _check_slowlog(self, instance, custom_tags):\n \"\"\"Retrieve length and entries from Redis' SLOWLOG\n\n This will parse through all entries of the SLOWLOG and select ones\n within the time range between the last seen entries and now\n\n \"\"\"\n conn = self._get_conn(instance)\n\n tags = self._get_tags(custom_tags, instance)\n\n if not instance.get(MAX_SLOW_ENTRIES_KEY):\n try:\n max_slow_entries = int(conn.config_get(MAX_SLOW_ENTRIES_KEY)[MAX_SLOW_ENTRIES_KEY])\n if max_slow_entries > DEFAULT_MAX_SLOW_ENTRIES:\n self.warning(\n \"Redis {0} is higher than {1}. Defaulting to {1}. \"\n \"If you need a higher value, please set {0} in your check config\".format(\n MAX_SLOW_ENTRIES_KEY, DEFAULT_MAX_SLOW_ENTRIES\n )\n )\n max_slow_entries = DEFAULT_MAX_SLOW_ENTRIES\n # No config on AWS Elasticache\n except redis.ResponseError:\n max_slow_entries = DEFAULT_MAX_SLOW_ENTRIES\n else:\n max_slow_entries = int(instance.get(MAX_SLOW_ENTRIES_KEY))\n\n # Generate a unique id for this instance to be persisted across runs\n ts_key = self._generate_instance_key(instance)\n\n # Get all slowlog entries\n\n slowlogs = conn.slowlog_get(max_slow_entries)\n\n # Find slowlog entries between last timestamp and now using start_time\n slowlogs = [s for s in slowlogs if s['start_time'] > self.last_timestamp_seen[ts_key]]\n\n max_ts = 0\n # Slowlog entry looks like:\n # {'command': 'LPOP somekey',\n # 'duration': 11238,\n # 'id': 496L,\n # 'start_time': 1422529869}\n for slowlog in slowlogs:\n if slowlog['start_time'] > max_ts:\n max_ts = slowlog['start_time']\n\n slowlog_tags = list(tags)\n command = slowlog['command'].split()\n # When the \"Garantia Data\" custom Redis is used, redis-py returns\n # an empty `command` field\n # FIXME when https://github.com/andymccurdy/redis-py/pull/622 is released in redis-py\n if command:\n slowlog_tags.append('command:{}'.format(ensure_unicode(command[0])))\n\n value = slowlog['duration']\n self.histogram('redis.slowlog.micros', value, tags=slowlog_tags)\n\n self.last_timestamp_seen[ts_key] = max_ts"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_command_stats(self, conn, tags):\n try:\n command_stats = conn.info(\"commandstats\")\n except Exception:\n self.warning('Could not retrieve command stats from Redis. INFO COMMANDSTATS only works with Redis >= 2.6.')\n return\n\n for key, stats in iteritems(command_stats):\n command = key.split('_', 1)[1]\n command_tags = tags + ['command:{}'.format(command)]\n\n # When `host:` is passed as a command, `calls` ends up having a leading `:`\n # see https://github.com/DataDog/integrations-core/issues/839\n calls = stats.get('calls') if command != 'host' else stats.get(':calls')\n\n self.gauge('redis.command.calls', calls, tags=command_tags)\n self.gauge('redis.command.usec_per_call', stats['usec_per_call'], tags=command_tags)", "response": "Get command - specific statistics from redis INFO COMMANDSTATS"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_generic_instances(self, instances):\n generic_instances = []\n for instance in instances:\n transformed_instance = self._create_core_dns_instance(instance)\n generic_instances.append(transformed_instance)\n\n return generic_instances", "response": "Transform each CoreDNS instance into OpenMetricsBaseCheck instance"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _create_core_dns_instance(self, instance):\n endpoint = instance.get('prometheus_url')\n if endpoint is None:\n raise ConfigurationError(\"Unable to find prometheus endpoint in config file.\")\n\n metrics = [DEFAULT_METRICS, GO_METRICS]\n metrics.extend(instance.get('metrics', []))\n\n instance.update({'prometheus_url': endpoint, 'namespace': 'coredns', 'metrics': metrics})\n\n return instance", "response": "Create a coredns instance from the config file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clean(ctx, check, compiled_only, all_matches, force, verbose):\n force_clean_root = False\n\n if check:\n path = resolve_path(os.path.join(get_root(), check))\n if not dir_exists(path):\n abort(\n 'Directory `{}` does not exist. Be sure to `ddev config set {repo} '\n 'path/to/integrations-{repo}`.'.format(path, repo=ctx.obj['repo_choice'])\n )\n else:\n path = os.getcwd()\n if basepath(path) in ('integrations-core', 'integrations-extras'):\n if force:\n force_clean_root = True\n else:\n echo_warning(\n 'You are running this from the root of the integrations project. '\n 'Should we remove everything, including: '\n '.cache, .coverage, .eggs, .pytest_cache, .tox, build, dist, and *.egg-info? '\n 'You can also use --force or -f to bypass this input.'\n )\n force_clean_root = click.confirm('Do you want to continue?')\n\n echo_waiting('Cleaning `{}`...'.format(path))\n if compiled_only:\n removed_paths = remove_compiled_scripts(path, detect_project=not all_matches)\n else:\n removed_paths = clean_package(path, detect_project=not all_matches, force_clean_root=force_clean_root)\n\n if verbose:\n if removed_paths:\n echo_success('Removed paths:')\n for p in removed_paths:\n echo_info(' {}'.format(p))\n\n if removed_paths:\n echo_success('Cleaned!')\n else:\n echo_success('Already clean!')", "response": "Removes a project s build artifacts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhitting a given URL and return the parsed json", "response": "def get(self, url, instance, service_check_tags, run_check=False):\n \"Hit a given URL and return the parsed json\"\n self.log.debug('Fetching CouchDB stats at url: %s' % url)\n\n auth = None\n if 'user' in instance and 'password' in instance:\n auth = (instance['user'], instance['password'])\n\n # Override Accept request header so that failures are not redirected to the Futon web-ui\n request_headers = headers(self.agentConfig)\n request_headers['Accept'] = 'text/json'\n\n try:\n r = requests.get(\n url, auth=auth, headers=request_headers, timeout=int(instance.get('timeout', self.TIMEOUT))\n )\n r.raise_for_status()\n if run_check:\n self.service_check(\n self.SERVICE_CHECK_NAME,\n AgentCheck.OK,\n tags=service_check_tags,\n message='Connection to %s was successful' % url,\n )\n except requests.exceptions.Timeout as e:\n self.service_check(\n self.SERVICE_CHECK_NAME,\n AgentCheck.CRITICAL,\n tags=service_check_tags,\n message=\"Request timeout: {0}, {1}\".format(url, e),\n )\n raise\n except requests.exceptions.HTTPError as e:\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=str(e))\n raise\n except Exception as e:\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=str(e))\n raise\n return r.json()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a SON document to a normal Python dictionary instance.", "response": "def to_dict(self):\n \"\"\"Convert a SON document to a normal Python dictionary instance.\n\n This is trickier than just *dict(...)* because it needs to be\n recursive.\n \"\"\"\n\n def transform_value(value):\n if isinstance(value, list):\n return [transform_value(v) for v in value]\n elif isinstance(value, collections.Mapping):\n return dict([\n (k, transform_value(v))\n for k, v in iteritems(value)])\n else:\n return value\n\n return transform_value(dict(self))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _cache_credentials(self, source, credentials, connect=False):\n # Don't let other threads affect this call's data.\n all_credentials = self.__all_credentials.copy()\n\n if source in all_credentials:\n # Nothing to do if we already have these credentials.\n if credentials == all_credentials[source]:\n return\n raise OperationFailure('Another user is already authenticated '\n 'to this database. You must logout first.')\n\n if connect:\n server = self._get_topology().select_server(\n writable_preferred_server_selector)\n\n # get_socket() logs out of the database if logged in with old\n # credentials, and logs in with new ones.\n with server.get_socket(all_credentials) as sock_info:\n sock_info.authenticate(credentials)\n\n # If several threads run _cache_credentials at once, last one wins.\n self.__all_credentials[source] = credentials", "response": "Save a set of authentication credentials."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cached(self, dbname, coll, index):\n cache = self.__index_cache\n now = datetime.datetime.utcnow()\n with self.__index_cache_lock:\n return (dbname in cache and\n coll in cache[dbname] and\n index in cache[dbname][coll] and\n now < cache[dbname][coll][index])", "response": "Test if index is cached."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\npurge an index from the index cache.", "response": "def _purge_index(self, database_name,\n collection_name=None, index_name=None):\n \"\"\"Purge an index from the index cache.\n\n If `index_name` is None purge an entire collection.\n\n If `collection_name` is None purge an entire database.\n \"\"\"\n with self.__index_cache_lock:\n if not database_name in self.__index_cache:\n return\n\n if collection_name is None:\n del self.__index_cache[database_name]\n return\n\n if not collection_name in self.__index_cache[database_name]:\n return\n\n if index_name is None:\n del self.__index_cache[database_name][collection_name]\n return\n\n if index_name in self.__index_cache[database_name][collection_name]:\n del self.__index_cache[database_name][collection_name][index_name]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef address(self):\n topology_type = self._topology._description.topology_type\n if topology_type == TOPOLOGY_TYPE.Sharded:\n raise InvalidOperation(\n 'Cannot use \"address\" property when load balancing among'\n ' mongoses, use \"nodes\" instead.')\n if topology_type not in (TOPOLOGY_TYPE.ReplicaSetWithPrimary,\n TOPOLOGY_TYPE.Single):\n return None\n return self._server_property('address')", "response": "Returns the value of the address property of the current node."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef nodes(self):\n description = self._topology.description\n return frozenset(s.address for s in description.known_servers)", "response": "Set of all currently connected servers."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _is_writable(self):\n topology = self._get_topology() # Starts monitors if necessary.\n try:\n svr = topology.select_server(writable_server_selector)\n\n # When directly connected to a secondary, arbiter, etc.,\n # select_server returns it, whatever the selector. Check\n # again if the server is writable.\n return svr.description.is_writable\n except ConnectionFailure:\n return False", "response": "Attempt to connect to a writable server and return True."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_cursor_manager(self, manager_class):\n warnings.warn(\n \"set_cursor_manager is Deprecated\",\n DeprecationWarning,\n stacklevel=2)\n manager = manager_class(self)\n if not isinstance(manager, CursorManager):\n raise TypeError(\"manager_class must be a subclass of \"\n \"CursorManager\")\n\n self.__cursor_manager = manager", "response": "Deprecated - Set this client s cursor manager."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _send_message_with_response(self, operation, read_preference=None,\n exhaust=False, address=None):\n \"\"\"Send a message to MongoDB and return a Response.\n\n :Parameters:\n - `operation`: a _Query or _GetMore object.\n - `read_preference` (optional): A ReadPreference.\n - `exhaust` (optional): If True, the socket used stays checked out.\n It is returned along with its Pool in the Response.\n - `address` (optional): Optional address when sending a message\n to a specific server, used for getMore.\n \"\"\"\n with self.__lock:\n # If needed, restart kill-cursors thread after a fork.\n self._kill_cursors_executor.open()\n\n topology = self._get_topology()\n if address:\n server = topology.select_server_by_address(address)\n if not server:\n raise AutoReconnect('server %s:%d no longer available'\n % address)\n else:\n selector = read_preference or writable_server_selector\n server = topology.select_server(selector)\n\n # A _Query's slaveOk bit is already set for queries with non-primary\n # read preference. If this is a direct connection to a mongod, override\n # and *always* set the slaveOk bit. See bullet point 2 in\n # server-selection.rst#topology-type-single.\n set_slave_ok = (\n topology.description.topology_type == TOPOLOGY_TYPE.Single\n and server.description.server_type != SERVER_TYPE.Mongos)\n\n return self._reset_on_error(\n server,\n server.send_message_with_response,\n operation,\n set_slave_ok,\n self.__all_credentials,\n self._event_listeners,\n exhaust)", "response": "Send a message to MongoDB and return a Response."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _reset_on_error(self, server, func, *args, **kwargs):\n try:\n return func(*args, **kwargs)\n except NetworkTimeout:\n # The socket has been closed. Don't reset the server.\n raise\n except ConnectionFailure:\n self.__reset_server(server.description.address)\n raise", "response": "Execute an operation on network error."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a kill cursors message soon with the given id.", "response": "def close_cursor(self, cursor_id, address=None):\n \"\"\"Send a kill cursors message soon with the given id.\n\n Raises :class:`TypeError` if `cursor_id` is not an instance of\n ``(int, long)``. What closing the cursor actually means\n depends on this client's cursor manager.\n\n This method may be called from a :class:`~pymongo.cursor.Cursor`\n destructor during garbage collection, so it isn't safe to take a\n lock or do network I/O. Instead, we schedule the cursor to be closed\n soon on a background thread.\n\n :Parameters:\n - `cursor_id`: id of cursor to close\n - `address` (optional): (host, port) pair of the cursor's server.\n If it is not provided, the client attempts to close the cursor on\n the primary or standalone, or a mongos server.\n\n .. versionchanged:: 3.0\n Added ``address`` parameter.\n \"\"\"\n if not isinstance(cursor_id, integer_types):\n raise TypeError(\"cursor_id must be an instance of (int, long)\")\n\n if self.__cursor_manager is not None:\n self.__cursor_manager.close(cursor_id, address)\n else:\n self.__kill_cursors_queue.append((address, [cursor_id]))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a kill cursors message with the given id.", "response": "def _close_cursor_now(self, cursor_id, address=None):\n \"\"\"Send a kill cursors message with the given id.\n\n What closing the cursor actually means depends on this client's\n cursor manager. If there is none, the cursor is closed synchronously\n on the current thread.\n \"\"\"\n if not isinstance(cursor_id, integer_types):\n raise TypeError(\"cursor_id must be an instance of (int, long)\")\n\n if self.__cursor_manager is not None:\n self.__cursor_manager.close(cursor_id, address)\n else:\n self._kill_cursors([cursor_id], address, self._get_topology())"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef kill_cursors(self, cursor_ids, address=None):\n warnings.warn(\n \"kill_cursors is deprecated.\",\n DeprecationWarning,\n stacklevel=2)\n\n if not isinstance(cursor_ids, list):\n raise TypeError(\"cursor_ids must be a list\")\n\n # \"Atomic\", needs no lock.\n self.__kill_cursors_queue.append((address, cursor_ids))", "response": "Sends a kill cursors message soon with the given ids."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef drop_database(self, name_or_database):\n name = name_or_database\n if isinstance(name, database.Database):\n name = name.name\n\n if not isinstance(name, string_type):\n raise TypeError(\"name_or_database must be an instance \"\n \"of %s or a Database\" % (string_type.__name__,))\n\n self._purge_index(name)\n with self._socket_for_reads(\n ReadPreference.PRIMARY) as (sock_info, slave_ok):\n self[name]._command(\n sock_info,\n \"dropDatabase\",\n slave_ok=slave_ok,\n read_preference=ReadPreference.PRIMARY,\n write_concern=self.write_concern,\n parse_write_concern_error=True)", "response": "Drop a database.\n\n Raises :class:`TypeError` if `name_or_database` is not an instance of\n :class:`basestring` (:class:`str` in python 3) or\n :class:`~pymongo.database.Database`.\n\n :Parameters:\n - `name_or_database`: the name of a database to drop, or a\n :class:`~pymongo.database.Database` instance representing the\n database to drop\n\n .. note:: The :attr:`~pymongo.mongo_client.MongoClient.write_concern` of\n this client is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this client's write concern automatically to this operation\n when connected to MongoDB >= 3.4."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndeprecate - Get the default database name.", "response": "def get_default_database(self):\n \"\"\"DEPRECATED - Get the database named in the MongoDB connection URI.\n\n >>> uri = 'mongodb://host/my_database'\n >>> client = MongoClient(uri)\n >>> db = client.get_default_database()\n >>> assert db.name == 'my_database'\n >>> db = client.get_database()\n >>> assert db.name == 'my_database'\n\n Useful in scripts where you want to choose which database to use\n based only on the URI in a configuration file.\n\n .. versionchanged:: 3.5\n Deprecated, use :meth:`get_database` instead.\n \"\"\"\n warnings.warn(\"get_default_database is deprecated. Use get_database \"\n \"instead.\", DeprecationWarning, stacklevel=2)\n if self.__default_database_name is None:\n raise ConfigurationError('No default database defined')\n\n return self[self.__default_database_name]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_database(self, name=None, codec_options=None, read_preference=None,\n write_concern=None, read_concern=None):\n \"\"\"Get a :class:`~pymongo.database.Database` with the given name and\n options.\n\n Useful for creating a :class:`~pymongo.database.Database` with\n different codec options, read preference, and/or write concern from\n this :class:`MongoClient`.\n\n >>> client.read_preference\n Primary()\n >>> db1 = client.test\n >>> db1.read_preference\n Primary()\n >>> from pymongo import ReadPreference\n >>> db2 = client.get_database(\n ... 'test', read_preference=ReadPreference.SECONDARY)\n >>> db2.read_preference\n Secondary(tag_sets=None)\n\n :Parameters:\n - `name` (optional): The name of the database - a string. If ``None``\n (the default) the database named in the MongoDB connection URI is\n returned.\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`. If ``None`` (the\n default) the :attr:`codec_options` of this :class:`MongoClient` is\n used.\n - `read_preference` (optional): The read preference to use. If\n ``None`` (the default) the :attr:`read_preference` of this\n :class:`MongoClient` is used. See :mod:`~pymongo.read_preferences`\n for options.\n - `write_concern` (optional): An instance of\n :class:`~pymongo.write_concern.WriteConcern`. If ``None`` (the\n default) the :attr:`write_concern` of this :class:`MongoClient` is\n used.\n - `read_concern` (optional): An instance of\n :class:`~pymongo.read_concern.ReadConcern`. If ``None`` (the\n default) the :attr:`read_concern` of this :class:`MongoClient` is\n used.\n\n .. versionchanged:: 3.5\n The `name` parameter is now optional, defaulting to the database\n named in the MongoDB connection URI.\n \"\"\"\n if name is None:\n if self.__default_database_name is None:\n raise ConfigurationError('No default database defined')\n name = self.__default_database_name\n\n return database.Database(\n self, name, codec_options, read_preference,\n write_concern, read_concern)", "response": "Get a database with the given name and options."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nflushes all pending writes to datafiles.", "response": "def fsync(self, **kwargs):\n \"\"\"Flush all pending writes to datafiles.\n\n :Parameters:\n\n Optional parameters can be passed as keyword arguments:\n\n - `lock`: If True lock the server to disallow writes.\n - `async`: If True don't block while synchronizing.\n\n .. warning:: `async` and `lock` can not be used together.\n\n .. warning:: MongoDB does not support the `async` option\n on Windows and will raise an exception on that\n platform.\n \"\"\"\n self.admin.command(\"fsync\",\n read_preference=ReadPreference.PRIMARY, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nunlocking a previously locked server.", "response": "def unlock(self):\n \"\"\"Unlock a previously locked server.\n \"\"\"\n cmd = {\"fsyncUnlock\": 1}\n with self._socket_for_writes() as sock_info:\n if sock_info.max_wire_version >= 4:\n try:\n sock_info.command(\"admin\", cmd)\n except OperationFailure as exc:\n # Ignore \"DB not locked\" to replicate old behavior\n if exc.code != 125:\n raise\n else:\n helpers._first_batch(sock_info, \"admin\", \"$cmd.sys.unlock\",\n {}, -1, True, self.codec_options,\n ReadPreference.PRIMARY, cmd, self._event_listeners)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check(self, instance):\n self.log.debug(\"Running instance: %s\", instance)\n custom_tags = instance.get('tags', [])\n\n # Validate the config.\n if not instance or self.PROC_NAME not in instance:\n raise GUnicornCheckError(\"instance must specify: %s\" % self.PROC_NAME)\n\n # Load the gunicorn master procedure.\n proc_name = instance.get(self.PROC_NAME)\n master_procs = self._get_master_proc_by_name(proc_name, custom_tags)\n\n # Fetch the worker procs and count their states.\n worker_procs = self._get_workers_from_procs(master_procs)\n working, idle = self._count_workers(worker_procs)\n\n # if no workers are running, alert CRITICAL, otherwise OK\n msg = \"%s working and %s idle workers for %s\" % (working, idle, proc_name)\n status = AgentCheck.CRITICAL if working == 0 and idle == 0 else AgentCheck.OK\n tags = ['app:' + proc_name] + custom_tags\n\n self.service_check(self.SVC_NAME, status, tags=tags, message=msg)\n\n # Submit the data.\n self.log.debug(\"instance %s procs - working:%s idle:%s\" % (proc_name, working, idle))\n self.gauge(\"gunicorn.workers\", working, tags + self.WORKING_TAGS)\n self.gauge(\"gunicorn.workers\", idle, tags + self.IDLE_TAGS)", "response": "Check the gunicorn master process and submit the data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_master_proc_by_name(self, name, tags):\n master_name = GUnicornCheck._get_master_proc_name(name)\n master_procs = [p for p in psutil.process_iter() if p.cmdline() and p.cmdline()[0] == master_name]\n if len(master_procs) == 0:\n # process not found, it's dead.\n self.service_check(\n self.SVC_NAME,\n AgentCheck.CRITICAL,\n tags=['app:' + name] + tags,\n message=\"No gunicorn process with name %s found\" % name,\n )\n raise GUnicornCheckError(\"Found no master process with name: %s\" % master_name)\n else:\n self.log.debug(\"There exist %s master process(es) with the name %s\" % (len(master_procs), name))\n return master_procs", "response": "Return a psutil process for the master gunicorn process with the given name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef batch_size(self, batch_size):\n if not isinstance(batch_size, integer_types):\n raise TypeError(\"batch_size must be an integer\")\n if batch_size < 0:\n raise ValueError(\"batch_size must be >= 0\")\n\n self.__batch_size = batch_size == 1 and 2 or batch_size\n return self", "response": "Limits the number of documents returned in one batch."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a getmore message and handle the response.", "response": "def __send_message(self, operation):\n \"\"\"Send a getmore message and handle the response.\n \"\"\"\n client = self.__collection.database.client\n listeners = client._event_listeners\n publish = listeners.enabled_for_commands\n try:\n response = client._send_message_with_response(\n operation, address=self.__address)\n except AutoReconnect:\n # Don't try to send kill cursors on another socket\n # or to another server. It can cause a _pinValue\n # assertion on some server releases if we get here\n # due to a socket timeout.\n self.__killed = True\n raise\n\n cmd_duration = response.duration\n rqst_id = response.request_id\n from_command = response.from_command\n\n if publish:\n start = datetime.datetime.now()\n try:\n doc = helpers._unpack_response(response.data,\n self.__id,\n self.__collection.codec_options)\n if from_command:\n helpers._check_command_response(doc['data'][0])\n\n except OperationFailure as exc:\n self.__killed = True\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, exc.details, \"getMore\", rqst_id, self.__address)\n\n raise\n except NotMasterError as exc:\n # Don't send kill cursors to another server after a \"not master\"\n # error. It's completely pointless.\n self.__killed = True\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, exc.details, \"getMore\", rqst_id, self.__address)\n\n client._reset_server_and_request_check(self.address)\n raise\n except Exception as exc:\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, _convert_exception(exc), \"getMore\", rqst_id,\n self.__address)\n raise\n\n if from_command:\n cursor = doc['data'][0]['cursor']\n documents = cursor['nextBatch']\n self.__id = cursor['id']\n self.__retrieved += len(documents)\n else:\n documents = doc[\"data\"]\n self.__id = doc[\"cursor_id\"]\n self.__retrieved += doc[\"number_returned\"]\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n # Must publish in getMore command response format.\n res = {\"cursor\": {\"id\": self.__id,\n \"ns\": self.__collection.full_name,\n \"nextBatch\": documents},\n \"ok\": 1}\n listeners.publish_command_success(\n duration, res, \"getMore\", rqst_id, self.__address)\n\n if self.__id == 0:\n self.__killed = True\n self.__data = deque(documents)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _refresh(self):\n if len(self.__data) or self.__killed:\n return len(self.__data)\n\n if self.__id: # Get More\n dbname, collname = self.__ns.split('.', 1)\n self.__send_message(\n _GetMore(dbname,\n collname,\n self.__batch_size,\n self.__id,\n self.__collection.codec_options))\n\n else: # Cursor id is zero nothing else to return\n self.__killed = True\n\n return len(self.__data)", "response": "Refreshes the cursor with more data from the server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _exclude_disk(self, device, file_system, mount_point):\n self.log.debug('_exclude_disk: {}, {}, {}'.format(device, file_system, mount_point))\n\n if not device or device == 'none':\n device = None\n\n # Allow no device if `all_partitions` is true so we can evaluate mount points\n if not self._all_partitions:\n return True\n\n # Hack for NFS secure mounts\n # Secure mounts might look like this: '/mypath (deleted)', we should\n # ignore all the bits not part of the mount point name. Take also into\n # account a space might be in the mount point.\n mount_point = mount_point.rsplit(' ', 1)[0]\n\n return self._partition_blacklisted(device, file_system, mount_point) or not self._partition_whitelisted(\n device, file_system, mount_point\n )", "response": "Return True if disks we don t want or that match regex in the config file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransforms the raw output for the df command into a normalized list devices.", "response": "def _list_devices(self, df_output):\n \"\"\"\n Given raw output for the df command, transform it into a normalized\n list devices. A 'device' is a list with fields corresponding to the\n output of df output on each platform.\n \"\"\"\n all_devices = [l.strip().split() for l in df_output.splitlines()]\n\n # Skip the header row and empty lines.\n raw_devices = [l for l in all_devices[1:] if l]\n\n # Flatten the disks that appear in the mulitple lines.\n flattened_devices = self._flatten_devices(raw_devices)\n\n # Filter fake or unwanteddisks.\n return [d for d in flattened_devices if self._keep_device(d)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _compile_tag_re(self):\n device_tag_list = []\n for regex_str, tags in iteritems(self._device_tag_re):\n try:\n device_tag_list.append([re.compile(regex_str, IGNORE_CASE), [t.strip() for t in tags.split(',')]])\n except TypeError:\n self.log.warning('{} is not a valid regular expression and will be ignored'.format(regex_str))\n self._device_tag_re = device_tag_list", "response": "Compile regex strings from device_tag_re option and return list of compiled regex / tag pairs"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfill the dict with a given key.", "response": "def fill(self, key, mor_dict):\n \"\"\"\n Set a dict mapping (resouce_type --> objects[]) for a given key\n \"\"\"\n with self._objects_queue_lock:\n self._objects_queue[key] = mor_dict"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef size(self, key, resource_type):\n with self._objects_queue_lock:\n return len(self._objects_queue[key].get(resource_type, []))", "response": "Return the size of the queue for a given key and resource type."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove an object from the list of objects for a given resource type from the cache.", "response": "def pop(self, key, resource_type):\n \"\"\"\n Extract an object from the list.\n If the key is not in the cache, this will raise a KeyError.\n If the list is empty, method will return None\n \"\"\"\n with self._objects_queue_lock:\n objects = self._objects_queue[key].get(resource_type, [])\n return objects.pop() if objects else None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexacts match of IP addresses.", "response": "def _ipaddress_match(ipname, host_ip):\n \"\"\"Exact matching of IP addresses.\n\n RFC 6125 explicitly doesn't define an algorithm for this\n (section 1.7.2 - \"Out of Scope\").\n \"\"\"\n # OpenSSL may add a trailing newline to a subjectAltName's IP address\n ip = ip_address(_unicode(ipname).rstrip())\n return ip == host_ip"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a TLS object to establish a secure connection to a server.", "response": "def _get_tls_object(self, ssl_params):\n \"\"\"\n Return a TLS object to establish a secure connection to a server\n \"\"\"\n if ssl_params is None:\n return None\n\n if not ssl_params[\"verify\"] and ssl_params[\"ca_certs\"]:\n self.warning(\n \"Incorrect configuration: trying to disable server certificate validation, \"\n \"while also specifying a capath. No validation will be performed. Fix your \"\n \"configuration to remove this warning\"\n )\n\n validate = ssl.CERT_REQUIRED if ssl_params[\"verify\"] else ssl.CERT_NONE\n\n if ssl_params[\"ca_certs\"] is None or os.path.isfile(ssl_params[\"ca_certs\"]):\n tls = ldap3.core.tls.Tls(\n local_private_key_file=ssl_params[\"key\"],\n local_certificate_file=ssl_params[\"cert\"],\n ca_certs_file=ssl_params[\"ca_certs\"],\n version=ssl.PROTOCOL_SSLv23,\n validate=validate,\n )\n elif os.path.isdir(ssl_params[\"ca_certs\"]):\n tls = ldap3.core.tls.Tls(\n local_private_key_file=ssl_params[\"key\"],\n local_certificate_file=ssl_params[\"cert\"],\n ca_certs_path=ssl_params[\"ca_certs\"],\n version=ssl.PROTOCOL_SSLv23,\n validate=validate,\n )\n else:\n raise ConfigurationError(\n 'Invalid path {} for ssl_ca_certs: no such file or directory'.format(ssl_params['ca_certs'])\n )\n return tls"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses instance configuration and perform minimal verification", "response": "def _get_instance_params(cls, instance):\n \"\"\"\n Parse instance configuration and perform minimal verification\n \"\"\"\n url = instance.get(\"url\")\n if url is None:\n raise ConfigurationError(\"You must specify a url for your instance in `conf.yaml`\")\n username = instance.get(\"username\")\n password = instance.get(\"password\")\n ssl_params = None\n if url.startswith(\"ldaps\"):\n ssl_params = {\n \"key\": instance.get(\"ssl_key\"),\n \"cert\": instance.get(\"ssl_cert\"),\n \"ca_certs\": instance.get(\"ssl_ca_certs\"),\n \"verify\": is_affirmative(instance.get(\"ssl_verify\", True)),\n }\n custom_queries = instance.get(\"custom_queries\", [])\n tags = list(instance.get(\"tags\", []))\n tags.append(\"url:{}\".format(url))\n\n return url, username, password, ssl_params, custom_queries, tags"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _collect_monitor_metrics(self, conn, tags):\n for entry in conn.entries:\n # Get metrics from monitor backend\n dn = entry.entry_dn.lower()\n if dn.endswith(self.CONNECTIONS_METRICS_DN):\n self._handle_connections_entry(entry, tags)\n elif dn.endswith(self.OPERATIONS_METRICS_DN):\n self._handle_operations_entry(entry, tags)\n elif dn.endswith(self.STATISTICS_METRICS_DN):\n self._handle_statistics_entry(entry, tags)\n elif dn.endswith(self.THREADS_METRICS_DN):\n self._handle_threads_entry(entry, tags)\n elif dn.endswith(self.TIME_METRICS_DN):\n self._handle_time_entry(entry, tags)\n elif dn.endswith(self.WAITERS_METRICS_DN):\n self._handle_waiters_entry(entry, tags)", "response": "Collect metrics from the monitor backend and add them to the list of tags."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _perform_custom_queries(self, conn, custom_queries, tags, instance):\n for query in custom_queries:\n name = query.get(\"name\")\n if name is None:\n self.log.error(\"`name` field is required for custom query\")\n continue\n search_base = query.get(\"search_base\")\n if search_base is None:\n self.log.error(\"`search_base` field is required for custom query #%s\", name)\n continue\n search_filter = query.get(\"search_filter\")\n if search_filter is None:\n self.log.error(\"`search_filter` field is required for custom query #%s\", name)\n continue\n attrs = query.get(\"attributes\")\n if \"username\" in query:\n username = query.get(\"username\")\n password = query.get(\"password\")\n if not username:\n # username is an empty string, we want anonymous bind\n username = None\n password = None\n else:\n # username not specified, we want to reuse the credentials for the monitor backend\n username = instance.get(\"username\")\n password = instance.get(\"password\")\n\n try:\n # Rebind with different credentials\n auth_method = ldap3.SIMPLE if username else ldap3.ANONYMOUS\n if username is None:\n conn.user = None\n res = conn.rebind(user=username, password=password, authentication=auth_method)\n if not res:\n raise ldap3.core.exceptions.LDAPBindError(\"Error binding to server: {}\".format(conn.result))\n except ldap3.core.exceptions.LDAPBindError:\n self.log.exception(\"Could not rebind to server at %s to perform query %s\", instance.get(\"url\"), name)\n continue\n\n try:\n # Perform the search query\n conn.search(search_base, search_filter, attributes=attrs)\n except ldap3.core.exceptions.LDAPException:\n self.log.exception(\"Unable to perform search query for %s\", name)\n continue\n\n query_tags = ['query:{}'.format(name)]\n query_tags.extend(tags)\n query_time = self._get_query_time(conn)\n results = len(conn.entries)\n self.gauge(\"{}.query.duration\".format(self.METRIC_PREFIX), query_time, tags=query_tags)\n self.gauge(\"{}.query.entries\".format(self.METRIC_PREFIX), results, tags=query_tags)", "response": "Perform custom queries on the server and collect additional metrics like number of result and duration of the query"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nextract first common name from DN that looks like cn = max file descriptors cn = connections cn = monitor", "response": "def _extract_common_name(cls, dn):\n \"\"\"\n extract first common name (cn) from DN that looks like \"cn=max file descriptors,cn=connections,cn=monitor\"\n \"\"\"\n dn = dn.lower().replace(\" \", \"_\")\n return dn.split(\",\")[0].split(\"=\")[1]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _process_cluster_health_data(self, node_name, node_stats, tags):\n\n # Tags for service check\n cluster_health_tags = list(tags) + ['node:{}'.format(node_name)]\n\n # Get the membership status of the node\n cluster_membership = node_stats.get('clusterMembership', None)\n membership_status = self.NODE_MEMBERSHIP_TRANSLATION.get(cluster_membership, AgentCheck.UNKNOWN)\n self.service_check(self.NODE_CLUSTER_SERVICE_CHECK_NAME, membership_status, tags=cluster_health_tags)\n\n # Get the health status of the node\n health = node_stats.get('status', None)\n health_status = self.NODE_HEALTH_TRANSLATION.get(health, AgentCheck.UNKNOWN)\n self.service_check(self.NODE_HEALTH_SERVICE_CHECK_NAME, health_status, tags=cluster_health_tags)", "response": "Process and send cluster health data"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate an event object from the message.", "response": "def _create_event(self, alert_type, msg_title, msg, server, tags=None):\n \"\"\"\n Create an event object\n \"\"\"\n msg_title = 'Couchbase {}: {}'.format(server, msg_title)\n msg = 'Couchbase instance {} {}'.format(server, msg)\n\n return {\n 'timestamp': int(time.time()),\n 'event_type': 'couchbase_rebalance',\n 'msg_text': msg,\n 'msg_title': msg_title,\n 'alert_type': alert_type,\n 'source_type_name': self.SOURCE_TYPE_NAME,\n 'aggregation_key': server,\n 'tags': tags,\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the parsed json from a given URL and return the parsed json.", "response": "def _get_stats(self, url, instance):\n \"\"\"\n Hit a given URL and return the parsed json.\n \"\"\"\n\n self.log.debug('Fetching Couchbase stats at url: {}'.format(url))\n\n ssl_verify = instance.get('ssl_verify', True)\n timeout = float(instance.get('timeout', DEFAULT_TIMEOUT))\n\n auth = None\n if 'user' in instance and 'password' in instance:\n auth = (instance['user'], instance['password'])\n\n r = requests.get(url, auth=auth, verify=ssl_verify, headers=headers(self.agentConfig), timeout=timeout)\n r.raise_for_status()\n return r.json()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the in - toto layout pubkeys of all the in - toto modules.", "response": "def __update_in_toto_layout_pubkeys(self):\n '''\n NOTE: We assume that all the public keys needed to verify any in-toto\n root layout, or sublayout, metadata file has been directly signed by\n the top-level TUF targets role using *OFFLINE* keys. This is a\n reasonable assumption, as TUF does not offer meaningful security\n guarantees if _ALL_ targets were signed using _online_ keys.\n '''\n\n target_relpaths = []\n targets = self.__updater.targets_of_role('targets')\n\n for target in targets:\n target_relpath = target['filepath']\n\n # Download this target only if it _looks_ like a public key.\n if target_relpath.endswith('.pub'):\n # NOTE: Avoid recursively downloading in-toto metadata for\n # in-toto root layout pubkeys themselves, and so on ad\n # infinitum.\n self.__get_target(target_relpath, download_in_toto_metadata=False)\n target_relpaths.append(target_relpath)\n\n return target_relpaths"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef download(self, target_relpath, download_in_toto_metadata=True):\n '''\n Returns:\n If download over TUF and in-toto is successful, this function will\n return the complete filepath to the desired target.\n '''\n return self.__get_target(target_relpath, download_in_toto_metadata=download_in_toto_metadata)", "response": "Downloads the file from the TUF to the target."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks the WMI metrics for a specific instance.", "response": "def check(self, instance):\n \"\"\"\n Fetch WMI metrics.\n \"\"\"\n # Connection information\n host = instance.get('host', \"localhost\")\n namespace = instance.get('namespace', \"root\\\\cimv2\")\n provider = instance.get('provider')\n username = instance.get('username', \"\")\n password = instance.get('password', \"\")\n\n # WMI instance\n wmi_class = instance.get('class')\n metrics = instance.get('metrics')\n filters = instance.get('filters')\n tag_by = instance.get('tag_by', \"\")\n tag_queries = instance.get('tag_queries', [])\n\n constant_tags = instance.get('constant_tags')\n custom_tags = instance.get('tags', [])\n if constant_tags is None:\n constant_tags = list(custom_tags)\n else:\n constant_tags.extend(custom_tags)\n self.log.warning(\"`constant_tags` is being deprecated, please use `tags`\")\n\n # Create or retrieve an existing WMISampler\n instance_hash = hash_mutable(instance)\n instance_key = self._get_instance_key(host, namespace, wmi_class, instance_hash)\n\n metric_name_and_type_by_property, properties = self._get_wmi_properties(instance_key, metrics, tag_queries)\n\n wmi_sampler = self._get_wmi_sampler(\n instance_key,\n wmi_class,\n properties,\n tag_by=tag_by,\n filters=filters,\n host=host,\n namespace=namespace,\n provider=provider,\n username=username,\n password=password,\n )\n\n # Sample, extract & submit metrics\n try:\n wmi_sampler.sample()\n metrics = self._extract_metrics(wmi_sampler, tag_by, tag_queries, constant_tags)\n except TimeoutException:\n self.log.warning(\n u\"WMI query timed out.\"\n u\" class={wmi_class} - properties={wmi_properties} -\"\n u\" filters={filters} - tag_queries={tag_queries}\".format(\n wmi_class=wmi_class, wmi_properties=properties, filters=filters, tag_queries=tag_queries\n )\n )\n else:\n self._submit_metrics(metrics, metric_name_and_type_by_property)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves all configuration for environments.", "response": "def prune(force):\n \"\"\"Remove all configuration for environments.\"\"\"\n if not force:\n echo_warning(\n 'Removing configuration of environments that may be in use will leave '\n 'them in a potentially unusable state. If you wish to proceed (e.g. you '\n 'have just restarted your machine), you may use the -f / --force flag.'\n )\n abort(code=2)\n\n checks = get_configured_checks()\n for check in checks:\n envs = get_configured_envs(check)\n\n if envs:\n echo_info('{}:'.format(check))\n for env in envs:\n echo_waiting('Removing `{}`... '.format(env), nl=False, indent=True)\n remove_env_data(check, env)\n echo_success('success!')\n\n remove_env_root(check)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a copy of the instance and set default values.", "response": "def _create_kubelet_prometheus_instance(self, instance):\n \"\"\"\n Create a copy of the instance and set default values.\n This is so the base class can create a scraper_config with the proper values.\n \"\"\"\n kubelet_instance = deepcopy(instance)\n kubelet_instance.update(\n {\n 'namespace': self.NAMESPACE,\n # We need to specify a prometheus_url so the base class can use it as the key for our config_map,\n # we specify a dummy url that will be replaced in the `check()` function. We append it with \"kubelet\"\n # so the key is different than the cadvisor scraper.\n 'prometheus_url': instance.get('kubelet_metrics_endpoint', 'dummy_url/kubelet'),\n 'metrics': [\n {\n 'apiserver_client_certificate_expiration_seconds': 'apiserver.certificate.expiration',\n 'rest_client_requests_total': 'rest.client.requests',\n 'rest_client_request_latency_seconds': 'rest.client.latency',\n 'kubelet_runtime_operations': 'kubelet.runtime.operations',\n 'kubelet_runtime_operations_errors': 'kubelet.runtime.errors',\n 'kubelet_network_plugin_operations_latency_microseconds': 'kubelet.network_plugin.latency',\n 'kubelet_volume_stats_available_bytes': 'kubelet.volume.stats.available_bytes',\n 'kubelet_volume_stats_capacity_bytes': 'kubelet.volume.stats.capacity_bytes',\n 'kubelet_volume_stats_used_bytes': 'kubelet.volume.stats.used_bytes',\n 'kubelet_volume_stats_inodes': 'kubelet.volume.stats.inodes',\n 'kubelet_volume_stats_inodes_free': 'kubelet.volume.stats.inodes_free',\n 'kubelet_volume_stats_inodes_used': 'kubelet.volume.stats.inodes_used',\n }\n ],\n # Defaults that were set when the Kubelet scraper was based on PrometheusScraper\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n return kubelet_instance"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nperforming and return a GET request against kubelet.", "response": "def perform_kubelet_query(self, url, verbose=True, timeout=10, stream=False):\n \"\"\"\n Perform and return a GET request against kubelet. Support auth and TLS validation.\n \"\"\"\n return requests.get(\n url,\n timeout=timeout,\n verify=self.kubelet_credentials.verify(),\n cert=self.kubelet_credentials.cert_pair(),\n headers=self.kubelet_credentials.headers(url),\n params={'verbose': verbose},\n stream=stream,\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the expiration datetime for the kubernetes pod.", "response": "def _compute_pod_expiration_datetime():\n \"\"\"\n Looks up the agent's kubernetes_pod_expiration_duration option and returns either:\n - None if expiration is disabled (set to 0)\n - A (timezone aware) datetime object to compare against\n \"\"\"\n try:\n seconds = int(get_config(\"kubernetes_pod_expiration_duration\"))\n if seconds == 0: # Expiration disabled\n return None\n return datetime.utcnow().replace(tzinfo=UTC) - timedelta(seconds=seconds)\n except (ValueError, TypeError):\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreports the number of running pods and containers in pods and the running containers in containers tagged by service and creator.", "response": "def _report_pods_running(self, pods, instance_tags):\n \"\"\"\n Reports the number of running pods on this node and the running\n containers in pods, tagged by service and creator.\n\n :param pods: pod list object\n :param instance_tags: list of tags\n \"\"\"\n pods_tag_counter = defaultdict(int)\n containers_tag_counter = defaultdict(int)\n for pod in pods['items']:\n # Containers reporting\n containers = pod.get('status', {}).get('containerStatuses', [])\n has_container_running = False\n for container in containers:\n container_id = container.get('containerID')\n if not container_id:\n self.log.debug('skipping container with no id')\n continue\n if \"running\" not in container.get('state', {}):\n continue\n has_container_running = True\n tags = tagger.tag(container_id, tagger.LOW) or None\n if not tags:\n continue\n tags += instance_tags\n hash_tags = tuple(sorted(tags))\n containers_tag_counter[hash_tags] += 1\n # Pod reporting\n if not has_container_running:\n continue\n pod_id = pod.get('metadata', {}).get('uid')\n if not pod_id:\n self.log.debug('skipping pod with no uid')\n continue\n tags = tagger.tag('kubernetes_pod://%s' % pod_id, tagger.LOW) or None\n if not tags:\n continue\n tags += instance_tags\n hash_tags = tuple(sorted(tags))\n pods_tag_counter[hash_tags] += 1\n for tags, count in iteritems(pods_tag_counter):\n self.gauge(self.NAMESPACE + '.pods.running', count, list(tags))\n for tags, count in iteritems(containers_tag_counter):\n self.gauge(self.NAMESPACE + '.containers.running', count, list(tags))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreporting the requests and limits of the containers in the pod list.", "response": "def _report_container_spec_metrics(self, pod_list, instance_tags):\n \"\"\"Reports pod requests & limits by looking at pod specs.\"\"\"\n for pod in pod_list['items']:\n pod_name = pod.get('metadata', {}).get('name')\n pod_phase = pod.get('status', {}).get('phase')\n if self._should_ignore_pod(pod_name, pod_phase):\n continue\n\n for ctr in pod['spec']['containers']:\n if not ctr.get('resources'):\n continue\n\n c_name = ctr.get('name', '')\n cid = None\n for ctr_status in pod['status'].get('containerStatuses', []):\n if ctr_status.get('name') == c_name:\n # it is already prefixed with 'runtime://'\n cid = ctr_status.get('containerID')\n break\n if not cid:\n continue\n\n pod_uid = pod.get('metadata', {}).get('uid')\n if self.pod_list_utils.is_excluded(cid, pod_uid):\n continue\n\n tags = tagger.tag('%s' % cid, tagger.HIGH) + instance_tags\n\n try:\n for resource, value_str in iteritems(ctr.get('resources', {}).get('requests', {})):\n value = self.parse_quantity(value_str)\n self.gauge('{}.{}.requests'.format(self.NAMESPACE, resource), value, tags)\n except (KeyError, AttributeError) as e:\n self.log.debug(\"Unable to retrieve container requests for %s: %s\", c_name, e)\n\n try:\n for resource, value_str in iteritems(ctr.get('resources', {}).get('limits', {})):\n value = self.parse_quantity(value_str)\n self.gauge('{}.{}.limits'.format(self.NAMESPACE, resource), value, tags)\n except (KeyError, AttributeError) as e:\n self.log.debug(\"Unable to retrieve container limits for %s: %s\", c_name, e)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _report_container_state_metrics(self, pod_list, instance_tags):\n if pod_list.get('expired_count'):\n self.gauge(self.NAMESPACE + '.pods.expired', pod_list.get('expired_count'), tags=instance_tags)\n\n for pod in pod_list['items']:\n pod_name = pod.get('metadata', {}).get('name')\n pod_uid = pod.get('metadata', {}).get('uid')\n\n if not pod_name or not pod_uid:\n continue\n\n for ctr_status in pod['status'].get('containerStatuses', []):\n c_name = ctr_status.get('name')\n cid = ctr_status.get('containerID')\n if not c_name or not cid:\n continue\n\n if self.pod_list_utils.is_excluded(cid, pod_uid):\n continue\n\n tags = tagger.tag('%s' % cid, tagger.ORCHESTRATOR) + instance_tags\n\n restart_count = ctr_status.get('restartCount', 0)\n self.gauge(self.NAMESPACE + '.containers.restarts', restart_count, tags)\n\n for (metric_name, field_name) in [('state', 'state'), ('last_state', 'lastState')]:\n c_state = ctr_status.get(field_name, {})\n\n for state_name in ['terminated', 'waiting']:\n state_reasons = WHITELISTED_CONTAINER_STATE_REASONS.get(state_name, [])\n self._submit_container_state_metric(metric_name, state_name, c_state, state_reasons, tags)", "response": "Reports container state & reasons by looking at container statuses"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the quantity of the resource in the resources spec.", "response": "def parse_quantity(string):\n \"\"\"\n Parse quantity allows to convert the value in the resources spec like:\n resources:\n requests:\n cpu: \"100m\"\n memory\": \"200Mi\"\n limits:\n memory: \"300Mi\"\n :param string: str\n :return: float\n \"\"\"\n number, unit = '', ''\n for char in string:\n if char.isdigit() or char == '.':\n number += char\n else:\n unit += char\n return float(number) * FACTORS.get(unit, 1)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a socket. error to ConnectionFailure and raise it.", "response": "def _raise_connection_failure(address, error):\n \"\"\"Convert a socket.error to ConnectionFailure and raise it.\"\"\"\n host, port = address\n # If connecting to a Unix socket, port will be None.\n if port is not None:\n msg = '%s:%d: %s' % (host, port, error)\n else:\n msg = '%s: %s' % (host, error)\n if isinstance(error, socket.timeout):\n raise NetworkTimeout(msg)\n elif isinstance(error, SSLError) and 'timed out' in str(error):\n # CPython 2.6, 2.7, PyPy 2.x, and PyPy3 do not distinguish network\n # timeouts from other SSLErrors (https://bugs.python.org/issue10272).\n # Luckily, we can work around this limitation because the phrase\n # 'timed out' appears in all the timeout related SSLErrors raised\n # on the above platforms. CPython >= 3.2 and PyPy3.3 correctly raise\n # socket.timeout.\n raise NetworkTimeout(msg)\n else:\n raise AutoReconnect(msg)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconnecting and return a socket object.", "response": "def _create_connection(address, options):\n \"\"\"Given (host, port) and PoolOptions, connect and return a socket object.\n\n Can raise socket.error.\n\n This is a modified version of create_connection from CPython >= 2.6.\n \"\"\"\n host, port = address\n\n # Check if dealing with a unix domain socket\n if host.endswith('.sock'):\n if not hasattr(socket, \"AF_UNIX\"):\n raise ConnectionFailure(\"UNIX-sockets are not supported \"\n \"on this system\")\n sock = socket.socket(socket.AF_UNIX)\n # SOCK_CLOEXEC not supported for Unix sockets.\n _set_non_inheritable_non_atomic(sock.fileno())\n try:\n sock.connect(host)\n return sock\n except socket.error:\n sock.close()\n raise\n\n # Don't try IPv6 if we don't support it. Also skip it if host\n # is 'localhost' (::1 is fine). Avoids slow connect issues\n # like PYTHON-356.\n family = socket.AF_INET\n if socket.has_ipv6 and host != 'localhost':\n family = socket.AF_UNSPEC\n\n err = None\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n af, socktype, proto, dummy, sa = res\n # SOCK_CLOEXEC was new in CPython 3.2, and only available on a limited\n # number of platforms (newer Linux and *BSD). Starting with CPython 3.4\n # all file descriptors are created non-inheritable. See PEP 446.\n try:\n sock = socket.socket(\n af, socktype | getattr(socket, 'SOCK_CLOEXEC', 0), proto)\n except socket.error:\n # Can SOCK_CLOEXEC be defined even if the kernel doesn't support\n # it?\n sock = socket.socket(af, socktype, proto)\n # Fallback when SOCK_CLOEXEC isn't available.\n _set_non_inheritable_non_atomic(sock.fileno())\n try:\n sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)\n sock.settimeout(options.connect_timeout)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE,\n options.socket_keepalive)\n if options.socket_keepalive:\n _set_keepalive_times(sock)\n sock.connect(sa)\n return sock\n except socket.error as e:\n err = e\n sock.close()\n\n if err is not None:\n raise err\n else:\n # This likely means we tried to connect to an IPv6 only\n # host with an OS/kernel or Python interpreter that doesn't\n # support IPv6. The test case is Jython2.5.1 which doesn't\n # support IPv6 at all.\n raise socket.error('getaddrinfo failed')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a host and PoolOptions return a configured socket.", "response": "def _configured_socket(address, options):\n \"\"\"Given (host, port) and PoolOptions, return a configured socket.\n\n Can raise socket.error, ConnectionFailure, or CertificateError.\n\n Sets socket's SSL and timeout options.\n \"\"\"\n sock = _create_connection(address, options)\n ssl_context = options.ssl_context\n\n if ssl_context is not None:\n host = address[0]\n try:\n # According to RFC6066, section 3, IPv4 and IPv6 literals are\n # not permitted for SNI hostname.\n if _HAVE_SNI and not is_ip_address(host):\n sock = ssl_context.wrap_socket(sock, server_hostname=host)\n else:\n sock = ssl_context.wrap_socket(sock)\n except IOError as exc:\n sock.close()\n raise ConnectionFailure(\"SSL handshake failed: %s\" % (str(exc),))\n if ssl_context.verify_mode and options.ssl_match_hostname:\n try:\n match_hostname(sock.getpeercert(), hostname=host)\n except CertificateError:\n sock.close()\n raise\n\n sock.settimeout(options.socket_timeout)\n return sock"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef command(self, dbname, spec, slave_ok=False,\n read_preference=ReadPreference.PRIMARY,\n codec_options=DEFAULT_CODEC_OPTIONS, check=True,\n allowable_errors=None, check_keys=False,\n read_concern=DEFAULT_READ_CONCERN,\n write_concern=None,\n parse_write_concern_error=False,\n collation=None):\n \"\"\"Execute a command or raise ConnectionFailure or OperationFailure.\n\n :Parameters:\n - `dbname`: name of the database on which to run the command\n - `spec`: a command document as a dict, SON, or mapping object\n - `slave_ok`: whether to set the SlaveOkay wire protocol bit\n - `read_preference`: a read preference\n - `codec_options`: a CodecOptions instance\n - `check`: raise OperationFailure if there are errors\n - `allowable_errors`: errors to ignore if `check` is True\n - `check_keys`: if True, check `spec` for invalid keys\n - `read_concern`: The read concern for this command.\n - `write_concern`: The write concern for this command.\n - `parse_write_concern_error`: Whether to parse the\n ``writeConcernError`` field in the command response.\n - `collation`: The collation for this command.\n \"\"\"\n if self.max_wire_version < 4 and not read_concern.ok_for_legacy:\n raise ConfigurationError(\n 'read concern level of %s is not valid '\n 'with a max wire version of %d.'\n % (read_concern.level, self.max_wire_version))\n if not (write_concern is None or write_concern.acknowledged or\n collation is None):\n raise ConfigurationError(\n 'Collation is unsupported for unacknowledged writes.')\n if self.max_wire_version >= 5 and write_concern:\n spec['writeConcern'] = write_concern.document\n elif self.max_wire_version < 5 and collation is not None:\n raise ConfigurationError(\n 'Must be connected to MongoDB 3.4+ to use a collation.')\n try:\n return command(self.sock, dbname, spec, slave_ok,\n self.is_mongos, read_preference, codec_options,\n check, allowable_errors, self.address,\n check_keys, self.listeners, self.max_bson_size,\n read_concern,\n parse_write_concern_error=parse_write_concern_error,\n collation=collation)\n except OperationFailure:\n raise\n # Catch socket.error, KeyboardInterrupt, etc. and close ourselves.\n except BaseException as error:\n self._raise_connection_failure(error)", "response": "Execute a command on the specified database."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a BSON message or raise ConnectionFailure.", "response": "def send_message(self, message, max_doc_size):\n \"\"\"Send a raw BSON message or raise ConnectionFailure.\n\n If a network exception is raised, the socket is closed.\n \"\"\"\n if (self.max_bson_size is not None\n and max_doc_size > self.max_bson_size):\n raise DocumentTooLarge(\n \"BSON document too large (%d bytes) - the connected server \"\n \"supports BSON document sizes up to %d bytes.\" %\n (max_doc_size, self.max_bson_size))\n\n try:\n self.sock.sendall(message)\n except BaseException as error:\n self._raise_connection_failure(error)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef receive_message(self, operation, request_id):\n try:\n return receive_message(\n self.sock, operation, request_id, self.max_message_size)\n except BaseException as error:\n self._raise_connection_failure(error)", "response": "Receive a BSON message or raise ConnectionFailure."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef legacy_write(self, request_id, msg, max_doc_size, with_last_error):\n if not with_last_error and not self.is_writable:\n # Write won't succeed, bail as if we'd done a getlasterror.\n raise NotMasterError(\"not master\")\n\n self.send_message(msg, max_doc_size)\n if with_last_error:\n response = self.receive_message(1, request_id)\n return helpers._check_gle_response(response)", "response": "Send a message to the server and return a dict."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends an insert etc. command to the server and return the response as a dict.", "response": "def write_command(self, request_id, msg):\n \"\"\"Send \"insert\" etc. command, returning response as a dict.\n\n Can raise ConnectionFailure or OperationFailure.\n\n :Parameters:\n - `request_id`: an int.\n - `msg`: bytes, the command message.\n \"\"\"\n self.send_message(msg, 0)\n response = helpers._unpack_response(self.receive_message(1, request_id))\n assert response['number_returned'] == 1\n result = response['data'][0]\n\n # Raises NotMasterError or OperationFailure.\n helpers._check_command_response(result)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_auth(self, all_credentials):\n if all_credentials or self.authset:\n cached = set(itervalues(all_credentials))\n authset = self.authset.copy()\n\n # Logout any credentials that no longer exist in the cache.\n for credentials in authset - cached:\n auth.logout(credentials.source, self)\n self.authset.discard(credentials)\n\n for credentials in cached - authset:\n auth.authenticate(credentials, self)\n self.authset.add(credentials)", "response": "Update this socket s authentication."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef authenticate(self, credentials):\n auth.authenticate(credentials, self)\n self.authset.add(credentials)", "response": "Log in to the server and store these credentials in authset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconnecting to the Mongo server and return a new SocketInfo object.", "response": "def connect(self):\n \"\"\"Connect to Mongo and return a new SocketInfo.\n\n Can raise ConnectionFailure or CertificateError.\n\n Note that the pool does not keep a reference to the socket -- you\n must call return_socket() when you're done with it.\n \"\"\"\n sock = None\n try:\n sock = _configured_socket(self.address, self.opts)\n if self.handshake:\n cmd = SON([\n ('ismaster', 1),\n ('client', self.opts.metadata)\n ])\n ismaster = IsMaster(\n command(sock,\n 'admin',\n cmd,\n False,\n False,\n ReadPreference.PRIMARY,\n DEFAULT_CODEC_OPTIONS))\n else:\n ismaster = None\n return SocketInfo(sock, self, ismaster, self.address)\n except socket.error as error:\n if sock is not None:\n sock.close()\n _raise_connection_failure(self.address, error)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a socket from the pool.", "response": "def get_socket(self, all_credentials, checkout=False):\n \"\"\"Get a socket from the pool. Use with a \"with\" statement.\n\n Returns a :class:`SocketInfo` object wrapping a connected\n :class:`socket.socket`.\n\n This method should always be used in a with-statement::\n\n with pool.get_socket(credentials, checkout) as socket_info:\n socket_info.send_message(msg)\n data = socket_info.receive_message(op_code, request_id)\n\n The socket is logged in or out as needed to match ``all_credentials``\n using the correct authentication mechanism for the server's wire\n protocol version.\n\n Can raise ConnectionFailure or OperationFailure.\n\n :Parameters:\n - `all_credentials`: dict, maps auth source to MongoCredential.\n - `checkout` (optional): keep socket checked out.\n \"\"\"\n # First get a socket, then attempt authentication. Simplifies\n # semaphore management in the face of network errors during auth.\n sock_info = self._get_socket_no_auth()\n try:\n sock_info.check_auth(all_credentials)\n yield sock_info\n except:\n # Exception in caller. Decrement semaphore.\n self.return_socket(sock_info)\n raise\n else:\n if not checkout:\n self.return_socket(sock_info)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget or create a SocketInfo. Can raise ConnectionFailure or CertificateError.", "response": "def _get_socket_no_auth(self):\n \"\"\"Get or create a SocketInfo. Can raise ConnectionFailure.\"\"\"\n # We use the pid here to avoid issues with fork / multiprocessing.\n # See test.test_client:TestClient.test_fork for an example of\n # what could go wrong otherwise\n if self.pid != os.getpid():\n self.reset()\n\n # Get a free socket or create one.\n if not self._socket_semaphore.acquire(\n True, self.opts.wait_queue_timeout):\n self._raise_wait_queue_timeout()\n with self.lock:\n self.active_sockets += 1\n\n # We've now acquired the semaphore and must release it on error.\n try:\n try:\n # set.pop() isn't atomic in Jython less than 2.7, see\n # http://bugs.jython.org/issue1854\n with self.lock:\n # Can raise ConnectionFailure.\n sock_info = self.sockets.pop()\n except KeyError:\n # Can raise ConnectionFailure or CertificateError.\n sock_info = self.connect()\n else:\n # Can raise ConnectionFailure.\n sock_info = self._check(sock_info)\n except:\n self._socket_semaphore.release()\n with self.lock:\n self.active_sockets -= 1\n raise\n\n return sock_info"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef return_socket(self, sock_info):\n if self.pid != os.getpid():\n self.reset()\n else:\n if sock_info.pool_id != self.pool_id:\n sock_info.close()\n elif not sock_info.closed:\n sock_info.last_checkin = _time()\n with self.lock:\n self.sockets.add(sock_info)\n\n self._socket_semaphore.release()\n with self.lock:\n self.active_sockets -= 1", "response": "Return the socket to the pool or discard it."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new TopologyDescription that is updated with the current topology description and the server description.", "response": "def updated_topology_description(topology_description, server_description):\n \"\"\"Return an updated copy of a TopologyDescription.\n\n :Parameters:\n - `topology_description`: the current TopologyDescription\n - `server_description`: a new ServerDescription that resulted from\n an ismaster call\n\n Called after attempting (successfully or not) to call ismaster on the\n server at server_description.address. Does not modify topology_description.\n \"\"\"\n address = server_description.address\n\n # These values will be updated, if necessary, to form the new\n # TopologyDescription.\n topology_type = topology_description.topology_type\n set_name = topology_description.replica_set_name\n max_set_version = topology_description.max_set_version\n max_election_id = topology_description.max_election_id\n server_type = server_description.server_type\n\n # Don't mutate the original dict of server descriptions; copy it.\n sds = topology_description.server_descriptions()\n\n # Replace this server's description with the new one.\n sds[address] = server_description\n\n if topology_type == TOPOLOGY_TYPE.Single:\n # Single type never changes.\n return TopologyDescription(\n TOPOLOGY_TYPE.Single,\n sds,\n set_name,\n max_set_version,\n max_election_id,\n topology_description._topology_settings)\n\n if topology_type == TOPOLOGY_TYPE.Unknown:\n if server_type == SERVER_TYPE.Standalone:\n sds.pop(address)\n\n elif server_type not in (SERVER_TYPE.Unknown, SERVER_TYPE.RSGhost):\n topology_type = _SERVER_TYPE_TO_TOPOLOGY_TYPE[server_type]\n\n if topology_type == TOPOLOGY_TYPE.Sharded:\n if server_type not in (SERVER_TYPE.Mongos, SERVER_TYPE.Unknown):\n sds.pop(address)\n\n elif topology_type == TOPOLOGY_TYPE.ReplicaSetNoPrimary:\n if server_type in (SERVER_TYPE.Standalone, SERVER_TYPE.Mongos):\n sds.pop(address)\n\n elif server_type == SERVER_TYPE.RSPrimary:\n (topology_type,\n set_name,\n max_set_version,\n max_election_id) = _update_rs_from_primary(sds,\n set_name,\n server_description,\n max_set_version,\n max_election_id)\n\n elif server_type in (\n SERVER_TYPE.RSSecondary,\n SERVER_TYPE.RSArbiter,\n SERVER_TYPE.RSOther):\n topology_type, set_name = _update_rs_no_primary_from_member(\n sds, set_name, server_description)\n\n elif topology_type == TOPOLOGY_TYPE.ReplicaSetWithPrimary:\n if server_type in (SERVER_TYPE.Standalone, SERVER_TYPE.Mongos):\n sds.pop(address)\n topology_type = _check_has_primary(sds)\n\n elif server_type == SERVER_TYPE.RSPrimary:\n (topology_type,\n set_name,\n max_set_version,\n max_election_id) = _update_rs_from_primary(sds,\n set_name,\n server_description,\n max_set_version,\n max_election_id)\n\n elif server_type in (\n SERVER_TYPE.RSSecondary,\n SERVER_TYPE.RSArbiter,\n SERVER_TYPE.RSOther):\n topology_type = _update_rs_with_primary_from_member(\n sds, set_name, server_description)\n\n else:\n # Server type is Unknown or RSGhost: did we just lose the primary?\n topology_type = _check_has_primary(sds)\n\n # Return updated copy.\n return TopologyDescription(topology_type,\n sds,\n set_name,\n max_set_version,\n max_election_id,\n topology_description._topology_settings)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _update_rs_from_primary(\n sds,\n replica_set_name,\n server_description,\n max_set_version,\n max_election_id):\n \"\"\"Update topology description from a primary's ismaster response.\n\n Pass in a dict of ServerDescriptions, current replica set name, the\n ServerDescription we are processing, and the TopologyDescription's\n max_set_version and max_election_id if any.\n\n Returns (new topology type, new replica_set_name, new max_set_version,\n new max_election_id).\n \"\"\"\n if replica_set_name is None:\n replica_set_name = server_description.replica_set_name\n\n elif replica_set_name != server_description.replica_set_name:\n # We found a primary but it doesn't have the replica_set_name\n # provided by the user.\n sds.pop(server_description.address)\n return (_check_has_primary(sds),\n replica_set_name,\n max_set_version,\n max_election_id)\n\n max_election_tuple = max_set_version, max_election_id\n if None not in server_description.election_tuple:\n if (None not in max_election_tuple and\n max_election_tuple > server_description.election_tuple):\n\n # Stale primary, set to type Unknown.\n address = server_description.address\n sds[address] = ServerDescription(address)\n return (_check_has_primary(sds),\n replica_set_name,\n max_set_version,\n max_election_id)\n\n max_election_id = server_description.election_id\n\n if (server_description.set_version is not None and\n (max_set_version is None or\n server_description.set_version > max_set_version)):\n\n max_set_version = server_description.set_version\n\n # We've heard from the primary. Is it the same primary as before?\n for server in sds.values():\n if (server.server_type is SERVER_TYPE.RSPrimary\n and server.address != server_description.address):\n\n # Reset old primary's type to Unknown.\n sds[server.address] = ServerDescription(server.address)\n\n # There can be only one prior primary.\n break\n\n # Discover new hosts from this primary's response.\n for new_address in server_description.all_hosts:\n if new_address not in sds:\n sds[new_address] = ServerDescription(new_address)\n\n # Remove hosts not in the response.\n for addr in set(sds) - server_description.all_hosts:\n sds.pop(addr)\n\n # If the host list differs from the seed list, we may not have a primary\n # after all.\n return (_check_has_primary(sds),\n replica_set_name,\n max_set_version,\n max_election_id)", "response": "Update the topology description from a primary s response."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the rs with a primary from a member.", "response": "def _update_rs_with_primary_from_member(\n sds,\n replica_set_name,\n server_description):\n \"\"\"RS with known primary. Process a response from a non-primary.\n\n Pass in a dict of ServerDescriptions, current replica set name, and the\n ServerDescription we are processing.\n\n Returns new topology type.\n \"\"\"\n assert replica_set_name is not None\n\n if replica_set_name != server_description.replica_set_name:\n sds.pop(server_description.address)\n elif (server_description.me and\n server_description.address != server_description.me):\n sds.pop(server_description.address)\n\n # Had this member been the primary?\n return _check_has_primary(sds)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _update_rs_no_primary_from_member(\n sds,\n replica_set_name,\n server_description):\n \"\"\"RS without known primary. Update from a non-primary's response.\n\n Pass in a dict of ServerDescriptions, current replica set name, and the\n ServerDescription we are processing.\n\n Returns (new topology type, new replica_set_name).\n \"\"\"\n topology_type = TOPOLOGY_TYPE.ReplicaSetNoPrimary\n if replica_set_name is None:\n replica_set_name = server_description.replica_set_name\n\n elif replica_set_name != server_description.replica_set_name:\n sds.pop(server_description.address)\n return topology_type, replica_set_name\n\n # This isn't the primary's response, so don't remove any servers\n # it doesn't report. Only add new servers.\n for address in server_description.all_hosts:\n if address not in sds:\n sds[address] = ServerDescription(address)\n\n if (server_description.me and\n server_description.address != server_description.me):\n sds.pop(server_description.address)\n\n return topology_type, replica_set_name", "response": "Update the rs with no primary from a non - primary response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if the current topology type is ReplicaSetWithPrimary.", "response": "def _check_has_primary(sds):\n \"\"\"Current topology type is ReplicaSetWithPrimary. Is primary still known?\n\n Pass in a dict of ServerDescriptions.\n\n Returns new topology type.\n \"\"\"\n for s in sds.values():\n if s.server_type == SERVER_TYPE.RSPrimary:\n return TOPOLOGY_TYPE.ReplicaSetWithPrimary\n else:\n return TOPOLOGY_TYPE.ReplicaSetNoPrimary"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef common_wire_version(self):\n servers = self.known_servers\n if servers:\n return min(s.max_wire_version for s in self.known_servers)\n\n return None", "response": "Minimum of all servers max wire versions or None."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef has_readable_server(self, read_preference=ReadPreference.PRIMARY):\n common.validate_read_preference(\"read_preference\", read_preference)\n return any(self.apply_selector(read_preference, None))", "response": "Returns True if there are readable servers available for the given read preference."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a human - friendly OS name.", "response": "def get_os():\n \"\"\"\n Human-friendly OS name\n \"\"\"\n if sys.platform == 'darwin':\n return 'mac'\n elif sys.platform.find('freebsd') != -1:\n return 'freebsd'\n elif sys.platform.find('linux') != -1:\n return 'linux'\n elif sys.platform.find('win32') != -1:\n return 'windows'\n elif sys.platform.find('sunos') != -1:\n return 'solaris'\n else:\n return sys.platform"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_bsd(name=None):\n name = name or sys.platform\n return Platform.is_darwin(name) or Platform.is_freebsd(name)", "response": "Return true if this is a BSD like operating system."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if the platform is a unix.", "response": "def is_unix(name=None):\n \"\"\" Return true if the platform is a unix, False otherwise. \"\"\"\n name = name or sys.platform\n return Platform.is_darwin(name) or Platform.is_linux(name) or Platform.is_freebsd(name)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_subprocess_output(command, log, raise_on_empty_output=True, log_debug=True):\n\n cmd_args = []\n if isinstance(command, string_types):\n for arg in command.split():\n cmd_args.append(arg)\n elif hasattr(type(command), '__iter__'):\n for arg in command:\n cmd_args.append(arg)\n else:\n raise TypeError('command must be a sequence or string')\n\n if log_debug:\n log.debug('Running get_subprocess_output with cmd: {}'.format(cmd_args))\n\n out, err, returncode = subprocess_output(cmd_args, raise_on_empty_output)\n\n log.debug(\n 'get_subprocess_output returned '\n '(len(out): {} ; len(err): {} ; returncode: {})'.format(len(out), len(err), returncode)\n )\n\n out = ensure_unicode(out) if out is not None else None\n err = ensure_unicode(err) if err is not None else None\n\n return out, err, returncode", "response": "Run a given subprocess command and return its output."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nshow changes since a specific date.", "response": "def changes(since, out_file, eager):\n \"\"\"Show changes since a specific date.\"\"\"\n root = get_root()\n history_data = defaultdict(lambda: {'lines': deque(), 'releasers': set()})\n\n with chdir(root):\n result = run_command(\n (\n 'git log \"--pretty=format:%H %s\" --date-order --date=iso8601 '\n '--since=\"{}T00:00:00\" */CHANGELOG.md'.format(since)\n ),\n capture=True,\n check=True,\n )\n\n for result_line in result.stdout.splitlines():\n commit_hash, commit_subject = result_line.split(' ', 1)\n\n if not eager and 'release' not in commit_subject.lower():\n continue\n\n result = run_command(\n 'git show \"--pretty=format:%an%n\" -U0 {} */CHANGELOG.md'.format(commit_hash), capture=True, check=True\n )\n\n # Example:\n #\n # \n # diff --git a//CHANGELOG.md b//CHANGELOG.md\n # index 89b5a3441..9534019a9 100644\n # --- a//CHANGELOG.md\n # +++ b//CHANGELOG.md\n # @@ -2,0 +3,5 @@\n # +## / \n # +\n # +* \n # +* \n # +\n # diff --git a//CHANGELOG.md b//CHANGELOG.md\n # index 89b5a3441..9534019a9 100644\n # --- a//CHANGELOG.md\n # +++ b//CHANGELOG.md\n # @@ -2,0 +3,4 @@\n # +## / \n # +\n # +* \n # +\n lines = deque(result.stdout.splitlines())\n author_name = lines.popleft().strip()\n\n patches = []\n for line in lines:\n if line:\n # New patch\n if line.startswith('diff --git'):\n patches.append([])\n patches[-1].append(line)\n\n for patch in patches:\n integration = patch[0].split('/')[-2].strip()\n history_data[integration]['releasers'].add(author_name)\n\n additions = deque()\n for line in reversed(patch):\n if line.startswith('+'):\n line = line[1:]\n # Demote releases to h3\n if line.startswith('##'):\n line = '#{}'.format(line)\n additions.append(line)\n elif line.startswith('@@'):\n break\n\n # Get rid of the header for new integrations\n if additions[-1].startswith('# '):\n additions.pop()\n\n # Get rid of blank lines to ensure consistency\n while not additions[0].strip():\n additions.popleft()\n while not additions[-1].strip():\n additions.pop()\n\n history_data[integration]['lines'].appendleft('')\n history_data[integration]['lines'].extendleft(additions)\n\n output_lines = ['# Changes since {}'.format(since), '']\n\n for integration, history in sorted(iteritems(history_data)):\n display_name = load_manifest(integration).get('display_name', integration)\n output_lines.append('## {}'.format(display_name))\n output_lines.append('released by: {}'.format(', '.join(sorted(history['releasers']))))\n\n output_lines.append('')\n output_lines.extend(history['lines'])\n\n output = '\\n'.join(output_lines)\n\n if out_file:\n write_file(out_file, output)\n else:\n echo_info(output)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef changelog(since, to, write, force):\n agent_tags = get_agent_tags(since, to)\n\n # store the changes in a mapping {agent_version --> {check_name --> current_version}}\n changes_per_agent = OrderedDict()\n\n # to keep indexing easy, we run the loop off-by-one\n for i in range(1, len(agent_tags)):\n req_file_name = os.path.basename(get_agent_release_requirements())\n current_tag = agent_tags[i - 1]\n # Requirements for current tag\n file_contents = git_show_file(req_file_name, current_tag)\n catalog_now = parse_agent_req_file(file_contents)\n # Requirements for previous tag\n file_contents = git_show_file(req_file_name, agent_tags[i])\n catalog_prev = parse_agent_req_file(file_contents)\n\n changes_per_agent[current_tag] = OrderedDict()\n\n for name, ver in iteritems(catalog_now):\n # at some point in the git history, the requirements file erroneusly\n # contained the folder name instead of the package name for each check,\n # let's be resilient\n old_ver = (\n catalog_prev.get(name)\n or catalog_prev.get(get_folder_name(name))\n or catalog_prev.get(get_package_name(name))\n )\n\n # normalize the package name to the check_name\n if name.startswith(DATADOG_PACKAGE_PREFIX):\n name = get_folder_name(name)\n\n if old_ver and old_ver != ver:\n # determine whether major version changed\n breaking = old_ver.split('.')[0] < ver.split('.')[0]\n changes_per_agent[current_tag][name] = (ver, breaking)\n elif not old_ver:\n # New integration\n changes_per_agent[current_tag][name] = (ver, False)\n\n # store the changelog in memory\n changelog_contents = StringIO()\n\n # prepare the links\n agent_changelog_url = 'https://github.com/DataDog/datadog-agent/blob/master/CHANGELOG.rst#{}'\n check_changelog_url = 'https://github.com/DataDog/integrations-core/blob/master/{}/CHANGELOG.md'\n\n # go through all the agent releases\n for agent, version_changes in iteritems(changes_per_agent):\n url = agent_changelog_url.format(agent.replace('.', '')) # Github removes dots from the anchor\n changelog_contents.write('## Datadog Agent version [{}]({})\\n\\n'.format(agent, url))\n\n if not version_changes:\n changelog_contents.write('* There were no integration updates for this version of the Agent.\\n\\n')\n else:\n for name, ver in iteritems(version_changes):\n # get the \"display name\" for the check\n manifest_file = os.path.join(get_root(), name, 'manifest.json')\n if os.path.exists(manifest_file):\n decoded = json.loads(read_file(manifest_file).strip(), object_pairs_hook=OrderedDict)\n display_name = decoded.get('display_name')\n else:\n display_name = name\n\n breaking_notice = \" **BREAKING CHANGE**\" if ver[1] else \"\"\n changelog_url = check_changelog_url.format(name)\n changelog_contents.write(\n '* {} [{}]({}){}\\n'.format(display_name, ver[0], changelog_url, breaking_notice)\n )\n # add an extra line to separate the release block\n changelog_contents.write('\\n')\n\n # save the changelog on disk if --write was passed\n if write:\n dest = get_agent_changelog()\n # don't overwrite an existing file\n if os.path.exists(dest) and not force:\n msg = \"Output file {} already exists, run the command again with --force to overwrite\"\n abort(msg.format(dest))\n\n write_file(dest, changelog_contents.getvalue())\n else:\n echo_info(changelog_contents.getvalue())", "response": "Generates a markdown file containing the list of checks that changed since Agent version 6. 3. 0."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating an empty instance if it doesn t exist.", "response": "def init_instance(self, key):\n \"\"\"\n Create an empty instance if it doesn't exist.\n If the instance already exists, this is a noop.\n \"\"\"\n with self._lock:\n if key not in self._metadata:\n self._metadata[key] = {}\n self._metric_ids[key] = []"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning whether a counter_id is present for a given instance key.", "response": "def contains(self, key, counter_id):\n \"\"\"\n Return whether a counter_id is present for a given instance key.\n If the key is not in the cache, raises a KeyError.\n \"\"\"\n with self._lock:\n return counter_id in self._metadata[key]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_metadata(self, key, metadata):\n with self._lock:\n self._metadata[key] = metadata", "response": "Store the metadata for the given instance key."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nstores the list of metric IDs that we want to collect for the given instance key", "response": "def set_metric_ids(self, key, metric_ids):\n \"\"\"\n Store the list of metric IDs we will want to collect for the given instance key\n \"\"\"\n with self._lock:\n self._metric_ids[key] = metric_ids"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_metadata(self, key, counter_id):\n with self._lock:\n metadata = self._metadata[key]\n try:\n return metadata[counter_id]\n except KeyError:\n raise MetadataNotFoundError(\"No metadata for counter id '{}' found in the cache.\".format(counter_id))", "response": "Get the metadata for the given instance key and counter_id."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef manifest(fix, include_extras):\n all_guids = {}\n\n root = get_root()\n root_name = basepath(get_root())\n\n ok_checks = 0\n failed_checks = 0\n fixed_checks = 0\n echo_info(\"Validating all manifest.json files...\")\n for check_name in sorted(os.listdir(root)):\n manifest_file = os.path.join(root, check_name, 'manifest.json')\n\n if file_exists(manifest_file):\n display_queue = []\n file_failures = 0\n file_fixed = False\n\n try:\n decoded = json.loads(read_file(manifest_file).strip(), object_pairs_hook=OrderedDict)\n except JSONDecodeError as e:\n failed_checks += 1\n echo_info(\"{}/manifest.json... \".format(check_name), nl=False)\n echo_failure(\"FAILED\")\n echo_failure(' invalid json: {}'.format(e))\n continue\n\n # attributes are valid\n attrs = set(decoded)\n for attr in sorted(attrs - ALL_ATTRIBUTES):\n file_failures += 1\n display_queue.append((echo_failure, ' Attribute `{}` is invalid'.format(attr)))\n for attr in sorted(REQUIRED_ATTRIBUTES - attrs):\n file_failures += 1\n display_queue.append((echo_failure, ' Attribute `{}` is required'.format(attr)))\n\n # guid\n guid = decoded.get('guid')\n if guid in all_guids:\n file_failures += 1\n output = ' duplicate `guid`: `{}` from `{}`'.format(guid, all_guids[guid])\n if fix:\n new_guid = uuid.uuid4()\n all_guids[new_guid] = check_name\n decoded['guid'] = new_guid\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `guid`: {}'.format(new_guid)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n elif not guid or not isinstance(guid, string_types):\n file_failures += 1\n output = ' required non-null string: guid'\n if fix:\n new_guid = uuid.uuid4()\n all_guids[new_guid] = check_name\n decoded['guid'] = new_guid\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `guid`: {}'.format(new_guid)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n else:\n all_guids[guid] = check_name\n\n # manifest_version\n correct_manifest_version = '1.0.0'\n manifest_version = decoded.get('manifest_version')\n version_parts = parse_version_parts(manifest_version)\n if len(version_parts) != 3:\n file_failures += 1\n\n if not manifest_version:\n output = ' required non-null string: manifest_version'\n else:\n output = ' invalid `manifest_version`: {}'.format(manifest_version)\n\n if fix:\n version_parts = parse_version_parts(correct_manifest_version)\n decoded['manifest_version'] = correct_manifest_version\n\n display_queue.append((echo_warning, output))\n display_queue.append(\n (echo_success, ' new `manifest_version`: {}'.format(correct_manifest_version))\n )\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n if len(version_parts) == 3:\n about_exists = os.path.isfile(\n os.path.join(root, check_name, 'datadog_checks', check_name, '__about__.py')\n )\n if version_parts >= [1, 0, 0]:\n if 'version' in decoded and about_exists:\n file_failures += 1\n output = ' outdated field: version'\n\n if fix:\n del decoded['version']\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' removed field: version'))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n elif about_exists:\n file_failures += 1\n output = ' outdated `manifest_version`: {}'.format(manifest_version)\n\n if fix:\n decoded['manifest_version'] = correct_manifest_version\n\n display_queue.append((echo_warning, output))\n display_queue.append(\n (echo_success, ' new `manifest_version`: {}'.format(correct_manifest_version))\n )\n\n if 'version' in decoded:\n del decoded['version']\n display_queue.append((echo_success, ' removed field: version'))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n else:\n version = decoded.get('version')\n version_parts = parse_version_parts(version)\n if len(version_parts) != 3:\n file_failures += 1\n\n if not version:\n display_queue.append((echo_failure, ' required non-null string: version'))\n else:\n display_queue.append((echo_failure, ' invalid `version`: {}'.format(version)))\n\n # integration_id\n integration_id = decoded.get('integration_id')\n if not re.search(INTEGRATION_ID_REGEX, integration_id):\n file_failures += 1\n output = 'integration_id contains invalid characters'\n display_queue.append((echo_failure, output))\n\n # maintainer\n if root_name == 'integrations-core':\n correct_maintainer = 'help@datadoghq.com'\n maintainer = decoded.get('maintainer')\n if maintainer != correct_maintainer:\n file_failures += 1\n output = ' incorrect `maintainer`: {}'.format(maintainer)\n\n if fix:\n decoded['maintainer'] = correct_maintainer\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `maintainer`: {}'.format(correct_maintainer)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n # name\n correct_name = check_name\n name = decoded.get('name')\n if not isinstance(name, string_types) or name.lower() != correct_name.lower():\n file_failures += 1\n output = ' incorrect `name`: {}'.format(name)\n\n if fix:\n decoded['name'] = correct_name\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `name`: {}'.format(correct_name)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n # short_description\n short_description = decoded.get('short_description')\n if not short_description or not isinstance(short_description, string_types):\n file_failures += 1\n display_queue.append((echo_failure, ' required non-null string: short_description'))\n if len(short_description) > 80:\n file_failures += 1\n display_queue.append((echo_failure, ' should contain 80 characters maximum: short_description'))\n\n # support\n correct_support = 'contrib' if root_name == 'integrations-extras' else 'core'\n support = decoded.get('support')\n if support != correct_support:\n file_failures += 1\n output = ' incorrect `support`: {}'.format(support)\n\n if fix:\n decoded['support'] = correct_support\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `support`: {}'.format(correct_support)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n if include_extras:\n # supported_os\n supported_os = decoded.get('supported_os')\n if not supported_os or not isinstance(supported_os, list):\n file_failures += 1\n display_queue.append((echo_failure, ' required non-null sequence: supported_os'))\n else:\n known_systems = {'linux', 'mac_os', 'windows'}\n unknown_systems = sorted(set(supported_os) - known_systems)\n if unknown_systems:\n file_failures += 1\n display_queue.append(\n (echo_failure, ' unknown `supported_os`: {}'.format(', '.join(unknown_systems)))\n )\n\n # public_title\n public_title = decoded.get('public_title')\n if not public_title or not isinstance(public_title, string_types):\n file_failures += 1\n display_queue.append((echo_failure, ' required non-null string: public_title'))\n else:\n title_start = 'Datadog-'\n title_end = ' Integration'\n section_char_set = set(public_title[len(title_start) : -len(title_end)].lower())\n check_name_char_set = set(check_name.lower())\n character_overlap = check_name_char_set & section_char_set\n\n correct_start = public_title.startswith(title_start)\n correct_end = public_title.endswith(title_end)\n overlap_enough = len(character_overlap) > int(len(check_name_char_set) * 0.5)\n\n if not (correct_start and correct_end and overlap_enough):\n file_failures += 1\n display_queue.append((echo_failure, ' invalid `public_title`: {}'.format(public_title)))\n\n # categories\n categories = decoded.get('categories')\n if not categories or not isinstance(categories, list):\n file_failures += 1\n display_queue.append((echo_failure, ' required non-null sequence: categories'))\n\n # type\n correct_integration_types = ['check', 'crawler']\n integration_type = decoded.get('type')\n if not integration_type or not isinstance(integration_type, string_types):\n file_failures += 1\n output = ' required non-null string: type'\n display_queue.append((echo_failure, output))\n elif integration_type not in correct_integration_types:\n file_failures += 1\n output = ' invalid `type`: {}'.format(integration_type)\n display_queue.append((echo_failure, output))\n\n # is_public\n correct_is_public = True\n is_public = decoded.get('is_public')\n if not isinstance(is_public, bool):\n file_failures += 1\n output = ' required boolean: is_public'\n\n if fix:\n decoded['is_public'] = correct_is_public\n\n display_queue.append((echo_warning, output))\n display_queue.append((echo_success, ' new `is_public`: {}'.format(correct_is_public)))\n\n file_failures -= 1\n file_fixed = True\n else:\n display_queue.append((echo_failure, output))\n\n if file_failures > 0:\n failed_checks += 1\n # Display detailed info if file invalid\n echo_info(\"{}/manifest.json... \".format(check_name), nl=False)\n echo_failure(\"FAILED\")\n for display_func, message in display_queue:\n display_func(message)\n elif not file_fixed:\n ok_checks += 1\n\n if fix and file_fixed:\n new_manifest = '{}\\n'.format(json.dumps(decoded, indent=2, separators=(',', ': ')))\n write_file(manifest_file, new_manifest)\n # Display detailed info if file has been completely fixed\n if file_failures == 0:\n fixed_checks += 1\n echo_info(\"{}/manifest.json... \".format(check_name), nl=False)\n echo_success(\"FIXED\")\n for display_func, message in display_queue:\n display_func(message)\n\n if ok_checks:\n echo_success(\"{} valid files\".format(ok_checks))\n if fixed_checks:\n echo_info(\"{} fixed files\".format(fixed_checks))\n if failed_checks:\n echo_failure(\"{} invalid files\".format(failed_checks))\n abort()", "response": "Validate all manifest. json files."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the python level of a log level.", "response": "def _get_py_loglevel(lvl):\n \"\"\"\n Map log levels to strings\n \"\"\"\n if not lvl:\n lvl = 'INFO'\n\n return LOG_LEVEL_MAP.get(lvl.upper(), logging.DEBUG)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitializes logging for the current state of the application", "response": "def init_logging():\n \"\"\"\n Initialize logging (set up forwarding to Go backend and sane defaults)\n \"\"\"\n # Forward to Go backend\n logging.addLevelName(TRACE_LEVEL, 'TRACE')\n logging.setLoggerClass(AgentLogger)\n rootLogger = logging.getLogger()\n rootLogger.addHandler(AgentLogHandler())\n rootLogger.setLevel(_get_py_loglevel(datadog_agent.get_config('log_level')))\n\n # `requests` (used in a lot of checks) imports `urllib3`, which logs a bunch of stuff at the info level\n # Therefore, pre emptively increase the default level of that logger to `WARN`\n urllib_logger = logging.getLogger(\"requests.packages.urllib3\")\n urllib_logger.setLevel(logging.WARN)\n urllib_logger.propagate = True"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread line - by - line and run callback on each line.", "response": "def tail(self, line_by_line=True, move_end=True):\n \"\"\"Read line-by-line and run callback on each line.\n line_by_line: yield each time a callback has returned True\n move_end: start from the last line of the log\"\"\"\n try:\n self._open_file(move_end=move_end)\n\n while True:\n pos = self._f.tell()\n line = self._f.readline()\n if line:\n line = line.strip(chr(0)) # a truncate may have create holes in the file\n if self._callback(line.rstrip(\"\\n\")):\n if line_by_line:\n yield True\n pos = self._f.tell()\n self._open_file(move_end=False, pos=pos)\n else:\n continue\n else:\n continue\n else:\n yield True\n assert pos == self._f.tell()\n self._open_file(move_end=False, pos=pos)\n\n except Exception as e:\n # log but survive\n self._log.exception(e)\n raise StopIteration(e)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef environment_run(up, down, sleep=None, endpoints=None, conditions=None, env_vars=None, wrapper=None):\n if not callable(up):\n raise TypeError('The custom setup `{}` is not callable.'.format(repr(up)))\n elif not callable(down):\n raise TypeError('The custom tear down `{}` is not callable.'.format(repr(down)))\n\n conditions = list(conditions) if conditions is not None else []\n\n if endpoints is not None:\n conditions.append(CheckEndpoints(endpoints))\n\n env_vars = mock_context_manager() if env_vars is None else EnvVars(env_vars)\n wrapper = mock_context_manager() if wrapper is None else wrapper\n\n result = None\n with env_vars, wrapper:\n try:\n if set_up_env():\n result = up()\n\n for condition in conditions:\n condition()\n\n if sleep:\n time.sleep(sleep)\n\n yield result\n finally:\n if tear_down_env():\n down()", "response": "A generator that runs the environment functions up and tear down."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a copy of the proxies dictionary that will disable the proxy if the uri provided is reached directly.", "response": "def config_proxy_skip(proxies, uri, skip_proxy=False):\n \"\"\"\n Returns an amended copy of the proxies dictionary - used by `requests`,\n it will disable the proxy if the uri provided is to be reached directly.\n :param proxies A dict with existing proxies: `https`, `http`, and `no` are potential keys.\n :param uri URI to determine if a proxy is necessary or not.\n :param skip_proxy If `True`, the returned proxy dictionary will disable all proxies.\n \"\"\"\n parsed_uri = urlparse(uri)\n\n # disable proxy if necessary\n if skip_proxy:\n proxies['http'] = ''\n proxies['https'] = ''\n elif proxies.get('no'):\n urls = []\n if isinstance(proxies['no'], string_types):\n urls = proxies['no'].replace(';', ',').split(\",\")\n elif isinstance(proxies['no'], list):\n urls = proxies['no']\n for url in urls:\n if url in parsed_uri.netloc:\n if 'http' in proxies:\n proxies.pop('http')\n if 'https' in proxies:\n proxies.pop('https')\n\n return proxies"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef close(self):\n if self._publish:\n self._events.put((self._listener.publish_server_closed,\n (self._description.address, self._topology_id)))\n self._monitor.close()\n self._pool.reset()", "response": "Close the connection pool and stop the monitor."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends an unacknowledged message to the MongoDB server.", "response": "def send_message(self, message, all_credentials):\n \"\"\"Send an unacknowledged message to MongoDB.\n\n Can raise ConnectionFailure.\n\n :Parameters:\n - `message`: (request_id, data).\n - `all_credentials`: dict, maps auth source to MongoCredential.\n \"\"\"\n _, data, max_doc_size = self._split_message(message)\n with self.get_socket(all_credentials) as sock_info:\n sock_info.send_message(data, max_doc_size)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a message to MongoDB and return a Response object.", "response": "def send_message_with_response(\n self,\n operation,\n set_slave_okay,\n all_credentials,\n listeners,\n exhaust=False):\n \"\"\"Send a message to MongoDB and return a Response object.\n\n Can raise ConnectionFailure.\n\n :Parameters:\n - `operation`: A _Query or _GetMore object.\n - `set_slave_okay`: Pass to operation.get_message.\n - `all_credentials`: dict, maps auth source to MongoCredential.\n - `listeners`: Instance of _EventListeners or None.\n - `exhaust` (optional): If True, the socket used stays checked out.\n It is returned along with its Pool in the Response.\n \"\"\"\n with self.get_socket(all_credentials, exhaust) as sock_info:\n\n duration = None\n publish = listeners.enabled_for_commands\n if publish:\n start = datetime.now()\n\n use_find_cmd = False\n if sock_info.max_wire_version >= 4:\n if not exhaust:\n use_find_cmd = True\n elif (isinstance(operation, _Query) and\n not operation.read_concern.ok_for_legacy):\n raise ConfigurationError(\n 'read concern level of %s is not valid '\n 'with a max wire version of %d.'\n % (operation.read_concern.level,\n sock_info.max_wire_version))\n if (isinstance(operation, _Query) and\n sock_info.max_wire_version < 5 and\n operation.collation is not None):\n raise ConfigurationError(\n 'Specifying a collation is unsupported with a max wire '\n 'version of %d.' % (sock_info.max_wire_version,))\n message = operation.get_message(\n set_slave_okay, sock_info.is_mongos, use_find_cmd)\n request_id, data, max_doc_size = self._split_message(message)\n\n if publish:\n encoding_duration = datetime.now() - start\n cmd, dbn = operation.as_command()\n listeners.publish_command_start(\n cmd, dbn, request_id, sock_info.address)\n start = datetime.now()\n\n try:\n sock_info.send_message(data, max_doc_size)\n response_data = sock_info.receive_message(1, request_id)\n except Exception as exc:\n if publish:\n duration = (datetime.now() - start) + encoding_duration\n failure = _convert_exception(exc)\n listeners.publish_command_failure(\n duration, failure, next(iter(cmd)), request_id,\n sock_info.address)\n raise\n\n if publish:\n duration = (datetime.now() - start) + encoding_duration\n\n if exhaust:\n return ExhaustResponse(\n data=response_data,\n address=self._description.address,\n socket_info=sock_info,\n pool=self._pool,\n duration=duration,\n request_id=request_id,\n from_command=use_find_cmd)\n else:\n return Response(\n data=response_data,\n address=self._description.address,\n duration=duration,\n request_id=request_id,\n from_command=use_find_cmd)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _split_message(self, message):\n if len(message) == 3:\n return message\n else:\n # get_more and kill_cursors messages don't include BSON documents.\n request_id, data = message\n return request_id, data, 0", "response": "Split a message into request_id data max_doc_size."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef upserted_ids(self):\n self._raise_if_unacknowledged(\"upserted_ids\")\n if self.__bulk_api_result:\n return dict((upsert[\"index\"], upsert[\"_id\"])\n for upsert in self.bulk_api_result[\"upserted\"])", "response": "A map of operation index to the _id of the upserted document."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nshows the contents of the config file.", "response": "def show(all_keys):\n \"\"\"Show the contents of the config file.\"\"\"\n if not config_file_exists():\n echo_info('No config file found! Please try `ddev config restore`.')\n else:\n if all_keys:\n echo_info(read_config_file().rstrip())\n else:\n echo_info(read_config_file_scrubbed().rstrip())"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_value(ctx, key, value):\n scrubbing = False\n if value is None:\n scrubbing = key in SECRET_KEYS\n value = click.prompt('Value for `{}`'.format(key), hide_input=scrubbing)\n\n if key in ('core', 'extras', 'agent') and not value.startswith('~'):\n value = os.path.abspath(value)\n\n user_config = new_config = ctx.obj\n user_config.pop('repo_choice', None)\n\n data = [value]\n data.extend(reversed(key.split('.')))\n key = data.pop()\n value = data.pop()\n\n # Use a separate mapping to show only what has changed in the end\n branch_config_root = branch_config = {}\n\n # Consider dots as keys\n while data:\n default_branch = {value: ''}\n branch_config[key] = default_branch\n branch_config = branch_config[key]\n\n new_value = new_config.get(key)\n if not hasattr(new_value, 'get'):\n new_value = default_branch\n\n new_config[key] = new_value\n new_config = new_config[key]\n\n key = value\n value = data.pop()\n\n value = string_to_toml_type(value)\n branch_config[key] = new_config[key] = value\n\n save_config(user_config)\n\n output_config = scrub_secrets(branch_config_root) if scrubbing else branch_config_root\n echo_success('New setting:')\n echo_info(toml.dumps(output_config).rstrip())", "response": "Assigns values to config file entries."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns this instance s socket to the connection pool.", "response": "def close(self):\n \"\"\"Return this instance's socket to the connection pool.\n \"\"\"\n if not self.__closed:\n self.__closed = True\n self.pool.return_socket(self.sock)\n self.sock, self.pool = None, None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrewinding this cursor to its unevaluated state.", "response": "def rewind(self):\n \"\"\"Rewind this cursor to its unevaluated state.\n\n Reset this cursor if it has been partially or completely evaluated.\n Any options that are present on the cursor will remain in effect.\n Future iterating performed on this cursor will cause new queries to\n be sent to the server, even if the resultant data has already been\n retrieved by this cursor.\n \"\"\"\n self.__data = deque()\n self.__id = None\n self.__address = None\n self.__retrieved = 0\n self.__killed = False\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __query_spec(self):\n operators = self.__modifiers.copy()\n if self.__ordering:\n operators[\"$orderby\"] = self.__ordering\n if self.__explain:\n operators[\"$explain\"] = True\n if self.__hint:\n operators[\"$hint\"] = self.__hint\n if self.__comment:\n operators[\"$comment\"] = self.__comment\n if self.__max_scan:\n operators[\"$maxScan\"] = self.__max_scan\n if self.__max_time_ms is not None:\n operators[\"$maxTimeMS\"] = self.__max_time_ms\n if self.__max:\n operators[\"$max\"] = self.__max\n if self.__min:\n operators[\"$min\"] = self.__min\n if self.__return_key:\n operators[\"$returnKey\"] = self.__return_key\n if self.__show_record_id:\n # This is upgraded to showRecordId for MongoDB 3.2+ \"find\" command.\n operators[\"$showDiskLoc\"] = self.__show_record_id\n if self.__snapshot:\n operators[\"$snapshot\"] = self.__snapshot\n\n if operators:\n # Make a shallow copy so we can cleanly rewind or clone.\n spec = self.__spec.copy()\n\n # White-listed commands must be wrapped in $query.\n if \"$query\" not in spec:\n # $query has to come first\n spec = SON([(\"$query\", spec)])\n\n if not isinstance(spec, SON):\n # Ensure the spec is SON. As order is important this will\n # ensure its set before merging in any extra operators.\n spec = SON(spec)\n\n spec.update(operators)\n return spec\n # Have to wrap with $query if \"query\" is the first key.\n # We can't just use $query anytime \"query\" is a key as\n # that breaks commands like count and find_and_modify.\n # Checking spec.keys()[0] covers the case that the spec\n # was passed as an instance of SON or OrderedDict.\n elif (\"query\" in self.__spec and\n (len(self.__spec) == 1 or\n next(iter(self.__spec)) == \"query\")):\n return SON({\"$query\": self.__spec})\n\n return self.__spec", "response": "Get the spec to use for a query."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets arbitrary query flags using a bitmask.", "response": "def add_option(self, mask):\n \"\"\"Set arbitrary query flags using a bitmask.\n\n To set the tailable flag:\n cursor.add_option(2)\n \"\"\"\n if not isinstance(mask, int):\n raise TypeError(\"mask must be an int\")\n self.__check_okay_to_chain()\n\n if mask & _QUERY_OPTIONS[\"exhaust\"]:\n if self.__limit:\n raise InvalidOperation(\"Can't use limit and exhaust together.\")\n if self.__collection.database.client.is_mongos:\n raise InvalidOperation('Exhaust cursors are '\n 'not supported by mongos')\n self.__exhaust = True\n\n self.__query_flags |= mask\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef limit(self, limit):\n if not isinstance(limit, integer_types):\n raise TypeError(\"limit must be an integer\")\n if self.__exhaust:\n raise InvalidOperation(\"Can't use limit and exhaust together.\")\n self.__check_okay_to_chain()\n\n self.__empty = False\n self.__limit = limit\n return self", "response": "Limits the number of results returned by this cursor."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nskipping the first skip results of this cursor.", "response": "def skip(self, skip):\n \"\"\"Skips the first `skip` results of this cursor.\n\n Raises :exc:`TypeError` if `skip` is not an integer. Raises\n :exc:`ValueError` if `skip` is less than ``0``. Raises\n :exc:`~pymongo.errors.InvalidOperation` if this :class:`Cursor` has\n already been used. The last `skip` applied to this cursor takes\n precedence.\n\n :Parameters:\n - `skip`: the number of results to skip\n \"\"\"\n if not isinstance(skip, integer_types):\n raise TypeError(\"skip must be an integer\")\n if skip < 0:\n raise ValueError(\"skip must be >= 0\")\n self.__check_okay_to_chain()\n\n self.__skip = skip\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the maximum time limit for a query operation.", "response": "def max_time_ms(self, max_time_ms):\n \"\"\"Specifies a time limit for a query operation. If the specified\n time is exceeded, the operation will be aborted and\n :exc:`~pymongo.errors.ExecutionTimeout` is raised. If `max_time_ms`\n is ``None`` no limit is applied.\n\n Raises :exc:`TypeError` if `max_time_ms` is not an integer or ``None``.\n Raises :exc:`~pymongo.errors.InvalidOperation` if this :class:`Cursor`\n has already been used.\n\n :Parameters:\n - `max_time_ms`: the time limit after which the operation is aborted\n \"\"\"\n if (not isinstance(max_time_ms, integer_types)\n and max_time_ms is not None):\n raise TypeError(\"max_time_ms must be an integer or None\")\n self.__check_okay_to_chain()\n\n self.__max_time_ms = max_time_ms\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the maximum time limit for a getMore operation on a single object.", "response": "def max_await_time_ms(self, max_await_time_ms):\n \"\"\"Specifies a time limit for a getMore operation on a\n :attr:`~pymongo.cursor.CursorType.TAILABLE_AWAIT` cursor. For all other\n types of cursor max_await_time_ms is ignored.\n\n Raises :exc:`TypeError` if `max_await_time_ms` is not an integer or\n ``None``. Raises :exc:`~pymongo.errors.InvalidOperation` if this\n :class:`Cursor` has already been used.\n\n .. note:: `max_await_time_ms` requires server version **>= 3.2**\n\n :Parameters:\n - `max_await_time_ms`: the time limit after which the operation is\n aborted\n\n .. versionadded:: 3.2\n \"\"\"\n if (not isinstance(max_await_time_ms, integer_types)\n and max_await_time_ms is not None):\n raise TypeError(\"max_await_time_ms must be an integer or None\")\n self.__check_okay_to_chain()\n\n # Ignore max_await_time_ms if not tailable or await_data is False.\n if self.__query_flags & CursorType.TAILABLE_AWAIT:\n self.__max_await_time_ms = max_await_time_ms\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef max(self, spec):\n if not isinstance(spec, (list, tuple)):\n raise TypeError(\"spec must be an instance of list or tuple\")\n\n self.__check_okay_to_chain()\n self.__max = SON(spec)\n return self", "response": "Adds an max operator to the chain."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd an operator that specifies the lower bound for all keys of a specific index.", "response": "def min(self, spec):\n \"\"\"Adds `min` operator that specifies lower bound for specific index.\n\n :Parameters:\n - `spec`: a list of field, limit pairs specifying the inclusive\n lower bound for all keys of a specific index in order.\n\n .. versionadded:: 2.7\n \"\"\"\n if not isinstance(spec, (list, tuple)):\n raise TypeError(\"spec must be an instance of list or tuple\")\n\n self.__check_okay_to_chain()\n self.__min = SON(spec)\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsorts this cursor s results.", "response": "def sort(self, key_or_list, direction=None):\n \"\"\"Sorts this cursor's results.\n\n Pass a field name and a direction, either\n :data:`~pymongo.ASCENDING` or :data:`~pymongo.DESCENDING`::\n\n for doc in collection.find().sort('field', pymongo.ASCENDING):\n print(doc)\n\n To sort by multiple fields, pass a list of (key, direction) pairs::\n\n for doc in collection.find().sort([\n ('field1', pymongo.ASCENDING),\n ('field2', pymongo.DESCENDING)]):\n print(doc)\n\n Beginning with MongoDB version 2.6, text search results can be\n sorted by relevance::\n\n cursor = db.test.find(\n {'$text': {'$search': 'some words'}},\n {'score': {'$meta': 'textScore'}})\n\n # Sort by 'score' field.\n cursor.sort([('score', {'$meta': 'textScore'})])\n\n for doc in cursor:\n print(doc)\n\n Raises :class:`~pymongo.errors.InvalidOperation` if this cursor has\n already been used. Only the last :meth:`sort` applied to this\n cursor has any effect.\n\n :Parameters:\n - `key_or_list`: a single key or a list of (key, direction)\n pairs specifying the keys to sort on\n - `direction` (optional): only used if `key_or_list` is a single\n key, if not given :data:`~pymongo.ASCENDING` is assumed\n \"\"\"\n self.__check_okay_to_chain()\n keys = helpers._index_list(key_or_list, direction)\n self.__ordering = helpers._index_document(keys)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef count(self, with_limit_and_skip=False):\n validate_boolean(\"with_limit_and_skip\", with_limit_and_skip)\n cmd = SON([(\"count\", self.__collection.name),\n (\"query\", self.__spec)])\n if self.__max_time_ms is not None:\n cmd[\"maxTimeMS\"] = self.__max_time_ms\n if self.__comment:\n cmd[\"$comment\"] = self.__comment\n\n if self.__hint is not None:\n cmd[\"hint\"] = self.__hint\n\n if with_limit_and_skip:\n if self.__limit:\n cmd[\"limit\"] = self.__limit\n if self.__skip:\n cmd[\"skip\"] = self.__skip\n\n return self.__collection._count(cmd, self.__collation)", "response": "Returns the number of documents in the results set for this query."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef distinct(self, key):\n options = {}\n if self.__spec:\n options[\"query\"] = self.__spec\n if self.__max_time_ms is not None:\n options['maxTimeMS'] = self.__max_time_ms\n if self.__comment:\n options['$comment'] = self.__comment\n if self.__collation is not None:\n options['collation'] = self.__collation\n\n return self.__collection.distinct(key, **options)", "response": "Returns a list of distinct values for key among all documents\nAttributeNames in this query."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an explain plan record for this cursor.", "response": "def explain(self):\n \"\"\"Returns an explain plan record for this cursor.\n\n .. mongodoc:: explain\n \"\"\"\n c = self.clone()\n c.__explain = True\n\n # always use a hard limit for explains\n if c.__limit:\n c.__limit = -abs(c.__limit)\n return next(c)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef where(self, code):\n self.__check_okay_to_chain()\n if not isinstance(code, Code):\n code = Code(code)\n\n self.__spec[\"$where\"] = code\n return self", "response": "Adds a $where clause to this query."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a collation to this cursor.", "response": "def collation(self, collation):\n \"\"\"Adds a :class:`~pymongo.collation.Collation` to this query.\n\n This option is only supported on MongoDB 3.4 and above.\n\n Raises :exc:`TypeError` if `collation` is not an instance of\n :class:`~pymongo.collation.Collation` or a ``dict``. Raises\n :exc:`~pymongo.errors.InvalidOperation` if this :class:`Cursor` has\n already been used. Only the last collation applied to this cursor has\n any effect.\n\n :Parameters:\n - `collation`: An instance of :class:`~pymongo.collation.Collation`.\n \"\"\"\n self.__check_okay_to_chain()\n self.__collation = validate_collation_or_none(collation)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __send_message(self, operation):\n client = self.__collection.database.client\n listeners = client._event_listeners\n publish = listeners.enabled_for_commands\n from_command = False\n\n if operation:\n kwargs = {\n \"read_preference\": self.__read_preference,\n \"exhaust\": self.__exhaust,\n }\n if self.__address is not None:\n kwargs[\"address\"] = self.__address\n\n try:\n response = client._send_message_with_response(operation,\n **kwargs)\n self.__address = response.address\n if self.__exhaust:\n # 'response' is an ExhaustResponse.\n self.__exhaust_mgr = _SocketManager(response.socket_info,\n response.pool)\n\n cmd_name = operation.name\n data = response.data\n cmd_duration = response.duration\n rqst_id = response.request_id\n from_command = response.from_command\n except AutoReconnect:\n # Don't try to send kill cursors on another socket\n # or to another server. It can cause a _pinValue\n # assertion on some server releases if we get here\n # due to a socket timeout.\n self.__killed = True\n raise\n else:\n # Exhaust cursor - no getMore message.\n rqst_id = 0\n cmd_name = 'getMore'\n if publish:\n # Fake a getMore command.\n cmd = SON([('getMore', self.__id),\n ('collection', self.__collection.name)])\n if self.__batch_size:\n cmd['batchSize'] = self.__batch_size\n if self.__max_time_ms:\n cmd['maxTimeMS'] = self.__max_time_ms\n listeners.publish_command_start(\n cmd, self.__collection.database.name, 0, self.__address)\n start = datetime.datetime.now()\n try:\n data = self.__exhaust_mgr.sock.receive_message(1, None)\n except Exception as exc:\n if publish:\n duration = datetime.datetime.now() - start\n listeners.publish_command_failure(\n duration, _convert_exception(exc), cmd_name, rqst_id,\n self.__address)\n if isinstance(exc, ConnectionFailure):\n self.__die()\n raise\n if publish:\n cmd_duration = datetime.datetime.now() - start\n\n if publish:\n start = datetime.datetime.now()\n try:\n doc = helpers._unpack_response(response=data,\n cursor_id=self.__id,\n codec_options=self.__codec_options)\n if from_command:\n helpers._check_command_response(doc['data'][0])\n except OperationFailure as exc:\n self.__killed = True\n\n # Make sure exhaust socket is returned immediately, if necessary.\n self.__die()\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, exc.details, cmd_name, rqst_id, self.__address)\n\n # If this is a tailable cursor the error is likely\n # due to capped collection roll over. Setting\n # self.__killed to True ensures Cursor.alive will be\n # False. No need to re-raise.\n if self.__query_flags & _QUERY_OPTIONS[\"tailable_cursor\"]:\n return\n raise\n except NotMasterError as exc:\n # Don't send kill cursors to another server after a \"not master\"\n # error. It's completely pointless.\n self.__killed = True\n\n # Make sure exhaust socket is returned immediately, if necessary.\n self.__die()\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, exc.details, cmd_name, rqst_id, self.__address)\n\n client._reset_server_and_request_check(self.__address)\n raise\n except Exception as exc:\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n listeners.publish_command_failure(\n duration, _convert_exception(exc), cmd_name, rqst_id,\n self.__address)\n raise\n\n if publish:\n duration = (datetime.datetime.now() - start) + cmd_duration\n # Must publish in find / getMore / explain command response format.\n if from_command:\n res = doc['data'][0]\n elif cmd_name == \"explain\":\n res = doc[\"data\"][0] if doc[\"number_returned\"] else {}\n else:\n res = {\"cursor\": {\"id\": doc[\"cursor_id\"],\n \"ns\": self.__collection.full_name},\n \"ok\": 1}\n if cmd_name == \"find\":\n res[\"cursor\"][\"firstBatch\"] = doc[\"data\"]\n else:\n res[\"cursor\"][\"nextBatch\"] = doc[\"data\"]\n listeners.publish_command_success(\n duration, res, cmd_name, rqst_id, self.__address)\n\n if from_command and cmd_name != \"explain\":\n cursor = doc['data'][0]['cursor']\n self.__id = cursor['id']\n if cmd_name == 'find':\n documents = cursor['firstBatch']\n else:\n documents = cursor['nextBatch']\n self.__data = deque(documents)\n self.__retrieved += len(documents)\n else:\n self.__id = doc[\"cursor_id\"]\n self.__data = deque(doc[\"data\"])\n self.__retrieved += doc[\"number_returned\"]\n\n if self.__id == 0:\n self.__killed = True\n\n\n if self.__limit and self.__id and self.__limit <= self.__retrieved:\n self.__die()\n\n # Don't wait for garbage collection to call __del__, return the\n # socket to the pool now.\n if self.__exhaust and self.__id == 0:\n self.__exhaust_mgr.close()", "response": "Send a query or getmore operation and handles the response."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _refresh(self):\n if len(self.__data) or self.__killed:\n return len(self.__data)\n\n if self.__id is None: # Query\n self.__send_message(_Query(self.__query_flags,\n self.__collection.database.name,\n self.__collection.name,\n self.__skip,\n self.__query_spec(),\n self.__projection,\n self.__codec_options,\n self.__read_preference,\n self.__limit,\n self.__batch_size,\n self.__read_concern,\n self.__collation))\n if not self.__id:\n self.__killed = True\n elif self.__id: # Get More\n if self.__limit:\n limit = self.__limit - self.__retrieved\n if self.__batch_size:\n limit = min(limit, self.__batch_size)\n else:\n limit = self.__batch_size\n\n # Exhaust cursors don't send getMore messages.\n if self.__exhaust:\n self.__send_message(None)\n else:\n self.__send_message(_GetMore(self.__collection.database.name,\n self.__collection.name,\n limit,\n self.__id,\n self.__codec_options,\n self.__max_await_time_ms))\n\n else: # Cursor id is zero nothing else to return\n self.__killed = True\n\n return len(self.__data)", "response": "Refreshes the cursor with more data from Mongo."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting an exception message to the log.", "response": "def print_exception(self, msg):\n \"\"\" Print exceptions happening in separate threads\n Prevent from logging a ton of them if a potentially big number of them fail the same way\n \"\"\"\n if self.exception_printed < 10:\n self.log.error(msg)\n self.exception_printed += 1"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the set of metrics needed to report the MORs.", "response": "def _compute_needed_metrics(self, instance, available_metrics):\n \"\"\" Compare the available metrics for one MOR we have computed and intersect them\n with the set of metrics we want to report\n \"\"\"\n i_key = self._instance_key(instance)\n if self.in_compatibility_mode(instance):\n if instance.get('all_metrics', False):\n return available_metrics\n\n wanted_metrics = []\n # Get only the basic metrics\n for counter_id in available_metrics:\n # No cache yet, skip it for now\n if not self.metadata_cache.contains(i_key, counter_id):\n self.log.debug(\n \"No metadata found for counter {}, will not collect it\".format(ensure_unicode(counter_id))\n )\n continue\n metadata = self.metadata_cache.get_metadata(i_key, counter_id)\n if metadata.get('name') in BASIC_METRICS:\n wanted_metrics.append(vim.PerformanceManager.MetricId(counterId=counter_id, instance=\"*\"))\n\n return wanted_metrics\n else:\n # The metadata cache contains only metrics of the desired level, so use it to filter the metrics to keep\n return [\n vim.PerformanceManager.MetricId(counterId=counter_id, instance=\"*\")\n for counter_id in available_metrics\n if self.metadata_cache.contains(i_key, counter_id)\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_external_host_tags(self):\n self.log.debug(\"Sending external_host_tags now\")\n external_host_tags = []\n for instance in self.instances:\n i_key = self._instance_key(instance)\n if not self.mor_cache.contains(i_key):\n self.log.warning(\"Unable to extract host tags for vSphere instance: {}\".format(i_key))\n continue\n\n for _, mor in self.mor_cache.mors(i_key):\n # Note: some mors have a None hostname\n hostname = mor.get('hostname')\n if hostname:\n external_host_tags.append((hostname, {SOURCE_TYPE: mor.get('tags')}))\n\n return external_host_tags", "response": "Returns a list of tags for every host that is detected by the vSphere\n integration."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_all_objs(\n self, server_instance, regexes=None, include_only_marked=False, tags=None, use_guest_hostname=False\n ):\n \"\"\"\n Explore vCenter infrastructure to discover hosts, virtual machines, etc.\n and compute their associated tags.\n Start at the vCenter `rootFolder`, so as to collect every objet.\n\n Example topology:\n ```\n rootFolder\n - datacenter1\n - compute_resource1 == cluster\n - host1\n - host2\n - host3\n - compute_resource2\n - host5\n - vm1\n - vm2\n ```\n\n If it's a node we want to query metric for, it will be enqueued at the\n instance level and will be processed by a subsequent job.\n \"\"\"\n start = time.time()\n if tags is None:\n tags = []\n obj_list = defaultdict(list)\n\n # Collect objects and their attributes\n all_objects = self._collect_mors_and_attributes(server_instance)\n\n # Add rootFolder since it is not explored by the propertyCollector\n rootFolder = server_instance.content.rootFolder\n all_objects[rootFolder] = {\"name\": rootFolder.name, \"parent\": None}\n\n for obj, properties in all_objects.items():\n instance_tags = []\n if not self._is_excluded(obj, properties, regexes, include_only_marked) and any(\n isinstance(obj, vimtype) for vimtype in RESOURCE_TYPE_METRICS\n ):\n if use_guest_hostname:\n hostname = properties.get(\"guest.hostName\", properties.get(\"name\", \"unknown\"))\n else:\n hostname = properties.get(\"name\", \"unknown\")\n if properties.get(\"parent\"):\n instance_tags += self._get_parent_tags(obj, all_objects)\n\n if isinstance(obj, vim.VirtualMachine):\n vsphere_type = 'vsphere_type:vm'\n vimtype = vim.VirtualMachine\n mor_type = \"vm\"\n power_state = properties.get(\"runtime.powerState\")\n if power_state != vim.VirtualMachinePowerState.poweredOn:\n self.log.debug(\"Skipping VM in state {}\".format(ensure_unicode(power_state)))\n continue\n host_mor = properties.get(\"runtime.host\")\n host = \"unknown\"\n if host_mor:\n host = ensure_unicode(all_objects.get(host_mor, {}).get(\"name\", \"unknown\"))\n instance_tags.append('vsphere_host:{}'.format(ensure_unicode(host)))\n elif isinstance(obj, vim.HostSystem):\n vsphere_type = 'vsphere_type:host'\n vimtype = vim.HostSystem\n mor_type = \"host\"\n elif isinstance(obj, vim.Datastore):\n vsphere_type = 'vsphere_type:datastore'\n instance_tags.append(\n 'vsphere_datastore:{}'.format(ensure_unicode(properties.get(\"name\", \"unknown\")))\n )\n hostname = None\n vimtype = vim.Datastore\n mor_type = \"datastore\"\n elif isinstance(obj, vim.Datacenter):\n vsphere_type = 'vsphere_type:datacenter'\n instance_tags.append(\n \"vsphere_datacenter:{}\".format(ensure_unicode(properties.get(\"name\", \"unknown\")))\n )\n hostname = None\n vimtype = vim.Datacenter\n mor_type = \"datacenter\"\n elif isinstance(obj, vim.ClusterComputeResource):\n vsphere_type = 'vsphere_type:cluster'\n instance_tags.append(\"vsphere_cluster:{}\".format(ensure_unicode(properties.get(\"name\", \"unknown\"))))\n hostname = None\n vimtype = vim.ClusterComputeResource\n mor_type = \"cluster\"\n else:\n vsphere_type = None\n\n if vsphere_type:\n instance_tags.append(vsphere_type)\n\n obj_list[vimtype].append(\n {\"mor_type\": mor_type, \"mor\": obj, \"hostname\": hostname, \"tags\": tags + instance_tags}\n )\n\n self.log.debug(\"All objects with attributes cached in {} seconds.\".format(time.time() - start))\n return obj_list", "response": "Get all objects from the vCenter and compute their associated tags."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _is_excluded(obj, properties, regexes, include_only_marked):\n # Host\n if isinstance(obj, vim.HostSystem):\n # Based on `host_include_only_regex`\n if regexes and regexes.get('host_include') is not None:\n match = re.search(regexes['host_include'], properties.get(\"name\", \"\"), re.IGNORECASE)\n if not match:\n return True\n\n # VirtualMachine\n elif isinstance(obj, vim.VirtualMachine):\n # Based on `vm_include_only_regex`\n if regexes and regexes.get('vm_include') is not None:\n match = re.search(regexes['vm_include'], properties.get(\"name\", \"\"), re.IGNORECASE)\n if not match:\n return True\n\n # Based on `include_only_marked`\n if include_only_marked:\n monitored = False\n for field in properties.get(\"customValue\", \"\"):\n if field.value == VM_MONITORING_FLAG:\n monitored = True\n break # we shall monitor\n if not monitored:\n return True\n\n return False", "response": "Return True if the given host or virtual machine is excluded by the user configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfill the Mor objects queue that will be asynchronously processed later. Resolve the vCenter `rootFolder` and initiate hosts and virtual machines discovery.", "response": "def _cache_morlist_raw(self, instance):\n \"\"\"\n Fill the Mor objects queue that will be asynchronously processed later.\n Resolve the vCenter `rootFolder` and initiate hosts and virtual machines\n discovery.\n \"\"\"\n i_key = self._instance_key(instance)\n self.log.debug(\"Caching the morlist for vcenter instance {}\".format(i_key))\n\n # If the queue is not completely empty, don't do anything\n for resource_type in RESOURCE_TYPE_METRICS:\n if self.mor_objects_queue.contains(i_key) and self.mor_objects_queue.size(i_key, resource_type):\n last = self.cache_config.get_last(CacheConfig.Morlist, i_key)\n self.log.debug(\n \"Skipping morlist collection: the objects queue for the \"\n \"resource type '{}' is still being processed \"\n \"(latest refresh was {}s ago)\".format(ensure_unicode(resource_type), time.time() - last)\n )\n return\n\n tags = [\"vcenter_server:{}\".format(ensure_unicode(instance.get('name')))]\n regexes = {\n 'host_include': instance.get('host_include_only_regex'),\n 'vm_include': instance.get('vm_include_only_regex'),\n }\n include_only_marked = is_affirmative(instance.get('include_only_marked', False))\n\n # Discover hosts and virtual machines\n server_instance = self._get_server_instance(instance)\n use_guest_hostname = is_affirmative(instance.get(\"use_guest_hostname\", False))\n all_objs = self._get_all_objs(\n server_instance, regexes, include_only_marked, tags, use_guest_hostname=use_guest_hostname\n )\n\n self.mor_objects_queue.fill(i_key, dict(all_objs))\n self.cache_config.set_last(CacheConfig.Morlist, i_key, time.time())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _process_mor_objects_queue_async(self, instance, mors):\n t = time.time()\n i_key = self._instance_key(instance)\n server_instance = self._get_server_instance(instance)\n perfManager = server_instance.content.perfManager\n\n # For non realtime metrics, we need to specifically ask which counters are available for which entity,\n # so we call perfManager.QueryAvailablePerfMetric for each cluster, datacenter, datastore\n # This should be okay since the number of such entities shouldn't be excessively large\n for mor in mors:\n mor_name = str(mor['mor'])\n available_metrics = {m.counterId for m in perfManager.QueryAvailablePerfMetric(entity=mor[\"mor\"])}\n try:\n self.mor_cache.set_metrics(i_key, mor_name, self._compute_needed_metrics(instance, available_metrics))\n except MorNotFoundError:\n self.log.error(\"Object '{}' is missing from the cache, skipping. \".format(ensure_unicode(mor_name)))\n continue\n\n # TEST-INSTRUMENTATION\n self.histogram(\n 'datadog.agent.vsphere.morlist_process_atomic.time', time.time() - t, tags=instance.get('tags', [])\n )", "response": "Process a batch of items popped from the objects queue by querying the availablePerfMetric for each MOR and storing them in the Mor cache."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess the Mor objects queue and fill the Mor cache with the result of the Mor objects queue.", "response": "def _process_mor_objects_queue(self, instance):\n \"\"\"\n Pops `batch_morlist_size` items from the mor objects queue and run asynchronously\n the _process_mor_objects_queue_async method to fill the Mor cache.\n \"\"\"\n i_key = self._instance_key(instance)\n self.mor_cache.init_instance(i_key)\n\n if not self.mor_objects_queue.contains(i_key):\n self.log.debug(\"Objects queue is not initialized yet for instance {}, skipping processing\".format(i_key))\n return\n\n for resource_type in RESOURCE_TYPE_METRICS:\n # Batch size can prevent querying large payloads at once if the environment is too large\n # If batch size is set to 0, process everything at once\n batch_size = self.batch_morlist_size or self.mor_objects_queue.size(i_key, resource_type)\n while self.mor_objects_queue.size(i_key, resource_type):\n mors = []\n for _ in range(batch_size):\n mor = self.mor_objects_queue.pop(i_key, resource_type)\n if mor is None:\n self.log.debug(\n \"No more objects of type '{}' left in the queue\".format(ensure_unicode(resource_type))\n )\n break\n\n mor_name = str(mor['mor'])\n mor['interval'] = REAL_TIME_INTERVAL if mor['mor_type'] in REALTIME_RESOURCES else None\n # Always update the cache to account for Mors that might have changed parent\n # in the meantime (e.g. a migrated VM).\n self.mor_cache.set_mor(i_key, mor_name, mor)\n\n # Only do this for non real-time resources i.e. datacenter, datastore and cluster\n # For hosts and VMs, we can rely on a precomputed list of metrics\n realtime_only = is_affirmative(instance.get(\"collect_realtime_only\", True))\n if mor[\"mor_type\"] not in REALTIME_RESOURCES and not realtime_only:\n mors.append(mor)\n\n # We will actually schedule jobs for non realtime resources only.\n if mors:\n self.pool.apply_async(self._process_mor_objects_queue_async, args=(instance, mors))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget all the performance counters metadata meaning name group and description...", "response": "def _cache_metrics_metadata(self, instance):\n \"\"\"\n Get all the performance counters metadata meaning name/group/description...\n from the server instance, attached with the corresponding ID\n \"\"\"\n # ## \n t = Timer()\n # ## \n\n i_key = self._instance_key(instance)\n self.metadata_cache.init_instance(i_key)\n self.log.info(\"Warming metrics metadata cache for instance {}\".format(i_key))\n server_instance = self._get_server_instance(instance)\n perfManager = server_instance.content.perfManager\n custom_tags = instance.get('tags', [])\n\n new_metadata = {}\n metric_ids = []\n # Use old behaviour with metrics to collect defined by our constants\n if self.in_compatibility_mode(instance, log_warning=True):\n for counter in perfManager.perfCounter:\n metric_name = self.format_metric_name(counter, compatibility=True)\n new_metadata[counter.key] = {'name': metric_name, 'unit': counter.unitInfo.key}\n # Build the list of metrics we will want to collect\n if instance.get(\"all_metrics\") or metric_name in BASIC_METRICS:\n metric_ids.append(vim.PerformanceManager.MetricId(counterId=counter.key, instance=\"*\"))\n else:\n collection_level = instance.get(\"collection_level\", 1)\n for counter in perfManager.QueryPerfCounterByLevel(collection_level):\n new_metadata[counter.key] = {\"name\": self.format_metric_name(counter), \"unit\": counter.unitInfo.key}\n # Build the list of metrics we will want to collect\n metric_ids.append(vim.PerformanceManager.MetricId(counterId=counter.key, instance=\"*\"))\n\n self.log.info(\"Finished metadata collection for instance {}\".format(i_key))\n # Reset metadata\n self.metadata_cache.set_metadata(i_key, new_metadata)\n self.metadata_cache.set_metric_ids(i_key, metric_ids)\n\n self.cache_config.set_last(CacheConfig.Metadata, i_key, time.time())\n\n # ## \n self.histogram('datadog.agent.vsphere.metric_metadata_collection.time', t.total(), tags=custom_tags)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving the value of a counter return the value with transformation if needed.", "response": "def _transform_value(self, instance, counter_id, value):\n \"\"\" Given the counter_id, look up for the metrics metadata to check the vsphere\n type of the counter and apply pre-reporting transformation if needed.\n \"\"\"\n i_key = self._instance_key(instance)\n try:\n metadata = self.metadata_cache.get_metadata(i_key, counter_id)\n if metadata[\"unit\"] == \"percent\":\n return float(value) / 100\n except MetadataNotFoundError:\n pass\n\n # Defaults to return the value without transformation\n return value"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _collect_metrics_async(self, instance, query_specs):\n # ## \n t = Timer()\n # ## \n i_key = self._instance_key(instance)\n server_instance = self._get_server_instance(instance)\n perfManager = server_instance.content.perfManager\n custom_tags = instance.get('tags', [])\n results = perfManager.QueryPerf(query_specs)\n if results:\n for mor_perfs in results:\n mor_name = str(mor_perfs.entity)\n try:\n mor = self.mor_cache.get_mor(i_key, mor_name)\n except MorNotFoundError:\n self.log.error(\n \"Trying to get metrics from object %s deleted from the cache, skipping. \"\n \"Consider increasing the parameter `clean_morlist_interval` to avoid that\",\n mor_name,\n )\n continue\n\n for result in mor_perfs.value:\n counter_id = result.id.counterId\n if not self.metadata_cache.contains(i_key, counter_id):\n self.log.debug(\n \"Skipping value for counter {}, because there is no metadata about it\".format(\n ensure_unicode(counter_id)\n )\n )\n continue\n\n # Metric types are absolute, delta, and rate\n metric_name = self.metadata_cache.get_metadata(i_key, result.id.counterId).get('name')\n\n if self.in_compatibility_mode(instance):\n if metric_name not in ALL_METRICS:\n self.log.debug(\"Skipping unknown `{}` metric.\".format(ensure_unicode(metric_name)))\n continue\n\n if not result.value:\n self.log.debug(\n \"Skipping `{}` metric because the value is empty\".format(ensure_unicode(metric_name))\n )\n continue\n\n instance_name = result.id.instance or \"none\"\n value = self._transform_value(instance, result.id.counterId, result.value[0])\n\n tags = ['instance:{}'.format(ensure_unicode(instance_name))]\n if not mor['hostname']: # no host tags available\n tags.extend(mor['tags'])\n\n # vsphere \"rates\" should be submitted as gauges (rate is\n # precomputed).\n self.gauge(\n \"vsphere.{}\".format(ensure_unicode(metric_name)),\n value,\n hostname=mor['hostname'],\n tags=tags + custom_tags,\n )\n\n # ## \n self.histogram('datadog.agent.vsphere.metric_colection.time', t.total(), tags=custom_tags)", "response": "This method is used to collect the metrics from the MORs in the cache. It is responsible for getting the metrics from the MORs in the cache."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncollect metrics for all MORs in the MOR list.", "response": "def collect_metrics(self, instance):\n \"\"\"\n Calls asynchronously _collect_metrics_async on all MORs, as the\n job queue is processed the Aggregator will receive the metrics.\n \"\"\"\n i_key = self._instance_key(instance)\n if not self.mor_cache.contains(i_key):\n self.log.debug(\"Not collecting metrics for instance '{}', nothing to do yet.\".format(i_key))\n return\n\n vm_count = 0\n custom_tags = instance.get('tags', [])\n tags = [\"vcenter_server:{}\".format(ensure_unicode(instance.get('name')))] + custom_tags\n\n n_mors = self.mor_cache.instance_size(i_key)\n if not n_mors:\n self.gauge('vsphere.vm.count', vm_count, tags=tags)\n self.log.debug(\"No Mor objects to process for instance '{}', skip...\".format(i_key))\n return\n\n self.log.debug(\"Collecting metrics for {} mors\".format(ensure_unicode(n_mors)))\n\n # Request metrics for several objects at once. We can limit the number of objects with batch_size\n # If batch_size is 0, process everything at once\n batch_size = self.batch_morlist_size or n_mors\n for batch in self.mor_cache.mors_batch(i_key, batch_size):\n query_specs = []\n for _, mor in iteritems(batch):\n if mor['mor_type'] == 'vm':\n vm_count += 1\n if mor['mor_type'] not in REALTIME_RESOURCES and ('metrics' not in mor or not mor['metrics']):\n continue\n\n query_spec = vim.PerformanceManager.QuerySpec()\n query_spec.entity = mor[\"mor\"]\n query_spec.intervalId = mor[\"interval\"]\n query_spec.maxSample = 1\n if mor['mor_type'] in REALTIME_RESOURCES:\n query_spec.metricId = self.metadata_cache.get_metric_ids(i_key)\n else:\n query_spec.metricId = mor[\"metrics\"]\n query_specs.append(query_spec)\n\n if query_specs:\n self.pool.apply_async(self._collect_metrics_async, args=(instance, query_specs))\n\n self.gauge('vsphere.vm.count', vm_count, tags=tags)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if the database we re targeting actually exists", "response": "def _check_db_exists(self, instance):\n \"\"\"\n Check if the database we're targeting actually exists\n If not then we won't do any checks\n This allows the same config to be installed on many servers but fail gracefully\n \"\"\"\n\n dsn, host, username, password, database, driver = self._get_access_info(instance, self.DEFAULT_DB_KEY)\n context = \"{} - {}\".format(host, database)\n if self.existing_databases is None:\n cursor = self.get_cursor(instance, None, self.DEFAULT_DATABASE)\n\n try:\n self.existing_databases = {}\n cursor.execute(DATABASE_EXISTS_QUERY)\n for row in cursor:\n self.existing_databases[row.name] = True\n\n except Exception as e:\n self.log.error(\"Failed to check if database {} exists: {}\".format(database, e))\n return False, context\n finally:\n self.close_cursor(cursor)\n\n return database in self.existing_databases, context"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _make_metric_list_to_collect(self, instance, custom_metrics):\n metrics_to_collect = []\n for name, counter_name, instance_name in self.METRICS:\n try:\n sql_type, base_name = self.get_sql_type(instance, counter_name)\n cfg = {}\n cfg['name'] = name\n cfg['counter_name'] = counter_name\n cfg['instance_name'] = instance_name\n\n metrics_to_collect.append(\n self.typed_metric(instance, cfg, DEFAULT_PERFORMANCE_TABLE, base_name, None, sql_type, None)\n )\n except SQLConnectionError:\n raise\n except Exception:\n self.log.warning(\"Can't load the metric {}, ignoring\".format(name), exc_info=True)\n continue\n\n # Load any custom metrics from conf.d/sqlserver.yaml\n for row in custom_metrics:\n db_table = row.get('table', DEFAULT_PERFORMANCE_TABLE)\n if db_table not in self.valid_tables:\n self.log.error('{} has an invalid table name: {}'.format(row['name'], db_table))\n continue\n\n if db_table == DEFAULT_PERFORMANCE_TABLE:\n user_type = row.get('type')\n if user_type is not None and user_type not in VALID_METRIC_TYPES:\n self.log.error('{} has an invalid metric type: {}'.format(row['name'], user_type))\n sql_type = None\n try:\n if user_type is None:\n sql_type, base_name = self.get_sql_type(instance, row['counter_name'])\n except Exception:\n self.log.warning(\"Can't load the metric {}, ignoring\".format(row['name']), exc_info=True)\n continue\n\n metrics_to_collect.append(\n self.typed_metric(instance, row, db_table, base_name, user_type, sql_type, None)\n )\n\n else:\n for column in row['columns']:\n metrics_to_collect.append(\n self.typed_metric(instance, row, db_table, base_name, None, sql_type, column)\n )\n\n instance_key = self._conn_key(instance, self.DEFAULT_DB_KEY)\n self.instances_metrics[instance_key] = metrics_to_collect\n simple_metrics = []\n fraction_metrics = []\n wait_stat_metrics = []\n vfs_metrics = []\n clerk_metrics = []\n self.log.debug(\"metrics to collect {}\".format(str(metrics_to_collect)))\n for m in metrics_to_collect:\n if type(m) is SqlSimpleMetric:\n self.log.debug(\"Adding simple metric %s\", m.sql_name)\n simple_metrics.append(m.sql_name)\n elif type(m) is SqlFractionMetric or type(m) is SqlIncrFractionMetric:\n self.log.debug(\"Adding fraction metric %s\", m.sql_name)\n fraction_metrics.append(m.sql_name)\n fraction_metrics.append(m.base_name)\n elif type(m) is SqlOsWaitStat:\n self.log.debug(\"Adding SqlOsWaitStat metric {}\".format(m.sql_name))\n wait_stat_metrics.append(m.sql_name)\n elif type(m) is SqlIoVirtualFileStat:\n self.log.debug(\"Adding SqlIoVirtualFileStat metric {}\".format(m.sql_name))\n vfs_metrics.append(m.sql_name)\n elif type(m) is SqlOsMemoryClerksStat:\n self.log.debug(\"Adding SqlOsMemoryClerksStat metric {}\".format(m.sql_name))\n clerk_metrics.append(m.sql_name)\n\n self.instances_per_type_metrics[instance_key][\"SqlSimpleMetric\"] = simple_metrics\n self.instances_per_type_metrics[instance_key][\"SqlFractionMetric\"] = fraction_metrics\n self.instances_per_type_metrics[instance_key][\"SqlOsWaitStat\"] = wait_stat_metrics\n self.instances_per_type_metrics[instance_key][\"SqlIoVirtualFileStat\"] = vfs_metrics\n self.instances_per_type_metrics[instance_key][\"SqlOsMemoryClerksStat\"] = clerk_metrics", "response": "Load the list of metrics to collect by instance_key."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef typed_metric(self, instance, cfg_inst, table, base_name, user_type, sql_type, column):\n '''\n Create the appropriate SqlServerMetric object, each implementing its method to\n fetch the metrics properly.\n If a `type` was specified in the config, it is used to report the value\n directly fetched from SQLServer. Otherwise, it is decided based on the\n sql_type, according to microsoft's documentation.\n '''\n if table == DEFAULT_PERFORMANCE_TABLE:\n metric_type_mapping = {\n PERF_COUNTER_BULK_COUNT: (self.rate, SqlSimpleMetric),\n PERF_COUNTER_LARGE_RAWCOUNT: (self.gauge, SqlSimpleMetric),\n PERF_LARGE_RAW_BASE: (self.gauge, SqlSimpleMetric),\n PERF_RAW_LARGE_FRACTION: (self.gauge, SqlFractionMetric),\n PERF_AVERAGE_BULK: (self.gauge, SqlIncrFractionMetric),\n }\n if user_type is not None:\n # user type overrides any other value\n metric_type = getattr(self, user_type)\n cls = SqlSimpleMetric\n\n else:\n metric_type, cls = metric_type_mapping[sql_type]\n else:\n table_type_mapping = {\n DM_OS_WAIT_STATS_TABLE: (self.gauge, SqlOsWaitStat),\n DM_OS_MEMORY_CLERKS_TABLE: (self.gauge, SqlOsMemoryClerksStat),\n DM_OS_VIRTUAL_FILE_STATS: (self.gauge, SqlIoVirtualFileStat),\n }\n metric_type, cls = table_type_mapping[table]\n\n return cls(self._get_connector(instance), cfg_inst, base_name, metric_type, column, self.log)", "response": "Create the appropriate SqlServerMetric object for a typed metric."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_access_info(self, instance, db_key, db_name=None):\n ''' Convenience method to extract info from instance\n '''\n dsn = instance.get('dsn')\n host = instance.get('host')\n username = instance.get('username')\n password = instance.get('password')\n database = instance.get(db_key) if db_name is None else db_name\n driver = instance.get('driver')\n if not dsn:\n if not host:\n host = '127.0.0.1,1433'\n if not database:\n database = self.DEFAULT_DATABASE\n if not driver:\n driver = self.DEFAULT_DRIVER\n return dsn, host, username, password, database, driver", "response": "Utility method to extract info from instance and db_key"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _conn_key(self, instance, db_key, db_name=None):\n ''' Return a key to use for the connection cache\n '''\n dsn, host, username, password, database, driver = self._get_access_info(instance, db_key, db_name)\n return '{}:{}:{}:{}:{}:{}'.format(dsn, host, username, password, database, driver)", "response": "Return a key to use for the connection cache\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _conn_string_odbc(self, db_key, instance=None, conn_key=None, db_name=None):\n ''' Return a connection string to use with odbc\n '''\n if instance:\n dsn, host, username, password, database, driver = self._get_access_info(instance, db_key, db_name)\n elif conn_key:\n dsn, host, username, password, database, driver = conn_key.split(\":\")\n\n conn_str = ''\n if dsn:\n conn_str = 'DSN={};'.format(dsn)\n\n if driver:\n conn_str += 'DRIVER={};'.format(driver)\n if host:\n conn_str += 'Server={};'.format(host)\n if database:\n conn_str += 'Database={};'.format(database)\n\n if username:\n conn_str += 'UID={};'.format(username)\n self.log.debug(\"Connection string (before password) {}\".format(conn_str))\n if password:\n conn_str += 'PWD={};'.format(password)\n return conn_str", "response": "Return a connection string to use with odbc\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _conn_string_adodbapi(self, db_key, instance=None, conn_key=None, db_name=None):\n ''' Return a connection string to use with adodbapi\n '''\n if instance:\n _, host, username, password, database, _ = self._get_access_info(instance, db_key, db_name)\n elif conn_key:\n _, host, username, password, database, _ = conn_key.split(\":\")\n\n p = self._get_adoprovider(instance)\n conn_str = 'Provider={};Data Source={};Initial Catalog={};'.format(p, host, database)\n\n if username:\n conn_str += 'User ID={};'.format(username)\n if password:\n conn_str += 'Password={};'.format(password)\n if not username and not password:\n conn_str += 'Integrated Security=SSPI;'\n return conn_str", "response": "Return a connection string to use with adodbapi"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_cursor(self, instance, db_key, db_name=None):\n '''\n Return a cursor to execute query against the db\n Cursor are cached in the self.connections dict\n '''\n conn_key = self._conn_key(instance, db_key, db_name)\n try:\n conn = self.connections[conn_key]['conn']\n except KeyError:\n # We catch KeyError to avoid leaking the auth info used to compose the key\n # FIXME: we should find a better way to compute unique keys to map opened connections other than\n # using auth info in clear text!\n raise SQLConnectionError(\"Cannot find an opened connection for host: {}\".format(instance.get('host')))\n return conn.cursor()", "response": "Return a cursor to execute query against the db\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_sql_type(self, instance, counter_name):\n '''\n Return the type of the performance counter so that we can report it to\n Datadog correctly\n If the sql_type is one that needs a base (PERF_RAW_LARGE_FRACTION and\n PERF_AVERAGE_BULK), the name of the base counter will also be returned\n '''\n with self.get_managed_cursor(instance, self.DEFAULT_DB_KEY) as cursor:\n cursor.execute(COUNTER_TYPE_QUERY, (counter_name,))\n (sql_type,) = cursor.fetchone()\n if sql_type == PERF_LARGE_RAW_BASE:\n self.log.warning(\"Metric {} is of type Base and shouldn't be reported this way\".format(counter_name))\n base_name = None\n if sql_type in [PERF_AVERAGE_BULK, PERF_RAW_LARGE_FRACTION]:\n # This is an ugly hack. For certains type of metric (PERF_RAW_LARGE_FRACTION\n # and PERF_AVERAGE_BULK), we need two metrics: the metrics specified and\n # a base metrics to get the ratio. There is no unique schema so we generate\n # the possible candidates and we look at which ones exist in the db.\n candidates = (\n counter_name + \" base\",\n counter_name.replace(\"(ms)\", \"base\"),\n counter_name.replace(\"Avg \", \"\") + \" base\",\n )\n try:\n cursor.execute(BASE_NAME_QUERY, candidates)\n base_name = cursor.fetchone().counter_name.strip()\n self.log.debug(\"Got base metric: {} for metric: {}\".format(base_name, counter_name))\n except Exception as e:\n self.log.warning(\"Could not get counter_name of base for metric: {}\".format(e))\n\n return sql_type, base_name", "response": "Get the SQL type of the performance counter and the base name of the base."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfetch the metrics from the sys. dm_os_performance_counters table.", "response": "def do_perf_counter_check(self, instance):\n \"\"\"\n Fetch the metrics from the sys.dm_os_performance_counters table\n \"\"\"\n custom_tags = instance.get('tags', [])\n if custom_tags is None:\n custom_tags = []\n instance_key = self._conn_key(instance, self.DEFAULT_DB_KEY)\n instance_by_key = self.instances_per_type_metrics[instance_key]\n with self.open_managed_db_connections(instance, self.DEFAULT_DB_KEY):\n # if the server was down at check __init__ key could be missing.\n if instance_key not in self.instances_metrics:\n self._make_metric_list_to_collect(instance, self.custom_metrics)\n metrics_to_collect = self.instances_metrics[instance_key]\n\n with self.get_managed_cursor(instance, self.DEFAULT_DB_KEY) as cursor:\n simple_rows = SqlSimpleMetric.fetch_all_values(cursor, instance_by_key[\"SqlSimpleMetric\"], self.log)\n fraction_results = SqlFractionMetric.fetch_all_values(\n cursor, instance_by_key[\"SqlFractionMetric\"], self.log\n )\n waitstat_rows, waitstat_cols = SqlOsWaitStat.fetch_all_values(\n cursor, instance_by_key[\"SqlOsWaitStat\"], self.log\n )\n vfs_rows, vfs_cols = SqlIoVirtualFileStat.fetch_all_values(\n cursor, instance_by_key[\"SqlIoVirtualFileStat\"], self.log\n )\n clerk_rows, clerk_cols = SqlOsMemoryClerksStat.fetch_all_values(\n cursor, instance_by_key[\"SqlOsMemoryClerksStat\"], self.log # noqa: E501\n )\n\n for metric in metrics_to_collect:\n try:\n if type(metric) is SqlSimpleMetric:\n metric.fetch_metric(cursor, simple_rows, custom_tags)\n elif type(metric) is SqlFractionMetric or type(metric) is SqlIncrFractionMetric:\n metric.fetch_metric(cursor, fraction_results, custom_tags)\n elif type(metric) is SqlOsWaitStat:\n metric.fetch_metric(cursor, waitstat_rows, waitstat_cols, custom_tags)\n elif type(metric) is SqlIoVirtualFileStat:\n metric.fetch_metric(cursor, vfs_rows, vfs_cols, custom_tags)\n elif type(metric) is SqlOsMemoryClerksStat:\n metric.fetch_metric(cursor, clerk_rows, clerk_cols, custom_tags)\n\n except Exception as e:\n self.log.warning(\"Could not fetch metric {} : {}\".format(metric.datadog_name, e))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall the stored procedure and checks the status of the stored procedure.", "response": "def do_stored_procedure_check(self, instance, proc):\n \"\"\"\n Fetch the metrics from the stored proc\n \"\"\"\n\n guardSql = instance.get('proc_only_if')\n custom_tags = instance.get(\"tags\", [])\n\n if (guardSql and self.proc_check_guard(instance, guardSql)) or not guardSql:\n self.open_db_connections(instance, self.DEFAULT_DB_KEY)\n cursor = self.get_cursor(instance, self.DEFAULT_DB_KEY)\n\n try:\n self.log.debug(\"Calling Stored Procedure : {}\".format(proc))\n if self._get_connector(instance) == 'adodbapi':\n cursor.callproc(proc)\n else:\n # pyodbc does not support callproc; use execute instead.\n # Reference: https://github.com/mkleehammer/pyodbc/wiki/Calling-Stored-Procedures\n call_proc = '{{CALL {}}}'.format(proc)\n cursor.execute(call_proc)\n\n rows = cursor.fetchall()\n self.log.debug(\"Row count ({}) : {}\".format(proc, cursor.rowcount))\n\n for row in rows:\n tags = [] if row.tags is None or row.tags == '' else row.tags.split(',')\n tags.extend(custom_tags)\n\n if row.type.lower() in self.proc_type_mapping:\n self.proc_type_mapping[row.type](row.metric, row.value, tags)\n else:\n self.log.warning(\n '{} is not a recognised type from procedure {}, metric {}'.format(\n row.type, proc, row.metric\n )\n )\n\n except Exception as e:\n self.log.warning(\"Could not call procedure {}: {}\".format(proc, e))\n\n self.close_cursor(cursor)\n self.close_db_connections(instance, self.DEFAULT_DB_KEY)\n else:\n self.log.info(\"Skipping call to {} due to only_if\".format(proc))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck to see if the guard SQL returns a single column containing 0 or 1", "response": "def proc_check_guard(self, instance, sql):\n \"\"\"\n check to see if the guard SQL returns a single column containing 0 or 1\n We return true if 1, else False\n \"\"\"\n self.open_db_connections(instance, self.PROC_GUARD_DB_KEY)\n cursor = self.get_cursor(instance, self.PROC_GUARD_DB_KEY)\n\n should_run = False\n try:\n cursor.execute(sql, ())\n result = cursor.fetchone()\n should_run = result[0] == 1\n except Exception as e:\n self.log.error(\"Failed to run proc_only_if sql {} : {}\".format(sql, e))\n\n self.close_cursor(cursor)\n self.close_db_connections(instance, self.PROC_GUARD_DB_KEY)\n return should_run"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef close_cursor(self, cursor):\n try:\n cursor.close()\n except Exception as e:\n self.log.warning(\"Could not close adodbapi cursor\\n{}\".format(e))", "response": "Closes the cursor explicitly b/c we had proven memory leaks\n We handle any exception from closing"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef close_db_connections(self, instance, db_key, db_name=None):\n conn_key = self._conn_key(instance, db_key, db_name)\n if conn_key not in self.connections:\n return\n\n try:\n self.connections[conn_key]['conn'].close()\n del self.connections[conn_key]\n except Exception as e:\n self.log.warning(\"Could not close adodbapi db connection\\n{0}\".format(e))", "response": "Closes the db connections for the current instance."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nopen the db connections.", "response": "def open_db_connections(self, instance, db_key, db_name=None):\n \"\"\"\n We open the db connections explicitly, so we can ensure they are open\n before we use them, and are closable, once we are finished. Open db\n connections keep locks on the db, presenting issues such as the SQL\n Server Agent being unable to stop.\n \"\"\"\n\n conn_key = self._conn_key(instance, db_key, db_name)\n timeout = int(instance.get('command_timeout', self.DEFAULT_COMMAND_TIMEOUT))\n\n dsn, host, username, password, database, driver = self._get_access_info(instance, db_key, db_name)\n custom_tags = instance.get(\"tags\", [])\n if custom_tags is None:\n custom_tags = []\n service_check_tags = ['host:{}'.format(host), 'db:{}'.format(database)]\n service_check_tags.extend(custom_tags)\n service_check_tags = list(set(service_check_tags))\n\n cs = instance.get('connection_string', '')\n cs += ';' if cs != '' else ''\n\n try:\n if self._get_connector(instance) == 'adodbapi':\n cs += self._conn_string_adodbapi(db_key, instance=instance, db_name=db_name)\n # autocommit: true disables implicit transaction\n rawconn = adodbapi.connect(cs, {'timeout': timeout, 'autocommit': True})\n else:\n cs += self._conn_string_odbc(db_key, instance=instance, db_name=db_name)\n rawconn = pyodbc.connect(cs, timeout=timeout)\n\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK, tags=service_check_tags)\n if conn_key not in self.connections:\n self.connections[conn_key] = {'conn': rawconn, 'timeout': timeout}\n else:\n try:\n # explicitly trying to avoid leaks...\n self.connections[conn_key]['conn'].close()\n except Exception as e:\n self.log.info(\"Could not close adodbapi db connection\\n{0}\".format(e))\n\n self.connections[conn_key]['conn'] = rawconn\n except Exception:\n cx = \"{} - {}\".format(host, database)\n message = \"Unable to connect to SQL Server for instance {}.\".format(cx)\n self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, tags=service_check_tags, message=message)\n\n password = instance.get('password')\n tracebk = traceback.format_exc()\n if password is not None:\n tracebk = tracebk.replace(password, \"*\" * 6)\n\n cxn_failure_exp = SQLConnectionError(\"{} \\n {}\".format(message, tracebk))\n raise cxn_failure_exp"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_generic_instances(self, instances):\n generic_instances = []\n for instance in instances:\n transformed_instance = self._create_kube_dns_instance(instance)\n generic_instances.append(transformed_instance)\n\n return generic_instances", "response": "Transform each Kube DNS instance into OpenMetricsBaseCheck instance"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates kube_dns instance from instance dict.", "response": "def _create_kube_dns_instance(self, instance):\n \"\"\"\n Set up kube_dns instance so it can be used in OpenMetricsBaseCheck\n \"\"\"\n kube_dns_instance = deepcopy(instance)\n\n # kube_dns uses 'prometheus_endpoint' and not 'prometheus_url', so we have to rename the key\n kube_dns_instance['prometheus_url'] = instance.get('prometheus_endpoint', None)\n\n kube_dns_instance.update(\n {\n 'namespace': 'kubedns',\n # Note: the count metrics were moved to specific functions list below to be submitted\n # as both gauges and monotonic_counts\n 'metrics': [\n {\n # metrics have been renamed to kubedns in kubernetes 1.6.0\n 'kubedns_kubedns_dns_response_size_bytes': 'response_size.bytes',\n 'kubedns_kubedns_dns_request_duration_seconds': 'request_duration.seconds',\n # metrics names for kubernetes < 1.6.0\n 'skydns_skydns_dns_response_size_bytes': 'response_size.bytes',\n 'skydns_skydns_dns_request_duration_seconds': 'request_duration.seconds',\n }\n ],\n # Defaults that were set when kube_dns was based on PrometheusCheck\n 'send_monotonic_counter': instance.get('send_monotonic_counter', False),\n 'health_service_check': instance.get('health_service_check', False),\n }\n )\n\n return kube_dns_instance"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef submit_as_gauge_and_monotonic_count(self, metric_suffix, metric, scraper_config):\n metric_name = scraper_config['namespace'] + metric_suffix\n for sample in metric.samples:\n # Explicit shallow copy of the instance tags\n _tags = list(scraper_config['custom_tags'])\n\n for label_name, label_value in iteritems(sample[self.SAMPLE_LABELS]):\n _tags.append('{}:{}'.format(label_name, label_value))\n # submit raw metric\n self.gauge(metric_name, sample[self.SAMPLE_VALUE], _tags)\n # submit rate metric\n self.monotonic_count(metric_name + '.count', sample[self.SAMPLE_VALUE], _tags)", "response": "Submit a kube_dns metric both as a gauge and as a monotonic_count"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new instance of this collection with the specified settings.", "response": "def with_options(\n self, codec_options=None, read_preference=None,\n write_concern=None, read_concern=None):\n \"\"\"Get a clone of this collection changing the specified settings.\n\n >>> coll1.read_preference\n Primary()\n >>> from pymongo import ReadPreference\n >>> coll2 = coll1.with_options(read_preference=ReadPreference.SECONDARY)\n >>> coll1.read_preference\n Primary()\n >>> coll2.read_preference\n Secondary(tag_sets=None)\n\n :Parameters:\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`. If ``None`` (the\n default) the :attr:`codec_options` of this :class:`Collection`\n is used.\n - `read_preference` (optional): The read preference to use. If\n ``None`` (the default) the :attr:`read_preference` of this\n :class:`Collection` is used. See :mod:`~pymongo.read_preferences`\n for options.\n - `write_concern` (optional): An instance of\n :class:`~pymongo.write_concern.WriteConcern`. If ``None`` (the\n default) the :attr:`write_concern` of this :class:`Collection`\n is used.\n - `read_concern` (optional): An instance of\n :class:`~pymongo.read_concern.ReadConcern`. If ``None`` (the\n default) the :attr:`read_concern` of this :class:`Collection`\n is used.\n \"\"\"\n return Collection(self.__database,\n self.__name,\n False,\n codec_options or self.codec_options,\n read_preference or self.read_preference,\n write_concern or self.write_concern,\n read_concern or self.read_concern)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninitialize an unordered batch of write operations.", "response": "def initialize_unordered_bulk_op(self, bypass_document_validation=False):\n \"\"\"**DEPRECATED** - Initialize an unordered batch of write operations.\n\n Operations will be performed on the server in arbitrary order,\n possibly in parallel. All operations will be attempted.\n\n :Parameters:\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n\n Returns a :class:`~pymongo.bulk.BulkOperationBuilder` instance.\n\n See :ref:`unordered_bulk` for examples.\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.5\n Deprecated. Use :meth:`~pymongo.collection.Collection.bulk_write`\n instead.\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 2.7\n \"\"\"\n warnings.warn(\"initialize_unordered_bulk_op is deprecated\",\n DeprecationWarning, stacklevel=2)\n return BulkOperationBuilder(self, False, bypass_document_validation)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef initialize_ordered_bulk_op(self, bypass_document_validation=False):\n warnings.warn(\"initialize_ordered_bulk_op is deprecated\",\n DeprecationWarning, stacklevel=2)\n return BulkOperationBuilder(self, True, bypass_document_validation)", "response": "Initialize an ordered bulk operation."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a batch of write operations to the server.", "response": "def bulk_write(self, requests, ordered=True,\n bypass_document_validation=False):\n \"\"\"Send a batch of write operations to the server.\n\n Requests are passed as a list of write operation instances (\n :class:`~pymongo.operations.InsertOne`,\n :class:`~pymongo.operations.UpdateOne`,\n :class:`~pymongo.operations.UpdateMany`,\n :class:`~pymongo.operations.ReplaceOne`,\n :class:`~pymongo.operations.DeleteOne`, or\n :class:`~pymongo.operations.DeleteMany`).\n\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'x': 1, u'_id': ObjectId('54f62e60fba5226811f634ef')}\n {u'x': 1, u'_id': ObjectId('54f62e60fba5226811f634f0')}\n >>> # DeleteMany, UpdateOne, and UpdateMany are also available.\n ...\n >>> from pymongo import InsertOne, DeleteOne, ReplaceOne\n >>> requests = [InsertOne({'y': 1}), DeleteOne({'x': 1}),\n ... ReplaceOne({'w': 1}, {'z': 1}, upsert=True)]\n >>> result = db.test.bulk_write(requests)\n >>> result.inserted_count\n 1\n >>> result.deleted_count\n 1\n >>> result.modified_count\n 0\n >>> result.upserted_ids\n {2: ObjectId('54f62ee28891e756a6e1abd5')}\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'x': 1, u'_id': ObjectId('54f62e60fba5226811f634f0')}\n {u'y': 1, u'_id': ObjectId('54f62ee2fba5226811f634f1')}\n {u'z': 1, u'_id': ObjectId('54f62ee28891e756a6e1abd5')}\n\n :Parameters:\n - `requests`: A list of write operations (see examples above).\n - `ordered` (optional): If ``True`` (the default) requests will be\n performed on the server serially, in the order provided. If an error\n occurs all remaining operations are aborted. If ``False`` requests\n will be performed on the server in arbitrary order, possibly in\n parallel, and all operations will be attempted.\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n\n :Returns:\n An instance of :class:`~pymongo.results.BulkWriteResult`.\n\n .. seealso:: :ref:`writes-and-ids`\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 3.0\n \"\"\"\n if not isinstance(requests, list):\n raise TypeError(\"requests must be a list\")\n\n blk = _Bulk(self, ordered, bypass_document_validation)\n for request in requests:\n try:\n request._add_to_bulk(blk)\n except AttributeError:\n raise TypeError(\"%r is not a valid request\" % (request,))\n\n bulk_api_result = blk.execute(self.write_concern.document)\n if bulk_api_result is not None:\n return BulkWriteResult(bulk_api_result, True)\n return BulkWriteResult({}, False)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninserting a single document into the database.", "response": "def insert_one(self, document, bypass_document_validation=False):\n \"\"\"Insert a single document.\n\n >>> db.test.count({'x': 1})\n 0\n >>> result = db.test.insert_one({'x': 1})\n >>> result.inserted_id\n ObjectId('54f112defba522406c9cc208')\n >>> db.test.find_one({'x': 1})\n {u'x': 1, u'_id': ObjectId('54f112defba522406c9cc208')}\n\n :Parameters:\n - `document`: The document to insert. Must be a mutable mapping\n type. If the document does not have an _id field one will be\n added automatically.\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n\n :Returns:\n - An instance of :class:`~pymongo.results.InsertOneResult`.\n\n .. seealso:: :ref:`writes-and-ids`\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 3.0\n \"\"\"\n common.validate_is_document_type(\"document\", document)\n if not (isinstance(document, RawBSONDocument) or \"_id\" in document):\n document[\"_id\"] = ObjectId()\n with self._socket_for_writes() as sock_info:\n return InsertOneResult(\n self._insert(sock_info, document,\n bypass_doc_val=bypass_document_validation),\n self.write_concern.acknowledged)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninsert a list of documents into the database.", "response": "def insert_many(self, documents, ordered=True,\n bypass_document_validation=False):\n \"\"\"Insert an iterable of documents.\n\n >>> db.test.count()\n 0\n >>> result = db.test.insert_many([{'x': i} for i in range(2)])\n >>> result.inserted_ids\n [ObjectId('54f113fffba522406c9cc20e'), ObjectId('54f113fffba522406c9cc20f')]\n >>> db.test.count()\n 2\n\n :Parameters:\n - `documents`: A iterable of documents to insert.\n - `ordered` (optional): If ``True`` (the default) documents will be\n inserted on the server serially, in the order provided. If an error\n occurs all remaining inserts are aborted. If ``False``, documents\n will be inserted on the server in arbitrary order, possibly in\n parallel, and all document inserts will be attempted.\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n\n :Returns:\n An instance of :class:`~pymongo.results.InsertManyResult`.\n\n .. seealso:: :ref:`writes-and-ids`\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 3.0\n \"\"\"\n if not isinstance(documents, collections.Iterable) or not documents:\n raise TypeError(\"documents must be a non-empty list\")\n inserted_ids = []\n def gen():\n \"\"\"A generator that validates documents and handles _ids.\"\"\"\n for document in documents:\n common.validate_is_document_type(\"document\", document)\n if not isinstance(document, RawBSONDocument):\n if \"_id\" not in document:\n document[\"_id\"] = ObjectId()\n inserted_ids.append(document[\"_id\"])\n yield (message._INSERT, document)\n\n blk = _Bulk(self, ordered, bypass_document_validation)\n blk.ops = [doc for doc in gen()]\n blk.execute(self.write_concern.document)\n return InsertManyResult(inserted_ids, self.write_concern.acknowledged)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreplace a single document matching the filter.", "response": "def replace_one(self, filter, replacement, upsert=False,\n bypass_document_validation=False, collation=None):\n \"\"\"Replace a single document matching the filter.\n\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'x': 1, u'_id': ObjectId('54f4c5befba5220aa4d6dee7')}\n >>> result = db.test.replace_one({'x': 1}, {'y': 1})\n >>> result.matched_count\n 1\n >>> result.modified_count\n 1\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'y': 1, u'_id': ObjectId('54f4c5befba5220aa4d6dee7')}\n\n The *upsert* option can be used to insert a new document if a matching\n document does not exist.\n\n >>> result = db.test.replace_one({'x': 1}, {'x': 1}, True)\n >>> result.matched_count\n 0\n >>> result.modified_count\n 0\n >>> result.upserted_id\n ObjectId('54f11e5c8891e756a6e1abd4')\n >>> db.test.find_one({'x': 1})\n {u'x': 1, u'_id': ObjectId('54f11e5c8891e756a6e1abd4')}\n\n :Parameters:\n - `filter`: A query that matches the document to replace.\n - `replacement`: The new document.\n - `upsert` (optional): If ``True``, perform an insert if no documents\n match the filter.\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`. This option is only supported\n on MongoDB 3.4 and above.\n\n :Returns:\n - An instance of :class:`~pymongo.results.UpdateResult`.\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 3.0\n \"\"\"\n common.validate_is_mapping(\"filter\", filter)\n common.validate_ok_for_replace(replacement)\n with self._socket_for_writes() as sock_info:\n result = self._update(sock_info, filter, replacement, upsert,\n bypass_doc_val=bypass_document_validation,\n collation=collation)\n return UpdateResult(result, self.write_concern.acknowledged)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates a single document matching the filter.", "response": "def update_one(self, filter, update, upsert=False,\n bypass_document_validation=False,\n collation=None):\n \"\"\"Update a single document matching the filter.\n\n >>> for doc in db.test.find():\n ... print(doc)\n ...\n {u'x': 1, u'_id': 0}\n {u'x': 1, u'_id': 1}\n {u'x': 1, u'_id': 2}\n >>> result = db.test.update_one({'x': 1}, {'$inc': {'x': 3}})\n >>> result.matched_count\n 1\n >>> result.modified_count\n 1\n >>> for doc in db.test.find():\n ... print(doc)\n ...\n {u'x': 4, u'_id': 0}\n {u'x': 1, u'_id': 1}\n {u'x': 1, u'_id': 2}\n\n :Parameters:\n - `filter`: A query that matches the document to update.\n - `update`: The modifications to apply.\n - `upsert` (optional): If ``True``, perform an insert if no documents\n match the filter.\n - `bypass_document_validation`: (optional) If ``True``, allows the\n write to opt-out of document level validation. Default is\n ``False``.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`. This option is only supported\n on MongoDB 3.4 and above.\n\n :Returns:\n - An instance of :class:`~pymongo.results.UpdateResult`.\n\n .. note:: `bypass_document_validation` requires server version\n **>= 3.2**\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n\n .. versionchanged:: 3.2\n Added bypass_document_validation support\n\n .. versionadded:: 3.0\n \"\"\"\n common.validate_is_mapping(\"filter\", filter)\n common.validate_ok_for_update(update)\n with self._socket_for_writes() as sock_info:\n result = self._update(sock_info, filter, update, upsert,\n check_keys=False,\n bypass_doc_val=bypass_document_validation,\n collation=collation)\n return UpdateResult(result, self.write_concern.acknowledged)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef delete_one(self, filter, collation=None):\n with self._socket_for_writes() as sock_info:\n return DeleteResult(self._delete(sock_info, filter, False,\n collation=collation),\n self.write_concern.acknowledged)", "response": "Delete a single document matching the filter."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete_many(self, filter, collation=None):\n with self._socket_for_writes() as sock_info:\n return DeleteResult(self._delete(sock_info, filter, True,\n collation=collation),\n self.write_concern.acknowledged)", "response": "Delete one or more documents matching the filter."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef find_one(self, filter=None, *args, **kwargs):\n if (filter is not None and not\n isinstance(filter, collections.Mapping)):\n filter = {\"_id\": filter}\n\n cursor = self.find(filter, *args, **kwargs)\n for result in cursor.limit(-1):\n return result\n return None", "response": "Return a single document from the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nscanning this entire collection in parallel. Returns a list of up to ``num_cursors`` cursors that can be iterated concurrently. As long as the collection is not modified during scanning, each document appears once in one of the cursors result sets. For example, to process each document in a collection using some thread-safe ``process_document()`` function: >>> def process_cursor(cursor): ... for document in cursor: ... # Some thread-safe processing function: ... process_document(document) >>> >>> # Get up to 4 cursors. ... >>> cursors = collection.parallel_scan(4) >>> threads = [ ... threading.Thread(target=process_cursor, args=(cursor,)) ... for cursor in cursors] >>> >>> for thread in threads: ... thread.start() >>> >>> for thread in threads: ... thread.join() >>> >>> # All documents have now been processed. The :meth:`parallel_scan` method obeys the :attr:`read_preference` of this :class:`Collection`. :Parameters: - `num_cursors`: the number of cursors to return - `**kwargs`: additional options for the parallelCollectionScan command can be passed as keyword arguments. .. note:: Requires server version **>= 2.5.5**. .. versionchanged:: 3.4 Added back support for arbitrary keyword arguments. MongoDB 3.4 adds support for maxTimeMS as an option to the parallelCollectionScan command. .. versionchanged:: 3.0 Removed support for arbitrary keyword arguments, since the parallelCollectionScan command has no optional arguments.", "response": "def parallel_scan(self, num_cursors, **kwargs):\n \"\"\"Scan this entire collection in parallel.\n\n Returns a list of up to ``num_cursors`` cursors that can be iterated\n concurrently. As long as the collection is not modified during\n scanning, each document appears once in one of the cursors result\n sets.\n\n For example, to process each document in a collection using some\n thread-safe ``process_document()`` function:\n\n >>> def process_cursor(cursor):\n ... for document in cursor:\n ... # Some thread-safe processing function:\n ... process_document(document)\n >>>\n >>> # Get up to 4 cursors.\n ...\n >>> cursors = collection.parallel_scan(4)\n >>> threads = [\n ... threading.Thread(target=process_cursor, args=(cursor,))\n ... for cursor in cursors]\n >>>\n >>> for thread in threads:\n ... thread.start()\n >>>\n >>> for thread in threads:\n ... thread.join()\n >>>\n >>> # All documents have now been processed.\n\n The :meth:`parallel_scan` method obeys the :attr:`read_preference` of\n this :class:`Collection`.\n\n :Parameters:\n - `num_cursors`: the number of cursors to return\n - `**kwargs`: additional options for the parallelCollectionScan\n command can be passed as keyword arguments.\n\n .. note:: Requires server version **>= 2.5.5**.\n\n .. versionchanged:: 3.4\n Added back support for arbitrary keyword arguments. MongoDB 3.4\n adds support for maxTimeMS as an option to the\n parallelCollectionScan command.\n\n .. versionchanged:: 3.0\n Removed support for arbitrary keyword arguments, since\n the parallelCollectionScan command has no optional arguments.\n \"\"\"\n cmd = SON([('parallelCollectionScan', self.__name),\n ('numCursors', num_cursors)])\n cmd.update(kwargs)\n\n with self._socket_for_reads() as (sock_info, slave_ok):\n result = self._command(sock_info, cmd, slave_ok,\n read_concern=self.read_concern)\n\n return [CommandCursor(self, cursor['cursor'], sock_info.address)\n for cursor in result['cursors']]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef count(self, filter=None, **kwargs):\n cmd = SON([(\"count\", self.__name)])\n if filter is not None:\n if \"query\" in kwargs:\n raise ConfigurationError(\"can't pass both filter and query\")\n kwargs[\"query\"] = filter\n if \"hint\" in kwargs and not isinstance(kwargs[\"hint\"], string_type):\n kwargs[\"hint\"] = helpers._index_document(kwargs[\"hint\"])\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n cmd.update(kwargs)\n return self._count(cmd, collation)", "response": "Returns the number of documents in this collection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate one or more indexes on this collection.", "response": "def create_indexes(self, indexes):\n \"\"\"Create one or more indexes on this collection.\n\n >>> from pymongo import IndexModel, ASCENDING, DESCENDING\n >>> index1 = IndexModel([(\"hello\", DESCENDING),\n ... (\"world\", ASCENDING)], name=\"hello_world\")\n >>> index2 = IndexModel([(\"goodbye\", DESCENDING)])\n >>> db.test.create_indexes([index1, index2])\n [\"hello_world\", \"goodbye_-1\"]\n\n :Parameters:\n - `indexes`: A list of :class:`~pymongo.operations.IndexModel`\n instances.\n\n .. note:: `create_indexes` uses the `createIndexes`_ command\n introduced in MongoDB **2.6** and cannot be used with earlier\n versions.\n\n .. note:: The :attr:`~pymongo.collection.Collection.write_concern` of\n this collection is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this collection's write concern automatically to this operation\n when connected to MongoDB >= 3.4.\n .. versionadded:: 3.0\n\n .. _createIndexes: https://docs.mongodb.com/manual/reference/command/createIndexes/\n \"\"\"\n if not isinstance(indexes, list):\n raise TypeError(\"indexes must be a list\")\n names = []\n def gen_indexes():\n for index in indexes:\n if not isinstance(index, IndexModel):\n raise TypeError(\"%r is not an instance of \"\n \"pymongo.operations.IndexModel\" % (index,))\n document = index.document\n names.append(document[\"name\"])\n yield document\n cmd = SON([('createIndexes', self.name),\n ('indexes', list(gen_indexes()))])\n with self._socket_for_writes() as sock_info:\n self._command(\n sock_info, cmd, read_preference=ReadPreference.PRIMARY,\n codec_options=_UNICODE_REPLACE_CODEC_OPTIONS,\n write_concern=self.write_concern,\n parse_write_concern_error=True)\n return names"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate an index on the collection.", "response": "def create_index(self, keys, **kwargs):\n \"\"\"Creates an index on this collection.\n\n Takes either a single key or a list of (key, direction) pairs.\n The key(s) must be an instance of :class:`basestring`\n (:class:`str` in python 3), and the direction(s) must be one of\n (:data:`~pymongo.ASCENDING`, :data:`~pymongo.DESCENDING`,\n :data:`~pymongo.GEO2D`, :data:`~pymongo.GEOHAYSTACK`,\n :data:`~pymongo.GEOSPHERE`, :data:`~pymongo.HASHED`,\n :data:`~pymongo.TEXT`).\n\n To create a single key ascending index on the key ``'mike'`` we just\n use a string argument::\n\n >>> my_collection.create_index(\"mike\")\n\n For a compound index on ``'mike'`` descending and ``'eliot'``\n ascending we need to use a list of tuples::\n\n >>> my_collection.create_index([(\"mike\", pymongo.DESCENDING),\n ... (\"eliot\", pymongo.ASCENDING)])\n\n All optional index creation parameters should be passed as\n keyword arguments to this method. For example::\n\n >>> my_collection.create_index([(\"mike\", pymongo.DESCENDING)],\n ... background=True)\n\n Valid options include, but are not limited to:\n\n - `name`: custom name to use for this index - if none is\n given, a name will be generated.\n - `unique`: if ``True`` creates a uniqueness constraint on the index.\n - `background`: if ``True`` this index should be created in the\n background.\n - `sparse`: if ``True``, omit from the index any documents that lack\n the indexed field.\n - `bucketSize`: for use with geoHaystack indexes.\n Number of documents to group together within a certain proximity\n to a given longitude and latitude.\n - `min`: minimum value for keys in a :data:`~pymongo.GEO2D`\n index.\n - `max`: maximum value for keys in a :data:`~pymongo.GEO2D`\n index.\n - `expireAfterSeconds`: Used to create an expiring (TTL)\n collection. MongoDB will automatically delete documents from\n this collection after seconds. The indexed field must\n be a UTC datetime or the data will not expire.\n - `partialFilterExpression`: A document that specifies a filter for\n a partial index.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`. This option is only supported\n on MongoDB 3.4 and above.\n\n See the MongoDB documentation for a full list of supported options by\n server version.\n\n .. warning:: `dropDups` is not supported by MongoDB 3.0 or newer. The\n option is silently ignored by the server and unique index builds\n using the option will fail if a duplicate value is detected.\n\n .. note:: `partialFilterExpression` requires server version **>= 3.2**\n\n .. note:: The :attr:`~pymongo.collection.Collection.write_concern` of\n this collection is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n :Parameters:\n - `keys`: a single key or a list of (key, direction)\n pairs specifying the index to create\n - `**kwargs` (optional): any additional index creation\n options (see the above list) should be passed as keyword\n arguments\n\n .. versionchanged:: 3.4\n Apply this collection's write concern automatically to this operation\n when connected to MongoDB >= 3.4. Support the `collation` option.\n .. versionchanged:: 3.2\n Added partialFilterExpression to support partial indexes.\n .. versionchanged:: 3.0\n Renamed `key_or_list` to `keys`. Removed the `cache_for` option.\n :meth:`create_index` no longer caches index names. Removed support\n for the drop_dups and bucket_size aliases.\n\n .. mongodoc:: indexes\n \"\"\"\n keys = helpers._index_list(keys)\n name = kwargs.setdefault(\"name\", helpers._gen_index_name(keys))\n self.__create_index(keys, kwargs)\n return name"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nensures that an index exists on this object.", "response": "def ensure_index(self, key_or_list, cache_for=300, **kwargs):\n \"\"\"**DEPRECATED** - Ensures that an index exists on this collection.\n\n .. versionchanged:: 3.0\n **DEPRECATED**\n \"\"\"\n warnings.warn(\"ensure_index is deprecated. Use create_index instead.\",\n DeprecationWarning, stacklevel=2)\n # The types supported by datetime.timedelta.\n if not (isinstance(cache_for, integer_types) or\n isinstance(cache_for, float)):\n raise TypeError(\"cache_for must be an integer or float.\")\n\n if \"drop_dups\" in kwargs:\n kwargs[\"dropDups\"] = kwargs.pop(\"drop_dups\")\n\n if \"bucket_size\" in kwargs:\n kwargs[\"bucketSize\"] = kwargs.pop(\"bucket_size\")\n\n keys = helpers._index_list(key_or_list)\n name = kwargs.setdefault(\"name\", helpers._gen_index_name(keys))\n\n # Note that there is a race condition here. One thread could\n # check if the index is cached and be preempted before creating\n # and caching the index. This means multiple threads attempting\n # to create the same index concurrently could send the index\n # to the server two or more times. This has no practical impact\n # other than wasted round trips.\n if not self.__database.client._cached(self.__database.name,\n self.__name, name):\n self.__create_index(keys, kwargs)\n self.__database.client._cache_index(self.__database.name,\n self.__name, name, cache_for)\n return name\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef drop_indexes(self):\n self.__database.client._purge_index(self.__database.name, self.__name)\n self.drop_index(\"*\")", "response": "Drops all indexes on this collection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndrop the specified index on this collection.", "response": "def drop_index(self, index_or_name):\n \"\"\"Drops the specified index on this collection.\n\n Can be used on non-existant collections or collections with no\n indexes. Raises OperationFailure on an error (e.g. trying to\n drop an index that does not exist). `index_or_name`\n can be either an index name (as returned by `create_index`),\n or an index specifier (as passed to `create_index`). An index\n specifier should be a list of (key, direction) pairs. Raises\n TypeError if index is not an instance of (str, unicode, list).\n\n .. warning::\n\n if a custom name was used on index creation (by\n passing the `name` parameter to :meth:`create_index` or\n :meth:`ensure_index`) the index **must** be dropped by name.\n\n :Parameters:\n - `index_or_name`: index (or name of index) to drop\n\n .. note:: The :attr:`~pymongo.collection.Collection.write_concern` of\n this collection is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this collection's write concern automatically to this operation\n when connected to MongoDB >= 3.4.\n\n \"\"\"\n name = index_or_name\n if isinstance(index_or_name, list):\n name = helpers._gen_index_name(index_or_name)\n\n if not isinstance(name, string_type):\n raise TypeError(\"index_or_name must be an index name or list\")\n\n self.__database.client._purge_index(\n self.__database.name, self.__name, name)\n cmd = SON([(\"dropIndexes\", self.__name), (\"index\", name)])\n with self._socket_for_writes() as sock_info:\n self._command(sock_info,\n cmd,\n read_preference=ReadPreference.PRIMARY,\n allowable_errors=[\"ns not found\"],\n write_concern=self.write_concern,\n parse_write_concern_error=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reindex(self):\n cmd = SON([(\"reIndex\", self.__name)])\n with self._socket_for_writes() as sock_info:\n return self._command(\n sock_info, cmd, read_preference=ReadPreference.PRIMARY,\n parse_write_concern_error=True)", "response": "Rebuilds all indexes on this collection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a cursor over the index documents for this collection.", "response": "def list_indexes(self):\n \"\"\"Get a cursor over the index documents for this collection.\n\n >>> for index in db.test.list_indexes():\n ... print(index)\n ...\n SON([(u'v', 1), (u'key', SON([(u'_id', 1)])),\n (u'name', u'_id_'), (u'ns', u'test.test')])\n\n :Returns:\n An instance of :class:`~pymongo.command_cursor.CommandCursor`.\n\n .. versionadded:: 3.0\n \"\"\"\n codec_options = CodecOptions(SON)\n coll = self.with_options(codec_options)\n with self._socket_for_primary_reads() as (sock_info, slave_ok):\n cmd = SON([(\"listIndexes\", self.__name), (\"cursor\", {})])\n if sock_info.max_wire_version > 2:\n try:\n cursor = self._command(sock_info, cmd, slave_ok,\n ReadPreference.PRIMARY,\n codec_options)[\"cursor\"]\n except OperationFailure as exc:\n # Ignore NamespaceNotFound errors to match the behavior\n # of reading from *.system.indexes.\n if exc.code != 26:\n raise\n cursor = {'id': 0, 'firstBatch': []}\n return CommandCursor(coll, cursor, sock_info.address)\n else:\n namespace = _UJOIN % (self.__database.name, \"system.indexes\")\n res = helpers._first_batch(\n sock_info, self.__database.name, \"system.indexes\",\n {\"ns\": self.__full_name}, 0, slave_ok, codec_options,\n ReadPreference.PRIMARY, cmd,\n self.database.client._event_listeners)\n data = res[\"data\"]\n cursor = {\n \"id\": res[\"cursor_id\"],\n \"firstBatch\": data,\n \"ns\": namespace,\n }\n # Note that a collection can only have 64 indexes, so we don't\n # technically have to pass len(data) here. There will never be\n # an OP_GET_MORE call.\n return CommandCursor(\n coll, cursor, sock_info.address, len(data))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef options(self):\n with self._socket_for_primary_reads() as (sock_info, slave_ok):\n if sock_info.max_wire_version > 2:\n criteria = {\"name\": self.__name}\n else:\n criteria = {\"name\": self.__full_name}\n cursor = self.__database._list_collections(sock_info,\n slave_ok,\n criteria)\n\n result = None\n for doc in cursor:\n result = doc\n break\n\n if not result:\n return {}\n\n options = result.get(\"options\", {})\n if \"create\" in options:\n del options[\"create\"]\n\n return options", "response": "Get the options set on this collection."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef aggregate(self, pipeline, **kwargs):\n if not isinstance(pipeline, list):\n raise TypeError(\"pipeline must be a list\")\n\n if \"explain\" in kwargs:\n raise ConfigurationError(\"The explain option is not supported. \"\n \"Use Database.command instead.\")\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n\n cmd = SON([(\"aggregate\", self.__name),\n (\"pipeline\", pipeline)])\n\n # Remove things that are not command options.\n batch_size = common.validate_positive_integer_or_none(\n \"batchSize\", kwargs.pop(\"batchSize\", None))\n use_cursor = common.validate_boolean(\n \"useCursor\", kwargs.pop(\"useCursor\", True))\n # If the server does not support the \"cursor\" option we\n # ignore useCursor and batchSize.\n with self._socket_for_reads() as (sock_info, slave_ok):\n if sock_info.max_wire_version > 0:\n if use_cursor:\n if \"cursor\" not in kwargs:\n kwargs[\"cursor\"] = {}\n if batch_size is not None:\n kwargs[\"cursor\"][\"batchSize\"] = batch_size\n\n dollar_out = pipeline and '$out' in pipeline[-1]\n if (sock_info.max_wire_version >= 5 and dollar_out and\n self.write_concern):\n cmd['writeConcern'] = self.write_concern.document\n\n cmd.update(kwargs)\n\n # Apply this Collection's read concern if $out is not in the\n # pipeline.\n if sock_info.max_wire_version >= 4 and 'readConcern' not in cmd:\n if dollar_out:\n result = self._command(sock_info, cmd, slave_ok,\n parse_write_concern_error=True,\n collation=collation)\n else:\n result = self._command(sock_info, cmd, slave_ok,\n read_concern=self.read_concern,\n collation=collation)\n else:\n result = self._command(sock_info, cmd, slave_ok,\n parse_write_concern_error=dollar_out,\n collation=collation)\n\n if \"cursor\" in result:\n cursor = result[\"cursor\"]\n else:\n # Pre-MongoDB 2.6. Fake a cursor.\n cursor = {\n \"id\": 0,\n \"firstBatch\": result[\"result\"],\n \"ns\": self.full_name,\n }\n return CommandCursor(\n self, cursor, sock_info.address).batch_size(batch_size or 0)", "response": "Perform an aggregate operation on the specified collection."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rename(self, new_name, **kwargs):\n if not isinstance(new_name, string_type):\n raise TypeError(\"new_name must be an \"\n \"instance of %s\" % (string_type.__name__,))\n\n if not new_name or \"..\" in new_name:\n raise InvalidName(\"collection names cannot be empty\")\n if new_name[0] == \".\" or new_name[-1] == \".\":\n raise InvalidName(\"collecion names must not start or end with '.'\")\n if \"$\" in new_name and not new_name.startswith(\"oplog.$main\"):\n raise InvalidName(\"collection names must not contain '$'\")\n\n new_name = \"%s.%s\" % (self.__database.name, new_name)\n cmd = SON([(\"renameCollection\", self.__full_name), (\"to\", new_name)])\n with self._socket_for_writes() as sock_info:\n if sock_info.max_wire_version >= 5 and self.write_concern:\n cmd['writeConcern'] = self.write_concern.document\n cmd.update(kwargs)\n sock_info.command('admin', cmd, parse_write_concern_error=True)", "response": "Rename this collection.\n\n If operating in auth mode, client must be authorized as an\n admin to perform this operation. Raises :class:`TypeError` if\n `new_name` is not an instance of :class:`basestring`\n (:class:`str` in python 3). Raises :class:`~pymongo.errors.InvalidName`\n if `new_name` is not a valid collection name.\n\n :Parameters:\n - `new_name`: new name for this collection\n - `**kwargs` (optional): additional arguments to the rename command\n may be passed as keyword arguments to this helper method\n (i.e. ``dropTarget=True``)\n\n .. note:: The :attr:`~pymongo.collection.Collection.write_concern` of\n this collection is automatically applied to this operation when using\n MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this collection's write concern automatically to this operation\n when connected to MongoDB >= 3.4."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a list of distinct values for key among all documents in this collection.", "response": "def distinct(self, key, filter=None, **kwargs):\n \"\"\"Get a list of distinct values for `key` among all documents\n in this collection.\n\n Raises :class:`TypeError` if `key` is not an instance of\n :class:`basestring` (:class:`str` in python 3).\n\n All optional distinct parameters should be passed as keyword arguments\n to this method. Valid options include:\n\n - `maxTimeMS` (int): The maximum amount of time to allow the count\n command to run, in milliseconds.\n - `collation` (optional): An instance of\n :class:`~pymongo.collation.Collation`. This option is only supported\n on MongoDB 3.4 and above.\n\n The :meth:`distinct` method obeys the :attr:`read_preference` of\n this :class:`Collection`.\n\n :Parameters:\n - `key`: name of the field for which we want to get the distinct\n values\n - `filter` (optional): A query document that specifies the documents\n from which to retrieve the distinct values.\n - `**kwargs` (optional): See list of options above.\n\n .. versionchanged:: 3.4\n Support the `collation` option.\n\n \"\"\"\n if not isinstance(key, string_type):\n raise TypeError(\"key must be an \"\n \"instance of %s\" % (string_type.__name__,))\n cmd = SON([(\"distinct\", self.__name),\n (\"key\", key)])\n if filter is not None:\n if \"query\" in kwargs:\n raise ConfigurationError(\"can't pass both filter and query\")\n kwargs[\"query\"] = filter\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n cmd.update(kwargs)\n with self._socket_for_reads() as (sock_info, slave_ok):\n return self._command(sock_info, cmd, slave_ok,\n read_concern=self.read_concern,\n collation=collation)[\"values\"]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nperform a mapReduce operation on this collection.", "response": "def map_reduce(self, map, reduce, out, full_response=False, **kwargs):\n \"\"\"Perform a map/reduce operation on this collection.\n\n If `full_response` is ``False`` (default) returns a\n :class:`~pymongo.collection.Collection` instance containing\n the results of the operation. Otherwise, returns the full\n response from the server to the `map reduce command`_.\n\n :Parameters:\n - `map`: map function (as a JavaScript string)\n - `reduce`: reduce function (as a JavaScript string)\n - `out`: output collection name or `out object` (dict). See\n the `map reduce command`_ documentation for available options.\n Note: `out` options are order sensitive. :class:`~bson.son.SON`\n can be used to specify multiple options.\n e.g. SON([('replace', ), ('db', )])\n - `full_response` (optional): if ``True``, return full response to\n this command - otherwise just return the result collection\n - `**kwargs` (optional): additional arguments to the\n `map reduce command`_ may be passed as keyword arguments to this\n helper method, e.g.::\n\n >>> db.test.map_reduce(map, reduce, \"myresults\", limit=2)\n\n .. note:: The :meth:`map_reduce` method does **not** obey the\n :attr:`read_preference` of this :class:`Collection`. To run\n mapReduce on a secondary use the :meth:`inline_map_reduce` method\n instead.\n\n .. note:: The :attr:`~pymongo.collection.Collection.write_concern` of\n this collection is automatically applied to this operation (if the\n output is not inline) when using MongoDB >= 3.4.\n\n .. versionchanged:: 3.4\n Apply this collection's write concern automatically to this operation\n when connected to MongoDB >= 3.4.\n\n .. seealso:: :doc:`/examples/aggregation`\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n .. versionchanged:: 2.2\n Removed deprecated arguments: merge_output and reduce_output\n\n .. _map reduce command: http://docs.mongodb.org/manual/reference/command/mapReduce/\n\n .. mongodoc:: mapreduce\n\n \"\"\"\n if not isinstance(out, (string_type, collections.Mapping)):\n raise TypeError(\"'out' must be an instance of \"\n \"%s or a mapping\" % (string_type.__name__,))\n\n cmd = SON([(\"mapreduce\", self.__name),\n (\"map\", map),\n (\"reduce\", reduce),\n (\"out\", out)])\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n cmd.update(kwargs)\n\n inline = 'inline' in cmd['out']\n with self._socket_for_primary_reads() as (sock_info, slave_ok):\n if (sock_info.max_wire_version >= 5 and self.write_concern and\n not inline):\n cmd['writeConcern'] = self.write_concern.document\n cmd.update(kwargs)\n if (sock_info.max_wire_version >= 4 and 'readConcern' not in cmd and\n inline):\n # No need to parse 'writeConcernError' here, since the command\n # is an inline map reduce.\n response = self._command(\n sock_info, cmd, slave_ok, ReadPreference.PRIMARY,\n read_concern=self.read_concern,\n collation=collation)\n else:\n response = self._command(\n sock_info, cmd, slave_ok, ReadPreference.PRIMARY,\n parse_write_concern_error=not inline,\n collation=collation)\n\n if full_response or not response.get('result'):\n return response\n elif isinstance(response['result'], dict):\n dbase = response['result']['db']\n coll = response['result']['collection']\n return self.__database.client[dbase][coll]\n else:\n return self.__database[response[\"result\"]]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform an inline map reduce operation on this collection.", "response": "def inline_map_reduce(self, map, reduce, full_response=False, **kwargs):\n \"\"\"Perform an inline map/reduce operation on this collection.\n\n Perform the map/reduce operation on the server in RAM. A result\n collection is not created. The result set is returned as a list\n of documents.\n\n If `full_response` is ``False`` (default) returns the\n result documents in a list. Otherwise, returns the full\n response from the server to the `map reduce command`_.\n\n The :meth:`inline_map_reduce` method obeys the :attr:`read_preference`\n of this :class:`Collection`.\n\n :Parameters:\n - `map`: map function (as a JavaScript string)\n - `reduce`: reduce function (as a JavaScript string)\n - `full_response` (optional): if ``True``, return full response to\n this command - otherwise just return the result collection\n - `**kwargs` (optional): additional arguments to the\n `map reduce command`_ may be passed as keyword arguments to this\n helper method, e.g.::\n\n >>> db.test.inline_map_reduce(map, reduce, limit=2)\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n\n \"\"\"\n cmd = SON([(\"mapreduce\", self.__name),\n (\"map\", map),\n (\"reduce\", reduce),\n (\"out\", {\"inline\": 1})])\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n cmd.update(kwargs)\n with self._socket_for_reads() as (sock_info, slave_ok):\n if sock_info.max_wire_version >= 4 and 'readConcern' not in cmd:\n res = self._command(sock_info, cmd, slave_ok,\n read_concern=self.read_concern,\n collation=collation)\n else:\n res = self._command(sock_info, cmd, slave_ok,\n collation=collation)\n\n if full_response:\n return res\n else:\n return res.get(\"results\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_one_and_replace(self, filter, replacement,\n projection=None, sort=None, upsert=False,\n return_document=ReturnDocument.BEFORE, **kwargs):\n \"\"\"Finds a single document and replaces it, returning either the\n original or the replaced document.\n\n The :meth:`find_one_and_replace` method differs from\n :meth:`find_one_and_update` by replacing the document matched by\n *filter*, rather than modifying the existing document.\n\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'x': 1, u'_id': 0}\n {u'x': 1, u'_id': 1}\n {u'x': 1, u'_id': 2}\n >>> db.test.find_one_and_replace({'x': 1}, {'y': 1})\n {u'x': 1, u'_id': 0}\n >>> for doc in db.test.find({}):\n ... print(doc)\n ...\n {u'y': 1, u'_id': 0}\n {u'x': 1, u'_id': 1}\n {u'x': 1, u'_id': 2}\n\n :Parameters:\n - `filter`: A query that matches the document to replace.\n - `replacement`: The replacement document.\n - `projection` (optional): A list of field names that should be\n returned in the result document or a mapping specifying the fields\n to include or exclude. If `projection` is a list \"_id\" will\n always be returned. Use a mapping to exclude fields from\n the result (e.g. projection={'_id': False}).\n - `sort` (optional): a list of (key, direction) pairs\n specifying the sort order for the query. If multiple documents\n match the query, they are sorted and the first is replaced.\n - `upsert` (optional): When ``True``, inserts a new document if no\n document matches the query. Defaults to ``False``.\n - `return_document`: If\n :attr:`ReturnDocument.BEFORE` (the default),\n returns the original document before it was replaced, or ``None``\n if no document matches. If\n :attr:`ReturnDocument.AFTER`, returns the replaced\n or inserted document.\n - `**kwargs` (optional): additional command arguments can be passed\n as keyword arguments (for example maxTimeMS can be used with\n recent server versions).\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n .. versionchanged:: 3.2\n Respects write concern.\n\n .. warning:: Starting in PyMongo 3.2, this command uses the\n :class:`~pymongo.write_concern.WriteConcern` of this\n :class:`~pymongo.collection.Collection` when connected to MongoDB >=\n 3.2. Note that using an elevated write concern with this command may\n be slower compared to using the default write concern.\n\n .. versionadded:: 3.0\n \"\"\"\n common.validate_ok_for_replace(replacement)\n kwargs['update'] = replacement\n return self.__find_and_modify(filter, projection,\n sort, upsert, return_document, **kwargs)", "response": "Find a single document and replaces it with replacement."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef find_one_and_update(self, filter, update,\n projection=None, sort=None, upsert=False,\n return_document=ReturnDocument.BEFORE, **kwargs):\n \"\"\"Finds a single document and updates it, returning either the\n original or the updated document.\n\n >>> db.test.find_one_and_update(\n ... {'_id': 665}, {'$inc': {'count': 1}, '$set': {'done': True}})\n {u'_id': 665, u'done': False, u'count': 25}}\n\n By default :meth:`find_one_and_update` returns the original version of\n the document before the update was applied. To return the updated\n version of the document instead, use the *return_document* option.\n\n >>> from pymongo import ReturnDocument\n >>> db.example.find_one_and_update(\n ... {'_id': 'userid'},\n ... {'$inc': {'seq': 1}},\n ... return_document=ReturnDocument.AFTER)\n {u'_id': u'userid', u'seq': 1}\n\n You can limit the fields returned with the *projection* option.\n\n >>> db.example.find_one_and_update(\n ... {'_id': 'userid'},\n ... {'$inc': {'seq': 1}},\n ... projection={'seq': True, '_id': False},\n ... return_document=ReturnDocument.AFTER)\n {u'seq': 2}\n\n The *upsert* option can be used to create the document if it doesn't\n already exist.\n\n >>> db.example.delete_many({}).deleted_count\n 1\n >>> db.example.find_one_and_update(\n ... {'_id': 'userid'},\n ... {'$inc': {'seq': 1}},\n ... projection={'seq': True, '_id': False},\n ... upsert=True,\n ... return_document=ReturnDocument.AFTER)\n {u'seq': 1}\n\n If multiple documents match *filter*, a *sort* can be applied.\n\n >>> for doc in db.test.find({'done': True}):\n ... print(doc)\n ...\n {u'_id': 665, u'done': True, u'result': {u'count': 26}}\n {u'_id': 701, u'done': True, u'result': {u'count': 17}}\n >>> db.test.find_one_and_update(\n ... {'done': True},\n ... {'$set': {'final': True}},\n ... sort=[('_id', pymongo.DESCENDING)])\n {u'_id': 701, u'done': True, u'result': {u'count': 17}}\n\n :Parameters:\n - `filter`: A query that matches the document to update.\n - `update`: The update operations to apply.\n - `projection` (optional): A list of field names that should be\n returned in the result document or a mapping specifying the fields\n to include or exclude. If `projection` is a list \"_id\" will\n always be returned. Use a dict to exclude fields from\n the result (e.g. projection={'_id': False}).\n - `sort` (optional): a list of (key, direction) pairs\n specifying the sort order for the query. If multiple documents\n match the query, they are sorted and the first is updated.\n - `upsert` (optional): When ``True``, inserts a new document if no\n document matches the query. Defaults to ``False``.\n - `return_document`: If\n :attr:`ReturnDocument.BEFORE` (the default),\n returns the original document before it was updated, or ``None``\n if no document matches. If\n :attr:`ReturnDocument.AFTER`, returns the updated\n or inserted document.\n - `**kwargs` (optional): additional command arguments can be passed\n as keyword arguments (for example maxTimeMS can be used with\n recent server versions).\n\n .. versionchanged:: 3.4\n Added the `collation` option.\n .. versionchanged:: 3.2\n Respects write concern.\n\n .. warning:: Starting in PyMongo 3.2, this command uses the\n :class:`~pymongo.write_concern.WriteConcern` of this\n :class:`~pymongo.collection.Collection` when connected to MongoDB >=\n 3.2. Note that using an elevated write concern with this command may\n be slower compared to using the default write concern.\n\n .. versionadded:: 3.0\n \"\"\"\n common.validate_ok_for_update(update)\n kwargs['update'] = update\n return self.__find_and_modify(filter, projection,\n sort, upsert, return_document, **kwargs)", "response": "Find a single document and update it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving a document in this collection.", "response": "def save(self, to_save, manipulate=True, check_keys=True, **kwargs):\n \"\"\"Save a document in this collection.\n\n **DEPRECATED** - Use :meth:`insert_one` or :meth:`replace_one` instead.\n\n .. versionchanged:: 3.0\n Removed the `safe` parameter. Pass ``w=0`` for unacknowledged write\n operations.\n \"\"\"\n warnings.warn(\"save is deprecated. Use insert_one or replace_one \"\n \"instead\", DeprecationWarning, stacklevel=2)\n common.validate_is_document_type(\"to_save\", to_save)\n\n write_concern = None\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n if kwargs:\n write_concern = WriteConcern(**kwargs)\n\n with self._socket_for_writes() as sock_info:\n if not (isinstance(to_save, RawBSONDocument) or \"_id\" in to_save):\n return self._insert(sock_info, to_save, True,\n check_keys, manipulate, write_concern)\n else:\n self._update(sock_info, {\"_id\": to_save[\"_id\"]}, to_save, True,\n check_keys, False, manipulate, write_concern,\n collation=collation)\n return to_save.get(\"_id\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninsert a set of documents into this collection.", "response": "def insert(self, doc_or_docs, manipulate=True,\n check_keys=True, continue_on_error=False, **kwargs):\n \"\"\"Insert a document(s) into this collection.\n\n **DEPRECATED** - Use :meth:`insert_one` or :meth:`insert_many` instead.\n\n .. versionchanged:: 3.0\n Removed the `safe` parameter. Pass ``w=0`` for unacknowledged write\n operations.\n \"\"\"\n warnings.warn(\"insert is deprecated. Use insert_one or insert_many \"\n \"instead.\", DeprecationWarning, stacklevel=2)\n write_concern = None\n if kwargs:\n write_concern = WriteConcern(**kwargs)\n with self._socket_for_writes() as sock_info:\n return self._insert(sock_info, doc_or_docs, not continue_on_error,\n check_keys, manipulate, write_concern)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates a document in this collection.", "response": "def update(self, spec, document, upsert=False, manipulate=False,\n multi=False, check_keys=True, **kwargs):\n \"\"\"Update a document(s) in this collection.\n\n **DEPRECATED** - Use :meth:`replace_one`, :meth:`update_one`, or\n :meth:`update_many` instead.\n\n .. versionchanged:: 3.0\n Removed the `safe` parameter. Pass ``w=0`` for unacknowledged write\n operations.\n \"\"\"\n warnings.warn(\"update is deprecated. Use replace_one, update_one or \"\n \"update_many instead.\", DeprecationWarning, stacklevel=2)\n common.validate_is_mapping(\"spec\", spec)\n common.validate_is_mapping(\"document\", document)\n if document:\n # If a top level key begins with '$' this is a modify operation\n # and we should skip key validation. It doesn't matter which key\n # we check here. Passing a document with a mix of top level keys\n # starting with and without a '$' is invalid and the server will\n # raise an appropriate exception.\n first = next(iter(document))\n if first.startswith('$'):\n check_keys = False\n\n write_concern = None\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n if kwargs:\n write_concern = WriteConcern(**kwargs)\n with self._socket_for_writes() as sock_info:\n return self._update(sock_info, spec, document, upsert,\n check_keys, multi, manipulate, write_concern,\n collation=collation)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove a document from this collection.", "response": "def remove(self, spec_or_id=None, multi=True, **kwargs):\n \"\"\"Remove a document(s) from this collection.\n\n **DEPRECATED** - Use :meth:`delete_one` or :meth:`delete_many` instead.\n\n .. versionchanged:: 3.0\n Removed the `safe` parameter. Pass ``w=0`` for unacknowledged write\n operations.\n \"\"\"\n warnings.warn(\"remove is deprecated. Use delete_one or delete_many \"\n \"instead.\", DeprecationWarning, stacklevel=2)\n if spec_or_id is None:\n spec_or_id = {}\n if not isinstance(spec_or_id, collections.Mapping):\n spec_or_id = {\"_id\": spec_or_id}\n write_concern = None\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n if kwargs:\n write_concern = WriteConcern(**kwargs)\n with self._socket_for_writes() as sock_info:\n return self._delete(sock_info, spec_or_id, multi, write_concern,\n collation=collation)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_and_modify(self, query={}, update=None,\n upsert=False, sort=None, full_response=False,\n manipulate=False, **kwargs):\n \"\"\"Update and return an object.\n\n **DEPRECATED** - Use :meth:`find_one_and_delete`,\n :meth:`find_one_and_replace`, or :meth:`find_one_and_update` instead.\n \"\"\"\n warnings.warn(\"find_and_modify is deprecated, use find_one_and_delete\"\n \", find_one_and_replace, or find_one_and_update instead\",\n DeprecationWarning, stacklevel=2)\n\n if not update and not kwargs.get('remove', None):\n raise ValueError(\"Must either update or remove\")\n\n if update and kwargs.get('remove', None):\n raise ValueError(\"Can't do both update and remove\")\n\n # No need to include empty args\n if query:\n kwargs['query'] = query\n if update:\n kwargs['update'] = update\n if upsert:\n kwargs['upsert'] = upsert\n if sort:\n # Accept a list of tuples to match Cursor's sort parameter.\n if isinstance(sort, list):\n kwargs['sort'] = helpers._index_document(sort)\n # Accept OrderedDict, SON, and dict with len == 1 so we\n # don't break existing code already using find_and_modify.\n elif (isinstance(sort, _ORDERED_TYPES) or\n isinstance(sort, dict) and len(sort) == 1):\n warnings.warn(\"Passing mapping types for `sort` is deprecated,\"\n \" use a list of (key, direction) pairs instead\",\n DeprecationWarning, stacklevel=2)\n kwargs['sort'] = sort\n else:\n raise TypeError(\"sort must be a list of (key, direction) \"\n \"pairs, a dict of len 1, or an instance of \"\n \"SON or OrderedDict\")\n\n fields = kwargs.pop(\"fields\", None)\n if fields is not None:\n kwargs[\"fields\"] = helpers._fields_list_to_dict(fields, \"fields\")\n\n collation = validate_collation_or_none(kwargs.pop('collation', None))\n\n cmd = SON([(\"findAndModify\", self.__name)])\n cmd.update(kwargs)\n with self._socket_for_writes() as sock_info:\n if sock_info.max_wire_version >= 4 and 'writeConcern' not in cmd:\n wc_doc = self.write_concern.document\n if wc_doc:\n cmd['writeConcern'] = wc_doc\n out = self._command(sock_info, cmd,\n read_preference=ReadPreference.PRIMARY,\n allowable_errors=[_NO_OBJ_ERROR],\n collation=collation)\n _check_write_command_response([(0, out)])\n\n if not out['ok']:\n if out[\"errmsg\"] == _NO_OBJ_ERROR:\n return None\n else:\n # Should never get here b/c of allowable_errors\n raise ValueError(\"Unexpected Error: %s\" % (out,))\n\n if full_response:\n return out\n else:\n document = out.get('value')\n if manipulate:\n document = self.__database._fix_outgoing(document, self)\n return document", "response": "Find and modify a set of items in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_native(cls, regex):\n if not isinstance(regex, RE_TYPE):\n raise TypeError(\n \"regex must be a compiled regular expression, not %s\"\n % type(regex))\n\n return Regex(regex.pattern, regex.flags)", "response": "Convert a Python regular expression into a Regex instance."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_int(data, position, dummy0, dummy1, dummy2):\n end = position + 4\n return _UNPACK_INT(data[position:end])[0], end", "response": "Decode a BSON int32 to python int."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndecoding a BSON double to python float.", "response": "def _get_float(data, position, dummy0, dummy1, dummy2):\n \"\"\"Decode a BSON double to python float.\"\"\"\n end = position + 8\n return _UNPACK_FLOAT(data[position:end])[0], end"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndecoding a BSON subdocument to opts. document_class or bson. dbref. DBRef.", "response": "def _get_object(data, position, obj_end, opts, dummy):\n \"\"\"Decode a BSON subdocument to opts.document_class or bson.dbref.DBRef.\"\"\"\n obj_size = _UNPACK_INT(data[position:position + 4])[0]\n end = position + obj_size - 1\n if data[end:position + obj_size] != b\"\\x00\":\n raise InvalidBSON(\"bad eoo\")\n if end >= obj_end:\n raise InvalidBSON(\"invalid object length\")\n if _raw_document_class(opts.document_class):\n return (opts.document_class(data[position:end + 1], opts),\n position + obj_size)\n\n obj = _elements_to_dict(data, position + 4, end, opts)\n\n position += obj_size\n if \"$ref\" in obj:\n return (DBRef(obj.pop(\"$ref\"), obj.pop(\"$id\", None),\n obj.pop(\"$db\", None), obj), position)\n return obj, position"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_array(data, position, obj_end, opts, element_name):\n size = _UNPACK_INT(data[position:position + 4])[0]\n end = position + size - 1\n if data[end:end + 1] != b\"\\x00\":\n raise InvalidBSON(\"bad eoo\")\n\n position += 4\n end -= 1\n result = []\n\n # Avoid doing global and attibute lookups in the loop.\n append = result.append\n index = data.index\n getter = _ELEMENT_GETTER\n\n while position < end:\n element_type = data[position:position + 1]\n # Just skip the keys.\n position = index(b'\\x00', position) + 1\n try:\n value, position = getter[element_type](\n data, position, obj_end, opts, element_name)\n except KeyError:\n _raise_unknown_type(element_type, element_name)\n append(value)\n\n if position != end + 1:\n raise InvalidBSON('bad array length')\n return result, position + 1", "response": "Decode a BSON array to python list."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndecode a BSON binary to bson. binary. Binary or python UUID.", "response": "def _get_binary(data, position, obj_end, opts, dummy1):\n \"\"\"Decode a BSON binary to bson.binary.Binary or python UUID.\"\"\"\n length, subtype = _UNPACK_LENGTH_SUBTYPE(data[position:position + 5])\n position += 5\n if subtype == 2:\n length2 = _UNPACK_INT(data[position:position + 4])[0]\n position += 4\n if length2 != length - 4:\n raise InvalidBSON(\"invalid binary (st 2) - lengths don't match!\")\n length = length2\n end = position + length\n if length < 0 or end > obj_end:\n raise InvalidBSON('bad binary object length')\n if subtype in (3, 4):\n # Java Legacy\n uuid_representation = opts.uuid_representation\n if uuid_representation == JAVA_LEGACY:\n java = data[position:end]\n value = uuid.UUID(bytes=java[0:8][::-1] + java[8:16][::-1])\n # C# legacy\n elif uuid_representation == CSHARP_LEGACY:\n value = uuid.UUID(bytes_le=data[position:end])\n # Python\n else:\n value = uuid.UUID(bytes=data[position:end])\n return value, end\n # Python3 special case. Decode subtype 0 to 'bytes'.\n if PY3 and subtype == 0:\n value = data[position:end]\n else:\n value = Binary(data[position:end], subtype)\n return value, end"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndecode a BSON ObjectId to bson. objectid. ObjectId.", "response": "def _get_oid(data, position, dummy0, dummy1, dummy2):\n \"\"\"Decode a BSON ObjectId to bson.objectid.ObjectId.\"\"\"\n end = position + 12\n return ObjectId(data[position:end]), end"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_boolean(data, position, dummy0, dummy1, dummy2):\n end = position + 1\n boolean_byte = data[position:end]\n if boolean_byte == b'\\x00':\n return False, end\n elif boolean_byte == b'\\x01':\n return True, end\n raise InvalidBSON('invalid boolean value: %r' % boolean_byte)", "response": "Decode a BSON boolean to python True or False."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecode a BSON datetime to python datetime. datetime.", "response": "def _get_date(data, position, dummy0, opts, dummy1):\n \"\"\"Decode a BSON datetime to python datetime.datetime.\"\"\"\n end = position + 8\n millis = _UNPACK_LONG(data[position:end])[0]\n return _millis_to_datetime(millis, opts), end"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndecode a BSON code to bson. code. Code.", "response": "def _get_code(data, position, obj_end, opts, element_name):\n \"\"\"Decode a BSON code to bson.code.Code.\"\"\"\n code, position = _get_string(data, position, obj_end, opts, element_name)\n return Code(code), position"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_code_w_scope(data, position, obj_end, opts, element_name):\n code_end = position + _UNPACK_INT(data[position:position + 4])[0]\n code, position = _get_string(\n data, position + 4, code_end, opts, element_name)\n scope, position = _get_object(data, position, code_end, opts, element_name)\n if position != code_end:\n raise InvalidBSON('scope outside of javascript code boundaries')\n return Code(code, scope), position", "response": "Decode a BSON code_w_scope to bson. code. Code."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_ref(data, position, obj_end, opts, element_name):\n collection, position = _get_string(\n data, position, obj_end, opts, element_name)\n oid, position = _get_oid(data, position, obj_end, opts, element_name)\n return DBRef(collection, oid), position", "response": "Decode a BSON DBPointer to bson. dbref. DBRef."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndecode a BSON timestamp to bson. timestamp. Timestamp.", "response": "def _get_timestamp(data, position, dummy0, dummy1, dummy2):\n \"\"\"Decode a BSON timestamp to bson.timestamp.Timestamp.\"\"\"\n end = position + 8\n inc, timestamp = _UNPACK_TIMESTAMP(data[position:end])\n return Timestamp(timestamp, inc), end"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_int64(data, position, dummy0, dummy1, dummy2):\n end = position + 8\n return Int64(_UNPACK_LONG(data[position:end])[0]), end", "response": "Decode a BSON int64 to bson. int64. Int64."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndecode a BSON decimal128 to bson. decimal128. Decimal128.", "response": "def _get_decimal128(data, position, dummy0, dummy1, dummy2):\n \"\"\"Decode a BSON decimal128 to bson.decimal128.Decimal128.\"\"\"\n end = position + 16\n return Decimal128.from_bid(data[position:end]), end"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _element_to_dict(data, position, obj_end, opts):\n element_type = data[position:position + 1]\n position += 1\n element_name, position = _get_c_string(data, position, opts)\n try:\n value, position = _ELEMENT_GETTER[element_type](data, position,\n obj_end, opts,\n element_name)\n except KeyError:\n _raise_unknown_type(element_type, element_name)\n return element_name, value, position", "response": "Decode a single key value pair."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndecoding a BSON document.", "response": "def _elements_to_dict(data, position, obj_end, opts):\n \"\"\"Decode a BSON document.\"\"\"\n result = opts.document_class()\n pos = position\n for key, value, pos in _iterate_elements(data, position, obj_end, opts):\n result[key] = value\n if pos != obj_end:\n raise InvalidBSON('bad object or element length')\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gen_list_name():\n for name in _LIST_NAMES:\n yield name\n\n counter = itertools.count(1000)\n while True:\n yield b(str(next(counter))) + b\"\\x00\"", "response": "Generate the list name for the current cache."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _make_c_string_check(string):\n if isinstance(string, bytes):\n if b\"\\x00\" in string:\n raise InvalidDocument(\"BSON keys / regex patterns must not \"\n \"contain a NUL character\")\n try:\n _utf_8_decode(string, None, True)\n return string + b\"\\x00\"\n except UnicodeError:\n raise InvalidStringData(\"strings in documents must be valid \"\n \"UTF-8: %r\" % string)\n else:\n if \"\\x00\" in string:\n raise InvalidDocument(\"BSON keys / regex patterns must not \"\n \"contain a NUL character\")\n return _utf_8_encode(string)[0] + b\"\\x00\"", "response": "Make a C string checking for embedded NUL characters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _make_c_string(string):\n if isinstance(string, bytes):\n try:\n _utf_8_decode(string, None, True)\n return string + b\"\\x00\"\n except UnicodeError:\n raise InvalidStringData(\"strings in documents must be valid \"\n \"UTF-8: %r\" % string)\n else:\n return _utf_8_encode(string)[0] + b\"\\x00\"", "response": "Make a C string."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _encode_dbref(name, value, check_keys, opts):\n buf = bytearray(b\"\\x03\" + name + b\"\\x00\\x00\\x00\\x00\")\n begin = len(buf) - 4\n\n buf += _name_value_to_bson(b\"$ref\\x00\",\n value.collection, check_keys, opts)\n buf += _name_value_to_bson(b\"$id\\x00\",\n value.id, check_keys, opts)\n if value.database is not None:\n buf += _name_value_to_bson(\n b\"$db\\x00\", value.database, check_keys, opts)\n for key, val in iteritems(value._DBRef__kwargs):\n buf += _element_to_bson(key, val, check_keys, opts)\n\n buf += b\"\\x00\"\n buf[begin:begin + 4] = _PACK_INT(len(buf) - begin)\n return bytes(buf)", "response": "Encode a bson. dbref. DBRef."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nencode a text field.", "response": "def _encode_text(name, value, dummy0, dummy1):\n \"\"\"Encode a python unicode (python 2.x) / str (python 3.x).\"\"\"\n value = _utf_8_encode(value)[0]\n return b\"\\x02\" + name + _PACK_INT(len(value) + 1) + value + b\"\\x00\""} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _encode_binary(name, value, dummy0, dummy1):\n subtype = value.subtype\n if subtype == 2:\n value = _PACK_INT(len(value)) + value\n return b\"\\x05\" + name + _PACK_LENGTH_SUBTYPE(len(value), subtype) + value", "response": "Encode bson. binary. Binary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _encode_bool(name, value, dummy0, dummy1):\n return b\"\\x08\" + name + (value and b\"\\x01\" or b\"\\x00\")", "response": "Encode a python boolean."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _encode_regex(name, value, dummy0, dummy1):\n flags = value.flags\n # Python 2 common case\n if flags == 0:\n return b\"\\x0B\" + name + _make_c_string_check(value.pattern) + b\"\\x00\"\n # Python 3 common case\n elif flags == re.UNICODE:\n return b\"\\x0B\" + name + _make_c_string_check(value.pattern) + b\"u\\x00\"\n else:\n sflags = b\"\"\n if flags & re.IGNORECASE:\n sflags += b\"i\"\n if flags & re.LOCALE:\n sflags += b\"l\"\n if flags & re.MULTILINE:\n sflags += b\"m\"\n if flags & re.DOTALL:\n sflags += b\"s\"\n if flags & re.UNICODE:\n sflags += b\"u\"\n if flags & re.VERBOSE:\n sflags += b\"x\"\n sflags += b\"\\x00\"\n return b\"\\x0B\" + name + _make_c_string_check(value.pattern) + sflags", "response": "Encode a python regex or bson. regex. Regex."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _encode_int(name, value, dummy0, dummy1):\n if -2147483648 <= value <= 2147483647:\n return b\"\\x10\" + name + _PACK_INT(value)\n else:\n try:\n return b\"\\x12\" + name + _PACK_LONG(value)\n except struct.error:\n raise OverflowError(\"BSON can only handle up to 8-byte ints\")", "response": "Encode a python int."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nencode bson. timestamp. Timestamp.", "response": "def _encode_timestamp(name, value, dummy0, dummy1):\n \"\"\"Encode bson.timestamp.Timestamp.\"\"\"\n return b\"\\x11\" + name + _PACK_TIMESTAMP(value.inc, value.time)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nencode a python long.", "response": "def _encode_long(name, value, dummy0, dummy1):\n \"\"\"Encode a python long (python 2.x)\"\"\"\n try:\n return b\"\\x12\" + name + _PACK_LONG(value)\n except struct.error:\n raise OverflowError(\"BSON can only handle up to 8-byte ints\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _name_value_to_bson(name, value, check_keys, opts):\n\n # First see if the type is already cached. KeyError will only ever\n # happen once per subtype.\n try:\n return _ENCODERS[type(value)](name, value, check_keys, opts)\n except KeyError:\n pass\n\n # Second, fall back to trying _type_marker. This has to be done\n # before the loop below since users could subclass one of our\n # custom types that subclasses a python built-in (e.g. Binary)\n marker = getattr(value, \"_type_marker\", None)\n if isinstance(marker, int) and marker in _MARKERS:\n func = _MARKERS[marker]\n # Cache this type for faster subsequent lookup.\n _ENCODERS[type(value)] = func\n return func(name, value, check_keys, opts)\n\n # If all else fails test each base type. This will only happen once for\n # a subtype of a supported base type.\n for base in _ENCODERS:\n if isinstance(value, base):\n func = _ENCODERS[base]\n # Cache this type for faster subsequent lookup.\n _ENCODERS[type(value)] = func\n return func(name, value, check_keys, opts)\n\n raise InvalidDocument(\"cannot convert value of type %s to bson\" %\n type(value))", "response": "Encode a single name value pair."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nencodes a single key value pair.", "response": "def _element_to_bson(key, value, check_keys, opts):\n \"\"\"Encode a single key, value pair.\"\"\"\n if not isinstance(key, string_type):\n raise InvalidDocument(\"documents must have only string keys, \"\n \"key was %r\" % (key,))\n if check_keys:\n if key.startswith(\"$\"):\n raise InvalidDocument(\"key %r must not start with '$'\" % (key,))\n if \".\" in key:\n raise InvalidDocument(\"key %r must not contain '.'\" % (key,))\n\n name = _make_name(key)\n return _name_value_to_bson(name, value, check_keys, opts)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nencodes a dictionary to BSON.", "response": "def _dict_to_bson(doc, check_keys, opts, top_level=True):\n \"\"\"Encode a document to BSON.\"\"\"\n if _raw_document_class(doc):\n return doc.raw\n try:\n elements = []\n if top_level and \"_id\" in doc:\n elements.append(_name_value_to_bson(b\"_id\\x00\", doc[\"_id\"],\n check_keys, opts))\n for (key, value) in iteritems(doc):\n if not top_level or key != \"_id\":\n elements.append(_element_to_bson(key, value,\n check_keys, opts))\n except AttributeError:\n raise TypeError(\"encoder expected a mapping type but got: %r\" % (doc,))\n\n encoded = b\"\".join(elements)\n return _PACK_INT(len(encoded) + 5) + encoded + b\"\\x00\""} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _millis_to_datetime(millis, opts):\n diff = ((millis % 1000) + 1000) % 1000\n seconds = (millis - diff) / 1000\n micros = diff * 1000\n if opts.tz_aware:\n dt = EPOCH_AWARE + datetime.timedelta(seconds=seconds,\n microseconds=micros)\n if opts.tzinfo:\n dt = dt.astimezone(opts.tzinfo)\n return dt\n else:\n return EPOCH_NAIVE + datetime.timedelta(seconds=seconds,\n microseconds=micros)", "response": "Convert milliseconds since epoch UTC to datetime."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef decode_all(data, codec_options=DEFAULT_CODEC_OPTIONS):\n if not isinstance(codec_options, CodecOptions):\n raise _CODEC_OPTIONS_TYPE_ERROR\n\n docs = []\n position = 0\n end = len(data) - 1\n use_raw = _raw_document_class(codec_options.document_class)\n try:\n while position < end:\n obj_size = _UNPACK_INT(data[position:position + 4])[0]\n if len(data) - position < obj_size:\n raise InvalidBSON(\"invalid object size\")\n obj_end = position + obj_size - 1\n if data[obj_end:position + obj_size] != b\"\\x00\":\n raise InvalidBSON(\"bad eoo\")\n if use_raw:\n docs.append(\n codec_options.document_class(\n data[position:obj_end + 1], codec_options))\n else:\n docs.append(_elements_to_dict(data,\n position + 4,\n obj_end,\n codec_options))\n position += obj_size\n return docs\n except InvalidBSON:\n raise\n except Exception:\n # Change exception type to InvalidBSON but preserve traceback.\n _, exc_value, exc_tb = sys.exc_info()\n reraise(InvalidBSON, exc_value, exc_tb)", "response": "Decode all documents in a BSON string into a list of documents."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef decode_iter(data, codec_options=DEFAULT_CODEC_OPTIONS):\n if not isinstance(codec_options, CodecOptions):\n raise _CODEC_OPTIONS_TYPE_ERROR\n\n position = 0\n end = len(data) - 1\n while position < end:\n obj_size = _UNPACK_INT(data[position:position + 4])[0]\n elements = data[position:position + obj_size]\n position += obj_size\n\n yield _bson_to_dict(elements, codec_options)", "response": "Decode a BSON string into a generator of _BSONDicts."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decode_file_iter(file_obj, codec_options=DEFAULT_CODEC_OPTIONS):\n while True:\n # Read size of next object.\n size_data = file_obj.read(4)\n if len(size_data) == 0:\n break # Finished with file normaly.\n elif len(size_data) != 4:\n raise InvalidBSON(\"cut off in middle of objsize\")\n obj_size = _UNPACK_INT(size_data)[0] - 4\n elements = size_data + file_obj.read(obj_size)\n yield _bson_to_dict(elements, codec_options)", "response": "Decode a file into multiple documents as a generator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking that the given string represents valid BSON data.", "response": "def is_valid(bson):\n \"\"\"Check that the given string represents valid :class:`BSON` data.\n\n Raises :class:`TypeError` if `bson` is not an instance of\n :class:`str` (:class:`bytes` in python 3). Returns ``True``\n if `bson` is valid :class:`BSON`, ``False`` otherwise.\n\n :Parameters:\n - `bson`: the data to be validated\n \"\"\"\n if not isinstance(bson, bytes):\n raise TypeError(\"BSON data must be an instance of a subclass of bytes\")\n\n try:\n _bson_to_dict(bson, DEFAULT_CODEC_OPTIONS)\n return True\n except Exception:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nencoding a dictionary into a new BSON object.", "response": "def encode(cls, document, check_keys=False,\n codec_options=DEFAULT_CODEC_OPTIONS):\n \"\"\"Encode a document to a new :class:`BSON` instance.\n\n A document can be any mapping type (like :class:`dict`).\n\n Raises :class:`TypeError` if `document` is not a mapping type,\n or contains keys that are not instances of\n :class:`basestring` (:class:`str` in python 3). Raises\n :class:`~bson.errors.InvalidDocument` if `document` cannot be\n converted to :class:`BSON`.\n\n :Parameters:\n - `document`: mapping type representing a document\n - `check_keys` (optional): check if keys start with '$' or\n contain '.', raising :class:`~bson.errors.InvalidDocument` in\n either case\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`.\n\n .. versionchanged:: 3.0\n Replaced `uuid_subtype` option with `codec_options`.\n \"\"\"\n if not isinstance(codec_options, CodecOptions):\n raise _CODEC_OPTIONS_TYPE_ERROR\n\n return cls(_dict_to_bson(document, check_keys, codec_options))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecodes this BSON data.", "response": "def decode(self, codec_options=DEFAULT_CODEC_OPTIONS):\n \"\"\"Decode this BSON data.\n\n By default, returns a BSON document represented as a Python\n :class:`dict`. To use a different :class:`MutableMapping` class,\n configure a :class:`~bson.codec_options.CodecOptions`::\n\n >>> import collections # From Python standard library.\n >>> import bson\n >>> from bson.codec_options import CodecOptions\n >>> data = bson.BSON.encode({'a': 1})\n >>> decoded_doc = bson.BSON.decode(data)\n \n >>> options = CodecOptions(document_class=collections.OrderedDict)\n >>> decoded_doc = bson.BSON.decode(data, codec_options=options)\n >>> type(decoded_doc)\n \n\n :Parameters:\n - `codec_options` (optional): An instance of\n :class:`~bson.codec_options.CodecOptions`.\n\n .. versionchanged:: 3.0\n Removed `compile_re` option: PyMongo now always represents BSON\n regular expressions as :class:`~bson.regex.Regex` objects. Use\n :meth:`~bson.regex.Regex.try_compile` to attempt to convert from a\n BSON regular expression to a Python regular expression object.\n\n Replaced `as_class`, `tz_aware`, and `uuid_subtype` options with\n `codec_options`.\n\n .. versionchanged:: 2.7\n Added `compile_re` option. If set to False, PyMongo represented BSON\n regular expressions as :class:`~bson.regex.Regex` objects instead of\n attempting to compile BSON regular expressions as Python native\n regular expressions, thus preventing errors for some incompatible\n patterns, see `PYTHON-500`_.\n\n .. _PYTHON-500: https://jira.mongodb.org/browse/PYTHON-500\n \"\"\"\n if not isinstance(codec_options, CodecOptions):\n raise _CODEC_OPTIONS_TYPE_ERROR\n\n return _bson_to_dict(self, codec_options)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the output of a command in a random order.", "response": "def subprocess_output(command, raise_on_empty_output):\n \"\"\"\n This is a stub to allow a check requiring `Popen` to run without an Agent (e.g. during tests or development),\n it's not supposed to be used anywhere outside the `datadog_checks.utils` package.\n \"\"\"\n\n # Use tempfile, allowing a larger amount of memory. The subprocess.Popen\n # docs warn that the data read is buffered in memory. They suggest not to\n # use subprocess.PIPE if the data size is large or unlimited.\n with tempfile.TemporaryFile() as stdout_f, tempfile.TemporaryFile() as stderr_f:\n proc = subprocess.Popen(command, stdout=stdout_f, stderr=stderr_f)\n proc.wait()\n stderr_f.seek(0)\n err = stderr_f.read()\n stdout_f.seek(0)\n output = stdout_f.read()\n\n if not output and raise_on_empty_output:\n raise SubprocessOutputEmptyError(\"get_subprocess_output expected output but had none.\")\n\n return output, err, proc.returncode"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _index_list(key_or_list, direction=None):\n if direction is not None:\n return [(key_or_list, direction)]\n else:\n if isinstance(key_or_list, string_type):\n return [(key_or_list, ASCENDING)]\n elif not isinstance(key_or_list, (list, tuple)):\n raise TypeError(\"if no direction is specified, \"\n \"key_or_list must be an instance of list\")\n return key_or_list", "response": "Helper to generate a list of ( key direction ) pairs."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _unpack_response(response,\n cursor_id=None,\n codec_options=_UNICODE_REPLACE_CODEC_OPTIONS):\n \"\"\"Unpack a response from the database.\n\n Check the response for errors and unpack, returning a dictionary\n containing the response data.\n\n Can raise CursorNotFound, NotMasterError, ExecutionTimeout, or\n OperationFailure.\n\n :Parameters:\n - `response`: byte string as returned from the database\n - `cursor_id` (optional): cursor_id we sent to get this response -\n used for raising an informative exception when we get cursor id not\n valid at server response\n - `codec_options` (optional): an instance of\n :class:`~bson.codec_options.CodecOptions`\n \"\"\"\n response_flag = struct.unpack(\" 1:\n self.log.warning(\"Several rows corresponding while the metric is supposed to be a scalar\")\n continue\n val = result[0][1]\n metric_tags = tags\n if metric.get('metric_tags'):\n metric_tags = metric_tags + metric.get('metric_tags')\n self.submit_metric(name, val, forced_type, metric_tags)\n elif 'OID' in metric:\n pass # This one is already handled by the other batch of requests\n else:\n raise Exception('Unsupported metric in config file: {}'.format(metric))", "response": "Given a list of metrics and a list of tags gather the tags requested in the instance conf for each row and submit the results to the aggregator."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_index_tags(self, index, results, index_tags, column_tags):\n '''\n Gather the tags for this row of the table (index) based on the\n results (all the results from the query).\n index_tags and column_tags are the tags to gather.\n - Those specified in index_tags contain the tag_group name and the\n index of the value we want to extract from the index tuple.\n cf. 1 for ipVersion in the IP-MIB::ipSystemStatsTable for example\n - Those specified in column_tags contain the name of a column, which\n could be a potential result, to use as a tage\n cf. ifDescr in the IF-MIB::ifTable for example\n '''\n tags = []\n for idx_tag in index_tags:\n tag_group = idx_tag[0]\n try:\n tag_value = index[idx_tag[1] - 1].prettyPrint()\n except IndexError:\n self.log.warning(\"Not enough indexes, skipping this tag\")\n continue\n tags.append(\"{}:{}\".format(tag_group, tag_value))\n for col_tag in column_tags:\n tag_group = col_tag[0]\n try:\n tag_value = results[col_tag[1]][index]\n except KeyError:\n self.log.warning(\"Column %s not present in the table, skipping this tag\", col_tag[1])\n continue\n if reply_invalid(tag_value):\n self.log.warning(\"Can't deduct tag from column for tag %s\", tag_group)\n continue\n tag_value = tag_value.prettyPrint()\n tags.append(\"{}:{}\".format(tag_group, tag_value))\n return tags", "response": "Gather the tags for this row based on the index and the column of the results."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef submit_metric(self, name, snmp_value, forced_type, tags=None):\n '''\n Convert the values reported as pysnmp-Managed Objects to values and\n report them to the aggregator\n '''\n tags = [] if tags is None else tags\n if reply_invalid(snmp_value):\n # Metrics not present in the queried object\n self.log.warning(\"No such Mib available: {}\".format(name))\n return\n\n metric_name = self.normalize(name, prefix=\"snmp\")\n\n if forced_type:\n if forced_type.lower() == \"gauge\":\n value = int(snmp_value)\n self.gauge(metric_name, value, tags)\n elif forced_type.lower() == \"counter\":\n value = int(snmp_value)\n self.rate(metric_name, value, tags)\n else:\n self.warning(\"Invalid forced-type specified: {} in {}\".format(forced_type, name))\n raise Exception(\"Invalid forced-type in config file: {}\".format(name))\n\n return\n\n # Ugly hack but couldn't find a cleaner way\n # Proper way would be to use the ASN1 method isSameTypeWith but it\n # wrongfully returns True in the case of CounterBasedGauge64\n # and Counter64 for example\n snmp_class = snmp_value.__class__.__name__\n if snmp_class in SNMP_COUNTERS:\n value = int(snmp_value)\n self.rate(metric_name, value, tags)\n return\n if snmp_class in SNMP_GAUGES:\n value = int(snmp_value)\n self.gauge(metric_name, value, tags)\n return\n\n self.log.warning(\"Unsupported metric type %s for %s\", snmp_class, metric_name)", "response": "Submit a metric to the aggregator."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_trace(self, frame=None):\n self.reset()\n\n if frame is None:\n frame = sys._getframe().f_back\n\n while frame:\n frame.f_trace = self.trace_dispatch\n self.botframe = frame\n frame = frame.f_back\n\n # Automatically proceed to next break point\n self.set_continue()\n\n sys.settrace(self.trace_dispatch)", "response": "Set the trace of the current thread."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncollecting stats for all reachable networks", "response": "def collect_networks_metrics(self, tags, network_ids, exclude_network_id_rules):\n \"\"\"\n Collect stats for all reachable networks\n \"\"\"\n networks = self.get_networks()\n filtered_networks = []\n if not network_ids:\n # Filter out excluded networks\n filtered_networks = [\n network\n for network in networks\n if not any([re.match(exclude_id, network.get('id')) for exclude_id in exclude_network_id_rules])\n ]\n else:\n for network in networks:\n if network.get('id') in network_ids:\n filtered_networks.append(network)\n\n for network in filtered_networks:\n network_id = network.get('id')\n service_check_tags = ['network:{}'.format(network_id)] + tags\n\n network_name = network.get('name')\n if network_name:\n service_check_tags.append('network_name:{}'.format(network_name))\n\n tenant_id = network.get('tenant_id')\n if tenant_id:\n service_check_tags.append('tenant_id:{}'.format(tenant_id))\n\n if network.get('admin_state_up'):\n self.service_check(self.NETWORK_SC, AgentCheck.OK, tags=service_check_tags)\n else:\n self.service_check(self.NETWORK_SC, AgentCheck.CRITICAL, tags=service_check_tags)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_uptime_string(self, uptime):\n uptime = uptime.strip()\n load_averages = uptime[uptime.find('load average:') :].split(':')[1].strip().split(',')\n load_averages = [float(load_avg) for load_avg in load_averages]\n return load_averages", "response": "Parse the uptime string into a list of load averages."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncollecting metrics for all hypervisors in the control plane.", "response": "def collect_hypervisors_metrics(\n self,\n servers,\n custom_tags=None,\n use_shortname=False,\n collect_hypervisor_metrics=True,\n collect_hypervisor_load=False,\n ):\n \"\"\"\n Submits stats for all hypervisors registered to this control plane\n Raises specific exceptions based on response code\n \"\"\"\n # Create a dictionary with hypervisor hostname as key and the list of project names as value\n hyp_project_names = defaultdict(set)\n for server in itervalues(servers):\n hypervisor_hostname = server.get('hypervisor_hostname')\n if not hypervisor_hostname:\n self.log.debug(\n \"hypervisor_hostname is None for server %s. Check that your user is an administrative users.\",\n server['server_id'],\n )\n else:\n hyp_project_names[hypervisor_hostname].add(server['project_name'])\n\n hypervisors = self.get_os_hypervisors_detail()\n for hyp in hypervisors:\n self.get_stats_for_single_hypervisor(\n hyp,\n hyp_project_names,\n custom_tags=custom_tags,\n use_shortname=use_shortname,\n collect_hypervisor_metrics=collect_hypervisor_metrics,\n collect_hypervisor_load=collect_hypervisor_load,\n )\n if not hypervisors:\n self.warning(\"Unable to collect any hypervisors from Nova response.\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef init_api(self, instance_config, custom_tags):\n custom_tags = custom_tags or []\n keystone_server_url = instance_config.get(\"keystone_server_url\")\n proxy_config = self.get_instance_proxy(instance_config, keystone_server_url)\n\n if self._api is None:\n # We are missing the entire instance scope either because it is the first time we initialize it or because\n # authentication previously failed and got removed from the cache\n # Let's populate it now\n try:\n self.log.debug(\"Fetch scope for instance {}\".format(self.instance_name))\n # Set keystone api with proper token\n self._api = ApiFactory.create(self.log, proxy_config, instance_config)\n self.service_check(\n self.IDENTITY_API_SC,\n AgentCheck.OK,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n except KeystoneUnreachable as e:\n self.warning(\n \"The agent could not contact the specified identity server at {} . \"\n \"Are you sure it is up at that address?\".format(keystone_server_url)\n )\n self.log.debug(\"Problem grabbing auth token: %s\", e)\n self.service_check(\n self.IDENTITY_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n\n # If Keystone is down/unreachable, we default the\n # Nova and Neutron APIs to UNKNOWN since we cannot access the service catalog\n self.service_check(\n self.NETWORK_API_SC,\n AgentCheck.UNKNOWN,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n self.service_check(\n self.COMPUTE_API_SC,\n AgentCheck.UNKNOWN,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n\n except MissingNovaEndpoint as e:\n self.warning(\"The agent could not find a compatible Nova endpoint in your service catalog!\")\n self.log.debug(\"Failed to get nova endpoint for response catalog: %s\", e)\n self.service_check(\n self.COMPUTE_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n\n except MissingNeutronEndpoint:\n self.warning(\"The agent could not find a compatible Neutron endpoint in your service catalog!\")\n self.service_check(\n self.NETWORK_API_SC,\n AgentCheck.CRITICAL,\n tags=[\"keystone_server: {}\".format(keystone_server_url)] + custom_tags,\n )\n\n if self._api is None:\n # Fast fail in the absence of an api\n raise IncompleteConfig()", "response": "Initializes the API for the current instance"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef detect_cadvisor(kubelet_url, cadvisor_port):\n if cadvisor_port == 0:\n raise ValueError(\"cAdvisor port set to 0 in configuration\")\n kubelet_hostname = urlparse(kubelet_url).hostname\n if not kubelet_hostname:\n raise ValueError(\"kubelet hostname empty\")\n url = \"http://{}:{}{}\".format(kubelet_hostname, cadvisor_port, LEGACY_CADVISOR_METRICS_PATH)\n\n # Test the endpoint is present\n r = requests.head(url, timeout=1)\n r.raise_for_status()\n\n return url", "response": "Attempts to connect to the cadvisor endpoint with given params\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nscrapes and submit metrics from a cadvisor instance.", "response": "def process_cadvisor(self, instance, cadvisor_url, pod_list, pod_list_utils):\n \"\"\"\n Scrape and submit metrics from cadvisor\n :param: instance: check instance object\n :param: cadvisor_url: valid cadvisor url, as returned by detect_cadvisor()\n :param: pod_list: fresh podlist object from the kubelet\n :param: pod_list_utils: already initialised PodListUtils object\n \"\"\"\n self.max_depth = instance.get('max_depth', DEFAULT_MAX_DEPTH)\n enabled_gauges = instance.get('enabled_gauges', DEFAULT_ENABLED_GAUGES)\n self.enabled_gauges = [\"{0}.{1}\".format(NAMESPACE, x) for x in enabled_gauges]\n enabled_rates = instance.get('enabled_rates', DEFAULT_ENABLED_RATES)\n self.enabled_rates = [\"{0}.{1}\".format(NAMESPACE, x) for x in enabled_rates]\n pod_level_metrics = instance.get('pod_level_metrics', DEFAULT_POD_LEVEL_METRICS)\n self.pod_level_metrics = [\"{0}.{1}\".format(NAMESPACE, x) for x in pod_level_metrics]\n\n self._update_metrics(instance, cadvisor_url, pod_list, pod_list_utils)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _publish_raw_metrics(self, metric, dat, tags, is_pod, depth=0):\n if depth >= self.max_depth:\n self.log.warning('Reached max depth on metric=%s' % metric)\n return\n\n if isinstance(dat, numbers.Number):\n # Pod level metric filtering\n is_pod_metric = False\n if self.pod_level_metrics and any([fnmatch(metric, pat) for pat in self.pod_level_metrics]):\n is_pod_metric = True\n if is_pod_metric != is_pod:\n return\n\n # Metric submission\n if self.enabled_rates and any([fnmatch(metric, pat) for pat in self.enabled_rates]):\n self.rate(metric, float(dat), tags)\n elif self.enabled_gauges and any([fnmatch(metric, pat) for pat in self.enabled_gauges]):\n self.gauge(metric, float(dat), tags)\n\n elif isinstance(dat, dict):\n for k, v in iteritems(dat):\n self._publish_raw_metrics(metric + '.%s' % k.lower(), v, tags, is_pod, depth + 1)\n\n elif isinstance(dat, list):\n self._publish_raw_metrics(metric, dat[-1], tags, is_pod, depth + 1)", "response": "Recusively parses and submit metrics for a given entity."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stats_for_version(version):\n metrics = dict(STATS_METRICS)\n\n # JVM additional metrics\n if version >= [0, 90, 10]:\n metrics.update(JVM_METRICS_POST_0_90_10)\n else:\n metrics.update(JVM_METRICS_PRE_0_90_10)\n\n # Additional Stats metrics\n if version >= [0, 90, 5]:\n metrics.update(ADDITIONAL_METRICS_POST_0_90_5)\n else:\n metrics.update(ADDITIONAL_METRICS_PRE_0_90_5)\n\n if version >= [1, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_1_0_0)\n\n if version < [2, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_PRE_2_0)\n if version >= [0, 90, 5]:\n metrics.update(ADDITIONAL_METRICS_POST_0_90_5_PRE_2_0)\n if version >= [1, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_1_x)\n\n if version >= [1, 3, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_1_3_0)\n\n if version >= [1, 4, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_1_4_0)\n\n if version >= [1, 5, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_1_5_0)\n\n if version >= [1, 6, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_1_6_0)\n\n if version >= [2, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_2_0)\n\n if version >= [2, 1, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_2_1)\n\n if version >= [5, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_5_x)\n\n if version < [5, 0, 0]:\n metrics.update(ADDITIONAL_METRICS_PRE_5_0_0)\n\n if version >= [6, 3, 0]:\n metrics.update(ADDITIONAL_METRICS_POST_6_3)\n else:\n metrics.update(ADDITIONAL_METRICS_PRE_6_3)\n\n return metrics", "response": "Returns the proper set of stats metrics for the specified ES version"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pshard_stats_for_version(version):\n pshard_stats_metrics = dict(PRIMARY_SHARD_METRICS)\n if version >= [1, 0, 0]:\n pshard_stats_metrics.update(PRIMARY_SHARD_METRICS_POST_1_0)\n\n return pshard_stats_metrics", "response": "Get the proper set of pshard metrics for the specified ES version"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the proper set of health metrics for the specified ES version", "response": "def health_stats_for_version(version):\n \"\"\"\n Get the proper set of health metrics for the specified ES version\n \"\"\"\n cluster_health_metrics = dict(CLUSTER_HEALTH_METRICS)\n if version >= [2, 4, 0]:\n cluster_health_metrics.update(CLUSTER_HEALTH_METRICS_POST_2_4)\n\n return cluster_health_metrics"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ready(ctx, quiet):\n user_config = ctx.obj\n cached_prs = {}\n\n for target in sorted(get_valid_checks()):\n # get the name of the current release tag\n cur_version = get_version_string(target)\n target_tag = get_release_tag_string(target, cur_version)\n\n # get the diff from HEAD\n diff_lines = get_commits_since(target, target_tag)\n\n # get the number of PRs that could be potentially released\n # Only show the ones that have a changelog label that isn't no-changelog\n pr_numbers = parse_pr_numbers(diff_lines)\n\n shippable_prs = 0\n for pr_num in pr_numbers:\n try:\n if pr_num in cached_prs:\n changelog_labels = cached_prs[pr_num]\n if len(changelog_labels) != 1:\n continue\n else:\n payload = get_pr(pr_num, user_config)\n changelog_labels = get_changelog_types(payload)\n cached_prs[pr_num] = changelog_labels\n\n if not changelog_labels:\n echo_warning(\n 'PR #{} has no changelog label attached, please add one! Skipping...'.format(pr_num)\n )\n continue\n\n if len(changelog_labels) > 1:\n echo_warning(\n 'Multiple changelog labels found attached to PR #{}, '\n 'please only use one! Skipping...'.format(pr_num)\n )\n continue\n\n if changelog_labels[0] != CHANGELOG_TYPE_NONE:\n shippable_prs += 1\n except Exception as e:\n echo_failure('Unable to fetch info for PR #{}: {}'.format(pr_num, e))\n continue\n\n if shippable_prs:\n if quiet:\n msg = target\n else:\n msg = 'Check {} has {} out of {} merged PRs that could be released' ''.format(\n target, shippable_prs, len(pr_numbers)\n )\n echo_info(msg)", "response": "Show all the checks that can be released."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef changes(ctx, check, dry_run):\n if not dry_run and check not in get_valid_checks():\n abort('Check `{}` is not an Agent-based Integration'.format(check))\n\n # get the name of the current release tag\n cur_version = get_version_string(check)\n target_tag = get_release_tag_string(check, cur_version)\n\n # get the diff from HEAD\n diff_lines = get_commits_since(check, target_tag)\n\n # for each PR get the title, we'll use it to populate the changelog\n pr_numbers = parse_pr_numbers(diff_lines)\n if not dry_run:\n echo_info('Found {} PRs merged since tag: {}'.format(len(pr_numbers), target_tag))\n\n user_config = ctx.obj\n if dry_run:\n changelog_types = []\n\n for pr_num in pr_numbers:\n try:\n payload = get_pr(pr_num, user_config)\n except Exception as e:\n echo_failure('Unable to fetch info for PR #{}: {}'.format(pr_num, e))\n continue\n\n current_changelog_types = get_changelog_types(payload)\n if not current_changelog_types:\n abort('No valid changelog labels found attached to PR #{}, please add one!'.format(pr_num))\n elif len(current_changelog_types) > 1:\n abort('Multiple changelog labels found attached to PR #{}, please only use one!'.format(pr_num))\n\n current_changelog_type = current_changelog_types[0]\n if current_changelog_type != 'no-changelog':\n changelog_types.append(current_changelog_type)\n\n return cur_version, changelog_types\n else:\n for pr_num in pr_numbers:\n try:\n payload = get_pr(pr_num, user_config)\n except Exception as e:\n echo_failure('Unable to fetch info for PR #{}: {}'.format(pr_num, e))\n continue\n\n changelog_types = get_changelog_types(payload)\n\n echo_success(payload.get('title'))\n echo_info(' * Url: {}'.format(payload.get('html_url')))\n\n echo_info(' * Changelog status: ', nl=False)\n if not changelog_types:\n echo_warning('WARNING! No changelog labels attached.\\n')\n elif len(changelog_types) > 1:\n echo_warning('WARNING! Too many changelog labels attached: {}\\n'.format(', '.join(changelog_types)))\n else:\n echo_success('{}\\n'.format(changelog_types[0]))", "response": "Show all the pending PRs for a given check."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntags the HEAD of the git repo with the current release number.", "response": "def tag(check, version, push, dry_run):\n \"\"\"Tag the HEAD of the git repo with the current release number for a\n specific check. The tag is pushed to origin by default.\n\n You can tag everything at once by setting the check to `all`.\n\n Notice: specifying a different version than the one in __about__.py is\n a maintenance task that should be run under very specific circumstances\n (e.g. re-align an old release performed on the wrong commit).\n \"\"\"\n tagging_all = check == 'all'\n\n valid_checks = get_valid_checks()\n if not tagging_all and check not in valid_checks:\n abort('Check `{}` is not an Agent-based Integration'.format(check))\n\n if tagging_all:\n if version:\n abort('You cannot tag every check with the same version')\n checks = sorted(valid_checks)\n else:\n checks = [check]\n\n # Check for any new tags\n tagged = False\n\n for check in checks:\n echo_info('{}:'.format(check))\n\n # get the current version\n if not version:\n version = get_version_string(check)\n\n # get the tag name\n release_tag = get_release_tag_string(check, version)\n echo_waiting('Tagging HEAD with {}... '.format(release_tag), indent=True, nl=False)\n\n if dry_run:\n version = None\n click.echo()\n continue\n\n result = git_tag(release_tag, push)\n\n if result.code == 128 or 'already exists' in result.stderr:\n echo_warning('already exists')\n elif result.code != 0:\n abort('\\n{}{}'.format(result.stdout, result.stderr), code=result.code)\n else:\n tagged = True\n echo_success('success!')\n\n # Reset version\n version = None\n\n if not tagged:\n abort(code=2)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef make(ctx, check, version, initial_release, skip_sign, sign_only):\n # Import lazily since in-toto runs a subprocess to check for gpg2 on load\n from ..signing import update_link_metadata, YubikeyException\n\n releasing_all = check == 'all'\n\n valid_checks = get_valid_checks()\n if not releasing_all and check not in valid_checks:\n abort('Check `{}` is not an Agent-based Integration'.format(check))\n\n # don't run the task on the master branch\n if get_current_branch() == 'master':\n abort('This task will commit, you do not want to add commits to master directly')\n\n if releasing_all:\n if version:\n abort('You cannot bump every check to the same version')\n checks = sorted(valid_checks)\n else:\n checks = [check]\n\n if initial_release:\n version = '1.0.0'\n\n for check in checks:\n if sign_only:\n break\n elif initial_release and check in BETA_PACKAGES:\n continue\n\n # Initial releases will only bump if not already 1.0.0 so no need to always output\n if not initial_release:\n echo_success('Check `{}`'.format(check))\n\n if version:\n # sanity check on the version provided\n cur_version = get_version_string(check)\n\n if version == 'final':\n # Remove any pre-release metadata\n version = finalize_version(cur_version)\n else:\n # Keep track of intermediate version bumps\n prev_version = cur_version\n for method in version.split(','):\n # Apply any supported version bumping methods. Chaining is required for going\n # from mainline releases to development releases since e.g. x.y.z > x.y.z-rc.A.\n # So for an initial bug fix dev release you can do `fix,rc`.\n if method in VERSION_BUMP:\n version = VERSION_BUMP[method](prev_version)\n prev_version = version\n\n p_version = parse_version_info(version)\n p_current = parse_version_info(cur_version)\n if p_version <= p_current:\n if initial_release:\n continue\n else:\n abort('Current version is {}, cannot bump to {}'.format(cur_version, version))\n else:\n cur_version, changelog_types = ctx.invoke(changes, check=check, dry_run=True)\n if not changelog_types:\n echo_warning('No changes for {}, skipping...'.format(check))\n continue\n bump_function = get_bump_function(changelog_types)\n version = bump_function(cur_version)\n\n if initial_release:\n echo_success('Check `{}`'.format(check))\n\n # update the version number\n echo_info('Current version of check {}: {}'.format(check, cur_version))\n echo_waiting('Bumping to {}... '.format(version), nl=False)\n update_version_module(check, cur_version, version)\n echo_success('success!')\n\n # update the CHANGELOG\n echo_waiting('Updating the changelog... ', nl=False)\n # TODO: Avoid double GitHub API calls when bumping all checks at once\n ctx.invoke(\n changelog,\n check=check,\n version=version,\n old_version=cur_version,\n initial=initial_release,\n quiet=True,\n dry_run=False,\n )\n echo_success('success!')\n\n commit_targets = [check]\n\n # update the list of integrations to be shipped with the Agent\n if check not in NOT_CHECKS:\n req_file = get_agent_release_requirements()\n commit_targets.append(os.path.basename(req_file))\n echo_waiting('Updating the Agent requirements file... ', nl=False)\n update_agent_requirements(req_file, check, get_agent_requirement_line(check, version))\n echo_success('success!')\n\n echo_waiting('Committing files...')\n\n # commit the changes.\n # do not use [ci skip] so releases get built https://docs.gitlab.com/ee/ci/yaml/#skipping-jobs\n msg = '[Release] Bumped {} version to {}'.format(check, version)\n git_commit(commit_targets, msg)\n\n if not initial_release:\n # Reset version\n version = None\n\n if sign_only or not skip_sign:\n echo_waiting('Updating release metadata...')\n echo_info('Please touch your Yubikey immediately after entering your PIN!')\n try:\n commit_targets = update_link_metadata(checks)\n git_commit(commit_targets, '[Release] Update metadata', force=True)\n except YubikeyException as e:\n abort('A problem occurred while signing metadata: {}'.format(e))\n\n # done\n echo_success('All done, remember to push to origin and open a PR to merge these changes on master')", "response": "This function generates a single check and returns the version of the check."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the changelog for a given check.", "response": "def changelog(ctx, check, version, old_version, initial, quiet, dry_run):\n \"\"\"Perform the operations needed to update the changelog.\n\n This method is supposed to be used by other tasks and not directly.\n \"\"\"\n if check not in get_valid_checks():\n abort('Check `{}` is not an Agent-based Integration'.format(check))\n\n # sanity check on the version provided\n cur_version = old_version or get_version_string(check)\n if parse_version_info(version) <= parse_version_info(cur_version):\n abort('Current version is {}, cannot bump to {}'.format(cur_version, version))\n\n if not quiet:\n echo_info('Current version of check {}: {}, bumping to: {}'.format(check, cur_version, version))\n\n # get the name of the current release tag\n target_tag = get_release_tag_string(check, cur_version)\n\n # get the diff from HEAD\n diff_lines = get_commits_since(check, None if initial else target_tag)\n\n # for each PR get the title, we'll use it to populate the changelog\n pr_numbers = parse_pr_numbers(diff_lines)\n if not quiet:\n echo_info('Found {} PRs merged since tag: {}'.format(len(pr_numbers), target_tag))\n\n if initial:\n # Only use the first one\n del pr_numbers[:-1]\n\n user_config = ctx.obj\n entries = []\n for pr_num in pr_numbers:\n try:\n payload = get_pr(pr_num, user_config)\n except Exception as e:\n echo_failure('Unable to fetch info for PR #{}: {}'.format(pr_num, e))\n continue\n\n changelog_labels = get_changelog_types(payload)\n\n if not changelog_labels:\n abort('No valid changelog labels found attached to PR #{}, please add one!'.format(pr_num))\n elif len(changelog_labels) > 1:\n abort('Multiple changelog labels found attached to PR #{}, please only use one!'.format(pr_num))\n\n changelog_type = changelog_labels[0]\n if changelog_type == CHANGELOG_TYPE_NONE:\n if not quiet:\n # No changelog entry for this PR\n echo_info('Skipping PR #{} from changelog due to label'.format(pr_num))\n continue\n\n author = payload.get('user', {}).get('login')\n author_url = payload.get('user', {}).get('html_url')\n title = '[{}] {}'.format(changelog_type, payload.get('title'))\n\n entry = ChangelogEntry(pr_num, title, payload.get('html_url'), author, author_url, from_contributor(payload))\n\n entries.append(entry)\n\n # store the new changelog in memory\n new_entry = StringIO()\n\n # the header contains version and date\n header = '## {} / {}\\n'.format(version, datetime.now().strftime('%Y-%m-%d'))\n new_entry.write(header)\n\n # one bullet point for each PR\n new_entry.write('\\n')\n for entry in entries:\n thanks_note = ''\n if entry.from_contributor:\n thanks_note = ' Thanks [{}]({}).'.format(entry.author, entry.author_url)\n new_entry.write('* {}. See [#{}]({}).{}\\n'.format(entry.title, entry.number, entry.url, thanks_note))\n new_entry.write('\\n')\n\n # read the old contents\n changelog_path = os.path.join(get_root(), check, 'CHANGELOG.md')\n old = list(stream_file_lines(changelog_path))\n\n # write the new changelog in memory\n changelog_buffer = StringIO()\n\n # preserve the title\n changelog_buffer.write(''.join(old[:2]))\n\n # prepend the new changelog to the old contents\n # make the command idempotent\n if header not in old:\n changelog_buffer.write(new_entry.getvalue())\n\n # append the rest of the old changelog\n changelog_buffer.write(''.join(old[2:]))\n\n # print on the standard out in case of a dry run\n if dry_run:\n echo_info(changelog_buffer.getvalue())\n else:\n # overwrite the old changelog\n write_file(changelog_path, changelog_buffer.getvalue())"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds a wheel for a check as it is on the repo HEAD", "response": "def build(check, sdist):\n \"\"\"Build a wheel for a check as it is on the repo HEAD\"\"\"\n if check in get_valid_checks():\n check_dir = os.path.join(get_root(), check)\n else:\n check_dir = resolve_path(check)\n if not dir_exists(check_dir):\n abort('`{}` is not an Agent-based Integration or Python package'.format(check))\n\n check = basepath(check_dir)\n\n echo_waiting('Building `{}`...'.format(check))\n\n dist_dir = os.path.join(check_dir, 'dist')\n remove_path(dist_dir)\n\n result = build_package(check_dir, sdist)\n if result.code != 0:\n abort(result.stdout, result.code)\n\n echo_info('Build done, artifact(s) in: {}'.format(dist_dir))\n echo_success('Success!')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nupload a specific check to PyPI.", "response": "def upload(ctx, check, sdist, dry_run):\n \"\"\"Release a specific check to PyPI as it is on the repo HEAD.\"\"\"\n if check in get_valid_checks():\n check_dir = os.path.join(get_root(), check)\n else:\n check_dir = resolve_path(check)\n if not dir_exists(check_dir):\n abort('`{}` is not an Agent-based Integration or Python package'.format(check))\n\n check = basepath(check_dir)\n\n # retrieve credentials\n pypi_config = ctx.obj.get('pypi', {})\n username = pypi_config.get('user') or os.getenv('TWINE_USERNAME')\n password = pypi_config.get('pass') or os.getenv('TWINE_PASSWORD')\n if not (username and password):\n abort('This requires pypi.user and pypi.pass configuration. Please see `ddev config -h`.')\n\n auth_env_vars = {'TWINE_USERNAME': username, 'TWINE_PASSWORD': password}\n echo_waiting('Building and publishing `{}` to PyPI...'.format(check))\n\n with chdir(check_dir, env_vars=auth_env_vars):\n result = build_package(check_dir, sdist)\n if result.code != 0:\n abort(result.stdout, result.code)\n echo_waiting('Uploading the package...')\n if not dry_run:\n result = run_command('twine upload --skip-existing dist{}*'.format(os.path.sep))\n if result.code != 0:\n abort(code=result.code)\n\n echo_success('Success!')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef metrics(self, name):\n return [\n MetricStub(\n ensure_unicode(stub.name),\n stub.type,\n stub.value,\n normalize_tags(stub.tags),\n ensure_unicode(stub.hostname),\n )\n for stub in self._metrics.get(to_string(name), [])\n ]", "response": "Returns the list of metrics received under the given name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef service_checks(self, name):\n return [\n ServiceCheckStub(\n ensure_unicode(stub.check_id),\n ensure_unicode(stub.name),\n stub.status,\n normalize_tags(stub.tags),\n ensure_unicode(stub.hostname),\n ensure_unicode(stub.message),\n )\n for stub in self._service_checks.get(to_string(name), [])\n ]", "response": "Returns the list of service checks received under the given name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef events(self):\n all_events = [{ensure_unicode(key): value for key, value in iteritems(ev)} for ev in self._events]\n\n for ev in all_events:\n to_decode = []\n for key, value in iteritems(ev):\n if isinstance(value, binary_type) and key != 'host':\n to_decode.append(key)\n for key in to_decode:\n ev[key] = ensure_unicode(ev[key])\n\n if ev.get('tags'):\n ev['tags'] = normalize_tags(ev['tags'])\n\n return all_events", "response": "Return all events in the cache."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nasserting a metric is tagged with tag", "response": "def assert_metric_has_tag(self, metric_name, tag, count=None, at_least=1):\n \"\"\"\n Assert a metric is tagged with tag\n \"\"\"\n self._asserted.add(metric_name)\n\n candidates = []\n for metric in self.metrics(metric_name):\n if tag in metric.tags:\n candidates.append(metric)\n\n if count is not None:\n assert len(candidates) == count\n else:\n assert len(candidates) >= at_least"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef assert_metric(self, name, value=None, tags=None, count=None, at_least=1, hostname=None, metric_type=None):\n self._asserted.add(name)\n tags = normalize_tags(tags, sort=True)\n\n candidates = []\n for metric in self.metrics(name):\n if value is not None and not self.is_aggregate(metric.type) and value != metric.value:\n continue\n\n if tags and tags != sorted(metric.tags):\n continue\n\n if hostname and hostname != metric.hostname:\n continue\n\n if metric_type is not None and metric_type != metric.type:\n continue\n\n candidates.append(metric)\n\n if value is not None and candidates and all(self.is_aggregate(m.type) for m in candidates):\n got = sum(m.value for m in candidates)\n msg = \"Expected count value for '{}': {}, got {}\".format(name, value, got)\n assert value == got, msg\n\n if count is not None:\n msg = \"Needed exactly {} candidates for '{}', got {}\".format(count, name, len(candidates))\n assert len(candidates) == count, msg\n else:\n msg = \"Needed at least {} candidates for '{}', got {}\".format(at_least, name, len(candidates))\n assert len(candidates) >= at_least, msg", "response": "Assert a metric was processed by this stub."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nasserts a service check was processed by this stub.", "response": "def assert_service_check(self, name, status=None, tags=None, count=None, at_least=1, hostname=None, message=None):\n \"\"\"\n Assert a service check was processed by this stub\n \"\"\"\n tags = normalize_tags(tags, sort=True)\n candidates = []\n for sc in self.service_checks(name):\n if status is not None and status != sc.status:\n continue\n\n if tags and tags != sorted(sc.tags):\n continue\n\n if hostname is not None and hostname != sc.hostname:\n continue\n\n if message is not None and message != sc.message:\n continue\n\n candidates.append(sc)\n\n if count is not None:\n msg = \"Needed exactly {} candidates for '{}', got {}\".format(count, name, len(candidates))\n assert len(candidates) == count, msg\n else:\n msg = \"Needed at least {} candidates for '{}', got {}\".format(at_least, name, len(candidates))\n assert len(candidates) >= at_least, msg"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the metrics assertion coverage in percent.", "response": "def metrics_asserted_pct(self):\n \"\"\"\n Return the metrics assertion coverage\n \"\"\"\n num_metrics = len(self._metrics)\n num_asserted = len(self._asserted)\n\n if num_metrics == 0:\n if num_asserted == 0:\n return 100\n else:\n return 0\n\n return num_asserted / num_metrics * 100"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexamining the response and raise an error if something is off", "response": "def _process_response(cls, response):\n \"\"\"\n Examine the response and raise an error is something is off\n \"\"\"\n if len(response) != 1:\n raise BadResponseError(\"Malformed response: {}\".format(response))\n\n stats = list(itervalues(response))[0]\n if not len(stats):\n raise BadResponseError(\"Malformed response for host: {}\".format(stats))\n\n return stats"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the items stats for the given key and value", "response": "def get_items_stats(key, value):\n \"\"\"\n Optional metric handler for 'items' stats\n\n key: \"items::\" format\n value: return untouched\n\n Like all optional metric handlers returns metric, tags, value\n \"\"\"\n itemized_key = key.split(':')\n slab_id = itemized_key[1]\n metric = itemized_key[2]\n\n tags = [\"slab:{}\".format(slab_id)]\n\n return metric, tags, value"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the stats for items with optional metric handlers.", "response": "def get_slabs_stats(key, value):\n \"\"\"\n Optional metric handler for 'items' stats\n\n key: \"items::\" format\n value: return untouched\n\n Like all optional metric handlers returns metric, tags, value\n \"\"\"\n slabbed_key = key.split(':')\n tags = []\n if len(slabbed_key) == 2:\n slab_id = slabbed_key[0]\n metric = slabbed_key[1]\n tags = [\"slab:{}\".format(slab_id)]\n else:\n metric = slabbed_key[0]\n\n return metric, tags, value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_stats(\n self,\n instance,\n base_url,\n object_type,\n max_detailed,\n filters,\n limit_vhosts,\n custom_tags,\n auth=None,\n ssl_verify=True,\n ):\n \"\"\"\n instance: the check instance\n base_url: the url of the rabbitmq management api (e.g. http://localhost:15672/api)\n object_type: either QUEUE_TYPE or NODE_TYPE or EXCHANGE_TYPE\n max_detailed: the limit of objects to collect for this type\n filters: explicit or regexes filters of specified queues or nodes (specified in the yaml file)\n \"\"\"\n instance_proxy = self.get_instance_proxy(instance, base_url)\n # Make a copy of this list as we will remove items from it at each\n # iteration\n explicit_filters = list(filters['explicit'])\n regex_filters = filters['regexes']\n\n data = []\n\n # only do this if vhosts were specified,\n # otherwise it'll just be making more queries for the same data\n if self._limit_vhosts(instance) and object_type == QUEUE_TYPE:\n for vhost in limit_vhosts:\n url = '{}/{}'.format(object_type, quote_plus(vhost))\n try:\n data += self._get_data(\n urljoin(base_url, url), auth=auth, ssl_verify=ssl_verify, proxies=instance_proxy\n )\n except Exception as e:\n self.log.debug(\"Couldn't grab queue data from vhost, {}: {}\".format(vhost, e))\n else:\n data = self._get_data(\n urljoin(base_url, object_type), auth=auth, ssl_verify=ssl_verify, proxies=instance_proxy\n )\n\n \"\"\" data is a list of nodes or queues:\n data = [\n {\n 'status': 'running',\n 'node': 'rabbit@host',\n 'name': 'queue1',\n 'consumers': 0,\n 'vhost': '/',\n 'backing_queue_status': {\n 'q1': 0,\n 'q3': 0,\n 'q2': 0,\n 'q4': 0,\n 'avg_ack_egress_rate': 0.0,\n 'ram_msg_count': 0,\n 'ram_ack_count': 0,\n 'len': 0,\n 'persistent_count': 0,\n 'target_ram_count': 'infinity',\n 'next_seq_id': 0,\n 'delta': ['delta', 'undefined', 0, 'undefined'],\n 'pending_acks': 0,\n 'avg_ack_ingress_rate': 0.0,\n 'avg_egress_rate': 0.0,\n 'avg_ingress_rate': 0.0\n },\n 'durable': True,\n 'idle_since': '2013-10-03 13:38:18',\n 'exclusive_consumer_tag': '',\n 'arguments': {},\n 'memory': 10956,\n 'policy': '',\n 'auto_delete': False\n },\n {\n 'status': 'running',\n 'node': 'rabbit@host,\n 'name': 'queue10',\n 'consumers': 0,\n 'vhost': '/',\n 'backing_queue_status': {\n 'q1': 0,\n 'q3': 0,\n 'q2': 0,\n 'q4': 0,\n 'avg_ack_egress_rate': 0.0,\n 'ram_msg_count': 0,\n 'ram_ack_count': 0,\n 'len': 0,\n 'persistent_count': 0,\n 'target_ram_count': 'infinity',\n 'next_seq_id': 0,\n 'delta': ['delta', 'undefined', 0, 'undefined'],\n 'pending_acks': 0,\n 'avg_ack_ingress_rate': 0.0,\n 'avg_egress_rate': 0.0, 'avg_ingress_rate': 0.0\n },\n 'durable': True,\n 'idle_since': '2013-10-03 13:38:18',\n 'exclusive_consumer_tag': '',\n 'arguments': {},\n 'memory': 10956,\n 'policy': '',\n 'auto_delete': False\n },\n {\n 'status': 'running',\n 'node': 'rabbit@host',\n 'name': 'queue11',\n 'consumers': 0,\n 'vhost': '/',\n 'backing_queue_status': {\n 'q1': 0,\n 'q3': 0,\n 'q2': 0,\n 'q4': 0,\n 'avg_ack_egress_rate': 0.0,\n 'ram_msg_count': 0,\n 'ram_ack_count': 0,\n 'len': 0,\n 'persistent_count': 0,\n 'target_ram_count': 'infinity',\n 'next_seq_id': 0,\n 'delta': ['delta', 'undefined', 0, 'undefined'],\n 'pending_acks': 0,\n 'avg_ack_ingress_rate': 0.0,\n 'avg_egress_rate': 0.0,\n 'avg_ingress_rate': 0.0\n },\n 'durable': True,\n 'idle_since': '2013-10-03 13:38:18',\n 'exclusive_consumer_tag': '',\n 'arguments': {},\n 'memory': 10956,\n 'policy': '',\n 'auto_delete': False\n },\n ...\n ]\n \"\"\"\n if len(explicit_filters) > max_detailed:\n raise Exception(\"The maximum number of {} you can specify is {}.\".format(object_type, max_detailed))\n\n # a list of queues/nodes is specified. We process only those\n data = self._filter_list(\n data, explicit_filters, regex_filters, object_type, instance.get(\"tag_families\", False)\n )\n\n # if no filters are specified, check everything according to the limits\n if len(data) > ALERT_THRESHOLD * max_detailed:\n # Post a message on the dogweb stream to warn\n self.alert(base_url, max_detailed, len(data), object_type, custom_tags)\n\n if len(data) > max_detailed:\n # Display a warning in the info page\n msg = (\n \"Too many items to fetch. \"\n \"You must choose the {} you are interested in by editing the rabbitmq.yaml configuration file\"\n \"or get in touch with Datadog support\"\n ).format(object_type)\n self.warning(msg)\n\n for data_line in data[:max_detailed]:\n # We truncate the list if it's above the limit\n self._get_metrics(data_line, object_type, custom_tags)\n\n # get a list of the number of bindings on a given queue\n # /api/queues/vhost/name/bindings\n if object_type is QUEUE_TYPE:\n self._get_queue_bindings_metrics(\n base_url, custom_tags, data, instance_proxy, instance, object_type, auth, ssl_verify\n )", "response": "This function returns the stats for a specific check instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_connections_stat(\n self, instance, base_url, object_type, vhosts, limit_vhosts, custom_tags, auth=None, ssl_verify=True\n ):\n \"\"\"\n Collect metrics on currently open connection per vhost.\n \"\"\"\n instance_proxy = self.get_instance_proxy(instance, base_url)\n\n grab_all_data = True\n\n if self._limit_vhosts(instance):\n grab_all_data = False\n data = []\n for vhost in vhosts:\n url = \"vhosts/{}/{}\".format(quote_plus(vhost), object_type)\n try:\n data += self._get_data(\n urljoin(base_url, url), auth=auth, ssl_verify=ssl_verify, proxies=instance_proxy\n )\n except Exception as e:\n # This will happen if there is no connection data to grab\n self.log.debug(\"Couldn't grab connection data from vhost, {}: {}\".format(vhost, e))\n\n # sometimes it seems to need to fall back to this\n if grab_all_data or not len(data):\n data = self._get_data(\n urljoin(base_url, object_type), auth=auth, ssl_verify=ssl_verify, proxies=instance_proxy\n )\n\n stats = {vhost: 0 for vhost in vhosts}\n connection_states = defaultdict(int)\n for conn in data:\n if conn['vhost'] in vhosts:\n stats[conn['vhost']] += 1\n # 'state' does not exist for direct type connections.\n connection_states[conn.get('state', 'direct')] += 1\n\n for vhost, nb_conn in iteritems(stats):\n self.gauge('rabbitmq.connections', nb_conn, tags=['{}_vhost:{}'.format(TAG_PREFIX, vhost)] + custom_tags)\n\n for conn_state, nb_conn in iteritems(connection_states):\n self.gauge(\n 'rabbitmq.connections.state',\n nb_conn,\n tags=['{}_conn_state:{}'.format(TAG_PREFIX, conn_state)] + custom_tags,\n )", "response": "Get statistics on currently open connections."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_aliveness(self, instance, base_url, vhosts, custom_tags, auth=None, ssl_verify=True):\n\n for vhost in vhosts:\n tags = ['vhost:{}'.format(vhost)] + custom_tags\n # We need to urlencode the vhost because it can be '/'.\n path = u'aliveness-test/{}'.format(quote_plus(vhost))\n aliveness_url = urljoin(base_url, path)\n aliveness_proxy = self.get_instance_proxy(instance, aliveness_url)\n aliveness_response = self._get_data(\n aliveness_url, auth=auth, ssl_verify=ssl_verify, proxies=aliveness_proxy\n )\n message = u\"Response from aliveness API: {}\".format(aliveness_response)\n\n if aliveness_response.get('status') == 'ok':\n status = AgentCheck.OK\n else:\n status = AgentCheck.CRITICAL\n\n self.service_check('rabbitmq.aliveness', status, tags, message=message)", "response": "Check the aliveness API against all or a subset of vhosts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a GridIn property.", "response": "def _grid_in_property(field_name, docstring, read_only=False,\n closed_only=False):\n \"\"\"Create a GridIn property.\"\"\"\n def getter(self):\n if closed_only and not self._closed:\n raise AttributeError(\"can only get %r on a closed file\" %\n field_name)\n # Protect against PHP-237\n if field_name == 'length':\n return self._file.get(field_name, 0)\n return self._file.get(field_name, None)\n\n def setter(self, value):\n if self._closed:\n self._coll.files.update_one({\"_id\": self._file[\"_id\"]},\n {\"$set\": {field_name: value}})\n self._file[field_name] = value\n\n if read_only:\n docstring += \"\\n\\nThis attribute is read-only.\"\n elif closed_only:\n docstring = \"%s\\n\\n%s\" % (docstring, \"This attribute is read-only and \"\n \"can only be read after :meth:`close` \"\n \"has been called.\")\n\n if not read_only and not closed_only:\n return property(getter, setter, doc=docstring)\n return property(getter, doc=docstring)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a GridOut property.", "response": "def _grid_out_property(field_name, docstring):\n \"\"\"Create a GridOut property.\"\"\"\n def getter(self):\n self._ensure_file()\n\n # Protect against PHP-237\n if field_name == 'length':\n return self._file.get(field_name, 0)\n return self._file.get(field_name, None)\n\n docstring += \"\\n\\nThis attribute is read-only.\"\n return property(getter, doc=docstring)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __flush_data(self, data):\n # Ensure the index, even if there's nothing to write, so\n # the filemd5 command always succeeds.\n self.__ensure_indexes()\n self._file['md5'].update(data)\n\n if not data:\n return\n assert(len(data) <= self.chunk_size)\n\n chunk = {\"files_id\": self._file[\"_id\"],\n \"n\": self._chunk_number,\n \"data\": Binary(data)}\n\n try:\n self._chunks.insert_one(chunk)\n except DuplicateKeyError:\n self._raise_file_exists(self._file['_id'])\n self._chunk_number += 1\n self._position += len(data)", "response": "Flushes the data to a chunk."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nflushing the buffer contents out to a chunk.", "response": "def __flush_buffer(self):\n \"\"\"Flush the buffer contents out to a chunk.\n \"\"\"\n self.__flush_data(self._buffer.getvalue())\n self._buffer.close()\n self._buffer = StringIO()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflush the file to the database.", "response": "def __flush(self):\n \"\"\"Flush the file to the database.\n \"\"\"\n try:\n self.__flush_buffer()\n\n self._file['md5'] = self._file[\"md5\"].hexdigest()\n self._file[\"length\"] = self._position\n self._file[\"uploadDate\"] = datetime.datetime.utcnow()\n\n return self._coll.files.insert_one(self._file)\n except DuplicateKeyError:\n self._raise_file_exists(self._id)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflush the file and close it.", "response": "def close(self):\n \"\"\"Flush the file and close it.\n\n A closed file cannot be written any more. Calling\n :meth:`close` more than once is allowed.\n \"\"\"\n if not self._closed:\n self.__flush()\n object.__setattr__(self, \"_closed\", True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef write(self, data):\n if self._closed:\n raise ValueError(\"cannot write to a closed file\")\n\n try:\n # file-like\n read = data.read\n except AttributeError:\n # string\n if not isinstance(data, (text_type, bytes)):\n raise TypeError(\"can only write strings or file-like objects\")\n if isinstance(data, text_type):\n try:\n data = data.encode(self.encoding)\n except AttributeError:\n raise TypeError(\"must specify an encoding for file in \"\n \"order to write %s\" % (text_type.__name__,))\n read = StringIO(data).read\n\n if self._buffer.tell() > 0:\n # Make sure to flush only when _buffer is complete\n space = self.chunk_size - self._buffer.tell()\n if space:\n try:\n to_write = read(space)\n except:\n self.abort()\n raise\n self._buffer.write(to_write)\n if len(to_write) < space:\n return # EOF or incomplete\n self.__flush_buffer()\n to_write = read(self.chunk_size)\n while to_write and len(to_write) == self.chunk_size:\n self.__flush_data(to_write)\n to_write = read(self.chunk_size)\n self._buffer.write(to_write)", "response": "Writes data to the file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads a single chunk at a time.", "response": "def readchunk(self):\n \"\"\"Reads a chunk at a time. If the current position is within a\n chunk the remainder of the chunk is returned.\n \"\"\"\n received = len(self.__buffer)\n chunk_data = EMPTY\n chunk_size = int(self.chunk_size)\n\n if received > 0:\n chunk_data = self.__buffer\n elif self.__position < int(self.length):\n chunk_number = int((received + self.__position) / chunk_size)\n chunk = self.__chunks.find_one({\"files_id\": self._id,\n \"n\": chunk_number})\n if not chunk:\n raise CorruptGridFile(\"no chunk #%d\" % chunk_number)\n\n chunk_data = chunk[\"data\"][self.__position % chunk_size:]\n\n if not chunk_data:\n raise CorruptGridFile(\"truncated chunk\")\n\n self.__position += len(chunk_data)\n self.__buffer = EMPTY\n return chunk_data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read(self, size=-1):\n self._ensure_file()\n\n if size == 0:\n return EMPTY\n\n remainder = int(self.length) - self.__position\n if size < 0 or size > remainder:\n size = remainder\n\n received = 0\n data = StringIO()\n while received < size:\n chunk_data = self.readchunk()\n received += len(chunk_data)\n data.write(chunk_data)\n\n # Detect extra chunks.\n max_chunk_n = math.ceil(self.length / float(self.chunk_size))\n chunk = self.__chunks.find_one({\"files_id\": self._id,\n \"n\": {\"$gte\": max_chunk_n}})\n # According to spec, ignore extra chunks if they are empty.\n if chunk is not None and len(chunk['data']):\n raise CorruptGridFile(\n \"Extra chunk found: expected %i chunks but found \"\n \"chunk with n=%i\" % (max_chunk_n, chunk['n']))\n\n self.__position -= received - size\n\n # Return 'size' bytes and store the rest.\n data.seek(size)\n self.__buffer = data.read()\n data.seek(0)\n return data.read(size)", "response": "Reads at most size bytes from the file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef seek(self, pos, whence=_SEEK_SET):\n if whence == _SEEK_SET:\n new_pos = pos\n elif whence == _SEEK_CUR:\n new_pos = self.__position + pos\n elif whence == _SEEK_END:\n new_pos = int(self.length) + pos\n else:\n raise IOError(22, \"Invalid value for `whence`\")\n\n if new_pos < 0:\n raise IOError(22, \"Invalid value for `pos` - must be positive\")\n\n self.__position = new_pos\n self.__buffer = EMPTY", "response": "Set the current position of the file to pos."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef provider(self, value):\n result = None\n\n # `None` defaults to `ProviderArchitecture.DEFAULT`\n defaulted_value = value or ProviderArchitecture.DEFAULT\n\n try:\n parsed_value = int(defaulted_value)\n except ValueError:\n pass\n else:\n if parsed_value in ProviderArchitecture:\n result = parsed_value\n\n if result is None:\n self.logger.error(u\"Invalid '%s' WMI Provider Architecture. The parameter is ignored.\", value)\n\n self._provider = result or ProviderArchitecture.DEFAULT", "response": "Validate and set a WMI provider. Default to ProviderArchitecture. DEFAULT"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef connection(self):\n return {'host': self.host, 'namespace': self.namespace, 'username': self.username, 'password': self.password}", "response": "A property to retrieve the sampler connection information."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef connection_key(self):\n return \"{host}:{namespace}:{username}\".format(host=self.host, namespace=self.namespace, username=self.username)", "response": "Return an index key used to cache the sampler connection."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of filters as a comprehensive WQL clause.", "response": "def formatted_filters(self):\n \"\"\"\n Cache and return filters as a comprehensive WQL clause.\n \"\"\"\n if not self._formatted_filters:\n filters = deepcopy(self.filters)\n self._formatted_filters = self._format_filter(filters, self._and_props)\n return self._formatted_filters"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the calculator for the given counter_type.", "response": "def _get_property_calculator(self, counter_type):\n \"\"\"\n Return the calculator for the given `counter_type`.\n Fallback with `get_raw`.\n \"\"\"\n calculator = get_raw\n try:\n calculator = get_calculator(counter_type)\n except UndefinedCalculator:\n self.logger.warning(\n u\"Undefined WMI calculator for counter_type {counter_type}.\"\n \" Values are reported as RAW.\".format(counter_type=counter_type)\n )\n\n return calculator"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nformatting the WMI Object s RAW data based on the previous sample and the current sample.", "response": "def _format_property_values(self, previous, current):\n \"\"\"\n Format WMI Object's RAW data based on the previous sample.\n\n Do not override the original WMI Object !\n \"\"\"\n formatted_wmi_object = CaseInsensitiveDict()\n\n for property_name, property_raw_value in iteritems(current):\n counter_type = self._property_counter_types.get(property_name)\n property_formatted_value = property_raw_value\n\n if counter_type:\n calculator = self._get_property_calculator(counter_type)\n property_formatted_value = calculator(previous, current, property_name)\n\n formatted_wmi_object[property_name] = property_formatted_value\n\n return formatted_wmi_object"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new connection to the WMI server.", "response": "def get_connection(self):\n \"\"\"\n Create a new WMI connection\n \"\"\"\n self.logger.debug(\n u\"Connecting to WMI server \"\n u\"(host={host}, namespace={namespace}, provider={provider}, username={username}).\".format(\n host=self.host, namespace=self.namespace, provider=self.provider, username=self.username\n )\n )\n\n # Initialize COM for the current thread\n # WARNING: any python COM object (locator, connection, etc) created in a thread\n # shouldn't be used in other threads (can lead to memory/handle leaks if done\n # without a deep knowledge of COM's threading model). Because of this and given\n # that we run each query in its own thread, we don't cache connections\n additional_args = []\n pythoncom.CoInitialize()\n\n if self.provider != ProviderArchitecture.DEFAULT:\n context = Dispatch(\"WbemScripting.SWbemNamedValueSet\")\n context.Add(\"__ProviderArchitecture\", self.provider)\n additional_args = [None, \"\", 128, context]\n\n locator = Dispatch(\"WbemScripting.SWbemLocator\")\n connection = locator.ConnectServer(self.host, self.namespace, self.username, self.password, *additional_args)\n\n return connection"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _format_filter(filters, and_props=[]):\n\n def build_where_clause(fltr):\n f = fltr.pop()\n wql = \"\"\n while f:\n prop, value = f.popitem()\n\n if isinstance(value, tuple):\n oper = value[0]\n value = value[1]\n elif isinstance(value, string_types) and '%' in value:\n oper = 'LIKE'\n else:\n oper = '='\n\n if isinstance(value, list):\n if not len(value):\n continue\n\n internal_filter = map(\n lambda x: (prop, x)\n if isinstance(x, tuple)\n else (prop, ('LIKE', x))\n if '%' in x\n else (prop, (oper, x)),\n value,\n )\n\n bool_op = ' OR '\n for p in and_props:\n if p.lower() in prop.lower():\n bool_op = ' AND '\n break\n\n clause = bool_op.join(\n [\n '{0} {1} \\'{2}\\''.format(k, v[0], v[1])\n if isinstance(v, tuple)\n else '{0} = \\'{1}\\''.format(k, v)\n for k, v in internal_filter\n ]\n )\n\n if bool_op.strip() == 'OR':\n wql += \"( {clause} )\".format(clause=clause)\n else:\n wql += \"{clause}\".format(clause=clause)\n\n else:\n wql += \"{property} {cmp} '{constant}'\".format(property=prop, cmp=oper, constant=value)\n if f:\n wql += \" AND \"\n\n # empty list skipped\n if wql.endswith(\" AND \"):\n wql = wql[:-5]\n\n if len(fltr) == 0:\n return \"( {clause} )\".format(clause=wql)\n\n return \"( {clause} ) OR {more}\".format(clause=wql, more=build_where_clause(fltr))\n\n if not filters:\n return \"\"\n\n return \" WHERE {clause}\".format(clause=build_where_clause(filters))", "response": "Transform a list of filters to a comprehensive WQL WHERE clause."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nqueries the class and get the list of objects.", "response": "def _query(self): # pylint: disable=E0202\n \"\"\"\n Query WMI using WMI Query Language (WQL) & parse the results.\n\n Returns: List of WMI objects or `TimeoutException`.\n \"\"\"\n formated_property_names = \",\".join(self.property_names)\n wql = \"Select {property_names} from {class_name}{filters}\".format(\n property_names=formated_property_names, class_name=self.class_name, filters=self.formatted_filters\n )\n self.logger.debug(u\"Querying WMI: {0}\".format(wql))\n\n try:\n # From: https://msdn.microsoft.com/en-us/library/aa393866(v=vs.85).aspx\n flag_return_immediately = 0x10 # Default flag.\n flag_forward_only = 0x20\n flag_use_amended_qualifiers = 0x20000\n\n query_flags = flag_return_immediately | flag_forward_only\n\n # For the first query, cache the qualifiers to determine each\n # propertie's \"CounterType\"\n includes_qualifiers = self.is_raw_perf_class and self._property_counter_types is None\n if includes_qualifiers:\n self._property_counter_types = CaseInsensitiveDict()\n query_flags |= flag_use_amended_qualifiers\n\n raw_results = self.get_connection().ExecQuery(wql, \"WQL\", query_flags)\n\n results = self._parse_results(raw_results, includes_qualifiers=includes_qualifiers)\n\n except pywintypes.com_error:\n self.logger.warning(u\"Failed to execute WMI query (%s)\", wql, exc_info=True)\n results = []\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _parse_results(self, raw_results, includes_qualifiers):\n results = []\n for res in raw_results:\n # Ensure all properties are available. Use case-insensitivity\n # because some properties are returned with different cases.\n item = CaseInsensitiveDict()\n for prop_name in self.property_names:\n item[prop_name] = None\n\n for wmi_property in res.Properties_:\n # IMPORTANT: To improve performance, only access the Qualifiers\n # if the \"CounterType\" hasn't already been cached.\n should_get_qualifier_type = (\n includes_qualifiers and wmi_property.Name not in self._property_counter_types\n )\n\n if should_get_qualifier_type:\n\n # Can't index into \"Qualifiers_\" for keys that don't exist\n # without getting an exception.\n qualifiers = dict((q.Name, q.Value) for q in wmi_property.Qualifiers_)\n\n # Some properties like \"Name\" and \"Timestamp_Sys100NS\" do\n # not have a \"CounterType\" (since they're not a counter).\n # Therefore, they're ignored.\n if \"CounterType\" in qualifiers:\n counter_type = qualifiers[\"CounterType\"]\n self._property_counter_types[wmi_property.Name] = counter_type\n\n self.logger.debug(\n u\"Caching property qualifier CounterType: \"\n \"{class_name}.{property_names} = {counter_type}\".format(\n class_name=self.class_name, property_names=wmi_property.Name, counter_type=counter_type\n )\n )\n else:\n self.logger.debug(\n u\"CounterType qualifier not found for {class_name}.{property_names}\".format(\n class_name=self.class_name, property_names=wmi_property.Name\n )\n )\n\n try:\n item[wmi_property.Name] = float(wmi_property.Value)\n except (TypeError, ValueError):\n item[wmi_property.Name] = wmi_property.Value\n\n results.append(item)\n return results", "response": "Parse WMI query results into a more comprehensive form."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _check_with_socket(self, sock_info, metadata=None):\n cmd = SON([('ismaster', 1)])\n if metadata is not None:\n cmd['client'] = metadata\n start = _time()\n request_id, msg, max_doc_size = message.query(\n 0, 'admin.$cmd', 0, -1, cmd,\n None, DEFAULT_CODEC_OPTIONS)\n\n # TODO: use sock_info.command()\n sock_info.send_message(msg, max_doc_size)\n raw_response = sock_info.receive_message(1, request_id)\n result = helpers._unpack_response(raw_response)\n return IsMaster(result['data'][0]), _time() - start", "response": "Check if a master is running on the socket."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef agent_reqs():\n\n echo_info(\"Validating requirements-agent-release.txt...\")\n agent_reqs_content = parse_agent_req_file(read_file(get_agent_release_requirements()))\n ok_checks = 0\n unreleased_checks = 0\n failed_checks = 0\n for check_name in get_valid_checks():\n if check_name not in AGENT_V5_ONLY | NOT_CHECKS:\n package_name = get_package_name(check_name)\n check_version = get_version_string(check_name)\n pinned_version = agent_reqs_content.get(package_name)\n if package_name not in agent_reqs_content:\n unreleased_checks += 1\n echo_warning('{} has not yet been released'.format(check_name))\n elif check_version != pinned_version:\n failed_checks += 1\n echo_failure(\"{} has version {} but is pinned to {}\".format(check_name, check_version, pinned_version))\n else:\n ok_checks += 1\n\n if ok_checks:\n echo_success(\"{} correctly pinned checks\".format(ok_checks))\n if unreleased_checks:\n echo_warning(\"{} unreleased checks\".format(unreleased_checks))\n if failed_checks:\n echo_failure(\"{} checks out of sync\".format(failed_checks))\n abort()", "response": "Verify that the checks versions are in sync with the agent - release. txt file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlog a deprecation notice at most once per AgentCheck instance for the pre - defined deprecation_key.", "response": "def _log_deprecation(self, deprecation_key):\n \"\"\"\n Logs a deprecation notice at most once per AgentCheck instance, for the pre-defined `deprecation_key`\n \"\"\"\n if not self._deprecations[deprecation_key][0]:\n self.log.warning(self._deprecations[deprecation_key][1])\n self._deprecations[deprecation_key][0] = True"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert_to_underscore_separated(self, name):\n metric_name = self.FIRST_CAP_RE.sub(br'\\1_\\2', ensure_bytes(name))\n metric_name = self.ALL_CAP_RE.sub(br'\\1_\\2', metric_name).lower()\n metric_name = self.METRIC_REPLACEMENT.sub(br'_', metric_name)\n return self.DOT_UNDERSCORE_CLEANUP.sub(br'.', metric_name).strip(b'_')", "response": "Convert from CamelCase to CamelCase and substitute illegal metric characters"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _normalize_tags_type(self, tags, device_name=None, metric_name=None):\n normalized_tags = []\n\n if device_name:\n self._log_deprecation('device_name')\n normalized_tags.append('device:{}'.format(ensure_unicode(device_name)))\n\n if tags is not None:\n for tag in tags:\n if tag is None:\n continue\n if not isinstance(tag, str):\n try:\n tag = tag.decode('utf-8')\n except Exception:\n self.log.warning(\n 'Error decoding tag `{}` as utf-8 for metric `{}`, ignoring tag'.format(tag, metric_name)\n )\n continue\n\n normalized_tags.append(tag)\n\n return normalized_tags", "response": "Normalizes tags contents and type of tags."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nnormalize a metric name into a well - formed metric name.", "response": "def normalize(self, metric, prefix=None, fix_case=False):\n \"\"\"\n Turn a metric into a well-formed metric name\n prefix.b.c\n :param metric The metric name to normalize\n :param prefix A prefix to to add to the normalized name, default None\n :param fix_case A boolean, indicating whether to make sure that the metric name returned is in \"snake_case\"\n \"\"\"\n if isinstance(metric, text_type):\n metric = unicodedata.normalize('NFKD', metric).encode('ascii', 'ignore')\n\n if fix_case:\n name = self.convert_to_underscore_separated(metric)\n if prefix is not None:\n prefix = self.convert_to_underscore_separated(prefix)\n else:\n name = re.sub(br\"[,\\+\\*\\-/()\\[\\]{}\\s]\", b\"_\", metric)\n # Eliminate multiple _\n name = re.sub(br\"__+\", b\"_\", name)\n # Don't start/end with _\n name = re.sub(br\"^_\", b\"\", name)\n name = re.sub(br\"_$\", b\"\", name)\n # Drop ._ and _.\n name = re.sub(br\"\\._\", b\".\", name)\n name = re.sub(br\"_\\.\", b\".\", name)\n\n if prefix is not None:\n return ensure_bytes(prefix) + b\".\" + name\n else:\n return name"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _normalize_tags_type(self, tags, device_name=None, metric_name=None):\n normalized_tags = []\n\n if device_name:\n self._log_deprecation(\"device_name\")\n device_tag = self._to_bytes(\"device:{}\".format(device_name))\n if device_tag is None:\n self.log.warning(\n 'Error encoding device name `{}` to utf-8 for metric `{}`, ignoring tag'.format(\n repr(device_name), repr(metric_name)\n )\n )\n else:\n normalized_tags.append(device_tag)\n\n if tags is not None:\n for tag in tags:\n encoded_tag = self._to_bytes(tag)\n if encoded_tag is None:\n self.log.warning(\n 'Error encoding tag `{}` to utf-8 for metric `{}`, ignoring tag'.format(\n repr(tag), repr(metric_name)\n )\n )\n continue\n normalized_tags.append(encoded_tag)\n\n return normalized_tags", "response": "Normalizes tags contents and type of tags."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _to_bytes(self, data):\n # TODO: On Python 3, move this `if` line to the `except` branch\n # as the common case will indeed no longer be bytes.\n if not isinstance(data, bytes):\n try:\n return data.encode('utf-8')\n except Exception:\n return None\n\n return data", "response": "Normalize a text data to bytes."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_instance(instance):\n url = instance.get('url')\n if not url:\n raise ConfigurationError(\"A URL must be specified in the instance\")\n\n pshard_stats = is_affirmative(instance.get('pshard_stats', False))\n pshard_graceful_to = is_affirmative(instance.get('pshard_graceful_timeout', False))\n node_name_as_host = is_affirmative(instance.get('node_name_as_host', False))\n index_stats = is_affirmative(instance.get('index_stats', False))\n cluster_stats = is_affirmative(instance.get('cluster_stats', False))\n if 'is_external' in instance:\n cluster_stats = is_affirmative(instance.get('is_external', False))\n pending_task_stats = is_affirmative(instance.get('pending_task_stats', True))\n admin_forwarder = is_affirmative(instance.get('admin_forwarder', False))\n\n # Support URLs that have a path in them from the config, for\n # backwards-compatibility.\n parsed = urlparse(url)\n if parsed[2] and not admin_forwarder:\n url = '{}://{}'.format(parsed[0], parsed[1])\n port = parsed.port\n host = parsed.hostname\n\n custom_tags = instance.get('tags', [])\n service_check_tags = ['host:{}'.format(host), 'port:{}'.format(port)]\n service_check_tags.extend(custom_tags)\n\n # Tag by URL so we can differentiate the metrics\n # from multiple instances\n tags = ['url:{}'.format(url)]\n tags.extend(custom_tags)\n\n timeout = instance.get('timeout') or DEFAULT_TIMEOUT\n\n config = ESInstanceConfig(\n admin_forwarder=admin_forwarder,\n pshard_stats=pshard_stats,\n pshard_graceful_to=pshard_graceful_to,\n node_name_as_host=node_name_as_host,\n cluster_stats=cluster_stats,\n index_stats=index_stats,\n password=instance.get('password'),\n service_check_tags=service_check_tags,\n health_tags=[],\n ssl_cert=instance.get('ssl_cert'),\n ssl_key=instance.get('ssl_key'),\n ssl_verify=instance.get('ssl_verify'),\n tags=tags,\n timeout=timeout,\n url=url,\n username=instance.get('username'),\n pending_task_stats=pending_task_stats,\n )\n return config", "response": "Create a config object from an instance dictionary."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndownload the dataset from the hosted Yellowbrick data store and save it to the location specified by get_data_home.", "response": "def download(self, replace=False):\n \"\"\"\n Download the dataset from the hosted Yellowbrick data store and save\n it to the location specified by ``get_data_home``. The downloader\n verifies the download completed successfully and safely by comparing\n the expected signature with the SHA 256 signature of the downloaded\n archive file.\n\n Parameters\n ----------\n replace : bool, default: False\n If the data archive already exists, replace the dataset. If this is\n False and the dataset exists, an exception is raised.\n \"\"\"\n download_data(\n self.url, self.signature, data_home=self.data_home,\n replace=replace, extract=True\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef contents(self):\n data = find_dataset_path(\n self.name, data_home=self.data_home, ext=None\n )\n return os.listdir(data)", "response": "Contents returns a list of the files in the data directory."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the contents of the README. md file that describes the dataset in detail and contains attribution information.", "response": "def README(self):\n \"\"\"\n Returns the contents of the README.md file that describes the dataset\n in detail and contains attribution information.\n \"\"\"\n path = find_dataset_path(\n self.name, data_home=self.data_home, fname=\"README.md\"\n )\n with open(path, 'r') as f:\n return f.read()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the contents of the meta. json file that describes important attributes about the dataset and modifies the behavior of the loader.", "response": "def meta(self):\n \"\"\"\n Returns the contents of the meta.json file that describes important\n attributes about the dataset and modifies the behavior of the loader.\n \"\"\"\n path = find_dataset_path(\n self.name, data_home=self.data_home, fname=\"meta.json\", raises=False\n )\n if path is None:\n return None\n\n with open(path, 'r') as f:\n return json.load(f)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef citation(self):\n path = find_dataset_path(\n self.name, data_home=self.data_home, fname=\"meta.json\", raises=False\n )\n if path is None:\n return None\n\n with open(path, 'r') as f:\n return f.read()", "response": "Returns the contents of the citation. bib file that describes the source\n and provenance of the dataset or None if the dataset is not in the citation. json file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the dataset as two numpy arrays X and y.", "response": "def to_numpy(self):\n \"\"\"\n Returns the dataset as two numpy arrays: X and y.\n\n Returns\n -------\n X : array-like with shape (n_instances, n_features)\n A numpy array describing the instance features.\n\n y : array-like with shape (n_instances,)\n A numpy array describing the target vector.\n \"\"\"\n path = find_dataset_path(self.name, ext=\".npz\", data_home=self.data_home)\n with np.load(path, allow_pickle=False) as npf:\n if \"X\" not in npf or \"y\" not in npf:\n raise DatasetsError((\n \"the downloaded dataset was improperly packaged without numpy arrays \"\n \"- please report this bug to the Yellowbrick maintainers!\"\n ))\n\n # TODO: How to handle the case where y is None?\n return npf[\"X\"], npf[\"y\"]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the dataset as two pandas objects X and y.", "response": "def to_pandas(self):\n \"\"\"\n Returns the dataset as two pandas objects: X and y.\n\n Returns\n -------\n X : DataFrame with shape (n_instances, n_features)\n A pandas DataFrame containing feature data and named columns.\n\n y : Series with shape (n_instances,)\n A pandas Series containing target data and an index that matches\n the feature DataFrame index.\n \"\"\"\n # Ensure the metadata is valid before continuing\n if self.meta is None:\n raise DatasetsError((\n \"the downloaded dataset was improperly packaged without meta.json \"\n \"- please report this bug to the Yellowbrick maintainers!\"\n ))\n\n if \"features\" not in self.meta or \"target\" not in self.meta:\n raise DatasetsError((\n \"the downloaded dataset was improperly packaged without features \"\n \"or target - please report this bug to the Yellowbrick maintainers!\"\n ))\n\n # Load data frame and return features and target\n # TODO: Return y as None if there is no self.meta[\"target\"]\n df = self.to_dataframe()\n return df[self.meta[\"features\"]], df[self.meta[\"target\"]]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the entire dataset as a single pandas DataFrame.", "response": "def to_dataframe(self):\n \"\"\"\n Returns the entire dataset as a single pandas DataFrame.\n\n Returns\n -------\n df : DataFrame with shape (n_instances, n_columns)\n A pandas DataFrame containing the complete original data table\n including all targets (specified by the meta data) and all\n features (including those that might have been filtered out).\n \"\"\"\n if pd is None:\n raise DatasetsError(\n \"pandas is required to load DataFrame, it can be installed with pip\"\n )\n\n path = find_dataset_path(self.name, ext=\".csv.gz\", data_home=self.data_home)\n return pd.read_csv(path, compression=\"gzip\")"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the unique labels assigned to the documents.", "response": "def labels(self):\n \"\"\"\n Return the unique labels assigned to the documents.\n \"\"\"\n return [\n name for name in os.listdir(self.root)\n if os.path.isdir(os.path.join(self.root, name))\n ]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the list of file names for all documents.", "response": "def files(self):\n \"\"\"\n Returns the list of file names for all documents.\n \"\"\"\n return [\n os.path.join(self.root, label, name)\n for label in self.labels\n for name in os.listdir(os.path.join(self.root, label))\n ]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread all of the documents from disk into an in - memory list.", "response": "def data(self):\n \"\"\"\n Read all of the documents from disk into an in-memory list.\n \"\"\"\n def read(path):\n with open(path, 'r', encoding='UTF-8') as f:\n return f.read()\n\n return [\n read(f) for f in self.files\n ]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef target(self):\n return [\n os.path.basename(os.path.dirname(f)) for f in self.files\n ]", "response": "Returns the target of each item in data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfits the model and generates the silhouette visualization.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n Fits the model and generates the silhouette visualization.\n \"\"\"\n # TODO: decide to use this method or the score method to draw.\n # NOTE: Probably this would be better in score, but the standard score\n # is a little different and I'm not sure how it's used.\n\n # Fit the wrapped estimator\n self.estimator.fit(X, y, **kwargs)\n\n # Get the properties of the dataset\n self.n_samples_ = X.shape[0]\n self.n_clusters_ = self.estimator.n_clusters\n\n # Compute the scores of the cluster\n labels = self.estimator.predict(X)\n self.silhouette_score_ = silhouette_score(X, labels)\n self.silhouette_samples_ = silhouette_samples(X, labels)\n\n # Draw the silhouette figure\n self.draw(labels)\n\n # Return the estimator\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(self, labels):\n\n # Track the positions of the lines being drawn\n y_lower = 10 # The bottom of the silhouette\n\n # Get the colors from the various properties\n # TODO: Use resolve_colors instead of this\n colors = color_palette(self.colormap, self.n_clusters_)\n\n # For each cluster, plot the silhouette scores\n for idx in range(self.n_clusters_):\n\n # Collect silhouette scores for samples in the current cluster .\n values = self.silhouette_samples_[labels == idx]\n values.sort()\n\n # Compute the size of the cluster and find upper limit\n size = values.shape[0]\n y_upper = y_lower + size\n\n color = colors[idx]\n self.ax.fill_betweenx(\n np.arange(y_lower, y_upper), 0, values,\n facecolor=color, edgecolor=color, alpha=0.5\n )\n\n # Label the silhouette plots with their cluster numbers\n self.ax.text(-0.05, y_lower + 0.5 * size, str(idx))\n\n # Compute the new y_lower for next plot\n y_lower = y_upper + 10\n\n # The vertical line for average silhouette score of all the values\n self.ax.axvline(\n x=self.silhouette_score_, color=\"red\", linestyle=\"--\"\n )\n\n return self.ax", "response": "Draws the silhouette scores for each sample and the average score of all the values in the current cluster."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef finalize(self):\n\n # Set the title\n self.set_title((\n \"Silhouette Plot of {} Clustering for {} Samples in {} Centers\"\n ).format(\n self.name, self.n_samples_, self.n_clusters_\n ))\n\n # Set the X and Y limits\n # The silhouette coefficient can range from -1, 1;\n # but here we scale the plot according to our visualizations\n\n # l_xlim and u_xlim are lower and upper limits of the x-axis,\n # set according to our calculated maximum and minimum silhouette score along with necessary padding\n l_xlim = max(-1, min(-0.1, round(min(self.silhouette_samples_) - 0.1, 1)))\n u_xlim = min(1, round(max(self.silhouette_samples_) + 0.1, 1))\n self.ax.set_xlim([l_xlim, u_xlim])\n\n # The (n_clusters_+1)*10 is for inserting blank space between\n # silhouette plots of individual clusters, to demarcate them clearly.\n self.ax.set_ylim([0, self.n_samples_ + (self.n_clusters_ + 1) * 10])\n\n # Set the x and y labels\n self.ax.set_xlabel(\"silhouette coefficient values\")\n self.ax.set_ylabel(\"cluster label\")\n\n # Set the ticks on the axis object.\n self.ax.set_yticks([]) # Clear the yaxis labels / ticks\n self.ax.xaxis.set_major_locator(ticker.MultipleLocator(0.1))", "response": "Finalize the figure by setting the title adjusting\n the limits and labels on the axes and adding the ticks on the yaxis."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the SHA256 signature of a file.", "response": "def sha256sum(path, blocksize=65536):\n \"\"\"\n Computes the SHA256 signature of a file to verify that the file has not\n been modified in transit and that it is the correct version of the data.\n \"\"\"\n sig = hashlib.sha256()\n with open(path, 'rb') as f:\n buf = f.read(blocksize)\n while len(buf) > 0:\n sig.update(buf)\n buf = f.read(blocksize)\n return sig.hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef draw_best_fit(X, y, ax, estimator='linear', **kwargs):\n\n # Estimators are the types of best fit lines that can be drawn.\n estimators = {\n LINEAR: fit_linear, # Uses OLS to fit the regression\n QUADRATIC: fit_quadratic, # Uses OLS with Polynomial order 2\n EXPONENTIAL: fit_exponential, # Not implemented yet\n LOG: fit_log, # Not implemented yet\n SELECT_BEST: fit_select_best, # Selects the best fit via MSE\n }\n\n # Check to make sure that a correct estimator value was passed in.\n if estimator not in estimators:\n raise YellowbrickValueError(\n \"'{}' not a valid type of estimator; choose from {}\".format(\n estimator, \", \".join(estimators.keys())\n )\n )\n\n # Then collect the estimator function from the mapping.\n estimator = estimators[estimator]\n\n # Ensure that X and y are the same length\n if len(X) != len(y):\n raise YellowbrickValueError((\n \"X and y must have same length:\"\n \" X len {} doesn't match y len {}!\"\n ).format(len(X), len(y)))\n\n # Ensure that X and y are np.arrays\n X = np.array(X)\n y = np.array(y)\n\n # Verify that X is a two dimensional array for Scikit-Learn esitmators\n # and that its dimensions are (n, 1) where n is the number of rows.\n if X.ndim < 2:\n X = X[:,np.newaxis] # Reshape X into the correct dimensions\n\n if X.ndim > 2:\n raise YellowbrickValueError(\n \"X must be a (1,) or (n,1) dimensional array not {}\".format(X.shape)\n )\n\n # Verify that y is a (n,) dimensional array\n if y.ndim > 1:\n raise YellowbrickValueError(\n \"y must be a (1,) dimensional array not {}\".format(y.shape)\n )\n\n # Uses the estimator to fit the data and get the model back.\n model = estimator(X, y)\n\n # Set the color if not passed in.\n if 'c' not in kwargs and 'color' not in kwargs:\n kwargs['color'] = LINE_COLOR\n\n # Get the current working axes\n ax = ax or plt.gca()\n\n # Plot line of best fit onto the axes that were passed in.\n # TODO: determine if xlim or X.min(), X.max() are better params\n xr = np.linspace(*ax.get_xlim(), num=100)\n ax.plot(xr, model.predict(xr[:,np.newaxis]), **kwargs)\n return ax", "response": "Draws a best fit line on the current axes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nselect the best fit of the estimators already implemented by choosing the smallest mean square error metric for the trained values.", "response": "def fit_select_best(X, y):\n \"\"\"\n Selects the best fit of the estimators already implemented by choosing the\n model with the smallest mean square error metric for the trained values.\n \"\"\"\n models = [fit(X,y) for fit in [fit_linear, fit_quadratic]]\n errors = map(lambda model: mse(y, model.predict(X)), models)\n\n return min(zip(models, errors), key=itemgetter(1))[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit_linear(X, y):\n model = linear_model.LinearRegression()\n model.fit(X, y)\n return model", "response": "Uses OLS to fit the regression."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit_quadratic(X, y):\n model = make_pipeline(\n PolynomialFeatures(2), linear_model.LinearRegression()\n )\n model.fit(X, y)\n return model", "response": "Fit a quadratic model to the data X and y."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndraw a 45 degree identity line such that y = x for all points within the current working axes x and y limits.", "response": "def draw_identity_line(ax=None, dynamic=True, **kwargs):\n \"\"\"\n Draws a 45 degree identity line such that y=x for all points within the\n given axes x and y limits. This function also registeres a callback so\n that as the figure is modified, the axes are updated and the line remains\n drawn correctly.\n\n Parameters\n ----------\n\n ax : matplotlib Axes, default: None\n The axes to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n dynamic : bool, default : True\n If the plot is dynamic, callbacks will be registered to update the\n identiy line as axes are changed.\n\n kwargs : dict\n Keyword arguments to pass to the matplotlib plot function to style the\n identity line.\n\n\n Returns\n -------\n\n ax : matplotlib Axes\n The axes with the line drawn on it.\n\n Notes\n -----\n\n .. seealso:: `StackOverflow discussion: Does matplotlib have a function for drawing diagonal lines in axis coordinates? `_\n \"\"\"\n\n # Get the current working axes\n ax = ax or plt.gca()\n\n # Define the standard line color\n if 'c' not in kwargs and 'color' not in kwargs:\n kwargs['color'] = LINE_COLOR\n\n # Define the standard opacity\n if 'alpha' not in kwargs:\n kwargs['alpha'] = 0.5\n\n # Draw the identity line\n identity, = ax.plot([],[], **kwargs)\n\n # Define the callback\n def callback(ax):\n # Get the x and y limits on the axes\n xlim = ax.get_xlim()\n ylim = ax.get_ylim()\n\n # Set the bounding range of the line\n data = (\n max(xlim[0], ylim[0]), min(xlim[1], ylim[1])\n )\n identity.set_data(data, data)\n\n # Register the callback and return\n callback(ax)\n\n if dynamic:\n ax.callbacks.connect('xlim_changed', callback)\n ax.callbacks.connect('ylim_changed', callback)\n\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing the confusion matrix based on the test data X with the true values specified by the target vector y.", "response": "def score(self, X, y):\n \"\"\"\n Draws a confusion matrix based on the test data supplied by comparing\n predictions on instances X with the true values specified by the\n target vector y.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n Returns\n -------\n\n score_ : float\n Global accuracy score\n \"\"\"\n # Create predictions from X (will raise not fitted error)\n y_pred = self.predict(X)\n\n # Encode the target with the supplied label encoder\n if self.label_encoder:\n try :\n y = self.label_encoder.inverse_transform(y)\n y_pred = self.label_encoder.inverse_transform(y_pred)\n except AttributeError:\n # if a mapping is passed to class apply it here.\n y = np.array([self.label_encoder[x] for x in y])\n y_pred = np.array([self.label_encoder[x] for x in y_pred])\n\n # Compute the confusion matrix and class counts\n self.confusion_matrix_ = confusion_matrix_metric(\n y, y_pred, labels=self.classes_, sample_weight=self.sample_weight\n )\n self.class_counts_ = self.class_counts(y)\n\n # Make array of only the classes actually being used.\n # Needed because sklearn confusion_matrix only returns counts for\n # selected classes but percent should be calculated on all classes\n selected_class_counts = []\n for c in self.classes_:\n try:\n selected_class_counts.append(self.class_counts_[c])\n except KeyError:\n selected_class_counts.append(0)\n self.class_counts_ = np.array(selected_class_counts)\n\n self.draw()\n\n # Retrieve and store the score attribute from the sklearn classifier\n self.score_ = self.estimator.score(X, y)\n\n return self.score_"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(self):\n\n # Perform display related manipulations on the confusion matrix data\n cm_display = self.confusion_matrix_\n\n # Convert confusion matrix to percent of each row, i.e. the\n # predicted as a percent of true in each class.\n if self.percent == True:\n # Note: div_safe function returns 0 instead of NAN.\n cm_display = div_safe(self.confusion_matrix_, self.class_counts_.reshape(-1,1))\n cm_display = np.round(cm_display* 100, decimals=0)\n\n # Y axis should be sorted top to bottom in pcolormesh\n cm_display = cm_display[::-1,::]\n\n # Set up the dimensions of the pcolormesh\n n_classes = len(self.classes_)\n X, Y = np.arange(n_classes+1), np.arange(n_classes+1)\n self.ax.set_ylim(bottom=0, top=cm_display.shape[0])\n self.ax.set_xlim(left=0, right=cm_display.shape[1])\n\n # Fetch the grid labels from the classes in correct order; set ticks.\n xticklabels = self.classes_\n yticklabels = self.classes_[::-1]\n ticks = np.arange(n_classes) + 0.5\n\n self.ax.set(xticks=ticks, yticks=ticks)\n self.ax.set_xticklabels(xticklabels, rotation=\"vertical\", fontsize=self.fontsize)\n self.ax.set_yticklabels(yticklabels, fontsize=self.fontsize)\n\n # Set data labels in the grid enumerating over all x,y class pairs.\n # NOTE: X and Y are one element longer than the confusion matrix, so\n # skip the last element in the enumeration to label grids.\n for x in X[:-1]:\n for y in Y[:-1]:\n\n # Extract the value and the text label\n value = cm_display[x,y]\n svalue = \"{:0.0f}\".format(value)\n if self.percent:\n svalue += \"%\"\n\n # Determine the grid and text colors\n base_color = self.cmap(value / cm_display.max())\n text_color = find_text_color(base_color)\n\n # Make zero values more subtle\n if cm_display[x,y] == 0:\n text_color = CMAP_MUTEDCOLOR\n\n # Add the label to the middle of the grid\n cx, cy = x+0.5, y+0.5\n self.ax.text(\n cy, cx, svalue, va='center', ha='center',\n color=text_color, fontsize=self.fontsize,\n )\n\n # Add a dark line on the grid with the diagonal. Note that the\n # tick labels have already been reversed.\n lc = 'k' if xticklabels[x] == yticklabels[y] else 'w'\n self._edgecolors.append(lc)\n\n\n # Draw the heatmap with colors bounded by vmin,vmax\n vmin = 0.00001\n vmax = 99.999 if self.percent == True else cm_display.max()\n self.ax.pcolormesh(\n X, Y, cm_display, vmin=vmin, vmax=vmax,\n edgecolor=self._edgecolors, cmap=self.cmap, linewidth='0.01'\n )\n\n # Return the axes being drawn on\n return self.ax", "response": "Draws the classification report for the given class."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pca_decomposition(X, y=None, ax=None, features=None, scale=True,\n proj_dim=2, proj_features=False, color=None,\n colormap=palettes.DEFAULT_SEQUENCE, alpha=0.75,\n random_state=None, **kwargs):\n \"\"\"Produce a two or three dimensional principal component plot of the data array ``X``\n projected onto it's largest sequential principal components. It is common practice to scale the\n data array ``X`` before applying a PC decomposition. Variable scaling can be controlled using\n the ``scale`` argument.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features.\n\n y : ndarray or Series of length n\n An array or series of target or class values.\n\n ax : matplotlib Axes, default: None\n The axes to plot the figure on. If None is passed in the current axes.\n will be used (or generated if required).\n\n features: list, default: None\n a list of feature names to use\n If a DataFrame is passed to fit and features is None, feature\n names are selected as the columns of the DataFrame.\n\n scale : bool, default: True\n Boolean that indicates if user wants to scale data.\n\n proj_dim : int, default: 2\n Dimension of the PCA visualizer.\n\n proj_features : bool, default: False\n Boolean that indicates if the user wants to project the features\n in the projected space. If True the plot will be similar to a biplot.\n\n color : list or tuple of colors, default: None\n Specify the colors for each individual class.\n\n colormap : string or cmap, default: None\n Optional string or matplotlib cmap to colorize lines.\n Use either color to colorize the lines on a per class basis or\n colormap to color them on a continuous scale.\n \n alpha : float, default: 0.75\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n random_state : int, RandomState instance or None, optional (default None)\n If input data is larger than 500x500 and the number of components to\n extract is lower than 80% of the smallest dimension of the data, then\n the more efficient `randomized` solver is enabled, this parameter sets\n the random state on this solver.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Examples\n --------\n >>> from sklearn import datasets\n >>> iris = datasets.load_iris()\n >>> X = iris.data\n >>> y = iris.target\n >>> pca_decomposition(X, color=y, proj_dim=3, colormap='RdBu_r')\n\n \"\"\"\n # Instantiate the visualizer\n visualizer = PCADecomposition(\n ax=ax, features=features, scale=scale, proj_dim=proj_dim,\n proj_features=proj_features, color=color, colormap=colormap,\n alpha=alpha, random_state=random_state, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y)\n visualizer.transform(X)\n visualizer.poof()\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Generates a two - dimensional principal component plot of the data array X and y."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fit(self, X, y=None, **kwargs):\n super(PCADecomposition, self).fit(X=X, y=y, **kwargs)\n self.pca_transformer.fit(X)\n self.pca_components_ = self.pca_transformer.named_steps['pca'].components_\n return self", "response": "Fits the PCA transformer transforms the data in X then draws the PCA decomposition in either 2D or 3D space as a scatter plot."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit(self, X, y=None):\n # The target determines what kind of estimator is fit\n ttype = type_of_target(y)\n if ttype.startswith(MULTICLASS):\n self.target_type_ = MULTICLASS\n self.estimator = OneVsRestClassifier(self.estimator)\n\n # Use label_binarize to create multi-label ouptut for OneVsRestClassifier\n self._target_labels = np.unique(y)\n Y = label_binarize(y, classes=self._target_labels)\n elif ttype.startswith(BINARY):\n self.target_type_ = BINARY\n\n # Different variable is used here to prevent transformation\n Y = y\n else:\n raise YellowbrickValueError((\n \"{} does not support target type '{}', \"\n \"please provide a binary or multiclass single-output target\"\n ).format(\n self.__class__.__name__, ttype\n ))\n\n # Fit the model and return self\n return super(PrecisionRecallCurve, self).fit(X, Y)", "response": "Fit the classification model to the data X and y."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef score(self, X, y=None):\n # If we don't do this check, then it is possible that OneVsRestClassifier\n # has not correctly been fitted for multi-class targets.\n if not hasattr(self, \"target_type_\"):\n raise NotFitted((\n \"{} cannot wrap an already fitted estimator\"\n ).format(\n self.__class__.__name__\n ))\n\n # Compute the prediction/threshold scores\n y_scores = self._get_y_scores(X)\n\n # Handle binary and multiclass cases to create correct data structure\n if self.target_type_ == BINARY:\n self.precision_, self.recall_, _ = sk_precision_recall_curve(y, y_scores)\n self.score_ = average_precision_score(y, y_scores)\n else:\n # Use label_binarize to create multi-label ouptut for OneVsRestClassifier\n Y = label_binarize(y, classes=self._target_labels)\n\n self.precision_, self.recall_, self.score_ = {}, {}, {}\n\n # Compute PRCurve for all classes\n for i, class_i in enumerate(self.classes_):\n self.precision_[class_i], self.recall_[class_i], _ = sk_precision_recall_curve(Y[:,i], y_scores[:,i])\n self.score_[class_i] = average_precision_score(Y[:,i], y_scores[:,i])\n\n # Compute micro average PR curve\n self.precision_[MICRO], self.recall_[MICRO], _ = sk_precision_recall_curve(\n Y.ravel(), y_scores.ravel()\n )\n self.score_[MICRO] = average_precision_score(Y, y_scores, average=MICRO)\n\n # Draw the figure\n self.draw()\n\n # Return a score between 0 and 1\n if self.target_type_ == BINARY:\n return self.score_\n return self.score_[MICRO]", "response": "Generates the Precision - Recall curve on the specified test data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(self):\n if self.iso_f1_curves:\n for f1 in self.iso_f1_values:\n x = np.linspace(0.01, 1)\n y = f1 * x / (2 * x - f1)\n self.ax.plot(x[y>=0], y[y>=0], color='#333333', alpha=0.2)\n self.ax.annotate('$f_1={:0.1f}$'.format(f1), xy=(0.9, y[45]+0.02))\n\n if self.target_type_ == BINARY:\n return self._draw_binary()\n return self._draw_multiclass()", "response": "Draws the precision - recall curves computed in score on the axes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndraw the precision - recall curves in the binary case.", "response": "def _draw_binary(self):\n \"\"\"\n Draw the precision-recall curves in the binary case\n \"\"\"\n self._draw_pr_curve(self.recall_, self.precision_, label=\"binary PR curve\")\n self._draw_ap_score(self.score_)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _draw_multiclass(self):\n # TODO: handle colors better with a mapping and user input\n if self.per_class:\n for cls in self.classes_:\n precision = self.precision_[cls]\n recall = self.recall_[cls]\n\n label = \"PR for class {} (area={:0.2f})\".format(cls, self.score_[cls])\n self._draw_pr_curve(recall, precision, label=label)\n\n if self.micro:\n precision = self.precision_[MICRO]\n recall = self.recall_[MICRO]\n self._draw_pr_curve(recall, precision)\n\n self._draw_ap_score(self.score_[MICRO])", "response": "Draw the precision - recall curves in the multiclass case."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _draw_pr_curve(self, recall, precision, label=None):\n self.ax.step(\n recall, precision, alpha=self.line_opacity, where='post', label=label\n )\n if self.fill_area:\n self.ax.fill_between(recall, precision, step='post', alpha=self.fill_opacity)", "response": "Helper function to draw a precision - recall curve with specified settings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _draw_ap_score(self, score, label=None):\n label = label or \"Avg Precision={:0.2f}\".format(score)\n if self.ap_score:\n self.ax.axhline(\n y=score, color=\"r\", ls=\"--\", label=label\n )", "response": "Helper function to draw the AP score annotation."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the scores of the target class X for the precision recall curve.", "response": "def _get_y_scores(self, X):\n \"\"\"\n The ``precision_recall_curve`` metric requires target scores that\n can either be the probability estimates of the positive class,\n confidence values, or non-thresholded measures of decisions (as\n returned by a \"decision function\").\n \"\"\"\n # TODO refactor shared method with ROCAUC\n\n # Resolution order of scoring functions\n attrs = (\n 'decision_function',\n 'predict_proba',\n )\n\n # Return the first resolved function\n for attr in attrs:\n try:\n method = getattr(self.estimator, attr, None)\n if method:\n # Compute the scores from the decision function\n y_scores = method(X)\n\n # Return only the positive class for binary predict_proba\n if self.target_type_ == BINARY and y_scores.ndim == 2:\n return y_scores[:,1]\n return y_scores\n\n except AttributeError:\n # Some Scikit-Learn estimators have both probability and\n # decision functions but override __getattr__ and raise an\n # AttributeError on access.\n continue\n\n # If we've gotten this far, we can't do anything\n raise ModelError((\n \"{} requires estimators with predict_proba or decision_function methods.\"\n ).format(self.__class__.__name__))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a balanced binning reference for the given data set.", "response": "def balanced_binning_reference(y, ax=None, target='Frequency', bins=4, **kwargs):\n \n \"\"\"\n BalancedBinningReference generates a histogram with vertical lines\n showing the recommended value point to bin your data so they can be evenly\n distributed in each bin.\n\n Parameters\n ----------\n y : an array of one dimension or a pandas Series\n \n ax : matplotlib Axes, default: None\n This is inherited from FeatureVisualizer and is defined within\n ``BalancedBinningReference``.\n\n target : string, default: \"Frequency\"\n The name of the ``y`` variable\n\n bins : number of bins to generate the histogram, default: 4\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n \"\"\"\n\n # Initialize the visualizer\n visualizer = BalancedBinningReference(ax=ax, bins=bins, target=target, **kwargs)\n \n # Fit and poof the visualizer\n visualizer.fit(y)\n visualizer.poof()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndraws a histogram with the reference value for binning as vertical lines.", "response": "def draw(self, y, **kwargs):\n \"\"\"\n Draws a histogram with the reference value for binning as vertical\n lines.\n\n Parameters\n ----------\n y : an array of one dimension or a pandas Series\n \"\"\"\n\n # draw the histogram\n hist, bin_edges = np.histogram(y, bins=self.bins)\n self.bin_edges_ = bin_edges\n self.ax.hist(y, bins=self.bins, color=kwargs.pop(\"color\", \"#6897bb\"), **kwargs)\n\n # add vetical line with binning reference values\n plt.vlines(bin_edges,0,max(hist),colors=kwargs.pop(\"colors\", \"r\"))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fit(self, y, **kwargs):\n\n #throw an error if y has more than 1 column\n if y.ndim > 1:\n raise YellowbrickValueError(\"y needs to be an array or Series with one dimension\") \n\n # Handle the target name if it is None.\n if self.target is None:\n self.target = 'Frequency'\n\n self.draw(y)\n return self", "response": "Fit the object to the histogram and checks draw."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n\n \"\"\"\n self.ax.set_xlabel(self.target)\n for tk in self.ax.get_xticklabels():\n tk.set_visible(True)\n\n for tk in self.ax.get_yticklabels():\n tk.set_visible(True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndownload all the example datasets to the data directory specified by get_data_home.", "response": "def download_all(data_home=None, replace=False):\n \"\"\"\n Downloads all the example datasets to the data directory specified by\n ``get_data_home``. This function ensures that all datasets are available\n for use with the examples.\n \"\"\"\n for _, meta in DATASETS.items():\n download_data(\n meta['url'], meta['signature'], data_home=data_home, replace=replace\n )\n\n print(\n \"Downloaded {} datasets to {}\".format(len(DATASETS), get_data_home(data_home))\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cleanup_all(data_home=None):\n removed = 0\n for name, meta in DATASETS.items():\n _, ext = os.path.splitext(meta['url'])\n removed += cleanup_dataset(name, data_home=data_home, ext=ext)\n\n print(\n \"Removed {} fixture objects from {}\".format(removed, get_data_home(data_home))\n )", "response": "Cleans up all the example datasets in the data directory specified by get_data_home."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfits the visualizer to the target variables y_train and y_test.", "response": "def fit(self, y_train, y_test=None):\n \"\"\"\n Fit the visualizer to the the target variables, which must be 1D\n vectors containing discrete (classification) data. Fit has two modes:\n\n 1. Balance mode: if only y_train is specified\n 2. Compare mode: if both train and test are specified\n\n In balance mode, the bar chart is displayed with each class as its own\n color. In compare mode, a side-by-side bar chart is displayed colored\n by train or test respectively.\n\n Parameters\n ----------\n y_train : array-like\n Array or list of shape (n,) that containes discrete data.\n\n y_test : array-like, optional\n Array or list of shape (m,) that contains discrete data. If\n specified, the bar chart will be drawn in compare mode.\n \"\"\"\n\n # check to make sure that y_train is not a 2D array, e.g. X\n if y_train.ndim == 2:\n raise YellowbrickValueError((\n \"fit has changed to only require a 1D array, y \"\n \"since version 0.9; please see the docs for more info\"\n ))\n\n # Check the target types for the y variables\n self._validate_target(y_train)\n self._validate_target(y_test)\n\n # Get the unique values from the dataset\n targets = (y_train,) if y_test is None else (y_train, y_test)\n self.classes_ = unique_labels(*targets)\n\n # Validate the classes with the class names\n if self.labels is not None:\n if len(self.labels) != len(self.classes_):\n raise YellowbrickValueError((\n \"discovered {} classes in the data, does not match \"\n \"the {} labels specified.\"\n ).format(len(self.classes_), len(self.labels)))\n\n # Determine if we're in compare or balance mode\n self._mode = BALANCE if y_test is None else COMPARE\n\n # Compute the support values\n if self._mode == BALANCE:\n self.support_ = np.array([\n (y_train == idx).sum() for idx in self.classes_\n ])\n\n else:\n self.support_ = np.array([\n [\n (y == idx).sum() for idx in self.classes_\n ]\n for y in targets\n ])\n\n # Draw the bar chart\n self.draw()\n\n # Fit returns self\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef draw(self):\n # Number of colors is either number of classes or 2\n colors = resolve_colors(len(self.support_))\n\n if self._mode == BALANCE:\n self.ax.bar(\n np.arange(len(self.support_)), self.support_,\n color=colors, align='center', width=0.5\n )\n\n # Compare mode\n else:\n bar_width = 0.35\n labels = [\"train\", \"test\"]\n\n for idx, support in enumerate(self.support_):\n index = np.arange(len(self.classes_))\n if idx > 0:\n index = index + bar_width\n\n self.ax.bar(\n index, support, bar_width,\n color=colors[idx], label=labels[idx]\n )\n\n return self.ax", "response": "Draws the class balance chart on the specified axes from support."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef finalize(self, **kwargs):\n # Set the title\n self.set_title(\n 'Class Balance for {:,} Instances'.format(self.support_.sum())\n )\n\n # Set the x ticks with the class names or labels if specified\n labels = self.labels if self.labels is not None else self.classes_\n xticks = np.arange(len(labels))\n if self._mode == COMPARE:\n xticks = xticks + (0.35/2)\n\n self.ax.set_xticks(xticks)\n self.ax.set_xticklabels(labels)\n\n # Compute the ceiling for the y limit\n cmax = self.support_.max()\n self.ax.set_ylim(0, cmax + cmax* 0.1)\n self.ax.set_ylabel(\"support\")\n\n # Remove the vertical grid\n self.ax.grid(False, axis=\"x\")\n\n # Add the legend if in compare mode:\n if self._mode == COMPARE:\n self.ax.legend(frameon=True)", "response": "Executes any subclass - specific axes finalization steps."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvalidating that the target is a classification target.", "response": "def _validate_target(self, y):\n \"\"\"\n Raises a value error if the target is not a classification target.\n \"\"\"\n # Ignore None values\n if y is None:\n return\n\n y_type = type_of_target(y)\n if y_type not in (\"binary\", \"multiclass\"):\n raise YellowbrickValueError((\n \"'{}' target type not supported, only binary and multiclass\"\n ).format(y_type))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the name of the model or pipeline.", "response": "def get_model_name(model):\n \"\"\"\n Detects the model name for a Scikit-Learn model or pipeline.\n\n Parameters\n ----------\n model: class or instance\n The object to determine the name for. If the model is an estimator it\n returns the class name; if it is a Pipeline it returns the class name\n of the final transformer or estimator in the Pipeline.\n\n Returns\n -------\n name : string\n The name of the model or pipeline.\n \"\"\"\n if not is_estimator(model):\n raise YellowbrickTypeError(\n \"Cannot detect the model name for non estimator: '{}'\".format(\n type(model)\n )\n )\n\n else:\n if isinstance(model, Pipeline):\n return get_model_name(model.steps[-1][-1])\n else:\n return model.__class__.__name__"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if numeric feature columns exist in ndarray", "response": "def has_ndarray_int_columns(features, X):\n \"\"\" Checks if numeric feature columns exist in ndarray \"\"\"\n _, ncols = X.shape\n if not all(d.isdigit() for d in features if isinstance(d, str)) or not isinstance(X, np.ndarray):\n return False\n ndarray_columns = np.arange(0, ncols)\n feature_cols = np.unique([int(d) for d in features])\n return all(np.in1d(feature_cols, ndarray_columns))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_monotonic(a, increasing=True):\n a = np.asarray(a) # ensure a is array-like\n\n if a.ndim > 1:\n raise ValueError(\"not supported for multi-dimensonal arrays\")\n\n if len(a) <= 1:\n return True\n\n if increasing:\n return np.all(a[1:] >= a[:-1], axis=0)\n return np.all(a[1:] <= a[:-1], axis=0)", "response": "Tests whether a vector a has monotonicity."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts an array of property values to a size value.", "response": "def prop_to_size(vals, mi=0.0, ma=5.0, power=0.5, log=False):\n \"\"\"\n Converts an array of property values (e.g. a metric or score) to values\n that are more useful for marker sizes, line widths, or other visual\n sizes. The new sizes are computed as:\n\n y = mi + (ma -mi)(\\frac{x_i - min(x){max(x) - min(x)})^{power}\n\n If ``log=True``, the natural logarithm of the property values is used instead.\n\n Parameters\n ----------\n prop : array-like, 1D\n An array of values of the property to scale between the size range.\n\n mi : float, default: 0.0\n The size to assign the smallest property (minimum size value).\n\n ma : float, default: 5.0\n The size to assign the largest property (maximum size value).\n\n power : float, default: 0.5\n Used to control how rapidly the size increases from smallest to largest.\n\n log : bool, default: False\n Use the natural logarithm to compute the property sizes\n\n Returns\n -------\n sizes : array, 1D\n The new size values, in the same shape as the input prop array\n \"\"\"\n # ensure that prop is an array\n vals = np.asarray(vals)\n\n # apply natural log if specified\n if log:\n vals = np.log(vals)\n\n # avoid division by zero error\n delta = vals.max() - vals.min()\n if delta == 0.0:\n delta = 1.0\n\n return mi + (ma-mi) * ((vals -vals.min()) / delta) ** power"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a normalized slug of the given text.", "response": "def slugify(text):\n \"\"\"\n Returns a slug of given text, normalizing unicode data for file-safe\n strings. Used for deciding where to write images to disk.\n\n Parameters\n ----------\n text : string\n The string to slugify\n\n Returns\n -------\n slug : string\n A normalized slug representation of the text\n\n .. seealso:: http://yashchandra.com/2014/05/08/how-to-generate-clean-url-or-a-slug-in-python/\n \"\"\"\n slug = re.sub(r'[^\\w]+', ' ', text)\n slug = \"-\".join(slug.lower().strip().split())\n return slug"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute the mean distortion of all samples X and labels.", "response": "def distortion_score(X, labels, metric='euclidean'):\n \"\"\"\n Compute the mean distortion of all samples.\n\n The distortion is computed as the the sum of the squared distances between\n each observation and its closest centroid. Logically, this is the metric\n that K-Means attempts to minimize as it is fitting the model.\n\n .. seealso:: http://kldavenport.com/the-cost-function-of-k-means/\n\n Parameters\n ----------\n X : array, shape = [n_samples, n_features] or [n_samples_a, n_samples_a]\n Array of pairwise distances between samples if metric == \"precomputed\"\n or a feature array for computing distances against the labels.\n\n labels : array, shape = [n_samples]\n Predicted labels for each sample\n\n metric : string\n The metric to use when calculating distance between instances in a\n feature array. If metric is a string, it must be one of the options\n allowed by `sklearn.metrics.pairwise.pairwise_distances\n `_\n\n .. todo:: add sample_size and random_state kwds similar to silhouette_score\n \"\"\"\n # Encode labels to get unique centers and groups\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n unique_labels = le.classes_\n\n # Sum of the distortions\n distortion = 0\n\n # Loop through each label (center) to compute the centroid\n for current_label in unique_labels:\n # Mask the instances that belong to the current label\n mask = labels == current_label\n instances = X[mask]\n\n # Compute the center of these instances\n center = instances.mean(axis=0)\n\n # NOTE: csc_matrix and csr_matrix mean returns a 2D array, numpy.mean\n # returns an array of 1 dimension less than the input. We expect\n # instances to be a 2D array, therefore to do pairwise computation we\n # require center to be a 2D array with a single row (the center).\n # See #370 for more detail.\n if not sp.issparse(instances):\n center = np.array([center])\n\n # Compute the square distances from the instances to the center\n distances = pairwise_distances(instances, center, metric=metric)\n distances = distances ** 2\n\n # Add the sum of square distance to the distortion\n distortion += distances.sum()\n\n return distortion"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfit n KMeans models where n is the length of self. k_values_ and stores the value of the KneeLocator and the score corresponding to the KneeLocator.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n Fits n KMeans models where n is the length of ``self.k_values_``,\n storing the silhouette scores in the ``self.k_scores_`` attribute.\n The \"elbow\" and silhouette score corresponding to it are stored in\n ``self.elbow_value`` and ``self.elbow_score`` respectively.\n This method finishes up by calling draw to create the plot.\n \"\"\"\n\n self.k_scores_ = []\n self.k_timers_ = []\n\n if self.locate_elbow:\n self.elbow_value_ = None\n self.elbow_score_ = None\n\n for k in self.k_values_:\n # Compute the start time for each model\n start = time.time()\n\n # Set the k value and fit the model\n self.estimator.set_params(n_clusters=k)\n self.estimator.fit(X)\n\n # Append the time and score to our plottable metrics\n self.k_timers_.append(time.time() - start)\n self.k_scores_.append(\n self.scoring_metric(X, self.estimator.labels_)\n ) \n\n if self.locate_elbow:\n locator_kwargs = {\n 'distortion': {'curve_nature': 'convex', 'curve_direction': 'decreasing'},\n 'silhouette': {'curve_nature': 'concave', 'curve_direction': 'increasing'},\n 'calinski_harabaz': {'curve_nature': 'concave', 'curve_direction': 'increasing'},\n }.get(self.metric, {}) \n elbow_locator = KneeLocator(self.k_values_,self.k_scores_,**locator_kwargs)\n self.elbow_value_ = elbow_locator.knee\n if self.elbow_value_ == None:\n warning_message=\\\n \"No 'knee' or 'elbow' point detected, \" \\\n \"pass `locate_elbow=False` to remove the warning\" \n warnings.warn(warning_message,YellowbrickWarning) \n else:\n self.elbow_score_ = self.k_scores_[self.k_values_.index(self.elbow_value_)]\n \n\n self.draw()\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef draw(self):\n # Plot the silhouette score against k\n self.ax.plot(self.k_values_, self.k_scores_, marker=\"D\")\n if self.locate_elbow and self.elbow_value_!=None:\n elbow_label = \"$elbow\\ at\\ k={}, score={:0.3f}$\".format(self.elbow_value_, self.elbow_score_)\n self.ax.axvline(self.elbow_value_, c=LINE_COLOR, linestyle=\"--\", label=elbow_label)\n \n # If we're going to plot the timings, create a twinx axis\n if self.timings:\n self.axes = [self.ax, self.ax.twinx()]\n self.axes[1].plot(\n self.k_values_, self.k_timers_, label=\"fit time\",\n c='g', marker=\"o\", linestyle=\"--\", alpha=0.75,\n )\n\n\n return self.ax", "response": "Draw the elbow curve for the specified scores and values of K."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef finalize(self):\n # Get the metric name\n metric = self.scoring_metric.__name__.replace(\"_\", \" \").title()\n\n # Set the title\n self.set_title(\n '{} Elbow for {} Clustering'.format(metric, self.name)\n )\n\n # Set the x and y labels\n self.ax.set_xlabel('k')\n self.ax.set_ylabel(metric.lower())\n \n #set the legend if locate_elbow=True\n if self.locate_elbow and self.elbow_value_!=None:\n self.ax.legend(loc='best', fontsize='medium')\n\n # Set the second y axis labels\n if self.timings:\n self.axes[1].grid(False)\n self.axes[1].set_ylabel(\"fit time (seconds)\", color='g')\n self.axes[1].tick_params('y', colors='g')", "response": "Finalize the figure by setting the title the x and y labels and adding the legend."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsave a figure to a gallery directory", "response": "def savefig(viz, name, gallery=GALLERY):\n \"\"\"\n Saves the figure to the gallery directory\n \"\"\"\n if not path.exists(gallery):\n os.makedirs(gallery)\n\n # Must save as png\n if len(name.split(\".\")) > 1:\n raise ValueError(\"name should not specify extension\")\n\n outpath = path.join(gallery, name+\".png\")\n viz.poof(outpath=outpath)\n print(\"created {}\".format(outpath))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef poof(self, outdir=None, ext=\".pdf\", **kwargs):\n for name, step in self.visual_steps.items():\n if outdir is not None:\n outpath = path.join(outdir, slugify(name) + ext)\n else:\n outpath = None\n\n step.poof(outpath=outpath, **kwargs)", "response": "This method writes the visualizations for all the backend context."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fit_transform_poof(self, X, y=None, outpath=None, **kwargs):\n self.fit_transform(X, y, **kwargs)\n self.poof(outpath, **kwargs)", "response": "Fit the model and transforms and call poof."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisplaying a projection of a vectorized corpus in two dimensions using UMAP (Uniform Manifold Approximation and Projection), a nonlinear dimensionality reduction method that is particularly well suited to embedding in two or three dimensions for visualization as a scatter plot. UMAP is a relatively new technique but is often used to visualize clusters or groups of data points and their relative proximities. It typically is fast, scalable, and can be applied directly to sparse matrices eliminating the need to run a truncatedSVD as a pre-processing step. The current default for UMAP is Euclidean distance. Hellinger distance would be a more appropriate distance function to use with CountVectorize data. That will be released in the next version of UMAP. In the meantime cosine distance is likely a better text default that Euclidean and can be set using the keyword arg metric='cosine'. Parameters ---------- X : ndarray or DataFrame of shape n x m A matrix of n instances with m features representing the corpus of vectorized documents to visualize with umap. y : ndarray or Series of length n An optional array or series of target or class values for instances. If this is specified, then the points will be colored according to their class. Often cluster labels are passed in to color the documents in cluster space, so this method is used both for classification and clustering methods. ax : matplotlib axes The axes to plot the figure on. classes : list of strings The names of the classes in the target, used to create a legend. colors : list or tuple of colors Specify the colors for each individual class colormap : string or matplotlib cmap Sequential colormap for continuous target alpha : float, default: 0.7 Specify a transparency where 1 is completely opaque and 0 is completely transparent. This property makes densely clustered points more visible. kwargs : dict Pass any additional keyword arguments to the UMAP transformer. Returns ------- ax : matplotlib axes Returns the axes that the parallel coordinates were drawn on.", "response": "def umap(X, y=None, ax=None, classes=None,\n colors=None, colormap=None, alpha=0.7, **kwargs):\n \"\"\"\n Display a projection of a vectorized corpus in two dimensions using UMAP (Uniform\n Manifold Approximation and Projection),\n a nonlinear dimensionality reduction method that is particularly well\n suited to embedding in two or three dimensions for visualization as a\n scatter plot. UMAP is a relatively new technique but is often used to visualize clusters or\n groups of data points and their relative proximities. It typically is fast, scalable,\n and can be applied directly to sparse matrices eliminating the need to run a truncatedSVD\n as a pre-processing step.\n\n The current default for UMAP is Euclidean distance. Hellinger distance would be a more appropriate distance\n function to use with CountVectorize data. That will be released in the next version of UMAP.\n In the meantime cosine distance is likely a better text default that Euclidean and can be set using\n the keyword arg metric='cosine'.\n\n Parameters\n ----------\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features representing the corpus of\n vectorized documents to visualize with umap.\n\n y : ndarray or Series of length n\n An optional array or series of target or class values for instances.\n If this is specified, then the points will be colored according to\n their class. Often cluster labels are passed in to color the documents\n in cluster space, so this method is used both for classification and\n clustering methods.\n\n ax : matplotlib axes\n The axes to plot the figure on.\n\n classes : list of strings\n The names of the classes in the target, used to create a legend.\n\n colors : list or tuple of colors\n Specify the colors for each individual class\n\n colormap : string or matplotlib cmap\n Sequential colormap for continuous target\n\n alpha : float, default: 0.7\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n kwargs : dict\n Pass any additional keyword arguments to the UMAP transformer.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = UMAPVisualizer(\n ax, classes, colors, colormap, alpha, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit_transform(X, y, **kwargs)\n #visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate an internal transformer pipeline that projects the data set into the 2D space using UMAP.", "response": "def make_transformer(self, umap_kwargs={}):\n \"\"\"\n Creates an internal transformer pipeline to project the data set into\n 2D space using UMAP. This method will reset the transformer on the\n class.\n\n Parameters\n ----------\n\n Returns\n -------\n transformer : Pipeline\n Pipelined transformer for UMAP projections\n \"\"\"\n\n # Create the pipeline steps\n steps = []\n\n # Add the UMAP manifold\n steps.append(('umap', UMAP(\n n_components=2, random_state=self.random_state, **umap_kwargs)))\n\n # return the pipeline\n return Pipeline(steps)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit(self, X, y=None, **kwargs):\n\n # Store the classes we observed in y\n if y is not None:\n self.classes_ = np.unique(y)\n elif y is None and self.labels is not None:\n self.classes_ = np.array([self.labels[0]])\n else:\n self.classes_ = np.array([self.NULL_CLASS])\n\n # Fit our internal transformer and transform the data.\n vecs = self.transformer_.fit_transform(X)\n self.n_instances_ = vecs.shape[0]\n\n # Draw the vectors\n self.draw(vecs, y, **kwargs)\n\n # Fit always returns self.\n return self", "response": "Fit the UMAP to the data X and draw the vectors and return the instance of the UMAP class."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw(self, points, target=None, **kwargs):\n # Resolve the labels with the classes\n labels = self.labels if self.labels is not None else self.classes_\n if len(labels) != len(self.classes_):\n raise YellowbrickValueError((\n \"number of supplied labels ({}) does not \"\n \"match the number of classes ({})\"\n ).format(len(labels), len(self.classes_)))\n\n\n # Create the color mapping for the labels.\n self.color_values_ = resolve_colors(\n n_colors=len(labels), colormap=self.colormap, colors=self.colors)\n colors = dict(zip(labels, self.color_values_))\n\n # Transform labels into a map of class to label\n labels = dict(zip(self.classes_, labels))\n\n # Expand the points into vectors of x and y for scatter plotting,\n # assigning them to their label if the label has been passed in.\n # Additionally, filter classes not specified directly by the user.\n series = defaultdict(lambda: {'x':[], 'y':[]})\n\n if target is not None:\n for t, point in zip(target, points):\n label = labels[t]\n series[label]['x'].append(point[0])\n series[label]['y'].append(point[1])\n else:\n label = self.classes_[0]\n for x,y in points:\n series[label]['x'].append(x)\n series[label]['y'].append(y)\n\n # Plot the points\n for label, points in series.items():\n self.ax.scatter(\n points['x'], points['y'], c=colors[label],\n alpha=self.alpha, label=label\n )", "response": "This method draws the UMAP scatter plot from a set of decomposed points."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate an explained variance plot of the dimensionality reduction algorithm using n = 1 to n_components dimensions. This is a single plot that shows the best trade off between the number of features retained within the data.", "response": "def explained_variance_visualizer(X, y=None, ax=None, scale=True, \n center=True, colormap=palettes.DEFAULT_SEQUENCE,\n **kwargs):\n \"\"\"Produce a plot of the explained variance produced by a dimensionality \n reduction algorithm using n=1 to n=n_components dimensions. This is a single \n plot to help identify the best trade off between number of dimensions\n and amount of information retained within the data.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n rows with m features\n\n y : ndarray or Series of length n\n An array or Series of target or class values\n \n ax : matplotlib Axes, default: None\n The aces to plot the figure on\n\n scale : bool, default: True\n Boolean that indicates if the values of X should be scaled.\n\n colormap : string or cmap, default: None\n optional string or matplotlib cmap to colorize lines\n Use either color to colorize the lines on a per class basis or\n colormap to color them on a continuous scale.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n \n Examples\n --------\n >>> from sklearn import datasets\n >>> bc = datasets.load_breast_cancer()\n >>> X = bc = bc.data\n >>> explained_variance_visualizer(X, scale=True, center=True, colormap='RdBu_r')\n \n \"\"\"\n\n # Instantiate the visualizer\n visualizer = ExplainedVariance(X=X)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.poof()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef roc_auc(model, X, y=None, ax=None, **kwargs):\n # Instantiate the visualizer\n visualizer = ROCAUC(model, ax, **kwargs)\n\n # Create the train and test splits\n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X_train, y_train, **kwargs)\n visualizer.score(X_test, y_test)\n visualizer.finalize()\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "This function is used to plot ROCAUC curves for the classifier with the same target vector and the same class. The ROC curves are plotted from the sum of all true positives and false positives across all classes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef score(self, X, y=None, **kwargs):\n\n # Compute the predictions for the test data\n y_pred = self._get_y_scores(X)\n\n # Note: In the above, _get_y_scores calls either a decision_function or\n # predict_proba, which should return a 2D array. But in a binary\n # classification using an estimator with only a decision_function, y_pred\n # will instead be 1D, meaning only one curve can be plotted. In this case,\n # we set the _binary_decision attribute to True to ensure only one curve is\n # computed and plotted later on.\n if y_pred.ndim == 1:\n self._binary_decision = True\n\n # Raise an error if it's a binary decision and user has set micro,\n # macro, or per_class to True\n if self.micro or self.macro or self.per_class:\n raise ModelError(\n \"Micro, macro, and per-class scores are not defined for \"\n \"binary classification for estimators with only \"\n \"decision_function methods; set micro, macro, and \"\n \"per-class params to False.\"\n )\n else:\n self._binary_decision = False\n # If it's not a binary decision, at least one of micro, macro, or\n # per_class must be True\n if not self.micro and not self.macro and not self.per_class:\n raise YellowbrickValueError(\n \"no curves will be drawn; specify micro, macro, or per_class\"\n )\n\n # Classes may be label encoded so only use what's in y to compute.\n # The self.classes_ attribute will be used as names for labels.\n classes = np.unique(y)\n n_classes = len(classes)\n\n # Store the false positive rate, true positive rate and curve info.\n self.fpr = dict()\n self.tpr = dict()\n self.roc_auc = dict()\n\n # If the decision is binary, compute the ROC curve and ROC area\n if self._binary_decision == True:\n self.fpr[0], self.tpr[0], _ = roc_curve(y, y_pred)\n self.roc_auc[0] = auc(self.fpr[0], self.tpr[0])\n else:\n # Otherwise compute the ROC curve and ROC area for each class\n for i, c in enumerate(classes):\n self.fpr[i], self.tpr[i], _ = roc_curve(y, y_pred[:,i], pos_label=c)\n self.roc_auc[i] = auc(self.fpr[i], self.tpr[i])\n\n # Compute micro average\n if self.micro:\n self._score_micro_average(y, y_pred, classes, n_classes)\n\n # Compute macro average\n if self.macro:\n self._score_macro_average(n_classes)\n\n # Draw the Curves\n self.draw()\n\n # Set score to micro average if specified\n if self.micro:\n self.score_ = self.roc_auc[MICRO]\n\n # Set score to macro average if not micro\n if self.macro:\n self.score_ = self.roc_auc[MACRO]\n\n # Set score to the base score if neither macro nor micro\n self.score_ = self.estimator.score(X, y)\n\n return self.score_", "response": "Generates the predicted target values using the Scikit - Learn - Critical Log - Learning estimator."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrender the ROC curves for the current class and returns the plotted figure.", "response": "def draw(self):\n \"\"\"\n Renders ROC-AUC plot.\n Called internally by score, possibly more than once\n\n Returns\n -------\n ax : the axis with the plotted figure\n \"\"\"\n colors = self.colors[0:len(self.classes_)]\n n_classes = len(colors)\n\n # If it's a binary decision, plot the single ROC curve\n if self._binary_decision == True:\n self.ax.plot(\n self.fpr[0], self.tpr[0],\n label='ROC for binary decision, AUC = {:0.2f}'.format(\n self.roc_auc[0]\n )\n )\n\n # If per-class plotting is requested, plot ROC curves for each class\n if self.per_class:\n for i, color in zip(range(n_classes), colors):\n self.ax.plot(\n self.fpr[i], self.tpr[i], color=color,\n label='ROC of class {}, AUC = {:0.2f}'.format(\n self.classes_[i], self.roc_auc[i],\n )\n )\n\n # If requested, plot the ROC curve for the micro average\n if self.micro:\n self.ax.plot(\n self.fpr[MICRO], self.tpr[MICRO], linestyle=\"--\",\n color= self.colors[len(self.classes_)-1],\n label='micro-average ROC curve, AUC = {:0.2f}'.format(\n self.roc_auc[\"micro\"],\n )\n )\n\n # If requested, plot the ROC curve for the macro average\n if self.macro:\n self.ax.plot(\n self.fpr[MACRO], self.tpr[MACRO], linestyle=\"--\",\n color= self.colors[len(self.classes_)-1],\n label='macro-average ROC curve, AUC = {:0.2f}'.format(\n self.roc_auc[\"macro\"],\n )\n )\n\n # Plot the line of no discrimination to compare the curve to.\n self.ax.plot([0,1], [0,1], linestyle=':', c=LINE_COLOR)\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef finalize(self, **kwargs):\n # Set the title and add the legend\n self.set_title('ROC Curves for {}'.format(self.name))\n self.ax.legend(loc='lower right', frameon=True)\n\n # Set the limits for the ROC/AUC (always between 0 and 1)\n self.ax.set_xlim([0.0, 1.0])\n self.ax.set_ylim([0.0, 1.0])\n\n # Set x and y axis labels\n self.ax.set_ylabel('True Postive Rate')\n self.ax.set_xlabel('False Positive Rate')", "response": "Executes any subclass - specific axes finalization steps."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_y_scores(self, X):\n # The resolution order of scoring functions\n attrs = (\n 'predict_proba',\n 'decision_function',\n )\n\n # Return the first resolved function\n for attr in attrs:\n try:\n method = getattr(self.estimator, attr, None)\n if method:\n return method(X)\n except AttributeError:\n # Some Scikit-Learn estimators have both probability and\n # decision functions but override __getattr__ and raise an\n # AttributeError on access.\n # Note that because of the ordering of our attrs above,\n # estimators with both will *only* ever use probability.\n continue\n\n # If we've gotten this far, raise an error\n raise ModelError(\n \"ROCAUC requires estimators with predict_proba or \"\n \"decision_function methods.\"\n )", "response": "This method computes the target scores for the ROCAUC metric."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _score_micro_average(self, y, y_pred, classes, n_classes):\n # Convert y to binarized array for micro and macro scores\n y = label_binarize(y, classes=classes)\n if n_classes == 2:\n y = np.hstack((1-y, y))\n\n # Compute micro-average\n self.fpr[MICRO], self.tpr[MICRO], _ = roc_curve(y.ravel(), y_pred.ravel())\n self.roc_auc[MICRO] = auc(self.fpr[MICRO], self.tpr[MICRO])", "response": "Compute the micro average scores for the ROCAUC curves."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _score_macro_average(self, n_classes):\n # Gather all FPRs\n all_fpr = np.unique(np.concatenate([self.fpr[i] for i in range(n_classes)]))\n avg_tpr = np.zeros_like(all_fpr)\n\n # Compute the averages per class\n for i in range(n_classes):\n avg_tpr += interp(all_fpr, self.fpr[i], self.tpr[i])\n\n # Finalize the average\n avg_tpr /= n_classes\n\n # Store the macro averages\n self.fpr[MACRO] = all_fpr\n self.tpr[MACRO] = avg_tpr\n self.roc_auc[MACRO] = auc(self.fpr[MACRO], self.tpr[MACRO])", "response": "Compute the macro average scores for the ROCAUC curves."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_estimator(model):\n if inspect.isclass(model):\n return issubclass(model, BaseEstimator)\n\n return isinstance(model, BaseEstimator)", "response": "Determines if a model is an estimator using issubclass and isinstance."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_gridsearch(estimator):\n\n from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n\n if inspect.isclass(estimator):\n return issubclass(estimator, (GridSearchCV, RandomizedSearchCV))\n\n return isinstance(estimator, (GridSearchCV, RandomizedSearchCV))", "response": "Returns True if the given estimator is a Scikit - Learn clusterer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if the given object is a Pandas Data Frame.", "response": "def is_dataframe(obj):\n \"\"\"\n Returns True if the given object is a Pandas Data Frame.\n\n Parameters\n ----------\n obj: instance\n The object to test whether or not is a Pandas DataFrame.\n \"\"\"\n try:\n # This is the best method of type checking\n from pandas import DataFrame\n return isinstance(obj, DataFrame)\n except ImportError:\n # Pandas is not a dependency, so this is scary\n return obj.__class__.__name__ == \"DataFrame\""} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_series(obj):\n try:\n # This is the best method of type checking\n from pandas import Series\n return isinstance(obj, Series)\n except ImportError:\n # Pandas is not a dependency, so this is scary\n return obj.__class__.__name__ == \"Series\"", "response": "Returns True if the given object is a Pandas Series."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn True if the given object is a Numpy Structured Array.", "response": "def is_structured_array(obj):\n \"\"\"\n Returns True if the given object is a Numpy Structured Array.\n\n Parameters\n ----------\n obj: instance\n The object to test whether or not is a Numpy Structured Array.\n \"\"\"\n if isinstance(obj, np.ndarray) and hasattr(obj, 'dtype'):\n if obj.dtype.names is not None:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nplotting the actual targets from the dataset against the predicted values generated by the model.", "response": "def prediction_error(model, X, y=None, ax=None, alpha=0.75, **kwargs):\n \"\"\"\n Quick method:\n\n Plot the actual targets from the dataset against the\n predicted values generated by our model(s).\n\n This helper function is a quick wrapper to utilize the PredictionError\n ScoreVisualizer for one-off analysis.\n\n Parameters\n ----------\n model : the Scikit-Learn estimator (should be a regressor)\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features.\n\n y : ndarray or Series of length n\n An array or series of target or class values.\n\n ax : matplotlib Axes\n The axes to plot the figure on.\n\n shared_limits : bool, default: True\n If shared_limits is True, the range of the X and Y axis limits will\n be identical, creating a square graphic with a true 45 degree line.\n In this form, it is easier to diagnose under- or over- prediction,\n though the figure will become more sparse. To localize points, set\n shared_limits to False, but note that this will distort the figure\n and should be accounted for during analysis.\n\n besfit : bool, default: True\n Draw a linear best fit line to estimate the correlation between the\n predicted and measured value of the target variable. The color of\n the bestfit line is determined by the ``line_color`` argument.\n\n identity: bool, default: True\n Draw the 45 degree identity line, y=x in order to better show the\n relationship or pattern of the residuals. E.g. to estimate if the\n model is over- or under- estimating the given values. The color of the\n identity line is a muted version of the ``line_color`` argument.\n\n point_color : color\n Defines the color of the error points; can be any matplotlib color.\n\n line_color : color\n Defines the color of the best fit line; can be any matplotlib color.\n\n alpha : float, default: 0.75\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Returns\n -------\n ax : matplotlib Axes\n Returns the axes that the prediction error plot was drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = PredictionError(model, ax, alpha=alpha, **kwargs)\n\n # Create the train and test splits\n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X_train, y_train, **kwargs)\n visualizer.score(X_test, y_test)\n visualizer.finalize()\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nplot the residuals for the given model on the given axes.", "response": "def residuals_plot(model,\n X,\n y,\n ax=None,\n hist=True,\n test_size=0.25,\n train_color='b',\n test_color='g',\n line_color=LINE_COLOR,\n random_state=None,\n train_alpha=0.75,\n test_alpha=0.75,\n **kwargs):\n \"\"\"Quick method:\n\n Divides the dataset X, y into a train and test split (the size of the\n splits determined by test_size) then plots the training and test residuals\n agains the predicted value for the given model.\n\n This helper function is a quick wrapper to utilize the ResidualsPlot\n ScoreVisualizer for one-off analysis.\n\n Parameters\n ----------\n model : a Scikit-Learn regressor\n Should be an instance of a regressor, otherwise will raise a\n YellowbrickTypeError exception on instantiation.\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features.\n\n y : ndarray or Series of length n\n An array or series of target or class values.\n\n ax : matplotlib Axes, default: None\n The axes to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n hist : {True, False, None, 'density', 'frequency'}, default: True\n Draw a histogram showing the distribution of the residuals on the\n right side of the figure. Requires Matplotlib >= 2.0.2.\n If set to 'density', the probability density function will be plotted.\n If set to True or 'frequency' then the frequency will be plotted.\n\n test_size : float, int default: 0.25\n If float, should be between 0.0 and 1.0 and represent the proportion\n of the dataset to include in the test split. If int, represents the\n absolute number of test samples.\n\n train_color : color, default: 'b'\n Residuals for training data are ploted with this color but also\n given an opacity of 0.5 to ensure that the test data residuals\n are more visible. Can be any matplotlib color.\n\n test_color : color, default: 'g'\n Residuals for test data are plotted with this color. In order to\n create generalizable models, reserved test data residuals are of\n the most analytical interest, so these points are highlighted by\n having full opacity. Can be any matplotlib color.\n\n line_color : color, default: dark grey\n Defines the color of the zero error line, can be any matplotlib color.\n\n random_state : int, RandomState instance or None, optional\n Passed to the train_test_split function.\n\n train_alpha : float, default: 0.75\n Specify a transparency for traininig data, where 1 is completely opaque \n and 0 is completely transparent. This property makes densely clustered \n points more visible.\n \n test_alpha : float, default: 0.75\n Specify a transparency for test data, where 1 is completely opaque and \n 0 is completely transparent. This property makes densely clustered \n points more visible.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the residuals plot was drawn on.\n \"\"\"\n # Instantiate the visualizer\n \n visualizer = ResidualsPlot(\n model=model, ax=ax, hist=hist, train_color=train_color,\n test_color=test_color, line_color=line_color, \n train_alpha=train_alpha,test_alpha=test_alpha,\n **kwargs\n )\n\n # Create the train and test splits\n X_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=test_size, random_state=random_state\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X_train, y_train, **kwargs)\n visualizer.score(X_test, y_test)\n visualizer.finalize()\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef draw(self, y, y_pred):\n \n \"\"\"\n Parameters\n ----------\n y : ndarray or Series of length n\n An array or series of target or class values\n\n y_pred : ndarray or Series of length n\n An array or series of predicted target values\n\n Returns\n ------\n ax : the axis with the plotted figure\n \"\"\"\n label = \"$R^2 = {:0.3f}$\".format(self.score_)\n self.ax.scatter(\n y,\n y_pred,\n c=self.colors['point'],\n alpha=self.alpha,\n label=label)\n\n # TODO If score is happening inside a loop, draw would get called multiple times.\n # Ideally we'd want the best fit line to be drawn only once\n if self.bestfit:\n draw_best_fit(\n y, y_pred, self.ax, 'linear', ls='--', lw=2,\n c=self.colors['line'], label='best fit'\n )\n\n # Set the axes limits based on the range of X and Y data\n # NOTE: shared_limits will be accounted for in finalize()\n # TODO: do better than add one for really small residuals\n self.ax.set_xlim(y.min()-1, y.max()+1)\n self.ax.set_ylim(y_pred.min()-1, y_pred.max()+1)\n\n return self.ax", "response": "Draws the n - term data for the current object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfinalize executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n \"\"\"\n # Set the title on the plot\n self.set_title(\n 'Prediction Error for {}'.format(self.name)\n )\n\n # Square the axes to ensure a 45 degree line\n if self.shared_limits:\n # Get the current limits\n ylim = self.ax.get_ylim()\n xlim = self.ax.get_xlim()\n\n # Find the range that captures all data\n bounds = (\n min(ylim[0], xlim[0]),\n max(ylim[1], xlim[1]),\n )\n\n # Reset the limits\n self.ax.set_xlim(bounds)\n self.ax.set_ylim(bounds)\n\n # Ensure the aspect ratio is square\n self.ax.set_aspect('equal', adjustable='box')\n\n # Draw the 45 degree line\n if self.identity:\n draw_identity_line(\n ax=self.ax, ls='--', lw=2, c=self.colors['line'],\n alpha=0.5, label=\"identity\"\n )\n\n # Set the axes labels\n self.ax.set_ylabel(r'$\\hat{y}$')\n self.ax.set_xlabel(r'$y$')\n\n # Set the legend\n # Note: it would be nice to be able to use the manual_legend utility\n # here, since if the user sets a low alpha value, the R2 color in the\n # legend will also become more translucent. Unfortunately this is a\n # bit tricky because adding a manual legend here would override the\n # best fit and 45 degree line legend components. In particular, the\n # best fit is plotted in draw because it depends on y and y_pred.\n self.ax.legend(loc='best', frameon=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef hax(self):\n if make_axes_locatable is None:\n raise YellowbrickValueError((\n \"residuals histogram requires matplotlib 2.0.2 or greater \"\n \"please upgrade matplotlib or set hist=False on the visualizer\"\n ))\n\n divider = make_axes_locatable(self.ax)\n\n hax = divider.append_axes(\"right\", size=1, pad=0.1, sharey=self.ax)\n hax.yaxis.tick_right()\n hax.grid(False, axis='x')\n\n return hax", "response": "Returns the axes that are used to create the histogram for the current instance of the class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfitting the residuals plot to the set of target values.", "response": "def fit(self, X, y, **kwargs):\n \"\"\"\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target values\n\n kwargs: keyword arguments passed to Scikit-Learn API.\n\n Returns\n -------\n self : visualizer instance\n \"\"\"\n super(ResidualsPlot, self).fit(X, y, **kwargs)\n self.score(X, y, train=True)\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef score(self, X, y=None, train=False, **kwargs):\n score = self.estimator.score(X, y, **kwargs)\n if train:\n self.train_score_ = score\n else:\n self.test_score_ = score\n\n y_pred = self.predict(X)\n scores = y_pred - y\n self.draw(y_pred, scores, train=train)\n \n return score", "response": "Generates predicted target values using the Scikit - Learn\n estimator."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef draw(self, y_pred, residuals, train=False, **kwargs):\n\n if train:\n color = self.colors['train_point']\n label = \"Train $R^2 = {:0.3f}$\".format(self.train_score_)\n alpha = self.alphas['train_point']\n else:\n color = self.colors['test_point']\n label = \"Test $R^2 = {:0.3f}$\".format(self.test_score_)\n alpha = self.alphas['test_point']\n \n # Update the legend information\n self._labels.append(label)\n self._colors.append(color)\n\n # Draw the residuals scatter plot\n self.ax.scatter(\n y_pred, residuals, c=color, alpha=alpha, label=label\n )\n\n # Add residuals histogram\n if self.hist in {True, 'frequency'}:\n self.hax.hist(residuals, bins=50, orientation=\"horizontal\", color=color)\n elif self.hist == 'density':\n self.hax.hist(\n residuals, bins=50, orientation=\"horizontal\", density=True, color=color\n )\n\n # Ensure the current axes is always the main residuals axes\n plt.sca(self.ax)\n return self.ax", "response": "Draw the residuals against the predicted value for the specified split."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfinalize executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n \"\"\"\n # Add the title to the plot\n self.set_title('Residuals for {} Model'.format(self.name))\n\n # Set the legend with full opacity patches using manual legend\n manual_legend(\n self, self._labels, self._colors, loc='best', frameon=True\n )\n\n # Create a full line across the figure at zero error.\n self.ax.axhline(y=0, c=self.colors['line'])\n\n # Set the axes labels\n self.ax.set_ylabel('Residuals')\n self.ax.set_xlabel(\"Predicted Value\")\n\n # Finalize the histogram axes\n if self.hist:\n self.hax.axhline(y=0, c=self.colors['line'])\n self.hax.set_xlabel(\"Distribution\")"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfits the visualizer to the given set of instances described by X and binary classes described in y.", "response": "def fit(self, X, y, **kwargs):\n \"\"\"\n Fit is the entry point for the visualizer. Given instances described\n by X and binary classes described in the target y, fit performs n\n trials by shuffling and splitting the dataset then computing the\n precision, recall, f1, and queue rate scores for each trial. The\n scores are aggregated by the quantiles expressed then drawn.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values. The target y must\n be a binary classification target.\n\n kwargs: dict\n keyword arguments passed to Scikit-Learn API.\n\n Returns\n -------\n self : instance\n Returns the instance of the visualizer\n\n raises: YellowbrickValueError\n If the target y is not a binary classification target.\n \"\"\"\n # Check target before metrics raise crazy exceptions\n if type_of_target(y) != 'binary':\n raise YellowbrickValueError(\"multiclass format is not supported\")\n\n # Make arrays indexable for cross validation\n X, y = indexable(X, y)\n\n # TODO: parallelize trials with joblib (using sklearn utility)\n # NOTE: parallelization with matplotlib is tricy at best!\n trials = [\n metric\n for idx in range(self.n_trials)\n for metric in self._split_fit_score_trial(X, y, idx)\n ]\n\n # Compute maximum number of uniform thresholds across all trials\n n_thresholds = np.array([len(t['thresholds']) for t in trials]).min()\n self.thresholds_ = np.linspace(0.0, 1.0, num=n_thresholds)\n\n # Filter metrics and collect values for uniform thresholds\n metrics = frozenset(METRICS) - self._check_exclude(self.exclude)\n uniform_metrics = defaultdict(list)\n\n for trial in trials:\n rows = defaultdict(list)\n for t in self.thresholds_:\n idx = bisect.bisect_left(trial['thresholds'], t)\n for metric in metrics:\n rows[metric].append(trial[metric][idx])\n\n for metric, row in rows.items():\n uniform_metrics[metric].append(row)\n\n # Convert metrics to metric arrays\n uniform_metrics = {\n metric: np.array(values)\n for metric, values in uniform_metrics.items()\n }\n\n # Perform aggregation and store cv_scores_\n quantiles = self._check_quantiles(self.quantiles)\n self.cv_scores_ = {}\n\n for metric, values in uniform_metrics.items():\n # Compute the lower, median, and upper plots\n lower, median, upper = mstats.mquantiles(\n values, prob=quantiles, axis=0\n )\n\n # Store the aggregates in cv scores\n self.cv_scores_[metric] = median\n self.cv_scores_[\"{}_lower\".format(metric)] = lower\n self.cv_scores_[\"{}_upper\".format(metric)] = upper\n\n # Draw and always return self\n self.draw()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsplitting the dataset X y and scores it by fitting the model and then scores it according to the required metrics.", "response": "def _split_fit_score_trial(self, X, y, idx=0):\n \"\"\"\n Splits the dataset, fits a clone of the estimator, then scores it\n according to the required metrics.\n\n The index of the split is added to the random_state if the\n random_state is not None; this ensures that every split is shuffled\n differently but in a deterministic fashion for testing purposes.\n \"\"\"\n random_state = self.random_state\n if random_state is not None:\n random_state += idx\n\n splitter = self._check_cv(self.cv, random_state)\n\n for train_index, test_index in splitter.split(X, y):\n # Safe indexing handles multiple types of inputs including\n # DataFrames and structured arrays - required for generic splits.\n X_train = safe_indexing(X, train_index)\n y_train = safe_indexing(y, train_index)\n X_test = safe_indexing(X, test_index)\n y_test = safe_indexing(y, test_index)\n\n model = clone(self.estimator)\n model.fit(X_train, y_train)\n\n if hasattr(model, \"predict_proba\"):\n # Get the probabilities for the positive class\n y_scores = model.predict_proba(X_test)[:,1]\n else:\n # Use the decision function to get the scores\n y_scores = model.decision_function(X_test)\n\n # Compute the curve metrics and thresholds\n curve_metrics = precision_recall_curve(y_test, y_scores)\n precision, recall, thresholds = curve_metrics\n\n # Compute the F1 score from precision and recall\n # Don't need to warn for F, precision/recall would have warned\n with np.errstate(divide='ignore', invalid='ignore'):\n beta = self.fbeta ** 2\n f_score = ((1 + beta) * precision * recall /\n (beta * precision + recall))\n\n # Ensure thresholds ends at 1\n thresholds = np.append(thresholds, 1)\n\n # Compute the queue rate\n queue_rate = np.array([\n (y_scores >= threshold).mean()\n for threshold in thresholds\n ])\n\n yield {\n 'thresholds': thresholds,\n 'precision': precision,\n 'recall': recall,\n 'fscore': f_score,\n 'queue_rate': queue_rate\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef draw(self):\n # Set the colors from the supplied values or reasonable defaults\n color_values = resolve_colors(n_colors=4, colors=self.color)\n\n for idx, metric in enumerate(METRICS):\n # Skip any excluded labels\n if metric not in self.cv_scores_:\n continue\n\n # Get the color ensuring every metric has a static color\n color = color_values[idx]\n\n # Make the label pretty\n if metric == \"fscore\":\n if self.fbeta == 1.0:\n label = \"$f_1$\"\n else:\n label = \"$f_{{\\beta={:0.1f}}}\".format(self.fbeta)\n else:\n label = metric.replace(\"_\", \" \")\n\n # Draw the metric values\n self.ax.plot(\n self.thresholds_, self.cv_scores_[metric],\n color=color, label=label\n )\n\n # Draw the upper and lower bounds\n lower = self.cv_scores_[\"{}_lower\".format(metric)]\n upper = self.cv_scores_[\"{}_upper\".format(metric)]\n\n self.ax.fill_between(\n self.thresholds_, upper, lower,\n alpha=0.35, linewidth=0, color=color\n )\n\n # Annotate the graph with the maximizing value\n if self.argmax.lower() == metric:\n argmax = self.cv_scores_[metric].argmax()\n threshold = self.thresholds_[argmax]\n self.ax.axvline(\n threshold, ls='--', c='k', lw=1,\n label=\"$t_{}={:0.2f}$\".format(metric[0], threshold)\n )\n\n return self.ax", "response": "Draws cv scores as a line chart on the current axes."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _check_quantiles(self, val):\n if len(val) != 3 or not is_monotonic(val) or not np.all(val < 1):\n raise YellowbrickValueError(\n \"quantiles must be a sequence of three \"\n \"monotonically increasing values less than 1\"\n )\n return np.asarray(val)", "response": "Validate the quantiles passed in. Returns the np array if valid."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nvalidates the cv method passed in. Returns the split strategy if validation exception is raised.", "response": "def _check_cv(self, val, random_state=None):\n \"\"\"\n Validate the cv method passed in. Returns the split strategy if no\n validation exception is raised.\n \"\"\"\n # Use default splitter in this case\n if val is None: val = 0.1\n\n if isinstance(val, float) and val <= 1.0:\n return ShuffleSplit(\n n_splits=1, test_size=val, random_state=random_state\n )\n\n if hasattr(val, \"split\") and hasattr(val, \"get_n_splits\"):\n if random_state is not None and hasattr(val, \"random_state\"):\n val.random_state = random_state\n return val\n\n raise YellowbrickValueError(\n \"'{}' is not a valid cv splitter\".format(type(val))\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_exclude(self, val):\n if val is None:\n exclude = frozenset()\n elif isinstance(val, str):\n exclude = frozenset([val.lower()])\n else:\n exclude = frozenset(map(lambda s: s.lower(), val))\n\n if len(exclude - frozenset(METRICS)) > 0:\n raise YellowbrickValueError(\n \"'{}' is not a valid metric to exclude\".format(repr(val))\n )\n\n return exclude", "response": "Validate the excluded metrics. Returns the set of excluded params."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef decisionviz(model,\n X,\n y,\n colors=None,\n classes=None,\n features=None,\n show_scatter=True,\n step_size=0.0025,\n markers=None,\n pcolormesh_alpha=0.8,\n scatter_alpha=1.0,\n title=None,\n **kwargs):\n \"\"\"DecisionBoundariesVisualizer is a bivariate data visualization algorithm\n that plots the decision boundaries of each class.\n\n This helper function is a quick wrapper to utilize the\n DecisionBoundariesVisualizers for one-off analysis.\n\n Parameters\n ----------\n model : the Scikit-Learn estimator, required\n Should be an instance of a classifier, else the __init__ will\n return an error.\n\n x : matrix, required\n The feature name that corresponds to a column name or index postion\n in the matrix that will be plotted against the x-axis\n\n y : array, required\n The feature name that corresponds to a column name or index postion\n in the matrix that will be plotted against the y-axis\n\n classes : a list of class names for the legend, default: None\n If classes is None and a y value is passed to fit then the classes\n are selected from the target vector.\n\n features : list of strings, default: None\n The names of the features or columns\n\n show_scatter : boolean, default: True\n If boolean is True, then a scatter plot with points will be drawn\n on top of the decision boundary graph\n\n step_size : float percentage, default: 0.0025\n Determines the step size for creating the numpy meshgrid that will\n later become the foundation of the decision boundary graph. The\n default value of 0.0025 means that the step size for constructing\n the meshgrid will be 0.25%% of differenes of the max and min of x\n and y for each feature.\n\n markers : iterable of strings, default: ,od*vh+\n Matplotlib style markers for points on the scatter plot points\n\n pcolormesh_alpha : float, default: 0.8\n Sets the alpha transparency for the meshgrid of model boundaries\n\n scatter_alpha : float, default: 1.0\n Sets the alpha transparency for the scatter plot points\n\n title : string, default: stringified feature_one and feature_two\n Sets the title of the visualization\n\n kwargs : keyword arguments passed to the super class.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the decision boundaries graph were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = DecisionBoundariesVisualizer(model,\n X,\n y,\n colors=colors,\n classes=classes,\n features=features,\n show_scatter=show_scatter,\n step_size=step_size,\n markers=markers,\n pcolormesh_alpha=pcolormesh_alpha,\n scatter_alpha=scatter_alpha,\n title=title,\n **kwargs)\n\n # Fit, draw and poof the visualizer\n visualizer.fit_draw_poof(X, y, **kwargs)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "This function is a quick wrapper to utilize the decision boundaries visualizer for one - off analysis."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_text_color(base_color, dark_color=\"black\", light_color=\"white\", coef_choice=0):\n\n #Coefficients:\n # option 0: http://www.nbdtech.com/Blog/archive/2008/04/27/Calculating-the-Perceived-Brightness-of-a-Color.aspx\n # option 1: http://stackoverflow.com/questions/596216/formula-to-determine-brightness-of-rgb-color\n coef_options = [np.array((.241, .691, .068, 0)),\n np.array((.299, .587, .114, 0))\n ]\n\n coefs= coef_options[coef_choice]\n rgb = np.array(base_color) * 255\n brightness = np.sqrt(np.dot(coefs, rgb**2))\n\n #Threshold from option 0 link; determined by trial and error.\n #base is light\n if brightness > 130:\n return dark_color\n return light_color", "response": "This function returns the appropriate light or dark text color for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef feature_importances(model, X, y=None, ax=None, labels=None,\n relative=True, absolute=False, xlabel=None,\n stack=False, **kwargs):\n \"\"\"\n Displays the most informative features in a model by showing a bar chart\n of features ranked by their importances. Although primarily a feature\n engineering mechanism, this visualizer requires a model that has either a\n ``coef_`` or ``feature_importances_`` parameter after fit.\n\n Parameters\n ----------\n model : Estimator\n A Scikit-Learn estimator that learns feature importances. Must support\n either ``coef_`` or ``feature_importances_`` parameters.\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n, optional\n An array or series of target or class values\n\n ax : matplotlib Axes, default: None\n The axis to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n labels : list, default: None\n A list of feature names to use. If a DataFrame is passed to fit and\n features is None, feature names are selected as the column names.\n\n relative : bool, default: True\n If true, the features are described by their relative importance as a\n percentage of the strongest feature component; otherwise the raw\n numeric description of the feature importance is shown.\n\n absolute : bool, default: False\n Make all coeficients absolute to more easily compare negative\n coeficients with positive ones.\n\n xlabel : str, default: None\n The label for the X-axis. If None is automatically determined by the\n underlying model and options provided.\n\n stack : bool, default: False\n If true and the classifier returns multi-class feature importance,\n then a stacked bar plot is plotted; otherwise the mean of the\n feature importance across classes are plotted.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = FeatureImportances(\n model, ax, labels, relative, absolute, xlabel, stack, **kwargs)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y)\n visualizer.finalize()\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Displays the most informative features in a model by ranking them by their importances."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X, y=None, **kwargs):\n super(FeatureImportances, self).fit(X, y, **kwargs)\n\n # Get the feature importances from the model\n self.feature_importances_ = self._find_importances_param()\n\n # Get the classes from the model\n if is_classifier(self):\n self.classes_ = self._find_classes_param()\n else:\n self.classes_ = None\n self.stack = False\n\n # If self.stack = True and feature importances is a multidim array,\n # we're expecting a shape of (n_classes, n_features)\n # therefore we flatten by taking the average by\n # column to get shape (n_features,) (see LogisticRegression)\n if not self.stack and self.feature_importances_.ndim > 1:\n self.feature_importances_ = np.mean(self.feature_importances_, axis=0)\n warnings.warn((\n \"detected multi-dimensional feature importances but stack=False, \"\n \"using mean to aggregate them.\"\n ), YellowbrickWarning)\n\n # Apply absolute value filter before normalization\n if self.absolute:\n self.feature_importances_ = np.abs(self.feature_importances_)\n\n # Normalize features relative to the maximum\n if self.relative:\n maxv = np.abs(self.feature_importances_).max()\n self.feature_importances_ /= maxv\n self.feature_importances_ *= 100.0\n\n # Create labels for the feature importances\n # NOTE: this code is duplicated from MultiFeatureVisualizer\n if self.labels is None:\n # Use column names if a dataframe\n if is_dataframe(X):\n self.features_ = np.array(X.columns)\n\n # Otherwise use the column index as the labels\n else:\n _, ncols = X.shape\n self.features_ = np.arange(0, ncols)\n else:\n self.features_ = np.array(self.labels)\n\n # Sort the features and their importances\n if self.stack:\n sort_idx = np.argsort(np.mean(self.feature_importances_, 0))\n self.features_ = self.features_[sort_idx]\n self.feature_importances_ = self.feature_importances_[:, sort_idx]\n else:\n sort_idx = np.argsort(self.feature_importances_)\n self.features_ = self.features_[sort_idx]\n self.feature_importances_ = self.feature_importances_[sort_idx]\n\n # Draw the feature importances\n self.draw()\n return self", "response": "Fits the estimator to discover the feature importances described by the data X then draws those importances as a bar plot."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndraws the feature importances as a bar chart ; called from fit.", "response": "def draw(self, **kwargs):\n \"\"\"\n Draws the feature importances as a bar chart; called from fit.\n \"\"\"\n # Quick validation\n for param in ('feature_importances_', 'features_'):\n if not hasattr(self, param):\n raise NotFitted(\"missing required param '{}'\".format(param))\n\n # Find the positions for each bar\n pos = np.arange(self.features_.shape[0]) + 0.5\n\n # Plot the bar chart\n if self.stack:\n colors = color_palette(kwargs.pop('colors', None),\n len(self.classes_))\n zeros = np.zeros(self.feature_importances_.shape[1])\n left_arr = np.zeros((self.feature_importances_.shape[1], 2))\n\n for idx in range(len(self.feature_importances_)):\n left = [\n left_arr[j, int(self.feature_importances_[idx][j] > 0)]\n for j in range(len(self.feature_importances_[idx]))\n ]\n\n self.ax.barh(pos, self.feature_importances_[idx], left=left,\n color=colors[idx], label=self.classes_[idx])\n\n left_arr[:, 0] += np.minimum(self.feature_importances_[idx],\n zeros)\n left_arr[:, 1] += np.maximum(self.feature_importances_[idx],\n zeros)\n else:\n self.ax.barh(pos, self.feature_importances_, align='center')\n\n # Set the labels for the bars\n self.ax.set_yticks(pos)\n self.ax.set_yticklabels(self.features_)\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinalizes the drawing setting labels and title.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize the drawing setting labels and title.\n \"\"\"\n # Set the title\n self.set_title('Feature Importances of {} Features using {}'.format(\n len(self.features_), self.name))\n\n # Set the xlabel\n self.ax.set_xlabel(self._get_xlabel())\n\n # Remove the ygrid\n self.ax.grid(False, axis='y')\n\n if self.stack:\n plt.legend(bbox_to_anchor=(1.04, 0.5), loc=\"center left\")\n # Ensure we have a tight fit\n plt.tight_layout()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch the wrapped model for the classes_ parameter.", "response": "def _find_classes_param(self):\n \"\"\"\n Searches the wrapped model for the classes_ parameter.\n \"\"\"\n for attr in [\"classes_\"]:\n try:\n return getattr(self.estimator, attr)\n except AttributeError:\n continue\n\n raise YellowbrickTypeError(\n \"could not find classes_ param on {}\".format(\n self.estimator.__class__.__name__\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_xlabel(self):\n # Return user-specified label\n if self.xlabel:\n return self.xlabel\n\n # Label for coefficients\n if hasattr(self.estimator, \"coef_\"):\n if self.relative:\n return \"relative coefficient magnitude\"\n return \"coefficient value\"\n\n # Default label for feature_importances_\n if self.relative:\n return \"relative importance\"\n return \"feature importance\"", "response": "Determines the xlabel based on the underlying data structure\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef target_color_type(y):\n ttype = type_of_target(y)\n\n if ttype.startswith(CONTINUOUS):\n return CONTINUOUS\n\n if ttype.startswith(\"binary\"):\n return DISCRETE\n\n if ttype.startswith(\"multiclass\"):\n if len(np.unique(y)) > MAX_DISCRETE_CLASSES:\n return CONTINUOUS\n return DISCRETE\n\n return UNKNOWN", "response": "Determines the type of color space that will best represent the target variable y."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating the grid layout for the joint plot.", "response": "def _layout(self):\n \"\"\"\n Creates the grid layout for the joint plot, adding new axes for the histograms\n if necessary and modifying the aspect ratio. Does not modify the axes or the\n layout if self.hist is False or None.\n \"\"\"\n # Ensure the axes are created if not hist, then return.\n if not self.hist:\n self.ax\n return\n\n # Ensure matplotlib version compatibility\n if make_axes_locatable is None:\n raise YellowbrickValueError((\n \"joint plot histograms requires matplotlib 2.0.2 or greater \"\n \"please upgrade matplotlib or set hist=False on the visualizer\"\n ))\n\n # Create the new axes for the histograms\n divider = make_axes_locatable(self.ax)\n self._xhax = divider.append_axes(\"top\", size=1, pad=0.1, sharex=self.ax)\n self._yhax = divider.append_axes(\"right\", size=1, pad=0.1, sharey=self.ax)\n\n # Modify the display of the axes\n self._xhax.xaxis.tick_top()\n self._yhax.yaxis.tick_right()\n self._xhax.grid(False, axis='y')\n self._yhax.grid(False, axis='x')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, X, y=None):\n # Convert python objects to numpy arrays\n if isinstance(X, (list, tuple)):\n X = np.array(X)\n\n if y is not None and isinstance(y, (list, tuple)):\n y = np.array(y)\n\n # Case where no columns are specified\n if self.columns is None:\n if (y is None and (X.ndim != 2 or X.shape[1] != 2)) or (y is not None and (X.ndim != 1 or y.ndim != 1)):\n raise YellowbrickValueError((\n \"when self.columns is None specify either X and y as 1D arrays \"\n \"or X as a matrix with 2 columns\"\n ))\n\n if y is None:\n # Draw the first column as x and the second column as y\n self.draw(X[:,0], X[:,1], xlabel=\"0\", ylabel=\"1\")\n return self\n\n # Draw x against y\n self.draw(X, y, xlabel=\"x\", ylabel=\"y\")\n return self\n\n # Case where a single string or int index is specified\n if isinstance(self.columns, (int,str)):\n if y is None:\n raise YellowbrickValueError(\n \"when self.columns is a single index, y must be specified\"\n )\n\n # fetch the index from X -- raising index error if not possible\n x = self._index_into(self.columns, X)\n self.draw(x, y, xlabel=str(self.columns), ylabel=\"target\")\n return self\n\n # Case where there is a double index for both columns\n columns = tuple(self.columns)\n if len(columns) != 2:\n raise YellowbrickValueError((\n \"'{}' contains too many indices or is invalid for joint plot\"\n ).format(columns))\n\n # TODO: color the points based on the target if it is given\n x = self._index_into(columns[0], X)\n y = self._index_into(columns[1], X)\n self.draw(x, y, xlabel=str(columns[0]), ylabel=str(columns[1]))\n return self", "response": "Fits the JointPlot to create a correlative visualization between the columns X and y."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndraws the main joint plot for the data in x and y.", "response": "def draw(self, x, y, xlabel=None, ylabel=None):\n \"\"\"\n Draw the joint plot for the data in x and y.\n\n Parameters\n ----------\n x, y : 1D array-like\n The data to plot for the x axis and the y axis\n\n xlabel, ylabel : str\n The labels for the x and y axes.\n \"\"\"\n # This is a little weird to be here, but it is the best place to perform\n # this computation given how fit calls draw and returns.\n self.corr_ = self.correlation_methods[self.correlation](x, y)\n\n # First draw the joint plot\n joint_kws = self.joint_kws or {}\n joint_kws.setdefault(\"alpha\", self.alpha)\n joint_kws.setdefault(\"label\", \"{}={:0.3f}\".format(self.correlation, self.corr_))\n\n # Draw scatter joint plot\n if self.kind == \"scatter\":\n self.ax.scatter(x, y, **joint_kws)\n\n # TODO: Draw best fit line (or should this be kind='reg'?)\n\n # Draw hexbin joint plot\n elif self.kind in ('hex', 'hexbin'):\n joint_kws.setdefault(\"mincnt\", 1)\n joint_kws.setdefault(\"gridsize\", 50)\n self.ax.hexbin(x, y, **joint_kws)\n\n # Something bad happened\n else:\n raise ValueError(\"unknown joint plot kind '{}'\".format(self.kind))\n\n # Set the X and Y axis labels on the plot\n self.ax.set_xlabel(xlabel)\n self.ax.set_ylabel(ylabel)\n\n # If we're not going to draw histograms, stop here\n if not self.hist:\n # Ensure the current axes is always the main joint plot axes\n plt.sca(self.ax)\n return self.ax\n\n # Draw the histograms\n hist_kws = self.hist_kws or {}\n hist_kws.setdefault(\"bins\", 50)\n if self.hist == \"density\":\n hist_kws.setdefault(\"density\", True)\n\n self.xhax.hist(x, **hist_kws)\n self.yhax.hist(y, orientation=\"horizontal\", **hist_kws)\n\n # Ensure the current axes is always the main joint plot axes\n plt.sca(self.ax)\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef finalize(self, **kwargs):\n # Set the aspect ratio to make the visualization square\n # TODO: still unable to make plot square using make_axes_locatable\n # x0,x1 = self.ax.get_xlim()\n # y0,y1 = self.ax.get_ylim()\n # self.ax.set_aspect(abs(x1-x0)/abs(y1-y0))\n\n # Add the title to the plot if the user has set one.\n self.set_title(\"\")\n\n # Set the legend with full opacity patches using manual legend.\n # Or Add the colorbar if this is a continuous plot.\n self.ax.legend(loc=\"best\", frameon=True)\n\n # Finalize the histograms\n if self.hist:\n plt.setp(self.xhax.get_xticklabels(), visible=False)\n plt.setp(self.yhax.get_yticklabels(), visible=False)\n plt.sca(self.ax)\n\n # Call tight layout to maximize readability\n plt.tight_layout()", "response": "Finalize executes any remaining image modifications making it ready to show."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nattempts to get the column from the data using the specified index raises an IndexError if this is not possible from this point in the stack.", "response": "def _index_into(self, idx, data):\n \"\"\"\n Attempts to get the column from the data using the specified index, raises an\n exception if this is not possible from this point in the stack.\n \"\"\"\n try:\n if is_dataframe(data):\n # Assume column indexing\n return data[idx]\n # Otherwise assume numpy array-like indexing\n return data[:,idx]\n except Exception as e:\n raise IndexError(\n \"could not index column '{}' into type {}: {}\".format(\n self.columns, data.__class__.__name__, e\n ))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit(self, X, y=None, **kwargs):\n if is_dataframe(X):\n self.X = X.values\n if self.features_ is None:\n self.features_ = X.columns\n else:\n self.X = X\n\n self.y = y\n\n super(MissingDataVisualizer, self).fit(X, y, **kwargs)", "response": "Fit the missing data visualization to the data X and y."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cv_scores(model, X, y, ax=None, cv=None, scoring=None, **kwargs):\n\n # Initialize the visualizer\n visualizer = CVScores(model, ax=ax, cv=cv, scoring=scoring)\n\n # Fit and poof the visualizer\n visualizer.fit(X, y)\n visualizer.poof(**kwargs)\n return visualizer.ax", "response": "Generates a CVScores visualizer for one - off analysis of the base class."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfits the learning curve with the wrapped model to the specified data. Draws training and test score curves and saves the scores to the estimator.", "response": "def fit(self, X, y, **kwargs):\n \"\"\"\n Fits the learning curve with the wrapped model to the specified data.\n Draws training and test score curves and saves the scores to the\n estimator.\n\n Parameters\n ----------\n X : array-like, shape (n_samples, n_features)\n Training vector, where n_samples is the number of samples and\n n_features is the number of features.\n\n y : array-like, shape (n_samples) or (n_samples, n_features), optional\n Target relative to X for classification or regression;\n None for unsupervised learning.\n\n Returns\n -------\n self : instance\n\n \"\"\"\n\n self.cv_scores_ = cross_val_score(\n self.estimator, X, y, cv=self.cv, scoring=self.scoring\n )\n self.cv_scores_mean_ = self.cv_scores_.mean()\n\n self.draw()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndraw the cross - validated scores for the current locale.", "response": "def draw(self, **kwargs):\n \"\"\"\n Creates the bar chart of the cross-validated scores generated from the\n fit method and places a dashed horizontal line that represents the\n average value of the scores.\n \"\"\"\n\n color = kwargs.pop(\"color\", \"b\")\n width = kwargs.pop(\"width\", 0.3)\n linewidth = kwargs.pop(\"linewidth\", 1)\n\n xvals = np.arange(1, len(self.cv_scores_) + 1, 1)\n self.ax.bar(xvals, self.cv_scores_, width=width)\n self.ax.axhline(\n self.cv_scores_mean_, color=color,\n label=\"Mean score = {:0.3f}\".format(self.cv_scores_mean_),\n linestyle='--', linewidth=linewidth\n )\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd the title, legend, and other visual final touches to the plot.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Add the title, legend, and other visual final touches to the plot.\n \"\"\"\n\n # Set the title of the figure\n self.set_title('Cross Validation Scores for {}'.format(self.name))\n\n # Add the legend\n loc = kwargs.pop(\"loc\", \"best\")\n edgecolor = kwargs.pop(\"edgecolor\", \"k\")\n self.ax.legend(frameon=True, loc=loc, edgecolor=edgecolor)\n\n # set spacing between the x ticks\n self.ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n\n # Set the axis labels\n self.ax.set_xlabel('Training Instances')\n self.ax.set_ylabel('Score')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef kendalltau(X):\n corrs = np.zeros((X.shape[1], X.shape[1]))\n for idx, cola in enumerate(X.T):\n for jdx, colb in enumerate(X.T):\n corrs[idx, jdx] = sp_kendalltau(cola,colb)[0]\n return corrs", "response": "Returns a correlation matrix so that each column is the variable and each row is the observations."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a 1D ranking of each feature with the algorithm and ranks them in a bar plot.", "response": "def rank1d(X, y=None, ax=None, algorithm='shapiro', features=None,\n orient='h', show_feature_names=True, **kwargs):\n \"\"\"Scores each feature with the algorithm and ranks them in a bar plot.\n\n This helper function is a quick wrapper to utilize the Rank1D Visualizer\n (Transformer) for one-off analysis.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n ax : matplotlib axes\n the axis to plot the figure on.\n\n algorithm : one of {'shapiro', }, default: 'shapiro'\n The ranking algorithm to use, default is 'Shapiro-Wilk.\n\n features : list\n A list of feature names to use.\n If a DataFrame is passed to fit and features is None, feature\n names are selected as the columns of the DataFrame.\n\n orient : 'h' or 'v'\n Specifies a horizontal or vertical bar chart.\n\n show_feature_names : boolean, default: True\n If True, the feature names are used to label the axis ticks in the\n plot.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n\n \"\"\"\n # Instantiate the visualizer\n visualizer = Rank1D(ax, algorithm, features, orient, show_feature_names,\n **kwargs)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rank2d(X, y=None, ax=None, algorithm='pearson', features=None,\n show_feature_names=True, colormap='RdBu_r', **kwargs):\n \"\"\"Displays pairwise comparisons of features with the algorithm and ranks\n them in a lower-left triangle heatmap plot.\n\n This helper function is a quick wrapper to utilize the Rank2D Visualizer\n (Transformer) for one-off analysis.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n ax : matplotlib axes\n the axis to plot the figure on.\n\n algorithm : one of {pearson, covariance, spearman, kendalltau}\n the ranking algorithm to use, default is Pearson correlation.\n\n features : list\n A list of feature names to use.\n If a DataFrame is passed to fit and features is None, feature\n names are selected as the columns of the DataFrame.\n\n show_feature_names : boolean, default: True\n If True, the feature names are used to label the axis ticks in the\n plot.\n\n colormap : string or cmap\n optional string or matplotlib cmap to colorize lines\n Use either color to colorize the lines on a per class basis or\n colormap to color them on a continuous scale.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n\n \"\"\"\n # Instantiate the visualizer\n visualizer = Rank2D(ax, algorithm, features, colormap, show_feature_names,\n **kwargs)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Displays pairwise comparisons of features with the algorithm and ranks them in a lower - left triangle heatmap plot."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rank(self, X, algorithm=None):\n algorithm = algorithm or self.ranking_\n algorithm = algorithm.lower()\n\n if algorithm not in self.ranking_methods:\n raise YellowbrickValueError(\n \"'{}' is unrecognized ranking method\".format(algorithm)\n )\n\n # Extract matrix from dataframe if necessary\n if is_dataframe(X):\n X = X.values\n\n return self.ranking_methods[algorithm](X)", "response": "Returns the feature ranking for the given set of n - dimensional n - dimensional n - dimensional molecular features."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef finalize(self, **kwargs):\n # Set the title\n self.set_title(\n \"{} Ranking of {} Features\".format(\n self.ranking_.title(), len(self.features_)\n )\n )", "response": "Executes any subclass - specific axes finalization steps."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef draw(self, **kwargs):\n if self.orientation_ == 'h':\n # Make the plot\n self.ax.barh(np.arange(len(self.ranks_)), self.ranks_, color='b')\n\n # Add ticks and tick labels\n self.ax.set_yticks(np.arange(len(self.ranks_)))\n if self.show_feature_names_:\n self.ax.set_yticklabels(self.features_)\n else:\n self.ax.set_yticklabels([])\n\n # Order the features from top to bottom on the y axis\n self.ax.invert_yaxis()\n\n # Turn off y grid lines\n self.ax.yaxis.grid(False)\n\n elif self.orientation_ == 'v':\n # Make the plot\n self.ax.bar(np.arange(len(self.ranks_)), self.ranks_, color='b')\n\n # Add ticks and tick labels\n self.ax.set_xticks(np.arange(len(self.ranks_)))\n if self.show_feature_names_:\n self.ax.set_xticklabels(self.features_, rotation=90)\n else:\n self.ax.set_xticklabels([])\n\n # Turn off x grid lines\n self.ax.xaxis.grid(False)\n\n else:\n raise YellowbrickValueError(\n \"Orientation must be 'h' or 'v'\"\n )", "response": "Draws the bar plot of the ranking array of features."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing the heatmap of the ranking matrix of variables.", "response": "def draw(self, **kwargs):\n \"\"\"\n Draws the heatmap of the ranking matrix of variables.\n \"\"\"\n # Set the axes aspect to be equal\n self.ax.set_aspect(\"equal\")\n\n # Generate a mask for the upper triangle\n mask = np.zeros_like(self.ranks_, dtype=np.bool)\n mask[np.triu_indices_from(mask)] = True\n\n # Draw the heatmap\n # TODO: Move mesh to a property so the colorbar can be finalized\n data = np.ma.masked_where(mask, self.ranks_)\n mesh = self.ax.pcolormesh(data, cmap=self.colormap, vmin=-1, vmax=1)\n\n # Set the Axis limits\n self.ax.set(\n xlim=(0, data.shape[1]), ylim=(0, data.shape[0])\n )\n\n # Add the colorbar\n cb = self.ax.figure.colorbar(mesh, None, self.ax)\n cb.outline.set_linewidth(0)\n\n # Reverse the rows to get the lower left triangle\n self.ax.invert_yaxis()\n\n # Add ticks and tick labels\n self.ax.set_xticks(np.arange(len(self.ranks_)) + 0.5)\n self.ax.set_yticks(np.arange(len(self.ranks_)) + 0.5)\n if self.show_feature_names_:\n self.ax.set_xticklabels(self.features_, rotation=90)\n self.ax.set_yticklabels(self.features_)\n else:\n self.ax.set_xticklabels([])\n self.ax.set_yticklabels([])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef class_prediction_error(\n model,\n X,\n y=None,\n ax=None,\n classes=None,\n test_size=0.2,\n random_state=None,\n **kwargs):\n \"\"\"Quick method:\n Divides the dataset X and y into train and test splits, fits the model on\n the train split, then scores the model on the test split. The visualizer\n displays the support for each class in the fitted classification model\n displayed as a stacked bar plot Each bar is segmented to show the\n distribution of predicted classes for each class.\n\n This helper function is a quick wrapper to utilize the ClassPredictionError\n ScoreVisualizer for one-off analysis.\n\n Parameters\n ----------\n model : the Scikit-Learn estimator (should be a classifier)\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features.\n\n y : ndarray or Series of length n\n An array or series of target or class values.\n\n ax : matplotlib axes\n The axes to plot the figure on.\n\n classes : list of strings\n The names of the classes in the target\n\n test_size : float, default=0.2\n The percentage of the data to reserve as test data.\n\n random_state : int or None, default=None\n The value to seed the random number generator for shuffling data.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the class prediction error plot was drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = ClassPredictionError(model, ax, classes, **kwargs)\n\n # Create the train and test splits\n X_train, X_test, y_train, y_test = tts(\n X, y, test_size=test_size, random_state=random_state\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X_train, y_train, **kwargs)\n visualizer.score(X_test, y_test)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Plots a ClassPredictionError plot on the specified axes."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef draw(self):\n\n indices = np.arange(len(self.classes_))\n prev = np.zeros(len(self.classes_))\n\n colors = resolve_colors(\n colors=self.colors,\n n_colors=len(self.classes_))\n\n for idx, row in enumerate(self.predictions_):\n self.ax.bar(indices, row, label=self.classes_[idx],\n bottom=prev, color=colors[idx])\n prev += row\n\n return self.ax", "response": "Draws the class prediction error across the axis."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinalizing executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n \"\"\"\n\n indices = np.arange(len(self.classes_))\n\n # Set the title\n self.set_title(\"Class Prediction Error for {}\".format(self.name))\n\n # Set the x ticks with the class names\n self.ax.set_xticks(indices)\n self.ax.set_xticklabels(self.classes_)\n\n # Set the axes labels\n self.ax.set_xlabel(\"actual class\")\n self.ax.set_ylabel(\"number of predicted class\")\n\n # Compute the ceiling for the y limit\n cmax = max([sum(predictions) for predictions in self.predictions_])\n self.ax.set_ylim(0, cmax + cmax * 0.1)\n\n # Put the legend outside of the graph\n plt.legend(bbox_to_anchor=(1.04, 0.5), loc=\"center left\")\n plt.tight_layout(rect=[0, 0, 0.85, 1])"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a MissingValues Bar visualizer that draws the missing values for each selected feature column.", "response": "def missing_bar(X, y=None, ax=None, classes=None, width=0.5, color='black', **kwargs):\n \"\"\"The MissingValues Bar visualizer creates a bar graph that lists the total\n count of missing values for each selected feature column.\n\n When y targets are supplied to fit, the output is a stacked bar chart where\n each color corresponds to the total NaNs for the feature in that column.\n\n Parameters\n ----------\n alpha : float, default: 0.5\n A value for bending elments with the background.\n\n marker : matplotlib marker, default: |\n The marker used for each element coordinate in the plot\n\n color : string, default: black\n The color for drawing the bar chart when the y targets are not passed to\n fit.\n\n colors : list, default: None\n The color pallette for drawing a stack bar chart when the y targets\n are passed to fit.\n\n classes : list, default: None\n A list of class names for the legend.\n If classes is None and a y value is passed to fit then the classes\n are selected from the target vector.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Attributes\n ----------\n features_ : np.array\n The feature labels ranked according to their importance\n\n classes_ : np.array\n The class labels for each of the target values\n\n Examples\n --------\n\n >>> from yellowbrick.contrib.missing import missing_bar\n >>> visualizer = missing_bar(X, y=y)\n \"\"\"\n # Instantiate the visualizer\n visualizer = MissingValuesBar(\n ax=ax, classes=classes, width=width, color=color, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y)\n visualizer.poof()\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef draw(self, X, y, **kwargs):\n nan_col_counts = self.get_nan_col_counts()\n\n # the x locations for the groups\n self.ind = np.arange(len(self.features_))\n\n if y is None:\n self.ax.barh(self.ind - self.width / 2, nan_col_counts, self.width,\n color=self.color, label=None)\n else:\n self.draw_stacked_bar(nan_col_counts)", "response": "Called from the fit method this method generates a horizontal bar chart for each nan count per\n target values."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndraws a horizontal stacked bar chart with different colors for each count of nan values per label.", "response": "def draw_stacked_bar(self, nan_col_counts):\n \"\"\"Draws a horizontal stacked bar chart with different colors\n for each count of nan values per label.\n \"\"\"\n for index, nan_values in enumerate(nan_col_counts):\n label, nan_col_counts = nan_values\n\n if index == 0:\n # first draw should be at zero\n bottom_chart = np.zeros(nan_col_counts.shape)\n\n # if features passed in then, label as such\n if self.classes_ is not None:\n label = self.classes_[index]\n\n color = self.colors[index]\n\n self.ax.barh(self.ind - self.width / 2, nan_col_counts, self.width,\n color=color, label=label, left=bottom_chart)\n\n # keep track of counts to build on stacked\n bottom_chart = nan_col_counts"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfinalize executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n\n \"\"\"\n # Set the title\n self.set_title(\n 'Count of Missing Values by Column'\n )\n tick_locations = np.arange(len(self.features_)) # the x locations for the groups\n self.ax.set_yticks(tick_locations)\n self.ax.set_yticklabels(self.get_feature_names())\n # Remove the ticks from the graph\n self.ax.set_xlabel('Count')\n\n self.ax.legend(loc='best')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef feature_correlation(X, y, ax=None, method='pearson',\n labels=None, sort=False, feature_index=None,\n feature_names=None, **kwargs):\n \"\"\"\n Displays the correlation between features and dependent variables.\n\n This visualizer can be used side-by-side with\n yellowbrick.features.JointPlotVisualizer that plots a feature\n against the target and shows the distribution of each via a\n histogram on each axis.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n ax : matplotlib Axes, default: None\n The axis to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n method : str, default: 'pearson'\n The method to calculate correlation between features and target.\n Options include:\n\n - 'pearson', which uses ``scipy.stats.pearsonr``\n - 'mutual_info-regression', which uses ``mutual_info-regression``\n from ``sklearn.feature_selection``\n - 'mutual_info-classification', which uses ``mutual_info_classif``\n from ``sklearn.feature_selection``\n 'mutual_info-classification'], default: 'pearson'\n\n labels : list, default: None\n A list of feature names to use. If a DataFrame is passed to fit and\n features is None, feature names are selected as the column names.\n\n sort : boolean, default: False\n If false, the features are are not sorted in the plot; otherwise\n features are sorted in ascending order of correlation.\n\n feature_index : list,\n A list of feature index to include in the plot.\n\n feature_names : list of feature names\n A list of feature names to include in the plot.\n Must have labels or the fitted data is a DataFrame with column names.\n If feature_index is provided, feature_names will be ignored.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n\n # Instantiate the visualizer\n viz = FeatureCorrelation(ax, method, labels, sort,\n feature_index, feature_names, **kwargs)\n\n # Fit and transform the visualizer (calls draw)\n viz.fit(X, y, **kwargs)\n viz.finalize()\n\n # Return the axes object on the visualizer\n return viz.ax", "response": "Visualize the feature correlation between two sets of target and class values."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit(self, X, y, **kwargs):\n self._create_labels_for_features(X)\n\n self._select_features_to_plot(X)\n\n # Calculate Features correlation with target variable\n if self.method == \"pearson\":\n self.scores_ = np.array(\n [pearsonr(x, y, **kwargs)[0] for x in np.asarray(X).T]\n )\n else:\n self.scores_ = np.array(\n self.correlation_methods[self.method](X, y, **kwargs)\n )\n\n # If feature indices are given, plot only the given features\n if self.feature_index:\n self.scores_ = self.scores_[self.feature_index]\n self.features_ = self.features_[self.feature_index]\n\n # Sort features by correlation\n if self.sort:\n sort_idx = np.argsort(self.scores_)\n self.scores_ = self.scores_[sort_idx]\n self.features_ = self.features_[sort_idx]\n\n self.draw()\n return self", "response": "Fits the estimator to calculate feature correlation to the dependent variable."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndraw the feature correlation to dependent variable called from fit.", "response": "def draw(self):\n \"\"\"\n Draws the feature correlation to dependent variable, called from fit.\n \"\"\"\n pos = np.arange(self.scores_.shape[0]) + 0.5\n\n self.ax.barh(pos, self.scores_)\n\n # Set the labels for the bars\n self.ax.set_yticks(pos)\n self.ax.set_yticklabels(self.features_)\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef finalize(self):\n self.set_title('Features correlation with dependent variable')\n\n self.ax.set_xlabel(self.correlation_labels[self.method])\n\n self.ax.grid(False, axis='y')", "response": "Finalize the drawing setting labels and title."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _create_labels_for_features(self, X):\n if self.labels is None:\n # Use column names if a dataframe\n if is_dataframe(X):\n self.features_ = np.array(X.columns)\n # Otherwise use the column index as the labels\n else:\n _, ncols = X.shape\n self.features_ = np.arange(0, ncols)\n else:\n self.features_ = np.array(self.labels)", "response": "Create labels for the features in the features_ attribute of the features_ attribute."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nselect features to plot.", "response": "def _select_features_to_plot(self, X):\n \"\"\"\n Select features to plot.\n\n feature_index is always used as the filter and\n if filter_names is supplied, a new feature_index\n is computed from those names.\n \"\"\"\n if self.feature_index:\n if self.feature_names:\n raise YellowbrickWarning(\n 'Both feature_index and feature_names '\n 'are specified. feature_names is ignored'\n )\n if (min(self.feature_index) < 0\n or max(self.feature_index) >= X.shape[1]):\n raise YellowbrickValueError('Feature index is out of range')\n elif self.feature_names:\n self.feature_index = []\n features_list = self.features_.tolist()\n for feature_name in self.feature_names:\n try:\n self.feature_index.append(\n features_list.index(feature_name)\n )\n except ValueError:\n raise YellowbrickValueError(\n '{} not in labels'.format(feature_name)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef feature_analysis(fname=\"feature_analysis.png\"):\n\n # Create side-by-side axes grid\n _, axes = plt.subplots(ncols=2, figsize=(18,6))\n\n # Draw RadViz on the left\n data = load_occupancy(split=False)\n oz = RadViz(ax=axes[0], classes=[\"unoccupied\", \"occupied\"])\n oz.fit(data.X, data.y)\n oz.finalize()\n\n # Draw Rank2D on the right\n data = load_concrete(split=False)\n oz = Rank2D(ax=axes[1])\n oz.fit_transform(data.X, data.y)\n oz.finalize()\n\n # Save figure\n path = os.path.join(FIGURES, fname)\n plt.tight_layout()\n plt.savefig(path)", "response": "Create figures for feature analysis\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef regression(fname=\"regression.png\"):\n _, axes = plt.subplots(ncols=2, figsize=(18, 6))\n alphas = np.logspace(-10, 1, 300)\n data = load_concrete(split=True)\n\n # Plot prediction error in the middle\n oz = PredictionError(LassoCV(alphas=alphas), ax=axes[0])\n oz.fit(data.X.train, data.y.train)\n oz.score(data.X.test, data.y.test)\n oz.finalize()\n\n # Plot residuals on the right\n oz = ResidualsPlot(RidgeCV(alphas=alphas), ax=axes[1])\n oz.fit(data.X.train, data.y.train)\n oz.score(data.X.test, data.y.test)\n oz.finalize()\n\n # Save figure\n path = os.path.join(FIGURES, fname)\n plt.tight_layout()\n plt.savefig(path)", "response": "Create figures for regression models\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ax(self):\n if not hasattr(self, \"_ax\") or self._ax is None:\n self._ax = plt.gca()\n return self._ax", "response": "The matplotlib axes that the visualizer draws upon."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the actual size in pixels as set by matplotlib or the user provided size if available.", "response": "def size(self):\n \"\"\"\n Returns the actual size in pixels as set by matplotlib, or\n the user provided size if available.\n \"\"\"\n if not hasattr(self, \"_size\") or self._size is None:\n fig = plt.gcf()\n self._size = fig.get_size_inches()*fig.dpi\n return self._size"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmaking the magic happen and a visualizer appear.", "response": "def poof(self, outpath=None, clear_figure=False, **kwargs):\n \"\"\"\n Poof makes the magic happen and a visualizer appear! You can pass in\n a path to save the figure to disk with various backends, or you can\n call it with no arguments to show the figure either in a notebook or\n in a GUI window that pops up on screen.\n\n Parameters\n ----------\n outpath: string, default: None\n path or None. Save figure to disk or if None show in window\n\n clear_figure: boolean, default: False\n When True, this flag clears the figure after saving to file or\n showing on screen. This is useful when making consecutive plots.\n\n kwargs: dict\n generic keyword arguments.\n\n Notes\n -----\n Developers of visualizers don't usually override poof, as it is\n primarily called by the user to render the visualization.\n \"\"\"\n # Ensure that draw has been called\n if self._ax is None:\n warn_message = (\n \"{} does not have a reference to a matplotlib.Axes \"\n \"the figure may not render as expected!\"\n )\n warnings.warn(\n warn_message.format(self.__class__.__name__), YellowbrickWarning\n )\n\n # Finalize the figure\n self.finalize()\n\n if outpath is not None:\n plt.savefig(outpath, **kwargs)\n else:\n plt.show()\n\n if clear_figure:\n plt.gcf().clear()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the title on the current axes.", "response": "def set_title(self, title=None):\n \"\"\"\n Sets the title on the current axes.\n\n Parameters\n ----------\n title: string, default: None\n Add title to figure or if None leave untitled.\n \"\"\"\n title = self.title or title\n if title is not None:\n self.ax.set_title(title)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating the subplots for the number of given models.", "response": "def generate_subplots(self):\n \"\"\"\n Generates the subplots for the number of given models.\n \"\"\"\n _, axes = plt.subplots(len(self.models), sharex=True, sharey=True)\n return axes"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef predict(self, X, y):\n for model in self.models:\n yield cvp(model, X, y, cv=12)", "response": "Returns a generator containing the predictions for each of the internal models in the set of target or class values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_corpus(path):\n\n # Check if the data exists, otherwise download or raise\n if not os.path.exists(path):\n raise ValueError((\n \"'{}' dataset has not been downloaded, \"\n \"use the yellowbrick.download module to fetch datasets\"\n ).format(path))\n\n # Read the directories in the directory as the categories.\n categories = [\n cat for cat in os.listdir(path)\n if os.path.isdir(os.path.join(path, cat))\n ]\n\n files = [] # holds the file names relative to the root\n data = [] # holds the text read from the file\n target = [] # holds the string of the category\n\n # Load the data from the files in the corpus\n for cat in categories:\n for name in os.listdir(os.path.join(path, cat)):\n files.append(os.path.join(path, cat, name))\n target.append(cat)\n\n with open(os.path.join(path, cat, name), 'r') as f:\n data.append(f.read())\n\n\n # Return the data bunch for use similar to the newsgroups example\n return Bunch(\n categories=categories,\n files=files,\n data=data,\n target=target,\n )", "response": "Loads and wrangles the passed in text corpus by path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_data_home(path=None):\n if path is None:\n path = os.environ.get('YELLOWBRICK_DATA', FIXTURES)\n\n path = os.path.expanduser(path)\n path = os.path.expandvars(path)\n\n if not os.path.exists(path):\n os.makedirs(path)\n\n return path", "response": "Returns the path of the Yellowbrick data directory."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlooks up the path to the dataset specified in the data home directory, which is found using the ``get_data_home`` function. By default data home is colocated with the code, but can be modified with the YELLOWBRICK_DATA environment variable, or passing in a different directory. The file returned will be by default, the name of the dataset in compressed CSV format. Other files and extensions can be passed in to locate other data types or auxilliary files. If the dataset is not found a ``DatasetsError`` is raised by default. Parameters ---------- dataset : str The name of the dataset; should either be a folder in data home or specified in the yellowbrick.datasets.DATASETS variable. data_home : str, optional The path on disk where data is stored. If not passed in, it is looked up from YELLOWBRICK_DATA or the default returned by ``get_data_home``. fname : str, optional The filename to look up in the dataset path, by default it will be the name of the dataset. The fname must include an extension. ext : str, default: \".csv.gz\" The extension of the data to look up in the dataset path, if the fname is specified then the ext parameter is ignored. If ext is None then the directory of the dataset will be returned. raises : bool, default: True If the path does not exist, raises a DatasetsError unless this flag is set to False, at which point None is returned (e.g. for checking if the path exists or not). Returns ------- path : str or None A path to the requested file, guaranteed to exist if an exception is not raised during processing of the request (unless None is returned). raises : DatasetsError If raise is True and the path does not exist, raises a DatasetsError.", "response": "def find_dataset_path(dataset, data_home=None, fname=None, ext=\".csv.gz\", raises=True):\n \"\"\"\n Looks up the path to the dataset specified in the data home directory,\n which is found using the ``get_data_home`` function. By default data home\n is colocated with the code, but can be modified with the YELLOWBRICK_DATA\n environment variable, or passing in a different directory.\n\n The file returned will be by default, the name of the dataset in compressed\n CSV format. Other files and extensions can be passed in to locate other data\n types or auxilliary files.\n\n If the dataset is not found a ``DatasetsError`` is raised by default.\n\n Parameters\n ----------\n dataset : str\n The name of the dataset; should either be a folder in data home or\n specified in the yellowbrick.datasets.DATASETS variable.\n\n data_home : str, optional\n The path on disk where data is stored. If not passed in, it is looked\n up from YELLOWBRICK_DATA or the default returned by ``get_data_home``.\n\n fname : str, optional\n The filename to look up in the dataset path, by default it will be the\n name of the dataset. The fname must include an extension.\n\n ext : str, default: \".csv.gz\"\n The extension of the data to look up in the dataset path, if the fname\n is specified then the ext parameter is ignored. If ext is None then\n the directory of the dataset will be returned.\n\n raises : bool, default: True\n If the path does not exist, raises a DatasetsError unless this flag is set\n to False, at which point None is returned (e.g. for checking if the\n path exists or not).\n\n Returns\n -------\n path : str or None\n A path to the requested file, guaranteed to exist if an exception is\n not raised during processing of the request (unless None is returned).\n\n raises : DatasetsError\n If raise is True and the path does not exist, raises a DatasetsError.\n \"\"\"\n # Figure out the root directory of the datasets\n data_home = get_data_home(data_home)\n\n # Figure out the relative path to the dataset\n if fname is None:\n if ext is None:\n path = os.path.join(data_home, dataset)\n else:\n path = os.path.join(data_home, dataset, \"{}{}\".format(dataset, ext))\n else:\n path = os.path.join(data_home, dataset, fname)\n\n # Determine if the path exists\n if not os.path.exists(path):\n\n # Suppress exceptions if required\n if not raises:\n return None\n\n raise DatasetsError((\n \"could not find dataset at {} - does it need to be downloaded?\"\n ).format(path))\n\n return path"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dataset_exists(dataset, data_home=None):\n data_home = get_data_home(data_home)\n path = os.path.join(data_home, dataset)\n\n return os.path.exists(path) and os.path.isdir(path)", "response": "Checks to see if a dataset exists in the data home directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dataset_archive(dataset, signature, data_home=None, ext=\".zip\"):\n data_home = get_data_home(data_home)\n path = os.path.join(data_home, dataset+ext)\n\n if os.path.exists(path) and os.path.isfile(path):\n return sha256sum(path) == signature\n\n return False", "response": "Checks to see if the dataset archive file exists in the data home directory and is the latest version of the dataset."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cleanup_dataset(dataset, data_home=None, ext=\".zip\"):\n removed = 0\n data_home = get_data_home(data_home)\n\n # Paths to remove\n datadir = os.path.join(data_home, dataset)\n archive = os.path.join(data_home, dataset+ext)\n\n # Remove directory and contents\n if os.path.exists(datadir):\n shutil.rmtree(datadir)\n removed += 1\n\n # Remove the archive file\n if os.path.exists(archive):\n os.remove(archive)\n removed += 1\n\n return removed", "response": "Removes the dataset directory and archive file from the data home directory."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit_transform_poof(self, X, y=None, **kwargs):\n Xp = self.fit_transform(X, y, **kwargs)\n self.poof(**kwargs)\n return Xp", "response": "Fit to data transform it then visualize it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfitting the object to the specified features.", "response": "def fit(self, X, y=None, **fit_params):\n \"\"\"\n This method performs preliminary computations in order to set up the\n figure or perform other analyses. It can also call drawing methods in\n order to set up various non-instance related figure elements.\n\n This method must return self.\n \"\"\"\n\n # Handle the feature names if they're None.\n if self.features_ is None:\n\n # If X is a data frame, get the columns off it.\n if is_dataframe(X):\n self.features_ = np.array(X.columns)\n\n # Otherwise create numeric labels for each column.\n else:\n _, ncols = X.shape\n self.features_ = np.arange(0, ncols)\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfitting the data to the target class and store the class labels for the target class.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n The fit method is the primary drawing input for the\n visualization since it has both the X and y data required for the\n viz and the transform method does not.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n kwargs : dict\n Pass generic arguments to the drawing method\n\n Returns\n -------\n self : instance\n Returns the instance of the transformer/visualizer\n \"\"\"\n super(DataVisualizer, self).fit(X, y, **kwargs)\n\n # Store the classes for the legend if they're None.\n if self.classes_ is None:\n # TODO: Is this the most efficient method?\n self.classes_ = [str(label) for label in np.unique(y)]\n\n # Draw the instances\n self.draw(X, y, **kwargs)\n\n # Fit always returns self.\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a color palette object with color definition and handling.", "response": "def color_palette(palette=None, n_colors=None):\n \"\"\"\n Return a color palette object with color definition and handling.\n\n Calling this function with ``palette=None`` will return the current\n matplotlib color cycle.\n\n This function can also be used in a ``with`` statement to temporarily\n set the color cycle for a plot or set of plots.\n\n Parameters\n ----------\n\n palette : None or str or sequence\n Name of a palette or ``None`` to return the current palette. If a\n sequence the input colors are used but possibly cycled.\n\n Available palette names from :py:mod:`yellowbrick.colors.palettes` are:\n\n .. hlist::\n :columns: 3\n\n * :py:const:`accent`\n * :py:const:`dark`\n * :py:const:`paired`\n * :py:const:`pastel`\n * :py:const:`bold`\n * :py:const:`muted`\n * :py:const:`colorblind`\n * :py:const:`sns_colorblind`\n * :py:const:`sns_deep`\n * :py:const:`sns_muted`\n * :py:const:`sns_pastel`\n * :py:const:`sns_bright`\n * :py:const:`sns_dark`\n * :py:const:`flatui`\n * :py:const:`neural_paint`\n\n n_colors : None or int\n Number of colors in the palette. If ``None``, the default will depend\n on how ``palette`` is specified. Named palettes default to 6 colors\n which allow the use of the names \"bgrmyck\", though others do have more\n or less colors; therefore reducing the size of the list can only be\n done by specifying this parameter. Asking for more colors than exist\n in the palette will cause it to cycle.\n\n Returns\n -------\n list(tuple)\n Returns a ColorPalette object, which behaves like a list, but can be\n used as a context manager and possesses functions to convert colors.\n\n .. seealso::\n\n :func:`.set_palette`\n Set the default color cycle for all plots.\n :func:`.set_color_codes`\n Reassign color codes like ``\"b\"``, ``\"g\"``, etc. to\n colors from one of the yellowbrick palettes.\n :func:`..colors.resolve_colors`\n Resolve a color map or listed sequence of colors.\n\n \"\"\"\n if palette is None:\n palette = get_color_cycle()\n if n_colors is None:\n n_colors = len(palette)\n\n elif not isinstance(palette, str):\n if n_colors is None:\n n_colors = len(palette)\n\n else:\n if palette.lower() not in PALETTES:\n raise YellowbrickValueError(\n \"'{}' is not a recognized palette!\".format(palette)\n )\n\n palette = PALETTES[palette.lower()]\n if n_colors is None:\n n_colors = len(palette)\n\n # Always return as many colors as we asked for\n pal_cycle = cycle(palette)\n palette = [next(pal_cycle) for _ in range(n_colors)]\n\n # Always return in RGB tuple format\n try:\n palette = map(mpl.colors.colorConverter.to_rgb, palette)\n palette = ColorPalette(palette)\n except ValueError:\n raise YellowbrickValueError(\n \"Could not generate a palette for %s\" % str(palette)\n )\n\n return palette"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the color codes of the base object in the current color cycle.", "response": "def set_color_codes(palette=\"accent\"):\n \"\"\"\n Change how matplotlib color shorthands are interpreted.\n\n Calling this will change how shorthand codes like \"b\" or \"g\"\n are interpreted by matplotlib in subsequent plots.\n\n Parameters\n ----------\n palette : str\n Named yellowbrick palette to use as the source of colors.\n\n See Also\n --------\n set_palette : Color codes can also be set through the function that\n sets the matplotlib color cycle.\n \"\"\"\n\n if palette not in PALETTES:\n raise YellowbrickValueError(\n \"'{}' is not a recognized palette!\".format(palette)\n )\n\n # Fetch the colors and adapt the length\n colors = PALETTES[palette]\n\n if len(colors) > 7:\n # Truncate colors that are longer than 7\n colors = colors[:7]\n\n elif len(colors) < 7:\n # Add the key (black) color to colors that are shorter than 7\n colors = colors + [YB_KEY]\n\n # Set the color codes on matplotlib\n for code, color in zip(\"bgrmyck\", colors):\n rgb = mpl.colors.colorConverter.to_rgb(color)\n mpl.colors.colorConverter.colors[code] = rgb\n mpl.colors.colorConverter.cache[code] = rgb"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a ListedColormap object from a named color sequence palette.", "response": "def color_sequence(palette=None, n_colors=None):\n \"\"\"\n Return a `ListedColormap` object from a named sequence palette. Useful\n for continuous color scheme values and color maps.\n\n Calling this function with ``palette=None`` will return the default\n color sequence: Color Brewer RdBu.\n\n Parameters\n ----------\n\n palette : None or str or sequence\n Name of a palette or ``None`` to return the default palette. If a\n sequence the input colors are used to create a ListedColormap.\n\n The currently implemented color sequences are from Color Brewer.\n\n Available palette names from :py:mod:`yellowbrick.colors.palettes` are:\n\n .. hlist::\n :columns: 3\n\n * :py:const: Blues\n * :py:const: BrBG\n * :py:const: BuGn\n * :py:const: BuPu\n * :py:const: GnBu\n * :py:const: Greens\n * :py:const: Greys\n * :py:const: OrRd\n * :py:const: Oranges\n * :py:const: PRGn\n * :py:const: PiYG\n * :py:const: PuBu\n * :py:const: PuBuGn\n * :py:const: PuOr\n * :py:const: PuRd\n * :py:const: Purples\n * :py:const: RdBu\n * :py:const: RdGy\n * :py:const: RdPu\n * :py:const: RdYlBu\n * :py:const: RdYlGn\n * :py:const: Reds\n * :py:const: Spectral\n * :py:const: YlGn\n * :py:const: YlGnBu\n * :py:const: YlOrBr\n * :py:const: YlOrRd\n * :py:const: ddl_heat\n\n n_colors : None or int\n Number of colors in the palette. If ``None``, the default will depend\n on how ``palette`` is specified - selecting the largest sequence for\n that palette name. Note that sequences have a minimum lenght of 3 - if\n a number of colors is specified that is not available for the sequence\n a ``ValueError`` is raised.\n\n Returns\n -------\n colormap\n Returns a ListedColormap object, an artist object from the matplotlib\n library that can be used wherever a colormap is necessary.\n \"\"\"\n # Select the default colormap if None is passed in.\n palette = palette or DEFAULT_SEQUENCE\n\n # Create a listed color map from the sequence\n if not isinstance(palette, str):\n return mplcol.ListedColormap(palette)\n\n # Otherwise perform a case-insensitive lookup\n sequences = {key.lower(): key for key in SEQUENCES.keys()}\n if palette.lower() not in sequences:\n raise YellowbrickValueError(\n \"'{}' is not a recognized palette!\".format(palette)\n )\n\n # Collect the palette into the dictionary of lists.\n n_palettes = SEQUENCES[sequences[palette.lower()]]\n\n # If no definitive color is passed in, maximize it.\n if n_colors is None:\n n_colors = max(n_palettes.keys())\n\n else:\n if n_colors not in n_palettes.keys():\n raise YellowbrickValueError(\n \"No {} palette of length {}\".format(palette, n_colors)\n )\n\n # Return the color map from the sequence\n return mplcol.ListedColormap(n_palettes[n_colors], name=palette, N=n_colors)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef as_hex(self):\n hex = [mpl.colors.rgb2hex(rgb) for rgb in self]\n return ColorPalette(hex)", "response": "Return a color palette with hex codes instead of RGB values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef as_rgb(self):\n rgb = [mpl.colors.colorConverter.to_rgb(hex) for hex in self]\n return ColorPalette(rgb)", "response": "Return a color palette with RGB values instead of hex codes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef plot(self, size=1):\n n = len(self)\n fig, ax = plt.subplots(1, 1, figsize=(n * size, size))\n ax.imshow(np.arange(n).reshape(1,n),\n cmap=mpl.colors.ListedColormap(list(self)),\n interpolation=\"nearest\", aspect=\"auto\")\n ax.set_xticks(np.arange(n) - .5)\n ax.set_yticks([-.5, .5])\n ax.set_xticklabels([])\n ax.set_yticklabels([])", "response": "Plot the values in the color palette as a horizontal array."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _make_path(self, path, name):\n if not os.path.exists(path):\n os.mkdirs(path)\n\n if os.path.isdir(path) :\n return os.path.join(path, name)\n\n return path", "response": "Make a path to the log file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plot_manifold_embedding(self, algorithm=\"lle\", path=\"images\"):\n _, ax = plt.subplots(figsize=(9,6))\n path = self._make_path(path, \"s_curve_{}_manifold.png\".format(algorithm))\n\n oz = Manifold(\n ax=ax, manifold=algorithm,\n target='continuous', colors='nipy_spectral'\n )\n\n oz.fit(self.X, self.y)\n oz.poof(outpath=path)", "response": "Draw the manifold embedding for the specified algorithm"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndownloading the zipped data set at the given URL and saves it to the data directory specified by get_data_home.", "response": "def download_data(url, signature, data_home=None, replace=False, extract=True):\n \"\"\"\n Downloads the zipped data set specified at the given URL, saving it to\n the data directory specified by ``get_data_home``. This function verifies\n the download with the given signature and extracts the archive.\n\n Parameters\n ----------\n url : str\n The URL of the dataset on the Internet to GET\n\n signature : str\n The SHA 256 hash of the dataset archive being downloaded to verify\n that the dataset has been correctly downloaded\n\n data_home : str, optional\n The path on disk where data is stored. If not passed in, it is looked\n up from YELLOWBRICK_DATA or the default returned by ``get_data_home``.\n\n replace : bool, default: False\n If the data archive already exists, replace the dataset. If this is\n False and the dataset exists, an exception is raised.\n\n extract : bool, default: True\n Extract the archive file after downloading it\n \"\"\"\n data_home = get_data_home(data_home)\n\n # Get the name of the file from the URL\n basename = os.path.basename(url)\n name, _ = os.path.splitext(basename)\n\n # Get the archive and data directory paths\n archive = os.path.join(data_home, basename)\n datadir = os.path.join(data_home, name)\n\n # If the archive exists cleanup or raise override exception\n if os.path.exists(archive):\n if not replace:\n raise DatasetsError((\n \"dataset already exists at {}, set replace=False to overwrite\"\n ).format(archive))\n\n cleanup_dataset(name, data_home=data_home)\n\n # Create the output directory if it does not exist\n if not os.path.exists(datadir):\n os.mkdir(datadir)\n\n # Fetch the response in a streaming fashion and write it to disk.\n response = urlopen(url)\n\n with open(archive, 'wb') as f:\n while True:\n chunk = response.read(CHUNK)\n if not chunk:\n break\n f.write(chunk)\n\n # Compare the signature of the archive to the expected one\n if sha256sum(archive) != signature:\n raise ValueError(\n \"Download signature does not match hardcoded signature!\"\n )\n\n # If extract, extract the zipfile.\n if extract:\n zf = zipfile.ZipFile(archive)\n zf.extractall(path=data_home)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_aesthetic(palette=\"yellowbrick\", font=\"sans-serif\", font_scale=1,\n color_codes=True, rc=None):\n \"\"\"\n Set aesthetic parameters in one step.\n\n Each set of parameters can be set directly or temporarily, see the\n referenced functions below for more information.\n\n Parameters\n ----------\n palette : string or sequence\n Color palette, see :func:`color_palette`\n font : string\n Font family, see matplotlib font manager.\n font_scale : float, optional\n Separate scaling factor to independently scale the size of the\n font elements.\n color_codes : bool\n If ``True`` and ``palette`` is a yellowbrick palette, remap the shorthand\n color codes (e.g. \"b\", \"g\", \"r\", etc.) to the colors from this palette.\n rc : dict or None\n Dictionary of rc parameter mappings to override the above.\n \"\"\"\n _set_context(font_scale)\n set_style(rc={\"font.family\": font})\n set_palette(palette, color_codes=color_codes)\n if rc is not None:\n mpl.rcParams.update(rc)", "response": "Set the parameters of the aesthetic for this base class."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _axes_style(style=None, rc=None):\n if isinstance(style, dict):\n style_dict = style\n\n else:\n # Define colors here\n dark_gray = \".15\"\n light_gray = \".8\"\n\n # Common parameters\n style_dict = {\n \"figure.facecolor\": \"white\",\n \"text.color\": dark_gray,\n \"axes.labelcolor\": dark_gray,\n \"legend.frameon\": False,\n \"legend.numpoints\": 1,\n \"legend.scatterpoints\": 1,\n \"xtick.direction\": \"out\",\n \"ytick.direction\": \"out\",\n \"xtick.color\": dark_gray,\n \"ytick.color\": dark_gray,\n \"axes.axisbelow\": True,\n \"image.cmap\": \"Greys\",\n \"font.family\": [\"sans-serif\"],\n \"font.sans-serif\": [\"Arial\", \"Liberation Sans\",\n \"Bitstream Vera Sans\", \"sans-serif\"],\n \"grid.linestyle\": \"-\",\n \"axes.grid\": True,\n \"lines.solid_capstyle\": \"round\",\n \"axes.facecolor\": \"white\",\n \"axes.edgecolor\": light_gray,\n \"axes.linewidth\": 1.25,\n \"grid.color\": light_gray,\n \"xtick.major.size\": 0,\n \"ytick.major.size\": 0,\n \"xtick.minor.size\": 0,\n \"ytick.minor.size\": 0,\n }\n\n # Override these settings with the provided rc dictionary\n if rc is not None:\n rc = {k: v for k, v in rc.items() if k in _style_keys}\n style_dict.update(rc)\n\n # Wrap in an _AxesStyle object so this can be used in a with statement\n style_object = _AxesStyle(style_dict)\n\n return style_object", "response": "Returns a parameter dictionary for the aesthetic style of the aesthetic."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the aesthetic style of the plots.", "response": "def set_style(style=None, rc=None):\n \"\"\"\n Set the aesthetic style of the plots.\n\n This affects things like the color of the axes, whether a grid is\n enabled by default, and other aesthetic elements.\n\n Parameters\n ----------\n style : dict, None, or one of {darkgrid, whitegrid, dark, white, ticks}\n A dictionary of parameters or the name of a preconfigured set.\n rc : dict, optional\n Parameter mappings to override the values in the preset seaborn\n style dictionaries. This only updates parameters that are\n considered part of the style definition.\n \"\"\"\n style_object = _axes_style(style, rc)\n mpl.rcParams.update(style_object)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _set_context(context=None, font_scale=1, rc=None):\n context_object = _plotting_context(context, font_scale, rc)\n mpl.rcParams.update(context_object)", "response": "Set the plotting context parameters for a base context."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the matplotlib color cycle using a seaborn color palette.", "response": "def set_palette(palette, n_colors=None, color_codes=False):\n \"\"\"\n Set the matplotlib color cycle using a seaborn palette.\n\n Parameters\n ----------\n palette : yellowbrick color palette | seaborn color palette (with ``sns_`` prepended)\n Palette definition. Should be something that :func:`color_palette`\n can process.\n n_colors : int\n Number of colors in the cycle. The default number of colors will depend\n on the format of ``palette``, see the :func:`color_palette`\n documentation for more information.\n color_codes : bool\n If ``True`` and ``palette`` is a seaborn palette, remap the shorthand\n color codes (e.g. \"b\", \"g\", \"r\", etc.) to the colors from this palette.\n \"\"\"\n colors = color_palette(palette, n_colors)\n if mpl_ge_150:\n from cycler import cycler\n cyl = cycler('color', colors)\n mpl.rcParams['axes.prop_cycle'] = cyl\n else:\n mpl.rcParams[\"axes.color_cycle\"] = list(colors)\n mpl.rcParams[\"patch.facecolor\"] = colors[0]\n if color_codes:\n set_color_codes(palette)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndisplaying a dispersion plot for words within a corpus.", "response": "def dispersion(words, corpus, y=None, ax=None, colors=None, colormap=None,\n labels=None, annotate_docs=False, ignore_case=False, **kwargs):\n \"\"\" Displays lexical dispersion plot for words in a corpus\n\n This helper function is a quick wrapper to utilize the DisperstionPlot\n Visualizer for one-off analysis\n\n Parameters\n ----------\n\n words : list\n A list of words whose dispersion will be examined within a corpus\n\n y : ndarray or Series of length n\n An optional array or series of target or class values for\n instances. If this is specified, then the points will be colored\n according to their class.\n\n corpus : list\n Should be provided as a list of documents that contain\n a list of words in the order they appear in the document.\n\n ax : matplotlib axes, default: None\n The axes to plot the figure on.\n\n labels : list of strings\n The names of the classes in the target, used to create a legend.\n Labels must match names of classes in sorted order.\n\n colors : list or tuple of colors\n Specify the colors for each individual class\n\n colormap : string or matplotlib cmap\n Qualitative colormap for discrete target\n\n annotate_docs : boolean, default: False\n Specify whether document boundaries will be displayed. Vertical lines\n are positioned at the end of each document.\n\n ignore_case : boolean, default: False\n\tSpecify whether input will be case-sensitive.\n\n kwargs : dict\n Pass any additional keyword arguments to the super class.\n\n Returns\n -------\n ax: matplotlib axes\n Returns the axes that the plot was drawn on\n \"\"\"\n\n # Instantiate the visualizer\n visualizer = DispersionPlot(\n words, ax=ax, colors=colors, colormap=colormap,\n ignore_case=ignore_case, labels=labels,\n annotate_docs=annotate_docs, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(corpus, y, **kwargs)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, X, y=None, **kwargs):\n\n if y is not None:\n self.classes_ = np.unique(y)\n elif y is None and self.labels is not None:\n self.classes_ = np.array([self.labels[0]])\n else:\n self.classes_ = np.array([self.NULL_CLASS])\n\n # Create an index (e.g. the y position) for the target words\n self.indexed_words_ = np.flip(self.target_words, axis=0)\n if self.ignore_case:\n self.indexed_words_ = np.array([w.lower() for w in self.indexed_words_])\n\n # Stack is used to create a 2D array from the generator\n try:\n points_target = np.stack(self._compute_dispersion(X, y))\n except ValueError:\n raise YellowbrickValueError((\n \"No indexed words were found in the corpus\"\n ))\n points = np.stack(zip(points_target[:,0].astype(int),\n points_target[:,1].astype(int)))\n\n self.target = points_target[:,2]\n\n self._check_missing_words(points)\n\n self.draw(points, self.target)\n return self", "response": "This method is the primary drawing input for the dispersion\n visualization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate the canvas and draws the plot on it.", "response": "def draw(self, points, target=None, **kwargs):\n \"\"\"\n Called from the fit method, this method creates the canvas and\n draws the plot on it.\n Parameters\n ----------\n kwargs: generic keyword arguments.\n \"\"\"\n\n # Resolve the labels with the classes\n labels = self.labels if self.labels is not None else self.classes_\n if len(labels) != len(self.classes_):\n raise YellowbrickValueError((\n \"number of supplied labels ({}) does not \"\n \"match the number of classes ({})\"\n ).format(len(labels), len(self.classes_)))\n\n # Create the color mapping for the labels.\n color_values = resolve_colors(\n n_colors=len(labels), colormap=self.colormap, colors=self.color)\n colors = dict(zip(labels, color_values))\n\n # Transform labels into a map of class to label\n labels = dict(zip(self.classes_, labels))\n\n # Define boundaries with a vertical line\n if self.annotate_docs:\n for xcoords in self.boundaries_:\n self.ax.axvline(x=xcoords, color='lightgray', linestyle='dashed')\n\n series = defaultdict(lambda: {'x':[], 'y':[]})\n\n if target is not None:\n for point, t in zip(points, target):\n label = labels[t]\n series[label]['x'].append(point[0])\n series[label]['y'].append(point[1])\n else:\n label = self.classes_[0]\n for x, y in points:\n series[label]['x'].append(x)\n series[label]['y'].append(y)\n\n for label, points in series.items():\n self.ax.scatter(points['x'], points['y'], marker='|',\n c=colors[label], zorder=100, label=label)\n\n self.ax.set_yticks(list(range(len(self.indexed_words_))))\n self.ax.set_yticklabels(self.indexed_words_)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef manifold(self, transformer):\n if not is_estimator(transformer):\n if transformer not in self.ALGORITHMS:\n raise YellowbrickValueError(\n \"could not create manifold for '%s'\".format(str(transformer))\n )\n\n # Create a new transformer with the specified params\n self._name = MANIFOLD_NAMES[transformer]\n transformer = clone(self.ALGORITHMS[transformer])\n params = {\n \"n_components\": 2,\n \"n_neighbors\": self.n_neighbors,\n \"random_state\": self.random_state,\n }\n\n for param in list(params.keys()):\n if param not in transformer.get_params():\n del params[param]\n\n transformer.set_params(**params)\n\n self._manifold = transformer\n if self._name is None:\n self._name = self._manifold.__class__.__name__", "response": "Creates the manifold estimator if a string value is passed in validates other objects passed in."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfit the manifold on X and transforms the data to plot it on the axes.", "response": "def fit_transform(self, X, y=None):\n \"\"\"\n Fits the manifold on X and transforms the data to plot it on the axes.\n The optional y specified can be used to declare discrete colors. If\n the target is set to 'auto', this method also determines the target\n type, and therefore what colors will be used.\n\n Note also that fit records the amount of time it takes to fit the\n manifold and reports that information in the visualization.\n\n Parameters\n ----------\n X : array-like of shape (n, m)\n A matrix or data frame with n instances and m features where m > 2.\n\n y : array-like of shape (n,), optional\n A vector or series with target values for each instance in X. This\n vector is used to determine the color of the points in X.\n\n Returns\n -------\n self : Manifold\n Returns the visualizer object.\n \"\"\"\n # Determine target type\n self._determine_target_color_type(y)\n\n # Compute classes and colors if target type is discrete\n if self._target_color_type == DISCRETE:\n self.classes_ = np.unique(y)\n\n color_kwargs = {'n_colors': len(self.classes_)}\n\n if isinstance(self.colors, str):\n color_kwargs['colormap'] = self.colors\n else:\n color_kwargs['colors'] = self.colors\n\n self._colors = resolve_colors(**color_kwargs)\n\n # Compute target range if colors are continuous\n elif self._target_color_type == CONTINUOUS:\n y = np.asarray(y)\n self.range_ = (y.min(), y.max())\n\n with Timer() as self.fit_time_:\n Xp = self.manifold.fit_transform(X)\n\n self.draw(Xp, y)\n return Xp"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transform(self, X):\n try:\n return self.manifold.transform(X)\n except AttributeError as e:\n raise AttributeError(str(e) + \" try using fit_transform instead.\")", "response": "Returns the transformed data points from the manifold embedding."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(self, X, y=None):\n scatter_kwargs = {\"alpha\": self.alpha}\n\n # Determine the colors\n if self._target_color_type == SINGLE:\n scatter_kwargs[\"c\"] = \"b\"\n\n elif self._target_color_type == DISCRETE:\n if y is None:\n raise YellowbrickValueError(\"y is required for discrete target\")\n\n scatter_kwargs[\"c\"] = [\n self._colors[np.searchsorted(self.classes_, (yi))] for yi in y\n ]\n\n elif self._target_color_type == CONTINUOUS:\n if y is None:\n raise YellowbrickValueError(\"y is required for continuous target\")\n\n # TODO manually make colorbar so we can draw it in finalize\n scatter_kwargs[\"c\"] = y\n scatter_kwargs[\"cmap\"] = self.colors or palettes.DEFAULT_SEQUENCE\n\n else:\n # Technically this should never be raised\n raise NotFitted(\"could not determine target color type\")\n\n # Draw the scatter plot with the associated colors and alpha\n self._scatter = self.ax.scatter(X[:,0], X[:,1], **scatter_kwargs)\n return self.ax", "response": "Draws the points described by X and colored by the points in y."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef finalize(self):\n self.set_title(\n '{} Manifold (fit in {:0.2f} seconds)'.format(\n self._name, self.fit_time_.interval\n )\n )\n self.ax.set_xticklabels([])\n self.ax.set_yticklabels([])\n\n if self._target_color_type == DISCRETE:\n # Add the legend\n manual_legend(self, self.classes_, self._colors, frameon=True)\n\n elif self._target_color_type == CONTINUOUS:\n # Add the color bar\n plt.colorbar(self._scatter, ax=self.ax)", "response": "Add title and modify axes to make the image ready for display."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _determine_target_color_type(self, y):\n if y is None:\n self._target_color_type = SINGLE\n elif self.target == \"auto\":\n # NOTE: See #73 for a generalization to use when implemented\n if len(np.unique(y)) < 10:\n self._target_color_type = DISCRETE\n else:\n self._target_color_type = CONTINUOUS\n else:\n self._target_color_type = self.target\n\n if self._target_color_type not in {SINGLE, DISCRETE, CONTINUOUS}:\n raise YellowbrickValueError((\n \"could not determine target color type \"\n \"from target='{}' to '{}'\"\n ).format(self.target, self._target_color_type))", "response": "Determines the target color type from the vector y."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rfecv(model, X, y, ax=None, step=1, groups=None, cv=None,\n scoring=None, **kwargs):\n \"\"\"\n Performs recursive feature elimination with cross-validation to determine\n an optimal number of features for a model. Visualizes the feature subsets\n with respect to the cross-validation score.\n\n This helper function is a quick wrapper to utilize the RFECV visualizer\n for one-off analysis.\n\n Parameters\n ----------\n model : a scikit-learn estimator\n An object that implements ``fit`` and provides information about the\n relative importance of features with either a ``coef_`` or\n ``feature_importances_`` attribute.\n\n Note that the object is cloned for each validation.\n\n X : array-like, shape (n_samples, n_features)\n Training vector, where n_samples is the number of samples and\n n_features is the number of features.\n\n y : array-like, shape (n_samples) or (n_samples, n_features), optional\n Target relative to X for classification or regression.\n\n ax : matplotlib.Axes object, optional\n The axes object to plot the figure on.\n\n step : int or float, optional (default=1)\n If greater than or equal to 1, then step corresponds to the (integer)\n number of features to remove at each iteration. If within (0.0, 1.0),\n then step corresponds to the percentage (rounded down) of features to\n remove at each iteration.\n\n groups : array-like, with shape (n_samples,), optional\n Group labels for the samples used while splitting the dataset into\n train/test set.\n\n cv : int, cross-validation generator or an iterable, optional\n Determines the cross-validation splitting strategy.\n Possible inputs for cv are:\n\n - None, to use the default 3-fold cross-validation,\n - integer, to specify the number of folds.\n - An object to be used as a cross-validation generator.\n - An iterable yielding train/test splits.\n\n see the scikit-learn\n `cross-validation guide `_\n for more information on the possible strategies that can be used here.\n\n scoring : string, callable or None, optional, default: None\n A string or scorer callable object / function with signature\n ``scorer(estimator, X, y)``. See scikit-learn model evaluation\n documentation for names of possible metrics.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers. These arguments are\n also passed to the `poof()` method, e.g. can pass a path to save the\n figure to.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the rfecv were drawn on.\n \"\"\"\n # Initialize the visualizer\n oz = RFECV(model, ax=ax, step=step, groups=groups, cv=cv, scoring=scoring)\n\n # Fit and poof the visualizer\n oz.fit(X, y)\n oz.poof(**kwargs)\n return oz.ax", "response": "This function is used to visualize the RFECV model for one - off analysis."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfits the RFECV with the wrapped model to the specified data and draws the RFECV curve with the optimal number of features found.", "response": "def fit(self, X, y=None):\n \"\"\"\n Fits the RFECV with the wrapped model to the specified data and draws\n the rfecv curve with the optimal number of features found.\n\n Parameters\n ----------\n X : array-like, shape (n_samples, n_features)\n Training vector, where n_samples is the number of samples and\n n_features is the number of features.\n\n y : array-like, shape (n_samples) or (n_samples, n_features), optional\n Target relative to X for classification or regression.\n\n Returns\n -------\n self : instance\n Returns the instance of the RFECV visualizer.\n \"\"\"\n X, y = check_X_y(X, y, \"csr\")\n n_features = X.shape[1]\n\n # This check is kind of unnecessary since RFE will do it, but it's\n # nice to get it out of the way ASAP and raise a meaningful error.\n if 0.0 < self.step < 1.0:\n step = int(max(1, self.step * n_features))\n else:\n step = int(self.step)\n\n if step < 0:\n raise YellowbrickValueError(\"step must be >0\")\n\n # Create the RFE model\n rfe = RFE(self.estimator, step=step)\n self.n_feature_subsets_ = np.arange(1, n_features+step, step)\n\n # Create the cross validation params\n # TODO: handle random state\n cv_params = {\n key: self.get_params()[key]\n for key in ('groups', 'cv', 'scoring')\n }\n\n # Perform cross-validation for each feature subset\n scores = []\n for n_features_to_select in self.n_feature_subsets_:\n rfe.set_params(n_features_to_select=n_features_to_select)\n scores.append(cross_val_score(rfe, X, y, **cv_params))\n\n # Convert scores to array\n self.cv_scores_ = np.array(scores)\n\n # Find the best RFE model\n bestidx = self.cv_scores_.mean(axis=1).argmax()\n self.n_features_ = self.n_feature_subsets_[bestidx]\n\n # Fit the final RFE model for the number of features\n self.rfe_estimator_ = rfe\n self.rfe_estimator_.set_params(n_features_to_select=self.n_features_)\n self.rfe_estimator_.fit(X, y)\n\n # Rewrap the visualizer to use the rfe estimator\n self._wrapped = self.rfe_estimator_\n\n # Hoist the RFE params to the visualizer\n self.support_ = self.rfe_estimator_.support_\n self.ranking_ = self.rfe_estimator_.ranking_\n\n self.draw()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef draw(self, **kwargs):\n # Compute the curves\n x = self.n_feature_subsets_\n means = self.cv_scores_.mean(axis=1)\n sigmas = self.cv_scores_.std(axis=1)\n\n\n # Plot one standard deviation above and below the mean\n self.ax.fill_between(x, means - sigmas, means+sigmas, alpha=0.25)\n\n # Plot the curve\n self.ax.plot(x, means, 'o-')\n\n # Plot the maximum number of features\n self.ax.axvline(\n self.n_features_, c='k', ls='--',\n label=\"n_features = {}\\nscore = {:0.3f}\".format(\n self.n_features_, self.cv_scores_.mean(axis=1).max()\n )\n )\n\n return self.ax", "response": "Renders the rfecv curve."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef scatterviz(X,\n y=None,\n ax=None,\n features=None,\n classes=None,\n color=None,\n colormap=None,\n markers=None,\n alpha=1.0,\n **kwargs):\n \"\"\"Displays a bivariate scatter plot.\n\n This helper function is a quick wrapper to utilize the ScatterVisualizer\n (Transformer) for one-off analysis.\n\n Parameters\n ----------\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n, default: None\n An array or series of target or class values\n\n ax : matplotlib axes, default: None\n The axes to plot the figure on.\n\n features : list of strings, default: None\n The names of two features or columns.\n More than that will raise an error.\n\n classes : list of strings, default: None\n The names of the classes in the target\n\n color : list or tuple of colors, default: None\n Specify the colors for each individual class\n\n colormap : string or matplotlib cmap, default: None\n Sequential colormap for continuous target\n\n markers : iterable of strings, default: ,+o*vhd\n Matplotlib style markers for points on the scatter plot points\n\n alpha : float, default: 1.0\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = ScatterVisualizer(ax=ax, features=features, classes=classes,\n color=color, colormap=colormap,\n markers=markers, alpha=alpha, **kwargs)\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Displays a bivariate scatter plot for one - off analysis."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfitting the transformer and visualizer to the target or class values of the object.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n The fit method is the primary drawing input for the parallel coords\n visualization since it has both the X and y data required for the\n viz and the transform method does not.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with 2 features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n kwargs : dict\n Pass generic arguments to the drawing method\n\n Returns\n -------\n self : instance\n Returns the instance of the transformer/visualizer\n \"\"\"\n _, ncols = X.shape\n\n if ncols == 2:\n X_two_cols = X\n if self.features_ is None:\n self.features_ = [\"Feature One\", \"Feature Two\"]\n\n # Handle the feature names if they're None.\n elif self.features_ is not None and is_dataframe(X):\n X_two_cols = X[self.features_].as_matrix()\n\n # handle numpy named/ structured array\n elif self.features_ is not None and is_structured_array(X):\n X_selected = X[self.features_]\n X_two_cols = X_selected.copy().view((np.float64, len(X_selected.dtype.names)))\n\n # handle features that are numeric columns in ndarray matrix\n elif self.features_ is not None and has_ndarray_int_columns(self.features_, X):\n f_one, f_two = self.features_\n X_two_cols = X[:, [int(f_one), int(f_two)]]\n\n else:\n raise YellowbrickValueError(\"\"\"\n ScatterVisualizer only accepts two features, please\n explicitly set these two features in the init kwargs or\n pass a matrix/ dataframe in with only two columns.\"\"\")\n\n # Store the classes for the legend if they're None.\n if self.classes_ is None:\n # TODO: Is this the most efficient method?\n self.classes_ = [str(label) for label in np.unique(y)]\n\n # Draw the instances\n self.draw(X_two_cols, y, **kwargs)\n\n # Fit always returns self.\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall from the fit method creates a scatter plot that draws each instance of the class or target colored point whose location is determined by the feature data set. This method creates a scatter plot that draws each instance of the class or target colored point whose location is determined by the feature data set.", "response": "def draw(self, X, y, **kwargs):\n \"\"\"Called from the fit method, this method creates a scatter plot that\n draws each instance as a class or target colored point, whose location\n is determined by the feature data set.\n \"\"\"\n # Set the axes limits\n self.ax.set_xlim([-1,1])\n self.ax.set_ylim([-1,1])\n\n # set the colors\n color_values = resolve_colors(\n n_colors=len(self.classes_),\n colormap=self.colormap,\n colors=self.color\n )\n\n colors = dict(zip(self.classes_, color_values))\n\n # Create a data structure to hold the scatter plot representations\n to_plot = {}\n for kls in self.classes_:\n to_plot[kls] = [[], []]\n\n # Add each row of the data set to to_plot for plotting\n # TODO: make this an independent function for override\n for i, row in enumerate(X):\n row_ = np.repeat(np.expand_dims(row, axis=1), 2, axis=1)\n x_, y_ = row_[0], row_[1]\n kls = self.classes_[y[i]]\n\n to_plot[kls][0].append(x_)\n to_plot[kls][1].append(y_)\n\n # Add the scatter plots from the to_plot function\n # TODO: store these plots to add more instances to later\n # TODO: make this a separate function\n for i, kls in enumerate(self.classes_):\n self.ax.scatter(\n to_plot[kls][0],\n to_plot[kls][1],\n marker=next(self.markers),\n color=colors[kls],\n label=str(kls),\n alpha=self.alpha,\n **kwargs)\n\n self.ax.axis('equal')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef finalize(self, **kwargs):\n # Divide out the two features\n feature_one, feature_two = self.features_\n\n # Set the title\n self.set_title('Scatter Plot: {0} vs {1}'.format(\n str(feature_one), str(feature_two)))\n # Add the legend\n self.ax.legend(loc='best')\n self.ax.set_xlabel(str(feature_one))\n self.ax.set_ylabel(str(feature_two))", "response": "Executes any subclass - specific axes finalization steps."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef gridsearch_color_plot(model, x_param, y_param, X=None, y=None, ax=None,\n **kwargs):\n \"\"\"Quick method:\n Create a color plot showing the best grid search scores across two\n parameters.\n\n This helper function is a quick wrapper to utilize GridSearchColorPlot\n for one-off analysis.\n\n If no `X` data is passed, the model is assumed to be fit already. This\n allows quick exploration without waiting for the grid search to re-run.\n\n Parameters\n ----------\n model : Scikit-Learn grid search object\n Should be an instance of GridSearchCV. If not, an exception is raised.\n The model may be fit or unfit.\n\n x_param : string\n The name of the parameter to be visualized on the horizontal axis.\n\n y_param : string\n The name of the parameter to be visualized on the vertical axis.\n\n metric : string (default 'mean_test_score')\n The field from the grid search's `cv_results` that we want to display.\n\n X : ndarray or DataFrame of shape n x m or None (default None)\n A matrix of n instances with m features. If not None, forces the\n GridSearchCV object to be fit.\n\n y : ndarray or Series of length n or None (default None)\n An array or series of target or class values.\n\n ax : matplotlib axes\n The axes to plot the figure on.\n\n classes : list of strings\n The names of the classes in the target\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the classification report was drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = GridSearchColorPlot(model, x_param, y_param, ax=ax, **kwargs)\n\n # Fit if necessary\n if X is not None:\n visualizer.fit(X, y)\n else:\n visualizer.draw()\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Create a color plot showing the best grid search scores across two parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef filter_missing(X, y=None):\n if y is not None:\n return filter_missing_X_and_y(X, y)\n else:\n return X[~np.isnan(X).any(axis=1)]", "response": "This function filters out any missing values in data and returns a tuple of the nans in the order they appear in X."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef filter_missing_X_and_y(X, y):\n y_nans = np.isnan(y)\n x_nans = np.isnan(X).any(axis=1)\n unioned_nans = np.logical_or(x_nans, y_nans)\n\n return X[~unioned_nans], y[~unioned_nans]", "response": "Remove rows from X and y where either contains nans."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef warn_if_nans_exist(X):\n null_count = count_rows_with_nans(X)\n total = len(X)\n percent = 100 * null_count / total\n\n if null_count > 0:\n warning_message = \\\n 'Warning! Found {} rows of {} ({:0.2f}%) with nan values. Only ' \\\n 'complete rows will be plotted.'.format(null_count, total, percent)\n warnings.warn(warning_message, DataWarning)", "response": "Warn if nans exist in a numpy array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef count_rows_with_nans(X):\n if X.ndim == 2:\n return np.where(np.isnan(X).sum(axis=1) != 0, 1, 0).sum()", "response": "Count the number of rows in 2D arrays that contain any nan values."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X, y):\n\n # note that GLM takes endog (y) and then exog (X):\n # this is the reverse of sklearn's methods\n self.glm_model = self.glm_partial(y, X)\n self.glm_results = self.glm_model.fit()\n return self", "response": "Fit the GLM model on the data X and y."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef missing_dispersion(X, y=None, ax=None, classes=None, alpha=0.5, marker=\"|\", **kwargs):\n # Instantiate the visualizer\n visualizer = MissingValuesDispersion(\n ax=ax, classes=classes, alpha=alpha, marker=marker, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y)\n visualizer.poof()\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Creates a Missing Values Dispersion visualizer that shows the locations of missing values in the feature dataset."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_nan_locs(self, **kwargs):\n if np.issubdtype(self.X.dtype, np.string_) or np.issubdtype(self.X.dtype, np.unicode_):\n mask = np.where( self.X == '' )\n nan_matrix = np.zeros(self.X.shape)\n nan_matrix[mask] = np.nan\n\n else:\n nan_matrix = self.X.astype(float)\n\n if self.y is None:\n return np.argwhere(np.isnan(nan_matrix))\n else:\n nan_locs = []\n for target_value in np.unique(self.y):\n indices = np.argwhere(self.y == target_value)\n target_matrix = nan_matrix[indices.flatten()]\n nan_target_locs = np.argwhere(np.isnan(target_matrix))\n nan_locs.append((target_value, nan_target_locs))\n\n return nan_locs", "response": "Gets the locations of nans in feature data and returns\n the coordinates in the matrix"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalling from the fit method this method creates a scatter plot that draws each instance of a class or target colored point whose location is determined by the feature data set.", "response": "def draw(self, X, y, **kwargs):\n \"\"\"Called from the fit method, this method creates a scatter plot that\n draws each instance as a class or target colored point, whose location\n is determined by the feature data set.\n\n If y is not None, then it draws a scatter plot where each class is in a\n different color.\n \"\"\"\n nan_locs = self.get_nan_locs()\n if y is None:\n x_, y_ = list(zip(*nan_locs))\n self.ax.scatter(x_, y_, alpha=self.alpha, marker=self.marker, label=None)\n else:\n self.draw_multi_dispersion_chart(nan_locs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndraws a multi dimensional dispersion chart for each color corresponds to a different target variable.", "response": "def draw_multi_dispersion_chart(self, nan_locs):\n \"\"\"Draws a multi dimensional dispersion chart, each color corresponds\n to a different target variable.\n \"\"\"\n for index, nan_values in enumerate(nan_locs):\n label, nan_locations = nan_values\n\n # if features passed in then, label as such\n if self.classes_ is not None:\n label = self.classes_[index]\n\n color = self.colors[index]\n\n x_, y_ = list(zip(*nan_locations))\n self.ax.scatter(x_, y_, alpha=self.alpha, marker=self.marker, color=color, label=label)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nproject the grid search results onto 2 dimensions. The display value is taken as the max over the non-displayed dimensions. Parameters ---------- cv_results : dict A dictionary of results from the `GridSearchCV` object's `cv_results_` attribute. x_param : string The name of the parameter to be visualized on the horizontal axis. y_param : string The name of the parameter to be visualized on the vertical axis. metric : string (default 'mean_test_score') The field from the grid search's `cv_results` that we want to display. Returns ------- unique_x_vals : list The parameter values that will be used to label the x axis. unique_y_vals: list The parameter values that will be used to label the y axis. best_scores: 2D numpy array (n_y by n_x) Array of scores to be displayed for each parameter value pair.", "response": "def param_projection(cv_results, x_param, y_param, metric='mean_test_score'):\n \"\"\"\n Projects the grid search results onto 2 dimensions.\n\n The display value is taken as the max over the non-displayed dimensions.\n\n Parameters\n ----------\n cv_results : dict\n A dictionary of results from the `GridSearchCV` object's `cv_results_`\n attribute.\n\n x_param : string\n The name of the parameter to be visualized on the horizontal axis.\n\n y_param : string\n The name of the parameter to be visualized on the vertical axis.\n\n metric : string (default 'mean_test_score')\n The field from the grid search's `cv_results` that we want to display.\n\n Returns\n -------\n unique_x_vals : list\n The parameter values that will be used to label the x axis.\n\n unique_y_vals: list\n The parameter values that will be used to label the y axis.\n\n best_scores: 2D numpy array (n_y by n_x)\n Array of scores to be displayed for each parameter value pair.\n \"\"\"\n # Extract the parameter values and score corresponding to each gridsearch\n # trial.\n # These are masked arrays where the cases where each parameter is\n # non-applicable are masked.\n try:\n x_vals = cv_results['param_' + x_param]\n except KeyError:\n raise YellowbrickKeyError(\"Parameter '{}' does not exist in the grid \"\n \"search results\".format(x_param))\n try:\n y_vals = cv_results['param_' + y_param]\n except KeyError:\n raise YellowbrickKeyError(\"Parameter '{}' does not exist in the grid \"\n \"search results\".format(y_param))\n\n if metric not in cv_results:\n raise YellowbrickKeyError(\"Metric '{}' does not exist in the grid \"\n \"search results\".format(metric))\n\n # Get unique, unmasked values of the two display parameters\n unique_x_vals = sorted(list(set(x_vals.compressed())))\n unique_y_vals = sorted(list(set(y_vals.compressed())))\n n_x = len(unique_x_vals)\n n_y = len(unique_y_vals)\n\n # Get mapping of each parameter value -> an integer index\n int_mapping_1 = {value: idx for idx, value in enumerate(unique_x_vals)}\n int_mapping_2 = {value: idx for idx, value in enumerate(unique_y_vals)}\n\n # Translate each gridsearch result to indices on the grid\n idx_x = [int_mapping_1[value] if value else None for value in x_vals]\n idx_y = [int_mapping_2[value] if value else None for value in y_vals]\n\n # Create an array of all scores for each value of the display parameters.\n # This is a n_x by n_y array of lists with `None` in place of empties\n # (my kingdom for a dataframe...)\n all_scores = [[None for _ in range(n_x)] for _ in range(n_y)]\n for x, y, score in zip(idx_x, idx_y, cv_results[metric]):\n if x is not None and y is not None:\n if all_scores[y][x] is None:\n all_scores[y][x] = []\n all_scores[y][x].append(score)\n\n # Get a numpy array consisting of the best scores for each parameter pair\n best_scores = np.empty((n_y, n_x))\n for x in range(n_x):\n for y in range(n_y):\n if all_scores[y][x] is None:\n best_scores[y, x] = np.nan\n else:\n try:\n best_scores[y, x] = max(all_scores[y][x])\n except ValueError:\n raise YellowbrickValueError(\n \"Cannot display grid search results for metric '{}': \"\n \"result values may not all be numeric\".format(metric)\n )\n\n return unique_x_vals, unique_y_vals, best_scores"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef param_projection(self, x_param, y_param, metric):\n return param_projection(self.estimator.cv_results_, x_param, y_param, metric)", "response": "This method projects the grid search results onto 2 dimensions and returns a 2D numpy array that contains the max over the non - displayed dimensions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __threshold(self, ymx_i):\n return ymx_i - (self.S * np.diff(self.xsn).mean())", "response": "Calculates the difference threshold for a specific local maximum."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding and returns the knee or elbow value normalized knee or None.", "response": "def find_knee(self, ):\n \"\"\"\n Finds and returns the \"knee\"or \"elbow\" value, the normalized knee\n value, and the x value where the knee is located.\n\n \"\"\"\n if not self.xmx_idx.size:\n warning_message = \\\n 'No \"knee\" or \"elbow point\" detected ' \\\n 'This could be due to bad clustering, no '\\\n 'actual clusters being formed etc.' \n warnings.warn(warning_message,YellowbrickWarning)\n return None, None, None\n\n mxmx_iter = np.arange(self.xmx_idx[0], len(self.xsn))\n xmx_idx_iter = np.append(self.xmx_idx, len(self.xsn))\n\n knee_, norm_knee_, knee_x = 0.0, 0.0, None\n for mxmx_i, mxmx in enumerate(xmx_idx_iter):\n # stopping criteria for exhasuting array\n if mxmx_i == len(xmx_idx_iter) - 1:\n break\n # indices between maxima/minima\n idxs = (mxmx_iter > xmx_idx_iter[mxmx_i]) * \\\n (mxmx_iter < xmx_idx_iter[mxmx_i + 1])\n between_local_mx = mxmx_iter[np.where(idxs)]\n\n for j in between_local_mx:\n if j in self.xmn_idx:\n # reached a minima, x indices are unique\n # only need to check if j is a min\n if self.yd[j + 1] > self.yd[j]:\n self.Tmx[mxmx_i] = 0\n knee_x = None # reset x where yd crossed Tmx\n elif self.yd[j + 1] <= self.yd[j]:\n warning_message=\"If this is a minima, \" \\\n \"how would you ever get here.\"\n warnings.warn(warning_message, YellowbrickWarning)\n if self.yd[j] < self.Tmx[mxmx_i] or self.Tmx[mxmx_i] < 0:\n # declare a knee\n if not knee_x:\n knee_x = j\n knee_ = self.x[self.xmx_idx[mxmx_i]]\n norm_knee_ = self.xsn[self.xmx_idx[mxmx_i]]\n return knee_, norm_knee_, knee_x"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nplotting the normalized curve the distance curve and the knee", "response": "def plot_knee_normalized(self, ):\n \"\"\"\n Plots the normalized curve, the distance curve (xd, ysn) and the\n knee, if it exists.\n\n \"\"\"\n import matplotlib.pyplot as plt\n\n plt.figure(figsize=(8, 8))\n plt.plot(self.xsn, self.ysn)\n plt.plot(self.xd, self.yd, 'r')\n plt.xticks(np.arange(min(self.xsn), max(self.xsn) + 0.1, 0.1))\n plt.yticks(np.arange(min(self.xd), max(self.ysn) + 0.1, 0.1))\n\n plt.vlines(self.norm_knee, plt.ylim()[0], plt.ylim()[1])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots the curve and the knee if it exists", "response": "def plot_knee(self, ):\n \"\"\"\n Plot the curve and the knee, if it exists\n \n \"\"\"\n import matplotlib.pyplot as plt\n\n plt.figure(figsize=(8, 8))\n plt.plot(self.x, self.y)\n plt.vlines(self.knee, plt.ylim()[0], plt.ylim()[1])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X, y, **kwargs):\n self.estimator.fit(X, y, **kwargs)\n self.draw()\n return self", "response": "A simple pass - through method that draws the alpha - error plot."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw(self):\n # Search for the correct parameters on the estimator.\n alphas = self._find_alphas_param()\n errors = self._find_errors_param()\n\n\n alpha = self.estimator.alpha_ # Get decision from the estimator\n name = self.name[:-2].lower() # Remove the CV from the label\n\n # Plot the alpha against the error\n self.ax.plot(alphas, errors, label=name)\n\n # Draw a dashed vline at the alpha\n label = \"$\\\\alpha={:0.3f}$\".format(alpha)\n self.ax.axvline(alpha, color='k', linestyle='dashed', label=label)\n\n return self.ax", "response": "Draws the alpha plot based on the values on the estimator."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsearches for the parameter on the estimator that contains the array of alphas that was used to produce the error selection.", "response": "def _find_alphas_param(self):\n \"\"\"\n Searches for the parameter on the estimator that contains the array of\n alphas that was used to produce the error selection. If it cannot find\n the parameter then a YellowbrickValueError is raised.\n \"\"\"\n\n # NOTE: The order of the search is very important!\n for attr in (\"cv_alphas_\", \"alphas_\", \"alphas\",):\n try:\n return getattr(self.estimator, attr)\n except AttributeError:\n continue\n\n raise YellowbrickValueError(\n \"could not find alphas param on {} estimator\".format(\n self.estimator.__class__.__name__\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsearching for the parameter that contains the array of errors that was used to determine the optimal alpha.", "response": "def _find_errors_param(self):\n \"\"\"\n Searches for the parameter on the estimator that contains the array of\n errors that was used to determine the optimal alpha. If it cannot find\n the parameter then a YellowbrickValueError is raised.\n \"\"\"\n\n # NOTE: The order of the search is very important!\n if hasattr(self.estimator, 'mse_path_'):\n return self.estimator.mse_path_.mean(1)\n\n if hasattr(self.estimator, 'cv_values_'):\n return self.estimator.cv_values_.mean(0)\n\n raise YellowbrickValueError(\n \"could not find errors param on {} estimator\".format(\n self.estimator.__class__.__name__\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndraws the alphas values against their associated error in a similar similar fashion to the AlphaSelection visualizer.", "response": "def draw(self):\n \"\"\"\n Draws the alphas values against their associated error in a similar\n fashion to the AlphaSelection visualizer.\n \"\"\"\n # Plot the alpha against the error\n self.ax.plot(self.alphas, self.errors, label=self.name.lower())\n\n # Draw a dashed vline at the alpha with maximal error\n alpha = self.alphas[np.where(self.errors == self.errors.max())][0]\n label = \"$\\\\alpha_{{max}}={:0.3f}$\".format(alpha)\n self.ax.axvline(alpha, color='k', linestyle='dashed', label=label)\n\n # Draw a dashed vline at the alpha with minimal error\n alpha = self.alphas[np.where(self.errors == self.errors.min())][0]\n label = \"$\\\\alpha_{{min}}={:0.3f}$\".format(alpha)\n self.ax.axvline(alpha, color='k', linestyle='dashed', label=label)\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef postag(\n X,\n ax=None,\n tagset=\"penn_treebank\",\n colormap=None,\n colors=None,\n frequency=False,\n **kwargs\n):\n \"\"\"\n Display a barchart with the counts of different parts of speech\n in X, which consists of a part-of-speech-tagged corpus, which the\n visualizer expects to be a list of lists of lists of (token, tag)\n tuples.\n\n Parameters\n ----------\n X : list or generator\n Should be provided as a list of documents or a generator\n that yields a list of documents that contain a list of\n sentences that contain (token, tag) tuples.\n ax : matplotlib axes\n The axes to plot the figure on.\n tagset: string\n The tagset that was used to perform part-of-speech tagging.\n Either \"penn_treebank\" or \"universal\", defaults to \"penn_treebank\".\n Use \"universal\" if corpus has been tagged using SpaCy.\n colors : list or tuple of colors\n Specify the colors for each individual part-of-speech.\n colormap : string or matplotlib cmap\n Specify a colormap to color the parts-of-speech.\n frequency: bool {True, False}, default: False\n If set to True, part-of-speech tags will be plotted according to frequency,\n from most to least frequent.\n kwargs : dict\n Pass any additional keyword arguments to the PosTagVisualizer.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes on which the PosTagVisualizer was drawn.\n \"\"\"\n # Instantiate the visualizer\n visualizer = PosTagVisualizer(\n ax=ax, tagset=tagset, colors=colors, colormap=colormap,\n frequency=frequency, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, **kwargs)\n\n # Return the axes object on the visualizer\n return visualizer", "response": "Display a barchart with the counts of different parts of speech\n in X."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfit the corpus to the appropriate tag map.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n Fits the corpus to the appropriate tag map.\n Text documents must be tokenized & tagged before passing to fit.\n\n Parameters\n ----------\n X : list or generator\n Should be provided as a list of documents or a generator\n that yields a list of documents that contain a list of\n sentences that contain (token, tag) tuples.\n\n y : ndarray or Series of length n\n An optional array of target values that are ignored by the\n visualizer.\n\n kwargs : dict\n Pass generic arguments to the drawing method\n\n Returns\n -------\n self : instance\n Returns the instance of the transformer/visualizer\n \"\"\"\n # TODO: add support for other tagsets?\n if self.tagset == \"penn_treebank\":\n self.pos_tag_counts_ = self._penn_tag_map()\n self._handle_treebank(X)\n\n elif self.tagset == \"universal\":\n self.pos_tag_counts_ = self._uni_tag_map()\n self._handle_universal(X)\n\n self.draw()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _handle_universal(self, X):\n jump = {\n # combine proper and regular nouns\n \"NOUN\": \"noun\", \"PROPN\": \"noun\",\n \"ADJ\": \"adjective\",\n \"VERB\": \"verb\",\n # include particles with adverbs\n \"ADV\": \"adverb\", \"PART\": \"adverb\",\n \"ADP\": \"adposition\",\n \"PRON\": \"pronoun\",\n \"CCONJ\": \"conjunction\",\n \"PUNCT\": \"punctuation\",\n \"DET\": \"determiner\",\n \"NUM\": \"number\",\n \"INTJ\": \"interjection\",\n \"SYM\": \"symbol\",\n }\n\n for tagged_doc in X:\n for tagged_sent in tagged_doc:\n for _, tag in tagged_sent:\n if tag == \"SPACE\":\n continue\n self.pos_tag_counts_[jump.get(tag, \"other\")] += 1", "response": "Handles Universal related information."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _handle_treebank(self, X):\n for tagged_doc in X:\n for tagged_sent in tagged_doc:\n for _, tag in tagged_sent:\n if tag.startswith(\"N\"):\n self.pos_tag_counts_[\"noun\"] += 1\n elif tag.startswith(\"J\"):\n self.pos_tag_counts_[\"adjective\"] += 1\n elif tag.startswith(\"V\"):\n self.pos_tag_counts_[\"verb\"] += 1\n # include particles with adverbs\n elif tag.startswith(\"RB\") or tag == \"RP\":\n self.pos_tag_counts_[\"adverb\"] += 1\n elif tag.startswith(\"PR\"):\n self.pos_tag_counts_[\"pronoun\"] += 1\n elif tag.startswith(\"W\"):\n self.pos_tag_counts_[\"wh- word\"] += 1\n elif tag == \"CC\":\n self.pos_tag_counts_[\"conjunction\"] += 1\n elif tag == \"CD\":\n self.pos_tag_counts_[\"digit\"] += 1\n # combine predeterminer and determiner\n elif tag in [\"DT\" or \"PDT\"]:\n self.pos_tag_counts_[\"determiner\"] += 1\n elif tag == \"EX\":\n self.pos_tag_counts_[\"existential\"] += 1\n elif tag == \"FW\":\n self.pos_tag_counts_[\"non-English\"] += 1\n elif tag == \"IN\":\n self.pos_tag_counts_[\"preposition\"] += 1\n elif tag == \"POS\":\n self.pos_tag_counts_[\"possessive\"] += 1\n elif tag == \"LS\":\n self.pos_tag_counts_[\"list\"] += 1\n elif tag == \"MD\":\n self.pos_tag_counts_[\"modal\"] += 1\n elif tag in self.punct_tags:\n self.pos_tag_counts_[\"punctuation\"] += 1\n elif tag == \"TO\":\n self.pos_tag_counts_[\"infinitive\"] += 1\n elif tag == \"UH\":\n self.pos_tag_counts_[\"interjection\"] += 1\n elif tag == \"SYM\":\n self.pos_tag_counts_[\"symbol\"] += 1\n else:\n self.pos_tag_counts_[\"other\"] += 1", "response": "This function handles the Penn Treebank tags and returns a dictionary of the part - of - speech tag mapping."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalling from the fit method, this method creates the canvas and draws the part-of-speech tag mapping as a bar chart. Parameters ---------- kwargs: dict generic keyword arguments. Returns ------- ax : matplotlib axes Axes on which the PosTagVisualizer was drawn.", "response": "def draw(self, **kwargs):\n \"\"\"\n Called from the fit method, this method creates the canvas and\n draws the part-of-speech tag mapping as a bar chart.\n\n Parameters\n ----------\n kwargs: dict\n generic keyword arguments.\n\n Returns\n -------\n ax : matplotlib axes\n Axes on which the PosTagVisualizer was drawn.\n \"\"\"\n colors = resolve_colors(\n n_colors=len(self.pos_tag_counts_),\n colormap=self.colormap,\n colors=self.colors,\n )\n\n if self.frequency:\n # Sort tags with respect to frequency in corpus\n sorted_tags = sorted(\n self.pos_tag_counts_, key=self.pos_tag_counts_.get, reverse=True\n )\n sorted_counts = [self.pos_tag_counts_[tag] for tag in sorted_tags]\n\n self.ax.bar(range(len(sorted_tags)), sorted_counts, color=colors)\n return self.ax\n\n self.ax.bar(\n range(len(self.pos_tag_counts_)),\n list(self.pos_tag_counts_.values()),\n color=colors,\n )\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef finalize(self, **kwargs):\n # NOTE: not deduping here, so this is total, not unique\n self.set_title(\n \"PosTag plot for {}-token corpus\".format(\n (sum(self.pos_tag_counts_.values()))\n )\n )\n\n self.ax.set_xticks(range(len(self.pos_tag_counts_)))\n self.ax.set_xticklabels(list(self.pos_tag_counts_.keys()), rotation=90)\n\n # Set the axis labels\n if self.frequency:\n self.ax.set_xlabel(\n \"{} part-of-speech tags, sorted by frequency\".format(self.tagset_names[self.tagset])\n )\n else:\n self.ax.set_xlabel(\n \"{} part-of-speech tags\".format(self.tagset_names[self.tagset])\n )\n self.ax.set_ylabel(\"Count\")", "response": "Finalize the plot with ticks labels and title"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tsne(X, y=None, ax=None, decompose='svd', decompose_by=50, classes=None,\n colors=None, colormap=None, alpha=0.7, **kwargs):\n \"\"\"\n Display a projection of a vectorized corpus in two dimensions using TSNE,\n a nonlinear dimensionality reduction method that is particularly well\n suited to embedding in two or three dimensions for visualization as a\n scatter plot. TSNE is widely used in text analysis to show clusters or\n groups of documents or utterances and their relative proximities.\n\n Parameters\n ----------\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features representing the corpus of\n vectorized documents to visualize with tsne.\n\n y : ndarray or Series of length n\n An optional array or series of target or class values for instances.\n If this is specified, then the points will be colored according to\n their class. Often cluster labels are passed in to color the documents\n in cluster space, so this method is used both for classification and\n clustering methods.\n\n ax : matplotlib axes\n The axes to plot the figure on.\n\n decompose : string or None\n A preliminary decomposition is often used prior to TSNE to make the\n projection faster. Specify `\"svd\"` for sparse data or `\"pca\"` for\n dense data. If decompose is None, the original data set will be used.\n\n decompose_by : int\n Specify the number of components for preliminary decomposition, by\n default this is 50; the more components, the slower TSNE will be.\n\n classes : list of strings\n The names of the classes in the target, used to create a legend.\n\n colors : list or tuple of colors\n Specify the colors for each individual class\n\n colormap : string or matplotlib cmap\n Sequential colormap for continuous target\n\n alpha : float, default: 0.7\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n kwargs : dict\n Pass any additional keyword arguments to the TSNE transformer.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = TSNEVisualizer(\n ax, decompose, decompose_by, classes, colors, colormap, alpha, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "This function is used to plot a vectorized corpus using TSNE."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a pipeline that will project the data set into the internal 2D space using TSNE.", "response": "def make_transformer(self, decompose='svd', decompose_by=50, tsne_kwargs={}):\n \"\"\"\n Creates an internal transformer pipeline to project the data set into\n 2D space using TSNE, applying an pre-decomposition technique ahead of\n embedding if necessary. This method will reset the transformer on the\n class, and can be used to explore different decompositions.\n\n Parameters\n ----------\n\n decompose : string or None, default: ``'svd'``\n A preliminary decomposition is often used prior to TSNE to make\n the projection faster. Specify ``\"svd\"`` for sparse data or ``\"pca\"``\n for dense data. If decompose is None, the original data set will\n be used.\n\n decompose_by : int, default: 50\n Specify the number of components for preliminary decomposition, by\n default this is 50; the more components, the slower TSNE will be.\n\n Returns\n -------\n\n transformer : Pipeline\n Pipelined transformer for TSNE projections\n \"\"\"\n\n # TODO: detect decompose by inferring from sparse matrix or dense or\n # If number of features > 50 etc.\n decompositions = {\n 'svd': TruncatedSVD,\n 'pca': PCA,\n }\n\n if decompose and decompose.lower() not in decompositions:\n raise YellowbrickValueError(\n \"'{}' is not a valid decomposition, use {}, or None\".format(\n decompose, \", \".join(decompositions.keys())\n )\n )\n\n # Create the pipeline steps\n steps = []\n\n # Add the pre-decomposition\n if decompose:\n klass = decompositions[decompose]\n steps.append((decompose, klass(\n n_components=decompose_by, random_state=self.random_state)))\n\n # Add the TSNE manifold\n steps.append(('tsne', TSNE(\n n_components=2, random_state=self.random_state, **tsne_kwargs)))\n\n # return the pipeline\n return Pipeline(steps)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef finalize(self, **kwargs):\n self.set_title(\n \"TSNE Projection of {} Documents\".format(self.n_instances_)\n )\n\n # Remove the ticks\n self.ax.set_yticks([])\n self.ax.set_xticks([])\n\n # Add the legend outside of the figure box.\n if not all(self.classes_ == np.array([self.NULL_CLASS])):\n box = self.ax.get_position()\n self.ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])\n manual_legend(\n self, self.classes_, self.color_values_,\n loc='center left', bbox_to_anchor=(1, 0.5)\n )", "response": "Finalize the drawing by removing ticks and axes objects that do not convey information about TNSE."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads the VERSION_PATH and returns the version string.", "response": "def get_version(path=VERSION_PATH):\n \"\"\"\n Reads the python file defined in the VERSION_PATH to find the get_version\n function, and executes it to ensure that it is loaded correctly. Separating\n the version in this way ensures no additional code is executed.\n \"\"\"\n namespace = {}\n exec(read(path), namespace)\n return namespace['get_version'](short=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nyielding a generator of requirements as defined by the REQUIRE_PATH which is the path to the requirements. txt file.", "response": "def get_requires(path=REQUIRE_PATH):\n \"\"\"\n Yields a generator of requirements as defined by the REQUIRE_PATH which\n should point to a requirements.txt output by `pip freeze`.\n \"\"\"\n for line in read(path).splitlines():\n line = line.strip()\n if line and not line.startswith('#'):\n yield line"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_description_type(path=PKG_DESCRIBE):\n _, ext = os.path.splitext(path)\n return {\n \".rst\": \"text/x-rst\",\n \".txt\": \"text/plain\",\n \".md\": \"text/markdown\",\n }[ext]", "response": "Returns the long_description_content_type based on the extension of the passed in path."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndisplays a learning curve based on number of samples vs training and cross validation scores. The learning curve aims to show how a model learns and improves with experience. This helper function is a quick wrapper to utilize the LearningCurve for one-off analysis. Parameters ---------- model : a scikit-learn estimator An object that implements ``fit`` and ``predict``, can be a classifier, regressor, or clusterer so long as there is also a valid associated scoring metric. Note that the object is cloned for each validation. X : array-like, shape (n_samples, n_features) Training vector, where n_samples is the number of samples and n_features is the number of features. y : array-like, shape (n_samples) or (n_samples, n_features), optional Target relative to X for classification or regression; None for unsupervised learning. ax : matplotlib.Axes object, optional The axes object to plot the figure on. groups : array-like, with shape (n_samples,) Optional group labels for the samples used while splitting the dataset into train/test sets. train_sizes : array-like, shape (n_ticks,) default: ``np.linspace(0.1,1.0,5)`` Relative or absolute numbers of training examples that will be used to generate the learning curve. If the dtype is float, it is regarded as a fraction of the maximum size of the training set, otherwise it is interpreted as absolute sizes of the training sets. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross-validation, - integer, to specify the number of folds. - An object to be used as a cross-validation generator. - An iterable yielding train/test splits. see the scikit-learn `cross-validation guide `_ for more information on the possible strategies that can be used here. scoring : string, callable or None, optional, default: None A string or scorer callable object / function with signature ``scorer(estimator, X, y)``. See scikit-learn model evaluation documentation for names of possible metrics. exploit_incremental_learning : boolean, default: False If the estimator supports incremental learning, this will be used to speed up fitting for different training set sizes. n_jobs : integer, optional Number of jobs to run in parallel (default 1). pre_dispatch : integer or string, optional Number of predispatched jobs for parallel execution (default is all). The option can reduce the allocated memory. The string can be an expression like '2*n_jobs'. shuffle : boolean, optional Whether to shuffle training data before taking prefixes of it based on``train_sizes``. random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. Used when ``shuffle`` is True. kwargs : dict Keyword arguments that are passed to the base class and may influence the visualization as defined in other Visualizers. These arguments are also passed to the `poof()` method, e.g. can pass a path to save the figure to. Returns ------- ax : matplotlib axes Returns the axes that the learning curve were drawn on.", "response": "def learning_curve(model, X, y, ax=None, groups=None,\n train_sizes=DEFAULT_TRAIN_SIZES, cv=None, scoring=None,\n exploit_incremental_learning=False, n_jobs=1,\n pre_dispatch=\"all\", shuffle=False, random_state=None,\n **kwargs):\n \"\"\"\n Displays a learning curve based on number of samples vs training and\n cross validation scores. The learning curve aims to show how a model\n learns and improves with experience.\n\n This helper function is a quick wrapper to utilize the LearningCurve\n for one-off analysis.\n\n Parameters\n ----------\n model : a scikit-learn estimator\n An object that implements ``fit`` and ``predict``, can be a\n classifier, regressor, or clusterer so long as there is also a valid\n associated scoring metric.\n\n Note that the object is cloned for each validation.\n\n X : array-like, shape (n_samples, n_features)\n Training vector, where n_samples is the number of samples and\n n_features is the number of features.\n\n y : array-like, shape (n_samples) or (n_samples, n_features), optional\n Target relative to X for classification or regression;\n None for unsupervised learning.\n\n ax : matplotlib.Axes object, optional\n The axes object to plot the figure on.\n\n groups : array-like, with shape (n_samples,)\n Optional group labels for the samples used while splitting the dataset\n into train/test sets.\n\n train_sizes : array-like, shape (n_ticks,)\n default: ``np.linspace(0.1,1.0,5)``\n\n Relative or absolute numbers of training examples that will be used to\n generate the learning curve. If the dtype is float, it is regarded as\n a fraction of the maximum size of the training set, otherwise it is\n interpreted as absolute sizes of the training sets.\n\n cv : int, cross-validation generator or an iterable, optional\n Determines the cross-validation splitting strategy.\n Possible inputs for cv are:\n\n - None, to use the default 3-fold cross-validation,\n - integer, to specify the number of folds.\n - An object to be used as a cross-validation generator.\n - An iterable yielding train/test splits.\n\n see the scikit-learn\n `cross-validation guide `_\n for more information on the possible strategies that can be used here.\n\n scoring : string, callable or None, optional, default: None\n A string or scorer callable object / function with signature\n ``scorer(estimator, X, y)``. See scikit-learn model evaluation\n documentation for names of possible metrics.\n\n exploit_incremental_learning : boolean, default: False\n If the estimator supports incremental learning, this will be used to\n speed up fitting for different training set sizes.\n\n n_jobs : integer, optional\n Number of jobs to run in parallel (default 1).\n\n pre_dispatch : integer or string, optional\n Number of predispatched jobs for parallel execution (default is\n all). The option can reduce the allocated memory. The string can\n be an expression like '2*n_jobs'.\n\n shuffle : boolean, optional\n Whether to shuffle training data before taking prefixes of it\n based on``train_sizes``.\n\n random_state : int, RandomState instance or None, optional (default=None)\n If int, random_state is the seed used by the random number generator;\n If RandomState instance, random_state is the random number generator;\n If None, the random number generator is the RandomState instance used\n by `np.random`. Used when ``shuffle`` is True.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers. These arguments are\n also passed to the `poof()` method, e.g. can pass a path to save the\n figure to.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the learning curve were drawn on.\n \"\"\"\n # Initialize the visualizer\n oz = LearningCurve(\n model, ax=ax, groups=groups, train_sizes=train_sizes, cv=cv,\n scoring=scoring, n_jobs=n_jobs, pre_dispatch=pre_dispatch,\n shuffle=shuffle, random_state=random_state,\n exploit_incremental_learning=exploit_incremental_learning,\n )\n\n # Fit and poof the visualizer\n oz.fit(X, y)\n oz.poof(**kwargs)\n return oz.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit(self, X, y=None):\n # arguments to pass to sk_learning_curve\n sklc_kwargs = {\n key: self.get_params()[key]\n for key in (\n 'groups', 'train_sizes', 'cv', 'scoring',\n 'exploit_incremental_learning', 'n_jobs',\n 'pre_dispatch', 'shuffle', 'random_state',\n )\n }\n\n # compute the learning curve and store the scores on the estimator\n curve = sk_learning_curve(self.estimator, X, y, **sklc_kwargs)\n self.train_sizes_, self.train_scores_, self.test_scores_ = curve\n\n # compute the mean and standard deviation of the training data\n self.train_scores_mean_ = np.mean(self.train_scores_, axis=1)\n self.train_scores_std_ = np.std(self.train_scores_, axis=1)\n\n # compute the mean and standard deviation of the test data\n self.test_scores_mean_ = np.mean(self.test_scores_, axis=1)\n self.test_scores_std_ = np.std(self.test_scores_, axis=1)\n\n # draw the curves on the current axes\n self.draw()\n return self", "response": "Fits the learning curve with the wrapped model to the specified data and saves the scores on the estimator."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw(self, **kwargs):\n # Specify the curves to draw and their labels\n labels = (\"Training Score\", \"Cross Validation Score\")\n curves = (\n (self.train_scores_mean_, self.train_scores_std_),\n (self.test_scores_mean_, self.test_scores_std_),\n )\n\n # Get the colors for the train and test curves\n colors = resolve_colors(n_colors=2)\n\n # Plot the fill betweens first so they are behind the curves.\n for idx, (mean, std) in enumerate(curves):\n # Plot one standard deviation above and below the mean\n self.ax.fill_between(\n self.train_sizes_, mean - std, mean+std, alpha=0.25,\n color=colors[idx],\n )\n\n # Plot the mean curves so they are in front of the variance fill\n for idx, (mean, _) in enumerate(curves):\n self.ax.plot(\n self.train_sizes_, mean, 'o-', color=colors[idx],\n label=labels[idx],\n )\n\n return self.ax", "response": "Renders the training and test learning curves."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisplaying each feature as a vertical axis and each instance as a line. This helper function is a quick wrapper to utilize the ParallelCoordinates Visualizer (Transformer) for one-off analysis. Parameters ---------- X : ndarray or DataFrame of shape n x m A matrix of n instances with m features y : ndarray or Series of length n An array or series of target or class values ax : matplotlib Axes, default: None The axis to plot the figure on. If None is passed in the current axes will be used (or generated if required). features : list, default: None a list of feature names to use If a DataFrame is passed to fit and features is None, feature names are selected as the columns of the DataFrame. classes : list, default: None a list of class names for the legend If classes is None and a y value is passed to fit then the classes are selected from the target vector. normalize : string or None, default: None specifies which normalization method to use, if any Current supported options are 'minmax', 'maxabs', 'standard', 'l1', and 'l2'. sample : float or int, default: 1.0 specifies how many examples to display from the data If int, specifies the maximum number of samples to display. If float, specifies a fraction between 0 and 1 to display. color : list or tuple, default: None optional list or tuple of colors to colorize lines Use either color to colorize the lines on a per class basis or colormap to color them on a continuous scale. colormap : string or cmap, default: None optional string or matplotlib cmap to colorize lines Use either color to colorize the lines on a per class basis or colormap to color them on a continuous scale. alpha : float, default: None Specify a transparency where 1 is completely opaque and 0 is completely transparent. This property makes densely clustered lines more visible. If None, the alpha is set to 0.5 in \"fast\" mode and 0.25 otherwise. fast : bool, default: False Fast mode improves the performance of the drawing time of parallel coordinates but produces an image that does not show the overlap of instances in the same class. Fast mode should be used when drawing all instances is too burdensome and sampling is not an option. vlines : boolean, default: True flag to determine vertical line display vlines_kwds : dict, default: None options to style or display the vertical lines, default: None kwargs : dict Keyword arguments that are passed to the base class and may influence the visualization as defined in other Visualizers. Returns ------- ax : matplotlib axes Returns the axes that the parallel coordinates were drawn on.", "response": "def parallel_coordinates(X, y, ax=None, features=None, classes=None,\n normalize=None, sample=1.0, color=None, colormap=None,\n alpha=None, fast=False, vlines=True, vlines_kwds=None,\n **kwargs):\n \"\"\"Displays each feature as a vertical axis and each instance as a line.\n\n This helper function is a quick wrapper to utilize the ParallelCoordinates\n Visualizer (Transformer) for one-off analysis.\n\n Parameters\n ----------\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n ax : matplotlib Axes, default: None\n The axis to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n features : list, default: None\n a list of feature names to use\n If a DataFrame is passed to fit and features is None, feature\n names are selected as the columns of the DataFrame.\n\n classes : list, default: None\n a list of class names for the legend\n If classes is None and a y value is passed to fit then the classes\n are selected from the target vector.\n\n normalize : string or None, default: None\n specifies which normalization method to use, if any\n Current supported options are 'minmax', 'maxabs', 'standard', 'l1',\n and 'l2'.\n\n sample : float or int, default: 1.0\n specifies how many examples to display from the data\n If int, specifies the maximum number of samples to display.\n If float, specifies a fraction between 0 and 1 to display.\n\n color : list or tuple, default: None\n optional list or tuple of colors to colorize lines\n Use either color to colorize the lines on a per class basis or\n colormap to color them on a continuous scale.\n\n colormap : string or cmap, default: None\n optional string or matplotlib cmap to colorize lines\n Use either color to colorize the lines on a per class basis or\n colormap to color them on a continuous scale.\n\n alpha : float, default: None\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered lines more visible.\n If None, the alpha is set to 0.5 in \"fast\" mode and 0.25 otherwise.\n\n fast : bool, default: False\n Fast mode improves the performance of the drawing time of parallel\n coordinates but produces an image that does not show the overlap of\n instances in the same class. Fast mode should be used when drawing all\n instances is too burdensome and sampling is not an option.\n\n vlines : boolean, default: True\n flag to determine vertical line display\n\n vlines_kwds : dict, default: None\n options to style or display the vertical lines, default: None\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = ParallelCoordinates(\n ax, features, classes, normalize, sample, color, colormap, alpha,\n fast, vlines, vlines_kwds, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfit the transformer to the target class and return the instance of the visualization.", "response": "def fit(self, X, y=None, **kwargs):\n \"\"\"\n The fit method is the primary drawing input for the\n visualization since it has both the X and y data required for the\n viz and the transform method does not.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n kwargs : dict\n Pass generic arguments to the drawing method\n\n Returns\n -------\n self : instance\n Returns the instance of the transformer/visualizer\n \"\"\"\n\n # Convert from pandas data types\n if is_dataframe(X):\n # Get column names before reverting to an np.ndarray\n if self.features_ is None:\n self.features_ = np.array(X.columns)\n\n X = X.values\n if is_series(y):\n y = y.values\n\n # Assign integer labels to the feature columns from the input\n if self.features_ is None:\n self.features_ = np.arange(0, X.shape[1])\n\n # Ensure that all classes are represented in the color mapping (before sample)\n # NOTE: np.unique also specifies the ordering of the classes\n if self.classes_ is None:\n self.classes_ = [str(label) for label in np.unique(y)]\n\n # Create the color mapping for each class\n # TODO: Allow both colormap, listed colors, and palette definition\n # TODO: Make this an independent function or property for override!\n color_values = resolve_colors(\n n_colors=len(self.classes_), colormap=self.colormap, colors=self.color\n )\n self._colors = dict(zip(self.classes_, color_values))\n\n # Ticks for each feature specified\n self._increments = np.arange(len(self.features_))\n\n # Subsample instances\n X, y = self._subsample(X, y)\n\n # Normalize instances\n if self.normalize is not None:\n X = self.NORMALIZERS[self.normalize].fit_transform(X)\n\n # the super method calls draw and returns self\n return super(ParallelCoordinates, self).fit(X, y, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncall from the fit method, this method creates the parallel coordinates canvas and draws each instance and vertical lines on it. Parameters ---------- X : ndarray of shape n x m A matrix of n instances with m features y : ndarray of length n An array or series of target or class values kwargs : dict Pass generic arguments to the drawing method", "response": "def draw(self, X, y, **kwargs):\n \"\"\"\n Called from the fit method, this method creates the parallel\n coordinates canvas and draws each instance and vertical lines on it.\n\n Parameters\n ----------\n X : ndarray of shape n x m\n A matrix of n instances with m features\n\n y : ndarray of length n\n An array or series of target or class values\n\n kwargs : dict\n Pass generic arguments to the drawing method\n\n \"\"\"\n if self.fast:\n return self.draw_classes(X, y, **kwargs)\n return self.draw_instances(X, y, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndraw the instances colored by the target y.", "response": "def draw_instances(self, X, y, **kwargs):\n \"\"\"\n Draw the instances colored by the target y such that each line is a\n single instance. This is the \"slow\" mode of drawing, since each\n instance has to be drawn individually. However, in so doing, the\n density of instances in braids is more apparent since lines have an\n independent alpha that is compounded in the figure.\n\n This is the default method of drawing.\n\n Parameters\n ----------\n X : ndarray of shape n x m\n A matrix of n instances with m features\n\n y : ndarray of length n\n An array or series of target or class values\n\n Notes\n -----\n This method can be used to draw additional instances onto the parallel\n coordinates before the figure is finalized.\n \"\"\"\n # Get alpha from param or default\n alpha = self.alpha or 0.25\n\n for idx in range(len(X)):\n Xi = X[idx]\n yi = y[idx]\n\n # TODO: generalize this duplicated code into a single function\n if isinstance(yi, str):\n label = yi\n else:\n # TODO: what happens if yi is not in classes?!\n label = self.classes_[yi]\n\n self.ax.plot(\n self._increments, Xi,\n color=self._colors[label], alpha=alpha, **kwargs\n )\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef draw_classes(self, X, y, **kwargs):\n # Get alpha from param or default\n alpha = self.alpha or 0.5\n\n # Prepare to flatten data within each class:\n # introduce separation between individual data points using None in\n # x-values and arbitrary value (one) in y-values\n X_separated = np.hstack([X, np.ones((X.shape[0], 1))])\n increments_separated = self._increments.tolist()\n increments_separated.append(None)\n\n # Get the classes that exist in the dataset, y\n y_values = np.unique(y)\n\n # Plot each class as a single line plot\n for yi in y_values:\n if isinstance(yi, str):\n label = yi\n else:\n # TODO: what happens if yi is not in classes?!\n label = self.classes_[yi]\n\n X_in_class = X_separated[y == yi, :]\n increments_in_class = increments_separated * len(X_in_class)\n if len(X_in_class) > 0:\n self.ax.plot(\n increments_in_class, X_in_class.flatten(), linewidth=1,\n color=self._colors[label], alpha=alpha, **kwargs\n )\n\n return self.ax", "response": "Draw the instances colored by the target y such that each line is a single class."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinalizing executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n\n \"\"\"\n # Set the title\n self.set_title(\n 'Parallel Coordinates for {} Features'.format(len(self.features_))\n )\n\n # Add the vertical lines\n # TODO: Make an independent function for override!\n if self.show_vlines:\n for idx in self._increments:\n self.ax.axvline(idx, **self.vlines_kwds)\n\n # Set the limits\n self.ax.set_xticks(self._increments)\n self.ax.set_xticklabels(self.features_)\n self.ax.set_xlim(self._increments[0], self._increments[-1])\n\n # Add the legend sorting classes by name\n labels = sorted(list(self._colors.keys()))\n colors = [self._colors[lbl] for lbl in labels]\n manual_legend(self, labels, colors, loc='best', frameon=True)\n\n # Add the grid view\n self.ax.grid()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef validation_curve(model, X, y, param_name, param_range, ax=None, logx=False,\n groups=None, cv=None, scoring=None, n_jobs=1,\n pre_dispatch=\"all\", **kwargs):\n \"\"\"\n Displays a validation curve for the specified param and values, plotting\n both the train and cross-validated test scores. The validation curve is a\n visual, single-parameter grid search used to tune a model to find the best\n balance between error due to bias and error due to variance.\n\n This helper function is a wrapper to use the ValidationCurve in a fast,\n visual analysis.\n\n Parameters\n ----------\n model : a scikit-learn estimator\n An object that implements ``fit`` and ``predict``, can be a\n classifier, regressor, or clusterer so long as there is also a valid\n associated scoring metric.\n\n Note that the object is cloned for each validation.\n\n X : array-like, shape (n_samples, n_features)\n Training vector, where n_samples is the number of samples and\n n_features is the number of features.\n\n y : array-like, shape (n_samples) or (n_samples, n_features), optional\n Target relative to X for classification or regression;\n None for unsupervised learning.\n\n param_name : string\n Name of the parameter that will be varied.\n\n param_range : array-like, shape (n_values,)\n The values of the parameter that will be evaluated.\n\n ax : matplotlib.Axes object, optional\n The axes object to plot the figure on.\n\n logx : boolean, optional\n If True, plots the x-axis with a logarithmic scale.\n\n groups : array-like, with shape (n_samples,)\n Optional group labels for the samples used while splitting the dataset\n into train/test sets.\n\n cv : int, cross-validation generator or an iterable, optional\n Determines the cross-validation splitting strategy.\n Possible inputs for cv are:\n\n - None, to use the default 3-fold cross-validation,\n - integer, to specify the number of folds.\n - An object to be used as a cross-validation generator.\n - An iterable yielding train/test splits.\n\n see the scikit-learn\n `cross-validation guide `_\n for more information on the possible strategies that can be used here.\n\n scoring : string, callable or None, optional, default: None\n A string or scorer callable object / function with signature\n ``scorer(estimator, X, y)``. See scikit-learn model evaluation\n documentation for names of possible metrics.\n\n n_jobs : integer, optional\n Number of jobs to run in parallel (default 1).\n\n pre_dispatch : integer or string, optional\n Number of predispatched jobs for parallel execution (default is\n all). The option can reduce the allocated memory. The string can\n be an expression like '2*n_jobs'.\n\n kwargs : dict\n Keyword arguments that are passed to the base class and may influence\n the visualization as defined in other Visualizers. These arguments are\n also passed to the `poof()` method, e.g. can pass a path to save the\n figure to.\n\n Returns\n -------\n ax : matplotlib.Axes\n The axes object that the validation curves were drawn on.\n \"\"\"\n\n # Initialize the visualizer\n oz = ValidationCurve(\n model, param_name, param_range, ax=ax, logx=logx, groups=groups,\n cv=cv, scoring=scoring, n_jobs=n_jobs, pre_dispatch=pre_dispatch\n )\n\n # Fit and poof the visualizer\n oz.fit(X, y)\n oz.poof(**kwargs)\n return oz.ax", "response": "This function is used to plot a validation curve for the specified parameter and values."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X, y=None):\n # arguments to pass to sk_validation_curve\n skvc_kwargs = {\n key: self.get_params()[key]\n for key in (\n 'param_name', 'param_range', 'groups', 'cv', 'scoring',\n 'n_jobs', 'pre_dispatch',\n )\n }\n\n # compute the validation curve and store scores\n curve = sk_validation_curve(self.estimator, X, y, **skvc_kwargs)\n self.train_scores_, self.test_scores_ = curve\n\n # compute the mean and standard deviation of the training data\n self.train_scores_mean_ = np.mean(self.train_scores_, axis=1)\n self.train_scores_std_ = np.std(self.train_scores_, axis=1)\n\n # compute the mean and standard deviation of the test data\n self.test_scores_mean_ = np.mean(self.test_scores_, axis=1)\n self.test_scores_std_ = np.std(self.test_scores_, axis=1)\n\n # draw the curves on the current axes\n self.draw()\n return self", "response": "Fits the validation curve with the wrapped estimator and parameter\n array to the specified data. Draws training and test score curves and saves the scores to the visualizer."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, X, y=None, **kwargs):\n # Fit the inner estimator\n self.estimator.fit(X, y)\n\n # Extract the classes from the estimator\n if self.classes_ is None:\n self.classes_ = self.estimator.classes_\n\n # Always return self from fit\n return self", "response": "Fit the inner estimator to the target and class values of the target or class and return the instance of the classification score visualizer."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _load_dataset(name, data_home=None, return_dataset=False):\n info = DATASETS[name]\n data = Dataset(name, data_home=data_home, **info)\n if return_dataset:\n return data\n return data.to_data()", "response": "Load a dataset by name and return specified format."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _load_corpus(name, data_home=None):\n info = DATASETS[name]\n return Corpus(name, data_home=data_home, **info)", "response": "Load a corpus object by name."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate a list of colors based on common color arguments for the given colormap and colors.", "response": "def resolve_colors(n_colors=None, colormap=None, colors=None):\n \"\"\"\n Generates a list of colors based on common color arguments, for example\n the name of a colormap or palette or another iterable of colors. The list\n is then truncated (or multiplied) to the specific number of requested\n colors.\n\n Parameters\n ----------\n n_colors : int, default: None\n Specify the length of the list of returned colors, which will either\n truncate or multiple the colors available. If None the length of the\n colors will not be modified.\n\n colormap : str, default: None\n The name of the matplotlib color map with which to generate colors.\n\n colors : iterable, default: None\n A collection of colors to use specifically with the plot.\n\n Returns\n -------\n colors : list\n A list of colors that can be used in matplotlib plots.\n\n Notes\n -----\n This function was originally based on a similar function in the pandas\n plotting library that has been removed in the new version of the library.\n \"\"\"\n\n # Work with the colormap if specified and colors is not\n if colormap is not None and colors is None:\n if isinstance(colormap, str):\n try:\n colormap = cm.get_cmap(colormap)\n except ValueError as e:\n raise YellowbrickValueError(e)\n\n\n n_colors = n_colors or len(get_color_cycle())\n _colors = list(map(colormap, np.linspace(0, 1, num=n_colors)))\n\n # Work with the color list\n elif colors is not None:\n\n # Warn if both colormap and colors is specified.\n if colormap is not None:\n warnings.warn(\n \"both colormap and colors specified; using colors\"\n )\n\n _colors = list(colors) # Ensure colors is a list\n\n # Get the default colors\n else:\n _colors = get_color_cycle()\n\n # Truncate or multiple the color list according to the number of colors\n if n_colors is not None and len(_colors) != n_colors:\n _colors = [\n _colors[idx % len(_colors)] for idx in np.arange(n_colors)\n ]\n\n return _colors"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef colors(self, value):\n if isinstance(value, str):\n # Must import here to avoid recursive import\n from .palettes import PALETTES\n\n if value not in PALETTES:\n raise YellowbrickValueError(\n \"'{}' is not a registered color palette\".format(value)\n )\n self._colors = copy(PALETTES[value])\n elif isinstance(value, list):\n self._colors = value\n else:\n self._colors = list(value)", "response": "Converts color strings into a color listing."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a property attribute for new - style classes that only calls its getter on the first access.", "response": "def memoized(fget):\n \"\"\"\n Return a property attribute for new-style classes that only calls its\n getter on the first access. The result is stored and on subsequent\n accesses is returned, preventing the need to call the getter any more.\n\n Parameters\n ----------\n fget: function\n The getter method to memoize for subsequent access.\n\n See also\n --------\n python-memoized-property\n `python-memoized-property `_\n \"\"\"\n attr_name = '_{0}'.format(fget.__name__)\n\n @wraps(fget)\n def fget_memoized(self):\n if not hasattr(self, attr_name):\n setattr(self, attr_name, fget(self))\n return getattr(self, attr_name)\n\n return property(fget_memoized)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef anscombe():\n _, ((axa, axb), (axc, axd)) = plt.subplots(2, 2, sharex='col', sharey='row')\n colors = get_color_cycle()\n\n for arr, ax, color in zip(ANSCOMBE, (axa, axb, axc, axd), colors):\n x = arr[0]\n y = arr[1]\n\n # Set the X and Y limits\n ax.set_xlim(0, 15)\n ax.set_ylim(0, 15)\n\n # Draw the points in the scatter plot\n ax.scatter(x, y, c=color)\n\n # Draw the linear best fit line on the plot\n draw_best_fit(x, y, ax, c=color)\n\n return (axa, axb, axc, axd)", "response": "Creates 2x2 grid plot of the 4 anscombe datasets for illustration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndisplay each feature as an axis around a circle surrounding a scatter plot whose points are each individual instance. This helper function is a quick wrapper to utilize the RadialVisualizer (Transformer) for one-off analysis. Parameters ---------- X : ndarray or DataFrame of shape n x m A matrix of n instances with m features y : ndarray or Series of length n An array or series of target or class values ax : matplotlib Axes, default: None The axes to plot the figure on. features : list of strings, default: None The names of the features or columns classes : list of strings, default: None The names of the classes in the target color : list or tuple of colors, default: None Specify the colors for each individual class colormap : string or matplotlib cmap, default: None Sequential colormap for continuous target alpha : float, default: 1.0 Specify a transparency where 1 is completely opaque and 0 is completely transparent. This property makes densely clustered points more visible. Returns ------- ax : matplotlib axes Returns the axes that the parallel coordinates were drawn on.", "response": "def radviz(X, y=None, ax=None, features=None, classes=None,\n color=None, colormap=None, alpha=1.0, **kwargs):\n \"\"\"\n Displays each feature as an axis around a circle surrounding a scatter\n plot whose points are each individual instance.\n\n This helper function is a quick wrapper to utilize the RadialVisualizer\n (Transformer) for one-off analysis.\n\n Parameters\n ----------\n\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n ax : matplotlib Axes, default: None\n The axes to plot the figure on.\n\n features : list of strings, default: None\n The names of the features or columns\n\n classes : list of strings, default: None\n The names of the classes in the target\n\n color : list or tuple of colors, default: None\n Specify the colors for each individual class\n\n colormap : string or matplotlib cmap, default: None\n Sequential colormap for continuous target\n\n alpha : float, default: 1.0\n Specify a transparency where 1 is completely opaque and 0 is completely\n transparent. This property makes densely clustered points more visible.\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the parallel coordinates were drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = RadialVisualizer(\n ax, features, classes, color, colormap, alpha, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nnormalize a matrix X to fit a matrix in the space [ 0 1 ) by column.", "response": "def normalize(X):\n \"\"\"\n MinMax normalization to fit a matrix in the space [0,1] by column.\n \"\"\"\n a = X.min(axis=0)\n b = X.max(axis=0)\n return (X - a[np.newaxis, :]) / ((b - a)[np.newaxis, :])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating the radviz canvas and draws each instance of the class or target colored point at each location in the class or target data set.", "response": "def draw(self, X, y, **kwargs):\n \"\"\"\n Called from the fit method, this method creates the radviz canvas and\n draws each instance as a class or target colored point, whose location\n is determined by the feature data set.\n \"\"\"\n # Convert from dataframe\n if is_dataframe(X):\n X = X.values\n\n # Clean out nans and warn that the user they aren't plotted\n nan_warnings.warn_if_nans_exist(X)\n X, y = nan_warnings.filter_missing(X, y)\n\n # Get the shape of the data\n nrows, ncols = X.shape\n\n # Set the axes limits\n self.ax.set_xlim([-1,1])\n self.ax.set_ylim([-1,1])\n\n # Create the colors\n # TODO: Allow both colormap, listed colors, and palette definition\n # TODO: Make this an independent function or property for override!\n color_values = resolve_colors(\n n_colors=len(self.classes_), colormap=self.colormap, colors=self.color\n )\n self._colors = dict(zip(self.classes_, color_values))\n\n # Create a data structure to hold scatter plot representations\n to_plot = {}\n for kls in self.classes_:\n to_plot[kls] = [[], []]\n\n # Compute the arcs around the circumference for each feature axis\n # TODO: make this an independent function for override\n s = np.array([\n (np.cos(t), np.sin(t))\n for t in [\n 2.0 * np.pi * (i / float(ncols))\n for i in range(ncols)\n ]\n ])\n\n # Compute the locations of the scatter plot for each class\n # Normalize the data first to plot along the 0, 1 axis\n for i, row in enumerate(self.normalize(X)):\n row_ = np.repeat(np.expand_dims(row, axis=1), 2, axis=1)\n xy = (s * row_).sum(axis=0) / row.sum()\n kls = self.classes_[y[i]]\n\n to_plot[kls][0].append(xy[0])\n to_plot[kls][1].append(xy[1])\n\n # Add the scatter plots from the to_plot function\n # TODO: store these plots to add more instances to later\n # TODO: make this a separate function\n for i, kls in enumerate(self.classes_):\n self.ax.scatter(\n to_plot[kls][0], to_plot[kls][1], color=self._colors[kls],\n label=str(kls), alpha=self.alpha, **kwargs\n )\n\n # Add the circular axis path\n # TODO: Make this a seperate function (along with labeling)\n self.ax.add_patch(patches.Circle(\n (0.0, 0.0), radius=1.0, facecolor='none', edgecolor='grey', linewidth=.5\n ))\n\n # Add the feature names\n for xy, name in zip(s, self.features_):\n # Add the patch indicating the location of the axis\n self.ax.add_patch(patches.Circle(xy, radius=0.025, facecolor='#777777'))\n\n # Add the feature names offset around the axis marker\n if xy[0] < 0.0 and xy[1] < 0.0:\n self.ax.text(xy[0] - 0.025, xy[1] - 0.025, name, ha='right', va='top', size='small')\n elif xy[0] < 0.0 and xy[1] >= 0.0:\n self.ax.text(xy[0] - 0.025, xy[1] + 0.025, name, ha='right', va='bottom', size='small')\n elif xy[0] >= 0.0 and xy[1] < 0.0:\n self.ax.text(xy[0] + 0.025, xy[1] - 0.025, name, ha='left', va='top', size='small')\n elif xy[0] >= 0.0 and xy[1] >= 0.0:\n self.ax.text(xy[0] + 0.025, xy[1] + 0.025, name, ha='left', va='bottom', size='small')\n\n self.ax.axis('equal')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfinalize executes any subclass - specific axes finalization steps.", "response": "def finalize(self, **kwargs):\n \"\"\"\n Finalize executes any subclass-specific axes finalization steps.\n The user calls poof and poof calls finalize.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n\n \"\"\"\n # Set the title\n self.set_title(\n 'RadViz for {} Features'.format(len(self.features_))\n )\n\n # Remove the ticks from the graph\n self.ax.set_yticks([])\n self.ax.set_xticks([])\n\n # Add the legend\n colors = [self._colors[c] for c in self.classes_]\n manual_legend(self, self.classes_, colors, loc='best')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the index of the value at the Qth percentile in array a.", "response": "def percentile_index(a, q):\n \"\"\"\n Returns the index of the value at the Qth percentile in array a.\n \"\"\"\n return np.where(\n a==np.percentile(a, q, interpolation='nearest')\n )[0][0]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nraise an exception if s is not in valid.", "response": "def validate_string_param(s, valid, param_name=\"param\"):\n \"\"\"\n Raises a well formatted exception if s is not in valid, otherwise does not\n raise an exception. Uses ``param_name`` to identify the parameter.\n \"\"\"\n if s.lower() not in valid:\n raise YellowbrickValueError(\n \"unknown {} '{}', chose from '{}'\".format(\n param_name, s, \", \".join(valid)\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef intercluster_distance(model, X, y=None, ax=None,\n min_size=400, max_size=25000,\n embedding='mds', scoring='membership',\n legend=True, legend_loc=\"lower left\", legend_size=1.5,\n random_state=None, **kwargs):\n \"\"\"Quick Method:\n\n Intercluster distance maps display an embedding of the cluster centers in\n 2 dimensions with the distance to other centers preserved. E.g. the closer\n to centers are in the visualization, the closer they are in the original\n feature space. The clusters are sized according to a scoring metric. By\n default, they are sized by membership, e.g. the number of instances that\n belong to each center. This gives a sense of the relative importance of\n clusters. Note however, that because two clusters overlap in the 2D space,\n it does not imply that they overlap in the original feature space.\n\n Parameters\n ----------\n model : a Scikit-Learn clusterer\n Should be an instance of a centroidal clustering algorithm (or a\n hierarchical algorithm with a specified number of clusters). Also\n accepts some other models like LDA for text clustering.\n If it is not a clusterer, an exception is raised.\n\n X : array-like of shape (n, m)\n A matrix or data frame with n instances and m features\n\n y : array-like of shape (n,), optional\n A vector or series representing the target for each instance\n\n ax : matplotlib Axes, default: None\n The axes to plot the figure on. If None is passed in the current axes\n will be used (or generated if required).\n\n min_size : int, default: 400\n The size, in points, of the smallest cluster drawn on the graph.\n Cluster sizes will be scaled between the min and max sizes.\n\n max_size : int, default: 25000\n The size, in points, of the largest cluster drawn on the graph.\n Cluster sizes will be scaled between the min and max sizes.\n\n embedding : default: 'mds'\n The algorithm used to embed the cluster centers in 2 dimensional space\n so that the distance between clusters is represented equivalently to\n their relationship in feature spaceself.\n Embedding algorithm options include:\n\n - **mds**: multidimensional scaling\n - **tsne**: stochastic neighbor embedding\n\n scoring : default: 'membership'\n The scoring method used to determine the size of the clusters drawn on\n the graph so that the relative importance of clusters can be viewed.\n Scoring method options include:\n\n - **membership**: number of instances belonging to each cluster\n\n legend : bool, default: True\n Whether or not to draw the size legend onto the graph, omit the legend\n to more easily see clusters that overlap.\n\n legend_loc : str, default: \"lower left\"\n The location of the legend on the graph, used to move the legend out\n of the way of clusters into open space. The same legend location\n options for matplotlib are used here.\n\n .. seealso:: https://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.legend\n\n legend_size : float, default: 1.5\n The size, in inches, of the size legend to inset into the graph.\n\n random_state : int or RandomState, default: None\n Fixes the random state for stochastic embedding algorithms.\n\n kwargs : dict\n Keyword arguments passed to the base class and may influence the\n feature visualization properties.\n\n Returns\n -------\n viz : InterclusterDistance\n The intercluster distance visualizer, fitted and finalized.\n \"\"\"\n oz = InterclusterDistance(\n model, ax=ax, min_size=min_size, max_size=max_size, embedding=embedding,\n scoring=scoring, legend=legend, legend_loc=legend_loc, legend_size=legend_size,\n random_state=random_state, **kwargs\n )\n\n oz.fit(X, y)\n oz.poof()\n return oz", "response": "Quick Method that displays an intercluster distance map for a set of clusters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lax(self):\n if inset_locator is None:\n raise YellowbrickValueError((\n \"intercluster distance map legend requires matplotlib 2.0.2 or greater \"\n \"please upgrade matplotlib or set legend=False on the visualizer\"\n ))\n\n lax = inset_locator.inset_axes(\n self.ax, width=self.legend_size, height=self.legend_size, loc=self.legend_loc\n )\n\n lax.set_frame_on(False)\n lax.set_facecolor(\"none\")\n lax.grid(False)\n lax.set_xlim(-1.4,1.4)\n lax.set_ylim(-1.4,1.4)\n lax.set_xticks([])\n lax.set_yticks([])\n\n for name in lax.spines:\n lax.spines[name].set_visible(False)\n\n return lax", "response": "Returns the legend axes creating it only on demand by creating a 2. 0. 2 inset axes that has no grid ticks spines and face frame."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the internal transformer that maps the cluster center s high dimensional space to its two dimensional space.", "response": "def transformer(self):\n \"\"\"\n Creates the internal transformer that maps the cluster center's high\n dimensional space to its two dimensional space.\n \"\"\"\n ttype = self.embedding.lower() # transformer method type\n\n if ttype == 'mds':\n return MDS(n_components=2, random_state=self.random_state)\n\n if ttype == 'tsne':\n return TSNE(n_components=2, random_state=self.random_state)\n\n raise YellowbrickValueError(\"unknown embedding '{}'\".format(ttype))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch for or creates cluster centers for the specified clustering algorithm.", "response": "def cluster_centers_(self):\n \"\"\"\n Searches for or creates cluster centers for the specified clustering\n algorithm. This algorithm ensures that that the centers are\n appropriately drawn and scaled so that distance between clusters are\n maintained.\n \"\"\"\n # TODO: Handle agglomerative clustering and LDA\n for attr in ('cluster_centers_',):\n try:\n return getattr(self.estimator, attr)\n except AttributeError:\n continue\n\n raise AttributeError(\n \"could not find or make cluster_centers_ for {}\".format(\n self.estimator.__class__.__name__\n ))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfit the clustering model computing the centers then embeds the centers into 2D space using the embedding method specified.", "response": "def fit(self, X, y=None):\n \"\"\"\n Fit the clustering model, computing the centers then embeds the centers\n into 2D space using the embedding method specified.\n \"\"\"\n with Timer() as self.fit_time_:\n # Fit the underlying estimator\n self.estimator.fit(X, y)\n\n # Get the centers\n # TODO: is this how sklearn stores all centers in the model?\n C = self.cluster_centers_\n\n # Embed the centers in 2D space and get the cluster scores\n self.embedded_centers_ = self.transformer.fit_transform(C)\n self.scores_ = self._score_clusters(X, y)\n\n # Draw the clusters and fit returns self\n self.draw()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef draw(self):\n # Compute the sizes of the markers from their score\n sizes = self._get_cluster_sizes()\n\n # Draw the scatter plots with associated sizes on the graph\n self.ax.scatter(\n self.embedded_centers_[:,0], self.embedded_centers_[:,1],\n s=sizes, c=self.facecolor, edgecolor=self.edgecolor, linewidth=1,\n )\n\n # Annotate the clusters with their labels\n for i, pt in enumerate(self.embedded_centers_):\n self.ax.text(\n s=str(i), x=pt[0], y=pt[1], va=\"center\", ha=\"center\", fontweight=\"bold\"\n )\n\n # Ensure the current axes is always the main residuals axes\n plt.sca(self.ax)\n return self.ax", "response": "Draw the embedded centers with their sizes on the visualization."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinalizing the visualization to create an origin grid feel instead of the default matplotlib feel. Set the title remove spines label the label of the grid with components and add a legend from the size if required.", "response": "def finalize(self):\n \"\"\"\n Finalize the visualization to create an \"origin grid\" feel instead of\n the default matplotlib feel. Set the title, remove spines, and label\n the grid with components. This function also adds a legend from the\n sizes if required.\n \"\"\"\n # Set the default title if a user hasn't supplied one\n self.set_title(\"{} Intercluster Distance Map (via {})\".format(\n self.estimator.__class__.__name__, self.embedding.upper()\n ))\n\n # Create the origin grid and minimalist display\n self.ax.set_xticks([0])\n self.ax.set_yticks([0])\n self.ax.set_xticklabels([])\n self.ax.set_yticklabels([])\n self.ax.set_xlabel(\"PC2\")\n self.ax.set_ylabel(\"PC1\")\n\n # Make the legend by creating an inset axes that shows relative sizing\n # based on the scoring metric supplied by the user.\n if self.legend:\n self._make_size_legend()\n\n return self.ax"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _score_clusters(self, X, y=None):\n stype = self.scoring.lower() # scoring method name\n\n if stype == \"membership\":\n return np.bincount(self.estimator.labels_)\n\n raise YellowbrickValueError(\"unknown scoring method '{}'\".format(stype))", "response": "Determines the scores of the clusters in X."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_cluster_sizes(self):\n # NOTE: log and power are hardcoded, should we allow the user to specify?\n return prop_to_size(\n self.scores_, mi=self.min_size, ma=self.max_size, log=False, power=0.5\n )", "response": "Returns the marker sizes based on the scores."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _make_size_legend(self):\n # Compute the size of the markers and scale them to our figure size\n # NOTE: the marker size is the area of the plot, we need to compute the\n # radius of the markers.\n areas = self._get_cluster_sizes()\n radii = np.sqrt(areas / np.pi)\n scaled = np.interp(radii, (radii.min(), radii.max()), (.1, 1))\n\n # Compute the locations of the 25th, 50th, and 75th percentile scores\n indices = np.array([\n percentile_index(self.scores_, p) for p in (25, 50, 75)\n ])\n\n # Draw size circles annotated with the percentile score as the legend.\n for idx in indices:\n # TODO: should the size circle's center be hard coded like this?\n center = (-0.30, 1-scaled[idx])\n c = Circle(\n center, scaled[idx], facecolor=\"none\", edgecolor=\"#2e7193\",\n linewidth=1.5, linestyle=\"--\"\n )\n self.lax.add_patch(c)\n\n # Add annotation to the size circle with the value of the score\n self.lax.annotate(\n self.scores_[idx], (-0.30, 1-(2*scaled[idx])), xytext=(1, 1-(2*scaled[idx])),\n arrowprops=dict(arrowstyle=\"wedge\", color=\"#2e7193\"), va='center', ha='center',\n )\n\n # Draw size legend title\n self.lax.text(s=\"membership\", x=0, y=1.2, va='center', ha='center')\n\n # Ensure the current axes is always the main axes after modifying the\n # inset axes and while drawing.\n plt.sca(self.ax)", "response": "Draw a legend that shows relative sizes of the clusters at the 25th 50th and 75th percentile based on the current scoring metric."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef manual_legend(g, labels, colors, **legend_kwargs):\n # Get access to the matplotlib Axes\n if isinstance(g, Visualizer):\n g = g.ax\n elif g is None:\n g = plt.gca()\n\n # Ensure that labels and colors are the same length to prevent odd behavior.\n if len(colors) != len(labels):\n raise YellowbrickValueError(\n \"please specify the same number of colors as labels!\"\n )\n\n # Create the legend handles with the associated colors and labels\n handles = [\n patches.Patch(color=color, label=label)\n for color, label in zip(colors, labels)\n ]\n\n # Return the Legend artist\n return g.legend(handles=handles, **legend_kwargs)", "response": "This function adds a manual legend to the current axes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef freqdist(X, y=None, ax=None, color=None, N=50, **kwargs):\n # Instantiate the visualizer\n visualizer = FreqDistVisualizer(\n ax, X, color, **kwargs\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X, y, **kwargs)\n visualizer.transform(X)\n\n # Return the axes object on the visualizer\n return visualizer.ax", "response": "Displays frequency distribution plot for text."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef count(self, X):\n # Sum on axis 0 (by columns), each column is a word\n # Convert the matrix to an array\n # Squeeze to remove the 1 dimension objects (like ravel)\n return np.squeeze(np.asarray(X.sum(axis=0)))", "response": "Returns the counts of all the words in X in the corpus and their corresponding frequency."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, X, y=None):\n\n # Compute the conditional word frequency\n if y is not None:\n # Fit the frequencies\n self.conditional_freqdist_ = {}\n\n # Conditional frequency distribution\n self.classes_ = [str(label) for label in set(y)]\n for label in self.classes_:\n self.conditional_freqdist_[label] = self.count(X[y == label])\n else:\n # No conditional frequencies\n self.conditional_freqdist_ = None\n\n # Frequency distribution of entire corpus.\n self.freqdist_ = self.count(X)\n self.sorted_ = self.freqdist_.argsort()[::-1] # Descending order\n\n # Compute the number of words, vocab, and hapaxes\n self.vocab_ = self.freqdist_.shape[0]\n self.words_ = self.freqdist_.sum()\n self.hapaxes_ = sum(1 for c in self.freqdist_ if c == 1)\n\n # Draw and ensure that we return self\n self.draw()\n return self", "response": "This method is used to fit the frequency distribution for a set of words and vocab. It is used to draw the frequency distribution for a set of words and vocab."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalling from the fit method, this method creates the canvas and draws the distribution plot on it. Parameters ---------- kwargs: generic keyword arguments.", "response": "def draw(self, **kwargs):\n \"\"\"\n Called from the fit method, this method creates the canvas and\n draws the distribution plot on it.\n\n Parameters\n ----------\n kwargs: generic keyword arguments.\n\n \"\"\"\n # Prepare the data\n bins = np.arange(self.N)\n words = [self.features[i] for i in self.sorted_[:self.N]]\n freqs = {}\n\n # Set up the bar plots\n if self.conditional_freqdist_:\n for label, values in sorted(self.conditional_freqdist_.items(), key=itemgetter(0)):\n freqs[label] = [\n values[i] for i in self.sorted_[:self.N]\n ]\n else:\n freqs['corpus'] = [\n self.freqdist_[i] for i in self.sorted_[:self.N]\n ]\n\n # Draw a horizontal barplot\n if self.orient == 'h':\n # Add the barchart, stacking if necessary\n for label, freq in freqs.items():\n self.ax.barh(bins, freq, label=label, align='center')\n\n # Set the y ticks to the words\n self.ax.set_yticks(bins)\n self.ax.set_yticklabels(words)\n\n # Order the features from top to bottom on the y axis\n self.ax.invert_yaxis()\n\n # Turn off y grid lines and turn on x grid lines\n self.ax.yaxis.grid(False)\n self.ax.xaxis.grid(True)\n\n # Draw a vertical barplot\n elif self.orient == 'v':\n # Add the barchart, stacking if necessary\n for label, freq in freqs.items():\n self.ax.bar(bins, freq, label=label, align='edge')\n\n # Set the y ticks to the words\n self.ax.set_xticks(bins)\n self.ax.set_xticklabels(words, rotation=90)\n\n # Turn off x grid lines and turn on y grid lines\n self.ax.yaxis.grid(True)\n self.ax.xaxis.grid(False)\n\n # Unknown state\n else:\n raise YellowbrickValueError(\n \"Orientation must be 'h' or 'v'\"\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef finalize(self, **kwargs):\n # Set the title\n self.set_title(\n 'Frequency Distribution of Top {} tokens'.format(self.N)\n )\n\n # Create the vocab, count, and hapaxes labels\n infolabel = \"vocab: {:,}\\nwords: {:,}\\nhapax: {:,}\".format(\n self.vocab_, self.words_, self.hapaxes_\n )\n\n self.ax.text(0.68, 0.97, infolabel, transform=self.ax.transAxes,\n fontsize=9, verticalalignment='top',\n bbox={'boxstyle':'round', 'facecolor':'white', 'alpha':.8})\n\n # Set the legend and the grid\n self.ax.legend(loc='upper right', frameon=True)", "response": "Sets the title and axes to the base class s title and the label."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a classification report for a single - off analysis.", "response": "def classification_report(model, X, y=None, ax=None, classes=None,\n random_state=None,**kwargs):\n \"\"\"Quick method:\n\n Displays precision, recall, F1, and support scores for the model.\n Integrates numerical scores as well as color-coded heatmap.\n\n This helper function is a quick wrapper to utilize the ClassificationReport\n ScoreVisualizer for one-off analysis.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features.\n\n y : ndarray or Series of length n\n An array or series of target or class values.\n\n ax : matplotlib axes\n The axes to plot the figure on.\n\n model : the Scikit-Learn estimator (should be a classifier)\n\n classes : list of strings\n The names of the classes in the target\n\n random_state: integer\n The seed value for a random generator\n\n Returns\n -------\n ax : matplotlib axes\n Returns the axes that the classification report was drawn on.\n \"\"\"\n # Instantiate the visualizer\n visualizer = ClassificationReport(model, ax, classes, **kwargs)\n\n # Create the train and test splits\n X_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.2, random_state=random_state\n )\n\n # Fit and transform the visualizer (calls draw)\n visualizer.fit(X_train, y_train, **kwargs)\n visualizer.score(X_test, y_test)\n\n # Return the axes object on the visualizer\n return visualizer.ax"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate the Scikit - Learn classification report.", "response": "def score(self, X, y=None, **kwargs):\n \"\"\"\n Generates the Scikit-Learn classification report.\n\n Parameters\n ----------\n X : ndarray or DataFrame of shape n x m\n A matrix of n instances with m features\n\n y : ndarray or Series of length n\n An array or series of target or class values\n\n Returns\n -------\n\n score_ : float\n Global accuracy score\n \"\"\"\n y_pred = self.predict(X)\n\n scores = precision_recall_fscore_support(y, y_pred)\n\n # Calculate the percentage for the support metric\n # and store the percent in place of raw support counts\n self.support_score_ = scores[-1]\n\n scores = list(scores)\n scores[-1] = scores[-1] / scores[-1].sum()\n\n # Create a mapping composed of precision, recall, F1, and support\n # to their respective values\n scores = map(lambda s: dict(zip(self.classes_, s)), scores)\n self.scores_ = dict(zip(SCORES_KEYS, scores))\n\n # Remove support scores if not required\n if not self.support:\n self.scores_.pop('support')\n\n self.draw()\n\n # Retrieve and store the score attribute from the sklearn classifier\n self.score_ = self.estimator.score(X, y)\n\n return self.score_"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef draw(self):\n # Create display grid\n cr_display = np.zeros((len(self.classes_), len(self._displayed_scores)))\n\n\n # For each class row, append columns for precision, recall, f1, and support\n for idx, cls in enumerate(self.classes_):\n for jdx, metric in enumerate(self._displayed_scores):\n cr_display[idx, jdx] = self.scores_[metric][cls]\n\n # Set up the dimensions of the pcolormesh\n # NOTE: pcolormesh accepts grids that are (N+1,M+1)\n X, Y = np.arange(len(self.classes_)+1), np.arange(len(self._displayed_scores)+1)\n self.ax.set_ylim(bottom=0, top=cr_display.shape[0])\n self.ax.set_xlim(left=0, right=cr_display.shape[1])\n\n # Set data labels in the grid, enumerating over class, metric pairs\n # NOTE: X and Y are one element longer than the classification report\n # so skip the last element to label the grid correctly.\n for x in X[:-1]:\n for y in Y[:-1]:\n\n # Extract the value and the text label\n value = cr_display[x, y]\n svalue = \"{:0.3f}\".format(value)\n\n # change the svalue for support (when y == 3) because we want\n # to label it as the actual support value, not the percentage\n if y == 3:\n if self.support != PERCENT:\n svalue = self.support_score_[x]\n\n # Determine the grid and text colors\n base_color = self.cmap(value)\n text_color = find_text_color(base_color)\n\n # Add the label to the middle of the grid\n cx, cy = x+0.5, y+0.5\n self.ax.text(\n cy, cx, svalue, va='center', ha='center', color=text_color\n )\n\n\n # Draw the heatmap with colors bounded by the min and max of the grid\n # NOTE: I do not understand why this is Y, X instead of X, Y it works\n # in this order but raises an exception with the other order.\n g = self.ax.pcolormesh(\n Y, X, cr_display, vmin=0, vmax=1, cmap=self.cmap, edgecolor='w',\n )\n\n # Add the color bar\n plt.colorbar(g, ax=self.ax)\n\n # Return the axes being drawn on\n return self.ax", "response": "Draws the classification report across each axis."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef finalize(self, **kwargs):\n # Set the title of the classifiation report\n self.set_title('{} Classification Report'.format(self.name))\n\n # Set the tick marks appropriately\n self.ax.set_xticks(np.arange(len(self._displayed_scores))+0.5)\n self.ax.set_yticks(np.arange(len(self.classes_))+0.5)\n\n self.ax.set_xticklabels(self._displayed_scores, rotation=45)\n self.ax.set_yticklabels(self.classes_)\n\n plt.tight_layout()", "response": "Finalize executes any subclass - specific axes finalization steps."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a deprecated __getitem__ method that tells users to use the getattr method instead.", "response": "def _deprecated_getitem_method(name, attrs):\n \"\"\"Create a deprecated ``__getitem__`` method that tells users to use\n getattr instead.\n\n Parameters\n ----------\n name : str\n The name of the object in the warning message.\n attrs : iterable[str]\n The set of allowed attributes.\n\n Returns\n -------\n __getitem__ : callable[any, str]\n The ``__getitem__`` method to put in the class dict.\n \"\"\"\n attrs = frozenset(attrs)\n msg = (\n \"'{name}[{attr!r}]' is deprecated, please use\"\n \" '{name}.{attr}' instead\"\n )\n\n def __getitem__(self, key):\n \"\"\"``__getitem__`` is deprecated, please use attribute access instead.\n \"\"\"\n warn(msg.format(name=name, attr=key), DeprecationWarning, stacklevel=2)\n if key in attrs:\n return self.__dict__[key]\n raise KeyError(key)\n\n return __getitem__"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _symbols2assets(self, symbols):\n '''\n Utility for debug/testing\n '''\n\n assets = {a.symbol: a for a in self.get_equities()}\n return [assets[symbol] for symbol in symbols if symbol in assets]", "response": "Utility for debug and testing\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninterfacing method. get_bars Return a DataFrame with columns MultiIndex open high low and volume for each asset.", "response": "def get_bars(self, assets, data_frequency, bar_count=500):\n '''\n Interface method.\n\n Return: pd.Dataframe() with columns MultiIndex [asset -> OHLCV]\n '''\n assets_is_scalar = not isinstance(assets, (list, set, tuple))\n is_daily = 'd' in data_frequency # 'daily' or '1d'\n if assets_is_scalar:\n symbols = [assets.symbol]\n else:\n symbols = [asset.symbol for asset in assets]\n\n symbol_bars = self._symbol_bars(\n symbols, 'day' if is_daily else 'minute', limit=bar_count)\n\n if is_daily:\n intra_bars = {}\n symbol_bars_minute = self._symbol_bars(\n symbols, 'minute', limit=1000)\n for symbol, df in symbol_bars_minute.items():\n agged = df.resample('1D').agg(dict(\n open='first',\n high='max',\n low='min',\n close='last',\n volume='sum',\n )).dropna()\n intra_bars[symbol] = agged\n\n dfs = []\n for asset in assets if not assets_is_scalar else [assets]:\n symbol = asset.symbol\n df = symbol_bars.get(symbol)\n if df is None:\n dfs.append(pd.DataFrame(\n [], columns=[\n 'open', 'high', 'low', 'close', 'volume']\n ))\n continue\n if is_daily:\n agged = intra_bars.get(symbol)\n if agged is not None and len(\n agged.index) > 0 and agged.index[-1] not in df.index:\n if not (agged.index[-1] > df.index[-1]):\n log.warn(\n ('agged.index[-1] = {}, df.index[-1] = {} '\n 'for {}').format(\n agged.index[-1], df.index[-1], symbol))\n df = df.append(agged.iloc[-1])\n df.columns = pd.MultiIndex.from_product([[asset, ], df.columns])\n dfs.append(df)\n\n return pd.concat(dfs, axis=1)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nqueries historic_agg for multiple symbols and return in dict.", "response": "def _symbol_bars(\n self,\n symbols,\n size,\n _from=None,\n to=None,\n limit=None):\n '''\n Query historic_agg either minute or day in parallel\n for multiple symbols, and return in dict.\n\n symbols: list[str]\n size: str ('day', 'minute')\n _from: str or pd.Timestamp\n to: str or pd.Timestamp\n limit: str or int\n\n return: dict[str -> pd.DataFrame]\n '''\n assert size in ('day', 'minute')\n\n # temp workaround for less bars after masking by\n # market hours\n query_limit = limit\n if query_limit is not None:\n query_limit *= 2\n\n @skip_http_error((404, 504))\n def fetch(symbol):\n df = self._api.polygon.historic_agg(\n size, symbol, _from, to, query_limit).df\n\n # zipline -> right label\n # API result -> left label (beginning of bucket)\n if size == 'minute':\n df.index += pd.Timedelta('1min')\n\n # mask out bars outside market hours\n mask = self._cal.minutes_in_range(\n df.index[0], df.index[-1],\n ).tz_convert(NY)\n df = df.reindex(mask)\n\n if limit is not None:\n df = df.iloc[-limit:]\n return df\n\n return parallelize(fetch)(symbols)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _symbol_trades(self, symbols):\n '''\n Query last_trade in parallel for multiple symbols and\n return in dict.\n\n symbols: list[str]\n\n return: dict[str -> polygon.Trade]\n '''\n\n @skip_http_error((404, 504))\n def fetch(symbol):\n return self._api.polygon.last_trade(symbol)\n\n return parallelize(fetch)(symbols)", "response": "Query last_trade in parallel for multiple symbols and\n\n return in dict.\n\n symbols is a list of symbols"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef record(self, *args, **kwargs):\n # Make 2 objects both referencing the same iterator\n args = [iter(args)] * 2\n\n # Zip generates list entries by calling `next` on each iterator it\n # receives. In this case the two iterators are the same object, so the\n # call to next on args[0] will also advance args[1], resulting in zip\n # returning (a,b) (c,d) (e,f) rather than (a,a) (b,b) (c,c) etc.\n positionals = zip(*args)\n for name, value in chain(positionals, kwargs.items()):\n self._recorded_vars[name] = value", "response": "Track and record values each day."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_all_orders(\n self,\n asset=None,\n before=None,\n status='all',\n days_back=None):\n '''\n If asset is unspecified or None, returns a dictionary keyed by\n asset ID. The dictionary contains a list of orders for each ID,\n oldest first. If an asset is specified, returns a list of open\n orders for that asset, oldest first. Orders submitted after\n before will not be returned. If provided, only orders of type\n status ('closed' or 'open') will be returned.\n '''\n orders = self._backend.all_orders(before, status, days_back)\n\n omap = {}\n sorted_orders = sorted([\n o for o in orders.values()\n ], key=lambda o: o.dt)\n for order in sorted_orders:\n key = order.asset\n if key not in omap:\n omap[key] = []\n omap[key].append(order.to_api_obj())\n\n if asset is None:\n return omap\n\n return omap.get(asset, [])", "response": "Returns a dictionary keyed by asset ID and list of orders for each ID."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef history(self, bar_count, frequency, field, ffill=True):\n\n return self.get_history_window(\n bar_count,\n frequency,\n self._calculate_universe(),\n field,\n ffill,\n )", "response": "Get the history window of a specific frequency and field."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates how many shares and contracts to order based on the type of asset being ordered.", "response": "def _calculate_order_value_amount(self, asset, value):\n \"\"\"\n Calculates how many shares/contracts to order based on the type of\n asset being ordered.\n \"\"\"\n if not self.executor.current_data.can_trade(asset):\n raise CannotOrderDelistedAsset(\n msg=\"Cannot order {0}, as it not tradable\".format(asset.symbol)\n )\n\n last_price = \\\n self.executor.current_data.current(asset, \"price\")\n\n if np.isnan(last_price):\n raise CannotOrderDelistedAsset(\n msg=\"Cannot order {0} on {1} as there is no last \"\n \"price for the security.\".format(asset.symbol,\n self.datetime)\n )\n\n if tolerant_equals(last_price, 0):\n zero_message = \"Price of 0 for {psid}; can't infer value\".format(\n psid=asset\n )\n log.debug(zero_message)\n # Don't place any order\n return 0\n\n return value / last_price"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_do_not_order_list(self, restricted_list, on_error='fail'):\n if isinstance(restricted_list, SecurityList):\n warnings.warn(\n \"`set_do_not_order_list(security_lists.leveraged_etf_list)` \"\n \"is deprecated. Use `set_asset_restrictions(\"\n \"security_lists.restrict_leveraged_etfs)` instead.\",\n category=DeprecationWarning,\n stacklevel=2\n )\n restrictions = SecurityListRestrictions(restricted_list)\n else:\n warnings.warn(\n \"`set_do_not_order_list(container_of_assets)` is deprecated. \"\n \"Create a zipline.finance.asset_restrictions.\"\n \"StaticRestrictions object with a container of assets and use \"\n \"`set_asset_restrictions(StaticRestrictions(\"\n \"container_of_assets))` instead.\",\n category=DeprecationWarning,\n stacklevel=2\n )\n restrictions = StaticRestrictions(restricted_list)\n\n self.set_asset_restrictions(restrictions, on_error)", "response": "Set a restriction on which assets can be ordered."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntranslate zipline script into pylivetrader script.", "response": "def translate(script):\n '''translate zipline script into pylivetrader script.\n '''\n tree = ast.parse(script)\n\n ZiplineImportVisitor().visit(tree)\n\n return astor.to_source(tree)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef make_eventrule(date_rule, time_rule, cal, half_days=True):\n\n # Insert the calendar in to the individual rules\n date_rule.cal = cal\n time_rule.cal = cal\n\n if half_days:\n inner_rule = date_rule & time_rule\n else:\n nhd_rule = NotHalfDay()\n nhd_rule.cal = cal\n inner_rule = date_rule & time_rule & nhd_rule\n\n return OncePerDay(rule=inner_rule)", "response": "Creates an event rule from the factory api."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_pipeline(context):\n\n # Filter for primary share equities. IsPrimaryShare is a built-in filter.\n primary_share = IsPrimaryShare()\n\n # Not when-issued equities.\n not_wi = ~IEXCompany.symbol.latest.endswith('.WI')\n\n # Equities without LP in their name, .matches does a match using a regular\n # expression\n not_lp_name = ~IEXCompany.companyName.latest.matches('.* L[. ]?P.?$')\n\n # Equities whose most recent Morningstar market cap is not null have\n # fundamental data and therefore are not ETFs.\n have_market_cap = IEXKeyStats.marketcap.latest >= 1\n\n # At least a certain price\n price = USEquityPricing.close.latest\n AtLeastPrice = (price >= context.MyLeastPrice)\n AtMostPrice = (price <= context.MyMostPrice)\n\n # Filter for stocks that pass all of our previous filters.\n tradeable_stocks = (\n primary_share\n & not_wi\n & not_lp_name\n & have_market_cap\n & AtLeastPrice\n & AtMostPrice\n )\n\n LowVar = 6\n HighVar = 40\n\n log.info(\n '''\nAlgorithm initialized variables:\n context.MaxCandidates %s\n LowVar %s\n HighVar %s''' %\n (context.MaxCandidates, LowVar, HighVar))\n\n # High dollar volume filter.\n base_universe = AverageDollarVolume(\n window_length=20,\n mask=tradeable_stocks\n ).percentile_between(LowVar, HighVar)\n\n # Short close price average.\n ShortAvg = SimpleMovingAverage(\n inputs=[USEquityPricing.close],\n window_length=3,\n mask=base_universe\n )\n\n # Long close price average.\n LongAvg = SimpleMovingAverage(\n inputs=[USEquityPricing.close],\n window_length=45,\n mask=base_universe\n )\n\n percent_difference = (ShortAvg - LongAvg) / LongAvg\n\n # Filter to select securities to long.\n stocks_worst = percent_difference.bottom(context.MaxCandidates)\n securities_to_trade = (stocks_worst)\n\n return Pipeline(\n columns={\n 'stocks_worst': stocks_worst\n },\n screen=(securities_to_trade),\n )", "response": "Create our pipeline for the current context."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrecord variables at the end of each day.", "response": "def my_record_vars(context, data):\n \"\"\"\n Record variables at the end of each day.\n \"\"\"\n\n # Record our variables.\n record(leverage=context.account.leverage)\n record(positions=len(context.portfolio.positions))\n if 0 < len(context.age):\n MaxAge = context.age[max(\n list(context.age.keys()), key=(lambda k: context.age[k]))]\n print(MaxAge)\n record(MaxAge=MaxAge)\n record(LowestPrice=context.LowestPrice)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef retrieve_all(self, sids, default_none=False):\n\n failures = set()\n hits = {}\n for sid in sids:\n try:\n hits[sid] = self._asset_cache[sid]\n except KeyError:\n if not default_none:\n failures.add(sid)\n else:\n hits[sid] = None\n\n if len(failures) > 0:\n raise SidsNotFound(sids=list(failures))\n\n return [hits[sid] for sid in sids]", "response": "Retrieve all assets in sids."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef retrieve_asset(self, sid, default_none=False):\n try:\n asset = self._asset_cache[sid]\n if asset is None and not default_none:\n raise SidsNotFound(sids=[sid])\n return asset\n except KeyError:\n raise SidsNotFound(sids=[sid])", "response": "Retrieve the Asset for a given sid."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef retrieve_equities(self, sids):\n cache = self._asset_cache\n\n try:\n return {\n k: cache[k]\n for k in sids\n }\n except KeyError:\n raise EquitiesNotFound(sids=sids)", "response": "Retrieve Equity objects for a list of sids."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if the asset s exchange calendar can be tradeed at the current simulation time.", "response": "def can_trade(self, assets):\n \"\"\"\n For the given asset or iterable of assets, returns true if all of the\n following are true:\n 1) the asset is alive for the session of the current simulation time\n (if current simulation time is not a market minute, we use the next\n session)\n 2) (if we are in minute mode) the asset's exchange is open at the\n current simulation time or at the simulation calendar's next market\n minute\n 3) there is a known last price for the asset.\n\n Notes\n -----\n The second condition above warrants some further explanation.\n - If the asset's exchange calendar is identical to the simulation\n calendar, then this condition always returns True.\n - If there are market minutes in the simulation calendar outside of\n this asset's exchange's trading hours (for example, if the simulation\n is running on the CME calendar but the asset is MSFT, which trades on\n the NYSE), during those minutes, this condition will return false\n (for example, 3:15 am Eastern on a weekday, during which the CME is\n open but the NYSE is closed).\n\n Parameters\n ----------\n assets: Asset or iterable of assets\n\n Returns\n -------\n can_trade : bool or pd.Series[bool] indexed by asset.\n \"\"\"\n dt = self.datetime\n\n if self._adjust_minutes:\n adjusted_dt = self._get_current_minute()\n else:\n adjusted_dt = dt\n\n data_portal = self.data_portal\n\n if isinstance(assets, Asset):\n return self._can_trade_for_asset(\n assets, dt, adjusted_dt, data_portal\n )\n else:\n def fetch(asset):\n return self._can_trade_for_asset(\n asset, dt, adjusted_dt, data_portal\n )\n tradeable = parallelize(fetch)(assets)\n return pd.Series(data=tradeable, index=assets, dtype=bool)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if the asset is alive and there is no trade data for the current simulation time.", "response": "def is_stale(self, assets):\n \"\"\"\n For the given asset or iterable of assets, returns true if the asset\n is alive and there is no trade data for the current simulation time.\n\n If the asset has never traded, returns False.\n\n If the current simulation time is not a valid market time, we use the\n current time to check if the asset is alive, but we use the last\n market minute/day for the trade data check.\n\n Parameters\n ----------\n assets: Asset or iterable of assets\n\n Returns\n -------\n boolean or Series of booleans, indexed by asset.\n \"\"\"\n dt = self.datetime\n if self._adjust_minutes:\n adjusted_dt = self._get_current_minute()\n else:\n adjusted_dt = dt\n\n data_portal = self.data_portal\n\n if isinstance(assets, Asset):\n return self._is_stale_for_asset(\n assets, dt, adjusted_dt, data_portal\n )\n else:\n return pd.Series(data={\n asset: self._is_stale_for_asset(\n asset, dt, adjusted_dt, data_portal\n )\n for asset in assets\n })"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef api_method(f):\n '''\n Redirect pylivetrader.api.* operations to the algorithm\n in the local context.\n '''\n\n @wraps(f)\n def wrapped(*args, **kwargs):\n # Get the instance and call the method\n algorithm = get_context()\n if algorithm is None:\n raise RuntimeError(\n '{} method must be called during live trading'.format(\n f.__name__))\n return getattr(algorithm, f.__name__)(*args, **kwargs)\n\n # register api redirection\n setattr(pylivetrader.api, f.__name__, wrapped)\n pylivetrader.api.__all__.append(f.__name__)\n f.is_api_method = True\n\n return f", "response": "Decorator for handling API methods."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new pipeline for the given context.", "response": "def make_pipeline(context):\n \"\"\"\n Create our pipeline.\n \"\"\"\n\n # Filter for primary share equities. IsPrimaryShare is a built-in filter.\n primary_share = IsPrimaryShare()\n\n # Equities listed as common stock (as opposed to, say, preferred stock).\n # 'ST00000001' indicates common stock.\n common_stock = morningstar.share_class_reference.security_type.latest.eq(\n 'ST00000001')\n\n # Non-depositary receipts. Recall that the ~ operator inverts filters,\n # turning Trues into Falses and vice versa\n not_depositary = ~morningstar.share_class_reference.is_depositary_receipt.latest\n\n # Equities not trading over-the-counter.\n not_otc = ~morningstar.share_class_reference.exchange_id.latest.startswith(\n 'OTC')\n\n # Not when-issued equities.\n not_wi = ~morningstar.share_class_reference.symbol.latest.endswith('.WI')\n\n # Equities without LP in their name, .matches does a match using a regular\n # expression\n not_lp_name = ~morningstar.company_reference.standard_name.latest.matches(\n '.* L[. ]?P.?$')\n\n # Equities with a null value in the limited_partnership Morningstar\n # fundamental field.\n not_lp_balance_sheet = morningstar.balance_sheet.limited_partnership.latest.isnull()\n\n # Equities whose most recent Morningstar market cap is not null have\n # fundamental data and therefore are not ETFs.\n have_market_cap = morningstar.valuation.market_cap.latest.notnull()\n\n # At least a certain price\n price = USEquityPricing.close.latest\n AtLeastPrice = (price >= context.MyLeastPrice)\n AtMostPrice = (price <= context.MyMostPrice)\n\n # Filter for stocks that pass all of our previous filters.\n tradeable_stocks = (\n primary_share\n & common_stock\n & not_depositary\n & not_otc\n & not_wi\n & not_lp_name\n & not_lp_balance_sheet\n & have_market_cap\n & AtLeastPrice\n & AtMostPrice\n )\n\n LowVar = 6\n HighVar = 40\n\n log.info(\n '''\nAlgorithm initialized variables:\n context.MaxCandidates %s\n LowVar %s\n HighVar %s''' %\n (context.MaxCandidates, LowVar, HighVar))\n\n # High dollar volume filter.\n base_universe = AverageDollarVolume(\n window_length=20,\n mask=tradeable_stocks\n ).percentile_between(LowVar, HighVar)\n\n # Short close price average.\n ShortAvg = SimpleMovingAverage(\n inputs=[USEquityPricing.close],\n window_length=3,\n mask=base_universe\n )\n\n # Long close price average.\n LongAvg = SimpleMovingAverage(\n inputs=[USEquityPricing.close],\n window_length=45,\n mask=base_universe\n )\n\n percent_difference = (ShortAvg - LongAvg) / LongAvg\n\n # Filter to select securities to long.\n stocks_worst = percent_difference.bottom(context.MaxCandidates)\n securities_to_trade = (stocks_worst)\n\n return Pipeline(\n columns={\n 'stocks_worst': stocks_worst\n },\n screen=(securities_to_trade),\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_adjusted_value(\n self,\n assets,\n field,\n dt,\n perspective_dt,\n data_frequency):\n '''\n TODO:\n for external data (fetch_csv) support, need to update logic here.\n '''\n return self.backend.get_spot_value(\n assets, field, dt, data_frequency, self.quantopian_compatible\n )", "response": "Get the adjusted value for the given object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_df_from_file(file_path, sep=\",\", header=0):\n with tf.gfile.Open(file_path) as infile:\n df = pd.read_csv(infile, sep=sep, header=header)\n return df", "response": "Wrapper around pandas read_csv."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef serialize_to_file(obj, file_name, append=False):\n logging.info(\"Serializing to file %s.\", file_name)\n with tf.gfile.Open(file_name, \"a+\" if append else \"wb\") as output_file:\n pickle.dump(obj, output_file)\n logging.info(\"Done serializing to file %s.\", file_name)", "response": "Pickle obj to file_name."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef savez_two_column(matrix, row_offset, file_name, append=False):\n logging.info(\"Saving obj to file in two column .npz format %s.\", file_name)\n tc = []\n for u, items in enumerate(matrix):\n user = row_offset + u\n for item in items:\n tc.append([user, item])\n \n np.savez_compressed(file_name, np.asarray(tc))\n logging.info(\"Done saving to file %s.\", file_name)", "response": "Save a compressed matrix to file_name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sorted_product_set(array_a, array_b):\n return np.sort(\n np.concatenate(\n [array_a[i] * array_b for i in xrange(len(array_a))], axis=0)\n )[::-1]", "response": "Compute the product set of array_a and array_b and sort it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the given list of matches that have only low - quality matches.", "response": "def set_low_quality_matches_(self, matches, all_matches, match_quality_matrix):\n \"\"\"\n Produce additional matches for predictions that have only low-quality matches.\n Specifically, for each ground-truth find the set of predictions that have\n maximum overlap with it (including ties); for each prediction in that set, if\n it is unmatched, then match it to the ground-truth with which it has the highest\n quality value.\n \"\"\"\n # For each gt, find the prediction with which it has highest quality\n highest_quality_foreach_gt, _ = match_quality_matrix.max(dim=1)\n # Find highest quality match available, even if it is low, including ties\n gt_pred_pairs_of_highest_quality = torch.nonzero(\n match_quality_matrix == highest_quality_foreach_gt[:, None]\n )\n # Example gt_pred_pairs_of_highest_quality:\n # tensor([[ 0, 39796],\n # [ 1, 32055],\n # [ 1, 32070],\n # [ 2, 39190],\n # [ 2, 40255],\n # [ 3, 40390],\n # [ 3, 41455],\n # [ 4, 45470],\n # [ 5, 45325],\n # [ 5, 46390]])\n # Each row is a (gt index, prediction index)\n # Note how gt items 1, 2, 3, and 5 each have two ties\n\n pred_inds_to_update = gt_pred_pairs_of_highest_quality[:, 1]\n matches[pred_inds_to_update] = all_matches[pred_inds_to_update]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef eval_detection_voc(pred_boxlists, gt_boxlists, iou_thresh=0.5, use_07_metric=False):\n assert len(gt_boxlists) == len(\n pred_boxlists\n ), \"Length of gt and pred lists need to be same.\"\n prec, rec = calc_detection_voc_prec_rec(\n pred_boxlists=pred_boxlists, gt_boxlists=gt_boxlists, iou_thresh=iou_thresh\n )\n ap = calc_detection_voc_ap(prec, rec, use_07_metric=use_07_metric)\n return {\"ap\": ap, \"map\": np.nanmean(ap)}", "response": "Evaluate on voc dataset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate precision and recall of a detection VOC based on evaluation code of PASCAL VOC Challenge.", "response": "def calc_detection_voc_prec_rec(gt_boxlists, pred_boxlists, iou_thresh=0.5):\n \"\"\"Calculate precision and recall based on evaluation code of PASCAL VOC.\n This function calculates precision and recall of\n predicted bounding boxes obtained from a dataset which has :math:`N`\n images.\n The code is based on the evaluation code used in PASCAL VOC Challenge.\n \"\"\"\n n_pos = defaultdict(int)\n score = defaultdict(list)\n match = defaultdict(list)\n for gt_boxlist, pred_boxlist in zip(gt_boxlists, pred_boxlists):\n pred_bbox = pred_boxlist.bbox.numpy()\n pred_label = pred_boxlist.get_field(\"labels\").numpy()\n pred_score = pred_boxlist.get_field(\"scores\").numpy()\n gt_bbox = gt_boxlist.bbox.numpy()\n gt_label = gt_boxlist.get_field(\"labels\").numpy()\n gt_difficult = gt_boxlist.get_field(\"difficult\").numpy()\n\n for l in np.unique(np.concatenate((pred_label, gt_label)).astype(int)):\n pred_mask_l = pred_label == l\n pred_bbox_l = pred_bbox[pred_mask_l]\n pred_score_l = pred_score[pred_mask_l]\n # sort by score\n order = pred_score_l.argsort()[::-1]\n pred_bbox_l = pred_bbox_l[order]\n pred_score_l = pred_score_l[order]\n\n gt_mask_l = gt_label == l\n gt_bbox_l = gt_bbox[gt_mask_l]\n gt_difficult_l = gt_difficult[gt_mask_l]\n\n n_pos[l] += np.logical_not(gt_difficult_l).sum()\n score[l].extend(pred_score_l)\n\n if len(pred_bbox_l) == 0:\n continue\n if len(gt_bbox_l) == 0:\n match[l].extend((0,) * pred_bbox_l.shape[0])\n continue\n\n # VOC evaluation follows integer typed bounding boxes.\n pred_bbox_l = pred_bbox_l.copy()\n pred_bbox_l[:, 2:] += 1\n gt_bbox_l = gt_bbox_l.copy()\n gt_bbox_l[:, 2:] += 1\n iou = boxlist_iou(\n BoxList(pred_bbox_l, gt_boxlist.size),\n BoxList(gt_bbox_l, gt_boxlist.size),\n ).numpy()\n gt_index = iou.argmax(axis=1)\n # set -1 if there is no matching ground truth\n gt_index[iou.max(axis=1) < iou_thresh] = -1\n del iou\n\n selec = np.zeros(gt_bbox_l.shape[0], dtype=bool)\n for gt_idx in gt_index:\n if gt_idx >= 0:\n if gt_difficult_l[gt_idx]:\n match[l].append(-1)\n else:\n if not selec[gt_idx]:\n match[l].append(1)\n else:\n match[l].append(0)\n selec[gt_idx] = True\n else:\n match[l].append(0)\n\n n_fg_class = max(n_pos.keys()) + 1\n prec = [None] * n_fg_class\n rec = [None] * n_fg_class\n\n for l in n_pos.keys():\n score_l = np.array(score[l])\n match_l = np.array(match[l], dtype=np.int8)\n\n order = score_l.argsort()[::-1]\n match_l = match_l[order]\n\n tp = np.cumsum(match_l == 1)\n fp = np.cumsum(match_l == 0)\n\n # If an element of fp + tp is 0,\n # the corresponding element of prec[l] is nan.\n prec[l] = tp / (fp + tp)\n # If n_pos[l] is 0, rec[l] is None.\n if n_pos[l] > 0:\n rec[l] = tp / n_pos[l]\n\n return prec, rec"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the average precisions of a given set of precisions and recalls.", "response": "def calc_detection_voc_ap(prec, rec, use_07_metric=False):\n \"\"\"Calculate average precisions based on evaluation code of PASCAL VOC.\n This function calculates average precisions\n from given precisions and recalls.\n The code is based on the evaluation code used in PASCAL VOC Challenge.\n Args:\n prec (list of numpy.array): A list of arrays.\n :obj:`prec[l]` indicates precision for class :math:`l`.\n If :obj:`prec[l]` is :obj:`None`, this function returns\n :obj:`numpy.nan` for class :math:`l`.\n rec (list of numpy.array): A list of arrays.\n :obj:`rec[l]` indicates recall for class :math:`l`.\n If :obj:`rec[l]` is :obj:`None`, this function returns\n :obj:`numpy.nan` for class :math:`l`.\n use_07_metric (bool): Whether to use PASCAL VOC 2007 evaluation metric\n for calculating average precision. The default value is\n :obj:`False`.\n Returns:\n ~numpy.ndarray:\n This function returns an array of average precisions.\n The :math:`l`-th value corresponds to the average precision\n for class :math:`l`. If :obj:`prec[l]` or :obj:`rec[l]` is\n :obj:`None`, the corresponding value is set to :obj:`numpy.nan`.\n \"\"\"\n\n n_fg_class = len(prec)\n ap = np.empty(n_fg_class)\n for l in range(n_fg_class):\n if prec[l] is None or rec[l] is None:\n ap[l] = np.nan\n continue\n\n if use_07_metric:\n # 11 point metric\n ap[l] = 0\n for t in np.arange(0.0, 1.1, 0.1):\n if np.sum(rec[l] >= t) == 0:\n p = 0\n else:\n p = np.max(np.nan_to_num(prec[l])[rec[l] >= t])\n ap[l] += p / 11\n else:\n # correct AP calculation\n # first append sentinel values at the end\n mpre = np.concatenate(([0], np.nan_to_num(prec[l]), [0]))\n mrec = np.concatenate(([0], rec[l], [1]))\n\n mpre = np.maximum.accumulate(mpre[::-1])[::-1]\n\n # to calculate area under PR curve, look for points\n # where X axis (recall) changes value\n i = np.where(mrec[1:] != mrec[:-1])[0]\n\n # and sum (\\Delta recall) * prec\n ap[l] = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])\n\n return ap"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_gt_proposals(self, proposals, targets):\n # Get the device we're operating on\n device = proposals[0].bbox.device\n\n gt_boxes = [target.copy_with_fields([]) for target in targets]\n\n # later cat of bbox requires all fields to be present for all bbox\n # so we need to add a dummy for objectness that's missing\n for gt_box in gt_boxes:\n gt_box.add_field(\"objectness\", torch.ones(len(gt_box), device=device))\n\n proposals = [\n cat_boxlist((proposal, gt_box))\n for proposal, gt_box in zip(proposals, gt_boxes)\n ]\n\n return proposals", "response": "Add the objectness of the given proposals to the target list."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef forward_for_single_feature_map(self, anchors, objectness, box_regression):\n device = objectness.device\n N, A, H, W = objectness.shape\n\n # put in the same format as anchors\n objectness = permute_and_flatten(objectness, N, A, 1, H, W).view(N, -1)\n objectness = objectness.sigmoid()\n\n box_regression = permute_and_flatten(box_regression, N, A, 4, H, W)\n\n num_anchors = A * H * W\n\n pre_nms_top_n = min(self.pre_nms_top_n, num_anchors)\n objectness, topk_idx = objectness.topk(pre_nms_top_n, dim=1, sorted=True)\n\n batch_idx = torch.arange(N, device=device)[:, None]\n box_regression = box_regression[batch_idx, topk_idx]\n\n image_shapes = [box.size for box in anchors]\n concat_anchors = torch.cat([a.bbox for a in anchors], dim=0)\n concat_anchors = concat_anchors.reshape(N, -1, 4)[batch_idx, topk_idx]\n\n proposals = self.box_coder.decode(\n box_regression.view(-1, 4), concat_anchors.view(-1, 4)\n )\n\n proposals = proposals.view(N, -1, 4)\n\n result = []\n for proposal, score, im_shape in zip(proposals, objectness, image_shapes):\n boxlist = BoxList(proposal, im_shape, mode=\"xyxy\")\n boxlist.add_field(\"objectness\", score)\n boxlist = boxlist.clip_to_image(remove_empty=False)\n boxlist = remove_small_boxes(boxlist, self.min_size)\n boxlist = boxlist_nms(\n boxlist,\n self.nms_thresh,\n max_proposals=self.post_nms_top_n,\n score_field=\"objectness\",\n )\n result.append(boxlist)\n return result", "response": "Forward for single feature map."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef forward(self, anchors, objectness, box_regression, targets=None):\n sampled_boxes = []\n num_levels = len(objectness)\n anchors = list(zip(*anchors))\n for a, o, b in zip(anchors, objectness, box_regression):\n sampled_boxes.append(self.forward_for_single_feature_map(a, o, b))\n\n boxlists = list(zip(*sampled_boxes))\n boxlists = [cat_boxlist(boxlist) for boxlist in boxlists]\n\n if num_levels > 1:\n boxlists = self.select_over_all_levels(boxlists)\n\n # append ground-truth bboxes to proposals\n if self.training and targets is not None:\n boxlists = self.add_gt_proposals(boxlists, targets)\n\n return boxlists", "response": "Forward the list of anchors objectness and box regression and return a list of boxlists."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _dropout_sparse_coo_matrix(sparse_matrix, rate,\n min_dropout_rate, max_dropout_rate):\n \"\"\"Drop values from a sparse matrix encoded as a SciPy coo matrix.\n\n Args:\n sparse_matrix: a SciPy coo sparse matrix.\n rate: if rate > 0 then non-zero elements of the input matrix\n will be droped uniformly at random.\n min_dropout_rate: minimum value for the dropout rate. If None\n FLAGS.min_dropout_rate is used. If dropout_rate is lower than\n min_dropout_rate it will clipped to min_dropout_rate.\n max_dropout_rate: minimum value for the dropout rate. If None\n FLAGS.max_dropout_rate is used. If dropout_rate is greater than\n max_dropout_rate it will clipped to max_dropout_rate.\n\n Returns:\n A SciPy coo matrix containing those non zero elements that have not been\n dropped out.\n \"\"\"\n if min_dropout_rate is None:\n min_dropout_rate = FLAGS.min_dropout_rate\n\n if max_dropout_rate is None:\n max_dropout_rate = FLAGS.max_dropout_rate\n\n if min_dropout_rate > max_dropout_rate:\n raise ValueError(\"min_dropout_rate (%f) should be less or equal to \"\n \"max_dropout_rate (%f)\"\n % (min_dropout_rate, max_dropout_rate))\n\n max_frac = 1.0 - min_dropout_rate\n min_frac = 1.0 - max_dropout_rate\n sampling_rate = 1.0 - rate\n\n sampled_fraction = min(max(sampling_rate, min_frac), max_frac)\n if sampled_fraction != sampling_rate:\n logging.warning(\"Minimum sampling rate is %2f.\", min_frac)\n logging.warning(\"Maximum sampling rate is %2f.\", max_frac)\n logging.warning(\"Desired sampling rate is %2f.\", sampling_rate)\n logging.warning(\"Desired sampling rate %2f clipped to %2f.\", sampling_rate,\n sampled_fraction)\n\n num_sampled = min(\n max(int(sparse_matrix.nnz * sampled_fraction), 1), sparse_matrix.nnz)\n sampled_indices = np.random.choice(sparse_matrix.nnz, size=num_sampled,\n replace=False)\n\n return sparse.coo_matrix((sparse_matrix.data[sampled_indices],\n (sparse_matrix.row[sampled_indices],\n sparse_matrix.col[sampled_indices])),\n shape=sparse_matrix.shape)", "response": "Drop values from a sparse matrix encoded as a SciPy coo matrix."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nshuffles a sparse matrix encoded as a SciPy csr_matrix.", "response": "def shuffle_sparse_coo_matrix(sparse_matrix, dropout_rate=0.0,\n min_dropout_rate=None, max_dropout_rate=None):\n \"\"\"Shuffle sparse matrix encoded as a SciPy coo matrix.\n\n Args:\n sparse_matrix: a SciPy coo sparse matrix.\n dropout_rate: if dropout_rate > 0 then non-zero elements of the input matrix\n will be droped uniformly at random.\n min_dropout_rate: minimum value for the dropout rate. If None\n FLAGS.min_dropout_rate is used.\n max_dropout_rate: minimum value for the dropout rate. If None\n FLAGS.max_dropout_rate is used.\n\n Returns:\n A SciPy csr_matrix entailing the randomized interactions.\n \"\"\"\n\n if (dropout_rate < 0.0) or (dropout_rate >= 1.0):\n raise ValueError(\"Dropout rate should be in [0, 1) but is %f\"\n % dropout_rate)\n\n (num_rows, num_cols) = sparse_matrix.shape\n shuffled_rows = shuffle(np.arange(num_rows))\n shuffled_cols = shuffle(np.arange(num_cols))\n\n if dropout_rate > 0.0:\n sparse_matrix = _dropout_sparse_coo_matrix(\n sparse_matrix, dropout_rate, min_dropout_rate, max_dropout_rate)\n\n new_row = np.take(shuffled_rows, sparse_matrix.row)\n new_col = np.take(shuffled_cols, sparse_matrix.col)\n\n return sparse.csr_matrix(\n (sparse_matrix.data, (new_row, new_col)), shape=(num_rows, num_cols))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nplays out a self - play match.", "response": "def play(network):\n \"\"\"Plays out a self-play match, returning a MCTSPlayer object containing:\n - the final position\n - the n x 362 tensor of floats representing the mcts search probabilities\n - the n-ary tensor of floats representing the original value-net estimate\n where n is the number of moves in the game\n \"\"\"\n readouts = FLAGS.num_readouts # defined in strategies.py\n # Disable resign in 5% of games\n if random.random() < FLAGS.resign_disable_pct:\n resign_threshold = -1.0\n else:\n resign_threshold = None\n\n player = MCTSPlayer(network, resign_threshold=resign_threshold)\n\n player.initialize_game()\n\n # Must run this once at the start to expand the root node.\n first_node = player.root.select_leaf()\n prob, val = network.run(first_node.position)\n first_node.incorporate_results(prob, val, first_node)\n\n while True:\n start = time.time()\n player.root.inject_noise()\n current_readouts = player.root.N\n # we want to do \"X additional readouts\", rather than \"up to X readouts\".\n while player.root.N < current_readouts + readouts:\n player.tree_search()\n\n if FLAGS.verbose >= 3:\n print(player.root.position)\n print(player.root.describe())\n\n if player.should_resign():\n player.set_result(-1 * player.root.position.to_play,\n was_resign=True)\n break\n move = player.pick_move()\n player.play_move(move)\n if player.root.is_done():\n player.set_result(player.root.position.result(), was_resign=False)\n break\n\n if (FLAGS.verbose >= 2) or (FLAGS.verbose >= 1 and player.root.position.n % 10 == 9):\n print(\"Q: {:.5f}\".format(player.root.Q))\n dur = time.time() - start\n print(\"%d: %d readouts, %.3f s/100. (%.2f sec)\" % (\n player.root.position.n, readouts, dur / readouts * 100.0, dur), flush=True)\n if FLAGS.verbose >= 3:\n print(\"Played >>\",\n coords.to_gtp(coords.from_flat(player.root.fmove)))\n\n if FLAGS.verbose >= 2:\n utils.dbg(\"%s: %.3f\" % (player.result_string, player.root.Q))\n utils.dbg(player.root.position, player.root.position.score())\n\n return player"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run_game(load_file, selfplay_dir=None, holdout_dir=None,\n sgf_dir=None, holdout_pct=0.05):\n \"\"\"Takes a played game and record results and game data.\"\"\"\n if sgf_dir is not None:\n minimal_sgf_dir = os.path.join(sgf_dir, 'clean')\n full_sgf_dir = os.path.join(sgf_dir, 'full')\n utils.ensure_dir_exists(minimal_sgf_dir)\n utils.ensure_dir_exists(full_sgf_dir)\n if selfplay_dir is not None:\n utils.ensure_dir_exists(selfplay_dir)\n utils.ensure_dir_exists(holdout_dir)\n\n with utils.logged_timer(\"Loading weights from %s ... \" % load_file):\n network = dual_net.DualNetwork(load_file)\n\n with utils.logged_timer(\"Playing game\"):\n player = play(network)\n\n output_name = '{}-{}'.format(int(time.time()), socket.gethostname())\n game_data = player.extract_data()\n if sgf_dir is not None:\n with gfile.GFile(os.path.join(minimal_sgf_dir, '{}.sgf'.format(output_name)), 'w') as f:\n f.write(player.to_sgf(use_comments=False))\n with gfile.GFile(os.path.join(full_sgf_dir, '{}.sgf'.format(output_name)), 'w') as f:\n f.write(player.to_sgf())\n\n tf_examples = preprocessing.make_dataset_from_selfplay(game_data)\n\n if selfplay_dir is not None:\n # Hold out 5% of games for validation.\n if random.random() < holdout_pct:\n fname = os.path.join(holdout_dir,\n \"{}.tfrecord.zz\".format(output_name))\n else:\n fname = os.path.join(selfplay_dir,\n \"{}.tfrecord.zz\".format(output_name))\n\n preprocessing.write_tf_examples(fname, tf_examples)", "response": "Takes a played game and record results and game data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef main(argv):\n del argv # Unused\n flags.mark_flag_as_required('load_file')\n\n run_game(\n load_file=FLAGS.load_file,\n selfplay_dir=FLAGS.selfplay_dir,\n holdout_dir=FLAGS.holdout_dir,\n holdout_pct=FLAGS.holdout_pct,\n sgf_dir=FLAGS.sgf_dir)", "response": "Entry point for running one selfplay game."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\napplying algorith 2 in https://arxiv.org/pdf/1901.08910.pdf. Args: usv: matrix to reduce given in SVD form with the spectrum s in increasing order. num_rows: number of rows in the output matrix. num_cols: number of columns in the output matrix. Returns: A resized version of (u, s, v) whose non zero singular values will be identical to the largest singular values in s.", "response": "def resize_matrix(usv, num_rows, num_cols):\n \"\"\"Apply algorith 2 in https://arxiv.org/pdf/1901.08910.pdf.\n\n Args:\n usv: matrix to reduce given in SVD form with the spectrum s in\n increasing order.\n num_rows: number of rows in the output matrix.\n num_cols: number of columns in the output matrix.\n Returns:\n A resized version of (u, s, v) whose non zero singular values will be\n identical to the largest singular values in s.\n \"\"\"\n u, s, v = usv\n k = min(num_rows, num_cols)\n\n u_random_proj = transform.resize(u[:, :k], (num_rows, k))\n v_random_proj = transform.resize(v[:k, :], (k, num_cols))\n\n u_random_proj_orth = _closest_column_orthogonal_matrix(u_random_proj)\n v_random_proj_orth = _closest_column_orthogonal_matrix(v_random_proj.T).T\n\n return np.matmul(u_random_proj_orth,\n np.matmul(np.diag(s[::-1][:k]), v_random_proj_orth))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfolding all values of the matrix into [ 0 1 ).", "response": "def normalize_matrix(matrix):\n \"\"\"Fold all values of the matrix into [0, 1].\"\"\"\n abs_matrix = np.abs(matrix.copy())\n return abs_matrix / abs_matrix.max()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_sorted_inputs(filename):\n with tf.gfile.Open(filename) as f:\n records = f.read().split(\"\\n\")\n inputs = [record.strip() for record in records]\n if not inputs[-1]:\n inputs.pop()\n\n input_lens = [(i, len(line.split())) for i, line in enumerate(inputs)]\n sorted_input_lens = sorted(input_lens, key=lambda x: x[1], reverse=True)\n\n sorted_inputs = []\n sorted_keys = {}\n for i, (index, _) in enumerate(sorted_input_lens):\n sorted_inputs.append(inputs[index])\n sorted_keys[index] = i\n return sorted_inputs, sorted_keys", "response": "Read and sort lines from the file sorted by decreasing length."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _trim_and_decode(ids, subtokenizer):\n try:\n index = list(ids).index(tokenizer.EOS_ID)\n return subtokenizer.decode(ids[:index])\n except ValueError: # No EOS found in sequence\n return subtokenizer.decode(ids)", "response": "Trim EOS and PAD tokens from ids and decode to return a string."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntranslate lines in file and save to output file if specified.", "response": "def translate_file(\n estimator, subtokenizer, input_file, output_file=None,\n print_all_translations=True):\n \"\"\"Translate lines in file, and save to output file if specified.\n\n Args:\n estimator: tf.Estimator used to generate the translations.\n subtokenizer: Subtokenizer object for encoding and decoding source and\n translated lines.\n input_file: file containing lines to translate\n output_file: file that stores the generated translations.\n print_all_translations: If true, all translations are printed to stdout.\n\n Raises:\n ValueError: if output file is invalid.\n \"\"\"\n batch_size = _DECODE_BATCH_SIZE\n\n # Read and sort inputs by length. Keep dictionary (original index-->new index\n # in sorted list) to write translations in the original order.\n sorted_inputs, sorted_keys = _get_sorted_inputs(input_file)\n num_decode_batches = (len(sorted_inputs) - 1) // batch_size + 1\n\n def input_generator():\n \"\"\"Yield encoded strings from sorted_inputs.\"\"\"\n for i, line in enumerate(sorted_inputs):\n if i % batch_size == 0:\n batch_num = (i // batch_size) + 1\n\n print(\"Decoding batch %d out of %d.\" % (batch_num, num_decode_batches))\n yield _encode_and_add_eos(line, subtokenizer)\n\n def input_fn():\n \"\"\"Created batched dataset of encoded inputs.\"\"\"\n ds = tf.data.Dataset.from_generator(\n input_generator, tf.int64, tf.TensorShape([None]))\n ds = ds.padded_batch(batch_size, [None])\n return ds\n\n translations = []\n for i, prediction in enumerate(estimator.predict(input_fn)):\n translation = _trim_and_decode(prediction[\"outputs\"], subtokenizer)\n translations.append(translation)\n\n if print_all_translations:\n print(\"Translating:\")\n print(\"\\tInput: %s\" % sorted_inputs[i])\n print(\"\\tOutput: %s\\n\" % translation)\n print(\"=\" * 100)\n\n # Write translations in the order they appeared in the original file.\n if output_file is not None:\n if tf.gfile.IsDirectory(output_file):\n raise ValueError(\"File output is a directory, will not save outputs to \"\n \"file.\")\n tf.logging.info(\"Writing to file %s\" % output_file)\n with tf.gfile.Open(output_file, \"w\") as f:\n for index in xrange(len(sorted_keys)):\n f.write(\"%s\\n\" % translations[sorted_keys[index]])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef translate_text(estimator, subtokenizer, txt):\n encoded_txt = _encode_and_add_eos(txt, subtokenizer)\n\n def input_fn():\n ds = tf.data.Dataset.from_tensors(encoded_txt)\n ds = ds.batch(_DECODE_BATCH_SIZE)\n return ds\n\n predictions = estimator.predict(input_fn)\n translation = next(predictions)[\"outputs\"]\n translation = _trim_and_decode(translation, subtokenizer)\n print(\"Translation of \\\"%s\\\": \\\"%s\\\"\" % (txt, translation))", "response": "Translate a single string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npad the vocabulary to a multiple of pad tokens.", "response": "def pad_vocabulary(self, vocab, pad):\n \"\"\"\n Pads vocabulary to a multiple of 'pad' tokens.\n\n :param vocab: list with vocabulary\n :param pad: integer\n \"\"\"\n vocab_size = len(vocab)\n padded_vocab_size = (vocab_size + pad - 1) // pad * pad\n for i in range(0, padded_vocab_size - vocab_size):\n token = f'madeupword{i:04d}'\n vocab.append(token)\n assert len(vocab) % pad == 0"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef segment(self, line):\n line = line.strip().split()\n entry = [self.tok2idx[i] for i in line]\n entry = [config.BOS] + entry + [config.EOS]\n return entry", "response": "Tokenizes a single sentence and adds special BOS and EOS tokens."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef detokenize(self, inputs, delim=' '):\n detok = delim.join([self.idx2tok[idx] for idx in inputs])\n detok = detok.replace(self.separator + ' ', '')\n detok = detok.replace(self.separator, '')\n\n detok = detok.replace(config.BOS_TOKEN, '')\n detok = detok.replace(config.EOS_TOKEN, '')\n detok = detok.replace(config.PAD_TOKEN, '')\n detok = detok.strip()\n return detok", "response": "Detokenizes a single sentence and removes token separator characters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef forward(self, x):\n last_inner = getattr(self, self.inner_blocks[-1])(x[-1])\n results = []\n results.append(getattr(self, self.layer_blocks[-1])(last_inner))\n for feature, inner_block, layer_block in zip(\n x[:-1][::-1], self.inner_blocks[:-1][::-1], self.layer_blocks[:-1][::-1]\n ):\n if not inner_block:\n continue\n inner_top_down = F.interpolate(last_inner, scale_factor=2, mode=\"nearest\")\n inner_lateral = getattr(self, inner_block)(feature)\n # TODO use size instead of scale to make it robust to different sizes\n # inner_top_down = F.upsample(last_inner, size=inner_lateral.shape[-2:],\n # mode='bilinear', align_corners=False)\n last_inner = inner_lateral + inner_top_down\n results.insert(0, getattr(self, layer_block)(last_inner))\n\n if isinstance(self.top_blocks, LastLevelP6P7):\n last_results = self.top_blocks(x[-1], results[-1])\n results.extend(last_results)\n elif isinstance(self.top_blocks, LastLevelMaxPool):\n last_results = self.top_blocks(results[-1])\n results.extend(last_results)\n\n return tuple(results)", "response": "Forward the feature maps to the next layer."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef upload_benchmark_run(self, dataset_name, table_name, run_id):\n expected_file = os.path.join(\n self._logging_dir, logger.BENCHMARK_RUN_LOG_FILE_NAME)\n with tf.gfile.GFile(expected_file) as f:\n benchmark_json = json.load(f)\n benchmark_json[\"model_id\"] = run_id\n table_ref = self._bq_client.dataset(dataset_name).table(table_name)\n errors = self._bq_client.insert_rows_json(table_ref, [benchmark_json])\n if errors:\n tf.logging.error(\n \"Failed to upload benchmark info to bigquery: {}\".format(errors))", "response": "Uploads benchmark run information to Bigquery."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupload metric information to Bigquery.", "response": "def upload_metric(self, dataset_name, table_name, run_id):\n \"\"\"Upload metric information to Bigquery.\n\n Args:\n dataset_name: string, the name of bigquery dataset where the data will be\n uploaded.\n table_name: string, the name of bigquery table under the dataset where\n the metric data will be uploaded. This is different from the\n benchmark_run table.\n run_id: string, a unique ID that will be attached to the data, usually\n this is a UUID4 format. This should be the same as the benchmark run_id.\n \"\"\"\n expected_file = os.path.join(\n self._logging_dir, logger.METRIC_LOG_FILE_NAME)\n with tf.gfile.GFile(expected_file) as f:\n lines = f.readlines()\n metrics = []\n for line in filter(lambda l: l.strip(), lines):\n metric = json.loads(line)\n metric[\"run_id\"] = run_id\n metrics.append(metric)\n table_ref = self._bq_client.dataset(dataset_name).table(table_name)\n errors = self._bq_client.insert_rows_json(table_ref, metrics)\n if errors:\n tf.logging.error(\n \"Failed to upload benchmark info to bigquery: {}\".format(errors))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef bleu_wrapper(ref_filename, hyp_filename, case_sensitive=False):\n ref_lines = tf.gfile.Open(ref_filename).read().strip().splitlines()\n hyp_lines = tf.gfile.Open(hyp_filename).read().strip().splitlines()\n\n if len(ref_lines) != len(hyp_lines):\n raise ValueError(\"Reference and translation files have different number of \"\n \"lines.\")\n if not case_sensitive:\n ref_lines = [x.lower() for x in ref_lines]\n hyp_lines = [x.lower() for x in hyp_lines]\n ref_tokens = [bleu_tokenize(x) for x in ref_lines]\n hyp_tokens = [bleu_tokenize(x) for x in hyp_lines]\n return metrics.compute_bleu(ref_tokens, hyp_tokens) * 100", "response": "Compute BLEU for two files."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef play_match(black_model, white_model, games, sgf_dir):\n with utils.logged_timer(\"Loading weights\"):\n black_net = dual_net.DualNetwork(black_model)\n white_net = dual_net.DualNetwork(white_model)\n\n readouts = FLAGS.num_readouts\n\n black = MCTSPlayer(black_net, two_player_mode=True)\n white = MCTSPlayer(white_net, two_player_mode=True)\n\n black_name = os.path.basename(black_net.save_file)\n white_name = os.path.basename(white_net.save_file)\n\n for i in range(games):\n num_move = 0 # The move number of the current game\n\n for player in [black, white]:\n player.initialize_game()\n first_node = player.root.select_leaf()\n prob, val = player.network.run(first_node.position)\n first_node.incorporate_results(prob, val, first_node)\n\n while True:\n start = time.time()\n active = white if num_move % 2 else black\n inactive = black if num_move % 2 else white\n\n current_readouts = active.root.N\n while active.root.N < current_readouts + readouts:\n active.tree_search()\n\n # print some stats on the search\n if FLAGS.verbose >= 3:\n print(active.root.position)\n\n # First, check the roots for hopeless games.\n if active.should_resign(): # Force resign\n active.set_result(-1 *\n active.root.position.to_play, was_resign=True)\n inactive.set_result(\n active.root.position.to_play, was_resign=True)\n\n if active.is_done():\n fname = \"{:d}-{:s}-vs-{:s}-{:d}.sgf\".format(int(time.time()),\n white_name, black_name, i)\n active.set_result(active.root.position.result(), was_resign=False)\n with gfile.GFile(os.path.join(sgf_dir, fname), 'w') as _file:\n sgfstr = sgf_wrapper.make_sgf(active.position.recent,\n active.result_string, black_name=black_name,\n white_name=white_name)\n _file.write(sgfstr)\n print(\"Finished game\", i, active.result_string)\n break\n\n move = active.pick_move()\n active.play_move(move)\n inactive.play_move(move)\n\n dur = time.time() - start\n num_move += 1\n\n if (FLAGS.verbose > 1) or (FLAGS.verbose == 1 and num_move % 10 == 9):\n timeper = (dur / readouts) * 100.0\n print(active.root.position)\n print(\"%d: %d readouts, %.3f s/100. (%.2f sec)\" % (num_move,\n readouts,\n timeper,\n dur))", "response": "Plays matches between two neural nets."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef main(argv):\n _, black_model, white_model = argv\n utils.ensure_dir_exists(FLAGS.eval_sgf_dir)\n play_match(black_model, white_model, FLAGS.num_evaluation_games, FLAGS.eval_sgf_dir)", "response": "Play matches between two neural nets."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main():\n args = parse_args()\n\n c = Counter()\n skipped = 0\n valid = 0\n data1 = []\n data2 = []\n\n with open(args.file1) as f1, open(args.file2) as f2:\n for idx, lines in enumerate(zip(f1, f2)):\n line1, line2 = lines\n if idx % 100000 == 1:\n print('Processed {} lines'.format(idx))\n try:\n line1.encode('latin1')\n line2.encode('latin1')\n except UnicodeEncodeError:\n skipped += 1\n else:\n data1.append(line1)\n data2.append(line2)\n valid += 1\n c.update(line1)\n\n ratio = valid / (skipped + valid)\n print('Skipped: {}, Valid: {}, Valid ratio {}'.format(skipped, valid, ratio))\n print('Character frequency:', c)\n\n save_output(args.file1, data1)\n save_output(args.file2, data2)", "response": "This function is used to get the most likely unicode glyphs and pairs of sentences which can t be decoded by latin - 1 encoder."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nencoding a set of proposals with respect to some reference boxes.", "response": "def encode(self, reference_boxes, proposals):\n \"\"\"\n Encode a set of proposals with respect to some\n reference boxes\n\n Arguments:\n reference_boxes (Tensor): reference boxes\n proposals (Tensor): boxes to be encoded\n \"\"\"\n\n TO_REMOVE = 1 # TODO remove\n ex_widths = proposals[:, 2] - proposals[:, 0] + TO_REMOVE\n ex_heights = proposals[:, 3] - proposals[:, 1] + TO_REMOVE\n ex_ctr_x = proposals[:, 0] + 0.5 * ex_widths\n ex_ctr_y = proposals[:, 1] + 0.5 * ex_heights\n\n gt_widths = reference_boxes[:, 2] - reference_boxes[:, 0] + TO_REMOVE\n gt_heights = reference_boxes[:, 3] - reference_boxes[:, 1] + TO_REMOVE\n gt_ctr_x = reference_boxes[:, 0] + 0.5 * gt_widths\n gt_ctr_y = reference_boxes[:, 1] + 0.5 * gt_heights\n\n wx, wy, ww, wh = self.weights\n targets_dx = wx * (gt_ctr_x - ex_ctr_x) / ex_widths\n targets_dy = wy * (gt_ctr_y - ex_ctr_y) / ex_heights\n targets_dw = ww * torch.log(gt_widths / ex_widths)\n targets_dh = wh * torch.log(gt_heights / ex_heights)\n\n targets = torch.stack((targets_dx, targets_dy, targets_dw, targets_dh), dim=1)\n return targets"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlaunches an evaluator job.", "response": "def launch_eval_job(tag, m1_path, m2_path, job_name, completions):\n \"\"\"Launches an evaluator job.\n tag: name for this eval job (used as top level folder name)\n m1_path, m2_path: full gs:// paths to the .pb files to match up\n job_name: string, appended to the container, used to differentiate the job\n names (e.g. 'minigo-cc-evaluator-v5-123-v7-456')\n completions: the number of completions desired (each completion is 2 games)\n \"\"\"\n print()\n if not re.match(r'[a-z0-9-]*$', tag, re.I):\n print(\"{} is not a valid tag\".format(tag))\n return\n\n # TODO: Change to minigo-pub\n sgf_bucket_path = \"sethtroisi-sandbox/experiments/eval/\" + tag\n assert not sgf_bucket_path.startswith(\"gs://\"), bucket_pat\n bucket_path = \"gs://\" + sgf_bucket_path\n\n\n metadata_path = os.path.join(bucket_path, 'metadata')\n assert not gfile.Exists(metadata_path), \"Already exists\"\n\n TS=str(int(time.time()))\n metadata = {\n 'timestamp': TS,\n 'date': datetime.datetime.now().isoformat(' '),\n 'model1': os.path.basename(m1_path),\n 'model2': os.path.basename(m2_path),\n 'model1_path': m1_path,\n 'model2_path': m2_path,\n 'job_name': job_name,\n 'completions': completions,\n 'launch_eval_version' : LAUNCH_EVAL_VERSION,\n }\n\n job_conf, resp_bw, resp_wb = launch_eval.launch_eval_job(\n m1_path, m2_path, job_name, sgf_bucket_path, completions)\n\n if not (resp_bw and resp_wb):\n print(\"launch_eval.py failed\")\n print(job_conf)\n print(resp_bw)\n print(resp_wb)\n print()\n assert False\n\n # Jobs were launched, record metadata to GCS.\n with gfile.GFile(metadata_path, \"w\") as metadata_file:\n json.dump(metadata, metadata_file)\n\n with gfile.GFile(os.path.join(bucket_path, 'commands'), \"w\") as f:\n f.write(str(sys.argv) + \"\\n\")\n\n\n with gfile.GFile(os.path.join(bucket_path, 'job_conf'), \"w\") as f:\n f.write(str(job_conf) + \"\\n\")"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef launch_eval_job(m1_path, m2_path, job_name,\n bucket_name, completions=5, flags_path=None):\n \"\"\"Launches an evaluator job.\n m1_path, m2_path: full gs:// paths to the .pb files to match up\n job_name: string, appended to the container, used to differentiate the job\n names (e.g. 'minigo-cc-evaluator-v5-123-v7-456')\n bucket_name: Where to write the sgfs, passed into the job as $BUCKET_NAME\n completions: the number of completions desired\n flags_path: the path to the eval flagfile to use (if any)\n \"\"\"\n if not all([m1_path, m2_path, job_name, bucket_name]):\n print(\"Provide all of m1_path, m2_path, job_name, and bucket_name \"\n \"params\")\n return\n api_instance = get_api()\n\n raw_job_conf = open(\"cluster/evaluator/cc-evaluator.yaml\").read()\n\n if flags_path:\n os.environ['EVAL_FLAGS_PATH'] = flags_path\n else:\n os.environ['EVAL_FLAGS_PATH'] = \"\"\n os.environ['BUCKET_NAME'] = bucket_name\n os.environ['MODEL_BLACK'] = m1_path\n os.environ['MODEL_WHITE'] = m2_path\n os.environ['JOBNAME'] = job_name\n env_job_conf = os.path.expandvars(raw_job_conf)\n\n job_conf = yaml.load(env_job_conf)\n job_conf['spec']['completions'] = completions\n\n response = api_instance.create_namespaced_job('default', body=job_conf)\n return job_conf, response", "response": "Launches an evaluator job."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef same_run_eval(black_num=0, white_num=0, completions=4):\n if black_num <= 0 or white_num <= 0:\n print(\"Need real model numbers\")\n return\n\n b = fsdb.get_model(black_num)\n w = fsdb.get_model(white_num)\n\n b_model_path = os.path.join(fsdb.models_dir(), b)\n w_model_path = os.path.join(fsdb.models_dir(), w)\n flags_path = fsdb.eval_flags_path()\n\n obj = launch_eval_job(b_model_path + \".pb\",\n w_model_path + \".pb\",\n \"{:d}-{:d}\".format(black_num, white_num),\n bucket_name=flags.FLAGS.bucket_name,\n flags_path=flags_path,\n completions=completions)\n\n # Fire spams the retval to stdout, so...\n return \"{} job launched ok\".format(obj[1].metadata.name)", "response": "Spawns a job matching up two models from the same run"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _append_pairs(new_pairs):\n desired_pairs = restore_pairs() or []\n desired_pairs += new_pairs\n print(\"Adding {} new pairs, queue has {} pairs\".format(len(new_pairs), len(desired_pairs)))\n save_pairs(desired_pairs)", "response": "Add new pairs to the pairlist"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds the top pairs to the working pairlist.", "response": "def add_top_pairs(dry_run=False, pair_now=False):\n \"\"\" Pairs up the top twenty models against each other.\n #1 plays 2,3,4,5, #2 plays 3,4,5,6 etc. for a total of 15*4 matches.\n\n Default behavior is to add the pairs to the working pairlist.\n `pair_now` will immediately create the pairings on the cluster.\n `dry_run` makes it only print the pairings that would be added\n \"\"\"\n top = ratings.top_n(15)\n new_pairs = []\n for idx, t in enumerate(top[:10]):\n new_pairs += [[t[0], o[0]] for o in top[idx+1:idx+5]]\n\n if dry_run:\n print(new_pairs)\n return\n\n if pair_now:\n maybe_enqueue(new_pairs)\n else:\n _append_pairs(new_pairs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef zoo_loop(sgf_dir=None, max_jobs=40):\n desired_pairs = restore_pairs() or []\n random.shuffle(desired_pairs)\n last_model_queued = restore_last_model()\n\n if sgf_dir:\n sgf_dir = os.path.abspath(sgf_dir)\n\n api_instance = get_api()\n toggle = True\n try:\n while True:\n last_model = fsdb.get_latest_pb()[0]\n if last_model_queued < last_model:\n print(\"Adding models {} to {} to be scheduled\".format(\n last_model_queued+1, last_model))\n for m in reversed(range(last_model_queued+1, last_model+1)):\n desired_pairs += make_pairs_for_model(m)\n last_model_queued = last_model\n save_last_model(last_model)\n\n cleanup(api_instance)\n random.shuffle(desired_pairs)\n r = api_instance.list_job_for_all_namespaces()\n if r.items:\n tasks = sum([item.spec.completions for item in r.items])\n else:\n tasks = 0\n if tasks < MAX_TASKS:\n if len(desired_pairs) == 0:\n if sgf_dir:\n if tasks > MIN_TASKS:\n time.sleep(60)\n continue\n print(\"Out of pairs! Syncing new eval games...\")\n ratings.sync(sgf_dir)\n print(\"Updating ratings and getting suggestions...\")\n if toggle:\n print(\"Pairing the top of the table.\")\n add_top_pairs()\n else:\n print(\"Pairing the least-known models.\")\n add_uncertain_pairs()\n toggle = not toggle\n for modelnum, rate in ratings.top_n():\n print(\"{:>30}: {:0.3f} ({:0.3f})\".format(modelnum, rate[0], rate[1]))\n desired_pairs = restore_pairs() or []\n else:\n print(\"Out of pairs. Sleeping ({} remain)\".format(len(r.items)))\n time.sleep(600)\n continue\n\n\n next_pair = desired_pairs.pop()\n failed = maybe_enqueue([next_pair])\n if failed != []:\n desired_pairs.extend(failed)\n save_pairs(sorted(desired_pairs))\n save_last_model(last_model)\n time.sleep(1)\n\n else:\n print(\"{}\\t {} finished / {} requested. \"\n \"({} jobs, {} pairs to be scheduled)\".format(\n time.strftime(\"%I:%M:%S %p\"),\n sum([i.status.succeeded or 0 for i in r.items]),\n tasks, len(r.items), len(desired_pairs)))\n time.sleep(60)\n except:\n print(\"Unfinished pairs:\")\n print(sorted(desired_pairs))\n save_pairs(sorted(desired_pairs))\n save_last_model(last_model)\n raise", "response": "This function loops through the list of possible match jobs and creates new ones and deletes them if they are too long."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cleanup(api_instance=None):\n api = api_instance or get_api()\n r = api.list_job_for_all_namespaces()\n delete_opts = kubernetes.client.V1DeleteOptions(\n propagation_policy=\"Background\")\n for job in r.items:\n if job.status.succeeded == job.spec.completions:\n print(job.metadata.name, \"finished!\")\n api.delete_namespaced_job(\n job.metadata.name, 'default', body=delete_opts)", "response": "Remove completed jobs from the cluster"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert to png and save json with path.", "response": "def convert_coco_stuff_mat(data_dir, out_dir):\n \"\"\"Convert to png and save json with path. This currently only contains\n the segmentation labels for objects+stuff in cocostuff - if we need to\n combine with other labels from original COCO that will be a TODO.\"\"\"\n sets = ['train', 'val']\n categories = []\n json_name = 'coco_stuff_%s.json'\n ann_dict = {}\n for data_set in sets:\n file_list = os.path.join(data_dir, '%s.txt')\n images = []\n with open(file_list % data_set) as f:\n for img_id, img_name in enumerate(f):\n img_name = img_name.replace('coco', 'COCO').strip('\\n')\n image = {}\n mat_file = os.path.join(\n data_dir, 'annotations/%s.mat' % img_name)\n data = h5py.File(mat_file, 'r')\n labelMap = data.get('S')\n if len(categories) == 0:\n labelNames = data.get('names')\n for idx, n in enumerate(labelNames):\n categories.append(\n {\"id\": idx, \"name\": ''.join(chr(i) for i in data[\n n[0]])})\n ann_dict['categories'] = categories\n scipy.misc.imsave(\n os.path.join(data_dir, img_name + '.png'), labelMap)\n image['width'] = labelMap.shape[0]\n image['height'] = labelMap.shape[1]\n image['file_name'] = img_name\n image['seg_file_name'] = img_name\n image['id'] = img_id\n images.append(image)\n ann_dict['images'] = images\n print(\"Num images: %s\" % len(images))\n with open(os.path.join(out_dir, json_name % data_set), 'wb') as outfile:\n outfile.write(json.dumps(ann_dict))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert from cityscapes format to COCO instance seg format - polygons", "response": "def convert_cityscapes_instance_only(\n data_dir, out_dir):\n \"\"\"Convert from cityscapes format to COCO instance seg format - polygons\"\"\"\n sets = [\n 'gtFine_val',\n 'gtFine_train',\n 'gtFine_test',\n\n # 'gtCoarse_train',\n # 'gtCoarse_val',\n # 'gtCoarse_train_extra'\n ]\n ann_dirs = [\n 'gtFine_trainvaltest/gtFine/val',\n 'gtFine_trainvaltest/gtFine/train',\n 'gtFine_trainvaltest/gtFine/test',\n\n # 'gtCoarse/train',\n # 'gtCoarse/train_extra',\n # 'gtCoarse/val'\n ]\n json_name = 'instancesonly_filtered_%s.json'\n ends_in = '%s_polygons.json'\n img_id = 0\n ann_id = 0\n cat_id = 1\n category_dict = {}\n\n category_instancesonly = [\n 'person',\n 'rider',\n 'car',\n 'truck',\n 'bus',\n 'train',\n 'motorcycle',\n 'bicycle',\n ]\n\n for data_set, ann_dir in zip(sets, ann_dirs):\n print('Starting %s' % data_set)\n ann_dict = {}\n images = []\n annotations = []\n ann_dir = os.path.join(data_dir, ann_dir)\n for root, _, files in os.walk(ann_dir):\n for filename in files:\n if filename.endswith(ends_in % data_set.split('_')[0]):\n if len(images) % 50 == 0:\n print(\"Processed %s images, %s annotations\" % (\n len(images), len(annotations)))\n json_ann = json.load(open(os.path.join(root, filename)))\n image = {}\n image['id'] = img_id\n img_id += 1\n\n image['width'] = json_ann['imgWidth']\n image['height'] = json_ann['imgHeight']\n image['file_name'] = filename[:-len(\n ends_in % data_set.split('_')[0])] + 'leftImg8bit.png'\n image['seg_file_name'] = filename[:-len(\n ends_in % data_set.split('_')[0])] + \\\n '%s_instanceIds.png' % data_set.split('_')[0]\n images.append(image)\n\n fullname = os.path.join(root, image['seg_file_name'])\n objects = cs.instances2dict_with_polygons(\n [fullname], verbose=False)[fullname]\n\n for object_cls in objects:\n if object_cls not in category_instancesonly:\n continue # skip non-instance categories\n\n for obj in objects[object_cls]:\n if obj['contours'] == []:\n print('Warning: empty contours.')\n continue # skip non-instance categories\n\n len_p = [len(p) for p in obj['contours']]\n if min(len_p) <= 4:\n print('Warning: invalid contours.')\n continue # skip non-instance categories\n\n ann = {}\n ann['id'] = ann_id\n ann_id += 1\n ann['image_id'] = image['id']\n ann['segmentation'] = obj['contours']\n\n if object_cls not in category_dict:\n category_dict[object_cls] = cat_id\n cat_id += 1\n ann['category_id'] = category_dict[object_cls]\n ann['iscrowd'] = 0\n ann['area'] = obj['pixelCount']\n ann['bbox'] = bboxs_util.xyxy_to_xywh(\n segms_util.polys_to_boxes(\n [ann['segmentation']])).tolist()[0]\n\n annotations.append(ann)\n\n ann_dict['images'] = images\n categories = [{\"id\": category_dict[name], \"name\": name} for name in\n category_dict]\n ann_dict['categories'] = categories\n ann_dict['annotations'] = annotations\n print(\"Num categories: %s\" % len(categories))\n print(\"Num images: %s\" % len(images))\n print(\"Num annotations: %s\" % len(annotations))\n with open(os.path.join(out_dir, json_name % data_set), 'w') as outfile:\n outfile.write(json.dumps(ann_dict))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the path to the MG file for a given model run and model number.", "response": "def get_mg_path(model_run, model_num):\n \"\"\"\n model_run = integer, e.g. 15, 16, corresponding to the v-number\n model_num = integer, e.g 939, for the model number in that run\n \"\"\"\n fsdb.switch_base(\"minigo-pub/v{:d}-19x19\".format(model_run))\n model = fsdb.get_model(model_num)\n return os.path.join(fsdb.models_dir(), model)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef make_single_array(ds, batch_size=8*1024):\n if isinstance(ds.output_types, tuple) or isinstance(ds.output_shapes, tuple):\n raise ValueError('Dataset must have a single type and shape')\n nshapes = len(ds.output_shapes)\n if nshapes > 0:\n raise ValueError('Dataset must be comprised of scalars (TensorShape=[])')\n batches = []\n with tf.Session() as sess:\n ds = ds.batch(batch_size)\n iterator = ds.make_initializable_iterator()\n sess.run(iterator.initializer)\n get_next = iterator.get_next()\n with tqdm(desc='Elements', unit_scale=1) as pbar:\n try:\n while True:\n batches.append(sess.run(get_next))\n pbar.update(len(batches[-1]))\n except tf.errors.OutOfRangeError:\n pass\n if batches:\n return np.concatenate(batches)\n return np.array([], dtype=ds.output_types.as_numpy_dtype)", "response": "Create a single numpy array from a TF Dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _histogram_move_keys_by_game(sess, ds, batch_size=8*1024):\n ds = ds.batch(batch_size)\n # Turns 'g_0000001234_m_133' into 'g_0000001234'\n ds = ds.map(lambda x: tf.strings.substr(x, 0, 12))\n iterator = ds.make_initializable_iterator()\n sess.run(iterator.initializer)\n get_next = iterator.get_next()\n h = collections.Counter()\n try:\n while True:\n h.update(sess.run(get_next))\n except tf.errors.OutOfRangeError:\n pass\n # NOTE: Cannot be truly sure the count is right till the end.\n return h", "response": "Given dataset of key names return histogram of moves and game players."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nturns keys of a Bigtable dataset into an array.", "response": "def _game_keys_as_array(ds):\n \"\"\"Turn keys of a Bigtable dataset into an array.\n\n Take g_GGG_m_MMM and create GGG.MMM numbers.\n\n Valuable when visualizing the distribution of a given dataset in\n the game keyspace.\n \"\"\"\n ds = ds.map(lambda row_key, cell: row_key)\n # want 'g_0000001234_m_133' is '0000001234.133' and so forth\n ds = ds.map(lambda x:\n tf.strings.to_number(tf.strings.substr(x, 2, 10) +\n '.' +\n tf.strings.substr(x, 15, 3),\n out_type=tf.float64))\n return make_single_array(ds)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndelete the given row keys from the given Bigtable.", "response": "def _delete_rows(args):\n \"\"\"Delete the given row keys from the given Bigtable.\n\n The args are (BigtableSpec, row_keys), but are passed\n as a single argument in order to work with\n multiprocessing.Pool.map. This is also the reason why this is a\n top-level function instead of a method.\n\n \"\"\"\n btspec, row_keys = args\n bt_table = bigtable.Client(btspec.project).instance(\n btspec.instance).table(btspec.table)\n rows = [bt_table.row(k) for k in row_keys]\n for r in rows:\n r.delete()\n bt_table.mutate_rows(rows)\n return row_keys"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting the metadata cell used to block until enough new games have been played.", "response": "def set_fresh_watermark(game_queue, count_from, window_size,\n fresh_fraction=0.05, minimum_fresh=20000):\n \"\"\"Sets the metadata cell used to block until some quantity of games have been played.\n\n This sets the 'freshness mark' on the `game_queue`, used to block training\n until enough new games have been played. The number of fresh games required\n is the larger of:\n - The fraction of the total window size\n - The `minimum_fresh` parameter\n The number of games required can be indexed from the 'count_from' parameter.\n Args:\n game_queue: A GameQueue object, on whose backing table will be modified.\n count_from: the index of the game to compute the increment from\n window_size: an integer indicating how many past games are considered\n fresh_fraction: a float in (0,1] indicating the fraction of games to wait for\n minimum_fresh: an integer indicating the lower bound on the number of new\n games.\n \"\"\"\n already_played = game_queue.latest_game_number - count_from\n print(\"== already_played: \", already_played, flush=True)\n if window_size > count_from: # How to handle the case when the window is not yet 'full'\n game_queue.require_fresh_games(int(minimum_fresh * .9))\n else:\n num_to_play = max(0, math.ceil(window_size * .9 * fresh_fraction) - already_played)\n print(\"== Num to play: \", num_to_play, flush=True)\n game_queue.require_fresh_games(num_to_play)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a dataset of serialized TFExamples from the last n games.", "response": "def get_unparsed_moves_from_last_n_games(games, games_nr, n,\n moves=2**21,\n shuffle=True,\n column_family=TFEXAMPLE,\n column='example',\n values_only=True):\n \"\"\"Get a dataset of serialized TFExamples from the last N games.\n\n Args:\n games, games_nr: GameQueues of the regular selfplay and calibration\n (aka 'no resign') games to sample from.\n n: an integer indicating how many past games should be sourced.\n moves: an integer indicating how many moves should be sampled\n from those N games.\n column_family: name of the column family containing move examples.\n column: name of the column containing move examples.\n shuffle: if True, shuffle the selected move examples.\n values_only: if True, return only column values, no row keys.\n\n Returns:\n A dataset containing no more than `moves` examples, sampled\n randomly from the last `n` games in the table.\n \"\"\"\n # The prefixes and suffixes below have the following meanings:\n # ct_: count\n # fr_: fraction\n # _r: resign (ordinary)\n # _nr: no-resign\n ct_r, ct_nr = 9, 1\n ct_total = ct_r + ct_nr\n fr_r = ct_r / ct_total\n fr_nr = ct_nr / ct_total\n resign = games.moves_from_last_n_games(\n math.ceil(n * fr_r),\n math.ceil(moves * fr_r),\n shuffle,\n column_family, column)\n no_resign = games_nr.moves_from_last_n_games(\n math.floor(n * fr_nr),\n math.floor(moves * fr_nr),\n shuffle,\n column_family, column)\n selection = np.array([0] * ct_r + [1] * ct_nr, dtype=np.int64)\n choice = tf.data.Dataset.from_tensor_slices(selection).repeat().take(moves)\n ds = tf.contrib.data.choose_from_datasets([resign, no_resign], choice)\n if shuffle:\n ds = ds.shuffle(len(selection) * 2)\n if values_only:\n ds = ds.map(lambda row_name, s: s)\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef count_elements_in_dataset(ds, batch_size=1*1024, parallel_batch=8):\n with tf.Session() as sess:\n dsc = ds.apply(tf.contrib.data.enumerate_dataset())\n dsc = dsc.apply(\n tf.contrib.data.map_and_batch(lambda c, v: c, batch_size,\n num_parallel_batches=parallel_batch))\n iterator = dsc.make_initializable_iterator()\n sess.run(iterator.initializer)\n get_next = iterator.get_next()\n counted = 0\n try:\n while True:\n # The numbers in the tensors are 0-based indicies,\n # so add 1 to get the number counted.\n counted = sess.run(tf.reduce_max(get_next)) + 1\n utils.dbg('Counted so far: %d' % counted)\n except tf.errors.OutOfRangeError:\n pass\n utils.dbg('Counted total: %d' % counted)\n return counted", "response": "Count and return all the elements in the given dataset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating the table underlying the queue.", "response": "def create(self):\n \"\"\"Create the table underlying the queue.\n\n Create the 'metadata' and 'tfexample' column families\n and their properties.\n \"\"\"\n if self.bt_table.exists():\n utils.dbg('Table already exists')\n return\n\n max_versions_rule = bigtable_column_family.MaxVersionsGCRule(1)\n self.bt_table.create(column_families={\n METADATA: max_versions_rule,\n TFEXAMPLE: max_versions_rule})"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngive a range of games return the games sorted by time.", "response": "def games_by_time(self, start_game, end_game):\n \"\"\"Given a range of games, return the games sorted by time.\n\n Returns [(time, game_number), ...]\n\n The time will be a `datetime.datetime` and the game\n number is the integer used as the basis of the row ID.\n\n Note that when a cluster of self-play nodes are writing\n concurrently, the game numbers may be out of order.\n \"\"\"\n move_count = b'move_count'\n rows = self.bt_table.read_rows(\n ROWCOUNT_PREFIX.format(start_game),\n ROWCOUNT_PREFIX.format(end_game),\n filter_=bigtable_row_filters.ColumnRangeFilter(\n METADATA, move_count, move_count))\n\n def parse(r):\n rk = str(r.row_key, 'utf-8')\n game = _game_from_counter.match(rk).groups()[0]\n return (r.cells[METADATA][move_count][0].timestamp, game)\n return sorted([parse(r) for r in rows], key=operator.itemgetter(0))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef delete_row_range(self, format_str, start_game, end_game):\n row_keys = make_single_array(\n self.tf_table.keys_by_range_dataset(\n format_str.format(start_game),\n format_str.format(end_game)))\n row_keys = list(row_keys)\n if not row_keys:\n utils.dbg('No rows left for games %d..%d' % (\n start_game, end_game))\n return\n utils.dbg('Deleting %d rows: %s..%s' % (\n len(row_keys), row_keys[0], row_keys[-1]))\n\n # Reverse the keys so that the queue is left in a more\n # sensible end state if you change your mind (say, due to a\n # mistake in the timestamp) and abort the process: there will\n # be a bit trimmed from the end, rather than a bit\n # trimmed out of the middle.\n row_keys.reverse()\n total_keys = len(row_keys)\n utils.dbg('Deleting total of %d keys' % total_keys)\n concurrency = min(MAX_BT_CONCURRENCY,\n multiprocessing.cpu_count() * 2)\n with multiprocessing.Pool(processes=concurrency) as pool:\n batches = []\n with tqdm(desc='Keys', unit_scale=2, total=total_keys) as pbar:\n for b in utils.iter_chunks(bigtable.row.MAX_MUTATIONS,\n row_keys):\n pbar.update(len(b))\n batches.append((self.btspec, b))\n if len(batches) >= concurrency:\n pool.map(_delete_rows, batches)\n batches = []\n pool.map(_delete_rows, batches)\n batches = []", "response": "Delete rows related to the given game range."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef trim_games_since(self, t, max_games=500000):\n latest = self.latest_game_number\n earliest = int(latest - max_games)\n gbt = self.games_by_time(earliest, latest)\n if not gbt:\n utils.dbg('No games between %d and %d' % (earliest, latest))\n return\n most_recent = gbt[-1]\n if isinstance(t, datetime.timedelta):\n target = most_recent[0] - t\n else:\n target = t\n i = bisect.bisect_right(gbt, (target,))\n if i >= len(gbt):\n utils.dbg('Last game is already at %s' % gbt[-1][0])\n return\n when, which = gbt[i]\n utils.dbg('Most recent: %s %s' % most_recent)\n utils.dbg(' Target: %s %s' % (when, which))\n which = int(which)\n self.delete_row_range(ROW_PREFIX, which, latest)\n self.delete_row_range(ROWCOUNT_PREFIX, which, latest)\n self.latest_game_number = which", "response": "Trim off the games since the given time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a range of games return the bleakest moves.", "response": "def bleakest_moves(self, start_game, end_game):\n \"\"\"Given a range of games, return the bleakest moves.\n\n Returns a list of (game, move, q) sorted by q.\n \"\"\"\n bleak = b'bleakest_q'\n rows = self.bt_table.read_rows(\n ROW_PREFIX.format(start_game),\n ROW_PREFIX.format(end_game),\n filter_=bigtable_row_filters.ColumnRangeFilter(\n METADATA, bleak, bleak))\n\n def parse(r):\n rk = str(r.row_key, 'utf-8')\n g, m = _game_row_key.match(rk).groups()\n q = r.cell_value(METADATA, bleak)\n return int(g), int(m), float(q)\n return sorted([parse(r) for r in rows], key=operator.itemgetter(2))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrequiring a given number of fresh games to be played. Args: number_fresh: integer, number of new fresh games needed Increments the cell `table_state=metadata:wait_for_game_number` by the given number of games. This will cause `self.wait_for_fresh_games()` to block until the game counter has reached this number.", "response": "def require_fresh_games(self, number_fresh):\n \"\"\"Require a given number of fresh games to be played.\n\n Args:\n number_fresh: integer, number of new fresh games needed\n\n Increments the cell `table_state=metadata:wait_for_game_number`\n by the given number of games. This will cause\n `self.wait_for_fresh_games()` to block until the game\n counter has reached this number.\n \"\"\"\n latest = self.latest_game_number\n table_state = self.bt_table.row(TABLE_STATE)\n table_state.set_cell(METADATA, WAIT_CELL, int(latest + number_fresh))\n table_state.commit()\n print(\"== Setting wait cell to \", int(latest + number_fresh), flush=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef wait_for_fresh_games(self, poll_interval=15.0):\n wait_until_game = self.read_wait_cell()\n if not wait_until_game:\n return\n latest_game = self.latest_game_number\n last_latest = latest_game\n while latest_game < wait_until_game:\n utils.dbg('Latest game {} not yet at required game {} '\n '(+{}, {:0.3f} games/sec)'.format(\n latest_game,\n wait_until_game,\n latest_game - last_latest,\n (latest_game - last_latest) / poll_interval\n ))\n time.sleep(poll_interval)\n last_latest = latest_game\n latest_game = self.latest_game_number", "response": "Block the caller until required new games have been played."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread the value of the cell holding the wait value", "response": "def read_wait_cell(self):\n \"\"\"Read the value of the cell holding the 'wait' value,\n\n Returns the int value of whatever it has, or None if the cell doesn't\n exist.\n \"\"\"\n\n table_state = self.bt_table.read_row(\n TABLE_STATE,\n filter_=bigtable_row_filters.ColumnRangeFilter(\n METADATA, WAIT_CELL, WAIT_CELL))\n if table_state is None:\n utils.dbg('No waiting for new games needed; '\n 'wait_for_game_number column not in table_state')\n return None\n value = table_state.cell_value(METADATA, WAIT_CELL)\n if not value:\n utils.dbg('No waiting for new games needed; '\n 'no value in wait_for_game_number cell '\n 'in table_state')\n return None\n return cbt_intvalue(value)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncounting the total number of moves in a game range.", "response": "def count_moves_in_game_range(self, game_begin, game_end):\n \"\"\"Count the total moves in a game range.\n\n Args:\n game_begin: integer, starting game\n game_end: integer, ending game\n\n Uses the `ct_` keyspace for rapid move summary.\n \"\"\"\n rows = self.bt_table.read_rows(\n ROWCOUNT_PREFIX.format(game_begin),\n ROWCOUNT_PREFIX.format(game_end),\n filter_=bigtable_row_filters.ColumnRangeFilter(\n METADATA, MOVE_COUNT, MOVE_COUNT))\n return sum([int(r.cell_value(METADATA, MOVE_COUNT)) for r in rows])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dataset of samples and shuffled moves from the given game range.", "response": "def moves_from_games(self, start_game, end_game, moves, shuffle,\n column_family, column):\n \"\"\"Dataset of samples and/or shuffled moves from game range.\n\n Args:\n n: an integer indicating how many past games should be sourced.\n moves: an integer indicating how many moves should be sampled\n from those N games.\n column_family: name of the column family containing move examples.\n column: name of the column containing move examples.\n shuffle: if True, shuffle the selected move examples.\n\n Returns:\n A dataset containing no more than `moves` examples, sampled\n randomly from the last `n` games in the table.\n \"\"\"\n start_row = ROW_PREFIX.format(start_game)\n end_row = ROW_PREFIX.format(end_game)\n # NOTE: Choose a probability high enough to guarantee at least the\n # required number of moves, by using a slightly lower estimate\n # of the total moves, then trimming the result.\n total_moves = self.count_moves_in_game_range(start_game, end_game)\n probability = moves / (total_moves * 0.99)\n utils.dbg('Row range: %s - %s; total moves: %d; probability %.3f; moves %d' % (\n start_row, end_row, total_moves, probability, moves))\n ds = self.tf_table.parallel_scan_range(start_row, end_row,\n probability=probability,\n columns=[(column_family, column)])\n if shuffle:\n utils.dbg('Doing a complete shuffle of %d moves' % moves)\n ds = ds.shuffle(moves)\n ds = ds.take(moves)\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef moves_from_last_n_games(self, n, moves, shuffle,\n column_family, column):\n \"\"\"Randomly choose a given number of moves from the last n games.\n\n Args:\n n: number of games at the end of this GameQueue to source.\n moves: number of moves to be sampled from `n` games.\n shuffle: if True, shuffle the selected moves.\n column_family: name of the column family containing move examples.\n column: name of the column containing move examples.\n\n Returns:\n a dataset containing the selected moves.\n \"\"\"\n self.wait_for_fresh_games()\n latest_game = self.latest_game_number\n utils.dbg('Latest game in %s: %s' % (self.btspec.table, latest_game))\n if latest_game == 0:\n raise ValueError('Cannot find a latest game in the table')\n\n start = int(max(0, latest_game - n))\n ds = self.moves_from_games(start, latest_game, moves, shuffle,\n column_family, column)\n return ds", "response": "Randomly choose a given number of moves from the last n games."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _write_move_counts(self, sess, h):\n def gen():\n for k, v in h.items():\n # The keys in the histogram may be of type 'bytes'\n k = str(k, 'utf-8')\n vs = str(v)\n yield (k.replace('g_', 'ct_') + '_%d' % v, vs)\n yield (k + '_m_000', vs)\n mc = tf.data.Dataset.from_generator(gen, (tf.string, tf.string))\n wr_op = self.tf_table.write(mc,\n column_families=[METADATA],\n columns=[MOVE_COUNT])\n sess.run(wr_op)", "response": "Add move counts from the given histogram to the table."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses to update the move_count cells for older games.", "response": "def update_move_counts(self, start_game, end_game, interval=1000):\n \"\"\"Used to update the move_count cell for older games.\n\n Should not be needed except for backfill or repair.\n\n move_count cells will be updated in both g__m_000 rows\n and ct__ rows.\n \"\"\"\n for g in range(start_game, end_game, interval):\n with tf.Session() as sess:\n start_row = ROW_PREFIX.format(g)\n end_row = ROW_PREFIX.format(g + interval)\n print('Range:', start_row, end_row)\n start_time = time.time()\n ds = self.tf_table.keys_by_range_dataset(start_row, end_row)\n h = _histogram_move_keys_by_game(sess, ds)\n self._write_move_counts(sess, h)\n end_time = time.time()\n elapsed = end_time - start_time\n print(' games/sec:', len(h)/elapsed)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_filenames(is_training, data_dir):\n if is_training:\n return [\n os.path.join(data_dir, 'train-%05d-of-01024' % i)\n for i in range(_NUM_TRAIN_FILES)]\n else:\n return [\n os.path.join(data_dir, 'validation-%05d-of-00128' % i)\n for i in range(128)]", "response": "Return filenames for dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nparses an example proto containing a training example of an image.", "response": "def _parse_example_proto(example_serialized):\n \"\"\"Parses an Example proto containing a training example of an image.\n\n The output of the build_image_data.py image preprocessing script is a dataset\n containing serialized Example protocol buffers. Each Example proto contains\n the following fields (values are included as examples):\n\n image/height: 462\n image/width: 581\n image/colorspace: 'RGB'\n image/channels: 3\n image/class/label: 615\n image/class/synset: 'n03623198'\n image/class/text: 'knee pad'\n image/object/bbox/xmin: 0.1\n image/object/bbox/xmax: 0.9\n image/object/bbox/ymin: 0.2\n image/object/bbox/ymax: 0.6\n image/object/bbox/label: 615\n image/format: 'JPEG'\n image/filename: 'ILSVRC2012_val_00041207.JPEG'\n image/encoded: \n\n Args:\n example_serialized: scalar Tensor tf.string containing a serialized\n Example protocol buffer.\n\n Returns:\n image_buffer: Tensor tf.string containing the contents of a JPEG file.\n label: Tensor tf.int32 containing the label.\n bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]\n where each coordinate is [0, 1) and the coordinates are arranged as\n [ymin, xmin, ymax, xmax].\n \"\"\"\n # Dense features in Example proto.\n feature_map = {\n 'image/encoded': tf.FixedLenFeature([], dtype=tf.string,\n default_value=''),\n 'image/class/label': tf.FixedLenFeature([], dtype=tf.int64,\n default_value=-1),\n 'image/class/text': tf.FixedLenFeature([], dtype=tf.string,\n default_value=''),\n }\n sparse_float32 = tf.VarLenFeature(dtype=tf.float32)\n # Sparse features in Example proto.\n feature_map.update(\n {k: sparse_float32 for k in ['image/object/bbox/xmin',\n 'image/object/bbox/ymin',\n 'image/object/bbox/xmax',\n 'image/object/bbox/ymax']})\n\n features = tf.parse_single_example(example_serialized, feature_map)\n label = tf.cast(features['image/class/label'], dtype=tf.int32)\n\n return features['image/encoded'], label"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_record(raw_record, is_training, dtype):\n image_buffer, label = _parse_example_proto(raw_record)\n\n image = imagenet_preprocessing.preprocess_image(\n image_buffer=image_buffer,\n output_height=_DEFAULT_IMAGE_SIZE,\n output_width=_DEFAULT_IMAGE_SIZE,\n num_channels=_NUM_CHANNELS,\n is_training=is_training)\n image = tf.cast(image, dtype)\n\n return image, label", "response": "Parses a record containing a training example of an image."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef input_fn(is_training, data_dir, batch_size, num_epochs=1, num_gpus=None,\n dtype=tf.float32):\n \"\"\"Input function which provides batches for train or eval.\n\n Args:\n is_training: A boolean denoting whether the input is for training.\n data_dir: The directory containing the input data.\n batch_size: The number of samples per batch.\n num_epochs: The number of epochs to repeat the dataset.\n num_gpus: The number of gpus used for training.\n dtype: Data type to use for images/features\n\n Returns:\n A dataset that can be used for iteration.\n \"\"\"\n mlperf_log.resnet_print(key=mlperf_log.INPUT_ORDER)\n filenames = get_filenames(is_training, data_dir)\n dataset = tf.data.Dataset.from_tensor_slices(filenames)\n\n if is_training:\n # Shuffle the input files\n dataset = dataset.shuffle(buffer_size=_NUM_TRAIN_FILES)\n\n # Convert to individual records\n dataset = dataset.flat_map(tf.data.TFRecordDataset)\n\n return resnet_run_loop.process_record_dataset(\n dataset=dataset,\n is_training=is_training,\n batch_size=batch_size,\n shuffle_buffer=_SHUFFLE_BUFFER,\n parse_record_fn=parse_record,\n num_epochs=num_epochs,\n num_gpus=num_gpus,\n examples_per_epoch=_NUM_IMAGES['train'] if is_training else None,\n dtype=dtype\n )", "response": "Input function which provides batches for training or evaluation."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nretrieving the size of each block_layer in the ResNet model. The number of block layers used for the Resnet model varies according to the size of the model. This helper grabs the layer set we want, throwing an error if a non-standard size has been selected. Args: resnet_size: The number of convolutional layers needed in the model. Returns: A list of block sizes to use in building the model. Raises: KeyError: if invalid resnet_size is received.", "response": "def _get_block_sizes(resnet_size):\n \"\"\"Retrieve the size of each block_layer in the ResNet model.\n\n The number of block layers used for the Resnet model varies according\n to the size of the model. This helper grabs the layer set we want, throwing\n an error if a non-standard size has been selected.\n\n Args:\n resnet_size: The number of convolutional layers needed in the model.\n\n Returns:\n A list of block sizes to use in building the model.\n\n Raises:\n KeyError: if invalid resnet_size is received.\n \"\"\"\n choices = {\n 18: [2, 2, 2, 2],\n 34: [3, 4, 6, 3],\n 50: [3, 4, 6, 3],\n 101: [3, 4, 23, 3],\n 152: [3, 8, 36, 3],\n 200: [3, 24, 36, 3]\n }\n\n try:\n return choices[resnet_size]\n except KeyError:\n err = ('Could not find layers for selected Resnet size.\\n'\n 'Size received: {}; sizes allowed: {}.'.format(\n resnet_size, choices.keys()))\n raise ValueError(err)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrap for MLPerf compliance logging calls.", "response": "def gnmt_print(*args, **kwargs):\n \"\"\"\n Wrapper for MLPerf compliance logging calls.\n All arguments but 'sync' are passed to mlperf_log.gnmt_print function.\n If 'sync' is set to True then the wrapper will synchronize all distributed\n workers. 'sync' should be set to True for all compliance tags that require\n accurate timing (RUN_START, RUN_STOP etc.)\n \"\"\"\n if kwargs.pop('sync'):\n barrier()\n if get_rank() == 0:\n kwargs['stack_offset'] = 2\n mlperf_log.gnmt_print(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninitializing weights and biases of the PyTorch LSTM layer.", "response": "def init_lstm_(lstm, init_weight=0.1):\n \"\"\"\n Initializes weights of LSTM layer.\n Weights and biases are initialized with uniform(-init_weight, init_weight)\n distribution.\n\n :param lstm: instance of torch.nn.LSTM\n :param init_weight: range for the uniform initializer\n \"\"\"\n # Initialize hidden-hidden weights\n init.uniform_(lstm.weight_hh_l0.data, -init_weight, init_weight)\n # Initialize input-hidden weights:\n init.uniform_(lstm.weight_ih_l0.data, -init_weight, init_weight)\n\n # Initialize bias. PyTorch LSTM has two biases, one for input-hidden GEMM\n # and the other for hidden-hidden GEMM. Here input-hidden bias is\n # initialized with uniform distribution and hidden-hidden bias is\n # initialized with zeros.\n init.uniform_(lstm.bias_ih_l0.data, -init_weight, init_weight)\n init.zeros_(lstm.bias_hh_l0.data)\n\n if lstm.bidirectional:\n init.uniform_(lstm.weight_hh_l0_reverse.data, -init_weight, init_weight)\n init.uniform_(lstm.weight_ih_l0_reverse.data, -init_weight, init_weight)\n\n init.uniform_(lstm.bias_ih_l0_reverse.data, -init_weight, init_weight)\n init.zeros_(lstm.bias_hh_l0_reverse.data)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setup_seeds(master_seed, epochs, device):\n if master_seed is None:\n # random master seed, random.SystemRandom() uses /dev/urandom on Unix\n master_seed = random.SystemRandom().randint(0, 2**32 - 1)\n if get_rank() == 0:\n # master seed is reported only from rank=0 worker, it's to avoid\n # confusion, seeds from rank=0 are later broadcasted to other\n # workers\n logging.info(f'Using random master seed: {master_seed}')\n else:\n # master seed was specified from command line\n logging.info(f'Using master seed from command line: {master_seed}')\n\n gnmt_print(key=mlperf_log.RUN_SET_RANDOM_SEED, value=master_seed,\n sync=False)\n\n # initialize seeding RNG\n seeding_rng = random.Random(master_seed)\n\n # generate worker seeds, one seed for every distributed worker\n worker_seeds = generate_seeds(seeding_rng, get_world_size())\n\n # generate seeds for data shuffling, one seed for every epoch\n shuffling_seeds = generate_seeds(seeding_rng, epochs)\n\n # broadcast seeds from rank=0 to other workers\n worker_seeds = broadcast_seeds(worker_seeds, device)\n shuffling_seeds = broadcast_seeds(shuffling_seeds, device)\n return worker_seeds, shuffling_seeds", "response": "Function creates seeds from one master_seed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwork as a temporary distributed barrier, currently pytorch doesn't implement barrier for NCCL backend. Calls all_reduce on dummy tensor and synchronizes with GPU.", "response": "def barrier():\n \"\"\"\n Works as a temporary distributed barrier, currently pytorch\n doesn't implement barrier for NCCL backend.\n Calls all_reduce on dummy tensor and synchronizes with GPU.\n \"\"\"\n if torch.distributed.is_available() and torch.distributed.is_initialized():\n torch.distributed.all_reduce(torch.cuda.FloatTensor(1))\n torch.cuda.synchronize()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets distributed rank or returns zero if distributed is not initialized", "response": "def get_rank():\n \"\"\"\n Gets distributed rank or returns zero if distributed is not initialized.\n \"\"\"\n if torch.distributed.is_available() and torch.distributed.is_initialized():\n rank = torch.distributed.get_rank()\n else:\n rank = 0\n return rank"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_world_size():\n if torch.distributed.is_available() and torch.distributed.is_initialized():\n world_size = torch.distributed.get_world_size()\n else:\n world_size = 1\n return world_size", "response": "Gets total number of workers or returns one if distributed is not initialized"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconfiguring logging. By default logs from all workers are printed to the console, entries are prefixed with \"N: \" where N is the rank of the worker. Logs printed to the console don't include timestaps. Full logs with timestamps are saved to the log_file file.", "response": "def setup_logging(log_file=os.devnull):\n \"\"\"\n Configures logging.\n By default logs from all workers are printed to the console, entries are\n prefixed with \"N: \" where N is the rank of the worker. Logs printed to the\n console don't include timestaps.\n Full logs with timestamps are saved to the log_file file.\n \"\"\"\n class RankFilter(logging.Filter):\n def __init__(self, rank):\n self.rank = rank\n\n def filter(self, record):\n record.rank = self.rank\n return True\n\n rank = get_rank()\n rank_filter = RankFilter(rank)\n\n logging_format = \"%(asctime)s - %(levelname)s - %(rank)s - %(message)s\"\n logging.basicConfig(level=logging.DEBUG,\n format=logging_format,\n datefmt=\"%Y-%m-%d %H:%M:%S\",\n filename=log_file,\n filemode='w')\n console = logging.StreamHandler(sys.stdout)\n console.setLevel(logging.INFO)\n formatter = logging.Formatter('%(rank)s: %(message)s')\n console.setFormatter(formatter)\n logging.getLogger('').addHandler(console)\n logging.getLogger('').addFilter(rank_filter)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the device based on local_rank and returns instance of torch. device.", "response": "def set_device(cuda, local_rank):\n \"\"\"\n Sets device based on local_rank and returns instance of torch.device.\n\n :param cuda: if True: use cuda\n :param local_rank: local rank of the worker\n \"\"\"\n if cuda:\n torch.cuda.set_device(local_rank)\n device = torch.device('cuda')\n else:\n device = torch.device('cpu')\n return device"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes distributed backend. :param cuda: (bool) if True initializes nccl backend, if False initializes gloo backend", "response": "def init_distributed(cuda):\n \"\"\"\n Initializes distributed backend.\n\n :param cuda: (bool) if True initializes nccl backend, if False initializes\n gloo backend\n \"\"\"\n world_size = int(os.environ.get('WORLD_SIZE', 1))\n distributed = (world_size > 1)\n if distributed:\n backend = 'nccl' if cuda else 'gloo'\n dist.init_process_group(backend=backend,\n init_method='env://')\n assert dist.is_initialized()\n return distributed"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting information about execution environment.", "response": "def log_env_info():\n \"\"\"\n Prints information about execution environment.\n \"\"\"\n logging.info('Collecting environment information...')\n env_info = torch.utils.collect_env.get_pretty_env_info()\n logging.info(f'{env_info}')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreducing the average value over all workers.", "response": "def reduce(self, op):\n \"\"\"\n Reduces average value over all workers.\n\n :param op: 'sum' or 'mean', reduction operator\n \"\"\"\n if op not in ('sum', 'mean'):\n raise NotImplementedError\n\n distributed = (get_world_size() > 1)\n if distributed:\n # Backward/forward compatibility around\n # https://github.com/pytorch/pytorch/commit/540ef9b1fc5506369a48491af8a285a686689b36 and\n # https://github.com/pytorch/pytorch/commit/044d00516ccd6572c0d6ab6d54587155b02a3b86\n # To accomodate change in Pytorch's distributed API\n if hasattr(dist, \"get_backend\"):\n _backend = dist.get_backend()\n if hasattr(dist, \"DistBackend\"):\n backend_enum_holder = dist.DistBackend\n else:\n backend_enum_holder = dist.Backend\n else:\n _backend = dist._backend\n backend_enum_holder = dist.dist_backend\n\n cuda = _backend == backend_enum_holder.NCCL\n\n if cuda:\n avg = torch.cuda.FloatTensor([self.avg])\n _sum = torch.cuda.FloatTensor([self.sum])\n else:\n avg = torch.FloatTensor([self.avg])\n _sum = torch.FloatTensor([self.sum])\n\n _reduce_op = dist.reduce_op if hasattr(dist, \"reduce_op\") else dist.ReduceOp\n dist.all_reduce(avg, op=_reduce_op.SUM)\n dist.all_reduce(_sum, op=_reduce_op.SUM)\n self.avg = avg.item()\n self.sum = _sum.item()\n\n if op == 'mean':\n self.avg /= get_world_size()\n self.sum /= get_world_size()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef build_dataset(dataset_list, transforms, dataset_catalog, is_train=True):\n if not isinstance(dataset_list, (list, tuple)):\n raise RuntimeError(\n \"dataset_list should be a list of strings, got {}\".format(dataset_list)\n )\n datasets = []\n total_datasets_size = 0\n for dataset_name in dataset_list:\n data = dataset_catalog.get(dataset_name)\n factory = getattr(D, data[\"factory\"])\n args = data[\"args\"]\n # for COCODataset, we want to remove images without annotations\n # during training\n if data[\"factory\"] == \"COCODataset\":\n args[\"remove_images_without_annotations\"] = is_train\n if data[\"factory\"] == \"PascalVOCDataset\":\n args[\"use_difficult\"] = not is_train\n args[\"transforms\"] = transforms\n # make dataset from factory\n dataset = factory(**args)\n total_datasets_size += len(dataset)\n datasets.append(dataset)\n\n # for testing, return a list of datasets\n if not is_train:\n return datasets, total_datasets_size\n\n # for training, concatenate all datasets into a single one\n dataset = datasets[0]\n if len(datasets) > 1:\n dataset = D.ConcatDataset(datasets)\n\n return [dataset], total_datasets_size", "response": "Builds a single dataset from a list of datasets."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a list of Tensors to a list of ImageLists.", "response": "def to_image_list(tensors, size_divisible=0):\n \"\"\"\n tensors can be an ImageList, a torch.Tensor or\n an iterable of Tensors. It can't be a numpy array.\n When tensors is an iterable of Tensors, it pads\n the Tensors with zeros so that they have the same\n shape\n \"\"\"\n if isinstance(tensors, torch.Tensor) and size_divisible > 0:\n tensors = [tensors]\n\n if isinstance(tensors, ImageList):\n return tensors\n elif isinstance(tensors, torch.Tensor):\n # single tensor shape can be inferred\n assert tensors.dim() == 4\n image_sizes = [tensor.shape[-2:] for tensor in tensors]\n return ImageList(tensors, image_sizes)\n elif isinstance(tensors, (tuple, list)):\n max_size = tuple(max(s) for s in zip(*[img.shape for img in tensors]))\n\n # TODO Ideally, just remove this and let me model handle arbitrary\n # input sizs\n if size_divisible > 0:\n import math\n\n stride = size_divisible\n max_size = list(max_size)\n max_size[1] = int(math.ceil(max_size[1] / stride) * stride)\n max_size[2] = int(math.ceil(max_size[2] / stride) * stride)\n max_size = tuple(max_size)\n\n batch_shape = (len(tensors),) + max_size\n batched_imgs = tensors[0].new(*batch_shape).zero_()\n for img, pad_img in zip(tensors, batched_imgs):\n pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img)\n\n image_sizes = [im.shape[-2:] for im in tensors]\n\n return ImageList(batched_imgs, image_sizes)\n else:\n raise TypeError(\"Unsupported type for to_image_list: {}\".format(type(tensors)))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef forward(self, features, proposals, targets=None):\n if self.training:\n with torch.no_grad():\n proposals = self.loss_evaluator.subsample(proposals, targets)\n\n x = self.feature_extractor(features, proposals)\n kp_logits = self.predictor(x)\n\n if not self.training:\n result = self.post_processor(kp_logits, proposals)\n return x, result, {}\n\n loss_kp = self.loss_evaluator(proposals, kp_logits)\n\n return x, proposals, dict(loss_kp=loss_kp)", "response": "Forward method for the base class."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_inference_input():\n return (tf.placeholder(tf.float32,\n [None, go.N, go.N, features_lib.NEW_FEATURES_PLANES],\n name='pos_tensor'),\n {'pi_tensor': tf.placeholder(tf.float32, [None, go.N * go.N + 1]),\n 'value_tensor': tf.placeholder(tf.float32, [None])})", "response": "Set up placeholders for input features and labels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef model_fn(features, labels, mode, params):\n\n policy_output, value_output, logits = model_inference_fn(\n features, mode == tf.estimator.ModeKeys.TRAIN, params)\n\n # train ops\n policy_cost = tf.reduce_mean(\n tf.nn.softmax_cross_entropy_with_logits_v2(\n logits=logits, labels=tf.stop_gradient(labels['pi_tensor'])))\n\n value_cost = params['value_cost_weight'] * tf.reduce_mean(\n tf.square(value_output - labels['value_tensor']))\n\n reg_vars = [v for v in tf.trainable_variables()\n if 'bias' not in v.name and 'beta' not in v.name]\n l2_cost = params['l2_strength'] * \\\n tf.add_n([tf.nn.l2_loss(v) for v in reg_vars])\n\n combined_cost = policy_cost + value_cost + l2_cost\n\n global_step = tf.train.get_or_create_global_step()\n learning_rate = tf.train.piecewise_constant(\n global_step, params['lr_boundaries'], params['lr_rates'])\n update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n\n # Insert quantization ops if requested\n if params['quantize']:\n if mode == tf.estimator.ModeKeys.TRAIN:\n tf.contrib.quantize.create_training_graph(\n quant_delay=params['quant_delay'])\n else:\n tf.contrib.quantize.create_eval_graph()\n\n optimizer = tf.train.MomentumOptimizer(\n learning_rate, params['sgd_momentum'])\n if params['use_tpu']:\n optimizer = tpu_optimizer.CrossShardOptimizer(optimizer)\n with tf.control_dependencies(update_ops):\n train_op = optimizer.minimize(combined_cost, global_step=global_step)\n\n # Computations to be executed on CPU, outside of the main TPU queues.\n def eval_metrics_host_call_fn(policy_output, value_output, pi_tensor, policy_cost,\n value_cost, l2_cost, combined_cost, step,\n est_mode=tf.estimator.ModeKeys.TRAIN):\n policy_entropy = -tf.reduce_mean(tf.reduce_sum(\n policy_output * tf.log(policy_output), axis=1))\n # pi_tensor is one_hot when generated from sgfs (for supervised learning)\n # and soft-max when using self-play records. argmax normalizes the two.\n policy_target_top_1 = tf.argmax(pi_tensor, axis=1)\n\n policy_output_in_top1 = tf.to_float(\n tf.nn.in_top_k(policy_output, policy_target_top_1, k=1))\n policy_output_in_top3 = tf.to_float(\n tf.nn.in_top_k(policy_output, policy_target_top_1, k=3))\n\n policy_top_1_confidence = tf.reduce_max(policy_output, axis=1)\n policy_target_top_1_confidence = tf.boolean_mask(\n policy_output,\n tf.one_hot(policy_target_top_1, tf.shape(policy_output)[1]))\n\n value_cost_normalized = value_cost / params['value_cost_weight']\n\n with tf.variable_scope(\"metrics\"):\n metric_ops = {\n 'policy_cost': tf.metrics.mean(policy_cost),\n 'value_cost': tf.metrics.mean(value_cost),\n 'value_cost_normalized': tf.metrics.mean(value_cost_normalized),\n 'l2_cost': tf.metrics.mean(l2_cost),\n 'policy_entropy': tf.metrics.mean(policy_entropy),\n 'combined_cost': tf.metrics.mean(combined_cost),\n\n 'policy_accuracy_top_1': tf.metrics.mean(policy_output_in_top1),\n 'policy_accuracy_top_3': tf.metrics.mean(policy_output_in_top3),\n 'policy_top_1_confidence': tf.metrics.mean(policy_top_1_confidence),\n 'policy_target_top_1_confidence': tf.metrics.mean(\n policy_target_top_1_confidence),\n 'value_confidence': tf.metrics.mean(tf.abs(value_output)),\n }\n\n if est_mode == tf.estimator.ModeKeys.EVAL:\n return metric_ops\n\n # NOTE: global_step is rounded to a multiple of FLAGS.summary_steps.\n eval_step = tf.reduce_min(step)\n\n # Create summary ops so that they show up in SUMMARIES collection\n # That way, they get logged automatically during training\n summary_writer = summary.create_file_writer(FLAGS.work_dir)\n with summary_writer.as_default(), \\\n summary.record_summaries_every_n_global_steps(\n params['summary_steps'], eval_step):\n for metric_name, metric_op in metric_ops.items():\n summary.scalar(metric_name, metric_op[1], step=eval_step)\n\n # Reset metrics occasionally so that they are mean of recent batches.\n reset_op = tf.variables_initializer(tf.local_variables(\"metrics\"))\n cond_reset_op = tf.cond(\n tf.equal(eval_step % params['summary_steps'], tf.to_int64(1)),\n lambda: reset_op,\n lambda: tf.no_op())\n\n return summary.all_summary_ops() + [cond_reset_op]\n\n metric_args = [\n policy_output,\n value_output,\n labels['pi_tensor'],\n tf.reshape(policy_cost, [1]),\n tf.reshape(value_cost, [1]),\n tf.reshape(l2_cost, [1]),\n tf.reshape(combined_cost, [1]),\n tf.reshape(global_step, [1]),\n ]\n\n predictions = {\n 'policy_output': policy_output,\n 'value_output': value_output,\n }\n\n eval_metrics_only_fn = functools.partial(\n eval_metrics_host_call_fn, est_mode=tf.estimator.ModeKeys.EVAL)\n host_call_fn = functools.partial(\n eval_metrics_host_call_fn, est_mode=tf.estimator.ModeKeys.TRAIN)\n\n tpu_estimator_spec = tpu_estimator.TPUEstimatorSpec(\n mode=mode,\n predictions=predictions,\n loss=combined_cost,\n train_op=train_op,\n eval_metrics=(eval_metrics_only_fn, metric_args),\n host_call=(host_call_fn, metric_args)\n )\n if params['use_tpu']:\n return tpu_estimator_spec\n else:\n return tpu_estimator_spec.as_estimator_spec()", "response": "Create the model for an estimator API."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild the inference part of the model graph.", "response": "def model_inference_fn(features, training, params):\n \"\"\"Builds just the inference part of the model graph.\n\n Args:\n features: input features tensor.\n training: True if the model is training.\n params: A dictionary\n\n Returns:\n (policy_output, value_output, logits) tuple of tensors.\n \"\"\"\n\n mg_batchn = functools.partial(\n tf.layers.batch_normalization,\n axis=-1,\n momentum=.95,\n epsilon=1e-5,\n center=True,\n scale=True,\n fused=True,\n training=training)\n\n mg_conv2d = functools.partial(\n tf.layers.conv2d,\n filters=params['conv_width'],\n kernel_size=3,\n padding=\"same\",\n data_format=\"channels_last\",\n use_bias=False)\n\n mg_global_avgpool2d = functools.partial(\n tf.layers.average_pooling2d,\n pool_size=go.N,\n strides=1,\n padding=\"valid\",\n data_format=\"channels_last\")\n\n def mg_activation(inputs):\n if FLAGS.use_swish:\n return tf.nn.swish(inputs)\n\n return tf.nn.relu(inputs)\n\n\n def residual_inner(inputs):\n conv_layer1 = mg_batchn(mg_conv2d(inputs))\n initial_output = mg_activation(conv_layer1)\n conv_layer2 = mg_batchn(mg_conv2d(initial_output))\n return conv_layer2\n\n def mg_res_layer(inputs):\n residual = residual_inner(inputs)\n output = mg_activation(inputs + residual)\n return output\n\n def mg_squeeze_excitation_layer(inputs):\n # Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-Excitation Networks.\n # 2018 IEEE/CVF Conference on Computer Vision, 7132-7141.\n # arXiv:1709.01507 [cs.CV]\n\n channels = params['conv_width']\n ratio = FLAGS.SE_ratio\n assert channels % ratio == 0\n\n residual = residual_inner(inputs)\n pool = mg_global_avgpool2d(residual)\n fc1 = tf.layers.dense(pool, units=channels // ratio)\n squeeze = mg_activation(fc1)\n\n if FLAGS.use_SE_bias:\n fc2 = tf.layers.dense(squeeze, units=2*channels)\n # Channels_last so axis = 3 = -1\n gamma, bias = tf.split(fc2, 2, axis=3)\n else:\n gamma = tf.layers.dense(squeeze, units=channels)\n bias = 0\n\n sig = tf.nn.sigmoid(gamma)\n # Explicitly signal the broadcast.\n scale = tf.reshape(sig, [-1, 1, 1, channels])\n\n excitation = tf.multiply(scale, residual) + bias\n return mg_activation(inputs + excitation)\n\n initial_block = mg_activation(mg_batchn(mg_conv2d(features)))\n\n # the shared stack\n shared_output = initial_block\n for _ in range(params['trunk_layers']):\n if FLAGS.use_SE or FLAGS.use_SE_bias:\n shared_output = mg_squeeze_excitation_layer(shared_output)\n else:\n shared_output = mg_res_layer(shared_output)\n\n # Policy head\n policy_conv = mg_conv2d(\n shared_output, filters=params['policy_conv_width'], kernel_size=1)\n policy_conv = mg_activation(mg_batchn(policy_conv, center=False, scale=False))\n logits = tf.layers.dense(\n tf.reshape(\n policy_conv, [-1, params['policy_conv_width'] * go.N * go.N]),\n go.N * go.N + 1)\n\n policy_output = tf.nn.softmax(logits, name='policy_output')\n\n # Value head\n value_conv = mg_conv2d(\n shared_output, filters=params['value_conv_width'], kernel_size=1)\n value_conv = mg_activation(mg_batchn(value_conv, center=False, scale=False))\n\n value_fc_hidden = mg_activation(tf.layers.dense(\n tf.reshape(value_conv, [-1, params['value_conv_width'] * go.N * go.N]),\n params['fc_width']))\n value_output = tf.nn.tanh(\n tf.reshape(tf.layers.dense(value_fc_hidden, 1), [-1]),\n name='value_output')\n\n return policy_output, value_output, logits"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild the model graph suitable for running on TPU.", "response": "def tpu_model_inference_fn(features):\n \"\"\"Builds the model graph suitable for running on TPU.\n\n It does two things:\n 1) Mark all weights as constant, which improves TPU inference performance\n because it prevents the weights being transferred to the TPU every call\n to Session.run().\n 2) Adds constant to the graph with a unique value and marks it as a\n dependency on the rest of the model. This works around a TensorFlow bug\n that prevents multiple models being run on a single TPU.\n\n Returns:\n (policy_output, value_output, logits) tuple of tensors.\n \"\"\"\n def custom_getter(getter, name, *args, **kwargs):\n with tf.control_dependencies(None):\n return tf.guarantee_const(\n getter(name, *args, **kwargs), name=name + \"/GuaranteeConst\")\n with tf.variable_scope(\"\", custom_getter=custom_getter):\n # TODO(tommadams): remove the tf.control_dependencies context manager\n # when a fixed version of TensorFlow is released.\n t = int(time.time())\n epoch_time = tf.constant(t, name='epoch_time_%d' % t)\n with tf.control_dependencies([epoch_time]):\n return model_inference_fn(features, False, FLAGS.flag_values_dict())"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninitializes a tf. Estimator run with random initial weights.", "response": "def bootstrap():\n \"\"\"Initialize a tf.Estimator run with random initial weights.\"\"\"\n # a bit hacky - forge an initial checkpoint with the name that subsequent\n # Estimator runs will expect to find.\n #\n # Estimator will do this automatically when you call train(), but calling\n # train() requires data, and I didn't feel like creating training data in\n # order to run the full train pipeline for 1 step.\n maybe_set_seed()\n initial_checkpoint_name = 'model.ckpt-1'\n save_file = os.path.join(FLAGS.work_dir, initial_checkpoint_name)\n sess = tf.Session(graph=tf.Graph())\n with sess.graph.as_default():\n features, labels = get_inference_input()\n model_fn(features, labels, tf.estimator.ModeKeys.PREDICT,\n params=FLAGS.flag_values_dict())\n sess.run(tf.global_variables_initializer())\n tf.train.Saver().save(sess, save_file)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef export_model(model_path):\n estimator = tf.estimator.Estimator(model_fn, model_dir=FLAGS.work_dir,\n params=FLAGS.flag_values_dict())\n latest_checkpoint = estimator.latest_checkpoint()\n all_checkpoint_files = tf.gfile.Glob(latest_checkpoint + '*')\n for filename in all_checkpoint_files:\n suffix = filename.partition(latest_checkpoint)[2]\n destination_path = model_path + suffix\n print(\"Copying {} to {}\".format(filename, destination_path))\n tf.gfile.Copy(filename, destination_path)", "response": "Take the latest checkpoint and copy it to model_path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninitializing the weights from the given save_file.", "response": "def initialize_weights(self, save_file):\n \"\"\"Initialize the weights from the given save_file.\n Assumes that the graph has been constructed, and the\n save_file contains weights that match the graph. Used\n to set the weights to a different version of the player\n without redifining the entire graph.\"\"\"\n tf.train.Saver().restore(self.sess, save_file)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses the output of helpfull.", "response": "def parse_helpfull_output(help_output, regex=FLAG_HELP_RE_PY):\n \"\"\"Parses the output of --helpfull.\n\n Args:\n help_output: str, the full output of --helpfull.\n\n Returns:\n A set of flags that are valid flags.\n \"\"\"\n valid_flags = set()\n for _, no_prefix, flag_name in regex.findall(help_output):\n valid_flags.add('--' + flag_name)\n if no_prefix:\n valid_flags.add('--no' + flag_name)\n return valid_flags"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef filter_flags(parsed_flags, valid_flags):\n def valid_argv(argv):\n \"\"\"Figures out if a flag parsed from the flagfile matches a flag in\n the command about to be run.\"\"\"\n flagname_match = FLAG_RE.match(argv)\n if not flagname_match:\n return True\n flagname = flagname_match.group()\n return flagname in valid_flags\n return list(filter(valid_argv, parsed_flags))", "response": "Return a subset of parsed_flags that are found in the list valid_flags."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprepare a subprocess command by running helpfull and masking flags.", "response": "def prepare_subprocess_cmd(subprocess_cmd):\n \"\"\"Prepares a subprocess command by running --helpfull and masking flags.\n\n Args:\n subprocess_cmd: List[str], what would be passed into subprocess.call()\n i.e. ['python', 'train.py', '--flagfile=flags']\n\n Returns:\n ['python', 'train.py', '--train_flag=blah', '--more_flags']\n \"\"\"\n help_cmd = subprocess_cmd + ['--helpfull']\n help_output = subprocess.run(help_cmd, stdout=subprocess.PIPE).stdout\n help_output = help_output.decode('ascii')\n if 'python' in subprocess_cmd[0]:\n valid_flags = parse_helpfull_output(help_output)\n else:\n valid_flags = parse_helpfull_output(help_output, regex=FLAG_HELP_RE_CC)\n parsed_flags = flags.FlagValues().read_flags_from_files(subprocess_cmd[1:])\n\n filtered_flags = filter_flags(parsed_flags, valid_flags)\n return [subprocess_cmd[0]] + filtered_flags"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprepares and run a subprocess cmd returning a CompletedProcess.", "response": "def run(cmd):\n \"\"\"Prepare and run a subprocess cmd, returning a CompletedProcess.\"\"\"\n print(\"Preparing the following cmd:\")\n cmd = prepare_subprocess_cmd(cmd)\n print(\"Running the following cmd:\")\n print('\\n'.join(cmd))\n return subprocess.run(cmd, stdout=sys.stdout, stderr=sys.stderr)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef checked_run(cmd):\n completed_process = run(cmd)\n if completed_process.returncode > 0:\n print(\"Command failed! Hanging around in case someone needs a \"\n \"docker connection. (Ctrl-C to quit now)\")\n time.sleep(300)\n raise RuntimeError\n return completed_process", "response": "Prepare and run a subprocess cmd checking for successful completion."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_record_dataset(dataset, is_training, batch_size, shuffle_buffer,\n parse_record_fn, num_epochs=1, num_gpus=None,\n examples_per_epoch=None, dtype=tf.float32):\n \"\"\"Given a Dataset with raw records, return an iterator over the records.\n\n Args:\n dataset: A Dataset representing raw records\n is_training: A boolean denoting whether the input is for training.\n batch_size: The number of samples per batch.\n shuffle_buffer: The buffer size to use when shuffling records. A larger\n value results in better randomness, but smaller values reduce startup\n time and use less memory.\n parse_record_fn: A function that takes a raw record and returns the\n corresponding (image, label) pair.\n num_epochs: The number of epochs to repeat the dataset.\n num_gpus: The number of gpus used for training.\n examples_per_epoch: The number of examples in an epoch.\n dtype: Data type to use for images/features.\n\n Returns:\n Dataset of (image, label) pairs ready for iteration.\n \"\"\"\n\n # We prefetch a batch at a time, This can help smooth out the time taken to\n # load input files as we go through shuffling and processing.\n dataset = dataset.prefetch(buffer_size=batch_size)\n if is_training:\n # Shuffle the records. Note that we shuffle before repeating to ensure\n # that the shuffling respects epoch boundaries.\n mlperf_log.resnet_print(key=mlperf_log.INPUT_ORDER)\n dataset = dataset.shuffle(buffer_size=shuffle_buffer)\n\n # If we are training over multiple epochs before evaluating, repeat the\n # dataset for the appropriate number of epochs.\n dataset = dataset.repeat(num_epochs)\n\n # Parse the raw records into images and labels. Testing has shown that setting\n # num_parallel_batches > 1 produces no improvement in throughput, since\n # batch_size is almost always much greater than the number of CPU cores.\n dataset = dataset.apply(\n tf.contrib.data.map_and_batch(\n lambda value: parse_record_fn(value, is_training, dtype),\n batch_size=batch_size,\n num_parallel_batches=1))\n\n # Operations between the final prefetch and the get_next call to the iterator\n # will happen synchronously during run time. We prefetch here again to\n # background all of the above processing work and keep it out of the\n # critical training path. Setting buffer_size to tf.contrib.data.AUTOTUNE\n # allows DistributionStrategies to adjust how many batches to fetch based\n # on how many devices are present.\n dataset = dataset.prefetch(buffer_size=tf.contrib.data.AUTOTUNE)\n\n return dataset", "response": "This function takes a Dataset representing raw records and returns an iterator over the records."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn an input function that returns a dataset with zeroes.", "response": "def get_synth_input_fn(height, width, num_channels, num_classes):\n \"\"\"Returns an input function that returns a dataset with zeroes.\n\n This is useful in debugging input pipeline performance, as it removes all\n elements of file reading and image preprocessing.\n\n Args:\n height: Integer height that will be used to create a fake image tensor.\n width: Integer width that will be used to create a fake image tensor.\n num_channels: Integer depth that will be used to create a fake image tensor.\n num_classes: Number of classes that should be represented in the fake labels\n tensor\n\n Returns:\n An input_fn that can be used in place of a real one to return a dataset\n that can be used for iteration.\n \"\"\"\n def input_fn(is_training, data_dir, batch_size, *args, **kwargs): # pylint: disable=unused-argument\n images = tf.zeros((batch_size, height, width, num_channels), tf.float32)\n labels = tf.zeros((batch_size, num_classes), tf.int32)\n return tf.data.Dataset.from_tensors((images, labels)).repeat()\n\n return input_fn"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a function that decays step - wise as training progresses.", "response": "def learning_rate_with_decay(\n batch_size, batch_denom, num_images, boundary_epochs, decay_rates,\n base_lr=0.1, enable_lars=False):\n \"\"\"Get a learning rate that decays step-wise as training progresses.\n\n Args:\n batch_size: the number of examples processed in each training batch.\n batch_denom: this value will be used to scale the base learning rate.\n `0.1 * batch size` is divided by this number, such that when\n batch_denom == batch_size, the initial learning rate will be 0.1.\n num_images: total number of images that will be used for training.\n boundary_epochs: list of ints representing the epochs at which we\n decay the learning rate.\n decay_rates: list of floats representing the decay rates to be used\n for scaling the learning rate. It should have one more element\n than `boundary_epochs`, and all elements should have the same type.\n base_lr: Initial learning rate scaled based on batch_denom.\n\n Returns:\n Returns a function that takes a single argument - the number of batches\n trained so far (global_step)- and returns the learning rate to be used\n for training the next batch.\n \"\"\"\n initial_learning_rate = base_lr * batch_size / batch_denom\n batches_per_epoch = num_images / batch_size\n\n # Multiply the learning rate by 0.1 at 100, 150, and 200 epochs.\n boundaries = [int(batches_per_epoch * epoch) for epoch in boundary_epochs]\n vals = [initial_learning_rate * decay for decay in decay_rates]\n\n def learning_rate_fn(global_step):\n lr = tf.train.piecewise_constant(global_step, boundaries, vals)\n warmup_steps = int(batches_per_epoch * 5)\n warmup_lr = (\n initial_learning_rate * tf.cast(global_step, tf.float32) / tf.cast(\n warmup_steps, tf.float32))\n return tf.cond(global_step < warmup_steps, lambda: warmup_lr, lambda: lr)\n\n def poly_rate_fn(global_step):\n \"\"\"Handles linear scaling rule, gradual warmup, and LR decay.\n\n The learning rate starts at 0, then it increases linearly per step. After\n flags.poly_warmup_epochs, we reach the base learning rate (scaled to account\n for batch size). The learning rate is then decayed using a polynomial rate\n decay schedule with power 2.0.\n\n Args:\n global_step: the current global_step\n\n Returns:\n returns the current learning rate\n \"\"\"\n\n # Learning rate schedule for LARS polynomial schedule\n if batch_size < 8192:\n plr = 5.0\n w_epochs = 5\n elif batch_size < 16384:\n plr = 10.0\n w_epochs = 5\n elif batch_size < 32768:\n plr = 25.0\n w_epochs = 5\n else:\n plr = 32.0\n w_epochs = 14\n\n w_steps = int(w_epochs * batches_per_epoch)\n wrate = (plr * tf.cast(global_step, tf.float32) / tf.cast(\n w_steps, tf.float32))\n\n # TODO(pkanwar): use a flag to help calc num_epochs.\n num_epochs = 90\n train_steps = batches_per_epoch * num_epochs\n\n min_step = tf.constant(1, dtype=tf.int64)\n decay_steps = tf.maximum(min_step, tf.subtract(global_step, w_steps))\n poly_rate = tf.train.polynomial_decay(\n plr,\n decay_steps,\n train_steps - w_steps + 1,\n power=2.0)\n return tf.where(global_step <= w_steps, wrate, poly_rate)\n\n # For LARS we have a new learning rate schedule\n if enable_lars:\n return poly_rate_fn\n\n return learning_rate_fn"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsharing functionality for different resnet model_fns. Initializes the ResnetModel representing the model layers and uses that model to build the necessary EstimatorSpecs for the `mode` in question. For training, this means building losses, the optimizer, and the train op that get passed into the EstimatorSpec. For evaluation and prediction, the EstimatorSpec is returned without a train op, but with the necessary parameters for the given mode. Args: features: tensor representing input images labels: tensor representing class labels for all input images mode: current estimator mode; should be one of `tf.estimator.ModeKeys.TRAIN`, `EVALUATE`, `PREDICT` model_class: a class representing a TensorFlow model that has a __call__ function. We assume here that this is a subclass of ResnetModel. resnet_size: A single integer for the size of the ResNet model. weight_decay: weight decay loss rate used to regularize learned variables. learning_rate_fn: function that returns the current learning rate given the current global_step momentum: momentum term used for optimization data_format: Input format ('channels_last', 'channels_first', or None). If set to None, the format is dependent on whether a GPU is available. version: Integer representing which version of the ResNet network to use. See README for details. Valid values: [1, 2] loss_scale: The factor to scale the loss for numerical stability. A detailed summary is present in the arg parser help text. loss_filter_fn: function that takes a string variable name and returns True if the var should be included in loss calculation, and False otherwise. If None, batch_normalization variables will be excluded from the loss. dtype: the TensorFlow dtype to use for calculations. Returns: EstimatorSpec parameterized according to the input params and the current mode.", "response": "def resnet_model_fn(features, labels, mode, model_class,\n resnet_size, weight_decay, learning_rate_fn, momentum,\n data_format, version, loss_scale, loss_filter_fn=None,\n dtype=resnet_model.DEFAULT_DTYPE,\n label_smoothing=0.0, enable_lars=False):\n \"\"\"Shared functionality for different resnet model_fns.\n\n Initializes the ResnetModel representing the model layers\n and uses that model to build the necessary EstimatorSpecs for\n the `mode` in question. For training, this means building losses,\n the optimizer, and the train op that get passed into the EstimatorSpec.\n For evaluation and prediction, the EstimatorSpec is returned without\n a train op, but with the necessary parameters for the given mode.\n\n Args:\n features: tensor representing input images\n labels: tensor representing class labels for all input images\n mode: current estimator mode; should be one of\n `tf.estimator.ModeKeys.TRAIN`, `EVALUATE`, `PREDICT`\n model_class: a class representing a TensorFlow model that has a __call__\n function. We assume here that this is a subclass of ResnetModel.\n resnet_size: A single integer for the size of the ResNet model.\n weight_decay: weight decay loss rate used to regularize learned variables.\n learning_rate_fn: function that returns the current learning rate given\n the current global_step\n momentum: momentum term used for optimization\n data_format: Input format ('channels_last', 'channels_first', or None).\n If set to None, the format is dependent on whether a GPU is available.\n version: Integer representing which version of the ResNet network to use.\n See README for details. Valid values: [1, 2]\n loss_scale: The factor to scale the loss for numerical stability. A detailed\n summary is present in the arg parser help text.\n loss_filter_fn: function that takes a string variable name and returns\n True if the var should be included in loss calculation, and False\n otherwise. If None, batch_normalization variables will be excluded\n from the loss.\n dtype: the TensorFlow dtype to use for calculations.\n\n Returns:\n EstimatorSpec parameterized according to the input params and the\n current mode.\n \"\"\"\n\n # Generate a summary node for the images\n tf.summary.image('images', features, max_outputs=6)\n\n # Checks that features/images have same data type being used for calculations.\n assert features.dtype == dtype\n\n features = tf.cast(features, dtype)\n\n model = model_class(resnet_size, data_format, version=version, dtype=dtype)\n\n logits = model(features, mode == tf.estimator.ModeKeys.TRAIN)\n\n # This acts as a no-op if the logits are already in fp32 (provided logits are\n # not a SparseTensor). If dtype is is low precision, logits must be cast to\n # fp32 for numerical stability.\n logits = tf.cast(logits, tf.float32)\n\n num_examples_metric = tf_mlperf_log.sum_metric(tensor=tf.shape(logits)[0], name=_NUM_EXAMPLES_NAME)\n\n predictions = {\n 'classes': tf.argmax(logits, axis=1),\n 'probabilities': tf.nn.softmax(logits, name='softmax_tensor')\n }\n\n\n if mode == tf.estimator.ModeKeys.PREDICT:\n # Return the predictions and the specification for serving a SavedModel\n return tf.estimator.EstimatorSpec(\n mode=mode,\n predictions=predictions,\n export_outputs={\n 'predict': tf.estimator.export.PredictOutput(predictions)\n })\n\n # Calculate loss, which includes softmax cross entropy and L2 regularization.\n mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_LOSS_FN, value=mlperf_log.CCE)\n\n if label_smoothing != 0.0:\n one_hot_labels = tf.one_hot(labels, 1001)\n cross_entropy = tf.losses.softmax_cross_entropy(\n logits=logits, onehot_labels=one_hot_labels,\n label_smoothing=label_smoothing)\n else:\n cross_entropy = tf.losses.sparse_softmax_cross_entropy(\n logits=logits, labels=labels)\n\n # Create a tensor named cross_entropy for logging purposes.\n tf.identity(cross_entropy, name='cross_entropy')\n tf.summary.scalar('cross_entropy', cross_entropy)\n\n # If no loss_filter_fn is passed, assume we want the default behavior,\n # which is that batch_normalization variables are excluded from loss.\n def exclude_batch_norm(name):\n return 'batch_normalization' not in name\n loss_filter_fn = loss_filter_fn or exclude_batch_norm\n\n mlperf_log.resnet_print(key=mlperf_log.MODEL_EXCLUDE_BN_FROM_L2,\n value=not loss_filter_fn('batch_normalization'))\n\n # Add weight decay to the loss.\n mlperf_log.resnet_print(key=mlperf_log.MODEL_L2_REGULARIZATION,\n value=weight_decay)\n l2_loss = weight_decay * tf.add_n(\n # loss is computed using fp32 for numerical stability.\n [tf.nn.l2_loss(tf.cast(v, tf.float32)) for v in tf.trainable_variables()\n if loss_filter_fn(v.name)])\n tf.summary.scalar('l2_loss', l2_loss)\n loss = cross_entropy + l2_loss\n\n if mode == tf.estimator.ModeKeys.TRAIN:\n global_step = tf.train.get_or_create_global_step()\n\n learning_rate = learning_rate_fn(global_step)\n\n log_id = mlperf_log.resnet_print(key=mlperf_log.OPT_LR, deferred=True)\n learning_rate = tf_mlperf_log.log_deferred(op=learning_rate, log_id=log_id,\n every_n=100)\n\n # Create a tensor named learning_rate for logging purposes\n tf.identity(learning_rate, name='learning_rate')\n tf.summary.scalar('learning_rate', learning_rate)\n\n mlperf_log.resnet_print(key=mlperf_log.OPT_NAME,\n value=mlperf_log.SGD_WITH_MOMENTUM)\n mlperf_log.resnet_print(key=mlperf_log.OPT_MOMENTUM, value=momentum)\n\n if enable_lars:\n optimizer = tf.contrib.opt.LARSOptimizer(\n learning_rate,\n momentum=momentum,\n weight_decay=weight_decay,\n skip_list=['batch_normalization', 'bias'])\n else:\n optimizer = tf.train.MomentumOptimizer(\n learning_rate=learning_rate,\n momentum=momentum\n )\n\n if loss_scale != 1:\n # When computing fp16 gradients, often intermediate tensor values are\n # so small, they underflow to 0. To avoid this, we multiply the loss by\n # loss_scale to make these tensor values loss_scale times bigger.\n scaled_grad_vars = optimizer.compute_gradients(loss * loss_scale)\n\n # Once the gradient computation is complete we can scale the gradients\n # back to the correct scale before passing them to the optimizer.\n unscaled_grad_vars = [(grad / loss_scale, var)\n for grad, var in scaled_grad_vars]\n minimize_op = optimizer.apply_gradients(unscaled_grad_vars, global_step)\n else:\n minimize_op = optimizer.minimize(loss, global_step)\n\n update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n train_op = tf.group(minimize_op, update_ops, num_examples_metric[1])\n else:\n train_op = None\n\n accuracy = tf.metrics.accuracy(labels, predictions['classes'])\n accuracy_top_5 = tf.metrics.mean(tf.nn.in_top_k(predictions=logits,\n targets=labels,\n k=5,\n name='top_5_op'))\n\n metrics = {'accuracy': accuracy,\n 'accuracy_top_5': accuracy_top_5,\n _NUM_EXAMPLES_NAME: num_examples_metric}\n\n # Create a tensor named train_accuracy for logging purposes\n tf.identity(accuracy[1], name='train_accuracy')\n tf.identity(accuracy_top_5[1], name='train_accuracy_top_5')\n tf.summary.scalar('train_accuracy', accuracy[1])\n tf.summary.scalar('train_accuracy_top_5', accuracy_top_5[1])\n\n return tf.estimator.EstimatorSpec(\n mode=mode,\n predictions=predictions,\n loss=loss,\n train_op=train_op,\n eval_metric_ops=metrics)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the batch size per device.", "response": "def per_device_batch_size(batch_size, num_gpus):\n \"\"\"For multi-gpu, batch-size must be a multiple of the number of GPUs.\n\n Note that this should eventually be handled by DistributionStrategies\n directly. Multi-GPU support is currently experimental, however,\n so doing the work here until that feature is in place.\n\n Args:\n batch_size: Global batch size to be divided among devices. This should be\n equal to num_gpus times the single-GPU batch_size for multi-gpu training.\n num_gpus: How many GPUs are used with DistributionStrategies.\n\n Returns:\n Batch size per device.\n\n Raises:\n ValueError: if batch_size is not divisible by number of devices\n \"\"\"\n if num_gpus <= 1:\n return batch_size\n\n remainder = batch_size % num_gpus\n if remainder:\n err = ('When running with multiple GPUs, batch size '\n 'must be a multiple of the number of available GPUs. Found {} '\n 'GPUs with a batch size of {}; try --batch_size={} instead.'\n ).format(num_gpus, batch_size, batch_size - remainder)\n raise ValueError(err)\n return int(batch_size / num_gpus)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nshare main loop for ResNet Models. Args: flags: FLAGS object that contains the params for running. See ResnetArgParser for created flags. model_function: the function that instantiates the Model and builds the ops for train/eval. This will be passed directly into the estimator. input_function: the function that processes the dataset and returns a dataset that the estimator can train on. This will be wrapped with all the relevant flags for running and passed to estimator. shape: list of ints representing the shape of the images used for training. This is only used if flags.export_dir is passed.", "response": "def resnet_main(seed, flags, model_function, input_function, shape=None):\n \"\"\"Shared main loop for ResNet Models.\n\n Args:\n flags: FLAGS object that contains the params for running. See\n ResnetArgParser for created flags.\n model_function: the function that instantiates the Model and builds the\n ops for train/eval. This will be passed directly into the estimator.\n input_function: the function that processes the dataset and returns a\n dataset that the estimator can train on. This will be wrapped with\n all the relevant flags for running and passed to estimator.\n shape: list of ints representing the shape of the images used for training.\n This is only used if flags.export_dir is passed.\n \"\"\"\n\n mlperf_log.resnet_print(key=mlperf_log.RUN_START)\n\n # Using the Winograd non-fused algorithms provides a small performance boost.\n os.environ['TF_ENABLE_WINOGRAD_NONFUSED'] = '1'\n\n # Create session config based on values of inter_op_parallelism_threads and\n # intra_op_parallelism_threads. Note that we default to having\n # allow_soft_placement = True, which is required for multi-GPU and not\n # harmful for other modes.\n session_config = tf.ConfigProto(\n inter_op_parallelism_threads=flags.inter_op_parallelism_threads,\n intra_op_parallelism_threads=flags.intra_op_parallelism_threads,\n allow_soft_placement=True)\n\n if flags.num_gpus == 0:\n distribution = tf.contrib.distribute.OneDeviceStrategy('device:CPU:0')\n elif flags.num_gpus == 1:\n distribution = tf.contrib.distribute.OneDeviceStrategy('device:GPU:0')\n else:\n distribution = tf.contrib.distribute.MirroredStrategy(\n num_gpus=flags.num_gpus\n )\n\n mlperf_log.resnet_print(key=mlperf_log.RUN_SET_RANDOM_SEED, value=seed)\n run_config = tf.estimator.RunConfig(train_distribute=distribution,\n session_config=session_config,\n tf_random_seed=seed)\n\n mlperf_log.resnet_print(key=mlperf_log.INPUT_BATCH_SIZE,\n value=flags.batch_size)\n classifier = tf.estimator.Estimator(\n model_fn=model_function, model_dir=flags.model_dir, config=run_config,\n params={\n 'resnet_size': flags.resnet_size,\n 'data_format': flags.data_format,\n 'batch_size': flags.batch_size,\n 'version': flags.version,\n 'loss_scale': flags.loss_scale,\n 'dtype': flags.dtype,\n 'label_smoothing': flags.label_smoothing,\n 'enable_lars': flags.enable_lars,\n 'weight_decay': flags.weight_decay,\n 'fine_tune': flags.fine_tune\n })\n\n if flags.benchmark_log_dir is not None:\n benchmark_logger = logger.BenchmarkLogger(flags.benchmark_log_dir)\n benchmark_logger.log_run_info('resnet')\n else:\n benchmark_logger = None\n\n mlperf_log.resnet_print(key=mlperf_log.TRAIN_LOOP)\n\n # The reference performs the first evaluation on the fourth epoch. (offset\n # eval by 3 epochs)\n mlperf_log.resnet_print(key=mlperf_log.EVAL_EPOCH_OFFSET, value=3)\n success = False\n for i in range(flags.train_epochs // flags.epochs_between_evals):\n # Data for epochs_between_evals (i.e. 4 epochs between evals) worth of\n # epochs is concatenated and run as a single block inside a session. For\n # this reason we declare all of the epochs that will be run at the start.\n # Submitters may report in a way which is reasonable for their control flow.\n for j in range(flags.epochs_between_evals):\n mlperf_log.resnet_print(key=mlperf_log.TRAIN_EPOCH,\n value=i * flags.epochs_between_evals + j)\n train_hooks = hooks_helper.get_train_hooks(\n flags.hooks,\n batch_size=flags.batch_size,\n benchmark_log_dir=flags.benchmark_log_dir)\n\n _log_cache = []\n def formatter(x):\n \"\"\"Abuse side effects to get tensors out of the model_fn.\"\"\"\n if _log_cache:\n _log_cache.pop()\n _log_cache.append(x.copy())\n return str(x)\n\n compliance_hook = tf.train.LoggingTensorHook(\n tensors={_NUM_EXAMPLES_NAME: _NUM_EXAMPLES_NAME},\n every_n_iter=int(1e10),\n at_end=True,\n formatter=formatter)\n\n print('Starting a training cycle.')\n\n def input_fn_train():\n return input_function(\n is_training=True,\n data_dir=flags.data_dir,\n batch_size=per_device_batch_size(flags.batch_size, flags.num_gpus),\n num_epochs=flags.epochs_between_evals,\n num_gpus=flags.num_gpus,\n dtype=flags.dtype\n )\n\n classifier.train(input_fn=input_fn_train, hooks=train_hooks + [compliance_hook],\n max_steps=flags.max_train_steps)\n\n train_examples = int(_log_cache.pop()[_NUM_EXAMPLES_NAME])\n mlperf_log.resnet_print(key=mlperf_log.INPUT_SIZE, value=train_examples)\n\n print('Starting to evaluate.')\n # Evaluate the model and print results\n def input_fn_eval():\n return input_function(\n is_training=False,\n data_dir=flags.data_dir,\n batch_size=per_device_batch_size(flags.batch_size, flags.num_gpus),\n num_epochs=1,\n dtype=flags.dtype\n )\n\n\n mlperf_log.resnet_print(key=mlperf_log.EVAL_START)\n # flags.max_train_steps is generally associated with testing and profiling.\n # As a result it is frequently called with synthetic data, which will\n # iterate forever. Passing steps=flags.max_train_steps allows the eval\n # (which is generally unimportant in those circumstances) to terminate.\n # Note that eval will run for max_train_steps each loop, regardless of the\n # global_step count.\n eval_results = classifier.evaluate(input_fn=input_fn_eval,\n steps=flags.max_train_steps)\n mlperf_log.resnet_print(key=mlperf_log.EVAL_STOP)\n mlperf_log.resnet_print(key=mlperf_log.EVAL_SIZE, value=int(eval_results[_NUM_EXAMPLES_NAME]))\n mlperf_log.resnet_print(key=mlperf_log.EVAL_ACCURACY, value=float(eval_results['accuracy']))\n mlperf_log.resnet_print(key=mlperf_log.EVAL_TARGET, value=flags.stop_threshold)\n print(eval_results)\n\n if benchmark_logger:\n benchmark_logger.log_estimator_evaluation_result(eval_results)\n\n if model_helpers.past_stop_threshold(\n flags.stop_threshold, eval_results['accuracy']):\n success = True\n break\n\n mlperf_log.resnet_print(key=mlperf_log.RUN_STOP, value={\"success\": success})\n mlperf_log.resnet_print(key=mlperf_log.RUN_FINAL)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compute_ratings(data=None):\n if data is None:\n with sqlite3.connect(\"ratings.db\") as db:\n data = db.execute(\"select model_winner, model_loser from wins\").fetchall()\n model_ids = set([d[0] for d in data]).union(set([d[1] for d in data]))\n\n # Map model_ids to a contiguous range.\n ordered = sorted(model_ids)\n new_id = {}\n for i, m in enumerate(ordered):\n new_id[m] = i\n\n # A function to rewrite the model_ids in our pairs\n def ilsr_data(d):\n p1, p2 = d\n p1 = new_id[p1]\n p2 = new_id[p2]\n return (p1, p2)\n\n pairs = list(map(ilsr_data, data))\n ilsr_param = choix.ilsr_pairwise(\n len(ordered),\n pairs,\n alpha=0.0001,\n max_iter=800)\n\n hessian = choix.opt.PairwiseFcts(pairs, penalty=.1).hessian(ilsr_param)\n std_err = np.sqrt(np.diagonal(np.linalg.inv(hessian)))\n\n # Elo conversion\n elo_mult = 400 / math.log(10)\n\n min_rating = min(ilsr_param)\n ratings = {}\n\n for model_id, param, err in zip(ordered, ilsr_param, std_err):\n ratings[model_id] = (elo_mult * (param - min_rating), elo_mult * err)\n\n return ratings", "response": "Computes the ratings for the current run of the current language."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef suggest_pairs(top_n=10, per_n=3, ignore_before=300):\n db = sqlite3.connect(\"ratings.db\")\n data = db.execute(\"select model_winner, model_loser from wins\").fetchall()\n bucket_ids = [id[0] for id in db.execute(\n \"select id from models where bucket = ?\", (fsdb.models_dir(),)).fetchall()]\n bucket_ids.sort()\n data = [d for d in data if d[0] in bucket_ids and d[1] in bucket_ids]\n\n ratings = [(model_num_for(k), v[0], v[1]) for k, v in compute_ratings(data).items()]\n ratings.sort()\n ratings = ratings[ignore_before:] # Filter off the first 100 models, which improve too fast.\n\n ratings.sort(key=lambda r: r[2], reverse=True)\n\n res = []\n for p1 in ratings[:top_n]:\n candidate_p2s = sorted(ratings, key=lambda p2_tup: abs(p1[1] - p2_tup[1]))[1:20]\n choices = random.sample(candidate_p2s, per_n)\n print(\"Pairing {}, sigma {:.2f} (Rating {:.2f})\".format(p1[0], p1[2], p1[1]))\n for p2 in choices:\n res.append([p1[0], p2[0]])\n print(\" {}, ratings delta {:.2f}\".format(p2[0], abs(p1[1] - p2[1])))\n return res", "response": "Suggests the best pairs of players to match up\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef time_recommendation(move_num, seconds_per_move=5, time_limit=15 * 60,\n decay_factor=0.98):\n \"\"\"Given the current move number and the 'desired' seconds per move, return\n how much time should actually be used. This is intended specifically for\n CGOS time controls, which has an absolute 15-minute time limit.\n\n The strategy is to spend the maximum possible moves using seconds_per_move,\n and then switch to an exponentially decaying time usage, calibrated so that\n we have enough time for an infinite number of moves.\"\"\"\n\n # Divide by two since you only play half the moves in a game.\n player_move_num = move_num / 2\n\n # Sum of geometric series maxes out at endgame_time seconds.\n endgame_time = seconds_per_move / (1 - decay_factor)\n\n if endgame_time > time_limit:\n # There is so little main time that we're already in 'endgame' mode.\n base_time = time_limit * (1 - decay_factor)\n core_moves = 0\n else:\n # Leave over endgame_time seconds for the end, and play at\n # seconds_per_move for as long as possible.\n base_time = seconds_per_move\n core_moves = (time_limit - endgame_time) / seconds_per_move\n\n return base_time * decay_factor ** max(player_move_num - core_moves, 0)", "response": "This function returns how much time should actually be used for a particular move number and the desired seconds per move."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsuggest a move based on the current position.", "response": "def suggest_move(self, position):\n \"\"\"Used for playing a single game.\n\n For parallel play, use initialize_move, select_leaf,\n incorporate_results, and pick_move\n \"\"\"\n start = time.time()\n\n if self.timed_match:\n while time.time() - start < self.seconds_per_move:\n self.tree_search()\n else:\n current_readouts = self.root.N\n while self.root.N < current_readouts + self.num_readouts:\n self.tree_search()\n if self.verbosity > 0:\n dbg(\"%d: Searched %d times in %.2f seconds\\n\\n\" % (\n position.n, self.num_readouts, time.time() - start))\n\n # print some stats on moves considered.\n if self.verbosity > 2:\n dbg(self.root.describe())\n dbg('\\n\\n')\n if self.verbosity > 3:\n dbg(self.root.position)\n\n return self.pick_move()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef play_move(self, c):\n if not self.two_player_mode:\n self.searches_pi.append(self.root.children_as_pi(\n self.root.position.n < self.temp_threshold))\n self.comments.append(self.root.describe())\n try:\n self.root = self.root.maybe_add_child(coords.to_flat(c))\n except go.IllegalMove:\n dbg(\"Illegal move\")\n if not self.two_player_mode:\n self.searches_pi.pop()\n self.comments.pop()\n raise\n\n self.position = self.root.position # for showboard\n del self.root.parent.children\n return True", "response": "Notable side effects:\n - finalizes the probability distribution according to\n this roots visit counts into the class' running tally, `searches_pi`\n - Makes the node associated with this move the root, for future\n `inject_noise` calls."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npicking a move to play based on MCTS readout statistics.", "response": "def pick_move(self):\n \"\"\"Picks a move to play, based on MCTS readout statistics.\n\n Highest N is most robust indicator. In the early stage of the game, pick\n a move weighted by visit count; later on, pick the absolute max.\"\"\"\n if self.root.position.n >= self.temp_threshold:\n fcoord = self.root.best_child()\n else:\n cdf = self.root.children_as_pi(squash=True).cumsum()\n cdf /= cdf[-2] # Prevents passing via softpick.\n selection = random.random()\n fcoord = cdf.searchsorted(selection)\n assert self.root.child_N[fcoord] != 0\n return coords.from_flat(fcoord)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting RecurrentAttention. :param inputs: tensor with inputs :param hidden: hidden state for LSTM layer :param context: context tensor from encoder :param context_len: vector of encoder sequence lengths :returns (rnn_outputs, hidden, attn_output, attn_scores)", "response": "def forward(self, inputs, hidden, context, context_len):\n \"\"\"\n Execute RecurrentAttention.\n\n :param inputs: tensor with inputs\n :param hidden: hidden state for LSTM layer\n :param context: context tensor from encoder\n :param context_len: vector of encoder sequence lengths\n\n :returns (rnn_outputs, hidden, attn_output, attn_scores)\n \"\"\"\n # set attention mask, sequences have different lengths, this mask\n # allows to include only valid elements of context in attention's\n # softmax\n self.attn.set_mask(context_len, context)\n\n inputs = self.dropout(inputs)\n rnn_outputs, hidden = self.rnn(inputs, hidden)\n attn_outputs, scores = self.attn(rnn_outputs, context)\n\n return rnn_outputs, hidden, attn_outputs, scores"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef init_hidden(self, hidden):\n if hidden is not None:\n # per-layer chunks\n hidden = hidden.chunk(self.num_layers)\n # (h, c) chunks for LSTM layer\n hidden = tuple(i.chunk(2) for i in hidden)\n else:\n hidden = [None] * self.num_layers\n\n self.next_hidden = []\n return hidden", "response": "Converts flattened hidden state into a tuple\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef package_hidden(self):\n if self.inference:\n hidden = torch.cat(tuple(itertools.chain(*self.next_hidden)))\n else:\n hidden = None\n return hidden", "response": "Flattens the hidden state from all LSTM layers into one tensor."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef forward(self, inputs, context, inference=False):\n self.inference = inference\n\n enc_context, enc_len, hidden = context\n hidden = self.init_hidden(hidden)\n\n x = self.embedder(inputs)\n\n x, h, attn, scores = self.att_rnn(x, hidden[0], enc_context, enc_len)\n self.append_hidden(h)\n\n x = torch.cat((x, attn), dim=2)\n x = self.dropout(x)\n x, h = self.rnn_layers[0](x, hidden[1])\n self.append_hidden(h)\n\n for i in range(1, len(self.rnn_layers)):\n residual = x\n x = torch.cat((x, attn), dim=2)\n x = self.dropout(x)\n x, h = self.rnn_layers[i](x, hidden[i + 1])\n self.append_hidden(h)\n x = x + residual\n\n x = self.classifier(x)\n hidden = self.package_hidden()\n\n return x, scores, [enc_context, enc_len, hidden]", "response": "Execute the decoder.\n\n :param inputs: tensor with inputs to the decoder\n :param context: state of encoder, encoder sequence lengths and hidden\n state of decoder's LSTM layers\n :param inference: if True stores and repackages hidden state"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cache_url(url, model_dir=None, progress=True):\n if model_dir is None:\n torch_home = os.path.expanduser(os.getenv('TORCH_HOME', '~/.torch'))\n model_dir = os.getenv('TORCH_MODEL_ZOO', os.path.join(torch_home, 'models'))\n if not os.path.exists(model_dir):\n os.makedirs(model_dir)\n parts = urlparse(url)\n filename = os.path.basename(parts.path)\n if filename == \"model_final.pkl\":\n # workaround as pre-trained Caffe2 models from Detectron have all the same filename\n # so make the full path the filename by replacing / with _\n filename = parts.path.replace(\"/\", \"_\")\n cached_file = os.path.join(model_dir, filename)\n if not os.path.exists(cached_file) and is_main_process():\n sys.stderr.write('Downloading: \"{}\" to {}\\n'.format(url, cached_file))\n hash_prefix = HASH_REGEX.search(filename)\n if hash_prefix is not None:\n hash_prefix = hash_prefix.group(1)\n # workaround: Caffe2 models don't have a hash, but follow the R-50 convention,\n # which matches the hash PyTorch uses. So we skip the hash matching\n # if the hash_prefix is less than 6 characters\n if len(hash_prefix) < 6:\n hash_prefix = None\n _download_url_to_file(url, cached_file, hash_prefix, progress=progress)\n synchronize()\n return cached_file", "response": "r Loads the Torch serialized object at the given URL and saves it in the given model_dir."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef split_heads(self, x):\n with tf.name_scope(\"split_heads\"):\n batch_size = tf.shape(x)[0]\n length = tf.shape(x)[1]\n\n # Calculate depth of last dimension after it has been split.\n depth = (self.hidden_size // self.num_heads)\n\n # Split the last dimension\n x = tf.reshape(x, [batch_size, length, self.num_heads, depth])\n\n # Transpose the result\n return tf.transpose(x, [0, 2, 1, 3])", "response": "Splits x into different heads and transpose the resulting value."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef combine_heads(self, x):\n with tf.name_scope(\"combine_heads\"):\n batch_size = tf.shape(x)[0]\n length = tf.shape(x)[2]\n x = tf.transpose(x, [0, 2, 1, 3]) # --> [batch, length, num_heads, depth]\n return tf.reshape(x, [batch_size, length, self.hidden_size])", "response": "Combine tensor that has been split."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\napplies attention mechanism to x and y.", "response": "def call(self, x, y, bias, cache=None):\n \"\"\"Apply attention mechanism to x and y.\n\n Args:\n x: a tensor with shape [batch_size, length_x, hidden_size]\n y: a tensor with shape [batch_size, length_y, hidden_size]\n bias: attention bias that will be added to the result of the dot product.\n cache: (Used during prediction) dictionary with tensors containing results\n of previous attentions. The dictionary must have the items:\n {\"k\": tensor with shape [batch_size, i, key_channels],\n \"v\": tensor with shape [batch_size, i, value_channels]}\n where i is the current decoded length.\n\n Returns:\n Attention layer output with shape [batch_size, length_x, hidden_size]\n \"\"\"\n # Linearly project the query (q), key (k) and value (v) using different\n # learned projections. This is in preparation of splitting them into\n # multiple heads. Multi-head attention uses multiple queries, keys, and\n # values rather than regular attention (which uses a single q, k, v).\n q = self.q_dense_layer(x)\n k = self.k_dense_layer(y)\n v = self.v_dense_layer(y)\n\n if cache is not None:\n # Combine cached keys and values with new keys and values.\n k = tf.concat([cache[\"k\"], k], axis=1)\n v = tf.concat([cache[\"v\"], v], axis=1)\n\n # Update cache\n cache[\"k\"] = k\n cache[\"v\"] = v\n\n # Split q, k, v into heads.\n q = self.split_heads(q)\n k = self.split_heads(k)\n v = self.split_heads(v)\n\n # Scale q to prevent the dot product between q and k from growing too large.\n depth = (self.hidden_size // self.num_heads)\n q *= depth ** -0.5\n\n # Calculate dot product attention\n logits = tf.matmul(q, k, transpose_b=True)\n logits += bias\n weights = tf.nn.softmax(logits, name=\"attention_weights\")\n if self.train:\n mlperf_log.transformer_print(\n key=mlperf_log.MODEL_HP_ATTENTION_DROPOUT,\n value=self.attention_dropout)\n weights = tf.nn.dropout(weights, 1.0 - self.attention_dropout)\n attention_output = tf.matmul(weights, v)\n\n # Recombine heads --> [batch_size, length, hidden_size]\n attention_output = self.combine_heads(attention_output)\n\n # Run the combined outputs through another linear projection layer.\n attention_output = self.output_dense_layer(attention_output)\n return attention_output"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _save_vocab_file(vocab_file, subtoken_list):\n with tf.gfile.Open(vocab_file, mode=\"w\") as f:\n for subtoken in subtoken_list:\n f.write(\"'%s'\\n\" % _unicode_to_native(subtoken))", "response": "Save subtokens to file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _load_vocab_file(vocab_file, reserved_tokens=None):\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n\n subtoken_list = []\n with tf.gfile.Open(vocab_file, mode=\"r\") as f:\n for line in f:\n subtoken = _native_to_unicode(line.strip())\n subtoken = subtoken[1:-1] # Remove surrounding single-quotes\n if subtoken in reserved_tokens:\n continue\n subtoken_list.append(_native_to_unicode(subtoken))\n return reserved_tokens + subtoken_list", "response": "Load vocabulary while ensuring reserved tokens are at the top."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert string to unicode.", "response": "def _native_to_unicode(s):\n \"\"\"Convert string to unicode (required in Python 2).\"\"\"\n if six.PY2:\n return s if isinstance(s, unicode) else s.decode(\"utf-8\")\n else:\n return s"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _unicode_to_native(s):\n if six.PY2:\n return s.encode(\"utf-8\") if isinstance(s, unicode) else s\n else:\n return s", "response": "Convert string from unicode to native format"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsplit text to a list of string tokens.", "response": "def _split_string_to_tokens(text):\n \"\"\"Splits text to a list of string tokens.\"\"\"\n if not text:\n return []\n ret = []\n token_start = 0\n # Classify each character in the input string\n is_alnum = [c in _ALPHANUMERIC_CHAR_SET for c in text]\n for pos in xrange(1, len(text)):\n if is_alnum[pos] != is_alnum[pos - 1]:\n token = text[token_start:pos]\n if token != u\" \" or token_start == 0:\n ret.append(token)\n token_start = pos\n final_token = text[token_start:]\n ret.append(final_token)\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _escape_token(token, alphabet):\n token = token.replace(u\"\\\\\", u\"\\\\\\\\\").replace(u\"_\", u\"\\\\u\")\n ret = [c if c in alphabet and c != u\"\\n\" else r\"\\%d;\" % ord(c) for c in token]\n return u\"\".join(ret) + \"_\"", "response": "Escapes a token in the alphabet and appends _ to the token."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the number of words in the file.", "response": "def _count_tokens(files, file_byte_limit=1e6):\n \"\"\"Return token counts of words in the files.\n\n Samples file_byte_limit bytes from each file, and counts the words that appear\n in the samples. The samples are semi-evenly distributed across the file.\n\n Args:\n files: List of filepaths\n file_byte_limit: Max number of bytes that will be read from each file.\n\n Returns:\n Dictionary mapping tokens to the number of times they appear in the sampled\n lines from the files.\n \"\"\"\n token_counts = collections.defaultdict(int)\n\n for filepath in files:\n with tf.gfile.Open(filepath, mode=\"r\") as reader:\n file_byte_budget = file_byte_limit\n counter = 0\n lines_to_skip = int(reader.size() / (file_byte_budget * 2))\n for line in reader:\n if counter < lines_to_skip:\n counter += 1\n else:\n if file_byte_budget < 0:\n break\n line = line.strip()\n file_byte_budget -= len(line)\n counter = 0\n\n # Add words to token counts\n for token in _split_string_to_tokens(_native_to_unicode(line)):\n token_counts[token] += 1\n return token_counts"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsplitting a token into subtokens defined in the subtoken dict.", "response": "def _split_token_to_subtokens(token, subtoken_dict, max_subtoken_length):\n \"\"\"Splits a token into subtokens defined in the subtoken dict.\"\"\"\n ret = []\n start = 0\n token_len = len(token)\n while start < token_len:\n # Find the longest subtoken, so iterate backwards.\n for end in xrange(min(token_len, start + max_subtoken_length), start, -1):\n subtoken = token[start:end]\n if subtoken in subtoken_dict:\n ret.append(subtoken)\n start = end\n break\n else: # Did not break\n # If there is no possible encoding of the escaped token then one of the\n # characters in the token is not in the alphabet. This should be\n # impossible and would be indicative of a bug.\n raise ValueError(\"Was unable to split token \\\"%s\\\" into subtokens.\" %\n token)\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates subtoken vocabulary with target size.", "response": "def _generate_subtokens_with_target_vocab_size(\n token_counts, alphabet, target_size, threshold, min_count=None,\n reserved_tokens=None):\n \"\"\"Generate subtoken vocabulary close to the target size.\"\"\"\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n\n if min_count is not None:\n tf.logging.info(\"Using min_count=%d to generate vocab with target size %d\" %\n (min_count, target_size))\n return _generate_subtokens(\n token_counts, alphabet, min_count, reserved_tokens=reserved_tokens)\n\n def bisect(min_val, max_val):\n \"\"\"Recursive function to binary search for subtoken vocabulary.\"\"\"\n cur_count = (min_val + max_val) // 2\n tf.logging.info(\"Binary search: trying min_count=%d (%d %d)\" %\n (cur_count, min_val, max_val))\n subtoken_list = _generate_subtokens(\n token_counts, alphabet, cur_count, reserved_tokens=reserved_tokens)\n\n val = len(subtoken_list)\n tf.logging.info(\"Binary search: min_count=%d resulted in %d tokens\" %\n (cur_count, val))\n\n within_threshold = abs(val - target_size) < threshold\n if within_threshold or min_val >= max_val or cur_count < 2:\n return subtoken_list\n if val > target_size:\n other_subtoken_list = bisect(cur_count + 1, max_val)\n else:\n other_subtoken_list = bisect(min_val, cur_count - 1)\n\n # Return vocabulary dictionary with the closest number of tokens.\n other_val = len(other_subtoken_list)\n if abs(other_val - target_size) < abs(val - target_size):\n return other_subtoken_list\n return subtoken_list\n\n tf.logging.info(\"Finding best min_count to get target size of %d\" %\n target_size)\n return bisect(_MIN_MIN_COUNT, _MAX_MIN_COUNT)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _generate_alphabet_dict(iterable, reserved_tokens=None):\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n alphabet = {c for token in iterable for c in token}\n alphabet |= {c for token in reserved_tokens for c in token}\n alphabet |= _ESCAPE_CHARS # Add escape characters to alphabet set.\n return alphabet", "response": "Create set of characters that appear in any element in the iterable."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _count_and_gen_subtokens(\n token_counts, alphabet, subtoken_dict, max_subtoken_length):\n \"\"\"Count number of times subtokens appear, and generate new subtokens.\n\n Args:\n token_counts: dict mapping tokens to the number of times they appear in the\n original files.\n alphabet: list of allowed characters. Used to escape the tokens, which\n guarantees that all tokens can be split into subtokens.\n subtoken_dict: dict mapping subtokens to ids.\n max_subtoken_length: maximum length of subtoken in subtoken_dict.\n\n Returns:\n A defaultdict mapping subtokens to the number of times they appear in the\n tokens. The dict may contain new subtokens.\n \"\"\"\n subtoken_counts = collections.defaultdict(int)\n for token, count in six.iteritems(token_counts):\n token = _escape_token(token, alphabet)\n subtokens = _split_token_to_subtokens(\n token, subtoken_dict, max_subtoken_length)\n\n # Generate new subtokens by taking substrings from token.\n start = 0\n for subtoken in subtokens:\n for end in xrange(start + 1, len(token) + 1):\n new_subtoken = token[start:end]\n subtoken_counts[new_subtoken] += count\n start += len(subtoken)\n\n return subtoken_counts", "response": "Count number of times subtokens appear and generate new subtokens."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of subtokens that are filtered by count.", "response": "def _filter_and_bucket_subtokens(subtoken_counts, min_count):\n \"\"\"Return a bucketed list of subtokens that are filtered by count.\n\n Args:\n subtoken_counts: defaultdict mapping subtokens to their counts\n min_count: int count used to filter subtokens\n\n Returns:\n List of subtoken sets, where subtokens in set i have the same length=i.\n \"\"\"\n # Create list of buckets, where subtokens in bucket i have length i.\n subtoken_buckets = []\n for subtoken, count in six.iteritems(subtoken_counts):\n if count < min_count: # Filter out subtokens that don't appear enough\n continue\n while len(subtoken_buckets) <= len(subtoken):\n subtoken_buckets.append(set())\n subtoken_buckets[len(subtoken)].add(subtoken)\n return subtoken_buckets"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _gen_new_subtoken_list(\n subtoken_counts, min_count, alphabet, reserved_tokens=None):\n \"\"\"Generate candidate subtokens ordered by count, and new max subtoken length.\n\n Add subtokens to the candiate list in order of length (longest subtokens\n first). When a subtoken is added, the counts of each of its prefixes are\n decreased. Prefixes that don't appear much outside the subtoken are not added\n to the candidate list.\n\n For example:\n subtoken being added to candidate list: 'translate'\n subtoken_counts: {'translate':10, 't':40, 'tr':16, 'tra':12, ...}\n min_count: 5\n\n When 'translate' is added, subtoken_counts is updated to:\n {'translate':0, 't':30, 'tr':6, 'tra': 2, ...}\n\n The subtoken 'tra' will not be added to the candidate list, because it appears\n twice (less than min_count) outside of 'translate'.\n\n Args:\n subtoken_counts: defaultdict mapping str subtokens to int counts\n min_count: int minumum count requirement for subtokens\n alphabet: set of characters. Each character is added to the subtoken list to\n guarantee that all tokens can be encoded.\n reserved_tokens: list of tokens that will be added to the beginning of the\n returned subtoken list.\n\n Returns:\n List of candidate subtokens in decreasing count order, and maximum subtoken\n length\n \"\"\"\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n\n # Create a list of (count, subtoken) for each candidate subtoken.\n subtoken_candidates = []\n\n # Use bucketted list to iterate through subtokens in order of length.\n # subtoken_buckets[i] = set(subtokens), where each subtoken has length i.\n subtoken_buckets = _filter_and_bucket_subtokens(subtoken_counts, min_count)\n max_subtoken_length = len(subtoken_buckets) - 1\n\n # Go through the list in reverse order to consider longer subtokens first.\n for subtoken_len in xrange(max_subtoken_length, 0, -1):\n for subtoken in subtoken_buckets[subtoken_len]:\n count = subtoken_counts[subtoken]\n\n # Possible if this subtoken is a prefix of another token.\n if count < min_count:\n continue\n\n # Ignore alphabet/reserved tokens, which will be added manually later.\n if subtoken not in alphabet and subtoken not in reserved_tokens:\n subtoken_candidates.append((count, subtoken))\n\n # Decrement count of the subtoken's prefixes (if a longer subtoken is\n # added, its prefixes lose priority to be added).\n for end in xrange(1, subtoken_len):\n subtoken_counts[subtoken[:end]] -= count\n\n # Add alphabet subtokens (guarantees that all strings are encodable).\n subtoken_candidates.extend((subtoken_counts.get(a, 0), a) for a in alphabet)\n\n # Order subtoken candidates by decreasing count.\n subtoken_list = [t for _, t in sorted(subtoken_candidates, reverse=True)]\n\n # Add reserved tokens to beginning of the list.\n subtoken_list = reserved_tokens + subtoken_list\n return subtoken_list, max_subtoken_length", "response": "Generate a new list of candidate subtokens ordered by count and new max subtoken length."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _generate_subtokens(\n token_counts, alphabet, min_count, num_iterations=4,\n reserved_tokens=None):\n \"\"\"Create a list of subtokens in decreasing order of frequency.\n\n Args:\n token_counts: dict mapping str tokens -> int count\n alphabet: set of characters\n min_count: int minimum number of times a subtoken must appear before it is\n added to the vocabulary.\n num_iterations: int number of iterations to generate new tokens.\n reserved_tokens: list of tokens that will be added to the beginning to the\n returned subtoken list.\n\n Returns:\n Sorted list of subtokens (most frequent first)\n \"\"\"\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n\n # Use alphabet set to create initial list of subtokens\n subtoken_list = reserved_tokens + list(alphabet)\n max_subtoken_length = 1\n\n # On each iteration, segment all words using the subtokens defined in\n # subtoken_dict, count how often the resulting subtokens appear, and update\n # the dictionary with subtokens w/ high enough counts.\n for i in xrange(num_iterations):\n tf.logging.info(\"\\tGenerating subtokens: iteration %d\" % i)\n # Generate new subtoken->id dictionary using the new subtoken list.\n subtoken_dict = _list_to_index_dict(subtoken_list)\n\n # Create dict mapping subtoken->count, with additional subtokens created\n # from substrings taken from the tokens.\n subtoken_counts = _count_and_gen_subtokens(\n token_counts, alphabet, subtoken_dict, max_subtoken_length)\n\n # Generate new list of subtokens sorted by subtoken count.\n subtoken_list, max_subtoken_length = _gen_new_subtoken_list(\n subtoken_counts, min_count, alphabet, reserved_tokens)\n\n tf.logging.info(\"\\tVocab size: %d\" % len(subtoken_list))\n return subtoken_list", "response": "Generate a list of subtokens in decreasing order of frequency."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a subtoken vocabulary from a list of files.", "response": "def init_from_files(\n vocab_file, files, target_vocab_size, threshold, min_count=None,\n file_byte_limit=1e6, reserved_tokens=None):\n \"\"\"Create subtoken vocabulary based on files, and save vocab to file.\n\n Args:\n vocab_file: String name of vocab file to store subtoken vocabulary.\n files: List of file paths that will be used to generate vocabulary.\n target_vocab_size: target vocabulary size to generate.\n threshold: int threshold of vocabulary size to accept.\n min_count: int minimum count to use for generating the vocabulary. The min\n count is the minimum number of times a subtoken should appear in the\n files before it is added to the vocabulary. If set to none, this value\n is found using binary search.\n file_byte_limit: (Default 1e6) Maximum number of bytes of sample text that\n will be drawn from the files.\n reserved_tokens: List of string tokens that are guaranteed to be at the\n beginning of the subtoken vocabulary list.\n\n Returns:\n Subtokenizer object\n \"\"\"\n if reserved_tokens is None:\n reserved_tokens = RESERVED_TOKENS\n\n if tf.gfile.Exists(vocab_file):\n tf.logging.info(\"Vocab file already exists (%s)\" % vocab_file)\n else:\n tf.logging.info(\"Begin steps to create subtoken vocabulary...\")\n token_counts = _count_tokens(files, file_byte_limit)\n alphabet = _generate_alphabet_dict(token_counts)\n subtoken_list = _generate_subtokens_with_target_vocab_size(\n token_counts, alphabet, target_vocab_size, threshold, min_count,\n reserved_tokens)\n tf.logging.info(\"Generated vocabulary with %d subtokens.\" %\n len(subtoken_list))\n mlperf_log.transformer_print(key=mlperf_log.PREPROC_VOCAB_SIZE,\n value=len(subtoken_list))\n _save_vocab_file(vocab_file, subtoken_list)\n return Subtokenizer(vocab_file)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nencoding a string into a list of int subtoken ids.", "response": "def encode(self, raw_string, add_eos=False):\n \"\"\"Encodes a string into a list of int subtoken ids.\"\"\"\n ret = []\n tokens = _split_string_to_tokens(_native_to_unicode(raw_string))\n for token in tokens:\n ret.extend(self._token_to_subtoken_ids(token))\n if add_eos:\n ret.append(EOS_ID)\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nencode a single token into a list of subtoken ids.", "response": "def _token_to_subtoken_ids(self, token):\n \"\"\"Encode a single token into a list of subtoken ids.\"\"\"\n cache_location = hash(token) % self._cache_size\n cache_key, cache_value = self._cache[cache_location]\n if cache_key == token:\n return cache_value\n\n ret = _split_token_to_subtokens(\n _escape_token(token, self.alphabet), self.subtoken_to_id_dict,\n self.max_subtoken_length)\n ret = [self.subtoken_to_id_dict[subtoken_id] for subtoken_id in ret]\n\n self._cache[cache_location] = (token, ret)\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decode(self, subtokens):\n if isinstance(subtokens, np.ndarray):\n # Note that list(subtokens) converts subtokens to a python list, but the\n # items remain as np.int32. This converts both the array and its items.\n subtokens = subtokens.tolist()\n\n if not subtokens:\n return \"\"\n\n assert isinstance(subtokens, list) and isinstance(subtokens[0], int), (\n \"Subtokens argument passed into decode() must be a list of integers.\")\n\n return _unicode_to_native(\n _join_tokens_to_string(self._subtoken_ids_to_tokens(subtokens)))", "response": "Converts list of int subtokens ids into a string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting list of int subtoken ids to list of string tokens.", "response": "def _subtoken_ids_to_tokens(self, subtokens):\n \"\"\"Convert list of int subtoken ids to a list of string tokens.\"\"\"\n escaped_tokens = \"\".join([\n self.subtoken_list[s] for s in subtokens\n if s < len(self.subtoken_list)])\n escaped_tokens = escaped_tokens.split(\"_\")\n\n # All tokens in the vocabulary list have been escaped (see _escape_token())\n # so each token must be unescaped when decoding.\n ret = []\n for token in escaped_tokens:\n if token:\n ret.append(_unescape_token(token))\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun the network on a batch of positions.", "response": "def batch_run_many(player, positions, batch_size=100):\n \"\"\"Used to avoid a memory oveflow issue when running the network\n on too many positions. TODO: This should be a member function of\n player.network?\"\"\"\n prob_list = []\n value_list = []\n for idx in range(0, len(positions), batch_size):\n probs, values = player.network.run_many(positions[idx:idx + batch_size])\n prob_list.append(probs)\n value_list.append(values)\n return np.concatenate(prob_list, axis=0), np.concatenate(value_list, axis=0)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nforwards computation of the mask logits.", "response": "def forward(self, x, boxes):\n \"\"\"\n Arguments:\n x (Tensor): the mask logits\n boxes (list[BoxList]): bounding boxes that are used as\n reference, one for ech image\n\n Returns:\n results (list[BoxList]): one BoxList for each image, containing\n the extra field mask\n \"\"\"\n mask_prob = x.sigmoid()\n\n # select masks coresponding to the predicted classes\n num_masks = x.shape[0]\n labels = [bbox.get_field(\"labels\") for bbox in boxes]\n labels = torch.cat(labels)\n index = torch.arange(num_masks, device=labels.device)\n mask_prob = mask_prob[index, labels][:, None]\n\n boxes_per_image = [len(box) for box in boxes]\n mask_prob = mask_prob.split(boxes_per_image, dim=0)\n\n if self.masker:\n mask_prob = self.masker(mask_prob, boxes)\n\n results = []\n for prob, box in zip(mask_prob, boxes):\n bbox = BoxList(box.bbox, box.size, mode=\"xyxy\")\n for field in box.fields():\n bbox.add_field(field, box.get_field(field))\n bbox.add_field(\"mask\", prob)\n results.append(bbox)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn file. py : lineno of your caller.", "response": "def get_caller(stack_index=2, root_dir=None):\n ''' Returns file.py:lineno of your caller. A stack_index of 2 will provide\n the caller of the function calling this function. Notice that stack_index\n of 2 or more will fail if called from global scope. '''\n caller = inspect.getframeinfo(inspect.stack()[stack_index][0])\n\n # Trim the filenames for readability.\n filename = caller.filename\n if root_dir is not None:\n filename = re.sub(\"^\" + root_dir + \"/\", \"\", filename)\n return \"%s:%d\" % (filename, caller.lineno)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _mlperf_print(key, value=None, benchmark=None, stack_offset=0,\n tag_set=None, deferred=False, root_dir=None,\n extra_print=False, prefix=\"\"):\n ''' Prints out an MLPerf Log Line.\n\n key: The MLPerf log key such as 'CLOCK' or 'QUALITY'. See the list of log keys in the spec.\n value: The value which contains no newlines.\n benchmark: The short code for the benchmark being run, see the MLPerf log spec.\n stack_offset: Increase the value to go deeper into the stack to find the callsite. For example, if this\n is being called by a wraper/helper you may want to set stack_offset=1 to use the callsite\n of the wraper/helper itself.\n tag_set: The set of tags in which key must belong.\n deferred: The value is not presently known. In that case, a unique ID will\n be assigned as the value of this call and will be returned. The\n caller can then include said unique ID when the value is known\n later.\n root_dir: Directory prefix which will be trimmed when reporting calling file\n for compliance logging.\n extra_print: Print a blank line before logging to clear any text in the line.\n prefix: String with which to prefix the log message. Useful for\n differentiating raw lines if stitching will be required.\n\n Example output:\n :::MLP-1537375353 MINGO[17] (eval.py:42) QUALITY: 43.7\n '''\n\n return_value = None\n\n if (tag_set is None and not PATTERN.match(key)) or key not in tag_set:\n raise ValueError('Invalid key for MLPerf print: ' + str(key))\n\n if value is not None and deferred:\n raise ValueError(\"deferred is set to True, but a value was provided\")\n\n if deferred:\n return_value = str(uuid.uuid4())\n value = \"DEFERRED: {}\".format(return_value)\n\n if value is None:\n tag = key\n else:\n str_json = json.dumps(value)\n tag = '{key}: {value}'.format(key=key, value=str_json)\n\n callsite = get_caller(2 + stack_offset, root_dir=root_dir)\n now = time.time()\n\n message = '{prefix}:::MLPv0.5.0 {benchmark} {secs:.9f} ({callsite}) {tag}'.format(\n prefix=prefix, secs=now, benchmark=benchmark, callsite=callsite, tag=tag)\n\n if extra_print:\n print() # There could be prior text on a line\n\n if tag in STDOUT_TAG_SET:\n LOGGER.info(message)\n else:\n LOGGER.debug(message)\n\n return return_value", "response": "Print an MLPerf log line."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a new model name given the model number.", "response": "def generate(model_num):\n \"\"\"Generates a new model name, given the model number.\"\"\"\n if model_num == 0:\n new_name = 'bootstrap'\n else:\n new_name = random.choice(NAMES)\n full_name = \"%06d-%s\" % (model_num, new_name)\n return full_name"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntaking a string related to a model name and extract its model number.", "response": "def detect_model_num(string):\n \"\"\"Takes a string related to a model name and extract its model number.\n\n For example:\n '000000-bootstrap.index' => 0\n \"\"\"\n match = re.match(MODEL_NUM_REGEX, string)\n if match:\n return int(match.group())\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef detect_model_name(string):\n match = re.match(MODEL_NAME_REGEX, string)\n if match:\n return match.group()\n return None", "response": "Takes a string related to a model name and extract its model name."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef batch_norm(inputs, training, data_format):\n # We set fused=True for a significant performance boost. See\n # https://www.tensorflow.org/performance/performance_guide#common_fused_ops\n outputs = tf.layers.batch_normalization(\n inputs=inputs, axis=1 if data_format == 'channels_first' else 3,\n momentum=_BATCH_NORM_DECAY, epsilon=_BATCH_NORM_EPSILON, center=True,\n scale=True, training=training, fused=True)\n\n resnet_log_helper.log_batch_norm(\n input_tensor=inputs, output_tensor=outputs, momentum=_BATCH_NORM_DECAY,\n epsilon=_BATCH_NORM_EPSILON, center=True, scale=True, training=training)\n\n return outputs", "response": "Performs a batch normalization using a standard set of parameters."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fixed_padding(inputs, kernel_size, data_format):\n pad_total = kernel_size - 1\n pad_beg = pad_total // 2\n pad_end = pad_total - pad_beg\n\n if data_format == 'channels_first':\n padded_inputs = tf.pad(inputs, [[0, 0], [0, 0],\n [pad_beg, pad_end], [pad_beg, pad_end]])\n else:\n padded_inputs = tf.pad(inputs, [[0, 0], [pad_beg, pad_end],\n [pad_beg, pad_end], [0, 0]])\n return padded_inputs", "response": "Pads the input along the spatial dimensions independently of input size."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstride 2 - D convolution with explicit padding.", "response": "def conv2d_fixed_padding(inputs, filters, kernel_size, strides, data_format):\n \"\"\"Strided 2-D convolution with explicit padding.\"\"\"\n # The padding is consistent and is based only on `kernel_size`, not on the\n # dimensions of `inputs` (as opposed to using `tf.layers.conv2d` alone).\n\n inputs_for_logging = inputs\n if strides > 1:\n inputs = fixed_padding(inputs, kernel_size, data_format)\n\n outputs = tf.layers.conv2d(\n inputs=inputs, filters=filters, kernel_size=kernel_size, strides=strides,\n padding=('SAME' if strides == 1 else 'VALID'), use_bias=False,\n kernel_initializer=tf.variance_scaling_initializer(\n distribution=\"truncated_normal\"),\n data_format=data_format)\n\n resnet_log_helper.log_conv2d(\n input_tensor=inputs_for_logging, output_tensor=outputs, stride=strides,\n filters=filters, initializer=mlperf_log.TRUNCATED_NORMAL, use_bias=False)\n\n return outputs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds a bottleneck block for ResNet v1.", "response": "def _bottleneck_block_v1(inputs, filters, training, projection_shortcut,\n strides, data_format):\n \"\"\"A single block for ResNet v1, with a bottleneck.\n\n Similar to _building_block_v1(), except using the \"bottleneck\" blocks\n described in:\n Convolution then batch normalization then ReLU as described by:\n Deep Residual Learning for Image Recognition\n https://arxiv.org/pdf/1512.03385.pdf\n by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Dec 2015.\n\n Args:\n inputs: A tensor of size [batch, channels, height_in, width_in] or\n [batch, height_in, width_in, channels] depending on data_format.\n filters: The number of filters for the convolutions.\n training: A Boolean for whether the model is in training or inference\n mode. Needed for batch normalization.\n projection_shortcut: The function to use for projection shortcuts\n (typically a 1x1 convolution when downsampling the input).\n strides: The block's stride. If greater than 1, this block will ultimately\n downsample the input.\n data_format: The input format ('channels_last' or 'channels_first').\n\n Returns:\n The output tensor of the block; shape should match inputs.\n \"\"\"\n resnet_log_helper.log_begin_block(\n input_tensor=inputs, block_type=mlperf_log.BOTTLENECK_BLOCK)\n\n shortcut = inputs\n\n if projection_shortcut is not None:\n shortcut = projection_shortcut(inputs)\n resnet_log_helper.log_projection(input_tensor=inputs,\n output_tensor=shortcut)\n shortcut = batch_norm(inputs=shortcut, training=training,\n data_format=data_format)\n\n inputs = conv2d_fixed_padding(\n inputs=inputs, filters=filters, kernel_size=1, strides=1,\n data_format=data_format)\n inputs = batch_norm(inputs, training, data_format)\n\n mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_RELU)\n inputs = tf.nn.relu(inputs)\n\n inputs = conv2d_fixed_padding(\n inputs=inputs, filters=filters, kernel_size=3, strides=strides,\n data_format=data_format)\n inputs = batch_norm(inputs, training, data_format)\n\n mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_RELU)\n inputs = tf.nn.relu(inputs)\n\n inputs = conv2d_fixed_padding(\n inputs=inputs, filters=4 * filters, kernel_size=1, strides=1,\n data_format=data_format)\n inputs = batch_norm(inputs, training, data_format)\n\n mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_SHORTCUT_ADD)\n inputs += shortcut\n\n mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_RELU)\n inputs = tf.nn.relu(inputs)\n\n resnet_log_helper.log_end_block(output_tensor=inputs)\n return inputs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a block layer for the ResNet model.", "response": "def block_layer(inputs, filters, bottleneck, block_fn, blocks, strides,\n training, name, data_format):\n \"\"\"Creates one layer of blocks for the ResNet model.\n\n Args:\n inputs: A tensor of size [batch, channels, height_in, width_in] or\n [batch, height_in, width_in, channels] depending on data_format.\n filters: The number of filters for the first convolution of the layer.\n bottleneck: Is the block created a bottleneck block.\n block_fn: The block to use within the model, either `building_block` or\n `bottleneck_block`.\n blocks: The number of blocks contained in the layer.\n strides: The stride to use for the first convolution of the layer. If\n greater than 1, this layer will ultimately downsample the input.\n training: Either True or False, whether we are currently training the\n model. Needed for batch norm.\n name: A string name for the tensor output of the block layer.\n data_format: The input format ('channels_last' or 'channels_first').\n\n Returns:\n The output tensor of the block layer.\n \"\"\"\n\n # Bottleneck blocks end with 4x the number of filters as they start with\n filters_out = filters * 4 if bottleneck else filters\n\n def projection_shortcut(inputs):\n return conv2d_fixed_padding(\n inputs=inputs, filters=filters_out, kernel_size=1, strides=strides,\n data_format=data_format)\n\n # Only the first block per block_layer uses projection_shortcut and strides\n inputs = block_fn(inputs, filters, training, projection_shortcut, strides,\n data_format)\n\n for _ in range(1, blocks):\n inputs = block_fn(inputs, filters, training, None, 1, data_format)\n\n return tf.identity(inputs, name)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _custom_dtype_getter(self, getter, name, shape=None, dtype=DEFAULT_DTYPE,\n *args, **kwargs):\n \"\"\"Creates variables in fp32, then casts to fp16 if necessary.\n\n This function is a custom getter. A custom getter is a function with the\n same signature as tf.get_variable, except it has an additional getter\n parameter. Custom getters can be passed as the `custom_getter` parameter of\n tf.variable_scope. Then, tf.get_variable will call the custom getter,\n instead of directly getting a variable itself. This can be used to change\n the types of variables that are retrieved with tf.get_variable.\n The `getter` parameter is the underlying variable getter, that would have\n been called if no custom getter was used. Custom getters typically get a\n variable with `getter`, then modify it in some way.\n\n This custom getter will create an fp32 variable. If a low precision\n (e.g. float16) variable was requested it will then cast the variable to the\n requested dtype. The reason we do not directly create variables in low\n precision dtypes is that applying small gradients to such variables may\n cause the variable not to change.\n\n Args:\n getter: The underlying variable getter, that has the same signature as\n tf.get_variable and returns a variable.\n name: The name of the variable to get.\n shape: The shape of the variable to get.\n dtype: The dtype of the variable to get. Note that if this is a low\n precision dtype, the variable will be created as a tf.float32 variable,\n then cast to the appropriate dtype\n *args: Additional arguments to pass unmodified to getter.\n **kwargs: Additional keyword arguments to pass unmodified to getter.\n\n Returns:\n A variable which is cast to fp16 if necessary.\n \"\"\"\n\n if dtype in CASTABLE_TYPES:\n var = getter(name, shape, tf.float32, *args, **kwargs)\n return tf.cast(var, dtype=dtype, name=name + '_cast')\n else:\n return getter(name, shape, dtype, *args, **kwargs)", "response": "Custom dtype getter for the base class"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new RNG with the current seed depends on current epoch idx.", "response": "def init_rng(self):\n \"\"\"\n Creates new RNG, seed depends on current epoch idx.\n \"\"\"\n rng = torch.Generator()\n seed = self.seeds[self.epoch]\n logging.info(f'Sampler for epoch {self.epoch} uses seed {seed}')\n rng.manual_seed(seed)\n return rng"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef distribute_batches(self, indices):\n assert len(indices) == self.num_samples\n\n indices = indices.view(-1, self.batch_size)\n indices = indices[self.rank::self.world_size].contiguous()\n indices = indices.view(-1)\n indices = indices.tolist()\n\n assert len(indices) == self.num_samples // self.world_size\n return indices", "response": "Assign batches to workers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reshuffle_batches(self, indices, rng):\n indices = indices.view(-1, self.global_batch_size)\n num_batches = indices.shape[0]\n order = torch.randperm(num_batches, generator=rng)\n indices = indices[order, :]\n indices = indices.view(-1)\n return indices", "response": "Reshuffle the global batches."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nforwards computation of the next set of images and features.", "response": "def forward(self, images, features, targets=None):\n \"\"\"\n Arguments:\n images (ImageList): images for which we want to compute the predictions\n features (list[Tensor]): features computed from the images that are\n used for computing the predictions. Each tensor in the list\n correspond to different feature levels\n targets (list[BoxList): ground-truth boxes present in the image (optional)\n\n Returns:\n boxes (list[BoxList]): the predicted boxes from the RPN, one BoxList per\n image.\n losses (dict[Tensor]): the losses for the model during training. During\n testing, it is an empty dict.\n \"\"\"\n box_cls, box_regression = self.head(features)\n anchors = self.anchor_generator(images, features)\n \n if self.training:\n return self._forward_train(anchors, box_cls, box_regression, targets)\n else:\n return self._forward_test(anchors, box_cls, box_regression)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nevaluates all positions with all models save the policy heatmaps as CSVs", "response": "def eval_policy(eval_positions):\n \"\"\"Evaluate all positions with all models save the policy heatmaps as CSVs\n\n CSV name is \"heatmap--.csv\"\n CSV format is: model number, value network output, policy network outputs\n\n position_name is taken from the SGF file\n Policy network outputs (19x19) are saved in flat order (see coord.from_flat)\n \"\"\"\n\n model_paths = oneoff_utils.get_model_paths(fsdb.models_dir())\n\n idx_start = FLAGS.idx_start\n eval_every = FLAGS.eval_every\n\n print(\"Evaluating models {}-{}, eval_every={}\".format(\n idx_start, len(model_paths), eval_every))\n\n player = None\n for i, idx in enumerate(tqdm(range(idx_start, len(model_paths), eval_every))):\n if player and i % 20 == 0:\n player.network.sess.close()\n tf.reset_default_graph()\n player = None\n\n if not player:\n player = oneoff_utils.load_player(model_paths[idx])\n else:\n oneoff_utils.restore_params(model_paths[idx], player)\n\n pos_names, positions = zip(*eval_positions)\n # This should be batched at somepoint.\n eval_probs, eval_values = player.network.run_many(positions)\n\n for pos_name, probs, value in zip(pos_names, eval_probs, eval_values):\n save_file = os.path.join(\n FLAGS.data_dir, \"heatmap-{}-{}.csv\".format(pos_name, idx))\n\n with open(save_file, \"w\") as data:\n data.write(\"{}, {}, {}\\n\".format(\n idx, value, \",\".join(map(str, probs))))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ensure_dir_exists(directory):\n \"Creates local directories if they don't exist.\"\n if directory.startswith('gs://'):\n return\n if not os.path.exists(directory):\n dbg(\"Making dir {}\".format(directory))\n os.makedirs(directory, exist_ok=True)", "response": "Creates local directories if they don t exist."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_game_result(result):\n \"Parse an SGF result string into value target.\"\n if re.match(r'[bB]\\+', result):\n return 1\n if re.match(r'[wW]\\+', result):\n return -1\n return 0", "response": "Parse an SGF result string into value target."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nyield from an iterator in chunks of chunk_size.", "response": "def iter_chunks(chunk_size, iterator):\n \"Yield from an iterator in chunks of chunk_size.\"\n iterator = iter(iterator)\n while True:\n next_chunk = _take_n(chunk_size, iterator)\n # If len(iterable) % chunk_size == 0, don't return an empty chunk.\n if next_chunk:\n yield next_chunk\n else:\n break"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef timer(message):\n \"Context manager for timing snippets of code.\"\n tick = time.time()\n yield\n tock = time.time()\n print(\"%s: %.3f seconds\" % (message, (tock - tick)))", "response": "Context manager for timing snippets of code."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_file(path, filename, max_depth=5):\n for root, dirs, files in os.walk(path):\n if filename in files:\n return os.path.join(root, filename)\n\n # Don't search past max_depth\n depth = root[len(path) + 1:].count(os.sep)\n if depth > max_depth:\n del dirs[:] # Clear dirs\n return None", "response": "Finds a file in path or a subdirectory."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_raw_files(raw_dir, data_source):\n raw_files = {\n \"inputs\": [],\n \"targets\": [],\n } # keys\n for d in data_source:\n input_file, target_file = download_and_extract(\n raw_dir, d[\"url\"], d[\"input\"], d[\"target\"])\n raw_files[\"inputs\"].append(input_file)\n raw_files[\"targets\"].append(target_file)\n return raw_files", "response": "Download and extract raw files from source."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\niterating through lines of file.", "response": "def txt_line_iterator(path):\n \"\"\"Iterate through lines of file.\"\"\"\n with tf.gfile.Open(path) as f:\n for line in f:\n yield line.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncompiles raw files into a single file for each language.", "response": "def compile_files(raw_dir, raw_files, tag):\n \"\"\"Compile raw files into a single file for each language.\n\n Args:\n raw_dir: Directory containing downloaded raw files.\n raw_files: Dict containing filenames of input and target data.\n {\"inputs\": list of files containing data in input language\n \"targets\": list of files containing corresponding data in target language\n }\n tag: String to append to the compiled filename.\n\n Returns:\n Full path of compiled input and target files.\n \"\"\"\n tf.logging.info(\"Compiling files with tag %s.\" % tag)\n filename = \"%s-%s\" % (_PREFIX, tag)\n input_compiled_file = os.path.join(raw_dir, filename + \".lang1\")\n target_compiled_file = os.path.join(raw_dir, filename + \".lang2\")\n\n with tf.gfile.Open(input_compiled_file, mode=\"w\") as input_writer:\n with tf.gfile.Open(target_compiled_file, mode=\"w\") as target_writer:\n for i in range(len(raw_files[\"inputs\"])):\n input_file = raw_files[\"inputs\"][i]\n target_file = raw_files[\"targets\"][i]\n\n tf.logging.info(\"Reading files %s and %s.\" % (input_file, target_file))\n write_file(input_writer, input_file)\n write_file(target_writer, target_file)\n return input_compiled_file, target_compiled_file"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite all of the lines from file using the writer.", "response": "def write_file(writer, filename):\n \"\"\"Write all of lines from file using the writer.\"\"\"\n for line in txt_line_iterator(filename):\n writer.write(line)\n writer.write(\"\\n\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nencode data from files as encoded Examples in TFrecord format.", "response": "def encode_and_save_files(\n subtokenizer, data_dir, raw_files, tag, total_shards):\n \"\"\"Save data from files as encoded Examples in TFrecord format.\n\n Args:\n subtokenizer: Subtokenizer object that will be used to encode the strings.\n data_dir: The directory in which to write the examples\n raw_files: A tuple of (input, target) data files. Each line in the input and\n the corresponding line in target file will be saved in a tf.Example.\n tag: String that will be added onto the file names.\n total_shards: Number of files to divide the data into.\n\n Returns:\n List of all files produced.\n \"\"\"\n # Create a file for each shard.\n filepaths = [shard_filename(data_dir, tag, n + 1, total_shards)\n for n in range(total_shards)]\n\n if all_exist(filepaths):\n tf.logging.info(\"Files with tag %s already exist.\" % tag)\n return filepaths\n\n tf.logging.info(\"Saving files with tag %s.\" % tag)\n input_file = raw_files[0]\n target_file = raw_files[1]\n\n # Write examples to each shard in round robin order.\n tmp_filepaths = [fname + \".incomplete\" for fname in filepaths]\n writers = [tf.python_io.TFRecordWriter(fname) for fname in tmp_filepaths]\n counter, shard = 0, 0\n for counter, (input_line, target_line) in enumerate(zip(\n txt_line_iterator(input_file), txt_line_iterator(target_file))):\n if counter > 0 and counter % 100000 == 0:\n tf.logging.info(\"\\tSaving case %d.\" % counter)\n example = dict_to_example(\n {\"inputs\": subtokenizer.encode(input_line, add_eos=True),\n \"targets\": subtokenizer.encode(target_line, add_eos=True)})\n writers[shard].write(example.SerializeToString())\n shard = (shard + 1) % total_shards\n for writer in writers:\n writer.close()\n\n for tmp_name, final_name in zip(tmp_filepaths, filepaths):\n tf.gfile.Rename(tmp_name, final_name)\n\n tf.logging.info(\"Saved %d Examples\", counter)\n return filepaths"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef shard_filename(path, tag, shard_num, total_shards):\n return os.path.join(\n path, \"%s-%s-%s-%.5d-of-%.5d\" % (_PREFIX, _ENCODE_TAG, tag, shard_num, total_shards))", "response": "Create filename for data shard."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef shuffle_records(fname):\n tf.logging.info(\"Shuffling records in file %s\" % fname)\n\n # Rename file prior to shuffling\n tmp_fname = fname + \".unshuffled\"\n tf.gfile.Rename(fname, tmp_fname)\n\n reader = tf.python_io.tf_record_iterator(tmp_fname)\n records = []\n for record in reader:\n records.append(record)\n if len(records) % 100000 == 0:\n tf.logging.info(\"\\tRead: %d\", len(records))\n\n random.shuffle(records)\n\n # Write shuffled records to original file name\n with tf.python_io.TFRecordWriter(fname) as w:\n for count, record in enumerate(records):\n w.write(record)\n if count > 0 and count % 100000 == 0:\n tf.logging.info(\"\\tWriting record: %d\" % count)\n\n tf.gfile.Remove(tmp_fname)", "response": "Shuffle records in a single file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dict_to_example(dictionary):\n features = {}\n for k, v in six.iteritems(dictionary):\n features[k] = tf.train.Feature(int64_list=tf.train.Int64List(value=v))\n return tf.train.Example(features=tf.train.Features(feature=features))", "response": "Converts a dictionary of string - > int to a tf. Example."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn true if all files in the list exist.", "response": "def all_exist(filepaths):\n \"\"\"Returns true if all files in the list exist.\"\"\"\n for fname in filepaths:\n if not tf.gfile.Exists(fname):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef main(unused_argv):\n tf.logging.set_verbosity(tf.logging.INFO)\n\n make_dir(FLAGS.raw_dir)\n make_dir(FLAGS.data_dir)\n\n # Get paths of download/extracted training and evaluation files.\n tf.logging.info(\"Step 1/4: Downloading data from source\")\n train_files = get_raw_files(FLAGS.raw_dir, _TRAIN_DATA_SOURCES)\n eval_files = get_raw_files(FLAGS.raw_dir, _EVAL_DATA_SOURCES)\n\n # Create subtokenizer based on the training files.\n tf.logging.info(\"Step 2/4: Creating subtokenizer and building vocabulary\")\n train_files_flat = train_files[\"inputs\"] + train_files[\"targets\"]\n vocab_file = os.path.join(FLAGS.data_dir, VOCAB_FILE)\n subtokenizer = tokenizer.Subtokenizer.init_from_files(\n vocab_file, train_files_flat, _TARGET_VOCAB_SIZE, _TARGET_THRESHOLD,\n min_count=None if FLAGS.search else _TRAIN_DATA_MIN_COUNT)\n\n tf.logging.info(\"Step 3/4: Compiling training and evaluation data\")\n compiled_train_files = compile_files(FLAGS.raw_dir, train_files, _TRAIN_TAG)\n compiled_eval_files = compile_files(FLAGS.raw_dir, eval_files, _EVAL_TAG)\n\n # Tokenize and save data as Examples in the TFRecord format.\n tf.logging.info(\"Step 4/4: Preprocessing and saving data\")\n train_tfrecord_files = encode_and_save_files(\n subtokenizer, FLAGS.data_dir, compiled_train_files, _TRAIN_TAG,\n _TRAIN_SHARDS)\n encode_and_save_files(\n subtokenizer, FLAGS.data_dir, compiled_eval_files, _EVAL_TAG,\n _EVAL_SHARDS)\n\n for fname in train_tfrecord_files:\n shuffle_records(fname)", "response": "Main function for the Transformer model."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates continuous representation of inputs.", "response": "def encode(self, inputs, attention_bias):\n \"\"\"Generate continuous representation for inputs.\n\n Args:\n inputs: int tensor with shape [batch_size, input_length].\n attention_bias: float tensor with shape [batch_size, 1, 1, input_length]\n\n Returns:\n float tensor with shape [batch_size, input_length, hidden_size]\n \"\"\"\n with tf.name_scope(\"encode\"):\n # Prepare inputs to the layer stack by adding positional encodings and\n # applying dropout.\n embedded_inputs = self.embedding_softmax_layer(inputs)\n inputs_padding = model_utils.get_padding(inputs)\n\n with tf.name_scope(\"add_pos_encoding\"):\n length = tf.shape(embedded_inputs)[1]\n pos_encoding = model_utils.get_position_encoding(\n length, self.params.hidden_size)\n encoder_inputs = embedded_inputs + pos_encoding\n\n if self.train:\n mlperf_log.transformer_print(\n key=mlperf_log.MODEL_HP_LAYER_POSTPROCESS_DROPOUT,\n value=self.params.layer_postprocess_dropout)\n encoder_inputs = tf.nn.dropout(\n encoder_inputs, 1 - self.params.layer_postprocess_dropout)\n\n return self.encoder_stack(encoder_inputs, attention_bias, inputs_padding)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndecoding the target sequence.", "response": "def decode(self, targets, encoder_outputs, attention_bias):\n \"\"\"Generate logits for each value in the target sequence.\n\n Args:\n targets: target values for the output sequence.\n int tensor with shape [batch_size, target_length]\n encoder_outputs: continuous representation of input sequence.\n float tensor with shape [batch_size, input_length, hidden_size]\n attention_bias: float tensor with shape [batch_size, 1, 1, input_length]\n\n Returns:\n float32 tensor with shape [batch_size, target_length, vocab_size]\n \"\"\"\n with tf.name_scope(\"decode\"):\n # Prepare inputs to decoder layers by shifting targets, adding positional\n # encoding and applying dropout.\n decoder_inputs = self.embedding_softmax_layer(targets)\n with tf.name_scope(\"shift_targets\"):\n # Shift targets to the right, and remove the last element\n decoder_inputs = tf.pad(\n decoder_inputs, [[0, 0], [1, 0], [0, 0]])[:, :-1, :]\n with tf.name_scope(\"add_pos_encoding\"):\n length = tf.shape(decoder_inputs)[1]\n decoder_inputs += model_utils.get_position_encoding(\n length, self.params.hidden_size)\n if self.train:\n mlperf_log.transformer_print(\n key=mlperf_log.MODEL_HP_LAYER_POSTPROCESS_DROPOUT,\n value=self.params.layer_postprocess_dropout)\n decoder_inputs = tf.nn.dropout(\n decoder_inputs, 1 - self.params.layer_postprocess_dropout)\n\n # Run values\n decoder_self_attention_bias = model_utils.get_decoder_self_attention_bias(\n length)\n outputs = self.decoder_stack(\n decoder_inputs, encoder_outputs, decoder_self_attention_bias,\n attention_bias)\n logits = self.embedding_softmax_layer.linear(outputs)\n return logits"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a decoding function that calculates logits of the next tokens.", "response": "def _get_symbols_to_logits_fn(self, max_decode_length):\n \"\"\"Returns a decoding function that calculates logits of the next tokens.\"\"\"\n\n timing_signal = model_utils.get_position_encoding(\n max_decode_length + 1, self.params.hidden_size)\n decoder_self_attention_bias = model_utils.get_decoder_self_attention_bias(\n max_decode_length)\n\n def symbols_to_logits_fn(ids, i, cache):\n \"\"\"Generate logits for next potential IDs.\n\n Args:\n ids: Current decoded sequences.\n int tensor with shape [batch_size * beam_size, i + 1]\n i: Loop index\n cache: dictionary of values storing the encoder output, encoder-decoder\n attention bias, and previous decoder attention values.\n\n Returns:\n Tuple of\n (logits with shape [batch_size * beam_size, vocab_size],\n updated cache values)\n \"\"\"\n # Set decoder input to the last generated IDs\n decoder_input = ids[:, -1:]\n\n # Preprocess decoder input by getting embeddings and adding timing signal.\n decoder_input = self.embedding_softmax_layer(decoder_input)\n decoder_input += timing_signal[i:i + 1]\n\n self_attention_bias = decoder_self_attention_bias[:, :, i:i + 1, :i + 1]\n decoder_outputs = self.decoder_stack(\n decoder_input, cache.get(\"encoder_outputs\"), self_attention_bias,\n cache.get(\"encoder_decoder_attention_bias\"), cache)\n logits = self.embedding_softmax_layer.linear(decoder_outputs)\n logits = tf.squeeze(logits, axis=[1])\n return logits, cache\n return symbols_to_logits_fn"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef predict(self, encoder_outputs, encoder_decoder_attention_bias):\n batch_size = tf.shape(encoder_outputs)[0]\n input_length = tf.shape(encoder_outputs)[1]\n max_decode_length = input_length + self.params.extra_decode_length\n\n symbols_to_logits_fn = self._get_symbols_to_logits_fn(max_decode_length)\n\n # Create initial set of IDs that will be passed into symbols_to_logits_fn.\n initial_ids = tf.zeros([batch_size], dtype=tf.int32)\n\n # Create cache storing decoder attention values for each layer.\n cache = {\n \"layer_%d\" % layer: {\n \"k\": tf.zeros([batch_size, 0, self.params.hidden_size]),\n \"v\": tf.zeros([batch_size, 0, self.params.hidden_size]),\n } for layer in range(self.params.num_hidden_layers)}\n\n # Add encoder output and attention bias to the cache.\n cache[\"encoder_outputs\"] = encoder_outputs\n cache[\"encoder_decoder_attention_bias\"] = encoder_decoder_attention_bias\n\n # Use beam search to find the top beam_size sequences and scores.\n mlperf_log.transformer_print(\n key=mlperf_log.MODEL_HP_SEQ_BEAM_SEARCH,\n value={\n \"vocab_size\": self.params.vocab_size,\n \"beam_size\": self.params.beam_size,\n \"alpha\": self.params.alpha,\n \"extra_decode_length\": self.params.extra_decode_length})\n decoded_ids, scores = beam_search.sequence_beam_search(\n symbols_to_logits_fn=symbols_to_logits_fn,\n initial_ids=initial_ids,\n initial_cache=cache,\n vocab_size=self.params.vocab_size,\n beam_size=self.params.beam_size,\n alpha=self.params.alpha,\n max_decode_length=max_decode_length,\n eos_id=EOS_ID)\n\n # Get the top sequence for each batch element\n top_decoded_ids = decoded_ids[:, 0, 1:]\n top_scores = scores[:, 0]\n\n return {\"outputs\": top_decoded_ids, \"scores\": top_scores}", "response": "Predict the sequence of the encoder outputs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _create_row_col_indices(ratings_df):\n user_id_to_user_idx = _create_index(ratings_df, \"userId\")\n item_id_to_item_idx = _create_index(ratings_df, \"movieId\")\n\n ratings_df[\"row\"] = ratings_df[\"userId\"].apply(\n lambda x: user_id_to_user_idx[x])\n ratings_df[\"col\"] = ratings_df[\"movieId\"].apply(\n lambda x: item_id_to_item_idx[x])\n\n return ratings_df", "response": "Maps user and item ids to their locations in the rating matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _preprocess_movie_lens(ratings_df):\n ratings_df[\"data\"] = 1.0\n num_timestamps = ratings_df[[\"userId\", \"timestamp\"]].groupby(\n \"userId\").nunique()\n last_user_timestamp = ratings_df[[\"userId\", \"timestamp\"]].groupby(\n \"userId\").max()\n\n ratings_df[\"numberOfTimestamps\"] = ratings_df[\"userId\"].apply(\n lambda x: num_timestamps[\"timestamp\"][x])\n ratings_df[\"lastTimestamp\"] = ratings_df[\"userId\"].apply(\n lambda x: last_user_timestamp[\"timestamp\"][x])\n\n ratings_df = ratings_df[ratings_df[\"numberOfTimestamps\"] > 2]\n\n ratings_df = _create_row_col_indices(ratings_df)\n\n train_ratings_df = ratings_df[\n ratings_df[\"timestamp\"] < ratings_df[\"lastTimestamp\"]]\n test_ratings_df = ratings_df[\n ratings_df[\"timestamp\"] == ratings_df[\"lastTimestamp\"]]\n\n return ratings_df, train_ratings_df, test_ratings_df", "response": "Separate the rating datafram into train set and test set."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getAnnIds(self, imgIds=[], catIds=[], areaRng=[], iscrowd=None):\n imgIds = imgIds if _isArrayLike(imgIds) else [imgIds]\n catIds = catIds if _isArrayLike(catIds) else [catIds]\n\n if len(imgIds) == len(catIds) == len(areaRng) == 0:\n anns = self.dataset['annotations']\n else:\n if not len(imgIds) == 0:\n lists = [self.imgToAnns[imgId] for imgId in imgIds if imgId in self.imgToAnns]\n anns = list(itertools.chain.from_iterable(lists))\n else:\n anns = self.dataset['annotations']\n anns = anns if len(catIds) == 0 else [ann for ann in anns if ann['category_id'] in catIds]\n anns = anns if len(areaRng) == 0 else [ann for ann in anns if ann['area'] > areaRng[0] and ann['area'] < areaRng[1]]\n if not iscrowd == None:\n ids = [ann['id'] for ann in anns if ann['iscrowd'] == iscrowd]\n else:\n ids = [ann['id'] for ann in anns]\n return ids", "response": "Get an ann ids that satisfy given filter conditions. default skips that filter\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the ids of the given categories", "response": "def getCatIds(self, catNms=[], supNms=[], catIds=[]):\n \"\"\"\n filtering parameters. default skips that filter.\n :param catNms (str array) : get cats for given cat names\n :param supNms (str array) : get cats for given supercategory names\n :param catIds (int array) : get cats for given cat ids\n :return: ids (int array) : integer array of cat ids\n \"\"\"\n catNms = catNms if _isArrayLike(catNms) else [catNms]\n supNms = supNms if _isArrayLike(supNms) else [supNms]\n catIds = catIds if _isArrayLike(catIds) else [catIds]\n\n if len(catNms) == len(supNms) == len(catIds) == 0:\n cats = self.dataset['categories']\n else:\n cats = self.dataset['categories']\n cats = cats if len(catNms) == 0 else [cat for cat in cats if cat['name'] in catNms]\n cats = cats if len(supNms) == 0 else [cat for cat in cats if cat['supercategory'] in supNms]\n cats = cats if len(catIds) == 0 else [cat for cat in cats if cat['id'] in catIds]\n ids = [cat['id'] for cat in cats]\n return ids"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getImgIds(self, imgIds=[], catIds=[]):\n '''\n Get img ids that satisfy given filter conditions.\n :param imgIds (int array) : get imgs for given ids\n :param catIds (int array) : get imgs with all given cats\n :return: ids (int array) : integer array of img ids\n '''\n imgIds = imgIds if _isArrayLike(imgIds) else [imgIds]\n catIds = catIds if _isArrayLike(catIds) else [catIds]\n\n if len(imgIds) == len(catIds) == 0:\n ids = self.imgs.keys()\n else:\n ids = set(imgIds)\n for i, catId in enumerate(catIds):\n if i == 0 and len(ids) == 0:\n ids = set(self.catToImgs[catId])\n else:\n ids &= set(self.catToImgs[catId])\n return list(ids)", "response": "Get img ids that satisfy given filter conditions."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload anns with the specified ids.", "response": "def loadAnns(self, ids=[]):\n \"\"\"\n Load anns with the specified ids.\n :param ids (int array) : integer ids specifying anns\n :return: anns (object array) : loaded ann objects\n \"\"\"\n if _isArrayLike(ids):\n return [self.anns[id] for id in ids]\n elif type(ids) == int:\n return [self.anns[ids]]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef loadCats(self, ids=[]):\n if _isArrayLike(ids):\n return [self.cats[id] for id in ids]\n elif type(ids) == int:\n return [self.cats[ids]]", "response": "Load cats with the specified ids."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef loadImgs(self, ids=[]):\n if _isArrayLike(ids):\n return [self.imgs[id] for id in ids]\n elif type(ids) == int:\n return [self.imgs[ids]]", "response": "Load an img objects with the specified ids."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndisplay the specified annotations.", "response": "def showAnns(self, anns):\n \"\"\"\n Display the specified annotations.\n :param anns (array of object): annotations to display\n :return: None\n \"\"\"\n if len(anns) == 0:\n return 0\n if 'segmentation' in anns[0] or 'keypoints' in anns[0]:\n datasetType = 'instances'\n elif 'caption' in anns[0]:\n datasetType = 'captions'\n else:\n raise Exception('datasetType not supported')\n if datasetType == 'instances':\n ax = plt.gca()\n ax.set_autoscale_on(False)\n polygons = []\n color = []\n for ann in anns:\n c = (np.random.random((1, 3))*0.6+0.4).tolist()[0]\n if 'segmentation' in ann:\n if type(ann['segmentation']) == list:\n # polygon\n for seg in ann['segmentation']:\n poly = np.array(seg).reshape((int(len(seg)/2), 2))\n polygons.append(Polygon(poly))\n color.append(c)\n else:\n # mask\n t = self.imgs[ann['image_id']]\n if type(ann['segmentation']['counts']) == list:\n rle = maskUtils.frPyObjects([ann['segmentation']], t['height'], t['width'])\n else:\n rle = [ann['segmentation']]\n m = maskUtils.decode(rle)\n img = np.ones( (m.shape[0], m.shape[1], 3) )\n if ann['iscrowd'] == 1:\n color_mask = np.array([2.0,166.0,101.0])/255\n if ann['iscrowd'] == 0:\n color_mask = np.random.random((1, 3)).tolist()[0]\n for i in range(3):\n img[:,:,i] = color_mask[i]\n ax.imshow(np.dstack( (img, m*0.5) ))\n if 'keypoints' in ann and type(ann['keypoints']) == list:\n # turn skeleton into zero-based index\n sks = np.array(self.loadCats(ann['category_id'])[0]['skeleton'])-1\n kp = np.array(ann['keypoints'])\n x = kp[0::3]\n y = kp[1::3]\n v = kp[2::3]\n for sk in sks:\n if np.all(v[sk]>0):\n plt.plot(x[sk],y[sk], linewidth=3, color=c)\n plt.plot(x[v>0], y[v>0],'o',markersize=8, markerfacecolor=c, markeredgecolor='k',markeredgewidth=2)\n plt.plot(x[v>1], y[v>1],'o',markersize=8, markerfacecolor=c, markeredgecolor=c, markeredgewidth=2)\n p = PatchCollection(polygons, facecolor=color, linewidths=0, alpha=0.4)\n ax.add_collection(p)\n p = PatchCollection(polygons, facecolor='none', edgecolors=color, linewidths=2)\n ax.add_collection(p)\n elif datasetType == 'captions':\n for ann in anns:\n print(ann['caption'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload a result file and return a result api object.", "response": "def loadRes(self, resFile):\n \"\"\"\n Load result file and return a result api object.\n :param resFile (str) : file name of result file\n :return: res (obj) : result api object\n \"\"\"\n res = COCO()\n res.dataset['images'] = [img for img in self.dataset['images']]\n\n print('Loading and preparing results...')\n tic = time.time()\n if type(resFile) == str: #or type(resFile) == unicode:\n anns = json.load(open(resFile))\n elif type(resFile) == np.ndarray:\n anns = self.loadNumpyAnnotations(resFile)\n else:\n anns = resFile\n assert type(anns) == list, 'results in not an array of objects'\n annsImgIds = [ann['image_id'] for ann in anns]\n assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \\\n 'Results do not correspond to current coco set'\n if 'caption' in anns[0]:\n imgIds = set([img['id'] for img in res.dataset['images']]) & set([ann['image_id'] for ann in anns])\n res.dataset['images'] = [img for img in res.dataset['images'] if img['id'] in imgIds]\n for id, ann in enumerate(anns):\n ann['id'] = id+1\n elif 'bbox' in anns[0] and not anns[0]['bbox'] == []:\n res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n for id, ann in enumerate(anns):\n bb = ann['bbox']\n x1, x2, y1, y2 = [bb[0], bb[0]+bb[2], bb[1], bb[1]+bb[3]]\n if not 'segmentation' in ann:\n ann['segmentation'] = [[x1, y1, x1, y2, x2, y2, x2, y1]]\n ann['area'] = bb[2]*bb[3]\n ann['id'] = id+1\n ann['iscrowd'] = 0\n elif 'segmentation' in anns[0]:\n res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n for id, ann in enumerate(anns):\n # now only support compressed RLE format as segmentation results\n ann['area'] = maskUtils.area(ann['segmentation'])\n if not 'bbox' in ann:\n ann['bbox'] = maskUtils.toBbox(ann['segmentation'])\n ann['id'] = id+1\n ann['iscrowd'] = 0\n elif 'keypoints' in anns[0]:\n res.dataset['categories'] = copy.deepcopy(self.dataset['categories'])\n for id, ann in enumerate(anns):\n s = ann['keypoints']\n x = s[0::3]\n y = s[1::3]\n x0,x1,y0,y1 = np.min(x), np.max(x), np.min(y), np.max(y)\n ann['area'] = (x1-x0)*(y1-y0)\n ann['id'] = id + 1\n ann['bbox'] = [x0,y0,x1-x0,y1-y0]\n print('DONE (t={:0.2f}s)'.format(time.time()- tic))\n\n res.dataset['annotations'] = anns\n res.createIndex()\n return res"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert annotation which can be polygons uncompressed RLE to binary mask", "response": "def annToRLE(self, ann):\n \"\"\"\n Convert annotation which can be polygons, uncompressed RLE to RLE.\n :return: binary mask (numpy 2D array)\n \"\"\"\n t = self.imgs[ann['image_id']]\n h, w = t['height'], t['width']\n segm = ann['segmentation']\n if type(segm) == list:\n # polygon -- a single object might consist of multiple parts\n # we merge all parts into one mask rle code\n rles = maskUtils.frPyObjects(segm, h, w)\n rle = maskUtils.merge(rles)\n elif type(segm['counts']) == list:\n # uncompressed RLE\n rle = maskUtils.frPyObjects(segm, h, w)\n else:\n # rle\n rle = ann['segmentation']\n return rle"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef annToMask(self, ann):\n rle = self.annToRLE(ann)\n m = maskUtils.decode(rle)\n return m", "response": "Convert an annotation which can be polygons uncompressed RLE or RLE to binary mask."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef forward(self, images, targets=None):\n if self.training and targets is None:\n raise ValueError(\"In training mode, targets should be passed\")\n images = to_image_list(images)\n features = self.backbone(images.tensors)\n proposals, proposal_losses = self.rpn(images, features, targets)\n if self.roi_heads:\n x, result, detector_losses = self.roi_heads(features, proposals, targets)\n else:\n # RPN-only models don't have roi_heads\n x = features\n result = proposals\n detector_losses = {}\n\n if self.training:\n losses = {}\n losses.update(detector_losses)\n losses.update(proposal_losses)\n return losses\n\n return result", "response": "Forward the model to the next set of images and targets."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sparse_svd(sparse_matrix, num_values, max_iter):\n\n if num_values <= 0:\n raise ValueError(\"num_values should be > 0 but instead is %d.\" % num_values)\n\n if max_iter is not None and max_iter < 0:\n raise ValueError(\"max_iter should be >= 0 but instead is %d.\" % max_iter)\n\n if max_iter is None:\n max_iter = FLAGS.max_iter_sparse_svd\n elif not max_iter:\n max_iter = None\n\n u, s, v = linalg.svds(\n sparse_matrix, k=num_values, maxiter=max_iter,\n return_singular_vectors=True)\n\n return (u, s, v)", "response": "Wrapper around SciPy s Singular Value Decomposition for sparse matrices."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef call(self, x):\n with tf.name_scope(\"embedding\"):\n embeddings = tf.gather(self.shared_weights, x)\n\n # Scale embedding by the sqrt of the hidden size\n embeddings *= self.hidden_size ** 0.5\n\n # Create binary array of size [batch_size, length]\n # where 1 = padding, 0 = not padding\n padding = model_utils.get_padding(x)\n\n # Set all padding embedding values to 0\n embeddings *= tf.expand_dims(1 - padding, -1)\n return embeddings", "response": "Get token embeddings of x."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef linear(self, x):\n with tf.name_scope(\"presoftmax_linear\"):\n batch_size = tf.shape(x)[0]\n length = tf.shape(x)[1]\n\n x = tf.reshape(x, [-1, self.hidden_size])\n logits = tf.matmul(x, self.shared_weights, transpose_b=True)\n\n return tf.reshape(logits, [batch_size, length, self.vocab_size])", "response": "Computes logits by running x through a linear layer."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform one iteration of the training and validation of the language.", "response": "def iterate(self, src, tgt, update=True, training=True):\n \"\"\"\n Performs one iteration of the training/validation.\n\n :param src: batch of examples from the source language\n :param tgt: batch of examples from the target language\n :param update: if True: optimizer does update of the weights\n :param training: if True: executes optimizer\n \"\"\"\n src, src_length = src\n tgt, tgt_length = tgt\n src_length = torch.LongTensor(src_length)\n tgt_length = torch.LongTensor(tgt_length)\n\n num_toks = {}\n num_toks['tgt'] = int(sum(tgt_length - 1))\n num_toks['src'] = int(sum(src_length))\n\n if self.cuda:\n src = src.cuda()\n src_length = src_length.cuda()\n tgt = tgt.cuda()\n\n if self.batch_first:\n output = self.model(src, src_length, tgt[:, :-1])\n tgt_labels = tgt[:, 1:]\n T, B = output.size(1), output.size(0)\n else:\n output = self.model(src, src_length, tgt[:-1])\n tgt_labels = tgt[1:]\n T, B = output.size(0), output.size(1)\n\n loss = self.criterion(output.view(T * B, -1),\n tgt_labels.contiguous().view(-1))\n\n loss_per_batch = loss.item()\n loss /= (B * self.iter_size)\n\n if training:\n self.fp_optimizer.step(loss, self.optimizer, self.scheduler,\n update)\n\n loss_per_token = loss_per_batch / num_toks['tgt']\n loss_per_sentence = loss_per_batch / B\n\n return loss_per_token, loss_per_sentence, num_toks"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nruns training or validation on batches of data.", "response": "def feed_data(self, data_loader, training=True):\n \"\"\"\n Runs training or validation on batches from data_loader.\n\n :param data_loader: data loader\n :param training: if True runs training else runs validation\n \"\"\"\n if training:\n assert self.optimizer is not None\n eval_fractions = np.linspace(0, 1, self.intra_epoch_eval+2)[1:-1]\n iters_with_update = len(data_loader) // self.iter_size\n eval_iters = (eval_fractions * iters_with_update).astype(int)\n eval_iters = eval_iters * self.iter_size\n eval_iters = set(eval_iters)\n\n batch_time = AverageMeter()\n data_time = AverageMeter()\n losses_per_token = AverageMeter(skip_first=False)\n losses_per_sentence = AverageMeter(skip_first=False)\n\n tot_tok_time = AverageMeter()\n src_tok_time = AverageMeter()\n tgt_tok_time = AverageMeter()\n\n batch_size = data_loader.batch_size\n\n end = time.time()\n for i, (src, tgt) in enumerate(data_loader):\n self.save_counter += 1\n # measure data loading time\n data_time.update(time.time() - end)\n\n update = False\n if i % self.iter_size == self.iter_size - 1:\n update = True\n\n # do a train/evaluate iteration\n stats = self.iterate(src, tgt, update, training=training)\n loss_per_token, loss_per_sentence, num_toks = stats\n\n # measure accuracy and record loss\n losses_per_token.update(loss_per_token, num_toks['tgt'])\n losses_per_sentence.update(loss_per_sentence, batch_size)\n\n # measure elapsed time\n elapsed = time.time() - end\n batch_time.update(elapsed)\n src_tok_time.update(num_toks['src'] / elapsed)\n tgt_tok_time.update(num_toks['tgt'] / elapsed)\n tot_num_toks = num_toks['tgt'] + num_toks['src']\n tot_tok_time.update(tot_num_toks / elapsed)\n self.loss = losses_per_token.avg\n\n if training and i in eval_iters:\n test_bleu, _ = self.translator.run(calc_bleu=True,\n epoch=self.epoch,\n iteration=i)\n\n log = []\n log += [f'TRAIN [{self.epoch}][{i}/{len(data_loader)}]']\n log += [f'BLEU: {test_bleu:.2f}']\n log = '\\t'.join(log)\n logging.info(log)\n\n self.model.train()\n self.preallocate(data_loader, training=True)\n\n if i % self.print_freq == 0:\n phase = 'TRAIN' if training else 'VALIDATION'\n log = []\n log += [f'{phase} [{self.epoch}][{i}/{len(data_loader)}]']\n log += [f'Time {batch_time.val:.3f} ({batch_time.avg:.3f})']\n log += [f'Data {data_time.val:.2e} ({data_time.avg:.2e})']\n log += [f'Tok/s {tot_tok_time.val:.0f} ({tot_tok_time.avg:.0f})']\n if self.verbose:\n log += [f'Src tok/s {src_tok_time.val:.0f} ({src_tok_time.avg:.0f})']\n log += [f'Tgt tok/s {tgt_tok_time.val:.0f} ({tgt_tok_time.avg:.0f})']\n log += [f'Loss/sentence {losses_per_sentence.val:.1f} ({losses_per_sentence.avg:.1f})']\n log += [f'Loss/tok {losses_per_token.val:.4f} ({losses_per_token.avg:.4f})']\n if training:\n lr = self.optimizer.param_groups[0]['lr']\n log += [f'LR {lr:.3e}']\n log = '\\t'.join(log)\n logging.info(log)\n\n save_chkpt = (self.save_counter % self.save_freq) == (self.save_freq - 1)\n if training and save_chkpt:\n self.save_counter = 0\n self.save_info['iteration'] = i\n identifier = next(self.checkpoint_counter, -1)\n if identifier != -1:\n with sync_workers() as rank:\n if rank == 0:\n self.save(identifier=identifier)\n\n end = time.time()\n\n tot_tok_time.reduce('sum')\n losses_per_token.reduce('mean')\n\n return losses_per_token.avg, tot_tok_time.avg"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef preallocate(self, data_loader, training):\n batch_size = data_loader.batch_size\n max_len = data_loader.dataset.max_len\n\n src_length = [max_len] * batch_size\n tgt_length = [max_len] * batch_size\n\n if self.batch_first:\n shape = (batch_size, max_len)\n else:\n shape = (max_len, batch_size)\n\n src = torch.full(shape, 4, dtype=torch.int64)\n tgt = torch.full(shape, 4, dtype=torch.int64)\n src = src, src_length\n tgt = tgt, tgt_length\n self.iterate(src, tgt, update=False, training=training)\n self.model.zero_grad()", "response": "Preallocate memory for the current cluster entry."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset model in training mode preallocates memory runs training on data provided by data_loader.", "response": "def optimize(self, data_loader):\n \"\"\"\n Sets model in training mode, preallocates memory and runs training on\n data provided by data_loader.\n\n :param data_loader: data loader\n \"\"\"\n torch.set_grad_enabled(True)\n self.model.train()\n torch.cuda.empty_cache()\n self.preallocate(data_loader, training=True)\n output = self.feed_data(data_loader, training=True)\n self.model.zero_grad()\n torch.cuda.empty_cache()\n return output"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef evaluate(self, data_loader):\n torch.set_grad_enabled(False)\n self.model.eval()\n torch.cuda.empty_cache()\n self.preallocate(data_loader, training=False)\n output = self.feed_data(data_loader, training=False)\n self.model.zero_grad()\n torch.cuda.empty_cache()\n return output", "response": "Evaluate the model on the data provided by data_loader."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(self, filename):\n if os.path.isfile(filename):\n checkpoint = torch.load(filename, map_location={'cuda:0': 'cpu'})\n if self.distributed:\n self.model.module.load_state_dict(checkpoint['state_dict'])\n else:\n self.model.load_state_dict(checkpoint['state_dict'])\n self.fp_optimizer.initialize_model(self.model)\n self.optimizer.load_state_dict(checkpoint['optimizer'])\n self.scheduler.load_state_dict(checkpoint['scheduler'])\n self.epoch = checkpoint['epoch']\n self.loss = checkpoint['loss']\n logging.info(f'Loaded checkpoint {filename} (epoch {self.epoch})')\n else:\n logging.error(f'Invalid checkpoint: {filename}')", "response": "Loads the current state of the current instance from a checkpoint file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsaving the current state of the model to a file.", "response": "def save(self, identifier=None, is_best=False, save_all=False):\n \"\"\"\n Stores checkpoint to a file.\n\n :param identifier: identifier for periodic checkpoint\n :param is_best: if True stores checkpoint to 'model_best.pth'\n :param save_all: if True stores checkpoint after completed training\n epoch\n \"\"\"\n\n def write_checkpoint(state, filename):\n filename = os.path.join(self.save_path, filename)\n logging.info(f'Saving model to {filename}')\n torch.save(state, filename)\n\n if self.distributed:\n model_state = self.model.module.state_dict()\n else:\n model_state = self.model.state_dict()\n\n state = {\n 'epoch': self.epoch,\n 'state_dict': model_state,\n 'optimizer': self.optimizer.state_dict(),\n 'scheduler': self.scheduler.state_dict(),\n 'loss': getattr(self, 'loss', None),\n }\n state = dict(list(state.items()) + list(self.save_info.items()))\n\n if identifier is not None:\n filename = self.checkpoint_filename % identifier\n write_checkpoint(state, filename)\n\n if is_best:\n filename = 'model_best.pth'\n write_checkpoint(state, filename)\n\n if save_all:\n filename = f'checkpoint_epoch_{self.epoch:03d}.pth'\n write_checkpoint(state, filename)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef keep_only_positive_boxes(boxes):\n assert isinstance(boxes, (list, tuple))\n assert isinstance(boxes[0], BoxList)\n assert boxes[0].has_field(\"labels\")\n positive_boxes = []\n positive_inds = []\n num_boxes = 0\n for boxes_per_image in boxes:\n labels = boxes_per_image.get_field(\"labels\")\n inds_mask = labels > 0\n inds = inds_mask.nonzero().squeeze(1)\n positive_boxes.append(boxes_per_image[inds])\n positive_inds.append(inds_mask)\n return positive_boxes, positive_inds", "response": "Returns a set of BoxList for which labels > 0."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nforwarding method for the base class.", "response": "def forward(self, features, proposals, targets=None):\n \"\"\"\n Arguments:\n features (list[Tensor]): feature-maps from possibly several levels\n proposals (list[BoxList]): proposal boxes\n targets (list[BoxList], optional): the ground-truth targets.\n\n Returns:\n x (Tensor): the result of the feature extractor\n proposals (list[BoxList]): during training, the original proposals\n are returned. During testing, the predicted boxlists are returned\n with the `mask` field set\n losses (dict[Tensor]): During training, returns the losses for the\n head. During testing, returns an empty dict.\n \"\"\"\n\n if self.training:\n # during training, only focus on positive boxes\n all_proposals = proposals\n proposals, positive_inds = keep_only_positive_boxes(proposals)\n if self.training and self.cfg.MODEL.ROI_MASK_HEAD.SHARE_BOX_FEATURE_EXTRACTOR:\n x = features\n x = x[torch.cat(positive_inds, dim=0)]\n else:\n x = self.feature_extractor(features, proposals)\n mask_logits = self.predictor(x)\n\n if not self.training:\n result = self.post_processor(mask_logits, proposals)\n return x, result, {}\n\n loss_mask = self.loss_evaluator(proposals, mask_logits, targets)\n\n return x, all_proposals, dict(loss_mask=loss_mask)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_95_percentile_bleak(games_nr, n_back=500):\n end_game = int(games_nr.latest_game_number)\n start_game = end_game - n_back if end_game >= n_back else 0\n moves = games_nr.bleakest_moves(start_game, end_game)\n evals = np.array([m[2] for m in moves])\n return np.percentile(evals, 5)", "response": "Gets the 95th percentile of bleakest_eval from bigtable"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_flagfile(flags_path, new_threshold):\n if abs(new_threshold) > 1:\n raise ValueError(\"Invalid new percentile for resign threshold\")\n with tf.gfile.GFile(flags_path) as f:\n lines = f.read()\n if new_threshold > 0:\n new_threshold *= -1\n if not RESIGN_FLAG_REGEX.search(lines):\n print(\"Resign threshold flag not found in flagfile {}! Aborting.\".format(flags_path))\n sys.exit(1)\n old_threshold = RESIGN_FLAG_REGEX.search(lines).groups(1)\n lines = re.sub(RESIGN_FLAG_REGEX, \"--resign_threshold={:.3f}\".format(new_threshold), lines)\n\n if abs(float(old_threshold[0]) - new_threshold) < 0.001:\n print(\"Not updating percentiles; {} ~= {:.3f}\".format(\n old_threshold[0], new_threshold), flush=True)\n else:\n print(\"Updated percentile from {} to {:.3f}\".format(\n old_threshold[0], new_threshold), flush=True)\n with tf.gfile.GFile(flags_path, 'w') as f:\n f.write(lines)", "response": "Updates the flagfile at flags_path to change the value for\n resign_threshold to new_threshold."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting the current search status to stderr.", "response": "def _minigui_report_search_status(self, leaves):\n \"\"\"Prints the current MCTS search status to stderr.\n\n Reports the current search path, root node's child_Q, root node's\n child_N, the most visited path in a format that can be parsed by\n one of the STDERR_HANDLERS in minigui.ts.\n\n Args:\n leaves: list of leaf MCTSNodes returned by tree_search().\n \"\"\"\n\n root = self._player.get_root()\n\n msg = {\n \"id\": hex(id(root)),\n \"n\": int(root.N),\n \"q\": float(root.Q),\n }\n\n msg[\"childQ\"] = [int(round(q * 1000)) for q in root.child_Q]\n msg[\"childN\"] = [int(n) for n in root.child_N]\n\n ranked_children = root.rank_children()\n variations = {}\n for i in ranked_children[:15]:\n if root.child_N[i] == 0 or i not in root.children:\n break\n c = coords.to_gtp(coords.from_flat(i))\n child = root.children[i]\n nodes = child.most_visited_path_nodes()\n moves = [coords.to_gtp(coords.from_flat(m.fmove)) for m in nodes]\n variations[c] = {\n \"n\": int(root.child_N[i]),\n \"q\": float(root.child_Q[i]),\n \"moves\": [c] + moves,\n }\n\n if leaves:\n path = []\n leaf = leaves[0]\n while leaf != root:\n path.append(leaf.fmove)\n leaf = leaf.parent\n if path:\n path.reverse()\n variations[\"live\"] = {\n \"n\": int(root.child_N[path[0]]),\n \"q\": float(root.child_Q[path[0]]),\n \"moves\": [coords.to_gtp(coords.from_flat(m)) for m in path]\n }\n\n if variations:\n msg[\"variations\"] = variations\n\n dbg(\"mg-update:%s\" % json.dumps(msg, sort_keys=True))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main():\n mlperf_log.ROOT_DIR_GNMT = os.path.dirname(os.path.abspath(__file__))\n mlperf_log.LOGGER.propagate = False\n\n args = parse_args()\n device = utils.set_device(args.cuda, args.local_rank)\n distributed = utils.init_distributed(args.cuda)\n gnmt_print(key=mlperf_log.RUN_START, sync=True)\n args.rank = utils.get_rank()\n\n if not args.cudnn:\n torch.backends.cudnn.enabled = False\n\n # create directory for results\n save_path = os.path.join(args.results_dir, args.save)\n args.save_path = save_path\n os.makedirs(save_path, exist_ok=True)\n\n # setup logging\n log_filename = f'log_rank_{utils.get_rank()}.log'\n utils.setup_logging(os.path.join(save_path, log_filename))\n\n if args.env:\n utils.log_env_info()\n\n logging.info(f'Saving results to: {save_path}')\n logging.info(f'Run arguments: {args}')\n\n # automatically set train_iter_size based on train_global_batch_size,\n # world_size and per-worker train_batch_size\n if args.train_global_batch_size is not None:\n global_bs = args.train_global_batch_size\n bs = args.train_batch_size\n world_size = utils.get_world_size()\n assert global_bs % (bs * world_size) == 0\n args.train_iter_size = global_bs // (bs * world_size)\n logging.info(f'Global batch size was set in the config, '\n f'Setting train_iter_size to {args.train_iter_size}')\n\n worker_seeds, shuffling_seeds = utils.setup_seeds(args.seed, args.epochs,\n device)\n worker_seed = worker_seeds[args.rank]\n logging.info(f'Worker {args.rank} is using worker seed: {worker_seed}')\n torch.manual_seed(worker_seed)\n\n # build tokenizer\n pad_vocab = utils.pad_vocabulary(args.math)\n tokenizer = Tokenizer(os.path.join(args.dataset_dir, config.VOCAB_FNAME),\n pad_vocab)\n\n # build datasets\n gnmt_print(key=mlperf_log.PREPROC_TOKENIZE_TRAINING, sync=False)\n gnmt_print(key=mlperf_log.TRAIN_HP_MAX_SEQ_LEN,\n value=args.max_length_train, sync=False)\n\n train_data = LazyParallelDataset(\n src_fname=os.path.join(args.dataset_dir, config.SRC_TRAIN_FNAME),\n tgt_fname=os.path.join(args.dataset_dir, config.TGT_TRAIN_FNAME),\n tokenizer=tokenizer,\n min_len=args.min_length_train,\n max_len=args.max_length_train,\n sort=False,\n max_size=args.max_size)\n\n gnmt_print(key=mlperf_log.PREPROC_NUM_TRAIN_EXAMPLES,\n value=len(train_data), sync=False)\n\n val_data = ParallelDataset(\n src_fname=os.path.join(args.dataset_dir, config.SRC_VAL_FNAME),\n tgt_fname=os.path.join(args.dataset_dir, config.TGT_VAL_FNAME),\n tokenizer=tokenizer,\n min_len=args.min_length_val,\n max_len=args.max_length_val,\n sort=True)\n\n gnmt_print(key=mlperf_log.PREPROC_TOKENIZE_EVAL, sync=False)\n\n test_data = TextDataset(\n src_fname=os.path.join(args.dataset_dir, config.SRC_TEST_FNAME),\n tokenizer=tokenizer,\n min_len=args.min_length_test,\n max_len=args.max_length_test,\n sort=True)\n\n gnmt_print(key=mlperf_log.PREPROC_NUM_EVAL_EXAMPLES,\n value=len(test_data), sync=False)\n\n vocab_size = tokenizer.vocab_size\n gnmt_print(key=mlperf_log.PREPROC_VOCAB_SIZE,\n value=vocab_size, sync=False)\n\n # build GNMT model\n model_config = {'hidden_size': args.hidden_size,\n 'num_layers': args.num_layers,\n 'dropout': args.dropout, 'batch_first': False,\n 'share_embedding': args.share_embedding}\n model = GNMT(vocab_size=vocab_size, **model_config)\n logging.info(model)\n\n batch_first = model.batch_first\n\n # define loss function (criterion) and optimizer\n criterion = build_criterion(vocab_size, config.PAD, args.smoothing)\n\n opt_config = {'optimizer': args.optimizer, 'lr': args.lr}\n opt_config.update(literal_eval(args.optimizer_extra))\n logging.info(f'Training optimizer config: {opt_config}')\n\n scheduler_config = {'warmup_steps': args.warmup_steps,\n 'remain_steps': args.remain_steps,\n 'decay_interval': args.decay_interval,\n 'decay_steps': args.decay_steps,\n 'decay_factor': args.decay_factor}\n\n logging.info(f'Training LR schedule config: {scheduler_config}')\n\n num_parameters = sum([l.nelement() for l in model.parameters()])\n logging.info(f'Number of parameters: {num_parameters}')\n\n batching_opt = {'shard_size': args.shard_size,\n 'num_buckets': args.num_buckets}\n # get data loaders\n train_loader = train_data.get_loader(batch_size=args.train_batch_size,\n seeds=shuffling_seeds,\n batch_first=batch_first,\n shuffle=True,\n batching=args.batching,\n batching_opt=batching_opt,\n num_workers=args.train_loader_workers)\n\n gnmt_print(key=mlperf_log.INPUT_BATCH_SIZE,\n value=args.train_batch_size * utils.get_world_size(),\n sync=False)\n gnmt_print(key=mlperf_log.INPUT_SIZE,\n value=train_loader.sampler.num_samples, sync=False)\n\n val_loader = val_data.get_loader(batch_size=args.val_batch_size,\n batch_first=batch_first,\n shuffle=False,\n num_workers=args.val_loader_workers)\n\n test_loader = test_data.get_loader(batch_size=args.test_batch_size,\n batch_first=batch_first,\n shuffle=False,\n pad=True,\n num_workers=args.test_loader_workers)\n\n gnmt_print(key=mlperf_log.EVAL_SIZE,\n value=len(test_loader.dataset), sync=False)\n\n translator = Translator(model=model,\n tokenizer=tokenizer,\n loader=test_loader,\n beam_size=args.beam_size,\n max_seq_len=args.max_length_test,\n len_norm_factor=args.len_norm_factor,\n len_norm_const=args.len_norm_const,\n cov_penalty_factor=args.cov_penalty_factor,\n cuda=args.cuda,\n print_freq=args.print_freq,\n dataset_dir=args.dataset_dir,\n target_bleu=args.target_bleu,\n save_path=args.save_path)\n\n # create trainer\n total_train_iters = len(train_loader) // args.train_iter_size * args.epochs\n save_info = {'model_config': model_config, 'config': args, 'tokenizer':\n tokenizer.get_state()}\n trainer_options = dict(\n criterion=criterion,\n grad_clip=args.grad_clip,\n iter_size=args.train_iter_size,\n save_path=save_path,\n save_freq=args.save_freq,\n save_info=save_info,\n opt_config=opt_config,\n scheduler_config=scheduler_config,\n train_iterations=total_train_iters,\n batch_first=batch_first,\n keep_checkpoints=args.keep_checkpoints,\n math=args.math,\n print_freq=args.print_freq,\n cuda=args.cuda,\n distributed=distributed,\n intra_epoch_eval=args.intra_epoch_eval,\n translator=translator)\n\n trainer_options['model'] = model\n trainer = trainers.Seq2SeqTrainer(**trainer_options)\n\n # optionally resume from a checkpoint\n if args.resume:\n checkpoint_file = args.resume\n if os.path.isdir(checkpoint_file):\n checkpoint_file = os.path.join(\n checkpoint_file, 'model_best.pth')\n if os.path.isfile(checkpoint_file):\n trainer.load(checkpoint_file)\n else:\n logging.error(f'No checkpoint found at {args.resume}')\n\n # training loop\n best_loss = float('inf')\n break_training = False\n test_bleu = None\n gnmt_print(key=mlperf_log.TRAIN_LOOP, sync=True)\n for epoch in range(args.start_epoch, args.epochs):\n logging.info(f'Starting epoch {epoch}')\n gnmt_print(key=mlperf_log.TRAIN_EPOCH,\n value=epoch, sync=True)\n\n train_loader.sampler.set_epoch(epoch)\n\n trainer.epoch = epoch\n train_loss, train_perf = trainer.optimize(train_loader)\n\n # evaluate on validation set\n if args.eval:\n logging.info(f'Running validation on dev set')\n val_loss, val_perf = trainer.evaluate(val_loader)\n\n # remember best prec@1 and save checkpoint\n gnmt_print(key=mlperf_log.TRAIN_CHECKPOINT, sync=False)\n if args.rank == 0:\n is_best = val_loss < best_loss\n best_loss = min(val_loss, best_loss)\n trainer.save(save_all=args.save_all, is_best=is_best)\n\n if args.eval:\n gnmt_print(key=mlperf_log.EVAL_START, value=epoch, sync=True)\n test_bleu, break_training = translator.run(calc_bleu=True,\n epoch=epoch)\n gnmt_print(key=mlperf_log.EVAL_ACCURACY,\n value={\"epoch\": epoch, \"value\": round(test_bleu, 2)},\n sync=False)\n gnmt_print(key=mlperf_log.EVAL_TARGET,\n value=args.target_bleu, sync=False)\n gnmt_print(key=mlperf_log.EVAL_STOP, sync=True)\n\n acc_log = []\n acc_log += [f'Summary: Epoch: {epoch}']\n acc_log += [f'Training Loss: {train_loss:.4f}']\n if args.eval:\n acc_log += [f'Validation Loss: {val_loss:.4f}']\n acc_log += [f'Test BLEU: {test_bleu:.2f}']\n\n perf_log = []\n perf_log += [f'Performance: Epoch: {epoch}']\n perf_log += [f'Training: {train_perf:.0f} Tok/s']\n if args.eval:\n perf_log += [f'Validation: {val_perf:.0f} Tok/s']\n\n if args.rank == 0:\n logging.info('\\t'.join(acc_log))\n logging.info('\\t'.join(perf_log))\n\n logging.info(f'Finished epoch {epoch}')\n if break_training:\n break\n\n gnmt_print(key=mlperf_log.RUN_STOP,\n value={\"success\": bool(break_training)}, sync=True)\n gnmt_print(key=mlperf_log.RUN_FINAL, sync=False)", "response": "Launches data - parallel multi - gpu training."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_holdout_selfplay():\n holdout_dirs = (os.path.join(fsdb.holdout_dir(), d)\n for d in reversed(gfile.ListDirectory(fsdb.holdout_dir()))\n if gfile.IsDirectory(os.path.join(fsdb.holdout_dir(), d))\n for f in gfile.ListDirectory(os.path.join(fsdb.holdout_dir(), d)))\n\n # This is a roundabout way of computing how many hourly directories we need\n # to read in order to encompass 20,000 holdout games.\n holdout_dirs = set(itertools.islice(holdout_dirs), 20000)\n cmd = ['python3', 'validate.py'] + list(holdout_dirs) + [\n '--use_tpu',\n '--tpu_name={}'.format(TPU_NAME),\n '--flagfile=rl_loop/distributed_flags',\n '--expand_validation_dirs']\n mask_flags.run(cmd)", "response": "Validate on held - out selfplay data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_pro():\n cmd = ['python3', 'validate.py', FLAGS.pro_dataset,\n '--use_tpu',\n '--tpu_name={}'.format(TPU_NAME),\n '--work_dir={}'.format(fsdb.working_dir()),\n '--flagfile=rl_loop/distributed_flags',\n '--validate_name=pro']\n mask_flags.run(cmd)", "response": "Validate on professional data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets number of groups used by GroupNorm based on number of channels.", "response": "def get_group_gn(dim, dim_per_gp, num_groups):\n \"\"\"get number of groups used by GroupNorm, based on number of channels.\"\"\"\n assert dim_per_gp == -1 or num_groups == -1, \\\n \"GroupNorm: can only specify G or C/G.\"\n\n if dim_per_gp > 0:\n assert dim % dim_per_gp == 0, \\\n \"dim: {}, dim_per_gp: {}\".format(dim, dim_per_gp)\n group_gn = dim // dim_per_gp\n else:\n assert dim % num_groups == 0, \\\n \"dim: {}, num_groups: {}\".format(dim, num_groups)\n group_gn = num_groups\n\n return group_gn"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_fc(dim_in, hidden_dim, use_gn=False):\n '''\n Caffe2 implementation uses XavierFill, which in fact\n corresponds to kaiming_uniform_ in PyTorch\n '''\n if use_gn:\n fc = nn.Linear(dim_in, hidden_dim, bias=False)\n nn.init.kaiming_uniform_(fc.weight, a=1)\n return nn.Sequential(fc, group_norm(hidden_dim))\n fc = nn.Linear(dim_in, hidden_dim)\n nn.init.kaiming_uniform_(fc.weight, a=1)\n nn.init.constant_(fc.bias, 0)\n return fc", "response": "Caffe2 implementation uses XavierFill which in fact\n corresponds to kaiming_uniform_ in PyTorch\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a resized copy of this bounding box with the requested size.", "response": "def resize(self, size, *args, **kwargs):\n \"\"\"\n Returns a resized copy of this bounding box\n\n :param size: The requested size in pixels, as a 2-tuple:\n (width, height).\n \"\"\"\n\n ratios = tuple(float(s) / float(s_orig) for s, s_orig in zip(size, self.size))\n if ratios[0] == ratios[1]:\n ratio = ratios[0]\n scaled_box = self.bbox * ratio\n bbox = BoxList(scaled_box, size, mode=self.mode)\n # bbox._copy_extra_fields(self)\n for k, v in self.extra_fields.items():\n if not isinstance(v, torch.Tensor):\n v = v.resize(size, *args, **kwargs)\n bbox.add_field(k, v)\n return bbox\n\n ratio_width, ratio_height = ratios\n xmin, ymin, xmax, ymax = self._split_into_xyxy()\n scaled_xmin = xmin * ratio_width\n scaled_xmax = xmax * ratio_width\n scaled_ymin = ymin * ratio_height\n scaled_ymax = ymax * ratio_height\n scaled_box = torch.cat(\n (scaled_xmin, scaled_ymin, scaled_xmax, scaled_ymax), dim=-1\n )\n bbox = BoxList(scaled_box, size, mode=\"xyxy\")\n # bbox._copy_extra_fields(self)\n for k, v in self.extra_fields.items():\n if not isinstance(v, torch.Tensor):\n v = v.resize(size, *args, **kwargs)\n bbox.add_field(k, v)\n\n return bbox.convert(self.mode)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transpose(self, method):\n if method not in (FLIP_LEFT_RIGHT, FLIP_TOP_BOTTOM):\n raise NotImplementedError(\n \"Only FLIP_LEFT_RIGHT and FLIP_TOP_BOTTOM implemented\"\n )\n\n image_width, image_height = self.size\n xmin, ymin, xmax, ymax = self._split_into_xyxy()\n if method == FLIP_LEFT_RIGHT:\n TO_REMOVE = 1\n transposed_xmin = image_width - xmax - TO_REMOVE\n transposed_xmax = image_width - xmin - TO_REMOVE\n transposed_ymin = ymin\n transposed_ymax = ymax\n elif method == FLIP_TOP_BOTTOM:\n transposed_xmin = xmin\n transposed_xmax = xmax\n transposed_ymin = image_height - ymax\n transposed_ymax = image_height - ymin\n\n transposed_boxes = torch.cat(\n (transposed_xmin, transposed_ymin, transposed_xmax, transposed_ymax), dim=-1\n )\n bbox = BoxList(transposed_boxes, self.size, mode=\"xyxy\")\n # bbox._copy_extra_fields(self)\n for k, v in self.extra_fields.items():\n if not isinstance(v, torch.Tensor):\n v = v.transpose(method)\n bbox.add_field(k, v)\n return bbox.convert(self.mode)", "response": "Transposes the image into a new bounding box."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncrop a rectangular region from this bounding box.", "response": "def crop(self, box):\n \"\"\"\n Cropss a rectangular region from this bounding box. The box is a\n 4-tuple defining the left, upper, right, and lower pixel\n coordinate.\n \"\"\"\n xmin, ymin, xmax, ymax = self._split_into_xyxy()\n w, h = box[2] - box[0], box[3] - box[1]\n cropped_xmin = (xmin - box[0]).clamp(min=0, max=w)\n cropped_ymin = (ymin - box[1]).clamp(min=0, max=h)\n cropped_xmax = (xmax - box[0]).clamp(min=0, max=w)\n cropped_ymax = (ymax - box[1]).clamp(min=0, max=h)\n\n # TODO should I filter empty boxes here?\n if False:\n is_empty = (cropped_xmin == cropped_xmax) | (cropped_ymin == cropped_ymax)\n\n cropped_box = torch.cat(\n (cropped_xmin, cropped_ymin, cropped_xmax, cropped_ymax), dim=-1\n )\n bbox = BoxList(cropped_box, (w, h), mode=\"xyxy\")\n # bbox._copy_extra_fields(self)\n for k, v in self.extra_fields.items():\n if not isinstance(v, torch.Tensor):\n v = v.crop(box)\n bbox.add_field(k, v)\n return bbox.convert(self.mode)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef main():\n args = parse_args()\n utils.set_device(args.cuda, args.local_rank)\n utils.init_distributed(args.cuda)\n setup_logging()\n\n if args.env:\n utils.log_env_info()\n\n logging.info(f'Run arguments: {args}')\n\n if not args.cuda and torch.cuda.is_available():\n warnings.warn('cuda is available but not enabled')\n if not args.cudnn:\n torch.backends.cudnn.enabled = False\n\n # load checkpoint and deserialize to CPU (to save GPU memory)\n checkpoint = torch.load(args.model, map_location={'cuda:0': 'cpu'})\n\n # build GNMT model\n tokenizer = Tokenizer()\n tokenizer.set_state(checkpoint['tokenizer'])\n vocab_size = tokenizer.vocab_size\n model_config = checkpoint['model_config']\n model_config['batch_first'] = args.batch_first\n model = GNMT(vocab_size=vocab_size, **model_config)\n model.load_state_dict(checkpoint['state_dict'])\n\n for (math, batch_size, beam_size) in product(args.math, args.batch_size,\n args.beam_size):\n logging.info(f'math: {math}, batch size: {batch_size}, '\n f'beam size: {beam_size}')\n if math == 'fp32':\n dtype = torch.FloatTensor\n if math == 'fp16':\n dtype = torch.HalfTensor\n model.type(dtype)\n\n if args.cuda:\n model = model.cuda()\n model.eval()\n\n # construct the dataset\n test_data = TextDataset(src_fname=args.input,\n tokenizer=tokenizer,\n sort=args.sort)\n\n # build the data loader\n test_loader = test_data.get_loader(batch_size=batch_size,\n batch_first=args.batch_first,\n shuffle=False,\n pad=True,\n num_workers=0)\n\n # build the translator object\n translator = Translator(model=model,\n tokenizer=tokenizer,\n loader=test_loader,\n beam_size=beam_size,\n max_seq_len=args.max_seq_len,\n len_norm_factor=args.len_norm_factor,\n len_norm_const=args.len_norm_const,\n cov_penalty_factor=args.cov_penalty_factor,\n cuda=args.cuda,\n print_freq=args.print_freq,\n dataset_dir=args.dataset_dir)\n\n # execute the inference\n translator.run(calc_bleu=args.bleu, eval_path=args.output,\n reference_path=args.reference, summary=True)", "response": "This function is executed on a single GPU. It is executed on a single GPU and is executed on a single GPU."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngive a list of numeric sequences returns the corresponding strings", "response": "def convert_to_strings(self, sequences, sizes=None):\n \"\"\"Given a list of numeric sequences, returns the corresponding strings\"\"\"\n strings = []\n for x in xrange(len(sequences)):\n seq_len = sizes[x] if sizes is not None else len(sequences[x])\n string = self._convert_to_string(sequences[x], seq_len)\n strings.append(string)\n return strings"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprocessing a list of strings and returns a list of strings.", "response": "def process_strings(self, sequences, remove_repetitions=False):\n \"\"\"\n Given a list of strings, removes blanks and replace space character with space.\n Option to remove repetitions (e.g. 'abbca' -> 'abca').\n\n Arguments:\n sequences: list of 1-d array of integers\n remove_repetitions (boolean, optional): If true, repeating characters\n are removed. Defaults to False.\n \"\"\"\n processed_strings = []\n for sequence in sequences:\n string = self.process_string(remove_repetitions, sequence).strip()\n processed_strings.append(string)\n return processed_strings"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wer(self, s1, s2):\n\n # build mapping of words to integers\n b = set(s1.split() + s2.split())\n word2char = dict(zip(b, range(len(b))))\n\n # map the words to a char array (Levenshtein packages only accepts\n # strings)\n w1 = [chr(word2char[w]) for w in s1.split()]\n w2 = [chr(word2char[w]) for w in s2.split()]\n\n return Lev.distance(''.join(w1), ''.join(w2))", "response": "Computes the edit distance between the two provided sentences after tokenizing to words."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nforward computation of the class logits and box regression from the model.", "response": "def forward(self, x, boxes):\n \"\"\"\n Arguments:\n x (tuple[tensor, tensor]): x contains the class logits\n and the box_regression from the model.\n boxes (list[BoxList]): bounding boxes that are used as\n reference, one for ech image\n\n Returns:\n results (list[BoxList]): one BoxList for each image, containing\n the extra fields labels and scores\n \"\"\"\n class_logits, box_regression = x\n class_prob = F.softmax(class_logits, -1)\n\n # TODO think about a representation of batch of boxes\n image_shapes = [box.size for box in boxes]\n boxes_per_image = [len(box) for box in boxes]\n concat_boxes = torch.cat([a.bbox for a in boxes], dim=0)\n\n if self.cls_agnostic_bbox_reg:\n box_regression = box_regression[:, -4:]\n proposals = self.box_coder.decode(\n box_regression.view(sum(boxes_per_image), -1), concat_boxes\n )\n if self.cls_agnostic_bbox_reg:\n proposals = proposals.repeat(1, class_prob.shape[1])\n\n num_classes = class_prob.shape[1]\n\n proposals = proposals.split(boxes_per_image, dim=0)\n class_prob = class_prob.split(boxes_per_image, dim=0)\n\n results = []\n for prob, boxes_per_img, image_shape in zip(\n class_prob, proposals, image_shapes\n ):\n boxlist = self.prepare_boxlist(boxes_per_img, prob, image_shape)\n boxlist = boxlist.clip_to_image(remove_empty=False)\n boxlist = self.filter_results(boxlist, num_classes)\n results.append(boxlist)\n return results"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\npreparing a boxlist for use in the object class store.", "response": "def prepare_boxlist(self, boxes, scores, image_shape):\n \"\"\"\n Returns BoxList from `boxes` and adds probability scores information\n as an extra field\n `boxes` has shape (#detections, 4 * #classes), where each row represents\n a list of predicted bounding boxes for each of the object classes in the\n dataset (including the background class). The detections in each row\n originate from the same object proposal.\n `scores` has shape (#detection, #classes), where each row represents a list\n of object detection confidence scores for each of the object classes in the\n dataset (including the background class). `scores[i, j]`` corresponds to the\n box at `boxes[i, j * 4:(j + 1) * 4]`.\n \"\"\"\n boxes = boxes.reshape(-1, 4)\n scores = scores.reshape(-1)\n boxlist = BoxList(boxes, image_shape, mode=\"xyxy\")\n boxlist.add_field(\"scores\", scores)\n return boxlist"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter_results(self, boxlist, num_classes):\n # unwrap the boxlist to avoid additional overhead.\n # if we had multi-class NMS, we could perform this directly on the boxlist\n boxes = boxlist.bbox.reshape(-1, num_classes * 4)\n scores = boxlist.get_field(\"scores\").reshape(-1, num_classes)\n\n device = scores.device\n result = []\n # Apply threshold on detection probabilities and apply NMS\n # Skip j = 0, because it's the background class\n inds_all = scores > self.score_thresh\n for j in range(1, num_classes):\n inds = inds_all[:, j].nonzero().squeeze(1)\n scores_j = scores[inds, j]\n boxes_j = boxes[inds, j * 4 : (j + 1) * 4]\n boxlist_for_class = BoxList(boxes_j, boxlist.size, mode=\"xyxy\")\n boxlist_for_class.add_field(\"scores\", scores_j)\n boxlist_for_class = boxlist_nms(\n boxlist_for_class, self.nms\n )\n num_labels = len(boxlist_for_class)\n boxlist_for_class.add_field(\n \"labels\", torch.full((num_labels,), j, dtype=torch.int64, device=device)\n )\n result.append(boxlist_for_class)\n\n result = cat_boxlist(result)\n number_of_detections = len(result)\n\n # Limit to max_per_image detections **over all classes**\n if number_of_detections > self.detections_per_img > 0:\n cls_scores = result.get_field(\"scores\")\n image_thresh, _ = torch.kthvalue(\n cls_scores.cpu(), number_of_detections - self.detections_per_img + 1\n )\n keep = cls_scores >= image_thresh.item()\n keep = torch.nonzero(keep).squeeze(1)\n result = result[keep]\n return result", "response": "Filter detection results by thresholding on scores and applying non - maximum suppression."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _determine_chunk_to_make(write_dir):\n models = fsdb.get_models()\n # Last model is N. N+1 (should be) training. We should gather games for N+2.\n chunk_to_make = os.path.join(write_dir, str(\n models[-1][0] + 1) + '.tfrecord.zz')\n if not tf.gfile.Exists(chunk_to_make):\n # N+1 is missing. Write it out ASAP\n print(\"Making chunk ASAP:\", chunk_to_make)\n return chunk_to_make, True\n chunk_to_make = os.path.join(write_dir, str(\n models[-1][0] + 2) + '.tfrecord.zz')\n while tf.gfile.Exists(chunk_to_make):\n print(\"Chunk for next model ({}) already exists. Sleeping.\".format(\n chunk_to_make))\n time.sleep(5 * 60)\n models = fsdb.get_models()\n chunk_to_make = os.path.join(write_dir, str(\n models[-1][0] + 2) + '.tfrecord.zz')\n print(\"Making chunk:\", chunk_to_make)\n\n return chunk_to_make, False", "response": "Determine the chunk to make and return the full path of the chunk to make and a boolean indicating whether we should wait for a new model and False == write immediately."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fill_and_wait_models(bufsize=EXAMPLES_PER_GENERATION,\n write_dir=None,\n threads=8,\n model_window=100,\n skip_first_rsync=False):\n \"\"\" Fills a ringbuffer with positions from the most recent games, then\n continually rsync's and updates the buffer until a new model is promoted.\n Once it detects a new model, iit then dumps its contents for training to\n immediately begin on the next model.\n \"\"\"\n write_dir = write_dir or fsdb.golden_chunk_dir()\n buf = ExampleBuffer(bufsize)\n models = fsdb.get_models()[-model_window:]\n if not skip_first_rsync:\n with timer(\"Rsync\"):\n smart_rsync(models[-1][0] - 6)\n files = tqdm(map(files_for_model, models), total=len(models))\n buf.parallel_fill(list(itertools.chain(*files)), threads=threads)\n\n print(\"Filled buffer, watching for new games\")\n while fsdb.get_latest_model()[0] == models[-1][0]:\n with timer(\"Rsync\"):\n smart_rsync(models[-1][0] - 2)\n new_files = tqdm(map(files_for_model, models[-2:]), total=len(models))\n buf.update(list(itertools.chain(*new_files)))\n time.sleep(60)\n latest = fsdb.get_latest_model()\n\n print(\"New model!\", latest[1], \"!=\", models[-1][1])\n print(buf)\n buf.flush(os.path.join(write_dir, str(latest[0] + 1) + '.tfrecord.zz'))", "response": "Fills a ringbuffer with positions from the most recent games then continually rsyncs and updates the buffer until a new model is promoted."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmake a golden chunk for a given model.", "response": "def make_chunk_for(output_dir=LOCAL_DIR,\n local_dir=LOCAL_DIR,\n game_dir=None,\n model_num=1,\n positions=EXAMPLES_PER_GENERATION,\n threads=8,\n sampling_frac=0.02):\n \"\"\"\n Explicitly make a golden chunk for a given model `model_num`\n (not necessarily the most recent one).\n\n While we haven't yet got enough samples (EXAMPLES_PER_GENERATION)\n Add samples from the games of previous model.\n \"\"\"\n game_dir = game_dir or fsdb.selfplay_dir()\n ensure_dir_exists(output_dir)\n models = [model for model in fsdb.get_models() if model[0] < model_num]\n buf = ExampleBuffer(positions, sampling_frac=sampling_frac)\n files = []\n for _, model in sorted(models, reverse=True):\n local_model_dir = os.path.join(local_dir, model)\n if not tf.gfile.Exists(local_model_dir):\n print(\"Rsyncing\", model)\n _rsync_dir(os.path.join(game_dir, model), local_model_dir)\n files.extend(tf.gfile.Glob(os.path.join(local_model_dir, '*.zz')))\n print(\"{}: {} games\".format(model, len(files)))\n if len(files) * 200 * sampling_frac > positions:\n break\n\n print(\"Filling from {} files\".format(len(files)))\n\n buf.parallel_fill(files, threads=threads)\n print(buf)\n output = os.path.join(output_dir, str(model_num) + '.tfrecord.zz')\n print(\"Writing to\", output)\n buf.flush(output)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntakes a list of. tfrecord. zz game records and fills the examples list with the results.", "response": "def parallel_fill(self, games, threads=8):\n \"\"\" games is a list of .tfrecord.zz game records. \"\"\"\n games.sort(key=os.path.basename)\n # A couple extra in case parsing fails\n max_games = int(self.max_size / self.sampling_frac / 200) + 480\n if len(games) > max_games:\n games = games[-max_games:]\n\n with mp.Pool(threads) as pool:\n res = tqdm(pool.imap(self.func, games), total=len(games))\n self.examples.extend(itertools.chain.from_iterable(res))\n print(\"Got\", len(self.examples), \"examples\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the internal list of new games.", "response": "def update(self, new_games):\n \"\"\" new_games is a list of .tfrecord.zz new game records. \"\"\"\n new_games.sort(key=os.path.basename)\n first_new_game = None\n for idx, game in enumerate(new_games):\n timestamp = file_timestamp(game)\n if timestamp <= self.examples[-1][0]:\n continue\n elif first_new_game is None:\n first_new_game = idx\n num_new_games = len(new_games) - idx\n print(\"Found {}/{} new games\".format(\n num_new_games, len(new_games)))\n self.total_updates += num_new_games\n self.examples.extend(self.func(game))\n if first_new_game is None:\n print(\"No new games\", file_timestamp(\n new_games[-1]), self.examples[-1][0])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nforwards for single feature map.", "response": "def forward_for_single_feature_map(\n self, anchors, box_cls, box_regression):\n \"\"\"\n Arguments:\n anchors: list[BoxList]\n box_cls: tensor of size N, A * C, H, W\n box_regression: tensor of size N, A * 4, H, W\n \"\"\"\n device = box_cls.device\n N, _, H, W = box_cls.shape\n A = box_regression.size(1) // 4\n C = box_cls.size(1) // A\n\n # put in the same format as anchors\n box_cls = permute_and_flatten(box_cls, N, A, C, H, W)\n box_cls = box_cls.sigmoid()\n\n box_regression = permute_and_flatten(box_regression, N, A, 4, H, W)\n box_regression = box_regression.reshape(N, -1, 4)\n\n num_anchors = A * H * W\n\n candidate_inds = box_cls > self.pre_nms_thresh\n\n pre_nms_top_n = candidate_inds.view(N, -1).sum(1)\n pre_nms_top_n = pre_nms_top_n.clamp(max=self.pre_nms_top_n)\n\n results = []\n for per_box_cls, per_box_regression, per_pre_nms_top_n, \\\n per_candidate_inds, per_anchors in zip(\n box_cls,\n box_regression,\n pre_nms_top_n,\n candidate_inds,\n anchors):\n\n # Sort and select TopN\n # TODO most of this can be made out of the loop for\n # all images. \n # TODO:Yang: Not easy to do. Because the numbers of detections are\n # different in each image. Therefore, this part needs to be done\n # per image. \n per_box_cls = per_box_cls[per_candidate_inds]\n \n per_box_cls, top_k_indices = \\\n per_box_cls.topk(per_pre_nms_top_n, sorted=False)\n\n per_candidate_nonzeros = \\\n per_candidate_inds.nonzero()[top_k_indices, :]\n\n per_box_loc = per_candidate_nonzeros[:, 0]\n per_class = per_candidate_nonzeros[:, 1]\n per_class += 1\n\n detections = self.box_coder.decode(\n per_box_regression[per_box_loc, :].view(-1, 4),\n per_anchors.bbox[per_box_loc, :].view(-1, 4)\n )\n\n boxlist = BoxList(detections, per_anchors.size, mode=\"xyxy\")\n boxlist.add_field(\"labels\", per_class)\n boxlist.add_field(\"scores\", per_box_cls)\n boxlist = boxlist.clip_to_image(remove_empty=False)\n boxlist = remove_small_boxes(boxlist, self.min_size)\n results.append(boxlist)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_tf_example(features, pi, value):\n return tf.train.Example(features=tf.train.Features(feature={\n 'x': tf.train.Feature(\n bytes_list=tf.train.BytesList(\n value=[features.tostring()])),\n 'pi': tf.train.Feature(\n bytes_list=tf.train.BytesList(\n value=[pi.tostring()])),\n 'outcome': tf.train.Feature(\n float_list=tf.train.FloatList(\n value=[value]))}))", "response": "Returns a tf. train. Example object for the given feature set and outcome."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting the examples to a file.", "response": "def write_tf_examples(filename, tf_examples, serialize=True):\n \"\"\"\n Args:\n filename: Where to write tf.records\n tf_examples: An iterable of tf.Example\n serialize: whether to serialize the examples.\n \"\"\"\n with tf.python_io.TFRecordWriter(\n filename, options=TF_RECORD_CONFIG) as writer:\n for ex in tf_examples:\n if serialize:\n writer.write(ex.SerializeToString())\n else:\n writer.write(ex)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef batch_parse_tf_example(batch_size, example_batch):\n features = {\n 'x': tf.FixedLenFeature([], tf.string),\n 'pi': tf.FixedLenFeature([], tf.string),\n 'outcome': tf.FixedLenFeature([], tf.float32),\n }\n parsed = tf.parse_example(example_batch, features)\n x = tf.decode_raw(parsed['x'], tf.uint8)\n x = tf.cast(x, tf.float32)\n x = tf.reshape(x, [batch_size, go.N, go.N,\n features_lib.NEW_FEATURES_PLANES])\n pi = tf.decode_raw(parsed['pi'], tf.float32)\n pi = tf.reshape(pi, [batch_size, go.N * go.N + 1])\n outcome = parsed['outcome']\n outcome.set_shape([batch_size])\n return x, {'pi_tensor': pi, 'value_tensor': outcome}", "response": "Parses a batch of tf. Example into a tuple of feature tensors pi and outcome tensors."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef read_tf_records(batch_size, tf_records, num_repeats=1,\n shuffle_records=True, shuffle_examples=True,\n shuffle_buffer_size=None, interleave=True,\n filter_amount=1.0):\n \"\"\"\n Args:\n batch_size: batch size to return\n tf_records: a list of tf_record filenames\n num_repeats: how many times the data should be read (default: One)\n shuffle_records: whether to shuffle the order of files read\n shuffle_examples: whether to shuffle the tf.Examples\n shuffle_buffer_size: how big of a buffer to fill before shuffling.\n interleave: iwhether to interleave examples from multiple tf_records\n filter_amount: what fraction of records to keep\n Returns:\n a tf dataset of batched tensors\n \"\"\"\n if shuffle_examples and not shuffle_buffer_size:\n raise ValueError(\"Must set shuffle buffer size if shuffling examples\")\n\n tf_records = list(tf_records)\n if shuffle_records:\n random.shuffle(tf_records)\n record_list = tf.data.Dataset.from_tensor_slices(tf_records)\n\n # compression_type here must agree with write_tf_examples\n map_func = functools.partial(\n tf.data.TFRecordDataset,\n buffer_size=8 * 1024 * 1024,\n compression_type='ZLIB')\n\n if interleave:\n # cycle_length = how many tfrecord files are read in parallel\n # The idea is to shuffle both the order of the files being read,\n # and the examples being read from the files.\n dataset = record_list.apply(tf.contrib.data.parallel_interleave(\n map_func, cycle_length=64, sloppy=True))\n else:\n dataset = record_list.flat_map(map_func)\n\n if filter_amount < 1.0:\n dataset = dataset.filter(\n lambda _: tf.random_uniform([]) < filter_amount)\n\n dataset = dataset.repeat(num_repeats)\n if shuffle_examples:\n dataset = dataset.shuffle(buffer_size=shuffle_buffer_size)\n\n dataset = dataset.batch(batch_size)\n return dataset", "response": "Reads a set of TFRecords into a single tree structure."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_input_tensors(batch_size, tf_records, num_repeats=1,\n shuffle_records=True, shuffle_examples=True,\n shuffle_buffer_size=None,\n filter_amount=0.05, random_rotation=True):\n \"\"\"Read tf.Records and prepare them for ingestion by dual_net.\n\n See `read_tf_records` for parameter documentation.\n\n Returns a dict of tensors (see return value of batch_parse_tf_example)\n \"\"\"\n print(\"Reading tf_records from {} inputs\".format(len(tf_records)))\n dataset = read_tf_records(\n batch_size,\n tf_records,\n num_repeats=num_repeats,\n shuffle_records=shuffle_records,\n shuffle_examples=shuffle_examples,\n shuffle_buffer_size=shuffle_buffer_size,\n filter_amount=filter_amount,\n interleave=True)\n dataset = dataset.filter(lambda t: tf.equal(tf.shape(t)[0], batch_size))\n dataset = dataset.map(\n functools.partial(batch_parse_tf_example, batch_size))\n if random_rotation:\n dataset = dataset.map(_random_rotation_pyfunc)\n\n return dataset.make_one_shot_iterator().get_next()", "response": "Read tf. Records and prepare them for ingestion by dual_net."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_dataset_from_selfplay(data_extracts):\n tf_examples = (make_tf_example(features_lib.extract_features(pos), pi, result)\n for pos, pi, result in data_extracts)\n return tf_examples", "response": "Returns an iterable of tf. Examples."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nforward computation of the next level of the CNN.", "response": "def forward(self, features, proposals, targets=None):\n \"\"\"\n Arguments:\n features (list[Tensor]): feature-maps from possibly several levels\n proposals (list[BoxList]): proposal boxes\n targets (list[BoxList], optional): the ground-truth targets.\n\n Returns:\n x (Tensor): the result of the feature extractor\n proposals (list[BoxList]): during training, the subsampled proposals\n are returned. During testing, the predicted boxlists are returned\n losses (dict[Tensor]): During training, returns the losses for the\n head. During testing, returns an empty dict.\n \"\"\"\n\n if self.training:\n # Faster R-CNN subsamples during training the proposals with a fixed\n # positive / negative ratio\n with torch.no_grad():\n proposals = self.loss_evaluator.subsample(proposals, targets)\n\n # extract features that will be fed to the final classifier. The\n # feature_extractor generally corresponds to the pooler + heads\n x = self.feature_extractor(features, proposals)\n # final classifier that converts the features into predictions\n class_logits, box_regression = self.predictor(x)\n\n if not self.training:\n result = self.post_processor((class_logits, box_regression), proposals)\n return x, result, {}\n\n loss_classifier, loss_box_reg = self.loss_evaluator(\n [class_logits], [box_regression]\n )\n return (\n x,\n proposals,\n dict(loss_classifier=loss_classifier, loss_box_reg=loss_box_reg),\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a boolean representing whether a model should stop at a given threshold.", "response": "def past_stop_threshold(stop_threshold, eval_metric):\n \"\"\"Return a boolean representing whether a model should be stopped.\n\n Args:\n stop_threshold: float, the threshold above which a model should stop\n training.\n eval_metric: float, the current value of the relevant metric to check.\n\n Returns:\n True if training should stop, False otherwise.\n\n Raises:\n ValueError: if either stop_threshold or eval_metric is not a number\n \"\"\"\n if stop_threshold is None:\n return False\n\n if not isinstance(stop_threshold, numbers.Number):\n raise ValueError(\"Threshold for checking stop conditions must be a number.\")\n if not isinstance(eval_metric, numbers.Number):\n raise ValueError(\"Eval metric being checked against stop conditions \"\n \"must be a number.\")\n\n if eval_metric >= stop_threshold:\n tf.logging.info(\n \"Stop threshold of {} was passed with metric value {}.\".format(\n stop_threshold, eval_metric))\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nforwards computation of the target and images for the next cluster.", "response": "def forward(self, images, features, targets=None):\n \"\"\"\n Arguments:\n images (ImageList): images for which we want to compute the predictions\n features (list[Tensor]): features computed from the images that are\n used for computing the predictions. Each tensor in the list\n correspond to different feature levels\n targets (list[BoxList): ground-truth boxes present in the image (optional)\n\n Returns:\n boxes (list[BoxList]): the predicted boxes from the RPN, one BoxList per\n image.\n losses (dict[Tensor]): the losses for the model during training. During\n testing, it is an empty dict.\n \"\"\"\n objectness, rpn_box_regression = self.head(features)\n anchors = self.anchor_generator(images, features)\n\n if self.training:\n return self._forward_train(anchors, objectness, rpn_box_regression, targets)\n else:\n return self._forward_test(anchors, objectness, rpn_box_regression)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning all_gather on arbitrary picklable data", "response": "def all_gather(data):\n \"\"\"\n Run all_gather on arbitrary picklable data (not necessarily tensors)\n Args:\n data: any picklable object\n Returns:\n list[data]: list of data gathered from each rank\n \"\"\"\n world_size = get_world_size()\n if world_size == 1:\n return [data]\n\n # serialized to a Tensor\n buffer = pickle.dumps(data)\n storage = torch.ByteStorage.from_buffer(buffer)\n tensor = torch.ByteTensor(storage).to(\"cuda\")\n\n # obtain Tensor size of each rank\n local_size = torch.IntTensor([tensor.numel()]).to(\"cuda\")\n size_list = [torch.IntTensor([0]).to(\"cuda\") for _ in range(world_size)]\n dist.all_gather(size_list, local_size)\n size_list = [int(size.item()) for size in size_list]\n max_size = max(size_list)\n\n # receiving Tensor from all ranks\n # we pad the tensor because torch all_gather does not support\n # gathering tensors of different shapes\n tensor_list = []\n for _ in size_list:\n tensor_list.append(torch.ByteTensor(size=(max_size,)).to(\"cuda\"))\n if local_size != max_size:\n padding = torch.ByteTensor(size=(max_size - local_size,)).to(\"cuda\")\n tensor = torch.cat((tensor, padding), dim=0)\n dist.all_gather(tensor_list, tensor)\n\n data_list = []\n for size, tensor in zip(size_list, tensor_list):\n buffer = tensor.cpu().numpy().tobytes()[:size]\n data_list.append(pickle.loads(buffer))\n\n return data_list"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreduce the values in the dictionary to the same size as the input_dict.", "response": "def reduce_dict(input_dict, average=True):\n \"\"\"\n Args:\n input_dict (dict): all the values will be reduced\n average (bool): whether to do average or sum\n Reduce the values in the dictionary from all processes so that process with rank\n 0 has the averaged results. Returns a dict with the same fields as\n input_dict, after reduction.\n \"\"\"\n world_size = get_world_size()\n if world_size < 2:\n return input_dict\n with torch.no_grad():\n names = []\n values = []\n # sort the keys so that they are consistent across processes\n for k in sorted(input_dict.keys()):\n names.append(k)\n values.append(input_dict[k])\n values = torch.stack(values, dim=0)\n dist.reduce(values, dst=0)\n if dist.get_rank() == 0 and average:\n # only main process gets accumulated, so only divide by\n # world_size in this case\n values /= world_size\n reduced_dict = {k: v for k, v in zip(names, values)}\n return reduced_dict"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef greedy_search(self, batch_size, initial_input, initial_context=None):\n max_seq_len = self.max_seq_len\n\n translation = torch.zeros(batch_size, max_seq_len, dtype=torch.int64)\n lengths = torch.ones(batch_size, dtype=torch.int64)\n active = torch.arange(0, batch_size, dtype=torch.int64)\n base_mask = torch.arange(0, batch_size, dtype=torch.int64)\n\n if self.cuda:\n translation = translation.cuda()\n lengths = lengths.cuda()\n active = active.cuda()\n base_mask = base_mask.cuda()\n\n translation[:, 0] = BOS\n words, context = initial_input, initial_context\n\n if self.batch_first:\n word_view = (-1, 1)\n ctx_batch_dim = 0\n else:\n word_view = (1, -1)\n ctx_batch_dim = 1\n\n counter = 0\n for idx in range(1, max_seq_len):\n if not len(active):\n break\n counter += 1\n\n words = words.view(word_view)\n output = self.model.generate(words, context, 1)\n words, logprobs, attn, context = output\n words = words.view(-1)\n\n translation[active, idx] = words\n lengths[active] += 1\n\n terminating = (words == EOS)\n\n if terminating.any():\n not_terminating = ~terminating\n\n mask = base_mask[:len(active)]\n mask = mask.masked_select(not_terminating)\n active = active.masked_select(not_terminating)\n\n words = words[mask]\n context[0] = context[0].index_select(ctx_batch_dim, mask)\n context[1] = context[1].index_select(0, mask)\n context[2] = context[2].index_select(1, mask)\n\n return translation, lengths, counter", "response": "Greedy search for the target tokens in the BOS language."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nbeam search decoder. :param batch_size: decoder batch size :param initial_input: initial input, usually tensor of BOS tokens :param initial_context: initial context, usually [encoder_context, src_seq_lengths, None] returns: (translation, lengths, counter) translation: (batch_size, max_seq_len) - indices of target tokens lengths: (batch_size) - lengths of generated translations counter: number of iterations of the decoding loop", "response": "def beam_search(self, batch_size, initial_input, initial_context=None):\n \"\"\"\n Beam search decoder.\n\n :param batch_size: decoder batch size\n :param initial_input: initial input, usually tensor of BOS tokens\n :param initial_context: initial context, usually [encoder_context,\n src_seq_lengths, None]\n\n returns: (translation, lengths, counter)\n translation: (batch_size, max_seq_len) - indices of target tokens\n lengths: (batch_size) - lengths of generated translations\n counter: number of iterations of the decoding loop\n \"\"\"\n beam_size = self.beam_size\n norm_const = self.len_norm_const\n norm_factor = self.len_norm_factor\n max_seq_len = self.max_seq_len\n cov_penalty_factor = self.cov_penalty_factor\n\n translation = torch.zeros(batch_size * beam_size, max_seq_len,\n dtype=torch.int64)\n lengths = torch.ones(batch_size * beam_size, dtype=torch.int64)\n scores = torch.zeros(batch_size * beam_size, dtype=torch.float32)\n\n active = torch.arange(0, batch_size * beam_size, dtype=torch.int64)\n base_mask = torch.arange(0, batch_size * beam_size, dtype=torch.int64)\n global_offset = torch.arange(0, batch_size * beam_size, beam_size,\n dtype=torch.int64)\n\n eos_beam_fill = torch.tensor([0] + (beam_size - 1) * [float('-inf')])\n\n if self.cuda:\n translation = translation.cuda()\n lengths = lengths.cuda()\n active = active.cuda()\n base_mask = base_mask.cuda()\n scores = scores.cuda()\n global_offset = global_offset.cuda()\n eos_beam_fill = eos_beam_fill.cuda()\n\n translation[:, 0] = BOS\n\n words, context = initial_input, initial_context\n\n if self.batch_first:\n word_view = (-1, 1)\n ctx_batch_dim = 0\n attn_query_dim = 1\n else:\n word_view = (1, -1)\n ctx_batch_dim = 1\n attn_query_dim = 0\n\n # replicate context\n if self.batch_first:\n # context[0] (encoder state): (batch, seq, feature)\n _, seq, feature = context[0].shape\n context[0].unsqueeze_(1)\n context[0] = context[0].expand(-1, beam_size, -1, -1)\n context[0] = context[0].contiguous().view(batch_size * beam_size,\n seq, feature)\n # context[0]: (batch * beam, seq, feature)\n else:\n # context[0] (encoder state): (seq, batch, feature)\n seq, _, feature = context[0].shape\n context[0].unsqueeze_(2)\n context[0] = context[0].expand(-1, -1, beam_size, -1)\n context[0] = context[0].contiguous().view(seq, batch_size *\n beam_size, feature)\n # context[0]: (seq, batch * beam, feature)\n\n # context[1] (encoder seq length): (batch)\n context[1].unsqueeze_(1)\n context[1] = context[1].expand(-1, beam_size)\n context[1] = context[1].contiguous().view(batch_size * beam_size)\n # context[1]: (batch * beam)\n\n accu_attn_scores = torch.zeros(batch_size * beam_size, seq)\n if self.cuda:\n accu_attn_scores = accu_attn_scores.cuda()\n\n counter = 0\n for idx in range(1, self.max_seq_len):\n if not len(active):\n break\n counter += 1\n\n eos_mask = (words == EOS)\n eos_mask = eos_mask.view(-1, beam_size)\n\n terminating, _ = eos_mask.min(dim=1)\n\n lengths[active[~eos_mask.view(-1)]] += 1\n\n output = self.model.generate(words, context, beam_size)\n words, logprobs, attn, context = output\n\n attn = attn.float().squeeze(attn_query_dim)\n attn = attn.masked_fill(eos_mask.view(-1).unsqueeze(1), 0)\n accu_attn_scores[active] += attn\n\n # words: (batch, beam, k)\n words = words.view(-1, beam_size, beam_size)\n words = words.masked_fill(eos_mask.unsqueeze(2), EOS)\n\n # logprobs: (batch, beam, k)\n logprobs = logprobs.float().view(-1, beam_size, beam_size)\n\n if eos_mask.any():\n logprobs[eos_mask] = eos_beam_fill\n\n active_scores = scores[active].view(-1, beam_size)\n # new_scores: (batch, beam, k)\n new_scores = active_scores.unsqueeze(2) + logprobs\n\n if idx == 1:\n new_scores[:, 1:, :].fill_(float('-inf'))\n\n new_scores = new_scores.view(-1, beam_size * beam_size)\n # index: (batch, beam)\n _, index = new_scores.topk(beam_size, dim=1)\n source_beam = index / beam_size\n\n new_scores = new_scores.view(-1, beam_size * beam_size)\n best_scores = torch.gather(new_scores, 1, index)\n scores[active] = best_scores.view(-1)\n\n words = words.view(-1, beam_size * beam_size)\n words = torch.gather(words, 1, index)\n\n # words: (1, batch * beam)\n words = words.view(word_view)\n\n offset = global_offset[:source_beam.shape[0]]\n source_beam += offset.unsqueeze(1)\n\n translation[active, :] = translation[active[source_beam.view(-1)], :]\n translation[active, idx] = words.view(-1)\n\n lengths[active] = lengths[active[source_beam.view(-1)]]\n\n context[2] = context[2].index_select(1, source_beam.view(-1))\n\n if terminating.any():\n not_terminating = ~terminating\n not_terminating = not_terminating.unsqueeze(1)\n not_terminating = not_terminating.expand(-1, beam_size).contiguous()\n\n normalization_mask = active.view(-1, beam_size)[terminating]\n\n # length normalization\n norm = lengths[normalization_mask].float()\n norm = (norm_const + norm) / (norm_const + 1.0)\n norm = norm ** norm_factor\n\n scores[normalization_mask] /= norm\n\n # coverage penalty\n penalty = accu_attn_scores[normalization_mask]\n penalty = penalty.clamp(0, 1)\n penalty = penalty.log()\n penalty[penalty == float('-inf')] = 0\n penalty = penalty.sum(dim=-1)\n\n scores[normalization_mask] += cov_penalty_factor * penalty\n\n mask = base_mask[:len(active)]\n mask = mask.masked_select(not_terminating.view(-1))\n\n words = words.index_select(ctx_batch_dim, mask)\n context[0] = context[0].index_select(ctx_batch_dim, mask)\n context[1] = context[1].index_select(0, mask)\n context[2] = context[2].index_select(1, mask)\n\n active = active.masked_select(not_terminating.view(-1))\n\n scores = scores.view(batch_size, beam_size)\n _, idx = scores.max(dim=1)\n\n translation = translation[idx + global_offset, :]\n lengths = lengths[idx + global_offset]\n\n return translation, lengths, counter"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef evaluate(dataset, predictions, output_folder, **kwargs):\n args = dict(\n dataset=dataset, predictions=predictions, output_folder=output_folder, **kwargs\n )\n if isinstance(dataset, datasets.COCODataset):\n return coco_evaluation(**args)\n elif isinstance(dataset, datasets.PascalVOCDataset):\n return voc_evaluation(**args)\n else:\n dataset_name = dataset.__class__.__name__\n raise NotImplementedError(\"Unsupported dataset type {}.\".format(dataset_name))", "response": "evaluate dataset using different methods based on dataset type."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates that examples are well formed.", "response": "def validate_examples(example_file):\n \"\"\"Validate that examples are well formed.\n\n Pi should sum to 1.0\n value should be {-1,1}\n\n Usage:\n validate_examples(\"../data/300.tfrecord.zz\")\n \"\"\"\n\n def test_example(raw):\n example = tf.train.Example()\n example.ParseFromString(raw)\n\n pi = np.frombuffer(example.features.feature['pi'].bytes_list.value[0], np.float32)\n value = example.features.feature['outcome'].float_list.value[0]\n assert abs(pi.sum() - 1) < 1e-4, pi.sum()\n assert value in (-1, 1), value\n\n opts = tf.python_io.TFRecordOptions(tf.python_io.TFRecordCompressionType.ZLIB)\n for record in tqdm(tf.python_io.tf_record_iterator(example_file, opts)):\n test_example(record)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_train_hooks(name_list, **kwargs):\n\n if not name_list:\n return []\n\n train_hooks = []\n for name in name_list:\n hook_name = HOOKS.get(name.strip().lower())\n if hook_name is None:\n raise ValueError('Unrecognized training hook requested: {}'.format(name))\n else:\n train_hooks.append(hook_name(**kwargs))\n\n return train_hooks", "response": "Factory for getting a list of TensorFlow hooks for training by name."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_logging_tensor_hook(every_n_iter=100, tensors_to_log=None, **kwargs): # pylint: disable=unused-argument\n if tensors_to_log is None:\n tensors_to_log = _TENSORS_TO_LOG\n\n return tf.train.LoggingTensorHook(\n tensors=tensors_to_log,\n every_n_iter=every_n_iter)", "response": "Function to get a LoggingTensorHook object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_examples_per_second_hook(every_n_steps=100,\n batch_size=128,\n warm_steps=5,\n **kwargs): # pylint: disable=unused-argument\n \"\"\"Function to get ExamplesPerSecondHook.\n\n Args:\n every_n_steps: `int`, print current and average examples per second every\n N steps.\n batch_size: `int`, total batch size used to calculate examples/second from\n global time.\n warm_steps: skip this number of steps before logging and running average.\n **kwargs: a dictionary of arguments to ExamplesPerSecondHook.\n\n Returns:\n Returns a ProfilerHook that writes out timelines that can be loaded into\n profiling tools like chrome://tracing.\n \"\"\"\n return hooks.ExamplesPerSecondHook(every_n_steps=every_n_steps,\n batch_size=batch_size,\n warm_steps=warm_steps)", "response": "Function to get ExamplesPerSecondHook."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_logging_metric_hook(benchmark_log_dir=None,\n tensors_to_log=None,\n every_n_secs=600,\n **kwargs): # pylint: disable=unused-argument\n \"\"\"Function to get LoggingMetricHook.\n\n Args:\n benchmark_log_dir: `string`, directory path to save the metric log.\n tensors_to_log: List of tensor names or dictionary mapping labels to tensor\n names. If not set, log _TENSORS_TO_LOG by default.\n every_n_secs: `int`, the frequency for logging the metric. Default to every\n 10 mins.\n\n Returns:\n Returns a ProfilerHook that writes out timelines that can be loaded into\n profiling tools like chrome://tracing.\n \"\"\"\n if benchmark_log_dir is None:\n raise ValueError(\"metric_log_dir should be provided to use metric logger\")\n if tensors_to_log is None:\n tensors_to_log = _TENSORS_TO_LOG\n return metric_hook.LoggingMetricHook(\n tensors=tensors_to_log,\n log_dir=benchmark_log_dir,\n every_n_secs=every_n_secs)", "response": "Function to get LoggingMetricHook."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a PASCAL VOC xml file into a list of dictionaries.", "response": "def parse_rec(filename):\n \"\"\" Parse a PASCAL VOC xml file \"\"\"\n tree = ET.parse(filename)\n objects = []\n for obj in tree.findall('object'):\n obj_struct = {}\n obj_struct['name'] = obj.find('name').text\n obj_struct['pose'] = obj.find('pose').text\n obj_struct['truncated'] = int(obj.find('truncated').text)\n obj_struct['difficult'] = int(obj.find('difficult').text)\n bbox = obj.find('bndbox')\n obj_struct['bbox'] = [int(bbox.find('xmin').text) - 1,\n int(bbox.find('ymin').text) - 1,\n int(bbox.find('xmax').text) - 1,\n int(bbox.find('ymax').text) - 1]\n objects.append(obj_struct)\n\n return objects"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef voc_eval(detpath,\n annopath,\n imagesetfile,\n classname,\n cachedir,\n ovthresh=0.5,\n use_07_metric=True):\n \"\"\"rec, prec, ap = voc_eval(detpath,\n annopath,\n imagesetfile,\n classname,\n [ovthresh],\n [use_07_metric])\nTop level function that does the PASCAL VOC evaluation.\ndetpath: Path to detections\n detpath.format(classname) should produce the detection results file.\nannopath: Path to annotations\n annopath.format(imagename) should be the xml annotations file.\nimagesetfile: Text file containing the list of images, one image per line.\nclassname: Category name (duh)\ncachedir: Directory for caching the annotations\n[ovthresh]: Overlap threshold (default = 0.5)\n[use_07_metric]: Whether to use VOC07's 11 point AP computation\n (default True)\n\"\"\"\n# assumes detections are in detpath.format(classname)\n# assumes annotations are in annopath.format(imagename)\n# assumes imagesetfile is a text file with each line an image name\n# cachedir caches the annotations in a pickle file\n# first load gt\n if not os.path.isdir(cachedir):\n os.mkdir(cachedir)\n cachefile = os.path.join(cachedir, 'annots.pkl')\n # read list of images\n with open(imagesetfile, 'r') as f:\n lines = f.readlines()\n imagenames = [x.strip() for x in lines]\n if not os.path.isfile(cachefile):\n # load annots\n recs = {}\n for i, imagename in enumerate(imagenames):\n recs[imagename] = parse_rec(annopath % (imagename))\n if i % 100 == 0:\n print('Reading annotation for {:d}/{:d}'.format(\n i + 1, len(imagenames)))\n # save\n print('Saving cached annotations to {:s}'.format(cachefile))\n with open(cachefile, 'wb') as f:\n pickle.dump(recs, f)\n else:\n # load\n with open(cachefile, 'rb') as f:\n recs = pickle.load(f)\n\n # extract gt objects for this class\n class_recs = {}\n npos = 0\n for imagename in imagenames:\n R = [obj for obj in recs[imagename] if obj['name'] == classname]\n bbox = np.array([x['bbox'] for x in R])\n difficult = np.array([x['difficult'] for x in R]).astype(np.bool)\n det = [False] * len(R)\n npos = npos + sum(~difficult)\n class_recs[imagename] = {'bbox': bbox,\n 'difficult': difficult,\n 'det': det}\n\n # read dets\n detfile = detpath.format(classname)\n with open(detfile, 'r') as f:\n lines = f.readlines()\n if any(lines) == 1:\n\n splitlines = [x.strip().split(' ') for x in lines]\n image_ids = [x[0] for x in splitlines]\n confidence = np.array([float(x[1]) for x in splitlines])\n BB = np.array([[float(z) for z in x[2:]] for x in splitlines])\n\n # sort by confidence\n sorted_ind = np.argsort(-confidence)\n sorted_scores = np.sort(-confidence)\n BB = BB[sorted_ind, :]\n image_ids = [image_ids[x] for x in sorted_ind]\n\n # go down dets and mark TPs and FPs\n nd = len(image_ids)\n tp = np.zeros(nd)\n fp = np.zeros(nd)\n for d in range(nd):\n R = class_recs[image_ids[d]]\n bb = BB[d, :].astype(float)\n ovmax = -np.inf\n BBGT = R['bbox'].astype(float)\n if BBGT.size > 0:\n # compute overlaps\n # intersection\n ixmin = np.maximum(BBGT[:, 0], bb[0])\n iymin = np.maximum(BBGT[:, 1], bb[1])\n ixmax = np.minimum(BBGT[:, 2], bb[2])\n iymax = np.minimum(BBGT[:, 3], bb[3])\n iw = np.maximum(ixmax - ixmin, 0.)\n ih = np.maximum(iymax - iymin, 0.)\n inters = iw * ih\n uni = ((bb[2] - bb[0]) * (bb[3] - bb[1]) +\n (BBGT[:, 2] - BBGT[:, 0]) *\n (BBGT[:, 3] - BBGT[:, 1]) - inters)\n overlaps = inters / uni\n ovmax = np.max(overlaps)\n jmax = np.argmax(overlaps)\n\n if ovmax > ovthresh:\n if not R['difficult'][jmax]:\n if not R['det'][jmax]:\n tp[d] = 1.\n R['det'][jmax] = 1\n else:\n fp[d] = 1.\n else:\n fp[d] = 1.\n\n # compute precision recall\n fp = np.cumsum(fp)\n tp = np.cumsum(tp)\n rec = tp / float(npos)\n # avoid divide by zero in case the first detection matches a difficult\n # ground truth\n prec = tp / np.maximum(tp + fp, np.finfo(np.float64).eps)\n ap = voc_ap(rec, prec, use_07_metric)\n else:\n rec = -1.\n prec = -1.\n ap = -1.\n\n return rec, prec, ap", "response": "This function evaluates the PASCAL VOC and returns the result of the evaluation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef build_collate_fn(batch_first=False, parallel=True, sort=False):\n def collate_seq(seq):\n \"\"\"\n Builds batches for training or inference.\n Batches are returned as pytorch tensors, with padding.\n\n :param seq: list of sequences\n \"\"\"\n lengths = [len(s) for s in seq]\n batch_length = max(lengths)\n\n shape = (batch_length, len(seq))\n seq_tensor = torch.full(shape, config.PAD, dtype=torch.int64)\n\n for i, s in enumerate(seq):\n end_seq = lengths[i]\n seq_tensor[:end_seq, i].copy_(s[:end_seq])\n\n if batch_first:\n seq_tensor = seq_tensor.t()\n\n return (seq_tensor, lengths)\n\n def parallel_collate(seqs):\n \"\"\"\n Builds batches from parallel dataset (src, tgt), optionally sorts batch\n by src sequence length.\n\n :param seqs: tuple of (src, tgt) sequences\n \"\"\"\n src_seqs, tgt_seqs = zip(*seqs)\n if sort:\n indices, src_seqs = zip(*sorted(enumerate(src_seqs),\n key=lambda item: len(item[1]),\n reverse=True))\n tgt_seqs = [tgt_seqs[idx] for idx in indices]\n\n return tuple([collate_seq(s) for s in [src_seqs, tgt_seqs]])\n\n def single_collate(src_seqs):\n \"\"\"\n Builds batches from text dataset, optionally sorts batch by src\n sequence length.\n\n :param src_seqs: source sequences\n \"\"\"\n if sort:\n indices, src_seqs = zip(*sorted(enumerate(src_seqs),\n key=lambda item: len(item[1]),\n reverse=True))\n else:\n indices = range(len(src_seqs))\n\n return collate_seq(src_seqs), tuple(indices)\n\n if parallel:\n return parallel_collate\n else:\n return single_collate", "response": "Build a collate_fn function for the internal use of the collate_fn functions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sort_by_length(self):\n self.lengths, indices = self.lengths.sort(descending=True)\n\n self.src = [self.src[idx] for idx in indices]\n self.indices = indices.tolist()\n self.sorted = True", "response": "Sorts the source dataset by the sequence length."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unsort(self, array):\n if self.sorted:\n inverse = sorted(enumerate(self.indices), key=itemgetter(1))\n array = [array[i[0]] for i in inverse]\n return array", "response": "\"Unsorts\" given array (restores original order of elements before\n dataset was sorted by sequence length).\n\n :param array: array to be \"unsorted\""} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef filter_data(self, min_len, max_len):\n logging.info(f'Filtering data, min len: {min_len}, max len: {max_len}')\n\n initial_len = len(self.src)\n filtered_src = []\n for src in self.src:\n if min_len <= len(src) <= max_len:\n filtered_src.append(src)\n\n self.src = filtered_src\n filtered_len = len(self.src)\n logging.info(f'Pairs before: {initial_len}, after: {filtered_len}')", "response": "Filter the data from the source list to only samples which satisfy the inequality min_len < max_len."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_data(self, fname, tokenizer, max_size):\n logging.info(f'Processing data from {fname}')\n data = []\n with open(fname) as dfile:\n for idx, line in enumerate(dfile):\n if max_size and idx == max_size:\n break\n entry = tokenizer.segment(line)\n entry = torch.tensor(entry)\n data.append(entry)\n return data", "response": "Loads data from the input file fname and returns a list of torch. Tensor objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsorting the source and target entries by the sequence length.", "response": "def sort_by_length(self):\n \"\"\"\n Sorts dataset by the sequence length.\n \"\"\"\n self.lengths, indices = self.lengths.sort(descending=True)\n\n self.src = [self.src[idx] for idx in indices]\n self.tgt = [self.tgt[idx] for idx in indices]\n self.src_lengths = [self.src_lengths[idx] for idx in indices]\n self.tgt_lengths = [self.tgt_lengths[idx] for idx in indices]\n self.indices = indices.tolist()\n self.sorted = True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter_data(self, min_len, max_len):\n logging.info(f'Filtering data, min len: {min_len}, max len: {max_len}')\n\n initial_len = len(self.src)\n filtered_src = []\n filtered_tgt = []\n for src, tgt in zip(self.src, self.tgt):\n if min_len <= len(src) <= max_len and \\\n min_len <= len(tgt) <= max_len:\n filtered_src.append(src)\n filtered_tgt.append(tgt)\n\n self.src = filtered_src\n self.tgt = filtered_tgt\n filtered_len = len(self.src)\n logging.info(f'Pairs before: {initial_len}, after: {filtered_len}')", "response": "Filter the data for the inequality of the source and target sequences."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_raw_data(self, fname, max_size):\n logging.info(f'Processing data from {fname}')\n data = []\n with open(fname) as dfile:\n for idx, line in enumerate(dfile):\n if max_size and idx == max_size:\n break\n data.append(line)\n return data", "response": "Loads the raw data from the input file fname."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfiltering out the raw data for the inequality of the source and target inequality.", "response": "def filter_raw_data(self, min_len, max_len):\n \"\"\"\n Preserves only samples which satisfy the following inequality:\n min_len <= src sample sequence length <= max_len AND\n min_len <= tgt sample sequence length <= max_len\n\n :param min_len: minimum sequence length\n :param max_len: maximum sequence length\n \"\"\"\n initial_len = len(self.raw_src)\n filtered_src = []\n filtered_tgt = []\n filtered_src_len = []\n filtered_tgt_len = []\n for src, tgt in zip(self.raw_src, self.raw_tgt):\n src_len = src.count(' ') + 1\n tgt_len = tgt.count(' ') + 1\n if min_len <= src_len <= max_len and \\\n min_len <= tgt_len <= max_len:\n filtered_src.append(src)\n filtered_tgt.append(tgt)\n filtered_src_len.append(src_len)\n filtered_tgt_len.append(tgt_len)\n\n self.raw_src = filtered_src\n self.raw_tgt = filtered_tgt\n self.src_len = filtered_src_len\n self.tgt_len = filtered_tgt_len\n filtered_len = len(self.raw_src)\n logging.info(f'Pairs before: {initial_len}, after: {filtered_len}')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef heatmaps_to_keypoints(maps, rois):\n # This function converts a discrete image coordinate in a HEATMAP_SIZE x\n # HEATMAP_SIZE image to a continuous keypoint coordinate. We maintain\n # consistency with keypoints_to_heatmap_labels by using the conversion from\n # Heckbert 1990: c = d + 0.5, where d is a discrete coordinate and c is a\n # continuous coordinate.\n offset_x = rois[:, 0]\n offset_y = rois[:, 1]\n\n widths = rois[:, 2] - rois[:, 0]\n heights = rois[:, 3] - rois[:, 1]\n widths = np.maximum(widths, 1)\n heights = np.maximum(heights, 1)\n widths_ceil = np.ceil(widths)\n heights_ceil = np.ceil(heights)\n\n # NCHW to NHWC for use with OpenCV\n maps = np.transpose(maps, [0, 2, 3, 1])\n min_size = 0 # cfg.KRCNN.INFERENCE_MIN_SIZE\n num_keypoints = maps.shape[3]\n xy_preds = np.zeros((len(rois), 3, num_keypoints), dtype=np.float32)\n end_scores = np.zeros((len(rois), num_keypoints), dtype=np.float32)\n for i in range(len(rois)):\n if min_size > 0:\n roi_map_width = int(np.maximum(widths_ceil[i], min_size))\n roi_map_height = int(np.maximum(heights_ceil[i], min_size))\n else:\n roi_map_width = widths_ceil[i]\n roi_map_height = heights_ceil[i]\n width_correction = widths[i] / roi_map_width\n height_correction = heights[i] / roi_map_height\n roi_map = cv2.resize(\n maps[i], (roi_map_width, roi_map_height), interpolation=cv2.INTER_CUBIC\n )\n # Bring back to CHW\n roi_map = np.transpose(roi_map, [2, 0, 1])\n # roi_map_probs = scores_to_probs(roi_map.copy())\n w = roi_map.shape[2]\n pos = roi_map.reshape(num_keypoints, -1).argmax(axis=1)\n x_int = pos % w\n y_int = (pos - x_int) // w\n # assert (roi_map_probs[k, y_int, x_int] ==\n # roi_map_probs[k, :, :].max())\n x = (x_int + 0.5) * width_correction\n y = (y_int + 0.5) * height_correction\n xy_preds[i, 0, :] = x + offset_x[i]\n xy_preds[i, 1, :] = y + offset_y[i]\n xy_preds[i, 2, :] = 1\n end_scores[i, :] = roi_map[np.arange(num_keypoints), y_int, x_int]\n\n return np.transpose(xy_preds, [0, 2, 1]), end_scores", "response": "Extract predicted keypoint locations from heatmaps. Output has shape ( x y logit prob ) where x y is the keypoint location of each keypoint."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds a function that can be used during serving.", "response": "def build_tensor_serving_input_receiver_fn(shape, dtype=tf.float32,\n batch_size=1):\n \"\"\"Returns a input_receiver_fn that can be used during serving.\n\n This expects examples to come through as float tensors, and simply\n wraps them as TensorServingInputReceivers.\n\n Arguably, this should live in tf.estimator.export. Testing here first.\n\n Args:\n shape: list representing target size of a single example.\n dtype: the expected datatype for the input example\n batch_size: number of input tensors that will be passed for prediction\n\n Returns:\n A function that itself returns a TensorServingInputReceiver.\n \"\"\"\n def serving_input_receiver_fn():\n # Prep a placeholder where the input example will be fed in\n features = tf.placeholder(\n dtype=dtype, shape=[batch_size] + shape, name='input_tensor')\n\n return tf.estimator.export.TensorServingInputReceiver(\n features=features, receiver_tensors=features)\n\n return serving_input_receiver_fn"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates which keypoints are contained inside a given box.", "response": "def _within_box(points, boxes):\n \"\"\"Validate which keypoints are contained inside a given box.\n points: NxKx2\n boxes: Nx4\n output: NxK\n \"\"\"\n x_within = (points[..., 0] >= boxes[:, 0, None]) & (\n points[..., 0] <= boxes[:, 2, None]\n )\n y_within = (points[..., 1] >= boxes[:, 1, None]) & (\n points[..., 1] <= boxes[:, 3, None]\n )\n return x_within & y_within"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef begin(self):\n self._global_step_tensor = tf.train.get_global_step()\n if self._global_step_tensor is None:\n raise RuntimeError(\n 'Global step should be created to use StepCounterHook.')", "response": "Called once before using the session to check global step."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef after_run(self, run_context, run_values): # pylint: disable=unused-argument\n global_step = run_values.results\n\n if self._timer.should_trigger_for_step(\n global_step) and global_step > self._warm_steps:\n elapsed_time, elapsed_steps = self._timer.update_last_triggered_step(\n global_step)\n if elapsed_time is not None:\n self._step_train_time += elapsed_time\n self._total_steps += elapsed_steps\n\n # average examples per second is based on the total (accumulative)\n # training steps and training time so far\n average_examples_per_sec = self._batch_size * (\n self._total_steps / self._step_train_time)\n # current examples per second is based on the elapsed training steps\n # and training time per batch\n current_examples_per_sec = self._batch_size * (\n elapsed_steps / elapsed_time)\n # Current examples/sec followed by average examples/sec\n tf.logging.info('Batch [%g]: current exp/sec = %g, average exp/sec = '\n '%g', self._total_steps, current_examples_per_sec,\n average_examples_per_sec)", "response": "Called after each run."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvisualizing keypoints in a single image.", "response": "def vis_keypoints(img, kps, kp_thresh=2, alpha=0.7):\n \"\"\"Visualizes keypoints (adapted from vis_one_image).\n kps has shape (4, #keypoints) where 4 rows are (x, y, logit, prob).\n \"\"\"\n dataset_keypoints = PersonKeypoints.NAMES\n kp_lines = PersonKeypoints.CONNECTIONS\n\n # Convert from plt 0-1 RGBA colors to 0-255 BGR colors for opencv.\n cmap = plt.get_cmap('rainbow')\n colors = [cmap(i) for i in np.linspace(0, 1, len(kp_lines) + 2)]\n colors = [(c[2] * 255, c[1] * 255, c[0] * 255) for c in colors]\n\n # Perform the drawing on a copy of the image, to allow for blending.\n kp_mask = np.copy(img)\n\n # Draw mid shoulder / mid hip first for better visualization.\n mid_shoulder = (\n kps[:2, dataset_keypoints.index('right_shoulder')] +\n kps[:2, dataset_keypoints.index('left_shoulder')]) / 2.0\n sc_mid_shoulder = np.minimum(\n kps[2, dataset_keypoints.index('right_shoulder')],\n kps[2, dataset_keypoints.index('left_shoulder')])\n mid_hip = (\n kps[:2, dataset_keypoints.index('right_hip')] +\n kps[:2, dataset_keypoints.index('left_hip')]) / 2.0\n sc_mid_hip = np.minimum(\n kps[2, dataset_keypoints.index('right_hip')],\n kps[2, dataset_keypoints.index('left_hip')])\n nose_idx = dataset_keypoints.index('nose')\n if sc_mid_shoulder > kp_thresh and kps[2, nose_idx] > kp_thresh:\n cv2.line(\n kp_mask, tuple(mid_shoulder), tuple(kps[:2, nose_idx]),\n color=colors[len(kp_lines)], thickness=2, lineType=cv2.LINE_AA)\n if sc_mid_shoulder > kp_thresh and sc_mid_hip > kp_thresh:\n cv2.line(\n kp_mask, tuple(mid_shoulder), tuple(mid_hip),\n color=colors[len(kp_lines) + 1], thickness=2, lineType=cv2.LINE_AA)\n\n # Draw the keypoints.\n for l in range(len(kp_lines)):\n i1 = kp_lines[l][0]\n i2 = kp_lines[l][1]\n p1 = kps[0, i1], kps[1, i1]\n p2 = kps[0, i2], kps[1, i2]\n if kps[2, i1] > kp_thresh and kps[2, i2] > kp_thresh:\n cv2.line(\n kp_mask, p1, p2,\n color=colors[l], thickness=2, lineType=cv2.LINE_AA)\n if kps[2, i1] > kp_thresh:\n cv2.circle(\n kp_mask, p1,\n radius=3, color=colors[l], thickness=-1, lineType=cv2.LINE_AA)\n if kps[2, i2] > kp_thresh:\n cv2.circle(\n kp_mask, p2,\n radius=3, color=colors[l], thickness=-1, lineType=cv2.LINE_AA)\n\n # Blend the keypoints.\n return cv2.addWeighted(img, 1.0 - alpha, kp_mask, alpha, 0)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_transform(self):\n cfg = self.cfg\n\n # we are loading images with OpenCV, so we don't need to convert them\n # to BGR, they are already! So all we need to do is to normalize\n # by 255 if we want to convert to BGR255 format, or flip the channels\n # if we want it to be in RGB in [0-1] range.\n if cfg.INPUT.TO_BGR255:\n to_bgr_transform = T.Lambda(lambda x: x * 255)\n else:\n to_bgr_transform = T.Lambda(lambda x: x[[2, 1, 0]])\n\n normalize_transform = T.Normalize(\n mean=cfg.INPUT.PIXEL_MEAN, std=cfg.INPUT.PIXEL_STD\n )\n\n transform = T.Compose(\n [\n T.ToPILImage(),\n T.Resize(self.min_image_size),\n T.ToTensor(),\n to_bgr_transform,\n normalize_transform,\n ]\n )\n return transform", "response": "Builds a basic transformation that was used to train the models."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_on_opencv_image(self, image):\n predictions = self.compute_prediction(image)\n top_predictions = self.select_top_predictions(predictions)\n\n result = image.copy()\n if self.show_mask_heatmaps:\n return self.create_mask_montage(result, top_predictions)\n result = self.overlay_boxes(result, top_predictions)\n if self.cfg.MODEL.MASK_ON:\n result = self.overlay_mask(result, top_predictions)\n if self.cfg.MODEL.KEYPOINT_ON:\n result = self.overlay_keypoints(result, top_predictions)\n result = self.overlay_class_names(result, top_predictions)\n\n return result", "response": "Runs the detection on an image."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes the prediction of a single image.", "response": "def compute_prediction(self, original_image):\n \"\"\"\n Arguments:\n original_image (np.ndarray): an image as returned by OpenCV\n\n Returns:\n prediction (BoxList): the detected objects. Additional information\n of the detection properties can be found in the fields of\n the BoxList via `prediction.fields()`\n \"\"\"\n # apply pre-processing to image\n image = self.transforms(original_image)\n # convert to an ImageList, padded so that it is divisible by\n # cfg.DATALOADER.SIZE_DIVISIBILITY\n image_list = to_image_list(image, self.cfg.DATALOADER.SIZE_DIVISIBILITY)\n image_list = image_list.to(self.device)\n # compute predictions\n with torch.no_grad():\n predictions = self.model(image_list)\n predictions = [o.to(self.cpu_device) for o in predictions]\n\n # always single image is passed at a time\n prediction = predictions[0]\n\n # reshape prediction (a BoxList) into the original image size\n height, width = original_image.shape[:-1]\n prediction = prediction.resize((width, height))\n\n if prediction.has_field(\"mask\"):\n # if we have masks, paste the masks in the right position\n # in the image, as defined by the bounding boxes\n masks = prediction.get_field(\"mask\")\n # always single image is passed at a time\n masks = self.masker([masks], [prediction])[0]\n prediction.add_field(\"mask\", masks)\n return prediction"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef select_top_predictions(self, predictions):\n scores = predictions.get_field(\"scores\")\n keep = torch.nonzero(scores > self.confidence_threshold).squeeze(1)\n predictions = predictions[keep]\n scores = predictions.get_field(\"scores\")\n _, idx = scores.sort(0, descending=True)\n return predictions[idx]", "response": "Select only predictions which have a score > self. confidence_threshold and returns the predictions in descending order of score\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compute_colors_for_labels(self, labels):\n colors = labels[:, None] * self.palette\n colors = (colors % 255).numpy().astype(\"uint8\")\n return colors", "response": "Simple function that adds fixed colors depending on the class\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\noverlays the boxes of the image with the predicted boxes.", "response": "def overlay_boxes(self, image, predictions):\n \"\"\"\n Adds the predicted boxes on top of the image\n\n Arguments:\n image (np.ndarray): an image as returned by OpenCV\n predictions (BoxList): the result of the computation by the model.\n It should contain the field `labels`.\n \"\"\"\n labels = predictions.get_field(\"labels\")\n boxes = predictions.bbox\n\n colors = self.compute_colors_for_labels(labels).tolist()\n\n for box, color in zip(boxes, colors):\n box = box.to(torch.int64)\n top_left, bottom_right = box[:2].tolist(), box[2:].tolist()\n image = cv2.rectangle(\n image, tuple(top_left), tuple(bottom_right), tuple(color), 1\n )\n\n return image"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\noverlaying the image with the mask and labels.", "response": "def overlay_mask(self, image, predictions):\n \"\"\"\n Adds the instances contours for each predicted object.\n Each label has a different color.\n\n Arguments:\n image (np.ndarray): an image as returned by OpenCV\n predictions (BoxList): the result of the computation by the model.\n It should contain the field `mask` and `labels`.\n \"\"\"\n masks = predictions.get_field(\"mask\").numpy()\n labels = predictions.get_field(\"labels\")\n\n colors = self.compute_colors_for_labels(labels).tolist()\n\n for mask, color in zip(masks, colors):\n thresh = mask[0, :, :, None]\n contours, hierarchy = cv2_util.findContours(\n thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE\n )\n image = cv2.drawContours(image, contours, -1, color, 3)\n\n composite = image\n\n return composite"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_mask_montage(self, image, predictions):\n masks = predictions.get_field(\"mask\")\n masks_per_dim = self.masks_per_dim\n masks = L.interpolate(\n masks.float(), scale_factor=1 / masks_per_dim\n ).byte()\n height, width = masks.shape[-2:]\n max_masks = masks_per_dim ** 2\n masks = masks[:max_masks]\n # handle case where we have less detections than max_masks\n if len(masks) < max_masks:\n masks_padded = torch.zeros(max_masks, 1, height, width, dtype=torch.uint8)\n masks_padded[: len(masks)] = masks\n masks = masks_padded\n masks = masks.reshape(masks_per_dim, masks_per_dim, height, width)\n result = torch.zeros(\n (masks_per_dim * height, masks_per_dim * width), dtype=torch.uint8\n )\n for y in range(masks_per_dim):\n start_y = y * height\n end_y = (y + 1) * height\n for x in range(masks_per_dim):\n start_x = x * width\n end_x = (x + 1) * width\n result[start_y:end_y, start_x:end_x] = masks[y, x]\n return cv2.applyColorMap(result.numpy(), cv2.COLORMAP_JET)", "response": "Create a montage showing the probability heatmaps for each one of the objects in the image."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef overlay_class_names(self, image, predictions):\n scores = predictions.get_field(\"scores\").tolist()\n labels = predictions.get_field(\"labels\").tolist()\n labels = [self.CATEGORIES[i] for i in labels]\n boxes = predictions.bbox\n\n template = \"{}: {:.2f}\"\n for box, score, label in zip(boxes, scores, labels):\n x, y = box[:2]\n s = template.format(label, score)\n cv2.putText(\n image, s, (x, y), cv2.FONT_HERSHEY_SIMPLEX, .5, (255, 255, 255), 1\n )\n\n return image", "response": "Overlays detected class names and scores in the positions defined by the top - left corner of the predicted bounding box."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncollects the CPU information for the local environment.", "response": "def _collect_cpu_info(run_info):\n \"\"\"Collect the CPU information for the local environment.\"\"\"\n cpu_info = {}\n\n cpu_info[\"num_cores\"] = multiprocessing.cpu_count()\n\n # Note: cpuinfo is not installed in the TensorFlow OSS tree.\n # It is installable via pip.\n import cpuinfo # pylint: disable=g-import-not-at-top\n\n info = cpuinfo.get_cpu_info()\n cpu_info[\"cpu_info\"] = info[\"brand\"]\n cpu_info[\"mhz_per_cpu\"] = info[\"hz_advertised_raw\"][0] / 1.0e6\n\n run_info[\"machine_config\"][\"cpu_info\"] = cpu_info"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncollects GPU information by TF device library.", "response": "def _collect_gpu_info(run_info):\n \"\"\"Collect local GPU information by TF device library.\"\"\"\n gpu_info = {}\n local_device_protos = device_lib.list_local_devices()\n\n gpu_info[\"count\"] = len([d for d in local_device_protos\n if d.device_type == \"GPU\"])\n # The device description usually is a JSON string, which contains the GPU\n # model info, eg:\n # \"device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0\"\n for d in local_device_protos:\n if d.device_type == \"GPU\":\n gpu_info[\"model\"] = _parse_gpu_model(d.physical_device_desc)\n # Assume all the GPU connected are same model\n break\n run_info[\"machine_config\"][\"gpu_info\"] = gpu_info"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef log_estimator_evaluation_result(self, eval_results):\n if not isinstance(eval_results, dict):\n tf.logging.warning(\"eval_results should be directory for logging. Got %s\",\n type(eval_results))\n return\n global_step = eval_results[tf.GraphKeys.GLOBAL_STEP]\n for key in sorted(eval_results):\n if key != tf.GraphKeys.GLOBAL_STEP:\n self.log_metric(key, eval_results[key], global_step=global_step)", "response": "Log the evaluation result for a estimator."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef log_metric(self, name, value, unit=None, global_step=None, extras=None):\n if not isinstance(value, numbers.Number):\n tf.logging.warning(\n \"Metric value to log should be a number. Got %s\", type(value))\n return\n if extras:\n extras = [{\"name\": k, \"value\": v} for k, v in sorted(extras.items())]\n else:\n extras = []\n with tf.gfile.GFile(\n os.path.join(self._logging_dir, METRIC_LOG_FILE_NAME), \"a\") as f:\n metric = {\n \"name\": name,\n \"value\": float(value),\n \"unit\": unit,\n \"global_step\": global_step,\n \"timestamp\": datetime.datetime.now().strftime(\n _DATE_TIME_FORMAT_PATTERN),\n \"extras\": extras}\n try:\n json.dump(metric, f)\n f.write(\"\\n\")\n except (TypeError, ValueError) as e:\n tf.logging.warning(\"Failed to dump metric to log file: \"\n \"name %s, value %s, error %s\", name, value, e)", "response": "Logs the metric to the local file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlog the run info for the local env.", "response": "def log_run_info(self, model_name):\n \"\"\"Collect most of the TF runtime information for the local env.\n\n The schema of the run info follows official/benchmark/datastore/schema.\n\n Args:\n model_name: string, the name of the model.\n \"\"\"\n run_info = {\n \"model_name\": model_name,\n \"machine_config\": {},\n \"run_date\": datetime.datetime.now().strftime(_DATE_TIME_FORMAT_PATTERN)}\n _collect_tensorflow_info(run_info)\n _collect_tensorflow_environment_variables(run_info)\n _collect_cpu_info(run_info)\n _collect_gpu_info(run_info)\n _collect_memory_info(run_info)\n\n with tf.gfile.GFile(os.path.join(\n self._logging_dir, BENCHMARK_RUN_LOG_FILE_NAME), \"w\") as f:\n try:\n json.dump(run_info, f)\n f.write(\"\\n\")\n except (TypeError, ValueError) as e:\n tf.logging.warning(\"Failed to dump benchmark run info to log file: %s\",\n e)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read_existing_paths(bt_table):\n rows = bt_table.read_rows(\n filter_=row_filters.ColumnRangeFilter(\n METADATA, SGF_FILENAME, SGF_FILENAME))\n names = (row.cell_value(METADATA, SGF_FILENAME).decode() for row in rows)\n processed = [os.path.splitext(os.path.basename(r))[0] for r in names]\n return processed", "response": "Return the SGF filename for each existing eval record."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef canonical_name(sgf_name):\n sgf_name = os.path.normpath(sgf_name)\n assert sgf_name.endswith('.sgf'), sgf_name\n # Strip off '.sgf'\n sgf_name = sgf_name[:-4]\n\n # Often eval is inside a folder with the run name.\n # include from folder before /eval/ if part of path.\n with_folder = re.search(r'/([^/]*/eval/.*)', sgf_name)\n if with_folder:\n return with_folder.group(1)\n\n # Return the filename\n return os.path.basename(sgf_name)", "response": "Keep filename and some date folders"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads all SGFs that match glob and extract relevant metadata for eval games table.", "response": "def read_games(glob, existing_paths):\n \"\"\"Read all SGFs that match glob\n\n Parse each game and extract relevant metadata for eval games table.\n \"\"\"\n\n globbed = sorted(gfile.Glob(glob))\n\n skipped = 0\n to_parse = []\n for sgf_name in tqdm(globbed):\n assert sgf_name.lower().endswith('.sgf'), sgf_name\n sgf_path = canonical_name(sgf_name)\n sgf_filename = os.path.basename(sgf_path)\n\n if sgf_path in existing_paths or sgf_filename in existing_paths:\n skipped += 1\n continue\n\n to_parse.append(sgf_name)\n\n game_data = []\n with multiprocessing.Pool() as pool:\n game_data = pool.map(process_game, tqdm(to_parse), 100)\n\n print(\"Read {} SGFs, {} new, {} existing\".format(\n len(globbed), len(game_data), skipped))\n return game_data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write_eval_records(bt_table, game_data, last_game):\n eval_num = last_game\n\n # Each column counts as a mutation so max rows is ~10000\n GAMES_PER_COMMIT = 2000\n for games in grouper(tqdm(game_data), GAMES_PER_COMMIT):\n assert bt_table.read_row(EVAL_PREFIX.format(eval_num)), \"Prev row doesn't exists\"\n assert bt_table.read_row(EVAL_PREFIX.format(eval_num+1)) is None, \"Row already exists\"\n\n rows = []\n for i, metadata in enumerate(games):\n eval_num += 1\n row_name = EVAL_PREFIX.format(eval_num)\n row = bt_table.row(row_name)\n for column, value in metadata:\n row.set_cell(METADATA, column, value)\n rows.append(row)\n # For each batch of games print a couple of the rows being added.\n if i < 5 or i + 5 > len(games):\n print(\"\\t\", i, row_name, metadata[6][1])\n\n if eval_num == last_game + len(games):\n test = input(\"Commit ('y'/'yes' required): \")\n if test.lower() not in ('y', 'yes'):\n break\n\n # TODO(derek): Figure out how to condition on atomic counter update.\n # Condition all updates on the current value of last_game\n\n game_num_update = bt_table.row(TABLE_STATE)\n game_num_update.set_cell(METADATA, EVAL_GAME_COUNTER, eval_num)\n print(TABLE_STATE, eval_num)\n\n response = bt_table.mutate_rows(rows)\n\n # validate that all rows written successfully\n any_bad = False\n for i, status in enumerate(response):\n if status.code is not 0:\n print(\"Row number {} failed to write {}\".format(i, status))\n any_bad = True\n if any_bad:\n break\n\n game_num_update.commit()", "response": "Write all eval records to eval_table."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ninitialize the reinforcement learning loop from a checkpoint.", "response": "def initialize_from_checkpoint(state):\n \"\"\"Initialize the reinforcement learning loop from a checkpoint.\"\"\"\n\n # The checkpoint's work_dir should contain the most recently trained model.\n model_paths = glob.glob(os.path.join(FLAGS.checkpoint_dir,\n 'work_dir/model.ckpt-*.pb'))\n if len(model_paths) != 1:\n raise RuntimeError('Expected exactly one model in the checkpoint work_dir, '\n 'got [{}]'.format(', '.join(model_paths)))\n start_model_path = model_paths[0]\n\n # Copy the latest trained model into the models directory and use it on the\n # first round of selfplay.\n state.best_model_name = 'checkpoint'\n shutil.copy(start_model_path,\n os.path.join(fsdb.models_dir(), state.best_model_name + '.pb'))\n\n # Copy the training chunks.\n golden_chunks_dir = os.path.join(FLAGS.checkpoint_dir, 'golden_chunks')\n for basename in os.listdir(golden_chunks_dir):\n path = os.path.join(golden_chunks_dir, basename)\n shutil.copy(path, fsdb.golden_chunk_dir())\n\n # Copy the training files.\n work_dir = os.path.join(FLAGS.checkpoint_dir, 'work_dir')\n for basename in os.listdir(work_dir):\n path = os.path.join(work_dir, basename)\n shutil.copy(path, fsdb.working_dir())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrun the given subprocess command in a coroutine.", "response": "async def run(*cmd):\n \"\"\"Run the given subprocess command in a coroutine.\n\n Args:\n *cmd: the command to run and its arguments.\n\n Returns:\n The output that the command wrote to stdout as a list of strings, one line\n per element (stderr output is piped to stdout).\n\n Raises:\n RuntimeError: if the command returns a non-zero result.\n \"\"\"\n\n stdout = await checked_run(*cmd)\n\n log_path = os.path.join(FLAGS.base_dir, get_cmd_name(cmd) + '.log')\n with gfile.Open(log_path, 'a') as f:\n f.write(expand_cmd_str(cmd))\n f.write('\\n')\n f.write(stdout)\n f.write('\\n')\n\n # Split stdout into lines.\n return stdout.split('\\n')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning up to num_records of golden chunks to train on.", "response": "def get_golden_chunk_records():\n \"\"\"Return up to num_records of golden chunks to train on.\n\n Returns:\n A list of golden chunks up to num_records in length, sorted by path.\n \"\"\"\n\n pattern = os.path.join(fsdb.golden_chunk_dir(), '*.zz')\n return sorted(tf.gfile.Glob(pattern), reverse=True)[:FLAGS.window_size]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def selfplay(state, flagfile='selfplay'):\n\n output_dir = os.path.join(fsdb.selfplay_dir(), state.output_model_name)\n holdout_dir = os.path.join(fsdb.holdout_dir(), state.output_model_name)\n\n lines = await run(\n 'bazel-bin/cc/selfplay',\n '--flagfile={}.flags'.format(os.path.join(FLAGS.flags_dir, flagfile)),\n '--model={}'.format(state.best_model_path),\n '--output_dir={}'.format(output_dir),\n '--holdout_dir={}'.format(holdout_dir),\n '--seed={}'.format(state.seed))\n result = '\\n'.join(lines[-6:])\n logging.info(result)\n stats = parse_win_stats_table(result, 1)[0]\n num_games = stats.total_wins\n logging.info('Black won %0.3f, white won %0.3f',\n stats.black_wins.total / num_games,\n stats.white_wins.total / num_games)\n\n # Write examples to a single record.\n pattern = os.path.join(output_dir, '*', '*.zz')\n random.seed(state.seed)\n tf.set_random_seed(state.seed)\n np.random.seed(state.seed)\n # TODO(tommadams): This method of generating one golden chunk per generation\n # is sub-optimal because each chunk gets reused multiple times for training,\n # introducing bias. Instead, a fresh dataset should be uniformly sampled out\n # of *all* games in the training window before the start of each training run.\n buffer = example_buffer.ExampleBuffer(sampling_frac=1.0)\n\n # TODO(tommadams): parallel_fill is currently non-deterministic. Make it not\n # so.\n logging.info('Writing golden chunk from \"{}\"'.format(pattern))\n buffer.parallel_fill(tf.gfile.Glob(pattern))\n buffer.flush(os.path.join(fsdb.golden_chunk_dir(),\n state.output_model_name + '.tfrecord.zz'))", "response": "Run selfplay and write a training chunk to the fsdb golden_chunk_dir."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntrain a new model on the RL loop.", "response": "async def train(state, tf_records):\n \"\"\"Run training and write a new model to the fsdb models_dir.\n\n Args:\n state: the RL loop State instance.\n tf_records: a list of paths to TensorFlow records to train on.\n \"\"\"\n\n model_path = os.path.join(fsdb.models_dir(), state.train_model_name)\n await run(\n 'python3', 'train.py', *tf_records,\n '--flagfile={}'.format(os.path.join(FLAGS.flags_dir, 'train.flags')),\n '--work_dir={}'.format(fsdb.working_dir()),\n '--export_path={}'.format(model_path),\n '--training_seed={}'.format(state.seed),\n '--freeze=true')\n # Append the time elapsed from when the RL was started to when this model\n # was trained.\n elapsed = time.time() - state.start_time\n timestamps_path = os.path.join(fsdb.models_dir(), 'train_times.txt')\n with gfile.Open(timestamps_path, 'a') as f:\n print('{:.3f} {}'.format(elapsed, state.train_model_name), file=f)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating the trained model against holdout games.", "response": "async def validate(state, holdout_glob):\n \"\"\"Validate the trained model against holdout games.\n\n Args:\n state: the RL loop State instance.\n holdout_glob: a glob that matches holdout games.\n \"\"\"\n\n if not glob.glob(holdout_glob):\n print('Glob \"{}\" didn\\'t match any files, skipping validation'.format(\n holdout_glob))\n else:\n await run(\n 'python3', 'validate.py', holdout_glob,\n '--flagfile={}'.format(os.path.join(FLAGS.flags_dir, 'validate.flags')),\n '--work_dir={}'.format(fsdb.working_dir()))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def evaluate_model(eval_model_path, target_model_path, sgf_dir, seed):\n\n lines = await run(\n 'bazel-bin/cc/eval',\n '--flagfile={}'.format(os.path.join(FLAGS.flags_dir, 'eval.flags')),\n '--model={}'.format(eval_model_path),\n '--model_two={}'.format(target_model_path),\n '--sgf_dir={}'.format(sgf_dir),\n '--seed={}'.format(seed))\n result = '\\n'.join(lines[-7:])\n logging.info(result)\n eval_stats, target_stats = parse_win_stats_table(result, 2)\n num_games = eval_stats.total_wins + target_stats.total_wins\n win_rate = eval_stats.total_wins / num_games\n logging.info('Win rate %s vs %s: %.3f', eval_stats.model_name,\n target_stats.model_name, win_rate)\n return win_rate", "response": "Evaluate one model against a target."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def evaluate_trained_model(state):\n\n return await evaluate_model(\n state.train_model_path, state.best_model_path,\n os.path.join(fsdb.eval_dir(), state.train_model_name), state.seed)", "response": "Evaluate the most recently trained model against the current best model."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rl_loop():\n\n state = State()\n\n if FLAGS.checkpoint_dir:\n # Start from a partially trained model.\n initialize_from_checkpoint(state)\n else:\n # Play the first round of selfplay games with a fake model that returns\n # random noise. We do this instead of playing multiple games using a single\n # model bootstrapped with random noise to avoid any initial bias.\n wait(selfplay(state, 'bootstrap'))\n\n # Train a real model from the random selfplay games.\n tf_records = get_golden_chunk_records()\n state.iter_num += 1\n wait(train(state, tf_records))\n\n # Select the newly trained model as the best.\n state.best_model_name = state.train_model_name\n state.gen_num += 1\n\n # Run selfplay using the new model.\n wait(selfplay(state))\n\n # Now start the full training loop.\n while state.iter_num <= FLAGS.iterations:\n # Build holdout glob before incrementing the iteration number because we\n # want to run validation on the previous generation.\n holdout_glob = os.path.join(fsdb.holdout_dir(), '%06d-*' % state.iter_num,\n '*')\n\n # Train on shuffled game data from recent selfplay rounds.\n tf_records = get_golden_chunk_records()\n state.iter_num += 1\n wait(train(state, tf_records))\n\n if FLAGS.parallel_post_train:\n # Run eval, validation & selfplay in parallel.\n model_win_rate, _, _ = wait([\n evaluate_trained_model(state),\n validate(state, holdout_glob),\n selfplay(state)])\n else:\n # Run eval, validation & selfplay sequentially.\n model_win_rate = wait(evaluate_trained_model(state))\n wait(validate(state, holdout_glob))\n wait(selfplay(state))\n\n if model_win_rate >= FLAGS.gating_win_rate:\n # Promote the trained model to the best model and increment the generation\n # number.\n state.best_model_name = state.train_model_name\n state.gen_num += 1", "response": "The main reinforcement learning loop."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun the reinforcement learning loop.", "response": "def main(unused_argv):\n \"\"\"Run the reinforcement learning loop.\"\"\"\n\n print('Wiping dir %s' % FLAGS.base_dir, flush=True)\n shutil.rmtree(FLAGS.base_dir, ignore_errors=True)\n dirs = [fsdb.models_dir(), fsdb.selfplay_dir(), fsdb.holdout_dir(),\n fsdb.eval_dir(), fsdb.golden_chunk_dir(), fsdb.working_dir()]\n for d in dirs:\n ensure_dir_exists(d);\n\n # Copy the flag files so there's no chance of them getting accidentally\n # overwritten while the RL loop is running.\n flags_dir = os.path.join(FLAGS.base_dir, 'flags')\n shutil.copytree(FLAGS.flags_dir, flags_dir)\n FLAGS.flags_dir = flags_dir\n\n # Copy the target model to the models directory so we can find it easily.\n shutil.copy(FLAGS.target_path, os.path.join(fsdb.models_dir(), 'target.pb'))\n\n logging.getLogger().addHandler(\n logging.FileHandler(os.path.join(FLAGS.base_dir, 'rl_loop.log')))\n formatter = logging.Formatter('[%(asctime)s] %(message)s',\n '%Y-%m-%d %H:%M:%S')\n for handler in logging.getLogger().handlers:\n handler.setFormatter(formatter)\n\n with logged_timer('Total time'):\n try:\n rl_loop()\n finally:\n asyncio.get_event_loop().close()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nyield a sequence of PositionWithContext objects for each player in the position.", "response": "def replay_position(position, result):\n \"\"\"\n Wrapper for a go.Position which replays its history.\n Assumes an empty start position! (i.e. no handicap, and history must be exhaustive.)\n\n Result must be passed in, since a resign cannot be inferred from position\n history alone.\n\n for position_w_context in replay_position(position):\n print(position_w_context.position)\n \"\"\"\n assert position.n == len(position.recent), \"Position history is incomplete\"\n pos = Position(komi=position.komi)\n for player_move in position.recent:\n color, next_move = player_move\n yield PositionWithContext(pos, next_move, result)\n pos = pos.play_move(next_move, color=color)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if c is surrounded on all sides by 1 color and return that color", "response": "def is_koish(board, c):\n 'Check if c is surrounded on all sides by 1 color, and return that color'\n if board[c] != EMPTY:\n return None\n neighbors = {board[n] for n in NEIGHBORS[c]}\n if len(neighbors) == 1 and EMPTY not in neighbors:\n return list(neighbors)[0]\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if c is an eye for the purpose of restricting MC rollouts.", "response": "def is_eyeish(board, c):\n 'Check if c is an eye, for the purpose of restricting MC rollouts.'\n # pass is fine.\n if c is None:\n return\n color = is_koish(board, c)\n if color is None:\n return None\n diagonal_faults = 0\n diagonals = DIAGONALS[c]\n if len(diagonals) < 4:\n diagonal_faults += 1\n for d in diagonals:\n if not board[d] in (color, EMPTY):\n diagonal_faults += 1\n if diagonal_faults > 1:\n return None\n else:\n return color"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_move_legal(self, move):\n 'Checks that a move is on an empty space, not on ko, and not suicide'\n if move is None:\n return True\n if self.board[move] != EMPTY:\n return False\n if move == self.ko:\n return False\n if self.is_move_suicidal(move):\n return False\n\n return True", "response": "Checks that a move is on an empty space not on ko and not suicide"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef all_legal_moves(self):\n 'Returns a np.array of size go.N**2 + 1, with 1 = legal, 0 = illegal'\n # by default, every move is legal\n legal_moves = np.ones([N, N], dtype=np.int8)\n # ...unless there is already a stone there\n legal_moves[self.board != EMPTY] = 0\n # calculate which spots have 4 stones next to them\n # padding is because the edge always counts as a lost liberty.\n adjacent = np.ones([N + 2, N + 2], dtype=np.int8)\n adjacent[1:-1, 1:-1] = np.abs(self.board)\n num_adjacent_stones = (adjacent[:-2, 1:-1] + adjacent[1:-1, :-2] +\n adjacent[2:, 1:-1] + adjacent[1:-1, 2:])\n # Surrounded spots are those that are empty and have 4 adjacent stones.\n surrounded_spots = np.multiply(\n (self.board == EMPTY),\n (num_adjacent_stones == 4))\n # Such spots are possibly illegal, unless they are capturing something.\n # Iterate over and manually check each spot.\n for coord in np.transpose(np.nonzero(surrounded_spots)):\n if self.is_move_suicidal(tuple(coord)):\n legal_moves[tuple(coord)] = 0\n\n # ...and retaking ko is always illegal\n if self.ko is not None:\n legal_moves[self.ko] = 0\n\n # and pass is always legal\n return np.concatenate([legal_moves.ravel(), [1]])", "response": "Returns a np. array of size go. N ** 2 + 1 with 1 = legal 0 = illegal"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn score from B perspective. If W is winning score is negative.", "response": "def score(self):\n 'Return score from B perspective. If W is winning, score is negative.'\n working_board = np.copy(self.board)\n while EMPTY in working_board:\n unassigned_spaces = np.where(working_board == EMPTY)\n c = unassigned_spaces[0][0], unassigned_spaces[1][0]\n territory, borders = find_reached(working_board, c)\n border_colors = set(working_board[b] for b in borders)\n X_border = BLACK in border_colors\n O_border = WHITE in border_colors\n if X_border and not O_border:\n territory_color = BLACK\n elif O_border and not X_border:\n territory_color = WHITE\n else:\n territory_color = UNKNOWN # dame, or seki\n place_stones(working_board, territory_color, territory)\n\n return np.count_nonzero(working_board == BLACK) - np.count_nonzero(working_board == WHITE) - self.komi"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_sgf_to_examples(sgf_path):\n\n return zip(*[(p.position, p.next_move, p.result)\n for p in sgf_wrapper.replay_sgf_file(sgf_path)])", "response": "Parse a SGF file and return a list of examples."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind all sgf files in base_dir with year >= min_year and komi", "response": "def find_and_filter_sgf_files(base_dir, min_year=None, komi=None):\n \"\"\"Finds all sgf files in base_dir with year >= min_year and komi\"\"\"\n sgf_files = []\n for dirpath, dirnames, filenames in os.walk(base_dir):\n for filename in filenames:\n if filename.endswith('.sgf'):\n path = os.path.join(dirpath, filename)\n sgf_files.append(path)\n\n if min_year == komi == None:\n print (\"Found {} sgf_files\".format(len(sgf_files)))\n return sgf_files\n\n f = filter_year_komi(min_year, komi)\n filtered_sgf_files = [sgf for sgf in tqdm(sgf_files) if f(sgf)]\n\n print(\"{} of {} .sgf files matched (min_year >= {}, komi = {})\".format(\n len(filtered_sgf_files), len(sgf_files), min_year, komi))\n return filtered_sgf_files"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_model_paths(model_dir):\n all_models = gfile.Glob(os.path.join(model_dir, '*.meta'))\n model_filenames = [os.path.basename(m) for m in all_models]\n model_numbers_names = [\n (shipname.detect_model_num(m), shipname.detect_model_name(m))\n for m in model_filenames]\n model_names = sorted(model_numbers_names)\n return [os.path.join(model_dir, name[1]) for name in model_names]", "response": "Returns all model paths in the model_dir."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nturning a game into SGF.", "response": "def make_sgf(\n move_history,\n result_string,\n ruleset=\"Chinese\",\n komi=7.5,\n white_name=PROGRAM_IDENTIFIER,\n black_name=PROGRAM_IDENTIFIER,\n comments=[]\n):\n \"\"\"Turn a game into SGF.\n\n Doesn't handle handicap games or positions with incomplete history.\n\n Args:\n move_history: iterable of PlayerMoves\n result_string: \"B+R\", \"W+0.5\", etc.\n comments: iterable of string/None. Will be zipped with move_history.\n \"\"\"\n boardsize = go.N\n game_moves = ''.join(translate_sgf_move(*z)\n for z in itertools.zip_longest(move_history, comments))\n result = result_string\n return SGF_TEMPLATE.format(**locals())"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef handle_node(pos, node):\n 'A node can either add B+W stones, play as B, or play as W.'\n props = node.properties\n black_stones_added = [coords.from_sgf(\n c) for c in props.get('AB', [])]\n white_stones_added = [coords.from_sgf(\n c) for c in props.get('AW', [])]\n if black_stones_added or white_stones_added:\n return add_stones(pos, black_stones_added, white_stones_added)\n # If B/W props are not present, then there is no move. But if it is present and equal to the empty string, then the move was a pass.\n elif 'B' in props:\n black_move = coords.from_sgf(props.get('B', [''])[0])\n return pos.play_move(black_move, color=go.BLACK)\n elif 'W' in props:\n white_move = coords.from_sgf(props.get('W', [''])[0])\n return pos.play_move(white_move, color=go.WHITE)\n else:\n return pos", "response": "A node can either add B + W stones play as B or play as W."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef replay_sgf(sgf_contents):\n root_node = get_sgf_root_node(sgf_contents)\n props = root_node.properties\n assert int(sgf_prop(props.get('GM', ['1']))) == 1, \"Not a Go SGF!\"\n\n komi = 0\n if props.get('KM') is not None:\n komi = float(sgf_prop(props.get('KM')))\n result = utils.parse_game_result(sgf_prop(props.get('RE', '')))\n\n pos = Position(komi=komi)\n current_node = root_node\n while pos is not None and current_node.next is not None:\n pos = handle_node(pos, current_node)\n maybe_correct_next(pos, current_node.next)\n next_move = get_next_move(current_node)\n yield PositionWithContext(pos, next_move, result)\n current_node = current_node.next", "response": "Wrapper for the replay_sgf function that returns a generator that yields PositionWithContext instances."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\napply the decoder to inputs given the context from the encoder.", "response": "def decode(self, inputs, context, inference=False):\n \"\"\"\n Applies the decoder to inputs, given the context from the encoder.\n\n :param inputs: tensor with inputs (batch, seq_len) if 'batch_first'\n else (seq_len, batch)\n :param context: context from the encoder\n :param inference: if True inference mode, if False training mode\n \"\"\"\n return self.decoder(inputs, context, inference)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate(self, inputs, context, beam_size):\n logits, scores, new_context = self.decode(inputs, context, True)\n logprobs = log_softmax(logits, dim=-1)\n logprobs, words = logprobs.topk(beam_size, dim=-1)\n return words, logprobs, scores, new_context", "response": "Generates a new decoder from the given inputs and context."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds the eval_path for the current iteration and epoch.", "response": "def build_eval_path(self, epoch, iteration):\n \"\"\"\n Appends index of the current epoch and index of the current iteration\n to the name of the file with results.\n\n :param epoch: index of the current epoch\n :param iteration: index of the current iteration\n \"\"\"\n if iteration is not None:\n eval_fname = f'eval_epoch_{epoch}_iter_{iteration}'\n else:\n eval_fname = f'eval_epoch_{epoch}'\n eval_path = os.path.join(self.save_path, eval_fname)\n return eval_path"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run(self, calc_bleu=True, epoch=None, iteration=None, eval_path=None,\n summary=False, reference_path=None):\n \"\"\"\n Runs translation on test dataset.\n\n :param calc_bleu: if True compares results with reference and computes\n BLEU score\n :param epoch: index of the current epoch\n :param iteration: index of the current iteration\n :param eval_path: path to the file for saving results\n :param summary: if True prints summary\n :param reference_path: path to the file with reference translation\n \"\"\"\n if self.cuda:\n test_bleu = torch.cuda.FloatTensor([0])\n break_training = torch.cuda.LongTensor([0])\n else:\n test_bleu = torch.FloatTensor([0])\n break_training = torch.LongTensor([0])\n\n if eval_path is None:\n eval_path = self.build_eval_path(epoch, iteration)\n detok_eval_path = eval_path + '.detok'\n\n with contextlib.suppress(FileNotFoundError):\n os.remove(eval_path)\n os.remove(detok_eval_path)\n\n rank = get_rank()\n logging.info(f'Running evaluation on test set')\n self.model.eval()\n torch.cuda.empty_cache()\n\n output = self.evaluate(epoch, iteration, summary)\n output = output[:len(self.loader.dataset)]\n output = self.loader.dataset.unsort(output)\n\n if rank == 0:\n with open(eval_path, 'a') as eval_file:\n eval_file.writelines(output)\n if calc_bleu:\n self.run_detokenizer(eval_path)\n test_bleu[0] = self.run_sacrebleu(detok_eval_path, reference_path)\n if summary:\n logging.info(f'BLEU on test dataset: {test_bleu[0]:.2f}')\n\n if self.target_bleu and test_bleu[0] >= self.target_bleu:\n logging.info(f'Target accuracy reached')\n break_training[0] = 1\n\n barrier()\n torch.cuda.empty_cache()\n logging.info(f'Finished evaluation on test set')\n\n if self.distributed:\n dist.broadcast(break_training, 0)\n dist.broadcast(test_bleu, 0)\n\n return test_bleu[0].item(), break_training[0].item()", "response": "Runs translation on the current test set."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrun evaluation on test dataset.", "response": "def evaluate(self, epoch, iteration, summary):\n \"\"\"\n Runs evaluation on test dataset.\n\n :param epoch: index of the current epoch\n :param iteration: index of the current iteration\n :param summary: if True prints summary\n \"\"\"\n batch_time = AverageMeter(False)\n tot_tok_per_sec = AverageMeter(False)\n iterations = AverageMeter(False)\n enc_seq_len = AverageMeter(False)\n dec_seq_len = AverageMeter(False)\n stats = {}\n\n output = []\n\n for i, (src, indices) in enumerate(self.loader):\n translate_timer = time.time()\n src, src_length = src\n\n batch_size = self.loader.batch_size\n global_batch_size = batch_size * get_world_size()\n beam_size = self.beam_size\n\n bos = [self.insert_target_start] * (batch_size * beam_size)\n bos = torch.LongTensor(bos)\n if self.batch_first:\n bos = bos.view(-1, 1)\n else:\n bos = bos.view(1, -1)\n\n src_length = torch.LongTensor(src_length)\n stats['total_enc_len'] = int(src_length.sum())\n\n if self.cuda:\n src = src.cuda()\n src_length = src_length.cuda()\n bos = bos.cuda()\n\n with torch.no_grad():\n context = self.model.encode(src, src_length)\n context = [context, src_length, None]\n\n if beam_size == 1:\n generator = self.generator.greedy_search\n else:\n generator = self.generator.beam_search\n preds, lengths, counter = generator(batch_size, bos, context)\n\n stats['total_dec_len'] = lengths.sum().item()\n stats['iters'] = counter\n\n indices = torch.tensor(indices).to(preds)\n preds = preds.scatter(0, indices.unsqueeze(1).expand_as(preds), preds)\n\n preds = gather_predictions(preds).cpu()\n\n for pred in preds:\n pred = pred.tolist()\n detok = self.tokenizer.detokenize(pred)\n output.append(detok + '\\n')\n\n elapsed = time.time() - translate_timer\n batch_time.update(elapsed, batch_size)\n\n total_tokens = stats['total_dec_len'] + stats['total_enc_len']\n ttps = total_tokens / elapsed\n tot_tok_per_sec.update(ttps, batch_size)\n\n iterations.update(stats['iters'])\n enc_seq_len.update(stats['total_enc_len'] / batch_size, batch_size)\n dec_seq_len.update(stats['total_dec_len'] / batch_size, batch_size)\n\n if i % self.print_freq == 0:\n log = []\n log += f'TEST '\n if epoch is not None:\n log += f'[{epoch}]'\n if iteration is not None:\n log += f'[{iteration}]'\n log += f'[{i}/{len(self.loader)}]\\t'\n log += f'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\\t'\n log += f'Decoder iters {iterations.val:.1f} ({iterations.avg:.1f})\\t'\n log += f'Tok/s {tot_tok_per_sec.val:.0f} ({tot_tok_per_sec.avg:.0f})'\n log = ''.join(log)\n logging.info(log)\n\n tot_tok_per_sec.reduce('sum')\n enc_seq_len.reduce('mean')\n dec_seq_len.reduce('mean')\n batch_time.reduce('mean')\n iterations.reduce('sum')\n\n if summary and get_rank() == 0:\n time_per_sentence = (batch_time.avg / global_batch_size)\n log = []\n log += f'TEST SUMMARY:\\n'\n log += f'Lines translated: {len(self.loader.dataset)}\\t'\n log += f'Avg total tokens/s: {tot_tok_per_sec.avg:.0f}\\n'\n log += f'Avg time per batch: {batch_time.avg:.3f} s\\t'\n log += f'Avg time per sentence: {1000*time_per_sentence:.3f} ms\\n'\n log += f'Avg encoder seq len: {enc_seq_len.avg:.2f}\\t'\n log += f'Avg decoder seq len: {dec_seq_len.avg:.2f}\\t'\n log += f'Total decoder iterations: {int(iterations.sum)}'\n log = ''.join(log)\n logging.info(log)\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nexecute moses detokenizer on eval_path file and saves result to eval_path +. detok", "response": "def run_detokenizer(self, eval_path):\n \"\"\"\n Executes moses detokenizer on eval_path file and saves result to\n eval_path + \".detok\" file.\n\n :param eval_path: path to the tokenized input\n \"\"\"\n logging.info('Running detokenizer')\n detok_path = os.path.join(self.dataset_dir, config.DETOKENIZER)\n detok_eval_path = eval_path + '.detok'\n\n with open(detok_eval_path, 'w') as detok_eval_file, \\\n open(eval_path, 'r') as eval_file:\n subprocess.run(['perl', f'{detok_path}'], stdin=eval_file,\n stdout=detok_eval_file, stderr=subprocess.DEVNULL)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_sacrebleu(self, detok_eval_path, reference_path):\n if reference_path is None:\n reference_path = os.path.join(self.dataset_dir,\n config.TGT_TEST_TARGET_FNAME)\n sacrebleu_params = '--score-only -lc --tokenize intl'\n logging.info(f'Running sacrebleu (parameters: {sacrebleu_params})')\n sacrebleu = subprocess.run([f'sacrebleu --input {detok_eval_path} \\\n {reference_path} {sacrebleu_params}'],\n stdout=subprocess.PIPE, shell=True)\n test_bleu = float(sacrebleu.stdout.strip())\n return test_bleu", "response": "Runs sacrebleu and returns BLEU score."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef configure(project=LOGGING_PROJECT):\n if not project:\n sys.stderr.write('!! Error: The $LOGGING_PROJECT enviroment '\n 'variable is required in order to set up cloud logging. '\n 'Cloud logging is disabled.\\n')\n return\n\n try:\n # if this fails, redirect stderr to /dev/null so no startup spam.\n with contextlib.redirect_stderr(io.StringIO()):\n client = glog.Client(project)\n client.setup_logging(logging.INFO)\n except:\n logging.basicConfig(level=logging.INFO)\n sys.stderr.write('!! Cloud logging disabled\\n')", "response": "Configures cloud logging with the given project."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nalign the state_dict with the loaded_state_dict and updates the state_dict with the model_state_dict.", "response": "def align_and_update_state_dicts(model_state_dict, loaded_state_dict):\n \"\"\"\n Strategy: suppose that the models that we will create will have prefixes appended\n to each of its keys, for example due to an extra level of nesting that the original\n pre-trained weights from ImageNet won't contain. For example, model.state_dict()\n might return backbone[0].body.res2.conv1.weight, while the pre-trained model contains\n res2.conv1.weight. We thus want to match both parameters together.\n For that, we look for each model weight, look among all loaded keys if there is one\n that is a suffix of the current weight name, and use it if that's the case.\n If multiple matches exist, take the one with longest size\n of the corresponding name. For example, for the same model as before, the pretrained\n weight file can contain both res2.conv1.weight, as well as conv1.weight. In this case,\n we want to match backbone[0].body.conv1.weight to conv1.weight, and\n backbone[0].body.res2.conv1.weight to res2.conv1.weight.\n \"\"\"\n current_keys = sorted(list(model_state_dict.keys()))\n loaded_keys = sorted(list(loaded_state_dict.keys()))\n # get a matrix of string matches, where each (i, j) entry correspond to the size of the\n # loaded_key string, if it matches\n match_matrix = [\n len(j) if i.endswith(j) else 0 for i in current_keys for j in loaded_keys\n ]\n match_matrix = torch.as_tensor(match_matrix).view(\n len(current_keys), len(loaded_keys)\n )\n max_match_size, idxs = match_matrix.max(1)\n # remove indices that correspond to no-match\n idxs[max_match_size == 0] = -1\n\n # used for logging\n max_size = max([len(key) for key in current_keys]) if current_keys else 1\n max_size_loaded = max([len(key) for key in loaded_keys]) if loaded_keys else 1\n log_str_template = \"{: <{}} loaded from {: <{}} of shape {}\"\n logger = logging.getLogger(__name__)\n for idx_new, idx_old in enumerate(idxs.tolist()):\n if idx_old == -1:\n continue\n key = current_keys[idx_new]\n key_old = loaded_keys[idx_old]\n model_state_dict[key] = loaded_state_dict[key_old]\n logger.info(\n log_str_template.format(\n key,\n max_size,\n key_old,\n max_size_loaded,\n tuple(loaded_state_dict[key_old].shape),\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the ratio of gradient norm to weight norm.", "response": "def compute_update_ratio(weight_tensors, before_weights, after_weights):\n \"\"\"Compute the ratio of gradient norm to weight norm.\"\"\"\n deltas = [after - before for after,\n before in zip(after_weights, before_weights)]\n delta_norms = [np.linalg.norm(d.ravel()) for d in deltas]\n weight_norms = [np.linalg.norm(w.ravel()) for w in before_weights]\n ratios = [d / w for d, w in zip(delta_norms, weight_norms)]\n all_summaries = [\n tf.Summary.Value(tag='update_ratios/' +\n tensor.name, simple_value=ratio)\n for tensor, ratio in zip(weight_tensors, ratios)]\n return tf.Summary(value=all_summaries)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef main(argv):\n tf_records = argv[1:]\n logging.info(\"Training on %s records: %s to %s\",\n len(tf_records), tf_records[0], tf_records[-1])\n with utils.logged_timer(\"Training\"):\n train(*tf_records)\n if FLAGS.export_path:\n dual_net.export_model(FLAGS.export_path)\n if FLAGS.freeze:\n if FLAGS.use_tpu:\n dual_net.freeze_graph_tpu(FLAGS.export_path)\n else:\n dual_net.freeze_graph(FLAGS.export_path)", "response": "Train on examples and export the updated model weights."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main(unused_argv):\n if FLAGS.use_tpu:\n dual_net.freeze_graph_tpu(FLAGS.model_path)\n else:\n dual_net.freeze_graph(FLAGS.model_path)", "response": "Freeze a model to a GraphDef proto."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef grouper(n, iterable):\n return (iterable[i:i + n] for i in range(0, len(iterable), n))", "response": "Returns a list of n elements from an iterable."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsample num_positions postions from each game in sgf_dir", "response": "def subsample():\n \"\"\"Sample num_positions postions from each game in sgf_dir\n\n Usage:\n python3 sharp_positions.py subsample --num_positions 10 --sgf_dir data/s\n\n NOTE(sethtroisi): see link for a script to truncate SGFs at move number\n https://github.com/sethtroisi/go-scripts\n \"\"\"\n\n sgf_files = oneoff_utils.find_and_filter_sgf_files(\n FLAGS.sgf_dir, None, None)\n\n with open(FLAG.collection, 'w') as collection:\n fails = 0\n for path in tqdm(sorted(sgf_files)):\n try:\n positions, moves, results = oneoff_utils.parse_sgf(path)\n except KeyboardInterrupt:\n raise\n except Exception as e:\n fails += 1\n print(\"Fail {}, while parsing {}: {}\".format(fails, path, e))\n continue\n\n moves = len(positions)\n indexes = random.sample(range(10, moves), FLAGS.num_positions)\n for index in sorted(indexes):\n collection.write('{}, {}\\n'.format(path, index))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef evaluate():\n\n def short_str(v):\n if isinstance(v, float):\n return \"{.3f}\".format(v)\n return str(v)\n\n # Load positons\n sgf_names, all_positions = get_final_positions()\n\n # Run and save some data about each position\n # Save to csv because that's easy\n model_paths = oneoff_utils.get_model_paths(FLAGS.model_dir)\n num_models = len(model_paths)\n print(\"Evaluating {} models: {} to {}\".format(\n num_models, model_paths[0], model_paths[-1]))\n print()\n\n with open(FLAGS.results, \"w\") as results:\n results.write(\",\".join(sgf_names) + \"\\n\")\n\n player = None\n for idx in tqdm(range(FLAGS.min_idx, num_models, 1), desc=\"model\"):\n model = model_paths[idx]\n\n if player and idx % 50 == 0:\n player.network.sess.close()\n tf.reset_default_graph()\n player = None\n\n if player:\n oneoff_utils.restore_params(model, player)\n else:\n player = oneoff_utils.load_player(model)\n\n row = [model]\n for positions in grouper(FLAGS.batch_size, all_positions):\n probs, values = player.network.run_many(positions)\n # NOTE(sethtroisi): For now we store the top n moves to shrink\n # the size of the recorded data.\n\n top_n = FLAGS.top_n\n top_policy_move = np.fliplr(np.argsort(probs))[:,:top_n]\n top_policy_value = np.fliplr(np.sort(probs))[:,:top_n]\n\n # One position at a time\n for v, m, p in zip(values, top_policy_move, top_policy_value):\n row.append(v)\n row.extend(itertools.chain.from_iterable(zip(m, p)))\n\n if len(positions) > 10:\n average_seen = top_policy_value.sum() / len(positions)\n if average_seen < 0.3:\n print(\"\\t\", average_seen, top_policy_value.sum(axis=-1))\n\n results.write(\",\".join(map(short_str, row)) + \"\\n\")", "response": "Evaluate the policy and value for each position\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds a subset of problems that maximal explains rating.", "response": "def minimize():\n \"\"\"Find a subset of problems that maximal explains rating.\n\n Usage:\n python3 sharp_positions.py minimize \\\n --model_dir models --sgf_dir data/s\n --rating_json ratings.json --results results.csv\n \"\"\"\n ########################### HYPER PARAMETERS ###############################\n\n # Stop when r2 is this much worse than full set of positions\n r2_stopping_percent = 0.96\n # for this many iterations\n stopping_iterations = 5\n\n # Limit SVM to a smaller number of positions to speed up code.\n max_positions_fit = 300\n # Filter any position that \"contributes\" less than this percent of max.\n filter_contribution_percent = 0.3\n # Never filter more than this many positions in one iterations\n filter_limit = 25\n\n ########################### HYPER PARAMETERS ###############################\n\n # Load positons\n model_paths = oneoff_utils.get_model_paths(FLAGS.model_dir)\n num_models = len(model_paths)\n assert num_models > 0, FLAGS.model_dir\n\n # Load model ratings\n # wget https://cloudygo.com/v12-19x19/json/ratings.json\n ratings = json.load(open(FLAGS.rating_json))\n raw_ratings = {int(r[0]): float(r[1]) for r in ratings}\n\n model_ratings = []\n for model in model_paths:\n model_idx = get_model_idx(model)\n if model_idx < FLAGS.min_idx:\n continue\n\n model_ratings.append(raw_ratings[model_idx])\n model_ratings = np.array(model_ratings)\n\n assert 0 < len(model_ratings) <= num_models, len(model_ratings)\n num_models = len(model_ratings)\n\n sgf_names, all_positions = get_final_positions()\n # Trim off common path prefix.\n common_path = os.path.commonpath(sgf_names)\n sgf_names = [name[len(common_path) + 1:] for name in sgf_names]\n\n print(\"Considering {} positions, {} models\".format(\n len(all_positions), num_models))\n print()\n\n # Load model data\n top_n = FLAGS.top_n\n positions = defaultdict(list)\n with open(FLAGS.results) as results:\n headers = results.readline().strip()\n assert headers.count(\",\") + 1 == len(sgf_names)\n\n # Row is + positions x [value, top_n x [move, move_policy]]\n for row in tqdm(results.readlines(), desc=\"result line\"):\n data = row.split(\",\")\n model_idx = get_model_idx(data.pop(0))\n if model_idx < FLAGS.min_idx:\n continue\n\n data_per = 1 + top_n * 2\n assert len(data) % data_per == 0, len(data)\n\n for position, position_data in enumerate(grouper(data_per, data)):\n value = float(position_data.pop(0))\n moves = list(map(int, position_data[0::2]))\n move_policy = list(map(float, position_data[1::2]))\n\n positions[position].append([value, moves, move_policy])\n\n def one_hot(n, i):\n one_hot = [0] * n\n if 0 <= i < n:\n one_hot[i] += 1\n return one_hot\n\n # NOTE: top_n isn't the same semantic value here and can be increased.\n one_hot_moves = top_n\n num_features = 1 + 5 + (one_hot_moves + 1)\n\n # Features by position\n features = []\n pos_top_moves = []\n for position, data in tqdm(positions.items(), desc=\"featurize\"):\n assert len(data) == num_models, len(data)\n\n top_moves = Counter([d[1][0] for d in data])\n top_n_moves = [m for m, c in top_moves.most_common(one_hot_moves)]\n if len(top_n_moves) < one_hot_moves:\n top_n_moves.extend([-1] * (one_hot_moves - len(top_n_moves)))\n assert len(top_n_moves) == one_hot_moves, \"pad with dummy moves\"\n pos_top_moves.append(top_n_moves)\n\n # Eventaully we want\n # [model 1 position 1 features, m1 p2 features, m1 p3 features, ... ]\n # [model 2 position 1 features, m2 p2 features, m2 p3 features, ... ]\n # [model 3 position 1 features, m3 p2 features, m3 p3 features, ... ]\n # ...\n # [model m position 1 features, mm p2 features, mm p3 features, ... ]\n\n # We'll do position selection by joining [model x position_feature]\n\n feature_columns = []\n for model, (v, m, mv) in enumerate(data):\n # Featurization (for each positions):\n # * Value (-1 to 1), Bucketed value\n # * Cluster all model by top_n moves (X,Y,Z or other)?\n # * value of that move for model\n # * policy value of top move\n model_features = []\n\n model_features.append(2 * v - 1)\n # NOTE(sethtroisi): Consider bucketize value by value percentiles.\n value_bucket = np.searchsorted((0.2, 0.4, 0.6, 0.8), v)\n model_features.extend(one_hot(5, value_bucket))\n\n # Policy weight for most common X moves (among all models).\n policy_weights = [0] * (one_hot_moves + 1)\n for move, policy_value in zip(m, mv):\n if move in top_n_moves:\n policy_weights[top_n_moves.index(move)] = policy_value\n else:\n policy_weights[-1] += policy_value\n model_features.extend(policy_weights)\n\n assert len(model_features) == num_features\n\n feature_columns.append(model_features)\n features.append(feature_columns)\n\n features = np.array(features)\n print(\"Feature shape\", features.shape)\n print()\n\n # Split the models to test / train\n train_size = int(num_models * 0.9)\n train_models = sorted(np.random.permutation(num_models)[:train_size])\n test_models = sorted(set(range(num_models)) - set(train_models))\n assert set(train_models + test_models) == set(range(num_models))\n features_train = features[:, train_models, :]\n features_test = features[:, test_models, :]\n\n labels_train = model_ratings[train_models]\n labels_test = model_ratings[test_models]\n\n # Choose some set of positions and see how well they explain ratings\n positions_to_use = set(positions.keys())\n linearSVM = svm.LinearSVR()\n best_test_r2 = 0\n below_threshold = 0\n\n for iteration in itertools.count(1):\n iter_positions = np.random.permutation(list(positions_to_use))\n iter_positions = sorted(iter_positions[:max_positions_fit])\n\n # Take this set of positions and build X\n X = np.concatenate(features_train[iter_positions], axis=1)\n Xtest = np.concatenate(features_test[iter_positions], axis=1)\n assert X.shape == (train_size, num_features * len(iter_positions))\n\n linearSVM.fit(X, labels_train)\n\n score_train = linearSVM.score(X, labels_train)\n score_test = linearSVM.score(Xtest, labels_test)\n print(\"iter {}, {}/{} included, R^2: {:.4f} train, {:.3f} test\".format(\n iteration, len(iter_positions), len(positions_to_use),\n score_train, score_test))\n\n # Determine the most and least useful position:\n # TODO(amj,brilee): Validate this math.\n assert len(linearSVM.coef_) == num_features * len(iter_positions)\n\n # The intercepts tell us how much this contributes to overall rating\n # but coef tell us how much different answers differentiate rating.\n coef_groups = list(grouper(num_features, linearSVM.coef_))\n position_coefs = [abs(sum(c)) for c in coef_groups]\n\n pos_value_idx = np.argsort(position_coefs)\n max_pos = pos_value_idx[-1]\n most_value = position_coefs[max_pos]\n\n print(\"\\tMost value {} => {:.1f} {}\".format(\n max_pos, most_value, sgf_names[iter_positions[max_pos]]))\n\n # Drop any positions that aren't very useful\n for dropped, pos_idx in enumerate(pos_value_idx[:filter_limit], 1):\n contribution = position_coefs[pos_idx]\n positions_to_use.remove(iter_positions[pos_idx])\n print(\"\\t\\tdropping({}): {:.1f} {}\".format(\n dropped, contribution, sgf_names[iter_positions[pos_idx]]))\n\n if contribution > filter_contribution_percent * most_value:\n break\n print()\n\n best_test_r2 = max(best_test_r2, score_test)\n if score_test > r2_stopping_percent * best_test_r2:\n below_threshold = 0\n else:\n below_threshold += 1\n if below_threshold == stopping_iterations:\n print(\"{}% decrease in R^2, stopping\".format(\n 100 - int(100 * r2_stopping_percent)))\n break\n\n # Write down the differentiating positions and their answers.\n svm_data = []\n for position_idx in list(reversed(pos_value_idx)):\n coefs = coef_groups[position_idx]\n\n # Global position index.\n position = iter_positions[position_idx]\n sgf_name = sgf_names[position]\n top_moves = pos_top_moves[position]\n\n svm_data.append([sgf_name, [top_moves, coefs.tolist()]])\n\n with open(FLAGS.SVM_json, \"w\") as svm_json:\n json.dump(svm_data, svm_json)\n print(\"Dumped data about {} positions to {}\".format(\n len(svm_data), FLAGS.SVM_json))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndownloading content from a url.", "response": "def download_from_url(path, url):\n \"\"\"Download content from a url.\n\n Args:\n path: string directory where file will be downloaded\n url: string url\n\n Returns:\n Full path to downloaded file\n \"\"\"\n filename = url.split(\"/\")[-1]\n found_file = find_file(path, filename, max_depth=0)\n if found_file is None:\n filename = os.path.join(path, filename)\n tf.logging.info(\"Downloading from %s to %s.\" % (url, filename))\n inprogress_filepath = filename + \".incomplete\"\n inprogress_filepath, _ = urllib.request.urlretrieve(\n url, inprogress_filepath, reporthook=download_report_hook)\n # Print newline to clear the carriage return from the download progress.\n print()\n tf.gfile.Rename(inprogress_filepath, filename)\n return filename\n else:\n tf.logging.info(\"Already downloaded: %s (at %s).\" % (url, found_file))\n return found_file"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive segmentation masks and the bounding boxes corresponding to the location of the masks in the image, this function crops and resizes the masks in the position defined by the boxes. This prepares the masks for them to be fed to the loss computation as the targets. Arguments: segmentation_masks: an instance of SegmentationMask proposals: an instance of BoxList", "response": "def project_masks_on_boxes(segmentation_masks, proposals, discretization_size):\n \"\"\"\n Given segmentation masks and the bounding boxes corresponding\n to the location of the masks in the image, this function\n crops and resizes the masks in the position defined by the\n boxes. This prepares the masks for them to be fed to the\n loss computation as the targets.\n\n Arguments:\n segmentation_masks: an instance of SegmentationMask\n proposals: an instance of BoxList\n \"\"\"\n masks = []\n M = discretization_size\n device = proposals.bbox.device\n proposals = proposals.convert(\"xyxy\")\n assert segmentation_masks.size == proposals.size, \"{}, {}\".format(\n segmentation_masks, proposals\n )\n # TODO put the proposals on the CPU, as the representation for the\n # masks is not efficient GPU-wise (possibly several small tensors for\n # representing a single instance mask)\n proposals = proposals.bbox.to(torch.device(\"cpu\"))\n for segmentation_mask, proposal in zip(segmentation_masks, proposals):\n # crop the masks, resize them to the desired resolution and\n # then convert them to the tensor representation,\n # instead of the list representation that was used\n cropped_mask = segmentation_mask.crop(proposal)\n scaled_mask = cropped_mask.resize((M, M))\n mask = scaled_mask.convert(mode=\"mask\")\n masks.append(mask)\n if len(masks) == 0:\n return torch.empty(0, dtype=torch.float32, device=device)\n return torch.stack(masks, dim=0).to(device, dtype=torch.float32)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nevaluate detection proposal recall metrics.", "response": "def evaluate_box_proposals(\n predictions, dataset, thresholds=None, area=\"all\", limit=None\n):\n \"\"\"Evaluate detection proposal recall metrics. This function is a much\n faster alternative to the official COCO API recall evaluation code. However,\n it produces slightly different results.\n \"\"\"\n # Record max overlap value for each gt box\n # Return vector of overlap values\n areas = {\n \"all\": 0,\n \"small\": 1,\n \"medium\": 2,\n \"large\": 3,\n \"96-128\": 4,\n \"128-256\": 5,\n \"256-512\": 6,\n \"512-inf\": 7,\n }\n area_ranges = [\n [0 ** 2, 1e5 ** 2], # all\n [0 ** 2, 32 ** 2], # small\n [32 ** 2, 96 ** 2], # medium\n [96 ** 2, 1e5 ** 2], # large\n [96 ** 2, 128 ** 2], # 96-128\n [128 ** 2, 256 ** 2], # 128-256\n [256 ** 2, 512 ** 2], # 256-512\n [512 ** 2, 1e5 ** 2],\n ] # 512-inf\n assert area in areas, \"Unknown area range: {}\".format(area)\n area_range = area_ranges[areas[area]]\n gt_overlaps = []\n num_pos = 0\n\n for image_id, prediction in enumerate(predictions):\n original_id = dataset.id_to_img_map[image_id]\n\n img_info = dataset.get_img_info(image_id)\n image_width = img_info[\"width\"]\n image_height = img_info[\"height\"]\n prediction = prediction.resize((image_width, image_height))\n\n # sort predictions in descending order\n # TODO maybe remove this and make it explicit in the documentation\n inds = prediction.get_field(\"objectness\").sort(descending=True)[1]\n prediction = prediction[inds]\n\n ann_ids = dataset.coco.getAnnIds(imgIds=original_id)\n anno = dataset.coco.loadAnns(ann_ids)\n gt_boxes = [obj[\"bbox\"] for obj in anno if obj[\"iscrowd\"] == 0]\n gt_boxes = torch.as_tensor(gt_boxes).reshape(-1, 4) # guard against no boxes\n gt_boxes = BoxList(gt_boxes, (image_width, image_height), mode=\"xywh\").convert(\n \"xyxy\"\n )\n gt_areas = torch.as_tensor([obj[\"area\"] for obj in anno if obj[\"iscrowd\"] == 0])\n\n if len(gt_boxes) == 0:\n continue\n\n valid_gt_inds = (gt_areas >= area_range[0]) & (gt_areas <= area_range[1])\n gt_boxes = gt_boxes[valid_gt_inds]\n\n num_pos += len(gt_boxes)\n\n if len(gt_boxes) == 0:\n continue\n\n if len(prediction) == 0:\n continue\n\n if limit is not None and len(prediction) > limit:\n prediction = prediction[:limit]\n\n overlaps = boxlist_iou(prediction, gt_boxes)\n\n _gt_overlaps = torch.zeros(len(gt_boxes))\n for j in range(min(len(prediction), len(gt_boxes))):\n # find which proposal box maximally covers each gt box\n # and get the iou amount of coverage for each gt box\n max_overlaps, argmax_overlaps = overlaps.max(dim=0)\n\n # find which gt box is 'best' covered (i.e. 'best' = most iou)\n gt_ovr, gt_ind = max_overlaps.max(dim=0)\n assert gt_ovr >= 0\n # find the proposal box that covers the best covered gt box\n box_ind = argmax_overlaps[gt_ind]\n # record the iou coverage of this gt box\n _gt_overlaps[j] = overlaps[box_ind, gt_ind]\n assert _gt_overlaps[j] == gt_ovr\n # mark the proposal box and the gt box as used\n overlaps[box_ind, :] = -1\n overlaps[:, gt_ind] = -1\n\n # append recorded iou coverage level\n gt_overlaps.append(_gt_overlaps)\n gt_overlaps = torch.cat(gt_overlaps, dim=0)\n gt_overlaps, _ = torch.sort(gt_overlaps)\n\n if thresholds is None:\n step = 0.05\n thresholds = torch.arange(0.5, 0.95 + 1e-5, step, dtype=torch.float32)\n recalls = torch.zeros_like(thresholds)\n # compute recall for each iou threshold\n for i, t in enumerate(thresholds):\n recalls[i] = (gt_overlaps >= t).float().sum() / float(num_pos)\n # ar = 2 * np.trapz(recalls, thresholds)\n ar = recalls.mean()\n return {\n \"ar\": ar,\n \"recalls\": recalls,\n \"thresholds\": thresholds,\n \"gt_overlaps\": gt_overlaps,\n \"num_pos\": num_pos,\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _decode_crop_and_flip(image_buffer, num_channels):\n # A large fraction of image datasets contain a human-annotated bounding box\n # delineating the region of the image containing the object of interest. We\n # choose to create a new bounding box for the object which is a randomly\n # distorted version of the human-annotated bounding box that obeys an\n # allowed range of aspect ratios, sizes and overlap with the human-annotated\n # bounding box. If no box is supplied, then we assume the bounding box is\n # the entire image.\n\n min_object_covered=0.1\n aspect_ratio_range=[0.75, 1.33]\n area_range=[0.05, 1.0]\n max_attempts=100\n\n mlperf_log.resnet_print(key=mlperf_log.INPUT_DISTORTED_CROP_MIN_OBJ_COV,\n value=min_object_covered)\n mlperf_log.resnet_print(key=mlperf_log.INPUT_DISTORTED_CROP_RATIO_RANGE,\n value=aspect_ratio_range)\n mlperf_log.resnet_print(key=mlperf_log.INPUT_DISTORTED_CROP_AREA_RANGE,\n value=area_range)\n mlperf_log.resnet_print(key=mlperf_log.INPUT_DISTORTED_CROP_MAX_ATTEMPTS,\n value=max_attempts)\n mlperf_log.resnet_print(key=mlperf_log.INPUT_CROP_USES_BBOXES, value=False)\n\n bbox = tf.constant([0.0, 0.0, 1.0, 1.0],\n dtype=tf.float32, shape=[1, 1, 4]) #From the entire image\n sample_distorted_bounding_box = tf.image.sample_distorted_bounding_box(\n tf.image.extract_jpeg_shape(image_buffer),\n bounding_boxes=bbox,\n min_object_covered=min_object_covered,\n aspect_ratio_range=aspect_ratio_range,\n area_range=area_range,\n max_attempts=max_attempts,\n use_image_if_no_bounding_boxes=True)\n bbox_begin, bbox_size, _ = sample_distorted_bounding_box\n\n # Reassemble the bounding box in the format the crop op requires.\n offset_y, offset_x, _ = tf.unstack(bbox_begin)\n target_height, target_width, _ = tf.unstack(bbox_size)\n crop_window = tf.stack([offset_y, offset_x, target_height, target_width])\n\n # Use the fused decode and crop op here, which is faster than each in series.\n cropped = tf.image.decode_and_crop_jpeg(\n image_buffer, crop_window, channels=num_channels)\n\n # Flip to add a little more random distortion in.\n mlperf_log.resnet_print(key=mlperf_log.INPUT_RANDOM_FLIP)\n cropped = tf.image.random_flip_left_right(cropped)\n return cropped", "response": "This function is used to decode a given image buffer and crop it to a random part of the image and randomly flips it."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nperforms central crop of the given image list.", "response": "def _central_crop(image, crop_height, crop_width):\n \"\"\"Performs central crops of the given image list.\n\n Args:\n image: a 3-D image tensor\n crop_height: the height of the image following the crop.\n crop_width: the width of the image following the crop.\n\n Returns:\n 3-D tensor with cropped image.\n \"\"\"\n shape = tf.shape(image)\n height, width = shape[0], shape[1]\n\n mlperf_log.resnet_print(key=mlperf_log.INPUT_CENTRAL_CROP,\n value=[crop_height, crop_width])\n\n amount_to_be_cropped_h = (height - crop_height)\n crop_top = amount_to_be_cropped_h // 2\n amount_to_be_cropped_w = (width - crop_width)\n crop_left = amount_to_be_cropped_w // 2\n return tf.slice(\n image, [crop_top, crop_left, 0], [crop_height, crop_width, -1])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _mean_image_subtraction(image, means, num_channels):\n if image.get_shape().ndims != 3:\n raise ValueError('Input must be of size [height, width, C>0]')\n\n if len(means) != num_channels:\n raise ValueError('len(means) must match the number of channels')\n\n mlperf_log.resnet_print(key=mlperf_log.INPUT_MEAN_SUBTRACTION,\n value=means)\n\n # We have a 1-D tensor of means; convert to 3-D.\n means = tf.expand_dims(tf.expand_dims(means, 0), 0)\n\n return image - means", "response": "Subtracts the given means from each image channel."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _smallest_size_at_least(height, width, resize_min):\n resize_min = tf.cast(resize_min, tf.float32)\n\n # Convert to floats to make subsequent calculations go smoothly.\n height, width = tf.cast(height, tf.float32), tf.cast(width, tf.float32)\n\n smaller_dim = tf.minimum(height, width)\n scale_ratio = resize_min / smaller_dim\n\n # Convert back to ints to make heights and widths that TF ops will accept.\n new_height = tf.cast(height * scale_ratio, tf.int32)\n new_width = tf.cast(width * scale_ratio, tf.int32)\n\n return new_height, new_width", "response": "Computes new shape with the smallest side equal to smallest_side."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _aspect_preserving_resize(image, resize_min):\n mlperf_log.resnet_print(key=mlperf_log.INPUT_RESIZE_ASPECT_PRESERVING,\n value={\"min\": resize_min})\n\n shape = tf.shape(image)\n height, width = shape[0], shape[1]\n\n new_height, new_width = _smallest_size_at_least(height, width, resize_min)\n\n return _resize_image(image, new_height, new_width)", "response": "Resize images preserving the original aspect ratio."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsearches for sequence of subtoken ids with the largest probability.", "response": "def sequence_beam_search(\n symbols_to_logits_fn, initial_ids, initial_cache, vocab_size, beam_size,\n alpha, max_decode_length, eos_id):\n \"\"\"Search for sequence of subtoken ids with the largest probability.\n\n Args:\n symbols_to_logits_fn: A function that takes in ids, index, and cache as\n arguments. The passed in arguments will have shape:\n ids -> [batch_size * beam_size, index]\n index -> [] (scalar)\n cache -> nested dictionary of tensors [batch_size * beam_size, ...]\n The function must return logits and new cache.\n logits -> [batch * beam_size, vocab_size]\n new cache -> same shape/structure as inputted cache\n initial_ids: Starting ids for each batch item.\n int32 tensor with shape [batch_size]\n initial_cache: dict containing starting decoder variables information\n vocab_size: int size of tokens\n beam_size: int number of beams\n alpha: float defining the strength of length normalization\n max_decode_length: maximum length to decoded sequence\n eos_id: int id of eos token, used to determine when a sequence has finished\n\n Returns:\n Top decoded sequences [batch_size, beam_size, max_decode_length]\n sequence scores [batch_size, beam_size]\n \"\"\"\n batch_size = tf.shape(initial_ids)[0]\n sbs = SequenceBeamSearch(symbols_to_logits_fn, vocab_size, batch_size,\n beam_size, alpha, max_decode_length, eos_id)\n return sbs.search(initial_ids, initial_cache)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of the tensor s shape and ensure no None values in list.", "response": "def _shape_list(tensor):\n \"\"\"Return a list of the tensor's shape, and ensure no None values in list.\"\"\"\n # Get statically known shape (may contain None's for unknown dimensions)\n shape = tensor.get_shape().as_list()\n\n # Ensure that the shape values are not None\n dynamic_shape = tf.shape(tensor)\n for i in range(len(shape)):\n if shape[i] is None:\n shape[i] = dynamic_shape[i]\n return shape"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _unflatten_beam_dim(tensor, batch_size, beam_size):\n shape = _shape_list(tensor)\n new_shape = [batch_size, beam_size] + shape[1:]\n return tf.reshape(tensor, new_shape)", "response": "Reshapes first dimension back to [ batch_size beam_size... )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _gather_beams(nested, beam_indices, batch_size, new_beam_size):\n # Computes the i'th coodinate that contains the batch index for gather_nd.\n # Batch pos is a tensor like [[0,0,0,0,],[1,1,1,1],..].\n batch_pos = tf.range(batch_size * new_beam_size) // new_beam_size\n batch_pos = tf.reshape(batch_pos, [batch_size, new_beam_size])\n\n # Create coordinates to be passed to tf.gather_nd. Stacking creates a tensor\n # with shape [batch_size, beam_size, 2], where the last dimension contains\n # the (i, j) gathering coordinates.\n coordinates = tf.stack([batch_pos, beam_indices], axis=2)\n\n return nest.map_structure(\n lambda state: tf.gather_nd(state, coordinates), nested)", "response": "This function is used to gather beams from nested structure of tensors."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngathers top k beams from nested structure.", "response": "def _gather_topk_beams(nested, score_or_log_prob, batch_size, beam_size):\n \"\"\"Gather top beams from nested structure.\"\"\"\n _, topk_indexes = tf.nn.top_k(score_or_log_prob, k=beam_size)\n return _gather_beams(nested, topk_indexes, batch_size, beam_size)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef search(self, initial_ids, initial_cache):\n state, state_shapes = self._create_initial_state(initial_ids, initial_cache)\n\n finished_state = tf.while_loop(\n self._continue_search, self._search_step, loop_vars=[state],\n shape_invariants=[state_shapes], parallel_iterations=1, back_prop=False)\n finished_state = finished_state[0]\n\n alive_seq = finished_state[_StateKeys.ALIVE_SEQ]\n alive_log_probs = finished_state[_StateKeys.ALIVE_LOG_PROBS]\n finished_seq = finished_state[_StateKeys.FINISHED_SEQ]\n finished_scores = finished_state[_StateKeys.FINISHED_SCORES]\n finished_flags = finished_state[_StateKeys.FINISHED_FLAGS]\n\n # Account for corner case where there are no finished sequences for a\n # particular batch item. In that case, return alive sequences for that batch\n # item.\n finished_seq = tf.where(\n tf.reduce_any(finished_flags, 1), finished_seq, alive_seq)\n finished_scores = tf.where(\n tf.reduce_any(finished_flags, 1), finished_scores, alive_log_probs)\n return finished_seq, finished_scores", "response": "Beam search for sequences with highest scores."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating the initial state dictionary and its shape invariants.", "response": "def _create_initial_state(self, initial_ids, initial_cache):\n \"\"\"Return initial state dictionary and its shape invariants.\n\n Args:\n initial_ids: initial ids to pass into the symbols_to_logits_fn.\n int tensor with shape [batch_size, 1]\n initial_cache: dictionary storing values to be passed into the\n symbols_to_logits_fn.\n\n Returns:\n state and shape invariant dictionaries with keys from _StateKeys\n \"\"\"\n # Current loop index (starts at 0)\n cur_index = tf.constant(0)\n\n # Create alive sequence with shape [batch_size, beam_size, 1]\n alive_seq = _expand_to_beam_size(initial_ids, self.beam_size)\n alive_seq = tf.expand_dims(alive_seq, axis=2)\n\n # Create tensor for storing initial log probabilities.\n # Assume initial_ids are prob 1.0\n initial_log_probs = tf.constant(\n [[0.] + [-float(\"inf\")] * (self.beam_size - 1)])\n alive_log_probs = tf.tile(initial_log_probs, [self.batch_size, 1])\n\n # Expand all values stored in the dictionary to the beam size, so that each\n # beam has a separate cache.\n alive_cache = nest.map_structure(\n lambda t: _expand_to_beam_size(t, self.beam_size), initial_cache)\n\n # Initialize tensor storing finished sequences with filler values.\n finished_seq = tf.zeros(tf.shape(alive_seq), tf.int32)\n\n # Set scores of the initial finished seqs to negative infinity.\n finished_scores = tf.ones([self.batch_size, self.beam_size]) * -INF\n\n # Initialize finished flags with all False values.\n finished_flags = tf.zeros([self.batch_size, self.beam_size], tf.bool)\n\n # Create state dictionary\n state = {\n _StateKeys.CUR_INDEX: cur_index,\n _StateKeys.ALIVE_SEQ: alive_seq,\n _StateKeys.ALIVE_LOG_PROBS: alive_log_probs,\n _StateKeys.ALIVE_CACHE: alive_cache,\n _StateKeys.FINISHED_SEQ: finished_seq,\n _StateKeys.FINISHED_SCORES: finished_scores,\n _StateKeys.FINISHED_FLAGS: finished_flags\n }\n\n # Create state invariants for each value in the state dictionary. Each\n # dimension must be a constant or None. A None dimension means either:\n # 1) the dimension's value is a tensor that remains the same but may\n # depend on the input sequence to the model (e.g. batch size).\n # 2) the dimension may have different values on different iterations.\n state_shape_invariants = {\n _StateKeys.CUR_INDEX: tf.TensorShape([]),\n _StateKeys.ALIVE_SEQ: tf.TensorShape([None, self.beam_size, None]),\n _StateKeys.ALIVE_LOG_PROBS: tf.TensorShape([None, self.beam_size]),\n _StateKeys.ALIVE_CACHE: nest.map_structure(\n _get_shape_keep_last_dim, alive_cache),\n _StateKeys.FINISHED_SEQ: tf.TensorShape([None, self.beam_size, None]),\n _StateKeys.FINISHED_SCORES: tf.TensorShape([None, self.beam_size]),\n _StateKeys.FINISHED_FLAGS: tf.TensorShape([None, self.beam_size])\n }\n\n return state, state_shape_invariants"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _continue_search(self, state):\n i = state[_StateKeys.CUR_INDEX]\n alive_log_probs = state[_StateKeys.ALIVE_LOG_PROBS]\n finished_scores = state[_StateKeys.FINISHED_SCORES]\n finished_flags = state[_StateKeys.FINISHED_FLAGS]\n\n not_at_max_decode_length = tf.less(i, self.max_decode_length)\n\n # Calculate largest length penalty (the larger penalty, the better score).\n max_length_norm = _length_normalization(self.alpha, self.max_decode_length)\n # Get the best possible scores from alive sequences.\n best_alive_scores = alive_log_probs[:, 0] / max_length_norm\n\n # Compute worst score in finished sequences for each batch element\n finished_scores *= tf.to_float(finished_flags) # set filler scores to zero\n lowest_finished_scores = tf.reduce_min(finished_scores, axis=1)\n\n # If there are no finished sequences in a batch element, then set the lowest\n # finished score to -INF for that element.\n finished_batches = tf.reduce_any(finished_flags, 1)\n lowest_finished_scores += (1. - tf.to_float(finished_batches)) * -INF\n\n worst_finished_score_better_than_best_alive_score = tf.reduce_all(\n tf.greater(lowest_finished_scores, best_alive_scores)\n )\n\n return tf.logical_and(\n not_at_max_decode_length,\n tf.logical_not(worst_finished_score_better_than_best_alive_score)\n )", "response": "Return whether to continue the search loop."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbeaming search loop body.", "response": "def _search_step(self, state):\n \"\"\"Beam search loop body.\n\n Grow alive sequences by a single ID. Sequences that have reached the EOS\n token are marked as finished. The alive and finished sequences with the\n highest log probabilities and scores are returned.\n\n A sequence's finished score is calculating by dividing the log probability\n by the length normalization factor. Without length normalization, the\n search is more likely to return shorter sequences.\n\n Args:\n state: A dictionary with the current loop state.\n\n Returns:\n new state dictionary.\n \"\"\"\n # Grow alive sequences by one token.\n new_seq, new_log_probs, new_cache = self._grow_alive_seq(state)\n # Collect top beam_size alive sequences\n alive_state = self._get_new_alive_state(new_seq, new_log_probs, new_cache)\n\n # Combine newly finished sequences with existing finished sequences, and\n # collect the top k scoring sequences.\n finished_state = self._get_new_finished_state(state, new_seq, new_log_probs)\n\n # Increment loop index and create new state dictionary\n new_state = {_StateKeys.CUR_INDEX: state[_StateKeys.CUR_INDEX] + 1}\n new_state.update(alive_state)\n new_state.update(finished_state)\n return [new_state]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngrow alive sequences by one token and collect top 2 * beam_size sequences.", "response": "def _grow_alive_seq(self, state):\n \"\"\"Grow alive sequences by one token, and collect top 2*beam_size sequences.\n\n 2*beam_size sequences are collected because some sequences may have reached\n the EOS token. 2*beam_size ensures that at least beam_size sequences are\n still alive.\n\n Args:\n state: A dictionary with the current loop state.\n Returns:\n Tuple of\n (Top 2*beam_size sequences [batch_size, 2 * beam_size, cur_index + 1],\n Scores of returned sequences [batch_size, 2 * beam_size],\n New alive cache, for each of the 2 * beam_size sequences)\n \"\"\"\n i = state[_StateKeys.CUR_INDEX]\n alive_seq = state[_StateKeys.ALIVE_SEQ]\n alive_log_probs = state[_StateKeys.ALIVE_LOG_PROBS]\n alive_cache = state[_StateKeys.ALIVE_CACHE]\n\n beams_to_keep = 2 * self.beam_size\n\n # Get logits for the next candidate IDs for the alive sequences. Get the new\n # cache values at the same time.\n flat_ids = _flatten_beam_dim(alive_seq) # [batch_size * beam_size]\n flat_cache = nest.map_structure(_flatten_beam_dim, alive_cache)\n\n flat_logits, flat_cache = self.symbols_to_logits_fn(flat_ids, i, flat_cache)\n\n # Unflatten logits to shape [batch_size, beam_size, vocab_size]\n logits = _unflatten_beam_dim(flat_logits, self.batch_size, self.beam_size)\n new_cache = nest.map_structure(\n lambda t: _unflatten_beam_dim(t, self.batch_size, self.beam_size),\n flat_cache)\n\n # Convert logits to normalized log probs\n candidate_log_probs = _log_prob_from_logits(logits)\n\n # Calculate new log probabilities if each of the alive sequences were\n # extended # by the the candidate IDs.\n # Shape [batch_size, beam_size, vocab_size]\n log_probs = candidate_log_probs + tf.expand_dims(alive_log_probs, axis=2)\n\n # Each batch item has beam_size * vocab_size candidate sequences. For each\n # batch item, get the k candidates with the highest log probabilities.\n flat_log_probs = tf.reshape(log_probs,\n [-1, self.beam_size * self.vocab_size])\n topk_log_probs, topk_indices = tf.nn.top_k(flat_log_probs, k=beams_to_keep)\n\n # Extract the alive sequences that generate the highest log probabilities\n # after being extended.\n topk_beam_indices = topk_indices // self.vocab_size\n topk_seq, new_cache = _gather_beams(\n [alive_seq, new_cache], topk_beam_indices, self.batch_size,\n beams_to_keep)\n\n # Append the most probable IDs to the topk sequences\n topk_ids = topk_indices % self.vocab_size\n topk_ids = tf.expand_dims(topk_ids, axis=2)\n topk_seq = tf.concat([topk_seq, topk_ids], axis=2)\n return topk_seq, topk_log_probs, new_cache"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_new_alive_state(self, new_seq, new_log_probs, new_cache):\n # To prevent finished sequences from being considered, set log probs to -INF\n new_finished_flags = tf.equal(new_seq[:, :, -1], self.eos_id)\n new_log_probs += tf.to_float(new_finished_flags) * -INF\n\n top_alive_seq, top_alive_log_probs, top_alive_cache = _gather_topk_beams(\n [new_seq, new_log_probs, new_cache], new_log_probs, self.batch_size,\n self.beam_size)\n\n return {\n _StateKeys.ALIVE_SEQ: top_alive_seq,\n _StateKeys.ALIVE_LOG_PROBS: top_alive_log_probs,\n _StateKeys.ALIVE_CACHE: top_alive_cache\n }", "response": "Gather the top k sequences that are still alive and return the new state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncombining new and old finished sequences and gather the top k sequences.", "response": "def _get_new_finished_state(self, state, new_seq, new_log_probs):\n \"\"\"Combine new and old finished sequences, and gather the top k sequences.\n\n Args:\n state: A dictionary with the current loop state.\n new_seq: New sequences generated by growing the current alive sequences\n int32 tensor with shape [batch_size, beam_size, i + 1]\n new_log_probs: Log probabilities of new sequences\n float32 tensor with shape [batch_size, beam_size]\n\n Returns:\n Dictionary with finished keys from _StateKeys:\n {Top beam_size finished sequences based on score,\n Scores of finished sequences,\n Finished flags of finished sequences}\n \"\"\"\n i = state[_StateKeys.CUR_INDEX]\n finished_seq = state[_StateKeys.FINISHED_SEQ]\n finished_scores = state[_StateKeys.FINISHED_SCORES]\n finished_flags = state[_StateKeys.FINISHED_FLAGS]\n\n # First append a column of 0-ids to finished_seq to increment the length.\n # New shape of finished_seq: [batch_size, beam_size, i + 1]\n finished_seq = tf.concat(\n [finished_seq,\n tf.zeros([self.batch_size, self.beam_size, 1], tf.int32)], axis=2)\n\n # Calculate new seq scores from log probabilities.\n length_norm = _length_normalization(self.alpha, i + 1)\n new_scores = new_log_probs / length_norm\n\n # Set the scores of the still-alive seq in new_seq to large negative values.\n new_finished_flags = tf.equal(new_seq[:, :, -1], self.eos_id)\n new_scores += (1. - tf.to_float(new_finished_flags)) * -INF\n\n # Combine sequences, scores, and flags.\n finished_seq = tf.concat([finished_seq, new_seq], axis=1)\n finished_scores = tf.concat([finished_scores, new_scores], axis=1)\n finished_flags = tf.concat([finished_flags, new_finished_flags], axis=1)\n\n # Return the finished sequences with the best scores.\n top_finished_seq, top_finished_scores, top_finished_flags = (\n _gather_topk_beams([finished_seq, finished_scores, finished_flags],\n finished_scores, self.batch_size, self.beam_size))\n\n return {\n _StateKeys.FINISHED_SEQ: top_finished_seq,\n _StateKeys.FINISHED_SCORES: top_finished_scores,\n _StateKeys.FINISHED_FLAGS: top_finished_flags\n }"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset initial random values for trainable parameters.", "response": "def reset_parameters(self, init_weight):\n \"\"\"\n Sets initial random values for trainable parameters.\n \"\"\"\n stdv = 1. / math.sqrt(self.num_units)\n self.linear_att.data.uniform_(-init_weight, init_weight)\n\n if self.normalize:\n self.normalize_scalar.data.fill_(stdv)\n self.normalize_bias.data.zero_()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_mask(self, context_len, context):\n\n if self.batch_first:\n max_len = context.size(1)\n else:\n max_len = context.size(0)\n\n indices = torch.arange(0, max_len, dtype=torch.int64,\n device=context.device)\n self.mask = indices >= (context_len.unsqueeze(1))", "response": "Sets self. mask which is applied before softmax\n ones for inactive context fields zeros for active context fields"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates Bahdanau score of a set of attributes.", "response": "def calc_score(self, att_query, att_keys):\n \"\"\"\n Calculate Bahdanau score\n\n :param att_query: b x t_q x n\n :param att_keys: b x t_k x n\n\n returns: b x t_q x t_k scores\n \"\"\"\n\n b, t_k, n = att_keys.size()\n t_q = att_query.size(1)\n\n att_query = att_query.unsqueeze(2).expand(b, t_q, t_k, n)\n att_keys = att_keys.unsqueeze(1).expand(b, t_q, t_k, n)\n sum_qk = att_query + att_keys\n\n if self.normalize:\n sum_qk = sum_qk + self.normalize_bias\n linear_att = self.linear_att / self.linear_att.norm()\n linear_att = linear_att * self.normalize_scalar\n else:\n linear_att = self.linear_att\n\n out = torch.tanh(sum_qk).matmul(linear_att)\n return out"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the positional encoding.", "response": "def get_position_encoding(\n length, hidden_size, min_timescale=1.0, max_timescale=1.0e4):\n \"\"\"Return positional encoding.\n\n Calculates the position encoding as a mix of sine and cosine functions with\n geometrically increasing wavelengths.\n Defined and formulized in Attention is All You Need, section 3.5.\n\n Args:\n length: Sequence length.\n hidden_size: Size of the\n min_timescale: Minimum scale that will be applied at each position\n max_timescale: Maximum scale that will be applied at each position\n\n Returns:\n Tensor with shape [length, hidden_size]\n \"\"\"\n position = tf.to_float(tf.range(length))\n num_timescales = hidden_size // 2\n log_timescale_increment = (\n math.log(float(max_timescale) / float(min_timescale)) /\n (tf.to_float(num_timescales) - 1))\n inv_timescales = min_timescale * tf.exp(\n tf.to_float(tf.range(num_timescales)) * -log_timescale_increment)\n scaled_time = tf.expand_dims(position, 1) * tf.expand_dims(inv_timescales, 0)\n signal = tf.concat([tf.sin(scaled_time), tf.cos(scaled_time)], axis=1)\n return signal"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_decoder_self_attention_bias(length):\n with tf.name_scope(\"decoder_self_attention_bias\"):\n valid_locs = tf.matrix_band_part(tf.ones([length, length]), -1, 0)\n valid_locs = tf.reshape(valid_locs, [1, 1, length, length])\n decoder_bias = _NEG_INF * (1.0 - valid_locs)\n return decoder_bias", "response": "Calculate bias for decoder that maintains model s autoregressive property."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_padding(x, padding_value=0):\n with tf.name_scope(\"padding\"):\n return tf.to_float(tf.equal(x, padding_value))", "response": "Returns float tensor representing the padding values in x."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_padding_bias(x):\n with tf.name_scope(\"attention_bias\"):\n padding = get_padding(x)\n attention_bias = padding * _NEG_INF\n attention_bias = tf.expand_dims(\n tf.expand_dims(attention_bias, axis=1), axis=1)\n return attention_bias", "response": "Calculate bias tensor from padding values in tensor."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a result to the dictionary.", "response": "def _add_result(self, dict_entry, entry, dt, start_time):\n \"\"\"Adds a result to the dictionary.\n\n Args:\n dict_entry: main dict to add entry\n entry: slot for this entry (likely an integer)\n dt: the timing for the entry\n start_time: when the entry started unix time float\n \"\"\"\n time_entry = {}\n time_entry['dt'] = dt\n time_entry['start_time'] = start_time\n dict_entry[entry] = time_entry"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsort the results and returns an array with only the values but sorted oldest first.", "response": "def _sorted_results(self, results_dicts):\n \"\"\"Sorts dict of results based on log start_time.\n\n Sorts the results and returns an array with only the values but sorted\n by oldest value first.value\n\n Args:\n results_dicts: List of result dicts\n\n Returns:\n List of only the time but sorted oldest first.\n \"\"\"\n print('results dicts:', results_dicts)\n sorted_dict = sorted(results_dicts, key=lambda k: k['start_time'])\n results = []\n for entry in sorted_dict:\n results.append(entry['dt'])\n return results"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_compliance(self, filename):\n print('Running Compliance Check on {}'.format(filename))\n print('#' * 80)\n start_time, status, dt, qual, target = mlp_compliance.l2_check_file_w_starttime(\n filename)\n print('#' * 80)\n\n if status:\n level = '2'\n else:\n start_time, status, dt, qual, target = mlp_compliance.l1_check_file_w_starttime(\n filename)\n print('#' * 80)\n if status:\n level = '1'\n else:\n level = '0'\n\n success = status and qual and target and qual >= target\n return start_time, level, dt, qual, success", "response": "Get the compliance level of the output file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nverifying and result and returns timing.", "response": "def verify_and_extract_time(self, log_file, division, result_name):\n \"\"\"Verifies and result and returns timing.\n\n Uses submodule mlp_compliance (https://github.com/bitfort/mlp_compliance)\n\n Args:\n log_file: Absolute path to result file.\n division: open, closed\n result_name: name of the benchmark, ncf, ssd, etc\n\n Returns:\n Time for the result or `INFINITE_TIME` if not a success\n\n Raises:\n Exception: If expected compliance level is not hit or cannot figure\n out expected compliance level.\n\n \"\"\"\n expected_level = constants.DIVISION_COMPLIANCE_CHECK_LEVEL.get(\n division, None)\n print(result_name)\n if expected_level is None:\n raise Exception('Unknown division: {}'.format(division))\n start_time, level, dt, _, success = self.get_compliance(log_file)\n print(float(start_time))\n if int(level) != expected_level:\n raise Exception('Error Level {} does not match needed level {}:{}'.format(\n level, expected_level, log_file))\n\n # Sets failure to converge to \"infinite time\" per the rules\n if success and dt:\n return dt, start_time\n else:\n print('Result was not a success set to INFINITE_TIME({})'.format(\n INFINITE_TIME))\n return INFINITE_TIME, start_time"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _compute_and_write_row_block(\n i, left_matrix, right_matrix, train_indices_out_path, test_indices_out_path,\n remove_empty_rows):\n \"\"\"Compute row block (shard) of expansion for row i of the left_matrix.\n\n Compute a shard of the randomized Kronecker product and dump it on the fly.\n A standard Kronecker product between matrices A and B produces\n [[a_11 B, ..., a_1n B],\n ...\n [a_m1 B, ..., a_mn B]]\n (if A's size is (m, n) and B's size is (p, q) then A Kronecker B has size\n (m p, n q)).\n Here we modify the standard Kronecker product expanding matrices in\n https://cs.stanford.edu/~jure/pubs/kronecker-jmlr10.pdf\n and randomize each block-wise operation a_ij B in the Kronecker product as\n in https://arxiv.org/pdf/1901.08910.pdf section III.4.\n The matrix we produce is\n [[F(a_11, B, w_11), ..., F(a_1n, B, w_1n)],\n ...\n [F(a_m1, B, w_m1), ... , F(a_mn, B, w_mn)]]\n where (w_ij) is a sequence of pseudo random numbers and F is randomized\n operator which will:\n 1) Shuffle rows and columns of B independently at random;\n 2) Dropout elements of B with a rate 1 - a_ij to compute\n F(a_ij, B, w_ij).\n (It is noteworthy that there is an abuse of notation above when writing\n F(a_ij, B, w_ij) as each block-wise operation will in fact consume\n multiple elements of the sequence (w_ij)).\n Each shard of index i consists of [F(a_i1, B, w_i1), ..., F(a_in, B, w_in)]\n\n Args:\n i: index of the shard. The rows i * m to (i + 1) * m of the full synthetic\n matrix matrix will be computed and dumpted to file.\n left_matrix: sparse SciPy csr matrix with values in [0, 1].\n right_matrix: sparse SciPy coo signed binary matrix. +1 values correspond\n to train set and -1 values correspond to test set.\n train_indices_out_path: path to output train file. The non zero indices of\n the resulting sparse matrix are dumped as a series of pickled records.\n '_i' will be used as a suffix for the shard's output file. The shard\n contains a pickled list of list each of which corresponds to a users.\n test_indices_out_path: path to output train file. The non zero indices of\n the resulting sparse matrix are dumped as a series of pickled records.\n '_i' will be used as a suffix for the shard's output file. The shard\n contains a pickled list of list each of which corresponds to a users.\n remove_empty_rows: whether to remove rows from the synthetic train and\n test matrices which are not present in the train or the test matrix.\n\n Returns:\n (num_removed_rows, metadata, train_metadata, test_metadata): an integer\n specifying the number of rows dropped because of dropout followed by\n a triplet of SparseMatrixMetadata corresponding to the overall shard,\n train shard and test shard.\n \"\"\"\n\n kron_blocks = []\n\n num_rows = 0\n num_removed_rows = 0\n num_interactions = 0\n num_train_interactions = 0\n num_test_interactions = 0\n\n # Construct blocks\n for j in xrange(left_matrix.shape[1]):\n\n dropout_rate = 1.0 - left_matrix[i, j]\n kron_block = shuffle_sparse_coo_matrix(right_matrix, dropout_rate)\n\n if not set(kron_block.data).issubset({1, -1}):\n raise ValueError(\"Values of sparse matrix should be -1 or 1 but are: \",\n set(kron_block.data))\n\n kron_blocks.append(kron_block)\n\n logging.info(\"Done with element (%d, %d)\", i, j)\n\n rows_to_write = sparse.hstack(kron_blocks).tocoo()\n\n train_rows_to_write = util.sparse_where_equal(rows_to_write, 1)\n test_rows_to_write = util.sparse_where_equal(rows_to_write, -1)\n\n logging.info(\"Producing data set row by row\")\n\n all_train_items_to_write = []\n all_test_items_to_write = []\n # Write Kronecker product line per line.\n for k in xrange(right_matrix.shape[0]):\n\n train_items_to_write = train_rows_to_write.getrow(k).indices\n test_items_to_write = test_rows_to_write.getrow(k).indices\n\n # for users with > 1 test items, keep only the first one\n if len(test_items_to_write) > 1:\n test_items_to_write = test_items_to_write[:1]\n\n num_train = train_items_to_write.shape[0]\n num_test = test_items_to_write.shape[0]\n\n if remove_empty_rows and ((not num_train) or (not num_test)):\n logging.info(\"Removed empty output row %d.\",\n i * left_matrix.shape[0] + k)\n num_removed_rows += 1\n continue\n\n num_rows += 1\n num_interactions += num_train + num_test\n num_train_interactions += num_train\n num_test_interactions += num_test\n\n all_train_items_to_write.append(train_items_to_write)\n all_test_items_to_write.append(test_items_to_write)\n\n if k % 1000 == 0:\n logging.info(\"Done producing data set row %d.\", k)\n\n logging.info(\"Done producing data set row by row.\")\n\n util.savez_two_column(\n all_train_items_to_write, \n row_offset=(i * right_matrix.shape[0]),\n file_name=train_indices_out_path + (\"_%d\" % i))\n util.savez_two_column(\n all_test_items_to_write, \n row_offset=(i * right_matrix.shape[0]),\n file_name=test_indices_out_path + (\"_%d\" % i))\n\n num_cols = rows_to_write.shape[1]\n metadata = SparseMatrixMetadata(num_interactions=num_interactions,\n num_rows=num_rows, num_cols=num_cols)\n train_metadata = SparseMatrixMetadata(num_interactions=num_train_interactions,\n num_rows=num_rows, num_cols=num_cols)\n test_metadata = SparseMatrixMetadata(num_interactions=num_test_interactions,\n num_rows=num_rows, num_cols=num_cols)\n\n logging.info(\"Done with left matrix row %d.\", i)\n logging.info(\"%d interactions written in shard.\", num_interactions)\n logging.info(\"%d rows removed in shard.\", num_removed_rows)\n logging.info(\"%d train interactions written in shard.\",\n num_train_interactions)\n logging.info(\"%d test interactions written in shard.\",\n num_test_interactions)\n\n return (num_removed_rows, metadata, train_metadata, test_metadata)", "response": "Compute and write the row block for expansion for row i of the right_matrix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute randomized Kronecker product and dump it on the fly. A standard Kronecker product between matrices A and B produces [[a_11 B, ..., a_1n B], ... [a_m1 B, ..., a_mn B]] (if A's size is (m, n) and B's size is (p, q) then A Kronecker B has size (m p, n q)). Here we modify the standard Kronecker product expanding matrices in https://cs.stanford.edu/~jure/pubs/kronecker-jmlr10.pdf and randomize each block-wise operation a_ij B in the Kronecker product as in https://arxiv.org/pdf/1901.08910.pdf section III.4. The matrix we produce is [[F(a_11, B, w_11), ..., F(a_1n, B, w_1n)], ... [F(a_m1, B, w_m1), ... , F(a_mn, B, w_mn)]] where (w_ij) is a sequence of pseudo random numbers and F is randomized operator which will: 1) Shuffle rows and columns of B independently at random; 2) Dropout elements of B with a rate 1 - a_ij to compute F(a_ij, B, w_ij). (It is noteworthy that there is an abuse of notation above when writing F(a_ij, B, w_ij) as each block-wise operation will in fact consume multiple elements of the sequence (w_ij)). Args: left_matrix: sparse SciPy csr matrix with values in [0, 1]. right_matrix: sparse SciPy coo signed binary matrix. +1 values correspond to train set and -1 values correspond to test set. train_indices_out_path: path to output train file. The non zero indices of the resulting sparse matrix are dumped as a series of pickled records. As many shard will be created as there are rows in left matrix. The shard corresponding to row i in the left matrix has the suffix _i appended to its file name. Each shard contains a pickled list of list each of which corresponds to a users. test_indices_out_path: path to output train file. The non zero indices of the resulting sparse matrix are dumped as a series of pickled records. As many shard will be created as there are rows in left matrix. The shard corresponding to row i in the left matrix has the suffix _i appended to its file name. Each shard contains a pickled list of list each of which corresponds to a users. train_metadata_out_path: path to optional complementary output file containing the number of train rows (r), columns (c) and non zeros (nnz) in a pickled SparseMatrixMetadata named tuple. test_metadata_out_path: path to optional complementary output file containing the number of test rows (r), columns (c) and non zeros (nnz) in a pickled SparseMatrixMetadata named tuple. remove_empty_rows: whether to remove rows from the synthetic train and test matrices which are not present in the train or the test matrix. Returns: (metadata, train_metadata, test_metadata) triplet of SparseMatrixMetadata corresponding to the overall data set, train data set and test data set.", "response": "def output_randomized_kronecker_to_pickle(\n left_matrix, right_matrix,\n train_indices_out_path, test_indices_out_path,\n train_metadata_out_path=None, test_metadata_out_path=None,\n remove_empty_rows=True):\n \"\"\"Compute randomized Kronecker product and dump it on the fly.\n\n A standard Kronecker product between matrices A and B produces\n [[a_11 B, ..., a_1n B],\n ...\n [a_m1 B, ..., a_mn B]]\n (if A's size is (m, n) and B's size is (p, q) then A Kronecker B has size\n (m p, n q)).\n Here we modify the standard Kronecker product expanding matrices in\n https://cs.stanford.edu/~jure/pubs/kronecker-jmlr10.pdf\n and randomize each block-wise operation a_ij B in the Kronecker product as\n in https://arxiv.org/pdf/1901.08910.pdf section III.4.\n The matrix we produce is\n [[F(a_11, B, w_11), ..., F(a_1n, B, w_1n)],\n ...\n [F(a_m1, B, w_m1), ... , F(a_mn, B, w_mn)]]\n where (w_ij) is a sequence of pseudo random numbers and F is randomized\n operator which will:\n 1) Shuffle rows and columns of B independently at random;\n 2) Dropout elements of B with a rate 1 - a_ij to compute\n F(a_ij, B, w_ij).\n (It is noteworthy that there is an abuse of notation above when writing\n F(a_ij, B, w_ij) as each block-wise operation will in fact consume\n multiple elements of the sequence (w_ij)).\n\n Args:\n left_matrix: sparse SciPy csr matrix with values in [0, 1].\n right_matrix: sparse SciPy coo signed binary matrix. +1 values correspond\n to train set and -1 values correspond to test set.\n train_indices_out_path: path to output train file. The non zero indices of\n the resulting sparse matrix are dumped as a series of pickled records.\n As many shard will be created as there are rows in left matrix. The shard\n corresponding to row i in the left matrix has the suffix _i appended to\n its file name. Each shard contains a pickled list of list each of which\n corresponds to a users.\n test_indices_out_path: path to output train file. The non zero indices of\n the resulting sparse matrix are dumped as a series of pickled records.\n As many shard will be created as there are rows in left matrix. The shard\n corresponding to row i in the left matrix has the suffix _i appended to\n its file name. Each shard contains a pickled list of list each of which\n corresponds to a users.\n train_metadata_out_path: path to optional complementary output file\n containing the number of train rows (r), columns (c) and non zeros (nnz)\n in a pickled SparseMatrixMetadata named tuple.\n test_metadata_out_path: path to optional complementary output file\n containing the number of test rows (r), columns (c) and non zeros (nnz)\n in a pickled SparseMatrixMetadata named tuple.\n remove_empty_rows: whether to remove rows from the synthetic train and\n test matrices which are not present in the train or the test matrix.\n\n Returns:\n (metadata, train_metadata, test_metadata) triplet of SparseMatrixMetadata\n corresponding to the overall data set, train data set and test data set.\n \"\"\"\n logging.info(\"Writing item sequences to pickle files %s and %s.\",\n train_indices_out_path, test_indices_out_path)\n\n num_rows = 0\n num_removed_rows = 0\n num_cols = left_matrix.shape[1] * right_matrix.shape[1]\n num_interactions = 0\n\n num_train_interactions = 0\n num_test_interactions = 0\n\n if not set(right_matrix.data).issubset({-1, 1}):\n raise ValueError(\n \"Values of sparse matrix should be -1 or 1 but are:\",\n set(right_matrix.data))\n\n for i in xrange(left_matrix.shape[0]):\n\n (shard_num_removed_rows, shard_metadata, shard_train_metadata,\n shard_test_metadata) = _compute_and_write_row_block(\n i, left_matrix, right_matrix, train_indices_out_path,\n test_indices_out_path, remove_empty_rows)\n\n num_rows += shard_metadata.num_rows\n num_removed_rows += shard_num_removed_rows\n num_interactions += shard_metadata.num_interactions\n num_train_interactions += shard_train_metadata.num_interactions\n num_test_interactions += shard_test_metadata.num_interactions\n\n logging.info(\"%d total interactions written.\", num_interactions)\n logging.info(\"%d total rows removed.\", num_removed_rows)\n logging.info(\"%d total train interactions written.\", num_train_interactions)\n logging.info(\"%d toal test interactions written.\", num_test_interactions)\n\n logging.info(\"Done writing.\")\n\n metadata = SparseMatrixMetadata(\n num_interactions=num_interactions,\n num_rows=num_rows, num_cols=num_cols)\n train_metadata = SparseMatrixMetadata(\n num_interactions=num_train_interactions,\n num_rows=num_rows, num_cols=num_cols)\n test_metadata = SparseMatrixMetadata(\n num_interactions=num_test_interactions,\n num_rows=num_rows, num_cols=num_cols)\n\n if train_metadata_out_path is not None:\n util.write_metadata_to_file(\n train_metadata, train_metadata_out_path, tag=\"train\")\n if test_metadata_out_path is not None:\n util.write_metadata_to_file(\n test_metadata, test_metadata_out_path, tag=\"test\")\n\n return metadata, train_metadata, test_metadata"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_gtp_instance(load_file, cgos_mode=False, kgs_mode=False,\n minigui_mode=False):\n \"\"\"Takes a path to model files and set up a GTP engine instance.\"\"\"\n n = DualNetwork(load_file)\n if cgos_mode:\n player = CGOSPlayer(network=n, seconds_per_move=5, timed_match=True,\n two_player_mode=True)\n else:\n player = MCTSPlayer(network=n, two_player_mode=True)\n\n name = \"Minigo-\" + os.path.basename(load_file)\n version = \"0.2\"\n\n engine = gtp_engine.Engine()\n engine.add_cmd_handler(\n gtp_engine.EngineCmdHandler(engine, name, version))\n\n if kgs_mode:\n engine.add_cmd_handler(KgsCmdHandler(player))\n engine.add_cmd_handler(RegressionsCmdHandler(player))\n engine.add_cmd_handler(GoGuiCmdHandler(player))\n if minigui_mode:\n engine.add_cmd_handler(MiniguiBasicCmdHandler(player, courtesy_pass=kgs_mode))\n else:\n engine.add_cmd_handler(BasicCmdHandler(player, courtesy_pass=kgs_mode))\n\n return engine", "response": "Takes a path to model files and sets up a GTP engine instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main(argv):\n del argv\n engine = make_gtp_instance(FLAGS.load_file,\n cgos_mode=FLAGS.cgos_mode,\n kgs_mode=FLAGS.kgs_mode,\n minigui_mode=FLAGS.minigui_mode)\n dbg(\"GTP engine ready\\n\")\n for msg in sys.stdin:\n if not engine.handle_msg(msg.strip()):\n break", "response": "Run Minigo in GTP mode."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate(*tf_records):\n if FLAGS.use_tpu:\n def _input_fn(params):\n return preprocessing.get_tpu_input_tensors(\n params['batch_size'], tf_records, filter_amount=1.0)\n else:\n def _input_fn():\n return preprocessing.get_input_tensors(\n FLAGS.train_batch_size, tf_records, filter_amount=1.0,\n shuffle_examples=False)\n\n steps = FLAGS.examples_to_validate // FLAGS.train_batch_size\n if FLAGS.use_tpu:\n steps //= FLAGS.num_tpu_cores\n\n estimator = dual_net.get_estimator()\n with utils.logged_timer(\"Validating\"):\n estimator.evaluate(_input_fn, steps=steps, name=FLAGS.validate_name)", "response": "Validate a model s performance on a set of holdout data."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef main(argv):\n _, *validation_paths = argv\n if FLAGS.expand_validation_dirs:\n tf_records = []\n with utils.logged_timer(\"Building lists of holdout files\"):\n for record_dir in validation_paths:\n tf_records.extend(gfile.Glob(os.path.join(record_dir, '*.zz')))\n else:\n tf_records = validation_paths\n\n if not tf_records:\n raise RuntimeError(\"Did not find any holdout files for validating!\")\n validate(*tf_records)", "response": "Validate a model s performance on a set of holdout data."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads custom environment setup from a Python source file and run the setup function.", "response": "def setup_custom_environment(custom_module_path):\n \"\"\"Load custom environment setup from a Python source file and run the setup\n function.\n \"\"\"\n module = import_file(\"maskrcnn_benchmark.utils.env.custom_module\", custom_module_path)\n assert hasattr(module, \"setup_environment\") and callable(\n module.setup_environment\n ), (\n \"Custom environment module defined in {} does not have the \"\n \"required callable attribute 'setup_environment'.\"\n ).format(\n custom_module_path\n )\n module.setup_environment()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds all models returning a list of model number and names sorted increasing.", "response": "def get_models():\n \"\"\"Finds all models, returning a list of model number and names\n sorted increasing.\n\n Returns: [(13, 000013-modelname), (17, 000017-modelname), ...etc]\n \"\"\"\n all_models = gfile.Glob(os.path.join(models_dir(), '*.meta'))\n model_filenames = [os.path.basename(m) for m in all_models]\n model_numbers_names = sorted([\n (shipname.detect_model_num(m), shipname.detect_model_name(m))\n for m in model_filenames])\n return model_numbers_names"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_hour_dirs(root=None):\n root = root or selfplay_dir()\n return list(filter(lambda s: re.match(r\"\\d{4}-\\d{2}-\\d{2}-\\d{2}\", s),\n gfile.ListDirectory(root)))", "response": "Gets the directories under selfplay_dir that match YYYY - MM - DD - HH."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints statistics for the most recent n_back models", "response": "def game_counts(n_back=20):\n \"\"\"Prints statistics for the most recent n_back models\"\"\"\n for _, model_name in get_models[-n_back:]:\n games = get_games(model_name)\n print(\"Model: {}, Games: {}\".format(model_name, len(games)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nperform non - maximum suppression on a boxlist.", "response": "def boxlist_nms(boxlist, nms_thresh, max_proposals=-1, score_field=\"scores\"):\n \"\"\"\n Performs non-maximum suppression on a boxlist, with scores specified\n in a boxlist field via score_field.\n\n Arguments:\n boxlist(BoxList)\n nms_thresh (float)\n max_proposals (int): if > 0, then only the top max_proposals are kept\n after non-maximum suppression\n score_field (str)\n \"\"\"\n if nms_thresh <= 0:\n return boxlist\n mode = boxlist.mode\n boxlist = boxlist.convert(\"xyxy\")\n boxes = boxlist.bbox\n score = boxlist.get_field(score_field)\n keep = _box_nms(boxes, score, nms_thresh)\n if max_proposals > 0:\n keep = keep[: max_proposals]\n boxlist = boxlist[keep]\n return boxlist.convert(mode)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves boxes with both sides >= min_size", "response": "def remove_small_boxes(boxlist, min_size):\n \"\"\"\n Only keep boxes with both sides >= min_size\n\n Arguments:\n boxlist (Boxlist)\n min_size (int)\n \"\"\"\n # TODO maybe add an API for querying the ws / hs\n xywh_boxes = boxlist.convert(\"xywh\").bbox\n _, _, ws, hs = xywh_boxes.unbind(dim=1)\n keep = (\n (ws >= min_size) & (hs >= min_size)\n ).nonzero().squeeze(1)\n return boxlist[keep]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef boxlist_iou(boxlist1, boxlist2):\n if boxlist1.size != boxlist2.size:\n raise RuntimeError(\n \"boxlists should have same image size, got {}, {}\".format(boxlist1, boxlist2))\n\n N = len(boxlist1)\n M = len(boxlist2)\n\n area1 = boxlist1.area()\n area2 = boxlist2.area()\n\n box1, box2 = boxlist1.bbox, boxlist2.bbox\n\n lt = torch.max(box1[:, None, :2], box2[:, :2]) # [N,M,2]\n rb = torch.min(box1[:, None, 2:], box2[:, 2:]) # [N,M,2]\n\n TO_REMOVE = 1\n\n wh = (rb - lt + TO_REMOVE).clamp(min=0) # [N,M,2]\n inter = wh[:, :, 0] * wh[:, :, 1] # [N,M]\n\n iou = inter / (area1[:, None] + area2 - inter)\n return iou", "response": "Compute the intersection over two set of boxes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconcatenate a list of BoxList into a", "response": "def cat_boxlist(bboxes):\n \"\"\"\n Concatenates a list of BoxList (having the same image size) into a\n single BoxList\n\n Arguments:\n bboxes (list[BoxList])\n \"\"\"\n assert isinstance(bboxes, (list, tuple))\n assert all(isinstance(bbox, BoxList) for bbox in bboxes)\n\n size = bboxes[0].size\n assert all(bbox.size == size for bbox in bboxes)\n\n mode = bboxes[0].mode\n assert all(bbox.mode == mode for bbox in bboxes)\n\n fields = set(bboxes[0].fields())\n assert all(set(bbox.fields()) == fields for bbox in bboxes)\n\n cat_boxes = BoxList(_cat([bbox.bbox for bbox in bboxes], dim=0), size, mode)\n\n for field in fields:\n data = _cat([bbox.get_field(field) for bbox in bboxes], dim=0)\n cat_boxes.add_field(field, data)\n\n return cat_boxes"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef forward(self, ploc, plabel, gloc, glabel):\n\n mask = glabel > 0\n pos_num = mask.sum(dim=1)\n\n vec_gd = self._loc_vec(gloc)\n\n # sum on four coordinates, and mask\n sl1 = self.sl1_loss(ploc, vec_gd).sum(dim=1)\n sl1 = (mask.float()*sl1).sum(dim=1)\n\n # hard negative mining\n con = self.con_loss(plabel, glabel)\n\n # postive mask will never selected\n con_neg = con.clone()\n con_neg[mask] = 0\n _, con_idx = con_neg.sort(dim=1, descending=True)\n _, con_rank = con_idx.sort(dim=1)\n\n # number of negative three times positive\n neg_num = torch.clamp(3*pos_num, max=mask.size(1)).unsqueeze(-1)\n neg_mask = con_rank < neg_num\n\n closs = (con*(mask.float() + neg_mask.float())).sum(dim=1)\n\n # avoid no object detected\n total_loss = sl1 + closs\n num_mask = (pos_num > 0).float()\n pos_num = pos_num.float().clamp(min=1e-6)\n\n ret = (total_loss*num_mask/pos_num).mean(dim=0)\n return ret", "response": "forward method for the internal method of the forward method."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset gradients of the parameters to the gradients of the parameters_with_grad.", "response": "def set_grads(params, params_with_grad):\n \"\"\"\n Copies gradients from param_with_grad to params\n\n :param params: dst parameters\n :param params_with_grad: src parameters\n \"\"\"\n for param, param_w_grad in zip(params, params_with_grad):\n if param.grad is None:\n param.grad = torch.nn.Parameter(torch.empty_like(param))\n param.grad.data.copy_(param_w_grad.grad.data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_weights(params, new_params):\n for param, new_param in zip(params, new_params):\n param.data.copy_(new_param.data)", "response": "Sets the weights of the parameters in the new_params to the ones in the new_params."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef initialize_model(self, model):\n logging.info('Initializing fp32 clone weights')\n self.fp16_model = model\n self.fp16_model.zero_grad()\n self.fp32_params = [param.to(torch.float32).detach()\n for param in model.parameters()]\n\n for param in self.fp32_params:\n param.requires_grad = True", "response": "Initializes internal state and build fp32 master copy of weights."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef step(self, loss, optimizer, scheduler, update=True):\n loss *= self.loss_scale\n loss.backward()\n\n if update:\n self.set_grads(self.fp32_params, self.fp16_model.parameters())\n if self.loss_scale != 1.0:\n for param in self.fp32_params:\n param.grad.data /= self.loss_scale\n\n norm = clip_grad_norm_(self.fp32_params, self.grad_clip)\n\n if math.isfinite(norm):\n scheduler.step()\n optimizer.step()\n self.set_weights(self.fp16_model.parameters(),\n self.fp32_params)\n self.since_last_invalid += 1\n else:\n self.loss_scale /= self.dls_downscale\n self.since_last_invalid = 0\n logging.info(f'Gradient norm: {norm}')\n logging.info(f'Skipped batch, new scale: {self.loss_scale}')\n\n if self.since_last_invalid >= self.dls_upscale_interval:\n self.loss_scale *= self.dls_upscale\n self.loss_scale = min(self.loss_scale, 8192.0)\n logging.info(f'Upscaling, new scale: {self.loss_scale}')\n self.since_last_invalid = 0\n\n self.fp16_model.zero_grad()", "response": "Performs one step of the optimizer."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nperforms one step of the optimizer.", "response": "def step(self, loss, optimizer, scheduler, update=True):\n \"\"\"\n Performs one step of the optimizer.\n\n :param loss: value of loss function\n :param optimizer: optimizer\n :param update: if True executes weight update\n \"\"\"\n loss.backward()\n if update:\n if self.grad_clip != float('inf'):\n clip_grad_norm_(self.model.parameters(), self.grad_clip)\n scheduler.step()\n optimizer.step()\n self.model.zero_grad()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn inputs and targets Tensors from a serialized tf. Example.", "response": "def _parse_example(serialized_example):\n \"\"\"Return inputs and targets Tensors from a serialized tf.Example.\"\"\"\n data_fields = {\n \"inputs\": tf.VarLenFeature(tf.int64),\n \"targets\": tf.VarLenFeature(tf.int64)\n }\n parsed = tf.parse_single_example(serialized_example, data_fields)\n inputs = tf.sparse_tensor_to_dense(parsed[\"inputs\"])\n targets = tf.sparse_tensor_to_dense(parsed[\"targets\"])\n return inputs, targets"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nindicates whether the example s length is lower than the maximum length.", "response": "def _filter_max_length(example, max_length=256):\n \"\"\"Indicates whether the example's length is lower than the maximum length.\"\"\"\n return tf.logical_and(tf.size(example[0]) <= max_length,\n tf.size(example[1]) <= max_length)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_example_length(example):\n length = tf.maximum(tf.shape(example[0])[0], tf.shape(example[1])[0])\n return length", "response": "Returns the maximum length between the example inputs and targets."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate min and max boundary lists.", "response": "def _create_min_max_boundaries(\n max_length, min_boundary=_MIN_BOUNDARY, boundary_scale=_BOUNDARY_SCALE):\n \"\"\"Create min and max boundary lists up to max_length.\n\n For example, when max_length=24, min_boundary=4 and boundary_scale=2, the\n returned values will be:\n buckets_min = [0, 4, 8, 16, 24]\n buckets_max = [4, 8, 16, 24, 25]\n\n Args:\n max_length: The maximum length of example in dataset.\n min_boundary: Minimum length in boundary.\n boundary_scale: Amount to scale consecutive boundaries in the list.\n\n Returns:\n min and max boundary lists\n\n \"\"\"\n # Create bucket boundaries list by scaling the previous boundary or adding 1\n # (to ensure increasing boundary sizes).\n bucket_boundaries = []\n x = min_boundary\n while x < max_length:\n bucket_boundaries.append(x)\n x = max(x + 1, int(x * boundary_scale))\n\n # Create min and max boundary lists from the initial list.\n buckets_min = [0] + bucket_boundaries\n buckets_max = bucket_boundaries + [max_length + 1]\n return buckets_min, buckets_max"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _batch_examples(dataset, batch_size, max_length):\n # Get min and max boundary lists for each example. These are used to calculate\n # the `bucket_id`, which is the index at which:\n # buckets_min[bucket_id] <= len(example) < buckets_max[bucket_id]\n # Note that using both min and max lists improves the performance.\n buckets_min, buckets_max = _create_min_max_boundaries(max_length)\n\n # Create list of batch sizes for each bucket_id, so that\n # bucket_batch_size[bucket_id] * buckets_max[bucket_id] <= batch_size\n bucket_batch_sizes = [batch_size // x for x in buckets_max]\n # bucket_id will be a tensor, so convert this list to a tensor as well.\n bucket_batch_sizes = tf.constant(bucket_batch_sizes, dtype=tf.int64)\n\n def example_to_bucket_id(example_input, example_target):\n \"\"\"Return int64 bucket id for this example, calculated based on length.\"\"\"\n seq_length = _get_example_length((example_input, example_target))\n\n # TODO: investigate whether removing code branching improves performance.\n conditions_c = tf.logical_and(\n tf.less_equal(buckets_min, seq_length),\n tf.less(seq_length, buckets_max))\n bucket_id = tf.reduce_min(tf.where(conditions_c))\n return bucket_id\n\n def window_size_fn(bucket_id):\n \"\"\"Return number of examples to be grouped when given a bucket id.\"\"\"\n return bucket_batch_sizes[bucket_id]\n\n def batching_fn(bucket_id, grouped_dataset):\n \"\"\"Batch and add padding to a dataset of elements with similar lengths.\"\"\"\n bucket_batch_size = window_size_fn(bucket_id)\n\n # Batch the dataset and add padding so that all input sequences in the\n # examples have the same length, and all target sequences have the same\n # lengths as well. Resulting lengths of inputs and targets can differ.\n return grouped_dataset.padded_batch(bucket_batch_size, ([None], [None]))\n\n return dataset.apply(tf.contrib.data.group_by_window(\n key_func=example_to_bucket_id,\n reduce_func=batching_fn,\n window_size=None,\n window_size_func=window_size_fn))", "response": "Group examples by similar lengths and return batched dataset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread and batch from files.", "response": "def _read_and_batch_from_files(\n file_pattern, batch_size, max_length, num_cpu_cores, shuffle, repeat):\n \"\"\"Create dataset where each item is a dict of \"inputs\" and \"targets\".\n\n Args:\n file_pattern: String used to match the input TFRecord files.\n batch_size: Maximum number of tokens per batch of examples\n max_length: Maximum number of tokens per example\n num_cpu_cores: Number of cpu cores for parallel input processing.\n shuffle: If true, randomizes order of elements.\n repeat: Number of times to repeat the dataset. If None, the dataset is\n repeated forever.\n\n Returns:\n tf.data.Dataset object containing examples loaded from the files.\n \"\"\"\n dataset = tf.data.Dataset.list_files(file_pattern)\n\n if shuffle:\n # Shuffle filenames\n mlperf_log.transformer_print(key=mlperf_log.INPUT_ORDER)\n dataset = dataset.shuffle(buffer_size=_FILE_SHUFFLE_BUFFER)\n\n # Read files and interleave results. When training, the order of the examples\n # will be non-deterministic.\n dataset = dataset.apply(\n tf.contrib.data.parallel_interleave(\n _load_records, sloppy=shuffle, cycle_length=num_cpu_cores))\n\n # Parse each tf.Example into a dictionary\n # TODO: Look into prefetch_input_elements for performance optimization.\n dataset = dataset.map(_parse_example,\n num_parallel_calls=num_cpu_cores)\n\n # Remove examples where the input or target length exceeds the maximum length,\n dataset = dataset.filter(lambda x, y: _filter_max_length((x, y), max_length))\n\n # Batch such that each batch has examples of similar length.\n mlperf_log.transformer_print(key=mlperf_log.INPUT_BATCH_SIZE,\n value=batch_size)\n mlperf_log.transformer_print(key=mlperf_log.INPUT_MAX_LENGTH,\n value=max_length)\n dataset = _batch_examples(dataset, batch_size, max_length)\n dataset = dataset.repeat(repeat)\n\n # Prefetch the next element to improve speed of input pipeline.\n dataset = dataset.prefetch(1)\n return dataset"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload and return dataset of batched examples for use during training.", "response": "def train_input_fn(params):\n \"\"\"Load and return dataset of batched examples for use during training.\"\"\"\n file_pattern = os.path.join(getattr(params, \"data_dir\", \"\"), \"*encoded-train*\")\n return _read_and_batch_from_files(\n file_pattern, params.batch_size, params.max_length, params.num_cpu_cores,\n shuffle=True, repeat=params.repeat_dataset)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nloading and return dataset of batched examples for use during evaluation.", "response": "def eval_input_fn(params):\n \"\"\"Load and return dataset of batched examples for use during evaluation.\"\"\"\n file_pattern = os.path.join(getattr(params, \"data_dir\", \"\"), \"*encoded-dev*\")\n return _read_and_batch_from_files(\n file_pattern, params.batch_size, params.max_length, params.num_cpu_cores,\n shuffle=False, repeat=1)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reduce_loss_dict(loss_dict):\n world_size = get_world_size()\n if world_size < 2:\n return loss_dict\n with torch.no_grad():\n loss_names = []\n all_losses = []\n for k in sorted(loss_dict.keys()):\n loss_names.append(k)\n all_losses.append(loss_dict[k])\n all_losses = torch.stack(all_losses, dim=0)\n dist.reduce(all_losses, dst=0)\n if dist.get_rank() == 0:\n # only main process gets accumulated, so only divide by\n # world_size in this case\n all_losses /= world_size\n reduced_losses = {k: v for k, v in zip(loss_names, all_losses)}\n return reduced_losses", "response": "Reduce the loss dictionary from all processes so that process with rank\n 0 has the averaged results. Returns a dict with the same fields as loss_dict."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting dtype string to tf dtype and set loss_scale default as needed.", "response": "def parse_dtype_info(flags):\n \"\"\"Convert dtype string to tf dtype, and set loss_scale default as needed.\n\n Args:\n flags: namespace object returned by arg parser.\n\n Raises:\n ValueError: If an invalid dtype is provided.\n \"\"\"\n if flags.dtype in (i[0] for i in DTYPE_MAP.values()):\n return # Make function idempotent\n\n try:\n flags.dtype, default_loss_scale = DTYPE_MAP[flags.dtype]\n except KeyError:\n raise ValueError(\"Invalid dtype: {}\".format(flags.dtype))\n\n flags.loss_scale = flags.loss_scale or default_loss_scale"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the IoU tensor based on two boxes tensor.", "response": "def calc_iou_tensor(box1, box2):\n \"\"\" Calculation of IoU based on two boxes tensor,\n Reference to https://github.com/kuangliu/pytorch-ssd\n input:\n box1 (N, 4)\n box2 (M, 4)\n output:\n IoU (N, M)\n \"\"\"\n N = box1.size(0)\n M = box2.size(0)\n\n be1 = box1.unsqueeze(1).expand(-1, M, -1)\n be2 = box2.unsqueeze(0).expand(N, -1, -1)\n\n # Left Top & Right Bottom\n lt = torch.max(be1[:, :, :2], be2[:, :, :2])\n # mask1 = (be1[:,:, 0] < be2[:,:, 0]) ^ (be1[:,:, 1] < be2[:,:, 1])\n # mask1 = ~mask1\n rb = torch.min(be1[:, :, 2:], be2[:, :, 2:])\n # mask2 = (be1[:,:, 2] < be2[:,:, 2]) ^ (be1[:,:, 3] < be2[:,:, 3])\n # mask2 = ~mask2\n\n delta = rb - lt\n delta[delta < 0] = 0\n intersect = delta[:, :, 0] * delta[:, :, 1]\n # *mask1.float()*mask2.float()\n\n delta1 = be1[:, :, 2:] - be1[:, :, :2]\n area1 = delta1[:, :, 0] * delta1[:, :, 1]\n delta2 = be2[:, :, 2:] - be2[:, :, :2]\n area2 = delta2[:, :, 0] * delta2[:, :, 1]\n\n iou = intersect / (area1 + area2 - intersect)\n return iou"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nscaling and transform from xywh to ltrb", "response": "def scale_back_batch(self, bboxes_in, scores_in):\n \"\"\"\n Do scale and transform from xywh to ltrb\n suppose input Nx4xnum_bbox Nxlabel_numxnum_bbox\n \"\"\"\n if bboxes_in.device == torch.device(\"cpu\"):\n self.dboxes = self.dboxes.cpu()\n self.dboxes_xywh = self.dboxes_xywh.cpu()\n else:\n self.dboxes = self.dboxes.cuda()\n self.dboxes_xywh = self.dboxes_xywh.cuda()\n\n bboxes_in = bboxes_in.permute(0, 2, 1)\n scores_in = scores_in.permute(0, 2, 1)\n # print(bboxes_in.device, scores_in.device, self.dboxes_xywh.device)\n\n bboxes_in[:, :, :2] = self.scale_xy * bboxes_in[:, :, :2]\n bboxes_in[:, :, 2:] = self.scale_wh * bboxes_in[:, :, 2:]\n\n bboxes_in[:, :, :2] = bboxes_in[:, :, :2] * self.dboxes_xywh[:, :,\n 2:] + self.dboxes_xywh[:, :,\n :2]\n bboxes_in[:, :, 2:] = bboxes_in[:, :, 2:].exp() * self.dboxes_xywh[:, :,\n 2:]\n\n # Transform format to ltrb\n l, t, r, b = bboxes_in[:, :, 0] - 0.5 * bboxes_in[:, :, 2], \\\n bboxes_in[:, :, 1] - 0.5 * bboxes_in[:, :, 3], \\\n bboxes_in[:, :, 0] + 0.5 * bboxes_in[:, :, 2], \\\n bboxes_in[:, :, 1] + 0.5 * bboxes_in[:, :, 3]\n\n bboxes_in[:, :, 0] = l\n bboxes_in[:, :, 1] = t\n bboxes_in[:, :, 2] = r\n bboxes_in[:, :, 3] = b\n\n return bboxes_in, F.softmax(scores_in, dim=-1)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_anchors(\n stride=16, sizes=(32, 64, 128, 256, 512), aspect_ratios=(0.5, 1, 2)\n):\n \"\"\"Generates a matrix of anchor boxes in (x1, y1, x2, y2) format. Anchors\n are centered on stride / 2, have (approximate) sqrt areas of the specified\n sizes, and aspect ratios as given.\n \"\"\"\n return _generate_anchors(\n stride,\n np.array(sizes, dtype=np.float) / stride,\n np.array(aspect_ratios, dtype=np.float),\n )", "response": "Generates a matrix of anchor boxes in the specified size and aspect ratios."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _generate_anchors(base_size, scales, aspect_ratios):\n anchor = np.array([1, 1, base_size, base_size], dtype=np.float) - 1\n anchors = _ratio_enum(anchor, aspect_ratios)\n anchors = np.vstack(\n [_scale_enum(anchors[i, :], scales) for i in range(anchors.shape[0])]\n )\n return torch.from_numpy(anchors)", "response": "Generate anchor windows by enumerating aspect ratios X\n scales wrt a reference"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts from a flattened coordinate to a Minigo coordinate.", "response": "def from_flat(flat):\n \"\"\"Converts from a flattened coordinate to a Minigo coordinate.\"\"\"\n if flat == go.N * go.N:\n return None\n return divmod(flat, go.N)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_flat(coord):\n if coord is None:\n return go.N * go.N\n return go.N * coord[0] + coord[1]", "response": "Converts from a Minigo coordinate to a flattened coordinate."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_sgf(sgfc):\n if sgfc is None or sgfc == '' or (go.N <= 19 and sgfc == 'tt'):\n return None\n return _SGF_COLUMNS.index(sgfc[1]), _SGF_COLUMNS.index(sgfc[0])", "response": "Converts from an SGF coordinate to a Minigo coordinate."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_gtp(gtpc):\n gtpc = gtpc.upper()\n if gtpc == 'PASS':\n return None\n col = _GTP_COLUMNS.index(gtpc[0])\n row_from_bottom = int(gtpc[1:])\n return go.N - row_from_bottom, col", "response": "Converts from a GTP coordinate to a Minigo coordinate."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting from a Minigo coordinate to a GTP coordinate.", "response": "def to_gtp(coord):\n \"\"\"Converts from a Minigo coordinate to a GTP coordinate.\"\"\"\n if coord is None:\n return 'pass'\n y, x = coord\n return '{}{}'.format(_GTP_COLUMNS[x], go.N - y)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwrapping for cv2. findContours to maintain compatiblity between versions 3 and 4", "response": "def findContours(*args, **kwargs):\n \"\"\"\n Wraps cv2.findContours to maintain compatiblity between versions\n 3 and 4\n\n Returns:\n contours, hierarchy\n \"\"\"\n if cv2.__version__.startswith('4'):\n contours, hierarchy = cv2.findContours(*args, **kwargs)\n elif cv2.__version__.startswith('3'):\n _, contours, hierarchy = cv2.findContours(*args, **kwargs)\n else:\n raise AssertionError(\n 'cv2 must be either version 3 or 4 to call this method')\n\n return contours, hierarchy"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef maybe_add_child(self, fcoord):\n if fcoord not in self.children:\n new_position = self.position.play_move(\n coords.from_flat(fcoord))\n self.children[fcoord] = MCTSNode(\n new_position, fmove=fcoord, parent=self)\n return self.children[fcoord]", "response": "Adds a child node for fcoord if it doesn t already exist."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_virtual_loss(self, up_to):\n self.losses_applied += 1\n # This is a \"win\" for the current node; hence a loss for its parent node\n # who will be deciding whether to investigate this node again.\n loss = self.position.to_play\n self.W += loss\n if self.parent is None or self is up_to:\n return\n self.parent.add_virtual_loss(up_to)", "response": "Propagate a virtual loss up to the root node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npropagate a value estimation up to the root node.", "response": "def backup_value(self, value, up_to):\n \"\"\"Propagates a value estimation up to the root node.\n\n Args:\n value: the value to be propagated (1 = black wins, -1 = white wins)\n up_to: the node to propagate until.\n \"\"\"\n self.N += 1\n self.W += value\n if self.parent is None or self is up_to:\n return\n self.parent.backup_value(value, up_to)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_done(self):\n return self.position.is_game_over() or self.position.n >= FLAGS.max_game_length", "response": "True if the last two moves were Pass or if the position is at a game over or greater than the max depth."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef children_as_pi(self, squash=False):\n probs = self.child_N\n if squash:\n probs = probs ** .98\n sum_probs = np.sum(probs)\n if sum_probs == 0:\n return probs\n return probs / np.sum(probs)", "response": "Returns the child visit counts as a probability distribution"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the most visited path in go - gui VAR format e. g. b r3 w c17...", "response": "def mvp_gg(self):\n \"\"\" Returns most visited path in go-gui VAR format e.g. 'b r3 w c17...\"\"\"\n output = []\n for node in self.most_visited_path_nodes():\n if max(node.child_N) <= 1:\n break\n output.append(coords.to_gtp(coords.from_flat(node.fmove)))\n return ' '.join(output)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _convert_args(handler, args):\n\n args = list(args)\n params = inspect.signature(handler).parameters\n for i, (arg, name) in enumerate(zip(args, params)):\n default = params[name].default\n annotation = params[name].annotation\n\n if annotation != inspect.Parameter.empty:\n if isinstance(annotation, type) and annotation != str:\n # The parameter is annotated with a type that isn't str: convert\n # the arg to that type.\n args[i] = annotation(arg)\n elif default != inspect.Parameter.empty:\n if default is not None and not isinstance(default, str):\n # The parameter has a default value that isn't None or a str:\n # convert the arg to the default value's type.\n args[i] = type(default)(arg)\n\n return args", "response": "Convert a list of command arguments to types specified by the handler."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_cmd_handler(self, handler_obj):\n for field in dir(handler_obj):\n if field.startswith(\"cmd_\"):\n cmd = field[4:]\n fn = getattr(handler_obj, field)\n if cmd in self.cmds:\n print('Replacing {} with {}'.format(\n _handler_name(self.cmds[cmd]), _handler_name(fn)),\n file=sys.stderr)\n self.cmds[cmd] = fn", "response": "Registers a new command handler object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\npadding x and y so that the results have the same length.", "response": "def _pad_tensors_to_same_length(x, y):\n \"\"\"Pad x and y so that the results have the same length (second dimension).\"\"\"\n with tf.name_scope(\"pad_to_same_length\"):\n x_length = tf.shape(x)[1]\n y_length = tf.shape(y)[1]\n\n max_length = tf.maximum(x_length, y_length)\n\n x = tf.pad(x, [[0, 0], [0, max_length - x_length], [0, 0]])\n y = tf.pad(y, [[0, 0], [0, max_length - y_length]])\n return x, y"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the padded cross entropy loss.", "response": "def padded_cross_entropy_loss(logits, labels, smoothing, vocab_size):\n \"\"\"Calculate cross entropy loss while ignoring padding.\n\n Args:\n logits: Tensor of size [batch_size, length_logits, vocab_size]\n labels: Tensor of size [batch_size, length_labels]\n smoothing: Label smoothing constant, used to determine the on and off values\n vocab_size: int size of the vocabulary\n Returns:\n Returns a float32 tensor with shape\n [batch_size, max(length_logits, length_labels)]\n \"\"\"\n with tf.name_scope(\"loss\", [logits, labels]):\n logits, labels = _pad_tensors_to_same_length(logits, labels)\n\n # Calculate smoothing cross entropy\n with tf.name_scope(\"smoothing_cross_entropy\", [logits, labels]):\n confidence = 1.0 - smoothing\n low_confidence = (1.0 - confidence) / tf.to_float(vocab_size - 1)\n soft_targets = tf.one_hot(\n tf.cast(labels, tf.int32),\n depth=vocab_size,\n on_value=confidence,\n off_value=low_confidence)\n xentropy = tf.nn.softmax_cross_entropy_with_logits_v2(\n logits=logits, labels=soft_targets)\n\n # Calculate the best (lowest) possible value of cross entropy, and\n # subtract from the cross entropy loss.\n normalizing_constant = -(\n confidence * tf.log(confidence) + tf.to_float(vocab_size - 1) *\n low_confidence * tf.log(low_confidence + 1e-20))\n xentropy -= normalizing_constant\n\n weights = tf.to_float(tf.not_equal(labels, 0))\n return xentropy * weights, weights"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrap a metric fn that returns scores and weights as an eval metric fn.", "response": "def _convert_to_eval_metric(metric_fn):\n \"\"\"Wrap a metric fn that returns scores and weights as an eval metric fn.\n\n The input metric_fn returns values for the current batch. The wrapper\n aggregates the return values collected over all of the batches evaluated.\n\n Args:\n metric_fn: function that returns scores and weights for the current batch's\n logits and predicted labels.\n\n Returns:\n function that aggregates the scores and weights from metric_fn.\n \"\"\"\n def problem_metric_fn(*args):\n \"\"\"Returns an aggregation of the metric_fn's returned values.\"\"\"\n (scores, weights) = metric_fn(*args)\n\n # The tf.metrics.mean function assures correct aggregation.\n return tf.metrics.mean(scores, weights)\n return problem_metric_fn"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn dictionary of model evaluation metrics.", "response": "def get_eval_metrics(logits, labels, params):\n \"\"\"Return dictionary of model evaluation metrics.\"\"\"\n metrics = {\n \"accuracy\": _convert_to_eval_metric(padded_accuracy)(logits, labels),\n \"accuracy_top5\": _convert_to_eval_metric(padded_accuracy_top5)(\n logits, labels),\n \"accuracy_per_sequence\": _convert_to_eval_metric(\n padded_sequence_accuracy)(logits, labels),\n \"neg_log_perplexity\": _convert_to_eval_metric(padded_neg_log_perplexity)(\n logits, labels, params.vocab_size),\n \"approx_bleu_score\": _convert_to_eval_metric(bleu_score)(logits, labels),\n \"rouge_2_fscore\": _convert_to_eval_metric(rouge_2_fscore)(logits, labels),\n \"rouge_L_fscore\": _convert_to_eval_metric(rouge_l_fscore)(logits, labels),\n }\n\n # Prefix each of the metric names with \"metrics/\". This allows the metric\n # graphs to display under the \"metrics\" category in TensorBoard.\n metrics = {\"metrics/%s\" % k: v for k, v in six.iteritems(metrics)}\n return metrics"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef padded_accuracy_topk(logits, labels, k):\n with tf.variable_scope(\"padded_accuracy_topk\", values=[logits, labels]):\n logits, labels = _pad_tensors_to_same_length(logits, labels)\n weights = tf.to_float(tf.not_equal(labels, 0))\n effective_k = tf.minimum(k, tf.shape(logits)[-1])\n _, outputs = tf.nn.top_k(logits, k=effective_k)\n outputs = tf.to_int32(outputs)\n padded_labels = tf.to_int32(labels)\n padded_labels = tf.expand_dims(padded_labels, axis=-1)\n padded_labels += tf.zeros_like(outputs) # Pad to same shape.\n same = tf.to_float(tf.equal(outputs, padded_labels))\n same_topk = tf.reduce_sum(same, axis=-1)\n return same_topk, weights", "response": "Percentage of times that top - k predictions matches labels on non - 0s."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef padded_sequence_accuracy(logits, labels):\n with tf.variable_scope(\"padded_sequence_accuracy\", values=[logits, labels]):\n logits, labels = _pad_tensors_to_same_length(logits, labels)\n weights = tf.to_float(tf.not_equal(labels, 0))\n outputs = tf.to_int32(tf.argmax(logits, axis=-1))\n padded_labels = tf.to_int32(labels)\n not_correct = tf.to_float(tf.not_equal(outputs, padded_labels)) * weights\n axis = list(range(1, len(outputs.get_shape())))\n correct_seq = 1.0 - tf.minimum(1.0, tf.reduce_sum(not_correct, axis=axis))\n return correct_seq, tf.constant(1.0)", "response": "Percentage of times that predictions matches labels everywhere ( non - 0 )."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef padded_neg_log_perplexity(logits, labels, vocab_size):\n num, den = padded_cross_entropy_loss(logits, labels, 0, vocab_size)\n return -num, den", "response": "Average log - perplexity excluding padding 0s. No smoothing."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef bleu_score(logits, labels):\n predictions = tf.to_int32(tf.argmax(logits, axis=-1))\n # TODO: Look into removing use of py_func\n bleu = tf.py_func(compute_bleu, (labels, predictions), tf.float32)\n return bleu, tf.constant(1.0)", "response": "Approximate BLEU score computation between labels and predictions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_ngrams_with_counter(segment, max_order):\n ngram_counts = collections.Counter()\n for order in xrange(1, max_order + 1):\n for i in xrange(0, len(segment) - order + 1):\n ngram = tuple(segment[i:i + order])\n ngram_counts[ngram] += 1\n return ngram_counts", "response": "Extracts all n - grams from a given text segment and returns a Counter containing the number of times each n - gram occurred."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute BLEU score of translated segments against one or more references.", "response": "def compute_bleu(reference_corpus, translation_corpus, max_order=4,\n use_bp=True):\n \"\"\"Computes BLEU score of translated segments against one or more references.\n\n Args:\n reference_corpus: list of references for each translation. Each\n reference should be tokenized into a list of tokens.\n translation_corpus: list of translations to score. Each translation\n should be tokenized into a list of tokens.\n max_order: Maximum n-gram order to use when computing BLEU score.\n use_bp: boolean, whether to apply brevity penalty.\n\n Returns:\n BLEU score.\n \"\"\"\n reference_length = 0\n translation_length = 0\n bp = 1.0\n geo_mean = 0\n\n matches_by_order = [0] * max_order\n possible_matches_by_order = [0] * max_order\n precisions = []\n\n for (references, translations) in zip(reference_corpus, translation_corpus):\n reference_length += len(references)\n translation_length += len(translations)\n ref_ngram_counts = _get_ngrams_with_counter(references, max_order)\n translation_ngram_counts = _get_ngrams_with_counter(translations, max_order)\n\n overlap = dict((ngram,\n min(count, translation_ngram_counts[ngram]))\n for ngram, count in ref_ngram_counts.items())\n\n for ngram in overlap:\n matches_by_order[len(ngram) - 1] += overlap[ngram]\n for ngram in translation_ngram_counts:\n possible_matches_by_order[len(ngram) - 1] += translation_ngram_counts[\n ngram]\n\n precisions = [0] * max_order\n smooth = 1.0\n\n for i in xrange(0, max_order):\n if possible_matches_by_order[i] > 0:\n precisions[i] = float(matches_by_order[i]) / possible_matches_by_order[i]\n if matches_by_order[i] > 0:\n precisions[i] = float(matches_by_order[i]) / possible_matches_by_order[\n i]\n else:\n smooth *= 2\n precisions[i] = 1.0 / (smooth * possible_matches_by_order[i])\n else:\n precisions[i] = 0.0\n\n if max(precisions) > 0:\n p_log_sum = sum(math.log(p) for p in precisions if p)\n geo_mean = math.exp(p_log_sum / max_order)\n\n if use_bp:\n ratio = translation_length / reference_length\n bp = math.exp(1 - 1. / ratio) if ratio < 1.0 else 1.0\n bleu = geo_mean * bp\n return np.float32(bleu)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrouge - 2 F1 score computation between labels and predictions.", "response": "def rouge_2_fscore(logits, labels):\n \"\"\"ROUGE-2 F1 score computation between labels and predictions.\n\n This is an approximate ROUGE scoring method since we do not glue word pieces\n or decode the ids and tokenize the output.\n\n Args:\n logits: tensor, model predictions\n labels: tensor, gold output.\n\n Returns:\n rouge2_fscore: approx rouge-2 f1 score.\n \"\"\"\n predictions = tf.to_int32(tf.argmax(logits, axis=-1))\n # TODO: Look into removing use of py_func\n rouge_2_f_score = tf.py_func(rouge_n, (predictions, labels), tf.float32)\n return rouge_2_f_score, tf.constant(1.0)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute ROUGE - N f1 score of two text collections of sentences.", "response": "def rouge_n(eval_sentences, ref_sentences, n=2):\n \"\"\"Computes ROUGE-N f1 score of two text collections of sentences.\n\n Source: https://www.microsoft.com/en-us/research/publication/\n rouge-a-package-for-automatic-evaluation-of-summaries/\n\n Args:\n eval_sentences: Predicted sentences.\n ref_sentences: Sentences from the reference set\n n: Size of ngram. Defaults to 2.\n\n Returns:\n f1 score for ROUGE-N\n \"\"\"\n f1_scores = []\n for eval_sentence, ref_sentence in zip(eval_sentences, ref_sentences):\n eval_ngrams = _get_ngrams(n, eval_sentence)\n ref_ngrams = _get_ngrams(n, ref_sentence)\n ref_count = len(ref_ngrams)\n eval_count = len(eval_ngrams)\n\n # Count the overlapping ngrams between evaluated and reference\n overlapping_ngrams = eval_ngrams.intersection(ref_ngrams)\n overlapping_count = len(overlapping_ngrams)\n\n # Handle edge case. This isn't mathematically correct, but it's good enough\n if eval_count == 0:\n precision = 0.0\n else:\n precision = float(overlapping_count) / eval_count\n if ref_count == 0:\n recall = 0.0\n else:\n recall = float(overlapping_count) / ref_count\n f1_scores.append(2.0 * ((precision * recall) / (precision + recall + 1e-8)))\n\n # return overlapping_count / reference_count\n return np.mean(f1_scores, dtype=np.float32)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rouge_l_fscore(predictions, labels):\n outputs = tf.to_int32(tf.argmax(predictions, axis=-1))\n rouge_l_f_score = tf.py_func(rouge_l_sentence_level, (outputs, labels),\n tf.float32)\n return rouge_l_f_score, tf.constant(1.0)", "response": "ROUGE scores computation between labels and predictions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_args():\n parser = ArgumentParser(description=\"PyTorch distributed training launch \"\n \"helper utilty that will spawn up \"\n \"multiple distributed processes\")\n\n # Optional arguments for the launch helper\n parser.add_argument(\"--nnodes\", type=int, default=1,\n help=\"The number of nodes to use for distributed \"\n \"training\")\n parser.add_argument(\"--node_rank\", type=int, default=0,\n help=\"The rank of the node for multi-node distributed \"\n \"training\")\n parser.add_argument(\"--nproc_per_node\", type=int, default=1,\n help=\"The number of processes to launch on each node, \"\n \"for GPU training, this is recommended to be set \"\n \"to the number of GPUs in your system so that \"\n \"each process can be bound to a single GPU.\")\n parser.add_argument(\"--master_addr\", default=\"127.0.0.1\", type=str,\n help=\"Master node (rank 0)'s address, should be either \"\n \"the IP address or the hostname of node 0, for \"\n \"single node multi-proc training, the \"\n \"--master_addr can simply be 127.0.0.1\")\n parser.add_argument(\"--master_port\", default=29500, type=int,\n help=\"Master node (rank 0)'s free port that needs to \"\n \"be used for communciation during distributed \"\n \"training\")\n parser.add_argument('--no_hyperthreads', action='store_true',\n help='Flag to disable binding to hyperthreads')\n parser.add_argument('--no_membind', action='store_true',\n help='Flag to disable memory binding')\n\n # non-optional arguments for binding\n parser.add_argument(\"--nsockets_per_node\", type=int, required=True,\n help=\"Number of CPU sockets on a node\")\n parser.add_argument(\"--ncores_per_socket\", type=int, required=True,\n help=\"Number of CPU cores per socket\")\n\n # positional\n parser.add_argument(\"training_script\", type=str,\n help=\"The full path to the single GPU training \"\n \"program/script to be launched in parallel, \"\n \"followed by all the arguments for the \"\n \"training script\")\n\n # rest from the training program\n parser.add_argument('training_script_args', nargs=REMAINDER)\n return parser.parse_args()", "response": "Parse command line options and return a parser object"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndownload and extract files from compressed archive file.", "response": "def download_and_extract(path, url, input_filename, target_filename):\n \"\"\"Extract files from downloaded compressed archive file.\n\n Args:\n path: string directory where the files will be downloaded\n url: url containing the compressed input and target files\n input_filename: name of file containing data in source language\n target_filename: name of file containing data in target language\n\n Returns:\n Full paths to extracted input and target files.\n\n Raises:\n OSError: if the the download/extraction fails.\n \"\"\"\n logging.info('Downloading and extracting data to: %s' % path)\n # Check if extracted files already exist in path\n input_file = find_file(path, input_filename)\n target_file = find_file(path, target_filename)\n if input_file and target_file:\n logging.info(\"Already downloaded and extracted %s.\" % url)\n return input_file, target_file\n\n # Download archive file if it doesn't already exist.\n compressed_file = download_from_url(path, url)\n\n # Extract compressed files\n logging.info(\"Extracting %s.\" % compressed_file)\n with tarfile.open(compressed_file, \"r:gz\") as corpus_tar:\n corpus_tar.extractall(path)\n\n # Return filepaths of the requested files.\n input_file = find_file(path, input_filename)\n target_file = find_file(path, target_filename)\n\n if input_file and target_file:\n return input_file, target_file\n\n raise OSError(\"Download/extraction failed for url %s to path %s\" %\n (url, path))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef main(unused_argv):\n make_dir(FLAGS.raw_dir)\n make_dir(FLAGS.data_dir)\n\n # Get paths of download/extracted training and evaluation files.\n print(\"Step 1/4: Downloading data from source\")\n train_files = get_raw_files(FLAGS.raw_dir, _TRAIN_DATA_SOURCES)\n eval_files = get_raw_files(FLAGS.raw_dir, _EVAL_DATA_SOURCES)", "response": "Download training and evaluation data for the Transformer model."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_comment_node(comment):\n # Example of a comment node. The resign threshold line appears only\n # for the first move in the game; it gets preprocessed by extract_game_data\n \"\"\"\n Resign Threshold: -0.88\n -0.0662\n D4 (100) ==> D16 (14) ==> Q16 (3) ==> Q4 (1) ==> Q: -0.07149\n move: action Q U P P-Dir N soft-N p-delta p-rel\n D4 : -0.028, -0.048, 0.020, 0.048, 0.064, 100 0.1096 0.06127 1.27\n D16 : -0.024, -0.043, 0.019, 0.044, 0.059, 96 0.1053 0.06135 1.40\n \"\"\"\n\n lines = comment.split('\\n')\n if lines[0].startswith('Resign'):\n lines = lines[1:]\n\n post_Q = float(lines[0])\n debug_rows = []\n comment_splitter = re.compile(r'[ :,]')\n for line in lines[3:]:\n if not line:\n continue\n columns = comment_splitter.split(line)\n columns = list(filter(bool, columns))\n coord, *other_columns = columns\n coord = coords.to_flat(coords.from_gtp(coord))\n debug_rows.append(DebugRow(coord, *map(float, other_columns)))\n if FLAGS.only_top_move:\n break\n return post_Q, debug_rows", "response": "Parse a comment node into a post - Q and debug rows."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef forward(self, inputs, lengths):\n x = self.embedder(inputs)\n\n # bidirectional layer\n x = self.dropout(x)\n x = pack_padded_sequence(x, lengths.cpu().numpy(),\n batch_first=self.batch_first)\n x, _ = self.rnn_layers[0](x)\n x, _ = pad_packed_sequence(x, batch_first=self.batch_first)\n\n # 1st unidirectional layer\n x = self.dropout(x)\n x, _ = self.rnn_layers[1](x)\n\n # the rest of unidirectional layers,\n # with residual connections starting from 3rd layer\n for i in range(2, len(self.rnn_layers)):\n residual = x\n x = self.dropout(x)\n x, _ = self.rnn_layers[i](x)\n x = x + residual\n\n return x", "response": "Execute the encoder.\n\n :param inputs: tensor with indices from the vocabulary\n :param lengths: vector with sequence lengths (excluding padding)\n\n returns: tensor with encoded sequences"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrunning the given subprocess command in a coroutine.", "response": "async def checked_run(*cmd):\n \"\"\"Run the given subprocess command in a coroutine.\n\n Args:\n *cmd: the command to run and its arguments.\n\n Returns:\n The output that the command wrote to stdout.\n\n Raises:\n RuntimeError: if the command returns a non-zero result.\n \"\"\"\n\n # Start the subprocess.\n logging.info('Running: %s', expand_cmd_str(cmd))\n with logged_timer('{} finished'.format(get_cmd_name(cmd))):\n p = await asyncio.create_subprocess_exec(\n *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT)\n\n # Stream output from the process stdout.\n chunks = []\n while True:\n chunk = await p.stdout.read(16 * 1024)\n if not chunk:\n break\n chunks.append(chunk)\n\n # Wait for the process to finish, check it was successful & build stdout.\n await p.wait()\n stdout = b''.join(chunks).decode()[:-1]\n if p.returncode:\n raise RuntimeError('Return code {} from process: {}\\n{}'.format(\n p.returncode, expand_cmd_str(cmd), stdout))\n\n return stdout"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef wait(aws):\n\n aws_list = aws if isinstance(aws, list) else [aws]\n results = asyncio.get_event_loop().run_until_complete(asyncio.gather(\n *aws_list, return_exceptions=True))\n # If any of the cmds failed, re-raise the error.\n for result in results:\n if isinstance(result, Exception):\n raise result\n return results if isinstance(aws, list) else results[0]", "response": "Waits for all of the awaitable objects in aws to finish."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_subdirs(parent_path):\n entries = os.listdir(parent_path)\n subdirs = [(entry, os.path.join(parent_path, entry))\n for entry in entries\n if os.path.isdir(entry)]\n return subdirs", "response": "Return a list of tuples of direct subdirectories of\n\n parent_path where each tuple corresponds to one subdirectory."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nforward the feature maps for each level and return the result.", "response": "def forward(self, x, boxes):\n \"\"\"\n Arguments:\n x (list[Tensor]): feature maps for each level\n boxes (list[BoxList]): boxes to be used to perform the pooling operation.\n Returns:\n result (Tensor)\n \"\"\"\n num_levels = len(self.poolers)\n rois = self.convert_to_roi_format(boxes)\n if num_levels == 1:\n return self.poolers[0](x[0], rois)\n\n levels = self.map_levels(boxes)\n\n num_rois = len(rois)\n num_channels = x[0].shape[1]\n output_size = self.output_size[0]\n\n dtype, device = x[0].dtype, x[0].device\n result = torch.zeros(\n (num_rois, num_channels, output_size, output_size),\n dtype=dtype,\n device=device,\n )\n for level, (per_level_feature, pooler) in enumerate(zip(x, self.poolers)):\n idx_in_level = torch.nonzero(levels == level).squeeze(1)\n rois_per_level = rois[idx_in_level]\n result[idx_in_level] = pooler(per_level_feature, rois_per_level)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\naugments the audio with sox.", "response": "def augment_audio_with_sox(path, sample_rate, tempo, gain):\n \"\"\"\n Changes tempo and gain of the recording with sox and loads it.\n \"\"\"\n with NamedTemporaryFile(suffix=\".wav\") as augmented_file:\n augmented_filename = augmented_file.name\n sox_augment_params = [\"tempo\", \"{:.3f}\".format(tempo), \"gain\", \"{:.3f}\".format(gain)]\n sox_params = \"sox \\\"{}\\\" -r {} -c 1 -b 16 {} {} >/dev/null 2>&1\".format(path, sample_rate,\n augmented_filename,\n \" \".join(sox_augment_params))\n os.system(sox_params)\n y = load_audio(augmented_filename)\n return y"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load_randomly_augmented_audio(path, sample_rate=16000, tempo_range=(0.85, 1.15),\n gain_range=(-6, 8)):\n \"\"\"\n Picks tempo and gain uniformly, applies it to the utterance by using sox utility.\n Returns the augmented utterance.\n \"\"\"\n low_tempo, high_tempo = tempo_range\n tempo_value = np.random.uniform(low=low_tempo, high=high_tempo)\n low_gain, high_gain = gain_range\n gain_value = np.random.uniform(low=low_gain, high=high_gain)\n audio = augment_audio_with_sox(path=path, sample_rate=sample_rate,\n tempo=tempo_value, gain=gain_value)\n return audio", "response": "Load audio from a randomly augmented utterance."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndefines how to train evaluate and predict from the transformer model.", "response": "def model_fn(features, labels, mode, params):\n \"\"\"Defines how to train, evaluate and predict from the transformer model.\"\"\"\n with tf.variable_scope(\"model\"):\n inputs, targets = features, labels\n\n # Create model and get output logits.\n model = transformer.Transformer(params, mode == tf.estimator.ModeKeys.TRAIN)\n\n output = model(inputs, targets)\n\n # When in prediction mode, the labels/targets is None. The model output\n # is the prediction\n if mode == tf.estimator.ModeKeys.PREDICT:\n return tf.estimator.EstimatorSpec(\n tf.estimator.ModeKeys.PREDICT,\n predictions=output)\n\n logits = output\n\n # Calculate model loss.\n xentropy, weights = metrics.padded_cross_entropy_loss(\n logits, targets, params.label_smoothing, params.vocab_size)\n loss = tf.reduce_sum(xentropy * weights) / tf.reduce_sum(weights)\n\n if mode == tf.estimator.ModeKeys.EVAL:\n return tf.estimator.EstimatorSpec(\n mode=mode, loss=loss, predictions={\"predictions\": logits},\n eval_metric_ops=metrics.get_eval_metrics(logits, labels, params))\n else:\n train_op = get_train_op(loss, params)\n return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate learning rate with linear warmup and rsqrt decay.", "response": "def get_learning_rate(learning_rate, hidden_size, learning_rate_warmup_steps):\n \"\"\"Calculate learning rate with linear warmup and rsqrt decay.\"\"\"\n with tf.name_scope(\"learning_rate\"):\n warmup_steps = tf.to_float(learning_rate_warmup_steps)\n step = tf.to_float(tf.train.get_or_create_global_step())\n\n learning_rate *= (hidden_size ** -0.5)\n # Apply linear warmup\n learning_rate *= tf.minimum(1.0, step / warmup_steps)\n # Apply rsqrt decay\n learning_rate *= tf.rsqrt(tf.maximum(step, warmup_steps))\n\n # Save learning rate value to TensorBoard summary.\n tf.summary.scalar(\"learning_rate\", learning_rate)\n\n return learning_rate"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates training operation that updates variables based on loss.", "response": "def get_train_op(loss, params):\n \"\"\"Generate training operation that updates variables based on loss.\"\"\"\n with tf.variable_scope(\"get_train_op\"):\n mlperf_log.transformer_print(\n key=mlperf_log.OPT_LR_WARMUP_STEPS,\n value=params.learning_rate_warmup_steps)\n learning_rate = get_learning_rate(\n params.learning_rate, params.hidden_size,\n params.learning_rate_warmup_steps)\n log_id = mlperf_log.resnet_print(key=mlperf_log.OPT_LR, deferred=True)\n learning_rate = tf_mlperf_log.log_deferred(op=learning_rate, log_id=log_id,\n every_n=100)\n\n # Create optimizer. Use LazyAdamOptimizer from TF contrib, which is faster\n # than the TF core Adam optimizer.\n mlperf_log.transformer_print(key=mlperf_log.OPT_NAME,\n value=mlperf_log.LAZY_ADAM)\n mlperf_log.transformer_print(key=mlperf_log.OPT_HP_ADAM_BETA1,\n value=params.optimizer_adam_beta1)\n mlperf_log.transformer_print(key=mlperf_log.OPT_HP_ADAM_BETA2,\n value=params.optimizer_adam_beta2)\n mlperf_log.transformer_print(key=mlperf_log.OPT_HP_ADAM_EPSILON,\n value=params.optimizer_adam_epsilon)\n optimizer = tf.contrib.opt.LazyAdamOptimizer(\n learning_rate,\n beta1=params.optimizer_adam_beta1,\n beta2=params.optimizer_adam_beta2,\n epsilon=params.optimizer_adam_epsilon)\n\n # Calculate and apply gradients using LazyAdamOptimizer.\n global_step = tf.train.get_global_step()\n tvars = tf.trainable_variables()\n gradients = optimizer.compute_gradients(\n loss, tvars, colocate_gradients_with_ops=True)\n train_op = optimizer.apply_gradients(\n gradients, global_step=global_step, name=\"train\")\n\n # Save gradient norm to Tensorboard\n tf.summary.scalar(\"global_norm/gradient_norm\",\n tf.global_norm(list(zip(*gradients))[0]))\n\n return train_op"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef translate_and_compute_bleu(estimator, subtokenizer, bleu_source, bleu_ref):\n # Create temporary file to store translation.\n tmp = tempfile.NamedTemporaryFile(delete=False)\n tmp_filename = tmp.name\n\n translate.translate_file(\n estimator, subtokenizer, bleu_source, output_file=tmp_filename,\n print_all_translations=False)\n\n # Compute uncased and cased bleu scores.\n uncased_score = compute_bleu.bleu_wrapper(bleu_ref, tmp_filename, False)\n cased_score = compute_bleu.bleu_wrapper(bleu_ref, tmp_filename, True)\n os.remove(tmp_filename)\n return uncased_score, cased_score", "response": "Translate file and report the cased and uncased bleu scores."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef evaluate_and_log_bleu(estimator, bleu_writer, bleu_source, bleu_ref):\n subtokenizer = tokenizer.Subtokenizer(\n os.path.join(FLAGS.data_dir, FLAGS.vocab_file))\n\n uncased_score, cased_score = translate_and_compute_bleu(\n estimator, subtokenizer, bleu_source, bleu_ref)\n\n print(\"Bleu score (uncased):\", uncased_score)\n print(\"Bleu score (cased):\", cased_score)\n\n summary = tf.Summary(value=[\n tf.Summary.Value(tag=\"bleu/uncased\", simple_value=uncased_score),\n tf.Summary.Value(tag=\"bleu/cased\", simple_value=cased_score),\n ])\n\n bleu_writer.add_summary(summary, get_global_step(estimator))\n bleu_writer.flush()\n return uncased_score, cased_score", "response": "Calculate and record the BLEU score."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntrains and evaluate model and optionally compute BLEU score.", "response": "def train_schedule(\n estimator, train_eval_iterations, single_iteration_train_steps=None,\n single_iteration_train_epochs=None, bleu_source=None, bleu_ref=None,\n bleu_threshold=None):\n \"\"\"Train and evaluate model, and optionally compute model's BLEU score.\n\n **Step vs. Epoch vs. Iteration**\n\n Steps and epochs are canonical terms used in TensorFlow and general machine\n learning. They are used to describe running a single process (train/eval):\n - Step refers to running the process through a single or batch of examples.\n - Epoch refers to running the process through an entire dataset.\n\n E.g. training a dataset with 100 examples. The dataset is\n divided into 20 batches with 5 examples per batch. A single training step\n trains the model on one batch. After 20 training steps, the model will have\n trained on every batch in the dataset, or, in other words, one epoch.\n\n Meanwhile, iteration is used in this implementation to describe running\n multiple processes (training and eval).\n - A single iteration:\n 1. trains the model for a specific number of steps or epochs.\n 2. evaluates the model.\n 3. (if source and ref files are provided) compute BLEU score.\n\n This function runs through multiple train+eval+bleu iterations.\n\n Args:\n estimator: tf.Estimator containing model to train.\n train_eval_iterations: Number of times to repeat the train+eval iteration.\n single_iteration_train_steps: Number of steps to train in one iteration.\n single_iteration_train_epochs: Number of epochs to train in one iteration.\n bleu_source: File containing text to be translated for BLEU calculation.\n bleu_ref: File containing reference translations for BLEU calculation.\n bleu_threshold: minimum BLEU score before training is stopped.\n\n Raises:\n ValueError: if both or none of single_iteration_train_steps and\n single_iteration_train_epochs were defined.\n \"\"\"\n # Ensure that exactly one of single_iteration_train_steps and\n # single_iteration_train_epochs is defined.\n if single_iteration_train_steps is None:\n if single_iteration_train_epochs is None:\n raise ValueError(\n \"Exactly one of single_iteration_train_steps or \"\n \"single_iteration_train_epochs must be defined. Both were none.\")\n else:\n if single_iteration_train_epochs is not None:\n raise ValueError(\n \"Exactly one of single_iteration_train_steps or \"\n \"single_iteration_train_epochs must be defined. Both were defined.\")\n\n evaluate_bleu = bleu_source is not None and bleu_ref is not None\n\n # Print out training schedule\n print(\"Training schedule:\")\n if single_iteration_train_epochs is not None:\n print(\"\\t1. Train for %d epochs.\" % single_iteration_train_epochs)\n else:\n print(\"\\t1. Train for %d steps.\" % single_iteration_train_steps)\n print(\"\\t2. Evaluate model.\")\n if evaluate_bleu:\n print(\"\\t3. Compute BLEU score.\")\n if bleu_threshold is not None:\n print(\"Repeat above steps until the BLEU score reaches\", bleu_threshold)\n if not evaluate_bleu or bleu_threshold is None:\n print(\"Repeat above steps %d times.\" % train_eval_iterations)\n\n if evaluate_bleu:\n # Set summary writer to log bleu score.\n bleu_writer = tf.summary.FileWriter(\n os.path.join(estimator.model_dir, BLEU_DIR))\n if bleu_threshold is not None:\n # Change loop stopping condition if bleu_threshold is defined.\n train_eval_iterations = INF\n\n # Loop training/evaluation/bleu cycles\n mlperf_log.transformer_print(key=mlperf_log.TRAIN_LOOP)\n for i in xrange(train_eval_iterations):\n print(\"Starting iteration\", i + 1)\n\n mlperf_log.transformer_print(key=mlperf_log.TRAIN_EPOCH,\n value=i * single_iteration_train_epochs + 1)\n\n # Train the model for single_iteration_train_steps or until the input fn\n # runs out of examples (if single_iteration_train_steps is None).\n estimator.train(dataset.train_input_fn, steps=single_iteration_train_steps)\n\n mlperf_log.transformer_print(key=mlperf_log.EVAL_START)\n eval_results = estimator.evaluate(dataset.eval_input_fn)\n print(\"Evaluation results (iter %d/%d):\" % (i + 1, train_eval_iterations),\n eval_results)\n\n if evaluate_bleu:\n uncased_score, _ = evaluate_and_log_bleu(\n estimator, bleu_writer, bleu_source, bleu_ref)\n if bleu_threshold is not None and uncased_score > bleu_threshold:\n bleu_writer.close()\n break\n mlperf_log.transformer_print(key=mlperf_log.EVAL_TARGET, value=bleu_threshold)\n mlperf_log.transformer_print(key=mlperf_log.EVAL_ACCURACY, value=uncased_score)\n mlperf_log.transformer_print(key=mlperf_log.EVAL_STOP)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_membership(self, member, role=github.GithubObject.NotSet):\n assert isinstance(member, github.NamedUser.NamedUser), member\n assert role is github.GithubObject.NotSet or isinstance(\n role, (str, unicode)), role\n if role is not github.GithubObject.NotSet:\n assert role in ['member', 'maintainer']\n put_parameters = {\n \"role\": role,\n }\n else:\n put_parameters = {\n \"role\": \"member\",\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/memberships/\" + member._identity,\n input=put_parameters\n )", "response": "Adds a member to the hierarchy of the user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd the current user s identity to the specified repository.", "response": "def add_to_repos(self, repo):\n \"\"\"\n :calls: `PUT /teams/:id/repos/:org/:repo `_\n :param repo: :class:`github.Repository.Repository`\n :rtype: None\n \"\"\"\n assert isinstance(repo, github.Repository.Repository), repo\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/repos/\" + repo._identity\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_repo_permission(self, repo, permission):\n assert isinstance(repo, github.Repository.Repository), repo\n put_parameters = {\n \"permission\": permission,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/repos/\" + repo._identity,\n input=put_parameters\n )", "response": "sets the permission for the specified repository"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef edit(self, name, description=github.GithubObject.NotSet, permission=github.GithubObject.NotSet, privacy=github.GithubObject.NotSet):\n assert isinstance(name, (str, unicode)), name\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert permission is github.GithubObject.NotSet or isinstance(permission, (str, unicode)), permission\n assert privacy is github.GithubObject.NotSet or isinstance(privacy, (str, unicode)), privacy\n post_parameters = {\n \"name\": name,\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if permission is not github.GithubObject.NotSet:\n post_parameters[\"permission\"] = permission\n if privacy is not github.GithubObject.NotSet:\n post_parameters[\"privacy\"] = privacy\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Edits the properties of the current object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a paginated list of members of the specified role.", "response": "def get_members(self, role=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /teams/:id/members `_\n :param role: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n assert role is github.GithubObject.NotSet or isinstance(role, (str, unicode)), role\n url_parameters = dict()\n if role is not github.GithubObject.NotSet:\n assert role in ['member', 'maintainer', 'all']\n url_parameters[\"role\"] = role\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/members\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_repos(self):\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n self.url + \"/repos\",\n None\n )", "response": "Returns a paginated list of repos for the current user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning true if the user has a team in the hierarchy.", "response": "def has_in_members(self, member):\n \"\"\"\n :calls: `GET /teams/:id/members/:user `_\n :param member: :class:`github.NamedUser.NamedUser`\n :rtype: bool\n \"\"\"\n assert isinstance(member, github.NamedUser.NamedUser), member\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/members/\" + member._identity\n )\n return status == 204"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef has_in_repos(self, repo):\n assert isinstance(repo, github.Repository.Repository), repo\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/repos/\" + repo._identity\n )\n return status == 204", "response": "Returns true if the user has a specific tag in the specified repository."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninitializes a new debug frame with requestHeader", "response": "def NEW_DEBUG_FRAME(self, requestHeader):\n \"\"\"\n Initialize a debug frame with requestHeader\n Frame count is updated and will be attached to respond header\n The structure of a frame: [requestHeader, statusCode, responseHeader, raw_data]\n Some of them may be None\n \"\"\"\n if self.DEBUG_FLAG: # pragma no branch (Flag always set in tests)\n new_frame = [requestHeader, None, None, None]\n if self._frameCount < self.DEBUG_FRAME_BUFFER_SIZE - 1: # pragma no branch (Should be covered)\n self._frameBuffer.append(new_frame)\n else:\n self._frameBuffer[0] = new_frame # pragma no cover (Should be covered)\n\n self._frameCount = len(self._frameBuffer) - 1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the current frame with responseHeader and data", "response": "def DEBUG_ON_RESPONSE(self, statusCode, responseHeader, data):\n '''\n Update current frame with response\n Current frame index will be attached to responseHeader\n '''\n if self.DEBUG_FLAG: # pragma no branch (Flag always set in tests)\n self._frameBuffer[self._frameCount][1:4] = [statusCode, responseHeader, data]\n responseHeader[self.DEBUG_HEADER_KEY] = self._frameCount"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_issue(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.issue_url\n )\n return github.Issue.Issue(self._requester, headers, data, completed=True)", "response": "returns a github. Issue. Issue object for the current issue"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a comment for a given branch.", "response": "def create_comment(self, body, commit_id, path, position):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/pulls/:number/comments `_\n :param body: string\n :param commit_id: :class:`github.Commit.Commit`\n :param path: string\n :param position: integer\n :rtype: :class:`github.PullRequestComment.PullRequestComment`\n \"\"\"\n return self.create_review_comment(body, commit_id, path, position)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a review comment for the specified branch.", "response": "def create_review_comment(self, body, commit_id, path, position):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/pulls/:number/comments `_\n :param body: string\n :param commit_id: :class:`github.Commit.Commit`\n :param path: string\n :param position: integer\n :rtype: :class:`github.PullRequestComment.PullRequestComment`\n \"\"\"\n assert isinstance(body, (str, unicode)), body\n assert isinstance(commit_id, github.Commit.Commit), commit_id\n assert isinstance(path, (str, unicode)), path\n assert isinstance(position, (int, long)), position\n post_parameters = {\n \"body\": body,\n \"commit_id\": commit_id._identity,\n \"path\": path,\n \"position\": position,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/comments\",\n input=post_parameters\n )\n return github.PullRequestComment.PullRequestComment(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new issue comment.", "response": "def create_issue_comment(self, body):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/issues/:number/comments `_\n :param body: string\n :rtype: :class:`github.IssueComment.IssueComment`\n \"\"\"\n assert isinstance(body, (str, unicode)), body\n post_parameters = {\n \"body\": body,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.issue_url + \"/comments\",\n input=post_parameters\n )\n return github.IssueComment.IssueComment(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_review(self, commit=github.GithubObject.NotSet, body=None, event=github.GithubObject.NotSet, comments=github.GithubObject.NotSet):\n assert commit is github.GithubObject.NotSet or isinstance(commit, github.Commit.Commit), commit\n assert isinstance(body, str), body\n assert event is github.GithubObject.NotSet or isinstance(event, str), event\n assert comments is github.GithubObject.NotSet or isinstance(comments, list), comments\n post_parameters = dict()\n if commit is not github.GithubObject.NotSet:\n post_parameters['commit_id'] = commit.sha\n post_parameters['body'] = body\n post_parameters['event'] = 'COMMENT' if event == github.GithubObject.NotSet else event\n if comments is github.GithubObject.NotSet:\n post_parameters['comments'] = []\n else:\n post_parameters['comments'] = comments\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/reviews\",\n input=post_parameters\n )\n self._useAttributes(data)\n return github.PullRequestReview.PullRequestReview(self._requester, headers, data, completed=True)", "response": "Creates a new review for the specified commit."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef delete_review_request(self, reviewers=github.GithubObject.NotSet, team_reviewers=github.GithubObject.NotSet):\n post_parameters = dict()\n if reviewers is not github.GithubObject.NotSet:\n assert all(isinstance(element, (str, unicode)) for element in reviewers), reviewers\n post_parameters[\"reviewers\"] = reviewers\n if team_reviewers is not github.GithubObject.NotSet:\n assert all(isinstance(element, (str, unicode)) for element in team_reviewers), team_reviewers\n post_parameters[\"team_reviewers\"] = team_reviewers\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.url + \"/requested_reviewers\",\n input=post_parameters\n )", "response": "Deletes the specified review request for the specified set of reviewers and team_reviewers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef edit(self, title=github.GithubObject.NotSet, body=github.GithubObject.NotSet, state=github.GithubObject.NotSet, base=github.GithubObject.NotSet):\n assert title is github.GithubObject.NotSet or isinstance(title, (str, unicode)), title\n assert body is github.GithubObject.NotSet or isinstance(body, (str, unicode)), body\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert base is github.GithubObject.NotSet or isinstance(base, (str, unicode)), base\n post_parameters = dict()\n if title is not github.GithubObject.NotSet:\n post_parameters[\"title\"] = title\n if body is not github.GithubObject.NotSet:\n post_parameters[\"body\"] = body\n if state is not github.GithubObject.NotSet:\n post_parameters[\"state\"] = state\n if base is not github.GithubObject.NotSet:\n post_parameters[\"base\"] = base\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Edits the object with the given title body state and base."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_review_comment(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self._parentUrl(self.url) + \"/comments/\" + str(id)\n )\n return github.PullRequestComment.PullRequestComment(self._requester, headers, data, completed=True)", "response": "returns a github. PullRequestComment object for the given ID"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a paginated list of github. PullRequestComment objects for the given issue ID since the given date.", "response": "def get_review_comments(self, since=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number/comments `_\n :param since: datetime.datetime format YYYY-MM-DDTHH:MM:SSZ\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.PullRequestComment.PullRequestComment`\n \"\"\"\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n url_parameters = dict()\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n return github.PaginatedList.PaginatedList(\n github.PullRequestComment.PullRequestComment,\n self._requester,\n self.url + \"/comments\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a paginated list of comments for a single review", "response": "def get_single_review_comments(self, id):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number/review/:id/comments `_\n :param id: integer\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.PullRequestComment.PullRequestComment`\n \"\"\"\n assert isinstance(id, (int, long)), id\n return github.PaginatedList.PaginatedList(\n github.PullRequestComment.PullRequestComment,\n self._requester,\n self.url + \"/reviews/\" + str(id) + \"/comments\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_commits(self):\n return github.PaginatedList.PaginatedList(\n github.Commit.Commit,\n self._requester,\n self.url + \"/commits\",\n None\n )", "response": "Returns a paginated list of all commits for this branch."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_files(self):\n return github.PaginatedList.PaginatedList(\n github.File.File,\n self._requester,\n self.url + \"/files\",\n None\n )", "response": "Returns a paginated list of files for this issue."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a github. IssueComment object for the given issue comment id", "response": "def get_issue_comment(self, id):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/issues/comments/:id `_\n :param id: integer\n :rtype: :class:`github.IssueComment.IssueComment`\n \"\"\"\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self._parentUrl(self.issue_url) + \"/comments/\" + str(id)\n )\n return github.IssueComment.IssueComment(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_issue_comments(self):\n return github.PaginatedList.PaginatedList(\n github.IssueComment.IssueComment,\n self._requester,\n self.issue_url + \"/comments\",\n None\n )", "response": "Returns a paginated list of all comments for this issue."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a github. PullRequestReview. PullRequestReview object for the given ID", "response": "def get_review(self, id):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number/reviews/:id `_\n :param id: integer\n :rtype: :class:`github.PullRequestReview.PullRequestReview`\n \"\"\"\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/reviews/\" + str(id),\n )\n return github.PullRequestReview.PullRequestReview(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_reviews(self):\n return github.PaginatedList.PaginatedList(\n github.PullRequestReview.PullRequestReview,\n self._requester,\n self.url + \"/reviews\",\n None,\n )", "response": "Returns a paginated list of all reviews for the current branch."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a tuple of github. NamedUser. NamedUser and github. Team. Team objects", "response": "def get_review_requests(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number/requested_reviewers `_\n :rtype: tuple of :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser` and of :class:`github.PaginatedList.PaginatedList` of :class:`github.Team.Team`\n \"\"\"\n return (\n github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/requested_reviewers\",\n None,\n list_item='users'\n ),\n github.PaginatedList.PaginatedList(\n github.Team.Team,\n self._requester,\n self.url + \"/requested_reviewers\",\n None,\n list_item='teams'\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_labels(self):\n return github.PaginatedList.PaginatedList(\n github.Label.Label,\n self._requester,\n self.issue_url + \"/labels\",\n None\n )", "response": "Returns a paginated list of all labels for the issue."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_from_labels(self, label):\n assert isinstance(label, (github.Label.Label, str, unicode)), label\n if isinstance(label, github.Label.Label):\n label = label._identity\n else:\n label = urllib.quote(label)\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.issue_url + \"/labels/\" + label\n )", "response": "Removes the issue from the labels."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the labels of the current issue.", "response": "def set_labels(self, *labels):\n \"\"\"\n :calls: `PUT /repos/:owner/:repo/issues/:number/labels `_\n :param labels: list of :class:`github.Label.Label` or strings\n :rtype: None\n \"\"\"\n assert all(isinstance(element, (github.Label.Label, str, unicode)) for element in labels), labels\n post_parameters = [label.name if isinstance(label, github.Label.Label) else label for label in labels]\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.issue_url + \"/labels\",\n input=post_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns true if the branch is merged", "response": "def is_merged(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number/merge `_\n :rtype: bool\n \"\"\"\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/merge\"\n )\n return status == 204"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a merge request for the specified branch.", "response": "def merge(self, commit_message=github.GithubObject.NotSet, commit_title=github.GithubObject.NotSet, merge_method=github.GithubObject.NotSet, sha=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PUT /repos/:owner/:repo/pulls/:number/merge `_\n :param commit_message: string\n :rtype: :class:`github.PullRequestMergeStatus.PullRequestMergeStatus`\n \"\"\"\n assert commit_message is github.GithubObject.NotSet or isinstance(commit_message, (str, unicode)), commit_message\n assert commit_title is github.GithubObject.NotSet or isinstance(commit_title, (str, unicode)), commit_title\n assert merge_method is github.GithubObject.NotSet or isinstance(merge_method, (str, unicode)), merge_method\n assert sha is github.GithubObject.NotSet or isinstance(sha, (str, unicode)), sha\n post_parameters = dict()\n if commit_message is not github.GithubObject.NotSet:\n post_parameters[\"commit_message\"] = commit_message\n if commit_title is not github.GithubObject.NotSet:\n post_parameters[\"commit_title\"] = commit_title\n if merge_method is not github.GithubObject.NotSet:\n post_parameters[\"merge_method\"] = merge_method\n if sha is not github.GithubObject.NotSet:\n post_parameters[\"sha\"] = sha\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/merge\",\n input=post_parameters\n )\n return github.PullRequestMergeStatus.PullRequestMergeStatus(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef repository(self):\n self._completeIfNotSet(self._repository)\n if self._repository is github.GithubObject.NotSet:\n # The repository was not set automatically, so it must be looked up by url.\n repo_url = \"/\".join(self.url.split(\"/\")[:-2])\n self._repository = github.GithubObject._ValuedAttribute(github.Repository.Repository(self._requester, self._headers, {'url': repo_url}, completed=False))\n return self._repository.value", "response": "Returns the repository attribute of the object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef as_pull_request(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/pulls/\".join(self.url.rsplit(\"/issues/\", 1))\n )\n return github.PullRequest.PullRequest(self._requester, headers, data, completed=True)", "response": "returns a github. PullRequest. PullRequest object for this issue"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_to_assignees(self, *assignees):\n assert all(isinstance(element, (github.NamedUser.NamedUser, str, unicode)) for element in assignees), assignees\n post_parameters = {\"assignees\": [assignee.login if isinstance(assignee, github.NamedUser.NamedUser) else assignee for assignee in assignees]}\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/assignees\",\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Adds the specified user to the set of assignees."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_comment(self, body):\n assert isinstance(body, (str, unicode)), body\n post_parameters = {\n \"body\": body,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/comments\",\n input=post_parameters\n )\n return github.IssueComment.IssueComment(self._requester, headers, data, completed=True)", "response": "Creates a comment for the current issue."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the issue metadata.", "response": "def edit(self, title=github.GithubObject.NotSet, body=github.GithubObject.NotSet, assignee=github.GithubObject.NotSet, state=github.GithubObject.NotSet, milestone=github.GithubObject.NotSet, labels=github.GithubObject.NotSet, assignees=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /repos/:owner/:repo/issues/:number `_\n :param title: string\n :param body: string\n :param assignee: string or :class:`github.NamedUser.NamedUser` or None\n :param assignees: list (of string or :class:`github.NamedUser.NamedUser`)\n :param state: string\n :param milestone: :class:`github.Milestone.Milestone` or None\n :param labels: list of string\n :rtype: None\n \"\"\"\n assert title is github.GithubObject.NotSet or isinstance(title, (str, unicode)), title\n assert body is github.GithubObject.NotSet or isinstance(body, (str, unicode)), body\n assert assignee is github.GithubObject.NotSet or assignee is None or isinstance(assignee, github.NamedUser.NamedUser) or isinstance(assignee, (str, unicode)), assignee\n assert assignees is github.GithubObject.NotSet or all(isinstance(element, github.NamedUser.NamedUser) or isinstance(element, (str, unicode)) for element in assignees), assignees\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert milestone is github.GithubObject.NotSet or milestone is None or isinstance(milestone, github.Milestone.Milestone), milestone\n assert labels is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in labels), labels\n post_parameters = dict()\n if title is not github.GithubObject.NotSet:\n post_parameters[\"title\"] = title\n if body is not github.GithubObject.NotSet:\n post_parameters[\"body\"] = body\n if assignee is not github.GithubObject.NotSet:\n if isinstance(assignee, (str, unicode)):\n post_parameters[\"assignee\"] = assignee\n else:\n post_parameters[\"assignee\"] = assignee._identity if assignee else ''\n if assignees is not github.GithubObject.NotSet:\n post_parameters[\"assignees\"] = [element._identity if isinstance(element, github.NamedUser.NamedUser) else element for element in assignees]\n if state is not github.GithubObject.NotSet:\n post_parameters[\"state\"] = state\n if milestone is not github.GithubObject.NotSet:\n post_parameters[\"milestone\"] = milestone._identity if milestone else ''\n if labels is not github.GithubObject.NotSet:\n post_parameters[\"labels\"] = labels\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_comments(self, since=github.GithubObject.NotSet):\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n url_parameters = dict()\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n return github.PaginatedList.PaginatedList(\n github.IssueComment.IssueComment,\n self._requester,\n self.url + \"/comments\",\n url_parameters\n )", "response": "returns a paginated list of all comments for the issue with the given since date."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a paginated list of all events for this issue.", "response": "def get_events(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/issues/:issue_number/events `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.IssueEvent.IssueEvent`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.IssueEvent.IssueEvent,\n self._requester,\n self.url + \"/events\",\n None,\n headers={'Accept': Consts.mediaTypeLockReasonPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_labels(self):\n return github.PaginatedList.PaginatedList(\n github.Label.Label,\n self._requester,\n self.url + \"/labels\",\n None\n )", "response": "Returns a paginated list of all labels for the current issue."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a paginated list of all reactions for the issue.", "response": "def get_reactions(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/issues/:number/reactions `_\n :return: :class: :class:`github.PaginatedList.PaginatedList` of :class:`github.Reaction.Reaction`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Reaction.Reaction,\n self._requester,\n self.url + \"/reactions\",\n None,\n headers={'Accept': Consts.mediaTypeReactionsPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_reaction(self, reaction_type):\n assert isinstance(reaction_type, (str, unicode)), \"reaction type should be a string\"\n assert reaction_type in [\"+1\", \"-1\", \"laugh\", \"confused\", \"heart\", \"hooray\"], \\\n \"Invalid reaction type (https://developer.github.com/v3/reactions/#reaction-types)\"\n\n post_parameters = {\n \"content\": reaction_type,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/reactions\",\n input=post_parameters,\n headers={'Accept': Consts.mediaTypeReactionsPreview}\n )\n return github.Reaction.Reaction(self._requester, headers, data, completed=True)", "response": "Creates a new reaction for the given type of issue."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef edit(self, title, state=github.GithubObject.NotSet, description=github.GithubObject.NotSet, due_on=github.GithubObject.NotSet):\n assert isinstance(title, (str, unicode)), title\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert due_on is github.GithubObject.NotSet or isinstance(due_on, datetime.date), due_on\n post_parameters = {\n \"title\": title,\n }\n if state is not github.GithubObject.NotSet:\n post_parameters[\"state\"] = state\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if due_on is not github.GithubObject.NotSet:\n post_parameters[\"due_on\"] = due_on.strftime(\"%Y-%m-%d\")\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Edits the content of the object with the given title state description and due_on."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a new entry to the public_members list.", "response": "def add_to_public_members(self, public_member):\n \"\"\"\n :calls: `PUT /orgs/:org/public_members/:user `_\n :param public_member: :class:`github.NamedUser.NamedUser`\n :rtype: None\n \"\"\"\n assert isinstance(public_member, github.NamedUser.NamedUser), public_member\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/public_members/\" + public_member._identity\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_fork(self, repo):\n assert isinstance(repo, github.Repository.Repository), repo\n url_parameters = {\n \"org\": self.login,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/repos/\" + repo.owner.login + \"/\" + repo.name + \"/forks\",\n parameters=url_parameters\n )\n return github.Repository.Repository(self._requester, headers, data, completed=True)", "response": "Creates a new branch for the given repository."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_hook(self, name, config, events=github.GithubObject.NotSet, active=github.GithubObject.NotSet):\n assert isinstance(name, (str, unicode)), name\n assert isinstance(config, dict), config\n assert events is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in events), events\n assert active is github.GithubObject.NotSet or isinstance(active, bool), active\n post_parameters = {\n \"name\": name,\n \"config\": config,\n }\n if events is not github.GithubObject.NotSet:\n post_parameters[\"events\"] = events\n if active is not github.GithubObject.NotSet:\n post_parameters[\"active\"] = active\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/hooks\",\n input=post_parameters\n )\n return github.Hook.Hook(self._requester, headers, data, completed=True)", "response": "Creates a new github. Hook. Hook object."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a new team with the specified name and repositories.", "response": "def create_team(self, name, repo_names=github.GithubObject.NotSet, permission=github.GithubObject.NotSet, privacy=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /orgs/:org/teams `_\n :param name: string\n :param repo_names: list of :class:`github.Repository.Repository`\n :param permission: string\n :param privacy: string\n :rtype: :class:`github.Team.Team`\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n assert repo_names is github.GithubObject.NotSet or all(isinstance(element, github.Repository.Repository) for element in repo_names), repo_names\n assert permission is github.GithubObject.NotSet or isinstance(permission, (str, unicode)), permission\n assert privacy is github.GithubObject.NotSet or isinstance(privacy, (str, unicode)), privacy\n post_parameters = {\n \"name\": name,\n }\n if repo_names is not github.GithubObject.NotSet:\n post_parameters[\"repo_names\"] = [element._identity for element in repo_names]\n if permission is not github.GithubObject.NotSet:\n post_parameters[\"permission\"] = permission\n if privacy is not github.GithubObject.NotSet:\n post_parameters['privacy'] = privacy\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/teams\",\n input=post_parameters\n )\n return github.Team.Team(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef delete_hook(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.url + \"/hooks/\" + str(id)\n )", "response": "Deletes a specific hook from the specified tenant."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef edit(self, billing_email=github.GithubObject.NotSet, blog=github.GithubObject.NotSet, company=github.GithubObject.NotSet, description=github.GithubObject.NotSet, email=github.GithubObject.NotSet, location=github.GithubObject.NotSet, name=github.GithubObject.NotSet):\n assert billing_email is github.GithubObject.NotSet or isinstance(billing_email, (str, unicode)), billing_email\n assert blog is github.GithubObject.NotSet or isinstance(blog, (str, unicode)), blog\n assert company is github.GithubObject.NotSet or isinstance(company, (str, unicode)), company\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert email is github.GithubObject.NotSet or isinstance(email, (str, unicode)), email\n assert location is github.GithubObject.NotSet or isinstance(location, (str, unicode)), location\n assert name is github.GithubObject.NotSet or isinstance(name, (str, unicode)), name\n post_parameters = dict()\n if billing_email is not github.GithubObject.NotSet:\n post_parameters[\"billing_email\"] = billing_email\n if blog is not github.GithubObject.NotSet:\n post_parameters[\"blog\"] = blog\n if company is not github.GithubObject.NotSet:\n post_parameters[\"company\"] = company\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if email is not github.GithubObject.NotSet:\n post_parameters[\"email\"] = email\n if location is not github.GithubObject.NotSet:\n post_parameters[\"location\"] = location\n if name is not github.GithubObject.NotSet:\n post_parameters[\"name\"] = name\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Updates the related object with the given parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_events(self):\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n self.url + \"/events\",\n None\n )", "response": "Returns a paginated list of events for the current user."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_hook(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/hooks/\" + str(id)\n )\n return github.Hook.Hook(self._requester, headers, data, completed=True)", "response": "returns a github. Hook. Hook object for the given ID"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_hooks(self):\n return github.PaginatedList.PaginatedList(\n github.Hook.Hook,\n self._requester,\n self.url + \"/hooks\",\n None\n )", "response": "Returns a paginated list of all hooks owned by the user."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_members(self, filter_=github.GithubObject.NotSet,\n role=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /orgs/:org/members `_\n :param filter_: string\n :param role: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n assert (filter_ is github.GithubObject.NotSet or\n isinstance(filter_, (str, unicode))), filter_\n assert (role is github.GithubObject.NotSet or\n isinstance(role, (str, unicode))), role\n\n url_parameters = {}\n if filter_ is not github.GithubObject.NotSet:\n url_parameters[\"filter\"] = filter_\n if role is not github.GithubObject.NotSet:\n url_parameters[\"role\"] = role\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/members\",\n url_parameters\n )", "response": "Returns a paginated list of members of the specified object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a paginated list of all projects in the organization.", "response": "def get_projects(self, state=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /orgs/:org/projects `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Project.Project`\n :param state: string\n \"\"\"\n \n url_parameters = dict()\n if state is not github.GithubObject.NotSet:\n url_parameters[\"state\"] = state\n \n return github.PaginatedList.PaginatedList(\n github.Project.Project,\n self._requester,\n self.url + \"/projects\",\n url_parameters,\n {\"Accept\": Consts.mediaTypeProjectsPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a paginated list of public members of the user.", "response": "def get_public_members(self):\n \"\"\"\n :calls: `GET /orgs/:org/public_members `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/public_members\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of the outside collaborators of the user.", "response": "def get_outside_collaborators(self, filter_=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /orgs/:org/outside_collaborators `_\n :param filter_: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n assert (filter_ is github.GithubObject.NotSet or\n isinstance(filter_, (str, unicode))), filter_\n\n url_parameters = {}\n if filter_ is not github.GithubObject.NotSet:\n url_parameters[\"filter\"] = filter_\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/outside_collaborators\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_outside_collaborator(self, collaborator):\n assert isinstance(collaborator, github.NamedUser.NamedUser), collaborator\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.url + \"/outside_collaborators/\" + collaborator._identity\n )", "response": "Removes the specified collaborator from the user s outside Collaborator set."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert_to_outside_collaborator(self, member):\n assert isinstance(member, github.NamedUser.NamedUser), member\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/outside_collaborators/\" + member._identity\n )", "response": "Converts a user to the outside Collaborator."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_repo(self, name):\n assert isinstance(name, (str, unicode)), name\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/repos/\" + self.login + \"/\" + name\n )\n return github.Repository.Repository(self._requester, headers, data, completed=True)", "response": "returns a github. Repository. Repository object for the given name"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_repos(self, type=github.GithubObject.NotSet):\n assert type is github.GithubObject.NotSet or isinstance(type, (str, unicode)), type\n url_parameters = dict()\n if type is not github.GithubObject.NotSet:\n url_parameters[\"type\"] = type\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n self.url + \"/repos\",\n url_parameters\n )", "response": "returns a paginated list of repositories for the current user."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_team(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/teams/\" + str(id)\n )\n return github.Team.Team(self._requester, headers, data, completed=True)", "response": "Returns a single team with the given ID."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of all teams for the current user.", "response": "def get_teams(self):\n \"\"\"\n :calls: `GET /orgs/:org/teams `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Team.Team`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Team.Team,\n self._requester,\n self.url + \"/teams\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef invite_user(self, user=github.GithubObject.NotSet, email=github.GithubObject.NotSet, role=github.GithubObject.NotSet, teams=github.GithubObject.NotSet):\n assert user is github.GithubObject.NotSet or isinstance(user, github.NamedUser.NamedUser), user\n assert email is github.GithubObject.NotSet or isinstance(email, (str, unicode)), email\n assert (email is github.GithubObject.NotSet) ^ (user is github.GithubObject.NotSet), \"specify only one of email or user\"\n parameters = {}\n if user is not github.GithubObject.NotSet:\n parameters[\"invitee_id\"] = user.id\n elif email is not github.GithubObject.NotSet:\n parameters[\"email\"] = email\n if role is not github.GithubObject.NotSet:\n assert isinstance(role, (str, unicode)), role\n assert role in ['admin', 'direct_member', 'billing_manager']\n parameters[\"role\"] = role\n if teams is not github.GithubObject.NotSet:\n assert all(isinstance(team, github.Team.Team) for team in teams)\n parameters[\"team_ids\"] = [t.id for t in teams]\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/invitations\",\n headers={'Accept': Consts.mediaTypeOrganizationInvitationPreview},\n input=parameters\n )", "response": "Invites a user to the organization."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef has_in_public_members(self, public_member):\n assert isinstance(public_member, github.NamedUser.NamedUser), public_member\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/public_members/\" + public_member._identity\n )\n return status == 204", "response": "Returns true if the user has access to the specified public_members."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_migrations(self):\n return github.PaginatedList.PaginatedList(\n github.Migration.Migration,\n self._requester,\n \"/orgs/\" + self.login + \"/migrations\",\n None,\n headers={\n \"Accept\": Consts.mediaTypeMigrationPreview\n }\n )", "response": "Returns a paginated list of all migrations for the user."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_user_push_restrictions(self):\n if self._user_push_restrictions is github.GithubObject.NotSet:\n return None\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self._user_push_restrictions,\n None\n )", "response": "Returns a list of github. PaginatedList of github. NamedUser objects."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_team_push_restrictions(self):\n if self._team_push_restrictions is github.GithubObject.NotSet:\n return None\n return github.PaginatedList.PaginatedList(\n github.Team.Team,\n self._requester,\n self._team_push_restrictions,\n None\n )", "response": "Returns a list of github. PaginatedList of github. Team. Team objects."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef name(self, value):\n self._completeIfNotSet(self._name)\n self._name.value = value", "response": "Sets the name of the current object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef label(self, value):\n self._completeIfNotSet(self._label)\n self._label.value = value", "response": "Sets the label of the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_asset(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.url\n )\n return True", "response": "Delete asset from the release."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_asset(self, name, label=\"\"):\n assert isinstance(name, (str, unicode)), name\n assert isinstance(label, (str, unicode)), label\n post_parameters = {\n \"name\": name,\n \"label\": label\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n return GitReleaseAsset(self._requester, headers, data, completed=True)", "response": "Update the asset metadata."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck and update the object with conditional request.", "response": "def update(self):\n '''\n Check and update the object with conditional request\n :rtype: Boolean value indicating whether the object is changed\n '''\n conditionalRequestHeader = dict()\n if self.etag is not None:\n conditionalRequestHeader[Consts.REQ_IF_NONE_MATCH] = self.etag\n if self.last_modified is not None:\n conditionalRequestHeader[Consts.REQ_IF_MODIFIED_SINCE] = self.last_modified\n\n status, responseHeaders, output = self._requester.requestJson(\n \"GET\",\n self._url.value,\n headers=conditionalRequestHeader\n )\n if status == 304:\n return False\n else:\n headers, data = self._requester._Requester__check(status, responseHeaders, output)\n self._storeAndUseAttributes(headers, data)\n self.__completed = True\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dismiss(self, message):\n assert isinstance(message, (str, unicode)), message\n post_parameters = {'message': message}\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.pull_request_url + \"/reviews/%s/dismissals\" % self.id,\n input=post_parameters\n )", "response": "dismisses the issue for this issue"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_to_collaborators(self, collaborator, permission=github.GithubObject.NotSet):\n assert isinstance(collaborator, github.NamedUser.NamedUser) or isinstance(collaborator, (str, unicode)), collaborator\n assert permission is github.GithubObject.NotSet or isinstance(permission, (str, unicode)), permission\n\n if isinstance(collaborator, github.NamedUser.NamedUser):\n collaborator = collaborator._identity\n\n if permission is not github.GithubObject.NotSet:\n put_parameters = {'permission': permission}\n else:\n put_parameters = None\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/collaborators/\" + collaborator,\n input=put_parameters\n )\n # return an invitation object if there's data returned by the API. If data is empty\n # there's a pending invitation for the given user.\n return github.Invitation.Invitation(self._requester, headers, data, completed=True) if \\\n data is not None else None", "response": "Adds a new collaborator to the set of collaborators."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_collaborator_permission(self, collaborator):\n assert isinstance(collaborator, github.NamedUser.NamedUser) or isinstance(collaborator, (str, unicode)), collaborator\n if isinstance(collaborator, github.NamedUser.NamedUser):\n collaborator = collaborator._identity\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/collaborators/\" + collaborator + \"/permission\",\n )\n return data[\"permission\"]", "response": "returns the permission of a collaborator"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a new comparison object for the base and head of the current branch.", "response": "def compare(self, base, head):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/compare/:base...:head `_\n :param base: string\n :param head: string\n :rtype: :class:`github.Comparison.Comparison`\n \"\"\"\n assert isinstance(base, (str, unicode)), base\n assert isinstance(head, (str, unicode)), head\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/compare/\" + base + \"...\" + head\n )\n return github.Comparison.Comparison(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_git_blob(self, content, encoding):\n assert isinstance(content, (str, unicode)), content\n assert isinstance(encoding, (str, unicode)), encoding\n post_parameters = {\n \"content\": content,\n \"encoding\": encoding,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/git/blobs\",\n input=post_parameters\n )\n return github.GitBlob.GitBlob(self._requester, headers, data, completed=True)", "response": "Creates a git blob for the given content and encoding."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a git commit for the given tree and parents.", "response": "def create_git_commit(self, message, tree, parents, author=github.GithubObject.NotSet, committer=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/git/commits `_\n :param message: string\n :param tree: :class:`github.GitTree.GitTree`\n :param parents: list of :class:`github.GitCommit.GitCommit`\n :param author: :class:`github.InputGitAuthor.InputGitAuthor`\n :param committer: :class:`github.InputGitAuthor.InputGitAuthor`\n :rtype: :class:`github.GitCommit.GitCommit`\n \"\"\"\n assert isinstance(message, (str, unicode)), message\n assert isinstance(tree, github.GitTree.GitTree), tree\n assert all(isinstance(element, github.GitCommit.GitCommit) for element in parents), parents\n assert author is github.GithubObject.NotSet or isinstance(author, github.InputGitAuthor), author\n assert committer is github.GithubObject.NotSet or isinstance(committer, github.InputGitAuthor), committer\n post_parameters = {\n \"message\": message,\n \"tree\": tree._identity,\n \"parents\": [element._identity for element in parents],\n }\n if author is not github.GithubObject.NotSet:\n post_parameters[\"author\"] = author._identity\n if committer is not github.GithubObject.NotSet:\n post_parameters[\"committer\"] = committer._identity\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/git/commits\",\n input=post_parameters\n )\n return github.GitCommit.GitCommit(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a git ref for the given ref and sha.", "response": "def create_git_ref(self, ref, sha):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/git/refs `_\n :param ref: string\n :param sha: string\n :rtype: :class:`github.GitRef.GitRef`\n \"\"\"\n assert isinstance(ref, (str, unicode)), ref\n assert isinstance(sha, (str, unicode)), sha\n post_parameters = {\n \"ref\": ref,\n \"sha\": sha,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/git/refs\",\n input=post_parameters\n )\n return github.GitRef.GitRef(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_git_release(self, tag, name, message, draft=False, prerelease=False, target_commitish=github.GithubObject.NotSet):\n assert isinstance(tag, (str, unicode)), tag\n assert isinstance(name, (str, unicode)), name\n assert isinstance(message, (str, unicode)), message\n assert isinstance(draft, bool), draft\n assert isinstance(prerelease, bool), prerelease\n assert target_commitish is github.GithubObject.NotSet or isinstance(target_commitish, (str, unicode, github.Branch.Branch, github.Commit.Commit, github.GitCommit.GitCommit)), target_commitish\n post_parameters = {\n \"tag_name\": tag,\n \"name\": name,\n \"body\": message,\n \"draft\": draft,\n \"prerelease\": prerelease,\n }\n if isinstance(target_commitish, (str, unicode)):\n post_parameters[\"target_commitish\"] = target_commitish\n elif isinstance(target_commitish, github.Branch.Branch):\n post_parameters[\"target_commitish\"] = target_commitish.name\n elif isinstance(target_commitish, (github.Commit.Commit, github.GitCommit.GitCommit)):\n post_parameters[\"target_commitish\"] = target_commitish.sha\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/releases\",\n input=post_parameters\n )\n return github.GitRelease.GitRelease(self._requester, headers, data, completed=True)", "response": "Creates a git release for the given tag name and message."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a git tag for the given object and type.", "response": "def create_git_tag(self, tag, message, object, type, tagger=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/git/tags `_\n :param tag: string\n :param message: string\n :param object: string\n :param type: string\n :param tagger: :class:`github.InputGitAuthor.InputGitAuthor`\n :rtype: :class:`github.GitTag.GitTag`\n \"\"\"\n assert isinstance(tag, (str, unicode)), tag\n assert isinstance(message, (str, unicode)), message\n assert isinstance(object, (str, unicode)), object\n assert isinstance(type, (str, unicode)), type\n assert tagger is github.GithubObject.NotSet or isinstance(tagger, github.InputGitAuthor), tagger\n post_parameters = {\n \"tag\": tag,\n \"message\": message,\n \"object\": object,\n \"type\": type,\n }\n if tagger is not github.GithubObject.NotSet:\n post_parameters[\"tagger\"] = tagger._identity\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/git/tags\",\n input=post_parameters\n )\n return github.GitTag.GitTag(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new GitTree with the specified tree and base_tree.", "response": "def create_git_tree(self, tree, base_tree=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/git/trees `_\n :param tree: list of :class:`github.InputGitTreeElement.InputGitTreeElement`\n :param base_tree: :class:`github.GitTree.GitTree`\n :rtype: :class:`github.GitTree.GitTree`\n \"\"\"\n assert all(isinstance(element, github.InputGitTreeElement) for element in tree), tree\n assert base_tree is github.GithubObject.NotSet or isinstance(base_tree, github.GitTree.GitTree), base_tree\n post_parameters = {\n \"tree\": [element._identity for element in tree],\n }\n if base_tree is not github.GithubObject.NotSet:\n post_parameters[\"base_tree\"] = base_tree._identity\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/git/trees\",\n input=post_parameters\n )\n return github.GitTree.GitTree(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new issue with the given title body and assignee and labels.", "response": "def create_issue(self, title, body=github.GithubObject.NotSet, assignee=github.GithubObject.NotSet, milestone=github.GithubObject.NotSet, labels=github.GithubObject.NotSet, assignees=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/issues `_\n :param title: string\n :param body: string\n :param assignee: string or :class:`github.NamedUser.NamedUser`\n :param assignees: list (of string or :class:`github.NamedUser.NamedUser`)\n :param milestone: :class:`github.Milestone.Milestone`\n :param labels: list of :class:`github.Label.Label`\n :rtype: :class:`github.Issue.Issue`\n \"\"\"\n assert isinstance(title, (str, unicode)), title\n assert body is github.GithubObject.NotSet or isinstance(body, (str, unicode)), body\n assert assignee is github.GithubObject.NotSet or isinstance(assignee, github.NamedUser.NamedUser) or isinstance(assignee, (str, unicode)), assignee\n assert assignees is github.GithubObject.NotSet or all(isinstance(element, github.NamedUser.NamedUser) or isinstance(element, (str, unicode)) for element in assignees), assignees\n assert milestone is github.GithubObject.NotSet or isinstance(milestone, github.Milestone.Milestone), milestone\n assert labels is github.GithubObject.NotSet or all(isinstance(element, github.Label.Label) or isinstance(element, (str, unicode)) for element in labels), labels\n\n post_parameters = {\n \"title\": title,\n }\n if body is not github.GithubObject.NotSet:\n post_parameters[\"body\"] = body\n if assignee is not github.GithubObject.NotSet:\n if isinstance(assignee, (str, unicode)):\n post_parameters[\"assignee\"] = assignee\n else:\n post_parameters[\"assignee\"] = assignee._identity\n if assignees is not github.GithubObject.NotSet:\n post_parameters[\"assignees\"] = [element._identity if isinstance(element, github.NamedUser.NamedUser) else element for element in assignees]\n if milestone is not github.GithubObject.NotSet:\n post_parameters[\"milestone\"] = milestone._identity\n if labels is not github.GithubObject.NotSet:\n post_parameters[\"labels\"] = [element.name if isinstance(element, github.Label.Label) else element for element in labels]\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/issues\",\n input=post_parameters\n )\n return github.Issue.Issue(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new repository key for the given title and key.", "response": "def create_key(self, title, key, read_only=False):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/keys `_\n :param title: string\n :param key: string\n :param read_only: bool\n :rtype: :class:`github.RepositoryKey.RepositoryKey`\n \"\"\"\n assert isinstance(title, (str, unicode)), title\n assert isinstance(key, (str, unicode)), key\n assert isinstance(read_only, bool), read_only\n post_parameters = {\n \"title\": title,\n \"key\": key,\n \"read_only\": read_only,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/keys\",\n input=post_parameters\n )\n return github.RepositoryKey.RepositoryKey(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_label(self, name, color, description=github.GithubObject.NotSet):\n assert isinstance(name, (str, unicode)), name\n assert isinstance(color, (str, unicode)), color\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n post_parameters = {\n \"name\": name,\n \"color\": color,\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/labels\",\n input=post_parameters,\n headers={'Accept': Consts.mediaTypeLabelDescriptionSearchPreview}\n )\n return github.Label.Label(self._requester, headers, data, completed=True)", "response": "Creates a new label for the given name color and description."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_milestone(self, title, state=github.GithubObject.NotSet, description=github.GithubObject.NotSet, due_on=github.GithubObject.NotSet):\n assert isinstance(title, (str, unicode)), title\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert due_on is github.GithubObject.NotSet or isinstance(due_on, (datetime.datetime, datetime.date)), due_on\n post_parameters = {\n \"title\": title,\n }\n if state is not github.GithubObject.NotSet:\n post_parameters[\"state\"] = state\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if due_on is not github.GithubObject.NotSet:\n if isinstance(due_on, datetime.date):\n post_parameters[\"due_on\"] = due_on.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n else:\n post_parameters[\"due_on\"] = due_on.isoformat()\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/milestones\",\n input=post_parameters\n )\n return github.Milestone.Milestone(self._requester, headers, data, completed=True)", "response": "Creates a new milestone with the given title state description and due_on."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new project with the given name and body", "response": "def create_project(self, name, body=github.GithubObject.NotSet):\n \"\"\"\n calls: `POST /repos/:owner/:repo/projects `_\n :param name: string\n :param body: string\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n assert body is github.GithubObject.NotSet or isinstance(body, (str, unicode)), body\n post_parameters = {\n \"name\": name,\n \"body\": body,\n }\n import_header = {\"Accept\": Consts.mediaTypeProjectsPreview}\n if body is not github.GithubObject.NotSet:\n post_parameters['body'] = body\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/projects\",\n headers=import_header,\n input=post_parameters\n )\n return github.Project.Project(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_pull(self, *args, **kwds):\n if len(args) + len(kwds) >= 4:\n return self.__create_pull_1(*args, **kwds)\n else:\n return self.__create_pull_2(*args, **kwds)", "response": "Creates a new pull request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_source_import(self, vcs, vcs_url, vcs_username=github.GithubObject.NotSet, vcs_password=github.GithubObject.NotSet):\n assert isinstance(vcs, (str, unicode)), vcs\n assert isinstance(vcs_url, (str, unicode)), vcs_url\n assert vcs_username is github.GithubObject.NotSet or isinstance(vcs_username, (str, unicode)), vcs_username\n assert vcs_password is github.GithubObject.NotSet or isinstance(vcs_password, (str, unicode)), vcs_password\n put_parameters = {\n \"vcs\": vcs,\n \"vcs_url\": vcs_url\n }\n\n if vcs_username is not github.GithubObject.NotSet:\n put_parameters[\"vcs_username\"] = vcs_username\n\n if vcs_password is not github.GithubObject.NotSet:\n put_parameters[\"vcs_password\"] = vcs_password\n\n import_header = {\"Accept\": Consts.mediaTypeImportPreview}\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/import\",\n headers=import_header,\n input=put_parameters\n )\n\n return github.SourceImport.SourceImport(self._requester, headers, data, completed=False)", "response": "Creates a new source import for the specified source."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef edit(self, name=None, description=github.GithubObject.NotSet, homepage=github.GithubObject.NotSet, private=github.GithubObject.NotSet, has_issues=github.GithubObject.NotSet, has_projects=github.GithubObject.NotSet, has_wiki=github.GithubObject.NotSet, has_downloads=github.GithubObject.NotSet, default_branch=github.GithubObject.NotSet, allow_squash_merge=github.GithubObject.NotSet, allow_merge_commit=github.GithubObject.NotSet, allow_rebase_merge=github.GithubObject.NotSet, archived=github.GithubObject.NotSet):\n if name is None:\n name = self.name\n assert isinstance(name, (str, unicode)), name\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert homepage is github.GithubObject.NotSet or isinstance(homepage, (str, unicode)), homepage\n assert private is github.GithubObject.NotSet or isinstance(private, bool), private\n assert has_issues is github.GithubObject.NotSet or isinstance(has_issues, bool), has_issues\n assert has_projects is github.GithubObject.NotSet or isinstance(has_projects, bool), has_projects\n assert has_wiki is github.GithubObject.NotSet or isinstance(has_wiki, bool), has_wiki\n assert has_downloads is github.GithubObject.NotSet or isinstance(has_downloads, bool), has_downloads\n assert default_branch is github.GithubObject.NotSet or isinstance(default_branch, (str, unicode)), default_branch\n assert allow_squash_merge is github.GithubObject.NotSet or isinstance(allow_squash_merge, bool), allow_squash_merge\n assert allow_merge_commit is github.GithubObject.NotSet or isinstance(allow_merge_commit, bool), allow_merge_commit\n assert allow_rebase_merge is github.GithubObject.NotSet or isinstance(allow_rebase_merge, bool), allow_rebase_merge\n assert archived is github.GithubObject.NotSet or (isinstance(archived, bool) and archived is True), archived\n post_parameters = {\n \"name\": name,\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if homepage is not github.GithubObject.NotSet:\n post_parameters[\"homepage\"] = homepage\n if private is not github.GithubObject.NotSet:\n post_parameters[\"private\"] = private\n if has_issues is not github.GithubObject.NotSet:\n post_parameters[\"has_issues\"] = has_issues\n if has_projects is not github.GithubObject.NotSet:\n post_parameters[\"has_projects\"] = has_projects\n if has_wiki is not github.GithubObject.NotSet:\n post_parameters[\"has_wiki\"] = has_wiki\n if has_downloads is not github.GithubObject.NotSet:\n post_parameters[\"has_downloads\"] = has_downloads\n if default_branch is not github.GithubObject.NotSet:\n post_parameters[\"default_branch\"] = default_branch\n if allow_squash_merge is not github.GithubObject.NotSet:\n post_parameters[\"allow_squash_merge\"] = allow_squash_merge\n if allow_merge_commit is not github.GithubObject.NotSet:\n post_parameters[\"allow_merge_commit\"] = allow_merge_commit\n if allow_rebase_merge is not github.GithubObject.NotSet:\n post_parameters[\"allow_rebase_merge\"] = allow_rebase_merge\n if archived is not github.GithubObject.NotSet:\n post_parameters[\"archived\"] = archived\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Updates the github object with the specified attributes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the link to the archive of the current object", "response": "def get_archive_link(self, archive_format, ref=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/:archive_format/:ref `_\n :param archive_format: string\n :param ref: string\n :rtype: string\n \"\"\"\n assert isinstance(archive_format, (str, unicode)), archive_format\n assert ref is github.GithubObject.NotSet or isinstance(ref, (str, unicode)), ref\n url = self.url + \"/\" + archive_format\n if ref is not github.GithubObject.NotSet:\n url += \"/\" + ref\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n url\n )\n return headers[\"location\"]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of all assignees of the issue.", "response": "def get_assignees(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/assignees `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/assignees\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a branch object for the branch with the given name", "response": "def get_branch(self, branch):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/branches/:branch `_\n :param branch: string\n :rtype: :class:`github.Branch.Branch`\n \"\"\"\n assert isinstance(branch, (str, unicode)), branch\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/branches/\" + branch\n )\n return github.Branch.Branch(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_branches(self):\n return github.PaginatedList.PaginatedList(\n github.Branch.Branch,\n self._requester,\n self.url + \"/branches\",\n None\n )", "response": "Returns a paginated list of branches for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_collaborators(self, affiliation=github.GithubObject.NotSet):\n\n url_parameters = dict()\n allowed_affiliations = ['outside', 'direct', 'all']\n if affiliation is not github.GithubObject.NotSet:\n assert isinstance(affiliation, str), affiliation\n assert affiliation in allowed_affiliations, \\\n 'Affiliation can be one of ' + ', '.join(allowed_affiliations)\n url_parameters['affiliation'] = affiliation\n\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/collaborators\",\n url_parameters\n )", "response": "returns a paginated list of collaborators for the current user"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_comments(self):\n return github.PaginatedList.PaginatedList(\n github.CommitComment.CommitComment,\n self._requester,\n self.url + \"/comments\",\n None\n )", "response": "Returns a paginated list of all comments for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a github. Commit. Commit object for the given sha", "response": "def get_commit(self, sha):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/commits/:sha `_\n :param sha: string\n :rtype: :class:`github.Commit.Commit`\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/commits/\" + sha\n )\n return github.Commit.Commit(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of commits for the given sha path since and until.", "response": "def get_commits(self, sha=github.GithubObject.NotSet, path=github.GithubObject.NotSet, since=github.GithubObject.NotSet, until=github.GithubObject.NotSet, author=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/commits `_\n :param sha: string\n :param path: string\n :param since: datetime.datetime\n :param until: datetime.datetime\n :param author: string or :class:`github.NamedUser.NamedUser` or :class:`github.AuthenticatedUser.AuthenticatedUser`\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Commit.Commit`\n \"\"\"\n assert sha is github.GithubObject.NotSet or isinstance(sha, (str, unicode)), sha\n assert path is github.GithubObject.NotSet or isinstance(path, (str, unicode)), path\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n assert until is github.GithubObject.NotSet or isinstance(until, datetime.datetime), until\n assert author is github.GithubObject.NotSet or isinstance(author, (str, unicode, github.NamedUser.NamedUser, github.AuthenticatedUser.AuthenticatedUser)), author\n url_parameters = dict()\n if sha is not github.GithubObject.NotSet:\n url_parameters[\"sha\"] = sha\n if path is not github.GithubObject.NotSet:\n url_parameters[\"path\"] = path\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n if until is not github.GithubObject.NotSet:\n url_parameters[\"until\"] = until.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n if author is not github.GithubObject.NotSet:\n if isinstance(author, (github.NamedUser.NamedUser, github.AuthenticatedUser.AuthenticatedUser)):\n url_parameters[\"author\"] = author.login\n else:\n url_parameters[\"author\"] = author\n return github.PaginatedList.PaginatedList(\n github.Commit.Commit,\n self._requester,\n self.url + \"/commits\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalls get_file_contents with path ref", "response": "def get_contents(self, path, ref=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/contents/:path `_\n :param path: string\n :param ref: string\n :rtype: :class:`github.ContentFile.ContentFile`\n \"\"\"\n return self.get_file_contents(path, ref)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_file_contents(self, path, ref=github.GithubObject.NotSet):\n assert isinstance(path, (str, unicode)), path\n assert ref is github.GithubObject.NotSet or isinstance(ref, (str, unicode)), ref\n url_parameters = dict()\n if ref is not github.GithubObject.NotSet:\n url_parameters[\"ref\"] = ref\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/contents/\" + urllib.quote(path),\n parameters=url_parameters\n )\n if isinstance(data, list):\n return [\n github.ContentFile.ContentFile(self._requester, headers, item, completed=False)\n for item in data\n ]\n return github.ContentFile.ContentFile(self._requester, headers, data, completed=True)", "response": "returns a list of github. ContentFile objects for a given path and ref"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of github. Referrer. Referrer objects for the top referrers of the current user.", "response": "def get_top_referrers(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/traffic/popular/referrers `_\n :rtype: :class:`list` of :class:`github.Referrer.Referrer`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/traffic/popular/referrers\"\n )\n if isinstance(data, list):\n return [\n github.Referrer.Referrer(self._requester, headers, item, completed=True)\n for item in data\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_top_paths(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/traffic/popular/paths\"\n )\n if isinstance(data, list):\n return [\n github.Path.Path(self._requester, headers, item, completed=True)\n for item in data\n ]", "response": "returns a list of github. Path. Path objects for all paths in the current node"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of github. View. View objects for the specified locale", "response": "def get_views_traffic(self, per=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/traffic/views `_\n :param per: string, must be one of day or week, day by default\n :rtype: None or list of :class:`github.View.View`\n \"\"\"\n assert per is github.GithubObject.NotSet or (isinstance(per, (str, unicode)) and (per == \"day\" or per == \"week\")), \"per must be day or week, day by default\"\n url_parameters = dict()\n if per is not github.GithubObject.NotSet:\n url_parameters[\"per\"] = per\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/traffic/views\",\n parameters=url_parameters\n )\n if (isinstance(data, dict)) and (\"views\" in data) and (isinstance(data[\"views\"], list)):\n data[\"views\"] = [\n github.View.View(self._requester, headers, item, completed=True)\n for item in data[\"views\"]\n ]\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a new file in this repository.", "response": "def create_file(self, path, message, content,\n branch=github.GithubObject.NotSet,\n committer=github.GithubObject.NotSet,\n author=github.GithubObject.NotSet):\n \"\"\"Create a file in this repository.\n\n :calls: `PUT /repos/:owner/:repo/contents/:path `_\n :param path: string, (required), path of the file in the repository\n :param message: string, (required), commit message\n :param content: string, (required), the actual data in the file\n :param branch: string, (optional), branch to create the commit on. Defaults to the default branch of the repository\n :param committer: InputGitAuthor, (optional), if no information is given the authenticated user's information will be used. You must specify both a name and email.\n :param author: InputGitAuthor, (optional), if omitted this will be filled in with committer information. If passed, you must specify both a name and email.\n :rtype: {\n 'content': :class:`ContentFile `:,\n 'commit': :class:`Commit `}\n \"\"\"\n assert isinstance(path, (str, unicode)), \\\n 'path must be str/unicode object'\n assert isinstance(message, (str, unicode)), \\\n 'message must be str/unicode object'\n assert isinstance(content, (str, unicode, bytes)), \\\n 'content must be a str/unicode object'\n assert branch is github.GithubObject.NotSet \\\n or isinstance(branch, (str, unicode)), \\\n 'branch must be a str/unicode object'\n assert author is github.GithubObject.NotSet \\\n or isinstance(author, github.InputGitAuthor), \\\n 'author must be a github.InputGitAuthor object'\n assert committer is github.GithubObject.NotSet \\\n or isinstance(committer, github.InputGitAuthor), \\\n 'committer must be a github.InputGitAuthor object'\n\n if atLeastPython3:\n if isinstance(content, str):\n content = content.encode('utf-8')\n content = b64encode(content).decode('utf-8')\n else:\n if isinstance(content, unicode):\n content = content.encode('utf-8')\n content = b64encode(content)\n put_parameters = {'message': message, 'content': content}\n\n if branch is not github.GithubObject.NotSet:\n put_parameters['branch'] = branch\n if author is not github.GithubObject.NotSet:\n put_parameters[\"author\"] = author._identity\n if committer is not github.GithubObject.NotSet:\n put_parameters[\"committer\"] = committer._identity\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/contents/\" + urllib.quote(path),\n input=put_parameters\n )\n\n return {'content': github.ContentFile.ContentFile(self._requester, headers, data[\"content\"], completed=False),\n 'commit': github.Commit.Commit(self._requester, headers, data[\"commit\"], completed=True)}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of github. ContentFile objects for a given path and ref", "response": "def get_dir_contents(self, path, ref=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/contents/:path `_\n :param path: string\n :param ref: string\n :rtype: list of :class:`github.ContentFile.ContentFile`\n \"\"\"\n assert isinstance(path, (str, unicode)), path\n assert ref is github.GithubObject.NotSet or isinstance(ref, (str, unicode)), ref\n url_parameters = dict()\n if ref is not github.GithubObject.NotSet:\n url_parameters[\"ref\"] = ref\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/contents/\" + urllib.quote(path),\n parameters=url_parameters\n )\n\n # Handle 302 redirect response\n if headers.get('status') == '302 Found' and headers.get('location'):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n headers['location'],\n parameters=url_parameters\n )\n\n return [\n github.ContentFile.ContentFile(self._requester, headers, attributes, completed=(attributes[\"type\"] != \"file\")) # Lazy completion only makes sense for files. See discussion here: https://github.com/jacquev6/PyGithub/issues/140#issuecomment-13481130\n for attributes in data\n ]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of all contributors for the specified object.", "response": "def get_contributors(self, anon=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/contributors `_\n :param anon: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n url_parameters = dict()\n if anon is not github.GithubObject.NotSet:\n url_parameters[\"anon\"] = anon\n\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/contributors\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_download(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/downloads/\" + str(id)\n )\n return github.Download.Download(self._requester, headers, data, completed=True)", "response": "returns a github. Download. Download object for the given ID"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of all downloads for the current user.", "response": "def get_downloads(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/downloads `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Download.Download`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Download.Download,\n self._requester,\n self.url + \"/downloads\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_forks(self):\n return github.PaginatedList.PaginatedList(\n Repository,\n self._requester,\n self.url + \"/forks\",\n None\n )", "response": "Returns a paginated list of all the forked repos."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a GitBlob object for the given sha", "response": "def get_git_blob(self, sha):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/git/blobs/:sha `_\n :param sha: string\n :rtype: :class:`github.GitBlob.GitBlob`\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/git/blobs/\" + sha\n )\n return github.GitBlob.GitBlob(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a github. GitCommit object for the given sha", "response": "def get_git_commit(self, sha):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/git/commits/:sha `_\n :param sha: string\n :rtype: :class:`github.GitCommit.GitCommit`\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/git/commits/\" + sha\n )\n return github.GitCommit.GitCommit(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a github. GitRef. GitRef object for the given ref", "response": "def get_git_ref(self, ref):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/git/refs/:ref `_\n :param ref: string\n :rtype: :class:`github.GitRef.GitRef`\n \"\"\"\n prefix = \"/git/refs/\"\n if not self._requester.FIX_REPO_GET_GIT_REF:\n prefix = \"/git/\"\n assert isinstance(ref, (str, unicode)), ref\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + prefix + ref\n )\n return github.GitRef.GitRef(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_git_refs(self):\n return github.PaginatedList.PaginatedList(\n github.GitRef.GitRef,\n self._requester,\n self.url + \"/git/refs\",\n None\n )", "response": "Returns a paginated list of GitRefs for the current locale."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a GitTag object for the given sha", "response": "def get_git_tag(self, sha):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/git/tags/:sha `_\n :param sha: string\n :rtype: :class:`github.GitTag.GitTag`\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/git/tags/\" + sha\n )\n return github.GitTag.GitTag(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a GitTree object for the given sha", "response": "def get_git_tree(self, sha, recursive=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/git/trees/:sha `_\n :param sha: string\n :param recursive: bool\n :rtype: :class:`github.GitTree.GitTree`\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n assert recursive is github.GithubObject.NotSet or isinstance(recursive, bool), recursive\n url_parameters = dict()\n if recursive is not github.GithubObject.NotSet and recursive:\n # GitHub API requires the recursive parameter be set to 1.\n url_parameters[\"recursive\"] = 1\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/git/trees/\" + sha,\n parameters=url_parameters\n )\n return github.GitTree.GitTree(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a single issue with the given number", "response": "def get_issue(self, number):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/issues/:number `_\n :param number: integer\n :rtype: :class:`github.Issue.Issue`\n \"\"\"\n assert isinstance(number, (int, long)), number\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/issues/\" + str(number)\n )\n return github.Issue.Issue(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of issues in the given state.", "response": "def get_issues(self, milestone=github.GithubObject.NotSet, state=github.GithubObject.NotSet, assignee=github.GithubObject.NotSet, mentioned=github.GithubObject.NotSet, labels=github.GithubObject.NotSet, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet, since=github.GithubObject.NotSet, creator=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/issues `_\n :param milestone: :class:`github.Milestone.Milestone` or \"none\" or \"*\"\n :param state: string. `open`, `closed`, or `all`. If this is not set the GitHub API default behavior will be used. At the moment this is to return only open issues. This might change anytime on GitHub API side and it could be clever to explicitly specify the state value.\n :param assignee: string or :class:`github.NamedUser.NamedUser` or \"none\" or \"*\"\n :param mentioned: :class:`github.NamedUser.NamedUser`\n :param labels: list of :class:`github.Label.Label`\n :param sort: string\n :param direction: string\n :param since: datetime.datetime\n :param creator: string or :class:`github.NamedUser.NamedUser`\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Issue.Issue`\n \"\"\"\n assert milestone is github.GithubObject.NotSet or milestone == \"*\" or milestone == \"none\" or isinstance(milestone, github.Milestone.Milestone), milestone\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert assignee is github.GithubObject.NotSet or isinstance(assignee, github.NamedUser.NamedUser) or isinstance(assignee, (str, unicode)), assignee\n assert mentioned is github.GithubObject.NotSet or isinstance(mentioned, github.NamedUser.NamedUser), mentioned\n assert labels is github.GithubObject.NotSet or all(isinstance(element, github.Label.Label) for element in labels), labels\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n assert creator is github.GithubObject.NotSet or isinstance(creator, github.NamedUser.NamedUser) or isinstance(creator, (str, unicode)), creator\n url_parameters = dict()\n if milestone is not github.GithubObject.NotSet:\n if isinstance(milestone, (str, unicode)):\n url_parameters[\"milestone\"] = milestone\n else:\n url_parameters[\"milestone\"] = milestone._identity\n if state is not github.GithubObject.NotSet:\n url_parameters[\"state\"] = state\n if assignee is not github.GithubObject.NotSet:\n if isinstance(assignee, (str, unicode)):\n url_parameters[\"assignee\"] = assignee\n else:\n url_parameters[\"assignee\"] = assignee._identity\n if mentioned is not github.GithubObject.NotSet:\n url_parameters[\"mentioned\"] = mentioned._identity\n if labels is not github.GithubObject.NotSet:\n url_parameters[\"labels\"] = \",\".join(label.name for label in labels)\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n if creator is not github.GithubObject.NotSet:\n if isinstance(creator, (str, unicode)):\n url_parameters[\"creator\"] = creator\n else:\n url_parameters[\"creator\"] = creator._identity\n return github.PaginatedList.PaginatedList(\n github.Issue.Issue,\n self._requester,\n self.url + \"/issues\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_issues_event(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/issues/events/\" + str(id),\n headers={'Accept': Consts.mediaTypeLockReasonPreview}\n )\n return github.IssueEvent.IssueEvent(self._requester, headers, data, completed=True)", "response": "returns a github. IssueEvent. IssueEvent object for the given id"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_keys(self):\n return github.PaginatedList.PaginatedList(\n github.RepositoryKey.RepositoryKey,\n self._requester,\n self.url + \"/keys\",\n None\n )", "response": "Returns a paginated list of all keys in the current repository."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a label object for the given name", "response": "def get_label(self, name):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/labels/:name `_\n :param name: string\n :rtype: :class:`github.Label.Label`\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/labels/\" + urllib.quote(name)\n )\n return github.Label.Label(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a dict of strings to integer", "response": "def get_languages(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/languages `_\n :rtype: dict of string to integer\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/languages\"\n )\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_license(self):\n\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/license\"\n )\n return github.ContentFile.ContentFile(self._requester, headers, data, completed=True)", "response": "returns a github. ContentFile. ContentFile object for the current node s license"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_milestone(self, number):\n assert isinstance(number, (int, long)), number\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/milestones/\" + str(number)\n )\n return github.Milestone.Milestone(self._requester, headers, data, completed=True)", "response": "returns a single milestone with the given number"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of milestones in the repository.", "response": "def get_milestones(self, state=github.GithubObject.NotSet, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/milestones `_\n :param state: string\n :param sort: string\n :param direction: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Milestone.Milestone`\n \"\"\"\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n url_parameters = dict()\n if state is not github.GithubObject.NotSet:\n url_parameters[\"state\"] = state\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n return github.PaginatedList.PaginatedList(\n github.Milestone.Milestone,\n self._requester,\n self.url + \"/milestones\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_network_events(self):\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n \"/networks/\" + self.owner.login + \"/\" + self.name + \"/events\",\n None\n )", "response": "Returns a paginated list of all events for this network."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a github. PullRequest object for the specified pull request number", "response": "def get_pull(self, number):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/:number `_\n :param number: integer\n :rtype: :class:`github.PullRequest.PullRequest`\n \"\"\"\n assert isinstance(number, (int, long)), number\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/pulls/\" + str(number)\n )\n return github.PullRequest.PullRequest(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_pulls(self, state=github.GithubObject.NotSet, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet, base=github.GithubObject.NotSet, head=github.GithubObject.NotSet):\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n assert base is github.GithubObject.NotSet or isinstance(base, (str, unicode)), base\n assert head is github.GithubObject.NotSet or isinstance(head, (str, unicode)), head\n url_parameters = dict()\n if state is not github.GithubObject.NotSet:\n url_parameters[\"state\"] = state\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n if base is not github.GithubObject.NotSet:\n url_parameters[\"base\"] = base\n if head is not github.GithubObject.NotSet:\n url_parameters[\"head\"] = head\n return github.PaginatedList.PaginatedList(\n github.PullRequest.PullRequest,\n self._requester,\n self.url + \"/pulls\",\n url_parameters\n )", "response": "Returns a list of pull requests for the specified state sort and direction."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a paginated list of all pull requests comments for the current repository", "response": "def get_pulls_comments(self, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet, since=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/pulls/comments `_\n :param sort: string\n :param direction: string\n :param since: datetime.datetime\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.PullRequestComment.PullRequestComment`\n \"\"\"\n return self.get_pulls_review_comments(sort, direction, since)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_pulls_review_comments(self, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet, since=github.GithubObject.NotSet):\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n url_parameters = dict()\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n return github.PaginatedList.PaginatedList(\n github.IssueComment.IssueComment,\n self._requester,\n self.url + \"/pulls/comments\",\n url_parameters\n )", "response": "Returns a paginated list of all comments for the given pull request."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_readme(self, ref=github.GithubObject.NotSet):\n assert ref is github.GithubObject.NotSet or isinstance(ref, (str, unicode)), ref\n url_parameters = dict()\n if ref is not github.GithubObject.NotSet:\n url_parameters[\"ref\"] = ref\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/readme\",\n parameters=url_parameters\n )\n return github.ContentFile.ContentFile(self._requester, headers, data, completed=True)", "response": "returns a github. ContentFile. ContentFile object for the specified ref"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a github. SourceImport. SourceImport object", "response": "def get_source_import(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/import `_\n :rtype: :class:`github.SourceImport.SourceImport`\n \"\"\"\n import_header = {\"Accept\": Consts.mediaTypeImportPreview}\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/import\",\n headers=import_header,\n )\n if not data:\n return None\n else:\n return github.SourceImport.SourceImport(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a paginated list of all the stargazers for the current user.", "response": "def get_stargazers(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/stargazers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/stargazers\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of all stargazers with dates.", "response": "def get_stargazers_with_dates(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/stargazers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Stargazer.Stargazer`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Stargazer.Stargazer,\n self._requester,\n self.url + \"/stargazers\",\n None,\n headers={'Accept': Consts.mediaTypeStarringPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_stats_contributors(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/stats/contributors\"\n )\n if not data:\n return None\n else:\n return [\n github.StatsContributor.StatsContributor(self._requester, headers, attributes, completed=True)\n for attributes in data\n ]", "response": "returns a list of github. StatsContributor. StatsContributor objects"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of github. StatsCommitActivity. StatsCommitActivity objects", "response": "def get_stats_commit_activity(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/stats/commit_activity `_\n :rtype: None or list of :class:`github.StatsCommitActivity.StatsCommitActivity`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/stats/commit_activity\"\n )\n if not data:\n return None\n else:\n return [\n github.StatsCommitActivity.StatsCommitActivity(self._requester, headers, attributes, completed=True)\n for attributes in data\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of github. StatsCodeFrequency. StatsCodeFrequency objects", "response": "def get_stats_code_frequency(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/stats/code_frequency `_\n :rtype: None or list of :class:`github.StatsCodeFrequency.StatsCodeFrequency`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/stats/code_frequency\"\n )\n if not data:\n return None\n else:\n return [\n github.StatsCodeFrequency.StatsCodeFrequency(self._requester, headers, attributes, completed=True)\n for attributes in data\n ]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_stats_participation(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/stats/participation\"\n )\n if not data:\n return None\n else:\n return github.StatsParticipation.StatsParticipation(self._requester, headers, data, completed=True)", "response": "returns a new instance of github. StatsParticipation. StatsParticipation."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new instance of github. StatsPunchCard. StatsPunchCard.", "response": "def get_stats_punch_card(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/stats/punch_card `_\n :rtype: None or :class:`github.StatsPunchCard.StatsPunchCard`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/stats/punch_card\"\n )\n if not data:\n return None\n else:\n return github.StatsPunchCard.StatsPunchCard(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a paginated list of subscribers for the current user.", "response": "def get_subscribers(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/subscribers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/subscribers\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_tags(self):\n return github.PaginatedList.PaginatedList(\n github.Tag.Tag,\n self._requester,\n self.url + \"/tags\",\n None\n )", "response": "Returns a paginated list of all tags for the current user."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_releases(self):\n return github.PaginatedList.PaginatedList(\n github.GitRelease.GitRelease,\n self._requester,\n self.url + \"/releases\",\n None\n )", "response": "Returns a paginated list of all releases for this user."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_release(self, id):\n if isinstance(id, int):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/releases/\" + str(id)\n )\n return github.GitRelease.GitRelease(self._requester, headers, data, completed=True)\n elif isinstance(id, (str, unicode)):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/releases/tags/\" + id\n )\n return github.GitRelease.GitRelease(self._requester, headers, data, completed=True)", "response": "returns a single release with the given id"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_topics(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/topics\",\n headers={'Accept': Consts.mediaTypeTopicsPreview}\n )\n return data['names']", "response": "returns a list of all topics for a given user"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a paginated list of all the watchers for this user.", "response": "def get_watchers(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/watchers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.url + \"/watchers\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn true if the issue has a specific issue with the given assignee.", "response": "def has_in_assignees(self, assignee):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/assignees/:assignee `_\n :param assignee: string or :class:`github.NamedUser.NamedUser`\n :rtype: bool\n \"\"\"\n assert isinstance(assignee, github.NamedUser.NamedUser) or isinstance(assignee, (str, unicode)), assignee\n\n if isinstance(assignee, github.NamedUser.NamedUser):\n assignee = assignee._identity\n\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/assignees/\" + assignee\n )\n return status == 204"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning true if the user has access to the collaborator.", "response": "def has_in_collaborators(self, collaborator):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/collaborators/:user `_\n :param collaborator: string or :class:`github.NamedUser.NamedUser`\n :rtype: bool\n \"\"\"\n assert isinstance(collaborator, github.NamedUser.NamedUser) or isinstance(collaborator, (str, unicode)), collaborator\n\n if isinstance(collaborator, github.NamedUser.NamedUser):\n collaborator = collaborator._identity\n\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/collaborators/\" + collaborator\n )\n return status == 204"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of github. Issue objects for the given state and keyword", "response": "def legacy_search_issues(self, state, keyword):\n \"\"\"\n :calls: `GET /legacy/issues/search/:owner/:repository/:state/:keyword `_\n :param state: \"open\" or \"closed\"\n :param keyword: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Issue.Issue`\n \"\"\"\n assert state in [\"open\", \"closed\"], state\n assert isinstance(keyword, (str, unicode)), keyword\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/legacy/issues/search/\" + self.owner.login + \"/\" + self.name + \"/\" + state + \"/\" + urllib.quote(keyword)\n )\n return [\n github.Issue.Issue(self._requester, headers, github.Legacy.convertIssue(element), completed=False)\n for element in data[\"issues\"]\n ]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmerge the base and head of the branch with the given commit message.", "response": "def merge(self, base, head, commit_message=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/merges `_\n :param base: string\n :param head: string\n :param commit_message: string\n :rtype: :class:`github.Commit.Commit`\n \"\"\"\n assert isinstance(base, (str, unicode)), base\n assert isinstance(head, (str, unicode)), head\n assert commit_message is github.GithubObject.NotSet or isinstance(commit_message, (str, unicode)), commit_message\n post_parameters = {\n \"base\": base,\n \"head\": head,\n }\n if commit_message is not github.GithubObject.NotSet:\n post_parameters[\"commit_message\"] = commit_message\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/merges\",\n input=post_parameters\n )\n if data is None:\n return None\n else:\n return github.Commit.Commit(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef replace_topics(self, topics):\n post_parameters = {\n 'names': topics\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.url + \"/topics\",\n headers={'Accept': Consts.mediaTypeTopicsPreview},\n input=post_parameters\n )", "response": "replace topics with new ones"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsubscribing to a hub event.", "response": "def subscribe_to_hub(self, event, callback, secret=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /hub `_\n :param event: string\n :param callback: string\n :param secret: string\n :rtype: None\n \"\"\"\n return self._hub(\"subscribe\", event, callback, secret)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef unsubscribe_from_hub(self, event, callback):\n return self._hub(\"unsubscribe\", event, callback, github.GithubObject.NotSet)", "response": "Unsubscribe from the hub."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_status(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url,\n headers={\n \"Accept\": Consts.mediaTypeMigrationPreview\n }\n )\n self._useAttributes(data)\n return self.state", "response": "returns the current state of the migration"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the url of the archive of the current user", "response": "def get_archive_url(self):\n \"\"\"\n :calls: `GET /user/migrations/:migration_id/archive`_\n :rtype: str\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/archive\",\n headers={\n \"Accept\": Consts.mediaTypeMigrationPreview\n }\n )\n return data[\"data\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unlock_repo(self, repo_name):\n assert isinstance(repo_name, (str, unicode)), repo_name\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n self.url + \"/repos/\" + repo_name + \"/lock\",\n headers={\n \"Accept\": Consts.mediaTypeMigrationPreview\n }\n )", "response": "unlocks the specified repository"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a paginated list of all columns for the current project.", "response": "def get_columns(self):\n \"\"\"\n :calls: `GET /projects/:project_id/columns `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.ProjectColumn.ProjectColumn`\n \"\"\"\n\n return github.PaginatedList.PaginatedList(\n github.ProjectColumn.ProjectColumn,\n self._requester,\n self.columns_url,\n None,\n {\"Accept\": Consts.mediaTypeProjectsPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new column in the project with the given name", "response": "def create_column(self, name):\n \"\"\"\n calls: `POST https://developer.github.com/v3/projects/columns/#create-a-project-column>`_\n :param name: string\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n post_parameters = {\"name\": name}\n import_header = {\"Accept\": Consts.mediaTypeProjectsPreview}\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/columns\",\n headers=import_header,\n input=post_parameters\n )\n return github.ProjectColumn.ProjectColumn(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_to_emails(self, *emails):\n assert all(isinstance(element, (str, unicode)) for element in emails), emails\n post_parameters = emails\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/user/emails\",\n input=post_parameters\n )", "response": "Adds the specified emails to the user s set of user s email addresses."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd the current user to the given user s followers.", "response": "def add_to_following(self, following):\n \"\"\"\n :calls: `PUT /user/following/:user `_\n :param following: :class:`github.NamedUser.NamedUser`\n :rtype: None\n \"\"\"\n assert isinstance(following, github.NamedUser.NamedUser), following\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n \"/user/following/\" + following._identity\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds the current user s ID to the starred user s list of active entries.", "response": "def add_to_starred(self, starred):\n \"\"\"\n :calls: `PUT /user/starred/:owner/:repo `_\n :param starred: :class:`github.Repository.Repository`\n :rtype: None\n \"\"\"\n assert isinstance(starred, github.Repository.Repository), starred\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n \"/user/starred/\" + starred._identity\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding the current entry to the watched entry.", "response": "def add_to_watched(self, watched):\n \"\"\"\n :calls: `PUT /repos/:owner/:repo/subscription `_\n :param watched: :class:`github.Repository.Repository`\n :rtype: None\n \"\"\"\n assert isinstance(watched, github.Repository.Repository), watched\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n \"/repos/\" + watched._identity + \"/subscription\",\n input={\"subscribed\": True}\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate an authorization object for the given note and url.", "response": "def create_authorization(self, scopes=github.GithubObject.NotSet, note=github.GithubObject.NotSet, note_url=github.GithubObject.NotSet, client_id=github.GithubObject.NotSet, client_secret=github.GithubObject.NotSet, onetime_password=None):\n \"\"\"\n :calls: `POST /authorizations `_\n :param scopes: list of string\n :param note: string\n :param note_url: string\n :param client_id: string\n :param client_secret: string\n :param onetime_password: string\n :rtype: :class:`github.Authorization.Authorization`\n \"\"\"\n assert scopes is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in scopes), scopes\n assert note is github.GithubObject.NotSet or isinstance(note, (str, unicode)), note\n assert note_url is github.GithubObject.NotSet or isinstance(note_url, (str, unicode)), note_url\n assert client_id is github.GithubObject.NotSet or isinstance(client_id, (str, unicode)), client_id\n assert client_secret is github.GithubObject.NotSet or isinstance(client_secret, (str, unicode)), client_secret\n assert onetime_password is None or isinstance(onetime_password, (str, unicode)), onetime_password\n post_parameters = dict()\n if scopes is not github.GithubObject.NotSet:\n post_parameters[\"scopes\"] = scopes\n if note is not github.GithubObject.NotSet:\n post_parameters[\"note\"] = note\n if note_url is not github.GithubObject.NotSet:\n post_parameters[\"note_url\"] = note_url\n if client_id is not github.GithubObject.NotSet:\n post_parameters[\"client_id\"] = client_id\n if client_secret is not github.GithubObject.NotSet:\n post_parameters[\"client_secret\"] = client_secret\n if onetime_password is not None:\n request_header = {Consts.headerOTP: onetime_password} # pragma no cover (Should be covered)\n else:\n request_header = None\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/authorizations\",\n input=post_parameters,\n headers=request_header,\n )\n return github.Authorization.Authorization(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_gist(self, public, files, description=github.GithubObject.NotSet):\n assert isinstance(public, bool), public\n assert all(isinstance(element, github.InputFileContent) for element in files.itervalues()), files\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n post_parameters = {\n \"public\": public,\n \"files\": {key: value._identity for key, value in files.iteritems()},\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/gists\",\n input=post_parameters\n )\n return github.Gist.Gist(self._requester, headers, data, completed=True)", "response": "Creates a new gist with the given files and description."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new user key for the given title and key.", "response": "def create_key(self, title, key):\n \"\"\"\n :calls: `POST /user/keys `_\n :param title: string\n :param key: string\n :rtype: :class:`github.UserKey.UserKey`\n \"\"\"\n assert isinstance(title, (str, unicode)), title\n assert isinstance(key, (str, unicode)), key\n post_parameters = {\n \"title\": title,\n \"key\": key,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/user/keys\",\n input=post_parameters\n )\n return github.UserKey.UserKey(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_repo(self, name, description=github.GithubObject.NotSet, homepage=github.GithubObject.NotSet,\n private=github.GithubObject.NotSet, has_issues=github.GithubObject.NotSet,\n has_wiki=github.GithubObject.NotSet, has_downloads=github.GithubObject.NotSet,\n has_projects=github.GithubObject.NotSet, auto_init=github.GithubObject.NotSet, license_template=github.GithubObject.NotSet,\n gitignore_template=github.GithubObject.NotSet, allow_squash_merge=github.GithubObject.NotSet,\n allow_merge_commit=github.GithubObject.NotSet, allow_rebase_merge=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /user/repos `_\n :param name: string\n :param description: string\n :param homepage: string\n :param private: bool\n :param has_issues: bool\n :param has_wiki: bool\n :param has_downloads: bool\n :param has_projects: bool\n :param auto_init: bool\n :param license_template: string\n :param gitignore_template: string\n :param allow_squash_merge: bool\n :param allow_merge_commit: bool\n :param allow_rebase_merge: bool\n :rtype: :class:`github.Repository.Repository`\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert homepage is github.GithubObject.NotSet or isinstance(homepage, (str, unicode)), homepage\n assert private is github.GithubObject.NotSet or isinstance(private, bool), private\n assert has_issues is github.GithubObject.NotSet or isinstance(has_issues, bool), has_issues\n assert has_wiki is github.GithubObject.NotSet or isinstance(has_wiki, bool), has_wiki\n assert has_downloads is github.GithubObject.NotSet or isinstance(has_downloads, bool), has_downloads\n assert has_projects is github.GithubObject.NotSet or isinstance(has_projects, bool), has_projects\n assert auto_init is github.GithubObject.NotSet or isinstance(auto_init, bool), auto_init\n assert license_template is github.GithubObject.NotSet or isinstance(license_template, (str, unicode)), license_template\n assert gitignore_template is github.GithubObject.NotSet or isinstance(gitignore_template, (str, unicode)), gitignore_template\n assert allow_squash_merge is github.GithubObject.NotSet or isinstance(allow_squash_merge, bool), allow_squash_merge\n assert allow_merge_commit is github.GithubObject.NotSet or isinstance(allow_merge_commit, bool), allow_merge_commit\n assert allow_rebase_merge is github.GithubObject.NotSet or isinstance(allow_rebase_merge, bool), allow_rebase_merge\n post_parameters = {\n \"name\": name,\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if homepage is not github.GithubObject.NotSet:\n post_parameters[\"homepage\"] = homepage\n if private is not github.GithubObject.NotSet:\n post_parameters[\"private\"] = private\n if has_issues is not github.GithubObject.NotSet:\n post_parameters[\"has_issues\"] = has_issues\n if has_wiki is not github.GithubObject.NotSet:\n post_parameters[\"has_wiki\"] = has_wiki\n if has_downloads is not github.GithubObject.NotSet:\n post_parameters[\"has_downloads\"] = has_downloads\n if has_projects is not github.GithubObject.NotSet:\n post_parameters[\"has_projects\"] = has_projects\n if auto_init is not github.GithubObject.NotSet:\n post_parameters[\"auto_init\"] = auto_init\n if license_template is not github.GithubObject.NotSet:\n post_parameters[\"license_template\"] = license_template\n if gitignore_template is not github.GithubObject.NotSet:\n post_parameters[\"gitignore_template\"] = gitignore_template\n if allow_squash_merge is not github.GithubObject.NotSet:\n post_parameters[\"allow_squash_merge\"] = allow_squash_merge\n if allow_merge_commit is not github.GithubObject.NotSet:\n post_parameters[\"allow_merge_commit\"] = allow_merge_commit\n if allow_rebase_merge is not github.GithubObject.NotSet:\n post_parameters[\"allow_rebase_merge\"] = allow_rebase_merge\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/user/repos\",\n input=post_parameters\n )\n return github.Repository.Repository(self._requester, headers, data, completed=True)", "response": "Creates a new repository with the given properties."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef edit(self, name=github.GithubObject.NotSet, email=github.GithubObject.NotSet, blog=github.GithubObject.NotSet, company=github.GithubObject.NotSet, location=github.GithubObject.NotSet, hireable=github.GithubObject.NotSet, bio=github.GithubObject.NotSet):\n assert name is github.GithubObject.NotSet or isinstance(name, (str, unicode)), name\n assert email is github.GithubObject.NotSet or isinstance(email, (str, unicode)), email\n assert blog is github.GithubObject.NotSet or isinstance(blog, (str, unicode)), blog\n assert company is github.GithubObject.NotSet or isinstance(company, (str, unicode)), company\n assert location is github.GithubObject.NotSet or isinstance(location, (str, unicode)), location\n assert hireable is github.GithubObject.NotSet or isinstance(hireable, bool), hireable\n assert bio is github.GithubObject.NotSet or isinstance(bio, (str, unicode)), bio\n post_parameters = dict()\n if name is not github.GithubObject.NotSet:\n post_parameters[\"name\"] = name\n if email is not github.GithubObject.NotSet:\n post_parameters[\"email\"] = email\n if blog is not github.GithubObject.NotSet:\n post_parameters[\"blog\"] = blog\n if company is not github.GithubObject.NotSet:\n post_parameters[\"company\"] = company\n if location is not github.GithubObject.NotSet:\n post_parameters[\"location\"] = location\n if hireable is not github.GithubObject.NotSet:\n post_parameters[\"hireable\"] = hireable\n if bio is not github.GithubObject.NotSet:\n post_parameters[\"bio\"] = bio\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n \"/user\",\n input=post_parameters\n )\n self._useAttributes(data)", "response": "Updates the user s identity."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a github. Authorization. Authorization object for the given id", "response": "def get_authorization(self, id):\n \"\"\"\n :calls: `GET /authorizations/:id `_\n :param id: integer\n :rtype: :class:`github.Authorization.Authorization`\n \"\"\"\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/authorizations/\" + str(id)\n )\n return github.Authorization.Authorization(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_authorizations(self):\n return github.PaginatedList.PaginatedList(\n github.Authorization.Authorization,\n self._requester,\n \"/authorizations\",\n None\n )", "response": "Returns a paginated list of all the authorizations for the current user."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a paginated list of all followers of the user.", "response": "def get_followers(self):\n \"\"\"\n :calls: `GET /user/followers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n \"/user/followers\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a paginated list of the users following the current user.", "response": "def get_following(self):\n \"\"\"\n :calls: `GET /user/following `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n \"/user/following\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a paginated list of gists for the given object.", "response": "def get_gists(self, since=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /gists `_\n :param since: datetime.datetime format YYYY-MM-DDTHH:MM:SSZ\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Gist.Gist`\n \"\"\"\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n url_parameters = dict()\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n return github.PaginatedList.PaginatedList(\n github.Gist.Gist,\n self._requester,\n \"/gists\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_issues(self, filter=github.GithubObject.NotSet, state=github.GithubObject.NotSet, labels=github.GithubObject.NotSet, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet, since=github.GithubObject.NotSet):\n assert filter is github.GithubObject.NotSet or isinstance(filter, (str, unicode)), filter\n assert state is github.GithubObject.NotSet or isinstance(state, (str, unicode)), state\n assert labels is github.GithubObject.NotSet or all(isinstance(element, github.Label.Label) for element in labels), labels\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n url_parameters = dict()\n if filter is not github.GithubObject.NotSet:\n url_parameters[\"filter\"] = filter\n if state is not github.GithubObject.NotSet:\n url_parameters[\"state\"] = state\n if labels is not github.GithubObject.NotSet:\n url_parameters[\"labels\"] = \",\".join(label.name for label in labels)\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n return github.PaginatedList.PaginatedList(\n github.Issue.Issue,\n self._requester,\n \"/issues\",\n url_parameters\n )", "response": "returns a list of issues in the repository"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_key(self, id):\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/user/keys/\" + str(id)\n )\n return github.UserKey.UserKey(self._requester, headers, data, completed=True)", "response": "Returns a single user key with the given id."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a single notification for the given id", "response": "def get_notification(self, id):\n \"\"\"\n :calls: `GET /notifications/threads/:id `_\n :rtype: :class:`github.Notification.Notification`\n \"\"\"\n\n assert isinstance(id, (str, unicode)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n \"/notifications/threads/\" + id\n )\n return github.Notification.Notification(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a paginated list of github. Notification. Notification objects for the specified resource.", "response": "def get_notifications(self, all=github.GithubObject.NotSet, participating=github.GithubObject.NotSet, since=github.GithubObject.NotSet, before=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /notifications `_\n :param all: bool\n :param participating: bool\n :param since: datetime.datetime\n :param before: datetime.datetime\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Notification.Notification`\n \"\"\"\n\n assert all is github.GithubObject.NotSet or isinstance(all, bool), all\n assert participating is github.GithubObject.NotSet or isinstance(participating, bool), participating\n assert since is github.GithubObject.NotSet or isinstance(since, datetime.datetime), since\n assert before is github.GithubObject.NotSet or isinstance(before, datetime.datetime), before\n\n params = dict()\n if all is not github.GithubObject.NotSet:\n params[\"all\"] = all\n if participating is not github.GithubObject.NotSet:\n params[\"participating\"] = participating\n if since is not github.GithubObject.NotSet:\n params[\"since\"] = since.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n if before is not github.GithubObject.NotSet:\n params[\"before\"] = before.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n\n return github.PaginatedList.PaginatedList(\n github.Notification.Notification,\n self._requester,\n \"/notifications\",\n params\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_organization_events(self, org):\n assert isinstance(org, github.Organization.Organization), org\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n \"/users/\" + self.login + \"/events/orgs/\" + org.login,\n None\n )", "response": "Returns a paginated list of all events for the user in the organization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a paginated list of organizations owned by the user.", "response": "def get_orgs(self):\n \"\"\"\n :calls: `GET /user/orgs `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Organization.Organization`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Organization.Organization,\n self._requester,\n \"/user/orgs\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_repos(self, visibility=github.GithubObject.NotSet, affiliation=github.GithubObject.NotSet, type=github.GithubObject.NotSet, sort=github.GithubObject.NotSet, direction=github.GithubObject.NotSet):\n assert visibility is github.GithubObject.NotSet or isinstance(visibility, (str, unicode)), visibility\n assert affiliation is github.GithubObject.NotSet or isinstance(affiliation, (str, unicode)), affiliation\n assert type is github.GithubObject.NotSet or isinstance(type, (str, unicode)), type\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n url_parameters = dict()\n if visibility is not github.GithubObject.NotSet:\n url_parameters[\"visibility\"] = visibility\n if affiliation is not github.GithubObject.NotSet:\n url_parameters[\"affiliation\"] = affiliation\n if type is not github.GithubObject.NotSet:\n url_parameters[\"type\"] = type\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n \"/user/repos\",\n url_parameters\n )", "response": "Returns a list of repos for the specified user."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_starred(self):\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n \"/user/starred\",\n None\n )", "response": "Returns a list of starred users."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a paginated list of starred gists.", "response": "def get_starred_gists(self):\n \"\"\"\n :calls: `GET /gists/starred `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Gist.Gist`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Gist.Gist,\n self._requester,\n \"/gists/starred\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns true if the user has in the following user.", "response": "def has_in_following(self, following):\n \"\"\"\n :calls: `GET /user/following/:user `_\n :param following: :class:`github.NamedUser.NamedUser`\n :rtype: bool\n \"\"\"\n assert isinstance(following, github.NamedUser.NamedUser), following\n status, headers, data = self._requester.requestJson(\n \"GET\",\n \"/user/following/\" + following._identity\n )\n return status == 204"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef has_in_starred(self, starred):\n assert isinstance(starred, github.Repository.Repository), starred\n status, headers, data = self._requester.requestJson(\n \"GET\",\n \"/user/starred/\" + starred._identity\n )\n return status == 204", "response": "Returns true if the user has in starred entries."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef has_in_subscriptions(self, subscription):\n assert isinstance(subscription, github.Repository.Repository), subscription\n status, headers, data = self._requester.requestJson(\n \"GET\",\n \"/user/subscriptions/\" + subscription._identity\n )\n return status == 204", "response": "Returns true if the user has access to the specified user s internal cache."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef has_in_watched(self, watched):\n assert isinstance(watched, github.Repository.Repository), watched\n status, headers, data = self._requester.requestJson(\n \"GET\",\n \"/repos/\" + watched._identity + \"/subscription\"\n )\n return status == 200", "response": "Returns true if the user has in the watched repository."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef mark_notifications_as_read(self, last_read_at=datetime.datetime.utcnow()):\n assert isinstance(last_read_at, datetime.datetime)\n put_parameters = {\n \"last_read_at\": last_read_at.strftime('%Y-%m-%dT%H:%M:%SZ')\n }\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n \"/notifications\",\n input=put_parameters\n )", "response": "marks the notifications as read."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_from_subscriptions(self, subscription):\n assert isinstance(subscription, github.Repository.Repository), subscription\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n \"/user/subscriptions/\" + subscription._identity\n )", "response": "Removes the user from the specified subscription."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving the entry from the watched list.", "response": "def remove_from_watched(self, watched):\n \"\"\"\n :calls: `DELETE /repos/:owner/:repo/subscription `_\n :param watched: :class:`github.Repository.Repository`\n :rtype: None\n \"\"\"\n assert isinstance(watched, github.Repository.Repository), watched\n headers, data = self._requester.requestJsonAndCheck(\n \"DELETE\",\n \"/repos/\" + watched._identity + \"/subscription\"\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef accept_invitation(self, invitation):\n assert isinstance(invitation, github.Invitation.Invitation) or isinstance(invitation, int)\n\n if isinstance(invitation, github.Invitation.Invitation):\n invitation = invitation.id\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n \"/user/repository_invitations/\" + str(invitation),\n input={}\n )", "response": ":calls: `PATCH /user/repository_invitations/:invitation_id `\n :param invitation: :class:`github.Invitation.Invitation` or int\n :rtype: None"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a migration for the specified repos.", "response": "def create_migration(self, repos, lock_repositories=github.GithubObject.NotSet, exclude_attachments=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /user/migrations`_\n :param repos: list or tuple of str\n :param lock_repositories: bool\n :param exclude_attachments: bool\n :rtype: :class:`github.Migration.Migration`\n \"\"\"\n assert isinstance(repos, (list, tuple)), repos\n assert all(isinstance(repo, (str, unicode)) for repo in repos), repos\n assert lock_repositories is github.GithubObject.NotSet or isinstance(lock_repositories, bool), lock_repositories\n assert exclude_attachments is github.GithubObject.NotSet or isinstance(exclude_attachments, bool), exclude_attachments\n post_parameters = {\n \"repositories\": repos\n }\n if lock_repositories is not github.GithubObject.NotSet:\n post_parameters[\"lock_repositories\"] = lock_repositories\n if exclude_attachments is not github.GithubObject.NotSet:\n post_parameters[\"exclude_attachments\"] = exclude_attachments\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n \"/user/migrations\",\n input=post_parameters,\n headers={\n \"Accept\": Consts.mediaTypeMigrationPreview\n }\n )\n return github.Migration.Migration(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a paginated list of repositories for the current language", "response": "def get_repos(self):\n \"\"\"\n :calls: `GET /installation/repositories `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Repository.Repository`\n \"\"\"\n url_parameters = dict()\n\n return github.PaginatedList.PaginatedList(\n contentClass=github.Repository.Repository,\n requester=self._requester,\n firstUrl=\"/installation/repositories\",\n firstParams=url_parameters,\n headers=INTEGRATION_PREVIEW_HEADERS,\n list_item='repositories'\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nchange the contents of the object with the given body", "response": "def edit(self, body):\n \"\"\"\n :calls: `PATCH /repos/:owner/:repo/pulls/comments/:number `_\n :param body: string\n :rtype: None\n \"\"\"\n assert isinstance(body, (str, unicode)), body\n post_parameters = {\n \"body\": body,\n }\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates the authorizations for the given authorizations.", "response": "def edit(self, scopes=github.GithubObject.NotSet, add_scopes=github.GithubObject.NotSet, remove_scopes=github.GithubObject.NotSet, note=github.GithubObject.NotSet, note_url=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /authorizations/:id `_\n :param scopes: list of string\n :param add_scopes: list of string\n :param remove_scopes: list of string\n :param note: string\n :param note_url: string\n :rtype: None\n \"\"\"\n assert scopes is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in scopes), scopes\n assert add_scopes is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in add_scopes), add_scopes\n assert remove_scopes is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in remove_scopes), remove_scopes\n assert note is github.GithubObject.NotSet or isinstance(note, (str, unicode)), note\n assert note_url is github.GithubObject.NotSet or isinstance(note_url, (str, unicode)), note_url\n post_parameters = dict()\n if scopes is not github.GithubObject.NotSet:\n post_parameters[\"scopes\"] = scopes\n if add_scopes is not github.GithubObject.NotSet:\n post_parameters[\"add_scopes\"] = add_scopes\n if remove_scopes is not github.GithubObject.NotSet:\n post_parameters[\"remove_scopes\"] = remove_scopes\n if note is not github.GithubObject.NotSet:\n post_parameters[\"note\"] = note\n if note_url is not github.GithubObject.NotSet:\n post_parameters[\"note_url\"] = note_url\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_cards(self, archived_state=github.GithubObject.NotSet):\n assert archived_state is github.GithubObject.NotSet or isinstance(archived_state, (str, unicode)), archived_state\n\n url_parameters = dict()\n if archived_state is not github.GithubObject.NotSet:\n url_parameters[\"archived_state\"] = archived_state\n\n return github.PaginatedList.PaginatedList(\n github.ProjectCard.ProjectCard,\n self._requester,\n self.url + \"/cards\",\n url_parameters,\n {\"Accept\": Consts.mediaTypeProjectsPreview}\n )", "response": "Returns a paginated list of all the cards in the specified column."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new project card for the given note content_id and content_type.", "response": "def create_card(self, note=github.GithubObject.NotSet,\n content_id=github.GithubObject.NotSet,\n content_type=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /projects/columns/:column_id/cards `_\n :param note: string\n :param content_id: integer\n :param content_type: string\n \"\"\"\n post_parameters = {}\n if isinstance(note, (str, unicode)):\n assert content_id is github.GithubObject.NotSet, content_id\n assert content_type is github.GithubObject.NotSet, content_type\n post_parameters = {\"note\": note}\n else:\n assert note is github.GithubObject.NotSet, note\n assert isinstance(content_id, int), content_id\n assert isinstance(content_type, (str, unicode)), content_type\n post_parameters = {\"content_id\": content_id,\n \"content_type\": content_type}\n\n import_header = {\"Accept\": Consts.mediaTypeProjectsPreview}\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/cards\",\n headers=import_header,\n input=post_parameters\n )\n return github.ProjectCard.ProjectCard(self._requester, headers,\n data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new commit comment for the current branch.", "response": "def create_comment(self, body, line=github.GithubObject.NotSet, path=github.GithubObject.NotSet, position=github.GithubObject.NotSet):\n \"\"\"\n :calls: `POST /repos/:owner/:repo/commits/:sha/comments `_\n :param body: string\n :param line: integer\n :param path: string\n :param position: integer\n :rtype: :class:`github.CommitComment.CommitComment`\n \"\"\"\n assert isinstance(body, (str, unicode)), body\n assert line is github.GithubObject.NotSet or isinstance(line, (int, long)), line\n assert path is github.GithubObject.NotSet or isinstance(path, (str, unicode)), path\n assert position is github.GithubObject.NotSet or isinstance(position, (int, long)), position\n post_parameters = {\n \"body\": body,\n }\n if line is not github.GithubObject.NotSet:\n post_parameters[\"line\"] = line\n if path is not github.GithubObject.NotSet:\n post_parameters[\"path\"] = path\n if position is not github.GithubObject.NotSet:\n post_parameters[\"position\"] = position\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/comments\",\n input=post_parameters\n )\n return github.CommitComment.CommitComment(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_status(self, state, target_url=github.GithubObject.NotSet, description=github.GithubObject.NotSet, context=github.GithubObject.NotSet):\n assert isinstance(state, (str, unicode)), state\n assert target_url is github.GithubObject.NotSet or isinstance(target_url, (str, unicode)), target_url\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert context is github.GithubObject.NotSet or isinstance(context, (str, unicode)), context\n post_parameters = {\n \"state\": state,\n }\n if target_url is not github.GithubObject.NotSet:\n post_parameters[\"target_url\"] = target_url\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if context is not github.GithubObject.NotSet:\n post_parameters[\"context\"] = context\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self._parentUrl(self._parentUrl(self.url)) + \"/statuses/\" + self.sha,\n input=post_parameters\n )\n return github.CommitStatus.CommitStatus(self._requester, headers, data, completed=True)", "response": "Creates a new status for the current branch."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_statuses(self):\n return github.PaginatedList.PaginatedList(\n github.CommitStatus.CommitStatus,\n self._requester,\n self._parentUrl(self._parentUrl(self.url)) + \"/statuses/\" + self.sha,\n None\n )", "response": "returns a paginated list of all statuses for this item"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_combined_status(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/status\"\n )\n return github.CommitCombinedStatus.CommitCombinedStatus(self._requester, headers, data, completed=True)", "response": "returns a CommitCombinedStatus object for this branch"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove the entry from the hierarchy", "response": "def delete(self):\n \"\"\"\n :calls: `DELETE /reactions/:id `_\n :rtype: None\n \"\"\"\n self._requester.requestJsonAndCheck(\n \"DELETE\",\n self._parentUrl(\"\") + \"/reactions/\" + str(self.id),\n headers={'Accept': Consts.mediaTypeReactionsPreview}\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_protection(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.protection_url,\n headers={'Accept': Consts.mediaTypeRequireMultipleApprovingReviews}\n )\n return github.BranchProtection.BranchProtection(self._requester, headers, data, completed=True)", "response": "returns a github. BranchProtection. BranchProtection object"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef edit_protection(self, strict=github.GithubObject.NotSet, contexts=github.GithubObject.NotSet, enforce_admins=github.GithubObject.NotSet, dismissal_users=github.GithubObject.NotSet, dismissal_teams=github.GithubObject.NotSet, dismiss_stale_reviews=github.GithubObject.NotSet, require_code_owner_reviews=github.GithubObject.NotSet, required_approving_review_count=github.GithubObject.NotSet, user_push_restrictions=github.GithubObject.NotSet, team_push_restrictions=github.GithubObject.NotSet):\n assert strict is github.GithubObject.NotSet or isinstance(strict, bool), strict\n assert contexts is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in contexts), contexts\n assert enforce_admins is github.GithubObject.NotSet or isinstance(enforce_admins, bool), enforce_admins\n assert dismissal_users is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in dismissal_users), dismissal_users\n assert dismissal_teams is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in dismissal_teams), dismissal_teams\n assert dismiss_stale_reviews is github.GithubObject.NotSet or isinstance(dismiss_stale_reviews, bool), dismiss_stale_reviews\n assert require_code_owner_reviews is github.GithubObject.NotSet or isinstance(require_code_owner_reviews, bool), require_code_owner_reviews\n assert required_approving_review_count is github.GithubObject.NotSet or isinstance(required_approving_review_count, int), required_approving_review_count\n\n post_parameters = {}\n if strict is not github.GithubObject.NotSet or contexts is not github.GithubObject.NotSet:\n if strict is github.GithubObject.NotSet:\n strict = False\n if contexts is github.GithubObject.NotSet:\n contexts = []\n post_parameters[\"required_status_checks\"] = {\"strict\": strict, \"contexts\": contexts}\n else:\n post_parameters[\"required_status_checks\"] = None\n\n if enforce_admins is not github.GithubObject.NotSet:\n post_parameters[\"enforce_admins\"] = enforce_admins\n else:\n post_parameters[\"enforce_admins\"] = None\n\n if dismissal_users is not github.GithubObject.NotSet or dismissal_teams is not github.GithubObject.NotSet or dismiss_stale_reviews is not github.GithubObject.NotSet or require_code_owner_reviews is not github.GithubObject.NotSet or required_approving_review_count is not github.GithubObject.NotSet:\n post_parameters[\"required_pull_request_reviews\"] = {}\n if dismiss_stale_reviews is not github.GithubObject.NotSet:\n post_parameters[\"required_pull_request_reviews\"][\"dismiss_stale_reviews\"] = dismiss_stale_reviews\n if require_code_owner_reviews is not github.GithubObject.NotSet:\n post_parameters[\"required_pull_request_reviews\"][\"require_code_owner_reviews\"] = require_code_owner_reviews\n if required_approving_review_count is not github.GithubObject.NotSet:\n post_parameters[\"required_pull_request_reviews\"][\"required_approving_review_count\"] = required_approving_review_count\n if dismissal_users is not github.GithubObject.NotSet:\n post_parameters[\"required_pull_request_reviews\"][\"dismissal_restrictions\"] = {\"users\": dismissal_users}\n if dismissal_teams is not github.GithubObject.NotSet:\n if \"dismissal_restrictions\" not in post_parameters[\"required_pull_request_reviews\"]:\n post_parameters[\"required_pull_request_reviews\"][\"dismissal_restrictions\"] = {}\n post_parameters[\"required_pull_request_reviews\"][\"dismissal_restrictions\"][\"teams\"] = dismissal_teams\n else:\n post_parameters[\"required_pull_request_reviews\"] = None\n if user_push_restrictions is not github.GithubObject.NotSet or team_push_restrictions is not github.GithubObject.NotSet:\n if user_push_restrictions is github.GithubObject.NotSet:\n user_push_restrictions = []\n if team_push_restrictions is github.GithubObject.NotSet:\n team_push_restrictions = []\n post_parameters[\"restrictions\"] = {\"users\": user_push_restrictions, \"teams\": team_push_restrictions}\n else:\n post_parameters[\"restrictions\"] = None\n\n headers, data = self._requester.requestJsonAndCheck(\n \"PUT\",\n self.protection_url,\n headers={'Accept': Consts.mediaTypeRequireMultipleApprovingReviews},\n input=post_parameters\n )", "response": "Edit protection of branch."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of github. RequiredStatusChecks. RequiredStatusChecks objects for the current branch.", "response": "def get_required_status_checks(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/branches/:branch/protection/required_status_checks `_\n :rtype: :class:`github.RequiredStatusChecks.RequiredStatusChecks`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.protection_url + \"/required_status_checks\"\n )\n return github.RequiredStatusChecks.RequiredStatusChecks(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef edit_required_status_checks(self, strict=github.GithubObject.NotSet, contexts=github.GithubObject.NotSet):\n assert strict is github.GithubObject.NotSet or isinstance(strict, bool), strict\n assert contexts is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in contexts), contexts\n\n post_parameters = {}\n if strict is not github.GithubObject.NotSet:\n post_parameters[\"strict\"] = strict\n if contexts is not github.GithubObject.NotSet:\n post_parameters[\"contexts\"] = contexts\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.protection_url + \"/required_status_checks\",\n input=post_parameters\n )", "response": "Edits the required status checks for the branch."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_required_pull_request_reviews(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.protection_url + \"/required_pull_request_reviews\",\n headers={'Accept': Consts.mediaTypeRequireMultipleApprovingReviews}\n )\n return github.RequiredPullRequestReviews.RequiredPullRequestReviews(self._requester, headers, data, completed=True)", "response": "returns a list of github. RequiredPullRequestReviews. RequiredPullRequestReviews objects for the branch."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef edit_required_pull_request_reviews(self, dismissal_users=github.GithubObject.NotSet, dismissal_teams=github.GithubObject.NotSet, dismiss_stale_reviews=github.GithubObject.NotSet, require_code_owner_reviews=github.GithubObject.NotSet, required_approving_review_count=github.GithubObject.NotSet):\n assert dismissal_users is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in dismissal_users), dismissal_users\n assert dismissal_teams is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in dismissal_teams), dismissal_teams\n assert dismiss_stale_reviews is github.GithubObject.NotSet or isinstance(dismiss_stale_reviews, bool), dismiss_stale_reviews\n assert require_code_owner_reviews is github.GithubObject.NotSet or isinstance(require_code_owner_reviews, bool), require_code_owner_reviews\n assert required_approving_review_count is github.GithubObject.NotSet or isinstance(required_approving_review_count, int), required_approving_review_count\n\n post_parameters = {}\n if dismissal_users is not github.GithubObject.NotSet:\n post_parameters[\"dismissal_restrictions\"] = {\"users\": dismissal_users}\n if dismissal_teams is not github.GithubObject.NotSet:\n if \"dismissal_restrictions\" not in post_parameters:\n post_parameters[\"dismissal_restrictions\"] = {}\n post_parameters[\"dismissal_restrictions\"][\"teams\"] = dismissal_teams\n if dismiss_stale_reviews is not github.GithubObject.NotSet:\n post_parameters[\"dismiss_stale_reviews\"] = dismiss_stale_reviews\n if require_code_owner_reviews is not github.GithubObject.NotSet:\n post_parameters[\"require_code_owner_reviews\"] = require_code_owner_reviews\n if required_approving_review_count is not github.GithubObject.NotSet:\n post_parameters[\"required_approving_review_count\"] = required_approving_review_count\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.protection_url + \"/required_pull_request_reviews\",\n headers={'Accept': Consts.mediaTypeRequireMultipleApprovingReviews},\n input=post_parameters\n )", "response": "Edits the required pull request reviews."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_admin_enforcement(self):\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.protection_url + \"/enforce_admins\"\n )\n return data[\"enabled\"]", "response": "returns true if admin access is enabled for this branch"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_user_push_restrictions(self):\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self._requester,\n self.protection_url + \"/restrictions/users\",\n None\n )", "response": "Returns a paginated list of GitHub User push restrictions for the current branch."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_team_push_restrictions(self):\n return github.PaginatedList.PaginatedList(\n github.Team.Team,\n self._requester,\n self.protection_url + \"/restrictions/teams\",\n None\n )", "response": "Returns a paginated list of teams that can push the current branch."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef edit_user_push_restrictions(self, *users):\n assert all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in users), users\n\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.protection_url + \"/restrictions/users\",\n input=users\n )", "response": "Edits the push restrictions for the specified users."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef edit_team_push_restrictions(self, *teams):\n assert all(isinstance(element, (str, unicode)) or isinstance(element, (str, unicode)) for element in teams), teams\n\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.protection_url + \"/restrictions/teams\",\n input=teams\n )", "response": "Edits the team push restrictions for the current branch."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a boolean indicating if the branch is required for this protection", "response": "def get_required_signatures(self):\n \"\"\"\n :calls: `GET /repos/:owner/:repo/branches/:branch/protection/required_signatures `\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.protection_url + \"/required_signatures\",\n headers={'Accept': Consts.signaturesProtectedBranchesPreview}\n )\n return data[\"enabled\"]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the properties of the named object in the repository.", "response": "def edit(self, name, config, events=github.GithubObject.NotSet, add_events=github.GithubObject.NotSet, remove_events=github.GithubObject.NotSet, active=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /repos/:owner/:repo/hooks/:id `_\n :param name: string\n :param config: dict\n :param events: list of string\n :param add_events: list of string\n :param remove_events: list of string\n :param active: bool\n :rtype: None\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n assert isinstance(config, dict), config\n assert events is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in events), events\n assert add_events is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in add_events), add_events\n assert remove_events is github.GithubObject.NotSet or all(isinstance(element, (str, unicode)) for element in remove_events), remove_events\n assert active is github.GithubObject.NotSet or isinstance(active, bool), active\n post_parameters = {\n \"name\": name,\n \"config\": config,\n }\n if events is not github.GithubObject.NotSet:\n post_parameters[\"events\"] = events\n if add_events is not github.GithubObject.NotSet:\n post_parameters[\"add_events\"] = add_events\n if remove_events is not github.GithubObject.NotSet:\n post_parameters[\"remove_events\"] = remove_events\n if active is not github.GithubObject.NotSet:\n post_parameters[\"active\"] = active\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a paginated list of the users following this entry.", "response": "def get_followers(self):\n \"\"\"\n :calls: `GET /users/:user/followers `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n NamedUser,\n self._requester,\n self.url + \"/followers\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a paginated list of the users following this entry.", "response": "def get_following(self):\n \"\"\"\n :calls: `GET /users/:user/following `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.NamedUser.NamedUser`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n NamedUser,\n self._requester,\n self.url + \"/following\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_keys(self):\n return github.PaginatedList.PaginatedList(\n github.UserKey.UserKey,\n self._requester,\n self.url + \"/keys\",\n None\n )", "response": "Returns a paginated list of all keys owned by the user."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_public_events(self):\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n self.url + \"/events/public\",\n None\n )", "response": "Returns a paginated list of public events for the current user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a paginated list of public events for the current user.", "response": "def get_public_received_events(self):\n \"\"\"\n :calls: `GET /users/:user/received_events/public `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Event.Event`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n self.url + \"/received_events/public\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_received_events(self):\n return github.PaginatedList.PaginatedList(\n github.Event.Event,\n self._requester,\n self.url + \"/received_events\",\n None\n )", "response": "Returns a paginated list of all events that have been received for this user."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of repositories in the cluster.", "response": "def get_repos(self, type=github.GithubObject.NotSet, sort=github.GithubObject.NotSet,\n direction=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /users/:user/repos `_\n :param type: string\n :param sort: string\n :param direction: string\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Repository.Repository`\n \"\"\"\n assert type is github.GithubObject.NotSet or isinstance(type, (str, unicode)), type\n assert sort is github.GithubObject.NotSet or isinstance(sort, (str, unicode)), sort\n assert direction is github.GithubObject.NotSet or isinstance(direction, (str, unicode)), direction\n url_parameters = dict()\n if type is not github.GithubObject.NotSet:\n url_parameters[\"type\"] = type\n if sort is not github.GithubObject.NotSet:\n url_parameters[\"sort\"] = sort\n if direction is not github.GithubObject.NotSet:\n url_parameters[\"direction\"] = direction\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n self.url + \"/repos\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_subscriptions(self):\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n self.url + \"/subscriptions\",\n None\n )", "response": "Returns a paginated list of all subscriptions for the current user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a paginated list of all the users who have been watched.", "response": "def get_watched(self):\n \"\"\"\n :calls: `GET /users/:user/watched `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Repository.Repository`\n \"\"\"\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self._requester,\n self.url + \"/watched\",\n None\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the release with the given name message.", "response": "def update_release(self, name, message, draft=False, prerelease=False,\n tag_name=github.GithubObject.NotSet,\n target_commitish=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /repos/:owner/:repo/releases/:release_id `_\n :rtype: :class:`github.GitRelease.GitRelease`\n \"\"\"\n assert tag_name is github.GithubObject.NotSet \\\n or isinstance(tag_name, (str, unicode)), \\\n 'tag_name must be a str/unicode object'\n assert target_commitish is github.GithubObject.NotSet \\\n or isinstance(target_commitish, (str, unicode)), \\\n 'target_commitish must be a str/unicode object'\n assert isinstance(name, (str, unicode)), name\n assert isinstance(message, (str, unicode)), message\n assert isinstance(draft, bool), draft\n assert isinstance(prerelease, bool), prerelease\n if tag_name is github.GithubObject.NotSet:\n tag_name = self.tag_name\n post_parameters = {\n \"tag_name\": tag_name,\n \"name\": name,\n \"body\": message,\n \"draft\": draft,\n \"prerelease\": prerelease,\n }\n # Do not set target_commitish to self.target_commitish when ommited, just don't send it\n # alltogether in that case, in order to match the Github API behaviour. Only send it when set.\n if target_commitish is not github.GithubObject.NotSet:\n post_parameters['target_commitish'] = target_commitish\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n return github.GitRelease.GitRelease(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupload an asset to the release.", "response": "def upload_asset(self, path, label=\"\", content_type=\"\"):\n \"\"\"\n :calls: `POST https:///repos/:owner/:repo/releases/:release_id/assets?name=foo.zip `_\n :rtype: :class:`github.GitReleaseAsset.GitReleaseAsset`\n \"\"\"\n assert isinstance(path, (str, unicode)), path\n assert isinstance(label, (str, unicode)), label\n\n post_parameters = {\n \"name\": basename(path),\n \"label\": label\n }\n headers = {}\n if len(content_type) > 0:\n headers[\"Content-Type\"] = content_type\n resp_headers, data = self._requester.requestBlobAndCheck(\n \"POST\",\n self.upload_url.split(\"{?\")[0],\n parameters=post_parameters,\n headers=headers,\n input=path\n )\n return github.GitReleaseAsset.GitReleaseAsset(self._requester, resp_headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_assets(self):\n return github.PaginatedList.PaginatedList(\n github.GitReleaseAsset.GitReleaseAsset,\n self._requester,\n self.url + \"/assets\",\n None\n )", "response": "Returns a paginated list of assets for the current release."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the object with the given sha.", "response": "def edit(self, sha, force=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /repos/:owner/:repo/git/refs/:ref `_\n :param sha: string\n :param force: bool\n :rtype: None\n \"\"\"\n assert isinstance(sha, (str, unicode)), sha\n assert force is github.GithubObject.NotSet or isinstance(force, bool), force\n post_parameters = {\n \"sha\": sha,\n }\n if force is not github.GithubObject.NotSet:\n post_parameters[\"force\"] = force\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef enable_console_debug_logging(): # pragma no cover (Function useful only outside test environment)\n\n logger = logging.getLogger(\"github\")\n logger.setLevel(logging.DEBUG)\n logger.addHandler(logging.StreamHandler())", "response": "This function is used to enable console debug logging."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rate_limiting(self):\n remaining, limit = self.__requester.rate_limiting\n if limit < 0:\n self.get_rate_limit()\n return self.__requester.rate_limiting", "response": "Get the current rate limiting value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rate_limiting_resettime(self):\n if self.__requester.rate_limiting_resettime == 0:\n self.get_rate_limit()\n return self.__requester.rate_limiting_resettime", "response": "Returns the current time when rate limiting will reset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a RateLimit object for the current rate limit.", "response": "def get_rate_limit(self):\n \"\"\"\n Rate limit status for different resources (core/search/graphql).\n\n :calls: `GET /rate_limit `_\n :rtype: :class:`github.RateLimit.RateLimit`\n \"\"\"\n headers, data = self.__requester.requestJsonAndCheck(\n 'GET',\n '/rate_limit'\n )\n return RateLimit.RateLimit(self.__requester, headers, data[\"resources\"], True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the license with the specified key.", "response": "def get_license(self, key=github.GithubObject.NotSet):\n \"\"\"\n :calls: `GET /license/:license `_\n :param key: string\n :rtype: :class:`github.License.License`\n \"\"\"\n\n assert isinstance(key, (str, unicode)), key\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/licenses/\" + key\n )\n return github.License.License(self.__requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a paginated list of all licenses for a given locale.", "response": "def get_licenses(self):\n \"\"\"\n :calls: `GET /licenses `_\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.License.License`\n \"\"\"\n\n url_parameters = dict()\n\n return github.PaginatedList.PaginatedList(\n github.License.License,\n self.__requester,\n \"/licenses\",\n url_parameters\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_user(self, login=github.GithubObject.NotSet):\n assert login is github.GithubObject.NotSet or isinstance(login, (str, unicode)), login\n if login is github.GithubObject.NotSet:\n return AuthenticatedUser.AuthenticatedUser(self.__requester, {}, {\"url\": \"/user\"}, completed=False)\n else:\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/users/\" + login\n )\n return github.NamedUser.NamedUser(self.__requester, headers, data, completed=True)", "response": "Returns a named user object for the specified login."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a github. Organization. Organization object for the login", "response": "def get_organization(self, login):\n \"\"\"\n :calls: `GET /orgs/:org `_\n :param login: string\n :rtype: :class:`github.Organization.Organization`\n \"\"\"\n assert isinstance(login, (str, unicode)), login\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/orgs/\" + login\n )\n return github.Organization.Organization(self.__requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_organizations(self, since=github.GithubObject.NotSet):\n assert since is github.GithubObject.NotSet or isinstance(since, (int, long)), since\n url_parameters = dict()\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self.__requester,\n \"/organizations\",\n url_parameters\n )", "response": "Returns a paginated list of organizations for the given node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall get_repo for a given repository name or ID", "response": "def get_repo(self, full_name_or_id, lazy=False):\n \"\"\"\n :calls: `GET /repos/:owner/:repo `_ or `GET /repositories/:id `_\n :rtype: :class:`github.Repository.Repository`\n \"\"\"\n assert isinstance(full_name_or_id, (str, unicode, int, long)), full_name_or_id\n url_base = \"/repositories/\" if isinstance(full_name_or_id, int) or isinstance(full_name_or_id, long) else \"/repos/\"\n url = \"%s%s\" % (url_base, full_name_or_id)\n if lazy:\n return Repository.Repository(self.__requester, {}, {\"url\": url}, completed=False)\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"%s%s\" % (url_base, full_name_or_id)\n )\n return Repository.Repository(self.__requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_repos(self, since=github.GithubObject.NotSet):\n assert since is github.GithubObject.NotSet or isinstance(since, (int, long)), since\n url_parameters = dict()\n if since is not github.GithubObject.NotSet:\n url_parameters[\"since\"] = since\n return github.PaginatedList.PaginatedList(\n github.Repository.Repository,\n self.__requester,\n \"/repositories\",\n url_parameters\n )", "response": "returns a paginated list of repositories for the given node ID"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a github. Project. Project object for the given id", "response": "def get_project(self, id):\n \"\"\"\n :calls: `GET /projects/:project_id `_\n :rtype: :class:`github.Project.Project`\n :param id: integer\n \"\"\"\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/projects/%d\" % (id),\n headers={\"Accept\": Consts.mediaTypeProjectsPreview}\n )\n return github.Project.Project(self.__requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a Gist object with the given ID", "response": "def get_gist(self, id):\n \"\"\"\n :calls: `GET /gists/:id `_\n :param id: string\n :rtype: :class:`github.Gist.Gist`\n \"\"\"\n assert isinstance(id, (str, unicode)), id\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/gists/\" + id\n )\n return github.Gist.Gist(self.__requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef search_users(self, query, sort=github.GithubObject.NotSet, order=github.GithubObject.NotSet, **qualifiers):\n assert isinstance(query, (str, unicode)), query\n url_parameters = dict()\n if sort is not github.GithubObject.NotSet:\n assert sort in ('followers', 'repositories', 'joined'), sort\n url_parameters[\"sort\"] = sort\n if order is not github.GithubObject.NotSet:\n assert order in ('asc', 'desc'), order\n url_parameters[\"order\"] = order\n\n query_chunks = []\n if query:\n query_chunks.append(query)\n\n for qualifier, value in qualifiers.items():\n query_chunks.append(\"%s:%s\" % (qualifier, value))\n\n url_parameters[\"q\"] = ' '.join(query_chunks)\n assert url_parameters[\"q\"], \"need at least one qualifier\"\n\n return github.PaginatedList.PaginatedList(\n github.NamedUser.NamedUser,\n self.__requester,\n \"/search/users\",\n url_parameters\n )", "response": "returns a paginated list of github. NamedUser objects for the specified query"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef search_code(self, query, sort=github.GithubObject.NotSet, order=github.GithubObject.NotSet, highlight=False, **qualifiers):\n assert isinstance(query, (str, unicode)), query\n url_parameters = dict()\n if sort is not github.GithubObject.NotSet: # pragma no branch (Should be covered)\n assert sort in ('indexed',), sort\n url_parameters[\"sort\"] = sort\n if order is not github.GithubObject.NotSet: # pragma no branch (Should be covered)\n assert order in ('asc', 'desc'), order\n url_parameters[\"order\"] = order\n\n query_chunks = []\n if query: # pragma no branch (Should be covered)\n query_chunks.append(query)\n\n for qualifier, value in qualifiers.items():\n query_chunks.append(\"%s:%s\" % (qualifier, value))\n\n url_parameters[\"q\"] = ' '.join(query_chunks)\n assert url_parameters[\"q\"], \"need at least one qualifier\"\n\n headers = {\"Accept\": Consts.highLightSearchPreview} if highlight else None\n\n return github.PaginatedList.PaginatedList(\n github.ContentFile.ContentFile,\n self.__requester,\n \"/search/code\",\n url_parameters,\n headers=headers\n )", "response": "returns a paginated list of all the codes that match the query"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef search_commits(self, query, sort=github.GithubObject.NotSet, order=github.GithubObject.NotSet, **qualifiers):\n assert isinstance(query, (str, unicode)), query\n url_parameters = dict()\n if sort is not github.GithubObject.NotSet: # pragma no branch (Should be covered)\n assert sort in ('author-date', 'committer-date'), sort\n url_parameters[\"sort\"] = sort\n if order is not github.GithubObject.NotSet: # pragma no branch (Should be covered)\n assert order in ('asc', 'desc'), order\n url_parameters[\"order\"] = order\n\n query_chunks = []\n if query: # pragma no branch (Should be covered)\n query_chunks.append(query)\n\n for qualifier, value in qualifiers.items():\n query_chunks.append(\"%s:%s\" % (qualifier, value))\n\n url_parameters[\"q\"] = ' '.join(query_chunks)\n assert url_parameters[\"q\"], \"need at least one qualifier\"\n\n return github.PaginatedList.PaginatedList(\n github.Commit.Commit,\n self.__requester,\n \"/search/commits\",\n url_parameters,\n headers={\n \"Accept\": Consts.mediaTypeCommitSearchPreview\n }\n )", "response": "returns a paginated list of commits for the given query"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a paginated list of topics that match the query", "response": "def search_topics(self, query, **qualifiers):\n \"\"\"\n :calls: `GET /search/topics `_\n :param query: string\n :param qualifiers: keyword dict query qualifiers\n :rtype: :class:`github.PaginatedList.PaginatedList` of :class:`github.Topic.Topic`\n \"\"\"\n assert isinstance(query, (str, unicode)), query\n url_parameters = dict()\n\n query_chunks = []\n if query: # pragma no branch (Should be covered)\n query_chunks.append(query)\n\n for qualifier, value in qualifiers.items():\n query_chunks.append(\"%s:%s\" % (qualifier, value))\n\n url_parameters[\"q\"] = ' '.join(query_chunks)\n assert url_parameters[\"q\"], \"need at least one qualifier\"\n\n return github.PaginatedList.PaginatedList(\n github.Topic.Topic,\n self.__requester,\n \"/search/topics\",\n url_parameters,\n headers={\n \"Accept\": Consts.mediaTypeTopicsPreview\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef render_markdown(self, text, context=github.GithubObject.NotSet):\n assert isinstance(text, (str, unicode)), text\n assert context is github.GithubObject.NotSet or isinstance(context, github.Repository.Repository), context\n post_parameters = {\n \"text\": text\n }\n if context is not github.GithubObject.NotSet:\n post_parameters[\"mode\"] = \"gfm\"\n post_parameters[\"context\"] = context._identity\n status, headers, data = self.__requester.requestJson(\n \"POST\",\n \"/markdown\",\n input=post_parameters\n )\n return data", "response": "Renders the markdown text for the specified object."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the description of a specific hook", "response": "def get_hook(self, name):\n \"\"\"\n :calls: `GET /hooks/:name `_\n :param name: string\n :rtype: :class:`github.HookDescription.HookDescription`\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n headers, attributes = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/hooks/\" + name\n )\n return HookDescription.HookDescription(self.__requester, headers, attributes, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of github. HookDescription objects", "response": "def get_hooks(self):\n \"\"\"\n :calls: `GET /hooks `_\n :rtype: list of :class:`github.HookDescription.HookDescription`\n \"\"\"\n headers, data = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/hooks\"\n )\n return [HookDescription.HookDescription(self.__requester, headers, attributes, completed=True) for attributes in data]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a GitignoreTemplate object for the given name", "response": "def get_gitignore_template(self, name):\n \"\"\"\n :calls: `GET /gitignore/templates/:name `_\n :rtype: :class:`github.GitignoreTemplate.GitignoreTemplate`\n \"\"\"\n assert isinstance(name, (str, unicode)), name\n headers, attributes = self.__requester.requestJsonAndCheck(\n \"GET\",\n \"/gitignore/templates/\" + name\n )\n return GitignoreTemplate.GitignoreTemplate(self.__requester, headers, attributes, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an object from raw_data and optionaly headers.", "response": "def create_from_raw_data(self, klass, raw_data, headers={}):\n \"\"\"\n Creates an object from raw_data previously obtained by :attr:`github.GithubObject.GithubObject.raw_data`,\n and optionaly headers previously obtained by :attr:`github.GithubObject.GithubObject.raw_headers`.\n\n :param klass: the class of the object to create\n :param raw_data: dict\n :param headers: dict\n :rtype: instance of class ``klass``\n \"\"\"\n return klass(self.__requester, headers, raw_data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dump(self, obj, file, protocol=0):\n pickle.dump((obj.__class__, obj.raw_data, obj.raw_headers), file, protocol)", "response": "Dumps a PyGithub object to a file - like object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_jwt(self, expiration=60):\n now = int(time.time())\n payload = {\n \"iat\": now,\n \"exp\": now + expiration,\n \"iss\": self.integration_id\n }\n encrypted = jwt.encode(\n payload,\n key=self.private_key,\n algorithm=\"RS256\"\n )\n\n if atLeastPython3:\n encrypted = encrypted.decode('utf-8')\n\n return encrypted", "response": "Creates a signed JWT for the current user."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_access_token(self, installation_id, user_id=None):\n body = {}\n if user_id:\n body = {\"user_id\": user_id}\n response = requests.post(\n \"https://api.github.com/app/installations/{}/access_tokens\".format(installation_id),\n headers={\n \"Authorization\": \"Bearer {}\".format(self.create_jwt()),\n \"Accept\": Consts.mediaTypeIntegrationPreview,\n \"User-Agent\": \"PyGithub/Python\"\n },\n json=body\n )\n\n if response.status_code == 201:\n return InstallationAuthorization.InstallationAuthorization(\n requester=None, # not required, this is a NonCompletableGithubObject\n headers={}, # not required, this is a NonCompletableGithubObject\n attributes=response.json(),\n completed=True\n )\n elif response.status_code == 403:\n raise GithubException.BadCredentialsException(\n status=response.status_code,\n data=response.text\n )\n elif response.status_code == 404:\n raise GithubException.UnknownObjectException(\n status=response.status_code,\n data=response.text\n )\n raise GithubException.GithubException(\n status=response.status_code,\n data=response.text\n )", "response": "Get an access token for the given installation id."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef edit(self, name, color, description=github.GithubObject.NotSet):\n assert isinstance(name, (str, unicode)), name\n assert isinstance(color, (str, unicode)), color\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n post_parameters = {\n \"name\": name,\n \"color\": color,\n }\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters,\n headers={'Accept': Consts.mediaTypeLabelDescriptionSearchPreview}\n )\n self._useAttributes(data)", "response": "Edits the label for a specific issue."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_content(self, content_type=github.GithubObject.NotSet):\n if self.content_url == None:\n return None\n \n if content_type == \"PullRequest\":\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.content_url.replace(\"issues\", \"pulls\")\n )\n return github.PullRequest.PullRequest(self._requester, headers, data, completed=True)\n elif content_type is github.GithubObject.NotSet or content_type == \"Issue\":\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.content_url\n )\n return github.Issue.Issue(self._requester, headers, data, completed=True)\n else:\n assert False, \"Unknown content type: %s\" % content_type", "response": "returns a github. Issue or github. PullRequest object"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new fork for the current entry.", "response": "def create_fork(self):\n \"\"\"\n :calls: `POST /gists/:id/forks `_\n :rtype: :class:`github.Gist.Gist`\n \"\"\"\n headers, data = self._requester.requestJsonAndCheck(\n \"POST\",\n self.url + \"/forks\"\n )\n return Gist(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nediting the entry with the given description and files.", "response": "def edit(self, description=github.GithubObject.NotSet, files=github.GithubObject.NotSet):\n \"\"\"\n :calls: `PATCH /gists/:id `_\n :param description: string\n :param files: dict of string to :class:`github.InputFileContent.InputFileContent`\n :rtype: None\n \"\"\"\n assert description is github.GithubObject.NotSet or isinstance(description, (str, unicode)), description\n assert files is github.GithubObject.NotSet or all(element is None or isinstance(element, github.InputFileContent) for element in files.itervalues()), files\n post_parameters = dict()\n if description is not github.GithubObject.NotSet:\n post_parameters[\"description\"] = description\n if files is not github.GithubObject.NotSet:\n post_parameters[\"files\"] = {key: None if value is None else value._identity for key, value in files.iteritems()}\n headers, data = self._requester.requestJsonAndCheck(\n \"PATCH\",\n self.url,\n input=post_parameters\n )\n self._useAttributes(data)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a single comment for the given id", "response": "def get_comment(self, id):\n \"\"\"\n :calls: `GET /gists/:gist_id/comments/:id `_\n :param id: integer\n :rtype: :class:`github.GistComment.GistComment`\n \"\"\"\n assert isinstance(id, (int, long)), id\n headers, data = self._requester.requestJsonAndCheck(\n \"GET\",\n self.url + \"/comments/\" + str(id)\n )\n return github.GistComment.GistComment(self._requester, headers, data, completed=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_comments(self):\n return github.PaginatedList.PaginatedList(\n github.GistComment.GistComment,\n self._requester,\n self.url + \"/comments\",\n None\n )", "response": "Returns a paginated list of comments for the current object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_starred(self):\n status, headers, data = self._requester.requestJson(\n \"GET\",\n self.url + \"/star\"\n )\n return status == 204", "response": "returns true if the user has starred the entry in the cache"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef send_KeyEvent(self, key, down):\n self.sendMessage(struct.pack('!BBxxI', 4, down, key))", "response": "Sends a key event to the remote server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a pointer event to the master.", "response": "def send_PointerEvent(self, x, y, buttonmask=0):\n \"\"\"Indicates either pointer movement or a pointer button press or\n release. The pointer is now at (x-position, y-position),\n and the current state of buttons 1 to 8 are represented by\n bits 0 to 7 of button-mask respectively, 0 meaning up, 1\n meaning down (pressed).\n \"\"\"\n self.sendMessage(struct.pack('!BBHH', 5, buttonmask, x, y))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend a message to the client to cut it in its clipboard.", "response": "def send_ClientCutText(self, message):\n \"\"\"The client has new text in its clipboard.\n \"\"\"\n self.sendMessage(struct.pack(\"!BxxxI\", 6, len(message)))\n self.sendMessage(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_env_info(self, env_state=None, env_id=None, episode_id=None, bump_past=None, fps=None):\n with self.cv:\n if env_id is None:\n env_id = self._env_id\n if env_state is None:\n env_state = self._env_state\n if fps is None:\n fps = self._fps\n self.cv.notifyAll()\n\n old_episode_id = self._episode_id\n if self.primary:\n current_id = parse_episode_id(self._episode_id)\n # Bump when changing from resetting -> running\n if bump_past is not None:\n bump_past_id = parse_episode_id(bump_past)\n current_id = max(bump_past_id+1, current_id+1)\n elif env_state == 'resetting':\n current_id += 1\n self._episode_id = generate_episode_id(current_id)\n assert self._fps or fps\n elif episode_id is False:\n # keep the same episode_id: this is just us proactive\n # setting the state to resetting after a done=True\n pass\n else:\n assert episode_id is not None, \"No episode_id provided. This likely indicates a misbehaving server, which did not send an episode_id\"\n self._episode_id = episode_id\n self._fps = fps\n logger.info('[%s] Changing env_state: %s (env_id=%s) -> %s (env_id=%s) (episode_id: %s->%s, fps=%s)', self.label, self._env_state, self._env_id, env_state, env_id, old_episode_id, self._episode_id, self._fps)\n self._env_state = env_state\n if env_id is not None:\n self._env_id = env_id\n\n return self.env_info()", "response": "Set the environment state tracking variables."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_ps(cls, client, dictionary, **kwargs):\n name = get_container_name(dictionary)\n if name is None:\n return None\n\n new_dictionary = {\n 'Id': dictionary['Id'],\n 'Image': dictionary['Image'],\n 'Name': '/' + name,\n }\n return cls(client, new_dictionary, **kwargs)", "response": "Construct a container object from the dictionary returned by the GET / containers API."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nattaching a log stream to the container.", "response": "def attach_log_stream(self):\n \"\"\"A log stream can only be attached if the container uses a json-file\n log driver.\n \"\"\"\n if self.has_api_logs:\n self.log_stream = self.attach(stdout=True, stderr=True, stream=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a value from the container or None if the value is not set.", "response": "def get(self, key):\n \"\"\"Return a value from the container or None if the value is not set.\n\n :param key: a string using dotted notation for nested dictionary\n lookups\n \"\"\"\n self.inspect_if_not_inspected()\n\n def get_value(dictionary, key):\n return (dictionary or {}).get(key)\n\n return reduce(get_value, key.split('.'), self.dictionary)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rename_to_tmp_name(self):\n self.client.rename(\n self.id,\n '%s_%s' % (self.short_id, self.name)\n )", "response": "Rename the container to a hopefully unique temporary container name by prepending the short id."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds a key combination such as ctrl - t .", "response": "def build(cls, keys, down=None):\n \"\"\"Build a key combination, such as:\n\n ctrl-t\n \"\"\"\n codes = []\n for key in keys.split('-'):\n key = keycode(key)\n codes.append(key)\n\n events = []\n if down is None or down:\n for code in codes:\n events.append(cls(code, down=True))\n\n if down is None or not down:\n for code in reversed(codes):\n events.append(cls(code, down=False))\n return events"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef runtime_spec(self, id):\n try:\n return self.runtimes[id]\n except KeyError:\n raise UnregisteredRuntime('No registered runtime with name: {}'.format(id))", "response": "Returns a DockerRuntime object describing the specified runtime"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nuse in the tests", "response": "def _manual_recv(self, method, body, headers={}):\n \"\"\"Used in the tests\"\"\"\n headers.setdefault('sent_at', time.time())\n return self.recv(self._make_context(), {'method': method, 'body': body, 'headers': headers})"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef exit_on_signal():\n def shutdown(signal, frame):\n logger.warn('Received signal %s: exiting', signal)\n sys.exit(128+signal)\n signal.signal(signal.SIGHUP, shutdown)\n signal.signal(signal.SIGINT, shutdown)\n signal.signal(signal.SIGTERM, shutdown)", "response": "Install a signal handler for HUP INT and TERM to call exit allowing clean shutdown."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread size bytes from the file and return them as a string.", "response": "def read_safe(self, size=None):\n \"\"\"\n We currently close our fbs files by killing them, so sometimes they end\n up with bad data at the end. Close our reader if we expect `size` bytes\n and get fewer.\n\n This is a hack and should be removed when we cleanly close our\n connections in fbs_writer.\n\n https://github.com/openai/universe-envs/issues/41\n \"\"\"\n bytes = self.file.read(size)\n if len(bytes) != size:\n # We unexpectedly got to the end of the file\n self.close()\n raise StopIteration\n return bytes"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalls from main thread. Returns the list of newly - available ( handle env ) pairs.", "response": "def pop(self, n=None):\n \"\"\"Call from main thread. Returns the list of newly-available (handle, env) pairs.\"\"\"\n self.error_buffer.check()\n\n envs = []\n\n if n is None:\n while True:\n try:\n envs += self.ready.get(block=False)\n except queue.Empty:\n break\n else:\n sync_timeout = 10 * 60\n start = time.time()\n\n wait_time = 1\n while len(envs) < n:\n try:\n extra_logger.info('[%s] Waiting for %d envs, currently at %d, sleeping for %d', self.label, n, len(envs), wait_time)\n envs += self.ready.get(timeout=wait_time)\n except queue.Empty:\n self.error_buffer.check()\n wait_time = min(wait_time * 2, 30)\n delta = time.time() - start\n if delta > sync_timeout:\n raise FatalError(\"Waited %.0fs to obtain envs, timeout was %.0fs. (Obtained %d/%d envs.)\" % (delta, sync_timeout, len(envs), n))\n\n return envs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef allocate(self, handles, initial=False, params={}):\n assert all(re.search('^\\d+$', h) for h in handles), \"All handles must be numbers: {}\".format(handles)\n self.requests.put(('allocate', (handles, initial, params)))", "response": "Initiate a request for more environments"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef configure(self, remotes=None,\n client_id=None,\n start_timeout=None, docker_image=None,\n ignore_clock_skew=False, disable_action_probes=False,\n vnc_driver=None, vnc_kwargs=None,\n rewarder_driver=None,\n replace_on_crash=False, allocate_sync=True,\n observer=False, api_key=None,\n record=False,\n sample_env_ids=None,\n ):\n \"\"\"Universe method to configure the environment.\n\n Args:\n\n ignore_clock_skew (bool): Assume remotes are on the same machine as us,\n for the purposes of diagnostics measurement.\n\n If true, we skip measuring the clock skew over the network,\n and skip generating diagnostics which rely on it.\n\n True when used by the rewarder to measure latency between\n the VNC frame and its calculation of reward for that\n frame. In this case we share a common clock with the env\n generating the VNC frame, so we don't need to send/receive\n probes. Clock skew is zero in this case.\n\n False when remotes are potentially different machines\n (such as an agent, or a demonstrator), and we will be\n sending probe keys and measuring network ping rountrip\n times to calculate clock skew.\n \"\"\"\n if self._started:\n raise error.Error('{} has already been started; cannot change configuration now.'.format(self))\n\n universe.configure_logging()\n\n twisty.start_once()\n\n if self.spec is not None:\n runtime = registration.runtime_spec(self.spec.tags['runtime'])\n # Let the user manually set the docker_image version\n if docker_image:\n # TODO: don't support this option?\n runtime.image = docker_image\n else:\n runtime = None\n\n if remotes is None:\n remotes = os.environ.get('GYM_VNC_REMOTES', '1')\n\n if client_id is None:\n client_id = default_client_id()\n\n if vnc_kwargs is None:\n vnc_kwargs = {}\n\n self.remote_manager, self.n = remotes_module.build(\n client_id=client_id,\n remotes=remotes, runtime=runtime, start_timeout=start_timeout,\n api_key=api_key,\n use_recorder_ports=record,\n )\n self.connection_names = [None] * self.n\n self.connection_labels = [None] * self.n\n self.crashed = {}\n\n self.allow_reconnect = replace_on_crash and self.remote_manager.supports_reconnect\n if self.remote_manager.connect_vnc:\n cls = vnc_session(vnc_driver)\n vnc_kwargs.setdefault('start_timeout', self.remote_manager.start_timeout)\n if runtime == 'gym-core':\n vnc_kwargs.setdefault('encoding', 'zrle')\n else:\n vnc_kwargs.setdefault('encoding', 'tight')\n vnc_kwargs.setdefault('fine_quality_level', 50)\n vnc_kwargs.setdefault('subsample_level', 2)\n # Filter out None values, since some drivers may not handle them correctly\n vnc_kwargs = {k: v for k, v in vnc_kwargs.items() if v is not None}\n logger.info('Using VNCSession arguments: %s. (Customize by running \"env.configure(vnc_kwargs={...})\"', vnc_kwargs)\n self.vnc_kwargs = vnc_kwargs\n self.vnc_session = cls()\n else:\n self.vnc_session = None\n\n self._observer = observer\n if self.remote_manager.connect_rewarder:\n cls = rewarder_session(rewarder_driver)\n self.rewarder_session = cls()\n else:\n self.rewarder_session = None\n\n if ignore_clock_skew:\n logger.info('Printed stats will ignore clock skew. (This usually makes sense only when the environment and agent are on the same machine.)')\n\n if self.rewarder_session or ignore_clock_skew:\n # Don't need rewarder session if we're ignoring clock skew\n if self.spec is not None:\n metadata_encoding = self.spec.tags.get('metadata_encoding')\n else:\n metadata_encoding = None\n self.diagnostics = diagnostics.Diagnostics(self.n, self._probe_key, ignore_clock_skew, metadata_encoding=metadata_encoding, disable_action_probes=disable_action_probes)\n else:\n self.diagnostics = None\n\n self._sample_env_ids = sample_env_ids\n\n self._reset_mask()\n self._started = True\n\n self.remote_manager.allocate([str(i) for i in range(self.n)], initial=True)\n if allocate_sync:\n # Block until we've fulfilled n environments\n self._handle_connect(n=self.n)\n else:\n # Handle any backends which synchronously fufill their\n # allocation.\n self._handle_connect()", "response": "Universe method to configure the environment.\n\n Args:\n\n ignore_clock_skew (bool): Assume remotes are on the same machine as us,\n for the purposes of diagnostics measurement.\n\n If true, we skip measuring the clock skew over the network,\n and skip generating diagnostics which rely on it.\n\n True when used by the rewarder to measure latency between\n the VNC frame and its calculation of reward for that\n frame. In this case we share a common clock with the env\n generating the VNC frame, so we don't need to send/receive\n probes. Clock skew is zero in this case.\n\n False when remotes are potentially different machines\n (such as an agent, or a demonstrator), and we will be\n sending probe keys and measuring network ping rountrip\n times to calculate clock skew."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the cpu times of all processes in the current process.", "response": "def cpu_times(self):\n ''' return {pid: {'user': 0.0, 'sys': 0.0}}, chrome_reset '''\n chrome_procs = self.get_chrome_procs()\n new_pids = {p.pid for p in chrome_procs}\n old_pids = {pid for pid in self.last_cpu_times}\n try:\n cpu_times = {p.pid: p.cpu_times() for p in chrome_procs}\n except psutil.NoSuchProcess:\n # Chrome restarted since fetching the new pids above. Better luck next time.\n return {}, True\n if new_pids != old_pids:\n # We don't know when the Chrome procs were restarted, so don't\n # return elapsed time until next run.\n self.last_cpu_times = cpu_times\n return {}, True\n # Same chrome pids as last run: measure the elapsed cpu times\n ordered_old_times = (self.last_cpu_times[p.pid] for p in chrome_procs)\n ordered_new_times = (cpu_times[p.pid] for p in chrome_procs)\n cpu_times_diff = {p.pid: {'user': (t[0] - l[0]) / self.interval, 'sys': (t[1] - l[1]) / self.interval}\n for (p, t, l) in zip(chrome_procs, ordered_new_times, ordered_old_times)}\n self.last_cpu_times = cpu_times\n return cpu_times_diff, False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef merge_infos(info1, info2):\n for key, value in six.iteritems(info2):\n if key in info1 and key.startswith('stats'):\n if key.startswith('stats.timers'):\n # timer\n info1[key] += value\n elif key.startswith('stats.gauges'):\n # gauge\n info1[key] = value\n else:\n # counter\n info1[key] += value\n else:\n info1[key] = value", "response": "Merge two info dicts together."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _merge_observation(accum_observation, observation):\n if observation is None:\n # We're currently masking. So accum_observation probably\n # belongs to the previous episode. We may lose a \"text\"\n # observation from the previous episode, but that's ok.\n return None\n elif accum_observation is None:\n # Nothing to merge together\n return observation\n\n accum_observation['vision'] = observation.get('vision')\n accum_observation['text'] = accum_observation.get('text', []) + observation.get('text', [])\n return accum_observation", "response": "Merge an observation into a single element."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbegins recording the rewards.", "response": "def begin_recording(self):\n \"\"\"\n Open the file and write the metadata header to describe this recording. Called after we establish an end-to-end connection\n This uses Version 1 of our protocol\n\n Version 0 can be seen here: https://github.com/openai/universe/blob/f85a7779c3847fa86ec7bb513a1da0d3158dda78/bin/recording_agent.py\n \"\"\"\n logger.info(\"[RewardProxyServer] [%d] Starting recording\", self.id)\n\n if self._closed:\n logger.error(\n \"[RewardProxyServer] [%d] Attempted to start writing although client connection is already closed. Aborting\", self.id)\n self.close()\n return\n\n if self._n_open_files != 0:\n logger.error(\"[RewardProxyServer] [%d] WARNING: n open rewards files = %s. This is unexpected. Dropping connection.\", self.id, self._n_open_files)\n self.close()\n return\n\n logfile_path = os.path.join(self.factory.logfile_dir, 'rewards.demo')\n logger.info('Recording to {}'.format(logfile_path))\n self.file = open(logfile_path, 'w')\n\n self._n_open_files += 1\n logger.info(\"[RewardProxyServer] [%d] n open rewards files incremented: %s\", self.id, self._n_open_files)\n\n self.file.write(json.dumps({\n 'version': 1,\n '_debug_version': '0.0.1', # Give this an internal version for debugging corrupt reward.demo files # TODO, pull this from setup.py or the host docker image\n }))\n self.file.write('\\n')\n self.file.flush()\n\n logger.info(\"[RewardProxyServer] [%d] Wrote version number\", self.id)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrecords a message to our rewards. demo file if it is opened", "response": "def record_message(self, msg, from_rewarder):\n \"\"\"Record a message to our rewards.demo file if it is has been opened\"\"\"\n if self.file:\n # Include an authoritative timestamp (because the `sent_at` from the server is likely to be different\n timestamped_message = {\n 'timestamp': time.time(),\n 'message': json.loads(msg),\n 'from_rewarder': from_rewarder,\n }\n self.file.write(json.dumps(timestamped_message))\n self.file.write('\\n')\n self.file.flush()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntakes a list of time_m_2 and returns a string a_mean - b_mean flooring out at 0.", "response": "def display_timestamps_pair_compact(time_m_2):\n \"\"\"Takes a list of the following form: [(a1, b1), (a2, b2), ...] and\n returns a string a_mean-b_mean, flooring out at 0.\n \"\"\"\n if len(time_m_2) == 0:\n return '(empty)'\n\n time_m_2 = np.array(time_m_2)\n\n low = time_m_2[:, 0].mean()\n high = time_m_2[:, 1].mean()\n\n low = max(low, 0)\n\n # Not sure if this'll always be true, and not worth crashing over\n if high < 0:\n logger.warn('Harmless warning: upper-bound on clock skew is negative: (%s, %s). Please let Greg know about this.', low, high)\n\n return '{}-{}'.format(display_timestamp(low), display_timestamp(high))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef display_timestamps_pair(time_m_2):\n if len(time_m_2) == 0:\n return '(empty)'\n\n time_m_2 = np.array(time_m_2)\n return '({}, {})'.format(\n display_timestamps(time_m_2[:, 0]),\n display_timestamps(time_m_2[:, 1]),\n )", "response": "Takes a list of time_m_2 and returns a string that contains the timestamps of the two times."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef modifiers_string(modifiers):\n '''Return a string describing a set of modifiers.\n\n Example::\n\n >>> modifiers_string(MOD_SHIFT | MOD_CTRL)\n 'MOD_SHIFT|MOD_CTRL'\n\n :Parameters:\n `modifiers` : int\n Bitwise combination of modifier constants.\n\n :rtype: str\n '''\n mod_names = []\n if modifiers & MOD_SHIFT:\n mod_names.append('MOD_SHIFT')\n if modifiers & MOD_CTRL:\n mod_names.append('MOD_CTRL')\n if modifiers & MOD_ALT:\n mod_names.append('MOD_ALT')\n if modifiers & MOD_CAPSLOCK:\n mod_names.append('MOD_CAPSLOCK')\n if modifiers & MOD_NUMLOCK:\n mod_names.append('MOD_NUMLOCK')\n if modifiers & MOD_SCROLLLOCK:\n mod_names.append('MOD_SCROLLLOCK')\n if modifiers & MOD_COMMAND:\n mod_names.append('MOD_COMMAND')\n if modifiers & MOD_OPTION:\n mod_names.append('MOD_OPTION')\n if modifiers & MOD_FUNCTION:\n mod_names.append('MOD_FUNCTION')\n return '|'.join(mod_names)", "response": "Return a string describing a set of modifiers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_metadata(self, observation_n, info_n, available_at=None):\n if self.instance_n is None:\n return\n\n with pyprofile.push('vnc_env.diagnostics.Diagnostics.add_metadata'):\n async = self.pool.imap_unordered(\n self._add_metadata_i,\n zip(self.instance_n, observation_n, info_n, [available_at] * len(observation_n)))\n list(async)", "response": "Mutates the info_n dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_metadata(self, observation, info, available_at=None):\n observation = observation['vision']\n if observation is None: return\n if self.network is not None and not self.network.active():\n return\n elif self.metadata_decoder is None:\n return\n elif observation is None:\n return\n # should return a dict with now/probe_received_at keys\n with pyprofile.push('vnc_env.diagnostics.DiagnosticsInstance.add_metadata.decode'):\n metadata = self.metadata_decoder.decode(observation, available_at=available_at)\n\n if metadata is False:\n # No metadata ready, though it doesn't mean parsing failed\n metadata = None\n elif metadata is None:\n if self.could_read_metadata:\n self.could_read_metadata = False\n extra_logger.info('[%s] Stopped being able to read metadata (expected when environment resets)', self.label)\n elif not self.could_read_metadata:\n self.could_read_metadata = True\n extra_logger.info('[%s] Started being able to read metadata', self.label)\n\n if self.metadata_decoder.flag_synchronous and metadata is not None:\n info['diagnostics.image_remote_time'] = metadata['now']\n\n local_now = time.time()\n\n if self.network is None:\n # Assume the clock skew is zero. Should only be run on the\n # same machine as the VNC server, such as the universe\n # instance inside of the environmenth containers.\n real_clock_skew = self.zero_clock_skew\n else:\n # Note: this is a 2-length vector of (min, max), so anything added to\n # it is also going to be a 2-length vector.\n # Most of the diagnostics below are, but you have to look carefully.\n real_clock_skew = self.network.reversed_clock_skew()\n\n # Store real clock skew here\n info['stats.gauges.diagnostics.clock_skew'] = real_clock_skew\n if self.ignore_clock_skew:\n clock_skew = self.zero_clock_skew\n else:\n clock_skew = real_clock_skew\n\n if metadata is not None:\n # We'll generally update the observation timestamp infrequently\n if self.last_observation_timestamp == metadata['now']:\n delta = None\n else:\n # We just got a new timestamp in the observation!\n self.last_observation_timestamp = metadata['now']\n observation_now = metadata['now']\n delta = observation_now - metadata['available_at']\n\n # Subtract *local* time it was received from the *remote* time\n # displayed. Negate and reverse order to fix time ordering.\n info['stats.gauges.diagnostics.lag.observation'] = -(delta + clock_skew)[[1, 0]]\n\n # if self.network is None:\n # # The rest of diagnostics need the network, so we're done here\n # return\n\n probe_received_at = metadata['probe_received_at']\n if probe_received_at == 0 or self.disable_action_probes:\n # Happens when the env first starts\n self.probe_received_at = None\n elif self.probe_received_at is None: # this also would work for the equality case\n self.probe_received_at = probe_received_at\n elif self.probe_received_at != probe_received_at and self.probe_sent_at is None:\n logger.info('[%s] Probe is marked as received at %s, but probe_sent_at is None. This is surprising. (HINT: do you have multiple universe instances talking to the same environment?)', self.label, probe_received_at)\n elif self.probe_received_at != probe_received_at:\n extra_logger.debug('[%s] Next probe received: old=%s new=%s', self.label, self.probe_received_at, probe_received_at)\n self.probe_received_at = probe_received_at\n # Subtract the *local* time we sent it from the *remote* time it was received\n self.action_latency_skewed = probe_received_at - self.probe_sent_at\n self.probe_sent_at = None\n\n if self.action_latency_skewed:\n action_lag = self.action_latency_skewed + clock_skew\n self.action_latency_skewed = None\n else:\n action_lag = None\n info['stats.gauges.diagnostics.lag.action'] = action_lag\n\n local_now = time.time()\n # Look at when the remote believed it parsed the score (not\n # all envs send this currently).\n #\n # Also, if we received no new rewards, then this values is\n # None. This could indicate a high reward latency (bad,\n # uncommon), or that the agent is calling step faster than new\n # rewards are coming in (good, common).\n remote_score_now = info.get('rewarder.lag.observation.timestamp')\n if remote_score_now is not None:\n delta = remote_score_now - local_now\n info['stats.gauges.diagnostics.lag.reward'] = -(delta + clock_skew)[[1, 0]]\n\n # Look at when the remote send the message, so we know how\n # long it's taking for messages to get to us.\n rewarder_message_now = info.get('reward_buffer.remote_time')\n if rewarder_message_now:\n delta = rewarder_message_now - local_now\n info['stats.gauges.diagnostics.lag.rewarder_message'] = -(delta + clock_skew)[[1, 0]]", "response": "Extract metadata from a pixel observation and add it to the info dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _print_if_needed(self):\n if self._in_txn or self.print_frequency is None:\n return\n elif self.last_export is not None and \\\n self.last_export + self.print_frequency > time.time():\n return\n\n self.export()", "response": "Returns the current version of the record if it is needed."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a generator which yields strings and a splitter function splits all input into chunks of size size.", "response": "def split_buffer(stream, splitter=None, decoder=lambda a: a):\n \"\"\"Given a generator which yields strings and a splitter function,\n joins all input, splits on the separator and yields each chunk.\n\n Unlike string.split(), each chunk includes the trailing\n separator, except for the last one if none was found on the end\n of the input.\n \"\"\"\n splitter = splitter or line_splitter\n buffered = six.text_type('')\n\n for data in stream_as_text(stream):\n buffered += data\n while True:\n buffer_split = splitter(buffered)\n if buffer_split is None:\n break\n\n item, buffered = buffer_split\n yield item\n\n if buffered:\n yield decoder(buffered)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconfigures the logging for the current user.", "response": "def configure_logging(path=None):\n \"\"\"\n Set up log levels, and split verbose logs to a file\n\n Configure the client-side environment logs to print\n to stdout at \"info\" level, and also to print to a\n verbose log file located at /tmp/universe-.log\n or another path you specify at \"debug\" level.\n We suggest calling this method at the beginning of\n your script.\n \"\"\"\n\n global _logging_configured\n if _logging_configured:\n return\n _logging_configured = True\n\n if path is False:\n # Disable logfile\n return\n elif path is None:\n path = '/tmp/universe-{}.log'.format(os.getpid())\n\n logger.info('Writing logs to file: %s', path)\n # Turn up extra_logger level\n extra_logger.setLevel(logging.DEBUG)\n if path == '-':\n return\n\n # Add file handler to root logger\n root_logger = logging.getLogger()\n formatter = logging.Formatter('[%(asctime)s] %(message)s')\n handler = logging.FileHandler(path, 'w', encoding='UTF-8')\n handler.setFormatter(formatter)\n root_logger.addHandler(handler)\n\n # Set extra_logger to *only* use file handler\n extra_logger.propagate = False\n extra_logger.addHandler(handler)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_client():\n info = {}\n host = os.environ.get('DOCKER_HOST')\n net_host = os.environ.get('DOCKER_NET_HOST')\n\n client_api_version = os.environ.get('DOCKER_API_VERSION')\n if not client_api_version:\n client_api_version = \"auto\"\n\n # IP to use for started containers\n if net_host:\n info['host'] = net_host\n elif host:\n info['host'] = urlparse.urlparse(host).netloc.split(':')[0]\n else:\n info['host'] = 'localhost'\n\n verify = os.environ.get('DOCKER_TLS_VERIFY') == '1'\n if verify: # use TLS\n assert_hostname = None\n cert_path = os.environ.get('DOCKER_CERT_PATH')\n if cert_path:\n client_cert = (os.path.join(cert_path, 'cert.pem'), os.path.join(cert_path, 'key.pem'))\n ca_cert = os.path.join(cert_path, 'ca.pem')\n else:\n client_cert = ca_cert = None\n\n tls_config = docker.tls.TLSConfig(\n client_cert=client_cert,\n ca_cert=ca_cert,\n verify=verify,\n assert_hostname=assert_hostname,\n )\n return docker.Client(base_url=host, tls=tls_config, version=client_api_version), info\n else:\n return docker.Client(base_url=host, version=client_api_version), info", "response": "Returns a docker client object for the given container."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pop(self, n=None):\n if self._popped:\n assert n is None\n return []\n self._popped = True\n\n envs = []\n for i, instance in enumerate(self.instances):\n env = remote.Remote(\n handle=self._handles[i],\n vnc_address='{}:{}'.format(instance.host, instance.vnc_port),\n vnc_password='openai',\n rewarder_address='{}:{}'.format(instance.host, instance.rewarder_port),\n rewarder_password='openai',\n )\n envs.append(env)\n return envs", "response": "Call from main thread. Returns the list of newly - available ( handle env ) pairs."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_writer(self, i):\n if self._recording_dir is None:\n return None\n if self._log_n is None:\n self._log_n = [None] * self.n\n if self._log_n[i] is None:\n self._log_n[i] = RecordingWriter(self._recording_dir, self._instance_id, i, async_write=self._async_write)\n return self._log_n[i]", "response": "Returns a RecordingWriter instance for the specified recording level."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _past_limit(self):\n if self._max_episode_steps is not None and self._max_episode_steps <= self._elapsed_steps:\n logger.debug(\"Env has passed the step limit defined by TimeLimit.\")\n return True\n\n if self._max_episode_seconds is not None and self._max_episode_seconds <= self._elapsed_seconds:\n logger.debug(\"Env has passed the seconds limit defined by TimeLimit.\")\n return True\n\n return False", "response": "Return true if we are past our limit."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef recvProxyData(self, data):\n if self.initialized:\n self.sendData(data)\n else:\n self.queued_data.append(data)", "response": "Write data to server if necessary."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sendData(self, data):\n # Not set up yet\n if self.client_log is None:\n self.client_log_buffer.append(data)\n else:\n self.client_log.write(data)\n self.transport.write(data)", "response": "Write data to the server."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef CropObservations(env):\n if env.spec.tags.get('flashgames', False):\n spec = runtime_spec('flashgames').server_registry[env.spec.id]\n return _CropObservations(env, x=18, y=84, height=spec[\"height\"], width=spec[\"width\"])\n elif (env.spec.tags.get('atari', False) and env.spec.tags.get('vnc', False)):\n return _CropObservations(env, height=194, width=160)\n else:\n # if unknown environment (or local atari), do nothing\n return env", "response": "Crops the visual observations of an environment so that they only contain the game screen."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apply_vnc_actions(self, vnc_actions):\n for event in vnc_actions:\n if isinstance(event, spaces.KeyEvent):\n if event.down:\n self._down_keysyms.add(event.key)\n else:\n self._down_keysyms.discard(event.key)\n\n logger.debug(\"AtariKeyState._down_keysyms: {}\".format(self._down_keysyms))", "response": "Apply a list of vnc_actions forward over the current keysyms state."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef recv_rpc(self, context, payload):\n logger.debug(\"Adding RPC payload to ControlBuffer queue: %s\", payload)\n self.buf.put(('rpc', (context, payload)))\n with self.cv:\n self.cv.notifyAll()", "response": "Called from any thread to add an RPC to the ControlBuffer queue."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get(self, *args, **kwargs):\n payload = self.buf.get(*args, **kwargs)\n logger.debug(\"Removing RPC payload from ControlBuffer queue: %s\", payload)\n return payload", "response": "Call from main thread."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends text to the agent", "response": "def send_env_text(self, text, episode_id):\n ''' text channel to communicate with the agent '''\n reactor.callFromThread(self._send_env_text, text, episode_id)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning an iterable of functions that can be used to format the logs output of a container.", "response": "def build_log_presenters(service_names, monochrome):\n \"\"\"Return an iterable of functions.\n\n Each function can be used to format the logs output of a container.\n \"\"\"\n prefix_width = max_name_width(service_names)\n\n def no_color(text):\n return text\n\n for color_func in cycle([no_color] if monochrome else colors.rainbow()):\n yield LogPresenter(prefix_width, color_func)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconsuming the queue by reading lines off of it and yielding them.", "response": "def consume_queue(queue, cascade_stop):\n \"\"\"Consume the queue by reading lines off of it and yielding them.\"\"\"\n while True:\n try:\n item = queue.get(timeout=0.1)\n except Empty:\n yield None\n continue\n # See https://github.com/docker/compose/issues/189\n except thread.error:\n raise ShutdownException()\n\n if item.exc:\n raise item.exc\n\n if item.is_stop:\n if cascade_stop:\n raise StopIteration\n else:\n continue\n\n yield item.item"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef setPadding(self, pad):\n\t\tif pad is not None:\n\t\t\tpad = self._guardAgainstUnicode(pad)\n\t\tself._padding = pad", "response": "setPadding - Set padding character."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef setIV(self, IV):\n\t\tif not IV or len(IV) != self.block_size:\n\t\t\traise ValueError(\"Invalid Initial Value (IV), must be a multiple of \" + str(self.block_size) + \" bytes\")\n\t\tIV = self._guardAgainstUnicode(IV)\n\t\tself._iv = IV", "response": "Will set the Initial Value used in conjunction with CBC mode"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __String_to_BitList(self, data):\n\t\tif _pythonMajorVersion < 3:\n\t\t\t# Turn the strings into integers. Python 3 uses a bytes\n\t\t\t# class, which already has this behaviour.\n\t\t\tdata = [ord(c) for c in data]\n\t\tl = len(data) * 8\n\t\tresult = [0] * l\n\t\tpos = 0\n\t\tfor ch in data:\n\t\t\ti = 7\n\t\t\twhile i >= 0:\n\t\t\t\tif ch & (1 << i) != 0:\n\t\t\t\t\tresult[pos] = 1\n\t\t\t\telse:\n\t\t\t\t\tresult[pos] = 0\n\t\t\t\tpos += 1\n\t\t\t\ti -= 1\n\n\t\treturn result", "response": "Turn the string data into a list of bits ( 1 0 ) s."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nturning the list of bits -> data into a string", "response": "def __BitList_to_String(self, data):\n\t\t\"\"\"Turn the list of bits -> data, into a string\"\"\"\n\t\tresult = []\n\t\tpos = 0\n\t\tc = 0\n\t\twhile pos < len(data):\n\t\t\tc += data[pos] << (7 - (pos % 8))\n\t\t\tif (pos % 8) == 7:\n\t\t\t\tresult.append(c)\n\t\t\t\tc = 0\n\t\t\tpos += 1\n\n\t\tif _pythonMajorVersion < 3:\n\t\t\treturn ''.join([ chr(c) for c in result ])\n\t\telse:\n\t\t\treturn bytes(result)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __create_sub_keys(self):\n\t\tkey = self.__permutate(des.__pc1, self.__String_to_BitList(self.getKey()))\n\t\ti = 0\n\t\t# Split into Left and Right sections\n\t\tself.L = key[:28]\n\t\tself.R = key[28:]\n\t\twhile i < 16:\n\t\t\tj = 0\n\t\t\t# Perform circular left shifts\n\t\t\twhile j < des.__left_rotations[i]:\n\t\t\t\tself.L.append(self.L[0])\n\t\t\t\tdel self.L[0]\n\n\t\t\t\tself.R.append(self.R[0])\n\t\t\t\tdel self.R[0]\n\n\t\t\t\tj += 1\n\n\t\t\t# Create one of the 16 subkeys through pc2 permutation\n\t\t\tself.Kn[i] = self.__permutate(des.__pc2, self.L + self.R)\n\n\t\t\ti += 1", "response": "Create the 16 subkeys K1 to K[16 from the given key"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encrypt(self, data, pad=None, padmode=None):\n\t\tdata = self._guardAgainstUnicode(data)\n\t\tif pad is not None:\n\t\t\tpad = self._guardAgainstUnicode(pad)\n\t\tdata = self._padData(data, pad, padmode)\n\t\treturn self.crypt(data, des.ENCRYPT)", "response": "Encrypt the data with the key."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef decrypt(self, data, pad=None, padmode=None):\n\t\tdata = self._guardAgainstUnicode(data)\n\t\tif pad is not None:\n\t\t\tpad = self._guardAgainstUnicode(pad)\n\t\tdata = self.crypt(data, des.DECRYPT)\n\t\treturn self._unpadData(data, pad, padmode)", "response": "decrypt(data, [pad], [padmode]) -> bytes\n\n\t\tdata : Bytes to be encrypted\n\t\tpad : Optional argument for decryption padding. Must only be one byte\n\t\tpadmode : Optional argument for overriding the padding mode.\n\n\t\tThe data must be a multiple of 8 bytes and will be decrypted\n\t\twith the already specified key. In PAD_NORMAL mode, if the\n\t\toptional padding character is supplied, then the un-encrypted\n\t\tdata will have the padding characters removed from the end of\n\t\tthe bytes. This pad removal only occurs on the last 8 bytes of\n\t\tthe data (last data block). In PAD_PKCS5 mode, the special\n\t\tpadding end markers will be removed from the data after decrypting."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setKey(self, key):\n\t\tself.key_size = 24 # Use DES-EDE3 mode\n\t\tif len(key) != self.key_size:\n\t\t\tif len(key) == 16: # Use DES-EDE2 mode\n\t\t\t\tself.key_size = 16\n\t\t\telse:\n\t\t\t\traise ValueError(\"Invalid triple DES key size. Key must be either 16 or 24 bytes long\")\n\t\tif self.getMode() == CBC:\n\t\t\tif not self.getIV():\n\t\t\t\t# Use the first 8 bytes of the key\n\t\t\t\tself._iv = key[:self.block_size]\n\t\t\tif len(self.getIV()) != self.block_size:\n\t\t\t\traise ValueError(\"Invalid IV, must be 8 bytes in length\")\n\t\tself.__key1 = des(key[:8], self._mode, self._iv,\n\t\t\t\t self._padding, self._padmode)\n\t\tself.__key2 = des(key[8:16], self._mode, self._iv,\n\t\t\t\t self._padding, self._padmode)\n\t\tif self.key_size == 16:\n\t\t\tself.__key3 = self.__key1\n\t\telse:\n\t\t\tself.__key3 = des(key[16:], self._mode, self._iv,\n\t\t\t\t\t self._padding, self._padmode)\n\t\t_baseDes.setKey(self, key)", "response": "Will set the crypting key for this object. Either 16 or 24 bytes long."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the type of crypting mode pyDes. ECB or pyDes. CBC", "response": "def setMode(self, mode):\n\t\t\"\"\"Sets the type of crypting mode, pyDes.ECB or pyDes.CBC\"\"\"\n\t\t_baseDes.setMode(self, mode)\n\t\tfor key in (self.__key1, self.__key2, self.__key3):\n\t\t\tkey.setMode(mode)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the type of padding mode pyDes. PAD_NORMAL or pyDes. PAD_PKCS5.", "response": "def setPadMode(self, mode):\n\t\t\"\"\"Sets the type of padding mode, pyDes.PAD_NORMAL or pyDes.PAD_PKCS5\"\"\"\n\t\t_baseDes.setPadMode(self, mode)\n\t\tfor key in (self.__key1, self.__key2, self.__key3):\n\t\t\tkey.setPadMode(mode)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef decrypt(self, data, pad=None, padmode=None):\n\t\tENCRYPT = des.ENCRYPT\n\t\tDECRYPT = des.DECRYPT\n\t\tdata = self._guardAgainstUnicode(data)\n\t\tif pad is not None:\n\t\t\tpad = self._guardAgainstUnicode(pad)\n\t\tif self.getMode() == CBC:\n\t\t\tself.__key1.setIV(self.getIV())\n\t\t\tself.__key2.setIV(self.getIV())\n\t\t\tself.__key3.setIV(self.getIV())\n\t\t\ti = 0\n\t\t\tresult = []\n\t\t\twhile i < len(data):\n\t\t\t\tiv = data[i:i+8]\n\t\t\t\tblock = self.__key3.crypt(iv, DECRYPT)\n\t\t\t\tblock = self.__key2.crypt(block, ENCRYPT)\n\t\t\t\tblock = self.__key1.crypt(block, DECRYPT)\n\t\t\t\tself.__key1.setIV(iv)\n\t\t\t\tself.__key2.setIV(iv)\n\t\t\t\tself.__key3.setIV(iv)\n\t\t\t\tresult.append(block)\n\t\t\t\ti += 8\n\t\t\tif _pythonMajorVersion < 3:\n\t\t\t\tdata = ''.join(result)\n\t\t\telse:\n\t\t\t\tdata = bytes.fromhex('').join(result)\n\t\telse:\n\t\t\tdata = self.__key3.crypt(data, DECRYPT)\n\t\t\tdata = self.__key2.crypt(data, ENCRYPT)\n\t\t\tdata = self.__key1.crypt(data, DECRYPT)\n\t\treturn self._unpadData(data, pad, padmode)", "response": "This method decrypts the data with the encryption key."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse payment result xml", "response": "def parse_payment_result(self, xml):\n \"\"\"\u89e3\u6790\u5fae\u4fe1\u652f\u4ed8\u7ed3\u679c\u901a\u77e5\"\"\"\n try:\n data = xmltodict.parse(xml)\n except (xmltodict.ParsingInterrupted, ExpatError):\n raise InvalidSignatureException()\n\n if not data or 'xml' not in data:\n raise InvalidSignatureException()\n\n data = data['xml']\n sign = data.pop('sign', None)\n real_sign = calculate_signature(data, self.api_key if not self.sandbox else self.sandbox_api_key)\n if sign != real_sign:\n raise InvalidSignatureException()\n\n for key in ('total_fee', 'settlement_total_fee', 'cash_fee', 'coupon_fee', 'coupon_count'):\n if key in data:\n data[key] = int(data[key])\n data['sign'] = sign\n return data"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_articles(self, articles):\n articles_data = []\n for article in articles:\n articles_data.append({\n 'thumb_media_id': article['thumb_media_id'],\n 'title': article['title'],\n 'content': article['content'],\n 'author': article.get('author', ''),\n 'content_source_url': article.get('content_source_url', ''),\n 'digest': article.get('digest', ''),\n 'show_cover_pic': article.get('show_cover_pic', 0),\n 'need_open_comment': int(article.get('need_open_comment', False)),\n 'only_fans_can_comment': int(article.get('only_fans_can_comment', False)),\n })\n return self._post(\n 'material/add_news',\n data={\n 'articles': articles_data\n }\n )", "response": "add_articles \u8be6\u60c5\u8bf7\u53c2\u8003 \u56fe\u6587\u7d20\u6750\u6570\u7ec4 \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\u542b\u56fe\u6587\u7d20\u6750\u6570\u7ec4"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add(self, media_type, media_file, title=None, introduction=None):\n params = {\n 'access_token': self.access_token,\n 'type': media_type\n }\n if media_type == 'video':\n assert title, 'Video title must be set'\n assert introduction, 'Video introduction must be set'\n description = {\n 'title': title,\n 'introduction': introduction\n }\n params['description'] = json.dumps(description)\n return self._post(\n 'material/add_material',\n params=params,\n files={\n 'media': media_file\n }\n )", "response": "Add a new entry to the cache."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, media_id):\n def _processor(res):\n if isinstance(res, dict):\n if 'news_item' in res:\n # \u56fe\u6587\u7d20\u6750\n return res['news_item']\n return res\n\n res = self._post(\n 'material/get_material',\n data={\n 'media_id': media_id\n },\n result_processor=_processor\n )\n return res", "response": "Get a specific entry in the cache."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the article in the national cache.", "response": "def update_article(self, media_id, index, article):\n \"\"\"\n \u4fee\u6539\u6c38\u4e45\u56fe\u6587\u7d20\u6750\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1444738732\n\n :param media_id: \u8981\u4fee\u6539\u7684\u56fe\u6587\u6d88\u606f\u7684 id\n :param index: \u8981\u66f4\u65b0\u7684\u6587\u7ae0\u5728\u56fe\u6587\u6d88\u606f\u4e2d\u7684\u4f4d\u7f6e\uff08\u591a\u56fe\u6587\u6d88\u606f\u65f6\uff0c\u6b64\u5b57\u6bb5\u624d\u6709\u610f\u4e49\uff09\uff0c\u7b2c\u4e00\u7bc7\u4e3a 0\n :param article: \u56fe\u6587\u7d20\u6750\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n article_data = {\n 'thumb_media_id': article['thumb_media_id'],\n 'title': article['title'],\n 'content': article['content'],\n 'author': article.get('author', ''),\n 'content_source_url': article.get('content_source_url', ''),\n 'digest': article.get('digest', ''),\n 'show_cover_pic': article.get('show_cover_pic', 0)\n }\n return self._post(\n 'material/update_news',\n data={\n 'media_id': media_id,\n 'index': index,\n 'articles': article_data\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_articles(self, media_id, index, articles):\n return self.update_article(media_id, index, articles[index])", "response": "\u4fee\u6539\u6c38\u4e45\u56fe\u6587\u7d20\u6750\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/4/19a59cba020d506e767360ca1be29450.html\n\n :param media_id: \u8981\u4fee\u6539\u7684\u56fe\u6587\u6d88\u606f\u7684 id\n :param index: \u8981\u66f4\u65b0\u7684\u6587\u7ae0\u5728\u56fe\u6587\u6d88\u606f\u4e2d\u7684\u4f4d\u7f6e\uff08\u591a\u56fe\u6587\u6d88\u606f\u65f6\uff0c\u6b64\u5b57\u6bb5\u624d\u6709\u610f\u4e49\uff09\uff0c\u7b2c\u4e00\u7bc7\u4e3a 0\n :param articles: \u56fe\u6587\u7d20\u6750\u6570\u7ec4\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef list_comment(self, msg_data_id, index=1, begin=0, count=50, type=0):\n return self._post(\n 'comment/list',\n data={\n 'msg_data_id': msg_data_id,\n 'index': index,\n 'begin': begin,\n 'count': count,\n 'type': type\n })", "response": "List all comments for a specific message data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef delete_comment(self, msg_data_id, index, user_comment_id):\n return self._post(\n 'comment/delete',\n data={\n 'msg_data_id': msg_data_id,\n 'index': index,\n 'user_comment_id': user_comment_id,\n })", "response": "Delete a user comment."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd a reply comment to a user.", "response": "def add_reply_comment(self, msg_data_id, index, user_comment_id, content):\n \"\"\"\n \u56de\u590d\u8bc4\u8bba\n \"\"\"\n return self._post(\n 'comment/reply/add',\n data={\n 'msg_data_id': msg_data_id,\n 'index': index,\n 'user_comment_id': user_comment_id,\n 'content': content\n })"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete_reply_comment(self, msg_data_id, index, user_comment_id):\n return self._post(\n 'comment/reply/delete',\n data={\n 'msg_data_id': msg_data_id,\n 'index': index,\n 'user_comment_id': user_comment_id,\n })", "response": "Delete a reply comment."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_provider_token(self, provider_secret):\n return self._post(\n 'service/get_provider_token',\n data={\n 'corpid': self._client.corp_id,\n 'provider_secret': provider_secret,\n }\n )", "response": "\u83b7\u53d6\u670d\u52a1\u5546\u51ed\u8bc1\n\n https://work.weixin.qq.com/api/doc#90001/90143/91200\n\n :param provider_secret: \u670d\u52a1\u5546\u7684secret\uff0c\u5728\u670d\u52a1\u5546\u7ba1\u7406\u540e\u53f0\u53ef\u89c1\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_login_url(self, login_ticket, target, agentid=None, provider_access_token=None):\n return self._post(\n 'service/get_login_url',\n params={\n 'provider_access_token': provider_access_token,\n },\n data={\n 'login_ticket': login_ticket,\n 'target': target,\n 'agentid': agentid,\n }\n )", "response": "\u83b7\u53d6\u767b\u5f55\u4f01\u4e1a\u53f7\u5b98\u7f51\u7684url\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=\u83b7\u53d6\u767b\u5f55\u4f01\u4e1a\u53f7\u5b98\u7f51\u7684url\n\n :param provider_access_token: \u670d\u52a1\u63d0\u4f9b\u5546\u7684 accesstoken\n :param login_ticket: \u901a\u8fc7get_login_info\u5f97\u5230\u7684login_ticket, 10\u5c0f\u65f6\u6709\u6548\n :param target: \u767b\u5f55\u8df3\u8f6c\u5230\u4f01\u4e1a\u53f7\u540e\u53f0\u7684\u76ee\u6807\u9875\u9762\n :param agentid: \u53ef\u9009\uff0c\u6388\u6743\u65b9\u5e94\u7528id\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransfers amount to a user.", "response": "def transfer(self, user_id, amount, desc, client_ip=None,\n check_name='OPTION_CHECK', real_name=None,\n out_trade_no=None, device_info=None):\n \"\"\"\n \u4f01\u4e1a\u4ed8\u6b3e\u63a5\u53e3\n\n :param user_id: \u63a5\u53d7\u6536\u7ea2\u5305\u7684\u7528\u6237\u5728\u516c\u4f17\u53f7\u4e0b\u7684 openid\n :param amount: \u4ed8\u6b3e\u91d1\u989d\uff0c\u5355\u4f4d\u5206\n :param desc: \u4ed8\u6b3e\u8bf4\u660e\n :param client_ip: \u53ef\u9009\uff0c\u8c03\u7528\u63a5\u53e3\u673a\u5668\u7684 IP \u5730\u5740\n :param check_name: \u53ef\u9009\uff0c\u6821\u9a8c\u7528\u6237\u59d3\u540d\u9009\u9879\uff0c\n NO_CHECK\uff1a\u4e0d\u6821\u9a8c\u771f\u5b9e\u59d3\u540d,\n FORCE_CHECK\uff1a\u5f3a\u6821\u9a8c\u771f\u5b9e\u59d3\u540d\uff08\u672a\u5b9e\u540d\u8ba4\u8bc1\u7684\u7528\u6237\u4f1a\u6821\u9a8c\u5931\u8d25\uff0c\u65e0\u6cd5\u8f6c\u8d26\uff09,\n OPTION_CHECK\uff1a\u9488\u5bf9\u5df2\u5b9e\u540d\u8ba4\u8bc1\u7684\u7528\u6237\u624d\u6821\u9a8c\u771f\u5b9e\u59d3\u540d\uff08\u672a\u5b9e\u540d\u8ba4\u8bc1\u7528\u6237\u4e0d\u6821\u9a8c\uff0c\u53ef\u4ee5\u8f6c\u8d26\u6210\u529f\uff09,\n \u9ed8\u8ba4\u4e3a OPTION_CHECK\n :param real_name: \u53ef\u9009\uff0c\u6536\u6b3e\u7528\u6237\u771f\u5b9e\u59d3\u540d\uff0c\n \u5982\u679ccheck_name\u8bbe\u7f6e\u4e3aFORCE_CHECK\u6216OPTION_CHECK\uff0c\u5219\u5fc5\u586b\u7528\u6237\u771f\u5b9e\u59d3\u540d\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u8ba2\u5355\u53f7\uff0c\u9700\u4fdd\u6301\u552f\u4e00\u6027\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :param device_info: \u53ef\u9009\uff0c\u5fae\u4fe1\u652f\u4ed8\u5206\u914d\u7684\u7ec8\u7aef\u8bbe\u5907\u53f7\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u4fe1\u606f\n \"\"\"\n if not out_trade_no:\n now = datetime.now()\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n data = {\n 'mch_appid': self.appid,\n 'mchid': self.mch_id,\n 'device_info': device_info,\n 'partner_trade_no': out_trade_no,\n 'openid': user_id,\n 'check_name': check_name,\n 're_user_name': real_name,\n 'amount': amount,\n 'desc': desc,\n 'spbill_create_ip': client_ip or get_external_ip(),\n }\n return self._post('mmpaymkttransfers/promotion/transfers', data=data)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transfer_bankcard(self, true_name, bank_card_no, bank_code, amount, desc=None, out_trade_no=None):\n if not out_trade_no:\n now = datetime.now()\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n data = {\n 'mch_id': self.mch_id,\n 'partner_trade_no': out_trade_no,\n 'amount': amount,\n 'desc': desc,\n 'enc_bank_no': self._rsa_encrypt(bank_card_no),\n 'enc_true_name': self._rsa_encrypt(true_name),\n 'bank_code': bank_code,\n }\n return self._post('mmpaysptrans/pay_bank', data=data)", "response": "Transfer a bank card."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query_bankcard(self, out_trade_no):\n data = {\n 'mch_id': self.mch_id,\n 'partner_trade_no': out_trade_no,\n }\n return self._post('mmpaysptrans/query_bank', data=data)", "response": "Query the Mpaysptrans s Bank Card."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the ID of the current user s template.", "response": "def get(self, template_id_short):\n \"\"\"\n \u83b7\u5f97\u6a21\u677fID\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1433751277\n\n :param template_id_short: \u6a21\u677f\u5e93\u4e2d\u6a21\u677f\u7684\u7f16\u53f7\uff0c\u6709\u201cTM**\u201d\u548c\u201cOPENTMTM**\u201d\u7b49\u5f62\u5f0f\n :return: \u6a21\u677f ID\n \"\"\"\n res = self._post(\n 'template/api_add_template',\n data={\n 'template_id_short': template_id_short\n },\n result_processor=lambda x: x['template_id']\n )\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef apply_device_id(self, quantity, reason, poi_id=None, comment=None):\n data = optionaldict()\n data['quantity'] = quantity\n data['apply_reason'] = reason\n data['poi_id'] = poi_id\n data['comment'] = comment\n res = self._post(\n 'shakearound/device/applyid',\n data=data,\n result_processor=lambda x: x['data']\n )\n return res", "response": "Apply device ID to the current state of the user."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the device identifier.", "response": "def update_device(self, device_id=None, uuid=None, major=None,\n minor=None, comment=None):\n \"\"\"\n \u66f4\u65b0\u8bbe\u5907\u4fe1\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/15/b9e012f917e3484b7ed02771156411f3.html\n\n :param device_id: \u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u586b\u4e86UUID\u3001major\u3001minor\uff0c\u5219\u53ef\u4e0d\u586b\u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u4e8c\u8005\u90fd\u586b\uff0c\u5219\u4ee5\u8bbe\u5907\u7f16\u53f7\u4e3a\u4f18\u5148\n :param uuid: UUID\n :param major: major\n :param minor: minor\n :param comment: \u8bbe\u5907\u7684\u5907\u6ce8\u4fe1\u606f\uff0c\u4e0d\u8d85\u8fc715\u4e2a\u6c49\u5b57\u621630\u4e2a\u82f1\u6587\u5b57\u6bcd\u3002\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = optionaldict()\n data['comment'] = comment\n data['device_identifier'] = {\n 'device_id': device_id,\n 'uuid': uuid,\n 'major': major,\n 'minor': minor\n }\n return self._post(\n 'shakearound/device/update',\n data=data\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef search_device(self, identifiers=None, apply_id=None,\n begin=0, count=10):\n \"\"\"\n \u67e5\u8be2\u8bbe\u5907\u5217\u8868\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/15/b9e012f917e3484b7ed02771156411f3.html\n\n :param identifiers: \u8bbe\u5907 ID \u4fe1\u606f\u5217\u8868\n :param apply_id: \u6279\u6b21ID\uff0c\u7533\u8bf7\u8bbe\u5907ID\u8d85\u51fa500\u4e2a\u65f6\u6240\u8fd4\u56de\u6279\u6b21ID\n :param begin: \u8bbe\u5907\u5217\u8868\u7684\u8d77\u59cb\u7d22\u5f15\u503c\n :param count: \u5f85\u67e5\u8be2\u7684\u8bbe\u5907\u4e2a\u6570\n :return: \u8bbe\u5907\u5217\u8868\n \"\"\"\n data = optionaldict()\n data['begin'] = begin\n data['count'] = count\n data['apply_id'] = apply_id\n if identifiers:\n data['device_identifiers'] = identifiers\n res = self._post(\n 'shakearound/device/search',\n data=data,\n result_processor=lambda x: x['data']\n )\n return res", "response": "Search for a device in the cache."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsearch for pages in a specific language.", "response": "def search_pages(self, page_ids=None, begin=0, count=10):\n \"\"\"\n \u67e5\u8be2\u9875\u9762\u5217\u8868\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/5/6626199ea8757c752046d8e46cf13251.html\n\n :param page_ids: \u6307\u5b9a\u9875\u9762\u7684id\u5217\u8868\n :param begin: \u9875\u9762\u5217\u8868\u7684\u8d77\u59cb\u7d22\u5f15\u503c\n :param count: \u5f85\u67e5\u8be2\u7684\u9875\u9762\u4e2a\u6570\n :return: \u9875\u9762\u67e5\u8be2\u7ed3\u679c\u4fe1\u606f\n \"\"\"\n if not page_ids:\n data = {\n 'type': 2,\n 'begin': begin,\n 'count': count\n }\n else:\n if not isinstance(page_ids, (tuple, list)):\n page_ids = [page_ids]\n data = {\n 'type': 1,\n 'page_ids': page_ids\n }\n\n res = self._post(\n 'shakearound/page/search',\n data=data,\n result_processor=lambda x: x['data']\n )\n return res"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_material(self, media_file, media_type='icon'):\n res = self._post(\n 'shakearound/material/add',\n files={\n 'media': media_file\n },\n params={\n 'type': media_type\n },\n result_processor=lambda x: x['data']\n )\n return res", "response": "add a new material to the set of known media"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef bind_device_pages(self, page_ids, bind, append, device_id=None,\n uuid=None, major=None, minor=None):\n \"\"\"\n \u914d\u7f6e\u8bbe\u5907\u4e0e\u9875\u9762\u7684\u5173\u8054\u5173\u7cfb\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/12/c8120214ec0ba08af5dfcc0da1a11400.html\n\n :param page_ids: \u5f85\u5173\u8054\u7684\u9875\u9762\u5217\u8868\n :param bind: \u5173\u8054\u64cd\u4f5c\u6807\u5fd7\u4f4d\uff0c 0\u4e3a\u89e3\u9664\u5173\u8054\u5173\u7cfb\uff0c1\u4e3a\u5efa\u7acb\u5173\u8054\u5173\u7cfb\n :param append: \u65b0\u589e\u64cd\u4f5c\u6807\u5fd7\u4f4d\uff0c 0\u4e3a\u8986\u76d6\uff0c1\u4e3a\u65b0\u589e\n :param device_id: \u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u586b\u4e86UUID\u3001major\u3001minor\uff0c\u5219\u53ef\u4e0d\u586b\u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u4e8c\u8005\u90fd\u586b\uff0c\u5219\u4ee5\u8bbe\u5907\u7f16\u53f7\u4e3a\u4f18\u5148\n :param uuid: UUID\n :param major: major\n :param minor: minor\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n if not isinstance(page_ids, (tuple, list)):\n page_ids = [page_ids]\n data = {\n 'page_ids': page_ids,\n 'bind': int(bind),\n 'append': int(append),\n 'device_identifier': {\n 'device_id': device_id,\n 'uuid': uuid,\n 'major': major,\n 'minor': minor\n }\n }\n return self._post(\n 'shakearound/device/bindpage',\n data=data\n )", "response": "bind_device_pages - Binds a list of page_ids to a device."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_device_statistics(self, begin_date, end_date, device_id=None,\n uuid=None, major=None, minor=None):\n \"\"\"\n \u4ee5\u8bbe\u5907\u4e3a\u7ef4\u5ea6\u7684\u6570\u636e\u7edf\u8ba1\u63a5\u53e3\n http://mp.weixin.qq.com/wiki/0/8a24bcacad40fe7ee98d1573cb8a6764.html\n\n :param begin_date: \u8d77\u59cb\u65f6\u95f4\uff0c\u6700\u957f\u65f6\u95f4\u8de8\u5ea6\u4e3a30\u5929\n :param end_date: \u7ed3\u675f\u65f6\u95f4\uff0c\u6700\u957f\u65f6\u95f4\u8de8\u5ea6\u4e3a30\u5929\n :param device_id: \u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u586b\u4e86UUID\u3001major\u3001minor\uff0c\u5219\u53ef\u4e0d\u586b\u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u4e8c\u8005\u90fd\u586b\uff0c\u5219\u4ee5\u8bbe\u5907\u7f16\u53f7\u4e3a\u4f18\u5148\n :param uuid: UUID\n :param major: major\n :param minor: minor\n \"\"\"\n data = {\n 'device_identifier': {\n 'device_id': device_id,\n 'uuid': uuid,\n 'major': major,\n 'minor': minor\n },\n 'begin_date': self._to_timestamp(begin_date),\n 'end_date': self._to_timestamp(end_date)\n }\n res = self._post(\n 'shakearound/statistics/device',\n data=data,\n result_processor=lambda x: x['data']\n )\n return res", "response": "\u4ee5\u8bbe\u5907\u4e3a\u7ef4\u5ea6\u7684\u6570\u636e\u7edf\u8ba1\u63a5\u53e3\n http://mp.weixin.qq.com/wiki/0/8a24bcacad40fe7ee98d1573cb8a6764.html\n\n :param begin_date: \u8d77\u59cb\u65f6\u95f4\uff0c\u6700\u957f\u65f6\u95f4\u8de8\u5ea6\u4e3a30\u5929\n :param end_date: \u7ed3\u675f\u65f6\u95f4\uff0c\u6700\u957f\u65f6\u95f4\u8de8\u5ea6\u4e3a30\u5929\n :param device_id: \u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u586b\u4e86UUID\u3001major\u3001minor\uff0c\u5219\u53ef\u4e0d\u586b\u8bbe\u5907\u7f16\u53f7\uff0c\u82e5\u4e8c\u8005\u90fd\u586b\uff0c\u5219\u4ee5\u8bbe\u5907\u7f16\u53f7\u4e3a\u4f18\u5148\n :param uuid: UUID\n :param major: major\n :param minor: minor"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_page_statistics(self, page_id, begin_date, end_date):\n res = self._post(\n 'shakearound/statistics/page',\n data={\n 'page_id': page_id,\n 'begin_date': self._to_timestamp(begin_date),\n 'end_date': self._to_timestamp(end_date),\n },\n result_processor=lambda x: x['data']\n )\n return res", "response": "Get statistics for a page."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_apply_status(self, apply_id):\n res = self._post(\n 'shakearound/device/applystatus',\n data={\n 'apply_id': apply_id,\n },\n result_processor=lambda x: x['data']\n )\n return res", "response": "\u67e5\u8be2\u8bbe\u5907ID\u7533\u8bf7\u5ba1\u6838\u72b6\u6001\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/15/b9e012f917e3484b7ed02771156411f3.html\n\n :param apply_id: \u6279\u6b21ID\uff0c\u7533\u8bf7\u8bbe\u5907ID\u65f6\u6240\u8fd4\u56de\u7684\u6279\u6b21ID\n :return: \u6279\u6b21\u72b6\u6001\u4fe1\u606f"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\napplies signing to the base payment.", "response": "def apply_signing(self, plan_id, contract_code, contract_display_account, notify_url,\n version=\"1.0\", clientip=None, deviceid=None, mobile=None, email=None, qq=None,\n request_serial=None, openid=None, creid=None, outerid=None):\n \"\"\"\n \u7533\u8bf7\u7b7e\u7ea6 api\n\n https://pay.weixin.qq.com/wiki/doc/api/pap.php?chapter=18_1&index=1\n\n :param plan_id: \u6a21\u677fid \u534f\u8bae\u6a21\u677fid\uff0c\u8bbe\u7f6e\u8def\u5f84\u89c1\u5f00\u53d1\u6b65\u9aa4\u3002\n :param contract_code: \u7b7e\u7ea6\u534f\u8bae\u53f7 \u5546\u6237\u4fa7\u7684\u7b7e\u7ea6\u534f\u8bae\u53f7\uff0c\u7531\u5546\u6237\u751f\u6210\n :param contract_display_account: \u7528\u6237\u8d26\u6237\u5c55\u793a\u540d\u79f0 \u7b7e\u7ea6\u7528\u6237\u7684\u540d\u79f0\uff0c\u7528\u4e8e\u9875\u9762\u5c55\u793a\uff0c\u9875\u9762\u6837\u4f8b\u53ef\u89c1\u6848\u4f8b\u4e0e\u89c4\u8303\n :param notify_url: \u56de\u8c03\u901a\u77e5url \u7528\u4e8e\u63a5\u6536\u7b7e\u7ea6\u6210\u529f\u6d88\u606f\u7684\u56de\u8c03\u901a\u77e5\u5730\u5740\uff0c\u4ee5http\u6216https\u5f00\u5934\u3002\n :param version: \u7248\u672c\u53f7 \u56fa\u5b9a\u503c1.0\n :param request_serial: \u53ef\u9009 \u8bf7\u6c42\u5e8f\u5217\u53f7 \u5546\u6237\u8bf7\u6c42\u7b7e\u7ea6\u65f6\u7684\u5e8f\u5217\u53f7\uff0c\u5546\u6237\u4fa7\u987b\u552f\u4e00\u3002\u5e8f\u5217\u53f7\u4e3b\u8981\u7528\u4e8e\u6392\u5e8f\uff0c\u4e0d\u4f5c\u4e3a\u67e5\u8be2\u6761\u4ef6\n :param clientip: \u53ef\u9009 \u5ba2\u6237\u7aef IP \u70b9\u5206IP\u683c\u5f0f(\u5ba2\u6237\u7aefIP)\n :param deviceid: \u53ef\u9009 \u8bbe\u5907ID android\u586bimei\u7684\u4e00\u6b21md5; ios\u586bidfa\u7684\u4e00\u6b21md5\n :param mobile: \u53ef\u9009 \u624b\u673a\u53f7 \u7528\u6237\u624b\u673a\u53f7\n :param email: \u53ef\u9009 \u90ae\u7bb1\u5730\u5740 \u7528\u6237\u90ae\u7bb1\u5730\u5740\n :param qq: \u53ef\u9009 QQ\u53f7 \u7528\u6237QQ\u53f7\n :param openid: \u53ef\u9009 \u5fae\u4fe1open ID \u7528\u6237\u5fae\u4fe1open ID\n :param creid: \u53ef\u9009 \u8eab\u4efd\u8bc1\u53f7 \u7528\u6237\u8eab\u4efd\u8bc1\u53f7\n :param outerid: \u53ef\u9009 \u5546\u6237\u4fa7\u7528\u6237\u6807\u8bc6 \u7528\u6237\u5728\u5546\u6237\u4fa7\u7684\u6807\u8bc6\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\u5b57\u5178\n \"\"\"\n timestamp = int(time.time())\n if request_serial is None:\n request_serial = int(time.time() * 1000)\n data = {\n \"appid\": self.appid,\n \"mch_id\": self.mch_id,\n \"sub_mch_id\": self.sub_mch_id,\n \"plan_id\": plan_id,\n \"contract_code\": contract_code,\n \"request_serial\": request_serial,\n \"contract_display_account\": contract_display_account,\n \"notify_url\": notify_url,\n \"version\": version,\n \"timestamp\": timestamp,\n \"clientip\": clientip,\n \"deviceid\": deviceid,\n \"mobile\": mobile,\n \"email\": email,\n \"qq\": qq,\n \"openid\": openid,\n \"creid\": creid,\n \"outerid\": outerid,\n }\n data = optionaldict(data)\n sign = calculate_signature(data, self._client.api_key)\n data[\"sign\"] = sign\n return {\n \"base_url\": \"{}papay/entrustweb\".format(self._client.API_BASE_URL),\n \"data\": data\n }"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nquery the API for signing.", "response": "def query_signing(self, contract_id=None, plan_id=None, contract_code=None, openid=None, version=\"1.0\"):\n \"\"\"\n \u67e5\u8be2\u7b7e\u7ea6\u5173\u7cfb api\n\n :param contract_id: \u53ef\u9009 \u59d4\u6258\u4ee3\u6263\u534f\u8baeid \u59d4\u6258\u4ee3\u6263\u7b7e\u7ea6\u6210\u529f\u540e\u7531\u5fae\u4fe1\u8fd4\u56de\u7684\u59d4\u6258\u4ee3\u6263\u534f\u8baeid\uff0c\u9009\u62e9contract_id\u67e5\u8be2\uff0c\u5219\u6b64\u53c2\u6570\u5fc5\u586b\n :param plan_id: \u53ef\u9009 \u6a21\u677fid \u5546\u6237\u5728\u5fae\u4fe1\u5546\u6237\u5e73\u53f0\u914d\u7f6e\u7684\u4ee3\u6263\u6a21\u677fid\uff0c\u9009\u62e9plan_id+contract_code\u67e5\u8be2\uff0c\u5219\u6b64\u53c2\u6570\u5fc5\u586b\n :param contract_code: \u53ef\u9009 \u7b7e\u7ea6\u534f\u8bae\u53f7 \u5546\u6237\u8bf7\u6c42\u7b7e\u7ea6\u65f6\u4f20\u5165\u7684\u7b7e\u7ea6\u534f\u8bae\u53f7\uff0c\u5546\u6237\u4fa7\u987b\u552f\u4e00\u3002\u9009\u62e9plan_id+contract_code\u67e5\u8be2\uff0c\u5219\u6b64\u53c2\u6570\u5fc5\u586b\n :param openid: \u53ef\u9009 openid \u7528\u6237\u6807\u8bc6\uff0c\u5fc5\u987b\u4fdd\u8bc1\u4e0e\u4f20\u5165appid\u5bf9\u5e94\n :param version: \u7248\u672c\u53f7 \u56fa\u5b9a\u503c1.0\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u4fe1\u606f\n \"\"\"\n if not contract_id and not (plan_id and contract_code) and not (plan_id and openid):\n raise ValueError(\"contract_id and (plan_id, contract_code) and (plan_id, openid) must be a choice.\")\n data = {\n \"appid\": self.appid,\n \"mch_id\": self.mch_id,\n \"contract_id\": contract_id,\n \"plan_id\": plan_id,\n \"contract_code\": contract_code,\n \"openid\": openid,\n \"version\": version,\n \"nonce_str\": None,\n }\n return self._post('papay/querycontract', data=data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef apply_deduct(self, body, total_fee, contract_id, notify_url, out_trade_no=None,\n detail=None, attach=None, fee_type='CNY', goods_tag=None, clientip=None, deviceid=None,\n mobile=None, email=None, qq=None, openid=None, creid=None, outerid=None):\n \"\"\"\n \u7533\u8bf7\u6263\u6b3e api\n\n :param body: \u5546\u54c1\u63cf\u8ff0 \u5546\u54c1\u6216\u652f\u4ed8\u5355\u7b80\u8981\u63cf\u8ff0\n :param out_trade_no: \u53ef\u9009 \u5546\u6237\u8ba2\u5355\u53f7 \u5546\u6237\u7cfb\u7edf\u5185\u90e8\u7684\u8ba2\u5355\u53f7,32\u4e2a\u5b57\u7b26\u5185\u3001\u53ef\u5305\u542b\u5b57\u6bcd, \u5176\u4ed6\u8bf4\u660e\u89c1\u5546\u6237\u8ba2\u5355\u53f7\n :param total_fee: \u603b\u91d1\u989d \u8ba2\u5355\u603b\u91d1\u989d\uff0c\u5355\u4f4d\u4e3a\u5206\uff0c\u53ea\u80fd\u4e3a\u6574\u6570\uff0c\u8be6\u89c1\u652f\u4ed8\u91d1\u989d\n :param contract_id: \u59d4\u6258\u4ee3\u6263\u534f\u8baeid \u7b7e\u7ea6\u6210\u529f\u540e\uff0c\u5fae\u4fe1\u8fd4\u56de\u7684\u59d4\u6258\u4ee3\u6263\u534f\u8baeid\n :param notify_url: \u56de\u8c03\u901a\u77e5url \u63a5\u53d7\u6263\u6b3e\u7ed3\u679c\u5f02\u6b65\u56de\u8c03\u901a\u77e5\u7684url\n :param detail: \u53ef\u9009 \u5546\u54c1\u8be6\u60c5 \u5546\u54c1\u540d\u79f0\u660e\u7ec6\u5217\u8868\n :param attach: \u53ef\u9009 \u9644\u52a0\u6570\u636e \u9644\u52a0\u6570\u636e\uff0c\u5728\u67e5\u8be2API\u548c\u652f\u4ed8\u901a\u77e5\u4e2d\u539f\u6837\u8fd4\u56de\uff0c\u8be5\u5b57\u6bb5\u4e3b\u8981\u7528\u4e8e\u5546\u6237\u643a\u5e26\u8ba2\u5355\u7684\u81ea\u5b9a\u4e49\u6570\u636e\n :param fee_type: \u53ef\u9009 \u8d27\u5e01\u7c7b\u578b \u7b26\u5408ISO 4217\u6807\u51c6\u7684\u4e09\u4f4d\u5b57\u6bcd\u4ee3\u7801\uff0c\u9ed8\u8ba4\u4eba\u6c11\u5e01\uff1aCNY\n :param goods_tag: \u53ef\u9009 \u5546\u54c1\u6807\u8bb0 \u5546\u54c1\u6807\u8bb0\uff0c\u4ee3\u91d1\u5238\u6216\u7acb\u51cf\u4f18\u60e0\u529f\u80fd\u7684\u53c2\u6570\uff0c\u8bf4\u660e\u8be6\u89c1\u4ee3\u91d1\u5238\u6216\u7acb\u51cf\u4f18\u60e0\n :param clientip: \u53ef\u9009 \u5ba2\u6237\u7aef IP \u70b9\u5206IP\u683c\u5f0f(\u5ba2\u6237\u7aefIP)\n :param deviceid: \u53ef\u9009 \u8bbe\u5907ID android\u586bimei\u7684\u4e00\u6b21md5; ios\u586bidfa\u7684\u4e00\u6b21md5\n :param mobile: \u53ef\u9009 \u624b\u673a\u53f7 \u7528\u6237\u624b\u673a\u53f7\n :param email: \u53ef\u9009 \u90ae\u7bb1\u5730\u5740 \u7528\u6237\u90ae\u7bb1\u5730\u5740\n :param qq: \u53ef\u9009 QQ\u53f7 \u7528\u6237QQ\u53f7\n :param openid: \u53ef\u9009 \u5fae\u4fe1open ID \u7528\u6237\u5fae\u4fe1open ID\n :param creid: \u53ef\u9009 \u8eab\u4efd\u8bc1\u53f7 \u7528\u6237\u8eab\u4efd\u8bc1\u53f7\n :param outerid: \u53ef\u9009 \u5546\u6237\u4fa7\u7528\u6237\u6807\u8bc6 \u7528\u6237\u5728\u5546\u6237\u4fa7\u7684\u6807\u8bc6\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u4fe1\u606f\n \"\"\"\n trade_type = 'PAP' # \u4ea4\u6613\u7c7b\u578b \u4ea4\u6613\u7c7b\u578bPAP-\u5fae\u4fe1\u59d4\u6258\u4ee3\u6263\u652f\u4ed8\n timestamp = int(time.time()) # 10\u4f4d\u65f6\u95f4\u6233\n spbill_create_ip = get_external_ip() # \u7ec8\u7aefIP \u8c03\u7528\u5fae\u4fe1\u652f\u4ed8API\u7684\u673a\u5668IP\n if not out_trade_no:\n now = datetime.fromtimestamp(time.time(), tz=timezone('Asia/Shanghai'))\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n\n data = {\n \"appid\": self.appid,\n \"mch_id\": self.mch_id,\n \"body\": body,\n \"out_trade_no\": out_trade_no,\n \"total_fee\": total_fee,\n \"trade_type\": trade_type,\n \"contract_id\": contract_id,\n \"notify_url\": notify_url,\n \"detail\": detail,\n \"attach\": attach,\n \"fee_type\": fee_type,\n \"goods_tag\": goods_tag,\n \"clientip\": clientip,\n \"deviceid\": deviceid,\n \"mobile\": mobile,\n \"email\": email,\n \"qq\": qq,\n \"openid\": openid,\n \"creid\": creid,\n \"outerid\": outerid,\n \"timestamp\": timestamp,\n \"spbill_create_ip\": spbill_create_ip,\n }\n return self._post(\"pay/pappayapply\", data=data)", "response": "Applydeduct to a base API"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef query_order(self, transaction_id=None, out_trade_no=None):\n if not transaction_id and not out_trade_no:\n raise ValueError(\"transaction_id and out_trade_no must be a choice.\")\n data = {\n \"appid\": self.appid,\n \"mch_id\": self.mch_id,\n \"transaction_id\": transaction_id,\n \"out_trade_no\": out_trade_no,\n }\n return self._post(\"pay/paporderquery\", data=data)", "response": "Query the order for a given transaction_id and out_trade_no."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef apply_cancel_signing(self, contract_id=None, plan_id=None, contract_code=None,\n contract_termination_remark=None, version=\"1.0\"):\n \"\"\"\n \u7533\u8bf7\u89e3\u7ea6\n\n https://pay.weixin.qq.com/wiki/doc/api/pap.php?chapter=18_4&index=6\n\n :param contract_id: \u5408\u540cID\n :param plan_id: \u6a21\u677fID\n :param contract_code: \u5408\u540c\u53f7\n :param contract_termination_remark: \u89e3\u7ea6\u539f\u56e0\n :param version: \u7248\u672c\u53f7\n :return:\n \"\"\"\n if not (contract_id or (plan_id and contract_code)):\n raise ValueError(\"contract_id and (plan_id, contract_code) must be a choice.\")\n data = {\n \"appid\": self.appid,\n \"mch_id\": self.mch_id,\n \"plan_id\": plan_id,\n \"contract_code\": contract_code,\n \"contract_id\": contract_id,\n \"contract_termination_remark\": contract_termination_remark,\n \"version\": version,\n \"nonce_str\": None,\n }\n return self._post(\"papay/deletecontract\", data=data)", "response": "Apply cancel signing to the current order."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef modify_product_status(self, standard, key, status):\n data = {\n 'keystandard': standard,\n 'keystr': key,\n 'status': status,\n }\n return self._post('product/modstatus', data=data)", "response": "Modify the status of a product."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list_product(self, offset=0, limit=10, status=None, key=None):\n data = optionaldict(\n offset=offset,\n limit=limit,\n status=status,\n keystr=key,\n )\n return self._post('product/getlist', data=data)", "response": "list_product - List all products in a specific order"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nclearing the product from the cache.", "response": "def clear_product(self, standard, key):\n \"\"\"\n \u6e05\u9664\u5546\u54c1\u4fe1\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/15/7fa787701295b884410b5163e13313af.html\n\n :param standard: \u5546\u54c1\u7f16\u7801\u6807\u51c6\n :param key: \u5546\u54c1\u7f16\u7801\u5185\u5bb9\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = {\n 'keystandard': standard,\n 'keystr': key,\n }\n return self._post('product/clear', data=data)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a message to the device.", "response": "def send_message(self, device_type, device_id, user_id, content):\n \"\"\"\n \u4e3b\u52a8\u53d1\u9001\u6d88\u606f\u7ed9\u8bbe\u5907\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://iot.weixin.qq.com/wiki/new/index.html?page=3-4-3\n\n :param device_type: \u8bbe\u5907\u7c7b\u578b\uff0c\u76ee\u524d\u4e3a\u201c\u516c\u4f17\u8d26\u53f7\u539f\u59cbID\u201d\n :param device_id: \u8bbe\u5907ID\n :param user_id: \u5fae\u4fe1\u7528\u6237\u8d26\u53f7\u7684openid\n :param content: \u6d88\u606f\u5185\u5bb9\uff0cBASE64\u7f16\u7801\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n content = to_text(base64.b64encode(to_binary(content)))\n return self._post(\n 'transmsg',\n data={\n 'device_type': device_type,\n 'device_id': device_id,\n 'open_id': user_id,\n 'content': content\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send_status_message(self, device_type, device_id, user_id, msg_type, device_status):\n return self._post(\n 'transmsg',\n data={\n 'device_type': device_type,\n 'device_id': device_id,\n 'open_id': user_id,\n 'msg_type': msg_type,\n 'device_status': device_status,\n }\n )", "response": "Send status message to the device."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets QRS Code URL", "response": "def get_qrcode_url(self, ticket, data=None):\n \"\"\"\n \u901a\u8fc7 ticket \u6362\u53d6\u4e8c\u7ef4\u7801\u5730\u5740\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://iot.weixin.qq.com/wiki/new/index.html?page=3-4-4\n\n :param ticket: \u4e8c\u7ef4\u7801 ticket\n :param data: \u989d\u5916\u6570\u636e\n :return: \u4e8c\u7ef4\u7801\u5730\u5740\n \"\"\"\n url = 'https://we.qq.com/d/{ticket}'.format(ticket=ticket)\n if data:\n if isinstance(data, (dict, tuple, list)):\n data = urllib.urlencode(data)\n data = to_text(base64.b64encode(to_binary(data)))\n url = '{base}#{data}'.format(base=url, data=data)\n return url"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nunbinding a user from a device.", "response": "def unbind(self, ticket, device_id, user_id):\n \"\"\"\n \u89e3\u7ed1\u8bbe\u5907\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://iot.weixin.qq.com/wiki/new/index.html?page=3-4-7\n\n :param ticket: \u7ed1\u5b9a\u64cd\u4f5c\u5408\u6cd5\u6027\u7684\u51ed\u8bc1\uff08\u7531\u5fae\u4fe1\u540e\u53f0\u751f\u6210\uff0c\u7b2c\u4e09\u65b9H5\u901a\u8fc7\u5ba2\u6237\u7aefjsapi\u83b7\u5f97\uff09\n :param device_id: \u8bbe\u5907id\n :param user_id: \u7528\u6237\u5bf9\u5e94\u7684openid\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._post(\n 'unbind',\n data={\n 'ticket': ticket,\n 'device_id': device_id,\n 'openid': user_id\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef authorize(self, devices, op_type=1):\n return self._post(\n 'authorize_device',\n data={\n 'device_num': len(devices),\n 'device_list': devices,\n 'op_type': op_type\n }\n )", "response": "Authorize a list of devices."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a user in the weixin.", "response": "def create(self, user_id, name, department=None, position=None,\n mobile=None, gender=0, tel=None, email=None,\n weixin_id=None, extattr=None):\n \"\"\"\n \u521b\u5efa\u6210\u5458\n\n https://work.weixin.qq.com/api/doc#90000/90135/90195\n \"\"\"\n user_data = optionaldict()\n user_data['userid'] = user_id\n user_data['name'] = name\n user_data['gender'] = gender\n user_data['department'] = department\n user_data['position'] = position\n user_data['mobile'] = mobile\n user_data['tel'] = tel\n user_data['email'] = email\n user_data['weixinid'] = weixin_id\n user_data['extattr'] = extattr\n\n return self._post(\n 'user/create',\n data=user_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a list of users from a user s company.", "response": "def list(self, department_id, fetch_child=False, status=0, simple=False):\n \"\"\"\n \u6279\u91cf\u83b7\u53d6\u90e8\u95e8\u6210\u5458 / \u6279\u91cf\u83b7\u53d6\u90e8\u95e8\u6210\u5458\u8be6\u60c5\n\n https://work.weixin.qq.com/api/doc#90000/90135/90200\n https://work.weixin.qq.com/api/doc#90000/90135/90201\n\n \u6b64\u63a5\u53e3\u548c `WeChatDepartment.get_users` \u662f\u540c\u4e00\u4e2a\u63a5\u53e3\uff0c\u533a\u522b\u4e3a simple \u7684\u9ed8\u8ba4\u503c\u4e0d\u540c\u3002\n \"\"\"\n url = 'user/simplelist' if simple else 'user/list'\n res = self._get(\n url,\n params={\n 'department_id': department_id,\n 'fetch_child': 1 if fetch_child else 0,\n 'status': status\n }\n )\n return res['userlist']"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convert_to_openid(self, user_id, agent_id=None):\n data = optionaldict(\n userid=user_id,\n agentid=agent_id\n )\n return self._post('user/convert_to_openid', data=data)", "response": "Convert user_id to OpenID."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create(self, chat_id=None, name=None, owner=None, user_list=None):\n data = optionaldict(\n chatid=chat_id,\n name=name,\n owner=owner,\n userlist=user_list,\n )\n return self._post('appchat/create', data=data)", "response": "create a new object"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef send(self, chat_id, msg_type, **kwargs):\n data = {\n 'chatid': chat_id,\n 'safe': kwargs.get('safe') or 0\n }\n data.update(self._build_msg_content(msg_type, **kwargs))\n\n return self._post('appchat/send', data=data)", "response": "Send a message to a chat."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndeprecates use send instead.", "response": "def send_msg(self, chat_id, msg_type, **kwargs):\n \"\"\" deprecated, use `send` instead \"\"\"\n return self.send(chat_id, msg_type, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send_text(self, chat_id, content, safe=0):\n return self.send(chat_id, 'text', safe=safe, content=content)", "response": "Send text message to chat"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds a message content", "response": "def _build_msg_content(self, msgtype='text', **kwargs):\n \"\"\"\n \u6784\u9020\u6d88\u606f\u5185\u5bb9\n\n :param content: \u6d88\u606f\u5185\u5bb9\uff0c\u6700\u957f\u4e0d\u8d85\u8fc72048\u4e2a\u5b57\u8282\n :param msgtype: \u6d88\u606f\u7c7b\u578b\uff0c\u53ef\u4ee5\u4e3atext/image/voice/video/file/textcard/news/mpnews/markdown\n :param kwargs: \u5177\u4f53\u6d88\u606f\u7c7b\u578b\u7684\u6269\u5c55\u53c2\u6570\n :return:\n \"\"\"\n data = {'msgtype': msgtype}\n if msgtype == 'text':\n data[msgtype] = {'content': kwargs.get('content')}\n elif msgtype == 'image' or msgtype == 'voice' or msgtype == 'file':\n data[msgtype] = {'media_id': kwargs.get('media_id')}\n elif msgtype == 'video':\n data[msgtype] = {\n 'media_id': kwargs.get('media_id'),\n 'title': kwargs.get('title'),\n 'description': kwargs.get('description')\n }\n elif msgtype == 'textcard':\n data[msgtype] = {\n 'title': kwargs.get('title'),\n 'description': kwargs.get('description'),\n 'url': kwargs.get('url'),\n 'btntxt': kwargs.get('btntxt'),\n }\n elif msgtype == 'news':\n # {\n # \"articles\" :\n # [\n # {\n # \"title\" : \"\u4e2d\u79cb\u8282\u793c\u54c1\u9886\u53d6\",\n # \"description\" : \"\u4eca\u5e74\u4e2d\u79cb\u8282\u516c\u53f8\u6709\u8c6a\u793c\u76f8\u9001\",\n # \"url\":\"https://zhidao.baidu.com/question/2073647112026042748.html\",\n # \"picurl\":\"http://res.mail.qq.com/node/ww/wwopenmng/images/independent/doc/test_pic_msg1.png\"\n # }\n # ]\n # }\n data[msgtype] = kwargs\n elif msgtype == 'mpnews':\n # {\n # \"articles\":[\n # {\n # \"title\": \"\u5730\u7403\u4e00\u5c0f\u65f6\",\n # \"thumb_media_id\": \"biz_get(image)\",\n # \"author\": \"Author\",\n # \"content_source_url\": \"https://work.weixin.qq.com\",\n # \"content\": \"3\u670824\u65e520:30-21:30 \\n\u529e\u516c\u533a\u5c06\u5173\u95ed\u7167\u660e\u4e00\u5c0f\u65f6\uff0c\u8bf7\u5404\u90e8\u95e8\u540c\u4e8b\u76f8\u4e92\u8f6c\u544a\",\n # \"digest\": \"3\u670824\u65e520:30-21:30 \\n\u529e\u516c\u533a\u5c06\u5173\u95ed\u7167\u660e\u4e00\u5c0f\u65f6\"\n # }\n # ]\n # }\n data[msgtype] = kwargs\n elif msgtype == 'markdown':\n # {\n # \"content\": \"\u60a8\u7684\u4f1a\u8bae\u5ba4\u5df2\u7ecf\u9884\u5b9a\uff0c\u7a0d\u540e\u4f1a\u540c\u6b65\u5230`\u90ae\u7bb1`\n # >**\u4e8b\u9879\u8be6\u60c5**\n # >\u4e8b\u3000\u9879\uff1a\u5f00\u4f1a\n # >\u7ec4\u7ec7\u8005\uff1a@miglioguan\n # >\u53c2\u4e0e\u8005\uff1a@miglioguan\u3001@kunliu\u3001@jamdeezhou\u3001@kanexiong\u3001@kisonwang\n # >\n # >\u4f1a\u8bae\u5ba4\uff1a\u5e7f\u5ddeTIT 1\u697c 301\n # >\u65e5\u3000\u671f\uff1a2018\u5e745\u670818\u65e5\n # >\u65f6\u3000\u95f4\uff1a\u4e0a\u53489:00-11:00\n # >\n # >\u8bf7\u51c6\u65f6\u53c2\u52a0\u4f1a\u8bae\u3002\n # >\n # >\u5982\u9700\u4fee\u6539\u4f1a\u8bae\u4fe1\u606f\uff0c\u8bf7\u70b9\u51fb\uff1a[\u4fee\u6539\u4f1a\u8bae\u4fe1\u606f](https://work.weixin.qq.com)\"\n # }\n data[msgtype] = kwargs\n else:\n raise TypeError('\u4e0d\u80fd\u8bc6\u522b\u7684msgtype: %s' % msgtype)\n return data"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the details of a specific QR Code", "response": "def show(self, ticket):\n \"\"\"\n \u901a\u8fc7ticket\u6362\u53d6\u4e8c\u7ef4\u7801\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1443433542\n\n :param ticket: \u4e8c\u7ef4\u7801 ticket \u3002\u53ef\u4ee5\u901a\u8fc7 :func:`create` \u83b7\u53d6\u5230\n :return: \u8fd4\u56de\u7684 Request \u5bf9\u8c61\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.qrcode.show('ticket data')\n\n \"\"\"\n if isinstance(ticket, dict):\n ticket = ticket['ticket']\n return requests.get(\n url='https://mp.weixin.qq.com/cgi-bin/showqrcode',\n params={\n 'ticket': ticket\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_url(cls, ticket):\n url = 'https://mp.weixin.qq.com/cgi-bin/showqrcode?ticket={ticket}'\n if isinstance(ticket, dict):\n ticket = ticket['ticket']\n ticket = six.moves.urllib.parse.quote(ticket)\n return url.format(ticket=ticket)", "response": "get url of the current node"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef access_token(self):\n access_token = self.session.get(self.access_token_key)\n if access_token:\n if not self.expires_at:\n # user provided access_token, just return it\n return access_token\n\n timestamp = time.time()\n if self.expires_at - timestamp > 60:\n return access_token\n\n self.fetch_access_token()\n return self.session.get(self.access_token_key)", "response": "Get the access token from the session."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create(self, card_data):\n result = self._post(\n 'card/create',\n data=card_data,\n result_processor=lambda x: x['card_id']\n )\n return result", "response": "\u521b\u5efa\u5361\u5238\n\n :param card_data: \u5361\u5238\u4fe1\u606f\n :return: \u521b\u5efa\u7684\u5361\u5238 ID"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbatching add locations to the cache.", "response": "def batch_add_locations(self, location_data):\n \"\"\"\n \u6279\u91cf\u5bfc\u5165\u95e8\u5e97\u4fe1\u606f\n\n :param location_data: \u95e8\u5e97\u4fe1\u606f\n :return: \u95e8\u5e97 ID \u5217\u8868\uff0c\u63d2\u5165\u5931\u8d25\u7684\u95e8\u5e97\u5143\u7d20\u503c\u4e3a -1\n \"\"\"\n result = self._post(\n 'card/location/batchadd',\n data=location_data,\n result_processor=lambda x: x['location_id_list']\n )\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_qrcode(self, qrcode_data):\n result = self._post(\n 'card/qrcode/create',\n data=qrcode_data,\n result_processor=lambda x: x['ticket']\n )\n return result", "response": "\u521b\u5efa\u5361\u5238\u4e8c\u7ef4\u7801\n\n :param qrcode_data: \u4e8c\u7ef4\u7801\u4fe1\u606f\n :return: \u4e8c\u7ef4\u7801 ticket\uff0c\u53ef\u4f7f\u7528 :func:show_qrcode \u6362\u53d6\u4e8c\u7ef4\u7801\u6587\u4ef6"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconsume a code from the local cache.", "response": "def consume_code(self, code, card_id=None):\n \"\"\"\n \u6d88\u8017 code\n \"\"\"\n card_data = {\n 'code': code\n }\n if card_id:\n card_data['card_id'] = card_id\n return self._post(\n 'card/code/consume',\n data=card_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_code(self, code, card_id=None, check_consume=True):\n card_data = {\n 'code': code\n }\n if card_id:\n card_data['card_id'] = card_id\n if not check_consume:\n card_data['check_consume'] = check_consume\n return self._post(\n 'card/code/get',\n data=card_data\n )", "response": "get code from a card"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a list of cards.", "response": "def get_card_list(self, openid, card_id=None):\n \"\"\"\n \u7528\u4e8e\u83b7\u53d6\u7528\u6237\u5361\u5305\u91cc\u7684\uff0c\u5c5e\u4e8e\u8be5appid\u4e0b\u7684\u5361\u5238\u3002\n \"\"\"\n card_data = {\n 'openid': openid\n }\n if card_id:\n card_data['card_id'] = card_id\n return self._post(\n 'card/user/getcardlist',\n data=card_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a specific resource", "response": "def get(self, card_id):\n \"\"\"\n \u67e5\u8be2\u5361\u5238\u8be6\u60c5\n \"\"\"\n result = self._post(\n 'card/get',\n data={\n 'card_id': card_id\n },\n result_processor=lambda x: x['card']\n )\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the code of a card.", "response": "def update_code(self, card_id, old_code, new_code):\n \"\"\"\n \u66f4\u65b0\u5361\u5238 code\n \"\"\"\n return self._post(\n 'card/code/update',\n data={\n 'card_id': card_id,\n 'code': old_code,\n 'new_code': new_code\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef activate_membercard(self, membership_number, code, **kwargs):\n kwargs['membership_number'] = membership_number\n kwargs['code'] = code\n return self._post(\n 'card/membercard/activate',\n data=kwargs\n )", "response": "\u6fc0\u6d3b\u4f1a\u5458\u5361 - \u63a5\u53e3\u6fc0\u6d3b\u65b9\u5f0f\n \u8be6\u60c5\u8bf7\u53c2\u89c1\n https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1451025283\n\n \u53c2\u6570\u793a\u4f8b\uff1a\n {\n \"init_bonus\": 100,\n \"init_bonus_record\":\"\u65e7\u79ef\u5206\u540c\u6b65\",\n \"init_balance\": 200,\n \"membership_number\": \"AAA00000001\",\n \"code\": \"12312313\",\n \"card_id\": \"xxxx_card_id\",\n \"background_pic_url\": \"https://mmbiz.qlogo.cn/mmbiz/0?wx_fmt=jpeg\",\n \"init_custom_field_value1\": \"xxxxx\",\n \"init_custom_field_value2\": \"xxxxx\",\n \"init_custom_field_value3\": \"xxxxx\"\n }\n\n \u8fd4\u56de\u793a\u4f8b\uff1a\n {\"errcode\":0, \"errmsg\":\"ok\"}\n\n :param membership_number: \u5fc5\u586b\uff0c\u4f1a\u5458\u5361\u7f16\u53f7\uff0c\u7531\u5f00\u53d1\u8005\u586b\u5165\uff0c\u4f5c\u4e3a\u5e8f\u5217\u53f7\u663e\u793a\u5728\u7528\u6237\u7684\u5361\u5305\u91cc\u3002\u53ef\u4e0eCode\u7801\u4fdd\u6301\u7b49\u503c\n :param code: \u5fc5\u586b\uff0c\u9886\u53d6\u4f1a\u5458\u5361\u7528\u6237\u83b7\u5f97\u7684code\n :param kwargs: \u5176\u4ed6\u975e\u5fc5\u586b\u5b57\u6bb5\uff0c\u5305\u542b\u5219\u66f4\u65b0\u5bf9\u5e94\u5b57\u6bb5\u3002\u8be6\u60c5\u53c2\u89c1\u5fae\u4fe1\u6587\u6863 \u201c6 \u6fc0\u6d3b\u4f1a\u5458\u5361\u201d \u90e8\u5206\n :return: \u53c2\u89c1\u8fd4\u56de\u793a\u4f8b"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a Pay gift card to the basket.", "response": "def add_pay_giftcard(self, base_info, extra_info, is_membercard):\n \"\"\"\n \u65b0\u589e\u652f\u4ed8\u540e\u6295\u653e\u5361\u5238\u7684\u89c4\u5219\uff0c\u652f\u6301\u652f\u4ed8\u540e\u9886\u5361\uff0c\u652f\u4ed8\u540e\u8d60\u5238\n \u8be6\u60c5\u8bf7\u53c2\u89c1\n https://mp.weixin.qq.com/wiki?id=mp1466494654_K9rNz\n\n :param base_info: \u8425\u9500\u89c4\u5219\u7ed3\u6784\u4f53\n :type base_info: dict\n :param extra_info: \u652f\u4ed8\u89c4\u5219\u7ed3\u6784\u4f53\n :type extra_info: dict\n :param is_membercard: \u672c\u6b21\u89c4\u5219\u662f\u5426\u662f\u9886\u5361\u3002\uff08\u9886\u5361\u4f20\u5165 True, \u8d60\u5238\u4f20\u5165 False\uff09\n :type is_membercard: bool\n :return: \u89c4\u5219 ID, \u8bbe\u7f6e\u6210\u529f\u7684\u5217\u8868\uff0c\u4ee5\u53ca\u8bbe\u7f6e\u5931\u8d25\u7684\u5217\u8868\n \"\"\"\n if is_membercard:\n rule_key = 'member_rule'\n rule_type = 'RULE_TYPE_PAY_MEMBER_CARD'\n else:\n rule_key = 'single_pay'\n rule_type = 'RULE_TYPE_SINGLE_PAY'\n return self._post(\n 'card/paygiftcard/add',\n data={\n 'rule_info': {\n 'type': rule_type,\n 'base_info': base_info,\n rule_key: extra_info,\n }\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_pay_giftcard(self, rule_id):\n return self._post(\n 'card/paygiftcard/getbyid',\n data={\n 'rule_id': rule_id,\n },\n result_processor=lambda x: x['rule_info'],\n )", "response": "Get pay giftcard by rule_id"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbatches get Pay Gift Card", "response": "def batch_get_pay_giftcard(self, effective=True, offset=0, count=10):\n \"\"\"\n \u6279\u91cf\u67e5\u8be2\u652f\u4ed8\u540e\u6295\u653e\u5361\u5238\u7684\u89c4\u5219\n \u8be6\u60c5\u8bf7\u53c2\u89c1\n https://mp.weixin.qq.com/wiki?id=mp1466494654_K9rNz\n\n\n :param effective: \u662f\u5426\u4ec5\u67e5\u8be2\u751f\u6548\u7684\u89c4\u5219\n :type effective: bool\n :param offset: \u8d77\u59cb\u504f\u79fb\u91cf\n :type offset: int\n :param count: \u67e5\u8be2\u7684\u6570\u91cf\n :type count: int\n :return: \u652f\u4ed8\u540e\u6295\u653e\u5361\u5238\u89c4\u5219\u7684\u603b\u6570\uff0c\u4ee5\u53ca\u67e5\u8be2\u5230\u7684\u5217\u8868\n \"\"\"\n return self._post(\n 'card/paygiftcard/batchget',\n data={\n 'type': 'RULE_TYPE_PAY_MEMBER_CARD',\n 'effective': effective,\n 'offset': offset,\n 'count': count,\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates a movie ticket.", "response": "def update_movie_ticket(self, code, ticket_class, show_time, duration,\n screening_room, seat_number, card_id=None):\n \"\"\"\n \u66f4\u65b0\u7535\u5f71\u7968\n \"\"\"\n ticket = {\n 'code': code,\n 'ticket_class': ticket_class,\n 'show_time': show_time,\n 'duration': duration,\n 'screening_room': screening_room,\n 'seat_number': seat_number\n }\n if card_id:\n ticket['card_id'] = card_id\n return self._post(\n 'card/movieticket/updateuser',\n data=ticket\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_luckymoney_balance(self, code, balance, card_id=None):\n card_data = {\n 'code': code,\n 'balance': balance\n }\n if card_id:\n card_data['card_id'] = card_id\n return self._post(\n 'card/luckymoney/updateuserbalance',\n data=card_data\n )", "response": "Update the user s lucky money balance."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking the codes for a specific card.", "response": "def check_code(self, card_id, codes):\n \"\"\"\n \u6838\u67e5code\n \"\"\"\n card_data = {\n 'card_id': card_id,\n 'code': codes\n }\n return self._post(\n 'card/code/checkcode',\n data=card_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef modify_stock(self, card_id, n):\n if n == 0:\n return\n card_data = {\n 'card_id': card_id,\n }\n if n > 0:\n card_data['increase_stock_value'] = n\n elif n < 0:\n card_data['reduce_stock_value'] = -n\n return self._post(\n 'card/modifystock',\n data=card_data\n )", "response": "Modify the stock value of a card"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the Activate URL for a specific membercard.", "response": "def get_activate_url(self, card_id, outer_str=None):\n \"\"\"\n \u83b7\u53d6\u5f00\u5361\u63d2\u4ef6 Url, \u5185\u542b\u8c03\u7528\u5f00\u5361\u63d2\u4ef6\u6240\u9700\u7684\u53c2\u6570\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1499332673_Unm7V\n\n :param card_id: \u4f1a\u5458\u5361\u7684card_id\n :param outer_str: \u6e20\u9053\u503c\uff0c\u7528\u4e8e\u7edf\u8ba1\u672c\u6b21\u9886\u53d6\u7684\u6e20\u9053\u53c2\u6570\n :return: \u5185\u542b\u8c03\u7528\u5f00\u5361\u63d2\u4ef6\u6240\u9700\u7684\u53c2\u6570\u7684 Url\n \"\"\"\n return self._post(\n 'card/membercard/activate/geturl',\n data={\n 'card_id': card_id,\n 'outer_str': outer_str,\n },\n result_processor=lambda x: x['url'],\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget info about the active user.", "response": "def get_activate_info(self, activate_ticket):\n \"\"\"\n \u83b7\u53d6\u7528\u6237\u5f00\u5361\u65f6\u63d0\u4ea4\u7684\u4fe1\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1499332673_Unm7V\n\n :param activate_ticket: \u8df3\u8f6c\u578b\u5f00\u5361\u7ec4\u4ef6\u5f00\u5361\u540e\u56de\u8c03\u4e2d\u7684\u6fc0\u6d3b\u7968\u636e\uff0c\u53ef\u4ee5\u7528\u6765\u83b7\u53d6\u7528\u6237\u5f00\u5361\u8d44\u6599\n :return: \u7528\u6237\u5f00\u5361\u65f6\u586b\u5199\u7684\u5b57\u6bb5\u503c\n \"\"\"\n return self._post(\n 'card/membercard/activatetempinfo/get',\n data={\n 'activate_ticket': activate_ticket,\n },\n result_processor=lambda x: x['info'],\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_jsapi_signature(self, prepay_id, timestamp=None, nonce_str=None):\n data = {\n 'appId': self.sub_appid or self.appid,\n 'timeStamp': timestamp or to_text(int(time.time())),\n 'nonceStr': nonce_str or random_string(32),\n 'signType': 'MD5',\n 'package': 'prepay_id={0}'.format(prepay_id),\n }\n return calculate_signature(\n data,\n self._client.api_key if not self._client.sandbox else self._client.sandbox_api_key\n )", "response": "\u83b7\u53d6 JSAPI \u7b7e\u540d\n\n :param prepay_id: \u7edf\u4e00\u4e0b\u5355\u63a5\u53e3\u8fd4\u56de\u7684 prepay_id \u53c2\u6570\u503c\n :param timestamp: \u53ef\u9009\uff0c\u65f6\u95f4\u6233\uff0c\u9ed8\u8ba4\u4e3a\u5f53\u524d\u65f6\u95f4\u6233\n :param nonce_str: \u53ef\u9009\uff0c\u968f\u673a\u5b57\u7b26\u4e32\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :return: \u7b7e\u540d"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_jsapi_params(self, prepay_id, timestamp=None, nonce_str=None, jssdk=False):\n data = {\n 'appId': self.sub_appid or self.appid,\n 'timeStamp': timestamp or to_text(int(time.time())),\n 'nonceStr': nonce_str or random_string(32),\n 'signType': 'MD5',\n 'package': 'prepay_id={0}'.format(prepay_id),\n }\n sign = calculate_signature(\n data,\n self._client.api_key if not self._client.sandbox else self._client.sandbox_api_key\n )\n logger.debug('JSAPI payment parameters: data = %s, sign = %s', data, sign)\n data['paySign'] = sign\n if jssdk:\n data['timestamp'] = data.pop('timeStamp')\n return data", "response": "\u83b7\u53d6 JSAPI \u53c2\u6570\n\n :param prepay_id: \u7edf\u4e00\u4e0b\u5355\u63a5\u53e3\u8fd4\u56de\u7684 prepay_id \u53c2\u6570\u503c\n :param timestamp: \u53ef\u9009\uff0c\u65f6\u95f4\u6233\uff0c\u9ed8\u8ba4\u4e3a\u5f53\u524d\u65f6\u95f4\u6233\n :param nonce_str: \u53ef\u9009\uff0c\u968f\u673a\u5b57\u7b26\u4e32\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :param jssdk: \u524d\u7aef\u8c03\u7528\u65b9\u5f0f\uff0c\u9ed8\u8ba4\u4f7f\u7528 WeixinJSBridge\n \u4f7f\u7528 jssdk \u8c03\u8d77\u652f\u4ed8\u7684\u8bdd\uff0ctimestamp \u7684 s \u4e3a\u5c0f\u5199\n \u4f7f\u7528 WeixinJSBridge \u8c03\u8d77\u652f\u4ed8\u7684\u8bdd\uff0ctimeStamp \u7684 S \u4e3a\u5927\u5199\n :return: \u53c2\u6570"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfetch access token from API", "response": "def fetch_access_token(self):\n \"\"\"\n \u83b7\u53d6 access token\n \u8be6\u60c5\u8bf7\u53c2\u8003 http://mp.weixin.qq.com/wiki/index.php?title=\u901a\u7528\u63a5\u53e3\u6587\u6863\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._fetch_access_token(\n url='https://api.weixin.qq.com/cgi-bin/token',\n params={\n 'grant_type': 'client_credential',\n 'appid': self.appid,\n 'secret': self.secret\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fetch_access_token(self):\n expires_in = 7200\n result = self.component.refresh_authorizer_token(\n self.appid, self.refresh_token)\n if 'expires_in' in result:\n expires_in = result['expires_in']\n self.session.set(\n self.access_token_key,\n result['authorizer_access_token'],\n expires_in\n )\n self.expires_at = int(time.time()) + expires_in\n return result", "response": "fetch_access_token \u83b7\u53d6 access token"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update(self, id, name=None, parent_id=None, order=None):\n data = optionaldict(\n id=id,\n name=name,\n parentid=parent_id,\n order=order\n )\n return self._post('department/update', data=data)", "response": "Update a user s log entry."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a single user s company", "response": "def get(self, id=None):\n \"\"\"\n \u83b7\u53d6\u6307\u5b9a\u90e8\u95e8\u5217\u8868\n\n https://work.weixin.qq.com/api/doc#90000/90135/90208\n\n \u6743\u9650\u8bf4\u660e\uff1a\n \u53ea\u80fd\u62c9\u53d6token\u5bf9\u5e94\u7684\u5e94\u7528\u7684\u6743\u9650\u8303\u56f4\u5185\u7684\u90e8\u95e8\u5217\u8868\n\n :param id: \u90e8\u95e8id\u3002\u83b7\u53d6\u6307\u5b9a\u90e8\u95e8\u53ca\u5176\u4e0b\u7684\u5b50\u90e8\u95e8\u3002 \u5982\u679c\u4e0d\u586b\uff0c\u9ed8\u8ba4\u83b7\u53d6\u5168\u91cf\u7ec4\u7ec7\u67b6\u6784\n :return: \u90e8\u95e8\u5217\u8868\n \"\"\"\n if id is None:\n res = self._get('department/list')\n else:\n res = self._get('department/list', params={'id': id})\n return res['department']"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_users(self, id, status=0, fetch_child=0, simple=True):\n url = 'user/simplelist' if simple else 'user/list'\n res = self._get(\n url,\n params={\n 'department_id': id,\n 'status': status,\n 'fetch_child': 1 if fetch_child else 0\n }\n )\n return res['userlist']", "response": "Get a list of users from a user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sync_user(self, url, token, encoding_aes_key, media_id, to_invite=True):\n return self._post(\n 'batch/syncuser',\n data={\n 'media_id': media_id,\n 'to_invite': to_invite,\n 'callback': {\n 'url': url,\n 'token': token,\n 'encodingaeskey': encoding_aes_key\n }\n }\n )", "response": "Sync a user with the given url."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef replace_party(self, url, token, encoding_aes_key, media_id):\n return self._post(\n 'batch/replaceparty',\n data={\n 'media_id': media_id,\n 'callback': {\n 'url': url,\n 'token': token,\n 'encodingaeskey': encoding_aes_key\n }\n }\n )", "response": "Replace the party with the given url."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninvites a user to a specific user.", "response": "def invite(self, user=None, party=None, tag=None):\n \"\"\"\n \u9080\u8bf7\u6210\u5458\n\n https://work.weixin.qq.com/api/doc#90000/90135/90975\n\n \u4f01\u4e1a\u53ef\u901a\u8fc7\u63a5\u53e3\u6279\u91cf\u9080\u8bf7\u6210\u5458\u4f7f\u7528\u4f01\u4e1a\u5fae\u4fe1\uff0c\u9080\u8bf7\u540e\u5c06\u901a\u8fc7\u77ed\u4fe1\u6216\u90ae\u4ef6\u4e0b\u53d1\u901a\u77e5\u3002\n\n :param user: \u6210\u5458ID\u5217\u8868, \u6700\u591a\u652f\u63011000\u4e2a\u3002\n :param party: \u6210\u5458ID\u5217\u8868, \u6700\u591a\u652f\u6301100\u4e2a\u3002\n :param tag: \u6210\u5458ID\u5217\u8868, \u6700\u591a\u652f\u6301100\u4e2a\u3002\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = optionaldict(user=user, party=party, tag=tag)\n return self._post('batch/invite', data=data)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef invite_user(self, url, token, encoding_aes_key, user_ids=None,\n party_ids=None, tag_ids=None, invite_tips=None):\n \"\"\"\n \u9080\u8bf7\u6210\u5458\u5173\u6ce8(deprecated)\n https://qydev.weixin.qq.com/wiki/index.php?title=\u5f02\u6b65\u4efb\u52a1\u63a5\u53e3\n\n :param url: \u4f01\u4e1a\u5e94\u7528\u63a5\u6536\u4f01\u4e1a\u5fae\u4fe1\u63a8\u9001\u8bf7\u6c42\u7684\u8bbf\u95ee\u534f\u8bae\u548c\u5730\u5740\uff0c\u652f\u6301http\u6216https\u534f\u8bae\n :param token: \u7528\u4e8e\u751f\u6210\u7b7e\u540d\n :param encoding_aes_key: \u7528\u4e8e\u6d88\u606f\u4f53\u7684\u52a0\u5bc6\uff0c\u662fAES\u5bc6\u94a5\u7684Base64\u7f16\u7801\n :param user_ids: \u53ef\u9009\uff0c\u6210\u5458ID\u5217\u8868\uff0c\u591a\u4e2a\u63a5\u6536\u8005\u7528\u2018|\u2019\u5206\u9694\uff0c\u6700\u591a\u652f\u63011000\u4e2a\u3002\n :param party_ids: \u53ef\u9009\uff0c\u90e8\u95e8ID\u5217\u8868\uff0c\u591a\u4e2a\u63a5\u6536\u8005\u7528\u2018|\u2019\u5206\u9694\uff0c\u6700\u591a\u652f\u6301100\u4e2a\u3002\n :param tag_ids: \u53ef\u9009\uff0c\u6807\u7b7eID\u5217\u8868\uff0c\u591a\u4e2a\u63a5\u6536\u8005\u7528\u2018|\u2019\u5206\u9694\u3002\n :param invite_tips: \u53ef\u9009\uff0c\u63a8\u9001\u5230\u5fae\u4fe1\u4e0a\u7684\u63d0\u793a\u8bed\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = optionaldict()\n data['callback'] = {\n 'url': url,\n 'token': token,\n 'encodingaeskey': encoding_aes_key\n }\n if isinstance(user_ids, (tuple, list)):\n user_ids = '|'.join(map(to_text, user_ids))\n if isinstance(party_ids, (tuple, list)):\n party_ids = '|'.join(map(to_text, party_ids))\n if isinstance(tag_ids, (tuple, list)):\n tag_ids = '|'.join(map(to_text, tag_ids))\n data['touser'] = user_ids\n data['toparty'] = party_ids\n data['totag'] = tag_ids\n data['invite_tips'] = invite_tips\n return self._post('batch/inviteuser', data=data)", "response": "Invite a user to a specific party."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new group", "response": "def create(self, name):\n \"\"\"\n \u521b\u5efa\u5206\u7ec4\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/0/56d992c605a97245eb7e617854b169fc.html\n\n :param name: \u5206\u7ec4\u540d\u5b57\uff0830\u4e2a\u5b57\u7b26\u4ee5\u5185\uff09\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.group.create('New Group')\n\n \"\"\"\n name = to_text(name)\n return self._post(\n 'groups/create',\n data={'group': {'name': name}}\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, user_id=None):\n if user_id is None:\n res = self._get(\n 'groups/get',\n result_processor=lambda x: x['groups']\n )\n else:\n res = self._post(\n 'groups/getid',\n data={'openid': user_id},\n result_processor=lambda x: x['groupid']\n )\n return res", "response": "Get a user s group info."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, group_id, name):\n name = to_text(name)\n return self._post(\n 'groups/update',\n data={\n 'group': {\n 'id': int(group_id),\n 'name': name\n }\n }\n )", "response": "\u4fee\u6539\u5206\u7ec4\u540d\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/0/56d992c605a97245eb7e617854b169fc.html\n\n :param group_id: \u5206\u7ec4id\uff0c\u7531\u5fae\u4fe1\u5206\u914d\n :param name: \u5206\u7ec4\u540d\u5b57\uff0830\u4e2a\u5b57\u7b26\u4ee5\u5185\uff09\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.group.update(1234, 'New Name')"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmoves user to group", "response": "def move_user(self, user_id, group_id):\n \"\"\"\n \u79fb\u52a8\u7528\u6237\u5206\u7ec4\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/0/56d992c605a97245eb7e617854b169fc.html\n\n :param user_id: \u7528\u6237 ID, \u53ef\u4ee5\u662f\u5355\u4e2a\u6216\u8005\u5217\u8868\uff0c\u4e3a\u5217\u8868\u65f6\u4e3a\u6279\u91cf\u79fb\u52a8\u7528\u6237\u5206\u7ec4\n :param group_id: \u5206\u7ec4 ID\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.group.move_user('openid', 1234)\n\n \"\"\"\n data = {'to_groupid': group_id}\n if isinstance(user_id, (tuple, list)):\n endpoint = 'groups/members/batchupdate'\n data['openid_list'] = user_id\n else:\n endpoint = 'groups/members/update'\n data['openid'] = user_id\n return self._post(endpoint, data=data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking weChat callback signature", "response": "def check_signature(token, signature, timestamp, nonce):\n \"\"\"Check WeChat callback signature, raises InvalidSignatureException\n if check failed.\n\n :param token: WeChat callback token\n :param signature: WeChat callback signature sent by WeChat server\n :param timestamp: WeChat callback timestamp sent by WeChat server\n :param nonce: WeChat callback nonce sent by WeChat sever\n \"\"\"\n signer = WeChatSigner()\n signer.add_data(token, timestamp, nonce)\n if signer.signature != signature:\n from wechatpy.exceptions import InvalidSignatureException\n\n raise InvalidSignatureException()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting value to unicode", "response": "def to_text(value, encoding='utf-8'):\n \"\"\"Convert value to unicode, default encoding is utf-8\n\n :param value: Value to be converted\n :param encoding: Desired encoding\n \"\"\"\n if not value:\n return ''\n if isinstance(value, six.text_type):\n return value\n if isinstance(value, six.binary_type):\n return value.decode(encoding)\n return six.text_type(value)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_binary(value, encoding='utf-8'):\n if not value:\n return b''\n if isinstance(value, six.binary_type):\n return value\n if isinstance(value, six.text_type):\n return value.encode(encoding)\n return to_text(value).encode(encoding)", "response": "Convert value to binary string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef timezone(zone):\n try:\n import pytz\n return pytz.timezone(zone)\n except ImportError:\n pass\n try:\n from dateutil.tz import gettz\n return gettz(zone)\n except ImportError:\n return None", "response": "Try to get timezone using pytz or python - dateutil. tz or None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_querystring(uri):\n parts = urlparse.urlsplit(uri)\n return urlparse.parse_qs(parts.query)", "response": "Get querystring info from uri."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd data to signer", "response": "def add_data(self, *args):\n \"\"\"Add data to signer\"\"\"\n for data in args:\n self._data.append(to_binary(data))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef signature(self):\n self._data.sort()\n str_to_sign = self._delimiter.join(self._data)\n return hashlib.sha1(str_to_sign).hexdigest()", "response": "Get the signature of the data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists all Shops in a specific page.", "response": "def list_shops(self, page_index=1, page_size=20):\n \"\"\"\n \u83b7\u53d6\u95e8\u5e97\u5217\u8868\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/15/bcfb5d4578ea818b89913472cf2bbf8f.html\n\n :param page_index: \u53ef\u9009\uff0c\u5206\u9875\u4e0b\u6807\uff0c\u9ed8\u8ba4\u4ece1\u5f00\u59cb\n :param page_size: \u53ef\u9009\uff0c\u6bcf\u9875\u7684\u4e2a\u6570\uff0c\u9ed8\u8ba420\u4e2a\uff0c\u6700\u592720\u4e2a\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n res = self._post(\n 'shop/list',\n data={\n 'pageindex': page_index,\n 'pagesize': page_size,\n },\n result_processor=lambda x: x['data']\n )\n return res"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_shop(self, shop_id=0):\n res = self._post(\n 'shop/get',\n data={\n 'shop_id': shop_id,\n },\n result_processor=lambda x: x['data']\n )\n return res", "response": "Get a specific shop from the API."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_device(self, shop_id, ssid, password, bssid):\n return self._post(\n 'device/add',\n data={\n 'shop_id': shop_id,\n 'ssid': ssid,\n 'password': password,\n 'bssid': bssid,\n }\n )", "response": "add_device - Add a new device to the database"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list_devices(self, shop_id=None, page_index=1, page_size=20):\n data = optionaldict(\n shop_id=shop_id,\n pageindex=page_index,\n pagesize=page_size\n )\n res = self._post(\n 'device/list',\n data=data,\n result_processor=lambda x: x['data']\n )\n return res", "response": "List all devices in a shop."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_qrcode_url(self, shop_id, img_id):\n res = self._post(\n 'qrcode/get',\n data={\n 'shop_id': shop_id,\n 'img_id': img_id,\n },\n result_processor=lambda x: x['data']['qrcode_url']\n )\n return res", "response": "Get QR Code URL."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_homepage(self, shop_id):\n res = self._post(\n 'homepage/get',\n data={'shop_id': shop_id},\n result_processor=lambda x: x['data']\n )\n return res", "response": "\u67e5\u8be2\u5546\u5bb6\u4e3b\u9875\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/6/2732f3cf83947e0e4971aa8797ee9d6a.html\n\n :param shop_id: \u95e8\u5e97 ID\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef list_statistics(self, begin_date, end_date, shop_id=-1):\n if isinstance(begin_date, (datetime, date)):\n begin_date = begin_date.strftime('%Y-%m-%d')\n if isinstance(end_date, (datetime, date)):\n end_date = end_date.strftime('%Y-%m-%d')\n res = self._post(\n 'statistics/list',\n data={\n 'begin_date': begin_date,\n 'end_date': end_date,\n 'shop_id': shop_id\n },\n result_processor=lambda x: x['data']\n )\n return res", "response": "List the statistics for a given date range."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef authorize_url(self, redirect_uri, state=None):\n redirect_uri = six.moves.urllib.parse.quote(redirect_uri, safe=b'')\n url_list = [\n self.OAUTH_BASE_URL,\n '?appid=',\n self._client.corp_id,\n '&redirect_uri=',\n redirect_uri,\n '&response_type=code&scope=snsapi_base',\n ]\n if state:\n url_list.extend(['&state=', state])\n url_list.append('#wechat_redirect')\n return ''.join(url_list)", "response": "URL to authorize the user"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create(self, body, total_fee, auth_code, client_ip=None, out_trade_no=None, detail=None, attach=None,\n fee_type='CNY', goods_tag=None, device_info=None, limit_pay=None):\n \"\"\"\n \u5237\u5361\u652f\u4ed8\u63a5\u53e3\n :param device_info: \u53ef\u9009\uff0c\u7ec8\u7aef\u8bbe\u5907\u53f7(\u5546\u6237\u81ea\u5b9a\u4e49\uff0c\u5982\u95e8\u5e97\u7f16\u53f7)\n :param body: \u5546\u54c1\u63cf\u8ff0\n :param detail: \u53ef\u9009\uff0c\u5546\u54c1\u8be6\u60c5\n :param attach: \u53ef\u9009\uff0c\u9644\u52a0\u6570\u636e\uff0c\u5728\u67e5\u8be2API\u548c\u652f\u4ed8\u901a\u77e5\u4e2d\u539f\u6837\u8fd4\u56de\uff0c\u8be5\u5b57\u6bb5\u4e3b\u8981\u7528\u4e8e\u5546\u6237\u643a\u5e26\u8ba2\u5355\u7684\u81ea\u5b9a\u4e49\u6570\u636e\n :param client_ip: \u53ef\u9009\uff0cAPP\u548c\u7f51\u9875\u652f\u4ed8\u63d0\u4ea4\u7528\u6237\u7aefip\uff0cNative\u652f\u4ed8\u586b\u8c03\u7528\u5fae\u4fe1\u652f\u4ed8API\u7684\u673a\u5668IP\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u8ba2\u5355\u53f7\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :param total_fee: \u603b\u91d1\u989d\uff0c\u5355\u4f4d\u5206\n :param fee_type: \u53ef\u9009\uff0c\u7b26\u5408ISO 4217\u6807\u51c6\u7684\u4e09\u4f4d\u5b57\u6bcd\u4ee3\u7801\uff0c\u9ed8\u8ba4\u4eba\u6c11\u5e01\uff1aCNY\n :param goods_tag: \u53ef\u9009\uff0c\u5546\u54c1\u6807\u8bb0\uff0c\u4ee3\u91d1\u5238\u6216\u7acb\u51cf\u4f18\u60e0\u529f\u80fd\u7684\u53c2\u6570\n :param limit_pay: \u53ef\u9009\uff0c\u6307\u5b9a\u652f\u4ed8\u65b9\u5f0f\uff0cno_credit--\u6307\u5b9a\u4e0d\u80fd\u4f7f\u7528\u4fe1\u7528\u5361\u652f\u4ed8\n :param auth_code: \u6388\u6743\u7801\uff0c\u626b\u7801\u652f\u4ed8\u6388\u6743\u7801\uff0c\u8bbe\u5907\u8bfb\u53d6\u7528\u6237\u5fae\u4fe1\u4e2d\u7684\u6761\u7801\u6216\u8005\u4e8c\u7ef4\u7801\u4fe1\u606f\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\n \"\"\"\n now = datetime.now()\n if not out_trade_no:\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n data = {\n 'appid': self.appid,\n 'device_info': device_info,\n 'body': body,\n 'detail': detail,\n 'attach': attach,\n 'out_trade_no': out_trade_no,\n 'total_fee': total_fee,\n 'fee_type': fee_type,\n 'spbill_create_ip': client_ip or get_external_ip(),\n 'goods_tag': goods_tag,\n 'limit_pay': limit_pay,\n 'auth_code': auth_code,\n }\n return self._post('pay/micropay', data=data)", "response": "Create a new object in the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new tag.", "response": "def create(self, name):\n \"\"\"\n \u521b\u5efa\u6807\u7b7e\n\n :param name: \u6807\u7b7e\u540d\uff0830\u4e2a\u5b57\u7b26\u4ee5\u5185\uff09\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \"\"\"\n name = to_text(name)\n return self._post(\n 'tags/create',\n data={'tag': {'name': name}},\n result_processor=lambda x: x['tag']\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update(self, tag_id, name):\n name = to_text(name)\n return self._post(\n 'tags/update',\n data={\n 'tag': {\n 'id': int(tag_id),\n 'name': name\n }\n }\n )", "response": "Update the tag with the given id and name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tag_user(self, tag_id, user_id):\n data = {'tagid': tag_id}\n if isinstance(user_id, (tuple, list)):\n data['openid_list'] = user_id\n else:\n data['openid_list'] = [user_id, ]\n return self._post('tags/members/batchtagging', data=data)", "response": "Tag a user with a specific ID."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef iter_tag_users(self, tag_id, first_user_id=None):\n while True:\n follower_data = self.get_tag_users(tag_id, first_user_id)\n if 'data' not in follower_data:\n return\n for openid in follower_data['data']['openid']:\n yield openid\n first_user_id = follower_data.get('next_openid')\n if not first_user_id:\n return", "response": "Iterate over tag_id user ids."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_black_list(self, begin_openid=None):\n data = {}\n if begin_openid:\n data['begin_openid'] = begin_openid\n return self._post(\n 'tags/members/getblacklist',\n data=data,\n )", "response": "Get the blacklist of the user."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets Wxa Code Unlimited.", "response": "def get_wxa_code_unlimited(self,\n scene,\n width=430,\n auto_color=False,\n line_color={\"r\": \"0\", \"g\": \"0\", \"b\": \"0\"},\n page=None,\n is_hyaline=False):\n \"\"\"\n \u521b\u5efa\u5c0f\u7a0b\u5e8f\u7801\uff08\u63a5\u53e3B\uff1a\u9002\u7528\u4e8e\u9700\u8981\u7684\u7801\u6570\u91cf\u6781\u591a\uff0c\u6216\u4ec5\u4e34\u65f6\u4f7f\u7528\u7684\u4e1a\u52a1\u573a\u666f\uff09\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/debug/wxadoc/dev/api/qrcode.html\n \"\"\"\n return self._post(\n 'wxa/getwxacodeunlimit',\n data=optionaldict(\n scene=scene,\n page=page,\n width=width,\n auto_color=auto_color,\n line_color=line_color,\n is_hyaline=is_hyaline,\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a template message to a user.", "response": "def send_template_message(self, user_id, template_id, data, form_id, page=None, color=None, emphasis_keyword=None):\n \"\"\"\n \u53d1\u9001\u6a21\u677f\u6d88\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/debug/wxadoc/dev/api/notice.html\n \"\"\"\n tpl_data = optionaldict(\n touser=user_id,\n template_id=template_id,\n page=page,\n form_id=form_id,\n data=data,\n color=color,\n emphasis_keyword=emphasis_keyword,\n )\n return self._post(\n 'cgi-bin/message/wxopen/template/send',\n data=tpl_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmodifies the current domain.", "response": "def modify_domain(self, action, request_domain=(), wsrequest_domain=(), upload_domain=(), download_domain=()):\n \"\"\"\n \u4fee\u6539\u5c0f\u7a0b\u5e8f\u670d\u52a1\u5668\u6388\u6743\u57df\u540d\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&id=open1489138143_WPbOO\n\n :param action: \u589e\u5220\u6539\u67e5\u7684\u64cd\u4f5c\u7c7b\u578b\uff0c\u4ec5\u652f\u6301 'add', 'delete', 'set', 'get'\n :param request_domain: request \u5408\u6cd5\u57df\u540d\n :param wsrequest_domain: socket \u5408\u6cd5\u57df\u540d\n :param upload_domain: upload file \u5408\u6cd5\u57df\u540d\n :param download_domain: download file \u5408\u6cd5\u57df\u540d\n \"\"\"\n return self._post(\n 'wxa/modify_domain',\n data={\n 'action': action,\n 'requestdomain': request_domain,\n 'wsrequestdomain': wsrequest_domain,\n 'uploaddomain': upload_domain,\n 'downloaddomain': download_domain,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncommit the current state of the current user.", "response": "def commit(self, template_id, ext_json, version, description):\n \"\"\"\n \u4e3a\u6388\u6743\u7684\u5c0f\u7a0b\u5e8f\u8d26\u53f7\u4e0a\u4f20\u5c0f\u7a0b\u5e8f\u4ee3\u7801\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&id=open1489140610_Uavc4\n\n :param template_id: \u4ee3\u7801\u5e93\u4e2d\u7684\u4ee3\u7801\u6a21\u677f ID\n :param ext_json: \u7b2c\u4e09\u65b9\u81ea\u5b9a\u4e49\u7684\u914d\u7f6e\n :param version: \u4ee3\u7801\u7248\u672c\u53f7\uff0c\u5f00\u53d1\u8005\u53ef\u81ea\u5b9a\u4e49\n :param description: \u4ee3\u7801\u63cf\u8ff0\uff0c\u5f00\u53d1\u8005\u53ef\u81ea\u5b9a\u4e49\n \"\"\"\n return self._post(\n 'wxa/commit',\n data={\n 'template_id': template_id,\n 'ext_json': ext_json,\n 'user_version': version,\n 'user_desc': description,\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef submit_audit(self, item_list):\n return self._post(\n 'wxa/submit_audit',\n data={\n 'item_list': item_list,\n },\n result_processor=lambda x: x['auditid'],\n )", "response": "Submit audit to the user."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists all templates in the current language.", "response": "def list_templates(self, offset=0, count=20):\n \"\"\"\n \u83b7\u53d6\u672c\u8d26\u53f7\u5185\u6240\u6709\u6a21\u677f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&id=open1500465446_j4CgR\n\n :param offset: \u7528\u4e8e\u5206\u9875\uff0c\u8868\u793a\u8d77\u59cb\u91cf\uff0c\u6700\u5c0f\u503c\u4e3a0\n :type offset: int\n :param count: \u7528\u4e8e\u5206\u9875\uff0c\u8868\u793a\u62c9\u53d6\u6570\u91cf\uff0c\u6700\u5927\u503c\u4e3a20\n :type count: int\n :return: \u6a21\u677f\u5217\u8868\n :rtype: list[dict]\n \"\"\"\n return self._post(\n 'cgi-bin/wxopen/template/list',\n data={\n 'offset': offset,\n 'count': count,\n },\n result_processor=lambda x: x['list'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_template(self, template_short_id, keyword_id_list):\n return self._post(\n 'cgi-bin/wxopen/template/add',\n data={\n 'id': template_short_id,\n 'keyword_id_list': keyword_id_list,\n },\n result_processor=lambda x: x['template_id'],\n )", "response": "Add a template to the list of keywords."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_open(self, appid):\n return self._post(\n 'cgi-bin/open/create',\n data={\n 'appid': appid,\n },\n result_processor=lambda x: x['open_appid'],\n )", "response": "\u521b\u5efa\u5f00\u653e\u5e73\u53f0\u8d26\u53f7\uff0c\u5e76\u7ed1\u5b9a\u516c\u4f17\u53f7/\u5c0f\u7a0b\u5e8f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://open.weixin.qq.com/cgi-bin/showdocument?action=dir_list&id=open1498704199_1bcax\n\n :param appid: \u6388\u6743\u516c\u4f17\u53f7\u6216\u5c0f\u7a0b\u5e8f\u7684 appid\n :return: \u5f00\u653e\u5e73\u53f0\u7684 appid"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_open(self, appid):\n return self._post(\n 'cgi-bin/open/get',\n data={\n 'appid': appid,\n },\n result_processor=lambda x: x['open_appid'],\n )", "response": "Get the open related items"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsearch for a given object.", "response": "def search(self,\n query,\n category,\n uid=None,\n latitude=None,\n longitude=None,\n city=None,\n region=None):\n \"\"\"\n \u53d1\u9001\u8bed\u4e49\u7406\u89e3\u8bf7\u6c42\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/0/0ce78b3c9524811fee34aba3e33f3448.html\n\n :param query: \u8f93\u5165\u6587\u672c\u4e32\n :param category: \u9700\u8981\u4f7f\u7528\u7684\u670d\u52a1\u7c7b\u578b\uff0c\u591a\u4e2a\u53ef\u4f20\u5165\u5217\u8868\n :param uid: \u53ef\u9009\uff0c\u7528\u6237\u552f\u4e00id\uff08\u975e\u5f00\u53d1\u8005id\uff09\uff0c\u7528\u6237\u533a\u5206\u516c\u4f17\u53f7\u4e0b\u7684\u4e0d\u540c\u7528\u6237\uff08\u5efa\u8bae\u586b\u5165\u7528\u6237openid\uff09\n :param latitude: \u53ef\u9009\uff0c\u7eac\u5ea6\u5750\u6807\uff0c\u4e0e\u7ecf\u5ea6\u540c\u65f6\u4f20\u5165\uff1b\u4e0e\u57ce\u5e02\u4e8c\u9009\u4e00\u4f20\u5165\n :param longitude: \u53ef\u9009\uff0c\u7ecf\u5ea6\u5750\u6807\uff0c\u4e0e\u7eac\u5ea6\u540c\u65f6\u4f20\u5165\uff1b\u4e0e\u57ce\u5e02\u4e8c\u9009\u4e00\u4f20\u5165\n :param city: \u53ef\u9009\uff0c\u57ce\u5e02\u540d\u79f0\uff0c\u4e0e\u7ecf\u7eac\u5ea6\u4e8c\u9009\u4e00\u4f20\u5165\n :param region: \u53ef\u9009\uff0c\u533a\u57df\u540d\u79f0\uff0c\u5728\u57ce\u5e02\u5b58\u5728\u7684\u60c5\u51b5\u4e0b\u53ef\u7701\uff1b\u4e0e\u7ecf\u7eac\u5ea6\u4e8c\u9009\u4e00\u4f20\u5165\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.semantic.search(\n '\u67e5\u4e00\u4e0b\u660e\u5929\u4ece\u5317\u4eac\u5230\u4e0a\u6d77\u7684\u5357\u822a\u673a\u7968',\n 'flight,hotel',\n city='\u5317\u4eac'\n )\n\n \"\"\"\n if isinstance(category, (tuple, list)):\n category = ','.join(category)\n\n data = optionaldict()\n data['query'] = query\n data['category'] = category\n data['uid'] = uid\n data['latitude'] = latitude\n data['longitude'] = longitude\n data['city'] = city\n data['region'] = region\n data['appid'] = self._client.appid\n return self._post(\n url='https://api.weixin.qq.com/semantic/semproxy/search',\n data=data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the object properties of the object.", "response": "def set(self,\n agent_id,\n name=None,\n description=None,\n redirect_domain=None,\n logo_media_id=None,\n report_location_flag=0,\n is_report_user=True,\n is_report_enter=True):\n \"\"\"\n \u8bbe\u7f6e\u5e94\u7528\n https://work.weixin.qq.com/api/doc#90000/90135/90228\n\n :param agent_id: \u4f01\u4e1a\u5e94\u7528\u7684id\n :param name: \u4f01\u4e1a\u5e94\u7528\u540d\u79f0\uff0c\u957f\u5ea6\u4e0d\u8d85\u8fc732\u4e2autf8\u5b57\u7b26\n :param description: \u4f01\u4e1a\u5e94\u7528\u8be6\u60c5\uff0c\u957f\u5ea6\u4e3a4\u81f3120\u4e2autf8\u5b57\u7b26\n :param redirect_domain: \u4f01\u4e1a\u5e94\u7528\u53ef\u4fe1\u57df\u540d\u3002\u6ce8\u610f\uff1a\u57df\u540d\u9700\u901a\u8fc7\u6240\u6709\u6743\u6821\u9a8c\uff0c\u5426\u5219jssdk\u529f\u80fd\u5c06\u53d7\u9650\uff0c\u6b64\u65f6\u8fd4\u56de\u9519\u8bef\u780185005\n :param logo_media_id: \u4f01\u4e1a\u5e94\u7528\u5934\u50cf\u7684mediaid\uff0c\u901a\u8fc7\u7d20\u6750\u7ba1\u7406\u63a5\u53e3\u4e0a\u4f20\u56fe\u7247\u83b7\u5f97mediaid\uff0c\u4e0a\u4f20\u540e\u4f1a\u81ea\u52a8\u88c1\u526a\u6210\u65b9\u5f62\u548c\u5706\u5f62\u4e24\u4e2a\u5934\u50cf\n :param report_location_flag: \u4f01\u4e1a\u5e94\u7528\u662f\u5426\u6253\u5f00\u5730\u7406\u4f4d\u7f6e\u4e0a\u62a5 0\uff1a\u4e0d\u4e0a\u62a5\uff1b1\uff1a\u8fdb\u5165\u4f1a\u8bdd\u4e0a\u62a5\uff1b\n :param is_report_enter: \u662f\u5426\u4e0a\u62a5\u7528\u6237\u8fdb\u5165\u5e94\u7528\u4e8b\u4ef6\u30020\uff1a\u4e0d\u63a5\u6536\uff1b1\uff1a\u63a5\u6536\u3002\n :param is_report_user: \u662f\u5426\u63a5\u6536\u7528\u6237\u53d8\u66f4\u901a\u77e5\u30020\uff1a\u4e0d\u63a5\u6536\uff1b1\uff1a\u63a5\u6536\u3002\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n agent_data = optionaldict()\n agent_data['agentid'] = agent_id\n agent_data['name'] = name\n agent_data['description'] = description\n agent_data['redirect_domain'] = redirect_domain\n agent_data['logo_mediaid'] = logo_media_id\n agent_data['report_location_flag'] = report_location_flag\n agent_data['isreportenter'] = 1 if is_report_enter else 0\n agent_data['isreportuser'] = 1 if is_report_user else 0\n return self._post(\n 'agent/set',\n data=agent_data\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_jsapi_signature(self, noncestr, ticket, timestamp, url):\n data = [\n 'noncestr={noncestr}'.format(noncestr=noncestr),\n 'jsapi_ticket={ticket}'.format(ticket=ticket),\n 'timestamp={timestamp}'.format(timestamp=timestamp),\n 'url={url}'.format(url=url),\n ]\n signer = WeChatSigner(delimiter=b'&')\n signer.add_data(*data)\n return signer.signature", "response": "Get the signature of a JS - API message."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\napplies the refund to the base transaction.", "response": "def apply(self, total_fee, refund_fee, out_refund_no, transaction_id=None,\n out_trade_no=None, fee_type='CNY', op_user_id=None,\n device_info=None, refund_account='REFUND_SOURCE_UNSETTLED_FUNDS',\n notify_url=None):\n \"\"\"\n \u7533\u8bf7\u9000\u6b3e\n\n :param total_fee: \u8ba2\u5355\u603b\u91d1\u989d\uff0c\u5355\u4f4d\u4e3a\u5206\n :param refund_fee: \u9000\u6b3e\u603b\u91d1\u989d\uff0c\u5355\u4f4d\u4e3a\u5206\n :param out_refund_no: \u5546\u6237\u7cfb\u7edf\u5185\u90e8\u7684\u9000\u6b3e\u5355\u53f7\uff0c\u5546\u6237\u7cfb\u7edf\u5185\u90e8\u552f\u4e00\uff0c\u540c\u4e00\u9000\u6b3e\u5355\u53f7\u591a\u6b21\u8bf7\u6c42\u53ea\u9000\u4e00\u7b14\n :param transaction_id: \u53ef\u9009\uff0c\u5fae\u4fe1\u8ba2\u5355\u53f7\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u7cfb\u7edf\u5185\u90e8\u7684\u8ba2\u5355\u53f7\uff0c\u4e0e transaction_id \u4e8c\u9009\u4e00\n :param fee_type: \u53ef\u9009\uff0c\u8d27\u5e01\u7c7b\u578b\uff0c\u7b26\u5408ISO 4217\u6807\u51c6\u7684\u4e09\u4f4d\u5b57\u6bcd\u4ee3\u7801\uff0c\u9ed8\u8ba4\u4eba\u6c11\u5e01\uff1aCNY\n :param op_user_id: \u53ef\u9009\uff0c\u64cd\u4f5c\u5458\u5e10\u53f7, \u9ed8\u8ba4\u4e3a\u5546\u6237\u53f7\n :param device_info: \u53ef\u9009\uff0c\u7ec8\u7aef\u8bbe\u5907\u53f7\n :param refund_account: \u53ef\u9009\uff0c\u9000\u6b3e\u8d44\u91d1\u6765\u6e90\uff0c\u4ec5\u9488\u5bf9\u8001\u8d44\u91d1\u6d41\u5546\u6237\u4f7f\u7528\uff0c\u9ed8\u8ba4\u4f7f\u7528\u672a\u7ed3\u7b97\u8d44\u91d1\u9000\u6b3e\n :param notify_url: \u53ef\u9009\uff0c\u5f02\u6b65\u63a5\u6536\u5fae\u4fe1\u652f\u4ed8\u9000\u6b3e\u7ed3\u679c\u901a\u77e5\u7684\u56de\u8c03\u5730\u5740\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\n \"\"\"\n data = {\n 'appid': self.appid,\n 'device_info': device_info,\n 'transaction_id': transaction_id,\n 'out_trade_no': out_trade_no,\n 'out_refund_no': out_refund_no,\n 'total_fee': total_fee,\n 'refund_fee': refund_fee,\n 'refund_fee_type': fee_type,\n 'op_user_id': op_user_id if op_user_id else self.mch_id,\n 'refund_account': refund_account,\n 'notify_url': notify_url,\n }\n return self._post('secapi/pay/refund', data=data)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef query(self, refund_id=None, out_refund_no=None, transaction_id=None,\n out_trade_no=None, device_info=None):\n \"\"\"\n \u67e5\u8be2\u9000\u6b3e\n\n :param refund_id: \u5fae\u4fe1\u9000\u6b3e\u5355\u53f7\n :param out_refund_no: \u5546\u6237\u9000\u6b3e\u5355\u53f7\n :param transaction_id: \u5fae\u4fe1\u8ba2\u5355\u53f7\n :param out_trade_no: \u5546\u6237\u7cfb\u7edf\u5185\u90e8\u7684\u8ba2\u5355\u53f7\n :param device_info: \u53ef\u9009\uff0c\u7ec8\u7aef\u8bbe\u5907\u53f7\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\n \"\"\"\n data = {\n 'appid': self.appid,\n 'device_info': device_info,\n 'transaction_id': transaction_id,\n 'out_trade_no': out_trade_no,\n 'out_refund_no': out_refund_no,\n 'refund_id': refund_id,\n }\n return self._post('pay/refundquery', data=data)", "response": "Query the refund_id and out_refund_no for a specific resource."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_stock(self, product_id, sku_info, quantity):\n return self._post(\n 'merchant/stock/add',\n data={\n \"product_id\": product_id,\n \"sku_info\": sku_info,\n \"quantity\": quantity\n }\n )", "response": "Add stock to a product"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reduce_stock(self, product_id, sku_info, quantity):\n return self._post(\n 'merchant/stock/reduce',\n data={\n \"product_id\": product_id,\n \"sku_info\": sku_info,\n \"quantity\": quantity\n }\n )", "response": "Reduce the stock for a product."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the delivery template for a specific exchange", "response": "def update_express(self, template_id, delivery_template):\n \"\"\"\n \u589e\u52a0\u90ae\u8d39\u6a21\u677f\n\n :param template_id: \u90ae\u8d39\u6a21\u677fID\n :param delivery_template: \u90ae\u8d39\u6a21\u677f\u4fe1\u606f(\u5b57\u6bb5\u8bf4\u660e\u8be6\u89c1\u589e\u52a0\u90ae\u8d39\u6a21\u677f)\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n delivery_template['template_id'] = template_id\n return self._post(\n 'merchant/express/update',\n data=delivery_template\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_group_property(self, group_id, group_properties):\n group_properties['group_id'] = group_id\n return self._post(\n 'merchant/group/propertymod',\n data=group_properties\n )", "response": "\u4fee\u6539\u5206\u7ec4\u5c5e\u6027\n\n :param group_id: \u5546\u54c1\u5206\u7ec4ID\n :param group_properties: \u5546\u54c1\u5206\u7ec4\u5c5e\u6027\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates the product of a group", "response": "def update_group_product(self, group_id, product_data):\n \"\"\"\n \u4fee\u6539\u5206\u7ec4\u5546\u54c1\n\n :param group_id: \u5546\u54c1\u5206\u7ec4ID\n :param product_data: \u5206\u7ec4\u5546\u54c1\u4fe1\u606f\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n product_data['group_id'] = group_id\n return self._post(\n 'merchant/group/productmod',\n data=product_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_shelf(self, shelf_id, shelf_data):\n shelf_data['shelf_id'] = shelf_id\n return self._post(\n 'merchant/shelf/mod',\n data=shelf_data\n )", "response": "Update the shelf data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nquery the order by status", "response": "def query_order(self, status=None, begintime=None, endtime=None):\n \"\"\"\n \u6839\u636e\u8ba2\u5355\u72b6\u6001/\u521b\u5efa\u65f6\u95f4\u83b7\u53d6\u8ba2\u5355\u8be6\u60c5\n\n :param status: \u8ba2\u5355\u72b6\u6001(\u4e0d\u5e26\u8be5\u5b57\u6bb5-\u5168\u90e8\u72b6\u6001, 2-\u5f85\u53d1\u8d27, 3-\u5df2\u53d1\u8d27, 5-\u5df2\u5b8c\u6210, 8-\u7ef4\u6743\u4e2d, )\n :param begintime: \u8ba2\u5355\u521b\u5efa\u65f6\u95f4\u8d77\u59cb\u65f6\u95f4(\u4e0d\u5e26\u8be5\u5b57\u6bb5\u5219\u4e0d\u6309\u7167\u65f6\u95f4\u505a\u7b5b\u9009)\n :param endtime: \u8ba2\u5355\u521b\u5efa\u65f6\u95f4\u7ec8\u6b62\u65f6\u95f4(\u4e0d\u5e26\u8be5\u5b57\u6bb5\u5219\u4e0d\u6309\u7167\u65f6\u95f4\u505a\u7b5b\u9009)\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._post(\n 'merchant/order/getbyfilter',\n data={\n 'status': status,\n 'begintime': begintime,\n 'endtime': endtime\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef download_bill(self, bill_date, bill_type='ALL', device_info=None):\n if isinstance(bill_date, (datetime, date)):\n bill_date = bill_date.strftime('%Y%m%d')\n\n data = {\n 'appid': self.appid,\n 'bill_date': bill_date,\n 'bill_type': bill_type,\n 'device_info': device_info,\n }\n return self._post('pay/downloadbill', data=data)", "response": "\u4e0b\u8f7d\u5bf9\u8d26\u5355\n\n :param bill_date: \u4e0b\u8f7d\u5bf9\u8d26\u5355\u7684\u65e5\u671f\n :param bill_type: \u8d26\u5355\u7c7b\u578b\uff0cALL\uff0c\u8fd4\u56de\u5f53\u65e5\u6240\u6709\u8ba2\u5355\u4fe1\u606f\uff0c\u9ed8\u8ba4\u503c\n SUCCESS\uff0c\u8fd4\u56de\u5f53\u65e5\u6210\u529f\u652f\u4ed8\u7684\u8ba2\u5355,\n REFUND\uff0c\u8fd4\u56de\u5f53\u65e5\u9000\u6b3e\u8ba2\u5355,\n REVOKED\uff0c\u5df2\u64a4\u9500\u7684\u8ba2\u5355\n :param device_info: \u5fae\u4fe1\u652f\u4ed8\u5206\u914d\u7684\u7ec8\u7aef\u8bbe\u5907\u53f7\uff0c\u586b\u5199\u6b64\u5b57\u6bb5\uff0c\u53ea\u4e0b\u8f7d\u8be5\u8bbe\u5907\u53f7\u7684\u5bf9\u8d26\u5355\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef download_fundflow(self, bill_date, account_type='Basic',\n tar_type=None):\n \"\"\"\n \u4e0b\u8f7d\u8d44\u91d1\u8d26\u5355\n https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=9_18&index=7\n\n :param bill_date: \u4e0b\u8f7d\u5bf9\u8d26\u5355\u7684\u65e5\u671f\n :param account_type: \u8d26\u5355\u7684\u8d44\u91d1\u6765\u6e90\u8d26\u6237\n Basic \u57fa\u672c\u8d26\u6237\n Operation \u8fd0\u8425\u8d26\u6237\n Fees \u624b\u7eed\u8d39\u8d26\u6237\n :param tar_type: \u975e\u5fc5\u4f20\u53c2\u6570\uff0c\u56fa\u5b9a\u503c\uff1aGZIP\uff0c\u8fd4\u56de\u683c\u5f0f\u4e3a.gzip\u7684\u538b\u7f29\u5305\u8d26\u5355\u3002\n \u4e0d\u4f20\u5219\u9ed8\u8ba4\u4e3a\u6570\u636e\u6d41\u5f62\u5f0f\u3002\n \"\"\"\n if isinstance(bill_date, (datetime, date)):\n bill_date = bill_date.strftime('%Y%m%d')\n\n data = {\n 'appid': self.appid,\n 'bill_date': bill_date,\n 'account_type': account_type,\n 'sign_type': 'HMAC-SHA256'\n }\n if tar_type is not None:\n data['tar_type'] = tar_type\n return self._post('pay/downloadfundflow', data=data)", "response": "\u4e0b\u8f7d\u8d44\u91d1\u8d26\u5355\n https://pay.weixin.qq.com/wiki/doc/api/jsapi.php?chapter=9_18&index=7\n\n :param bill_date: \u4e0b\u8f7d\u5bf9\u8d26\u5355\u7684\u65e5\u671f\n :param account_type: \u8d26\u5355\u7684\u8d44\u91d1\u6765\u6e90\u8d26\u6237\n Basic \u57fa\u672c\u8d26\u6237\n Operation \u8fd0\u8425\u8d26\u6237\n Fees \u624b\u7eed\u8d39\u8d26\u6237\n :param tar_type: \u975e\u5fc5\u4f20\u53c2\u6570\uff0c\u56fa\u5b9a\u503c\uff1aGZIP\uff0c\u8fd4\u56de\u683c\u5f0f\u4e3a.gzip\u7684\u538b\u7f29\u5305\u8d26\u5355\u3002\n \u4e0d\u4f20\u5219\u9ed8\u8ba4\u4e3a\u6570\u636e\u6d41\u5f62\u5f0f\u3002"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching the access token from the API", "response": "def fetch_access_token(self):\n \"\"\" Fetch access token\"\"\"\n return self._fetch_access_token(\n url='https://qyapi.weixin.qq.com/cgi-bin/gettoken',\n params={\n 'corpid': self.corp_id,\n 'corpsecret': self.secret\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_message(xml):\n if not xml:\n return\n message = xmltodict.parse(to_text(xml))['xml']\n message_type = message['MsgType'].lower()\n event_type = None\n if message_type == 'event' or message_type.startswith('device_'):\n if 'Event' in message:\n event_type = message['Event'].lower()\n # special event type for device_event\n if event_type is None and message_type.startswith('device_'):\n event_type = message_type\n elif message_type.startswith('device_'):\n event_type = 'device_{event}'.format(event=event_type)\n\n if event_type == 'subscribe' and message.get('EventKey'):\n event_key = message['EventKey']\n if event_key.startswith(('scanbarcode|', 'scanimage|')):\n event_type = 'subscribe_scan_product'\n message['Event'] = event_type\n elif event_key.startswith('qrscene_'):\n # Scan to subscribe with scene id event\n event_type = 'subscribe_scan'\n message['Event'] = event_type\n message['EventKey'] = event_key[len('qrscene_'):]\n message_class = EVENT_TYPES.get(event_type, UnknownMessage)\n else:\n message_class = MESSAGE_TYPES.get(message_type, UnknownMessage)\n return message_class(message)", "response": "Parse a message from XML."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend text message to user", "response": "def send_text(self, user_id, content, account=None):\n \"\"\"\n \u53d1\u9001\u6587\u672c\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param content: \u6d88\u606f\u6b63\u6587\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.message.send_text('openid', 'text')\n\n \"\"\"\n data = {\n 'touser': user_id,\n 'msgtype': 'text',\n 'text': {'content': content}\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending an image message to a user.", "response": "def send_image(self, user_id, media_id, account=None):\n \"\"\"\n \u53d1\u9001\u56fe\u7247\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param media_id: \u56fe\u7247\u7684\u5a92\u4f53ID\u3002 \u53ef\u4ee5\u901a\u8fc7 :func:`upload_media` \u4e0a\u4f20\u3002\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.message.send_image('openid', 'media_id')\n\n \"\"\"\n data = {\n 'touser': user_id,\n 'msgtype': 'image',\n 'image': {\n 'media_id': media_id\n }\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsends voice message to user", "response": "def send_voice(self, user_id, media_id, account=None):\n \"\"\"\n \u53d1\u9001\u8bed\u97f3\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param media_id: \u53d1\u9001\u7684\u8bed\u97f3\u7684\u5a92\u4f53ID\u3002 \u53ef\u4ee5\u901a\u8fc7 :func:`upload_media` \u4e0a\u4f20\u3002\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.message.send_voice('openid', 'media_id')\n\n \"\"\"\n data = {\n 'touser': user_id,\n 'msgtype': 'voice',\n 'voice': {\n 'media_id': media_id\n }\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend a video message to a user.", "response": "def send_video(self, user_id, media_id, title=None,\n description=None, account=None):\n \"\"\"\n \u53d1\u9001\u89c6\u9891\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param media_id: \u53d1\u9001\u7684\u89c6\u9891\u7684\u5a92\u4f53ID\u3002 \u53ef\u4ee5\u901a\u8fc7 :func:`upload_media` \u4e0a\u4f20\u3002\n :param title: \u89c6\u9891\u6d88\u606f\u7684\u6807\u9898\n :param description: \u89c6\u9891\u6d88\u606f\u7684\u63cf\u8ff0\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n res = client.message.send_video('openid', 'media_id', 'title', 'description')\n \"\"\"\n video_data = {\n 'media_id': media_id,\n }\n if title:\n video_data['title'] = title\n if description:\n video_data['description'] = description\n\n data = {\n 'touser': user_id,\n 'msgtype': 'video',\n 'video': video_data\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending music to user.", "response": "def send_music(self, user_id, url, hq_url, thumb_media_id,\n title=None, description=None, account=None):\n \"\"\"\n \u53d1\u9001\u97f3\u4e50\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param url: \u97f3\u4e50\u94fe\u63a5\n :param hq_url: \u9ad8\u54c1\u8d28\u97f3\u4e50\u94fe\u63a5\uff0cwifi\u73af\u5883\u4f18\u5148\u4f7f\u7528\u8be5\u94fe\u63a5\u64ad\u653e\u97f3\u4e50\n :param thumb_media_id: \u7f29\u7565\u56fe\u7684\u5a92\u4f53ID\u3002 \u53ef\u4ee5\u901a\u8fc7 :func:`upload_media` \u4e0a\u4f20\u3002\n :param title: \u97f3\u4e50\u6807\u9898\n :param description: \u97f3\u4e50\u63cf\u8ff0\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n music_data = {\n 'musicurl': url,\n 'hqmusicurl': hq_url,\n 'thumb_media_id': thumb_media_id\n }\n if title:\n music_data['title'] = title\n if description:\n music_data['description'] = description\n\n data = {\n 'touser': user_id,\n 'msgtype': 'music',\n 'music': music_data\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend articles to user", "response": "def send_articles(self, user_id, articles, account=None):\n \"\"\"\n \u53d1\u9001\u56fe\u6587\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/7/12a5a320ae96fecdf0e15cb06123de9f.html\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param articles: \u4e00\u4e2a\u5305\u542b\u81f3\u591a10\u4e2a\u56fe\u6587\u7684\u6570\u7ec4, \u6216\u8005\u5fae\u4fe1\u56fe\u6587\u6d88\u606f\u7d20\u6750 media_id\n :param account: \u53ef\u9009\uff0c\u5ba2\u670d\u8d26\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n if isinstance(articles, (tuple, list)):\n articles_data = []\n for article in articles:\n articles_data.append({\n 'title': article['title'],\n 'description': article['description'],\n 'url': article['url'],\n 'picurl': article.get('image', article.get('picurl')),\n })\n data = {\n 'touser': user_id,\n 'msgtype': 'news',\n 'news': {\n 'articles': articles_data\n }\n }\n else:\n data = {\n 'touser': user_id,\n 'msgtype': 'mpnews',\n 'mpnews': {\n 'media_id': articles,\n }\n }\n return self._send_custom_message(data, account=account)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send_card(self, user_id, card_id, card_ext=None, account=None):\n wxcard = {\n 'card_id': card_id,\n }\n if card_ext:\n wxcard['card_ext'] = card_ext\n data = {\n 'touser': user_id,\n 'msgtype': 'wxcard',\n 'wxcard': wxcard,\n }\n return self._send_custom_message(data, account=account)", "response": "Send a card message to a user"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send_mini_program_page(self, user_id, miniprogrampage, account=None):\n data = {\n 'touser': user_id,\n 'msgtype': 'miniprogrampage',\n 'miniprogrampage': miniprogrampage\n }\n return self._send_custom_message(data, account=account)", "response": "Send a mini program page to a user."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send_mass_text(self, group_or_users, content,\n is_to_all=False, preview=False,\n send_ignore_reprint=0, client_msg_id=None):\n \"\"\"\n \u7fa4\u53d1\u6587\u672c\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1481187827_i0l21\n\n :param group_or_users: \u503c\u4e3a\u6574\u578b\u6570\u5b57\u65f6\u4e3a\u6309\u5206\u7ec4\u7fa4\u53d1\uff0c\u503c\u4e3a\u5217\u8868/\u5143\u7ec4\u65f6\u4e3a\u6309 OpenID \u5217\u8868\u7fa4\u53d1\n \u5f53 is_to_all \u4e3a True \u65f6\uff0c\u4f20\u5165 None \u5373\u5bf9\u6240\u6709\u7528\u6237\u53d1\u9001\u3002\n :param content: \u6d88\u606f\u6b63\u6587\n :param is_to_all: \u7528\u4e8e\u8bbe\u5b9a\u662f\u5426\u5411\u5168\u90e8\u7528\u6237\u53d1\u9001\uff0c\u503c\u4e3atrue\u6216false\uff0c\u9009\u62e9true\u8be5\u6d88\u606f\u7fa4\u53d1\u7ed9\u6240\u6709\u7528\u6237\n \u9009\u62e9false\u53ef\u6839\u636egroup_id\u53d1\u9001\u7ed9\u6307\u5b9a\u7fa4\u7ec4\u7684\u7528\u6237\n :type is_to_all: bool\n :param preview: \u662f\u5426\u53d1\u9001\u9884\u89c8\uff0c\u6b64\u65f6 group_or_users \u53c2\u6570\u5e94\u4e3a\u4e00\u4e2aopenid\u5b57\u7b26\u4e32\n :type preview: bool\n :param send_ignore_reprint: \u6307\u5b9a\u5f85\u7fa4\u53d1\u7684\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u662f\u5426\u7ee7\u7eed\u7fa4\u53d1\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a1\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u4e14\u539f\u521b\u6587\u5141\u8bb8\u8f6c\u8f7d\u65f6\uff0c\u5c06\u7ee7\u7eed\u8fdb\u884c\u7fa4\u53d1\u64cd\u4f5c\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a0\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u5c06\u505c\u6b62\u7fa4\u53d1\u64cd\u4f5c\u3002\n send_ignore_reprint \u9ed8\u8ba4\u4e3a0\u3002\n :type send_ignore_reprint: int\n :param client_msg_id: \u5f00\u53d1\u8005\u4fa7\u7fa4\u53d1 msgid\uff0c\u957f\u5ea6\u9650\u5236 64 \u5b57\u8282\n :type client_msg_id: str\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._send_mass_message(\n group_or_users,\n 'text',\n {\n 'text': {\n 'content': content\n }\n },\n is_to_all,\n preview,\n send_ignore_reprint,\n client_msg_id,\n )", "response": "Send mass text to a group or users."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_mass_video(self, group_or_users, media_id, title=None,\n description=None, is_to_all=False, preview=False,\n send_ignore_reprint=0, client_msg_id=None):\n \"\"\"\n \u7fa4\u53d1\u89c6\u9891\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1481187827_i0l21\n\n :param group_or_users: \u503c\u4e3a\u6574\u578b\u6570\u5b57\u65f6\u4e3a\u6309\u5206\u7ec4\u7fa4\u53d1\uff0c\u503c\u4e3a\u5217\u8868/\u5143\u7ec4\u65f6\u4e3a\u6309 OpenID \u5217\u8868\u7fa4\u53d1\n \u5f53 is_to_all \u4e3a True \u65f6\uff0c\u4f20\u5165 None \u5373\u5bf9\u6240\u6709\u7528\u6237\u53d1\u9001\u3002\n :param media_id: \u89c6\u9891\u7684\u5a92\u4f53 ID\u3002\u53ef\u4ee5\u901a\u8fc7 :func:`upload_video` \u4e0a\u4f20\u3002\n :param title: \u89c6\u9891\u6807\u9898\n :param description: \u89c6\u9891\u63cf\u8ff0\n :param is_to_all: \u7528\u4e8e\u8bbe\u5b9a\u662f\u5426\u5411\u5168\u90e8\u7528\u6237\u53d1\u9001\uff0c\u503c\u4e3atrue\u6216false\uff0c\u9009\u62e9true\u8be5\u6d88\u606f\u7fa4\u53d1\u7ed9\u6240\u6709\u7528\u6237\n \u9009\u62e9false\u53ef\u6839\u636egroup_id\u53d1\u9001\u7ed9\u6307\u5b9a\u7fa4\u7ec4\u7684\u7528\u6237\n :type is_to_all: bool\n :param preview: \u662f\u5426\u53d1\u9001\u9884\u89c8\uff0c\u6b64\u65f6 group_or_users \u53c2\u6570\u5e94\u4e3a\u4e00\u4e2aopenid\u5b57\u7b26\u4e32\n :type preview: bool\n :param send_ignore_reprint: \u6307\u5b9a\u5f85\u7fa4\u53d1\u7684\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u662f\u5426\u7ee7\u7eed\u7fa4\u53d1\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a1\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u4e14\u539f\u521b\u6587\u5141\u8bb8\u8f6c\u8f7d\u65f6\uff0c\u5c06\u7ee7\u7eed\u8fdb\u884c\u7fa4\u53d1\u64cd\u4f5c\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a0\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u5c06\u505c\u6b62\u7fa4\u53d1\u64cd\u4f5c\u3002\n send_ignore_reprint \u9ed8\u8ba4\u4e3a0\u3002\n :type send_ignore_reprint: int\n :param client_msg_id: \u5f00\u53d1\u8005\u4fa7\u7fa4\u53d1 msgid\uff0c\u957f\u5ea6\u9650\u5236 64 \u5b57\u8282\n :type client_msg_id: str\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n video_data = {\n 'media_id': media_id\n }\n if title:\n video_data['title'] = title\n if description:\n video_data['description'] = description\n return self._send_mass_message(\n group_or_users,\n 'mpvideo',\n {\n 'mpvideo': video_data\n },\n is_to_all,\n preview,\n send_ignore_reprint,\n client_msg_id,\n )", "response": "Send a mass video to a group."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef send_mass_article(self, group_or_users, media_id,\n is_to_all=False, preview=False,\n send_ignore_reprint=0, client_msg_id=None):\n \"\"\"\n \u7fa4\u53d1\u56fe\u6587\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1481187827_i0l21\n\n :param group_or_users: \u503c\u4e3a\u6574\u578b\u6570\u5b57\u65f6\u4e3a\u6309\u5206\u7ec4\u7fa4\u53d1\uff0c\u503c\u4e3a\u5217\u8868/\u5143\u7ec4\u65f6\u4e3a\u6309 OpenID \u5217\u8868\u7fa4\u53d1\n \u5f53 is_to_all \u4e3a True \u65f6\uff0c\u4f20\u5165 None \u5373\u5bf9\u6240\u6709\u7528\u6237\u53d1\u9001\u3002\n :param media_id: \u56fe\u6587\u7684\u5a92\u4f53 ID\u3002\u53ef\u4ee5\u901a\u8fc7 :func:`upload_articles` \u4e0a\u4f20\u3002\n :param is_to_all: \u7528\u4e8e\u8bbe\u5b9a\u662f\u5426\u5411\u5168\u90e8\u7528\u6237\u53d1\u9001\uff0c\u503c\u4e3atrue\u6216false\uff0c\u9009\u62e9true\u8be5\u6d88\u606f\u7fa4\u53d1\u7ed9\u6240\u6709\u7528\u6237\n \u9009\u62e9false\u53ef\u6839\u636egroup_id\u53d1\u9001\u7ed9\u6307\u5b9a\u7fa4\u7ec4\u7684\u7528\u6237\n :type is_to_all: bool\n :param preview: \u662f\u5426\u53d1\u9001\u9884\u89c8\uff0c\u6b64\u65f6 group_or_users \u53c2\u6570\u5e94\u4e3a\u4e00\u4e2aopenid\u5b57\u7b26\u4e32\n :type preview: bool\n :param send_ignore_reprint: \u6307\u5b9a\u5f85\u7fa4\u53d1\u7684\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u662f\u5426\u7ee7\u7eed\u7fa4\u53d1\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a1\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u4e14\u539f\u521b\u6587\u5141\u8bb8\u8f6c\u8f7d\u65f6\uff0c\u5c06\u7ee7\u7eed\u8fdb\u884c\u7fa4\u53d1\u64cd\u4f5c\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a0\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u5c06\u505c\u6b62\u7fa4\u53d1\u64cd\u4f5c\u3002\n send_ignore_reprint \u9ed8\u8ba4\u4e3a0\u3002\n :type send_ignore_reprint: int\n :param client_msg_id: \u5f00\u53d1\u8005\u4fa7\u7fa4\u53d1 msgid\uff0c\u957f\u5ea6\u9650\u5236 64 \u5b57\u8282\n :type client_msg_id: str\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._send_mass_message(\n group_or_users,\n 'mpnews',\n {\n 'mpnews': {\n 'media_id': media_id\n }\n },\n is_to_all,\n preview,\n send_ignore_reprint,\n client_msg_id,\n )", "response": "Send a mass article to the server."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a template to a user", "response": "def send_template(self, user_id, template_id, data, url=None, mini_program=None):\n \"\"\"\n \u53d1\u9001\u6a21\u677f\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1445241432&lang=zh_CN\n\n :param user_id: \u7528\u6237 ID \u3002 \u5c31\u662f\u4f60\u6536\u5230\u7684 `Message` \u7684 source\n :param template_id: \u6a21\u677f ID\u3002\u5728\u516c\u4f17\u5e73\u53f0\u7ebf\u4e0a\u6a21\u677f\u5e93\u4e2d\u9009\u7528\u6a21\u677f\u83b7\u5f97\n :param url: \u94fe\u63a5\u5730\u5740\n :param data: \u6a21\u677f\u6d88\u606f\u6570\u636e\n :param mini_program: \u8df3\u5c0f\u7a0b\u5e8f\u6240\u9700\u6570\u636e, \u5982\uff1a`{'appid': 'appid', 'pagepath': 'index?foo=bar'}`\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n tpl_data = optionaldict(\n touser=user_id,\n template_id=template_id,\n url=url,\n miniprogram=mini_program,\n data=data,\n )\n return self._post(\n 'message/template/send',\n data=tpl_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_mass_card(self, group_or_users, card_id,\n is_to_all=False, preview=False,\n send_ignore_reprint=0, client_msg_id=None):\n \"\"\"\n \u7fa4\u53d1\u5361\u5238\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1481187827_i0l21\n\n :param group_or_users: \u503c\u4e3a\u6574\u578b\u6570\u5b57\u65f6\u4e3a\u6309\u5206\u7ec4\u7fa4\u53d1\uff0c\u503c\u4e3a\u5217\u8868/\u5143\u7ec4\u65f6\u4e3a\u6309 OpenID \u5217\u8868\u7fa4\u53d1\n \u5f53 is_to_all \u4e3a True \u65f6\uff0c\u4f20\u5165 None \u5373\u5bf9\u6240\u6709\u7528\u6237\u53d1\u9001\u3002\n :param card_id: \u5361\u5238 ID\n :param is_to_all: \u7528\u4e8e\u8bbe\u5b9a\u662f\u5426\u5411\u5168\u90e8\u7528\u6237\u53d1\u9001\uff0c\u503c\u4e3atrue\u6216false\uff0c\u9009\u62e9true\u8be5\u6d88\u606f\u7fa4\u53d1\u7ed9\u6240\u6709\u7528\u6237\n \u9009\u62e9false\u53ef\u6839\u636egroup_id\u53d1\u9001\u7ed9\u6307\u5b9a\u7fa4\u7ec4\u7684\u7528\u6237\n :type is_to_all: bool\n :param preview: \u662f\u5426\u53d1\u9001\u9884\u89c8\uff0c\u6b64\u65f6 group_or_users \u53c2\u6570\u5e94\u4e3a\u4e00\u4e2aopenid\u5b57\u7b26\u4e32\n :type preview: bool\n :param send_ignore_reprint: \u6307\u5b9a\u5f85\u7fa4\u53d1\u7684\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u662f\u5426\u7ee7\u7eed\u7fa4\u53d1\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a1\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u4e14\u539f\u521b\u6587\u5141\u8bb8\u8f6c\u8f7d\u65f6\uff0c\u5c06\u7ee7\u7eed\u8fdb\u884c\u7fa4\u53d1\u64cd\u4f5c\u3002\n \u5f53 send_ignore_reprint \u53c2\u6570\u8bbe\u7f6e\u4e3a0\u65f6\uff0c\u6587\u7ae0\u88ab\u5224\u5b9a\u4e3a\u8f6c\u8f7d\u65f6\uff0c\u5c06\u505c\u6b62\u7fa4\u53d1\u64cd\u4f5c\u3002\n send_ignore_reprint \u9ed8\u8ba4\u4e3a0\u3002\n :type send_ignore_reprint: int\n :param client_msg_id: \u5f00\u53d1\u8005\u4fa7\u7fa4\u53d1 msgid\uff0c\u957f\u5ea6\u9650\u5236 64 \u5b57\u8282\n :type client_msg_id: str\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._send_mass_message(\n group_or_users,\n 'wxcard',\n {\n 'wxcard': {\n 'card_id': card_id\n }\n },\n is_to_all,\n preview,\n send_ignore_reprint,\n client_msg_id,\n )", "response": "Send a mass card to a group or users."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsends a subscribe template to the user.", "response": "def send_subscribe_template(self, openid, template_id, scene, title, data, url=None):\n \"\"\"\n \u4e00\u6b21\u6027\u8ba2\u9605\u6d88\u606f\uff0c\u901a\u8fc7API\u63a8\u9001\u8ba2\u9605\u6a21\u677f\u6d88\u606f\u7ed9\u5230\u6388\u6743\u5fae\u4fe1\u7528\u6237\u3002\n \u8be6\u60c5\u8bf7\u53c2\u9605\uff1a\n https://mp.weixin.qq.com/wiki?id=mp1500374289_66bvB\n\n :param openid: \u586b\u63a5\u6536\u6d88\u606f\u7684\u7528\u6237openid\n :param template_id: \u8ba2\u9605\u6d88\u606f\u6a21\u677fID\n :param scene: \u8ba2\u9605\u573a\u666f\u503c\uff0c\u5f00\u53d1\u8005\u53ef\u4ee5\u586b0-10000\u7684\u6574\u5f62\u503c\uff0c\u7528\u6765\u6807\u8bc6\u8ba2\u9605\u573a\u666f\u503c\n :type scene: int\n :param title: \u6d88\u606f\u6807\u9898\uff0c15\u5b57\u4ee5\u5185\n :param data: \u6d88\u606f\u6b63\u6587\uff0cvalue\u4e3a\u6d88\u606f\u5185\u5bb9\uff0ccolor\u4e3a\u989c\u8272\uff0c200\u5b57\u4ee5\u5185\n :type data: dict\n :param url: \u70b9\u51fb\u6d88\u606f\u8df3\u8f6c\u7684\u94fe\u63a5\uff0c\u9700\u8981\u6709ICP\u5907\u6848\n \"\"\"\n post_data = {\n 'touser': openid,\n 'template_id': template_id,\n 'url': url,\n 'scene': scene,\n 'title': title,\n 'data': data,\n }\n if url is not None:\n post_data['url'] = url\n return self._post(\n 'message/template/subscribe',\n data=post_data,\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding an account to the Kfaccount database.", "response": "def add_account(self, account, nickname, password):\n \"\"\"\n \u6dfb\u52a0\u5ba2\u670d\u8d26\u53f7\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/1/70a29afed17f56d537c833f89be979c9.html\n\n :param account: \u5b8c\u6574\u5ba2\u670d\u8d26\u53f7\uff0c\u683c\u5f0f\u4e3a\uff1a\u8d26\u53f7\u524d\u7f00@\u516c\u4f17\u53f7\u5fae\u4fe1\u53f7\n :param nickname: \u5ba2\u670d\u6635\u79f0\uff0c\u6700\u957f6\u4e2a\u6c49\u5b57\u621612\u4e2a\u82f1\u6587\u5b57\u7b26\n :param password: \u5ba2\u670d\u8d26\u53f7\u767b\u5f55\u5bc6\u7801\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n password = to_binary(password)\n password = hashlib.md5(password).hexdigest()\n return self._post(\n 'https://api.weixin.qq.com/customservice/kfaccount/add',\n data={\n 'kf_account': account,\n 'nickname': nickname,\n 'password': password\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting a user s Kfaccount.", "response": "def delete_account(self, account):\n \"\"\"\n \u5220\u9664\u5ba2\u670d\u8d26\u53f7\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/1/70a29afed17f56d537c833f89be979c9.html\n\n :param account: \u5b8c\u6574\u5ba2\u670d\u8d26\u53f7\uff0c\u683c\u5f0f\u4e3a\uff1a\u8d26\u53f7\u524d\u7f00@\u516c\u4f17\u53f7\u5fae\u4fe1\u53f7\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n params_data = [\n 'access_token={0}'.format(quote(self.access_token)),\n 'kf_account={0}'.format(quote(to_binary(account), safe=b'/@')),\n ]\n params = '&'.join(params_data)\n return self._get(\n 'https://api.weixin.qq.com/customservice/kfaccount/del',\n params=params\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef upload_headimg(self, account, media_file):\n return self._post(\n 'https://api.weixin.qq.com/customservice/kfaccount/uploadheadimg',\n params={\n 'kf_account': account\n },\n files={\n 'media': media_file\n }\n )", "response": "Upload a headimg to the specified account."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_session(self, openid, account, text=None):\n data = optionaldict(\n openid=openid,\n kf_account=account,\n text=text\n )\n return self._post(\n 'https://api.weixin.qq.com/customservice/kfsession/create',\n data=data\n )", "response": "\u591a\u5ba2\u670d\u521b\u5efa\u4f1a\u8bdd\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/2/6c20f3e323bdf5986cfcb33cbd3b829a.html\n\n :param openid: \u5ba2\u6237 openid\n :param account: \u5b8c\u6574\u5ba2\u670d\u8d26\u53f7\n :param text: \u9644\u52a0\u4fe1\u606f\uff0c\u53ef\u9009\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_session_list(self, account):\n res = self._get(\n 'https://api.weixin.qq.com/customservice/kfsession/getsessionlist',\n params={'kf_account': account},\n result_processor=lambda x: x['sessionlist']\n )\n return res", "response": "\u83b7\u53d6\u5ba2\u670d\u7684\u4f1a\u8bdd\u5217\u8868\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/2/6c20f3e323bdf5986cfcb33cbd3b829a.html\n\n :param account: \u5b8c\u6574\u5ba2\u670d\u8d26\u53f7\n :return: \u5ba2\u670d\u7684\u4f1a\u8bdd\u5217\u8868"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_records(self, start_time, end_time, msgid=1,\n number=10000):\n \"\"\"\n \u83b7\u53d6\u5ba2\u670d\u804a\u5929\u8bb0\u5f55\n\n :param start_time: \u67e5\u8be2\u5f00\u59cb\u65f6\u95f4\uff0cUNIX \u65f6\u95f4\u6233\n :param end_time: \u67e5\u8be2\u7ed3\u675f\u65f6\u95f4\uff0cUNIX \u65f6\u95f4\u6233\uff0c\u6bcf\u6b21\u67e5\u8be2\u4e0d\u80fd\u8de8\u65e5\u67e5\u8be2\n :param msgid: \u6d88\u606fid\u987a\u5e8f\u4ece\u5c0f\u5230\u5927\uff0c\u4ece1\u5f00\u59cb\n :param number: \u6bcf\u6b21\u83b7\u53d6\u6761\u6570\uff0c\u6700\u591a10000\u6761\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n if isinstance(start_time, datetime.datetime):\n start_time = time.mktime(start_time.timetuple())\n if isinstance(end_time, datetime.datetime):\n end_time = time.mktime(end_time.timetuple())\n record_data = {\n 'starttime': int(start_time),\n 'endtime': int(end_time),\n 'msgid': msgid,\n 'number': number\n }\n res = self._post(\n 'https://api.weixin.qq.com/customservice/msgrecord/getmsglist',\n data=record_data,\n )\n return res", "response": "Get records from a user s custom service."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_reply(reply, message=None, render=False):\n r = None\n if not reply:\n r = EmptyReply()\n elif isinstance(reply, BaseReply):\n r = reply\n if message:\n r.source = message.target\n r.target = message.source\n elif isinstance(reply, six.string_types):\n r = TextReply(\n message=message,\n content=reply\n )\n elif isinstance(reply, (tuple, list)):\n if len(reply) > 10:\n raise AttributeError(\"Can't add more than 10 articles\"\n \" in an ArticlesReply\")\n r = ArticlesReply(\n message=message,\n articles=reply\n )\n if r and render:\n return r.render()\n return r", "response": "Create a reply quickly\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef deserialize_reply(xml, update_time=False):\n if not xml:\n return EmptyReply()\n\n try:\n reply_dict = xmltodict.parse(xml)[\"xml\"]\n msg_type = reply_dict[\"MsgType\"]\n except (xmltodict.expat.ExpatError, KeyError):\n raise ValueError(\"bad reply xml\")\n if msg_type not in REPLY_TYPES:\n raise ValueError(\"unknown reply type\")\n\n cls = REPLY_TYPES[msg_type]\n kwargs = dict()\n for attr, field in cls._fields.items():\n if field.name in reply_dict:\n str_value = reply_dict[field.name]\n kwargs[attr] = field.from_xml(str_value)\n\n if update_time:\n kwargs[\"time\"] = time.time()\n\n return cls(**kwargs)", "response": "Deserialize reply xml into a new object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef render(self):\n tpl = '\\n{data}\\n'\n nodes = []\n msg_type = ''.format(\n msg_type=self.type\n )\n nodes.append(msg_type)\n for name, field in self._fields.items():\n value = getattr(self, name, field.default)\n node_xml = field.to_xml(value)\n nodes.append(node_xml)\n data = '\\n'.join(nodes)\n return tpl.format(data=data)", "response": "Render reply from Python object to XML string"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fetch_access_token(self):\n url = '{0}{1}'.format(\n self.API_BASE_URL,\n '/component/api_component_token'\n )\n return self._fetch_access_token(\n url=url,\n data=json.dumps({\n 'component_appid': self.component_appid,\n 'component_appsecret': self.component_appsecret,\n 'component_verify_ticket': self.component_verify_ticket\n })\n )", "response": "Get component access token"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _fetch_access_token(self, url, data):\n logger.info('Fetching component access token')\n res = self._http.post(\n url=url,\n data=data\n )\n try:\n res.raise_for_status()\n except requests.RequestException as reqe:\n raise WeChatClientException(\n errcode=None,\n errmsg=None,\n client=self,\n request=reqe.request,\n response=reqe.response\n )\n result = res.json()\n if 'errcode' in result and result['errcode'] != 0:\n raise WeChatClientException(\n result['errcode'],\n result['errmsg'],\n client=self,\n request=res.request,\n response=res\n )\n\n expires_in = 7200\n if 'expires_in' in result:\n expires_in = result['expires_in']\n self.session.set(\n 'component_access_token',\n result['component_access_token'],\n expires_in\n )\n self.expires_at = int(time.time()) + expires_in\n return result", "response": "Fetch the component access token from the server"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_pre_auth_url_m(self, redirect_uri):\n url = \"https://mp.weixin.qq.com/safe/bindcomponent?action=bindcomponent&auth_type=3&no_scan=1&\"\n redirect_uri = quote(redirect_uri, safe='')\n return \"{0}component_appid={1}&pre_auth_code={2}&redirect_uri={3}\".format(\n url, self.component_appid, self.create_preauthcode()['pre_auth_code'], redirect_uri\n )", "response": "get pre auth url"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrefresh the authorizer token for the specified user.", "response": "def refresh_authorizer_token(\n self, authorizer_appid, authorizer_refresh_token):\n \"\"\"\n \u83b7\u53d6\uff08\u5237\u65b0\uff09\u6388\u6743\u516c\u4f17\u53f7\u7684\u4ee4\u724c\n\n :params authorizer_appid: \u6388\u6743\u65b9appid\n :params authorizer_refresh_token: \u6388\u6743\u65b9\u7684\u5237\u65b0\u4ee4\u724c\n \"\"\"\n return self.post(\n '/component/api_authorizer_token',\n data={\n 'component_appid': self.component_appid,\n 'authorizer_appid': authorizer_appid,\n 'authorizer_refresh_token': authorizer_refresh_token\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_authorizer_option(self, authorizer_appid, option_name):\n return self.post(\n '/component/api_get_authorizer_option',\n data={\n 'component_appid': self.component_appid,\n 'authorizer_appid': authorizer_appid,\n 'option_name': option_name\n }\n )", "response": "This method returns the authorizer option for a given option name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_client_by_authorization_code(self, authorization_code):\n warnings.warn('`get_client_by_authorization_code` method of `WeChatComponent` is deprecated,'\n 'Use `parse_message` parse message and '\n 'Use `get_client_by_appid` instead',\n DeprecationWarning, stacklevel=2)\n result = self.query_auth(authorization_code)\n access_token = result['authorization_info']['authorizer_access_token']\n refresh_token = result['authorization_info']['authorizer_refresh_token'] # NOQA\n authorizer_appid = result['authorization_info']['authorizer_appid'] # noqa\n return WeChatComponentClient(\n authorizer_appid, self, access_token, refresh_token,\n session=self.session\n )", "response": "Get a client by authorization code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_client_by_appid(self, authorizer_appid):\n access_token_key = '{0}_access_token'.format(authorizer_appid)\n refresh_token_key = '{0}_refresh_token'.format(authorizer_appid)\n access_token = self.session.get(access_token_key)\n refresh_token = self.session.get(refresh_token_key)\n assert refresh_token\n\n if not access_token:\n ret = self.refresh_authorizer_token(\n authorizer_appid,\n refresh_token\n )\n access_token = ret['authorizer_access_token']\n refresh_token = ret['authorizer_refresh_token']\n access_token_key = '{0}_access_token'.format(authorizer_appid)\n expires_in = 7200\n if 'expires_in' in ret:\n expires_in = ret['expires_in']\n self.session.set(access_token_key, access_token, expires_in)\n\n return WeChatComponentClient(\n authorizer_appid,\n self,\n session=self.session\n )", "response": "get_client_by_appid - Gets a client by the appid"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing message and return ComponentUnknownMessage", "response": "def parse_message(self, msg, msg_signature, timestamp, nonce):\n \"\"\"\n \u5904\u7406 wechat server \u63a8\u9001\u6d88\u606f\n\n :params msg: \u52a0\u5bc6\u5185\u5bb9\n :params msg_signature: \u6d88\u606f\u7b7e\u540d\n :params timestamp: \u65f6\u95f4\u6233\n :params nonce: \u968f\u673a\u6570\n \"\"\"\n content = self.crypto.decrypt_message(msg, msg_signature, timestamp, nonce)\n message = xmltodict.parse(to_text(content))['xml']\n message_type = message['InfoType'].lower()\n message_class = COMPONENT_MESSAGE_TYPES.get(message_type, ComponentUnknownMessage)\n msg = message_class(message)\n if msg.type == 'component_verify_ticket':\n self.session.set(msg.type, msg.verify_ticket)\n elif msg.type in ('authorized', 'updateauthorized'):\n msg.query_auth_result = self.query_auth(msg.authorization_code)\n return msg"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cache_component_verify_ticket(self, msg, signature, timestamp, nonce):\n warnings.warn('`cache_component_verify_ticket` method of `WeChatComponent` is deprecated,'\n 'Use `parse_message` instead',\n DeprecationWarning, stacklevel=2)\n content = self.crypto.decrypt_message(msg, signature, timestamp, nonce)\n message = xmltodict.parse(to_text(content))['xml']\n o = ComponentVerifyTicketMessage(message)\n self.session.set(o.type, o.verify_ticket)", "response": "This method is used to cache component verify ticket."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_unauthorized(self, msg, signature, timestamp, nonce):\n warnings.warn('`get_unauthorized` method of `WeChatComponent` is deprecated,'\n 'Use `parse_message` instead',\n DeprecationWarning, stacklevel=2)\n content = self.crypto.decrypt_message(msg, signature, timestamp, nonce)\n message = xmltodict.parse(to_text(content))['xml']\n return ComponentUnauthorizedMessage(message)", "response": "This method is used to decrypt a message and return a ComponentUnauthorizedMessage object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fetch_access_token(self, code):\n res = self._get(\n 'sns/oauth2/component/access_token',\n params={\n 'appid': self.app_id,\n 'component_appid': self.component.component_appid,\n 'component_access_token': self.component.access_token,\n 'code': code,\n 'grant_type': 'authorization_code',\n }\n )\n self.access_token = res['access_token']\n self.open_id = res['openid']\n self.refresh_token = res['refresh_token']\n self.expires_in = res['expires_in']\n self.scope = res['scope']\n return res", "response": "fetch_access_token - Fetches access_token from ns_oauth2_component_access_token URL"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the URL to authorize the user.", "response": "def authorize_url(self):\n \"\"\"\u83b7\u53d6\u6388\u6743\u8df3\u8f6c\u5730\u5740\n\n :return: URL \u5730\u5740\n \"\"\"\n redirect_uri = quote(self.redirect_uri, safe=b'')\n url_list = [\n self.OAUTH_BASE_URL,\n 'oauth2/authorize?appid=',\n self.app_id,\n '&redirect_uri=',\n redirect_uri,\n '&response_type=code&scope=',\n self.scope\n ]\n if self.state:\n url_list.extend(['&state=', self.state])\n url_list.append('#wechat_redirect')\n return ''.join(url_list)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_access_token(self, openid=None, access_token=None):\n openid = openid or self.open_id\n access_token = access_token or self.access_token\n res = self._get(\n 'sns/auth',\n params={\n 'access_token': access_token,\n 'openid': openid\n }\n )\n if res['errcode'] == 0:\n return True\n return False", "response": "Check if an access token is valid."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a new entry to the cache", "response": "def add(self, agent_id, media_type, media_file):\n \"\"\"\n \u65b0\u589e\u5176\u5b83\u7c7b\u578b\u6c38\u4e45\u7d20\u6750\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=%E4%B8%8A%E4%BC%A0%E6%B0%B8%E4%B9%85%E7%B4%A0%E6%9D%90\n\n :param agent_id: \u4f01\u4e1a\u5e94\u7528\u7684id\n :param media_type: \u5a92\u4f53\u6587\u4ef6\u7c7b\u578b\uff0c\u5206\u522b\u6709\u56fe\u7247\uff08image\uff09\u3001\u8bed\u97f3\uff08voice\uff09\u3001\u89c6\u9891\uff08video\uff09\u666e\u901a\u6587\u4ef6\uff08file\uff09\n :param media_file: \u8981\u4e0a\u4f20\u7684\u6587\u4ef6\uff0c\u4e00\u4e2a File-object\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n params = {\n 'agentid': agent_id,\n 'type': media_type,\n }\n return self._post(\n url='material/add_material',\n params=params,\n files={\n 'media': media_file\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets url for a specific ID", "response": "def get_url(self, agent_id, media_id):\n \"\"\"\n \u83b7\u53d6\u6c38\u4e45\u7d20\u6750\u4e0b\u8f7d\u5730\u5740\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=%E8%8E%B7%E5%8F%96%E6%B0%B8%E4%B9%85%E7%B4%A0%E6%9D%90\n\n :param agent_id: \u4f01\u4e1a\u5e94\u7528\u7684id\n :param media_id: \u5a92\u4f53\u6587\u4ef6 ID\n :return: \u4e34\u65f6\u7d20\u6750\u4e0b\u8f7d\u5730\u5740\n \"\"\"\n parts = (\n 'https://qyapi.weixin.qq.com/cgi-bin/material/get',\n '?access_token=',\n self.access_token,\n '&media_id=',\n media_id,\n '&agentid=',\n agent_id,\n )\n return ''.join(parts)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get(self, agent_id, media_id):\n return requests.get(self.get_url(agent_id, media_id))", "response": "Get the ID of the current object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_articles(self, agent_id, media_id, articles):\n articles_data = []\n for article in articles:\n articles_data.append({\n 'thumb_media_id': article['thumb_media_id'],\n 'title': article['title'],\n 'content': article['content'],\n 'author': article.get('author', ''),\n 'content_source_url': article.get('content_source_url', ''),\n 'digest': article.get('digest', ''),\n 'show_cover_pic': article.get('show_cover_pic', 0)\n })\n return self._post(\n 'material/update_news',\n data={\n 'agentid': agent_id,\n 'media_id': media_id,\n 'articles': articles_data\n }\n )", "response": "Update articles in a media."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbatches get a set of items from the specified agent.", "response": "def batchget(self, agent_id, media_type, offset=0, count=20):\n \"\"\"\n \u6279\u91cf\u83b7\u53d6\u6c38\u4e45\u7d20\u6750\u5217\u8868\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=%E8%8E%B7%E5%8F%96%E7%B4%A0%E6%9D%90%E5%88%97%E8%A1%A8\n\n :param agent_id: \u4f01\u4e1a\u5e94\u7528\u7684id\n :param media_type: \u5a92\u4f53\u6587\u4ef6\u7c7b\u578b\uff0c\u5206\u522b\u6709\u56fe\u6587\uff08mpnews\uff09\u3001\u56fe\u7247\uff08image\uff09\u3001\n \u8bed\u97f3\uff08voice\uff09\u3001\u89c6\u9891\uff08video\uff09\u548c\u6587\u4ef6\uff08file\uff09\n :param offset: \u4ece\u5168\u90e8\u7d20\u6750\u7684\u8be5\u504f\u79fb\u4f4d\u7f6e\u5f00\u59cb\u8fd4\u56de\uff0c0 \u8868\u793a\u4ece\u7b2c\u4e00\u4e2a\u7d20\u6750\u8fd4\u56de\n :param count: \u8fd4\u56de\u7d20\u6750\u7684\u6570\u91cf\uff0c\u53d6\u503c\u57281\u523020\u4e4b\u95f4\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._post(\n 'material/batchget',\n data={\n 'agent_id': agent_id,\n 'type': media_type,\n 'offset': offset,\n 'count': count\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new item in the cache.", "response": "def create(self, agent_id, menu_data):\n \"\"\"\n \u521b\u5efa\u83dc\u5355\n\n https://work.weixin.qq.com/api/doc#90000/90135/90231\n\n :param agent_id: \u5e94\u7528id\n \"\"\"\n return self._post(\n 'menu/create',\n params={\n 'agentid': agent_id\n },\n data=menu_data\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget jsapi card params", "response": "def get_jsapi_card_params(self, card_ticket, card_type, **kwargs):\n \"\"\"\n \u53c2\u6570\u610f\u4e49\u89c1\u5fae\u4fe1\u6587\u6863\u5730\u5740\uff1ahttp://mp.weixin.qq.com/wiki/7/aaa137b55fb2e0456bf8dd9148dd613f.html\n :param card_ticket: \u7528\u4e8e\u5361\u5238\u7684\u5fae\u4fe1 api_ticket\n :param card_type: \n :param kwargs: \u975e\u5fc5\u987b\u53c2\u6570\uff1anoncestr, timestamp, code, openid, fixed_begintimestamp, outer_str\n :return: \u5305\u542b\u8c03\u7528jssdk\u6240\u6709\u6240\u9700\u53c2\u6570\u7684 dict\n \"\"\"\n card_signature_dict = {\n 'card_type': card_type,\n 'noncestr': kwargs.get('noncestr', random_string()),\n 'api_ticket': card_ticket,\n 'appid': self.appid,\n 'timestamp': kwargs.get('timestamp', str(int(time.time()))),\n }\n list_before_sign = sorted([str(x) for x in card_signature_dict.values()])\n str_to_sign = \"\".join(list_before_sign).encode()\n card_signature_dict['sign'] = hashlib.sha1(str_to_sign).hexdigest()\n return card_signature_dict"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new card.", "response": "def create_card(self, base_info, payee, invoice_type, detail=None):\n \"\"\"\n \u521b\u5efa\u53d1\u7968\u5361\u5238\u6a21\u677f\n \u6ce8\u610f\u8fd9\u91cc\u7684\u5bf9\u8c61\u548c\u4f1a\u5458\u5361\u6709\u7c7b\u4f3c\u4e4b\u5904\uff0c\u4f46\u662f\u542b\u4e49\u6709\u4e0d\u540c\u3002\u521b\u5efa\u53d1\u7968\u5361\u5238\u6a21\u677f\u662f\u521b\u5efa\u53d1\u7968\u5361\u5238\u7684\u57fa\u7840\u3002\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1496561481_1TyO7\n\n :param base_info:\u53d1\u7968\u5361\u5238\u6a21\u677f\u57fa\u7840\u4fe1\u606f\n :type base_info: dict\n :param payee: \u6536\u6b3e\u65b9\uff08\u5f00\u7968\u65b9\uff09\u5168\u79f0\uff0c\u663e\u793a\u5728\u53d1\u7968\u8be6\u60c5\u5185\u3002\u5efa\u8bae\u4e00\u4e2a\u6536\u6b3e\u65b9\u5bf9\u5e94\u4e00\u4e2a\u53d1\u7968\u5361\u5238\u6a21\u677f\n :param invoice_type: \u53d1\u7968\u7c7b\u578b\u63cf\u8ff0\n :param detail: \u5907\u6ce8\u8be6\u60c5\n :return: \u53d1\u7968\u5361\u5238\u6a21\u677f\u7684\u7f16\u53f7\uff0c\u7528\u4e8e\u540e\u7eed\u8be5\u5546\u6237\u53d1\u7968\u751f\u6210\u540e\uff0c\u4f5c\u4e3a\u5fc5\u586b\u53c2\u6570\u5728\u8c03\u7528\u63d2\u5361\u63a5\u53e3\u65f6\u4f20\u5165\n \"\"\"\n return self._post(\n 'platform/createcard',\n data={\n 'invoice_info': {\n 'base_info': base_info,\n 'payee': payee,\n 'type': invoice_type,\n 'detail': detail,\n },\n },\n result_processor=lambda x: x['card_id'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the auth field of an user.", "response": "def set_auth_field(self, user_field, biz_field):\n \"\"\"\n \u8bbe\u7f6e\u6388\u6743\u9875\u5b57\u6bb5\u4fe1\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1497082828_r1cI2\n\n :param user_field: \u6388\u6743\u9875\u4e2a\u4eba\u53d1\u7968\u5b57\u6bb5\n :type user_field: dict\n :param biz_field: \u6388\u6743\u9875\u5355\u4f4d\u53d1\u7968\u5b57\u6bb5\n :type biz_field: dict\n \"\"\"\n return self._post(\n 'setbizattr',\n params={\n 'action': 'set_auth_field',\n },\n data={\n 'auth_field': {\n 'user_field': user_field,\n 'biz_field': biz_field,\n },\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reject_insert(self, s_pappid, order_id, reason, redirect_url=None):\n return self._post(\n 'rejectinsert',\n data={\n 's_pappid': s_pappid,\n 'order_id': order_id,\n 'reason': reason,\n 'url': redirect_url,\n },\n )", "response": "\u62d2\u7edd\u7528\u6237\u7684\u5f00\u53d1\u7968\u8bf7\u6c42\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1497082828_r1cI2\n\n :param s_pappid: \u5f00\u7968\u5e73\u53f0\u5728\u5fae\u4fe1\u7684\u6807\u8bc6\u53f7\uff0c\u5546\u6237\u9700\u8981\u627e\u5f00\u7968\u5e73\u53f0\u63d0\u4f9b\n :param order_id: \u8ba2\u5355id\uff0c\u5728\u5546\u6237\u5185\u5355\u7b14\u5f00\u7968\u8bf7\u6c42\u7684\u552f\u4e00\u8bc6\u522b\u53f7\n :param reason: \u62d2\u7edd\u539f\u56e0\n :param redirect_url: \u8df3\u8f6c\u94fe\u63a5"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninserts a new entry into the cache.", "response": "def insert(self, order_id, card_id, appid, card_ext):\n \"\"\"\n \u5236\u4f5c\u53d1\u7968\u5361\u5238\uff0c\u5e76\u653e\u5165\u7528\u6237\u5361\u5305\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1497082828_r1cI2\n\n :param order_id: \u8ba2\u5355id\uff0c\u5728\u5546\u6237\u5185\u5355\u7b14\u5f00\u7968\u8bf7\u6c42\u7684\u552f\u4e00\u8bc6\u522b\u53f7\n :param card_id: \u53d1\u7968\u5361\u5238\u6a21\u677f\u7684\u7f16\u53f7\n :param appid: \u5546\u6237 AppID\n :param card_ext: \u53d1\u7968\u5177\u4f53\u5185\u5bb9\n :type card_ext: dict\n :return: \u968f\u673a\u9632\u91cd\u5b57\u7b26\u4e32\uff0c\u4ee5\u53ca\u7528\u6237 Open ID\n \"\"\"\n return self._post(\n 'insert',\n data={\n 'order_id': order_id,\n 'card_id': card_id,\n 'appid': appid,\n 'card_ext': card_ext,\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nuploads a PDF to the ACS.", "response": "def upload_pdf(self, pdf):\n \"\"\"\n \u4e0a\u4f20\u7535\u5b50\u53d1\u7968\u4e2d\u7684\u6d88\u8d39\u51ed\u8bc1 PDF\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1497082828_r1cI2\n\n :param pdf: \u8981\u4e0a\u4f20\u7684 PDF \u6587\u4ef6\uff0c\u4e00\u4e2a File-object\n :return: 64\u4f4d\u6574\u6570\uff0c\u5728\u5c06\u53d1\u7968\u5361\u5238\u63d2\u5165\u7528\u6237\u5361\u5305\u65f6\u4f7f\u7528\u7528\u4e8e\u5173\u8054pdf\u548c\u53d1\u7968\u5361\u5238\u3002\u6709\u6548\u671f\u4e3a3\u5929\u3002\n \"\"\"\n return self._post(\n 'platform/setpdf',\n files={\n 'pdf': pdf,\n },\n result_processor=lambda x: x['s_media_id'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_status(self, card_id, code, reimburse_status):\n return self._post(\n 'platform/updatestatus',\n data={\n 'card_id': card_id,\n 'code': code,\n 'reimburse_status': reimburse_status,\n },\n )", "response": "Update the status of a specific resource."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_pay_mch(self, mchid, s_pappid):\n return self._post(\n 'setbizattr',\n params={\n 'action': 'set_pay_mch',\n },\n data={\n 'paymch_info': {\n 'mchid': mchid,\n 's_pappid': s_pappid,\n },\n },\n )", "response": "Set the Mchid and S_Pappid of an invoice."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the status of a reimburse.", "response": "def update_reimburse(self, card_id, encrypt_code, reimburse_status):\n \"\"\"\n \u62a5\u9500\u65b9\u66f4\u65b0\u53d1\u7968\u4fe1\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1496561749_f7T6D\n\n :param card_id: \u53d1\u7968\u5361\u5238\u7684 Card ID\n :param encrypt_code: \u53d1\u7968\u5361\u5238\u7684\u52a0\u5bc6 Code\n :param reimburse_status: \u53d1\u7968\u62a5\u9500\u72b6\u6001\n \"\"\"\n return self._post(\n 'reimburse/updateinvoicestatus',\n data={\n 'card_id': card_id,\n 'encrypt_code': encrypt_code,\n 'reimburse_status': reimburse_status,\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbatching update reimburse status.", "response": "def batch_update_reimburse(self, openid, reimburse_status, invoice_list):\n \"\"\"\n \u62a5\u9500\u65b9\u6279\u91cf\u66f4\u65b0\u53d1\u7968\u4fe1\u606f\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1496561749_f7T6D\n\n :param openid: \u7528\u6237\u7684 Open ID\n :param reimburse_status: \u53d1\u7968\u62a5\u9500\u72b6\u6001\n :param invoice_list: \u53d1\u7968\u5217\u8868\n :type invoice_list: list[dict]\n \"\"\"\n return self._post(\n 'reimburse/updatestatusbatch',\n data={\n 'openid': openid,\n 'reimburse_status': reimburse_status,\n 'invoice_list': invoice_list,\n },\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the url to the user s title.", "response": "def get_user_title_url(\n self, user_fill, title=None, phone=None, tax_no=None, addr=None, bank_type=None, bank_no=None,\n out_title_id=None):\n \"\"\"\n \u83b7\u53d6\u6dfb\u52a0\u53d1\u7968\u94fe\u63a5\n \u83b7\u53d6\u94fe\u63a5\uff0c\u53d1\u9001\u7ed9\u7528\u6237\u3002\u7528\u6237\u540c\u610f\u4ee5\u540e\uff0c\u53d1\u7968\u62ac\u5934\u4fe1\u606f\u5c06\u4f1a\u5f55\u5165\u5230\u7528\u6237\u5fae\u4fe1\u4e2d\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1496554912_vfWU0\n\n :param user_fill: \u4f01\u4e1a\u8bbe\u7f6e\u62ac\u5934\u4e3a0\uff0c\u7528\u6237\u81ea\u5df1\u586b\u5199\u62ac\u5934\u4e3a1\n :type user_fill: bool\n :param title: \u62ac\u5934\uff0c\u5f53 user_fill \u4e3a False \u65f6\u5fc5\u586b\n :param phone: \u8054\u7cfb\u65b9\u5f0f\n :param tax_no: \u7a0e\u53f7\n :param addr: \u5730\u5740\n :param bank_type: \u94f6\u884c\u7c7b\u578b\n :param bank_no: \u94f6\u884c\u53f7\u7801\n :param out_title_id: \u5f00\u7968\u7801\n :return: \u6dfb\u52a0\u53d1\u7968\u7684\u94fe\u63a5\n \"\"\"\n if user_fill and title is None:\n raise ValueError('title is required when user_fill is False')\n return self._post(\n 'biz/getusertitleurl',\n data={\n 'user_fill': int(user_fill),\n 'title': title,\n 'phone': phone,\n 'tax_no': tax_no,\n 'addr': addr,\n 'bank_type': bank_type,\n 'bank_no': bank_no,\n 'out_title_id': out_title_id,\n },\n result_processor=lambda x: x['url'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget select title url.", "response": "def get_select_title_url(self, attach=None):\n \"\"\"\n \u83b7\u53d6\u5546\u6237\u4e13\u5c5e\u5f00\u7968\u94fe\u63a5\n \u5546\u6237\u8c03\u7528\u63a5\u53e3\uff0c\u83b7\u53d6\u94fe\u63a5\u3002\u7528\u6237\u626b\u7801\uff0c\u53ef\u4ee5\u9009\u62e9\u62ac\u5934\u53d1\u7ed9\u5546\u6237\u3002\u53ef\u4ee5\u5c06\u94fe\u63a5\u8f6c\u6210\u4e8c\u7ef4\u7801\uff0c\u7acb\u5728\u6536\u94f6\u53f0\u3002\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?id=mp1496554912_vfWU0\n\n :param attach: \u9644\u52a0\u5b57\u6bb5\uff0c\u7528\u6237\u63d0\u4ea4\u53d1\u7968\u65f6\u4f1a\u53d1\u9001\u7ed9\u5546\u6237\n :return: \u5546\u6237\u4e13\u5c5e\u5f00\u7968\u94fe\u63a5\n \"\"\"\n return self._post(\n 'biz/getselecttitleurl',\n data={\n 'attach': attach,\n },\n result_processor=lambda x: x['url'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuploads a media file to the Alien.", "response": "def upload(self, media_type, media_file):\n \"\"\"\n \u4e0a\u4f20\u4e34\u65f6\u7d20\u6750\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://mp.weixin.qq.com/wiki/5/963fc70b80dc75483a271298a76a8d59.html\n\n :param media_type: \u5a92\u4f53\u6587\u4ef6\u7c7b\u578b\uff0c\u5206\u522b\u6709\u56fe\u7247\uff08image\uff09\u3001\u8bed\u97f3\uff08voice\uff09\u3001\u89c6\u9891\uff08video\uff09\u548c\u7f29\u7565\u56fe\uff08thumb\uff09\n :param media_file: \u8981\u4e0a\u4f20\u7684\u6587\u4ef6\uff0c\u4e00\u4e2a File-object\n\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._post(\n url='media/upload',\n params={\n 'type': media_type\n },\n files={\n 'media': media_file\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef upload_video(self, media_id, title, description):\n return self._post(\n url='media/uploadvideo',\n data={\n 'media_id': media_id,\n 'title': title,\n 'description': description\n }\n )", "response": "Upload a video to the media."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef upload_image(self, media_file):\n res = self._post(\n url='media/uploadimg',\n files={\n 'media': media_file\n },\n result_processor=lambda x: x['url']\n )\n return res", "response": "Upload an image to the ACS."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a message to the server.", "response": "def send(self, user_id, total_amount, send_name, act_name,\n wishing, remark, total_num=1, client_ip=None,\n out_trade_no=None, scene_id=None, consume_mch_id=None):\n \"\"\"\n \u53d1\u9001\u73b0\u91d1\u7ea2\u5305\n\n :param user_id: \u63a5\u6536\u7ea2\u5305\u7684\u7528\u6237\u5728\u516c\u4f17\u53f7\u4e0b\u7684 openid\n :param total_amount: \u7ea2\u5305\u91d1\u989d\uff0c\u5355\u4f4d\u5206\n :param send_name: \u5546\u6237\u540d\u79f0\n :param act_name: \u6d3b\u52a8\u540d\u79f0\n :param wishing: \u7ea2\u5305\u795d\u798f\u8bed\n :param remark: \u5907\u6ce8\n :param client_ip: \u53ef\u9009\uff0c\u8c03\u7528\u63a5\u53e3\u7684\u673a\u5668 IP \u5730\u5740\n :param total_num: \u53ef\u9009\uff0c\u7ea2\u5305\u53d1\u653e\u603b\u4eba\u6570\uff0c\u9ed8\u8ba4\u4e3a 1\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u8ba2\u5355\u53f7\uff0c\u9ed8\u8ba4\u4f1a\u81ea\u52a8\u751f\u6210\n :param scene_id: \u53ef\u9009\uff0c\u53d1\u653e\u7ea2\u5305\u4f7f\u7528\u573a\u666f\uff0c\u7ea2\u5305\u91d1\u989d\u5927\u4e8e200\u65f6\u5fc5\u4f20\n :param consume_mch_id: \u53ef\u9009\uff0c\u8d44\u91d1\u6388\u6743\u5546\u6237\u53f7\u3002\u670d\u52a1\u5546\u66ff\u7279\u7ea6\u5546\u6237\u53d1\u653e\u65f6\u4f7f\u7528\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\u5b57\u5178\n \"\"\"\n if not out_trade_no:\n now = datetime.now()\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n data = {\n 'wxappid': self.appid,\n 're_openid': user_id,\n 'total_amount': total_amount,\n 'send_name': send_name,\n 'act_name': act_name,\n 'wishing': wishing,\n 'remark': remark,\n 'client_ip': client_ip or get_external_ip(),\n 'total_num': total_num,\n 'mch_billno': out_trade_no,\n 'scene_id': scene_id,\n 'risk_info': None,\n 'consume_mch_id': consume_mch_id\n }\n return self._post('mmpaymkttransfers/sendredpack', data=data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nquerying the MCHT related data.", "response": "def query(self, out_trade_no, bill_type='MCHT'):\n \"\"\"\n \u67e5\u8be2\u7ea2\u5305\u53d1\u653e\u8bb0\u5f55\n\n :param out_trade_no: \u5546\u6237\u8ba2\u5355\u53f7\n :param bill_type: \u53ef\u9009\uff0c\u8ba2\u5355\u7c7b\u578b\uff0c\u76ee\u524d\u56fa\u5b9a\u4e3a MCHT\n :return: \u8fd4\u56de\u7684\u7ea2\u5305\u53d1\u653e\u8bb0\u5f55\u4fe1\u606f\n \"\"\"\n data = {\n 'mch_billno': out_trade_no,\n 'bill_type': bill_type,\n 'appid': self.appid,\n }\n return self._post('mmpaymkttransfers/gethbinfo', data=data)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new object", "response": "def create(self, chat_id, name, owner, user_list):\n \"\"\"\n \u521b\u5efa\u4f1a\u8bdd\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=\u4f01\u4e1a\u4f1a\u8bdd\u63a5\u53e3\u8bf4\u660e\n\n :param chat_id: \u4f1a\u8bddid\u3002\u5b57\u7b26\u4e32\u7c7b\u578b\uff0c\u6700\u957f32\u4e2a\u5b57\u7b26\u3002\u53ea\u5141\u8bb8\u5b57\u7b260-9\u53ca\u5b57\u6bcda-zA-Z,\n \u5982\u679c\u503c\u5185\u5bb9\u4e3a64bit\u65e0\u7b26\u53f7\u6574\u578b\uff1a\u8981\u6c42\u503c\u8303\u56f4\u5728[1, 2^63)\u4e4b\u95f4\uff0c\n [2^63, 2^64)\u4e3a\u7cfb\u7edf\u5206\u914d\u4f1a\u8bddid\u533a\u95f4\n :param name: \u4f1a\u8bdd\u6807\u9898\n :param owner: \u7ba1\u7406\u5458userid\uff0c\u5fc5\u987b\u662f\u8be5\u4f1a\u8bdduserlist\u7684\u6210\u5458\u4e4b\u4e00\n :param user_list: \u4f1a\u8bdd\u6210\u5458\u5217\u8868\uff0c\u6210\u5458\u7528userid\u6765\u6807\u8bc6\u3002\u4f1a\u8bdd\u6210\u5458\u5fc5\u987b\u57283\u4eba\u6216\u4ee5\u4e0a\uff0c1000\u4eba\u4ee5\u4e0b\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self._post(\n 'chat/create',\n data={\n 'chatid': chat_id,\n 'name': name,\n 'owner': owner,\n 'userlist': user_list,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update(self, chat_id, op_user, name=None, owner=None,\n add_user_list=None, del_user_list=None):\n \"\"\"\n \u4fee\u6539\u4f1a\u8bdd\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=\u4f01\u4e1a\u4f1a\u8bdd\u63a5\u53e3\u8bf4\u660e\n\n :param chat_id: \u4f1a\u8bdd ID\n :param op_user: \u64cd\u4f5c\u4eba userid\n :param name: \u4f1a\u8bdd\u6807\u9898\n :param owner: \u7ba1\u7406\u5458userid\uff0c\u5fc5\u987b\u662f\u8be5\u4f1a\u8bdduserlist\u7684\u6210\u5458\u4e4b\u4e00\n :param add_user_list: \u4f1a\u8bdd\u65b0\u589e\u6210\u5458\u5217\u8868\uff0c\u6210\u5458\u7528userid\u6765\u6807\u8bc6\n :param del_user_list: \u4f1a\u8bdd\u9000\u51fa\u6210\u5458\u5217\u8868\uff0c\u6210\u5458\u7528userid\u6765\u6807\u8bc6\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = optionaldict(\n chatid=chat_id,\n op_user=op_user,\n name=name,\n owner=owner,\n add_user_list=add_user_list,\n del_user_list=del_user_list,\n )\n return self._post('chat/update', data=data)", "response": "Update the chat s internal cache."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef clear_notify(self, op_user, type, id):\n return self._post(\n 'chat/clearnotify',\n data={\n 'op_user': op_user,\n 'chat': {\n 'type': type,\n 'id': id,\n }\n }\n )", "response": "Clear the user s notify"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef send_text(self, sender, receiver_type, receiver_id, content):\n data = {\n 'receiver': {\n 'type': receiver_type,\n 'id': receiver_id,\n },\n 'sender': sender,\n 'msgtype': 'text',\n 'text': {\n 'content': content,\n }\n }\n return self._post('chat/send', data=data)", "response": "Send text to a user"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_single_text(self, sender, receiver, content):\n return self.send_text(sender, 'single', receiver, content)", "response": "Send a single text message to the receiver."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a group text message to the receiver.", "response": "def send_group_text(self, sender, receiver, content):\n \"\"\"\n \u53d1\u9001\u7fa4\u804a\u6587\u672c\u6d88\u606f\n\n :param sender: \u53d1\u9001\u4eba\n :param receiver: \u4f1a\u8bdd ID\n :param content: \u6d88\u606f\u5185\u5bb9\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self.send_text(sender, 'group', receiver, content)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsends an image message to a specific sender", "response": "def send_image(self, sender, receiver_type, receiver_id, media_id):\n \"\"\"\n \u53d1\u9001\u56fe\u7247\u6d88\u606f\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://qydev.weixin.qq.com/wiki/index.php?title=\u4f01\u4e1a\u4f1a\u8bdd\u63a5\u53e3\u8bf4\u660e\n\n :param sender: \u53d1\u9001\u4eba\n :param receiver_type: \u63a5\u6536\u4eba\u7c7b\u578b\uff1asingle|group\uff0c\u5206\u522b\u8868\u793a\uff1a\u5355\u804a|\u7fa4\u804a\n :param receiver_id: \u63a5\u6536\u4eba\u7684\u503c\uff0c\u4e3auserid|chatid\uff0c\u5206\u522b\u8868\u793a\uff1a\u6210\u5458id|\u4f1a\u8bddid\n :param media_id: \u56fe\u7247\u5a92\u4f53\u6587\u4ef6id\uff0c\u53ef\u4ee5\u8c03\u7528\u4e0a\u4f20\u7d20\u6750\u6587\u4ef6\u63a5\u53e3\u83b7\u53d6\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n data = {\n 'receiver': {\n 'type': receiver_type,\n 'id': receiver_id,\n },\n 'sender': sender,\n 'msgtype': 'image',\n 'image': {\n 'media_id': media_id,\n }\n }\n return self._post('chat/send', data=data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_group_image(self, sender, receiver, media_id):\n return self.send_image(sender, 'group', receiver, media_id)", "response": "Send a group image"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_file(self, sender, receiver_type, receiver_id, media_id):\n data = {\n 'receiver': {\n 'type': receiver_type,\n 'id': receiver_id,\n },\n 'sender': sender,\n 'msgtype': 'file',\n 'file': {\n 'media_id': media_id,\n }\n }\n return self._post('chat/send', data=data)", "response": "Send a file to a specific receiver"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_single_file(self, sender, receiver, media_id):\n return self.send_file(sender, 'single', receiver, media_id)", "response": "send_file - Sends a single file to a receiver"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends a file to a group", "response": "def send_group_file(self, sender, receiver, media_id):\n \"\"\"\n \u53d1\u9001\u7fa4\u804a\u6587\u4ef6\u6d88\u606f\n\n :param sender: \u53d1\u9001\u4eba\n :param receiver: \u4f1a\u8bdd ID\n :param media_id: \u6587\u4ef6id\uff0c\u53ef\u4ee5\u8c03\u7528\u4e0a\u4f20\u7d20\u6750\u6587\u4ef6\u63a5\u53e3\u83b7\u53d6, \u6587\u4ef6\u987b\u5927\u4e8e4\u5b57\u8282\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n \"\"\"\n return self.send_file(sender, 'group', receiver, media_id)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send(self, user_id, stock_id, op_user_id=None, device_info=None,\n out_trade_no=None):\n \"\"\"\n \u53d1\u653e\u4ee3\u91d1\u5238\n\n :param user_id: \u7528\u6237\u5728\u516c\u4f17\u53f7\u4e0b\u7684 openid\n :param stock_id: \u4ee3\u91d1\u5238\u6279\u6b21 ID\n :param op_user_id: \u53ef\u9009\uff0c\u64cd\u4f5c\u5458\u8d26\u53f7\uff0c\u9ed8\u8ba4\u4e3a\u5546\u6237\u53f7\n :param device_info: \u53ef\u9009\uff0c\u5fae\u4fe1\u652f\u4ed8\u5206\u914d\u7684\u7ec8\u7aef\u8bbe\u5907\u53f7\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u8ba2\u5355\u53f7\uff0c\u9700\u4fdd\u6301\u552f\u4e00\u6027\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u4fe1\u606f\n \"\"\"\n if not out_trade_no:\n now = datetime.now()\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n data = {\n 'appid': self.appid,\n 'coupon_stock_id': stock_id,\n 'openid': user_id,\n 'openid_count': 1,\n 'partner_trade_no': out_trade_no,\n 'op_user_id': op_user_id,\n 'device_info': device_info,\n 'version': '1.0',\n 'type': 'XML',\n }\n return self._post('mmpaymkttransfers/send_coupon', data=data)", "response": "Send a new invoice to the Mch."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef query_stock(self, stock_id, op_user_id=None, device_info=None):\n data = {\n 'appid': self.appid,\n 'coupon_stock_id': stock_id,\n 'op_user_id': op_user_id,\n 'device_info': device_info,\n 'version': '1.0',\n 'type': 'XML',\n }\n return self._post('mmpaymkttransfers/query_coupon_stock', data=data)", "response": "Query the stock for a given coupon."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds user action sets to a user.", "response": "def add_user_action_sets(self, _type, name, description, version='v1.0'):\n \"\"\"\n \u521b\u5efa\u6570\u636e\u6e90\n https://wximg.qq.com/wxp/pdftool/get.html?id=rkalQXDBM&pa=39\n\n :param _type: \u7528\u6237\u884c\u4e3a\u6e90\u7c7b\u578b\n :param name: \u7528\u6237\u884c\u4e3a\u6e90\u540d\u79f0 \u5fc5\u586b\n :param description: \u7528\u6237\u884c\u4e3a\u6e90\u63cf\u8ff0\uff0c\u5b57\u6bb5\u957f\u5ea6\u6700\u5c0f 1 \u5b57\u8282\uff0c\u957f\u5ea6\u6700\u5927 128 \u5b57\u8282\n :param version: \u7248\u672c\u53f7 v1.0\n :return: \u6570\u636e\u6e90\u552f\u4e00ID\n \"\"\"\n return self._post(\n 'user_action_sets/add',\n params={'version': version},\n json=optionaldict(\n type=_type,\n name=name,\n description=description,\n version=version\n ),\n result_processor=lambda x: x['data']['user_action_set_id']\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets user action sets.", "response": "def get_user_action_sets(self, user_action_set_id, version='v1.0'):\n \"\"\"\n \u83b7\u53d6\u6570\u636e\u6e90\u4fe1\u606f\n\n :param user_action_set_id: \u6570\u636e\u6e90\u552f\u4e00ID\n :param version: \u7248\u672c\u53f7 v1.0\n \"\"\"\n return self._get(\n 'user_action_sets/get',\n params={'version': version, 'user_action_set_id': user_action_set_id},\n result_processor=lambda x: x['data']['list']\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_user_actions(self, actions=(), version='v1.0'):\n return self._post(\n 'user_actions/add',\n params={'version': version},\n json={'actions': actions}\n )", "response": "Add user actions to a user"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a list of ad leads.", "response": "def get_ad_leads(self, start_date=None, end_date=None, filtering=(), page=1, page_size=100, version='v1.0'):\n \"\"\"\n \u83b7\u53d6\u670b\u53cb\u5708\u9500\u552e\u7ebf\u7d22\u6570\u636e\u63a5\u53e3\n\n :param start_date: \u5f00\u59cb\u65e5\u671f \u9ed8\u8ba4\u4eca\u5929\n :param end_date: \u7ed3\u675f\u65e5\u671f \u9ed8\u8ba4\u4eca\u5929\n :param filtering: \u8fc7\u6ee4\u6761\u4ef6 [{field: \u8fc7\u6ee4\u5b57\u6bb5, operator: \u64cd\u4f5c\u7b26, values: \u5b57\u6bb5\u53d6\u503c}]\n :param page: \u9875\u7801\uff0c\u83b7\u53d6\u6307\u5b9a\u9875\u6570\u636e\n :param page_size: \u4e00\u9875\u83b7\u53d6\u7684\u6570\u636e\u6761\u6570(1-100)\n :param version: \u7248\u672c\u53f7 v1.0\n \"\"\"\n today = datetime.date.today()\n if start_date is None:\n start_date = today\n if end_date is None:\n end_date = today\n if isinstance(start_date, datetime.date):\n start_date = start_date.strftime(\"%Y-%m-%d\")\n if isinstance(end_date, datetime.date):\n end_date = end_date.strftime(\"%Y-%m-%d\")\n\n return self._get(\n 'wechat_ad_leads/get',\n params=optionaldict(\n date_range=json.dumps({'start_date': start_date, 'end_date': end_date}),\n filtering=json.dumps(filtering) if filtering else None,\n page=page,\n page_size=page_size,\n version=version\n ),\n result_processor=lambda x: x['data']\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create(self, trade_type, body, total_fee, notify_url, client_ip=None,\n user_id=None, out_trade_no=None, detail=None, attach=None,\n fee_type='CNY', time_start=None, time_expire=None, goods_tag=None,\n product_id=None, device_info=None, limit_pay=None, scene_info=None, sub_user_id=None):\n \"\"\"\n \u7edf\u4e00\u4e0b\u5355\u63a5\u53e3\n\n :param trade_type: \u4ea4\u6613\u7c7b\u578b\uff0c\u53d6\u503c\u5982\u4e0b\uff1aJSAPI\uff0cNATIVE\uff0cAPP\uff0cWAP, MWEB\n :param body: \u5546\u54c1\u63cf\u8ff0\n :param total_fee: \u603b\u91d1\u989d\uff0c\u5355\u4f4d\u5206\n :param notify_url: \u63a5\u6536\u5fae\u4fe1\u652f\u4ed8\u5f02\u6b65\u901a\u77e5\u56de\u8c03\u5730\u5740\n :param client_ip: \u53ef\u9009\uff0cAPP\u548c\u7f51\u9875\u652f\u4ed8\u63d0\u4ea4\u7528\u6237\u7aefip\uff0cNative\u652f\u4ed8\u586b\u8c03\u7528\u5fae\u4fe1\u652f\u4ed8API\u7684\u673a\u5668IP\n :param user_id: \u53ef\u9009\uff0c\u7528\u6237\u5728\u5546\u6237appid\u4e0b\u7684\u552f\u4e00\u6807\u8bc6\u3002trade_type=JSAPI\u548cappid\u5df2\u8bbe\u5b9a\uff0c\u6b64\u53c2\u6570\u5fc5\u4f20\n :param sub_user_id: \u53ef\u9009\uff0c\u5c0f\u7a0b\u5e8fappid\u4e0b\u7684\u552f\u4e00\u6807\u8bc6\u3002trade_type=JSAPI\u548csub_appid\u5df2\u8bbe\u5b9a\uff0c\u6b64\u53c2\u6570\u5fc5\u4f20\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u8ba2\u5355\u53f7\uff0c\u9ed8\u8ba4\u81ea\u52a8\u751f\u6210\n :param detail: \u53ef\u9009\uff0c\u5546\u54c1\u8be6\u60c5\n :param attach: \u53ef\u9009\uff0c\u9644\u52a0\u6570\u636e\uff0c\u5728\u67e5\u8be2API\u548c\u652f\u4ed8\u901a\u77e5\u4e2d\u539f\u6837\u8fd4\u56de\uff0c\u8be5\u5b57\u6bb5\u4e3b\u8981\u7528\u4e8e\u5546\u6237\u643a\u5e26\u8ba2\u5355\u7684\u81ea\u5b9a\u4e49\u6570\u636e\n :param fee_type: \u53ef\u9009\uff0c\u7b26\u5408ISO 4217\u6807\u51c6\u7684\u4e09\u4f4d\u5b57\u6bcd\u4ee3\u7801\uff0c\u9ed8\u8ba4\u4eba\u6c11\u5e01\uff1aCNY\n :param time_start: \u53ef\u9009\uff0c\u8ba2\u5355\u751f\u6210\u65f6\u95f4\uff0c\u9ed8\u8ba4\u4e3a\u5f53\u524d\u65f6\u95f4\n :param time_expire: \u53ef\u9009\uff0c\u8ba2\u5355\u5931\u6548\u65f6\u95f4\uff0c\u9ed8\u8ba4\u4e3a\u8ba2\u5355\u751f\u6210\u65f6\u95f4\u540e\u4e24\u5c0f\u65f6\n :param goods_tag: \u53ef\u9009\uff0c\u5546\u54c1\u6807\u8bb0\uff0c\u4ee3\u91d1\u5238\u6216\u7acb\u51cf\u4f18\u60e0\u529f\u80fd\u7684\u53c2\u6570\n :param product_id: \u53ef\u9009\uff0ctrade_type=NATIVE\uff0c\u6b64\u53c2\u6570\u5fc5\u4f20\u3002\u6b64id\u4e3a\u4e8c\u7ef4\u7801\u4e2d\u5305\u542b\u7684\u5546\u54c1ID\uff0c\u5546\u6237\u81ea\u884c\u5b9a\u4e49\n :param device_info: \u53ef\u9009\uff0c\u7ec8\u7aef\u8bbe\u5907\u53f7(\u95e8\u5e97\u53f7\u6216\u6536\u94f6\u8bbe\u5907ID)\uff0c\u6ce8\u610f\uff1aPC\u7f51\u9875\u6216\u516c\u4f17\u53f7\u5185\u652f\u4ed8\u8bf7\u4f20\"WEB\"\n :param limit_pay: \u53ef\u9009\uff0c\u6307\u5b9a\u652f\u4ed8\u65b9\u5f0f\uff0cno_credit--\u6307\u5b9a\u4e0d\u80fd\u4f7f\u7528\u4fe1\u7528\u5361\u652f\u4ed8\n :param scene_info: \u53ef\u9009\uff0c\u4e0a\u62a5\u652f\u4ed8\u7684\u573a\u666f\u4fe1\u606f\n :type scene_info: dict\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e\n \"\"\"\n now = datetime.fromtimestamp(time.time(), tz=timezone('Asia/Shanghai'))\n hours_later = now + timedelta(hours=2)\n if time_start is None:\n time_start = now\n if time_expire is None:\n time_expire = hours_later\n if not out_trade_no:\n out_trade_no = '{0}{1}{2}'.format(\n self.mch_id,\n now.strftime('%Y%m%d%H%M%S'),\n random.randint(1000, 10000)\n )\n if scene_info is not None:\n scene_info = json.dumps(scene_info, ensure_ascii=False)\n data = {\n 'appid': self.appid,\n 'sub_appid': self.sub_appid,\n 'device_info': device_info,\n 'body': body,\n 'detail': detail,\n 'attach': attach,\n 'out_trade_no': out_trade_no,\n 'fee_type': fee_type,\n 'total_fee': total_fee,\n 'spbill_create_ip': client_ip or get_external_ip(),\n 'time_start': time_start.strftime('%Y%m%d%H%M%S'),\n 'time_expire': time_expire.strftime('%Y%m%d%H%M%S'),\n 'goods_tag': goods_tag,\n 'notify_url': notify_url,\n 'trade_type': trade_type,\n 'limit_pay': limit_pay,\n 'product_id': product_id,\n 'openid': user_id,\n 'sub_openid': sub_user_id,\n 'scene_info': scene_info,\n }\n return self._post('pay/unifiedorder', data=data)", "response": "Create a new node."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef reverse(self, transaction_id=None, out_trade_no=None):\n data = {\n 'appid': self.appid,\n 'transaction_id': transaction_id,\n 'out_trade_no': out_trade_no,\n }\n return self._post('secapi/pay/reverse', data=data)", "response": "\u64a4\u9500\u8ba2\u5355\n\n :param transaction_id: \u53ef\u9009\uff0c\u5fae\u4fe1\u7684\u8ba2\u5355\u53f7\uff0c\u4f18\u5148\u4f7f\u7528\n :param out_trade_no: \u53ef\u9009\uff0c\u5546\u6237\u7cfb\u7edf\u5185\u90e8\u7684\u8ba2\u5355\u53f7,\n transaction_id\u3001out_trade_no\u4e8c\u9009\u4e00\uff0c\n \u5982\u679c\u540c\u65f6\u5b58\u5728\u4f18\u5148\u7ea7\uff1atransaction_id> out_trade_no\n :return: \u8fd4\u56de\u7684\u7ed3\u679c\u6570\u636e"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, user_id, lang='zh_CN'):\n assert lang in ('zh_CN', 'zh_TW', 'en'), 'lang can only be one of \\\n zh_CN, zh_TW, en language codes'\n return self._get(\n 'user/info',\n params={\n 'openid': user_id,\n 'lang': lang\n }\n )", "response": "\u83b7\u53d6\u7528\u6237\u57fa\u672c\u4fe1\u606f\uff08\u5305\u62ecUnionID\u673a\u5236\uff09\n \u8be6\u60c5\u8bf7\u53c2\u8003\n https://mp.weixin.qq.com/wiki?t=resource/res_main&id=mp1421140839\n\n :param user_id: \u666e\u901a\u7528\u6237\u7684\u6807\u8bc6\uff0c\u5bf9\u5f53\u524d\u516c\u4f17\u53f7\u552f\u4e00\n :param lang: \u8fd4\u56de\u56fd\u5bb6\u5730\u533a\u8bed\u8a00\u7248\u672c\uff0czh_CN \u7b80\u4f53\uff0czh_TW \u7e41\u4f53\uff0cen \u82f1\u8bed\n :return: \u8fd4\u56de\u7684 JSON \u6570\u636e\u5305\n\n \u4f7f\u7528\u793a\u4f8b::\n\n from wechatpy import WeChatClient\n\n client = WeChatClient('appid', 'secret')\n user = client.user.get('openid')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_followers(self, first_user_id=None):\n params = {}\n if first_user_id:\n params['next_openid'] = first_user_id\n return self._get(\n 'user/get',\n params=params\n )", "response": "Get list of followers."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_batch(self, user_list):\n if all((isinstance(x, six.string_types) for x in user_list)):\n user_list = [{'openid': oid} for oid in user_list]\n res = self._post(\n 'user/info/batchget',\n data={'user_list': user_list},\n result_processor=lambda x: x['user_info_list']\n )\n return res", "response": "Get a list of user_list"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef change_openid(self, from_appid, openid_list):\n '''\u5fae\u4fe1\u516c\u4f17\u53f7\u4e3b\u4f53\u53d8\u66f4\u8fc1\u79fb\u7528\u6237 openid\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://kf.qq.com/faq/170221aUnmmU170221eUZJNf.html\n\n :param from_appid: \u539f\u516c\u4f17\u53f7\u7684 appid\n :param openid_list: \u9700\u8981\u8f6c\u6362\u7684openid\uff0c\u8fd9\u4e9b\u5fc5\u987b\u662f\u65e7\u8d26\u53f7\u76ee\u524d\u5173\u6ce8\u7684\u624d\u884c\uff0c\u5426\u5219\u4f1a\u51fa\u9519\uff1b\u4e00\u6b21\u6700\u591a100\u4e2a\n :return: \u8f6c\u6362\u540e\u7684 openid \u4fe1\u606f\u5217\u8868\n '''\n return self._post(\n 'changeopenid',\n data={'from_appid': from_appid, 'openid_list': openid_list},\n result_processor=lambda x: x['result_list']\n )", "response": "\u5fae\u4fe1\u516c\u4f17\u53f7\u4e3b\u4f53\u53d8\u66f4\u8fc1\u79fb\u7528\u6237 openid\n\n \u8be6\u60c5\u8bf7\u53c2\u8003\n http://kf.qq.com/faq/170221aUnmmU170221eUZJNf.html\n\n :param from_appid: \u539f\u516c\u4f17\u53f7\u7684 appid\n :param openid_list: \u9700\u8981\u8f6c\u6362\u7684openid\uff0c\u8fd9\u4e9b\u5fc5\u987b\u662f\u65e7\u8d26\u53f7\u76ee\u524d\u5173\u6ce8\u7684\u624d\u884c\uff0c\u5426\u5219\u4f1a\u51fa\u9519\uff1b\u4e00\u6b21\u6700\u591a100\u4e2a\n :return: \u8f6c\u6362\u540e\u7684 openid \u4fe1\u606f\u5217\u8868"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send(self, agent_id, user_ids, party_ids='',\n tag_ids='', msg=None):\n \"\"\"\n \u901a\u7528\u7684\u6d88\u606f\u53d1\u9001\u63a5\u53e3\u3002msg \u5185\u9700\u8981\u6307\u5b9a msgtype \u548c\u5bf9\u5e94\u7c7b\u578b\u6d88\u606f\u5fc5\u987b\u7684\u5b57\u6bb5\u3002\n \u5982\u679c\u90e8\u5206\u63a5\u6536\u4eba\u65e0\u6743\u9650\u6216\u4e0d\u5b58\u5728\uff0c\u53d1\u9001\u4ecd\u7136\u6267\u884c\uff0c\u4f46\u4f1a\u8fd4\u56de\u65e0\u6548\u7684\u90e8\u5206\uff08\u5373invaliduser\u6216invalidparty\u6216invalidtag\uff09\uff0c\u5e38\u89c1\u7684\u539f\u56e0\u662f\u63a5\u6536\u4eba\u4e0d\u5728\u5e94\u7528\u7684\u53ef\u89c1\u8303\u56f4\u5185\u3002\n user_ids\u3001party_ids\u3001tag_ids \u4e0d\u80fd\u540c\u65f6\u4e3a\u7a7a\uff0c\u540e\u9762\u4e0d\u518d\u5f3a\u8c03\u3002\n\n :param agent_id: \u5fc5\u586b\uff0c\u4f01\u4e1a\u5e94\u7528\u7684id\uff0c\u6574\u578b\u3002\u53ef\u5728\u5e94\u7528\u7684\u8bbe\u7f6e\u9875\u9762\u67e5\u770b\u3002\n :param user_ids: \u6210\u5458ID\u5217\u8868\u3002\n :param party_ids: \u90e8\u95e8ID\u5217\u8868\u3002\n :param tag_ids: \u6807\u7b7eID\u5217\u8868\u3002\n :param msg: \u53d1\u9001\u6d88\u606f\u7684 dict \u5bf9\u8c61\n :type msg: dict | None\n :return: \u63a5\u53e3\u8c03\u7528\u7ed3\u679c\n \"\"\"\n msg = msg or {}\n if isinstance(user_ids, (tuple, list)):\n user_ids = '|'.join(user_ids)\n if isinstance(party_ids, (tuple, list)):\n party_ids = '|'.join(party_ids)\n if isinstance(tag_ids, (tuple, list)):\n tag_ids = '|'.join(tag_ids)\n\n data = {\n 'touser': user_ids,\n 'toparty': party_ids,\n 'totag': tag_ids,\n 'agentid': agent_id\n }\n data.update(msg)\n return self._post('message/send', data=data)", "response": "Send a message to the specified agent."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a textcard to the given user_ids.", "response": "def send_text_card(self, agent_id, user_ids, title, description, url, btntxt='\u8be6\u60c5',\n party_ids='', tag_ids=''):\n \"\"\" \u6587\u672c\u5361\u7247\u6d88\u606f\n\n https://work.weixin.qq.com/api/doc#90000/90135/90236/\u6587\u672c\u5361\u7247\u6d88\u606f\n\n \u8bf7\u6c42\u793a\u4f8b\uff1a\n {\n \"touser\" : \"UserID1|UserID2|UserID3\",\n \"toparty\" : \"PartyID1 | PartyID2\",\n \"totag\" : \"TagID1 | TagID2\",\n \"msgtype\" : \"textcard\",\n \"agentid\" : 1,\n \"textcard\" : {\n \"title\" : \"\u9886\u5956\u901a\u77e5\",\n \"description\" : \"
2016\u5e749\u670826\u65e5
\u606d\u559c\u4f60\u62bd\u4e2diPhone 7\u4e00\u53f0\uff0c\n \u9886\u5956\u7801\uff1axxxx
\u8bf7\u4e8e2016\u5e7410\u670810\u65e5\u524d\u8054\u7cfb\u884c\u653f\u540c\u4e8b\u9886\u53d6
\",\n \"url\" : \"URL\",\n \"btntxt\":\"\u66f4\u591a\"\n }\n }\n\n \u7279\u6b8a\u8bf4\u660e\uff1a\n \u5361\u7247\u6d88\u606f\u7684\u5c55\u73b0\u5f62\u5f0f\u975e\u5e38\u7075\u6d3b\uff0c\u652f\u6301\u4f7f\u7528br\u6807\u7b7e\u6216\u8005\u7a7a\u683c\u6765\u8fdb\u884c\u6362\u884c\u5904\u7406\uff0c\u4e5f\u652f\u6301\u4f7f\u7528div\u6807\u7b7e\u6765\u4f7f\u7528\u4e0d\u540c\u7684\u5b57\u4f53\u989c\u8272\uff0c\n \u76ee\u524d\u5185\u7f6e\u4e863\u79cd\u6587\u5b57\u989c\u8272\uff1a\u7070\u8272(gray)\u3001\u9ad8\u4eae(highlight)\u3001\u9ed8\u8ba4\u9ed1\u8272(normal)\uff0c\u5c06\u5176\u4f5c\u4e3adiv\u6807\u7b7e\u7684class\u5c5e\u6027\u5373\u53ef\uff0c\n \u5177\u4f53\u7528\u6cd5\u8bf7\u53c2\u8003\u4e0a\u9762\u7684\u793a\u4f8b\u3002\n\n :param agent_id: \u5fc5\u586b\uff0c\u4f01\u4e1a\u5e94\u7528\u7684id\uff0c\u6574\u578b\u3002\u53ef\u5728\u5e94\u7528\u7684\u8bbe\u7f6e\u9875\u9762\u67e5\u770b\u3002\n :param user_ids: \u6210\u5458ID\u5217\u8868\uff08\u6d88\u606f\u63a5\u6536\u8005\uff0c\u591a\u4e2a\u63a5\u6536\u8005\u7528\u2018|\u2019\u5206\u9694\uff0c\u6700\u591a\u652f\u63011000\u4e2a\uff09\u3002\n :param title: \u6807\u9898\uff0c\u4e0d\u8d85\u8fc7128\u4e2a\u5b57\u8282\uff0c\u8d85\u8fc7\u4f1a\u81ea\u52a8\u622a\u65ad\u3002\n :param description: \u5fc5\u586b\uff0c\u63cf\u8ff0\uff0c\u4e0d\u8d85\u8fc7512\u4e2a\u5b57\u8282\uff0c\u8d85\u8fc7\u4f1a\u81ea\u52a8\u622a\u65ad\n :param url: \u5fc5\u586b\uff0c\u70b9\u51fb\u540e\u8df3\u8f6c\u7684\u94fe\u63a5\u3002\n :param btntxt: \u6309\u94ae\u6587\u5b57\u3002 \u9ed8\u8ba4\u4e3a\u201c\u8be6\u60c5\u201d\uff0c \u4e0d\u8d85\u8fc74\u4e2a\u6587\u5b57\uff0c\u8d85\u8fc7\u81ea\u52a8\u622a\u65ad\u3002\n :param party_ids: \u90e8\u95e8ID\u5217\u8868\u3002\n :param tag_ids: \u6807\u7b7eID\u5217\u8868\u3002\n \"\"\"\n return self.send(\n agent_id,\n user_ids,\n party_ids,\n tag_ids,\n msg={\n 'msgtype': 'textcard',\n 'textcard': {\n 'title': title,\n 'description': description,\n 'url': url,\n 'btntxt': btntxt,\n },\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_markdown(self, agent_id, user_ids, content, party_ids='', tag_ids=''):\n msg = {\n \"msgtype\": \"markdown\",\n \"markdown\": {\"content\": content}\n }\n return self._send_message(\n agent_id=agent_id,\n user_ids=user_ids,\n party_ids=party_ids,\n tag_ids=tag_ids,\n msg=msg\n )", "response": "Send markdown to the specified user."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef multivariate_normal(x, mu, L):\n mu_ndims = mu.shape.ndims\n x_ndims = x.shape.ndims\n if x_ndims is not None and x_ndims != 2:\n raise ValueError('Shape of x must be 2D.')\n if mu_ndims is not None and mu_ndims != 2:\n raise ValueError('Shape of mu must be 2D.')\n\n d = x - mu\n alpha = tf.matrix_triangular_solve(L, d, lower=True)\n num_dims = tf.cast(tf.shape(d)[0], L.dtype)\n p = - 0.5 * tf.reduce_sum(tf.square(alpha), 0)\n p -= 0.5 * num_dims * np.log(2 * np.pi)\n p -= tf.reduce_sum(tf.log(tf.matrix_diag_part(L)))\n return p", "response": "Computes the log - density of a multivariate normal distribution."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmaking Tensorflow optimization tensor. This method builds optimization tensor and initializes all necessary variables created by optimizer. :param model: GPflow model. :param session: Tensorflow session. :param var_list: List of variables for training. :param kwargs: Dictionary of extra parameters passed to Tensorflow optimizer's minimize method. :return: Tensorflow optimization tensor or operation.", "response": "def make_optimize_tensor(self, model, session=None, var_list=None, **kwargs):\n \"\"\"\n Make Tensorflow optimization tensor.\n This method builds optimization tensor and initializes all necessary variables\n created by optimizer.\n\n :param model: GPflow model.\n :param session: Tensorflow session.\n :param var_list: List of variables for training.\n :param kwargs: Dictionary of extra parameters passed to Tensorflow\n optimizer's minimize method.\n :return: Tensorflow optimization tensor or operation.\n \"\"\"\n session = model.enquire_session(session)\n objective = model.objective\n full_var_list = self._gen_var_list(model, var_list)\n # Create optimizer variables before initialization.\n with session.as_default():\n minimize = self.optimizer.minimize(objective, var_list=full_var_list, **kwargs)\n model.initialize(session=session)\n self._initialize_optimizer(session)\n return minimize"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_optimize_action(self, model, session=None, var_list=None, **kwargs):\n if model is None or not isinstance(model, Model):\n raise ValueError('Unknown type passed for optimization.')\n session = model.enquire_session(session)\n feed_dict = kwargs.pop('feed_dict', None)\n feed_dict_update = self._gen_feed_dict(model, feed_dict)\n run_kwargs = {} if feed_dict_update is None else {'feed_dict': feed_dict_update}\n optimizer_tensor = self.make_optimize_tensor(model, session, var_list=var_list, **kwargs)\n opt = Optimization()\n opt.with_optimizer(self)\n opt.with_model(model)\n opt.with_optimizer_tensor(optimizer_tensor)\n opt.with_run_kwargs(**run_kwargs)\n return opt", "response": "Build Optimization action task with Tensorflow optimizer."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef minimize(self, model, session=None, var_list=None, feed_dict=None,\n maxiter=1000, initialize=False, anchor=True, step_callback=None, **kwargs):\n \"\"\"\n Minimizes objective function of the model.\n\n :param model: GPflow model with objective tensor.\n :param session: Session where optimization will be run.\n :param var_list: List of extra variables which should be trained during optimization.\n :param feed_dict: Feed dictionary of tensors passed to session run method.\n :param maxiter: Number of run interation.\n :param initialize: If `True` model parameters will be re-initialized even if they were\n initialized before for gotten session.\n :param anchor: If `True` trained variable values computed during optimization at\n particular session will be synchronized with internal parameter values.\n :param step_callback: A callback function to execute at each optimization step.\n The callback should accept variable argument list, where first argument is\n optimization step number.\n :type step_callback: Callable[[], None]\n :param kwargs: This is a dictionary of extra parameters for session run method.\n \"\"\"\n\n if model is None or not isinstance(model, Model):\n raise ValueError('The `model` argument must be a GPflow model.')\n\n opt = self.make_optimize_action(model,\n session=session,\n var_list=var_list,\n feed_dict=feed_dict, **kwargs)\n\n self._model = opt.model\n self._minimize_operation = opt.optimizer_tensor\n\n session = model.enquire_session(session)\n with session.as_default():\n for step in range(maxiter):\n opt()\n if step_callback is not None:\n step_callback(step)\n\n if anchor:\n opt.model.anchor(session)", "response": "Minimizes objective function of the model."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a tf. Session with the given arguments and kwargs.", "response": "def get_session(*args, **kwargs):\n \"\"\"\n Pass session configuration options\n \"\"\"\n if 'config' not in kwargs:\n kwargs['config'] = tf.ConfigProto(**settings.session)\n if settings.profiling.dump_timeline:\n def fill_kwargs(key, value):\n \"\"\"\n Internal function for filling default None values with meaningful\n values from gpflow settings.\n \"\"\"\n if kwargs.get(key) is None:\n kwargs[key] = value\n fill_kwargs('output_file_name', settings.profiling.output_file_name)\n fill_kwargs('output_directory', settings.profiling.output_directory)\n fill_kwargs('each_time', settings.profiling.each_time)\n return TracerSession(*args, **kwargs)\n kwargs.pop(\"output_file_name\", None)\n kwargs.pop(\"output_directory\", None)\n kwargs.pop(\"each_time\", None)\n return tf.Session(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes SciPy optimization tensor. The `make_optimize_tensor` method builds optimization tensor and initializes all necessary variables created by optimizer. :param model: GPflow model. :param session: Tensorflow session. :param var_list: List of variables for training. :param kwargs: Scipy optional optimization parameters, - `maxiter`, maximal number of iterations to perform. - `disp`, if True, prints convergence messages. :return: Tensorflow operation.", "response": "def make_optimize_tensor(self, model, session=None, var_list=None, **kwargs):\n \"\"\"\n Make SciPy optimization tensor.\n The `make_optimize_tensor` method builds optimization tensor and initializes\n all necessary variables created by optimizer.\n\n :param model: GPflow model.\n :param session: Tensorflow session.\n :param var_list: List of variables for training.\n :param kwargs: Scipy optional optimization parameters,\n - `maxiter`, maximal number of iterations to perform.\n - `disp`, if True, prints convergence messages.\n :return: Tensorflow operation.\n \"\"\"\n session = model.enquire_session(session)\n with session.as_default():\n var_list = self._gen_var_list(model, var_list)\n optimizer_kwargs = self._optimizer_kwargs.copy()\n options = optimizer_kwargs.get('options', {})\n options.update(kwargs)\n optimizer_kwargs.update(dict(options=options))\n objective = model.objective\n optimizer = external_optimizer.ScipyOptimizerInterface(\n objective, var_list=var_list, **optimizer_kwargs)\n model.initialize(session=session)\n return optimizer"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nminimizing objective function of the model.", "response": "def minimize(self, model, session=None, var_list=None, feed_dict=None, maxiter=1000,\n disp=False, initialize=False, anchor=True, step_callback=None, **kwargs):\n \"\"\"\n Minimizes objective function of the model.\n\n :param model: GPflow model with objective tensor.\n :param session: Session where optimization will be run.\n :param var_list: List of extra variables which should be trained during optimization.\n :param feed_dict: Feed dictionary of tensors passed to session run method.\n :param maxiter: Number of run interation. Note: scipy optimizer can do early stopping\n if model converged.\n :param disp: ScipyOptimizer option. Set to True to print convergence messages.\n :param initialize: If `True` model parameters will be re-initialized even if they were\n initialized before for gotten session.\n :param anchor: If `True` trained parameters computed during optimization at\n particular session will be synchronized with internal parameter values.\n :param step_callback: A function to be called at each optimization step;\n arguments are the current values of all optimization variables\n flattened into a single vector.\n :type step_callback: Callable[[np.ndarray], None]\n :param kwargs: This is a dictionary of extra parameters for session run method.\n \"\"\"\n if model is None or not isinstance(model, Model):\n raise ValueError('Unknown type passed for optimization.')\n\n if model.is_built_coherence() is Build.NO:\n raise GPflowError('Model is not built.')\n\n session = model.enquire_session(session)\n self._model = model\n optimizer = self.make_optimize_tensor(model, session,\n var_list=var_list, maxiter=maxiter, disp=disp)\n self._optimizer = optimizer\n feed_dict = self._gen_feed_dict(model, feed_dict)\n optimizer.minimize(session=session, feed_dict=feed_dict, step_callback=step_callback,\n **kwargs)\n if anchor:\n model.anchor(session)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompile the GPMC into a dictionary of parameters.", "response": "def compile(self, session=None):\n \"\"\"\n Before calling the standard compile function, check to see if the size\n of the data has changed and add parameters appropriately.\n\n This is necessary because the shape of the parameters depends on the\n shape of the data.\n \"\"\"\n if not self.num_data == self.X.shape[0]:\n self.num_data = self.X.shape[0]\n self.V = Parameter(np.zeros((self.num_data, self.num_latent)))\n self.V.prior = Gaussian(0., 1.)\n\n return super(GPMC, self).compile(session=session)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _build_likelihood(self):\n K = self.kern.K(self.X)\n L = tf.cholesky(\n K + tf.eye(tf.shape(self.X)[0], dtype=settings.float_type) * settings.numerics.jitter_level)\n F = tf.matmul(L, self.V) + self.mean_function(self.X)\n\n return tf.reduce_sum(self.likelihood.logp(F, self.Y))", "response": "r Construct a tf function to compute the likelihood of a general GP\n model."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild the predict function for the new set of entries in the cluster.", "response": "def _build_predict(self, Xnew, full_cov=False):\n \"\"\"\n Xnew is a data matrix, point at which we want to predict\n\n This method computes\n\n p(F* | (F=LV) )\n\n where F* are points on the GP at Xnew, F=LV are points on the GP at X.\n\n \"\"\"\n mu, var = conditional(Xnew, self.X, self.kern, self.V,\n full_cov=full_cov,\n q_sqrt=None, white=True)\n return mu + self.mean_function(Xnew), var"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _init_variational_parameters(self, num_inducing, q_mu, q_sqrt, q_diag):\n q_mu = np.zeros((num_inducing, self.num_latent)) if q_mu is None else q_mu\n self.q_mu = Parameter(q_mu, dtype=settings.float_type) # M x P\n\n if q_sqrt is None:\n if self.q_diag:\n self.q_sqrt = Parameter(np.ones((num_inducing, self.num_latent), dtype=settings.float_type),\n transform=transforms.positive) # M x P\n else:\n q_sqrt = np.array([np.eye(num_inducing, dtype=settings.float_type) for _ in range(self.num_latent)])\n self.q_sqrt = Parameter(q_sqrt, transform=transforms.LowerTriangular(num_inducing, self.num_latent)) # P x M x M\n else:\n if q_diag:\n assert q_sqrt.ndim == 2\n self.num_latent = q_sqrt.shape[1]\n self.q_sqrt = Parameter(q_sqrt, transform=transforms.positive) # M x L/P\n else:\n assert q_sqrt.ndim == 3\n self.num_latent = q_sqrt.shape[0]\n num_inducing = q_sqrt.shape[1]\n self.q_sqrt = Parameter(q_sqrt, transform=transforms.LowerTriangular(num_inducing, self.num_latent))", "response": "Initializes the variational parameters for the current class."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _build_likelihood(self):\n\n # Get prior KL.\n KL = self.build_prior_KL()\n\n # Get conditionals\n fmean, fvar = self._build_predict(self.X, full_cov=False, full_output_cov=False)\n\n # Get variational expectations.\n var_exp = self.likelihood.variational_expectations(fmean, fvar, self.Y)\n\n # re-scale for minibatch size\n scale = tf.cast(self.num_data, settings.float_type) / tf.cast(tf.shape(self.X)[0], settings.float_type)\n\n return tf.reduce_sum(var_exp) * scale - KL", "response": "Build the likelihood for the current model."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compile(self, session=None):\n session = self.enquire_session(session)\n if self.is_built_coherence(session.graph) is Build.NO:\n with session.graph.as_default():\n self.build()\n self.initialize(session, force=True)", "response": "This method compiles the node for the passed session."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninitializing TensorFlow variables which are returned by initializables property and uses feed dictionary returned by initializable_feeds property.", "response": "def initialize(self, session=None, force=False):\n \"\"\"\n Initializes TensorFlow variables, which are returned by `initializables` property and\n uses feed dictionary returned by `initializable_feeds` property defined at ICompilable\n interface and implemented by descendants.\n\n :param session: TensorFlow session used for initializing. In case when session is None,\n default TensorFlow session will be checked first, if session is still None, then\n default GPflowFlow session will used, but there is *no garuantee* that GPflow\n session's graph is compliant with node's tensors graph.\n :param force: inidicates either the initialized TensorFlow variables must be\n re-initialized or not.\n\n :raises: GPflowError exception if session's graph is different from the graph\n used by node tensors.\n \"\"\"\n session = self.enquire_session(session)\n initializables = self.initializables\n if initializables:\n misc.initialize_variables(\n variables=initializables,\n session=session,\n force=force,\n feed_dict=self.initializable_feeds)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clear(self):\n parent = self.parent\n if parent is not self and parent.is_built_coherence(self.graph) is Build.YES:\n raise GPflowError('Clear method cannot be started. Upper nodes are built.')\n self._clear()", "response": "Calls _clear abstract method which must be implemented by descendants."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nverify and returns relevant TensorFlow graph.", "response": "def enquire_graph(self, graph=None):\n \"\"\"\n Verifies and returns relevant TensorFlow graph. If non-None graph were passed,\n the same graph is returned. Otherwise, nodes's graph is exposed and it is\n undefined the default TensorFlow graph is used.\n\n :param graph: TensorFlow graph or None. Default is None.\n :return: TensorFlow graph.\n \"\"\"\n if graph is None:\n graph = self.root.graph if self.graph is None else self.graph\n if graph is None:\n graph = tf.get_default_graph()\n return graph"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nensures and returns relevant TensorFlow session.", "response": "def enquire_session(self, session=None):\n \"\"\"\n Verifies and returns relevant TensorFlow session. If non-None session\n were passed, session is checked for graph compliance and returned back.\n Otherwise, default TensorFlow session is returned. When TensorFlow default\n session is not set up, GPflow session's manager creates or uses existing\n one for returning.\n\n :param session: TensorFlow session or None. Default value is None.\n :return: TensorFlow session.\n :raises GPflowError: Session's graph is not compilable with node's graph.\n \"\"\"\n if session is None:\n session = tf.get_default_session()\n if session is None:\n session = session_manager.get_default_session()\n self.is_built_coherence(session.graph)\n return session"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_built_coherence(self, graph=None):\n graph = self.enquire_graph(graph=graph)\n is_built = self.is_built(graph)\n if is_built is Build.NOT_COMPATIBLE_GRAPH:\n raise GPflowError('Tensor \"{}\" uses different graph.'.format(self.pathname))\n return is_built", "response": "Checks that node was built using input graph."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build(self):\n if self.is_built_coherence() is Build.NO:\n with tf.name_scope(self.tf_name_scope):\n self._build()", "response": "Implementation for ICompilable interface build method. Builds tensors within the current node s name scope using parentable node s name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tf_name_scope(self):\n if self.parent is self:\n leader_name = self.name\n leader_index = self.index\n if leader_index is None:\n return leader_name\n return \"{name}-{index}\".format(name=leader_name, index=leader_index)\n return self.pathname.rsplit('/')[-1]", "response": "Auxilary method for composing the tree name scopes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tf_pathname(self):\n if self.parent is self:\n return self.tf_name_scope\n tail = self.pathname.split('/', 1)[-1]\n leader = self.root.tf_name_scope\n return \"{leader_name}/{tail_name}\".format(leader_name=leader, tail_name=tail)", "response": "Returns the full path name for this tensor at build time."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef autobuild_decorate(func, caller):\n evaldict = dict(_call_=caller, _func_=func)\n fun = AutobuildFunctionMaker.create(\n func, \"return _call_(_func_, %(shortsignature)s)\",\n evaldict, __wrapped__=func)\n if hasattr(func, '__qualname__'):\n fun.__qualname__ = func.__qualname__\n return fun", "response": "Decorates a function using a caller."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _build_prior(self, prior_tensors):\n # TODO(@awav): What prior must represent empty list of parameters?\n if not prior_tensors:\n return tf.constant(0, dtype=settings.float_type)\n return tf.add_n(prior_tensors, name='prior')", "response": "Builds a tf expression for the prior by summing all child - parameter priors."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef make_deprecated_class(oldname, NewClass):\n msg = (\"{module}.{} has been renamed to {module}.{}\"\n .format(oldname, NewClass.__name__, module=NewClass.__module__))\n\n class OldClass(NewClass):\n def __new__(cls, *args, **kwargs):\n raise NotImplementedError(msg)\n OldClass.__doc__ = msg\n OldClass.__qualname__ = OldClass.__name__ = oldname\n return OldClass", "response": "Returns a class that raises NotImplementedError on instantiation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _validate_ard_shape(self, name, value, ARD=None):\n if ARD is None:\n ARD = np.asarray(value).squeeze().shape != ()\n\n if ARD:\n # accept float or array:\n value = value * np.ones(self.input_dim, dtype=settings.float_type)\n\n if self.input_dim == 1 or not ARD:\n correct_shape = ()\n else:\n correct_shape = (self.input_dim,)\n\n if np.asarray(value).squeeze().shape != correct_shape:\n raise ValueError(\"shape of {} does not match input_dim\".format(name))\n\n return value, ARD", "response": "Validates the shape of an ARD hyperparameter."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_separate_dims(self, other_kernel):\n if isinstance(self.active_dims, slice) or isinstance(other_kernel.active_dims, slice):\n # Be very conservative for kernels defined over slices of dimensions\n return False\n\n if np.any(self.active_dims.reshape(-1, 1) == other_kernel.active_dims.reshape(1, -1)):\n return False\n\n return True", "response": "Checks if the kernels defined on different dimensions overlap."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _slice(self, X, X2):\n if isinstance(self.active_dims, slice):\n X = X[..., self.active_dims]\n if X2 is not None:\n X2 = X2[..., self.active_dims]\n else:\n X = tf.gather(X, self.active_dims, axis=-1)\n if X2 is not None:\n X2 = tf.gather(X2, self.active_dims, axis=-1)\n\n input_dim_shape = tf.shape(X)[-1]\n input_dim = tf.convert_to_tensor(self.input_dim, dtype=settings.int_type)\n with tf.control_dependencies([tf.assert_equal(input_dim_shape, input_dim)]):\n X = tf.identity(X)\n\n return X, X2", "response": "Slice the correct dimensions for use in the kernel."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _slice_cov(self, cov):\n cov = tf.cond(tf.equal(tf.rank(cov), 2), lambda: tf.matrix_diag(cov), lambda: cov)\n\n if isinstance(self.active_dims, slice):\n cov = cov[..., self.active_dims, self.active_dims]\n else:\n cov_shape = tf.shape(cov)\n covr = tf.reshape(cov, [-1, cov_shape[-1], cov_shape[-1]])\n gather1 = tf.gather(tf.transpose(covr, [2, 1, 0]), self.active_dims)\n gather2 = tf.gather(tf.transpose(gather1, [1, 0, 2]), self.active_dims)\n cov = tf.reshape(tf.transpose(gather2, [2, 0, 1]),\n tf.concat([cov_shape[:-2], [len(self.active_dims), len(self.active_dims)]], 0))\n return cov", "response": "Slice the correct dimensions for use in the kernel for use in the kernel."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _scaled_square_dist(self, X, X2):\n\n X = X / self.lengthscales\n\n if X2 is None:\n Xs = tf.reduce_sum(tf.square(X), axis=-1, keepdims=True)\n dist = -2 * tf.matmul(X, X, transpose_b=True)\n dist += Xs + tf.matrix_transpose(Xs)\n return dist\n\n Xs = tf.reduce_sum(tf.square(X), axis=-1)\n X2 = X2 / self.lengthscales\n X2s = tf.reduce_sum(tf.square(X2), axis=-1)\n dist = -2 * tf.tensordot(X, X2, [[-1], [-1]])\n dist += _broadcasting_elementwise_op(tf.add, Xs, X2s)\n return dist", "response": "Returns the scaled square distance between two sets of entries."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef scaled_euclid_dist(self, X, X2): # pragma: no cover\n warnings.warn('scaled_euclid_dist is deprecated and will be removed '\n 'in GPflow version 1.4.0. For stationary kernels, '\n 'define K_r(r) instead.',\n DeprecationWarning)\n r2 = self.scaled_square_dist(X, X2)\n return self._clipped_sqrt(r2)", "response": "Returns the Euclidean distance between X and X2."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef K(self, X, X2=None, presliced=False):\n if not presliced:\n X, X2 = self._slice(X, X2)\n return self.K_r2(self.scaled_square_dist(X, X2))", "response": "Calculates the kernel matrix for a set of stationary kernels."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the kernel evaluated on r2 which is the scaled squared distance.", "response": "def K_r2(self, r2):\n \"\"\"\n Returns the kernel evaluated on `r2`, which is the scaled squared distance.\n Will call self.K_r(r=sqrt(r2)), or can be overwritten directly (and should operate element-wise on r2).\n \"\"\"\n r = self._clipped_sqrt(r2)\n return self.K_r(r)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the Jacobian of the object in the order dependent family of functions defined in equations 4 to 7 in the reference paper.", "response": "def _J(self, theta):\n \"\"\"\n Implements the order dependent family of functions defined in equations\n 4 to 7 in the reference paper.\n \"\"\"\n if self.order == 0:\n return np.pi - theta\n elif self.order == 1:\n return tf.sin(theta) + (np.pi - theta) * tf.cos(theta)\n elif self.order == 2:\n return 3. * tf.sin(theta) * tf.cos(theta) + \\\n (np.pi - theta) * (1. + 2. * tf.cos(theta) ** 2)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking whether the kernels in the combination act on disjoint subsets of dimensions.", "response": "def on_separate_dimensions(self):\n \"\"\"\n Checks whether the kernels in the combination act on disjoint subsets\n of dimensions. Currently, it is hard to asses whether two slice objects\n will overlap, so this will always return False.\n :return: Boolean indicator.\n \"\"\"\n if np.any([isinstance(k.active_dims, slice) for k in self.kernels]):\n # Be conservative in the case of a slice object\n return False\n else:\n dimlist = [k.active_dims for k in self.kernels]\n overlapping = False\n for i, dims_i in enumerate(dimlist):\n for dims_j in dimlist[i + 1:]:\n if np.any(dims_i.reshape(-1, 1) == dims_j.reshape(1, -1)):\n overlapping = True\n return not overlapping"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntransforming from the packed to unpacked representations ( numpy ) N", "response": "def forward(self, x):\n \"\"\"\n Transforms from the packed to unpacked representations (numpy)\n \n :param x: packed numpy array. Must have shape `self.num_matrices x triangular_number\n :return: Reconstructed numpy array y of shape self.num_matrices x N x N\n \"\"\"\n fwd = np.zeros((self.num_matrices, self.N, self.N), settings.float_type)\n indices = np.tril_indices(self.N, 0)\n z = np.zeros(len(indices[0])).astype(int)\n for i in range(self.num_matrices):\n fwd[(z + i,) + indices] = x[i, :]\n return fwd.squeeze(axis=0) if self.squeeze else fwd"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef backward(self, y):\n if self.squeeze:\n y = y[None, :, :]\n ind = np.tril_indices(self.N)\n return np.vstack([y_i[ind] for y_i in y])", "response": "Transforms a series of triangular matrices y to the packed representation x"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef forward_tensor(self, x):\n fwd = vec_to_tri(x, self.N)\n return tf.squeeze(fwd, axis=0) if self.squeeze else fwd", "response": "Transforms from the packed representation of N x N to unpacked representations of N x N"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef backward_tensor(self, y):\n if self.squeeze:\n y = tf.expand_dims(y, axis=0)\n indices = np.vstack(np.tril_indices(self.N)).T\n indices = itertools.product(np.arange(self.num_matrices), indices)\n indices = np.array([np.hstack(x) for x in indices])\n triangular = tf.gather_nd(y, indices)\n return tf.reshape(triangular, [self.num_matrices, (self.N**2 + self.N) // 2])", "response": "Transforms a series of triangular matrices y to the packed representation x."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextracting a dictionary with tensors specified by name from an object existing dictionary with tensors specified by name.", "response": "def get_autoflow(cls, obj, name):\n \"\"\"\n Extracts from an object existing dictionary with tensors specified by name.\n If there is no such object then new one will be created. Intenally, it appends\n autoflow prefix to the name and saves it as an attribute.\n\n :param obj: target GPflow object.\n :param name: unique part of autoflow attribute's name.\n\n :raises: ValueError exception if `name` is not a string.\n \"\"\"\n if not isinstance(name, str):\n raise ValueError('Name must be string.')\n prefix = cls.__autoflow_prefix__\n autoflow_name = prefix + name\n store = misc.get_attribute(obj, autoflow_name, allow_fail=True, default={})\n if not store:\n setattr(obj, autoflow_name, store)\n return store"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef clear_autoflow(cls, obj, name=None):\n if name is not None and not isinstance(name, str):\n raise ValueError('Name must be a string.')\n prefix = cls.__autoflow_prefix__\n if name:\n prefix = \"\" if name.startswith(prefix) else prefix\n delattr(obj, prefix + name)\n else:\n keys = [attr for attr in obj.__dict__ if attr.startswith(prefix)]\n for key in keys:\n delattr(obj, key)", "response": "Clears autoflow s tensor storage."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncompiling the VGP into a single parameter vector.", "response": "def compile(self, session=None):\n \"\"\"\n Before calling the standard compile function, check to see if the size\n of the data has changed and add variational parameters appropriately.\n\n This is necessary because the shape of the parameters depends on the\n shape of the data.\n \"\"\"\n if not self.num_data == self.X.shape[0]:\n self.num_data = self.X.shape[0]\n self.q_mu = Parameter(np.zeros((self.num_data, self.num_latent)))\n self.q_sqrt = Parameter(np.eye(self.num_data)[:, :, None] *\n np.ones((1, 1, self.num_latent)))\n\n return super(VGP, self).compile(session=session)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compile(self, session=None):\n if not self.num_data == self.X.shape[0]:\n self.num_data = self.X.shape[0]\n self.q_alpha = Parameter(np.zeros((self.num_data, self.num_latent)))\n self.q_lambda = Parameter(np.ones((self.num_data, self.num_latent)),\n transforms.positive)\n return super(VGP_opper_archambeau, self).compile(session=session)", "response": "Compile the VGP_opper_archambeau object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nbuild predict and variance of the new object.", "response": "def _build_predict(self, Xnew, full_cov=False):\n \"\"\"\n The posterior variance of F is given by\n q(f) = N(f | K alpha + mean, [K^-1 + diag(lambda**2)]^-1)\n Here we project this to F*, the values of the GP at Xnew which is given\n by\n q(F*) = N ( F* | K_{*F} alpha + mean, K_{**} - K_{*f}[K_{ff} +\n diag(lambda**-2)]^-1 K_{f*} )\n \"\"\"\n\n # compute kernel things\n Kx = self.kern.K(self.X, Xnew)\n K = self.kern.K(self.X)\n\n # predictive mean\n f_mean = tf.matmul(Kx, self.q_alpha, transpose_a=True) + self.mean_function(Xnew)\n\n # predictive var\n A = K + tf.matrix_diag(tf.transpose(1. / tf.square(self.q_lambda)))\n L = tf.cholesky(A)\n Kx_tiled = tf.tile(tf.expand_dims(Kx, 0), [self.num_latent, 1, 1])\n LiKx = tf.matrix_triangular_solve(L, Kx_tiled)\n if full_cov:\n f_var = self.kern.K(Xnew) - tf.matmul(LiKx, LiKx, transpose_a=True)\n else:\n f_var = self.kern.Kdiag(Xnew) - tf.reduce_sum(tf.square(LiKx), 1)\n return f_mean, tf.transpose(f_var)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _build_likelihood(self):\n # get the (marginals of) q(f): exactly predicting!\n fmean, fvar = self._build_predict(self.X, full_cov=False)\n return tf.reduce_sum(self.likelihood.variational_expectations(fmean, fvar, self.Y))", "response": "This function computes the optimal density for v up to a constant\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse(string):\n if not isinstance(string, str):\n raise ValueError('Config value \"{0}\" expected to be string.'\n .format(string))\n if string in ['true', 'True']:\n return True\n elif string in ['false', 'False']:\n return False\n elif string in ['float64', 'float32', 'float16',\n 'int64', 'int32', 'int16']:\n return getattr(np, string)\n else:\n try:\n return int(string)\n except ValueError:\n pass\n try:\n return float(string)\n except ValueError:\n return string", "response": "Parses a string into a base64 - encoded version of the attribute."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntake a dictionary and returns a nested tuple of named tuples.", "response": "def _namedtuplify(mapping):\n \"\"\"\n Make the dictionary into a nested series of named tuples.\n This is what allows accessing by attribute: settings.numerics.jitter\n Thank you https://gist.github.com/hangtwenty/5960435\n \"\"\"\n if isinstance(mapping, collections.Mapping):\n for key, value in list(mapping.items()):\n mapping[key] = _namedtuplify(value)\n try:\n mapping.pop('__name__')\n except KeyError:\n pass\n # return collections.namedtuple('settingsa', dict(**mapping))(**mapping)\n return _MutableNamedTuple(mapping)\n return _parse(mapping)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_config_file(path=None):\n cfg = configparser.ConfigParser()\n\n if path is None: # pragma: no cover\n dirs = [os.curdir, os.path.expanduser('~'),\n os.path.dirname(os.path.realpath(__file__))]\n locations = map(os.path.abspath, dirs)\n for loc in locations:\n if cfg.read(os.path.join(loc, 'gpflowrc')):\n break\n if cfg.read(os.path.join(loc, '.gpflowrc')):\n break\n else:\n if not cfg.read(path):\n raise RuntimeError(\"Config at '{0}' cannot be read\".format(path))\n return cfg", "response": "Reads config file.\n First look for config file in the current directory, then in the\n user's home directory, then in the same directory as this file.\n Tries to find config file both with and without preceeding 'dot'\n for hidden files (prefer non-hidden)."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the evaluation locations x and weights wn for a multivariate Gauss - Hermite quadrature.", "response": "def mvhermgauss(H: int, D: int):\n \"\"\"\n Return the evaluation locations 'xn', and weights 'wn' for a multivariate\n Gauss-Hermite quadrature.\n\n The outputs can be used to approximate the following type of integral:\n int exp(-x)*f(x) dx ~ sum_i w[i,:]*f(x[i,:])\n\n :param H: Number of Gauss-Hermite evaluation points.\n :param D: Number of input dimensions. Needs to be known at call-time.\n :return: eval_locations 'x' (H**DxD), weights 'w' (H**D)\n \"\"\"\n gh_x, gh_w = hermgauss(H)\n x = np.array(list(itertools.product(*(gh_x,) * D))) # H**DxD\n w = np.prod(np.array(list(itertools.product(*(gh_w,) * D))), 1) # H**D\n return x, w"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes N Gaussian expectation integrals of a single function f.", "response": "def mvnquad(func, means, covs, H: int, Din: int=None, Dout=None):\n \"\"\"\n Computes N Gaussian expectation integrals of a single function 'f'\n using Gauss-Hermite quadrature.\n :param f: integrand function. Takes one input of shape ?xD.\n :param means: NxD\n :param covs: NxDxD\n :param H: Number of Gauss-Hermite evaluation points.\n :param Din: Number of input dimensions. Needs to be known at call-time.\n :param Dout: Number of output dimensions. Defaults to (). Dout is assumed\n to leave out the item index, i.e. f actually maps (?xD)->(?x*Dout).\n :return: quadratures (N,*Dout)\n \"\"\"\n # Figure out input shape information\n if Din is None:\n Din = means.shape[1] if type(means.shape) is tuple else means.shape[1].value\n\n if Din is None:\n raise GPflowError(\"If `Din` is passed as `None`, `means` must have a known shape. \"\n \"Running mvnquad in `autoflow` without specifying `Din` and `Dout` \"\n \"is problematic. Consider using your own session.\") # pragma: no cover\n\n xn, wn = mvhermgauss(H, Din)\n N = tf.shape(means)[0]\n\n # transform points based on Gaussian parameters\n cholXcov = tf.cholesky(covs) # NxDxD\n Xt = tf.matmul(cholXcov, tf.tile(xn[None, :, :], (N, 1, 1)), transpose_b=True) # NxDxH**D\n X = 2.0 ** 0.5 * Xt + tf.expand_dims(means, 2) # NxDxH**D\n Xr = tf.reshape(tf.transpose(X, [2, 0, 1]), (-1, Din)) # (H**D*N)xD\n\n # perform quadrature\n fevals = func(Xr)\n if Dout is None:\n Dout = tuple((d if type(d) is int else d.value) for d in fevals.shape[1:])\n\n if any([d is None for d in Dout]):\n raise GPflowError(\"If `Dout` is passed as `None`, the output of `func` must have known \"\n \"shape. Running mvnquad in `autoflow` without specifying `Din` and `Dout` \"\n \"is problematic. Consider using your own session.\") # pragma: no cover\n fX = tf.reshape(fevals, (H ** Din, N,) + Dout)\n wr = np.reshape(wn * np.pi ** (-Din * 0.5),\n (-1,) + (1,) * (1 + len(Dout)))\n return tf.reduce_sum(fX * wr, 0)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ndiagquad(funcs, H: int, Fmu, Fvar, logspace: bool=False, **Ys):\n def unify(f_list):\n \"\"\"\n Stack a list of means/vars into a full block\n \"\"\"\n return tf.reshape(\n tf.concat([tf.reshape(f, (-1, 1)) for f in f_list], axis=1),\n (-1, 1, Din))\n\n if isinstance(Fmu, (tuple, list)):\n Din = len(Fmu)\n shape = tf.shape(Fmu[0])\n Fmu, Fvar = map(unify, [Fmu, Fvar]) # both N x 1 x Din\n else:\n Din = 1\n shape = tf.shape(Fmu)\n Fmu, Fvar = [tf.reshape(f, (-1, 1, 1)) for f in [Fmu, Fvar]]\n\n xn, wn = mvhermgauss(H, Din)\n # xn: H**Din x Din, wn: H**Din\n\n gh_x = xn.reshape(1, -1, Din) # 1 x H**Din x Din\n Xall = gh_x * tf.sqrt(2.0 * Fvar) + Fmu # N x H**Din x Din\n Xs = [Xall[:, :, i] for i in range(Din)] # N x H**Din each\n\n gh_w = wn * np.pi ** (-0.5 * Din) # H**Din x 1\n\n for name, Y in Ys.items():\n Y = tf.reshape(Y, (-1, 1))\n Y = tf.tile(Y, [1, H**Din]) # broadcast Y to match X\n # without the tiling, some calls such as tf.where() (in bernoulli) fail\n Ys[name] = Y # now N x H**Din\n\n def eval_func(f):\n feval = f(*Xs, **Ys) # f should be elementwise: return shape N x H**Din\n if logspace:\n log_gh_w = np.log(gh_w.reshape(1, -1))\n result = tf.reduce_logsumexp(feval + log_gh_w, axis=1)\n else:\n result = tf.matmul(feval, gh_w.reshape(-1, 1))\n return tf.reshape(result, shape)\n\n if isinstance(funcs, Iterable):\n return [eval_func(f) for f in funcs]\n else:\n return eval_func(funcs)", "response": "This function calculates the N Gaussian expectation integrals of one or more functions and returns the shape of the resulting array."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ndiag_mc(funcs, S: int, Fmu, Fvar, logspace: bool=False, epsilon=None, **Ys):\n N, D = tf.shape(Fmu)[0], tf.shape(Fvar)[1]\n\n if epsilon is None:\n epsilon = tf.random_normal((S, N, D), dtype=settings.float_type)\n\n mc_x = Fmu[None, :, :] + tf.sqrt(Fvar[None, :, :]) * epsilon\n mc_Xr = tf.reshape(mc_x, (S * N, D))\n\n for name, Y in Ys.items():\n D_out = tf.shape(Y)[1]\n # we can't rely on broadcasting and need tiling\n mc_Yr = tf.tile(Y[None, ...], [S, 1, 1]) # S x N x D_out\n Ys[name] = tf.reshape(mc_Yr, (S * N, D_out)) # S * N x D_out\n\n def eval_func(func):\n feval = func(mc_Xr, **Ys)\n feval = tf.reshape(feval, (S, N, -1))\n if logspace:\n log_S = tf.log(tf.cast(S, settings.float_type))\n return tf.reduce_logsumexp(feval, axis=0) - log_S # N x D\n else:\n return tf.reduce_mean(feval, axis=0)\n\n if isinstance(funcs, Iterable):\n return [eval_func(f) for f in funcs]\n else:\n return eval_func(funcs)", "response": "Computes N Gaussian expectation integrals of one or more functions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef params_as_tensors(method):\n @functools.wraps(method)\n def tensor_mode_wrapper(obj, *args, **kwargs):\n if not isinstance(obj, Parameterized):\n raise GPflowError(\n 'Tensor mode works only for Parameterized object.')\n prev_value = _params_as_tensors_enter(obj, True)\n try:\n result = method(obj, *args, **kwargs)\n finally:\n _params_as_tensors_exit(obj, prev_value)\n return result\n return tensor_mode_wrapper", "response": "A function decorator that converts representation for parameters into\n their unconstrained tensors and data holders to their data holders."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the KL divergence for a single set of independent distributions.", "response": "def gauss_kl(q_mu, q_sqrt, K=None, *, K_cholesky=None):\n \"\"\"\n Compute the KL divergence KL[q || p] between\n\n q(x) = N(q_mu, q_sqrt^2)\n and\n p(x) = N(0, K)\n\n We assume N multiple independent distributions, given by the columns of\n q_mu and the last dimension of q_sqrt. Returns the sum of the divergences.\n\n q_mu is a matrix [M, L], each column contains a mean.\n\n q_sqrt can be a 3D tensor [L, M, M], each matrix within is a lower\n triangular square-root matrix of the covariance of q.\n q_sqrt can be a matrix [M, L], each column represents the diagonal of a\n square-root matrix of the covariance of q.\n\n K is the covariance of p, [M, M] or [L, M, M]\n K_cholesky is the cholesky of the covariance of p, [M, M] or [L, M, M]\n \n Note: if no K matrix is given (both `K` and `K_cholesky` are None),\n `gauss_kl` computes the KL divergence from p(x) = N(0, I) instead.\n The K matrix can be passed either directly as `K`, or as its Cholesky factor, `K_cholesky`. \n In either case, it can be a single matrix [M, M], in which case the sum of the L KL divergences \n is computed by broadcasting, or L different covariances [L, M, M].\n \"\"\"\n\n if (K is not None) and (K_cholesky is not None):\n raise ValueError(\"Ambiguous arguments: gauss_kl() must only \"\n \"be passed one of `K` or `K_cholesky`.\")\n\n white = (K is None) and (K_cholesky is None)\n diag = q_sqrt.get_shape().ndims == 2\n\n M, B = tf.shape(q_mu)[0], tf.shape(q_mu)[1]\n\n if white:\n alpha = q_mu # [M, B]\n else:\n if K is not None:\n Lp = tf.cholesky(K) # [B, M, M] or [M, M]\n elif K_cholesky is not None:\n Lp = K_cholesky # [B, M, M] or [M, M]\n\n batched = Lp.get_shape().ndims == 3\n q_mu = tf.transpose(q_mu)[:, :, None] if batched else q_mu # [B, M, 1] or [M, B]\n alpha = tf.matrix_triangular_solve(Lp, q_mu, lower=True) # [B, M, 1] or [M, B]\n\n if diag:\n Lq = Lq_diag = q_sqrt\n Lq_full = tf.matrix_diag(tf.transpose(q_sqrt)) # [B, M, M]\n else:\n Lq = Lq_full = tf.matrix_band_part(q_sqrt, -1, 0) # force lower triangle # [B, M, M]\n Lq_diag = tf.matrix_diag_part(Lq) # [M, B]\n\n # Mahalanobis term: \u03bcq\u1d40 \u03a3p\u207b\u00b9 \u03bcq\n mahalanobis = tf.reduce_sum(tf.square(alpha))\n\n # Constant term: - B * M\n constant = - tf.cast(tf.size(q_mu, out_type=tf.int64), dtype=settings.float_type)\n\n # Log-determinant of the covariance of q(x):\n logdet_qcov = tf.reduce_sum(tf.log(tf.square(Lq_diag)))\n\n # Trace term: tr(\u03a3p\u207b\u00b9 \u03a3q)\n if white:\n trace = tf.reduce_sum(tf.square(Lq))\n else:\n if diag and not batched:\n # K is [M, M] and q_sqrt is [M, B]: fast specialisation\n LpT = tf.transpose(Lp) # [M, M]\n Lp_inv = tf.matrix_triangular_solve(Lp, tf.eye(M, dtype=settings.float_type),lower=True) # [M, M]\n K_inv = tf.matrix_diag_part(tf.matrix_triangular_solve(LpT, Lp_inv, lower=False))[:, None] # [M, M] -> [M, 1]\n trace = tf.reduce_sum(K_inv * tf.square(q_sqrt))\n else:\n # TODO: broadcast instead of tile when tf allows (not implemented in tf <= 1.12)\n Lp_full = Lp if batched else tf.tile(tf.expand_dims(Lp, 0), [B, 1, 1])\n LpiLq = tf.matrix_triangular_solve(Lp_full, Lq_full, lower=True)\n trace = tf.reduce_sum(tf.square(LpiLq))\n\n twoKL = mahalanobis + constant - logdet_qcov + trace\n\n # Log-determinant of the covariance of p(x):\n if not white:\n log_sqdiag_Lp = tf.log(tf.square(tf.matrix_diag_part(Lp)))\n sum_log_sqdiag_Lp = tf.reduce_sum(log_sqdiag_Lp)\n # If K is B x M x M, num_latent is no longer implicit, no need to multiply the single kernel logdet\n scale = 1.0 if batched else tf.cast(B, settings.float_type)\n twoKL += scale * sum_log_sqdiag_Lp\n\n return 0.5 * twoKL"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nbuild a tensorflow function to compute the bound on the marginallikelihood.", "response": "def _build_likelihood(self):\n \"\"\"\n Construct a tensorflow function to compute the bound on the marginal\n likelihood.\n \"\"\"\n pX = DiagonalGaussian(self.X_mean, self.X_var)\n\n num_inducing = len(self.feature)\n psi0 = tf.reduce_sum(expectation(pX, self.kern))\n psi1 = expectation(pX, (self.kern, self.feature))\n psi2 = tf.reduce_sum(expectation(pX, (self.kern, self.feature), (self.kern, self.feature)), axis=0)\n Kuu = features.Kuu(self.feature, self.kern, jitter=settings.jitter)\n L = tf.cholesky(Kuu)\n sigma2 = self.likelihood.variance\n sigma = tf.sqrt(sigma2)\n\n # Compute intermediate matrices\n A = tf.matrix_triangular_solve(L, tf.transpose(psi1), lower=True) / sigma\n tmp = tf.matrix_triangular_solve(L, psi2, lower=True)\n AAT = tf.matrix_triangular_solve(L, tf.transpose(tmp), lower=True) / sigma2\n B = AAT + tf.eye(num_inducing, dtype=settings.float_type)\n LB = tf.cholesky(B)\n log_det_B = 2. * tf.reduce_sum(tf.log(tf.matrix_diag_part(LB)))\n c = tf.matrix_triangular_solve(LB, tf.matmul(A, self.Y), lower=True) / sigma\n\n # KL[q(x) || p(x)]\n dX_var = self.X_var if len(self.X_var.get_shape()) == 2 else tf.matrix_diag_part(self.X_var)\n NQ = tf.cast(tf.size(self.X_mean), settings.float_type)\n D = tf.cast(tf.shape(self.Y)[1], settings.float_type)\n KL = -0.5 * tf.reduce_sum(tf.log(dX_var)) \\\n + 0.5 * tf.reduce_sum(tf.log(self.X_prior_var)) \\\n - 0.5 * NQ \\\n + 0.5 * tf.reduce_sum((tf.square(self.X_mean - self.X_prior_mean) + dX_var) / self.X_prior_var)\n\n # compute log marginal bound\n ND = tf.cast(tf.size(self.Y), settings.float_type)\n bound = -0.5 * ND * tf.log(2 * np.pi * sigma2)\n bound += -0.5 * D * log_det_B\n bound += -0.5 * tf.reduce_sum(tf.square(self.Y)) / sigma2\n bound += 0.5 * tf.reduce_sum(tf.square(c))\n bound += -0.5 * D * (tf.reduce_sum(psi0) / sigma2 -\n tf.reduce_sum(tf.matrix_diag_part(AAT)))\n bound -= KL\n return bound"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _build_predict(self, Xnew, full_cov=False):\n pX = DiagonalGaussian(self.X_mean, self.X_var)\n\n num_inducing = len(self.feature)\n psi1 = expectation(pX, (self.kern, self.feature))\n psi2 = tf.reduce_sum(expectation(pX, (self.kern, self.feature), (self.kern, self.feature)), axis=0)\n Kuu = features.Kuu(self.feature, self.kern, jitter=settings.numerics.jitter_level)\n Kus = features.Kuf(self.feature, self.kern, Xnew)\n sigma2 = self.likelihood.variance\n sigma = tf.sqrt(sigma2)\n L = tf.cholesky(Kuu)\n\n A = tf.matrix_triangular_solve(L, tf.transpose(psi1), lower=True) / sigma\n tmp = tf.matrix_triangular_solve(L, psi2, lower=True)\n AAT = tf.matrix_triangular_solve(L, tf.transpose(tmp), lower=True) / sigma2\n B = AAT + tf.eye(num_inducing, dtype=settings.float_type)\n LB = tf.cholesky(B)\n c = tf.matrix_triangular_solve(LB, tf.matmul(A, self.Y), lower=True) / sigma\n tmp1 = tf.matrix_triangular_solve(L, Kus, lower=True)\n tmp2 = tf.matrix_triangular_solve(LB, tmp1, lower=True)\n mean = tf.matmul(tmp2, c, transpose_a=True)\n if full_cov:\n var = self.kern.K(Xnew) + tf.matmul(tmp2, tmp2, transpose_a=True) \\\n - tf.matmul(tmp1, tmp1, transpose_a=True)\n shape = tf.stack([1, 1, tf.shape(self.Y)[1]])\n var = tf.tile(tf.expand_dims(var, 2), shape)\n else:\n var = self.kern.Kdiag(Xnew) + tf.reduce_sum(tf.square(tmp2), 0) \\\n - tf.reduce_sum(tf.square(tmp1), 0)\n shape = tf.stack([1, tf.shape(self.Y)[1]])\n var = tf.tile(tf.expand_dims(var, 1), shape)\n return mean + self.mean_function(Xnew), var", "response": "Compute the mean and variance of the latent function at some new points."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a single - output conditional for a single - output case.", "response": "def _conditional(Xnew, feat, kern, f, *, full_cov=False, full_output_cov=False, q_sqrt=None, white=False):\n \"\"\"\n Multi-output GP with independent GP priors.\n Number of latent processes equals the number of outputs (L = P).\n\n The covariance matrices used to calculate the conditional have the following shape:\n - Kuu: P x M x M\n - Kuf: P x M x N\n - Kff: P x N or P x N x N\n\n Further reference\n -----------------\n - See `gpflow.conditionals._conditional` for a detailed explanation of\n conditional in the single-output case.\n - See the multiouput notebook for more information about the multiouput framework.\n - See above for the parameters and the return value.\n \"\"\"\n\n logger.debug(\"conditional: object, SharedIndependentMof, SeparateIndependentMok, object\")\n # Following are: P x M x M - P x M x N - P x N(x N)\n Kmms = Kuu(feat, kern, jitter=settings.numerics.jitter_level) # P x M x M\n Kmns = Kuf(feat, kern, Xnew) # P x M x N\n kern_list = kern.kernels if isinstance(kern, Combination) else [kern.kern] * len(feat.feat_list)\n Knns = tf.stack([k.K(Xnew) if full_cov else k.Kdiag(Xnew) for k in kern_list], axis=0)\n fs = tf.transpose(f)[:, :, None] # P x M x 1\n # P x 1 x M x M or P x M x 1\n q_sqrts = tf.transpose(q_sqrt)[:, :, None] if q_sqrt.shape.ndims == 2 else q_sqrt[:, None, :, :]\n\n def single_gp_conditional(t):\n Kmm, Kmn, Knn, f, q_sqrt = t\n return base_conditional(Kmn, Kmm, Knn, f, full_cov=full_cov, q_sqrt=q_sqrt, white=white)\n\n rmu, rvar = tf.map_fn(single_gp_conditional,\n (Kmms, Kmns, Knns, fs, q_sqrts),\n (settings.float_type, settings.float_type)) # P x N x 1, P x 1 x N x N or P x N x 1\n\n fmu = _rollaxis_left(rmu[..., 0], 1) # N x P\n\n if full_cov:\n fvar = rvar[..., 0, :, :] # P x N x N\n else:\n fvar = _rollaxis_left(rvar[..., 0], 1) # N x P\n\n return fmu, _expand_independent_outputs(fvar, full_cov, full_output_cov)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsamples from a conditional distribution.", "response": "def _sample_conditional(Xnew, feat, kern, f, *, full_cov=False, full_output_cov=False, q_sqrt=None, white=False, num_samples=None):\n \"\"\"\n `sample_conditional` will return a sample from the conditinoal distribution.\n In most cases this means calculating the conditional mean m and variance v and then\n returning m + sqrt(v) * eps, with eps ~ N(0, 1).\n However, for some combinations of Mok and Mof more efficient sampling routines exists.\n The dispatcher will make sure that we use the most efficent one.\n\n :return: N x P (full_output_cov = False) or N x P x P (full_output_cov = True)\n \"\"\"\n logger.debug(\"sample conditional: (MixedKernelSharedMof, MixedKernelSeparateMof), SeparateMixedMok\")\n if full_cov:\n raise NotImplementedError(\"full_cov not yet implemented\")\n if full_output_cov:\n raise NotImplementedError(\"full_output_cov not yet implemented\")\n independent_cond = conditional.dispatch(object, SeparateIndependentMof, SeparateIndependentMok, object)\n g_mu, g_var = independent_cond(Xnew, feat, kern, f, white=white, q_sqrt=q_sqrt,\n full_output_cov=False, full_cov=False) # [..., N, L], [..., N, L]\n g_sample = _sample_mvn(g_mu, g_var, \"diag\", num_samples=num_samples) # [..., (S), N, L]\n with params_as_tensors_for(kern):\n f_mu, f_var = _mix_latent_gp(kern.W, g_mu, g_var, full_cov, full_output_cov)\n f_sample = tf.tensordot(g_sample, kern.W, [[-1], [-1]]) # [..., N, P]\n return f_sample, f_mu, f_var"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _mix_latent_gp(W, g_mu, g_var, full_cov, full_output_cov):\n f_mu = tf.tensordot(g_mu, W, [[-1], [-1]]) # [..., N, P]\n\n rk = tf.rank(g_var)\n leading_dims = (rk - 3) if full_cov else (rk - 2)\n\n if full_cov and full_output_cov: # g_var is [L, ..., N, N]\n # this branch is practically never taken\n g_var = _rollaxis_left(g_var, 1) # [..., N, N, L]\n g_var = tf.expand_dims(g_var, axis=-2) # [..., N, N, 1, L]\n g_var_W = g_var * W # [..., N, P, L]\n f_var = tf.tensordot(g_var_W, W, [[-1], [-1]]) # [..., N, N, P, P]\n f_var = misc.leading_transpose(f_var, [..., -4, -2, -3, -1]) # [..., N, P, N, P]\n\n elif full_cov and not full_output_cov: # g_var is [L, ..., N, N]\n # this branch is practically never taken\n f_var = tf.tensordot(g_var, W**2, [[0], [-1]]) # [..., N, N, P]\n f_var = misc.leading_transpose(f_var, [..., -1, -3, -2]) # [..., P, N, N]\n\n elif not full_cov and full_output_cov: # g_var is [..., N, L]\n g_var = tf.expand_dims(g_var, axis=-2) # [..., N, 1, L]\n g_var_W = g_var * W # [..., N, P, L]\n f_var = tf.tensordot(g_var_W, W, [[-1], [-1]]) # [..., N, P, P]\n\n elif not full_cov and not full_output_cov: # g_var is [..., N, L]\n W_squared = W**2 # [P, L]\n f_var = tf.tensordot(g_var, W_squared, [[-1], [-1]]) # [..., N, P]\n\n return f_mu, f_var", "response": "r Makes the mean and variance of an uncorrelated L - dimensional latent GP\n and returns the mean and variance of the mixed GP."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef size(self):\n return np.multiply.reduce(self.shape, dtype=np.int32)", "response": "The size of this parameter equivalent to self. value. size"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tf_compilation_index(self):\n if self.parameter_tensor is None:\n return None\n name = self.parameter_tensor.name\n return name.split('-', 1)[-1].split('/')[0]", "response": "Takes out index from the parameter s tensor name. E. g. parameter tensor name is \n GPR - 0 - 0 - lengthcales"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _build_prior(self, unconstrained_tensor, constrained_tensor):\n if not misc.is_tensor(unconstrained_tensor):\n raise GPflowError(\"Unconstrained input must be a tensor.\")\n\n if not misc.is_tensor(constrained_tensor):\n raise GPflowError(\"Constrained input must be a tensor.\")\n\n prior_name = 'prior'\n\n if self.prior is None:\n return tf.constant(0.0, settings.float_type, name=prior_name)\n\n log_jacobian = self.transform.log_jacobian_tensor(unconstrained_tensor)\n logp_var = self.prior.logp(constrained_tensor)\n return tf.squeeze(tf.add(logp_var, log_jacobian, name=prior_name))", "response": "Builds a tensorflow representation of the prior density."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nminimize a scalar Tensor.", "response": "def minimize(self,\n session=None,\n feed_dict=None,\n fetches=None,\n step_callback=None,\n loss_callback=None,\n **run_kwargs):\n \"\"\"Minimize a scalar `Tensor`.\n\n Variables subject to optimization are updated in-place at the end of\n optimization.\n\n Note that this method does *not* just return a minimization `Op`, unlike\n `Optimizer.minimize()`; instead it actually performs minimization by\n executing commands to control a `Session`.\n\n Args:\n session: A `Session` instance.\n feed_dict: A feed dict to be passed to calls to `session.run`.\n fetches: A list of `Tensor`s to fetch and supply to `loss_callback`\n as positional arguments.\n step_callback: A function to be called at each optimization step;\n arguments are the current values of all optimization variables\n flattened into a single vector.\n loss_callback: A function to be called every time the loss and gradients\n are computed, with evaluated fetches supplied as positional arguments.\n **run_kwargs: kwargs to pass to `session.run`.\n \"\"\"\n session = session or ops.get_default_session()\n feed_dict = feed_dict or {}\n fetches = fetches or []\n\n loss_callback = loss_callback or (lambda *fetches: None)\n step_callback = step_callback or (lambda xk: None)\n\n self._initialize_updated_shapes(session)\n\n # Construct loss function and associated gradient.\n loss_grad_func = self._make_eval_func([self._loss,\n self._packed_loss_grad], session,\n feed_dict, fetches, loss_callback)\n\n # Construct equality constraint functions and associated gradients.\n equality_funcs = self._make_eval_funcs(self._equalities, session, feed_dict,\n fetches)\n equality_grad_funcs = self._make_eval_funcs(self._packed_equality_grads,\n session, feed_dict, fetches)\n\n # Construct inequality constraint functions and associated gradients.\n inequality_funcs = self._make_eval_funcs(self._inequalities, session,\n feed_dict, fetches)\n inequality_grad_funcs = self._make_eval_funcs(self._packed_inequality_grads,\n session, feed_dict, fetches)\n\n # Get initial value from TF session.\n initial_packed_var_val = session.run(self._packed_var)\n\n # Perform minimization.\n packed_var_val = self._minimize(\n initial_val=initial_packed_var_val,\n loss_grad_func=loss_grad_func,\n equality_funcs=equality_funcs,\n equality_grad_funcs=equality_grad_funcs,\n inequality_funcs=inequality_funcs,\n inequality_grad_funcs=inequality_grad_funcs,\n packed_bounds=self._packed_bounds,\n step_callback=step_callback,\n optimizer_kwargs=self.optimizer_kwargs)\n var_vals = [\n packed_var_val[packing_slice] for packing_slice in self._packing_slices\n ]\n\n # Set optimization variables to their new values.\n session.run(\n self._var_updates,\n feed_dict=dict(zip(self._update_placeholders, var_vals)),\n **run_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _minimize(self, initial_val, loss_grad_func, equality_funcs,\n equality_grad_funcs, inequality_funcs, inequality_grad_funcs,\n packed_bounds, step_callback, optimizer_kwargs):\n \"\"\"Wrapper for a particular optimization algorithm implementation.\n\n It would be appropriate for a subclass implementation of this method to\n raise `NotImplementedError` if unsupported arguments are passed: e.g. if an\n algorithm does not support constraints but `len(equality_funcs) > 0`.\n\n Args:\n initial_val: A NumPy vector of initial values.\n loss_grad_func: A function accepting a NumPy packed variable vector and\n returning two outputs, a loss value and the gradient of that loss with\n respect to the packed variable vector.\n equality_funcs: A list of functions each of which specifies a scalar\n quantity that an optimizer should hold exactly zero.\n equality_grad_funcs: A list of gradients of equality_funcs.\n inequality_funcs: A list of functions each of which specifies a scalar\n quantity that an optimizer should hold >= 0.\n inequality_grad_funcs: A list of gradients of inequality_funcs.\n packed_bounds: A list of bounds for each index, or `None`.\n step_callback: A callback function to execute at each optimization step,\n supplied with the current value of the packed variable vector.\n optimizer_kwargs: Other key-value arguments available to the optimizer.\n\n Returns:\n The optimal variable vector as a NumPy vector.\n \"\"\"\n raise NotImplementedError(\n 'To use ExternalOptimizerInterface, subclass from it and implement '\n 'the _minimize() method.')", "response": "Wrapper for the external optimizer interface to minimize an internal key - value vector."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npack a list of Tensor s into a single flattened rank - 1 Tensor.", "response": "def _pack(cls, tensors):\n \"\"\"Pack a list of `Tensor`s into a single, flattened, rank-1 `Tensor`.\"\"\"\n if not tensors:\n return None\n elif len(tensors) == 1:\n return array_ops.reshape(tensors[0], [-1])\n else:\n flattened = [array_ops.reshape(tensor, [-1]) for tensor in tensors]\n return array_ops.concat(flattened, 0)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconstructs a function that evaluates a Tensor or list of Tensors.", "response": "def _make_eval_func(self, tensors, session, feed_dict, fetches,\n callback=None):\n \"\"\"Construct a function that evaluates a `Tensor` or list of `Tensor`s.\"\"\"\n if not isinstance(tensors, list):\n tensors = [tensors]\n num_tensors = len(tensors)\n\n def eval_func(x):\n \"\"\"Function to evaluate a `Tensor`.\"\"\"\n shapes = dict(zip(self._vars, self._var_shapes))\n augmented_feed_dict = {\n var: x[packing_slice].reshape(shapes[var])\n for var, packing_slice in zip(self._vars, self._packing_slices)\n }\n augmented_feed_dict.update(feed_dict)\n augmented_fetches = tensors + fetches\n\n augmented_fetch_vals = session.run(\n augmented_fetches, feed_dict=augmented_feed_dict)\n\n if callable(callback):\n callback(*augmented_fetch_vals[num_tensors:])\n\n return augmented_fetch_vals[:num_tensors]\n\n return eval_func"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_global_step(session: tf.Session) -> tf.Variable:\n global_step_tensor = tf.Variable(0, trainable=False, name=\"global_step\")\n session.run(global_step_tensor.initializer)\n return global_step_tensor", "response": "Creates the Tensorflow global_step variable."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef restore_session(session: tf.Session, checkpoint_dir: str,\n saver: Optional[tf.train.Saver] = None) -> None:\n \"\"\"\n Restores Tensorflow session from the latest checkpoint.\n :param session: The TF session\n :param checkpoint_dir: checkpoint files directory.\n :param saver: The saver object, if not provided a default saver object will be created.\n \"\"\"\n checkpoint_path = tf.train.latest_checkpoint(checkpoint_dir)\n logger = settings.logger()\n if logger.isEnabledFor(logging.INFO):\n logger.info(\"Restoring session from `%s`.\", checkpoint_path)\n\n saver = saver or get_default_saver()\n saver.restore(session, checkpoint_path)", "response": "Restores Tensorflow session from the latest checkpoint."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_default_saver(max_to_keep: int=3) -> tf.train.Saver:\n return tf.train.Saver(max_to_keep=max_to_keep)", "response": "Returns a Tensorflow Saver object with default values."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the state of the current state of the current optimiser.", "response": "def update_optimiser(context, *args, **kwargs) -> None:\n \"\"\"\n Writes optimiser state into corresponding TensorFlow variables. This may need to be done\n for optimisers like ScipyOptimiser that work with their own copies of the variables.\n Normally the source variables would be updated only when the optimiser has finished the\n minimisation. This function may be called from the callback in order to get the TensorFlow\n variables up-to-date so that they can be written into TensorBoard for example.\n\n The expected state parameters and the procedure of writing them into variables is specific\n to each optimiser. Currently it is implemented only for ScipyOptimiser.\n\n After the state is updated a flag is set to the context to prevent multiple updates in\n the same callback.\n\n :param context: Monitor context\n :param args: Optimiser's state passed to the callback\n :param kwargs: Optimiser's state passed to the callback\n \"\"\"\n\n if context.optimiser is None or context.optimiser_updated:\n return\n\n if isinstance(context.optimiser, ScipyOptimizer) and len(args) > 0:\n\n optimizer = context.optimiser.optimizer # get access to ExternalOptimizerInterface\n var_vals = [args[0][packing_slice] for packing_slice in optimizer._packing_slices]\n context.session.run(optimizer._var_updates,\n feed_dict=dict(zip(optimizer._update_placeholders, var_vals)))\n context.optimiser_updated = True"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nevaluates the value of the global step variable.", "response": "def global_step(self) -> int:\n \"\"\"\n Evaluates the value of the global step variable if it is set, otherwise returns the\n current iteration number.\n \"\"\"\n if self.session is None or self.global_step_tensor is None:\n return self.iteration_no + self.init_global_step\n else:\n return self.session.run(self.global_step_tensor)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the condition that will be evaluated during the optimisation cycle.", "response": "def with_condition(self, condition: Callable[[MonitorContext], bool]) -> 'MonitorTask':\n \"\"\"\n Sets the task running condition that will be evaluated during the optimisation cycle.\n \"\"\"\n self._condition = condition\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the flag indicating that the task should also run after the optimisation is ended.", "response": "def with_exit_condition(self, exit_condition: Optional[bool]=True) -> 'MonitorTask':\n \"\"\"\n Sets the flag indicating that the task should also run after the optimisation is ended.\n \"\"\"\n self._exit_condition = exit_condition\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start_monitoring(self) -> None:\n self._context.init_global_step = self._context.global_step\n self._start_timestamp = get_hr_time()\n self._last_timestamp = self._start_timestamp", "response": "Start monitoring for the current context."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nstops monitoring the current context.", "response": "def stop_monitoring(self) -> None:\n \"\"\"\n The recommended way of using Monitor is opening it with the `with` statement. In this case\n the user doesn't need to call this function explicitly. Otherwise the function should be\n called when the optimisation is done.\n\n The function sets the optimisation completed flag in the monitoring context and runs the\n tasks once more. If the monitor was created with the `print_summary` option it prints the\n tasks' timing summary.\n \"\"\"\n\n self._context.optimisation_finished = True\n self._on_iteration()\n\n if self._print_summary:\n self.print_summary()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef print_summary(self) -> None:\n print(\"Tasks execution time summary:\")\n for mon_task in self._monitor_tasks:\n print(\"%s:\\t%.4f (sec)\" % (mon_task.task_name, mon_task.total_time))", "response": "Prints the tasks execution time summary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalling at each iteration.", "response": "def _on_iteration(self, *args, **kwargs) -> None:\n \"\"\"\n Called at each iteration.\n\n This function does time measurements, updates timing in the monitoring context and calls\n all monitoring tasks.\n \"\"\"\n\n # Update timing and the iteration number in the monitoring context.\n current_timestamp = get_hr_time()\n self._context.optimisation_time += current_timestamp - self._last_timestamp\n self._context.total_time = current_timestamp - self._start_timestamp\n self._context.optimiser_updated = False\n if not self._context.optimisation_finished:\n self._context.iteration_no += 1\n\n # Call all monitoring functions\n for func in self._monitor_tasks:\n func(self._context, *args, **kwargs)\n\n # Remember the time when the control is returned back to the optimiser, so that the next\n # optimisation step can be accurately timed.\n self._last_timestamp = get_hr_time()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _growing_step_sequence(interval_growth, max_interval, init_interval, start_level=None):\n interval = init_interval\n next_level = start_level or init_interval\n while True:\n yield next_level\n interval = min(interval * interval_growth, max_interval)\n next_level += interval", "response": "Generates an iterator that yields a sequence of trigger levels with growing intervals."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlock the location used by this writer.", "response": "def __lock_location(self) -> None:\n \"\"\"\n Attempts to lock the location used by this writer. Will raise an error if the location is\n already locked by another writer. Will do nothing if the location is already locked by\n this writer.\n \"\"\"\n if not self._is_active:\n if self._location in LogdirWriter._locked_locations:\n raise RuntimeError('TensorBoard event file in directory %s with suffix %s '\n 'is already in use. At present multiple TensoBoard file writers '\n 'cannot write data into the same file.' % self._location)\n LogdirWriter._locked_locations.add(self._location)\n self._is_active = True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __release_location(self) -> None:\n if self._is_active:\n LogdirWriter._locked_locations.remove(self._location)\n self._is_active = False", "response": "Release the lock on the location used by this writer."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the flag indicating that the event file should be flushed at each call.", "response": "def with_flush_immediately(self, flush_immediately: Optional[bool]=True)\\\n -> 'BaseTensorBoardTask':\n \"\"\"\n Sets the flag indicating that the event file should be flushed at each call.\n \"\"\"\n self._flush_immediately = flush_immediately\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _eval_summary(self, context: MonitorContext, feed_dict: Optional[Dict]=None) -> None:\n\n if self._summary is None:\n raise RuntimeError('TensorBoard monitor task should set the Tensorflow.Summary object')\n\n if context.session is None:\n raise RuntimeError('To run a TensorBoard monitor task the TF session object'\n ' must be provided when creating an instance of the Monitor')\n\n summary = context.session.run(self._summary, feed_dict=feed_dict)\n self._file_writer.add_summary(summary, context.global_step)\n if self._flush_immediately:\n self.flush()", "response": "Evaluates the summary tensor and writes the result to the event file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn two n - by - n matrices. The first one contains all the x values and the second all the y values . The second one contains all the x values and the second all the y values .", "response": "def make_grid(xx, yy):\n \"\"\"\n Returns two n-by-n matrices. The first one contains all the x values \n and the second all the y values of a cartesian product between `xx` and `yy`.\n \"\"\"\n n = len(xx)\n xx, yy = np.meshgrid(xx, yy)\n grid = np.array([xx.ravel(), yy.ravel()]).T\n x = grid[:, 0].reshape(n, n)\n y = grid[:, 1].reshape(n, n)\n return x, y"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef elapsed(self):\n if self._stop is None:\n return timer() - self._start\n return self._stop - self._start", "response": "Returns the elapsed time between start and stop timestamps."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef watcher(self) -> Watcher:\n if not hasattr(self, \"_watcher\"):\n self._watcher = Watcher()\n return self._watcher", "response": "Returns an instance of Watcher that can be used to access the action s watcher."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef with_settings(self,\n stop: Optional[int] = None,\n start: int = 0,\n step: int = 1) -> 'Loop':\n \"\"\"\n Set start, stop and step loop configuration.\n\n :param stop: Looop stop iteration integer. If None then loop\n becomes infinite.\n :param start: Loop iteration start integer.\n :param step: Loop iteration interval integer.\n :return: Loop itself.\n \"\"\"\n self.start = start\n self.stop = stop\n self.step = step\n return self", "response": "Sets start stop and step loop configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun performs loop iterations. :param context: Action context.", "response": "def run(self, context: ActionContext):\n \"\"\"\n Run performs loop iterations.\n\n :param context: Action context.\n \"\"\"\n iterator = itertools.count(start=self.start, step=self.step)\n for i in iterator:\n self.with_iteration(i)\n if self.stop is not None and i >= self.stop:\n break\n try:\n self._action(context)\n except Loop.Continue:\n continue\n except Loop.Break:\n break"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreplaces optimizer tensor with new model.", "response": "def with_optimizer_tensor(self, tensor: Union[tf.Tensor, tf.Operation]) -> 'Optimization':\n \"\"\"\n Replace optimizer tensor.\n\n :param model: Tensorflow tensor.\n :return: Optimization instance self reference.\n \"\"\"\n self._optimizer_tensor = tensor\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreplacing Tensorflow session run kwargs with kwargs.", "response": "def with_run_kwargs(self, **kwargs: Dict[str, Any]) -> 'Optimization':\n \"\"\"\n Replace Tensorflow session run kwargs.\n Check Tensorflow session run [documentation](https://www.tensorflow.org/api_docs/python/tf/Session).\n\n :param kwargs: Dictionary of tensors as keys and numpy arrays or\n primitive python types as values.\n :return: Optimization instance self reference.\n \"\"\"\n self._run_kwargs = kwargs\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _conditional(Xnew, feat, kern, f, *, full_cov=False, full_output_cov=False, q_sqrt=None, white=False):\n logger.debug(\"Conditional: Inducing Feature - Kernel\")\n Kmm = Kuu(feat, kern, jitter=settings.numerics.jitter_level) # M x M\n Kmn = Kuf(feat, kern, Xnew) # M x N\n Knn = kern.K(Xnew) if full_cov else kern.Kdiag(Xnew)\n\n fmean, fvar = base_conditional(Kmn, Kmm, Knn, f, full_cov=full_cov,\n q_sqrt=q_sqrt, white=white) # N x R, R x N x N or N x R\n return fmean, _expand_independent_outputs(fvar, full_cov, full_output_cov)", "response": "Function to calculate the covariance matrix of a single - output GP conditional."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving f, representing the GP at the points X, produce the mean and (co-)variance of the GP at the points Xnew. Additionally, there may be Gaussian uncertainty about f as represented by q_sqrt. In this case `f` represents the mean of the distribution and q_sqrt the square-root of the covariance. Additionally, the GP may have been centered (whitened) so that p(v) = N(0, I) f = L v thus p(f) = N(0, LL^T) = N(0, K). In this case `f` represents the values taken by v. The method can either return the diagonals of the covariance matrix for each output (default) or the full covariance matrix (full_cov=True). We assume R independent GPs, represented by the columns of f (and the first dimension of q_sqrt). :param Xnew: data matrix, size N x D. Evaluate the GP at these new points :param X: data points, size M x D. :param kern: GPflow kernel. :param f: data matrix, M x R, representing the function values at X, for R functions. :param q_sqrt: matrix of standard-deviations or Cholesky matrices, size M x R or R x M x M. :param white: boolean of whether to use the whitened representation as described above. :return: - mean: N x R - variance: N x R (full_cov = False), R x N x N (full_cov = True)", "response": "def _conditional(Xnew, X, kern, f, *, full_cov=False, q_sqrt=None, white=False, full_output_cov=None):\n \"\"\"\n Given f, representing the GP at the points X, produce the mean and\n (co-)variance of the GP at the points Xnew.\n\n Additionally, there may be Gaussian uncertainty about f as represented by\n q_sqrt. In this case `f` represents the mean of the distribution and\n q_sqrt the square-root of the covariance.\n\n Additionally, the GP may have been centered (whitened) so that\n p(v) = N(0, I)\n f = L v\n thus\n p(f) = N(0, LL^T) = N(0, K).\n In this case `f` represents the values taken by v.\n\n The method can either return the diagonals of the covariance matrix for\n each output (default) or the full covariance matrix (full_cov=True).\n\n We assume R independent GPs, represented by the columns of f (and the\n first dimension of q_sqrt).\n\n :param Xnew: data matrix, size N x D. Evaluate the GP at these new points\n :param X: data points, size M x D.\n :param kern: GPflow kernel.\n :param f: data matrix, M x R, representing the function values at X,\n for R functions.\n :param q_sqrt: matrix of standard-deviations or Cholesky matrices,\n size M x R or R x M x M.\n :param white: boolean of whether to use the whitened representation as\n described above.\n :return:\n - mean: N x R\n - variance: N x R (full_cov = False), R x N x N (full_cov = True)\n \"\"\"\n logger.debug(\"Conditional: Kernel\")\n num_data = tf.shape(X)[-2] # M\n Kmm = kern.K(X) + tf.eye(num_data, dtype=settings.float_type) * settings.numerics.jitter_level # [..., M, M]\n Kmn = kern.K(X, Xnew) # [M, ..., N]\n\n if full_cov:\n Knn = kern.K(Xnew) # [...,N,N]\n else:\n Knn = kern.Kdiag(Xnew) # [...,N]\n\n mean, var = base_conditional(Kmn, Kmm, Knn, f, full_cov=full_cov, q_sqrt=q_sqrt, white=white)\n\n return mean, var"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsample from a conditional distribution.", "response": "def _sample_conditional(Xnew, feat, kern, f, *, full_cov=False, full_output_cov=False, q_sqrt=None, white=False, num_samples=None):\n \"\"\"\n `sample_conditional` will return a sample from the conditional distribution.\n In most cases this means calculating the conditional mean m and variance v and then\n returning m + sqrt(v) * eps, with eps ~ N(0, 1).\n However, for some combinations of Mok and Mof more efficient sampling routines exists.\n The dispatcher will make sure that we use the most efficient one.\n\n :return: samples, mean, cov\n samples has shape [num_samples, N, P] or [N, P] if num_samples is None\n mean and cov as for conditional()\n \"\"\"\n if full_cov and full_output_cov:\n raise NotImplementedError(\"The combination of both full_cov and full_output_cov is not \"\n \"implemented for sample_conditional.\")\n\n logger.debug(\"sample conditional: InducingFeature Kernel\")\n mean, cov = conditional(Xnew, feat, kern, f, q_sqrt=q_sqrt, white=white,\n full_cov=full_cov, full_output_cov=full_output_cov)\n if full_cov:\n # mean: [..., N, P]\n # cov: [..., P, N, N]\n mean_PN = tf.matrix_transpose(mean) # [..., P, N]\n samples = _sample_mvn(mean_PN, cov, 'full', num_samples=num_samples) # [..., (S), P, N]\n samples = tf.matrix_transpose(samples) # [..., (S), P, N]\n\n else:\n # mean: [..., N, P]\n # cov: [..., N, P] or [..., N, P, P]\n cov_structure = \"full\" if full_output_cov else \"diag\"\n samples = _sample_mvn(mean, cov, cov_structure, num_samples=num_samples) # [..., (S), P, N]\n\n return samples, mean, cov"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef base_conditional(Kmn, Kmm, Knn, f, *, full_cov=False, q_sqrt=None, white=False):\n logger.debug(\"base conditional\")\n # compute kernel stuff\n num_func = tf.shape(f)[-1] # R\n N = tf.shape(Kmn)[-1]\n M = tf.shape(f)[-2]\n\n # get the leadings dims in Kmn to the front of the tensor\n # if Kmn has rank two, i.e. [M, N], this is the identity op.\n K = tf.rank(Kmn)\n perm = tf.concat([tf.reshape(tf.range(1, K-1), [K-2]), # leading dims (...)\n tf.reshape(0, [1]), # [M]\n tf.reshape(K-1, [1])], 0) # [N]\n Kmn = tf.transpose(Kmn, perm) # ... x M x N\n\n leading_dims = tf.shape(Kmn)[:-2]\n Lm = tf.cholesky(Kmm) # [M,M]\n\n # Compute the projection matrix A\n Lm = tf.broadcast_to(Lm, tf.concat([leading_dims, tf.shape(Lm)], 0)) # [...,M,M]\n A = tf.matrix_triangular_solve(Lm, Kmn, lower=True) # [...,M,N]\n # compute the covariance due to the conditioning\n if full_cov:\n fvar = Knn - tf.matmul(A, A, transpose_a=True) # [...,N,N]\n cov_shape = tf.concat([leading_dims, [num_func, N, N]], 0)\n fvar = tf.broadcast_to(tf.expand_dims(fvar, -3), cov_shape) # [...,R,N,N]\n else:\n fvar = Knn - tf.reduce_sum(tf.square(A), -2) # [...,N]\n cov_shape = tf.concat([leading_dims, [num_func, N]], 0) # [...,R,N]\n fvar = tf.broadcast_to(tf.expand_dims(fvar, -2), cov_shape) # [...,R,N]\n\n # another backsubstitution in the unwhitened case\n if not white:\n A = tf.matrix_triangular_solve(tf.matrix_transpose(Lm), A, lower=False)\n\n # construct the conditional mean\n f_shape = tf.concat([leading_dims, [M, num_func]], 0) # [...,M,R]\n f = tf.broadcast_to(f, f_shape) # [...,M,R]\n fmean = tf.matmul(A, f, transpose_a=True) # [...,N,R]\n\n if q_sqrt is not None:\n if q_sqrt.get_shape().ndims == 2:\n LTA = A * tf.expand_dims(tf.transpose(q_sqrt), 2) # R x M x N\n elif q_sqrt.get_shape().ndims == 3:\n L = q_sqrt\n L = tf.broadcast_to(L, tf.concat([leading_dims, tf.shape(L)], 0))\n\n shape = tf.concat([leading_dims, [num_func, M, N]], 0)\n A_tiled = tf.broadcast_to(tf.expand_dims(A, -3), shape)\n LTA = tf.matmul(L, A_tiled, transpose_a=True) # R x M x N\n else: # pragma: no cover\n raise ValueError(\"Bad dimension for q_sqrt: %s\" %\n str(q_sqrt.get_shape().ndims))\n if full_cov:\n fvar = fvar + tf.matmul(LTA, LTA, transpose_a=True) # R x N x N\n else:\n fvar = fvar + tf.reduce_sum(tf.square(LTA), -2) # R x N\n\n if not full_cov:\n fvar = tf.matrix_transpose(fvar) # N x R\n\n return fmean, fvar", "response": "r Base conditional for the KNN and Kmm."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the conditional for uncertain inputs Xnew mu Xnew_var.", "response": "def uncertain_conditional(Xnew_mu, Xnew_var, feat, kern, q_mu, q_sqrt, *,\n mean_function=None, full_output_cov=False, full_cov=False, white=False):\n \"\"\"\n Calculates the conditional for uncertain inputs Xnew, p(Xnew) = N(Xnew_mu, Xnew_var).\n See ``conditional`` documentation for further reference.\n\n :param Xnew_mu: mean of the inputs, size N x Din\n :param Xnew_var: covariance matrix of the inputs, size N x Din x Din\n :param feat: gpflow.InducingFeature object, only InducingPoints is supported\n :param kern: gpflow kernel object.\n :param q_mu: mean inducing points, size M x Dout\n :param q_sqrt: cholesky of the covariance matrix of the inducing points, size Dout x M x M\n :param full_output_cov: boolean wheter to compute covariance between output dimension.\n Influences the shape of return value ``fvar``. Default is False\n :param white: boolean whether to use whitened representation. Default is False.\n\n :return fmean, fvar: mean and covariance of the conditional, size ``fmean`` is N x Dout,\n size ``fvar`` depends on ``full_output_cov``: if True ``f_var`` is N x Dout x Dout,\n if False then ``f_var`` is N x Dout\n \"\"\"\n\n # TODO(VD): Tensorflow 1.7 doesn't support broadcasting in``tf.matmul`` and\n # ``tf.matrix_triangular_solve``. This is reported in issue 216.\n # As a temporary workaround, we are using ``tf.einsum`` for the matrix\n # multiplications and tiling in the triangular solves.\n # The code that should be used once the bug is resolved is added in comments.\n\n if not isinstance(feat, InducingPoints):\n raise NotImplementedError\n\n if full_cov:\n # TODO(VD): ``full_cov`` True would return a ``fvar`` of shape N x N x D x D,\n # encoding the covariance between input datapoints as well.\n # This is not implemented as this feature is only used for plotting purposes.\n raise NotImplementedError\n\n pXnew = Gaussian(Xnew_mu, Xnew_var)\n\n num_data = tf.shape(Xnew_mu)[0] # number of new inputs (N)\n num_ind = tf.shape(q_mu)[0] # number of inducing points (M)\n num_func = tf.shape(q_mu)[1] # output dimension (D)\n\n q_sqrt_r = tf.matrix_band_part(q_sqrt, -1, 0) # D x M x M\n\n eKuf = tf.transpose(expectation(pXnew, (kern, feat))) # M x N (psi1)\n Kuu = features.Kuu(feat, kern, jitter=settings.jitter) # M x M\n Luu = tf.cholesky(Kuu) # M x M\n\n if not white:\n q_mu = tf.matrix_triangular_solve(Luu, q_mu, lower=True)\n Luu_tiled = tf.tile(Luu[None, :, :], [num_func, 1, 1]) # remove line once issue 216 is fixed\n q_sqrt_r = tf.matrix_triangular_solve(Luu_tiled, q_sqrt_r, lower=True)\n\n Li_eKuf = tf.matrix_triangular_solve(Luu, eKuf, lower=True) # M x N\n fmean = tf.matmul(Li_eKuf, q_mu, transpose_a=True)\n\n eKff = expectation(pXnew, kern) # N (psi0)\n eKuffu = expectation(pXnew, (kern, feat), (kern, feat)) # N x M x M (psi2)\n Luu_tiled = tf.tile(Luu[None, :, :], [num_data, 1, 1]) # remove this line, once issue 216 is fixed\n Li_eKuffu = tf.matrix_triangular_solve(Luu_tiled, eKuffu, lower=True)\n Li_eKuffu_Lit = tf.matrix_triangular_solve(Luu_tiled, tf.matrix_transpose(Li_eKuffu), lower=True) # N x M x M\n cov = tf.matmul(q_sqrt_r, q_sqrt_r, transpose_b=True) # D x M x M\n\n if mean_function is None or isinstance(mean_function, mean_functions.Zero):\n e_related_to_mean = tf.zeros((num_data, num_func, num_func), dtype=settings.float_type)\n else:\n # Update mean: \\mu(x) + m(x)\n fmean = fmean + expectation(pXnew, mean_function)\n\n # Calculate: m(x) m(x)^T + m(x) \\mu(x)^T + \\mu(x) m(x)^T,\n # where m(x) is the mean_function and \\mu(x) is fmean\n e_mean_mean = expectation(pXnew, mean_function, mean_function) # N x D x D\n Lit_q_mu = tf.matrix_triangular_solve(Luu, q_mu, adjoint=True)\n e_mean_Kuf = expectation(pXnew, mean_function, (kern, feat)) # N x D x M\n # einsum isn't able to infer the rank of e_mean_Kuf, hence we explicitly set the rank of the tensor:\n e_mean_Kuf = tf.reshape(e_mean_Kuf, [num_data, num_func, num_ind])\n e_fmean_mean = tf.einsum(\"nqm,mz->nqz\", e_mean_Kuf, Lit_q_mu) # N x D x D\n e_related_to_mean = e_fmean_mean + tf.matrix_transpose(e_fmean_mean) + e_mean_mean\n\n if full_output_cov:\n fvar = (\n tf.matrix_diag(tf.tile((eKff - tf.trace(Li_eKuffu_Lit))[:, None], [1, num_func])) +\n tf.matrix_diag(tf.einsum(\"nij,dji->nd\", Li_eKuffu_Lit, cov)) +\n # tf.matrix_diag(tf.trace(tf.matmul(Li_eKuffu_Lit, cov))) +\n tf.einsum(\"ig,nij,jh->ngh\", q_mu, Li_eKuffu_Lit, q_mu) -\n # tf.matmul(q_mu, tf.matmul(Li_eKuffu_Lit, q_mu), transpose_a=True) -\n fmean[:, :, None] * fmean[:, None, :] +\n e_related_to_mean\n )\n else:\n fvar = (\n (eKff - tf.trace(Li_eKuffu_Lit))[:, None] +\n tf.einsum(\"nij,dji->nd\", Li_eKuffu_Lit, cov) +\n tf.einsum(\"ig,nij,jg->ng\", q_mu, Li_eKuffu_Lit, q_mu) -\n fmean ** 2 +\n tf.matrix_diag_part(e_related_to_mean)\n )\n\n return fmean, fvar"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _sample_mvn(mean, cov, cov_structure=None, num_samples=None):\n mean_shape = tf.shape(mean)\n S = num_samples if num_samples is not None else 1\n D = mean_shape[-1]\n leading_dims = mean_shape[:-2]\n num_leading_dims = tf.size(leading_dims)\n\n if cov_structure == \"diag\":\n # mean: [..., N, D] and cov [..., N, D]\n with tf.control_dependencies([tf.assert_equal(tf.rank(mean), tf.rank(cov))]):\n eps_shape = tf.concat([leading_dims, [S], mean_shape[-2:]], 0)\n eps = tf.random_normal(eps_shape, dtype=settings.float_type) # [..., S, N, D]\n samples = mean[..., None, :, :] + tf.sqrt(cov)[..., None, :, :] * eps # [..., S, N, D]\n elif cov_structure == \"full\":\n # mean: [..., N, D] and cov [..., N, D, D]\n with tf.control_dependencies([tf.assert_equal(tf.rank(mean) + 1, tf.rank(cov))]):\n jittermat = (\n tf.eye(D, batch_shape=mean_shape[:-1], dtype=settings.float_type)\n * settings.jitter\n ) # [..., N, D, D]\n eps_shape = tf.concat([mean_shape, [S]], 0)\n eps = tf.random_normal(eps_shape, dtype=settings.float_type) # [..., N, D, S]\n chol = tf.cholesky(cov + jittermat) # [..., N, D, D]\n samples = mean[..., None] + tf.matmul(chol, eps) # [..., N, D, S]\n samples = misc.leading_transpose(samples, [..., -1, -3, -2]) # [..., S, N, D]\n else:\n raise NotImplementedError # pragma: no cover\n\n if num_samples is None:\n return samples[..., 0, :, :] # [..., N, D]\n return samples", "response": "Sample from a D - dimensional Multivariate Normal distribution."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _expand_independent_outputs(fvar, full_cov, full_output_cov):\n if full_cov and full_output_cov:\n fvar = tf.matrix_diag(tf.transpose(fvar)) # N x N x P x P\n fvar = tf.transpose(fvar, [0, 2, 1, 3]) # N x P x N x P\n if not full_cov and full_output_cov:\n fvar = tf.matrix_diag(fvar) # N x P x P\n if full_cov and not full_output_cov:\n pass # P x N x N\n if not full_cov and not full_output_cov:\n pass # N x P\n\n return fvar", "response": "Reshapes fvar to the correct shape specified by full_cov and full_output_cov."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrolling the tensor A forward num_rolls times", "response": "def _rollaxis_right(A, num_rolls):\n \"\"\" Roll the tensor `A` forward `num_rolls` times \"\"\"\n assert num_rolls > 0\n rank = tf.rank(A)\n perm = tf.concat([rank - num_rolls + tf.range(num_rolls), tf.range(rank - num_rolls)], 0)\n return tf.transpose(A, perm)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwraps for the inducingpoint method.", "response": "def inducingpoint_wrapper(feat, Z):\n \"\"\"\n Models which used to take only Z can now pass `feat` and `Z` to this method. This method will\n check for consistency and return the correct feature. This allows backwards compatibility in\n for the methods.\n \"\"\"\n if feat is not None and Z is not None:\n raise ValueError(\"Cannot pass both an InducingFeature instance and Z values\") # pragma: no cover\n elif feat is None and Z is None:\n raise ValueError(\"You must pass either an InducingFeature instance or Z values\") # pragma: no cover\n elif Z is not None:\n feat = InducingPoints(Z)\n elif isinstance(feat, np.ndarray):\n feat = InducingPoints(feat)\n else:\n assert isinstance(feat, InducingFeature) # pragma: no cover\n return feat"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _build_likelihood(self):\n\n num_inducing = len(self.feature)\n num_data = tf.cast(tf.shape(self.Y)[0], settings.float_type)\n output_dim = tf.cast(tf.shape(self.Y)[1], settings.float_type)\n\n err = self.Y - self.mean_function(self.X)\n Kdiag = self.kern.Kdiag(self.X)\n Kuf = features.Kuf(self.feature, self.kern, self.X)\n Kuu = features.Kuu(self.feature, self.kern, jitter=settings.numerics.jitter_level)\n L = tf.cholesky(Kuu)\n sigma = tf.sqrt(self.likelihood.variance)\n\n # Compute intermediate matrices\n A = tf.matrix_triangular_solve(L, Kuf, lower=True) / sigma\n AAT = tf.matmul(A, A, transpose_b=True)\n B = AAT + tf.eye(num_inducing, dtype=settings.float_type)\n LB = tf.cholesky(B)\n Aerr = tf.matmul(A, err)\n c = tf.matrix_triangular_solve(LB, Aerr, lower=True) / sigma\n\n # compute log marginal bound\n bound = -0.5 * num_data * output_dim * np.log(2 * np.pi)\n bound += tf.negative(output_dim) * tf.reduce_sum(tf.log(tf.matrix_diag_part(LB)))\n bound -= 0.5 * num_data * output_dim * tf.log(self.likelihood.variance)\n bound += -0.5 * tf.reduce_sum(tf.square(err)) / self.likelihood.variance\n bound += 0.5 * tf.reduce_sum(tf.square(c))\n bound += -0.5 * output_dim * tf.reduce_sum(Kdiag) / self.likelihood.variance\n bound += 0.5 * output_dim * tf.reduce_sum(tf.matrix_diag_part(AAT))\n\n return bound", "response": "Builds a tensorflow function to compute the bound on the marginal log likelihood of the associated entry."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbuilding the predict term for a new set of points.", "response": "def _build_predict(self, Xnew, full_cov=False):\n \"\"\"\n Compute the mean and variance of the latent function at some new points\n Xnew. For a derivation of the terms in here, see the associated SGPR\n notebook. \n \"\"\"\n num_inducing = len(self.feature)\n err = self.Y - self.mean_function(self.X)\n Kuf = features.Kuf(self.feature, self.kern, self.X)\n Kuu = features.Kuu(self.feature, self.kern, jitter=settings.numerics.jitter_level)\n Kus = features.Kuf(self.feature, self.kern, Xnew)\n sigma = tf.sqrt(self.likelihood.variance)\n L = tf.cholesky(Kuu)\n A = tf.matrix_triangular_solve(L, Kuf, lower=True) / sigma\n B = tf.matmul(A, A, transpose_b=True) + tf.eye(num_inducing, dtype=settings.float_type)\n LB = tf.cholesky(B)\n Aerr = tf.matmul(A, err)\n c = tf.matrix_triangular_solve(LB, Aerr, lower=True) / sigma\n tmp1 = tf.matrix_triangular_solve(L, Kus, lower=True)\n tmp2 = tf.matrix_triangular_solve(LB, tmp1, lower=True)\n mean = tf.matmul(tmp2, c, transpose_a=True)\n if full_cov:\n var = self.kern.K(Xnew) + tf.matmul(tmp2, tmp2, transpose_a=True) \\\n - tf.matmul(tmp1, tmp1, transpose_a=True)\n var = tf.tile(var[None, ...], [self.num_latent, 1, 1]) # P x N x N\n else:\n var = self.kern.Kdiag(Xnew) + tf.reduce_sum(tf.square(tmp2), 0) \\\n - tf.reduce_sum(tf.square(tmp1), 0)\n var = tf.tile(var[:, None], [1, self.num_latent])\n return mean + self.mean_function(Xnew), var"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the mean and variance of the variational distribution on the entry set.", "response": "def compute_qu(self):\n \"\"\"\n Computes the mean and variance of q(u), the variational distribution on\n inducing outputs. SVGP with this q(u) should predict identically to\n SGPR.\n :return: mu, A\n \"\"\"\n Kuf = features.Kuf(self.feature, self.kern, self.X)\n Kuu = features.Kuu(self.feature, self.kern, jitter=settings.jitter)\n\n Sig = Kuu + (self.likelihood.variance ** -1) * tf.matmul(Kuf, Kuf, transpose_b=True)\n Sig_sqrt = tf.cholesky(Sig)\n\n Sig_sqrt_Kuu = tf.matrix_triangular_solve(Sig_sqrt, Kuu)\n\n A = tf.matmul(Sig_sqrt_Kuu, Sig_sqrt_Kuu, transpose_a=True)\n mu = tf.matmul(Sig_sqrt_Kuu,\n tf.matrix_triangular_solve(Sig_sqrt, tf.matmul(Kuf, self.Y - self.mean_function(self.X))),\n transpose_a=True) * self.likelihood.variance ** -1.0\n\n return mu, A"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _build_likelihood(self):\n\n # FITC approximation to the log marginal likelihood is\n # log ( normal( y | mean, K_fitc ) )\n # where K_fitc = Qff + diag( \\nu )\n # where Qff = Kfu Kuu^{-1} Kuf\n # with \\nu_i = Kff_{i,i} - Qff_{i,i} + \\sigma^2\n\n # We need to compute the Mahalanobis term -0.5* err^T K_fitc^{-1} err\n # (summed over functions).\n\n # We need to deal with the matrix inverse term.\n # K_fitc^{-1} = ( Qff + \\diag( \\nu ) )^{-1}\n # = ( V^T V + \\diag( \\nu ) )^{-1}\n # Applying the Woodbury identity we obtain\n # = \\diag( \\nu^{-1} ) - \\diag( \\nu^{-1} ) V^T ( I + V \\diag( \\nu^{-1} ) V^T )^{-1) V \\diag(\\nu^{-1} )\n # Let \\beta = \\diag( \\nu^{-1} ) err\n # and let \\alpha = V \\beta\n # then Mahalanobis term = -0.5* ( \\beta^T err - \\alpha^T Solve( I + V \\diag( \\nu^{-1} ) V^T, alpha ) )\n\n err, nu, Luu, L, alpha, beta, gamma = self._build_common_terms()\n\n mahalanobisTerm = -0.5 * tf.reduce_sum(tf.square(err) / tf.expand_dims(nu, 1)) \\\n + 0.5 * tf.reduce_sum(tf.square(gamma))\n\n # We need to compute the log normalizing term -N/2 \\log 2 pi - 0.5 \\log \\det( K_fitc )\n\n # We need to deal with the log determinant term.\n # \\log \\det( K_fitc ) = \\log \\det( Qff + \\diag( \\nu ) )\n # = \\log \\det( V^T V + \\diag( \\nu ) )\n # Applying the determinant lemma we obtain\n # = \\log [ \\det \\diag( \\nu ) \\det( I + V \\diag( \\nu^{-1} ) V^T ) ]\n # = \\log [ \\det \\diag( \\nu ) ] + \\log [ \\det( I + V \\diag( \\nu^{-1} ) V^T ) ]\n\n constantTerm = -0.5 * self.num_data * tf.log(tf.constant(2. * np.pi, settings.float_type))\n logDeterminantTerm = -0.5 * tf.reduce_sum(tf.log(nu)) - tf.reduce_sum(tf.log(tf.matrix_diag_part(L)))\n logNormalizingTerm = constantTerm + logDeterminantTerm\n\n return mahalanobisTerm + logNormalizingTerm * self.num_latent", "response": "Constructs a tensorflow function to compute the marginal likelihood of the log marginal log log."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the mean and variance of the latent function at some new points Xnew.", "response": "def _build_predict(self, Xnew, full_cov=False):\n \"\"\"\n Compute the mean and variance of the latent function at some new points\n Xnew.\n \"\"\"\n _, _, Luu, L, _, _, gamma = self._build_common_terms()\n Kus = features.Kuf(self.feature, self.kern, Xnew) # size M x Xnew\n\n w = tf.matrix_triangular_solve(Luu, Kus, lower=True) # size M x Xnew\n\n tmp = tf.matrix_triangular_solve(tf.transpose(L), gamma, lower=False)\n mean = tf.matmul(w, tmp, transpose_a=True) + self.mean_function(Xnew)\n intermediateA = tf.matrix_triangular_solve(L, w, lower=True)\n\n if full_cov:\n var = self.kern.K(Xnew) - tf.matmul(w, w, transpose_a=True) \\\n + tf.matmul(intermediateA, intermediateA, transpose_a=True)\n var = tf.tile(var[None, ...], [self.num_latent, 1, 1]) # P x N x N\n else:\n var = self.kern.Kdiag(Xnew) - tf.reduce_sum(tf.square(w), 0) \\\n + tf.reduce_sum(tf.square(intermediateA), 0) # size Xnew,\n var = tf.tile(var[:, None], [1, self.num_latent])\n\n return mean, var"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntranspose tensors with leading dimensions.", "response": "def leading_transpose(tensor: tf.Tensor, perm: List[Union[int, type(...)]]) -> tf.Tensor:\n \"\"\"\n Transposes tensors with leading dimensions. Leading dimensions in\n permutation list represented via ellipsis `...`.\n\n When leading dimensions are found, `transpose` method\n considers them as a single grouped element indexed by 0 in `perm` list. So, passing\n `perm=[-2, ..., -1]`, you assume that your input tensor has [..., A, B] shape,\n and you want to move leading dims between A and B dimensions.\n Dimension indices in permutation list can be negative or positive. Valid positive\n indices start from 1 up to the tensor rank, viewing leading dimensions `...` as zero\n index.\n\n Example:\n a = tf.random.normal((1, 2, 3, 4, 5, 6))\n b = leading_transpose(a, [5, -3, ..., -2])\n sess.run(b).shape\n output> (6, 4, 1, 2, 3, 5)\n\n :param tensor: TensorFlow tensor.\n :param perm: List of permutation indices.\n\n :returns: TensorFlow tensor.\n :raises: ValueError when `...` cannot be found.\n \"\"\"\n perm = copy.copy(perm)\n idx = perm.index(...)\n perm[idx] = 0\n\n rank = tf.rank(tensor)\n perm_tf = perm % rank\n\n leading_dims = tf.range(rank - len(perm) + 1)\n perm = tf.concat([perm_tf[:idx], leading_dims, perm_tf[idx+1:]], 0)\n return tf.transpose(tensor, perm)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef normalize_num_type(num_type):\n if isinstance(num_type, tf.DType):\n num_type = num_type.as_numpy_dtype.type\n\n if num_type in [np.float32, np.float64]: # pylint: disable=E1101\n num_type = settings.float_type\n elif num_type in [np.int16, np.int32, np.int64]:\n num_type = settings.int_type\n else:\n raise ValueError('Unknown dtype \"{0}\" passed to normalizer.'.format(num_type))\n\n return num_type", "response": "Normalizes the number of types for the base array."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef vec_to_tri(vectors, N):\n indices = list(zip(*np.tril_indices(N)))\n indices = tf.constant([list(i) for i in indices], dtype=tf.int64)\n\n def vec_to_tri_vector(vector):\n return tf.scatter_nd(indices=indices, shape=[N, N], updates=vector)\n\n return tf.map_fn(vec_to_tri_vector, vectors)", "response": "Takes a D x M tensor vectors and maps it to a D x matrix_size X matrix_sizetensor\n where the lower triangle of each matrix_size x matrix_size matrix is returned."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert between GPflow indexing and tensorflow indexing method is a function that broadcasts over the first dimension and the final dimension.", "response": "def swap_dimensions(method):\n \"\"\"\n Converts between GPflow indexing and tensorflow indexing\n `method` is a function that broadcasts over the first dimension (i.e. like all tensorflow matrix ops):\n `method` inputs DN1, DNN\n `method` outputs DN1, DNN\n :return: Function that broadcasts over the final dimension (i.e. compatible with GPflow):\n inputs: ND, DNN\n outputs: ND, DNN\n \"\"\"\n @functools.wraps(method)\n def wrapper(a_nd, b_dnn, swap=True):\n if swap:\n if a_nd.get_shape().ndims != 2: # pragma: no cover\n raise ValueError(\"The `a_nd` input must have 2 dimensions.\")\n a_dn1 = tf.transpose(a_nd)[:, :, None]\n A_dn1, B_dnn = method(a_dn1, b_dnn)\n A_nd = tf.transpose(A_dn1[:, :, 0])\n return A_nd, B_dnn\n else:\n return method(a_nd, b_dnn)\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntakes inverse of lower triangular (e.g. Cholesky) matrix. This function broadcasts over the first index. :param M: Tensor with lower triangular structure of shape DxNxN :return: The inverse of the Cholesky decomposition. Same shape as input.", "response": "def _inverse_lower_triangular(M):\n \"\"\"\n Take inverse of lower triangular (e.g. Cholesky) matrix. This function\n broadcasts over the first index.\n\n :param M: Tensor with lower triangular structure of shape DxNxN\n :return: The inverse of the Cholesky decomposition. Same shape as input.\n \"\"\"\n if M.get_shape().ndims != 3: # pragma: no cover\n raise ValueError(\"Number of dimensions for input is required to be 3.\")\n D, N = tf.shape(M)[0], tf.shape(M)[1]\n I_DNN = tf.eye(N, dtype=M.dtype)[None, :, :] * tf.ones((D, 1, 1), dtype=M.dtype)\n return tf.matrix_triangular_solve(M, I_DNN)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a Tensorflow optimization tensor.", "response": "def make_optimize_tensor(self, model, session=None, var_list=None):\n \"\"\"\n Make Tensorflow optimization tensor.\n This method builds natural gradients optimization tensor and initializes all\n necessary variables created by the optimizer.\n\n :param model: GPflow model.\n :param session: Tensorflow session.\n :param var_list: List of tuples of variational parameters.\n :return: Tensorflow natural gradient operation.\n \"\"\"\n session = model.enquire_session(session)\n with session.as_default(), tf.name_scope(self.name):\n # Create optimizer variables before initialization.\n return self._build_natgrad_step_ops(model, *var_list)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nforwarding - mode pushforward analogous to the pullback defined by tf. gradients.", "response": "def _forward_gradients(ys, xs, d_xs):\n \"\"\"\n Forward-mode pushforward analogous to the pullback defined by tf.gradients.\n With tf.gradients, grad_ys is the vector being pulled back, and here d_xs is\n the vector being pushed forward, i.e. this computes (\u2202ys / \u2202xs)^T \u2202xs.\n\n This is adapted from https://github.com/HIPS/autograd/pull/175#issuecomment-306984338\n\n :param ys: list of variables being differentiated (tensor)\n :param xs: list of variables to differentiate wrt (tensor)\n :param d_xs: list of gradients to push forward (same shapes as ys)\n :return: the specified moment of the variational distribution\n \"\"\"\n # this should be v = [tf.placeholder(y.dtype) for y in ys], but tensorflow\n # wants a value for the placeholder, even though it never gets used\n v = [tf.placeholder_with_default(tf.zeros(y.get_shape(), dtype=y.dtype),\n shape=y.get_shape()) for y in ys]\n\n g = tf.gradients(ys, xs, grad_ys=v)\n return tf.gradients(g, v, grad_ys=d_xs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nminimize the model at a specific session.", "response": "def minimize(self, model, session=None, var_list=None, feed_dict=None,\n maxiter=1000, initialize=True, anchor=True, step_callback=None, **kwargs):\n \"\"\"\n :param model: GPflow model with objective tensor.\n :param session: Session where optimization will be run.\n :param var_list: List of extra variables which should be trained during optimization.\n :param feed_dict: Feed dictionary of tensors passed to session run method.\n :param maxiter: Number of run interation.\n :param initialize: If `True` model parameters will be re-initialized even if they were\n initialized before in the specified session.\n :param anchor: If `True` trained variable values computed during optimization at\n particular session will be synchronized with internal parameter values.\n :param step_callback: A callback function to execute at each optimization step.\n Callback takes an arbitrary list of arguments. Input arguments depend on\n interface implementation.\n :param kwargs: This is a dictionary of extra parameters for session run method.\n\n \"\"\"\n raise NotImplementedError()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstoring compilation flag at specified index in context s shared data.", "response": "def _check_index_in_compilations(context: BaseContext, index: str):\n \"\"\"Store compilation flag at specified index in context's shared data.\"\"\"\n compilations = 'compilations'\n if compilations not in context.shared_data:\n return False\n return index in context.shared_data[compilations]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding a structured numpy array.", "response": "def struct(cls, type_name: int, data: np.ndarray, data_dtype: np.dtype = None, shape: Tuple = None):\n \"\"\"Build structured numpy array.\n\n :param int type_name: StructType enum converted to integer.\n :param data: encoded data as a numpy array or a structured numpy array.\n :param data_dtype: numpy dtype.\n :param shape: in case when a list of numpy arrays is passed the length (shape) of\n that array is required.\n\n :return: structured numpy array.\n {\n '__type__': ,\n '__data__': \n }\n \"\"\"\n\n data_dtype = data.dtype if data_dtype is None else data_dtype\n shape = data.shape if shape is None and data.shape else shape\n data_dtype = [data_dtype]\n if shape:\n data_dtype = [data_dtype[0], shape]\n dtype = np.dtype([type_pattern(), (StructField.DATA.value, *data_dtype)])\n return np.array((type_name, data), dtype=dtype)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntakes snapshot of the object and replaces _parent property value on None.", "response": "def _take_values(self, item: Node) -> DictBasicType:\n \"\"\"Takes snapshot of the object and replaces _parent property value on None to avoid\n infitinite recursion in GPflow tree traversing.\n\n :param item: GPflow node object.\n :return: dictionary snapshot of the node object.\"\"\"\n\n values = super()._take_values(item)\n values['_parent'] = None\n return values"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _take_values(self, item: Parameter) -> DictBasicType:\n\n session = self.context.session\n values = super()._take_values(item)\n cached_value = np.array(item.read_value(session=session))\n values['_value'] = cached_value\n return values", "response": "Uses the super method to take values from the cache and replace the value with the value assossiated with the context s session."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _take_extras(self, item: Parameter) -> Optional[bool]:\n\n index = item.tf_compilation_index()\n if index is not None:\n if item.index == index:\n return True\n _add_index_to_compilations(self.context, index)\n return None", "response": "Return either this GPflow objects requires compilation at decoding time."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _take_values(self, item: Parameterized) -> DictBasicType:\n\n values = super()._take_values(item)\n values = {k: v for k, v in values.items() if not k.startswith(AutoFlow.__autoflow_prefix__)}\n return values", "response": "Uses the super method to take values from the parameter object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef coders(self):\n return (PrimitiveTypeCoder,\n TensorFlowCoder,\n FunctionCoder,\n ListCoder,\n DictCoder,\n SliceCoder,\n ParameterCoder,\n ParamListCoder,\n ParameterizedCoder,\n TransformCoder,\n PriorCoder)", "response": "List of default supported coders."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the name of the object s path name.", "response": "def pathname(self):\n \"\"\"\n Path name is a recursive representation parent path name plus the name\n which was assigned to this object by its parent. In other words, it is\n stack of parent name where top is always parent's original name:\n `parent.pathname + parent.childname` and stop condition is root's\n name.\n\n For example, the pathname of an instance with the two parents may look\n like `parent0/parent1/childname_at_parent1`.\n Top parent's name equals its original name `parent0.name == parent0.pathname`.\n \"\"\"\n if self.parent is self:\n return self.name\n parent = self._parent\n return misc.tensor_name(parent.pathname, parent.childname(self))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn an iterator over all node descendants.", "response": "def _descendants(self):\n \"\"\"\n Scans full list of node descendants.\n\n :return: Generator of nodes.\n \"\"\"\n children = self._children\n if children is not None:\n for child in children.values():\n yield from child._descendants\n yield child"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the child object.", "response": "def _set_child(self, name, child):\n \"\"\"\n Set child.\n\n :param name: Child name.\n :param child: Parentable object.\n \"\"\"\n if not isinstance(child, Parentable):\n raise ValueError('Parentable child object expected, not {child}'.format(child=child))\n child._set_parent(self)\n self._store_child(name, child)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _unset_child(self, name, child):\n if name not in self._children or self._children[name] is not child:\n msg = 'Child {child} with name \"{name}\" is not found'\n raise ValueError(msg.format(child=child, name=name))\n child._set_parent(None)\n self._remove_child(name, child)", "response": "Unsets the child from parent."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _set_parent(self, parent=None):\n if parent is not None:\n if not isinstance(parent, Parentable):\n raise ValueError('Parent object must implement Parentable interface.')\n if parent is self or parent._contains(self):\n raise ValueError('Self references are not allowed.')\n self._parent = parent if parent is not None else None", "response": "Sets the parent of the object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nproduces samples from the posterior latent function at the points Xnew.", "response": "def predict_f_samples(self, Xnew, num_samples):\n \"\"\"\n Produce samples from the posterior latent function(s) at the points\n Xnew.\n \"\"\"\n mu, var = self._build_predict(Xnew, full_cov=True) # N x P, # P x N x N\n jitter = tf.eye(tf.shape(mu)[0], dtype=settings.float_type) * settings.jitter\n samples = []\n for i in range(self.num_latent):\n L = tf.cholesky(var[i, :, :] + jitter)\n shape = tf.stack([tf.shape(L)[0], num_samples])\n V = tf.random_normal(shape, dtype=settings.float_type)\n samples.append(mu[:, i:i + 1] + tf.matmul(L, V))\n return tf.transpose(tf.stack(samples))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef predict_y(self, Xnew):\n pred_f_mean, pred_f_var = self._build_predict(Xnew)\n return self.likelihood.predict_mean_and_var(pred_f_mean, pred_f_var)", "response": "Compute the mean and variance of held - out data at the points Xnew\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef predict_density(self, Xnew, Ynew):\n pred_f_mean, pred_f_var = self._build_predict(Xnew)\n return self.likelihood.predict_density(pred_f_mean, pred_f_var, Ynew)", "response": "Compute the log density of the data Ynew at the points Xnew and Ynew."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _build_likelihood(self):\n K = self.kern.K(self.X) + tf.eye(tf.shape(self.X)[0], dtype=settings.float_type) * self.likelihood.variance\n L = tf.cholesky(K)\n m = self.mean_function(self.X)\n logpdf = multivariate_normal(self.Y, m, L) # (R,) log-likelihoods for each independent dimension of Y\n\n return tf.reduce_sum(logpdf)", "response": "r Construct a tensorflow function to compute the likelihood."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nbuild the predict function for the new set of points.", "response": "def _build_predict(self, Xnew, full_cov=False):\n \"\"\"\n Xnew is a data matrix, the points at which we want to predict.\n\n This method computes\n\n p(F* | Y)\n\n where F* are points on the GP at Xnew, Y are noisy observations at X.\n\n \"\"\"\n y = self.Y - self.mean_function(self.X)\n Kmn = self.kern.K(self.X, Xnew)\n Kmm_sigma = self.kern.K(self.X) + tf.eye(tf.shape(self.X)[0], dtype=settings.float_type) * self.likelihood.variance\n Knn = self.kern.K(Xnew) if full_cov else self.kern.Kdiag(Xnew)\n f_mean, f_var = base_conditional(Kmn, Kmm_sigma, Knn, y, full_cov=full_cov, white=False) # N x P, N x P or P x N x N\n return f_mean + self.mean_function(Xnew), f_var"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef quadrature_expectation(p, obj1, obj2=None, num_gauss_hermite_points=None):\n if isinstance(p, tuple):\n assert len(p) == 2\n\n if p[1].shape.ndims == 2:\n p = DiagonalGaussian(*p)\n elif p[1].shape.ndims == 3:\n p = Gaussian(*p)\n elif p[1].shape.ndims == 4:\n p = MarkovGaussian(*p)\n\n if isinstance(obj1, tuple):\n obj1, feat1 = obj1\n else:\n feat1 = None\n\n if isinstance(obj2, tuple):\n obj2, feat2 = obj2\n else:\n feat2 = None\n\n return _quadrature_expectation(p, obj1, feat1, obj2, feat2, num_gauss_hermite_points)", "response": "Compute the expectation _p"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_eval_func(obj, feature, slice=np.s_[...]):\n if feature is not None:\n # kernel + feature combination\n if not isinstance(feature, InducingFeature) or not isinstance(obj, kernels.Kernel):\n raise TypeError(\"If `feature` is supplied, `obj` must be a kernel.\")\n return lambda x: tf.transpose(Kuf(feature, obj, x))[slice]\n elif isinstance(obj, mean_functions.MeanFunction):\n return lambda x: obj(x)[slice]\n elif isinstance(obj, kernels.Kernel):\n return lambda x: obj.Kdiag(x)\n else:\n raise NotImplementedError()", "response": "Returns the function of interest for the expectation\n depending on the type of obj and feature."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _quadrature_expectation(p, obj1, feature1, obj2, feature2, num_gauss_hermite_points):\n num_gauss_hermite_points = 100 if num_gauss_hermite_points is None else num_gauss_hermite_points\n\n if obj2 is None:\n eval_func = lambda x: get_eval_func(obj1, feature1)(x)\n elif obj1 is None:\n raise NotImplementedError(\"First object cannot be None.\")\n else:\n eval_func = lambda x: (get_eval_func(obj1, feature1, np.s_[:, :, None])(x) *\n get_eval_func(obj2, feature2, np.s_[:, None, :])(x))\n\n if isinstance(p, DiagonalGaussian):\n if isinstance(obj1, kernels.Kernel) and isinstance(obj2, kernels.Kernel) \\\n and obj1.on_separate_dims(obj2): # no joint expectations required\n\n eKxz1 = quadrature_expectation(p, (obj1, feature1),\n num_gauss_hermite_points=num_gauss_hermite_points)\n eKxz2 = quadrature_expectation(p, (obj2, feature2),\n num_gauss_hermite_points=num_gauss_hermite_points)\n return eKxz1[:, :, None] * eKxz2[:, None, :]\n\n else:\n cov = tf.matrix_diag(p.cov)\n else:\n cov = p.cov\n return mvnquad(eval_func, p.mu, cov, num_gauss_hermite_points)", "response": "Internal method that returns the quadrature expectations for a single object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling of quadrature expectations for Markov Gaussians (useful for time series) Fallback method for missing analytic expectations wrt Markov Gaussians Nota Bene: obj1 is always associated with x_n, whereas obj2 always with x_{n+1} if one requires e.g. _p(x_{n:n+1}), compute the transpose and then transpose the result of the expectation", "response": "def _quadrature_expectation(p, obj1, feature1, obj2, feature2, num_gauss_hermite_points):\n \"\"\"\n Handling of quadrature expectations for Markov Gaussians (useful for time series)\n Fallback method for missing analytic expectations wrt Markov Gaussians\n Nota Bene: obj1 is always associated with x_n, whereas obj2 always with x_{n+1}\n if one requires e.g. _p(x_{n:n+1}), compute the\n transpose and then transpose the result of the expectation\n \"\"\"\n num_gauss_hermite_points = 40 if num_gauss_hermite_points is None else num_gauss_hermite_points\n\n if obj2 is None:\n eval_func = lambda x: get_eval_func(obj1, feature1)(x)\n mu, cov = p.mu[:-1], p.cov[0, :-1] # cross covariances are not needed\n elif obj1 is None:\n eval_func = lambda x: get_eval_func(obj2, feature2)(x)\n mu, cov = p.mu[1:], p.cov[0, 1:] # cross covariances are not needed\n else:\n eval_func = lambda x: (get_eval_func(obj1, feature1, np.s_[:, :, None])(tf.split(x, 2, 1)[0]) *\n get_eval_func(obj2, feature2, np.s_[:, None, :])(tf.split(x, 2, 1)[1]))\n mu = tf.concat((p.mu[:-1, :], p.mu[1:, :]), 1) # Nx2D\n cov_top = tf.concat((p.cov[0, :-1, :, :], p.cov[1, :-1, :, :]), 2) # NxDx2D\n cov_bottom = tf.concat((tf.matrix_transpose(p.cov[1, :-1, :, :]), p.cov[0, 1:, :, :]), 2)\n cov = tf.concat((cov_top, cov_bottom), 1) # Nx2Dx2D\n\n return mvnquad(eval_func, mu, cov, num_gauss_hermite_points)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef expectation(p, obj1, obj2=None, nghp=None):\n if isinstance(p, tuple):\n assert len(p) == 2\n\n if p[1].shape.ndims == 2:\n p = DiagonalGaussian(*p)\n elif p[1].shape.ndims == 3:\n p = Gaussian(*p)\n elif p[1].shape.ndims == 4:\n p = MarkovGaussian(*p)\n\n if isinstance(obj1, tuple):\n obj1, feat1 = obj1\n else:\n feat1 = None\n\n if isinstance(obj2, tuple):\n obj2, feat2 = obj2\n else:\n feat2 = None\n\n try:\n return _expectation(p, obj1, feat1, obj2, feat2, nghp=nghp)\n except NotImplementedError as e: # pragma: no cover\n\n warn_msg = \"Quadrature is used to calculate the expectation. \" + str(e)\n logger.warning(warn_msg)\n\n return _quadrature_expectation(p, obj1, feat1, obj2, feat2, nghp)", "response": "Compute the expectation of obj1 obj2 x."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _expectation(p, kern, none1, none2, none3, nghp=None):\n return kern.Kdiag(p.mu)", "response": "Compute the expectation of a kernel"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the expectation for a single entry in the current kernel.", "response": "def _expectation(p, kern, feat, none1, none2, nghp=None):\n \"\"\"\n Compute the expectation:\n _p(X)\n - K_{.,.} :: RBF kernel\n\n :return: NxM\n \"\"\"\n with params_as_tensors_for(kern, feat):\n # use only active dimensions\n Xcov = kern._slice_cov(p.cov)\n Z, Xmu = kern._slice(feat.Z, p.mu)\n D = tf.shape(Xmu)[1]\n if kern.ARD:\n lengthscales = kern.lengthscales\n else:\n lengthscales = tf.zeros((D,), dtype=settings.float_type) + kern.lengthscales\n\n chol_L_plus_Xcov = tf.cholesky(tf.matrix_diag(lengthscales ** 2) + Xcov) # NxDxD\n\n all_diffs = tf.transpose(Z) - tf.expand_dims(Xmu, 2) # NxDxM\n exponent_mahalanobis = tf.matrix_triangular_solve(chol_L_plus_Xcov, all_diffs, lower=True) # NxDxM\n exponent_mahalanobis = tf.reduce_sum(tf.square(exponent_mahalanobis), 1) # NxM\n exponent_mahalanobis = tf.exp(-0.5 * exponent_mahalanobis) # NxM\n\n sqrt_det_L = tf.reduce_prod(lengthscales)\n sqrt_det_L_plus_Xcov = tf.exp(tf.reduce_sum(tf.log(tf.matrix_diag_part(chol_L_plus_Xcov)), axis=1))\n determinants = sqrt_det_L / sqrt_det_L_plus_Xcov # N\n\n return kern.variance * (determinants[:, None] * exponent_mahalanobis)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _expectation(p, mean, none, kern, feat, nghp=None):\n Xmu, Xcov = p.mu, p.cov\n\n with tf.control_dependencies([tf.assert_equal(\n tf.shape(Xmu)[1], tf.constant(kern.input_dim, settings.int_type),\n message=\"Currently cannot handle slicing in exKxz.\")]):\n Xmu = tf.identity(Xmu)\n\n with params_as_tensors_for(kern, feat):\n D = tf.shape(Xmu)[1]\n lengthscales = kern.lengthscales if kern.ARD \\\n else tf.zeros((D,), dtype=settings.float_type) + kern.lengthscales\n\n chol_L_plus_Xcov = tf.cholesky(tf.matrix_diag(lengthscales ** 2) + Xcov) # NxDxD\n all_diffs = tf.transpose(feat.Z) - tf.expand_dims(Xmu, 2) # NxDxM\n\n sqrt_det_L = tf.reduce_prod(lengthscales)\n sqrt_det_L_plus_Xcov = tf.exp(tf.reduce_sum(tf.log(tf.matrix_diag_part(chol_L_plus_Xcov)), axis=1))\n determinants = sqrt_det_L / sqrt_det_L_plus_Xcov # N\n\n exponent_mahalanobis = tf.cholesky_solve(chol_L_plus_Xcov, all_diffs) # NxDxM\n non_exponent_term = tf.matmul(Xcov, exponent_mahalanobis, transpose_a=True)\n non_exponent_term = tf.expand_dims(Xmu, 2) + non_exponent_term # NxDxM\n\n exponent_mahalanobis = tf.reduce_sum(all_diffs * exponent_mahalanobis, 1) # NxM\n exponent_mahalanobis = tf.exp(-0.5 * exponent_mahalanobis) # NxM\n\n return kern.variance * (determinants[:, None] * exponent_mahalanobis)[:, None, :] * non_exponent_term", "response": "Compute the expectation of a single entry in the current kernel."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the expectation of a kernel with the same parameters.", "response": "def _expectation(p, kern1, feat1, kern2, feat2, nghp=None):\n \"\"\"\n Compute the expectation:\n expectation[n] = _p(x_n)\n - Ka_{.,.}, Kb_{.,.} :: RBF kernels\n Ka and Kb as well as Z1 and Z2 can differ from each other.\n\n :return: N x dim(Z1) x dim(Z2)\n \"\"\"\n\n if kern1.on_separate_dims(kern2) and isinstance(p, DiagonalGaussian):\n # no joint expectations required\n eKxz1 = expectation(p, (kern1, feat1))\n eKxz2 = expectation(p, (kern2, feat2))\n return eKxz1[:, :, None] * eKxz2[:, None, :]\n\n Ka, Kb = kern1, kern2\n\n with params_as_tensors_for(Ka, feat1, Kb, feat2):\n\n # use only active dimensions\n Xcov = Ka._slice_cov(tf.matrix_diag(p.cov) if isinstance(p, DiagonalGaussian) else p.cov)\n\n Z1, Xmu = Ka._slice(feat1.Z, p.mu)\n\n N = tf.shape(Xmu)[0]\n D = tf.shape(Xmu)[1]\n\n def get_squared_length_scales(kern):\n squared_lengthscales = kern.lengthscales ** 2. if kern.ARD \\\n else tf.zeros((D,), dtype=settings.float_type) + kern.lengthscales ** 2.\n return squared_lengthscales\n\n if Ka == Kb:\n La = get_squared_length_scales(Ka)\n Lb = La\n half_mean_L = La * 0.5 # average length scale\n else:\n La, Lb = map(get_squared_length_scales, (Ka, Kb))\n half_mean_L = La * Lb / (La + Lb) # average length scale\n\n sqrt_det_L = tf.reduce_prod(half_mean_L) ** 0.5\n C = tf.cholesky(tf.matrix_diag(half_mean_L) + Xcov) # [N, D, D]\n dets = sqrt_det_L / tf.exp(tf.reduce_sum(tf.log(tf.matrix_diag_part(C)), axis=1)) # N\n\n # for mahalanobis computation we need Z\u1d40 (CC\u1d40)\u207b\u00b9 Z as well as C\u207b\u00b9 Z\n # with Z = Z\u2081, Z\u2082 for two rbf kernels\n def get_cholesky_solve_terms(Z, C=C):\n C_inv_z = tf.matrix_triangular_solve(\n C, tf.tile(tf.expand_dims(tf.transpose(Z), 0),\n [N, 1, 1]), lower=True) # [N, D, M]\n z_CC_inv_z = tf.reduce_sum(tf.square(C_inv_z), 1) # [N, M]\n\n return C_inv_z, z_CC_inv_z\n\n C_inv_mu = tf.matrix_triangular_solve(C, tf.expand_dims(Xmu, 2), lower=True) # [N, D, 1]\n mu_CC_inv_mu = tf.expand_dims(tf.reduce_sum(tf.square(C_inv_mu), 1), 2) # [N, 1, 1]\n\n C_inv_z1, z1_CC_inv_z1 = get_cholesky_solve_terms(Z1 / La * half_mean_L)\n z1_CC_inv_mu = 2 * tf.matmul(C_inv_z1, C_inv_mu, transpose_a=True)[:, :, 0] # [N, M1]\n\n if feat1 == feat2 and Ka == Kb:\n # in this case Z2==Z1 so we can reuse the Z1 terms\n C_inv_z2, z2_CC_inv_z2 = C_inv_z1, z1_CC_inv_z1\n z2_CC_inv_mu = z1_CC_inv_mu # [N, M]\n Z2 = Z1\n else:\n # compute terms related to Z2\n Z2, _ = Kb._slice(feat2.Z, p.mu)\n C_inv_z2, z2_CC_inv_z2 = get_cholesky_solve_terms(Z2 / Lb * half_mean_L)\n z2_CC_inv_mu = 2 * tf.matmul(C_inv_z2, C_inv_mu, transpose_a=True)[:, :, 0] # [N, M2]\n\n z1_CC_inv_z2 = tf.matmul(C_inv_z1, C_inv_z2, transpose_a=True) # [N, M1, M2]\n\n # expand dims for broadcasting\n # along M1\n z2_CC_inv_mu = tf.expand_dims(z2_CC_inv_mu, 1) # [N, 1, M2]\n z2_CC_inv_z2 = tf.expand_dims(z2_CC_inv_z2, 1)\n\n # along M2\n z1_CC_inv_mu = tf.expand_dims(z1_CC_inv_mu, 2) # [N, M1, 1]\n z1_CC_inv_z1 = tf.expand_dims(z1_CC_inv_z1, 2)\n\n # expanded version of ((Z1 + Z2)-mu) (CCT)-1 ((Z1 + Z2)-mu)\n mahalanobis = mu_CC_inv_mu + z2_CC_inv_z2 + \\\n z1_CC_inv_z1 + 2 * z1_CC_inv_z2 - \\\n z1_CC_inv_mu - z2_CC_inv_mu # [N, M1, M2]\n\n exp_mahalanobis = tf.exp(-0.5 * mahalanobis) # [N, M1, M2]\n\n if Z1 == Z2:\n # CAVEAT : Compute sqrt(self.K(Z)) explicitly\n # to prevent automatic gradient from\n # being NaN sometimes, see pull request #615\n sqrt_exp_dist = tf.exp(-0.25 * Ka.scaled_square_dist(Z1, None))\n else:\n # Compute exp( -.5 (Z-Z')^top (L_1+L_2)^{-1} (Z-Z') )\n lengthscales_rms = tf.sqrt(La + Lb)\n Z1 = Z1 / lengthscales_rms\n Z1sqr = tf.reduce_sum(tf.square(Z1), axis=1)\n Z2 = Z2 / lengthscales_rms\n Z2sqr = tf.reduce_sum(tf.square(Z2), axis=1)\n dist = -2 * tf.matmul(Z1, Z2, transpose_b=True) \\\n + tf.reshape(Z1sqr, (-1, 1)) + tf.reshape(Z2sqr, (1, -1))\n sqrt_exp_dist = tf.exp(-0.5 * dist) # M1 x M2\n\n return Ka.variance * Kb.variance * sqrt_exp_dist * \\\n tf.reshape(dets, [N, 1, 1]) * exp_mahalanobis"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _expectation(p, kern, none1, none2, none3, nghp=None):\n with params_as_tensors_for(kern):\n # use only active dimensions\n Xmu, _ = kern._slice(p.mu, None)\n Xcov = kern._slice_cov(p.cov)\n\n return tf.reduce_sum(kern.variance * (tf.matrix_diag_part(Xcov) + tf.square(Xmu)), 1)", "response": "Compute the expectation of a n - tuple kernel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the expectation for a given feature.", "response": "def _expectation(p, kern, feat, none1, none2, nghp=None):\n \"\"\"\n Compute the expectation:\n _p(X)\n - K_{.,.} :: Linear kernel\n\n :return: NxM\n \"\"\"\n with params_as_tensors_for(kern, feat):\n # use only active dimensions\n Z, Xmu = kern._slice(feat.Z, p.mu)\n\n return tf.matmul(Xmu, Z * kern.variance, transpose_b=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the expectation of a single entry in a Markov kernel.", "response": "def _expectation(p, kern, feat, mean, none, nghp=None):\n \"\"\"\n Compute the expectation:\n expectation[n] = _p(x_{n:n+1})\n - K_{.,.} :: Linear kernel\n - p :: MarkovGaussian distribution (p.cov 2x(N+1)xDxD)\n\n :return: NxMxD\n \"\"\"\n Xmu, Xcov = p.mu, p.cov\n\n with tf.control_dependencies([tf.assert_equal(\n tf.shape(Xmu)[1], tf.constant(kern.input_dim, settings.int_type),\n message=\"Currently cannot handle slicing in exKxz.\")]):\n Xmu = tf.identity(Xmu)\n\n with params_as_tensors_for(kern, feat):\n N = tf.shape(Xmu)[0] - 1\n var_Z = kern.variance * feat.Z # MxD\n tiled_Z = tf.tile(tf.expand_dims(var_Z, 0), (N, 1, 1)) # NxMxD\n eXX = Xcov[1, :-1] + (Xmu[:-1][..., None] * Xmu[1:][:, None, :]) # NxDxD\n return tf.matmul(tiled_Z, eXX)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _expectation(p, kern1, feat1, kern2, feat2, nghp=None):\n if kern1.on_separate_dims(kern2) and isinstance(p, DiagonalGaussian): # no joint expectations required\n eKxz1 = expectation(p, (kern1, feat1))\n eKxz2 = expectation(p, (kern2, feat2))\n return eKxz1[:, :, None] * eKxz2[:, None, :]\n\n if kern1 != kern2 or feat1 != feat2:\n raise NotImplementedError(\"The expectation over two kernels has only an \"\n \"analytical implementation if both kernels are equal.\")\n\n kern = kern1\n feat = feat1\n\n with params_as_tensors_for(kern, feat):\n # use only active dimensions\n Xcov = kern._slice_cov(tf.matrix_diag(p.cov) if isinstance(p, DiagonalGaussian) else p.cov)\n Z, Xmu = kern._slice(feat.Z, p.mu)\n\n N = tf.shape(Xmu)[0]\n var_Z = kern.variance * Z\n tiled_Z = tf.tile(tf.expand_dims(var_Z, 0), (N, 1, 1)) # NxMxD\n XX = Xcov + tf.expand_dims(Xmu, 1) * tf.expand_dims(Xmu, 2) # NxDxD\n return tf.matmul(tf.matmul(tiled_Z, XX), tiled_Z, transpose_b=True)", "response": "Compute the expectation of two kernels."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the expectation for a given kernel.", "response": "def _expectation(p, mean, none, kern, feat, nghp=None):\n \"\"\"\n Compute the expectation:\n expectation[n] = _p(x_n)\n - K_{.,} :: Linear kernel\n or the equivalent for MarkovGaussian\n\n :return: NxDxM\n \"\"\"\n return tf.matrix_transpose(expectation(p, (kern, feat), mean))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the expectation of a given parameter", "response": "def _expectation(p, constant_mean, none, kern, feat, nghp=None):\n \"\"\"\n Compute the expectation:\n expectation[n] = _p(x_n)\n - m(x_i) = c :: Constant function\n - K_{.,.} :: Kernel function\n\n :return: NxQxM\n \"\"\"\n with params_as_tensors_for(constant_mean):\n c = constant_mean(p.mu) # NxQ\n eKxz = expectation(p, (kern, feat), nghp=nghp) # NxM\n\n return c[..., None] * eKxz[:, None, :]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _expectation(p, linear_mean, none, kern, feat, nghp=None):\n with params_as_tensors_for(linear_mean):\n N = p.mu.shape[0].value\n D = p.mu.shape[1].value\n exKxz = expectation(p, mean_functions.Identity(D), (kern, feat), nghp=nghp)\n eKxz = expectation(p, (kern, feat), nghp=nghp)\n eAxKxz = tf.matmul(tf.tile(linear_mean.A[None, :, :], (N, 1, 1)),\n exKxz, transpose_a=True)\n ebKxz = linear_mean.b[None, :, None] * eKxz[:, None, :]\n return eAxKxz + ebKxz", "response": "Compute the expectation of a NxM object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the expectation of the next iteration of the sequence.", "response": "def _expectation(p, mean, none1, none2, none3, nghp=None):\n \"\"\"\n Compute the expectation:\n _p(X)\n - m(x) :: Linear, Identity or Constant mean function\n\n :return: NxQ\n \"\"\"\n return mean(p.mu)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _expectation(p, mean1, none1, mean2, none2, nghp=None):\n return mean1(p.mu)[:, :, None] * mean2(p.mu)[:, None, :]", "response": "Compute the expectation of a sequence of parameters."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _expectation(p, mean1, none1, mean2, none2, nghp=None):\n with params_as_tensors_for(mean2):\n N = tf.shape(p.mu)[0]\n e_xxt = p.cov + (p.mu[:, :, None] * p.mu[:, None, :]) # NxDxD\n e_xxt_A = tf.matmul(e_xxt, tf.tile(mean2.A[None, ...], (N, 1, 1))) # NxDxQ\n e_x_bt = p.mu[:, :, None] * mean2.b[None, None, :] # NxDxQ\n\n return e_xxt_A + e_x_bt", "response": "Compute the expectation of a cluster."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the expectation for a single resource.", "response": "def _expectation(p, mean1, none1, mean2, none2, nghp=None):\n \"\"\"\n Compute the expectation:\n expectation[n] = _p(x_n)\n - m1(.) :: Linear mean function\n - m2(.) :: Identity mean function\n\n :return: NxQxD\n \"\"\"\n with params_as_tensors_for(mean1):\n N = tf.shape(p.mu)[0]\n e_xxt = p.cov + (p.mu[:, :, None] * p.mu[:, None, :]) # NxDxD\n e_A_xxt = tf.matmul(tf.tile(mean1.A[None, ...], (N, 1, 1)), e_xxt, transpose_a=True) # NxQxD\n e_b_xt = mean1.b[None, :, None] * p.mu[:, None, :] # NxQxD\n\n return e_A_xxt + e_b_xt"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _expectation(p, mean1, none1, mean2, none2, nghp=None):\n with params_as_tensors_for(mean1, mean2):\n e_xxt = p.cov + (p.mu[:, :, None] * p.mu[:, None, :]) # NxDxD\n e_A1t_xxt_A2 = tf.einsum(\"iq,nij,jz->nqz\", mean1.A, e_xxt, mean2.A) # NxQ1xQ2\n e_A1t_x_b2t = tf.einsum(\"iq,ni,z->nqz\", mean1.A, p.mu, mean2.b) # NxQ1xQ2\n e_b1_xt_A2 = tf.einsum(\"q,ni,iz->nqz\", mean1.b, p.mu, mean2.A) # NxQ1xQ2\n e_b1_b2t = mean1.b[:, None] * mean2.b[None, :] # Q1xQ2\n\n return e_A1t_xxt_A2 + e_A1t_x_b2t + e_b1_xt_A2 + e_b1_b2t", "response": "Compute the expectation of a cluster."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _expectation(p, mean, none, kern, feat, nghp=None):\n return functools.reduce(tf.add, [\n expectation(p, mean, (k, feat), nghp=nghp) for k in kern.kernels])", "response": "r Compute the expectation for the NxQxM\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _expectation(p, kern1, feat1, kern2, feat2, nghp=None):\n crossexps = []\n\n if kern1 == kern2 and feat1 == feat2: # avoid duplicate computation by using transposes\n for i, k1 in enumerate(kern1.kernels):\n crossexps.append(expectation(p, (k1, feat1), (k1, feat1), nghp=nghp))\n\n for k2 in kern1.kernels[:i]:\n eKK = expectation(p, (k1, feat1), (k2, feat2), nghp=nghp)\n eKK += tf.matrix_transpose(eKK)\n crossexps.append(eKK)\n else:\n for k1, k2 in it.product(kern1.kernels, kern2.kernels):\n crossexps.append(expectation(p, (k1, feat1), (k2, feat2), nghp=nghp))\n\n return functools.reduce(tf.add, crossexps)", "response": "r Compute the expectation of a two kernels."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _expectation(p, rbf_kern, feat1, lin_kern, feat2, nghp=None):\n if rbf_kern.on_separate_dims(lin_kern) and isinstance(p, DiagonalGaussian): # no joint expectations required\n eKxz1 = expectation(p, (rbf_kern, feat1))\n eKxz2 = expectation(p, (lin_kern, feat2))\n return eKxz1[:, :, None] * eKxz2[:, None, :]\n\n if feat1 != feat2:\n raise NotImplementedError(\"Features have to be the same for both kernels.\")\n\n if rbf_kern.active_dims != lin_kern.active_dims:\n raise NotImplementedError(\"active_dims have to be the same for both kernels.\")\n\n with params_as_tensors_for(rbf_kern, lin_kern, feat1, feat2):\n # use only active dimensions\n Xcov = rbf_kern._slice_cov(tf.matrix_diag(p.cov) if isinstance(p, DiagonalGaussian) else p.cov)\n Z, Xmu = rbf_kern._slice(feat1.Z, p.mu)\n\n N = tf.shape(Xmu)[0]\n D = tf.shape(Xmu)[1]\n\n lin_kern_variances = lin_kern.variance if lin_kern.ARD \\\n else tf.zeros((D,), dtype=settings.float_type) + lin_kern.variance\n\n rbf_kern_lengthscales = rbf_kern.lengthscales if rbf_kern.ARD \\\n else tf.zeros((D,), dtype=settings.float_type) + rbf_kern.lengthscales ## Begin RBF eKxz code:\n\n chol_L_plus_Xcov = tf.cholesky(tf.matrix_diag(rbf_kern_lengthscales ** 2) + Xcov) # NxDxD\n\n Z_transpose = tf.transpose(Z)\n all_diffs = Z_transpose - tf.expand_dims(Xmu, 2) # NxDxM\n exponent_mahalanobis = tf.matrix_triangular_solve(chol_L_plus_Xcov, all_diffs, lower=True) # NxDxM\n exponent_mahalanobis = tf.reduce_sum(tf.square(exponent_mahalanobis), 1) # NxM\n exponent_mahalanobis = tf.exp(-0.5 * exponent_mahalanobis) # NxM\n\n sqrt_det_L = tf.reduce_prod(rbf_kern_lengthscales)\n sqrt_det_L_plus_Xcov = tf.exp(tf.reduce_sum(tf.log(tf.matrix_diag_part(chol_L_plus_Xcov)), axis=1))\n determinants = sqrt_det_L / sqrt_det_L_plus_Xcov # N\n eKxz_rbf = rbf_kern.variance * (determinants[:, None] * exponent_mahalanobis) ## NxM <- End RBF eKxz code\n\n tiled_Z = tf.tile(tf.expand_dims(Z_transpose, 0), (N, 1, 1)) # NxDxM\n z_L_inv_Xcov = tf.matmul(tiled_Z, Xcov / rbf_kern_lengthscales[:, None] ** 2., transpose_a=True) # NxMxD\n\n cross_eKzxKxz = tf.cholesky_solve(\n chol_L_plus_Xcov, (lin_kern_variances * rbf_kern_lengthscales ** 2.)[..., None] * tiled_Z) # NxDxM\n\n cross_eKzxKxz = tf.matmul((z_L_inv_Xcov + Xmu[:, None, :]) * eKxz_rbf[..., None], cross_eKzxKxz) # NxMxM\n return cross_eKzxKxz", "response": "Compute the expectation of a single kernel."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _expectation(p, lin_kern, feat1, rbf_kern, feat2, nghp=None):\n return tf.matrix_transpose(expectation(p, (rbf_kern, feat2), (lin_kern, feat1)))", "response": "Compute the expectation matrix for a single kernel."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _expectation(p, kern, none1, none2, none3, nghp=None):\n if not kern.on_separate_dimensions:\n raise NotImplementedError(\n \"Product currently needs to be defined on separate dimensions.\") # pragma: no cover\n\n return functools.reduce(tf.multiply, [\n expectation(p, k, nghp=nghp) for k in kern.kernels])", "response": "r Compute the expectation for a given kernel."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _expectation(p, obj1, feat1, obj2, feat2, nghp=None):\n if obj2 is None:\n gaussian = Gaussian(p.mu[:-1], p.cov[0, :-1])\n return expectation(gaussian, (obj1, feat1), nghp=nghp)\n elif obj1 is None:\n gaussian = Gaussian(p.mu[1:], p.cov[0, 1:])\n return expectation(gaussian, (obj2, feat2), nghp=nghp)\n else:\n return expectation(p, (obj1, feat1), (obj2, feat2), nghp=nghp)", "response": "This function is used to compute the expectation of two objects."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates samples from a single model using a straight - forward HMC implementation.", "response": "def sample(self, model, num_samples, epsilon,\n lmin=1, lmax=1, thin=1, burn=0,\n session=None, initialize=True, anchor=True,\n logprobs=True):\n \"\"\"\n A straight-forward HMC implementation. The mass matrix is assumed to be the\n identity.\n\n The gpflow model must implement `build_objective` method to build `f` function\n (tensor) which in turn based on model's internal trainable parameters `x`.\n\n f(x) = E(x)\n\n we then generate samples from the distribution\n\n pi(x) = exp(-E(x))/Z\n\n The total number of iterations is given by:\n\n burn + thin * num_samples\n\n The leafrog (Verlet) integrator works by picking a random number of steps\n uniformly between lmin and lmax, and taking steps of length epsilon.\n\n :param model: gpflow model with `build_objective` method implementation.\n :param num_samples: number of samples to generate.\n :param epsilon: HMC tuning parameter - stepsize.\n :param lmin: HMC tuning parameter - lowest integer `a` of uniform `[a, b]` distribution\n used for drawing number of leapfrog iterations.\n :param lmax: HMC tuning parameter - largest integer `b` from uniform `[a, b]` distribution\n used for drawing number of leapfrog iterations.\n :param thin: an integer which specifies the thinning interval.\n :param burn: an integer which specifies how many initial samples to discard.\n :param session: TensorFlow session. The default session or cached GPflow session\n will be used if it is none.\n :param initialize: indication either TensorFlow initialization is required or not.\n :param anchor: dump live trainable values computed within specified TensorFlow\n session to actual parameters (in python scope).\n :param logprobs: indicates either logprob values shall be included in output or not.\n\n :return: data frame with `num_samples` traces, where columns are full names of\n trainable parameters except last column, which is `logprobs`.\n Trainable parameters are represented as constrained values in output.\n\n :raises: ValueError exception in case when wrong parameter ranges were passed.\n \"\"\"\n\n if lmax <= 0 or lmin <= 0:\n raise ValueError('The lmin and lmax parameters must be greater zero.')\n if thin <= 0:\n raise ValueError('The thin parameter must be greater zero.')\n if burn < 0:\n raise ValueError('The burn parameter must be equal or greater zero.')\n\n lmax += 1\n session = model.enquire_session(session)\n\n model.initialize(session=session, force=initialize)\n\n with tf.name_scope('hmc'):\n params = list(model.trainable_parameters)\n xs = list(model.trainable_tensors)\n\n def logprob_grads():\n logprob = tf.negative(model.build_objective())\n grads = tf.gradients(logprob, xs)\n return logprob, grads\n\n thin_args = [logprob_grads, xs, thin, epsilon, lmin, lmax]\n\n if burn > 0:\n burn_op = _burning(burn, *thin_args)\n session.run(burn_op, feed_dict=model.feeds)\n\n xs_dtypes = _map(lambda x: x.dtype, xs)\n logprob_dtype = model.objective.dtype\n dtypes = _flat(xs_dtypes, [logprob_dtype])\n indices = np.arange(num_samples)\n\n def map_body(_):\n xs_sample, logprob_sample = _thinning(*thin_args)\n return _flat(xs_sample, [logprob_sample])\n\n hmc_output = tf.map_fn(map_body, indices, dtype=dtypes,\n back_prop=False, parallel_iterations=1)\n with tf.control_dependencies(hmc_output):\n unconstrained_trace, logprob_trace = hmc_output[:-1], hmc_output[-1]\n constrained_trace = _map(lambda x, param: param.transform.forward_tensor(x),\n unconstrained_trace, params)\n hmc_output = constrained_trace + [logprob_trace]\n\n names = [param.pathname for param in params]\n raw_traces = session.run(hmc_output, feed_dict=model.feeds)\n\n if anchor:\n model.anchor(session)\n\n traces = dict(zip(names, map(list, raw_traces[:-1])))\n if logprobs:\n traces.update({'logprobs': raw_traces[-1]})\n return pd.DataFrame(traces)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef predict_density(self, Fmu, Fvar, Y):\n return ndiagquad(self.logp,\n self.num_gauss_hermite_points,\n Fmu, Fvar, logspace=True, Y=Y)", "response": "r This method computes the log predictive density of a specific object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef variational_expectations(self, Fmu, Fvar, Y):\n return ndiagquad(self.logp,\n self.num_gauss_hermite_points,\n Fmu, Fvar, Y=Y)", "response": "r Compute the expected log density of the data given a Gaussian\n distribution for the function values Fmu and Fvar and Y."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _partition_and_stitch(self, args, func_name):\n # get the index from Y\n Y = args[-1]\n ind = Y[:, -1]\n ind = tf.cast(ind, tf.int32)\n Y = Y[:, :-1]\n args[-1] = Y\n\n # split up the arguments into chunks corresponding to the relevant likelihoods\n args = zip(*[tf.dynamic_partition(X, ind, self.num_likelihoods) for X in args])\n\n # apply the likelihood-function to each section of the data\n with params_as_tensors_for(self, convert=False):\n funcs = [getattr(lik, func_name) for lik in self.likelihood_list]\n results = [f(*args_i) for f, args_i in zip(funcs, args)]\n\n # stitch the results back together\n partitions = tf.dynamic_partition(tf.range(0, tf.size(ind)), ind, self.num_likelihoods)\n results = tf.dynamic_stitch(partitions, results)\n\n return results", "response": "This function splits up the args into partitions and stitches the results back together."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _make_phi(self, F):\n scaled_bins_left = tf.concat([self.bin_edges / self.sigma, np.array([np.inf])], 0)\n scaled_bins_right = tf.concat([np.array([-np.inf]), self.bin_edges / self.sigma], 0)\n return inv_probit(scaled_bins_left - tf.reshape(F, (-1, 1)) / self.sigma) \\\n - inv_probit(scaled_bins_right - tf.reshape(F, (-1, 1)) / self.sigma)", "response": "Constructs a probability matrix where each row output the probability of the corresponding\n label and each row output the probability of the corresponding\n label and the rows match the entries of F."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef predict_mean_and_var(self, Fmu, Fvar, epsilon=None):\n integrand2 = lambda *X: self.conditional_variance(*X) + tf.square(self.conditional_mean(*X))\n E_y, E_y2 = self._mc_quadrature([self.conditional_mean, integrand2],\n Fmu, Fvar, epsilon=epsilon)\n V_y = E_y2 - tf.square(E_y)\n return E_y, V_y", "response": "r Given a Normal distribution for the latent function Fmu and Fvar return the mean and variance of Y\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nabsorbs a Z^t gate into a W", "response": "def _absorb_z_into_w(moment_index: int,\n op: ops.Operation,\n state: _OptimizerState) -> None:\n \"\"\"Absorbs a Z^t gate into a W(a) flip.\n\n [Where W(a) is shorthand for PhasedX(phase_exponent=a).]\n\n Uses the following identity:\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500Z^t\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500W(a)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500Z^t/2\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500Z^t/2\u2500\u2500\u2500 (split Z)\n \u2261 \u2500\u2500\u2500W(a)\u2500\u2500\u2500W(a)\u2500\u2500\u2500Z^-t/2\u2500\u2500\u2500W(a)\u2500\u2500\u2500Z^t/2\u2500\u2500\u2500 (flip Z)\n \u2261 \u2500\u2500\u2500W(a)\u2500\u2500\u2500W(a)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500W(a+t/2)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 (phase W)\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500W(a+t/2)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 (cancel Ws)\n \u2261 \u2500\u2500\u2500W(a+t/2)\u2500\u2500\u2500\n \"\"\"\n t = cast(float, _try_get_known_z_half_turns(op))\n q = op.qubits[0]\n state.held_w_phases[q] = cast(float, state.held_w_phases[q]) + t / 2\n state.deletions.append((moment_index, op))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if a held W gate crosses the whole W gate.", "response": "def _potential_cross_whole_w(moment_index: int,\n op: ops.Operation,\n tolerance: float,\n state: _OptimizerState) -> None:\n \"\"\"Grabs or cancels a held W gate against an existing W gate.\n\n [Where W(a) is shorthand for PhasedX(phase_exponent=a).]\n\n Uses the following identity:\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500W(b)\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500X\u2500\u2500\u2500Z^a\u2500\u2500\u2500Z^-b\u2500\u2500\u2500X\u2500\u2500\u2500Z^b\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500Z^-a\u2500\u2500\u2500Z^b\u2500\u2500\u2500X\u2500\u2500\u2500X\u2500\u2500\u2500Z^b\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500Z^-a\u2500\u2500\u2500Z^b\u2500\u2500\u2500Z^b\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500Z^2(b-a)\u2500\u2500\u2500\n \"\"\"\n state.deletions.append((moment_index, op))\n\n _, phase_exponent = cast(Tuple[float, float],\n _try_get_known_phased_pauli(op))\n q = op.qubits[0]\n a = state.held_w_phases.get(q)\n b = phase_exponent\n\n if a is None:\n # Collect the gate.\n state.held_w_phases[q] = b\n else:\n # Cancel the gate.\n state.held_w_phases[q] = None\n t = 2*(b - a)\n if not decompositions.is_negligible_turn(t / 2, tolerance):\n leftover_phase = ops.Z(q)**t\n state.inline_intos.append((moment_index, leftover_phase))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncross the held W over a partial W gate.", "response": "def _potential_cross_partial_w(moment_index: int,\n op: ops.Operation,\n state: _OptimizerState) -> None:\n \"\"\"Cross the held W over a partial W gate.\n\n [Where W(a) is shorthand for PhasedX(phase_exponent=a).]\n\n Uses the following identity:\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500W(b)^t\u2500\u2500\u2500\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500X\u2500\u2500\u2500Z^a\u2500\u2500\u2500W(b)^t\u2500\u2500\u2500\u2500\u2500\u2500 (expand W(a))\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500X\u2500\u2500\u2500W(b-a)^t\u2500\u2500\u2500Z^a\u2500\u2500\u2500\u2500 (move Z^a across, phasing axis)\n \u2261 \u2500\u2500\u2500Z^-a\u2500\u2500\u2500W(a-b)^t\u2500\u2500\u2500X\u2500\u2500\u2500Z^a\u2500\u2500\u2500\u2500 (move X across, negating axis angle)\n \u2261 \u2500\u2500\u2500W(2a-b)^t\u2500\u2500\u2500Z^-a\u2500\u2500\u2500X\u2500\u2500\u2500Z^a\u2500\u2500\u2500 (move Z^-a across, phasing axis)\n \u2261 \u2500\u2500\u2500W(2a-b)^t\u2500\u2500\u2500W(a)\u2500\u2500\u2500\n \"\"\"\n a = state.held_w_phases.get(op.qubits[0])\n if a is None:\n return\n exponent, phase_exponent = cast(Tuple[float, float],\n _try_get_known_phased_pauli(op))\n new_op = ops.PhasedXPowGate(\n exponent=exponent,\n phase_exponent=2 * a - phase_exponent).on(op.qubits[0])\n state.deletions.append((moment_index, op))\n state.inline_intos.append((moment_index, new_op))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncrossing exactly one W flip over a partial CZ.", "response": "def _single_cross_over_cz(moment_index: int,\n op: ops.Operation,\n qubit_with_w: ops.Qid,\n state: _OptimizerState) -> None:\n \"\"\"Crosses exactly one W flip over a partial CZ.\n\n [Where W(a) is shorthand for PhasedX(phase_exponent=a).]\n\n Uses the following identity:\n\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\n \u2502\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500@^t\u2500\u2500\u2500\n\n\n \u2261 \u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500O\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n | | \u2502 (split into on/off cases)\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500W(a)\u2500\u2500\u2500@^t\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n\n \u2261 \u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500O\u2500\u2500\u2500\u2500\u2500\u2500\n | \u2502 | (off doesn't interact with on)\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^t\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500W(a)\u2500\u2500\u2500\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500Z^t\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500O\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 | | (crossing causes kickback)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^-t\u2500\u2500\u2500W(a)\u2500\u2500\u2500W(a)\u2500\u2500\u2500 (X Z^t X Z^-t = exp(pi t) I)\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500Z^t\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 (merge on/off cases)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^-t\u2500\u2500\u2500W(a)\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n\n \u2261 \u2500\u2500\u2500Z^t\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^-t\u2500\u2500\u2500W(a)\u2500\u2500\u2500\u2500\n \"\"\"\n t = cast(float, _try_get_known_cz_half_turns(op))\n other_qubit = op.qubits[0] if qubit_with_w == op.qubits[1] else op.qubits[1]\n negated_cz = ops.CZ(*op.qubits)**-t\n kickback = ops.Z(other_qubit)**t\n\n state.deletions.append((moment_index, op))\n state.inline_intos.append((moment_index, negated_cz))\n state.insertions.append((moment_index, kickback))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _double_cross_over_cz(op: ops.Operation,\n state: _OptimizerState) -> None:\n \"\"\"Crosses two W flips over a partial CZ.\n\n [Where W(a) is shorthand for PhasedX(phase_exponent=a).]\n\n Uses the following identity:\n\n \u2500\u2500\u2500W(a)\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\n \u2502\n \u2500\u2500\u2500W(b)\u2500\u2500\u2500@^t\u2500\u2500\u2500\n\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500W(a)\u2500\u2500\u2500\n \u2502 (single-cross top W over CZ)\n \u2500\u2500\u2500W(b)\u2500\u2500\u2500@^-t\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500Z^t\u2500\u2500\u2500\u2500\n\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500Z^-t\u2500\u2500\u2500W(a)\u2500\u2500\u2500\n \u2502 (single-cross bottom W over CZ)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^t\u2500\u2500\u2500W(b)\u2500\u2500\u2500Z^t\u2500\u2500\u2500\u2500\n\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500W(a)\u2500\u2500\u2500Z^t\u2500\u2500\u2500\u2500\n \u2502 (flip over Z^-t)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^t\u2500\u2500\u2500W(b)\u2500\u2500\u2500Z^t\u2500\u2500\u2500\u2500\n\n\n \u2261 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500W(a+t/2)\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 (absorb Zs into Ws)\n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@^t\u2500\u2500\u2500W(b+t/2)\u2500\u2500\u2500\u2500\u2500\u2500\n\n \u2261 \u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500W(a+t/2)\u2500\u2500\u2500\n \u2502\n \u2500\u2500\u2500@^t\u2500\u2500\u2500W(b+t/2)\u2500\u2500\u2500\n \"\"\"\n t = cast(float, _try_get_known_cz_half_turns(op))\n for q in op.qubits:\n state.held_w_phases[q] = cast(float, state.held_w_phases[q]) + t / 2", "response": "Crosses two W flips over a partial CZ."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_probability(p: float, p_str: str) -> float:\n if p < 0:\n raise ValueError('{} was less than 0.'.format(p_str))\n elif p > 1:\n raise ValueError('{} was greater than 1.'.format(p_str))\n return p", "response": "Validates that a probability is between 0 and 1 inclusively."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef phase_by(val: Any, phase_turns: float, qubit_index: int,\n default: TDefault = RaiseTypeErrorIfNotProvided):\n \"\"\"Returns a phased version of the effect.\n\n For example, an X gate phased by 90 degrees would be a Y gate.\n This works by calling `val`'s _phase_by_ method and returning\n the result.\n\n Args:\n val: The value to describe with a unitary matrix.\n phase_turns: The amount to phase the gate, in fractions of a whole\n turn. Divide by 2pi to get radians.\n qubit_index: The index of the target qubit the phasing applies to. For\n operations this is the index of the qubit within the operation's\n qubit list. For gates it's the index of the qubit within the tuple\n of qubits taken by the gate's `on` method.\n default: The default value to return if `val` can't be phased. If not\n specified, an error is raised when `val` can't be phased.\n\n Returns:\n If `val` has a _phase_by_ method and its result is not NotImplemented,\n that result is returned. Otherwise, the function will return the\n default value provided or raise a TypeError if none was provided.\n\n Raises:\n TypeError:\n `val` doesn't have a _phase_by_ method (or that method returned\n NotImplemented) and no `default` was specified.\n \"\"\"\n getter = getattr(val, '_phase_by_', None)\n result = NotImplemented if getter is None else getter(\n phase_turns, qubit_index)\n\n if result is not NotImplemented:\n return result\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if getter is None:\n raise TypeError(\"object of type '{}' \"\n \"has no _phase_by_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _phase_by_ method, \"\n \"but it returned NotImplemented.\".format(type(val)))", "response": "Returns a phased version of the effect."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef on_each(self, *targets: raw_types.Qid) -> op_tree.OP_TREE:\n return [self.on(target) for target in targets]", "response": "Returns a list of operations applied this gate to each of the target qubits."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef engine_from_environment() -> Engine:\n api_key = os.environ.get(ENV_API_KEY)\n if not api_key:\n raise EnvironmentError(\n 'Environment variable {} is not set.'.format(ENV_API_KEY))\n\n default_project_id = os.environ.get(ENV_DEFAULT_PROJECT_ID)\n\n return Engine(api_key=api_key, default_project_id=default_project_id)", "response": "Returns an Engine instance configured using the environment variables."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot_state_histogram(result: trial_result.TrialResult) -> np.ndarray:\n\n # pyplot import is deferred because it requires a system dependency\n # (python3-tk) that `python -m pip install cirq` can't handle for the user.\n # This allows cirq to be usable without python3-tk.\n import matplotlib.pyplot as plt\n\n num_qubits = len(result.measurements.keys())\n states = 2**num_qubits\n values = np.zeros(states)\n\n # measurements is a dict of {measurement gate key:\n # array(repetitions, boolean result)}\n # Convert this to an array of repetitions, each with an array of booleans.\n # e.g. {q1: array([[True, True]]), q2: array([[False, False]])}\n # --> array([[True, False], [True, False]])\n measurement_by_result = np.array([\n v.transpose()[0] for k, v in result.measurements.items()]).transpose()\n\n for meas in measurement_by_result:\n # Convert each array of booleans to a string representation.\n # e.g. [True, False] -> [1, 0] -> '10' -> 2\n state_ind = int(''.join([str(x) for x in [int(x) for x in meas]]), 2)\n values[state_ind] += 1\n\n plot_labels = [bin(x)[2:].zfill(num_qubits) for x in range(states)]\n plt.bar(np.arange(states), values, tick_label=plot_labels)\n plt.xlabel('qubit state')\n plt.ylabel('result count')\n plt.show()\n\n return values", "response": "Plot the state histogram from a single result with repetitions."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a sweep into an equivalent protobuf representation.", "response": "def sweep_to_proto_dict(sweep: Sweep, repetitions: int=1) -> Dict:\n \"\"\"Converts sweep into an equivalent protobuf representation.\"\"\"\n msg = {} # type: Dict\n if not sweep == UnitSweep:\n sweep = _to_zip_product(sweep)\n msg['sweep'] = {\n 'factors': [_sweep_zip_to_proto_dict(cast(Zip, factor)) for factor\n in sweep.factors]\n }\n msg['repetitions'] = repetitions\n return msg"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _to_zip_product(sweep: Sweep) -> Product:\n if not isinstance(sweep, Product):\n sweep = Product(sweep)\n if not all(isinstance(f, Zip) for f in sweep.factors):\n factors = [f if isinstance(f, Zip) else Zip(f) for f in sweep.factors]\n sweep = Product(*factors)\n for factor in sweep.factors:\n for term in cast(Zip, factor).sweeps:\n if not isinstance(term, SingleSweep):\n raise ValueError('cannot convert to zip-product form: {}'\n .format(sweep))\n return sweep", "response": "Converts sweep to a product of zips of single sweeps if possible."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef remove_redundant_acquaintance_opportunities(\n strategy: circuits.Circuit) -> int:\n \"\"\"Removes redundant acquaintance opportunities.\"\"\"\n if not is_acquaintance_strategy(strategy):\n raise TypeError('not is_acquaintance_strategy(circuit)')\n\n qubits = sorted(strategy.all_qubits())\n mapping = {q: i for i, q in enumerate(qubits)}\n\n expose_acquaintance_gates(strategy)\n annotated_strategy = strategy.copy()\n LogicalAnnotator(mapping)(annotated_strategy)\n\n new_moments = [] # type: List[ops.Moment]\n acquaintance_opps = set() # type: Set[FrozenSet[int]]\n n_removed = 0\n for moment in annotated_strategy:\n new_moment = [] # type: List[ops.Operation]\n for op in moment:\n if isinstance(op, AcquaintanceOperation):\n opp = frozenset(cast(Sequence[int], op.logical_indices))\n if opp not in acquaintance_opps:\n acquaintance_opps.add(opp)\n new_moment.append(acquaint(*op.qubits))\n else:\n n_removed += 1\n else:\n new_moment.append(op)\n new_moments.append(ops.Moment(new_moment))\n strategy._moments = new_moments\n return n_removed", "response": "Removes redundant acquaintance opportunities."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nruns the supplied Circuit or Schedule and returns a list of TrialResult objects.", "response": "def run_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: study.Sweepable,\n repetitions: int = 1,\n ) -> List[study.TrialResult]:\n \"\"\"Runs the supplied Circuit or Schedule, mimicking quantum hardware.\n\n In contrast to run, this allows for sweeping over different parameter\n values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n repetitions: The number of repetitions to simulate.\n\n Returns:\n TrialResult list for this run; one for each possible parameter\n resolver.\n \"\"\"\n circuit = (program if isinstance(program, circuits.Circuit)\n else program.to_circuit())\n param_resolvers = study.to_resolvers(params)\n\n trial_results = [] # type: List[study.TrialResult]\n for param_resolver in param_resolvers:\n measurements = self._run(circuit=circuit,\n param_resolver=param_resolver,\n repetitions=repetitions)\n trial_results.append(study.TrialResult(params=param_resolver,\n repetitions=repetitions,\n measurements=measurements))\n return trial_results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _run(\n self,\n circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n repetitions: int\n ) -> Dict[str, np.ndarray]:\n \"\"\"Run a simulation, mimicking quantum hardware.\n\n Args:\n circuit: The circuit to simulate.\n param_resolver: Parameters to run with the program.\n repetitions: Number of times to repeat the run.\n\n Returns:\n A dictionary from measurement gate key to measurement\n results. Measurement results are stored in a 2-dimensional\n numpy array, the first dimension corresponding to the repetition\n and the second to the actual boolean measurement results (ordered\n by the qubits being measured.)\n \"\"\"\n raise NotImplementedError()", "response": "Runs a simulation on the given time series."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing SamplesDisplays in the supplied Circuit or Schedule.", "response": "def compute_samples_displays(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n param_resolver: 'study.ParamResolverOrSimilarType' = None,\n ) -> study.ComputeDisplaysResult:\n \"\"\"Computes SamplesDisplays in the supplied Circuit or Schedule.\n\n Args:\n program: The circuit or schedule to simulate.\n param_resolver: Parameters to run with the program.\n\n Returns:\n ComputeDisplaysResult for the simulation.\n \"\"\"\n return self.compute_samples_displays_sweep(\n program,\n study.ParamResolver(param_resolver))[0]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compute_samples_displays_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: Optional[study.Sweepable] = None\n ) -> List[study.ComputeDisplaysResult]:\n \"\"\"Computes SamplesDisplays in the supplied Circuit or Schedule.\n\n In contrast to `compute_displays`, this allows for sweeping\n over different parameter values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n\n Returns:\n List of ComputeDisplaysResults for this run, one for each\n possible parameter resolver.\n \"\"\"\n circuit = (program if isinstance(program, circuits.Circuit)\n else program.to_circuit())\n param_resolvers = study.to_resolvers(params or study.ParamResolver({}))\n\n compute_displays_results = [] # type: List[study.ComputeDisplaysResult]\n for param_resolver in param_resolvers:\n display_values = {} # type: Dict[Hashable, Any]\n preceding_circuit = circuits.Circuit()\n for i, moment in enumerate(circuit):\n displays = (op for op in moment\n if isinstance(op, ops.SamplesDisplay))\n for display in displays:\n measurement_key = str(display.key)\n measurement_circuit = circuits.Circuit.from_ops(\n display.measurement_basis_change(),\n ops.measure(*display.qubits,\n key=measurement_key)\n )\n measurements = self._run(\n preceding_circuit + measurement_circuit,\n param_resolver,\n display.num_samples)\n display_values[display.key] = (\n display.value_derived_from_samples(\n measurements[measurement_key]))\n preceding_circuit.append(circuit[i])\n compute_displays_results.append(study.ComputeDisplaysResult(\n params=param_resolver,\n display_values=display_values))\n\n return compute_displays_results", "response": "Computes SamplesDisplays in the supplied Circuit or Schedule."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef simulate(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n param_resolver: 'study.ParamResolverOrSimilarType' = None,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Any = None,\n ) -> 'SimulationTrialResult':\n \"\"\"Simulates the supplied Circuit or Schedule.\n\n This method returns a result which allows access to the entire\n wave function.\n\n Args:\n program: The circuit or schedule to simulate.\n param_resolver: Parameters to run with the program.\n qubit_order: Determines the canonical ordering of the qubits. This\n is often used in specifying the initial state, i.e. the\n ordering of the computational basis states.\n initial_state: The initial state for the simulation. The form of\n this state depends on the simulation implementation. See\n documentation of the implementing class for details.\n\n Returns:\n SimulationTrialResults for the simulation. Includes the final state.\n \"\"\"\n return self.simulate_sweep(\n program,\n study.ParamResolver(param_resolver),\n qubit_order,\n initial_state)[0]", "response": "Simulates the supplied Circuit or Schedule and returns a simulation result."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef simulate_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: study.Sweepable,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Any = None,\n ) -> List['SimulationTrialResult']:\n \"\"\"Simulates the supplied Circuit or Schedule.\n\n This method returns a result which allows access to the entire\n wave function. In contrast to simulate, this allows for sweeping\n over different parameter values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n qubit_order: Determines the canonical ordering of the qubits. This\n is often used in specifying the initial state, i.e. the\n ordering of the computational basis states.\n initial_state: The initial state for the simulation. The form of\n this state depends on the simulation implementation. See\n documentation of the implementing class for details.\n\n Returns:\n List of SimulationTrialResults for this run, one for each\n possible parameter resolver.\n \"\"\"\n raise NotImplementedError()", "response": "Simulates the supplied Circuit or Schedule."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsimulate the supplied Circuit or Schedule.", "response": "def simulate_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: study.Sweepable,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Any = None,\n ) -> List['SimulationTrialResult']:\n \"\"\"Simulates the supplied Circuit or Schedule.\n\n This method returns a result which allows access to the entire\n wave function. In contrast to simulate, this allows for sweeping\n over different parameter values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n qubit_order: Determines the canonical ordering of the qubits. This\n is often used in specifying the initial state, i.e. the\n ordering of the computational basis states.\n initial_state: The initial state for the simulation. The form of\n this state depends on the simulation implementation. See\n documentation of the implementing class for details.\n\n Returns:\n List of SimulationTrialResults for this run, one for each\n possible parameter resolver.\n \"\"\"\n circuit = (program if isinstance(program, circuits.Circuit)\n else program.to_circuit())\n param_resolvers = study.to_resolvers(params)\n\n trial_results = []\n qubit_order = ops.QubitOrder.as_qubit_order(qubit_order)\n for param_resolver in param_resolvers:\n all_step_results = self.simulate_moment_steps(circuit,\n param_resolver,\n qubit_order,\n initial_state)\n measurements = {} # type: Dict[str, np.ndarray]\n for step_result in all_step_results:\n for k, v in step_result.measurements.items():\n measurements[k] = np.array(v, dtype=bool)\n trial_results.append(\n self._create_simulator_trial_result(\n params=param_resolver,\n measurements=measurements,\n final_simulator_state=step_result.simulator_state()))\n return trial_results"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn an iterator of StepResults for each moment simulated.", "response": "def simulate_moment_steps(\n self,\n circuit: circuits.Circuit,\n param_resolver: 'study.ParamResolverOrSimilarType' = None,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Any = None\n ) -> Iterator:\n \"\"\"Returns an iterator of StepResults for each moment simulated.\n\n If the circuit being simulated is empty, a single step result should\n be returned with the state being set to the initial state.\n\n Args:\n circuit: The Circuit to simulate.\n param_resolver: A ParamResolver for determining values of Symbols.\n qubit_order: Determines the canonical ordering of the qubits. This\n is often used in specifying the initial state, i.e. the\n ordering of the computational basis states.\n initial_state: The initial state for the simulation. The form of\n this state depends on the simulation implementation. See\n documentation of the implementing class for details.\n\n Returns:\n Iterator that steps through the simulation, simulating each\n moment and returning a StepResult for each moment.\n \"\"\"\n return self._simulator_iterator(\n circuit,\n study.ParamResolver(param_resolver),\n qubit_order,\n initial_state)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning an iterator over StepResult from Moments of a Circuit.", "response": "def _simulator_iterator(\n self,\n circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n qubit_order: ops.QubitOrderOrList,\n initial_state: Any,\n ) -> Iterator:\n \"\"\"Iterator over StepResult from Moments of a Circuit.\n\n Args:\n circuit: The circuit to simulate.\n param_resolver: A ParamResolver for determining values of\n Symbols.\n qubit_order: Determines the canonical ordering of the qubits. This\n is often used in specifying the initial state, i.e. the\n ordering of the computational basis states.\n initial_state: The initial state for the simulation. The form of\n this state depends on the simulation implementation. See\n documentation of the implementing class for details.\n\n Yields:\n StepResults from simulating a Moment of the Circuit.\n \"\"\"\n raise NotImplementedError()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsamples from the system at this point in the computation.", "response": "def sample(self,\n qubits: List[ops.Qid],\n repetitions: int = 1) -> np.ndarray:\n \"\"\"Samples from the system at this point in the computation.\n\n Note that this does not collapse the wave function.\n\n Args:\n qubits: The qubits to be sampled in an order that influence the\n returned measurement results.\n repetitions: The number of samples to take.\n\n Returns:\n Measurement results with True corresponding to the ``|1\u27e9`` state.\n The outer list is for repetitions, and the inner corresponds to\n measurements ordered by the supplied qubits. These lists\n are wrapped as an numpy ndarray.\n \"\"\"\n raise NotImplementedError()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsampling from the system at this point in the computation.", "response": "def sample_measurement_ops(\n self,\n measurement_ops: List[ops.GateOperation],\n repetitions: int = 1) -> Dict[str, np.ndarray]:\n \"\"\"Samples from the system at this point in the computation.\n\n Note that this does not collapse the wave function.\n\n In contrast to `sample` which samples qubits, this takes a list of\n `cirq.GateOperation` instances whose gates are `cirq.MeasurementGate`\n instances and then returns a mapping from the key in the measurement\n gate to the resulting bit strings. Different measurement operations must\n not act on the same qubits.\n\n Args:\n measurement_ops: `GateOperation` instances whose gates are\n `MeasurementGate` instances to be sampled form.\n repetitions: The number of samples to take.\n\n Returns: A dictionary from measurement gate key to measurement\n results. Measurement results are stored in a 2-dimensional\n numpy array, the first dimension corresponding to the repetition\n and the second to the actual boolean measurement results (ordered\n by the qubits being measured.)\n\n Raises:\n ValueError: If the operation's gates are not `MeasurementGate`\n instances or a qubit is acted upon multiple times by different\n operations from `measurement_ops`.\n \"\"\"\n bounds = {} # type: Dict[str, Tuple]\n all_qubits = [] # type: List[ops.Qid]\n current_index = 0\n for op in measurement_ops:\n gate = op.gate\n if not isinstance(gate, ops.MeasurementGate):\n raise ValueError('{} was not a MeasurementGate'.format(gate))\n key = protocols.measurement_key(gate)\n if key in bounds:\n raise ValueError(\n 'Duplicate MeasurementGate with key {}'.format(key))\n bounds[key] = (current_index, current_index + len(op.qubits))\n all_qubits.extend(op.qubits)\n current_index += len(op.qubits)\n indexed_sample = self.sample(all_qubits, repetitions)\n return {k: np.array([x[s:e] for x in indexed_sample]) for k, (s, e) in\n bounds.items()}"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\noverrides sympy and numpy returning repr strings that don t parse.", "response": "def proper_repr(value: Any) -> str:\n \"\"\"Overrides sympy and numpy returning repr strings that don't parse.\"\"\"\n\n if isinstance(value, sympy.Basic):\n result = sympy.srepr(value)\n\n # HACK: work around https://github.com/sympy/sympy/issues/16074\n # (only handles a few cases)\n fixed_tokens = [\n 'Symbol', 'pi', 'Mul', 'Add', 'Mod', 'Integer', 'Float', 'Rational'\n ]\n for token in fixed_tokens:\n result = result.replace(token, 'sympy.' + token)\n\n return result\n\n if isinstance(value, np.ndarray):\n return 'np.array({!r})'.format(value.tolist())\n return repr(value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning a Rabi oscillation experiment.", "response": "def rabi_oscillations(sampler: sim.Sampler,\n qubit: devices.GridQubit,\n max_angle: float = 2 * np.pi,\n *,\n repetitions: int = 1000,\n num_points: int = 200) -> RabiResult:\n \"\"\"Runs a Rabi oscillation experiment.\n\n Rotates a qubit around the x-axis of the Bloch sphere by a sequence of Rabi\n angles evenly spaced between 0 and max_angle. For each rotation, repeat\n the circuit a number of times and measure the average probability of the\n qubit being in the |1> state.\n\n Args:\n sampler: The quantum engine or simulator to run the circuits.\n qubit: The qubit under test.\n max_angle: The final Rabi angle in radians.\n repetitions: The number of repetitions of the circuit for each Rabi\n angle.\n num_points: The number of Rabi angles.\n\n Returns:\n A RabiResult object that stores and plots the result.\n \"\"\"\n theta = sympy.Symbol('theta')\n circuit = circuits.Circuit.from_ops(ops.X(qubit) ** theta)\n circuit.append(ops.measure(qubit, key='z'))\n sweep = study.Linspace(key='theta', start=0.0, stop=max_angle / np.pi,\n length=num_points)\n results = sampler.run_sweep(circuit, params=sweep, repetitions=repetitions)\n angles = np.linspace(0.0, max_angle, num_points)\n excited_state_probs = np.zeros(num_points)\n for i in range(num_points):\n excited_state_probs[i] = np.mean(results[i].measurements['z'])\n\n return RabiResult(angles, excited_state_probs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate a randomized benchmark of a single qubit.", "response": "def single_qubit_randomized_benchmarking(\n sampler: sim.Sampler,\n qubit: devices.GridQubit,\n use_xy_basis: bool = True,\n *,\n num_clifford_range: Sequence[int] = range(10, 100, 10),\n num_circuits: int = 20,\n repetitions: int = 1000) -> RandomizedBenchMarkResult:\n \"\"\"Clifford-based randomized benchmarking (RB) of a single qubit.\n\n A total of num_circuits random circuits are generated, each of which\n contains a fixed number of single-qubit Clifford gates plus one\n additional Clifford that inverts the whole sequence and a measurement in\n the z-basis. Each circuit is repeated a number of times and the average\n |0> state population is determined from the measurement outcomes of all\n of the circuits.\n\n The above process is done for different circuit lengths specified by the\n integers in num_clifford_range. For example, an integer 10 means the\n random circuits will contain 10 Clifford gates each plus one inverting\n Clifford. The user may use the result to extract an average gate fidelity,\n by analyzing the change in the average |0> state population at different\n circuit lengths. For actual experiments, one should choose\n num_clifford_range such that a clear exponential decay is observed in the\n results.\n\n See Barends et al., Nature 508, 500 for details.\n\n Args:\n sampler: The quantum engine or simulator to run the circuits.\n qubit: The qubit under test.\n use_xy_basis: Determines if the Clifford gates are built with x and y\n rotations (True) or x and z rotations (False).\n num_clifford_range: The different numbers of Cliffords in the RB study.\n num_circuits: The number of random circuits generated for each\n number of Cliffords.\n repetitions: The number of repetitions of each circuit.\n\n Returns:\n A RandomizedBenchMarkResult object that stores and plots the result.\n \"\"\"\n\n cliffords = _single_qubit_cliffords()\n c1 = cliffords.c1_in_xy if use_xy_basis else cliffords.c1_in_xz\n cfd_mats = np.array([_gate_seq_to_mats(gates) for gates in c1])\n\n gnd_probs = []\n for num_cfds in num_clifford_range:\n excited_probs_l = []\n for _ in range(num_circuits):\n circuit = _random_single_q_clifford(qubit, num_cfds, c1, cfd_mats)\n circuit.append(ops.measure(qubit, key='z'))\n results = sampler.run(circuit, repetitions=repetitions)\n excited_probs_l.append(np.mean(results.measurements['z']))\n gnd_probs.append(1.0 - np.mean(excited_probs_l))\n\n return RandomizedBenchMarkResult(num_clifford_range, gnd_probs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef single_qubit_state_tomography(sampler: sim.Sampler,\n qubit: devices.GridQubit,\n circuit: circuits.Circuit,\n repetitions: int = 1000) -> TomographyResult:\n \"\"\"Single-qubit state tomography.\n\n The density matrix of the output state of a circuit is measured by first\n doing projective measurements in the z-basis, which determine the\n diagonal elements of the matrix. A X/2 or Y/2 rotation is then added before\n the z-basis measurement, which determines the imaginary and real parts of\n the off-diagonal matrix elements, respectively.\n\n See Vandersypen and Chuang, Rev. Mod. Phys. 76, 1037 for details.\n\n Args:\n sampler: The quantum engine or simulator to run the circuits.\n qubit: The qubit under test.\n circuit: The circuit to execute on the qubit before tomography.\n repetitions: The number of measurements for each basis rotation.\n\n Returns:\n A TomographyResult object that stores and plots the density matrix.\n \"\"\"\n circuit_z = circuit + circuits.Circuit.from_ops(ops.measure(qubit, key='z'))\n results = sampler.run(circuit_z, repetitions=repetitions)\n rho_11 = np.mean(results.measurements['z'])\n rho_00 = 1.0 - rho_11\n\n circuit_x = circuits.Circuit.from_ops(circuit, ops.X(qubit) ** 0.5,\n ops.measure(qubit, key='z'))\n results = sampler.run(circuit_x, repetitions=repetitions)\n rho_01_im = np.mean(results.measurements['z']) - 0.5\n\n circuit_y = circuits.Circuit.from_ops(circuit, ops.Y(qubit) ** -0.5,\n ops.measure(qubit, key='z'))\n results = sampler.run(circuit_y, repetitions=repetitions)\n rho_01_re = 0.5 - np.mean(results.measurements['z'])\n\n rho_01 = rho_01_re + 1j * rho_01_im\n rho_10 = np.conj(rho_01)\n\n rho = np.array([[rho_00, rho_01], [rho_10, rho_11]])\n\n return TomographyResult(rho)", "response": "Single - qubit state tomography."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a two - qubit Clifford gate.", "response": "def _two_qubit_clifford(q_0: devices.GridQubit, q_1: devices.GridQubit,\n idx: int,\n cliffords: Cliffords) -> Iterator[ops.OP_TREE]:\n \"\"\"Generates a two-qubit Clifford gate.\n\n An integer (idx) from 0 to 11519 is used to generate a two-qubit Clifford\n gate which is constructed with single-qubit X and Y rotations and CZ gates.\n The decomposition of the Cliffords follow those described in the appendix\n of Barends et al., Nature 508, 500.\n\n The integer idx is first decomposed into idx_0 (which ranges from 0 to\n 23), idx_1 (ranging from 0 to 23) and idx_2 (ranging from 0 to 19). idx_0\n and idx_1 determine the two single-qubit rotations which happen at the\n beginning of all two-qubit Clifford gates. idx_2 determines the\n subsequent gates in the following:\n\n a) If idx_2 = 0, do nothing so the Clifford is just two single-qubit\n Cliffords (total of 24*24 = 576 possibilities).\n\n b) If idx_2 = 1, perform a CZ, followed by -Y/2 on q_0 and Y/2 on q_1,\n followed by another CZ, followed by Y/2 on q_0 and -Y/2 on q_1, followed\n by one more CZ and finally a Y/2 on q_1. The Clifford is then a member of\n the SWAP-like class (total of 24*24 = 576 possibilities).\n\n c) If 2 <= idx_2 <= 10, perform a CZ followed by a member of the S_1\n group on q_0 and a member of the S_1^(Y/2) group on q_1. The Clifford is\n a member of the CNOT-like class (a total of 3*3*24*24 = 5184 possibilities).\n\n d) If 11 <= idx_2 <= 19, perform a CZ, followed by Y/2 on q_0 and -X/2 on\n q_1, followed by another CZ, and finally a member of the S_1^(Y/2) group on\n q_0 and a member of the S_1^(X/2) group on q_1. The Clifford is a member\n of the iSWAP-like class (a total of 3*3*24*24 = 5184 possibilities).\n\n Through the above process, all 11520 members of the two-qubit Clifford\n group may be generated.\n\n Args:\n q_0: The first qubit under test.\n q_1: The second qubit under test.\n idx: An integer from 0 to 11519.\n cliffords: A NamedTuple that contains single-qubit Cliffords from the\n C1, S1, S_1^(X/2) and S_1^(Y/2) groups.\n \"\"\"\n c1 = cliffords.c1_in_xy\n s1 = cliffords.s1\n s1_x = cliffords.s1_x\n s1_y = cliffords.s1_y\n\n idx_0 = int(idx / 480)\n idx_1 = int((idx % 480) / 20)\n idx_2 = idx - idx_0 * 480 - idx_1 * 20\n yield _single_qubit_gates(c1[idx_0], q_0)\n yield _single_qubit_gates(c1[idx_1], q_1)\n if idx_2 == 1:\n yield ops.CZ(q_0, q_1)\n yield ops.Y(q_0) ** -0.5\n yield ops.Y(q_1) ** 0.5\n yield ops.CZ(q_0, q_1)\n yield ops.Y(q_0) ** 0.5\n yield ops.Y(q_1) ** -0.5\n yield ops.CZ(q_0, q_1)\n yield ops.Y(q_1) ** 0.5\n elif 2 <= idx_2 <= 10:\n yield ops.CZ(q_0, q_1)\n idx_3 = int((idx_2 - 2) / 3)\n idx_4 = (idx_2 - 2) % 3\n yield _single_qubit_gates(s1[idx_3], q_0)\n yield _single_qubit_gates(s1_y[idx_4], q_1)\n elif idx_2 >= 11:\n yield ops.CZ(q_0, q_1)\n yield ops.Y(q_0) ** 0.5\n yield ops.X(q_1) ** -0.5\n yield ops.CZ(q_0, q_1)\n idx_3 = int((idx_2 - 11) / 3)\n idx_4 = (idx_2 - 11) % 3\n yield _single_qubit_gates(s1_y[idx_3], q_0)\n yield _single_qubit_gates(s1_x[idx_4], q_1)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a sequence of tuples with the first item being a Rabi angle and the second item being the corresponding excited state probability.", "response": "def data(self) -> Sequence[Tuple[float, float]]:\n \"\"\"Returns a sequence of tuple pairs with the first item being a Rabi\n angle and the second item being the corresponding excited state\n probability.\n \"\"\"\n return [(angle, prob) for angle, prob in zip(self._rabi_angles,\n self._excited_state_probs)]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plot(self, **plot_kwargs: Any) -> None:\n fig = plt.figure()\n plt.plot(self._rabi_angles, self._excited_state_probs, 'ro-',\n figure=fig, **plot_kwargs)\n plt.xlabel(r\"Rabi Angle (Radian)\", figure=fig)\n plt.ylabel('Excited State Probability', figure=fig)\n fig.show()", "response": "Plots excited state probability vs the Rabi angle."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a sequence of tuples with the first item being the number of Cliffords and the second item being the ground state probability.", "response": "def data(self) -> Sequence[Tuple[int, float]]:\n \"\"\"Returns a sequence of tuple pairs with the first item being a\n number of Cliffords and the second item being the corresponding average\n ground state probability.\n \"\"\"\n return [(num, prob) for num, prob in zip(self._num_cfds_seq,\n self._gnd_state_probs)]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nplots the average ground state probability vs the number of cliffords in the RB study.", "response": "def plot(self, **plot_kwargs: Any) -> None:\n \"\"\"Plots the average ground state probability vs the number of\n Cliffords in the RB study.\n\n Args:\n **plot_kwargs: Arguments to be passed to matplotlib.pyplot.plot.\n \"\"\"\n fig = plt.figure()\n plt.plot(self._num_cfds_seq, self._gnd_state_probs, 'ro-',\n figure=fig, **plot_kwargs)\n plt.xlabel(r\"Number of Cliffords\", figure=fig)\n plt.ylabel('Ground State Probability', figure=fig)\n fig.show()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reflection_matrix_pow(reflection_matrix: np.ndarray, exponent: float):\n\n # The eigenvalues are x and -x for some complex unit x. Determine x.\n squared_phase = np.dot(reflection_matrix[:, 0],\n reflection_matrix[0, :])\n phase = complex(np.sqrt(squared_phase))\n\n # Extract +x and -x eigencomponents of the matrix.\n i = np.eye(reflection_matrix.shape[0]) * phase\n pos_part = (i + reflection_matrix) * 0.5\n neg_part = (i - reflection_matrix) * 0.5\n\n # Raise the matrix to a power by raising its eigencomponents to that power.\n pos_factor = phase**(exponent - 1)\n neg_factor = pos_factor * complex(-1)**exponent\n pos_part_raised = pos_factor * pos_part\n neg_part_raised = neg_part * neg_factor\n return pos_part_raised + neg_part_raised", "response": "Raises a matrix with two opposing eigenvalues to a power."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the phase of two numpy arrays that agree on the global phase of one entry.", "response": "def match_global_phase(a: np.ndarray,\n b: np.ndarray\n ) -> Tuple[np.ndarray, np.ndarray]:\n \"\"\"Phases the given matrices so that they agree on the phase of one entry.\n\n To maximize precision, the position with the largest entry from one of the\n matrices is used when attempting to compute the phase difference between\n the two matrices.\n\n Args:\n a: A numpy array.\n b: Another numpy array.\n\n Returns:\n A tuple (a', b') where a' == b' implies a == b*exp(i t) for some t.\n \"\"\"\n\n # Not much point when they have different shapes.\n if a.shape != b.shape:\n return a, b\n\n # Find the entry with the largest magnitude in one of the matrices.\n k = max(np.ndindex(*a.shape), key=lambda t: abs(b[t]))\n\n def dephase(v):\n r = np.real(v)\n i = np.imag(v)\n\n # Avoid introducing floating point error when axis-aligned.\n if i == 0:\n return -1 if r < 0 else 1\n if r == 0:\n return 1j if i < 0 else -1j\n\n return np.exp(-1j * np.arctan2(i, r))\n\n # Zero the phase at this entry in both matrices.\n return a * dephase(a[k]), b * dephase(b[k])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef targeted_left_multiply(left_matrix: np.ndarray,\n right_target: np.ndarray,\n target_axes: Sequence[int],\n out: Optional[np.ndarray] = None\n ) -> np.ndarray:\n \"\"\"Left-multiplies the given axes of the target tensor by the given matrix.\n\n Note that the matrix must have a compatible tensor structure.\n\n For example, if you have an 6-qubit state vector `input_state` with shape\n (2, 2, 2, 2, 2, 2), and a 2-qubit unitary operation `op` with shape\n (2, 2, 2, 2), and you want to apply `op` to the 5'th and 3'rd qubits\n within `input_state`, then the output state vector is computed as follows:\n\n output_state = cirq.targeted_left_multiply(op, input_state, [5, 3])\n\n This method also works when the right hand side is a matrix instead of a\n vector. If a unitary circuit's matrix is `old_effect`, and you append\n a CNOT(q1, q4) operation onto the circuit, where the control q1 is the qubit\n at offset 1 and the target q4 is the qubit at offset 4, then the appended\n circuit's unitary matrix is computed as follows:\n\n new_effect = cirq.targeted_left_multiply(\n left_matrix=cirq.unitary(cirq.CNOT).reshape((2, 2, 2, 2)),\n right_target=old_effect,\n target_axes=[1, 4])\n\n Args:\n left_matrix: What to left-multiply the target tensor by.\n right_target: A tensor to carefully broadcast a left-multiply over.\n target_axes: Which axes of the target are being operated on.\n out: The buffer to store the results in. If not specified or None, a new\n buffer is used. Must have the same shape as right_target.\n\n Returns:\n The output tensor.\n \"\"\"\n k = len(target_axes)\n d = len(right_target.shape)\n work_indices = tuple(range(k))\n data_indices = tuple(range(k, k + d))\n used_data_indices = tuple(data_indices[q] for q in target_axes)\n input_indices = work_indices + used_data_indices\n output_indices = list(data_indices)\n for w, t in zip(work_indices, target_axes):\n output_indices[t] = w\n\n all_indices = set(input_indices + data_indices + tuple(output_indices))\n\n return np.einsum(left_matrix, input_indices,\n right_target, data_indices,\n output_indices,\n # We would prefer to omit 'optimize=' (it's faster),\n # but this is a workaround for a bug in numpy:\n # https://github.com/numpy/numpy/issues/10926\n optimize=len(all_indices) >= 26,\n # And this is workaround for *another* bug!\n # Supposed to be able to just say 'old=old'.\n **({'out': out} if out is not None else {}))", "response": "Left - multiplies the given axes of the target tensor by the given matrix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nleaving - multiplies a matrix onto N slices.", "response": "def apply_matrix_to_slices(\n target: np.ndarray,\n matrix: np.ndarray,\n slices: List[_TSlice],\n *,\n out: Optional[np.ndarray] = None) -> np.ndarray:\n \"\"\"Left-multiplies an NxN matrix onto N slices of a numpy array.\n\n Example:\n The 4x4 matrix of a fractional SWAP gate can be expressed as\n\n [ 1 ]\n [ X**t ]\n [ 1 ]\n\n Where X is the 2x2 Pauli X gate and t is the power of the swap with t=1\n being a full swap. X**t is a power of the Pauli X gate's matrix.\n Applying the fractional swap is equivalent to applying a fractional X\n within the inner 2x2 subspace; the rest of the matrix is identity. This\n can be expressed using `apply_matrix_to_slices` as follows:\n\n def fractional_swap(target):\n assert target.shape == (4,)\n return apply_matrix_to_slices(\n target=target,\n matrix=cirq.unitary(cirq.X**t),\n slices=[1, 2]\n )\n\n Args:\n target: The input array with slices that need to be left-multiplied.\n matrix: The linear operation to apply to the subspace defined by the\n slices.\n slices: The parts of the tensor that correspond to the \"vector entries\"\n that the matrix should operate on. May be integers or complicated\n multi-dimensional slices into a tensor. The slices must refer to\n non-overlapping sections of the input all with the same shape.\n out: Where to write the output. If not specified, a new numpy array is\n created, with the same shape and dtype as the target, to store the\n output.\n\n Returns:\n The transformed array.\n \"\"\"\n # Validate arguments.\n if out is target:\n raise ValueError(\"Can't write output over the input.\")\n if matrix.shape != (len(slices), len(slices)):\n raise ValueError(\"matrix.shape != (len(slices), len(slices))\")\n\n # Fill in default values and prepare space.\n if out is None:\n out = np.copy(target)\n else:\n out[...] = target[...]\n\n # Apply operation.\n for i, s_i in enumerate(slices):\n out[s_i] *= matrix[i, i]\n for j, s_j in enumerate(slices):\n if i != j:\n out[s_i] += target[s_j] * matrix[i, j]\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntake the partial trace of a given tensor.", "response": "def partial_trace(tensor: np.ndarray,\n keep_indices: List[int]) -> np.ndarray:\n \"\"\"Takes the partial trace of a given tensor.\n\n The input tensor must have shape `(d_0, ..., d_{k-1}, d_0, ..., d_{k-1})`.\n The trace is done over all indices that are not in keep_indices. The\n resulting tensor has shape `(d_{i_0}, ..., d_{i_r}, d_{i_0}, ..., d_{i_r})`\n where `i_j` is the `j`th element of `keep_indices`.\n\n Args:\n tensor: The tensor to sum over. This tensor must have a shape\n `(d_0, ..., d_{k-1}, d_0, ..., d_{k-1})`.\n keep_indices: Which indices to not sum over. These are only the indices\n of the first half of the tensors indices (i.e. all elements must\n be between `0` and `tensor.ndims / 2 - 1` inclusive).\n\n Raises:\n ValueError: if the tensor is not of the correct shape or the indices\n are not from the first half of valid indices for the tensor.\n \"\"\"\n ndim = tensor.ndim // 2\n if not all(tensor.shape[i] == tensor.shape[i + ndim] for i in range(ndim)):\n raise ValueError('Tensors must have shape (d_0,...,d_{{k-1}},d_0,...,'\n 'd_{{k-1}}) but had shape ({}).'.format(tensor.shape))\n if not all(i < ndim for i in keep_indices):\n raise ValueError('keep_indices were {} but must be in first half, '\n 'i.e. have index less that {}.'.format(keep_indices,\n ndim))\n keep_set = set(keep_indices)\n keep_map = dict(zip(keep_indices, sorted(keep_indices)))\n left_indices = [keep_map[i] if i in keep_set else i for i in range(ndim)]\n right_indices = [ndim + i if i in keep_set else i for i in left_indices]\n return np.einsum(tensor, left_indices + right_indices)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef mul(lhs: Any, rhs: Any, default: Any = RaiseTypeErrorIfNotProvided) -> Any:\n # Use left-hand-side's __mul__.\n left_mul = getattr(lhs, '__mul__', None)\n result = NotImplemented if left_mul is None else left_mul(rhs)\n\n # Fallback to right-hand-side's __rmul__.\n if result is NotImplemented:\n right_mul = getattr(rhs, '__rmul__', None)\n result = NotImplemented if right_mul is None else right_mul(lhs)\n\n # Don't build up factors of 1.0 vs sympy Symbols.\n if lhs == 1 and is_parameterized(rhs):\n result = rhs\n if rhs == 1 and is_parameterized(lhs):\n result = lhs\n if lhs == -1 and is_parameterized(rhs):\n result = -rhs\n if rhs == -1 and is_parameterized(lhs):\n result = -lhs\n\n # Output.\n if result is not NotImplemented:\n return result\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n raise TypeError(\"unsupported operand type(s) for *: '{}' and '{}'\".format(\n type(lhs), type(rhs)))", "response": "Returns the product of two sets of elements."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndetermine if a matrix is approximately diagonal.", "response": "def is_diagonal(matrix: np.ndarray, *, atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is a approximately diagonal.\n\n A matrix is diagonal if i!=j implies m[i,j]==0.\n\n Args:\n matrix: The matrix to check.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the matrix is diagonal within the given tolerance.\n \"\"\"\n matrix = np.copy(matrix)\n for i in range(min(matrix.shape)):\n matrix[i, i] = 0\n return tolerance.all_near_zero(matrix, atol=atol)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_hermitian(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is approximately Hermitian.\n\n A matrix is Hermitian if it's square and equal to its adjoint.\n\n Args:\n matrix: The matrix to check.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the matrix is Hermitian within the given tolerance.\n \"\"\"\n return (matrix.shape[0] == matrix.shape[1] and\n np.allclose(matrix, np.conj(matrix.T), rtol=rtol, atol=atol))", "response": "Determines if a matrix is approximately Hermitian."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetermine if a matrix is approximately orthogonal.", "response": "def is_orthogonal(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is approximately orthogonal.\n\n A matrix is orthogonal if it's square and real and its transpose is its\n inverse.\n\n Args:\n matrix: The matrix to check.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the matrix is orthogonal within the given tolerance.\n \"\"\"\n return (matrix.shape[0] == matrix.shape[1] and\n np.all(np.imag(matrix) == 0) and\n np.allclose(matrix.dot(matrix.T), np.eye(matrix.shape[0]),\n rtol=rtol,\n atol=atol))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_special_orthogonal(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is approximately special orthogonal.\n\n A matrix is special orthogonal if it is square and real and its transpose\n is its inverse and its determinant is one.\n\n Args:\n matrix: The matrix to check.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the matrix is special orthogonal within the given tolerance.\n \"\"\"\n return (is_orthogonal(matrix, rtol=rtol, atol=atol) and\n (matrix.shape[0] == 0 or\n np.allclose(np.linalg.det(matrix), 1, rtol=rtol, atol=atol)))", "response": "Determines if a matrix is approximately special orthogonal."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_unitary(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is approximately unitary.\n\n A matrix is unitary if it's square and its adjoint is its inverse.\n\n Args:\n matrix: The matrix to check.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the matrix is unitary within the given tolerance.\n \"\"\"\n return (matrix.shape[0] == matrix.shape[1] and\n np.allclose(matrix.dot(np.conj(matrix.T)), np.eye(matrix.shape[0]),\n rtol=rtol,\n atol=atol))", "response": "Determines if a matrix is approximately unitary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_special_unitary(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if a matrix is approximately unitary with unit determinant.\n\n A matrix is special-unitary if it is square and its adjoint is its inverse\n and its determinant is one.\n\n Args:\n matrix: The matrix to check.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n Returns:\n Whether the matrix is unitary with unit determinant within the given\n tolerance.\n \"\"\"\n return (is_unitary(matrix, rtol=rtol, atol=atol) and\n (matrix.shape[0] == 0 or\n np.allclose(np.linalg.det(matrix), 1, rtol=rtol, atol=atol)))", "response": "Determines if a matrix is approximately unitary with unit determinant."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndetermining if two matrices approximately commute.", "response": "def commutes(\n m1: np.ndarray,\n m2: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> bool:\n \"\"\"Determines if two matrices approximately commute.\n\n Two matrices A and B commute if they are square and have the same size and\n AB = BA.\n\n Args:\n m1: One of the matrices.\n m2: The other matrix.\n rtol: The per-matrix-entry relative tolerance on equality.\n atol: The per-matrix-entry absolute tolerance on equality.\n\n Returns:\n Whether the two matrices have compatible sizes and a commutator equal\n to zero within tolerance.\n \"\"\"\n return (m1.shape[0] == m1.shape[1] and\n m1.shape == m2.shape and\n np.allclose(m1.dot(m2), m2.dot(m1), rtol=rtol, atol=atol))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndetermines if two arrays are all close to global phase.", "response": "def allclose_up_to_global_phase(\n a: np.ndarray,\n b: np.ndarray,\n *,\n rtol: float = 1.e-5,\n atol: float = 1.e-8,\n equal_nan: bool = False\n) -> bool:\n \"\"\"Determines if a ~= b * exp(i t) for some t.\n\n Args:\n a: A numpy array.\n b: Another numpy array.\n rtol: Relative error tolerance.\n atol: Absolute error tolerance.\n equal_nan: Whether or not NaN entries should be considered equal to\n other NaN entries.\n \"\"\"\n\n a, b = transformations.match_global_phase(a, b)\n\n # Should now be equivalent.\n return np.allclose(a=a, b=b, rtol=rtol, atol=atol, equal_nan=equal_nan)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an index corresponding to a desired subset of an np. ndarray.", "response": "def slice_for_qubits_equal_to(target_qubit_axes: Sequence[int],\n little_endian_qureg_value: int,\n *, # Forces keyword args.\n num_qubits: int = None\n ) -> Tuple[Union[slice, int, 'ellipsis'], ...]:\n \"\"\"Returns an index corresponding to a desired subset of an np.ndarray.\n\n It is assumed that the np.ndarray's shape is of the form (2, 2, 2, ..., 2).\n\n Example:\n\n ```python\n # A '4 qubit' tensor with values from 0 to 15.\n r = np.array(range(16)).reshape((2,) * 4)\n\n # We want to index into the subset where qubit #1 and qubit #3 are ON.\n s = cirq.slice_for_qubits_equal_to([1, 3], 0b11)\n print(s)\n # (slice(None, None, None), 1, slice(None, None, None), 1, Ellipsis)\n\n # Get that subset. It corresponds to numbers of the form 0b*1*1.\n # where here '*' indicates any possible value.\n print(r[s])\n # [[ 5 7]\n # [13 15]]\n ```\n\n Args:\n target_qubit_axes: The qubits that are specified by the index bits. All\n other axes of the slice are unconstrained.\n little_endian_qureg_value: An integer whose bits specify what value is\n desired for of the target qubits. The integer is little endian\n w.r.t. the target quit axes, meaning the low bit of the integer\n determines the desired value of the first targeted qubit, and so\n forth with the k'th targeted qubit's value set to\n bool(qureg_value & (1 << k)).\n num_qubits: If specified the slices will extend all the way up to\n this number of qubits, otherwise if it is None, the final element\n return will be Ellipsis. Optional and defaults to using Ellipsis.\n\n Returns:\n An index object that will slice out a mutable view of the desired subset\n of a tensor.\n \"\"\"\n n = num_qubits if num_qubits is not None else (\n max(target_qubit_axes) if target_qubit_axes else -1)\n result = [slice(None)] * (n + 2 * (\n num_qubits is None)) # type: List[Union[slice, int, ellipsis]]\n for k, axis in enumerate(target_qubit_axes):\n result[axis] = (little_endian_qureg_value >> k) & 1\n if num_qubits is None:\n result[-1] = Ellipsis\n return tuple(result)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a scheduled operation at a specific time.", "response": "def op_at_on(operation: ops.Operation,\n time: Timestamp,\n device: Device):\n \"\"\"Creates a scheduled operation with a device-determined duration.\"\"\"\n return ScheduledOperation(time,\n device.duration_of(operation),\n operation)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef num_qubits(self) -> Optional[int]:\n if not self:\n return None\n any_gate = next(iter(self))\n return any_gate.num_qubits()", "response": "Returns the number of qubits in the domain if known None otherwise."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreconstructs matrix of self using unitaries of underlying gates.", "response": "def matrix(self) -> np.ndarray:\n \"\"\"Reconstructs matrix of self using unitaries of underlying gates.\n\n Raises:\n TypeError: if any of the gates in self does not provide a unitary.\n \"\"\"\n num_qubits = self.num_qubits()\n if num_qubits is None:\n raise ValueError('Unknown number of qubits')\n num_dim = 2 ** num_qubits\n result = np.zeros((num_dim, num_dim), dtype=np.complex128)\n for gate, coefficient in self.items():\n result += protocols.unitary(gate) * coefficient\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef box_draw_character(first: Optional[BoxDrawCharacterSet],\n second: BoxDrawCharacterSet,\n *,\n top: int = 0,\n bottom: int = 0,\n left: int = 0,\n right: int = 0) -> Optional[str]:\n \"\"\"Finds a box drawing character based on its connectivity.\n\n For example:\n\n box_draw_character(\n NORMAL_BOX_CHARS,\n BOLD_BOX_CHARS,\n top=-1,\n right=+1)\n\n evaluates to '\u2515', which has a normal upward leg and bold rightward leg.\n\n Args:\n first: The character set to use for legs set to -1. If set to None,\n defaults to the same thing as the second character set.\n second: The character set to use for legs set to +1.\n top: Whether the upward leg should be present.\n bottom: Whether the bottom leg should be present.\n left: Whether the left leg should be present.\n right: Whether the right leg should be present.\n\n Returns:\n A box drawing character approximating the desired properties, or None\n if all legs are set to 0.\n \"\"\"\n if first is None:\n first = second\n sign = +1\n combo = None\n\n # Known combinations.\n if first is NORMAL_BOX_CHARS and second is BOLD_BOX_CHARS:\n combo = NORMAL_THEN_BOLD_MIXED_BOX_CHARS\n if first is BOLD_BOX_CHARS and second is NORMAL_BOX_CHARS:\n combo = NORMAL_THEN_BOLD_MIXED_BOX_CHARS\n sign = -1\n\n if combo is None:\n choice = second if +1 in [top, bottom, left, right] else first\n return choice.char(top=bool(top),\n bottom=bool(bottom),\n left=bool(left),\n right=bool(right))\n\n return combo.char(top=top * sign,\n bottom=bottom * sign,\n left=left * sign,\n right=right * sign)", "response": "Finds a box drawing character based on its connectivity."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning an application of this gate to the given qubits.", "response": "def on(self,\n *qubits: raw_types.Qid) -> 'SingleQubitPauliStringGateOperation':\n \"\"\"Returns an application of this gate to the given qubits.\n\n Args:\n *qubits: The collection of qubits to potentially apply the gate to.\n \"\"\"\n if len(qubits) != 1:\n raise ValueError(\n 'Expected a single qubit, got <{!r}>.'.format(qubits))\n from cirq.ops.pauli_string import SingleQubitPauliStringGateOperation\n return SingleQubitPauliStringGateOperation(self, qubits[0])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef decompose(\n val: TValue,\n *,\n intercepting_decomposer: Callable[['cirq.Operation'],\n Union[None,\n NotImplementedType,\n 'cirq.OP_TREE']] = None,\n fallback_decomposer: Callable[['cirq.Operation'],\n Union[None,\n NotImplementedType,\n 'cirq.OP_TREE']] = None,\n keep: Callable[['cirq.Operation'], bool] = None,\n on_stuck_raise: Union[None,\n Exception,\n Callable[['cirq.Operation'],\n Union[None, Exception]]]\n = _value_error_describing_bad_operation\n) -> List['cirq.Operation']:\n \"\"\"Recursively decomposes a value into `cirq.Operation`s meeting a criteria.\n\n Args:\n val: The value to decompose into operations.\n intercepting_decomposer: An optional method that is called before the\n default decomposer (the value's `_decompose_` method). If\n `intercepting_decomposer` is specified and returns a result that\n isn't `NotImplemented` or `None`, that result is used. Otherwise the\n decomposition falls back to the default decomposer.\n\n Note that `val` will be passed into `intercepting_decomposer`, even\n if `val` isn't a `cirq.Operation`.\n fallback_decomposer: An optional decomposition that used after the\n `intercepting_decomposer` and the default decomposer (the value's\n `_decompose_` method) both fail.\n keep: A predicate that determines if the initial operation or\n intermediate decomposed operations should be kept or else need to be\n decomposed further. If `keep` isn't specified, it defaults to \"value\n can't be decomposed anymore\".\n on_stuck_raise: If there is an operation that can't be decomposed and\n also can't be kept, `on_stuck_raise` is used to determine what error\n to raise. `on_stuck_raise` can either directly be an `Exception`, or\n a method that takes the problematic operation and returns an\n `Exception`. If `on_stuck_raise` is set to `None` or a method that\n returns `None`, undecomposable operations are simply silently kept.\n `on_stuck_raise` defaults to a `ValueError` describing the unwanted\n undecomposable operation.\n\n Returns:\n A list of operations that the given value was decomposed into. If\n `on_stuck_raise` isn't set to None, all operations in the list will\n satisfy the predicate specified by `keep`.\n\n Raises:\n TypeError:\n `val` isn't a `cirq.Operation` and can't be decomposed even once.\n (So it's not possible to return a list of operations.)\n\n ValueError:\n Default type of error raised if there's an undecomposable operation\n that doesn't satisfy the given `keep` predicate.\n\n TError:\n Custom type of error raised if there's an undecomposable operation\n that doesn't satisfy the given `keep` predicate.\n \"\"\"\n from cirq import ops # HACK: Avoids circular dependencies.\n\n if (on_stuck_raise is not _value_error_describing_bad_operation and\n keep is None):\n raise ValueError(\n \"Must specify 'keep' if specifying 'on_stuck_raise', because it's \"\n \"not possible to get stuck if you don't have a criteria on what's \"\n \"acceptable to keep.\")\n\n decomposers = [d\n for d in [intercepting_decomposer,\n _default_decomposer,\n fallback_decomposer]\n if d]\n\n def decomposer(op):\n for d in decomposers:\n r = d(op)\n if r is not NotImplemented and r is not None:\n return r\n return NotImplemented\n\n output = []\n queue = [val] # type: List[Any]\n while queue:\n item = queue.pop(0)\n\n if isinstance(item, ops.Operation) and keep is not None and keep(item):\n output.append(item)\n continue\n\n decomposed = decomposer(item)\n if decomposed is not NotImplemented and decomposed is not None:\n queue[:0] = ops.flatten_op_tree(decomposed)\n continue\n\n if (not isinstance(item, ops.Operation) and\n isinstance(item, collections.Iterable)):\n queue[:0] = ops.flatten_op_tree(item)\n continue\n\n if keep is not None and on_stuck_raise is not None:\n if isinstance(on_stuck_raise, Exception):\n raise on_stuck_raise\n elif callable(on_stuck_raise):\n error = on_stuck_raise(item)\n if error is not None:\n raise error\n\n output.append(item)\n\n return output", "response": "Recursively decompose a value into a list of cirq. Operation objects."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decompose_once(val: Any,\n default=RaiseTypeErrorIfNotProvided,\n **kwargs):\n \"\"\"Decomposes a value into operations, if possible.\n\n This method decomposes the value exactly once, instead of decomposing it\n and then continuing to decomposing the decomposed operations recursively\n until some criteria is met (which is what `cirq.decompose` does).\n\n Args:\n val: The value to call `_decompose_` on, if possible.\n default: A default result to use if the value doesn't have a\n `_decompose_` method or that method returns `NotImplemented` or\n `None`. If not specified, undecomposable values cause a `TypeError`.\n kwargs: Arguments to forward into the `_decompose_` method of `val`.\n For example, this is used to tell gates what qubits they are being\n applied to.\n\n Returns:\n The result of `val._decompose_(**kwargs)`, if `val` has a `_decompose_`\n method and it didn't return `NotImplemented` or `None`. Otherwise\n `default` is returned, if it was specified. Otherwise an error is\n raised.\n\n TypeError:\n `val` didn't have a `_decompose_` method (or that method returned\n `NotImplemented` or `None`) and `default` wasn't set.\n \"\"\"\n method = getattr(val, '_decompose_', None)\n decomposed = NotImplemented if method is None else method(**kwargs)\n\n if decomposed is not NotImplemented and decomposed is not None:\n from cirq import ops # HACK: Avoids circular dependencies.\n return list(ops.flatten_op_tree(decomposed))\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n if method is None:\n raise TypeError(\"object of type '{}' \"\n \"has no _decompose_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _decompose_ method, \"\n \"but it returned NotImplemented or None.\".format(type(val)))", "response": "Decomposes a value into operations recursively returning a list of the result of decomposing it."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decompose_once_with_qubits(val: Any,\n qubits: Iterable['cirq.Qid'],\n default=RaiseTypeErrorIfNotProvided):\n \"\"\"Decomposes a value into operations on the given qubits.\n\n This method is used when decomposing gates, which don't know which qubits\n they are being applied to unless told. It decomposes the gate exactly once,\n instead of decomposing it and then continuing to decomposing the decomposed\n operations recursively until some criteria is met.\n\n Args:\n val: The value to call `._decompose_(qubits=qubits)` on, if possible.\n qubits: The value to pass into the named `qubits` parameter of\n `val._decompose_`.\n default: A default result to use if the value doesn't have a\n `_decompose_` method or that method returns `NotImplemented` or\n `None`. If not specified, undecomposable values cause a `TypeError`.\n\n Returns:\n The result of `val._decompose_(qubits=qubits)`, if `val` has a\n `_decompose_` method and it didn't return `NotImplemented` or `None`.\n Otherwise `default` is returned, if it was specified. Otherwise an error\n is raised.\n\n TypeError:\n `val` didn't have a `_decompose_` method (or that method returned\n `NotImplemented` or `None`) and `default` wasn't set.\n \"\"\"\n return decompose_once(val, default, qubits=tuple(qubits))", "response": "Decomposes a value into operations on the given qubits."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn an explicit basis set that forces the given qubits in the given order.", "response": "def explicit(fixed_qubits: Iterable[raw_types.Qid],\n fallback: Optional['QubitOrder']=None) -> 'QubitOrder':\n \"\"\"A basis that contains exactly the given qubits in the given order.\n\n Args:\n fixed_qubits: The qubits in basis order.\n fallback: A fallback order to use for extra qubits not in the\n fixed_qubits list. Extra qubits will always come after the\n fixed_qubits, but will be ordered based on the fallback. If no\n fallback is specified, a ValueError is raised when extra qubits\n are specified.\n\n Returns:\n A Basis instance that forces the given qubits in the given order.\n \"\"\"\n result = tuple(fixed_qubits)\n if len(set(result)) < len(result):\n raise ValueError(\n 'Qubits appear in fixed_order twice: {}.'.format(result))\n\n def func(qubits):\n remaining = set(qubits) - set(fixed_qubits)\n if not remaining:\n return result\n if not fallback:\n raise ValueError(\n 'Unexpected extra qubits: {}.'.format(remaining))\n return result + fallback.order_for(remaining)\n\n return QubitOrder(func)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef order_for(self, qubits: Iterable[raw_types.Qid]\n ) -> Tuple[raw_types.Qid, ...]:\n \"\"\"Returns a qubit tuple ordered corresponding to the basis.\n\n Args:\n qubits: Qubits that should be included in the basis. (Additional\n qubits may be added into the output by the basis.)\n\n Returns:\n A tuple of qubits in the same order that their single-qubit\n matrices would be passed into `np.kron` when producing a matrix for\n the entire system.\n \"\"\"\n return self._explicit_func(qubits)", "response": "Returns a qubit tuple ordered corresponding to the basis."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef as_qubit_order(val: 'qubit_order_or_list.QubitOrderOrList'\n ) -> 'QubitOrder':\n \"\"\"Converts a value into a basis.\n\n Args:\n val: An iterable or a basis.\n\n Returns:\n The basis implied by the value.\n \"\"\"\n if isinstance(val, collections.Iterable):\n return QubitOrder.explicit(val)\n if isinstance(val, QubitOrder):\n return val\n raise ValueError(\n \"Don't know how to interpret <{}> as a Basis.\".format(val))", "response": "Converts a value into a basis set."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntransform the Basis so that it applies to the internal and external qubits.", "response": "def map(self,\n internalize: Callable[[TExternalQubit], TInternalQubit],\n externalize: Callable[[TInternalQubit], TExternalQubit]\n ) -> 'QubitOrder':\n \"\"\"Transforms the Basis so that it applies to wrapped qubits.\n\n Args:\n externalize: Converts an internal qubit understood by the underlying\n basis into an external qubit understood by the caller.\n internalize: Converts an external qubit understood by the caller\n into an internal qubit understood by the underlying basis.\n\n Returns:\n A basis that transforms qubits understood by the caller into qubits\n understood by an underlying basis, uses that to order the qubits,\n then wraps the ordered qubits back up for the caller.\n \"\"\"\n\n def func(qubits):\n unwrapped_qubits = [internalize(q) for q in qubits]\n unwrapped_result = self.order_for(unwrapped_qubits)\n return tuple(externalize(q) for q in unwrapped_result)\n\n return QubitOrder(func)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef range(*args, prefix: str):\n return [NamedQubit(prefix + str(i)) for i in range(*args)]", "response": "Returns a range of NamedQubits.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef channel(val: Any,\n default: Any = RaiseTypeErrorIfNotProvided\n ) -> Union[Tuple[np.ndarray], Sequence[TDefault]]:\n r\"\"\"Returns a list of matrices describing the channel for the given value.\n\n These matrices are the terms in the operator sum representation of\n a quantum channel. If the returned matrices are {A_0,A_1,..., A_{r-1}},\n then this describes the channel:\n \\rho \\rightarrow \\sum_{k=0}^{r-1} A_0 \\rho A_0^\\dagger\n These matrices are required to satisfy the trace preserving condition\n \\sum_{k=0}^{r-1} A_i^\\dagger A_i = I\n where I is the identity matrix. The matrices A_i are sometimes called\n Krauss or noise operators.\n\n Args:\n val: The value to describe by a channel.\n default: Determines the fallback behavior when `val` doesn't have\n a channel. If `default` is not set, a TypeError is raised. If\n default is set to a value, that value is returned.\n\n Returns:\n If `val` has a `_channel_` method and its result is not NotImplemented,\n that result is returned. Otherwise, if `val` has a `_mixture_` method\n and its results is not NotImplement a tuple made up of channel\n corresponding to that mixture being a probabilistic mixture of unitaries\n is returned. Otherwise, if `val` has a `_unitary_` method and\n its result is not NotImplemented a tuple made up of that result is\n returned. Otherwise, if a default value was specified, the default\n value is returned.\n\n Raises:\n TypeError: `val` doesn't have a _channel_ or _unitary_ method (or that\n method returned NotImplemented) and also no default value was\n specified.\n \"\"\"\n channel_getter = getattr(val, '_channel_', None)\n channel_result = (\n NotImplemented if channel_getter is None else channel_getter())\n if channel_result is not NotImplemented:\n return tuple(channel_result)\n\n mixture_getter = getattr(val, '_mixture_', None)\n mixture_result = (\n NotImplemented if mixture_getter is None else mixture_getter())\n if mixture_result is not NotImplemented:\n return tuple(np.sqrt(p) * u for p, u in mixture_result)\n\n unitary_getter = getattr(val, '_unitary_', None)\n unitary_result = (\n NotImplemented if unitary_getter is None else unitary_getter())\n if unitary_result is not NotImplemented:\n return (unitary_result,)\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if (channel_getter is None and unitary_getter is None\n and mixture_getter is None):\n raise TypeError(\"object of type '{}' has no _channel_ or _mixture_ or \"\n \"_unitary_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _channel_, _mixture_ or \"\n \"_unitary_ method, but it returned NotImplemented.\"\n .format(type(val)))", "response": "r Returns a list of matrices describing the channel for the given value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef has_channel(val: Any) -> bool:\n channel_getter = getattr(val, '_has_channel_', None)\n result = NotImplemented if channel_getter is None else channel_getter()\n\n if result is not NotImplemented:\n return result\n\n result = has_mixture_channel(val)\n if result is not NotImplemented and result:\n return result\n\n # No has methods, use `_channel_` or delegates instead.\n return channel(val, None) is not None", "response": "Returns whether the value has a channel representation."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef two_qubit_matrix_to_operations(q0: ops.Qid,\n q1: ops.Qid,\n mat: np.ndarray,\n allow_partial_czs: bool,\n atol: float = 1e-8,\n clean_operations: bool = True,\n ) -> List[ops.Operation]:\n \"\"\"Decomposes a two-qubit operation into Z/XY/CZ gates.\n\n Args:\n q0: The first qubit being operated on.\n q1: The other qubit being operated on.\n mat: Defines the operation to apply to the pair of qubits.\n allow_partial_czs: Enables the use of Partial-CZ gates.\n atol: A limit on the amount of absolute error introduced by the\n construction.\n clean_operations: Enables optimizing resulting operation list by\n merging operations and ejecting phased Paulis and Z operations.\n\n Returns:\n A list of operations implementing the matrix.\n \"\"\"\n kak = linalg.kak_decomposition(mat, atol=atol)\n operations = _kak_decomposition_to_operations(\n q0, q1, kak, allow_partial_czs, atol=atol)\n if clean_operations:\n return _cleanup_operations(operations)\n else:\n return operations", "response": "Decomposes a two - qubit operation into Z and XY gates."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of operations that are part of the decomposition.", "response": "def _kak_decomposition_to_operations(q0: ops.Qid,\n q1: ops.Qid,\n kak: linalg.KakDecomposition,\n allow_partial_czs: bool,\n atol: float = 1e-8\n ) -> List[ops.Operation]:\n \"\"\"Assumes that the decomposition is canonical.\"\"\"\n b0, b1 = kak.single_qubit_operations_before\n pre = [_do_single_on(b0, q0, atol=atol), _do_single_on(b1, q1, atol=atol)]\n a0, a1 = kak.single_qubit_operations_after\n post = [_do_single_on(a0, q0, atol=atol), _do_single_on(a1, q1, atol=atol)]\n\n return list(cast(Iterable[ops.Operation], ops.flatten_op_tree([\n pre,\n _non_local_part(q0,\n q1,\n kak.interaction_coefficients,\n allow_partial_czs,\n atol=atol),\n post,\n ])))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntest if a circuit for an operator expi * rad is trivial.", "response": "def _is_trivial_angle(rad: float, atol: float) -> bool:\n \"\"\"Tests if a circuit for an operator exp(i*rad*XX) (or YY, or ZZ) can\n be performed with a whole CZ.\n\n Args:\n rad: The angle in radians, assumed to be in the range [-pi/4, pi/4]\n \"\"\"\n return abs(rad) < atol or abs(abs(rad) - np.pi / 4) < atol"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _parity_interaction(q0: ops.Qid,\n q1: ops.Qid,\n rads: float,\n atol: float,\n gate: Optional[ops.Gate] = None):\n \"\"\"Yields a ZZ interaction framed by the given operation.\"\"\"\n if abs(rads) < atol:\n return\n\n h = rads * -2 / np.pi\n if gate is not None:\n g = cast(ops.Gate, gate)\n yield g.on(q0), g.on(q1)\n\n # If rads is \u00b1pi/4 radians within tolerance, single full-CZ suffices.\n if _is_trivial_angle(rads, atol):\n yield ops.CZ.on(q0, q1)\n else:\n yield ops.CZ(q0, q1) ** (-2 * h)\n\n yield ops.Z(q0)**h\n yield ops.Z(q1)**h\n if gate is not None:\n g = protocols.inverse(gate)\n yield g.on(q0), g.on(q1)", "response": "Yields a ZZ interaction framed by the given operation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _non_local_part(q0: ops.Qid,\n q1: ops.Qid,\n interaction_coefficients: Tuple[float, float, float],\n allow_partial_czs: bool,\n atol: float = 1e-8):\n \"\"\"Yields non-local operation of KAK decomposition.\"\"\"\n\n x, y, z = interaction_coefficients\n\n if (allow_partial_czs or\n all(_is_trivial_angle(e, atol) for e in [x, y, z])):\n return [\n _parity_interaction(q0, q1, x, atol, ops.Y**-0.5),\n _parity_interaction(q0, q1, y, atol, ops.X**0.5),\n _parity_interaction(q0, q1, z, atol)\n ]\n\n if abs(z) >= atol:\n return _xx_yy_zz_interaction_via_full_czs(q0, q1, x, y, z)\n\n if y >= atol:\n return _xx_yy_interaction_via_full_czs(q0, q1, x, y)\n\n return _xx_interaction_via_full_czs(q0, q1, x)", "response": "Yields non - local part of the KAK decomposition."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _run(\n self,\n circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n repetitions: int) -> Dict[str, List[np.ndarray]]:\n \"\"\"See definition in `cirq.SimulatesSamples`.\"\"\"\n param_resolver = param_resolver or study.ParamResolver({})\n resolved_circuit = protocols.resolve_parameters(circuit, param_resolver)\n def measure_or_mixture(op):\n return protocols.is_measurement(op) or protocols.has_mixture(op)\n if circuit.are_all_matches_terminal(measure_or_mixture):\n return self._run_sweep_sample(resolved_circuit, repetitions)\n else:\n return self._run_sweep_repeat(resolved_circuit, repetitions)", "response": "Runs the simulation for the given set of resources."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsimulates an operation that has a unitary.", "response": "def _simulate_unitary(self, op: ops.Operation, data: _StateAndBuffer,\n indices: List[int]) -> None:\n \"\"\"Simulate an op that has a unitary.\"\"\"\n result = protocols.apply_unitary(\n op,\n args=protocols.ApplyUnitaryArgs(\n data.state,\n data.buffer,\n indices))\n if result is data.buffer:\n data.buffer = data.state\n data.state = result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _simulate_measurement(self, op: ops.Operation, data: _StateAndBuffer,\n indices: List[int], measurements: Dict[str, List[bool]],\n num_qubits: int) -> None:\n \"\"\"Simulate an op that is a measurement in the computataional basis.\"\"\"\n meas = ops.op_gate_of_type(op, ops.MeasurementGate)\n # TODO: support measurement outside computational basis.\n if meas:\n invert_mask = meas.invert_mask or num_qubits * (False,)\n # Measure updates inline.\n bits, _ = wave_function.measure_state_vector(data.state,\n indices,\n data.state)\n corrected = [bit ^ mask for bit, mask in\n zip(bits, invert_mask)]\n key = protocols.measurement_key(meas)\n measurements[key].extend(corrected)", "response": "Simulate an operation that is a measurement in the computataional basis."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _simulate_mixture(self, op: ops.Operation, data: _StateAndBuffer,\n indices: List[int]) -> None:\n \"\"\"Simulate an op that is a mixtures of unitaries.\"\"\"\n probs, unitaries = zip(*protocols.mixture(op))\n # We work around numpy barfing on choosing from a list of\n # numpy arrays (which is not `one-dimensional`) by selecting\n # the index of the unitary.\n index = np.random.choice(range(len(unitaries)), p=probs)\n shape = (2,) * (2 * len(indices))\n unitary = unitaries[index].astype(self._dtype).reshape(shape)\n result = linalg.targeted_left_multiply(unitary, data.state, indices,\n out=data.buffer)\n data.buffer = data.state\n data.state = result", "response": "Simulate an op that is a mixtures of unitaries."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cubic_acquaintance_strategy(\n qubits: Iterable[ops.Qid],\n swap_gate: ops.Gate=ops.SWAP\n ) -> circuits.Circuit:\n \"\"\"Acquaints every triple of qubits.\n\n Exploits the fact that in a simple linear swap network every pair of\n logical qubits that starts at distance two remains so (except temporarily\n near the edge), and that every third one `goes through` the pair at some\n point in the network. The strategy then iterates through a series of\n mappings in which qubits i and i + k are placed at distance two, for k = 1\n through n / 2. Linear swap networks are used in between to effect the\n permutation.\n \"\"\"\n\n qubits = tuple(qubits)\n n_qubits = len(qubits)\n\n swap_gate = SwapPermutationGate(swap_gate)\n\n moments = []\n index_order = tuple(range(n_qubits))\n max_separation = max(((n_qubits - 1) // 2) + 1, 2)\n for separation in range(1, max_separation):\n stepped_indices_concatenated = tuple(itertools.chain(*(\n range(offset, n_qubits, separation)\n for offset in range(separation))))\n new_index_order = skip_and_wrap_around(stepped_indices_concatenated)\n permutation = {i: new_index_order.index(j)\n for i, j in enumerate(index_order)}\n permutation_gate = LinearPermutationGate(\n n_qubits, permutation, swap_gate)\n moments.append(ops.Moment([permutation_gate(*qubits)]))\n for i in range(n_qubits + 1):\n for offset in range(3):\n moment = ops.Moment(acquaint(*qubits[j:j+3])\n for j in range(offset, n_qubits - 2, 3))\n moments.append(moment)\n if i < n_qubits:\n moment = ops.Moment(swap_gate(*qubits[j:j+2])\n for j in range(i % 2, n_qubits - 1, 2))\n moments.append(moment)\n index_order = new_index_order[::-1]\n return circuits.Circuit(moments, device=UnconstrainedAcquaintanceDevice)", "response": "A cubic acquaintance strategy for a set of logical qubits."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse ASCIIart device layout into info about qubits and connectivity.", "response": "def _parse_device(s: str) -> Tuple[List[GridQubit], Dict[str, Set[GridQubit]]]:\n \"\"\"Parse ASCIIart device layout into info about qubits and connectivity.\n\n Args:\n s: String representing the qubit layout. Each line represents a row,\n and each character in the row is a qubit, or a blank site if the\n character is a hyphen '-'. Different letters for the qubit specify\n which measurement line that qubit is connected to, e.g. all 'A'\n qubits share a measurement line. Leading and trailing spaces on\n each line are ignored.\n\n Returns:\n A list of qubits and a dict mapping measurement line name to the qubits\n on that measurement line.\n \"\"\"\n lines = s.strip().split('\\n')\n qubits = [] # type: List[GridQubit]\n measurement_lines = {} # type: Dict[str, Set[GridQubit]]\n for row, line in enumerate(lines):\n for col, c in enumerate(line.strip()):\n if c != '-':\n qubit = GridQubit(row, col)\n qubits.append(qubit)\n measurement_line = measurement_lines.setdefault(c, set())\n measurement_line.add(qubit)\n return qubits, measurement_lines"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\niterate over arguments and calls approx_eq recursively.", "response": "def _approx_eq_iterables(val: Any, other: Any, *,\n atol: Union[int, float]) -> bool:\n \"\"\"Iterates over arguments and calls approx_eq recursively.\n\n Types of `val` and `other` does not necessarily needs to match each other.\n They just need to be iterable of the same length and have the same\n structure, approx_eq() will be called on each consecutive element of `val`\n and `other`.\n\n Args:\n val: Source for approximate comparison.\n other: Target for approximate comparison.\n atol: The minimum absolute tolerance. See np.isclose() documentation for\n details.\n\n Returns:\n True if objects are approximately equal, False otherwise. Returns\n NotImplemented when approximate equality is not implemented for given\n types.\n \"\"\"\n\n def get_iter(iterable):\n try:\n return iter(iterable)\n except TypeError:\n return None\n\n val_it = get_iter(val)\n other_it = get_iter(other)\n\n if val_it is not None and other_it is not None:\n while True:\n try:\n val_next = next(val_it)\n except StopIteration:\n try:\n next(other_it)\n return False\n except StopIteration:\n return True\n\n try:\n other_next = next(other_it)\n except StopIteration:\n return False\n\n result = approx_eq(val_next, other_next, atol=atol)\n if result is not True:\n return result\n\n return NotImplemented"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncomplete an acquaintance strategy for a set of qubits.", "response": "def complete_acquaintance_strategy(qubit_order: Sequence[ops.Qid],\n acquaintance_size: int=0,\n ) -> circuits.Circuit:\n \"\"\"\n Returns an acquaintance strategy capable of executing a gate corresponding\n to any set of at most acquaintance_size qubits.\n\n Args:\n qubit_order: The qubits on which the strategy should be defined.\n acquaintance_size: The maximum number of qubits to be acted on by\n an operation.\n\n Returns:\n An circuit capable of implementing any set of k-local\n operation.\n \"\"\"\n if acquaintance_size < 0:\n raise ValueError('acquaintance_size must be non-negative.')\n elif acquaintance_size == 0:\n return circuits.Circuit(device=UnconstrainedAcquaintanceDevice)\n\n if acquaintance_size > len(qubit_order):\n return circuits.Circuit(device=UnconstrainedAcquaintanceDevice)\n if acquaintance_size == len(qubit_order):\n return circuits.Circuit.from_ops(\n acquaint(*qubit_order), device=UnconstrainedAcquaintanceDevice)\n\n strategy = circuits.Circuit.from_ops(\n (acquaint(q) for q in qubit_order),\n device=UnconstrainedAcquaintanceDevice)\n for size_to_acquaint in range(2, acquaintance_size + 1):\n expose_acquaintance_gates(strategy)\n replace_acquaintance_with_swap_network(\n strategy, qubit_order, size_to_acquaint)\n return strategy"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd the specified number of input and output qubits.", "response": "def set_io_qubits(qubit_count):\n \"\"\"Add the specified number of input and output qubits.\"\"\"\n input_qubits = [cirq.GridQubit(i, 0) for i in range(qubit_count)]\n output_qubit = cirq.GridQubit(qubit_count, 0)\n return (input_qubits, output_qubit)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nimplement function f = 1 if x == x", "response": "def make_oracle(input_qubits, output_qubit, x_bits):\n \"\"\"Implement function {f(x) = 1 if x==x', f(x) = 0 if x!= x'}.\"\"\"\n # Make oracle.\n # for (1, 1) it's just a Toffoli gate\n # otherwise negate the zero-bits.\n yield(cirq.X(q) for (q, bit) in zip(input_qubits, x_bits) if not bit)\n yield(cirq.TOFFOLI(input_qubits[0], input_qubits[1], output_qubit))\n yield(cirq.X(q) for (q, bit) in zip(input_qubits, x_bits) if not bit)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_grover_circuit(input_qubits, output_qubit, oracle):\n # For 2 input qubits, that means using Grover operator only once.\n c = cirq.Circuit()\n\n # Initialize qubits.\n c.append([\n cirq.X(output_qubit),\n cirq.H(output_qubit),\n cirq.H.on_each(*input_qubits),\n ])\n\n # Query oracle.\n c.append(oracle)\n\n # Construct Grover operator.\n c.append(cirq.H.on_each(*input_qubits))\n c.append(cirq.X.on_each(*input_qubits))\n c.append(cirq.H.on(input_qubits[1]))\n c.append(cirq.CNOT(input_qubits[0], input_qubits[1]))\n c.append(cirq.H.on(input_qubits[1]))\n c.append(cirq.X.on_each(*input_qubits))\n c.append(cirq.H.on_each(*input_qubits))\n\n # Measure the result.\n c.append(cirq.measure(*input_qubits, key='result'))\n\n return c", "response": "Construct a Grover circuit."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\niterate through relevant python files within the given directory and returns an iterable of all python files that are not generated by the user.", "response": "def get_unhidden_ungenerated_python_files(directory: str) -> Iterable[str]:\n \"\"\"Iterates through relevant python files within the given directory.\n\n Args:\n directory: The top-level directory to explore.\n\n Yields:\n File paths.\n \"\"\"\n for dirpath, dirnames, filenames in os.walk(directory, topdown=True):\n if os.path.split(dirpath)[-1].startswith('.'):\n dirnames.clear()\n continue\n\n for filename in filenames:\n if filename.endswith('.py') and not filename.endswith('_pb2.py'):\n yield os.path.join(dirpath, filename)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_virtual_env(venv_path: str,\n requirements_paths: Iterable[str],\n python_path: str,\n verbose: bool) -> None:\n \"\"\"Creates a new virtual environment and then installs dependencies.\n\n Args:\n venv_path: Where to put the virtual environment's state.\n requirements_paths: Location of requirements files to -r install.\n python_path: The python binary to use.\n verbose: When set, more progress output is produced.\n \"\"\"\n shell_tools.run_cmd('virtualenv',\n None if verbose else '--quiet',\n '-p',\n python_path,\n venv_path,\n out=sys.stderr)\n pip_path = os.path.join(venv_path, 'bin', 'pip')\n for req_path in requirements_paths:\n shell_tools.run_cmd(pip_path,\n 'install',\n None if verbose else '--quiet',\n '-r',\n req_path,\n out=sys.stderr)", "response": "Creates a new virtual environment and installs dependencies."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef derive_temporary_python2_environment(\n destination_directory: str,\n python3_environment: PreparedEnv,\n verbose: bool,\n env_name: str = '.test_virtualenv_py2',\n python_path: str = \"/usr/bin/python2.7\") -> PreparedEnv:\n \"\"\"Creates a python 2.7 environment starting from a prepared python 3 one.\n\n Args:\n destination_directory: Where to put the python 2 environment.\n python3_environment: The prepared environment to start from.\n verbose: When set, more progress output is produced.\n env_name: The name to use for the virtualenv directory.\n python_path: The python binary to use.\n\n Returns:\n A description of the environment that was prepared.\n \"\"\"\n\n shutil.rmtree(destination_directory)\n input_directory = cast(str, python3_environment.destination_directory)\n os.chdir(input_directory)\n conversion_script_path = os.path.join(\n input_directory,\n 'dev_tools',\n 'python2.7-generate.sh')\n shell_tools.run_cmd('bash',\n conversion_script_path,\n destination_directory,\n input_directory,\n python3_environment.virtual_env_path,\n out=sys.stderr)\n os.chdir(destination_directory)\n\n # Create virtual environment.\n env_path = os.path.join(destination_directory, env_name)\n # (These files are output by dev_tools/python2.7-generate.sh.)\n req_path = os.path.join(destination_directory, 'requirements.txt')\n dev_req_path = os.path.join(destination_directory,\n 'pip-list-test-tools.txt')\n contrib_req_path = os.path.join(destination_directory,\n 'cirq',\n 'contrib',\n 'contrib-requirements.txt')\n req_paths = [req_path, dev_req_path, contrib_req_path]\n create_virtual_env(venv_path=env_path,\n python_path=python_path,\n requirements_paths=req_paths,\n verbose=verbose)\n\n return PreparedEnv(github_repo=python3_environment.repository,\n actual_commit_id=python3_environment.actual_commit_id,\n compare_commit_id=python3_environment.compare_commit_id,\n destination_directory=destination_directory,\n virtual_env_path=env_path)", "response": "Derives a temporary python 2. 7 environment from a prepared python 3 one."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_estimate(unknown_gate, qnum, repetitions):\n\n qubits = [None] * qnum\n for i in range(len(qubits)):\n qubits[i] = cirq.GridQubit(0, i)\n ancilla = cirq.GridQubit(0, len(qubits))\n\n circuit = cirq.Circuit.from_ops(\n cirq.H.on_each(*qubits),\n [cirq.ControlledGate(unknown_gate**(2**i)).on(qubits[qnum-i-1], ancilla)\n for i in range(qnum)],\n QftInverse(qnum)(*qubits),\n cirq.measure(*qubits, key='phase'))\n simulator = cirq.Simulator()\n result = simulator.run(circuit, repetitions=repetitions)\n return result", "response": "Runs the phase estimator circuit and execute simulations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef experiment(qnum, repetitions=100):\n\n def example_gate(phi):\n \"\"\"An example unitary 1-qubit gate U with an eigen vector |0> and an\n eigen value exp(2*Pi*i*phi)\n \"\"\"\n\n gate = cirq.SingleQubitMatrixGate(\n matrix=np.array([[np.exp(2*np.pi*1.0j*phi), 0], [0, 1]]))\n return gate\n\n print('Estimation with {}qubits.'.format(qnum))\n print('Actual, Estimation (Raw binary)')\n errors = []\n fold_func = lambda ms: ''.join(np.flip(ms, 0).astype(int).astype(str))\n for phi in np.arange(0, 1, 0.1):\n result = run_estimate(example_gate(phi), qnum, repetitions)\n hist = result.histogram(key='phase', fold_func=fold_func)\n estimate_bin = hist.most_common(1)[0][0]\n estimate = (sum([float(s)*0.5**(order+1)\n for order, s in enumerate(estimate_bin)]))\n print('{:0.4f}, {:0.4f} ({})'.format(phi, estimate, estimate_bin))\n errors.append((phi-estimate)**2)\n print('RMS Error: {:0.4f}\\n'.format(np.sqrt(sum(errors)/len(errors))))", "response": "Execute the phase estimator cirquit with multiple settings and\n show results."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _decompose_(self, qubits):\n\n qubits = list(qubits)\n while len(qubits) > 0:\n q_head = qubits.pop(0)\n yield cirq.H(q_head)\n for i, qubit in enumerate(qubits):\n yield (cirq.CZ**(-1/2.0**(i+1)))(qubit, q_head)", "response": "A generator that yields the quantum circuit."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsearching for linear sequence of qubits on a device.", "response": "def line_on_device(\n device: 'cirq.google.XmonDevice',\n length: int,\n method: LinePlacementStrategy = greedy.GreedySequenceSearchStrategy()\n) -> GridQubitLineTuple:\n \"\"\"Searches for linear sequence of qubits on device.\n\n Args:\n device: Google Xmon device instance.\n length: Desired number of qubits making up the line.\n method: Line placement method. Defaults to\n cirq.greedy.GreedySequenceSearchMethod.\n\n Returns:\n Line sequences search results.\n \"\"\"\n return method.place_line(device, length)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the value ** factor of the given value.", "response": "def pow(val: Any,\n exponent: Any,\n default: Any = RaiseTypeErrorIfNotProvided) -> Any:\n \"\"\"Returns `val**factor` of the given value, if defined.\n\n Values define an extrapolation by defining a __pow__(self, exponent) method.\n Note that the method may return NotImplemented to indicate a particular\n extrapolation can't be done.\n\n Args:\n val: The value or iterable of values to invert.\n exponent: The extrapolation factor. For example, if this is 0.5 and val\n is a gate then the caller is asking for a square root of the gate.\n default: Determines the fallback behavior when `val` doesn't have\n an extrapolation defined. If `default` is not set and that occurs,\n a TypeError is raised instead.\n\n Returns:\n If `val` has a __pow__ method that returns something besides\n NotImplemented, that result is returned. Otherwise, if a default value\n was specified, the default value is returned.\n\n Raises:\n TypeError: `val` doesn't have a __pow__ method (or that method returned\n NotImplemented) and no `default` value was specified.\n \"\"\"\n raiser = getattr(val, '__pow__', None)\n result = NotImplemented if raiser is None else raiser(exponent)\n if result is not NotImplemented:\n return result\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n if raiser is None:\n raise TypeError(\"object of type '{}' \"\n \"has no __pow__ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a __pow__ method, \"\n \"but it returned NotImplemented.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sample(program: Union[circuits.Circuit, schedules.Schedule],\n *,\n noise: devices.NoiseModel = devices.NO_NOISE,\n param_resolver: Optional[study.ParamResolver] = None,\n repetitions: int = 1,\n dtype: Type[np.number] = np.complex64) -> study.TrialResult:\n \"\"\"Simulates sampling from the given circuit or schedule.\n\n Args:\n program: The circuit or schedule to sample from.\n noise: Noise model to use while running the simulation.\n param_resolver: Parameters to run with the program.\n repetitions: The number of samples to take.\n dtype: The `numpy.dtype` used by the simulation. Typically one of\n `numpy.complex64` or `numpy.complex128`.\n Favors speed over precision by default, i.e. uses `numpy.complex64`.\n \"\"\"\n\n # State vector simulation is much faster, but only works if no randomness.\n if noise == devices.NO_NOISE and protocols.has_unitary(program):\n return sparse_simulator.Simulator(dtype=dtype).run(\n program=program,\n param_resolver=param_resolver,\n repetitions=repetitions)\n\n return density_matrix_simulator.DensityMatrixSimulator(\n dtype=dtype, noise=noise).run(program=program,\n param_resolver=param_resolver,\n repetitions=repetitions)", "response": "Simulates sampling from the given circuit or schedule."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sample_sweep(program: Union[circuits.Circuit, schedules.Schedule],\n params: study.Sweepable,\n *,\n noise: devices.NoiseModel = devices.NO_NOISE,\n repetitions: int = 1,\n dtype: Type[np.number] = np.complex64\n ) -> List[study.TrialResult]:\n \"\"\"Runs the supplied Circuit or Schedule, mimicking quantum hardware.\n\n In contrast to run, this allows for sweeping over different parameter\n values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n noise: Noise model to use while running the simulation.\n repetitions: The number of repetitions to simulate, per set of\n parameter values.\n dtype: The `numpy.dtype` used by the simulation. Typically one of\n `numpy.complex64` or `numpy.complex128`.\n Favors speed over precision by default, i.e. uses `numpy.complex64`.\n\n Returns:\n TrialResult list for this run; one for each possible parameter\n resolver.\n \"\"\"\n circuit = (program if isinstance(program, circuits.Circuit)\n else program.to_circuit())\n param_resolvers = study.to_resolvers(params)\n\n trial_results = [] # type: List[study.TrialResult]\n for param_resolver in param_resolvers:\n measurements = sample(circuit,\n noise=noise,\n param_resolver=param_resolver,\n repetitions=repetitions,\n dtype=dtype)\n trial_results.append(measurements)\n return trial_results", "response": "Runs the supplied Circuit or Schedule and returns a list of trial results."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef moment_by_moment_schedule(device: Device, circuit: Circuit):\n schedule = Schedule(device)\n t = Timestamp()\n for moment in circuit:\n if not moment.operations:\n continue\n for op in moment.operations:\n scheduled_op = ScheduledOperation.op_at_on(op, t, device)\n # Raises a ValueError describing the problem if this cannot be\n # scheduled.\n schedule.include(scheduled_operation=scheduled_op)\n # Raises ValueError at first sign of a device conflict.\n device.validate_scheduled_operation(schedule, scheduled_op)\n # Increment time for next moment by max of ops during this moment.\n max_duration = max(device.duration_of(op) for op in moment.operations)\n t += max_duration\n return schedule", "response": "Returns a schedule aligned with the moment structure of the circuit."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef min_height(self) -> int:\n return max(\n len(self.content.split('\\n')) if self.content else 0,\n # Only vertical lines can cross 0 height blocks.\n int(any([self.left, self.right]))\n )", "response": "Minimum height necessary to render the block s contents."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndraw lines in the box using the given character set.", "response": "def draw_curve(self,\n grid_characters: BoxDrawCharacterSet,\n *,\n top: bool = False,\n left: bool = False,\n right: bool = False,\n bottom: bool = False,\n crossing_char: Optional[str] = None):\n \"\"\"Draws lines in the box using the given character set.\n\n Supports merging the new lines with the lines from a previous call to\n draw_curve, including when they have different character sets (assuming\n there exist characters merging the two).\n\n Args:\n grid_characters: The character set to draw the curve with.\n top: Draw topward leg?\n left: Draw leftward leg?\n right: Draw rightward leg?\n bottom: Draw downward leg?\n crossing_char: Overrides the all-legs-present character. Useful for\n ascii diagrams, where the + doesn't always look the clearest.\n \"\"\"\n if not any([top, left, right, bottom]):\n return\n\n # Remember which legs are new, old, or missing.\n sign_top = +1 if top else -1 if self.top else 0\n sign_bottom = +1 if bottom else -1 if self.bottom else 0\n sign_left = +1 if left else -1 if self.left else 0\n sign_right = +1 if right else -1 if self.right else 0\n\n # Add new segments.\n if top:\n self.top = grid_characters.top_bottom\n if bottom:\n self.bottom = grid_characters.top_bottom\n if left:\n self.left = grid_characters.left_right\n if right:\n self.right = grid_characters.left_right\n\n # Fill center.\n if not all([crossing_char,\n self.top, self.bottom, self.left, self.right]):\n crossing_char = box_draw_character(\n self._prev_curve_grid_chars,\n grid_characters,\n top=sign_top,\n bottom=sign_bottom,\n left=sign_left,\n right=sign_right)\n self.center = crossing_char or ''\n\n self._prev_curve_grid_chars = grid_characters"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of text lines representing the block s contents.", "response": "def render(self, width: int, height: int) -> List[str]:\n \"\"\"Returns a list of text lines representing the block's contents.\n\n Args:\n width: The width of the output text. Must be at least as large as\n the block's minimum width.\n height: The height of the output text. Must be at least as large as\n the block's minimum height.\n\n Returns:\n Text pre-split into lines.\n \"\"\"\n if width == 0 or height == 0:\n return [''] * height\n\n out_chars = [[' '] * width for _ in range(height)]\n\n mid_x = int((width - 1) * self.horizontal_alignment)\n mid_y = (height - 1) // 2\n\n # Horizontal line legs.\n if self.left:\n out_chars[mid_y][:mid_x + 1] = self.left * (mid_x + 1)\n if self.right:\n out_chars[mid_y][mid_x:] = self.right * (width - mid_x)\n\n # Vertical line legs.\n if self.top:\n for y in range(mid_y + 1):\n out_chars[y][mid_x] = self.top\n if self.bottom:\n for y in range(mid_y, height):\n out_chars[y][mid_x] = self.bottom\n\n # Central content.\n mid = self.content or self.center\n if self.content or self.center:\n content_lines = mid.split('\\n')\n y = mid_y - (len(content_lines) - 1) // 2\n for dy, content_line in enumerate(content_lines):\n s = int((len(content_line) - 1) * self.horizontal_alignment)\n x = mid_x - s\n for dx, c in enumerate(content_line):\n out_chars[y + dy][x + dx] = c\n\n return [''.join(line) for line in out_chars]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the mutable block at the given position.", "response": "def mutable_block(self, x: int, y: int) -> Block:\n \"\"\"Returns the block at (x, y) so it can be edited.\"\"\"\n if x < 0 or y < 0:\n raise IndexError('x < 0 or y < 0')\n return self._blocks[(x, y)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_col_min_width(self, x: int, min_width: int):\n if x < 0:\n raise IndexError('x < 0')\n self._min_widths[x] = min_width", "response": "Sets a minimum width for blocks in the column with coordinate x."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset a minimum height for blocks in the row with coordinate y.", "response": "def set_row_min_height(self, y: int, min_height: int):\n \"\"\"Sets a minimum height for blocks in the row with coordinate y.\"\"\"\n if y < 0:\n raise IndexError('y < 0')\n self._min_heights[y] = min_height"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef render(self,\n *,\n block_span_x: Optional[int] = None,\n block_span_y: Optional[int] = None,\n min_block_width: int = 0,\n min_block_height: int = 0) -> str:\n \"\"\"Outputs text containing the diagram.\n\n Args:\n block_span_x: The width of the diagram in blocks. Set to None to\n default to using the smallest width that would include all\n accessed blocks and columns with a specified minimum width.\n block_span_y: The height of the diagram in blocks. Set to None to\n default to using the smallest height that would include all\n accessed blocks and rows with a specified minimum height.\n min_block_width: A global minimum width for all blocks.\n min_block_height: A global minimum height for all blocks.\n\n Returns:\n The diagram as a string.\n \"\"\"\n\n # Determine desired size of diagram in blocks.\n if block_span_x is None:\n block_span_x = 1 + max(\n max(x for x, _ in self._blocks.keys()),\n max(self._min_widths.keys()),\n )\n if block_span_y is None:\n block_span_y = 1 + max(\n max(y for _, y in self._blocks.keys()),\n max(self._min_heights.keys()),\n )\n\n # Method for accessing blocks without creating new entries.\n empty = Block()\n def block(x: int, y: int) -> Block:\n return self._blocks.get((x, y), empty)\n\n # Determine the width of every column and the height of every row.\n widths = {\n x: max(\n max(block(x, y).min_width() for y in range(block_span_y)),\n self._min_widths.get(x, 0),\n min_block_width,\n )\n for x in range(block_span_x)\n }\n heights = {\n y: max(\n max(block(x, y).min_height() for x in range(block_span_x)),\n self._min_heights.get(y, 0),\n min_block_height,\n )\n for y in range(block_span_y)\n }\n\n # Get the individually rendered blocks.\n block_renders = {\n (x, y): block(x, y).render(widths[x], heights[y])\n for x in range(block_span_x)\n for y in range(block_span_y)\n }\n\n # Paste together all of the rows of rendered block content.\n out_lines = [] # type: List[str]\n for y in range(block_span_y):\n for by in range(heights[y]):\n out_line_chunks = [] # type: List[str]\n for x in range(block_span_x):\n out_line_chunks.extend(block_renders[x, y][by])\n out_lines.append(''.join(out_line_chunks).rstrip())\n\n # Then paste together the rows.\n return '\\n'.join(out_lines)", "response": "Outputs text containing the diagram.\n\n Args:\n block_span_x: The width of the diagram in blocks. Set to None to\n default to using the smallest width that would include all\n accessed blocks and columns with a specified minimum width.\n block_span_y: The height of the diagram in blocks. Set to None to\n default to using the smallest height that would include all\n accessed blocks and rows with a specified minimum height.\n min_block_width: A global minimum width for all blocks.\n min_block_height: A global minimum height for all blocks.\n\n Returns:\n The diagram as a string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndemonstrate Quantum Fourier transform.", "response": "def main():\n \"\"\"Demonstrates Quantum Fourier transform.\n \"\"\"\n # Create circuit\n qft_circuit = generate_2x2_grid_qft_circuit()\n print('Circuit:')\n print(qft_circuit)\n # Simulate and collect final_state\n simulator = cirq.Simulator()\n result = simulator.simulate(qft_circuit)\n print()\n print('FinalState')\n print(np.around(result.final_state, 3))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef map_qubits(self: TSelf_PauliStringGateOperation,\n qubit_map: Dict[raw_types.Qid, raw_types.Qid]\n ) -> TSelf_PauliStringGateOperation:\n \"\"\"Return an equivalent operation on new qubits with its Pauli string\n mapped to new qubits.\n\n new_pauli_string = self.pauli_string.map_qubits(qubit_map)\n \"\"\"", "response": "Return an equivalent operation on new qubits with its Pauli string\n mapped to new qubits."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreporting the status of the current branch to github.", "response": "def report_status_to_github(self,\n state: str,\n description: str,\n context: str,\n target_url: Optional[str] = None):\n \"\"\"Sets a commit status indicator on github.\n\n If not running from a pull request (i.e. repository is None), then this\n just prints to stderr.\n\n Args:\n state: The state of the status indicator.\n Must be 'error', 'failure', 'pending', or 'success'.\n description: A summary of why the state is what it is,\n e.g. '5 lint errors' or 'tests passed!'.\n context: The name of the status indicator, e.g. 'pytest' or 'lint'.\n target_url: Optional location where additional details about the\n status can be found, e.g. an online test results page.\n\n Raises:\n ValueError: Not one of the allowed states.\n IOError: The HTTP post request failed, or the response didn't have\n a 201 code indicating success in the expected way.\n \"\"\"\n if state not in ['error', 'failure', 'pending', 'success']:\n raise ValueError('Unrecognized state: {!r}'.format(state))\n\n if self.repository is None or self.repository.access_token is None:\n return\n\n print(repr(('report_status',\n context,\n state,\n description,\n target_url)), file=sys.stderr)\n\n payload = {\n 'state': state,\n 'description': description,\n 'context': context,\n }\n if target_url is not None:\n payload['target_url'] = target_url\n\n url = (\n \"https://api.github.com/repos/{}/{}/statuses/{}?access_token={}\"\n .format(self.repository.organization,\n self.repository.name,\n self.actual_commit_id,\n self.repository.access_token))\n\n response = requests.post(url, json=payload)\n\n if response.status_code != 201:\n raise IOError('Request failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the files changed on one git branch vs another.", "response": "def get_changed_files(self) -> List[str]:\n \"\"\"Get the files changed on one git branch vs another.\n\n Returns:\n List[str]: File paths of changed files, relative to the git repo\n root.\n \"\"\"\n out = shell_tools.output_of(\n 'git',\n 'diff',\n '--name-only',\n self.compare_commit_id,\n self.actual_commit_id,\n '--',\n cwd=self.destination_directory)\n return [e for e in out.split('\\n') if e.strip()]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a Duration from datetime. timedelta.", "response": "def create(cls, duration: Union['Duration', timedelta]) -> 'Duration':\n \"\"\"Creates a Duration from datetime.timedelta if necessary\"\"\"\n if isinstance(duration, cls):\n return duration\n elif isinstance(duration, timedelta):\n duration_in_picos = duration.total_seconds() * 10**12\n return cls(picos=duration_in_picos)\n else:\n raise TypeError(\n 'Only datetime.timedelta and cirq.Duration are supported.')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _create_array(self, arr: np.ndarray) -> int:\n if not isinstance(arr, np.ndarray):\n raise ValueError('Array is not a numpy ndarray.')\n try:\n c_arr = np.ctypeslib.as_ctypes(arr)\n except (KeyError, NotImplementedError):\n raise ValueError(\n 'Array has unsupported dtype {}.'.format(arr.dtype))\n\n # pylint: disable=protected-access\n raw_arr = RawArray(c_arr._type_, c_arr)\n\n with self._lock:\n if self._count >= len(self._arrays):\n self._arrays += len(self._arrays) * [None]\n\n self._get_next_free()\n\n # Note storing the shape is a workaround for an issue encountered\n # when upgrading to numpy 1.15.\n # See https://github.com/numpy/numpy/issues/11636\n self._arrays[self._current] = (raw_arr, arr.shape)\n\n self._count += 1\n\n return self._current", "response": "Creates a RawArray from the given numpy array."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfrees the memory for the given array with the given handle.", "response": "def _free_array(self, handle: int):\n \"\"\"Frees the memory for the array with the given handle.\n\n Args:\n handle: The handle of the array whose memory should be freed. This\n handle must come from the _create_array method.\n \"\"\"\n with self._lock:\n if self._arrays[handle] is not None:\n self._arrays[handle] = None\n self._count -= 1"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the numpy array with the given handle.", "response": "def _get_array(self, handle: int) -> np.ndarray:\n \"\"\"Returns the array with the given handle.\n\n Args:\n handle: The handle of the array whose memory should be freed. This\n handle must come from the _create_array method.\n\n Returns:\n The numpy ndarray with the handle given from _create_array.\n \"\"\"\n tup = self._arrays[handle]\n assert tup is not None\n c_arr, shape = tup\n with warnings.catch_warnings():\n warnings.simplefilter('ignore', RuntimeWarning)\n result = np.ctypeslib.as_array(c_arr)\n result.shape = shape\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite QASM output to a file specified by path.", "response": "def save(self, path: Union[str, bytes, int]) -> None:\n \"\"\"Write QASM output to a file specified by path.\"\"\"\n with open(path, 'w') as f:\n def write(s: str) -> None:\n f.write(s)\n self._write_qasm(write)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrectify the acquaintance strategy of the given circuit.", "response": "def rectify_acquaintance_strategy(\n circuit: circuits.Circuit,\n acquaint_first: bool=True\n ) -> None:\n \"\"\"Splits moments so that they contain either only acquaintance gates\n or only permutation gates. Orders resulting moments so that the first one\n is of the same type as the previous one.\n\n Args:\n circuit: The acquaintance strategy to rectify.\n acquaint_first: Whether to make acquaintance moment first in when\n splitting the first mixed moment.\n \"\"\"\n\n if not is_acquaintance_strategy(circuit):\n raise TypeError('not is_acquaintance_strategy(circuit)')\n\n rectified_moments = []\n for moment in circuit:\n gate_type_to_ops = collections.defaultdict(list\n ) # type: Dict[bool, List[ops.GateOperation]]\n for op in moment.operations:\n gate_type_to_ops[isinstance(op.gate, AcquaintanceOpportunityGate)\n ].append(op)\n if len(gate_type_to_ops) == 1:\n rectified_moments.append(moment)\n continue\n for acquaint_first in sorted(gate_type_to_ops.keys(),\n reverse=acquaint_first):\n rectified_moments.append(\n ops.Moment(gate_type_to_ops[acquaint_first]))\n circuit._moments = rectified_moments"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef replace_acquaintance_with_swap_network(\n circuit: circuits.Circuit,\n qubit_order: Sequence[ops.Qid],\n acquaintance_size: Optional[int] = 0,\n swap_gate: ops.Gate = ops.SWAP\n ) -> bool:\n \"\"\"\n Replace every moment containing acquaintance gates (after\n rectification) with a generalized swap network, with the partition\n given by the acquaintance gates in that moment (and singletons for the\n free qubits). Accounts for reversing effect of swap networks.\n\n Args:\n circuit: The acquaintance strategy.\n qubit_order: The qubits, in order, on which the replacing swap network\n gate acts on.\n acquaintance_size: The acquaintance size of the new swap network gate.\n swap_gate: The gate used to swap logical indices.\n\n Returns: Whether or not the overall effect of the inserted swap network\n gates is to reverse the order of the qubits, i.e. the parity of the\n number of swap network gates inserted.\n\n Raises:\n TypeError: circuit is not an acquaintance strategy.\n \"\"\"\n\n if not is_acquaintance_strategy(circuit):\n raise TypeError('not is_acquaintance_strategy(circuit)')\n\n rectify_acquaintance_strategy(circuit)\n reflected = False\n reverse_map = {q: r for q, r in zip(qubit_order, reversed(qubit_order))}\n for moment_index, moment in enumerate(circuit):\n if reflected:\n moment = moment.transform_qubits(reverse_map.__getitem__)\n if all(isinstance(op.gate, AcquaintanceOpportunityGate)\n for op in moment.operations):\n swap_network_gate = SwapNetworkGate.from_operations(\n qubit_order, moment.operations,\n acquaintance_size, swap_gate)\n swap_network_op = swap_network_gate(*qubit_order)\n moment = ops.Moment([swap_network_op])\n reflected = not reflected\n circuit._moments[moment_index] = moment\n return reflected", "response": "Replaces all acquaintance gates in the given circuit with a generalized swap network."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if a gate is a native ion gate.", "response": "def is_native_ion_gate(gate: ops.Gate) -> bool:\n \"\"\"Check if a gate is a native ion gate.\n\n Args:\n gate: Input gate.\n\n Returns:\n True if the gate is native to the ion, false otherwise.\n \"\"\"\n return isinstance(gate, (ops.XXPowGate,\n ops.MeasurementGate,\n ops.XPowGate,\n ops.YPowGate,\n ops.ZPowGate))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convert_one(self, op: ops.Operation) -> ops.OP_TREE:\n\n # Known gate name\n if not isinstance(op, ops.GateOperation):\n raise TypeError(\"{!r} is not a gate operation.\".format(op))\n\n if is_native_ion_gate(op.gate):\n return [op]\n # one choice of known Hadamard gate decomposition\n if isinstance(op.gate, ops.HPowGate) and op.gate.exponent == 1:\n return [ops.Rx(np.pi).on(op.qubits[0]),\n ops.Ry(-1 * np.pi/2).on(op.qubits[0])]\n # one choice of known CNOT gate decomposition\n if isinstance(op.gate, ops.CNotPowGate) and op.gate.exponent == 1:\n return [ops.Ry(np.pi/2).on(op.qubits[0]),\n MS(np.pi/4).on(op.qubits[0], op.qubits[1]),\n ops.Rx(-1*np.pi/2).on(op.qubits[0]),\n ops.Rx(-1*np.pi/2).on(op.qubits[1]),\n ops.Ry(-1*np.pi/2).on(op.qubits[0])]\n # Known matrix\n mat = protocols.unitary(op, None) if len(\n op.qubits) <= 2 else None\n if mat is not None and len(op.qubits) == 1:\n gates = optimizers.single_qubit_matrix_to_phased_x_z(mat)\n return [g.on(op.qubits[0]) for g in gates]\n elif mat is not None and len(op.qubits) == 2:\n return two_qubit_matrix_to_ion_operations(\n op.qubits[0], op.qubits[1], mat)\n else:\n if self.ignore_failures:\n return [op]\n else:\n raise TypeError(\n \"Don't know how to work with {!r}. \"\n \"It isn't a native Ion Trap operation, \"\n \"a 1 or 2 qubit gate with a known unitary, \"\n \"or composite.\".format(op.gate))", "response": "Convert a single operation into an ion trap native gates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef diagonalize_real_symmetric_matrix(\n matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> np.ndarray:\n \"\"\"Returns an orthogonal matrix that diagonalizes the given matrix.\n\n Args:\n matrix: A real symmetric matrix to diagonalize.\n rtol: float = 1e-5,\n atol: float = 1e-8\n\n Returns:\n An orthogonal matrix P such that P.T @ matrix @ P is diagonal.\n\n Raises:\n ValueError: Matrix isn't real symmetric.\n \"\"\"\n\n # TODO: Determine if thresholds should be passed into is_hermitian\n if np.any(np.imag(matrix) != 0) or not predicates.is_hermitian(matrix):\n raise ValueError('Input must be real and symmetric.')\n\n _, result = np.linalg.eigh(matrix)\n\n return result", "response": "Returns an orthogonal matrix that diagonalizes the given matrix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsplitting a range into approximate equivalence classes.", "response": "def _contiguous_groups(\n length: int,\n comparator: Callable[[int, int], bool]\n) -> List[Tuple[int, int]]:\n \"\"\"Splits range(length) into approximate equivalence classes.\n\n Args:\n length: The length of the range to split.\n comparator: Determines if two indices have approximately equal items.\n\n Returns:\n A list of (inclusive_start, exclusive_end) range endpoints. Each\n corresponds to a run of approximately-equivalent items.\n \"\"\"\n result = []\n start = 0\n while start < length:\n past = start + 1\n while past < length and comparator(start, past):\n past += 1\n result.append((start, past))\n start = past\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef diagonalize_real_symmetric_and_sorted_diagonal_matrices(\n symmetric_matrix: np.ndarray,\n diagonal_matrix: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8,\n check_preconditions: bool = True) -> np.ndarray:\n \"\"\"Returns an orthogonal matrix that diagonalizes both given matrices.\n\n The given matrices must commute.\n Guarantees that the sorted diagonal matrix is not permuted by the\n diagonalization (except for nearly-equal values).\n\n Args:\n symmetric_matrix: A real symmetric matrix.\n diagonal_matrix: A real diagonal matrix with entries along the diagonal\n sorted into descending order.\n rtol: Relative numeric error threshold.\n atol: Absolute numeric error threshold.\n check_preconditions: If set, verifies that the input matrices commute\n and are respectively symmetric and diagonal descending.\n\n Returns:\n An orthogonal matrix P such that P.T @ symmetric_matrix @ P is diagonal\n and P.T @ diagonal_matrix @ P = diagonal_matrix (up to tolerance).\n\n Raises:\n ValueError: Matrices don't meet preconditions (e.g. not symmetric).\n \"\"\"\n\n # Verify preconditions.\n if check_preconditions:\n if (np.any(np.imag(symmetric_matrix)) or\n not predicates.is_hermitian(symmetric_matrix,\n rtol=rtol,\n atol=atol)):\n raise ValueError('symmetric_matrix must be real symmetric.')\n if (not predicates.is_diagonal(diagonal_matrix, atol=atol) or\n np.any(np.imag(diagonal_matrix)) or\n np.any(diagonal_matrix[:-1, :-1] < diagonal_matrix[1:, 1:])):\n raise ValueError(\n 'diagonal_matrix must be real diagonal descending.')\n if not predicates.commutes(diagonal_matrix,\n symmetric_matrix,\n rtol=rtol,\n atol=atol):\n raise ValueError('Given matrices must commute.')\n\n def similar_singular(i, j):\n return np.allclose(diagonal_matrix[i, i],\n diagonal_matrix[j, j], rtol=rtol)\n\n # Because the symmetric matrix commutes with the diagonal singulars matrix,\n # the symmetric matrix should be block-diagonal with a block boundary\n # wherever the singular values happen change. So we can use the singular\n # values to extract blocks that can be independently diagonalized.\n ranges = _contiguous_groups(diagonal_matrix.shape[0], similar_singular)\n\n # Build the overall diagonalization by diagonalizing each block.\n p = np.zeros(symmetric_matrix.shape, dtype=np.float64)\n for start, end in ranges:\n block = symmetric_matrix[start:end, start:end]\n p[start:end, start:end] = diagonalize_real_symmetric_matrix(\n block, rtol=rtol, atol=atol)\n\n return p", "response": "Returns an orthogonal matrix that diagonalizes both given matrices."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef bidiagonalize_real_matrix_pair_with_symmetric_products(\n mat1: np.ndarray,\n mat2: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8,\n check_preconditions: bool = True) -> Tuple[np.ndarray, np.ndarray]:\n \"\"\"Finds orthogonal matrices that diagonalize both mat1 and mat2.\n\n Requires mat1 and mat2 to be real.\n Requires mat1.T @ mat2 to be symmetric.\n Requires mat1 @ mat2.T to be symmetric.\n\n Args:\n mat1: One of the real matrices.\n mat2: The other real matrix.\n rtol: Relative numeric error threshold.\n atol: Absolute numeric error threshold.\n check_preconditions: If set, verifies that the inputs are real, and that\n mat1.T @ mat2 and mat1 @ mat2.T are both symmetric. Defaults to set.\n\n Returns:\n A tuple (L, R) of two orthogonal matrices, such that both L @ mat1 @ R\n and L @ mat2 @ R are diagonal matrices.\n\n Raises:\n ValueError: Matrices don't meet preconditions (e.g. not real).\n \"\"\"\n\n if check_preconditions:\n if np.any(np.imag(mat1) != 0):\n raise ValueError('mat1 must be real.')\n if np.any(np.imag(mat2) != 0):\n raise ValueError('mat2 must be real.')\n if not predicates.is_hermitian(mat1.dot(mat2.T), rtol=rtol, atol=atol):\n raise ValueError('mat1 @ mat2.T must be symmetric.')\n if not predicates.is_hermitian(mat1.T.dot(mat2), rtol=rtol, atol=atol):\n raise ValueError('mat1.T @ mat2 must be symmetric.')\n\n # Use SVD to bi-diagonalize the first matrix.\n base_left, base_diag, base_right = _svd_handling_empty(np.real(mat1))\n base_diag = np.diag(base_diag)\n\n # Determine where we switch between diagonalization-fixup strategies.\n dim = base_diag.shape[0]\n rank = dim\n while rank > 0 and tolerance.all_near_zero(base_diag[rank - 1, rank - 1],\n atol=atol):\n rank -= 1\n base_diag = base_diag[:rank, :rank]\n\n # Try diagonalizing the second matrix with the same factors as the first.\n semi_corrected = base_left.T.dot(np.real(mat2)).dot(base_right.T)\n\n # Fix up the part of the second matrix's diagonalization that's matched\n # against non-zero diagonal entries in the first matrix's diagonalization\n # by performing simultaneous diagonalization.\n overlap = semi_corrected[:rank, :rank]\n overlap_adjust = diagonalize_real_symmetric_and_sorted_diagonal_matrices(\n overlap,\n base_diag,\n rtol=rtol,\n atol=atol,\n check_preconditions=check_preconditions)\n\n # Fix up the part of the second matrix's diagonalization that's matched\n # against zeros in the first matrix's diagonalization by performing an SVD.\n extra = semi_corrected[rank:, rank:]\n extra_left_adjust, _, extra_right_adjust = _svd_handling_empty(extra)\n\n # Merge the fixup factors into the initial diagonalization.\n left_adjust = combinators.block_diag(overlap_adjust, extra_left_adjust)\n right_adjust = combinators.block_diag(overlap_adjust.T,\n extra_right_adjust)\n left = left_adjust.T.dot(base_left.T)\n right = base_right.T.dot(right_adjust.T)\n\n return left, right", "response": "This function returns the orthogonalized matrix that diagonalize both mat1 and mat2 and returns the first orthogonal matrix that is symmetric."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds orthogonal matrices L, R such that L @ matrix @ R is diagonal. Args: mat: A unitary matrix. rtol: Relative numeric error threshold. atol: Absolute numeric error threshold. check_preconditions: If set, verifies that the input is a unitary matrix (to the given tolerances). Defaults to set. Returns: A triplet (L, d, R) such that L @ mat @ R = diag(d). Both L and R will be orthogonal matrices with determinant equal to 1. Raises: ValueError: Matrices don't meet preconditions (e.g. not real).", "response": "def bidiagonalize_unitary_with_special_orthogonals(\n mat: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8,\n check_preconditions: bool = True\n) -> Tuple[np.ndarray, np.array, np.ndarray]:\n \"\"\"Finds orthogonal matrices L, R such that L @ matrix @ R is diagonal.\n\n Args:\n mat: A unitary matrix.\n rtol: Relative numeric error threshold.\n atol: Absolute numeric error threshold.\n check_preconditions: If set, verifies that the input is a unitary matrix\n (to the given tolerances). Defaults to set.\n\n Returns:\n A triplet (L, d, R) such that L @ mat @ R = diag(d). Both L and R will\n be orthogonal matrices with determinant equal to 1.\n\n Raises:\n ValueError: Matrices don't meet preconditions (e.g. not real).\n \"\"\"\n\n if check_preconditions:\n if not predicates.is_unitary(mat, rtol=rtol, atol=atol):\n raise ValueError('matrix must be unitary.')\n\n # Note: Because mat is unitary, setting A = real(mat) and B = imag(mat)\n # guarantees that both A @ B.T and A.T @ B are Hermitian.\n left, right = bidiagonalize_real_matrix_pair_with_symmetric_products(\n np.real(mat),\n np.imag(mat),\n rtol=rtol,\n atol=atol,\n check_preconditions=check_preconditions)\n\n # Convert to special orthogonal w/o breaking diagonalization.\n if np.linalg.det(left) < 0:\n left[0, :] *= -1\n if np.linalg.det(right) < 0:\n right[:, 0] *= -1\n\n diag = combinators.dot(left, mat, right)\n\n return left, np.diag(diag), right"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef asymmetric_depolarize(\n p_x: float, p_y: float, p_z: float\n) -> AsymmetricDepolarizingChannel:\n r\"\"\"Returns a AsymmetricDepolarizingChannel with given parameter.\n\n This channel evolves a density matrix via\n\n $$\n \\rho \\rightarrow (1 - p_x - p_y - p_z) \\rho\n + p_x X \\rho X + p_y Y \\rho Y + p_z Z \\rho Z\n $$\n\n Args:\n p_x: The probability that a Pauli X and no other gate occurs.\n p_y: The probability that a Pauli Y and no other gate occurs.\n p_z: The probability that a Pauli Z and no other gate occurs.\n\n Raises:\n ValueError: if the args or the sum of the args are not probabilities.\n \"\"\"\n return AsymmetricDepolarizingChannel(p_x, p_y, p_z)", "response": "r Returns an AsymmetricDepolarizingChannel with given parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nqueries the scheduled operations by time and qubit.", "response": "def query(\n self,\n *, # Forces keyword args.\n time: Timestamp,\n duration: Union[Duration, timedelta] = Duration(),\n qubits: Iterable[Qid] = None,\n include_query_end_time=False,\n include_op_end_times=False) -> List[ScheduledOperation]:\n \"\"\"Finds operations by time and qubit.\n\n Args:\n time: Operations must end after this time to be returned.\n duration: Operations must start by time+duration to be\n returned.\n qubits: If specified, only operations touching one of the included\n qubits will be returned.\n include_query_end_time: Determines if the query interval includes\n its end time. Defaults to no.\n include_op_end_times: Determines if the scheduled operation\n intervals include their end times or not. Defaults to no.\n\n Returns:\n A list of scheduled operations meeting the specified conditions.\n \"\"\"\n duration = Duration.create(duration)\n earliest_time = time - self._max_duration\n end_time = time + duration\n qubits = None if qubits is None else frozenset(qubits)\n\n def overlaps_interval(op):\n if not include_op_end_times and op.time + op.duration == time:\n return False\n if not include_query_end_time and op.time == end_time:\n return False\n return op.time + op.duration >= time and op.time <= end_time\n\n def overlaps_qubits(op):\n if qubits is None:\n return True\n return not qubits.isdisjoint(op.operation.qubits)\n\n potential_matches = self.scheduled_operations.irange_key(earliest_time,\n end_time)\n return [op\n for op in potential_matches\n if overlaps_interval(op) and overlaps_qubits(op)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind operations happening at the same time as the given operation.", "response": "def operations_happening_at_same_time_as(\n self, scheduled_operation: ScheduledOperation\n ) -> List[ScheduledOperation]:\n \"\"\"Finds operations happening at the same time as the given operation.\n\n Args:\n scheduled_operation: The operation specifying the time to query.\n\n Returns:\n Scheduled operations that overlap with the given operation.\n \"\"\"\n overlaps = self.query(\n time=scheduled_operation.time,\n duration=scheduled_operation.duration)\n return [e for e in overlaps if e != scheduled_operation]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef include(self, scheduled_operation: ScheduledOperation):\n collisions = self.query(time=scheduled_operation.time,\n duration=scheduled_operation.duration,\n qubits=scheduled_operation.operation.qubits)\n if collisions:\n raise ValueError('Operation {} has collisions: {}'.format(\n scheduled_operation.operation, collisions))\n self.scheduled_operations.add(scheduled_operation)\n self._max_duration = max(self._max_duration,\n scheduled_operation.duration)", "response": "Adds a scheduled operation to the schedule."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef exclude(self, scheduled_operation: ScheduledOperation) -> bool:\n try:\n self.scheduled_operations.remove(scheduled_operation)\n return True\n except ValueError:\n return False", "response": "Removes a scheduled operation from the schedule if present."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_circuit(self) -> Circuit:\n circuit = Circuit(device=self.device)\n time = None # type: Optional[Timestamp]\n for so in self.scheduled_operations:\n if so.time != time:\n circuit.append(so.operation,\n strategy=InsertStrategy.NEW_THEN_INLINE)\n time = so.time\n else:\n circuit.append(so.operation,\n strategy=InsertStrategy.INLINE)\n return circuit", "response": "Convert the schedule to a circuit."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nacquainting each of the qubits with another set of qubits with another set of qubits.", "response": "def acquaint_insides(swap_gate: ops.Gate,\n acquaintance_gate: ops.Operation,\n qubits: Sequence[ops.Qid],\n before: bool,\n layers: Layers,\n mapping: Dict[ops.Qid, int]\n ) -> None:\n \"\"\"Acquaints each of the qubits with another set specified by an\n acquaintance gate.\n\n Args:\n qubits: The list of qubits of which half are individually acquainted\n with another list of qubits.\n layers: The layers to put gates into.\n acquaintance_gate: The acquaintance gate that acquaints the end qubit\n with another list of qubits.\n before: Whether the acquainting is done before the shift.\n swap_gate: The gate used to swap logical indices.\n mapping: The mapping from qubits to logical indices. Used to keep track\n of the effect of inside-acquainting swaps.\n \"\"\"\n\n max_reach = _get_max_reach(len(qubits), round_up=before)\n reaches = itertools.chain(range(1, max_reach + 1),\n range(max_reach, -1, -1))\n offsets = (0, 1) * max_reach\n swap_gate = SwapPermutationGate(swap_gate)\n ops = []\n for offset, reach in zip(offsets, reaches):\n if offset == before:\n ops.append(acquaintance_gate)\n for dr in range(offset, reach, 2):\n ops.append(swap_gate(*qubits[dr:dr + 2]))\n intrastitial_layer = getattr(layers, 'pre' if before else 'post')\n intrastitial_layer += ops\n\n # add interstitial gate\n interstitial_layer = getattr(layers,\n ('prior' if before else 'posterior') + '_interstitial')\n interstitial_layer.append(acquaintance_gate)\n\n # update mapping\n reached_qubits = qubits[:max_reach + 1]\n positions = list(mapping[q] for q in reached_qubits)\n mapping.update(zip(reached_qubits, reversed(positions)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef acquaint_and_shift(parts: Tuple[List[ops.Qid], List[ops.Qid]],\n layers: Layers,\n acquaintance_size: Optional[int],\n swap_gate: ops.Gate,\n mapping: Dict[ops.Qid, int]):\n \"\"\"Acquaints and shifts a pair of lists of qubits. The first part is\n acquainted with every qubit individually in the second part, and vice\n versa. Operations are grouped into several layers:\n * prior_interstitial: The first layer of acquaintance gates.\n * prior: The combination of acquaintance gates and swaps that acquaints\n the inner halves.\n * intra: The shift gate.\n * post: The combination of acquaintance gates and swaps that acquaints\n the outer halves.\n * posterior_interstitial: The last layer of acquaintance gates.\n\n Args:\n parts: The two lists of qubits to acquaint.\n layers: The layers to put gates into.\n acquaintance_size: The number of qubits to acquaint at a time. If None,\n after each pair of parts is shifted the union thereof is\n acquainted.\n swap_gate: The gate used to swap logical indices.\n mapping: The mapping from qubits to logical indices. Used to keep track\n of the effect of inside-acquainting swaps.\n \"\"\"\n left_part, right_part = parts\n left_size, right_size = len(left_part), len(right_part)\n assert not (set(left_part) & set(right_part))\n qubits = left_part + right_part\n shift = CircularShiftGate(len(qubits),\n left_size,\n swap_gate=swap_gate)(\n *qubits)\n if acquaintance_size is None:\n layers.intra.append(shift)\n layers.post.append(acquaint(*qubits))\n shift.gate.update_mapping(mapping, qubits)\n elif max(left_size, right_size) != acquaintance_size - 1:\n layers.intra.append(shift)\n shift.gate.update_mapping(mapping, qubits)\n elif acquaintance_size == 2:\n layers.prior_interstitial.append(acquaint(*qubits))\n layers.intra.append(shift)\n shift.gate.update_mapping(mapping, qubits)\n else:\n # before\n if left_size == acquaintance_size - 1:\n # right part\n pre_acquaintance_gate = acquaint(*qubits[:acquaintance_size])\n acquaint_insides(\n swap_gate=swap_gate,\n acquaintance_gate=pre_acquaintance_gate,\n qubits=right_part,\n before=True,\n layers=layers,\n mapping=mapping)\n\n if right_size == acquaintance_size - 1:\n # left part\n pre_acquaintance_gate = acquaint(*qubits[-acquaintance_size:])\n acquaint_insides(\n swap_gate=swap_gate,\n acquaintance_gate=pre_acquaintance_gate,\n qubits=left_part[::-1],\n before=True,\n layers=layers,\n mapping=mapping)\n\n layers.intra.append(shift)\n shift.gate.update_mapping(mapping, qubits)\n\n # after\n if ((left_size == acquaintance_size - 1) and\n (right_size > 1)):\n # right part\n post_acquaintance_gate = acquaint(*qubits[-acquaintance_size:])\n\n new_left_part = qubits[right_size - 1::-1]\n acquaint_insides(\n swap_gate=swap_gate,\n acquaintance_gate=post_acquaintance_gate,\n qubits=new_left_part,\n before=False,\n layers=layers,\n mapping=mapping)\n\n if ((right_size == acquaintance_size - 1) and\n (left_size > 1)):\n # left part\n\n post_acquaintance_gate = acquaint(*qubits[:acquaintance_size])\n acquaint_insides(\n swap_gate=swap_gate,\n acquaintance_gate=post_acquaintance_gate,\n qubits=qubits[right_size:],\n before=False,\n layers=layers,\n mapping=mapping)", "response": "Acquaints and shifts a pair of parts into a single set of layers."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef circuit_to_pdf_using_qcircuit_via_tex(circuit: circuits.Circuit,\n filepath: str,\n pdf_kwargs=None,\n qcircuit_kwargs=None,\n clean_ext=('dvi', 'ps'),\n documentclass='article'):\n \"\"\"Compiles the QCircuit-based latex diagram of the given circuit.\n\n Args:\n circuit: The circuit to produce a pdf of.\n filepath: Where to output the pdf.\n pdf_kwargs: The arguments to pass to generate_pdf.\n qcircuit_kwargs: The arguments to pass to\n circuit_to_latex_using_qcircuit.\n clean_ext: The file extensions to clean up after compilation. By\n default, latexmk is used with the '-pdfps' flag, which produces\n intermediary dvi and ps files.\n documentclass: The documentclass of the latex file.\n \"\"\"\n pdf_kwargs = {'compiler': 'latexmk', 'compiler_args': ['-pdfps'],\n **({} if pdf_kwargs is None else pdf_kwargs)}\n qcircuit_kwargs = {} if qcircuit_kwargs is None else qcircuit_kwargs\n tex = circuit_to_latex_using_qcircuit(circuit, **qcircuit_kwargs)\n doc = Document(documentclass=documentclass, document_options='dvips')\n doc.packages.append(Package('amsmath'))\n doc.packages.append(Package('qcircuit'))\n doc.append(NoEscape(tex))\n doc.generate_pdf(filepath, **pdf_kwargs)\n for ext in clean_ext:\n try:\n os.remove(filepath + '.' + ext)\n except (OSError, IOError) as e:\n if e.errno != errno.ENOENT:\n raise", "response": "Compiles the QCircuit - based latex diagram of the given circuit and outputs a PDF of the result."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef deconstruct_single_qubit_matrix_into_angles(\n mat: np.ndarray) -> Tuple[float, float, float]:\n \"\"\"Breaks down a 2x2 unitary into more useful ZYZ angle parameters.\n\n Args:\n mat: The 2x2 unitary matrix to break down.\n\n Returns:\n A tuple containing the amount to phase around Z, then rotate around Y,\n then phase around Z (all in radians).\n \"\"\"\n # Anti-cancel left-vs-right phase along top row.\n right_phase = cmath.phase(mat[0, 1] * np.conj(mat[0, 0])) + math.pi\n mat = np.dot(mat, _phase_matrix(-right_phase))\n\n # Cancel top-vs-bottom phase along left column.\n bottom_phase = cmath.phase(mat[1, 0] * np.conj(mat[0, 0]))\n mat = np.dot(_phase_matrix(-bottom_phase), mat)\n\n # Lined up for a rotation. Clear the off-diagonal cells with one.\n rotation = math.atan2(abs(mat[1, 0]), abs(mat[0, 0]))\n mat = np.dot(_rotation_matrix(-rotation), mat)\n\n # Cancel top-left-vs-bottom-right phase.\n diagonal_phase = cmath.phase(mat[1, 1] * np.conj(mat[0, 0]))\n\n # Note: Ignoring global phase.\n return right_phase + diagonal_phase, rotation * 2, bottom_phase", "response": "Breaks down a 2x2 unitary matrix into more useful ZYZ angle parameters."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _group_similar(items: List[T],\n comparer: Callable[[T, T], bool]) -> List[List[T]]:\n \"\"\"Combines similar items into groups.\n\n Args:\n items: The list of items to group.\n comparer: Determines if two items are similar.\n\n Returns:\n A list of groups of items.\n \"\"\"\n groups = [] # type: List[List[T]]\n used = set() # type: Set[int]\n for i in range(len(items)):\n if i not in used:\n group = [items[i]]\n for j in range(i + 1, len(items)):\n if j not in used and comparer(items[i], items[j]):\n used.add(j)\n group.append(items[j])\n groups.append(group)\n return groups", "response": "Combines similar items into groups."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _perp_eigendecompose(matrix: np.ndarray,\n rtol: float = 1e-5,\n atol: float = 1e-8,\n ) -> Tuple[np.array, List[np.ndarray]]:\n \"\"\"An eigendecomposition that ensures eigenvectors are perpendicular.\n\n numpy.linalg.eig doesn't guarantee that eigenvectors from the same\n eigenspace will be perpendicular. This method uses Gram-Schmidt to recover\n a perpendicular set. It further checks that all eigenvectors are\n perpendicular and raises an ArithmeticError otherwise.\n\n Args:\n matrix: The matrix to decompose.\n rtol: Relative threshold for determining whether eigenvalues are from\n the same eigenspace and whether eigenvectors are perpendicular.\n atol: Absolute threshold for determining whether eigenvalues are from\n the same eigenspace and whether eigenvectors are perpendicular.\n\n Returns:\n The eigenvalues and column eigenvectors. The i'th eigenvalue is\n associated with the i'th column eigenvector.\n\n Raises:\n ArithmeticError: Failed to find perpendicular eigenvectors.\n \"\"\"\n vals, cols = np.linalg.eig(matrix)\n vecs = [cols[:, i] for i in range(len(cols))]\n\n # Convert list of row arrays to list of column arrays.\n for i in range(len(vecs)):\n vecs[i] = np.reshape(vecs[i], (len(vecs[i]), vecs[i].ndim))\n\n # Group by similar eigenvalue.\n n = len(vecs)\n groups = _group_similar(\n list(range(n)),\n lambda k1, k2: np.allclose(vals[k1], vals[k2], rtol=rtol))\n\n # Remove overlap between eigenvectors with the same eigenvalue.\n for g in groups:\n q, _ = np.linalg.qr(np.hstack([vecs[i] for i in g]))\n for i in range(len(g)):\n vecs[g[i]] = q[:, i]\n\n return vals, vecs", "response": "Decomposes a matrix of the ith column eigenvectors into a perpendicular set of ith eigenvalues."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef map_eigenvalues(\n matrix: np.ndarray,\n func: Callable[[complex], complex],\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> np.ndarray:\n \"\"\"Applies a function to the eigenvalues of a matrix.\n\n Given M = sum_k a_k |v_k> Tuple[complex, np.ndarray, np.ndarray]:\n \"\"\"Splits a 4x4 matrix U = kron(A, B) into A, B, and a global factor.\n\n Requires the matrix to be the kronecker product of two 2x2 unitaries.\n Requires the matrix to have a non-zero determinant.\n Giving an incorrect matrix will cause garbage output.\n\n Args:\n matrix: The 4x4 unitary matrix to factor.\n\n Returns:\n A scalar factor and a pair of 2x2 unit-determinant matrices. The\n kronecker product of all three is equal to the given matrix.\n\n Raises:\n ValueError:\n The given matrix can't be tensor-factored into 2x2 pieces.\n \"\"\"\n\n # Use the entry with the largest magnitude as a reference point.\n a, b = max(\n ((i, j) for i in range(4) for j in range(4)),\n key=lambda t: abs(matrix[t]))\n\n # Extract sub-factors touching the reference cell.\n f1 = np.zeros((2, 2), dtype=np.complex128)\n f2 = np.zeros((2, 2), dtype=np.complex128)\n for i in range(2):\n for j in range(2):\n f1[(a >> 1) ^ i, (b >> 1) ^ j] = matrix[a ^ (i << 1), b ^ (j << 1)]\n f2[(a & 1) ^ i, (b & 1) ^ j] = matrix[a ^ i, b ^ j]\n\n # Rescale factors to have unit determinants.\n f1 /= (np.sqrt(np.linalg.det(f1)) or 1)\n f2 /= (np.sqrt(np.linalg.det(f2)) or 1)\n\n # Determine global phase.\n g = matrix[a, b] / (f1[a >> 1, b >> 1] * f2[a & 1, b & 1])\n if np.real(g) < 0:\n f1 *= -1\n g = -g\n\n return g, f1, f2", "response": "Splits a 4x4 matrix U into A B and a pair of 2x2 unitaries."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef so4_to_magic_su2s(\n mat: np.ndarray,\n *,\n rtol: float = 1e-5,\n atol: float = 1e-8,\n check_preconditions: bool = True\n) -> Tuple[np.ndarray, np.ndarray]:\n \"\"\"Finds 2x2 special-unitaries A, B where mat = Mag.H @ kron(A, B) @ Mag.\n\n Mag is the magic basis matrix:\n\n 1 0 0 i\n 0 i 1 0\n 0 i -1 0 (times sqrt(0.5) to normalize)\n 1 0 0 -i\n\n Args:\n mat: A real 4x4 orthogonal matrix.\n rtol: Per-matrix-entry relative tolerance on equality.\n atol: Per-matrix-entry absolute tolerance on equality.\n check_preconditions: When set, the code verifies that the given\n matrix is from SO(4). Defaults to set.\n\n Returns:\n A pair (A, B) of matrices in SU(2) such that Mag.H @ kron(A, B) @ Mag\n is approximately equal to the given matrix.\n\n Raises:\n ValueError: Bad matrix.\n \"\"\"\n if check_preconditions:\n if mat.shape != (4, 4) or not predicates.is_special_orthogonal(\n mat, atol=atol, rtol=rtol):\n raise ValueError('mat must be 4x4 special orthogonal.')\n\n ab = combinators.dot(MAGIC, mat, MAGIC_CONJ_T)\n _, a, b = kron_factor_4x4_to_2x2s(ab)\n\n return a, b", "response": "This function converts a 4x4 orthogonal matrix to a 2x2 special - unitaries matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef kak_canonicalize_vector(x: float, y: float, z: float) -> KakDecomposition:\n\n phase = [complex(1)] # Accumulated global phase.\n left = [np.eye(2)] * 2 # Per-qubit left factors.\n right = [np.eye(2)] * 2 # Per-qubit right factors.\n v = [x, y, z] # Remaining XX/YY/ZZ interaction vector.\n\n # These special-unitary matrices flip the X, Y, and Z axes respectively.\n flippers = [\n np.array([[0, 1], [1, 0]]) * 1j,\n np.array([[0, -1j], [1j, 0]]) * 1j,\n np.array([[1, 0], [0, -1]]) * 1j\n ]\n\n # Each of these special-unitary matrices swaps two the roles of two axes.\n # The matrix at index k swaps the *other two* axes (e.g. swappers[1] is a\n # Hadamard operation that swaps X and Z).\n swappers = [\n np.array([[1, -1j], [1j, -1]]) * 1j * np.sqrt(0.5),\n np.array([[1, 1], [1, -1]]) * 1j * np.sqrt(0.5),\n np.array([[0, 1 - 1j], [1 + 1j, 0]]) * 1j * np.sqrt(0.5)\n ]\n\n # Shifting strength by \u00bd\u03c0 is equivalent to local ops (e.g. exp(i\u00bd\u03c0 XX)\u221dXX).\n def shift(k, step):\n v[k] += step * np.pi / 2\n phase[0] *= 1j ** step\n right[0] = combinators.dot(flippers[k] ** (step % 4), right[0])\n right[1] = combinators.dot(flippers[k] ** (step % 4), right[1])\n\n # Two negations is equivalent to temporarily flipping along the other axis.\n def negate(k1, k2):\n v[k1] *= -1\n v[k2] *= -1\n phase[0] *= -1\n s = flippers[3 - k1 - k2] # The other axis' flipper.\n left[1] = combinators.dot(left[1], s)\n right[1] = combinators.dot(s, right[1])\n\n # Swapping components is equivalent to temporarily swapping the two axes.\n def swap(k1, k2):\n v[k1], v[k2] = v[k2], v[k1]\n s = swappers[3 - k1 - k2] # The other axis' swapper.\n left[0] = combinators.dot(left[0], s)\n left[1] = combinators.dot(left[1], s)\n right[0] = combinators.dot(s, right[0])\n right[1] = combinators.dot(s, right[1])\n\n # Shifts an axis strength into the range (-\u03c0/4, \u03c0/4].\n def canonical_shift(k):\n while v[k] <= -np.pi / 4:\n shift(k, +1)\n while v[k] > np.pi / 4:\n shift(k, -1)\n\n # Sorts axis strengths into descending order by absolute magnitude.\n def sort():\n if abs(v[0]) < abs(v[1]):\n swap(0, 1)\n if abs(v[1]) < abs(v[2]):\n swap(1, 2)\n if abs(v[0]) < abs(v[1]):\n swap(0, 1)\n\n # Get all strengths to (-\u00bc\u03c0, \u00bc\u03c0] in descending order by absolute magnitude.\n canonical_shift(0)\n canonical_shift(1)\n canonical_shift(2)\n sort()\n\n # Move all negativity into z.\n if v[0] < 0:\n negate(0, 2)\n if v[1] < 0:\n negate(1, 2)\n canonical_shift(2)\n\n return KakDecomposition(\n global_phase=phase[0],\n single_qubit_operations_after=(left[1], left[0]),\n interaction_coefficients=(v[0], v[1], v[2]),\n single_qubit_operations_before=(right[1], right[0]))", "response": "Canonicalizes an XX Y Z interaction by swap - negation and shift - ing axes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecompose a 2 - qubit unitary into 1 - qubit ops and XX YZ interactions.", "response": "def kak_decomposition(\n mat: np.ndarray,\n rtol: float = 1e-5,\n atol: float = 1e-8) -> KakDecomposition:\n \"\"\"Decomposes a 2-qubit unitary into 1-qubit ops and XX/YY/ZZ interactions.\n\n Args:\n mat: The 4x4 unitary matrix to decompose.\n rtol: Per-matrix-entry relative tolerance on equality.\n atol: Per-matrix-entry absolute tolerance on equality.\n\n Returns:\n A `cirq.KakDecomposition` canonicalized such that the interaction\n coefficients x, y, z satisfy:\n\n 0 \u2264 abs(z) \u2264 y \u2264 x \u2264 \u03c0/4\n z \u2260 -\u03c0/4\n\n Raises:\n ValueError: Bad matrix.\n ArithmeticError: Failed to perform the decomposition.\n\n References:\n 'An Introduction to Cartan's KAK Decomposition for QC Programmers'\n https://arxiv.org/abs/quant-ph/0507171\n \"\"\"\n magic = np.array([[1, 0, 0, 1j],\n [0, 1j, 1, 0],\n [0, 1j, -1, 0],\n [1, 0, 0, -1j]]) * np.sqrt(0.5)\n gamma = np.array([[1, 1, 1, 1],\n [1, 1, -1, -1],\n [-1, 1, -1, 1],\n [1, -1, -1, 1]]) * 0.25\n\n # Diagonalize in magic basis.\n left, d, right = diagonalize.bidiagonalize_unitary_with_special_orthogonals(\n combinators.dot(np.conj(magic.T), mat, magic),\n atol=atol,\n rtol=rtol,\n check_preconditions=False)\n\n # Recover pieces.\n a1, a0 = so4_to_magic_su2s(left.T,\n atol=atol,\n rtol=rtol,\n check_preconditions=False)\n b1, b0 = so4_to_magic_su2s(right.T,\n atol=atol,\n rtol=rtol,\n check_preconditions=False)\n w, x, y, z = gamma.dot(np.vstack(np.angle(d))).flatten()\n g = np.exp(1j * w)\n\n # Canonicalize.\n inner_cannon = kak_canonicalize_vector(x, y, z)\n\n b1 = np.dot(inner_cannon.single_qubit_operations_before[0], b1)\n b0 = np.dot(inner_cannon.single_qubit_operations_before[1], b0)\n a1 = np.dot(a1, inner_cannon.single_qubit_operations_after[0])\n a0 = np.dot(a0, inner_cannon.single_qubit_operations_after[1])\n return KakDecomposition(\n interaction_coefficients=inner_cannon.interaction_coefficients,\n global_phase=g * inner_cannon.global_phase,\n single_qubit_operations_before=(b1, b0),\n single_qubit_operations_after=(a1, a0))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_mapping(mapping: Dict[ops.Qid, LogicalIndex],\n operations: ops.OP_TREE\n ) -> None:\n \"\"\"Updates a mapping (in place) from qubits to logical indices according to\n a set of permutation gates. Any gates other than permutation gates are\n ignored.\n\n Args:\n mapping: The mapping to update.\n operations: The operations to update according to.\n \"\"\"\n for op in ops.flatten_op_tree(operations):\n if (isinstance(op, ops.GateOperation) and\n isinstance(op.gate, PermutationGate)):\n op.gate.update_mapping(mapping, op.qubits)", "response": "Updates a mapping from qubits to logical indices according to the given operations."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating a mapping from qubits to logical indices.", "response": "def update_mapping(self, mapping: Dict[ops.Qid, LogicalIndex],\n keys: Sequence[ops.Qid]\n ) -> None:\n \"\"\"Updates a mapping (in place) from qubits to logical indices.\n\n Args:\n mapping: The mapping to update.\n keys: The qubits acted on by the gate.\n \"\"\"\n permutation = self.permutation()\n indices = tuple(permutation.keys())\n new_keys = [keys[permutation[i]] for i in indices]\n old_elements = [mapping[keys[i]] for i in indices]\n mapping.update(zip(new_keys, old_elements))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding text to the given location.", "response": "def write(self,\n x: int,\n y: int,\n text: str,\n transposed_text: 'Optional[str]' = None):\n \"\"\"Adds text to the given location.\n\n Args:\n x: The column in which to write the text.\n y: The row in which to write the text.\n text: The text to write at location (x, y).\n transposed_text: Optional text to write instead, if the text\n diagram is transposed.\n \"\"\"\n entry = self.entries.get((x, y), _DiagramText('', ''))\n self.entries[(x, y)] = _DiagramText(\n entry.text + text,\n entry.transposed_text + (transposed_text if transposed_text\n else text))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef content_present(self, x: int, y: int) -> bool:\n\n # Text?\n if (x, y) in self.entries:\n return True\n\n # Vertical line?\n if any(v.x == x and v.y1 < y < v.y2 for v in self.vertical_lines):\n return True\n\n # Horizontal line?\n if any(line_y == y and x1 < x < x2\n for line_y, x1, x2, _ in self.horizontal_lines):\n return True\n\n return False", "response": "Determines if a line or printed text is at the given location."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds a horizontal or vertical line from x1 y1 to x2 y2.", "response": "def grid_line(self, x1: int, y1: int, x2: int, y2: int,\n emphasize: bool = False):\n \"\"\"Adds a vertical or horizontal line from (x1, y1) to (x2, y2).\n\n Horizontal line is selected on equality in the second coordinate and\n vertical line is selected on equality in the first coordinate.\n\n Raises:\n ValueError: If line is neither horizontal nor vertical.\n \"\"\"\n if x1 == x2:\n self.vertical_line(x1, y1, y2, emphasize)\n elif y1 == y2:\n self.horizontal_line(y1, x1, x2, emphasize)\n else:\n raise ValueError(\"Line is neither horizontal nor vertical\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef vertical_line(self,\n x: Union[int, float],\n y1: Union[int, float],\n y2: Union[int, float],\n emphasize: bool = False\n ) -> None:\n \"\"\"Adds a line from (x, y1) to (x, y2).\"\"\"\n y1, y2 = sorted([y1, y2])\n self.vertical_lines.append(_VerticalLine(x, y1, y2, emphasize))", "response": "Adds a vertical line from x y1 y2 to x y2."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a horizontal line to the log.", "response": "def horizontal_line(self,\n y: Union[int, float],\n x1: Union[int, float],\n x2: Union[int, float],\n emphasize: bool = False\n ) -> None:\n \"\"\"Adds a line from (x1, y) to (x2, y).\"\"\"\n x1, x2 = sorted([x1, x2])\n self.horizontal_lines.append(_HorizontalLine(y, x1, x2, emphasize))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transpose(self) -> 'TextDiagramDrawer':\n out = TextDiagramDrawer()\n out.entries = {(y, x): _DiagramText(v.transposed_text, v.text)\n for (x, y), v in self.entries.items()}\n out.vertical_lines = [_VerticalLine(*e)\n for e in self.horizontal_lines]\n out.horizontal_lines = [_HorizontalLine(*e)\n for e in self.vertical_lines]\n out.vertical_padding = self.horizontal_padding.copy()\n out.horizontal_padding = self.vertical_padding.copy()\n return out", "response": "Returns the same diagram but mirrored across its diagonal."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef width(self) -> int:\n max_x = -1.0\n for x, _ in self.entries.keys():\n max_x = max(max_x, x)\n for v in self.vertical_lines:\n max_x = max(max_x, v.x)\n for h in self.horizontal_lines:\n max_x = max(max_x, h.x1, h.x2)\n return 1 + int(max_x)", "response": "Determines how many entry columns are in the diagram."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndetermines how many entry rows are in the diagram.", "response": "def height(self) -> int:\n \"\"\"Determines how many entry rows are in the diagram.\"\"\"\n max_y = -1.0\n for _, y in self.entries.keys():\n max_y = max(max_y, y)\n for h in self.horizontal_lines:\n max_y = max(max_y, h.y)\n for v in self.vertical_lines:\n max_y = max(max_y, v.y1, v.y2)\n return 1 + int(max_y)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchanging the padding after the given column.", "response": "def force_horizontal_padding_after(\n self, index: int, padding: Union[int, float]) -> None:\n \"\"\"Change the padding after the given column.\"\"\"\n self.horizontal_padding[index] = padding"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef force_vertical_padding_after(\n self, index: int, padding: Union[int, float]) -> None:\n \"\"\"Change the padding after the given row.\"\"\"\n self.vertical_padding[index] = padding", "response": "Change the padding after the given row."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef insert_empty_columns(self, x: int, amount: int = 1) -> None:\n def transform_columns(\n column: Union[int, float],\n row: Union[int, float]\n ) -> Tuple[Union[int, float], Union[int, float]]:\n return column + (amount if column >= x else 0), row\n self._transform_coordinates(transform_columns)", "response": "Insert a number of columns after the given column."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef insert_empty_rows(self, y: int, amount: int = 1) -> None:\n def transform_rows(\n column: Union[int, float],\n row: Union[int, float]\n ) -> Tuple[Union[int, float], Union[int, float]]:\n return column, row + (amount if row >= y else 0)\n self._transform_coordinates(transform_rows)", "response": "Insert a number of rows after the given row."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\noutput text containing the diagram.", "response": "def render(self,\n horizontal_spacing: int = 1,\n vertical_spacing: int = 1,\n crossing_char: str = None,\n use_unicode_characters: bool = True) -> str:\n \"\"\"Outputs text containing the diagram.\"\"\"\n\n block_diagram = BlockDiagramDrawer()\n\n w = self.width()\n h = self.height()\n\n # Communicate padding into block diagram.\n for x in range(0, w - 1):\n block_diagram.set_col_min_width(\n x*2 + 1,\n # Horizontal separation looks narrow, so partials round up.\n int(np.ceil(self.horizontal_padding.get(x, horizontal_spacing)))\n )\n block_diagram.set_col_min_width(x*2, 1)\n for y in range(0, h - 1):\n block_diagram.set_row_min_height(\n y*2 + 1,\n # Vertical separation looks wide, so partials round down.\n int(np.floor(self.vertical_padding.get(y, vertical_spacing)))\n )\n block_diagram.set_row_min_height(y*2, 1)\n\n # Draw vertical lines.\n for x_b, y1_b, y2_b, emphasize in self.vertical_lines:\n x = int(x_b * 2)\n y1, y2 = int(min(y1_b, y2_b) * 2), int(max(y1_b, y2_b) * 2)\n charset = pick_charset(use_unicode_characters, emphasize)\n\n # Caps.\n block_diagram.mutable_block(x, y1).draw_curve(\n charset, bottom=True)\n block_diagram.mutable_block(x, y2).draw_curve(\n charset, top=True)\n\n # Span.\n for y in range(y1 + 1, y2):\n block_diagram.mutable_block(x, y).draw_curve(\n charset, top=True, bottom=True)\n\n # Draw horizontal lines.\n for y_b, x1_b, x2_b, emphasize in self.horizontal_lines:\n y = int(y_b * 2)\n x1, x2 = int(min(x1_b, x2_b) * 2), int(max(x1_b, x2_b) * 2)\n charset = pick_charset(use_unicode_characters, emphasize)\n\n # Caps.\n block_diagram.mutable_block(x1, y).draw_curve(\n charset, right=True)\n block_diagram.mutable_block(x2, y).draw_curve(\n charset, left=True)\n\n # Span.\n for x in range(x1 + 1, x2):\n block_diagram.mutable_block(x, y).draw_curve(\n charset, left=True, right=True, crossing_char=crossing_char)\n\n # Place entries.\n for (x, y), v in self.entries.items():\n x *= 2\n y *= 2\n block_diagram.mutable_block(x, y).content = v.text\n\n return block_diagram.render()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef all_near_zero(a: Union[float, complex, Iterable[float], np.ndarray],\n *,\n atol: float = 1e-8) -> bool:\n \"\"\"Checks if the tensor's elements are all near zero.\n\n Args:\n a: Tensor of elements that could all be near zero.\n atol: Absolute tolerance.\n \"\"\"\n return np.all(np.less_equal(np.abs(a), atol))", "response": "Checks if the tensor s elements are all near zero."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if the elements of a tensor are all near zero.", "response": "def all_near_zero_mod(a: Union[float, complex, Iterable[float], np.ndarray],\n period: float,\n *,\n atol: float = 1e-8) -> bool:\n \"\"\"Checks if the tensor's elements are all near multiples of the period.\n\n Args:\n a: Tensor of elements that could all be near multiples of the period.\n period: The period, e.g. 2 pi when working in radians.\n atol: Absolute tolerance.\n \"\"\"\n b = (np.asarray(a) + period / 2) % period - period / 2\n return np.all(np.less_equal(np.abs(b), atol))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _git_fetch_for_comparison(remote: str,\n actual_branch: str,\n compare_branch: str,\n verbose: bool) -> prepared_env.PreparedEnv:\n \"\"\"Fetches two branches including their common ancestor.\n\n Limits the depth of the fetch to avoid unnecessary work. Scales up the\n depth exponentially and tries again when the initial guess is not deep\n enough.\n\n Args:\n remote: The location of the remote repository, in a format that the\n git command will understand.\n actual_branch: A remote branch or ref to fetch,\n compare_branch: Another remote branch or ref to fetch,\n verbose: When set, more progress output is produced.\n\n Returns:\n A ComparableCommits containing the commit id of the actual branch and\n a the id of a commit to compare against (e.g. for when doing incremental\n checks).\n \"\"\"\n actual_id = ''\n base_id = ''\n for depth in [10, 100, 1000, None]:\n depth_str = '' if depth is None else '--depth={}'.format(depth)\n\n shell_tools.run_cmd(\n 'git',\n 'fetch',\n None if verbose else '--quiet',\n remote,\n actual_branch,\n depth_str,\n log_run_to_stderr=verbose)\n actual_id = shell_tools.output_of('git', 'rev-parse', 'FETCH_HEAD')\n\n shell_tools.run_cmd('git',\n 'fetch',\n None if verbose else '--quiet',\n remote,\n compare_branch,\n depth_str,\n log_run_to_stderr=verbose)\n base_id = shell_tools.output_of('git', 'rev-parse', 'FETCH_HEAD')\n\n try:\n base_id = shell_tools.output_of(\n 'git',\n 'merge-base',\n actual_id,\n base_id)\n break\n except subprocess.CalledProcessError:\n # No common ancestor. We need to dig deeper.\n pass\n\n return prepared_env.PreparedEnv(None, actual_id, base_id, None, None)", "response": "Fetches two branches including their common ancestor and returns the actual branch and base branch."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fetch_github_pull_request(destination_directory: str,\n repository: github_repository.GithubRepository,\n pull_request_number: int,\n verbose: bool\n ) -> prepared_env.PreparedEnv:\n \"\"\"Uses content from github to create a dir for testing and comparisons.\n\n Args:\n destination_directory: The location to fetch the contents into.\n repository: The github repository that the commit lives under.\n pull_request_number: The id of the pull request to clone. If None, then\n the master branch is cloned instead.\n verbose: When set, more progress output is produced.\n\n Returns:\n Commit ids corresponding to content to test/compare.\n \"\"\"\n\n branch = 'pull/{}/head'.format(pull_request_number)\n os.chdir(destination_directory)\n print('chdir', destination_directory, file=sys.stderr)\n\n shell_tools.run_cmd(\n 'git',\n 'init',\n None if verbose else '--quiet',\n out=sys.stderr)\n result = _git_fetch_for_comparison(remote=repository.as_remote(),\n actual_branch=branch,\n compare_branch='master',\n verbose=verbose)\n shell_tools.run_cmd(\n 'git',\n 'branch',\n None if verbose else '--quiet',\n 'compare_commit',\n result.compare_commit_id,\n log_run_to_stderr=verbose)\n shell_tools.run_cmd(\n 'git',\n 'checkout',\n None if verbose else '--quiet',\n '-b',\n 'actual_commit',\n result.actual_commit_id,\n log_run_to_stderr=verbose)\n return prepared_env.PreparedEnv(\n github_repo=repository,\n actual_commit_id=result.actual_commit_id,\n compare_commit_id=result.compare_commit_id,\n destination_directory=destination_directory,\n virtual_env_path=None)", "response": "Uses content from github to create a dir for testing and comparison."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fetch_local_files(destination_directory: str,\n verbose: bool) -> prepared_env.PreparedEnv:\n \"\"\"Uses local files to create a directory for testing and comparisons.\n\n Args:\n destination_directory: The directory where the copied files should go.\n verbose: When set, more progress output is produced.\n\n Returns:\n Commit ids corresponding to content to test/compare.\n \"\"\"\n staging_dir = destination_directory + '-staging'\n try:\n shutil.copytree(get_repo_root(), staging_dir)\n os.chdir(staging_dir)\n if verbose:\n print('chdir', staging_dir, file=sys.stderr)\n\n shell_tools.run_cmd(\n 'git',\n 'add',\n '--all',\n out=sys.stderr,\n log_run_to_stderr=verbose)\n\n shell_tools.run_cmd(\n 'git',\n 'commit',\n '-m', 'working changes',\n '--allow-empty',\n '--no-gpg-sign',\n None if verbose else '--quiet',\n out=sys.stderr,\n log_run_to_stderr=verbose)\n\n cur_commit = shell_tools.output_of('git', 'rev-parse', 'HEAD')\n\n os.chdir(destination_directory)\n if verbose:\n print('chdir', destination_directory, file=sys.stderr)\n shell_tools.run_cmd('git',\n 'init',\n None if verbose else '--quiet',\n out=sys.stderr,\n log_run_to_stderr=verbose)\n result = _git_fetch_for_comparison(staging_dir,\n cur_commit,\n 'master',\n verbose=verbose)\n finally:\n shutil.rmtree(staging_dir, ignore_errors=True)\n\n shell_tools.run_cmd(\n 'git',\n 'branch',\n None if verbose else '--quiet',\n 'compare_commit',\n result.compare_commit_id,\n log_run_to_stderr=verbose)\n shell_tools.run_cmd(\n 'git',\n 'checkout',\n None if verbose else '--quiet',\n '-b',\n 'actual_commit',\n result.actual_commit_id,\n log_run_to_stderr=verbose)\n return prepared_env.PreparedEnv(\n github_repo=None,\n actual_commit_id=result.actual_commit_id,\n compare_commit_id=result.compare_commit_id,\n destination_directory=destination_directory,\n virtual_env_path=None)", "response": "Uses local files to create a directory for testing and comparison."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef circuit_to_tensorflow_runnable(\n circuit: circuits.Circuit,\n initial_state: Union[int, np.ndarray] = 0,\n ) -> ComputeFuncAndFeedDict:\n \"\"\"Returns a compute function and feed_dict for a `cirq.Circuit`'s output.\n\n `result.compute()` will return a `tensorflow.Tensor` with\n `tensorflow.placeholder` objects to be filled in by `result.feed_dict`, at\n which point it will evaluate to the output state vector of the circuit.\n\n You can apply further operations to the tensor returned by\n `result.compute`. This allows, for example, for the final result to be\n a small amount of computed data (e.g. an expectation value) instead of the\n gigantic raw state vector.\n\n The tensor returned by `result.compute` is intended to run efficiently\n on cloud TPUs. It will have dtype complex64 and a shape of (2**n,) where n\n is the number of qubits.\n\n Examples:\n To simulate the circuit with tensorflow in a normal session, forward\n this method's output into `tensorflow.Session.run` as follows:\n\n import tensorflow as tf\n r = circuit_to_tensorflow_runnable(...)\n with tf.Session() as session:\n output = session.run(r.compute(), feed_dict=r.feed_dict)\n print(output)\n\n Note that you can use the returned tensor in further computations. For\n example, to compute the chance of the system ending up in the first 128\n computational basis states you can use `tf.norm(tensor[:128], 2)`:\n\n import tensorflow as tf\n r = circuit_to_tensorflow_runnable(...)\n expectation = lambda: tf.norm(r.compute()[:128], 2)\n with tf.Session() as session:\n output = session.run(expectation, feed_dict=r.feed_dict)\n print(output)\n\n For documentation on running against cloud TPUs, see\n\n https://cloud.google.com/tpu/docs/quickstart#run_example\n\n Generally speaking, from within a cloud instance, you use\n `tf.contrib.tpu.rewrite` to convert the tensor into a TPU compatible\n form, initialize the TPU system, then run the rewritten tensor:\n\n import tensorflow as tf\n TPU_TARGET = ???????\n r = circuit_to_tensorflow_runnable(...YOUR_CIRCUIT...)\n rewritten_for_tpu = tf.contrib.tpu.rewrite(r.compute)\n\n with tf.Session(target=TPU_TARGET) as session:\n session.run(tf.contrib.tpu.initialize_system())\n output = session.run(rewritten_for_tpu, feed_dict=r.feed_dict)\n print(output)\n\n Args:\n circuit: The circuit to apply to `initial_state` to produce an output\n state vector.\n initial_state: The input into the circuit. If this is an integer, it\n indicates that the input state is a computational basis state where\n the k'th qubit is set by the k'th bit of the integer. If this is\n a numpy array, it should directly encode a normalized wavefunction.\n\n Returns:\n A ComputeFuncAndFeedDict, which is a named tuple whose first element is\n a function that returns a Tensor representing the output state vector\n that results from applying the given circuit to the given, and whose\n second element is a feed_dict containing important parameters describing\n that tensor.\n \"\"\"\n if not circuit.are_all_measurements_terminal():\n raise ValueError('not circuit.are_all_measurements_terminal()')\n\n t = _TensorCircuit(circuit, initial_state)\n return ComputeFuncAndFeedDict(t.compute, t.feed_dict)", "response": "Returns a compute function and feed_dict for a single circuit."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _circuit_as_layers(circuit: circuits.Circuit,\n grouping: _QubitGrouping) -> List[_TransformsThenCzs]:\n \"\"\"Transforms a circuit into a series of GroupMatrix+CZ layers.\n\n Args:\n circuit: The circuit to transform.\n grouping: How the circuit's qubits are combined into groups.\n\n Returns:\n A list of layers. Each layer has a matrix to apply to each group of\n qubits, and a list of CZs to apply to pairs of qubits crossing\n between groups.\n \"\"\"\n frontier = {q: 0 for q in circuit.all_qubits()}\n\n layers = []\n while True:\n # Pull within-group operations into per-group matrices.\n any_group_matrices = False\n group_matrices = []\n for g in grouping.groups:\n # Scan for reachable operations contained within the group qubits.\n start_frontier = {q: frontier[q] for q in g}\n end_frontier = circuit.reachable_frontier_from(start_frontier)\n mergeable_ops = circuit.findall_operations_between(start_frontier,\n end_frontier)\n\n # Advance frontier.\n for q, v in end_frontier.items():\n frontier[q] = v\n\n # Fold reachable operations into a single group matrix.\n group_matrix = np.eye(1 << len(g)).reshape((2, 2) * len(g))\n if mergeable_ops:\n any_group_matrices = True\n for _, op in mergeable_ops:\n group_matrix = linalg.targeted_left_multiply(\n left_matrix=protocols.unitary(op).reshape(\n (2, 2) * len(op.qubits)),\n right_target=group_matrix,\n target_axes=[grouping.loc(q)[1] for q in op.qubits])\n group_matrices.append(np.transpose(group_matrix.reshape(\n 1 << len(g), 1 << len(g))))\n\n # Scan for reachable CZ operations between groups.\n end_frontier = circuit.reachable_frontier_from(\n frontier,\n is_blocker=lambda op: grouping.all_in_same_group(*op.qubits))\n cz_ops = circuit.findall_operations_between(frontier, end_frontier)\n\n # Advance frontier.\n frontier = end_frontier\n\n # List out qubit index pairs for each CZ.\n cz_indices = []\n for _, cz in cz_ops:\n a, b = cz.qubits\n assert cz == ops.CZ(a, b)\n cz_indices.append((grouping.ind(a), grouping.ind(b)))\n\n # Combine group and CZ operations into a simulation layer.\n if not any_group_matrices and not cz_indices:\n break\n layer = _TransformsThenCzs(group_matrices=group_matrices,\n cz_indices=cz_indices)\n layers.append(layer)\n\n # We should have processed the whole circuit.\n assert frontier == {q: len(circuit) for q in circuit.all_qubits()}\n\n return layers", "response": "Transforms a circuit into a series of GroupMatrix + CZs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _deref(tensor: tf.Tensor, index: tf.Tensor) -> tf.Tensor:\n assert tensor.shape[0] > 0\n return _deref_helper(lambda i: tensor[i, ...],\n index,\n 0,\n tensor.shape[0] - 1)", "response": "Equivalent to tensor [ index... ]."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _multi_deref(tensors: List[tf.Tensor], index: tf.Tensor) -> List[tf.Tensor]:\n assert tensors\n assert tensors[0].shape[0] > 0\n return _deref_helper(lambda i: [tensor[i, ...] for tensor in tensors],\n index,\n 0,\n tensors[0].shape[0] - 1)", "response": "Equivalent to _deref for multi - tensor."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning the given circuit or schedule and returns the first result.", "response": "def run(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n param_resolver: 'study.ParamResolverOrSimilarType' = None,\n repetitions: int = 1,\n ) -> study.TrialResult:\n \"\"\"Samples from the given Circuit or Schedule.\n\n Args:\n program: The circuit or schedule to simulate.\n param_resolver: Parameters to run with the program.\n repetitions: The number of repetitions to simulate.\n\n Returns:\n TrialResult for a run.\n \"\"\"\n return self.run_sweep(program, study.ParamResolver(param_resolver),\n repetitions)[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: study.Sweepable,\n repetitions: int = 1,\n ) -> List[study.TrialResult]:\n \"\"\"Samples from the given Circuit or Schedule.\n\n In contrast to run, this allows for sweeping over different parameter\n values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n repetitions: The number of repetitions to simulate.\n\n Returns:\n TrialResult list for this run; one for each possible parameter\n resolver.\n \"\"\"", "response": "Runs the sweeping procedure over the given parameter set."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef measure(*qubits: raw_types.Qid,\n key: Optional[str] = None,\n invert_mask: Tuple[bool, ...] = ()\n ) -> gate_operation.GateOperation:\n \"\"\"Returns a single MeasurementGate applied to all the given qubits.\n\n The qubits are measured in the computational basis.\n\n Args:\n *qubits: The qubits that the measurement gate should measure.\n key: The string key of the measurement. If this is None, it defaults\n to a comma-separated list of the target qubits' str values.\n invert_mask: A list of Truthy or Falsey values indicating whether\n the corresponding qubits should be flipped. None indicates no\n inverting should be done.\n\n Returns:\n An operation targeting the given qubits with a measurement.\n\n Raises:\n ValueError if the qubits are not instances of Qid.\n \"\"\"\n for qubit in qubits:\n if isinstance(qubit, np.ndarray):\n raise ValueError(\n 'measure() was called a numpy ndarray. Perhaps you meant '\n 'to call measure_state_vector on numpy array?'\n )\n elif not isinstance(qubit, raw_types.Qid):\n raise ValueError(\n 'measure() was called with type different than Qid.')\n\n if key is None:\n key = _default_measurement_key(qubits)\n return MeasurementGate(len(qubits), key, invert_mask).on(*qubits)", "response": "Returns a MeasurementGate applied to all the given qubits."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of operations individually measuring the given qubits.", "response": "def measure_each(*qubits: raw_types.Qid,\n key_func: Callable[[raw_types.Qid], str] = str\n ) -> List[gate_operation.GateOperation]:\n \"\"\"Returns a list of operations individually measuring the given qubits.\n\n The qubits are measured in the computational basis.\n\n Args:\n *qubits: The qubits to measure.\n key_func: Determines the key of the measurements of each qubit. Takes\n the qubit and returns the key for that qubit. Defaults to str.\n\n Returns:\n A list of operations individually measuring the given qubits.\n \"\"\"\n return [MeasurementGate(1, key_func(q)).on(q) for q in qubits]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Rx(rads: Union[float, sympy.Basic]) -> XPowGate:\n pi = sympy.pi if protocols.is_parameterized(rads) else np.pi\n return XPowGate(exponent=rads / pi, global_shift=-0.5)", "response": "Returns a gate with the matrix e^ - i X rads / 2."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a gate with the matrix e^Y rads / 2.", "response": "def Ry(rads: Union[float, sympy.Basic]) -> YPowGate:\n \"\"\"Returns a gate with the matrix e^{-i Y rads / 2}.\"\"\"\n pi = sympy.pi if protocols.is_parameterized(rads) else np.pi\n return YPowGate(exponent=rads / pi, global_shift=-0.5)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Rz(rads: Union[float, sympy.Basic]) -> ZPowGate:\n pi = sympy.pi if protocols.is_parameterized(rads) else np.pi\n return ZPowGate(exponent=rads / pi, global_shift=-0.5)", "response": "Returns a ZPowGate with the matrix e^ - i Z rads."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _decompose_(self, qubits):\n a, b = qubits\n yield CNOT(a, b)\n yield CNOT(b, a) ** self._exponent\n yield CNOT(a, b)", "response": "Decomposes the set of qubits into a set of unique entries."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_args(self, qubits: Sequence[Qid]) -> None:\n if len(qubits) == 0:\n raise ValueError(\n \"Applied a gate to an empty set of qubits. Gate: {}\".format(\n repr(self)))\n\n if len(qubits) != self.num_qubits():\n raise ValueError(\n 'Wrong number of qubits for <{!r}>. '\n 'Expected {} qubits but got <{!r}>.'.format(\n self,\n self.num_qubits(),\n qubits))\n\n if any([not isinstance(qubit, Qid)\n for qubit in qubits]):\n raise ValueError(\n 'Gate was called with type different than Qid.')", "response": "Checks if this gate can be applied to the given qubits."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns an application of this gate to the given qubits.", "response": "def on(self, *qubits: Qid) -> 'gate_operation.GateOperation':\n \"\"\"Returns an application of this gate to the given qubits.\n\n Args:\n *qubits: The collection of qubits to potentially apply the gate to.\n \"\"\"\n # Avoids circular import.\n from cirq.ops import gate_operation\n return gate_operation.GateOperation(self, list(qubits))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a controlled version of this gate.", "response": "def controlled_by(self, *control_qubits: Qid) -> 'Gate':\n \"\"\"Returns a controlled version of this gate.\n\n Args:\n control_qubits: Optional qubits to control the gate by.\n \"\"\"\n # Avoids circular import.\n from cirq.ops import ControlledGate\n return ControlledGate(self, control_qubits,\n len(control_qubits) if control_qubits is not None\n else 1)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the same operation but with different qubits.", "response": "def transform_qubits(self: TSelf_Operation,\n func: Callable[[Qid], Qid]) -> TSelf_Operation:\n \"\"\"Returns the same operation, but with different qubits.\n\n Args:\n func: The function to use to turn each current qubit into a desired\n new qubit.\n\n Returns:\n The receiving operation but with qubits transformed by the given\n function.\n \"\"\"\n return self.with_qubits(*(func(q) for q in self.qubits))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef controlled_by(self, *control_qubits: Qid) -> 'Operation':\n # Avoids circular import.\n from cirq.ops import ControlledOperation\n if control_qubits is None or len(control_qubits) is 0:\n raise ValueError(\n \"Can't get controlled operation without control qubit. Op: {}\"\n .format(repr(self)))\n else:\n return ControlledOperation(control_qubits, self)", "response": "Returns a controlled version of this operation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef perform_check(self,\n env: env_tools.PreparedEnv,\n verbose: bool) -> Tuple[bool, str]:\n \"\"\"Evaluates the status check and returns a pass/fail with message.\n\n Args:\n env: Describes a prepared python 3 environment in which to run.\n verbose: When set, more progress output is produced.\n\n Returns:\n A tuple containing a pass/fail boolean and then a details message.\n \"\"\"", "response": "Evaluates the status check and returns a pass or fail with a details message."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run(self,\n env: env_tools.PreparedEnv,\n verbose: bool,\n previous_failures: Set['Check']) -> CheckResult:\n \"\"\"Evaluates this check.\n\n Args:\n env: The prepared python environment to run the check in.\n verbose: When set, more progress output is produced.\n previous_failures: Checks that have already run and failed.\n\n Returns:\n A CheckResult instance.\n \"\"\"\n\n # Skip if a dependency failed.\n if previous_failures.intersection(self.dependencies):\n print(shell_tools.highlight(\n 'Skipped ' + self.command_line_switch(),\n shell_tools.YELLOW))\n return CheckResult(\n self, False, 'Skipped due to dependency failing.', None)\n\n print(shell_tools.highlight(\n 'Running ' + self.command_line_switch(),\n shell_tools.GREEN))\n try:\n success, message = self.perform_check(env, verbose=verbose)\n result = CheckResult(self, success, message, None)\n except Exception as ex:\n result = CheckResult(self, False, 'Unexpected error.', ex)\n\n print(shell_tools.highlight(\n 'Finished ' + self.command_line_switch(),\n shell_tools.GREEN if result.success else shell_tools.RED))\n if verbose:\n print(result)\n\n return result", "response": "Runs the check in the given python environment."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pick_env_and_run_and_report(self,\n env: env_tools.PreparedEnv,\n env_py2: Optional[env_tools.PreparedEnv],\n verbose: bool,\n previous_failures: Set['Check']\n ) -> CheckResult:\n \"\"\"Evaluates this check in python 3 or 2.7, and reports to github.\n\n If the prepared environments are not linked to a github repository,\n with a known access token, reporting to github is skipped.\n\n Args:\n env: A prepared python 3 environment.\n env_py2: A prepared python 2.7 environment.\n verbose: When set, more progress output is produced.\n previous_failures: Checks that have already run and failed.\n\n Returns:\n A CheckResult instance.\n \"\"\"\n env.report_status_to_github('pending', 'Running...', self.context())\n chosen_env = cast(env_tools.PreparedEnv,\n env_py2 if self.needs_python2_env() else env)\n os.chdir(cast(str, chosen_env.destination_directory))\n\n result = self.run(chosen_env, verbose, previous_failures)\n\n if result.unexpected_error is not None:\n env.report_status_to_github('error',\n 'Unexpected error.',\n self.context())\n else:\n env.report_status_to_github(\n 'success' if result.success else 'failure',\n result.message,\n self.context())\n\n return result", "response": "Selects the environment and runs it and reports the result to github."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_resolvers(sweepable: Sweepable) -> List[ParamResolver]:\n if isinstance(sweepable, ParamResolver):\n return [sweepable]\n elif isinstance(sweepable, Sweep):\n return list(sweepable)\n elif isinstance(sweepable, collections.Iterable):\n iterable = cast(collections.Iterable, sweepable)\n return list(iterable) if isinstance(next(iter(iterable)),\n ParamResolver) else sum(\n [list(s) for s in iterable], [])\n raise TypeError('Unexpected Sweepable type.')", "response": "Convert a Sweepable to a list of ParamResolvers."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns whether the object is parameterized with any unresolved Symbols.", "response": "def is_parameterized(val: Any) -> bool:\n \"\"\"Returns whether the object is parameterized with any Symbols.\n\n A value is parameterized when it has an `_is_parameterized_` method and\n that method returns a truthy value, or if the value is an instance of\n sympy.Basic.\n\n Returns:\n True if the gate has any unresolved Symbols\n and False otherwise. If no implementation of the magic\n method above exists or if that method returns NotImplemented,\n this will default to False.\n \"\"\"\n if isinstance(val, sympy.Basic):\n return True\n\n getter = getattr(val, '_is_parameterized_', None)\n result = NotImplemented if getter is None else getter()\n\n if result is not NotImplemented:\n return result\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef resolve_parameters(\n val: Any,\n param_resolver: 'cirq.ParamResolverOrSimilarType') -> Any:\n \"\"\"Resolves symbol parameters in the effect using the param resolver.\n\n This function will use the `_resolve_parameters_` magic method\n of `val` to resolve any Symbols with concrete values from the given\n parameter resolver.\n\n Args:\n val: The object to resolve (e.g. the gate, operation, etc)\n param_resolver: the object to use for resolving all symbols\n\n Returns:\n a gate or operation of the same type, but with all Symbols\n replaced with floats according to the given ParamResolver.\n If `val` has no `_resolve_parameters_` method or if it returns\n NotImplemented, `val` itself is returned.\n \"\"\"\n if not param_resolver:\n return val\n\n # Ensure its a dictionary wrapped in a ParamResolver.\n from cirq import ParamResolver # HACK: break cycle.\n param_resolver = ParamResolver(param_resolver)\n if isinstance(val, sympy.Basic):\n return param_resolver.value_of(val)\n\n getter = getattr(val, '_resolve_parameters_', None)\n result = NotImplemented if getter is None else getter(param_resolver)\n\n if result is not NotImplemented:\n return result\n else:\n return val", "response": "Resolves symbol parameters in the effect using the given ParamResolver."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef circuit_to_latex_using_qcircuit(\n circuit: circuits.Circuit,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT) -> str:\n \"\"\"Returns a QCircuit-based latex diagram of the given circuit.\n\n Args:\n circuit: The circuit to represent in latex.\n qubit_order: Determines the order of qubit wires in the diagram.\n\n Returns:\n Latex code for the diagram.\n \"\"\"\n diagram = circuit.to_text_diagram_drawer(\n qubit_namer=qcircuit_qubit_namer,\n qubit_order=qubit_order,\n get_circuit_diagram_info=get_qcircuit_diagram_info)\n return _render(diagram)", "response": "Returns a LaTeX code for the given circuit."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef kron(*matrices: np.ndarray) -> np.ndarray:\n product = np.eye(1)\n for m in matrices:\n product = np.kron(product, m)\n return np.array(product)", "response": "Computes the kronecker product of a sequence of matrices."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the kronecker product of a sequence of matrices and controls.", "response": "def kron_with_controls(*matrices: np.ndarray) -> np.ndarray:\n \"\"\"Computes the kronecker product of a sequence of matrices and controls.\n\n Use linalg.CONTROL_TAG to represent controls. Any entry of the output\n matrix corresponding to a situation where the control is not satisfied will\n be overwritten by identity matrix elements.\n\n The control logic works by imbuing NaN with the meaning \"failed to meet one\n or more controls\". The normal kronecker product then spreads the per-item\n NaNs to all the entries in the product that need to be replaced by identity\n matrix elements. This method rewrites those NaNs. Thus CONTROL_TAG can be\n the matrix [[NaN, 0], [0, 1]] or equivalently [[NaN, NaN], [NaN, 1]].\n\n Because this method re-interprets NaNs as control-failed elements, it won't\n propagate error-indicating NaNs from its input to its output in the way\n you'd otherwise expect.\n\n Args:\n *matrices: The matrices and controls to combine with the kronecker\n product.\n\n Returns:\n The resulting matrix.\n \"\"\"\n\n product = kron(*matrices)\n\n # The NaN from CONTROL_TAG spreads to everywhere identity belongs.\n for i in range(product.shape[0]):\n for j in range(product.shape[1]):\n if np.isnan(product[i, j]):\n product[i, j] = 1 if i == j else 0\n\n return product"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes the dot/matrix product of a sequence of values. A *args version of np.linalg.multi_dot. Args: *values: The values to combine with the dot/matrix product. Returns: The resulting value or matrix.", "response": "def dot(*values: Union[float, complex, np.ndarray]\n ) -> Union[float, complex, np.ndarray]:\n \"\"\"Computes the dot/matrix product of a sequence of values.\n\n A *args version of np.linalg.multi_dot.\n\n Args:\n *values: The values to combine with the dot/matrix product.\n\n Returns:\n The resulting value or matrix.\n \"\"\"\n if len(values) == 1:\n if isinstance(values[0], np.ndarray):\n return np.array(values[0])\n return values[0]\n return np.linalg.multi_dot(values)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef block_diag(*blocks: np.ndarray) -> np.ndarray:\n for b in blocks:\n if b.shape[0] != b.shape[1]:\n raise ValueError('Blocks must be square.')\n\n if not blocks:\n return np.zeros((0, 0), dtype=np.complex128)\n\n n = sum(b.shape[0] for b in blocks)\n dtype = functools.reduce(_merge_dtypes, (b.dtype for b in blocks))\n\n result = np.zeros(shape=(n, n), dtype=dtype)\n i = 0\n for b in blocks:\n j = i + b.shape[0]\n result[i:j, i:j] = b\n i = j\n\n return result", "response": "Concatenates blocks into a block diagonal matrix."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind the first index of a target item within a list of lists.", "response": "def index_2d(seqs: List[List[Any]], target: Any) -> Tuple[int, int]:\n \"\"\"Finds the first index of a target item within a list of lists.\n\n Args:\n seqs: The list of lists to search.\n target: The item to find.\n\n Raises:\n ValueError: Item is not present.\n \"\"\"\n for i in range(len(seqs)):\n for j in range(len(seqs[i])):\n if seqs[i][j] == target:\n return i, j\n raise ValueError('Item not present.')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nissue new linear sequence search.", "response": "def search(\n self,\n trace_func: Callable[\n [List[LineSequence], float, float, float, bool],\n None] = None) -> List[LineSequence]:\n \"\"\"Issues new linear sequence search.\n\n Each call to this method starts new search.\n\n Args:\n trace_func: Optional callable which will be called for each simulated\n annealing step with arguments: solution candidate (list of linear\n sequences on the chip), current temperature (float), candidate cost\n (float), probability of accepting candidate (float), and acceptance\n decision (boolean).\n\n Returns:\n List of linear sequences on the chip found by this method.\n \"\"\"\n\n def search_trace(state: _STATE, temp: float,\n cost: float, probability: float, accepted: bool):\n if trace_func:\n trace_seqs, _ = state\n trace_func(trace_seqs, temp, cost, probability, accepted)\n\n seqs, _ = optimization.anneal_minimize(\n self._create_initial_solution(),\n self._quadratic_sum_cost,\n self._force_edges_active_move,\n self._rand.random_sample,\n trace_func=search_trace)\n return seqs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _quadratic_sum_cost(self, state: _STATE) -> float:\n cost = 0.0\n total_len = float(len(self._c))\n seqs, _ = state\n for seq in seqs:\n cost += (len(seq) / total_len) ** 2\n return -cost", "response": "Function that sums squares of lengths of sequences."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmove function which repeats _force_edge_active_move a few times.", "response": "def _force_edges_active_move(self, state: _STATE) -> _STATE:\n \"\"\"Move function which repeats _force_edge_active_move a few times.\n\n Args:\n state: Search state, not mutated.\n\n Returns:\n New search state which consists of incremental changes of the\n original state.\n \"\"\"\n for _ in range(self._rand.randint(1, 4)):\n state = self._force_edge_active_move(state)\n return state"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _force_edge_active_move(self, state: _STATE) -> _STATE:\n seqs, edges = state\n unused_edges = edges.copy()\n\n # List edges which do not belong to any linear sequence.\n for seq in seqs:\n for i in range(1, len(seq)):\n unused_edges.remove(self._normalize_edge((seq[i - 1], seq[i])))\n\n edge = self._choose_random_edge(unused_edges)\n if not edge:\n return seqs, edges\n\n return (\n self._force_edge_active(seqs,\n edge,\n lambda: bool(self._rand.randint(2))),\n edges)", "response": "Move which forces a random edge to appear on some sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _force_edge_active(self, seqs: List[List[GridQubit]], edge: EDGE,\n sample_bool: Callable[[], bool]\n ) -> List[List[GridQubit]]:\n \"\"\"Move which forces given edge to appear on some sequence.\n\n Args:\n seqs: List of linear sequences covering chip.\n edge: Edge to be activated.\n sample_bool: Callable returning random bool.\n\n Returns:\n New list of linear sequences with given edge on some of the\n sequences.\n \"\"\"\n\n n0, n1 = edge\n\n # Make a copy of original sequences.\n seqs = list(seqs)\n\n # Localize edge nodes within current solution.\n i0, j0 = index_2d(seqs, n0)\n i1, j1 = index_2d(seqs, n1)\n s0 = seqs[i0]\n s1 = seqs[i1]\n\n # Handle case when nodes belong to different linear sequences,\n # separately from the case where they belong to a single linear\n # sequence.\n if i0 != i1:\n\n # Split s0 and s1 in two parts: s0 in parts before n0, and after n0\n # (without n0); s1 in parts before n1, and after n1 (without n1).\n part = [s0[:j0], s0[j0 + 1:]], [s1[:j1], s1[j1 + 1:]]\n\n # Remove both sequences from original list.\n del seqs[max(i0, i1)]\n del seqs[min(i0, i1)]\n\n # Choose part of s0 which will be attached to n0, and make sure it\n # can be attached in the end.\n c0 = 0 if not part[0][1] else 1 if not part[0][\n 0] else sample_bool()\n if c0:\n part[0][c0].reverse()\n\n # Choose part of s1 which will be attached to n1, and make sure it\n # can be attached in the beginning.\n c1 = 0 if not part[1][1] else 1 if not part[1][\n 0] else sample_bool()\n if not c1:\n part[1][c1].reverse()\n\n # Append newly formed sequence from the chosen parts and new edge.\n seqs.append(part[0][c0] + [n0, n1] + part[1][c1])\n\n # Append the left-overs to the solution, if they exist.\n other = [1, 0]\n seqs.append(part[0][other[c0]])\n seqs.append(part[1][other[c1]])\n else:\n # Swap nodes so that n0 always preceeds n1 on sequence.\n if j0 > j1:\n j0, j1 = j1, j0\n n0, n1 = n1, n0\n\n # Split sequence in three parts, without nodes n0 an n1 present:\n # head might end with n0, inner might begin with n0 and end with\n # n1, tail might begin with n1.\n head = s0[:j0]\n inner = s0[j0 + 1:j1]\n tail = s0[j1 + 1:]\n\n # Remove original sequence from sequences list.\n del seqs[i0]\n\n # Either append edge to inner section, or attach it between head\n # and tail.\n if sample_bool():\n # Append edge either before or after inner section.\n if sample_bool():\n seqs.append(inner + [n1, n0] + head[::-1])\n seqs.append(tail)\n else:\n seqs.append(tail[::-1] + [n1, n0] + inner)\n seqs.append(head)\n else:\n # Form a new sequence from head, tail, and new edge.\n seqs.append(head + [n0, n1] + tail)\n seqs.append(inner)\n\n return [e for e in seqs if e]", "response": "Move which forces given edge to appear on some of the linear sequences."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _create_initial_solution(self) -> _STATE:\n\n def extract_sequences() -> List[List[GridQubit]]:\n \"\"\"Creates list of sequcenes for initial state.\n\n Returns:\n List of lists of sequences constructed on the chip.\n \"\"\"\n seqs = []\n prev = None\n seq = None\n for node in self._c:\n if prev is None:\n seq = [node]\n else:\n if node in self._c_adj[prev]:\n # Expand current sequence.\n seq.append(node)\n else:\n # Create new sequence, there is no connection between\n # nodes.\n seqs.append(seq)\n seq = [node]\n prev = node\n if seq:\n seqs.append(seq)\n return seqs\n\n def assemble_edges() -> Set[EDGE]:\n \"\"\"Creates list of edges for initial state.\n\n Returns:\n List of all possible edges.\n \"\"\"\n nodes_set = set(self._c)\n edges = set()\n for n in self._c:\n if above(n) in nodes_set:\n edges.add(self._normalize_edge((n, above(n))))\n if right_of(n) in nodes_set:\n edges.add(self._normalize_edge((n, right_of(n))))\n return edges\n\n return extract_sequences(), assemble_edges()", "response": "Creates an initial solution based on the chip description."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _normalize_edge(self, edge: EDGE) -> EDGE:\n\n def lower(n: GridQubit, m: GridQubit) -> bool:\n return n.row < m.row or (n.row == m.row and n.col < m.col)\n\n n1, n2 = edge\n return (n1, n2) if lower(n1, n2) else (n2, n1)", "response": "Normalizes an edge to be unique and return it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\npicking a random edge from the set of edges.", "response": "def _choose_random_edge(self, edges: Set[EDGE]) -> Optional[EDGE]:\n \"\"\"Picks random edge from the set of edges.\n\n Args:\n edges: Set of edges to pick from.\n\n Returns:\n Random edge from the supplied set, or None for empty set.\n \"\"\"\n if edges:\n index = self._rand.randint(len(edges))\n for e in edges:\n if not index:\n return e\n index -= 1\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nruns an annealing sequence search.", "response": "def place_line(self,\n device: 'cirq.google.XmonDevice',\n length: int) -> GridQubitLineTuple:\n \"\"\"Runs line sequence search.\n\n Args:\n device: Chip description.\n length: Required line length.\n\n Returns:\n List of linear sequences on the chip found by simulated annealing\n method.\n \"\"\"\n seqs = AnnealSequenceSearch(device, self.seed).search(self.trace_func)\n return GridQubitLineTuple.best_of(seqs, length)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _decompose_inside_control(self,\n target1: raw_types.Qid,\n control: raw_types.Qid,\n target2: raw_types.Qid\n ) -> op_tree.OP_TREE:\n \"\"\"A decomposition assuming the control separates the targets.\n\n target1: \u2500@\u2500X\u2500\u2500\u2500\u2500\u2500\u2500\u2500T\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500X\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500X^-0.5\u2500\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n control: \u2500X\u2500@\u2500X\u2500\u2500\u2500\u2500\u2500@\u2500T^-1\u2500X\u2500@\u2500T\u2500\u2500\u2500\u2500X\u2500@\u2500X^0.5\u2500@\u2500@\u2500X\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n target2: \u2500\u2500\u2500\u2500\u2500@\u2500H\u2500T\u2500X\u2500T\u2500\u2500\u2500\u2500\u2500\u2500X\u2500T^-1\u2500\u2500\u2500X\u2500T^-1\u2500\u2500\u2500\u2500X\u2500\u2500\u2500X\u2500H\u2500S^-1\u2500\u2500\u2500\n \"\"\"\n a, b, c = target1, control, target2\n yield common_gates.CNOT(a, b)\n yield common_gates.CNOT(b, a)\n yield common_gates.CNOT(c, b)\n yield common_gates.H(c)\n yield common_gates.T(c)\n yield common_gates.CNOT(b, c)\n yield common_gates.T(a)\n yield common_gates.T(b)**-1\n yield common_gates.T(c)\n yield common_gates.CNOT(a, b)\n yield common_gates.CNOT(b, c)\n yield common_gates.T(b)\n yield common_gates.T(c)**-1\n yield common_gates.CNOT(a, b)\n yield common_gates.CNOT(b, c)\n yield pauli_gates.X(b)**0.5\n yield common_gates.T(c)**-1\n yield common_gates.CNOT(b, a)\n yield common_gates.CNOT(b, c)\n yield common_gates.CNOT(a, b)\n yield common_gates.CNOT(b, c)\n yield common_gates.H(c)\n yield common_gates.S(c)**-1\n yield pauli_gates.X(a)**-0.5", "response": "A decomposition assuming the control separates the targets."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _decompose_outside_control(self,\n control: raw_types.Qid,\n near_target: raw_types.Qid,\n far_target: raw_types.Qid\n ) -> op_tree.OP_TREE:\n \"\"\"A decomposition assuming one of the targets is in the middle.\n\n control: \u2500\u2500\u2500T\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502 \u2502 \u2502\n near: \u2500X\u2500T\u2500\u2500\u2500\u2500\u2500\u2500X\u2500@\u2500T^-1\u2500X\u2500@\u2500X\u2500\u2500\u2500\u2500@\u2500X^0.5\u2500X\u2500@\u2500X^0.5\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502 \u2502 \u2502 \u2502\n far: \u2500@\u2500Y^-0.5\u2500T\u2500X\u2500T\u2500\u2500\u2500\u2500\u2500\u2500X\u2500T^-1\u2500X\u2500T^-1\u2500\u2500\u2500\u2500X\u2500S\u2500\u2500\u2500\u2500\u2500X^-0.5\u2500\n \"\"\"\n a, b, c = control, near_target, far_target\n\n t = common_gates.T\n sweep_abc = [common_gates.CNOT(a, b),\n common_gates.CNOT(b, c)]\n\n yield common_gates.CNOT(c, b)\n yield pauli_gates.Y(c)**-0.5\n yield t(a), t(b), t(c)\n yield sweep_abc\n yield t(b) ** -1, t(c)\n yield sweep_abc\n yield t(c) ** -1\n yield sweep_abc\n yield t(c) ** -1\n yield pauli_gates.X(b)**0.5\n yield sweep_abc\n yield common_gates.S(c)\n yield pauli_gates.X(b)**0.5\n yield pauli_gates.X(c)**-0.5", "response": "A decomposition assuming one of the targets is in the middle."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mixture(\n val: Any,\n default: Any = RaiseTypeErrorIfNotProvided) -> Sequence[Tuple[float, Any]]:\n \"\"\"Return a sequence of tuples representing a probabilistic combination.\n\n A mixture is described by an iterable of tuples of the form\n\n (probability of object, object)\n\n The probability components of the tuples must sum to 1.0 and be between\n 0 and 1 (inclusive).\n\n Args:\n val: The value whose mixture is being computed.\n default: A default value if val does not support mixture.\n\n Returns:\n An iterable of tuples of size 2. The first element of the tuple is a\n probability (between 0 and 1) and the second is the object that occurs\n with that probability in the mixture. The probabilities will sum to 1.0.\n \"\"\"\n\n getter = getattr(val, '_mixture_', None)\n result = NotImplemented if getter is None else getter()\n\n if result is not NotImplemented:\n return result\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if getter is None:\n raise TypeError(\n \"object of type '{}' has no _mixture_ method.\".format(type(val)))\n\n raise TypeError(\"object of type '{}' does have a _mixture_ method, \"\n \"but it returned NotImplemented.\".format(type(val)))", "response": "Returns a sequence of tuples representing a mixture of objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef has_mixture(val: Any) -> bool:\n getter = getattr(val, '_has_mixture_', None)\n result = NotImplemented if getter is None else getter()\n if result is not NotImplemented:\n return result\n\n # No _has_mixture_ function, use _mixture_ instead\n return mixture(val, None) is not None", "response": "Returns whether the value has a mixture representation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mixture_channel(\n val: Any,\n default: Any = RaiseTypeErrorIfNotProvided) -> Sequence[\n Tuple[float, np.ndarray]]:\n \"\"\"Return a sequence of tuples for a channel that is a mixture of unitaries.\n\n In contrast to `mixture` this method falls back to `unitary` if `_mixture_`\n is not implemented.\n\n A mixture channel is described by an iterable of tuples of the form\n\n (probability of unitary, unitary)\n\n The probability components of the tuples must sum to 1.0 and be between\n 0 and 1 (inclusive) and the `unitary` must be a unitary matrix.\n\n Args:\n val: The value whose mixture_channel is being computed.\n default: A default value if val does not support mixture.\n\n Returns:\n An iterable of tuples of size 2. The first element of the tuple is a\n probability (between 0 and 1) and the second is the unitary that occurs\n with that probability. The probabilities will sum to 1.0.\n \"\"\"\n mixture_getter = getattr(val, '_mixture_', None)\n result = NotImplemented if mixture_getter is None else mixture_getter()\n if result is not NotImplemented:\n return result\n\n unitary_getter = getattr(val, '_unitary_', None)\n result = NotImplemented if unitary_getter is None else unitary_getter()\n if result is not NotImplemented:\n return ((1.0, result),)\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if mixture_getter is None and unitary_getter is None:\n raise TypeError(\n \"object of type '{}' has no _mixture_ or _unitary_ method.\"\n .format(type(val)))\n\n raise TypeError(\"object of type '{}' does have a _mixture_ or _unitary_ \"\n \"method, but it returned NotImplemented.\".format(type(val)))", "response": "Returns a sequence of tuples for a mixture of unitaries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning whether the value has a mixture channel representation.", "response": "def has_mixture_channel(val: Any) -> bool:\n \"\"\"Returns whether the value has a mixture channel representation.\n\n In contrast to `has_mixture` this method falls back to checking whether\n the value has a unitary representation via `has_channel`.\n\n Returns:\n If `val` has a `_has_mixture_` method and its result is not\n NotImplemented, that result is returned. Otherwise, if `val` has a\n `_has_unitary_` method and its results is not NotImplemented, that\n result is returned. Otherwise, if the value has a `_mixture_` method\n that is not a non-default value, True is returned. Returns False if none\n of these functions.\n \"\"\"\n mixture_getter = getattr(val, '_has_mixture_', None)\n result = NotImplemented if mixture_getter is None else mixture_getter()\n if result is not NotImplemented:\n return result\n\n result = has_unitary(val)\n if result is not NotImplemented and result:\n return result\n\n # No _has_mixture_ or _has_unitary_ function, use _mixture_ instead.\n return mixture_channel(val, None) is not None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating that the mixture s tuple are valid probabilities.", "response": "def validate_mixture(supports_mixture: SupportsMixture):\n \"\"\"Validates that the mixture's tuple are valid probabilities.\"\"\"\n mixture_tuple = mixture(supports_mixture, None)\n if mixture_tuple is None:\n raise TypeError('{}_mixture did not have a _mixture_ method'.format(\n supports_mixture))\n\n def validate_probability(p, p_str):\n if p < 0:\n raise ValueError('{} was less than 0.'.format(p_str))\n elif p > 1:\n raise ValueError('{} was greater than 1.'.format(p_str))\n\n total = 0.0\n for p, val in mixture_tuple:\n validate_probability(p, '{}\\'s probability'.format(str(val)))\n total += p\n if not np.isclose(total, 1.0):\n raise ValueError('Sum of probabilities of a mixture was not 1.0')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the display states for the given base - class and returns the result.", "response": "def compute_displays(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n param_resolver: study.ParamResolver = study.ParamResolver({}),\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Union[int, np.ndarray] = 0,\n ) -> study.ComputeDisplaysResult:\n \"\"\"Computes displays in the supplied Circuit or Schedule.\n\n Args:\n program: The circuit or schedule to simulate.\n param_resolver: Parameters to run with the program.\n qubit_order: Determines the canonical ordering of the qubits used\n to define the order of amplitudes in the wave function.\n initial_state: If an int, the state is set to the computational\n basis state corresponding to this state. Otherwise if this\n is a np.ndarray it is the full initial state. In this case it\n must be the correct size, be normalized (an L2 norm of 1), and\n be safely castable to an appropriate dtype for the simulator.\n\n Returns:\n ComputeDisplaysResult for the simulation.\n \"\"\"\n return self.compute_displays_sweep(\n program, [param_resolver], qubit_order, initial_state)[0]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\naccumulate single qubit phase gates into the scratch shards.", "response": "def _single_qubit_accumulate_into_scratch(args: Dict[str, Any]):\n \"\"\"Accumulates single qubit phase gates into the scratch shards.\"\"\"\n index = args['indices'][0]\n shard_num = args['shard_num']\n half_turns = args['half_turns']\n num_shard_qubits = args['num_shard_qubits']\n scratch = _scratch_shard(args)\n\n # ExpZ = exp(-i pi Z half_turns / 2).\n if index >= num_shard_qubits:\n # Acts on prefix qubits.\n sign = 1 - 2 * _kth_bit(shard_num, index - num_shard_qubits)\n scratch -= half_turns * sign\n else:\n # Acts on shard qubits.\n scratch -= half_turns * _pm_vects(args)[index]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _one_projector(args: Dict[str, Any], index: int) -> Union[int, np.ndarray]:\n num_shard_qubits = args['num_shard_qubits']\n shard_num = args['shard_num']\n if index >= num_shard_qubits:\n return _kth_bit(shard_num, index - num_shard_qubits)\n return _zero_one_vects(args)[index]", "response": "Returns a projector onto the |1 > subspace of the index - th qubit."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _two_qubit_accumulate_into_scratch(args: Dict[str, Any]):\n index0, index1 = args['indices']\n half_turns = args['half_turns']\n scratch = _scratch_shard(args)\n\n projector = _one_projector(args, index0) * _one_projector(args, index1)\n # Exp11 = exp(-i pi |11><11| half_turns), but we accumulate phases as\n # pi / 2.\n scratch += 2 * half_turns * projector", "response": "Accumulates two qubit phase gates into the scratch shards."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _apply_scratch_as_phase(args: Dict[str, Any]):\n state = _state_shard(args)\n state *= np.exp(I_PI_OVER_2 * _scratch_shard(args))", "response": "Takes scratch shards and applies them as exponentiated phase to state.\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply a W gate when the gate acts only within a shard.", "response": "def _w_within_shard(args: Dict[str, Any]):\n \"\"\"Applies a W gate when the gate acts only within a shard.\"\"\"\n index = args['index']\n half_turns = args['half_turns']\n axis_half_turns = args['axis_half_turns']\n state = _state_shard(args)\n pm_vect = _pm_vects(args)[index]\n num_shard_qubits = args['num_shard_qubits']\n shard_size = 2 ** num_shard_qubits\n\n reshape_tuple = (2 ** (num_shard_qubits - 1 - index), 2, 2 ** index)\n perm_state = np.reshape(\n np.reshape(state, reshape_tuple)[:, ::-1, :], shard_size)\n cos = np.cos(-0.5 * np.pi * half_turns)\n sin = np.sin(-0.5 * np.pi * half_turns)\n\n cos_axis = np.cos(np.pi * axis_half_turns)\n sin_axis = np.sin(np.pi * axis_half_turns)\n\n new_state = cos * state + 1j * sin * perm_state * (\n cos_axis - 1j * sin_axis * pm_vect)\n np.copyto(state, new_state)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _w_between_shards(args: Dict[str, Any]):\n shard_num = args['shard_num']\n state = _state_shard(args)\n num_shard_qubits = args['num_shard_qubits']\n index = args['index']\n half_turns = args['half_turns']\n\n axis_half_turns = args['axis_half_turns']\n\n perm_index = shard_num ^ (1 << (index - num_shard_qubits))\n perm_state = mem_manager.SharedMemManager.get_array(\n args['state_handle']).view(np.complex64)[perm_index]\n\n cos = np.cos(-0.5 * np.pi * half_turns)\n sin = np.sin(-0.5 * np.pi * half_turns)\n\n cos_axis = np.cos(np.pi * axis_half_turns)\n sin_axis = np.sin(np.pi * axis_half_turns)\n\n scratch = _scratch_shard(args)\n z_op = (1 - 2 * _kth_bit(shard_num, index - num_shard_qubits))\n np.copyto(scratch, state * cos + 1j * sin * perm_state *\n (cos_axis - 1j * sin_axis * z_op))", "response": "Applies a W gate when the gate acts between shards."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _copy_scratch_to_state(args: Dict[str, Any]):\n np.copyto(_state_shard(args), _scratch_shard(args))", "response": "Copes scratch shards to state shards."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _one_prob_per_shard(args: Dict[str, Any]) -> float:\n index = args['index']\n\n state = _state_shard(args) * _one_projector(args, index)\n norm = np.linalg.norm(state)\n return norm * norm", "response": "Returns the probability of getting a one measurement on a state shard."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _norm_squared(args: Dict[str, Any]) -> float:\n state = _state_shard(args)\n return np.sum(np.abs(state) ** 2)", "response": "Returns the norm for each state shard."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nproject state shards onto the appropriate post measurement state.", "response": "def _collapse_state(args: Dict[str, Any]):\n \"\"\"Projects state shards onto the appropriate post measurement state.\n\n This function makes no assumptions about the interpretation of quantum\n theory.\n\n Args:\n args: The args from shard_num_args.\n \"\"\"\n index = args['index']\n result = args['result']\n prob_one = args['prob_one']\n\n state = _state_shard(args)\n normalization = np.sqrt(prob_one if result else 1 - prob_one)\n state *= (_one_projector(args, index) * result +\n (1 - _one_projector(args, index)) * (1 - result))\n state /= normalization"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef init_z_vects(self):\n shard_size = 2 ** self._num_shard_qubits\n\n a, b = np.indices((shard_size, self._num_shard_qubits))\n a >>= b\n a &= 1\n zero_one_vects = np.ascontiguousarray(a.transpose())\n zero_one_vects_handle = mem_manager.SharedMemManager.create_array(\n zero_one_vects)\n self._shared_mem_dict['zero_one_vects_handle'] = zero_one_vects_handle\n\n pm_vects = 1 - 2 * zero_one_vects\n pm_vects_handle = mem_manager.SharedMemManager.create_array(pm_vects)\n self._shared_mem_dict['pm_vects_handle'] = pm_vects_handle", "response": "Initializes the bitwise vectors which are precomputed in shared memory."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize a scratch pad equal in size to the wavefunction.", "response": "def _init_scratch(self):\n \"\"\"Initializes a scratch pad equal in size to the wavefunction.\"\"\"\n scratch = np.zeros((self._num_shards, self._shard_size),\n dtype=np.complex64)\n scratch_handle = mem_manager.SharedMemManager.create_array(\n scratch.view(dtype=np.float32))\n self._shared_mem_dict['scratch_handle'] = scratch_handle"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninitialize a the shard wavefunction and sets the initial state.", "response": "def _init_state(self, initial_state: Union[int, np.ndarray]):\n \"\"\"Initializes a the shard wavefunction and sets the initial state.\"\"\"\n state = np.reshape(\n sim.to_valid_state_vector(initial_state, self._num_qubits),\n (self._num_shards, self._shard_size))\n state_handle = mem_manager.SharedMemManager.create_array(\n state.view(dtype=np.float32))\n self._shared_mem_dict['state_handle'] = state_handle"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsimulates a set of phases on the xmon architecture.", "response": "def simulate_phases(self, phase_map: Dict[Tuple[int, ...], float]):\n \"\"\"Simulate a set of phase gates on the xmon architecture.\n\n Args:\n phase_map: A map from a tuple of indices to a value, one for each\n phase gate being simulated. If the tuple key has one index, then\n this is a Z phase gate on the index-th qubit with a rotation\n angle of pi times the value of the map. If the tuple key has two\n indices, then this is a |11> phasing gate, acting on the qubits\n at the two indices, and a rotation angle of pi times the value\n of the map.\n \"\"\"\n self._pool.map(_clear_scratch, self._shard_num_args())\n # Iterate over the map of phase data.\n for indices, half_turns in phase_map.items():\n args = self._shard_num_args(\n {'indices': indices, 'half_turns': half_turns})\n if len(indices) == 1:\n self._pool.map(_single_qubit_accumulate_into_scratch, args)\n elif len(indices) == 2:\n self._pool.map(_two_qubit_accumulate_into_scratch, args)\n # Exponentiate the phases and add them into the state.\n self._pool.map(_apply_scratch_as_phase, self._shard_num_args())"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsimulate a single qubit rotation gate about a X + b Y.", "response": "def simulate_w(self,\n index: int,\n half_turns: float,\n axis_half_turns: float):\n \"\"\"Simulate a single qubit rotation gate about a X + b Y.\n\n The gate simulated is U = exp(-i pi/2 W half_turns)\n where W = cos(pi axis_half_turns) X + sin(pi axis_half_turns) Y\n\n Args:\n index: The qubit to act on.\n half_turns: The amount of the overall rotation, see the formula\n above.\n axis_half_turns: The angle between the pauli X and Y operators,\n see the formula above.\n \"\"\"\n args = self._shard_num_args({\n 'index': index,\n 'half_turns': half_turns,\n 'axis_half_turns': axis_half_turns\n })\n if index >= self._num_shard_qubits:\n # W gate spans shards.\n self._pool.map(_clear_scratch, args)\n self._pool.map(_w_between_shards, args)\n self._pool.map(_copy_scratch_to_state, args)\n else:\n # W gate is within a shard.\n self._pool.map(_w_within_shard, args)\n\n # Normalize after every w.\n norm_squared = np.sum(self._pool.map(_norm_squared, args))\n args = self._shard_num_args({\n 'norm_squared': norm_squared\n })\n self._pool.map(_renorm, args)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsimulating a single qubit measurement in the computational basis.", "response": "def simulate_measurement(self, index: int) -> bool:\n \"\"\"Simulates a single qubit measurement in the computational basis.\n\n Args:\n index: Which qubit is measured.\n\n Returns:\n True iff the measurement result corresponds to the |1> state.\n \"\"\"\n args = self._shard_num_args({'index': index})\n prob_one = np.sum(self._pool.map(_one_prob_per_shard, args))\n result = bool(np.random.random() <= prob_one)\n\n args = self._shard_num_args({\n 'index': index,\n 'result': result,\n 'prob_one': prob_one\n })\n self._pool.map(_collapse_state, args)\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsamples from measurements in the computational basis.", "response": "def sample_measurements(\n self,\n indices: List[int],\n repetitions: int=1) -> List[List[bool]]:\n \"\"\"Samples from measurements in the computational basis.\n\n Note that this does not collapse the wave function.\n\n Args:\n indices: Which qubits are measured.\n\n Returns:\n Measurement results with True corresponding to the |1> state.\n The outer list is for repetitions, and the inner corresponds to\n measurements ordered by the input indices.\n\n Raises:\n ValueError if repetitions is less than one.\n \"\"\"\n # Stepper uses little endian while sample_state uses big endian.\n reversed_indices = [self._num_qubits - 1 - index for index in indices]\n return sim.sample_state_vector(self._current_state(), reversed_indices,\n repetitions)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndecomposes the Fermionic SWAP gate into two single - qubit gates and one iSWAP gate.", "response": "def fswap(p, q):\n \"\"\"Decompose the Fermionic SWAP gate into two single-qubit gates and\n one iSWAP gate.\n\n Args:\n p: the id of the first qubit\n q: the id of the second qubit\n \"\"\"\n\n yield cirq.ISWAP(q, p), cirq.Z(p) ** 1.5\n yield cirq.Z(q) ** 1.5"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef bogoliubov_trans(p, q, theta):\n\n # The iSWAP gate corresponds to evolve under the Hamiltonian XX+YY for\n # time -pi/4.\n expo = -4 * theta / np.pi\n\n yield cirq.X(p)\n yield cirq.S(p)\n yield cirq.ISWAP(p, q)**expo\n yield cirq.S(p) ** 1.5\n yield cirq.X(p)", "response": "A generator that yields the 2 - mode Bogoliubov transformation."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fermi_fourier_trans_2(p, q):\n\n yield cirq.Z(p)**1.5\n yield cirq.ISWAP(q, p)**0.5\n yield cirq.Z(p)**1.5", "response": "A 2 - mode fermionic Fourier transformation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fermi_fourier_trans_inverse_4(qubits):\n\n yield fswap(qubits[1], qubits[2]),\n yield fermi_fourier_trans_2(qubits[0], qubits[1])\n yield fermi_fourier_trans_2(qubits[2], qubits[3])\n yield fswap(qubits[1], qubits[2])\n yield fermi_fourier_trans_2(qubits[0], qubits[1])\n yield cirq.S(qubits[2])\n yield fermi_fourier_trans_2(qubits[2], qubits[3])\n yield fswap(qubits[1], qubits[2])", "response": "A fermionic Fourier transformation on 4 qubits."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fermi_fourier_trans_inverse_conjugate_4(qubits):\n\n yield fswap(qubits[1], qubits[2]),\n yield fermi_fourier_trans_2(qubits[0], qubits[1])\n yield fermi_fourier_trans_2(qubits[2], qubits[3])\n yield fswap(qubits[1], qubits[2])\n yield fermi_fourier_trans_2(qubits[0], qubits[1])\n yield cirq.S(qubits[2]) ** 3\n yield fermi_fourier_trans_2(qubits[2], qubits[3])\n yield fswap(qubits[1], qubits[2])", "response": "This function implements the inverse conjugate of the Fourier transform."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate the parameters for the BCS ground state i. e. the delta and bog_theta of the Bogoliubov transformation.", "response": "def bcs_parameters(n_site, n_fermi, u, t) :\n \"\"\"Generate the parameters for the BCS ground state, i.e., the\n superconducting gap and the rotational angles in the Bogoliubov\n transformation.\n\n Args:\n n_site: the number of sites in the Hubbard model\n n_fermi: the number of fermions\n u: the interaction strength\n t: the tunneling strength\n\n Returns:\n float delta, List[float] bog_theta\n \"\"\"\n\n # The wave numbers satisfy the periodic boundary condition.\n wave_num = np.linspace(0, 1, n_site, endpoint=False)\n # The hopping energy as a function of wave numbers\n hop_erg = -2 * t * np.cos(2 * np.pi * wave_num)\n # Finding the Fermi energy\n fermi_erg = hop_erg[n_fermi // 2]\n # Set the Fermi energy to zero\n hop_erg = hop_erg - fermi_erg\n\n def _bcs_gap(x):\n \"\"\"Defines the self-consistent equation for the BCS wavefunction.\n\n Args:\n x: the superconducting gap\n \"\"\"\n\n s = 0.\n for i in range(n_site):\n s += 1. / np.sqrt(hop_erg[i] ** 2 + x ** 2)\n return 1 + s * u / (2 * n_site)\n\n # Superconducting gap\n delta = scipy.optimize.bisect(_bcs_gap, 0.01, 10000. * abs(u))\n # The amplitude of the double excitation state\n bcs_v = np.sqrt(0.5 * (1 - hop_erg / np.sqrt(hop_erg ** 2 + delta ** 2)))\n # The rotational angle in the Bogoliubov transformation.\n bog_theta = np.arcsin(bcs_v)\n\n return delta, bog_theta"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the bloch vector of a qubit.", "response": "def bloch_vector_from_state_vector(state: Sequence, index: int) -> np.ndarray:\n \"\"\"Returns the bloch vector of a qubit.\n\n Calculates the bloch vector of the qubit at index\n in the wavefunction given by state, assuming state follows\n the standard Kronecker convention of numpy.kron.\n\n Args:\n state: A sequence representing a wave function in which\n the ordering mapping to qubits follows the standard Kronecker\n convention of numpy.kron.\n index: index of qubit who's bloch vector we want to find.\n follows the standard Kronecker convention of numpy.kron.\n\n Returns:\n A length 3 numpy array representing the qubit's bloch vector.\n\n Raises:\n ValueError: if the size of state is not a power of 2.\n ValueError: if the size of the state represents more than 25 qubits.\n IndexError: if index is out of range for the number of qubits\n corresponding to the state.\n \"\"\"\n rho = density_matrix_from_state_vector(state, [index])\n v = np.zeros(3, dtype=np.float32)\n v[0] = 2*np.real(rho[0][1])\n v[1] = 2*np.imag(rho[1][0])\n v[2] = np.real(rho[0][0] - rho[1][1])\n\n return v"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dirac_notation(state: Sequence, decimals: int=2) -> str:\n perm_list = [\"\".join(seq) for seq in itertools.product(\n \"01\", repeat=int(len(state)).bit_length() - 1)]\n components = []\n ket = \"|{}\u27e9\"\n for x in range(len(perm_list)):\n format_str = \"({:.\" + str(decimals) + \"g})\"\n # Python 2 rounds imaginary numbers to 0, so need to round separately.\n val = (round(state[x].real, decimals)\n + 1j * round(state[x].imag, decimals))\n if round(val.real, decimals) == 0 and round(val.imag, decimals) != 0:\n val = val.imag\n format_str = \"{:.\" + str(decimals) + \"g}j\"\n elif round(val.imag, decimals) == 0 and round(val.real, decimals) != 0:\n val = val.real\n format_str = \"{:.\" + str(decimals) + \"g}\"\n if val != 0:\n if round(state[x], decimals) == 1:\n components.append(ket.format(perm_list[x]))\n else:\n components.append((format_str + ket).format(val, perm_list[x]))\n if not components:\n return '0'\n\n return ' + '.join(components).replace(' + -', ' - ')", "response": "Returns the wavefunction as a string in Dirac notation."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nverifying the state_rep is valid and converts it to ndarray form.", "response": "def to_valid_state_vector(state_rep: Union[int, np.ndarray],\n num_qubits: int,\n dtype: Type[np.number] = np.complex64) -> np.ndarray:\n \"\"\"Verifies the state_rep is valid and converts it to ndarray form.\n\n This method is used to support passing in an integer representing a\n computational basis state or a full wave function as a representation of\n a state.\n\n Args:\n state_rep: If an int, the state returned is the state corresponding to\n a computational basis state. If an numpy array this is the full\n wave function. Both of these are validated for the given number\n of qubits, and the state must be properly normalized and of the\n appropriate dtype.\n num_qubits: The number of qubits for the state. The state_rep must be\n valid for this number of qubits.\n dtype: The numpy dtype of the state, will be used when creating the\n state for a computational basis state, or validated against if\n state_rep is a numpy array.\n\n Returns:\n A numpy ndarray corresponding to the state on the given number of\n qubits.\n\n Raises:\n ValueError if the state is not valid.\n \"\"\"\n if isinstance(state_rep, np.ndarray):\n if len(state_rep) != 2 ** num_qubits:\n raise ValueError(\n 'initial state was of size {} '\n 'but expected state for {} qubits'.format(\n len(state_rep), num_qubits))\n state = state_rep\n elif isinstance(state_rep, int):\n if state_rep < 0:\n raise ValueError('initial_state must be positive')\n elif state_rep >= 2 ** num_qubits:\n raise ValueError(\n 'initial state was {} but expected state for {} qubits'.format(\n state_rep, num_qubits))\n else:\n state = np.zeros(2 ** num_qubits, dtype=dtype)\n state[state_rep] = 1.0\n else:\n raise TypeError('initial_state was not of type int or ndarray')\n validate_normalized_state(state, num_qubits, dtype)\n return state"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_normalized_state(state: np.ndarray,\n num_qubits: int,\n dtype: Type[np.number] = np.complex64) -> None:\n \"\"\"Validates that the given state is a valid wave function.\"\"\"\n if state.size != 1 << num_qubits:\n raise ValueError(\n 'State has incorrect size. Expected {} but was {}.'.format(\n 1 << num_qubits, state.size))\n if state.dtype != dtype:\n raise ValueError(\n 'State has invalid dtype. Expected {} but was {}'.format(\n dtype, state.dtype))\n norm = np.sum(np.abs(state) ** 2)\n if not np.isclose(norm, 1):\n raise ValueError('State is not normalized instead had norm %s' % norm)", "response": "Validates that the given state is a valid wave function."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsamples repeatedly from measurements in the computational basis.", "response": "def sample_state_vector(state: np.ndarray,\n indices: List[int],\n repetitions: int=1) -> np.ndarray:\n \"\"\"Samples repeatedly from measurements in the computational basis.\n\n Note that this does not modify the passed in state.\n\n Args:\n state: The multi-qubit wavefunction to be sampled. This is an array of\n 2 to the power of the number of qubit complex numbers, and so\n state must be of size ``2**integer``. The state can be a vector of\n size ``2**integer`` or a tensor of shape ``(2, 2, ..., 2)``.\n indices: Which qubits are measured. The state is assumed to be supplied\n in big endian order. That is the xth index of v, when expressed as\n a bitstring, has its largest values in the 0th index.\n repetitions: The number of times to sample the state.\n\n Returns:\n Measurement results with True corresponding to the ``|1\u27e9`` state.\n The outer list is for repetitions, and the inner corresponds to\n measurements ordered by the supplied qubits. These lists\n are wrapped as an numpy ndarray.\n\n Raises:\n ValueError: ``repetitions`` is less than one or size of ``state`` is not\n a power of 2.\n IndexError: An index from ``indices`` is out of range, given the number\n of qubits corresponding to the state.\n \"\"\"\n if repetitions < 0:\n raise ValueError('Number of repetitions cannot be negative. Was {}'\n .format(repetitions))\n num_qubits = _validate_num_qubits(state)\n _validate_indices(num_qubits, indices)\n\n if repetitions == 0 or len(indices) == 0:\n return np.zeros(shape=(repetitions, len(indices)))\n\n # Calculate the measurement probabilities.\n probs = _probs(state, indices, num_qubits)\n\n # We now have the probability vector, correctly ordered, so sample over\n # it. Note that we us ints here, since numpy's choice does not allow for\n # choosing from a list of tuples or list of lists.\n result = np.random.choice(len(probs), size=repetitions, p=probs)\n # Convert to bools and rearrange to match repetition being the outer list.\n return np.transpose([(1 & (result >> i)).astype(np.bool) for i in\n range(len(indices))])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the probabilities for a measurement on the given indices.", "response": "def _probs(state: np.ndarray, indices: List[int],\n num_qubits: int) -> List[float]:\n \"\"\"Returns the probabilities for a measurement on the given indices.\"\"\"\n # Tensor of squared amplitudes, shaped a rank [2, 2, .., 2] tensor.\n tensor = np.reshape(state, [2] * num_qubits)\n\n # Calculate the probabilities for measuring the particular results.\n probs = [\n np.linalg.norm(\n tensor[linalg.slice_for_qubits_equal_to(indices, b)]) ** 2\n for b in range(2 ** len(indices))]\n\n # To deal with rounding issues, ensure that the probabilities sum to 1.\n probs /= sum(probs) # type: ignore\n return probs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates that state s size is a power of 2 returning number of qubits.", "response": "def _validate_num_qubits(state: np.ndarray) -> int:\n \"\"\"Validates that state's size is a power of 2, returning number of qubits.\n \"\"\"\n size = state.size\n if size & (size - 1):\n raise ValueError('state.size ({}) is not a power of two.'.format(size))\n return size.bit_length() - 1"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate that the indices have values within range of num_qubits.", "response": "def _validate_indices(num_qubits: int, indices: List[int]) -> None:\n \"\"\"Validates that the indices have values within range of num_qubits.\"\"\"\n if any(index < 0 for index in indices):\n raise IndexError('Negative index in indices: {}'.format(indices))\n if any(index >= num_qubits for index in indices):\n raise IndexError('Out of range indices, must be less than number of '\n 'qubits but was {}'.format(indices))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef density_matrix_of(self, qubits: List[ops.Qid] = None) -> np.ndarray:\n return density_matrix_from_state_vector(\n self.state_vector(),\n [self.qubit_map[q] for q in qubits] if qubits is not None else None\n )", "response": "Returns the density matrix of the state."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the bloch vector of a given qubit in the state given by self. state_vector.", "response": "def bloch_vector_of(self, qubit: ops.Qid) -> np.ndarray:\n \"\"\"Returns the bloch vector of a qubit in the state.\n\n Calculates the bloch vector of the given qubit\n in the state given by self.state_vector(), given that\n self.state_vector() follows the standard Kronecker convention of\n numpy.kron.\n\n Args:\n qubit: qubit who's bloch vector we want to find.\n\n Returns:\n A length 3 numpy array representing the qubit's bloch vector.\n\n Raises:\n ValueError: if the size of the state represents more than 25 qubits.\n IndexError: if index is out of range for the number of qubits\n corresponding to the state.\n \"\"\"\n return bloch_vector_from_state_vector(self.state_vector(),\n self.qubit_map[qubit])"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_acquaintance_size(obj: Union[circuits.Circuit, ops.Operation]) -> int:\n if isinstance(obj, circuits.Circuit):\n if not is_acquaintance_strategy(obj):\n raise TypeError('not is_acquaintance_strategy(circuit)')\n return max(tuple(get_acquaintance_size(op)\n for op in obj.all_operations()) or (0,))\n if not isinstance(obj, ops.Operation):\n raise TypeError('not isinstance(obj, (Circuit, Operation))')\n if not isinstance(obj, ops.GateOperation):\n return 0\n if isinstance(obj.gate, AcquaintanceOpportunityGate):\n return len(obj.qubits)\n if isinstance(obj.gate, BipartiteSwapNetworkGate):\n return 2\n if isinstance(obj.gate, ShiftSwapNetworkGate):\n return obj.gate.acquaintance_size()\n if isinstance(obj.gate, SwapNetworkGate):\n if obj.gate.acquaintance_size is None:\n return sum(sorted(obj.gate.part_lens)[-2:])\n if (obj.gate.acquaintance_size - 1) in obj.gate.part_lens:\n return obj.gate.acquaintance_size\n sizer = getattr(obj.gate, '_acquaintance_size_', None)\n return 0 if sizer is None else sizer(len(obj.qubits))", "response": "Returns the number of qubits to be acquainted with each other."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_or_search(self) -> List[GridQubit]:\n if not self._sequence:\n self._sequence = self._find_sequence()\n return self._sequence", "response": "Starts the search or gives previously calculated sequence."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _find_sequence(self) -> List[GridQubit]:\n # Run the first pass and drop starting qubit from the found sequence.\n tail = self._sequence_search(self._start, [])\n tail.pop(0)\n\n # Run the second pass and keep the starting qubit.\n head = self._sequence_search(self._start, tail)\n head.reverse()\n\n return self._expand_sequence(head + tail)", "response": "Searches for a sequence starting at a given qubit."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _sequence_search(self, start: GridQubit,\n current: List[GridQubit]) -> List[GridQubit]:\n \"\"\"Search for the continuous linear sequence from the given qubit.\n\n This method is called twice for the same starting qubit, so that\n sequences that begin and end on this qubit are searched for.\n\n Args:\n start: The first qubit, where search should be trigerred from.\n current: Previously found linear sequence, which qubits are\n forbidden to use during the search.\n\n Returns:\n Continuous linear sequence that begins with the starting qubit and\n does not contain any qubits from the current list.\n \"\"\"\n used = set(current)\n seq = []\n n = start # type: Optional[GridQubit]\n while n is not None:\n # Append qubit n to the sequence and mark it is as visited.\n seq.append(n)\n used.add(n)\n # Advance search to the next qubit.\n n = self._choose_next_qubit(n, used)\n return seq", "response": "Search for the continuous linear sequence from the given qubit."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _find_path_between(self, p: GridQubit, q: GridQubit,\n used: Set[GridQubit]) -> Optional[List[GridQubit]]:\n \"\"\"Searches for continuous sequence between two qubits.\n\n This method runs two BFS algorithms in parallel (alternating variable s\n in each iteration); the first one starting from qubit p, and the second\n one starting from qubit q. If at some point a qubit reachable from p is\n found to be on the set of qubits already reached from q (or vice versa),\n the search is stopped and new path returned.\n\n Args:\n p: The first qubit, start of the sequence.\n q: The second qubit, end of the sequence.\n used: Set of forbidden qubits which cannot appear on the sequence.\n\n Returns:\n Continues sequence of qubits with new path between p and q, or None\n if no path was found.\n \"\"\"\n\n def assemble_path(n: GridQubit, parent: Dict[GridQubit, GridQubit]):\n path = [n]\n while n in parent:\n n = parent[n]\n path.append(n)\n return path\n\n other = {p: q, q: p}\n parents = {p: dict(), q: dict()} \\\n # type: Dict[GridQubit, Dict[GridQubit, GridQubit]]\n visited = {p: set(), q: set()} # type: Dict[GridQubit, Set[GridQubit]]\n\n queue = collections.deque([(p, p), (q, q)])\n\n # Run two BFSs simultaneously.\n while queue:\n n, s = queue.popleft()\n for n_adj in self._c_adj[n]:\n if n_adj in visited[other[s]]:\n # Connection has been found, construct the path and return.\n path_s = assemble_path(n, parents[s])[-2::-1]\n path_other = assemble_path(n_adj, parents[other[s]])[:-1]\n path = path_s + path_other\n if s == q:\n path.reverse()\n return path\n elif n_adj not in used and n_adj not in visited[s]:\n # Append n_adj to the end of queue of qubit s.\n queue.append((n_adj, s))\n visited[s].add(n_adj)\n parents[s][n_adj] = n\n\n return None", "response": "Searches for continuous sequence between two qubits."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _collect_unused(self, start: GridQubit,\n used: Set[GridQubit]) -> Set[GridQubit]:\n \"\"\"Lists all the qubits that are reachable from given qubit.\n\n Args:\n start: The first qubit for which connectivity should be calculated.\n Might be a member of used set.\n used: Already used qubits, which cannot be used during the\n collection.\n\n Returns:\n Set of qubits that are reachable from starting qubit without\n traversing any of the used qubits.\n \"\"\"\n\n def collect(n: GridQubit, visited: Set[GridQubit]):\n visited.add(n)\n for m in self._c_adj[n]:\n if m not in used and m not in visited:\n collect(m, visited)\n\n visited = set() # type: Set[GridQubit]\n collect(start, visited)\n return visited", "response": "Returns a list of all the qubits that are reachable from start and which are not used during the collection."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nruns line sequence search.", "response": "def place_line(self,\n device: 'cirq.google.XmonDevice',\n length: int) -> GridQubitLineTuple:\n \"\"\"Runs line sequence search.\n\n Args:\n device: Chip description.\n length: Required line length.\n\n Returns:\n Linear sequences found on the chip.\n\n Raises:\n ValueError: If search algorithm passed on initialization is not\n recognized.\n \"\"\"\n\n if not device.qubits:\n return GridQubitLineTuple()\n\n start = min(device.qubits) # type: GridQubit\n sequences = [] # type: List[LineSequence]\n greedy_search = {\n 'minimal_connectivity': [\n _PickFewestNeighbors(device, start),\n ],\n 'largest_area': [\n _PickLargestArea(device, start),\n ],\n 'best': [\n _PickFewestNeighbors(device, start),\n _PickLargestArea(device, start),\n ]\n } # type: Dict[str, List[GreedySequenceSearch]]\n\n algos = greedy_search.get(self.algorithm)\n if algos is None:\n raise ValueError(\n \"Unknown greedy search algorithm %s\" % self.algorithm)\n\n for algorithm in algos:\n sequences.append(algorithm.get_or_search())\n\n return GridQubitLineTuple.best_of(sequences, length)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a maximum on the trace distance between this effect s input and output.", "response": "def trace_distance_bound(val: Any) -> float:\n \"\"\"Returns a maximum on the trace distance between this effect's input\n and output. This method makes use of the effect's `_trace_distance_bound_`\n method to determine the maximum bound on the trace difference between\n before and after the effect.\n\n Args:\n val: The effect of which the bound should be calculated\n\n Returns:\n If `val` has a _trace_distance_bound_ method and its result is not\n NotImplemented, that result is returned. Otherwise, 1 is returned.\n Result is capped at a maximum of 1, even if the underlying function\n produces a result greater than 1.\n\n \"\"\"\n getter = getattr(val, '_trace_distance_bound_', None)\n result = NotImplemented if getter is None else getter()\n\n if result is not NotImplemented and result < 1.0:\n return result\n return 1.0"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndetermines if the moment has operations touching the given qubits.", "response": "def operates_on(self, qubits: Iterable[raw_types.Qid]) -> bool:\n \"\"\"Determines if the moment has operations touching the given qubits.\n\n Args:\n qubits: The qubits that may or may not be touched by operations.\n\n Returns:\n Whether this moment has operations involving the qubits.\n \"\"\"\n return any(q in qubits for q in self.qubits)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef without_operations_touching(self, qubits: Iterable[raw_types.Qid]):\n qubits = frozenset(qubits)\n if not self.operates_on(qubits):\n return self\n return Moment(\n operation for operation in self.operations\n if qubits.isdisjoint(frozenset(operation.qubits)))", "response": "Returns an equal moment but without operations on the given qubits."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _approx_eq_(self, other: Any, atol: Union[int, float]) -> bool:\n if not isinstance(other, type(self)):\n return NotImplemented\n return approx_eq(self.operations, other.operations, atol=atol)", "response": "See cirq. protocols. SupportsApproximateEquality."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns true only if the operations have qubits in common.", "response": "def _disjoint_qubits(op1: ops.Operation, op2: ops.Operation) -> bool:\n \"\"\"Returns true only if the operations have qubits in common.\"\"\"\n return not set(op1.qubits) & set(op2.qubits)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndescribes how to change operations near the given location.", "response": "def optimization_at(self,\n circuit: Circuit,\n index: int,\n op: ops.Operation\n ) -> Optional[PointOptimizationSummary]:\n \"\"\"Describes how to change operations near the given location.\n\n For example, this method could realize that the given operation is an\n X gate and that in the very next moment there is a Z gate. It would\n indicate that they should be combined into a Y gate by returning\n PointOptimizationSummary(clear_span=2,\n clear_qubits=op.qubits,\n new_operations=cirq.Y(op.qubits[0]))\n\n Args:\n circuit: The circuit to improve.\n index: The index of the moment with the operation to focus on.\n op: The operation to focus improvements upon.\n\n Returns:\n A description of the optimization to perform, or else None if no\n change should be made.\n \"\"\""} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a schedule into an iterable of proto dictionaries.", "response": "def schedule_to_proto_dicts(schedule: Schedule) -> Iterable[Dict]:\n \"\"\"Convert a schedule into an iterable of proto dictionaries.\n\n Args:\n schedule: The schedule to convert to a proto dict. Must contain only\n gates that can be cast to xmon gates.\n\n Yields:\n A proto dictionary corresponding to an Operation proto.\n \"\"\"\n last_time_picos = None # type: Optional[int]\n for so in schedule.scheduled_operations:\n op = gate_to_proto_dict(\n cast(ops.GateOperation, so.operation).gate,\n so.operation.qubits)\n time_picos = so.time.raw_picos()\n if last_time_picos is None:\n op['incremental_delay_picoseconds'] = time_picos\n else:\n op['incremental_delay_picoseconds'] = time_picos - last_time_picos\n last_time_picos = time_picos\n yield op"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef schedule_from_proto_dicts(\n device: 'xmon_device.XmonDevice',\n ops: Iterable[Dict],\n) -> Schedule:\n \"\"\"Convert proto dictionaries into a Schedule for the given device.\"\"\"\n scheduled_ops = []\n last_time_picos = 0\n for op in ops:\n delay_picos = 0\n if 'incremental_delay_picoseconds' in op:\n delay_picos = op['incremental_delay_picoseconds']\n time_picos = last_time_picos + delay_picos\n last_time_picos = time_picos\n xmon_op = xmon_op_from_proto_dict(op)\n scheduled_ops.append(ScheduledOperation.op_at_on(\n operation=xmon_op,\n time=Timestamp(picos=time_picos),\n device=device,\n ))\n return Schedule(device, scheduled_ops)", "response": "Convert a list of proto dictionaries into a Schedule object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pack_results(measurements: Sequence[Tuple[str, np.ndarray]]) -> bytes:\n if not measurements:\n return b''\n\n shapes = [(key, np.shape(data)) for key, data in measurements]\n if not all(len(shape) == 2 for _, shape in shapes):\n raise ValueError(\"Expected 2-D data: shapes={}\".format(shapes))\n\n reps = shapes[0][1][0]\n if not all(shape[0] == reps for _, shape in shapes):\n raise ValueError(\n \"Expected same reps for all keys: shapes={}\".format(shapes))\n\n bits = np.hstack([np.asarray(data, dtype=bool) for _, data in measurements])\n bits = bits.reshape(-1)\n\n # Pad length to multiple of 8 if needed.\n remainder = len(bits) % 8\n if remainder:\n bits = np.pad(bits, (0, 8 - remainder), 'constant')\n\n # Pack in little-endian bit order.\n bits = bits.reshape((-1, 8))[:, ::-1]\n byte_arr = np.packbits(bits, axis=1).reshape(-1)\n\n return byte_arr.tobytes()", "response": "Pack measurement results into a byte string."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nunpacks data from a bitstring into individual measurement results.", "response": "def unpack_results(\n data: bytes,\n repetitions: int,\n key_sizes: Sequence[Tuple[str, int]]\n) -> Dict[str, np.ndarray]:\n \"\"\"Unpack data from a bitstring into individual measurement results.\n\n Args:\n data: Packed measurement results, in the form ...\n where each repetition is .....\n with bits packed in little-endian order in each byte.\n repetitions: number of repetitions.\n key_sizes: Keys and sizes of the measurements in the data.\n\n Returns:\n Dict mapping measurement key to a 2D array of boolean results. Each\n array has shape (repetitions, size) with size for that measurement.\n \"\"\"\n bits_per_rep = sum(size for _, size in key_sizes)\n total_bits = repetitions * bits_per_rep\n\n byte_arr = np.frombuffer(data, dtype='uint8').reshape((len(data), 1))\n bits = np.unpackbits(byte_arr, axis=1)[:, ::-1].reshape(-1).astype(bool)\n bits = bits[:total_bits].reshape((repetitions, bits_per_rep))\n\n results = {}\n ofs = 0\n for key, size in key_sizes:\n results[key] = bits[:, ofs:ofs + size]\n ofs += size\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_native_xmon_op(op: ops.Operation) -> bool:\n return (isinstance(op, ops.GateOperation) and\n is_native_xmon_gate(op.gate))", "response": "Checks if the operation is a native xmon gate."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if a gate is a native xmon gate.", "response": "def is_native_xmon_gate(gate: ops.Gate) -> bool:\n \"\"\"Check if a gate is a native xmon gate.\n\n Args:\n gate: Input gate.\n\n Returns:\n True if the gate is native to the xmon, false otherwise.\n \"\"\"\n return isinstance(gate, (ops.CZPowGate,\n ops.MeasurementGate,\n ops.PhasedXPowGate,\n ops.XPowGate,\n ops.YPowGate,\n ops.ZPowGate))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef xmon_op_from_proto_dict(proto_dict: Dict) -> ops.Operation:\n\n def raise_missing_fields(gate_name: str):\n raise ValueError(\n '{} missing required fields: {}'.format(gate_name, proto_dict))\n param = _parameterized_value_from_proto_dict\n qubit = devices.GridQubit.from_proto_dict\n if 'exp_w' in proto_dict:\n exp_w = proto_dict['exp_w']\n if ('half_turns' not in exp_w or 'axis_half_turns' not in exp_w\n or 'target' not in exp_w):\n raise_missing_fields('ExpW')\n return ops.PhasedXPowGate(\n exponent=param(exp_w['half_turns']),\n phase_exponent=param(exp_w['axis_half_turns']),\n ).on(qubit(exp_w['target']))\n elif 'exp_z' in proto_dict:\n exp_z = proto_dict['exp_z']\n if 'half_turns' not in exp_z or 'target' not in exp_z:\n raise_missing_fields('ExpZ')\n return ops.Z(qubit(exp_z['target']))**param(exp_z['half_turns'])\n elif 'exp_11' in proto_dict:\n exp_11 = proto_dict['exp_11']\n if ('half_turns' not in exp_11 or 'target1' not in exp_11\n or 'target2' not in exp_11):\n raise_missing_fields('Exp11')\n return ops.CZ(qubit(exp_11['target1']),\n qubit(exp_11['target2']))**param(exp_11['half_turns'])\n elif 'measurement' in proto_dict:\n meas = proto_dict['measurement']\n invert_mask = cast(Tuple[Any, ...], ())\n if 'invert_mask' in meas:\n invert_mask = tuple(json.loads(x) for x in meas['invert_mask'])\n if 'key' not in meas or 'targets' not in meas:\n raise_missing_fields('Measurement')\n return ops.MeasurementGate(\n num_qubits=len(meas['targets']),\n key=meas['key'],\n invert_mask=invert_mask\n ).on(*[qubit(q) for q in meas['targets']])\n else:\n raise ValueError('invalid operation: {}'.format(proto_dict))", "response": "Convert the dictionary representation of a single operation into an XMon operation."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nremove terms with coefficients of absolute value atol or less.", "response": "def clean(self: 'TSelf', *, atol: float=1e-9) -> 'TSelf':\n \"\"\"Remove terms with coefficients of absolute value atol or less.\"\"\"\n negligible = [v for v, c in self._terms.items() if abs(c) <= atol]\n for v in negligible:\n del self._terms[v]\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking whether two linear combinations are approximately equal.", "response": "def _approx_eq_(self, other: Any, atol: float) -> bool:\n \"\"\"Checks whether two linear combinations are approximately equal.\"\"\"\n if not isinstance(other, LinearDict):\n return NotImplemented\n\n all_vs = set(self.keys()) | set(other.keys())\n return all(abs(self[v] - other[v]) < atol for v in all_vs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef enqueue(self, priority: int, item: TItem) -> bool:\n if self._drop_set is not None:\n if (priority, item) in self._drop_set:\n return False\n self._drop_set.add((priority, item))\n\n # First enqueue initializes self._offset.\n if not self._buckets:\n self._buckets.append([item])\n self._offset = priority\n self._len = 1\n return True\n\n # Where is the bucket this item is supposed to go into?\n i = priority - self._offset\n\n # Extend bucket list backwards if needed.\n if i < 0:\n self._buckets[:0] = [[] for _ in range(-i)]\n self._offset = priority\n i = 0\n\n # Extend bucket list forwards if needed.\n while i >= len(self._buckets):\n self._buckets.append([])\n\n # Finish by adding item to the intended bucket's list.\n self._buckets[i].append(item)\n self._len += 1\n return True", "response": "Enqueue an item into the priority queue."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove and returns an item from the priority queue.", "response": "def dequeue(self) -> Tuple[int, TItem]:\n \"\"\"Removes and returns an item from the priority queue.\n\n Returns:\n A tuple whose first element is the priority of the dequeued item\n and whose second element is the dequeued item.\n\n Raises:\n ValueError:\n The queue is empty.\n \"\"\"\n if self._len == 0:\n raise ValueError('BucketPriorityQueue is empty.')\n\n # Drop empty buckets at the front of the queue.\n while self._buckets and not self._buckets[0]:\n self._buckets.pop(0)\n self._offset += 1\n\n # Pull item out of the front bucket.\n item = self._buckets[0].pop(0)\n priority = self._offset\n self._len -= 1\n if self._drop_set is not None:\n self._drop_set.remove((priority, item))\n\n # Note: do not eagerly clear out empty buckets after pulling the item!\n # Doing so increases the worst case complexity of \"monotonic\" use from\n # O(N+P) to O(N*P).\n\n return priority, item"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun the supplied Circuit or Schedule via Quantum Engine.", "response": "def run(\n self,\n *, # Force keyword args.\n program: Union[circuits.Circuit, Schedule],\n job_config: Optional[JobConfig] = None,\n param_resolver: ParamResolver = ParamResolver({}),\n repetitions: int = 1,\n priority: int = 50,\n processor_ids: Sequence[str] = ('xmonsim',)) -> TrialResult:\n \"\"\"Runs the supplied Circuit or Schedule via Quantum Engine.\n\n Args:\n program: The Circuit or Schedule to execute. If a circuit is\n provided, a moment by moment schedule will be used.\n job_config: Configures the names of programs and jobs.\n param_resolver: Parameters to run with the program.\n repetitions: The number of repetitions to simulate.\n priority: The priority to run at, 0-100.\n processor_ids: The engine processors to run against.\n\n Returns:\n A single TrialResult for this run.\n \"\"\"\n return list(\n self.run_sweep(program=program,\n job_config=job_config,\n params=[param_resolver],\n repetitions=repetitions,\n priority=priority,\n processor_ids=processor_ids))[0]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run_sweep(\n self,\n *, # Force keyword args.\n program: Union[circuits.Circuit, Schedule],\n job_config: Optional[JobConfig] = None,\n params: Sweepable = None,\n repetitions: int = 1,\n priority: int = 500,\n processor_ids: Sequence[str] = ('xmonsim',)) -> 'EngineJob':\n \"\"\"Runs the supplied Circuit or Schedule via Quantum Engine.\n\n In contrast to run, this runs across multiple parameter sweeps, and\n does not block until a result is returned.\n\n Args:\n program: The Circuit or Schedule to execute. If a circuit is\n provided, a moment by moment schedule will be used.\n job_config: Configures the names of programs and jobs.\n params: Parameters to run with the program.\n repetitions: The number of circuit repetitions to run.\n priority: The priority to run at, 0-100.\n processor_ids: The engine processors to run against.\n\n Returns:\n An EngineJob. If this is iterated over it returns a list of\n TrialResults, one for each parameter sweep.\n \"\"\"\n\n job_config = self.implied_job_config(job_config)\n schedule = self.program_as_schedule(program)\n\n # Check program to run and program parameters.\n if not 0 <= priority < 1000:\n raise ValueError('priority must be between 0 and 1000')\n\n schedule.device.validate_schedule(schedule)\n\n # Create program.\n sweeps = _sweepable_to_sweeps(params or ParamResolver({}))\n program_dict = {} # type: Dict[str, Any]\n\n program_dict['parameter_sweeps'] = [\n sweep_to_proto_dict(sweep, repetitions) for\n sweep in sweeps]\n program_dict['operations'] = [op for op in\n schedule_to_proto_dicts(schedule)]\n code = {\n '@type': 'type.googleapis.com/cirq.api.google.v1.Program'}\n code.update(program_dict)\n request = {\n 'name': 'projects/%s/programs/%s' % (job_config.project_id,\n job_config.program_id,),\n 'gcs_code_location': {'uri': job_config.gcs_program},\n 'code': code,\n }\n response = self.service.projects().programs().create(\n parent='projects/%s' % job_config.project_id,\n body=request).execute()\n\n # Create job.\n request = {\n 'name': '%s/jobs/%s' % (response['name'], job_config.job_id),\n 'output_config': {\n 'gcs_results_location': {\n 'uri': job_config.gcs_results\n }\n },\n 'scheduling_config': {\n 'priority': priority,\n 'processor_selector': {\n 'processor_names': [\n 'projects/%s/processors/%s' %\n (job_config.project_id, processor_id)\n for processor_id in processor_ids\n ]\n }\n },\n }\n response = self.service.projects().programs().jobs().create(\n parent=response['name'], body=request).execute()\n\n return EngineJob(job_config, response, self)", "response": "Runs a single parameter sweep."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the previously created quantum program.", "response": "def get_program(self, program_resource_name: str) -> Dict:\n \"\"\"Returns the previously created quantum program.\n\n Params:\n program_resource_name: A string of the form\n `projects/project_id/programs/program_id`.\n\n Returns:\n A dictionary containing the metadata and the program.\n \"\"\"\n return self.service.projects().programs().get(\n name=program_resource_name).execute()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the metadata about a previously created job.", "response": "def get_job(self, job_resource_name: str) -> Dict:\n \"\"\"Returns metadata about a previously created job.\n\n See get_job_result if you want the results of the job and not just\n metadata about the job.\n\n Params:\n job_resource_name: A string of the form\n `projects/project_id/programs/program_id/jobs/job_id`.\n\n Returns:\n A dictionary containing the metadata.\n \"\"\"\n return self.service.projects().programs().jobs().get(\n name=job_resource_name).execute()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_job_results(self, job_resource_name: str) -> List[TrialResult]:\n response = self.service.projects().programs().jobs().getResult(\n parent=job_resource_name).execute()\n trial_results = []\n for sweep_result in response['result']['sweepResults']:\n sweep_repetitions = sweep_result['repetitions']\n key_sizes = [(m['key'], len(m['qubits']))\n for m in sweep_result['measurementKeys']]\n for result in sweep_result['parameterizedResults']:\n data = base64.standard_b64decode(result['measurementResults'])\n measurements = unpack_results(data, sweep_repetitions,\n key_sizes)\n\n trial_results.append(TrialResult(\n params=ParamResolver(\n result.get('params', {}).get('assignments', {})),\n repetitions=sweep_repetitions,\n measurements=measurements))\n return trial_results", "response": "Returns the actual results of a completed job."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncancels the given job.", "response": "def cancel_job(self, job_resource_name: str):\n \"\"\"Cancels the given job.\n\n See also the cancel method on EngineJob.\n\n Params:\n job_resource_name: A string of the form\n `projects/project_id/programs/program_id/jobs/job_id`.\n \"\"\"\n self.service.projects().programs().jobs().cancel(\n name=job_resource_name, body={}).execute()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef results(self) -> List[TrialResult]:\n if not self._results:\n job = self._update_job()\n for _ in range(1000):\n if job['executionStatus']['state'] in TERMINAL_STATES:\n break\n time.sleep(0.5)\n job = self._update_job()\n if job['executionStatus']['state'] != 'SUCCESS':\n raise RuntimeError(\n 'Job %s did not succeed. It is in state %s.' % (\n job['name'], job['executionStatus']['state']))\n self._results = self._engine.get_job_results(\n self.job_resource_name)\n return self._results", "response": "Returns the job results blocking until the job is complete."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef flatten_op_tree(root: OP_TREE,\n preserve_moments: bool = False\n ) -> Iterable[Union[Operation, Moment]]:\n \"\"\"Performs an in-order iteration of the operations (leaves) in an OP_TREE.\n\n Args:\n root: The operation or tree of operations to iterate.\n preserve_moments: Whether to yield Moments intact instead of\n flattening them\n\n Yields:\n Operations from the tree.\n\n Raises:\n TypeError: root isn't a valid OP_TREE.\n \"\"\"\n if (isinstance(root, Operation)\n or preserve_moments and isinstance(root, Moment)):\n yield root\n return\n\n if isinstance(root, collections.Iterable):\n for subtree in root:\n for item in flatten_op_tree(subtree, preserve_moments):\n yield item\n return\n\n raise TypeError('Not a collections.Iterable or an Operation: {} {}'.format(\n type(root), root))", "response": "Returns an in - order iteration of the operations in an OP_TREE."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transform_op_tree(\n root: OP_TREE,\n op_transformation: Callable[[Operation], OP_TREE]=lambda e: e,\n iter_transformation: Callable[[Iterable[OP_TREE]], OP_TREE]=lambda e: e,\n preserve_moments: bool = False\n) -> OP_TREE:\n \"\"\"Maps transformation functions onto the nodes of an OP_TREE.\n\n Args:\n root: The operation or tree of operations to transform.\n op_transformation: How to transform the operations (i.e. leaves).\n iter_transformation: How to transform the iterables (i.e. internal\n nodes).\n preserve_moments: Whether to leave Moments alone. If True, the\n transformation functions will not be applied to Moments or the\n operations within them.\n\n Returns:\n A transformed operation tree.\n\n Raises:\n TypeError: root isn't a valid OP_TREE.\n \"\"\"\n if isinstance(root, Operation):\n return op_transformation(root)\n\n if preserve_moments and isinstance(root, Moment):\n return root\n\n if isinstance(root, collections.Iterable):\n return iter_transformation(\n transform_op_tree(subtree,\n op_transformation,\n iter_transformation,\n preserve_moments)\n for subtree in root)\n\n raise TypeError(\n 'Not a collections.Iterable or an Operation: {}'.format(root))", "response": "Transforms an OP_TREE into a single element - wise tree of internalCOOKIE entries."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef circuit_to_quirk_url(circuit: circuits.Circuit,\n prefer_unknown_gate_to_failure: bool=False,\n escape_url=True) -> str:\n \"\"\"Returns a Quirk URL for the given circuit.\n\n Args:\n circuit: The circuit to open in Quirk.\n prefer_unknown_gate_to_failure: If not set, gates that fail to convert\n will cause this function to raise an error. If set, a URL\n containing bad gates will be generated. (Quirk will open the URL,\n and replace the bad gates with parse errors, but still get the rest\n of the circuit.)\n escape_url: If set, the generated URL will have special characters such\n as quotes escaped using %. This makes it possible to paste the URL\n into forums and the command line and etc and have it properly\n parse. If not set, the generated URL will be more compact and human\n readable (and can still be pasted directly into a browser's address\n bar).\n\n Returns:\n\n \"\"\"\n circuit = circuit.copy()\n linearize_circuit_qubits(circuit)\n\n cols = [] # Type: List[List[Any]]\n for moment in circuit:\n can_merges = []\n for op in moment.operations:\n for col, can_merge in _to_quirk_cols(\n op,\n prefer_unknown_gate_to_failure):\n if can_merge:\n can_merges.append(col)\n else:\n cols.append(col)\n if can_merges:\n merged_col = [1] * max(len(e) for e in can_merges)\n for col in can_merges:\n for i in range(len(col)):\n if col[i] != 1:\n merged_col[i] = col[i]\n cols.append(merged_col)\n\n circuit_json = json.JSONEncoder(ensure_ascii=False,\n separators=(',', ':'),\n sort_keys=True).encode({'cols': cols})\n if escape_url:\n suffix = urllib.parse.quote(circuit_json)\n else:\n suffix = circuit_json\n return 'http://algassert.com/quirk#circuit={}'.format(suffix)", "response": "Converts a circuit into a Quirk URL."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndetermines if two qubits are adjacent qubits.", "response": "def is_adjacent(self, other: ops.Qid) -> bool:\n \"\"\"Determines if two qubits are adjacent qubits.\"\"\"\n return (isinstance(other, GridQubit) and\n abs(self.row - other.row) + abs(self.col - other.col) == 1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a GridQubit object from a protobuf dictionary.", "response": "def from_proto_dict(proto_dict: Dict) -> 'GridQubit':\n \"\"\"Proto dict must have 'row' and 'col' keys.\"\"\"\n if 'row' not in proto_dict or 'col' not in proto_dict:\n raise ValueError(\n 'Proto dict does not contain row or col: {}'.format(proto_dict))\n return GridQubit(row=proto_dict['row'], col=proto_dict['col'])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the logical operations to apply to qubits.", "response": "def get_operations(self,\n indices: Sequence[LogicalIndex],\n qubits: Sequence[ops.Qid]\n ) -> ops.OP_TREE:\n \"\"\"Gets the logical operations to apply to qubits.\"\"\""} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge_single_qubit_gates_into_phased_x_z(\n circuit: circuits.Circuit,\n atol: float = 1e-8) -> None:\n \"\"\"Canonicalizes runs of single-qubit rotations in a circuit.\n\n Specifically, any run of non-parameterized circuits will be replaced by an\n optional PhasedX operation followed by an optional Z operation.\n\n Args:\n circuit: The circuit to rewrite. This value is mutated in-place.\n atol: Absolute tolerance to angle error. Larger values allow more\n negligible gates to be dropped, smaller values increase accuracy.\n \"\"\"\n\n def synth(qubit: ops.Qid, matrix: np.ndarray) -> List[ops.Operation]:\n out_gates = decompositions.single_qubit_matrix_to_phased_x_z(\n matrix, atol)\n return [gate(qubit) for gate in out_gates]\n\n MergeSingleQubitGates(synthesizer=synth).optimize_circuit(circuit)", "response": "This function merges single - qubit gates into a single - qubit molecular x - z - gate."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrap the given string with terminal color codes.", "response": "def highlight(text: str, color_code: int, bold: bool=False) -> str:\n \"\"\"Wraps the given string with terminal color codes.\n\n Args:\n text: The content to highlight.\n color_code: The color to highlight with, e.g. 'shelltools.RED'.\n bold: Whether to bold the content in addition to coloring.\n\n Returns:\n The highlighted string.\n \"\"\"\n return '{}\\033[{}m{}\\033[0m'.format(\n '\\033[1m' if bold else '',\n color_code,\n text,)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprint and captures output from the given asynchronous iterable.", "response": "async def _async_forward(async_chunks: collections.AsyncIterable,\n out: Optional[Union[TeeCapture, IO[str]]]\n ) -> Optional[str]:\n \"\"\"Prints/captures output from the given asynchronous iterable.\n\n Args:\n async_chunks: An asynchronous source of bytes or str.\n out: Where to put the chunks.\n\n Returns:\n The complete captured output, or else None if the out argument wasn't a\n TeeCapture instance.\n \"\"\"\n capture = isinstance(out, TeeCapture)\n out_pipe = out.out_pipe if isinstance(out, TeeCapture) else out\n\n chunks = [] if capture else None # type: Optional[List[str]]\n async for chunk in async_chunks:\n if not isinstance(chunk, str):\n chunk = chunk.decode()\n if out_pipe:\n print(chunk, file=out_pipe, end='')\n if chunks is not None:\n chunks.append(chunk)\n\n return ''.join(chunks) if chunks is not None else None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\nasync def _async_wait_for_process(\n future_process: Any,\n out: Optional[Union[TeeCapture, IO[str]]] = sys.stdout,\n err: Optional[Union[TeeCapture, IO[str]]] = sys.stderr\n) -> CommandOutput:\n \"\"\"Awaits the creation and completion of an asynchronous process.\n\n Args:\n future_process: The eventually created process.\n out: Where to write stuff emitted by the process' stdout.\n err: Where to write stuff emitted by the process' stderr.\n\n Returns:\n A (captured output, captured error output, return code) triplet.\n \"\"\"\n process = await future_process\n future_output = _async_forward(process.stdout, out)\n future_err_output = _async_forward(process.stderr, err)\n output, err_output = await asyncio.gather(future_output, future_err_output)\n await process.wait()\n\n return CommandOutput(output, err_output, process.returncode)", "response": "A wrapper around asyncio. Popen that waits for the process to finish."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_cmd(*cmd: Optional[str],\n out: Optional[Union[TeeCapture, IO[str]]] = sys.stdout,\n err: Optional[Union[TeeCapture, IO[str]]] = sys.stderr,\n raise_on_fail: bool = True,\n log_run_to_stderr: bool = True,\n abbreviate_non_option_arguments: bool = False,\n **kwargs\n ) -> CommandOutput:\n \"\"\"Invokes a subprocess and waits for it to finish.\n\n Args:\n cmd: Components of the command to execute, e.g. [\"echo\", \"dog\"].\n out: Where to write the process' stdout. Defaults to sys.stdout. Can be\n anything accepted by print's 'file' parameter, or None if the\n output should be dropped, or a TeeCapture instance. If a TeeCapture\n instance is given, the first element of the returned tuple will be\n the captured output.\n err: Where to write the process' stderr. Defaults to sys.stderr. Can be\n anything accepted by print's 'file' parameter, or None if the\n output should be dropped, or a TeeCapture instance. If a TeeCapture\n instance is given, the second element of the returned tuple will be\n the captured error output.\n raise_on_fail: If the process returns a non-zero error code\n and this flag is set, a CalledProcessError will be raised.\n Otherwise the return code is the third element of the returned\n tuple.\n log_run_to_stderr: Determines whether the fact that this shell command\n was executed is logged to sys.stderr or not.\n abbreviate_non_option_arguments: When logging to stderr, this cuts off\n the potentially-huge tail of the command listing off e.g. hundreds\n of file paths. No effect if log_run_to_stderr is not set.\n\n **kwargs: Extra arguments for asyncio.create_subprocess_shell, such as\n a cwd (current working directory) argument.\n\n Returns:\n A (captured output, captured error output, return code) triplet. The\n captured outputs will be None if the out or err parameters were not set\n to an instance of TeeCapture.\n\n Raises:\n subprocess.CalledProcessError: The process returned a non-zero error\n code and raise_on_fail was set.\n \"\"\"\n kept_cmd = tuple(cast(str, e) for e in cmd if e is not None)\n if log_run_to_stderr:\n cmd_desc = kept_cmd\n if abbreviate_non_option_arguments:\n cmd_desc = abbreviate_command_arguments_after_switches(cmd_desc)\n print('run:', cmd_desc, file=sys.stderr)\n result = asyncio.get_event_loop().run_until_complete(\n _async_wait_for_process(\n asyncio.create_subprocess_exec(\n *kept_cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n **kwargs),\n out,\n err))\n if raise_on_fail and result[2]:\n raise subprocess.CalledProcessError(result[2], kept_cmd)\n return result", "response": "Invokes a subprocess and waits for it to finish."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef run_shell(cmd: str,\n out: Optional[Union[TeeCapture, IO[str]]] = sys.stdout,\n err: Optional[Union[TeeCapture, IO[str]]] = sys.stderr,\n raise_on_fail: bool = True,\n log_run_to_stderr: bool = True,\n **kwargs\n ) -> CommandOutput:\n \"\"\"Invokes a shell command and waits for it to finish.\n\n Args:\n cmd: The command line string to execute, e.g. \"echo dog | cat > file\".\n out: Where to write the process' stdout. Defaults to sys.stdout. Can be\n anything accepted by print's 'file' parameter, or None if the\n output should be dropped, or a TeeCapture instance. If a TeeCapture\n instance is given, the first element of the returned tuple will be\n the captured output.\n err: Where to write the process' stderr. Defaults to sys.stderr. Can be\n anything accepted by print's 'file' parameter, or None if the\n output should be dropped, or a TeeCapture instance. If a TeeCapture\n instance is given, the second element of the returned tuple will be\n the captured error output.\n raise_on_fail: If the process returns a non-zero error code\n and this flag is set, a CalledProcessError will be raised.\n Otherwise the return code is the third element of the returned\n tuple.\n log_run_to_stderr: Determines whether the fact that this shell command\n was executed is logged to sys.stderr or not.\n **kwargs: Extra arguments for asyncio.create_subprocess_shell, such as\n a cwd (current working directory) argument.\n\n Returns:\n A (captured output, captured error output, return code) triplet. The\n captured outputs will be None if the out or err parameters were not set\n to an instance of TeeCapture.\n\n Raises:\n subprocess.CalledProcessError: The process returned a non-zero error\n code and raise_on_fail was set.\n \"\"\"\n if log_run_to_stderr:\n print('shell:', cmd, file=sys.stderr)\n result = asyncio.get_event_loop().run_until_complete(\n _async_wait_for_process(\n asyncio.create_subprocess_shell(\n cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n **kwargs),\n out,\n err))\n if raise_on_fail and result[2]:\n raise subprocess.CalledProcessError(result[2], cmd)\n return result", "response": "Invokes a shell command and waits for it to finish."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef output_of(*cmd: Optional[str], **kwargs) -> str:\n result = cast(str, run_cmd(*cmd,\n log_run_to_stderr=False,\n out=TeeCapture(),\n **kwargs).out)\n\n # Strip final newline.\n if result.endswith('\\n'):\n result = result[:-1]\n\n return result", "response": "Invokes a subprocess and returns its output as a string."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef chosen_angle_to_half_turns(\n half_turns: Optional[Union[sympy.Basic, float]] = None,\n rads: Optional[float] = None,\n degs: Optional[float] = None,\n default: float = 1.0,\n) -> Union[sympy.Basic, float]:\n \"\"\"Returns a half_turns value based on the given arguments.\n\n At most one of half_turns, rads, degs must be specified. If none are\n specified, the output defaults to half_turns=1.\n\n Args:\n half_turns: The number of half turns to rotate by.\n rads: The number of radians to rotate by.\n degs: The number of degrees to rotate by\n default: The half turns angle to use if nothing else is specified.\n\n Returns:\n A number of half turns.\n \"\"\"\n\n if len([1 for e in [half_turns, rads, degs] if e is not None]) > 1:\n raise ValueError('Redundant angle specification. '\n 'Use ONE of half_turns, rads, or degs.')\n\n if rads is not None:\n return rads / np.pi\n\n if degs is not None:\n return degs / 180\n\n if half_turns is not None:\n return half_turns\n\n return default", "response": "Returns a half_turns value based on the given arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef chosen_angle_to_canonical_half_turns(\n half_turns: Optional[Union[sympy.Basic, float]] = None,\n rads: Optional[float] = None,\n degs: Optional[float] = None,\n default: float = 1.0,\n) -> Union[sympy.Basic, float]:\n \"\"\"Returns a canonicalized half_turns based on the given arguments.\n\n At most one of half_turns, rads, degs must be specified. If none are\n specified, the output defaults to half_turns=1.\n\n Args:\n half_turns: The number of half turns to rotate by.\n rads: The number of radians to rotate by.\n degs: The number of degrees to rotate by\n default: The half turns angle to use if nothing else is specified.\n\n Returns:\n A number of half turns.\n \"\"\"\n return canonicalize_half_turns(\n chosen_angle_to_half_turns(\n half_turns=half_turns,\n rads=rads,\n degs=degs,\n default=default))", "response": "Returns a canonicalized half_turns based on the given arguments."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef canonicalize_half_turns(\n half_turns: Union[sympy.Basic, float]\n) -> Union[sympy.Basic, float]:\n \"\"\"Wraps the input into the range (-1, +1].\"\"\"\n if isinstance(half_turns, sympy.Basic):\n return half_turns\n half_turns %= 2\n if half_turns > 1:\n half_turns -= 2\n return half_turns", "response": "Wraps the input into the range - 1 - 1."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nimplement a single - qubit operation with few rotations.", "response": "def single_qubit_matrix_to_pauli_rotations(\n mat: np.ndarray, atol: float = 0\n) -> List[Tuple[ops.Pauli, float]]:\n \"\"\"Implements a single-qubit operation with few rotations.\n\n Args:\n mat: The 2x2 unitary matrix of the operation to implement.\n atol: A limit on the amount of absolute error introduced by the\n construction.\n\n Returns:\n A list of (Pauli, half_turns) tuples that, when applied in order,\n perform the desired operation.\n \"\"\"\n\n def is_clifford_rotation(half_turns):\n return near_zero_mod(half_turns, 0.5, atol=atol)\n\n def to_quarter_turns(half_turns):\n return round(2 * half_turns) % 4\n\n def is_quarter_turn(half_turns):\n return (is_clifford_rotation(half_turns) and\n to_quarter_turns(half_turns) % 2 == 1)\n\n def is_half_turn(half_turns):\n return (is_clifford_rotation(half_turns) and\n to_quarter_turns(half_turns) == 2)\n\n def is_no_turn(half_turns):\n return (is_clifford_rotation(half_turns) and\n to_quarter_turns(half_turns) == 0)\n\n # Decompose matrix\n z_rad_before, y_rad, z_rad_after = (\n linalg.deconstruct_single_qubit_matrix_into_angles(mat))\n z_ht_before = z_rad_before / np.pi - 0.5\n m_ht = y_rad / np.pi\n m_pauli = ops.pauli_gates.X # type: ops.pauli_gates.Pauli\n z_ht_after = z_rad_after / np.pi + 0.5\n\n # Clean up angles\n if is_clifford_rotation(z_ht_before):\n if ((is_quarter_turn(z_ht_before) or is_quarter_turn(z_ht_after)) ^\n (is_half_turn(m_ht) and is_no_turn(z_ht_before-z_ht_after))):\n z_ht_before += 0.5\n z_ht_after -= 0.5\n m_pauli = ops.pauli_gates.Y\n if is_half_turn(z_ht_before) or is_half_turn(z_ht_after):\n z_ht_before -= 1\n z_ht_after += 1\n m_ht = -m_ht\n if is_no_turn(m_ht):\n z_ht_before += z_ht_after\n z_ht_after = 0\n elif is_half_turn(m_ht):\n z_ht_after -= z_ht_before\n z_ht_before = 0\n\n # Generate operations\n rotation_list = [\n (ops.pauli_gates.Z, z_ht_before),\n (m_pauli, m_ht),\n (ops.pauli_gates.Z, z_ht_after)]\n return [(pauli, ht) for pauli, ht in rotation_list if not is_no_turn(ht)]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef single_qubit_matrix_to_gates(\n mat: np.ndarray, tolerance: float = 0\n) -> List[ops.SingleQubitGate]:\n \"\"\"Implements a single-qubit operation with few gates.\n\n Args:\n mat: The 2x2 unitary matrix of the operation to implement.\n tolerance: A limit on the amount of error introduced by the\n construction.\n\n Returns:\n A list of gates that, when applied in order, perform the desired\n operation.\n \"\"\"\n rotations = single_qubit_matrix_to_pauli_rotations(mat, tolerance)\n return [cast(ops.SingleQubitGate, pauli)**ht for pauli, ht in rotations]", "response": "Implements a single - qubit operation with few gates."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef single_qubit_op_to_framed_phase_form(\n mat: np.ndarray) -> Tuple[np.ndarray, complex, complex]:\n \"\"\"Decomposes a 2x2 unitary M into U^-1 * diag(1, r) * U * diag(g, g).\n\n U translates the rotation axis of M to the Z axis.\n g fixes a global phase factor difference caused by the translation.\n r's phase is the amount of rotation around M's rotation axis.\n\n This decomposition can be used to decompose controlled single-qubit\n rotations into controlled-Z operations bordered by single-qubit operations.\n\n Args:\n mat: The qubit operation as a 2x2 unitary matrix.\n\n Returns:\n A 2x2 unitary U, the complex relative phase factor r, and the complex\n global phase factor g. Applying M is equivalent (up to global phase) to\n applying U, rotating around the Z axis to apply r, then un-applying U.\n When M is controlled, the control must be rotated around the Z axis to\n apply g.\n \"\"\"\n vals, vecs = np.linalg.eig(mat)\n u = np.conj(vecs).T\n r = vals[1] / vals[0]\n g = vals[0]\n return u, r, g", "response": "Decomposes a 2x2 unitary M into a 2x2 complex relative phase factor r and complex global phase factor g."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _deconstruct_single_qubit_matrix_into_gate_turns(\n mat: np.ndarray) -> Tuple[float, float, float]:\n \"\"\"Breaks down a 2x2 unitary into gate parameters.\n\n Args:\n mat: The 2x2 unitary matrix to break down.\n\n Returns:\n A tuple containing the amount to rotate around an XY axis, the phase of\n that axis, and the amount to phase around Z. All results will be in\n fractions of a whole turn, with values canonicalized into the range\n [-0.5, 0.5).\n \"\"\"\n pre_phase, rotation, post_phase = (\n linalg.deconstruct_single_qubit_matrix_into_angles(mat))\n\n # Figure out parameters of the actual gates we will do.\n tau = 2 * np.pi\n xy_turn = rotation / tau\n xy_phase_turn = 0.25 - pre_phase / tau\n total_z_turn = (post_phase + pre_phase) / tau\n\n # Normalize turns into the range [-0.5, 0.5).\n return (_signed_mod_1(xy_turn), _signed_mod_1(xy_phase_turn),\n _signed_mod_1(total_z_turn))", "response": "Breaks down a 2x2 unitary matrix into gate parameters."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef single_qubit_matrix_to_phased_x_z(\n mat: np.ndarray,\n atol: float = 0\n) -> List[ops.SingleQubitGate]:\n \"\"\"Implements a single-qubit operation with a PhasedX and Z gate.\n\n If one of the gates isn't needed, it will be omitted.\n\n Args:\n mat: The 2x2 unitary matrix of the operation to implement.\n atol: A limit on the amount of error introduced by the\n construction.\n\n Returns:\n A list of gates that, when applied in order, perform the desired\n operation.\n \"\"\"\n\n xy_turn, xy_phase_turn, total_z_turn = (\n _deconstruct_single_qubit_matrix_into_gate_turns(mat))\n\n # Build the intended operation out of non-negligible XY and Z rotations.\n result = [\n ops.PhasedXPowGate(exponent=2 * xy_turn,\n phase_exponent=2 * xy_phase_turn),\n ops.Z**(2 * total_z_turn)\n ]\n result = [\n g for g in result\n if protocols.trace_distance_bound(g) > atol\n ]\n\n # Special case: XY half-turns can absorb Z rotations.\n if len(result) == 2 and abs(xy_turn) >= 0.5 - atol:\n return [\n ops.PhasedXPowGate(phase_exponent=2 * xy_phase_turn + total_z_turn)\n ]\n\n return result", "response": "Implements a single - qubit operation with a PhasedX and Z gate."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a unitary matrix describing the given value.", "response": "def unitary(val: Any,\n default: TDefault = RaiseTypeErrorIfNotProvided\n ) -> Union[np.ndarray, TDefault]:\n \"\"\"Returns a unitary matrix describing the given value.\n\n Args:\n val: The value to describe with a unitary matrix.\n default: Determines the fallback behavior when `val` doesn't have\n a unitary matrix. If `default` is not set, a TypeError is raised. If\n default is set to a value, that value is returned.\n\n Returns:\n If `val` has a _unitary_ method and its result is not NotImplemented,\n that result is returned. Otherwise, if `val` is a cirq.Gate or\n cirq.Operation, decomposition will be attempted and the resulting\n unitary is returned if unitaries exist for all operations of the\n decompostion. If the result is still NotImplemented and a default value\n was specified, the default value is returned.\n\n Raises:\n TypeError: `val` doesn't have a _unitary_ method (or that method\n returned NotImplemented) and also no default value was specified.\n \"\"\"\n from cirq import Gate, Operation # HACK: Avoids circular dependencies.\n\n getter = getattr(val, '_unitary_', None)\n result = NotImplemented if getter is None else getter()\n if result is not NotImplemented:\n return result\n\n # Fallback to decomposition for gates and operations\n if isinstance(val, (Gate, Operation)):\n decomposed_unitary = _decompose_and_get_unitary(val)\n if decomposed_unitary is not None:\n return decomposed_unitary\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if getter is None:\n raise TypeError(\"object of type '{}' \"\n \"has no _unitary_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _unitary_ method, \"\n \"but it returned NotImplemented.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef has_unitary(val: Any) -> bool:\n from cirq.protocols.decompose import (decompose_once,\n decompose_once_with_qubits)\n # HACK: Avoids circular dependencies.\n from cirq import Gate, Operation, LineQubit\n\n getter = getattr(val, '_has_unitary_', None)\n result = NotImplemented if getter is None else getter()\n if result is not NotImplemented:\n return result\n\n # Fallback to explicit _unitary_\n unitary_getter = getattr(val, '_unitary_', None)\n if unitary_getter is not None and unitary_getter() is not NotImplemented:\n return True\n\n # Fallback to decomposition for gates and operations\n if isinstance(val, Gate):\n # Since gates don't know about qubits, we need to create some\n decomposed_val = decompose_once_with_qubits(val,\n LineQubit.range(val.num_qubits()),\n default=None)\n if decomposed_val is not None:\n return all(has_unitary(v) for v in decomposed_val)\n elif isinstance(val, Operation):\n decomposed_val = decompose_once(val, default=None)\n if decomposed_val is not None:\n return all(has_unitary(v) for v in decomposed_val)\n\n # Finally, fallback to full unitary method, including decomposition\n return unitary(val, None) is not None", "response": "Returns whether the value has a unitary matrix representation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _decompose_and_get_unitary(val: Union['cirq.Operation', 'cirq.Gate']\n ) -> np.ndarray:\n \"\"\"Try to decompose a cirq.Operation or cirq.Gate, and return its unitary\n if it exists.\n\n Returns:\n If `val` can be decomposed into unitaries, calculate the resulting\n unitary and return it. If it doesn't exist, None is returned.\n \"\"\"\n from cirq.protocols.apply_unitary import apply_unitary, ApplyUnitaryArgs\n from cirq.protocols.decompose import (decompose_once,\n decompose_once_with_qubits)\n from cirq import Gate, LineQubit, Operation\n\n if isinstance(val, Operation):\n qubits = val.qubits\n decomposed_val = decompose_once(val, default=None)\n elif isinstance(val, Gate):\n # Since gates don't know about qubits, we need to create some\n qubits = tuple(LineQubit.range(val.num_qubits()))\n decomposed_val = decompose_once_with_qubits(val,\n qubits,\n default=None)\n\n if decomposed_val is not None:\n # Calculate the resulting unitary (if it exists)\n n = len(qubits)\n state = np.eye(1 << n, dtype=np.complex128)\n state.shape = (2,) * (2 * n)\n buffer = np.zeros(state.shape, dtype=np.complex128)\n qubit_map = {q: i for i, q in enumerate(qubits)}\n result = state\n for op in decomposed_val:\n indices = [qubit_map[q] for q in op.qubits]\n result = apply_unitary(\n unitary_value=op,\n args=ApplyUnitaryArgs(state, buffer, indices),\n default=None)\n if result is None:\n return None\n if result is buffer:\n buffer = state\n state = result\n if result is not None:\n return result.reshape((1 << n, 1 << n))", "response": "Decomposes a cirq. Operation or cirq. Gate and returns its unitary if it exists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if the user has write permission.", "response": "def check_collaborator_has_write(repo: GithubRepository, username: str\n ) -> Optional[CannotAutomergeError]:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/events/#list-events-for-an-issue\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/collaborators/{}/permission\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n username,\n repo.access_token))\n\n response = requests.get(url)\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Collaborator check failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n\n payload = json.JSONDecoder().decode(response.content.decode())\n if payload['permission'] not in ['admin', 'write']:\n return CannotAutomergeError(\n 'Only collaborators with write permission can use automerge.')\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_auto_merge_labeler(repo: GithubRepository, pull_id: int\n ) -> Optional[CannotAutomergeError]:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/events/#list-events-for-an-issue\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/issues/{}/events\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n repo.access_token))\n\n response = requests.get(url)\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Event check failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n\n payload = json.JSONDecoder().decode(response.content.decode())\n relevant = [event\n for event in payload\n if event['event'] == 'labeled' and\n event['label']['name'] in AUTO_MERGE_LABELS]\n if not relevant:\n return CannotAutomergeError('\"automerge\" label was never added.')\n\n return check_collaborator_has_write(repo, relevant[-1]['actor']['login'])", "response": "Check if a labeler is relevant to a new issue."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_comment(repo: GithubRepository, pull_id: int, text: str) -> None:\n url = (\"https://api.github.com/repos/{}/{}/issues/{}/comments\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n repo.access_token))\n data = {\n 'body': text\n }\n response = requests.post(url, json=data)\n\n if response.status_code != 201:\n raise RuntimeError('Add comment failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))", "response": "Adds a comment to a node."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef edit_comment(repo: GithubRepository, text: str, comment_id: int) -> None:\n url = (\"https://api.github.com/repos/{}/{}/issues/comments/{}\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n comment_id,\n repo.access_token))\n data = {\n 'body': text\n }\n response = requests.patch(url, json=data)\n\n if response.status_code != 200:\n raise RuntimeError('Edit comment failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))", "response": "Edit a comment in a node."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef classify_pr_synced_state(pr: PullRequestDetails) -> Optional[bool]:\n state = pr.payload['mergeable_state'].lower()\n classification = {\n 'behind': False,\n 'clean': True,\n }\n return classification.get(state, None)", "response": "Classifies the state of a single pull request."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the status of a pull request.", "response": "def get_pr_review_status(pr: PullRequestDetails) -> Any:\n \"\"\"\n References:\n https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/pulls/{}/reviews\"\n \"?access_token={}\".format(pr.repo.organization,\n pr.repo.name,\n pr.pull_id,\n pr.repo.access_token))\n response = requests.get(url)\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Get review failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n\n return json.JSONDecoder().decode(response.content.decode())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting check - runs for a specific pull request.", "response": "def get_pr_checks(pr: PullRequestDetails) -> Dict[str, Any]:\n \"\"\"\n References:\n https://developer.github.com/v3/checks/runs/#list-check-runs-for-a-specific-ref\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/commits/{}/check-runs\"\n \"?access_token={}\".format(pr.repo.organization,\n pr.repo.name,\n pr.branch_sha,\n pr.repo.access_token))\n response = requests.get(\n url,\n headers={'Accept': 'application/vnd.github.antiope-preview+json'})\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Get check-runs failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n\n return json.JSONDecoder().decode(response.content.decode())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_repo_ref(repo: GithubRepository, ref: str) -> Dict[str, Any]:\n\n url = (\"https://api.github.com/repos/{}/{}/git/refs/{}\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n ref,\n repo.access_token))\n response = requests.get(url)\n if response.status_code != 200:\n raise RuntimeError(\n 'Refs get failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n payload = json.JSONDecoder().decode(response.content.decode())\n return payload", "response": "Get a single reference from a repository."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nlist the comments on an issue on a GitHub repository.", "response": "def list_pr_comments(repo: GithubRepository, pull_id: int\n ) -> List[Dict[str, Any]]:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/comments/#list-comments-on-an-issue\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/issues/{}/comments\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n repo.access_token))\n response = requests.get(url)\n if response.status_code != 200:\n raise RuntimeError(\n 'Comments get failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n payload = json.JSONDecoder().decode(response.content.decode())\n return payload"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting a single comment from a branch.", "response": "def delete_comment(repo: GithubRepository, comment_id: int) -> None:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/comments/#delete-a-comment\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/issues/comments/{}\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n comment_id,\n repo.access_token))\n response = requests.delete(url)\n if response.status_code != 204:\n raise RuntimeError(\n 'Comment delete failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntry to merge the branch with the master branch.", "response": "def attempt_sync_with_master(pr: PullRequestDetails\n ) -> Union[bool, CannotAutomergeError]:\n \"\"\"\n References:\n https://developer.github.com/v3/repos/merging/#perform-a-merge\n \"\"\"\n master_sha = get_master_sha(pr.repo)\n remote = pr.remote_repo\n url = (\"https://api.github.com/repos/{}/{}/merges\"\n \"?access_token={}\".format(remote.organization,\n remote.name,\n remote.access_token))\n data = {\n 'base': pr.branch_name,\n 'head': master_sha,\n 'commit_message': 'Update branch (automerge)'.format(pr.branch_name)\n }\n response = requests.post(url, json=data)\n\n if response.status_code == 201:\n # Merge succeeded.\n log('Synced #{} ({!r}) with master.'.format(pr.pull_id, pr.title))\n return True\n\n if response.status_code == 204:\n # Already merged.\n return False\n\n if response.status_code == 409:\n # Merge conflict.\n return CannotAutomergeError(\"There's a merge conflict.\")\n\n if response.status_code == 403:\n # Permission denied.\n return CannotAutomergeError(\n \"Spurious failure. Github API requires me to be an admin on the \"\n \"fork repository to merge master into the PR branch. Hit \"\n \"'Update Branch' for me before trying again.\")\n\n raise RuntimeError('Sync with master failed for unknown reason. '\n 'Code: {}. Content: {}.'.format(response.status_code,\n response.content))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nattempting to merge a single branch into a single branch.", "response": "def attempt_squash_merge(pr: PullRequestDetails\n ) -> Union[bool, CannotAutomergeError]:\n \"\"\"\n References:\n https://developer.github.com/v3/pulls/#merge-a-pull-request-merge-button\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/pulls/{}/merge\"\n \"?access_token={}\".format(pr.repo.organization,\n pr.repo.name,\n pr.pull_id,\n pr.repo.access_token))\n data = {\n 'commit_title': '{} (#{})'.format(pr.title, pr.pull_id),\n 'commit_message': pr.body,\n 'sha': pr.branch_sha,\n 'merge_method': 'squash'\n }\n response = requests.put(url, json=data)\n\n if response.status_code == 200:\n # Merge succeeded.\n log('Merged PR#{} ({!r}):\\n{}\\n'.format(\n pr.pull_id,\n pr.title,\n indent(pr.body)))\n return True\n\n if response.status_code == 405:\n return CannotAutomergeError(\"Pull Request is not mergeable.\")\n\n if response.status_code == 409:\n # Need to sync.\n return False\n\n raise RuntimeError('Merge failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndelete a single branch from the remote repository.", "response": "def auto_delete_pr_branch(pr: PullRequestDetails) -> bool:\n \"\"\"\n References:\n https://developer.github.com/v3/git/refs/#delete-a-reference\n \"\"\"\n\n open_pulls = list_open_pull_requests(pr.repo, base_branch=pr.branch_name)\n if any(open_pulls):\n log('Not deleting branch {!r}. It is used elsewhere.'.format(\n pr.branch_name))\n return False\n\n remote = pr.remote_repo\n if pr.is_on_fork():\n log('Not deleting branch {!r}. It belongs to a fork ({}/{}).'.format(\n pr.branch_name,\n pr.remote_repo.organization,\n pr.remote_repo.name))\n return False\n\n url = (\"https://api.github.com/repos/{}/{}/git/refs/heads/{}\"\n \"?access_token={}\".format(remote.organization,\n remote.name,\n pr.branch_name,\n remote.access_token))\n response = requests.delete(url)\n\n if response.status_code == 204:\n # Delete succeeded.\n log('Deleted branch {!r}.'.format(pr.branch_name))\n return True\n\n log('Delete failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding labels to an issue.", "response": "def add_labels_to_pr(repo: GithubRepository,\n pull_id: int,\n *labels: str,\n override_token: str = None) -> None:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/labels/#add-labels-to-an-issue\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/issues/{}/labels\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n override_token or repo.access_token))\n response = requests.post(url, json=list(labels))\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Add labels failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_label_from_pr(repo: GithubRepository,\n pull_id: int,\n label: str) -> bool:\n \"\"\"\n References:\n https://developer.github.com/v3/issues/labels/#remove-a-label-from-an-issue\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/issues/{}/labels/{}\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n label,\n repo.access_token))\n response = requests.delete(url)\n\n if response.status_code == 404:\n payload = json.JSONDecoder().decode(response.content.decode())\n if payload['message'] == 'Label does not exist':\n return False\n\n if response.status_code == 200:\n # Removed the label.\n return True\n\n raise RuntimeError(\n 'Label remove failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))", "response": "Removes a label from a pull request."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef drop_temporary(pr: PullRequestDetails,\n problem: Optional[CannotAutomergeError],\n prev_seen_times: Dict[int, datetime.datetime],\n next_seen_times: Dict[int, datetime.datetime],\n ) -> Optional[CannotAutomergeError]:\n \"\"\"Filters out problems that may be temporary.\"\"\"\n\n if problem is not None and problem.may_be_temporary:\n since = prev_seen_times.get(pr.pull_id, datetime.datetime.utcnow())\n if is_recent_date(since):\n next_seen_times[pr.pull_id] = since\n return None\n\n return problem", "response": "Filter out problems that may be temporary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_github(repo: GithubRepository,\n pull_id: int) -> 'PullRequestDetails':\n \"\"\"\n References:\n https://developer.github.com/v3/pulls/#get-a-single-pull-request\n \"\"\"\n url = (\"https://api.github.com/repos/{}/{}/pulls/{}\"\n \"?access_token={}\".format(repo.organization,\n repo.name,\n pull_id,\n repo.access_token))\n\n response = requests.get(url)\n\n if response.status_code != 200:\n raise RuntimeError(\n 'Pull check failed. Code: {}. Content: {}.'.format(\n response.status_code, response.content))\n\n payload = json.JSONDecoder().decode(response.content.decode())\n return PullRequestDetails(payload, repo)", "response": "Returns a PullRequestDetails object from a github repository and pull_id."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a M\u00f8lmer\u2013S\u00f8rensen gate rotating by the desired amount.", "response": "def MS(rads: float) -> ops.XXPowGate:\n \"\"\"The M\u00f8lmer\u2013S\u00f8rensen gate, a native two-qubit operation in ion traps.\n\n A rotation around the XX axis in the two-qubit bloch sphere.\n\n The gate implements the following unitary:\n\n exp(-i t XX) = [ cos(t) 0 0 -isin(t)]\n [ 0 cos(t) -isin(t) 0 ]\n [ 0 -isin(t) cos(t) 0 ]\n [-isin(t) 0 0 cos(t) ]\n\n Args:\n rads: The rotation angle in radians.\n\n Returns:\n M\u00f8lmer\u2013S\u00f8rensen gate rotating by the desired amount.\n \"\"\"\n return ops.XXPowGate(exponent=rads*2/np.pi, global_shift=-0.5)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate probability and draws if solution should be accepted.", "response": "def _accept(random_sample: float, cost_diff: float,\n temp: float) -> Tuple[bool, float]:\n \"\"\"Calculates probability and draws if solution should be accepted.\n\n Based on exp(-Delta*E/T) formula.\n\n Args:\n random_sample: Uniformly distributed random number in the range [0, 1).\n cost_diff: Cost difference between new and previous solutions.\n temp: Current temperature.\n\n Returns:\n Tuple of boolean and float, with boolean equal to True if solution is\n accepted, and False otherwise. The float value is acceptance\n probability.\n \"\"\"\n exponent = -cost_diff / temp\n if exponent >= 0.0:\n return True, 1.0\n else:\n probability = math.exp(exponent)\n return probability > random_sample, probability"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nminimizes the current annealing candidate.", "response": "def anneal_minimize(\n initial: T,\n cost_func: Callable[[T], float],\n move_func: Callable[[T], T],\n random_sample: Callable[[], float],\n temp_initial: float = 1.0e-2,\n temp_final: float = 1e-6,\n cooling_factor: float = 0.99,\n repeat: int = 100,\n trace_func: Callable[\n [T, float, float, float, bool], None] = None) -> T:\n \"\"\"Minimize solution using Simulated Annealing meta-heuristic.\n\n Args:\n initial: Initial solution of type T to the problem.\n cost_func: Callable which takes current solution of type T, evaluates it\n and returns float with the cost estimate. The better solution is,\n the lower resulting value should be; negative values are allowed.\n move_func: Callable which takes current solution of type T and returns a\n new solution candidate of type T which is random iteration over\n input solution. The input solution, which is argument to this\n callback should not be mutated.\n random_sample: Callable which gives uniformly sampled random value from\n the [0, 1) interval on each call.\n temp_initial: Optional initial temperature for simulated annealing\n optimization. Scale of this value is cost_func-dependent.\n temp_final: Optional final temperature for simulated annealing\n optimization, where search should be stopped. Scale of this value is\n cost_func-dependent.\n cooling_factor: Optional factor to be applied to the current temperature\n and give the new temperature, this must be strictly greater than 0\n and strictly lower than 1.\n repeat: Optional number of iterations to perform at each given\n temperature.\n trace_func: Optional callback for tracing simulated annealing progress.\n This is going to be called at each algorithm step for the arguments:\n solution candidate (T), current temperature (float), candidate cost\n (float), probability of accepting candidate (float), and acceptance\n decision (boolean).\n\n Returns:\n The best solution found.\n\n Raises:\n ValueError: When supplied arguments are invalid.\n \"\"\"\n\n if not 0.0 < cooling_factor < 1.0:\n raise ValueError(\"Cooling factor must be within (0, 1) range\")\n\n temp = temp_initial\n sol = initial\n sol_cost = cost_func(initial)\n best = sol\n best_cost = sol_cost\n\n if trace_func:\n trace_func(sol, temp, sol_cost, 1.0, True)\n\n while temp > temp_final:\n for _ in range(0, repeat):\n # Find a new solution candidate and evaluate its cost.\n cand = move_func(sol)\n cand_cost = cost_func(cand)\n\n # Store the best solution, regardless if it is accepted or not.\n if best_cost > cand_cost:\n best = cand\n best_cost = cand_cost\n\n accepted, probability = _accept(random_sample(),\n cand_cost - sol_cost,\n temp)\n if accepted:\n sol = cand\n sol_cost = cand_cost\n\n if trace_func:\n trace_func(cand, temp, cand_cost, probability, accepted)\n\n temp *= cooling_factor\n\n return best"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef op_gate_of_type(op: raw_types.Operation,\n gate_type: Type[TV]) -> Optional[TV]:\n \"\"\"Returns the gate of given type, if the op has that gate otherwise None.\n \"\"\"\n if isinstance(op, GateOperation) and isinstance(op.gate, gate_type):\n return op.gate\n return None", "response": "Returns the gate of given type if the op has that gate otherwise None."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntake a sequence of qubit pairs and returns a sequence in which every pair is at distance two.", "response": "def qubit_pairs_to_qubit_order(\n qubit_pairs: Sequence[Sequence[ops.Qid]]\n ) -> List[ops.Qid]:\n \"\"\"Takes a sequence of qubit pairs and returns a sequence in which every\n pair is at distance two.\n\n Specifically, given pairs (1a, 1b), (2a, 2b), etc. returns\n (1a, 2a, 1b, 2b, 3a, 4a, 3b, 4b, ...).\n \"\"\"\n\n if set(len(qubit_pair) for qubit_pair in qubit_pairs) != set((2,)):\n raise ValueError(\n 'set(len(qubit_pair) for qubit_pair in qubit_pairs) != '\n 'set((2,))')\n n_pairs = len(qubit_pairs)\n qubits = [] # type: List[ops.Qid]\n for i in range(0, 2 * (n_pairs // 2), 2):\n qubits += [qubit_pairs[i][0], qubit_pairs[i + 1][0],\n qubit_pairs[i][1], qubit_pairs[i + 1][1]]\n if n_pairs % 2:\n qubits += list(qubit_pairs[-1])\n return qubits"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef quartic_paired_acquaintance_strategy(\n qubit_pairs: Iterable[Tuple[ops.Qid, ops.Qid]]\n ) -> Tuple[circuits.Circuit, Sequence[ops.Qid]]:\n \"\"\"Acquaintance strategy for pairs of pairs.\n\n Implements UpCCGSD ansatz from arXiv:1810.02327.\n \"\"\"\n\n qubit_pairs = tuple(\n cast(Tuple[ops.Qid, ops.Qid], tuple(qubit_pair))\n for qubit_pair in qubit_pairs)\n qubits = qubit_pairs_to_qubit_order(qubit_pairs)\n n_qubits = len(qubits)\n swap_network = SwapNetworkGate((1,) * n_qubits, 2)(*qubits)\n strategy = circuits.Circuit.from_ops(swap_network,\n device=UnconstrainedAcquaintanceDevice)\n expose_acquaintance_gates(strategy)\n for i in reversed(range(0, n_qubits, 2)):\n moment = ops.Moment([acquaint(*qubits[j: j + 4])\n for j in range(i % 4, n_qubits - 3, 4)])\n strategy.insert(2 * i, moment)\n return strategy, qubits", "response": "Acquaintance strategy for pairs of pairs."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef circuit_diagram_info(val: Any,\n args: Optional[CircuitDiagramInfoArgs] = None,\n default=RaiseTypeErrorIfNotProvided):\n \"\"\"Requests information on drawing an operation in a circuit diagram.\n\n Calls _circuit_diagram_info_ on `val`. If `val` doesn't have\n _circuit_diagram_info_, or it returns NotImplemented, that indicates that\n diagram information is not available.\n\n Args:\n val: The operation or gate that will need to be drawn.\n args: A CircuitDiagramInfoArgs describing the desired drawing style.\n default: A default result to return if the value doesn't have circuit\n diagram information. If not specified, a TypeError is raised\n instead.\n\n Returns:\n If `val` has no _circuit_diagram_info_ method or it returns\n NotImplemented, then `default` is returned (or a TypeError is\n raised if no `default` is specified).\n\n Otherwise, the value returned by _circuit_diagram_info_ is returned.\n\n Raises:\n TypeError:\n `val` doesn't have circuit diagram information and `default` was\n not specified.\n \"\"\"\n\n # Attempt.\n if args is None:\n args = CircuitDiagramInfoArgs.UNINFORMED_DEFAULT\n getter = getattr(val, '_circuit_diagram_info_', None)\n result = NotImplemented if getter is None else getter(args)\n\n # Success?\n if isinstance(result, str):\n return CircuitDiagramInfo(wire_symbols=(result,))\n if isinstance(result, collections.Iterable):\n return CircuitDiagramInfo(wire_symbols=tuple(result))\n if result is not NotImplemented:\n return result\n\n # Failure.\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n if getter is None:\n raise TypeError(\n \"object of type '{}' \"\n \"has no _circuit_diagram_info_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _circuit_diagram_info_ \"\n \"method, but it returned NotImplemented.\".format(type(val)))", "response": "Returns a CircuitDiagramInfo object that can be used to draw an operation in a circuit diagram."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the Pauli expansion of val in the Pauli basis.", "response": "def pauli_expansion(\n val: Any,\n *,\n default: Union[value.LinearDict[str], TDefault]\n = RaiseTypeErrorIfNotProvided,\n atol: float = 1e-9\n) -> Union[value.LinearDict[str], TDefault]:\n \"\"\"Returns coefficients of the expansion of val in the Pauli basis.\n\n Args:\n val: The value whose Pauli expansion is to returned.\n default: Determines what happens when `val` does not have methods that\n allow Pauli expansion to be obtained (see below). If set, the value\n is returned in that case. Otherwise, TypeError is raised.\n atol: Ignore coefficients whose absolute value is smaller than this.\n\n Returns:\n If `val` has a _pauli_expansion_ method, then its result is returned.\n Otherwise, if `val` has a small unitary then that unitary is expanded\n in the Pauli basis and coefficients are returned. Otherwise, if default\n is set to None or other value then default is returned. Otherwise,\n TypeError is raised.\n\n Raises:\n TypeError if `val` has none of the methods necessary to obtain its Pauli\n expansion and no default value has been provided.\n \"\"\"\n method = getattr(val, '_pauli_expansion_', None)\n expansion = NotImplemented if method is None else method()\n\n if expansion is not NotImplemented:\n return expansion.clean(atol=atol)\n\n matrix = unitary(val, default=None)\n if matrix is None:\n if default is RaiseTypeErrorIfNotProvided:\n raise TypeError('No Pauli expansion for object {} of type {}'\n .format(val, type(val)))\n return default\n\n num_qubits = matrix.shape[0].bit_length() - 1\n basis = operator_spaces.kron_bases(operator_spaces.PAULI_BASIS,\n repeat=num_qubits)\n\n expansion = operator_spaces.expand_matrix_in_orthogonal_basis(matrix, basis)\n return expansion.clean(atol=atol)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsees definition in `cirq.SimulatesSamples`.", "response": "def _run(self, circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n repetitions: int) -> Dict[str, np.ndarray]:\n \"\"\"See definition in `cirq.SimulatesSamples`.\"\"\"\n param_resolver = param_resolver or study.ParamResolver({})\n resolved_circuit = protocols.resolve_parameters(circuit,\n param_resolver)\n\n if circuit.are_all_measurements_terminal():\n return self._run_sweep_sample(resolved_circuit, repetitions)\n else:\n return self._run_sweep_repeat(resolved_circuit, repetitions)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn an iterator over the state of the simulation.", "response": "def _simulator_iterator(self, circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n qubit_order: ops.QubitOrderOrList,\n initial_state: Union[int, np.ndarray]) -> Iterator:\n \"\"\"See definition in `cirq.SimulatesIntermediateState`.\n\n If the initial state is an int, the state is set to the computational\n basis state corresponding to this state. Otherwise if the initial\n state is a np.ndarray it is the full initial state, either a pure state\n or the full density matrix. If it is the pure state it must be the\n correct size, be normalized (an L2 norm of 1), and be safely castable\n to an appropriate dtype for the simulator. If it is a mixed state\n it must be correctly sized and positive semidefinite with trace one.\n \"\"\"\n param_resolver = param_resolver or study.ParamResolver({})\n resolved_circuit = protocols.resolve_parameters(circuit, param_resolver)\n actual_initial_state = 0 if initial_state is None else initial_state\n return self._base_iterator(resolved_circuit,\n qubit_order,\n actual_initial_state)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compute_displays_sweep(\n self,\n program: Union[circuits.Circuit, schedules.Schedule],\n params: Optional[study.Sweepable] = None,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n initial_state: Union[int, np.ndarray] = 0,\n ) -> List[study.ComputeDisplaysResult]:\n \"\"\"Computes displays in the supplied Circuit or Schedule.\n\n In contrast to `compute_displays`, this allows for sweeping\n over different parameter values.\n\n Args:\n program: The circuit or schedule to simulate.\n params: Parameters to run with the program.\n qubit_order: Determines the canonical ordering of the qubits used to\n define the order of amplitudes in the wave function.\n initial_state: If an int, the state is set to the computational\n basis state corresponding to this state. Otherwise if it is a\n np.ndarray it is the full initial state, either a pure state\n or the full density matrix. If it is the pure state it must be\n the correct size, be normalized (an L2 norm of 1), and be\n safely castable to an appropriate dtype for the simulator.\n If it is a mixed state it must be correctly sized and\n positive semidefinite with trace one.\n\n Returns:\n List of ComputeDisplaysResults for this run, one for each\n possible parameter resolver.\n \"\"\"\n circuit = (program if isinstance(program, circuits.Circuit)\n else program.to_circuit())\n param_resolvers = study.to_resolvers(params or study.ParamResolver({}))\n qubit_order = ops.QubitOrder.as_qubit_order(qubit_order)\n qubits = qubit_order.order_for(circuit.all_qubits())\n\n compute_displays_results = [] # type: List[study.ComputeDisplaysResult]\n for param_resolver in param_resolvers:\n display_values = {} # type: Dict[Hashable, Any]\n\n # Compute the displays in the first Moment\n moment = circuit[0]\n matrix = density_matrix_utils.to_valid_density_matrix(\n initial_state, num_qubits=len(qubits), dtype=self._dtype)\n qubit_map = {q: i for i, q in enumerate(qubits)}\n _enter_moment_display_values_into_dictionary(\n display_values, moment, matrix, qubit_order, qubit_map)\n\n # Compute the displays in the rest of the Moments\n all_step_results = self.simulate_moment_steps(\n circuit,\n param_resolver,\n qubit_order,\n initial_state)\n for step_result, moment in zip(all_step_results, circuit[1:]):\n _enter_moment_display_values_into_dictionary(\n display_values,\n moment,\n step_result.density_matrix(),\n qubit_order,\n step_result._qubit_map)\n\n compute_displays_results.append(study.ComputeDisplaysResult(\n params=param_resolver,\n display_values=display_values))\n\n return compute_displays_results", "response": "Computes the displays in the supplied circuit or schedule."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the density matrix to a new density matrix.", "response": "def set_density_matrix(self, density_matrix_repr: Union[int, np.ndarray]):\n \"\"\"Set the density matrix to a new density matrix.\n\n Args:\n density_matrix_repr: If this is an int, the density matrix is set to\n the computational basis state corresponding to this state. Otherwise\n if this is a np.ndarray it is the full state, either a pure state\n or the full density matrix. If it is the pure state it must be the\n correct size, be normalized (an L2 norm of 1), and be safely\n castable to an appropriate dtype for the simulator. If it is a\n mixed state it must be correctly sized and positive semidefinite\n with trace one.\n \"\"\"\n density_matrix = density_matrix_utils.to_valid_density_matrix(\n density_matrix_repr, len(self._qubit_map), self._dtype)\n density_matrix = np.reshape(density_matrix,\n self.simulator_state().density_matrix.shape)\n np.copyto(dst=self.simulator_state().density_matrix, src=density_matrix)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef density_matrix(self):\n size = 2 ** len(self._qubit_map)\n return np.reshape(self._density_matrix, (size, size))", "response": "Returns the density matrix at this step in the simulation."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating tensor product of bases.", "response": "def kron_bases(*bases: Dict[str, np.ndarray],\n repeat: int = 1) -> Dict[str, np.ndarray]:\n \"\"\"Creates tensor product of bases.\"\"\"\n product_basis = {'': 1}\n for basis in bases * repeat:\n product_basis = {\n name1 + name2: np.kron(matrix1, matrix2)\n for name1, matrix1 in product_basis.items()\n for name2, matrix2 in basis.items()\n }\n return product_basis"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing Hilbert - Schmidt inner product of two matrices. Linear in second argument.", "response": "def hilbert_schmidt_inner_product(m1: np.ndarray, m2: np.ndarray) -> complex:\n \"\"\"Computes Hilbert-Schmidt inner product of two matrices.\n\n Linear in second argument.\n \"\"\"\n return np.einsum('ij,ij', m1.conj(), m2)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef expand_matrix_in_orthogonal_basis(\n m: np.ndarray,\n basis: Dict[str, np.ndarray],\n) -> value.LinearDict[str]:\n \"\"\"Computes coefficients of expansion of m in basis.\n\n We require that basis be orthogonal w.r.t. the Hilbert-Schmidt inner\n product. We do not require that basis be orthonormal. Note that Pauli\n basis (I, X, Y, Z) is orthogonal, but not orthonormal.\n \"\"\"\n return value.LinearDict({\n name: (hilbert_schmidt_inner_product(b, m) /\n hilbert_schmidt_inner_product(b, b))\n for name, b in basis.items()\n })", "response": "Computes coefficients of expansion of m in basis."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomputing linear combination of basis vectors with given coefficients.", "response": "def matrix_from_basis_coefficients(expansion: value.LinearDict[str],\n basis: Dict[str, np.ndarray]) -> np.ndarray:\n \"\"\"Computes linear combination of basis vectors with given coefficients.\"\"\"\n some_element = next(iter(basis.values()))\n result = np.zeros_like(some_element, dtype=np.complex128)\n for name, coefficient in expansion.items():\n result += coefficient * basis[name]\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef converted_gate_set(circuit: circuits.Circuit,\n no_clifford_gates: bool = False,\n atol: float = 1e-8,\n ) -> circuits.Circuit:\n \"\"\"Returns a new, equivalent circuit using the gate set\n {SingleQubitCliffordGate,\n CZ/PauliInteractionGate, PauliStringPhasor}.\n \"\"\"\n conv_circuit = circuits.Circuit(circuit)\n optimizers.ConvertToCzAndSingleGates().optimize_circuit(conv_circuit)\n optimizers.MergeSingleQubitGates().optimize_circuit(conv_circuit)\n ConvertToPauliStringPhasors(ignore_failures=True,\n keep_clifford=not no_clifford_gates,\n atol=atol,\n ).optimize_circuit(conv_circuit)\n optimizers.DropEmptyMoments().optimize_circuit(conv_circuit)\n return conv_circuit", "response": "Converts a circuit using the gate set\n CZ and PauliStringPhasor."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a new circuit with optimized operations applied to the given xmon gate set.", "response": "def optimized_for_xmon(\n circuit: circuits.Circuit,\n new_device: Optional[xmon_device.XmonDevice] = None,\n qubit_map: Callable[[ops.Qid], devices.GridQubit] =\n lambda e: cast(devices.GridQubit, e),\n allow_partial_czs: bool = False,\n) -> circuits.Circuit:\n \"\"\"Optimizes a circuit with XmonDevice in mind.\n\n Starts by converting the circuit's operations to the xmon gate set, then\n begins merging interactions and rotations, ejecting pi-rotations and phasing\n operations, dropping unnecessary operations, and pushing operations earlier.\n\n Args:\n circuit: The circuit to optimize.\n new_device: The device the optimized circuit should be targeted at. If\n set to None, the circuit's current device is used.\n qubit_map: Transforms the qubits (e.g. so that they are GridQubits).\n allow_partial_czs: If true, the optimized circuit may contain partial CZ\n gates. Otherwise all partial CZ gates will be converted to full CZ\n gates. At worst, two CZ gates will be put in place of each partial\n CZ from the input.\n\n Returns:\n The optimized circuit.\n \"\"\"\n copy = circuit.copy()\n opts = _OPTIMIZERS_PART_CZ if allow_partial_czs else _OPTIMIZERS\n for optimizer in opts:\n optimizer(copy)\n\n return circuits.Circuit.from_ops(\n (op.transform_qubits(qubit_map) for op in copy.all_operations()),\n strategy=circuits.InsertStrategy.EARLIEST,\n device=new_device or copy.device)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\napplying a channel effect to the target tensor.", "response": "def apply_channel(val: Any,\n args: ApplyChannelArgs,\n default: TDefault = RaiseTypeErrorIfNotProvided\n) -> Union[np.ndarray, TDefault]:\n \"\"\"High performance evolution under a channel evolution.\n\n If `val` defines an `_apply_channel_` method, that method will be\n used to apply `val`'s channel effect to the target tensor. Otherwise, if\n `val` defines an `_apply_unitary_` method, that method will be used to\n apply `val`s channel effect to the target tensor. Otherwise, if `val`\n returns a non-default channel with `cirq.channel`, that channel will be\n applied using a generic method. If none of these cases apply, an\n exception is raised or the specified default value is returned.\n\n\n Args:\n val: The value with a channel to apply to the target.\n args: A mutable `cirq.ApplyChannelArgs` object describing the target\n tensor, available workspace, and left and right axes to operate on.\n The attributes of this object will be mutated as part of computing\n the result.\n default: What should be returned if `val` doesn't have a channel. If\n not specified, a TypeError is raised instead of returning a default\n value.\n\n Returns:\n If the receiving object is not able to apply a channel,\n the specified default value is returned (or a TypeError is raised). If\n this occurs, then `target_tensor` should not have been mutated.\n\n If the receiving object was able to work inline, directly\n mutating `target_tensor` it will return `target_tensor`. The caller is\n responsible for checking if the result is `target_tensor`.\n\n If the receiving object wrote its output over `out_buffer`, the\n result will be `out_buffer`. The caller is responsible for\n checking if the result is `out_buffer` (and e.g. swapping\n the buffer for the target tensor before the next call).\n\n Note that it is an error for the return object to be either of the\n auxiliary buffers, and the method will raise an AssertionError if\n this contract is violated.\n\n The receiving object may also write its output over a new buffer\n that it created, in which case that new array is returned.\n\n Raises:\n TypeError: `val` doesn't have a channel and `default` wasn't specified.\n AssertionError: if the\n \"\"\"\n # Check if the specialized method is present.\n func = getattr(val, '_apply_channel_', None)\n if func is not None:\n result = func(args)\n if result is not NotImplemented and result is not None:\n def err_str(buf_num_str):\n return (\n \"Object of type '{}' returned a result object equal to \"\n \"auxiliary_buffer{}. This type violates the contract \"\n \"that appears in apply_unitary's documentation.\".format(\n type(val), buf_num_str))\n assert result is not args.auxiliary_buffer0, err_str('0')\n assert result is not args.auxiliary_buffer1, err_str('1')\n return result\n\n # Possibly use `apply_unitary`.\n result = _apply_unitary(val, args)\n if result is not None:\n return result\n\n # Fallback to using the object's `_channel_` matrices.\n krauss = channel(val, None)\n if krauss is not None:\n return _apply_krauss(krauss, args)\n\n # Don't know how to apply channel. Fallback to specified default behavior.\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n raise TypeError(\n \"object of type '{}' has no _apply_channel_, _apply_unitary_, \"\n \"_unitary_, or _channel_ methods (or they returned None or \"\n \"NotImplemented).\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _apply_unitary(val: Any, args: 'ApplyChannelArgs') -> Optional[np.ndarray]:\n left_args = ApplyUnitaryArgs(target_tensor=args.target_tensor,\n available_buffer=args.auxiliary_buffer0,\n axes=args.left_axes)\n left_result = apply_unitary(val, left_args, None)\n if left_result is None:\n return None\n right_args = ApplyUnitaryArgs(\n target_tensor=np.conjugate(left_result),\n available_buffer=args.out_buffer,\n axes=args.right_axes)\n right_result = apply_unitary(val, right_args)\n np.conjugate(right_result, out=right_result)\n return right_result", "response": "Attempt to use apply_unitary and return the result."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _apply_krauss(krauss: Union[Tuple[np.ndarray], Sequence[Any]],\n args: 'ApplyChannelArgs') -> np.ndarray:\n \"\"\"Directly apply the kraus operators to the target tensor.\"\"\"\n # Initialize output.\n args.out_buffer[:] = 0\n # Stash initial state into buffer0.\n np.copyto(dst=args.auxiliary_buffer0, src=args.target_tensor)\n\n # Special case for single-qubit operations.\n if krauss[0].shape == (2, 2):\n return _apply_krauss_single_qubit(krauss, args)\n # Fallback to np.einsum for the general case.\n return _apply_krauss_multi_qubit(krauss, args)", "response": "Directly apply the kraus operators to the target tensor."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses slicing to apply single qubit channel.", "response": "def _apply_krauss_single_qubit(krauss: Union[Tuple[Any], Sequence[Any]],\n args: 'ApplyChannelArgs') -> np.ndarray:\n \"\"\"Use slicing to apply single qubit channel.\"\"\"\n zero_left = linalg.slice_for_qubits_equal_to(args.left_axes, 0)\n one_left = linalg.slice_for_qubits_equal_to(args.left_axes, 1)\n zero_right = linalg.slice_for_qubits_equal_to(args.right_axes, 0)\n one_right = linalg.slice_for_qubits_equal_to(args.right_axes, 1)\n for krauss_op in krauss:\n np.copyto(dst=args.target_tensor,\n src=args.auxiliary_buffer0)\n linalg.apply_matrix_to_slices(\n args.target_tensor,\n krauss_op,\n [zero_left, one_left],\n out=args.auxiliary_buffer1)\n # No need to transpose as we are acting on the tensor\n # representation of matrix, so transpose is done for us.\n linalg.apply_matrix_to_slices(\n args.auxiliary_buffer1,\n np.conjugate(krauss_op),\n [zero_right, one_right],\n out=args.target_tensor)\n args.out_buffer += args.target_tensor\n return args.out_buffer"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nusing numpy s einsum to apply a multi - qubit channel.", "response": "def _apply_krauss_multi_qubit(krauss: Union[Tuple[Any], Sequence[Any]],\n args: 'ApplyChannelArgs') -> np.ndarray:\n \"\"\"Use numpy's einsum to apply a multi-qubit channel.\"\"\"\n for krauss_op in krauss:\n np.copyto(dst=args.target_tensor, src=args.auxiliary_buffer0)\n krauss_tensor = np.reshape(\n krauss_op.astype(args.target_tensor.dtype),\n (2,) * len(args.left_axes) * 2)\n linalg.targeted_left_multiply(\n krauss_tensor,\n args.target_tensor,\n args.left_axes,\n out=args.auxiliary_buffer1)\n # No need to transpose as we are acting on the tensor\n # representation of matrix, so transpose is done for us.\n linalg.targeted_left_multiply(\n np.conjugate(krauss_tensor),\n args.auxiliary_buffer1,\n args.right_axes,\n out=args.target_tensor)\n args.out_buffer += args.target_tensor\n return args.out_buffer"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert any circuit into two circuits where the left and right gates are equivalent to the given circuit.", "response": "def convert_and_separate_circuit(circuit: circuits.Circuit,\n leave_cliffords: bool = True,\n atol: float = 1e-8,\n ) -> Tuple[circuits.Circuit, circuits.Circuit]:\n \"\"\"Converts any circuit into two circuits where (circuit_left+circuit_right)\n is equivalent to the given circuit.\n\n Args:\n circuit: Any Circuit that cirq.google.optimized_for_xmon() supports.\n All gates should either provide a decomposition or have a known one\n or two qubit unitary matrix.\n\n Returns:\n (circuit_left, circuit_right)\n\n circuit_left contains only PauliStringPhasor operations.\n\n circuit_right is a Clifford circuit which contains only\n SingleQubitCliffordGate and PauliInteractionGate gates.\n It also contains MeasurementGates if the\n given circuit contains measurements.\n \"\"\"\n circuit = converted_gate_set(circuit,\n no_clifford_gates=not leave_cliffords,\n atol=atol)\n return pauli_string_half(circuit), regular_half(circuit)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef regular_half(circuit: circuits.Circuit) -> circuits.Circuit:\n return circuits.Circuit(\n ops.Moment(op\n for op in moment.operations\n if not isinstance(op, ops.PauliStringPhasor))\n for moment in circuit)", "response": "Return only the Clifford part of a circuit."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn only the non - Clifford part of a circuit.", "response": "def pauli_string_half(circuit: circuits.Circuit) -> circuits.Circuit:\n \"\"\"Return only the non-Clifford part of a circuit. See\n convert_and_separate_circuit().\n\n Args:\n circuit: A Circuit with the gate set {SingleQubitCliffordGate,\n PauliInteractionGate, PauliStringPhasor}.\n\n Returns:\n A Circuit with only PauliStringPhasor operations.\n \"\"\"\n return circuits.Circuit.from_ops(\n _pull_non_clifford_before(circuit),\n strategy=circuits.InsertStrategy.EARLIEST)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nverify the density matrix_rep is valid and converts it to ndarray form.", "response": "def to_valid_density_matrix(\n density_matrix_rep: Union[int, np.ndarray],\n num_qubits: int,\n dtype: Type[np.number] = np.complex64) -> np.ndarray:\n \"\"\"Verifies the density_matrix_rep is valid and converts it to ndarray form.\n\n This method is used to support passing a matrix, a vector (wave function),\n or a computational basis state as a representation of a state.\n\n Args:\n density_matrix_rep: If an numpy array, if it is of rank 2 (a matrix),\n then this is the density matrix. If it is a numpy array of rank 1\n (a vector) then this is a wave function. If this is an int,\n then this is the computation basis state.\n num_qubits: The number of qubits for the density matrix. The\n density_matrix_rep must be valid for this number of qubits.\n dtype: The numpy dtype of the density matrix, will be used when creating\n the state for a computational basis state (int), or validated\n against if density_matrix_rep is a numpy array.\n\n Returns:\n A numpy matrix corresponding to the density matrix on the given number\n of qubits.\n\n Raises:\n ValueError if the density_matrix_rep is not valid.\n \"\"\"\n if (isinstance(density_matrix_rep, np.ndarray)\n and density_matrix_rep.ndim == 2):\n if density_matrix_rep.shape != (2 ** num_qubits, 2 ** num_qubits):\n raise ValueError(\n 'Density matrix was not square and of size 2 ** num_qubit, '\n 'instead was {}'.format(density_matrix_rep.shape))\n if not np.allclose(density_matrix_rep,\n np.transpose(np.conj(density_matrix_rep))):\n raise ValueError('The density matrix is not hermitian.')\n if not np.isclose(np.trace(density_matrix_rep), 1.0):\n raise ValueError(\n 'Density matrix did not have trace 1 but instead {}'.format(\n np.trace(density_matrix_rep)))\n if density_matrix_rep.dtype != dtype:\n raise ValueError(\n 'Density matrix had dtype {} but expected {}'.format(\n density_matrix_rep.dtype, dtype))\n if not np.all(np.linalg.eigvalsh(density_matrix_rep) > -1e-8):\n raise ValueError('The density matrix is not positive semidefinite.')\n return density_matrix_rep\n\n state_vector = wave_function.to_valid_state_vector(density_matrix_rep,\n num_qubits, dtype)\n return np.outer(state_vector, np.conj(state_vector))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform a measurement of the density matrix in the computational basis.", "response": "def measure_density_matrix(\n density_matrix: np.ndarray,\n indices: List[int],\n out: np.ndarray = None) -> Tuple[List[bool], np.ndarray]:\n \"\"\"Performs a measurement of the density matrix in the computational basis.\n\n This does not modify `density_matrix` unless the optional `out` is\n `density_matrix`.\n\n Args:\n density_matrix: The density matrix to be measured. This matrix is\n assumed to be positive semidefinite and trace one. The matrix is\n assumed to be of shape (2 ** integer, 2 ** integer) or\n (2, 2, ..., 2).\n indices: Which qubits are measured. The matrix is assumed to be supplied\n in big endian order. That is the xth index of v, when expressed as\n a bitstring, has the largest values in the 0th index.\n out: An optional place to store the result. If `out` is the same as\n the `density_matrix` parameter, then `density_matrix` will be\n modified inline. If `out` is not None, then the result is put into\n `out`. If `out` is None a new value will be allocated. In all of\n these cases `out` will be the same as the returned ndarray of the\n method. The shape and dtype of `out` will match that of\n `density_matrix` if `out` is None, otherwise it will match the\n shape and dtype of `out`.\n\n Returns:\n A tuple of a list and an numpy array. The list is an array of booleans\n corresponding to the measurement values (ordered by the indices). The\n numpy array is the post measurement matrix. This matrix has the same\n shape and dtype as the input matrix.\n\n Raises:\n ValueError if the dimension of the matrix is not compatible with a\n matrix of n qubits.\n IndexError if the indices are out of range for the number of qubits\n corresponding to the density matrix.\n \"\"\"\n num_qubits = _validate_num_qubits(density_matrix)\n _validate_indices(num_qubits, indices)\n\n if len(indices) == 0:\n if out is None:\n out = np.copy(density_matrix)\n elif out is not density_matrix:\n np.copyto(dst=out, src=density_matrix)\n return ([], out)\n # Final else: if out is matrix then matrix will be modified in place.\n\n # Cache initial shape.\n initial_shape = density_matrix.shape\n\n # Calculate the measurement probabilities and then make the measurement.\n probs = _probs(density_matrix, indices, num_qubits)\n result = np.random.choice(len(probs), p=probs)\n measurement_bits = [(1 & (result >> i)) for i in range(len(indices))]\n\n # Calculate the slice for the measurement result.\n result_slice = linalg.slice_for_qubits_equal_to(indices, result,\n num_qubits=num_qubits)\n # Create a mask which is False for only the slice.\n mask = np.ones([2] * 2 * num_qubits, dtype=bool)\n # Remove ellipses from last element of\n mask[result_slice * 2] = False\n\n if out is None:\n out = np.copy(density_matrix)\n elif out is not density_matrix:\n np.copyto(dst=out, src=density_matrix)\n # Final else: if out is matrix then matrix will be modified in place.\n\n # Potentially reshape to tensor, and then set masked values to 0.\n out.shape = [2] * num_qubits * 2\n out[mask] = 0\n\n # Restore original shape (if necessary) and renormalize.\n out.shape = initial_shape\n out /= probs[result]\n\n return measurement_bits, out"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the probabilities for a measurement on the given indices.", "response": "def _probs(density_matrix: np.ndarray, indices: List[int],\n num_qubits: int) -> List[float]:\n \"\"\"Returns the probabilities for a measurement on the given indices.\"\"\"\n # Only diagonal elements matter.\n all_probs = np.diagonal(\n np.reshape(density_matrix, (2 ** num_qubits, 2 ** num_qubits)))\n # Shape into a tensor\n tensor = np.reshape(all_probs, [2] * num_qubits)\n\n # Calculate the probabilities for measuring the particular results.\n probs = [\n np.sum(np.abs(tensor[linalg.slice_for_qubits_equal_to(indices, b)]))\n for b in range(2 ** len(indices))]\n\n # To deal with rounding issues, ensure that the probabilities sum to 1.\n probs /= np.sum(probs) # type: ignore\n return probs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _validate_num_qubits(density_matrix: np.ndarray) -> int:\n shape = density_matrix.shape\n half_index = len(shape) // 2\n row_size = np.prod(shape[:half_index]) if len(shape) != 0 else 0\n col_size = np.prod(shape[half_index:]) if len(shape) != 0 else 0\n if row_size != col_size:\n raise ValueError(\n 'Matrix was not square. Shape was {}'.format(shape))\n if row_size & (row_size - 1):\n raise ValueError(\n 'Matrix could not be shaped into a square matrix with dimensions '\n 'not a power of two. Shape was {}'.format(shape)\n )\n if len(shape) > 2 and not np.allclose(shape, 2):\n raise ValueError(\n 'Matrix is a tensor of rank greater than 2, but had dimensions '\n 'that are not powers of two. Shape was {}'.format(shape)\n )\n return int(row_size).bit_length() - 1", "response": "Validates that matrix s shape is a valid shape for qubits."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _apply_unitary_circuit(circuit: Circuit,\n state: np.ndarray,\n qubits: Tuple[ops.Qid, ...],\n dtype: Type[np.number]) -> np.ndarray:\n \"\"\"Applies a circuit's unitary effect to the given vector or matrix.\n\n This method assumes that the caller wants to ignore measurements.\n\n Args:\n circuit: The circuit to simulate. All operations must have a known\n matrix or decompositions leading to known matrices. Measurements\n are allowed to be in the circuit, but they will be ignored.\n state: The initial state tensor (i.e. superposition or unitary matrix).\n This is what will be left-multiplied by the circuit's effective\n unitary. If this is a state vector, it must have shape\n (2,) * num_qubits. If it is a unitary matrix it should have shape\n (2,) * (2*num_qubits).\n qubits: The qubits in the state tensor. Determines which axes operations\n apply to. An operation targeting the k'th qubit in this list will\n operate on the k'th axis of the state tensor.\n dtype: The numpy dtype to use for applying the unitary. Must be a\n complex dtype.\n\n Returns:\n The left-multiplied state tensor.\n \"\"\"\n qubit_map = {q: i for i, q in enumerate(qubits)}\n buffer = np.zeros(state.shape, dtype=dtype)\n\n def on_stuck(bad_op):\n return TypeError(\n 'Operation without a known matrix or decomposition: {!r}'.format(\n bad_op))\n\n unitary_ops = protocols.decompose(\n circuit.all_operations(),\n keep=protocols.has_unitary,\n intercepting_decomposer=_decompose_measurement_inversions,\n on_stuck_raise=on_stuck)\n\n for op in unitary_ops:\n indices = [qubit_map[q] for q in op.qubits]\n result = protocols.apply_unitary(\n unitary_value=op,\n args=protocols.ApplyUnitaryArgs(state, buffer, indices))\n if result is buffer:\n buffer = state\n state = result\n return state", "response": "Applies a circuit s unitary effect to the given vector or matrix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngroup runs of items that are identical according to a keying function.", "response": "def _group_until_different(items: Iterable[TIn],\n key: Callable[[TIn], TKey],\n value=lambda e: e):\n \"\"\"Groups runs of items that are identical according to a keying function.\n\n Args:\n items: The items to group.\n key: If two adjacent items produce the same output from this function,\n they will be grouped.\n value: Maps each item into a value to put in the group. Defaults to the\n item itself.\n\n Examples:\n _group_until_different(range(11), key=is_prime) yields\n (False, [0, 1])\n (True, [2, 3])\n (False, [4])\n (True, [5])\n (False, [6])\n (True, [7])\n (False, [8, 9, 10])\n\n Yields:\n Tuples containing the group key and item values.\n \"\"\"\n return ((k, [value(i) for i in v]) for (k, v) in groupby(items, key))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_ops(*operations: ops.OP_TREE,\n strategy: InsertStrategy = InsertStrategy.EARLIEST,\n device: devices.Device = devices.UnconstrainedDevice\n ) -> 'Circuit':\n \"\"\"Creates an empty circuit and appends the given operations.\n\n Args:\n operations: The operations to append to the new circuit.\n strategy: How to append the operations.\n device: Hardware that the circuit should be able to run on.\n\n Returns:\n The constructed circuit containing the operations.\n \"\"\"\n result = Circuit(device=device)\n result.append(operations, strategy)\n return result", "response": "Creates a new circuit with the given operations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _approx_eq_(self, other: Any, atol: Union[int, float]) -> bool:\n if not isinstance(other, type(self)):\n return NotImplemented\n return cirq.protocols.approx_eq(\n self._moments,\n other._moments,\n atol=atol\n ) and self._device == other._device", "response": "See cirq. protocols. SupportsApproximateEquality."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef with_device(\n self,\n new_device: devices.Device,\n qubit_mapping: Callable[[ops.Qid], ops.Qid] = lambda e: e,\n ) -> 'Circuit':\n \"\"\"Maps the current circuit onto a new device, and validates.\n\n Args:\n new_device: The new device that the circuit should be on.\n qubit_mapping: How to translate qubits from the old device into\n qubits on the new device.\n\n Returns:\n The translated circuit.\n \"\"\"\n return Circuit(\n moments=[ops.Moment(operation.transform_qubits(qubit_mapping)\n for operation in moment.operations)\n for moment in self._moments],\n device=new_device\n )", "response": "Returns a new circuit with the current circuit mapped onto a new device."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _repr_pretty_(self, p: Any, cycle: bool) -> None:\n if cycle:\n # There should never be a cycle. This is just in case.\n p.text('Circuit(...)')\n else:\n p.text(self.to_text_diagram())", "response": "Print ASCII diagram in Jupyter."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef next_moment_operating_on(self,\n qubits: Iterable[ops.Qid],\n start_moment_index: int = 0,\n max_distance: int = None) -> Optional[int]:\n \"\"\"Finds the index of the next moment that touches the given qubits.\n\n Args:\n qubits: We're looking for operations affecting any of these qubits.\n start_moment_index: The starting point of the search.\n max_distance: The number of moments (starting from the start index\n and moving forward) to check. Defaults to no limit.\n\n Returns:\n None if there is no matching moment, otherwise the index of the\n earliest matching moment.\n\n Raises:\n ValueError: negative max_distance.\n \"\"\"\n max_circuit_distance = len(self._moments) - start_moment_index\n if max_distance is None:\n max_distance = max_circuit_distance\n elif max_distance < 0:\n raise ValueError('Negative max_distance: {}'.format(max_distance))\n else:\n max_distance = min(max_distance, max_circuit_distance)\n\n return self._first_moment_operating_on(\n qubits,\n range(start_moment_index, start_moment_index + max_distance))", "response": "Finds the index of the next moment that touches the given qubits."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef next_moments_operating_on(self,\n qubits: Iterable[ops.Qid],\n start_moment_index: int = 0\n ) -> Dict[ops.Qid, int]:\n \"\"\"Finds the index of the next moment that touches each qubit.\n\n Args:\n qubits: The qubits to find the next moments acting on.\n start_moment_index: The starting point of the search.\n\n Returns:\n The index of the next moment that touches each qubit. If there\n is no such moment, the next moment is specified as the number of\n moments in the circuit. Equivalently, can be characterized as one\n plus the index of the last moment after start_moment_index\n (inclusive) that does *not* act on a given qubit.\n \"\"\"\n next_moments = {}\n for q in qubits:\n next_moment = self.next_moment_operating_on(\n [q], start_moment_index)\n next_moments[q] = (len(self._moments) if next_moment is None else\n next_moment)\n return next_moments", "response": "Finds the index of the next moments that act on a given qubit."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind the index of the previous matching moment that touches the given qubits.", "response": "def prev_moment_operating_on(\n self,\n qubits: Sequence[ops.Qid],\n end_moment_index: Optional[int] = None,\n max_distance: Optional[int] = None) -> Optional[int]:\n \"\"\"Finds the index of the next moment that touches the given qubits.\n\n Args:\n qubits: We're looking for operations affecting any of these qubits.\n end_moment_index: The moment index just after the starting point of\n the reverse search. Defaults to the length of the list of\n moments.\n max_distance: The number of moments (starting just before from the\n end index and moving backward) to check. Defaults to no limit.\n\n Returns:\n None if there is no matching moment, otherwise the index of the\n latest matching moment.\n\n Raises:\n ValueError: negative max_distance.\n \"\"\"\n if end_moment_index is None:\n end_moment_index = len(self._moments)\n\n if max_distance is None:\n max_distance = len(self._moments)\n elif max_distance < 0:\n raise ValueError('Negative max_distance: {}'.format(max_distance))\n else:\n max_distance = min(end_moment_index, max_distance)\n\n # Don't bother searching indices past the end of the list.\n if end_moment_index > len(self._moments):\n d = end_moment_index - len(self._moments)\n end_moment_index -= d\n max_distance -= d\n if max_distance <= 0:\n return None\n\n return self._first_moment_operating_on(qubits,\n (end_moment_index - k - 1\n for k in range(max_distance)))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndetermine how far can be reached into a given start frontier.", "response": "def reachable_frontier_from(\n self,\n start_frontier: Dict[ops.Qid, int],\n *,\n is_blocker: Callable[[ops.Operation], bool] = lambda op: False\n ) -> Dict[ops.Qid, int]:\n \"\"\"Determines how far can be reached into a circuit under certain rules.\n\n The location L = (qubit, moment_index) is *reachable* if and only if:\n\n a) L is one of the items in `start_frontier`.\n\n OR\n\n b) There is no operation at L and prev(L) = (qubit, moment_index-1)\n is reachable and L is within the bounds of the circuit.\n\n OR\n\n c) There is an operation P covering L and, for every location\n M = (q', moment_index) that P covers, the location\n prev(M) = (q', moment_index-1) is reachable. Also, P must not be\n classified as a blocker by the given `is_blocker` argument.\n\n In other words, the reachable region extends forward through time along\n each qubit until it hits a blocked operation or an operation that\n crosses into the set of not-involved-at-the-moment qubits.\n\n For each qubit q in `start_frontier`, the reachable locations will\n correspond to a contiguous range starting at start_frontier[q] and\n ending just before some index end_q. The result of this method is a\n dictionary, and that dictionary maps each qubit q to its end_q.\n\n Examples:\n\n If start_frontier is {\n cirq.LineQubit(0): 6,\n cirq.LineQubit(1): 2,\n cirq.LineQubit(2): 2,\n } then the reachable wire locations in the following circuit are\n highlighted with '\u2588' characters:\n\n 0 1 2 3 4 5 6 7 8 9 10 11 12 13\n 0: \u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500\n \u2502 \u2502\n 1: \u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2588\u2588H\u2588\u2588\u2588@\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502\n 2: \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2588\u2588\u2588\u2588\u2588\u2588@\u2588\u2588\u2588H\u2588\u2588\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502\n 3: \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n\n And the computed end_frontier is {\n cirq.LineQubit(0): 11,\n cirq.LineQubit(1): 9,\n cirq.LineQubit(2): 6,\n }\n\n Note that the frontier indices (shown above the circuit) are\n best thought of (and shown) as happening *between* moment indices.\n\n If we specify a blocker as follows:\n\n is_blocker=lambda: op == cirq.CZ(cirq.LineQubit(1),\n cirq.LineQubit(2))\n\n and use this start_frontier:\n\n {\n cirq.LineQubit(0): 0,\n cirq.LineQubit(1): 0,\n cirq.LineQubit(2): 0,\n cirq.LineQubit(3): 0,\n }\n\n Then this is the reachable area:\n\n 0 1 2 3 4 5 6 7 8 9 10 11 12 13\n 0: \u2500\u2588\u2588H\u2588\u2588\u2588@\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500\n \u2502 \u2502\n 1: \u2500\u2588\u2588\u2588\u2588\u2588\u2588@\u2588\u2588\u2588H\u2588\u2588\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502\n 2: \u2500\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n \u2502 \u2502\n 3: \u2500\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2500@\u2500\u2500\u2500H\u2500\u2500\u2500@\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n\n and the computed end_frontier is:\n\n {\n cirq.LineQubit(0): 11,\n cirq.LineQubit(1): 3,\n cirq.LineQubit(2): 3,\n cirq.LineQubit(3): 5,\n }\n\n Args:\n start_frontier: A starting set of reachable locations.\n is_blocker: A predicate that determines if operations block\n reachability. Any location covered by an operation that causes\n `is_blocker` to return True is considered to be an unreachable\n location.\n\n Returns:\n An end_frontier dictionary, containing an end index for each qubit q\n mapped to a start index by the given `start_frontier` dictionary.\n\n To determine if a location (q, i) was reachable, you can use\n this expression:\n\n q in start_frontier and start_frontier[q] <= i < end_frontier[q]\n\n where i is the moment index, q is the qubit, and end_frontier is the\n result of this method.\n \"\"\"\n active = set() # type: Set[ops.Qid]\n end_frontier = {}\n queue = BucketPriorityQueue[ops.Operation](drop_duplicate_entries=True)\n\n def enqueue_next(qubit: ops.Qid, moment: int) -> None:\n next_moment = self.next_moment_operating_on([qubit], moment)\n if next_moment is None:\n end_frontier[qubit] = max(len(self), start_frontier[qubit])\n if qubit in active:\n active.remove(qubit)\n else:\n next_op = self.operation_at(qubit, next_moment)\n assert next_op is not None\n queue.enqueue(next_moment, next_op)\n\n for start_qubit, start_moment in start_frontier.items():\n enqueue_next(start_qubit, start_moment)\n\n while queue:\n cur_moment, cur_op = queue.dequeue()\n for q in cur_op.qubits:\n if (q in start_frontier and\n cur_moment >= start_frontier[q] and\n q not in end_frontier):\n active.add(q)\n\n continue_past = (\n cur_op is not None and\n active.issuperset(cur_op.qubits) and\n not is_blocker(cur_op)\n )\n if continue_past:\n for q in cur_op.qubits:\n enqueue_next(q, cur_moment + 1)\n else:\n for q in cur_op.qubits:\n if q in active:\n end_frontier[q] = cur_moment\n active.remove(q)\n\n return end_frontier"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding operations between two given frontiers.", "response": "def findall_operations_between(self,\n start_frontier: Dict[ops.Qid, int],\n end_frontier: Dict[ops.Qid, int],\n omit_crossing_operations: bool = False\n ) -> List[Tuple[int, ops.Operation]]:\n \"\"\"Finds operations between the two given frontiers.\n\n If a qubit is in `start_frontier` but not `end_frontier`, its end index\n defaults to the end of the circuit. If a qubit is in `end_frontier` but\n not `start_frontier`, its start index defaults to the start of the\n circuit. Operations on qubits not mentioned in either frontier are not\n included in the results.\n\n Args:\n start_frontier: Just before where to start searching for operations,\n for each qubit of interest. Start frontier indices are\n inclusive.\n end_frontier: Just before where to stop searching for operations,\n for each qubit of interest. End frontier indices are exclusive.\n omit_crossing_operations: Determines whether or not operations that\n cross from a location between the two frontiers to a location\n outside the two frontiers are included or excluded. (Operations\n completely inside are always included, and operations completely\n outside are always excluded.)\n\n Returns:\n A list of tuples. Each tuple describes an operation found between\n the two frontiers. The first item of each tuple is the index of the\n moment containing the operation, and the second item is the\n operation itself. The list is sorted so that the moment index\n increases monotonically.\n \"\"\"\n result = BucketPriorityQueue[ops.Operation](\n drop_duplicate_entries=True)\n\n involved_qubits = set(start_frontier.keys()) | set(end_frontier.keys())\n # Note: only sorted to ensure a deterministic result ordering.\n for q in sorted(involved_qubits):\n for i in range(start_frontier.get(q, 0),\n end_frontier.get(q, len(self))):\n op = self.operation_at(q, i)\n if op is None:\n continue\n if (omit_crossing_operations and\n not involved_qubits.issuperset(op.qubits)):\n continue\n result.enqueue(i, op)\n\n return list(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef findall_operations_until_blocked(\n self,\n start_frontier: Dict[ops.Qid, int],\n is_blocker: Callable[[ops.Operation], bool] = lambda op: False\n ) -> List[Tuple[int, ops.Operation]]:\n \"\"\"\n Finds all operations until a blocking operation is hit. This returns\n a list of all operations from the starting frontier until a blocking\n operation is encountered. An operation is part of the list if\n it is involves a qubit in the start_frontier dictionary, comes after\n the moment listed in that dictionary, and before any blocking\n operationi that involve that qubit. Operations are only considered\n to blocking the qubits that they operate on, so a blocking operation\n that does not operate on any qubit in the starting frontier is not\n actually considered blocking. See `reachable_frontier_from` for a more\n in depth example of reachable states.\n\n Args:\n start_frontier: A starting set of reachable locations.\n is_blocker: A predicate that determines if operations block\n reachability. Any location covered by an operation that causes\n `is_blocker` to return True is considered to be an unreachable\n location.\n\n Returns:\n A list of tuples. Each tuple describes an operation found between\n the start frontier and a blocking operation. The first item of\n each tuple is the index of the moment containing the operation,\n and the second item is the operation itself.\n \"\"\"\n op_list = []\n max_index = len(self._moments)\n for qubit in start_frontier:\n current_index = start_frontier[qubit]\n if current_index < 0:\n current_index = 0\n while current_index < max_index:\n if self[current_index].operates_on_single_qubit(qubit):\n next_op = self.operation_at(qubit, current_index)\n if next_op is not None:\n if is_blocker(next_op):\n break\n op_list.append((current_index,next_op))\n current_index+=1\n return op_list", "response": "This function finds all operations until a blocking operation is hit."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the operation on a qubit within a moment.", "response": "def operation_at(self,\n qubit: ops.Qid,\n moment_index: int) -> Optional[ops.Operation]:\n \"\"\"Finds the operation on a qubit within a moment, if any.\n\n Args:\n qubit: The qubit to check for an operation on.\n moment_index: The index of the moment to check for an operation\n within. Allowed to be beyond the end of the circuit.\n\n Returns:\n None if there is no operation on the qubit at the given moment, or\n else the operation.\n \"\"\"\n if not 0 <= moment_index < len(self._moments):\n return None\n for op in self._moments[moment_index].operations:\n if qubit in op.qubits:\n return op\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the locations of all operations that satisfy a given condition.", "response": "def findall_operations(self, predicate: Callable[[ops.Operation], bool]\n ) -> Iterable[Tuple[int, ops.Operation]]:\n \"\"\"Find the locations of all operations that satisfy a given condition.\n\n This returns an iterator of (index, operation) tuples where each\n operation satisfies op_cond(operation) is truthy. The indices are\n in order of the moments and then order of the ops within that moment.\n\n Args:\n predicate: A method that takes an Operation and returns a Truthy\n value indicating the operation meets the find condition.\n\n Returns:\n An iterator (index, operation)'s that satisfy the op_condition.\n \"\"\"\n for index, moment in enumerate(self._moments):\n for op in moment.operations:\n if predicate(op):\n yield index, op"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind the locations of all gate operations of a given type.", "response": "def findall_operations_with_gate_type(\n self,\n gate_type: Type[T_DESIRED_GATE_TYPE]\n ) -> Iterable[Tuple[int,\n ops.GateOperation,\n T_DESIRED_GATE_TYPE]]:\n \"\"\"Find the locations of all gate operations of a given type.\n\n Args:\n gate_type: The type of gate to find, e.g. XPowGate or\n MeasurementGate.\n\n Returns:\n An iterator (index, operation, gate)'s for operations with the given\n gate type.\n \"\"\"\n result = self.findall_operations(lambda operation: bool(\n ops.op_gate_of_type(operation, gate_type)))\n for index, op in result:\n gate_op = cast(ops.GateOperation, op)\n yield index, gate_op, cast(T_DESIRED_GATE_TYPE, gate_op.gate)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef are_all_matches_terminal(self,\n predicate: Callable[[ops.Operation], bool]):\n \"\"\"Check whether all of the ops that satisfy a predicate are terminal.\n\n Args:\n predicate: A predicate on ops.Operations which is being checked.\n\n Returns:\n Whether or not all `Operation` s in a circuit that satisfy the\n given predicate are terminal.\n \"\"\"\n return all(\n self.next_moment_operating_on(op.qubits, i + 1) is None for\n (i, op) in self.findall_operations(predicate)\n )", "response": "Returns True if all of the operations in a circuit that satisfy a predicate are terminal."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetermine and prepares where an insertion should occur at the specified splitter index.", "response": "def _pick_or_create_inserted_op_moment_index(\n self, splitter_index: int, op: ops.Operation,\n strategy: InsertStrategy) -> int:\n \"\"\"Determines and prepares where an insertion will occur.\n\n Args:\n splitter_index: The index to insert at.\n op: The operation that will be inserted.\n strategy: The insertion strategy.\n\n Returns:\n The index of the (possibly new) moment where the insertion should\n occur.\n\n Raises:\n ValueError: Unrecognized append strategy.\n \"\"\"\n\n if (strategy is InsertStrategy.NEW or\n strategy is InsertStrategy.NEW_THEN_INLINE):\n self._moments.insert(splitter_index, ops.Moment())\n return splitter_index\n\n if strategy is InsertStrategy.INLINE:\n if (0 <= splitter_index - 1 < len(self._moments) and\n self._can_add_op_at(splitter_index - 1, op)):\n return splitter_index - 1\n\n return self._pick_or_create_inserted_op_moment_index(\n splitter_index, op, InsertStrategy.NEW)\n\n if strategy is InsertStrategy.EARLIEST:\n if self._can_add_op_at(splitter_index, op):\n p = self._prev_moment_available(op, splitter_index)\n return p or 0\n\n return self._pick_or_create_inserted_op_moment_index(\n splitter_index, op, InsertStrategy.INLINE)\n\n raise ValueError('Unrecognized append strategy: {}'.format(strategy))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninserting operations into the circuit.", "response": "def insert(\n self,\n index: int,\n moment_or_operation_tree: Union[ops.Moment, ops.OP_TREE],\n strategy: InsertStrategy = InsertStrategy.EARLIEST) -> int:\n \"\"\" Inserts operations into the circuit.\n Operations are inserted into the moment specified by the index and\n 'InsertStrategy'.\n Moments within the operation tree are inserted intact.\n\n Args:\n index: The index to insert all of the operations at.\n moment_or_operation_tree: The moment or operation tree to insert.\n strategy: How to pick/create the moment to put operations into.\n\n Returns:\n The insertion index that will place operations just after the\n operations that were inserted by this method.\n\n Raises:\n ValueError: Bad insertion strategy.\n \"\"\"\n moments_and_operations = list(ops.flatten_op_tree(\n ops.transform_op_tree(moment_or_operation_tree,\n self._device.decompose_operation,\n preserve_moments=True),\n preserve_moments=True))\n\n for moment_or_op in moments_and_operations:\n if isinstance(moment_or_op, ops.Moment):\n self._device.validate_moment(cast(ops.Moment, moment_or_op))\n else:\n self._device.validate_operation(\n cast(ops.Operation, moment_or_op))\n\n # limit index to 0..len(self._moments), also deal with indices smaller 0\n k = max(min(index if index >= 0 else len(self._moments) + index,\n len(self._moments)), 0)\n for moment_or_op in moments_and_operations:\n if isinstance(moment_or_op, ops.Moment):\n self._moments.insert(k, moment_or_op)\n k += 1\n else:\n p = self._pick_or_create_inserted_op_moment_index(\n k, moment_or_op, strategy)\n while p >= len(self._moments):\n self._moments.append(ops.Moment())\n self._moments[p] = self._moments[p].with_operation(moment_or_op)\n self._device.validate_moment(self._moments[p])\n k = max(k, p + 1)\n if strategy is InsertStrategy.NEW_THEN_INLINE:\n strategy = InsertStrategy.INLINE\n return k"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef insert_into_range(self,\n operations: ops.OP_TREE,\n start: int,\n end: int) -> int:\n \"\"\"Writes operations inline into an area of the circuit.\n\n Args:\n start: The start of the range (inclusive) to write the\n given operations into.\n end: The end of the range (exclusive) to write the given\n operations into. If there are still operations remaining,\n new moments are created to fit them.\n operations: An operation or tree of operations to insert.\n\n Returns:\n An insertion index that will place operations after the operations\n that were inserted by this method.\n\n Raises:\n IndexError: Bad inline_start and/or inline_end.\n \"\"\"\n if not 0 <= start <= end <= len(self):\n raise IndexError('Bad insert indices: [{}, {})'.format(\n start, end))\n\n operations = list(ops.flatten_op_tree(operations))\n for op in operations:\n self._device.validate_operation(op)\n\n i = start\n op_index = 0\n while op_index < len(operations):\n op = operations[op_index]\n while i < end and not self._device.can_add_operation_into_moment(\n op, self._moments[i]):\n i += 1\n if i >= end:\n break\n self._moments[i] = self._moments[i].with_operation(op)\n op_index += 1\n\n if op_index >= len(operations):\n return end\n\n return self.insert(end, operations[op_index:])", "response": "Inserts operations inline into an area of the circuit."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _pick_inserted_ops_moment_indices(operations: Sequence[ops.Operation],\n start: int = 0,\n frontier: Dict[ops.Qid,\n int] = None\n ) -> Tuple[Sequence[int],\n Dict[ops.Qid, int]]:\n \"\"\"Greedily assigns operations to moments.\n\n Args:\n operations: The operations to assign to moments.\n start: The first moment to consider assignment to.\n frontier: The first moment to which an operation acting on a qubit\n can be assigned. Updated in place as operations are assigned.\n\n Returns:\n The frontier giving the index of the moment after the last one to\n which an operation that acts on each qubit is assigned. If a\n frontier was specified as an argument, this is the same object.\n \"\"\"\n if frontier is None:\n frontier = defaultdict(lambda: 0)\n moment_indices = []\n for op in operations:\n op_start = max(start, max(frontier[q] for q in op.qubits))\n moment_indices.append(op_start)\n for q in op.qubits:\n frontier[q] = max(frontier[q], op_start + 1)\n\n return moment_indices, frontier", "response": "Greedily assigns operations to moments."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninsert moments to separate two frontiers.", "response": "def _push_frontier(self,\n early_frontier: Dict[ops.Qid, int],\n late_frontier: Dict[ops.Qid, int],\n update_qubits: Iterable[ops.Qid] = None\n ) -> Tuple[int, int]:\n \"\"\"Inserts moments to separate two frontiers.\n\n After insertion n_new moments, the following holds:\n for q in late_frontier:\n early_frontier[q] <= late_frontier[q] + n_new\n for q in update_qubits:\n early_frontier[q] the identifies the same moment as before\n (but whose index may have changed if this moment is after\n those inserted).\n\n Args:\n early_frontier: The earlier frontier. For qubits not in the later\n frontier, this is updated to account for the newly inserted\n moments.\n late_frontier: The later frontier. This is not modified.\n update_qubits: The qubits for which to update early_frontier to\n account for the newly inserted moments.\n\n Returns:\n (index at which new moments were inserted, how many new moments\n were inserted) if new moments were indeed inserted. (0, 0)\n otherwise.\n \"\"\"\n if update_qubits is None:\n update_qubits = set(early_frontier).difference(late_frontier)\n n_new_moments = (max(early_frontier.get(q, 0) - late_frontier[q]\n for q in late_frontier)\n if late_frontier else 0)\n if n_new_moments > 0:\n insert_index = min(late_frontier.values())\n self._moments[insert_index:insert_index] = (\n [ops.Moment()] * n_new_moments)\n for q in update_qubits:\n if early_frontier.get(q, 0) > insert_index:\n early_frontier[q] += n_new_moments\n return insert_index, n_new_moments\n return (0, 0)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _insert_operations(self,\n operations: Sequence[ops.Operation],\n insertion_indices: Sequence[int]) -> None:\n \"\"\"Inserts operations at the specified moments. Appends new moments if\n necessary.\n\n Args:\n operations: The operations to insert.\n insertion_indices: Where to insert them, i.e. operations[i] is\n inserted into moments[insertion_indices[i].\n\n Raises:\n ValueError: operations and insert_indices have different lengths.\n\n NB: It's on the caller to ensure that the operations won't conflict\n with operations already in the moment or even each other.\n \"\"\"\n if len(operations) != len(insertion_indices):\n raise ValueError('operations and insertion_indices must have the'\n 'same length.')\n self._moments += [\n ops.Moment() for _ in range(1 + max(insertion_indices) - len(self))\n ]\n moment_to_ops = defaultdict(list\n ) # type: Dict[int, List[ops.Operation]]\n for op_index, moment_index in enumerate(insertion_indices):\n moment_to_ops[moment_index].append(operations[op_index])\n for moment_index, new_ops in moment_to_ops.items():\n self._moments[moment_index] = ops.Moment(\n self._moments[moment_index].operations + tuple(new_ops))", "response": "Inserts operations at the specified moments. Appends new moments if necessary."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef insert_at_frontier(self,\n operations: ops.OP_TREE,\n start: int,\n frontier: Dict[ops.Qid, int] = None\n ) -> Dict[ops.Qid, int]:\n \"\"\"Inserts operations inline at frontier.\n\n Args:\n operations: the operations to insert\n start: the moment at which to start inserting the operations\n frontier: frontier[q] is the earliest moment in which an operation\n acting on qubit q can be placed.\n \"\"\"\n if frontier is None:\n frontier = defaultdict(lambda: 0)\n operations = tuple(ops.flatten_op_tree(operations))\n if not operations:\n return frontier\n qubits = set(q for op in operations for q in op.qubits)\n if any(frontier[q] > start for q in qubits):\n raise ValueError('The frontier for qubits on which the operations'\n 'to insert act cannot be after start.')\n\n next_moments = self.next_moments_operating_on(qubits, start)\n\n insertion_indices, _ = self._pick_inserted_ops_moment_indices(\n operations, start, frontier)\n\n self._push_frontier(frontier, next_moments)\n\n self._insert_operations(operations, insertion_indices)\n\n return frontier", "response": "Inserts operations inline at frontier."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving several operations from a circuit.", "response": "def batch_remove(self,\n removals: Iterable[Tuple[int, ops.Operation]]) -> None:\n \"\"\"Removes several operations from a circuit.\n\n Args:\n removals: A sequence of (moment_index, operation) tuples indicating\n operations to delete from the moments that are present. All\n listed operations must actually be present or the edit will\n fail (without making any changes to the circuit).\n\n ValueError:\n One of the operations to delete wasn't present to start with.\n\n IndexError:\n Deleted from a moment that doesn't exist.\n \"\"\"\n copy = self.copy()\n for i, op in removals:\n if op not in copy._moments[i].operations:\n raise ValueError(\n \"Can't remove {} @ {} because it doesn't exist.\".format(\n op, i))\n copy._moments[i] = ops.Moment(\n old_op\n for old_op in copy._moments[i].operations\n if op != old_op)\n self._device.validate_circuit(copy)\n self._moments = copy._moments"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninserting operations into empty spaces in existing moments.", "response": "def batch_insert_into(self,\n insert_intos: Iterable[Tuple[int, ops.Operation]]\n ) -> None:\n \"\"\"Inserts operations into empty spaces in existing moments.\n\n If any of the insertions fails (due to colliding with an existing\n operation), this method fails without making any changes to the circuit.\n\n Args:\n insert_intos: A sequence of (moment_index, new_operation)\n pairs indicating a moment to add a new operation into.\n\n ValueError:\n One of the insertions collided with an existing operation.\n\n IndexError:\n Inserted into a moment index that doesn't exist.\n \"\"\"\n copy = self.copy()\n for i, op in insert_intos:\n copy._moments[i] = copy._moments[i].with_operation(op)\n self._device.validate_circuit(copy)\n self._moments = copy._moments"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\napply a batched insert operation to the circuit.", "response": "def batch_insert(self,\n insertions: Iterable[Tuple[int, ops.OP_TREE]]) -> None:\n \"\"\"Applies a batched insert operation to the circuit.\n\n Transparently handles the fact that earlier insertions may shift\n the index that later insertions should occur at. For example, if you\n insert an operation at index 2 and at index 4, but the insert at index 2\n causes a new moment to be created, then the insert at \"4\" will actually\n occur at index 5 to account for the shift from the new moment.\n\n All insertions are done with the strategy 'EARLIEST'.\n\n When multiple inserts occur at the same index, the gates from the later\n inserts end up before the gates from the earlier inserts (exactly as if\n you'd called list.insert several times with the same index: the later\n inserts shift the earliest inserts forward).\n\n Args:\n insertions: A sequence of (insert_index, operations) pairs\n indicating operations to add into the circuit at specific\n places.\n \"\"\"\n # Work on a copy in case validation fails halfway through.\n copy = self.copy()\n shift = 0\n # Note: python `sorted` is guaranteed to be stable. This matters.\n insertions = sorted(insertions, key=lambda e: e[0])\n groups = _group_until_different(insertions,\n key=lambda e: e[0],\n value=lambda e: e[1])\n for i, group in groups:\n insert_index = i + shift\n next_index = copy.insert(insert_index,\n reversed(group),\n InsertStrategy.EARLIEST)\n if next_index > insert_index:\n shift += next_index - insert_index\n self._moments = copy._moments"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef append(\n self,\n moment_or_operation_tree: Union[ops.Moment, ops.OP_TREE],\n strategy: InsertStrategy = InsertStrategy.EARLIEST):\n \"\"\"Appends operations onto the end of the circuit.\n\n Moments within the operation tree are appended intact.\n\n Args:\n moment_or_operation_tree: The moment or operation tree to append.\n strategy: How to pick/create the moment to put operations into.\n \"\"\"\n self.insert(len(self._moments), moment_or_operation_tree, strategy)", "response": "Appends operations onto the end of the circuit."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nclears operations that are touching given qubits at given moments.", "response": "def clear_operations_touching(self,\n qubits: Iterable[ops.Qid],\n moment_indices: Iterable[int]):\n \"\"\"Clears operations that are touching given qubits at given moments.\n\n Args:\n qubits: The qubits to check for operations on.\n moment_indices: The indices of moments to check for operations\n within.\n \"\"\"\n qubits = frozenset(qubits)\n for k in moment_indices:\n if 0 <= k < len(self._moments):\n self._moments[k] = self._moments[k].without_operations_touching(\n qubits)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the set of all qubits acted upon by Operations in this circuit.", "response": "def all_qubits(self) -> FrozenSet[ops.Qid]:\n \"\"\"Returns the qubits acted upon by Operations in this circuit.\"\"\"\n return frozenset(q for m in self._moments for q in m.qubits)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\niterates over the operations applied by this circuit.", "response": "def all_operations(self) -> Iterator[ops.Operation]:\n \"\"\"Iterates over the operations applied by this circuit.\n\n Operations from earlier moments will be iterated over first. Operations\n within a moment are iterated in the order they were given to the\n moment's constructor.\n \"\"\"\n return (op for moment in self for op in moment.operations)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting the circuit into a unitary matrix if possible.", "response": "def _unitary_(self) -> Union[np.ndarray, NotImplementedType]:\n \"\"\"Converts the circuit into a unitary matrix, if possible.\n\n If the circuit contains any non-terminal measurements, the conversion\n into a unitary matrix fails (i.e. returns NotImplemented). Terminal\n measurements are ignored when computing the unitary matrix. The unitary\n matrix is the product of the unitary matrix of all operations in the\n circuit (after expanding them to apply to the whole system).\n \"\"\"\n if not self._has_unitary_():\n return NotImplemented\n return self.to_unitary_matrix(ignore_terminal_measurements=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert the given state to a unitary matrix.", "response": "def to_unitary_matrix(\n self,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n qubits_that_should_be_present: Iterable[ops.Qid] = (),\n ignore_terminal_measurements: bool = True,\n dtype: Type[np.number] = np.complex128) -> np.ndarray:\n \"\"\"Converts the circuit into a unitary matrix, if possible.\n\n Args:\n qubit_order: Determines how qubits are ordered when passing matrices\n into np.kron.\n qubits_that_should_be_present: Qubits that may or may not appear\n in operations within the circuit, but that should be included\n regardless when generating the matrix.\n ignore_terminal_measurements: When set, measurements at the end of\n the circuit are ignored instead of causing the method to\n fail.\n dtype: The numpy dtype for the returned unitary. Defaults to\n np.complex128. Specifying np.complex64 will run faster at the\n cost of precision. `dtype` must be a complex np.dtype, unless\n all operations in the circuit have unitary matrices with\n exclusively real coefficients (e.g. an H + TOFFOLI circuit).\n\n Returns:\n A (possibly gigantic) 2d numpy array corresponding to a matrix\n equivalent to the circuit's effect on a quantum state.\n\n Raises:\n ValueError: The circuit contains measurement gates that are not\n ignored.\n TypeError: The circuit contains gates that don't have a known\n unitary matrix, e.g. gates parameterized by a Symbol.\n \"\"\"\n\n if not ignore_terminal_measurements and any(\n protocols.is_measurement(op)\n for op in self.all_operations()):\n raise ValueError('Circuit contains a measurement.')\n\n if not self.are_all_measurements_terminal():\n raise ValueError('Circuit contains a non-terminal measurement.')\n\n qs = ops.QubitOrder.as_qubit_order(qubit_order).order_for(\n self.all_qubits().union(qubits_that_should_be_present))\n n = len(qs)\n\n state = np.eye(1 << n, dtype=np.complex128)\n state.shape = (2,) * (2 * n)\n\n result = _apply_unitary_circuit(self, state, qs, dtype)\n return result.reshape((1 << n, 1 << n))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a TextDiagramDrawer instance with the circuit drawn into it.", "response": "def to_text_diagram_drawer(\n self,\n *,\n use_unicode_characters: bool = True,\n qubit_namer: Optional[Callable[[ops.Qid], str]] = None,\n transpose: bool = False,\n precision: Optional[int] = 3,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n get_circuit_diagram_info:\n Optional[Callable[[ops.Operation,\n protocols.CircuitDiagramInfoArgs],\n protocols.CircuitDiagramInfo]]=None\n ) -> TextDiagramDrawer:\n \"\"\"Returns a TextDiagramDrawer with the circuit drawn into it.\n\n Args:\n use_unicode_characters: Determines if unicode characters are\n allowed (as opposed to ascii-only diagrams).\n qubit_namer: Names qubits in diagram. Defaults to str.\n transpose: Arranges qubit wires vertically instead of horizontally.\n precision: Number of digits to use when representing numbers.\n qubit_order: Determines how qubits are ordered in the diagram.\n get_circuit_diagram_info: Gets circuit diagram info. Defaults to\n protocol with fallback.\n\n Returns:\n The TextDiagramDrawer instance.\n \"\"\"\n qubits = ops.QubitOrder.as_qubit_order(qubit_order).order_for(\n self.all_qubits())\n qubit_map = {qubits[i]: i for i in range(len(qubits))}\n\n if qubit_namer is None:\n qubit_namer = lambda q: str(q) + ('' if transpose else ': ')\n diagram = TextDiagramDrawer()\n for q, i in qubit_map.items():\n diagram.write(0, i, qubit_namer(q))\n\n moment_groups = [] # type: List[Tuple[int, int]]\n for moment in self._moments:\n _draw_moment_in_diagram(moment,\n use_unicode_characters,\n qubit_map,\n diagram,\n precision,\n moment_groups,\n get_circuit_diagram_info)\n\n w = diagram.width()\n for i in qubit_map.values():\n diagram.horizontal_line(i, 0, w)\n\n if moment_groups:\n _draw_moment_groups_in_diagram(moment_groups,\n use_unicode_characters,\n diagram)\n\n if transpose:\n diagram = diagram.transpose()\n\n return diagram"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a QASM object equivalent to the circuit.", "response": "def _to_qasm_output(\n self,\n header: Optional[str] = None,\n precision: int = 10,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n ) -> QasmOutput:\n \"\"\"Returns a QASM object equivalent to the circuit.\n\n Args:\n header: A multi-line string that is placed in a comment at the top\n of the QASM. Defaults to a cirq version specifier.\n precision: Number of digits to use when representing numbers.\n qubit_order: Determines how qubits are ordered in the QASM\n register.\n \"\"\"\n if header is None:\n header = 'Generated from Cirq v{}'.format(\n cirq._version.__version__)\n qubits = ops.QubitOrder.as_qubit_order(qubit_order).order_for(\n self.all_qubits())\n return QasmOutput(operations=self.all_operations(),\n qubits=qubits,\n header=header,\n precision=precision,\n version='2.0')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the QASM equivalent to the circuit.", "response": "def to_qasm(self,\n header: Optional[str] = None,\n precision: int = 10,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n ) -> str:\n \"\"\"Returns QASM equivalent to the circuit.\n\n Args:\n header: A multi-line string that is placed in a comment at the top\n of the QASM. Defaults to a cirq version specifier.\n precision: Number of digits to use when representing numbers.\n qubit_order: Determines how qubits are ordered in the QASM\n register.\n \"\"\"\n return str(self._to_qasm_output(header, precision, qubit_order))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving a QASM file equivalent to the circuit.", "response": "def save_qasm(self,\n file_path: Union[str, bytes, int],\n header: Optional[str] = None,\n precision: int = 10,\n qubit_order: ops.QubitOrderOrList = ops.QubitOrder.DEFAULT,\n ) -> None:\n \"\"\"Save a QASM file equivalent to the circuit.\n\n Args:\n file_path: The location of the file where the qasm will be written.\n header: A multi-line string that is placed in a comment at the top\n of the QASM. Defaults to a cirq version specifier.\n precision: Number of digits to use when representing numbers.\n qubit_order: Determines how qubits are ordered in the QASM\n register.\n \"\"\"\n self._to_qasm_output(header, precision, qubit_order).save(file_path)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef qasm(val: Any,\n *,\n args: Optional[QasmArgs] = None,\n qubits: Optional[Iterable['cirq.Qid']] = None,\n default: TDefault = RaiseTypeErrorIfNotProvided\n ) -> Union[str, TDefault]:\n \"\"\"Returns QASM code for the given value, if possible.\n\n Different values require different sets of arguments. The general rule of\n thumb is that circuits don't need any, operations need a `QasmArgs`, and\n gates need both a `QasmArgs` and `qubits`.\n\n Args:\n val: The value to turn into QASM code.\n args: A `QasmArgs` object to pass into the value's `_qasm_` method.\n This is for needed for objects that only have a local idea of what's\n going on, e.g. a `cirq.Operation` in a bigger `cirq.Circuit`\n involving qubits that the operation wouldn't otherwise know about.\n qubits: A list of qubits that the value is being applied to. This is\n needed for `cirq.Gate` values, which otherwise wouldn't know what\n qubits to talk about.\n default: A default result to use if the value doesn't have a\n `_qasm_` method or that method returns `NotImplemented` or\n `None`. If not specified, undecomposable values cause a `TypeError`.\n\n Returns:\n The result of `val._qasm_(...)`, if `val` has a `_qasm_`\n method and it didn't return `NotImplemented` or `None`. Otherwise\n `default` is returned, if it was specified. Otherwise an error is\n raised.\n\n TypeError:\n `val` didn't have a `_qasm_` method (or that method returned\n `NotImplemented` or `None`) and `default` wasn't set.\n \"\"\"\n method = getattr(val, '_qasm_', None)\n result = NotImplemented\n if method is not None:\n kwargs = {} # type: Dict[str, Any]\n if args is not None:\n kwargs['args'] = args\n if qubits is not None:\n kwargs['qubits'] = tuple(qubits)\n result = method(**kwargs)\n if result is not None and result is not NotImplemented:\n return result\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n if method is None:\n raise TypeError(\"object of type '{}' \"\n \"has no _qasm_ method.\".format(type(val)))\n raise TypeError(\"object of type '{}' does have a _qasm_ method, \"\n \"but it returned NotImplemented or None.\".format(type(val)))", "response": "Returns the QASM code for the given value."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format_field(self, value: Any, spec: str) -> str:\n from cirq import ops # HACK: avoids cyclic dependency.\n if isinstance(value, (float, int)):\n if isinstance(value, float):\n value = round(value, self.precision)\n if spec == 'half_turns':\n value = 'pi*{}'.format(value) if value != 0 else '0'\n spec = ''\n elif isinstance(value, ops.Qid):\n value = self.qubit_id_map[value]\n elif isinstance(value, str) and spec == 'meas':\n value = self.meas_key_id_map[value]\n spec = ''\n return super().format_field(value, spec)", "response": "Method of string. Formatter that specifies the output of format_field."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates the oracle of the given input_qubits and outputs the result.", "response": "def make_oracle(input_qubits,\n output_qubit,\n secret_factor_bits,\n secret_bias_bit):\n \"\"\"Gates implementing the function f(a) = a\u00b7factors + bias (mod 2).\"\"\"\n\n if secret_bias_bit:\n yield cirq.X(output_qubit)\n\n for qubit, bit in zip(input_qubits, secret_factor_bits):\n if bit:\n yield cirq.CNOT(qubit, output_qubit)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef make_bernstein_vazirani_circuit(input_qubits, output_qubit, oracle):\n\n c = cirq.Circuit()\n\n # Initialize qubits.\n c.append([\n cirq.X(output_qubit),\n cirq.H(output_qubit),\n cirq.H.on_each(*input_qubits),\n ])\n\n # Query oracle.\n c.append(oracle)\n\n # Measure in X basis.\n c.append([\n cirq.H.on_each(*input_qubits),\n cirq.measure(*input_qubits, key='result')\n ])\n\n return c", "response": "Creates a Bernstein Vazirani circuit."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the big - endian integers specified by groups of bits.", "response": "def _tuple_of_big_endian_int(bit_groups: Tuple[np.ndarray, ...]\n ) -> Tuple[int, ...]:\n \"\"\"Returns the big-endian integers specified by groups of bits.\n\n Args:\n bit_groups: Groups of descending bits, each specifying a big endian\n integer with the 1s bit at the end.\n\n Returns:\n A tuple containing the integer for each group.\n \"\"\"\n return tuple(_big_endian_int(bits) for bits in bit_groups)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _big_endian_int(bits: np.ndarray) -> int:\n result = 0\n for e in bits:\n result <<= 1\n if e:\n result |= 1\n return result", "response": "Returns the big - endian integer specified by the given bits."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_oracle(q0, q1, secret_function):\n\n # coverage: ignore\n if secret_function[0]:\n yield [CNOT(q0, q1), X(q1)]\n\n if secret_function[1]:\n yield CNOT(q0, q1)", "response": "Yields the oracle of the secret function f."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef chip_as_adjacency_list(device: 'cirq.google.XmonDevice',\n ) -> Dict[GridQubit, List[GridQubit]]:\n \"\"\"Gives adjacency list representation of a chip.\n\n The adjacency list is constructed in order of above, left_of, below and\n right_of consecutively.\n\n Args:\n device: Chip to be converted.\n\n Returns:\n Map from nodes to list of qubits which represent all the neighbours of\n given qubit.\n \"\"\"\n c_set = set(device.qubits)\n c_adj = {} # type: Dict[GridQubit, List[GridQubit]]\n for n in device.qubits:\n c_adj[n] = []\n for m in [above(n), left_of(n), below(n), right_of(n)]:\n if m in c_set:\n c_adj[n].append(m)\n return c_adj", "response": "Converts a chip into an adjacency list representation of a chip."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef hhl_circuit(A, C, t, register_size, *input_prep_gates):\n\n ancilla = cirq.GridQubit(0, 0)\n # to store eigenvalues of the matrix\n register = [cirq.GridQubit(i+1, 0) for i in range(register_size)]\n # to store input and output vectors\n memory = cirq.GridQubit(register_size+1, 0)\n\n c = cirq.Circuit()\n hs = HamiltonianSimulation(A, t)\n pe = PhaseEstimation(register_size+1, hs)\n c.append([gate(memory) for gate in input_prep_gates])\n c.append([\n pe(*(register + [memory])),\n EigenRotation(register_size+1, C, t)(*(register+[ancilla])),\n pe(*(register + [memory]))**-1,\n cirq.measure(ancilla)\n ])\n\n # Pauli observable display\n c.append([\n cirq.pauli_string_expectation(\n cirq.PauliString({ancilla: cirq.Z}),\n key='a'\n ),\n cirq.pauli_string_expectation(\n cirq.PauliString({memory: cirq.X}),\n key='x'\n ),\n cirq.pauli_string_expectation(\n cirq.PauliString({memory: cirq.Y}),\n key='y'\n ),\n cirq.pauli_string_expectation(\n cirq.PauliString({memory: cirq.Z}),\n key='z'\n ),\n ])\n\n return c", "response": "Constructs the HHL circuit."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef main():\n\n # Eigendecomposition:\n # (4.537, [-0.971555, -0.0578339+0.229643j])\n # (0.349, [-0.236813, 0.237270-0.942137j])\n # |b> = (0.64510-0.47848j, 0.35490-0.47848j)\n # |x> = (-0.0662724-0.214548j, 0.784392-0.578192j)\n A = np.array([[4.30213466-6.01593490e-08j,\n 0.23531802+9.34386156e-01j],\n [0.23531882-9.34388383e-01j,\n 0.58386534+6.01593489e-08j]])\n t = 0.358166*math.pi\n register_size = 4\n input_prep_gates = [cirq.Rx(1.276359), cirq.Rz(1.276359)]\n expected = (0.144130, 0.413217, -0.899154)\n\n # Set C to be the smallest eigenvalue that can be represented by the\n # circuit.\n C = 2*math.pi / (2**register_size * t)\n\n # Simulate circuit\n print(\"Expected observable outputs:\")\n print(\"X =\", expected[0])\n print(\"Y =\", expected[1])\n print(\"Z =\", expected[2])\n print(\"Actual: \")\n simulate(hhl_circuit(A, C, t, register_size, *input_prep_gates))", "response": "Simulates HHL with matrix input and outputs Pauli observables of the\n resulting qubit state x."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrunning the sweeps for the given set of time - related objects.", "response": "def _run(\n self,\n circuit: circuits.Circuit,\n param_resolver: study.ParamResolver,\n repetitions: int,\n ) -> Dict[str, List[np.ndarray]]:\n \"\"\"See definition in `cirq.SimulatesSamples`.\"\"\"\n\n circuit = protocols.resolve_parameters(circuit, param_resolver)\n _verify_xmon_circuit(circuit)\n\n # Delegate to appropriate method based on contents.\n if circuit.are_all_measurements_terminal():\n return self._run_sweep_sample(circuit, repetitions)\n else:\n return self._run_sweep_repeat(circuit, repetitions)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _base_iterator(\n self,\n circuit: circuits.Circuit,\n qubit_order: ops.QubitOrderOrList,\n initial_state: Union[int, np.ndarray],\n perform_measurements: bool=True,\n ) -> Iterator['XmonStepResult']:\n \"\"\"See definition in `cirq.SimulatesIntermediateState`.\n\n If the initial state is an int, the state is set to the computational\n basis state corresponding to this state. Otherwise if the initial\n state is a np.ndarray it is the full initial state. In this case it\n must be the correct size, be normalized (an L2 norm of 1), and\n be safely castable to an appropriate dtype for the simulator.\n \"\"\"\n qubits = ops.QubitOrder.as_qubit_order(qubit_order).order_for(\n circuit.all_qubits())\n qubit_map = {q: i for i, q in enumerate(reversed(qubits))}\n if isinstance(initial_state, np.ndarray):\n initial_state = initial_state.astype(dtype=np.complex64,\n casting='safe')\n\n with xmon_stepper.Stepper(\n num_qubits=len(qubits),\n num_prefix_qubits=self.options.num_prefix_qubits,\n initial_state=initial_state,\n min_qubits_before_shard=self.options.min_qubits_before_shard,\n use_processes=self.options.use_processes\n ) as stepper:\n if len(circuit) == 0:\n yield XmonStepResult(stepper, qubit_map, {})\n for moment in circuit:\n measurements = collections.defaultdict(\n list) # type: Dict[str, List[bool]]\n phase_map = {} # type: Dict[Tuple[int, ...], float]\n for op in moment.operations:\n gate = cast(ops.GateOperation, op).gate\n if isinstance(gate, ops.ZPowGate):\n index = qubit_map[op.qubits[0]]\n phase_map[(index,)] = cast(float, gate.exponent)\n elif isinstance(gate, ops.CZPowGate):\n index0 = qubit_map[op.qubits[0]]\n index1 = qubit_map[op.qubits[1]]\n phase_map[(index0, index1)] = cast(float,\n gate.exponent)\n elif isinstance(gate, ops.XPowGate):\n index = qubit_map[op.qubits[0]]\n stepper.simulate_w(\n index=index,\n half_turns=gate.exponent,\n axis_half_turns=0)\n elif isinstance(gate, ops.YPowGate):\n index = qubit_map[op.qubits[0]]\n stepper.simulate_w(\n index=index,\n half_turns=gate.exponent,\n axis_half_turns=0.5)\n elif isinstance(gate, ops.PhasedXPowGate):\n index = qubit_map[op.qubits[0]]\n stepper.simulate_w(\n index=index,\n half_turns=gate.exponent,\n axis_half_turns=gate.phase_exponent)\n elif isinstance(gate, ops.MeasurementGate):\n if perform_measurements:\n invert_mask = (\n gate.invert_mask or len(op.qubits) * (False,))\n for qubit, invert in zip(op.qubits, invert_mask):\n index = qubit_map[qubit]\n result = stepper.simulate_measurement(index)\n if invert:\n result = not result\n key = protocols.measurement_key(gate)\n measurements[key].append(result)\n else:\n # coverage: ignore\n raise TypeError('{!r} is not supported by the '\n 'xmon simulator.'.format(gate))\n stepper.simulate_phases(phase_map)\n yield XmonStepResult(stepper, qubit_map, measurements)", "response": "A base iterator for the internal state of the system."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates the state of the simulator to the given state.", "response": "def set_state_vector(self, state: Union[int, np.ndarray]):\n \"\"\"Updates the state of the simulator to the given new state.\n\n Args:\n state: If this is an int, then this is the state to reset\n the stepper to, expressed as an integer of the computational basis.\n Integer to bitwise indices is little endian. Otherwise if this is\n a np.ndarray this must be the correct size and have dtype of\n np.complex64.\n\n Raises:\n ValueError if the state is incorrectly sized or not of the correct\n dtype.\n \"\"\"\n self._stepper.reset_state(state)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsampling from the wave function at this point in the computation.", "response": "def sample(self, qubits: List[ops.Qid], repetitions: int=1):\n \"\"\"Samples from the wave function at this point in the computation.\n\n Note that this does not collapse the wave function.\n\n Returns:\n Measurement results with True corresponding to the `|1>` state.\n The outer list is for repetitions, and the inner corresponds to\n measurements ordered by the supplied qubits.\n \"\"\"\n return self._stepper.sample_measurements(\n indices=[self.qubit_map[q] for q in qubits],\n repetitions=repetitions)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding possibly stateful noise to a series of moments.", "response": "def noisy_moments(self, moments: 'Iterable[cirq.Moment]',\n system_qubits: Sequence['cirq.Qid']\n ) -> Sequence['cirq.OP_TREE']:\n \"\"\"Adds possibly stateful noise to a series of moments.\n\n Args:\n moments: The moments to add noise to.\n system_qubits: A list of all qubits in the system.\n\n Returns:\n A sequence of OP_TREEs, with the k'th tree corresponding to the\n noisy operations for the k'th moment.\n \"\"\"\n if not hasattr(self.noisy_moment, '_not_overridden'):\n result = []\n for moment in moments:\n result.append(self.noisy_moment(moment, system_qubits))\n return result\n\n if not hasattr(self.noisy_operation, '_not_overridden'):\n result = []\n for moment in moments:\n result.append([self.noisy_operation(op) for op in moment])\n return result\n\n assert False, 'Should be unreachable.'"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds noise to the operations from a moment.", "response": "def noisy_moment(self, moment: 'cirq.Moment',\n system_qubits: Sequence['cirq.Qid']) -> 'cirq.OP_TREE':\n \"\"\"Adds noise to the operations from a moment.\n\n Args:\n moment: The moment to add noise to.\n system_qubits: A list of all qubits in the system.\n\n Returns:\n An OP_TREE corresponding to the noisy operations for the moment.\n \"\"\"\n if not hasattr(self.noisy_moments, '_not_overridden'):\n return self.noisy_moments([moment], system_qubits)\n\n if not hasattr(self.noisy_operation, '_not_overridden'):\n return [self.noisy_operation(op) for op in moment]\n\n assert False, 'Should be unreachable.'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd noise to an individual operation.", "response": "def noisy_operation(self, operation: 'cirq.Operation') -> 'cirq.OP_TREE':\n \"\"\"Adds noise to an individual operation.\n\n Args:\n operation: The operation to make noisy.\n\n Returns:\n An OP_TREE corresponding to the noisy operations implementing the\n noisy version of the given operation.\n \"\"\"\n if not hasattr(self.noisy_moments, '_not_overridden'):\n return self.noisy_moments([ops.Moment([operation])],\n operation.qubits)\n\n if not hasattr(self.noisy_moment, '_not_overridden'):\n return self.noisy_moment(ops.Moment([operation]), operation.qubits)\n\n assert False, 'Should be unreachable.'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the measurement key for the given value.", "response": "def measurement_key(\n val: Any,\n default: Any = RaiseTypeErrorIfNotProvided):\n \"\"\"Get the measurement key for the given value.\n\n Args:\n val: The value which has the measurement key..\n default: Determines the fallback behavior when `val` doesn't have\n a measurement key. If `default` is not set, a TypeError is raised.\n If default is set to a value, that value is returned if the value\n does not have `_measurement_key_`.\n\n Returns:\n If `val` has a `_measurement_key_` method and its result is not\n `NotImplemented`, that result is returned. Otherwise, if a default\n value was specified, the default value is returned.\n\n Raises:\n TypeError: `val` doesn't have a _measurement_key_ method (or that method\n returned NotImplemented) and also no default value was specified.\n \"\"\"\n getter = getattr(val, '_measurement_key_', None)\n result = NotImplemented if getter is None else getter()\n if result is not NotImplemented:\n return result\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if getter is None:\n raise TypeError(\n \"object of type '{}' has no _measurement_key_ method.\"\n .format(type(val)))\n\n raise TypeError(\"object of type '{}' does have a _measurement_key_ method, \"\n \"but it returned NotImplemented.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a Google Random Circuits v2 for the given set of qubits.", "response": "def generate_supremacy_circuit_google_v2(qubits: Iterable[devices.GridQubit],\n cz_depth: int,\n seed: int) -> circuits.Circuit:\n \"\"\"\n Generates Google Random Circuits v2 as in github.com/sboixo/GRCS cz_v2.\n See also https://arxiv.org/abs/1807.10749\n\n Args:\n qubits: qubit grid in which to generate the circuit.\n cz_depth: number of layers with CZ gates.\n seed: seed for the random instance.\n\n Returns:\n A circuit corresponding to instance\n inst_{n_rows}x{n_cols}_{cz_depth+1}_{seed}\n\n The mapping of qubits is cirq.GridQubit(j,k) -> q[j*n_cols+k]\n (as in the QASM mapping)\n \"\"\"\n\n non_diagonal_gates = [ops.pauli_gates.X**(1/2), ops.pauli_gates.Y**(1/2)]\n rand_gen = random.Random(seed).random\n\n circuit = circuits.Circuit()\n\n # Add an initial moment of Hadamards\n circuit.append(ops.common_gates.H(qubit) for qubit in qubits)\n\n layer_index = 0\n if cz_depth:\n layer_index = _add_cz_layer(layer_index, circuit)\n # In the first moment, add T gates when possible\n for qubit in qubits:\n if not circuit.operation_at(qubit, 1):\n circuit.append(ops.common_gates.T(qubit),\n strategy=InsertStrategy.EARLIEST)\n\n for moment_index in range(2, cz_depth+1):\n layer_index = _add_cz_layer(layer_index, circuit)\n # Add single qubit gates in the same moment\n for qubit in qubits:\n if not circuit.operation_at(qubit, moment_index):\n last_op = circuit.operation_at(qubit, moment_index-1)\n if last_op:\n gate = cast(ops.GateOperation, last_op).gate\n # Add a random non diagonal gate after a CZ\n if gate == ops.CZ:\n circuit.append(_choice(rand_gen,\n non_diagonal_gates).on(qubit),\n strategy=InsertStrategy.EARLIEST)\n # Add a T gate after a non diagonal gate\n elif not gate == ops.T:\n circuit.append(ops.common_gates.T(qubit),\n strategy=InsertStrategy.EARLIEST)\n\n # Add a final moment of Hadamards\n circuit.append([ops.common_gates.H(qubit) for qubit in qubits],\n strategy=InsertStrategy.NEW_THEN_INLINE)\n\n return circuit"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a 2D supremacy circuit for a random instance of the class.", "response": "def generate_supremacy_circuit_google_v2_grid(n_rows: int, n_cols: int,\n cz_depth: int, seed: int\n ) -> circuits.Circuit:\n \"\"\"\n Generates Google Random Circuits v2 as in github.com/sboixo/GRCS cz_v2.\n See also https://arxiv.org/abs/1807.10749\n\n Args:\n n_rows: number of rows of a 2D lattice.\n n_cols: number of columns.\n cz_depth: number of layers with CZ gates.\n seed: seed for the random instance.\n\n Returns:\n A circuit corresponding to instance\n inst_{n_rows}x{n_cols}_{cz_depth+1}_{seed}\n\n The mapping of qubits is cirq.GridQubit(j,k) -> q[j*n_cols+k]\n (as in the QASM mapping)\n \"\"\"\n qubits = [devices.GridQubit(i, j) for i in range(n_rows)\n for j in range(n_cols)]\n return generate_supremacy_circuit_google_v2(qubits, cz_depth, seed)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef generate_supremacy_circuit_google_v2_bristlecone(n_rows: int,\n cz_depth: int, seed: int\n ) -> circuits.Circuit:\n \"\"\"\n Generates Google Random Circuits v2 in Bristlecone.\n See also https://arxiv.org/abs/1807.10749\n\n Args:\n n_rows: number of rows in a Bristlecone lattice.\n Note that we do not include single qubit corners.\n cz_depth: number of layers with CZ gates.\n seed: seed for the random instance.\n\n Returns:\n A circuit with given size and seed.\n \"\"\"\n def get_qubits(n_rows):\n def count_neighbors(qubits, qubit):\n \"\"\"Counts the qubits that the given qubit can interact with.\"\"\"\n possibles = [\n devices.GridQubit(qubit.row + 1, qubit.col),\n devices.GridQubit(qubit.row - 1, qubit.col),\n devices.GridQubit(qubit.row, qubit.col + 1),\n devices.GridQubit(qubit.row, qubit.col - 1),\n ]\n return len(list(e for e in possibles if e in qubits))\n\n assert 1 <= n_rows <= 11\n max_row = n_rows - 1\n dev = google.Bristlecone\n # we need a consistent order of qubits\n qubits = list(dev.qubits)\n qubits.sort()\n qubits = [q for q in qubits\n if q.row <= max_row and q.row + q.col < n_rows + 6\n and q.row - q.col < n_rows - 5]\n qubits = [q for q in qubits if count_neighbors(qubits, q) > 1]\n return qubits\n\n qubits = get_qubits(n_rows)\n return generate_supremacy_circuit_google_v2(qubits, cz_depth, seed)", "response": "Generates a supremacy circuit for a given number of rows in a Bristlecone lattice."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _choice(rand_gen: Callable[[], float], sequence: Sequence[T]) -> T:\n return sequence[int(rand_gen() * len(sequence))]", "response": "Choose a random element from a non - empty sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a CZ layer for the given set of qubits.", "response": "def _make_cz_layer(qubits: Iterable[devices.GridQubit], layer_index: int\n ) -> Iterable[ops.Operation]:\n \"\"\"\n Each layer index corresponds to a shift/transpose of this CZ pattern:\n\n \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf . . .\n\n \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf . . .\n\n \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf . . .\n\n \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf . . .\n\n \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf . . .\n\n \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf \u25cf \u25cf \u25cf\u2500\u2500\u2500\u25cf . . .\n . . . . . . . . .\n . . . . . . . . .\n . . . . . . . . .\n\n Labelled edges, showing the exact index-to-CZs mapping (mod 8):\n\n \u25cf\u25000\u2500\u25cf\u25002\u2500\u25cf\u25004\u2500\u25cf\u25006\u2500\u25cf\u25000\u2500. . .\n 3\u2502 7\u2502 3\u2502 7\u2502 3\u2502\n \u25cf\u25004\u2500\u25cf\u25006\u2500\u25cf\u25000\u2500\u25cf\u25002\u2500\u25cf\u25004\u2500. . .\n 1\u2502 5\u2502 1\u2502 5\u2502 1\u2502\n \u25cf\u25000\u2500\u25cf\u25002\u2500\u25cf\u25004\u2500\u25cf\u25006\u2500\u25cf\u25000\u2500. . .\n 7\u2502 3\u2502 7\u2502 3\u2502 7\u2502\n \u25cf\u25004\u2500\u25cf\u25006\u2500\u25cf\u25000\u2500\u25cf\u25002\u2500\u25cf\u25004\u2500. . .\n 5\u2502 1\u2502 5\u2502 1\u2502 5\u2502\n \u25cf\u25000\u2500\u25cf\u25002\u2500\u25cf\u25004\u2500\u25cf\u25006\u2500\u25cf\u25000\u2500. . .\n 3\u2502 7\u2502 3\u2502 7\u2502 3\u2502\n . . . . . .\n . . . . . .\n . . . . . .\n\n Note that, for small devices, some layers will be empty because the layer\n only contains edges not present on the device.\n \"\"\"\n\n # map to an internal layer index to match the cycle order of public circuits\n layer_index_map = [0, 3, 2, 1, 4, 7, 6, 5]\n internal_layer_index = layer_index_map[layer_index % 8]\n\n dir_row = internal_layer_index % 2\n dir_col = 1 - dir_row\n shift = (internal_layer_index >> 1) % 4\n\n for q in qubits:\n q2 = devices.GridQubit(q.row + dir_row, q.col + dir_col)\n if q2 not in qubits:\n continue # This edge isn't on the device.\n if (q.row * (2 - dir_row) + q.col * (2 - dir_col)) % 4 != shift:\n continue # No CZ along this edge for this layer.\n\n yield ops.common_gates.CZ(q, q2)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndetermine the effect of an operation on the given qubits and returns a matrix equivalent to the effect of the operation on the given qubits.", "response": "def _op_to_matrix(self,\n op: Optional[ops.Operation],\n qubits: Tuple[ops.Qid, ...]\n ) -> Optional[np.ndarray]:\n \"\"\"Determines the effect of an operation on the given qubits.\n\n If the operation is a 1-qubit operation on one of the given qubits,\n or a 2-qubit operation on both of the given qubits, and also the\n operation has a known matrix, then a matrix is returned. Otherwise None\n is returned.\n\n Args:\n op: The operation to understand.\n qubits: The qubits we care about. Order determines matrix tensor\n order.\n\n Returns:\n None, or else a matrix equivalent to the effect of the operation.\n \"\"\"\n q1, q2 = qubits\n\n matrix = protocols.unitary(op, None)\n if matrix is None:\n return None\n\n assert op is not None\n if op.qubits == qubits:\n return matrix\n if op.qubits == (q2, q1):\n return MergeInteractions._flip_kron_order(matrix)\n if op.qubits == (q1,):\n return np.kron(matrix, np.eye(2))\n if op.qubits == (q2,):\n return np.kron(np.eye(2), matrix)\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nscans the two qubit operations into a matrix.", "response": "def _scan_two_qubit_ops_into_matrix(\n self,\n circuit: circuits.Circuit,\n index: Optional[int],\n qubits: Tuple[ops.Qid, ...]\n ) -> Tuple[List[ops.Operation], List[int], np.ndarray]:\n \"\"\"Accumulates operations affecting the given pair of qubits.\n\n The scan terminates when it hits the end of the circuit, finds an\n operation without a known matrix, or finds an operation that interacts\n the given qubits with other qubits.\n\n Args:\n circuit: The circuit to scan for operations.\n index: The index to start scanning forward from.\n qubits: The pair of qubits we care about.\n\n Returns:\n A tuple containing:\n 0. The operations.\n 1. The moment indices those operations were on.\n 2. A matrix equivalent to the effect of the scanned operations.\n \"\"\"\n\n product = np.eye(4, dtype=np.complex128)\n all_operations = []\n touched_indices = []\n\n while index is not None:\n operations = list({circuit.operation_at(q, index) for q in qubits})\n op_data = [\n self._op_to_matrix(op, qubits)\n for op in operations\n if op is not None\n ]\n\n # Stop at any non-constant or non-local interaction.\n if any(e is None for e in op_data):\n break\n present_ops = [op for op in operations if op]\n present_op_data = cast(List[np.ndarray], op_data)\n\n for op_mat in present_op_data:\n product = np.dot(op_mat, product)\n all_operations.extend(present_ops)\n\n touched_indices.append(index)\n index = circuit.next_moment_operating_on(qubits, index + 1)\n\n return all_operations, touched_indices, product"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _flip_kron_order(mat4x4: np.ndarray) -> np.ndarray:\n result = np.array([[0] * 4] * 4, dtype=np.complex128)\n order = [0, 2, 1, 3]\n for i in range(4):\n for j in range(4):\n result[order[i], order[j]] = mat4x4[i, j]\n return result", "response": "Given mat4x4 returns M = sum of krons of a_i and b_i."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef control(controllee: Union['cirq.Gate', op_tree.OP_TREE],\n control_qubits: Sequence['cirq.Qid'] = None,\n default: Any = RaiseTypeErrorIfNotProvided) -> Any:\n \"\"\"Returns a Controlled version of the given value, if defined.\n\n Controllees define how to be controlled by defining a method\n controlled_by(self, control_qubits). Note that the method may return\n NotImplemented to indicate a particular controlling can't be done.\n\n Args:\n controllee: The gate, operation or iterable of operations to control.\n control_qubits: A list of Qids that would control this controllee.\n default: Determines the fallback behavior when `controllee` doesn't\n have a controlling defined. If `default` is not set and the\n fallback occurs, a TypeError is raised instead.\n\n Returns:\n If `controllee` has a controlled_by method that returns something\n besides NotImplemented, that result is returned. For an OP_TREE,\n transformation is applied at the leaf. Otherwise, if a default value\n was specified, the default value is returned.\n\n Raises:\n TypeError: `controllee` doesn't have a controlled_by method (or that\n method returned NotImplemented) and no `default` was specified.\n \"\"\"\n if control_qubits is None:\n control_qubits = []\n controller = getattr(controllee, 'controlled_by', None)\n result = NotImplemented if controller is None else controller(\n *control_qubits)\n if result is not NotImplemented:\n return result\n\n if isinstance(controllee, collections.Iterable):\n return op_tree.transform_op_tree(\n controllee,\n op_transformation=lambda op: control(op, control_qubits))\n\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n\n if controller is None:\n raise TypeError(\"object of type '{}' has no controlled_by \"\n \"method.\".format(type(controllee)))\n raise TypeError(\"object of type '{}' does have a controlled_by method, \"\n \"but it returned NotImplemented.\".format(type(controllee)))", "response": "Returns a new version of the given value if the given controlling is defined."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the inverse of the given value if defined.", "response": "def inverse(val: Any, default: Any = RaiseTypeErrorIfNotProvided) -> Any:\n \"\"\"Returns the inverse `val**-1` of the given value, if defined.\n\n An object can define an inverse by defining a __pow__(self, exponent) method\n that returns something besides NotImplemented when given the exponent -1.\n The inverse of iterables is by default defined to be the iterable's items,\n each inverted, in reverse order.\n\n Args:\n val: The value (or iterable of invertible values) to invert.\n default: Determines the fallback behavior when `val` doesn't have\n an inverse defined. If `default` is not set, a TypeError is raised.\n If `default` is set to a value, that value is returned.\n\n Returns:\n If `val` has a __pow__ method that returns something besides\n NotImplemented when given an exponent of -1, that result is returned.\n Otherwise, if `val` is iterable, the result is a tuple with the same\n items as `val` but in reverse order and with each item inverted.\n Otherwise, if a `default` argument was specified, it is returned.\n\n Raises:\n TypeError: `val` doesn't have a __pow__ method, or that method returned\n NotImplemented when given -1. Furthermore `val` isn't an\n iterable containing invertible items. Also, no `default` argument\n was specified.\n \"\"\"\n from cirq import ops # HACK: avoid circular import\n\n # Check if object defines an inverse via __pow__.\n raiser = getattr(val, '__pow__', None)\n result = NotImplemented if raiser is None else raiser(-1)\n if result is not NotImplemented:\n return result\n\n # Maybe it's an iterable of invertible items?\n # Note: we avoid str because 'a'[0] == 'a', which creates an infinite loop.\n if (isinstance(val, collections.Iterable) and\n not isinstance(val, (str, ops.Operation))):\n unique_indicator = [] # type: List[Any]\n results = tuple(inverse(e, unique_indicator) for e in val)\n if all(e is not unique_indicator for e in results):\n return results[::-1]\n\n # Can't invert.\n if default is not RaiseTypeErrorIfNotProvided:\n return default\n raise TypeError(\n \"object of type '{}' isn't invertible. \"\n \"It has no __pow__ method (or the method returned NotImplemented) \"\n \"and it isn't an iterable of invertible objects.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndecomposing a two - qubit operation into MS - single - qubit rotation gates.", "response": "def two_qubit_matrix_to_ion_operations(q0: ops.Qid,\n q1: ops.Qid,\n mat: np.ndarray,\n atol: float = 1e-8\n ) -> List[ops.Operation]:\n \"\"\"Decomposes a two-qubit operation into MS/single-qubit rotation gates.\n\n Args:\n q0: The first qubit being operated on.\n q1: The other qubit being operated on.\n mat: Defines the operation to apply to the pair of qubits.\n tolerance: A limit on the amount of error introduced by the\n construction.\n\n Returns:\n A list of operations implementing the matrix.\n \"\"\"\n kak = linalg.kak_decomposition(mat, atol=atol)\n operations = _kak_decomposition_to_operations(q0,\n q1, kak, atol)\n return _cleanup_operations(operations)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nyielding an XX interaction framed by the given operation.", "response": "def _parity_interaction(q0: ops.Qid,\n q1: ops.Qid,\n rads: float,\n atol: float,\n gate: Optional[ops.Gate] = None):\n \"\"\"Yields an XX interaction framed by the given operation.\"\"\"\n\n if abs(rads) < atol:\n return\n\n if gate is not None:\n g = cast(ops.Gate, gate)\n yield g.on(q0), g.on(q1)\n\n yield MS(-1 * rads).on(q0, q1)\n\n if gate is not None:\n g = protocols.inverse(gate)\n yield g.on(q0), g.on(q1)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _non_local_part(q0: ops.Qid,\n q1: ops.Qid,\n interaction_coefficients: Tuple[float, float, float],\n atol: float = 1e-8):\n \"\"\"Yields non-local operation of KAK decomposition.\"\"\"\n\n x, y, z = interaction_coefficients\n\n return [\n _parity_interaction(q0, q1, x, atol),\n _parity_interaction(q0, q1, y, atol, ops.Z ** -0.5),\n _parity_interaction(q0, q1, z, atol, ops.Y ** 0.5)]", "response": "Yields non - local part of KAK decomposition."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nattempting to resolve a Symbol or name or float to its assigned value.", "response": "def value_of(\n self,\n value: Union[sympy.Basic, float, str]\n ) -> Union[sympy.Basic, float]:\n \"\"\"Attempt to resolve a Symbol or name or float to its assigned value.\n\n If unable to resolve a sympy.Symbol, returns it unchanged.\n If unable to resolve a name, returns a sympy.Symbol with that name.\n\n Args:\n value: The sympy.Symbol or name or float to try to resolve into just\n a float.\n\n Returns:\n The value of the parameter as resolved by this resolver.\n \"\"\"\n if isinstance(value, str):\n return self.param_dict.get(value, sympy.Symbol(value))\n if isinstance(value, sympy.Basic):\n if sys.version_info.major < 3:\n # coverage: ignore\n # HACK: workaround https://github.com/sympy/sympy/issues/16087\n d = {k.encode(): v for k, v in self.param_dict.items()}\n v = value.subs(d)\n else:\n v = value.subs(self.param_dict)\n return v if v.free_symbols else float(v)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef value_equality(cls: type = None,\n *,\n unhashable: bool = False,\n distinct_child_types: bool = False,\n manual_cls: bool = False,\n approximate: bool = False\n ) -> Union[Callable[[type], type], type]:\n \"\"\"Implements __eq__/__ne__/__hash__ via a _value_equality_values_ method.\n\n _value_equality_values_ is a method that the decorated class must implement.\n\n _value_equality_approximate_values_ is a method that the decorated class\n might implement if special support for approximate equality is required.\n This is only used when approximate argument is set. When approximate\n argument is set and _value_equality_approximate_values_ is not defined,\n _value_equality_values_ values are used for approximate equality.\n For example, this can be used to compare periodic values like angles: the\n angle value can be wrapped with `PeriodicValue`. When returned as part of\n approximate values a special normalization will be done automatically to\n guarantee correctness.\n\n Note that the type of the decorated value is included as part of the value\n equality values. This is so that completely separate classes with identical\n equality values (e.g. a Point2D and a Vector2D) don't compare as equal.\n Further note that this means that child types of the decorated type will be\n considered equal to each other, though this behavior can be changed via\n the 'distinct_child_types` argument. The type logic is implemented behind\n the scenes by a `_value_equality_values_cls_` method added to the class.\n\n Args:\n cls: The type to decorate. Automatically passed in by python when using\n the @cirq.value_equality decorator notation on a class.\n unhashable: When set, the __hash__ method will be set to None instead of\n to a hash of the equality class and equality values. Useful for\n mutable types such as dictionaries.\n distinct_child_types: When set, classes that inherit from the decorated\n class will not be considered equal to it. Also, different child\n classes will not be considered equal to each other. Useful for when\n the decorated class is an abstract class or trait that is helping to\n define equality for many conceptually distinct concrete classes.\n manual_cls: When set, the method '_value_equality_values_cls_' must be\n implemented. This allows a new class to compare as equal to another\n existing class that is also using value equality, by having the new\n class return the existing class' type.\n Incompatible with `distinct_child_types`.\n approximate: When set, the decorated class will be enhanced with\n `_approx_eq_` implementation and thus start to support the\n `SupportsApproximateEquality` protocol.\n \"\"\"\n\n # If keyword arguments were specified, python invokes the decorator method\n # without a `cls` argument, then passes `cls` into the result.\n if cls is None:\n return lambda deferred_cls: value_equality(deferred_cls,\n unhashable=unhashable,\n manual_cls=manual_cls,\n distinct_child_types=\n distinct_child_types,\n approximate=approximate)\n\n if distinct_child_types and manual_cls:\n raise ValueError(\"'distinct_child_types' is \"\n \"incompatible with 'manual_cls\")\n\n values_getter = getattr(cls, '_value_equality_values_', None)\n if values_getter is None:\n raise TypeError('The @cirq.value_equality decorator requires a '\n '_value_equality_values_ method to be defined.')\n\n if distinct_child_types:\n setattr(cls, '_value_equality_values_cls_', lambda self: type(self))\n elif manual_cls:\n cls_getter = getattr(cls, '_value_equality_values_cls_', None)\n if cls_getter is None:\n raise TypeError('The @cirq.value_equality decorator requires a '\n '_value_equality_values_cls_ method to be defined '\n 'when \"manual_cls\" is set.')\n else:\n setattr(cls, '_value_equality_values_cls_', lambda self: cls)\n setattr(cls, '__hash__', None if unhashable else _value_equality_hash)\n setattr(cls, '__eq__', _value_equality_eq)\n setattr(cls, '__ne__', _value_equality_ne)\n\n if approximate:\n if not hasattr(cls, '_value_equality_approximate_values_'):\n setattr(cls, '_value_equality_approximate_values_', values_getter)\n setattr(cls, '_approx_eq_', _value_equality_approx_eq)\n\n return cls", "response": "Returns a function that can be used to compare two value types of a given class."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a set of interesting lines from a GNU unified diff format.", "response": "def diff_to_new_interesting_lines(unified_diff_lines: List[str]\n ) -> Dict[int, str]:\n \"\"\"\n Extracts a set of 'interesting' lines out of a GNU unified diff format.\n\n Format:\n gnu.org/software/diffutils/manual/html_node/Detailed-Unified.html\n\n @@ from-line-numbers to-line-numbers @@\n line-from-either-file\n ...\n @@ start,count start,count @@\n line-from-either-file\n ...\n @@ single start,count @@\n line-from-either-file\n ...\n Examples:\n Deleted line (5 is the deleted LOC, 7 is the guessed would-have-been loc\n in the updated file given other changes pushing the line around):\n @@ 5 7,0 @@\n - content-of-line\n\n Added line:\n @@ 5,0 7 @@\n + content-of-line\n\n Modified chunk:\n @@ 10,15 11,5 @@\n - removed-line\n + added-line\n ...\n\n Args:\n unified_diff_lines: Lines of output from git diff.\n Returns:\n A dictionary of \"touched lines\", with key equal to the line number and\n value equal to the reason the line was touched. Includes added lines\n and lines near changes (including removals).\n \"\"\"\n interesting_lines = dict()\n for diff_line in unified_diff_lines:\n # Parse the 'new file' range parts of the unified diff.\n if not diff_line.startswith('@@ '):\n continue\n change = diff_line[3:diff_line.index(' @@', 3)]\n new = change.split(' ')[1]\n start = int(new.split(',')[0])\n count = 1 if ',' not in new else int(new.split(',')[1])\n\n # The lines before and after a deletion should still be covered.\n if count == 0:\n for i in range(start, start + 2):\n interesting_lines[i] = 'is near a removal'\n else:\n for i in range(start, start + count):\n interesting_lines[i] = 'is new or changed'\n return interesting_lines"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of tuples of indices content and reason - for - including of those uncovered lines in the given file.", "response": "def get_incremental_uncovered_lines(abs_path: str,\n base_commit: str,\n actual_commit: Optional[str]\n ) -> List[Tuple[int, str, str]]:\n \"\"\"\n Uses git diff and the annotation files created by\n `pytest --cov-report annotate` to find touched but uncovered lines in the\n given file.\n\n Args:\n abs_path: The path of a file to look for uncovered lines in.\n base_commit: Old state to diff against.\n actual_commit: Current state. Use None to use local uncommitted files.\n\n Returns:\n A list of the indices, content, and reason-for-including of\n 'interesting' uncovered lines. An interesting uncovered line is one\n involved with the diff.\n \"\"\"\n # Deleted files don't have any lines that need to be covered.\n if not os.path.isfile(abs_path):\n return []\n\n unified_diff_lines_str = shell_tools.output_of(\n 'git',\n 'diff',\n '--unified=0',\n base_commit,\n actual_commit,\n '--',\n abs_path)\n unified_diff_lines = [e\n for e in unified_diff_lines_str.split('\\n')\n if e.strip()]\n\n touched_lines = diff_to_new_interesting_lines(unified_diff_lines)\n\n with open(abs_path, 'r') as actual_file:\n ignored_lines = determine_ignored_lines(actual_file.read())\n\n cover_path = abs_path + ',cover'\n has_cover_file = os.path.isfile(cover_path)\n content_file = cover_path if has_cover_file else abs_path\n with open(content_file, 'r') as annotated_coverage_file:\n return [(i, fix_line_from_coverage_file(line), touched_lines[i])\n for i, line in enumerate(annotated_coverage_file, start=1)\n if i in touched_lines and i not in ignored_lines\n if line_counts_as_uncovered(line, has_cover_file)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef line_content_counts_as_uncovered_manual(content: str) -> bool:\n # Omit empty lines.\n if not content:\n return False\n\n # Omit declarations.\n for keyword in ['def', 'class']:\n if content.startswith(keyword) and content.endswith(':'):\n return False\n\n # TODO: multiline comments, multiline strings, etc, etc.\n return True", "response": "Returns True if the line content counts as uncovered in the coverage report."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if the line counts as uncovered.", "response": "def line_counts_as_uncovered(line: str,\n is_from_cover_annotation_file: bool) -> bool:\n \"\"\"\n Args:\n line: The line of code (including coverage annotation).\n is_from_cover_annotation_file: Whether this line has been annotated.\n Returns:\n Does the line count as uncovered?\n \"\"\"\n\n # Ignore this line?\n if is_from_cover_annotation_file:\n # Already covered, or the tool deemed it not relevant for coverage.\n if not line.startswith('! '):\n return False\n\n content = line[2:]\n else:\n content = line\n\n # Ignore surrounding whitespace.\n content = content.strip()\n\n # Ignore end-of-line comments.\n # TODO: avoid # in strings, etc.\n if '#' in content:\n content = content[:content.index('#')].strip()\n\n # Ignored line pattern?\n if any(re.search(pat, content) for pat in IGNORED_LINE_PATTERNS):\n return False\n\n return (is_from_cover_annotation_file or\n line_content_counts_as_uncovered_manual(content))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_applicable_python_file(rel_path: str) -> bool:\n return (rel_path.endswith('.py') and\n not any(re.search(pat, rel_path) for pat in IGNORED_FILE_PATTERNS))", "response": "Determines if a file should be included in incremental coverage analysis."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the approximate common period of the given list of periods.", "response": "def _approximate_common_period(periods: List[float],\n approx_denom: int = 60,\n reject_atol: float = 1e-8) -> Optional[float]:\n \"\"\"Finds a value that is nearly an integer multiple of multiple periods.\n\n The returned value should be the smallest non-negative number with this\n property. If `approx_denom` is too small the computation can fail to satisfy\n the `reject_atol` criteria and return `None`. This is actually desirable\n behavior, since otherwise the code would e.g. return a nonsense value when\n asked to compute the common period of `np.e` and `np.pi`.\n\n Args:\n periods: The result must be an approximate integer multiple of each of\n these.\n approx_denom: Determines how the floating point values are rounded into\n rational values (so that integer methods such as lcm can be used).\n Each floating point value f_k will be rounded to a rational number\n of the form n_k / approx_denom. If you want to recognize rational\n periods of the form i/d then d should divide `approx_denom`.\n reject_atol: If the computed approximate common period is at least this\n far from an integer multiple of any of the given periods, then it\n is discarded and `None` is returned instead.\n\n Returns:\n The approximate common period, or else `None` if the given\n `approx_denom` wasn't sufficient to approximate the common period to\n within the given `reject_atol`.\n \"\"\"\n if not periods:\n return None\n if any(e == 0 for e in periods):\n return None\n if len(periods) == 1:\n return abs(periods[0])\n approx_rational_periods = [\n fractions.Fraction(int(np.round(abs(p) * approx_denom)), approx_denom)\n for p in periods\n ]\n common = float(_common_rational_period(approx_rational_periods))\n\n for p in periods:\n if p != 0 and abs(p * np.round(common / p) - common) > reject_atol:\n return None\n\n return common"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds the least common integer multiple of some fractions. rational_periods", "response": "def _common_rational_period(rational_periods: List[fractions.Fraction]\n ) -> fractions.Fraction:\n \"\"\"Finds the least common integer multiple of some fractions.\n\n The solution is the smallest positive integer c such that there\n exists integers n_k satisfying p_k * n_k = c for all k.\n \"\"\"\n assert rational_periods, \"no well-defined solution for an empty list\"\n common_denom = _lcm(p.denominator for p in rational_periods)\n int_periods = [p.numerator * common_denom // p.denominator\n for p in rational_periods]\n int_common_period = _lcm(int_periods)\n return fractions.Fraction(int_common_period, common_denom)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fit(self, X):\n X = column_or_1d(X, warn=True)\n _check_numpy_unicode_bug(X)\n self.classes_ = np.unique(X)\n return self", "response": "Fit label encoder to a set of target values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfitting label encoder and return encoded labels.", "response": "def fit_transform(self, X, y=None, **fit_params):\n \"\"\"Fit label encoder and return encoded labels\n\n Parameters\n ----------\n y : array-like of shape [n_samples]\n Target values.\n\n Returns\n -------\n y : array-like of shape [n_samples]\n \"\"\"\n y = column_or_1d(X, warn=True)\n _check_numpy_unicode_bug(X)\n self.classes_, X = np.unique(X, return_inverse=True)\n return X"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntransform labels to normalized encoding.", "response": "def transform(self, y):\n \"\"\"Transform labels to normalized encoding.\n\n Parameters\n ----------\n y : array-like of shape [n_samples]\n Target values.\n\n Returns\n -------\n y : array-like of shape [n_samples]\n \"\"\"\n check_is_fitted(self, 'classes_')\n y = column_or_1d(y, warn=True)\n\n classes = np.unique(y)\n _check_numpy_unicode_bug(classes)\n if len(np.intersect1d(classes, self.classes_)) < len(classes):\n diff = np.setdiff1d(classes, self.classes_)\n raise ValueError(\"y contains new labels: %s\" % str(diff))\n return np.searchsorted(self.classes_, y)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntransform labels back to original encoding.", "response": "def inverse_transform(self, y):\n \"\"\"Transform labels back to original encoding.\n\n Parameters\n ----------\n y : numpy array of shape [n_samples]\n Target values.\n\n Returns\n -------\n y : numpy array of shape [n_samples]\n \"\"\"\n check_is_fitted(self, 'classes_')\n\n diff = np.setdiff1d(y, np.arange(len(self.classes_)))\n if diff:\n raise ValueError(\"y contains new labels: %s\" % str(diff))\n y = np.asarray(y)\n return self.classes_[y]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntransform features per specified math function.", "response": "def transform(self, y):\n \"\"\"\n Transform features per specified math function.\n :param y:\n :return:\n \"\"\"\n if self.transform_type == 'log':\n return np.log(y)\n elif self.transform_type == 'exp':\n return np.exp(y)\n elif self.transform_type == 'sqrt':\n return np.sqrt(y)\n elif self.transform_type == 'sin':\n return np.sin(y)\n elif self.transform_type == 'cos':\n return np.cos(y)\n elif self.transform_type == 'tan':\n return np.tan(y)\n elif self.transform_type == 'abs':\n return np.abs(y)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fit(self, X, y=None, **fit_params):\n if self.transform_type not in self.valid_transforms:\n warnings.warn(\"Invalid transform type.\", stacklevel=2)\n return self", "response": "Fit a binary math operator on the data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntransforms features per specified math function.", "response": "def transform(self, y):\n \"\"\"\n Transform features per specified math function.\n :param y:\n :return:\n \"\"\"\n if isinstance(y, pd.DataFrame):\n x = y.ix[:,0]\n y = y.ix[:,1]\n else:\n x = y[:,0]\n y = y[:,1]\n if self.transform_type == 'add':\n return pd.DataFrame(np.add(x, y))\n elif self.transform_type == 'sub':\n return pd.DataFrame(np.subtract(x, y))\n elif self.transform_type == 'mul':\n return pd.DataFrame(np.multiply(x, y))\n elif self.transform_type == 'div':\n return pd.DataFrame(np.divide(x, y))\n elif self.transform_type == 'rem':\n return pd.DataFrame(np.remainder(x, y))\n elif self.transform_type == 'pow':\n return pd.DataFrame(x**y)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef serialize_tree(tree, feature_names, outfile):\n\n tree_ = tree.tree_\n feature_name = [feature_names[i] if i != _tree.TREE_UNDEFINED else 'n/a' for i in tree_.feature]\n\n def traverse(node, depth, outfile):\n if tree_.feature[node] != _tree.TREE_UNDEFINED:\n name = feature_name[node]\n threshold = tree_.threshold[node]\n\n # Define internal node for serialization\n internal_node = {\n 'type': 'internal',\n 'split': {\n 'type': 'continuous',\n 'featureIndex': feature_names.index(name),\n 'threshold': threshold\n }\n }\n\n # Serialize the internal Node\n json.dump(internal_node, outfile)\n outfile.write('\\n')\n\n # Traverse Left\n traverse(tree_.children_left[node], depth + 1, outfile)\n\n # Traverse Rigiht\n traverse(tree_.children_right[node], depth + 1, outfile)\n else:\n leaf_node = {\n 'type': 'leaf',\n 'values': tree_.value[node].tolist()[0]\n }\n\n # Serialize the leaf node\n json.dump(leaf_node, outfile)\n outfile.write('\\n')\n\n traverse(0, 1, outfile)", "response": "Serialize the tree to a file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nserializes the decision tree to a bundle.", "response": "def serialize_to_bundle(self, transformer, path, model_name, serialize_node=True):\n \"\"\"\n :type transformer: sklearn.tree.tree.BaseDecisionTree\n :type path: str\n :type model_name: str\n :type serialize_node: bool\n :param transformer:\n :param path:\n :param model_name:\n :return:\n \"\"\"\n\n # Define attributes\n attributes = list()\n attributes.append(('num_features', transformer.n_features_))\n if isinstance(transformer, DecisionTreeClassifier):\n attributes.append(('num_classes', int(transformer.n_classes_)))\n\n inputs = []\n outputs = []\n if serialize_node:\n # define node inputs and outputs\n inputs = [{\n \"name\": transformer.input_features,\n \"port\": \"features\"\n }]\n\n outputs = [{\n \"name\": transformer.prediction_column,\n \"port\": \"prediction\"\n }]\n\n self.serialize(transformer, path, model_name, attributes, inputs, outputs, node=serialize_node)\n\n # Serialize tree.json\n tree_path = \"{}/{}.node/tree.json\".format(path, model_name)\n if not serialize_node:\n tree_path = \"{}/{}/tree.json\".format(path, model_name)\n with open(tree_path, 'w') as outfile:\n self.serialize_tree(transformer, transformer.feature_names, outfile)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sent2vec(self, words, transformer):\n sent_vec = np.zeros(transformer.vector_size)\n numw = 0\n for w in words:\n try:\n sent_vec = np.add(sent_vec, transformer.wv[w])\n numw += 1\n except:\n continue\n return sent_vec / np.sqrt(sent_vec.dot(sent_vec))", "response": "Used with sqrt kernel\n Used with sqrt kernel\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef serialize_to_bundle(self, transformer, path, model):\n\n # Define Node Inputs and Outputs\n inputs = [{\n \"name\": transformer.input_features,\n \"port\": \"features\"\n }]\n\n outputs = list()\n outputs.append({\n \"name\": transformer.prediction_column,\n \"port\": \"prediction\"\n })\n\n outputs.append({\n \"name\": \"raw_prediction\",\n \"port\": \"raw_prediction\"\n })\n\n outputs.append({\n \"name\": \"probability\",\n \"port\": \"probability\"\n })\n\n # compile tuples of model attributes to serialize\n tree_weights = Vector([1.0 for x in range(0, len(transformer.estimators_))])\n attributes = list()\n attributes.append(('num_features', transformer.n_features_))\n attributes.append(('tree_weights', tree_weights))\n attributes.append(('trees', [\"tree{}\".format(x) for x in range(0, len(transformer.estimators_))]))\n if isinstance(transformer, RandomForestClassifier):\n attributes.append(('num_classes', transformer.n_classes_)) # TODO: get number of classes from the transformer\n\n self.serialize(transformer, path, model, attributes, inputs, outputs)\n\n rf_path = \"{}/{}.node\".format(path, model)\n\n estimators = transformer.estimators_\n\n i = 0\n for estimator in estimators:\n estimator.mlinit(input_features = transformer.input_features, prediction_column = transformer.prediction_column, feature_names=transformer.feature_names)\n model_name = \"tree{}\".format(i)\n estimator.serialize_to_bundle(rf_path, model_name, serialize_node=False)\n\n i += 1", "response": "This method serializes the model to a sklearn bundle."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_mleap_model(self, transformer, attributes_to_serialize=None):\n js = {\n 'op': transformer.op\n }\n\n # If the transformer doesn't have any attributes, return just the op name\n if attributes_to_serialize is None:\n return js\n\n attributes = {}\n\n for name, value in attributes_to_serialize:\n if isinstance(value, float):\n attributes[name] = {\n \"double\": value\n }\n\n elif isinstance(value, bool) and value in [True, False]:\n attributes[name] = {\n \"boolean\": value\n }\n\n elif isinstance(value, int):\n attributes[name] = {\n \"long\": value\n }\n elif isinstance(value, Vector):\n attributes[name] = {\n \"type\": \"list\",\n \"double\": value.values\n }\n elif isinstance(value, list) and (isinstance(value[0], np.float64) or isinstance(value[0], np.float32) or isinstance(value[0], float)):\n base = type(value[0])\n attributes[name] = {\n _type_map[base]: value,\n \"shape\": {\n \"dimensions\": [{\n \"size\": len(value),\n \"name\": \"\"\n }]\n },\n \"type\": \"tensor\"\n }\n elif isinstance(value, list) and isinstance(value[0], str):\n attributes[name] = {\n \"type\": \"list\",\n \"string\": value\n }\n\n elif isinstance(value, np.ndarray):\n attributes[name] = {\n \"double\": list(value.flatten()),\n \"shape\": {\n \"dimensions\": [{\n \"size\": list(value.shape),\n \"name\": \"\"\n }]\n },\n \"type\": \"tensor\"\n }\n\n elif isinstance(value, str):\n attributes[name] = {\n 'string': value\n }\n\n elif isinstance(value, dict):\n shapes = list()\n for shape in value['data_shape']:\n if shape['shape'] == 'scalar':\n shapes.append(({\"base\": \"scalar\",\n \"isNullable\": False}))\n elif shape['shape'] == 'tensor':\n shapes.append(({\"base\": \"tensor\",\n \"isNullable\": False,\n \"tensorShape\": {\n \"dimensions\": [{\n \"size\": shape['tensor_shape']['dimensions'][0]['size'],\n \"name\": \"\"\n }]\n }\n }\n ))\n attributes[name] = {\n 'type': 'list',\n 'data_shape': shapes\n }\n\n js['attributes'] = attributes\n\n return js", "response": "Generates the model. json given a list of attributes which are a tuple comprised of name value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef wait_until(what, times=-1):\n while times:\n logger.info('Waiting times left %d', times)\n try:\n if what() is True:\n return True\n except:\n logger.exception('Wait failed')\n else:\n logger.warning('Trial[%d] failed', times)\n times -= 1\n time.sleep(1)\n\n return False", "response": "Wait until what returns True."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _init_devices(self):\n if not settings.PDU_CONTROLLER_TYPE:\n if settings.AUTO_DUT:\n return\n\n for device in settings.GOLDEN_DEVICES:\n port, _ = device\n try:\n with OpenThreadController(port) as otc:\n logger.info('Resetting %s', port)\n otc.reset()\n except:\n logger.exception('Failed to reset device %s', port)\n self.history.mark_bad_golden_device(device)\n\n return\n\n tries = 3\n pdu_factory = PduControllerFactory()\n\n while True:\n try:\n pdu = pdu_factory.create_pdu_controller(settings.PDU_CONTROLLER_TYPE)\n pdu.open(**settings.PDU_CONTROLLER_OPEN_PARAMS)\n except EOFError:\n logger.warning('Failed to connect to telnet')\n tries = tries - 1\n if tries:\n time.sleep(10)\n continue\n else:\n logger.error('Fatal error: cannot connect to apc')\n raise\n else:\n pdu.reboot(**settings.PDU_CONTROLLER_REBOOT_PARAMS)\n pdu.close()\n break\n\n time.sleep(len(settings.GOLDEN_DEVICES))", "response": "Initialize all usb devices."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrestart the harness backend service.", "response": "def _init_harness(self):\n \"\"\"Restart harness backend service.\n\n Please start the harness controller before running the cases, otherwise, nothing happens\n \"\"\"\n self._hc = HarnessController(self.result_dir)\n self._hc.stop()\n time.sleep(1)\n self._hc.start()\n time.sleep(2)\n\n harness_config = ConfigParser.ConfigParser()\n harness_config.read('%s\\\\Config\\\\Configuration.ini' % settings.HARNESS_HOME)\n if harness_config.has_option('THREAD_HARNESS_CONFIG', 'BrowserAutoNavigate') and \\\n harness_config.getboolean('THREAD_HARNESS_CONFIG', 'BrowserAutoNavigate'):\n logger.error('BrowserAutoNavigate in Configuration.ini should be False')\n raise FailError('BrowserAutoNavigate in Configuration.ini should be False')\n if settings.MIXED_DEVICE_TYPE:\n if harness_config.has_option('THREAD_HARNESS_CONFIG', 'EnableDeviceSelection') and \\\n not harness_config.getboolean('THREAD_HARNESS_CONFIG', 'EnableDeviceSelection'):\n logger.error('EnableDeviceSelection in Configuration.ini should be True')\n raise FailError('EnableDeviceSelection in Configuration.ini should be True')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _init_dut(self):\n if self.auto_dut:\n self.dut = None\n return\n\n dut_port = settings.DUT_DEVICE[0]\n dut = OpenThreadController(dut_port)\n self.dut = dut", "response": "Initialize the DUT.\n\n DUT will be restarted. and openthread will started."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting up the Harness instance.", "response": "def setUp(self):\n \"\"\"Prepare to run test case.\n\n Start harness service, init golden devices, reset DUT and open browser.\n \"\"\"\n if self.__class__ is HarnessCase:\n return\n\n logger.info('Setting up')\n # clear files\n logger.info('Deleting all .pdf')\n os.system('del /q \"%HOMEDRIVE%%HOMEPATH%\\\\Downloads\\\\NewPdf_*.pdf\"')\n logger.info('Deleting all .xlsx')\n os.system('del /q \"%HOMEDRIVE%%HOMEPATH%\\\\Downloads\\\\ExcelReport*.xlsx\"')\n logger.info('Deleting all .pcapng')\n os.system('del /q \"%s\\\\Captures\\\\*.pcapng\"' % settings.HARNESS_HOME)\n\n # using temp files to fix excel downloading fail\n logger.info('Empty files in temps')\n os.system('del /q \"%s\\\\Thread_Harness\\\\temp\\\\*.*\"' % settings.HARNESS_HOME)\n\n # create directory\n os.system('mkdir %s' % self.result_dir)\n self._init_harness()\n self._init_devices()\n self._init_dut()\n self._init_rf_shield()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tearDown(self):\n if self.__class__ is HarnessCase:\n return\n\n logger.info('Tearing down')\n self._destroy_harness()\n self._destroy_browser()\n self._destroy_dut()\n self._destroy_rf_shield()", "response": "Clean up after each case."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _setup_page(self):\n if not self.started:\n self.started = time.time()\n\n if time.time() - self.started > 5*len(settings.GOLDEN_DEVICES):\n self._browser.refresh()\n return\n\n # Detect Sniffer\n try:\n dialog = self._browser.find_element_by_id('capture-Setup-modal')\n except:\n logger.exception('Failed to get dialog.')\n else:\n if dialog and dialog.get_attribute('aria-hidden') == 'false':\n times = 100\n while times:\n status = dialog.find_element_by_class_name('status-notify').text\n if 'Searching' in status:\n logger.info('Still detecting..')\n elif 'Not' in status:\n logger.warning('Sniffer device not verified!')\n button = dialog.find_element_by_id('snifferAutoDetectBtn')\n button.click()\n elif 'Verified' in status:\n logger.info('Verified!')\n button = dialog.find_element_by_id('saveCaptureSettings')\n button.click()\n break\n else:\n logger.warning('Unexpected sniffer verification status')\n\n times = times - 1\n time.sleep(1)\n\n if not times:\n raise Exception('Unable to detect sniffer device')\n\n time.sleep(1)\n\n try:\n skip_button = self._browser.find_element_by_id('SkipPrepareDevice')\n if skip_button.is_enabled():\n skip_button.click()\n time.sleep(1)\n except:\n logger.info('Still detecting sniffers')\n\n try:\n next_button = self._browser.find_element_by_id('nextButton')\n except:\n logger.exception('Failed to finish setup')\n return\n\n if not next_button.is_enabled():\n logger.info('Harness is still not ready')\n return\n\n # General Setup\n try:\n if self.child_timeout or self.sed_polling_interval:\n logger.info('finding general Setup button')\n button = self._browser.find_element_by_id('general-Setup')\n button.click()\n time.sleep(2)\n\n dialog = self._browser.find_element_by_id('general-Setup-modal')\n if dialog.get_attribute('aria-hidden') != 'false':\n raise Exception('Missing General Setup dialog')\n\n field = dialog.find_element_by_id('inp_general_child_update_wait_time')\n field.clear()\n if self.child_timeout:\n field.send_keys(str(self.child_timeout))\n\n field = dialog.find_element_by_id('inp_general_sed_polling_rate')\n field.clear()\n if self.sed_polling_interval:\n field.send_keys(str(self.sed_polling_interval))\n\n button = dialog.find_element_by_id('saveGeneralSettings')\n button.click()\n time.sleep(1)\n\n except:\n logger.info('general setup exception')\n logger.exception('Failed to do general setup')\n return\n\n # Finish this page\n next_button.click()\n time.sleep(1)", "response": "Do sniffer settings and general setup"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _select_case(self, role, case):\n # select the case\n elem = Select(self._browser.find_element_by_id('select-dut'))\n elem.select_by_value(str(role))\n time.sleep(1)\n\n checkbox = None\n wait_until(lambda: self._browser.find_elements_by_css_selector('.tree-node .tree-title') and True)\n elems = self._browser.find_elements_by_css_selector('.tree-node .tree-title')\n finder = re.compile(r'.*\\b' + case + r'\\b')\n finder_dotted = re.compile(r'.*\\b' + case.replace(' ', r'\\.') + r'\\b')\n for elem in elems:\n action_chains = ActionChains(self._browser)\n action_chains.move_to_element(elem)\n action_chains.perform()\n logger.debug(elem.text)\n if finder.match(elem.text) or finder_dotted.match(elem.text):\n parent = elem.find_element_by_xpath('..')\n checkbox = parent.find_element_by_class_name('tree-checkbox')\n break\n\n if not checkbox:\n time.sleep(5)\n raise Exception('Failed to find the case')\n\n self._browser.execute_script(\"$('.overview').css('left', '0')\")\n checkbox.click()\n time.sleep(1)\n\n elem = self._browser.find_element_by_id('runTest')\n elem.click()\n if not wait_until(lambda: self._browser.find_element_by_id('stopTest') and True, 10):\n raise Exception('Failed to start test case')", "response": "Select the test case."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncollect test result. Generate PDF, excel and pcap file", "response": "def _collect_result(self):\n \"\"\"Collect test result.\n\n Generate PDF, excel and pcap file\n \"\"\"\n # generate pdf\n self._browser.find_element_by_class_name('save-pdf').click()\n time.sleep(1)\n try:\n dialog = self._browser.find_element_by_id('Testinfo')\n except:\n logger.exception('Failed to get test info dialog.')\n else:\n if dialog.get_attribute('aria-hidden') != 'false':\n raise Exception('Test information dialog not ready')\n\n version = self.auto_dut and settings.DUT_VERSION or self.dut.version\n dialog.find_element_by_id('inp_dut_manufacturer').send_keys(settings.DUT_MANUFACTURER)\n dialog.find_element_by_id('inp_dut_firmware_version').send_keys(version)\n dialog.find_element_by_id('inp_tester_name').send_keys(settings.TESTER_NAME)\n dialog.find_element_by_id('inp_remarks').send_keys(settings.TESTER_REMARKS)\n dialog.find_element_by_id('generatePdf').click()\n\n time.sleep(1)\n main_window = self._browser.current_window_handle\n\n # generate excel\n self._browser.find_element_by_class_name('save-excel').click()\n time.sleep(1)\n for window_handle in self._browser.window_handles:\n if window_handle != main_window:\n self._browser.switch_to.window(window_handle)\n self._browser.close()\n self._browser.switch_to.window(main_window)\n\n # save pcap\n self._browser.find_element_by_class_name('save-wireshark').click()\n time.sleep(1)\n for window_handle in self._browser.window_handles:\n if window_handle != main_window:\n self._browser.switch_to.window(window_handle)\n self._browser.close()\n self._browser.switch_to.window(main_window)\n\n os.system('copy \"%%HOMEPATH%%\\\\Downloads\\\\NewPdf_*.pdf\" %s\\\\'\n % self.result_dir)\n os.system('copy \"%%HOMEPATH%%\\\\Downloads\\\\ExcelReport_*.xlsx\" %s\\\\'\n % self.result_dir)\n os.system('copy \"%s\\\\Captures\\\\*.pcapng\" %s\\\\'\n % (settings.HARNESS_HOME, self.result_dir))\n os.system('copy \"%s\\\\Thread_Harness\\\\temp\\\\*.*\" \"%s\"'\n % (settings.HARNESS_HOME, self.result_dir))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwaiting for dialogs and handle them until done.", "response": "def _wait_dialog(self):\n \"\"\"Wait for dialogs and handle them until done.\n \"\"\"\n logger.debug('waiting for dialog')\n done = False\n error = False\n\n logger.info(\"self timeout %d\",self.timeout)\n while not done and self.timeout:\n try:\n dialog = self._browser.find_element_by_id('RemoteConfirm')\n except:\n logger.exception('Failed to get dialog.')\n else:\n if dialog and dialog.get_attribute('aria-hidden') == 'false':\n title = dialog.find_element_by_class_name('modal-title').text\n time.sleep(1)\n logger.info('Handling dialog[%s]', title)\n\n try:\n done = self._handle_dialog(dialog, title)\n except:\n logger.exception('Error handling dialog: %s', title)\n error = True\n\n if done is None:\n raise FailError('Unexpected dialog occurred')\n\n dialog.find_element_by_id('ConfirmOk').click()\n\n time.sleep(1)\n\n try:\n stop_button = self._browser.find_element_by_id('stopTest')\n if done:\n stop_button.click()\n # wait for stop procedure end\n time.sleep(10)\n except NoSuchElementException:\n logger.info('Test stopped')\n time.sleep(5)\n done = True\n\n self.timeout -= 1\n\n # check if already ended capture\n if self.timeout % 10 == 0:\n lines = self._hc.tail()\n if 'SUCCESS: The process \"dumpcap.exe\" with PID ' in lines:\n logger.info('Tshark should be ended now, lets wait at most 30 seconds.')\n if not wait_until(lambda: 'tshark.exe' not in subprocess.check_output('tasklist'), 30):\n res = subprocess.check_output('taskkill /t /f /im tshark.exe',\n stderr=subprocess.STDOUT, shell=True)\n logger.info(res)\n\n # Wait until case really stopped\n wait_until(lambda: self._browser.find_element_by_id('runTest') and True, 30)\n\n if error:\n raise FailError('Fail for previous exceptions')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _handle_dialog(self, dialog, title):\n done = self.on_dialog(dialog, title)\n if isinstance(done, bool):\n return done\n\n if title.startswith('Start DUT'):\n body = dialog.find_element_by_id('cnfrmMsg').text\n if 'Sleepy End Device' in body:\n self.dut.mode = 's'\n self.dut.child_timeout = self.child_timeout\n elif 'End Device' in body:\n self.dut.mode = 'rsn'\n self.dut.child_timeout = self.child_timeout\n else:\n self.dut.mode = 'rsdn'\n\n if 'at channel' in body:\n self.channel = int(body.split(':')[1])\n\n self.dut.channel = self.channel\n self.dut.panid = settings.THREAD_PANID\n self.dut.networkname = settings.THREAD_NETWORKNAME\n self.dut.extpanid = settings.THREAD_EXTPANID\n self.dut.start()\n\n elif (title.startswith('MAC Address Required')\n or title.startswith('DUT Random Extended MAC Address Required')):\n mac = self.dut.mac\n inp = dialog.find_element_by_id('cnfrmInpText')\n inp.clear()\n inp.send_keys('0x%s' % mac)\n\n elif title.startswith('LL64 Address'):\n ll64 = None\n for addr in self.dut.addrs:\n addr = addr.lower()\n if addr.startswith('fe80') and not re.match('.+ff:fe00:[0-9a-f]{0,4}$', addr):\n ll64 = addr\n break\n\n if not ll64:\n raise FailError('No link local address found')\n\n logger.info('Link local address is %s', ll64)\n inp = dialog.find_element_by_id('cnfrmInpText')\n inp.clear()\n inp.send_keys(ll64)\n\n elif title.startswith('Enter Channel'):\n self.dut.channel = self.channel\n inp = dialog.find_element_by_id('cnfrmInpText')\n inp.clear()\n inp.send_keys(str(self.dut.channel))\n\n elif title.startswith('User Action Needed'):\n body = dialog.find_element_by_id('cnfrmMsg').text\n if body.startswith('Power Down the DUT'):\n self.dut.stop()\n return True\n\n elif title.startswith('Short Address'):\n short_addr = '0x%s' % self.dut.short_addr\n inp = dialog.find_element_by_id('cnfrmInpText')\n inp.clear()\n inp.send_keys(short_addr)\n\n elif title.startswith('ML64 Address'):\n ml64 = None\n for addr in self.dut.addrs:\n if addr.startswith('fd') and not re.match('.+ff:fe00:[0-9a-f]{0,4}$', addr):\n ml64 = addr\n break\n\n if not ml64:\n raise Exception('No mesh local address found')\n\n logger.info('Mesh local address is %s', ml64)\n inp = dialog.find_element_by_id('cnfrmInpText')\n inp.clear()\n inp.send_keys(ml64)\n\n elif title.startswith('Shield Devices') or title.startswith('Sheild DUT'):\n if self.rf_shield:\n logger.info('Shielding devices')\n with self.rf_shield:\n self.rf_shield.shield()\n elif self.dut and settings.SHIELD_SIMULATION:\n self.dut.channel = (self.channel == THREAD_CHANNEL_MAX\n and THREAD_CHANNEL_MIN) or (self.channel + 1)\n else:\n raw_input('Shield DUT and press enter to continue..')\n\n elif title.startswith('Unshield Devices') or title.startswith('Bring DUT Back to network'):\n if self.rf_shield:\n logger.info('Unshielding devices')\n with self.rf_shield:\n self.rf_shield.unshield()\n elif self.dut and settings.SHIELD_SIMULATION:\n self.dut.channel = self.channel\n else:\n raw_input('Bring DUT and press enter to continue..')\n\n elif title.startswith('Configure Prefix on DUT'):\n body = dialog.find_element_by_id('cnfrmMsg').text\n body = body.split(': ')[1]\n params = reduce(lambda params, param: params.update(((param[0].strip(' '), param[1]),)) or params,\n [it.split('=') for it in body.split(', ')], {})\n prefix = params['P_Prefix'].strip('\\0\\r\\n\\t ')\n flags = []\n if params.get('P_slaac_preferred', 0) == '1':\n flags.append('p')\n flags.append('ao')\n if params.get('P_stable', 0) == '1':\n flags.append('s')\n if params.get('P_default', 0) == '1':\n flags.append('r')\n prf = 'high'\n self.dut.add_prefix(prefix, ''.join(flags), prf)\n\n return False", "response": "Handle a dialog.\n\n Returns:\n bool True if no more dialogs expected,\n False if more dialogs needed, and\n None if not handled"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists devices in settings file and print versions", "response": "def list_devices(names=None, continue_from=None, **kwargs):\n \"\"\"List devices in settings file and print versions\"\"\"\n\n if not names:\n names = [device for device, _type in settings.GOLDEN_DEVICES if _type == 'OpenThread']\n\n if continue_from:\n continue_from = names.index(continue_from)\n else:\n continue_from = 0\n\n for port in names[continue_from:]:\n try:\n with OpenThreadController(port) as otc:\n print('%s: %s' % (port, otc.version))\n except:\n logger.exception('failed to get version of %s' % port)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndiscovering all test cases and skip them", "response": "def discover(names=None, pattern=['*.py'], skip='efp', dry_run=False, blacklist=None, name_greps=None,\n manual_reset=False, delete_history=False, max_devices=0,\n continue_from=None, result_file='./result.json', auto_reboot=False, keep_explorer=False,\n add_all_devices=False):\n \"\"\"Discover all test cases and skip those passed\n\n Args:\n pattern (str): Pattern to match case modules, refer python's unittest\n documentation for more details\n skip (str): types cases to skip\n \"\"\"\n if not os.path.exists(settings.OUTPUT_PATH):\n os.mkdir(settings.OUTPUT_PATH)\n\n if delete_history:\n os.system('del history.json')\n\n if blacklist:\n try:\n excludes = [line.strip('\\n') for line in open(blacklist, 'r').readlines()\n if not line.startswith('#')]\n except:\n logger.exception('Failed to open test case black list file')\n raise\n else:\n excludes = []\n\n log = None\n if os.path.isfile(result_file):\n try:\n log = json.load(open(result_file, 'r'))\n except:\n logger.exception('Failed to open result file')\n\n if not log:\n log = {}\n json.dump(log, open(result_file, 'w'), indent=2)\n\n suite = unittest.TestSuite()\n discovered = unittest.defaultTestLoader.discover('cases', pattern)\n\n if names and continue_from:\n names = names[names.index(continue_from):]\n\n for s1 in discovered:\n for s2 in s1:\n for case in s2:\n if case.__class__ is HarnessCase:\n continue\n case_name = unicode(case.__class__.__name__)\n\n # grep name\n if name_greps and not any(fnmatch.fnmatch(case_name, name_grep) for name_grep in name_greps):\n logger.info('case[%s] skipped by name greps', case_name)\n continue\n\n # whitelist\n if len(names) and case_name not in names:\n logger.info('case[%s] skipped', case_name)\n continue\n\n # skip cases\n if case_name in log.keys():\n if (log[case_name]['passed'] and ('p' in skip)) \\\n or (log[case_name]['passed'] is False and ('f' in skip)) \\\n or (log[case_name]['passed'] is None and ('e' in skip)):\n logger.warning('case[%s] skipped for its status[%s]', case_name, log[case_name]['passed'])\n continue\n\n # continue from\n if continue_from:\n if continue_from != case_name:\n logger.warning('case[%s] skipped for continue from[%s]', case_name, continue_from)\n continue\n else:\n continue_from = None\n\n # black list\n if case_name in excludes:\n logger.warning('case[%s] skipped for blacklist', case_name)\n continue\n\n # max devices\n if max_devices and case.golden_devices_required > max_devices:\n logger.warning('case[%s] skipped for exceeding max golden devices allowed[%d]', case_name, max_devices)\n continue\n\n suite.addTest(case)\n logger.info('case[%s] added', case_name)\n\n if auto_reboot:\n argv = []\n argv.append('\"%s\"' % os.sep.join([os.getcwd(), 'start.bat']))\n argv.extend(['-p', pattern])\n argv.extend(['-k', skip])\n argv.extend(['-o', result_file])\n argv.append('-a')\n\n if manual_reset:\n argv.append('-m')\n\n if delete_history:\n argv.append('-d')\n\n auto_reboot_args = argv + names\n else:\n auto_reboot_args = None\n os.system('del \"%s\"' % RESUME_SCRIPT_PATH)\n\n # manual reset\n if manual_reset:\n settings.PDU_CONTROLLER_TYPE = 'MANUAL_PDU_CONTROLLER'\n settings.PDU_CONTROLLER_OPEN_PARAMS = {}\n settings.PDU_CONTROLLER_REBOOT_PARAMS = {}\n\n result = SimpleTestResult(result_file, auto_reboot_args, keep_explorer, add_all_devices)\n for case in suite:\n logger.info(case.__class__.__name__)\n\n if dry_run:\n return\n\n suite.run(result)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding the expected string within times.", "response": "def _expect(self, expected, times=50):\n \"\"\"Find the `expected` line within `times` trials.\n\n Args:\n expected str: the expected string\n times int: number of trials\n \"\"\"\n print '[%s] Expecting [%s]' % (self.port, expected)\n\n retry_times = 10\n while times > 0 and retry_times > 0:\n line = self._readline()\n print '[%s] Got line [%s]' % (self.port, line)\n\n if line == expected:\n print '[%s] Expected [%s]' % (self.port, expected)\n return\n\n if not line:\n retry_times -= 1\n time.sleep(0.1)\n\n times -= 1\n\n raise Exception('failed to find expected string[%s]' % expected)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _readline(self):\n logging.info('%s: reading line', self.port)\n if len(self._lines) > 1:\n return self._lines.pop(0)\n\n tail = ''\n if len(self._lines):\n tail = self._lines.pop()\n\n try:\n tail += self._read()\n except socket.error:\n logging.exception('%s: No new data', self.port)\n time.sleep(0.1)\n\n self._lines += LINESEPX.split(tail)\n if len(self._lines) > 1:\n return self._lines.pop(0)", "response": "Read exactly one line from the device returning the first line on success Returns None on no data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending exactly one line to the device", "response": "def _sendline(self, line):\n \"\"\"Send exactly one line to the device\n\n Args:\n line str: data send to device\n \"\"\"\n logging.info('%s: sending line', self.port)\n # clear buffer\n self._lines = []\n try:\n self._read()\n except socket.error:\n logging.debug('%s: Nothing cleared', self.port)\n\n print 'sending [%s]' % line\n self._write(line + '\\r\\n')\n\n # wait for write to complete\n time.sleep(0.1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend specific command to the OpenThread CLI string and parse the response", "response": "def __sendCommand(self, cmd):\n \"\"\"send specific command to reference unit over serial port\n\n Args:\n cmd: OpenThread CLI string\n\n Returns:\n Done: successfully send the command to reference unit and parse it\n Value: successfully retrieve the desired value from reference unit\n Error: some errors occur, indicates by the followed specific error number\n \"\"\"\n logging.info('%s: sendCommand[%s]', self.port, cmd)\n if self.logThreadStatus == self.logStatus['running']:\n self.logThreadStatus = self.logStatus['pauseReq']\n while self.logThreadStatus != self.logStatus['paused'] and self.logThreadStatus != self.logStatus['stop']:\n pass\n\n try:\n # command retransmit times\n retry_times = 3\n while retry_times > 0:\n retry_times -= 1\n try:\n self._sendline(cmd)\n self._expect(cmd)\n except Exception as e:\n logging.exception('%s: failed to send command[%s]: %s', self.port, cmd, str(e))\n if retry_times == 0:\n raise\n else:\n break\n\n line = None\n response = []\n retry_times = 10\n while retry_times > 0:\n line = self._readline()\n logging.info('%s: the read line is[%s]', self.port, line)\n if line:\n response.append(line)\n if line == 'Done':\n break\n else:\n retry_times -= 1\n time.sleep(0.2)\n if line != 'Done':\n raise Exception('%s: failed to find end of response' % self.port)\n logging.info('%s: send command[%s] done!', self.port, cmd)\n return response\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"sendCommand() Error: \" + str(e))\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting specific type of IPv6 address configured on thread device", "response": "def __getIp6Address(self, addressType):\n \"\"\"get specific type of IPv6 address configured on thread device\n\n Args:\n addressType: the specific type of IPv6 address\n\n link local: link local unicast IPv6 address that's within one-hop scope\n global: global unicast IPv6 address\n rloc: mesh local unicast IPv6 address for routing in thread network\n mesh EID: mesh Endpoint Identifier\n\n Returns:\n IPv6 address string\n \"\"\"\n addrType = ['link local', 'global', 'rloc', 'mesh EID']\n addrs = []\n globalAddr = []\n linkLocal64Addr = ''\n rlocAddr = ''\n meshEIDAddr = ''\n\n addrs = self.__sendCommand('ipaddr')\n for ip6Addr in addrs:\n if ip6Addr == 'Done':\n break\n\n ip6AddrPrefix = ip6Addr.split(':')[0]\n if ip6AddrPrefix == 'fe80':\n # link local address\n if ip6Addr.split(':')[4] != '0':\n linkLocal64Addr = ip6Addr\n elif ip6Addr.startswith(self.meshLocalPrefix):\n # mesh local address\n if ip6Addr.split(':')[4] == '0':\n # rloc\n rlocAddr = ip6Addr\n else:\n # mesh EID\n meshEIDAddr = ip6Addr\n else:\n # global ipv6 address\n if ip6Addr != None:\n globalAddr.append(ip6Addr)\n else:\n pass\n\n if addressType == addrType[0]:\n return linkLocal64Addr\n elif addressType == addrType[1]:\n return globalAddr\n elif addressType == addrType[2]:\n return rlocAddr\n elif addressType == addrType[3]:\n return meshEIDAddr\n else:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __setRouterUpgradeThreshold(self, iThreshold):\n print 'call __setRouterUpgradeThreshold'\n try:\n cmd = 'routerupgradethreshold %s' % str(iThreshold)\n print cmd\n return self.__sendCommand(cmd) == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setRouterUpgradeThreshold() Error: \" + str(e))", "response": "set the router upgrade threshold"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __setRouterSelectionJitter(self, iRouterJitter):\n print 'call _setRouterSelectionJitter'\n try:\n cmd = 'routerselectionjitter %s' % str(iRouterJitter)\n print cmd\n return self.__sendCommand(cmd) == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setRouterSelectionJitter() Error: \" + str(e))", "response": "set the ROUTER_SELECTION_JITTER parameter for REED to upgrade to Router"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __setAddressfilterMode(self, mode):\n print 'call setAddressFilterMode() ' + mode\n try:\n cmd = 'macfilter addr ' + mode\n if self.__sendCommand(cmd)[0] == 'Done':\n return True\n return False\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"__setAddressFilterMode() Error: \" + str(e))", "response": "set the mode of the mac filter addr"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __startOpenThread(self):\n print 'call startOpenThread'\n try:\n if self.hasActiveDatasetToCommit:\n if self.__sendCommand('dataset commit active')[0] != 'Done':\n raise Exception('failed to commit active dataset')\n else:\n self.hasActiveDatasetToCommit = False\n\n # restore whitelist/blacklist address filter mode if rejoin after reset\n if self.isPowerDown:\n if self._addressfilterMode == 'whitelist':\n if self.__setAddressfilterMode('whitelist'):\n for addr in self._addressfilterSet:\n self.addAllowMAC(addr)\n elif self._addressfilterMode == 'blacklist':\n if self.__setAddressfilterMode('blacklist'):\n for addr in self._addressfilterSet:\n self.addBlockedMAC(addr)\n\n if self.deviceRole in [Thread_Device_Role.Leader, Thread_Device_Role.Router, Thread_Device_Role.REED]:\n self.__setRouterSelectionJitter(1)\n\n if self.__sendCommand('ifconfig up')[0] == 'Done':\n if self.__sendCommand('thread start')[0] == 'Done':\n self.isPowerDown = False\n return True\n else:\n return False\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"startOpenThread() Error: \" + str(e))", "response": "start OpenThread stack and thread interface up\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __stopOpenThread(self):\n print 'call stopOpenThread'\n try:\n if self.__sendCommand('thread stop')[0] == 'Done':\n return self.__sendCommand('ifconfig down')[0] == 'Done'\n else:\n return False\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"stopOpenThread() Error: \" + str(e))", "response": "stop OpenThread stack and thread interface down\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __convertRlocToRouterId(self, xRloc16):\n routerList = []\n routerList = self.__sendCommand('router list')[0].split()\n print routerList\n print xRloc16\n\n for index in routerList:\n router = []\n cmd = 'router %s' % index\n router = self.__sendCommand(cmd)\n\n for line in router:\n if 'Done' in line:\n break\n elif 'Router ID' in line:\n routerid = line.split()[2]\n elif 'Rloc' in line:\n rloc16 = line.split()[1]\n else:\n pass\n\n # process input rloc16\n if isinstance(xRloc16, str):\n rloc16 = '0x' + rloc16\n if rloc16 == xRloc16:\n return routerid\n elif isinstance(xRloc16, int):\n if int(rloc16, 16) == xRloc16:\n return routerid\n else:\n pass\n\n return None", "response": "This function converts Rloc to Router ID."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __convertIp6PrefixStringToIp6Address(self, strIp6Prefix):\n prefix1 = strIp6Prefix.rstrip('L')\n prefix2 = prefix1.lstrip(\"0x\")\n hexPrefix = str(prefix2).ljust(16,'0')\n hexIter = iter(hexPrefix)\n finalMac = ':'.join(a + b + c + d for a,b,c,d in zip(hexIter, hexIter,hexIter,hexIter))\n prefix = str(finalMac)\n strIp6Prefix = prefix[:20]\n return strIp6Prefix +':'", "response": "convert IPv6 prefix string to IPv6 dotted - quad format"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a long hex integer to string", "response": "def __convertLongToString(self, iValue):\n \"\"\"convert a long hex integer to string\n remove '0x' and 'L' return string\n\n Args:\n iValue: long integer in hex format\n\n Returns:\n string of this long integer without \"0x\" and \"L\"\n \"\"\"\n string = ''\n strValue = str(hex(iValue))\n\n string = strValue.lstrip('0x')\n string = string.rstrip('L')\n\n return string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __readCommissioningLogs(self, durationInSeconds):\n self.logThreadStatus = self.logStatus['running']\n logs = Queue()\n t_end = time.time() + durationInSeconds\n while time.time() < t_end:\n time.sleep(0.3)\n\n if self.logThreadStatus == self.logStatus['pauseReq']:\n self.logThreadStatus = self.logStatus['paused']\n\n if self.logThreadStatus != self.logStatus['running']:\n continue\n\n try:\n line = self._readline()\n if line:\n print line\n logs.put(line)\n\n if \"Join success\" in line:\n self.joinCommissionedStatus = self.joinStatus['succeed']\n break\n elif \"Join failed\" in line:\n self.joinCommissionedStatus = self.joinStatus['failed']\n break\n\n except Exception:\n pass\n\n self.logThreadStatus = self.logStatus['stop']\n return logs", "response": "read logs during the commissioning process"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __convertChannelMask(self, channelsArray):\n maskSet = 0\n\n for eachChannel in channelsArray:\n mask = 1 << eachChannel\n maskSet = (maskSet | mask)\n\n return maskSet", "response": "convert channelsArray to bitmask format corresponding to a given channel array"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __setKeySwitchGuardTime(self, iKeySwitchGuardTime):\n print '%s call setKeySwitchGuardTime' % self.port\n print iKeySwitchGuardTime\n try:\n cmd = 'keysequence guardtime %s' % str(iKeySwitchGuardTime)\n if self.__sendCommand(cmd)[0] == 'Done':\n time.sleep(1)\n return True\n else:\n return False\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setKeySwitchGuardTime() Error; \" + str(e))", "response": "set the Key switch guard time"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclose current serial port connection", "response": "def closeConnection(self):\n \"\"\"close current serial port connection\"\"\"\n print '%s call closeConnection' % self.port\n try:\n if self.handle:\n self.handle.close()\n self.handle = None\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"closeConnection() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninitializes the serial port with baudrate timeout parameters", "response": "def intialize(self):\n \"\"\"initialize the serial port with baudrate, timeout parameters\"\"\"\n print '%s call intialize' % self.port\n try:\n self.deviceConnected = False\n\n # init serial port\n self._connect()\n\n if self.firmwarePrefix in self.UIStatusMsg:\n self.deviceConnected = True\n else:\n self.UIStatusMsg = \"Firmware Not Matching Expecting \" + self.firmwarePrefix + \" Now is \" + self.UIStatusMsg\n ModuleHelper.WriteIntoDebugLogger(\"Err: OpenThread device Firmware not matching..\")\n\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"intialize() Error: \" + str(e))\n self.deviceConnected = False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setNetworkName(self, networkName='GRL'):\n print '%s call setNetworkName' % self.port\n print networkName\n try:\n cmd = 'networkname %s' % networkName\n datasetCmd = 'dataset networkname %s' % networkName\n self.hasActiveDatasetToCommit = True\n return self.__sendCommand(cmd)[0] == 'Done' and self.__sendCommand(datasetCmd)[0] == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setNetworkName() Error: \" + str(e))", "response": "set Thread Network name"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef setChannel(self, channel=11):\n print '%s call setChannel' % self.port\n print channel\n try:\n cmd = 'channel %s' % channel\n datasetCmd = 'dataset channel %s' % channel\n self.hasActiveDatasetToCommit = True\n return self.__sendCommand(cmd)[0] == 'Done' and self.__sendCommand(datasetCmd)[0] == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setChannel() Error: \" + str(e))", "response": "set the channel of Thread device operates on"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getMAC(self, bType=MacType.RandomMac):\n print '%s call getMAC' % self.port\n print bType\n # if power down happens, return extended address assigned previously\n if self.isPowerDown:\n macAddr64 = self.mac\n else:\n if bType == MacType.FactoryMac:\n macAddr64 = self.__sendCommand('eui64')[0]\n elif bType == MacType.HashMac:\n macAddr64 = self.__sendCommand('joinerid')[0]\n else:\n macAddr64 = self.__sendCommand('extaddr')[0]\n print macAddr64\n\n return int(macAddr64, 16)", "response": "get one specific type of MAC address"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef getRloc16(self):\n print '%s call getRloc16' % self.port\n rloc16 = self.__sendCommand('rloc16')[0]\n return int(rloc16, 16)", "response": "get rloc16 short address"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getNetworkFragmentID(self):\n print '%s call getNetworkFragmentID' % self.port\n if not self.__isOpenThreadRunning():\n print 'OpenThread is not running'\n return None\n\n leaderData = []\n leaderData = self.__sendCommand('leaderdata')\n return int(leaderData[0].split()[2], 16)", "response": "get current partition id of Thread Network Partition from LeaderData\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getParentAddress(self):\n print '%s call getParentAddress' % self.port\n parentInfo = []\n parentInfo = self.__sendCommand('parent')\n\n for line in parentInfo:\n if 'Done' in line:\n break\n elif 'Ext Addr' in line:\n eui = line.split()[2]\n print eui\n #elif 'Rloc' in line:\n # rloc16 = line.split()[1]\n # print rloc16\n else:\n pass\n\n return int(eui, 16)", "response": "get Thread device s parent extended address and rloc16 short address"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\npowering down the Thread device", "response": "def powerDown(self):\n \"\"\"power down the Thread device\"\"\"\n print '%s call powerDown' % self.port\n self._sendline('reset')\n self.isPowerDown = True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef powerUp(self):\n print '%s call powerUp' % self.port\n if not self.handle:\n self._connect()\n\n self.isPowerDown = False\n\n if not self.__isOpenThreadRunning():\n self.__startOpenThread()", "response": "power up the Thread device"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nresets and rejoin Thread Network without any timeout", "response": "def reboot(self):\n \"\"\"reset and rejoin to Thread Network without any timeout\n\n Returns:\n True: successful to reset and rejoin the Thread Network\n False: fail to reset and rejoin the Thread Network\n \"\"\"\n print '%s call reboot' % self.port\n try:\n self._sendline('reset')\n self.isPowerDown = True\n time.sleep(3)\n\n self.__startOpenThread()\n time.sleep(3)\n\n if self.__sendCommand('state')[0] == 'disabled':\n print '[FAIL] reboot'\n return False\n else:\n return True\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"reboot() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend ICMPv6 echo request with a given length to a unicast destination", "response": "def ping(self, destination, length=20):\n \"\"\" send ICMPv6 echo request with a given length to a unicast destination\n address\n\n Args:\n destination: the unicast destination address of ICMPv6 echo request\n length: the size of ICMPv6 echo request payload\n \"\"\"\n print '%s call ping' % self.port\n print 'destination: %s' %destination\n try:\n cmd = 'ping %s %s' % (destination, str(length))\n print cmd\n self._sendline(cmd)\n self._expect(cmd)\n # wait echo reply\n time.sleep(1)\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"ping() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nresetting the internal state of the object", "response": "def reset(self):\n \"\"\"factory reset\"\"\"\n print '%s call reset' % self.port\n try:\n self._sendline('factoryreset')\n self._read()\n\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"reset() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget data polling rate for sleepy end device", "response": "def getPollingRate(self):\n \"\"\"get data polling rate for sleepy end device\"\"\"\n print '%s call getPollingRate' % self.port\n sPollingRate = self.__sendCommand('pollperiod')[0]\n try:\n iPollingRate = int(sPollingRate)/1000\n fPollingRate = round(float(sPollingRate)/1000, 3)\n return fPollingRate if fPollingRate > iPollingRate else iPollingRate\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"getPollingRate() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef setLinkQuality(self, EUIadr, LinkQuality):\n print '%s call setLinkQuality' % self.port\n print EUIadr\n print LinkQuality\n try:\n # process EUIadr\n euiHex = hex(EUIadr)\n euiStr = str(euiHex)\n euiStr = euiStr.rstrip('L')\n address64 = ''\n if '0x' in euiStr:\n address64 = euiStr.lstrip('0x')\n # prepend 0 at the beginning\n if len(address64) < 16:\n address64 = address64.zfill(16)\n print address64\n\n cmd = 'macfilter rss add-lqi %s %s' % (address64, str(LinkQuality))\n print cmd\n return self.__sendCommand(cmd)[0] == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setLinkQuality() Error: \" + str(e))", "response": "set custom link quality for all receiving messages from the specified EUIadr"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets custom link quality for all receiving messages from the any address", "response": "def setOutBoundLinkQuality(self, LinkQuality):\n \"\"\"set custom LinkQualityIn for all receiving messages from the any address\n\n Args:\n LinkQuality: a given custom link quality\n link quality/link margin mapping table\n 3: 21 - 255 (dB)\n 2: 11 - 20 (dB)\n 1: 3 - 9 (dB)\n 0: 0 - 2 (dB)\n\n Returns:\n True: successful to set the link quality\n False: fail to set the link quality\n \"\"\"\n print '%s call setOutBoundLinkQuality' % self.port\n print LinkQuality\n try:\n cmd = 'macfilter rss add-lqi * %s' % str(LinkQuality)\n print cmd\n return self.__sendCommand(cmd)[0] == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setOutBoundLinkQuality() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef removeRouterPrefix(self, prefixEntry):\n print '%s call removeRouterPrefix' % self.port\n print prefixEntry\n prefix = self.__convertIp6PrefixStringToIp6Address(str(prefixEntry))\n try:\n prefixLen = 64\n cmd = 'prefix remove %s/%d' % (prefix, prefixLen)\n print cmd\n if self.__sendCommand(cmd)[0] == 'Done':\n # send server data ntf to leader\n return self.__sendCommand('netdataregister')[0] == 'Done'\n else:\n return False\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"removeRouterPrefix() Error: \" + str(e))", "response": "remove the configured router prefix on a border"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nresetting and rejoin Thread Network with a given timeout delay", "response": "def resetAndRejoin(self, timeout):\n \"\"\"reset and join back Thread Network with a given timeout delay\n\n Args:\n timeout: a timeout interval before rejoin Thread Network\n\n Returns:\n True: successful to reset and rejoin Thread Network\n False: fail to reset and rejoin the Thread Network\n \"\"\"\n print '%s call resetAndRejoin' % self.port\n print timeout\n try:\n self._sendline('reset')\n self.isPowerDown = True\n time.sleep(timeout)\n\n if self.deviceRole == Thread_Device_Role.SED:\n self.setPollingRate(self.sedPollingRate)\n\n self.__startOpenThread()\n time.sleep(3)\n\n if self.__sendCommand('state')[0] == 'disabled':\n print '[FAIL] reset and rejoin'\n return False\n return True\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"resetAndRejoin() Error: \" + str(e))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting networkid timeout for Thread device", "response": "def setNetworkIDTimeout(self, iNwkIDTimeOut):\n \"\"\"set networkid timeout for Thread device\n\n Args:\n iNwkIDTimeOut: a given NETWORK_ID_TIMEOUT\n\n Returns:\n True: successful to set NETWORK_ID_TIMEOUT\n False: fail to set NETWORK_ID_TIMEOUT\n \"\"\"\n print '%s call setNetworkIDTimeout' % self.port\n print iNwkIDTimeOut\n iNwkIDTimeOut /= 1000\n try:\n cmd = 'networkidtimeout %s' % str(iNwkIDTimeOut)\n print cmd\n return self.__sendCommand(cmd)[0] == 'Done'\n except Exception, e:\n ModuleHelper.WriteIntoDebugLogger(\"setNetworkIDTimeout() Error: \" + str(e))"}