{"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef read_and_write(self, reader, writer, chunk_size):\n data = reader.recv(chunk_size)\n if len(data) == 0:\n return True\n writer.sendall(data)", "response": "Read chunk_size from reader and write result to writer."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_local_user():\n # TODO: I don't understand why these lines were added outside the\n # try/except, since presumably it means the attempt at catching ImportError\n # wouldn't work. However, that's how the contributing user committed it.\n # Need an older Windows box to test it out, most likely.\n import getpass\n\n username = None\n # All Unix and most Windows systems support the getpass module.\n try:\n username = getpass.getuser()\n # Some SaaS platforms raise KeyError, implying there is no real user\n # involved. They get the default value of None.\n except KeyError:\n pass\n # Older (?) Windows systems don't support getpass well; they should\n # have the `win32` module instead.\n except ImportError: # pragma: nocover\n if win32:\n import win32api\n import win32security # noqa\n import win32profile # noqa\n\n username = win32api.GetUserName()\n return username", "response": "Return the local executing username or None if one can t be found."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parameterize(self, call, host):\n debug(\"Parameterizing {!r} for host {!r}\".format(call, host))\n # Generate a custom ConnectionCall that knows how to yield a Connection\n # in its make_context(), specifically one to the host requested here.\n clone = call.clone(into=ConnectionCall)\n # TODO: using bag-of-attrs is mildly gross but whatever, I'll take it.\n clone.host = host\n return clone", "response": "Parameterize a Call with its Context set to a per - host Config."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef mixed_use_of_local_and_run(self):\n cxn = Connection(\"localhost\")\n result = cxn.local(\"echo foo\", hide=True)\n assert result.stdout == \"foo\\n\"\n assert not cxn.is_connected # meh way of proving it didn't use SSH yet\n result = cxn.run(\"echo foo\", hide=True)\n assert cxn.is_connected # NOW it's using SSH\n assert result.stdout == \"foo\\n\"", "response": "mixed use of local and run command truly over SSH via localhost"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nopen an SSH connection to the host and port.", "response": "def open(self):\n \"\"\"\n Initiate an SSH connection to the host/port this object is bound to.\n\n This may include activating the configured gateway connection, if one\n is set.\n\n Also saves a handle to the now-set Transport object for easier access.\n\n Various connect-time settings (and/or their corresponding :ref:`SSH\n config options `) are utilized here in the call to\n `SSHClient.connect `. (For details,\n see :doc:`the configuration docs `.)\n\n .. versionadded:: 2.0\n \"\"\"\n # Short-circuit\n if self.is_connected:\n return\n err = \"Refusing to be ambiguous: connect() kwarg '{}' was given both via regular arg and via connect_kwargs!\" # noqa\n # These may not be given, period\n for key in \"\"\"\n hostname\n port\n username\n \"\"\".split():\n if key in self.connect_kwargs:\n raise ValueError(err.format(key))\n # These may be given one way or the other, but not both\n if (\n \"timeout\" in self.connect_kwargs\n and self.connect_timeout is not None\n ):\n raise ValueError(err.format(\"timeout\"))\n # No conflicts -> merge 'em together\n kwargs = dict(\n self.connect_kwargs,\n username=self.user,\n hostname=self.host,\n port=self.port,\n )\n if self.gateway:\n kwargs[\"sock\"] = self.open_gateway()\n if self.connect_timeout:\n kwargs[\"timeout\"] = self.connect_timeout\n # Strip out empty defaults for less noisy debugging\n if \"key_filename\" in kwargs and not kwargs[\"key_filename\"]:\n del kwargs[\"key_filename\"]\n # Actually connect!\n self.client.connect(**kwargs)\n self.transport = self.client.get_transport()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open_gateway(self):\n # ProxyCommand is faster to set up, so do it first.\n if isinstance(self.gateway, string_types):\n # Leverage a dummy SSHConfig to ensure %h/%p/etc are parsed.\n # TODO: use real SSH config once loading one properly is\n # implemented.\n ssh_conf = SSHConfig()\n dummy = \"Host {}\\n ProxyCommand {}\"\n ssh_conf.parse(StringIO(dummy.format(self.host, self.gateway)))\n return ProxyCommand(ssh_conf.lookup(self.host)[\"proxycommand\"])\n # Handle inner-Connection gateway type here.\n # TODO: logging\n self.gateway.open()\n # TODO: expose the opened channel itself as an attribute? (another\n # possible argument for separating the two gateway types...) e.g. if\n # someone wanted to piggyback on it for other same-interpreter socket\n # needs...\n # TODO: and the inverse? allow users to supply their own socket/like\n # object they got via $WHEREEVER?\n # TODO: how best to expose timeout param? reuse general connection\n # timeout from config?\n return self.gateway.transport.open_channel(\n kind=\"direct-tcpip\",\n dest_addr=(self.host, int(self.port)),\n # NOTE: src_addr needs to be 'empty but not None' values to\n # correctly encode into a network message. Theoretically Paramiko\n # could auto-interpret None sometime & save us the trouble.\n src_addr=(\"\", 0),\n )", "response": "Open a new connection from the gateway."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nterminate the network connection to the remote end if open.", "response": "def close(self):\n \"\"\"\n Terminate the network connection to the remote end, if open.\n\n If no connection is open, this method does nothing.\n\n .. versionadded:: 2.0\n \"\"\"\n if self.is_connected:\n self.client.close()\n if self.forward_agent and self._agent_handler is not None:\n self._agent_handler.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a shell command on the remote end of this connection.", "response": "def run(self, command, **kwargs):\n \"\"\"\n Execute a shell command on the remote end of this connection.\n\n This method wraps an SSH-capable implementation of\n `invoke.runners.Runner.run`; see its documentation for details.\n\n .. warning::\n There are a few spots where Fabric departs from Invoke's default\n settings/behaviors; they are documented under\n `.Config.global_defaults`.\n\n .. versionadded:: 2.0\n \"\"\"\n runner = self.config.runners.remote(self)\n return self._run(runner, command, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sudo(self, command, **kwargs):\n runner = self.config.runners.remote(self)\n return self._sudo(runner, command, **kwargs)", "response": "Execute a shell command on the remote end."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nexecutes a shell command on the local system.", "response": "def local(self, *args, **kwargs):\n \"\"\"\n Execute a shell command on the local system.\n\n This method is effectively a wrapper of `invoke.run`; see its docs for\n details and call signature.\n\n .. versionadded:: 2.0\n \"\"\"\n # Superclass run() uses runners.local, so we can literally just call it\n # straight.\n return super(Connection, self).run(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new instance of the SFTP client.", "response": "def sftp(self):\n \"\"\"\n Return a `~paramiko.sftp_client.SFTPClient` object.\n\n If called more than one time, memoizes the first result; thus, any\n given `.Connection` instance will only ever have a single SFTP client,\n and state (such as that managed by\n `~paramiko.sftp_client.SFTPClient.chdir`) will be preserved.\n\n .. versionadded:: 2.0\n \"\"\"\n if self._sftp is None:\n self._sftp = self.client.open_sftp()\n return self._sftp"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef forward_local(\n self,\n local_port,\n remote_port=None,\n remote_host=\"localhost\",\n local_host=\"localhost\",\n ):\n \"\"\"\n Open a tunnel connecting ``local_port`` to the server's environment.\n\n For example, say you want to connect to a remote PostgreSQL database\n which is locked down and only accessible via the system it's running\n on. You have SSH access to this server, so you can temporarily make\n port 5432 on your local system act like port 5432 on the server::\n\n import psycopg2\n from fabric import Connection\n\n with Connection('my-db-server').forward_local(5432):\n db = psycopg2.connect(\n host='localhost', port=5432, database='mydb'\n )\n # Do things with 'db' here\n\n This method is analogous to using the ``-L`` option of OpenSSH's\n ``ssh`` program.\n\n :param int local_port: The local port number on which to listen.\n\n :param int remote_port:\n The remote port number. Defaults to the same value as\n ``local_port``.\n\n :param str local_host:\n The local hostname/interface on which to listen. Default:\n ``localhost``.\n\n :param str remote_host:\n The remote hostname serving the forwarded remote port. Default:\n ``localhost`` (i.e., the host this `.Connection` is connected to.)\n\n :returns:\n Nothing; this method is only useful as a context manager affecting\n local operating system state.\n\n .. versionadded:: 2.0\n \"\"\"\n if not remote_port:\n remote_port = local_port\n\n # TunnelManager does all of the work, sitting in the background (so we\n # can yield) and spawning threads every time somebody connects to our\n # local port.\n finished = Event()\n manager = TunnelManager(\n local_port=local_port,\n local_host=local_host,\n remote_port=remote_port,\n remote_host=remote_host,\n # TODO: not a huge fan of handing in our transport, but...?\n transport=self.transport,\n finished=finished,\n )\n manager.start()\n\n # Return control to caller now that things ought to be operational\n try:\n yield\n # Teardown once user exits block\n finally:\n # Signal to manager that it should close all open tunnels\n finished.set()\n # Then wait for it to do so\n manager.join()\n # Raise threading errors from within the manager, which would be\n # one of:\n # - an inner ThreadException, which was created by the manager on\n # behalf of its Tunnels; this gets directly raised.\n # - some other exception, which would thus have occurred in the\n # manager itself; we wrap this in a new ThreadException.\n # NOTE: in these cases, some of the metadata tracking in\n # ExceptionHandlingThread/ExceptionWrapper/ThreadException (which\n # is useful when dealing with multiple nearly-identical sibling IO\n # threads) is superfluous, but it doesn't feel worth breaking\n # things up further; we just ignore it for now.\n wrapper = manager.exception()\n if wrapper is not None:\n if wrapper.type is ThreadException:\n raise wrapper.value\n else:\n raise ThreadException([wrapper])", "response": "Forward a local entry to the local operating system."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef forward_remote(\n self,\n remote_port,\n local_port=None,\n remote_host=\"127.0.0.1\",\n local_host=\"localhost\",\n ):\n \"\"\"\n Open a tunnel connecting ``remote_port`` to the local environment.\n\n For example, say you're running a daemon in development mode on your\n workstation at port 8080, and want to funnel traffic to it from a\n production or staging environment.\n\n In most situations this isn't possible as your office/home network\n probably blocks inbound traffic. But you have SSH access to this\n server, so you can temporarily make port 8080 on that server act like\n port 8080 on your workstation::\n\n from fabric import Connection\n\n c = Connection('my-remote-server')\n with c.forward_remote(8080):\n c.run(\"remote-data-writer --port 8080\")\n # Assuming remote-data-writer runs until interrupted, this will\n # stay open until you Ctrl-C...\n\n This method is analogous to using the ``-R`` option of OpenSSH's\n ``ssh`` program.\n\n :param int remote_port: The remote port number on which to listen.\n\n :param int local_port:\n The local port number. Defaults to the same value as\n ``remote_port``.\n\n :param str local_host:\n The local hostname/interface the forwarded connection talks to.\n Default: ``localhost``.\n\n :param str remote_host:\n The remote interface address to listen on when forwarding\n connections. Default: ``127.0.0.1`` (i.e. only listen on the remote\n localhost).\n\n :returns:\n Nothing; this method is only useful as a context manager affecting\n local operating system state.\n\n .. versionadded:: 2.0\n \"\"\"\n if not local_port:\n local_port = remote_port\n # Callback executes on each connection to the remote port and is given\n # a Channel hooked up to said port. (We don't actually care about the\n # source/dest host/port pairs at all; only whether the channel has data\n # to read and suchlike.)\n # We then pair that channel with a new 'outbound' socket connection to\n # the local host/port being forwarded, in a new Tunnel.\n # That Tunnel is then added to a shared data structure so we can track\n # & close them during shutdown.\n #\n # TODO: this approach is less than ideal because we have to share state\n # between ourselves & the callback handed into the transport's own\n # thread handling (which is roughly analogous to our self-controlled\n # TunnelManager for local forwarding). See if we can use more of\n # Paramiko's API (or improve it and then do so) so that isn't\n # necessary.\n tunnels = []\n\n def callback(channel, src_addr_tup, dst_addr_tup):\n sock = socket.socket()\n # TODO: handle connection failure such that channel, etc get closed\n sock.connect((local_host, local_port))\n # TODO: we don't actually need to generate the Events at our level,\n # do we? Just let Tunnel.__init__ do it; all we do is \"press its\n # button\" on shutdown...\n tunnel = Tunnel(channel=channel, sock=sock, finished=Event())\n tunnel.start()\n # Communication between ourselves & the Paramiko handling subthread\n tunnels.append(tunnel)\n\n # Ask Paramiko (really, the remote sshd) to call our callback whenever\n # connections are established on the remote iface/port.\n # transport.request_port_forward(remote_host, remote_port, callback)\n try:\n self.transport.request_port_forward(\n address=remote_host, port=remote_port, handler=callback\n )\n yield\n finally:\n # TODO: see above re: lack of a TunnelManager\n # TODO: and/or also refactor with TunnelManager re: shutdown logic.\n # E.g. maybe have a non-thread TunnelManager-alike with a method\n # that acts as the callback? At least then there's a tiny bit more\n # encapsulation...meh.\n for tunnel in tunnels:\n tunnel.finished.set()\n tunnel.join()\n self.transport.cancel_port_forward(\n address=remote_host, port=remote_port\n )", "response": "Forward a new connection to the remote server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dedup_list(l):\n dedup = set()\n return [ x for x in l if not (x in dedup or dedup.add(x))]", "response": "Given a list l will remove duplicates from the list and return a list with the original order of the list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse(s, g):\n # The CYK table. Indexed with a 2-tuple: (start pos, end pos)\n table = defaultdict(set)\n # Top-level structure is similar to the CYK table. Each cell is a dict from\n # rule name to the best (lightest) tree for that rule.\n trees = defaultdict(dict)\n # Populate base case with existing terminal production rules\n for i, w in enumerate(s):\n for terminal, rules in g.terminal_rules.items():\n if match(terminal, w):\n for rule in rules:\n table[(i, i)].add(rule)\n if (rule.lhs not in trees[(i, i)] or\n rule.weight < trees[(i, i)][rule.lhs].weight):\n trees[(i, i)][rule.lhs] = RuleNode(rule, [T(w)], weight=rule.weight)\n\n # Iterate over lengths of sub-sentences\n for l in xrange(2, len(s) + 1):\n # Iterate over sub-sentences with the given length\n for i in xrange(len(s) - l + 1):\n # Choose partition of the sub-sentence in [1, l)\n for p in xrange(i + 1, i + l):\n span1 = (i, p - 1)\n span2 = (p, i + l - 1)\n for r1, r2 in itertools.product(table[span1], table[span2]):\n for rule in g.nonterminal_rules.get((r1.lhs, r2.lhs), []):\n table[(i, i + l - 1)].add(rule)\n r1_tree = trees[span1][r1.lhs]\n r2_tree = trees[span2][r2.lhs]\n rule_total_weight = rule.weight + r1_tree.weight + r2_tree.weight\n if (rule.lhs not in trees[(i, i + l - 1)]\n or rule_total_weight < trees[(i, i + l - 1)][rule.lhs].weight):\n trees[(i, i + l - 1)][rule.lhs] = RuleNode(rule, [r1_tree, r2_tree], weight=rule_total_weight)\n return table, trees", "response": "Parses a sentence s using CNF grammar g."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_any_nt_unit_rule(g):\n for rule in g.rules:\n if len(rule.rhs) == 1 and isinstance(rule.rhs[0], NT):\n return rule\n return None", "response": "Returns a non - terminal unit rule from g or None if there is none."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nremoving a rule from g without changing the langugage produced by g.", "response": "def _remove_unit_rule(g, rule):\n \"\"\"Removes 'rule' from 'g' without changing the langugage produced by 'g'.\"\"\"\n new_rules = [x for x in g.rules if x != rule]\n refs = [x for x in g.rules if x.lhs == rule.rhs[0]]\n new_rules += [build_unit_skiprule(rule, ref) for ref in refs]\n return Grammar(new_rules)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsplit a rule into shorter rules.", "response": "def _split(rule):\n \"\"\"Splits a rule whose len(rhs) > 2 into shorter rules.\"\"\"\n rule_str = str(rule.lhs) + '__' + '_'.join(str(x) for x in rule.rhs)\n rule_name = '__SP_%s' % (rule_str) + '_%d'\n yield Rule(rule.lhs, [rule.rhs[0], NT(rule_name % 1)], weight=rule.weight, alias=rule.alias)\n for i in xrange(1, len(rule.rhs) - 2):\n yield Rule(NT(rule_name % i), [rule.rhs[i], NT(rule_name % (i + 1))], weight=0, alias='Split')\n yield Rule(NT(rule_name % (len(rule.rhs) - 2)), rule.rhs[-2:], weight=0, alias='Split')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\napplying the TERM rule on g.", "response": "def _term(g):\n \"\"\"Applies the TERM rule on 'g' (see top comment).\"\"\"\n all_t = {x for rule in g.rules for x in rule.rhs if isinstance(x, T)}\n t_rules = {t: Rule(NT('__T_%s' % str(t)), [t], weight=0, alias='Term') for t in all_t}\n new_rules = []\n for rule in g.rules:\n if len(rule.rhs) > 1 and any(isinstance(x, T) for x in rule.rhs):\n new_rhs = [t_rules[x].lhs if isinstance(x, T) else x for x in rule.rhs]\n new_rules.append(Rule(rule.lhs, new_rhs, weight=rule.weight, alias=rule.alias))\n new_rules.extend(v for k, v in t_rules.items() if k in rule.rhs)\n else:\n new_rules.append(rule)\n return Grammar(new_rules)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\napplying the BIN rule to g.", "response": "def _bin(g):\n \"\"\"Applies the BIN rule to 'g' (see top comment).\"\"\"\n new_rules = []\n for rule in g.rules:\n if len(rule.rhs) > 2:\n new_rules += _split(rule)\n else:\n new_rules.append(rule)\n return Grammar(new_rules)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\napply the UNIT rule to g.", "response": "def _unit(g):\n \"\"\"Applies the UNIT rule to 'g' (see top comment).\"\"\"\n nt_unit_rule = get_any_nt_unit_rule(g)\n while nt_unit_rule:\n g = _remove_unit_rule(g, nt_unit_rule)\n nt_unit_rule = get_any_nt_unit_rule(g)\n return g"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef revert_cnf(node):\n if isinstance(node, T):\n return node\n # Reverts TERM rule.\n if node.rule.lhs.name.startswith('__T_'):\n return node.children[0]\n else:\n children = []\n for child in map(revert_cnf, node.children):\n # Reverts BIN rule.\n if isinstance(child, RuleNode) and child.rule.lhs.name.startswith('__SP_'):\n children += child.children\n else:\n children.append(child)\n # Reverts UNIT rule.\n if isinstance(node.rule, UnitSkipRule):\n return unroll_unit_skiprule(node.rule.lhs, node.rule.rhs,\n node.rule.skipped_rules, children,\n node.rule.weight, node.rule.alias)\n else:\n return RuleNode(node.rule, children)", "response": "Reverts a parse tree ( RuleNode to its original non - CNF form ( Node )."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a lark rule from a Lark VM to a Rule.", "response": "def _to_rule(self, lark_rule):\n \"\"\"Converts a lark rule, (lhs, rhs, callback, options), to a Rule.\"\"\"\n assert isinstance(lark_rule.origin, NT)\n assert all(isinstance(x, Symbol) for x in lark_rule.expansion)\n return Rule(\n lark_rule.origin, lark_rule.expansion,\n weight=lark_rule.options.priority if lark_rule.options and lark_rule.options.priority else 0,\n alias=lark_rule)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse(self, tokenized): # pylint: disable=invalid-name\n table, trees = _parse(tokenized, self.grammar)\n # Check if the parse succeeded.\n if all(r.lhs != self.start for r in table[(0, len(tokenized) - 1)]):\n raise ParseError('Parsing failed.')\n parse = trees[(0, len(tokenized) - 1)][self.start]\n return self._to_tree(revert_cnf(parse))", "response": "Parses input which is a list of tokens."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _to_tree(self, rule_node):\n orig_rule = self.orig_rules[rule_node.rule.alias]\n children = []\n for child in rule_node.children:\n if isinstance(child, RuleNode):\n children.append(self._to_tree(child))\n else:\n assert isinstance(child.name, Token)\n children.append(child.name)\n t = Tree(orig_rule.origin, children)\n t.rule=orig_rule\n return t", "response": "Converts a RuleNode parse tree to a lark Tree."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pydot__tree_to_png(tree, filename, rankdir=\"LR\"):\n\n import pydot\n graph = pydot.Dot(graph_type='digraph', rankdir=rankdir)\n\n i = [0]\n\n def new_leaf(leaf):\n node = pydot.Node(i[0], label=repr(leaf))\n i[0] += 1\n graph.add_node(node)\n return node\n\n def _to_pydot(subtree):\n color = hash(subtree.data) & 0xffffff\n color |= 0x808080\n\n subnodes = [_to_pydot(child) if isinstance(child, Tree) else new_leaf(child)\n for child in subtree.children]\n node = pydot.Node(i[0], style=\"filled\", fillcolor=\"#%x\"%color, label=subtree.data)\n i[0] += 1\n graph.add_node(node)\n\n for subnode in subnodes:\n graph.add_edge(pydot.Edge(node, subnode))\n\n return node\n\n _to_pydot(tree)\n graph.write_png(filename)", "response": "Writes a colorful image that represents the tree."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexpanding ( inline ) children at the given indices", "response": "def expand_kids_by_index(self, *indices):\n \"Expand (inline) children at the given indices\"\n for i in sorted(indices, reverse=True): # reverse so that changing tail won't affect indices\n kid = self.children[i]\n self.children[i:i+1] = kid.children"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a parser instance and a dictionary mapping some label with the same name and a list of malformed syntax examples it ll return the label for the other examples that bests matches the current error.", "response": "def match_examples(self, parse_fn, examples):\n \"\"\" Given a parser instance and a dictionary mapping some label with\n some malformed syntax examples, it'll return the label for the\n example that bests matches the current error.\n \"\"\"\n assert self.state is not None, \"Not supported for this exception\"\n\n candidate = None\n for label, example in examples.items():\n assert not isinstance(example, STRING_TYPE)\n\n for malformed in example:\n try:\n parse_fn(malformed)\n except UnexpectedInput as ut:\n if ut.state == self.state:\n try:\n if ut.token == self.token: # Try exact match first\n return label\n except AttributeError:\n pass\n if not candidate:\n candidate = label\n\n return candidate"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef feed(self, token, test_newline=True):\n if test_newline:\n newlines = token.count(self.newline_char)\n if newlines:\n self.line += newlines\n self.line_start_pos = self.char_pos + token.rindex(self.newline_char) + 1\n\n self.char_pos += len(token)\n self.column = self.char_pos - self.line_start_pos + 1", "response": "Consume a token and calculate the new line & column."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates an instance of Lark with the grammar given by its filename.", "response": "def open(cls, grammar_filename, rel_to=None, **options):\n \"\"\"Create an instance of Lark with the grammar given by its filename\n\n If rel_to is provided, the function will find the grammar filename in relation to it.\n\n Example:\n\n >>> Lark.open(\"grammar_file.lark\", rel_to=__file__, parser=\"lalr\")\n Lark(...)\n\n \"\"\"\n if rel_to:\n basepath = os.path.dirname(rel_to)\n grammar_filename = os.path.join(basepath, grammar_filename)\n with open(grammar_filename, encoding='utf8') as f:\n return cls(f, **options)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate FOLLOW sets. Adapted from: http://lara.epfl.ch/w/cc09:algorithm_for_first_and_follow_sets", "response": "def calculate_sets(rules):\n \"\"\"Calculate FOLLOW sets.\n\n Adapted from: http://lara.epfl.ch/w/cc09:algorithm_for_first_and_follow_sets\"\"\"\n symbols = {sym for rule in rules for sym in rule.expansion} | {rule.origin for rule in rules}\n\n # foreach grammar rule X ::= Y(1) ... Y(k)\n # if k=0 or {Y(1),...,Y(k)} subset of NULLABLE then\n # NULLABLE = NULLABLE union {X}\n # for i = 1 to k\n # if i=1 or {Y(1),...,Y(i-1)} subset of NULLABLE then\n # FIRST(X) = FIRST(X) union FIRST(Y(i))\n # for j = i+1 to k\n # if i=k or {Y(i+1),...Y(k)} subset of NULLABLE then\n # FOLLOW(Y(i)) = FOLLOW(Y(i)) union FOLLOW(X)\n # if i+1=j or {Y(i+1),...,Y(j-1)} subset of NULLABLE then\n # FOLLOW(Y(i)) = FOLLOW(Y(i)) union FIRST(Y(j))\n # until none of NULLABLE,FIRST,FOLLOW changed in last iteration\n\n NULLABLE = set()\n FIRST = {}\n FOLLOW = {}\n for sym in symbols:\n FIRST[sym]={sym} if sym.is_term else set()\n FOLLOW[sym]=set()\n\n # Calculate NULLABLE and FIRST\n changed = True\n while changed:\n changed = False\n\n for rule in rules:\n if set(rule.expansion) <= NULLABLE:\n if update_set(NULLABLE, {rule.origin}):\n changed = True\n\n for i, sym in enumerate(rule.expansion):\n if set(rule.expansion[:i]) <= NULLABLE:\n if update_set(FIRST[rule.origin], FIRST[sym]):\n changed = True\n\n # Calculate FOLLOW\n changed = True\n while changed:\n changed = False\n\n for rule in rules:\n for i, sym in enumerate(rule.expansion):\n if i==len(rule.expansion)-1 or set(rule.expansion[i+1:]) <= NULLABLE:\n if update_set(FOLLOW[sym], FOLLOW[rule.origin]):\n changed = True\n\n for j in range(i+1, len(rule.expansion)):\n if set(rule.expansion[i+1:j]) <= NULLABLE:\n if update_set(FOLLOW[sym], FIRST[rule.expansion[j]]):\n changed = True\n\n return FIRST, FOLLOW, NULLABLE"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all init_ptrs accessible by rule ( recursive )", "response": "def expand_rule(self, rule):\n \"Returns all init_ptrs accessible by rule (recursive)\"\n init_ptrs = set()\n def _expand_rule(rule):\n assert not rule.is_term, rule\n\n for r in self.rules_by_origin[rule]:\n init_ptr = RulePtr(r, 0)\n init_ptrs.add(init_ptr)\n\n if r.expansion: # if not empty rule\n new_r = init_ptr.next\n if not new_r.is_term:\n yield new_r\n\n for _ in bfs([rule], _expand_rule):\n pass\n\n return fzset(init_ptrs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn all rules and terminals of grammar prepended by a namespace prefix except for those which are aliased.", "response": "def import_from_grammar_into_namespace(grammar, namespace, aliases):\n \"\"\"Returns all rules and terminals of grammar, prepended\n with a 'namespace' prefix, except for those which are aliased.\n \"\"\"\n\n imported_terms = dict(grammar.term_defs)\n imported_rules = {n:(n,deepcopy(t),o) for n,t,o in grammar.rule_defs}\n\n term_defs = []\n rule_defs = []\n\n def rule_dependencies(symbol):\n if symbol.type != 'RULE':\n return []\n try:\n _, tree, _ = imported_rules[symbol]\n except KeyError:\n raise GrammarError(\"Missing symbol '%s' in grammar %s\" % (symbol, namespace))\n return tree.scan_values(lambda x: x.type in ('RULE', 'TERMINAL'))\n\n def get_namespace_name(name):\n try:\n return aliases[name].value\n except KeyError:\n if name[0] == '_':\n return '_%s__%s' % (namespace, name[1:])\n return '%s__%s' % (namespace, name)\n\n to_import = list(bfs(aliases, rule_dependencies))\n for symbol in to_import:\n if symbol.type == 'TERMINAL':\n term_defs.append([get_namespace_name(symbol), imported_terms[symbol]])\n else:\n assert symbol.type == 'RULE'\n rule = imported_rules[symbol]\n for t in rule[1].iter_subtrees():\n for i, c in enumerate(t.children):\n if isinstance(c, Token) and c.type in ('RULE', 'TERMINAL'):\n t.children[i] = Token(c.type, get_namespace_name(c))\n rule_defs.append((get_namespace_name(symbol), rule[1], rule[2]))\n\n return term_defs, rule_defs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing grammar_text verify and create Grammar object. Display nice messages on error.", "response": "def load_grammar(self, grammar_text, grammar_name=''):\n \"Parse grammar_text, verify, and create Grammar object. Display nice messages on error.\"\n\n try:\n tree = self.canonize_tree.transform( self.parser.parse(grammar_text+'\\n') )\n except UnexpectedCharacters as e:\n context = e.get_context(grammar_text)\n raise GrammarError(\"Unexpected input at line %d column %d in %s: \\n\\n%s\" %\n (e.line, e.column, grammar_name, context))\n except UnexpectedToken as e:\n context = e.get_context(grammar_text)\n error = e.match_examples(self.parser.parse, {\n 'Unclosed parenthesis': ['a: (\\n'],\n 'Umatched closing parenthesis': ['a: )\\n', 'a: [)\\n', 'a: (]\\n'],\n 'Expecting rule or terminal definition (missing colon)': ['a\\n', 'a->\\n', 'A->\\n', 'a A\\n'],\n 'Alias expects lowercase name': ['a: -> \"a\"\\n'],\n 'Unexpected colon': ['a::\\n', 'a: b:\\n', 'a: B:\\n', 'a: \"a\":\\n'],\n 'Misplaced operator': ['a: b??', 'a: b(?)', 'a:+\\n', 'a:?\\n', 'a:*\\n', 'a:|*\\n'],\n 'Expecting option (\"|\") or a new rule or terminal definition': ['a:a\\n()\\n'],\n '%import expects a name': ['%import \"a\"\\n'],\n '%ignore expects a value': ['%ignore %import\\n'],\n })\n if error:\n raise GrammarError(\"%s at line %s column %s\\n\\n%s\" % (error, e.line, e.column, context))\n elif 'STRING' in e.expected:\n raise GrammarError(\"Expecting a value at line %s column %s\\n\\n%s\" % (e.line, e.column, context))\n raise\n\n tree = PrepareGrammar().transform(tree)\n\n # Extract grammar items\n defs = classify(tree.children, lambda c: c.data, lambda c: c.children)\n term_defs = defs.pop('term', [])\n rule_defs = defs.pop('rule', [])\n statements = defs.pop('statement', [])\n assert not defs\n\n term_defs = [td if len(td)==3 else (td[0], 1, td[1]) for td in term_defs]\n term_defs = [(name.value, (t, int(p))) for name, p, t in term_defs]\n rule_defs = [options_from_rule(*x) for x in rule_defs]\n\n # Execute statements\n ignore, imports = [], {}\n for (stmt,) in statements:\n if stmt.data == 'ignore':\n t ,= stmt.children\n ignore.append(t)\n elif stmt.data == 'import':\n if len(stmt.children) > 1:\n path_node, arg1 = stmt.children\n else:\n path_node, = stmt.children\n arg1 = None\n\n if isinstance(arg1, Tree): # Multi import\n dotted_path = tuple(path_node.children)\n names = arg1.children\n aliases = dict(zip(names, names)) # Can't have aliased multi import, so all aliases will be the same as names\n else: # Single import\n dotted_path = tuple(path_node.children[:-1])\n name = path_node.children[-1] # Get name from dotted path\n aliases = {name: arg1 or name} # Aliases if exist\n\n if path_node.data == 'import_lib': # Import from library\n base_paths = []\n else: # Relative import\n if grammar_name == '': # Import relative to script file path if grammar is coded in script\n try:\n base_file = os.path.abspath(sys.modules['__main__'].__file__)\n except AttributeError:\n base_file = None\n else:\n base_file = grammar_name # Import relative to grammar file path if external grammar file\n if base_file:\n base_paths = [os.path.split(base_file)[0]]\n else:\n base_paths = [os.path.abspath(os.path.curdir)]\n\n try:\n import_base_paths, import_aliases = imports[dotted_path]\n assert base_paths == import_base_paths, 'Inconsistent base_paths for %s.' % '.'.join(dotted_path)\n import_aliases.update(aliases)\n except KeyError:\n imports[dotted_path] = base_paths, aliases\n\n elif stmt.data == 'declare':\n for t in stmt.children:\n term_defs.append([t.value, (None, None)])\n else:\n assert False, stmt\n\n # import grammars\n for dotted_path, (base_paths, aliases) in imports.items():\n grammar_path = os.path.join(*dotted_path) + EXT\n g = import_grammar(grammar_path, base_paths=base_paths)\n new_td, new_rd = import_from_grammar_into_namespace(g, '__'.join(dotted_path), aliases)\n\n term_defs += new_td\n rule_defs += new_rd\n\n # Verify correctness 1\n for name, _ in term_defs:\n if name.startswith('__'):\n raise GrammarError('Names starting with double-underscore are reserved (Error at %s)' % name)\n\n # Handle ignore tokens\n # XXX A slightly hacky solution. Recognition of %ignore TERMINAL as separate comes from the lexer's\n # inability to handle duplicate terminals (two names, one value)\n ignore_names = []\n for t in ignore:\n if t.data=='expansions' and len(t.children) == 1:\n t2 ,= t.children\n if t2.data=='expansion' and len(t2.children) == 1:\n item ,= t2.children\n if item.data == 'value':\n item ,= item.children\n if isinstance(item, Token) and item.type == 'TERMINAL':\n ignore_names.append(item.value)\n continue\n\n name = '__IGNORE_%d'% len(ignore_names)\n ignore_names.append(name)\n term_defs.append((name, (t, 1)))\n\n # Verify correctness 2\n terminal_names = set()\n for name, _ in term_defs:\n if name in terminal_names:\n raise GrammarError(\"Terminal '%s' defined more than once\" % name)\n terminal_names.add(name)\n\n if set(ignore_names) > terminal_names:\n raise GrammarError(\"Terminals %s were marked to ignore but were not defined!\" % (set(ignore_names) - terminal_names))\n\n resolve_term_references(term_defs)\n\n rules = rule_defs\n\n rule_names = set()\n for name, _x, _o in rules:\n if name.startswith('__'):\n raise GrammarError('Names starting with double-underscore are reserved (Error at %s)' % name)\n if name in rule_names:\n raise GrammarError(\"Rule '%s' defined more than once\" % name)\n rule_names.add(name)\n\n for name, expansions, _o in rules:\n used_symbols = {t for x in expansions.find_data('expansion')\n for t in x.scan_values(lambda t: t.type in ('RULE', 'TERMINAL'))}\n for sym in used_symbols:\n if sym.type == 'TERMINAL':\n if sym not in terminal_names:\n raise GrammarError(\"Token '%s' used but not defined (in rule %s)\" % (sym, name))\n else:\n if sym not in rule_names:\n raise GrammarError(\"Rule '%s' used but not defined (in rule %s)\" % (sym, name))\n\n return Grammar(rules, term_defs, ignore_names)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\napplying the style rules to the radio group.", "response": "def applyStyleRules(self):\r\n \"\"\"\r\n Conditionally disabled/enables form fields based on the current\r\n section in the radio group\r\n \"\"\"\r\n for button, widget in zip(self.radioButtons, self.widgets):\r\n if isinstance(widget, CheckBox):\r\n widget.hideInput()\r\n if not button.GetValue(): # not checked\r\n widget.widget.Disable()\r\n else:\r\n widget.widget.Enable()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef handleImplicitCheck(self):\r\n for button, widget in zip(self.radioButtons, self.widgets):\r\n if isinstance(widget, CheckBox):\r\n if button.GetValue(): # checked\r\n widget.setValue(True)\r\n else:\r\n widget.setValue(False)", "response": "Handles implicit check of the Radio button."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef createWidgets(self):\r\n from gooey.gui.components import widgets\r\n return [getattr(widgets, item['type'])(self, item)\r\n for item in getin(self.widgetInfo, ['data', 'widgets'], [])]", "response": "Instantiate the Gooey Widgets that are used within the RadioGroup"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef onStart(self, *args, **kwarg):\r\n with transactUI(self):\r\n config = self.navbar.getActiveConfig()\r\n config.resetErrors()\r\n if config.isValid():\r\n self.clientRunner.run(self.buildCliString())\r\n self.showConsole()\r\n else:\r\n config.displayErrors()\r\n self.Layout()", "response": "Called when the user is ready to start the client."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef onEdit(self):\r\n with transactUI(self):\r\n if self.buildSpec['poll_external_updates']:\r\n self.fetchExternalUpdates()\r\n self.showSettings()", "response": "Return the user to the settings screen for further editing"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef buildCliString(self):\r\n config = self.navbar.getActiveConfig()\r\n group = self.buildSpec['widgets'][self.navbar.getSelectedGroup()]\r\n positional = config.getPositionalArgs()\r\n optional = config.getOptionalArgs()\r\n print(cli.buildCliString(\r\n self.buildSpec['target'],\r\n group['command'],\r\n positional,\r\n optional\r\n ))\r\n return cli.buildCliString(\r\n self.buildSpec['target'],\r\n group['command'],\r\n positional,\r\n optional\r\n )", "response": "Collect all of the required information from the config screen and build a CLI string which can be used to invoke the client program"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall when the client is finished.", "response": "def onComplete(self, *args, **kwargs):\r\n \"\"\"\r\n Display the appropriate screen based on the success/fail of the\r\n host program\r\n \"\"\"\r\n with transactUI(self):\r\n if self.clientRunner.was_success():\r\n if self.buildSpec.get('return_to_config', False):\r\n self.showSettings()\r\n else:\r\n self.showSuccess()\r\n if self.buildSpec.get('show_success_modal', True):\r\n wx.CallAfter(modals.showSuccess)\r\n else:\r\n if self.clientRunner.wasForcefullyStopped:\r\n self.showForceStopped()\r\n else:\r\n self.showError()\r\n wx.CallAfter(modals.showFailure)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fetchExternalUpdates(self):\r\n seeds = seeder.fetchDynamicProperties(\r\n self.buildSpec['target'],\r\n self.buildSpec['encoding']\r\n )\r\n for config in self.configs:\r\n config.seedUI(seeds)", "response": "Fetch the seed values from the seeder and update the UI"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the appropriate layout navigation component based on user preferences", "response": "def buildNavigation(self):\r\n \"\"\"\r\n Chooses the appropriate layout navigation component based on user prefs\r\n \"\"\"\r\n if self.buildSpec['navigation'] == constants.TABBED:\r\n navigation = Tabbar(self, self.buildSpec, self.configs)\r\n else:\r\n navigation = Sidebar(self, self.buildSpec, self.configs)\r\n if self.buildSpec['navigation'] == constants.HIDDEN:\r\n navigation.Hide()\r\n return navigation"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getFontFace(self):\r\n userFace = self.buildSpec['terminal_font_family'] or self.defaultFont.GetFaceName()\r\n return (''\r\n if self.buildSpec['monospace_display']\r\n else userFace)", "response": "Choose the best font face available given the user options"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _forward_stdout(self, process):\r\n '''\r\n Reads the stdout of `process` and forwards lines and progress\r\n to any interested subscribers\r\n '''\r\n while True:\r\n line = process.stdout.readline()\r\n if not line:\r\n break\r\n pub.send_message(events.CONSOLE_UPDATE, msg=line.decode(self.encoding))\r\n pub.send_message(events.PROGRESS_UPDATE,\r\n progress=self._extract_progress(line))\r\n pub.send_message(events.EXECUTION_COMPLETE)", "response": "Reads the stdout of process and forwards lines and progress to any interested subscribers."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract the progress information from the text using the progress_regex and calculate instructions", "response": "def _extract_progress(self, text):\r\n '''\r\n Finds progress information in the text using the\r\n user-supplied regex and calculation instructions\r\n '''\r\n # monad-ish dispatch to avoid the if/else soup\r\n find = partial(re.search, string=text.strip().decode(self.encoding))\r\n regex = unit(self.progress_regex)\r\n match = bind(regex, find)\r\n result = bind(match, self._calculate_progress)\r\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the final progress value found by the regex match", "response": "def _calculate_progress(self, match):\r\n '''\r\n Calculates the final progress value found by the regex\r\n '''\r\n if not self.progress_expr:\r\n return safe_float(match.group(1))\r\n else:\r\n return self._eval_progress(match)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nhides and show configuration panels based on the currently selected option in the sidebar", "response": "def swapConfigPanels(self, event):\r\n \"\"\"Hide/show configuration panels based on the currently selected\r\n option in the sidebar \"\"\"\r\n for id, panel in enumerate(self.configPanels):\r\n panel.Hide()\r\n self.activeSelection = event.Selection\r\n self.configPanels[event.Selection].Show()\r\n self._parent.Layout()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef updateProgressBar(self, *args, **kwargs):\r\n '''\r\n value, disable_animation=False\r\n :param args:\r\n :param kwargs:\r\n :return:\r\n '''\r\n value = kwargs.get('progress')\r\n pb = self.progress_bar\r\n if value is None:\r\n return\r\n if value < 0:\r\n pb.Pulse()\r\n else:\r\n value = min(int(value), pb.GetRange())\r\n if pb.GetValue() != value:\r\n # Windows 7 progress bar animation hack\r\n # http://stackoverflow.com/questions/5332616/disabling-net-progressbar-animation-when-changing-value\r\n if self.buildSpec['disable_progress_bar_animation'] \\\r\n and sys.platform.startswith(\"win\"):\r\n if pb.GetRange() == value:\r\n pb.SetValue(value)\r\n pb.SetValue(value - 1)\r\n else:\r\n pb.SetValue(value + 1)\r\n pb.SetValue(value)", "response": "Updates the progress bar with the new value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fetchDynamicProperties(target, encoding):\r\n cmd = '{} {}'.format(target, 'gooey-seed-ui --ignore-gooey')\r\n proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\r\n if proc.returncode != 0:\r\n out, _ = proc.communicate()\r\n return json.loads(out.decode(encoding))\r\n else:\r\n # TODO: useful feedback\r\n return {}", "response": "Sends a gooey - seed - ui request to the client program it retrieve dynamically generated defaults with which to seed the UI"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a group of items and returns a wx. BoxSizer instance that can be used to create the group of items.", "response": "def makeGroup(self, parent, thissizer, group, *args):\r\n '''\r\n Messily builds the (potentially) nested and grouped layout\r\n\r\n Note! Mutates `self.reifiedWidgets` in place with the widgets as they're\r\n instantiated! I cannot figure out how to split out the creation of the\r\n widgets from their styling without WxPython violently exploding\r\n\r\n TODO: sort out the WX quirks and clean this up.\r\n '''\r\n\r\n # determine the type of border , if any, the main sizer will use\r\n if getin(group, ['options', 'show_border'], False):\r\n boxDetails = wx.StaticBox(parent, -1, group['name'] or '')\r\n boxSizer = wx.StaticBoxSizer(boxDetails, wx.VERTICAL)\r\n else:\r\n boxSizer = wx.BoxSizer(wx.VERTICAL)\r\n boxSizer.AddSpacer(10)\r\n if group['name']:\r\n boxSizer.Add(wx_util.h1(parent, group['name'] or ''), 0, wx.TOP | wx.BOTTOM | wx.LEFT, 8)\r\n\r\n group_description = getin(group, ['description'])\r\n if group_description:\r\n description = wx.StaticText(parent, label=group_description)\r\n boxSizer.Add(description, 0, wx.EXPAND | wx.LEFT, 10)\r\n\r\n # apply an underline when a grouping border is not specified\r\n if not getin(group, ['options', 'show_border'], False) and group['name']:\r\n boxSizer.Add(wx_util.horizontal_rule(parent), 0, wx.EXPAND | wx.LEFT, 10)\r\n\r\n ui_groups = self.chunkWidgets(group)\r\n\r\n for uigroup in ui_groups:\r\n sizer = wx.BoxSizer(wx.HORIZONTAL)\r\n for item in uigroup:\r\n widget = self.reifyWidget(parent, item)\r\n # !Mutate the reifiedWidgets instance variable in place\r\n self.reifiedWidgets.append(widget)\r\n sizer.Add(widget, 1, wx.ALL, 5)\r\n boxSizer.Add(sizer, 0, wx.ALL | wx.EXPAND, 5)\r\n\r\n # apply the same layout rules recursively for subgroups\r\n hs = wx.BoxSizer(wx.HORIZONTAL)\r\n for e, subgroup in enumerate(group['groups']):\r\n self.makeGroup(parent, hs, subgroup, 1, wx.ALL | wx.EXPAND, 5)\r\n if e % getin(group, ['options', 'columns'], 2) \\\r\n or e == len(group['groups']):\r\n boxSizer.Add(hs, *args)\r\n hs = wx.BoxSizer(wx.HORIZONTAL)\r\n\r\n thissizer.Add(boxSizer, *args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nchunk the widgets up into groups based on their sizing hints", "response": "def chunkWidgets(self, group):\r\n ''' chunk the widgets up into groups based on their sizing hints '''\r\n ui_groups = []\r\n subgroup = []\r\n for index, item in enumerate(group['items']):\r\n if getin(item, ['options', 'full_width'], False):\r\n ui_groups.append(subgroup)\r\n ui_groups.append([item])\r\n subgroup = []\r\n else:\r\n subgroup.append(item)\r\n if len(subgroup) == getin(group, ['options', 'columns'], 2) \\\r\n or item == group['items'][-1]:\r\n ui_groups.append(subgroup)\r\n subgroup = []\r\n return ui_groups"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a JSON description of a widget into a WxObject", "response": "def reifyWidget(self, parent, item):\r\n ''' Convert a JSON description of a widget into a WxObject '''\r\n from gooey.gui.components import widgets\r\n widgetClass = getattr(widgets, item['type'])\r\n return widgetClass(parent, item)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns true if the given action is required.", "response": "def is_required(action):\r\n '''\r\n _actions possessing the `required` flag and not implicitly optional\r\n through `nargs` being '*' or '?'\r\n '''\r\n return not isinstance(action, _SubParsersAction) and (\r\n action.required == True and action.nargs not in ['*', '?'])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the given action is standard.", "response": "def is_standard(action):\r\n \"\"\" actions which are general \"store\" instructions.\r\n e.g. anything which has an argument style like:\r\n $ script.py -f myfilename.txt\r\n \"\"\"\r\n boolean_actions = (\r\n _StoreConstAction, _StoreFalseAction,\r\n _StoreTrueAction\r\n )\r\n return (not action.choices\r\n and not isinstance(action, _CountAction)\r\n and not isinstance(action, _HelpAction)\r\n and type(action) not in boolean_actions)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if the given action is a flag.", "response": "def is_flag(action):\r\n \"\"\" _actions which are either storeconst, store_bool, etc.. \"\"\"\r\n action_types = [_StoreTrueAction, _StoreFalseAction, _StoreConstAction]\r\n return any(list(map(lambda Action: isinstance(action, Action), action_types)))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the string that will be used to drop down the next value", "response": "def counter(metatdata, value):\r\n '''\r\n Returns\r\n str(option_string * DropDown Value)\r\n e.g.\r\n -vvvvv\r\n '''\r\n if not str(value).isdigit():\r\n return None\r\n arg = str(metatdata['commands'][0]).replace('-', '')\r\n repeated_args = arg * int(value)\r\n return '-' + repeated_args"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load(language_dir, filename, encoding):\r\n ''' Open and return the supplied json file '''\r\n global _DICTIONARY\r\n try:\r\n json_file = filename + '.json'\r\n with io.open(os.path.join(language_dir, json_file), 'r', encoding=encoding) as f:\r\n _DICTIONARY = json.load(f)\r\n except IOError:\r\n raise IOError('{0} Language file not found at location {1}. '\r\n 'Make sure that your translation file is in the '\r\n 'listed language directory'.format(filename.title(), language_dir))", "response": "Open and return the supplied json file"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the value in a nested dict", "response": "def getin(m, path, default=None):\r\n \"\"\"returns the value in a nested dict\"\"\"\r\n keynotfound = ':com.gooey-project/not-found'\r\n result = reduce(lambda acc, val: acc.get(val, {keynotfound: None}), path, m)\r\n # falsey values like 0 would incorrectly trigger the default to be returned\r\n # so the keynotfound val is used to signify a miss vs just a falesy val\r\n if isinstance(result, dict) and keynotfound in result:\r\n return default\r\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncopies - on - write associates a value in a dict", "response": "def assoc(m, key, val):\r\n \"\"\"Copy-on-write associates a value in a dict\"\"\"\r\n cpy = deepcopy(m)\r\n cpy[key] = val\r\n return cpy"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncopy - on - write associates a value in a nested dict", "response": "def associn(m, path, value):\r\n \"\"\" Copy-on-write associates a value in a nested dict \"\"\"\r\n def assoc_recursively(m, path, value):\r\n if not path:\r\n return value\r\n p = path[0]\r\n return assoc(m, p, assoc_recursively(m.get(p,{}), path[1:], value))\r\n return assoc_recursively(m, path, value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmerging all maps left to right", "response": "def merge(*maps):\r\n \"\"\"Merge all maps left to right\"\"\"\r\n copies = map(deepcopy, maps)\r\n return reduce(lambda acc, val: acc.update(val) or acc, copies)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef findfirst(f, coll):\r\n result = list(dropwhile(f, coll))\r\n return result[0] if result else None", "response": "Return first occurrence matching f otherwise None"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate the input_tensor that minimizes the weighted losses.", "response": "def visualize_activation_with_losses(input_tensor, losses, wrt_tensor=None,\n seed_input=None, input_range=(0, 255),\n **optimizer_params):\n \"\"\"Generates the `input_tensor` that minimizes the weighted `losses`. This function is intended for advanced\n use cases where a custom loss is desired.\n\n Args:\n input_tensor: An input tensor of shape: `(samples, channels, image_dims...)` if `image_data_format=\n channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.\n wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.\n When None, this is assumed to be the same as `input_tensor` (Default value: None)\n losses: List of ([Loss](vis.losses#Loss), weight) tuples.\n seed_input: Seeds the optimization with a starting image. Initialized with a random value when set to None.\n (Default value = None)\n input_range: Specifies the input range as a `(min, max)` tuple. This is used to rescale the\n final optimized input to the given range. (Default value=(0, 255))\n optimizer_params: The **kwargs for optimizer [params](vis.optimizer#optimizerminimize). Will default to\n reasonable values when required keys are not found.\n\n Returns:\n The model input that minimizes the weighted `losses`.\n \"\"\"\n # Default optimizer kwargs.\n optimizer_params = utils.add_defaults_to_kwargs({\n 'seed_input': seed_input,\n 'max_iter': 200,\n 'verbose': False\n }, **optimizer_params)\n\n opt = Optimizer(input_tensor, losses, input_range, wrt_tensor=wrt_tensor)\n img = opt.minimize(**optimizer_params)[0]\n\n # If range has integer numbers, cast to 'uint8'\n if isinstance(input_range[0], int) and isinstance(input_range[1], int):\n img = np.clip(img, input_range[0], input_range[1]).astype('uint8')\n\n if K.image_data_format() == 'channels_first':\n img = np.moveaxis(img, 0, -1)\n return img"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef visualize_activation(model, layer_idx, filter_indices=None, wrt_tensor=None,\n seed_input=None, input_range=(0, 255),\n backprop_modifier=None, grad_modifier=None,\n act_max_weight=1, lp_norm_weight=10, tv_weight=10,\n **optimizer_params):\n \"\"\"Generates the model input that maximizes the output of all `filter_indices` in the given `layer_idx`.\n\n Args:\n model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`\n if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if\n `image_data_format=channels_last`.\n layer_idx: The layer index within `model.layers` whose filters needs to be visualized.\n filter_indices: filter indices within the layer to be maximized.\n If None, all filters are visualized. (Default value = None)\n For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.\n If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for\n 'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.\n wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.\n When None, this is assumed to be the same as `input_tensor` (Default value: None)\n seed_input: Seeds the optimization with a starting input. Initialized with a random value when set to None.\n (Default value = None)\n input_range: Specifies the input range as a `(min, max)` tuple. This is used to rescale the\n final optimized input to the given range. (Default value=(0, 255))\n backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't\n specify anything, no backprop modification is applied. (Default value = None)\n grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't\n specify anything, gradients are unchanged (Default value = None)\n act_max_weight: The weight param for `ActivationMaximization` loss. Not used if 0 or None. (Default value = 1)\n lp_norm_weight: The weight param for `LPNorm` regularization loss. Not used if 0 or None. (Default value = 10)\n tv_weight: The weight param for `TotalVariation` regularization loss. Not used if 0 or None. (Default value = 10)\n optimizer_params: The **kwargs for optimizer [params](vis.optimizer#optimizerminimize). Will default to\n reasonable values when required keys are not found.\n\n Example:\n If you wanted to visualize the input image that would maximize the output index 22, say on\n final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer_idx = dense_layer_idx`.\n\n If `filter_indices = [22, 23]`, then it should generate an input image that shows features of both classes.\n\n Returns:\n The model input that maximizes the output of `filter_indices` in the given `layer_idx`.\n \"\"\"\n if backprop_modifier is not None:\n modifier_fn = get(backprop_modifier)\n model = modifier_fn(model)\n\n losses = [\n (ActivationMaximization(model.layers[layer_idx], filter_indices), act_max_weight),\n (LPNorm(model.input), lp_norm_weight),\n (TotalVariation(model.input), tv_weight)\n ]\n\n # Add grad_filter to optimizer_params.\n optimizer_params = utils.add_defaults_to_kwargs({\n 'grad_modifier': grad_modifier\n }, **optimizer_params)\n\n return visualize_activation_with_losses(model.input, losses, wrt_tensor,\n seed_input, input_range, **optimizer_params)", "response": "Generates the model input that maximizes the output of all filter_indices in the given layer_idx."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nuse RMSProp to compute step from gradients.", "response": "def _rmsprop(self, grads, cache=None, decay_rate=0.95):\n \"\"\"Uses RMSProp to compute step from gradients.\n\n Args:\n grads: numpy array of gradients.\n cache: numpy array of same shape as `grads` as RMSProp cache\n decay_rate: How fast to decay cache\n\n Returns:\n A tuple of\n step: numpy array of the same shape as `grads` giving the step.\n Note that this does not yet take the learning rate into account.\n cache: Updated RMSProp cache.\n \"\"\"\n if cache is None:\n cache = np.zeros_like(grads)\n cache = decay_rate * cache + (1 - decay_rate) * grads ** 2\n step = -grads / np.sqrt(cache + K.epsilon())\n return step, cache"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a random seed_input if None. Otherwise returns a random seed_input.", "response": "def _get_seed_input(self, seed_input):\n \"\"\"Creates a random `seed_input` if None. Otherwise:\n - Ensures batch_size dim on provided `seed_input`.\n - Shuffle axis according to expected `image_data_format`.\n \"\"\"\n desired_shape = (1, ) + K.int_shape(self.input_tensor)[1:]\n if seed_input is None:\n return utils.random_array(desired_shape, mean=np.mean(self.input_range),\n std=0.05 * (self.input_range[1] - self.input_range[0]))\n\n # Add batch dim if needed.\n if len(seed_input.shape) != len(desired_shape):\n seed_input = np.expand_dims(seed_input, 0)\n\n # Only possible if channel idx is out of place.\n if seed_input.shape[-1] != desired_shape[-1] and \\\n seed_input.shape[1] != desired_shape[1]:\n seed_input = np.moveaxis(seed_input, -1, 1)\n return seed_input.astype(K.floatx())"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nperform gradient descent on the input image with respect to defined losses.", "response": "def minimize(self, seed_input=None, max_iter=200,\n input_modifiers=None, grad_modifier=None,\n callbacks=None, verbose=True):\n \"\"\"Performs gradient descent on the input image with respect to defined losses.\n\n Args:\n seed_input: An N-dim numpy array of shape: `(samples, channels, image_dims...)` if `image_data_format=\n channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.\n Seeded with random noise if set to None. (Default value = None)\n max_iter: The maximum number of gradient descent iterations. (Default value = 200)\n input_modifiers: A list of [InputModifier](vis.input_modifiers#inputmodifier) instances specifying\n how to make `pre` and `post` changes to the optimized input during the optimization process.\n `pre` is applied in list order while `post` is applied in reverse order. For example,\n `input_modifiers = [f, g]` means that `pre_input = g(f(inp))` and `post_input = f(g(inp))`\n grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't\n specify anything, gradients are unchanged. (Default value = None)\n callbacks: A list of [OptimizerCallback](vis.callbacks#optimizercallback) instances to trigger.\n verbose: Logs individual losses at the end of every gradient descent iteration.\n Very useful to estimate loss weight factor(s). (Default value = True)\n\n Returns:\n The tuple of `(optimized input, grads with respect to wrt, wrt_value)` after gradient descent iterations.\n \"\"\"\n seed_input = self._get_seed_input(seed_input)\n input_modifiers = input_modifiers or []\n grad_modifier = _identity if grad_modifier is None else get(grad_modifier)\n\n callbacks = callbacks or []\n if verbose:\n callbacks.append(_PRINT_CALLBACK)\n\n cache = None\n best_loss = float('inf')\n best_input = None\n\n grads = None\n wrt_value = None\n\n for i in range(max_iter):\n # Apply modifiers `pre` step\n for modifier in input_modifiers:\n seed_input = modifier.pre(seed_input)\n\n # 0 learning phase for 'test'\n computed_values = self.compute_fn([seed_input, 0])\n losses = computed_values[:len(self.loss_names)]\n named_losses = list(zip(self.loss_names, losses))\n overall_loss, grads, wrt_value = computed_values[len(self.loss_names):]\n\n # TODO: theano grads shape is inconsistent for some reason. Patch for now and investigate later.\n if grads.shape != wrt_value.shape:\n grads = np.reshape(grads, wrt_value.shape)\n\n # Apply grad modifier.\n grads = grad_modifier(grads)\n\n # Trigger callbacks\n for c in callbacks:\n c.callback(i, named_losses, overall_loss, grads, wrt_value)\n\n # Gradient descent update.\n # It only makes sense to do this if wrt_tensor is input_tensor. Otherwise shapes wont match for the update.\n if self.wrt_tensor_is_input_tensor:\n step, cache = self._rmsprop(grads, cache)\n seed_input += step\n\n # Apply modifiers `post` step\n for modifier in reversed(input_modifiers):\n seed_input = modifier.post(seed_input)\n\n if overall_loss < best_loss:\n best_loss = overall_loss.copy()\n best_input = seed_input.copy()\n\n # Trigger on_end\n for c in callbacks:\n c.on_end()\n\n return utils.deprocess_input(best_input[0], self.input_range), grads, wrt_value"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef order_by_line_nos(objs, line_nos):\n ordering = sorted(range(len(line_nos)), key=line_nos.__getitem__)\n return [objs[i] for i in ordering]", "response": "Orders the set of objects by line_nos"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_md_file(string, filename, out_path=\".\"):\n md_file = \"%s.md\" % filename\n with open(os.path.join(out_path, md_file), \"w\") as f:\n f.write(string)\n print(\"wrote {}.\".format(md_file))", "response": "Write a string with line breaks to a. md file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a src path string with line info for use as markdown link.", "response": "def get_src_path(self, obj, append_base=True):\n \"\"\"Creates a src path string with line info for use as markdown link.\n \"\"\"\n path = getsourcefile(obj)\n if self.src_root not in path:\n # this can happen with e.g.\n # inlinefunc-wrapped functions\n if hasattr(obj, \"__module__\"):\n path = \"%s.%s\" % (obj.__module__, obj.__name__)\n else:\n path = obj.__name__\n path = path.replace(\".\", \"/\")\n pre, post = path.rsplit(self.src_root + \"/\", 1)\n\n lineno = self.get_line_no(obj)\n lineno = \"\" if lineno is None else \"#L{}\".format(lineno)\n\n path = self.src_root + \"/\" + post + lineno\n if append_base:\n path = os.path.join(self.github_link, path)\n return path"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef doc2md(self, func):\n doc = getdoc(func) or \"\"\n blockindent = 0\n argindent = 1\n out = []\n\n for line in doc.split(\"\\n\"):\n indent = len(line) - len(line.lstrip())\n line = line.lstrip()\n if _RE_BLOCKSTART.match(line):\n # start of a new block\n blockindent = indent\n out.append(\"\\n*{}*\\n\".format(line))\n elif indent > blockindent:\n if _RE_ARGSTART.match(line):\n # start of new argument\n out.append(\"\\n\" + \" \" * blockindent + \" - \" + _RE_ARGSTART.sub(r\"**\\1** (\\2): \\3\", line))\n argindent = indent\n elif _RE_EXCSTART.match(line):\n # start of an exception-type block\n out.append(\"\\n\" + \" \" * blockindent + \" - \" + _RE_EXCSTART.sub(r\"**\\1**: \\2\", line))\n argindent = indent\n elif indent > argindent:\n out.append(\"\\n\" + \" \" * (blockindent + 2) + line)\n else:\n out.append(\"\\n\" + line)\n else:\n out.append(\"\\n\" + line)\n\n return \"\".join(out)", "response": "Parse the docstring of a function and return a markdown string."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntakes a function or method and returns a string that contains the contents of the function and its docstring.", "response": "def func2md(self, func, clsname=None, names=None, depth=3):\n \"\"\"Takes a function (or method) and documents it.\n\n Args:\n clsname (str, optional): class name to prepend to funcname.\n depth (int, optional): number of ### to append to function name\n\n \"\"\"\n section = \"#\" * depth\n if names is None:\n names = [func.__name__]\n\n funcname = \", \".join(names)\n escfuncname = \", \".join([\"`%s`\" % funcname if funcname.startswith(\"_\") else funcname for funcname in names])\n header = \"%s%s\" % (\"%s.\" % clsname if clsname else \"\", escfuncname)\n\n path = self.get_src_path(func)\n doc = self.doc2md(func)\n\n args, kwargs = [], []\n spec = getargspec(func)\n vargsname, kwargsname = spec.varargs, spec.keywords\n vargs = list(make_iter(spec.args)) if spec.args else []\n defaults = list(make_iter(spec.defaults)) if spec.defaults else []\n\n while vargs:\n if vargs and vargs[0] == \"self\":\n args.append(vargs.pop(0))\n elif len(vargs) > len(defaults):\n args.append(vargs.pop(0))\n else:\n default = defaults.pop(0)\n if isinstance(default, str):\n default = \"\\\"%s\\\"\" % default\n else:\n default = \"%s\" % str(default)\n\n kwargs.append((vargs.pop(0), default))\n\n if args:\n args = \", \".join(\"%s\" % arg for arg in args)\n if kwargs:\n kwargs = \", \".join(\"%s=%s\" % kwarg for kwarg in kwargs)\n if args:\n kwargs = \", \" + kwargs\n if vargsname:\n vargsname = \"*%s\" % vargsname\n if args or kwargs:\n vargsname = \", \" + vargsname\n if kwargsname:\n kwargsname = \"**%s\" % kwargsname\n if args or kwargs or vargsname:\n kwargsname = \", \" + kwargsname\n\n _FUNCDEF = \"{funcname}({args}{kwargs}{vargs}{vkwargs})\"\n funcdef = _FUNCDEF.format(funcname=funcname,\n args=args or \"\",\n kwargs=kwargs or \"\",\n vargs=vargsname or \"\",\n vkwargs=kwargsname or \"\")\n\n # split the function definition if it is too long\n lmax = 90\n if len(funcdef) > lmax:\n # wrap in the args list\n split = funcdef.split(\"(\", 1)\n # we gradually build the string again\n rest = split[1]\n args = rest.split(\", \")\n\n funcname = \"(\".join(split[:1]) + \"(\"\n lline = len(funcname)\n parts = []\n for arg in args:\n larg = len(arg)\n if larg > lmax - 5:\n # not much to do if arg is so long\n parts.append(arg)\n elif lline + larg > lmax:\n # the next arg is too long, break the line\n parts.append(\"\\\\\\n \" + arg)\n lline = 0\n else:\n parts.append(arg)\n lline += len(parts[-1])\n funcdef = funcname + \", \".join(parts)\n\n # build the signature\n string = FUNC_TEMPLATE.format(section=section,\n header=header,\n funcdef=funcdef,\n path=path,\n doc=doc if doc else \"*No documentation found.*\")\n return string"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntakes a class and creates markdown text to document its methods and variables.", "response": "def class2md(self, cls, depth=2):\n \"\"\"Takes a class and creates markdown text to document its methods and variables.\n \"\"\"\n\n section = \"#\" * depth\n subsection = \"#\" * (depth + 2)\n clsname = cls.__name__\n modname = cls.__module__\n header = clsname\n path = self.get_src_path(cls)\n doc = self.doc2md(cls)\n\n try:\n init = self.func2md(cls.__init__, clsname=clsname)\n except (ValueError, TypeError):\n # this happens if __init__ is outside the repo\n init = \"\"\n\n variables = []\n for name, obj in getmembers(cls, lambda a: not (inspect.isroutine(a) or inspect.ismethod(a))):\n if not name.startswith(\"_\") and type(obj) == property:\n comments = self.doc2md(obj) or inspect.getcomments(obj)\n comments = \"\\n %s\" % comments if comments else \"\"\n variables.append(\"\\n%s %s.%s%s\\n\" % (subsection, clsname, name, comments))\n\n handlers = []\n for name, obj in getmembers(cls, inspect.ismethoddescriptor):\n if not name.startswith(\"_\") and hasattr(obj, \"__module__\") and obj.__module__ == modname:\n handlers.append(\"\\n%s %s.%s\\n *Handler*\" % (subsection, clsname, name))\n\n methods = []\n for name, obj in getmembers(cls, inspect.ismethod):\n if not name.startswith(\"_\") and hasattr(obj,\n \"__module__\") and obj.__module__ == modname and name not in handlers:\n methods.append(self.func2md(obj, clsname=clsname, depth=depth + 1))\n\n string = CLASS_TEMPLATE.format(section=section,\n header=header,\n path=path,\n doc=doc if doc else \"\",\n init=init,\n variables=\"\".join(variables),\n handlers=\"\".join(handlers),\n methods=\"\".join(methods))\n return string"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntake an imported module object and creates a Markdown string containing functions and classes and classes.", "response": "def module2md(self, module):\n \"\"\"Takes an imported module object and create a Markdown string containing functions and classes.\n \"\"\"\n modname = module.__name__\n path = self.get_src_path(module, append_base=False)\n path = \"[{}]({})\".format(path, os.path.join(self.github_link, path))\n found = set()\n\n classes = []\n line_nos = []\n for name, obj in getmembers(module, inspect.isclass):\n # handle classes\n found.add(name)\n if not name.startswith(\"_\") and hasattr(obj, \"__module__\") and obj.__module__ == modname:\n classes.append(self.class2md(obj))\n line_nos.append(self.get_line_no(obj) or 0)\n classes = order_by_line_nos(classes, line_nos)\n\n # Since functions can have multiple aliases.\n func2names = defaultdict(list)\n for name, obj in getmembers(module, inspect.isfunction):\n func2names[obj].append(name)\n\n functions = []\n line_nos = []\n for obj in func2names:\n names = func2names[obj]\n found.update(names)\n\n # Include if within module or included modules within __init__.py and exclude from global variables\n is_module_within_init = '__init__.py' in path and obj.__module__.startswith(modname)\n if is_module_within_init:\n found.add(obj.__module__.replace(modname + '.', ''))\n\n if hasattr(obj, \"__module__\") and (obj.__module__ == modname or is_module_within_init):\n names = list(filter(lambda name: not name.startswith(\"_\"), names))\n if len(names) > 0:\n functions.append(self.func2md(obj, names=names))\n line_nos.append(self.get_line_no(obj) or 0)\n functions = order_by_line_nos(functions, line_nos)\n\n variables = []\n line_nos = []\n for name, obj in module.__dict__.items():\n if not name.startswith(\"_\") and name not in found:\n if hasattr(obj, \"__module__\") and obj.__module__ != modname:\n continue\n if hasattr(obj, \"__name__\") and not obj.__name__.startswith(modname):\n continue\n\n comments = inspect.getcomments(obj)\n comments = \": %s\" % comments if comments else \"\"\n variables.append(\"- **%s**%s\" % (name, comments))\n line_nos.append(self.get_line_no(obj) or 0)\n\n variables = order_by_line_nos(variables, line_nos)\n if variables:\n new_list = [\"**Global Variables**\", \"---------------\"]\n new_list.extend(variables)\n variables = new_list\n\n string = MODULE_TEMPLATE.format(path=path,\n global_vars=\"\\n\".join(variables) if variables else \"\",\n functions=\"\\n\".join(functions) if functions else \"\",\n classes=\"\".join(classes) if classes else \"\")\n return string"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _find_penultimate_layer(model, layer_idx, penultimate_layer_idx):\n if penultimate_layer_idx is None:\n for idx, layer in utils.reverse_enumerate(model.layers[:layer_idx - 1]):\n if isinstance(layer, Wrapper):\n layer = layer.layer\n if isinstance(layer, (_Conv, _Pooling1D, _Pooling2D, _Pooling3D)):\n penultimate_layer_idx = idx\n break\n\n if penultimate_layer_idx is None:\n raise ValueError('Unable to determine penultimate `Conv` or `Pooling` '\n 'layer for layer_idx: {}'.format(layer_idx))\n\n # Handle negative indexing otherwise the next check can fail.\n if layer_idx < 0:\n layer_idx = len(model.layers) + layer_idx\n if penultimate_layer_idx > layer_idx:\n raise ValueError('`penultimate_layer_idx` needs to be before `layer_idx`')\n\n return model.layers[penultimate_layer_idx]", "response": "Searches for the nearest penultimate CNN or Pooling layer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating an attention heatmap over the seed_input with respect to weighted losses.", "response": "def visualize_saliency_with_losses(input_tensor, losses, seed_input, wrt_tensor=None, grad_modifier='absolute', keepdims=False):\n \"\"\"Generates an attention heatmap over the `seed_input` by using positive gradients of `input_tensor`\n with respect to weighted `losses`.\n\n This function is intended for advanced use cases where a custom loss is desired. For common use cases,\n refer to `visualize_class_saliency` or `visualize_regression_saliency`.\n\n For a full description of saliency, see the paper:\n [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps]\n (https://arxiv.org/pdf/1312.6034v2.pdf)\n\n Args:\n input_tensor: An input tensor of shape: `(samples, channels, image_dims...)` if `image_data_format=\n channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.\n losses: List of ([Loss](vis.losses#Loss), weight) tuples.\n seed_input: The model input for which activation map needs to be visualized.\n wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.\n When None, this is assumed to be the same as `input_tensor` (Default value: None)\n grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). By default `absolute`\n value of gradients are used. To visualize positive or negative gradients, use `relu` and `negate`\n respectively. (Default value = 'absolute')\n keepdims: A boolean, whether to keep the dimensions or not.\n If keepdims is False, the channels axis is deleted.\n If keepdims is True, the grad with same shape as input_tensor is returned. (Default value: False)\n\n Returns:\n The normalized gradients of `seed_input` with respect to weighted `losses`.\n \"\"\"\n opt = Optimizer(input_tensor, losses, wrt_tensor=wrt_tensor, norm_grads=False)\n grads = opt.minimize(seed_input=seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)[1]\n\n if not keepdims:\n channel_idx = 1 if K.image_data_format() == 'channels_first' else -1\n grads = np.max(grads, axis=channel_idx)\n return utils.normalize(grads)[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a visualised Saliency layer over the seed_input for maximizing filter_indices.", "response": "def visualize_saliency(model, layer_idx, filter_indices, seed_input, wrt_tensor=None,\n backprop_modifier=None, grad_modifier='absolute', keepdims=False):\n \"\"\"Generates an attention heatmap over the `seed_input` for maximizing `filter_indices`\n output in the given `layer_idx`.\n\n Args:\n model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`\n if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if\n `image_data_format=channels_last`.\n layer_idx: The layer index within `model.layers` whose filters needs to be visualized.\n filter_indices: filter indices within the layer to be maximized.\n If None, all filters are visualized. (Default value = None)\n For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.\n If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for\n 'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.\n seed_input: The model input for which activation map needs to be visualized.\n wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.\n When None, this is assumed to be the same as `input_tensor` (Default value: None)\n backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't\n specify anything, no backprop modification is applied. (Default value = None)\n grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). By default `absolute`\n value of gradients are used. To visualize positive or negative gradients, use `relu` and `negate`\n respectively. (Default value = 'absolute')\n keepdims: A boolean, whether to keep the dimensions or not.\n If keepdims is False, the channels axis is deleted.\n If keepdims is True, the grad with same shape as input_tensor is returned. (Default value: False)\n\n Example:\n If you wanted to visualize attention over 'bird' category, say output index 22 on the\n final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer = dense_layer`.\n\n One could also set filter indices to more than one value. For example, `filter_indices = [22, 23]` should\n (hopefully) show attention map that corresponds to both 22, 23 output categories.\n\n Returns:\n The heatmap image indicating the `seed_input` regions whose change would most contribute towards\n maximizing the output of `filter_indices`.\n \"\"\"\n if backprop_modifier is not None:\n modifier_fn = get(backprop_modifier)\n model = modifier_fn(model)\n\n # `ActivationMaximization` loss reduces as outputs get large, hence negative gradients indicate the direction\n # for increasing activations. Multiply with -1 so that positive gradients indicate increase instead.\n losses = [\n (ActivationMaximization(model.layers[layer_idx], filter_indices), -1)\n ]\n return visualize_saliency_with_losses(model.input, losses, seed_input, wrt_tensor, grad_modifier, keepdims)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef visualize_cam_with_losses(input_tensor, losses, seed_input, penultimate_layer, grad_modifier=None):\n penultimate_output = penultimate_layer.output\n opt = Optimizer(input_tensor, losses, wrt_tensor=penultimate_output, norm_grads=False)\n _, grads, penultimate_output_value = opt.minimize(seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)\n\n # For numerical stability. Very small grad values along with small penultimate_output_value can cause\n # w * penultimate_output_value to zero out, even for reasonable fp precision of float32.\n grads = grads / (np.max(grads) + K.epsilon())\n\n # Average pooling across all feature maps.\n # This captures the importance of feature map (channel) idx to the output.\n channel_idx = 1 if K.image_data_format() == 'channels_first' else -1\n other_axis = np.delete(np.arange(len(grads.shape)), channel_idx)\n weights = np.mean(grads, axis=tuple(other_axis))\n\n # Generate heatmap by computing weight * output over feature maps\n output_dims = utils.get_img_shape(penultimate_output)[2:]\n heatmap = np.zeros(shape=output_dims, dtype=K.floatx())\n for i, w in enumerate(weights):\n if channel_idx == -1:\n heatmap += w * penultimate_output_value[0, ..., i]\n else:\n heatmap += w * penultimate_output_value[0, i, ...]\n\n # ReLU thresholding to exclude pattern mismatch information (negative gradients).\n heatmap = np.maximum(heatmap, 0)\n\n # The penultimate feature map size is definitely smaller than input image.\n input_dims = utils.get_img_shape(input_tensor)[2:]\n\n # Figure out the zoom factor.\n zoom_factor = [i / (j * 1.0) for i, j in iter(zip(input_dims, output_dims))]\n heatmap = zoom(heatmap, zoom_factor)\n return utils.normalize(heatmap)", "response": "Generates a gradient based class activation map by using positive gradients of input_tensor with respect to weighted losses."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a gradient based class activation map that maximizes the outputs of filter_indices in layer_idx.", "response": "def visualize_cam(model, layer_idx, filter_indices,\n seed_input, penultimate_layer_idx=None,\n backprop_modifier=None, grad_modifier=None):\n \"\"\"Generates a gradient based class activation map (grad-CAM) that maximizes the outputs of\n `filter_indices` in `layer_idx`.\n\n Args:\n model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`\n if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if\n `image_data_format=channels_last`.\n layer_idx: The layer index within `model.layers` whose filters needs to be visualized.\n filter_indices: filter indices within the layer to be maximized.\n If None, all filters are visualized. (Default value = None)\n For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.\n If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for\n 'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.\n seed_input: The input image for which activation map needs to be visualized.\n penultimate_layer_idx: The pre-layer to `layer_idx` whose feature maps should be used to compute gradients\n wrt filter output. If not provided, it is set to the nearest penultimate `Conv` or `Pooling` layer.\n backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't\n specify anything, no backprop modification is applied. (Default value = None)\n grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't\n specify anything, gradients are unchanged (Default value = None)\n\n Example:\n If you wanted to visualize attention over 'bird' category, say output index 22 on the\n final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer = dense_layer`.\n\n One could also set filter indices to more than one value. For example, `filter_indices = [22, 23]` should\n (hopefully) show attention map that corresponds to both 22, 23 output categories.\n\n Returns:\n The heatmap image indicating the input regions whose change would most contribute towards\n maximizing the output of `filter_indices`.\n \"\"\"\n if backprop_modifier is not None:\n modifier_fn = get(backprop_modifier)\n model = modifier_fn(model)\n\n penultimate_layer = _find_penultimate_layer(model, layer_idx, penultimate_layer_idx)\n\n # `ActivationMaximization` outputs negative gradient values for increase in activations. Multiply with -1\n # so that positive gradients indicate increase instead.\n losses = [\n (ActivationMaximization(model.layers[layer_idx], filter_indices), -1)\n ]\n return visualize_cam_with_losses(model.input, losses, seed_input, penultimate_layer, grad_modifier)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef modify_model_backprop(model, backprop_modifier):\n # The general strategy is as follows:\n # - Save original model so that upstream callers don't see unexpected results with their models.\n # - Call backend specific function that registers the custom op and loads the model under modified context manager.\n # - Maintain cache to save this expensive process on subsequent calls.\n # - Load model with custom context modifying backprop behavior.\n #\n # The reason for this round about way is because the graph needs to be rebuild when any of its layer builder\n # functions are changed. This is very complicated to do in Keras and makes the implementation very tightly bound\n # with keras internals. By saving and loading models, we dont have to worry about future compatibility.\n #\n # The only exception to this is the way advanced activations are handled which makes use of some keras internal\n # knowledge and might break in the future.\n # ADD on 22 Jul 2018:\n # In fact, it has broken. Currently, advanced activations are not supported.\n\n # 0. Retrieve from cache if previously computed.\n modified_model = _MODIFIED_MODEL_CACHE.get((model, backprop_modifier))\n if modified_model is not None:\n return modified_model\n\n model_path = os.path.join(tempfile.gettempdir(), next(tempfile._get_candidate_names()) + '.h5')\n try:\n # 1. Save original model\n model.save(model_path)\n\n # 2. Register modifier and load modified model under custom context.\n modifier_fn = _BACKPROP_MODIFIERS.get(backprop_modifier)\n if modifier_fn is None:\n raise ValueError(\"'{}' modifier is not supported\".format(backprop_modifier))\n modifier_fn(backprop_modifier)\n\n # 3. Create graph under custom context manager.\n with tf.get_default_graph().gradient_override_map({'Relu': backprop_modifier}):\n # This should rebuild graph with modifications.\n modified_model = load_model(model_path)\n\n # Cache to improve subsequent call performance.\n _MODIFIED_MODEL_CACHE[(model, backprop_modifier)] = modified_model\n return modified_model\n finally:\n os.remove(model_path)", "response": "Creates a copy of model with modified activations."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef normalize(input_tensor, output_tensor):\n image_dims = utils.get_img_shape(input_tensor)[1:]\n return output_tensor / np.prod(image_dims)", "response": "Normalizes the output_tensor with respect to input_tensor dimensions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef build_loss(self):\n image_dims = K.ndim(self.img) - 2\n\n # Constructing slice [1:] + [:-1] * (image_dims - 1) and [:-1] * (image_dims)\n start_slice = [slice(1, None, None)] + [slice(None, -1, None) for _ in range(image_dims - 1)]\n end_slice = [slice(None, -1, None) for _ in range(image_dims)]\n samples_channels_slice = [slice(None, None, None), slice(None, None, None)]\n\n # Compute pixel diffs by rolling slices to the right per image dim.\n tv = None\n for i in range(image_dims):\n ss = tuple(samples_channels_slice + start_slice)\n es = tuple(samples_channels_slice + end_slice)\n diff_square = K.square(self.img[utils.slicer[ss]] - self.img[utils.slicer[es]])\n tv = diff_square if tv is None else tv + diff_square\n\n # Roll over to next image dim\n start_slice = np.roll(start_slice, 1).tolist()\n end_slice = np.roll(end_slice, 1).tolist()\n\n tv = K.sum(K.pow(tv, self.beta / 2.))\n return normalize(self.img, tv)", "response": "Implements the N - dim version of function\n $$TV^{ beta } ( x ) = \\ sum_{whc } \\ right"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _find_font_file(query):\n return list(filter(lambda path: query.lower() in os.path.basename(path).lower(), fontman.findSystemFonts()))", "response": "Utility to find font file.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_defaults_to_kwargs(defaults, **kwargs):\n defaults = dict(defaults)\n defaults.update(kwargs)\n return defaults", "response": "Updates kwargs with dict of defaults"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\napplying modifications to the model layers to create a new Graph.", "response": "def apply_modifications(model, custom_objects=None):\n \"\"\"Applies modifications to the model layers to create a new Graph. For example, simply changing\n `model.layers[idx].activation = new activation` does not change the graph. The entire graph needs to be updated\n with modified inbound and outbound tensors because of change in layer building function.\n\n Args:\n model: The `keras.models.Model` instance.\n\n Returns:\n The modified model with changes applied. Does not mutate the original `model`.\n \"\"\"\n # The strategy is to save the modified model and load it back. This is done because setting the activation\n # in a Keras layer doesnt actually change the graph. We have to iterate the entire graph and change the\n # layer inbound and outbound nodes with modified tensors. This is doubly complicated in Keras 2.x since\n # multiple inbound and outbound nodes are allowed with the Graph API.\n model_path = os.path.join(tempfile.gettempdir(), next(tempfile._get_candidate_names()) + '.h5')\n try:\n model.save(model_path)\n return load_model(model_path, custom_objects=custom_objects)\n finally:\n os.remove(model_path)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef random_array(shape, mean=128., std=20.):\n x = np.random.random(shape)\n # normalize around mean=0, std=1\n x = (x - np.mean(x)) / (np.std(x) + K.epsilon())\n # and then around the desired mean/std\n x = (x * std) + mean\n return x", "response": "Creates a uniformly distributed random array of the given shape with the given mean and std."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlook up the layer index corresponding to layer_name from model.", "response": "def find_layer_idx(model, layer_name):\n \"\"\"Looks up the layer index corresponding to `layer_name` from `model`.\n\n Args:\n model: The `keras.models.Model` instance.\n layer_name: The name of the layer to lookup.\n\n Returns:\n The layer index if found. Raises an exception otherwise.\n \"\"\"\n layer_idx = None\n for idx, layer in enumerate(model.layers):\n if layer.name == layer_name:\n layer_idx = idx\n break\n\n if layer_idx is None:\n raise ValueError(\"No layer with name '{}' within the model\".format(layer_name))\n return layer_idx"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_img_shape(img):\n if isinstance(img, np.ndarray):\n shape = img.shape\n else:\n shape = K.int_shape(img)\n\n if K.image_data_format() == 'channels_last':\n shape = list(shape)\n shape.insert(1, shape[-1])\n shape = tuple(shape[:-1])\n return shape", "response": "Returns image shape in a backend agnostic manner."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_img(path, grayscale=False, target_size=None):\n img = io.imread(path, grayscale)\n if target_size:\n img = transform.resize(img, target_size, preserve_range=True).astype('uint8')\n return img", "response": "Utility function to load an image from disk."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef lookup_imagenet_labels(indices):\n global _CLASS_INDEX\n if _CLASS_INDEX is None:\n with open(os.path.join(os.path.dirname(__file__), '../../resources/imagenet_class_index.json')) as f:\n _CLASS_INDEX = json.load(f)\n\n indices = listify(indices)\n return [_CLASS_INDEX[str(idx)][1] for idx in indices]", "response": "Utility function to return the image net label corresponding to the image category."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndraws text over the image. Requires PIL.", "response": "def draw_text(img, text, position=(10, 10), font='FreeSans.ttf', font_size=14, color=(0, 0, 0)):\n \"\"\"Draws text over the image. Requires PIL.\n\n Args:\n img: The image to use.\n text: The text string to overlay.\n position: The text (x, y) position. (Default value = (10, 10))\n font: The ttf or open type font to use. (Default value = 'FreeSans.ttf')\n font_size: The text font size. (Default value = 12)\n color: The (r, g, b) values for text color. (Default value = (0, 0, 0))\n\n Returns: Image overlayed with text.\n \"\"\"\n _check_pil()\n\n font_files = _find_font_file(font)\n if len(font_files) == 0:\n logger.warn(\"Failed to lookup font '{}', falling back to default\".format(font))\n font = ImageFont.load_default()\n else:\n font = ImageFont.truetype(font_files[0], font_size)\n\n # Don't mutate original image\n img = Image.fromarray(img)\n draw = ImageDraw.Draw(img)\n draw.text(position, text, fill=color, font=font)\n return np.asarray(img)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef normalize(array, min_value=0., max_value=1.):\n arr_min = np.min(array)\n arr_max = np.max(array)\n normalized = (array - arr_min) / (arr_max - arr_min + K.epsilon())\n return (max_value - min_value) * normalized + min_value", "response": "Normalizes the numpy array to range between min_value and max_value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_num_filters(layer):\n # Handle layers with no channels.\n if K.ndim(layer.output) == 2:\n return K.int_shape(layer.output)[-1]\n\n channel_idx = 1 if K.image_data_format() == 'channels_first' else -1\n return K.int_shape(layer.output)[channel_idx]", "response": "Determines the number of filters within the given keras layer."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef overlay(array1, array2, alpha=0.5):\n if alpha < 0. or alpha > 1.:\n raise ValueError(\"`alpha` needs to be between [0, 1]\")\n if array1.shape != array2.shape:\n raise ValueError('`array1` and `array2` must have the same shapes')\n\n return (array1 * alpha + array2 * (1. - alpha)).astype(array1.dtype)", "response": "Overlays array1 onto array2 using alpha blending."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninfers the coordinates and dimensions of a new DataArray.", "response": "def _infer_coords_and_dims(shape, coords, dims):\n \"\"\"All the logic for creating a new DataArray\"\"\"\n\n if (coords is not None and not utils.is_dict_like(coords) and\n len(coords) != len(shape)):\n raise ValueError('coords is not dict-like, but it has %s items, '\n 'which does not match the %s dimensions of the '\n 'data' % (len(coords), len(shape)))\n\n if isinstance(dims, str):\n dims = (dims,)\n\n if dims is None:\n dims = ['dim_%s' % n for n in range(len(shape))]\n if coords is not None and len(coords) == len(shape):\n # try to infer dimensions from coords\n if utils.is_dict_like(coords):\n # deprecated in GH993, removed in GH1539\n raise ValueError('inferring DataArray dimensions from '\n 'dictionary like ``coords`` is no longer '\n 'supported. Use an explicit list of '\n '``dims`` instead.')\n for n, (dim, coord) in enumerate(zip(dims, coords)):\n coord = as_variable(coord,\n name=dims[n]).to_index_variable()\n dims[n] = coord.name\n dims = tuple(dims)\n else:\n for d in dims:\n if not isinstance(d, str):\n raise TypeError('dimension %s is not a string' % d)\n\n new_coords = OrderedDict()\n\n if utils.is_dict_like(coords):\n for k, v in coords.items():\n new_coords[k] = as_variable(v, name=k)\n elif coords is not None:\n for dim, coord in zip(dims, coords):\n var = as_variable(coord, name=dim)\n var.dims = (dim,)\n new_coords[dim] = var\n\n sizes = dict(zip(dims, shape))\n for k, v in new_coords.items():\n if any(d not in dims for d in v.dims):\n raise ValueError('coordinate %s has dimensions %s, but these '\n 'are not a subset of the DataArray '\n 'dimensions %s' % (k, v.dims, dims))\n\n for d, s in zip(v.dims, v.shape):\n if s != sizes[d]:\n raise ValueError('conflicting sizes for dimension %r: '\n 'length %s on the data but length %s on '\n 'coordinate %r' % (d, sizes[d], s, k))\n\n if k in sizes and v.shape != (sizes[k],):\n raise ValueError('coordinate %r is a DataArray dimension, but '\n 'it has shape %r rather than expected shape %r '\n 'matching the dimension size'\n % (k, v.shape, (sizes[k],)))\n\n assert_unique_multiindex_level_names(new_coords)\n\n return new_coords, dims"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_dataset(self, dim=None, name=None):\n if dim is not None and dim not in self.dims:\n warnings.warn('the order of the arguments on DataArray.to_dataset '\n 'has changed; you now need to supply ``name`` as '\n 'a keyword argument',\n FutureWarning, stacklevel=2)\n name = dim\n dim = None\n\n if dim is not None:\n if name is not None:\n raise TypeError('cannot supply both dim and name arguments')\n return self._to_dataset_split(dim)\n else:\n return self._to_dataset_whole(name)", "response": "Convert a DataArray into a Dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a mapping of all MultiIndex levels and their corresponding coordinate name.", "response": "def _level_coords(self):\n \"\"\"Return a mapping of all MultiIndex levels and their corresponding\n coordinate name.\n \"\"\"\n level_coords = OrderedDict()\n for cname, var in self._coords.items():\n if var.ndim == 1 and isinstance(var, IndexVariable):\n level_names = var.level_names\n if level_names is not None:\n dim, = var.dims\n level_coords.update({lname: dim for lname in level_names})\n return level_coords"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef indexes(self):\n if self._indexes is None:\n self._indexes = default_indexes(self._coords, self.dims)\n return Indexes(self._indexes)", "response": "Mapping of pandas. Index objects used for label based indexing."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nresetting the coordinates of the object to become variables.", "response": "def reset_coords(self, names=None, drop=False, inplace=None):\n \"\"\"Given names of coordinates, reset them to become variables.\n\n Parameters\n ----------\n names : str or list of str, optional\n Name(s) of non-index coordinates in this dataset to reset into\n variables. By default, all non-index coordinates are reset.\n drop : bool, optional\n If True, remove coordinates instead of converting them into\n variables.\n inplace : bool, optional\n If True, modify this dataset inplace. Otherwise, create a new\n object.\n\n Returns\n -------\n Dataset, or DataArray if ``drop == True``\n \"\"\"\n inplace = _check_inplace(inplace)\n if inplace and not drop:\n raise ValueError('cannot reset coordinates in-place on a '\n 'DataArray without ``drop == True``')\n if names is None:\n names = set(self.coords) - set(self.dims)\n dataset = self.coords.to_dataset().reset_coords(names, drop)\n if drop:\n if inplace:\n self._coords = dataset._variables\n else:\n return self._replace(coords=dataset._variables)\n else:\n if self.name is None:\n raise ValueError('cannot reset_coords with drop=False '\n 'on an unnamed DataArrray')\n dataset[self.name] = self.variable\n return dataset"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load(self, **kwargs):\n ds = self._to_temp_dataset().load(**kwargs)\n new = self._from_temp_dataset(ds)\n self._variable = new._variable\n self._coords = new._coords\n return self", "response": "Manually trigger loading of this array s data from disk or a\n remote source into memory and return this array."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ntriggering computation in constituent dask arrays This keeps them as dask arrays but encourages them as dask arrays but encourages them as dask arrays.", "response": "def persist(self, **kwargs):\n \"\"\" Trigger computation in constituent dask arrays\n\n This keeps them as dask arrays but encourages them to keep data in\n memory. This is particularly useful when on a distributed machine.\n When on a single machine consider using ``.compute()`` instead.\n\n Parameters\n ----------\n **kwargs : dict\n Additional keyword arguments passed on to ``dask.persist``.\n\n See Also\n --------\n dask.persist\n \"\"\"\n ds = self._to_temp_dataset().persist(**kwargs)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef copy(self, deep=True, data=None):\n variable = self.variable.copy(deep=deep, data=data)\n coords = OrderedDict((k, v.copy(deep=deep))\n for k, v in self._coords.items())\n return self._replace(variable, coords)", "response": "Returns a shallow copy of the current array."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncoerce this array s data into a dask array with the given chunks.", "response": "def chunk(self, chunks=None, name_prefix='xarray-', token=None,\n lock=False):\n \"\"\"Coerce this array's data into a dask arrays with the given chunks.\n\n If this variable is a non-dask array, it will be converted to dask\n array. If it's a dask array, it will be rechunked to the given chunk\n sizes.\n\n If neither chunks is not provided for one or more dimensions, chunk\n sizes along that dimension will not be updated; non-dask arrays will be\n converted into dask arrays with a single block.\n\n Parameters\n ----------\n chunks : int, tuple or dict, optional\n Chunk sizes along each dimension, e.g., ``5``, ``(5, 5)`` or\n ``{'x': 5, 'y': 5}``.\n name_prefix : str, optional\n Prefix for the name of the new dask array.\n token : str, optional\n Token uniquely identifying this array.\n lock : optional\n Passed on to :py:func:`dask.array.from_array`, if the array is not\n already as dask array.\n\n Returns\n -------\n chunked : xarray.DataArray\n \"\"\"\n if isinstance(chunks, (list, tuple)):\n chunks = dict(zip(self.dims, chunks))\n\n ds = self._to_temp_dataset().chunk(chunks, name_prefix=name_prefix,\n token=token, lock=lock)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef isel(self, indexers=None, drop=False, **indexers_kwargs):\n indexers = either_dict_or_kwargs(indexers, indexers_kwargs, 'isel')\n ds = self._to_temp_dataset().isel(drop=drop, indexers=indexers)\n return self._from_temp_dataset(ds)", "response": "Return a new DataArray whose dataset is given by integer indexing\n along the specified dimension."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sel(self, indexers=None, method=None, tolerance=None, drop=False,\n **indexers_kwargs):\n \"\"\"Return a new DataArray whose dataset is given by selecting\n index labels along the specified dimension(s).\n\n .. warning::\n\n Do not try to assign values when using any of the indexing methods\n ``isel`` or ``sel``::\n\n da = xr.DataArray([0, 1, 2, 3], dims=['x'])\n # DO NOT do this\n da.isel(x=[0, 1, 2])[1] = -1\n\n Assigning values with the chained indexing using ``.sel`` or\n ``.isel`` fails silently.\n\n See Also\n --------\n Dataset.sel\n DataArray.isel\n\n \"\"\"\n ds = self._to_temp_dataset().sel(\n indexers=indexers, drop=drop, method=method, tolerance=tolerance,\n **indexers_kwargs)\n return self._from_temp_dataset(ds)", "response": "Return a new DataArray whose dataset is given by selecting\n index labels along the specified dimension."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new DataArray whose dataset is given by pointwise integer indexing along the specified dimension.", "response": "def isel_points(self, dim='points', **indexers):\n \"\"\"Return a new DataArray whose dataset is given by pointwise integer\n indexing along the specified dimension(s).\n\n See Also\n --------\n Dataset.isel_points\n \"\"\"\n ds = self._to_temp_dataset().isel_points(dim=dim, **indexers)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sel_points(self, dim='points', method=None, tolerance=None,\n **indexers):\n \"\"\"Return a new DataArray whose dataset is given by pointwise selection\n of index labels along the specified dimension(s).\n\n See Also\n --------\n Dataset.sel_points\n \"\"\"\n ds = self._to_temp_dataset().sel_points(\n dim=dim, method=method, tolerance=tolerance, **indexers)\n return self._from_temp_dataset(ds)", "response": "Return a new DataArray whose dataset is given by pointwise selection\n of index labels along the specified dimension."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconforming this object onto a new set of indexes.", "response": "def reindex(self, indexers=None, method=None, tolerance=None, copy=True,\n **indexers_kwargs):\n \"\"\"Conform this object onto a new set of indexes, filling in\n missing values with NaN.\n\n Parameters\n ----------\n indexers : dict, optional\n Dictionary with keys given by dimension names and values given by\n arrays of coordinates tick labels. Any mis-matched coordinate\n values will be filled in with NaN, and any mis-matched dimension\n names will simply be ignored.\n One of indexers or indexers_kwargs must be provided.\n copy : bool, optional\n If ``copy=True``, data in the return value is always copied. If\n ``copy=False`` and reindexing is unnecessary, or can be performed\n with only slice operations, then the output may share memory with\n the input. In either case, a new xarray object is always returned.\n method : {None, 'nearest', 'pad'/'ffill', 'backfill'/'bfill'}, optional\n Method to use for filling index values in ``indexers`` not found on\n this data array:\n\n * None (default): don't fill gaps\n * pad / ffill: propagate last valid index value forward\n * backfill / bfill: propagate next valid index value backward\n * nearest: use nearest valid index value (requires pandas>=0.16)\n tolerance : optional\n Maximum distance between original and new labels for inexact\n matches. The values of the index at the matching locations must\n satisfy the equation ``abs(index[indexer] - target) <= tolerance``.\n **indexers_kwarg : {dim: indexer, ...}, optional\n The keyword arguments form of ``indexers``.\n One of indexers or indexers_kwargs must be provided.\n\n Returns\n -------\n reindexed : DataArray\n Another dataset array, with this array's data but replaced\n coordinates.\n\n See Also\n --------\n DataArray.reindex_like\n align\n \"\"\"\n indexers = either_dict_or_kwargs(\n indexers, indexers_kwargs, 'reindex')\n ds = self._to_temp_dataset().reindex(\n indexers=indexers, method=method, tolerance=tolerance, copy=copy)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninterpolate the multidimensional array with the given coordinates.", "response": "def interp(self, coords=None, method='linear', assume_sorted=False,\n kwargs={}, **coords_kwargs):\n \"\"\" Multidimensional interpolation of variables.\n\n coords : dict, optional\n Mapping from dimension names to the new coordinates.\n new coordinate can be an scalar, array-like or DataArray.\n If DataArrays are passed as new coordates, their dimensions are\n used for the broadcasting.\n method: {'linear', 'nearest'} for multidimensional array,\n {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic'}\n for 1-dimensional array.\n assume_sorted: boolean, optional\n If False, values of x can be in any order and they are sorted\n first. If True, x has to be an array of monotonically increasing\n values.\n kwargs: dictionary\n Additional keyword passed to scipy's interpolator.\n **coords_kwarg : {dim: coordinate, ...}, optional\n The keyword arguments form of ``coords``.\n One of coords or coords_kwargs must be provided.\n\n Returns\n -------\n interpolated: xr.DataArray\n New dataarray on the new coordinates.\n\n Notes\n -----\n scipy is required.\n\n See Also\n --------\n scipy.interpolate.interp1d\n scipy.interpolate.interpn\n\n Examples\n --------\n >>> da = xr.DataArray([1, 3], [('x', np.arange(2))])\n >>> da.interp(x=0.5)\n \n array(2.0)\n Coordinates:\n x float64 0.5\n \"\"\"\n if self.dtype.kind not in 'uifc':\n raise TypeError('interp only works for a numeric type array. '\n 'Given {}.'.format(self.dtype))\n\n ds = self._to_temp_dataset().interp(\n coords, method=method, kwargs=kwargs, assume_sorted=assume_sorted,\n **coords_kwargs)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef interp_like(self, other, method='linear', assume_sorted=False,\n kwargs={}):\n \"\"\"Interpolate this object onto the coordinates of another object,\n filling out of range values with NaN.\n\n Parameters\n ----------\n other : Dataset or DataArray\n Object with an 'indexes' attribute giving a mapping from dimension\n names to an 1d array-like, which provides coordinates upon\n which to index the variables in this dataset.\n method: string, optional.\n {'linear', 'nearest'} for multidimensional array,\n {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic'}\n for 1-dimensional array. 'linear' is used by default.\n assume_sorted: boolean, optional\n If False, values of coordinates that are interpolated over can be\n in any order and they are sorted first. If True, interpolated\n coordinates are assumed to be an array of monotonically increasing\n values.\n kwargs: dictionary, optional\n Additional keyword passed to scipy's interpolator.\n\n Returns\n -------\n interpolated: xr.DataArray\n Another dataarray by interpolating this dataarray's data along the\n coordinates of the other object.\n\n Notes\n -----\n scipy is required.\n If the dataarray has object-type coordinates, reindex is used for these\n coordinates instead of the interpolation.\n\n See Also\n --------\n DataArray.interp\n DataArray.reindex_like\n \"\"\"\n if self.dtype.kind not in 'uifc':\n raise TypeError('interp only works for a numeric type array. '\n 'Given {}.'.format(self.dtype))\n\n ds = self._to_temp_dataset().interp_like(\n other, method=method, kwargs=kwargs, assume_sorted=assume_sorted)\n return self._from_temp_dataset(ds)", "response": "Interpolate this object onto the coordinates of another object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new DataArray with renamed coordinates or a new name.", "response": "def rename(self, new_name_or_name_dict=None, **names):\n \"\"\"Returns a new DataArray with renamed coordinates or a new name.\n\n Parameters\n ----------\n new_name_or_name_dict : str or dict-like, optional\n If the argument is dict-like, it it used as a mapping from old\n names to new names for coordinates. Otherwise, use the argument\n as the new name for this array.\n **names, optional\n The keyword arguments form of a mapping from old names to\n new names for coordinates.\n One of new_name_or_name_dict or names must be provided.\n\n\n Returns\n -------\n renamed : DataArray\n Renamed array or array with renamed coordinates.\n\n See Also\n --------\n Dataset.rename\n DataArray.swap_dims\n \"\"\"\n if names or utils.is_dict_like(new_name_or_name_dict):\n name_dict = either_dict_or_kwargs(\n new_name_or_name_dict, names, 'rename')\n dataset = self._to_temp_dataset().rename(name_dict)\n return self._from_temp_dataset(dataset)\n else:\n return self._replace(name=new_name_or_name_dict)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new Dataset containing swapped dimensions.", "response": "def swap_dims(self, dims_dict):\n \"\"\"Returns a new DataArray with swapped dimensions.\n\n Parameters\n ----------\n dims_dict : dict-like\n Dictionary whose keys are current dimension names and whose values\n are new names. Each value must already be a coordinate on this\n array.\n\n Returns\n -------\n renamed : Dataset\n DataArray with swapped dimensions.\n\n See Also\n --------\n\n DataArray.rename\n Dataset.swap_dims\n \"\"\"\n ds = self._to_temp_dataset().swap_dims(dims_dict)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new object with an additional dimension inserted at the corresponding position in the array.", "response": "def expand_dims(self, dim=None, axis=None, **dim_kwargs):\n \"\"\"Return a new object with an additional axis (or axes) inserted at\n the corresponding position in the array shape.\n\n If dim is already a scalar coordinate, it will be promoted to a 1D\n coordinate consisting of a single value.\n\n Parameters\n ----------\n dim : str, sequence of str, dict, or None\n Dimensions to include on the new variable.\n If provided as str or sequence of str, then dimensions are inserted\n with length 1. If provided as a dict, then the keys are the new\n dimensions and the values are either integers (giving the length of\n the new dimensions) or sequence/ndarray (giving the coordinates of\n the new dimensions). **WARNING** for python 3.5, if ``dim`` is\n dict-like, then it must be an ``OrderedDict``. This is to ensure\n that the order in which the dims are given is maintained.\n axis : integer, list (or tuple) of integers, or None\n Axis position(s) where new axis is to be inserted (position(s) on\n the result array). If a list (or tuple) of integers is passed,\n multiple axes are inserted. In this case, dim arguments should be\n same length list. If axis=None is passed, all the axes will be\n inserted to the start of the result array.\n **dim_kwargs : int or sequence/ndarray\n The keywords are arbitrary dimensions being inserted and the values\n are either the lengths of the new dims (if int is given), or their\n coordinates. Note, this is an alternative to passing a dict to the\n dim kwarg and will only be used if dim is None. **WARNING** for\n python 3.5 ``dim_kwargs`` is not available.\n\n Returns\n -------\n expanded : same type as caller\n This object, but with an additional dimension(s).\n \"\"\"\n if isinstance(dim, int):\n raise TypeError('dim should be str or sequence of strs or dict')\n elif isinstance(dim, str):\n dim = OrderedDict(((dim, 1),))\n elif isinstance(dim, (list, tuple)):\n if len(dim) != len(set(dim)):\n raise ValueError('dims should not contain duplicate values.')\n dim = OrderedDict(((d, 1) for d in dim))\n\n # TODO: get rid of the below code block when python 3.5 is no longer\n # supported.\n python36_plus = sys.version_info[0] == 3 and sys.version_info[1] > 5\n not_ordereddict = dim is not None and not isinstance(dim, OrderedDict)\n if not python36_plus and not_ordereddict:\n raise TypeError(\"dim must be an OrderedDict for python <3.6\")\n elif not python36_plus and dim_kwargs:\n raise ValueError(\"dim_kwargs isn't available for python <3.6\")\n dim_kwargs = OrderedDict(dim_kwargs)\n\n dim = either_dict_or_kwargs(dim, dim_kwargs, 'expand_dims')\n ds = self._to_temp_dataset().expand_dims(dim, axis)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the index of the data array using one or more existing coordinates.", "response": "def set_index(self, indexes=None, append=False, inplace=None,\n **indexes_kwargs):\n \"\"\"Set DataArray (multi-)indexes using one or more existing\n coordinates.\n\n Parameters\n ----------\n indexes : {dim: index, ...}\n Mapping from names matching dimensions and values given\n by (lists of) the names of existing coordinates or variables to set\n as new (multi-)index.\n append : bool, optional\n If True, append the supplied index(es) to the existing index(es).\n Otherwise replace the existing index(es) (default).\n inplace : bool, optional\n If True, set new index(es) in-place. Otherwise, return a new\n DataArray object.\n **indexes_kwargs: optional\n The keyword arguments form of ``indexes``.\n One of indexes or indexes_kwargs must be provided.\n\n Returns\n -------\n obj : DataArray\n Another dataarray, with this data but replaced coordinates.\n\n See Also\n --------\n DataArray.reset_index\n \"\"\"\n inplace = _check_inplace(inplace)\n indexes = either_dict_or_kwargs(indexes, indexes_kwargs, 'set_index')\n coords, _ = merge_indexes(indexes, self._coords, set(), append=append)\n if inplace:\n self._coords = coords\n else:\n return self._replace(coords=coords)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nresets the index of the dataarray to the specified index or multi - index level.", "response": "def reset_index(self, dims_or_levels, drop=False, inplace=None):\n \"\"\"Reset the specified index(es) or multi-index level(s).\n\n Parameters\n ----------\n dims_or_levels : str or list\n Name(s) of the dimension(s) and/or multi-index level(s) that will\n be reset.\n drop : bool, optional\n If True, remove the specified indexes and/or multi-index levels\n instead of extracting them as new coordinates (default: False).\n inplace : bool, optional\n If True, modify the dataarray in-place. Otherwise, return a new\n DataArray object.\n\n Returns\n -------\n obj : DataArray\n Another dataarray, with this dataarray's data but replaced\n coordinates.\n\n See Also\n --------\n DataArray.set_index\n \"\"\"\n inplace = _check_inplace(inplace)\n coords, _ = split_indexes(dims_or_levels, self._coords, set(),\n self._level_coords, drop=drop)\n if inplace:\n self._coords = coords\n else:\n return self._replace(coords=coords)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrearranging index levels using input order.", "response": "def reorder_levels(self, dim_order=None, inplace=None,\n **dim_order_kwargs):\n \"\"\"Rearrange index levels using input order.\n\n Parameters\n ----------\n dim_order : optional\n Mapping from names matching dimensions and values given\n by lists representing new level orders. Every given dimension\n must have a multi-index.\n inplace : bool, optional\n If True, modify the dataarray in-place. Otherwise, return a new\n DataArray object.\n **dim_order_kwargs: optional\n The keyword arguments form of ``dim_order``.\n One of dim_order or dim_order_kwargs must be provided.\n\n Returns\n -------\n obj : DataArray\n Another dataarray, with this dataarray's data but replaced\n coordinates.\n \"\"\"\n inplace = _check_inplace(inplace)\n dim_order = either_dict_or_kwargs(dim_order, dim_order_kwargs,\n 'reorder_levels')\n replace_coords = {}\n for dim, order in dim_order.items():\n coord = self._coords[dim]\n index = coord.to_index()\n if not isinstance(index, pd.MultiIndex):\n raise ValueError(\"coordinate %r has no MultiIndex\" % dim)\n replace_coords[dim] = IndexVariable(coord.dims,\n index.reorder_levels(order))\n coords = self._coords.copy()\n coords.update(replace_coords)\n if inplace:\n self._coords = coords\n else:\n return self._replace(coords=coords)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstacking any number of existing dimensions into a single new dimension.", "response": "def stack(self, dimensions=None, **dimensions_kwargs):\n \"\"\"\n Stack any number of existing dimensions into a single new dimension.\n\n New dimensions will be added at the end, and the corresponding\n coordinate variables will be combined into a MultiIndex.\n\n Parameters\n ----------\n dimensions : Mapping of the form new_name=(dim1, dim2, ...)\n Names of new dimensions, and the existing dimensions that they\n replace.\n **dimensions_kwargs:\n The keyword arguments form of ``dimensions``.\n One of dimensions or dimensions_kwargs must be provided.\n\n Returns\n -------\n stacked : DataArray\n DataArray with stacked data.\n\n Examples\n --------\n\n >>> arr = DataArray(np.arange(6).reshape(2, 3),\n ... coords=[('x', ['a', 'b']), ('y', [0, 1, 2])])\n >>> arr\n \n array([[0, 1, 2],\n [3, 4, 5]])\n Coordinates:\n * x (x) |S1 'a' 'b'\n * y (y) int64 0 1 2\n >>> stacked = arr.stack(z=('x', 'y'))\n >>> stacked.indexes['z']\n MultiIndex(levels=[['a', 'b'], [0, 1, 2]],\n labels=[[0, 0, 0, 1, 1, 1], [0, 1, 2, 0, 1, 2]],\n names=['x', 'y'])\n\n See also\n --------\n DataArray.unstack\n \"\"\"\n ds = self._to_temp_dataset().stack(dimensions, **dimensions_kwargs)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef unstack(self, dim=None):\n ds = self._to_temp_dataset().unstack(dim)\n return self._from_temp_dataset(ds)", "response": "Unstack existing dimensions into multiple new dimensions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new DataArray with transposed dimensions.", "response": "def transpose(self, *dims) -> 'DataArray':\n \"\"\"Return a new DataArray object with transposed dimensions.\n\n Parameters\n ----------\n *dims : str, optional\n By default, reverse the dimensions. Otherwise, reorder the\n dimensions to this order.\n\n Returns\n -------\n transposed : DataArray\n The returned DataArray's array is transposed.\n\n Notes\n -----\n This operation returns a view of this array's data. It is\n lazy for dask-backed DataArrays but not for numpy-backed DataArrays\n -- the data will be fully loaded.\n\n See Also\n --------\n numpy.transpose\n Dataset.transpose\n \"\"\"\n variable = self.variable.transpose(*dims)\n return self._replace(variable)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndrop coordinates or index labels from this DataArray.", "response": "def drop(self, labels, dim=None):\n \"\"\"Drop coordinates or index labels from this DataArray.\n\n Parameters\n ----------\n labels : scalar or list of scalars\n Name(s) of coordinate variables or index labels to drop.\n dim : str, optional\n Dimension along which to drop index labels. By default (if\n ``dim is None``), drops coordinates rather than index labels.\n\n Returns\n -------\n dropped : DataArray\n \"\"\"\n if utils.is_scalar(labels):\n labels = [labels]\n ds = self._to_temp_dataset().drop(labels, dim)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dropna(self, dim, how='any', thresh=None):\n ds = self._to_temp_dataset().dropna(dim, how=how, thresh=thresh)\n return self._from_temp_dataset(ds)", "response": "Returns a new array with dropped labels for missing values along the provided dimension."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fillna(self, value):\n if utils.is_dict_like(value):\n raise TypeError('cannot provide fill value as a dictionary with '\n 'fillna on a DataArray')\n out = ops.fillna(self, value)\n return out", "response": "Fill missing values in this array with value."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfill NaN values by propogating values forward", "response": "def ffill(self, dim, limit=None):\n '''Fill NaN values by propogating values forward\n\n *Requires bottleneck.*\n\n Parameters\n ----------\n dim : str\n Specifies the dimension along which to propagate values when\n filling.\n limit : int, default None\n The maximum number of consecutive NaN values to forward fill. In\n other words, if there is a gap with more than this number of\n consecutive NaNs, it will only be partially filled. Must be greater\n than 0 or None for no limit.\n\n Returns\n -------\n DataArray\n '''\n from .missing import ffill\n return ffill(self, dim, limit=limit)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bfill(self, dim, limit=None):\n '''Fill NaN values by propogating values backward\n\n *Requires bottleneck.*\n\n Parameters\n ----------\n dim : str\n Specifies the dimension along which to propagate values when\n filling.\n limit : int, default None\n The maximum number of consecutive NaN values to backward fill. In\n other words, if there is a gap with more than this number of\n consecutive NaNs, it will only be partially filled. Must be greater\n than 0 or None for no limit.\n\n Returns\n -------\n DataArray\n '''\n from .missing import bfill\n return bfill(self, dim, limit=limit)", "response": "Fill NaN values by propogating values backward\n filling."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reduce(self, func, dim=None, axis=None, keep_attrs=None, **kwargs):\n\n var = self.variable.reduce(func, dim, axis, keep_attrs, **kwargs)\n return self._replace_maybe_drop_dims(var)", "response": "Reduce this array by applying func along some dimension."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_pandas(self):\n # TODO: consolidate the info about pandas constructors and the\n # attributes that correspond to their indexes into a separate module?\n constructors = {0: lambda x: x,\n 1: pd.Series,\n 2: pd.DataFrame,\n 3: pd.Panel}\n try:\n constructor = constructors[self.ndim]\n except KeyError:\n raise ValueError('cannot convert arrays with %s dimensions into '\n 'pandas objects' % self.ndim)\n indexes = [self.get_index(dim) for dim in self.dims]\n return constructor(self.values, *indexes)", "response": "Convert this array into a pandas object with the same shape."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert this array and its coordinates into a tidy pandas. DataFrame.", "response": "def to_dataframe(self, name=None):\n \"\"\"Convert this array and its coordinates into a tidy pandas.DataFrame.\n\n The DataFrame is indexed by the Cartesian product of index coordinates\n (in the form of a :py:class:`pandas.MultiIndex`).\n\n Other coordinates are included as columns in the DataFrame.\n \"\"\"\n if name is None:\n name = self.name\n if name is None:\n raise ValueError('cannot convert an unnamed DataArray to a '\n 'DataFrame: use the ``name`` parameter')\n\n dims = OrderedDict(zip(self.dims, self.shape))\n # By using a unique name, we can convert a DataArray into a DataFrame\n # even if it shares a name with one of its coordinates.\n # I would normally use unique_name = object() but that results in a\n # dataframe with columns in the wrong order, for reasons I have not\n # been able to debug (possibly a pandas bug?).\n unique_name = '__unique_name_identifier_z98xfz98xugfg73ho__'\n ds = self._to_dataset_whole(name=unique_name)\n df = ds._to_dataframe(dims)\n df.columns = [name if c == unique_name else c\n for c in df.columns]\n return df"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_series(self):\n index = self.coords.to_index()\n return pd.Series(self.values.reshape(-1), index=index, name=self.name)", "response": "Convert this array into a pandas. Series."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_masked_array(self, copy=True):\n isnull = pd.isnull(self.values)\n return np.ma.MaskedArray(data=self.values, mask=isnull, copy=copy)", "response": "Convert this array into a numpy. ma. MaskedArray."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite the contents of the array to a netCDF file.", "response": "def to_netcdf(self, *args, **kwargs):\n \"\"\"Write DataArray contents to a netCDF file.\n\n Parameters\n ----------\n path : str or Path, optional\n Path to which to save this dataset. If no path is provided, this\n function returns the resulting netCDF file as a bytes object; in\n this case, we need to use scipy.io.netcdf, which does not support\n netCDF version 4 (the default format becomes NETCDF3_64BIT).\n mode : {'w', 'a'}, optional\n Write ('w') or append ('a') mode. If mode='w', any existing file at\n this location will be overwritten.\n format : {'NETCDF4', 'NETCDF4_CLASSIC', 'NETCDF3_64BIT',\n 'NETCDF3_CLASSIC'}, optional\n File format for the resulting netCDF file:\n\n * NETCDF4: Data is stored in an HDF5 file, using netCDF4 API\n features.\n * NETCDF4_CLASSIC: Data is stored in an HDF5 file, using only\n netCDF 3 compatible API features.\n * NETCDF3_64BIT: 64-bit offset version of the netCDF 3 file format,\n which fully supports 2+ GB files, but is only compatible with\n clients linked against netCDF version 3.6.0 or later.\n * NETCDF3_CLASSIC: The classic netCDF 3 file format. It does not\n handle 2+ GB files very well.\n\n All formats are supported by the netCDF4-python library.\n scipy.io.netcdf only supports the last two formats.\n\n The default format is NETCDF4 if you are saving a file to disk and\n have the netCDF4-python library available. Otherwise, xarray falls\n back to using scipy to write netCDF files and defaults to the\n NETCDF3_64BIT format (scipy does not support netCDF4).\n group : str, optional\n Path to the netCDF4 group in the given file to open (only works for\n format='NETCDF4'). The group(s) will be created if necessary.\n engine : {'netcdf4', 'scipy', 'h5netcdf'}, optional\n Engine to use when writing netCDF files. If not provided, the\n default engine is chosen based on available dependencies, with a\n preference for 'netcdf4' if writing to a file on disk.\n encoding : dict, optional\n Nested dictionary with variable names as keys and dictionaries of\n variable specific encodings as values, e.g.,\n ``{'my_variable': {'dtype': 'int16', 'scale_factor': 0.1,\n 'zlib': True}, ...}``\n\n Notes\n -----\n Only xarray.Dataset objects can be written to netCDF files, so\n the xarray.DataArray is converted to a xarray.Dataset object\n containing a single variable. If the DataArray has no name, or if the\n name is the same as a co-ordinate name, then it is given the name\n '__xarray_dataarray_variable__'.\n\n All parameters are passed directly to `xarray.Dataset.to_netcdf`.\n \"\"\"\n from ..backends.api import DATAARRAY_NAME, DATAARRAY_VARIABLE\n\n if self.name is None:\n # If no name is set then use a generic xarray name\n dataset = self.to_dataset(name=DATAARRAY_VARIABLE)\n elif self.name in self.coords or self.name in self.dims:\n # The name is the same as one of the coords names, which netCDF\n # doesn't support, so rename it but keep track of the old name\n dataset = self.to_dataset(name=DATAARRAY_VARIABLE)\n dataset.attrs[DATAARRAY_NAME] = self.name\n else:\n # No problems with the name - so we're fine!\n dataset = self.to_dataset()\n\n return dataset.to_netcdf(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts this xarray. DataArray into a dictionary following xarray. NameConventions.", "response": "def to_dict(self, data=True):\n \"\"\"\n Convert this xarray.DataArray into a dictionary following xarray\n naming conventions.\n\n Converts all variables and attributes to native Python objects.\n Useful for coverting to json. To avoid datetime incompatibility\n use decode_times=False kwarg in xarrray.open_dataset.\n\n Parameters\n ----------\n data : bool, optional\n Whether to include the actual data in the dictionary. When set to\n False, returns just the schema.\n\n See also\n --------\n DataArray.from_dict\n \"\"\"\n d = self.variable.to_dict(data=data)\n d.update({'coords': {}, 'name': self.name})\n for k in self.coords:\n d['coords'][k] = self.coords[k].variable.to_dict(data=data)\n return d"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_dict(cls, d):\n coords = None\n if 'coords' in d:\n try:\n coords = OrderedDict([(k, (v['dims'],\n v['data'],\n v.get('attrs')))\n for k, v in d['coords'].items()])\n except KeyError as e:\n raise ValueError(\n \"cannot convert dict when coords are missing the key \"\n \"'{dims_data}'\".format(dims_data=str(e.args[0])))\n try:\n data = d['data']\n except KeyError:\n raise ValueError(\"cannot convert dict without the key 'data''\")\n else:\n obj = cls(data, coords, d.get('dims'), d.get('name'),\n d.get('attrs'))\n return obj", "response": "Convert a dictionary into an xarray. DataArray\n object."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a pandas. Series into an xarray. DataArray.", "response": "def from_series(cls, series):\n \"\"\"Convert a pandas.Series into an xarray.DataArray.\n\n If the series's index is a MultiIndex, it will be expanded into a\n tensor product of one-dimensional coordinates (filling in missing\n values with NaN). Thus this operation should be the inverse of the\n `to_series` method.\n \"\"\"\n # TODO: add a 'name' parameter\n name = series.name\n df = pd.DataFrame({name: series})\n ds = Dataset.from_dataframe(df)\n return ds[name]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlike equals but also checks the array name attributes and attributes on all coordinates.", "response": "def identical(self, other):\n \"\"\"Like equals, but also checks the array name and attributes, and\n attributes on all coordinates.\n\n See Also\n --------\n DataArray.broadcast_equals\n DataArray.equal\n \"\"\"\n try:\n return (self.name == other.name and\n self._all_compat(other, 'identical'))\n except (TypeError, AttributeError):\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _title_for_slice(self, truncate=50):\n one_dims = []\n for dim, coord in self.coords.items():\n if coord.size == 1:\n one_dims.append('{dim} = {v}'.format(\n dim=dim, v=format_item(coord.values)))\n\n title = ', '.join(one_dims)\n if len(title) > truncate:\n title = title[:(truncate - 3)] + '...'\n\n return title", "response": "Returns a string that can be used for plot titles for a slice of the dataarray."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the n - th order discrete difference along given axis.", "response": "def diff(self, dim, n=1, label='upper'):\n \"\"\"Calculate the n-th order discrete difference along given axis.\n\n Parameters\n ----------\n dim : str, optional\n Dimension over which to calculate the finite difference.\n n : int, optional\n The number of times values are differenced.\n label : str, optional\n The new coordinate in dimension ``dim`` will have the\n values of either the minuend's or subtrahend's coordinate\n for values 'upper' and 'lower', respectively. Other\n values are not supported.\n\n Returns\n -------\n difference : same type as caller\n The n-th order finite difference of this object.\n\n Examples\n --------\n >>> arr = xr.DataArray([5, 5, 6, 6], [[1, 2, 3, 4]], ['x'])\n >>> arr.diff('x')\n \n array([0, 1, 0])\n Coordinates:\n * x (x) int64 2 3 4\n >>> arr.diff('x', 2)\n \n array([ 1, -1])\n Coordinates:\n * x (x) int64 3 4\n\n See Also\n --------\n DataArray.differentiate\n \"\"\"\n ds = self._to_temp_dataset().diff(n=n, dim=dim, label=label)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef shift(self, shifts=None, fill_value=dtypes.NA, **shifts_kwargs):\n variable = self.variable.shift(\n shifts=shifts, fill_value=fill_value, **shifts_kwargs)\n return self._replace(variable=variable)", "response": "Shifts this array by an offset along one or more dimensions."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrolls this array by an offset along one or more dimensions.", "response": "def roll(self, shifts=None, roll_coords=None, **shifts_kwargs):\n \"\"\"Roll this array by an offset along one or more dimensions.\n\n Unlike shift, roll may rotate all variables, including coordinates\n if specified. The direction of rotation is consistent with\n :py:func:`numpy.roll`.\n\n Parameters\n ----------\n roll_coords : bool\n Indicates whether to roll the coordinates by the offset\n The current default of roll_coords (None, equivalent to True) is\n deprecated and will change to False in a future version.\n Explicitly pass roll_coords to silence the warning.\n **shifts : keyword arguments of the form {dim: offset}\n Integer offset to rotate each of the given dimensions. Positive\n offsets roll to the right; negative offsets roll to the left.\n\n Returns\n -------\n rolled : DataArray\n DataArray with the same attributes but rolled data and coordinates.\n\n See also\n --------\n shift\n\n Examples\n --------\n\n >>> arr = xr.DataArray([5, 6, 7], dims='x')\n >>> arr.roll(x=1)\n \n array([7, 5, 6])\n Coordinates:\n * x (x) int64 2 0 1\n \"\"\"\n ds = self._to_temp_dataset().roll(\n shifts=shifts, roll_coords=roll_coords, **shifts_kwargs)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dot(self, other, dims=None):\n if isinstance(other, Dataset):\n raise NotImplementedError('dot products are not yet supported '\n 'with Dataset objects.')\n if not isinstance(other, DataArray):\n raise TypeError('dot only operates on DataArrays.')\n\n return computation.dot(self, other, dims=dims)", "response": "Perform the dot product of two DataArrays along their shared dims."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sortby(self, variables, ascending=True):\n ds = self._to_temp_dataset().sortby(variables, ascending=ascending)\n return self._from_temp_dataset(ds)", "response": "Sort the data array by the specified variables."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef quantile(self, q, dim=None, interpolation='linear', keep_attrs=None):\n\n ds = self._to_temp_dataset().quantile(\n q, dim=dim, keep_attrs=keep_attrs, interpolation=interpolation)\n return self._from_temp_dataset(ds)", "response": "Compute the qth quantile of the data along the specified dimension."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rank(self, dim, pct=False, keep_attrs=None):\n\n ds = self._to_temp_dataset().rank(dim, pct=pct, keep_attrs=keep_attrs)\n return self._from_temp_dataset(ds)", "response": "Ranks the data array."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndifferentiating the array with the second order accurate central differences. .. note:: This feature is limited to simple cartesian geometry, i.e. coord must be one dimensional. Parameters ---------- coord: str The coordinate to be used to compute the gradient. edge_order: 1 or 2. Default 1 N-th order accurate differences at the boundaries. datetime_unit: None or any of {'Y', 'M', 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns', 'ps', 'fs', 'as'} Unit to compute gradient. Only valid for datetime coordinate. Returns ------- differentiated: DataArray See also -------- numpy.gradient: corresponding numpy function Examples -------- >>> da = xr.DataArray(np.arange(12).reshape(4, 3), dims=['x', 'y'], ... coords={'x': [0, 0.1, 1.1, 1.2]}) >>> da array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]]) Coordinates: * x (x) float64 0.0 0.1 1.1 1.2 Dimensions without coordinates: y >>> >>> da.differentiate('x') array([[30. , 30. , 30. ], [27.545455, 27.545455, 27.545455], [27.545455, 27.545455, 27.545455], [30. , 30. , 30. ]]) Coordinates: * x (x) float64 0.0 0.1 1.1 1.2 Dimensions without coordinates: y", "response": "def differentiate(self, coord, edge_order=1, datetime_unit=None):\n \"\"\" Differentiate the array with the second order accurate central\n differences.\n\n .. note::\n This feature is limited to simple cartesian geometry, i.e. coord\n must be one dimensional.\n\n Parameters\n ----------\n coord: str\n The coordinate to be used to compute the gradient.\n edge_order: 1 or 2. Default 1\n N-th order accurate differences at the boundaries.\n datetime_unit: None or any of {'Y', 'M', 'W', 'D', 'h', 'm', 's', 'ms',\n 'us', 'ns', 'ps', 'fs', 'as'}\n Unit to compute gradient. Only valid for datetime coordinate.\n\n Returns\n -------\n differentiated: DataArray\n\n See also\n --------\n numpy.gradient: corresponding numpy function\n\n Examples\n --------\n\n >>> da = xr.DataArray(np.arange(12).reshape(4, 3), dims=['x', 'y'],\n ... coords={'x': [0, 0.1, 1.1, 1.2]})\n >>> da\n \n array([[ 0, 1, 2],\n [ 3, 4, 5],\n [ 6, 7, 8],\n [ 9, 10, 11]])\n Coordinates:\n * x (x) float64 0.0 0.1 1.1 1.2\n Dimensions without coordinates: y\n >>>\n >>> da.differentiate('x')\n \n array([[30. , 30. , 30. ],\n [27.545455, 27.545455, 27.545455],\n [27.545455, 27.545455, 27.545455],\n [30. , 30. , 30. ]])\n Coordinates:\n * x (x) float64 0.0 0.1 1.1 1.2\n Dimensions without coordinates: y\n \"\"\"\n ds = self._to_temp_dataset().differentiate(\n coord, edge_order, datetime_unit)\n return self._from_temp_dataset(ds)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef integrate(self, dim, datetime_unit=None):\n ds = self._to_temp_dataset().integrate(dim, datetime_unit)\n return self._from_temp_dataset(ds)", "response": "Integrate the array with the trapezoidal rule."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef construct(self, window_dim, stride=1, fill_value=dtypes.NA):\n\n from .dataarray import DataArray\n\n window = self.obj.variable.rolling_window(self.dim, self.window,\n window_dim, self.center,\n fill_value=fill_value)\n result = DataArray(window, dims=self.obj.dims + (window_dim,),\n coords=self.obj.coords)\n return result.isel(**{self.dim: slice(None, None, stride)})", "response": "Convert this rolling object to xr. DataArray where the window dimension is stacked as a new dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reduce(self, func, **kwargs):\n rolling_dim = utils.get_temp_dimname(self.obj.dims, '_rolling_dim')\n windows = self.construct(rolling_dim)\n result = windows.reduce(func, dim=rolling_dim, **kwargs)\n\n # Find valid windows based on count.\n counts = self._counts()\n return result.where(counts >= self._min_periods)", "response": "Reduce the items in this group by applying func along some tier dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nnumbers of non - nan entries in each rolling window.", "response": "def _counts(self):\n \"\"\" Number of non-nan entries in each rolling window. \"\"\"\n\n rolling_dim = utils.get_temp_dimname(self.obj.dims, '_rolling_dim')\n # We use False as the fill_value instead of np.nan, since boolean\n # array is faster to be reduced than object array.\n # The use of skipna==False is also faster since it does not need to\n # copy the strided array.\n counts = (self.obj.notnull()\n .rolling(center=self.center, **{self.dim: self.window})\n .construct(rolling_dim, fill_value=False)\n .sum(dim=rolling_dim, skipna=False))\n return counts"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a wrapped function for any function func for the n - tuple class.", "response": "def _reduce_method(cls, func):\n \"\"\"\n Methods to return a wrapped function for any function `func` for\n numpy methods.\n \"\"\"\n\n def wrapped_func(self, **kwargs):\n return self.reduce(func, **kwargs)\n return wrapped_func"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef reduce(self, func, **kwargs):\n from .dataset import Dataset\n reduced = OrderedDict()\n for key, da in self.obj.data_vars.items():\n if self.dim in da.dims:\n reduced[key] = self.rollings[key].reduce(func, **kwargs)\n else:\n reduced[key] = self.obj[key]\n return Dataset(reduced, coords=self.obj.coords)", "response": "Reduce the items in this group by applying func along some tier dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a wrapped function for injecting numpy and bottoleneck methods. is the name of the method that is used to reduce the object.", "response": "def _reduce_method(cls, func):\n \"\"\"\n Return a wrapped function for injecting numpy and bottoleneck methods.\n see ops.inject_datasetrolling_methods\n \"\"\"\n\n def wrapped_func(self, **kwargs):\n from .dataset import Dataset\n reduced = OrderedDict()\n for key, da in self.obj.data_vars.items():\n if self.dim in da.dims:\n reduced[key] = getattr(self.rollings[key],\n func.__name__)(**kwargs)\n else:\n reduced[key] = self.obj[key]\n return Dataset(reduced, coords=self.obj.coords)\n return wrapped_func"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef construct(self, window_dim, stride=1, fill_value=dtypes.NA):\n\n from .dataset import Dataset\n\n dataset = OrderedDict()\n for key, da in self.obj.data_vars.items():\n if self.dim in da.dims:\n dataset[key] = self.rollings[key].construct(\n window_dim, fill_value=fill_value)\n else:\n dataset[key] = da\n return Dataset(dataset, coords=self.obj.coords).isel(\n **{self.dim: slice(None, None, stride)})", "response": "Convert this rolling object to xr. Dataset where the dimension is stacked as a new dimension."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _reduce_method(cls, func):\n def wrapped_func(self, **kwargs):\n from .dataarray import DataArray\n\n reduced = self.obj.variable.coarsen(\n self.windows, func, self.boundary, self.side)\n coords = {}\n for c, v in self.obj.coords.items():\n if c == self.obj.name:\n coords[c] = reduced\n else:\n if any(d in self.windows for d in v.dims):\n coords[c] = v.variable.coarsen(\n self.windows, self.coord_func[c],\n self.boundary, self.side)\n else:\n coords[c] = v\n return DataArray(reduced, dims=self.obj.dims, coords=coords)\n\n return wrapped_func", "response": "Returns a wrapped function for injecting numpy methods."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a wrapped function for injecting numpy methods.", "response": "def _reduce_method(cls, func):\n \"\"\"\n Return a wrapped function for injecting numpy methods.\n see ops.inject_coarsen_methods\n \"\"\"\n def wrapped_func(self, **kwargs):\n from .dataset import Dataset\n\n reduced = OrderedDict()\n for key, da in self.obj.data_vars.items():\n reduced[key] = da.variable.coarsen(\n self.windows, func, self.boundary, self.side)\n\n coords = {}\n for c, v in self.obj.coords.items():\n if any(d in self.windows for d in v.dims):\n coords[c] = v.variable.coarsen(\n self.windows, self.coord_func[c],\n self.boundary, self.side)\n else:\n coords[c] = v.variable\n return Dataset(reduced, coords=coords)\n\n return wrapped_func"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ensure_fixed_length_bytes(var):\n dims, data, attrs, encoding = unpack_for_encoding(var)\n if check_vlen_dtype(data.dtype) == bytes:\n # TODO: figure out how to handle this with dask\n data = np.asarray(data, dtype=np.string_)\n return Variable(dims, data, attrs, encoding)", "response": "Ensure that a variable with vlen bytes is converted to fixed width."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting numpy arrays from fixed width bytes to characters.", "response": "def bytes_to_char(arr):\n \"\"\"Convert numpy/dask arrays from fixed width bytes to characters.\"\"\"\n if arr.dtype.kind != 'S':\n raise ValueError('argument must have a fixed-width bytes dtype')\n\n if isinstance(arr, dask_array_type):\n import dask.array as da\n return da.map_blocks(_numpy_bytes_to_char, arr,\n dtype='S1',\n chunks=arr.chunks + ((arr.dtype.itemsize,)),\n new_axis=[arr.ndim])\n else:\n return _numpy_bytes_to_char(arr)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _numpy_bytes_to_char(arr):\n # ensure the array is contiguous\n arr = np.array(arr, copy=False, order='C', dtype=np.string_)\n return arr.reshape(arr.shape + (1,)).view('S1')", "response": "Like netCDF4. stringtochar but faster and more flexible.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef char_to_bytes(arr):\n if arr.dtype != 'S1':\n raise ValueError(\"argument must have dtype='S1'\")\n\n if not arr.ndim:\n # no dimension to concatenate along\n return arr\n\n size = arr.shape[-1]\n\n if not size:\n # can't make an S0 dtype\n return np.zeros(arr.shape[:-1], dtype=np.string_)\n\n if isinstance(arr, dask_array_type):\n import dask.array as da\n\n if len(arr.chunks[-1]) > 1:\n raise ValueError('cannot stacked dask character array with '\n 'multiple chunks in the last dimension: {}'\n .format(arr))\n\n dtype = np.dtype('S' + str(arr.shape[-1]))\n return da.map_blocks(_numpy_char_to_bytes, arr,\n dtype=dtype,\n chunks=arr.chunks[:-1],\n drop_axis=[arr.ndim - 1])\n else:\n return StackedBytesArray(arr)", "response": "Convert numpy arrays from characters to fixed width bytes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nliking netCDF4. chartostring but faster and more flexible.", "response": "def _numpy_char_to_bytes(arr):\n \"\"\"Like netCDF4.chartostring, but faster and more flexible.\n \"\"\"\n # based on: http://stackoverflow.com/a/10984878/809705\n arr = np.array(arr, copy=False, order='C')\n dtype = 'S' + str(arr.shape[-1])\n return arr.view(dtype).reshape(arr.shape[:-1])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef safe_cast_to_index(array: Any) -> pd.Index:\n if isinstance(array, pd.Index):\n index = array\n elif hasattr(array, 'to_index'):\n index = array.to_index()\n else:\n kwargs = {}\n if hasattr(array, 'dtype') and array.dtype.kind == 'O':\n kwargs['dtype'] = object\n index = pd.Index(np.asarray(array), **kwargs)\n return _maybe_cast_to_cftimeindex(index)", "response": "Given an array safely cast it to a pandas. Index."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a MultiIndex from a product with refactorizing levels.", "response": "def multiindex_from_product_levels(levels: Sequence[pd.Index],\n names: Optional[Sequence[str]] = None\n ) -> pd.MultiIndex:\n \"\"\"Creating a MultiIndex from a product without refactorizing levels.\n\n Keeping levels the same gives back the original labels when we unstack.\n\n Parameters\n ----------\n levels : sequence of pd.Index\n Values for each MultiIndex level.\n names : optional sequence of objects\n Names for each level.\n\n Returns\n -------\n pandas.MultiIndex\n \"\"\"\n if any(not isinstance(lev, pd.Index) for lev in levels):\n raise TypeError('levels must be a list of pd.Index objects')\n\n split_labels, levels = zip(*[lev.factorize() for lev in levels])\n labels_mesh = np.meshgrid(*split_labels, indexing='ij')\n labels = [x.ravel() for x in labels_mesh]\n return pd.MultiIndex(levels, labels, sortorder=0, names=names)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef maybe_wrap_array(original, new_array):\n # in case func lost array's metadata\n if isinstance(new_array, np.ndarray) and new_array.shape == original.shape:\n return original.__array_wrap__(new_array)\n else:\n return new_array", "response": "Wrap a transformed array with __array_wrap__ is it can be done safely."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncompares two objects for equivalence ( identity or equality", "response": "def equivalent(first: T, second: T) -> bool:\n \"\"\"Compare two objects for equivalence (identity or equality), using\n array_equiv if either object is an ndarray\n \"\"\"\n # TODO: refactor to avoid circular import\n from . import duck_array_ops\n if isinstance(first, np.ndarray) or isinstance(second, np.ndarray):\n return duck_array_ops.array_equiv(first, second)\n else:\n return ((first is second) or\n (first == second) or\n (pd.isnull(first) and pd.isnull(second)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef peek_at(iterable: Iterable[T]) -> Tuple[T, Iterator[T]]:\n gen = iter(iterable)\n peek = next(gen)\n return peek, itertools.chain([peek], gen)", "response": "Returns the first value from iterable as well as a new iterator with\n the same content as the original iterable"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_safety_check(first_dict: MutableMapping[K, V],\n second_dict: Mapping[K, V],\n compat: Callable[[V, V], bool] = equivalent) -> None:\n \"\"\"Check the safety of updating one dictionary with another.\n\n Raises ValueError if dictionaries have non-compatible values for any key,\n where compatibility is determined by identity (they are the same item) or\n the `compat` function.\n\n Parameters\n ----------\n first_dict, second_dict : dict-like\n All items in the second dictionary are checked against for conflicts\n against items in the first dictionary.\n compat : function, optional\n Binary operator to determine if two values are compatible. By default,\n checks for equivalence.\n \"\"\"\n for k, v in second_dict.items():\n if k in first_dict and not compat(v, first_dict[k]):\n raise ValueError('unsafe to merge dictionaries without '\n 'overriding values; conflicting key %r' % k)", "response": "Check the safety of updating one dictionary with another."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_incompatible_items(first_dict: MutableMapping[K, V],\n second_dict: Mapping[K, V],\n compat: Callable[[V, V], bool] = equivalent\n ) -> None:\n \"\"\"Remove incompatible items from the first dictionary in-place.\n\n Items are retained if their keys are found in both dictionaries and the\n values are compatible.\n\n Parameters\n ----------\n first_dict, second_dict : dict-like\n Mappings to merge.\n compat : function, optional\n Binary operator to determine if two values are compatible. By default,\n checks for equivalence.\n \"\"\"\n for k in list(first_dict):\n if k not in second_dict or not compat(first_dict[k], second_dict[k]):\n del first_dict[k]", "response": "Removes incompatible items from the first dictionary in - place."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_scalar(value: Any) -> bool:\n return (\n getattr(value, 'ndim', None) == 0 or\n isinstance(value, (str, bytes)) or not\n isinstance(value, (Iterable, ) + dask_array_type))", "response": "Whether a value is a scalar."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_0d_object_array(value: Any) -> np.ndarray:\n result = np.empty((), dtype=object)\n result[()] = value\n return result", "response": "Given a value wrap it in a 0 - D numpy. ndarray with dtype = object.\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_0d_array(value: Any) -> np.ndarray:\n if np.isscalar(value) or (isinstance(value, np.ndarray) and\n value.ndim == 0):\n return np.array(value)\n else:\n return to_0d_object_array(value)", "response": "Given a value wrap it in a 0 - D numpy. ndarray.\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntest equivalence of two dict - like objects.", "response": "def dict_equiv(first: Mapping[K, V], second: Mapping[K, V],\n compat: Callable[[V, V], bool] = equivalent) -> bool:\n \"\"\"Test equivalence of two dict-like objects. If any of the values are\n numpy arrays, compare them correctly.\n\n Parameters\n ----------\n first, second : dict-like\n Dictionaries to compare for equality\n compat : function, optional\n Binary operator to determine if two values are compatible. By default,\n checks for equivalence.\n\n Returns\n -------\n equals : bool\n True if the dictionaries are equal\n \"\"\"\n for k in first:\n if k not in second or not compat(first[k], second[k]):\n return False\n for k in second:\n if k not in first:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the intersection of two dictionaries.", "response": "def ordered_dict_intersection(first_dict: Mapping[K, V],\n second_dict: Mapping[K, V],\n compat: Callable[[V, V], bool] = equivalent\n ) -> MutableMapping[K, V]:\n \"\"\"Return the intersection of two dictionaries as a new OrderedDict.\n\n Items are retained if their keys are found in both dictionaries and the\n values are compatible.\n\n Parameters\n ----------\n first_dict, second_dict : dict-like\n Mappings to merge.\n compat : function, optional\n Binary operator to determine if two values are compatible. By default,\n checks for equivalence.\n\n Returns\n -------\n intersection : OrderedDict\n Intersection of the contents.\n \"\"\"\n new_dict = OrderedDict(first_dict)\n remove_incompatible_items(new_dict, second_dict, compat)\n return new_dict"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_uniform_spaced(arr, **kwargs) -> bool:\n arr = np.array(arr, dtype=float)\n diffs = np.diff(arr)\n return bool(np.isclose(diffs.min(), diffs.max(), **kwargs))", "response": "Return True if values of an array are uniformly spaced and sorted."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert numpy objects to native Python objects", "response": "def decode_numpy_dict_values(attrs: Mapping[K, V]) -> Dict[K, V]:\n \"\"\"Convert attribute values from numpy objects to native Python objects,\n for use in to_dict\n \"\"\"\n attrs = dict(attrs)\n for k, v in attrs.items():\n if isinstance(v, np.ndarray):\n attrs[k] = v.tolist()\n elif isinstance(v, np.generic):\n attrs[k] = v.item()\n return attrs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ensure_us_time_resolution(val):\n if np.issubdtype(val.dtype, np.datetime64):\n val = val.astype('datetime64[us]')\n elif np.issubdtype(val.dtype, np.timedelta64):\n val = val.astype('timedelta64[us]')\n return val", "response": "Convert val out of numpy time for use in to_dict.\n Needed because of numpy bug GH#7619"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting a new dimension name based on new_dim.", "response": "def get_temp_dimname(dims: Container[Hashable], new_dim: Hashable) -> Hashable:\n \"\"\" Get an new dimension name based on new_dim, that is not used in dims.\n If the same name exists, we add an underscore(s) in the head.\n\n Example1:\n dims: ['a', 'b', 'c']\n new_dim: ['_rolling']\n -> ['_rolling']\n Example2:\n dims: ['a', 'b', 'c', '_rolling']\n new_dim: ['_rolling']\n -> ['__rolling']\n \"\"\"\n while new_dim in dims:\n new_dim = '_' + str(new_dim)\n return new_dim"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive an object array with no missing values infer its dtype from its first element", "response": "def _infer_dtype(array, name=None):\n \"\"\"Given an object array with no missing values, infer its dtype from its\n first element\n \"\"\"\n if array.dtype.kind != 'O':\n raise TypeError('infer_type must be called on a dtype=object array')\n\n if array.size == 0:\n return np.dtype(float)\n\n element = array[(0,) * array.ndim]\n if isinstance(element, (bytes, str)):\n return strings.create_vlen_dtype(type(element))\n\n dtype = np.array(element).dtype\n if dtype.kind != 'O':\n return dtype\n\n raise ValueError('unable to infer dtype on variable {!r}; xarray '\n 'cannot serialize arbitrary Python objects'\n .format(name))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a copy of an array with the given dtype.", "response": "def _copy_with_dtype(data, dtype):\n \"\"\"Create a copy of an array with the given dtype.\n\n We use this instead of np.array() to ensure that custom object dtypes end\n up on the resulting array.\n \"\"\"\n result = np.empty(data.shape, dtype)\n result[...] = data\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencode a variable into a CF - encoded version of the variable.", "response": "def encode_cf_variable(var, needs_copy=True, name=None):\n \"\"\"\n Converts an Variable into an Variable which follows some\n of the CF conventions:\n\n - Nans are masked using _FillValue (or the deprecated missing_value)\n - Rescaling via: scale_factor and add_offset\n - datetimes are converted to the CF 'units since time' format\n - dtype encodings are enforced.\n\n Parameters\n ----------\n var : xarray.Variable\n A variable holding un-encoded data.\n\n Returns\n -------\n out : xarray.Variable\n A variable which has been encoded as described above.\n \"\"\"\n ensure_not_multiindex(var, name=name)\n\n for coder in [times.CFDatetimeCoder(),\n times.CFTimedeltaCoder(),\n variables.CFScaleOffsetCoder(),\n variables.CFMaskCoder(),\n variables.UnsignedIntegerCoder()]:\n var = coder.encode(var, name=name)\n\n # TODO(shoyer): convert all of these to use coders, too:\n var = maybe_encode_nonstring_dtype(var, name=name)\n var = maybe_default_fill_value(var)\n var = maybe_encode_bools(var)\n var = ensure_dtype_not_object(var, name=name)\n return var"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef decode_cf_variable(name, var, concat_characters=True, mask_and_scale=True,\n decode_times=True, decode_endianness=True,\n stack_char_dim=True, use_cftime=None):\n \"\"\"\n Decodes a variable which may hold CF encoded information.\n\n This includes variables that have been masked and scaled, which\n hold CF style time variables (this is almost always the case if\n the dataset has been serialized) and which have strings encoded\n as character arrays.\n\n Parameters\n ----------\n name: str\n Name of the variable. Used for better error messages.\n var : Variable\n A variable holding potentially CF encoded information.\n concat_characters : bool\n Should character arrays be concatenated to strings, for\n example: ['h', 'e', 'l', 'l', 'o'] -> 'hello'\n mask_and_scale: bool\n Lazily scale (using scale_factor and add_offset) and mask\n (using _FillValue). If the _Unsigned attribute is present\n treat integer arrays as unsigned.\n decode_times : bool\n Decode cf times ('hours since 2000-01-01') to np.datetime64.\n decode_endianness : bool\n Decode arrays from non-native to native endianness.\n stack_char_dim : bool\n Whether to stack characters into bytes along the last dimension of this\n array. Passed as an argument because we need to look at the full\n dataset to figure out if this is appropriate.\n use_cftime: bool, optional\n Only relevant if encoded dates come from a standard calendar\n (e.g. 'gregorian', 'proleptic_gregorian', 'standard', or not\n specified). If None (default), attempt to decode times to\n ``np.datetime64[ns]`` objects; if this is not possible, decode times to\n ``cftime.datetime`` objects. If True, always decode times to\n ``cftime.datetime`` objects, regardless of whether or not they can be\n represented using ``np.datetime64[ns]`` objects. If False, always\n decode times to ``np.datetime64[ns]`` objects; if this is not possible\n raise an error.\n\n Returns\n -------\n out : Variable\n A variable holding the decoded equivalent of var.\n \"\"\"\n var = as_variable(var)\n original_dtype = var.dtype\n\n if concat_characters:\n if stack_char_dim:\n var = strings.CharacterArrayCoder().decode(var, name=name)\n var = strings.EncodedStringCoder().decode(var)\n\n if mask_and_scale:\n for coder in [variables.UnsignedIntegerCoder(),\n variables.CFMaskCoder(),\n variables.CFScaleOffsetCoder()]:\n var = coder.decode(var, name=name)\n\n if decode_times:\n for coder in [times.CFTimedeltaCoder(),\n times.CFDatetimeCoder(use_cftime=use_cftime)]:\n var = coder.decode(var, name=name)\n\n dimensions, data, attributes, encoding = (\n variables.unpack_for_decoding(var))\n # TODO(shoyer): convert everything below to use coders\n\n if decode_endianness and not data.dtype.isnative:\n # do this last, so it's only done if we didn't already unmask/scale\n data = NativeEndiannessArray(data)\n original_dtype = data.dtype\n\n encoding.setdefault('dtype', original_dtype)\n\n if 'dtype' in attributes and attributes['dtype'] == 'bool':\n del attributes['dtype']\n data = BoolTypeArray(data)\n\n if not isinstance(data, dask_array_type):\n data = indexing.LazilyOuterIndexedArray(data)\n\n return Variable(dimensions, data, attributes, encoding=encoding)", "response": "Decodes a variable in CF - encoded form into a new variable in the CF - encoded form."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the time attributes of the time variables that are in the time bounds variable.", "response": "def _update_bounds_attributes(variables):\n \"\"\"Adds time attributes to time bounds variables.\n\n Variables handling time bounds (\"Cell boundaries\" in the CF\n conventions) do not necessarily carry the necessary attributes to be\n decoded. This copies the attributes from the time variable to the\n associated boundaries.\n\n See Also:\n\n http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/\n cf-conventions.html#cell-boundaries\n\n https://github.com/pydata/xarray/issues/2565\n \"\"\"\n\n # For all time variables with bounds\n for v in variables.values():\n attrs = v.attrs\n has_date_units = 'units' in attrs and 'since' in attrs['units']\n if has_date_units and 'bounds' in attrs:\n if attrs['bounds'] in variables:\n bounds_attrs = variables[attrs['bounds']].attrs\n bounds_attrs.setdefault('units', attrs['units'])\n if 'calendar' in attrs:\n bounds_attrs.setdefault('calendar', attrs['calendar'])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecodes several CF encoded variables.", "response": "def decode_cf_variables(variables, attributes, concat_characters=True,\n mask_and_scale=True, decode_times=True,\n decode_coords=True, drop_variables=None,\n use_cftime=None):\n \"\"\"\n Decode several CF encoded variables.\n\n See: decode_cf_variable\n \"\"\"\n dimensions_used_by = defaultdict(list)\n for v in variables.values():\n for d in v.dims:\n dimensions_used_by[d].append(v)\n\n def stackable(dim):\n # figure out if a dimension can be concatenated over\n if dim in variables:\n return False\n for v in dimensions_used_by[dim]:\n if v.dtype.kind != 'S' or dim != v.dims[-1]:\n return False\n return True\n\n coord_names = set()\n\n if isinstance(drop_variables, str):\n drop_variables = [drop_variables]\n elif drop_variables is None:\n drop_variables = []\n drop_variables = set(drop_variables)\n\n # Time bounds coordinates might miss the decoding attributes\n if decode_times:\n _update_bounds_attributes(variables)\n\n new_vars = OrderedDict()\n for k, v in variables.items():\n if k in drop_variables:\n continue\n stack_char_dim = (concat_characters and v.dtype == 'S1' and\n v.ndim > 0 and stackable(v.dims[-1]))\n new_vars[k] = decode_cf_variable(\n k, v, concat_characters=concat_characters,\n mask_and_scale=mask_and_scale, decode_times=decode_times,\n stack_char_dim=stack_char_dim, use_cftime=use_cftime)\n if decode_coords:\n var_attrs = new_vars[k].attrs\n if 'coordinates' in var_attrs:\n coord_str = var_attrs['coordinates']\n var_coord_names = coord_str.split()\n if all(k in variables for k in var_coord_names):\n new_vars[k].encoding['coordinates'] = coord_str\n del var_attrs['coordinates']\n coord_names.update(var_coord_names)\n\n if decode_coords and 'coordinates' in attributes:\n attributes = OrderedDict(attributes)\n coord_names.update(attributes.pop('coordinates').split())\n\n return new_vars, attributes, coord_names"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndecode the given Dataset or DataStore into a new Dataset.", "response": "def decode_cf(obj, concat_characters=True, mask_and_scale=True,\n decode_times=True, decode_coords=True, drop_variables=None,\n use_cftime=None):\n \"\"\"Decode the given Dataset or Datastore according to CF conventions into\n a new Dataset.\n\n Parameters\n ----------\n obj : Dataset or DataStore\n Object to decode.\n concat_characters : bool, optional\n Should character arrays be concatenated to strings, for\n example: ['h', 'e', 'l', 'l', 'o'] -> 'hello'\n mask_and_scale: bool, optional\n Lazily scale (using scale_factor and add_offset) and mask\n (using _FillValue).\n decode_times : bool, optional\n Decode cf times (e.g., integers since 'hours since 2000-01-01') to\n np.datetime64.\n decode_coords : bool, optional\n Use the 'coordinates' attribute on variable (or the dataset itself) to\n identify coordinates.\n drop_variables: string or iterable, optional\n A variable or list of variables to exclude from being parsed from the\n dataset. This may be useful to drop variables with problems or\n inconsistent values.\n use_cftime: bool, optional\n Only relevant if encoded dates come from a standard calendar\n (e.g. 'gregorian', 'proleptic_gregorian', 'standard', or not\n specified). If None (default), attempt to decode times to\n ``np.datetime64[ns]`` objects; if this is not possible, decode times to\n ``cftime.datetime`` objects. If True, always decode times to\n ``cftime.datetime`` objects, regardless of whether or not they can be\n represented using ``np.datetime64[ns]`` objects. If False, always\n decode times to ``np.datetime64[ns]`` objects; if this is not possible\n raise an error.\n\n Returns\n -------\n decoded : Dataset\n \"\"\"\n from .core.dataset import Dataset\n from .backends.common import AbstractDataStore\n\n if isinstance(obj, Dataset):\n vars = obj._variables\n attrs = obj.attrs\n extra_coords = set(obj.coords)\n file_obj = obj._file_obj\n encoding = obj.encoding\n elif isinstance(obj, AbstractDataStore):\n vars, attrs = obj.load()\n extra_coords = set()\n file_obj = obj\n encoding = obj.get_encoding()\n else:\n raise TypeError('can only decode Dataset or DataStore objects')\n\n vars, attrs, coord_names = decode_cf_variables(\n vars, attrs, concat_characters, mask_and_scale, decode_times,\n decode_coords, drop_variables=drop_variables, use_cftime=use_cftime)\n ds = Dataset(vars, attrs=attrs)\n ds = ds.set_coords(coord_names.union(extra_coords).intersection(vars))\n ds._file_obj = file_obj\n ds.encoding = encoding\n\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cf_decoder(variables, attributes,\n concat_characters=True, mask_and_scale=True,\n decode_times=True):\n \"\"\"\n Decode a set of CF encoded variables and attributes.\n\n See Also, decode_cf_variable\n\n Parameters\n ----------\n variables : dict\n A dictionary mapping from variable name to xarray.Variable\n attributes : dict\n A dictionary mapping from attribute name to value\n concat_characters : bool\n Should character arrays be concatenated to strings, for\n example: ['h', 'e', 'l', 'l', 'o'] -> 'hello'\n mask_and_scale: bool\n Lazily scale (using scale_factor and add_offset) and mask\n (using _FillValue).\n decode_times : bool\n Decode cf times ('hours since 2000-01-01') to np.datetime64.\n\n Returns\n -------\n decoded_variables : dict\n A dictionary mapping from variable name to xarray.Variable objects.\n decoded_attributes : dict\n A dictionary mapping from attribute name to values.\n \"\"\"\n variables, attributes, _ = decode_cf_variables(\n variables, attributes, concat_characters, mask_and_scale, decode_times)\n return variables, attributes", "response": "Decode a set of CF encoded variables and attributes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nencoding coordinates on the given dataset object into variable specific and global attributes.", "response": "def encode_dataset_coordinates(dataset):\n \"\"\"Encode coordinates on the given dataset object into variable specific\n and global attributes.\n\n When possible, this is done according to CF conventions.\n\n Parameters\n ----------\n dataset : Dataset\n Object to encode.\n\n Returns\n -------\n variables : dict\n attrs : dict\n \"\"\"\n non_dim_coord_names = set(dataset.coords) - set(dataset.dims)\n return _encode_coordinates(dataset._variables, dataset.attrs,\n non_dim_coord_names=non_dim_coord_names)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cf_encoder(variables, attributes):\n new_vars = OrderedDict((k, encode_cf_variable(v, name=k))\n for k, v in variables.items())\n return new_vars, attributes", "response": "A function which takes a dict of variables and attributes and encodes them to conform to CF conventions as much\n as possible."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef coerce_nc3_dtype(arr):\n dtype = str(arr.dtype)\n if dtype in _nc3_dtype_coercions:\n new_dtype = _nc3_dtype_coercions[dtype]\n # TODO: raise a warning whenever casting the data-type instead?\n cast_arr = arr.astype(new_dtype)\n if not (cast_arr == arr).all():\n raise ValueError('could not safely cast array from dtype %s to %s'\n % (dtype, new_dtype))\n arr = cast_arr\n return arr", "response": "Coerce an array to a data type that can be stored in a netCDF - 3 file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_valid_nc3_name(s):\n if not isinstance(s, str):\n return False\n if not isinstance(s, str):\n s = s.decode('utf-8')\n num_bytes = len(s.encode('utf-8'))\n return ((unicodedata.normalize('NFC', s) == s) and\n (s not in _reserved_names) and\n (num_bytes >= 0) and\n ('/' not in s) and\n (s[-1] != ' ') and\n (_isalnumMUTF8(s[0]) or (s[0] == '_')) and\n all((_isalnumMUTF8(c) or c in _specialchars for c in s)))", "response": "Test whether an object can be converted to a netCDF - 3 name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_variable(obj, name=None) -> 'Union[Variable, IndexVariable]':\n from .dataarray import DataArray\n\n # TODO: consider extending this method to automatically handle Iris and\n if isinstance(obj, DataArray):\n # extract the primary Variable from DataArrays\n obj = obj.variable\n\n if isinstance(obj, Variable):\n obj = obj.copy(deep=False)\n elif isinstance(obj, tuple):\n try:\n obj = Variable(*obj)\n except (TypeError, ValueError) as error:\n # use .format() instead of % because it handles tuples consistently\n raise error.__class__('Could not convert tuple of form '\n '(dims, data[, attrs, encoding]): '\n '{} to Variable.'.format(obj))\n elif utils.is_scalar(obj):\n obj = Variable([], obj)\n elif isinstance(obj, (pd.Index, IndexVariable)) and obj.name is not None:\n obj = Variable(obj.name, obj)\n elif isinstance(obj, (set, dict)):\n raise TypeError(\n \"variable %r has invalid type %r\" % (name, type(obj)))\n elif name is not None:\n data = as_compatible_data(obj)\n if data.ndim != 1:\n raise MissingDimensionsError(\n 'cannot set variable %r with %r-dimensional data '\n 'without explicit dimension names. Pass a tuple of '\n '(dims, data) instead.' % (name, data.ndim))\n obj = Variable(name, data, fastpath=True)\n else:\n raise TypeError('unable to convert object into a variable without an '\n 'explicit list of dimensions: %r' % obj)\n\n if name is not None and name in obj.dims:\n # convert the Variable into an Index\n if obj.ndim != 1:\n raise MissingDimensionsError(\n '%r has more than 1-dimension and the same name as one of its '\n 'dimensions %r. xarray disallows such variables because they '\n 'conflict with the coordinates used to label '\n 'dimensions.' % (name, obj.dims))\n obj = obj.to_index_variable()\n\n return obj", "response": "Convert an object into a Variable."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _possibly_convert_objects(values):\n return np.asarray(pd.Series(values.ravel())).reshape(values.shape)", "response": "Convert arrays of datetime. datetime and datetime. timedelta objects into\n datetime64 and timedelta64 according to the pandas convention."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef as_compatible_data(data, fastpath=False):\n if fastpath and getattr(data, 'ndim', 0) > 0:\n # can't use fastpath (yet) for scalars\n return _maybe_wrap_data(data)\n\n if isinstance(data, Variable):\n return data.data\n\n if isinstance(data, NON_NUMPY_SUPPORTED_ARRAY_TYPES):\n return _maybe_wrap_data(data)\n\n if isinstance(data, tuple):\n data = utils.to_0d_object_array(data)\n\n if isinstance(data, pd.Timestamp):\n # TODO: convert, handle datetime objects, too\n data = np.datetime64(data.value, 'ns')\n\n if isinstance(data, timedelta):\n data = np.timedelta64(getattr(data, 'value', data), 'ns')\n\n # we don't want nested self-described arrays\n data = getattr(data, 'values', data)\n\n if isinstance(data, np.ma.MaskedArray):\n mask = np.ma.getmaskarray(data)\n if mask.any():\n dtype, fill_value = dtypes.maybe_promote(data.dtype)\n data = np.asarray(data, dtype=dtype)\n data[mask] = fill_value\n else:\n data = np.asarray(data)\n\n # validate whether the data is valid data types\n data = np.asarray(data)\n\n if isinstance(data, np.ndarray):\n if data.dtype.kind == 'O':\n data = _possibly_convert_objects(data)\n elif data.dtype.kind == 'M':\n data = np.asarray(data, 'datetime64[ns]')\n elif data.dtype.kind == 'm':\n data = np.asarray(data, 'timedelta64[ns]')\n\n return _maybe_wrap_data(data)", "response": "Prepare and wrap data to put in a Variable."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _as_array_or_item(data):\n data = np.asarray(data)\n if data.ndim == 0:\n if data.dtype.kind == 'M':\n data = np.datetime64(data, 'ns')\n elif data.dtype.kind == 'm':\n data = np.timedelta64(data, 'ns')\n return data", "response": "Return the given values as a numpy array or as an individual item if the given values are 0d datetime64 or timedelta64 array."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate broadcast compatible variables with the same dimensions.", "response": "def _broadcast_compat_variables(*variables):\n \"\"\"Create broadcast compatible variables, with the same dimensions.\n\n Unlike the result of broadcast_variables(), some variables may have\n dimensions of size 1 instead of the the size of the broadcast dimension.\n \"\"\"\n dims = tuple(_unified_dims(variables))\n return tuple(var.set_dims(dims) if var.dims != dims else var\n for var in variables)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives any number of variables return variables with matching dimensions and broadcast data.", "response": "def broadcast_variables(*variables):\n \"\"\"Given any number of variables, return variables with matching dimensions\n and broadcast data.\n\n The data on the returned variables will be a view of the data on the\n corresponding original arrays, but dimensions will be reordered and\n inserted so that both broadcast arrays have the same dimensions. The new\n dimensions are sorted in order of appearance in the first variable's\n dimensions followed by the second variable's dimensions.\n \"\"\"\n dims_map = _unified_dims(variables)\n dims_tuple = tuple(dims_map)\n return tuple(var.set_dims(dims_map) if var.dims != dims_tuple else var\n for var in variables)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef concat(variables, dim='concat_dim', positions=None, shortcut=False):\n variables = list(variables)\n if all(isinstance(v, IndexVariable) for v in variables):\n return IndexVariable.concat(variables, dim, positions, shortcut)\n else:\n return Variable.concat(variables, dim, positions, shortcut)", "response": "Concatenate a list of Variable objects along a new or existing dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef assert_unique_multiindex_level_names(variables):\n level_names = defaultdict(list)\n all_level_names = set()\n for var_name, var in variables.items():\n if isinstance(var._data, PandasIndexAdapter):\n idx_level_names = var.to_index_variable().level_names\n if idx_level_names is not None:\n for n in idx_level_names:\n level_names[n].append('%r (%s)' % (n, var_name))\n if idx_level_names:\n all_level_names.update(idx_level_names)\n\n for k, v in level_names.items():\n if k in variables:\n v.append('(%s)' % k)\n\n duplicate_names = [v for v in level_names.values() if len(v) > 1]\n if duplicate_names:\n conflict_str = '\\n'.join([', '.join(v) for v in duplicate_names])\n raise ValueError('conflicting MultiIndex level name(s):\\n%s'\n % conflict_str)\n # Check confliction between level names and dimensions GH:2299\n for k, v in variables.items():\n for d in v.dims:\n if d in all_level_names:\n raise ValueError('conflicting level / dimension names. {} '\n 'already exists as a level name.'.format(d))", "response": "Check for uniqueness of MultiIndex level names in all given variables."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_base_variable(self):\n return Variable(self.dims, self._data, self._attrs,\n encoding=self._encoding, fastpath=True)", "response": "Return this variable as a base xarray. Variable"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_index_variable(self):\n return IndexVariable(self.dims, self._data, self._attrs,\n encoding=self._encoding, fastpath=True)", "response": "Return this variable as an xarray. IndexVariable"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_dict(self, data=True):\n item = {'dims': self.dims,\n 'attrs': decode_numpy_dict_values(self.attrs)}\n if data:\n item['data'] = ensure_us_time_resolution(self.values).tolist()\n else:\n item.update({'dtype': str(self.dtype), 'shape': self.shape})\n return item", "response": "Returns a dictionary representation of the object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _broadcast_indexes(self, key):\n key = self._item_key_to_tuple(key) # key is a tuple\n # key is a tuple of full size\n key = indexing.expanded_indexer(key, self.ndim)\n # Convert a scalar Variable to an integer\n key = tuple(\n k.data.item() if isinstance(k, Variable) and k.ndim == 0 else k\n for k in key)\n # Convert a 0d-array to an integer\n key = tuple(\n k.item() if isinstance(k, np.ndarray) and k.ndim == 0 else k\n for k in key)\n\n if all(isinstance(k, BASIC_INDEXING_TYPES) for k in key):\n return self._broadcast_indexes_basic(key)\n\n self._validate_indexers(key)\n # Detect it can be mapped as an outer indexer\n # If all key is unlabeled, or\n # key can be mapped as an OuterIndexer.\n if all(not isinstance(k, Variable) for k in key):\n return self._broadcast_indexes_outer(key)\n\n # If all key is 1-dimensional and there are no duplicate labels,\n # key can be mapped as an OuterIndexer.\n dims = []\n for k, d in zip(key, self.dims):\n if isinstance(k, Variable):\n if len(k.dims) > 1:\n return self._broadcast_indexes_vectorized(key)\n dims.append(k.dims[0])\n elif not isinstance(k, integer_types):\n dims.append(d)\n if len(set(dims)) == len(dims):\n return self._broadcast_indexes_outer(key)\n\n return self._broadcast_indexes_vectorized(key)", "response": "This method is used to broadcast the indexes of the items in the item."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake sanity checks for the indexes.", "response": "def _validate_indexers(self, key):\n \"\"\" Make sanity checks \"\"\"\n for dim, k in zip(self.dims, key):\n if isinstance(k, BASIC_INDEXING_TYPES):\n pass\n else:\n if not isinstance(k, Variable):\n k = np.asarray(k)\n if k.ndim > 1:\n raise IndexError(\n \"Unlabeled multi-dimensional array cannot be \"\n \"used for indexing: {}\".format(k))\n if k.dtype.kind == 'b':\n if self.shape[self.get_axis_num(dim)] != len(k):\n raise IndexError(\n \"Boolean array size {0:d} is used to index array \"\n \"with shape {1:s}.\".format(len(k),\n str(self.shape)))\n if k.ndim > 1:\n raise IndexError(\"{}-dimensional boolean indexing is \"\n \"not supported. \".format(k.ndim))\n if getattr(k, 'dims', (dim, )) != (dim, ):\n raise IndexError(\n \"Boolean indexer should be unlabeled or on the \"\n \"same dimension to the indexed array. Indexer is \"\n \"on {0:s} but the target dimension is \"\n \"{1:s}.\".format(str(k.dims), dim))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _finalize_indexing_result(self, dims, data):\n return type(self)(dims, data, self._attrs, self._encoding,\n fastpath=True)", "response": "Used by IndexVariable to return IndexVariable objects when possible."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nindexing this Variable with - 1 remapped to fill_value.", "response": "def _getitem_with_mask(self, key, fill_value=dtypes.NA):\n \"\"\"Index this Variable with -1 remapped to fill_value.\"\"\"\n # TODO(shoyer): expose this method in public API somewhere (isel?) and\n # use it for reindex.\n # TODO(shoyer): add a sanity check that all other integers are\n # non-negative\n # TODO(shoyer): add an optimization, remapping -1 to an adjacent value\n # that is actually indexed rather than mapping it to the last value\n # along each axis.\n\n if fill_value is dtypes.NA:\n fill_value = dtypes.get_fill_value(self.dtype)\n\n dims, indexer, new_order = self._broadcast_indexes(key)\n\n if self.size:\n if isinstance(self._data, dask_array_type):\n # dask's indexing is faster this way; also vindex does not\n # support negative indices yet:\n # https://github.com/dask/dask/pull/2967\n actual_indexer = indexing.posify_mask_indexer(indexer)\n else:\n actual_indexer = indexer\n\n data = as_indexable(self._data)[actual_indexer]\n chunks_hint = getattr(data, 'chunks', None)\n mask = indexing.create_mask(indexer, self.shape, chunks_hint)\n data = duck_array_ops.where(mask, fill_value, data)\n else:\n # array cannot be indexed along dimensions of size 0, so just\n # build the mask directly instead.\n mask = indexing.create_mask(indexer, self.shape)\n data = np.broadcast_to(fill_value, getattr(mask, 'shape', ()))\n\n if new_order:\n data = np.moveaxis(data, range(len(new_order)), new_order)\n return self._finalize_indexing_result(dims, data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef copy(self, deep=True, data=None):\n if data is None:\n data = self._data\n\n if isinstance(data, indexing.MemoryCachedArray):\n # don't share caching between copies\n data = indexing.MemoryCachedArray(data.array)\n\n if deep:\n if isinstance(data, dask_array_type):\n data = data.copy()\n elif not isinstance(data, PandasIndexAdapter):\n # pandas.Index is immutable\n data = np.array(data)\n else:\n data = as_compatible_data(data)\n if self.shape != data.shape:\n raise ValueError(\"Data shape {} must match shape of object {}\"\n .format(data.shape, self.shape))\n\n # note:\n # dims is already an immutable tuple\n # attributes and encoding will be copied when the new Array is created\n return type(self)(self.dims, data, self._attrs, self._encoding,\n fastpath=True)", "response": "Returns a copy of this object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef chunk(self, chunks=None, name=None, lock=False):\n import dask.array as da\n\n if utils.is_dict_like(chunks):\n chunks = dict((self.get_axis_num(dim), chunk)\n for dim, chunk in chunks.items())\n\n if chunks is None:\n chunks = self.chunks or self.shape\n\n data = self._data\n if isinstance(data, da.Array):\n data = data.rechunk(chunks)\n else:\n if utils.is_dict_like(chunks):\n chunks = tuple(chunks.get(n, s)\n for n, s in enumerate(self.shape))\n # da.from_array works by using lazily indexing with a tuple of\n # slices. Using OuterIndexer is a pragmatic choice: dask does not\n # yet handle different indexing types in an explicit way:\n # https://github.com/dask/dask/issues/2883\n data = indexing.ImplicitToExplicitIndexingAdapter(\n data, indexing.OuterIndexer)\n data = da.from_array(data, chunks, name=name, lock=lock)\n\n return type(self)(self.dims, data, self._attrs, self._encoding,\n fastpath=True)", "response": "Coerce this array s data into a dask array with the given chunks."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new array indexed along the specified dimension.", "response": "def isel(self, indexers=None, drop=False, **indexers_kwargs):\n \"\"\"Return a new array indexed along the specified dimension(s).\n\n Parameters\n ----------\n **indexers : {dim: indexer, ...}\n Keyword arguments with names matching dimensions and values given\n by integers, slice objects or arrays.\n\n Returns\n -------\n obj : Array object\n A new Array with the selected data and dimensions. In general,\n the new variable's data will be a view of this variable's data,\n unless numpy fancy indexing was triggered by using an array\n indexer, in which case the data will be a copy.\n \"\"\"\n indexers = either_dict_or_kwargs(indexers, indexers_kwargs, 'isel')\n\n invalid = [k for k in indexers if k not in self.dims]\n if invalid:\n raise ValueError(\"dimensions %r do not exist\" % invalid)\n\n key = [slice(None)] * self.ndim\n for i, dim in enumerate(self.dims):\n if dim in indexers:\n key[i] = indexers[dim]\n return self[tuple(key)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new object with squeezed data.", "response": "def squeeze(self, dim=None):\n \"\"\"Return a new object with squeezed data.\n\n Parameters\n ----------\n dim : None or str or tuple of str, optional\n Selects a subset of the length one dimensions. If a dimension is\n selected with length greater than one, an error is raised. If\n None, all length one dimensions are squeezed.\n\n Returns\n -------\n squeezed : same type as caller\n This object, but with with all or a subset of the dimensions of\n length 1 removed.\n\n See Also\n --------\n numpy.squeeze\n \"\"\"\n dims = common.get_squeeze_dims(self, dim)\n return self.isel({d: 0 for d in dims})"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a new Variable with shifted data.", "response": "def shift(self, shifts=None, fill_value=dtypes.NA, **shifts_kwargs):\n \"\"\"\n Return a new Variable with shifted data.\n\n Parameters\n ----------\n shifts : mapping of the form {dim: offset}\n Integer offset to shift along each of the given dimensions.\n Positive offsets shift to the right; negative offsets shift to the\n left.\n fill_value: scalar, optional\n Value to use for newly missing values\n **shifts_kwargs:\n The keyword arguments form of ``shifts``.\n One of shifts or shifts_kwarg must be provided.\n\n Returns\n -------\n shifted : Variable\n Variable with the same dimensions and attributes but shifted data.\n \"\"\"\n shifts = either_dict_or_kwargs(shifts, shifts_kwargs, 'shift')\n result = self\n for dim, count in shifts.items():\n result = result._shift_one_dim(dim, count, fill_value=fill_value)\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a new Variable with paddings padded to the edges of each dimension.", "response": "def pad_with_fill_value(self, pad_widths=None, fill_value=dtypes.NA,\n **pad_widths_kwargs):\n \"\"\"\n Return a new Variable with paddings.\n\n Parameters\n ----------\n pad_width: Mapping of the form {dim: (before, after)}\n Number of values padded to the edges of each dimension.\n **pad_widths_kwargs:\n Keyword argument for pad_widths\n \"\"\"\n pad_widths = either_dict_or_kwargs(pad_widths, pad_widths_kwargs,\n 'pad')\n\n if fill_value is dtypes.NA:\n dtype, fill_value = dtypes.maybe_promote(self.dtype)\n else:\n dtype = self.dtype\n\n if isinstance(self.data, dask_array_type):\n array = self.data\n\n # Dask does not yet support pad. We manually implement it.\n # https://github.com/dask/dask/issues/1926\n for d, pad in pad_widths.items():\n axis = self.get_axis_num(d)\n before_shape = list(array.shape)\n before_shape[axis] = pad[0]\n before_chunks = list(array.chunks)\n before_chunks[axis] = (pad[0], )\n after_shape = list(array.shape)\n after_shape[axis] = pad[1]\n after_chunks = list(array.chunks)\n after_chunks[axis] = (pad[1], )\n\n arrays = []\n if pad[0] > 0:\n arrays.append(da.full(before_shape, fill_value,\n dtype=dtype, chunks=before_chunks))\n arrays.append(array)\n if pad[1] > 0:\n arrays.append(da.full(after_shape, fill_value,\n dtype=dtype, chunks=after_chunks))\n if len(arrays) > 1:\n array = da.concatenate(arrays, axis=axis)\n else:\n pads = [(0, 0) if d not in pad_widths else pad_widths[d]\n for d in self.dims]\n array = np.pad(self.data.astype(dtype, copy=False), pads,\n mode='constant', constant_values=fill_value)\n return type(self)(self.dims, array)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef roll(self, shifts=None, **shifts_kwargs):\n shifts = either_dict_or_kwargs(shifts, shifts_kwargs, 'roll')\n\n result = self\n for dim, count in shifts.items():\n result = result._roll_one_dim(dim, count)\n return result", "response": "Return a new Variable with rolld data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef transpose(self, *dims) -> 'Variable':\n if len(dims) == 0:\n dims = self.dims[::-1]\n axes = self.get_axis_num(dims)\n if len(dims) < 2: # no need to transpose if only one dimension\n return self.copy(deep=False)\n\n data = as_indexable(self._data).transpose(axes)\n return type(self)(dims, data, self._attrs, self._encoding,\n fastpath=True)", "response": "Return a new Variable object with transposed dimensions."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a new Variable with given set of dimensions.", "response": "def set_dims(self, dims, shape=None):\n \"\"\"Return a new variable with given set of dimensions.\n This method might be used to attach new dimension(s) to variable.\n\n When possible, this operation does not copy this variable's data.\n\n Parameters\n ----------\n dims : str or sequence of str or dict\n Dimensions to include on the new variable. If a dict, values are\n used to provide the sizes of new dimensions; otherwise, new\n dimensions are inserted with length 1.\n\n Returns\n -------\n Variable\n \"\"\"\n if isinstance(dims, str):\n dims = [dims]\n\n if shape is None and utils.is_dict_like(dims):\n shape = dims.values()\n\n missing_dims = set(self.dims) - set(dims)\n if missing_dims:\n raise ValueError('new dimensions %r must be a superset of '\n 'existing dimensions %r' % (dims, self.dims))\n\n self_dims = set(self.dims)\n expanded_dims = tuple(\n d for d in dims if d not in self_dims) + self.dims\n\n if self.dims == expanded_dims:\n # don't use broadcast_to unless necessary so the result remains\n # writeable if possible\n expanded_data = self.data\n elif shape is not None:\n dims_map = dict(zip(dims, shape))\n tmp_shape = tuple(dims_map[d] for d in expanded_dims)\n expanded_data = duck_array_ops.broadcast_to(self.data, tmp_shape)\n else:\n expanded_data = self.data[\n (None,) * (len(expanded_dims) - self.ndim)]\n\n expanded_var = Variable(expanded_dims, expanded_data, self._attrs,\n self._encoding, fastpath=True)\n return expanded_var.transpose(*dims)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstacks any number of existing dimensions into a single new dimension.", "response": "def stack(self, dimensions=None, **dimensions_kwargs):\n \"\"\"\n Stack any number of existing dimensions into a single new dimension.\n\n New dimensions will be added at the end, and the order of the data\n along each new dimension will be in contiguous (C) order.\n\n Parameters\n ----------\n dimensions : Mapping of form new_name=(dim1, dim2, ...)\n Names of new dimensions, and the existing dimensions that they\n replace.\n **dimensions_kwargs:\n The keyword arguments form of ``dimensions``.\n One of dimensions or dimensions_kwargs must be provided.\n\n Returns\n -------\n stacked : Variable\n Variable with the same attributes but stacked data.\n\n See also\n --------\n Variable.unstack\n \"\"\"\n dimensions = either_dict_or_kwargs(dimensions, dimensions_kwargs,\n 'stack')\n result = self\n for new_dim, dims in dimensions.items():\n result = result._stack_once(dims, new_dim)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreducing this array by applying func along some dimension.", "response": "def reduce(self, func, dim=None, axis=None,\n keep_attrs=None, allow_lazy=False, **kwargs):\n \"\"\"Reduce this array by applying `func` along some dimension(s).\n\n Parameters\n ----------\n func : function\n Function which can be called in the form\n `func(x, axis=axis, **kwargs)` to return the result of reducing an\n np.ndarray over an integer valued axis.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply `func`.\n axis : int or sequence of int, optional\n Axis(es) over which to apply `func`. Only one of the 'dim'\n and 'axis' arguments can be supplied. If neither are supplied, then\n the reduction is calculated over the flattened array (by calling\n `func(x)` without an axis argument).\n keep_attrs : bool, optional\n If True, the variable's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n **kwargs : dict\n Additional keyword arguments passed on to `func`.\n\n Returns\n -------\n reduced : Array\n Array with summarized data and the indicated dimension(s)\n removed.\n \"\"\"\n if dim is common.ALL_DIMS:\n dim = None\n if dim is not None and axis is not None:\n raise ValueError(\"cannot supply both 'axis' and 'dim' arguments\")\n\n if dim is not None:\n axis = self.get_axis_num(dim)\n input_data = self.data if allow_lazy else self.values\n if axis is not None:\n data = func(input_data, axis=axis, **kwargs)\n else:\n data = func(input_data, **kwargs)\n\n if getattr(data, 'shape', ()) == self.shape:\n dims = self.dims\n else:\n removed_axes = (range(self.ndim) if axis is None\n else np.atleast_1d(axis) % self.ndim)\n dims = [adim for n, adim in enumerate(self.dims)\n if n not in removed_axes]\n\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n attrs = self._attrs if keep_attrs else None\n\n return Variable(dims, data, attrs=attrs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef concat(cls, variables, dim='concat_dim', positions=None,\n shortcut=False):\n \"\"\"Concatenate variables along a new or existing dimension.\n\n Parameters\n ----------\n variables : iterable of Array\n Arrays to stack together. Each variable is expected to have\n matching dimensions and shape except for along the stacked\n dimension.\n dim : str or DataArray, optional\n Name of the dimension to stack along. This can either be a new\n dimension name, in which case it is added along axis=0, or an\n existing dimension name, in which case the location of the\n dimension is unchanged. Where to insert the new dimension is\n determined by the first variable.\n positions : None or list of integer arrays, optional\n List of integer arrays which specifies the integer positions to\n which to assign each dataset along the concatenated dimension.\n If not supplied, objects are concatenated in the provided order.\n shortcut : bool, optional\n This option is used internally to speed-up groupby operations.\n If `shortcut` is True, some checks of internal consistency between\n arrays to concatenate are skipped.\n\n Returns\n -------\n stacked : Variable\n Concatenated Variable formed by stacking all the supplied variables\n along the given dimension.\n \"\"\"\n if not isinstance(dim, str):\n dim, = dim.dims\n\n # can't do this lazily: we need to loop through variables at least\n # twice\n variables = list(variables)\n first_var = variables[0]\n\n arrays = [v.data for v in variables]\n\n if dim in first_var.dims:\n axis = first_var.get_axis_num(dim)\n dims = first_var.dims\n data = duck_array_ops.concatenate(arrays, axis=axis)\n if positions is not None:\n # TODO: deprecate this option -- we don't need it for groupby\n # any more.\n indices = nputils.inverse_permutation(\n np.concatenate(positions))\n data = duck_array_ops.take(data, indices, axis=axis)\n else:\n axis = 0\n dims = (dim,) + first_var.dims\n data = duck_array_ops.stack(arrays, axis=axis)\n\n attrs = OrderedDict(first_var.attrs)\n encoding = OrderedDict(first_var.encoding)\n if not shortcut:\n for var in variables:\n if var.dims != first_var.dims:\n raise ValueError('inconsistent dimensions')\n utils.remove_incompatible_items(attrs, var.attrs)\n\n return cls(dims, data, attrs, encoding)", "response": "Concatenate a list of ArrayLogEntry objects along a new or existing dimension."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntrue if two Variables have the same dimensions and values ; otherwise False.", "response": "def equals(self, other, equiv=duck_array_ops.array_equiv):\n \"\"\"True if two Variables have the same dimensions and values;\n otherwise False.\n\n Variables can still be equal (like pandas objects) if they have NaN\n values in the same locations.\n\n This method is necessary because `v1 == v2` for Variables\n does element-wise comparisons (like numpy.ndarrays).\n \"\"\"\n other = getattr(other, 'variable', other)\n try:\n return (self.dims == other.dims and\n (self._data is other._data or\n equiv(self.data, other.data)))\n except (TypeError, AttributeError):\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef broadcast_equals(self, other, equiv=duck_array_ops.array_equiv):\n try:\n self, other = broadcast_variables(self, other)\n except (ValueError, AttributeError):\n return False\n return self.equals(other, equiv=equiv)", "response": "True if two Variables have the values after being broadcast against each other."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlikes equals but also checks attributes.", "response": "def identical(self, other):\n \"\"\"Like equals, but also checks attributes.\n \"\"\"\n try:\n return (utils.dict_equiv(self.attrs, other.attrs) and\n self.equals(other))\n except (TypeError, AttributeError):\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the qth quantile of the data along the specified dimension.", "response": "def quantile(self, q, dim=None, interpolation='linear'):\n \"\"\"Compute the qth quantile of the data along the specified dimension.\n\n Returns the qth quantiles(s) of the array elements.\n\n Parameters\n ----------\n q : float in range of [0,1] (or sequence of floats)\n Quantile to compute, which must be between 0 and 1\n inclusive.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply quantile.\n interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}\n This optional parameter specifies the interpolation method to\n use when the desired quantile lies between two data points\n ``i < j``:\n * linear: ``i + (j - i) * fraction``, where ``fraction`` is\n the fractional part of the index surrounded by ``i`` and\n ``j``.\n * lower: ``i``.\n * higher: ``j``.\n * nearest: ``i`` or ``j``, whichever is nearest.\n * midpoint: ``(i + j) / 2``.\n\n Returns\n -------\n quantiles : Variable\n If `q` is a single quantile, then the result\n is a scalar. If multiple percentiles are given, first axis of\n the result corresponds to the quantile and a quantile dimension\n is added to the return array. The other dimensions are the\n dimensions that remain after the reduction of the array.\n\n See Also\n --------\n numpy.nanpercentile, pandas.Series.quantile, Dataset.quantile,\n DataArray.quantile\n \"\"\"\n if isinstance(self.data, dask_array_type):\n raise TypeError(\"quantile does not work for arrays stored as dask \"\n \"arrays. Load the data via .compute() or .load() \"\n \"prior to calling this method.\")\n\n q = np.asarray(q, dtype=np.float64)\n\n new_dims = list(self.dims)\n if dim is not None:\n axis = self.get_axis_num(dim)\n if utils.is_scalar(dim):\n new_dims.remove(dim)\n else:\n for d in dim:\n new_dims.remove(d)\n else:\n axis = None\n new_dims = []\n\n # only add the quantile dimension if q is array like\n if q.ndim != 0:\n new_dims = ['quantile'] + new_dims\n\n qs = np.nanpercentile(self.data, q * 100., axis=axis,\n interpolation=interpolation)\n return Variable(new_dims, qs)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rank(self, dim, pct=False):\n import bottleneck as bn\n\n if isinstance(self.data, dask_array_type):\n raise TypeError(\"rank does not work for arrays stored as dask \"\n \"arrays. Load the data via .compute() or .load() \"\n \"prior to calling this method.\")\n\n axis = self.get_axis_num(dim)\n func = bn.nanrankdata if self.dtype.kind == 'f' else bn.rankdata\n ranked = func(self.data, axis=axis)\n if pct:\n count = np.sum(~np.isnan(self.data), axis=axis, keepdims=True)\n ranked /= count\n return Variable(self.dims, ranked)", "response": "Ranks the data.\n\n Equal values are assigned a rank that is the average of the ranks that\n would have been otherwise assigned to all of the values within that\n set. Ranks begin at 1, not 0. If `pct`, computes percentage ranks.\n\n NaNs in the input array are returned as NaNs.\n\n The `bottleneck` library is required.\n\n Parameters\n ----------\n dim : str\n Dimension over which to compute rank.\n pct : bool, optional\n If True, compute percentage ranks, otherwise compute integer ranks.\n\n Returns\n -------\n ranked : Variable\n\n See Also\n --------\n Dataset.rank, DataArray.rank"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rolling_window(self, dim, window, window_dim, center=False,\n fill_value=dtypes.NA):\n \"\"\"\n Make a rolling_window along dim and add a new_dim to the last place.\n\n Parameters\n ----------\n dim: str\n Dimension over which to compute rolling_window\n window: int\n Window size of the rolling\n window_dim: str\n New name of the window dimension.\n center: boolean. default False.\n If True, pad fill_value for both ends. Otherwise, pad in the head\n of the axis.\n fill_value:\n value to be filled.\n\n Returns\n -------\n Variable that is a view of the original array with a added dimension of\n size w.\n The return dim: self.dims + (window_dim, )\n The return shape: self.shape + (window, )\n\n Examples\n --------\n >>> v=Variable(('a', 'b'), np.arange(8).reshape((2,4)))\n >>> v.rolling_window(x, 'b', 3, 'window_dim')\n \n array([[[nan, nan, 0], [nan, 0, 1], [0, 1, 2], [1, 2, 3]],\n [[nan, nan, 4], [nan, 4, 5], [4, 5, 6], [5, 6, 7]]])\n\n >>> v.rolling_window(x, 'b', 3, 'window_dim', center=True)\n \n array([[[nan, 0, 1], [0, 1, 2], [1, 2, 3], [2, 3, nan]],\n [[nan, 4, 5], [4, 5, 6], [5, 6, 7], [6, 7, nan]]])\n \"\"\"\n if fill_value is dtypes.NA: # np.nan is passed\n dtype, fill_value = dtypes.maybe_promote(self.dtype)\n array = self.astype(dtype, copy=False).data\n else:\n dtype = self.dtype\n array = self.data\n\n new_dims = self.dims + (window_dim, )\n return Variable(new_dims, duck_array_ops.rolling_window(\n array, axis=self.get_axis_num(dim), window=window,\n center=center, fill_value=fill_value))", "response": "This function creates a rolling_window of the array along dim and adds a new_dim to the last place."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef coarsen(self, windows, func, boundary='exact', side='left'):\n windows = {k: v for k, v in windows.items() if k in self.dims}\n if not windows:\n return self.copy()\n\n reshaped, axes = self._coarsen_reshape(windows, boundary, side)\n if isinstance(func, str):\n name = func\n func = getattr(duck_array_ops, name, None)\n if func is None:\n raise NameError('{} is not a valid method.'.format(name))\n return type(self)(self.dims, func(reshaped, axis=axes), self._attrs)", "response": "Return a new array with coarsened data."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconstructs a reshaped array for corsen .", "response": "def _coarsen_reshape(self, windows, boundary, side):\n \"\"\"\n Construct a reshaped-array for corsen\n \"\"\"\n if not utils.is_dict_like(boundary):\n boundary = {d: boundary for d in windows.keys()}\n\n if not utils.is_dict_like(side):\n side = {d: side for d in windows.keys()}\n\n # remove unrelated dimensions\n boundary = {k: v for k, v in boundary.items() if k in windows}\n side = {k: v for k, v in side.items() if k in windows}\n\n for d, window in windows.items():\n if window <= 0:\n raise ValueError('window must be > 0. Given {}'.format(window))\n\n variable = self\n for d, window in windows.items():\n # trim or pad the object\n size = variable.shape[self._get_axis_num(d)]\n n = int(size / window)\n if boundary[d] == 'exact':\n if n * window != size:\n raise ValueError(\n 'Could not coarsen a dimension of size {} with '\n 'window {}'.format(size, window))\n elif boundary[d] == 'trim':\n if side[d] == 'left':\n variable = variable.isel({d: slice(0, window * n)})\n else:\n excess = size - window * n\n variable = variable.isel({d: slice(excess, None)})\n elif boundary[d] == 'pad': # pad\n pad = window * n - size\n if pad < 0:\n pad += window\n if side[d] == 'left':\n pad_widths = {d: (0, pad)}\n else:\n pad_widths = {d: (pad, 0)}\n variable = variable.pad_with_fill_value(pad_widths)\n else:\n raise TypeError(\n \"{} is invalid for boundary. Valid option is 'exact', \"\n \"'trim' and 'pad'\".format(boundary[d]))\n\n shape = []\n axes = []\n axis_count = 0\n for i, d in enumerate(variable.dims):\n if d in windows:\n size = variable.shape[i]\n shape.append(int(size / windows[d]))\n shape.append(windows[d])\n axis_count += 1\n axes.append(i + axis_count)\n else:\n shape.append(variable.shape[i])\n\n return variable.data.reshape(shape), tuple(axes)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _to_numeric(self, offset=None, datetime_unit=None, dtype=float):\n numeric_array = duck_array_ops.datetime_to_numeric(\n self.data, offset, datetime_unit, dtype)\n return type(self)(self.dims, numeric_array, self._attrs)", "response": "Convert the object to numeric."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nspecialize version of Variable. concat for IndexVariable objects.", "response": "def concat(cls, variables, dim='concat_dim', positions=None,\n shortcut=False):\n \"\"\"Specialized version of Variable.concat for IndexVariable objects.\n\n This exists because we want to avoid converting Index objects to NumPy\n arrays, if possible.\n \"\"\"\n if not isinstance(dim, str):\n dim, = dim.dims\n\n variables = list(variables)\n first_var = variables[0]\n\n if any(not isinstance(v, cls) for v in variables):\n raise TypeError('IndexVariable.concat requires that all input '\n 'variables be IndexVariable objects')\n\n indexes = [v._data.array for v in variables]\n\n if not indexes:\n data = []\n else:\n data = indexes[0].append(indexes[1:])\n\n if positions is not None:\n indices = nputils.inverse_permutation(\n np.concatenate(positions))\n data = data.take(indices)\n\n attrs = OrderedDict(first_var.attrs)\n if not shortcut:\n for var in variables:\n if var.dims != first_var.dims:\n raise ValueError('inconsistent dimensions')\n utils.remove_incompatible_items(attrs, var.attrs)\n\n return cls(first_var.dims, data, attrs)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a copy of this VariableReservedEntry object.", "response": "def copy(self, deep=True, data=None):\n \"\"\"Returns a copy of this object.\n\n `deep` is ignored since data is stored in the form of\n pandas.Index, which is already immutable. Dimensions, attributes\n and encodings are always copied.\n\n Use `data` to create a new object with the same structure as\n original but entirely new data.\n\n Parameters\n ----------\n deep : bool, optional\n Deep is always ignored.\n data : array_like, optional\n Data to use in the new object. Must have same shape as original.\n\n Returns\n -------\n object : Variable\n New object with dimensions, attributes, encodings, and optionally\n data copied from original.\n \"\"\"\n if data is None:\n data = self._data\n else:\n data = as_compatible_data(data)\n if self.shape != data.shape:\n raise ValueError(\"Data shape {} must match shape of object {}\"\n .format(data.shape, self.shape))\n return type(self)(self.dims, data, self._attrs,\n self._encoding, fastpath=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts this variable to a pandas. Index.", "response": "def to_index(self):\n \"\"\"Convert this variable to a pandas.Index\"\"\"\n # n.b. creating a new pandas.Index from an old pandas.Index is\n # basically free as pandas.Index objects are immutable\n assert self.ndim == 1\n index = self._data.array\n if isinstance(index, pd.MultiIndex):\n # set default names for multi-index unnamed levels so that\n # we can safely rename dimension / coordinate later\n valid_level_names = [name or '{}_level_{}'.format(self.dims[0], i)\n for i, name in enumerate(index.names)]\n index = index.set_names(valid_level_names)\n else:\n index = index.set_names(self.name)\n return index"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef level_names(self):\n index = self.to_index()\n if isinstance(index, pd.MultiIndex):\n return index.names\n else:\n return None", "response": "Return the level names of this IndexVariable or None if this IndexVariable has no\n MultiIndex."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_level_variable(self, level):\n if self.level_names is None:\n raise ValueError(\"IndexVariable %r has no MultiIndex\" % self.name)\n index = self.to_index()\n return type(self)(self.dims, index.get_level_values(level))", "response": "Return a new IndexVariable from a given MultiIndex level."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parsed_string_to_bounds(date_type, resolution, parsed):\n if resolution == 'year':\n return (date_type(parsed.year, 1, 1),\n date_type(parsed.year + 1, 1, 1) - timedelta(microseconds=1))\n elif resolution == 'month':\n if parsed.month == 12:\n end = date_type(parsed.year + 1, 1, 1) - timedelta(microseconds=1)\n else:\n end = (date_type(parsed.year, parsed.month + 1, 1) -\n timedelta(microseconds=1))\n return date_type(parsed.year, parsed.month, 1), end\n elif resolution == 'day':\n start = date_type(parsed.year, parsed.month, parsed.day)\n return start, start + timedelta(days=1, microseconds=-1)\n elif resolution == 'hour':\n start = date_type(parsed.year, parsed.month, parsed.day, parsed.hour)\n return start, start + timedelta(hours=1, microseconds=-1)\n elif resolution == 'minute':\n start = date_type(parsed.year, parsed.month, parsed.day, parsed.hour,\n parsed.minute)\n return start, start + timedelta(minutes=1, microseconds=-1)\n elif resolution == 'second':\n start = date_type(parsed.year, parsed.month, parsed.day, parsed.hour,\n parsed.minute, parsed.second)\n return start, start + timedelta(seconds=1, microseconds=-1)\n else:\n raise KeyError", "response": "Generalization of pandas. tseries. index. DatetimeIndex. _parsed_string_to_bounds\n for use with non - standard calendars and cftime. datetime\n objects."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_date_field(datetimes, field):\n return np.array([getattr(date, field) for date in datetimes])", "response": "Adapted from pandas. tslib. get_date_field"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a property that can be used to access the given field.", "response": "def _field_accessor(name, docstring=None, min_cftime_version='0.0'):\n \"\"\"Adapted from pandas.tseries.index._field_accessor\"\"\"\n\n def f(self, min_cftime_version=min_cftime_version):\n import cftime\n\n version = cftime.__version__\n\n if LooseVersion(version) >= LooseVersion(min_cftime_version):\n return get_date_field(self._data, name)\n else:\n raise ImportError('The {!r} accessor requires a minimum '\n 'version of cftime of {}. Found an '\n 'installed version of {}.'.format(\n name, min_cftime_version, version))\n\n f.__name__ = name\n f.__doc__ = docstring\n return property(f)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a numpy array from an array of strings.", "response": "def _parse_array_of_cftime_strings(strings, date_type):\n \"\"\"Create a numpy array from an array of strings.\n\n For use in generating dates from strings for use with interp. Assumes the\n array is either 0-dimensional or 1-dimensional.\n\n Parameters\n ----------\n strings : array of strings\n Strings to convert to dates\n date_type : cftime.datetime type\n Calendar type to use for dates\n\n Returns\n -------\n np.array\n \"\"\"\n return np.array([_parse_iso8601_without_reso(date_type, s)\n for s in strings.ravel()]).reshape(strings.shape)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _partial_date_slice(self, resolution, parsed):\n start, end = _parsed_string_to_bounds(self.date_type, resolution,\n parsed)\n\n times = self._data\n\n if self.is_monotonic:\n if (len(times) and ((start < times[0] and end < times[0]) or\n (start > times[-1] and end > times[-1]))):\n # we are out of range\n raise KeyError\n\n # a monotonic (sorted) series can be sliced\n left = times.searchsorted(start, side='left')\n right = times.searchsorted(end, side='right')\n return slice(left, right)\n\n lhs_mask = times >= start\n rhs_mask = times <= end\n return np.flatnonzero(lhs_mask & rhs_mask)", "response": "Adapted from pandas. tseries. index. DatetimeIndex. _partial_date_slice\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadapts from pandas. tseries. index. DatetimeIndex. _get_string_slice", "response": "def _get_string_slice(self, key):\n \"\"\"Adapted from pandas.tseries.index.DatetimeIndex._get_string_slice\"\"\"\n parsed, resolution = _parse_iso8601_with_reso(self.date_type, key)\n try:\n loc = self._partial_date_slice(resolution, parsed)\n except KeyError:\n raise KeyError(key)\n return loc"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_loc(self, key, method=None, tolerance=None):\n if isinstance(key, str):\n return self._get_string_slice(key)\n else:\n return pd.Index.get_loc(self, key, method=method,\n tolerance=tolerance)", "response": "Adapted from pandas. tseries. index. DatetimeIndex. get_loc"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadapt from pandas. tseries. index. DatetimeIndex. _maybe_cast_slice_bound", "response": "def _maybe_cast_slice_bound(self, label, side, kind):\n \"\"\"Adapted from\n pandas.tseries.index.DatetimeIndex._maybe_cast_slice_bound\"\"\"\n if isinstance(label, str):\n parsed, resolution = _parse_iso8601_with_reso(self.date_type,\n label)\n start, end = _parsed_string_to_bounds(self.date_type, resolution,\n parsed)\n if self.is_monotonic_decreasing and len(self) > 1:\n return end if side == 'left' else start\n return start if side == 'left' else end\n else:\n return label"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_value(self, series, key):\n if np.asarray(key).dtype == np.dtype(bool):\n return series.iloc[key]\n elif isinstance(key, slice):\n return series.iloc[self.slice_indexer(\n key.start, key.stop, key.step)]\n else:\n return series.iloc[self.get_loc(key)]", "response": "Adapted from pandas. tseries. index. DatetimeIndex. get_value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nshift the CFTimeIndex a multiple of the given frequency.", "response": "def shift(self, n, freq):\n \"\"\"Shift the CFTimeIndex a multiple of the given frequency.\n\n See the documentation for :py:func:`~xarray.cftime_range` for a\n complete listing of valid frequency strings.\n\n Parameters\n ----------\n n : int\n Periods to shift by\n freq : str or datetime.timedelta\n A frequency string or datetime.timedelta object to shift by\n\n Returns\n -------\n CFTimeIndex\n\n See also\n --------\n pandas.DatetimeIndex.shift\n\n Examples\n --------\n >>> index = xr.cftime_range('2000', periods=1, freq='M')\n >>> index\n CFTimeIndex([2000-01-31 00:00:00], dtype='object')\n >>> index.shift(1, 'M')\n CFTimeIndex([2000-02-29 00:00:00], dtype='object')\n \"\"\"\n from .cftime_offsets import to_offset\n\n if not isinstance(n, int):\n raise TypeError(\"'n' must be an int, got {}.\".format(n))\n if isinstance(freq, timedelta):\n return self + n * freq\n elif isinstance(freq, str):\n return self + n * to_offset(freq)\n else:\n raise TypeError(\n \"'freq' must be of type \"\n \"str or datetime.timedelta, got {}.\".format(freq))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_datetimeindex(self, unsafe=False):\n nptimes = cftime_to_nptime(self)\n calendar = infer_calendar_name(self)\n if calendar not in _STANDARD_CALENDARS and not unsafe:\n warnings.warn(\n 'Converting a CFTimeIndex with dates from a non-standard '\n 'calendar, {!r}, to a pandas.DatetimeIndex, which uses dates '\n 'from the standard calendar. This may lead to subtle errors '\n 'in operations that depend on the length of time between '\n 'dates.'.format(calendar), RuntimeWarning, stacklevel=2)\n return pd.DatetimeIndex(nptimes)", "response": "Convert this index to a pandas. DatetimeIndex."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a function that dispatches to dask for dask array inputs.", "response": "def _dask_or_eager_func(name, eager_module=np, dask_module=dask_array,\n list_of_args=False, array_args=slice(1),\n requires_dask=None):\n \"\"\"Create a function that dispatches to dask for dask array inputs.\"\"\"\n if dask_module is not None:\n def f(*args, **kwargs):\n if list_of_args:\n dispatch_args = args[0]\n else:\n dispatch_args = args[array_args]\n if any(isinstance(a, dask_array.Array) for a in dispatch_args):\n try:\n wrapped = getattr(dask_module, name)\n except AttributeError as e:\n raise AttributeError(\"%s: requires dask >=%s\" %\n (e, requires_dask))\n else:\n wrapped = getattr(eager_module, name)\n return wrapped(*args, **kwargs)\n else:\n def f(*args, **kwargs):\n return getattr(eager_module, name)(*args, **kwargs)\n return f"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef as_shared_dtype(scalars_or_arrays):\n arrays = [asarray(x) for x in scalars_or_arrays]\n # Pass arrays directly instead of dtypes to result_type so scalars\n # get handled properly.\n # Note that result_type() safely gets the dtype from dask arrays without\n # evaluating them.\n out_type = dtypes.result_type(*arrays)\n return [x.astype(out_type, copy=False) for x in arrays]", "response": "Cast a list of scalars to a shared dtype using xarray s type promotion rules."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef allclose_or_equiv(arr1, arr2, rtol=1e-5, atol=1e-8):\n arr1, arr2 = as_like_arrays(arr1, arr2)\n if arr1.shape != arr2.shape:\n return False\n return bool(\n isclose(arr1, arr2, rtol=rtol, atol=atol, equal_nan=True).all())", "response": "Like np. allclose but also allows values to be NaN in both arrays\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef array_equiv(arr1, arr2):\n arr1, arr2 = as_like_arrays(arr1, arr2)\n if arr1.shape != arr2.shape:\n return False\n\n with warnings.catch_warnings():\n warnings.filterwarnings('ignore', \"In the future, 'NAT == x'\")\n\n flag_array = (arr1 == arr2)\n flag_array |= (isnull(arr1) & isnull(arr2))\n\n return bool(flag_array.all())", "response": "Like np. array_equal but also allows values to be NaN in both arrays\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncounting the number of non - NA in this array along the given axis or axes.", "response": "def count(data, axis=None):\n \"\"\"Count the number of non-NA in this array along the given axis or axes\n \"\"\"\n return np.sum(np.logical_not(isnull(data)), axis=axis)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef datetime_to_numeric(array, offset=None, datetime_unit=None, dtype=float):\n # TODO: make this function dask-compatible?\n if offset is None:\n offset = array.min()\n array = array - offset\n\n if not hasattr(array, 'dtype'): # scalar is converted to 0d-array\n array = np.array(array)\n\n if array.dtype.kind in 'O':\n # possibly convert object array containing datetime.timedelta\n array = np.asarray(pd.Series(array.ravel())).reshape(array.shape)\n\n if datetime_unit:\n array = array / np.timedelta64(1, datetime_unit)\n\n # convert np.NaT to np.nan\n if array.dtype.kind in 'mM':\n return np.where(isnull(array), np.nan, array.astype(dtype))\n return array.astype(dtype)", "response": "Convert an array containing datetime - like data to an array of floats."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the first non - NA elements in this array along the given axis", "response": "def first(values, axis, skipna=None):\n \"\"\"Return the first non-NA elements in this array along the given axis\n \"\"\"\n if (skipna or skipna is None) and values.dtype.kind not in 'iSU':\n # only bother for dtypes that can hold NaN\n _fail_on_dask_array_input_skipna(values)\n return nanfirst(values, axis)\n return take(values, 0, axis=axis)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef last(values, axis, skipna=None):\n if (skipna or skipna is None) and values.dtype.kind not in 'iSU':\n # only bother for dtypes that can hold NaN\n _fail_on_dask_array_input_skipna(values)\n return nanlast(values, axis)\n return take(values, -1, axis=axis)", "response": "Return the last non - NA elements in this array along the given axis"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rolling_window(array, axis, window, center, fill_value):\n if isinstance(array, dask_array_type):\n return dask_array_ops.rolling_window(\n array, axis, window, center, fill_value)\n else: # np.ndarray\n return nputils.rolling_window(\n array, axis, window, center, fill_value)", "response": "Make an ndarray with a rolling window of axis - th dimension."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef concat(objs, dim=None, data_vars='all', coords='different',\n compat='equals', positions=None, indexers=None, mode=None,\n concat_over=None):\n \"\"\"Concatenate xarray objects along a new or existing dimension.\n\n Parameters\n ----------\n objs : sequence of Dataset and DataArray objects\n xarray objects to concatenate together. Each object is expected to\n consist of variables and coordinates with matching shapes except for\n along the concatenated dimension.\n dim : str or DataArray or pandas.Index\n Name of the dimension to concatenate along. This can either be a new\n dimension name, in which case it is added along axis=0, or an existing\n dimension name, in which case the location of the dimension is\n unchanged. If dimension is provided as a DataArray or Index, its name\n is used as the dimension to concatenate along and the values are added\n as a coordinate.\n data_vars : {'minimal', 'different', 'all' or list of str}, optional\n These data variables will be concatenated together:\n * 'minimal': Only data variables in which the dimension already\n appears are included.\n * 'different': Data variables which are not equal (ignoring\n attributes) across all datasets are also concatenated (as well as\n all for which dimension already appears). Beware: this option may\n load the data payload of data variables into memory if they are not\n already loaded.\n * 'all': All data variables will be concatenated.\n * list of str: The listed data variables will be concatenated, in\n addition to the 'minimal' data variables.\n If objects are DataArrays, data_vars must be 'all'.\n coords : {'minimal', 'different', 'all' or list of str}, optional\n These coordinate variables will be concatenated together:\n * 'minimal': Only coordinates in which the dimension already appears\n are included.\n * 'different': Coordinates which are not equal (ignoring attributes)\n across all datasets are also concatenated (as well as all for which\n dimension already appears). Beware: this option may load the data\n payload of coordinate variables into memory if they are not already\n loaded.\n * 'all': All coordinate variables will be concatenated, except\n those corresponding to other dimensions.\n * list of str: The listed coordinate variables will be concatenated,\n in addition the 'minimal' coordinates.\n compat : {'equals', 'identical'}, optional\n String indicating how to compare non-concatenated variables and\n dataset global attributes for potential conflicts. 'equals' means\n that all variable values and dimensions must be the same;\n 'identical' means that variable attributes and global attributes\n must also be equal.\n positions : None or list of integer arrays, optional\n List of integer arrays which specifies the integer positions to which\n to assign each dataset along the concatenated dimension. If not\n supplied, objects are concatenated in the provided order.\n indexers, mode, concat_over : deprecated\n\n Returns\n -------\n concatenated : type of objs\n\n See also\n --------\n merge\n auto_combine\n \"\"\"\n # TODO: add join and ignore_index arguments copied from pandas.concat\n # TODO: support concatenating scalar coordinates even if the concatenated\n # dimension already exists\n from .dataset import Dataset\n from .dataarray import DataArray\n\n try:\n first_obj, objs = utils.peek_at(objs)\n except StopIteration:\n raise ValueError('must supply at least one object to concatenate')\n\n if dim is None:\n warnings.warn('the `dim` argument to `concat` will be required '\n 'in a future version of xarray; for now, setting it to '\n \"the old default of 'concat_dim'\",\n FutureWarning, stacklevel=2)\n dim = 'concat_dims'\n\n if indexers is not None: # pragma: nocover\n warnings.warn('indexers has been renamed to positions; the alias '\n 'will be removed in a future version of xarray',\n FutureWarning, stacklevel=2)\n positions = indexers\n\n if mode is not None:\n raise ValueError('`mode` is no longer a valid argument to '\n 'xarray.concat; it has been split into the '\n '`data_vars` and `coords` arguments')\n if concat_over is not None:\n raise ValueError('`concat_over` is no longer a valid argument to '\n 'xarray.concat; it has been split into the '\n '`data_vars` and `coords` arguments')\n\n if isinstance(first_obj, DataArray):\n f = _dataarray_concat\n elif isinstance(first_obj, Dataset):\n f = _dataset_concat\n else:\n raise TypeError('can only concatenate xarray Dataset and DataArray '\n 'objects, got %s' % type(first_obj))\n return f(objs, dim, data_vars, coords, compat, positions)", "response": "Concatenate two or more xarray objects along a new or existing dimension."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the name and 1d coordinate variable for concatenating along the new dimension.", "response": "def _calc_concat_dim_coord(dim):\n \"\"\"\n Infer the dimension name and 1d coordinate variable (if appropriate)\n for concatenating along the new dimension.\n \"\"\"\n from .dataarray import DataArray\n\n if isinstance(dim, str):\n coord = None\n elif not isinstance(dim, (DataArray, Variable)):\n dim_name = getattr(dim, 'name', None)\n if dim_name is None:\n dim_name = 'concat_dim'\n coord = IndexVariable(dim_name, dim)\n dim = dim_name\n elif not isinstance(dim, DataArray):\n coord = as_variable(dim).to_index_variable()\n dim, = coord.dims\n else:\n coord = dim\n dim, = coord.dims\n return dim, coord"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _calc_concat_over(datasets, dim, data_vars, coords):\n # Return values\n concat_over = set()\n equals = {}\n\n if dim in datasets[0]:\n concat_over.add(dim)\n for ds in datasets:\n concat_over.update(k for k, v in ds.variables.items()\n if dim in v.dims)\n\n def process_subset_opt(opt, subset):\n if isinstance(opt, str):\n if opt == 'different':\n # all nonindexes that are not the same in each dataset\n for k in getattr(datasets[0], subset):\n if k not in concat_over:\n # Compare the variable of all datasets vs. the one\n # of the first dataset. Perform the minimum amount of\n # loads in order to avoid multiple loads from disk\n # while keeping the RAM footprint low.\n v_lhs = datasets[0].variables[k].load()\n # We'll need to know later on if variables are equal.\n computed = []\n for ds_rhs in datasets[1:]:\n v_rhs = ds_rhs.variables[k].compute()\n computed.append(v_rhs)\n if not v_lhs.equals(v_rhs):\n concat_over.add(k)\n equals[k] = False\n # computed variables are not to be re-computed\n # again in the future\n for ds, v in zip(datasets[1:], computed):\n ds.variables[k].data = v.data\n break\n else:\n equals[k] = True\n\n elif opt == 'all':\n concat_over.update(set(getattr(datasets[0], subset)) -\n set(datasets[0].dims))\n elif opt == 'minimal':\n pass\n else:\n raise ValueError(\"unexpected value for %s: %s\" % (subset, opt))\n else:\n invalid_vars = [k for k in opt\n if k not in getattr(datasets[0], subset)]\n if invalid_vars:\n if subset == 'coords':\n raise ValueError(\n 'some variables in coords are not coordinates on '\n 'the first dataset: %s' % (invalid_vars,))\n else:\n raise ValueError(\n 'some variables in data_vars are not data variables '\n 'on the first dataset: %s' % (invalid_vars,))\n concat_over.update(opt)\n\n process_subset_opt(data_vars, 'data_vars')\n process_subset_opt(coords, 'coords')\n return concat_over, equals", "response": "Calculates which dataset variables need to be concatenated in the result."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _dataset_concat(datasets, dim, data_vars, coords, compat, positions):\n from .dataset import Dataset\n\n if compat not in ['equals', 'identical']:\n raise ValueError(\"compat=%r invalid: must be 'equals' \"\n \"or 'identical'\" % compat)\n\n dim, coord = _calc_concat_dim_coord(dim)\n # Make sure we're working on a copy (we'll be loading variables)\n datasets = [ds.copy() for ds in datasets]\n datasets = align(*datasets, join='outer', copy=False, exclude=[dim])\n\n concat_over, equals = _calc_concat_over(datasets, dim, data_vars, coords)\n\n def insert_result_variable(k, v):\n assert isinstance(v, Variable)\n if k in datasets[0].coords:\n result_coord_names.add(k)\n result_vars[k] = v\n\n # create the new dataset and add constant variables\n result_vars = OrderedDict()\n result_coord_names = set(datasets[0].coords)\n result_attrs = datasets[0].attrs\n result_encoding = datasets[0].encoding\n\n for k, v in datasets[0].variables.items():\n if k not in concat_over:\n insert_result_variable(k, v)\n\n # check that global attributes and non-concatenated variables are fixed\n # across all datasets\n for ds in datasets[1:]:\n if (compat == 'identical' and\n not utils.dict_equiv(ds.attrs, result_attrs)):\n raise ValueError('dataset global attributes not equal')\n for k, v in ds.variables.items():\n if k not in result_vars and k not in concat_over:\n raise ValueError('encountered unexpected variable %r' % k)\n elif (k in result_coord_names) != (k in ds.coords):\n raise ValueError('%r is a coordinate in some datasets but not '\n 'others' % k)\n elif k in result_vars and k != dim:\n # Don't use Variable.identical as it internally invokes\n # Variable.equals, and we may already know the answer\n if compat == 'identical' and not utils.dict_equiv(\n v.attrs, result_vars[k].attrs):\n raise ValueError(\n 'variable %s not identical across datasets' % k)\n\n # Proceed with equals()\n try:\n # May be populated when using the \"different\" method\n is_equal = equals[k]\n except KeyError:\n result_vars[k].load()\n is_equal = v.equals(result_vars[k])\n if not is_equal:\n raise ValueError(\n 'variable %s not equal across datasets' % k)\n\n # we've already verified everything is consistent; now, calculate\n # shared dimension sizes so we can expand the necessary variables\n dim_lengths = [ds.dims.get(dim, 1) for ds in datasets]\n non_concat_dims = {}\n for ds in datasets:\n non_concat_dims.update(ds.dims)\n non_concat_dims.pop(dim, None)\n\n def ensure_common_dims(vars):\n # ensure each variable with the given name shares the same\n # dimensions and the same shape for all of them except along the\n # concat dimension\n common_dims = tuple(pd.unique([d for v in vars for d in v.dims]))\n if dim not in common_dims:\n common_dims = (dim,) + common_dims\n for var, dim_len in zip(vars, dim_lengths):\n if var.dims != common_dims:\n common_shape = tuple(non_concat_dims.get(d, dim_len)\n for d in common_dims)\n var = var.set_dims(common_dims, common_shape)\n yield var\n\n # stack up each variable to fill-out the dataset (in order)\n for k in datasets[0].variables:\n if k in concat_over:\n vars = ensure_common_dims([ds.variables[k] for ds in datasets])\n combined = concat_vars(vars, dim, positions)\n insert_result_variable(k, combined)\n\n result = Dataset(result_vars, attrs=result_attrs)\n result = result.set_coords(result_coord_names)\n result.encoding = result_encoding\n\n if coord is not None:\n # add concat dimension last to ensure that its in the final Dataset\n result[coord.name] = coord\n\n return result", "response": "Concatenate a sequence of datasets along a new or existing dimension."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _infer_tile_ids_from_nested_list(entry, current_pos):\n\n if isinstance(entry, list):\n for i, item in enumerate(entry):\n for result in _infer_tile_ids_from_nested_list(item,\n current_pos + (i,)):\n yield result\n else:\n yield current_pos, entry", "response": "A generator function that yields tuples containing the tile IDs of each object in the nested list."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _combine_nd(combined_ids, concat_dims, data_vars='all',\n coords='different', compat='no_conflicts'):\n \"\"\"\n Concatenates and merges an N-dimensional structure of datasets.\n\n No checks are performed on the consistency of the datasets, concat_dims or\n tile_IDs, because it is assumed that this has already been done.\n\n Parameters\n ----------\n combined_ids : Dict[Tuple[int, ...]], xarray.Dataset]\n Structure containing all datasets to be concatenated with \"tile_IDs\" as\n keys, which specify position within the desired final combined result.\n concat_dims : sequence of str\n The dimensions along which the datasets should be concatenated. Must be\n in order, and the length must match\n\n Returns\n -------\n combined_ds : xarray.Dataset\n \"\"\"\n\n # Perform N-D dimensional concatenation\n # Each iteration of this loop reduces the length of the tile_ids tuples\n # by one. It always combines along the first dimension, removing the first\n # element of the tuple\n for concat_dim in concat_dims:\n combined_ids = _auto_combine_all_along_first_dim(combined_ids,\n dim=concat_dim,\n data_vars=data_vars,\n coords=coords,\n compat=compat)\n combined_ds = list(combined_ids.values())[0]\n return combined_ds", "response": "Concatenates a set of datasets into a single N - dimensional structure."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _auto_combine(datasets, concat_dims, compat, data_vars, coords,\n infer_order_from_coords, ids):\n \"\"\"\n Calls logic to decide concatenation order before concatenating.\n \"\"\"\n\n # Arrange datasets for concatenation\n if infer_order_from_coords:\n raise NotImplementedError\n # TODO Use coordinates to determine tile_ID for each dataset in N-D\n # Ignore how they were ordered previously\n # Should look like:\n # combined_ids, concat_dims = _infer_tile_ids_from_coords(datasets,\n # concat_dims)\n else:\n # Use information from the shape of the user input\n if not ids:\n # Determine tile_IDs by structure of input in N-D\n # (i.e. ordering in list-of-lists)\n combined_ids, concat_dims = _infer_concat_order_from_positions(\n datasets, concat_dims)\n else:\n # Already sorted so just use the ids already passed\n combined_ids = OrderedDict(zip(ids, datasets))\n\n # Check that the inferred shape is combinable\n _check_shape_tile_ids(combined_ids)\n\n # Repeatedly concatenate then merge along each dimension\n combined = _combine_nd(combined_ids, concat_dims, compat=compat,\n data_vars=data_vars, coords=coords)\n return combined", "response": "Combines the datasets into a single N - D tree."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef auto_combine(datasets, concat_dim=_CONCAT_DIM_DEFAULT,\n compat='no_conflicts', data_vars='all', coords='different'):\n \"\"\"Attempt to auto-magically combine the given datasets into one.\n This method attempts to combine a list of datasets into a single entity by\n inspecting metadata and using a combination of concat and merge.\n It does not concatenate along more than one dimension or sort data under\n any circumstances. It does align coordinates, but different variables on\n datasets can cause it to fail under some scenarios. In complex cases, you\n may need to clean up your data and use ``concat``/``merge`` explicitly.\n ``auto_combine`` works well if you have N years of data and M data\n variables, and each combination of a distinct time period and set of data\n variables is saved its own dataset.\n\n Parameters\n ----------\n datasets : sequence of xarray.Dataset\n Dataset objects to merge.\n concat_dim : str or DataArray or Index, optional\n Dimension along which to concatenate variables, as used by\n :py:func:`xarray.concat`. You only need to provide this argument if\n the dimension along which you want to concatenate is not a dimension\n in the original datasets, e.g., if you want to stack a collection of\n 2D arrays along a third dimension.\n By default, xarray attempts to infer this argument by examining\n component files. Set ``concat_dim=None`` explicitly to disable\n concatenation.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n String indicating how to compare variables of the same name for\n potential conflicts:\n\n - 'broadcast_equals': all values must be equal when variables are\n broadcast against each other to ensure common dimensions.\n - 'equals': all values and dimensions must be the same.\n - 'identical': all values, dimensions and attributes must be the\n same.\n - 'no_conflicts': only values which are not null in both datasets\n must be equal. The returned dataset then contains the combination\n of all non-null values.\n data_vars : {'minimal', 'different', 'all' or list of str}, optional\n Details are in the documentation of concat\n coords : {'minimal', 'different', 'all' or list of str}, optional\n Details are in the documentation of conca\n\n Returns\n -------\n combined : xarray.Dataset\n\n See also\n --------\n concat\n Dataset.merge\n \"\"\" # noqa\n\n # Coerce 1D input into ND to maintain backwards-compatible API until API\n # for N-D combine decided\n # (see https://github.com/pydata/xarray/pull/2553/#issuecomment-445892746)\n if concat_dim is None or concat_dim == _CONCAT_DIM_DEFAULT:\n concat_dims = concat_dim\n elif not isinstance(concat_dim, list):\n concat_dims = [concat_dim]\n else:\n concat_dims = concat_dim\n infer_order_from_coords = False\n\n # The IDs argument tells _auto_combine that the datasets are not yet sorted\n return _auto_combine(datasets, concat_dims=concat_dims, compat=compat,\n data_vars=data_vars, coords=coords,\n infer_order_from_coords=infer_order_from_coords,\n ids=False)", "response": "This method attempts to auto - magically combine the given datasets into one."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_date_type(calendar):\n try:\n import cftime\n except ImportError:\n raise ImportError(\n 'cftime is required for dates with non-standard calendars')\n else:\n calendars = {\n 'noleap': cftime.DatetimeNoLeap,\n '360_day': cftime.Datetime360Day,\n '365_day': cftime.DatetimeNoLeap,\n '366_day': cftime.DatetimeAllLeap,\n 'gregorian': cftime.DatetimeGregorian,\n 'proleptic_gregorian': cftime.DatetimeProlepticGregorian,\n 'julian': cftime.DatetimeJulian,\n 'all_leap': cftime.DatetimeAllLeap,\n 'standard': cftime.DatetimeGregorian\n }\n return calendars[calendar]", "response": "Return the cftime date type for a given calendar name."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the day in the month that satisfies a BaseCFTimeOffset s onOffset policy.", "response": "def _get_day_of_month(other, day_option):\n \"\"\"Find the day in `other`'s month that satisfies a BaseCFTimeOffset's\n onOffset policy, as described by the `day_option` argument.\n\n Parameters\n ----------\n other : cftime.datetime\n day_option : 'start', 'end'\n 'start': returns 1\n 'end': returns last day of the month\n\n Returns\n -------\n day_of_month : int\n\n \"\"\"\n\n if day_option == 'start':\n return 1\n elif day_option == 'end':\n days_in_month = _days_in_month(other)\n return days_in_month\n elif day_option is None:\n # Note: unlike `_shift_month`, _get_day_of_month does not\n # allow day_option = None\n raise NotImplementedError\n else:\n raise ValueError(day_option)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _days_in_month(date):\n if date.month == 12:\n reference = type(date)(date.year + 1, 1, 1)\n else:\n reference = type(date)(date.year, date.month + 1, 1)\n return (reference - timedelta(days=1)).day", "response": "The number of days in the month of the given date"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _adjust_n_months(other_day, n, reference_day):\n if n > 0 and other_day < reference_day:\n n = n - 1\n elif n <= 0 and other_day > reference_day:\n n = n + 1\n return n", "response": "Adjust the number of times a monthly offset is applied based\n on the day of a given date."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadjusts the number of times an annual offset is applied based on another date and the reference day provided", "response": "def _adjust_n_years(other, n, month, reference_day):\n \"\"\"Adjust the number of times an annual offset is applied based on\n another date, and the reference day provided\"\"\"\n if n > 0:\n if other.month < month or (other.month == month and\n other.day < reference_day):\n n -= 1\n else:\n if other.month > month or (other.month == month and\n other.day > reference_day):\n n += 1\n return n"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _shift_month(date, months, day_option='start'):\n delta_year = (date.month + months) // 12\n month = (date.month + months) % 12\n\n if month == 0:\n month = 12\n delta_year = delta_year - 1\n year = date.year + delta_year\n\n if day_option == 'start':\n day = 1\n elif day_option == 'end':\n reference = type(date)(year, month, 1)\n day = _days_in_month(reference)\n else:\n raise ValueError(day_option)\n # dayofwk=-1 is required to update the dayofwk and dayofyr attributes of\n # the returned date object in versions of cftime between 1.0.2 and\n # 1.0.3.4. It can be removed for versions of cftime greater than\n # 1.0.3.4.\n return date.replace(year=year, month=month, day=day, dayofwk=-1)", "response": "Shift the date to a given number of months away."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef roll_qtrday(other, n, month, day_option, modby=3):\n\n months_since = other.month % modby - month % modby\n\n if n > 0:\n if months_since < 0 or (\n months_since == 0 and\n other.day < _get_day_of_month(other, day_option)):\n # pretend to roll back if on same month but\n # before compare_day\n n -= 1\n else:\n if months_since > 0 or (\n months_since == 0 and\n other.day > _get_day_of_month(other, day_option)):\n # make sure to roll forward, so negate\n n += 1\n return n", "response": "Roll the number of periods to a given day in a given month."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_offset(freq):\n if isinstance(freq, BaseCFTimeOffset):\n return freq\n else:\n try:\n freq_data = re.match(_PATTERN, freq).groupdict()\n except AttributeError:\n raise ValueError('Invalid frequency string provided')\n\n freq = freq_data['freq']\n multiples = freq_data['multiple']\n if multiples is None:\n multiples = 1\n else:\n multiples = int(multiples)\n\n return _FREQUENCIES[freq](n=multiples)", "response": "Convert a frequency string to the appropriate subclass of BaseCFTimeOffset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating an equally - spaced sequence of cftime. datetime objects between start and end.", "response": "def _generate_linear_range(start, end, periods):\n \"\"\"Generate an equally-spaced sequence of cftime.datetime objects between\n and including two dates (whose length equals the number of periods).\"\"\"\n import cftime\n\n total_seconds = (end - start).total_seconds()\n values = np.linspace(0., total_seconds, periods, endpoint=True)\n units = 'seconds since {}'.format(format_cftime_datetime(start))\n calendar = start.calendar\n return cftime.num2date(values, units=units, calendar=calendar,\n only_use_cftime_datetimes=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _generate_range(start, end, periods, offset):\n if start:\n start = offset.rollforward(start)\n\n if end:\n end = offset.rollback(end)\n\n if periods is None and end < start:\n end = None\n periods = 0\n\n if end is None:\n end = start + (periods - 1) * offset\n\n if start is None:\n start = end - (periods - 1) * offset\n\n current = start\n if offset.n >= 0:\n while current <= end:\n yield current\n\n next_date = current + offset\n if next_date <= current:\n raise ValueError('Offset {offset} did not increment date'\n .format(offset=offset))\n current = next_date\n else:\n while current >= end:\n yield current\n\n next_date = current + offset\n if next_date >= current:\n raise ValueError('Offset {offset} did not decrement date'\n .format(offset=offset))\n current = next_date", "response": "Generate a regular range of cftime. datetime objects with a specific time offset."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a fixed frequency CFTimeIndex for the given start and end dates.", "response": "def cftime_range(start=None, end=None, periods=None, freq='D',\n normalize=False, name=None, closed=None,\n calendar='standard'):\n \"\"\"Return a fixed frequency CFTimeIndex.\n\n Parameters\n ----------\n start : str or cftime.datetime, optional\n Left bound for generating dates.\n end : str or cftime.datetime, optional\n Right bound for generating dates.\n periods : integer, optional\n Number of periods to generate.\n freq : str, default 'D', BaseCFTimeOffset, or None\n Frequency strings can have multiples, e.g. '5H'.\n normalize : bool, default False\n Normalize start/end dates to midnight before generating date range.\n name : str, default None\n Name of the resulting index\n closed : {None, 'left', 'right'}, optional\n Make the interval closed with respect to the given frequency to the\n 'left', 'right', or both sides (None, the default).\n calendar : str\n Calendar type for the datetimes (default 'standard').\n\n Returns\n -------\n CFTimeIndex\n\n Notes\n -----\n\n This function is an analog of ``pandas.date_range`` for use in generating\n sequences of ``cftime.datetime`` objects. It supports most of the\n features of ``pandas.date_range`` (e.g. specifying how the index is\n ``closed`` on either side, or whether or not to ``normalize`` the start and\n end bounds); however, there are some notable exceptions:\n\n - You cannot specify a ``tz`` (time zone) argument.\n - Start or end dates specified as partial-datetime strings must use the\n `ISO-8601 format `_.\n - It supports many, but not all, frequencies supported by\n ``pandas.date_range``. For example it does not currently support any of\n the business-related, semi-monthly, or sub-second frequencies.\n - Compound sub-monthly frequencies are not supported, e.g. '1H1min', as\n these can easily be written in terms of the finest common resolution,\n e.g. '61min'.\n\n Valid simple frequency strings for use with ``cftime``-calendars include\n any multiples of the following.\n\n +--------+--------------------------+\n | Alias | Description |\n +========+==========================+\n | A, Y | Year-end frequency |\n +--------+--------------------------+\n | AS, YS | Year-start frequency |\n +--------+--------------------------+\n | Q | Quarter-end frequency |\n +--------+--------------------------+\n | QS | Quarter-start frequency |\n +--------+--------------------------+\n | M | Month-end frequency |\n +--------+--------------------------+\n | MS | Month-start frequency |\n +--------+--------------------------+\n | D | Day frequency |\n +--------+--------------------------+\n | H | Hour frequency |\n +--------+--------------------------+\n | T, min | Minute frequency |\n +--------+--------------------------+\n | S | Second frequency |\n +--------+--------------------------+\n\n Any multiples of the following anchored offsets are also supported.\n\n +----------+--------------------------------------------------------------------+\n | Alias | Description |\n +==========+====================================================================+\n | A(S)-JAN | Annual frequency, anchored at the end (or beginning) of January |\n +----------+--------------------------------------------------------------------+\n | A(S)-FEB | Annual frequency, anchored at the end (or beginning) of February |\n +----------+--------------------------------------------------------------------+\n | A(S)-MAR | Annual frequency, anchored at the end (or beginning) of March |\n +----------+--------------------------------------------------------------------+\n | A(S)-APR | Annual frequency, anchored at the end (or beginning) of April |\n +----------+--------------------------------------------------------------------+\n | A(S)-MAY | Annual frequency, anchored at the end (or beginning) of May |\n +----------+--------------------------------------------------------------------+\n | A(S)-JUN | Annual frequency, anchored at the end (or beginning) of June |\n +----------+--------------------------------------------------------------------+\n | A(S)-JUL | Annual frequency, anchored at the end (or beginning) of July |\n +----------+--------------------------------------------------------------------+\n | A(S)-AUG | Annual frequency, anchored at the end (or beginning) of August |\n +----------+--------------------------------------------------------------------+\n | A(S)-SEP | Annual frequency, anchored at the end (or beginning) of September |\n +----------+--------------------------------------------------------------------+\n | A(S)-OCT | Annual frequency, anchored at the end (or beginning) of October |\n +----------+--------------------------------------------------------------------+\n | A(S)-NOV | Annual frequency, anchored at the end (or beginning) of November |\n +----------+--------------------------------------------------------------------+\n | A(S)-DEC | Annual frequency, anchored at the end (or beginning) of December |\n +----------+--------------------------------------------------------------------+\n | Q(S)-JAN | Quarter frequency, anchored at the end (or beginning) of January |\n +----------+--------------------------------------------------------------------+\n | Q(S)-FEB | Quarter frequency, anchored at the end (or beginning) of February |\n +----------+--------------------------------------------------------------------+\n | Q(S)-MAR | Quarter frequency, anchored at the end (or beginning) of March |\n +----------+--------------------------------------------------------------------+\n | Q(S)-APR | Quarter frequency, anchored at the end (or beginning) of April |\n +----------+--------------------------------------------------------------------+\n | Q(S)-MAY | Quarter frequency, anchored at the end (or beginning) of May |\n +----------+--------------------------------------------------------------------+\n | Q(S)-JUN | Quarter frequency, anchored at the end (or beginning) of June |\n +----------+--------------------------------------------------------------------+\n | Q(S)-JUL | Quarter frequency, anchored at the end (or beginning) of July |\n +----------+--------------------------------------------------------------------+\n | Q(S)-AUG | Quarter frequency, anchored at the end (or beginning) of August |\n +----------+--------------------------------------------------------------------+\n | Q(S)-SEP | Quarter frequency, anchored at the end (or beginning) of September |\n +----------+--------------------------------------------------------------------+\n | Q(S)-OCT | Quarter frequency, anchored at the end (or beginning) of October |\n +----------+--------------------------------------------------------------------+\n | Q(S)-NOV | Quarter frequency, anchored at the end (or beginning) of November |\n +----------+--------------------------------------------------------------------+\n | Q(S)-DEC | Quarter frequency, anchored at the end (or beginning) of December |\n +----------+--------------------------------------------------------------------+\n\n\n Finally, the following calendar aliases are supported.\n\n +--------------------------------+---------------------------------------+\n | Alias | Date type |\n +================================+=======================================+\n | standard, gregorian | ``cftime.DatetimeGregorian`` |\n +--------------------------------+---------------------------------------+\n | proleptic_gregorian | ``cftime.DatetimeProlepticGregorian`` |\n +--------------------------------+---------------------------------------+\n | noleap, 365_day | ``cftime.DatetimeNoLeap`` |\n +--------------------------------+---------------------------------------+\n | all_leap, 366_day | ``cftime.DatetimeAllLeap`` |\n +--------------------------------+---------------------------------------+\n | 360_day | ``cftime.Datetime360Day`` |\n +--------------------------------+---------------------------------------+\n | julian | ``cftime.DatetimeJulian`` |\n +--------------------------------+---------------------------------------+\n\n Examples\n --------\n\n This function returns a ``CFTimeIndex``, populated with ``cftime.datetime``\n objects associated with the specified calendar type, e.g.\n\n >>> xr.cftime_range(start='2000', periods=6, freq='2MS', calendar='noleap')\n CFTimeIndex([2000-01-01 00:00:00, 2000-03-01 00:00:00, 2000-05-01 00:00:00,\n 2000-07-01 00:00:00, 2000-09-01 00:00:00, 2000-11-01 00:00:00],\n dtype='object')\n\n As in the standard pandas function, three of the ``start``, ``end``,\n ``periods``, or ``freq`` arguments must be specified at a given time, with\n the other set to ``None``. See the `pandas documentation\n `_\n for more examples of the behavior of ``date_range`` with each of the\n parameters.\n\n See Also\n --------\n pandas.date_range\n \"\"\" # noqa: E501\n # Adapted from pandas.core.indexes.datetimes._generate_range.\n if _count_not_none(start, end, periods, freq) != 3:\n raise ValueError(\n \"Of the arguments 'start', 'end', 'periods', and 'freq', three \"\n \"must be specified at a time.\")\n\n if start is not None:\n start = to_cftime_datetime(start, calendar)\n start = _maybe_normalize_date(start, normalize)\n if end is not None:\n end = to_cftime_datetime(end, calendar)\n end = _maybe_normalize_date(end, normalize)\n\n if freq is None:\n dates = _generate_linear_range(start, end, periods)\n else:\n offset = to_offset(freq)\n dates = np.array(list(_generate_range(start, end, periods, offset)))\n\n left_closed = False\n right_closed = False\n\n if closed is None:\n left_closed = True\n right_closed = True\n elif closed == 'left':\n left_closed = True\n elif closed == 'right':\n right_closed = True\n else:\n raise ValueError(\"Closed must be either 'left', 'right' or None\")\n\n if (not left_closed and len(dates) and\n start is not None and dates[0] == start):\n dates = dates[1:]\n if (not right_closed and len(dates) and\n end is not None and dates[-1] == end):\n dates = dates[:-1]\n\n return CFTimeIndex(dates, name=name)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if the given date is in the set of possible dates created using a length - one version of this offset class.", "response": "def onOffset(self, date):\n \"\"\"Check if the given date is in the set of possible dates created\n using a length-one version of this offset class.\"\"\"\n mod_month = (date.month - self.month) % 3\n return mod_month == 0 and date.day == self._get_offset_day(date)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrolls date forward to nearest start of quarter", "response": "def rollforward(self, date):\n \"\"\"Roll date forward to nearest start of quarter\"\"\"\n if self.onOffset(date):\n return date\n else:\n return date + QuarterBegin(month=self.month)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rollback(self, date):\n if self.onOffset(date):\n return date\n else:\n return date - QuarterBegin(month=self.month)", "response": "Roll date backward to nearest start of quarter"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rollforward(self, date):\n if self.onOffset(date):\n return date\n else:\n return date + QuarterEnd(month=self.month)", "response": "Roll date forward to nearest end of quarter"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nroll date backward to nearest end of quarter", "response": "def rollback(self, date):\n \"\"\"Roll date backward to nearest end of quarter\"\"\"\n if self.onOffset(date):\n return date\n else:\n return date - QuarterEnd(month=self.month)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rollforward(self, date):\n if self.onOffset(date):\n return date\n else:\n return date + YearBegin(month=self.month)", "response": "Roll date forward to nearest start of year"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rollback(self, date):\n if self.onOffset(date):\n return date\n else:\n return date - YearBegin(month=self.month)", "response": "Roll date backward to nearest start of year"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if the given date is in the set of possible dates created using a length - one version of this offset class.", "response": "def onOffset(self, date):\n \"\"\"Check if the given date is in the set of possible dates created\n using a length-one version of this offset class.\"\"\"\n return date.day == _days_in_month(date) and date.month == self.month"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrolling date forward to nearest end of year", "response": "def rollforward(self, date):\n \"\"\"Roll date forward to nearest end of year\"\"\"\n if self.onOffset(date):\n return date\n else:\n return date + YearEnd(month=self.month)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrolls date backward to nearest end of year", "response": "def rollback(self, date):\n \"\"\"Roll date backward to nearest end of year\"\"\"\n if self.onOffset(date):\n return date\n else:\n return date - YearEnd(month=self.month)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving an object x and numchars long format it so that it is numchars long and truncate it with trailing spaces as necessary.", "response": "def pretty_print(x, numchars):\n \"\"\"Given an object `x`, call `str(x)` and format the returned string so\n that it is numchars long, padding with trailing spaces or truncating with\n ellipses as necessary\n \"\"\"\n s = maybe_truncate(x, numchars)\n return s + ' ' * max(numchars - len(s), 0)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef first_n_items(array, n_desired):\n # Unfortunately, we can't just do array.flat[:n_desired] here because it\n # might not be a numpy.ndarray. Moreover, access to elements of the array\n # could be very expensive (e.g. if it's only available over DAP), so go out\n # of our way to get them in a single call to __getitem__ using only slices.\n if n_desired < 1:\n raise ValueError('must request at least one item')\n\n if array.size == 0:\n # work around for https://github.com/numpy/numpy/issues/5195\n return []\n\n if n_desired < array.size:\n indexer = _get_indexer_at_least_n_items(array.shape, n_desired,\n from_end=False)\n array = array[indexer]\n return np.asarray(array).flat[:n_desired]", "response": "Returns the first n_desired items of an array"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the last n_desired items of an array", "response": "def last_n_items(array, n_desired):\n \"\"\"Returns the last n_desired items of an array\"\"\"\n # Unfortunately, we can't just do array.flat[-n_desired:] here because it\n # might not be a numpy.ndarray. Moreover, access to elements of the array\n # could be very expensive (e.g. if it's only available over DAP), so go out\n # of our way to get them in a single call to __getitem__ using only slices.\n if (n_desired == 0) or (array.size == 0):\n return []\n\n if n_desired < array.size:\n indexer = _get_indexer_at_least_n_items(array.shape, n_desired,\n from_end=True)\n array = array[indexer]\n return np.asarray(array).flat[-n_desired:]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef last_item(array):\n if array.size == 0:\n # work around for https://github.com/numpy/numpy/issues/5195\n return []\n\n indexer = (slice(-1, None),) * array.ndim\n return np.ravel(array[indexer]).tolist()", "response": "Returns the last item of an array in a list or an empty list."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef format_timestamp(t):\n # Timestamp is only valid for 1678 to 2262\n try:\n datetime_str = str(pd.Timestamp(t))\n except OutOfBoundsDatetime:\n datetime_str = str(t)\n\n try:\n date_str, time_str = datetime_str.split()\n except ValueError:\n # catch NaT and others that don't split nicely\n return datetime_str\n else:\n if time_str == '00:00:00':\n return date_str\n else:\n return '{}T{}'.format(date_str, time_str)", "response": "Cast given object to a Timestamp and return a nicely formatted string"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncasting given object to a Timestamp and return a nicely formatted string", "response": "def format_timedelta(t, timedelta_format=None):\n \"\"\"Cast given object to a Timestamp and return a nicely formatted string\"\"\"\n timedelta_str = str(pd.Timedelta(t))\n try:\n days_str, time_str = timedelta_str.split(' days ')\n except ValueError:\n # catch NaT and others that don't split nicely\n return timedelta_str\n else:\n if timedelta_format == 'date':\n return days_str + ' days'\n elif timedelta_format == 'time':\n return time_str\n else:\n return timedelta_str"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a succinct summary of an object as a string", "response": "def format_item(x, timedelta_format=None, quote_strings=True):\n \"\"\"Returns a succinct summary of an object as a string\"\"\"\n if isinstance(x, (np.datetime64, datetime)):\n return format_timestamp(x)\n if isinstance(x, (np.timedelta64, timedelta)):\n return format_timedelta(x, timedelta_format=timedelta_format)\n elif isinstance(x, (str, bytes)):\n return repr(x) if quote_strings else x\n elif isinstance(x, (float, np.float)):\n return '{0:.4}'.format(x)\n else:\n return str(x)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a succinct summaries of all items in a sequence as strings", "response": "def format_items(x):\n \"\"\"Returns a succinct summaries of all items in a sequence as strings\"\"\"\n x = np.asarray(x)\n timedelta_format = 'datetime'\n if np.issubdtype(x.dtype, np.timedelta64):\n x = np.asarray(x, dtype='timedelta64[ns]')\n day_part = (x[~pd.isnull(x)]\n .astype('timedelta64[D]')\n .astype('timedelta64[ns]'))\n time_needed = x[~pd.isnull(x)] != day_part\n day_needed = day_part != np.timedelta64(0, 'ns')\n if np.logical_not(day_needed).all():\n timedelta_format = 'time'\n elif np.logical_not(time_needed).all():\n timedelta_format = 'date'\n\n formatted = [format_item(xi, timedelta_format) for xi in x]\n return formatted"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a string for as many items in the flattened version of the array that will fit within max_width characters.", "response": "def format_array_flat(array, max_width):\n \"\"\"Return a formatted string for as many items in the flattened version of\n array that will fit within max_width characters.\n \"\"\"\n # every item will take up at least two characters, but we always want to\n # print at least first and last items\n max_possibly_relevant = min(max(array.size, 1),\n max(int(np.ceil(max_width / 2.)), 2))\n relevant_front_items = format_items(\n first_n_items(array, (max_possibly_relevant + 1) // 2))\n relevant_back_items = format_items(\n last_n_items(array, max_possibly_relevant // 2))\n # interleave relevant front and back items:\n # [a, b, c] and [y, z] -> [a, z, b, y, c]\n relevant_items = sum(zip_longest(relevant_front_items,\n reversed(relevant_back_items)),\n ())[:max_possibly_relevant]\n\n cum_len = np.cumsum([len(s) + 1 for s in relevant_items]) - 1\n if (array.size > 2) and ((max_possibly_relevant < array.size) or\n (cum_len > max_width).any()):\n padding = ' ... '\n count = min(array.size,\n max(np.argmax(cum_len + len(padding) - 1 > max_width), 2))\n else:\n count = array.size\n padding = '' if (count <= 1) else ' '\n\n num_front = (count + 1) // 2\n num_back = count - num_front\n # note that num_back is 0 <--> array.size is 0 or 1\n # <--> relevant_back_items is []\n pprint_str = (' '.join(relevant_front_items[:num_front]) +\n padding +\n ' '.join(relevant_back_items[-num_back:]))\n return pprint_str"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets all column items to format including both keys of mapping and MultiIndex levels if any.", "response": "def _get_col_items(mapping):\n \"\"\"Get all column items to format, including both keys of `mapping`\n and MultiIndex levels if any.\n \"\"\"\n from .variable import IndexVariable\n\n col_items = []\n for k, v in mapping.items():\n col_items.append(k)\n var = getattr(v, 'variable', v)\n if isinstance(var, IndexVariable):\n level_names = var.to_index_variable().level_names\n if level_names is not None:\n col_items += list(level_names)\n return col_items"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef short_dask_repr(array, show_dtype=True):\n chunksize = tuple(c[0] for c in array.chunks)\n if show_dtype:\n return 'dask.array'.format(\n array.shape, array.dtype, chunksize)\n else:\n return 'dask.array'.format(\n array.shape, chunksize)", "response": "Similar to dask. array. DataArray. __repr__ but without\n redundant information that s already printed by the repr\n function of the xarray wrapper."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _upsample(self, method, *args, **kwargs):\n\n upsampled_index = self._full_index\n\n # Drop non-dimension coordinates along the resampled dimension\n for k, v in self._obj.coords.items():\n if k == self._dim:\n continue\n if self._dim in v.dims:\n self._obj = self._obj.drop(k)\n\n if method == 'asfreq':\n return self.mean(self._dim)\n\n elif method in ['pad', 'ffill', 'backfill', 'bfill', 'nearest']:\n kwargs = kwargs.copy()\n kwargs.update(**{self._dim: upsampled_index})\n return self._obj.reindex(method=method, *args, **kwargs)\n\n elif method == 'interpolate':\n return self._interpolate(*args, **kwargs)\n\n else:\n raise ValueError('Specified method was \"{}\" but must be one of'\n '\"asfreq\", \"ffill\", \"bfill\", or \"interpolate\"'\n .format(method))", "response": "Dispatch function to call appropriate up - sampling methods on the object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _interpolate(self, kind='linear'):\n # drop any existing non-dimension coordinates along the resampling\n # dimension\n dummy = self._obj.copy()\n for k, v in self._obj.coords.items():\n if k != self._dim and self._dim in v.dims:\n dummy = dummy.drop(k)\n return dummy.interp(assume_sorted=True, method=kind,\n kwargs={'bounds_error': False},\n **{self._dim: self._full_index})", "response": "Apply scipy. interpolate. interp1d along resampling dimension."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef apply(self, func, shortcut=False, args=(), **kwargs):\n combined = super(DataArrayResample, self).apply(\n func, shortcut=shortcut, args=args, **kwargs)\n\n # If the aggregation function didn't drop the original resampling\n # dimension, then we need to do so before we can rename the proxy\n # dimension we used.\n if self._dim in combined.coords:\n combined = combined.drop(self._dim)\n\n if self._resample_dim in combined.dims:\n combined = combined.rename({self._resample_dim: self._dim})\n\n return combined", "response": "Apply a function over each array in the group and concatenate them together into a new array."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\napplying a function over each Dataset in the groups generated for resampling and concatenate them together into a new Dataset.", "response": "def apply(self, func, args=(), **kwargs):\n \"\"\"Apply a function over each Dataset in the groups generated for\n resampling and concatenate them together into a new Dataset.\n\n `func` is called like `func(ds, *args, **kwargs)` for each dataset `ds`\n in this group.\n\n Apply uses heuristics (like `pandas.GroupBy.apply`) to figure out how\n to stack together the datasets. The rule is:\n 1. If the dimension along which the group coordinate is defined is\n still in the first grouped item after applying `func`, then stack\n over this dimension.\n 2. Otherwise, stack over the new dimension given by name of this\n grouping (the argument to the `groupby` function).\n\n Parameters\n ----------\n func : function\n Callable to apply to each sub-dataset.\n args : tuple, optional\n Positional arguments passed on to `func`.\n **kwargs\n Used to call `func(ds, **kwargs)` for each sub-dataset `ar`.\n\n Returns\n -------\n applied : Dataset or DataArray\n The result of splitting, applying and combining this dataset.\n \"\"\"\n kwargs.pop('shortcut', None) # ignore shortcut if set (for now)\n applied = (func(ds, *args, **kwargs) for ds in self._iter_grouped())\n combined = self._combine(applied)\n\n return combined.rename({self._resample_dim: self._dim})"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreduces the items in this group by applying func along the specified resampling dimension.", "response": "def reduce(self, func, dim=None, keep_attrs=None, **kwargs):\n \"\"\"Reduce the items in this group by applying `func` along the\n pre-defined resampling dimension.\n\n Parameters\n ----------\n func : function\n Function which can be called in the form\n `func(x, axis=axis, **kwargs)` to return the result of collapsing\n an np.ndarray over an integer valued axis.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply `func`.\n keep_attrs : bool, optional\n If True, the datasets's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n **kwargs : dict\n Additional keyword arguments passed on to `func`.\n\n Returns\n -------\n reduced : Array\n Array with summarized data and the indicated dimension(s)\n removed.\n \"\"\"\n if dim == DEFAULT_DIMS:\n dim = None\n\n return super(DatasetResample, self).reduce(\n func, dim, keep_attrs, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef default_indexes(\n coords: Mapping[Any, Variable],\n dims: Iterable,\n) -> 'OrderedDict[Any, pd.Index]':\n \"\"\"Default indexes for a Dataset/DataArray.\n\n Parameters\n ----------\n coords : Mapping[Any, xarray.Variable]\n Coordinate variables from which to draw default indexes.\n dims : iterable\n Iterable of dimension names.\n\n Returns\n -------\n Mapping from indexing keys (levels/dimension names) to indexes used for\n indexing along that dimension.\n \"\"\"\n return OrderedDict((key, coords[key].to_index())\n for key in dims if key in coords)", "response": "Returns a default indexes for a Dataset or DataArray."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef isel_variable_and_index(\n name: Hashable,\n variable: Variable,\n index: pd.Index,\n indexers: Mapping[Any, Union[slice, Variable]],\n) -> Tuple[Variable, Optional[pd.Index]]:\n \"\"\"Index a Variable and pandas.Index together.\"\"\"\n if not indexers:\n # nothing to index\n return variable.copy(deep=False), index\n\n if len(variable.dims) > 1:\n raise NotImplementedError(\n 'indexing multi-dimensional variable with indexes is not '\n 'supported yet')\n\n new_variable = variable.isel(indexers)\n\n if new_variable.dims != (name,):\n # can't preserve a index if result has new dimensions\n return new_variable, None\n\n # we need to compute the new index\n (dim,) = variable.dims\n indexer = indexers[dim]\n if isinstance(indexer, Variable):\n indexer = indexer.data\n new_index = index[indexer]\n return new_variable, new_index", "response": "Index a Variable and pandas. Index together."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_output_coords(\n args: list,\n signature: _UFuncSignature,\n exclude_dims: AbstractSet = frozenset(),\n) -> 'List[OrderedDict[Any, Variable]]':\n \"\"\"Build output coordinates for an operation.\n\n Parameters\n ----------\n args : list\n List of raw operation arguments. Any valid types for xarray operations\n are OK, e.g., scalars, Variable, DataArray, Dataset.\n signature : _UfuncSignature\n Core dimensions signature for the operation.\n exclude_dims : optional set\n Dimensions excluded from the operation. Coordinates along these\n dimensions are dropped.\n\n Returns\n -------\n OrderedDict of Variable objects with merged coordinates.\n \"\"\"\n input_coords = _get_coord_variables(args)\n\n if exclude_dims:\n input_coords = [OrderedDict((k, v) for k, v in coord_vars.items()\n if exclude_dims.isdisjoint(v.dims))\n for coord_vars in input_coords]\n\n if len(input_coords) == 1:\n # we can skip the expensive merge\n unpacked_input_coords, = input_coords\n merged = OrderedDict(unpacked_input_coords)\n else:\n merged = expand_and_merge_variables(input_coords)\n\n output_coords = []\n for output_dims in signature.output_core_dims:\n dropped_dims = signature.all_input_core_dims - set(output_dims)\n if dropped_dims:\n filtered = OrderedDict((k, v) for k, v in merged.items()\n if dropped_dims.isdisjoint(v.dims))\n else:\n filtered = merged\n output_coords.append(filtered)\n\n return output_coords", "response": "Builds the output coordinates for an operation."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\napply a variable level function over DataArray Variable and ndarray objects.", "response": "def apply_dataarray_vfunc(\n func,\n *args,\n signature,\n join='inner',\n exclude_dims=frozenset(),\n keep_attrs=False\n):\n \"\"\"Apply a variable level function over DataArray, Variable and/or ndarray\n objects.\n \"\"\"\n from .dataarray import DataArray\n\n if len(args) > 1:\n args = deep_align(args, join=join, copy=False, exclude=exclude_dims,\n raise_on_invalid=False)\n\n if keep_attrs and hasattr(args[0], 'name'):\n name = args[0].name\n else:\n name = result_name(args)\n result_coords = build_output_coords(args, signature, exclude_dims)\n\n data_vars = [getattr(a, 'variable', a) for a in args]\n result_var = func(*data_vars)\n\n if signature.num_outputs > 1:\n out = tuple(DataArray(variable, coords, name=name, fastpath=True)\n for variable, coords in zip(result_var, result_coords))\n else:\n coords, = result_coords\n out = DataArray(result_var, coords, name=name, fastpath=True)\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef apply_dict_of_variables_vfunc(\n func, *args, signature, join='inner', fill_value=None\n):\n \"\"\"Apply a variable level function over dicts of DataArray, DataArray,\n Variable and ndarray objects.\n \"\"\"\n args = [_as_variables_or_variable(arg) for arg in args]\n names = join_dict_keys(args, how=join)\n grouped_by_name = collect_dict_values(args, names, fill_value)\n\n result_vars = OrderedDict()\n for name, variable_args in zip(names, grouped_by_name):\n result_vars[name] = func(*variable_args)\n\n if signature.num_outputs > 1:\n return _unpack_dict_tuples(result_vars, signature.num_outputs)\n else:\n return result_vars", "response": "Apply a variable level function over dicts of DataArray Variable and ndarray objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _fast_dataset(\n variables: 'OrderedDict[Any, Variable]',\n coord_variables: Mapping[Any, Variable],\n) -> 'Dataset':\n \"\"\"Create a dataset as quickly as possible.\n\n Beware: the `variables` OrderedDict is modified INPLACE.\n \"\"\"\n from .dataset import Dataset\n variables.update(coord_variables)\n coord_names = set(coord_variables)\n return Dataset._from_vars_and_coord_names(variables, coord_names)", "response": "Create a Dataset as quickly as possible."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\napply a variable level function over Dataset dict of DataArray Variable and ndarray objects.", "response": "def apply_dataset_vfunc(\n func,\n *args,\n signature,\n join='inner',\n dataset_join='exact',\n fill_value=_NO_FILL_VALUE,\n exclude_dims=frozenset(),\n keep_attrs=False\n):\n \"\"\"Apply a variable level function over Dataset, dict of DataArray,\n DataArray, Variable and/or ndarray objects.\n \"\"\"\n from .dataset import Dataset\n first_obj = args[0] # we'll copy attrs from this in case keep_attrs=True\n\n if (dataset_join not in _JOINS_WITHOUT_FILL_VALUES and\n fill_value is _NO_FILL_VALUE):\n raise TypeError('to apply an operation to datasets with different '\n 'data variables with apply_ufunc, you must supply the '\n 'dataset_fill_value argument.')\n\n if len(args) > 1:\n args = deep_align(args, join=join, copy=False, exclude=exclude_dims,\n raise_on_invalid=False)\n\n list_of_coords = build_output_coords(args, signature, exclude_dims)\n args = [getattr(arg, 'data_vars', arg) for arg in args]\n\n result_vars = apply_dict_of_variables_vfunc(\n func, *args, signature=signature, join=dataset_join,\n fill_value=fill_value)\n\n if signature.num_outputs > 1:\n out = tuple(_fast_dataset(*args)\n for args in zip(result_vars, list_of_coords))\n else:\n coord_vars, = list_of_coords\n out = _fast_dataset(result_vars, coord_vars)\n\n if keep_attrs and isinstance(first_obj, Dataset):\n if isinstance(out, tuple):\n out = tuple(ds._copy_attrs_from(first_obj) for ds in out)\n else:\n out._copy_attrs_from(first_obj)\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\niterate over selections of an xarray object in the provided order.", "response": "def _iter_over_selections(obj, dim, values):\n \"\"\"Iterate over selections of an xarray object in the provided order.\"\"\"\n from .groupby import _dummy_copy\n\n dummy = None\n for value in values:\n try:\n obj_sel = obj.sel(**{dim: value})\n except (KeyError, IndexError):\n if dummy is None:\n dummy = _dummy_copy(obj)\n obj_sel = dummy\n yield obj_sel"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply a function over a GroupBy Dataset DataArray or ndarray objects.", "response": "def apply_groupby_func(func, *args):\n \"\"\"Apply a dataset or datarray level function over GroupBy, Dataset,\n DataArray, Variable and/or ndarray objects.\n \"\"\"\n from .groupby import GroupBy, peek_at\n from .variable import Variable\n\n groupbys = [arg for arg in args if isinstance(arg, GroupBy)]\n assert groupbys, 'must have at least one groupby to iterate over'\n first_groupby = groupbys[0]\n if any(not first_groupby._group.equals(gb._group) for gb in groupbys[1:]):\n raise ValueError('apply_ufunc can only perform operations over '\n 'multiple GroupBy objets at once if they are all '\n 'grouped the same way')\n\n grouped_dim = first_groupby._group.name\n unique_values = first_groupby._unique_coord.values\n\n iterators = []\n for arg in args:\n if isinstance(arg, GroupBy):\n iterator = (value for _, value in arg)\n elif hasattr(arg, 'dims') and grouped_dim in arg.dims:\n if isinstance(arg, Variable):\n raise ValueError(\n 'groupby operations cannot be performed with '\n 'xarray.Variable objects that share a dimension with '\n 'the grouped dimension')\n iterator = _iter_over_selections(arg, grouped_dim, unique_values)\n else:\n iterator = itertools.repeat(arg)\n iterators.append(iterator)\n\n applied = (func(*zipped_args) for zipped_args in zip(*iterators))\n applied_example, applied = peek_at(applied)\n combine = first_groupby._combine\n if isinstance(applied_example, tuple):\n combined = tuple(combine(output) for output in zip(*applied))\n else:\n combined = combine(applied)\n return combined"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\napply a function over a Variable and ndarray objects.", "response": "def apply_variable_ufunc(\n func,\n *args,\n signature,\n exclude_dims=frozenset(),\n dask='forbidden',\n output_dtypes=None,\n output_sizes=None,\n keep_attrs=False\n):\n \"\"\"Apply a ndarray level function over Variable and/or ndarray objects.\n \"\"\"\n from .variable import Variable, as_compatible_data\n\n dim_sizes = unified_dim_sizes((a for a in args if hasattr(a, 'dims')),\n exclude_dims=exclude_dims)\n broadcast_dims = tuple(dim for dim in dim_sizes\n if dim not in signature.all_core_dims)\n output_dims = [broadcast_dims + out for out in signature.output_core_dims]\n\n input_data = [broadcast_compat_data(arg, broadcast_dims, core_dims)\n if isinstance(arg, Variable)\n else arg\n for arg, core_dims in zip(args, signature.input_core_dims)]\n\n if any(isinstance(array, dask_array_type) for array in input_data):\n if dask == 'forbidden':\n raise ValueError('apply_ufunc encountered a dask array on an '\n 'argument, but handling for dask arrays has not '\n 'been enabled. Either set the ``dask`` argument '\n 'or load your data into memory first with '\n '``.load()`` or ``.compute()``')\n elif dask == 'parallelized':\n input_dims = [broadcast_dims + dims\n for dims in signature.input_core_dims]\n numpy_func = func\n\n def func(*arrays):\n return _apply_with_dask_atop(\n numpy_func, arrays, input_dims, output_dims,\n signature, output_dtypes, output_sizes)\n elif dask == 'allowed':\n pass\n else:\n raise ValueError('unknown setting for dask array handling in '\n 'apply_ufunc: {}'.format(dask))\n result_data = func(*input_data)\n\n if signature.num_outputs == 1:\n result_data = (result_data,)\n elif (not isinstance(result_data, tuple) or\n len(result_data) != signature.num_outputs):\n raise ValueError('applied function does not have the number of '\n 'outputs specified in the ufunc signature. '\n 'Result is not a tuple of {} elements: {!r}'\n .format(signature.num_outputs, result_data))\n\n output = []\n for dims, data in zip(output_dims, result_data):\n data = as_compatible_data(data)\n if data.ndim != len(dims):\n raise ValueError(\n 'applied function returned data with unexpected '\n 'number of dimensions: {} vs {}, for dimensions {}'\n .format(data.ndim, len(dims), dims))\n\n var = Variable(dims, data, fastpath=True)\n for dim, new_size in var.sizes.items():\n if dim in dim_sizes and new_size != dim_sizes[dim]:\n raise ValueError(\n 'size of dimension {!r} on inputs was unexpectedly '\n 'changed by applied function from {} to {}. Only '\n 'dimensions specified in ``exclude_dims`` with '\n 'xarray.apply_ufunc are allowed to change size.'\n .format(dim, dim_sizes[dim], new_size))\n\n if keep_attrs and isinstance(args[0], Variable):\n var.attrs.update(args[0].attrs)\n output.append(var)\n\n if signature.num_outputs == 1:\n return output[0]\n else:\n return tuple(output)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apply_array_ufunc(func, *args, dask='forbidden'):\n if any(isinstance(arg, dask_array_type) for arg in args):\n if dask == 'forbidden':\n raise ValueError('apply_ufunc encountered a dask array on an '\n 'argument, but handling for dask arrays has not '\n 'been enabled. Either set the ``dask`` argument '\n 'or load your data into memory first with '\n '``.load()`` or ``.compute()``')\n elif dask == 'parallelized':\n raise ValueError(\"cannot use dask='parallelized' for apply_ufunc \"\n 'unless at least one input is an xarray object')\n elif dask == 'allowed':\n pass\n else:\n raise ValueError('unknown setting for dask array handling: {}'\n .format(dask))\n return func(*args)", "response": "Apply a ndarray level function over ndarray objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\napply a function to unlabeled arrays on xarray objects.", "response": "def apply_ufunc(\n func: Callable,\n *args: Any,\n input_core_dims: Optional[Sequence[Sequence]] = None,\n output_core_dims: Optional[Sequence[Sequence]] = ((),),\n exclude_dims: AbstractSet = frozenset(),\n vectorize: bool = False,\n join: str = 'exact',\n dataset_join: str = 'exact',\n dataset_fill_value: object = _NO_FILL_VALUE,\n keep_attrs: bool = False,\n kwargs: Mapping = None,\n dask: str = 'forbidden',\n output_dtypes: Optional[Sequence] = None,\n output_sizes: Optional[Mapping[Any, int]] = None\n) -> Any:\n \"\"\"Apply a vectorized function for unlabeled arrays on xarray objects.\n\n The function will be mapped over the data variable(s) of the input\n arguments using xarray's standard rules for labeled computation, including\n alignment, broadcasting, looping over GroupBy/Dataset variables, and\n merging of coordinates.\n\n Parameters\n ----------\n func : callable\n Function to call like ``func(*args, **kwargs)`` on unlabeled arrays\n (``.data``) that returns an array or tuple of arrays. If multiple\n arguments with non-matching dimensions are supplied, this function is\n expected to vectorize (broadcast) over axes of positional arguments in\n the style of NumPy universal functions [1]_ (if this is not the case,\n set ``vectorize=True``). If this function returns multiple outputs, you\n must set ``output_core_dims`` as well.\n *args : Dataset, DataArray, GroupBy, Variable, numpy/dask arrays or scalars\n Mix of labeled and/or unlabeled arrays to which to apply the function.\n input_core_dims : Sequence[Sequence], optional\n List of the same length as ``args`` giving the list of core dimensions\n on each input argument that should not be broadcast. By default, we\n assume there are no core dimensions on any input arguments.\n\n For example, ``input_core_dims=[[], ['time']]`` indicates that all\n dimensions on the first argument and all dimensions other than 'time'\n on the second argument should be broadcast.\n\n Core dimensions are automatically moved to the last axes of input\n variables before applying ``func``, which facilitates using NumPy style\n generalized ufuncs [2]_.\n output_core_dims : List[tuple], optional\n List of the same length as the number of output arguments from\n ``func``, giving the list of core dimensions on each output that were\n not broadcast on the inputs. By default, we assume that ``func``\n outputs exactly one array, with axes corresponding to each broadcast\n dimension.\n\n Core dimensions are assumed to appear as the last dimensions of each\n output in the provided order.\n exclude_dims : set, optional\n Core dimensions on the inputs to exclude from alignment and\n broadcasting entirely. Any input coordinates along these dimensions\n will be dropped. Each excluded dimension must also appear in\n ``input_core_dims`` for at least one argument. Only dimensions listed\n here are allowed to change size between input and output objects.\n vectorize : bool, optional\n If True, then assume ``func`` only takes arrays defined over core\n dimensions as input and vectorize it automatically with\n :py:func:`numpy.vectorize`. This option exists for convenience, but is\n almost always slower than supplying a pre-vectorized function.\n Using this option requires NumPy version 1.12 or newer.\n join : {'outer', 'inner', 'left', 'right', 'exact'}, optional\n Method for joining the indexes of the passed objects along each\n dimension, and the variables of Dataset objects with mismatched\n data variables:\n\n - 'outer': use the union of object indexes\n - 'inner': use the intersection of object indexes\n - 'left': use indexes from the first object with each dimension\n - 'right': use indexes from the last object with each dimension\n - 'exact': raise `ValueError` instead of aligning when indexes to be\n aligned are not equal\n dataset_join : {'outer', 'inner', 'left', 'right', 'exact'}, optional\n Method for joining variables of Dataset objects with mismatched\n data variables.\n\n - 'outer': take variables from both Dataset objects\n - 'inner': take only overlapped variables\n - 'left': take only variables from the first object\n - 'right': take only variables from the last object\n - 'exact': data variables on all Dataset objects must match exactly\n dataset_fill_value : optional\n Value used in place of missing variables on Dataset inputs when the\n datasets do not share the exact same ``data_vars``. Required if\n ``dataset_join not in {'inner', 'exact'}``, otherwise ignored.\n keep_attrs: boolean, Optional\n Whether to copy attributes from the first argument to the output.\n kwargs: dict, optional\n Optional keyword arguments passed directly on to call ``func``.\n dask: 'forbidden', 'allowed' or 'parallelized', optional\n How to handle applying to objects containing lazy data in the form of\n dask arrays:\n\n - 'forbidden' (default): raise an error if a dask array is encountered.\n - 'allowed': pass dask arrays directly on to ``func``.\n - 'parallelized': automatically parallelize ``func`` if any of the\n inputs are a dask array. If used, the ``output_dtypes`` argument must\n also be provided. Multiple output arguments are not yet supported.\n output_dtypes : list of dtypes, optional\n Optional list of output dtypes. Only used if dask='parallelized'.\n output_sizes : dict, optional\n Optional mapping from dimension names to sizes for outputs. Only used\n if dask='parallelized' and new dimensions (not found on inputs) appear\n on outputs.\n\n Returns\n -------\n Single value or tuple of Dataset, DataArray, Variable, dask.array.Array or\n numpy.ndarray, the first type on that list to appear on an input.\n\n Examples\n --------\n\n Calculate the vector magnitude of two arguments:\n\n >>> def magnitude(a, b):\n ... func = lambda x, y: np.sqrt(x ** 2 + y ** 2)\n ... return xr.apply_ufunc(func, a, b)\n\n You can now apply ``magnitude()`` to ``xr.DataArray`` and ``xr.Dataset``\n objects, with automatically preserved dimensions and coordinates, e.g.,\n\n >>> array = xr.DataArray([1, 2, 3], coords=[('x', [0.1, 0.2, 0.3])])\n >>> magnitude(array, -array)\n \n array([1.414214, 2.828427, 4.242641])\n Coordinates:\n * x (x) float64 0.1 0.2 0.3\n\n Plain scalars, numpy arrays and a mix of these with xarray objects is also\n supported:\n\n >>> magnitude(4, 5)\n 5.0\n >>> magnitude(3, np.array([0, 4]))\n array([3., 5.])\n >>> magnitude(array, 0)\n \n array([1., 2., 3.])\n Coordinates:\n * x (x) float64 0.1 0.2 0.3\n\n Other examples of how you could use ``apply_ufunc`` to write functions to\n (very nearly) replicate existing xarray functionality:\n\n Compute the mean (``.mean``) over one dimension::\n\n def mean(obj, dim):\n # note: apply always moves core dimensions to the end\n return apply_ufunc(np.mean, obj,\n input_core_dims=[[dim]],\n kwargs={'axis': -1})\n\n Inner product over a specific dimension (like ``xr.dot``)::\n\n def _inner(x, y):\n result = np.matmul(x[..., np.newaxis, :], y[..., :, np.newaxis])\n return result[..., 0, 0]\n\n def inner_product(a, b, dim):\n return apply_ufunc(_inner, a, b, input_core_dims=[[dim], [dim]])\n\n Stack objects along a new dimension (like ``xr.concat``)::\n\n def stack(objects, dim, new_coord):\n # note: this version does not stack coordinates\n func = lambda *x: np.stack(x, axis=-1)\n result = apply_ufunc(func, *objects,\n output_core_dims=[[dim]],\n join='outer',\n dataset_fill_value=np.nan)\n result[dim] = new_coord\n return result\n\n If your function is not vectorized but can be applied only to core\n dimensions, you can use ``vectorize=True`` to turn into a vectorized\n function. This wraps :py:func:`numpy.vectorize`, so the operation isn't\n terribly fast. Here we'll use it to calculate the distance between\n empirical samples from two probability distributions, using a scipy\n function that needs to be applied to vectors::\n\n import scipy.stats\n\n def earth_mover_distance(first_samples,\n second_samples,\n dim='ensemble'):\n return apply_ufunc(scipy.stats.wasserstein_distance,\n first_samples, second_samples,\n input_core_dims=[[dim], [dim]],\n vectorize=True)\n\n Most of NumPy's builtin functions already broadcast their inputs\n appropriately for use in `apply`. You may find helper functions such as\n numpy.broadcast_arrays helpful in writing your function. `apply_ufunc` also\n works well with numba's vectorize and guvectorize. Further explanation with\n examples are provided in the xarray documentation [3].\n\n See also\n --------\n numpy.broadcast_arrays\n numba.vectorize\n numba.guvectorize\n\n References\n ----------\n .. [1] http://docs.scipy.org/doc/numpy/reference/ufuncs.html\n .. [2] http://docs.scipy.org/doc/numpy/reference/c-api.generalized-ufuncs.html\n .. [3] http://xarray.pydata.org/en/stable/computation.html#wrapping-custom-computation\n \"\"\" # noqa: E501 # don't error on that URL one line up\n from .groupby import GroupBy\n from .dataarray import DataArray\n from .variable import Variable\n\n if input_core_dims is None:\n input_core_dims = ((),) * (len(args))\n elif len(input_core_dims) != len(args):\n raise ValueError(\n 'input_core_dims must be None or a tuple with the length same to '\n 'the number of arguments. Given input_core_dims: {}, '\n 'number of args: {}.'.format(input_core_dims, len(args)))\n\n if kwargs is None:\n kwargs = {}\n\n signature = _UFuncSignature(input_core_dims, output_core_dims)\n\n if exclude_dims and not exclude_dims <= signature.all_core_dims:\n raise ValueError('each dimension in `exclude_dims` must also be a '\n 'core dimension in the function signature')\n\n if kwargs:\n func = functools.partial(func, **kwargs)\n\n if vectorize:\n if signature.all_core_dims:\n # we need the signature argument\n if LooseVersion(np.__version__) < '1.12': # pragma: no cover\n raise NotImplementedError(\n 'numpy 1.12 or newer required when using vectorize=True '\n 'in xarray.apply_ufunc with non-scalar output core '\n 'dimensions.')\n func = np.vectorize(func,\n otypes=output_dtypes,\n signature=signature.to_gufunc_string())\n else:\n func = np.vectorize(func, otypes=output_dtypes)\n\n variables_vfunc = functools.partial(apply_variable_ufunc, func,\n signature=signature,\n exclude_dims=exclude_dims,\n keep_attrs=keep_attrs,\n dask=dask,\n output_dtypes=output_dtypes,\n output_sizes=output_sizes)\n\n if any(isinstance(a, GroupBy) for a in args):\n this_apply = functools.partial(apply_ufunc, func,\n input_core_dims=input_core_dims,\n output_core_dims=output_core_dims,\n exclude_dims=exclude_dims,\n join=join,\n dataset_join=dataset_join,\n dataset_fill_value=dataset_fill_value,\n keep_attrs=keep_attrs,\n dask=dask)\n return apply_groupby_func(this_apply, *args)\n elif any(is_dict_like(a) for a in args):\n return apply_dataset_vfunc(variables_vfunc, *args,\n signature=signature,\n join=join,\n exclude_dims=exclude_dims,\n dataset_join=dataset_join,\n fill_value=dataset_fill_value,\n keep_attrs=keep_attrs)\n elif any(isinstance(a, DataArray) for a in args):\n return apply_dataarray_vfunc(variables_vfunc, *args,\n signature=signature,\n join=join,\n exclude_dims=exclude_dims,\n keep_attrs=keep_attrs)\n elif any(isinstance(a, Variable) for a in args):\n return variables_vfunc(*args)\n else:\n return apply_array_ufunc(func, *args, dask=dask)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngeneralize dot product for xarray objects.", "response": "def dot(*arrays, dims=None, **kwargs):\n \"\"\"Generalized dot product for xarray objects. Like np.einsum, but\n provides a simpler interface based on array dimensions.\n\n Parameters\n ----------\n arrays: DataArray (or Variable) objects\n Arrays to compute.\n dims: str or tuple of strings, optional\n Which dimensions to sum over.\n If not speciified, then all the common dimensions are summed over.\n **kwargs: dict\n Additional keyword arguments passed to numpy.einsum or\n dask.array.einsum\n\n Returns\n -------\n dot: DataArray\n\n Examples\n --------\n\n >>> da_a = xr.DataArray(np.arange(3 * 4).reshape(3, 4), dims=['a', 'b'])\n >>> da_b = xr.DataArray(np.arange(3 * 4 * 5).reshape(3, 4, 5),\n >>> dims=['a', 'b', 'c'])\n >>> da_c = xr.DataArray(np.arange(5 * 6).reshape(5, 6), dims=['c', 'd'])\n >>>\n >>> xr.dot(da_a, da_b, dims=['a', 'b']).dims\n ('c', )\n >>> xr.dot(da_a, da_b, dims=['a']).dims\n ('b', 'c')\n >>> xr.dot(da_a, da_b, da_c, dims=['b', 'c']).dims\n ('a', 'd')\n \"\"\"\n from .dataarray import DataArray\n from .variable import Variable\n\n if any(not isinstance(arr, (Variable, DataArray)) for arr in arrays):\n raise TypeError('Only xr.DataArray and xr.Variable are supported.'\n 'Given {}.'.format([type(arr) for arr in arrays]))\n\n if len(arrays) == 0:\n raise TypeError('At least one array should be given.')\n\n if isinstance(dims, str):\n dims = (dims, )\n\n common_dims = set.intersection(*[set(arr.dims) for arr in arrays])\n all_dims = []\n for arr in arrays:\n all_dims += [d for d in arr.dims if d not in all_dims]\n\n einsum_axes = 'abcdefghijklmnopqrstuvwxyz'\n dim_map = {d: einsum_axes[i] for i, d in enumerate(all_dims)}\n\n if dims is None:\n # find dimensions that occur more than one times\n dim_counts = Counter()\n for arr in arrays:\n dim_counts.update(arr.dims)\n dims = tuple(d for d, c in dim_counts.items() if c > 1)\n\n dims = tuple(dims) # make dims a tuple\n\n # dimensions to be parallelized\n broadcast_dims = tuple(d for d in all_dims\n if d in common_dims and d not in dims)\n input_core_dims = [[d for d in arr.dims if d not in broadcast_dims]\n for arr in arrays]\n output_core_dims = [tuple(d for d in all_dims if d not in\n dims + broadcast_dims)]\n\n # older dask than 0.17.4, we use tensordot if possible.\n if isinstance(arr.data, dask_array_type):\n import dask\n if LooseVersion(dask.__version__) < LooseVersion('0.17.4'):\n if len(broadcast_dims) == 0 and len(arrays) == 2:\n axes = [[arr.get_axis_num(d) for d in arr.dims if d in dims]\n for arr in arrays]\n return apply_ufunc(duck_array_ops.tensordot, *arrays,\n dask='allowed',\n input_core_dims=input_core_dims,\n output_core_dims=output_core_dims,\n kwargs={'axes': axes})\n\n # construct einsum subscripts, such as '...abc,...ab->...c'\n # Note: input_core_dims are always moved to the last position\n subscripts_list = ['...' + ''.join([dim_map[d] for d in ds]) for ds\n in input_core_dims]\n subscripts = ','.join(subscripts_list)\n subscripts += '->...' + ''.join([dim_map[d] for d in output_core_dims[0]])\n\n # subscripts should be passed to np.einsum as arg, not as kwargs. We need\n # to construct a partial function for apply_ufunc to work.\n func = functools.partial(duck_array_ops.einsum, subscripts, **kwargs)\n result = apply_ufunc(func, *arrays,\n input_core_dims=input_core_dims,\n output_core_dims=output_core_dims,\n dask='allowed')\n return result.transpose(*[d for d in all_dims if d in result.dims])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns elements from x or y depending on cond.", "response": "def where(cond, x, y):\n \"\"\"Return elements from `x` or `y` depending on `cond`.\n\n Performs xarray-like broadcasting across input arguments.\n\n Parameters\n ----------\n cond : scalar, array, Variable, DataArray or Dataset with boolean dtype\n When True, return values from `x`, otherwise returns values from `y`.\n x, y : scalar, array, Variable, DataArray or Dataset\n Values from which to choose. All dimension coordinates on these objects\n must be aligned with each other and with `cond`.\n\n Returns\n -------\n In priority order: Dataset, DataArray, Variable or array, whichever\n type appears as an input argument.\n\n Examples\n --------\n\n >>> cond = xr.DataArray([True, False], dims=['x'])\n >>> x = xr.DataArray([1, 2], dims=['y'])\n >>> xr.where(cond, x, 0)\n \n array([[1, 2],\n [0, 0]])\n Dimensions without coordinates: x, y\n\n See also\n --------\n numpy.where : corresponding numpy function\n Dataset.where, DataArray.where : equivalent methods\n \"\"\"\n # alignment for three arguments is complicated, so don't support it yet\n return apply_ufunc(duck_array_ops.where,\n cond, x, y,\n join='exact',\n dataset_join='exact',\n dask='allowed')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_gufunc_string(self):\n all_dims = self.all_core_dims\n dims_map = dict(zip(sorted(all_dims), range(len(all_dims))))\n input_core_dims = [['dim%d' % dims_map[dim] for dim in core_dims]\n for core_dims in self.input_core_dims]\n output_core_dims = [['dim%d' % dims_map[dim] for dim in core_dims]\n for core_dims in self.output_core_dims]\n alt_signature = type(self)(input_core_dims, output_core_dims)\n return str(alt_signature)", "response": "Create an equivalent signature string for a NumPy gufunc.\n object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns indices for an inverse permutation.", "response": "def inverse_permutation(indices):\n \"\"\"Return indices for an inverse permutation.\n\n Parameters\n ----------\n indices : 1D np.ndarray with dtype=int\n Integer positions to assign elements to.\n\n Returns\n -------\n inverse_permutation : 1D np.ndarray with dtype=int\n Integer indices to take from the original array to create the\n permutation.\n \"\"\"\n # use intp instead of int64 because of windows :(\n inverse_permutation = np.empty(len(indices), dtype=np.intp)\n inverse_permutation[indices] = np.arange(len(indices), dtype=np.intp)\n return inverse_permutation"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _is_contiguous(positions):\n previous = positions[0]\n for current in positions[1:]:\n if current != previous + 1:\n return False\n previous = current\n return True", "response": "Given a non - empty list returns True if all of the elements in the list are contiguous integers."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _advanced_indexer_subspaces(key):\n if not isinstance(key, tuple):\n key = (key,)\n advanced_index_positions = [i for i, k in enumerate(key)\n if not isinstance(k, slice)]\n\n if (not advanced_index_positions or\n not _is_contiguous(advanced_index_positions)):\n # Nothing to reorder: dimensions on the indexing result are already\n # ordered like vindex. See NumPy's rule for \"Combining advanced and\n # basic indexing\":\n # https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#combining-advanced-and-basic-indexing\n return (), ()\n\n non_slices = [k for k in key if not isinstance(k, slice)]\n ndim = len(np.broadcast(*non_slices).shape)\n mixed_positions = advanced_index_positions[0] + np.arange(ndim)\n vindex_positions = np.arange(ndim)\n return mixed_positions, vindex_positions", "response": "Indices of the advanced indexes subspaces for mixed indexing and vindex."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nroll window with padding.", "response": "def rolling_window(a, axis, window, center, fill_value):\n \"\"\" rolling window with padding. \"\"\"\n pads = [(0, 0) for s in a.shape]\n if center:\n start = int(window / 2) # 10 -> 5, 9 -> 4\n end = window - 1 - start\n pads[axis] = (start, end)\n else:\n pads[axis] = (window - 1, 0)\n a = np.pad(a, pads, mode='constant', constant_values=fill_value)\n return _rolling_window(a, window, axis)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns an array_like object with a rolling window along an axis.", "response": "def _rolling_window(a, window, axis=-1):\n \"\"\"\n Make an ndarray with a rolling window along axis.\n\n Parameters\n ----------\n a : array_like\n Array to add rolling window to\n axis: int\n axis position along which rolling window will be applied.\n window : int\n Size of rolling window\n\n Returns\n -------\n Array that is a view of the original array with a added dimension\n of size w.\n\n Examples\n --------\n >>> x=np.arange(10).reshape((2,5))\n >>> np.rolling_window(x, 3, axis=-1)\n array([[[0, 1, 2], [1, 2, 3], [2, 3, 4]],\n [[5, 6, 7], [6, 7, 8], [7, 8, 9]]])\n\n Calculate rolling mean of last dimension:\n >>> np.mean(np.rolling_window(x, 3, axis=-1), -1)\n array([[ 1., 2., 3.],\n [ 6., 7., 8.]])\n\n This function is taken from https://github.com/numpy/numpy/pull/31\n but slightly modified to accept axis option.\n \"\"\"\n axis = _validate_axis(a, axis)\n a = np.swapaxes(a, axis, -1)\n\n if window < 1:\n raise ValueError(\n \"`window` must be at least 1. Given : {}\".format(window))\n if window > a.shape[-1]:\n raise ValueError(\"`window` is too long. Given : {}\".format(window))\n\n shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)\n strides = a.strides + (a.strides[-1],)\n rolling = np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides,\n writeable=False)\n return np.swapaxes(rolling, -2, axis)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_virtual_variable(variables, key, level_vars=None, dim_sizes=None):\n if level_vars is None:\n level_vars = {}\n if dim_sizes is None:\n dim_sizes = {}\n\n if key in dim_sizes:\n data = pd.Index(range(dim_sizes[key]), name=key)\n variable = IndexVariable((key,), data)\n return key, key, variable\n\n if not isinstance(key, str):\n raise KeyError(key)\n\n split_key = key.split('.', 1)\n if len(split_key) == 2:\n ref_name, var_name = split_key\n elif len(split_key) == 1:\n ref_name, var_name = key, None\n else:\n raise KeyError(key)\n\n if ref_name in level_vars:\n dim_var = variables[level_vars[ref_name]]\n ref_var = dim_var.to_index_variable().get_level_variable(ref_name)\n else:\n ref_var = variables[ref_name]\n\n if var_name is None:\n virtual_var = ref_var\n var_name = key\n else:\n if _contains_datetime_like_objects(ref_var):\n ref_var = xr.DataArray(ref_var)\n data = getattr(ref_var.dt, var_name).data\n else:\n data = getattr(ref_var, var_name).data\n virtual_var = Variable(ref_var.dims, data)\n\n return ref_name, var_name, virtual_var", "response": "Get a virtual variable from a dict of xarray. Variable objects or MultiIndex variables."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the dimensions corresponding to a set of variables.", "response": "def calculate_dimensions(variables):\n \"\"\"Calculate the dimensions corresponding to a set of variables.\n\n Returns dictionary mapping from dimension names to sizes. Raises ValueError\n if any of the dimension sizes conflict.\n \"\"\"\n dims = OrderedDict()\n last_used = {}\n scalar_vars = set(k for k, v in variables.items() if not v.dims)\n for k, var in variables.items():\n for dim, size in zip(var.dims, var.shape):\n if dim in scalar_vars:\n raise ValueError('dimension %r already exists as a scalar '\n 'variable' % dim)\n if dim not in dims:\n dims[dim] = size\n last_used[dim] = k\n elif dims[dim] != size:\n raise ValueError('conflicting sizes for dimension %r: '\n 'length %s on %r and length %s on %r' %\n (dim, size, k, dims[dim], last_used[dim]))\n return dims"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmerge the given indexes into a single multi - index.", "response": "def merge_indexes(\n indexes, # type: Dict[Any, Union[Any, List[Any]]]\n variables, # type: Dict[Any, Variable]\n coord_names, # type: Set\n append=False, # type: bool\n):\n # type: (...) -> Tuple[OrderedDict[Any, Variable], Set]\n \"\"\"Merge variables into multi-indexes.\n\n Not public API. Used in Dataset and DataArray set_index\n methods.\n \"\"\"\n vars_to_replace = {} # Dict[Any, Variable]\n vars_to_remove = [] # type: list\n\n for dim, var_names in indexes.items():\n if isinstance(var_names, str):\n var_names = [var_names]\n\n names, codes, levels = [], [], [] # type: (list, list, list)\n current_index_variable = variables.get(dim)\n\n for n in var_names:\n var = variables[n]\n if (current_index_variable is not None and\n var.dims != current_index_variable.dims):\n raise ValueError(\n \"dimension mismatch between %r %s and %r %s\"\n % (dim, current_index_variable.dims, n, var.dims))\n\n if current_index_variable is not None and append:\n current_index = current_index_variable.to_index()\n if isinstance(current_index, pd.MultiIndex):\n try:\n current_codes = current_index.codes\n except AttributeError:\n # fpr pandas<0.24\n current_codes = current_index.labels\n names.extend(current_index.names)\n codes.extend(current_codes)\n levels.extend(current_index.levels)\n else:\n names.append('%s_level_0' % dim)\n cat = pd.Categorical(current_index.values, ordered=True)\n codes.append(cat.codes)\n levels.append(cat.categories)\n\n if not len(names) and len(var_names) == 1:\n idx = pd.Index(variables[var_names[0]].values)\n\n else:\n for n in var_names:\n names.append(n)\n var = variables[n]\n cat = pd.Categorical(var.values, ordered=True)\n codes.append(cat.codes)\n levels.append(cat.categories)\n\n idx = pd.MultiIndex(levels, codes, names=names)\n\n vars_to_replace[dim] = IndexVariable(dim, idx)\n vars_to_remove.extend(var_names)\n\n new_variables = OrderedDict([(k, v) for k, v in variables.items()\n if k not in vars_to_remove])\n new_variables.update(vars_to_replace)\n new_coord_names = coord_names | set(vars_to_replace)\n new_coord_names -= set(vars_to_remove)\n\n return new_variables, new_coord_names"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef split_indexes(\n dims_or_levels, # type: Union[Any, List[Any]]\n variables, # type: OrderedDict[Any, Variable]\n coord_names, # type: Set\n level_coords, # type: Dict[Any, Any]\n drop=False, # type: bool\n):\n # type: (...) -> Tuple[OrderedDict[Any, Variable], Set]\n \"\"\"Extract (multi-)indexes (levels) as variables.\n\n Not public API. Used in Dataset and DataArray reset_index\n methods.\n \"\"\"\n if isinstance(dims_or_levels, str):\n dims_or_levels = [dims_or_levels]\n\n dim_levels = defaultdict(list) # type: Dict[Any, list]\n dims = []\n for k in dims_or_levels:\n if k in level_coords:\n dim_levels[level_coords[k]].append(k)\n else:\n dims.append(k)\n\n vars_to_replace = {}\n vars_to_create = OrderedDict() # type: OrderedDict[Any, Variable]\n vars_to_remove = []\n\n for d in dims:\n index = variables[d].to_index()\n if isinstance(index, pd.MultiIndex):\n dim_levels[d] = index.names\n else:\n vars_to_remove.append(d)\n if not drop:\n vars_to_create[d + '_'] = Variable(d, index)\n\n for d, levs in dim_levels.items():\n index = variables[d].to_index()\n if len(levs) == index.nlevels:\n vars_to_remove.append(d)\n else:\n vars_to_replace[d] = IndexVariable(d, index.droplevel(levs))\n\n if not drop:\n for lev in levs:\n idx = index.get_level_values(lev)\n vars_to_create[idx.name] = Variable(d, idx)\n\n new_variables = variables.copy()\n for v in set(vars_to_remove):\n del new_variables[v]\n new_variables.update(vars_to_replace)\n new_variables.update(vars_to_create)\n new_coord_names = (coord_names | set(vars_to_create)) - set(vars_to_remove)\n\n return new_variables, new_coord_names", "response": "Splits the indexes of a single object into two sets of variables."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef as_dataset(obj):\n if hasattr(obj, 'to_dataset'):\n obj = obj.to_dataset()\n if not isinstance(obj, Dataset):\n obj = Dataset(obj)\n return obj", "response": "Cast the given object to a Dataset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _ipython_key_completions_(self):\n return [key for key in self._dataset._ipython_key_completions_()\n if key not in self._dataset._coord_names]", "response": "Provide method for the key - completer in IPython."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _set_init_vars_and_dims(self, data_vars, coords, compat):\n both_data_and_coords = [k for k in data_vars if k in coords]\n if both_data_and_coords:\n raise ValueError('variables %r are found in both data_vars and '\n 'coords' % both_data_and_coords)\n\n if isinstance(coords, Dataset):\n coords = coords.variables\n\n variables, coord_names, dims = merge_data_and_coords(\n data_vars, coords, compat=compat)\n\n self._variables = variables\n self._coord_names = coord_names\n self._dims = dims", "response": "Set the initial value of Dataset variables and dimensions\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new dataset from the contents of a backends. DataStore object.", "response": "def load_store(cls, store, decoder=None):\n \"\"\"Create a new dataset from the contents of a backends.*DataStore\n object\n \"\"\"\n variables, attributes = store.load()\n if decoder:\n variables, attributes = decoder(variables, attributes)\n obj = cls(variables, attrs=attributes)\n obj._file_obj = store\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _persist_inplace(self, **kwargs):\n # access .data to coerce everything to numpy or dask arrays\n lazy_data = {k: v._data for k, v in self.variables.items()\n if isinstance(v._data, dask_array_type)}\n if lazy_data:\n import dask\n\n # evaluate all the dask arrays simultaneously\n evaluated_data = dask.persist(*lazy_data.values(), **kwargs)\n\n for k, data in zip(lazy_data, evaluated_data):\n self.variables[k].data = data\n\n return self", "response": "Persist all the Dask arrays in memory"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef persist(self: T, **kwargs) -> T:\n new = self.copy(deep=False)\n return new._persist_inplace(**kwargs)", "response": "Persist the current state of the object to the distributed memory."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconstruct a new object of the same class with the given variables coord_names dims attrs indexes and file_obj.", "response": "def _construct_direct(cls, variables, coord_names, dims, attrs=None,\n indexes=None, encoding=None, file_obj=None):\n \"\"\"Shortcut around __init__ for internal use when we want to skip\n costly validation\n \"\"\"\n obj = object.__new__(cls)\n obj._variables = variables\n obj._coord_names = coord_names\n obj._dims = dims\n obj._indexes = indexes\n obj._attrs = attrs\n obj._file_obj = file_obj\n obj._encoding = encoding\n obj._initialized = True\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreplace variables with recalculated dimensions.", "response": "def _replace_with_new_dims( # type: ignore\n self: T,\n variables: 'OrderedDict[Any, Variable]' = None,\n coord_names: set = None,\n attrs: 'Optional[OrderedDict]' = __default,\n indexes: 'Optional[OrderedDict[Any, pd.Index]]' = __default,\n inplace: bool = False,\n ) -> T:\n \"\"\"Replace variables with recalculated dimensions.\"\"\"\n dims = dict(calculate_dimensions(variables))\n return self._replace(\n variables, coord_names, dims, attrs, indexes, inplace=inplace)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a shallow copy of the current dataset.", "response": "def copy(self: T, deep: bool = False, data: Mapping = None) -> T:\n \"\"\"Returns a copy of this dataset.\n\n If `deep=True`, a deep copy is made of each of the component variables.\n Otherwise, a shallow copy of each of the component variable is made, so\n that the underlying memory region of the new dataset is the same as in\n the original dataset.\n\n Use `data` to create a new object with the same structure as\n original but entirely new data.\n\n Parameters\n ----------\n deep : bool, optional\n Whether each component variable is loaded into memory and copied onto\n the new object. Default is False.\n data : dict-like, optional\n Data to use in the new object. Each item in `data` must have same\n shape as corresponding data variable in original. When `data` is\n used, `deep` is ignored for the data variables and only used for\n coords.\n\n Returns\n -------\n object : Dataset\n New object with dimensions, attributes, coordinates, name, encoding,\n and optionally data copied from original.\n\n Examples\n --------\n\n Shallow copy versus deep copy\n\n >>> da = xr.DataArray(np.random.randn(2, 3))\n >>> ds = xr.Dataset({'foo': da, 'bar': ('x', [-1, 2])},\n coords={'x': ['one', 'two']})\n >>> ds.copy()\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Coordinates:\n * x (x) >> ds_0 = ds.copy(deep=False)\n >>> ds_0['foo'][0, 0] = 7\n >>> ds_0\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Coordinates:\n * x (x) >> ds\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Coordinates:\n * x (x) >> ds.copy(data={'foo': np.arange(6).reshape(2, 3), 'bar': ['a', 'b']})\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Coordinates:\n * x (x) >> ds\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Coordinates:\n * x (x) T:\n \"\"\"Create a new Dataset with the listed variables from this dataset and\n the all relevant coordinates. Skips all validation.\n \"\"\"\n variables = OrderedDict() # type: OrderedDict[Any, Variable]\n coord_names = set()\n indexes = OrderedDict() # type: OrderedDict[Any, pd.Index]\n\n for name in names:\n try:\n variables[name] = self._variables[name]\n except KeyError:\n ref_name, var_name, var = _get_virtual_variable(\n self._variables, name, self._level_coords, self.dims)\n variables[var_name] = var\n if ref_name in self._coord_names or ref_name in self.dims:\n coord_names.add(var_name)\n if (var_name,) == var.dims:\n indexes[var_name] = var.to_index()\n\n needed_dims = set() # type: set\n for v in variables.values():\n needed_dims.update(v.dims)\n\n dims = dict((k, self.dims[k]) for k in needed_dims)\n\n for k in self._coord_names:\n if set(self.variables[k].dims) <= needed_dims:\n variables[k] = self._variables[k]\n coord_names.add(k)\n if k in self.indexes:\n indexes[k] = self.indexes[k]\n\n return self._replace(variables, coord_names, dims, indexes=indexes)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconstructs a DataArray by indexing this dataset", "response": "def _construct_dataarray(self, name) -> 'DataArray':\n \"\"\"Construct a DataArray by indexing this dataset\n \"\"\"\n from .dataarray import DataArray\n\n try:\n variable = self._variables[name]\n except KeyError:\n _, name, variable = _get_virtual_variable(\n self._variables, name, self._level_coords, self.dims)\n\n needed_dims = set(variable.dims)\n\n coords = OrderedDict() # type: OrderedDict[Any, Variable]\n for k in self.coords:\n if set(self.variables[k].dims) <= needed_dims:\n coords[k] = self.variables[k]\n\n if self._indexes is None:\n indexes = None\n else:\n indexes = OrderedDict((k, v) for k, v in self._indexes.items()\n if k in coords)\n\n return DataArray(variable, coords, name=name, indexes=indexes,\n fastpath=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nlisting of items to look - up for key - completion", "response": "def _item_sources(self):\n \"\"\"List of places to look-up items for key-completion\"\"\"\n return [self.data_vars, self.coords, {d: self[d] for d in self.dims},\n LevelCoordinatesSource(self)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmapping of pandas. Index objects used for label based indexing.", "response": "def indexes(self) -> 'Mapping[Any, pd.Index]':\n \"\"\"Mapping of pandas.Index objects used for label based indexing\n \"\"\"\n if self._indexes is None:\n self._indexes = default_indexes(self._variables, self._dims)\n return Indexes(self._indexes)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_coords(self, names, inplace=None):\n # TODO: allow inserting new coordinates with this method, like\n # DataFrame.set_index?\n # nb. check in self._variables, not self.data_vars to insure that the\n # operation is idempotent\n inplace = _check_inplace(inplace)\n if isinstance(names, str):\n names = [names]\n self._assert_all_in_dataset(names)\n obj = self if inplace else self.copy()\n obj._coord_names.update(names)\n return obj", "response": "Set the names of one or more variables as coordinates in this dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reset_coords(self, names=None, drop=False, inplace=None):\n inplace = _check_inplace(inplace)\n if names is None:\n names = self._coord_names - set(self.dims)\n else:\n if isinstance(names, str):\n names = [names]\n self._assert_all_in_dataset(names)\n bad_coords = set(names) & set(self.dims)\n if bad_coords:\n raise ValueError(\n 'cannot remove index coordinates with reset_coords: %s'\n % bad_coords)\n obj = self if inplace else self.copy()\n obj._coord_names.difference_update(names)\n if drop:\n for name in names:\n del obj._variables[name]\n return obj", "response": "Reset the coordinates of the specified names of variables in this dataset to become variables."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dump_to_store(self, store, **kwargs):\n from ..backends.api import dump_to_store\n # TODO: rename and/or cleanup this method to make it more consistent\n # with to_netcdf()\n return dump_to_store(self, store, **kwargs)", "response": "Store dataset contents to a backends. DataStore object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_netcdf(self, path=None, mode='w', format=None, group=None,\n engine=None, encoding=None, unlimited_dims=None,\n compute=True):\n \"\"\"Write dataset contents to a netCDF file.\n\n Parameters\n ----------\n path : str, Path or file-like object, optional\n Path to which to save this dataset. File-like objects are only\n supported by the scipy engine. If no path is provided, this\n function returns the resulting netCDF file as bytes; in this case,\n we need to use scipy, which does not support netCDF version 4 (the\n default format becomes NETCDF3_64BIT).\n mode : {'w', 'a'}, optional\n Write ('w') or append ('a') mode. If mode='w', any existing file at\n this location will be overwritten. If mode='a', existing variables\n will be overwritten.\n format : {'NETCDF4', 'NETCDF4_CLASSIC', 'NETCDF3_64BIT',\n 'NETCDF3_CLASSIC'}, optional\n File format for the resulting netCDF file:\n\n * NETCDF4: Data is stored in an HDF5 file, using netCDF4 API\n features.\n * NETCDF4_CLASSIC: Data is stored in an HDF5 file, using only\n netCDF 3 compatible API features.\n * NETCDF3_64BIT: 64-bit offset version of the netCDF 3 file format,\n which fully supports 2+ GB files, but is only compatible with\n clients linked against netCDF version 3.6.0 or later.\n * NETCDF3_CLASSIC: The classic netCDF 3 file format. It does not\n handle 2+ GB files very well.\n\n All formats are supported by the netCDF4-python library.\n scipy.io.netcdf only supports the last two formats.\n\n The default format is NETCDF4 if you are saving a file to disk and\n have the netCDF4-python library available. Otherwise, xarray falls\n back to using scipy to write netCDF files and defaults to the\n NETCDF3_64BIT format (scipy does not support netCDF4).\n group : str, optional\n Path to the netCDF4 group in the given file to open (only works for\n format='NETCDF4'). The group(s) will be created if necessary.\n engine : {'netcdf4', 'scipy', 'h5netcdf'}, optional\n Engine to use when writing netCDF files. If not provided, the\n default engine is chosen based on available dependencies, with a\n preference for 'netcdf4' if writing to a file on disk.\n encoding : dict, optional\n Nested dictionary with variable names as keys and dictionaries of\n variable specific encodings as values, e.g.,\n ``{'my_variable': {'dtype': 'int16', 'scale_factor': 0.1,\n 'zlib': True}, ...}``\n\n The `h5netcdf` engine supports both the NetCDF4-style compression\n encoding parameters ``{'zlib': True, 'complevel': 9}`` and the h5py\n ones ``{'compression': 'gzip', 'compression_opts': 9}``.\n This allows using any compression plugin installed in the HDF5\n library, e.g. LZF.\n\n unlimited_dims : sequence of str, optional\n Dimension(s) that should be serialized as unlimited dimensions.\n By default, no dimensions are treated as unlimited dimensions.\n Note that unlimited_dims may also be set via\n ``dataset.encoding['unlimited_dims']``.\n compute: boolean\n If true compute immediately, otherwise return a\n ``dask.delayed.Delayed`` object that can be computed later.\n \"\"\"\n if encoding is None:\n encoding = {}\n from ..backends.api import to_netcdf\n return to_netcdf(self, path, mode, format=format, group=group,\n engine=engine, encoding=encoding,\n unlimited_dims=unlimited_dims,\n compute=compute)", "response": "Write the contents of this object to a netCDF file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_zarr(self, store=None, mode='w-', synchronizer=None, group=None,\n encoding=None, compute=True, consolidated=False):\n \"\"\"Write dataset contents to a zarr group.\n\n .. note:: Experimental\n The Zarr backend is new and experimental. Please report any\n unexpected behavior via github issues.\n\n Parameters\n ----------\n store : MutableMapping or str, optional\n Store or path to directory in file system.\n mode : {'w', 'w-'}\n Persistence mode: 'w' means create (overwrite if exists);\n 'w-' means create (fail if exists).\n synchronizer : object, optional\n Array synchronizer\n group : str, obtional\n Group path. (a.k.a. `path` in zarr terminology.)\n encoding : dict, optional\n Nested dictionary with variable names as keys and dictionaries of\n variable specific encodings as values, e.g.,\n ``{'my_variable': {'dtype': 'int16', 'scale_factor': 0.1,}, ...}``\n compute: bool, optional\n If True compute immediately, otherwise return a\n ``dask.delayed.Delayed`` object that can be computed later.\n consolidated: bool, optional\n If True, apply zarr's `consolidate_metadata` function to the store\n after writing.\n\n References\n ----------\n https://zarr.readthedocs.io/\n \"\"\"\n if encoding is None:\n encoding = {}\n if mode not in ['w', 'w-']:\n # TODO: figure out how to handle 'r+' and 'a'\n raise ValueError(\"The only supported options for mode are 'w' \"\n \"and 'w-'.\")\n from ..backends.api import to_zarr\n return to_zarr(self, store=store, mode=mode, synchronizer=synchronizer,\n group=group, encoding=encoding, compute=compute,\n consolidated=consolidated)", "response": "Write the contents of this object to a zarr group."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef info(self, buf=None):\n\n if buf is None: # pragma: no cover\n buf = sys.stdout\n\n lines = []\n lines.append('xarray.Dataset {')\n lines.append('dimensions:')\n for name, size in self.dims.items():\n lines.append('\\t{name} = {size} ;'.format(name=name, size=size))\n lines.append('\\nvariables:')\n for name, da in self.variables.items():\n dims = ', '.join(da.dims)\n lines.append('\\t{type} {name}({dims}) ;'.format(\n type=da.dtype, name=name, dims=dims))\n for k, v in da.attrs.items():\n lines.append('\\t\\t{name}:{k} = {v} ;'.format(name=name, k=k,\n v=v))\n lines.append('\\n// global attributes:')\n for k, v in self.attrs.items():\n lines.append('\\t:{k} = {v} ;'.format(k=k, v=v))\n lines.append('}')\n\n buf.write('\\n'.join(lines))", "response": "Prints out a summary of the current state of the object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef chunks(self):\n chunks = {}\n for v in self.variables.values():\n if v.chunks is not None:\n for dim, c in zip(v.dims, v.chunks):\n if dim in chunks and c != chunks[dim]:\n raise ValueError('inconsistent chunks')\n chunks[dim] = c\n return Frozen(SortedKeysDict(chunks))", "response": "Returns a Frozen dictionary of the block dimensions for this dataset s data or None if it s not a dask\n array."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncoerces all arrays in this dataset into dask arrays with the given chunk sizes along each dimension.", "response": "def chunk(self, chunks=None, name_prefix='xarray-', token=None,\n lock=False):\n \"\"\"Coerce all arrays in this dataset into dask arrays with the given\n chunks.\n\n Non-dask arrays in this dataset will be converted to dask arrays. Dask\n arrays will be rechunked to the given chunk sizes.\n\n If neither chunks is not provided for one or more dimensions, chunk\n sizes along that dimension will not be updated; non-dask arrays will be\n converted into dask arrays with a single block.\n\n Parameters\n ----------\n chunks : int or dict, optional\n Chunk sizes along each dimension, e.g., ``5`` or\n ``{'x': 5, 'y': 5}``.\n name_prefix : str, optional\n Prefix for the name of any new dask arrays.\n token : str, optional\n Token uniquely identifying this dataset.\n lock : optional\n Passed on to :py:func:`dask.array.from_array`, if the array is not\n already as dask array.\n\n Returns\n -------\n chunked : xarray.Dataset\n \"\"\"\n try:\n from dask.base import tokenize\n except ImportError:\n # raise the usual error if dask is entirely missing\n import dask # noqa\n raise ImportError('xarray requires dask version 0.9 or newer')\n\n if isinstance(chunks, Number):\n chunks = dict.fromkeys(self.dims, chunks)\n\n if chunks is not None:\n bad_dims = [d for d in chunks if d not in self.dims]\n if bad_dims:\n raise ValueError('some chunks keys are not dimensions on this '\n 'object: %s' % bad_dims)\n\n def selkeys(dict_, keys):\n if dict_ is None:\n return None\n return dict((d, dict_[d]) for d in keys if d in dict_)\n\n def maybe_chunk(name, var, chunks):\n chunks = selkeys(chunks, var.dims)\n if not chunks:\n chunks = None\n if var.ndim > 0:\n token2 = tokenize(name, token if token else var._data)\n name2 = '%s%s-%s' % (name_prefix, name, token2)\n return var.chunk(chunks, name=name2, lock=lock)\n else:\n return var\n\n variables = OrderedDict([(k, maybe_chunk(k, v, chunks))\n for k, v in self.variables.items()])\n return self._replace(variables)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_indexers_coords_and_indexes(self, indexers):\n from .dataarray import DataArray\n\n coord_list = []\n indexes = OrderedDict()\n for k, v in indexers.items():\n if isinstance(v, DataArray):\n v_coords = v.coords\n if v.dtype.kind == 'b':\n if v.ndim != 1: # we only support 1-d boolean array\n raise ValueError(\n '{:d}d-boolean array is used for indexing along '\n 'dimension {!r}, but only 1d boolean arrays are '\n 'supported.'.format(v.ndim, k))\n # Make sure in case of boolean DataArray, its\n # coordinate also should be indexed.\n v_coords = v[v.values.nonzero()[0]].coords\n\n coord_list.append({d: v_coords[d].variable for d in v.coords})\n indexes.update(v.indexes)\n\n # we don't need to call align() explicitly or check indexes for\n # alignment, because merge_variables already checks for exact alignment\n # between dimension coordinates\n coords = merge_variables(coord_list)\n assert_coordinate_consistent(self, coords)\n\n # silently drop the conflicted variables.\n attached_coords = OrderedDict(\n (k, v) for k, v in coords.items() if k not in self._variables\n )\n attached_indexes = OrderedDict(\n (k, v) for k, v in indexes.items() if k not in self._variables\n )\n return attached_coords, attached_indexes", "response": "Extract coordinates from indexers. indexers returns an OrderedDict mapping from coordinate name to the corresponding coordinate variable."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a new Dataset with each array indexed along the specified dimension.", "response": "def isel(self, indexers=None, drop=False, **indexers_kwargs):\n \"\"\"Returns a new dataset with each array indexed along the specified\n dimension(s).\n\n This method selects values from each array using its `__getitem__`\n method, except this method does not require knowing the order of\n each array's dimensions.\n\n Parameters\n ----------\n indexers : dict, optional\n A dict with keys matching dimensions and values given\n by integers, slice objects or arrays.\n indexer can be a integer, slice, array-like or DataArray.\n If DataArrays are passed as indexers, xarray-style indexing will be\n carried out. See :ref:`indexing` for the details.\n One of indexers or indexers_kwargs must be provided.\n drop : bool, optional\n If ``drop=True``, drop coordinates variables indexed by integers\n instead of making them scalar.\n **indexers_kwarg : {dim: indexer, ...}, optional\n The keyword arguments form of ``indexers``.\n One of indexers or indexers_kwargs must be provided.\n\n Returns\n -------\n obj : Dataset\n A new Dataset with the same contents as this dataset, except each\n array and dimension is indexed by the appropriate indexers.\n If indexer DataArrays have coordinates that do not conflict with\n this object, then these coordinates will be attached.\n In general, each array's data will be a view of the array's data\n in this dataset, unless vectorized indexing was triggered by using\n an array indexer, in which case the data will be a copy.\n\n See Also\n --------\n Dataset.sel\n DataArray.isel\n \"\"\"\n\n indexers = either_dict_or_kwargs(indexers, indexers_kwargs, 'isel')\n\n indexers_list = self._validate_indexers(indexers)\n\n variables = OrderedDict()\n indexes = OrderedDict()\n for name, var in self.variables.items():\n var_indexers = {k: v for k, v in indexers_list if k in var.dims}\n if drop and name in var_indexers:\n continue # drop this variable\n\n if name in self.indexes:\n new_var, new_index = isel_variable_and_index(\n name, var, self.indexes[name], var_indexers)\n if new_index is not None:\n indexes[name] = new_index\n else:\n new_var = var.isel(indexers=var_indexers)\n\n variables[name] = new_var\n\n coord_names = set(variables).intersection(self._coord_names)\n selected = self._replace_with_new_dims(\n variables, coord_names, indexes)\n\n # Extract coordinates from indexers\n coord_vars, new_indexes = (\n selected._get_indexers_coords_and_indexes(indexers))\n variables.update(coord_vars)\n indexes.update(new_indexes)\n coord_names = (set(variables)\n .intersection(self._coord_names)\n .union(coord_vars))\n return self._replace_with_new_dims(\n variables, coord_names, indexes=indexes)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sel(self, indexers=None, method=None, tolerance=None, drop=False,\n **indexers_kwargs):\n \"\"\"Returns a new dataset with each array indexed by tick labels\n along the specified dimension(s).\n\n In contrast to `Dataset.isel`, indexers for this method should use\n labels instead of integers.\n\n Under the hood, this method is powered by using pandas's powerful Index\n objects. This makes label based indexing essentially just as fast as\n using integer indexing.\n\n It also means this method uses pandas's (well documented) logic for\n indexing. This means you can use string shortcuts for datetime indexes\n (e.g., '2000-01' to select all values in January 2000). It also means\n that slices are treated as inclusive of both the start and stop values,\n unlike normal Python indexing.\n\n Parameters\n ----------\n indexers : dict, optional\n A dict with keys matching dimensions and values given\n by scalars, slices or arrays of tick labels. For dimensions with\n multi-index, the indexer may also be a dict-like object with keys\n matching index level names.\n If DataArrays are passed as indexers, xarray-style indexing will be\n carried out. See :ref:`indexing` for the details.\n One of indexers or indexers_kwargs must be provided.\n method : {None, 'nearest', 'pad'/'ffill', 'backfill'/'bfill'}, optional\n Method to use for inexact matches (requires pandas>=0.16):\n\n * None (default): only exact matches\n * pad / ffill: propagate last valid index value forward\n * backfill / bfill: propagate next valid index value backward\n * nearest: use nearest valid index value\n tolerance : optional\n Maximum distance between original and new labels for inexact\n matches. The values of the index at the matching locations must\n satisfy the equation ``abs(index[indexer] - target) <= tolerance``.\n Requires pandas>=0.17.\n drop : bool, optional\n If ``drop=True``, drop coordinates variables in `indexers` instead\n of making them scalar.\n **indexers_kwarg : {dim: indexer, ...}, optional\n The keyword arguments form of ``indexers``.\n One of indexers or indexers_kwargs must be provided.\n\n Returns\n -------\n obj : Dataset\n A new Dataset with the same contents as this dataset, except each\n variable and dimension is indexed by the appropriate indexers.\n If indexer DataArrays have coordinates that do not conflict with\n this object, then these coordinates will be attached.\n In general, each array's data will be a view of the array's data\n in this dataset, unless vectorized indexing was triggered by using\n an array indexer, in which case the data will be a copy.\n\n\n See Also\n --------\n Dataset.isel\n DataArray.sel\n \"\"\"\n indexers = either_dict_or_kwargs(indexers, indexers_kwargs, 'sel')\n pos_indexers, new_indexes = remap_label_indexers(\n self, indexers=indexers, method=method, tolerance=tolerance)\n result = self.isel(indexers=pos_indexers, drop=drop)\n return result._overwrite_indexes(new_indexes)", "response": "Returns a new dataset with each array indexed by tick labels and optionally by tick labels."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef isel_points(self, dim='points', **indexers):\n # type: (...) -> Dataset\n \"\"\"Returns a new dataset with each array indexed pointwise along the\n specified dimension(s).\n\n This method selects pointwise values from each array and is akin to\n the NumPy indexing behavior of `arr[[0, 1], [0, 1]]`, except this\n method does not require knowing the order of each array's dimensions.\n\n Parameters\n ----------\n dim : str or DataArray or pandas.Index or other list-like object, optional\n Name of the dimension to concatenate along. If dim is provided as a\n string, it must be a new dimension name, in which case it is added\n along axis=0. If dim is provided as a DataArray or Index or\n list-like object, its name, which must not be present in the\n dataset, is used as the dimension to concatenate along and the\n values are added as a coordinate.\n **indexers : {dim: indexer, ...}\n Keyword arguments with names matching dimensions and values given\n by array-like objects. All indexers must be the same length and\n 1 dimensional.\n\n Returns\n -------\n obj : Dataset\n A new Dataset with the same contents as this dataset, except each\n array and dimension is indexed by the appropriate indexers. With\n pointwise indexing, the new Dataset will always be a copy of the\n original.\n\n See Also\n --------\n Dataset.sel\n Dataset.isel\n Dataset.sel_points\n DataArray.isel_points\n \"\"\" # noqa\n warnings.warn('Dataset.isel_points is deprecated: use Dataset.isel()'\n 'instead.', DeprecationWarning, stacklevel=2)\n\n indexer_dims = set(indexers)\n\n def take(variable, slices):\n # Note: remove helper function when once when numpy\n # supports vindex https://github.com/numpy/numpy/pull/6075\n if hasattr(variable.data, 'vindex'):\n # Special case for dask backed arrays to use vectorised list\n # indexing\n sel = variable.data.vindex[slices]\n else:\n # Otherwise assume backend is numpy array with 'fancy' indexing\n sel = variable.data[slices]\n return sel\n\n def relevant_keys(mapping):\n return [k for k, v in mapping.items()\n if any(d in indexer_dims for d in v.dims)]\n\n coords = relevant_keys(self.coords)\n indexers = [(k, np.asarray(v)) # type: ignore\n for k, v in indexers.items()]\n indexers_dict = dict(indexers)\n non_indexed_dims = set(self.dims) - indexer_dims\n non_indexed_coords = set(self.coords) - set(coords)\n\n # All the indexers should be iterables\n # Check that indexers are valid dims, integers, and 1D\n for k, v in indexers:\n if k not in self.dims:\n raise ValueError(\"dimension %s does not exist\" % k)\n if v.dtype.kind != 'i': # type: ignore\n raise TypeError('Indexers must be integers')\n if v.ndim != 1: # type: ignore\n raise ValueError('Indexers must be 1 dimensional')\n\n # all the indexers should have the same length\n lengths = set(len(v) for k, v in indexers)\n if len(lengths) > 1:\n raise ValueError('All indexers must be the same length')\n\n # Existing dimensions are not valid choices for the dim argument\n if isinstance(dim, str):\n if dim in self.dims:\n # dim is an invalid string\n raise ValueError('Existing dimension names are not valid '\n 'choices for the dim argument in sel_points')\n\n elif hasattr(dim, 'dims'):\n # dim is a DataArray or Coordinate\n if dim.name in self.dims:\n # dim already exists\n raise ValueError('Existing dimensions are not valid choices '\n 'for the dim argument in sel_points')\n\n # Set the new dim_name, and optionally the new dim coordinate\n # dim is either an array-like or a string\n if not utils.is_scalar(dim):\n # dim is array like get name or assign 'points', get as variable\n dim_name = 'points' if not hasattr(dim, 'name') else dim.name\n dim_coord = as_variable(dim, name=dim_name)\n else:\n # dim is a string\n dim_name = dim\n dim_coord = None # type: ignore\n\n reordered = self.transpose(\n *(list(indexer_dims) + list(non_indexed_dims)))\n\n variables = OrderedDict() # type: ignore\n\n for name, var in reordered.variables.items():\n if name in indexers_dict or any(\n d in indexer_dims for d in var.dims):\n # slice if var is an indexer or depends on an indexed dim\n slc = [indexers_dict[k]\n if k in indexers_dict\n else slice(None) for k in var.dims]\n\n var_dims = [dim_name] + [d for d in var.dims\n if d in non_indexed_dims]\n selection = take(var, tuple(slc))\n var_subset = type(var)(var_dims, selection, var.attrs)\n variables[name] = var_subset\n else:\n # If not indexed just add it back to variables or coordinates\n variables[name] = var\n\n coord_names = (set(coords) & set(variables)) | non_indexed_coords\n\n dset = self._replace_vars_and_dims(variables, coord_names=coord_names)\n # Add the dim coord to the new dset. Must be done after creation\n # because_replace_vars_and_dims can only access existing coords,\n # not add new ones\n if dim_coord is not None:\n dset.coords[dim_name] = dim_coord\n return dset", "response": "Returns a new Dataset with each array indexed pointwise along the\n specified dimension."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a new Dataset with each array indexed pointwise by tick labels along the specified dimension.", "response": "def sel_points(self, dim='points', method=None, tolerance=None,\n **indexers):\n \"\"\"Returns a new dataset with each array indexed pointwise by tick\n labels along the specified dimension(s).\n\n In contrast to `Dataset.isel_points`, indexers for this method should\n use labels instead of integers.\n\n In contrast to `Dataset.sel`, this method selects points along the\n diagonal of multi-dimensional arrays, not the intersection.\n\n Parameters\n ----------\n dim : str or DataArray or pandas.Index or other list-like object, optional\n Name of the dimension to concatenate along. If dim is provided as a\n string, it must be a new dimension name, in which case it is added\n along axis=0. If dim is provided as a DataArray or Index or\n list-like object, its name, which must not be present in the\n dataset, is used as the dimension to concatenate along and the\n values are added as a coordinate.\n method : {None, 'nearest', 'pad'/'ffill', 'backfill'/'bfill'}, optional\n Method to use for inexact matches (requires pandas>=0.16):\n\n * None (default): only exact matches\n * pad / ffill: propagate last valid index value forward\n * backfill / bfill: propagate next valid index value backward\n * nearest: use nearest valid index value\n tolerance : optional\n Maximum distance between original and new labels for inexact\n matches. The values of the index at the matching locations must\n satisfy the equation ``abs(index[indexer] - target) <= tolerance``.\n Requires pandas>=0.17.\n **indexers : {dim: indexer, ...}\n Keyword arguments with names matching dimensions and values given\n by array-like objects. All indexers must be the same length and\n 1 dimensional.\n\n Returns\n -------\n obj : Dataset\n A new Dataset with the same contents as this dataset, except each\n array and dimension is indexed by the appropriate indexers. With\n pointwise indexing, the new Dataset will always be a copy of the\n original.\n\n See Also\n --------\n Dataset.sel\n Dataset.isel\n Dataset.isel_points\n DataArray.sel_points\n \"\"\" # noqa\n warnings.warn('Dataset.sel_points is deprecated: use Dataset.sel()'\n 'instead.', DeprecationWarning, stacklevel=2)\n\n pos_indexers, _ = indexing.remap_label_indexers(\n self, indexers, method=method, tolerance=tolerance\n )\n return self.isel_points(dim=dim, **pos_indexers)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reindex_like(self, other, method=None, tolerance=None, copy=True):\n indexers = alignment.reindex_like_indexers(self, other)\n return self.reindex(indexers=indexers, method=method, copy=copy,\n tolerance=tolerance)", "response": "Conform this object onto the indexes of another object filling missing values with NaN."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reindex(self, indexers=None, method=None, tolerance=None, copy=True,\n **indexers_kwargs):\n \"\"\"Conform this object onto a new set of indexes, filling in\n missing values with NaN.\n\n Parameters\n ----------\n indexers : dict. optional\n Dictionary with keys given by dimension names and values given by\n arrays of coordinates tick labels. Any mis-matched coordinate\n values will be filled in with NaN, and any mis-matched dimension\n names will simply be ignored.\n One of indexers or indexers_kwargs must be provided.\n method : {None, 'nearest', 'pad'/'ffill', 'backfill'/'bfill'}, optional\n Method to use for filling index values in ``indexers`` not found in\n this dataset:\n\n * None (default): don't fill gaps\n * pad / ffill: propagate last valid index value forward\n * backfill / bfill: propagate next valid index value backward\n * nearest: use nearest valid index value (requires pandas>=0.16)\n tolerance : optional\n Maximum distance between original and new labels for inexact\n matches. The values of the index at the matching locations must\n satisfy the equation ``abs(index[indexer] - target) <= tolerance``.\n Requires pandas>=0.17.\n copy : bool, optional\n If ``copy=True``, data in the return value is always copied. If\n ``copy=False`` and reindexing is unnecessary, or can be performed\n with only slice operations, then the output may share memory with\n the input. In either case, a new xarray object is always returned.\n **indexers_kwarg : {dim: indexer, ...}, optional\n Keyword arguments in the same form as ``indexers``.\n One of indexers or indexers_kwargs must be provided.\n\n Returns\n -------\n reindexed : Dataset\n Another dataset, with this dataset's data but replaced coordinates.\n\n See Also\n --------\n Dataset.reindex_like\n align\n pandas.Index.get_indexer\n \"\"\"\n indexers = utils.either_dict_or_kwargs(indexers, indexers_kwargs,\n 'reindex')\n\n bad_dims = [d for d in indexers if d not in self.dims]\n if bad_dims:\n raise ValueError('invalid reindex dimensions: %s' % bad_dims)\n\n variables, indexes = alignment.reindex_variables(\n self.variables, self.sizes, self.indexes, indexers, method,\n tolerance, copy=copy)\n coord_names = set(self._coord_names)\n coord_names.update(indexers)\n return self._replace_with_new_dims(\n variables, coord_names, indexes=indexes)", "response": "Conform this object onto a new set of indexes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef interp(self, coords=None, method='linear', assume_sorted=False,\n kwargs={}, **coords_kwargs):\n \"\"\" Multidimensional interpolation of Dataset.\n\n Parameters\n ----------\n coords : dict, optional\n Mapping from dimension names to the new coordinates.\n New coordinate can be a scalar, array-like or DataArray.\n If DataArrays are passed as new coordates, their dimensions are\n used for the broadcasting.\n method: string, optional.\n {'linear', 'nearest'} for multidimensional array,\n {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic'}\n for 1-dimensional array. 'linear' is used by default.\n assume_sorted: boolean, optional\n If False, values of coordinates that are interpolated over can be\n in any order and they are sorted first. If True, interpolated\n coordinates are assumed to be an array of monotonically increasing\n values.\n kwargs: dictionary, optional\n Additional keyword passed to scipy's interpolator.\n **coords_kwarg : {dim: coordinate, ...}, optional\n The keyword arguments form of ``coords``.\n One of coords or coords_kwargs must be provided.\n\n Returns\n -------\n interpolated: xr.Dataset\n New dataset on the new coordinates.\n\n Notes\n -----\n scipy is required.\n\n See Also\n --------\n scipy.interpolate.interp1d\n scipy.interpolate.interpn\n \"\"\"\n from . import missing\n\n coords = either_dict_or_kwargs(coords, coords_kwargs, 'interp')\n indexers = OrderedDict(self._validate_indexers(coords))\n\n obj = self if assume_sorted else self.sortby([k for k in coords])\n\n def maybe_variable(obj, k):\n # workaround to get variable for dimension without coordinate.\n try:\n return obj._variables[k]\n except KeyError:\n return as_variable((k, range(obj.dims[k])))\n\n def _validate_interp_indexer(x, new_x):\n # In the case of datetimes, the restrictions placed on indexers\n # used with interp are stronger than those which are placed on\n # isel, so we need an additional check after _validate_indexers.\n if (_contains_datetime_like_objects(x) and\n not _contains_datetime_like_objects(new_x)):\n raise TypeError('When interpolating over a datetime-like '\n 'coordinate, the coordinates to '\n 'interpolate to must be either datetime '\n 'strings or datetimes. '\n 'Instead got\\n{}'.format(new_x))\n else:\n return (x, new_x)\n\n variables = OrderedDict()\n for name, var in obj._variables.items():\n if name not in indexers:\n if var.dtype.kind in 'uifc':\n var_indexers = {\n k: _validate_interp_indexer(maybe_variable(obj, k), v)\n for k, v in indexers.items()\n if k in var.dims\n }\n variables[name] = missing.interp(\n var, var_indexers, method, **kwargs)\n elif all(d not in indexers for d in var.dims):\n # keep unrelated object array\n variables[name] = var\n\n coord_names = set(variables).intersection(obj._coord_names)\n indexes = OrderedDict(\n (k, v) for k, v in obj.indexes.items() if k not in indexers)\n selected = self._replace_with_new_dims(\n variables.copy(), coord_names, indexes=indexes)\n\n # attach indexer as coordinate\n variables.update(indexers)\n indexes.update(\n (k, v.to_index()) for k, v in indexers.items() if v.dims == (k,)\n )\n\n # Extract coordinates from indexers\n coord_vars, new_indexes = (\n selected._get_indexers_coords_and_indexes(coords))\n variables.update(coord_vars)\n indexes.update(new_indexes)\n\n coord_names = (set(variables)\n .intersection(obj._coord_names)\n .union(coord_vars))\n return self._replace_with_new_dims(\n variables, coord_names, indexes=indexes)", "response": "Interpolate a multidimensional array over a set of coordinates."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ninterpolate this object onto the coordinates of another object.", "response": "def interp_like(self, other, method='linear', assume_sorted=False,\n kwargs={}):\n \"\"\"Interpolate this object onto the coordinates of another object,\n filling the out of range values with NaN.\n\n Parameters\n ----------\n other : Dataset or DataArray\n Object with an 'indexes' attribute giving a mapping from dimension\n names to an 1d array-like, which provides coordinates upon\n which to index the variables in this dataset.\n method: string, optional.\n {'linear', 'nearest'} for multidimensional array,\n {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic'}\n for 1-dimensional array. 'linear' is used by default.\n assume_sorted: boolean, optional\n If False, values of coordinates that are interpolated over can be\n in any order and they are sorted first. If True, interpolated\n coordinates are assumed to be an array of monotonically increasing\n values.\n kwargs: dictionary, optional\n Additional keyword passed to scipy's interpolator.\n\n Returns\n -------\n interpolated: xr.Dataset\n Another dataset by interpolating this dataset's data along the\n coordinates of the other object.\n\n Notes\n -----\n scipy is required.\n If the dataset has object-type coordinates, reindex is used for these\n coordinates instead of the interpolation.\n\n See Also\n --------\n Dataset.interp\n Dataset.reindex_like\n \"\"\"\n coords = alignment.reindex_like_indexers(self, other)\n\n numeric_coords = OrderedDict()\n object_coords = OrderedDict()\n for k, v in coords.items():\n if v.dtype.kind in 'uifcMm':\n numeric_coords[k] = v\n else:\n object_coords[k] = v\n\n ds = self\n if object_coords:\n # We do not support interpolation along object coordinate.\n # reindex instead.\n ds = self.reindex(object_coords)\n return ds.interp(numeric_coords, method, assume_sorted, kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rename(self, name_dict=None, inplace=None, **names):\n # TODO: add separate rename_vars and rename_dims methods.\n inplace = _check_inplace(inplace)\n name_dict = either_dict_or_kwargs(name_dict, names, 'rename')\n for k, v in name_dict.items():\n if k not in self and k not in self.dims:\n raise ValueError(\"cannot rename %r because it is not a \"\n \"variable or dimension in this dataset\" % k)\n\n variables, coord_names, dims, indexes = self._rename_all(\n name_dict=name_dict, dim_dict=name_dict)\n return self._replace(variables, coord_names, dims=dims,\n indexes=indexes, inplace=inplace)", "response": "Returns a new Dataset object with renamed variables and dimensions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef swap_dims(self, dims_dict, inplace=None):\n # TODO: deprecate this method in favor of a (less confusing)\n # rename_dims() method that only renames dimensions.\n inplace = _check_inplace(inplace)\n for k, v in dims_dict.items():\n if k not in self.dims:\n raise ValueError('cannot swap from dimension %r because it is '\n 'not an existing dimension' % k)\n if self.variables[v].dims != (k,):\n raise ValueError('replacement dimension %r is not a 1D '\n 'variable along the old dimension %r'\n % (v, k))\n\n result_dims = set(dims_dict.get(dim, dim) for dim in self.dims)\n\n coord_names = self._coord_names.copy()\n coord_names.update(dims_dict.values())\n\n variables = OrderedDict()\n indexes = OrderedDict()\n for k, v in self.variables.items():\n dims = tuple(dims_dict.get(dim, dim) for dim in v.dims)\n if k in result_dims:\n var = v.to_index_variable()\n if k in self.indexes:\n indexes[k] = self.indexes[k]\n else:\n indexes[k] = var.to_index()\n else:\n var = v.to_base_variable()\n var.dims = dims\n variables[k] = var\n\n return self._replace_with_new_dims(variables, coord_names,\n indexes=indexes, inplace=inplace)", "response": "Returns a new Dataset containing swapped dimensions."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new object with an additional dimension inserted at the corresponding position in the array.", "response": "def expand_dims(self, dim=None, axis=None, **dim_kwargs):\n \"\"\"Return a new object with an additional axis (or axes) inserted at\n the corresponding position in the array shape.\n\n If dim is already a scalar coordinate, it will be promoted to a 1D\n coordinate consisting of a single value.\n\n Parameters\n ----------\n dim : str, sequence of str, dict, or None\n Dimensions to include on the new variable.\n If provided as str or sequence of str, then dimensions are inserted\n with length 1. If provided as a dict, then the keys are the new\n dimensions and the values are either integers (giving the length of\n the new dimensions) or sequence/ndarray (giving the coordinates of\n the new dimensions). **WARNING** for python 3.5, if ``dim`` is\n dict-like, then it must be an ``OrderedDict``. This is to ensure\n that the order in which the dims are given is maintained.\n axis : integer, list (or tuple) of integers, or None\n Axis position(s) where new axis is to be inserted (position(s) on\n the result array). If a list (or tuple) of integers is passed,\n multiple axes are inserted. In this case, dim arguments should be\n same length list. If axis=None is passed, all the axes will be\n inserted to the start of the result array.\n **dim_kwargs : int or sequence/ndarray\n The keywords are arbitrary dimensions being inserted and the values\n are either the lengths of the new dims (if int is given), or their\n coordinates. Note, this is an alternative to passing a dict to the\n dim kwarg and will only be used if dim is None. **WARNING** for\n python 3.5 ``dim_kwargs`` is not available.\n\n Returns\n -------\n expanded : same type as caller\n This object, but with an additional dimension(s).\n \"\"\"\n if isinstance(dim, int):\n raise TypeError('dim should be str or sequence of strs or dict')\n elif isinstance(dim, str):\n dim = OrderedDict(((dim, 1),))\n elif isinstance(dim, (list, tuple)):\n if len(dim) != len(set(dim)):\n raise ValueError('dims should not contain duplicate values.')\n dim = OrderedDict(((d, 1) for d in dim))\n\n # TODO: get rid of the below code block when python 3.5 is no longer\n # supported.\n python36_plus = sys.version_info[0] == 3 and sys.version_info[1] > 5\n not_ordereddict = dim is not None and not isinstance(dim, OrderedDict)\n if not python36_plus and not_ordereddict:\n raise TypeError(\"dim must be an OrderedDict for python <3.6\")\n elif not python36_plus and dim_kwargs:\n raise ValueError(\"dim_kwargs isn't available for python <3.6\")\n\n dim = either_dict_or_kwargs(dim, dim_kwargs, 'expand_dims')\n\n if axis is not None and not isinstance(axis, (list, tuple)):\n axis = [axis]\n\n if axis is None:\n axis = list(range(len(dim)))\n\n if len(dim) != len(axis):\n raise ValueError('lengths of dim and axis should be identical.')\n for d in dim:\n if d in self.dims:\n raise ValueError(\n 'Dimension {dim} already exists.'.format(dim=d))\n if (d in self._variables and\n not utils.is_scalar(self._variables[d])):\n raise ValueError(\n '{dim} already exists as coordinate or'\n ' variable name.'.format(dim=d))\n\n variables = OrderedDict()\n coord_names = self._coord_names.copy()\n # If dim is a dict, then ensure that the values are either integers\n # or iterables.\n for k, v in dim.items():\n if hasattr(v, \"__iter__\"):\n # If the value for the new dimension is an iterable, then\n # save the coordinates to the variables dict, and set the\n # value within the dim dict to the length of the iterable\n # for later use.\n variables[k] = xr.IndexVariable((k,), v)\n coord_names.add(k)\n dim[k] = variables[k].size\n elif isinstance(v, int):\n pass # Do nothing if the dimensions value is just an int\n else:\n raise TypeError('The value of new dimension {k} must be '\n 'an iterable or an int'.format(k=k))\n\n for k, v in self._variables.items():\n if k not in dim:\n if k in coord_names: # Do not change coordinates\n variables[k] = v\n else:\n result_ndim = len(v.dims) + len(axis)\n for a in axis:\n if a < -result_ndim or result_ndim - 1 < a:\n raise IndexError(\n 'Axis {a} is out of bounds of the expanded'\n ' dimension size {dim}.'.format(\n a=a, v=k, dim=result_ndim))\n\n axis_pos = [a if a >= 0 else result_ndim + a\n for a in axis]\n if len(axis_pos) != len(set(axis_pos)):\n raise ValueError('axis should not contain duplicate'\n ' values.')\n # We need to sort them to make sure `axis` equals to the\n # axis positions of the result array.\n zip_axis_dim = sorted(zip(axis_pos, dim.items()))\n\n all_dims = list(zip(v.dims, v.shape))\n for d, c in zip_axis_dim:\n all_dims.insert(d, c)\n all_dims = OrderedDict(all_dims)\n\n variables[k] = v.set_dims(all_dims)\n else:\n # If dims includes a label of a non-dimension coordinate,\n # it will be promoted to a 1D coordinate with a single value.\n variables[k] = v.set_dims(k)\n\n new_dims = self._dims.copy()\n new_dims.update(dim)\n\n return self._replace_vars_and_dims(\n variables, dims=new_dims, coord_names=coord_names)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_index(self, indexes=None, append=False, inplace=None,\n **indexes_kwargs):\n \"\"\"Set Dataset (multi-)indexes using one or more existing coordinates\n or variables.\n\n Parameters\n ----------\n indexes : {dim: index, ...}\n Mapping from names matching dimensions and values given\n by (lists of) the names of existing coordinates or variables to set\n as new (multi-)index.\n append : bool, optional\n If True, append the supplied index(es) to the existing index(es).\n Otherwise replace the existing index(es) (default).\n inplace : bool, optional\n If True, set new index(es) in-place. Otherwise, return a new\n Dataset object.\n **indexes_kwargs: optional\n The keyword arguments form of ``indexes``.\n One of indexes or indexes_kwargs must be provided.\n\n Returns\n -------\n obj : Dataset\n Another dataset, with this dataset's data but replaced coordinates.\n\n See Also\n --------\n Dataset.reset_index\n Dataset.swap_dims\n \"\"\"\n inplace = _check_inplace(inplace)\n indexes = either_dict_or_kwargs(indexes, indexes_kwargs, 'set_index')\n variables, coord_names = merge_indexes(indexes, self._variables,\n self._coord_names,\n append=append)\n return self._replace_vars_and_dims(variables, coord_names=coord_names,\n inplace=inplace)", "response": "Set the index of the data for the given locations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reset_index(self, dims_or_levels, drop=False, inplace=None):\n inplace = _check_inplace(inplace)\n variables, coord_names = split_indexes(dims_or_levels, self._variables,\n self._coord_names,\n self._level_coords, drop=drop)\n return self._replace_vars_and_dims(variables, coord_names=coord_names,\n inplace=inplace)", "response": "Reset the index of the specified dimension or multi - index levels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reorder_levels(self, dim_order=None, inplace=None,\n **dim_order_kwargs):\n \"\"\"Rearrange index levels using input order.\n\n Parameters\n ----------\n dim_order : optional\n Mapping from names matching dimensions and values given\n by lists representing new level orders. Every given dimension\n must have a multi-index.\n inplace : bool, optional\n If True, modify the dataset in-place. Otherwise, return a new\n DataArray object.\n **dim_order_kwargs: optional\n The keyword arguments form of ``dim_order``.\n One of dim_order or dim_order_kwargs must be provided.\n\n Returns\n -------\n obj : Dataset\n Another dataset, with this dataset's data but replaced\n coordinates.\n \"\"\"\n inplace = _check_inplace(inplace)\n dim_order = either_dict_or_kwargs(dim_order, dim_order_kwargs,\n 'reorder_levels')\n replace_variables = {}\n for dim, order in dim_order.items():\n coord = self._variables[dim]\n index = coord.to_index()\n if not isinstance(index, pd.MultiIndex):\n raise ValueError(\"coordinate %r has no MultiIndex\" % dim)\n replace_variables[dim] = IndexVariable(coord.dims,\n index.reorder_levels(order))\n variables = self._variables.copy()\n variables.update(replace_variables)\n return self._replace_vars_and_dims(variables, inplace=inplace)", "response": "Rearrange index levels using input order."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, other, inplace=None):\n inplace = _check_inplace(inplace, default=True)\n variables, coord_names, dims = dataset_update_method(self, other)\n\n return self._replace_vars_and_dims(variables, coord_names, dims,\n inplace=inplace)", "response": "Update this dataset s variables with those from another dataset."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmerge two set of exception - related datasets into a single dataset.", "response": "def merge(self, other, inplace=None, overwrite_vars=frozenset(),\n compat='no_conflicts', join='outer'):\n \"\"\"Merge the arrays of two datasets into a single dataset.\n\n This method generally not allow for overriding data, with the exception\n of attributes, which are ignored on the second dataset. Variables with\n the same name are checked for conflicts via the equals or identical\n methods.\n\n Parameters\n ----------\n other : Dataset or castable to Dataset\n Dataset or variables to merge with this dataset.\n inplace : bool, optional\n If True, merge the other dataset into this dataset in-place.\n Otherwise, return a new dataset object.\n overwrite_vars : str or sequence, optional\n If provided, update variables of these name(s) without checking for\n conflicts in this dataset.\n compat : {'broadcast_equals', 'equals', 'identical',\n 'no_conflicts'}, optional\n String indicating how to compare variables of the same name for\n potential conflicts:\n - 'broadcast_equals': all values must be equal when variables are\n broadcast against each other to ensure common dimensions.\n - 'equals': all values and dimensions must be the same.\n - 'identical': all values, dimensions and attributes must be the\n same.\n - 'no_conflicts': only values which are not null in both datasets\n must be equal. The returned dataset then contains the combination\n of all non-null values.\n join : {'outer', 'inner', 'left', 'right', 'exact'}, optional\n Method for joining ``self`` and ``other`` along shared dimensions:\n\n - 'outer': use the union of the indexes\n - 'inner': use the intersection of the indexes\n - 'left': use indexes from ``self``\n - 'right': use indexes from ``other``\n - 'exact': error instead of aligning non-equal indexes\n\n Returns\n -------\n merged : Dataset\n Merged dataset.\n\n Raises\n ------\n MergeError\n If any variables conflict (see ``compat``).\n \"\"\"\n inplace = _check_inplace(inplace)\n variables, coord_names, dims = dataset_merge_method(\n self, other, overwrite_vars=overwrite_vars, compat=compat,\n join=join)\n\n return self._replace_vars_and_dims(variables, coord_names, dims,\n inplace=inplace)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndrops variables or index labels from this dataset.", "response": "def drop(self, labels, dim=None):\n \"\"\"Drop variables or index labels from this dataset.\n\n Parameters\n ----------\n labels : scalar or list of scalars\n Name(s) of variables or index labels to drop.\n dim : None or str, optional\n Dimension along which to drop index labels. By default (if\n ``dim is None``), drops variables rather than index labels.\n\n Returns\n -------\n dropped : Dataset\n \"\"\"\n if utils.is_scalar(labels):\n labels = [labels]\n if dim is None:\n return self._drop_vars(labels)\n else:\n try:\n index = self.indexes[dim]\n except KeyError:\n raise ValueError(\n 'dimension %r does not have coordinate labels' % dim)\n new_index = index.drop(labels)\n return self.loc[{dim: new_index}]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef drop_dims(self, drop_dims):\n if utils.is_scalar(drop_dims):\n drop_dims = [drop_dims]\n\n missing_dimensions = [d for d in drop_dims if d not in self.dims]\n if missing_dimensions:\n raise ValueError('Dataset does not contain the dimensions: %s'\n % missing_dimensions)\n\n drop_vars = set(k for k, v in self._variables.items()\n for d in v.dims if d in drop_dims)\n\n variables = OrderedDict((k, v) for k, v in self._variables.items()\n if k not in drop_vars)\n coord_names = set(k for k in self._coord_names if k in variables)\n\n return self._replace_with_new_dims(variables, coord_names)", "response": "Drop dimensions and associated variables from this dataset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a new Dataset object with all array dimensions transposed.", "response": "def transpose(self, *dims):\n \"\"\"Return a new Dataset object with all array dimensions transposed.\n\n Although the order of dimensions on each array will change, the dataset\n dimensions themselves will remain in fixed (sorted) order.\n\n Parameters\n ----------\n *dims : str, optional\n By default, reverse the dimensions on each array. Otherwise,\n reorder the dimensions to this order.\n\n Returns\n -------\n transposed : Dataset\n Each array in the dataset (including) coordinates will be\n transposed to the given order.\n\n Notes\n -----\n This operation returns a view of each array's data. It is\n lazy for dask-backed DataArrays but not for numpy-backed DataArrays\n -- the data will be fully loaded into memory.\n\n See Also\n --------\n numpy.transpose\n DataArray.transpose\n \"\"\"\n if dims:\n if set(dims) ^ set(self.dims):\n raise ValueError('arguments to transpose (%s) must be '\n 'permuted dataset dimensions (%s)'\n % (dims, tuple(self.dims)))\n ds = self.copy()\n for name, var in self._variables.items():\n var_dims = tuple(dim for dim in dims if dim in var.dims)\n ds._variables[name] = var.transpose(*var_dims)\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dropna(self, dim, how='any', thresh=None, subset=None):\n # TODO: consider supporting multiple dimensions? Or not, given that\n # there are some ugly edge cases, e.g., pandas's dropna differs\n # depending on the order of the supplied axes.\n\n if dim not in self.dims:\n raise ValueError('%s must be a single dataset dimension' % dim)\n\n if subset is None:\n subset = list(self.data_vars)\n\n count = np.zeros(self.dims[dim], dtype=np.int64)\n size = 0\n\n for k in subset:\n array = self._variables[k]\n if dim in array.dims:\n dims = [d for d in array.dims if d != dim]\n count += np.asarray(array.count(dims))\n size += np.prod([self.dims[d] for d in dims])\n\n if thresh is not None:\n mask = count >= thresh\n elif how == 'any':\n mask = count == size\n elif how == 'all':\n mask = count > 0\n elif how is not None:\n raise ValueError('invalid how option: %s' % how)\n else:\n raise TypeError('must specify how or thresh')\n\n return self.isel({dim: mask})", "response": "Returns a new dataset with dropped labels for missing values along the provided dimension."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfilling missing values in this object with the given value.", "response": "def fillna(self, value):\n \"\"\"Fill missing values in this object.\n\n This operation follows the normal broadcasting and alignment rules that\n xarray uses for binary arithmetic, except the result is aligned to this\n object (``join='left'``) instead of aligned to the intersection of\n index coordinates (``join='inner'``).\n\n Parameters\n ----------\n value : scalar, ndarray, DataArray, dict or Dataset\n Used to fill all matching missing values in this dataset's data\n variables. Scalars, ndarrays or DataArrays arguments are used to\n fill all data with aligned coordinates (for DataArrays).\n Dictionaries or datasets match data variables and then align\n coordinates if necessary.\n\n Returns\n -------\n Dataset\n \"\"\"\n if utils.is_dict_like(value):\n value_keys = getattr(value, 'data_vars', value).keys()\n if not set(value_keys) <= set(self.data_vars.keys()):\n raise ValueError('all variables in the argument to `fillna` '\n 'must be contained in the original dataset')\n out = ops.fillna(self, value)\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninterpolate values along a specified dimension.", "response": "def interpolate_na(self, dim=None, method='linear', limit=None,\n use_coordinate=True,\n **kwargs):\n \"\"\"Interpolate values according to different methods.\n\n Parameters\n ----------\n dim : str\n Specifies the dimension along which to interpolate.\n method : {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic',\n 'polynomial', 'barycentric', 'krog', 'pchip',\n 'spline'}, optional\n String indicating which method to use for interpolation:\n\n - 'linear': linear interpolation (Default). Additional keyword\n arguments are passed to ``numpy.interp``\n - 'nearest', 'zero', 'slinear', 'quadratic', 'cubic',\n 'polynomial': are passed to ``scipy.interpolate.interp1d``. If\n method=='polynomial', the ``order`` keyword argument must also be\n provided.\n - 'barycentric', 'krog', 'pchip', 'spline': use their respective\n ``scipy.interpolate`` classes.\n use_coordinate : boolean or str, default True\n Specifies which index to use as the x values in the interpolation\n formulated as `y = f(x)`. If False, values are treated as if\n eqaully-spaced along `dim`. If True, the IndexVariable `dim` is\n used. If use_coordinate is a string, it specifies the name of a\n coordinate variariable to use as the index.\n limit : int, default None\n Maximum number of consecutive NaNs to fill. Must be greater than 0\n or None for no limit.\n\n Returns\n -------\n Dataset\n\n See also\n --------\n numpy.interp\n scipy.interpolate\n \"\"\"\n from .missing import interp_na, _apply_over_vars_with_dim\n\n new = _apply_over_vars_with_dim(interp_na, self, dim=dim,\n method=method, limit=limit,\n use_coordinate=use_coordinate,\n **kwargs)\n return new"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfilling NaN values by propogating values forward", "response": "def ffill(self, dim, limit=None):\n '''Fill NaN values by propogating values forward\n\n *Requires bottleneck.*\n\n Parameters\n ----------\n dim : str\n Specifies the dimension along which to propagate values when\n filling.\n limit : int, default None\n The maximum number of consecutive NaN values to forward fill. In\n other words, if there is a gap with more than this number of\n consecutive NaNs, it will only be partially filled. Must be greater\n than 0 or None for no limit.\n\n Returns\n -------\n Dataset\n '''\n from .missing import ffill, _apply_over_vars_with_dim\n\n new = _apply_over_vars_with_dim(ffill, self, dim=dim, limit=limit)\n return new"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef bfill(self, dim, limit=None):\n '''Fill NaN values by propogating values backward\n\n *Requires bottleneck.*\n\n Parameters\n ----------\n dim : str\n Specifies the dimension along which to propagate values when\n filling.\n limit : int, default None\n The maximum number of consecutive NaN values to backward fill. In\n other words, if there is a gap with more than this number of\n consecutive NaNs, it will only be partially filled. Must be greater\n than 0 or None for no limit.\n\n Returns\n -------\n Dataset\n '''\n from .missing import bfill, _apply_over_vars_with_dim\n\n new = _apply_over_vars_with_dim(bfill, self, dim=dim, limit=limit)\n return new", "response": "Fill NaN values by propogating values backward\n filling."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef combine_first(self, other):\n out = ops.fillna(self, other, join=\"outer\", dataset_join=\"outer\")\n return out", "response": "Combine two Datasets default to data_vars of self."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reduce(self, func, dim=None, keep_attrs=None, numeric_only=False,\n allow_lazy=False, **kwargs):\n \"\"\"Reduce this dataset by applying `func` along some dimension(s).\n\n Parameters\n ----------\n func : function\n Function which can be called in the form\n `f(x, axis=axis, **kwargs)` to return the result of reducing an\n np.ndarray over an integer valued axis.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply `func`. By default `func` is\n applied over all dimensions.\n keep_attrs : bool, optional\n If True, the dataset's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n numeric_only : bool, optional\n If True, only apply ``func`` to variables with a numeric dtype.\n **kwargs : dict\n Additional keyword arguments passed on to ``func``.\n\n Returns\n -------\n reduced : Dataset\n Dataset with this object's DataArrays replaced with new DataArrays\n of summarized data and the indicated dimension(s) removed.\n \"\"\"\n if dim is ALL_DIMS:\n dim = None\n if isinstance(dim, str):\n dims = set([dim])\n elif dim is None:\n dims = set(self.dims)\n else:\n dims = set(dim)\n\n missing_dimensions = [d for d in dims if d not in self.dims]\n if missing_dimensions:\n raise ValueError('Dataset does not contain the dimensions: %s'\n % missing_dimensions)\n\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n\n variables = OrderedDict()\n for name, var in self._variables.items():\n reduce_dims = [d for d in var.dims if d in dims]\n if name in self.coords:\n if not reduce_dims:\n variables[name] = var\n else:\n if (not numeric_only or\n np.issubdtype(var.dtype, np.number) or\n (var.dtype == np.bool_)):\n if len(reduce_dims) == 1:\n # unpack dimensions for the benefit of functions\n # like np.argmin which can't handle tuple arguments\n reduce_dims, = reduce_dims\n elif len(reduce_dims) == var.ndim:\n # prefer to aggregate over axis=None rather than\n # axis=(0, 1) if they will be equivalent, because\n # the former is often more efficient\n reduce_dims = None\n variables[name] = var.reduce(func, dim=reduce_dims,\n keep_attrs=keep_attrs,\n allow_lazy=allow_lazy,\n **kwargs)\n\n coord_names = set(k for k in self.coords if k in variables)\n attrs = self.attrs if keep_attrs else None\n return self._replace_vars_and_dims(variables, coord_names, attrs=attrs)", "response": "Reduces this Dataset by applying func along some dimension."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\napply a function over the data variables in this dataset and return a new dataset with the result.", "response": "def apply(self, func, keep_attrs=None, args=(), **kwargs):\n \"\"\"Apply a function over the data variables in this dataset.\n\n Parameters\n ----------\n func : function\n Function which can be called in the form `func(x, *args, **kwargs)`\n to transform each DataArray `x` in this dataset into another\n DataArray.\n keep_attrs : bool, optional\n If True, the dataset's attributes (`attrs`) will be copied from\n the original object to the new one. If False, the new object will\n be returned without attributes.\n args : tuple, optional\n Positional arguments passed on to `func`.\n **kwargs : dict\n Keyword arguments passed on to `func`.\n\n Returns\n -------\n applied : Dataset\n Resulting dataset from applying ``func`` over each data variable.\n\n Examples\n --------\n >>> da = xr.DataArray(np.random.randn(2, 3))\n >>> ds = xr.Dataset({'foo': da, 'bar': ('x', [-1, 2])})\n >>> ds\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Dimensions without coordinates: dim_0, dim_1, x\n Data variables:\n foo (dim_0, dim_1) float64 -0.3751 -1.951 -1.945 0.2948 0.711 -0.3948\n bar (x) int64 -1 2\n >>> ds.apply(np.fabs)\n \n Dimensions: (dim_0: 2, dim_1: 3, x: 2)\n Dimensions without coordinates: dim_0, dim_1, x\n Data variables:\n foo (dim_0, dim_1) float64 0.3751 1.951 1.945 0.2948 0.711 0.3948\n bar (x) float64 1.0 2.0\n \"\"\" # noqa\n variables = OrderedDict(\n (k, maybe_wrap_array(v, func(v, *args, **kwargs)))\n for k, v in self.data_vars.items())\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n attrs = self.attrs if keep_attrs else None\n return type(self)(variables, attrs=attrs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef assign(self, variables=None, **variables_kwargs):\n variables = either_dict_or_kwargs(\n variables, variables_kwargs, 'assign')\n data = self.copy()\n # do all calculations first...\n results = data._calc_assign_results(variables)\n # ... and then assign\n data.update(results)\n return data", "response": "Assign new data variables to a Dataset returning a new Dataset with all the original variables added to the new ones."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_array(self, dim='variable', name=None):\n from .dataarray import DataArray\n\n data_vars = [self.variables[k] for k in self.data_vars]\n broadcast_vars = broadcast_variables(*data_vars)\n data = duck_array_ops.stack([b.data for b in broadcast_vars], axis=0)\n\n coords = dict(self.coords)\n coords[dim] = list(self.data_vars)\n\n dims = (dim,) + broadcast_vars[0].dims\n\n return DataArray(data, coords, dims, attrs=self.attrs, name=name)", "response": "Convert this dataset into an xarray. DataArray."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting this dataset into a dask. dataframe. DataFrame.", "response": "def to_dask_dataframe(self, dim_order=None, set_index=False):\n \"\"\"\n Convert this dataset into a dask.dataframe.DataFrame.\n\n The dimensions, coordinates and data variables in this dataset form\n the columns of the DataFrame.\n\n Parameters\n ----------\n dim_order : list, optional\n Hierarchical dimension order for the resulting dataframe. All\n arrays are transposed to this order and then written out as flat\n vectors in contiguous order, so the last dimension in this list\n will be contiguous in the resulting DataFrame. This has a major\n influence on which operations are efficient on the resulting dask\n dataframe.\n\n If provided, must include all dimensions on this dataset. By\n default, dimensions are sorted alphabetically.\n set_index : bool, optional\n If set_index=True, the dask DataFrame is indexed by this dataset's\n coordinate. Since dask DataFrames to not support multi-indexes,\n set_index only works if the dataset only contains one dimension.\n\n Returns\n -------\n dask.dataframe.DataFrame\n \"\"\"\n\n import dask.array as da\n import dask.dataframe as dd\n\n if dim_order is None:\n dim_order = list(self.dims)\n elif set(dim_order) != set(self.dims):\n raise ValueError(\n 'dim_order {} does not match the set of dimensions on this '\n 'Dataset: {}'.format(dim_order, list(self.dims)))\n\n ordered_dims = OrderedDict((k, self.dims[k]) for k in dim_order)\n\n columns = list(ordered_dims)\n columns.extend(k for k in self.coords if k not in self.dims)\n columns.extend(self.data_vars)\n\n series_list = []\n for name in columns:\n try:\n var = self.variables[name]\n except KeyError:\n # dimension without a matching coordinate\n size = self.dims[name]\n data = da.arange(size, chunks=size, dtype=np.int64)\n var = Variable((name,), data)\n\n # IndexVariable objects have a dummy .chunk() method\n if isinstance(var, IndexVariable):\n var = var.to_base_variable()\n\n dask_array = var.set_dims(ordered_dims).chunk(self.chunks).data\n series = dd.from_array(dask_array.reshape(-1), columns=[name])\n series_list.append(series)\n\n df = dd.concat(series_list, axis=1)\n\n if set_index:\n if len(dim_order) == 1:\n (dim,) = dim_order\n df = df.set_index(dim)\n else:\n # triggers an error about multi-indexes, even if only one\n # dimension is passed\n df = df.set_index(dim_order)\n\n return df"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts this dataset to a dictionary following xarray naming conventions.", "response": "def to_dict(self, data=True):\n \"\"\"\n Convert this dataset to a dictionary following xarray naming\n conventions.\n\n Converts all variables and attributes to native Python objects\n Useful for coverting to json. To avoid datetime incompatibility\n use decode_times=False kwarg in xarrray.open_dataset.\n\n Parameters\n ----------\n data : bool, optional\n Whether to include the actual data in the dictionary. When set to\n False, returns just the schema.\n\n See also\n --------\n Dataset.from_dict\n \"\"\"\n d = {'coords': {}, 'attrs': decode_numpy_dict_values(self.attrs),\n 'dims': dict(self.dims), 'data_vars': {}}\n for k in self.coords:\n d['coords'].update({k: self[k].variable.to_dict(data=data)})\n for k in self.data_vars:\n d['data_vars'].update({k: self[k].variable.to_dict(data=data)})\n return d"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a dictionary into an xarray. Dataset.", "response": "def from_dict(cls, d):\n \"\"\"\n Convert a dictionary into an xarray.Dataset.\n\n Input dict can take several forms::\n\n d = {'t': {'dims': ('t'), 'data': t},\n 'a': {'dims': ('t'), 'data': x},\n 'b': {'dims': ('t'), 'data': y}}\n\n d = {'coords': {'t': {'dims': 't', 'data': t,\n 'attrs': {'units':'s'}}},\n 'attrs': {'title': 'air temperature'},\n 'dims': 't',\n 'data_vars': {'a': {'dims': 't', 'data': x, },\n 'b': {'dims': 't', 'data': y}}}\n\n where 't' is the name of the dimesion, 'a' and 'b' are names of data\n variables and t, x, and y are lists, numpy.arrays or pandas objects.\n\n Parameters\n ----------\n d : dict, with a minimum structure of {'var_0': {'dims': [..], \\\n 'data': [..]}, \\\n ...}\n\n Returns\n -------\n obj : xarray.Dataset\n\n See also\n --------\n Dataset.to_dict\n DataArray.from_dict\n \"\"\"\n\n if not set(['coords', 'data_vars']).issubset(set(d)):\n variables = d.items()\n else:\n import itertools\n variables = itertools.chain(d.get('coords', {}).items(),\n d.get('data_vars', {}).items())\n try:\n variable_dict = OrderedDict([(k, (v['dims'],\n v['data'],\n v.get('attrs'))) for\n k, v in variables])\n except KeyError as e:\n raise ValueError(\n \"cannot convert dict without the key \"\n \"'{dims_data}'\".format(dims_data=str(e.args[0])))\n obj = cls(variable_dict)\n\n # what if coords aren't dims?\n coords = set(d.get('coords', {})) - set(d.get('dims', {}))\n obj = obj.set_coords(coords)\n\n obj.attrs.update(d.get('attrs', {}))\n\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the n - th order discrete difference along a given dimension.", "response": "def diff(self, dim, n=1, label='upper'):\n \"\"\"Calculate the n-th order discrete difference along given axis.\n\n Parameters\n ----------\n dim : str, optional\n Dimension over which to calculate the finite difference.\n n : int, optional\n The number of times values are differenced.\n label : str, optional\n The new coordinate in dimension ``dim`` will have the\n values of either the minuend's or subtrahend's coordinate\n for values 'upper' and 'lower', respectively. Other\n values are not supported.\n\n Returns\n -------\n difference : same type as caller\n The n-th order finite difference of this object.\n\n Examples\n --------\n >>> ds = xr.Dataset({'foo': ('x', [5, 5, 6, 6])})\n >>> ds.diff('x')\n \n Dimensions: (x: 3)\n Coordinates:\n * x (x) int64 1 2 3\n Data variables:\n foo (x) int64 0 1 0\n >>> ds.diff('x', 2)\n \n Dimensions: (x: 2)\n Coordinates:\n * x (x) int64 2 3\n Data variables:\n foo (x) int64 1 -1\n\n See Also\n --------\n Dataset.differentiate\n \"\"\"\n if n == 0:\n return self\n if n < 0:\n raise ValueError('order `n` must be non-negative but got {0}'\n ''.format(n))\n\n # prepare slices\n kwargs_start = {dim: slice(None, -1)}\n kwargs_end = {dim: slice(1, None)}\n\n # prepare new coordinate\n if label == 'upper':\n kwargs_new = kwargs_end\n elif label == 'lower':\n kwargs_new = kwargs_start\n else:\n raise ValueError('The \\'label\\' argument has to be either '\n '\\'upper\\' or \\'lower\\'')\n\n variables = OrderedDict()\n\n for name, var in self.variables.items():\n if dim in var.dims:\n if name in self.data_vars:\n variables[name] = (var.isel(**kwargs_end) -\n var.isel(**kwargs_start))\n else:\n variables[name] = var.isel(**kwargs_new)\n else:\n variables[name] = var\n\n difference = self._replace_vars_and_dims(variables)\n\n if n > 1:\n return difference.diff(dim, n - 1)\n else:\n return difference"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nshift this Dataset by an offset along one or more dimensions.", "response": "def shift(self, shifts=None, fill_value=dtypes.NA, **shifts_kwargs):\n \"\"\"Shift this dataset by an offset along one or more dimensions.\n\n Only data variables are moved; coordinates stay in place. This is\n consistent with the behavior of ``shift`` in pandas.\n\n Parameters\n ----------\n shifts : Mapping with the form of {dim: offset}\n Integer offset to shift along each of the given dimensions.\n Positive offsets shift to the right; negative offsets shift to the\n left.\n fill_value: scalar, optional\n Value to use for newly missing values\n **shifts_kwargs:\n The keyword arguments form of ``shifts``.\n One of shifts or shifts_kwarg must be provided.\n\n Returns\n -------\n shifted : Dataset\n Dataset with the same coordinates and attributes but shifted data\n variables.\n\n See also\n --------\n roll\n\n Examples\n --------\n\n >>> ds = xr.Dataset({'foo': ('x', list('abcde'))})\n >>> ds.shift(x=2)\n \n Dimensions: (x: 5)\n Coordinates:\n * x (x) int64 0 1 2 3 4\n Data variables:\n foo (x) object nan nan 'a' 'b' 'c'\n \"\"\"\n shifts = either_dict_or_kwargs(shifts, shifts_kwargs, 'shift')\n invalid = [k for k in shifts if k not in self.dims]\n if invalid:\n raise ValueError(\"dimensions %r do not exist\" % invalid)\n\n variables = OrderedDict()\n for name, var in self.variables.items():\n if name in self.data_vars:\n var_shifts = {k: v for k, v in shifts.items()\n if k in var.dims}\n variables[name] = var.shift(\n fill_value=fill_value, shifts=var_shifts)\n else:\n variables[name] = var\n\n return self._replace_vars_and_dims(variables)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef roll(self, shifts=None, roll_coords=None, **shifts_kwargs):\n shifts = either_dict_or_kwargs(shifts, shifts_kwargs, 'roll')\n invalid = [k for k in shifts if k not in self.dims]\n if invalid:\n raise ValueError(\"dimensions %r do not exist\" % invalid)\n\n if roll_coords is None:\n warnings.warn(\"roll_coords will be set to False in the future.\"\n \" Explicitly set roll_coords to silence warning.\",\n FutureWarning, stacklevel=2)\n roll_coords = True\n\n unrolled_vars = () if roll_coords else self.coords\n\n variables = OrderedDict()\n for k, v in self.variables.items():\n if k not in unrolled_vars:\n variables[k] = v.roll(**{k: s for k, s in shifts.items()\n if k in v.dims})\n else:\n variables[k] = v\n\n return self._replace_vars_and_dims(variables)", "response": "Roll this Dataset by an offset along one or more dimensions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsorting the dataset by the specified variables.", "response": "def sortby(self, variables, ascending=True):\n \"\"\"\n Sort object by labels or values (along an axis).\n\n Sorts the dataset, either along specified dimensions,\n or according to values of 1-D dataarrays that share dimension\n with calling object.\n\n If the input variables are dataarrays, then the dataarrays are aligned\n (via left-join) to the calling object prior to sorting by cell values.\n NaNs are sorted to the end, following Numpy convention.\n\n If multiple sorts along the same dimension is\n given, numpy's lexsort is performed along that dimension:\n https://docs.scipy.org/doc/numpy/reference/generated/numpy.lexsort.html\n and the FIRST key in the sequence is used as the primary sort key,\n followed by the 2nd key, etc.\n\n Parameters\n ----------\n variables: str, DataArray, or list of either\n 1D DataArray objects or name(s) of 1D variable(s) in\n coords/data_vars whose values are used to sort the dataset.\n ascending: boolean, optional\n Whether to sort by ascending or descending order.\n\n Returns\n -------\n sorted: Dataset\n A new dataset where all the specified dims are sorted by dim\n labels.\n \"\"\"\n from .dataarray import DataArray\n\n if not isinstance(variables, list):\n variables = [variables]\n else:\n variables = variables\n variables = [v if isinstance(v, DataArray) else self[v]\n for v in variables]\n aligned_vars = align(self, *variables, join='left')\n aligned_self = aligned_vars[0]\n aligned_other_vars = aligned_vars[1:]\n vars_by_dim = defaultdict(list)\n for data_array in aligned_other_vars:\n if data_array.ndim != 1:\n raise ValueError(\"Input DataArray is not 1-D.\")\n if (data_array.dtype == object and\n LooseVersion(np.__version__) < LooseVersion('1.11.0')):\n raise NotImplementedError(\n 'sortby uses np.lexsort under the hood, which requires '\n 'numpy 1.11.0 or later to support object data-type.')\n (key,) = data_array.dims\n vars_by_dim[key].append(data_array)\n\n indices = {}\n for key, arrays in vars_by_dim.items():\n order = np.lexsort(tuple(reversed(arrays)))\n indices[key] = order if ascending else order[::-1]\n return aligned_self.isel(**indices)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef quantile(self, q, dim=None, interpolation='linear',\n numeric_only=False, keep_attrs=None):\n \"\"\"Compute the qth quantile of the data along the specified dimension.\n\n Returns the qth quantiles(s) of the array elements for each variable\n in the Dataset.\n\n Parameters\n ----------\n q : float in range of [0,1] (or sequence of floats)\n Quantile to compute, which must be between 0 and 1 inclusive.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply quantile.\n interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}\n This optional parameter specifies the interpolation method to\n use when the desired quantile lies between two data points\n ``i < j``:\n\n * linear: ``i + (j - i) * fraction``, where ``fraction`` is\n the fractional part of the index surrounded by ``i`` and\n ``j``.\n * lower: ``i``.\n * higher: ``j``.\n * nearest: ``i`` or ``j``, whichever is nearest.\n * midpoint: ``(i + j) / 2``.\n keep_attrs : bool, optional\n If True, the dataset's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n numeric_only : bool, optional\n If True, only apply ``func`` to variables with a numeric dtype.\n\n Returns\n -------\n quantiles : Dataset\n If `q` is a single quantile, then the result is a scalar for each\n variable in data_vars. If multiple percentiles are given, first\n axis of the result corresponds to the quantile and a quantile\n dimension is added to the return Dataset. The other dimensions are\n the dimensions that remain after the reduction of the array.\n\n See Also\n --------\n numpy.nanpercentile, pandas.Series.quantile, DataArray.quantile\n \"\"\"\n\n if isinstance(dim, str):\n dims = set([dim])\n elif dim is None:\n dims = set(self.dims)\n else:\n dims = set(dim)\n\n _assert_empty([d for d in dims if d not in self.dims],\n 'Dataset does not contain the dimensions: %s')\n\n q = np.asarray(q, dtype=np.float64)\n\n variables = OrderedDict()\n for name, var in self.variables.items():\n reduce_dims = [d for d in var.dims if d in dims]\n if reduce_dims or not var.dims:\n if name not in self.coords:\n if (not numeric_only or\n np.issubdtype(var.dtype, np.number) or\n var.dtype == np.bool_):\n if len(reduce_dims) == var.ndim:\n # prefer to aggregate over axis=None rather than\n # axis=(0, 1) if they will be equivalent, because\n # the former is often more efficient\n reduce_dims = None\n variables[name] = var.quantile(\n q, dim=reduce_dims, interpolation=interpolation)\n\n else:\n variables[name] = var\n\n # construct the new dataset\n coord_names = set(k for k in self.coords if k in variables)\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n attrs = self.attrs if keep_attrs else None\n new = self._replace_vars_and_dims(variables, coord_names, attrs=attrs)\n if 'quantile' in new.dims:\n new.coords['quantile'] = Variable('quantile', q)\n else:\n new.coords['quantile'] = q\n return new", "response": "Compute the qth quantile of the data along the specified dimension."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rank(self, dim, pct=False, keep_attrs=None):\n if dim not in self.dims:\n raise ValueError(\n 'Dataset does not contain the dimension: %s' % dim)\n\n variables = OrderedDict()\n for name, var in self.variables.items():\n if name in self.data_vars:\n if dim in var.dims:\n variables[name] = var.rank(dim, pct=pct)\n else:\n variables[name] = var\n\n coord_names = set(self.coords)\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n attrs = self.attrs if keep_attrs else None\n return self._replace_vars_and_dims(variables, coord_names, attrs=attrs)", "response": "Ranks the data.\n\n Equal values are assigned a rank that is the average of the ranks that\n would have been otherwise assigned to all of the values within\n that set.\n Ranks begin at 1, not 0. If pct is True, computes percentage ranks.\n\n NaNs in the input array are returned as NaNs.\n\n The `bottleneck` library is required.\n\n Parameters\n ----------\n dim : str\n Dimension over which to compute rank.\n pct : bool, optional\n If True, compute percentage ranks, otherwise compute integer ranks.\n keep_attrs : bool, optional\n If True, the dataset's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n\n Returns\n -------\n ranked : Dataset\n Variables that do not depend on `dim` are dropped."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef differentiate(self, coord, edge_order=1, datetime_unit=None):\n from .variable import Variable\n\n if coord not in self.variables and coord not in self.dims:\n raise ValueError('Coordinate {} does not exist.'.format(coord))\n\n coord_var = self[coord].variable\n if coord_var.ndim != 1:\n raise ValueError('Coordinate {} must be 1 dimensional but is {}'\n ' dimensional'.format(coord, coord_var.ndim))\n\n dim = coord_var.dims[0]\n if _contains_datetime_like_objects(coord_var):\n if coord_var.dtype.kind in 'mM' and datetime_unit is None:\n datetime_unit, _ = np.datetime_data(coord_var.dtype)\n elif datetime_unit is None:\n datetime_unit = 's' # Default to seconds for cftime objects\n coord_var = coord_var._to_numeric(datetime_unit=datetime_unit)\n\n variables = OrderedDict()\n for k, v in self.variables.items():\n if (k in self.data_vars and dim in v.dims and\n k not in self.coords):\n if _contains_datetime_like_objects(v):\n v = v._to_numeric(datetime_unit=datetime_unit)\n grad = duck_array_ops.gradient(\n v.data, coord_var, edge_order=edge_order,\n axis=v.get_axis_num(dim))\n variables[k] = Variable(v.dims, grad)\n else:\n variables[k] = v\n return self._replace_vars_and_dims(variables)", "response": "Differentiate with the second order accurate central centralCOOKIE entries."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nintegrating the array with the trapezoidal rule.", "response": "def integrate(self, coord, datetime_unit=None):\n \"\"\" integrate the array with the trapezoidal rule.\n\n .. note::\n This feature is limited to simple cartesian geometry, i.e. coord\n must be one dimensional.\n\n Parameters\n ----------\n dim: str, or a sequence of str\n Coordinate(s) used for the integration.\n datetime_unit\n Can be specify the unit if datetime coordinate is used. One of\n {'Y', 'M', 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns', 'ps', 'fs',\n 'as'}\n\n Returns\n -------\n integrated: Dataset\n\n See also\n --------\n DataArray.integrate\n numpy.trapz: corresponding numpy function\n \"\"\"\n if not isinstance(coord, (list, tuple)):\n coord = (coord, )\n result = self\n for c in coord:\n result = result._integrate_one(c, datetime_unit=datetime_unit)\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter_by_attrs(self, **kwargs):\n selection = []\n for var_name, variable in self.data_vars.items():\n has_value_flag = False\n for attr_name, pattern in kwargs.items():\n attr_value = variable.attrs.get(attr_name)\n if ((callable(pattern) and pattern(attr_value)) or\n attr_value == pattern):\n has_value_flag = True\n else:\n has_value_flag = False\n break\n if has_value_flag is True:\n selection.append(var_name)\n return self[selection]", "response": "Returns a new Dataset with variables that match specific conditions."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _apply_mask(\n data: np.ndarray,\n encoded_fill_values: list,\n decoded_fill_value: Any,\n dtype: Any,\n) -> np.ndarray:\n \"\"\"Mask all matching values in a NumPy arrays.\"\"\"\n data = np.asarray(data, dtype=dtype)\n condition = False\n for fv in encoded_fill_values:\n condition |= data == fv\n return np.where(condition, decoded_fill_value, data)", "response": "Mask all matching values in a NumPy arrays."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a float dtype that can losslessly represent dtype values.", "response": "def _choose_float_dtype(dtype, has_offset):\n \"\"\"Return a float dtype that can losslessly represent `dtype` values.\"\"\"\n # Keep float32 as-is. Upcast half-precision to single-precision,\n # because float16 is \"intended for storage but not computation\"\n if dtype.itemsize <= 4 and np.issubdtype(dtype, np.floating):\n return np.float32\n # float32 can exactly represent all integers up to 24 bits\n if dtype.itemsize <= 2 and np.issubdtype(dtype, np.integer):\n # A scale factor is entirely safe (vanishing into the mantissa),\n # but a large integer offset could lead to loss of precision.\n # Sensitivity analysis can be tricky, so we just use a float64\n # if there's any offset at all - better unoptimised than wrong!\n if not has_offset:\n return np.float32\n # For all other types and circumstances, we just use float64.\n # (safe because eg. complex numbers are not supported in NetCDF)\n return np.float64"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _apply_over_vars_with_dim(func, self, dim=None, **kwargs):\n '''wrapper for datasets'''\n\n ds = type(self)(coords=self.coords, attrs=self.attrs)\n\n for name, var in self.data_vars.items():\n if dim in var.dims:\n ds[name] = func(var, dim=dim, **kwargs)\n else:\n ds[name] = var\n\n return ds", "response": "wrapper for datasets with dim = None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_clean_interp_index(arr, dim, use_coordinate=True, **kwargs):\n '''get index to use for x values in interpolation.\n\n If use_coordinate is True, the coordinate that shares the name of the\n dimension along which interpolation is being performed will be used as the\n x values.\n\n If use_coordinate is False, the x values are set as an equally spaced\n sequence.\n '''\n if use_coordinate:\n if use_coordinate is True:\n index = arr.get_index(dim)\n else:\n index = arr.coords[use_coordinate]\n if index.ndim != 1:\n raise ValueError(\n 'Coordinates used for interpolation must be 1D, '\n '%s is %dD.' % (use_coordinate, index.ndim))\n\n # raise if index cannot be cast to a float (e.g. MultiIndex)\n try:\n index = index.values.astype(np.float64)\n except (TypeError, ValueError):\n # pandas raises a TypeError\n # xarray/nuppy raise a ValueError\n raise TypeError('Index must be castable to float64 to support'\n 'interpolation, got: %s' % type(index))\n # check index sorting now so we can skip it later\n if not (np.diff(index) > 0).all():\n raise ValueError(\"Index must be monotonicly increasing\")\n else:\n axis = arr.get_axis_num(dim)\n index = np.arange(arr.shape[axis], dtype=np.float64)\n\n return index", "response": "get index to use for x values in interpolation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef interp_na(self, dim=None, use_coordinate=True, method='linear', limit=None,\n **kwargs):\n '''Interpolate values according to different methods.'''\n\n if dim is None:\n raise NotImplementedError('dim is a required argument')\n\n if limit is not None:\n valids = _get_valid_fill_mask(self, dim, limit)\n\n # method\n index = get_clean_interp_index(self, dim, use_coordinate=use_coordinate,\n **kwargs)\n interp_class, kwargs = _get_interpolator(method, **kwargs)\n interpolator = partial(func_interpolate_na, interp_class, **kwargs)\n\n with warnings.catch_warnings():\n warnings.filterwarnings('ignore', 'overflow', RuntimeWarning)\n warnings.filterwarnings('ignore', 'invalid value', RuntimeWarning)\n arr = apply_ufunc(interpolator, index, self,\n input_core_dims=[[dim], [dim]],\n output_core_dims=[[dim]],\n output_dtypes=[self.dtype],\n dask='parallelized',\n vectorize=True,\n keep_attrs=True).transpose(*self.dims)\n\n if limit is not None:\n arr = arr.where(valids)\n\n return arr", "response": "Interpolate values according to different methods."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef func_interpolate_na(interpolator, x, y, **kwargs):\n '''helper function to apply interpolation along 1 dimension'''\n # it would be nice if this wasn't necessary, works around:\n # \"ValueError: assignment destination is read-only\" in assignment below\n out = y.copy()\n\n nans = pd.isnull(y)\n nonans = ~nans\n\n # fast track for no-nans and all-nans cases\n n_nans = nans.sum()\n if n_nans == 0 or n_nans == len(y):\n return y\n\n f = interpolator(x[nonans], y[nonans], **kwargs)\n out[nans] = f(x[nans])\n return out", "response": "helper function to apply interpolation along 1 dimension"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ffill(arr, dim=None, limit=None):\n '''forward fill missing values'''\n import bottleneck as bn\n\n axis = arr.get_axis_num(dim)\n\n # work around for bottleneck 178\n _limit = limit if limit is not None else arr.shape[axis]\n\n return apply_ufunc(bn.push, arr,\n dask='parallelized',\n keep_attrs=True,\n output_dtypes=[arr.dtype],\n kwargs=dict(n=_limit, axis=axis)).transpose(*arr.dims)", "response": "forward fill missing values"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_interpolator(method, vectorizeable_only=False, **kwargs):\n '''helper function to select the appropriate interpolator class\n\n returns interpolator class and keyword arguments for the class\n '''\n interp1d_methods = ['linear', 'nearest', 'zero', 'slinear', 'quadratic',\n 'cubic', 'polynomial']\n valid_methods = interp1d_methods + ['barycentric', 'krog', 'pchip',\n 'spline', 'akima']\n\n has_scipy = True\n try:\n from scipy import interpolate\n except ImportError:\n has_scipy = False\n\n # prioritize scipy.interpolate\n if (method == 'linear' and not\n kwargs.get('fill_value', None) == 'extrapolate' and\n not vectorizeable_only):\n kwargs.update(method=method)\n interp_class = NumpyInterpolator\n\n elif method in valid_methods:\n if not has_scipy:\n raise ImportError(\n 'Interpolation with method `%s` requires scipy' % method)\n\n if method in interp1d_methods:\n kwargs.update(method=method)\n interp_class = ScipyInterpolator\n elif vectorizeable_only:\n raise ValueError('{} is not a vectorizeable interpolator. '\n 'Available methods are {}'.format(\n method, interp1d_methods))\n elif method == 'barycentric':\n interp_class = interpolate.BarycentricInterpolator\n elif method == 'krog':\n interp_class = interpolate.KroghInterpolator\n elif method == 'pchip':\n interp_class = interpolate.PchipInterpolator\n elif method == 'spline':\n kwargs.update(method=method)\n interp_class = SplineInterpolator\n elif method == 'akima':\n interp_class = interpolate.Akima1DInterpolator\n else:\n raise ValueError('%s is not a valid scipy interpolator' % method)\n else:\n raise ValueError('%s is not a valid interpolator' % method)\n\n return interp_class, kwargs", "response": "helper function to select the appropriate interpolator class for the appropriate class\n returns interpolator class and keyword arguments for the class\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_interpolator_nd(method, **kwargs):\n '''helper function to select the appropriate interpolator class\n\n returns interpolator class and keyword arguments for the class\n '''\n valid_methods = ['linear', 'nearest']\n\n try:\n from scipy import interpolate\n except ImportError:\n raise ImportError(\n 'Interpolation with method `%s` requires scipy' % method)\n\n if method in valid_methods:\n kwargs.update(method=method)\n interp_class = interpolate.interpn\n else:\n raise ValueError('%s is not a valid interpolator for interpolating '\n 'over multiple dimensions.' % method)\n\n return interp_class, kwargs", "response": "helper function to select the appropriate interpolator class and keyword arguments for the class\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_valid_fill_mask(arr, dim, limit):\n '''helper function to determine values that can be filled when limit is not\n None'''\n kw = {dim: limit + 1}\n # we explicitly use construct method to avoid copy.\n new_dim = utils.get_temp_dimname(arr.dims, '_window')\n return (arr.isnull().rolling(min_periods=1, **kw)\n .construct(new_dim, fill_value=False)\n .sum(new_dim, skipna=False)) <= limit", "response": "helper function to determine values that can be filled when limit is not\n None"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlocalize a variable by linear and nearest neighbor method.", "response": "def _localize(var, indexes_coords):\n \"\"\" Speed up for linear and nearest neighbor method.\n Only consider a subspace that is needed for the interpolation\n \"\"\"\n indexes = {}\n for dim, [x, new_x] in indexes_coords.items():\n index = x.to_index()\n imin = index.get_loc(np.min(new_x.values), method='nearest')\n imax = index.get_loc(np.max(new_x.values), method='nearest')\n\n indexes[dim] = slice(max(imin - 2, 0), imax + 2)\n indexes_coords[dim] = (x[indexes[dim]], new_x)\n return var.isel(**indexes), indexes_coords"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmaking x and new_x float.", "response": "def _floatize_x(x, new_x):\n \"\"\" Make x and new_x float.\n This is particulary useful for datetime dtype.\n x, new_x: tuple of np.ndarray\n \"\"\"\n x = list(x)\n new_x = list(new_x)\n for i in range(len(x)):\n if _contains_datetime_like_objects(x[i]):\n # Scipy casts coordinates to np.float64, which is not accurate\n # enough for datetime64 (uses 64bit integer).\n # We assume that the most of the bits are used to represent the\n # offset (min(x)) and the variation (x - min(x)) can be\n # represented by float.\n xmin = x[i].values.min()\n x[i] = x[i]._to_numeric(offset=xmin, dtype=np.float64)\n new_x[i] = new_x[i]._to_numeric(offset=xmin, dtype=np.float64)\n return x, new_x"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef interp(var, indexes_coords, method, **kwargs):\n if not indexes_coords:\n return var.copy()\n\n # simple speed up for the local interpolation\n if method in ['linear', 'nearest']:\n var, indexes_coords = _localize(var, indexes_coords)\n\n # default behavior\n kwargs['bounds_error'] = kwargs.get('bounds_error', False)\n\n # target dimensions\n dims = list(indexes_coords)\n x, new_x = zip(*[indexes_coords[d] for d in dims])\n destination = broadcast_variables(*new_x)\n\n # transpose to make the interpolated axis to the last position\n broadcast_dims = [d for d in var.dims if d not in dims]\n original_dims = broadcast_dims + dims\n new_dims = broadcast_dims + list(destination[0].dims)\n interped = interp_func(var.transpose(*original_dims).data,\n x, destination, method, kwargs)\n\n result = Variable(new_dims, interped, attrs=var.attrs)\n\n # dimension of the output array\n out_dims = OrderedSet()\n for d in var.dims:\n if d in dims:\n out_dims.update(indexes_coords[d][1].dims)\n else:\n out_dims.add(d)\n return result.transpose(*tuple(out_dims))", "response": "Interpolate a Variable with a set of indexes coordinates."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ninterpolating a 1 - dimensional array - like or array - like into a new array - like.", "response": "def interp_func(var, x, new_x, method, kwargs):\n \"\"\"\n multi-dimensional interpolation for array-like. Interpolated axes should be\n located in the last position.\n\n Parameters\n ----------\n var: np.ndarray or dask.array.Array\n Array to be interpolated. The final dimension is interpolated.\n x: a list of 1d array.\n Original coordinates. Should not contain NaN.\n new_x: a list of 1d array\n New coordinates. Should not contain NaN.\n method: string\n {'linear', 'nearest', 'zero', 'slinear', 'quadratic', 'cubic'} for\n 1-dimensional itnterpolation.\n {'linear', 'nearest'} for multidimensional interpolation\n **kwargs:\n Optional keyword arguments to be passed to scipy.interpolator\n\n Returns\n -------\n interpolated: array\n Interpolated array\n\n Note\n ----\n This requiers scipy installed.\n\n See Also\n --------\n scipy.interpolate.interp1d\n \"\"\"\n if not x:\n return var.copy()\n\n if len(x) == 1:\n func, kwargs = _get_interpolator(method, vectorizeable_only=True,\n **kwargs)\n else:\n func, kwargs = _get_interpolator_nd(method, **kwargs)\n\n if isinstance(var, dask_array_type):\n import dask.array as da\n\n _assert_single_chunk(var, range(var.ndim - len(x), var.ndim))\n chunks = var.chunks[:-len(x)] + new_x[0].shape\n drop_axis = range(var.ndim - len(x), var.ndim)\n new_axis = range(var.ndim - len(x), var.ndim - len(x) + new_x[0].ndim)\n return da.map_blocks(_interpnd, var, x, new_x, func, kwargs,\n dtype=var.dtype, chunks=chunks,\n new_axis=new_axis, drop_axis=drop_axis)\n\n return _interpnd(var, x, new_x, func, kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _nicetitle(coord, value, maxchar, template):\n prettyvalue = format_item(value, quote_strings=False)\n title = template.format(coord=coord, value=prettyvalue)\n\n if len(title) > maxchar:\n title = title[:(maxchar - 3)] + '...'\n\n return title", "response": "Return a nicetitle for a given coordinate and value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _easy_facetgrid(data, plotfunc, kind, x=None, y=None, row=None,\n col=None, col_wrap=None, sharex=True, sharey=True,\n aspect=None, size=None, subplot_kws=None, **kwargs):\n \"\"\"\n Convenience method to call xarray.plot.FacetGrid from 2d plotting methods\n\n kwargs are the arguments to 2d plotting method\n \"\"\"\n ax = kwargs.pop('ax', None)\n figsize = kwargs.pop('figsize', None)\n if ax is not None:\n raise ValueError(\"Can't use axes when making faceted plots.\")\n if aspect is None:\n aspect = 1\n if size is None:\n size = 3\n elif figsize is not None:\n raise ValueError('cannot provide both `figsize` and `size` arguments')\n\n g = FacetGrid(data=data, col=col, row=row, col_wrap=col_wrap,\n sharex=sharex, sharey=sharey, figsize=figsize,\n aspect=aspect, size=size, subplot_kws=subplot_kws)\n\n if kind == 'line':\n return g.map_dataarray_line(plotfunc, x, y, **kwargs)\n\n if kind == 'dataarray':\n return g.map_dataarray(plotfunc, x, y, **kwargs)", "response": "This method is used to make faceted plots from 2d plotting."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef map_dataarray(self, func, x, y, **kwargs):\n\n if kwargs.get('cbar_ax', None) is not None:\n raise ValueError('cbar_ax not supported by FacetGrid.')\n\n cmap_params, cbar_kwargs = _process_cmap_cbar_kwargs(\n func, kwargs, self.data.values)\n\n self._cmap_extend = cmap_params.get('extend')\n\n # Order is important\n func_kwargs = kwargs.copy()\n func_kwargs.update(cmap_params)\n func_kwargs.update({'add_colorbar': False, 'add_labels': False})\n\n # Get x, y labels for the first subplot\n x, y = _infer_xy_labels(\n darray=self.data.loc[self.name_dicts.flat[0]], x=x, y=y,\n imshow=func.__name__ == 'imshow', rgb=kwargs.get('rgb', None))\n\n for d, ax in zip(self.name_dicts.flat, self.axes.flat):\n # None is the sentinel value\n if d is not None:\n subset = self.data.loc[d]\n mappable = func(subset, x=x, y=y, ax=ax, **func_kwargs)\n self._mappables.append(mappable)\n\n self._cmap_extend = cmap_params.get('extend')\n self._finalize_grid(x, y)\n\n if kwargs.get('add_colorbar', True):\n self.add_colorbar(**cbar_kwargs)\n\n return self", "response": "Applies a function to a 2d xarray containing a subset of the data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfinalize the annotations and layout.", "response": "def _finalize_grid(self, *axlabels):\n \"\"\"Finalize the annotations and layout.\"\"\"\n if not self._finalized:\n self.set_axis_labels(*axlabels)\n self.set_titles()\n self.fig.tight_layout()\n\n for ax, namedict in zip(self.axes.flat, self.name_dicts.flat):\n if namedict is None:\n ax.set_visible(False)\n\n self._finalized = True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing a colorbar for the current object", "response": "def add_colorbar(self, **kwargs):\n \"\"\"Draw a colorbar\n \"\"\"\n kwargs = kwargs.copy()\n if self._cmap_extend is not None:\n kwargs.setdefault('extend', self._cmap_extend)\n if 'label' not in kwargs:\n kwargs.setdefault('label', label_from_attrs(self.data))\n self.cbar = self.fig.colorbar(self._mappables[-1],\n ax=list(self.axes.flat),\n **kwargs)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_axis_labels(self, x_var=None, y_var=None):\n if x_var is not None:\n if x_var in self.data.coords:\n self._x_var = x_var\n self.set_xlabels(label_from_attrs(self.data[x_var]))\n else:\n # x_var is a string\n self.set_xlabels(x_var)\n\n if y_var is not None:\n if y_var in self.data.coords:\n self._y_var = y_var\n self.set_ylabels(label_from_attrs(self.data[y_var]))\n else:\n self.set_ylabels(y_var)\n return self", "response": "Set axis labels on the left column and bottom row of the grid."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlabeling the x axis on the bottom row of the grid.", "response": "def set_xlabels(self, label=None, **kwargs):\n \"\"\"Label the x axis on the bottom row of the grid.\"\"\"\n if label is None:\n label = label_from_attrs(self.data[self._x_var])\n for ax in self._bottom_axes:\n ax.set_xlabel(label, **kwargs)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_ylabels(self, label=None, **kwargs):\n if label is None:\n label = label_from_attrs(self.data[self._y_var])\n for ax in self._left_axes:\n ax.set_ylabel(label, **kwargs)\n return self", "response": "Label the y axis on the left column of the grid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_titles(self, template=\"{coord} = {value}\", maxchar=30,\n **kwargs):\n \"\"\"\n Draw titles either above each facet or on the grid margins.\n\n Parameters\n ----------\n template : string\n Template for plot titles containing {coord} and {value}\n maxchar : int\n Truncate titles at maxchar\n kwargs : keyword args\n additional arguments to matplotlib.text\n\n Returns\n -------\n self: FacetGrid object\n\n \"\"\"\n import matplotlib as mpl\n\n kwargs[\"size\"] = kwargs.pop(\"size\", mpl.rcParams[\"axes.labelsize\"])\n\n nicetitle = functools.partial(_nicetitle, maxchar=maxchar,\n template=template)\n\n if self._single_group:\n for d, ax in zip(self.name_dicts.flat, self.axes.flat):\n # Only label the ones with data\n if d is not None:\n coord, value = list(d.items()).pop()\n title = nicetitle(coord, value, maxchar=maxchar)\n ax.set_title(title, **kwargs)\n else:\n # The row titles on the right edge of the grid\n for ax, row_name in zip(self.axes[:, -1], self.row_names):\n title = nicetitle(coord=self._row_var, value=row_name,\n maxchar=maxchar)\n ax.annotate(title, xy=(1.02, .5), xycoords=\"axes fraction\",\n rotation=270, ha=\"left\", va=\"center\", **kwargs)\n\n # The column titles on the top row\n for ax, col_name in zip(self.axes[0, :], self.col_names):\n title = nicetitle(coord=self._col_var, value=col_name,\n maxchar=maxchar)\n ax.set_title(title, **kwargs)\n\n return self", "response": "Draw titles on the grid and the grid margins."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset and control tick behavior on x and y axes.", "response": "def set_ticks(self, max_xticks=_NTICKS, max_yticks=_NTICKS,\n fontsize=_FONTSIZE):\n \"\"\"\n Set and control tick behavior\n\n Parameters\n ----------\n max_xticks, max_yticks : int, optional\n Maximum number of labeled ticks to plot on x, y axes\n fontsize : string or int\n Font size as used by matplotlib text\n\n Returns\n -------\n self : FacetGrid object\n\n \"\"\"\n from matplotlib.ticker import MaxNLocator\n\n # Both are necessary\n x_major_locator = MaxNLocator(nbins=max_xticks)\n y_major_locator = MaxNLocator(nbins=max_yticks)\n\n for ax in self.axes.flat:\n ax.xaxis.set_major_locator(x_major_locator)\n ax.yaxis.set_major_locator(y_major_locator)\n for tick in itertools.chain(ax.xaxis.get_major_ticks(),\n ax.yaxis.get_major_ticks()):\n tick.label.set_fontsize(fontsize)\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef map(self, func, *args, **kwargs):\n plt = import_matplotlib_pyplot()\n\n for ax, namedict in zip(self.axes.flat, self.name_dicts.flat):\n if namedict is not None:\n data = self.data.loc[namedict]\n plt.sca(ax)\n innerargs = [data[a].values for a in args]\n maybe_mappable = func(*innerargs, **kwargs)\n # TODO: better way to verify that an artist is mappable?\n # https://stackoverflow.com/questions/33023036/is-it-possible-to-detect-if-a-matplotlib-artist-is-a-mappable-suitable-for-use-w#33023522\n if (maybe_mappable and\n hasattr(maybe_mappable, 'autoscale_None')):\n self._mappables.append(maybe_mappable)\n\n self._finalize_grid(*args[:2])\n\n return self", "response": "Applies a function to each facet s subset of the data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_time_bins(index, freq, closed, label, base):\n\n if not isinstance(index, CFTimeIndex):\n raise TypeError('index must be a CFTimeIndex, but got '\n 'an instance of %r' % type(index).__name__)\n if len(index) == 0:\n datetime_bins = labels = CFTimeIndex(data=[], name=index.name)\n return datetime_bins, labels\n\n first, last = _get_range_edges(index.min(), index.max(), freq,\n closed=closed,\n base=base)\n datetime_bins = labels = cftime_range(freq=freq,\n start=first,\n end=last,\n name=index.name)\n\n datetime_bins, labels = _adjust_bin_edges(datetime_bins, freq, closed,\n index, labels)\n\n if label == 'right':\n labels = labels[1:]\n else:\n labels = labels[:-1]\n\n # TODO: when CFTimeIndex supports missing values, if the reference index\n # contains missing values, insert the appropriate NaN value at the\n # beginning of the datetime_bins and labels indexes.\n\n return datetime_bins, labels", "response": "Returns the bin labels and the corresponding datetime_bins for resampling operations."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadjusts the bin edges of a single time series to match the given bin frequencies.", "response": "def _adjust_bin_edges(datetime_bins, offset, closed, index, labels):\n \"\"\"This is required for determining the bin edges resampling with\n daily frequencies greater than one day, month end, and year end\n frequencies.\n\n Consider the following example. Let's say you want to downsample the\n time series with the following coordinates to month end frequency:\n\n CFTimeIndex([2000-01-01 12:00:00, 2000-01-31 12:00:00,\n 2000-02-01 12:00:00], dtype='object')\n\n Without this adjustment, _get_time_bins with month-end frequency will\n return the following index for the bin edges (default closed='right' and\n label='right' in this case):\n\n CFTimeIndex([1999-12-31 00:00:00, 2000-01-31 00:00:00,\n 2000-02-29 00:00:00], dtype='object')\n\n If 2000-01-31 is used as a bound for a bin, the value on\n 2000-01-31T12:00:00 (at noon on January 31st), will not be included in the\n month of January. To account for this, pandas adds a day minus one worth\n of microseconds to the bin edges generated by cftime range, so that we do\n bin the value at noon on January 31st in the January bin. This results in\n an index with bin edges like the following:\n\n CFTimeIndex([1999-12-31 23:59:59, 2000-01-31 23:59:59,\n 2000-02-29 23:59:59], dtype='object')\n\n The labels are still:\n\n CFTimeIndex([2000-01-31 00:00:00, 2000-02-29 00:00:00], dtype='object')\n\n This is also required for daily frequencies longer than one day and\n year-end frequencies.\n \"\"\"\n is_super_daily = (isinstance(offset, (MonthEnd, QuarterEnd, YearEnd)) or\n (isinstance(offset, Day) and offset.n > 1))\n if is_super_daily:\n if closed == 'right':\n datetime_bins = datetime_bins + datetime.timedelta(days=1,\n microseconds=-1)\n if datetime_bins[-2] > index.max():\n datetime_bins = datetime_bins[:-1]\n labels = labels[:-1]\n\n return datetime_bins, labels"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the correct starting and ending datetimes for the resampled CFTimeIndex range.", "response": "def _get_range_edges(first, last, offset, closed='left', base=0):\n \"\"\" Get the correct starting and ending datetimes for the resampled\n CFTimeIndex range.\n\n Parameters\n ----------\n first : cftime.datetime\n Uncorrected starting datetime object for resampled CFTimeIndex range.\n Usually the min of the original CFTimeIndex.\n last : cftime.datetime\n Uncorrected ending datetime object for resampled CFTimeIndex range.\n Usually the max of the original CFTimeIndex.\n offset : xarray.coding.cftime_offsets.BaseCFTimeOffset\n The offset object representing target conversion a.k.a. resampling\n frequency. Contains information on offset type (e.g. Day or 'D') and\n offset magnitude (e.g., n = 3).\n closed : 'left' or 'right', optional\n Which side of bin interval is closed. Defaults to 'left'.\n base : int, optional\n For frequencies that evenly subdivide 1 day, the \"origin\" of the\n aggregated intervals. For example, for '5min' frequency, base could\n range from 0 through 4. Defaults to 0.\n\n Returns\n -------\n first : cftime.datetime\n Corrected starting datetime object for resampled CFTimeIndex range.\n last : cftime.datetime\n Corrected ending datetime object for resampled CFTimeIndex range.\n \"\"\"\n if isinstance(offset, CFTIME_TICKS):\n first, last = _adjust_dates_anchored(first, last, offset,\n closed=closed, base=base)\n return first, last\n else:\n first = normalize_date(first)\n last = normalize_date(last)\n\n if closed == 'left':\n first = offset.rollback(first)\n else:\n first = first - offset\n\n last = last + offset\n return first, last"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _adjust_dates_anchored(first, last, offset, closed='right', base=0):\n\n base = base % offset.n\n start_day = normalize_date(first)\n base_td = type(offset)(n=base).as_timedelta()\n start_day += base_td\n foffset = exact_cftime_datetime_difference(\n start_day, first) % offset.as_timedelta()\n loffset = exact_cftime_datetime_difference(\n start_day, last) % offset.as_timedelta()\n if closed == 'right':\n if foffset.total_seconds() > 0:\n fresult = first - foffset\n else:\n fresult = first - offset.as_timedelta()\n\n if loffset.total_seconds() > 0:\n lresult = last + (offset.as_timedelta() - loffset)\n else:\n lresult = last\n else:\n if foffset.total_seconds() > 0:\n fresult = first - foffset\n else:\n fresult = first\n\n if loffset.total_seconds() > 0:\n lresult = last + (offset.as_timedelta() - loffset)\n else:\n lresult = last + offset.as_timedelta()\n return fresult, lresult", "response": "Adjusts the dates of the first and last time series to fix resampling errors."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexact computation of a and b.", "response": "def exact_cftime_datetime_difference(a, b):\n \"\"\"Exact computation of b - a\n\n Assumes:\n\n a = a_0 + a_m\n b = b_0 + b_m\n\n Here a_0, and b_0 represent the input dates rounded\n down to the nearest second, and a_m, and b_m represent\n the remaining microseconds associated with date a and\n date b.\n\n We can then express the value of b - a as:\n\n b - a = (b_0 + b_m) - (a_0 + a_m) = b_0 - a_0 + b_m - a_m\n\n By construction, we know that b_0 - a_0 must be a round number\n of seconds. Therefore we can take the result of b_0 - a_0 using\n ordinary cftime.datetime arithmetic and round to the nearest\n second. b_m - a_m is the remainder, in microseconds, and we\n can simply add this to the rounded timedelta.\n\n Parameters\n ----------\n a : cftime.datetime\n Input datetime\n b : cftime.datetime\n Input datetime\n\n Returns\n -------\n datetime.timedelta\n \"\"\"\n seconds = b.replace(microsecond=0) - a.replace(microsecond=0)\n seconds = int(round(seconds.total_seconds()))\n microseconds = b.microsecond - a.microsecond\n return datetime.timedelta(seconds=seconds, microseconds=microseconds)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef first_items(self, index):\n\n datetime_bins, labels = _get_time_bins(index, self.freq, self.closed,\n self.label, self.base)\n if self.loffset is not None:\n if isinstance(self.loffset, datetime.timedelta):\n labels = labels + self.loffset\n else:\n labels = labels + to_offset(self.loffset)\n\n # check binner fits data\n if index[0] < datetime_bins[0]:\n raise ValueError(\"Value falls before first bin\")\n if index[-1] > datetime_bins[-1]:\n raise ValueError(\"Value falls after last bin\")\n\n integer_bins = np.searchsorted(\n index, datetime_bins, side=self.closed)[:-1]\n first_items = pd.Series(integer_bins, labels)\n\n # Mask duplicate values with NaNs, preserving the last values\n non_duplicate = ~first_items.duplicated('last')\n return first_items.where(non_duplicate)", "response": "Meant to reproduce the results of the following\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef align(*objects, **kwargs):\n join = kwargs.pop('join', 'inner')\n copy = kwargs.pop('copy', True)\n indexes = kwargs.pop('indexes', None)\n exclude = kwargs.pop('exclude', _DEFAULT_EXCLUDE)\n if indexes is None:\n indexes = {}\n if kwargs:\n raise TypeError('align() got unexpected keyword arguments: %s'\n % list(kwargs))\n\n if not indexes and len(objects) == 1:\n # fast path for the trivial case\n obj, = objects\n return (obj.copy(deep=copy),)\n\n all_indexes = defaultdict(list)\n unlabeled_dim_sizes = defaultdict(set)\n for obj in objects:\n for dim in obj.dims:\n if dim not in exclude:\n try:\n index = obj.indexes[dim]\n except KeyError:\n unlabeled_dim_sizes[dim].add(obj.sizes[dim])\n else:\n all_indexes[dim].append(index)\n\n # We don't reindex over dimensions with all equal indexes for two reasons:\n # - It's faster for the usual case (already aligned objects).\n # - It ensures it's possible to do operations that don't require alignment\n # on indexes with duplicate values (which cannot be reindexed with\n # pandas). This is useful, e.g., for overwriting such duplicate indexes.\n joiner = _get_joiner(join)\n joined_indexes = {}\n for dim, matching_indexes in all_indexes.items():\n if dim in indexes:\n index = utils.safe_cast_to_index(indexes[dim])\n if (any(not index.equals(other) for other in matching_indexes) or\n dim in unlabeled_dim_sizes):\n joined_indexes[dim] = index\n else:\n if (any(not matching_indexes[0].equals(other)\n for other in matching_indexes[1:]) or\n dim in unlabeled_dim_sizes):\n if join == 'exact':\n raise ValueError(\n 'indexes along dimension {!r} are not equal'\n .format(dim))\n index = joiner(matching_indexes)\n joined_indexes[dim] = index\n else:\n index = matching_indexes[0]\n\n if dim in unlabeled_dim_sizes:\n unlabeled_sizes = unlabeled_dim_sizes[dim]\n labeled_size = index.size\n if len(unlabeled_sizes | {labeled_size}) > 1:\n raise ValueError(\n 'arguments without labels along dimension %r cannot be '\n 'aligned because they have different dimension size(s) %r '\n 'than the size of the aligned dimension labels: %r'\n % (dim, unlabeled_sizes, labeled_size))\n\n for dim in unlabeled_dim_sizes:\n if dim not in all_indexes:\n sizes = unlabeled_dim_sizes[dim]\n if len(sizes) > 1:\n raise ValueError(\n 'arguments without labels along dimension %r cannot be '\n 'aligned because they have different dimension sizes: %r'\n % (dim, sizes))\n\n result = []\n for obj in objects:\n valid_indexers = {k: v for k, v in joined_indexes.items()\n if k in obj.dims}\n if not valid_indexers:\n # fast path for no reindexing necessary\n new_obj = obj.copy(deep=copy)\n else:\n new_obj = obj.reindex(copy=copy, **valid_indexers)\n new_obj.encoding = obj.encoding\n result.append(new_obj)\n\n return tuple(result)", "response": "Aligns the objects in the input list with the same size as the input list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef deep_align(objects, join='inner', copy=True, indexes=None,\n exclude=frozenset(), raise_on_invalid=True):\n \"\"\"Align objects for merging, recursing into dictionary values.\n\n This function is not public API.\n \"\"\"\n from .dataarray import DataArray\n from .dataset import Dataset\n\n if indexes is None:\n indexes = {}\n\n def is_alignable(obj):\n return isinstance(obj, (DataArray, Dataset))\n\n positions = []\n keys = []\n out = []\n targets = []\n no_key = object()\n not_replaced = object()\n for n, variables in enumerate(objects):\n if is_alignable(variables):\n positions.append(n)\n keys.append(no_key)\n targets.append(variables)\n out.append(not_replaced)\n elif is_dict_like(variables):\n for k, v in variables.items():\n if is_alignable(v) and k not in indexes:\n # Skip variables in indexes for alignment, because these\n # should to be overwritten instead:\n # https://github.com/pydata/xarray/issues/725\n positions.append(n)\n keys.append(k)\n targets.append(v)\n out.append(OrderedDict(variables))\n elif raise_on_invalid:\n raise ValueError('object to align is neither an xarray.Dataset, '\n 'an xarray.DataArray nor a dictionary: %r'\n % variables)\n else:\n out.append(variables)\n\n aligned = align(*targets, join=join, copy=copy, indexes=indexes,\n exclude=exclude)\n\n for position, key, aligned_obj in zip(positions, keys, aligned):\n if key is no_key:\n out[position] = aligned_obj\n else:\n out[position][key] = aligned_obj\n\n # something went wrong: we should have replaced all sentinel values\n assert all(arg is not not_replaced for arg in out)\n\n return out", "response": "Align objects for merging into dictionary values."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reindex_like_indexers(target, other):\n indexers = {k: v for k, v in other.indexes.items() if k in target.dims}\n\n for dim in other.dims:\n if dim not in indexers and dim in target.dims:\n other_size = other.sizes[dim]\n target_size = target.sizes[dim]\n if other_size != target_size:\n raise ValueError('different size for unlabeled '\n 'dimension on argument %r: %r vs %r'\n % (dim, other_size, target_size))\n return indexers", "response": "Extract indexers to align target with other."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reindex_variables(\n variables: Mapping[Any, Variable],\n sizes: Mapping[Any, int],\n indexes: Mapping[Any, pd.Index],\n indexers: Mapping,\n method: Optional[str] = None,\n tolerance: Any = None,\n copy: bool = True,\n) -> 'Tuple[OrderedDict[Any, Variable], OrderedDict[Any, pd.Index]]':\n \"\"\"Conform a dictionary of aligned variables onto a new set of variables,\n filling in missing values with NaN.\n\n Not public API.\n\n Parameters\n ----------\n variables : dict-like\n Dictionary of xarray.Variable objects.\n sizes : dict-like\n Dictionary from dimension names to integer sizes.\n indexes : dict-like\n Dictionary of indexes associated with variables.\n indexers : dict\n Dictionary with keys given by dimension names and values given by\n arrays of coordinates tick labels. Any mis-matched coordinate values\n will be filled in with NaN, and any mis-matched dimension names will\n simply be ignored.\n method : {None, 'nearest', 'pad'/'ffill', 'backfill'/'bfill'}, optional\n Method to use for filling index values in ``indexers`` not found in\n this dataset:\n * None (default): don't fill gaps\n * pad / ffill: propagate last valid index value forward\n * backfill / bfill: propagate next valid index value backward\n * nearest: use nearest valid index value\n tolerance : optional\n Maximum distance between original and new labels for inexact matches.\n The values of the index at the matching locations must satisfy the\n equation ``abs(index[indexer] - target) <= tolerance``.\n copy : bool, optional\n If ``copy=True``, data in the return values is always copied. If\n ``copy=False`` and reindexing is unnecessary, or can be performed\n with only slice operations, then the output may share memory with\n the input. In either case, new xarray objects are always returned.\n\n Returns\n -------\n reindexed : OrderedDict\n Dict of reindexed variables.\n new_indexes : OrderedDict\n Dict of indexes associated with the reindexed variables.\n \"\"\"\n from .dataarray import DataArray\n\n # create variables for the new dataset\n reindexed = OrderedDict() # type: OrderedDict[Any, Variable]\n\n # build up indexers for assignment along each dimension\n int_indexers = {}\n new_indexes = OrderedDict(indexes)\n masked_dims = set()\n unchanged_dims = set()\n\n for dim, indexer in indexers.items():\n if isinstance(indexer, DataArray) and indexer.dims != (dim,):\n warnings.warn(\n \"Indexer has dimensions {0:s} that are different \"\n \"from that to be indexed along {1:s}. \"\n \"This will behave differently in the future.\".format(\n str(indexer.dims), dim),\n FutureWarning, stacklevel=3)\n\n target = new_indexes[dim] = utils.safe_cast_to_index(indexers[dim])\n\n if dim in indexes:\n index = indexes[dim]\n\n if not index.is_unique:\n raise ValueError(\n 'cannot reindex or align along dimension %r because the '\n 'index has duplicate values' % dim)\n\n int_indexer = get_indexer_nd(index, target, method, tolerance)\n\n # We uses negative values from get_indexer_nd to signify\n # values that are missing in the index.\n if (int_indexer < 0).any():\n masked_dims.add(dim)\n elif np.array_equal(int_indexer, np.arange(len(index))):\n unchanged_dims.add(dim)\n\n int_indexers[dim] = int_indexer\n\n if dim in variables:\n var = variables[dim]\n args = (var.attrs, var.encoding) # type: tuple\n else:\n args = ()\n reindexed[dim] = IndexVariable((dim,), target, *args)\n\n for dim in sizes:\n if dim not in indexes and dim in indexers:\n existing_size = sizes[dim]\n new_size = indexers[dim].size\n if existing_size != new_size:\n raise ValueError(\n 'cannot reindex or align along dimension %r without an '\n 'index because its size %r is different from the size of '\n 'the new index %r' % (dim, existing_size, new_size))\n\n for name, var in variables.items():\n if name not in indexers:\n key = tuple(slice(None)\n if d in unchanged_dims\n else int_indexers.get(d, slice(None))\n for d in var.dims)\n needs_masking = any(d in masked_dims for d in var.dims)\n\n if needs_masking:\n new_var = var._getitem_with_mask(key)\n elif all(is_full_slice(k) for k in key):\n # no reindexing necessary\n # here we need to manually deal with copying data, since\n # we neither created a new ndarray nor used fancy indexing\n new_var = var.copy(deep=copy)\n else:\n new_var = var[key]\n\n reindexed[name] = new_var\n\n return reindexed, new_indexes", "response": "Conform a dictionary of aligned variables onto a new set of variables."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef assert_coordinate_consistent(obj, coords):\n for k in obj.dims:\n # make sure there are no conflict in dimension coordinates\n if k in coords and k in obj.coords:\n if not coords[k].equals(obj[k].variable):\n raise IndexError(\n 'dimension coordinate {!r} conflicts between '\n 'indexed and indexing objects:\\n{}\\nvs.\\n{}'\n .format(k, obj[k], coords[k]))", "response": "Maeke sure the dimension coordinate of obj is\n consistent with coords."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remap_label_indexers(obj, indexers=None, method=None, tolerance=None,\n **indexers_kwargs):\n \"\"\"\n Remap **indexers from obj.coords.\n If indexer is an instance of DataArray and it has coordinate, then this\n coordinate will be attached to pos_indexers.\n\n Returns\n -------\n pos_indexers: Same type of indexers.\n np.ndarray or Variable or DataArra\n new_indexes: mapping of new dimensional-coordinate.\n \"\"\"\n from .dataarray import DataArray\n indexers = either_dict_or_kwargs(\n indexers, indexers_kwargs, 'remap_label_indexers')\n\n v_indexers = {k: v.variable.data if isinstance(v, DataArray) else v\n for k, v in indexers.items()}\n\n pos_indexers, new_indexes = indexing.remap_label_indexers(\n obj, v_indexers, method=method, tolerance=tolerance\n )\n # attach indexer's coordinate to pos_indexers\n for k, v in indexers.items():\n if isinstance(v, Variable):\n pos_indexers[k] = Variable(v.dims, pos_indexers[k])\n elif isinstance(v, DataArray):\n # drop coordinates found in indexers since .sel() already\n # ensures alignments\n coords = OrderedDict((k, v) for k, v in v._coords.items()\n if k not in indexers)\n pos_indexers[k] = DataArray(pos_indexers[k],\n coords=coords, dims=v.dims)\n return pos_indexers, new_indexes", "response": "Remap indexers from obj. coords to new dimensions."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert all index coordinates into a pandas. Index.", "response": "def to_index(self, ordered_dims=None):\n \"\"\"Convert all index coordinates into a :py:class:`pandas.Index`.\n\n Parameters\n ----------\n ordered_dims : sequence, optional\n Possibly reordered version of this object's dimensions indicating\n the order in which dimensions should appear on the result.\n\n Returns\n -------\n pandas.Index\n Index subclass corresponding to the outer-product of all dimension\n coordinates. This will be a MultiIndex if this object is has more\n than more dimension.\n \"\"\"\n if ordered_dims is None:\n ordered_dims = self.dims\n elif set(ordered_dims) != set(self.dims):\n raise ValueError('ordered_dims must match dims, but does not: '\n '{} vs {}'.format(ordered_dims, self.dims))\n\n if len(ordered_dims) == 0:\n raise ValueError('no valid index for a 0-dimensional object')\n elif len(ordered_dims) == 1:\n (dim,) = ordered_dims\n return self._data.get_index(dim)\n else:\n indexes = [self._data.get_index(k) for k in ordered_dims]\n names = list(ordered_dims)\n return pd.MultiIndex.from_product(indexes, names=names)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _merge_inplace(self, other):\n if other is None:\n yield\n else:\n # don't include indexes in priority_vars, because we didn't align\n # first\n priority_vars = OrderedDict(\n kv for kv in self.variables.items() if kv[0] not in self.dims)\n variables = merge_coords_for_inplace_math(\n [self.variables, other.variables], priority_vars=priority_vars)\n yield\n self._update_coords(variables)", "response": "For use with in - place binary arithmetic."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmerge two sets of coordinates to create a new Dataset The method implements the logic used for joining coordinates in the result of a binary operation performed on xarray objects: - If two index coordinates conflict (are not equal), an exception is raised. You must align your data before passing it to this method. - If an index coordinate and a non-index coordinate conflict, the non- index coordinate is dropped. - If two non-index coordinates conflict, both are dropped. Parameters ---------- other : DatasetCoordinates or DataArrayCoordinates The coordinates from another dataset or data array. Returns ------- merged : Dataset A new Dataset with merged coordinates.", "response": "def merge(self, other):\n \"\"\"Merge two sets of coordinates to create a new Dataset\n\n The method implements the logic used for joining coordinates in the\n result of a binary operation performed on xarray objects:\n\n - If two index coordinates conflict (are not equal), an exception is\n raised. You must align your data before passing it to this method.\n - If an index coordinate and a non-index coordinate conflict, the non-\n index coordinate is dropped.\n - If two non-index coordinates conflict, both are dropped.\n\n Parameters\n ----------\n other : DatasetCoordinates or DataArrayCoordinates\n The coordinates from another dataset or data array.\n\n Returns\n -------\n merged : Dataset\n A new Dataset with merged coordinates.\n \"\"\"\n from .dataset import Dataset\n\n if other is None:\n return self.to_dataset()\n else:\n other_vars = getattr(other, 'variables', other)\n coords = expand_and_merge_variables([self.variables, other_vars])\n return Dataset._from_vars_and_coord_names(coords, set(coords))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _ipython_key_completions_(self):\n return [key for key in self._data._ipython_key_completions_()\n if key not in self._data.data_vars]", "response": "Provide method for the key - completerions in IPython."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwraps to apply bottleneck moving window funcs on dask arrays", "response": "def dask_rolling_wrapper(moving_func, a, window, min_count=None, axis=-1):\n '''wrapper to apply bottleneck moving window funcs on dask arrays'''\n dtype, fill_value = dtypes.maybe_promote(a.dtype)\n a = a.astype(dtype)\n # inputs for overlap\n if axis < 0:\n axis = a.ndim + axis\n depth = {d: 0 for d in range(a.ndim)}\n depth[axis] = (window + 1) // 2\n boundary = {d: fill_value for d in range(a.ndim)}\n # Create overlap array.\n ag = overlap(a, depth=depth, boundary=boundary)\n # apply rolling func\n out = ag.map_blocks(moving_func, window, min_count=min_count,\n axis=axis, dtype=a.dtype)\n # trim array\n result = trim_internal(out, depth)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rolling_window(a, axis, window, center, fill_value):\n orig_shape = a.shape\n if axis < 0:\n axis = a.ndim + axis\n depth = {d: 0 for d in range(a.ndim)}\n depth[axis] = int(window / 2)\n # For evenly sized window, we need to crop the first point of each block.\n offset = 1 if window % 2 == 0 else 0\n\n if depth[axis] > min(a.chunks[axis]):\n raise ValueError(\n \"For window size %d, every chunk should be larger than %d, \"\n \"but the smallest chunk size is %d. Rechunk your array\\n\"\n \"with a larger chunk size or a chunk size that\\n\"\n \"more evenly divides the shape of your array.\" %\n (window, depth[axis], min(a.chunks[axis])))\n\n # Although dask.overlap pads values to boundaries of the array,\n # the size of the generated array is smaller than what we want\n # if center == False.\n if center:\n start = int(window / 2) # 10 -> 5, 9 -> 4\n end = window - 1 - start\n else:\n start, end = window - 1, 0\n pad_size = max(start, end) + offset - depth[axis]\n drop_size = 0\n # pad_size becomes more than 0 when the overlapped array is smaller than\n # needed. In this case, we need to enlarge the original array by padding\n # before overlapping.\n if pad_size > 0:\n if pad_size < depth[axis]:\n # overlapping requires each chunk larger than depth. If pad_size is\n # smaller than the depth, we enlarge this and truncate it later.\n drop_size = depth[axis] - pad_size\n pad_size = depth[axis]\n shape = list(a.shape)\n shape[axis] = pad_size\n chunks = list(a.chunks)\n chunks[axis] = (pad_size, )\n fill_array = da.full(shape, fill_value, dtype=a.dtype, chunks=chunks)\n a = da.concatenate([fill_array, a], axis=axis)\n\n boundary = {d: fill_value for d in range(a.ndim)}\n\n # create overlap arrays\n ag = overlap(a, depth=depth, boundary=boundary)\n\n # apply rolling func\n def func(x, window, axis=-1):\n x = np.asarray(x)\n rolling = nputils._rolling_window(x, window, axis)\n return rolling[(slice(None), ) * axis + (slice(offset, None), )]\n\n chunks = list(a.chunks)\n chunks.append(window)\n out = ag.map_blocks(func, dtype=a.dtype, new_axis=a.ndim, chunks=chunks,\n window=window, axis=axis)\n\n # crop boundary.\n index = (slice(None),) * axis + (slice(drop_size,\n drop_size + orig_shape[axis]), )\n return out[index]", "response": "Dask s equivalence to np. utils. rolling_window"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndetermining the size of the zarr array based on the encoding and variable data.", "response": "def _determine_zarr_chunks(enc_chunks, var_chunks, ndim):\n \"\"\"\n Given encoding chunks (possibly None) and variable chunks (possibly None)\n \"\"\"\n\n # zarr chunk spec:\n # chunks : int or tuple of ints, optional\n # Chunk shape. If not provided, will be guessed from shape and dtype.\n\n # if there are no chunks in encoding and the variable data is a numpy\n # array, then we let zarr use its own heuristics to pick the chunks\n if var_chunks is None and enc_chunks is None:\n return None\n\n # if there are no chunks in encoding but there are dask chunks, we try to\n # use the same chunks in zarr\n # However, zarr chunks needs to be uniform for each array\n # http://zarr.readthedocs.io/en/latest/spec/v1.html#chunks\n # while dask chunks can be variable sized\n # http://dask.pydata.org/en/latest/array-design.html#chunks\n if var_chunks and enc_chunks is None:\n if any(len(set(chunks[:-1])) > 1 for chunks in var_chunks):\n raise ValueError(\n \"Zarr requires uniform chunk sizes except for final chunk.\"\n \" Variable dask chunks %r are incompatible. Consider \"\n \"rechunking using `chunk()`.\" % (var_chunks,))\n if any((chunks[0] < chunks[-1]) for chunks in var_chunks):\n raise ValueError(\n \"Final chunk of Zarr array must be the same size or smaller \"\n \"than the first. Variable Dask chunks %r are incompatible. \"\n \"Consider rechunking using `chunk()`.\" % var_chunks)\n # return the first chunk for each dimension\n return tuple(chunk[0] for chunk in var_chunks)\n\n # from here on, we are dealing with user-specified chunks in encoding\n # zarr allows chunks to be an integer, in which case it uses the same chunk\n # size on each dimension.\n # Here we re-implement this expansion ourselves. That makes the logic of\n # checking chunk compatibility easier\n\n if isinstance(enc_chunks, integer_types):\n enc_chunks_tuple = ndim * (enc_chunks,)\n else:\n enc_chunks_tuple = tuple(enc_chunks)\n\n if len(enc_chunks_tuple) != ndim:\n # throw away encoding chunks, start over\n return _determine_zarr_chunks(None, var_chunks, ndim)\n\n for x in enc_chunks_tuple:\n if not isinstance(x, int):\n raise TypeError(\"zarr chunks must be an int or a tuple of ints. \"\n \"Instead found %r\" % (enc_chunks_tuple,))\n\n # if there are chunks in encoding and the variable data is a numpy array,\n # we use the specified chunks\n if var_chunks is None:\n return enc_chunks_tuple\n\n # the hard case\n # DESIGN CHOICE: do not allow multiple dask chunks on a single zarr chunk\n # this avoids the need to get involved in zarr synchronization / locking\n # From zarr docs:\n # \"If each worker in a parallel computation is writing to a separate\n # region of the array, and if region boundaries are perfectly aligned\n # with chunk boundaries, then no synchronization is required.\"\n # TODO: incorporate synchronizer to allow writes from multiple dask\n # threads\n if var_chunks and enc_chunks_tuple:\n for zchunk, dchunks in zip(enc_chunks_tuple, var_chunks):\n for dchunk in dchunks[:-1]:\n if dchunk % zchunk:\n raise NotImplementedError(\n \"Specified zarr chunks %r would overlap multiple dask \"\n \"chunks %r. This is not implemented in xarray yet. \"\n \" Consider rechunking the data using \"\n \"`chunk()` or specifying different chunks in encoding.\"\n % (enc_chunks_tuple, var_chunks))\n if dchunks[-1] > zchunk:\n raise ValueError(\n \"Final chunk of Zarr array must be the same size or \"\n \"smaller than the first. The specified Zarr chunk \"\n \"encoding is %r, but %r in variable Dask chunks %r is \"\n \"incompatible. Consider rechunking using `chunk()`.\"\n % (enc_chunks_tuple, dchunks, var_chunks))\n return enc_chunks_tuple\n\n raise AssertionError(\n \"We should never get here. Function logic must be wrong.\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef encode_zarr_variable(var, needs_copy=True, name=None):\n\n var = conventions.encode_cf_variable(var, name=name)\n\n # zarr allows unicode, but not variable-length strings, so it's both\n # simpler and more compact to always encode as UTF-8 explicitly.\n # TODO: allow toggling this explicitly via dtype in encoding.\n coder = coding.strings.EncodedStringCoder(allows_unicode=False)\n var = coder.encode(var, name=name)\n var = coding.strings.ensure_fixed_length_bytes(var)\n\n return var", "response": "Encodes a variable into a zarr - compatible variable."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads and decode a dataset from a Zarr store.", "response": "def open_zarr(store, group=None, synchronizer=None, chunks='auto',\n decode_cf=True, mask_and_scale=True, decode_times=True,\n concat_characters=True, decode_coords=True,\n drop_variables=None, consolidated=False,\n overwrite_encoded_chunks=False, **kwargs):\n \"\"\"Load and decode a dataset from a Zarr store.\n\n .. note:: Experimental\n The Zarr backend is new and experimental. Please report any\n unexpected behavior via github issues.\n\n The `store` object should be a valid store for a Zarr group. `store`\n variables must contain dimension metadata encoded in the\n `_ARRAY_DIMENSIONS` attribute.\n\n Parameters\n ----------\n store : MutableMapping or str\n A MutableMapping where a Zarr Group has been stored or a path to a\n directory in file system where a Zarr DirectoryStore has been stored.\n synchronizer : object, optional\n Array synchronizer provided to zarr\n group : str, obtional\n Group path. (a.k.a. `path` in zarr terminology.)\n chunks : int or dict or tuple or {None, 'auto'}, optional\n Chunk sizes along each dimension, e.g., ``5`` or\n ``{'x': 5, 'y': 5}``. If `chunks='auto'`, dask chunks are created\n based on the variable's zarr chunks. If `chunks=None`, zarr array\n data will lazily convert to numpy arrays upon access. This accepts\n all the chunk specifications as Dask does.\n overwrite_encoded_chunks: bool, optional\n Whether to drop the zarr chunks encoded for each variable when a\n dataset is loaded with specified chunk sizes (default: False)\n decode_cf : bool, optional\n Whether to decode these variables, assuming they were saved according\n to CF conventions.\n mask_and_scale : bool, optional\n If True, replace array values equal to `_FillValue` with NA and scale\n values according to the formula `original_values * scale_factor +\n add_offset`, where `_FillValue`, `scale_factor` and `add_offset` are\n taken from variable attributes (if they exist). If the `_FillValue` or\n `missing_value` attribute contains multiple values a warning will be\n issued and all array values matching one of the multiple values will\n be replaced by NA.\n decode_times : bool, optional\n If True, decode times encoded in the standard NetCDF datetime format\n into datetime objects. Otherwise, leave them encoded as numbers.\n concat_characters : bool, optional\n If True, concatenate along the last dimension of character arrays to\n form string arrays. Dimensions will only be concatenated over (and\n removed) if they have no corresponding variable and if they are only\n used as the last dimension of character arrays.\n decode_coords : bool, optional\n If True, decode the 'coordinates' attribute to identify coordinates in\n the resulting dataset.\n drop_variables : string or iterable, optional\n A variable or list of variables to exclude from being parsed from the\n dataset. This may be useful to drop variables with problems or\n inconsistent values.\n consolidated : bool, optional\n Whether to open the store using zarr's consolidated metadata\n capability. Only works for stores that have already been consolidated.\n\n Returns\n -------\n dataset : Dataset\n The newly created dataset.\n\n See Also\n --------\n open_dataset\n\n References\n ----------\n http://zarr.readthedocs.io/\n \"\"\"\n if 'auto_chunk' in kwargs:\n auto_chunk = kwargs.pop('auto_chunk')\n if auto_chunk:\n chunks = 'auto' # maintain backwards compatibility\n else:\n chunks = None\n\n warnings.warn(\"auto_chunk is deprecated. Use chunks='auto' instead.\",\n FutureWarning, stacklevel=2)\n\n if kwargs:\n raise TypeError(\"open_zarr() got unexpected keyword arguments \" +\n \",\".join(kwargs.keys()))\n\n if not isinstance(chunks, (int, dict)):\n if chunks != 'auto' and chunks is not None:\n raise ValueError(\"chunks must be an int, dict, 'auto', or None. \"\n \"Instead found %s. \" % chunks)\n\n if not decode_cf:\n mask_and_scale = False\n decode_times = False\n concat_characters = False\n decode_coords = False\n\n def maybe_decode_store(store, lock=False):\n ds = conventions.decode_cf(\n store, mask_and_scale=mask_and_scale, decode_times=decode_times,\n concat_characters=concat_characters, decode_coords=decode_coords,\n drop_variables=drop_variables)\n\n # TODO: this is where we would apply caching\n\n return ds\n\n # Zarr supports a wide range of access modes, but for now xarray either\n # reads or writes from a store, never both. For open_zarr, we only read\n mode = 'r'\n zarr_store = ZarrStore.open_group(store, mode=mode,\n synchronizer=synchronizer,\n group=group, consolidated=consolidated)\n ds = maybe_decode_store(zarr_store)\n\n # auto chunking needs to be here and not in ZarrStore because variable\n # chunks do not survive decode_cf\n # return trivial case\n if not chunks:\n return ds\n\n # adapted from Dataset.Chunk()\n if isinstance(chunks, int):\n chunks = dict.fromkeys(ds.dims, chunks)\n\n if isinstance(chunks, tuple) and len(chunks) == len(ds.dims):\n chunks = dict(zip(ds.dims, chunks))\n\n def get_chunk(name, var, chunks):\n chunk_spec = dict(zip(var.dims, var.encoding.get('chunks')))\n\n # Coordinate labels aren't chunked\n if var.ndim == 1 and var.dims[0] == name:\n return chunk_spec\n\n if chunks == 'auto':\n return chunk_spec\n\n for dim in var.dims:\n if dim in chunks:\n spec = chunks[dim]\n if isinstance(spec, int):\n spec = (spec,)\n if isinstance(spec, (tuple, list)) and chunk_spec[dim]:\n if any(s % chunk_spec[dim] for s in spec):\n warnings.warn(\"Specified Dask chunks %r would \"\n \"separate Zarr chunk shape %r for \"\n \"dimension %r. This significantly \"\n \"degrades performance. Consider \"\n \"rechunking after loading instead.\"\n % (chunks[dim], chunk_spec[dim], dim),\n stacklevel=2)\n chunk_spec[dim] = chunks[dim]\n return chunk_spec\n\n def maybe_chunk(name, var, chunks):\n from dask.base import tokenize\n\n chunk_spec = get_chunk(name, var, chunks)\n\n if (var.ndim > 0) and (chunk_spec is not None):\n # does this cause any data to be read?\n token2 = tokenize(name, var._data)\n name2 = 'zarr-%s' % token2\n var = var.chunk(chunk_spec, name=name2, lock=None)\n if overwrite_encoded_chunks and var.chunks is not None:\n var.encoding['chunks'] = tuple(x[0] for x in var.chunks)\n return var\n else:\n return var\n\n variables = OrderedDict([(k, maybe_chunk(k, v, chunks))\n for k, v in ds.variables.items()])\n return ds._replace_vars_and_dims(variables)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nextracting dimension sizes from a dictionary of variables.", "response": "def broadcast_dimension_size(\n variables: List[Variable],\n) -> 'OrderedDict[Any, int]':\n \"\"\"Extract dimension sizes from a dictionary of variables.\n\n Raises ValueError if any dimensions have different sizes.\n \"\"\"\n dims = OrderedDict() # type: OrderedDict[Any, int]\n for var in variables:\n for dim, size in zip(var.dims, var.shape):\n if dim in dims and size != dims[dim]:\n raise ValueError('index %r not aligned' % dim)\n dims[dim] = size\n return dims"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef unique_variable(name, variables, compat='broadcast_equals'):\n # type: (Any, List[Variable], str) -> Variable\n \"\"\"Return the unique variable from a list of variables or raise MergeError.\n\n Parameters\n ----------\n name : hashable\n Name for this variable.\n variables : list of xarray.Variable\n List of Variable objects, all of which go by the same name in different\n inputs.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n Type of equality check to use.\n\n Returns\n -------\n Variable to use in the result.\n\n Raises\n ------\n MergeError: if any of the variables are not equal.\n \"\"\" # noqa\n out = variables[0]\n if len(variables) > 1:\n combine_method = None\n\n if compat == 'minimal':\n compat = 'broadcast_equals'\n\n if compat == 'broadcast_equals':\n dim_lengths = broadcast_dimension_size(variables)\n out = out.set_dims(dim_lengths)\n\n if compat == 'no_conflicts':\n combine_method = 'fillna'\n\n for var in variables[1:]:\n if not getattr(out, compat)(var):\n raise MergeError('conflicting values for variable %r on '\n 'objects to be combined:\\n'\n 'first value: %r\\nsecond value: %r'\n % (name, out, var))\n if combine_method:\n # TODO: add preservation of attrs into fillna\n out = getattr(out, combine_method)(var)\n out.attrs = var.attrs\n\n return out", "response": "Returns the unique variable from a list of variables."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge_variables(\n list_of_variables_dicts, # type: List[Mapping[Any, Variable]]\n priority_vars=None, # type: Optional[Mapping[Any, Variable]]\n compat='minimal', # type: str\n):\n # type: (...) -> OrderedDict[Any, Variable]\n \"\"\"Merge dicts of variables, while resolving conflicts appropriately.\n\n Parameters\n ----------\n lists_of_variables_dicts : list of mappings with Variable values\n List of mappings for which each value is a xarray.Variable object.\n priority_vars : mapping with Variable or None values, optional\n If provided, variables are always taken from this dict in preference to\n the input variable dictionaries, without checking for conflicts.\n compat : {'identical', 'equals', 'broadcast_equals', 'minimal', 'no_conflicts'}, optional\n Type of equality check to use when checking for conflicts.\n\n Returns\n -------\n OrderedDict with keys taken by the union of keys on list_of_variable_dicts,\n and Variable values corresponding to those that should be found on the\n merged result.\n \"\"\" # noqa\n if priority_vars is None:\n priority_vars = {}\n\n _assert_compat_valid(compat)\n dim_compat = min(compat, 'equals', key=_VALID_COMPAT.get)\n\n lookup = OrderedDefaultDict(list)\n for variables in list_of_variables_dicts:\n for name, var in variables.items():\n lookup[name].append(var)\n\n # n.b. it's important to fill up merged in the original order in which\n # variables appear\n merged = OrderedDict() # type: OrderedDict[Any, Variable]\n\n for name, var_list in lookup.items():\n if name in priority_vars:\n # one of these arguments (e.g., the first for in-place arithmetic\n # or the second for Dataset.update) takes priority\n merged[name] = priority_vars[name]\n else:\n dim_variables = [var for var in var_list if (name,) == var.dims]\n if dim_variables:\n # if there are dimension coordinates, these must be equal (or\n # identical), and they take priority over non-dimension\n # coordinates\n merged[name] = unique_variable(name, dim_variables, dim_compat)\n else:\n try:\n merged[name] = unique_variable(name, var_list, compat)\n except MergeError:\n if compat != 'minimal':\n # we need more than \"minimal\" compatibility (for which\n # we drop conflicting coordinates)\n raise\n\n return merged", "response": "Merges the variables in the list of variables_dicts into a single dict."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a list of dicts with xarray object values expand the values.", "response": "def expand_variable_dicts(\n list_of_variable_dicts: 'List[Union[Dataset, OrderedDict]]',\n) -> 'List[Mapping[Any, Variable]]':\n \"\"\"Given a list of dicts with xarray object values, expand the values.\n\n Parameters\n ----------\n list_of_variable_dicts : list of dict or Dataset objects\n Each value for the mappings must be of the following types:\n - an xarray.Variable\n - a tuple `(dims, data[, attrs[, encoding]])` that can be converted in\n an xarray.Variable\n - or an xarray.DataArray\n\n Returns\n -------\n A list of ordered dictionaries corresponding to inputs, or coordinates from\n an input's values. The values of each ordered dictionary are all\n xarray.Variable objects.\n \"\"\"\n from .dataarray import DataArray\n from .dataset import Dataset\n\n var_dicts = []\n\n for variables in list_of_variable_dicts:\n if isinstance(variables, Dataset):\n var_dicts.append(variables.variables)\n continue\n\n # append coords to var_dicts before appending sanitized_vars,\n # because we want coords to appear first\n sanitized_vars = OrderedDict() # type: OrderedDict[Any, Variable]\n\n for name, var in variables.items():\n if isinstance(var, DataArray):\n # use private API for speed\n coords = var._coords.copy()\n # explicitly overwritten variables should take precedence\n coords.pop(name, None)\n var_dicts.append(coords)\n\n var = as_variable(var, name=name)\n sanitized_vars[name] = var\n\n var_dicts.append(sanitized_vars)\n\n return var_dicts"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a list of dicts with xarray object values identify coordinates.", "response": "def determine_coords(list_of_variable_dicts):\n # type: (List[Dict]) -> Tuple[Set, Set]\n \"\"\"Given a list of dicts with xarray object values, identify coordinates.\n\n Parameters\n ----------\n list_of_variable_dicts : list of dict or Dataset objects\n Of the same form as the arguments to expand_variable_dicts.\n\n Returns\n -------\n coord_names : set of variable names\n noncoord_names : set of variable names\n All variable found in the input should appear in either the set of\n coordinate or non-coordinate names.\n \"\"\"\n from .dataarray import DataArray\n from .dataset import Dataset\n\n coord_names = set() # type: set\n noncoord_names = set() # type: set\n\n for variables in list_of_variable_dicts:\n if isinstance(variables, Dataset):\n coord_names.update(variables.coords)\n noncoord_names.update(variables.data_vars)\n else:\n for name, var in variables.items():\n if isinstance(var, DataArray):\n coords = set(var._coords) # use private API for speed\n # explicitly overwritten variables should take precedence\n coords.discard(name)\n coord_names.update(coords)\n\n return coord_names, noncoord_names"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef coerce_pandas_values(objects):\n from .dataset import Dataset\n from .dataarray import DataArray\n\n out = []\n for obj in objects:\n if isinstance(obj, Dataset):\n variables = obj\n else:\n variables = OrderedDict()\n if isinstance(obj, PANDAS_TYPES):\n obj = OrderedDict(obj.iteritems())\n for k, v in obj.items():\n if isinstance(v, PANDAS_TYPES):\n v = DataArray(v)\n variables[k] = v\n out.append(variables)\n return out", "response": "Convert pandas values found in a list of labeled objects into native xarray objects."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmerge coordinate variables without worrying about alignment. This function is used for merging variables in coordinates.py.", "response": "def merge_coords_for_inplace_math(objs, priority_vars=None):\n \"\"\"Merge coordinate variables without worrying about alignment.\n\n This function is used for merging variables in coordinates.py.\n \"\"\"\n expanded = expand_variable_dicts(objs)\n variables = merge_variables(expanded, priority_vars)\n assert_unique_multiindex_level_names(variables)\n return variables"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nextracting the priority variable from a list of mappings. We need this method because in some cases the priority argument itself might have conflicting values (e.g., if it is a dict with two DataArray values with conflicting coordinate values). Parameters ---------- objects : list of dictionaries of variables Dictionaries in which to find the priority variables. priority_arg : int or None Integer object whose variable should take priority. compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional Compatibility checks to use when merging variables. Returns ------- None, if priority_arg is None, or an OrderedDict with Variable objects as values indicating priority variables.", "response": "def _get_priority_vars(objects, priority_arg, compat='equals'):\n \"\"\"Extract the priority variable from a list of mappings.\n\n We need this method because in some cases the priority argument itself\n might have conflicting values (e.g., if it is a dict with two DataArray\n values with conflicting coordinate values).\n\n Parameters\n ----------\n objects : list of dictionaries of variables\n Dictionaries in which to find the priority variables.\n priority_arg : int or None\n Integer object whose variable should take priority.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n Compatibility checks to use when merging variables.\n\n Returns\n -------\n None, if priority_arg is None, or an OrderedDict with Variable objects as\n values indicating priority variables.\n \"\"\" # noqa\n if priority_arg is None:\n priority_vars = {}\n else:\n expanded = expand_variable_dicts([objects[priority_arg]])\n priority_vars = merge_variables(expanded, compat=compat)\n return priority_vars"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexpanding variables with worrying about alignment.", "response": "def expand_and_merge_variables(objs, priority_arg=None):\n \"\"\"Merge coordinate variables without worrying about alignment.\n\n This function is used for merging variables in computation.py.\n \"\"\"\n expanded = expand_variable_dicts(objs)\n priority_vars = _get_priority_vars(objs, priority_arg)\n variables = merge_variables(expanded, priority_vars)\n return variables"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmerging coordinate variables. See merge_core below for argument descriptions. This works similarly to merge_core, except everything we don't worry about whether variables are coordinates or not.", "response": "def merge_coords(objs, compat='minimal', join='outer', priority_arg=None,\n indexes=None):\n \"\"\"Merge coordinate variables.\n\n See merge_core below for argument descriptions. This works similarly to\n merge_core, except everything we don't worry about whether variables are\n coordinates or not.\n \"\"\"\n _assert_compat_valid(compat)\n coerced = coerce_pandas_values(objs)\n aligned = deep_align(coerced, join=join, copy=False, indexes=indexes)\n expanded = expand_variable_dicts(aligned)\n priority_vars = _get_priority_vars(aligned, priority_arg, compat=compat)\n variables = merge_variables(expanded, priority_vars, compat=compat)\n assert_unique_multiindex_level_names(variables)\n\n return variables"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses in Dataset. __init__.", "response": "def merge_data_and_coords(data, coords, compat='broadcast_equals',\n join='outer'):\n \"\"\"Used in Dataset.__init__.\"\"\"\n objs = [data, coords]\n explicit_coords = coords.keys()\n indexes = dict(extract_indexes(coords))\n return merge_core(objs, compat, join, explicit_coords=explicit_coords,\n indexes=indexes)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nyields the name & index of valid indexes from a mapping of coords", "response": "def extract_indexes(coords):\n \"\"\"Yields the name & index of valid indexes from a mapping of coords\"\"\"\n for name, variable in coords.items():\n variable = as_variable(variable, name=name)\n if variable.dims == (name,):\n yield name, variable.to_index()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvalidating explicit coordinate names/dims. Raise a MergeError if an explicit coord shares a name with a dimension but is comprised of arbitrary dimensions.", "response": "def assert_valid_explicit_coords(variables, dims, explicit_coords):\n \"\"\"Validate explicit coordinate names/dims.\n\n Raise a MergeError if an explicit coord shares a name with a dimension\n but is comprised of arbitrary dimensions.\n \"\"\"\n for coord_name in explicit_coords:\n if coord_name in dims and variables[coord_name].dims != (coord_name,):\n raise MergeError(\n 'coordinate %s shares a name with a dataset dimension, but is '\n 'not a 1D variable along that dimension. This is disallowed '\n 'by the xarray data model.' % coord_name)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef merge_core(objs,\n compat='broadcast_equals',\n join='outer',\n priority_arg=None,\n explicit_coords=None,\n indexes=None):\n \"\"\"Core logic for merging labeled objects.\n\n This is not public API.\n\n Parameters\n ----------\n objs : list of mappings\n All values must be convertable to labeled arrays.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n Compatibility checks to use when merging variables.\n join : {'outer', 'inner', 'left', 'right'}, optional\n How to combine objects with different indexes.\n priority_arg : integer, optional\n Optional argument in `objs` that takes precedence over the others.\n explicit_coords : set, optional\n An explicit list of variables from `objs` that are coordinates.\n indexes : dict, optional\n Dictionary with values given by pandas.Index objects.\n\n Returns\n -------\n variables : OrderedDict\n Ordered dictionary of Variable objects.\n coord_names : set\n Set of coordinate names.\n dims : dict\n Dictionary mapping from dimension names to sizes.\n\n Raises\n ------\n MergeError if the merge cannot be done successfully.\n \"\"\" # noqa\n from .dataset import calculate_dimensions\n\n _assert_compat_valid(compat)\n\n coerced = coerce_pandas_values(objs)\n aligned = deep_align(coerced, join=join, copy=False, indexes=indexes)\n expanded = expand_variable_dicts(aligned)\n\n coord_names, noncoord_names = determine_coords(coerced)\n\n priority_vars = _get_priority_vars(aligned, priority_arg, compat=compat)\n variables = merge_variables(expanded, priority_vars, compat=compat)\n assert_unique_multiindex_level_names(variables)\n\n dims = calculate_dimensions(variables)\n\n if explicit_coords is not None:\n assert_valid_explicit_coords(variables, dims, explicit_coords)\n coord_names.update(explicit_coords)\n\n for dim, size in dims.items():\n if dim in variables:\n coord_names.add(dim)\n\n ambiguous_coords = coord_names.intersection(noncoord_names)\n if ambiguous_coords:\n raise MergeError('unable to determine if these variables should be '\n 'coordinates or not in the merged result: %s'\n % ambiguous_coords)\n\n return variables, coord_names, dict(dims)", "response": "This function is not public API. It will merge labeled arrays."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmerges any number of xarray objects into a single Dataset.", "response": "def merge(objects, compat='no_conflicts', join='outer'):\n \"\"\"Merge any number of xarray objects into a single Dataset as variables.\n\n Parameters\n ----------\n objects : Iterable[Union[xarray.Dataset, xarray.DataArray, dict]]\n Merge together all variables from these objects. If any of them are\n DataArray objects, they must have a name.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n String indicating how to compare variables of the same name for\n potential conflicts:\n\n - 'broadcast_equals': all values must be equal when variables are\n broadcast against each other to ensure common dimensions.\n - 'equals': all values and dimensions must be the same.\n - 'identical': all values, dimensions and attributes must be the\n same.\n - 'no_conflicts': only values which are not null in both datasets\n must be equal. The returned dataset then contains the combination\n of all non-null values.\n join : {'outer', 'inner', 'left', 'right', 'exact'}, optional\n How to combine objects with different indexes.\n\n Returns\n -------\n Dataset\n Dataset with combined variables from each object.\n\n Examples\n --------\n >>> arrays = [xr.DataArray(n, name='var%d' % n) for n in range(5)]\n >>> xr.merge(arrays)\n \n Dimensions: ()\n Coordinates:\n *empty*\n Data variables:\n var0 int64 0\n var1 int64 1\n var2 int64 2\n var3 int64 3\n var4 int64 4\n\n Raises\n ------\n xarray.MergeError\n If any variables with the same name have conflicting values.\n\n See also\n --------\n concat\n \"\"\" # noqa\n from .dataarray import DataArray\n from .dataset import Dataset\n\n dict_like_objects = [\n obj.to_dataset() if isinstance(obj, DataArray) else obj\n for obj in objects]\n\n variables, coord_names, dims = merge_core(dict_like_objects, compat, join)\n # TODO: don't always recompute indexes\n merged = Dataset._construct_direct(\n variables, coord_names, dims, indexes=None)\n\n return merged"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dataset_merge_method(dataset, other, overwrite_vars, compat, join):\n\n # we are locked into supporting overwrite_vars for the Dataset.merge\n # method due for backwards compatibility\n # TODO: consider deprecating it?\n\n if isinstance(overwrite_vars, str):\n overwrite_vars = set([overwrite_vars])\n overwrite_vars = set(overwrite_vars)\n\n if not overwrite_vars:\n objs = [dataset, other]\n priority_arg = None\n elif overwrite_vars == set(other):\n objs = [dataset, other]\n priority_arg = 1\n else:\n other_overwrite = OrderedDict()\n other_no_overwrite = OrderedDict()\n for k, v in other.items():\n if k in overwrite_vars:\n other_overwrite[k] = v\n else:\n other_no_overwrite[k] = v\n objs = [dataset, other_no_overwrite, other_overwrite]\n priority_arg = 2\n\n return merge_core(objs, compat, join, priority_arg=priority_arg)", "response": "Guts of the Dataset. merge method."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dataset_update_method(dataset, other):\n from .dataset import Dataset\n from .dataarray import DataArray\n\n if not isinstance(other, Dataset):\n other = OrderedDict(other)\n for key, value in other.items():\n if isinstance(value, DataArray):\n # drop conflicting coordinates\n coord_names = [c for c in value.coords\n if c not in value.dims and c in dataset.coords]\n if coord_names:\n other[key] = value.drop(coord_names)\n\n return merge_core([dataset, other], priority_arg=1,\n indexes=dataset.indexes)", "response": "Guts of the Dataset. update method."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreplaces nan in a by val and returns the replaced array and the nan position", "response": "def _replace_nan(a, val):\n \"\"\"\n replace nan in a by val, and returns the replaced array and the nan\n position\n \"\"\"\n mask = isnull(a)\n return where_method(val, mask, a), mask"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _maybe_null_out(result, axis, mask, min_count=1):\n if hasattr(axis, '__len__'): # if tuple or list\n raise ValueError('min_count is not available for reduction '\n 'with more than one dimensions.')\n\n if axis is not None and getattr(result, 'ndim', False):\n null_mask = (mask.shape[axis] - mask.sum(axis) - min_count) < 0\n if null_mask.any():\n dtype, fill_value = dtypes.maybe_promote(result.dtype)\n result = result.astype(dtype)\n result[null_mask] = fill_value\n\n elif getattr(result, 'dtype', None) not in dtypes.NAT_TYPES:\n null_mask = mask.size - mask.sum()\n if null_mask < min_count:\n result = np.nan\n\n return result", "response": "Internal function to handle null out of an array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _disable_auto_decode_variable(var):\n var.set_auto_maskandscale(False)\n\n # only added in netCDF4-python v1.2.8\n with suppress(AttributeError):\n var.set_auto_chartostring(False)", "response": "Disable automatic decoding on a netCDF4. Variable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning an appropriate positive infinity value for this dtype.", "response": "def get_pos_infinity(dtype):\n \"\"\"Return an appropriate positive infinity for this dtype.\n\n Parameters\n ----------\n dtype : np.dtype\n\n Returns\n -------\n fill_value : positive infinity value corresponding to this dtype.\n \"\"\"\n if issubclass(dtype.type, (np.floating, np.integer)):\n return np.inf\n\n if issubclass(dtype.type, np.complexfloating):\n return np.inf + 1j * np.inf\n\n return INF"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_neg_infinity(dtype):\n if issubclass(dtype.type, (np.floating, np.integer)):\n return -np.inf\n\n if issubclass(dtype.type, np.complexfloating):\n return -np.inf - 1j * np.inf\n\n return NINF", "response": "Return an appropriate positive infinity value for this dtype."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_datetime_like(dtype):\n return (np.issubdtype(dtype, np.datetime64) or\n np.issubdtype(dtype, np.timedelta64))", "response": "Check if a dtype is a subclass of the numpy datetime types\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef result_type(*arrays_and_dtypes):\n types = {np.result_type(t).type for t in arrays_and_dtypes}\n\n for left, right in PROMOTE_TO_OBJECT:\n if (any(issubclass(t, left) for t in types) and\n any(issubclass(t, right) for t in types)):\n return np.dtype(object)\n\n return np.result_type(*arrays_and_dtypes)", "response": "Like np. result_type but with type promotion rules matching pandas."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse_envi(meta):\n\n def parsevec(s):\n return np.fromstring(s.strip('{}'), dtype='float', sep=',')\n\n def default(s):\n return s.strip('{}')\n\n parse = {'wavelength': parsevec,\n 'fwhm': parsevec}\n parsed_meta = {k: parse.get(k, default)(v) for k, v in meta.items()}\n return parsed_meta", "response": "Parse the ENVI metadata into a dictionary of dicts."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef open_rasterio(filename, parse_coordinates=None, chunks=None, cache=None,\n lock=None):\n \"\"\"Open a file with rasterio (experimental).\n\n This should work with any file that rasterio can open (most often:\n geoTIFF). The x and y coordinates are generated automatically from the\n file's geoinformation, shifted to the center of each pixel (see\n `\"PixelIsArea\" Raster Space\n `_\n for more information).\n\n You can generate 2D coordinates from the file's attributes with::\n\n from affine import Affine\n da = xr.open_rasterio('path_to_file.tif')\n transform = Affine.from_gdal(*da.attrs['transform'])\n nx, ny = da.sizes['x'], da.sizes['y']\n x, y = np.meshgrid(np.arange(nx)+0.5, np.arange(ny)+0.5) * transform\n\n\n Parameters\n ----------\n filename : str, rasterio.DatasetReader, or rasterio.WarpedVRT\n Path to the file to open. Or already open rasterio dataset.\n parse_coordinates : bool, optional\n Whether to parse the x and y coordinates out of the file's\n ``transform`` attribute or not. The default is to automatically\n parse the coordinates only if they are rectilinear (1D).\n It can be useful to set ``parse_coordinates=False``\n if your files are very large or if you don't need the coordinates.\n chunks : int, tuple or dict, optional\n Chunk sizes along each dimension, e.g., ``5``, ``(5, 5)`` or\n ``{'x': 5, 'y': 5}``. If chunks is provided, it used to load the new\n DataArray into a dask array.\n cache : bool, optional\n If True, cache data loaded from the underlying datastore in memory as\n NumPy arrays when accessed to avoid reading from the underlying data-\n store multiple times. Defaults to True unless you specify the `chunks`\n argument to use dask, in which case it defaults to False.\n lock : False, True or threading.Lock, optional\n If chunks is provided, this argument is passed on to\n :py:func:`dask.array.from_array`. By default, a global lock is\n used to avoid issues with concurrent access to the same file when using\n dask's multithreaded backend.\n\n Returns\n -------\n data : DataArray\n The newly created DataArray.\n \"\"\"\n import rasterio\n from rasterio.vrt import WarpedVRT\n vrt_params = None\n if isinstance(filename, rasterio.io.DatasetReader):\n filename = filename.name\n elif isinstance(filename, rasterio.vrt.WarpedVRT):\n vrt = filename\n filename = vrt.src_dataset.name\n vrt_params = dict(crs=vrt.crs.to_string(),\n resampling=vrt.resampling,\n src_nodata=vrt.src_nodata,\n dst_nodata=vrt.dst_nodata,\n tolerance=vrt.tolerance,\n transform=vrt.transform,\n width=vrt.width,\n height=vrt.height,\n warp_extras=vrt.warp_extras)\n\n if lock is None:\n lock = RASTERIO_LOCK\n\n manager = CachingFileManager(rasterio.open, filename, lock=lock, mode='r')\n riods = manager.acquire()\n if vrt_params is not None:\n riods = WarpedVRT(riods, **vrt_params)\n\n if cache is None:\n cache = chunks is None\n\n coords = OrderedDict()\n\n # Get bands\n if riods.count < 1:\n raise ValueError('Unknown dims')\n coords['band'] = np.asarray(riods.indexes)\n\n # Get coordinates\n if LooseVersion(rasterio.__version__) < '1.0':\n transform = riods.affine\n else:\n transform = riods.transform\n if transform.is_rectilinear:\n # 1d coordinates\n parse = True if parse_coordinates is None else parse_coordinates\n if parse:\n nx, ny = riods.width, riods.height\n # xarray coordinates are pixel centered\n x, _ = (np.arange(nx) + 0.5, np.zeros(nx) + 0.5) * transform\n _, y = (np.zeros(ny) + 0.5, np.arange(ny) + 0.5) * transform\n coords['y'] = y\n coords['x'] = x\n else:\n # 2d coordinates\n parse = False if (parse_coordinates is None) else parse_coordinates\n if parse:\n warnings.warn(\n \"The file coordinates' transformation isn't \"\n \"rectilinear: xarray won't parse the coordinates \"\n \"in this case. Set `parse_coordinates=False` to \"\n \"suppress this warning.\",\n RuntimeWarning, stacklevel=3)\n\n # Attributes\n attrs = dict()\n # Affine transformation matrix (always available)\n # This describes coefficients mapping pixel coordinates to CRS\n # For serialization store as tuple of 6 floats, the last row being\n # always (0, 0, 1) per definition (see\n # https://github.com/sgillies/affine)\n attrs['transform'] = tuple(transform)[:6]\n if hasattr(riods, 'crs') and riods.crs:\n # CRS is a dict-like object specific to rasterio\n # If CRS is not None, we convert it back to a PROJ4 string using\n # rasterio itself\n try:\n attrs['crs'] = riods.crs.to_proj4()\n except AttributeError:\n attrs['crs'] = riods.crs.to_string()\n if hasattr(riods, 'res'):\n # (width, height) tuple of pixels in units of CRS\n attrs['res'] = riods.res\n if hasattr(riods, 'is_tiled'):\n # Is the TIF tiled? (bool)\n # We cast it to an int for netCDF compatibility\n attrs['is_tiled'] = np.uint8(riods.is_tiled)\n if hasattr(riods, 'nodatavals'):\n # The nodata values for the raster bands\n attrs['nodatavals'] = tuple(\n np.nan if nodataval is None else nodataval\n for nodataval in riods.nodatavals)\n\n # Parse extra metadata from tags, if supported\n parsers = {'ENVI': _parse_envi}\n\n driver = riods.driver\n if driver in parsers:\n meta = parsers[driver](riods.tags(ns=driver))\n\n for k, v in meta.items():\n # Add values as coordinates if they match the band count,\n # as attributes otherwise\n if (isinstance(v, (list, np.ndarray))\n and len(v) == riods.count):\n coords[k] = ('band', np.asarray(v))\n else:\n attrs[k] = v\n\n data = indexing.LazilyOuterIndexedArray(\n RasterioArrayWrapper(manager, lock, vrt_params))\n\n # this lets you write arrays loaded with rasterio\n data = indexing.CopyOnWriteArray(data)\n if cache and chunks is None:\n data = indexing.MemoryCachedArray(data)\n\n result = DataArray(data=data, dims=('band', 'y', 'x'),\n coords=coords, attrs=attrs)\n\n if chunks is not None:\n from dask.base import tokenize\n # augment the token with the file modification time\n try:\n mtime = os.path.getmtime(filename)\n except OSError:\n # the filename is probably an s3 bucket rather than a regular file\n mtime = None\n token = tokenize(filename, mtime, chunks)\n name_prefix = 'open_rasterio-%s' % token\n result = result.chunk(chunks, name_prefix=name_prefix, token=token)\n\n # Make the file closeable\n result._file_obj = manager\n\n return result", "response": "Open a file with rasterio."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_indexer(self, key):\n assert len(key) == 3, 'rasterio datasets should always be 3D'\n\n # bands cannot be windowed but they can be listed\n band_key = key[0]\n np_inds = []\n # bands (axis=0) cannot be windowed but they can be listed\n if isinstance(band_key, slice):\n start, stop, step = band_key.indices(self.shape[0])\n band_key = np.arange(start, stop, step)\n # be sure we give out a list\n band_key = (np.asarray(band_key) + 1).tolist()\n if isinstance(band_key, list): # if band_key is not a scalar\n np_inds.append(slice(None))\n\n # but other dims can only be windowed\n window = []\n squeeze_axis = []\n for i, (k, n) in enumerate(zip(key[1:], self.shape[1:])):\n if isinstance(k, slice):\n # step is always positive. see indexing.decompose_indexer\n start, stop, step = k.indices(n)\n np_inds.append(slice(None, None, step))\n elif is_scalar(k):\n # windowed operations will always return an array\n # we will have to squeeze it later\n squeeze_axis.append(- (2 - i))\n start = k\n stop = k + 1\n else:\n start, stop = np.min(k), np.max(k) + 1\n np_inds.append(k - start)\n window.append((start, stop))\n\n if isinstance(key[1], np.ndarray) and isinstance(key[2], np.ndarray):\n # do outer-style indexing\n np_inds[-2:] = np.ix_(*np_inds[-2:])\n\n return band_key, tuple(window), tuple(squeeze_axis), tuple(np_inds)", "response": "Get indexer for rasterio array."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngroup an array by its unique values.", "response": "def unique_value_groups(ar, sort=True):\n \"\"\"Group an array by its unique values.\n\n Parameters\n ----------\n ar : array-like\n Input array. This will be flattened if it is not already 1-D.\n sort : boolean, optional\n Whether or not to sort unique values.\n\n Returns\n -------\n values : np.ndarray\n Sorted, unique values as returned by `np.unique`.\n indices : list of lists of int\n Each element provides the integer indices in `ar` with values given by\n the corresponding value in `unique_values`.\n \"\"\"\n inverse, values = pd.factorize(ar, sort=sort)\n groups = [[] for _ in range(len(values))]\n for n, g in enumerate(inverse):\n if g >= 0:\n # pandas uses -1 to mark NaN, but doesn't include them in values\n groups[g].append(n)\n return values, groups"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _consolidate_slices(slices):\n result = []\n last_slice = slice(None)\n for slice_ in slices:\n if not isinstance(slice_, slice):\n raise ValueError('list element is not a slice: %r' % slice_)\n if (result and last_slice.stop == slice_.start and\n _is_one_or_none(last_slice.step) and\n _is_one_or_none(slice_.step)):\n last_slice = slice(last_slice.start, slice_.stop, slice_.step)\n result[-1] = last_slice\n else:\n result.append(slice_)\n last_slice = slice_\n return result", "response": "Consolidate adjacent slices in a list of slices."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlike inverse_permutation but also handles slices.", "response": "def _inverse_permutation_indices(positions):\n \"\"\"Like inverse_permutation, but also handles slices.\n\n Parameters\n ----------\n positions : list of np.ndarray or slice objects.\n If slice objects, all are assumed to be slices.\n\n Returns\n -------\n np.ndarray of indices or None, if no permutation is necessary.\n \"\"\"\n if not positions:\n return None\n\n if isinstance(positions[0], slice):\n positions = _consolidate_slices(positions)\n if positions == slice(None):\n return None\n positions = [np.arange(sl.start, sl.stop, sl.step) for sl in positions]\n\n indices = nputils.inverse_permutation(np.concatenate(positions))\n return indices"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _apply_loffset(grouper, result):\n\n needs_offset = (\n isinstance(grouper.loffset, (pd.DateOffset, datetime.timedelta))\n and isinstance(result.index, pd.DatetimeIndex)\n and len(result.index) > 0\n )\n\n if needs_offset:\n result.index = result.index + grouper.loffset\n\n grouper.loffset = None", "response": "Applies the loffset to the result DataFrame."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\niterates over each element in this group", "response": "def _iter_grouped(self):\n \"\"\"Iterate over each element in this group\"\"\"\n for indices in self._group_indices:\n yield self._obj.isel(**{self._group_dim: indices})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrestore empty groups if needed.", "response": "def _maybe_restore_empty_groups(self, combined):\n \"\"\"Our index contained empty groups (e.g., from a resampling). If we\n reduced on that dimension, we want to restore the full index.\n \"\"\"\n if (self._full_index is not None and\n self._group.name in combined.dims):\n indexers = {self._group.name: self._full_index}\n combined = combined.reindex(**indexers)\n return combined"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _maybe_unstack(self, obj):\n if self._stacked_dim is not None and self._stacked_dim in obj.dims:\n obj = obj.unstack(self._stacked_dim)\n for dim in self._inserted_dims:\n if dim in obj.coords:\n del obj.coords[dim]\n return obj", "response": "Unstacks the array with the specified dimension if there is one."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the first element of each group along the group dimension", "response": "def first(self, skipna=None, keep_attrs=None):\n \"\"\"Return the first element of each group along the group dimension\n \"\"\"\n return self._first_or_last(duck_array_ops.first, skipna, keep_attrs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the last element of each group along the group dimension", "response": "def last(self, skipna=None, keep_attrs=None):\n \"\"\"Return the last element of each group along the group dimension\n \"\"\"\n return self._first_or_last(duck_array_ops.last, skipna, keep_attrs)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfasts version of _iter_grouped that yields Variables without metadata", "response": "def _iter_grouped_shortcut(self):\n \"\"\"Fast version of `_iter_grouped` that yields Variables without\n metadata\n \"\"\"\n var = self._obj.variable\n for indices in self._group_indices:\n yield var[{self._group_dim: indices}]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apply(self, func, shortcut=False, args=(), **kwargs):\n if shortcut:\n grouped = self._iter_grouped_shortcut()\n else:\n grouped = self._iter_grouped()\n applied = (maybe_wrap_array(arr, func(arr, *args, **kwargs))\n for arr in grouped)\n return self._combine(applied, shortcut=shortcut)", "response": "Apply a function over each array in the group and concatenate them together into a new array."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _combine(self, applied, shortcut=False):\n applied_example, applied = peek_at(applied)\n coord, dim, positions = self._infer_concat_args(applied_example)\n if shortcut:\n combined = self._concat_shortcut(applied, dim, positions)\n else:\n combined = concat(applied, dim)\n combined = _maybe_reorder(combined, dim, positions)\n\n if isinstance(combined, type(self._obj)):\n # only restore dimension order for arrays\n combined = self._restore_dim_order(combined)\n if coord is not None:\n if shortcut:\n combined._coords[coord.name] = as_variable(coord)\n else:\n combined.coords[coord.name] = coord\n combined = self._maybe_restore_empty_groups(combined)\n combined = self._maybe_unstack(combined)\n return combined", "response": "Recombine the applied objects like the original."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reduce(self, func, dim=None, axis=None,\n keep_attrs=None, shortcut=True, **kwargs):\n \"\"\"Reduce the items in this group by applying `func` along some\n dimension(s).\n\n Parameters\n ----------\n func : function\n Function which can be called in the form\n `func(x, axis=axis, **kwargs)` to return the result of collapsing\n an np.ndarray over an integer valued axis.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply `func`.\n axis : int or sequence of int, optional\n Axis(es) over which to apply `func`. Only one of the 'dimension'\n and 'axis' arguments can be supplied. If neither are supplied, then\n `func` is calculated over all dimension for each group item.\n keep_attrs : bool, optional\n If True, the datasets's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n **kwargs : dict\n Additional keyword arguments passed on to `func`.\n\n Returns\n -------\n reduced : Array\n Array with summarized data and the indicated dimension(s)\n removed.\n \"\"\"\n if dim == DEFAULT_DIMS:\n dim = ALL_DIMS\n # TODO change this to dim = self._group_dim after\n # the deprecation process\n if self._obj.ndim > 1:\n warnings.warn(\n \"Default reduction dimension will be changed to the \"\n \"grouped dimension in a future version of xarray. To \"\n \"silence this warning, pass dim=xarray.ALL_DIMS \"\n \"explicitly.\",\n FutureWarning, stacklevel=2)\n\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n\n def reduce_array(ar):\n return ar.reduce(func, dim, axis, keep_attrs=keep_attrs, **kwargs)\n return self.apply(reduce_array, shortcut=shortcut)", "response": "Reduce the items in this group by applying func along some dimension and axis."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef apply(self, func, args=(), **kwargs):\n kwargs.pop('shortcut', None) # ignore shortcut if set (for now)\n applied = (func(ds, *args, **kwargs) for ds in self._iter_grouped())\n return self._combine(applied)", "response": "Apply a function over each Dataset in the group and concatenate them\n together into a new Dataset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrecombine the applied objects like the original.", "response": "def _combine(self, applied):\n \"\"\"Recombine the applied objects like the original.\"\"\"\n applied_example, applied = peek_at(applied)\n coord, dim, positions = self._infer_concat_args(applied_example)\n combined = concat(applied, dim)\n combined = _maybe_reorder(combined, dim, positions)\n if coord is not None:\n combined[coord.name] = coord\n combined = self._maybe_restore_empty_groups(combined)\n combined = self._maybe_unstack(combined)\n return combined"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreduces the items in this group by applying func along some dimension.", "response": "def reduce(self, func, dim=None, keep_attrs=None, **kwargs):\n \"\"\"Reduce the items in this group by applying `func` along some\n dimension(s).\n\n Parameters\n ----------\n func : function\n Function which can be called in the form\n `func(x, axis=axis, **kwargs)` to return the result of collapsing\n an np.ndarray over an integer valued axis.\n dim : str or sequence of str, optional\n Dimension(s) over which to apply `func`.\n axis : int or sequence of int, optional\n Axis(es) over which to apply `func`. Only one of the 'dimension'\n and 'axis' arguments can be supplied. If neither are supplied, then\n `func` is calculated over all dimension for each group item.\n keep_attrs : bool, optional\n If True, the datasets's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n **kwargs : dict\n Additional keyword arguments passed on to `func`.\n\n Returns\n -------\n reduced : Array\n Array with summarized data and the indicated dimension(s)\n removed.\n \"\"\"\n if dim == DEFAULT_DIMS:\n dim = ALL_DIMS\n # TODO change this to dim = self._group_dim after\n # the deprecation process. Do not forget to remove _reduce_method\n warnings.warn(\n \"Default reduction dimension will be changed to the \"\n \"grouped dimension in a future version of xarray. To \"\n \"silence this warning, pass dim=xarray.ALL_DIMS \"\n \"explicitly.\",\n FutureWarning, stacklevel=2)\n elif dim is None:\n dim = self._group_dim\n\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n\n def reduce_dataset(ds):\n return ds.reduce(func, dim, keep_attrs, **kwargs)\n return self.apply(reduce_dataset)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns attrs that are not in ignored_attrs", "response": "def _filter_attrs(attrs, ignored_attrs):\n \"\"\" Return attrs that are not in ignored_attrs\n \"\"\"\n return dict((k, v) for k, v in attrs.items() if k not in ignored_attrs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_cdms2(variable):\n values = np.asarray(variable)\n name = variable.id\n dims = variable.getAxisIds()\n coords = {}\n for axis in variable.getAxisList():\n coords[axis.id] = DataArray(\n np.asarray(axis), dims=[axis.id],\n attrs=_filter_attrs(axis.attributes, cdms2_ignored_attrs))\n grid = variable.getGrid()\n if grid is not None:\n ids = [a.id for a in grid.getAxisList()]\n for axis in grid.getLongitude(), grid.getLatitude():\n if axis.id not in variable.getAxisIds():\n coords[axis.id] = DataArray(\n np.asarray(axis[:]), dims=ids,\n attrs=_filter_attrs(axis.attributes,\n cdms2_ignored_attrs))\n attrs = _filter_attrs(variable.attributes, cdms2_ignored_attrs)\n dataarray = DataArray(values, dims=dims, coords=coords, name=name,\n attrs=attrs)\n return decode_cf(dataarray.to_dataset())[dataarray.name]", "response": "Convert a cdms2 variable into a DataArray\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a DataArray into a cdms2 variable and its axes and grids.", "response": "def to_cdms2(dataarray, copy=True):\n \"\"\"Convert a DataArray into a cdms2 variable\n \"\"\"\n # we don't want cdms2 to be a hard dependency\n import cdms2\n\n def set_cdms2_attrs(var, attrs):\n for k, v in attrs.items():\n setattr(var, k, v)\n\n # 1D axes\n axes = []\n for dim in dataarray.dims:\n coord = encode(dataarray.coords[dim])\n axis = cdms2.createAxis(coord.values, id=dim)\n set_cdms2_attrs(axis, coord.attrs)\n axes.append(axis)\n\n # Data\n var = encode(dataarray)\n cdms2_var = cdms2.createVariable(var.values, axes=axes, id=dataarray.name,\n mask=pd.isnull(var.values), copy=copy)\n\n # Attributes\n set_cdms2_attrs(cdms2_var, var.attrs)\n\n # Curvilinear and unstructured grids\n if dataarray.name not in dataarray.coords:\n\n cdms2_axes = OrderedDict()\n for coord_name in set(dataarray.coords.keys()) - set(dataarray.dims):\n\n coord_array = dataarray.coords[coord_name].to_cdms2()\n\n cdms2_axis_cls = (cdms2.coord.TransientAxis2D\n if coord_array.ndim else\n cdms2.auxcoord.TransientAuxAxis1D)\n cdms2_axis = cdms2_axis_cls(coord_array)\n if cdms2_axis.isLongitude():\n cdms2_axes['lon'] = cdms2_axis\n elif cdms2_axis.isLatitude():\n cdms2_axes['lat'] = cdms2_axis\n\n if 'lon' in cdms2_axes and 'lat' in cdms2_axes:\n if len(cdms2_axes['lon'].shape) == 2:\n cdms2_grid = cdms2.hgrid.TransientCurveGrid(\n cdms2_axes['lat'], cdms2_axes['lon'])\n else:\n cdms2_grid = cdms2.gengrid.AbstractGenericGrid(\n cdms2_axes['lat'], cdms2_axes['lon'])\n for axis in cdms2_grid.getAxisList():\n cdms2_var.setAxis(cdms2_var.getAxisIds().index(axis.id), axis)\n cdms2_var.setGrid(cdms2_grid)\n\n return cdms2_var"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn attrs with keys in keys list", "response": "def _pick_attrs(attrs, keys):\n \"\"\" Return attrs with keys in keys list\n \"\"\"\n return dict((k, v) for k, v in attrs.items() if k in keys)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_iris_args(attrs):\n # iris.unit is deprecated in Iris v1.9\n import cf_units\n args = {'attributes': _filter_attrs(attrs, iris_forbidden_keys)}\n args.update(_pick_attrs(attrs, ('standard_name', 'long_name',)))\n unit_args = _pick_attrs(attrs, ('calendar',))\n if 'units' in attrs:\n args['units'] = cf_units.Unit(attrs['units'], **unit_args)\n return args", "response": "Converts the xarray attrs into args that can be passed into Iris\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_iris(dataarray):\n # Iris not a hard dependency\n import iris\n from iris.fileformats.netcdf import parse_cell_methods\n\n dim_coords = []\n aux_coords = []\n\n for coord_name in dataarray.coords:\n coord = encode(dataarray.coords[coord_name])\n coord_args = _get_iris_args(coord.attrs)\n coord_args['var_name'] = coord_name\n axis = None\n if coord.dims:\n axis = dataarray.get_axis_num(coord.dims)\n if coord_name in dataarray.dims:\n try:\n iris_coord = iris.coords.DimCoord(coord.values, **coord_args)\n dim_coords.append((iris_coord, axis))\n except ValueError:\n iris_coord = iris.coords.AuxCoord(coord.values, **coord_args)\n aux_coords.append((iris_coord, axis))\n else:\n iris_coord = iris.coords.AuxCoord(coord.values, **coord_args)\n aux_coords.append((iris_coord, axis))\n\n args = _get_iris_args(dataarray.attrs)\n args['var_name'] = dataarray.name\n args['dim_coords_and_dims'] = dim_coords\n args['aux_coords_and_dims'] = aux_coords\n if 'cell_methods' in dataarray.attrs:\n args['cell_methods'] = \\\n parse_cell_methods(dataarray.attrs['cell_methods'])\n\n masked_data = duck_array_ops.masked_invalid(dataarray.data)\n cube = iris.cube.Cube(masked_data, **args)\n\n return cube", "response": "Convert a DataArray into a Iris Cube\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _iris_obj_to_attrs(obj):\n attrs = {'standard_name': obj.standard_name,\n 'long_name': obj.long_name}\n if obj.units.calendar:\n attrs['calendar'] = obj.units.calendar\n if obj.units.origin != '1' and not obj.units.is_unknown():\n attrs['units'] = obj.units.origin\n attrs.update(obj.attributes)\n return dict((k, v) for k, v in attrs.items() if v is not None)", "response": "Return a dictionary of attrs when given a Iris object"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _iris_cell_methods_to_str(cell_methods_obj):\n cell_methods = []\n for cell_method in cell_methods_obj:\n names = ''.join(['{}: '.format(n) for n in cell_method.coord_names])\n intervals = ' '.join(['interval: {}'.format(interval)\n for interval in cell_method.intervals])\n comments = ' '.join(['comment: {}'.format(comment)\n for comment in cell_method.comments])\n extra = ' '.join([intervals, comments]).strip()\n if extra:\n extra = ' ({})'.format(extra)\n cell_methods.append(names + cell_method.method + extra)\n return ' '.join(cell_methods)", "response": "Converts a Iris cell methods into a string"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _name(iris_obj, default='unknown'):\n return (iris_obj.var_name or iris_obj.standard_name or\n iris_obj.long_name or default)", "response": "Mimicks name method but with different name resolution order."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a Iris cube into a DataArray .", "response": "def from_iris(cube):\n \"\"\" Convert a Iris cube into an DataArray\n \"\"\"\n import iris.exceptions\n from xarray.core.pycompat import dask_array_type\n\n name = _name(cube)\n if name == 'unknown':\n name = None\n dims = []\n for i in range(cube.ndim):\n try:\n dim_coord = cube.coord(dim_coords=True, dimensions=(i,))\n dims.append(_name(dim_coord))\n except iris.exceptions.CoordinateNotFoundError:\n dims.append(\"dim_{}\".format(i))\n\n if len(set(dims)) != len(dims):\n duplicates = [k for k, v in Counter(dims).items() if v > 1]\n raise ValueError('Duplicate coordinate name {}.'.format(duplicates))\n\n coords = OrderedDict()\n\n for coord in cube.coords():\n coord_attrs = _iris_obj_to_attrs(coord)\n coord_dims = [dims[i] for i in cube.coord_dims(coord)]\n if coord_dims:\n coords[_name(coord)] = (coord_dims, coord.points, coord_attrs)\n else:\n coords[_name(coord)] = ((), coord.points.item(), coord_attrs)\n\n array_attrs = _iris_obj_to_attrs(cube)\n cell_methods = _iris_cell_methods_to_str(cube.cell_methods)\n if cell_methods:\n array_attrs['cell_methods'] = cell_methods\n\n # Deal with iris 1.* and 2.*\n cube_data = cube.core_data() if hasattr(cube, 'core_data') else cube.data\n\n # Deal with dask and numpy masked arrays\n if isinstance(cube_data, dask_array_type):\n from dask.array import ma as dask_ma\n filled_data = dask_ma.filled(cube_data, get_fill_value(cube.dtype))\n elif isinstance(cube_data, np.ma.MaskedArray):\n filled_data = np.ma.filled(cube_data, get_fill_value(cube.dtype))\n else:\n filled_data = cube_data\n\n dataarray = DataArray(filled_data, coords=coords, name=name,\n attrs=array_attrs, dims=dims)\n decoded_ds = decode_cf(dataarray._to_temp_dataset())\n return dataarray._from_temp_dataset(decoded_ds)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngive an array of numeric dates in netCDF format convert it into numpy array of date time objects.", "response": "def decode_cf_datetime(num_dates, units, calendar=None, use_cftime=None):\n \"\"\"Given an array of numeric dates in netCDF format, convert it into a\n numpy array of date time objects.\n\n For standard (Gregorian) calendars, this function uses vectorized\n operations, which makes it much faster than cftime.num2date. In such a\n case, the returned array will be of type np.datetime64.\n\n Note that time unit in `units` must not be smaller than microseconds and\n not larger than days.\n\n See also\n --------\n cftime.num2date\n \"\"\"\n num_dates = np.asarray(num_dates)\n flat_num_dates = num_dates.ravel()\n if calendar is None:\n calendar = 'standard'\n\n if use_cftime is None:\n try:\n dates = _decode_datetime_with_pandas(flat_num_dates, units,\n calendar)\n except (OutOfBoundsDatetime, OverflowError):\n dates = _decode_datetime_with_cftime(\n flat_num_dates.astype(np.float), units, calendar)\n\n if (dates[np.nanargmin(num_dates)].year < 1678 or\n dates[np.nanargmax(num_dates)].year >= 2262):\n if calendar in _STANDARD_CALENDARS:\n warnings.warn(\n 'Unable to decode time axis into full '\n 'numpy.datetime64 objects, continuing using '\n 'cftime.datetime objects instead, reason: dates out '\n 'of range', SerializationWarning, stacklevel=3)\n else:\n if calendar in _STANDARD_CALENDARS:\n dates = cftime_to_nptime(dates)\n elif use_cftime:\n dates = _decode_datetime_with_cftime(\n flat_num_dates.astype(np.float), units, calendar)\n else:\n dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)\n\n return dates.reshape(num_dates.shape)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives an array of numeric timedeltas in netCDF format convert it into a numpy timedelta64 [ ns ] array.", "response": "def decode_cf_timedelta(num_timedeltas, units):\n \"\"\"Given an array of numeric timedeltas in netCDF format, convert it into a\n numpy timedelta64[ns] array.\n \"\"\"\n num_timedeltas = np.asarray(num_timedeltas)\n units = _netcdf_to_numpy_timeunit(units)\n\n shape = num_timedeltas.shape\n num_timedeltas = num_timedeltas.ravel()\n\n result = pd.to_timedelta(num_timedeltas, unit=units, box=False)\n # NaT is returned unboxed with wrong units; this should be fixed in pandas\n if result.dtype != 'timedelta64[ns]':\n result = result.astype('timedelta64[ns]')\n return result.reshape(shape)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef infer_calendar_name(dates):\n if np.asarray(dates).dtype == 'datetime64[ns]':\n return 'proleptic_gregorian'\n else:\n return np.asarray(dates).ravel()[0].calendar", "response": "Given an array of datetimes infer the CF calendar name"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef infer_datetime_units(dates):\n dates = np.asarray(dates).ravel()\n if np.asarray(dates).dtype == 'datetime64[ns]':\n dates = pd.to_datetime(dates, box=False)\n dates = dates[pd.notnull(dates)]\n reference_date = dates[0] if len(dates) > 0 else '1970-01-01'\n reference_date = pd.Timestamp(reference_date)\n else:\n reference_date = dates[0] if len(dates) > 0 else '1970-01-01'\n reference_date = format_cftime_datetime(reference_date)\n unique_timedeltas = np.unique(np.diff(dates))\n if unique_timedeltas.dtype == np.dtype('O'):\n # Convert to np.timedelta64 objects using pandas to work around a\n # NumPy casting bug: https://github.com/numpy/numpy/issues/11096\n unique_timedeltas = pd.to_timedelta(unique_timedeltas, box=False)\n units = _infer_time_units_from_diff(unique_timedeltas)\n return '%s since %s' % (units, reference_date)", "response": "Given an array of datetimes returns a CF compatible time - unit string of\n the form days hours minutes or seconds"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef format_cftime_datetime(date):\n return '{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:06d}'.format(\n date.year, date.month, date.day, date.hour, date.minute, date.second,\n date.microsecond)", "response": "Converts a cftime. datetime object to a string with the format YYYY - MM - DD. UUUUUU\n YYYY - MM - DD. UUUUUUU\n YYYY - MM - DD. UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef infer_timedelta_units(deltas):\n deltas = pd.to_timedelta(np.asarray(deltas).ravel(), box=False)\n unique_timedeltas = np.unique(deltas[pd.notnull(deltas)])\n units = _infer_time_units_from_diff(unique_timedeltas)\n return units", "response": "Given an array of timedeltas returns a CF compatible time - unit from that array."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cftime_to_nptime(times):\n times = np.asarray(times)\n new = np.empty(times.shape, dtype='M8[ns]')\n for i, t in np.ndenumerate(times):\n try:\n # Use pandas.Timestamp in place of datetime.datetime, because\n # NumPy casts it safely it np.datetime64[ns] for dates outside\n # 1678 to 2262 (this is not currently the case for\n # datetime.datetime).\n dt = pd.Timestamp(t.year, t.month, t.day, t.hour, t.minute,\n t.second, t.microsecond)\n except ValueError as e:\n raise ValueError('Cannot convert date {} to a date in the '\n 'standard calendar. Reason: {}.'.format(t, e))\n new[i] = np.datetime64(dt)\n return new", "response": "Given an array of cftime. datetime objects return an array of numpy. datetime64 objects of the same size"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _encode_datetime_with_cftime(dates, units, calendar):\n cftime = _import_cftime()\n\n if np.issubdtype(dates.dtype, np.datetime64):\n # numpy's broken datetime conversion only works for us precision\n dates = dates.astype('M8[us]').astype(datetime)\n\n def encode_datetime(d):\n return np.nan if d is None else cftime.date2num(d, units, calendar)\n\n return np.vectorize(encode_datetime)(dates)", "response": "Fallback method for encoding dates using cftime.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef encode_cf_datetime(dates, units=None, calendar=None):\n dates = np.asarray(dates)\n\n if units is None:\n units = infer_datetime_units(dates)\n else:\n units = _cleanup_netcdf_time_units(units)\n\n if calendar is None:\n calendar = infer_calendar_name(dates)\n\n delta, ref_date = _unpack_netcdf_time_units(units)\n try:\n if calendar not in _STANDARD_CALENDARS or dates.dtype.kind == 'O':\n # parse with cftime instead\n raise OutOfBoundsDatetime\n assert dates.dtype == 'datetime64[ns]'\n\n delta_units = _netcdf_to_numpy_timeunit(delta)\n time_delta = np.timedelta64(1, delta_units).astype('timedelta64[ns]')\n ref_date = pd.Timestamp(ref_date)\n\n # If the ref_date Timestamp is timezone-aware, convert to UTC and\n # make it timezone-naive (GH 2649).\n if ref_date.tz is not None:\n ref_date = ref_date.tz_convert(None)\n\n # Wrap the dates in a DatetimeIndex to do the subtraction to ensure\n # an OverflowError is raised if the ref_date is too far away from\n # dates to be encoded (GH 2272).\n num = (pd.DatetimeIndex(dates.ravel()) - ref_date) / time_delta\n num = num.values.reshape(dates.shape)\n\n except (OutOfBoundsDatetime, OverflowError):\n num = _encode_datetime_with_cftime(dates, units, calendar)\n\n num = cast_to_int_if_safe(num)\n return (num, units, calendar)", "response": "Given an array of datetime objects returns the tuple num units and calendar suitable for a CF compliant time variable."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _validate_dataset_names(dataset):\n def check_name(name):\n if isinstance(name, str):\n if not name:\n raise ValueError('Invalid name for DataArray or Dataset key: '\n 'string must be length 1 or greater for '\n 'serialization to netCDF files')\n elif name is not None:\n raise TypeError('DataArray.name or Dataset key must be either a '\n 'string or None for serialization to netCDF files')\n\n for k in dataset.variables:\n check_name(k)", "response": "Validate the names of the DataArray and Dataset keys."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _validate_attrs(dataset):\n def check_attr(name, value):\n if isinstance(name, str):\n if not name:\n raise ValueError('Invalid name for attr: string must be '\n 'length 1 or greater for serialization to '\n 'netCDF files')\n else:\n raise TypeError(\"Invalid name for attr: {} must be a string for \"\n \"serialization to netCDF files\".format(name))\n\n if not isinstance(value, (str, Number, np.ndarray, np.number,\n list, tuple)):\n raise TypeError('Invalid value for attr: {} must be a number, '\n 'a string, an ndarray or a list/tuple of '\n 'numbers/strings for serialization to netCDF '\n 'files'.format(value))\n\n # Check attrs on the dataset itself\n for k, v in dataset.attrs.items():\n check_attr(k, v)\n\n # Check attrs on each variable within the dataset\n for variable in dataset.variables.values():\n for k, v in variable.attrs.items():\n check_attr(k, v)", "response": "Validate the attrs of the dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads and decode a dataset from a file or file - like object.", "response": "def open_dataset(filename_or_obj, group=None, decode_cf=True,\n mask_and_scale=None, decode_times=True, autoclose=None,\n concat_characters=True, decode_coords=True, engine=None,\n chunks=None, lock=None, cache=None, drop_variables=None,\n backend_kwargs=None, use_cftime=None):\n \"\"\"Load and decode a dataset from a file or file-like object.\n\n Parameters\n ----------\n filename_or_obj : str, Path, file or xarray.backends.*DataStore\n Strings and Path objects are interpreted as a path to a netCDF file\n or an OpenDAP URL and opened with python-netCDF4, unless the filename\n ends with .gz, in which case the file is gunzipped and opened with\n scipy.io.netcdf (only netCDF3 supported). Byte-strings or file-like\n objects are opened by scipy.io.netcdf (netCDF3) or h5py (netCDF4/HDF).\n group : str, optional\n Path to the netCDF4 group in the given file to open (only works for\n netCDF4 files).\n decode_cf : bool, optional\n Whether to decode these variables, assuming they were saved according\n to CF conventions.\n mask_and_scale : bool, optional\n If True, replace array values equal to `_FillValue` with NA and scale\n values according to the formula `original_values * scale_factor +\n add_offset`, where `_FillValue`, `scale_factor` and `add_offset` are\n taken from variable attributes (if they exist). If the `_FillValue` or\n `missing_value` attribute contains multiple values a warning will be\n issued and all array values matching one of the multiple values will\n be replaced by NA. mask_and_scale defaults to True except for the\n pseudonetcdf backend.\n decode_times : bool, optional\n If True, decode times encoded in the standard NetCDF datetime format\n into datetime objects. Otherwise, leave them encoded as numbers.\n autoclose : bool, optional\n If True, automatically close files to avoid OS Error of too many files\n being open. However, this option doesn't work with streams, e.g.,\n BytesIO.\n concat_characters : bool, optional\n If True, concatenate along the last dimension of character arrays to\n form string arrays. Dimensions will only be concatenated over (and\n removed) if they have no corresponding variable and if they are only\n used as the last dimension of character arrays.\n decode_coords : bool, optional\n If True, decode the 'coordinates' attribute to identify coordinates in\n the resulting dataset.\n engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio', 'cfgrib', \\\n 'pseudonetcdf'}, optional\n Engine to use when reading files. If not provided, the default engine\n is chosen based on available dependencies, with a preference for\n 'netcdf4'.\n chunks : int or dict, optional\n If chunks is provided, it used to load the new dataset into dask\n arrays. ``chunks={}`` loads the dataset with dask using a single\n chunk for all arrays.\n lock : False or duck threading.Lock, optional\n Resource lock to use when reading data from disk. Only relevant when\n using dask or another form of parallelism. By default, appropriate\n locks are chosen to safely read and write files with the currently\n active dask scheduler.\n cache : bool, optional\n If True, cache data loaded from the underlying datastore in memory as\n NumPy arrays when accessed to avoid reading from the underlying data-\n store multiple times. Defaults to True unless you specify the `chunks`\n argument to use dask, in which case it defaults to False. Does not\n change the behavior of coordinates corresponding to dimensions, which\n always load their data from disk into a ``pandas.Index``.\n drop_variables: string or iterable, optional\n A variable or list of variables to exclude from being parsed from the\n dataset. This may be useful to drop variables with problems or\n inconsistent values.\n backend_kwargs: dictionary, optional\n A dictionary of keyword arguments to pass on to the backend. This\n may be useful when backend options would improve performance or\n allow user control of dataset processing.\n use_cftime: bool, optional\n Only relevant if encoded dates come from a standard calendar\n (e.g. 'gregorian', 'proleptic_gregorian', 'standard', or not\n specified). If None (default), attempt to decode times to\n ``np.datetime64[ns]`` objects; if this is not possible, decode times to\n ``cftime.datetime`` objects. If True, always decode times to\n ``cftime.datetime`` objects, regardless of whether or not they can be\n represented using ``np.datetime64[ns]`` objects. If False, always\n decode times to ``np.datetime64[ns]`` objects; if this is not possible\n raise an error.\n\n Returns\n -------\n dataset : Dataset\n The newly created dataset.\n\n Notes\n -----\n ``open_dataset`` opens the file with read-only access. When you modify\n values of a Dataset, even one linked to files on disk, only the in-memory\n copy you are manipulating in xarray is modified: the original file on disk\n is never touched.\n\n See Also\n --------\n open_mfdataset\n \"\"\"\n engines = [None, 'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio',\n 'cfgrib', 'pseudonetcdf']\n if engine not in engines:\n raise ValueError('unrecognized engine for open_dataset: {}\\n'\n 'must be one of: {}'\n .format(engine, engines))\n\n if autoclose is not None:\n warnings.warn(\n 'The autoclose argument is no longer used by '\n 'xarray.open_dataset() and is now ignored; it will be removed in '\n 'a future version of xarray. If necessary, you can control the '\n 'maximum number of simultaneous open files with '\n 'xarray.set_options(file_cache_maxsize=...).',\n FutureWarning, stacklevel=2)\n\n if mask_and_scale is None:\n mask_and_scale = not engine == 'pseudonetcdf'\n\n if not decode_cf:\n mask_and_scale = False\n decode_times = False\n concat_characters = False\n decode_coords = False\n\n if cache is None:\n cache = chunks is None\n\n if backend_kwargs is None:\n backend_kwargs = {}\n\n def maybe_decode_store(store, lock=False):\n ds = conventions.decode_cf(\n store, mask_and_scale=mask_and_scale, decode_times=decode_times,\n concat_characters=concat_characters, decode_coords=decode_coords,\n drop_variables=drop_variables, use_cftime=use_cftime)\n\n _protect_dataset_variables_inplace(ds, cache)\n\n if chunks is not None:\n from dask.base import tokenize\n # if passed an actual file path, augment the token with\n # the file modification time\n if (isinstance(filename_or_obj, str) and\n not is_remote_uri(filename_or_obj)):\n mtime = os.path.getmtime(filename_or_obj)\n else:\n mtime = None\n token = tokenize(filename_or_obj, mtime, group, decode_cf,\n mask_and_scale, decode_times, concat_characters,\n decode_coords, engine, chunks, drop_variables,\n use_cftime)\n name_prefix = 'open_dataset-%s' % token\n ds2 = ds.chunk(chunks, name_prefix=name_prefix, token=token)\n ds2._file_obj = ds._file_obj\n else:\n ds2 = ds\n\n return ds2\n\n if isinstance(filename_or_obj, Path):\n filename_or_obj = str(filename_or_obj)\n\n if isinstance(filename_or_obj, backends.AbstractDataStore):\n store = filename_or_obj\n\n elif isinstance(filename_or_obj, str):\n filename_or_obj = _normalize_path(filename_or_obj)\n\n if engine is None:\n engine = _get_default_engine(filename_or_obj,\n allow_remote=True)\n if engine == 'netcdf4':\n store = backends.NetCDF4DataStore.open(\n filename_or_obj, group=group, lock=lock, **backend_kwargs)\n elif engine == 'scipy':\n store = backends.ScipyDataStore(filename_or_obj, **backend_kwargs)\n elif engine == 'pydap':\n store = backends.PydapDataStore.open(\n filename_or_obj, **backend_kwargs)\n elif engine == 'h5netcdf':\n store = backends.H5NetCDFStore(\n filename_or_obj, group=group, lock=lock, **backend_kwargs)\n elif engine == 'pynio':\n store = backends.NioDataStore(\n filename_or_obj, lock=lock, **backend_kwargs)\n elif engine == 'pseudonetcdf':\n store = backends.PseudoNetCDFDataStore.open(\n filename_or_obj, lock=lock, **backend_kwargs)\n elif engine == 'cfgrib':\n store = backends.CfGribDataStore(\n filename_or_obj, lock=lock, **backend_kwargs)\n\n else:\n if engine not in [None, 'scipy', 'h5netcdf']:\n raise ValueError(\"can only read bytes or file-like objects \"\n \"with engine='scipy' or 'h5netcdf'\")\n engine = _get_engine_from_magic_number(filename_or_obj)\n if engine == 'scipy':\n store = backends.ScipyDataStore(filename_or_obj, **backend_kwargs)\n elif engine == 'h5netcdf':\n store = backends.H5NetCDFStore(filename_or_obj, group=group,\n lock=lock, **backend_kwargs)\n\n with close_on_error(store):\n ds = maybe_decode_store(store)\n\n # Ensure source filename always stored in dataset object (GH issue #2550)\n if 'source' not in ds.encoding:\n if isinstance(filename_or_obj, str):\n ds.encoding['source'] = filename_or_obj\n\n return ds"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open_dataarray(filename_or_obj, group=None, decode_cf=True,\n mask_and_scale=None, decode_times=True, autoclose=None,\n concat_characters=True, decode_coords=True, engine=None,\n chunks=None, lock=None, cache=None, drop_variables=None,\n backend_kwargs=None, use_cftime=None):\n \"\"\"Open an DataArray from a netCDF file containing a single data variable.\n\n This is designed to read netCDF files with only one data variable. If\n multiple variables are present then a ValueError is raised.\n\n Parameters\n ----------\n filename_or_obj : str, Path, file or xarray.backends.*DataStore\n Strings and Paths are interpreted as a path to a netCDF file or an\n OpenDAP URL and opened with python-netCDF4, unless the filename ends\n with .gz, in which case the file is gunzipped and opened with\n scipy.io.netcdf (only netCDF3 supported). Byte-strings or file-like\n objects are opened by scipy.io.netcdf (netCDF3) or h5py (netCDF4/HDF).\n group : str, optional\n Path to the netCDF4 group in the given file to open (only works for\n netCDF4 files).\n decode_cf : bool, optional\n Whether to decode these variables, assuming they were saved according\n to CF conventions.\n mask_and_scale : bool, optional\n If True, replace array values equal to `_FillValue` with NA and scale\n values according to the formula `original_values * scale_factor +\n add_offset`, where `_FillValue`, `scale_factor` and `add_offset` are\n taken from variable attributes (if they exist). If the `_FillValue` or\n `missing_value` attribute contains multiple values a warning will be\n issued and all array values matching one of the multiple values will\n be replaced by NA. mask_and_scale defaults to True except for the\n pseudonetcdf backend.\n decode_times : bool, optional\n If True, decode times encoded in the standard NetCDF datetime format\n into datetime objects. Otherwise, leave them encoded as numbers.\n concat_characters : bool, optional\n If True, concatenate along the last dimension of character arrays to\n form string arrays. Dimensions will only be concatenated over (and\n removed) if they have no corresponding variable and if they are only\n used as the last dimension of character arrays.\n decode_coords : bool, optional\n If True, decode the 'coordinates' attribute to identify coordinates in\n the resulting dataset.\n engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio', 'cfgrib'}, \\\n optional\n Engine to use when reading files. If not provided, the default engine\n is chosen based on available dependencies, with a preference for\n 'netcdf4'.\n chunks : int or dict, optional\n If chunks is provided, it used to load the new dataset into dask\n arrays.\n lock : False or duck threading.Lock, optional\n Resource lock to use when reading data from disk. Only relevant when\n using dask or another form of parallelism. By default, appropriate\n locks are chosen to safely read and write files with the currently\n active dask scheduler.\n cache : bool, optional\n If True, cache data loaded from the underlying datastore in memory as\n NumPy arrays when accessed to avoid reading from the underlying data-\n store multiple times. Defaults to True unless you specify the `chunks`\n argument to use dask, in which case it defaults to False. Does not\n change the behavior of coordinates corresponding to dimensions, which\n always load their data from disk into a ``pandas.Index``.\n drop_variables: string or iterable, optional\n A variable or list of variables to exclude from being parsed from the\n dataset. This may be useful to drop variables with problems or\n inconsistent values.\n backend_kwargs: dictionary, optional\n A dictionary of keyword arguments to pass on to the backend. This\n may be useful when backend options would improve performance or\n allow user control of dataset processing.\n use_cftime: bool, optional\n Only relevant if encoded dates come from a standard calendar\n (e.g. 'gregorian', 'proleptic_gregorian', 'standard', or not\n specified). If None (default), attempt to decode times to\n ``np.datetime64[ns]`` objects; if this is not possible, decode times to\n ``cftime.datetime`` objects. If True, always decode times to\n ``cftime.datetime`` objects, regardless of whether or not they can be\n represented using ``np.datetime64[ns]`` objects. If False, always\n decode times to ``np.datetime64[ns]`` objects; if this is not possible\n raise an error.\n\n Notes\n -----\n This is designed to be fully compatible with `DataArray.to_netcdf`. Saving\n using `DataArray.to_netcdf` and then loading with this function will\n produce an identical result.\n\n All parameters are passed directly to `xarray.open_dataset`. See that\n documentation for further details.\n\n See also\n --------\n open_dataset\n \"\"\"\n\n dataset = open_dataset(filename_or_obj, group=group, decode_cf=decode_cf,\n mask_and_scale=mask_and_scale,\n decode_times=decode_times, autoclose=autoclose,\n concat_characters=concat_characters,\n decode_coords=decode_coords, engine=engine,\n chunks=chunks, lock=lock, cache=cache,\n drop_variables=drop_variables,\n backend_kwargs=backend_kwargs,\n use_cftime=use_cftime)\n\n if len(dataset.data_vars) != 1:\n raise ValueError('Given file dataset contains more than one data '\n 'variable. Please read with xarray.open_dataset and '\n 'then select the variable you want.')\n else:\n data_array, = dataset.data_vars.values()\n\n data_array._file_obj = dataset._file_obj\n\n # Reset names if they were changed during saving\n # to ensure that we can 'roundtrip' perfectly\n if DATAARRAY_NAME in dataset.attrs:\n data_array.name = dataset.attrs[DATAARRAY_NAME]\n del dataset.attrs[DATAARRAY_NAME]\n\n if data_array.name == DATAARRAY_VARIABLE:\n data_array.name = None\n\n return data_array", "response": "Open an array from a netCDF file containing a single data variable."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef open_mfdataset(paths, chunks=None, concat_dim=_CONCAT_DIM_DEFAULT,\n compat='no_conflicts', preprocess=None, engine=None,\n lock=None, data_vars='all', coords='different',\n autoclose=None, parallel=False, **kwargs):\n \"\"\"Open multiple files as a single dataset.\n\n Requires dask to be installed. See documentation for details on dask [1].\n Attributes from the first dataset file are used for the combined dataset.\n\n Parameters\n ----------\n paths : str or sequence\n Either a string glob in the form \"path/to/my/files/*.nc\" or an explicit\n list of files to open. Paths can be given as strings or as pathlib\n Paths.\n chunks : int or dict, optional\n Dictionary with keys given by dimension names and values given by chunk\n sizes. In general, these should divide the dimensions of each dataset.\n If int, chunk each dimension by ``chunks``.\n By default, chunks will be chosen to load entire input files into\n memory at once. This has a major impact on performance: please see the\n full documentation for more details [2].\n concat_dim : None, str, DataArray or Index, optional\n Dimension to concatenate files along. This argument is passed on to\n :py:func:`xarray.auto_combine` along with the dataset objects. You only\n need to provide this argument if the dimension along which you want to\n concatenate is not a dimension in the original datasets, e.g., if you\n want to stack a collection of 2D arrays along a third dimension.\n By default, xarray attempts to infer this argument by examining\n component files. Set ``concat_dim=None`` explicitly to disable\n concatenation.\n compat : {'identical', 'equals', 'broadcast_equals', 'no_conflicts'}, optional\n String indicating how to compare variables of the same name for\n potential conflicts when merging:\n * 'broadcast_equals': all values must be equal when variables are\n broadcast against each other to ensure common dimensions.\n * 'equals': all values and dimensions must be the same.\n * 'identical': all values, dimensions and attributes must be the\n same.\n * 'no_conflicts': only values which are not null in both datasets\n must be equal. The returned dataset then contains the combination\n of all non-null values.\n preprocess : callable, optional\n If provided, call this function on each dataset prior to concatenation.\n You can find the file-name from which each dataset was loaded in\n ``ds.encoding['source']``.\n engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio', 'cfgrib'}, \\\n optional\n Engine to use when reading files. If not provided, the default engine\n is chosen based on available dependencies, with a preference for\n 'netcdf4'.\n lock : False or duck threading.Lock, optional\n Resource lock to use when reading data from disk. Only relevant when\n using dask or another form of parallelism. By default, appropriate\n locks are chosen to safely read and write files with the currently\n active dask scheduler.\n data_vars : {'minimal', 'different', 'all' or list of str}, optional\n These data variables will be concatenated together:\n\n * 'minimal': Only data variables in which the dimension already\n appears are included.\n * 'different': Data variables which are not equal (ignoring\n attributes) across all datasets are also concatenated (as well as\n all for which dimension already appears). Beware: this option may\n load the data payload of data variables into memory if they are not\n already loaded.\n * 'all': All data variables will be concatenated.\n * list of str: The listed data variables will be concatenated, in\n addition to the 'minimal' data variables.\n coords : {'minimal', 'different', 'all' o list of str}, optional\n These coordinate variables will be concatenated together:\n\n * 'minimal': Only coordinates in which the dimension already appears\n are included.\n * 'different': Coordinates which are not equal (ignoring attributes)\n across all datasets are also concatenated (as well as all for which\n dimension already appears). Beware: this option may load the data\n payload of coordinate variables into memory if they are not already\n loaded.\n * 'all': All coordinate variables will be concatenated, except\n those corresponding to other dimensions.\n * list of str: The listed coordinate variables will be concatenated,\n in addition the 'minimal' coordinates.\n parallel : bool, optional\n If True, the open and preprocess steps of this function will be\n performed in parallel using ``dask.delayed``. Default is False.\n **kwargs : optional\n Additional arguments passed on to :py:func:`xarray.open_dataset`.\n\n Returns\n -------\n xarray.Dataset\n\n Notes\n -----\n ``open_mfdataset`` opens files with read-only access. When you modify values\n of a Dataset, even one linked to files on disk, only the in-memory copy you\n are manipulating in xarray is modified: the original file on disk is never\n touched.\n\n See Also\n --------\n auto_combine\n open_dataset\n\n References\n ----------\n\n .. [1] http://xarray.pydata.org/en/stable/dask.html\n .. [2] http://xarray.pydata.org/en/stable/dask.html#chunking-and-performance\n \"\"\" # noqa\n if isinstance(paths, str):\n if is_remote_uri(paths):\n raise ValueError(\n 'cannot do wild-card matching for paths that are remote URLs: '\n '{!r}. Instead, supply paths as an explicit list of strings.'\n .format(paths))\n paths = sorted(glob(paths))\n else:\n paths = [str(p) if isinstance(p, Path) else p for p in paths]\n\n if not paths:\n raise IOError('no files to open')\n\n # Coerce 1D input into ND to maintain backwards-compatible API until API\n # for N-D combine decided\n # (see https://github.com/pydata/xarray/pull/2553/#issuecomment-445892746)\n if concat_dim is None or concat_dim is _CONCAT_DIM_DEFAULT:\n concat_dims = concat_dim\n elif not isinstance(concat_dim, list):\n concat_dims = [concat_dim]\n else:\n concat_dims = concat_dim\n infer_order_from_coords = False\n\n # If infer_order_from_coords=True then this is unnecessary, but quick.\n # If infer_order_from_coords=False then this creates a flat list which is\n # easier to iterate over, while saving the originally-supplied structure\n combined_ids_paths, concat_dims = _infer_concat_order_from_positions(\n paths, concat_dims)\n ids, paths = (\n list(combined_ids_paths.keys()), list(combined_ids_paths.values()))\n\n open_kwargs = dict(engine=engine, chunks=chunks or {}, lock=lock,\n autoclose=autoclose, **kwargs)\n\n if parallel:\n import dask\n # wrap the open_dataset, getattr, and preprocess with delayed\n open_ = dask.delayed(open_dataset)\n getattr_ = dask.delayed(getattr)\n if preprocess is not None:\n preprocess = dask.delayed(preprocess)\n else:\n open_ = open_dataset\n getattr_ = getattr\n\n datasets = [open_(p, **open_kwargs) for p in paths]\n file_objs = [getattr_(ds, '_file_obj') for ds in datasets]\n if preprocess is not None:\n datasets = [preprocess(ds) for ds in datasets]\n\n if parallel:\n # calling compute here will return the datasets/file_objs lists,\n # the underlying datasets will still be stored as dask arrays\n datasets, file_objs = dask.compute(datasets, file_objs)\n\n # Close datasets in case of a ValueError\n try:\n if infer_order_from_coords:\n # Discard ordering because it should be redone from coordinates\n ids = False\n\n combined = _auto_combine(\n datasets, concat_dims=concat_dims,\n compat=compat,\n data_vars=data_vars, coords=coords,\n infer_order_from_coords=infer_order_from_coords,\n ids=ids)\n except ValueError:\n for ds in datasets:\n ds.close()\n raise\n\n combined._file_obj = _MultiFileCloser(file_objs)\n combined.attrs = datasets[0].attrs\n return combined", "response": "Open multiple files as a single dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_netcdf(dataset, path_or_file=None, mode='w', format=None, group=None,\n engine=None, encoding=None, unlimited_dims=None, compute=True,\n multifile=False):\n \"\"\"This function creates an appropriate datastore for writing a dataset to\n disk as a netCDF file\n\n See `Dataset.to_netcdf` for full API docs.\n\n The ``multifile`` argument is only for the private use of save_mfdataset.\n \"\"\"\n if isinstance(path_or_file, Path):\n path_or_file = str(path_or_file)\n\n if encoding is None:\n encoding = {}\n\n if path_or_file is None:\n if engine is None:\n engine = 'scipy'\n elif engine != 'scipy':\n raise ValueError('invalid engine for creating bytes with '\n 'to_netcdf: %r. Only the default engine '\n \"or engine='scipy' is supported\" % engine)\n if not compute:\n raise NotImplementedError(\n 'to_netcdf() with compute=False is not yet implemented when '\n 'returning bytes')\n elif isinstance(path_or_file, str):\n if engine is None:\n engine = _get_default_engine(path_or_file)\n path_or_file = _normalize_path(path_or_file)\n else: # file-like object\n engine = 'scipy'\n\n # validate Dataset keys, DataArray names, and attr keys/values\n _validate_dataset_names(dataset)\n _validate_attrs(dataset)\n\n try:\n store_open = WRITEABLE_STORES[engine]\n except KeyError:\n raise ValueError('unrecognized engine for to_netcdf: %r' % engine)\n\n if format is not None:\n format = format.upper()\n\n # handle scheduler specific logic\n scheduler = _get_scheduler()\n have_chunks = any(v.chunks for v in dataset.variables.values())\n\n autoclose = have_chunks and scheduler in ['distributed', 'multiprocessing']\n if autoclose and engine == 'scipy':\n raise NotImplementedError(\"Writing netCDF files with the %s backend \"\n \"is not currently supported with dask's %s \"\n \"scheduler\" % (engine, scheduler))\n\n target = path_or_file if path_or_file is not None else BytesIO()\n kwargs = dict(autoclose=True) if autoclose else {}\n store = store_open(target, mode, format, group, **kwargs)\n\n if unlimited_dims is None:\n unlimited_dims = dataset.encoding.get('unlimited_dims', None)\n if isinstance(unlimited_dims, str):\n unlimited_dims = [unlimited_dims]\n\n writer = ArrayWriter()\n\n # TODO: figure out how to refactor this logic (here and in save_mfdataset)\n # to avoid this mess of conditionals\n try:\n # TODO: allow this work (setting up the file for writing array data)\n # to be parallelized with dask\n dump_to_store(dataset, store, writer, encoding=encoding,\n unlimited_dims=unlimited_dims)\n if autoclose:\n store.close()\n\n if multifile:\n return writer, store\n\n writes = writer.sync(compute=compute)\n\n if path_or_file is None:\n store.sync()\n return target.getvalue()\n finally:\n if not multifile and compute:\n store.close()\n\n if not compute:\n import dask\n return dask.delayed(_finalize_store)(writes, store)", "response": "This function creates an appropriate datastore for writing a dataset to a netCDF file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nstore dataset contents to a backends. DataStore object.", "response": "def dump_to_store(dataset, store, writer=None, encoder=None,\n encoding=None, unlimited_dims=None):\n \"\"\"Store dataset contents to a backends.*DataStore object.\"\"\"\n if writer is None:\n writer = ArrayWriter()\n\n if encoding is None:\n encoding = {}\n\n variables, attrs = conventions.encode_dataset_coordinates(dataset)\n\n check_encoding = set()\n for k, enc in encoding.items():\n # no need to shallow copy the variable again; that already happened\n # in encode_dataset_coordinates\n variables[k].encoding = enc\n check_encoding.add(k)\n\n if encoder:\n variables, attrs = encoder(variables, attrs)\n\n store.store(variables, attrs, check_encoding, writer,\n unlimited_dims=unlimited_dims)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef save_mfdataset(datasets, paths, mode='w', format=None, groups=None,\n engine=None, compute=True):\n \"\"\"Write multiple datasets to disk as netCDF files simultaneously.\n\n This function is intended for use with datasets consisting of dask.array\n objects, in which case it can write the multiple datasets to disk\n simultaneously using a shared thread pool.\n\n When not using dask, it is no different than calling ``to_netcdf``\n repeatedly.\n\n Parameters\n ----------\n datasets : list of xarray.Dataset\n List of datasets to save.\n paths : list of str or list of Paths\n List of paths to which to save each corresponding dataset.\n mode : {'w', 'a'}, optional\n Write ('w') or append ('a') mode. If mode='w', any existing file at\n these locations will be overwritten.\n format : {'NETCDF4', 'NETCDF4_CLASSIC', 'NETCDF3_64BIT',\n 'NETCDF3_CLASSIC'}, optional\n\n File format for the resulting netCDF file:\n\n * NETCDF4: Data is stored in an HDF5 file, using netCDF4 API\n features.\n * NETCDF4_CLASSIC: Data is stored in an HDF5 file, using only\n netCDF 3 compatible API features.\n * NETCDF3_64BIT: 64-bit offset version of the netCDF 3 file format,\n which fully supports 2+ GB files, but is only compatible with\n clients linked against netCDF version 3.6.0 or later.\n * NETCDF3_CLASSIC: The classic netCDF 3 file format. It does not\n handle 2+ GB files very well.\n\n All formats are supported by the netCDF4-python library.\n scipy.io.netcdf only supports the last two formats.\n\n The default format is NETCDF4 if you are saving a file to disk and\n have the netCDF4-python library available. Otherwise, xarray falls\n back to using scipy to write netCDF files and defaults to the\n NETCDF3_64BIT format (scipy does not support netCDF4).\n groups : list of str, optional\n Paths to the netCDF4 group in each corresponding file to which to save\n datasets (only works for format='NETCDF4'). The groups will be created\n if necessary.\n engine : {'netcdf4', 'scipy', 'h5netcdf'}, optional\n Engine to use when writing netCDF files. If not provided, the\n default engine is chosen based on available dependencies, with a\n preference for 'netcdf4' if writing to a file on disk.\n See `Dataset.to_netcdf` for additional information.\n compute: boolean\n If true compute immediately, otherwise return a\n ``dask.delayed.Delayed`` object that can be computed later.\n\n Examples\n --------\n\n Save a dataset into one netCDF per year of data:\n\n >>> years, datasets = zip(*ds.groupby('time.year'))\n >>> paths = ['%s.nc' % y for y in years]\n >>> xr.save_mfdataset(datasets, paths)\n \"\"\"\n if mode == 'w' and len(set(paths)) < len(paths):\n raise ValueError(\"cannot use mode='w' when writing multiple \"\n 'datasets to the same path')\n\n for obj in datasets:\n if not isinstance(obj, Dataset):\n raise TypeError('save_mfdataset only supports writing Dataset '\n 'objects, received type %s' % type(obj))\n\n if groups is None:\n groups = [None] * len(datasets)\n\n if len(set([len(datasets), len(paths), len(groups)])) > 1:\n raise ValueError('must supply lists of the same length for the '\n 'datasets, paths and groups arguments to '\n 'save_mfdataset')\n\n writers, stores = zip(*[\n to_netcdf(ds, path, mode, format, group, engine, compute=compute,\n multifile=True)\n for ds, path, group in zip(datasets, paths, groups)])\n\n try:\n writes = [w.sync(compute=compute) for w in writers]\n finally:\n if compute:\n for store in stores:\n store.close()\n\n if not compute:\n import dask\n return dask.delayed([dask.delayed(_finalize_store)(w, s)\n for w, s in zip(writes, stores)])", "response": "Save multiple datasets to disk as netCDF files simultaneously."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_zarr(dataset, store=None, mode='w-', synchronizer=None, group=None,\n encoding=None, compute=True, consolidated=False):\n \"\"\"This function creates an appropriate datastore for writing a dataset to\n a zarr ztore\n\n See `Dataset.to_zarr` for full API docs.\n \"\"\"\n if isinstance(store, Path):\n store = str(store)\n if encoding is None:\n encoding = {}\n\n # validate Dataset keys, DataArray names, and attr keys/values\n _validate_dataset_names(dataset)\n _validate_attrs(dataset)\n\n zstore = backends.ZarrStore.open_group(store=store, mode=mode,\n synchronizer=synchronizer,\n group=group,\n consolidate_on_close=consolidated)\n\n writer = ArrayWriter()\n # TODO: figure out how to properly handle unlimited_dims\n dump_to_store(dataset, zstore, writer, encoding=encoding)\n writes = writer.sync(compute=compute)\n\n if compute:\n _finalize_store(writes, zstore)\n else:\n import dask\n return dask.delayed(_finalize_store)(writes, zstore)\n\n return zstore", "response": "This function creates an appropriate datastore for writing a dataset to a zarr ztore\n \\'' + store + '\\'."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a new MultiIndex that contains all the unused levels in the current MultiIndex.", "response": "def remove_unused_levels(self):\n \"\"\"\n create a new MultiIndex from the current that removing\n unused levels, meaning that they are not expressed in the labels\n The resulting MultiIndex will have the same outward\n appearance, meaning the same .values and ordering. It will also\n be .equals() to the original.\n .. versionadded:: 0.20.0\n Returns\n -------\n MultiIndex\n Examples\n --------\n >>> i = pd.MultiIndex.from_product([range(2), list('ab')])\n MultiIndex(levels=[[0, 1], ['a', 'b']],\n labels=[[0, 0, 1, 1], [0, 1, 0, 1]])\n >>> i[2:]\n MultiIndex(levels=[[0, 1], ['a', 'b']],\n labels=[[1, 1], [0, 1]])\n The 0 from the first level is not represented\n and can be removed\n >>> i[2:].remove_unused_levels()\n MultiIndex(levels=[[1], ['a', 'b']],\n labels=[[0, 0], [0, 1]])\n \"\"\"\n import pandas.core.algorithms as algos\n\n new_levels = []\n new_labels = []\n\n changed = False\n for lev, lab in zip(self.levels, self.labels):\n\n # Since few levels are typically unused, bincount() is more\n # efficient than unique() - however it only accepts positive values\n # (and drops order):\n uniques = np.where(np.bincount(lab + 1) > 0)[0] - 1\n has_na = int(len(uniques) and (uniques[0] == -1))\n\n if len(uniques) != len(lev) + has_na:\n # We have unused levels\n changed = True\n\n # Recalculate uniques, now preserving order.\n # Can easily be cythonized by exploiting the already existing\n # \"uniques\" and stop parsing \"lab\" when all items are found:\n uniques = algos.unique(lab)\n if has_na:\n na_idx = np.where(uniques == -1)[0]\n # Just ensure that -1 is in first position:\n uniques[[0, na_idx[0]]] = uniques[[na_idx[0], 0]]\n\n # labels get mapped from uniques to 0:len(uniques)\n # -1 (if present) is mapped to last position\n label_mapping = np.zeros(len(lev) + has_na)\n # ... and reassigned value -1:\n label_mapping[uniques] = np.arange(len(uniques)) - has_na\n\n lab = label_mapping[lab]\n\n # new levels are simple\n lev = lev.take(uniques[has_na:])\n\n new_levels.append(lev)\n new_labels.append(lab)\n\n result = self._shallow_copy()\n\n if changed:\n result._reset_identity()\n result._set_levels(new_levels, validate=False)\n result._set_labels(new_labels, validate=False)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef robust_getitem(array, key, catch=Exception, max_retries=6,\n initial_delay=500):\n \"\"\"\n Robustly index an array, using retry logic with exponential backoff if any\n of the errors ``catch`` are raised. The initial_delay is measured in ms.\n\n With the default settings, the maximum delay will be in the range of 32-64\n seconds.\n \"\"\"\n assert max_retries >= 0\n for n in range(max_retries + 1):\n try:\n return array[key]\n except catch:\n if n == max_retries:\n raise\n base_delay = initial_delay * 2 ** n\n next_delay = base_delay + np.random.randint(base_delay)\n msg = ('getitem failed, waiting %s ms before trying again '\n '(%s tries remaining). Full traceback: %s' %\n (next_delay, max_retries - n, traceback.format_exc()))\n logger.debug(msg)\n time.sleep(1e-3 * next_delay)", "response": "Robustly index an array using exponential backoff."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nencode the variables and attributes in this store", "response": "def encode(self, variables, attributes):\n \"\"\"\n Encode the variables and attributes in this store\n\n Parameters\n ----------\n variables : dict-like\n Dictionary of key/value (variable name / xr.Variable) pairs\n attributes : dict-like\n Dictionary of key/value (attribute name / attribute) pairs\n\n Returns\n -------\n variables : dict-like\n attributes : dict-like\n\n \"\"\"\n variables = OrderedDict([(k, self.encode_variable(v))\n for k, v in variables.items()])\n attributes = OrderedDict([(k, self.encode_attribute(v))\n for k, v in attributes.items()])\n return variables, attributes"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntops level method for putting data on this store, this method: - encodes variables/attributes - sets dimensions - sets variables Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs attributes : dict-like Dictionary of key/value (attribute name / attribute) pairs check_encoding_set : list-like List of variables that should be checked for invalid encoding values writer : ArrayWriter unlimited_dims : list-like List of dimension names that should be treated as unlimited dimensions.", "response": "def store(self, variables, attributes, check_encoding_set=frozenset(),\n writer=None, unlimited_dims=None):\n \"\"\"\n Top level method for putting data on this store, this method:\n - encodes variables/attributes\n - sets dimensions\n - sets variables\n\n Parameters\n ----------\n variables : dict-like\n Dictionary of key/value (variable name / xr.Variable) pairs\n attributes : dict-like\n Dictionary of key/value (attribute name / attribute) pairs\n check_encoding_set : list-like\n List of variables that should be checked for invalid encoding\n values\n writer : ArrayWriter\n unlimited_dims : list-like\n List of dimension names that should be treated as unlimited\n dimensions.\n \"\"\"\n if writer is None:\n writer = ArrayWriter()\n\n variables, attributes = self.encode(variables, attributes)\n\n self.set_attributes(attributes)\n self.set_dimensions(variables, unlimited_dims=unlimited_dims)\n self.set_variables(variables, check_encoding_set, writer,\n unlimited_dims=unlimited_dims)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_attributes(self, attributes):\n for k, v in attributes.items():\n self.set_attribute(k, v)", "response": "This method sets the attributes on the object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_dimensions(self, variables, unlimited_dims=None):\n if unlimited_dims is None:\n unlimited_dims = set()\n\n existing_dims = self.get_dimensions()\n\n dims = OrderedDict()\n for v in unlimited_dims: # put unlimited_dims first\n dims[v] = None\n for v in variables.values():\n dims.update(dict(zip(v.dims, v.shape)))\n\n for dim, length in dims.items():\n if dim in existing_dims and length != existing_dims[dim]:\n raise ValueError(\n \"Unable to update size for existing dimension\"\n \"%r (%d != %d)\" % (dim, length, existing_dims[dim]))\n elif dim not in existing_dims:\n is_unlimited = dim in unlimited_dims\n self.set_dimension(dim, length, is_unlimited)", "response": "This method sets the dimensions of the data store."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _season_from_months(months):\n # TODO: Move \"season\" accessor upstream into pandas\n seasons = np.array(['DJF', 'MAM', 'JJA', 'SON'])\n months = np.asarray(months)\n return seasons[(months // 3) % 4]", "response": "Compute season from month ordinal\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _access_through_cftimeindex(values, name):\n from ..coding.cftimeindex import CFTimeIndex\n values_as_cftimeindex = CFTimeIndex(values.ravel())\n if name == 'season':\n months = values_as_cftimeindex.month\n field_values = _season_from_months(months)\n else:\n field_values = getattr(values_as_cftimeindex, name)\n return field_values.reshape(values.shape)", "response": "Coerce an array of datetime - like values to a CFTimeIndex\n and access requested datetime component\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _access_through_series(values, name):\n values_as_series = pd.Series(values.ravel())\n if name == \"season\":\n months = values_as_series.dt.month.values\n field_values = _season_from_months(months)\n else:\n field_values = getattr(values_as_series.dt, name).values\n return field_values.reshape(values.shape)", "response": "Coerce an array of datetime - like values to a pandas Series and\n access requested datetime component\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_date_field(values, name, dtype):\n if is_np_datetime_like(values.dtype):\n access_method = _access_through_series\n else:\n access_method = _access_through_cftimeindex\n\n if isinstance(values, dask_array_type):\n from dask.array import map_blocks\n return map_blocks(access_method,\n values, name, dtype=dtype)\n else:\n return access_method(values, name)", "response": "Indirectly access pandas libts. get_date_field by wrapping data\n as a Series and calling through. dt attribute."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncoerce an array of datetime - like values to a pandas Series and apply requested rounding", "response": "def _round_series(values, name, freq):\n \"\"\"Coerce an array of datetime-like values to a pandas Series and\n apply requested rounding\n \"\"\"\n values_as_series = pd.Series(values.ravel())\n method = getattr(values_as_series.dt, name)\n field_values = method(freq=freq).values\n\n return field_values.reshape(values.shape)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fillna(data, other, join=\"left\", dataset_join=\"left\"):\n from .computation import apply_ufunc\n\n return apply_ufunc(duck_array_ops.fillna, data, other,\n join=join,\n dask=\"allowed\",\n dataset_join=dataset_join,\n dataset_fill_value=np.nan,\n keep_attrs=True)", "response": "Fill missing values in this object with data from other object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef where_method(self, cond, other=dtypes.NA):\n from .computation import apply_ufunc\n # alignment for three arguments is complicated, so don't support it yet\n join = 'inner' if other is dtypes.NA else 'exact'\n return apply_ufunc(duck_array_ops.where_method,\n self, cond, other,\n join=join,\n dataset_join=join,\n dask='allowed',\n keep_attrs=True)", "response": "Return elements from self or other depending on cond."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nopen a netcdf dataset from the online repository.", "response": "def open_dataset(name, cache=True, cache_dir=_default_cache_dir,\n github_url='https://github.com/pydata/xarray-data',\n branch='master', **kws):\n \"\"\"\n Load a dataset from the online repository (requires internet).\n\n If a local copy is found then always use that to avoid network traffic.\n\n Parameters\n ----------\n name : str\n Name of the netcdf file containing the dataset\n ie. 'air_temperature'\n cache_dir : string, optional\n The directory in which to search for and write cached data.\n cache : boolean, optional\n If True, then cache data locally for use on subsequent calls\n github_url : string\n Github repository where the data is stored\n branch : string\n The git branch to download from\n kws : dict, optional\n Passed to xarray.open_dataset\n\n See Also\n --------\n xarray.open_dataset\n\n \"\"\"\n longdir = _os.path.expanduser(cache_dir)\n fullname = name + '.nc'\n localfile = _os.sep.join((longdir, fullname))\n md5name = name + '.md5'\n md5file = _os.sep.join((longdir, md5name))\n\n if not _os.path.exists(localfile):\n\n # This will always leave this directory on disk.\n # May want to add an option to remove it.\n if not _os.path.isdir(longdir):\n _os.mkdir(longdir)\n\n url = '/'.join((github_url, 'raw', branch, fullname))\n urlretrieve(url, localfile)\n url = '/'.join((github_url, 'raw', branch, md5name))\n urlretrieve(url, md5file)\n\n localmd5 = file_md5_checksum(localfile)\n with open(md5file, 'r') as f:\n remotemd5 = f.read()\n if localmd5 != remotemd5:\n _os.remove(localfile)\n msg = \"\"\"\n MD5 checksum does not match, try downloading dataset again.\n \"\"\"\n raise IOError(msg)\n\n ds = _open_dataset(localfile, **kws)\n\n if not cache:\n ds = ds.load()\n _os.remove(localfile)\n\n return ds"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmaking a key for caching files in the LRU cache.", "response": "def _make_key(self):\n \"\"\"Make a key for caching files in the LRU cache.\"\"\"\n value = (self._opener,\n self._args,\n 'a' if self._mode == 'w' else self._mode,\n tuple(sorted(self._kwargs.items())))\n return _HashedSequence(value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nacquire a file object from the manager.", "response": "def acquire(self, needs_lock=True):\n \"\"\"Acquiring a file object from the manager.\n\n A new file is only opened if it has expired from the\n least-recently-used cache.\n\n This method uses a lock, which ensures that it is thread-safe. You can\n safely acquire a file in multiple threads at the same time, as long as\n the underlying file object is thread-safe.\n\n Returns\n -------\n An open file object, as returned by ``opener(*args, **kwargs)``.\n \"\"\"\n with self._optional_lock(needs_lock):\n try:\n file = self._cache[self._key]\n except KeyError:\n kwargs = self._kwargs\n if self._mode is not _DEFAULT_MODE:\n kwargs = kwargs.copy()\n kwargs['mode'] = self._mode\n file = self._opener(*self._args, **kwargs)\n if self._mode == 'w':\n # ensure file doesn't get overriden when opened again\n self._mode = 'a'\n self._cache[self._key] = file\n return file"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _enforce_size_limit(self, capacity):\n while len(self._cache) > capacity:\n key, value = self._cache.popitem(last=False)\n if self._on_evict is not None:\n self._on_evict(key, value)", "response": "Shrink the cache if necessary evicting the oldest items."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef maxsize(self, size):\n if size < 0:\n raise ValueError('maxsize must be non-negative')\n with self._lock:\n self._enforce_size_limit(size)\n self._maxsize = size", "response": "Resize the cache evicting the oldest items if necessary."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning system information as a dict", "response": "def get_sys_info():\n \"Returns system information as a dict\"\n\n blob = []\n\n # get full commit hash\n commit = None\n if os.path.isdir(\".git\") and os.path.isdir(\"xarray\"):\n try:\n pipe = subprocess.Popen('git log --format=\"%H\" -n 1'.split(\" \"),\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n so, serr = pipe.communicate()\n except Exception:\n pass\n else:\n if pipe.returncode == 0:\n commit = so\n try:\n commit = so.decode('utf-8')\n except ValueError:\n pass\n commit = commit.strip().strip('\"')\n\n blob.append(('commit', commit))\n\n try:\n (sysname, nodename, release,\n version, machine, processor) = platform.uname()\n blob.extend([\n (\"python\", sys.version),\n (\"python-bits\", struct.calcsize(\"P\") * 8),\n (\"OS\", \"%s\" % (sysname)),\n (\"OS-release\", \"%s\" % (release)),\n # (\"Version\", \"%s\" % (version)),\n (\"machine\", \"%s\" % (machine)),\n (\"processor\", \"%s\" % (processor)),\n (\"byteorder\", \"%s\" % sys.byteorder),\n (\"LC_ALL\", \"%s\" % os.environ.get('LC_ALL', \"None\")),\n (\"LANG\", \"%s\" % os.environ.get('LANG', \"None\")),\n (\"LOCALE\", \"%s.%s\" % locale.getlocale()),\n\n ])\n except Exception:\n pass\n\n return blob"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a key for indexing an ndarray return an equivalent key which is a tuple with length equal to the number of dimensions.", "response": "def expanded_indexer(key, ndim):\n \"\"\"Given a key for indexing an ndarray, return an equivalent key which is a\n tuple with length equal to the number of dimensions.\n\n The expansion is done by replacing all `Ellipsis` items with the right\n number of full slices and then padding the key with full slices so that it\n reaches the appropriate dimensionality.\n \"\"\"\n if not isinstance(key, tuple):\n # numpy treats non-tuple keys equivalent to tuples of length 1\n key = (key,)\n new_key = []\n # handling Ellipsis right is a little tricky, see:\n # http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing\n found_ellipsis = False\n for k in key:\n if k is Ellipsis:\n if not found_ellipsis:\n new_key.extend((ndim + 1 - len(key)) * [slice(None)])\n found_ellipsis = True\n else:\n new_key.append(slice(None))\n else:\n new_key.append(k)\n if len(new_key) > ndim:\n raise IndexError('too many indices')\n new_key.extend((ndim - len(new_key)) * [slice(None)])\n return tuple(new_key)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting values into a numpy array of at most 1 - dimension while preserving tuples.", "response": "def _asarray_tuplesafe(values):\n \"\"\"\n Convert values into a numpy array of at most 1-dimension, while preserving\n tuples.\n\n Adapted from pandas.core.common._asarray_tuplesafe\n \"\"\"\n if isinstance(values, tuple):\n result = utils.to_0d_object_array(values)\n else:\n result = np.asarray(values)\n if result.ndim == 2:\n result = np.empty(len(values), dtype=object)\n result[:] = values\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the indexer for the N - dimensional index.", "response": "def get_indexer_nd(index, labels, method=None, tolerance=None):\n \"\"\" Call pd.Index.get_indexer(labels). \"\"\"\n kwargs = _index_method_kwargs(method, tolerance)\n\n flat_labels = np.ravel(labels)\n flat_indexer = index.get_indexer(flat_labels, **kwargs)\n indexer = flat_indexer.reshape(labels.shape)\n return indexer"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_label_indexer(index, label, index_name='', method=None,\n tolerance=None):\n \"\"\"Given a pandas.Index and labels (e.g., from __getitem__) for one\n dimension, return an indexer suitable for indexing an ndarray along that\n dimension. If `index` is a pandas.MultiIndex and depending on `label`,\n return a new pandas.Index or pandas.MultiIndex (otherwise return None).\n \"\"\"\n new_index = None\n\n if isinstance(label, slice):\n if method is not None or tolerance is not None:\n raise NotImplementedError(\n 'cannot use ``method`` argument if any indexers are '\n 'slice objects')\n indexer = index.slice_indexer(_sanitize_slice_element(label.start),\n _sanitize_slice_element(label.stop),\n _sanitize_slice_element(label.step))\n if not isinstance(indexer, slice):\n # unlike pandas, in xarray we never want to silently convert a\n # slice indexer into an array indexer\n raise KeyError('cannot represent labeled-based slice indexer for '\n 'dimension %r with a slice over integer positions; '\n 'the index is unsorted or non-unique' % index_name)\n\n elif is_dict_like(label):\n is_nested_vals = _is_nested_tuple(tuple(label.values()))\n if not isinstance(index, pd.MultiIndex):\n raise ValueError('cannot use a dict-like object for selection on '\n 'a dimension that does not have a MultiIndex')\n elif len(label) == index.nlevels and not is_nested_vals:\n indexer = index.get_loc(tuple((label[k] for k in index.names)))\n else:\n for k, v in label.items():\n # index should be an item (i.e. Hashable) not an array-like\n if not isinstance(v, Hashable):\n raise ValueError('Vectorized selection is not '\n 'available along level variable: ' + k)\n indexer, new_index = index.get_loc_level(\n tuple(label.values()), level=tuple(label.keys()))\n\n # GH2619. Raise a KeyError if nothing is chosen\n if indexer.dtype.kind == 'b' and indexer.sum() == 0:\n raise KeyError('{} not found'.format(label))\n\n elif isinstance(label, tuple) and isinstance(index, pd.MultiIndex):\n if _is_nested_tuple(label):\n indexer = index.get_locs(label)\n elif len(label) == index.nlevels:\n indexer = index.get_loc(label)\n else:\n indexer, new_index = index.get_loc_level(\n label, level=list(range(len(label)))\n )\n else:\n label = (label if getattr(label, 'ndim', 1) > 1 # vectorized-indexing\n else _asarray_tuplesafe(label))\n if label.ndim == 0:\n if isinstance(index, pd.MultiIndex):\n indexer, new_index = index.get_loc_level(label.item(), level=0)\n else:\n indexer = get_loc(index, label.item(), method, tolerance)\n elif label.dtype.kind == 'b':\n indexer = label\n else:\n if isinstance(index, pd.MultiIndex) and label.ndim > 1:\n raise ValueError('Vectorized selection is not available along '\n 'MultiIndex variable: ' + index_name)\n indexer = get_indexer_nd(index, label, method, tolerance)\n if np.any(indexer < 0):\n raise KeyError('not all values found in index %r'\n % index_name)\n return indexer, new_index", "response": "Converts a pandas. Index and labels to a pandas. MultiIndex."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives a xarray data object and label based indexers return a mapping of label indexers with only dimension names as keys.", "response": "def get_dim_indexers(data_obj, indexers):\n \"\"\"Given a xarray data object and label based indexers, return a mapping\n of label indexers with only dimension names as keys.\n\n It groups multiple level indexers given on a multi-index dimension\n into a single, dictionary indexer for that dimension (Raise a ValueError\n if it is not possible).\n \"\"\"\n invalid = [k for k in indexers\n if k not in data_obj.dims and k not in data_obj._level_coords]\n if invalid:\n raise ValueError(\"dimensions or multi-index levels %r do not exist\"\n % invalid)\n\n level_indexers = defaultdict(dict)\n dim_indexers = {}\n for key, label in indexers.items():\n dim, = data_obj[key].dims\n if key != dim:\n # assume here multi-index level indexer\n level_indexers[dim][key] = label\n else:\n dim_indexers[key] = label\n\n for dim, level_labels in level_indexers.items():\n if dim_indexers.get(dim, False):\n raise ValueError(\"cannot combine multi-index level indexers \"\n \"with an indexer for dimension %s\" % dim)\n dim_indexers[dim] = level_labels\n\n return dim_indexers"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive an xarray data object and label based indexers return a mapping of updated pandas index objects.", "response": "def remap_label_indexers(data_obj, indexers, method=None, tolerance=None):\n \"\"\"Given an xarray data object and label based indexers, return a mapping\n of equivalent location based indexers. Also return a mapping of updated\n pandas index objects (in case of multi-index level drop).\n \"\"\"\n if method is not None and not isinstance(method, str):\n raise TypeError('``method`` must be a string')\n\n pos_indexers = {}\n new_indexes = {}\n\n dim_indexers = get_dim_indexers(data_obj, indexers)\n for dim, label in dim_indexers.items():\n try:\n index = data_obj.indexes[dim]\n except KeyError:\n # no index for this dimension: reuse the provided labels\n if method is not None or tolerance is not None:\n raise ValueError('cannot supply ``method`` or ``tolerance`` '\n 'when the indexed dimension does not have '\n 'an associated coordinate.')\n pos_indexers[dim] = label\n else:\n idxr, new_idx = convert_label_indexer(index, label,\n dim, method, tolerance)\n pos_indexers[dim] = idxr\n if new_idx is not None:\n new_indexes[dim] = new_idx\n\n return pos_indexers, new_indexes"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives a slice and the size of the dimension to which it will be applied return a new slice equivalent to applying the slices sequentially", "response": "def slice_slice(old_slice, applied_slice, size):\n \"\"\"Given a slice and the size of the dimension to which it will be applied,\n index it with another slice to return a new slice equivalent to applying\n the slices sequentially\n \"\"\"\n step = (old_slice.step or 1) * (applied_slice.step or 1)\n\n # For now, use the hack of turning old_slice into an ndarray to reconstruct\n # the slice start and stop. This is not entirely ideal, but it is still\n # definitely better than leaving the indexer as an array.\n items = _expand_slice(old_slice, size)[applied_slice]\n if len(items) > 0:\n start = items[0]\n stop = items[-1] + int(np.sign(step))\n if stop < 0:\n stop = None\n else:\n start = 0\n stop = 0\n return slice(start, stop, step)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a new object that can be indexed by the given array.", "response": "def as_indexable(array):\n \"\"\"\n This function always returns a ExplicitlyIndexed subclass,\n so that the vectorized indexing is always possible with the returned\n object.\n \"\"\"\n if isinstance(array, ExplicitlyIndexed):\n return array\n if isinstance(array, np.ndarray):\n return NumpyIndexingAdapter(array)\n if isinstance(array, pd.Index):\n return PandasIndexAdapter(array)\n if isinstance(array, dask_array_type):\n return DaskIndexingAdapter(array)\n raise TypeError('Invalid array type: {}'.format(type(array)))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts an OuterIndexer into a VectorizedIndexer.", "response": "def _outer_to_vectorized_indexer(key, shape):\n \"\"\"Convert an OuterIndexer into an vectorized indexer.\n\n Parameters\n ----------\n key : Outer/Basic Indexer\n An indexer to convert.\n shape : tuple\n Shape of the array subject to the indexing.\n\n Returns\n -------\n VectorizedIndexer\n Tuple suitable for use to index a NumPy array with vectorized indexing.\n Each element is an array: broadcasting them together gives the shape\n of the result.\n \"\"\"\n key = key.tuple\n\n n_dim = len([k for k in key if not isinstance(k, integer_types)])\n i_dim = 0\n new_key = []\n for k, size in zip(key, shape):\n if isinstance(k, integer_types):\n new_key.append(np.array(k).reshape((1,) * n_dim))\n else: # np.ndarray or slice\n if isinstance(k, slice):\n k = np.arange(*k.indices(size))\n assert k.dtype.kind in {'i', 'u'}\n shape = [(1,) * i_dim + (k.size, ) +\n (1,) * (n_dim - i_dim - 1)]\n new_key.append(k.reshape(*shape))\n i_dim += 1\n return VectorizedIndexer(tuple(new_key))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _outer_to_numpy_indexer(key, shape):\n if len([k for k in key.tuple if not isinstance(k, slice)]) <= 1:\n # If there is only one vector and all others are slice,\n # it can be safely used in mixed basic/advanced indexing.\n # Boolean index should already be converted to integer array.\n return key.tuple\n else:\n return _outer_to_vectorized_indexer(key, shape).tuple", "response": "Convert an OuterIndexer into a NumPy indexer."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncombine two indexers. Parameters ---------- old_key: ExplicitIndexer The first indexer for the original array shape: tuple of ints Shape of the original array to be indexed by old_key new_key: The second indexer for indexing original[old_key]", "response": "def _combine_indexers(old_key, shape, new_key):\n \"\"\" Combine two indexers.\n\n Parameters\n ----------\n old_key: ExplicitIndexer\n The first indexer for the original array\n shape: tuple of ints\n Shape of the original array to be indexed by old_key\n new_key:\n The second indexer for indexing original[old_key]\n \"\"\"\n if not isinstance(old_key, VectorizedIndexer):\n old_key = _outer_to_vectorized_indexer(old_key, shape)\n if len(old_key.tuple) == 0:\n return new_key\n\n new_shape = np.broadcast(*old_key.tuple).shape\n if isinstance(new_key, VectorizedIndexer):\n new_key = _arrayize_vectorized_indexer(new_key, new_shape)\n else:\n new_key = _outer_to_vectorized_indexer(new_key, new_shape)\n\n return VectorizedIndexer(tuple(o[new_key.tuple] for o in\n np.broadcast_arrays(*old_key.tuple)))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsupport explicit indexing by delegating to a raw indexing method.", "response": "def explicit_indexing_adapter(\n key, shape, indexing_support, raw_indexing_method):\n \"\"\"Support explicit indexing by delegating to a raw indexing method.\n\n Outer and/or vectorized indexers are supported by indexing a second time\n with a NumPy array.\n\n Parameters\n ----------\n key : ExplicitIndexer\n Explicit indexing object.\n shape : Tuple[int, ...]\n Shape of the indexed array.\n indexing_support : IndexingSupport enum\n Form of indexing supported by raw_indexing_method.\n raw_indexing_method: callable\n Function (like ndarray.__getitem__) that when called with indexing key\n in the form of a tuple returns an indexed array.\n\n Returns\n -------\n Indexing result, in the form of a duck numpy-array.\n \"\"\"\n raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)\n result = raw_indexing_method(raw_key.tuple)\n if numpy_indices.tuple:\n # index the loaded np.ndarray\n result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]\n return result"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _decompose_slice(key, size):\n start, stop, step = key.indices(size)\n if step > 0:\n # If key already has a positive step, use it as is in the backend\n return key, slice(None)\n else:\n # determine stop precisely for step > 1 case\n # e.g. [98:2:-2] -> [98:3:-2]\n stop = start + int((stop - start - 1) / step) * step + 1\n start, stop = stop + 1, start + 1\n return slice(start, stop, -step), slice(None, None, -1)", "response": "decompose a slice to successive two slices."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _decompose_vectorized_indexer(indexer, shape, indexing_support):\n assert isinstance(indexer, VectorizedIndexer)\n\n if indexing_support is IndexingSupport.VECTORIZED:\n return indexer, BasicIndexer(())\n\n backend_indexer = []\n np_indexer = []\n # convert negative indices\n indexer = [np.where(k < 0, k + s, k) if isinstance(k, np.ndarray) else k\n for k, s in zip(indexer.tuple, shape)]\n\n for k, s in zip(indexer, shape):\n if isinstance(k, slice):\n # If it is a slice, then we will slice it as-is\n # (but make its step positive) in the backend,\n # and then use all of it (slice(None)) for the in-memory portion.\n bk_slice, np_slice = _decompose_slice(k, s)\n backend_indexer.append(bk_slice)\n np_indexer.append(np_slice)\n else:\n # If it is a (multidimensional) np.ndarray, just pickup the used\n # keys without duplication and store them as a 1d-np.ndarray.\n oind, vind = np.unique(k, return_inverse=True)\n backend_indexer.append(oind)\n np_indexer.append(vind.reshape(*k.shape))\n\n backend_indexer = OuterIndexer(tuple(backend_indexer))\n np_indexer = VectorizedIndexer(tuple(np_indexer))\n\n if indexing_support is IndexingSupport.OUTER:\n return backend_indexer, np_indexer\n\n # If the backend does not support outer indexing,\n # backend_indexer (OuterIndexer) is also decomposed.\n backend_indexer, np_indexer1 = _decompose_outer_indexer(\n backend_indexer, shape, indexing_support)\n np_indexer = _combine_indexers(np_indexer1, shape, np_indexer)\n return backend_indexer, np_indexer", "response": "Decomposes vectorized indexer to the successive two indexers where the first one is used to index the backend arrays while the second one is used to index loaded on - memory np. ndarray."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecompose an outer indexer to the successive two indexers where the first one is used to index backend arrays while the second one is used to index backend arrays while the second one is used to index backend arrays.", "response": "def _decompose_outer_indexer(indexer, shape, indexing_support):\n \"\"\"\n Decompose outer indexer to the successive two indexers, where the\n first indexer will be used to index backend arrays, while the second one\n is used to index the loaded on-memory np.ndarray.\n\n Parameters\n ----------\n indexer: VectorizedIndexer\n indexing_support: One of the entries of IndexingSupport\n\n Returns\n -------\n backend_indexer: OuterIndexer or BasicIndexer\n np_indexers: an ExplicitIndexer (OuterIndexer / BasicIndexer)\n\n Notes\n -----\n This function is used to realize the vectorized indexing for the backend\n arrays that only support basic or outer indexing.\n\n As an example, let us consider to index a few elements from a backend array\n with a orthogonal indexer ([0, 3, 1], [2, 3, 2]).\n Even if the backend array only supports basic indexing, it is more\n efficient to load a subslice of the array than loading the entire array,\n\n >>> backend_indexer = BasicIndexer(slice(0, 3), slice(2, 3))\n >>> array = array[backend_indexer] # load subslice of the array\n >>> np_indexer = OuterIndexer([0, 2, 1], [0, 1, 0])\n >>> array[np_indexer] # outer indexing for on-memory np.ndarray.\n \"\"\"\n if indexing_support == IndexingSupport.VECTORIZED:\n return indexer, BasicIndexer(())\n assert isinstance(indexer, (OuterIndexer, BasicIndexer))\n\n backend_indexer = []\n np_indexer = []\n # make indexer positive\n pos_indexer = []\n for k, s in zip(indexer.tuple, shape):\n if isinstance(k, np.ndarray):\n pos_indexer.append(np.where(k < 0, k + s, k))\n elif isinstance(k, integer_types) and k < 0:\n pos_indexer.append(k + s)\n else:\n pos_indexer.append(k)\n indexer = pos_indexer\n\n if indexing_support is IndexingSupport.OUTER_1VECTOR:\n # some backends such as h5py supports only 1 vector in indexers\n # We choose the most efficient axis\n gains = [(np.max(k) - np.min(k) + 1.0) / len(np.unique(k))\n if isinstance(k, np.ndarray) else 0 for k in indexer]\n array_index = np.argmax(np.array(gains)) if len(gains) > 0 else None\n\n for i, (k, s) in enumerate(zip(indexer, shape)):\n if isinstance(k, np.ndarray) and i != array_index:\n # np.ndarray key is converted to slice that covers the entire\n # entries of this key.\n backend_indexer.append(slice(np.min(k), np.max(k) + 1))\n np_indexer.append(k - np.min(k))\n elif isinstance(k, np.ndarray):\n # Remove duplicates and sort them in the increasing order\n pkey, ekey = np.unique(k, return_inverse=True)\n backend_indexer.append(pkey)\n np_indexer.append(ekey)\n elif isinstance(k, integer_types):\n backend_indexer.append(k)\n else: # slice: convert positive step slice for backend\n bk_slice, np_slice = _decompose_slice(k, s)\n backend_indexer.append(bk_slice)\n np_indexer.append(np_slice)\n\n return (OuterIndexer(tuple(backend_indexer)),\n OuterIndexer(tuple(np_indexer)))\n\n if indexing_support == IndexingSupport.OUTER:\n for k, s in zip(indexer, shape):\n if isinstance(k, slice):\n # slice: convert positive step slice for backend\n bk_slice, np_slice = _decompose_slice(k, s)\n backend_indexer.append(bk_slice)\n np_indexer.append(np_slice)\n elif isinstance(k, integer_types):\n backend_indexer.append(k)\n elif isinstance(k, np.ndarray) and (np.diff(k) >= 0).all():\n backend_indexer.append(k)\n np_indexer.append(slice(None))\n else:\n # Remove duplicates and sort them in the increasing order\n oind, vind = np.unique(k, return_inverse=True)\n backend_indexer.append(oind)\n np_indexer.append(vind.reshape(*k.shape))\n\n return (OuterIndexer(tuple(backend_indexer)),\n OuterIndexer(tuple(np_indexer)))\n\n # basic indexer\n assert indexing_support == IndexingSupport.BASIC\n\n for k, s in zip(indexer, shape):\n if isinstance(k, np.ndarray):\n # np.ndarray key is converted to slice that covers the entire\n # entries of this key.\n backend_indexer.append(slice(np.min(k), np.max(k) + 1))\n np_indexer.append(k - np.min(k))\n elif isinstance(k, integer_types):\n backend_indexer.append(k)\n else: # slice: convert positive step slice for backend\n bk_slice, np_slice = _decompose_slice(k, s)\n backend_indexer.append(bk_slice)\n np_indexer.append(np_slice)\n\n return (BasicIndexer(tuple(backend_indexer)),\n OuterIndexer(tuple(np_indexer)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _arrayize_vectorized_indexer(indexer, shape):\n slices = [v for v in indexer.tuple if isinstance(v, slice)]\n if len(slices) == 0:\n return indexer\n\n arrays = [v for v in indexer.tuple if isinstance(v, np.ndarray)]\n n_dim = arrays[0].ndim if len(arrays) > 0 else 0\n i_dim = 0\n new_key = []\n for v, size in zip(indexer.tuple, shape):\n if isinstance(v, np.ndarray):\n new_key.append(np.reshape(v, v.shape + (1, ) * len(slices)))\n else: # slice\n shape = ((1,) * (n_dim + i_dim) + (-1,) +\n (1,) * (len(slices) - i_dim - 1))\n new_key.append(np.arange(*v.indices(size)).reshape(shape))\n i_dim += 1\n return VectorizedIndexer(tuple(new_key))", "response": "Return an identical vindex but slices are replaced by arrays"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _dask_array_with_chunks_hint(array, chunks):\n import dask.array as da\n if len(chunks) < array.ndim:\n raise ValueError('not enough chunks in hint')\n new_chunks = []\n for chunk, size in zip(chunks, array.shape):\n new_chunks.append(chunk if size > 1 else (1,))\n return da.from_array(array, new_chunks)", "response": "Create a dask array using the chunks hint for dimensions of size > 1."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_mask(indexer, shape, chunks_hint=None):\n if isinstance(indexer, OuterIndexer):\n key = _outer_to_vectorized_indexer(indexer, shape).tuple\n assert not any(isinstance(k, slice) for k in key)\n mask = _masked_result_drop_slice(key, chunks_hint)\n\n elif isinstance(indexer, VectorizedIndexer):\n key = indexer.tuple\n base_mask = _masked_result_drop_slice(key, chunks_hint)\n slice_shape = tuple(np.arange(*k.indices(size)).size\n for k, size in zip(key, shape)\n if isinstance(k, slice))\n expanded_mask = base_mask[\n (Ellipsis,) + (np.newaxis,) * len(slice_shape)]\n mask = duck_array_ops.broadcast_to(\n expanded_mask, base_mask.shape + slice_shape)\n\n elif isinstance(indexer, BasicIndexer):\n mask = any(k == -1 for k in indexer.tuple)\n\n else:\n raise TypeError('unexpected key type: {}'.format(type(indexer)))\n\n return mask", "response": "Create a mask for indexing with a fill - value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting masked indices in a flat array to nearest unmasked index.", "response": "def _posify_mask_subindexer(index):\n \"\"\"Convert masked indices in a flat array to the nearest unmasked index.\n\n Parameters\n ----------\n index : np.ndarray\n One dimensional ndarray with dtype=int.\n\n Returns\n -------\n np.ndarray\n One dimensional ndarray with all values equal to -1 replaced by an\n adjacent non-masked element.\n \"\"\"\n masked = index == -1\n unmasked_locs = np.flatnonzero(~masked)\n if not unmasked_locs.size:\n # indexing unmasked_locs is invalid\n return np.zeros_like(index)\n masked_locs = np.flatnonzero(masked)\n prev_value = np.maximum(0, np.searchsorted(unmasked_locs, masked_locs) - 1)\n new_index = index.copy()\n new_index[masked_locs] = index[unmasked_locs[prev_value]]\n return new_index"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting masked values in an indexer to nearest unmasked values.", "response": "def posify_mask_indexer(indexer):\n \"\"\"Convert masked values (-1) in an indexer to nearest unmasked values.\n\n This routine is useful for dask, where it can be much faster to index\n adjacent points than arbitrary points from the end of an array.\n\n Parameters\n ----------\n indexer : ExplicitIndexer\n Input indexer.\n\n Returns\n -------\n ExplicitIndexer\n Same type of input, with all values in ndarray keys equal to -1\n replaced by an adjacent non-masked element.\n \"\"\"\n key = tuple(_posify_mask_subindexer(k.ravel()).reshape(k.shape)\n if isinstance(k, np.ndarray) else k\n for k in indexer.tuple)\n return type(indexer)(key)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nimporting seaborn and handle deprecation of apionly module", "response": "def import_seaborn():\n '''import seaborn and handle deprecation of apionly module'''\n with warnings.catch_warnings(record=True) as w:\n warnings.simplefilter(\"always\")\n try:\n import seaborn.apionly as sns\n if (w and issubclass(w[-1].category, UserWarning) and\n (\"seaborn.apionly module\" in str(w[-1].message))):\n raise ImportError\n except ImportError:\n import seaborn as sns\n finally:\n warnings.resetwarnings()\n return sns"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild a discrete colormap and normalization of the data.", "response": "def _build_discrete_cmap(cmap, levels, extend, filled):\n \"\"\"\n Build a discrete colormap and normalization of the data.\n \"\"\"\n import matplotlib as mpl\n\n if not filled:\n # non-filled contour plots\n extend = 'max'\n\n if extend == 'both':\n ext_n = 2\n elif extend in ['min', 'max']:\n ext_n = 1\n else:\n ext_n = 0\n\n n_colors = len(levels) + ext_n - 1\n pal = _color_palette(cmap, n_colors)\n\n new_cmap, cnorm = mpl.colors.from_levels_and_colors(\n levels, pal, extend=extend)\n # copy the old cmap name, for easier testing\n new_cmap.name = getattr(cmap, 'name', cmap)\n\n return new_cmap, cnorm"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _determine_cmap_params(plot_data, vmin=None, vmax=None, cmap=None,\n center=None, robust=False, extend=None,\n levels=None, filled=True, norm=None):\n \"\"\"\n Use some heuristics to set good defaults for colorbar and range.\n\n Parameters\n ==========\n plot_data: Numpy array\n Doesn't handle xarray objects\n\n Returns\n =======\n cmap_params : dict\n Use depends on the type of the plotting function\n \"\"\"\n import matplotlib as mpl\n\n calc_data = np.ravel(plot_data[np.isfinite(plot_data)])\n\n # Handle all-NaN input data gracefully\n if calc_data.size == 0:\n # Arbitrary default for when all values are NaN\n calc_data = np.array(0.0)\n\n # Setting center=False prevents a divergent cmap\n possibly_divergent = center is not False\n\n # Set center to 0 so math below makes sense but remember its state\n center_is_none = False\n if center is None:\n center = 0\n center_is_none = True\n\n # Setting both vmin and vmax prevents a divergent cmap\n if (vmin is not None) and (vmax is not None):\n possibly_divergent = False\n\n # Setting vmin or vmax implies linspaced levels\n user_minmax = (vmin is not None) or (vmax is not None)\n\n # vlim might be computed below\n vlim = None\n\n # save state; needed later\n vmin_was_none = vmin is None\n vmax_was_none = vmax is None\n\n if vmin is None:\n if robust:\n vmin = np.percentile(calc_data, ROBUST_PERCENTILE)\n else:\n vmin = calc_data.min()\n elif possibly_divergent:\n vlim = abs(vmin - center)\n\n if vmax is None:\n if robust:\n vmax = np.percentile(calc_data, 100 - ROBUST_PERCENTILE)\n else:\n vmax = calc_data.max()\n elif possibly_divergent:\n vlim = abs(vmax - center)\n\n if possibly_divergent:\n # kwargs not specific about divergent or not: infer defaults from data\n divergent = ((vmin < 0) and (vmax > 0)) or not center_is_none\n else:\n divergent = False\n\n # A divergent map should be symmetric around the center value\n if divergent:\n if vlim is None:\n vlim = max(abs(vmin - center), abs(vmax - center))\n vmin, vmax = -vlim, vlim\n\n # Now add in the centering value and set the limits\n vmin += center\n vmax += center\n\n # now check norm and harmonize with vmin, vmax\n if norm is not None:\n if norm.vmin is None:\n norm.vmin = vmin\n else:\n if not vmin_was_none and vmin != norm.vmin:\n raise ValueError('Cannot supply vmin and a norm'\n + ' with a different vmin.')\n vmin = norm.vmin\n\n if norm.vmax is None:\n norm.vmax = vmax\n else:\n if not vmax_was_none and vmax != norm.vmax:\n raise ValueError('Cannot supply vmax and a norm'\n + ' with a different vmax.')\n vmax = norm.vmax\n\n # if BoundaryNorm, then set levels\n if isinstance(norm, mpl.colors.BoundaryNorm):\n levels = norm.boundaries\n\n # Choose default colormaps if not provided\n if cmap is None:\n if divergent:\n cmap = OPTIONS['cmap_divergent']\n else:\n cmap = OPTIONS['cmap_sequential']\n\n # Handle discrete levels\n if levels is not None and norm is None:\n if is_scalar(levels):\n if user_minmax:\n levels = np.linspace(vmin, vmax, levels)\n elif levels == 1:\n levels = np.asarray([(vmin + vmax) / 2])\n else:\n # N in MaxNLocator refers to bins, not ticks\n ticker = mpl.ticker.MaxNLocator(levels - 1)\n levels = ticker.tick_values(vmin, vmax)\n vmin, vmax = levels[0], levels[-1]\n\n if extend is None:\n extend = _determine_extend(calc_data, vmin, vmax)\n\n if levels is not None or isinstance(norm, mpl.colors.BoundaryNorm):\n cmap, newnorm = _build_discrete_cmap(cmap, levels, extend, filled)\n norm = newnorm if norm is None else norm\n\n return dict(vmin=vmin, vmax=vmax, cmap=cmap, extend=extend,\n levels=levels, norm=norm)", "response": "Determine the parameters of a colorbar and range of a colorbar."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _infer_xy_labels_3d(darray, x, y, rgb):\n assert rgb is None or rgb != x\n assert rgb is None or rgb != y\n # Start by detecting and reporting invalid combinations of arguments\n assert darray.ndim == 3\n not_none = [a for a in (x, y, rgb) if a is not None]\n if len(set(not_none)) < len(not_none):\n raise ValueError(\n 'Dimension names must be None or unique strings, but imshow was '\n 'passed x=%r, y=%r, and rgb=%r.' % (x, y, rgb))\n for label in not_none:\n if label not in darray.dims:\n raise ValueError('%r is not a dimension' % (label,))\n\n # Then calculate rgb dimension if certain and check validity\n could_be_color = [label for label in darray.dims\n if darray[label].size in (3, 4) and label not in (x, y)]\n if rgb is None and not could_be_color:\n raise ValueError(\n 'A 3-dimensional array was passed to imshow(), but there is no '\n 'dimension that could be color. At least one dimension must be '\n 'of size 3 (RGB) or 4 (RGBA), and not given as x or y.')\n if rgb is None and len(could_be_color) == 1:\n rgb = could_be_color[0]\n if rgb is not None and darray[rgb].size not in (3, 4):\n raise ValueError('Cannot interpret dim %r of size %s as RGB or RGBA.'\n % (rgb, darray[rgb].size))\n\n # If rgb dimension is still unknown, there must be two or three dimensions\n # in could_be_color. We therefore warn, and use a heuristic to break ties.\n if rgb is None:\n assert len(could_be_color) in (2, 3)\n rgb = could_be_color[-1]\n warnings.warn(\n 'Several dimensions of this array could be colors. Xarray '\n 'will use the last possible dimension (%r) to match '\n 'matplotlib.pyplot.imshow. You can pass names of x, y, '\n 'and/or rgb dimensions to override this guess.' % rgb)\n assert rgb is not None\n\n # Finally, we pick out the red slice and delegate to the 2D version:\n return _infer_xy_labels(darray.isel(**{rgb: 0}), x, y)", "response": "Infer x and y labels for showing RGB images."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninfers x and y labels for a 2D data array.", "response": "def _infer_xy_labels(darray, x, y, imshow=False, rgb=None):\n \"\"\"\n Determine x and y labels. For use in _plot2d\n\n darray must be a 2 dimensional data array, or 3d for imshow only.\n \"\"\"\n assert x is None or x != y\n if imshow and darray.ndim == 3:\n return _infer_xy_labels_3d(darray, x, y, rgb)\n\n if x is None and y is None:\n if darray.ndim != 2:\n raise ValueError('DataArray must be 2d')\n y, x = darray.dims\n elif x is None:\n if y not in darray.dims and y not in darray.coords:\n raise ValueError('y must be a dimension name if x is not supplied')\n x = darray.dims[0] if y == darray.dims[1] else darray.dims[1]\n elif y is None:\n if x not in darray.dims and x not in darray.coords:\n raise ValueError('x must be a dimension name if y is not supplied')\n y = darray.dims[0] if x == darray.dims[1] else darray.dims[1]\n elif any(k not in darray.coords and k not in darray.dims for k in (x, y)):\n raise ValueError('x and y must be coordinate variables')\n return x, y"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef label_from_attrs(da, extra=''):\n ''' Makes informative labels if variable metadata (attrs) follows\n CF conventions. '''\n\n if da.attrs.get('long_name'):\n name = da.attrs['long_name']\n elif da.attrs.get('standard_name'):\n name = da.attrs['standard_name']\n elif da.name is not None:\n name = da.name\n else:\n name = ''\n\n if da.attrs.get('units'):\n units = ' [{}]'.format(da.attrs['units'])\n else:\n units = ''\n\n return '\\n'.join(textwrap.wrap(name + extra + units, 30))", "response": "Makes informative labels if variable metadata ( attrs ) follows\n CF conventions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _interval_to_bound_points(array):\n\n array_boundaries = np.array([x.left for x in array])\n array_boundaries = np.concatenate(\n (array_boundaries, np.array([array[-1].right])))\n\n return array_boundaries", "response": "Helper function which returns an array containing the Intervals boundaries."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _resolve_intervals_2dplot(val, func_name):\n label_extra = ''\n if _valid_other_type(val, [pd.Interval]):\n if func_name == 'pcolormesh':\n val = _interval_to_bound_points(val)\n else:\n val = _interval_to_mid_points(val)\n label_extra = '_center'\n\n return val, label_extra", "response": "Helper function to resolve the values of a coordinate array containing pd. Interval with their mid - points."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if x has a type from types?", "response": "def _valid_other_type(x, types):\n \"\"\"\n Do all elements of x have a type from types?\n \"\"\"\n return all(any(isinstance(el, t) for t in types) for el in np.ravel(x))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _valid_numpy_subdtype(x, numpy_types):\n # If any of the types given in numpy_types is understood as numpy.generic,\n # all possible x will be considered valid. This is probably unwanted.\n for t in numpy_types:\n assert not np.issubdtype(np.generic, t)\n\n return any(np.issubdtype(x.dtype, t) for t in numpy_types)", "response": "Check if x is a subdtype of numpy_types."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _ensure_plottable(*args):\n numpy_types = [np.floating, np.integer, np.timedelta64, np.datetime64]\n other_types = [datetime]\n try:\n import cftime\n cftime_datetime = [cftime.datetime]\n except ImportError:\n cftime_datetime = []\n other_types = other_types + cftime_datetime\n for x in args:\n if not (_valid_numpy_subdtype(np.array(x), numpy_types)\n or _valid_other_type(np.array(x), other_types)):\n raise TypeError('Plotting requires coordinates to be numeric '\n 'or dates of type np.datetime64, '\n 'datetime.datetime, cftime.datetime or '\n 'pd.Interval.')\n if (_valid_other_type(np.array(x), cftime_datetime)\n and not nc_time_axis_available):\n raise ImportError('Plotting of arrays of cftime.datetime '\n 'objects or arrays indexed by '\n 'cftime.datetime objects requires the '\n 'optional `nc-time-axis` (v1.2.0 or later) '\n 'package.')", "response": "Raise exception if any of the arguments are not plottable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating axes with provided parameters.", "response": "def _update_axes(ax, xincrease, yincrease,\n xscale=None, yscale=None,\n xticks=None, yticks=None,\n xlim=None, ylim=None):\n \"\"\"\n Update axes with provided parameters\n \"\"\"\n if xincrease is None:\n pass\n elif xincrease and ax.xaxis_inverted():\n ax.invert_xaxis()\n elif not xincrease and not ax.xaxis_inverted():\n ax.invert_xaxis()\n\n if yincrease is None:\n pass\n elif yincrease and ax.yaxis_inverted():\n ax.invert_yaxis()\n elif not yincrease and not ax.yaxis_inverted():\n ax.invert_yaxis()\n\n # The default xscale, yscale needs to be None.\n # If we set a scale it resets the axes formatters,\n # This means that set_xscale('linear') on a datetime axis\n # will remove the date labels. So only set the scale when explicitly\n # asked to. https://github.com/matplotlib/matplotlib/issues/8740\n if xscale is not None:\n ax.set_xscale(xscale)\n if yscale is not None:\n ax.set_yscale(yscale)\n\n if xticks is not None:\n ax.set_xticks(xticks)\n if yticks is not None:\n ax.set_yticks(yticks)\n\n if xlim is not None:\n ax.set_xlim(xlim)\n if ylim is not None:\n ax.set_ylim(ylim)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _is_monotonic(coord, axis=0):\n if coord.shape[axis] < 3:\n return True\n else:\n n = coord.shape[axis]\n delta_pos = (coord.take(np.arange(1, n), axis=axis) >=\n coord.take(np.arange(0, n - 1), axis=axis))\n delta_neg = (coord.take(np.arange(1, n), axis=axis) <=\n coord.take(np.arange(0, n - 1), axis=axis))\n return np.all(delta_pos) or np.all(delta_neg)", "response": "Return True if the given array is monotonic."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _infer_interval_breaks(coord, axis=0, check_monotonic=False):\n coord = np.asarray(coord)\n\n if check_monotonic and not _is_monotonic(coord, axis=axis):\n raise ValueError(\"The input coordinate is not sorted in increasing \"\n \"order along axis %d. This can lead to unexpected \"\n \"results. Consider calling the `sortby` method on \"\n \"the input DataArray. To plot data with categorical \"\n \"axes, consider using the `heatmap` function from \"\n \"the `seaborn` statistical plotting library.\" % axis)\n\n deltas = 0.5 * np.diff(coord, axis=axis)\n if deltas.size == 0:\n deltas = np.array(0.0)\n first = np.take(coord, [0], axis=axis) - np.take(deltas, [0], axis=axis)\n last = np.take(coord, [-1], axis=axis) + np.take(deltas, [-1], axis=axis)\n trim_last = tuple(slice(None, -1) if n == axis else slice(None)\n for n in range(coord.ndim))\n return np.concatenate([first, coord[trim_last] + deltas, last], axis=axis)", "response": "Infer the interval breaks of a given coordinate."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _process_cmap_cbar_kwargs(func, kwargs, data):\n\n cmap = kwargs.pop('cmap', None)\n colors = kwargs.pop('colors', None)\n\n cbar_kwargs = kwargs.pop('cbar_kwargs', {})\n cbar_kwargs = {} if cbar_kwargs is None else dict(cbar_kwargs)\n\n levels = kwargs.pop('levels', None)\n if 'contour' in func.__name__ and levels is None:\n levels = 7 # this is the matplotlib default\n\n # colors is mutually exclusive with cmap\n if cmap and colors:\n raise ValueError(\"Can't specify both cmap and colors.\")\n\n # colors is only valid when levels is supplied or the plot is of type\n # contour or contourf\n if colors and (('contour' not in func.__name__) and (not levels)):\n raise ValueError(\"Can only specify colors with contour or levels\")\n\n # we should not be getting a list of colors in cmap anymore\n # is there a better way to do this test?\n if isinstance(cmap, (list, tuple)):\n warnings.warn(\"Specifying a list of colors in cmap is deprecated. \"\n \"Use colors keyword instead.\",\n DeprecationWarning, stacklevel=3)\n\n cmap_kwargs = {'plot_data': data,\n 'levels': levels,\n 'cmap': colors if colors else cmap,\n 'filled': func.__name__ != 'contour'}\n\n cmap_args = getfullargspec(_determine_cmap_params).args\n cmap_kwargs.update((a, kwargs[a]) for a in cmap_args if a in kwargs)\n cmap_params = _determine_cmap_params(**cmap_kwargs)\n\n return cmap_params, cbar_kwargs", "response": "Process kwargs that need to be parsed from kwargs and return cmap_params cbar_kwargs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots a DataArray using matplotlib. pyplot.", "response": "def plot(darray, row=None, col=None, col_wrap=None, ax=None, hue=None,\n rtol=0.01, subplot_kws=None, **kwargs):\n \"\"\"\n Default plot of DataArray using matplotlib.pyplot.\n\n Calls xarray plotting function based on the dimensions of\n darray.squeeze()\n\n =============== ===========================\n Dimensions Plotting function\n --------------- ---------------------------\n 1 :py:func:`xarray.plot.line`\n 2 :py:func:`xarray.plot.pcolormesh`\n Anything else :py:func:`xarray.plot.hist`\n =============== ===========================\n\n Parameters\n ----------\n darray : DataArray\n row : string, optional\n If passed, make row faceted plots on this dimension name\n col : string, optional\n If passed, make column faceted plots on this dimension name\n hue : string, optional\n If passed, make faceted line plots with hue on this dimension name\n col_wrap : integer, optional\n Use together with ``col`` to wrap faceted plots\n ax : matplotlib axes, optional\n If None, uses the current axis. Not applicable when using facets.\n rtol : number, optional\n Relative tolerance used to determine if the indexes\n are uniformly spaced. Usually a small positive number.\n subplot_kws : dict, optional\n Dictionary of keyword arguments for matplotlib subplots. Only applies\n to FacetGrid plotting.\n **kwargs : optional\n Additional keyword arguments to matplotlib\n\n \"\"\"\n darray = darray.squeeze()\n\n plot_dims = set(darray.dims)\n plot_dims.discard(row)\n plot_dims.discard(col)\n plot_dims.discard(hue)\n\n ndims = len(plot_dims)\n\n error_msg = ('Only 1d and 2d plots are supported for facets in xarray. '\n 'See the package `Seaborn` for more options.')\n\n if ndims in [1, 2]:\n if row or col:\n kwargs['row'] = row\n kwargs['col'] = col\n kwargs['col_wrap'] = col_wrap\n kwargs['subplot_kws'] = subplot_kws\n if ndims == 1:\n plotfunc = line\n kwargs['hue'] = hue\n elif ndims == 2:\n if hue:\n plotfunc = line\n kwargs['hue'] = hue\n else:\n plotfunc = pcolormesh\n else:\n if row or col or hue:\n raise ValueError(error_msg)\n plotfunc = hist\n\n kwargs['ax'] = ax\n\n return plotfunc(darray, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction to plot a single line of the given DataArray index against values", "response": "def line(darray, *args, **kwargs):\n \"\"\"\n Line plot of DataArray index against values\n\n Wraps :func:`matplotlib:matplotlib.pyplot.plot`\n\n Parameters\n ----------\n darray : DataArray\n Must be 1 dimensional\n figsize : tuple, optional\n A tuple (width, height) of the figure in inches.\n Mutually exclusive with ``size`` and ``ax``.\n aspect : scalar, optional\n Aspect ratio of plot, so that ``aspect * size`` gives the width in\n inches. Only used if a ``size`` is provided.\n size : scalar, optional\n If provided, create a new figure for the plot with the given size.\n Height (in inches) of each plot. See also: ``aspect``.\n ax : matplotlib axes object, optional\n Axis on which to plot this figure. By default, use the current axis.\n Mutually exclusive with ``size`` and ``figsize``.\n hue : string, optional\n Dimension or coordinate for which you want multiple lines plotted.\n If plotting against a 2D coordinate, ``hue`` must be a dimension.\n x, y : string, optional\n Dimensions or coordinates for x, y axis.\n Only one of these may be specified.\n The other coordinate plots values from the DataArray on which this\n plot method is called.\n xscale, yscale : 'linear', 'symlog', 'log', 'logit', optional\n Specifies scaling for the x- and y-axes respectively\n xticks, yticks : Specify tick locations for x- and y-axes\n xlim, ylim : Specify x- and y-axes limits\n xincrease : None, True, or False, optional\n Should the values on the x axes be increasing from left to right?\n if None, use the default for the matplotlib function.\n yincrease : None, True, or False, optional\n Should the values on the y axes be increasing from top to bottom?\n if None, use the default for the matplotlib function.\n add_legend : boolean, optional\n Add legend with y axis coordinates (2D inputs only).\n *args, **kwargs : optional\n Additional arguments to matplotlib.pyplot.plot\n\n \"\"\"\n\n # Handle facetgrids first\n row = kwargs.pop('row', None)\n col = kwargs.pop('col', None)\n if row or col:\n allargs = locals().copy()\n allargs.update(allargs.pop('kwargs'))\n allargs.pop('darray')\n return _easy_facetgrid(darray, line, kind='line', **allargs)\n\n ndims = len(darray.dims)\n if ndims > 2:\n raise ValueError('Line plots are for 1- or 2-dimensional DataArrays. '\n 'Passed DataArray has {ndims} '\n 'dimensions'.format(ndims=ndims))\n\n # Ensures consistency with .plot method\n figsize = kwargs.pop('figsize', None)\n aspect = kwargs.pop('aspect', None)\n size = kwargs.pop('size', None)\n ax = kwargs.pop('ax', None)\n hue = kwargs.pop('hue', None)\n x = kwargs.pop('x', None)\n y = kwargs.pop('y', None)\n xincrease = kwargs.pop('xincrease', None) # default needs to be None\n yincrease = kwargs.pop('yincrease', None)\n xscale = kwargs.pop('xscale', None) # default needs to be None\n yscale = kwargs.pop('yscale', None)\n xticks = kwargs.pop('xticks', None)\n yticks = kwargs.pop('yticks', None)\n xlim = kwargs.pop('xlim', None)\n ylim = kwargs.pop('ylim', None)\n add_legend = kwargs.pop('add_legend', True)\n _labels = kwargs.pop('_labels', True)\n if args is ():\n args = kwargs.pop('args', ())\n\n ax = get_axis(figsize, size, aspect, ax)\n xplt, yplt, hueplt, xlabel, ylabel, huelabel = \\\n _infer_line_data(darray, x, y, hue)\n\n # Remove pd.Intervals if contained in xplt.values.\n if _valid_other_type(xplt.values, [pd.Interval]):\n # Is it a step plot? (see matplotlib.Axes.step)\n if kwargs.get('linestyle', '').startswith('steps-'):\n xplt_val, yplt_val = _interval_to_double_bound_points(xplt.values,\n yplt.values)\n # Remove steps-* to be sure that matplotlib is not confused\n kwargs['linestyle'] = (kwargs['linestyle']\n .replace('steps-pre', '')\n .replace('steps-post', '')\n .replace('steps-mid', ''))\n if kwargs['linestyle'] == '':\n kwargs.pop('linestyle')\n else:\n xplt_val = _interval_to_mid_points(xplt.values)\n yplt_val = yplt.values\n xlabel += '_center'\n else:\n xplt_val = xplt.values\n yplt_val = yplt.values\n\n _ensure_plottable(xplt_val, yplt_val)\n\n primitive = ax.plot(xplt_val, yplt_val, *args, **kwargs)\n\n if _labels:\n if xlabel is not None:\n ax.set_xlabel(xlabel)\n\n if ylabel is not None:\n ax.set_ylabel(ylabel)\n\n ax.set_title(darray._title_for_slice())\n\n if darray.ndim == 2 and add_legend:\n ax.legend(handles=primitive,\n labels=list(hueplt.values),\n title=huelabel)\n\n # Rotate dates on xlabels\n # Do this without calling autofmt_xdate so that x-axes ticks\n # on other subplots (if any) are not deleted.\n # https://stackoverflow.com/questions/17430105/autofmt-xdate-deletes-x-axis-labels-of-all-subplots\n if np.issubdtype(xplt.dtype, np.datetime64):\n for xlabels in ax.get_xticklabels():\n xlabels.set_rotation(30)\n xlabels.set_ha('right')\n\n _update_axes(ax, xincrease, yincrease, xscale, yscale,\n xticks, yticks, xlim, ylim)\n\n return primitive"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef step(darray, *args, **kwargs):\n if ('ls' in kwargs.keys()) and ('linestyle' not in kwargs.keys()):\n kwargs['linestyle'] = kwargs.pop('ls')\n\n where = kwargs.pop('where', 'pre')\n\n if where not in ('pre', 'post', 'mid'):\n raise ValueError(\"'where' argument to step must be \"\n \"'pre', 'post' or 'mid'\")\n\n kwargs['linestyle'] = 'steps-' + where + kwargs.get('linestyle', '')\n\n return line(darray, *args, **kwargs)", "response": "This function is a convenience function for plotting a single step plot of DataArray index against values\nFormula"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef hist(darray, figsize=None, size=None, aspect=None, ax=None, **kwargs):\n ax = get_axis(figsize, size, aspect, ax)\n\n xincrease = kwargs.pop('xincrease', None) # default needs to be None\n yincrease = kwargs.pop('yincrease', None)\n xscale = kwargs.pop('xscale', None) # default needs to be None\n yscale = kwargs.pop('yscale', None)\n xticks = kwargs.pop('xticks', None)\n yticks = kwargs.pop('yticks', None)\n xlim = kwargs.pop('xlim', None)\n ylim = kwargs.pop('ylim', None)\n\n no_nan = np.ravel(darray.values)\n no_nan = no_nan[pd.notnull(no_nan)]\n\n primitive = ax.hist(no_nan, **kwargs)\n\n ax.set_title('Histogram')\n ax.set_xlabel(label_from_attrs(darray))\n\n _update_axes(ax, xincrease, yincrease, xscale, yscale,\n xticks, yticks, xlim, ylim)\n\n return primitive", "response": "Plots a histogram of the given DataArray."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _plot2d(plotfunc):\n commondoc = \"\"\"\n Parameters\n ----------\n darray : DataArray\n Must be 2 dimensional, unless creating faceted plots\n x : string, optional\n Coordinate for x axis. If None use darray.dims[1]\n y : string, optional\n Coordinate for y axis. If None use darray.dims[0]\n figsize : tuple, optional\n A tuple (width, height) of the figure in inches.\n Mutually exclusive with ``size`` and ``ax``.\n aspect : scalar, optional\n Aspect ratio of plot, so that ``aspect * size`` gives the width in\n inches. Only used if a ``size`` is provided.\n size : scalar, optional\n If provided, create a new figure for the plot with the given size.\n Height (in inches) of each plot. See also: ``aspect``.\n ax : matplotlib axes object, optional\n Axis on which to plot this figure. By default, use the current axis.\n Mutually exclusive with ``size`` and ``figsize``.\n row : string, optional\n If passed, make row faceted plots on this dimension name\n col : string, optional\n If passed, make column faceted plots on this dimension name\n col_wrap : integer, optional\n Use together with ``col`` to wrap faceted plots\n xscale, yscale : 'linear', 'symlog', 'log', 'logit', optional\n Specifies scaling for the x- and y-axes respectively\n xticks, yticks : Specify tick locations for x- and y-axes\n xlim, ylim : Specify x- and y-axes limits\n xincrease : None, True, or False, optional\n Should the values on the x axes be increasing from left to right?\n if None, use the default for the matplotlib function.\n yincrease : None, True, or False, optional\n Should the values on the y axes be increasing from top to bottom?\n if None, use the default for the matplotlib function.\n add_colorbar : Boolean, optional\n Adds colorbar to axis\n add_labels : Boolean, optional\n Use xarray metadata to label axes\n norm : ``matplotlib.colors.Normalize`` instance, optional\n If the ``norm`` has vmin or vmax specified, the corresponding kwarg\n must be None.\n vmin, vmax : floats, optional\n Values to anchor the colormap, otherwise they are inferred from the\n data and other keyword arguments. When a diverging dataset is inferred,\n setting one of these values will fix the other by symmetry around\n ``center``. Setting both values prevents use of a diverging colormap.\n If discrete levels are provided as an explicit list, both of these\n values are ignored.\n cmap : matplotlib colormap name or object, optional\n The mapping from data values to color space. If not provided, this\n will be either be ``viridis`` (if the function infers a sequential\n dataset) or ``RdBu_r`` (if the function infers a diverging dataset).\n When `Seaborn` is installed, ``cmap`` may also be a `seaborn`\n color palette. If ``cmap`` is seaborn color palette and the plot type\n is not ``contour`` or ``contourf``, ``levels`` must also be specified.\n colors : discrete colors to plot, optional\n A single color or a list of colors. If the plot type is not ``contour``\n or ``contourf``, the ``levels`` argument is required.\n center : float, optional\n The value at which to center the colormap. Passing this value implies\n use of a diverging colormap. Setting it to ``False`` prevents use of a\n diverging colormap.\n robust : bool, optional\n If True and ``vmin`` or ``vmax`` are absent, the colormap range is\n computed with 2nd and 98th percentiles instead of the extreme values.\n extend : {'neither', 'both', 'min', 'max'}, optional\n How to draw arrows extending the colorbar beyond its limits. If not\n provided, extend is inferred from vmin, vmax and the data limits.\n levels : int or list-like object, optional\n Split the colormap (cmap) into discrete color intervals. If an integer\n is provided, \"nice\" levels are chosen based on the data range: this can\n imply that the final number of levels is not exactly the expected one.\n Setting ``vmin`` and/or ``vmax`` with ``levels=N`` is equivalent to\n setting ``levels=np.linspace(vmin, vmax, N)``.\n infer_intervals : bool, optional\n Only applies to pcolormesh. If True, the coordinate intervals are\n passed to pcolormesh. If False, the original coordinates are used\n (this can be useful for certain map projections). The default is to\n always infer intervals, unless the mesh is irregular and plotted on\n a map projection.\n subplot_kws : dict, optional\n Dictionary of keyword arguments for matplotlib subplots. Only applies\n to FacetGrid plotting.\n cbar_ax : matplotlib Axes, optional\n Axes in which to draw the colorbar.\n cbar_kwargs : dict, optional\n Dictionary of keyword arguments to pass to the colorbar.\n **kwargs : optional\n Additional arguments to wrapped matplotlib function\n\n Returns\n -------\n artist :\n The same type of primitive artist that the wrapped matplotlib\n function returns\n \"\"\"\n\n # Build on the original docstring\n plotfunc.__doc__ = '%s\\n%s' % (plotfunc.__doc__, commondoc)\n\n @functools.wraps(plotfunc)\n def newplotfunc(darray, x=None, y=None, figsize=None, size=None,\n aspect=None, ax=None, row=None, col=None,\n col_wrap=None, xincrease=True, yincrease=True,\n add_colorbar=None, add_labels=True, vmin=None, vmax=None,\n cmap=None, center=None, robust=False, extend=None,\n levels=None, infer_intervals=None, colors=None,\n subplot_kws=None, cbar_ax=None, cbar_kwargs=None,\n xscale=None, yscale=None, xticks=None, yticks=None,\n xlim=None, ylim=None, norm=None, **kwargs):\n # All 2d plots in xarray share this function signature.\n # Method signature below should be consistent.\n\n # Decide on a default for the colorbar before facetgrids\n if add_colorbar is None:\n add_colorbar = plotfunc.__name__ != 'contour'\n imshow_rgb = (\n plotfunc.__name__ == 'imshow' and\n darray.ndim == (3 + (row is not None) + (col is not None)))\n if imshow_rgb:\n # Don't add a colorbar when showing an image with explicit colors\n add_colorbar = False\n # Matplotlib does not support normalising RGB data, so do it here.\n # See eg. https://github.com/matplotlib/matplotlib/pull/10220\n if robust or vmax is not None or vmin is not None:\n darray = _rescale_imshow_rgb(darray, vmin, vmax, robust)\n vmin, vmax, robust = None, None, False\n\n # Handle facetgrids first\n if row or col:\n allargs = locals().copy()\n allargs.pop('imshow_rgb')\n allargs.update(allargs.pop('kwargs'))\n allargs.pop('darray')\n # Need the decorated plotting function\n allargs['plotfunc'] = globals()[plotfunc.__name__]\n return _easy_facetgrid(darray, kind='dataarray', **allargs)\n\n plt = import_matplotlib_pyplot()\n\n rgb = kwargs.pop('rgb', None)\n if rgb is not None and plotfunc.__name__ != 'imshow':\n raise ValueError('The \"rgb\" keyword is only valid for imshow()')\n elif rgb is not None and not imshow_rgb:\n raise ValueError('The \"rgb\" keyword is only valid for imshow()'\n 'with a three-dimensional array (per facet)')\n\n xlab, ylab = _infer_xy_labels(\n darray=darray, x=x, y=y, imshow=imshow_rgb, rgb=rgb)\n\n # better to pass the ndarrays directly to plotting functions\n xval = darray[xlab].values\n yval = darray[ylab].values\n\n # check if we need to broadcast one dimension\n if xval.ndim < yval.ndim:\n xval = np.broadcast_to(xval, yval.shape)\n\n if yval.ndim < xval.ndim:\n yval = np.broadcast_to(yval, xval.shape)\n\n # May need to transpose for correct x, y labels\n # xlab may be the name of a coord, we have to check for dim names\n if imshow_rgb:\n # For RGB[A] images, matplotlib requires the color dimension\n # to be last. In Xarray the order should be unimportant, so\n # we transpose to (y, x, color) to make this work.\n yx_dims = (ylab, xlab)\n dims = yx_dims + tuple(d for d in darray.dims if d not in yx_dims)\n if dims != darray.dims:\n darray = darray.transpose(*dims)\n elif darray[xlab].dims[-1] == darray.dims[0]:\n darray = darray.transpose()\n\n # Pass the data as a masked ndarray too\n zval = darray.to_masked_array(copy=False)\n\n # Replace pd.Intervals if contained in xval or yval.\n xplt, xlab_extra = _resolve_intervals_2dplot(xval, plotfunc.__name__)\n yplt, ylab_extra = _resolve_intervals_2dplot(yval, plotfunc.__name__)\n\n _ensure_plottable(xplt, yplt)\n\n cmap_params, cbar_kwargs = _process_cmap_cbar_kwargs(\n plotfunc, locals(), zval.data)\n\n if 'contour' in plotfunc.__name__:\n # extend is a keyword argument only for contour and contourf, but\n # passing it to the colorbar is sufficient for imshow and\n # pcolormesh\n kwargs['extend'] = cmap_params['extend']\n kwargs['levels'] = cmap_params['levels']\n # if colors == a single color, matplotlib draws dashed negative\n # contours. we lose this feature if we pass cmap and not colors\n if isinstance(colors, str):\n cmap_params['cmap'] = None\n kwargs['colors'] = colors\n\n if 'pcolormesh' == plotfunc.__name__:\n kwargs['infer_intervals'] = infer_intervals\n\n if 'imshow' == plotfunc.__name__ and isinstance(aspect, str):\n # forbid usage of mpl strings\n raise ValueError(\"plt.imshow's `aspect` kwarg is not available \"\n \"in xarray\")\n\n ax = get_axis(figsize, size, aspect, ax)\n primitive = plotfunc(xplt, yplt, zval, ax=ax, cmap=cmap_params['cmap'],\n vmin=cmap_params['vmin'],\n vmax=cmap_params['vmax'],\n norm=cmap_params['norm'],\n **kwargs)\n\n # Label the plot with metadata\n if add_labels:\n ax.set_xlabel(label_from_attrs(darray[xlab], xlab_extra))\n ax.set_ylabel(label_from_attrs(darray[ylab], ylab_extra))\n ax.set_title(darray._title_for_slice())\n\n if add_colorbar:\n if add_labels and 'label' not in cbar_kwargs:\n cbar_kwargs['label'] = label_from_attrs(darray)\n cbar = _add_colorbar(primitive, ax, cbar_ax, cbar_kwargs,\n cmap_params)\n\n elif (cbar_ax is not None or cbar_kwargs):\n # inform the user about keywords which aren't used\n raise ValueError(\"cbar_ax and cbar_kwargs can't be used with \"\n \"add_colorbar=False.\")\n\n # origin kwarg overrides yincrease\n if 'origin' in kwargs:\n yincrease = None\n\n _update_axes(ax, xincrease, yincrease, xscale, yscale,\n xticks, yticks, xlim, ylim)\n\n # Rotate dates on xlabels\n # Do this without calling autofmt_xdate so that x-axes ticks\n # on other subplots (if any) are not deleted.\n # https://stackoverflow.com/questions/17430105/autofmt-xdate-deletes-x-axis-labels-of-all-subplots\n if np.issubdtype(xplt.dtype, np.datetime64):\n for xlabels in ax.get_xticklabels():\n xlabels.set_rotation(30)\n xlabels.set_ha('right')\n\n return primitive\n\n # For use as DataArray.plot.plotmethod\n @functools.wraps(newplotfunc)\n def plotmethod(_PlotMethods_obj, x=None, y=None, figsize=None, size=None,\n aspect=None, ax=None, row=None, col=None, col_wrap=None,\n xincrease=True, yincrease=True, add_colorbar=None,\n add_labels=True, vmin=None, vmax=None, cmap=None,\n colors=None, center=None, robust=False, extend=None,\n levels=None, infer_intervals=None, subplot_kws=None,\n cbar_ax=None, cbar_kwargs=None,\n xscale=None, yscale=None, xticks=None, yticks=None,\n xlim=None, ylim=None, norm=None, **kwargs):\n \"\"\"\n The method should have the same signature as the function.\n\n This just makes the method work on Plotmethods objects,\n and passes all the other arguments straight through.\n \"\"\"\n allargs = locals()\n allargs['darray'] = _PlotMethods_obj._da\n allargs.update(kwargs)\n for arg in ['_PlotMethods_obj', 'newplotfunc', 'kwargs']:\n del allargs[arg]\n return newplotfunc(**allargs)\n\n # Add to class _PlotMethods\n setattr(_PlotMethods, plotmethod.__name__, plotmethod)\n\n return newplotfunc", "response": "Decorator for common 2d plotting logic"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nplotting a 2D array of pixels on a matplotlib axes.", "response": "def imshow(x, y, z, ax, **kwargs):\n \"\"\"\n Image plot of 2d DataArray using matplotlib.pyplot\n\n Wraps :func:`matplotlib:matplotlib.pyplot.imshow`\n\n While other plot methods require the DataArray to be strictly\n two-dimensional, ``imshow`` also accepts a 3D array where some\n dimension can be interpreted as RGB or RGBA color channels and\n allows this dimension to be specified via the kwarg ``rgb=``.\n\n Unlike matplotlib, Xarray can apply ``vmin`` and ``vmax`` to RGB or RGBA\n data, by applying a single scaling factor and offset to all bands.\n Passing ``robust=True`` infers ``vmin`` and ``vmax``\n :ref:`in the usual way `.\n\n .. note::\n This function needs uniformly spaced coordinates to\n properly label the axes. Call DataArray.plot() to check.\n\n The pixels are centered on the coordinates values. Ie, if the coordinate\n value is 3.2 then the pixels for those coordinates will be centered on 3.2.\n \"\"\"\n\n if x.ndim != 1 or y.ndim != 1:\n raise ValueError('imshow requires 1D coordinates, try using '\n 'pcolormesh or contour(f)')\n\n # Centering the pixels- Assumes uniform spacing\n try:\n xstep = (x[1] - x[0]) / 2.0\n except IndexError:\n # Arbitrary default value, similar to matplotlib behaviour\n xstep = .1\n try:\n ystep = (y[1] - y[0]) / 2.0\n except IndexError:\n ystep = .1\n left, right = x[0] - xstep, x[-1] + xstep\n bottom, top = y[-1] + ystep, y[0] - ystep\n\n defaults = {'origin': 'upper',\n 'interpolation': 'nearest'}\n\n if not hasattr(ax, 'projection'):\n # not for cartopy geoaxes\n defaults['aspect'] = 'auto'\n\n # Allow user to override these defaults\n defaults.update(kwargs)\n\n if defaults['origin'] == 'upper':\n defaults['extent'] = [left, right, bottom, top]\n else:\n defaults['extent'] = [left, right, top, bottom]\n\n if z.ndim == 3:\n # matplotlib imshow uses black for missing data, but Xarray makes\n # missing data transparent. We therefore add an alpha channel if\n # there isn't one, and set it to transparent where data is masked.\n if z.shape[-1] == 3:\n alpha = np.ma.ones(z.shape[:2] + (1,), dtype=z.dtype)\n if np.issubdtype(z.dtype, np.integer):\n alpha *= 255\n z = np.ma.concatenate((z, alpha), axis=2)\n else:\n z = z.copy()\n z[np.any(z.mask, axis=-1), -1] = 0\n\n primitive = ax.imshow(z, **defaults)\n\n return primitive"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef contour(x, y, z, ax, **kwargs):\n primitive = ax.contour(x, y, z, **kwargs)\n return primitive", "response": "Contour plot of 2d DataArray"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef contourf(x, y, z, ax, **kwargs):\n primitive = ax.contourf(x, y, z, **kwargs)\n return primitive", "response": "Wrapper around matplotlib. pyplot. contourf"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pcolormesh(x, y, z, ax, infer_intervals=None, **kwargs):\n\n # decide on a default for infer_intervals (GH781)\n x = np.asarray(x)\n if infer_intervals is None:\n if hasattr(ax, 'projection'):\n if len(x.shape) == 1:\n infer_intervals = True\n else:\n infer_intervals = False\n else:\n infer_intervals = True\n\n if (infer_intervals and\n ((np.shape(x)[0] == np.shape(z)[1]) or\n ((x.ndim > 1) and (np.shape(x)[1] == np.shape(z)[1])))):\n if len(x.shape) == 1:\n x = _infer_interval_breaks(x, check_monotonic=True)\n else:\n # we have to infer the intervals on both axes\n x = _infer_interval_breaks(x, axis=1)\n x = _infer_interval_breaks(x, axis=0)\n\n if (infer_intervals and\n (np.shape(y)[0] == np.shape(z)[0])):\n if len(y.shape) == 1:\n y = _infer_interval_breaks(y, check_monotonic=True)\n else:\n # we have to infer the intervals on both axes\n y = _infer_interval_breaks(y, axis=1)\n y = _infer_interval_breaks(y, axis=0)\n\n primitive = ax.pcolormesh(x, y, z, **kwargs)\n\n # by default, pcolormesh picks \"round\" values for bounds\n # this results in ugly looking plots with lots of surrounding whitespace\n if not hasattr(ax, 'projection') and x.ndim == 1 and y.ndim == 1:\n # not a cartopy geoaxis\n ax.set_xlim(x[0], x[-1])\n ax.set_ylim(y[0], y[-1])\n\n return primitive", "response": "A wrapper around matplotlib. pyplot. pcolormesh that takes a 2d array x y z and returns a 2d array x y z."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_scheduler(get=None, collection=None):\n try:\n # dask 0.18.1 and later\n from dask.base import get_scheduler\n actual_get = get_scheduler(get, collection)\n except ImportError:\n try:\n from dask.utils import effective_get\n actual_get = effective_get(get, collection)\n except ImportError:\n return None\n\n try:\n from dask.distributed import Client\n if isinstance(actual_get.__self__, Client):\n return 'distributed'\n except (ImportError, AttributeError):\n try:\n import dask.multiprocessing\n if actual_get == dask.multiprocessing.get:\n return 'multiprocessing'\n else:\n return 'threaded'\n except ImportError:\n return 'threaded'", "response": "Determine the dask scheduler that is being used."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef acquire(lock, blocking=True):\n if blocking:\n # no arguments needed\n return lock.acquire()\n elif DistributedLock is not None and isinstance(lock, DistributedLock):\n # distributed.Lock doesn't support the blocking argument yet:\n # https://github.com/dask/distributed/pull/2412\n return lock.acquire(timeout=0)\n else:\n # \"blocking\" keyword argument not supported for:\n # - threading.Lock on Python 2.\n # - dask.SerializableLock with dask v1.0.0 or earlier.\n # - multiprocessing.Lock calls the argument \"block\" instead.\n return lock.acquire(blocking)", "response": "Acquire a lock possibly in a non - blocking fashion."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncombining a sequence of locks into a single lock.", "response": "def combine_locks(locks):\n \"\"\"Combine a sequence of locks into a single lock.\"\"\"\n all_locks = []\n for lock in locks:\n if isinstance(lock, CombinedLock):\n all_locks.extend(lock.locks)\n elif lock is not None:\n all_locks.append(lock)\n\n num_locks = len(all_locks)\n if num_locks > 1:\n return CombinedLock(all_locks)\n elif num_locks == 1:\n return all_locks[0]\n else:\n return DummyLock()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a list of dimensions to squeeze out.", "response": "def get_squeeze_dims(xarray_obj,\n dim: Union[Hashable, Iterable[Hashable], None] = None,\n axis: Union[int, Iterable[int], None] = None\n ) -> List[Hashable]:\n \"\"\"Get a list of dimensions to squeeze out.\n \"\"\"\n if dim is not None and axis is not None:\n raise ValueError('cannot use both parameters `axis` and `dim`')\n if dim is None and axis is None:\n return [d for d, s in xarray_obj.sizes.items() if s == 1]\n\n if isinstance(dim, Iterable) and not isinstance(dim, str):\n dim = list(dim)\n elif dim is not None:\n dim = [dim]\n else:\n assert axis is not None\n if isinstance(axis, int):\n axis = [axis]\n axis = list(axis)\n if any(not isinstance(a, int) for a in axis):\n raise TypeError(\n 'parameter `axis` must be int or iterable of int.')\n alldims = list(xarray_obj.sizes.keys())\n dim = [alldims[a] for a in axis]\n\n if any(xarray_obj.sizes[k] > 1 for k in dim):\n raise ValueError('cannot select a dimension to squeeze out '\n 'which has length greater than one')\n return dim"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef full_like(other, fill_value, dtype: Union[str, np.dtype, None] = None):\n from .dataarray import DataArray\n from .dataset import Dataset\n from .variable import Variable\n\n if isinstance(other, Dataset):\n data_vars = OrderedDict(\n (k, _full_like_variable(v, fill_value, dtype))\n for k, v in other.data_vars.items())\n return Dataset(data_vars, coords=other.coords, attrs=other.attrs)\n elif isinstance(other, DataArray):\n return DataArray(\n _full_like_variable(other.variable, fill_value, dtype),\n dims=other.dims, coords=other.coords, attrs=other.attrs,\n name=other.name)\n elif isinstance(other, Variable):\n return _full_like_variable(other, fill_value, dtype)\n else:\n raise TypeError(\"Expected DataArray, Dataset, or Variable\")", "response": "Return a new object with the same shape and type as a given object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if a dtype is a subclass of the numpy datetime types", "response": "def is_np_datetime_like(dtype: Union[str, np.dtype]) -> bool:\n \"\"\"Check if a dtype is a subclass of the numpy datetime types\n \"\"\"\n return (np.issubdtype(dtype, np.datetime64) or\n np.issubdtype(dtype, np.timedelta64))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _contains_cftime_datetimes(array) -> bool:\n try:\n from cftime import datetime as cftime_datetime\n except ImportError:\n return False\n else:\n if array.dtype == np.dtype('O') and array.size > 0:\n sample = array.ravel()[0]\n if isinstance(sample, dask_array_type):\n sample = sample.compute()\n if isinstance(sample, np.ndarray):\n sample = sample.item()\n return isinstance(sample, cftime_datetime)\n else:\n return False", "response": "Check if an array contains cftime. datetime objects\nInvitements"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_axis_num(self, dim: Union[Hashable, Iterable[Hashable]]\n ) -> Union[int, Tuple[int, ...]]:\n \"\"\"Return axis number(s) corresponding to dimension(s) in this array.\n\n Parameters\n ----------\n dim : str or iterable of str\n Dimension name(s) for which to lookup axes.\n\n Returns\n -------\n int or tuple of int\n Axis number or numbers corresponding to the given dimensions.\n \"\"\"\n if isinstance(dim, Iterable) and not isinstance(dim, str):\n return tuple(self._get_axis_num(d) for d in dim)\n else:\n return self._get_axis_num(dim)", "response": "Returns the axis number or tuple corresponding to the given dimension."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sizes(self: Any) -> Mapping[Hashable, int]:\n return Frozen(OrderedDict(zip(self.dims, self.shape)))", "response": "Ordered mapping from dimension names to lengths."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprovides method for the key - autocompletions in IPython.", "response": "def _ipython_key_completions_(self) -> List[str]:\n \"\"\"Provide method for the key-autocompletions in IPython.\n See http://ipython.readthedocs.io/en/stable/config/integrating.html#tab-completion\n For the details.\n \"\"\" # noqa\n item_lists = [item\n for sublist in self._item_sources\n for item in sublist\n if isinstance(item, str)]\n return list(set(item_lists))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new object with squeezed data.", "response": "def squeeze(self, dim: Union[Hashable, Iterable[Hashable], None] = None,\n drop: bool = False,\n axis: Union[int, Iterable[int], None] = None):\n \"\"\"Return a new object with squeezed data.\n\n Parameters\n ----------\n dim : None or Hashable or iterable of Hashable, optional\n Selects a subset of the length one dimensions. If a dimension is\n selected with length greater than one, an error is raised. If\n None, all length one dimensions are squeezed.\n drop : bool, optional\n If ``drop=True``, drop squeezed coordinates instead of making them\n scalar.\n axis : None or int or iterable of int, optional\n Like dim, but positional.\n\n Returns\n -------\n squeezed : same type as caller\n This object, but with with all or a subset of the dimensions of\n length 1 removed.\n\n See Also\n --------\n numpy.squeeze\n \"\"\"\n dims = get_squeeze_dims(self, dim, axis)\n return self.isel(drop=drop, **{d: 0 for d in dims})"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_index(self, key: Hashable) -> pd.Index:\n if key not in self.dims:\n raise KeyError(key)\n\n try:\n return self.indexes[key]\n except KeyError:\n # need to ensure dtype=int64 in case range is empty on Python 2\n return pd.Index(range(self.sizes[key]), name=key, dtype=np.int64)", "response": "Get an index for a dimension with fall - back to a default RangeIndex\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nassigns new coordinates to the current object.", "response": "def assign_coords(self, **kwargs):\n \"\"\"Assign new coordinates to this object.\n\n Returns a new object with all the original data in addition to the new\n coordinates.\n\n Parameters\n ----------\n kwargs : keyword, value pairs\n keywords are the variables names. If the values are callable, they\n are computed on this object and assigned to new coordinate\n variables. If the values are not callable, (e.g. a DataArray,\n scalar, or array), they are simply assigned.\n\n Returns\n -------\n assigned : same type as caller\n A new object with the new coordinates in addition to the existing\n data.\n\n Examples\n --------\n\n Convert longitude coordinates from 0-359 to -180-179:\n\n >>> da = xr.DataArray(np.random.rand(4),\n ... coords=[np.array([358, 359, 0, 1])],\n ... dims='lon')\n >>> da\n \n array([0.28298 , 0.667347, 0.657938, 0.177683])\n Coordinates:\n * lon (lon) int64 358 359 0 1\n >>> da.assign_coords(lon=(((da.lon + 180) % 360) - 180))\n \n array([0.28298 , 0.667347, 0.657938, 0.177683])\n Coordinates:\n * lon (lon) int64 -2 -1 0 1\n\n Notes\n -----\n Since ``kwargs`` is a dictionary, the order of your arguments may not\n be preserved, and so the order of the new variables is not well\n defined. Assigning multiple variables within the same ``assign_coords``\n is possible, but you cannot reference other variables created within\n the same ``assign_coords`` call.\n\n See also\n --------\n Dataset.assign\n Dataset.swap_dims\n \"\"\"\n data = self.copy(deep=False)\n results = self._calc_assign_results(kwargs)\n data.coords.update(results)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nassign new attrs to this object.", "response": "def assign_attrs(self, *args, **kwargs):\n \"\"\"Assign new attrs to this object.\n\n Returns a new object equivalent to self.attrs.update(*args, **kwargs).\n\n Parameters\n ----------\n args : positional arguments passed into ``attrs.update``.\n kwargs : keyword arguments passed into ``attrs.update``.\n\n Returns\n -------\n assigned : same type as caller\n A new object with the new attrs in addition to the existing data.\n\n See also\n --------\n Dataset.assign\n \"\"\"\n out = self.copy(deep=False)\n out.attrs.update(*args, **kwargs)\n return out"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\napply func(self, *args, **kwargs) This method replicates the pandas method of the same name. Parameters ---------- func : function function to apply to this xarray object (Dataset/DataArray). ``args``, and ``kwargs`` are passed into ``func``. Alternatively a ``(callable, data_keyword)`` tuple where ``data_keyword`` is a string indicating the keyword of ``callable`` that expects the xarray object. args : positional arguments passed into ``func``. kwargs : a dictionary of keyword arguments passed into ``func``. Returns ------- object : the return type of ``func``. Notes ----- Use ``.pipe`` when chaining together functions that expect xarray or pandas objects, e.g., instead of writing >>> f(g(h(ds), arg1=a), arg2=b, arg3=c) You can write >>> (ds.pipe(h) ... .pipe(g, arg1=a) ... .pipe(f, arg2=b, arg3=c) ... ) If you have a function that takes the data as (say) the second argument, pass a tuple indicating which keyword expects the data. For example, suppose ``f`` takes its data as ``arg2``: >>> (ds.pipe(h) ... .pipe(g, arg1=a) ... .pipe((f, 'arg2'), arg1=a, arg3=c) ... ) See Also -------- pandas.DataFrame.pipe", "response": "def pipe(self, func: Union[Callable[..., T], Tuple[Callable[..., T], str]],\n *args, **kwargs) -> T:\n \"\"\"\n Apply func(self, *args, **kwargs)\n\n This method replicates the pandas method of the same name.\n\n Parameters\n ----------\n func : function\n function to apply to this xarray object (Dataset/DataArray).\n ``args``, and ``kwargs`` are passed into ``func``.\n Alternatively a ``(callable, data_keyword)`` tuple where\n ``data_keyword`` is a string indicating the keyword of\n ``callable`` that expects the xarray object.\n args : positional arguments passed into ``func``.\n kwargs : a dictionary of keyword arguments passed into ``func``.\n\n Returns\n -------\n object : the return type of ``func``.\n\n Notes\n -----\n\n Use ``.pipe`` when chaining together functions that expect\n xarray or pandas objects, e.g., instead of writing\n\n >>> f(g(h(ds), arg1=a), arg2=b, arg3=c)\n\n You can write\n\n >>> (ds.pipe(h)\n ... .pipe(g, arg1=a)\n ... .pipe(f, arg2=b, arg3=c)\n ... )\n\n If you have a function that takes the data as (say) the second\n argument, pass a tuple indicating which keyword expects the\n data. For example, suppose ``f`` takes its data as ``arg2``:\n\n >>> (ds.pipe(h)\n ... .pipe(g, arg1=a)\n ... .pipe((f, 'arg2'), arg1=a, arg3=c)\n ... )\n\n See Also\n --------\n pandas.DataFrame.pipe\n \"\"\"\n if isinstance(func, tuple):\n func, target = func\n if target in kwargs:\n raise ValueError('%s is both the pipe target and a keyword '\n 'argument' % target)\n kwargs[target] = self\n return func(*args, **kwargs)\n else:\n return func(self, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef groupby(self, group, squeeze: bool = True):\n return self._groupby_cls(self, group, squeeze=squeeze)", "response": "Returns a GroupBy object for performing grouped operations on the a\n array."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef groupby_bins(self, group, bins, right: bool = True, labels=None,\n precision: int = 3, include_lowest: bool = False,\n squeeze: bool = True):\n \"\"\"Returns a GroupBy object for performing grouped operations.\n\n Rather than using all unique values of `group`, the values are discretized\n first by applying `pandas.cut` [1]_ to `group`.\n\n Parameters\n ----------\n group : str, DataArray or IndexVariable\n Array whose binned values should be used to group this array. If a\n string, must be the name of a variable contained in this dataset.\n bins : int or array of scalars\n If bins is an int, it defines the number of equal-width bins in the\n range of x. However, in this case, the range of x is extended by .1%\n on each side to include the min or max values of x. If bins is a\n sequence it defines the bin edges allowing for non-uniform bin\n width. No extension of the range of x is done in this case.\n right : boolean, optional\n Indicates whether the bins include the rightmost edge or not. If\n right == True (the default), then the bins [1,2,3,4] indicate\n (1,2], (2,3], (3,4].\n labels : array or boolean, default None\n Used as labels for the resulting bins. Must be of the same length as\n the resulting bins. If False, string bin labels are assigned by\n `pandas.cut`.\n precision : int\n The precision at which to store and display the bins labels.\n include_lowest : bool\n Whether the first interval should be left-inclusive or not.\n squeeze : boolean, optional\n If \"group\" is a dimension of any arrays in this dataset, `squeeze`\n controls whether the subarrays have a dimension of length 1 along\n that dimension or if the dimension is squeezed out.\n\n Returns\n -------\n grouped : GroupBy\n A `GroupBy` object patterned after `pandas.GroupBy` that can be\n iterated over in the form of `(unique_value, grouped_array)` pairs.\n The name of the group has the added suffix `_bins` in order to\n distinguish it from the original variable.\n\n References\n ----------\n .. [1] http://pandas.pydata.org/pandas-docs/stable/generated/pandas.cut.html\n \"\"\" # noqa\n return self._groupby_cls(self, group, squeeze=squeeze, bins=bins,\n cut_kwargs={'right': right, 'labels': labels,\n 'precision': precision,\n 'include_lowest': include_lowest})", "response": "Returns a GroupBy object for performing grouped operations on the avec of unique values of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the avec of the values."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a rolling window object for the given date and time.", "response": "def rolling(self, dim: Optional[Mapping[Hashable, int]] = None,\n min_periods: Optional[int] = None, center: bool = False,\n **dim_kwargs: int):\n \"\"\"\n Rolling window object.\n\n Parameters\n ----------\n dim: dict, optional\n Mapping from the dimension name to create the rolling iterator\n along (e.g. `time`) to its moving window size.\n min_periods : int, default None\n Minimum number of observations in window required to have a value\n (otherwise result is NA). The default, None, is equivalent to\n setting min_periods equal to the size of the window.\n center : boolean, default False\n Set the labels at the center of the window.\n **dim_kwargs : optional\n The keyword arguments form of ``dim``.\n One of dim or dim_kwargs must be provided.\n\n Returns\n -------\n Rolling object (core.rolling.DataArrayRolling for DataArray,\n core.rolling.DatasetRolling for Dataset.)\n\n Examples\n --------\n Create rolling seasonal average of monthly data e.g. DJF, JFM, ..., SON:\n\n >>> da = xr.DataArray(np.linspace(0, 11, num=12),\n ... coords=[pd.date_range('15/12/1999',\n ... periods=12, freq=pd.DateOffset(months=1))],\n ... dims='time')\n >>> da\n \n array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 2000-01-15 2000-02-15 ...\n >>> da.rolling(time=3, center=True).mean()\n \n array([nan, 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., nan])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 2000-01-15 2000-02-15 ...\n\n Remove the NaNs using ``dropna()``:\n\n >>> da.rolling(time=3, center=True).mean().dropna('time')\n \n array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.])\n Coordinates:\n * time (time) datetime64[ns] 2000-01-15 2000-02-15 2000-03-15 ...\n\n See Also\n --------\n core.rolling.DataArrayRolling\n core.rolling.DatasetRolling\n \"\"\" # noqa\n dim = either_dict_or_kwargs(dim, dim_kwargs, 'rolling')\n return self._rolling_cls(self, dim, min_periods=min_periods,\n center=center)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef coarsen(self, dim: Optional[Mapping[Hashable, int]] = None,\n boundary: str = 'exact',\n side: Union[str, Mapping[Hashable, str]] = 'left',\n coord_func: str = 'mean',\n **dim_kwargs: int):\n \"\"\"\n Coarsen object.\n\n Parameters\n ----------\n dim: dict, optional\n Mapping from the dimension name to the window size.\n dim : str\n Name of the dimension to create the rolling iterator\n along (e.g., `time`).\n window : int\n Size of the moving window.\n boundary : 'exact' | 'trim' | 'pad'\n If 'exact', a ValueError will be raised if dimension size is not a\n multiple of the window size. If 'trim', the excess entries are\n dropped. If 'pad', NA will be padded.\n side : 'left' or 'right' or mapping from dimension to 'left' or 'right'\n coord_func: function (name) that is applied to the coordintes,\n or a mapping from coordinate name to function (name).\n\n Returns\n -------\n Coarsen object (core.rolling.DataArrayCoarsen for DataArray,\n core.rolling.DatasetCoarsen for Dataset.)\n\n Examples\n --------\n Coarsen the long time series by averaging over every four days.\n\n >>> da = xr.DataArray(np.linspace(0, 364, num=364),\n ... dims='time',\n ... coords={'time': pd.date_range(\n ... '15/12/1999', periods=364)})\n >>> da\n \n array([ 0. , 1.002755, 2.00551 , ..., 361.99449 , 362.997245,\n 364. ])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 1999-12-16 ... 2000-12-12\n >>>\n >>> da.coarsen(time=3, boundary='trim').mean()\n \n array([ 1.002755, 4.011019, 7.019284, ..., 358.986226,\n 361.99449 ])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-16 1999-12-19 ... 2000-12-10\n >>>\n\n See Also\n --------\n core.rolling.DataArrayCoarsen\n core.rolling.DatasetCoarsen\n \"\"\"\n dim = either_dict_or_kwargs(dim, dim_kwargs, 'coarsen')\n return self._coarsen_cls(\n self, dim, boundary=boundary, side=side,\n coord_func=coord_func)", "response": "Returns a new object that coarsen the long time series."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a Resample object for performing downsampling and upsampling operations.", "response": "def resample(self, indexer: Optional[Mapping[Hashable, str]] = None,\n skipna=None, closed: Optional[str] = None,\n label: Optional[str] = None,\n base: int = 0, keep_attrs: Optional[bool] = None,\n loffset=None,\n **indexer_kwargs: str):\n \"\"\"Returns a Resample object for performing resampling operations.\n\n Handles both downsampling and upsampling. If any intervals contain no\n values from the original object, they will be given the value ``NaN``.\n\n Parameters\n ----------\n indexer : {dim: freq}, optional\n Mapping from the dimension name to resample frequency.\n skipna : bool, optional\n Whether to skip missing values when aggregating in downsampling.\n closed : 'left' or 'right', optional\n Side of each interval to treat as closed.\n label : 'left or 'right', optional\n Side of each interval to use for labeling.\n base : int, optional\n For frequencies that evenly subdivide 1 day, the \"origin\" of the\n aggregated intervals. For example, for '24H' frequency, base could\n range from 0 through 23.\n loffset : timedelta or str, optional\n Offset used to adjust the resampled time labels. Some pandas date\n offset strings are supported.\n keep_attrs : bool, optional\n If True, the object's attributes (`attrs`) will be copied from\n the original object to the new one. If False (default), the new\n object will be returned without attributes.\n **indexer_kwargs : {dim: freq}\n The keyword arguments form of ``indexer``.\n One of indexer or indexer_kwargs must be provided.\n\n Returns\n -------\n resampled : same type as caller\n This object resampled.\n\n Examples\n --------\n Downsample monthly time-series data to seasonal data:\n\n >>> da = xr.DataArray(np.linspace(0, 11, num=12),\n ... coords=[pd.date_range('15/12/1999',\n ... periods=12, freq=pd.DateOffset(months=1))],\n ... dims='time')\n >>> da\n \n array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 2000-01-15 2000-02-15 ...\n >>> da.resample(time=\"QS-DEC\").mean()\n \n array([ 1., 4., 7., 10.])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-01 2000-03-01 2000-06-01 2000-09-01\n\n Upsample monthly time-series data to daily data:\n\n >>> da.resample(time='1D').interpolate('linear')\n \n array([ 0. , 0.032258, 0.064516, ..., 10.935484, 10.967742, 11. ])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 1999-12-16 1999-12-17 ...\n\n Limit scope of upsampling method\n >>> da.resample(time='1D').nearest(tolerance='1D')\n \n array([ 0., 0., nan, ..., nan, 11., 11.])\n Coordinates:\n * time (time) datetime64[ns] 1999-12-15 1999-12-16 ... 2000-11-15\n\n References\n ----------\n\n .. [1] http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases\n \"\"\" # noqa\n # TODO support non-string indexer after removing the old API.\n\n from .dataarray import DataArray\n from .resample import RESAMPLE_DIM\n from ..coding.cftimeindex import CFTimeIndex\n\n if keep_attrs is None:\n keep_attrs = _get_keep_attrs(default=False)\n\n # note: the second argument (now 'skipna') use to be 'dim'\n if ((skipna is not None and not isinstance(skipna, bool))\n or ('how' in indexer_kwargs and 'how' not in self.dims)\n or ('dim' in indexer_kwargs and 'dim' not in self.dims)):\n raise TypeError(\n 'resample() no longer supports the `how` or '\n '`dim` arguments. Instead call methods on resample '\n \"objects, e.g., data.resample(time='1D').mean()\")\n\n indexer = either_dict_or_kwargs(indexer, indexer_kwargs, 'resample')\n if len(indexer) != 1:\n raise ValueError(\n \"Resampling only supported along single dimensions.\"\n )\n dim, freq = next(iter(indexer.items()))\n\n dim_name = dim\n dim_coord = self[dim]\n\n if isinstance(self.indexes[dim_name], CFTimeIndex):\n from .resample_cftime import CFTimeGrouper\n grouper = CFTimeGrouper(freq, closed, label, base, loffset)\n else:\n # TODO: to_offset() call required for pandas==0.19.2\n grouper = pd.Grouper(freq=freq, closed=closed, label=label,\n base=base,\n loffset=pd.tseries.frequencies.to_offset(\n loffset))\n group = DataArray(dim_coord, coords=dim_coord.coords,\n dims=dim_coord.dims, name=RESAMPLE_DIM)\n resampler = self._resample_cls(self, group=group, dim=dim_name,\n grouper=grouper,\n resample_dim=RESAMPLE_DIM)\n\n return resampler"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef where(self, cond, other=dtypes.NA, drop: bool = False):\n from .alignment import align\n from .dataarray import DataArray\n from .dataset import Dataset\n\n if drop:\n if other is not dtypes.NA:\n raise ValueError('cannot set `other` if drop=True')\n\n if not isinstance(cond, (Dataset, DataArray)):\n raise TypeError(\"cond argument is %r but must be a %r or %r\" %\n (cond, Dataset, DataArray))\n\n # align so we can use integer indexing\n self, cond = align(self, cond)\n\n # get cond with the minimal size needed for the Dataset\n if isinstance(cond, Dataset):\n clipcond = cond.to_array().any('variable')\n else:\n clipcond = cond\n\n # clip the data corresponding to coordinate dims that are not used\n nonzeros = zip(clipcond.dims, np.nonzero(clipcond.values))\n indexers = {k: np.unique(v) for k, v in nonzeros}\n\n self = self.isel(**indexers)\n cond = cond.isel(**indexers)\n\n return ops.where_method(self, cond, other)", "response": "Filter elements from this object according to a condition."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncloses any files linked to this object.", "response": "def close(self: Any) -> None:\n \"\"\"Close any files linked to this object\n \"\"\"\n if self._file_obj is not None:\n self._file_obj.close()\n self._file_obj = None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef isin(self, test_elements):\n from .computation import apply_ufunc\n from .dataset import Dataset\n from .dataarray import DataArray\n from .variable import Variable\n\n if isinstance(test_elements, Dataset):\n raise TypeError(\n 'isin() argument must be convertible to an array: {}'\n .format(test_elements))\n elif isinstance(test_elements, (Variable, DataArray)):\n # need to explicitly pull out data to support dask arrays as the\n # second argument\n test_elements = test_elements.data\n\n return apply_ufunc(\n duck_array_ops.isin,\n self,\n kwargs=dict(test_elements=test_elements),\n dask='allowed',\n )", "response": "Tests each value in the array for whether it is in test elements."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomparing two docker versions", "response": "def compare_version(v1, v2):\n \"\"\"Compare docker versions\n\n >>> v1 = '1.9'\n >>> v2 = '1.10'\n >>> compare_version(v1, v2)\n 1\n >>> compare_version(v2, v1)\n -1\n >>> compare_version(v2, v2)\n 0\n \"\"\"\n s1 = StrictVersion(v1)\n s2 = StrictVersion(v2)\n if s1 == s2:\n return 0\n elif s1 > s2:\n return -1\n else:\n return 1"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef datetime_to_timestamp(dt):\n delta = dt - datetime.utcfromtimestamp(0)\n return delta.seconds + delta.days * 24 * 3600", "response": "Convert a UTC datetime to a Unix timestamp"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading a line - separated environment file.", "response": "def parse_env_file(env_file):\n \"\"\"\n Reads a line-separated environment file.\n The format of each line should be \"key=value\".\n \"\"\"\n environment = {}\n\n with open(env_file, 'r') as f:\n for line in f:\n\n if line[0] == '#':\n continue\n\n line = line.strip()\n if not line:\n continue\n\n parse_line = line.split('=', 1)\n if len(parse_line) == 2:\n k, v = parse_line\n environment[k] = v\n else:\n raise errors.DockerException(\n 'Invalid line in environment file {0}:\\n{1}'.format(\n env_file, line))\n\n return environment"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconfigures a plugin. Args: name (string): The name of the plugin. The ``:latest`` tag is optional, and is the default if omitted. options (dict): A key-value mapping of options Returns: ``True`` if successful", "response": "def configure_plugin(self, name, options):\n \"\"\"\n Configure a plugin.\n\n Args:\n name (string): The name of the plugin. The ``:latest`` tag is\n optional, and is the default if omitted.\n options (dict): A key-value mapping of options\n\n Returns:\n ``True`` if successful\n \"\"\"\n url = self._url('/plugins/{0}/set', name)\n data = options\n if isinstance(data, dict):\n data = ['{0}={1}'.format(k, v) for k, v in six.iteritems(data)]\n res = self._post_json(url, data=data)\n self._raise_for_status(res)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new plugin.", "response": "def create_plugin(self, name, plugin_data_dir, gzip=False):\n \"\"\"\n Create a new plugin.\n\n Args:\n name (string): The name of the plugin. The ``:latest`` tag is\n optional, and is the default if omitted.\n plugin_data_dir (string): Path to the plugin data directory.\n Plugin data directory must contain the ``config.json``\n manifest file and the ``rootfs`` directory.\n gzip (bool): Compress the context using gzip. Default: False\n\n Returns:\n ``True`` if successful\n \"\"\"\n url = self._url('/plugins/create')\n\n with utils.create_archive(\n root=plugin_data_dir, gzip=gzip,\n files=set(utils.build.walk(plugin_data_dir, []))\n ) as archv:\n res = self._post(url, params={'name': name}, data=archv)\n self._raise_for_status(res)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef disable_plugin(self, name):\n url = self._url('/plugins/{0}/disable', name)\n res = self._post(url)\n self._raise_for_status(res)\n return True", "response": "Disable an installed plugin."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nenables an installed plugin.", "response": "def enable_plugin(self, name, timeout=0):\n \"\"\"\n Enable an installed plugin.\n\n Args:\n name (string): The name of the plugin. The ``:latest`` tag is\n optional, and is the default if omitted.\n timeout (int): Operation timeout (in seconds). Default: 0\n\n Returns:\n ``True`` if successful\n \"\"\"\n url = self._url('/plugins/{0}/enable', name)\n params = {'timeout': timeout}\n res = self._post(url, params=params)\n self._raise_for_status(res)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef inspect_plugin(self, name):\n url = self._url('/plugins/{0}/json', name)\n return self._result(self._get(url), True)", "response": "Retrieve plugin metadata.\n\n Args:\n name (string): The name of the plugin. The ``:latest`` tag is\n optional, and is the default if omitted.\n\n Returns:\n A dict containing plugin info"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plugins(self):\n url = self._url('/plugins')\n return self._result(self._get(url), True)", "response": "Retrieve a list of installed plugins."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nretrieve list of privileges granted to a remote plugin.", "response": "def plugin_privileges(self, name):\n \"\"\"\n Retrieve list of privileges to be granted to a plugin.\n\n Args:\n name (string): Name of the remote plugin to examine. The\n ``:latest`` tag is optional, and is the default if omitted.\n\n Returns:\n A list of dictionaries representing the plugin's\n permissions\n\n \"\"\"\n params = {\n 'remote': name,\n }\n\n headers = {}\n registry, repo_name = auth.resolve_repository_name(name)\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n\n url = self._url('/plugins/privileges')\n return self._result(\n self._get(url, params=params, headers=headers), True\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\npushing a new plugin to the registry.", "response": "def push_plugin(self, name):\n \"\"\"\n Push a plugin to the registry.\n\n Args:\n name (string): Name of the plugin to upload. The ``:latest``\n tag is optional, and is the default if omitted.\n\n Returns:\n ``True`` if successful\n \"\"\"\n url = self._url('/plugins/{0}/pull', name)\n\n headers = {}\n registry, repo_name = auth.resolve_repository_name(name)\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n res = self._post(url, headers=headers)\n self._raise_for_status(res)\n return self._stream_helper(res, decode=True)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove an installed plugin.", "response": "def remove_plugin(self, name, force=False):\n \"\"\"\n Remove an installed plugin.\n\n Args:\n name (string): Name of the plugin to remove. The ``:latest``\n tag is optional, and is the default if omitted.\n force (bool): Disable the plugin before removing. This may\n result in issues if the plugin is in use by a container.\n\n Returns:\n ``True`` if successful\n \"\"\"\n url = self._url('/plugins/{0}', name)\n res = self._delete(url, params={'force': force})\n self._raise_for_status(res)\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef upgrade_plugin(self, name, remote, privileges):\n\n url = self._url('/plugins/{0}/upgrade', name)\n params = {\n 'remote': remote,\n }\n\n headers = {}\n registry, repo_name = auth.resolve_repository_name(remote)\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n response = self._post_json(\n url, params=params, headers=headers, data=privileges,\n stream=True\n )\n self._raise_for_status(response)\n return self._stream_helper(response, decode=True)", "response": "Upgrade an installed plugin."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_env(cls, **kwargs):\n timeout = kwargs.pop('timeout', DEFAULT_TIMEOUT_SECONDS)\n version = kwargs.pop('version', None)\n return cls(\n timeout=timeout, version=version, **kwargs_from_env(**kwargs)\n )", "response": "Returns a new instance of the class from environment variables."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef close(self):\n\n if not self._response.raw.closed:\n # find the underlying socket object\n # based on api.client._get_raw_response_socket\n\n sock_fp = self._response.raw._fp.fp\n\n if hasattr(sock_fp, 'raw'):\n sock_raw = sock_fp.raw\n\n if hasattr(sock_raw, 'sock'):\n sock = sock_raw.sock\n\n elif hasattr(sock_raw, '_sock'):\n sock = sock_raw._sock\n\n elif hasattr(sock_fp, 'channel'):\n # We're working with a paramiko (SSH) channel, which doesn't\n # support cancelable streams with the current implementation\n raise DockerException(\n 'Cancellable streams not supported for the SSH protocol'\n )\n else:\n sock = sock_fp._sock\n\n if hasattr(urllib3.contrib, 'pyopenssl') and isinstance(\n sock, urllib3.contrib.pyopenssl.WrappedSocket):\n sock = sock.socket\n\n sock.shutdown(socket.SHUT_RDWR)\n sock.close()", "response": "Closes the event streaming."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread at most n bytes from socket returning a tuple of the n - byte integers.", "response": "def read(socket, n=4096):\n \"\"\"\n Reads at most n bytes from socket\n \"\"\"\n\n recoverable_errors = (errno.EINTR, errno.EDEADLK, errno.EWOULDBLOCK)\n\n if six.PY3 and not isinstance(socket, NpipeSocket):\n select.select([socket], [], [])\n\n try:\n if hasattr(socket, 'recv'):\n return socket.recv(n)\n if six.PY3 and isinstance(socket, getattr(pysocket, 'SocketIO')):\n return socket.read(n)\n return os.read(socket.fileno(), n)\n except EnvironmentError as e:\n if e.errno not in recoverable_errors:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads exactly n bytes from socket returning a binary string.", "response": "def read_exactly(socket, n):\n \"\"\"\n Reads exactly n bytes from socket\n Raises SocketError if there isn't enough data\n \"\"\"\n data = six.binary_type()\n while len(data) < n:\n next_data = read(socket, n - len(data))\n if not next_data:\n raise SocketError(\"Unexpected EOF\")\n data += next_data\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef next_frame_header(socket):\n try:\n data = read_exactly(socket, 8)\n except SocketError:\n return (-1, -1)\n\n stream, actual = struct.unpack('>BxxxL', data)\n return (stream, actual)", "response": "Reads the next frame of data from the socket and returns the stream and size of the next frame."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef frames_iter(socket, tty):\n if tty:\n return ((STDOUT, frame) for frame in frames_iter_tty(socket))\n else:\n return frames_iter_no_tty(socket)", "response": "Return an iterator of frames read from socket."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef frames_iter_no_tty(socket):\n while True:\n (stream, n) = next_frame_header(socket)\n if n < 0:\n break\n while n > 0:\n result = read(socket, n)\n if result is None:\n continue\n data_length = len(result)\n if data_length == 0:\n # We have reached EOF\n return\n n -= data_length\n yield (stream, result)", "response": "Returns a generator of data read from the socket when tty setting is\n is not enabled."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\niterating through frames read from the socket and return the result. Args: demux (bool): If False, stdout and stderr are multiplexed, and the result is the concatenation of all the frames. If True, the streams are demultiplexed, and the result is a 2-tuple where each item is the concatenation of frames belonging to the same stream.", "response": "def consume_socket_output(frames, demux=False):\n \"\"\"\n Iterate through frames read from the socket and return the result.\n\n Args:\n\n demux (bool):\n If False, stdout and stderr are multiplexed, and the result is the\n concatenation of all the frames. If True, the streams are\n demultiplexed, and the result is a 2-tuple where each item is the\n concatenation of frames belonging to the same stream.\n \"\"\"\n if demux is False:\n # If the streams are multiplexed, the generator returns strings, that\n # we just need to concatenate.\n return six.binary_type().join(frames)\n\n # If the streams are demultiplexed, the generator yields tuples\n # (stdout, stderr)\n out = [None, None]\n for frame in frames:\n # It is guaranteed that for each frame, one and only one stream\n # is not None.\n assert frame != (None, None)\n if frame[0] is not None:\n if out[0] is None:\n out[0] = frame[0]\n else:\n out[0] += frame[0]\n else:\n if out[1] is None:\n out[1] = frame[1]\n else:\n out[1] += frame[1]\n return tuple(out)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding a new instance of a new node.", "response": "def build(self, path=None, tag=None, quiet=False, fileobj=None,\n nocache=False, rm=False, timeout=None,\n custom_context=False, encoding=None, pull=False,\n forcerm=False, dockerfile=None, container_limits=None,\n decode=False, buildargs=None, gzip=False, shmsize=None,\n labels=None, cache_from=None, target=None, network_mode=None,\n squash=None, extra_hosts=None, platform=None, isolation=None,\n use_config_proxy=False):\n \"\"\"\n Similar to the ``docker build`` command. Either ``path`` or ``fileobj``\n needs to be set. ``path`` can be a local path (to a directory\n containing a Dockerfile) or a remote URL. ``fileobj`` must be a\n readable file-like object to a Dockerfile.\n\n If you have a tar file for the Docker build context (including a\n Dockerfile) already, pass a readable file-like object to ``fileobj``\n and also pass ``custom_context=True``. If the stream is compressed\n also, set ``encoding`` to the correct value (e.g ``gzip``).\n\n Example:\n >>> from io import BytesIO\n >>> from docker import APIClient\n >>> dockerfile = '''\n ... # Shared Volume\n ... FROM busybox:buildroot-2014.02\n ... VOLUME /data\n ... CMD [\"/bin/sh\"]\n ... '''\n >>> f = BytesIO(dockerfile.encode('utf-8'))\n >>> cli = APIClient(base_url='tcp://127.0.0.1:2375')\n >>> response = [line for line in cli.build(\n ... fileobj=f, rm=True, tag='yourname/volume'\n ... )]\n >>> response\n ['{\"stream\":\" ---\\\\u003e a9eb17255234\\\\n\"}',\n '{\"stream\":\"Step 1 : VOLUME /data\\\\n\"}',\n '{\"stream\":\" ---\\\\u003e Running in abdc1e6896c6\\\\n\"}',\n '{\"stream\":\" ---\\\\u003e 713bca62012e\\\\n\"}',\n '{\"stream\":\"Removing intermediate container abdc1e6896c6\\\\n\"}',\n '{\"stream\":\"Step 2 : CMD [\\\\\"/bin/sh\\\\\"]\\\\n\"}',\n '{\"stream\":\" ---\\\\u003e Running in dba30f2a1a7e\\\\n\"}',\n '{\"stream\":\" ---\\\\u003e 032b8b2855fc\\\\n\"}',\n '{\"stream\":\"Removing intermediate container dba30f2a1a7e\\\\n\"}',\n '{\"stream\":\"Successfully built 032b8b2855fc\\\\n\"}']\n\n Args:\n path (str): Path to the directory containing the Dockerfile\n fileobj: A file object to use as the Dockerfile. (Or a file-like\n object)\n tag (str): A tag to add to the final image\n quiet (bool): Whether to return the status\n nocache (bool): Don't use the cache when set to ``True``\n rm (bool): Remove intermediate containers. The ``docker build``\n command now defaults to ``--rm=true``, but we have kept the old\n default of `False` to preserve backward compatibility\n timeout (int): HTTP timeout\n custom_context (bool): Optional if using ``fileobj``\n encoding (str): The encoding for a stream. Set to ``gzip`` for\n compressing\n pull (bool): Downloads any updates to the FROM image in Dockerfiles\n forcerm (bool): Always remove intermediate containers, even after\n unsuccessful builds\n dockerfile (str): path within the build context to the Dockerfile\n buildargs (dict): A dictionary of build arguments\n container_limits (dict): A dictionary of limits applied to each\n container created by the build process. Valid keys:\n\n - memory (int): set memory limit for build\n - memswap (int): Total memory (memory + swap), -1 to disable\n swap\n - cpushares (int): CPU shares (relative weight)\n - cpusetcpus (str): CPUs in which to allow execution, e.g.,\n ``\"0-3\"``, ``\"0,1\"``\n decode (bool): If set to ``True``, the returned stream will be\n decoded into dicts on the fly. Default ``False``\n shmsize (int): Size of `/dev/shm` in bytes. The size must be\n greater than 0. If omitted the system uses 64MB\n labels (dict): A dictionary of labels to set on the image\n cache_from (:py:class:`list`): A list of images used for build\n cache resolution\n target (str): Name of the build-stage to build in a multi-stage\n Dockerfile\n network_mode (str): networking mode for the run commands during\n build\n squash (bool): Squash the resulting images layers into a\n single layer.\n extra_hosts (dict): Extra hosts to add to /etc/hosts in building\n containers, as a mapping of hostname to IP address.\n platform (str): Platform in the format ``os[/arch[/variant]]``\n isolation (str): Isolation technology used during build.\n Default: `None`.\n use_config_proxy (bool): If ``True``, and if the docker client\n configuration file (``~/.docker/config.json`` by default)\n contains a proxy configuration, the corresponding environment\n variables will be set in the container being built.\n\n Returns:\n A generator for the build output.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n ``TypeError``\n If neither ``path`` nor ``fileobj`` is specified.\n \"\"\"\n remote = context = None\n headers = {}\n container_limits = container_limits or {}\n buildargs = buildargs or {}\n if path is None and fileobj is None:\n raise TypeError(\"Either path or fileobj needs to be provided.\")\n if gzip and encoding is not None:\n raise errors.DockerException(\n 'Can not use custom encoding if gzip is enabled'\n )\n\n for key in container_limits.keys():\n if key not in constants.CONTAINER_LIMITS_KEYS:\n raise errors.DockerException(\n 'Invalid container_limits key {0}'.format(key)\n )\n\n if custom_context:\n if not fileobj:\n raise TypeError(\"You must specify fileobj with custom_context\")\n context = fileobj\n elif fileobj is not None:\n context = utils.mkbuildcontext(fileobj)\n elif path.startswith(('http://', 'https://',\n 'git://', 'github.com/', 'git@')):\n remote = path\n elif not os.path.isdir(path):\n raise TypeError(\"You must specify a directory to build in path\")\n else:\n dockerignore = os.path.join(path, '.dockerignore')\n exclude = None\n if os.path.exists(dockerignore):\n with open(dockerignore, 'r') as f:\n exclude = list(filter(\n lambda x: x != '' and x[0] != '#',\n [l.strip() for l in f.read().splitlines()]\n ))\n dockerfile = process_dockerfile(dockerfile, path)\n context = utils.tar(\n path, exclude=exclude, dockerfile=dockerfile, gzip=gzip\n )\n encoding = 'gzip' if gzip else encoding\n\n u = self._url('/build')\n params = {\n 't': tag,\n 'remote': remote,\n 'q': quiet,\n 'nocache': nocache,\n 'rm': rm,\n 'forcerm': forcerm,\n 'pull': pull,\n 'dockerfile': dockerfile,\n }\n params.update(container_limits)\n\n if use_config_proxy:\n proxy_args = self._proxy_configs.get_environment()\n for k, v in proxy_args.items():\n buildargs.setdefault(k, v)\n if buildargs:\n params.update({'buildargs': json.dumps(buildargs)})\n\n if shmsize:\n if utils.version_gte(self._version, '1.22'):\n params.update({'shmsize': shmsize})\n else:\n raise errors.InvalidVersion(\n 'shmsize was only introduced in API version 1.22'\n )\n\n if labels:\n if utils.version_gte(self._version, '1.23'):\n params.update({'labels': json.dumps(labels)})\n else:\n raise errors.InvalidVersion(\n 'labels was only introduced in API version 1.23'\n )\n\n if cache_from:\n if utils.version_gte(self._version, '1.25'):\n params.update({'cachefrom': json.dumps(cache_from)})\n else:\n raise errors.InvalidVersion(\n 'cache_from was only introduced in API version 1.25'\n )\n\n if target:\n if utils.version_gte(self._version, '1.29'):\n params.update({'target': target})\n else:\n raise errors.InvalidVersion(\n 'target was only introduced in API version 1.29'\n )\n\n if network_mode:\n if utils.version_gte(self._version, '1.25'):\n params.update({'networkmode': network_mode})\n else:\n raise errors.InvalidVersion(\n 'network_mode was only introduced in API version 1.25'\n )\n\n if squash:\n if utils.version_gte(self._version, '1.25'):\n params.update({'squash': squash})\n else:\n raise errors.InvalidVersion(\n 'squash was only introduced in API version 1.25'\n )\n\n if extra_hosts is not None:\n if utils.version_lt(self._version, '1.27'):\n raise errors.InvalidVersion(\n 'extra_hosts was only introduced in API version 1.27'\n )\n\n if isinstance(extra_hosts, dict):\n extra_hosts = utils.format_extra_hosts(extra_hosts)\n params.update({'extrahosts': extra_hosts})\n\n if platform is not None:\n if utils.version_lt(self._version, '1.32'):\n raise errors.InvalidVersion(\n 'platform was only introduced in API version 1.32'\n )\n params['platform'] = platform\n\n if isolation is not None:\n if utils.version_lt(self._version, '1.24'):\n raise errors.InvalidVersion(\n 'isolation was only introduced in API version 1.24'\n )\n params['isolation'] = isolation\n\n if context is not None:\n headers = {'Content-Type': 'application/tar'}\n if encoding:\n headers['Content-Encoding'] = encoding\n\n self._set_auth_headers(headers)\n\n response = self._post(\n u,\n data=context,\n params=params,\n headers=headers,\n stream=True,\n timeout=timeout,\n )\n\n if context is not None and not custom_context:\n context.close()\n\n return self._stream_helper(response, decode=decode)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes the builder cache", "response": "def prune_builds(self):\n \"\"\"\n Delete the builder cache\n\n Returns:\n (dict): A dictionary containing information about the operation's\n result. The ``SpaceReclaimed`` key indicates the amount of\n bytes of disk space reclaimed.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url(\"/build/prune\")\n return self._result(self._post(url), True)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_swarm_spec(self, *args, **kwargs):\n ext_ca = kwargs.pop('external_ca', None)\n if ext_ca:\n kwargs['external_cas'] = [ext_ca]\n return types.SwarmSpec(self._version, *args, **kwargs)", "response": "Creates a SwarmSpec instance that can be used as the swarm_spec argument."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes a new Swarm using the current connected engine as the first Swarm node.", "response": "def init_swarm(self, advertise_addr=None, listen_addr='0.0.0.0:2377',\n force_new_cluster=False, swarm_spec=None,\n default_addr_pool=None, subnet_size=None):\n \"\"\"\n Initialize a new Swarm using the current connected engine as the first\n node.\n\n Args:\n advertise_addr (string): Externally reachable address advertised\n to other nodes. This can either be an address/port combination\n in the form ``192.168.1.1:4567``, or an interface followed by a\n port number, like ``eth0:4567``. If the port number is omitted,\n the port number from the listen address is used. If\n ``advertise_addr`` is not specified, it will be automatically\n detected when possible. Default: None\n listen_addr (string): Listen address used for inter-manager\n communication, as well as determining the networking interface\n used for the VXLAN Tunnel Endpoint (VTEP). This can either be\n an address/port combination in the form ``192.168.1.1:4567``,\n or an interface followed by a port number, like ``eth0:4567``.\n If the port number is omitted, the default swarm listening port\n is used. Default: '0.0.0.0:2377'\n force_new_cluster (bool): Force creating a new Swarm, even if\n already part of one. Default: False\n swarm_spec (dict): Configuration settings of the new Swarm. Use\n ``APIClient.create_swarm_spec`` to generate a valid\n configuration. Default: None\n default_addr_pool (list of strings): Default Address Pool specifies\n default subnet pools for global scope networks. Each pool\n should be specified as a CIDR block, like '10.0.0.0/8'.\n Default: None\n subnet_size (int): SubnetSize specifies the subnet size of the\n networks created from the default subnet pool. Default: None\n\n Returns:\n ``True`` if successful.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n url = self._url('/swarm/init')\n if swarm_spec is not None and not isinstance(swarm_spec, dict):\n raise TypeError('swarm_spec must be a dictionary')\n\n if default_addr_pool is not None:\n if utils.version_lt(self._version, '1.39'):\n raise errors.InvalidVersion(\n 'Address pool is only available for API version >= 1.39'\n )\n # subnet_size becomes 0 if not set with default_addr_pool\n if subnet_size is None:\n subnet_size = DEFAULT_SWARM_SUBNET_SIZE\n\n if subnet_size is not None:\n if utils.version_lt(self._version, '1.39'):\n raise errors.InvalidVersion(\n 'Subnet size is only available for API version >= 1.39'\n )\n # subnet_size is ignored if set without default_addr_pool\n if default_addr_pool is None:\n default_addr_pool = DEFAULT_SWARM_ADDR_POOL\n\n data = {\n 'AdvertiseAddr': advertise_addr,\n 'ListenAddr': listen_addr,\n 'DefaultAddrPool': default_addr_pool,\n 'SubnetSize': subnet_size,\n 'ForceNewCluster': force_new_cluster,\n 'Spec': swarm_spec,\n }\n response = self._post_json(url, data=data)\n self._raise_for_status(response)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef inspect_swarm(self):\n url = self._url('/swarm')\n return self._result(self._get(url), True)", "response": "Retrieve low - level information about the current swarm."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nretrieve low - level information about a swarm node.", "response": "def inspect_node(self, node_id):\n \"\"\"\n Retrieve low-level information about a swarm node\n\n Args:\n node_id (string): ID of the node to be inspected.\n\n Returns:\n A dictionary containing data about this node.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url('/nodes/{0}', node_id)\n return self._result(self._get(url), True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes this Engine join a swarm that has already been created. Args: remote_addrs (:py:class:`list`): Addresses of one or more manager nodes already participating in the Swarm to join. join_token (string): Secret token for joining this Swarm. listen_addr (string): Listen address used for inter-manager communication if the node gets promoted to manager, as well as determining the networking interface used for the VXLAN Tunnel Endpoint (VTEP). Default: ``'0.0.0.0:2377`` advertise_addr (string): Externally reachable address advertised to other nodes. This can either be an address/port combination in the form ``192.168.1.1:4567``, or an interface followed by a port number, like ``eth0:4567``. If the port number is omitted, the port number from the listen address is used. If AdvertiseAddr is not specified, it will be automatically detected when possible. Default: ``None`` Returns: ``True`` if the request went through. Raises: :py:class:`docker.errors.APIError` If the server returns an error.", "response": "def join_swarm(self, remote_addrs, join_token, listen_addr='0.0.0.0:2377',\n advertise_addr=None):\n \"\"\"\n Make this Engine join a swarm that has already been created.\n\n Args:\n remote_addrs (:py:class:`list`): Addresses of one or more manager\n nodes already participating in the Swarm to join.\n join_token (string): Secret token for joining this Swarm.\n listen_addr (string): Listen address used for inter-manager\n communication if the node gets promoted to manager, as well as\n determining the networking interface used for the VXLAN Tunnel\n Endpoint (VTEP). Default: ``'0.0.0.0:2377``\n advertise_addr (string): Externally reachable address advertised\n to other nodes. This can either be an address/port combination\n in the form ``192.168.1.1:4567``, or an interface followed by a\n port number, like ``eth0:4567``. If the port number is omitted,\n the port number from the listen address is used. If\n AdvertiseAddr is not specified, it will be automatically\n detected when possible. Default: ``None``\n\n Returns:\n ``True`` if the request went through.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n data = {\n \"RemoteAddrs\": remote_addrs,\n \"ListenAddr\": listen_addr,\n \"JoinToken\": join_token,\n \"AdvertiseAddr\": advertise_addr,\n }\n url = self._url('/swarm/join')\n response = self._post_json(url, data=data)\n self._raise_for_status(response)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nleaves a swarm. Args: force (bool): Leave the swarm even if this node is a manager. Default: ``False`` Returns: ``True`` if the request went through. Raises: :py:class:`docker.errors.APIError` If the server returns an error.", "response": "def leave_swarm(self, force=False):\n \"\"\"\n Leave a swarm.\n\n Args:\n force (bool): Leave the swarm even if this node is a manager.\n Default: ``False``\n\n Returns:\n ``True`` if the request went through.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url('/swarm/leave')\n response = self._post(url, params={'force': force})\n # Ignore \"this node is not part of a swarm\" error\n if force and response.status_code == http_client.NOT_ACCEPTABLE:\n return True\n # FIXME: Temporary workaround for 1.13.0-rc bug\n # https://github.com/docker/docker/issues/29192\n if force and response.status_code == http_client.SERVICE_UNAVAILABLE:\n return True\n self._raise_for_status(response)\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_node(self, node_id, force=False):\n url = self._url('/nodes/{0}', node_id)\n params = {\n 'force': force\n }\n res = self._delete(url, params=params)\n self._raise_for_status(res)\n return True", "response": "Removes a node from the swarm."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nunlock a locked swarm.", "response": "def unlock_swarm(self, key):\n \"\"\"\n Unlock a locked swarm.\n\n Args:\n key (string): The unlock key as provided by\n :py:meth:`get_unlock_key`\n\n Raises:\n :py:class:`docker.errors.InvalidArgument`\n If the key argument is in an incompatible format\n\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Returns:\n `True` if the request was successful.\n\n Example:\n\n >>> key = client.get_unlock_key()\n >>> client.unlock_node(key)\n\n \"\"\"\n if isinstance(key, dict):\n if 'UnlockKey' not in key:\n raise errors.InvalidArgument('Invalid unlock key format')\n else:\n key = {'UnlockKey': key}\n\n url = self._url('/swarm/unlock')\n res = self._post_json(url, data=key)\n self._raise_for_status(res)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_node(self, node_id, version, node_spec=None):\n url = self._url('/nodes/{0}/update?version={1}', node_id, str(version))\n res = self._post_json(url, data=node_spec)\n self._raise_for_status(res)\n return True", "response": "Update the node s configuration with the new version number."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_swarm(self, version, swarm_spec=None, rotate_worker_token=False,\n rotate_manager_token=False):\n \"\"\"\n Update the Swarm's configuration\n\n Args:\n version (int): The version number of the swarm object being\n updated. This is required to avoid conflicting writes.\n swarm_spec (dict): Configuration settings to update. Use\n :py:meth:`~docker.api.swarm.SwarmApiMixin.create_swarm_spec` to\n generate a valid configuration. Default: ``None``.\n rotate_worker_token (bool): Rotate the worker join token. Default:\n ``False``.\n rotate_manager_token (bool): Rotate the manager join token.\n Default: ``False``.\n\n Returns:\n ``True`` if the request went through.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n url = self._url('/swarm/update')\n response = self._post_json(url, data=swarm_spec, params={\n 'rotateWorkerToken': rotate_worker_token,\n 'rotateManagerToken': rotate_manager_token,\n 'version': version\n })\n self._raise_for_status(response)\n return True", "response": "Update the Swarm s configuration with the new version number."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a secret. Args: secret_id (str): Secret ID. Returns: (:py:class:`Secret`): The secret. Raises: :py:class:`docker.errors.NotFound` If the secret does not exist. :py:class:`docker.errors.APIError` If the server returns an error.", "response": "def get(self, secret_id):\n \"\"\"\n Get a secret.\n\n Args:\n secret_id (str): Secret ID.\n\n Returns:\n (:py:class:`Secret`): The secret.\n\n Raises:\n :py:class:`docker.errors.NotFound`\n If the secret does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.prepare_model(self.client.api.inspect_secret(secret_id))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlisting secrets. Similar to the docker secret ls command.", "response": "def list(self, **kwargs):\n \"\"\"\n List secrets. Similar to the ``docker secret ls`` command.\n\n Args:\n filters (dict): Server-side list filtering options.\n\n Returns:\n (list of :py:class:`Secret`): The secrets.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.secrets(**kwargs)\n return [self.prepare_model(obj) for obj in resp]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlist the names and ids of all networks in the cluster.", "response": "def networks(self, names=None, ids=None, filters=None):\n \"\"\"\n List networks. Similar to the ``docker networks ls`` command.\n\n Args:\n names (:py:class:`list`): List of names to filter by\n ids (:py:class:`list`): List of ids to filter by\n filters (dict): Filters to be processed on the network list.\n Available filters:\n - ``driver=[]`` Matches a network's driver.\n - ``label=[]`` or ``label=[=]``.\n - ``type=[\"custom\"|\"builtin\"]`` Filters networks by type.\n\n Returns:\n (dict): List of network objects.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n if filters is None:\n filters = {}\n if names:\n filters['name'] = names\n if ids:\n filters['id'] = ids\n params = {'filters': utils.convert_filters(filters)}\n url = self._url(\"/networks\")\n res = self._get(url, params=params)\n return self._result(res, json=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a network using the given parameters.", "response": "def create_network(self, name, driver=None, options=None, ipam=None,\n check_duplicate=None, internal=False, labels=None,\n enable_ipv6=False, attachable=None, scope=None,\n ingress=None):\n \"\"\"\n Create a network. Similar to the ``docker network create``.\n\n Args:\n name (str): Name of the network\n driver (str): Name of the driver used to create the network\n options (dict): Driver options as a key-value dictionary\n ipam (IPAMConfig): Optional custom IP scheme for the network.\n check_duplicate (bool): Request daemon to check for networks with\n same name. Default: ``None``.\n internal (bool): Restrict external access to the network. Default\n ``False``.\n labels (dict): Map of labels to set on the network. Default\n ``None``.\n enable_ipv6 (bool): Enable IPv6 on the network. Default ``False``.\n attachable (bool): If enabled, and the network is in the global\n scope, non-service containers on worker nodes will be able to\n connect to the network.\n scope (str): Specify the network's scope (``local``, ``global`` or\n ``swarm``)\n ingress (bool): If set, create an ingress network which provides\n the routing-mesh in swarm mode.\n\n Returns:\n (dict): The created network reference object\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n A network using the bridge driver:\n\n >>> client.create_network(\"network1\", driver=\"bridge\")\n\n You can also create more advanced networks with custom IPAM\n configurations. For example, setting the subnet to\n ``192.168.52.0/24`` and gateway address to ``192.168.52.254``.\n\n .. code-block:: python\n\n >>> ipam_pool = docker.types.IPAMPool(\n subnet='192.168.52.0/24',\n gateway='192.168.52.254'\n )\n >>> ipam_config = docker.types.IPAMConfig(\n pool_configs=[ipam_pool]\n )\n >>> docker_client.create_network(\"network1\", driver=\"bridge\",\n ipam=ipam_config)\n \"\"\"\n if options is not None and not isinstance(options, dict):\n raise TypeError('options must be a dictionary')\n\n data = {\n 'Name': name,\n 'Driver': driver,\n 'Options': options,\n 'IPAM': ipam,\n 'CheckDuplicate': check_duplicate,\n }\n\n if labels is not None:\n if version_lt(self._version, '1.23'):\n raise InvalidVersion(\n 'network labels were introduced in API 1.23'\n )\n if not isinstance(labels, dict):\n raise TypeError('labels must be a dictionary')\n data[\"Labels\"] = labels\n\n if enable_ipv6:\n if version_lt(self._version, '1.23'):\n raise InvalidVersion(\n 'enable_ipv6 was introduced in API 1.23'\n )\n data['EnableIPv6'] = True\n\n if internal:\n if version_lt(self._version, '1.22'):\n raise InvalidVersion('Internal networks are not '\n 'supported in API version < 1.22')\n data['Internal'] = True\n\n if attachable is not None:\n if version_lt(self._version, '1.24'):\n raise InvalidVersion(\n 'attachable is not supported in API version < 1.24'\n )\n data['Attachable'] = attachable\n\n if ingress is not None:\n if version_lt(self._version, '1.29'):\n raise InvalidVersion(\n 'ingress is not supported in API version < 1.29'\n )\n\n data['Ingress'] = ingress\n\n if scope is not None:\n if version_lt(self._version, '1.30'):\n raise InvalidVersion(\n 'scope is not supported in API version < 1.30'\n )\n data['Scope'] = scope\n\n url = self._url(\"/networks/create\")\n res = self._post_json(url, data=data)\n return self._result(res, json=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_network(self, net_id):\n url = self._url(\"/networks/{0}\", net_id)\n res = self._delete(url)\n self._raise_for_status(res)", "response": "Remove a network from the cache."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef inspect_network(self, net_id, verbose=None, scope=None):\n params = {}\n if verbose is not None:\n if version_lt(self._version, '1.28'):\n raise InvalidVersion('verbose was introduced in API 1.28')\n params['verbose'] = verbose\n if scope is not None:\n if version_lt(self._version, '1.31'):\n raise InvalidVersion('scope was introduced in API 1.31')\n params['scope'] = scope\n\n url = self._url(\"/networks/{0}\", net_id)\n res = self._get(url, params=params)\n return self._result(res, json=True)", "response": "Get detailed information about a network."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconnect a container to a network.", "response": "def connect_container_to_network(self, container, net_id,\n ipv4_address=None, ipv6_address=None,\n aliases=None, links=None,\n link_local_ips=None):\n \"\"\"\n Connect a container to a network.\n\n Args:\n container (str): container-id/name to be connected to the network\n net_id (str): network id\n aliases (:py:class:`list`): A list of aliases for this endpoint.\n Names in that list can be used within the network to reach the\n container. Defaults to ``None``.\n links (:py:class:`list`): A list of links for this endpoint.\n Containers declared in this list will be linked to this\n container. Defaults to ``None``.\n ipv4_address (str): The IP address of this container on the\n network, using the IPv4 protocol. Defaults to ``None``.\n ipv6_address (str): The IP address of this container on the\n network, using the IPv6 protocol. Defaults to ``None``.\n link_local_ips (:py:class:`list`): A list of link-local\n (IPv4/IPv6) addresses.\n \"\"\"\n data = {\n \"Container\": container,\n \"EndpointConfig\": self.create_endpoint_config(\n aliases=aliases, links=links, ipv4_address=ipv4_address,\n ipv6_address=ipv6_address, link_local_ips=link_local_ips\n ),\n }\n\n url = self._url(\"/networks/{0}/connect\", net_id)\n res = self._post_json(url, data=data)\n self._raise_for_status(res)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef disconnect_container_from_network(self, container, net_id,\n force=False):\n \"\"\"\n Disconnect a container from a network.\n\n Args:\n container (str): container ID or name to be disconnected from the\n network\n net_id (str): network ID\n force (bool): Force the container to disconnect from a network.\n Default: ``False``\n \"\"\"\n data = {\"Container\": container}\n if force:\n if version_lt(self._version, '1.22'):\n raise InvalidVersion(\n 'Forced disconnect was introduced in API 1.22'\n )\n data['Force'] = force\n url = self._url(\"/networks/{0}/disconnect\", net_id)\n res = self._post_json(url, data=data)\n self._raise_for_status(res)", "response": "Disconnect a container from a network."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init(self, advertise_addr=None, listen_addr='0.0.0.0:2377',\n force_new_cluster=False, default_addr_pool=None,\n subnet_size=None, **kwargs):\n \"\"\"\n Initialize a new swarm on this Engine.\n\n Args:\n advertise_addr (str): Externally reachable address advertised to\n other nodes. This can either be an address/port combination in\n the form ``192.168.1.1:4567``, or an interface followed by a\n port number, like ``eth0:4567``. If the port number is omitted,\n the port number from the listen address is used.\n\n If not specified, it will be automatically detected when\n possible.\n listen_addr (str): Listen address used for inter-manager\n communication, as well as determining the networking interface\n used for the VXLAN Tunnel Endpoint (VTEP). This can either be\n an address/port combination in the form ``192.168.1.1:4567``,\n or an interface followed by a port number, like ``eth0:4567``.\n If the port number is omitted, the default swarm listening port\n is used. Default: ``0.0.0.0:2377``\n force_new_cluster (bool): Force creating a new Swarm, even if\n already part of one. Default: False\n default_addr_pool (list of str): Default Address Pool specifies\n default subnet pools for global scope networks. Each pool\n should be specified as a CIDR block, like '10.0.0.0/8'.\n Default: None\n subnet_size (int): SubnetSize specifies the subnet size of the\n networks created from the default subnet pool. Default: None\n task_history_retention_limit (int): Maximum number of tasks\n history stored.\n snapshot_interval (int): Number of logs entries between snapshot.\n keep_old_snapshots (int): Number of snapshots to keep beyond the\n current snapshot.\n log_entries_for_slow_followers (int): Number of log entries to\n keep around to sync up slow followers after a snapshot is\n created.\n heartbeat_tick (int): Amount of ticks (in seconds) between each\n heartbeat.\n election_tick (int): Amount of ticks (in seconds) needed without a\n leader to trigger a new election.\n dispatcher_heartbeat_period (int): The delay for an agent to send\n a heartbeat to the dispatcher.\n node_cert_expiry (int): Automatic expiry for nodes certificates.\n external_ca (dict): Configuration for forwarding signing requests\n to an external certificate authority. Use\n ``docker.types.SwarmExternalCA``.\n name (string): Swarm's name\n labels (dict): User-defined key/value metadata.\n signing_ca_cert (str): The desired signing CA certificate for all\n swarm node TLS leaf certificates, in PEM format.\n signing_ca_key (str): The desired signing CA key for all swarm\n node TLS leaf certificates, in PEM format.\n ca_force_rotate (int): An integer whose purpose is to force swarm\n to generate a new signing CA certificate and key, if none have\n been specified.\n autolock_managers (boolean): If set, generate a key and use it to\n lock data stored on the managers.\n log_driver (DriverConfig): The default log driver to use for tasks\n created in the orchestrator.\n\n Returns:\n ``True`` if the request went through.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> client.swarm.init(\n advertise_addr='eth0', listen_addr='0.0.0.0:5000',\n force_new_cluster=False, default_addr_pool=['10.20.0.0/16],\n subnet_size=24, snapshot_interval=5000,\n log_entries_for_slow_followers=1200\n )\n\n \"\"\"\n init_kwargs = {\n 'advertise_addr': advertise_addr,\n 'listen_addr': listen_addr,\n 'force_new_cluster': force_new_cluster,\n 'default_addr_pool': default_addr_pool,\n 'subnet_size': subnet_size\n }\n init_kwargs['swarm_spec'] = self.client.api.create_swarm_spec(**kwargs)\n self.client.api.init_swarm(**init_kwargs)\n self.reload()\n return True", "response": "Initialize a new Swarm on this Engine."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update(self, rotate_worker_token=False, rotate_manager_token=False,\n **kwargs):\n \"\"\"\n Update the swarm's configuration.\n\n It takes the same arguments as :py:meth:`init`, except\n ``advertise_addr``, ``listen_addr``, and ``force_new_cluster``. In\n addition, it takes these arguments:\n\n Args:\n rotate_worker_token (bool): Rotate the worker join token. Default:\n ``False``.\n rotate_manager_token (bool): Rotate the manager join token.\n Default: ``False``.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n \"\"\"\n # this seems to have to be set\n if kwargs.get('node_cert_expiry') is None:\n kwargs['node_cert_expiry'] = 7776000000000000\n\n return self.client.api.update_swarm(\n version=self.version,\n swarm_spec=self.client.api.create_swarm_spec(**kwargs),\n rotate_worker_token=rotate_worker_token,\n rotate_manager_token=rotate_manager_token\n )", "response": "Update the current configuration of the swarm."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_volume(self, name=None, driver=None, driver_opts=None,\n labels=None):\n \"\"\"\n Create and register a named volume\n\n Args:\n name (str): Name of the volume\n driver (str): Name of the driver used to create the volume\n driver_opts (dict): Driver options as a key-value dictionary\n labels (dict): Labels to set on the volume\n\n Returns:\n (dict): The created volume reference object\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> volume = cli.create_volume(name='foobar', driver='local',\n driver_opts={'foo': 'bar', 'baz': 'false'},\n labels={\"key\": \"value\"})\n >>> print(volume)\n {u'Driver': u'local',\n u'Labels': {u'key': u'value'},\n u'Mountpoint': u'/var/lib/docker/volumes/foobar/_data',\n u'Name': u'foobar',\n u'Scope': u'local'}\n\n \"\"\"\n url = self._url('/volumes/create')\n if driver_opts is not None and not isinstance(driver_opts, dict):\n raise TypeError('driver_opts must be a dictionary')\n\n data = {\n 'Name': name,\n 'Driver': driver,\n 'DriverOpts': driver_opts,\n }\n\n if labels is not None:\n if utils.compare_version('1.23', self._version) < 0:\n raise errors.InvalidVersion(\n 'volume labels were introduced in API 1.23'\n )\n if not isinstance(labels, dict):\n raise TypeError('labels must be a dictionary')\n data[\"Labels\"] = labels\n\n return self._result(self._post_json(url, data=data), True)", "response": "Create and register a named volume in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef inspect_volume(self, name):\n url = self._url('/volumes/{0}', name)\n return self._result(self._get(url), True)", "response": "Retrieve the information of a specific volume."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting unused volumes from the server.", "response": "def prune_volumes(self, filters=None):\n \"\"\"\n Delete unused volumes\n\n Args:\n filters (dict): Filters to process on the prune list.\n\n Returns:\n (dict): A dict containing a list of deleted volume names and\n the amount of disk space reclaimed in bytes.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n params = {}\n if filters:\n params['filters'] = utils.convert_filters(filters)\n url = self._url('/volumes/prune')\n return self._result(self._post(url, params=params), True)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_volume(self, name, force=False):\n params = {}\n if force:\n if utils.version_lt(self._version, '1.25'):\n raise errors.InvalidVersion(\n 'force removal was introduced in API 1.25'\n )\n params = {'force': force}\n\n url = self._url('/volumes/{0}', name, params=params)\n resp = self._delete(url)\n self._raise_for_status(resp)", "response": "Removes a specific image volume from the cache."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a new config in the neccesary way.", "response": "def create_config(self, name, data, labels=None):\n \"\"\"\n Create a config\n\n Args:\n name (string): Name of the config\n data (bytes): Config data to be stored\n labels (dict): A mapping of labels to assign to the config\n\n Returns (dict): ID of the newly created config\n \"\"\"\n if not isinstance(data, bytes):\n data = data.encode('utf-8')\n\n data = base64.b64encode(data)\n if six.PY3:\n data = data.decode('ascii')\n body = {\n 'Data': data,\n 'Name': name,\n 'Labels': labels\n }\n\n url = self._url('/configs/create')\n return self._result(\n self._post_json(url, data=body), True\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef inspect_config(self, id):\n url = self._url('/configs/{0}', id)\n return self._result(self._get(url), True)", "response": "Retrieve config metadata\n\n Args:\n id (string): Full ID of the config to inspect\n\n Returns (dict): A dictionary of metadata\n\n Raises:\n :py:class:`docker.errors.NotFound`\n if no config with that ID exists"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_config(self, id):\n url = self._url('/configs/{0}', id)\n res = self._delete(url)\n self._raise_for_status(res)\n return True", "response": "Removes a config from the cache"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef configs(self, filters=None):\n url = self._url('/configs')\n params = {}\n if filters:\n params['filters'] = utils.convert_filters(filters)\n return self._result(self._get(url, params=params), True)", "response": "Returns a list of all the available configuration items for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a secret in the secret store", "response": "def create_secret(self, name, data, labels=None, driver=None):\n \"\"\"\n Create a secret\n\n Args:\n name (string): Name of the secret\n data (bytes): Secret data to be stored\n labels (dict): A mapping of labels to assign to the secret\n driver (DriverConfig): A custom driver configuration. If\n unspecified, the default ``internal`` driver will be used\n\n Returns (dict): ID of the newly created secret\n \"\"\"\n if not isinstance(data, bytes):\n data = data.encode('utf-8')\n\n data = base64.b64encode(data)\n if six.PY3:\n data = data.decode('ascii')\n body = {\n 'Data': data,\n 'Name': name,\n 'Labels': labels\n }\n\n if driver is not None:\n if utils.version_lt(self._version, '1.31'):\n raise errors.InvalidVersion(\n 'Secret driver is only available for API version > 1.31'\n )\n\n body['Driver'] = driver\n\n url = self._url('/secrets/create')\n return self._result(\n self._post_json(url, data=body), True\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving secret metadata Args: id (string): Full ID of the secret to remove Returns (dict): A dictionary of metadata Raises: :py:class:`docker.errors.NotFound` if no secret with that ID exists", "response": "def inspect_secret(self, id):\n \"\"\"\n Retrieve secret metadata\n\n Args:\n id (string): Full ID of the secret to remove\n\n Returns (dict): A dictionary of metadata\n\n Raises:\n :py:class:`docker.errors.NotFound`\n if no secret with that ID exists\n \"\"\"\n url = self._url('/secrets/{0}', id)\n return self._result(self._get(url), True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update(self, node_spec):\n return self.client.api.update_node(self.id, self.version, node_spec)", "response": "Update the node s configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving this node from the swarm.", "response": "def remove(self, force=False):\n \"\"\"\n Remove this node from the swarm.\n\n Args:\n force (bool): Force remove an active node. Default: `False`\n\n Returns:\n `True` if the request was successful.\n\n Raises:\n :py:class:`docker.errors.NotFound`\n If the node doesn't exist in the swarm.\n\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.remove_node(self.id, force=force)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get(self, node_id):\n return self.prepare_model(self.client.api.inspect_node(node_id))", "response": "Get a node s related attributes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list(self, *args, **kwargs):\n return [\n self.prepare_model(n)\n for n in self.client.api.nodes(*args, **kwargs)\n ]", "response": "List swarm nodes.\n\n Args:\n filters (dict): Filters to process on the nodes list. Valid\n filters: ``id``, ``name``, ``membership`` and ``role``.\n Default: ``None``\n\n Returns:\n A list of :py:class:`Node` objects.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> client.nodes.list(filters={'role': 'manager'})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nattempt to parse a json object from a buffer.", "response": "def json_splitter(buffer):\n \"\"\"Attempt to parse a json object from a buffer. If there is at least one\n object, return it and the rest of the buffer, otherwise return None.\n \"\"\"\n buffer = buffer.strip()\n try:\n obj, index = json_decoder.raw_decode(buffer)\n rest = buffer[json.decoder.WHITESPACE.match(buffer, index).end():]\n return obj, rest\n except ValueError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconfigures the current node s settings.", "response": "def configure(self, options):\n \"\"\"\n Update the plugin's settings.\n\n Args:\n options (dict): A key-value mapping of options.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n self.client.api.configure_plugin(self.name, options)\n self.reload()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndisable the plugin. Raises: :py:class:`docker.errors.APIError` If the server returns an error.", "response": "def disable(self):\n \"\"\"\n Disable the plugin.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n self.client.api.disable_plugin(self.name)\n self.reload()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef enable(self, timeout=0):\n self.client.api.enable_plugin(self.name, timeout)\n self.reload()", "response": "Enable the plugin.\n\n Args:\n timeout (int): Timeout in seconds. Default: 0\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving the plugin from the server.", "response": "def remove(self, force=False):\n \"\"\"\n Remove the plugin from the server.\n\n Args:\n force (bool): Remove even if the plugin is enabled.\n Default: False\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.remove_plugin(self.name, force=force)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef upgrade(self, remote=None):\n if self.enabled:\n raise errors.DockerError(\n 'Plugin must be disabled before upgrading.'\n )\n\n if remote is None:\n remote = self.name\n privileges = self.client.api.plugin_privileges(remote)\n for d in self.client.api.upgrade_plugin(self.name, remote, privileges):\n yield d\n self._reload()", "response": "Upgrade the plugin.\n\n Args:\n remote (string): Remote reference to upgrade to. The\n ``:latest`` tag is optional and is the default if omitted.\n Default: this plugin's name.\n\n Returns:\n A generator streaming the decoded API logs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create(self, name, plugin_data_dir, gzip=False):\n self.client.api.create_plugin(name, plugin_data_dir, gzip)\n return self.get(name)", "response": "Create a new plugin."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get(self, name):\n return self.prepare_model(self.client.api.inspect_plugin(name))", "response": "Gets a plugin s internal cache id."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninstalling a plugin from the remote server.", "response": "def install(self, remote_name, local_name=None):\n \"\"\"\n Pull and install a plugin.\n\n Args:\n remote_name (string): Remote reference for the plugin to\n install. The ``:latest`` tag is optional, and is the\n default if omitted.\n local_name (string): Local name for the pulled plugin.\n The ``:latest`` tag is optional, and is the default if\n omitted. Optional.\n\n Returns:\n (:py:class:`Plugin`): The installed plugin\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n privileges = self.client.api.plugin_privileges(remote_name)\n it = self.client.api.pull_plugin(remote_name, privileges, local_name)\n for data in it:\n pass\n return self.get(local_name or remote_name)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlist the installed plugins on the server.", "response": "def list(self):\n \"\"\"\n List plugins installed on the server.\n\n Returns:\n (list of :py:class:`Plugin`): The plugins.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.plugins()\n return [self.prepare_model(r) for r in resp]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting data usage information.", "response": "def df(self):\n \"\"\"\n Get data usage information.\n\n Returns:\n (dict): A dictionary representing different resource categories\n and their respective data usage.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url('/system/df')\n return self._result(self._get(url), True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting real - time events from the server. Similar to the docker events command.", "response": "def events(self, since=None, until=None, filters=None, decode=None):\n \"\"\"\n Get real-time events from the server. Similar to the ``docker events``\n command.\n\n Args:\n since (UTC datetime or int): Get events from this point\n until (UTC datetime or int): Get events until this point\n filters (dict): Filter the events by event time, container or image\n decode (bool): If set to true, stream will be decoded into dicts on\n the fly. False by default.\n\n Returns:\n A :py:class:`docker.types.daemon.CancellableStream` generator\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> for event in client.events(decode=True)\n ... print(event)\n {u'from': u'image/with:tag',\n u'id': u'container-id',\n u'status': u'start',\n u'time': 1423339459}\n ...\n\n or\n\n >>> events = client.events()\n >>> for event in events:\n ... print(event)\n >>> # and cancel from another thread\n >>> events.close()\n \"\"\"\n\n if isinstance(since, datetime):\n since = utils.datetime_to_timestamp(since)\n\n if isinstance(until, datetime):\n until = utils.datetime_to_timestamp(until)\n\n if filters:\n filters = utils.convert_filters(filters)\n\n params = {\n 'since': since,\n 'until': until,\n 'filters': filters\n }\n url = self._url('/events')\n\n response = self._get(url, params=params, stream=True, timeout=None)\n stream = self._stream_helper(response, decode=decode)\n\n return types.CancellableStream(stream, response)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef login(self, username, password=None, email=None, registry=None,\n reauth=False, dockercfg_path=None):\n \"\"\"\n Authenticate with a registry. Similar to the ``docker login`` command.\n\n Args:\n username (str): The registry username\n password (str): The plaintext password\n email (str): The email for the registry account\n registry (str): URL to the registry. E.g.\n ``https://index.docker.io/v1/``\n reauth (bool): Whether or not to refresh existing authentication on\n the Docker server.\n dockercfg_path (str): Use a custom path for the Docker config file\n (default ``$HOME/.docker/config.json`` if present,\n otherwise``$HOME/.dockercfg``)\n\n Returns:\n (dict): The response from the login request\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n # If we don't have any auth data so far, try reloading the config file\n # one more time in case anything showed up in there.\n # If dockercfg_path is passed check to see if the config file exists,\n # if so load that config.\n if dockercfg_path and os.path.exists(dockercfg_path):\n self._auth_configs = auth.load_config(\n dockercfg_path, credstore_env=self.credstore_env\n )\n elif not self._auth_configs or self._auth_configs.is_empty:\n self._auth_configs = auth.load_config(\n credstore_env=self.credstore_env\n )\n\n authcfg = self._auth_configs.resolve_authconfig(registry)\n # If we found an existing auth config for this registry and username\n # combination, we can return it immediately unless reauth is requested.\n if authcfg and authcfg.get('username', None) == username \\\n and not reauth:\n return authcfg\n\n req_data = {\n 'username': username,\n 'password': password,\n 'email': email,\n 'serveraddress': registry,\n }\n\n response = self._post_json(self._url('/auth'), data=req_data)\n if response.status_code == 200:\n self._auth_configs.add_auth(registry or auth.INDEX_NAME, req_data)\n return self._result(response, json=True)", "response": "Authenticate with a registry."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef version(self, api_version=True):\n url = self._url(\"/version\", versioned_api=api_version)\n return self._result(self._get(url), json=True)", "response": "Returns version information from the server. Similar to the docker\n version command."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_service(\n self, task_template, name=None, labels=None, mode=None,\n update_config=None, networks=None, endpoint_config=None,\n endpoint_spec=None, rollback_config=None\n ):\n \"\"\"\n Create a service.\n\n Args:\n task_template (TaskTemplate): Specification of the task to start as\n part of the new service.\n name (string): User-defined name for the service. Optional.\n labels (dict): A map of labels to associate with the service.\n Optional.\n mode (ServiceMode): Scheduling mode for the service (replicated\n or global). Defaults to replicated.\n update_config (UpdateConfig): Specification for the update strategy\n of the service. Default: ``None``\n rollback_config (RollbackConfig): Specification for the rollback\n strategy of the service. Default: ``None``\n networks (:py:class:`list`): List of network names or IDs to attach\n the service to. Default: ``None``.\n endpoint_spec (EndpointSpec): Properties that can be configured to\n access and load balance a service. Default: ``None``.\n\n Returns:\n A dictionary containing an ``ID`` key for the newly created\n service.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n _check_api_features(\n self._version, task_template, update_config, endpoint_spec,\n rollback_config\n )\n\n url = self._url('/services/create')\n headers = {}\n image = task_template.get('ContainerSpec', {}).get('Image', None)\n if image is None:\n raise errors.DockerException(\n 'Missing mandatory Image key in ContainerSpec'\n )\n if mode and not isinstance(mode, dict):\n mode = ServiceMode(mode)\n\n registry, repo_name = auth.resolve_repository_name(image)\n auth_header = auth.get_config_header(self, registry)\n if auth_header:\n headers['X-Registry-Auth'] = auth_header\n if utils.version_lt(self._version, '1.25'):\n networks = networks or task_template.pop('Networks', None)\n data = {\n 'Name': name,\n 'Labels': labels,\n 'TaskTemplate': task_template,\n 'Mode': mode,\n 'Networks': utils.convert_service_networks(networks),\n 'EndpointSpec': endpoint_spec\n }\n\n if update_config is not None:\n data['UpdateConfig'] = update_config\n\n if rollback_config is not None:\n data['RollbackConfig'] = rollback_config\n\n return self._result(\n self._post_json(url, data=data, headers=headers), True\n )", "response": "Creates a new service in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef inspect_service(self, service, insert_defaults=None):\n url = self._url('/services/{0}', service)\n params = {}\n if insert_defaults is not None:\n if utils.version_lt(self._version, '1.29'):\n raise errors.InvalidVersion(\n 'insert_defaults is not supported in API version < 1.29'\n )\n params['insertDefaults'] = insert_defaults\n\n return self._result(self._get(url, params=params), True)", "response": "Return information about a service."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving information about a task.", "response": "def inspect_task(self, task):\n \"\"\"\n Retrieve information about a task.\n\n Args:\n task (str): Task ID\n\n Returns:\n (dict): Information about the task.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url('/tasks/{0}', task)\n return self._result(self._get(url), True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_service(self, service):\n\n url = self._url('/services/{0}', service)\n resp = self._delete(url)\n self._raise_for_status(resp)\n return True", "response": "Stop and remove a service."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef service_logs(self, service, details=False, follow=False, stdout=False,\n stderr=False, since=0, timestamps=False, tail='all',\n is_tty=None):\n \"\"\"\n Get log stream for a service.\n Note: This endpoint works only for services with the ``json-file``\n or ``journald`` logging drivers.\n\n Args:\n service (str): ID or name of the service\n details (bool): Show extra details provided to logs.\n Default: ``False``\n follow (bool): Keep connection open to read logs as they are\n sent by the Engine. Default: ``False``\n stdout (bool): Return logs from ``stdout``. Default: ``False``\n stderr (bool): Return logs from ``stderr``. Default: ``False``\n since (int): UNIX timestamp for the logs staring point.\n Default: 0\n timestamps (bool): Add timestamps to every log line.\n tail (string or int): Number of log lines to be returned,\n counting from the current end of the logs. Specify an\n integer or ``'all'`` to output all log lines.\n Default: ``all``\n is_tty (bool): Whether the service's :py:class:`ContainerSpec`\n enables the TTY option. If omitted, the method will query\n the Engine for the information, causing an additional\n roundtrip.\n\n Returns (generator): Logs for the service.\n \"\"\"\n params = {\n 'details': details,\n 'follow': follow,\n 'stdout': stdout,\n 'stderr': stderr,\n 'since': since,\n 'timestamps': timestamps,\n 'tail': tail\n }\n\n url = self._url('/services/{0}/logs', service)\n res = self._get(url, params=params, stream=True)\n if is_tty is None:\n is_tty = self.inspect_service(\n service\n )['Spec']['TaskTemplate']['ContainerSpec'].get('TTY', False)\n return self._get_result_tty(True, res, is_tty)", "response": "Returns a generator that returns the logs for a given service."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating a service in the cluster.", "response": "def update_service(self, service, version, task_template=None, name=None,\n labels=None, mode=None, update_config=None,\n networks=None, endpoint_config=None,\n endpoint_spec=None, fetch_current_spec=False,\n rollback_config=None):\n \"\"\"\n Update a service.\n\n Args:\n service (string): A service identifier (either its name or service\n ID).\n version (int): The version number of the service object being\n updated. This is required to avoid conflicting writes.\n task_template (TaskTemplate): Specification of the updated task to\n start as part of the service.\n name (string): New name for the service. Optional.\n labels (dict): A map of labels to associate with the service.\n Optional.\n mode (ServiceMode): Scheduling mode for the service (replicated\n or global). Defaults to replicated.\n update_config (UpdateConfig): Specification for the update strategy\n of the service. Default: ``None``.\n rollback_config (RollbackConfig): Specification for the rollback\n strategy of the service. Default: ``None``\n networks (:py:class:`list`): List of network names or IDs to attach\n the service to. Default: ``None``.\n endpoint_spec (EndpointSpec): Properties that can be configured to\n access and load balance a service. Default: ``None``.\n fetch_current_spec (boolean): Use the undefined settings from the\n current specification of the service. Default: ``False``\n\n Returns:\n A dictionary containing a ``Warnings`` key.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n _check_api_features(\n self._version, task_template, update_config, endpoint_spec,\n rollback_config\n )\n\n if fetch_current_spec:\n inspect_defaults = True\n if utils.version_lt(self._version, '1.29'):\n inspect_defaults = None\n current = self.inspect_service(\n service, insert_defaults=inspect_defaults\n )['Spec']\n\n else:\n current = {}\n\n url = self._url('/services/{0}/update', service)\n data = {}\n headers = {}\n\n data['Name'] = current.get('Name') if name is None else name\n\n data['Labels'] = current.get('Labels') if labels is None else labels\n\n if mode is not None:\n if not isinstance(mode, dict):\n mode = ServiceMode(mode)\n data['Mode'] = mode\n else:\n data['Mode'] = current.get('Mode')\n\n data['TaskTemplate'] = _merge_task_template(\n current.get('TaskTemplate', {}), task_template\n )\n\n container_spec = data['TaskTemplate'].get('ContainerSpec', {})\n image = container_spec.get('Image', None)\n if image is not None:\n registry, repo_name = auth.resolve_repository_name(image)\n auth_header = auth.get_config_header(self, registry)\n if auth_header:\n headers['X-Registry-Auth'] = auth_header\n\n if update_config is not None:\n data['UpdateConfig'] = update_config\n else:\n data['UpdateConfig'] = current.get('UpdateConfig')\n\n if rollback_config is not None:\n data['RollbackConfig'] = rollback_config\n else:\n data['RollbackConfig'] = current.get('RollbackConfig')\n\n if networks is not None:\n converted_networks = utils.convert_service_networks(networks)\n if utils.version_lt(self._version, '1.25'):\n data['Networks'] = converted_networks\n else:\n data['TaskTemplate']['Networks'] = converted_networks\n elif utils.version_lt(self._version, '1.25'):\n data['Networks'] = current.get('Networks')\n elif data['TaskTemplate'].get('Networks') is None:\n current_task_template = current.get('TaskTemplate', {})\n current_networks = current_task_template.get('Networks')\n if current_networks is None:\n current_networks = current.get('Networks')\n if current_networks is not None:\n data['TaskTemplate']['Networks'] = current_networks\n\n if endpoint_spec is not None:\n data['EndpointSpec'] = endpoint_spec\n else:\n data['EndpointSpec'] = current.get('EndpointSpec')\n\n resp = self._post_json(\n url, data=data, params={'version': version}, headers=headers\n )\n return self._result(resp, json=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a suitable APIError from requests. exceptions. HTTPError.", "response": "def create_api_error_from_http_exception(e):\n \"\"\"\n Create a suitable APIError from requests.exceptions.HTTPError.\n \"\"\"\n response = e.response\n try:\n explanation = response.json()['message']\n except ValueError:\n explanation = (response.content or '').strip()\n cls = APIError\n if response.status_code == 404:\n if explanation and ('No such image' in str(explanation) or\n 'not found: does not exist or no pull access'\n in str(explanation) or\n 'repository does not exist' in str(explanation)):\n cls = ImageNotFound\n else:\n cls = NotFound\n raise cls(e, response=response, explanation=explanation)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_dict(config):\n '''\n Instantiate a new ProxyConfig from a dictionary that represents a\n client configuration, as described in `the documentation`_.\n\n .. _the documentation:\n https://docs.docker.com/network/proxy/#configure-the-docker-client\n '''\n return ProxyConfig(\n http=config.get('httpProxy'),\n https=config.get('httpsProxy'),\n ftp=config.get('ftpProxy'),\n no_proxy=config.get('noProxy'),\n )", "response": "Instantiate a new ProxyConfig from a dictionary that represents a\n client configuration."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a dictionary representing the environment variables used to get the proxy settings.", "response": "def get_environment(self):\n '''\n Return a dictionary representing the environment variables used to\n set the proxy settings.\n '''\n env = {}\n if self.http:\n env['http_proxy'] = env['HTTP_PROXY'] = self.http\n if self.https:\n env['https_proxy'] = env['HTTPS_PROXY'] = self.https\n if self.ftp:\n env['ftp_proxy'] = env['FTP_PROXY'] = self.ftp\n if self.no_proxy:\n env['no_proxy'] = env['NO_PROXY'] = self.no_proxy\n return env"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a list of strings representing environment variables prepend the environment variables corresponding to the proxy settings.", "response": "def inject_proxy_environment(self, environment):\n '''\n Given a list of strings representing environment variables, prepend the\n environment variables corresponding to the proxy settings.\n '''\n if not self:\n return environment\n\n proxy_env = format_environment(self.get_environment())\n if not environment:\n return proxy_env\n # It is important to prepend our variables, because we want the\n # variables defined in \"environment\" to take precedence.\n return proxy_env + environment"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef order(self):\n # non-GA releases should appear before GA releases\n # Order: tp -> beta -> rc -> GA\n if self.stage:\n for st in STAGES:\n if st in self.stage:\n stage = (STAGES.index(st), self.stage)\n break\n else:\n stage = (len(STAGES),)\n\n return (int(self.major), int(self.minor), int(self.patch)) + stage", "response": "Return a representation that allows this object to be sorted\n correctly with the default comparator."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef remove(self, force=False):\n return self.client.api.remove_volume(self.id, force=force)", "response": "Remove this volume.\n\n Args:\n force (bool): Force removal of volumes that were already removed\n out of band by the volume driver plugin.\n Raises:\n :py:class:`docker.errors.APIError`\n If volume failed to remove."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create(self, name=None, **kwargs):\n obj = self.client.api.create_volume(name, **kwargs)\n return self.prepare_model(obj)", "response": "Create a new volume."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, volume_id):\n return self.prepare_model(self.client.api.inspect_volume(volume_id))", "response": "Get a specific volume."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list(self, **kwargs):\n resp = self.client.api.volumes(**kwargs)\n if not resp.get('Volumes'):\n return []\n return [self.prepare_model(obj) for obj in resp['Volumes']]", "response": "List volumes. Similar to the docker volume ls command."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fnmatch(name, pat):\n\n name = name.lower()\n pat = pat.lower()\n return fnmatchcase(name, pat)", "response": "Test whether FILENAME matches PATTERN."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, config_id):\n return self.prepare_model(self.client.api.inspect_config(config_id))", "response": "Get a config.\n\n Args:\n config_id (str): Config ID.\n\n Returns:\n (:py:class:`Config`): The config.\n\n Raises:\n :py:class:`docker.errors.NotFound`\n If the config does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_connection(self, *args, **kwargs):\n conn = super(SSLHTTPAdapter, self).get_connection(*args, **kwargs)\n if conn.assert_hostname != self.assert_hostname:\n conn.assert_hostname = self.assert_hostname\n return conn", "response": "Override the default get_connection method to set the assert_hostname to the value of the assert_hostname attribute."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef exclude_paths(root, patterns, dockerfile=None):\n\n if dockerfile is None:\n dockerfile = 'Dockerfile'\n\n patterns.append('!' + dockerfile)\n pm = PatternMatcher(patterns)\n return set(pm.walk(root))", "response": "Returns a set of all paths that do not match any of the patterns in the root directory path."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_image(self, image, chunk_size=DEFAULT_DATA_CHUNK_SIZE):\n res = self._get(self._url(\"/images/{0}/get\", image), stream=True)\n return self._stream_raw_result(res, chunk_size, False)", "response": "Get a tarball of an image. Similar to the docker get command."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef history(self, image):\n res = self._get(self._url(\"/images/{0}/history\", image))\n return self._result(res, True)", "response": "Show the history of an image."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef images(self, name=None, quiet=False, all=False, filters=None):\n params = {\n 'filter': name,\n 'only_ids': 1 if quiet else 0,\n 'all': 1 if all else 0,\n }\n if filters:\n params['filters'] = utils.convert_filters(filters)\n res = self._result(self._get(self._url(\"/images/json\"), params=params),\n True)\n if quiet:\n return [x['Id'] for x in res]\n return res", "response": "List images belonging to a repository."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nimport an image from a file or URL.", "response": "def import_image(self, src=None, repository=None, tag=None, image=None,\n changes=None, stream_src=False):\n \"\"\"\n Import an image. Similar to the ``docker import`` command.\n\n If ``src`` is a string or unicode string, it will first be treated as a\n path to a tarball on the local system. If there is an error reading\n from that file, ``src`` will be treated as a URL instead to fetch the\n image from. You can also pass an open file handle as ``src``, in which\n case the data will be read from that file.\n\n If ``src`` is unset but ``image`` is set, the ``image`` parameter will\n be taken as the name of an existing image to import from.\n\n Args:\n src (str or file): Path to tarfile, URL, or file-like object\n repository (str): The repository to create\n tag (str): The tag to apply\n image (str): Use another image like the ``FROM`` Dockerfile\n parameter\n \"\"\"\n if not (src or image):\n raise errors.DockerException(\n 'Must specify src or image to import from'\n )\n u = self._url('/images/create')\n\n params = _import_image_params(\n repository, tag, image,\n src=(src if isinstance(src, six.string_types) else None),\n changes=changes\n )\n headers = {'Content-Type': 'application/tar'}\n\n if image or params.get('fromSrc') != '-': # from image or URL\n return self._result(\n self._post(u, data=None, params=params)\n )\n elif isinstance(src, six.string_types): # from file path\n with open(src, 'rb') as f:\n return self._result(\n self._post(\n u, data=f, params=params, headers=headers, timeout=None\n )\n )\n else: # from raw data\n if stream_src:\n headers['Transfer-Encoding'] = 'chunked'\n return self._result(\n self._post(u, data=src, params=params, headers=headers)\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nliking :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but allows importing in-memory bytes data. Args: data (bytes collection): Bytes collection containing valid tar data repository (str): The repository to create tag (str): The tag to apply", "response": "def import_image_from_data(self, data, repository=None, tag=None,\n changes=None):\n \"\"\"\n Like :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but\n allows importing in-memory bytes data.\n\n Args:\n data (bytes collection): Bytes collection containing valid tar data\n repository (str): The repository to create\n tag (str): The tag to apply\n \"\"\"\n\n u = self._url('/images/create')\n params = _import_image_params(\n repository, tag, src='-', changes=changes\n )\n headers = {'Content-Type': 'application/tar'}\n return self._result(\n self._post(\n u, data=data, params=params, headers=headers, timeout=None\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef import_image_from_file(self, filename, repository=None, tag=None,\n changes=None):\n \"\"\"\n Like :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but only\n supports importing from a tar file on disk.\n\n Args:\n filename (str): Full path to a tar file.\n repository (str): The repository to create\n tag (str): The tag to apply\n\n Raises:\n IOError: File does not exist.\n \"\"\"\n\n return self.import_image(\n src=filename, repository=repository, tag=tag, changes=changes\n )", "response": "Imports an image from a tar file on disk."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef import_image_from_url(self, url, repository=None, tag=None,\n changes=None):\n \"\"\"\n Like :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but only\n supports importing from a URL.\n\n Args:\n url (str): A URL pointing to a tar file.\n repository (str): The repository to create\n tag (str): The tag to apply\n \"\"\"\n return self.import_image(\n src=url, repository=repository, tag=tag, changes=changes\n )", "response": "Imports an image from a URL."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlike :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but only supports importing from another image, like the ``FROM`` Dockerfile parameter. Args: image (str): Image name to import from repository (str): The repository to create tag (str): The tag to apply", "response": "def import_image_from_image(self, image, repository=None, tag=None,\n changes=None):\n \"\"\"\n Like :py:meth:`~docker.api.image.ImageApiMixin.import_image`, but only\n supports importing from another image, like the ``FROM`` Dockerfile\n parameter.\n\n Args:\n image (str): Image name to import from\n repository (str): The repository to create\n tag (str): The tag to apply\n \"\"\"\n return self.import_image(\n image=image, repository=repository, tag=tag, changes=changes\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef inspect_image(self, image):\n return self._result(\n self._get(self._url(\"/images/{0}/json\", image)), True\n )", "response": "Get detailed information about an image. Similar to the docker inspect command but only for images."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets image digest and platform information by contacting the registry.", "response": "def inspect_distribution(self, image, auth_config=None):\n \"\"\"\n Get image digest and platform information by contacting the registry.\n\n Args:\n image (str): The image name to inspect\n auth_config (dict): Override the credentials that are found in the\n config for this request. ``auth_config`` should contain the\n ``username`` and ``password`` keys to be valid.\n\n Returns:\n (dict): A dict containing distribution data\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n registry, _ = auth.resolve_repository_name(image)\n\n headers = {}\n if auth_config is None:\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n else:\n log.debug('Sending supplied auth config')\n headers['X-Registry-Auth'] = auth.encode_header(auth_config)\n\n url = self._url(\"/distribution/{0}/json\", image)\n\n return self._result(\n self._get(url, headers=headers), True\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_image(self, data, quiet=None):\n params = {}\n\n if quiet is not None:\n if utils.version_lt(self._version, '1.23'):\n raise errors.InvalidVersion(\n 'quiet is not supported in API version < 1.23'\n )\n params['quiet'] = quiet\n\n res = self._post(\n self._url(\"/images/load\"), data=data, params=params, stream=True\n )\n if utils.version_gte(self._version, '1.23'):\n return self._stream_helper(res, decode=True)\n\n self._raise_for_status(res)", "response": "Load an image from the server."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pull(self, repository, tag=None, stream=False, auth_config=None,\n decode=False, platform=None):\n \"\"\"\n Pulls an image. Similar to the ``docker pull`` command.\n\n Args:\n repository (str): The repository to pull\n tag (str): The tag to pull\n stream (bool): Stream the output as a generator. Make sure to\n consume the generator, otherwise pull might get cancelled.\n auth_config (dict): Override the credentials that are found in the\n config for this request. ``auth_config`` should contain the\n ``username`` and ``password`` keys to be valid.\n decode (bool): Decode the JSON data from the server into dicts.\n Only applies with ``stream=True``\n platform (str): Platform in the format ``os[/arch[/variant]]``\n\n Returns:\n (generator or str): The output\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> for line in cli.pull('busybox', stream=True, decode=True):\n ... print(json.dumps(line, indent=4))\n {\n \"status\": \"Pulling image (latest) from busybox\",\n \"progressDetail\": {},\n \"id\": \"e72ac664f4f0\"\n }\n {\n \"status\": \"Pulling image (latest) from busybox, endpoint: ...\",\n \"progressDetail\": {},\n \"id\": \"e72ac664f4f0\"\n }\n\n \"\"\"\n if not tag:\n repository, tag = utils.parse_repository_tag(repository)\n registry, repo_name = auth.resolve_repository_name(repository)\n\n params = {\n 'tag': tag,\n 'fromImage': repository\n }\n headers = {}\n\n if auth_config is None:\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n else:\n log.debug('Sending supplied auth config')\n headers['X-Registry-Auth'] = auth.encode_header(auth_config)\n\n if platform is not None:\n if utils.version_lt(self._version, '1.32'):\n raise errors.InvalidVersion(\n 'platform was only introduced in API version 1.32'\n )\n params['platform'] = platform\n\n response = self._post(\n self._url('/images/create'), params=params, headers=headers,\n stream=stream, timeout=None\n )\n\n self._raise_for_status(response)\n\n if stream:\n return self._stream_helper(response, decode=decode)\n\n return self._result(response)", "response": "Pulls an image from the specified repository."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef push(self, repository, tag=None, stream=False, auth_config=None,\n decode=False):\n \"\"\"\n Push an image or a repository to the registry. Similar to the ``docker\n push`` command.\n\n Args:\n repository (str): The repository to push to\n tag (str): An optional tag to push\n stream (bool): Stream the output as a blocking generator\n auth_config (dict): Override the credentials that are found in the\n config for this request. ``auth_config`` should contain the\n ``username`` and ``password`` keys to be valid.\n decode (bool): Decode the JSON data from the server into dicts.\n Only applies with ``stream=True``\n\n Returns:\n (generator or str): The output from the server.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n >>> for line in cli.push('yourname/app', stream=True, decode=True):\n ... print(line)\n {'status': 'Pushing repository yourname/app (1 tags)'}\n {'status': 'Pushing','progressDetail': {}, 'id': '511136ea3c5a'}\n {'status': 'Image already pushed, skipping', 'progressDetail':{},\n 'id': '511136ea3c5a'}\n ...\n\n \"\"\"\n if not tag:\n repository, tag = utils.parse_repository_tag(repository)\n registry, repo_name = auth.resolve_repository_name(repository)\n u = self._url(\"/images/{0}/push\", repository)\n params = {\n 'tag': tag\n }\n headers = {}\n\n if auth_config is None:\n header = auth.get_config_header(self, registry)\n if header:\n headers['X-Registry-Auth'] = header\n else:\n log.debug('Sending supplied auth config')\n headers['X-Registry-Auth'] = auth.encode_header(auth_config)\n\n response = self._post_json(\n u, None, headers=headers, stream=stream, params=params\n )\n\n self._raise_for_status(response)\n\n if stream:\n return self._stream_helper(response, decode=decode)\n\n return self._result(response)", "response": "Push an image or a repository to the registry."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_image(self, image, force=False, noprune=False):\n params = {'force': force, 'noprune': noprune}\n res = self._delete(self._url(\"/images/{0}\", image), params=params)\n return self._result(res, True)", "response": "Remove an image from the cluster."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef search(self, term):\n return self._result(\n self._get(self._url(\"/images/search\"), params={'term': term}),\n True\n )", "response": "Search for images on Docker Hub. Similar to the docker search image command."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntagging an image into a repository.", "response": "def tag(self, image, repository, tag=None, force=False):\n \"\"\"\n Tag an image into a repository. Similar to the ``docker tag`` command.\n\n Args:\n image (str): The image to tag\n repository (str): The repository to set for the tag\n tag (str): The tag name\n force (bool): Force\n\n Returns:\n (bool): ``True`` if successful\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> client.tag('ubuntu', 'localhost:5000/ubuntu', 'latest',\n force=True)\n \"\"\"\n params = {\n 'tag': tag,\n 'repo': repository,\n 'force': 1 if force else 0\n }\n url = self._url(\"/images/{0}/tag\", image)\n res = self._post(url, params=params)\n self._raise_for_status(res)\n return res.status_code == 201"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef attach(self, container, stdout=True, stderr=True,\n stream=False, logs=False, demux=False):\n \"\"\"\n Attach to a container.\n\n The ``.logs()`` function is a wrapper around this method, which you can\n use instead if you want to fetch/stream container output without first\n retrieving the entire backlog.\n\n Args:\n container (str): The container to attach to.\n stdout (bool): Include stdout.\n stderr (bool): Include stderr.\n stream (bool): Return container output progressively as an iterator\n of strings, rather than a single string.\n logs (bool): Include the container's previous output.\n demux (bool): Keep stdout and stderr separate.\n\n Returns:\n By default, the container's output as a single string (two if\n ``demux=True``: one for stdout and one for stderr).\n\n If ``stream=True``, an iterator of output strings. If\n ``demux=True``, two iterators are returned: one for stdout and one\n for stderr.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n params = {\n 'logs': logs and 1 or 0,\n 'stdout': stdout and 1 or 0,\n 'stderr': stderr and 1 or 0,\n 'stream': stream and 1 or 0\n }\n\n headers = {\n 'Connection': 'Upgrade',\n 'Upgrade': 'tcp'\n }\n\n u = self._url(\"/containers/{0}/attach\", container)\n response = self._post(u, headers=headers, params=params, stream=True)\n\n output = self._read_from_socket(\n response, stream, self._check_is_tty(container), demux=demux)\n\n if stream:\n return CancellableStream(output, response)\n else:\n return output", "response": "Attaches a container to a specific container."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlikes attach but returns the underlying socket - like object for the HTTP request.", "response": "def attach_socket(self, container, params=None, ws=False):\n \"\"\"\n Like ``attach``, but returns the underlying socket-like object for the\n HTTP request.\n\n Args:\n container (str): The container to attach to.\n params (dict): Dictionary of request parameters (e.g. ``stdout``,\n ``stderr``, ``stream``).\n For ``detachKeys``, ~/.docker/config.json is used by default.\n ws (bool): Use websockets instead of raw HTTP.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if params is None:\n params = {\n 'stdout': 1,\n 'stderr': 1,\n 'stream': 1\n }\n\n if 'detachKeys' not in params \\\n and 'detachKeys' in self._general_configs:\n\n params['detachKeys'] = self._general_configs['detachKeys']\n\n if ws:\n return self._attach_websocket(container, params)\n\n headers = {\n 'Connection': 'Upgrade',\n 'Upgrade': 'tcp'\n }\n\n u = self._url(\"/containers/{0}/attach\", container)\n return self._get_raw_response_socket(\n self.post(\n u, None, params=self._attach_params(params), stream=True,\n headers=headers\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef commit(self, container, repository=None, tag=None, message=None,\n author=None, changes=None, conf=None):\n \"\"\"\n Commit a container to an image. Similar to the ``docker commit``\n command.\n\n Args:\n container (str): The image hash of the container\n repository (str): The repository to push the image to\n tag (str): The tag to push\n message (str): A commit message\n author (str): The name of the author\n changes (str): Dockerfile instructions to apply while committing\n conf (dict): The configuration for the container. See the\n `Engine API documentation\n `_\n for full details.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n params = {\n 'container': container,\n 'repo': repository,\n 'tag': tag,\n 'comment': message,\n 'author': author,\n 'changes': changes\n }\n u = self._url(\"/commit\")\n return self._result(\n self._post_json(u, data=conf, params=params), json=True\n )", "response": "Commits a container to an image. Similar to the docker commit command."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef containers(self, quiet=False, all=False, trunc=False, latest=False,\n since=None, before=None, limit=-1, size=False,\n filters=None):\n \"\"\"\n List containers. Similar to the ``docker ps`` command.\n\n Args:\n quiet (bool): Only display numeric Ids\n all (bool): Show all containers. Only running containers are shown\n by default\n trunc (bool): Truncate output\n latest (bool): Show only the latest created container, include\n non-running ones.\n since (str): Show only containers created since Id or Name, include\n non-running ones\n before (str): Show only container created before Id or Name,\n include non-running ones\n limit (int): Show `limit` last created containers, include\n non-running ones\n size (bool): Display sizes\n filters (dict): Filters to be processed on the image list.\n Available filters:\n\n - `exited` (int): Only containers with specified exit code\n - `status` (str): One of ``restarting``, ``running``,\n ``paused``, ``exited``\n - `label` (str): format either ``\"key\"`` or ``\"key=value\"``\n - `id` (str): The id of the container.\n - `name` (str): The name of the container.\n - `ancestor` (str): Filter by container ancestor. Format of\n ``[:tag]``, ````, or\n ````.\n - `before` (str): Only containers created before a particular\n container. Give the container name or id.\n - `since` (str): Only containers created after a particular\n container. Give container name or id.\n\n A comprehensive list can be found in the documentation for\n `docker ps\n `_.\n\n Returns:\n A list of dicts, one per container\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n params = {\n 'limit': 1 if latest else limit,\n 'all': 1 if all else 0,\n 'size': 1 if size else 0,\n 'trunc_cmd': 1 if trunc else 0,\n 'since': since,\n 'before': before\n }\n if filters:\n params['filters'] = utils.convert_filters(filters)\n u = self._url(\"/containers/json\")\n res = self._result(self._get(u, params=params), True)\n\n if quiet:\n return [{'Id': x['Id']} for x in res]\n if trunc:\n for x in res:\n x['Id'] = x['Id'][:12]\n return res", "response": "List containers in the system."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a Docker container.", "response": "def create_container(self, image, command=None, hostname=None, user=None,\n detach=False, stdin_open=False, tty=False, ports=None,\n environment=None, volumes=None,\n network_disabled=False, name=None, entrypoint=None,\n working_dir=None, domainname=None, host_config=None,\n mac_address=None, labels=None, stop_signal=None,\n networking_config=None, healthcheck=None,\n stop_timeout=None, runtime=None,\n use_config_proxy=False):\n \"\"\"\n Creates a container. Parameters are similar to those for the ``docker\n run`` command except it doesn't support the attach options (``-a``).\n\n The arguments that are passed directly to this function are\n host-independent configuration options. Host-specific configuration\n is passed with the `host_config` argument. You'll normally want to\n use this method in combination with the :py:meth:`create_host_config`\n method to generate ``host_config``.\n\n **Port bindings**\n\n Port binding is done in two parts: first, provide a list of ports to\n open inside the container with the ``ports`` parameter, then declare\n bindings with the ``host_config`` parameter. For example:\n\n .. code-block:: python\n\n container_id = cli.create_container(\n 'busybox', 'ls', ports=[1111, 2222],\n host_config=cli.create_host_config(port_bindings={\n 1111: 4567,\n 2222: None\n })\n )\n\n\n You can limit the host address on which the port will be exposed like\n such:\n\n .. code-block:: python\n\n cli.create_host_config(port_bindings={1111: ('127.0.0.1', 4567)})\n\n Or without host port assignment:\n\n .. code-block:: python\n\n cli.create_host_config(port_bindings={1111: ('127.0.0.1',)})\n\n If you wish to use UDP instead of TCP (default), you need to declare\n ports as such in both the config and host config:\n\n .. code-block:: python\n\n container_id = cli.create_container(\n 'busybox', 'ls', ports=[(1111, 'udp'), 2222],\n host_config=cli.create_host_config(port_bindings={\n '1111/udp': 4567, 2222: None\n })\n )\n\n To bind multiple host ports to a single container port, use the\n following syntax:\n\n .. code-block:: python\n\n cli.create_host_config(port_bindings={\n 1111: [1234, 4567]\n })\n\n You can also bind multiple IPs to a single container port:\n\n .. code-block:: python\n\n cli.create_host_config(port_bindings={\n 1111: [\n ('192.168.0.100', 1234),\n ('192.168.0.101', 1234)\n ]\n })\n\n **Using volumes**\n\n Volume declaration is done in two parts. Provide a list of\n paths to use as mountpoints inside the container with the\n ``volumes`` parameter, and declare mappings from paths on the host\n in the ``host_config`` section.\n\n .. code-block:: python\n\n container_id = cli.create_container(\n 'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],\n host_config=cli.create_host_config(binds={\n '/home/user1/': {\n 'bind': '/mnt/vol2',\n 'mode': 'rw',\n },\n '/var/www': {\n 'bind': '/mnt/vol1',\n 'mode': 'ro',\n }\n })\n )\n\n You can alternatively specify binds as a list. This code is equivalent\n to the example above:\n\n .. code-block:: python\n\n container_id = cli.create_container(\n 'busybox', 'ls', volumes=['/mnt/vol1', '/mnt/vol2'],\n host_config=cli.create_host_config(binds=[\n '/home/user1/:/mnt/vol2',\n '/var/www:/mnt/vol1:ro',\n ])\n )\n\n **Networking**\n\n You can specify networks to connect the container to by using the\n ``networking_config`` parameter. At the time of creation, you can\n only connect a container to a single networking, but you\n can create more connections by using\n :py:meth:`~connect_container_to_network`.\n\n For example:\n\n .. code-block:: python\n\n networking_config = docker_client.create_networking_config({\n 'network1': docker_client.create_endpoint_config(\n ipv4_address='172.28.0.124',\n aliases=['foo', 'bar'],\n links=['container2']\n )\n })\n\n ctnr = docker_client.create_container(\n img, command, networking_config=networking_config\n )\n\n Args:\n image (str): The image to run\n command (str or list): The command to be run in the container\n hostname (str): Optional hostname for the container\n user (str or int): Username or UID\n detach (bool): Detached mode: run container in the background and\n return container ID\n stdin_open (bool): Keep STDIN open even if not attached\n tty (bool): Allocate a pseudo-TTY\n ports (list of ints): A list of port numbers\n environment (dict or list): A dictionary or a list of strings in\n the following format ``[\"PASSWORD=xxx\"]`` or\n ``{\"PASSWORD\": \"xxx\"}``.\n volumes (str or list): List of paths inside the container to use\n as volumes.\n network_disabled (bool): Disable networking\n name (str): A name for the container\n entrypoint (str or list): An entrypoint\n working_dir (str): Path to the working directory\n domainname (str): The domain name to use for the container\n host_config (dict): A dictionary created with\n :py:meth:`create_host_config`.\n mac_address (str): The Mac Address to assign the container\n labels (dict or list): A dictionary of name-value labels (e.g.\n ``{\"label1\": \"value1\", \"label2\": \"value2\"}``) or a list of\n names of labels to set with empty values (e.g.\n ``[\"label1\", \"label2\"]``)\n stop_signal (str): The stop signal to use to stop the container\n (e.g. ``SIGINT``).\n stop_timeout (int): Timeout to stop the container, in seconds.\n Default: 10\n networking_config (dict): A networking configuration generated\n by :py:meth:`create_networking_config`.\n runtime (str): Runtime to use with this container.\n healthcheck (dict): Specify a test to perform to check that the\n container is healthy.\n use_config_proxy (bool): If ``True``, and if the docker client\n configuration file (``~/.docker/config.json`` by default)\n contains a proxy configuration, the corresponding environment\n variables will be set in the container being created.\n\n Returns:\n A dictionary with an image 'Id' key and a 'Warnings' key.\n\n Raises:\n :py:class:`docker.errors.ImageNotFound`\n If the specified image does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if isinstance(volumes, six.string_types):\n volumes = [volumes, ]\n\n if isinstance(environment, dict):\n environment = utils.utils.format_environment(environment)\n\n if use_config_proxy:\n environment = self._proxy_configs.inject_proxy_environment(\n environment\n )\n\n config = self.create_container_config(\n image, command, hostname, user, detach, stdin_open, tty,\n ports, environment, volumes,\n network_disabled, entrypoint, working_dir, domainname,\n host_config, mac_address, labels,\n stop_signal, networking_config, healthcheck,\n stop_timeout, runtime\n )\n return self.create_container_from_config(config, name)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_host_config(self, *args, **kwargs):\n if not kwargs:\n kwargs = {}\n if 'version' in kwargs:\n raise TypeError(\n \"create_host_config() got an unexpected \"\n \"keyword argument 'version'\"\n )\n kwargs['version'] = self._version\n return HostConfig(*args, **kwargs)", "response": "Create a dictionary for the host_config argument to\n ."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef diff(self, container):\n return self._result(\n self._get(self._url(\"/containers/{0}/changes\", container)), True\n )", "response": "Inspect changes on a container s filesystem."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef export(self, container, chunk_size=DEFAULT_DATA_CHUNK_SIZE):\n res = self._get(\n self._url(\"/containers/{0}/export\", container), stream=True\n )\n return self._stream_raw_result(res, chunk_size, False)", "response": "Export the contents of a container as a tar archive."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_archive(self, container, path, chunk_size=DEFAULT_DATA_CHUNK_SIZE):\n params = {\n 'path': path\n }\n url = self._url('/containers/{0}/archive', container)\n res = self._get(url, params=params, stream=True)\n self._raise_for_status(res)\n encoded_stat = res.headers.get('x-docker-container-path-stat')\n return (\n self._stream_raw_result(res, chunk_size, False),\n utils.decode_json_header(encoded_stat) if encoded_stat else None\n )", "response": "Retrieve a file or folder from a container in the form of a tar archive."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef inspect_container(self, container):\n return self._result(\n self._get(self._url(\"/containers/{0}/json\", container)), True\n )", "response": "Returns a dictionary containing information about the container s attributes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef kill(self, container, signal=None):\n url = self._url(\"/containers/{0}/kill\", container)\n params = {}\n if signal is not None:\n if not isinstance(signal, six.string_types):\n signal = int(signal)\n params['signal'] = signal\n res = self._post(url, params=params)\n\n self._raise_for_status(res)", "response": "Kill a container or send a signal to a container."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef logs(self, container, stdout=True, stderr=True, stream=False,\n timestamps=False, tail='all', since=None, follow=None,\n until=None):\n \"\"\"\n Get logs from a container. Similar to the ``docker logs`` command.\n\n The ``stream`` parameter makes the ``logs`` function return a blocking\n generator you can iterate over to retrieve log output as it happens.\n\n Args:\n container (str): The container to get logs from\n stdout (bool): Get ``STDOUT``. Default ``True``\n stderr (bool): Get ``STDERR``. Default ``True``\n stream (bool): Stream the response. Default ``False``\n timestamps (bool): Show timestamps. Default ``False``\n tail (str or int): Output specified number of lines at the end of\n logs. Either an integer of number of lines or the string\n ``all``. Default ``all``\n since (datetime or int): Show logs since a given datetime or\n integer epoch (in seconds)\n follow (bool): Follow log output. Default ``False``\n until (datetime or int): Show logs that occurred before the given\n datetime or integer epoch (in seconds)\n\n Returns:\n (generator or str)\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if follow is None:\n follow = stream\n params = {'stderr': stderr and 1 or 0,\n 'stdout': stdout and 1 or 0,\n 'timestamps': timestamps and 1 or 0,\n 'follow': follow and 1 or 0,\n }\n if tail != 'all' and (not isinstance(tail, int) or tail < 0):\n tail = 'all'\n params['tail'] = tail\n\n if since is not None:\n if isinstance(since, datetime):\n params['since'] = utils.datetime_to_timestamp(since)\n elif (isinstance(since, int) and since > 0):\n params['since'] = since\n else:\n raise errors.InvalidArgument(\n 'since value should be datetime or positive int, '\n 'not {}'.format(type(since))\n )\n\n if until is not None:\n if utils.version_lt(self._version, '1.35'):\n raise errors.InvalidVersion(\n 'until is not supported for API version < 1.35'\n )\n if isinstance(until, datetime):\n params['until'] = utils.datetime_to_timestamp(until)\n elif (isinstance(until, int) and until > 0):\n params['until'] = until\n else:\n raise errors.InvalidArgument(\n 'until value should be datetime or positive int, '\n 'not {}'.format(type(until))\n )\n\n url = self._url(\"/containers/{0}/logs\", container)\n res = self._get(url, params=params, stream=stream)\n output = self._get_result(container, stream, res)\n\n if stream:\n return CancellableStream(output, res)\n else:\n return output", "response": "Get logs from a container."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pause(self, container):\n url = self._url('/containers/{0}/pause', container)\n res = self._post(url)\n self._raise_for_status(res)", "response": "Pauses all processes within a container."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the public - facing port that is NAT - ed to private_port.", "response": "def port(self, container, private_port):\n \"\"\"\n Lookup the public-facing port that is NAT-ed to ``private_port``.\n Identical to the ``docker port`` command.\n\n Args:\n container (str): The container to look up\n private_port (int): The private port to inspect\n\n Returns:\n (list of dict): The mapping for the host ports\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n .. code-block:: bash\n\n $ docker run -d -p 80:80 ubuntu:14.04 /bin/sleep 30\n 7174d6347063a83f412fad6124c99cffd25ffe1a0807eb4b7f9cec76ac8cb43b\n\n .. code-block:: python\n\n >>> cli.port('7174d6347063', 80)\n [{'HostIp': '0.0.0.0', 'HostPort': '80'}]\n \"\"\"\n res = self._get(self._url(\"/containers/{0}/json\", container))\n self._raise_for_status(res)\n json_ = res.json()\n private_port = str(private_port)\n h_ports = None\n\n # Port settings is None when the container is running with\n # network_mode=host.\n port_settings = json_.get('NetworkSettings', {}).get('Ports')\n if port_settings is None:\n return None\n\n if '/' in private_port:\n return port_settings.get(private_port)\n\n for protocol in ['tcp', 'udp', 'sctp']:\n h_ports = port_settings.get(private_port + '/' + protocol)\n if h_ports:\n break\n\n return h_ports"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef put_archive(self, container, path, data):\n params = {'path': path}\n url = self._url('/containers/{0}/archive', container)\n res = self._put(url, params=params, data=data)\n self._raise_for_status(res)\n return res.status_code == 200", "response": "Insert a file or folder in an existing container using a tar archive as\n source."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_container(self, container, v=False, link=False, force=False):\n params = {'v': v, 'link': link, 'force': force}\n res = self._delete(\n self._url(\"/containers/{0}\", container), params=params\n )\n self._raise_for_status(res)", "response": "Removes a container from the cluster."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrename a container. Similar to the docker rename command.", "response": "def rename(self, container, name):\n \"\"\"\n Rename a container. Similar to the ``docker rename`` command.\n\n Args:\n container (str): ID of the container to rename\n name (str): New name for the container\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url(\"/containers/{0}/rename\", container)\n params = {'name': name}\n res = self._post(url, params=params)\n self._raise_for_status(res)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef resize(self, container, height, width):\n params = {'h': height, 'w': width}\n url = self._url(\"/containers/{0}/resize\", container)\n res = self._post(url, params=params)\n self._raise_for_status(res)", "response": "Resize the tty session."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef restart(self, container, timeout=10):\n params = {'t': timeout}\n url = self._url(\"/containers/{0}/restart\", container)\n conn_timeout = self.timeout\n if conn_timeout is not None:\n conn_timeout += timeout\n res = self._post(url, params=params, timeout=conn_timeout)\n self._raise_for_status(res)", "response": "Restarts a container. Similar to the docker restart command."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start(self, container, *args, **kwargs):\n if args or kwargs:\n raise errors.DeprecatedMethod(\n 'Providing configuration in the start() method is no longer '\n 'supported. Use the host_config param in create_container '\n 'instead.'\n )\n url = self._url(\"/containers/{0}/start\", container)\n res = self._post(url)\n self._raise_for_status(res)", "response": "Start a container. Similar to the ``docker start`` command, but\n doesn't support attach options.\n\n **Deprecation warning:** Passing configuration options in ``start`` is\n no longer supported. Users are expected to provide host config options\n in the ``host_config`` parameter of\n :py:meth:`~ContainerApiMixin.create_container`.\n\n\n Args:\n container (str): The container to start\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n :py:class:`docker.errors.DeprecatedMethod`\n If any argument besides ``container`` are provided.\n\n Example:\n\n >>> container = cli.create_container(\n ... image='busybox:latest',\n ... command='/bin/sleep 30')\n >>> cli.start(container=container.get('Id'))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stats(self, container, decode=None, stream=True):\n url = self._url(\"/containers/{0}/stats\", container)\n if stream:\n return self._stream_helper(self._get(url, stream=True),\n decode=decode)\n else:\n if decode:\n raise errors.InvalidArgument(\n \"decode is only available in conjuction with stream=True\"\n )\n return self._result(self._get(url, params={'stream': False}),\n json=True)", "response": "Returns a dictionary of the current statistics for a specific container."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndisplaying the running processes of a container.", "response": "def top(self, container, ps_args=None):\n \"\"\"\n Display the running processes of a container.\n\n Args:\n container (str): The container to inspect\n ps_args (str): An optional arguments passed to ps (e.g. ``aux``)\n\n Returns:\n (str): The output of the top\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n u = self._url(\"/containers/{0}/top\", container)\n params = {}\n if ps_args is not None:\n params['ps_args'] = ps_args\n return self._result(self._get(u, params=params), True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unpause(self, container):\n url = self._url('/containers/{0}/unpause', container)\n res = self._post(url)\n self._raise_for_status(res)", "response": "Unpause all processes within a container."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_container(\n self, container, blkio_weight=None, cpu_period=None, cpu_quota=None,\n cpu_shares=None, cpuset_cpus=None, cpuset_mems=None, mem_limit=None,\n mem_reservation=None, memswap_limit=None, kernel_memory=None,\n restart_policy=None\n ):\n \"\"\"\n Update resource configs of one or more containers.\n\n Args:\n container (str): The container to inspect\n blkio_weight (int): Block IO (relative weight), between 10 and 1000\n cpu_period (int): Limit CPU CFS (Completely Fair Scheduler) period\n cpu_quota (int): Limit CPU CFS (Completely Fair Scheduler) quota\n cpu_shares (int): CPU shares (relative weight)\n cpuset_cpus (str): CPUs in which to allow execution\n cpuset_mems (str): MEMs in which to allow execution\n mem_limit (int or str): Memory limit\n mem_reservation (int or str): Memory soft limit\n memswap_limit (int or str): Total memory (memory + swap), -1 to\n disable swap\n kernel_memory (int or str): Kernel memory limit\n restart_policy (dict): Restart policy dictionary\n\n Returns:\n (dict): Dictionary containing a ``Warnings`` key.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n url = self._url('/containers/{0}/update', container)\n data = {}\n if blkio_weight:\n data['BlkioWeight'] = blkio_weight\n if cpu_period:\n data['CpuPeriod'] = cpu_period\n if cpu_shares:\n data['CpuShares'] = cpu_shares\n if cpu_quota:\n data['CpuQuota'] = cpu_quota\n if cpuset_cpus:\n data['CpusetCpus'] = cpuset_cpus\n if cpuset_mems:\n data['CpusetMems'] = cpuset_mems\n if mem_limit:\n data['Memory'] = utils.parse_bytes(mem_limit)\n if mem_reservation:\n data['MemoryReservation'] = utils.parse_bytes(mem_reservation)\n if memswap_limit:\n data['MemorySwap'] = utils.parse_bytes(memswap_limit)\n if kernel_memory:\n data['KernelMemory'] = utils.parse_bytes(kernel_memory)\n if restart_policy:\n if utils.version_lt(self._version, '1.23'):\n raise errors.InvalidVersion(\n 'restart policy update is not supported '\n 'for API version < 1.23'\n )\n data['RestartPolicy'] = restart_policy\n\n res = self._post_json(url, data=data)\n return self._result(res, True)", "response": "Updates the resource configs of one or more containers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wait(self, container, timeout=None, condition=None):\n url = self._url(\"/containers/{0}/wait\", container)\n params = {}\n if condition is not None:\n if utils.version_lt(self._version, '1.30'):\n raise errors.InvalidVersion(\n 'wait condition is not supported for API version < 1.30'\n )\n params['condition'] = condition\n\n res = self._post(url, timeout=timeout, params=params)\n return self._result(res, True)", "response": "Block until a container stops."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tags(self):\n tags = self.attrs.get('RepoTags')\n if tags is None:\n tags = []\n return [tag for tag in tags if tag != ':']", "response": "A list of the tags of this image."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a tarball of an image. Similar to the docker save command.", "response": "def save(self, chunk_size=DEFAULT_DATA_CHUNK_SIZE, named=False):\n \"\"\"\n Get a tarball of an image. Similar to the ``docker save`` command.\n\n Args:\n chunk_size (int): The generator will return up to that much data\n per iteration, but may return less. If ``None``, data will be\n streamed as it is received. Default: 2 MB\n named (str or bool): If ``False`` (default), the tarball will not\n retain repository and tag information for this image. If set\n to ``True``, the first tag in the :py:attr:`~tags` list will\n be used to identify the image. Alternatively, any element of\n the :py:attr:`~tags` list can be used as an argument to use\n that specific tag as the saved identifier.\n\n Returns:\n (generator): A stream of raw archive data.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n\n >>> image = cli.get_image(\"busybox:latest\")\n >>> f = open('/tmp/busybox-latest.tar', 'wb')\n >>> for chunk in image:\n >>> f.write(chunk)\n >>> f.close()\n \"\"\"\n img = self.id\n if named:\n img = self.tags[0] if self.tags else img\n if isinstance(named, six.string_types):\n if named not in self.tags:\n raise InvalidArgument(\n \"{} is not a valid tag for this image\".format(named)\n )\n img = named\n\n return self.client.api.get_image(img, chunk_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tag(self, repository, tag=None, **kwargs):\n return self.client.api.tag(self.id, repository, tag=tag, **kwargs)", "response": "Tag this image into a repository."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\npull the image digest.", "response": "def pull(self, platform=None):\n \"\"\"\n Pull the image digest.\n\n Args:\n platform (str): The platform to pull the image for.\n Default: ``None``\n\n Returns:\n (:py:class:`Image`): A reference to the pulled image.\n \"\"\"\n repository, _ = parse_repository_tag(self.image_name)\n return self.collection.pull(repository, tag=self.id, platform=platform)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks whether the given platform identifier is available for this resource.", "response": "def has_platform(self, platform):\n \"\"\"\n Check whether the given platform identifier is available for this\n digest.\n\n Args:\n platform (str or dict): A string using the ``os[/arch[/variant]]``\n format, or a platform dictionary.\n\n Returns:\n (bool): ``True`` if the platform is recognized as available,\n ``False`` otherwise.\n\n Raises:\n :py:class:`docker.errors.InvalidArgument`\n If the platform argument is not a valid descriptor.\n \"\"\"\n if platform and not isinstance(platform, dict):\n parts = platform.split('/')\n if len(parts) > 3 or len(parts) < 1:\n raise InvalidArgument(\n '\"{0}\" is not a valid platform descriptor'.format(platform)\n )\n platform = {'os': parts[0]}\n if len(parts) > 2:\n platform['variant'] = parts[2]\n if len(parts) > 1:\n platform['architecture'] = parts[1]\n return normalize_platform(\n platform, self.client.version()\n ) in self.attrs['Platforms']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build(self, **kwargs):\n resp = self.client.api.build(**kwargs)\n if isinstance(resp, six.string_types):\n return self.get(resp)\n last_event = None\n image_id = None\n result_stream, internal_stream = itertools.tee(json_stream(resp))\n for chunk in internal_stream:\n if 'error' in chunk:\n raise BuildError(chunk['error'], result_stream)\n if 'stream' in chunk:\n match = re.search(\n r'(^Successfully built |sha256:)([0-9a-f]+)$',\n chunk['stream']\n )\n if match:\n image_id = match.group(2)\n last_event = chunk\n if image_id:\n return (self.get(image_id), result_stream)\n raise BuildError(last_event or 'Unknown', result_stream)", "response": "Builds an image and returns it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get(self, name):\n return self.prepare_model(self.client.api.inspect_image(name))", "response": "Gets an image.\n\n Args:\n name (str): The name of the image.\n\n Returns:\n (:py:class:`Image`): The image.\n\n Raises:\n :py:class:`docker.errors.ImageNotFound`\n If the image does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_registry_data(self, name, auth_config=None):\n return RegistryData(\n image_name=name,\n attrs=self.client.api.inspect_distribution(name, auth_config),\n client=self.client,\n collection=self,\n )", "response": "Gets the registry data for an image."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlist the images in the repository.", "response": "def list(self, name=None, all=False, filters=None):\n \"\"\"\n List images on the server.\n\n Args:\n name (str): Only show images belonging to the repository ``name``\n all (bool): Show intermediate image layers. By default, these are\n filtered out.\n filters (dict): Filters to be processed on the image list.\n Available filters:\n - ``dangling`` (bool)\n - ``label`` (str): format either ``key`` or ``key=value``\n\n Returns:\n (list of :py:class:`Image`): The images.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.images(name=name, all=all, filters=filters)\n return [self.get(r[\"Id\"]) for r in resp]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load(self, data):\n resp = self.client.api.load_image(data)\n images = []\n for chunk in resp:\n if 'stream' in chunk:\n match = re.search(\n r'(^Loaded image ID: |^Loaded image: )(.+)$',\n chunk['stream']\n )\n if match:\n image_id = match.group(2)\n images.append(image_id)\n if 'error' in chunk:\n raise ImageLoadError(chunk['error'])\n\n return [self.get(i) for i in images]", "response": "Load an image from a binary image stream."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pull(self, repository, tag=None, **kwargs):\n if not tag:\n repository, tag = parse_repository_tag(repository)\n\n if 'stream' in kwargs:\n warnings.warn(\n '`stream` is not a valid parameter for this method'\n ' and will be overridden'\n )\n del kwargs['stream']\n\n pull_log = self.client.api.pull(\n repository, tag=tag, stream=True, **kwargs\n )\n for _ in pull_log:\n # We don't do anything with the logs, but we need\n # to keep the connection alive and wait for the image\n # to be pulled.\n pass\n if tag:\n return self.get('{0}{2}{1}'.format(\n repository, tag, '@' if tag.startswith('sha256:') else ':'\n ))\n return self.list(repository)", "response": "Pull an image of the given name and return it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting arguments to create and create_container.", "response": "def _create_container_args(kwargs):\n \"\"\"\n Convert arguments to create() to arguments to create_container().\n \"\"\"\n # Copy over kwargs which can be copied directly\n create_kwargs = {}\n for key in copy.copy(kwargs):\n if key in RUN_CREATE_KWARGS:\n create_kwargs[key] = kwargs.pop(key)\n host_config_kwargs = {}\n for key in copy.copy(kwargs):\n if key in RUN_HOST_CONFIG_KWARGS:\n host_config_kwargs[key] = kwargs.pop(key)\n\n # Process kwargs which are split over both create and host_config\n ports = kwargs.pop('ports', {})\n if ports:\n host_config_kwargs['port_bindings'] = ports\n\n volumes = kwargs.pop('volumes', {})\n if volumes:\n host_config_kwargs['binds'] = volumes\n\n network = kwargs.pop('network', None)\n if network:\n create_kwargs['networking_config'] = {network: None}\n host_config_kwargs['network_mode'] = network\n\n # All kwargs should have been consumed by this point, so raise\n # error if any are left\n if kwargs:\n raise create_unexpected_kwargs_error('run', kwargs)\n\n create_kwargs['host_config'] = HostConfig(**host_config_kwargs)\n\n # Fill in any kwargs which need processing by create_host_config first\n port_bindings = create_kwargs['host_config'].get('PortBindings')\n if port_bindings:\n # sort to make consistent for tests\n create_kwargs['ports'] = [tuple(p.split('/', 1))\n for p in sorted(port_bindings.keys())]\n if volumes:\n if isinstance(volumes, dict):\n create_kwargs['volumes'] = [\n v.get('bind') for v in volumes.values()\n ]\n else:\n create_kwargs['volumes'] = [\n _host_volume_from_bind(v) for v in volumes\n ]\n return create_kwargs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef image(self):\n image_id = self.attrs.get('ImageID', self.attrs['Image'])\n if image_id is None:\n return None\n return self.client.images.get(image_id.split(':')[1])", "response": "Returns the image of the container."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef status(self):\n if isinstance(self.attrs['State'], dict):\n return self.attrs['State']['Status']\n return self.attrs['State']", "response": "Returns the status of the container. For example running or exited."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef attach(self, **kwargs):\n return self.client.api.attach(self.id, **kwargs)", "response": "Attach to this container."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef attach_socket(self, **kwargs):\n return self.client.api.attach_socket(self.id, **kwargs)", "response": "Like attach but returns the underlying socket - like object\n for the HTTP request."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef commit(self, repository=None, tag=None, **kwargs):\n\n resp = self.client.api.commit(self.id, repository=repository, tag=tag,\n **kwargs)\n return self.client.images.get(resp['Id'])", "response": "Commits a container to an image. Similar to the docker commit command."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef exec_run(self, cmd, stdout=True, stderr=True, stdin=False, tty=False,\n privileged=False, user='', detach=False, stream=False,\n socket=False, environment=None, workdir=None, demux=False):\n \"\"\"\n Run a command inside this container. Similar to\n ``docker exec``.\n\n Args:\n cmd (str or list): Command to be executed\n stdout (bool): Attach to stdout. Default: ``True``\n stderr (bool): Attach to stderr. Default: ``True``\n stdin (bool): Attach to stdin. Default: ``False``\n tty (bool): Allocate a pseudo-TTY. Default: False\n privileged (bool): Run as privileged.\n user (str): User to execute command as. Default: root\n detach (bool): If true, detach from the exec command.\n Default: False\n stream (bool): Stream response data. Default: False\n socket (bool): Return the connection socket to allow custom\n read/write operations. Default: False\n environment (dict or list): A dictionary or a list of strings in\n the following format ``[\"PASSWORD=xxx\"]`` or\n ``{\"PASSWORD\": \"xxx\"}``.\n workdir (str): Path to working directory for this exec session\n demux (bool): Return stdout and stderr separately\n\n Returns:\n (ExecResult): A tuple of (exit_code, output)\n exit_code: (int):\n Exit code for the executed command or ``None`` if\n either ``stream`` or ``socket`` is ``True``.\n output: (generator, bytes, or tuple):\n If ``stream=True``, a generator yielding response chunks.\n If ``socket=True``, a socket object for the connection.\n If ``demux=True``, a tuple of two bytes: stdout and stderr.\n A bytestring containing response data otherwise.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.exec_create(\n self.id, cmd, stdout=stdout, stderr=stderr, stdin=stdin, tty=tty,\n privileged=privileged, user=user, environment=environment,\n workdir=workdir,\n )\n exec_output = self.client.api.exec_start(\n resp['Id'], detach=detach, tty=tty, stream=stream, socket=socket,\n demux=demux\n )\n if socket or stream:\n return ExecResult(None, exec_output)\n\n return ExecResult(\n self.client.api.exec_inspect(resp['Id'])['ExitCode'],\n exec_output\n )", "response": "Runs a command inside this container. Similar to docker exec."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef export(self, chunk_size=DEFAULT_DATA_CHUNK_SIZE):\n return self.client.api.export(self.id, chunk_size)", "response": "Export the contents of the container s filesystem as a tar archive."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_archive(self, path, chunk_size=DEFAULT_DATA_CHUNK_SIZE):\n return self.client.api.get_archive(self.id, path, chunk_size)", "response": "Retrieve a file or folder from the container in the form of a tar archive."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef kill(self, signal=None):\n\n return self.client.api.kill(self.id, signal=signal)", "response": "Kill or send a signal to the container."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef logs(self, **kwargs):\n return self.client.api.logs(self.id, **kwargs)", "response": "Returns a generator that returns the logs from the container."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef put_archive(self, path, data):\n return self.client.api.put_archive(self.id, path, data)", "response": "Insert a file or folder in this container using a tar archive as\n source."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove(self, **kwargs):\n return self.client.api.remove_container(self.id, **kwargs)", "response": "Remove this container. Similar to the docker rm command."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrename this container. Similar to the docker rename command.", "response": "def rename(self, name):\n \"\"\"\n Rename this container. Similar to the ``docker rename`` command.\n\n Args:\n name (str): New name for the container\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.rename(self.id, name)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef resize(self, height, width):\n return self.client.api.resize(self.id, height, width)", "response": "Resize the tty session."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef restart(self, **kwargs):\n return self.client.api.restart(self.id, **kwargs)", "response": "Restarts this container. Similar to the docker restart command."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstarts this container. Similar to the docker start command but causes an error if the server returns an error.", "response": "def start(self, **kwargs):\n \"\"\"\n Start this container. Similar to the ``docker start`` command, but\n doesn't support attach options.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.start(self.id, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the current statistics for this container.", "response": "def stats(self, **kwargs):\n \"\"\"\n Stream statistics for this container. Similar to the\n ``docker stats`` command.\n\n Args:\n decode (bool): If set to true, stream will be decoded into dicts\n on the fly. Only applicable if ``stream`` is True.\n False by default.\n stream (bool): If set to false, only the current stats will be\n returned instead of a stream. True by default.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.stats(self.id, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop(self, **kwargs):\n return self.client.api.stop(self.id, **kwargs)", "response": "Stops a container. Similar to the docker stop command."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndisplays the running processes of the container.", "response": "def top(self, **kwargs):\n \"\"\"\n Display the running processes of the container.\n\n Args:\n ps_args (str): An optional arguments passed to ps (e.g. ``aux``)\n\n Returns:\n (str): The output of the top\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.top(self.id, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the resource configuration of the containers.", "response": "def update(self, **kwargs):\n \"\"\"\n Update resource configuration of the containers.\n\n Args:\n blkio_weight (int): Block IO (relative weight), between 10 and 1000\n cpu_period (int): Limit CPU CFS (Completely Fair Scheduler) period\n cpu_quota (int): Limit CPU CFS (Completely Fair Scheduler) quota\n cpu_shares (int): CPU shares (relative weight)\n cpuset_cpus (str): CPUs in which to allow execution\n cpuset_mems (str): MEMs in which to allow execution\n mem_limit (int or str): Memory limit\n mem_reservation (int or str): Memory soft limit\n memswap_limit (int or str): Total memory (memory + swap), -1 to\n disable swap\n kernel_memory (int or str): Kernel memory limit\n restart_policy (dict): Restart policy dictionary\n\n Returns:\n (dict): Dictionary containing a ``Warnings`` key.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return self.client.api.update_container(self.id, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wait(self, **kwargs):\n return self.client.api.wait(self.id, **kwargs)", "response": "Block until the container stops then return its exit code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(self, image, command=None, stdout=True, stderr=False,\n remove=False, **kwargs):\n \"\"\"\n Run a container. By default, it will wait for the container to finish\n and return its logs, similar to ``docker run``.\n\n If the ``detach`` argument is ``True``, it will start the container\n and immediately return a :py:class:`Container` object, similar to\n ``docker run -d``.\n\n Example:\n Run a container and get its output:\n\n >>> import docker\n >>> client = docker.from_env()\n >>> client.containers.run('alpine', 'echo hello world')\n b'hello world\\\\n'\n\n Run a container and detach:\n\n >>> container = client.containers.run('bfirsh/reticulate-splines',\n detach=True)\n >>> container.logs()\n 'Reticulating spline 1...\\\\nReticulating spline 2...\\\\n'\n\n Args:\n image (str): The image to run.\n command (str or list): The command to run in the container.\n auto_remove (bool): enable auto-removal of the container on daemon\n side when the container's process exits.\n blkio_weight_device: Block IO weight (relative device weight) in\n the form of: ``[{\"Path\": \"device_path\", \"Weight\": weight}]``.\n blkio_weight: Block IO weight (relative weight), accepts a weight\n value between 10 and 1000.\n cap_add (list of str): Add kernel capabilities. For example,\n ``[\"SYS_ADMIN\", \"MKNOD\"]``.\n cap_drop (list of str): Drop kernel capabilities.\n cgroup_parent (str): Override the default parent cgroup.\n cpu_count (int): Number of usable CPUs (Windows only).\n cpu_percent (int): Usable percentage of the available CPUs\n (Windows only).\n cpu_period (int): The length of a CPU period in microseconds.\n cpu_quota (int): Microseconds of CPU time that the container can\n get in a CPU period.\n cpu_rt_period (int): Limit CPU real-time period in microseconds.\n cpu_rt_runtime (int): Limit CPU real-time runtime in microseconds.\n cpu_shares (int): CPU shares (relative weight).\n cpuset_cpus (str): CPUs in which to allow execution (``0-3``,\n ``0,1``).\n cpuset_mems (str): Memory nodes (MEMs) in which to allow execution\n (``0-3``, ``0,1``). Only effective on NUMA systems.\n detach (bool): Run container in the background and return a\n :py:class:`Container` object.\n device_cgroup_rules (:py:class:`list`): A list of cgroup rules to\n apply to the container.\n device_read_bps: Limit read rate (bytes per second) from a device\n in the form of: `[{\"Path\": \"device_path\", \"Rate\": rate}]`\n device_read_iops: Limit read rate (IO per second) from a device.\n device_write_bps: Limit write rate (bytes per second) from a\n device.\n device_write_iops: Limit write rate (IO per second) from a device.\n devices (:py:class:`list`): Expose host devices to the container,\n as a list of strings in the form\n ``::``.\n\n For example, ``/dev/sda:/dev/xvda:rwm`` allows the container\n to have read-write access to the host's ``/dev/sda`` via a\n node named ``/dev/xvda`` inside the container.\n dns (:py:class:`list`): Set custom DNS servers.\n dns_opt (:py:class:`list`): Additional options to be added to the\n container's ``resolv.conf`` file.\n dns_search (:py:class:`list`): DNS search domains.\n domainname (str or list): Set custom DNS search domains.\n entrypoint (str or list): The entrypoint for the container.\n environment (dict or list): Environment variables to set inside\n the container, as a dictionary or a list of strings in the\n format ``[\"SOMEVARIABLE=xxx\"]``.\n extra_hosts (dict): Additional hostnames to resolve inside the\n container, as a mapping of hostname to IP address.\n group_add (:py:class:`list`): List of additional group names and/or\n IDs that the container process will run as.\n healthcheck (dict): Specify a test to perform to check that the\n container is healthy.\n hostname (str): Optional hostname for the container.\n init (bool): Run an init inside the container that forwards\n signals and reaps processes\n init_path (str): Path to the docker-init binary\n ipc_mode (str): Set the IPC mode for the container.\n isolation (str): Isolation technology to use. Default: `None`.\n kernel_memory (int or str): Kernel memory limit\n labels (dict or list): A dictionary of name-value labels (e.g.\n ``{\"label1\": \"value1\", \"label2\": \"value2\"}``) or a list of\n names of labels to set with empty values (e.g.\n ``[\"label1\", \"label2\"]``)\n links (dict): Mapping of links using the\n ``{'container': 'alias'}`` format. The alias is optional.\n Containers declared in this dict will be linked to the new\n container using the provided alias. Default: ``None``.\n log_config (LogConfig): Logging configuration.\n lxc_conf (dict): LXC config.\n mac_address (str): MAC address to assign to the container.\n mem_limit (int or str): Memory limit. Accepts float values\n (which represent the memory limit of the created container in\n bytes) or a string with a units identification char\n (``100000b``, ``1000k``, ``128m``, ``1g``). If a string is\n specified without a units character, bytes are assumed as an\n intended unit.\n mem_reservation (int or str): Memory soft limit\n mem_swappiness (int): Tune a container's memory swappiness\n behavior. Accepts number between 0 and 100.\n memswap_limit (str or int): Maximum amount of memory + swap a\n container is allowed to consume.\n mounts (:py:class:`list`): Specification for mounts to be added to\n the container. More powerful alternative to ``volumes``. Each\n item in the list is expected to be a\n :py:class:`docker.types.Mount` object.\n name (str): The name for this container.\n nano_cpus (int): CPU quota in units of 1e-9 CPUs.\n network (str): Name of the network this container will be connected\n to at creation time. You can connect to additional networks\n using :py:meth:`Network.connect`. Incompatible with\n ``network_mode``.\n network_disabled (bool): Disable networking.\n network_mode (str): One of:\n\n - ``bridge`` Create a new network stack for the container on\n on the bridge network.\n - ``none`` No networking for this container.\n - ``container:`` Reuse another container's network\n stack.\n - ``host`` Use the host network stack.\n\n Incompatible with ``network``.\n oom_kill_disable (bool): Whether to disable OOM killer.\n oom_score_adj (int): An integer value containing the score given\n to the container in order to tune OOM killer preferences.\n pid_mode (str): If set to ``host``, use the host PID namespace\n inside the container.\n pids_limit (int): Tune a container's pids limit. Set ``-1`` for\n unlimited.\n platform (str): Platform in the format ``os[/arch[/variant]]``.\n Only used if the method needs to pull the requested image.\n ports (dict): Ports to bind inside the container.\n\n The keys of the dictionary are the ports to bind inside the\n container, either as an integer or a string in the form\n ``port/protocol``, where the protocol is either ``tcp``,\n ``udp``, or ``sctp``.\n\n The values of the dictionary are the corresponding ports to\n open on the host, which can be either:\n\n - The port number, as an integer. For example,\n ``{'2222/tcp': 3333}`` will expose port 2222 inside the\n container as port 3333 on the host.\n - ``None``, to assign a random host port. For example,\n ``{'2222/tcp': None}``.\n - A tuple of ``(address, port)`` if you want to specify the\n host interface. For example,\n ``{'1111/tcp': ('127.0.0.1', 1111)}``.\n - A list of integers, if you want to bind multiple host ports\n to a single container port. For example,\n ``{'1111/tcp': [1234, 4567]}``.\n\n privileged (bool): Give extended privileges to this container.\n publish_all_ports (bool): Publish all ports to the host.\n read_only (bool): Mount the container's root filesystem as read\n only.\n remove (bool): Remove the container when it has finished running.\n Default: ``False``.\n restart_policy (dict): Restart the container when it exits.\n Configured as a dictionary with keys:\n\n - ``Name`` One of ``on-failure``, or ``always``.\n - ``MaximumRetryCount`` Number of times to restart the\n container on failure.\n\n For example:\n ``{\"Name\": \"on-failure\", \"MaximumRetryCount\": 5}``\n\n runtime (str): Runtime to use with this container.\n security_opt (:py:class:`list`): A list of string values to\n customize labels for MLS systems, such as SELinux.\n shm_size (str or int): Size of /dev/shm (e.g. ``1G``).\n stdin_open (bool): Keep ``STDIN`` open even if not attached.\n stdout (bool): Return logs from ``STDOUT`` when ``detach=False``.\n Default: ``True``.\n stderr (bool): Return logs from ``STDERR`` when ``detach=False``.\n Default: ``False``.\n stop_signal (str): The stop signal to use to stop the container\n (e.g. ``SIGINT``).\n storage_opt (dict): Storage driver options per container as a\n key-value mapping.\n stream (bool): If true and ``detach`` is false, return a log\n generator instead of a string. Ignored if ``detach`` is true.\n Default: ``False``.\n sysctls (dict): Kernel parameters to set in the container.\n tmpfs (dict): Temporary filesystems to mount, as a dictionary\n mapping a path inside the container to options for that path.\n\n For example:\n\n .. code-block:: python\n\n {\n '/mnt/vol2': '',\n '/mnt/vol1': 'size=3G,uid=1000'\n }\n\n tty (bool): Allocate a pseudo-TTY.\n ulimits (:py:class:`list`): Ulimits to set inside the container,\n as a list of :py:class:`docker.types.Ulimit` instances.\n use_config_proxy (bool): If ``True``, and if the docker client\n configuration file (``~/.docker/config.json`` by default)\n contains a proxy configuration, the corresponding environment\n variables will be set in the container being built.\n user (str or int): Username or UID to run commands as inside the\n container.\n userns_mode (str): Sets the user namespace mode for the container\n when user namespace remapping option is enabled. Supported\n values are: ``host``\n uts_mode (str): Sets the UTS namespace mode for the container.\n Supported values are: ``host``\n version (str): The version of the API to use. Set to ``auto`` to\n automatically detect the server's version. Default: ``1.35``\n volume_driver (str): The name of a volume driver/plugin.\n volumes (dict or list): A dictionary to configure volumes mounted\n inside the container. The key is either the host path or a\n volume name, and the value is a dictionary with the keys:\n\n - ``bind`` The path to mount the volume inside the container\n - ``mode`` Either ``rw`` to mount the volume read/write, or\n ``ro`` to mount it read-only.\n\n For example:\n\n .. code-block:: python\n\n {'/home/user1/': {'bind': '/mnt/vol2', 'mode': 'rw'},\n '/var/www': {'bind': '/mnt/vol1', 'mode': 'ro'}}\n\n volumes_from (:py:class:`list`): List of container names or IDs to\n get volumes from.\n working_dir (str): Path to the working directory.\n\n Returns:\n The container logs, either ``STDOUT``, ``STDERR``, or both,\n depending on the value of the ``stdout`` and ``stderr`` arguments.\n\n ``STDOUT`` and ``STDERR`` may be read only if either ``json-file``\n or ``journald`` logging driver used. Thus, if you are using none of\n these drivers, a ``None`` object is returned instead. See the\n `Engine API documentation\n `_\n for full details.\n\n If ``detach`` is ``True``, a :py:class:`Container` object is\n returned instead.\n\n Raises:\n :py:class:`docker.errors.ContainerError`\n If the container exits with a non-zero exit code and\n ``detach`` is ``False``.\n :py:class:`docker.errors.ImageNotFound`\n If the specified image does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if isinstance(image, Image):\n image = image.id\n stream = kwargs.pop('stream', False)\n detach = kwargs.pop('detach', False)\n platform = kwargs.pop('platform', None)\n\n if detach and remove:\n if version_gte(self.client.api._version, '1.25'):\n kwargs[\"auto_remove\"] = True\n else:\n raise RuntimeError(\"The options 'detach' and 'remove' cannot \"\n \"be used together in api versions < 1.25.\")\n\n if kwargs.get('network') and kwargs.get('network_mode'):\n raise RuntimeError(\n 'The options \"network\" and \"network_mode\" can not be used '\n 'together.'\n )\n\n try:\n container = self.create(image=image, command=command,\n detach=detach, **kwargs)\n except ImageNotFound:\n self.client.images.pull(image, platform=platform)\n container = self.create(image=image, command=command,\n detach=detach, **kwargs)\n\n container.start()\n\n if detach:\n return container\n\n logging_driver = container.attrs['HostConfig']['LogConfig']['Type']\n\n out = None\n if logging_driver == 'json-file' or logging_driver == 'journald':\n out = container.logs(\n stdout=stdout, stderr=stderr, stream=True, follow=True\n )\n\n exit_status = container.wait()['StatusCode']\n if exit_status != 0:\n out = None\n if not kwargs.get('auto_remove'):\n out = container.logs(stdout=False, stderr=True)\n\n if remove:\n container.remove()\n if exit_status != 0:\n raise ContainerError(\n container, exit_status, command, image, out\n )\n\n return out if stream or out is None else b''.join(\n [line for line in out]\n )", "response": "Runs a container and returns its output."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create(self, image, command=None, **kwargs):\n if isinstance(image, Image):\n image = image.id\n kwargs['image'] = image\n kwargs['command'] = command\n kwargs['version'] = self.client.api._version\n create_kwargs = _create_container_args(kwargs)\n resp = self.client.api.create_container(**create_kwargs)\n return self.get(resp['Id'])", "response": "Create a container without starting it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget a container by name or ID.", "response": "def get(self, container_id):\n \"\"\"\n Get a container by name or ID.\n\n Args:\n container_id (str): Container name or ID.\n\n Returns:\n A :py:class:`Container` object.\n\n Raises:\n :py:class:`docker.errors.NotFound`\n If the container does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.inspect_container(container_id)\n return self.prepare_model(resp)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list(self, all=False, before=None, filters=None, limit=-1, since=None,\n sparse=False, ignore_removed=False):\n \"\"\"\n List containers. Similar to the ``docker ps`` command.\n\n Args:\n all (bool): Show all containers. Only running containers are shown\n by default\n since (str): Show only containers created since Id or Name, include\n non-running ones\n before (str): Show only container created before Id or Name,\n include non-running ones\n limit (int): Show `limit` last created containers, include\n non-running ones\n filters (dict): Filters to be processed on the image list.\n Available filters:\n\n - `exited` (int): Only containers with specified exit code\n - `status` (str): One of ``restarting``, ``running``,\n ``paused``, ``exited``\n - `label` (str): format either ``\"key\"`` or ``\"key=value\"``\n - `id` (str): The id of the container.\n - `name` (str): The name of the container.\n - `ancestor` (str): Filter by container ancestor. Format of\n ``[:tag]``, ````, or\n ````.\n - `before` (str): Only containers created before a particular\n container. Give the container name or id.\n - `since` (str): Only containers created after a particular\n container. Give container name or id.\n\n A comprehensive list can be found in the documentation for\n `docker ps\n `_.\n\n sparse (bool): Do not inspect containers. Returns partial\n information, but guaranteed not to block. Use\n :py:meth:`Container.reload` on resulting objects to retrieve\n all attributes. Default: ``False``\n ignore_removed (bool): Ignore failures due to missing containers\n when attempting to inspect containers from the original list.\n Set to ``True`` if race conditions are likely. Has no effect\n if ``sparse=True``. Default: ``False``\n\n Returns:\n (list of :py:class:`Container`)\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n resp = self.client.api.containers(all=all, before=before,\n filters=filters, limit=limit,\n since=since)\n if sparse:\n return [self.prepare_model(r) for r in resp]\n else:\n containers = []\n for r in resp:\n try:\n containers.append(self.get(r['Id']))\n # a container may have been removed while iterating\n except NotFound:\n if not ignore_removed:\n raise\n return containers", "response": "List the containers in the system."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of tasks in this service.", "response": "def tasks(self, filters=None):\n \"\"\"\n List the tasks in this service.\n\n Args:\n filters (dict): A map of filters to process on the tasks list.\n Valid filters: ``id``, ``name``, ``node``,\n ``label``, and ``desired-state``.\n\n Returns:\n :py:class:`list`: List of task dictionaries.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if filters is None:\n filters = {}\n filters['service'] = self.id\n return self.client.api.tasks(filters=filters)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update(self, **kwargs):\n # Image is required, so if it hasn't been set, use current image\n if 'image' not in kwargs:\n spec = self.attrs['Spec']['TaskTemplate']['ContainerSpec']\n kwargs['image'] = spec['Image']\n\n if kwargs.get('force_update') is True:\n task_template = self.attrs['Spec']['TaskTemplate']\n current_value = int(task_template.get('ForceUpdate', 0))\n kwargs['force_update'] = current_value + 1\n\n create_kwargs = _get_create_service_kwargs('update', kwargs)\n\n return self.client.api.update_service(\n self.id,\n self.version,\n **create_kwargs\n )", "response": "Updates the service s configuration."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef logs(self, **kwargs):\n is_tty = self.attrs['Spec']['TaskTemplate']['ContainerSpec'].get(\n 'TTY', False\n )\n return self.client.api.service_logs(self.id, is_tty=is_tty, **kwargs)", "response": "Get the logs for the service."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nscaling this service container.", "response": "def scale(self, replicas):\n \"\"\"\n Scale service container.\n\n Args:\n replicas (int): The number of containers that should be running.\n\n Returns:\n bool: ``True`` if successful.\n \"\"\"\n\n if 'Global' in self.attrs['Spec']['Mode'].keys():\n raise InvalidArgument('Cannot scale a global container')\n\n service_mode = ServiceMode('replicated', replicas)\n return self.client.api.update_service(self.id, self.version,\n mode=service_mode,\n fetch_current_spec=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new service in the container system.", "response": "def create(self, image, command=None, **kwargs):\n \"\"\"\n Create a service. Similar to the ``docker service create`` command.\n\n Args:\n image (str): The image name to use for the containers.\n command (list of str or str): Command to run.\n args (list of str): Arguments to the command.\n constraints (list of str): :py:class:`~docker.types.Placement`\n constraints.\n preferences (list of tuple): :py:class:`~docker.types.Placement`\n preferences.\n platforms (list of tuple): A list of platform constraints\n expressed as ``(arch, os)`` tuples.\n container_labels (dict): Labels to apply to the container.\n endpoint_spec (EndpointSpec): Properties that can be configured to\n access and load balance a service. Default: ``None``.\n env (list of str): Environment variables, in the form\n ``KEY=val``.\n hostname (string): Hostname to set on the container.\n init (boolean): Run an init inside the container that forwards\n signals and reaps processes\n isolation (string): Isolation technology used by the service's\n containers. Only used for Windows containers.\n labels (dict): Labels to apply to the service.\n log_driver (str): Log driver to use for containers.\n log_driver_options (dict): Log driver options.\n mode (ServiceMode): Scheduling mode for the service.\n Default:``None``\n mounts (list of str): Mounts for the containers, in the form\n ``source:target:options``, where options is either\n ``ro`` or ``rw``.\n name (str): Name to give to the service.\n networks (list of str): List of network names or IDs to attach\n the service to. Default: ``None``.\n resources (Resources): Resource limits and reservations.\n restart_policy (RestartPolicy): Restart policy for containers.\n secrets (list of :py:class:`docker.types.SecretReference`): List\n of secrets accessible to containers for this service.\n stop_grace_period (int): Amount of time to wait for\n containers to terminate before forcefully killing them.\n update_config (UpdateConfig): Specification for the update strategy\n of the service. Default: ``None``\n rollback_config (RollbackConfig): Specification for the rollback\n strategy of the service. Default: ``None``\n user (str): User to run commands as.\n workdir (str): Working directory for commands to run.\n tty (boolean): Whether a pseudo-TTY should be allocated.\n groups (:py:class:`list`): A list of additional groups that the\n container process will run as.\n open_stdin (boolean): Open ``stdin``\n read_only (boolean): Mount the container's root filesystem as read\n only.\n stop_signal (string): Set signal to stop the service's containers\n healthcheck (Healthcheck): Healthcheck\n configuration for this service.\n hosts (:py:class:`dict`): A set of host to IP mappings to add to\n the container's `hosts` file.\n dns_config (DNSConfig): Specification for DNS\n related configurations in resolver configuration file.\n configs (:py:class:`list`): List of :py:class:`ConfigReference`\n that will be exposed to the service.\n privileges (Privileges): Security options for the service's\n containers.\n\n Returns:\n :py:class:`Service`: The created service.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n kwargs['image'] = image\n kwargs['command'] = command\n create_kwargs = _get_create_service_kwargs('create', kwargs)\n service_id = self.client.api.create_service(**create_kwargs)\n return self.get(service_id)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a service s metadata.", "response": "def get(self, service_id, insert_defaults=None):\n \"\"\"\n Get a service.\n\n Args:\n service_id (str): The ID of the service.\n insert_defaults (boolean): If true, default values will be merged\n into the output.\n\n Returns:\n :py:class:`Service`: The service.\n\n Raises:\n :py:class:`docker.errors.NotFound`\n If the service does not exist.\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n :py:class:`docker.errors.InvalidVersion`\n If one of the arguments is not supported with the current\n API version.\n \"\"\"\n return self.prepare_model(\n self.client.api.inspect_service(service_id, insert_defaults)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nlists the services in the cluster.", "response": "def list(self, **kwargs):\n \"\"\"\n List services.\n\n Args:\n filters (dict): Filters to process on the nodes list. Valid\n filters: ``id``, ``name`` , ``label`` and ``mode``.\n Default: ``None``.\n\n Returns:\n list of :py:class:`Service`: The services.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n return [\n self.prepare_model(s)\n for s in self.client.api.services(**kwargs)\n ]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reload(self):\n new_model = self.collection.get(self.id)\n self.attrs = new_model.attrs", "response": "Reload the object from the server and update the attrs attribute with the new data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a model from a set of attributes.", "response": "def prepare_model(self, attrs):\n \"\"\"\n Create a model from a set of attributes.\n \"\"\"\n if isinstance(attrs, Model):\n attrs.client = self.client\n attrs.collection = self\n return attrs\n elif isinstance(attrs, dict):\n return self.model(attrs=attrs, client=self.client, collection=self)\n else:\n raise Exception(\"Can't create %s from %s\" %\n (self.model.__name__, attrs))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of all containers that are connected to the network as a list of container objects.", "response": "def containers(self):\n \"\"\"\n The containers that are connected to the network, as a list of\n :py:class:`~docker.models.containers.Container` objects.\n \"\"\"\n return [\n self.client.containers.get(cid) for cid in\n (self.attrs.get('Containers') or {}).keys()\n ]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef connect(self, container, *args, **kwargs):\n if isinstance(container, Container):\n container = container.id\n return self.client.api.connect_container_to_network(\n container, self.id, *args, **kwargs\n )", "response": "Connects a container to this network."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef disconnect(self, container, *args, **kwargs):\n if isinstance(container, Container):\n container = container.id\n return self.client.api.disconnect_container_from_network(\n container, self.id, *args, **kwargs\n )", "response": "Disconnect a container from this network."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new network in the cluster.", "response": "def create(self, name, *args, **kwargs):\n \"\"\"\n Create a network. Similar to the ``docker network create``.\n\n Args:\n name (str): Name of the network\n driver (str): Name of the driver used to create the network\n options (dict): Driver options as a key-value dictionary\n ipam (IPAMConfig): Optional custom IP scheme for the network.\n check_duplicate (bool): Request daemon to check for networks with\n same name. Default: ``None``.\n internal (bool): Restrict external access to the network. Default\n ``False``.\n labels (dict): Map of labels to set on the network. Default\n ``None``.\n enable_ipv6 (bool): Enable IPv6 on the network. Default ``False``.\n attachable (bool): If enabled, and the network is in the global\n scope, non-service containers on worker nodes will be able to\n connect to the network.\n scope (str): Specify the network's scope (``local``, ``global`` or\n ``swarm``)\n ingress (bool): If set, create an ingress network which provides\n the routing-mesh in swarm mode.\n\n Returns:\n (:py:class:`Network`): The network that was created.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n\n Example:\n A network using the bridge driver:\n\n >>> client.networks.create(\"network1\", driver=\"bridge\")\n\n You can also create more advanced networks with custom IPAM\n configurations. For example, setting the subnet to\n ``192.168.52.0/24`` and gateway address to ``192.168.52.254``.\n\n .. code-block:: python\n\n >>> ipam_pool = docker.types.IPAMPool(\n subnet='192.168.52.0/24',\n gateway='192.168.52.254'\n )\n >>> ipam_config = docker.types.IPAMConfig(\n pool_configs=[ipam_pool]\n )\n >>> client.networks.create(\n \"network1\",\n driver=\"bridge\",\n ipam=ipam_config\n )\n\n \"\"\"\n resp = self.client.api.create_network(name, *args, **kwargs)\n return self.get(resp['Id'])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get(self, network_id, *args, **kwargs):\n return self.prepare_model(\n self.client.api.inspect_network(network_id, *args, **kwargs)\n )", "response": "Get a specific network by its ID."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef list(self, *args, **kwargs):\n greedy = kwargs.pop('greedy', False)\n resp = self.client.api.networks(*args, **kwargs)\n networks = [self.prepare_model(item) for item in resp]\n if greedy and version_gte(self.client.api._version, '1.28'):\n for net in networks:\n net.reload()\n return networks", "response": "List the networks on the server."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing the authentication entries and returns a dictionary of authentication entries.", "response": "def parse_auth(cls, entries, raise_on_error=False):\n \"\"\"\n Parses authentication entries\n\n Args:\n entries: Dict of authentication entries.\n raise_on_error: If set to true, an invalid format will raise\n InvalidConfigFile\n\n Returns:\n Authentication registry.\n \"\"\"\n\n conf = {}\n for registry, entry in six.iteritems(entries):\n if not isinstance(entry, dict):\n log.debug(\n 'Config entry for key {0} is not auth config'.format(\n registry\n )\n )\n # We sometimes fall back to parsing the whole config as if it\n # was the auth config by itself, for legacy purposes. In that\n # case, we fail silently and return an empty conf if any of the\n # keys is not formatted properly.\n if raise_on_error:\n raise errors.InvalidConfigFile(\n 'Invalid configuration for registry {0}'.format(\n registry\n )\n )\n return {}\n if 'identitytoken' in entry:\n log.debug(\n 'Found an IdentityToken entry for registry {0}'.format(\n registry\n )\n )\n conf[registry] = {\n 'IdentityToken': entry['identitytoken']\n }\n continue # Other values are irrelevant if we have a token\n\n if 'auth' not in entry:\n # Starting with engine v1.11 (API 1.23), an empty dictionary is\n # a valid value in the auths config.\n # https://github.com/docker/compose/issues/3265\n log.debug(\n 'Auth data for {0} is absent. Client might be using a '\n 'credentials store instead.'.format(registry)\n )\n conf[registry] = {}\n continue\n\n username, password = decode_auth(entry['auth'])\n log.debug(\n 'Found entry (registry={0}, username={1})'\n .format(repr(registry), repr(username))\n )\n\n conf[registry] = {\n 'username': username,\n 'password': password,\n 'email': entry.get('email'),\n 'serveraddress': registry,\n }\n return conf"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nresolve the authentication configuration for a specific registry.", "response": "def resolve_authconfig(self, registry=None):\n \"\"\"\n Returns the authentication data from the given auth configuration for a\n specific registry. As with the Docker client, legacy entries in the\n config with full URLs are stripped down to hostnames before checking\n for a match. Returns None if no match was found.\n \"\"\"\n\n if self.creds_store or self.cred_helpers:\n store_name = self.get_credential_store(registry)\n if store_name is not None:\n log.debug(\n 'Using credentials store \"{0}\"'.format(store_name)\n )\n cfg = self._resolve_authconfig_credstore(registry, store_name)\n if cfg is not None:\n return cfg\n log.debug('No entry in credstore - fetching from auth dict')\n\n # Default to the public index server\n registry = resolve_index_name(registry) if registry else INDEX_NAME\n log.debug(\"Looking for auth entry for {0}\".format(repr(registry)))\n\n if registry in self.auths:\n log.debug(\"Found {0}\".format(repr(registry)))\n return self.auths[registry]\n\n for key, conf in six.iteritems(self.auths):\n if resolve_index_name(key) == registry:\n log.debug(\"Found {0}\".format(repr(key)))\n return conf\n\n log.debug(\"No entry found\")\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate an exec instance in a running container.", "response": "def exec_create(self, container, cmd, stdout=True, stderr=True,\n stdin=False, tty=False, privileged=False, user='',\n environment=None, workdir=None, detach_keys=None):\n \"\"\"\n Sets up an exec instance in a running container.\n\n Args:\n container (str): Target container where exec instance will be\n created\n cmd (str or list): Command to be executed\n stdout (bool): Attach to stdout. Default: ``True``\n stderr (bool): Attach to stderr. Default: ``True``\n stdin (bool): Attach to stdin. Default: ``False``\n tty (bool): Allocate a pseudo-TTY. Default: False\n privileged (bool): Run as privileged.\n user (str): User to execute command as. Default: root\n environment (dict or list): A dictionary or a list of strings in\n the following format ``[\"PASSWORD=xxx\"]`` or\n ``{\"PASSWORD\": \"xxx\"}``.\n workdir (str): Path to working directory for this exec session\n detach_keys (str): Override the key sequence for detaching\n a container. Format is a single character `[a-Z]`\n or `ctrl-` where `` is one of:\n `a-z`, `@`, `^`, `[`, `,` or `_`.\n ~/.docker/config.json is used by default.\n\n Returns:\n (dict): A dictionary with an exec ``Id`` key.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n\n if environment is not None and utils.version_lt(self._version, '1.25'):\n raise errors.InvalidVersion(\n 'Setting environment for exec is not supported in API < 1.25'\n )\n\n if isinstance(cmd, six.string_types):\n cmd = utils.split_command(cmd)\n\n if isinstance(environment, dict):\n environment = utils.utils.format_environment(environment)\n\n data = {\n 'Container': container,\n 'User': user,\n 'Privileged': privileged,\n 'Tty': tty,\n 'AttachStdin': stdin,\n 'AttachStdout': stdout,\n 'AttachStderr': stderr,\n 'Cmd': cmd,\n 'Env': environment,\n }\n\n if workdir is not None:\n if utils.version_lt(self._version, '1.35'):\n raise errors.InvalidVersion(\n 'workdir is not supported for API version < 1.35'\n )\n data['WorkingDir'] = workdir\n\n if detach_keys:\n data['detachKeys'] = detach_keys\n elif 'detachKeys' in self._general_configs:\n data['detachKeys'] = self._general_configs['detachKeys']\n\n url = self._url('/containers/{0}/exec', container)\n res = self._post_json(url, data=data)\n return self._result(res, True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns low - level information about an exec command.", "response": "def exec_inspect(self, exec_id):\n \"\"\"\n Return low-level information about an exec command.\n\n Args:\n exec_id (str): ID of the exec instance\n\n Returns:\n (dict): Dictionary of values returned by the endpoint.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n if isinstance(exec_id, dict):\n exec_id = exec_id.get('Id')\n res = self._get(self._url(\"/exec/{0}/json\", exec_id))\n return self._result(res, True)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstart a previously set up exec command.", "response": "def exec_start(self, exec_id, detach=False, tty=False, stream=False,\n socket=False, demux=False):\n \"\"\"\n Start a previously set up exec instance.\n\n Args:\n exec_id (str): ID of the exec instance\n detach (bool): If true, detach from the exec command.\n Default: False\n tty (bool): Allocate a pseudo-TTY. Default: False\n stream (bool): Stream response data. Default: False\n socket (bool): Return the connection socket to allow custom\n read/write operations.\n demux (bool): Return stdout and stderr separately\n\n Returns:\n\n (generator or str or tuple): If ``stream=True``, a generator\n yielding response chunks. If ``socket=True``, a socket object for\n the connection. A string containing response data otherwise. If\n ``demux=True``, a tuple with two elements of type byte: stdout and\n stderr.\n\n Raises:\n :py:class:`docker.errors.APIError`\n If the server returns an error.\n \"\"\"\n # we want opened socket if socket == True\n\n data = {\n 'Tty': tty,\n 'Detach': detach\n }\n\n headers = {} if detach else {\n 'Connection': 'Upgrade',\n 'Upgrade': 'tcp'\n }\n\n res = self._post_json(\n self._url('/exec/{0}/start', exec_id),\n headers=headers,\n data=data,\n stream=True\n )\n if detach:\n return self._result(res)\n if socket:\n return self._get_raw_response_socket(res)\n return self._read_from_socket(res, stream, tty=tty, demux=demux)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _raise_for_status(self, response):\n try:\n response.raise_for_status()\n except requests.exceptions.HTTPError as e:\n raise create_api_error_from_http_exception(e)", "response": "Raises stored : class : APIError if one occurred."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstreams result for TTY - enabled container and raw binary data", "response": "def _stream_raw_result(self, response, chunk_size=1, decode=True):\n ''' Stream result for TTY-enabled container and raw binary data'''\n self._raise_for_status(response)\n for out in response.iter_content(chunk_size, decode):\n yield out"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _disable_socket_timeout(self, socket):\n sockets = [socket, getattr(socket, '_sock', None)]\n\n for s in sockets:\n if not hasattr(s, 'settimeout'):\n continue\n\n timeout = -1\n\n if hasattr(s, 'gettimeout'):\n timeout = s.gettimeout()\n\n # Don't change the timeout if it is already disabled.\n if timeout is None or timeout == 0.0:\n continue\n\n s.settimeout(None)", "response": "Disables the socket timeout."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reload_config(self, dockercfg_path=None):\n self._auth_configs = auth.load_config(\n dockercfg_path, credstore_env=self.credstore_env\n )", "response": "Reloads the auth configuration for the current user"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef acquire(self, blocking=None, blocking_timeout=None, token=None):\n sleep = self.sleep\n if token is None:\n token = uuid.uuid1().hex.encode()\n else:\n encoder = self.redis.connection_pool.get_encoder()\n token = encoder.encode(token)\n if blocking is None:\n blocking = self.blocking\n if blocking_timeout is None:\n blocking_timeout = self.blocking_timeout\n stop_trying_at = None\n if blocking_timeout is not None:\n stop_trying_at = mod_time.time() + blocking_timeout\n while True:\n if self.do_acquire(token):\n self.local.token = token\n return True\n if not blocking:\n return False\n if stop_trying_at is not None and mod_time.time() > stop_trying_at:\n return False\n mod_time.sleep(sleep)", "response": "Acquire a shared distributed lock."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef owned(self):\n stored_token = self.redis.get(self.name)\n # need to always compare bytes to bytes\n # TODO: this can be simplified when the context manager is finished\n if stored_token and not isinstance(stored_token, bytes):\n encoder = self.redis.connection_pool.get_encoder()\n stored_token = encoder.encode(stored_token)\n return self.local.token is not None and \\\n stored_token == self.local.token", "response": "Returns True if this key is owned by this lock otherwise False."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nresets a TTL of an already acquired lock back to a timeout value.", "response": "def reacquire(self):\n \"\"\"\n Resets a TTL of an already acquired lock back to a timeout value.\n \"\"\"\n if self.local.token is None:\n raise LockError(\"Cannot reacquire an unlocked lock\")\n if self.timeout is None:\n raise LockError(\"Cannot reacquire a lock with no timeout\")\n return self.do_reacquire()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a unix timestamp to a Python datetime object", "response": "def timestamp_to_datetime(response):\n \"Converts a unix timestamp to a Python datetime object\"\n if not response:\n return None\n try:\n response = int(response)\n except ValueError:\n return None\n return datetime.datetime.fromtimestamp(response)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pairs_to_dict(response, decode_keys=False):\n \"Create a dict given a list of key/value pairs\"\n if response is None:\n return {}\n if decode_keys:\n # the iter form is faster, but I don't know how to make that work\n # with a nativestr() map\n return dict(izip(imap(nativestr, response[::2]), response[1::2]))\n else:\n it = iter(response)\n return dict(izip(it, it))", "response": "Create a dict given a list of key / value pairs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a response to a list of score pairs.", "response": "def zset_score_pairs(response, **options):\n \"\"\"\n If ``withscores`` is specified in the options, return the response as\n a list of (value, score) pairs\n \"\"\"\n if not response or not options.get('withscores'):\n return response\n score_cast_func = options.get('score_cast_func', float)\n it = iter(response)\n return list(izip(it, imap(score_cast_func, it)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsorting the response as a list of n - element tuples with n being the value found in options. groups.", "response": "def sort_return_tuples(response, **options):\n \"\"\"\n If ``groups`` is specified, return the response as a list of\n n-element tuples with n being the value found in options['groups']\n \"\"\"\n if not response or not options.get('groups'):\n return response\n n = options['groups']\n return list(izip(*[response[i::n] for i in range(n)]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef from_url(cls, url, db=None, **kwargs):\n connection_pool = ConnectionPool.from_url(url, db=db, **kwargs)\n return cls(connection_pool=connection_pool)", "response": "Returns a Redis client object configured from the given URL."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef lock(self, name, timeout=None, sleep=0.1, blocking_timeout=None,\n lock_class=None, thread_local=True):\n \"\"\"\n Return a new Lock object using key ``name`` that mimics\n the behavior of threading.Lock.\n\n If specified, ``timeout`` indicates a maximum life for the lock.\n By default, it will remain locked until release() is called.\n\n ``sleep`` indicates the amount of time to sleep per loop iteration\n when the lock is in blocking mode and another client is currently\n holding the lock.\n\n ``blocking_timeout`` indicates the maximum amount of time in seconds to\n spend trying to acquire the lock. A value of ``None`` indicates\n continue trying forever. ``blocking_timeout`` can be specified as a\n float or integer, both representing the number of seconds to wait.\n\n ``lock_class`` forces the specified lock implementation.\n\n ``thread_local`` indicates whether the lock token is placed in\n thread-local storage. By default, the token is placed in thread local\n storage so that a thread only sees its token, not a token set by\n another thread. Consider the following timeline:\n\n time: 0, thread-1 acquires `my-lock`, with a timeout of 5 seconds.\n thread-1 sets the token to \"abc\"\n time: 1, thread-2 blocks trying to acquire `my-lock` using the\n Lock instance.\n time: 5, thread-1 has not yet completed. redis expires the lock\n key.\n time: 5, thread-2 acquired `my-lock` now that it's available.\n thread-2 sets the token to \"xyz\"\n time: 6, thread-1 finishes its work and calls release(). if the\n token is *not* stored in thread local storage, then\n thread-1 would see the token value as \"xyz\" and would be\n able to successfully release the thread-2's lock.\n\n In some use cases it's necessary to disable thread local storage. For\n example, if you have code where one thread acquires a lock and passes\n that lock instance to a worker thread to release later. If thread\n local storage isn't disabled in this case, the worker thread won't see\n the token set by the thread that acquired the lock. Our assumption\n is that these cases aren't common and as such default to using\n thread local storage. \"\"\"\n if lock_class is None:\n lock_class = Lock\n return lock_class(self, name, timeout=timeout, sleep=sleep,\n blocking_timeout=blocking_timeout,\n thread_local=thread_local)", "response": "This method returns a new Lock object with the specified key name and timeout and sleep."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute a command and return a parsed response", "response": "def execute_command(self, *args, **options):\n \"Execute a command and return a parsed response\"\n pool = self.connection_pool\n command_name = args[0]\n connection = pool.get_connection(command_name, **options)\n try:\n connection.send_command(*args)\n return self.parse_response(connection, command_name, **options)\n except (ConnectionError, TimeoutError) as e:\n connection.disconnect()\n if not (connection.retry_on_timeout and\n isinstance(e, TimeoutError)):\n raise\n connection.send_command(*args)\n return self.parse_response(connection, command_name, **options)\n finally:\n pool.release(connection)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_response(self, connection, command_name, **options):\n \"Parses a response from the Redis server\"\n try:\n response = connection.read_response()\n except ResponseError:\n if EMPTY_RESPONSE in options:\n return options[EMPTY_RESPONSE]\n raise\n if command_name in self.response_callbacks:\n return self.response_callbacks[command_name](response, **options)\n return response", "response": "Parses a response from the Redis server"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef client_kill_filter(self, _id=None, _type=None, addr=None, skipme=None):\n args = []\n if _type is not None:\n client_types = ('normal', 'master', 'slave', 'pubsub')\n if str(_type).lower() not in client_types:\n raise DataError(\"CLIENT KILL type must be one of %r\" % (\n client_types,))\n args.extend((Token.get_token('TYPE'), _type))\n if skipme is not None:\n if not isinstance(skipme, bool):\n raise DataError(\"CLIENT KILL skipme must be a bool\")\n if skipme:\n args.extend((Token.get_token('SKIPME'),\n Token.get_token('YES')))\n else:\n args.extend((Token.get_token('SKIPME'),\n Token.get_token('NO')))\n if _id is not None:\n args.extend((Token.get_token('ID'), _id))\n if addr is not None:\n args.extend((Token.get_token('ADDR'), addr))\n if not args:\n raise DataError(\"CLIENT KILL ... ... \"\n \" must specify at least one filter\")\n return self.execute_command('CLIENT KILL', *args)", "response": "Kills a client by a variety of filter options."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of currently connected clients.", "response": "def client_list(self, _type=None):\n \"\"\"\n Returns a list of currently connected clients.\n If type of client specified, only that type will be returned.\n :param _type: optional. one of the client types (normal, master,\n replica, pubsub)\n \"\"\"\n \"Returns a list of currently connected clients\"\n if _type is not None:\n client_types = ('normal', 'master', 'replica', 'pubsub')\n if str(_type).lower() not in client_types:\n raise DataError(\"CLIENT LIST _type must be one of %r\" % (\n client_types,))\n return self.execute_command('CLIENT LIST', Token.get_token('TYPE'),\n _type)\n return self.execute_command('CLIENT LIST')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef client_unblock(self, client_id, error=False):\n args = ['CLIENT UNBLOCK', int(client_id)]\n if error:\n args.append(Token.get_token('ERROR'))\n return self.execute_command(*args)", "response": "Unblocks a connection by its client id."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef client_pause(self, timeout):\n if not isinstance(timeout, (int, long)):\n raise DataError(\"CLIENT PAUSE timeout must be an integer\")\n return self.execute_command('CLIENT PAUSE', str(timeout))", "response": "Suspend all the Redis clients for the specified amount of time."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flushall(self, asynchronous=False):\n args = []\n if asynchronous:\n args.append(Token.get_token('ASYNC'))\n return self.execute_command('FLUSHALL', *args)", "response": "Delete all keys in all databases on the current host."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef migrate(self, host, port, keys, destination_db, timeout,\n copy=False, replace=False, auth=None):\n \"\"\"\n Migrate 1 or more keys from the current Redis server to a different\n server specified by the ``host``, ``port`` and ``destination_db``.\n\n The ``timeout``, specified in milliseconds, indicates the maximum\n time the connection between the two servers can be idle before the\n command is interrupted.\n\n If ``copy`` is True, the specified ``keys`` are NOT deleted from\n the source server.\n\n If ``replace`` is True, this operation will overwrite the keys\n on the destination server if they exist.\n\n If ``auth`` is specified, authenticate to the destination server with\n the password provided.\n \"\"\"\n keys = list_or_args(keys, [])\n if not keys:\n raise DataError('MIGRATE requires at least one key')\n pieces = []\n if copy:\n pieces.append(Token.get_token('COPY'))\n if replace:\n pieces.append(Token.get_token('REPLACE'))\n if auth:\n pieces.append(Token.get_token('AUTH'))\n pieces.append(auth)\n pieces.append(Token.get_token('KEYS'))\n pieces.extend(keys)\n return self.execute_command('MIGRATE', host, port, '', destination_db,\n timeout, *pieces)", "response": "Migrate one or more keys from the current Redis server to a different Redis server."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef object(self, infotype, key):\n \"Return the encoding, idletime, or refcount about the key\"\n return self.execute_command('OBJECT', infotype, key, infotype=infotype)", "response": "Return the encoding idletime or refcount about the key"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the total memory usage for the specified key and its value and associated administrative overheads.", "response": "def memory_usage(self, key, samples=None):\n \"\"\"\n Return the total memory usage for key, its value and associated\n administrative overheads.\n\n For nested data structures, ``samples`` is the number of elements to\n sample. If left unspecified, the server's default is 5. Use 0 to sample\n all elements.\n \"\"\"\n args = []\n if isinstance(samples, int):\n args.extend([Token.get_token('SAMPLES'), samples])\n return self.execute_command('MEMORY USAGE', key, *args)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef shutdown(self, save=False, nosave=False):\n if save and nosave:\n raise DataError('SHUTDOWN save and nosave cannot both be set')\n args = ['SHUTDOWN']\n if save:\n args.append('SAVE')\n if nosave:\n args.append('NOSAVE')\n try:\n self.execute_command(*args)\n except ConnectionError:\n # a ConnectionError here is expected\n return\n raise RedisError(\"SHUTDOWN seems to have failed.\")", "response": "Shutdown the Redis server."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef slaveof(self, host=None, port=None):\n if host is None and port is None:\n return self.execute_command('SLAVEOF', Token.get_token('NO'),\n Token.get_token('ONE'))\n return self.execute_command('SLAVEOF', host, port)", "response": "Set the server to be a replicated slave of the instance identified by the host and port."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nperforms a bitwise operation between keys and store the result in dest.", "response": "def bitop(self, operation, dest, *keys):\n \"\"\"\n Perform a bitwise operation using ``operation`` between ``keys`` and\n store the result in ``dest``.\n \"\"\"\n return self.execute_command('BITOP', operation, dest, *keys)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the position of the first set bit in a string.", "response": "def bitpos(self, key, bit, start=None, end=None):\n \"\"\"\n Return the position of the first bit set to 1 or 0 in a string.\n ``start`` and ``end`` difines search range. The range is interpreted\n as a range of bytes and not a range of bits, so start=0 and end=2\n means to look at the first three bytes.\n \"\"\"\n if bit not in (0, 1):\n raise DataError('bit must be 0 or 1')\n params = [key, bit]\n\n start is not None and params.append(start)\n\n if start is not None and end is not None:\n params.append(end)\n elif start is None and end is not None:\n raise DataError(\"start argument is not set, \"\n \"when end is specified\")\n return self.execute_command('BITPOS', *params)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef expireat(self, name, when):\n if isinstance(when, datetime.datetime):\n when = int(mod_time.mktime(when.timetuple()))\n return self.execute_command('EXPIREAT', name, when)", "response": "Set an expire flag on key name. When can be represented as an integer indicating unix time. When can be represented as an integer indicating unix time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a list of values ordered identically to keys", "response": "def mget(self, keys, *args):\n \"\"\"\n Returns a list of values ordered identically to ``keys``\n \"\"\"\n args = list_or_args(keys, args)\n options = {}\n if not args:\n options[EMPTY_RESPONSE] = []\n return self.execute_command('MGET', *args, **options)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef msetnx(self, mapping):\n items = []\n for pair in iteritems(mapping):\n items.extend(pair)\n return self.execute_command('MSETNX', *items)", "response": "Sets the key - value pairs for the specified key - value pairs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset an expire flag on key name for time milliseconds.", "response": "def pexpire(self, name, time):\n \"\"\"\n Set an expire flag on key ``name`` for ``time`` milliseconds.\n ``time`` can be represented by an integer or a Python timedelta\n object.\n \"\"\"\n if isinstance(time, datetime.timedelta):\n time = int(time.total_seconds() * 1000)\n return self.execute_command('PEXPIRE', name, time)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pexpireat(self, name, when):\n if isinstance(when, datetime.datetime):\n ms = int(when.microsecond / 1000)\n when = int(mod_time.mktime(when.timetuple())) * 1000 + ms\n return self.execute_command('PEXPIREAT', name, when)", "response": "Set an expire flag on key name when time. When can be represented as an integer representing unix time in milliseconds."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the value of key name to value that expires in time_ms milliseconds.", "response": "def psetex(self, name, time_ms, value):\n \"\"\"\n Set the value of key ``name`` to ``value`` that expires in ``time_ms``\n milliseconds. ``time_ms`` can be represented by an integer or a Python\n timedelta object\n \"\"\"\n if isinstance(time_ms, datetime.timedelta):\n time_ms = int(time_ms.total_seconds() * 1000)\n return self.execute_command('PSETEX', name, time_ms, value)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef restore(self, name, ttl, value, replace=False):\n params = [name, ttl, value]\n if replace:\n params.append('REPLACE')\n return self.execute_command('RESTORE', *params)", "response": "Restores a key from the given serialized value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set(self, name, value, ex=None, px=None, nx=False, xx=False):\n pieces = [name, value]\n if ex is not None:\n pieces.append('EX')\n if isinstance(ex, datetime.timedelta):\n ex = int(ex.total_seconds())\n pieces.append(ex)\n if px is not None:\n pieces.append('PX')\n if isinstance(px, datetime.timedelta):\n px = int(px.total_seconds() * 1000)\n pieces.append(px)\n\n if nx:\n pieces.append('NX')\n if xx:\n pieces.append('XX')\n return self.execute_command('SET', *pieces)", "response": "Set the value at key name to value."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef setbit(self, name, offset, value):\n value = value and 1 or 0\n return self.execute_command('SETBIT', name, offset, value)", "response": "Set the value of the bit at offset in name to value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setex(self, name, time, value):\n if isinstance(time, datetime.timedelta):\n time = int(time.total_seconds())\n return self.execute_command('SETEX', name, time, value)", "response": "Set the value of key name to value that expires in time."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the value of the key name to value starting at offset.", "response": "def setrange(self, name, offset, value):\n \"\"\"\n Overwrite bytes in the value of ``name`` starting at ``offset`` with\n ``value``. If ``offset`` plus the length of ``value`` exceeds the\n length of the original value, the new value will be larger than before.\n If ``offset`` exceeds the length of the original value, null bytes\n will be used to pad between the end of the previous value and the start\n of what's being injected.\n\n Returns the length of the new string.\n \"\"\"\n return self.execute_command('SETRANGE', name, offset, value)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\npopping and returns the first element of the list named in keys. If timeout is not specified then block indefinitely for timeout seconds.", "response": "def blpop(self, keys, timeout=0):\n \"\"\"\n LPOP a value off of the first non-empty list\n named in the ``keys`` list.\n\n If none of the lists in ``keys`` has a value to LPOP, then block\n for ``timeout`` seconds, or until a value gets pushed on to one\n of the lists.\n\n If timeout is 0, then block indefinitely.\n \"\"\"\n if timeout is None:\n timeout = 0\n keys = list_or_args(keys, None)\n keys.append(timeout)\n return self.execute_command('BLPOP', *keys)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npops and returns the first element of the list named in keys. If timeout is not specified then block indefinitely for timeout seconds.", "response": "def brpop(self, keys, timeout=0):\n \"\"\"\n RPOP a value off of the first non-empty list\n named in the ``keys`` list.\n\n If none of the lists in ``keys`` has a value to RPOP, then block\n for ``timeout`` seconds, or until a value gets pushed on to one\n of the lists.\n\n If timeout is 0, then block indefinitely.\n \"\"\"\n if timeout is None:\n timeout = 0\n keys = list_or_args(keys, None)\n keys.append(timeout)\n return self.execute_command('BRPOP', *keys)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lrange(self, name, start, end):\n return self.execute_command('LRANGE', name, start, end)", "response": "Return a slice of the list name between start and end"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lrem(self, name, count, value):\n return self.execute_command('LREM', name, count, value)", "response": "Remove the first count occurrences of elements equal to value from the list stored at name."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset position of list name to value", "response": "def lset(self, name, index, value):\n \"Set ``position`` of list ``name`` to ``value``\"\n return self.execute_command('LSET', name, index, value)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsorting and return the list set or sorted set at name.", "response": "def sort(self, name, start=None, num=None, by=None, get=None,\n desc=False, alpha=False, store=None, groups=False):\n \"\"\"\n Sort and return the list, set or sorted set at ``name``.\n\n ``start`` and ``num`` allow for paging through the sorted data\n\n ``by`` allows using an external key to weight and sort the items.\n Use an \"*\" to indicate where in the key the item value is located\n\n ``get`` allows for returning items from external keys rather than the\n sorted data itself. Use an \"*\" to indicate where int he key\n the item value is located\n\n ``desc`` allows for reversing the sort\n\n ``alpha`` allows for sorting lexicographically rather than numerically\n\n ``store`` allows for storing the result of the sort into\n the key ``store``\n\n ``groups`` if set to True and if ``get`` contains at least two\n elements, sort will return a list of tuples, each containing the\n values fetched from the arguments to ``get``.\n\n \"\"\"\n if (start is not None and num is None) or \\\n (num is not None and start is None):\n raise DataError(\"``start`` and ``num`` must both be specified\")\n\n pieces = [name]\n if by is not None:\n pieces.append(Token.get_token('BY'))\n pieces.append(by)\n if start is not None and num is not None:\n pieces.append(Token.get_token('LIMIT'))\n pieces.append(start)\n pieces.append(num)\n if get is not None:\n # If get is a string assume we want to get a single value.\n # Otherwise assume it's an interable and we want to get multiple\n # values. We can't just iterate blindly because strings are\n # iterable.\n if isinstance(get, (bytes, basestring)):\n pieces.append(Token.get_token('GET'))\n pieces.append(get)\n else:\n for g in get:\n pieces.append(Token.get_token('GET'))\n pieces.append(g)\n if desc:\n pieces.append(Token.get_token('DESC'))\n if alpha:\n pieces.append(Token.get_token('ALPHA'))\n if store is not None:\n pieces.append(Token.get_token('STORE'))\n pieces.append(store)\n\n if groups:\n if not get or isinstance(get, (bytes, basestring)) or len(get) < 2:\n raise DataError('when using \"groups\" the \"get\" argument '\n 'must be specified and contain at least '\n 'two keys')\n\n options = {'groups': len(get) if groups else None}\n return self.execute_command('SORT', *pieces, **options)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning an iterator over the items in the cache for the specified pattern and number of returns.", "response": "def scan_iter(self, match=None, count=None):\n \"\"\"\n Make an iterator using the SCAN command so that the client doesn't\n need to remember the cursor position.\n\n ``match`` allows for filtering the keys by pattern\n\n ``count`` allows for hint the minimum number of returns\n \"\"\"\n cursor = '0'\n while cursor != 0:\n cursor, data = self.scan(cursor=cursor, match=match, count=count)\n for item in data:\n yield item"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef smove(self, src, dst, value):\n \"Move ``value`` from set ``src`` to set ``dst`` atomically\"\n return self.execute_command('SMOVE', src, dst, value)", "response": "Move value from set src to set dst atomically"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef spop(self, name, count=None):\n \"Remove and return a random member of set ``name``\"\n args = (count is not None) and [count] or []\n return self.execute_command('SPOP', name, *args)", "response": "Remove and return a random member of set name"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nacknowledging the successful processing of one or more messages.", "response": "def xack(self, name, groupname, *ids):\n \"\"\"\n Acknowledges the successful processing of one or more messages.\n name: name of the stream.\n groupname: name of the consumer group.\n *ids: message ids to acknowlege.\n \"\"\"\n return self.execute_command('XACK', name, groupname, *ids)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef xadd(self, name, fields, id='*', maxlen=None, approximate=True):\n pieces = []\n if maxlen is not None:\n if not isinstance(maxlen, (int, long)) or maxlen < 1:\n raise DataError('XADD maxlen must be a positive integer')\n pieces.append(Token.get_token('MAXLEN'))\n if approximate:\n pieces.append(Token.get_token('~'))\n pieces.append(str(maxlen))\n pieces.append(id)\n if not isinstance(fields, dict) or len(fields) == 0:\n raise DataError('XADD fields must be a non-empty dict')\n for pair in iteritems(fields):\n pieces.extend(pair)\n return self.execute_command('XADD', name, *pieces)", "response": "Add a new entry to a stream."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef xclaim(self, name, groupname, consumername, min_idle_time, message_ids,\n idle=None, time=None, retrycount=None, force=False,\n justid=False):\n \"\"\"\n Changes the ownership of a pending message.\n name: name of the stream.\n groupname: name of the consumer group.\n consumername: name of a consumer that claims the message.\n min_idle_time: filter messages that were idle less than this amount of\n milliseconds\n message_ids: non-empty list or tuple of message IDs to claim\n idle: optional. Set the idle time (last time it was delivered) of the\n message in ms\n time: optional integer. This is the same as idle but instead of a\n relative amount of milliseconds, it sets the idle time to a specific\n Unix time (in milliseconds).\n retrycount: optional integer. set the retry counter to the specified\n value. This counter is incremented every time a message is delivered\n again.\n force: optional boolean, false by default. Creates the pending message\n entry in the PEL even if certain specified IDs are not already in the\n PEL assigned to a different client.\n justid: optional boolean, false by default. Return just an array of IDs\n of messages successfully claimed, without returning the actual message\n \"\"\"\n if not isinstance(min_idle_time, (int, long)) or min_idle_time < 0:\n raise DataError(\"XCLAIM min_idle_time must be a non negative \"\n \"integer\")\n if not isinstance(message_ids, (list, tuple)) or not message_ids:\n raise DataError(\"XCLAIM message_ids must be a non empty list or \"\n \"tuple of message IDs to claim\")\n\n kwargs = {}\n pieces = [name, groupname, consumername, str(min_idle_time)]\n pieces.extend(list(message_ids))\n\n if idle is not None:\n if not isinstance(idle, (int, long)):\n raise DataError(\"XCLAIM idle must be an integer\")\n pieces.extend((Token.get_token('IDLE'), str(idle)))\n if time is not None:\n if not isinstance(time, (int, long)):\n raise DataError(\"XCLAIM time must be an integer\")\n pieces.extend((Token.get_token('TIME'), str(time)))\n if retrycount is not None:\n if not isinstance(retrycount, (int, long)):\n raise DataError(\"XCLAIM retrycount must be an integer\")\n pieces.extend((Token.get_token('RETRYCOUNT'), str(retrycount)))\n\n if force:\n if not isinstance(force, bool):\n raise DataError(\"XCLAIM force must be a boolean\")\n pieces.append(Token.get_token('FORCE'))\n if justid:\n if not isinstance(justid, bool):\n raise DataError(\"XCLAIM justid must be a boolean\")\n pieces.append(Token.get_token('JUSTID'))\n kwargs['parse_justid'] = True\n return self.execute_command('XCLAIM', *pieces, **kwargs)", "response": "This method is used to claim a pending message in a consumer group."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a new consumer group associated with a stream.", "response": "def xgroup_create(self, name, groupname, id='$', mkstream=False):\n \"\"\"\n Create a new consumer group associated with a stream.\n name: name of the stream.\n groupname: name of the consumer group.\n id: ID of the last item in the stream to consider already delivered.\n \"\"\"\n pieces = ['XGROUP CREATE', name, groupname, id]\n if mkstream:\n pieces.append(Token.get_token('MKSTREAM'))\n return self.execute_command(*pieces)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef xgroup_delconsumer(self, name, groupname, consumername):\n return self.execute_command('XGROUP DELCONSUMER', name, groupname,\n consumername)", "response": "Remove a specific consumer from a consumer group."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the last item in the consumer group to something else.", "response": "def xgroup_setid(self, name, groupname, id):\n \"\"\"\n Set the consumer group last delivered ID to something else.\n name: name of the stream.\n groupname: name of the consumer group.\n id: ID of the last item in the stream to consider already delivered.\n \"\"\"\n return self.execute_command('XGROUP SETID', name, groupname, id)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning information about pending messages in a range.", "response": "def xpending_range(self, name, groupname, min, max, count,\n consumername=None):\n \"\"\"\n Returns information about pending messages, in a range.\n name: name of the stream.\n groupname: name of the consumer group.\n min: minimum stream ID.\n max: maximum stream ID.\n count: number of messages to return\n consumername: name of a consumer to filter by (optional).\n \"\"\"\n pieces = [name, groupname]\n if min is not None or max is not None or count is not None:\n if min is None or max is None or count is None:\n raise DataError(\"XPENDING must be provided with min, max \"\n \"and count parameters, or none of them. \")\n if not isinstance(count, (int, long)) or count < -1:\n raise DataError(\"XPENDING count must be a integer >= -1\")\n pieces.extend((min, max, str(count)))\n if consumername is not None:\n if min is None or max is None or count is None:\n raise DataError(\"if XPENDING is provided with consumername,\"\n \" it must be provided with min, max and\"\n \" count parameters\")\n pieces.append(consumername)\n return self.execute_command('XPENDING', *pieces, parse_detail=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading stream values within an interval.", "response": "def xrange(self, name, min='-', max='+', count=None):\n \"\"\"\n Read stream values within an interval.\n name: name of the stream.\n start: first stream ID. defaults to '-',\n meaning the earliest available.\n finish: last stream ID. defaults to '+',\n meaning the latest available.\n count: if set, only return this many items, beginning with the\n earliest available.\n \"\"\"\n pieces = [min, max]\n if count is not None:\n if not isinstance(count, (int, long)) or count < 1:\n raise DataError('XRANGE count must be a positive integer')\n pieces.append(Token.get_token('COUNT'))\n pieces.append(str(count))\n\n return self.execute_command('XRANGE', name, *pieces)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xread(self, streams, count=None, block=None):\n pieces = []\n if block is not None:\n if not isinstance(block, (int, long)) or block < 0:\n raise DataError('XREAD block must be a non-negative integer')\n pieces.append(Token.get_token('BLOCK'))\n pieces.append(str(block))\n if count is not None:\n if not isinstance(count, (int, long)) or count < 1:\n raise DataError('XREAD count must be a positive integer')\n pieces.append(Token.get_token('COUNT'))\n pieces.append(str(count))\n if not isinstance(streams, dict) or len(streams) == 0:\n raise DataError('XREAD streams must be a non empty dict')\n pieces.append(Token.get_token('STREAMS'))\n keys, values = izip(*iteritems(streams))\n pieces.extend(keys)\n pieces.extend(values)\n return self.execute_command('XREAD', *pieces)", "response": "Read many items from a set of streams."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads from a consumer group via a consumer group.", "response": "def xreadgroup(self, groupname, consumername, streams, count=None,\n block=None, noack=False):\n \"\"\"\n Read from a stream via a consumer group.\n groupname: name of the consumer group.\n consumername: name of the requesting consumer.\n streams: a dict of stream names to stream IDs, where\n IDs indicate the last ID already seen.\n count: if set, only return this many items, beginning with the\n earliest available.\n block: number of milliseconds to wait, if nothing already present.\n noack: do not add messages to the PEL\n \"\"\"\n pieces = [Token.get_token('GROUP'), groupname, consumername]\n if count is not None:\n if not isinstance(count, (int, long)) or count < 1:\n raise DataError(\"XREADGROUP count must be a positive integer\")\n pieces.append(Token.get_token(\"COUNT\"))\n pieces.append(str(count))\n if block is not None:\n if not isinstance(block, (int, long)) or block < 0:\n raise DataError(\"XREADGROUP block must be a non-negative \"\n \"integer\")\n pieces.append(Token.get_token(\"BLOCK\"))\n pieces.append(str(block))\n if noack:\n pieces.append(Token.get_token(\"NOACK\"))\n if not isinstance(streams, dict) or len(streams) == 0:\n raise DataError('XREADGROUP streams must be a non empty dict')\n pieces.append(Token.get_token('STREAMS'))\n pieces.extend(streams.keys())\n pieces.extend(streams.values())\n return self.execute_command('XREADGROUP', *pieces)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntrimming old messages from a stream.", "response": "def xtrim(self, name, maxlen, approximate=True):\n \"\"\"\n Trims old messages from a stream.\n name: name of the stream.\n maxlen: truncate old stream messages beyond this size\n approximate: actual stream length may be slightly more than maxlen\n \"\"\"\n pieces = [Token.get_token('MAXLEN')]\n if approximate:\n pieces.append(Token.get_token('~'))\n pieces.append(maxlen)\n return self.execute_command('XTRIM', name, *pieces)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds any number of elements to the specified key.", "response": "def zadd(self, name, mapping, nx=False, xx=False, ch=False, incr=False):\n \"\"\"\n Set any number of element-name, score pairs to the key ``name``. Pairs\n are specified as a dict of element-names keys to score values.\n\n ``nx`` forces ZADD to only create new elements and not to update\n scores for elements that already exist.\n\n ``xx`` forces ZADD to only update scores of elements that already\n exist. New elements will not be added.\n\n ``ch`` modifies the return value to be the numbers of elements changed.\n Changed elements include new elements that were added and elements\n whose scores changed.\n\n ``incr`` modifies ZADD to behave like ZINCRBY. In this mode only a\n single element/score pair can be specified and the score is the amount\n the existing score will be incremented by. When using this mode the\n return value of ZADD will be the new score of the element.\n\n The return value of ZADD varies based on the mode specified. With no\n options, ZADD returns the number of new elements added to the sorted\n set.\n \"\"\"\n if not mapping:\n raise DataError(\"ZADD requires at least one element/score pair\")\n if nx and xx:\n raise DataError(\"ZADD allows either 'nx' or 'xx', not both\")\n if incr and len(mapping) != 1:\n raise DataError(\"ZADD option 'incr' only works when passing a \"\n \"single element/score pair\")\n pieces = []\n options = {}\n if nx:\n pieces.append(Token.get_token('NX'))\n if xx:\n pieces.append(Token.get_token('XX'))\n if ch:\n pieces.append(Token.get_token('CH'))\n if incr:\n pieces.append(Token.get_token('INCR'))\n options['as_score'] = True\n for pair in iteritems(mapping):\n pieces.append(pair[1])\n pieces.append(pair[0])\n return self.execute_command('ZADD', name, *pieces, **options)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the number of elements in the sorted set at key name with a score between min and max.", "response": "def zcount(self, name, min, max):\n \"\"\"\n Returns the number of elements in the sorted set at key ``name`` with\n a score between ``min`` and ``max``.\n \"\"\"\n return self.execute_command('ZCOUNT', name, min, max)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nincrements the score of value in sorted set name by amount", "response": "def zincrby(self, name, amount, value):\n \"Increment the score of ``value`` in sorted set ``name`` by ``amount``\"\n return self.execute_command('ZINCRBY', name, amount, value)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nintersecting multiple sorted sets specified by keys into a new sorted set dest.", "response": "def zinterstore(self, dest, keys, aggregate=None):\n \"\"\"\n Intersect multiple sorted sets specified by ``keys`` into\n a new sorted set, ``dest``. Scores in the destination will be\n aggregated based on the ``aggregate``, or SUM if none is provided.\n \"\"\"\n return self._zaggregate('ZINTERSTORE', dest, keys, aggregate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the number of items in the sorted set name between the lexicographical range min and max.", "response": "def zlexcount(self, name, min, max):\n \"\"\"\n Return the number of items in the sorted set ``name`` between the\n lexicographical range ``min`` and ``max``.\n \"\"\"\n return self.execute_command('ZLEXCOUNT', name, min, max)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef zpopmax(self, name, count=None):\n args = (count is not None) and [count] or []\n options = {\n 'withscores': True\n }\n return self.execute_command('ZPOPMAX', name, *args, **options)", "response": "Remove and return up to count members with the highest scores\n from the sorted set name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef zpopmin(self, name, count=None):\n args = (count is not None) and [count] or []\n options = {\n 'withscores': True\n }\n return self.execute_command('ZPOPMIN', name, *args, **options)", "response": "Remove and return up to count members with the lowest scores\n from the sorted set name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bzpopmax(self, keys, timeout=0):\n if timeout is None:\n timeout = 0\n keys = list_or_args(keys, None)\n keys.append(timeout)\n return self.execute_command('BZPOPMAX', *keys)", "response": "Remove and return the highest priority set from the sorted set at the given keys. If timeout is not set then block indefinitely until a member gets added to the first non - empty set at the given keys. If timeout is set then block indefinitely until a member gets added to the first non - empty set at the given keys."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef zrangebylex(self, name, min, max, start=None, num=None):\n if (start is not None and num is None) or \\\n (num is not None and start is None):\n raise DataError(\"``start`` and ``num`` must both be specified\")\n pieces = ['ZRANGEBYLEX', name, min, max]\n if start is not None and num is not None:\n pieces.extend([Token.get_token('LIMIT'), start, num])\n return self.execute_command(*pieces)", "response": "Return the lexicographical range of values from sorted set name between min and max."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving all elements in the sorted set name between the lexicographical range specified by min and max.", "response": "def zremrangebylex(self, name, min, max):\n \"\"\"\n Remove all elements in the sorted set ``name`` between the\n lexicographical range specified by ``min`` and ``max``.\n\n Returns the number of elements removed.\n \"\"\"\n return self.execute_command('ZREMRANGEBYLEX', name, min, max)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef zremrangebyrank(self, name, min, max):\n return self.execute_command('ZREMRANGEBYRANK', name, min, max)", "response": "Remove all elements in the sorted set name with ranks between min and max."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a range of values from the sorted set name with scores between min and max in descending order.", "response": "def zrevrangebyscore(self, name, max, min, start=None, num=None,\n withscores=False, score_cast_func=float):\n \"\"\"\n Return a range of values from the sorted set ``name`` with scores\n between ``min`` and ``max`` in descending order.\n\n If ``start`` and ``num`` are specified, then return a slice\n of the range.\n\n ``withscores`` indicates to return the scores along with the values.\n The return type is a list of (value, score) pairs\n\n ``score_cast_func`` a callable used to cast the score return value\n \"\"\"\n if (start is not None and num is None) or \\\n (num is not None and start is None):\n raise DataError(\"``start`` and ``num`` must both be specified\")\n pieces = ['ZREVRANGEBYSCORE', name, max, min]\n if start is not None and num is not None:\n pieces.extend([Token.get_token('LIMIT'), start, num])\n if withscores:\n pieces.append(Token.get_token('WITHSCORES'))\n options = {\n 'withscores': withscores,\n 'score_cast_func': score_cast_func\n }\n return self.execute_command(*pieces, **options)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef zunionstore(self, dest, keys, aggregate=None):\n return self._zaggregate('ZUNIONSTORE', dest, keys, aggregate)", "response": "Union multiple sorted sets specified by keys into a new sorted set dest."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef hmget(self, name, keys, *args):\n \"Returns a list of values ordered identically to ``keys``\"\n args = list_or_args(keys, args)\n return self.execute_command('HMGET', name, *args)", "response": "Returns a list of values ordered identically to keys"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a Lua script and return the result.", "response": "def eval(self, script, numkeys, *keys_and_args):\n \"\"\"\n Execute the Lua ``script``, specifying the ``numkeys`` the script\n will touch and the key names and argument values in ``keys_and_args``.\n Returns the result of the script.\n\n In practice, use the object returned by ``register_script``. This\n function exists purely for Redis API completion.\n \"\"\"\n return self.execute_command('EVAL', script, numkeys, *keys_and_args)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef evalsha(self, sha, numkeys, *keys_and_args):\n return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)", "response": "Execute a Lua script with the given sha."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef geoadd(self, name, *values):\n if len(values) % 3 != 0:\n raise DataError(\"GEOADD requires places with lon, lat and name\"\n \" values\")\n return self.execute_command('GEOADD', name, *values)", "response": "Add the specified geospatial items to the specified key identified by the name argument."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef geodist(self, name, place1, place2, unit=None):\n pieces = [name, place1, place2]\n if unit and unit not in ('m', 'km', 'mi', 'ft'):\n raise DataError(\"GEODIST invalid unit\")\n elif unit:\n pieces.append(unit)\n return self.execute_command('GEODIST', *pieces)", "response": "Return the distance between place1 and place2 members of the key name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the response from a publish / subscribe command", "response": "def parse_response(self, block=True, timeout=0):\n \"Parse the response from a publish/subscribe command\"\n connection = self.connection\n if connection is None:\n raise RuntimeError(\n 'pubsub connection not set: '\n 'did you forget to call subscribe() or psubscribe()?')\n if not block and not connection.can_read(timeout=timeout):\n return None\n return self._execute(connection, connection.read_response)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nnormalizes channel names to be either bytes or strings based on whether responses are automatically decoded.", "response": "def _normalize_keys(self, data):\n \"\"\"\n normalize channel/pattern names to be either bytes or strings\n based on whether responses are automatically decoded. this saves us\n from coercing the value for each message coming in.\n \"\"\"\n encode = self.encoder.encode\n decode = self.encoder.decode\n return {decode(encode(k)): v for k, v in iteritems(data)}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsubscribe to channel patterns.", "response": "def psubscribe(self, *args, **kwargs):\n \"\"\"\n Subscribe to channel patterns. Patterns supplied as keyword arguments\n expect a pattern name as the key and a callable as the value. A\n pattern's callable will be invoked automatically when a message is\n received on that pattern rather than producing a message via\n ``listen()``.\n \"\"\"\n if args:\n args = list_or_args(args[0], args[1:])\n new_patterns = dict.fromkeys(args)\n new_patterns.update(kwargs)\n ret_val = self.execute_command('PSUBSCRIBE', *iterkeys(new_patterns))\n # update the patterns dict AFTER we send the command. we don't want to\n # subscribe twice to these patterns, once for the command and again\n # for the reconnection.\n new_patterns = self._normalize_keys(new_patterns)\n self.patterns.update(new_patterns)\n self.pending_unsubscribe_patterns.difference_update(new_patterns)\n return ret_val"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef punsubscribe(self, *args):\n if args:\n args = list_or_args(args[0], args[1:])\n patterns = self._normalize_keys(dict.fromkeys(args))\n else:\n patterns = self.patterns\n self.pending_unsubscribe_patterns.update(patterns)\n return self.execute_command('PUNSUBSCRIBE', *args)", "response": "Unsubscribe from the supplied patterns."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef subscribe(self, *args, **kwargs):\n if args:\n args = list_or_args(args[0], args[1:])\n new_channels = dict.fromkeys(args)\n new_channels.update(kwargs)\n ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))\n # update the channels dict AFTER we send the command. we don't want to\n # subscribe twice to these channels, once for the command and again\n # for the reconnection.\n new_channels = self._normalize_keys(new_channels)\n self.channels.update(new_channels)\n self.pending_unsubscribe_channels.difference_update(new_channels)\n return ret_val", "response": "Subscribe to channels. Channels supplied as keyword arguments expect\n a channel name as the key and a callable as the value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef unsubscribe(self, *args):\n if args:\n args = list_or_args(args[0], args[1:])\n channels = self._normalize_keys(dict.fromkeys(args))\n else:\n channels = self.channels\n self.pending_unsubscribe_channels.update(channels)\n return self.execute_command('UNSUBSCRIBE', *args)", "response": "Unsubscribe from the supplied channels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef handle_message(self, response, ignore_subscribe_messages=False):\n message_type = nativestr(response[0])\n if message_type == 'pmessage':\n message = {\n 'type': message_type,\n 'pattern': response[1],\n 'channel': response[2],\n 'data': response[3]\n }\n elif message_type == 'pong':\n message = {\n 'type': message_type,\n 'pattern': None,\n 'channel': None,\n 'data': response[1]\n }\n else:\n message = {\n 'type': message_type,\n 'pattern': None,\n 'channel': response[1],\n 'data': response[2]\n }\n\n # if this is an unsubscribe message, remove it from memory\n if message_type in self.UNSUBSCRIBE_MESSAGE_TYPES:\n if message_type == 'punsubscribe':\n pattern = response[1]\n if pattern in self.pending_unsubscribe_patterns:\n self.pending_unsubscribe_patterns.remove(pattern)\n self.patterns.pop(pattern, None)\n else:\n channel = response[1]\n if channel in self.pending_unsubscribe_channels:\n self.pending_unsubscribe_channels.remove(channel)\n self.channels.pop(channel, None)\n\n if message_type in self.PUBLISH_MESSAGE_TYPES:\n # if there's a message handler, invoke it\n if message_type == 'pmessage':\n handler = self.patterns.get(message['pattern'], None)\n else:\n handler = self.channels.get(message['channel'], None)\n if handler:\n handler(message)\n return None\n elif message_type != 'pong':\n # this is a subscribe/unsubscribe message. ignore if we don't\n # want them\n if ignore_subscribe_messages or self.ignore_subscribe_messages:\n return None\n\n return message", "response": "Parses a pub / sub message and returns a parsed version of the message."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef multi(self):\n if self.explicit_transaction:\n raise RedisError('Cannot issue nested calls to MULTI')\n if self.command_stack:\n raise RedisError('Commands without an initial WATCH have already '\n 'been issued')\n self.explicit_transaction = True", "response": "This method is used to issue multiple commands at once."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pipeline_execute_command(self, *args, **options):\n self.command_stack.append((args, options))\n return self", "response": "Execute a command in the pipeline."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reset(self):\n self.operations = []\n self._last_overflow = 'WRAP'\n self.overflow(self._default_overflow or self._last_overflow)", "response": "Reset the state of the instance to when it was constructed."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the overflow algorithm of successive INCRBY operations", "response": "def overflow(self, overflow):\n \"\"\"\n Update the overflow algorithm of successive INCRBY operations\n :param overflow: Overflow algorithm, one of WRAP, SAT, FAIL. See the\n Redis docs for descriptions of these algorithmsself.\n :returns: a :py:class:`BitFieldOperation` instance.\n \"\"\"\n overflow = overflow.upper()\n if overflow != self._last_overflow:\n self._last_overflow = overflow\n self.operations.append(('OVERFLOW', overflow))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get(self, fmt, offset):\n self.operations.append(('GET', fmt, offset))\n return self", "response": "Adds a GET operation to the bitfield."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting the value of a given bitfield at the given offset.", "response": "def set(self, fmt, offset, value):\n \"\"\"\n Set the value of a given bitfield.\n :param fmt: format-string for the bitfield being read, e.g. 'u8' for\n an unsigned 8-bit integer.\n :param offset: offset (in number of bits). If prefixed with a\n '#', this is an offset multiplier, e.g. given the arguments\n fmt='u8', offset='#2', the offset will be 16.\n :param int value: value to set at the given position.\n :returns: a :py:class:`BitFieldOperation` instance.\n \"\"\"\n self.operations.append(('SET', fmt, offset, value))\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef execute(self):\n command = self.command\n self.reset()\n return self.client.execute_command(*command)", "response": "Execute the command in a single BITFIELD command."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a cached token object or creates a new one if not already cached", "response": "def get_token(cls, value):\n \"Gets a cached token object or creates a new one if not already cached\"\n\n # Use try/except because after running for a short time most tokens\n # should already be cached\n try:\n return cls._cache[value]\n except KeyError:\n token = Token(value)\n cls._cache[value] = token\n return token"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a bytestring representation of the value", "response": "def encode(self, value):\n \"Return a bytestring representation of the value\"\n if isinstance(value, Token):\n return value.encoded_value\n elif isinstance(value, bytes):\n return value\n elif isinstance(value, bool):\n # special case bool since it is a subclass of int\n raise DataError(\"Invalid input of type: 'bool'. Convert to a \"\n \"byte, string or number first.\")\n elif isinstance(value, float):\n value = repr(value).encode()\n elif isinstance(value, (int, long)):\n # python 2 repr() on longs is '123L', so use str() instead\n value = str(value).encode()\n elif not isinstance(value, basestring):\n # a value we don't know how to deal with. throw an error\n typename = type(value).__name__\n raise DataError(\"Invalid input of type: '%s'. Convert to a \"\n \"byte, string or number first.\" % typename)\n if isinstance(value, unicode):\n value = value.encode(self.encoding, self.encoding_errors)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a unicode string from the byte representation", "response": "def decode(self, value, force=False):\n \"Return a unicode string from the byte representation\"\n if (self.decode_responses or force) and isinstance(value, bytes):\n value = value.decode(self.encoding, self.encoding_errors)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse an error response", "response": "def parse_error(self, response):\n \"Parse an error response\"\n error_code = response.split(' ')[0]\n if error_code in self.EXCEPTION_CLASSES:\n response = response[len(error_code) + 1:]\n exception_class = self.EXCEPTION_CLASSES[error_code]\n if isinstance(exception_class, dict):\n exception_class = exception_class.get(response, ResponseError)\n return exception_class(response)\n return ResponseError(response)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef on_connect(self, connection):\n \"Called when the socket connects\"\n self._sock = connection._sock\n self._buffer = SocketBuffer(self._sock, self.socket_read_size)\n self.encoder = connection.encoder", "response": "Called when the socket connects"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef on_disconnect(self):\n \"Called when the socket disconnects\"\n self._sock = None\n if self._buffer is not None:\n self._buffer.close()\n self._buffer = None\n self.encoder = None", "response": "Called when the socket disconnects"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconnect to the Redis server if not already connected", "response": "def connect(self):\n \"Connects to the Redis server if not already connected\"\n if self._sock:\n return\n try:\n sock = self._connect()\n except socket.timeout:\n raise TimeoutError(\"Timeout connecting to server\")\n except socket.error:\n e = sys.exc_info()[1]\n raise ConnectionError(self._error_message(e))\n\n self._sock = sock\n self._selector = DefaultSelector(sock)\n try:\n self.on_connect()\n except RedisError:\n # clean up after any error in on_connect\n self.disconnect()\n raise\n\n # run any user callbacks. right now the only internal callback\n # is for pubsub channel/pattern resubscription\n for callback in self._connect_callbacks:\n callback(self)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndisconnecting from the Redis server", "response": "def disconnect(self):\n \"Disconnects from the Redis server\"\n self._parser.on_disconnect()\n if self._sock is None:\n return\n if self._selector is not None:\n self._selector.close()\n self._selector = None\n try:\n if os.getpid() == self.pid:\n self._sock.shutdown(socket.SHUT_RDWR)\n self._sock.close()\n except socket.error:\n pass\n self._sock = None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsends an already packed command to the Redis server", "response": "def send_packed_command(self, command):\n \"Send an already packed command to the Redis server\"\n if not self._sock:\n self.connect()\n try:\n if isinstance(command, str):\n command = [command]\n for item in command:\n self._sock.sendall(item)\n except socket.timeout:\n self.disconnect()\n raise TimeoutError(\"Timeout writing to socket\")\n except socket.error:\n e = sys.exc_info()[1]\n self.disconnect()\n if len(e.args) == 1:\n errno, errmsg = 'UNKNOWN', e.args[0]\n else:\n errno = e.args[0]\n errmsg = e.args[1]\n raise ConnectionError(\"Error %s while writing to socket. %s.\" %\n (errno, errmsg))\n except: # noqa: E722\n self.disconnect()\n raise"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npoll the socket to see if there s data that can be read.", "response": "def can_read(self, timeout=0):\n \"Poll the socket to see if there's data that can be read.\"\n sock = self._sock\n if not sock:\n self.connect()\n sock = self._sock\n return self._parser.can_read() or self._selector.can_read(timeout)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_response(self):\n \"Read the response from a previously sent command\"\n try:\n response = self._parser.read_response()\n except socket.timeout:\n self.disconnect()\n raise TimeoutError(\"Timeout reading from %s:%s\" %\n (self.host, self.port))\n except socket.error:\n self.disconnect()\n e = sys.exc_info()[1]\n raise ConnectionError(\"Error while reading from %s:%s : %s\" %\n (self.host, self.port, e.args))\n except: # noqa: E722\n self.disconnect()\n raise\n if isinstance(response, ResponseError):\n raise response\n return response", "response": "Read the response from a previously sent command"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pack_command(self, *args):\n \"Pack a series of arguments into the Redis protocol\"\n output = []\n # the client might have included 1 or more literal arguments in\n # the command name, e.g., 'CONFIG GET'. The Redis server expects these\n # arguments to be sent separately, so split the first argument\n # manually. All of these arguements get wrapped in the Token class\n # to prevent them from being encoded.\n command = args[0]\n if ' ' in command:\n args = tuple(Token.get_token(s)\n for s in command.split()) + args[1:]\n else:\n args = (Token.get_token(command),) + args[1:]\n\n buff = SYM_EMPTY.join((SYM_STAR, str(len(args)).encode(), SYM_CRLF))\n\n buffer_cutoff = self._buffer_cutoff\n for arg in imap(self.encoder.encode, args):\n # to avoid large string mallocs, chunk the command into the\n # output list if we're sending large values\n if len(buff) > buffer_cutoff or len(arg) > buffer_cutoff:\n buff = SYM_EMPTY.join(\n (buff, SYM_DOLLAR, str(len(arg)).encode(), SYM_CRLF))\n output.append(buff)\n output.append(arg)\n buff = SYM_CRLF\n else:\n buff = SYM_EMPTY.join(\n (buff, SYM_DOLLAR, str(len(arg)).encode(),\n SYM_CRLF, arg, SYM_CRLF))\n output.append(buff)\n return output", "response": "Pack a series of arguments into the Redis protocol"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _connect(self):\n \"Wrap the socket with SSL support\"\n sock = super(SSLConnection, self)._connect()\n if hasattr(ssl, \"create_default_context\"):\n context = ssl.create_default_context()\n context.check_hostname = False\n context.verify_mode = self.cert_reqs\n if self.certfile and self.keyfile:\n context.load_cert_chain(certfile=self.certfile,\n keyfile=self.keyfile)\n if self.ca_certs:\n context.load_verify_locations(self.ca_certs)\n sock = context.wrap_socket(sock, server_hostname=self.host)\n else:\n # In case this code runs in a version which is older than 2.7.9,\n # we want to fall back to old code\n sock = ssl.wrap_socket(sock,\n cert_reqs=self.cert_reqs,\n keyfile=self.keyfile,\n certfile=self.certfile,\n ca_certs=self.ca_certs)\n return sock", "response": "Wrap the socket with SSL support"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a Unix domain socket connection", "response": "def _connect(self):\n \"Create a Unix domain socket connection\"\n sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)\n sock.settimeout(self.socket_timeout)\n sock.connect(self.path)\n return sock"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new ConnectionPool configured from a given URL.", "response": "def from_url(cls, url, db=None, decode_components=False, **kwargs):\n \"\"\"\n Return a connection pool configured from the given URL.\n\n For example::\n\n redis://[:password]@localhost:6379/0\n rediss://[:password]@localhost:6379/0\n unix://[:password]@/path/to/socket.sock?db=0\n\n Three URL schemes are supported:\n\n - ```redis://``\n `_ creates a\n normal TCP socket connection\n - ```rediss://``\n `_ creates\n a SSL wrapped TCP socket connection\n - ``unix://`` creates a Unix Domain Socket connection\n\n There are several ways to specify a database number. The parse function\n will return the first specified option:\n 1. A ``db`` querystring option, e.g. redis://localhost?db=0\n 2. If using the redis:// scheme, the path argument of the url, e.g.\n redis://localhost/0\n 3. The ``db`` argument to this function.\n\n If none of these options are specified, db=0 is used.\n\n The ``decode_components`` argument allows this function to work with\n percent-encoded URLs. If this argument is set to ``True`` all ``%xx``\n escapes will be replaced by their single-character equivalents after\n the URL has been parsed. This only applies to the ``hostname``,\n ``path``, and ``password`` components.\n\n Any additional querystring arguments and keyword arguments will be\n passed along to the ConnectionPool class's initializer. The querystring\n arguments ``socket_connect_timeout`` and ``socket_timeout`` if supplied\n are parsed as float values. The arguments ``socket_keepalive`` and\n ``retry_on_timeout`` are parsed to boolean values that accept\n True/False, Yes/No values to indicate state. Invalid types cause a\n ``UserWarning`` to be raised. In the case of conflicting arguments,\n querystring arguments always win.\n\n \"\"\"\n url = urlparse(url)\n url_options = {}\n\n for name, value in iteritems(parse_qs(url.query)):\n if value and len(value) > 0:\n parser = URL_QUERY_ARGUMENT_PARSERS.get(name)\n if parser:\n try:\n url_options[name] = parser(value[0])\n except (TypeError, ValueError):\n warnings.warn(UserWarning(\n \"Invalid value for `%s` in connection URL.\" % name\n ))\n else:\n url_options[name] = value[0]\n\n if decode_components:\n password = unquote(url.password) if url.password else None\n path = unquote(url.path) if url.path else None\n hostname = unquote(url.hostname) if url.hostname else None\n else:\n password = url.password\n path = url.path\n hostname = url.hostname\n\n # We only support redis://, rediss:// and unix:// schemes.\n if url.scheme == 'unix':\n url_options.update({\n 'password': password,\n 'path': path,\n 'connection_class': UnixDomainSocketConnection,\n })\n\n elif url.scheme in ('redis', 'rediss'):\n url_options.update({\n 'host': hostname,\n 'port': int(url.port or 6379),\n 'password': password,\n })\n\n # If there's a path argument, use it as the db argument if a\n # querystring value wasn't specified\n if 'db' not in url_options and path:\n try:\n url_options['db'] = int(path.replace('/', ''))\n except (AttributeError, ValueError):\n pass\n\n if url.scheme == 'rediss':\n url_options['connection_class'] = SSLConnection\n else:\n valid_schemes = ', '.join(('redis://', 'rediss://', 'unix://'))\n raise ValueError('Redis URL must specify one of the following'\n 'schemes (%s)' % valid_schemes)\n\n # last shot at the db value\n url_options['db'] = int(url_options.get('db', db or 0))\n\n # update the arguments from the URL values\n kwargs.update(url_options)\n\n # backwards compatability\n if 'charset' in kwargs:\n warnings.warn(DeprecationWarning(\n '\"charset\" is deprecated. Use \"encoding\" instead'))\n kwargs['encoding'] = kwargs.pop('charset')\n if 'errors' in kwargs:\n warnings.warn(DeprecationWarning(\n '\"errors\" is deprecated. Use \"encoding_errors\" instead'))\n kwargs['encoding_errors'] = kwargs.pop('errors')\n\n return cls(**kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a connection from the pool", "response": "def get_connection(self, command_name, *keys, **options):\n \"Get a connection from the pool\"\n self._checkpid()\n try:\n connection = self._available_connections.pop()\n except IndexError:\n connection = self.make_connection()\n self._in_use_connections.add(connection)\n try:\n # ensure this connection is connected to Redis\n connection.connect()\n # connections that the pool provides should be ready to send\n # a command. if not, the connection was either returned to the\n # pool before all data has been read or the socket has been\n # closed. either way, reconnect and verify everything is good.\n if not connection.is_ready_for_command():\n connection.disconnect()\n connection.connect()\n if not connection.is_ready_for_command():\n raise ConnectionError('Connection not ready')\n except: # noqa: E722\n # release the connection back to the pool so that we don't leak it\n self.release(connection)\n raise\n\n return connection"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_encoder(self):\n \"Return an encoder based on encoding settings\"\n kwargs = self.connection_kwargs\n return Encoder(\n encoding=kwargs.get('encoding', 'utf-8'),\n encoding_errors=kwargs.get('encoding_errors', 'strict'),\n decode_responses=kwargs.get('decode_responses', False)\n )", "response": "Return an encoder based on encoding settings"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_connection(self):\n \"Create a new connection\"\n if self._created_connections >= self.max_connections:\n raise ConnectionError(\"Too many connections\")\n self._created_connections += 1\n return self.connection_class(**self.connection_kwargs)", "response": "Create a new connection"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndisconnect all connections in the pool", "response": "def disconnect(self):\n \"Disconnects all connections in the pool\"\n self._checkpid()\n all_conns = chain(self._available_connections,\n self._in_use_connections)\n for connection in all_conns:\n connection.disconnect()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_connection(self):\n \"Make a fresh connection.\"\n connection = self.connection_class(**self.connection_kwargs)\n self._connections.append(connection)\n return connection", "response": "Make a fresh connection."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a connection from the pool.", "response": "def get_connection(self, command_name, *keys, **options):\n \"\"\"\n Get a connection, blocking for ``self.timeout`` until a connection\n is available from the pool.\n\n If the connection returned is ``None`` then creates a new connection.\n Because we use a last-in first-out queue, the existing connections\n (having been returned to the pool after the initial ``None`` values\n were added) will be returned before ``None`` values. This means we only\n create new connections when we need to, i.e.: the actual number of\n connections will only increase in response to demand.\n \"\"\"\n # Make sure we haven't changed process.\n self._checkpid()\n\n # Try and get a connection from the pool. If one isn't available within\n # self.timeout then raise a ``ConnectionError``.\n connection = None\n try:\n connection = self.pool.get(block=True, timeout=self.timeout)\n except Empty:\n # Note that this is not caught by the redis client and will be\n # raised unless handled by application code. If you want never to\n raise ConnectionError(\"No connection available.\")\n\n # If the ``connection`` is actually ``None`` then that's a cue to make\n # a new connection to add to the pool.\n if connection is None:\n connection = self.make_connection()\n\n try:\n # ensure this connection is connected to Redis\n connection.connect()\n # connections that the pool provides should be ready to send\n # a command. if not, the connection was either returned to the\n # pool before all data has been read or the socket has been\n # closed. either way, reconnect and verify everything is good.\n if not connection.is_ready_for_command():\n connection.disconnect()\n connection.connect()\n if not connection.is_ready_for_command():\n raise ConnectionError('Connection not ready')\n except: # noqa: E722\n # release the connection back to the pool so that we don't leak it\n self.release(connection)\n raise\n\n return connection"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrelease the connection back to the pool.", "response": "def release(self, connection):\n \"Releases the connection back to the pool.\"\n # Make sure we haven't changed process.\n self._checkpid()\n if connection.pid != self.pid:\n return\n\n # Put the connection back into the pool.\n try:\n self.pool.put_nowait(connection)\n except Full:\n # perhaps the pool has been reset() after a fork? regardless,\n # we don't want this connection\n pass"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npack a series of arguments into a value Redis command", "response": "def pack_command(self, *args):\n \"Pack a series of arguments into a value Redis command\"\n args_output = SYM_EMPTY.join([\n SYM_EMPTY.join(\n (SYM_DOLLAR, str(len(k)).encode(), SYM_CRLF, k, SYM_CRLF))\n for k in imap(self.encoder.encode, args)])\n output = SYM_EMPTY.join(\n (SYM_STAR, str(len(args)).encode(), SYM_CRLF, args_output))\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetermine if the current platform has the selector available", "response": "def has_selector(selector):\n \"Determine if the current platform has the selector available\"\n try:\n if selector == 'poll':\n # the select module offers the poll selector even if the platform\n # doesn't support it. Attempt to poll for nothing to make sure\n # poll is available\n p = select.poll()\n p.poll(0)\n else:\n # the other selectors will fail when instantiated\n getattr(select, selector)().close()\n return True\n except (OSError, AttributeError):\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef DefaultSelector(sock):\n \"Return the best selector for the platform\"\n global _DEFAULT_SELECTOR\n if _DEFAULT_SELECTOR is None:\n if has_selector('poll'):\n _DEFAULT_SELECTOR = PollSelector\n elif hasattr(select, 'select'):\n _DEFAULT_SELECTOR = SelectSelector\n else:\n raise RedisError('Platform does not support any selectors')\n return _DEFAULT_SELECTOR(sock)", "response": "Return the best selector for the platform"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if data is ready to be read from the socket False otherwise.", "response": "def can_read(self, timeout=0):\n \"\"\"\n Return True if data is ready to be read from the socket,\n otherwise False.\n\n This doesn't guarentee that the socket is still connected, just that\n there is data to read.\n\n Automatically retries EINTR errors based on PEP 475.\n \"\"\"\n while True:\n try:\n return self.check_can_read(timeout)\n except (select.error, IOError) as ex:\n if self.errno_from_exception(ex) == errno.EINTR:\n continue\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if the socket is ready to send a command.", "response": "def is_ready_for_command(self, timeout=0):\n \"\"\"\n Return True if the socket is ready to send a command,\n otherwise False.\n\n Automatically retries EINTR errors based on PEP 475.\n \"\"\"\n while True:\n try:\n return self.check_is_ready_for_command(timeout)\n except (select.error, IOError) as ex:\n if self.errno_from_exception(ex) == errno.EINTR:\n continue\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the error number from an exception object", "response": "def errno_from_exception(self, ex):\n \"\"\"\n Get the error number from an exception\n \"\"\"\n if hasattr(ex, 'errno'):\n return ex.errno\n elif ex.args:\n return ex.args[0]\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_url(url, db=None, **kwargs):\n from redis.client import Redis\n return Redis.from_url(url, db, **kwargs)", "response": "Returns an active Redis client generated from the given database URL."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rotate_slaves(self):\n \"Round-robin slave balancer\"\n slaves = self.sentinel_manager.discover_slaves(self.service_name)\n if slaves:\n if self.slave_rr_counter is None:\n self.slave_rr_counter = random.randint(0, len(slaves) - 1)\n for _ in xrange(len(slaves)):\n self.slave_rr_counter = (\n self.slave_rr_counter + 1) % len(slaves)\n slave = slaves[self.slave_rr_counter]\n yield slave\n # Fallback to the master connection\n try:\n yield self.get_master_address()\n except MasterNotFoundError:\n pass\n raise SlaveNotFoundError('No slave found for %r' % (self.service_name))", "response": "Round - robin slave balancer"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndiscovering the master s address corresponding to the service labeled service_name.", "response": "def discover_master(self, service_name):\n \"\"\"\n Asks sentinel servers for the Redis master's address corresponding\n to the service labeled ``service_name``.\n\n Returns a pair (address, port) or raises MasterNotFoundError if no\n master is found.\n \"\"\"\n for sentinel_no, sentinel in enumerate(self.sentinels):\n try:\n masters = sentinel.sentinel_masters()\n except (ConnectionError, TimeoutError):\n continue\n state = masters.get(service_name)\n if state and self.check_master_state(state, service_name):\n # Put this sentinel at the top of the list\n self.sentinels[0], self.sentinels[sentinel_no] = (\n sentinel, self.sentinels[0])\n return state['ip'], state['port']\n raise MasterNotFoundError(\"No master found for %r\" % (service_name,))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves slaves that are in an ODOWN or SDOWN state", "response": "def filter_slaves(self, slaves):\n \"Remove slaves that are in an ODOWN or SDOWN state\"\n slaves_alive = []\n for slave in slaves:\n if slave['is_odown'] or slave['is_sdown']:\n continue\n slaves_alive.append((slave['ip'], slave['port']))\n return slaves_alive"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef discover_slaves(self, service_name):\n \"Returns a list of alive slaves for service ``service_name``\"\n for sentinel in self.sentinels:\n try:\n slaves = sentinel.sentinel_slaves(service_name)\n except (ConnectionError, ResponseError, TimeoutError):\n continue\n slaves = self.filter_slaves(slaves)\n if slaves:\n return slaves\n return []", "response": "Returns a list of alive slaves for service service_name"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef explain_weights_lightning(estimator, vec=None, top=20, target_names=None,\n targets=None, feature_names=None,\n coef_scale=None):\n \"\"\" Return an explanation of a lightning estimator weights \"\"\"\n return explain_weights_lightning_not_supported(estimator)", "response": "Return an explanation of a lightning estimator weights"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef explain_prediction_lightning(estimator, doc, vec=None, top=None,\n target_names=None, targets=None,\n feature_names=None, vectorized=False,\n coef_scale=None):\n \"\"\" Return an explanation of a lightning estimator predictions \"\"\"\n return explain_weights_lightning_not_supported(estimator, doc)", "response": "Return an explanation of a lightning estimator prediction"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef format_as_html(explanation, # type: Explanation\n include_styles=True, # type: bool\n force_weights=True, # type: bool\n show=fields.ALL,\n preserve_density=None, # type: Optional[bool]\n highlight_spaces=None, # type: Optional[bool]\n horizontal_layout=True, # type: bool\n show_feature_values=False # type: bool\n ):\n # type: (...) -> str\n \"\"\" Format explanation as html.\n Most styles are inline, but some are included separately in