Fix several ResourceWarnings in test_requests.py

Hello,

I do not know if you are interested in fixing `ResourceWarning`s in tests, but I will take my chance :)
There was 10 unclosed files in `test_requests.py`.

@@ -1022,12 +1022,13 @@ def test_form_encoded_post_query_multivalued_element(self, httpbin):
         assert prep.body == "test=foo&test=baz"

     def test_different_encodings_dont_break_post(self, httpbin):
-        r = requests.post(
-            httpbin("post"),
-            data={"stuff": json.dumps({"a": 123})},
-            params={"blah": "asdf1234"},
-            files={"file": ("test_requests.py", open(__file__, "rb"))},
-        )
+        with open(__file__, "rb") as f:
+            r = requests.post(
+                httpbin("post"),
+                data={"stuff": json.dumps({"a": 123})},
+                params={"blah": "asdf1234"},
+                files={"file": ("test_requests.py", f)},
+            )
         assert r.status_code == 200

     [USER_REDACTED].mark.parametrize(
@@ -1040,35 +1041,44 @@ def test_different_encodings_dont_break_post(self, httpbin):
         ),
     )
     def test_unicode_multipart_post(self, httpbin, data):
-        r = requests.post(
-            httpbin("post"),
-            data=data,
-            files={"file": ("test_requests.py", open(__file__, "rb"))},
-        )
+        with open(__file__, "rb") as f:
+            r = requests.post(
+                httpbin("post"),
+                data=data,
+                files={"file": ("test_requests.py", f)},
+            )
         assert r.status_code == 200

     def test_unicode_multipart_post_fieldnames(self, httpbin):
         filename = os.path.splitext(__file__)[0] + ".py"
-        r = requests.Request(
-            method="POST",
-            url=httpbin("post"),
-            data={b"stuff": "elixr"},
-            files={"file": ("test_requests.py", open(filename, "rb"))},
-        )
-        prep = r.prepare()
+        with open(filename, "rb") as f:
+            r = requests.Request(
+                method="POST",
+                url=httpbin("post"),
+                data={b"stuff": "elixr"},
+                files={"file": ("test_requests.py", f)},
+            )
+            prep = r.prepare()
+
         assert b'name="stuff"' in prep.body
         assert b"name=\"b'stuff'\"" not in prep.body

     def test_unicode_method_name(self, httpbin):
-        files = {"file": open(__file__, "rb")}
-        r = requests.request(method="POST", url=httpbin("post"), files=files)
+        with open(__file__, "rb") as f:
+            files = {"file": f}
+            r = requests.request(
+                method="POST",
+                url=httpbin("post"),
+                files=files,
+            )
         assert r.status_code == 200

     def test_unicode_method_name_with_request_object(self, httpbin):
-        files = {"file": open(__file__, "rb")}
         s = requests.Session()
-        req = requests.Request("POST", httpbin("post"), files=files)
-        prep = s.prepare_request(req)
+        with open(__file__, "rb") as f:
+            files = {"file": f}
+            req = requests.Request("POST", httpbin("post"), files=files)
+            prep = s.prepare_request(req)
         assert isinstance(prep.method, builtin_str)
         assert prep.method == "POST"

@@ -1084,18 +1094,14 @@ def test_non_prepared_request_error(self):
         assert str(e.value) == "You can only send PreparedRequests."

     def test_custom_content_type(self, httpbin):
-        r = requests.post(
-            httpbin("post"),
-            data={"stuff": json.dumps({"a": 123})},
-            files={
-                "file1": ("test_requests.py", open(__file__, "rb")),
-                "file2": (
-                    "test_requests",
-                    open(__file__, "rb"),
-                    "text/py-content-type",
-                ),
-            },
-        )
+        with open(__file__, "rb") as f1:
+            with open(__file__, "rb") as f2:
+                data={"stuff": json.dumps({"a": 123})}
+                files = {
+                    "file1": ("test_requests.py", f1),
+                    "file2": ("test_requests", f2, "text/py-content-type"),
+                }
+                r = requests.post(httpbin("post"), data=data, files=files)
         assert r.status_code == 200
         assert b"text/py-content-type" in r.request.body

@@ -1484,9 +1490,9 @@ def test_prepared_request_is_pickleable(self, httpbin):
         assert resp.status_code == 200

     def test_prepared_request_with_file_is_pickleable(self, httpbin):
-        files = {"file": open(__file__, "rb")}
-        r = requests.Request("POST", httpbin("post"), files=files)
-        p = r.prepare()
+        with open(__file__, "rb") as f:
+            r = requests.Request("POST", httpbin("post"), files={"file": f})
+            p = r.prepare()

         # [NAME_REDACTED] can be pickled and unpickled
         r = pickle.loads(pickle.dumps(p))

Hi [USER_REDACTED], thanks for looking into this. I don't have an issue adding the context handlers, but we should probably avoid dumping all of the logic inside of them.

If we want to encourage best practices, let's make sure anything not relevant to the file handler stays at its original indentation. Once that's done, if you can rebase onto master, I think I'm happy with this.

Hi [USER_REDACTED], I've rebased your change onto the top of our main branch and resolved some conflicts. This seems like a reasonable fix still and I think we're happy with its current state. Thanks again for the contribution!

Let’s pull the `assert` out to the same indentation level as `with`.

Same here.

Let's just pass this directly to the `files` parameter.

We can unindent this to align with `with`.

These asserts can be done at the same indentation as the `with` block.


Document that params and data can be list of tuples everywhere

Fix a few inconsistencies in the doc found while working on [MechanicalSoup](https://github.com/[USER_REDACTED]nicalSoup/MechanicalSoup/pull/219).

@@ -18,8 +18,10 @@ def request(method, url, **kwargs):

     :param method: method for the new :class:`Request` object.
     :param url: URL for the new :class:`Request` object.
-    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
-    :param data: (optional) Dictionary or list of tuples ``[(key, value)]`` (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
+    :param params: (optional) Dictionary, list of tuples or bytes to send
+        in the body of the :class:`Request`.
+    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+        object to send in the body of the :class:`Request`.
     :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
     :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
     :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
@@ -62,7 +64,8 @@ def get(url, params=None, **kwargs):
     r"""Sends a GET request.

     :param url: URL for the new :class:`Request` object.
-    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
+    :param params: (optional) Dictionary, list of tuples or bytes to send
+        in the body of the :class:`Request`.
     :param \*\*kwargs: Optional arguments that ``request`` takes.
     :return: :class:`Response <Response>` object
     :rtype: requests.Response
@@ -102,7 +105,8 @@ def post(url, data=None, json=None, **kwargs):
     r"""Sends a POST request.

     :param url: URL for the new :class:`Request` object.
-    :param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
+    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+        object to send in the body of the :class:`Request`.
     :param json: (optional) json data to send in the body of the :class:`Request`.
     :param \*\*kwargs: Optional arguments that ``request`` takes.
     :return: :class:`Response <Response>` object
@@ -116,7 +120,8 @@ def put(url, data=None, **kwargs):
     r"""Sends a PUT request.

     :param url: URL for the new :class:`Request` object.
-    :param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
+    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+        object to send in the body of the :class:`Request`.
     :param json: (optional) json data to send in the body of the :class:`Request`.
     :param \*\*kwargs: Optional arguments that ``request`` takes.
     :return: :class:`Response <Response>` object
@@ -130,7 +135,8 @@ def patch(url, data=None, **kwargs):
     r"""Sends a PATCH request.

     :param url: URL for the new :class:`Request` object.
-    :param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
+    :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+        object to send in the body of the :class:`Request`.
     :param json: (optional) json data to send in the body of the :class:`Request`.
     :param \*\*kwargs: Optional arguments that ``request`` takes.
     :return: :class:`Response <Response>` object

@@ -204,9 +204,13 @@ class Request(RequestHooksMixin):
     :param url: URL to send.
     :param headers: dictionary of headers to send.
     :param files: dictionary of {filename: fileobject} files to multipart upload.
-    :param data: the body to attach to the request. If a dictionary is provided, form-encoding will take place.
+    :param data: the body to attach to the request. If a dictionary or
+        list of tuples ``[(key, value)]`` is provided, form-encoding will
+        take place.
     :param json: json for the body to attach to the request (if files or data is not specified).
-    :param params: dictionary of URL parameters to append to the URL.
+    :param params: URL parameters to append to the URL. If a dictionary or
+        list of tuples ``[(key, value)]`` is provided, form-encoding will
+        take place.
     :param auth: Auth handler or (user, pass) tuple.
     :param cookies: dictionary or CookieJar of cookies to attach to this request.
     :param hooks: dictionary of callback hooks, for internal usage.

@@ -453,8 +453,8 @@ def request(self, method, url,
         :param url: URL for the new :class:`Request` object.
         :param params: (optional) Dictionary or bytes to be sent in the query
             string for the :class:`Request`.
-        :param data: (optional) Dictionary, bytes, or file-like object to send
-            in the body of the :class:`Request`.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
         :param json: (optional) json to send in the body of the
             :class:`Request`.
         :param headers: (optional) Dictionary of HTTP Headers to send with the
@@ -550,7 +550,8 @@ def post(self, url, data=None, json=None, **kwargs):
         r"""Sends a POST request. Returns :class:`Response` object.

         :param url: URL for the new :class:`Request` object.
-        :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
         :param json: (optional) json to send in the body of the :class:`Request`.
         :param \*\*kwargs: Optional arguments that ``request`` takes.
         :rtype: requests.Response
@@ -562,7 +563,8 @@ def put(self, url, data=None, **kwargs):
         r"""Sends a PUT request. Returns :class:`Response` object.

         :param url: URL for the new :class:`Request` object.
-        :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
         :param \*\*kwargs: Optional arguments that ``request`` takes.
         :rtype: requests.Response
         """
@@ -573,7 +575,8 @@ def patch(self, url, data=None, **kwargs):
         r"""Sends a PATCH request. Returns :class:`Response` object.

         :param url: URL for the new :class:`Request` object.
-        :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
         :param \*\*kwargs: Optional arguments that ``request`` takes.
         :rtype: requests.Response
         """

Hi [USER_REDACTED], thanks for submitting this PR! I think we can make the change on the base `request` method to be consistent between `data` and `params`.

As for the others, it feels unnecessarily redundant for a smaller edge case. The tuple list is a alternative encapsulation of the dictionary concept that is allowing us to use the same key multiple times, and we have documentation around this in the main docs if needed.

Ok, I think I’m happy with this. Thanks [USER_REDACTED]!

This can probably be two lines instead of three, following the line length of the other entries.

To use 2 lines, one needs to cut the ` ``literal`` ` block, and I've seen some RST processor yield incorrect results in this case so I preferred avoiding this cut. I can change it if you prefer.

I don’t think we need to cut the literal block. This line is almost identical to the one below it which only occupies a single line.

Indeed: my text editor wraps at 72 characters so it was cutting in the middle of the literal block. I make lines a bit longer, but in most cases it doesn't change the number of lines. Other docstrings fit on one line because they're e.g. 171 characters long.

[USER_REDACTED] yes, so if we split it across two lines it will be < 100 characters per line which is consistent with this projects typical line length.

I'm a new contributor so I can't talk about the established practices, but the doc says "Line-length can exceed 79 characters, to 100, when convenient.", so I understand that I can make an exception to the 79 characters when I have a good reason ("when convenient"). I don't see the benefit of making an exception here.


[NAME_REDACTED] header whenever root URL changes

Previously the header was stripped only if the hostname changed, but in
an https -> http redirect that can leak the credentials on the wire
(#4716). Based on with RFC 7235 section 2.2, the header is now stripped
if the "canonical root URL" (scheme+authority) has changed.

Closes #4716.

@@ -115,6 +115,22 @@ def get_redirect_target(self, resp):
             return to_native_string(location, 'utf8')
         return None

+    def should_strip_auth(self, old_url, new_url):
+        """Decide whether Authorization header should be removed when redirecting"""
+        old_parsed = urlparse(old_url)
+        new_parsed = urlparse(new_url)
+        if old_parsed.hostname != new_parsed.hostname:
+            return True
+        # Special case: allow http -> https redirect when using the standard
+        # ports. This isn't specified by RFC 7235, but is kept to avoid
+        # breaking backwards compatibility with older versions of requests
+        # that allowed any redirects on the same host.
+        if (old_parsed.scheme == 'http' and old_parsed.port in (80, None)
+                and new_parsed.scheme == 'https' and new_parsed.port in (443, None)):
+            return False
+        # Standard case: root URI must match
+        return old_parsed.port != new_parsed.port or old_parsed.scheme != new_parsed.scheme
+
     def resolve_redirects(self, resp, req, stream=False, timeout=None,
                           verify=True, cert=None, proxies=None, yield_requests=False, **adapter_kwargs):
         """Receives a Response. Returns a generator of Responses or Requests."""
@@ -236,14 +252,10 @@ def rebuild_auth(self, prepared_request, response):
         headers = prepared_request.headers
         url = prepared_request.url

-        if 'Authorization' in headers:
+        if 'Authorization' in headers and self.should_strip_auth(response.request.url, url):
             # If we get redirected to a new host, we should strip out any
             # authentication headers.
-            original_parsed = urlparse(response.request.url)
-            redirect_parsed = urlparse(url)
-
-            if (original_parsed.hostname != redirect_parsed.hostname):
-                del headers['Authorization']
+            del headers['Authorization']

         # .netrc might have more auth for us on our new host.
         new_auth = get_netrc_auth(url) if self.trust_env else None

@@ -1573,15 +1573,15 @@ def test_nonhttp_schemes_dont_check_URLs(self):
             preq = req.prepare()
             assert test_url == preq.url

-    [USER_REDACTED].mark.xfail(raises=ConnectionError)
-    def test_auth_is_stripped_on_redirect_off_host(self, httpbin):
+    def test_auth_is_stripped_on_http_downgrade(self, httpbin, httpbin_secure, httpbin_ca_bundle):
         r = requests.get(
-            httpbin('redirect-to'),
-            params={'url': 'http://www.google.co.uk'},
+            httpbin_secure('redirect-to'),
+            params={'url': httpbin('get')},
             auth=('user', 'pass'),
+            verify=httpbin_ca_bundle
         )
         assert r.history[0].request.headers['Authorization']
-        assert not r.request.headers.get('Authorization', '')
+        assert 'Authorization' not in r.request.headers

     def test_auth_is_retained_for_redirect_on_host(self, httpbin):
         r = requests.get(httpbin('redirect/1'), auth=('user', 'pass'))
@@ -1590,6 +1590,27 @@ def test_auth_is_retained_for_redirect_on_host(self, httpbin):

         assert h1 == h2

+    def test_should_strip_auth_host_change(self):
+        s = requests.Session()
+        assert s.should_strip_auth('http://example.com/foo', 'http://another.example.com/')
+
+    def test_should_strip_auth_http_downgrade(self):
+        s = requests.Session()
+        assert s.should_strip_auth('https://example.com/foo', 'http://example.com/bar')
+
+    def test_should_strip_auth_https_upgrade(self):
+        s = requests.Session()
+        assert not s.should_strip_auth('http://example.com/foo', 'https://example.com/bar')
+        assert not s.should_strip_auth('http://example.com:80/foo', 'https://example.com/bar')
+        assert not s.should_strip_auth('http://example.com/foo', 'https://example.com:443/bar')
+        # Non-standard ports should trigger stripping
+        assert s.should_strip_auth('http://example.com:8080/foo', 'https://example.com/bar')
+        assert s.should_strip_auth('http://example.com/foo', 'https://example.com:8443/bar')
+
+    def test_should_strip_auth_port_change(self):
+        s = requests.Session()
+        assert s.should_strip_auth('http://example.com:1234/foo', 'https://example.com:4321/bar')
+
     def test_manual_redirect_with_partial_body_read(self, httpbin):
         s = requests.Session()
         r1 = s.get(httpbin('redirect/2'), allow_redirects=False, stream=True)

Hi [USER_REDACTED], thanks for contributing to Requests! I agree that this behavior does seem wrong and we ideally shouldn’t be sending credentials across the wire in clear view. I took a quick look but was surprisingly unable to find this discussed before. While I’m inclined to merge this, there’s likely some concern about breaking users relying on us not doing the right thing here. I’d like [USER_REDACTED] to weigh in with his thoughts before we move forward.

In the meantime, I’ve left a couple minor notes on some of the changes.

Ok, I think this is set to go. Thanks for driving this, [USER_REDACTED]!

Ok, this still looks good. Thanks again for driving this [USER_REDACTED]!

We may want to do the explicit check on scheme, hostname, and port here. Otherwise a url with credentials that’s redirected without credentials may have the Authorization header unnecessarily stripped.

We should be explicit here and validate that `Authorization` is not in the request headers rather than checking if the value is falsey.

Fair enough, I hadn't spotted that the username and password are still present in response.request.url.

I was just copying the test above it, but I agree that would be an improvement. Will fix it tomorrow.

`r.json()['headers']` should also be a way to verify that it wasn't *sent*

I think this might need to be a bit different. I think if someone sent credentials over `HTTP` and we're being redirected to the same domain but it's `HTTPS` we should allow that. That's a secure upgrade. The opposite should not be allowed.

I did it this way based on RFC 7235:

>  A protection space is defined by [NAME_REDACTED] (the scheme
   and authority components of the effective request URI; see Section
   5.5 of [RFC7230]) of the server being accessed, in combination with
   the realm value if present.

...

> The protection space determines the domain over which credentials can
   be automatically applied.  If a prior request has been authorized,
   the user agent MAY reuse the same credentials for all other requests
   within that protection space for a period of time determined by [NAME_REDACTED], parameters, and/or user preferences (such as a
   configurable inactivity timeout).

While it's not explicit, the implication is that a user agent may not automatically reuse the credentials outside the protection space, and a change from http -> https is a change in protection space.

[USER_REDACTED], just to be clear, the spec you’re referencing is specifically about the realm attribute, which is different that what we’re doing here.

If we look at browsers for reference here, upgrade redirects maintain Authorization headers. Breaking this will make a common pattern pretty painful for most users.

Strange, I thought I'd tested the http -> https case with Firefox and Chromium and neither had sent the Authorization header. I'll take a closer look at my test when I'm in the office tomorrow.

Just to check that I'm testing the same thing that you're referring to: my test is a localhost Apache server listening on http and https. On http it is set up to require basic auth, and on some path it is set up to send a 302 redirect to somewhere on https. I point the browser at the http port, I get a WWW-Authenticate challenge and I type in username+password, I then get redirected to https and the browser either does or doesn't automatically send the same credentials immediately without another WWW-Authenticate challenge?

I've redone my test, using `openssl s_server` to see the request headers that the browser sends to the https port after the redirect. Firefox definitely isn't preserving the Authorization header across the redirect. Let me know if you'd like more details to reproduce the test (httpd.conf etc) using Docker.

Hi [USER_REDACTED], it looks like Firefox does adhere to this rule. I'm a bit worried that adding the stripping from HTTP->HTTPS is going to be breaking for some people. I think for this patch, we should focus on removing the immediate security risk, and then we can add a todo for Requests3 to make this more in line with how browsers are behaving.

Thanks for reminding me about this PR. Okay, I can add an exception to allow http -> https redirects, with a comment to explain why.

How do you want to handle port changes? For example, `http://example.com` -> `https://example.com` is one thing (which implicitly changes the port from 80 to 443), but what about `http://example.com:8080` -> `https://example.com:12345` or `http://example.com` -> `http://example.com:12345`? That last one seems like a security risk since a non-root user could listen on an arbitrary port.

I bring up the port changes because the unit test necessarily runs httpbin on non-standard ports and so there is no way to test redirects with implicit port numbers.

Any change in a URLs origin should probably result in stripping of security headers. Origin for `http/https` URLs is scheme, hostname, and port. The only cases we should keep headers on a cross-origin redirect IMO are `http://example.com:80 -> https://example.com:443` and `http://example.com:1234 -> https://example.com:1234`. Thoughts?

I think that matches the logic I started implementing (with the exception that it will only allow upgrade from 80->443 when both are implicit):
```python
https_upgrade = original_parsed.scheme == 'http' and redirect_parsed.scheme == 'https'
if (original_parsed.hostname != redirect_parsed.hostname
         or original_parsed.port != redirect_parsed.port
         or (original_parsed.scheme != redirect_parsed.scheme and not https_upgrade)):
```
Does that look reasonable?

The catch is that it's not really possible to test with httpbin, because we don't get to pick the port numbers that httpbin runs on. One option would be to move this logic into its own function (that just takes two URLs and decides whether to strip) and unit test that function separately from the end-to-end test through httpbin. The existing end-to-end tests should be sufficient to check that the actual stripping is done correctly.

Question because I don't know the behavior of the URL parser, if you parse `http://example.com` and `http://example.com:80` do you get a different value of `redirect_parsed.port`, is it `None` on implicit?

Yes
```python
>>> print(urllib.parse.urlparse('http://example.com/').port)
None
>>> print(urllib.parse.urlparse('http://example.com:80/').port)
80
```

So we'd have to account for that in our comparison above as we want `http://example.com:80` -> `https://example.com` to work the same as `http://example.com` -> `https://example.com`.

At this point there’s enough complexity in this implementation we should probably move it into it’s own check function.

I think Seth’s assessment is right here. We want to check the following:

1. The hostnames match.
2. The schemes match...
    * Except in the case of http->https
    * If scheme change is http->https, origin port may be 80 or None and destination port may be 443 or None.
3. Ports match, except in above scenario.
4. Same scheme and hostname but varied port results in stripped auth.

Sounds sensible to me. I've implemented that in the latest push.

I've left the logic in a public method of SessionRedirectMixin, because that gives the user the option to override it in a subclass, but let me know if I should move it to utils.

Ping? I think I've addressed the comments so far.


Add official Python 3.7 support

Python 3.7 was just released [1]. This is a small change to
enable support in requests.

[1] https://docs.python.org/3.7/whatsnew/3.7.html

Signed-off-by: [NAME_EMAIL_REDACTED]

@@ -1,14 +1,5 @@
 sudo: false
 language: python
-python:
-  # - "2.6"
-  - "2.7"
-  - "3.4"
-  - "3.5"
-  - "3.6"
-  - "3.7-dev"
-  # - "pypy"  -- appears to hang
-  # - "pypy3"
 # command to install dependencies
 install: "make"
 # command to run tests
@@ -21,9 +12,31 @@ jobs:
   include:
     - stage: test
       script:
-        - |
-          if [[ "$TRAVIS_PYTHON_VERSION" != "2.6" ]] ; then make test-readme; fi
+        - make test-readme
         - make ci
+      python: '2.7'
+    - stage: test
+      script:
+        - make test-readme
+        - make ci
+      python: '3.4'
+    - stage: test
+      script:
+        - make test-readme
+        - make ci
+      python: '3.5'
+    - stage: test
+      script:
+        - make test-readme
+        - make ci
+      python: '3.6'
+    - stage: test
+      script:
+        - make test-readme
+        - make ci
+      python: '3.7'
+      dist: xenial
+      sudo: true
     - stage: coverage
       python: 3.6
       script: codecov

@@ -25,6 +25,11 @@ environment:
       PYTHON_ARCH: "64"
       TOXENV: "py36"

+    - PYTHON: "C:\\Python37-x64"
+      PYTHON_VERSION: "3.7.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py37"
+
 install:
   # [NAME_REDACTED] (from the official .msi of http://python.org) and pip when
   # not already installed.

@@ -95,6 +95,7 @@ def run_tests(self):
         '[NAME_REDACTED] :: Python :: 3.4',
         '[NAME_REDACTED] :: Python :: 3.5',
         '[NAME_REDACTED] :: Python :: 3.6',
+        '[NAME_REDACTED] :: Python :: 3.7',
         '[NAME_REDACTED] :: Python :: Implementation :: CPython',
         '[NAME_REDACTED] :: Python :: Implementation :: PyPy'
     ),

Alright, I think this looks good to go. Thanks for seeing this all the way through, [USER_REDACTED]!

This could drop the `|` and fit on one line

Same.

Same.

Same.

Same.

Why do we need sudo here and does this conflict with the sudo: False declaration above?

Could we get a quick explanation of the benefits/trade offs of moving from the matrix approach to this new setup?

No, it does not. This is the only way to use xenial on Travis is to use sudo on this job alone. And xenial is the only target that has true Python 3.7. So it's not ideal (because enabling sudo is *slow*) but it's the best we have right now

You mean moving from the old configuration to the more explicit matrix definition? Other than greater flexibility?


Documentation and tests for form-encoded multivalued elements as lists

@@ -188,3 +188,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])

@@ -244,12 +244,16 @@ dictionary of data will automatically be form-encoded when the request is made::
       ...
     }

-You can also pass a list of tuples to the ``data`` argument. This is particularly
-useful when the form has multiple elements that use the same key::
-
-    >>> payload = (('key1', 'value1'), ('key1', 'value2'))
-    >>> r = requests.post('http://httpbin.org/post', data=payload)
-    >>> print(r.text)
+The ``data`` argument can also have multiple values for each key. This can be
+done by making ``data`` either a list of tuples or a dictionary with lists
+as values. This is particularly useful when the form has multiple elements that
+use the same key::
+
+    >>> payload_tuples = [('key1', 'value1'), ('key1', 'value2')]
+    >>> r1 = requests.post('http://httpbin.org/post', data=payload_tuples)
+    >>> payload_dict = {'key1': ['value1', 'value2']}
+    >>> r2 = requests.post('http://httpbin.org/post', data=payload_dict)
+    >>> print(r1.text)
     {
       ...
       "form": {
@@ -260,6 +264,8 @@ useful when the form has multiple elements that use the same key::
       },
       ...
     }
+    >>> r1.text == r2.text
+    True

 There are times that you may want to send data that is not form-encoded. If
 you pass in a ``string`` instead of a ``dict``, that data will be posted directly.

@@ -868,6 +868,12 @@ def test_urlencoded_get_query_multivalued_param(self, httpbin):
         assert r.status_code == 200
         assert r.url == httpbin('get?test=foo&test=baz')

+    def test_form_encoded_post_query_multivalued_element(self, httpbin):
+        r = requests.Request(method='POST', url=httpbin('post'),
+                             data=dict(test=['foo', 'baz']))
+        prep = r.prepare()
+        assert prep.body == 'test=foo&test=baz'
+
     def test_different_encodings_dont_break_post(self, httpbin):
         r = requests.post(httpbin('post'),
             data={'stuff': json.dumps({'a': 123})},

Hi [USER_REDACTED], thanks for contributing to Requests! I actually didn't know this would work, so thanks for teaching me something new.

I left a quick comment about consolidating this with the current tuple example. Let me know what you think about that and we can work forward from there.

I think this looks good, thanks [USER_REDACTED]!

It may be better to roll this into the example above. They are both performing the same action, just with different syntactic sugar.

I rolled the examples into one in c61f36e. Does that look like what you were thinking about?


Separate collections from collections.abc

In Python 3.7, there is a warning that the abstract bases in collections.abc will no longer be accessible through the regular collections module in Python 3.8.  This patch future-proofs requests.

I wasn't able to run the test suite locally, so this needs a thorough test going forward.

@@ -43,6 +43,7 @@
     import cookielib
     from Cookie import Morsel
     from StringIO import StringIO
+    from collections import Callable, Mapping, MutableMapping

     from urllib3.packages.ordered_dict import OrderedDict

@@ -60,6 +61,7 @@
     from http.cookies import Morsel
     from io import StringIO
     from collections import OrderedDict
+    from collections.abc import Callable, Mapping, MutableMapping

     builtin_str = str
     str = str

@@ -12,10 +12,9 @@
 import copy
 import time
 import calendar
-import collections

 from ._internal_utils import to_native_string
-from .compat import cookielib, urlparse, urlunparse, Morsel
+from .compat import cookielib, urlparse, urlunparse, Morsel, MutableMapping

 try:
     import threading
@@ -169,7 +168,7 @@ class CookieConflictError(RuntimeError):
     """

-class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
+class RequestsCookieJar(cookielib.CookieJar, MutableMapping):
     """Compatibility class; is a cookielib.CookieJar, but exposes a dict
     interface.

@@ -7,7 +7,6 @@
 This module contains the primary objects that power Requests.
 """

-import collections
 import datetime
 import sys

@@ -37,6 +36,7 @@
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
     iter_slices, guess_json_utf, super_len, check_header_validity)
 from .compat import (
+    Callable, Mapping,
     cookielib, urlunparse, urlsplit, urlencode, str, bytes,
     is_py2, chardet, builtin_str, basestring)
 from .compat import json as complexjson
@@ -174,10 +174,10 @@ def register_hook(self, event, hook):
         if event not in self.hooks:
             raise ValueError('Unsupported event specified, with event name "%s"' % (event))

-        if isinstance(hook, collections.Callable):
+        if isinstance(hook, Callable):
             self.hooks[event].append(hook)
         elif hasattr(hook, '__iter__'):
-            self.hooks[event].extend(h for h in hook if isinstance(h, collections.Callable))
+            self.hooks[event].extend(h for h in hook if isinstance(h, Callable))

     def deregister_hook(self, event, hook):
         """Deregister a previously registered hook.
@@ -461,7 +461,7 @@ def prepare_body(self, data, files, json=None):

         is_stream = all([
             hasattr(data, '__iter__'),
-            not isinstance(data, (basestring, list, tuple, collections.Mapping))
+            not isinstance(data, (basestring, list, tuple, Mapping))
         ])

         try:

@@ -10,11 +10,10 @@
 import os
 import sys
 import time
-from collections import Mapping
 from datetime import timedelta

 from .auth import _basic_auth_str
-from .compat import cookielib, is_py3, OrderedDict, urljoin, urlparse
+from .compat import cookielib, is_py3, OrderedDict, urljoin, urlparse, Mapping
 from .cookies import (
     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT

@@ -7,16 +7,14 @@
 Data structures that power Requests.
 """

-import collections
+from .compat import OrderedDict, Mapping, MutableMapping

-from .compat import OrderedDict

-
-class CaseInsensitiveDict(collections.MutableMapping):
+class CaseInsensitiveDict(MutableMapping):
     """A case-insensitive ``dict``-like object.

     Implements all methods and operations of
-    ``collections.MutableMapping`` as well as dict's ``copy``. Also
+    ``MutableMapping`` as well as dict's ``copy``. Also
     provides ``lower_items``.

     All keys are expected to be strings. The structure remembers the
@@ -71,7 +69,7 @@ def lower_items(self):
         )

     def __eq__(self, other):
-        if isinstance(other, collections.Mapping):
+        if isinstance(other, Mapping):
             other = CaseInsensitiveDict(other)
         else:
             return NotImplemented

@@ -9,7 +9,6 @@
 """

 import codecs
-import collections
 import contextlib
 import io
 import os
@@ -29,7 +28,7 @@
 from .compat import (
     quote, urlparse, bytes, str, OrderedDict, unquote, getproxies,
     proxy_bypass, urlunparse, basestring, integer_types, is_py3,
-    proxy_bypass_environment, getproxies_environment)
+    proxy_bypass_environment, getproxies_environment, Mapping)
 from .cookies import cookiejar_from_dict
 from .structures import CaseInsensitiveDict
 from .exceptions import (
@@ -301,7 +300,7 @@ def to_key_val_list(value):
     if isinstance(value, (str, bytes, bool, int)):
         raise ValueError('cannot encode objects that are not 2-tuples')

-    if isinstance(value, collections.Mapping):
+    if isinstance(value, Mapping):
         value = value.items()

     return list(value)

@@ -29,6 +29,7 @@
 from requests.sessions import SessionRedirectMixin
 from requests.models import urlencode
 from requests.hooks import default_hooks
+from requests.compat import MutableMapping

 from .compat import StringIO, u
 from .utils import override_environ
@@ -726,7 +727,7 @@ def test_POSTBIN_GET_POST_FILES_WITH_DATA(self, httpbin):
             requests.post(url, files=['bad file data'])

     def test_post_with_custom_mapping(self, httpbin):
-        class CustomMapping(collections.MutableMapping):
+        class CustomMapping(MutableMapping):
             def __init__(self, *args, **kwargs):
                 self.data = dict(*args, **kwargs)

Hi [USER_REDACTED], thanks for contributing to Requests! This looks like it will resolve #4494 by [NAME_REDACTED] for Python 3.7. I'd like another maintainer to approve this PR before merging, but I believe this looks good from my end.

[USER_REDACTED] on which versions of Python 3 will this actually succeed on? It was my impression that `collections.abc` was introduced in 3.6 or 3.7.

[USER_REDACTED] I'd tested it back to 3.4 without issue, so I think this should work.

This works back to Python 3.3.


Clarify docs that Requests always uses certs from certifi

Since commit 0d7de6430eef0cf09f9662746daf0c28d83f144e, certifi is always used for certificates. Certify became a hard dependency of the package in 628633143d5b8590b1dbdf5371fe81fb8250dffd.

Now update the docs to clarify that Request will always use certificates from certifi.

@@ -274,15 +274,14 @@ If you specify a wrong path or an invalid cert, you'll get a SSLError::
 CA Certificates
 ---------------

-By default, Requests bundles a set of root CAs that it trusts, sourced from the
-`Mozilla trust store`_. However, these are only updated once for each Requests
-version. This means that if you pin a Requests version your certificates can
-become extremely out of date.
-
-[NAME_REDACTED] version 2.4.0 onwards, Requests will attempt to use certificates
-from `certifi`_ if it is present on the system. This allows for users to update
-their trusted certificates without having to change the code that runs on their
-system.
+Requests uses certificates from the package `certifi`_. This allows for users
+to update their trusted certificates without changing the version of Requests.
+
+Before version 2.16, Requests bundled a set of root CAs that it trusted,
+sourced from the `Mozilla trust store`_. The certificates were only updated
+once for each Requests version. When ``certifi`` was not installed, this led to
+extremely out-of-date certificate bundles when using significantly older
+versions of Requests.

 For the sake of security we recommend upgrading certifi frequently!

If you feel like it, we could probably remove "Starting with version 2.4.0, certifi became an optional dependency and Requests would attempt to use its certificates if it was present. Since version 2.16, certifi is a mandatory dependency." but I think this looks good. Thanks [USER_REDACTED]!

Alright, I think this looks good. Thanks [USER_REDACTED]!

They would be out of date if and only if users didn't separately install certifi. This is a blanket statement that is plainly untrue.

This is why I included "could". It is true under some conditions, but not others.

How would you like to see it altered?

What about?

> This meant that the certificates of a pinned version of Requests could become extremely out of date if certi was not installed.

Or should the sentence be removed entirely? Any other suggestions?

I think the `could` is too subtle. What about:

```
When `certifi` was not installed, this led to extremely out-of-date certificate bundles when using significantly older versions of Requests.
```

Sounds good and works for me. I've changed the wording as suggested in the latest revision of the PR.


append previous url fragment on redirect

This PR should address #4443 by [NAME_REDACTED] fragment to subsequent redirects, unless a new fragment is provided in the Location header. This behaviour should bring us into compliance with [RFC 7230 § 7.1.2](https://tools.ietf.org/html/rfc7231#section-7.1.2). I also wrote a test to confirm we don't send fragment information to new servers, our underlying implementation strips that.

One outstanding question I do have though is about chained redirects.

-----

Let's say we make a request (`http://url#alice`) and get a 302 response (`http://new_url#bob`). We would leave `#bob` as the fragment for the second request. When we request `http://new_url#bob`, we get back another 301 to `http://final_url/`. Do we append `alice` or `bob` at this point?

I went with the assumption that the "original" request is the first one in the chain. I could see an argument that all requests are stateless though, and that the `new_url#bob` is the only originating request in scope for the second redirect.

@@ -123,6 +123,7 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,
         hist = []  # keep track of history

         url = self.get_redirect_target(resp)
+        previous_fragment = urlparse(req.url).fragment
         while url:
             prepared_request = req.copy()

@@ -147,8 +148,12 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,
                 parsed_rurl = urlparse(resp.url)
                 url = '%s:%s' % (to_native_string(parsed_rurl.scheme), url)

-            # The scheme should be lower case...
+            # Normalize url case and attach previous fragment if needed (RFC 7231 7.1.2)
             parsed = urlparse(url)
+            if parsed.fragment == '' and previous_fragment:
+                parsed = parsed._replace(fragment=previous_fragment)
+            elif parsed.fragment:
+                previous_fragment = parsed.fragment
             url = parsed.geturl()

             # Facilitate relative 'location' headers, as allowed by RFC 7231.

@@ -235,3 +235,75 @@ def redirect_resp_handler(sock):
         assert r.url == u'{0}/{1}'.format(url, expected_path.decode('ascii'))

         close_server.set()
+
+def test_fragment_not_sent_with_request():
+    """Verify that the fragment portion of a URI isn't sent to the server."""
+    def response_handler(sock):
+        req = consume_socket_content(sock, timeout=0.5)
+        sock.send(
+            b'HTTP/1.1 200 OK\r\n'
+            b'Content-Length: '+bytes(len(req))+b'\r\n'
+            b'\r\n'+req
+        )
+
+    close_server = threading.Event()
+    server = Server(response_handler, wait_to_close_event=close_server)
+
+    with server as (host, port):
+        url = 'http://{0}:{1}/path/to/thing/#view=edit&token=hunter2'.format(host, port)
+        r = requests.get(url)
+        raw_request = r.content
+
+        assert r.status_code == 200
+        headers, body = raw_request.split(b'\r\n\r\n', 1)
+        status_line, headers = headers.split(b'\r\n', 1)
+
+        assert status_line == b'GET /path/to/thing/ HTTP/1.1'
+        for frag in (b'view', b'edit', b'token', b'hunter2'):
+            assert frag not in headers
+            assert frag not in body
+
+        close_server.set()
+
+def test_fragment_update_on_redirect():
+    """Verify we only append previous fragment if one doesn't exist on new
+    location. If a new fragment is encounterd in a Location header, it should
+    be added to all subsequent requests.
+    """
+
+    def response_handler(sock):
+        consume_socket_content(sock, timeout=0.5)
+        sock.send(
+            b'HTTP/1.1 302 FOUND\r\n'
+            b'Content-Length: 0\r\n'
+            b'Location: /get#relevant-section\r\n\r\n'
+        )
+        consume_socket_content(sock, timeout=0.5)
+        sock.send(
+            b'HTTP/1.1 302 FOUND\r\n'
+            b'Content-Length: 0\r\n'
+            b'Location: /final-url/\r\n\r\n'
+        )
+        consume_socket_content(sock, timeout=0.5)
+        sock.send(
+            b'HTTP/1.1 200 OK\r\n\r\n'
+        )
+
+    close_server = threading.Event()
+    server = Server(response_handler, wait_to_close_event=close_server)
+
+    with server as (host, port):
+        url = 'http://{0}:{1}/path/to/thing/#view=edit&token=hunter2'.format(host, port)
+        r = requests.get(url)
+        raw_request = r.content
+
+        assert r.status_code == 200
+        assert len(r.history) == 2
+        assert r.history[0].request.url == url
+
+        # Verify we haven't overwritten the location with our previous fragment.
+        assert r.history[1].request.url == 'http://{0}:{1}/get#relevant-section'.format(host, port)
+        # Verify previous fragment is used and not the original.
+        assert r.url == 'http://{0}:{1}/final-url/#relevant-section'.format(host, port)
+
+        close_server.set()

@@ -294,6 +294,14 @@ def test_transfer_enc_removal_on_redirect(self, httpbin):
         for header in purged_headers:
             assert header not in next_resp.request.headers

+    def test_fragment_maintained_on_redirect(self, httpbin):
+        fragment = "#view=edit&token=hunter2"
+        r = requests.get(httpbin('redirect-to?url=get')+fragment)
+
+        assert len(r.history) > 0
+        assert r.history[0].request.url == httpbin('redirect-to?url=get')+fragment
+        assert r.url == httpbin('get')+fragment
+
     def test_HTTP_200_OK_GET_WITH_PARAMS(self, httpbin):
         heads = {'User-agent': 'Mozilla/5.0'}

Once you rebase this, [USER_REDACTED] it should be good to merge. 🎉

I'm confused how this import is working right now. Maybe it's too late for me to be reviewing code?

😬 this is me running `git add tests` and not paying attention to garbage in the directory. Disregard this file.

I await your force-push to remove it 😄

Ok, we should be updated now.

Can we do `parsed._replace(fragment='previous_fragment')`?

My concern is that `url` might end with `#` which should be equivalent to `parsed.fragment == ''`. What do we do there?

Yep, this is definitely a gap. It looks like `geturl()` will also strip empty fragments from redirect urls which probably isn’t the behavior we want.

I’m a little leery of using private methods, but given that `_replace` has been stable for a decade it’s probably safe. I’m wondering given the trailing hash issue if it may be easier to append the fragment option (`’’`, `#`, `#fragment`) after we call `geturl`. I’ll try to come up with something tonight.

`_replace` is actually part of `namedtuple` which `ParseResult` inherits from. If we wanted to, we could switch wholesale to github.com/hyper/rfc3986 which has a proper public API for this but that is definitely more work than this.


Refs #36644 -- Documented no pk ordering in first()/last() after empty order_by().

#### Trac ticket number
ticket-36644

#### Branch description
Although I still think we can regard this as a bugfix, on second thought I think it deserves just a little documentation to clarify.

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [x] **No AI tools were used** in preparing this PR.
- [ ] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

#### Checklist
- [x] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [x] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -2774,8 +2774,9 @@ the direction is changed.
 *Asynchronous version*: ``afirst()``

 Returns the first object matched by [NAME_REDACTED], or ``None`` if there
-is no matching object. If the ``QuerySet`` has no ordering defined, then the
-queryset is automatically ordered by [NAME_REDACTED] can affect
+is no matching object. If the ``QuerySet`` has no ordering defined (and has not
+had ordering forcibly cleared by calling :meth:`order_by` with no arguments),
+then the queryset is automatically ordered by [NAME_REDACTED] can affect
 aggregation results as described in :ref:`aggregation-ordering-interaction`.

 Example::
@@ -2790,6 +2791,12 @@ equivalent to the above example::
     except IndexError:
         p = None

+.. versionchanged:: 6.1
+
+    ``first()`` and :meth:`last` no longer order by [NAME_REDACTED]
+    ordering has been forcibly cleared by calling :meth:`order_by` with no
+    arguments.
+
 ``last()``
 ~~~~~~~~~~

@@ -422,6 +422,10 @@ Miscellaneous
   ``null``, to match the behavior of :lookup:`exact=None <exact>` on key
   transforms. Previously, it was interpreted as an :lookup:`isnull` lookup.

+* :meth:`~.QuerySet.first` and :meth:`~.QuerySet.last` no longer order by the
+  primary key when a ``QuerySet``'s ordering has been forcibly cleared by
+  calling :meth:`~.QuerySet.order_by` with no arguments.
+
 .. _deprecated-features-6.1:

 Features deprecated in 6.1

This reads a bit better to me like this:

```suggestion
is no matching object. If the ``QuerySet`` has no ordering defined (and has not
had ordering forcibly cleared by calling :meth:`order_by` with no arguments),
```

I think this reads a bit better like this:

```suggestion
    ``first()`` and :meth:`last` no longer order by [NAME_REDACTED]
    ordering has been forcibly cleared by calling :meth:`order_by` with no
    arguments.
```


Clarified optional nature of [NAME_REDACTED].

It's not clear that CLAs are needed to ensure contributors are assenting to our license (the "inbound=outbound" agreement), but we can keep them around for contributors who would like to (or are required by [NAME_REDACTED]) to submit one, without investing additional resources in checking every single contribution.

See https://forum.djangoproject.com/t/cla-vs-dco-for-django-contributors/42399 and recent board minutes.

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [x] **No AI tools were used** in preparing this PR.
- [ ] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

#### TODO
Update flatpages on the website:
- [x] https://www.djangoproject.com/foundation/cla/faq/
- [x] https://www.djangoproject.com/foundation/cla/

@@ -69,8 +69,8 @@ Sign the [NAME_REDACTED]
 --------------------------------------

 The code that you write belongs to you or your employer. If your contribution
-is more than one or two lines of code, you need to sign the `CLA`_. See the
-`[NAME_REDACTED] FAQ`_ for a more thorough explanation.
+is more than one or two lines of code, you have the option to sign the `CLA`_.
+See the `[NAME_REDACTED] FAQ`_ for a more thorough explanation.

 .. _CLA: https://www.djangoproject.com/foundation/cla/
 .. _Contributor License Agreement FAQ: https://www.djangoproject.com/foundation/cla/faq/

@@ -59,11 +59,10 @@ and time availability), claim it by [NAME_REDACTED]:
 * Finally click the "Submit changes" button at the bottom to save.

 .. note::
-    The Django software foundation requests that anyone contributing more than
-    a :ref:`trivial change <trivial-change>`, to Django sign and submit a
-    `[NAME_REDACTED]`_, this ensures that the Django Software
-    Foundation has clear license to all contributions allowing for a clear
-    license for all users.
+    If your change is not :ref:`trivial <trivial-change>`, you have the option
+    to sign and submit a `[NAME_REDACTED]`_ clarifying the status
+    of your contribution. This ensures that the [NAME_REDACTED] has
+    clear license to your contribution.

 .. _Login using your GitHub account: https://code.djangoproject.com/github/login
 .. _Create an account: https://www.djangoproject.com/accounts/register/
@@ -508,7 +507,8 @@ All tickets
 * Is the pull request a single squashed commit with a message that follows our
   :ref:`commit message format <committing-guidelines>`?
 * Are you the patch author and a new contributor? Please add yourself to the
-  :source:`AUTHORS` file and submit a `[NAME_REDACTED]`_.
+  :source:`AUTHORS` file. At your option, submit a
+  `[NAME_REDACTED]`_.
 * Does this have an accepted ticket on Trac? All contributions require a ticket
   unless the :ref:`change is considered trivial <trivial-change>`.

LGTM overall! added some small suggestions. Thanks! 🌟

Should we be more lenient and say:

```suggestion
    If your change is not :ref:`trivial <trivial-change>`, you have the option to
```

Same here?


Refs #33579 -- Added extra tests for NotUpdated exception.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number. -->
<!-- Or delete the line and write "N/A" if this is a trivial PR. -->

ticket-33579

#### Branch description
When `NotUpdated` was added in ab148c02cedbac492f29930dcd5346e1af052635, these additional tests that have equivalents for the `DoesNotExist` and `MultipleObjectsReturned` exceptions were missed.

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [x] **No AI tools were used** in preparing this PR.
- [ ] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

#### Checklist
- [x] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [x] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -449,6 +449,12 @@ def test_inherited_multiple_objects_returned_exception(self):
         with self.assertRaises(Place.MultipleObjectsReturned):
             Restaurant.objects.get()

+    def test_inherited_not_updated_exception(self):
+        # NotUpdated is also inherited.
+        obj = Restaurant(id=999)
+        with self.assertRaises(Place.NotUpdated):
+            obj.save(update_fields={"name"})
+
     def test_related_objects_for_inherited_models(self):
         # Related objects work just as they normally do.
         s1 = Supplier.objects.create(name="[NAME_REDACTED]", address="123 [NAME_REDACTED]")

@@ -2,7 +2,7 @@
 from django.contrib.auth.models import User as AuthUser
 from django.contrib.contenttypes.models import ContentType
 from django.core import checks, management
-from django.db import DEFAULT_DB_ALIAS, models
+from django.db import DEFAULT_DB_ALIAS, models, transaction
 from django.db.models import signals
 from django.test import TestCase, override_settings
 from django.test.utils import isolate_apps
@@ -104,8 +104,8 @@ def test_correct_type_proxy_of_proxy(self):

     def test_proxy_included_in_ancestors(self):
         """
-        Proxy models are included in the ancestors for a model's DoesNotExist
-        and MultipleObjectsReturned
+        Proxy models are included in the ancestors for a model's DoesNotExist,
+        MultipleObjectsReturned, and NotUpdated
         """
         Person.objects.create(name="[NAME_REDACTED]")
         MyPerson.objects.create(name="Bazza del Frob")
@@ -118,6 +118,8 @@ def test_proxy_included_in_ancestors(self):
             MyPersonProxy.objects.get(id__lt=max_id + 1)
         with self.assertRaises(Person.DoesNotExist):
             StatusPerson.objects.get(name="Zathras")
+        with self.assertRaises(Person.NotUpdated), transaction.atomic():
+            StatusPerson(id=999).save(update_fields={"name"})

         StatusPerson.objects.create(name="[NAME_REDACTED].")
         StatusPerson.objects.create(name="[NAME_REDACTED].")

@@ -70,6 +70,10 @@ def test_multipleobjectsreturned_class(self):
         klass = Event.MultipleObjectsReturned
         self.assertIs(pickle.loads(pickle.dumps(klass)), klass)

+    def test_not_updated_class(self):
+        klass = Event.NotUpdated
+        self.assertIs(pickle.loads(pickle.dumps(klass)), klass)
+
     def test_forward_relatedobjectdoesnotexist_class(self):
         # ForwardManyToOneDescriptor
         klass = Event.group.RelatedObjectDoesNotExist

Curiously this works for `update_fields`, but not for `force_update` which gets overridden somehow as part of the MTI handling. Either that's a bug, or it needs to be documented... 🤔

Ditto.

What exactly do you mean? What doesn't work?

Sorry - Using `obj.save(force_update=True)` doesn't raise `NotUpdated` where we are using multi-table inheritance because the parent(s) is inserted first and that overrides the `force_insert` flag on the child:

https://github.com/[USER_REDACTED]ngo/django/blob/24a14860ced5c456522d69c16afd2c631cc0456f/django/db/models/base.py#L980-L995

This means that `_save_table()` here is called with both `force_insert` and `force_update` set to `True`, but we don't do any further validation from this point onward. As `force_insert` is `True`, all handling of `force_update` is then ignored in `_save_table()` as we don't take the branch that handles that.

When using `update_fields`, however, that is passed through to `_save_parents()` which then calls `_save_table()` and handles it properly.

I suspect the fix would be to pass `force_update` through `_save_parents()`, but I haven't looked at that yet.

That seems like a legitimate bug, if that's the case. But oh dear, that's gonna break people's code :grimacing:

It might be possible to fix - we'd need to raise a deprecation warning first for a number of releases. Anyway - will try to write that up as another ticket later.

That's a nice catch but I suspect the intersections of users that

1. Are creating model instances pointing at primary keys they expect to exist
2. Are making use of MTI
3. Are passing `force_update=True`

Is pretty small which might explain why this slipped under the radar for so long (the ORM has been silently doing inserts in this case for a long time)

If we are going to address this issue we should also test interactions with `force_insert: tuple[type[Model]]` and `force_update=True` as I assume it's not a well covered area either.

This causes the next operation to fail on databases that don't support savepoints:
```
======================================================================
ERROR: test_proxy_included_in_ancestors (proxy_models.tests.ProxyModelTests.test_proxy_included_in_ancestors)
Proxy models are included in the ancestors for a model's DoesNotExist,
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.12/unittest/case.py", line 634, in run
    self._callTestMethod(testMethod)
    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
    if method() is not None:
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/tests/proxy_models/tests.py", line 124, in test_proxy_included_in_ancestors
    StatusPerson.objects.create(name="[NAME_REDACTED].")
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/query.py", line 690, in create
    obj.save(force_insert=True, using=self.db)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/base.py", line 896, in save
    self.save_base(
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/base.py", line 985, in save_base
    parent_inserted = self._save_parents(
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/base.py", line 1038, in _save_parents
    updated = self._save_table(
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/base.py", line 1191, in _save_table
    results = self._do_insert(
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/base.py", line 1243, in _do_insert
    return manager._insert(
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/query.py", line 2013, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/models/sql/compiler.py", line 1939, in execute_sql
    cursor.execute(sql, params)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/backends/utils.py", line 99, in _execute
    self.db.validate_no_broken_transaction()
    ^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-snowflake/django-snowflake/django_repo/django/db/backends/base/base.py", line 521, in validate_no_broken_transaction
    raise TransactionManagementError(
    ^^^^^^^^^^^^^^^^^
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
```
Possible solutions:
- Skip this block if `DatabaseFeatures.uses_savepoints=False`.
- Move this block to a separate test.

The whole test was a bit overloaded, so I split each thing being tested into separate tests: https://github.com/[USER_REDACTED]ngo/django/pull/20669


Fixed #34352 -- Unified terms in Signals docs.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number. -->
<!-- Or delete the line and write "N/A" if this is a trivial PR. -->

ticket-34352

#### Branch description
removed callback from the signals document for consistency & to avoid confusion

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [x] **No AI tools were used** in preparing this PR.
- [ ] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

#### Checklist
- [x] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [x] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -18,15 +18,15 @@ changes::
     from django.core.signals import setting_changed

-    def my_callback(sender, **kwargs):
+    def my_receiver(sender, **kwargs):
         print("Setting changed!")

     class MyAppConfig(AppConfig):
         ...

         def ready(self):
-            setting_changed.connect(my_callback)
+            setting_changed.connect(my_receiver)

 Django's :doc:`built-in signals </ref/signals>` let user code get notified of
 certain actions.
@@ -58,7 +58,7 @@ the order they were registered.
     :param sender: Specifies a particular sender to receive signals from. See
         :ref:`connecting-to-specific-signals` for more information.

-    :param weak: Django stores signal handlers as weak references by
+    :param weak: Django stores signal receivers as weak references by
         default. Thus, if your receiver is a local function, it may be
         garbage collected. To prevent this, pass ``weak=False`` when you call
         the signal's ``connect()`` method.
@@ -79,19 +79,19 @@ Receiver functions
 First, we need to define a receiver function. A receiver can be any Python
 function or method::

-    def my_callback(sender, **kwargs):
+    def my_receiver(sender, **kwargs):
         print("Request finished!")

 Notice that the function takes a ``sender`` argument, along with wildcard
-keyword arguments (``**kwargs``); all signal handlers must take these
+keyword arguments (``**kwargs``); all signal receivers must take these
 arguments.

 We'll look at senders :ref:`a bit later <connecting-to-specific-signals>`, but
 right now look at the ``**kwargs`` argument. All signals send keyword
 arguments, and may change those keyword arguments at any time. In the case of
 :data:`~django.core.signals.request_finished`, it's documented as sending no
 arguments, which means we might be tempted to write our signal handling as
-``my_callback(sender)``.
+``my_receiver(sender)``.

 This would be wrong -- in fact, Django will throw an error if you do so. That's
 because at any point arguments could get added to the signal and your receiver
@@ -100,7 +100,7 @@ must be able to handle those new arguments.
 Receivers may also be asynchronous functions, with the same signature but
 declared using ``async def``::

-    async def my_callback(sender, **kwargs):
+    async def my_receiver(sender, **kwargs):
         await asyncio.sleep(5)
         print("Request finished!")

@@ -118,7 +118,7 @@ manual connect route::

     from django.core.signals import request_finished

-    request_finished.connect(my_callback)
+    request_finished.connect(my_receiver)

 Alternatively, you can use a :func:`receiver` decorator:

@@ -135,24 +135,24 @@ Here's how you connect with the decorator::

     [USER_REDACTED](request_finished)
-    def my_callback(sender, **kwargs):
+    def my_receiver(sender, **kwargs):
         print("Request finished!")

-Now, our ``my_callback`` function will be called each time a request finishes.
+Now, our ``my_receiver`` function will be called each time a request finishes.

 .. admonition:: Where should this code live?

     Strictly speaking, signal handling and registration code can live anywhere
     you like, although it's recommended to avoid the application's root module
     and its ``models`` module to minimize side-effects of importing code.

-    In practice, signal handlers are usually defined in a ``signals``
+    In practice, signal receivers are usually defined in a ``signals``
     submodule of the application they relate to. Signal receivers are
     connected in the :meth:`~django.apps.AppConfig.ready` method of your
     application :ref:`configuration class <configuring-applications-ref>`. If
     you're using the :func:`receiver` decorator, import the ``signals``
     submodule inside :meth:`~django.apps.AppConfig.ready`, this will implicitly
-    connect signal handlers::
+    connect signal receivers::

         from django.apps import AppConfig
         from django.core.signals import request_finished
@@ -162,11 +162,11 @@ Now, our ``my_callback`` function will be called each time a request finishes.
             ...

             def ready(self):
-                # Implicitly connect signal handlers decorated with [USER_REDACTED].
+                # Implicitly connect signal receivers decorated with [USER_REDACTED].
                 from . import signals

                 # Explicitly connect a signal handler.
-                request_finished.connect(signals.my_callback)
+                request_finished.connect(signals.my_receiver)

 .. note::

@@ -228,7 +228,7 @@ bound to the signal once for each unique ``dispatch_uid`` value::

     from django.core.signals import request_finished

-    request_finished.connect(my_callback, dispatch_uid="my_unique_identifier")
+    request_finished.connect(my_receiver, dispatch_uid="my_unique_identifier")

 .. _defining-and-sending-signals:

I would make a single exception here and continue to use "callback", as otherwise, we're explaining the term X by [NAME_REDACTED] By using a synonym, there's a chance a user can rely on some prior knowledge or gain new knowledge.

Another possibility is to use both, e.g. "The receiver (callback) function that will ..."


Fixed #36890 -- [NAME_REDACTED](distinct=True) on SQLite with the default delimiter.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number. -->
<!-- Or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36890

#### Branch description
Currently, the generic StringAgg function raises an error on SQLite when distinct=True

This change enables support for distinct aggregation on SQLite by [NAME_REDACTED] in the generated SQL when it matches the default comma.

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [ ] **No AI tools were used** in preparing this PR.
- [X] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

I used [NAME_REDACTED] to assist with writing tests.

#### Checklist
- [X] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [X] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [X] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [X] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [X] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -369,6 +369,24 @@ def as_mysql(self, compiler, connection, **extra_context):
         return sql, (*params, *delimiter_params)

     def as_sqlite(self, compiler, connection, **extra_context):
+        if (
+            self.distinct
+            and isinstance(self.delimiter.value, Value)
+            and self.delimiter.value.value == ","
+        ):
+            clone = self.copy()
+            source_expressions = clone.get_source_expressions()
+            clone.set_source_expressions(
+                source_expressions[:1] + source_expressions[2:]
+            )
+
+            return clone.as_sql(
+                compiler,
+                connection,
+                function="GROUP_CONCAT",
+                **extra_context,
+            )
+
         if connection.get_database_version() < (3, 44):
             return self.as_sql(
                 compiler,

@@ -4175,7 +4175,14 @@ by [NAME_REDACTED]
     .. attribute:: delimiter

         A ``Value`` or expression representing the string that should separate
-        each of the values. For example, ``Value(",")``.
+        each of the values. For example, ``Value(",")``. (On SQLite, the
+        literal delimiter ``Value(",")`` is the only delimiter compatible with
+        ``distinct=True``.)
+
+        .. versionchanged:: 6.1
+
+            Support for using ``distinct=True`` with a delimiter of
+            ``Value(",")`` on SQLite was added.

 Query-related tools
 ===================

@@ -284,6 +284,9 @@ Models
 * The :data:`~django.db.models.signals.m2m_changed` signal now receives a
   ``raw`` argument.

+* :class:`~django.db.models.StringAgg` now supports ``distinct=True`` on SQLite
+  when using the default delimiter ``Value(",")`` only.
+
 Pagination
 ~~~~~~~~~~

@@ -3,6 +3,7 @@
 import re
 from decimal import Decimal
 from itertools import chain
+from unittest import skipUnless

 from django.core.exceptions import FieldError
 from django.db import NotSupportedError, connection
@@ -579,6 +580,16 @@ def test_distinct_on_stringagg(self):
         )
         self.assertCountEqual(books["ratings"].split(","), ["3", "4", "4.5", "5"])

+    [USER_REDACTED](connection.vendor == "sqlite", "Special default case for SQLite.")
+    def test_distinct_on_stringagg_sqlite_special_case(self):
+        """
+        Value(",") is the only delimiter usable on SQLite with distinct=True.
+        """
+        books = Book.objects.aggregate(
+            ratings=StringAgg(Cast(F("rating"), CharField()), Value(","), distinct=True)
+        )
+        self.assertCountEqual(books["ratings"].split(","), ["3.0", "4.0", "4.5", "5.0"])
+
     [USER_REDACTED]("supports_aggregate_distinct_multiple_argument")
     def test_raises_error_on_multiple_argument_distinct(self):
         message = (
@@ -589,7 +600,7 @@ def test_raises_error_on_multiple_argument_distinct(self):
             Book.objects.aggregate(
                 ratings=StringAgg(
                     Cast(F("rating"), CharField()),
-                    Value(","),
+                    Value(";"),
                     distinct=True,
                 )
             )

Thanks 👍

This lacks `order_by` and `filter`

```suggestion
        if (
            self.distinct
            and isinstance(self.delimiter.value, Value)
            and self.delimiter.value.value == ","
        ):
            return ...
```

It's too bad we can't provide a better hint for SQLite users in this case, but I don't know what we can do about it without departing from our existing patterns.

Simon's prior feedback for you was to avoid asserting over the exact stringified values, but once I realized that we _don't_ want this test to run on all databases (what about other databases besides SQLite where supports_aggregate_distinct_multiple_argument is also False, but the default delimiter trick may not work), I realized we were only testing a SQLite specific case.

Given that, I shuffled this test location and made it as parallel as possible to the prior one.


Fixed #36893 -- Serialized elidable kwarg for RunSQL and RunPython operations.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number. -->
<!-- Or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36893

#### Branch description

Fixed an issue where `elidable=True` was not being serialized for Run-SQL and Run-Python operations which caused these operations to lose their elidable status when written to a migration file (e.g., during squashing), reverting to the default `False`.

-Updated deconstruct() for both Run-SQL and Run-Python to correctly include 'elidable' keyword when set to 'True'
-[NAME_REDACTED] are test_run_sql_elidable , test_run_python_elidable and test_operations.py

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [ ] **No AI tools were used** in preparing this PR.
- [x] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.
Assistance provided by [NAME_REDACTED] I used the agent to understand the issue, implement the fix, and add regression tests.
#### Checklist
- [x] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [x] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -93,6 +93,8 @@ def deconstruct(self):
             kwargs["state_operations"] = self.state_operations
         if self.hints:
             kwargs["hints"] = self.hints
+        if self.elidable:
+            kwargs["elidable"] = self.elidable
         return (self.__class__.__qualname__, [], kwargs)

     [USER_REDACTED]
@@ -173,6 +175,8 @@ def deconstruct(self):
             kwargs["atomic"] = self.atomic
         if self.hints:
             kwargs["hints"] = self.hints
+        if self.elidable:
+            kwargs["elidable"] = self.elidable
         return (self.__class__.__qualname__, [], kwargs)

     [USER_REDACTED]

@@ -5536,6 +5536,10 @@ def test_run_sql(self):
         elidable_operation = migrations.RunSQL("SELECT 1 FROM void;", elidable=True)
         self.assertEqual(elidable_operation.reduce(operation, []), [operation])

+        # Test elidable deconstruction
+        definition = elidable_operation.deconstruct()
+        self.assertIs(definition[2]["elidable"], True)
+
     def test_run_sql_params(self):
         """
         #23426 - RunSQL should accept parameters.
@@ -5789,6 +5793,10 @@ def create_shetlandponies(models, schema_editor):
         elidable_operation = migrations.RunPython(inner_method, elidable=True)
         self.assertEqual(elidable_operation.reduce(operation, []), [operation])

+        # Test elidable deconstruction
+        definition = elidable_operation.deconstruct()
+        self.assertIs(definition[2]["elidable"], True)
+
     def test_run_python_invalid_reverse_code(self):
         msg = "RunPython must be supplied with callable arguments"
         with self.assertRaisesMessage(ValueError, msg):

Thanks [USER_REDACTED] and [USER_REDACTED] for the review.

Very nearly there, one small cleanup suggestion. And please update the PR title to match the commit. Thanks.

Thanks 👍

Probs don't need an explicit test for just `elidable` param.
Is there already a test somewhere that checks _all_ serialized params for RunSQL? I'd probs ensure we have one that does that, and include elidable as part of its check.

Thanks for the feedback! I've updated the PR to remove the separate test case. I found that test_run_sqland and test_run_python
were already creating elidable operations for reduction testing, so I added assertions there to verify they also deconstruct correctly with the elidable keyword.

Similar change here.

One assertion for this is fine, also please use `assertIs` for first-party values:
```suggestion
        self.assertIs(definition[2]["elidable"], True)
```


Fixed spell checks in GH docs workflow.

#### Branch description
The spelling check job was passing even with spelling errors because
the system spell checker (`aspell`) was not installed on the [NAME_REDACTED] runner. While `sphinxcontrib.spelling` and `PyEnchant` were
installed via pip, they require a system-level spell checker backend
to function.

Once installed, these errors were reported:

```
WARNING: internals/security.txt:50: : Spell check: runnable: Include a runnable proof of concept.
WARNING: ref/contrib/postgres/search.txt:292: : Spell check: lexeme: an untrusted source. The content of each lexeme is escaped so that any.
WARNING: ref/contrib/postgres/search.txt:295: : Spell check: lexemes: You can combine lexemes with other lexemes using the .
WARNING: ref/contrib/postgres/search.txt:295: : Spell check: lexemes: You can combine lexemes with other lexemes using the .
WARNING: ref/contrib/postgres/search.txt:314: : Spell check: Lexeme: Lexeme objects also support term weighting and prefixes:.
WARNING: ref/models/database-functions.txt:1897: : Spell check: ai: 23ai/26ai (23.9) or later..
WARNING: ref/models/database-functions.txt:1897: : Spell check: ai: 23ai/26ai (23.9) or later..
WARNING: ref/models/expressions.txt:439: : Spell check: positionally:  can be supplied positionally or only by keyword. For.
WARNING: ref/models/fields.txt:1339: : Spell check: ai: PostgreSQL < 18 only supports persisted columns. Oracle < 23ai/26ai (23.7).
WARNING: ref/models/fields.txt:1339: : Spell check: ai: PostgreSQL < 18 only supports persisted columns. Oracle < 23ai/26ai (23.7).
WARNING: ref/models/fields.txt:1344: : Spell check: ai: s was added on Oracle 23ai/26ai.
WARNING: ref/models/fields.txt:1344: : Spell check: ai: s was added on Oracle 23ai/26ai.
WARNING: releases/4.2.21.txt:24: : Spell check: unclosed: exception if it encounters an unusually large number of unclosed opening tags..
WARNING: releases/5.1.9.txt:24: : Spell check: unclosed: exception if it encounters an unusually large number of unclosed opening tags..
WARNING: releases/5.2.1.txt:24: : Spell check: unclosed: exception if it encounters an unusually large number of unclosed opening tags..
WARNING: releases/6.1.txt:244: : Spell check: mistyped: suggestions for mistyped subcommand names and argument choices..
WARNING: releases/6.1.txt:281: : Spell check: ai: Oracle 23ai/26ai (23.7+)..
WARNING: releases/6.1.txt:281: : Spell check: ai: Oracle 23ai/26ai (23.7+)..
WARNING: releases/6.1.txt:343: : Spell check: durations: durations expressed in weeks (.
WARNING: Found 19 misspelled words
build finished with problems, 20 warnings (with warnings treated as errors).
```

#### AI [NAME_REDACTED] (REQUIRED)
<!-- Please select exactly ONE of the following: -->
- [ ] **No AI tools were used** in preparing this PR.
- [X] **If AI tools were used**, I have disclosed which ones, and fully reviewed and verified their output.

I have used [NAME_REDACTED] with model Sonnet 4.5 for debugging the root cause of this issue.

#### Checklist
- [X] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [X] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [X] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [X] The commit message is written in past tense, mentions the ticket number, and ends with a period.

@@ -35,6 +35,8 @@ jobs:
           python-version: '3.14'
           cache: 'pip'
           cache-dependency-path: 'docs/requirements.txt'
+      - name: Install system spell checker
+        run: sudo apt update && sudo apt install -y aspell aspell-en
       - run: python -m pip install -r docs/requirements.txt
       - name: Build docs
         run: |

@@ -178,8 +178,9 @@ Spelling check
 ~~~~~~~~~~~~~~

 Before you commit your docs, it's a good idea to run the spelling checker.
-You'll need to install :pypi:`sphinxcontrib-spelling` first. Then from the
-``docs`` directory, run:
+You'll need to install :pypi:`sphinxcontrib-spelling` first. The spell checker
+also requires a system-level spell checking backend such as `Aspell
+<http://aspell.net/>`__. Then from the ``docs`` directory, run:

 .. console::

@@ -46,8 +46,8 @@ the industry-standard 90 days. Confirmed vulnerabilities with a
 Reporting guidelines
 --------------------

-Include a runnable proof of concept
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Include a working proof of concept
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Please privately share a minimal Django project or code snippet that
 demonstrates the potential vulnerability. Include clear instructions on how to

@@ -438,11 +438,11 @@ return type.

     Built-in database functions (such as
     :class:`~django.db.models.functions.Cast`) vary in whether arguments such
-    as ``output_field`` can be supplied positionally or only by keyword. For
-    ``output_field`` and several other cases, the input ultimately reaches
-    ``Func()`` as a keyword argument, so the advice to avoid constructing
-    keyword arguments from untrusted user input applies as equally to these
-    arguments as it does to ``**extra``.
+    as ``output_field`` can be supplied as positional arguments or only by
+    keyword. For ``output_field`` and several other cases, the input ultimately
+    reaches ``Func()`` as a keyword argument, so the advice to avoid
+    constructing keyword arguments from untrusted user input applies as equally
+    to these arguments as it does to ``**extra``.

 ``Aggregate()`` expressions
 ---------------------------

@@ -242,7 +242,7 @@ [NAME_REDACTED]

 * Management commands now set :class:`~argparse.ArgumentParser`\'s
   ``suggest_on_error`` argument to ``True`` by [NAME_REDACTED] 3.14, enabling
-  suggestions for mistyped subcommand names and argument choices.
+  suggestions for incorrectly typed subcommand names and argument choices.

 * The :djadmin:`loaddata` command now calls
   :data:`~django.db.models.signals.m2m_changed` signals with ``raw=True`` when
@@ -340,7 +340,7 @@ Utilities
 ~~~~~~~~~

 * :func:`~django.utils.dateparse.parse_duration` now supports ISO 8601
-  durations expressed in weeks (``PnW``).
+  time periods expressed in weeks (``PnW``).

 Validators
 ~~~~~~~~~~

@@ -4,6 +4,7 @@ accessors
 Aceh
 admindocs
 affordances
+ai
 [NAME_REDACTED]
 allowlist
@@ -245,6 +246,8 @@ kwargs
 Kyrgyz
 latin
 lawrence
+lexeme
+lexemes
 Libera
 lifecycle
 lifecycles
@@ -531,6 +534,7 @@ unapplied
 unapplying
 uncategorized
 unclaim
+unclosed
 uncopyable
 unencoded
 unencrypted

positional parameters -> positional arguments

Wonderful. Are you up for updating the section on this in writing-documentation.txt that aspell is required?

On mac, sphinxcontrib.spelling exits with an instruction to install enchant, which, at least via homebrew, pulls in aspell. Guess not on linux?

Applied, thanks!

I added a new commit with this adding. Good call! :dart: