doc updates

Various doc updates for clarity. Some of these may not match [USER_REDACTED]'s aesthetic sensibilities, which may require an inversion of the proposed change. I was mainly trying to make things consistent, input on preferences would be greatly appreciated.

Some notes:
- I think at least the faq.rst changes _need_ to be merged because there is still live documentation stating that Requests supports 3.1 and 3.2. I can open a separate PR if we want those changes quicker.
- What was ScraperWiki, is now found at quickcode.io and appears to either be a paid or account-walled service. [NAME_REDACTED] is showing the site as a free service when the documentation was created, so I'm not sure if Kenneth still wants this included. At the very least, the URL should be updated.
- ~~Does Requests support pypy 2.2 officially? faq.rst says yes, but it's other versions were wrong.~~
- While I realize it's common to use `class.method` as a pattern when describing code, I felt like `r.json` is ambiguous when referenced next to `r.text`. Calling exactly `r.json` won't function as described in the provided `ValueError` example. Explicitly defining the call as `r.json()` reduces the chance of misinterpretation.
- api.rst has a pared down set of Exceptions and classes. I assume this is to keep extraneous information from bloating the page to a point of being unhelpful. I do think it may be worth including `BaseAdapter` for easy reference since anyone looking into a custom _non-HTTP_ [NAME_REDACTED] will likely need to start there.

@@ -63,6 +63,9 @@ [NAME_REDACTED]
 .. autoclass:: requests.PreparedRequest
    :inherited-members:

+.. autoclass:: requests.adapters.BaseAdapter
+   :inherited-members:
+
 .. autoclass:: requests.adapters.HTTPAdapter
    :inherited-members:

@@ -1,17 +1,6 @@
 Integrations
 ============

-ScraperWiki
-------------
-
-`ScraperWiki <https://scraperwiki.com/>`_ is an excellent service that allows
-you to run Python, Ruby, and PHP scraper scripts on the web. Now, Requests
-v0.6.1 is available to use in your scrapers!
-
-To give it a try, simply::
-
-    import requests
-
 Python for iOS
 --------------

@@ -41,7 +41,7 @@ Requests currently supports the following versions of Python:
 - Python 3.3
 - Python 3.4
 - Python 3.5
-- PyPy 1.9
+- [NAME_REDACTED] is not officially supported although support is available
 with the `Requests-Toolbelt`_.

@@ -87,11 +87,11 @@ See the :ref:`Session API Docs <sessionapi>` to learn more.
 Request and [NAME_REDACTED]
 ----------------------------

-Whenever a call is made to ``requests.get()`` and friends you are doing two
+Whenever a call is made to ``requests.get()`` and friends, you are doing two
 major things. First, you are constructing a ``Request`` object which will be
 sent off to a server to request or query some resource. Second, a ``Response``
-object is generated once ``requests`` gets a response back from the server.
-The Response object contains all of the information returned by [NAME_REDACTED]
+object is generated once Requests gets a response back from the server.
+The ``Response`` object contains all of the information returned by [NAME_REDACTED] contains the ``Request`` object you created originally. Here is a simple
 request to get some very important information from Wikipedia's servers::

@@ -213,9 +213,7 @@ You can pass ``verify`` the path to a CA_BUNDLE file or directory with certifica

 This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`` environment variable.

-Requests can also ignore verifying the SSL certificate if you set ``verify`` to False.
-
-::
+Requests can also ignore verifying the SSL certificate if you set ``verify`` to False::

     >>> requests.get('https://kennethreitz.com', verify=False)
     <Response [200]>
@@ -242,7 +240,7 @@ If you specify a wrong path or an invalid cert, you'll get a SSLError::
 CA Certificates
 ---------------

-By [NAME_REDACTED] a set of root CAs that it trusts, sourced from the
+By default, Requests bundles a set of root CAs that it trusts, sourced from the
 `Mozilla trust store`_. However, these are only updated once for each Requests
 version. This means that if you pin a Requests version your certificates can
 become extremely out of date.
@@ -266,7 +264,7 @@ [NAME_REDACTED]

 By default, when you make a request, the body of the response is downloaded
 immediately. You can override this behaviour and defer downloading the response
-body until you access the :class:`Response.content <requests.Response.content>`
+body until you access the :attr:`Response.content <requests.Response.content>`
 attribute with the ``stream`` parameter::

     tarball_url = 'https://github.com/[USER_REDACTED]nethreitz/requests/tarball/master'
@@ -279,15 +277,15 @@ remains open, hence allowing us to make content retrieval conditional::
       content = r.content
       ...

-You can further control the workflow by [NAME_REDACTED] :class:`Response.iter_content <requests.Response.iter_content>`
-and :class:`Response.iter_lines <requests.Response.iter_lines>` methods.
+You can further control the workflow by [NAME_REDACTED] :meth:`Response.iter_content() <requests.Response.iter_content>`
+and :meth:`Response.iter_lines() <requests.Response.iter_lines>` methods.
 Alternatively, you can read the undecoded body from the underlying
 urllib3 :class:`urllib3.HTTPResponse <urllib3.response.HTTPResponse>` at
-:class:`Response.raw <requests.Response.raw>`.
+:attr:`Response.raw <requests.Response.raw>`.

 If you set ``stream`` to ``True`` when making a request, Requests cannot
 release the connection back to the pool unless you consume all the data or call
-:class:`Response.close <requests.Response.close>`. This can lead to
+:meth:`Response.close <requests.Response.close>`. This can lead to
 inefficiency with connections. If you find yourself partially reading request
 bodies (or not reading them at all) while using ``stream=True``, you should
 consider using ``contextlib.closing`` (`documented here`_), like this::
@@ -349,11 +347,11 @@ a length) for your body::
     requests.post('http://some.url/chunked', data=gen())

 For chunked encoded responses, it's best to iterate over the data using
-:meth:`Response.iter_content() <requests.models.Response.iter_content>`. In
+:meth:`Response.iter_content() <requests.Response.iter_content>`. In
 an ideal situation you'll have set ``stream=True`` on the request, in which
-case you can iterate chunk-by-chunk by calling ``iter_content`` with a chunk
-size parameter of ``None``. If you want to set a maximum size of the chunk,
-you can set a chunk size parameter to any integer.
+case you can iterate chunk-by-chunk by calling ``iter_content`` with a ``chunk_size``
+parameter of ``None``. If you want to set a maximum size of the chunk,
+you can set a ``chunk_size`` parameter to any integer.

 .. _multipart:
@@ -440,9 +438,10 @@ Requests allows you to use specify your own authentication mechanism.
 Any callable which is passed as the ``auth`` argument to a request method will
 have the opportunity to modify the request before it is dispatched.

-Authentication implementations are subclasses of ``requests.auth.AuthBase``,
+Authentication implementations are subclasses of :class:`AuthBase <requests.auth.AuthBase>`,
 and are easy to define. Requests provides two common authentication scheme
-implementations in ``requests.auth``: ``HTTPBasicAuth`` and ``HTTPDigestAuth``.
+implementations in ``requests.auth``: :class:`HTTPBasicAuth <requests.auth.HTTPBasicAuth>` and
+:class:`HTTPDigestAuth <requests.auth.HTTPDigestAuth>`.

 Let's pretend that we have a web service that will only respond if the
 ``X-Pizza`` header is set to a password value. Unlikely, but just go with it.
@@ -472,11 +471,11 @@ Then, we can make a request using our [NAME_REDACTED]::
 [NAME_REDACTED]
 ------------------

-With :class:`requests.Response.iter_lines()` you can easily
+With :meth:`Response.iter_lines() <requests.Response.iter_lines>` you can easily
 iterate over streaming APIs such as the `[NAME_REDACTED]
 API <https://dev.twitter.com/streaming/overview>`_. Simply
 set ``stream`` to ``True`` and iterate over the response with
-:class:`~requests.Response.iter_lines()`::
+:meth:`~requests.Response.iter_lines()`::

     import json
     import requests
@@ -491,7 +490,7 @@ set ``stream`` to ``True`` and iterate over the response with

 .. warning::

-    :class:`~requests.Response.iter_lines()` is not reentrant safe.
+    :meth:`~requests.Response.iter_lines()` is not reentrant safe.
     Calling this method multiple times causes some of the received data
     being lost. In case you need to call it from multiple places, use
     the resulting iterator object instead::
@@ -669,8 +668,9 @@ commits is POST, which creates a new commit. As we're using the Requests repo,
 we should probably avoid making ham-handed POSTS to it. Instead, let's play
 with the Issues feature of GitHub.

-This documentation was added in response to Issue #482. Given that this issue
-already exists, we will use it as an example. Let's start by [NAME_REDACTED]
+This documentation was added in response to
+`Issue #482 <https://github.com/[USER_REDACTED]nethreitz/requests/issues/482>`_. Given that
+this issue already exists, we will use it as an example. Let's start by [NAME_REDACTED]

 ::

@@ -845,8 +845,8 @@ with the given prefix will use the given [NAME_REDACTED] of the details of implementing a [NAME_REDACTED] are beyond the scope of
 this documentation, but take a look at the next example for a simple SSL use-
-case. For more than that, you might look at subclassing
-``requests.adapters.BaseAdapter``.
+case. For more than that, you might look at subclassing the
+:class:`BaseAdapter <requests.adapters.BaseAdapter>`.

 Example: Specific SSL Version
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -858,10 +858,8 @@ that uses a version that isn't compatible with the default.

 You can use [NAME_REDACTED] for this by [NAME_REDACTED] existing
 implementation of HTTPAdapter, and adding a parameter *ssl_version* that gets
-passed-through to `urllib3`. We'll make a TA that instructs the library to use
-SSLv3:
-
-::
+passed-through to `urllib3`. We'll make a [NAME_REDACTED] that instructs the
+library to use SSLv3::

     import ssl

@@ -904,7 +902,7 @@ [NAME_REDACTED] unusual circumstances you may want to provide headers in an ordered manner. If you pass an ``OrderedDict`` to the ``headers`` keyword argument, that will provide the headers with an ordering. *However*, the ordering of the default headers used by [NAME_REDACTED], which means that if you override default headers in the ``headers`` keyword argument, they may appear out of order compared to other headers in that keyword argument.

-If this is problematic, users should consider setting the default headers on a :class:`Session <requests.Session>` object, by setting :data:`Session <requests.Session.headers>` to a custom ``OrderedDict``. That ordering will always be preferred.
+If this is problematic, users should consider setting the default headers on a :class:`Session <requests.Session>` object, by setting :attr:`Session <requests.Session.headers>` to a custom ``OrderedDict``. That ordering will always be preferred.

 .. _timeouts:

@@ -940,7 +938,7 @@ If the remote server is very slow, you can tell Requests to wait forever for
 a response, by [NAME_REDACTED] a timeout value and then retrieving a cup of
 coffee.

-.. code-block:: python
+::

     r = requests.get('https://github.com', timeout=None)

@@ -27,7 +27,7 @@ Begin by [NAME_REDACTED]::
     >>> import requests

 Now, let's try to get a webpage. For this example, let's get GitHub's public
-timeline ::
+timeline::

     >>> r = requests.get('https://api.github.com/events')

@@ -148,11 +148,11 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::
     >>> r.json()
     [{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...

-In case the JSON decoding fails, ``r.json`` raises an exception. For example, if
+In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 ([NAME_REDACTED]), or if the response contains invalid JSON,
-attempting ``r.json`` raises ``ValueError: No JSON object could be decoded``.
+attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.

-It should be noted that the success of the call to ``r.json`` does **not**
+It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a
 failed response (e.g. error details with HTTP 500). Such JSON will be decoded
 and returned. To check that a request is successful, use
@@ -439,7 +439,7 @@ HEAD.

 We can use the ``history`` property of the Response object to track redirection.

-The :meth:`Response.history <requests.Response.history>` list contains the
+The :attr:`Response.history <requests.Response.history>` list contains the
 :class:`Response <requests.Response>` objects that were created in order to
 complete the request. The list is sorted from the oldest to the most recent
 response.
@@ -504,20 +504,20 @@ Errors and Exceptions
 ---------------------

 In the event of a network problem (e.g. DNS failure, refused connection, etc),
-Requests will raise a :class:`~requests.exceptions.ConnectionError` exception.
+Requests will raise a :exc:`~requests.exceptions.ConnectionError` exception.

 :meth:`Response.raise_for_status() <requests.Response.raise_for_status>` will
-raise an :class:`~requests.exceptions.HTTPError` if the HTTP request
+raise an :exc:`~requests.exceptions.HTTPError` if the HTTP request
 returned an unsuccessful status code.

-If a request times out, a :class:`~requests.exceptions.Timeout` exception is
+If a request times out, a :exc:`~requests.exceptions.Timeout` exception is
 raised.

 If a request exceeds the configured number of maximum redirections, a
-:class:`~requests.exceptions.TooManyRedirects` exception is raised.
+:exc:`~requests.exceptions.TooManyRedirects` exception is raised.

 All exceptions that Requests explicitly raises inherit from
-:class:`requests.exceptions.RequestException`.
+:exc:`requests.exceptions.RequestException`.

 -----------------------

I think there is no value in being specific about PyPy versions: we just support PyPy.

Why was this removed?

:: produces the same output and is used everywhere else in the documentation. I couldn't find any additional HTML markup generated by `.. code-block:: python` v.s. `::` and the [documentation](http://www.sphinx-doc.org/en/stable/markup/code.html) states it's only necessary when displaying a language not in the default config (which is python for Requests).

It looks like this all came from the same commit from Kevin though, so he may have a reason for this that I'm missing.

Great, I'll update that.

One more note on this. It looks like this does affect the GitHub highlighting but not Sphinx (Pygments).

The two previous code snippets above this one do not currently display with any syntax highlighting, but this one does. I don't have a strong opinion on which direction to go with this, but I do think it looks a little odd if highlighting isn't uniform across grouped examples.


adding passthrough in close() for non-urllib3-like Responses

This is minor, but when testing `resolve_redirects` I encountered an `AttributeError` when passing a non-urllib3 Response. This is because the method unconditionally calls [`close`](https://github.com/[USER_REDACTED]nethreitz/requests/blob/master/requests/models.py#L864) on the Response which in turn calls `release_conn` on the `Response.raw` attribute.

If the desire for requests is to maintain support for non-urllib3 objects in custom adapters, I think this would be a simple and useful addition. If urllib3 is going to be a de facto pattern for Responses though, this is probably just unnecessary code bloat.

@@ -870,4 +870,6 @@ def close(self):
         if not self._content_consumed:
             self.raw.close()

-        return self.raw.release_conn()
+        release_conn = getattr(self.raw, 'release_conn', None)
+        if release_conn is not None:
+            release_conn()

@@ -1353,6 +1353,15 @@ def test_response_json_when_content_is_None(self, httpbin):
         with pytest.raises(ValueError):
             r.json()

+    def test_response_without_release_conn(self):
+        """Test `close` call for non-urllib3-like raw objects.
+        Should work when `release_conn` attr doesn't exist on `response.raw`.
+        """
+        resp = requests.Response()
+        resp.raw = StringIO.StringIO('test')
+        assert not resp.raw.closed
+        resp.close()
+        assert resp.raw.closed

 class TestCaseInsensitiveDict:

Can we make some kind of positive assertion here? I don't want us to get to a place where we fix this by having `close` be a no-op without this test failing. =)

I'd rather do an `if` here than create this temporary lambda every single time. =)

Do we really need to return here?

No we don't, I was just trying to change as little of the existing code as needed. I'm fine with removing it.


#3490 removing Content-Type and Transfer-Encoding headers on redirect

This will remove additional headers (`Content-Type` and `Transfer-Encoding`) from non-temporary/non-permanent redirects in `resolve_redirects` to address #3490.

This functionality isn't currently being tested but I'm happy to add a test or two for it if desired.

@@ -140,9 +140,10 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,

             # https://github.com/[USER_REDACTED]nethreitz/requests/issues/1084
             if resp.status_code not in (codes.temporary_redirect, codes.permanent_redirect):
-                if 'Content-Length' in prepared_request.headers:
-                    del prepared_request.headers['Content-Length']
-
+                # https://github.com/[USER_REDACTED]nethreitz/requests/issues/3490
+                purged_headers = ('Content-Length', 'Content-Type', 'Transfer-Encoding')
+                for header in purged_headers:
+                    prepared_request.headers.pop(header, None)
                 prepared_request.body = None

             headers = prepared_request.headers

@@ -221,6 +221,46 @@ def test_http_303_doesnt_change_head_to_get(self, httpbin):
         assert r.history[0].status_code == 303
         assert r.history[0].is_redirect

+    def test_header_and_body_removal_on_redirect(self, httpbin):
+        purged_headers = ('Content-Length', 'Content-Type')
+        ses = requests.Session()
+        req = requests.Request('POST', httpbin('post'), data={'test': 'data'})
+        prep = ses.prepare_request(req)
+        resp = ses.send(prep)
+
+        # Mimic a redirect response
+        resp.status_code = 302
+        resp.headers['location'] = 'get'
+
+        # Run request through resolve_redirects
+        next_resp = next(ses.resolve_redirects(resp, prep))
+        assert next_resp.request.body is None
+        for header in purged_headers:
+            assert header not in next_resp.request.headers
+
+    def test_transfer_enc_removal_on_redirect(self, httpbin):
+        purged_headers = ('Transfer-Encoding', 'Content-Type')
+        ses = requests.Session()
+        req = requests.Request('POST', httpbin('post'), data=(b'x' for x in range(1)))
+        prep = ses.prepare_request(req)
+        assert 'Transfer-Encoding' in prep.headers
+
+        # [NAME_REDACTED] to avoid https://github.com/[USER_REDACTED]n1024/pytest-httpbin/issues/33
+        resp = requests.Response()
+        resp.raw = io.BytesIO(b'the content')
+        resp.request = prep
+        setattr(resp.raw, 'release_conn', lambda *args: args)
+
+        # Mimic a redirect response
+        resp.status_code = 302
+        resp.headers['location'] = httpbin('get')
+
+        # Run request through resolve_redirect
+        next_resp = next(ses.resolve_redirects(resp, prep))
+        assert next_resp.request.body is None
+        for header in purged_headers:
+            assert header not in next_resp.request.headers
+
     def test_HTTP_200_OK_GET_WITH_PARAMS(self, httpbin):
         heads = {'User-agent': 'Mozilla/5.0'}

I know I wrote this poorly the first time, but why don't we do:

``` py
prepared_request.headers.pop('Content-Length', None)
prepared_request.headers.pop('Content-Type', None)
```

Or if you really want to shave the yak

``` py
for part in ('Length', 'Type'):
    prepared_request.headers.pop('Content-' + part, None)
```

Just to be clear, I think the second suggestion is _very_ bad, but I realize some people might actually prefer that.

I originally used `pop` over `del` when writing it, but reverted to keep with the surrounding code. If you're onboard with using `pop` instead, let's do it.

This test doesn't actually cover 'Transfer-Encoding' since 'Content-Length' is present. I can add a smaller separate test for 'Transfer-Encoding' if it's deemed that we definitely want to remove it.

This does ensure that as long as `purged_headers` contains 'Transfer-Encoding' in `resolve_redirects`, that it will be removed if it occurs. It just won't fail for accidental removal from the `purged_headers` tuple.

Transfer-Encoding should be easily covered: a generator body will trigger it.

Transfer-Encoding test code added.

Can we split this into two tests that look very similar, rather than have one really complex test?

Sure thing. Tests split up now.

I reduced the `range` call to size 1 because we were hitting [this bug](https://github.com/[USER_REDACTED]n1024/pytest-httpbin/issues/33) when sending more than one chunk. After some more testing, this only passes _most of the time_, which likely isn't something we want to introduce.

The failures seem random and are tied specifically to the httpbin module. Calling the live http://httpbin.org/post endpoint won't produce the failure.

As I see it, two options are either calling the httpbin.org/post endpoint directly (probably not the best solution) or adding a try/except around `resp = ses.send(prep)` and manually creating the Response on failure.

Thoughts?

There's a question about whether we should just manually create the response unconditionally, to be honest. That might be the most sensible thing to do.

Alright, I'll make a note and create the `Response` manually for this.

Ok, test updated with manual `Response`. If it looks good I'll squash things down.


adding in pickling tests for PreparedRequest

This is a continuation of [USER_REDACTED]'s work in #2757 and addresses issue #1558 regarding pickling Request objects (specifically `PreparedRequest` objects in this PR).

These tests ensure that a standard `PreparedRequest`, a `PreparedRequest` with a file object as the body, and a `PreparedRequest` with hooks defined outside of the `locals` scope will all pickle properly.

This also works for data passed to the `json` parameter and I can add my test for that as well if it's deemed helpful.

Hooks defined inside of a method (such as in the [original test](https://github.com/[USER_REDACTED]nethreitz/requests/pull/2757/files#diff-56c2d754173a4a158ce8f445834c8fe8R843) by [USER_REDACTED]) will fail because you can't pickle local objects. Is this a use case that needs to be handled?

@@ -1053,6 +1053,52 @@ def test_request_and_response_are_pickleable(self, httpbin):
         assert r.request.url == pr.request.url
         assert r.request.headers == pr.request.headers

+    def test_prepared_request_is_pickleable(self, httpbin):
+        p = requests.Request('GET', httpbin('get')).prepare()
+
+        # [NAME_REDACTED] can be pickled and unpickled
+        r = pickle.loads(pickle.dumps(p))
+        assert r.url == p.url
+        assert r.headers == p.headers
+        assert r.body == p.body
+
+        # Verify unpickled PreparedRequest sends properly
+        s = requests.Session()
+        resp = s.send(r)
+        assert resp.status_code == 200
+
+    def test_prepared_request_with_file_is_pickleable(self, httpbin):
+        files = {'file': open(__file__, 'rb')}
+        r = requests.Request('POST', httpbin('post'), files=files)
+        p = r.prepare()
+
+        # [NAME_REDACTED] can be pickled and unpickled
+        r = pickle.loads(pickle.dumps(p))
+        assert r.url == p.url
+        assert r.headers == p.headers
+        assert r.body == p.body
+
+        # Verify unpickled PreparedRequest sends properly
+        s = requests.Session()
+        resp = s.send(r)
+        assert resp.status_code == 200
+
+    def test_prepared_request_with_hook_is_pickleable(self, httpbin):
+        r = requests.Request('GET', httpbin('get'), hooks=default_hooks())
+        p = r.prepare()
+
+        # [NAME_REDACTED] can be pickled
+        r = pickle.loads(pickle.dumps(p))
+        assert r.url == p.url
+        assert r.headers == p.headers
+        assert r.body == p.body
+        assert r.hooks == p.hooks
+
+        # Verify unpickled PreparedRequest sends properly
+        s = requests.Session()
+        resp = s.send(r)
+        assert resp.status_code == 200
+
     def test_cannot_send_unprepared_requests(self, httpbin):
         r = requests.Request(url=httpbin())
         with pytest.raises(ValueError):

Alright, cool, let's merge these. More tests are good!

This doesn't actually test that a file object in the body is pickleable: `r.prepare()` will read the whole file object into memory as a string. I don't know that we actually do support having a file object as the body when pickling/unpickling, though it'd be interesting to see.

Yeah, we 100% do not support that because it's not a capability of any version of pickle as far as I can tell. The only reason `files` works while `data` doesn't is because we consume the file object during `prepare_body`. It may be worth changing the name or adding a docstring here for clarification on why this works.


raise InvalidHeader on multiple Location values

Addresses issue raised in #2939 with a fix for multiple Location headers in response. This currently breaks on `test_requests_are_updated_each_time` because the [`RedirectSession`](https://github.com/[USER_REDACTED]nethreitz/requests/blob/proposed/3.0.0/tests/test_requests.py#L1564) populates it's `raw` attribute with `StringIO` rather than a urllib3 `HTTPResponse`. I can convert the `_build_raw` method to fix the test, if this looks like a workable solution.

@@ -83,6 +83,10 @@ class InvalidURL(RequestException, ValueError):
     """ The URL provided was somehow invalid. """

+class InvalidHeader(RequestException, ValueError):
+    """The header value provided was somehow invalid."""
+
+
 class ChunkedEncodingError(RequestException):
     """The server declared chunked encoding but sent an invalid chunk."""

@@ -20,15 +20,16 @@
 from .hooks import default_hooks, dispatch_hook
 from .utils import to_key_val_list, default_headers, to_native_string
 from .exceptions import (
-    TooManyRedirects, InvalidScheme, ChunkedEncodingError, ContentDecodingError)
+    TooManyRedirects, InvalidScheme, ChunkedEncodingError,
+    ContentDecodingError, InvalidHeader)
 from .packages.urllib3._collections import RecentlyUsedContainer
 from .structures import CaseInsensitiveDict

 from .adapters import HTTPAdapter

 from .utils import (
     requote_uri, get_environ_proxies, get_netrc_auth, should_bypass_proxies,
-    get_auth_from_url
+    get_auth_from_url, is_valid_location
 )

 from .status_codes import codes
@@ -98,6 +99,10 @@ def resolve_redirects(self, response, stream=False, timeout=None,
         request = response.request

         while response.is_redirect:
+            if not is_valid_location(response):
+                raise InvalidHeader('Response contains multiple Location headers. '
+                                    'Unable to perform redirect.')
+
             prepared_request = request.copy()

             if redirect_count > 0:

@@ -665,6 +665,17 @@ def parse_header_links(value):

     return links

+def is_valid_location(response):
+    """Verify that multiple Location headers weren't
+    returned from the last response.
+    """
+    headers = getattr(response.raw, 'headers', None)
+    if headers is not None:
+        getlist = getattr(headers, 'getlist', None)
+        if getlist is not None:
+            return len(getlist('location')) <= 1
+    # If response.raw isn't urllib3-like we can't reliably check this
+    return True

 # Null bytes; no need to recreate these on each call to guess_json_utf
 _null = '\x00'.encode('ascii')  # encoding to ASCII for Python 3

@@ -22,7 +22,7 @@
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidScheme, InvalidURL,
     MissingScheme, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError)
+    ProxyError, InvalidHeader)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -222,6 +222,20 @@ def test_http_303_doesnt_change_head_to_get(self, httpbin):
         assert r.history[0].status_code == 303
         assert r.history[0].is_redirect

+    def test_multiple_location_headers(self, httpbin):
+        headers = [('Location', 'http://example.com'),
+                   ('Location', 'https://example.com/1')]
+        params = '&'.join(['%s=%s' % (k, v) for k, v in headers])
+        ses = requests.Session()
+        req = requests.Request('GET', httpbin('response-headers?%s' % params))
+        prep = ses.prepare_request(req)
+        resp = ses.send(prep)
+        # change response to redirect
+        resp.status_code = 302
+        with pytest.raises(InvalidHeader):
+            # next triggers yield on generator
+            next(ses.resolve_redirects(resp, prep))
+
     # def test_HTTP_302_ALLOW_REDIRECT_POST(self):
     #     r = requests.post(httpbin('status', '302'), data={'some': 'data'})
     #     self.assertEqual(r.status_code, 200)
@@ -1612,7 +1626,7 @@ def test_requests_are_updated_each_time(httpbin):
     r0 = session.send(prep)
     assert r0.request.method == 'POST'
     assert session.calls[-1] == SendCall((r0.request,), {})
-    redirect_generator = session.resolve_redirects(r0, prep)
+    redirect_generator = session.resolve_redirects(r0)
     default_keyword_args = {
         'stream': False,
         'verify': True,

This will need to include a `getattr` check. Frankly, it's a bit implementation detaily, so we may want to hide this in a utility function.

Going through a few libraries that implement different adapters, the `raw` field seems considerably varied in it's use.

In the event we end up with something file-like, such as `StringIO`, we can parse the headers out, but only if it's the complete response. In the limited testing I did though, this was typically just the response body. I'm not sure if we can do anything useful in that case. Also if it is the response body and happens to contain contents mimicking an HTTP response, then we could in theory flag this incorrectly.

I'm just seeing a lot of unknowns here that I'm not sure have a clean fix. Perhaps just returning True when we can't make a reasonable discernment would be best.

Here we'll likely have a header value similar to `'http://foo.com, http://foo.com'` which could be parsed but probably not reliably. I recognize that URLs can contain unencoded `,` but header values are also defined to be comma separated, so I'm not sure how the case of a multi-value header with values containing commas is handled. A cursory google search wasn't terribly informative.

Browsers seem to catch this problem at what would be our urllib3 layer before the field merge is ever done. Patching it there is null though, because here we're concerned about how to handle the issue for non-urllib3 responses. Perhaps this is another case we can't handle, but then that puts us back at ground-zero for the pull request (other than preventing the exception for StringIO). Perhaps that what [USER_REDACTED] was asking for though and I severely overthought the problem.

I think this is more than we need. If we can't get headers in the above case, don't worry about it.

Mostly what I'm concerned about here is not exploding in the non-urllib3 case: we don't need to be too clever, or even really "handle" it in any meaningful way.

Ok, great, that helps narrow the scope considerably. I'll shorten this down to just catching the urllib3 case then. Thanks!

So we shouldn't use hasattr, we should use getattr. This is because [of this](https://hynek.me/articles/hasattr/).

To be clear, I suggest the `getattr` pattern in this case.

Per usual, I prefer the Python 2 behavior here hahaha

Yeah, I'm onboard with this. I updated the PR with a somewhat verbose implementation of `getattr`.


Support responses like `HTTP/1.1 404 öööööööö`

Currently this code raises UnicodeDecodeError:

`requests.get('http://www.voice.fi//lampaan-kiveksia-ja-coca-cola-hulluutta-5-asiaa-joita-et-tiennyt-islannista-122015?utm_source=feed&utm_medium=feed&utm_content=article_link&utm_campaign=feed').ok`

Here is their status:
`HTTP/1.1 404 Komponenttia ei löydy`

This pull request allow module to handle URLs like this correctly

@@ -849,12 +849,16 @@ def raise_for_status(self):
         """Raises stored :class:`HTTPError`, if one occurred."""

         http_error_msg = ''
+        if isinstance(self.reason, bytes):
+            reason = self.reason.decode('utf-8', 'ignore')
+        else:
+            reason = self.reason

         if 400 <= self.status_code < 500:
-            http_error_msg = '%s [NAME_REDACTED]: %s for url: %s' % (self.status_code, self.reason, self.url)
+            http_error_msg = u'%s [NAME_REDACTED]: %s for url: %s' % (self.status_code, reason, self.url)

         elif 500 <= self.status_code < 600:
-            http_error_msg = '%s [NAME_REDACTED]: %s for url: %s' % (self.status_code, self.reason, self.url)
+            http_error_msg = u'%s [NAME_REDACTED]: %s for url: %s' % (self.status_code, reason, self.url)

         if http_error_msg:
             raise HTTPError(http_error_msg, response=self)

@@ -987,6 +987,15 @@ def test_response_decode_unicode(self):
         chunks = r.iter_content(decode_unicode=True)
         assert all(isinstance(chunk, str) for chunk in chunks)

+    def test_response_reason_unicode(self):
+        # check for unicode HTTP status
+        r = requests.Response()
+        r.url = u'unicode URL'
+        r.reason = u'Komponenttia ei löydy'.encode('utf-8')
+        r.status_code = 404
+        r.encoding = None
+        assert not r.ok # old behaviour - crashes here
+
     def test_response_chunk_size_type(self):
         """Ensure that chunk_size is passed as None or an integer, otherwise
         raise a TypeError.

Can you move this to a new test, please?

Can you change this line to decode with the error handling mode `replace`, please?

I'd still like this to be its own test, please. =)

[USER_REDACTED] done and updated PR

We did officially drop 3.2 support, right? I know we stopped running the tests, but we were supporting it regardless before this.

We dropped it like it was hot.

Perhaps I'm missing something, why not `u'Komponenttia ei löydy'.encode('utf-8')`?

[USER_REDACTED] python < 3.3 does not support u'' prefix

[USER_REDACTED] I can add six.u for it.

As we discussed above, we don't support 3.0, 3.1, or 3.2. 2.7 absolutely supports `u''`. And you're using `u''` in your pull request inside requests.


check for headers containing return characters

Addresses issue raised in #2947.

This likely needs some tweaking. I added the check in `adapters.py` because it was the lowest level I could find header modification being done before sending. I'm not sure it actually needs to be this low though since it doesn't seem easily reachable with a non-`PreparedRequest` object.

Alternatively, we could move the check to the [`prepare_headers`](https://github.com/[USER_REDACTED]nethreitz/requests/blob/master/requests/models.py#L406) method in `PreparedRequest`.

I also ignored `Accept` headers in the check because I the requirements in [RFC7230](https://tools.ietf.org/html/rfc7230#page-24) seems a bit vague.

> This specification deprecates such line folding except within the message/http media type
>    (Section 8.3.1).  A sender MUST NOT generate a message that includes line folding (i.e., that has any    field-value that contains a match to the obs-fold rule) unless the message is intended for packaging
>    within the message/http media type.

This may be a poor interpretation on my part though.

@@ -166,3 +166,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - piotrjurkiewicz
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])

@@ -80,7 +80,11 @@ class InvalidSchema(RequestException, ValueError):

 class InvalidURL(RequestException, ValueError):
-    """ The URL provided was somehow invalid. """
+    """The URL provided was somehow invalid."""
+
+
+class InvalidHeader(RequestException, ValueError):
+    """The header value provided was somehow invalid."""

 class ChunkedEncodingError(RequestException):

@@ -27,7 +27,8 @@
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
-    iter_slices, guess_json_utf, super_len, to_native_string)
+    iter_slices, guess_json_utf, super_len, to_native_string,
+    check_header_validity)
 from .compat import (
     cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
     is_py2, chardet, builtin_str, basestring)
@@ -403,10 +404,13 @@ def prepare_url(self, url, params):
     def prepare_headers(self, headers):
         """Prepares the given HTTP headers."""

+        self.headers = CaseInsensitiveDict()
         if headers:
-            self.headers = CaseInsensitiveDict((to_native_string(name), value) for name, value in headers.items())
-        else:
-            self.headers = CaseInsensitiveDict()
+            for header in headers.items():
+                # Raise exception on invalid header value.
+                check_header_validity(header)
+                name, value = header
+                self.headers[to_native_string(name)] = value

     def prepare_body(self, data, files, json=None):
         """Prepares the given HTTP body data."""

@@ -27,7 +27,7 @@
                      basestring)
 from .cookies import RequestsCookieJar, cookiejar_from_dict
 from .structures import CaseInsensitiveDict
-from .exceptions import InvalidURL, FileModeWarning
+from .exceptions import InvalidURL, InvalidHeader, FileModeWarning

 _hush_pyflakes = (RequestsCookieJar,)

@@ -732,6 +732,24 @@ def to_native_string(string, encoding='ascii'):

     return out

+# Moved outside of function to avoid recompile every call
+_CLEAN_HEADER_REGEX_BYTE = re.compile(b'^\\S[^\\r\\n]*$|^$')
+_CLEAN_HEADER_REGEX_STR = re.compile(r'^\S[^\r\n]*$|^$')
+
+def check_header_validity(header):
+    """Verifies that header value doesn't contain leading whitespace or
+    return characters. This prevents unintended header injection.
+
+    :param header: tuple, in the format (name, value).
+    """
+    name, value = header
+
+    if isinstance(value, bytes):
+        pat = _CLEAN_HEADER_REGEX_BYTE
+    else:
+        pat = _CLEAN_HEADER_REGEX_STR
+    if not pat.match(value):
+        raise InvalidHeader("Invalid return character or leading space in header: %s" % name)

 def urldefragauth(url):
     """

@@ -23,7 +23,7 @@
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError)
+    ProxyError, InvalidHeader)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -1128,6 +1128,47 @@ def test_header_keys_are_native(self, httpbin):
         assert 'unicode' in p.headers.keys()
         assert 'byte' in p.headers.keys()

+    def test_header_validation(self,httpbin):
+        """Ensure prepare_headers regex isn't flagging valid header contents."""
+        headers_ok = {'foo': 'bar baz qux',
+                      'bar': '1',
+                      'baz': '',
+                      'qux': str.encode(u'fbbq')}
+        r = requests.get(httpbin('get'), headers=headers_ok)
+        assert r.request.headers['foo'] == headers_ok['foo']
+
+    def test_header_no_return_chars(self, httpbin):
+        """Ensure that a header containing return character sequences raise an
+        exception. Otherwise, multiple headers are created from single string.
+        """
+        headers_ret = {'foo': 'bar\r\nbaz: qux'}
+        headers_lf = {'foo': 'bar\nbaz: qux'}
+        headers_cr = {'foo': 'bar\rbaz: qux'}
+
+        # Test for newline
+        with pytest.raises(InvalidHeader):
+            r = requests.get(httpbin('get'), headers=headers_ret)
+        # Test for line feed
+        with pytest.raises(InvalidHeader):
+            r = requests.get(httpbin('get'), headers=headers_lf)
+        # Test for carriage return
+        with pytest.raises(InvalidHeader):
+            r = requests.get(httpbin('get'), headers=headers_cr)
+
+    def test_header_no_leading_space(self, httpbin):
+        """Ensure headers containing leading whitespace raise
+        [NAME_REDACTED] before sending.
+        """
+        headers_space = {'foo': ' bar'}
+        headers_tab = {'foo': '   bar'}
+
+        # Test for whitespace
+        with pytest.raises(InvalidHeader):
+            r = requests.get(httpbin('get'), headers=headers_space)
+        # Test for tab
+        with pytest.raises(InvalidHeader):
+            r = requests.get(httpbin('get'), headers=headers_tab)
+
     [USER_REDACTED].mark.parametrize('files', ('foo', b'foo', bytearray(b'foo')))
     def test_can_send_objects_with_files(self, httpbin, files):
         data = {'a': 'this is a string'}

comment here would be good

Good call on that, I hadn't thought out the regex completely. It was missing strings of length 0. Comment and test updated.

We need to bring `to_native_string` up to before the match on this regex. This is because we allow users to provide headers as unicode and byte strings, which will misbehave with a native string regex.

To be clear, `to_native_string` should probably go above this line.

Can we say 'leading space', not just 'space'?

Moved the `to_native_string` call up and added it for `val` too, since that's what we're actually matching the regex on.

I'm a little bit worried about this, actually. `val` could be an opaque bytestring that doesn't decode well on Python 3. Let's change this to use a bytestring regex (prefixed with `rb`) and then elide the call to `to_native_string` on `val`.

One space after the `#` character, please. =)

These need to be separated from each other by [NAME_REDACTED], not one.

[USER_REDACTED] Just to clarify, do we want to leave `name = to_native_string(name)` on line 413 to prevent a byte string from being passed to the exception, or did you want it moved back to where it was originally on line 416?

[USER_REDACTED] Also the rb'string' syntax doesn't appear to work for python 2.6 or 2.7.

I don't really mind if it gets moved. =)

As to the rb string syntax, in that case I recommend not using a raw string at all and just escaping the backslashes and using bytestrings.. It's annoying, but works on all platforms.

Hmm, so that worked fine on my python2 testing but it looks like it fails for python3 with the following error:

`TypeError: can't use a bytes pattern on a string-like object`

I'm digging around for a more universal option.

We don't need this import any more.

Frankly, I don't love that this hides the call to `to_native_string`. I understand why it's here, but I'd rather that it didn't. Can we just turn this into a function that we can pass to `filter`?

So using `re.compile` is great here, but we lose the advantage because these are done in the function. Can you hoist both these REs up to module scope and give them neat names, and then have this function choose between them?

That's what I was thinking too (with performance), I just wasn't sure where to declare them. You're suggesting throwing something like this at the top of `utils.py`, yeah?

```
CLEAN_HEADER_REGEX_BYTES = re.compile(b'^\S[^\\r\\n]*$|^$')
CLEAN_HEADER_REGEX_STR = re.compile(r'^\S[^\r\n]*$|^$')
```

Or even just hoist them directly above this function. Let's prefix their names with underscores just to discourage people relying on them.

I'm not particularly fond of having to iterate over the header list twice, but I couldn't devise a way to wrap name in to_native_string() while using `is_valid_header` as a filter function. Perhaps I'm missing an easy solution though.

I mean, `filter` is kind of a lie here (it's really `map`, and even there we aren't transforming the data), so you could have just one loop that calls `is_valid_header` then sets the kvp in the CaseInsensitiveDict.

This seems slightly kludgy to me, since we're not doing a transform or using this as a conditional. Maybe it's just a naming issue though. Renaming it `check_header_validity` and removing the `return True` may be a better option to describe it's current functionality. Otherwise, I'll leave it in, as is, to use as a generalized utility check elsewhere if needed.

Yeah, let's do that.

Alright, updated. If it's looking ok, I'll squash everything back down.

I think _strictly_ we're missing a backslash in this regex. You have `\S`, but I think you want `\\S`.

Right on, completely missed that. Thanks! Pushed up the additional \ and will squash the commits down if this is looking ok.


added in type check for chunk_size

This should raise a type error if a non-integer chunk_size is passed. I'm assuming we're not concerned about a chunk_size of type long, but that may be wrong. Resolves issue #3364

@@ -685,6 +685,8 @@ def generate():

         if self._content_consumed and isinstance(self._content, bool):
             raise StreamConsumedError()
+        elif not isinstance(chunk_size, int):
+            raise TypeError("chunk_size must be an int, it is instead a %s." % type(chunk_size))
         # simulate reading small chunks of the content
         reused_chunks = iter_slices(self._content, chunk_size)

@@ -980,6 +980,20 @@ def test_response_decode_unicode(self):
         chunks = r.iter_content(decode_unicode=True)
         assert all(isinstance(chunk, str) for chunk in chunks)

+    def test_response_chunk_size_int(self):
+        """Ensure that chunk_size is passed as an integer, otherwise
+        raise a TypeError.
+        """
+        r = requests.Response()
+        r.raw = io.BytesIO(b'the content')
+        chunks = r.iter_content(1)
+        assert all(len(chunk) == 1 for chunk in chunks)
+
+        r = requests.Response()
+        r.raw = io.BytesIO(b'the content')
+        with pytest.raises(TypeError):
+            chunks = r.iter_content("1024")
+
     def test_request_and_response_are_pickleable(self, httpbin):
         r = requests.get(httpbin('get'))

More simply:

> "chunk_size must be an int, it is instead a %s" % type(chunk_size)

Ok, things should be reworded. I didn't line wrap, as per the contributing docs, but let me know if it needs any other formatting tweaks.


adding asserted_encoding check on None type encoding to match text() …

…behavior. Resolves #3359

@@ -358,13 +358,20 @@ def get_encoding_from_headers(headers):

 def stream_decode_response_unicode(iterator, r):
     """Stream decodes a iterator."""
+    encoding = r.encoding

-    if r.encoding is None:
-        for item in iterator:
-            yield item
-        return
+    if encoding is None:
+        encoding = r.apparent_encoding
+
+    try:
+        decoder = codecs.getincrementaldecoder(encoding)(errors='replace')
+    except (LookupError, TypeError):
+        # A LookupError is raised if the encoding was not found which could
+        # indicate a misspelling or similar mistake.
+        #
+        # A TypeError can be raised if encoding is None
+        raise UnicodeError("Unable to decode contents with encoding %s." % encoding)

-    decoder = codecs.getincrementaldecoder(r.encoding)(errors='replace')
     for chunk in iterator:
         rv = decoder.decode(chunk)
         if rv:

@@ -980,6 +980,13 @@ def test_response_decode_unicode(self):
         chunks = r.iter_content(decode_unicode=True)
         assert all(isinstance(chunk, str) for chunk in chunks)

+        # check for encoding value of None
+        r = requests.Response()
+        r.raw = io.BytesIO(b'the content')
+        r.encoding = None
+        chunks = r.iter_content(decode_unicode=True)
+        assert all(isinstance(chunk, str) for chunk in chunks)
+
     def test_response_chunk_size_int(self):
         """Ensure that chunk_size is passed as an integer, otherwise
         raise a TypeError.

Why not just use `encoding` instead of `r.encoding`?

Can we add the same `try...except` as in `text, except catching both`LookupError`*and*`TypeError` and, if either is raised, raising a more appropriate error? This code cannot function with a fallback, so we need to raise a useful exception here.

Are you referring to the reuse of "r.encoding" on line 362? I was trying to follow the pattern established with the [`text()`](https://github.com/[USER_REDACTED]nethreitz/requests/blob/49b69c4751689df7aca4435ef1223a58871a64e2/requests/models.py#L770) function

I added in a `UnicodeError`, not sure how descriptive you'd like the text. `stream_decode_response_unicode()` doesn't seem to currently have any coverage, but I can add tests for the failures if you think that would be useful.

Attribute accesses are slower than local variable look-ups though, so we might as well use `encoding` since it already exists.

Alright, should be swapped to use the local `encoding` variable now.


Fixed #36921 -- Fixed KeyError when adding inline instances of models not registered with admin.

#### Trac ticket number
ticket-36921

#### Branch description
Added an if statement to avoid key error if model not registered in admin.

Note this bug was introduced in another pull request I worked on [here](https://github.com/[USER_REDACTED]ngo/django/pull/18934/).

I added the example Jacob shared to the same example project that had been used with that issue [here](https://github.com/reergymerej/ticket_13883).

#### AI [NAME_REDACTED] (REQUIRED)
- [ ] **No AI tools were used** in preparing this PR.
- [x] **If AI tools were used** - [NAME_REDACTED] plugin in VS Code

#### Checklist
- [x] This PR follows the [contribution guidelines](https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/submitting-patches/).
- [x] This PR **does not** disclose a security vulnerability (see [vulnerability reporting](https://docs.djangoproject.com/en/stable/internals/security/)).
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [x] I have attached screenshots in both light and dark modes for any UI changes.

@@ -1419,7 +1419,7 @@ def response_add(self, request, obj, post_url_continue=None):

             # Find the optgroup for the new item, if available
             source_model_name = request.POST.get(SOURCE_MODEL_VAR)
-
+            source_admin = None
             if source_model_name:
                 app_label, model_name = source_model_name.split(".", 1)
                 try:
@@ -1428,21 +1428,23 @@ def response_add(self, request, obj, post_url_continue=None):
                     msg = _('The app "%s" could not be found.') % source_model_name
                     self.message_user(request, msg, messages.ERROR)
                 else:
-                    source_admin = self.admin_site._registry[source_model]
-                    form = source_admin.get_form(request)()
-                    if self.opts.verbose_name_plural in form.fields:
-                        field = form.fields[self.opts.verbose_name_plural]
-                        for option_value, option_label in field.choices:
-                            # Check if this is an optgroup (label is a sequence
-                            # of choices rather than a single string value).
-                            if isinstance(option_label, (list, tuple)):
-                                # It's an optgroup:
-                                # (group_name, [(value, label), ...])
-                                optgroup_label = option_value
-                                for choice_value, choice_display in option_label:
-                                    if choice_display == str(obj):
-                                        popup_response["optgroup"] = str(optgroup_label)
-                                        break
+                    source_admin = self.admin_site._registry.get(source_model)
+
+            if source_admin:
+                form = source_admin.get_form(request)()
+                if self.opts.verbose_name_plural in form.fields:
+                    field = form.fields[self.opts.verbose_name_plural]
+                    for option_value, option_label in field.choices:
+                        # Check if this is an optgroup (label is a sequence
+                        # of choices rather than a single string value).
+                        if isinstance(option_label, (list, tuple)):
+                            # It's an optgroup:
+                            # (group_name, [(value, label), ...])
+                            optgroup_label = option_value
+                            for choice_value, choice_display in option_label:
+                                if choice_display == str(obj):
+                                    popup_response["optgroup"] = str(optgroup_label)
+                                    break

             popup_response_data = json.dumps(popup_response)
             return TemplateResponse(

@@ -589,6 +589,31 @@ def test_popup_add_POST_with_invalid_source_model(self):
         self.assertIn("admin_views.nonexistent", str(messages[0]))
         self.assertIn("could not be found", str(messages[0]))

+    def test_popup_add_POST_with_unregistered_source_model(self):
+        """
+        Popup add where source_model is a valid Django model but is not
+        registered in the admin site (e.g. a model only used as an inline)
+        should succeed without raising a KeyError.
+        """
+        post_data = {
+            IS_POPUP_VAR: "1",
+            # Chapter exists as a model but is not registered in site (only
+            # in site6), simulating a model used only as an inline.
+            SOURCE_MODEL_VAR: "admin_views.chapter",
+            "title": "Test Article",
+            "content": "some content",
+            "date_0": "2010-09-10",
+            "date_1": "14:55:39",
+        }
+        response = self.client.post(reverse("admin:admin_views_article_add"), post_data)
+        self.assertEqual(response.status_code, 200)
+        self.assertContains(response, "data-popup-response")
+        # No error messages - unregistered model is silently skipped.
+        messages = list(response.wsgi_request._messages)
+        self.assertEqual(len(messages), 0)
+        # No optgroup in the response.
+        self.assertNotContains(response, ""optgroup"")
+
     def test_basic_edit_POST(self):
         """
         A smoke test to ensure POST on edit_view works.

Wonderful, thanks for jumping on this!

This might be time to fight indentation inflation 😄

What do you think about something like:
```py
            source_admin = None
            if source_model_name:
                app_label, model_name = source_model_name.split(".", 1)
                try:
                    source_model = apps.get_model(app_label, model_name)
                except LookupError:
                    msg = _('The app "%s" could not be found.') % source_model_name
                    self.message_user(request, msg, messages.ERROR)
                else:
                    source_admin = self.admin_site._registry.get(source_model)

            if source_admin:
                form = source_admin.get_form(request)()
                ...
```

Oh my gosh yes I'm embarrassed you had to suggest I had the same thought and then forgot to come back to it. Thank you. That is done.


Refs #35809 -- Fixed test_selectbox_selected_rows() on macOS.

ticket-35809

This was failing for me on Mac because, as the on-screen help text advises, on mac you use Command, not Control.

To check:
```
./runtests.py --screenshots --selenium=chrome -k selected_rows --parallel=1
```

@@ -1,6 +1,7 @@
 import datetime
 import os
 import re
+import sys
 import unittest
 import zoneinfo
 from unittest import mock
@@ -5983,6 +5984,12 @@ def setUp(self):
             title="A [NAME_REDACTED]", published=True, slug="a-long-title"
         )

+    [USER_REDACTED]
+    def modifier_key(self):
+        from selenium.webdriver.common.keys import Keys
+
+        return Keys.COMMAND if sys.platform == "darwin" else Keys.CONTROL
+
     [USER_REDACTED]_cases(["desktop_size", "mobile_size", "rtl", "dark", "high_contrast"])
     def test_login_button_centered(self):
         from selenium.webdriver.common.by [NAME_REDACTED]
@@ -6406,8 +6413,8 @@ def test_selectbox_selected_rows(self):
             elem = self.selenium.find_element(
                 By.CSS_SELECTOR, f"#id_user_permissions_from option[value='{perm.id}']"
             )
-            ActionChains(self.selenium).key_down(Keys.CONTROL).click(elem).key_up(
-                Keys.CONTROL
+            ActionChains(self.selenium).key_down(self.modifier_key).click(elem).key_up(
+                self.modifier_key
             ).perform()

         # Move focus to other element.
@@ -6425,8 +6432,8 @@ def test_selectbox_selected_rows(self):
             elem = self.selenium.find_element(
                 By.CSS_SELECTOR, f"#id_user_permissions_to option[value='{perm.id}']"
             )
-            ActionChains(self.selenium).key_down(Keys.CONTROL).click(elem).key_up(
-                Keys.CONTROL
+            ActionChains(self.selenium).key_down(self.modifier_key).click(elem).key_up(
+                self.modifier_key
             ).perform()

         # Move focus to other element.

Looks good, added a small request to reduce memory footprint (not that the gain is massive, but the precedent important).

Despite the historical usage of `setUp` for object creation, I usually recommend using helpers to get/create interesting values used by tests. The reason is that everything that is attached to `self` in `setUp`, is "alive" for the whole test suite run (not the test case, but the whooooole test suite run). So I would suggest:

```suggestion

    [USER_REDACTED]
    def modifier_key(self):
        return Keys.COMMAND if sys.platform == "darwin" else Keys.CONTROL
```

I like it!


Refs #24928 -- Added introspection support for PostgreSQL HStoreField.

@@ -65,6 +65,11 @@ def ready(self):
                         3910: "django.contrib.postgres.fields.DateTimeRangeField",
                         3912: "django.contrib.postgres.fields.DateRangeField",
                         3926: "django.contrib.postgres.fields.BigIntegerRangeField",
+                        # PostgreSQL OIDs may vary depending on the
+                        # installation, especially for datatypes from
+                        # extensions, e.g. "hstore". In such cases, the
+                        # type_display attribute (psycopg 3.2+) should be used.
+                        "hstore": "django.contrib.postgres.fields.HStoreField",
                     }
                 )
                 if conn.connection is not None:

@@ -8,6 +8,7 @@
 import threading
 import warnings
 from contextlib import contextmanager
+from functools import lru_cache

 from django.conf import settings
 from django.core.exceptions import ImproperlyConfigured
@@ -29,6 +30,7 @@
     raise ImproperlyConfigured("Error loading psycopg2 or psycopg module")

+[USER_REDACTED]_cache
 def psycopg_version():
     version = Database.__version__.split(" ", 1)[0]
     return get_version_tuple(version)

@@ -3,6 +3,7 @@
 from django.db.backends.base.introspection import BaseDatabaseIntrospection
 from django.db.backends.base.introspection import FieldInfo as BaseFieldInfo
 from django.db.backends.base.introspection import TableInfo as BaseTableInfo
+from django.db.backends.postgresql.base import psycopg_version
 from django.db.models import DB_CASCADE, DB_SET_DEFAULT, DB_SET_NULL, DO_NOTHING, [NAME_REDACTED] = namedtuple("FieldInfo", [*BaseFieldInfo._fields, "is_autofield", "comment"])
@@ -120,10 +121,19 @@ def get_table_description(self, cursor, table_name):
         cursor.execute(
             "SELECT * FROM %s LIMIT 1" % self.connection.ops.quote_name(table_name)
         )
+
+        # PostgreSQL OIDs may vary depending on the installation, especially
+        # for datatypes from extensions, e.g. "hstore". In such cases, the
+        # type_display attribute (psycopg 3.2+) should be used.
+        type_display_available = psycopg_version() >= (3, 2)
         return [
             FieldInfo(
                 line.name,
-                line.type_code,
+                (
+                    line.type_display
+                    if type_display_available and line.type_display == "hstore"
+                    else line.type_code
+                ),
                 # display_size is always None on psycopg2.
                 line.internal_size if line.display_size is None else line.display_size,
                 line.internal_size,

@@ -126,7 +126,9 @@ Minor features
 :mod:`django.contrib.postgres`
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-* ...
+* :djadmin:`inspectdb` now introspects
+  :class:`~django.contrib.postgres.fields.HStoreField` when ``psycopg`` 3.2+ is
+  installed and ``django.contrib.postgres`` is in :setting:`INSTALLED_APPS`.

 :mod:`django.contrib.redirects`
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -517,8 +517,11 @@ def test_lookup_cast_isnull_noop(self):
     def test_correct_extraction_psycopg_version(self):
         from django.db.backends.postgresql.base import Database, psycopg_version

+        psycopg_version.cache_clear()
         with mock.patch.object(Database, "__version__", "4.2.1 (dt dec pq3 ext lo64)"):
+            self.addCleanup(psycopg_version.cache_clear)
             self.assertEqual(psycopg_version(), (4, 2, 1))
+        psycopg_version.cache_clear()
         with mock.patch.object(
             Database, "__version__", "4.2b0.dev1 (dt dec pq3 ext lo64)"
         ):

@@ -33,3 +33,16 @@ def test_range_fields(self):
                 "null=True)",
             ],
         )
+
+    def test_hstore_field(self):
+        from django.db.backends.postgresql.base import psycopg_version
+
+        if psycopg_version() < (3, 2):
+            self.skipTest("psycopg 3.2+ is required.")
+        self.assertFieldsInModel(
+            "postgres_tests_hstoremodel",
+            [
+                "field = django.contrib.postgres.fields.HStoreField(blank=True, "
+                "null=True)",
+            ],
+        )

Thanks!

```suggestion
  is installed and ``django.contrib.postgres`` is in :setting:`INSTALLED_APPS`.
```


Refs #36680 -- Avoided manipulating PATH in AdminScriptTestCase.

#### Trac ticket number
ticket-36680

#### Branch description
This mostly reverts 6436ec321073bf0622af815e0af08f54c97f9b30, which was fragile. Instead, if `black` is present, we use it to format the expected result, instead of hard-coding the expected formatted value.

See #20034 and #20054 for problems with the prior approach, including comments there about even further cases revealed on the no-requirements build where a path component could be `None`.

Thanks [USER_REDACTED] and [USER_REDACTED] for reports.
#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [n/a] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [x] I have attached screenshots in both light and dark modes for any UI changes.

@@ -33,13 +33,11 @@
 from django.core.management.commands.loaddata import Command as LoaddataCommand
 from django.core.management.commands.runserver import Command as RunserverCommand
 from django.core.management.commands.testserver import Command as TestserverCommand
-from django.core.management.utils import find_formatters
 from django.db import ConnectionHandler, connection
 from django.db.migrations.recorder import MigrationRecorder
 from django.test import LiveServerTestCase, SimpleTestCase, TestCase, override_settings
 from django.test.utils import captured_stderr, captured_stdout
 from django.urls import path
-from django.utils.functional import cached_property
 from django.utils.version import PY313, get_docs_version
 from django.views.static import serve

@@ -49,6 +47,8 @@

 SYSTEM_CHECK_MSG = "System check identified no issues"

+HAS_BLACK = shutil.which("black")
+

 class AdminScriptTestCase(SimpleTestCase):
     def setUp(self):
@@ -112,21 +112,7 @@ def _ext_backend_paths(self):
                 paths.append(os.path.dirname(backend_dir))
         return paths

-    [USER_REDACTED]_property
-    def path_without_formatters(self):
-        return os.pathsep.join(
-            [
-                path_component
-                for path_component in os.environ.get("PATH", "").split(os.pathsep)
-                for formatter_path in find_formatters().values()
-                if formatter_path
-                and os.path.commonpath([path_component, formatter_path]) == os.sep
-            ]
-        )
-
-    def run_test(
-        self, args, settings_file=None, apps=None, umask=-1, discover_formatters=False
-    ):
+    def run_test(self, args, settings_file=None, apps=None, umask=-1):
         base_dir = os.path.dirname(self.test_dir)
         # The base dir for Django's tests is one level up.
         tests_dir = os.path.dirname(os.path.dirname(__file__))
@@ -148,8 +134,6 @@ def run_test(
         python_path.extend(ext_backend_base_dirs)
         test_environ["PYTHONPATH"] = os.pathsep.join(python_path)
         test_environ["PYTHONWARNINGS"] = ""
-        if not discover_formatters:
-            test_environ["PATH"] = self.path_without_formatters

         p = subprocess.run(
             [sys.executable, *args],
@@ -161,19 +145,10 @@ def run_test(
         )
         return p.stdout, p.stderr

-    def run_django_admin(
-        self, args, settings_file=None, umask=-1, discover_formatters=False
-    ):
-        return self.run_test(
-            ["-m", "django", *args],
-            settings_file,
-            umask=umask,
-            discover_formatters=discover_formatters,
-        )
+    def run_django_admin(self, args, settings_file=None, umask=-1):
+        return self.run_test(["-m", "django", *args], settings_file, umask=umask)

-    def run_manage(
-        self, args, settings_file=None, manage_py=None, discover_formatters=False
-    ):
+    def run_manage(self, args, settings_file=None, manage_py=None):
         template_manage_py = (
             os.path.join(os.path.dirname(__file__), manage_py)
             if manage_py
@@ -192,11 +167,17 @@ def run_manage(
         with open(test_manage_py, "w") as fp:
             fp.write(manage_py_contents)

-        return self.run_test(
-            ["./manage.py", *args],
-            settings_file,
-            discover_formatters=discover_formatters,
-        )
+        return self.run_test(["./manage.py", *args], settings_file)
+
+    def assertInAfterFormatting(self, member, container, msg=None):
+        if HAS_BLACK:
+            import black
+
+            # Black does not have a stable API, but this is still less fragile
+            # than attempting to filter out all paths where it is available.
+            member = black.format_str(member, mode=black.FileMode())
+
+        self.assertIn(member, container, msg=msg)

     def assertNoOutput(self, stream):
         "Utility assertion: assert that the given stream is empty"
@@ -773,7 +754,7 @@ def test_setup_environ(self):
         with open(os.path.join(app_path, "apps.py")) as f:
             content = f.read()
             self.assertIn("class SettingsTestConfig(AppConfig)", content)
-            self.assertIn("name = 'settings_test'", content)
+            self.assertInAfterFormatting("name = 'settings_test'", content)

     def test_setup_environ_custom_template(self):
         """
@@ -798,7 +779,7 @@ def test_startapp_unicode_name(self):
         with open(os.path.join(app_path, "apps.py"), encoding="utf8") as f:
             content = f.read()
             self.assertIn("class こんにちはConfig(AppConfig)", content)
-            self.assertIn("name = 'こんにちは'", content)
+            self.assertInAfterFormatting("name = 'こんにちは'", content)

     def test_builtin_command(self):
         """
@@ -1960,7 +1941,7 @@ def setUp(self):
     def test_version(self):
         "version is handled as a special case"
         args = ["version"]
-        out, err = self.run_manage(args, discover_formatters=True)
+        out, err = self.run_manage(args)
         self.assertNoOutput(err)
         self.assertOutput(out, get_version())

@@ -2713,7 +2694,7 @@ def test_custom_project_template_non_python_files_not_formatted(self):
         args = ["startproject", "--template", template_path, "customtestproject"]
         testproject_dir = os.path.join(self.test_dir, "customtestproject")

-        _, err = self.run_django_admin(args, discover_formatters=True)
+        _, err = self.run_django_admin(args)
         self.assertNoOutput(err)
         with open(
             os.path.join(template_path, "additional_dir", "requirements.in")
@@ -2808,7 +2789,7 @@ def serve_template(request, *args, **kwargs):
                 f"{self.live_server_url}/user_agent_check/project_template.tgz"
             )
             args = ["startproject", "--template", template_url, "urltestproject"]
-            _, err = self.run_django_admin(args, discover_formatters=True)
+            _, err = self.run_django_admin(args)

             self.assertNoOutput(err)
             self.assertIn("Django/%s" % get_version(), user_agent)
@@ -2879,8 +2860,10 @@ def test_custom_project_template_context_variables(self):
         test_manage_py = os.path.join(testproject_dir, "manage.py")
         with open(test_manage_py) as fp:
             content = fp.read()
-            self.assertIn('project_name = "another_project"', content)
-            self.assertIn('project_directory = "%s"' % testproject_dir, content)
+            self.assertInAfterFormatting('project_name = "another_project"', content)
+            self.assertInAfterFormatting(
+                'project_directory = "%s"' % testproject_dir, content
+            )

     def test_no_escaping_of_project_variables(self):
         "Make sure template context variables are not html escaped"
@@ -2990,7 +2973,7 @@ def test_custom_project_template_hidden_directory_included(self):
         self.assertNoOutput(err)
         render_py_path = os.path.join(testproject_dir, ".hidden", "render.py")
         with open(render_py_path) as fp:
-            self.assertIn(
+            self.assertInAfterFormatting(
                 f"# The {project_name} should be rendered.",
                 fp.read(),
             )
@@ -3150,7 +3133,7 @@ def test_template(self):
         with open(os.path.join(app_path, "apps.py")) as f:
             content = f.read()
             self.assertIn("class NewAppConfig(AppConfig)", content)
-            self.assertIn("name = 'new_app'", content)
+            self.assertInAfterFormatting("name = 'new_app'", content)

     def test_creates_directory_when_custom_app_destination_missing(self):
         args = [

Thank you for this work! Added some feedback, LMKWYT!

This looks great, thank you so much [USER_REDACTED]! 🌟

For reference to others, this is a partial revert of 6436ec321073bf0622af815e0af08f54c97f9b30 and the diff compared to the original code looks like this:
```diff
diff --git a/tests/admin_scripts/tests.py b/tests/admin_scripts/tests.py
index 3718354dc7..0c27194568 100644
--- a/tests/admin_scripts/tests.py
+++ b/tests/admin_scripts/tests.py
@@ -169,6 +169,16 @@ class AdminScriptTestCase(SimpleTestCase):

         return self.run_test(["./manage.py", *args], settings_file)

+    def assertInAfterFormatting(self, member, container, msg=None):
+        if HAS_BLACK:
+            import black
+
+            # Black does not have a stable API, but this is still less fragile
+            # than attempting to filter out all paths where it is available.
+            member = black.format_str(member, mode=black.FileMode())
+
+        self.assertIn(member, container, msg=msg)
+
     def assertNoOutput(self, stream):
         "Utility assertion: assert that the given stream is empty"
         self.assertEqual(
@@ -744,10 +754,7 @@ class DjangoAdminSettingsDirectory(AdminScriptTestCase):
         with open(os.path.join(app_path, "apps.py")) as f:
             content = f.read()
             self.assertIn("class SettingsTestConfig(AppConfig)", content)
-            self.assertIn(
-                'name = "settings_test"' if HAS_BLACK else "name = 'settings_test'",
-                content,
-            )
+            self.assertInAfterFormatting("name = 'settings_test'", content)

     def test_setup_environ_custom_template(self):
         """
@@ -772,9 +779,7 @@ class DjangoAdminSettingsDirectory(AdminScriptTestCase):
         with open(os.path.join(app_path, "apps.py"), encoding="utf8") as f:
             content = f.read()
             self.assertIn("class こんにちはConfig(AppConfig)", content)
-            self.assertIn(
-                'name = "こんにちは"' if HAS_BLACK else "name = 'こんにちは'", content
-            )
+            self.assertInAfterFormatting("name = 'こんにちは'", content)

     def test_builtin_command(self):
         """
@@ -2855,8 +2860,10 @@ class StartProject(LiveServerTestCase, AdminScriptTestCase):
         test_manage_py = os.path.join(testproject_dir, "manage.py")
         with open(test_manage_py) as fp:
             content = fp.read()
-            self.assertIn('project_name = "another_project"', content)
-            self.assertIn('project_directory = "%s"' % testproject_dir, content)
+            self.assertInAfterFormatting('project_name = "another_project"', content)
+            self.assertInAfterFormatting(
+                'project_directory = "%s"' % testproject_dir, content
+            )

     def test_no_escaping_of_project_variables(self):
         "Make sure template context variables are not html escaped"
@@ -2966,7 +2973,7 @@ class StartProject(LiveServerTestCase, AdminScriptTestCase):
         self.assertNoOutput(err)
         render_py_path = os.path.join(testproject_dir, ".hidden", "render.py")
         with open(render_py_path) as fp:
-            self.assertIn(
+            self.assertInAfterFormatting(
                 f"# The {project_name} should be rendered.",
                 fp.read(),
             )
@@ -3126,10 +3133,7 @@ class StartApp(AdminScriptTestCase):
         with open(os.path.join(app_path, "apps.py")) as f:
             content = f.read()
             self.assertIn("class NewAppConfig(AppConfig)", content)
-            self.assertIn(
-                'name = "new_app"' if HAS_BLACK else "name = 'new_app'",
-                content,
-            )
+            self.assertInAfterFormatting("name = 'new_app'", content)

     def test_creates_directory_when_custom_app_destination_missing(self):
         args = [
```

IMHO, all these assertIn need migration to the new helper. See failed Python 3.14 tests:

```
======================================================================
FAIL: test_custom_project_template_context_variables (admin_scripts.tests.StartProject.test_custom_project_template_context_variables)
Make sure template context variables are rendered with proper values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 669, in run
    self._callTestMethod(testMethod)

  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 615, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "/home/runner/work/django/django/tests/admin_scripts/tests.py", line 2865, in test_custom_project_template_context_variables
    self.assertIn('project_directory = "%s"' % testproject_dir, content)
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 1192, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 750, in fail
    raise self.failureException(msg)
    ^^^^^^^^^^^^^^^
AssertionError: 'project_directory = "/tmp/django_guy55sry/django_lmr2wqmu/tmp1ypnoyz2/test_project/project_dir"' not found in '# The manage.py of the another_project test project\n\n# template context:\nproject_name = "another_project"\nproject_directory = (\n    "/tmp/django_guy55sry/django_lmr2wqmu/tmp1ypnoyz2/test_project/project_dir"\n)\nsecret_key = "django-insecure-ld1&pwsm40q-$r$p*cf03(g#&0hwhpunw3dnv*$#_-qs85j0z9"\n'
```

My original suggestion is to only format the `member` since `container` is already formatted by [NAME_REDACTED]

Moreover, I would rename this helper to `assertInOutput` and use it consistently to replace "most" `assertIn` calls, so the current (and future) failing tests, like `test_custom_project_template_context_variables`,  are brittle no more:
```
======================================================================
FAIL: test_custom_project_template_context_variables (admin_scripts.tests.StartProject.test_custom_project_template_context_variables)
Make sure template context variables are rendered with proper values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 669, in run
    self._callTestMethod(testMethod)

  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 615, in _callTestMethod
    result = method()
    ^^^^^^^^^^^^^^^
  File "/home/runner/work/django/django/tests/admin_scripts/tests.py", line 2865, in test_custom_project_template_context_variables
    self.assertIn('project_directory = "%s"' % testproject_dir, content)
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 1192, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))
    ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.14.0/x64/lib/python3.14/unittest/case.py", line 750, in fail
    raise self.failureException(msg)
    ^^^^^^^^^^^^^^^
AssertionError: 'project_directory = "/tmp/django_guy55sry/django_lmr2wqmu/tmp1ypnoyz2/test_project/project_dir"' not found in '# The manage.py of the another_project test project\n\n# template context:\nproject_name = "another_project"\nproject_directory = (\n    "/tmp/django_guy55sry/django_lmr2wqmu/tmp1ypnoyz2/test_project/project_dir"\n)\nsecret_key = "django-insecure-ld1&pwsm40q-$r$p*cf03(g#&0hwhpunw3dnv*$#_-qs85j0z9"\n'
```

Yep, missed a handful here. Most of the assertIn's are actually asserting against streams, not templated files, so black isn't relevant there.

Re:
> Moreover, I would rename this helper to assertInOutput

I hear that, but the tricky part is we have an `assertOutput` already for streams. How's `assertInTemplatedFile`?

We can't use it blindly, since the blind approach gives on an input of `<&>`:
```py
black.parsing.InvalidInput: Cannot parse: 1:0: <&>
```

Given that, I kind of like `assertInAfterFormatting` or similar after all so that you don't end up with:
```py
            self.assertIn("class NewAppConfig(AppConfig)", content)
            self.assertInTemplatedFile("name = 'new_app'", content)
```

[Black is in the requirements.txt](https://github.com/[USER_REDACTED]ngo/django/blob/74564946c3b42a2ef7d087047e49873847a7e1d9/tests/requirements/py3.txt#L5) that the test instructions and CI jobs install. Unless you're explicitly wanting to support running tests in other contexts, I think the tests could assume black is installed and perhaps explicitly test the case where it doesn't by mocking `find_formatters()` to return `None`?

(Just passing by [NAME_REDACTED] in my MR — I think your approach here to avoid changing the PATH is sensible!)

Thank you, this is in my radar, priority for early next week.

Thank you [USER_REDACTED] for taking another look. Django explicitly support development environments (and Django general usage) without having `black` available, so we can't just assume `black` is installed.

For example, in our Jenkins CI we define one job that runs the tests without any external dependency (https://djangoci.com/job/main-no-requirements/).