Remove support for Python 2.7, 3.6 and Pypy3.6

This PR is a proposal to officially remove support for Python 2.7 as discussed in #6023. The next minor release (Requests 2.28.0) will no longer provide support for any of the end of life Python runtimes. Users are encouraged to upgrade to Python 3.8 or later. If you're unable to do so, we'd recommend pinning to `requests<2.28`.

This will be the first of a few PRs that will be doing some general code clean up now that we're moving to a Python 3-only code base. I'll post the others as drafts shortly.

@@ -9,14 +9,11 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["2.7", "3.6", "3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
         os: [ubuntu-18.04, macOS-latest, windows-latest]
         include:
-          # pypy3 on Mac OS currently fails trying to compile
+          # pypy-3.7 on Mac OS currently fails trying to compile
           # brotlipy. Moving pypy3 to only test linux.
-          - python-version: pypy3
-            os: ubuntu-latest
-            experimental: false
           - python-version: pypy-3.7
             os: ubuntu-latest
             experimental: false

@@ -11,6 +11,11 @@ dev
 - Fixed urllib3 exception leak, wrapping `urllib3.exceptions.SSLError` with
   `requests.exceptions.SSLError` for `content` and `iter_content`.

+**Deprecations**
+
+- ⚠ Requests has officially dropped support for Python 2.7. ⚠
+- Requests has officially dropped support for Python 3.6 (including pypy3).
+
 2.27.1 (2022-01-05)
 -------------------

@@ -33,7 +33,7 @@ Requests is available on PyPI:
 $ python -m pip install requests
 ```

-Requests officially supports Python 2.7 & 3.6+.
+Requests officially supports Python 3.7+.

 ## [NAME_REDACTED] & Best–Practices

@@ -55,16 +55,16 @@ [NAME_REDACTED] gave an excellent summary on
 Python 3 Support?
 -----------------

-Yes! Requests officially supports Python 2.7 & 3.6+ and PyPy.
+Yes! Requests officially supports Python 3.7+ and PyPy.

 Python 2 Support?
 -----------------

-Yes! We understand that we have a large user base with varying needs. Through
-**at least** Requests 2.27.x, we will be providing continued support for Python
-2.7. However, this support is likely to end some time in 2022.
+No! As of Requests 2.28.0, Requests no longer supports Python 2.7. Users who
+have been unable to migrate should pin to `requests<2.28`. Full information
+can be found in `psf/requests#6023 <https://github.com/psf/requests/issues/6023>`_.

-It is *highly* recommended users migrate to Python 3.7+ now since Python
+It is *highly* recommended users migrate to Python 3.8+ now since Python
 2.7 is no longer receiving bug fixes or security updates as of January 1, 2020.

 What are "hostname doesn't match" errors?
@@ -83,10 +83,7 @@ when servers are using `[NAME_REDACTED]`_. When such servers are hosting
 more than one SSL site they need to be able to return the appropriate
 certificate based on the hostname the client is connecting to.

-Python3 and Python 2.7.9+ include native support for SNI in their SSL modules.
-For information on using SNI with Requests on Python < 2.7.9 refer to this
-`Stack Overflow answer`_.
+Python 3 already includes native support for SNI in their SSL modules.

 .. _`Server-Name-Indication`: https://en.wikipedia.org/wiki/Server_Name_Indication
 .. _`virtual hosting`: https://en.wikipedia.org/wiki/Virtual_hosting
-.. _`Stack Overflow answer`: https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484

@@ -72,7 +72,7 @@ Requests is ready for today's web.
 - [NAME_REDACTED]
 - ``.netrc`` Support

-Requests officially supports Python 2.7 & 3.6+, and runs great on PyPy.
+Requests officially supports Python 3.7+, and runs great on PyPy.

 The User Guide

@@ -8,7 +8,7 @@
 which depend on extremely few external helpers (such as compat)
 """

-from .compat import is_py2, builtin_str, str
+from .compat import builtin_str

 def to_native_string(string, encoding='ascii'):
@@ -19,10 +19,7 @@ def to_native_string(string, encoding='ascii'):
     if isinstance(string, builtin_str):
         out = string
     else:
-        if is_py2:
-            out = string.encode(encoding)
-        else:
-            out = string.decode(encoding)
+        out = string.decode(encoding)

     return out

@@ -477,12 +477,7 @@ def send(self, request, stream=False, timeout=None, verify=True, cert=None, prox
                     low_conn.send(b'0\r\n\r\n')

                     # Receive the response from the server
-                    try:
-                        # For Python 2.7, use buffering of HTTP responses
-                        r = low_conn.getresponse(buffering=True)
-                    except TypeError:
-                        # For compatibility with Python 3.3+
-                        r = low_conn.getresponse()
+                    r = low_conn.getresponse()

                     resp = HTTPResponse.from_httplib(
                         r,

@@ -4,8 +4,9 @@
 requests.compat
 ~~~~~~~~~~~~~~~

-This module handles import compatibility issues between Python 2 and
-Python 3.
+This module previously handled import compatibility issues
+between Python 2 and Python 3. It remains for backwards
+compatibility until the next major version.
 """

 try:
@@ -28,54 +29,35 @@
 #: Python 3.x?
 is_py3 = (_ver[0] == 3)

+# json/simplejson module import resolution
 has_simplejson = False
 try:
     import simplejson as json
     has_simplejson = True
 except ImportError:
     import json

+if has_simplejson:
+    from simplejson import JSONDecodeError
+else:
+    from json import JSONDecodeError
+
 # ---------
-# Specifics
+# [NAME_REDACTED]
 # ---------
-
-if is_py2:
-    from urllib import (
-        quote, unquote, quote_plus, unquote_plus, urlencode, getproxies,
-        proxy_bypass, proxy_bypass_environment, getproxies_environment)
-    from urlparse import urlparse, urlunparse, urljoin, urlsplit, urldefrag
-    from urllib2 import parse_http_list
-    import cookielib
-    from Cookie import Morsel
-    from StringIO import StringIO
-    # [NAME_REDACTED] for backwards compatibility.
-    from collections import Callable, Mapping, MutableMapping, OrderedDict
-
-    builtin_str = str
-    bytes = str
-    str = unicode
-    basestring = basestring
-    numeric_types = (int, long, float)
-    integer_types = (int, long)
-    JSONDecodeError = ValueError
-
-elif is_py3:
-    from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
-    from urllib.request import parse_http_list, getproxies, proxy_bypass, proxy_bypass_environment, getproxies_environment
-    from http import cookiejar as cookielib
-    from http.cookies import Morsel
-    from io import StringIO
-    # [NAME_REDACTED] for backwards compatibility.
-    from collections import OrderedDict
-    from collections.abc import Callable, Mapping, MutableMapping
-    if has_simplejson:
-        from simplejson import JSONDecodeError
-    else:
-        from json import JSONDecodeError
-
-    builtin_str = str
-    str = str
-    bytes = bytes
-    basestring = (str, bytes)
-    numeric_types = (int, float)
-    integer_types = (int,)
+from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
+from urllib.request import parse_http_list, getproxies, proxy_bypass, proxy_bypass_environment, getproxies_environment
+from http import cookiejar as cookielib
+from http.cookies import Morsel
+from io import StringIO
+
+# [NAME_REDACTED] for backwards compatibility.
+from collections import OrderedDict
+from collections.abc import Callable, Mapping, MutableMapping
+
+builtin_str = str
+str = str
+bytes = bytes
+basestring = (str, bytes)
+numeric_types = (int, float)
+integer_types = (int,)

@@ -36,8 +36,8 @@ def _implementation():
     """Return a dict with the Python implementation and version.

     Provide both the name and the version of the Python implementation
-    currently running. For example, on CPython 2.7.5 it will return
-    {'name': 'CPython', 'version': '2.7.5'}.
+    currently running. For example, on CPython 3.10.3 it will return
+    {'name': 'CPython', 'version': '3.10.3'}.

     This function works best on CPython and PyPy: in particular, it probably
     doesn't work for Jython or IronPython. Future investigation should be done

@@ -8,7 +8,6 @@
 """

 import datetime
-import sys

 # Import encoding now, to avoid implicit import later.
 # Implicit import within threads may cause LookupError when standard library is in a ZIP,
@@ -45,8 +44,8 @@
     iter_slices, guess_json_utf, super_len, check_header_validity)
 from .compat import (
     Callable, Mapping,
-    cookielib, urlunparse, urlsplit, urlencode, str, bytes,
-    is_py2, chardet, builtin_str, basestring, JSONDecodeError)
+    cookielib, urlunparse, urlsplit, urlencode,
+    chardet, builtin_str, basestring, JSONDecodeError)
 from .compat import json as complexjson
 from .status_codes import codes

@@ -373,7 +372,7 @@ def prepare_url(self, url, params):
         if isinstance(url, bytes):
             url = url.decode('utf8')
         else:
-            url = unicode(url) if is_py2 else str(url)
+            url = str(url)

         # Remove leading whitespaces from url
         url = url.lstrip()
@@ -424,18 +423,6 @@ def prepare_url(self, url, params):
         if not path:
             path = '/'

-        if is_py2:
-            if isinstance(scheme, str):
-                scheme = scheme.encode('utf-8')
-            if isinstance(netloc, str):
-                netloc = netloc.encode('utf-8')
-            if isinstance(path, str):
-                path = path.encode('utf-8')
-            if isinstance(query, str):
-                query = query.encode('utf-8')
-            if isinstance(fragment, str):
-                fragment = fragment.encode('utf-8')
-
         if isinstance(params, (str, bytes)):
             params = to_native_string(params)

@@ -919,10 +906,7 @@ def json(self, **kwargs):
         except JSONDecodeError as e:
             # Catch JSON-related errors and raise as requests.JSONDecodeError
             # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
-            if is_py2: # e is a ValueError
-                raise RequestsJSONDecodeError(e.message)
-            else:
-                raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
+            raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

     [USER_REDACTED]
     def links(self):

@@ -14,7 +14,7 @@
 from collections import OrderedDict

 from .auth import _basic_auth_str
-from .compat import cookielib, is_py3, urljoin, urlparse, Mapping
+from .compat import cookielib, urljoin, urlparse, Mapping
 from .cookies import (
     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT
@@ -39,10 +39,7 @@

 # Preferred clock, based on which one is more accurate on a given system.
 if sys.platform == 'win32':
-    try:  # Python 3.4+
-        preferred_clock = time.perf_counter
-    except AttributeError:  # Earlier than Python 3.
-        preferred_clock = time.clock
+    preferred_clock = time.perf_counter
 else:
     preferred_clock = time.time

@@ -111,8 +108,7 @@ def get_redirect_target(self, resp):
             # It is more likely to get UTF8 header rather than latin1.
             # This causes incorrect handling of UTF8 encoded location headers.
             # To solve this, we re-encode the location in latin1.
-            if is_py3:
-                location = location.encode('latin1')
+            location = location.encode('latin1')
             return to_native_string(location, 'utf8')
         return None

@@ -30,7 +30,7 @@
 from .compat import parse_http_list as _parse_list_header
 from .compat import (
     quote, urlparse, bytes, str, unquote, getproxies,
-    proxy_bypass, urlunparse, basestring, integer_types, is_py3,
+    proxy_bypass, urlunparse, basestring, integer_types,
     proxy_bypass_environment, getproxies_environment, Mapping)
 from .cookies import cookiejar_from_dict
 from .structures import CaseInsensitiveDict
@@ -54,10 +54,7 @@

     def proxy_bypass_registry(host):
         try:
-            if is_py3:
-                import winreg
-            else:
-                import _winreg as winreg
+            import winreg
         except ImportError:
             return False

@@ -281,12 +278,11 @@ def extract_zipped_paths(path):
 [USER_REDACTED].contextmanager
 def atomic_open(filename):
     """Write a file to the disk in an atomic fashion"""
-    replacer = os.rename if sys.version_info[0] == 2 else os.replace
     tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
     try:
         with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
             yield tmp_handler
-        replacer(tmp_name, filename)
+        os.replace(tmp_name, filename)
     except BaseException:
         os.remove(tmp_name)
         raise

@@ -1,5 +1,10 @@
-[bdist_wheel]
-universal = 1
-
 [metadata]
 license_file = LICENSE
+provides-extra =
+    socks
+    use_chardet_on_py3
+requires-dist =
+    certifi>=2017.4.17
+    charset_normalizer~=2.0.0
+    idna>=2.5,<4
+    urllib3>=1.21.1,<1.27

@@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# Learn more: https://github.com/[USER_REDACTED]nethreitz/setup.py
 import os
 import sys

@@ -8,7 +7,27 @@
 from setuptools import setup
 from setuptools.command.test import test as TestCommand

-here = os.path.abspath(os.path.dirname(__file__))
+
+CURRENT_PYTHON = sys.version_info[:2]
+REQUIRED_PYTHON = (3, 7)
+
+if CURRENT_PYTHON < REQUIRED_PYTHON:
+    sys.stderr.write(
+        """
+==========================
+[NAME_REDACTED] version
+==========================
+This version of Requests requires at least Python {}.{}, but
+you're trying to install it on Python {}.{}. To resolve this,
+consider upgrading to a supported Python version.
+
+If you can't upgrade your Python version, you'll need to
+pin to an older version of Requests (<2.28).
+""".format(
+            *(REQUIRED_PYTHON + CURRENT_PYTHON)
+        )
+    )
+    sys.exit(1)

 class PyTest(TestCommand):
     user_options = [('pytest-args=', 'a', "Arguments to pass into py.test")]
@@ -38,27 +57,23 @@ def run_tests(self):
     os.system('twine upload dist/*')
     sys.exit()

-packages = ['requests']
-
 requires = [
-    'charset_normalizer~=2.0.0; python_version >= "3"',
-    'chardet>=3.0.2,<5; python_version < "3"',
-    'idna>=2.5,<3; python_version < "3"',
-    'idna>=2.5,<4; python_version >= "3"',
+    'charset_normalizer~=2.0.0',
+    'idna>=2.5,<4',
     'urllib3>=1.21.1,<1.27',
-    'certifi>=2017.4.17'
-
+    'certifi>=2017.4.17',
 ]
 test_requirements = [
     'pytest-httpbin==0.0.7',
     'pytest-cov',
     'pytest-mock',
     'pytest-xdist',
     'PySocks>=1.5.6, !=1.5.7',
-    'pytest>=3'
+    'pytest>=3',
 ]

 about = {}
+here = os.path.abspath(os.path.dirname(__file__))
 with open(os.path.join(here, 'requests', '__version__.py'), 'r', 'utf-8') as f:
     exec(f.read(), about)

@@ -74,11 +89,11 @@ def run_tests(self):
     author=about['__author__'],
     author_email=about['__author_email__'],
     url=about['__url__'],
-    packages=packages,
+    packages=['requests'],
     package_data={'': ['LICENSE', 'NOTICE']},
     package_dir={'requests': 'requests'},
     include_package_data=True,
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
+    python_requires=">=3.7, <4",
     install_requires=requires,
     license=about['__license__'],
     zip_safe=False,
@@ -90,14 +105,13 @@ def run_tests(self):
         '[NAME_REDACTED] :: English',
         '[NAME_REDACTED] :: OS Independent',
         '[NAME_REDACTED] :: Python',
-        '[NAME_REDACTED] :: Python :: 2',
-        '[NAME_REDACTED] :: Python :: 2.7',
         '[NAME_REDACTED] :: Python :: 3',
-        '[NAME_REDACTED] :: Python :: 3.6',
         '[NAME_REDACTED] :: Python :: 3.7',
         '[NAME_REDACTED] :: Python :: 3.8',
         '[NAME_REDACTED] :: Python :: 3.9',
         '[NAME_REDACTED] :: Python :: 3.10',
+        '[NAME_REDACTED] :: Python :: 3.11',
+        '[NAME_REDACTED] :: Python :: 3 :: Only',
         '[NAME_REDACTED] :: Python :: Implementation :: CPython',
         '[NAME_REDACTED] :: Python :: Implementation :: PyPy',
         'Topic :: Internet :: WWW/HTTP',
@@ -108,7 +122,6 @@ def run_tests(self):
     extras_require={
         'security': [],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
-        'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
         'use_chardet_on_py3': ['chardet>=3.0.2,<5']
     },
     project_urls={

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
-
-from requests.compat import is_py3
-
+import warnings

 try:
     import StringIO
@@ -13,9 +11,14 @@
 except ImportError:
     cStringIO = None

-if is_py3:
-    def u(s):
-        return s
-else:
-    def u(s):
-        return s.decode('unicode-escape')
+
+def u(s):
+    warnings.warn(
+        (
+            "This helper function is no longer relevant in Python 3. "
+            "Usage of this alias should be discontinued as it will be "
+            "removed in a future release of Requests."
+        ),
+        DeprecationWarning,
+    )
+    return s

@@ -2,7 +2,6 @@

 """Tests for Requests."""

-from __future__ import division
 import json
 import os
 import pickle
@@ -18,7 +17,7 @@
 from requests.adapters import HTTPAdapter
 from requests.auth import HTTPDigestAuth, _basic_auth_str
 from requests.compat import (
-    Morsel, cookielib, getproxies, str, urlparse,
+    Morsel, cookielib, getproxies, urlparse,
     builtin_str)
 from requests.cookies import (
     cookiejar_from_dict, morsel_to_cookie)
@@ -47,9 +46,9 @@
 from requests.sessions import SessionRedirectMixin
 from requests.models import urlencode
 from requests.hooks import default_hooks
-from requests.compat import JSONDecodeError, is_py3, MutableMapping
+from requests.compat import JSONDecodeError, MutableMapping

-from .compat import StringIO, u
+from .compat import StringIO
 from .utils import override_environ
 from urllib3.util import Timeout as Urllib3Timeout

@@ -519,7 +518,7 @@ def test_BASICAUTH_TUPLE_HTTP_200_OK_GET(self, httpbin):
     [USER_REDACTED].mark.parametrize(
         'username, password', (
             ('user', 'pass'),
-            (u'имя'.encode('utf-8'), u'пароль'.encode('utf-8')),
+            ('имя'.encode('utf-8'), 'пароль'.encode('utf-8')),
             (42, 42),
             (None, None),
         ))
@@ -843,8 +842,6 @@ def test_conflicting_post_params(self, httpbin):
         with open('requirements-dev.txt') as f:
             with pytest.raises(ValueError):
                 requests.post(url, data='[{"some": "data"}]', files={'some': f})
-            with pytest.raises(ValueError):
-                requests.post(url, data=u('[{"some": "data"}]'), files={'some': f})

     def test_request_ok_set(self, httpbin):
         r = requests.get(httpbin('status', '404'))
@@ -989,8 +986,8 @@ def test_different_encodings_dont_break_post(self, httpbin):

     [USER_REDACTED].mark.parametrize(
         'data', (
-            {'stuff': u('ëlïxr')},
-            {'stuff': u('ëlïxr').encode('utf-8')},
+            {'stuff': 'ëlïxr'},
+            {'stuff': 'ëlïxr'.encode('utf-8')},
             {'stuff': 'elixr'},
             {'stuff': 'elixr'.encode('utf-8')},
         ))
@@ -1013,13 +1010,13 @@ def test_unicode_multipart_post_fieldnames(self, httpbin):
     def test_unicode_method_name(self, httpbin):
         files = {'file': open(__file__, 'rb')}
         r = requests.request(
-            method=u('POST'), url=httpbin('post'), files=files)
+            method='POST', url=httpbin('post'), files=files)
         assert r.status_code == 200

     def test_unicode_method_name_with_request_object(self, httpbin):
         files = {'file': open(__file__, 'rb')}
         s = requests.Session()
-        req = requests.Request(u('POST'), httpbin('post'), files=files)
+        req = requests.Request('POST', httpbin('post'), files=files)
         prep = s.prepare_request(req)
         assert isinstance(prep.method, builtin_str)
         assert prep.method == 'POST'
@@ -1029,7 +1026,7 @@ def test_unicode_method_name_with_request_object(self, httpbin):

     def test_non_prepared_request_error(self):
         s = requests.Session()
-        req = requests.Request(u('POST'), '/')
+        req = requests.Request('POST', '/')

         with pytest.raises(ValueError) as e:
             s.send(req)
@@ -1336,8 +1333,8 @@ def test_response_decode_unicode(self):
     def test_response_reason_unicode(self):
         # check for unicode HTTP status
         r = requests.Response()
-        r.url = u'unicode URL'
-        r.reason = u'Komponenttia ei löydy'.encode('utf-8')
+        r.url = 'unicode URL'
+        r.reason = 'Komponenttia ei löydy'.encode('utf-8')
         r.status_code = 404
         r.encoding = None
         assert not r.ok  # old behaviour - crashes here
@@ -1346,7 +1343,7 @@ def test_response_reason_unicode_fallback(self):
         # check raise_status falls back to ISO-8859-1
         r = requests.Response()
         r.url = 'some url'
-        reason = u'Komponenttia ei löydy'
+        reason = 'Komponenttia ei löydy'
         r.reason = reason.encode('latin-1')
         r.status_code = 500
         r.encoding = None
@@ -1593,7 +1590,7 @@ def test_long_authinfo_in_url(self):
         assert r.url == url

     def test_header_keys_are_native(self, httpbin):
-        headers = {u('unicode'): 'blah', 'byte'.encode('ascii'): 'blah'}
+        headers = {'unicode': 'blah', 'byte'.encode('ascii'): 'blah'}
         r = requests.Request('GET', httpbin('get'), headers=headers)
         p = r.prepare()

@@ -1605,7 +1602,7 @@ def test_header_keys_are_native(self, httpbin):
     def test_header_validation(self, httpbin):
         """Ensure prepare_headers regex isn't flagging valid header contents."""
         headers_ok = {'foo': 'bar baz qux',
-                      'bar': u'fbbq'.encode('utf8'),
+                      'bar': 'fbbq'.encode('utf8'),
                       'baz': '',
                       'qux': '1'}
         r = requests.get(httpbin('get'), headers=headers_ok)
@@ -1885,7 +1882,7 @@ def test_redirect_with_wrong_gzipped_header(self, httpbin):
     [USER_REDACTED].mark.parametrize(
         'username, password, auth_str', (
             ('test', 'test', 'Basic dGVzdDp0ZXN0'),
-            (u'имя'.encode('utf-8'), u'пароль'.encode('utf-8'), 'Basic 0LjQvNGPOtC/0LDRgNC+0LvRjA=='),
+            ('имя'.encode('utf-8'), 'пароль'.encode('utf-8'), 'Basic 0LjQvNGPOtC/0LDRgNC+0LvRjA=='),
         ))
     def test_basic_auth_str_is_always_native(self, username, password, auth_str):
         s = _basic_auth_str(username, password)
@@ -2469,7 +2466,7 @@ def test_data_argument_accepts_tuples(data):
         },
         {
             'method': 'GET',
-            'url': u('http://www.example.com/üniçø∂é')
+            'url': 'http://www.example.com/üniçø∂é'
         },
     ))
 def test_prepared_copy(kwargs):
@@ -2507,39 +2504,39 @@ class TestPreparingURLs(object):
         'url,expected',
         (
             ('http://google.com', 'http://google.com/'),
-            (u'http://ジェーピーニック.jp', u'http://xn--hckqz9bzb1cyrb.jp/'),
-            (u'http://xn--n3h.net/', u'http://xn--n3h.net/'),
+            ('http://ジェーピーニック.jp', 'http://xn--hckqz9bzb1cyrb.jp/'),
+            ('http://xn--n3h.net/', 'http://xn--n3h.net/'),
             (
-                u'http://ジェーピーニック.jp'.encode('utf-8'),
-                u'http://xn--hckqz9bzb1cyrb.jp/'
+                'http://ジェーピーニック.jp'.encode('utf-8'),
+                'http://xn--hckqz9bzb1cyrb.jp/'
             ),
             (
-                u'http://straße.de/straße',
-                u'http://xn--strae-oqa.de/stra%C3%9Fe'
+                'http://straße.de/straße',
+                'http://xn--strae-oqa.de/stra%C3%9Fe'
             ),
             (
-                u'http://straße.de/straße'.encode('utf-8'),
-                u'http://xn--strae-oqa.de/stra%C3%9Fe'
+                'http://straße.de/straße'.encode('utf-8'),
+                'http://xn--strae-oqa.de/stra%C3%9Fe'
             ),
             (
-                u'http://Königsgäßchen.de/straße',
-                u'http://xn--knigsgchen-b4a3dun.de/stra%C3%9Fe'
+                'http://Königsgäßchen.de/straße',
+                'http://xn--knigsgchen-b4a3dun.de/stra%C3%9Fe'
             ),
             (
-                u'http://Königsgäßchen.de/straße'.encode('utf-8'),
-                u'http://xn--knigsgchen-b4a3dun.de/stra%C3%9Fe'
+                'http://Königsgäßchen.de/straße'.encode('utf-8'),
+                'http://xn--knigsgchen-b4a3dun.de/stra%C3%9Fe'
             ),
             (
                 b'http://xn--n3h.net/',
-                u'http://xn--n3h.net/'
+                'http://xn--n3h.net/'
             ),
             (
                 b'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/',
-                u'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/'
+                'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/'
             ),
             (
-                u'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/',
-                u'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/'
+                'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/',
+                'http://[1200:0000:ab00:1234:0000:2552:7777:1313]:12345/'
             )
         )
     )
@@ -2664,7 +2661,6 @@ def test_json_decode_compatibility(self, httpbin):
         assert isinstance(excinfo.value, JSONDecodeError)
         assert r.text not in str(excinfo.value)

-    [USER_REDACTED].mark.skipif(not is_py3, reason="doc attribute is only present on py3")
     def test_json_decode_persists_doc_attr(self, httpbin):
         r = requests.get(httpbin('bytes/20'))
         with pytest.raises(requests.exceptions.JSONDecodeError) as excinfo:

@@ -749,10 +749,7 @@ def test_should_bypass_proxies_win_registry(url, expected, override,
     """
     if override is None:
         override = '192.168.*;127.0.0.1;localhost.localdomain;172.16.1.1'
-    if compat.is_py3:
-        import winreg
-    else:
-        import _winreg as winreg
+    import winreg

     class RegHandle:
         def Close(self):

@@ -78,9 +78,7 @@ def run(self):
     def _create_socket_and_bind(self):
         sock = socket.socket()
         sock.bind((self.host, self.port))
-        # NB: when Python 2.7 is no longer supported, the argument
-        # can be removed to use a default backlog size
-        sock.listen(5)
+        sock.listen()
         return sock

     def _close_server_sock_ignore_errors(self):

@@ -1,5 +1,5 @@
 [tox]
-envlist = py{27,36,37,38,39}-{default,use_chardet_on_py3}
+envlist = py{37,38,39,310}-{default, use_chardet_on_py3}

 [testenv]
 deps = -rrequirements-dev.txt

This is so exciting! Here's a handful of comments/questions:

A few other bits to clean up:

`tox.ini`: add `310`, remove `27` and `36`

Several files: clean up `use_chardet_on_py3`

`requests/utils.py`: `replacer = os.rename if sys.version_info[0] == 2 else os.replace`

`tests/testserver/server.py`:

```python
        # NB: when Python 2.7 is no longer supported, the argument
        # can be removed to use a default backlog size
        sock.listen(5)
```

LGTM! :shipit:

Since we're now Python 3.7+ I wonder if we can start using `time.perf_counter` everywhere? Might need to do more research into why Windows was given only perf_counter/clock instead of `time.time()`

[Finally free...](https://pypi.org/project/win-inet-pton) :smiling_face_with_tear:

Should we go through and remove all uses of `u(...)` in the tests to simplify them?

Do we need `u'...'`?

No we don't, I have a separate linting PR that removes these lines entirely since they're redundant. I can move that into this one.

I left the definition in case it's in use externally, but yes, it should be removed from all of our tests. I'll do a second pass if I missed some.

It was a performance issue for Windows where the clock wasn't granular enough for CI to pass on Appveyor (#3988). I can take a look at standardizing everything onto perf_counter but that's probably a separate deep dive.

`pyupgrade **/*.py --py37-plus` will deal with a bunch of `u''`s and other stuff.

Yep, that's what's in the linting commit currently. Unfortunately, it just makes the two declarations identical so it will still require manual removal in one of the PRs.

```suggestion
```

Perhaps add a deprecation warning for any external use?

We discussed this in the urllib3 Discord with [USER_REDACTED] and [USER_REDACTED] and figured out that this series of steps isn't always going to solve the problem because pip *also* dropped Python versions over time.

There were a few solutions proposed:
- Keep the current message
- Always recommend installing pip<21 (last version that supported Python 2 and 3.5)
- Recommend installing [pip-with-requires-python](https://pypi.org/project/pip-with-requires-python)
- Not give a recommendation on *how* to upgrade pip and setuptools?

I also raised the point of recommending users upgrade pip/setuptools would likely break OS installations.

I think `pip-with-requires-python` is likely the safest option. Alternatively, we'd vend the requires values from the package in our setup.py and compose the install message from those values as [USER_REDACTED] suggested. That avoids any risk for Python 3.4 users who still comprise a significant number of downloads.

[USER_REDACTED] Noted that new Python versions come with a "good" pip version out of the box. Maybe our strategy should be to instruct the user to either pin to <2.28 or upgrade to at least Python 3.7 (since they'll need that version to install Requests anyways) and then we don't need to mention pip/setuptools at all.

I went ahead and [ripped out the whole of touching pip versions](https://github.com/psf/requests/pull/6091/commits/febcd5199c278d99232124e22b3ab2d912b7cc16) after further discussion. If users want to upgrade their pip, that's something they can manage, otherwise the simplest option is to upgrade their Python version which will provide this automatically.

Other users are free to pin to older supported versions as discussed in the original deprecation announcement.

```suggestion
pin to an older version of Requests (<2.28).
```

Let's just go all the way here. :P

Fair enough :D


Limit size of payload in JSONDecodeError

2.27 introduced a change in behavior where now the exception raised by [NAME_REDACTED] json contains the full body of the invalid response. This gets included it's string representation. This can cause problems when the response is very large. This PR tries to limit the size of the response that we store this way, to what might be around the reported error position. But we could also just return to first n bytes or remove the response altogether and let users fetch it, if needed from the error.response object.

@@ -34,6 +34,16 @@ class InvalidJSONError(RequestException):
 class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
     """Couldn't decode the text into json"""

+    def __init__(self, *args, **kwargs):
+        """
+        Construct the JSONDecodeError instance first with all
+        args. Then use it's args to construct the IOError so that
+        the json specific args aren't used as IOError specific args
+        and the error message from JSONDecodeError is preserved.
+        """
+        CompatJSONDecodeError.__init__(self, *args)
+        InvalidJSONError.__init__(self, *self.args, **kwargs)
+

 class HTTPError(RequestException):
     """An HTTP error occurred."""

@@ -23,14 +23,14 @@
     cookiejar_from_dict, morsel_to_cookie)
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
-    MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
+    MissingSchema, ReadTimeout, Timeout, RetryError, RequestException, TooManyRedirects,
     ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
 from requests.models import urlencode
 from requests.hooks import default_hooks
-from requests.compat import MutableMapping
+from requests.compat import JSONDecodeError, is_py3, MutableMapping

 from .compat import StringIO, u
 from .utils import override_environ
@@ -2585,5 +2585,15 @@ def test_post_json_nan(self, httpbin):

     def test_json_decode_compatibility(self, httpbin):
         r = requests.get(httpbin('bytes/20'))
-        with pytest.raises(requests.exceptions.JSONDecodeError):
+        with pytest.raises(requests.exceptions.JSONDecodeError) as excinfo:
             r.json()
+        assert isinstance(excinfo.value, RequestException)
+        assert isinstance(excinfo.value, JSONDecodeError)
+        assert r.text not in str(excinfo.value)
+
+    [USER_REDACTED].mark.skipif(not is_py3, reason="doc attribute is only present on py3")
+    def test_json_decode_persists_doc_attr(self, httpbin):
+        r = requests.get(httpbin('bytes/20'))
+        with pytest.raises(requests.exceptions.JSONDecodeError) as excinfo:
+            r.json()
+        assert excinfo.value.doc == r.text

Are there any tests we can add to verify this doesn't revert back to containing the entirety of `doc` in the `__repr__`? It's also unclear how this prevents that from happening on a visual inspection

Left a few small notes but I think we're close to wrapping this up if [USER_REDACTED] doesn't have more concerns.

Great, thanks for the updates [USER_REDACTED]! I think there were a couple more things we discussed in the last review I left comments on. Let me know if you have any questions.

Looks good to me, thanks [USER_REDACTED]!

Seems good to me.

I'm still not convinced this is maintainable or correct but I also understand that it's probably a tonne of work to get compatibility on python 2 and 3 that isn't a scary nightmare

Can we add comments here explaining why:

1. We're using super old Python 2.5 methods of calling parent `__init__` methods
2. We're calling the `__init__`s in this particular order - presuming the order matters
3. This fixes the issue of `.doc` ending up in the message/default `__repr__`

Can we add an empty line prior to the `__init__` here? That's already the style of this sub-module if not the entirety of `requests`

nit; This might be phrased "[...] the error message from JSONDecodeError is preserved."

Also note the missing period in the current sentence.

Do we need to use `.value` in these assertions? Just `excinfo` should be sufficient and that's how the typical user will probably interact with them.

Also somewhat of a nit. It may be better to just make this its own test with a `[USER_REDACTED].skipIf(not is_py3)` at the top.

e.g.

```python
[USER_REDACTED].skipIf(not is_py3)
def test_json_decode_persists_doc_attr(self, httpbin):
    r = requests.get(httpbin('bytes/20'))
    with pytest.raises(requests.exceptions.JSONDecodeError) as e:
        r.json()
    assert e.doc == r.text
```

don't we want to test that the py2 object is also what we expect? Or if we don't can i can make it py3 only?

excinfo is a pytest thing that wraps the actual exception, i think we have to look at value to get the original object

updated

The original test doesn't assert anything about Python 2 either. So we don't lose anything with this change. I suppose we could assert doc _isn't_ on the Python 2 exception but I'm not sure how meaningful that is.

Edit: to be clear I'm proposing two tests. The one you have and moving the Python 3 specifics to its own test.

ok, i think i updated it with what you had in mind.

I think the last sentence still needs a period at the end.

We probably don't need to reassert this since it's already handled in the test above.

nit; since we're only using this once we can probably just use r.text in the final assertion as shown in the original proposal.

I liked having both in the same test, assures us that we are testing with the right string. but i'll take it out.


Refactor rebuild_proxies to separate resolution and auth handling

This is a proposal for handling #5888 that's come out of discussion in an alternative PR (#5893). Given that we've stalled out on progress there for a few weeks, this PR is to help drive a conclusion. The approach taken here is to mitigate the incorrect behavior of stripping `Proxy-Authorization` headers off all requests sent with `Session.send`. This will not address the performance concerns from #5891 as that's a more significant problem without a clear answer.

The goal of driving this separately from #5891 is that one is an unintended breakage in basic behavior of the library and the second is an unfortunate performance regression. I'd like to address the former more quickly if possible instead of letting the problem fester.

@@ -29,7 +29,7 @@

 from .utils import (
     requote_uri, get_environ_proxies, get_netrc_auth, should_bypass_proxies,
-    get_auth_from_url, rewind_body
+    get_auth_from_url, rewind_body, resolve_proxies
 )

 from .status_codes import codes
@@ -269,7 +269,6 @@ def rebuild_auth(self, prepared_request, response):
         if new_auth is not None:
             prepared_request.prepare_auth(new_auth)

-
     def rebuild_proxies(self, prepared_request, proxies):
         """This method re-evaluates the proxy configuration by [NAME_REDACTED] If we are redirected to a URL covered by
@@ -282,21 +281,9 @@ def rebuild_proxies(self, prepared_request, proxies):

         :rtype: dict
         """
-        proxies = proxies if proxies is not None else {}
         headers = prepared_request.headers
-        url = prepared_request.url
-        scheme = urlparse(url).scheme
-        new_proxies = proxies.copy()
-        no_proxy = proxies.get('no_proxy')
-
-        bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)
-        if self.trust_env and not bypass_proxy:
-            environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
-
-            proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
-
-            if proxy:
-                new_proxies.setdefault(scheme, proxy)
+        scheme = urlparse(prepared_request.url).scheme
+        new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env)

         if 'Proxy-Authorization' in headers:
             del headers['Proxy-Authorization']
@@ -633,7 +620,10 @@ def send(self, request, **kwargs):
         kwargs.setdefault('stream', self.stream)
         kwargs.setdefault('verify', self.verify)
         kwargs.setdefault('cert', self.cert)
-        kwargs.setdefault('proxies', self.rebuild_proxies(request, self.proxies))
+        if 'proxies' not in kwargs:
+            kwargs['proxies'] = resolve_proxies(
+                request, self.proxies, self.trust_env
+            )

         # It's possible that users might accidentally send a Request object.
         # Guard against that specific failure case.

@@ -830,6 +830,34 @@ def select_proxy(url, proxies):
     return proxy

+def resolve_proxies(request, proxies, trust_env=True):
+    """This method takes proxy information from a request and configuration
+    input to resolve a mapping of target proxies. This will consider settings
+    such a NO_PROXY to strip proxy configurations.
+
+    :param request: Request or PreparedRequest
+    :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs
+    :param trust_env: Boolean declaring whether to trust environment configs
+
+    :rtype: dict
+    """
+    proxies = proxies if proxies is not None else {}
+    url = request.url
+    scheme = urlparse(url).scheme
+    no_proxy = proxies.get('no_proxy')
+    new_proxies = proxies.copy()
+
+    bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)
+    if trust_env and not bypass_proxy:
+        environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
+
+        proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
+
+        if proxy:
+            new_proxies.setdefault(scheme, proxy)
+    return new_proxies
+
+
 def default_user_agent(name="python-requests"):
     """
     Return a string representing the default user agent.

@@ -590,6 +590,15 @@ def test_respect_proxy_env_on_request(self, httpbin):
                 session = requests.Session()
                 session.request(method='GET', url=httpbin())

+    def test_proxy_authorization_preserved_on_request(self, httpbin):
+        proxy_auth_value = "Bearer XXX"
+        session = requests.Session()
+        session.headers.update({"Proxy-Authorization": proxy_auth_value})
+        resp = session.request(method='GET', url=httpbin('get'))
+        sent_headers = resp.json().get('headers', {})
+
+        assert sent_headers.get("Proxy-Authorization") == proxy_auth_value
+
     def test_basicauth_with_netrc(self, httpbin):
         auth = ('user', 'pass')
         wrong_auth = ('wronguser', 'wrongpass')
@@ -2575,4 +2584,4 @@ def test_post_json_nan(self, httpbin):
     def test_json_decode_compatibility(self, httpbin):
         r = requests.get(httpbin('bytes/20'))
         with pytest.raises(requests.exceptions.JSONDecodeError):
-            r.json()
\ No newline at end of file
+            r.json()

Hey [USER_REDACTED], thanks for this!
I added a few comments following up on our discussions.

This change should actually fix the performance regression in most cases (`Session.request` - which I assume is the most common flow - always sets `proxies` on `kwargs`).

So now we're:
1. No longer stripping the proxy-auth header. Sounds right to me.
2. No longer setting the proxy-auth header at all. Is this the desired behavior? What if someone uses a proxy and passes the username + password in the url? (though I'm not sure if that flow is supported or not)

Didn't we want to unify 850 and 851 so we don't call `should_bypass_proxies` if `trust_env` is `False`?

For 2.) we were never doing that before this change and I don't think that was an intended byproduct. We should still support our normal proxy auth flows that were available prior to #5681.

We do, I was going to have [USER_REDACTED] rebase their change onto whatever we merge so they can still have a commit in the repo.

Yep, I think `proxies` will be the escape hatch when performance is a concern here :)

Sounds good to me.


Move from urlparse to parse_url for prepending schemes

This addresses the base problem raised in #5855 with the `urlparse` changes in Python 3.9 and potential issues in future 3.7/3.8 releases. We've avoided changing other uses of `urlparse` for now due to issues with `parse_url`'s strict validation requirements when performing parsing.

@@ -21,6 +21,7 @@
 import zipfile
 from collections import OrderedDict
 from urllib3.util import make_headers
+from urllib3.util import parse_url

 from .__version__ import __version__
 from . import certs
@@ -963,15 +964,23 @@ def prepend_scheme_if_needed(url, new_scheme):

     :rtype: str
     """
-    scheme, netloc, path, params, query, fragment = urlparse(url, new_scheme)
-
-    # urlparse is a finicky beast, and sometimes decides that there isn't a
-    # netloc present. Assume that it's being over-cautious, and switch netloc
-    # and path if urlparse decided there was no netloc.
+    parsed = parse_url(url)
+    scheme, auth, host, port, path, query, fragment = parsed
+
+    # A defect in urlparse determines that there isn't a netloc present in some
+    # urls. We previously assumed parsing was overly cautious, and swapped the
+    # netloc and path. Due to a lack of tests on the original defect, this is
+    # maintained with parse_url for backwards compatibility.
+    netloc = parsed.netloc
     if not netloc:
         netloc, path = path, netloc

-    return urlunparse((scheme, netloc, path, params, query, fragment))
+    if scheme is None:
+        scheme = new_scheme
+    if path is None:
+        path = ''
+
+    return urlunparse((scheme, netloc, path, '', query, fragment))

 def get_auth_from_url(url):

@@ -601,6 +601,7 @@ def test_parse_header_links(value, expected):
     'value, expected', (
         ('example.com/path', 'http://example.com/path'),
         ('//example.com/path', 'http://example.com/path'),
+        ('example.com:80', 'http://example.com:80'),
     ))
 def test_prepend_scheme_if_needed(value, expected):
     assert prepend_scheme_if_needed(value, 'http') == expected

Uhhh, what? This doesn't make any sense to me, is this something that's only needed for `urlparse`?

Yeah, this is only a quirk for `urlparse`. I'd thought I repro'd with `parse_url` initially, but I'm unable to now.

I'll reword the whole comment for clarity. I'm not confident the test suite covers all of the initial issue though. I think I'm still in favor of leaving the swap in place as a fallback for anything we're missing.

[USER_REDACTED] I've reworded things to be a bit clearer.


Add thread locker workflow

@@ -0,0 +1,15 @@
+name: '[NAME_REDACTED]'
+
+on:
+  schedule:
+    - cron: '0 0 * * *'
+
+permissions:
+  issues: write
+  pull-requests: write
+
+jobs:
+  action:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: dessant/lock-threads@v2

Just a minor fix needed and then I think this looks good. Thanks, [USER_REDACTED]!

It looks like this got clipped during copy. `me`->`name`

Actually one more thought while we're here. It looks like some other projects have [shortened the window](https://github.com/pyca/pyopenssl/blob/main/.github/workflows/lock.yml#L13-L14) from 365 days to 90. That may be more reasonable to prevent unrelated pings resetting such a long timer.

Oh, I missed this. Hold on. #5911


Fix inconsistent exception type in response.json() method

## Summary
Fixes the inconsistency of errors thrown in the `response.json()` method in `models.py`, while preserving backwards compatibility.

### What we used to have
Depending on whether or not `simplejson` was installed in the user's library, and whether the user was running Python 3+ versus Python 2, a different exception would get thrown when there was an issue with decoding response text as JSON. If `simplejson` was installed, `simplejson.JSONDecodeError` would get thrown. If not, `json.JSONDecodeError` would get thrown for Python 3+ users, and `ValueError` for Python 2 users. Thus, depending on the scenario, users would find themselves having to catch either `simplejson.JSONDecodeError`, `json.JSONDecodeError`, or `ValueError` in the case of the `response.json()` method failing. This inconsistency is not ideal.

### What we have now
There is now one error class in `exceptions.py` that will represent all errors that can be thrown or caught in the `response.json()` method: `requests.JSONDecodeError`. All `simplejson` functionality was replaced with `json` functionality, but the new error type aliases both `json.JSONDecodeError` and `simplejson.JSONDecodeError`. If `simplejson` is not installed, its `JSONDecodError` is replaced with a plain `Exception` when aliased. If the user is running Python 3+, `json.JSONDecodeError` is aliased, but if the user is running Python 2, that is replaced with `ValueError` when aliased, as the `JSONDecodeError` was not previously a part of the `json` library. Now, all five error types, `json.JSONDecodeError`, `simplejson.JSONDecodeError`, `ValueError`, and `requests.JSONDecodeError` and its parent class `requests.RequestException`, will be caught from the `requests.JSONDecodeError` that will be raised.

Fixes #5794
[USER_REDACTED]

@@ -4,7 +4,12 @@ Release History
 dev
 ---

--   \[Short description of non-trivial change.\]
+- \[Short description of non-trivial change.\]
+
+- Added a `requests.exceptions.JSONDecodeError` to decrease inconsistencies
+  in the library. This gets raised in the `response.json()` method, and is
+  backwards compatible as it inherits from previously thrown exceptions.
+  Can be caught from `requests.exceptions.RequestException` as well.

 2.26.0 (2021-07-13)
 -------------------

@@ -153,9 +153,9 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::

 In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 ([NAME_REDACTED]), or if the response contains invalid JSON,
-attempting ``r.json()`` raises ``simplejson.JSONDecodeError`` if simplejson is
-installed or raises ``ValueError: No JSON object could be decoded`` on Python 2 or
-``json.JSONDecodeError`` on Python 3.
+attempting ``r.json()`` raises ``requests.exceptions.JSONDecodeError``. This wrapper exception
+provides interoperability for multiple exceptions that may be thrown by different
+python versions and json serialization libraries.

 It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a

@@ -139,7 +139,7 @@ def _check_cryptography(cryptography_version):
 from .exceptions import (
     RequestException, Timeout, URLRequired,
     TooManyRedirects, HTTPError, ConnectionError,
-    FileModeWarning, ConnectTimeout, ReadTimeout
+    FileModeWarning, ConnectTimeout, ReadTimeout, JSONDecodeError
 )

 # Set default logging handler to avoid "No handler found" warnings.

@@ -28,8 +28,10 @@
 #: Python 3.x?
 is_py3 = (_ver[0] == 3)

+has_simplejson = False
 try:
     import simplejson as json
+    has_simplejson = True
 except ImportError:
     import json

@@ -49,13 +51,13 @@
     # [NAME_REDACTED] for backwards compatibility.
     from collections import Callable, Mapping, MutableMapping, OrderedDict

-
     builtin_str = str
     bytes = str
     str = unicode
     basestring = basestring
     numeric_types = (int, long, float)
     integer_types = (int, long)
+    JSONDecodeError = ValueError

 elif is_py3:
     from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
@@ -66,6 +68,10 @@
     # [NAME_REDACTED] for backwards compatibility.
     from collections import OrderedDict
     from collections.abc import Callable, Mapping, MutableMapping
+    if has_simplejson:
+        from simplejson import JSONDecodeError
+    else:
+        from json import JSONDecodeError

     builtin_str = str
     str = str

@@ -8,6 +8,8 @@
 """
 from urllib3.exceptions import HTTPError as BaseHTTPError

+from .compat import JSONDecodeError as CompatJSONDecodeError
+

 class RequestException(IOError):
     """There was an ambiguous exception that occurred while handling your
@@ -29,6 +31,10 @@ class InvalidJSONError(RequestException):
     """A JSON error occurred."""

+class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
+    """Couldn't decode the text into json"""
+
+
 class HTTPError(RequestException):
     """An HTTP error occurred."""

@@ -29,7 +29,9 @@
 from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .exceptions import (
     HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError, InvalidJSONError)
+    ContentDecodingError, ConnectionError, StreamConsumedError,
+    InvalidJSONError)
+from .exceptions import JSONDecodeError as RequestsJSONDecodeError
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
@@ -38,7 +40,7 @@
 from .compat import (
     Callable, Mapping,
     cookielib, urlunparse, urlsplit, urlencode, str, bytes,
-    is_py2, chardet, builtin_str, basestring)
+    is_py2, chardet, builtin_str, basestring, JSONDecodeError)
 from .compat import json as complexjson
 from .status_codes import codes

@@ -468,9 +470,9 @@ def prepare_body(self, data, files, json=None):
             content_type = 'application/json'

             try:
-              body = complexjson.dumps(json, allow_nan=False)
+                body = complexjson.dumps(json, allow_nan=False)
             except ValueError as ve:
-              raise InvalidJSONError(ve, request=self)
+                raise InvalidJSONError(ve, request=self)

             if not isinstance(body, bytes):
                 body = body.encode('utf-8')
@@ -882,12 +884,8 @@ def json(self, **kwargs):
         r"""Returns the json-encoded content of a response, if any.

         :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
-        :raises simplejson.JSONDecodeError: If the response body does not
-            contain valid json and simplejson is installed.
-        :raises json.JSONDecodeError: If the response body does not contain
-            valid json and simplejson is not installed on Python 3.
-        :raises ValueError: If the response body does not contain valid
-            json and simplejson is not installed on Python 2.
+        :raises requests.exceptions.JSONDecodeError: If the response body does not
+            contain valid json.
         """

         if not self.encoding and self.content and len(self.content) > 3:
@@ -907,7 +905,16 @@ def json(self, **kwargs):
                     # and the server didn't bother to tell us what codec *was*
                     # used.
                     pass
-        return complexjson.loads(self.text, **kwargs)
+
+        try:
+            return complexjson.loads(self.text, **kwargs)
+        except JSONDecodeError as e:
+            # Catch JSON-related errors and raise as requests.JSONDecodeError
+            # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
+            if is_py2: # e is a ValueError
+                raise RequestsJSONDecodeError(e.message)
+            else:
+                raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

     [USER_REDACTED]
     def links(self):

@@ -2570,4 +2570,9 @@ def test_parameters_for_nonstandard_schemes(self, input, params, expected):
     def test_post_json_nan(self, httpbin):
         data = {"foo": float("nan")}
         with pytest.raises(requests.exceptions.InvalidJSONError):
-          r = requests.post(httpbin('post'), json=data)
\ No newline at end of file
+          r = requests.post(httpbin('post'), json=data)
+
+    def test_json_decode_compatibility(self, httpbin):
+        r = requests.get(httpbin('bytes/20'))
+        with pytest.raises(requests.exceptions.JSONDecodeError):
+            r.json()
\ No newline at end of file

Thanks for the updates, [USER_REDACTED]. I added a couple of comments for discussion. I'm not convinced they're necessarily the right direction, but it does feel like the simplejson/json problems are potentially being spread more places than needed.

Let me know what you think, thanks!

Cool, I think this looks like we’re ready now. Thanks for seeing this through, [USER_REDACTED].

For merging, I’m not sure if we want to put this on a 2.27.x branch while we wait for the next release, or if we merge now knowing we won’t do another patch release for 2.26.x. Any thoughts [USER_REDACTED]?

This is backwards incompatible and needs to be reverted

I forgot we had this exception, where do we use it? Should we sub-class this in `JSONDecodeError` as well?

Why did you change this?

Please revert this. It's tangential to the change

Please revert this as well

Please revert this

Please revert this

Please revert this

Please revert this

Please revert this

Please revert this

Please revert this

Please document this as `requests.exceptions.JSONDecodeError`. Eventually I desperately want to stop importing every exception class into `requests/__init__.py`

We should probably preserve the original exception instance in this as well and proxy attributes down to it. That way folks can find the original exception. I'm not certain `simplejson` and `json` have the same exception attributes with the same meaning

Please revert this

Please revert this

We must be clear that this is _new_ in the next version of Requests (add it to our changelog too) and that it is backwards compatible as it inherits from the exceptions raised previously

Do you want me to add it to the changelog now or do you like to do that?

I added this exception to sub-class both `JSONDecodeError`s

Please add it to the changelog now

I'm talking about the `InvalidJSONError` exception here (the line I commented on)

We probably don't want to add all the information in the last sentence as it will become stale quickly. Let's simplfy it to something closer to:

> This wrapper exception provides interoperability for multiple exceptions that may be thrown by [NAME_REDACTED] json serialization libraries.

These spacing changes aren't really related to the current PR, can we remove them.

Our custom exceptions in Requests should ideally fall under a catch-all parent exception, `RequestException`. That allows users to catch everything thrown by Requests.

I think we're still waiting on a response here, [USER_REDACTED].

I'm not sure I'm following how this will work in the Python 2 `json` case. Where is `ValueError` being handled?

https://github.com/psf/requests/pull/5856/checks?check_run_id=3031103164 Yeah, this is broken. We need to be able to handle this a different way.

I see your last comment was made outside of this thread, [USER_REDACTED]. To clarify, while `json.JSONDecodeError` does inherit from `ValueError`, it's not going to work in this case. The exception doesn't exist in Python 2, so we fall over with an attribute error when handling the exception. You'll need to take another approach to handle this.

Yes, I can subclass that in `JSONDecodeError` as well.

Giving the user 0 useful information is unacceptable. Why can't we use `e.message` there?

```py
Python 2.7.18 (default, May 19 2021, 00:00:00)
[GCC 11.1.1 20210428 ([NAME_REDACTED] 11.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> try:
...   json.loads("foo")
... except ValueError as ve:
...   pass
...
>>> dir(ve)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', 'args', 'message']
>>> ve.message
'No JSON object could be decoded'
>>> ve.args
('No JSON object could be decoded',)
>>>
```

It doesn't look like we use this anywhere except to declare the exceptions. We're already setting either dependency to be imported under the `json` namespace. Can we not just do `from json import JSONDecodeError` under the py3 section and have it automatically resolve? Perhaps adding a short comment above the import saying it's using simplejson or json depending on what was resolved?

Why did you remove this?

2.26.0 was released. This can not be added to those notes as this was not released with 2.26.0 please move it above the `2.26.0 (2021-07-13)` line.

These changes in this file are unrelated and should be in a separate PR

We have logic for which simplejson/json exceptions we're using spread across a few places now. Is there a compelling reason not to simply import JSONDecodeError from compat since it's already been resolved and supplying it as a single parent class?

[USER_REDACTED] We want to know which properties the `JSONDecodeError` has, since it may have different properties depending on compatibility. That requires an extra check on import.

[USER_REDACTED] I'm not sure I'm following, could you clarify which properties you're referring to? We have 3 parent classes that we'd ever use in our environment, `json.JSONDecodeError`, `simplejson.JSONDecodeError`, and `ValueError`. Only one of those can be relevant in a given execution of Requests.

What I was asking about was if we did:

```python
from .compat import JSONDecodeError as CompatJSONDecodeError

[...]

class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
```

We've already resolved these values once in compat, so I wanted to know why we're doing it again here.

[USER_REDACTED] For Python 3, in the `compat.py` file itself, reference to the `json` package directly, whether aliased as `json` or actually the `json` package itself, would always look for the `json` package rather than whatever was aliased as it. So, just in the `compat.py` file itself, I used the check. From all other scripts that have `from .compat import json`, however, it works as expected.

Did you have any more questions about the code?

Hey [USER_REDACTED], I think there may still be some confusion. If you open the _Files changed_ tab, you should be able to see the full context of this thread. I've written out the [full diff](https://github.com/nateprewitt/requests/commit/665cf30861fb356d84103808e240d6940e8ff3d6) from the suggestion above to minimize ambiguity. You should find this passes the test suite without issue, so please let me know if there were other concerns you had.

While we're here it would also be good to add a test exercising this change. Something to the effect of the example below should be sufficient:

```
def test_json_decode_compatibility(self, httpbin):
    r = requests.get(httpbin('bytes/20'))
    with pytest.raises(requests.exceptions.JSONDecodeError):
        r.json()
```

This won't cover simplejson in our general test suite, but I've done a one off run to ensure that's working.

Thank you, that makes total sense. I made those changes and I'm running workflows right now.


Throw value error when serializing JSON object with NaN value

I am attempting to fix #5767 with this PR. Feedback is really appreciated.

@@ -25,6 +25,10 @@ def __init__(self, *args, **kwargs):
         super(RequestException, self).__init__(*args, **kwargs)

+class InvalidJSONError(RequestException):
+    """A JSON error occurred."""
+
+
 class HTTPError(RequestException):
     """An HTTP error occurred."""

@@ -29,7 +29,7 @@
 from .cookies import cookiejar_from_dict, get_cookie_header, _copy_cookie_jar
 from .exceptions import (
     HTTPError, MissingSchema, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError)
+    ContentDecodingError, ConnectionError, StreamConsumedError, InvalidJSONError)
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
@@ -466,7 +466,12 @@ def prepare_body(self, data, files, json=None):
             # urllib3 requires a bytes-like body. Python 2's json.dumps
             # provides this natively, but Python 3 gives a Unicode string.
             content_type = 'application/json'
-            body = complexjson.dumps(json)
+
+            try:
+              body = complexjson.dumps(json, allow_nan=False)
+            except ValueError as ve:
+              raise InvalidJSONError(ve, request=self)
+
             if not isinstance(body, bytes):
                 body = body.encode('utf-8')

@@ -24,7 +24,7 @@
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidSchema, InvalidURL,
     MissingSchema, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL)
+    ProxyError, InvalidHeader, UnrewindableBodyError, SSLError, InvalidProxyURL, InvalidJSONError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -2566,3 +2566,8 @@ def test_parameters_for_nonstandard_schemes(self, input, params, expected):
         r = requests.Request('GET', url=input, params=params)
         p = r.prepare()
         assert p.url == expected
+
+    def test_post_json_nan(self, httpbin):
+        data = {"foo": float("nan")}
+        with pytest.raises(requests.exceptions.InvalidJSONError):
+          r = requests.post(httpbin('post'), json=data)
\ No newline at end of file

I don't believe a `request()` method will throw a ValueError ever and so this constitutes a backwards incompatible change. We should create a new exception, however, that inherits from `RequestsException` to cover this case and do something like

```py
try:
    body = complexjson.dumps(json, allow_nan=False)
except ValueError as ve:
    raise InvalidJSONError(json, original_exc=ve)
```

[USER_REDACTED] is this okay? Couldn't do it the exact way you specified, but `request` contains the JSON info.

I have an instinct of "No" but that's because people do not read the docs and sometimes pass open files and such _in addition to_ the `json=` keyword and keeping a reference to those might not be _the best_ but that's a "The user has shot themselves in the foot" situation only for high traffic (lots of requests doing the wrong thing explicitly that has undefined behaviour), poorly designed software (because it's potentially not releasing/decref'ing the exception which doesn't decref the request which doesn't decref the file obj).

tl;dr - This should be fine


Switch LGPL'd chardet for MIT licensed charset_normalizer

At least for Python 3 -- charset_normalizer doesn't support Python2, so for that chardet is still used -- this means the "have chardet" path is also still tested.

Although using the (non-vendored) chardet library is fine for requests itself, but using a LGPL dependency the story is a lot less clear for downstream projects, particularly ones that might like to bundle requests (and thus chardet) in to a single binary -- think something similar to what docker-compose is doing. By [NAME_REDACTED] it is no longer clear if the resulting artefact must also be LGPL'd.

By [NAME_REDACTED] for one under MIT we remove all license ambiguity.

As an "escape hatch" I have made the code so that it will use chardet first if it is installed, but we no longer depend upon it directly, although there is a new extra added, `requests[lgpl]`. This should minimize the impact to users, and give them an escape hatch if charset_normalizer turns out to be not as good. (In my non-exhaustive tests it detects the same encoding as chartdet in every case I threw at it)

I've read https://github.com/psf/requests/pull/4115, https://github.com/psf/requests/issues/3389, and https://github.com/chardet/chardet/issues/36#issuecomment-768281452 so I'm aware of the history, but I hope that the approach in this PR will allow this to be merged, as right now, the [NAME_REDACTED] doesn't allow projects to depend upon LGPL'd code (this is something I'm trying to get changed, but it is a _very_ slow process)

@@ -23,6 +23,12 @@ env/

 .workon

+# in case you work with IntelliJ/PyCharm
+.idea
+*.iml
+.python-version
+
+
 t.py

 t2.py

@@ -6,6 +6,22 @@ dev

 -   \[Short description of non-trivial change.\]

+**Dependencies**
+
+- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
+  to remove license ambiguity for projects bundling requests. If `chardet`
+  is already installed on your machine it will be used instead of `charset_normalizer`
+  to keep backwards compatibility.
+
+  You can also install `chardet` while installing requests by
+  specifying `[use_chardet_on_py3]` extra as follows:
+
+    ```shell
+    pip install "requests[use_chardet_on_py3]"
+    ```
+
+  Python2 still depends upon the `chardet` module.
+
 2.25.1 (2020-12-16)
 -------------------

@@ -1707,4 +1723,3 @@ This is not a backwards compatible change.

 -   Frustration
 -   Conception
-

@@ -697,10 +697,22 @@ [NAME_REDACTED] you receive a response, Requests makes a guess at the encoding to
 use for decoding the response when you access the :attr:`Response.text
 <requests.Response.text>` attribute. Requests will first check for an
-encoding in the HTTP header, and if none is present, will use `chardet
-<https://pypi.org/project/chardet/>`_ to attempt to guess the encoding.
-
-The only time Requests will not do this is if no explicit charset
+encoding in the HTTP header, and if none is present, will use
+`charset_normalizer <https://pypi.org/project/charset_normalizer/>`_
+or `chardet <https://github.com/chardet/chardet>`_ to attempt to
+guess the encoding.
+
+If ``chardet`` is installed, ``requests`` uses it, however for python3
+``chardet`` is no longer a mandatory dependency. The ``chardet``
+library is an LGPL-licenced dependency and some users of requests
+cannot depend on mandatory LGPL-licensed dependencies.
+
+When you install ``request`` without specifying ``[use_chardet_on_py3]]`` extra,
+and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
+(MIT-licensed) to guess the encoding. For Python 2, ``requests`` uses only
+``chardet`` and is a mandatory dependency there.
+
+The only time Requests will not guess the encoding is if no explicit charset
 is present in the HTTP headers **and** the ``Content-Type``
 header contains ``text``. In this situation, `RFC 2616
 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies

@@ -41,12 +41,20 @@
 """

 import urllib3
-import chardet
 import warnings
 from .exceptions import RequestsDependencyWarning

+try:
+    from charset_normalizer import __version__ as charset_normalizer_version
+except ImportError:
+    charset_normalizer_version = None

-def check_compatibility(urllib3_version, chardet_version):
+try:
+    from chardet import __version__ as chardet_version
+except ImportError:
+    chardet_version = None
+
+def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
     urllib3_version = urllib3_version.split('.')
     assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.

@@ -62,12 +70,19 @@ def check_compatibility(urllib3_version, chardet_version):
     assert minor >= 21
     assert minor <= 26

-    # Check chardet for compatibility.
-    major, minor, patch = chardet_version.split('.')[:3]
-    major, minor, patch = int(major), int(minor), int(patch)
-    # chardet >= 3.0.2, < 5.0.0
-    assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
-
+    # Check charset_normalizer for compatibility.
+    if chardet_version:
+        major, minor, patch = chardet_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # chardet_version >= 3.0.2, < 5.0.0
+        assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
+    elif charset_normalizer_version:
+        major, minor, patch = charset_normalizer_version.split('.')[:3]
+        major, minor, patch = int(major), int(minor), int(patch)
+        # charset_normalizer >= 2.0.0 < 3.0.0
+        assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
+    else:
+        raise Exception("You need either charset_normalizer or chardet installed")

 def _check_cryptography(cryptography_version):
     # cryptography < 1.3.4
@@ -82,10 +97,10 @@ def _check_cryptography(cryptography_version):

 # Check imported dependencies for compatibility.
 try:
-    check_compatibility(urllib3.__version__, chardet.__version__)
+    check_compatibility(urllib3.__version__, chardet_version, charset_normalizer_version)
 except (AssertionError, ValueError):
-    warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
-                  "version!".format(urllib3.__version__, chardet.__version__),
+    warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
+                  "version!".format(urllib3.__version__, chardet_version, charset_normalizer_version),
                   RequestsDependencyWarning)

 # Attempt to enable urllib3's fallback for SNI support

@@ -8,7 +8,10 @@
 Python 3.
 """

-import chardet
+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet

 import sys

@@ -8,10 +8,19 @@

 import idna
 import urllib3
-import chardet

 from . import __version__ as requests_version

+try:
+    import charset_normalizer
+except ImportError:
+    charset_normalizer = None
+
+try:
+    import chardet
+except ImportError:
+    chardet = None
+
 try:
     from urllib3.contrib import pyopenssl
 except ImportError:
@@ -71,7 +80,12 @@ def info():

     implementation_info = _implementation()
     urllib3_info = {'version': urllib3.__version__}
-    chardet_info = {'version': chardet.__version__}
+    charset_normalizer_info = {'version': None}
+    chardet_info = {'version': None}
+    if charset_normalizer:
+        charset_normalizer_info = {'version': charset_normalizer.__version__}
+    if chardet:
+        chardet_info = {'version': chardet.__version__}

     pyopenssl_info = {
         'version': None,
@@ -99,9 +113,11 @@ def info():
         'implementation': implementation_info,
         'system_ssl': system_ssl_info,
         'using_pyopenssl': pyopenssl is not None,
+        'using_charset_normalizer': chardet is None,
         'pyOpenSSL': pyopenssl_info,
         'urllib3': urllib3_info,
         'chardet': chardet_info,
+        'charset_normalizer': charset_normalizer_info,
         'cryptography': cryptography_info,
         'idna': idna_info,
         'requests': {

@@ -731,7 +731,7 @@ def next(self):

     [USER_REDACTED]
     def apparent_encoding(self):
-        """The apparent encoding, provided by [NAME_REDACTED]"""
+        """The apparent encoding, provided by [NAME_REDACTED]_normalizer or chardet libraries."""
         return chardet.detect(self.content)['encoding']

     def iter_content(self, chunk_size=1, decode_unicode=False):
@@ -845,7 +845,7 @@ def text(self):
         """Content of the response, in unicode.

         If Response.encoding is None, encoding will be guessed using
-        ``chardet``.
+        ``charset_normalizer`` or ``chardet``.

         The encoding of the response content is determined based solely on HTTP
         headers, following RFC 2616 to the letter. If you can take advantage of
@@ -893,7 +893,7 @@ def json(self, **kwargs):
         if not self.encoding and self.content and len(self.content) > 3:
             # No encoding set. JSON RFC 4627 section 3 states we should expect
             # UTF-8, -16 or -32. Detect which one to use; If the detection or
-            # decoding fails, fall back to `self.text` (using chardet to make
+            # decoding fails, fall back to `self.text` (using charset_normalizer to make
             # a best guess).
             encoding = guess_json_utf(self.content)
             if encoding is not None:

@@ -1,14 +1,26 @@
 import sys

+try:
+    import chardet
+except ImportError:
+    import charset_normalizer as chardet
+    import warnings
+
+    warnings.filterwarnings('ignore', 'Trying to detect', module='charset_normalizer')
+
 # This code exists for backwards compatibility reasons.
 # I don't like it either. Just look the other way. :)

-for package in ('urllib3', 'idna', 'chardet'):
+for package in ('urllib3', 'idna'):
     locals()[package] = __import__(package)
     # This traversal is apparently necessary such that the identities are
     # preserved (requests.packages.urllib3.* is urllib3.*)
     for mod in list(sys.modules):
         if mod == package or mod.startswith(package + '.'):
             sys.modules['requests.packages.' + mod] = sys.modules[mod]

+target = chardet.__name__
+for mod in list(sys.modules):
+    if mod == target or mod.startswith(target + '.'):
+        sys.modules['requests.packages.' + target.replace(target, 'chardet')] = sys.modules[mod]
 # Kinda cool, though, right?

@@ -41,7 +41,8 @@ def run_tests(self):
 packages = ['requests']

 requires = [
-    'chardet>=3.0.2,<5',
+    'charset_normalizer~=2.0.0; python_version >= "3"',
+    'chardet>=3.0.2,<5; python_version < "3"',
     'idna>=2.5,<3',
     'urllib3>=1.21.1,<1.27',
     'certifi>=2017.4.17'
@@ -103,6 +104,7 @@ def run_tests(self):
         'security': ['pyOpenSSL >= 0.14', 'cryptography>=1.3.4'],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
         'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
+        'use_chardet_on_py3': ['chardet>=3.0.2,<5']
     },
     project_urls={
         'Documentation': 'https://requests.readthedocs.io',

@@ -1,7 +1,18 @@
 [tox]
-envlist = py27,py35,py36,py37,py38
+envlist = py{27,35,36,37,38}-{default,use_chardet_on_py3}

 [testenv]
-
+deps = -rrequirements-dev.txt
+extras =
+    security
+    socks
 commands =
-    python setup.py test
+    pytest tests
+
+[testenv:default]
+
+[testenv:use_chardet_on_py3]
+extras =
+    security
+    socks
+    use_chardet_on_py3

This PR looks good, thank you for all the work and patience. I have one concern with the recent release of v2.0.0:

Looks like we're set on the testing infra fixes. I'll leave final merge to [USER_REDACTED], but I think this looks alright.

```suggestion
    'charset_normalizer~=1.4.0,<2; python_version >= "3"',
```

```suggestion
    'charset_normalizer~=1.4.0; python_version >= "3"',
```

This seems somewhat disingenuous. For one thing, it's not the _only_ way to get `lgpl` dependencies. For another, it really exists to just force `chardet` to be installed on py3 so should be named something more like `use_chardet_on_py3`

Maybe precise that if chardet is installed, it will still be used as it could confuse users.

Maybe change the min to `(1, 4, 0)`.

I would add a bit of explanation there.

- What happens if chardet is installed
- Keep both lib name as python 2 is keeping chardet by default

Done

Added.

Changed to 1.4.1 to account for logging fix just added by [USER_REDACTED]

Thanks!

Updated.

[USER_REDACTED] Missed the replacement there `use_chardet_on_py3`

Fixed. Thanks!

```suggestion
        # charset_normalizer >= 1.4.1, < 2.0.0
```

```suggestion
- Instead of `chardet`, use the MIT-licensed `charset_normalizer` for Python3
  to remove license ambiguity for projects bundling requests. If `chardet`
  is already installed on your machine it will be used instead of `charset_normalizer`
  to keep backwards compatibility.

  You can also install `chardet` while installing requests by
  specifying `[use_chardet_on_py3]` extra as follows:

  ```shell
  pip install "requests[use_chardet_on_py3]"
  ```
```

```suggestion
and ``chardet`` is not already installed, ``requests`` uses ``charset-normalizer``
```

Something went wrong with that suggestion .. the ending triple quotes disappeared somehow, try this

```suggestion
    ```shell
    pip install "requests[use_chardet_on_py3]"
    ```
```

[USER_REDACTED] missed a « lgpl » here. It is the last one.

Too much reliance on Ci (which is not running because of workflow awaiting approval :)

This time I actually run it before last fixup locally.

If charset_normalizer v2.0.0 is out do we need to update this to be >2?

Do we need to update this with v2.0.0?

We haven't tested v2. I'd rather get something imperfect out than something perfect. Today's literally the only time I've had to review anything

Makes sense, I/we can handle the upgrade in a follow-up PR.


Add support for brotli decoding

When the `brotli` or `brotlicffi` packages are installed, `urllib3.util.make_headers()` inserts ',br' in the Accept-Encoding header and decodes br from the answers.

@@ -5,6 +5,8 @@ dev
 ---

 -   \[Short description of non-trivial change.\]
+- [NAME_REDACTED] compression, if either the `brotli` or `brotlicffi` package
+  is installed.

 2.25.1 (2020-12-16)
 -------------------

@@ -11,6 +11,9 @@ [NAME_REDACTED]?
 Requests automatically decompresses gzip-encoded responses, and does
 its best to decode response content to unicode when possible.

+When either the `brotli <https://pypi.org/project/Brotli/>`_ or `brotlicffi <https://pypi.org/project/brotlicffi/>`_
+package is installed, requests also decodes Brotli-encoded responses.
+
 You can get direct access to the raw response (and even the socket),
 if needed as well.

@@ -128,6 +128,9 @@ You can also access the response body as bytes, for non-text requests::

 The ``gzip`` and ``deflate`` transfer-encodings are automatically decoded for you.

+The ``br``  transfer-encoding is automatically decoded for you if a Brotli library
+like `brotli <https://pypi.org/project/brotli>`_ or `brotlicffi <https://pypi.org/project/brotli>`_ is installed.
+
 For example, to create an image from binary data returned by a request, you can
 use the following code::

@@ -20,6 +20,7 @@
 import warnings
 import zipfile
 from collections import OrderedDict
+from urllib3.util import make_headers

 from .__version__ import __version__
 from . import certs
@@ -41,6 +42,11 @@

 DEFAULT_PORTS = {'http': 80, 'https': 443}

+# Ensure that ', ' is used to preserve previous delimiter behavior.
+DEFAULT_ACCEPT_ENCODING = ", ".join(
+    re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"])
+)
+

 if sys.platform == 'win32':
     # provide a proxy_bypass version on Windows without DNS lookups
@@ -820,7 +826,7 @@ def default_headers():
     """
     return CaseInsensitiveDict({
         'User-Agent': default_user_agent(),
-        'Accept-Encoding': ', '.join(('gzip', 'deflate')),
+        'Accept-Encoding': DEFAULT_ACCEPT_ENCODING,
         'Accept': '*/*',
         'Connection': 'keep-alive',
     })

This looks mostly good, few comments for you. Also can we add a test for this?

I'm now happy with my changes but I'd like another reviewer to take a quick peek now that I'm the author [NAME_REDACTED] diff. cc [USER_REDACTED] [USER_REDACTED]

Can we link to the individual packages here?

nit: Brotli is a proper name so should be capitalized here and in the changelog

Should we generate this header once and store the result so it doesn't need to be called every time? The value is determined by [NAME_REDACTED] changing often.

I have capitalized Brotli-decompression on both places.

This optimization makes sense.  I kindly ask you to implement it in the way, you think fits best.


quickstart.rst: r.json() can raise JSONDecodeError on Py3

% `python2 -c "import requests ; requests.get('https://github.com').json()"`
% `python3 -c "import requests ; requests.get('https://github.com').json()"`

Python | Simplejson is installed  | Simplejson is NOT installed
--- | --- | ---
Python 2 | simplejson.JSONDecodeError | ValueError
Python 3 | simplejson.JSONDecodeError | json.JSONDecodeError

@@ -150,7 +150,9 @@ There's also a builtin JSON decoder, in case you're dealing with JSON data::

 In case the JSON decoding fails, ``r.json()`` raises an exception. For example, if
 the response gets a 204 ([NAME_REDACTED]), or if the response contains invalid JSON,
-attempting ``r.json()`` raises ``ValueError: No JSON object could be decoded``.
+attempting ``r.json()`` raises ``simplejson.JSONDecodeError`` if simplejson is
+installed or raises ``ValueError: No JSON object could be decoded`` on Python 2 or
+``json.JSONDecodeError`` on Python 3.

 It should be noted that the success of the call to ``r.json()`` does **not**
 indicate the success of the response. Some servers may return a JSON object in a

@@ -877,7 +877,12 @@ def json(self, **kwargs):
         r"""Returns the json-encoded content of a response, if any.

         :param \*\*kwargs: Optional arguments that ``json.loads`` takes.
-        :raises ValueError: If the response body does not contain valid json.
+        :raises simplejson.JSONDecodeError: If the response body does not
+            contain valid json and simplejson is installed.
+        :raises json.JSONDecodeError: If the response body does not contain
+            valid json and simplejson is not installed on Python 3.
+        :raises ValueError: If the response body does not contain valid
+            json and simplejson is not installed on Python 2.
         """

         if not self.encoding and self.content and len(self.content) > 3:

Does `simplejson` raise a `ValueError` on Py2?

Yes.  Good catch.  Thanks.


Avoid zip extract racing condition by [NAME_REDACTED]+write instead extract

Extract also creates the folder hierarchy, however we do not need that,
the file itself being extracted to a temporary folder is good enough.
Instead we read the content of the zip and then write it. The write is
not locked but it's OK to update the same file multiple times given the
update operation will not alter the content of the file. By [NAME_REDACTED] hierarchy (default via extract) we no longer can run into the
problem of two parallel extracts both trying to create the folder
hierarchy without exists ok flag, and one must fail.

Resolves #5223.

Signed-off-by: Bernát Gábor <[EMAIL_REDACTED]>

@@ -256,13 +256,28 @@ def extract_zipped_paths(path):

     # we have a valid zip archive and a valid member of that archive
     tmp = tempfile.gettempdir()
-    extracted_path = os.path.join(tmp, *member.split('/'))
+    extracted_path = os.path.join(tmp, member.split('/')[-1])
     if not os.path.exists(extracted_path):
-        extracted_path = zip_file.extract(member, path=tmp)
-
+        # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition
+        with atomic_open(extracted_path) as file_handler:
+            file_handler.write(zip_file.read(member))
     return extracted_path

+[USER_REDACTED].contextmanager
+def atomic_open(filename):
+    """Write a file to the disk in an atomic fashion"""
+    replacer = os.rename if sys.version_info[0] == 2 else os.replace
+    tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename))
+    try:
+        with os.fdopen(tmp_descriptor, 'wb') as tmp_handler:
+            yield tmp_handler
+        replacer(tmp_name, filename)
+    except BaseException:
+        os.remove(tmp_name)
+        raise
+
+
 def from_key_val_list(value):
     """Take an object and test to see if it can be represented as a
     dictionary. Unless it can not be represented as such, return an

Just to handle a further race condition that can happen between opening and writing the file.


5677: Rebuild proxies on Session#send

- Adds some test cases to highlight the issue, some of those seem excessive, I would like suggestions on how to combine them.
- Rebuilds the proxy configuration on `send`, making it behave like `requests.Session#request` that already did that.

@@ -633,7 +633,7 @@ def send(self, request, **kwargs):
         kwargs.setdefault('stream', self.stream)
         kwargs.setdefault('verify', self.verify)
         kwargs.setdefault('cert', self.cert)
-        kwargs.setdefault('proxies', self.proxies)
+        kwargs.setdefault('proxies', self.rebuild_proxies(request, self.proxies))

         # It's possible that users might accidentally send a Request object.
         # Guard against that specific failure case.

@@ -40,6 +40,9 @@
 # listening on that port)
 TARPIT = 'http://10.255.255.1'

+# This is to avoid waiting the timeout of using TARPIT
+INVALID_PROXY='http://localhost:1'
+
 try:
     from ssl import SSLContext
     del SSLContext
@@ -551,6 +554,42 @@ def test_proxy_error_on_bad_url(self, httpbin, httpbin_secure):
         with pytest.raises(InvalidProxyURL):
             requests.get(httpbin(), proxies={'http': 'http:///example.com:8080'})

+    def test_respect_proxy_env_on_send_self_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_send_session_prepared_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                request = requests.Request('GET', httpbin())
+                prepared = session.prepare_request(request)
+                session.send(prepared)
+
+    def test_respect_proxy_env_on_send_with_redirects(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                url = httpbin('redirect/1')
+                print(url)
+                request = requests.Request('GET', url)
+                session.send(request.prepare())
+
+    def test_respect_proxy_env_on_get(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.get(httpbin())
+
+    def test_respect_proxy_env_on_request(self, httpbin):
+        with override_environ(http_proxy=INVALID_PROXY):
+            with pytest.raises(ProxyError):
+                session = requests.Session()
+                session.request(method='GET', url=httpbin())
+
     def test_basicauth_with_netrc(self, httpbin):
         auth = ('user', 'pass')
         wrong_auth = ('wronguser', 'wrongpass')

I'm 99% certain this isn't actually correct. This does work regardless of whether it's used or not. We should do:

```py
if kwargs['proxies']:
    kwargs['proxies'] = self.rebuild_proxies(request, kwargs['proxies'])
```

After this line though

What do you mean by "this"?

The problem or the proposed solution?

For the problem, I've added a fell test cases, which when I remove this line change those were not passing.

As for the solution.

Should the `proxies` parameter be overwritten by [NAME_REDACTED]?

If we `rebuild_proxies` there, it will load the environment variables, instead of using what is being passed on the arguments, my solution intentionally considers the arguments as the highest priority.

[USER_REDACTED]

Does my answer help you understand the problem in any way?

I'm facing this problem in production and the tests do show the inconsistent use of proxy environment variables by `requests.session`.

I don't recall 😆 but your logic makes sense.