Add something to the docs about hooks on Session()

I had to work this out by [NAME_REDACTED] for `Session` – better to put it front-and-centre in the docs.

@@ -436,7 +436,7 @@ You can assign a hook function on a per-request basis by passing a
 ``{hook_name: callback_function}`` dictionary to the ``hooks`` request
 parameter::

-    hooks=dict(response=print_url)
+    hooks={'response': print_url}

 That ``callback_function`` will receive a chunk of data as its first
 argument.
@@ -452,12 +452,36 @@ If the callback function returns a value, it is assumed that it is to
 replace the data that was passed in. If the function doesn't return
 anything, nothing else is effected.

+::
+
+    def record_hook(r, *args, **kwargs):
+        r.hook_called = True
+        return r
+
 Let's print some request method arguments at runtime::

-    >>> requests.get('http://httpbin.org', hooks=dict(response=print_url))
+    >>> requests.get('http://httpbin.org', hooks={'response': print_url})
+    http://httpbin.org
+    <Response [200]>
+
+You can add multiple hooks to a single request.  Let's call two hooks at once::
+
+    >>> r = requests.get('http://httpbin.org', hooks={'response': [print_url, record_hook]})
+    >>> r.hook_called
+    True
+
+You can also add hooks to a ``Session`` instance.  Any hooks you add will then
+be called on every request made to the session.  For example::
+
+   >>> s = requests.Session()
+   >>> s.hooks['response'].append(print_url)
+   >>> s.get('http://httpbin.org')
     http://httpbin.org
     <Response [200]>

+A ``Session`` can have multiple hooks, which will be called in the order
+they are added.
+
 .. _custom-auth:

 [NAME_REDACTED]

We need to be more clear about what a session's hooks dictionary looks like. This should also be clarified above by [NAME_REDACTED] also provide *multiple* hooks. For example

```py
requests.get('http://httpbinorg', hooks={'response': [hook0, hook1]})
```

Maybe we should also stop using `dict()` since that's so incredibly slow.

I'm happy with this.

I am too, thanks [USER_REDACTED]!

This should be:

```
s.hooks['response'].append(print_url)
```

As we keep a list of hooks


Warn about possible slowness on older versions of pyOpenSSL

Give a warning when `cryptography < 1.3.4` and `pyOpenSSL < 16.0.0` as discussed in #4309

@@ -178,3 +178,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])

@@ -8,6 +8,8 @@ dev

 **Improvements**

+- Warn user about possible slowdown when using cryptography version < 1.3.4
+
 **Bugfixes**

 - Parsing empty ``Link`` headers with ``parse_header_links()`` no longer return one bogus entry

@@ -71,6 +71,17 @@ def check_compatibility(urllib3_version, chardet_version):
     assert patch >= 2

+def _check_cryptography(cryptography_version):
+    # cryptography < 1.3.4
+    try:
+        cryptography_version = list(map(int, cryptography_version.split('.')))
+    except ValueError:
+        return
+
+    if cryptography_version < [1, 3, 4]:
+        warning = 'Old version of cryptography ({0}) may cause slowdown.'.format(cryptography_version)
+        warnings.warn(warning, RequestsDependencyWarning)
+
 # Check imported dependencies for compatibility.
 try:
     check_compatibility(urllib3.__version__, chardet.__version__)
@@ -83,6 +94,10 @@ def check_compatibility(urllib3_version, chardet_version):
 try:
     from urllib3.contrib import pyopenssl
     pyopenssl.inject_into_urllib3()
+
+    # Check cryptography version
+    from cryptography import __version__ as cryptography_version
+    _check_cryptography(cryptography_version)
 except ImportError:
     pass

Thanks for getting this updated [USER_REDACTED]! This is moving in the right direction. I've left a few more notes inline that we may want to address.

This is looking much better! I left one last minor comment. If you could also squash your commits into one, that would be really helpful. Thanks for taking the time to work through this, [USER_REDACTED]!

This looks good to me, thanks [USER_REDACTED]!

Cool, looks good. Mind updating the changelog? Then we should be good to go.

Cool, thanks a bunch!

This should be a private function. We really shouldn't be exposing this as part of our API.

I feel like there's a bifurcation. Below we check that `'OpenSSL' in sys.modules` why not do this check there too? Further, is there another `OpenSSL` module that we might import that isn't `pyOpenSSL`? What is this check protecting against?

This will always be false since no relesaed version of cryptography is `16.0.0` or higher. I think this was meant to be `pyopenssl_version`

Is the issue for old versions of `pyOpenSSL` or `cryptography`.

To clarify, I believe the comment says we want to verify `pyOpenSSL` is installed, and if it is check that `cryptography` is at least 1.3.4 which is our minimum requirement. We haven't done any testing to see how far back you need to go with `cryptography` for the slowdown to occur, but 1.3.4 should be sufficient.

I also don't think we actually need the check to be this complicated. We're already checking for `pyOpenSSL` with our `inject_pyopenssl_into_urllib3` above. A simple helper function that checks cryptography's version meets or exceeds our threshold (supposedly 1.3.4), and otherwise displays a warning, should be sufficient.

This should really be _check_cryptography because that's what we're concerned about.

So we don't really care about the version of pyOpenSSL, the only reason it's significant is 16.0 is where a version constraint for cryptography > 1.0 is put in place. Let's remove this check and just launch the warning if cryptography doesn't meet our requirement. That's the source of the slowdown.

No need to pass both of these. We don't actually care about the pyOpenSSL version as I stated above, just that it's being imported for urllib3.

We can do the check with something like:
```
from cryptography import __version__ as cryptography_version
```

This allows us to have to only worry about the import error.

Let's move this into the `_check_cryptography` function.

One petty nit and we're set here I think. Mind lowecasing the cryptography package name here, below in the warning message, and in the try/except comment?

Sorry before we squash this import is also not being used anymore.

And cryptography is still capitalized here.

This check will break if for any reason cryptography has a non-numerical version string. As in other places, we should aim to tolerate that by [NAME_REDACTED] version numbers will be fine.


Remove redundant Python 2.6 stuff

Same as https://github.com/requests/requests/pull/4326 but for the proposed/3.0.0 branch.

The good news is nearly everything was already in this branch from https://github.com/requests/requests/pull/4118, except for tidying up some OrderedDict compat stuff.

Also, pytest>=3.0.0 is required for the Python 3.3 build, otherwise:
```
py.test -n 8 --boxed --junitxml=report.xml
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.3.6/bin/py.test", line 11, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/config.py", line 39, in main
    config = _prepareconfig(args, plugins)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/config.py", line 118, in _prepareconfig
    pluginmanager=pluginmanager, args=args)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
    wrap_controller.send(call_outcome)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/helpconfig.py", line 28, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
    self.result = func()
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/config.py", line 861, in pytest_cmdline_parse
    self.parse(args)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/config.py", line 966, in parse
    self._preparse(args, addopts=addopts)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/config.py", line 927, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/_pytest/vendored_packages/pluggy.py", line 501, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/pkg_resources/__init__.py", line 2404, in load
    self.require(*args, **kwargs)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/pkg_resources/__init__.py", line 2427, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages/pkg_resources/__init__.py", line 872, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pytest 2.9.2 (/home/travis/virtualenv/python3.3.6/lib/python3.3/site-packages), Requirement.parse('pytest>=3.0.0'))
```
https://travis-ci.org/hugovk/requests/jobs/284048190

The other CI failures are related to cookies and also occur in latest master and fresh builds of proposed/3.0.0.

@@ -3,6 +3,9 @@

 - Support for Python 2.6 has been dropped.

+  - The ``OrderedDict`` import no longer exists in compat.py because it is part
+    of ``collections`` in Python 2.7 and newer.
+
 - Simplified logic for determining Content-Length and Transfer-Encoding.
   Requests will now avoid setting both headers on the same request, and
   raise an exception if this is done manually by a user.

@@ -182,3 +182,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
+- Hugo van Kemenade ([CONTRIBUTOR_REDACTED])

@@ -39,8 +39,6 @@
     from Cookie import Morsel
     from StringIO import StringIO

-    from urllib3.packages.ordered_dict import OrderedDict
-
     builtin_str = str
     bytes = str
     str = unicode
@@ -54,7 +52,6 @@
     from http import cookiejar as cookielib
     from http.cookies import Morsel
     from io import StringIO
-    from collections import OrderedDict

     builtin_str = str
     str = str

@@ -10,11 +10,11 @@
 import os
 import platform
 import time
-from collections import Mapping
+from collections import Mapping, OrderedDict
 from datetime import timedelta

 from .auth import _basic_auth_str
-from .compat import cookielib, OrderedDict, urljoin, urlparse, is_py3, str
+from .compat import cookielib, urljoin, urlparse, is_py3, str
 from .cookies import (
     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar,
     merge_cookies, _copy_cookie_jar)

@@ -9,8 +9,6 @@

 import collections

-from .compat import OrderedDict
-

 class CaseInsensitiveDict(collections.MutableMapping):
     """A case-insensitive ``dict``-like object.
@@ -40,7 +38,7 @@ class CaseInsensitiveDict(collections.MutableMapping):
     """

     def __init__(self, data=None, **kwargs):
-        self._store = OrderedDict()
+        self._store = collections.OrderedDict()
         if data is None:
             data = {}
         self.update(data, **kwargs)

@@ -26,7 +26,7 @@
 from ._internal_utils import to_native_string
 from .compat import parse_http_list as _parse_list_header
 from .compat import (
-    quote, urlparse, bytes, str, OrderedDict, unquote, getproxies,
+    quote, urlparse, bytes, str, unquote, getproxies,
     proxy_bypass, urlunparse, basestring, integer_types, is_py2, is_py3,
     proxy_bypass_environment, getproxies_environment)
 from .cookies import cookiejar_from_dict
@@ -238,7 +238,7 @@ def from_key_val_list(value):
     if isinstance(value, (str, bytes, bool, int)):
         raise ValueError('cannot encode objects that are not 2-tuples')

-    return OrderedDict(value)
+    return collections.OrderedDict(value)

 def to_key_val_list(value):

@@ -18,7 +18,7 @@
 from requests.auth import HTTPDigestAuth, _basic_auth_str
 from requests.compat import (
     Morsel, cookielib, getproxies, str, urlparse,
-    builtin_str, OrderedDict)
+    builtin_str)
 from requests.cookies import (
     cookiejar_from_dict, morsel_to_cookie)
 from requests.exceptions import (
@@ -144,7 +144,8 @@ def test_params_are_added_before_fragment(self, url, expected):
         assert request.url == expected

     def test_params_original_order_is_preserved_by_default(self):
-        param_ordered_dict = OrderedDict((('z', 1), ('a', 1), ('k', 1), ('d', 1)))
+        param_ordered_dict = collections.OrderedDict(
+            (('z', 1), ('a', 1), ('k', 1), ('d', 1)))
         session = requests.Session()
         request = requests.Request('GET', 'http://example.com/', params=param_ordered_dict)
         prep = session.prepare_request(request)
@@ -539,11 +540,11 @@ def test_headers_on_session_with_None_are_not_sent(self, httpbin):
     def test_headers_preserve_order(self, httpbin):
         """Preserve order when headers provided as OrderedDict."""
         ses = requests.Session()
-        ses.headers = OrderedDict()
+        ses.headers = collections.OrderedDict()
         ses.headers['Accept-Encoding'] = 'identity'
         ses.headers['First'] = '1'
         ses.headers['Second'] = '2'
-        headers = OrderedDict([('Third', '3'), ('Fourth', '4')])
+        headers = collections.OrderedDict([('Third', '3'), ('Fourth', '4')])
         headers['Fifth'] = '5'
         headers['Second'] = '222'
         req = requests.Request('GET', httpbin('get'), headers=headers)
@@ -2103,7 +2104,7 @@ def test_chunked_upload_with_manually_set_content_length_header_raises_error(sel
         """Ensure that if a user manually sets a content length header, when
         the data is chunked, that an InvalidHeader error is raised.
         """
-        data = (i for i in [b'a', b'b', b'c'])
+        data = (i for i in [b'a', b'b', b'c'])
         url = httpbin('post')
         with pytest.raises(InvalidHeader):
             r = requests.post(url, data=data, headers={'Content-Length': 'foo'})

Alright, I think this is sensible since we were only using the special `OrdredDict` import for 2.6 and already import `collections` in most places.

A few final notes, you'll need to rebase this branch onto the current `proposed/3.0.0` branch and wait for a patched version of `pytest-httpbin` to be released so we can get green tests. We could also use a CHANGELOG entry stating that the `OrderedDict` import no longer exists in `compat.py`.

I'd also encourage you to add yourself to the AUTHORS.rst for this branch if you'd like.

If anyone else from [USER_REDACTED]/core want to give this a glace, I think this is about ready to go.

Thanks again for putting this together, [USER_REDACTED]!

Looks good. Merge when you feel satisfied [USER_REDACTED].

Found one more thing in the final pass.

We've removed the requirement for python 3.3 in the 3.0.0 branch, so lets save this change for a separate PR if it's really needed. (Although, I think the requirements.txt was removed on master and that should now be reflected on this branch)

I've removed this commit and rebased onto current proposed/3.0.0 and update AUTHORS.rst on this branch.

It looks like this is a cli example. We don't want the "collection" lead here. The interpreter is just showing the returned objects repr. Same with the line below this.


Adding header name to exception

Adds the name of the header to the invalid header exception raised on TypeError.

Fixes #4239

@@ -7,6 +7,7 @@ dev
 +++

 **Improvements**
+- Error messages for invalid headers now include the header name for easier debugging

 **Bugfixes**

@@ -868,8 +868,8 @@ def check_header_validity(header):
         if not pat.match(value):
             raise InvalidHeader("Invalid return character or leading space in header: %s" % name)
     except TypeError:
-        raise InvalidHeader("Header value %s must be of type str or bytes, "
-                            "not %s" % (value, type(value)))
+        raise InvalidHeader("Value for header {%s: %s} must be of type str or "
+                            "bytes, not %s" % (name, value, type(value)))

 def urldefragauth(url):

@@ -1401,14 +1401,17 @@ def test_header_value_not_str(self, httpbin):
         headers_list = {'baz': ['foo', 'bar']}

         # Test for int
-        with pytest.raises(InvalidHeader):
+        with pytest.raises(InvalidHeader) as excinfo:
             r = requests.get(httpbin('get'), headers=headers_int)
+        assert 'foo' in str(excinfo.value)
         # Test for dict
-        with pytest.raises(InvalidHeader):
+        with pytest.raises(InvalidHeader) as excinfo:
             r = requests.get(httpbin('get'), headers=headers_dict)
+        assert 'bar' in str(excinfo.value)
         # Test for list
-        with pytest.raises(InvalidHeader):
+        with pytest.raises(InvalidHeader) as excinfo:
             r = requests.get(httpbin('get'), headers=headers_list)
+        assert 'baz' in str(excinfo.value)

     def test_header_no_return_chars(self, httpbin):
         """Ensure that a header containing return character sequences raise an

I left a suggestion on possible wording changes but this looks good to me. Thanks [USER_REDACTED]!

One minor tweak and I think this will be set 😊

Cool, looks good! Mind adding a note to `HISTORY.rst`? When we have that I'll merge. 😁

This may read a little clear if the name and/or value are wrapped in quotes.

Perhaps even adjusting the wording to something like "Header {name}: {value} must have a value of string or bytes, not {type}".

Makes sense, I'll add that in really quick. Thanks for the feedback!

Mind adding a space after the colon?

Mind adding a space after the colon?

On it!


Fix help ssl check

This will resolve the test failures discovered in #4182. We were retrieving the `OPENSSL_VERSION_NUMBER` from the system `ssl` module in all cases before now. This attribute doesn't exist in Python 2.6 causing new tests for the `info()` method to fail. This patch will return an empty value for Python 2.6 since we don't seem to have a way to reliably check the openssl version.

[USER_REDACTED] [USER_REDACTED], I've pushed this branch up to the requests repo directly, so feel free to chop this up or do anything you feel is necessary to get #4182 moving. I don't know that the provided test is especially useful, but it does show that the fix here works.

@@ -85,12 +85,16 @@ def info():
         'version': getattr(cryptography, '__version__', ''),
     }

+    # OPENSSL_VERSION_NUMBER doesn't exist in the Python 2.6 ssl module.
+    system_ssl = getattr(ssl, 'OPENSSL_VERSION_NUMBER', None)
+    system_ssl_info = {
+        'version': '%x' % system_ssl if system_ssl is not None else ''
+    }
+
     return {
         'platform': platform_info,
         'implementation': implementation_info,
-        'system_ssl': {
-            'version': '%x' % ssl.OPENSSL_VERSION_NUMBER,
-        },
+        'system_ssl': system_ssl_info,
         'using_pyopenssl': pyopenssl is not None,
         'pyOpenSSL': pyopenssl_info,
         'urllib3': urllib3_info,

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8
+
+import sys
+
+import pytest
+
+from requests.help import info
+
+
+[USER_REDACTED].mark.skipif(sys.version_info[:2] != (2,6), reason="Only run on Python 2.6")
+def test_system_ssl_py26():
+    """OPENSSL_VERSION_NUMBER isn't provided in Python 2.6, verify we don't
+    blow up in this case.
+    """
+    assert info()['system_ssl'] == {'version': ''}
+
+
+[USER_REDACTED].mark.skipif(sys.version_info < (2,7), reason="Only run on Python 2.7+")
+def test_system_ssl():
+    """Verify we're actually setting system_ssl when it should be available."""
+    assert info()['system_ssl']['version'] != ''

This looks sensible, thanks [USER_REDACTED]!

This test file would also pass if there was never anything interesting in `system_ssl`.

How about a second test that asserts that on Pythons >2.6, the `version` key is non-empty?

Yep, that seems reasonable if we're going to be testing for this specifically. I've added that into the test commit, if we decide to merge that part of the PR.


Add idna version info to requests.help

Resolves #4180.

@@ -8,6 +8,8 @@ dev

 **Improvements**

+- Running ``$ python -m requests.help`` now includes the installed version of idna.
+
 **Bugfixes**

 2.18.2 (2017-07-25)

@@ -6,6 +6,7 @@
 import sys
 import ssl

+import idna
 import urllib3
 import chardet

@@ -84,6 +85,9 @@ def info():
     cryptography_info = {
         'version': getattr(cryptography, '__version__', ''),
     }
+    idna_info = {
+        'version': getattr(idna, '__version__', ''),
+    }

     # OPENSSL_VERSION_NUMBER doesn't exist in the Python 2.6 ssl module.
     system_ssl = getattr(ssl, 'OPENSSL_VERSION_NUMBER', None)
@@ -100,6 +104,7 @@ def info():
         'urllib3': urllib3_info,
         'chardet': chardet_info,
         'cryptography': cryptography_info,
+        'idna': idna_info,
         'requests': {
             'version': requests_version,
         },

@@ -19,3 +19,22 @@ def test_system_ssl_py26():
 def test_system_ssl():
     """Verify we're actually setting system_ssl when it should be available."""
     assert info()['system_ssl']['version'] != ''
+
+
+class VersionedPackage(object):
+    def __init__(self, version):
+        self.__version__ = version
+
+
+def test_idna_without_version_attribute(mocker):
+    """Older versions of IDNA don't provide a __version__ attribute, verify
+    that if we have such a package, we don't blow up.
+    """
+    mocker.patch('requests.help.idna', new=None)
+    assert info()['idna'] == {'version': ''}
+
+
+def test_idna_with_version_attribute(mocker):
+    """Verify we're actually setting idna version when it should be available."""
+    mocker.patch('requests.help.idna', new=VersionedPackage('2.6'))
+    assert info()['idna'] == {'version': '2.6'}

Alright, this looks good to me. Thanks [USER_REDACTED]! ✨

Sorry, one final note and I think we're set here. Could we get the second underscore for the closing dunder on version (`__version_` -> `__version__`). Once that's done I think I'm happy with this.

Fixed.


Pass pool_kwargs rather than updating connection_pool_kw

This addresses an issue where making HTTPS through proxies used the
default urllib3 connection pool settings.

fixes #3633

@@ -10,10 +10,6 @@

 import os.path
 import socket
-try:
-    from threading import RLock
-except ImportError:  # threading is an optional module and may not be present.
-    from dummy_threading import RLock

 from urllib3.poolmanager import PoolManager, proxy_from_url
 from urllib3.response import HTTPResponse
@@ -53,6 +49,67 @@ def SOCKSProxyManager(*args, **kwargs):
 DEFAULT_POOL_TIMEOUT = None

+def _pool_kwargs(verify, cert):
+    """Create a dictionary of keyword arguments to pass to a
+    :class:`PoolManager <urllib3.poolmanager.PoolManager>` with the
+    necessary SSL configuration.
+
+    :param verify: Whether we should actually verify the certificate;
+                   optionally a path to a CA certificate bundle or
+                   directory of CA certificates.
+    :param cert: The path to the client certificate and key, if any.
+                 This can either be the path to the certificate and
+                 key concatenated in a single file, or as a tuple of
+                 (cert_file, key_file).
+    """
+    pool_kwargs = {}
+    if verify:
+
+        cert_loc = None
+
+        # Allow self-specified cert location.
+        if verify is not True:
+            cert_loc = verify
+
+        if not cert_loc:
+            cert_loc = DEFAULT_CA_BUNDLE_PATH
+
+        if not cert_loc or not os.path.exists(cert_loc):
+            raise IOError("Could not find a suitable TLS CA certificate bundle, "
+                          "invalid path: {0}".format(cert_loc))
+
+        pool_kwargs['cert_reqs'] = 'CERT_REQUIRED'
+
+        if not os.path.isdir(cert_loc):
+            pool_kwargs['ca_certs'] = cert_loc
+            pool_kwargs['ca_cert_dir'] = None
+        else:
+            pool_kwargs['ca_cert_dir'] = cert_loc
+            pool_kwargs['ca_certs'] = None
+    else:
+        pool_kwargs['cert_reqs'] = 'CERT_NONE'
+        pool_kwargs['ca_certs'] = None
+        pool_kwargs['ca_cert_dir'] = None
+
+    if cert:
+        if not isinstance(cert, basestring):
+            pool_kwargs['cert_file'] = cert[0]
+            pool_kwargs['key_file'] = cert[1]
+        else:
+            pool_kwargs['cert_file'] = cert
+            pool_kwargs['key_file'] = None
+
+        cert_file = pool_kwargs['cert_file']
+        key_file = pool_kwargs['key_file']
+        if cert_file and not os.path.exists(cert_file):
+            raise IOError("Could not find the TLS certificate file, "
+                          "invalid path: {0}".format(cert_file))
+        if key_file and not os.path.exists(key_file):
+            raise IOError("Could not find the TLS key file, "
+                          "invalid path: {0}".format(key_file))
+    return pool_kwargs
+
+
 class BaseAdapter(object):
     """The Base Transport Adapter"""

@@ -126,7 +183,6 @@ def __init__(self, pool_connections=DEFAULT_POOLSIZE,
         self._pool_connections = pool_connections
         self._pool_maxsize = pool_maxsize
         self._pool_block = pool_block
-        self._pool_kw_lock = RLock()

         self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

@@ -139,7 +195,6 @@ def __setstate__(self, state):
         # self.poolmanager uses a lambda function, which isn't pickleable.
         self.proxy_manager = {}
         self.config = {}
-        self._pool_kw_lock = RLock()

         for attr, value in state.items():
             setattr(self, attr, value)
@@ -204,65 +259,6 @@ def proxy_manager_for(self, proxy, **proxy_kwargs):

         return manager

-    def _update_poolmanager_ssl_kw(self, verify, cert):
-        """Update the :class:`PoolManager <urllib3.poolmanager.PoolManager>`
-        connection_pool_kw with the necessary SSL configuration. This method
-        should not be called from user code, and is only exposed for use when
-        subclassing the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
-
-        :param verify: Whether we should actually verify the certificate;
-                       optionally a path to a CA certificate bundle or
-                       directory of CA certificates.
-        :param cert: The path to the client certificate and key, if any.
-                     This can either be the path to the certificate and
-                     key concatenated in a single file, or as a tuple of
-                     (cert_file, key_file).
-        """
-        if verify:
-
-            cert_loc = None
-
-            # Allow self-specified cert location.
-            if verify is not True:
-                cert_loc = verify
-
-            if not cert_loc:
-                cert_loc = DEFAULT_CA_BUNDLE_PATH
-
-            if not cert_loc or not os.path.exists(cert_loc):
-                raise IOError("Could not find a suitable TLS CA certificate bundle, "
-                              "invalid path: {0}".format(cert_loc))
-
-            self.poolmanager.connection_pool_kw['cert_reqs'] = 'CERT_REQUIRED'
-
-            if not os.path.isdir(cert_loc):
-                self.poolmanager.connection_pool_kw['ca_certs'] = cert_loc
-                self.poolmanager.connection_pool_kw['ca_cert_dir'] = None
-            else:
-                self.poolmanager.connection_pool_kw['ca_cert_dir'] = cert_loc
-                self.poolmanager.connection_pool_kw['ca_certs'] = None
-        else:
-            self.poolmanager.connection_pool_kw['cert_reqs'] = 'CERT_NONE'
-            self.poolmanager.connection_pool_kw['ca_certs'] = None
-            self.poolmanager.connection_pool_kw['ca_cert_dir'] = None
-
-        if cert:
-            if not isinstance(cert, basestring):
-                self.poolmanager.connection_pool_kw['cert_file'] = cert[0]
-                self.poolmanager.connection_pool_kw['key_file'] = cert[1]
-            else:
-                self.poolmanager.connection_pool_kw['cert_file'] = cert
-                self.poolmanager.connection_pool_kw['key_file'] = None
-
-            cert_file = self.poolmanager.connection_pool_kw['cert_file']
-            key_file = self.poolmanager.connection_pool_kw['key_file']
-            if cert_file and not os.path.exists(cert_file):
-                raise IOError("Could not find the TLS certificate file, "
-                              "invalid path: {0}".format(cert_file))
-            if key_file and not os.path.exists(key_file):
-                raise IOError("Could not find the TLS key file, "
-                              "invalid path: {0}".format(key_file))
-
     def build_response(self, req, resp):
         """Builds a :class:`Response <requests.Response>` object from a urllib3
         response. This should not be called from user code, and is only exposed
@@ -309,21 +305,18 @@ def get_connection(self, url, proxies=None, verify=None, cert=None):
         :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
         :rtype: urllib3.ConnectionPool
         """
-        with self._pool_kw_lock:
-            if url.lower().startswith('https'):
-                self._update_poolmanager_ssl_kw(verify, cert)
-
-            proxy = select_proxy(url, proxies)
+        pool_kwargs = _pool_kwargs(verify, cert)
+        proxy = select_proxy(url, proxies)

-            if proxy:
-                proxy = prepend_scheme_if_needed(proxy, 'http')
-                proxy_manager = self.proxy_manager_for(proxy)
-                conn = proxy_manager.connection_from_url(url)
-            else:
-                # Only scheme should be lower case
-                parsed = urlparse(url)
-                url = parsed.geturl()
-                conn = self.poolmanager.connection_from_url(url)
+        if proxy:
+            proxy = prepend_scheme_if_needed(proxy, 'http')
+            proxy_manager = self.proxy_manager_for(proxy)
+            conn = proxy_manager.connection_from_url(url, pool_kwargs=pool_kwargs)
+        else:
+            # Only scheme should be lower case
+            parsed = urlparse(url)
+            url = parsed.geturl()
+            conn = self.poolmanager.connection_from_url(url, pool_kwargs=pool_kwargs)

         return conn

@@ -31,6 +31,7 @@
 from requests.sessions import SessionRedirectMixin
 from requests.models import urlencode
 from requests.hooks import default_hooks
+from requests.utils import DEFAULT_CA_BUNDLE_PATH

 from .compat import StringIO, u
 from .utils import override_environ
@@ -2765,3 +2766,200 @@ def test_parameters_for_nonstandard_schemes(self, input, params, expected):
         r = requests.Request('GET', url=input, params=params)
         p = r.prepare()
         assert p.url == expected
+
+
+class TestGetConnection(object):
+    """
+    Tests for the :meth:`requests.adapters.HTTPAdapter.get_connection` that assert
+    the connections are correctly configured.
+    """
+    [USER_REDACTED].mark.parametrize(
+        'proxies, verify, cert, expected',
+        (
+            (
+                {},
+                True,
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                False,
+                None,
+                {
+                    'cert_reqs': 'CERT_NONE',
+                    'ca_certs': None,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                __file__,
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': __file__,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                os.path.dirname(__file__),
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': None,
+                    'ca_cert_dir': os.path.dirname(__file__),
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                True,
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                True,
+                __file__,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': __file__,
+                    'key_file': None,
+                },
+            ),
+            (
+                {},
+                True,
+                (__file__, __file__),
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': __file__,
+                    'key_file': __file__,
+                },
+            ),
+            (
+                {},
+                True,
+                (__file__, __file__),
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': __file__,
+                    'key_file': __file__,
+                },
+            ),
+            (
+                {'http': 'http://proxy.example.com', 'https': 'http://proxy.example.com'},
+                True,
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {'http': 'http://proxy.example.com', 'https': 'http://proxy.example.com'},
+                os.path.dirname(__file__),
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': None,
+                    'ca_cert_dir': os.path.dirname(__file__),
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {'http': 'http://proxy.example.com', 'https': 'http://proxy.example.com'},
+                __file__,
+                None,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': __file__,
+                    'ca_cert_dir': None,
+                    'cert_file': None,
+                    'key_file': None,
+                },
+            ),
+            (
+                {'http': 'http://proxy.example.com', 'https': 'http://proxy.example.com'},
+                True,
+                __file__,
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': __file__,
+                    'key_file': None,
+                },
+            ),
+            (
+                {'http': 'http://proxy.example.com', 'https': 'http://proxy.example.com'},
+                True,
+                (__file__, __file__),
+                {
+                    'cert_reqs': 'CERT_REQUIRED',
+                    'ca_certs': DEFAULT_CA_BUNDLE_PATH,
+                    'ca_cert_dir': None,
+                    'cert_file': __file__,
+                    'key_file': __file__,
+                },
+            ),
+        )
+    )
+    def test_get_https_connection(self, proxies, verify, cert, expected):
+        """Assert connections are configured correctly."""
+        adapter = requests.adapters.HTTPAdapter()
+        connection = adapter.get_connection(
+            'https://example.com', proxies=proxies, verify=verify, cert=cert)
+        actual_config = {}
+        for key, value in connection.__dict__.items():
+            if key in expected:
+                actual_config[key] = value
+        assert actual_config == expected
+
+    [USER_REDACTED].mark.parametrize(
+        'verify, cert',
+        (
+            ('a/path/that/does/not/exist', None),
+            (True, 'a/path/that/does/not/exist'),
+            (True, (__file__, 'a/path/that/does/not/exist')),
+            (True, ('a/path/that/does/not/exist', __file__)),
+        )
+    )
+    def test_cert_files_missing(self, verify, cert):
+        """
+        Assert an IOError is raised when one of the certificate files or
+        directories can't be found.
+        """
+        adapter = requests.adapters.HTTPAdapter()
+        with pytest.raises(IOError) as excinfo:
+            adapter.get_connection('https://example.com', verify=verify, cert=cert)
+        excinfo.match('invalid path: a/path/that/does/not/exist')

Hrm, this might be a breaking change. Not sure though: our docstring implies we wanted users to override that method, but maybe the underscore in front means we didn't mean it?

LGTM. Thanks!

Don't have static methods: they're just not very useful. Either we should decide users can't override this, in which case this should be a function, or we should decide they can and it becomes an instance method.

I can't think of a particularly good reason for someone to want to override this, so a function sounds fine me. If someone really objects to the behaviour for some unforeseen reason, they can override ``get_connection`` and build the pool kwargs their own way.

Yeah, that works for me. Function is the way to go then!


Convert version compatibility checks to warnings

This patch will convert the dependency exceptions into warnings, preventing the forced halt while importing Requests. I condensed the message into a single response, but we can expand that out if needed. It just seemed like another place for updates to be forgotten.

I'd also like to test this function, but due to the requirements changing with each release, it's hard to write something that won't need to be updated in parallel. We could look at parsing the requirements out of `requires` but that seems less than ideal. If I'm being honest, I'm not crazy about expanding the code around this any more than we absolutely have to.

@@ -40,34 +40,44 @@
 :license: Apache 2.0, see LICENSE for more details.
 """

-# Check urllib3 for compatibility.
 import urllib3
-urllib3_version = urllib3.__version__.split('.')
-# Sometimes, urllib3 only reports its version as 16.1.
-if len(urllib3_version) == 2:
-    urllib3_version.append('0')
-major, minor, patch = urllib3_version
-major, minor, patch = int(major), int(minor), int(patch)
-# urllib3 >= 1.21.1, < 1.22
-try:
+import chardet
+import warnings
+from .exceptions import RequestsDependencyWarning
+
+
+def check_compatibility(urllib3_version, chardet_version):
+    urllib3_version = urllib3_version.split('.')
+    assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.
+
+    # Sometimes, urllib3 only reports its version as 16.1.
+    if len(urllib3_version) == 2:
+        urllib3_version.append('0')
+
+    # Check urllib3 for compatibility.
+    major, minor, patch = urllib3_version  # noqa: F811
+    major, minor, patch = int(major), int(minor), int(patch)
+    # urllib3 >= 1.21.1, < 1.22
     assert major == 1
     assert minor >= 21
     assert minor <= 22
-except AssertionError:
-    raise RuntimeError('Requests dependency \'urllib3\' must be version >= 1.21.1, < 1.22!')
-

-# Check chardet for compatibility.
-import chardet
-major, minor, patch = chardet.__version__.split('.')[:3]
-major, minor, patch = int(major), int(minor), int(patch)
-# chardet >= 3.0.2, < 3.1.0
-try:
+    # Check chardet for compatibility.
+    major, minor, patch = chardet_version.split('.')[:3]
+    major, minor, patch = int(major), int(minor), int(patch)
+    # chardet >= 3.0.2, < 3.1.0
     assert major == 3
     assert minor < 1
     assert patch >= 2
-except AssertionError:
-    raise RuntimeError('Requests dependency \'chardet\' must be version >= 3.0.2, < 3.1.0!')
+
+
+# Check imported dependencies for compatibility.
+try:
+    check_compatibility(urllib3.__version__, chardet.__version__)
+except (AssertionError, ValueError):
+    warnings.warn("urllib3 ({0}) or chardet ({1}) doesn't match a supported "
+                  "version!".format(urllib3.__version__, chardet.__version__),
+                  RequestsDependencyWarning)

 # Attempt to enable urllib3's SNI support, if possible
 try:
@@ -76,8 +86,6 @@
 except ImportError:
     pass

-import warnings
-
 # urllib3's DependencyWarnings should be silenced.
 from urllib3.exceptions import DependencyWarning
 warnings.simplefilter('ignore', DependencyWarning)

@@ -115,3 +115,8 @@ class RequestsWarning(Warning):
 class FileModeWarning(RequestsWarning, DeprecationWarning):
     """A file was opened in text mode, but Requests determined its binary length."""
     pass
+
+
+class RequestsDependencyWarning(RequestsWarning):
+    """An imported dependency doesn't match the expected version range."""
+    pass

Cool, if you can clean up the builds I think I'll be happy with this!

Still good for me. =)

Also, my complaint about using assert statements stands. Anyone using `python -O` will skip these checks and any version incompatibilities we have will shoot them in the face.


Raise exceptions in utils.set_environ

fixes #4127

@@ -184,3 +184,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])

@@ -15,6 +15,7 @@ dev

 - Resolve installation failure if multiprocessing is not available
 - Resolve tests crash if multiprocessing is not able to determine the number of CPU cores
+- Resolve error swallowing in utils set_environ generator

 2.17.3 (2017-05-29)
@@ -1481,4 +1482,3 @@ This is not a backwards compatible change.

 * Frustration
 * Conception
-

@@ -612,18 +612,18 @@ def set_environ(env_name, value):
     the environment variable 'env_name'.

     If 'value' is None, do nothing"""
-    if value is not None:
+    value_changed = value is not None
+    if value_changed:
         old_value = os.environ.get(env_name)
         os.environ[env_name] = value
     try:
         yield
     finally:
-        if value is None:
-            return
-        if old_value is None:
-            del os.environ[env_name]
-        else:
-            os.environ[env_name] = old_value
+        if value_changed:
+            if old_value is None:
+                del os.environ[env_name]
+            else:
+                os.environ[env_name] = old_value

 def should_bypass_proxies(url, no_proxy):

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-

 import os
+import copy
 from io import BytesIO

 import pytest
@@ -17,7 +18,7 @@
     requote_uri, select_proxy, should_bypass_proxies, super_len,
     to_key_val_list, to_native_string,
     unquote_header_value, unquote_unreserved,
-    urldefragauth, add_dict_to_cookiejar)
+    urldefragauth, add_dict_to_cookiejar, set_environ)
 from requests._internal_utils import unicode_is_ascii

 from .compat import StringIO, cStringIO
@@ -651,4 +652,29 @@ def QueryValueEx(key, value_name):
     monkeypatch.setenv('NO_PROXY', '')
     monkeypatch.setattr(winreg, 'OpenKey', OpenKey)
     monkeypatch.setattr(winreg, 'QueryValueEx', QueryValueEx)
-    assert should_bypass_proxies(url, no_proxy=None) == expected
+
+
+[USER_REDACTED].mark.parametrize(
+    'env_name, value', (
+            ('no_proxy', '192.168.0.0/24,127.0.0.1,localhost.localdomain'),
+            ('no_proxy', None),
+            ('a_new_key', '192.168.0.0/24,127.0.0.1,localhost.localdomain'),
+            ('a_new_key', None),
+    ))
+def test_set_environ(env_name, value):
+    """Tests set_environ will set environ values and will restore the environ."""
+    environ_copy = copy.deepcopy(os.environ)
+    with set_environ(env_name, value):
+        assert os.environ.get(env_name) == value
+
+    assert os.environ == environ_copy
+
+
+def test_set_environ_raises_exception():
+    """Tests set_environ will raise exceptions in context when the
+    value parameter is None."""
+    with pytest.raises(Exception) as exception:
+        with set_environ('test1', None):
+            raise Exception('Expected exception')
+
+    assert 'Expected exception' in str(exception.value)

Thanks for putting this together, [USER_REDACTED]! Overall, this looks good and seems to be a fairly sensible approach to the problem. The only thing that I might suggest adding is a simple test showing exceptions are properly raised from inside `set_environ` now.

We're currently having some issues with our windows CI environment, so we'll need to wait for that to be resolved before merging. I'd also like [USER_REDACTED] or [USER_REDACTED] to give this a glance if/when they have a free moment.

Feel free to add yourself to the AUTHORS.rst if you'd like. Thanks for you first contribution to Requests!

Thanks for this! As [USER_REDACTED] suggested, a test here would be really valuable to make sure we haven't regressed the behaviour.

Cool, just some small notes, but this looks much better: great work!

Cool, looks good. If you can add a changelog entry to `HISTORY.rst` and then squash your commits down we'll be good to go.

Hey [USER_REDACTED], this is looking good! I have a couple of minor notes on docstring cleanup, and an outstanding question. I wouldn't change anything with the conditional until [USER_REDACTED] determines if I'm overly paranoid :)

Alright, I think I'm happy with this, thanks [USER_REDACTED]! Once [USER_REDACTED] has a chance to re-review, we should be ready to merge.

Looks good to me!

Hrm, I don't love the two separate `yield` statements here. Do you mind refactoring the old `finally` statement to invert the conditional on `if value is None: return` and to instead have `if value is not None` then do the dance with `old_value`.

[USER_REDACTED] will set this refactor up and add a test

[USER_REDACTED] refactored the finally statement, let me know if this is more what you would like to see there, also added tests

We don't need this newline.

`None` is implicit in `get`, you can leave it out.

[USER_REDACTED] what are your thoughts on adding `value_changed = value is not None`, and using that single value for both conditionals? The fact these two checks are bound is implicit and has the potential to get out of sync as soon as one changes.

Could we end this with a period and either bring the `"""` up to the same line or wrap the text down? We don't have a hard cap at 80 characters in Requests.

Works for me!


Drop python2.6 support

Drop support for python2.6. Fixes #3928

Note that this targets the 3.0.0 branch, not master.

@@ -1,7 +1,6 @@
 sudo: false
 language: python
 python:
-  - "2.6"
   - "2.7"
   - "3.3"
   - "3.4"
@@ -10,20 +9,21 @@ python:
   # - "3.7-dev"
   # - "pypy"  -- appears to hang
   # - "pypy3"
+matrix:
+  allow_failures:
+    - python: 3.7-dev
 # command to install dependencies
 install: "make"
 # command to run tests
 script:
-  - |
-    if [[ "$TRAVIS_PYTHON_VERSION" != "2.6" ]] ; then make test-readme; fi
+  - make test-readme
   - make ci
 cache: pip
 jobs:
   include:
     - stage: test
       script:
-        - |
-          if [[ "$TRAVIS_PYTHON_VERSION" != "2.6" ]] ; then make test-readme; fi
+        - make test-readme
         - make ci
     - stage: coverage
       python: 3.6

@@ -1,6 +1,8 @@
 3.0.0 (2017-xx-xx)
 ++++++++++++++++++

+- Support for Python 2.6 has been dropped.
+
 - Simplified logic for determining Content-Length and Transfer-Encoding.
   Requests will now avoid setting both headers on the same request, and
   raise an exception if this is done manually by a user.

@@ -128,7 +128,7 @@ Patches and Suggestions
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
-- [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
 - Łukasz Langa <[EMAIL_REDACTED]>
 - [NAME_REDACTED] <[EMAIL_REDACTED]>
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])

@@ -81,7 +81,7 @@ Requests is ready for today's web.
 - ``.netrc`` Support
 - [NAME_REDACTED]

-Requests officially supports Python 2.6–2.7 & 3.3–3.7, and runs great on PyPy.
+Requests officially supports Python 2.7 & 3.3–3.7, and runs great on PyPy.

 Installation
 ------------

@@ -54,7 +54,6 @@ Python 3 Support?
 Yes! Here's a list of Python platforms that are officially
 supported:

-* Python 2.6
 * Python 2.7
 * Python 3.3
 * Python 3.4
@@ -69,7 +68,7 @@ These errors occur when :ref:`SSL certificate verification <verification>`
 fails to match the certificate the server responds with to the hostname
 Requests thinks it's contacting. If you're certain the server's SSL setup is
 correct (for example, because you can visit the site with your browser) and
-you're using Python 2.6 or 2.7, a possible explanation is that you need
+you're using Python 2.7, a possible explanation is that you need
 Server-Name-Indication.

 `Server-Name-Indication`_, or SNI, is an official extension to SSL where the

@@ -49,7 +49,6 @@ [NAME_REDACTED] currently supports the following versions of Python:

-- Python 2.6
 - Python 2.7
 - Python 3.3
 - Python 3.4

@@ -104,7 +104,7 @@ Requests is ready for today's web.
 - [NAME_REDACTED]
 - ``.netrc`` Support

-Requests officially supports Python 2.6–2.7 & 3.3–3.7, and runs great on PyPy.
+Requests officially supports Python 2.7 & 3.3–3.7, and runs great on PyPy.

 The User Guide

@@ -100,12 +100,7 @@

 # Set default logging handler to avoid "No handler found" warnings.
 import logging
-try:  # Python 2.7+
-    from logging import NullHandler
-except ImportError:
-    class NullHandler(logging.Handler):
-        def emit(self, record):
-            pass
+from logging import NullHandler

 logging.getLogger(__name__).addHandler(NullHandler())

@@ -187,8 +187,7 @@ def __init__(self, pool_connections=DEFAULT_POOLSIZE,
         self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

     def __getstate__(self):
-        return dict((attr, getattr(self, attr, None)) for attr in
-                    self.__attrs__)
+        return {attr: getattr(self, attr, None) for attr in self.__attrs__}

     def __setstate__(self, state):
         # Can't handle by adding 'proxy_manager' to self.__attrs__ because
@@ -478,11 +477,10 @@ def send(self, request, stream=False, timeout=None, verify=True, cert=None, prox

                     # Receive the response from the server
                     try:
-                        # For Python 2.7+ versions, use buffering of HTTP
-                        # responses
+                        # For Python 2.7, use buffering of HTTP responses
                         r = low_conn.getresponse(buffering=True)
                     except TypeError:
-                        # For compatibility with Python 2.6 versions and back
+                        # For Python 3.3+ versions, this is the default
                         r = low_conn.getresponse()

                     resp = HTTPResponse.from_httplib(

@@ -440,20 +440,21 @@ def create_cookie(name, value, **kwargs):
     By default, the pair of `name` and `value` will be set for the domain ''
     and sent on every request (this is sometimes called a "supercookie").
     """
-    result = dict(
-        version=0,
-        name=name,
-        value=value,
-        port=None,
-        domain='',
-        path='/',
-        secure=False,
-        expires=None,
-        discard=True,
-        comment=None,
-        comment_url=None,
-        rest={'HttpOnly': None},
-        rfc2109=False,)
+    result = {
+        'version': 0,
+        'name': name,
+        'value': value,
+        'port': None,
+        'domain': '',
+        'path': '/',
+        'secure': False,
+        'expires': None,
+        'discard': True,
+        'comment': None,
+        'comment_url': None,
+        'rest': {'HttpOnly': None},
+        'rfc2109': False,
+    }

     badargs = set(kwargs) - set(result)
     if badargs:

@@ -15,14 +15,14 @@

 def default_hooks():
-    return dict((event, []) for event in HOOKS)
+    return {event: [] for event in HOOKS}

 # TODO: response is the only one

 def dispatch_hook(key, hooks, hook_data, **kwargs):
     """Dispatches a hook dictionary on a given piece of data."""
-    hooks = hooks or dict()
+    hooks = hooks or {}
     hooks = hooks.get(key)
     if hooks:
         if hasattr(hooks, '__call__'):

@@ -653,10 +653,7 @@ def __getstate__(self):
         if not self._content_consumed:
             self.content

-        return dict(
-            (attr, getattr(self, attr, None))
-            for attr in self.__attrs__
-        )
+        return {attr: getattr(self, attr, None) for attr in self.__attrs__}

     def __setstate__(self, state):
         for name, value in state.items():

@@ -736,7 +736,7 @@ def mount(self, prefix, adapter):
             self.adapters[key] = self.adapters.pop(key)

     def __getstate__(self):
-        state = dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
+        state = {attr: getattr(self, attr, None) for attr in self.__attrs__}
         return state

     def __setstate__(self, state):

@@ -679,22 +679,8 @@ def should_bypass_proxies(url, no_proxy):
                     # to apply the proxies on this URL.
                     return True

-    # If the system proxy settings indicate that this URL should be bypassed,
-    # don't proxy.
-    # The proxy_bypass function is incredibly buggy on OS X in early versions
-    # of Python 2.6, so allow this call to fail. Only catch the specific
-    # exceptions we've seen, though: this call failing in other ways can reveal
-    # legitimate problems.
     with set_environ('no_proxy', no_proxy_arg):
-        try:
-            bypass = proxy_bypass(netloc)
-        except (TypeError, socket.gaierror):
-            bypass = False
-
-    if bypass:
-        return True
-
-    return False
+        return bool(proxy_bypass(netloc))

 def get_environ_proxies(url, no_proxy=None):

@@ -80,7 +80,7 @@ def run_tests(self):
         '[NAME_REDACTED] :: English',
         'License :: OSI Approved :: [NAME_REDACTED]',
         '[NAME_REDACTED] :: Python',
-        '[NAME_REDACTED] :: Python :: 2.6',
+        '[NAME_REDACTED] :: Python :: 2',
         '[NAME_REDACTED] :: Python :: 2.7',
         '[NAME_REDACTED] :: Python :: 3',
         '[NAME_REDACTED] :: Python :: 3.3',
@@ -95,7 +95,6 @@ def run_tests(self):
     extras_require={
         'security': ['pyOpenSSL>=0.14', 'cryptography>=1.3.4', 'idna>=2.0.0'],
         'socks': ['PySocks>=1.5.6, !=1.5.7'],
-        'socks:sys_platform == "win32" and (python_version == "2.7" or python_version == "2.6")': ['win_inet_pton'],
+        'socks:sys_platform == "win32" and python_version == "2.7"': ['win_inet_pton'],
     },
 )
-

@@ -933,7 +933,7 @@ def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):

     def test_urlencoded_get_query_multivalued_param(self, httpbin):

-        r = requests.get(httpbin('get'), params=dict(test=['foo', 'baz']))
+        r = requests.get(httpbin('get'), params={'test': ['foo', 'baz']})
         assert r.status_code == 200
         assert r.url == httpbin('get?test=foo&test=baz')

@@ -566,7 +566,7 @@ def test_add_dict_to_cookiejar(cookiejar):
     cookiedict = {'test': 'cookies',
                   'good': 'cookies'}
     cj = add_dict_to_cookiejar(cookiejar, cookiedict)
-    cookies = dict((cookie.name, cookie.value) for cookie in cj)
+    cookies = {cookie.name: cookie.value for cookie in cj}
     assert cookiedict == cookies

@@ -1,8 +1,8 @@
 [tox]
-envlist = py26,py27,py33,py34,py35,py36
+envlist = py27,py33,py34,py35,py36

 [testenv]

 commands =
     pip install -e .[socks]
-    python setup.py test
\ No newline at end of file
+    python setup.py test

I believe you can drop the [`NullHandler`](https://github.com/hobarrera/requests/blob/drop-py26/requests/__init__.py#L76-L81) workaround.

---

The code could now use set literals and dict/set comprehension. Thoughts on converting the code as part of this commit? Using [flake8-comprehensions](https://pypi.python.org/pypi/flake8-comprehensions) I discovered the following:

```
./requests/models.py:659:16: C402 Unnecessary generator - rewrite as a dict comprehension.
./requests/hooks.py:18:12: C402 Unnecessary generator - rewrite as a dict comprehension.
./requests/hooks.py:25:22: C408 Unnecessary dict call - rewrite as a literal.
./requests/adapters.py:133:16: C402 Unnecessary generator - rewrite as a dict comprehension.
./requests/sessions.py:758:17: C402 Unnecessary generator - rewrite as a dict comprehension.
./requests/cookies.py:441:14: C408 Unnecessary dict call - rewrite as a literal.
./tests/test_utils.py:569:15: C402 Unnecessary generator - rewrite as a dict comprehension.
./tests/test_requests.py:933:49: C408 Unnecessary dict call - rewrite as a literal.
```

Alright, I think we're close to calling this good. For the time being, it looks like we'll need to work around 2.6 builds failing on proposed/3.0.0.

[USER_REDACTED], [USER_REDACTED], any final thoughts here? I think this looks good from my end.

Good that this one was removed, but there should exist a trove classifier for `'[NAME_REDACTED] :: Python :: 2'` as requests still supports Python 2 in some capacity.

Added `'[NAME_REDACTED] :: Python :: 2'` Keep in mind that `2.7` is still there as well.

The comments for this bit were oddly out of date; only this branch works on python3, while the above fails (with `TypeError`).


Catch error if multiprocessing is not available or is unable to determine the number of CPUs

resolves #4113

@@ -8,6 +8,10 @@ dev

 **Bugfixes**

+- Resolve installation failure if multiprocessing is not available
+- Resolve tests crash if multiprocessing is not able to determine the number of CPU cores
+
+
 2.17.3 (2017-05-29)
 +++++++++++++++++++

@@ -8,7 +8,6 @@

 from setuptools import setup
 from setuptools.command.test import test as TestCommand
-from multiprocessing import cpu_count

 here = os.path.abspath(os.path.dirname(__file__))

@@ -17,7 +16,11 @@ class PyTest(TestCommand):

     def initialize_options(self):
         TestCommand.initialize_options(self)
-        self.pytest_args = ['-n', str(cpu_count()), '--boxed']
+        try:
+            from multiprocessing import cpu_count
+            self.pytest_args = ['-n', str(cpu_count()), '--boxed']
+        except (ImportError, NotImplementedError):
+            self.pytest_args = ['-n', '1', '--boxed']

     def finalize_options(self):
         TestCommand.finalize_options(self)

Thanks for this, it looks good! Can you rebase onto the current master and add a note to the changelog please?

Cool, this is almost ready to go: there's a small typo though.

Cool, this looks like a good bugfix. I'll try to get it out by Thursday, but I want to confirm that there's nothing else lurking before I do that.

This should be `installation`.


Remove session redirect cache

Addresses #4055. Essentially, this just strips out the redirect cache in `sessions.py`.

I ran the tests and didn't find any side effects of removing the cache. This is my first PR to requests, so let me know if there's any additional steps that need to be done.

Thanks!

@@ -13,8 +13,6 @@
 from collections import Mapping
 from datetime import timedelta

-from urllib3._collections import RecentlyUsedContainer
-
 from .auth import _basic_auth_str
 from .compat import cookielib, is_py3, OrderedDict, urljoin, urlparse
 from .cookies import (
@@ -39,8 +37,6 @@
 # formerly defined here, reexposed here for backward compatibility
 from .models import REDIRECT_STATI

-REDIRECT_CACHE_SIZE = 1000
-
 # Preferred clock, based on which one is more accurate on a given system.
 if platform.system() == 'Windows':
     try:  # Python 3.3+
@@ -158,9 +154,6 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,
                 url = requote_uri(url)

             prepared_request.url = to_native_string(url)
-            # Cache the url, unless it redirects to itself.
-            if resp.is_permanent_redirect and req.url != prepared_request.url:
-                self.redirect_cache[req.url] = prepared_request.url

             self.rebuild_method(prepared_request, resp)

@@ -393,9 +386,6 @@ def __init__(self):
         self.mount('https://', HTTPAdapter())
         self.mount('http://', HTTPAdapter())

-        # Only store 1000 redirects to prevent using infinite memory
-        self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
-
     def __enter__(self):
         return self

@@ -623,16 +613,6 @@ def send(self, request, **kwargs):
         stream = kwargs.get('stream')
         hooks = request.hooks

-        # Resolve URL in redirect cache, if available.
-        if allow_redirects:
-            checked_urls = set()
-            while request.url in self.redirect_cache:
-                checked_urls.add(request.url)
-                new_url = self.redirect_cache.get(request.url)
-                if new_url in checked_urls:
-                    break
-                request.url = new_url
-
         # Get the appropriate adapter to use
         adapter = self.get_adapter(url=request.url)

@@ -745,18 +725,12 @@ def mount(self, prefix, adapter):

     def __getstate__(self):
         state = dict((attr, getattr(self, attr, None)) for attr in self.__attrs__)
-        state['redirect_cache'] = dict(self.redirect_cache)
         return state

     def __setstate__(self, state):
-        redirect_cache = state.pop('redirect_cache', {})
         for attr, value in state.items():
             setattr(self, attr, value)

-        self.redirect_cache = RecentlyUsedContainer(REDIRECT_CACHE_SIZE)
-        for redirect, to in redirect_cache.items():
-            self.redirect_cache[redirect] = to
-

 def session():
     """

Thanks for contributing to Requests, [USER_REDACTED]! We'll wait on merging until one of the other maintainers signs off, but this looks from my end.

Great work! ✨

I hated this code. So glad it's going.

Hope I'm not responsible for it landing in the codebase. I probably am :)


API improvements

- `Response.next`

Related: #4065.

@@ -593,6 +593,7 @@ def __init__(self):

         self._content = False
         self._content_consumed = False
+        self._next = None

         #: [NAME_REDACTED] of responded HTTP Status, e.g. 404 or 200.
         self.status_code = None
@@ -706,12 +707,17 @@ def is_redirect(self):

     [USER_REDACTED]
     def is_permanent_redirect(self):
-        """True if this Response one of the permanent versions of redirect"""
+        """True if this Response one of the permanent versions of redirect."""
         return ('location' in self.headers and self.status_code in (codes.moved_permanently, codes.permanent_redirect))

+    [USER_REDACTED]
+    def next(self):
+        """Returns a PreparedRequest for the next request in a redirect chain, if there is one."""
+        return self._next
+
     [USER_REDACTED]
     def apparent_encoding(self):
-        """The apparent encoding, provided by [NAME_REDACTED]"""
+        """The apparent encoding, provided by [NAME_REDACTED]"""
         return chardet.detect(self.content)['encoding']

     def iter_content(self, chunk_size=1, decode_unicode=False):

@@ -114,8 +114,8 @@ def get_redirect_target(self, resp):
         return None

     def resolve_redirects(self, resp, req, stream=False, timeout=None,
-                          verify=True, cert=None, proxies=None, **adapter_kwargs):
-        """Receives a Response. Returns a generator of Responses."""
+                          verify=True, cert=None, proxies=None, yield_requests=False, **adapter_kwargs):
+        """Receives a Response. Returns a generator of Responses or Requests."""

         hist = [] # keep track of history

@@ -203,22 +203,26 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,
             # Override the original request.
             req = prepared_request

-            resp = self.send(
-                req,
-                stream=stream,
-                timeout=timeout,
-                verify=verify,
-                cert=cert,
-                proxies=proxies,
-                allow_redirects=False,
-                **adapter_kwargs
-            )
+            if yield_requests:
+                yield req
+            else:

-            extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)
+                resp = self.send(
+                    req,
+                    stream=stream,
+                    timeout=timeout,
+                    verify=verify,
+                    cert=cert,
+                    proxies=proxies,
+                    allow_redirects=False,
+                    **adapter_kwargs
+                )

-            # extract redirect url, if any, for the next loop
-            url = self.get_redirect_target(resp)
-            yield resp
+                extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)
+
+                # extract redirect url, if any, for the next loop
+                url = self.get_redirect_target(resp)
+                yield resp

     def rebuild_auth(self, prepared_request, response):
         """When being redirected we may want to strip authentication from the
@@ -597,8 +601,7 @@ def delete(self, url, **kwargs):
         return self.request('DELETE', url, **kwargs)

     def send(self, request, **kwargs):
-        """
-        Send a given PreparedRequest.
+        """Send a given PreparedRequest.

         :rtype: requests.Response
         """
@@ -668,6 +671,13 @@ def send(self, request, **kwargs):
             r = history.pop()
             r.history = history

+        # If redirects aren't being followed, store the response on the Request for Response.next().
+        if not allow_redirects:
+            try:
+                r._next = self.resolve_redirects(r, request, yield_requests=True, **kwargs).next()
+            except StopIteration:
+                pass
+
         if not stream:
             r.content

Why have this when we already have `session.send`?

more intuitive. i dabbled about it alot when i was writing it originally, and i opted out, decided that i could always add it back in later — i think it's time for it to go in. it's just a nice shortcut.

`Response.next().send(session)` is kinda nice

a lot better than `session.send(Response.next())` imo

Yield requests maybe?

i can peel that out if you're really against it — i think it's harmless though, and just smooths out the api a bit

sure!

I see no difference, except that one is more intuitive for people already using that API. Sessions send requests, requests don't send sessions.

If this isn't doing anything, why isn't it just a property?

hmmmm

"Harmless" seems in conflict with "Extra surface area that doesn't seem to provide much benefit"

Also, `next_request` would be more intuitive.

i remember why i left this off now.

if responses recorded their sent arguments, this would have made sense.

Ok, so I just stumbled across this in the code, and this is *very* confusing. Despite this being a generator, it is a generator that can literally be iterated only once in this mode before it starts doing something stupid (where "stupid" is that it repeatedly runs the same processing). That's not actually written down anywhere, so it would probably help us to at least say that the "yield requests" mode can only yield *one* request.


AttributeError error raised when `files` file-pointer (fp) resolves to None

The library raises NoneType error when file-pointer (fp) resolves to None.

```
>>> from requests import post
>>> r = post("https://example.com", files={"file-name": None})
AttributeError: 'NoneType' object has no attribute 'read'
```

However, when a param value or json field is None they are not included in the request body.

```
>>> from requests import get
>>> r = get("https://example.com", params={"file-name": None})
>>> r.request.url
```

This commit makes the beahviour consistent for files.

@@ -187,3 +187,4 @@ Patches and Suggestions
 - Nehal J Wani ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])

@@ -23,6 +23,7 @@ dev
 - DNS resolution in proxy bypass no longer includes the username and password in
   the request. This also fixes the issue of DNS queries failing on macOS.
 - Properly normalize adapter prefixes for url comparison.
+- Passing ``None`` as a file pointer to the ``files`` param no longer raises an exception.

 2.18.4 (2017-08-15)
 +++++++++++++++++++

@@ -155,8 +155,12 @@ def _encode_files(files, data):

             if isinstance(fp, (str, bytes, bytearray)):
                 fdata = fp
-            else:
+            elif hasattr(fp, 'read'):
                 fdata = fp.read()
+            elif fp is None:
+                continue
+            else:
+                fdata = fp

             rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
             rf.make_multipart(content_type=ft)

@@ -676,6 +676,14 @@ def test_POSTBIN_GET_POST_FILES(self, httpbin):
         with pytest.raises(ValueError):
             requests.post(url, files=['bad file data'])

+    def test_invalid_files_input(self, httpbin):
+
+        url = httpbin('post')
+        post = requests.post(url,
+                             files={"random-file-1": None, "random-file-2": 1})
+        assert b'name="random-file-1"' not in post.request.body
+        assert b'name="random-file-2"' in post.request.body
+
     def test_POSTBIN_SEEKED_OBJECT_WITH_NO_ITER(self, httpbin):

         class TestStream(object):

This seems pretty reasonable! Do you mind also adding a test?

Hey [USER_REDACTED], I've created a branch on the Requests repository titled "none_filepointer".

You can use `git fetch` to pull down the branch locally and cherry pick your commit. If you don't have a remote url for Requests, you can use [this guide](https://help.github.com/articles/adding-a-remote/) to get one set up.

I've added one additional comment that I'd like [USER_REDACTED] to weigh in on, and then I think we're set.

I'd rather we explicitly look for `None`. Do you mind making this change?

Sorry that I randomly come out of nowhere once in a blue moon to give review comments so ignore my comment if you like :) But hasattr has a small edge case which can cause an unexpected error in Python 2. You can refer to the link here:
https://hynek.me/articles/hasattr/

The article does mention that if you know what you are doing then its all good. My recommendation is to use getattr with default value of None to avoid AttributeError

[USER_REDACTED] that's because properties in python2 aren't attributes. very known and undertstood functionality.

In this case, are we expecting the objects `__str__` representation to be sufficient? We currently blow up on things like `datetime` or `dict`, but this may have unexpected results.

If we're OK with the above, would it make more sense to reorg the conditionals to check for `read`, `None` and then everything else would be `fdata = fp`?

e.g.
```python
if hasattr(fp, 'read'):
    fdata = fp.read()
elif fp is None:
    continue
else:
    fdata = fp
```

So the code that will eventually handle this is [here](https://github.com/urllib3/urllib3/blob/c8c98c473ab30f427fe05057f2ceded10ec93153/urllib3/filepost.py#L78..L90).

Honestly, at this point, we *could* simplify the conditional but I'd rather leave it and let it blow up for folks. In 3.0, I'd love to have this else branch raise an exception, though.

Alright, I'm on board with that. [USER_REDACTED] would you mind removing this `else` conditional to keep our existing functionality. We'll look at adding a more explicit exception in 3.0. Thanks!

[USER_REDACTED]

```python
else:
    fdata = fp

rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
```

Getting rid of else part will result in fdata being an unbound variable. Should we set it to fdata=None in else and let it blow up or leave things as is?

Urgh, yep. Ok so [USER_REDACTED] we may just want to put the exception in now. Non-file objects were already exploding on the read call so they never worked.

We *could* make sure the exception inherits from AttributeError if we want to ensure we’re not breaking anyone currently relying on this defect for flow control.

[USER_REDACTED] I'd rather never inherit from `AttributeError` if possible. That is just a world of pain. We've done this in the past, but that means if people are putting too much code in their `try/except` block, we may cause them to enter an exception handling block.

In short, I think the code as is may be safe right now. If we want to be extra magical, we could do `fdata = str(fp)` but I really don't want to add that behaviour at all. Let's just leave this as is, I guess and plan to add the exception in 3.0

Sounds good! I think I misread your first response, so we should be set then.


Pin sphinx

This will fix the build issues in Python 2.6 due to Sphinx's new requirement of `typing` in versions 1.6+. This pinning (capping) should only be in place until a solution for representing markers is introduced in Pipfile.

While we're here doing cleanup, I've also removed the no longer needed `pipenv lock` before each build. Some preliminary tests show it saving 15-20% on the total build time for CI. We can scrap that though if it's not inline with the goal of the PR, getting the build working.

@@ -2,7 +2,6 @@

 init:
 	pip install pipenv
-	pipenv lock
 	pipenv install --dev
 	pipenv run pip install -e .[socks]

@@ -6,8 +6,8 @@ verify_ssl = true
 pytest = ">=2.8.0"
 codecov = "*"
 pytest-httpbin = "==0.0.7"
-sphinx = "*"
 pytest-mock = "*"
 pytest-cov = "*"
-pysocks = "*"
 alabaster = "*"
+Sphinx = "<=1.5.5"
+PySocks = "*"

Is this necessary? This should only be run on `init`.

Sorry, are you asking if the removal is necessary, or the locking?

We can leave this in, but it's an extra 4-5 minutes of CI time that doesn't actually need to run.  Pipenv is no longer worrying about hashes on a package specific level, so platform conflicts aren't a problem now. If there is an actual change to the Pipfile not represented, Pipenv will rebuild things without this command, otherwise it uses the current Pipfile.lock.

Like I said, we can not include this commit here, but this line should be removed at some point.

Edit: `init` runs every time we start a build with `make` which means we perform the same expensive action on every build.

Ok, I don't mind removing it. =)


Test that the readme renders in CI.

Resolves #4030.

@@ -14,6 +14,8 @@ python:
 install: "make"
 # command to run tests
 script:
+  - |
+    if [[ "$TRAVIS_PYTHON_VERSION" != "2.6" ]] ; then make test-readme; fi
   - make coverage
 after_success:
   - pipenv run codecov

@@ -47,7 +47,7 @@ Release History
 **Bugfixes**

 - Much improved handling of non-ASCII ``Location`` header values in redirects.
-  Fewer ``UnicodeDecodeError``s are encountered on Python 2, and Python 3 now
+  Fewer ``UnicodeDecodeErrors`` are encountered on Python 2, and Python 3 now
   correctly understands that Latin-1 is unlikely to be the correct encoding.
 - If an attempt to ``seek`` file to find out its length fails, we now
   appropriately handle that by [NAME_REDACTED]

@@ -10,6 +10,9 @@ test:
 	# the -k flag, like "py.test -k test_path_is_not_double_encoded"
 	pipenv run py.test tests

+test-readme:
+	pipenv run python setup.py check -r -s
+
 coverage:
 	pipenv run py.test --cov-config .coveragerc --verbose --cov-report term --cov-report xml --cov=requests tests

@@ -9,5 +9,6 @@ pytest-httpbin = "==0.0.7"
 pytest-mock = "*"
 pytest-cov = "*"
 alabaster = "*"
+readme_renderer = "*"
 Sphinx = "<=1.5.5"
-PySocks = "*"
+PySocks = "*"
\ No newline at end of file

This shouldn't be run on Python 2.6. It seems `readme-renderer` doesn't support 2.6.

Agh, that's a pain. Ok.


Remove some unused imports.

@@ -19,7 +19,7 @@
 from .compat import urlparse, basestring
 from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
                     prepend_scheme_if_needed, get_auth_from_url, urldefragauth,
-                    select_proxy, to_native_string)
+                    select_proxy)
 from .structures import CaseInsensitiveDict
 from .packages.urllib3.exceptions import ClosedPoolError
 from .packages.urllib3.exceptions import ConnectTimeoutError

@@ -20,7 +20,6 @@
 from .cookies import extract_cookies_to_jar
 from ._internal_utils import to_native_string
 from .utils import parse_dict_header
-from .status_codes import codes

 CONTENT_TYPE_FORM_URLENCODED = 'application/x-www-form-urlencoded'
 CONTENT_TYPE_MULTI_PART = 'multipart/form-data'

@@ -16,7 +16,7 @@
 # such as in [NAME_REDACTED] https://github.com/[USER_REDACTED]nethreitz/requests/issues/3578.
 import encodings.idna

-from io import BytesIO, UnsupportedOperation
+from io import UnsupportedOperation
 from .hooks import default_hooks
 from .structures import CaseInsensitiveDict

@@ -36,7 +36,7 @@
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
     iter_slices, guess_json_utf, super_len, check_header_validity)
 from .compat import (
-    cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
+    cookielib, urlunparse, urlsplit, urlencode, str, bytes,
     is_py2, chardet, builtin_str, basestring)
 from .compat import json as complexjson
 from .status_codes import codes

@@ -27,8 +27,8 @@
 from .compat import (
     quote, urlparse, bytes, str, OrderedDict, unquote, getproxies,
     proxy_bypass, urlunparse, basestring, integer_types)
-from .cookies import RequestsCookieJar, cookiejar_from_dict
-from .structures import CaseInsensitiveDict, TimedCache, TimedCacheManaged
+from .cookies import cookiejar_from_dict
+from .structures import CaseInsensitiveDict, TimedCacheManaged
 from .exceptions import (
     InvalidURL, InvalidHeader, FileModeWarning, UnrewindableBodyError)

Looks good aside from the chardet change.

Thanks [USER_REDACTED]

Indeed, thanks! :sparkles: :cake: :sparkles:

Can you remove the changes from this file please? This is a third-party package we vendor unchanged.

Thanks, I hadn't spotted that.


Fix the additional newline generated by iter_lines() caused by a '\r\n' pair being separated in two different chunks.

When a "\r\n" CRLF pair is splitted into two chunks accidentally, iter_lines() generates two line breaks instead of one. This patch fixes the incorrect behavior and closes issue #3980.

@@ -776,7 +776,11 @@ def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None, delimiter=

         .. note:: This method is not reentrant safe.
         """
+        carriage_return = u'\r' if decode_unicode else b'\r'
+        line_feed = u'\n' if decode_unicode else b'\n'
+
         pending = None
+        last_chunk_ends_with_cr = False

         for chunk in self.iter_content(chunk_size=chunk_size,
                                        decode_unicode=decode_unicode):
@@ -796,6 +800,23 @@ def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None, delimiter=
             if delimiter:
                 lines = chunk.split(delimiter)
             else:
+                # Python splitlines() supports the universal newline (PEP 278).
+                # That means, '\r', '\n', and '\r\n' are all treated as end of
+                # line. If the last chunk ends with '\r', and the current chunk
+                # starts with '\n', they should be merged and treated as only
+                # *one* new line separator '\r\n' by splitlines().
+                # This rule only applies when splitlines() is used.
+
+                # The last chunk ends with '\r', so the '\n' at chunk[0]
+                # is just the second half of a '\r\n' pair rather than a
+                # new line break. Just skip it.
+                skip_first_char = last_chunk_ends_with_cr and chunk.startswith(line_feed)
+                last_chunk_ends_with_cr = chunk.endswith(carriage_return)
+                if skip_first_char:
+                    chunk = chunk[1:]
+                    # it's possible that after stripping the '\n' then chunk becomes empty
+                    if not chunk:
+                        continue
                 lines = chunk.splitlines()

             # Calling `.split(delimiter)` will always end with whatever text

@@ -1303,57 +1303,76 @@ def test_response_lines(self):
         might not be lined up ideally.
         """
         mock_chunks = [
-            'This \r\n',
-            '',
-            'is\r',
-            '\n',
-            'a',
-            ' ',
-            '',
-            '',
-            'test.',
-            '\r',
-            '\n',
-            'end.',
+            b'This \r\n',
+            b'',
+            b'is\r',
+            b'\n',
+            b'a',
+            b' ',
+            b'',
+            b'',
+            b'test.',
+            b'\r',
+            b'\n',
+            b'end.',
         ]
-        mock_data = ''.join(mock_chunks)
+        mock_data = b''.join(mock_chunks)
+        unicode_mock_data = mock_data.decode('utf-8')

-        mock_iter_content = lambda *args, **kwargs: (e for e in mock_chunks)
+        def mock_iter_content(*args, **kwargs):
+            if kwargs.get("decode_unicode"):
+                return (e.decode('utf-8') for e in mock_chunks)
+            return (e for e in mock_chunks)

         r = requests.Response()
         r._content_consumed = True
         r.iter_content = mock_iter_content

-        assert list(r.iter_lines(delimiter='\r\n')) == mock_data.split('\r\n')
+        # decode_unicode=None, output raw bytes
+        assert list(r.iter_lines(delimiter=b'\r\n')) == mock_data.split(b'\r\n')

-        # Because '\n' is a single line-end, when `iter_lines()` receives
-        # the chunks containing a single '\n', it emits '' as a line -- whereas
-        # `.splitlines()` combines with the '\r' and splits on `\r\n`.
+        # decode_unicode=True, output unicode strings
+        assert list(r.iter_lines(decode_unicode=True, delimiter=u'\r\n')) == unicode_mock_data.split(u'\r\n')
+
+        # When delimiter is None, we should yield the same result as splitlines()
+        # which supports the universal newline.
+        # '\r', '\n', and '\r\n' are all treated as one line break.
+
+        # decode_unicode=None, output raw bytes
         result = list(r.iter_lines())
-        assert result != mock_data.splitlines()
-        assert result[2] == ''
-        assert result[4] == ''
+        assert result == mock_data.splitlines()
+
+        # decode_unicode=True, output unicode strings
+        result = list(r.iter_lines(decode_unicode=True))
+        assert result == unicode_mock_data.splitlines()
+
         # If we change all the line breaks to `\r`, we should be okay.
-        mock_chunks = [chunk.replace('\n', '\r') for chunk in mock_chunks]
-        mock_data = ''.join(mock_chunks)
+        # decode_unicode=None, output raw bytes
+        mock_chunks = [chunk.replace(b'\n', b'\r') for chunk in mock_chunks]
+        mock_data = b''.join(mock_chunks)
         assert list(r.iter_lines()) == mock_data.splitlines()

+        # decode_unicode=True, output unicode strings
+        unicode_mock_data = mock_data.decode('utf-8')
+        assert list(r.iter_lines(decode_unicode=True)) == unicode_mock_data.splitlines()
+

     [USER_REDACTED].mark.parametrize(
         'content, expected_no_delimiter, expected_delimiter', (
-            ([''], [], []),
-            (['line\n'], ['line'], ['line\n']),
-            (['line', '\n'], ['line'], ['line\n']),
-            (['line\r\n'], ['line'], ['line', '']),
+            ([b''], [], []),
+            ([b'line\n'], [u'line'], [u'line\n']),
+            ([b'line', b'\n'], [u'line'], [u'line\n']),
+            ([b'line\r\n'], [u'line'], [u'line', u'']),
             # Empty chunk in the end of stream, same behavior as the previous
-            (['line\r\n', ''], ['line'], ['line', '']),
-            (['line', '\r\n'], ['line'], ['line', '']),
-            (['a\r', '\nb\r'], ['a', '', 'b'], ['a', 'b\r']),
-            (['a\n', '\nb'], ['a', '', 'b'], ['a\n\nb']),
-            (['a\r\n','\rb\n'], ['a', '', 'b'], ['a', '\rb\n']),
-            (['a\nb', 'c'], ['a', 'bc'], ['a\nbc']),
-            (['a\n', '\rb', '\r\nc'], ['a', '', 'b', 'c'], ['a\n\rb', 'c']),
-            (['a\r\nb', '', 'c'], ['a', 'bc'], ['a', 'bc'])  # Empty chunk with pending data
+            ([b'line\r\n', b''], [u'line'], [u'line', u'']),
+            ([b'line', b'\r\n'], [u'line'], [u'line', u'']),
+            ([b'a\r', b'\nb\r'], [u'a', u'b'], [u'a', u'b\r']),
+            ([b'a\r', b'\n', b'\nb'], [u'a', u'', u'b'], [u'a', u'\nb']),
+            ([b'a\n', b'\nb'], [u'a', u'', u'b'], [u'a\n\nb']),
+            ([b'a\r\n', b'\rb\n'], [u'a', u'', u'b'], [u'a', u'\rb\n']),
+            ([b'a\nb', b'c'], [u'a', u'bc'], [u'a\nbc']),
+            ([b'a\n', b'\rb', b'\r\nc'], [u'a', u'', u'b', u'c'], [u'a\n\rb', u'c']),
+            ([b'a\r\nb', b'', b'c'], [u'a', u'bc'], [u'a', u'bc'])  # Empty chunk with pending data
         ))
     def test_response_lines_parametrized(self, content, expected_no_delimiter, expected_delimiter):
         """
@@ -1363,13 +1382,22 @@ def test_response_lines_parametrized(self, content, expected_no_delimiter, expec
         https://github.com/[USER_REDACTED]nethreitz/requests/pull/2431#issuecomment-72333964
         """
         mock_chunks = content
-        mock_iter_content = lambda *args, **kwargs: (e for e in mock_chunks)
+        def mock_iter_content(*args, **kwargs):
+            if kwargs.get("decode_unicode"):
+                return (e.decode('utf-8') for e in mock_chunks)
+            return (e for e in mock_chunks)

         r = requests.Response()
         r._content_consumed = True
         r.iter_content = mock_iter_content
-        assert list(r.iter_lines()) == expected_no_delimiter
-        assert list(r.iter_lines(delimiter='\r\n')) == expected_delimiter
+
+        # decode_unicode=True, output unicode strings
+        assert list(r.iter_lines(decode_unicode=True)) == expected_no_delimiter
+        assert list(r.iter_lines(decode_unicode=True, delimiter='\r\n')) == expected_delimiter
+
+        # decode_unicode=None, output raw bytes
+        assert list(r.iter_lines()) == [line.encode('utf-8') for line in expected_no_delimiter]
+        assert list(r.iter_lines(delimiter=b'\r\n')) == [line.encode('utf-8') for line in expected_delimiter]

     def test_prepared_request_is_pickleable(self, httpbin):
         p = requests.Request('GET', httpbin('get')).prepare()

Cool, this patch looks broadly right! I've left a few minor structural notes in the diff.

Additionally, it'd be good if you could add a test that reproduces this problem and demonstrates that it's fixed. Should be easily enough done with careful choice of body and chunk size.

Cool, this is looking really good. A few minor stylistic notes for you.

So this is now looking good except for my possible concern about the readability of our two alternative approaches. I'd like either [USER_REDACTED] or [USER_REDACTED] to weigh in there, if possible.

This looks fine to me.

Looks good to me too. Do you want to squash the commits down just for cleanliness sake? Then I'll go ahead and merge.

Not at all, thank you for working with us to get this ready to go. Well done! ✨ :cake: :sparkles:

Let's hoist the result of these two `else` calls to the top of the function, rather than check it twice each time around the loop:

```python
carriage_return = u'\r' if decode_unicode else b'\r'
line_feed = u'\n' if decode_unicode else b'\r'
```

We should expand this comment to explain that this only affects the `splitlines` case because `splitlines` will treat any of `\r`, `\r\n`, and `\n` as newlines, and so splitting `\r\n` into two chunks will get misleading results.

It would also be good to hoist this into the block that holds the `splitlines` call, as we don't need to do this processing in any other situation.

OK, will do that.

One thing to keep in mind here is the final line of the file ending in `\r\n`. If we split at `\r`, we get a final one byte line of `\n`. This code makes chunk equivalent to `''` and `''.splitlines()` will result in an empty list. You'll get an index out of bounds with the call `lines[-1]` below.

Probably a rarer edge case but something we should cover.

Fixed. Thanks.

Why has the condition been doubled up here? Under what circumstances will `line[-1][-1]` be falsey?

Again, this should be brought into the if branch above.

These tests run on Python 2, so Unicode strings must be prefixed with `u`

Again, your test seems to show that all the unprefixed strings are intended to be Unicode: they need to be prefixed

Sorry I forgot to remove them. It should be prevented by [NAME_REDACTED][1:].

Wasn't universal newline support dropped in Python 3? I know you can no longer get a TextIOBuffer object that supports that. How does that affect splitting strings?

Why is there an extra space between `and` and `lines[-1][-1]`?

Well, splitlines() actually supports a superset of universal newline.
FYI: https://docs.python.org/3/library/stdtypes.html#str.splitlines
The unicode strings supports a superset of universal newline.
However, the new python 3 bytes behaves differently.
https://docs.python.org/3/library/stdtypes.html#bytes.splitlines
In both cases, however, the '\r', '\n', and '\r\n' rule still apply.

removed.

Minor nit: as there are inverted commas around `'\r'` in the line above, they should be added here around `\n`.

Minor nit: should be `*one*`, not `"one"`.

Oooh, just noticed a subtle misbehaviour. Imagine we get the following chunks:

```python
[b'this is a string\r', b'\n', b'\nso is this']
```

This will not do the right thing: we'll strip *two* newline characters, rather than emit an empty line. That is, we'll emit: `[b'this is a string', b'so is this']` instead of `[b'this is a string', b'', b'so is this']`, which is what we *should* emit.

This is because a chunk that is *just* a `\n` causes an early continue, above, without resetting the value of the `last_chunk_ends_with_cr` boolean. Essentially it gets completely ignored. That's not right.

It'd be good to add a test case (or modify an existing one) to include that case, and then fix it up by [NAME_REDACTED] before the `if not chunk: continue` clause.

We can probably collapse this line into the one above it as with the other assertions:

```python
assert list(r.iter_lines()) == mock_data.splitlines()
```

This piece of code seems to be outdated. I didnt' see it in the current HEAD.

[USER_REDACTED] Nice catch! Sorry that I overlooked such a simple check. Just add a simple fix along with a test case for it. BTW, I did not hoist the flag per your request. It will look somewhat like this:
```
                # a temp variable is needed here and I cannot find a good variable naming that is not confusing
                old_last_chunk_ends_with_cr = last_chunk_ends_with_cr
                last_chunk_ends_with_cr = chunk.endswith(carriage_return)
                if old_last_chunk_ends_with_cr and chunk.startswith(line_feed):
                    chunk = chunk[1:]
                    if not chunk:
                        continue
```
Also this yields an unnecessary method call endwiths('\r') since when chun[1:] is empty we already know the result is False. So I prefer just setting the flag to False though it looks like code duplication at first glance.
```
if not chunk:
    last_chunk_ends_with_cr = False
```
Simple and readable.

Hrm. I don't know if I think it's more readable to do it this way. It means there are two places in the code that set this variable, instead of just one, which forces the reader to ask themselves *why* it was done this way. I don't know that the efficiency gains are worth it. 🤔

Yup, you're right.

Now that I understand what concerns [USER_REDACTED] has, I agree. I think this could be better written as:

```py
chunk_startswith_line_feed = last_chunk_ends_with_cr and chunk.startswith(line_feed)
last_chunk_ends_with_cr = chunk.endswith(carriage_return)
if chunk_startswith_line_feed:
    chunk = chunk[1:]
    if not chunk:
        continue
lines = chunk.splitlines()
```

This properly *explains* the condition on this line with a descriptive variable name. It avoids the `old_last_chunk_ends_with_cr` variable and it keeps things concise and easy to reason about.

Yep, I'm on board with [USER_REDACTED]'s suggestion. That seems the most readable option out of what's been suggested.

The only minor input I have is standardizing the use of `_` in `ends_with` vs `startswith` in the variable names.