Add appveyor testing of Requests.

In general it'd be good to have more testing, but it's also possible that via PR #3979 we'll get unexpected different behaviour on Windows than on other platforms. For this reason, we should start also testing on Windows.

This PR takes the configuration used for urllib3, updates it for Requests, and then provides it. We'll still need to actually enable Appveyor so that it'll try to build, which I cannot do as I don't have admin on the repo. [USER_REDACTED], mind giving that a try?

@@ -0,0 +1,229 @@
+# Sample script to install Python and pip under Windows
+# Authors: [NAME_REDACTED], [NAME_REDACTED], [NAME_REDACTED], and [NAME_REDACTED]
+# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+
+$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
+$BASE_URL = "https://www.python.org/ftp/python/"
+$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
+$GET_PIP_PATH = "C:\get-pip.py"
+
+$PYTHON_PRERELEASE_REGEX = @"
+(?x)
+(?<major>\d+)
+\.
+(?<minor>\d+)
+\.
+(?<micro>\d+)
+(?<prerelease>[a-z]{1,2}\d+)
+"@
+
+
+function Download ($filename, $url) {
+    $webclient = [NAME_REDACTED]
+
+    $basedir = $pwd.Path + "\"
+    $filepath = $basedir + $filename
+    if (Test-Path $filename) {
+        Write-Host "Reusing" $filepath
+        return $filepath
+    }
+
+    # Download and retry up to 3 times in case of network transient errors.
+    Write-Host "Downloading" $filename "from" $url
+    $retry_attempts = 2
+    for ($i = 0; $i -lt $retry_attempts; $i++) {
+        try {
+            $webclient.DownloadFile($url, $filepath)
+            break
+        }
+        Catch [Exception]{
+            Start-Sleep 1
+        }
+    }
+    if (Test-Path $filepath) {
+        Write-Host "File saved at" $filepath
+    } else {
+        # Retry once to get the error message if any at the last try
+        $webclient.DownloadFile($url, $filepath)
+    }
+    return $filepath
+}
+
+
+function ParsePythonVersion ($python_version) {
+    if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
+        return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
+                $matches.prerelease)
+    }
+    $version_obj = [version]$python_version
+    return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
+}
+
+
+function DownloadPython ($python_version, $platform_suffix) {
+    $major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
+
+    if (($major -le 2 -and $micro -eq 0) `
+        -or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
+        ) {
+        $dir = "$major.$minor"
+        $python_version = "$major.$minor$prerelease"
+    } else {
+        $dir = "$major.$minor.$micro"
+    }
+
+    if ($prerelease) {
+        if (($major -le 2) `
+            -or ($major -eq 3 -and $minor -eq 1) `
+            -or ($major -eq 3 -and $minor -eq 2) `
+            -or ($major -eq 3 -and $minor -eq 3) `
+            ) {
+            $dir = "$dir/prev"
+        }
+    }
+
+    if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
+        $ext = "msi"
+        if ($platform_suffix) {
+            $platform_suffix = ".$platform_suffix"
+        }
+    } else {
+        $ext = "exe"
+        if ($platform_suffix) {
+            $platform_suffix = "-$platform_suffix"
+        }
+    }
+
+    $filename = "python-$python_version$platform_suffix.$ext"
+    $url = "$BASE_URL$dir/$filename"
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallPython ($python_version, $architecture, $python_home) {
+    Write-Host "[NAME_REDACTED]" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = ""
+    } else {
+        $platform_suffix = "amd64"
+    }
+    $installer_path = DownloadPython $python_version $platform_suffix
+    $installer_ext = [System.IO.Path]::GetExtension($installer_path)
+    Write-Host "Installing $installer_path to $python_home"
+    $install_log = $python_home + ".log"
+    if ($installer_ext -eq '.msi') {
+        InstallPythonMSI $installer_path $python_home $install_log
+    } else {
+        InstallPythonEXE $installer_path $python_home $install_log
+    }
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallPythonEXE ($exepath, $python_home, $install_log) {
+    $install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
+    RunCommand $exepath $install_args
+}
+
+
+function InstallPythonMSI ($msipath, $python_home, $install_log) {
+    $install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
+    $uninstall_args = "/qn /x $msipath"
+    RunCommand "msiexec.exe" $install_args
+    if (-not(Test-Path $python_home)) {
+        Write-Host "Python seems to be installed else-where, reinstalling."
+        RunCommand "msiexec.exe" $uninstall_args
+        RunCommand "msiexec.exe" $install_args
+    }
+}
+
+function RunCommand ($command, $command_args) {
+    Write-Host $command $command_args
+    Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
+}
+
+
+function InstallPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $python_path = $python_home + "\python.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $webclient = [NAME_REDACTED]
+        $webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
+        Write-Host "Executing:" $python_path $GET_PIP_PATH
+        & $python_path $GET_PIP_PATH
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+
+function DownloadMiniconda ($python_version, $platform_suffix) {
+    if ($python_version -eq "3.4") {
+        $filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
+    } else {
+        $filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
+    }
+    $url = $MINICONDA_URL + $filename
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallMiniconda ($python_version, $architecture, $python_home) {
+    Write-Host "[NAME_REDACTED]" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = "x86"
+    } else {
+        $platform_suffix = "x86_64"
+    }
+    $filepath = DownloadMiniconda $python_version $platform_suffix
+    Write-Host "Installing" $filepath "to" $python_home
+    $install_log = $python_home + ".log"
+    $args = "/S /D=$python_home"
+    Write-Host $filepath $args
+    Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallMinicondaPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $conda_path = $python_home + "\Scripts\conda.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $args = "install --yes pip"
+        Write-Host $conda_path $args
+        Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+function main () {
+    InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
+    InstallPip $env:PYTHON
+}
+
+main
\ No newline at end of file

@@ -0,0 +1,61 @@
+# AppVeyor.yml from https://github.com/ogrisel/python-appveyor-demo
+# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+
+build: off
+
+environment:
+  matrix:
+    - PYTHON: "C:\\Python266-x64"
+      PYTHON_VERSION: "2.6.6"
+      PYTHON_ARCH: "64"
+      TOXENV: "py26"
+
+    - PYTHON: "C:\\Python27-x64"
+      PYTHON_VERSION: "2.7.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py27"
+
+    - PYTHON: "C:\\Python33-x64"
+      PYTHON_VERSION: "3.3.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py33"
+
+    - PYTHON: "C:\\Python34-x64"
+      PYTHON_VERSION: "3.4.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py34"
+
+    - PYTHON: "C:\\Python35-x64"
+      PYTHON_VERSION: "3.5.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py35"
+
+    - PYTHON: "C:\\Python36-x64"
+      PYTHON_VERSION: "3.6.x"
+      PYTHON_ARCH: "64"
+      TOXENV: "py36"
+
+install:
+  # [NAME_REDACTED] (from the official .msi of http://python.org) and pip when
+  # not already installed.
+  - ps: if (-not(Test-Path($env:PYTHON))) { & _appveyor\install.ps1 }
+
+  # Prepend newly installed Python to the PATH of this build (this cannot be
+  # done from inside the powershell script as it would require to restart
+  # the parent CMD process).
+  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+
+  # Check that we have the expected version and architecture for Python
+  - "python --version"
+  - "python -c \"import struct; print(struct.calcsize('P') * 8)\""
+
+  # Upgrade to the latest version of pip to avoid it displaying warnings
+  # about it being out of date.
+  - "pip install --disable-pip-version-check --user --upgrade pip"
+  - "mingw32-make"
+
+test_script:
+  - "mingw32-make coverage"
+
+on_success:
+  - "pipenv run codecov"

Looks good to me if [USER_REDACTED] is happy.

Didn't we drop support for 2.6?

Is 3.6 not supported on AppVeyor?

Not until 3.0 was the decision I believe.

Correct.

It should be, this was just an oversight.

Ok, added


improve proxy bypass on Windows

See #3976

@@ -15,6 +15,8 @@ Release History
   resolution on Windows.
 - Added ``win_inet_pton`` as conditional dependency for the ``[socks]`` extra
   on Windows with Python 2.7.
+- Changed the proxy bypass implementation on Windows: the proxy bypass
+  check doesn't use forward and reverse DNS requests anymore

 2.13.0 (2017-01-24)
 +++++++++++++++++++

@@ -37,7 +37,9 @@
 # ---------

 if is_py2:
-    from urllib import quote, unquote, quote_plus, unquote_plus, urlencode, getproxies, proxy_bypass
+    from urllib import (
+        quote, unquote, quote_plus, unquote_plus, urlencode, getproxies,
+        proxy_bypass, proxy_bypass_environment, getproxies_environment)
     from urlparse import urlparse, urlunparse, urljoin, urlsplit, urldefrag
     from urllib2 import parse_http_list
     import cookielib
@@ -54,7 +56,7 @@

 elif is_py3:
     from urllib.parse import urlparse, urlunparse, urljoin, urlsplit, urlencode, quote, unquote, quote_plus, unquote_plus, urldefrag
-    from urllib.request import parse_http_list, getproxies, proxy_bypass
+    from urllib.request import parse_http_list, getproxies, proxy_bypass, proxy_bypass_environment, getproxies_environment
     from http import cookiejar as cookielib
     from http.cookies import Morsel
     from io import StringIO

@@ -8,12 +8,9 @@
 """

 import collections
-import time

 from .compat import OrderedDict

-current_time = getattr(time, 'monotonic', time.time)
-

 class CaseInsensitiveDict(collections.MutableMapping):
     """A case-insensitive ``dict``-like object.
@@ -106,89 +103,3 @@ def __getitem__(self, key):

     def get(self, key, default=None):
         return self.__dict__.get(key, default)
-
-
-class TimedCacheManaged(object):
-    """
-    Wrap a function call in a timed cache
-    """
-    def __init__(self, fnc):
-        self.fnc = fnc
-        self.cache = TimedCache()
-
-    def __call__(self, *args, **kwargs):
-        key = args[0]
-        found = None
-        try:
-            found = self.cache[key]
-        except KeyError:
-            found = self.fnc(key, **kwargs)
-            self.cache[key] = found
-
-        return found
-
-
-class TimedCache(collections.MutableMapping):
-    """
-    Evicts entries after expiration_secs. If none are expired and maxlen is hit,
-    will evict the oldest cached entry
-    """
-    def __init__(self, maxlen=32, expiration_secs=60):
-        """
-        :param maxlen: most number of entries to hold on to
-        :param expiration_secs: the number of seconds to hold on
-        to entries
-        """
-        self.maxlen = maxlen
-        self.expiration_secs = expiration_secs
-        self._dict = OrderedDict()
-
-    def __repr__(self):
-        return '<TimedCache maxlen:%d len:%d expiration_secs:%d>' % \
-            (self.maxlen, len(self._dict), self.expiration_secs)
-
-    def __iter__(self):
-        return ((key, value[1]) for key, value in self._dict.items())
-
-    def __delitem__(self, item):
-        del self._dict[item]
-
-    def __getitem__(self, key):
-        """
-        Look up an item in the cache. If the item
-        has already expired, it will be invalidated and not returned
-
-        :param key: which entry to look up
-        :return: the value in the cache, or None
-        """
-        occurred, value = self._dict[key]
-        now = int(current_time())
-
-        if now - occurred > self.expiration_secs:
-            del self._dict[key]
-            raise KeyError(key)
-        else:
-            return value
-
-    def __setitem__(self, key, value):
-        """
-        Locates the value at lookup key, if cache is full, will evict the
-        oldest entry
-
-        :param key: the key to search the cache for
-        :param value: the value to be added to the cache
-        """
-        now = int(current_time())
-
-        while len(self._dict) >= self.maxlen:
-            self._dict.popitem(last=False)
-
-        self._dict[key] = (now, value)
-
-    def __len__(self):
-        """:return: the length of the cache"""
-        return len(self._dict)
-
-    def clear(self):
-        """Clears the cache"""
-        return self._dict.clear()

@@ -14,6 +14,7 @@
 import contextlib
 import io
 import os
+import platform
 import re
 import socket
 import struct
@@ -26,9 +27,10 @@
 from .compat import parse_http_list as _parse_list_header
 from .compat import (
     quote, urlparse, bytes, str, OrderedDict, unquote, getproxies,
-    proxy_bypass, urlunparse, basestring, integer_types)
+    proxy_bypass, urlunparse, basestring, integer_types, is_py3,
+    proxy_bypass_environment, getproxies_environment)
 from .cookies import cookiejar_from_dict
-from .structures import CaseInsensitiveDict, TimedCacheManaged
+from .structures import CaseInsensitiveDict
 from .exceptions import (
     InvalidURL, InvalidHeader, FileModeWarning, UnrewindableBodyError)

@@ -37,6 +39,54 @@
 DEFAULT_CA_BUNDLE_PATH = certs.where()

+if platform.system() == 'Windows':
+    # provide a proxy_bypass version on Windows without DNS lookups
+
+    def proxy_bypass_registry(host):
+        if is_py3:
+            import winreg
+        else:
+            import _winreg as winreg
+        try:
+            internetSettings = winreg.OpenKey(winreg.HKEY_CURRENT_USER,
+                r'Software\Microsoft\Windows\CurrentVersion\[NAME_REDACTED]')
+            proxyEnable = winreg.QueryValueEx(internetSettings,
+                                              'ProxyEnable')[0]
+            proxyOverride = winreg.QueryValueEx(internetSettings,
+                                                'ProxyOverride')[0]
+        except OSError:
+            return False
+        if not proxyEnable or not proxyOverride:
+            return False
+
+        # make a check value list from the registry entry: replace the
+        # '<local>' string by [NAME_REDACTED] the corresponding
+        # canonical entry.
+        proxyOverride = proxyOverride.split(';')
+        # now check if we match one of the registry values.
+        for test in proxyOverride:
+            if test == '<local>':
+                if '.' not in host:
+                    return True
+            test = test.replace(".", r"\.")     # mask dots
+            test = test.replace("*", r".*")     # change glob sequence
+            test = test.replace("?", r".")      # change glob char
+            if re.match(test, host, re.I):
+                return True
+        return False
+
+    def proxy_bypass(host):  # noqa
+        """[NAME_REDACTED], if the host should be bypassed.
+
+        Checks proxy settings gathered from the environment, if specified,
+        or the registry.
+        """
+        if getproxies_environment():
+            return proxy_bypass_environment(host)
+        else:
+            return proxy_bypass_registry(host)
+
+
 def dict_to_sequence(d):
     """Returns an internal sequence dictionary update."""

@@ -577,16 +627,6 @@ def set_environ(env_name, value):
             os.environ[env_name] = old_value

-[USER_REDACTED]
-def _proxy_bypass_cached(netloc):
-    """
-    Looks for netloc in the cache, if not found, will call proxy_bypass
-    for the netloc and store its result in the cache
-
-    :rtype: bool
-    """
-    return proxy_bypass(netloc)
-
 def should_bypass_proxies(url, no_proxy):
     """
     Returns whether we should bypass proxies or not.
@@ -634,7 +674,7 @@ def should_bypass_proxies(url, no_proxy):
     # legitimate problems.
     with set_environ('no_proxy', no_proxy_arg):
         try:
-            bypass = _proxy_bypass_cached(netloc)
+            bypass = proxy_bypass(netloc)
         except (TypeError, socket.gaierror):
             bypass = False

@@ -2,7 +2,7 @@

 import pytest

-from requests.structures import CaseInsensitiveDict, LookupDict, TimedCache, TimedCacheManaged
+from requests.structures import CaseInsensitiveDict, LookupDict

 class TestCaseInsensitiveDict:
@@ -74,75 +74,3 @@ def test_getitem(self, key, value):
     [USER_REDACTED]_item_parameters
     def test_get(self, key, value):
         assert self.lookup_dict.get(key) == value
-
-
-class TestTimedCache(object):
-    [USER_REDACTED].fixture(autouse=True)
-    def setup(self):
-        self.any_value = 'some value'
-        self.expiration_secs = 60
-        self.cache = TimedCache(expiration_secs=self.expiration_secs)
-        yield
-        self.cache.clear()
-
-    def test_get(self):
-        self.cache['a'] = self.any_value
-        assert self.cache['a'] is self.any_value
-
-    def test_repr(self):
-        repr = str(self.cache)
-        assert repr == '<TimedCache maxlen:32 len:0 expiration_secs:60>'
-
-    def test_get_expired_item(self, mocker):
-        self.cache = TimedCache(maxlen=1, expiration_secs=self.expiration_secs)
-
-        mocker.patch('requests.structures.current_time', lambda: 0)
-        self.cache['a'] = self.any_value
-        mocker.patch('requests.structures.current_time', lambda: self.expiration_secs + 1)
-        assert self.cache.get('a') is None
-
-    def test_evict_first_entry_when_full(self, mocker):
-        self.cache = TimedCache(maxlen=2, expiration_secs=2)
-        mocker.patch('requests.structures.current_time', lambda: 0)
-        self.cache['a'] = self.any_value
-        mocker.patch('requests.structures.current_time', lambda: 1)
-        self.cache['b'] = self.any_value
-        mocker.patch('requests.structures.current_time', lambda: 3)
-        self.cache['c'] = self.any_value
-        assert len(self.cache) is 2
-        with pytest.raises(KeyError, message='Expected key not found'):
-            self.cache['a']
-        assert self.cache['b'] is self.any_value
-        assert self.cache['c'] is self.any_value
-
-    def test_delete_item_removes_item(self):
-        self.cache['a'] = self.any_value
-        del self.cache['a']
-        with pytest.raises(KeyError, message='Expected key not found'):
-            self.cache['a']
-
-    def test_iterating_hides_timestamps(self):
-        self.cache['a'] = 1
-        self.cache['b'] = 2
-        expected = [('a', 1), ('b', 2)]
-        actual = [(key, val) for key, val in self.cache]
-        assert expected == actual
-
-
-class TestTimedCacheManagedDecorator(object):
-    def test_caches_repeated_calls(self, mocker):
-        mocker.patch('requests.structures.current_time', lambda: 0)
-
-        nonlocals = {'value': 0}
-
-        [USER_REDACTED]
-        def some_method(x):
-            nonlocals['value'] = nonlocals['value'] + x
-            return nonlocals['value']
-
-        first_result = some_method(1)
-        assert first_result is 1
-        second_result = some_method(1)
-        assert second_result is 1
-        third_result = some_method(2)
-        assert third_result is 3

@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-

+import os
 from io import BytesIO

 import pytest
@@ -599,3 +600,55 @@ def test_should_bypass_proxies_no_proxy(
     no_proxy = '192.168.0.0/24,127.0.0.1,localhost.localdomain,172.16.1.1'
     # Test 'no_proxy' argument
     assert should_bypass_proxies(url, no_proxy=no_proxy) == expected
+
+
+[USER_REDACTED].mark.skipif(os.name != 'nt', reason='Test only on Windows')
+[USER_REDACTED].mark.parametrize(
+    'url, expected, override', (
+            ('http://192.168.0.1:5000/', True, None),
+            ('http://192.168.0.1/', True, None),
+            ('http://172.16.1.1/', True, None),
+            ('http://172.16.1.1:5000/', True, None),
+            ('http://localhost.localdomain:5000/v1.0/', True, None),
+            ('http://172.16.1.22/', False, None),
+            ('http://172.16.1.22:5000/', False, None),
+            ('http://google.com:5000/v1.0/', False, None),
+            ('http://mylocalhostname:5000/v1.0/', True, '<local>'),
+            ('http://192.168.0.1/', False, ''),
+    ))
+def test_should_bypass_proxies_win_registry(url, expected, override,
+                                            monkeypatch):
+    """Tests for function should_bypass_proxies to check if proxy
+    can be bypassed or not with Windows registry settings
+    """
+    if override is None:
+        override = '192.168.*;127.0.0.1;localhost.localdomain;172.16.1.1'
+    if compat.is_py3:
+        import winreg
+    else:
+        import _winreg as winreg
+
+    class RegHandle:
+        def Close(self):
+            pass
+
+    ie_settings = RegHandle()
+
+    def OpenKey(key, subkey):
+        return ie_settings
+
+    def QueryValueEx(key, value_name):
+        if key is ie_settings:
+            if value_name == 'ProxyEnable':
+                return [1]
+            elif value_name == 'ProxyOverride':
+                return [override]
+
+    monkeypatch.setenv('http_proxy', '')
+    monkeypatch.setenv('https_proxy', '')
+    monkeypatch.setenv('ftp_proxy', '')
+    monkeypatch.setenv('no_proxy', '')
+    monkeypatch.setenv('NO_PROXY', '')
+    monkeypatch.setattr(winreg, 'OpenKey', OpenKey)
+    monkeypatch.setattr(winreg, 'QueryValueEx', QueryValueEx)
+    assert should_bypass_proxies(url, no_proxy=None) == expected

Hrm, we may need to enable appveyor to ensure that this stuff actually works.

This looks like a good start [USER_REDACTED]! I left a couple notes with some outstanding questions from my end. A couple are just stylistic nits but the others are on functionality.

Cool, so this is looking really good! I'd love to see if we can get the test coverage up a bit higher, and I've left a note in the diff. :smile:

Ok, I think I'm happy. Time to see what [USER_REDACTED] thinks.

Looks good to me!

Ok then, one last request. Can you add notes to the changelog to note that a) we've changed our strategy for looking up proxy bypass information on Windows, and b) we've removed the proxy bypass cache as it's no longer needed? Just want to make sure we have a good reference in case these things have unforseen bugs, and I bet I'll forget that there were two things in this PR when it comes to writing the changelog later. :grin:

Alright, let's do it!

This comment is confusing: what is it trying to say?

We may as well just use booleans.

What is this for?

We can probably unify these two lines.

May as well just import this directly from compat.

Then we would need a platform check in compat and winreg could be `None` on non Windows platforms...

Is there a reason this is `1` and not `True` like the other return values? If so, we should probably add a comment explaining why.

It may be better to follow something closer to the import convention of the other modules, rather than use `\`.
```python
from urllib import (
    quote, unquote, quote_plus, unquote_plus, urlencode, getproxies,
    proxy_bypass, proxy_bypass_environment, getproxies_environment)
```

Could we remove this extra line here.

This seems to return a boolean but the docstring states it returns a dictionary of scheme to proxy mappings. It may be a good idea to clarify this case.

Nope, I forgot this one 🙈

Somehow I copied the wrong docstring, I'll fix this :)

Rather than use `os.name`, might it be better to use `platform.system`?

[USER_REDACTED] Should I remove this one, I don't think this is really necessary...

Yeah, let's do that: there's no need to tolerate this `ImportError` I don't think.


fix handle of non-ascii location on redirects issue #3888

fix for #3888

@@ -12,7 +12,7 @@
 from datetime import datetime

 from .auth import _basic_auth_str
-from .compat import cookielib, OrderedDict, urljoin, urlparse
+from .compat import cookielib, is_py3, OrderedDict, urljoin, urlparse
 from .cookies import (
     cookiejar_from_dict, extract_cookies_to_jar, RequestsCookieJar, merge_cookies)
 from .models import Request, PreparedRequest, DEFAULT_REDIRECT_LIMIT
@@ -90,7 +90,16 @@ class SessionRedirectMixin(object):
     def get_redirect_target(self, resp):
         """Receives a Response. Returns a redirect URI or ``None``"""
         if resp.is_redirect:
-            return resp.headers['location']
+            location = resp.headers['location']
+            # Currently the underlying http module on py3 decode headers
+            # in latin1, but empirical evidence suggests that latin1 is very
+            # rarely used with non-ASCII characters in HTTP headers.
+            # It is more likely to get UTF8 header rather than latin1.
+            # This causes incorrect handling of UTF8 encoded location headers.
+            # To solve this, we re-encode the location in latin1.
+            if is_py3:
+                location = location.encode('latin1')
+            return to_native_string(location, 'utf8')
         return None

     def resolve_redirects(self, resp, req, stream=False, timeout=None,

@@ -3,7 +3,6 @@
 import pytest
 import threading
 import requests
-from requests.compat import quote, is_py3

 from tests.testserver.server import Server, consume_socket_content

@@ -208,8 +207,7 @@ def test_use_proxy_from_environment(httpbin, var, scheme):

 def test_redirect_rfc1808_to_non_ascii_location():
     path = u'š'
-    expected_path = quote(path.encode('utf8')).encode('ascii')
-    expected_path_py3 = b'%C3%85%C2%A1'
+    expected_path = b'%C5%A1'
     redirect_request = []  # stores the second request to the server

     def redirect_resp_handler(sock):
@@ -233,11 +231,7 @@ def redirect_resp_handler(sock):
         assert r.status_code == 200
         assert len(r.history) == 1
         assert r.history[0].status_code == 301
-
-        # currently Python3 not handling non-ASCII redirects (issue #3888)
-        if is_py3:
-            assert redirect_request[0].startswith(b'GET /' + expected_path_py3 + b' HTTP/1.1')
-        else:
-            assert redirect_request[0].startswith(b'GET /' + expected_path + b' HTTP/1.1')
+        assert redirect_request[0].startswith(b'GET /' + expected_path + b' HTTP/1.1')
+        assert r.url == u'{0}/{1}'.format(url, expected_path.decode('ascii'))

         close_server.set()

Cool, so in general I'm happy with this. I have some notes about the code comments, and I'd also like [USER_REDACTED] and [USER_REDACTED] to weigh in if they can.

For the most part this looks good to me 😊 I left a couple comments on the test but otherwise, I think things are good. Thanks for putting this together [USER_REDACTED]!

Cool, so I'm happy!

s/underling/underlying/

So I think this comment needs to have a bit more elaboration. Specifically, we need to talk about why we think we're more likely to get UTF-8 encoded location headers than Latin-1: specifically, because empirical evidence suggests that latin-1 is very rarely used with non-ASCII characters in HTTP headers.

It seems like it might be more resilient/explicit to hard code the expected path here? It would divest us from relying on `quote` in the test, saves us an import, and hopefully clear up this line a bit.

At the very least, a comment on what we're doing here seems good.

We're kind of testing the test here. Perhaps also adding a check that `r.url` is what we expect would be helpful?

I agree with [USER_REDACTED]. Also this is a minor nit, but capitals and periods would be great here 😊

added more info about the issue.

yes :)

added capitals and periods [USER_REDACTED], im not native english so feel free to help me rephrase it.

agree changed

agree, added assert for url


fix unicode decode error on py2 when handling redirect without scheme

as discussed on #3959 , fix the UnicodeDecodeError happening because implicit decoding of str string into unicode while using python2.

This issue happens when handling redirection without scheme  (i.e. Location: //URL) on python2 and only if the location is unicode.

First time applying a PR on GitHub so let me know if there is something I can do to make it better :).

Thanks,
Shmulik

@@ -179,4 +179,4 @@ Patches and Suggestions
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
-
+- [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])

@@ -122,7 +122,7 @@ def resolve_redirects(self, resp, req, stream=False, timeout=None,
             # Handle redirection without scheme (see: RFC 1808 Section 4)
             if url.startswith('//'):
                 parsed_rurl = urlparse(resp.url)
-                url = '%s:%s' % (parsed_rurl.scheme, url)
+                url = '%s:%s' % (to_native_string(parsed_rurl.scheme), url)

             # The scheme should be lower case...
             parsed = urlparse(url)

@@ -1,9 +1,9 @@
 # -*- coding: utf-8 -*-

-import os
 import pytest
 import threading
 import requests
+from requests.compat import quote, is_py3

 from tests.testserver.server import Server, consume_socket_content

@@ -204,3 +204,40 @@ def test_use_proxy_from_environment(httpbin, var, scheme):

         # it had actual content (not checking for SOCKS protocol for now)
         assert len(fake_proxy.handler_results[0]) > 0
+
+
+def test_redirect_rfc1808_to_non_ascii_location():
+    path = u'š'
+    expected_path = quote(path.encode('utf8')).encode('ascii')
+    expected_path_py3 = b'%C3%85%C2%A1'
+    redirect_request = []  # stores the second request to the server
+
+    def redirect_resp_handler(sock):
+        consume_socket_content(sock, timeout=0.5)
+        location = u'//{0}:{1}/{2}'.format(host, port, path)
+        sock.send(
+            b'HTTP/1.1 301 Moved Permanently\r\n'
+            b'Content-Length: 0\r\n'
+            b'Location: ' + location.encode('utf8') + b'\r\n'
+            b'\r\n'
+        )
+        redirect_request.append(consume_socket_content(sock, timeout=0.5))
+        sock.send(b'HTTP/1.1 200 OK\r\n\r\n')
+
+    close_server = threading.Event()
+    server = Server(redirect_resp_handler, wait_to_close_event=close_server)
+
+    with server as (host, port):
+        url = u'http://{0}:{1}'.format(host, port)
+        r = requests.get(url=url, allow_redirects=True)
+        assert r.status_code == 200
+        assert len(r.history) == 1
+        assert r.history[0].status_code == 301
+
+        # currently Python3 not handling non-ASCII redirects (issue #3888)
+        if is_py3:
+            assert redirect_request[0].startswith(b'GET /' + expected_path_py3 + b' HTTP/1.1')
+        else:
+            assert redirect_request[0].startswith(b'GET /' + expected_path + b' HTTP/1.1')
+
+        close_server.set()

Hey [USER_REDACTED], thanks so much for opening your first PR with Requests! I've left a couple notes inline on things that'll need to be fixed, but this is a really good start. Let us know if you have any questions.

Agreed, this is a good start! Just a few extra notes on top of [USER_REDACTED]'s review.

Ok cool, I think I'm happy. Thanks for working with us through the code review. This is a nice improvement!

[USER_REDACTED] could we confirm that the redirected URL is what we expect here?

Unfortunately, neither `%` or `format` work on byte strings in Python 3 prior to 3.5. This means this string will need to be constructed another way. Either using `join` on multiple byte strings, or the `+` operator will probably be the best way to approach this for all Python versions.

Any reason this shouldn't be a get?

Hey [USER_REDACTED] thanks for the quick review!

I would be glad to make it happen but the issue is that current decoding into native string of headers makes Location header on python2 encoded string and on python3 decoded string. so this test won't be consistent across python version.

before this PR we get for the same test the following redirected URL:
python3: http://localhost:58792/?%C3%85%C2%A0
python2: UnicodeDecodeError (due to implicit decoding into unicode on string formatting)

this PR only fixes the python2 UnicodeDecodeError (python3 behaves the same).

there is an encoding issue #3888 which leads to inconsistency between python versions as each python version handles different type of strings.

I agree that this PR is far from solving #3888 (which I'll be happy to discuss about possible solutions).

My ideas are (just note a few, regardless of order):

1. keep this PR as is
2. add xfail for this test
3. test only `the resolve_redirects()` function
4. discuss possible for solution for #3888 that I might implement in the same PR
5. leave all as is and ignore this PR

of course I am open to any suggestions :smile:

Thanks for the suggestions, [USER_REDACTED]! It's a bit unfortunate that we're further exposing the inconsistent functionality between versions by [NAME_REDACTED], this fixes the problem it's intended to address so I guess we can pass on this check until #3888 is fixed.

I'd be ok adding a check on Python version to let us make the correct assertion.

add the assertion for url.
please let me know if there is a better way or more common way to ignore this assertion for python3.

please let me know if any other changes needed :smile:

Right now we should probably assert what Python 3 *does* do, even if it's wrong, and just put a note on that assertion to fix it up when we change #3888.

agree, changed it

Ok, one last note. It's slightly nicer to save this into a mutable object in the enclosing scope (e.g. a list) and then put the assertion in the enclosing function rather than the socket handler. assertions in the socket handler *do* work, but it's usually nicer to let the socket handler run to completion rather than blow it up and possibly leak it.

once again this is probably better :smile: . I hope I understand you correctly (changed it again).


HTTPAdapter.send() timeout defaults

This PR allows to use urllib3 ``Timeout`` objects as ``timeout`` argument in ``HTTPAdapter.send()``.

Besides it omits the ``timeout`` argument when calling ``conn.urlopen()`` if ``HTTPAdapter.send()`` is called without a timeout. This allows setting a default timeout at connection pool level:
`adapter.poolmanager.connection_pool_kw['timeout'] = urllib3.Timeout(...)`

@@ -391,7 +391,7 @@ def send(self, request, stream=False, timeout=None, verify=True, cert=None, prox
         :param timeout: (optional) How long to wait for the server to send
             data before giving up, as a float, or a :ref:`(connect timeout,
             read timeout) <timeouts>` tuple.
-        :type timeout: float or tuple
+        :type timeout: float or tuple or urllib3 Timeout object
         :param verify: (optional) Either a boolean, in which case it controls whether
             we verify the server's TLS certificate, or a string, in which case it
             must be a path to a CA bundle to use
@@ -418,6 +418,8 @@ def send(self, request, stream=False, timeout=None, verify=True, cert=None, prox
                        "timeout tuple, or a single float to set "
                        "both timeouts to the same value".format(timeout))
                 raise ValueError(err)
+        elif isinstance(timeout, TimeoutSauce):
+            pass
         else:
             timeout = TimeoutSauce(connect=timeout, read=timeout)

@@ -33,6 +33,7 @@

 from .compat import StringIO, u
 from .utils import override_environ
+from requests.packages.urllib3.util import Timeout as Urllib3Timeout

 # Requests to this URL should always fail with a connection timeout (nothing
 # listening on that port)
@@ -2013,7 +2014,12 @@ def test_invalid_timeout(self, httpbin, timeout, error_text):
             requests.get(httpbin('get'), timeout=timeout)
         assert error_text in str(e)

-    def test_none_timeout(self, httpbin):
+    [USER_REDACTED].mark.parametrize(
+        'timeout', (
+            None,
+            Urllib3Timeout(connect=None, read=None)
+        ))
+    def test_none_timeout(self, httpbin, timeout):
         """Check that you can set None as a valid timeout value.

         To actually test this behavior, we'd want to check that setting the
@@ -2022,27 +2028,42 @@ def test_none_timeout(self, httpbin):
         Instead we verify that setting the timeout to None does not prevent the
         request from succeeding.
         """
-        r = requests.get(httpbin('get'), timeout=None)
+        r = requests.get(httpbin('get'), timeout=timeout)
         assert r.status_code == 200

-    def test_read_timeout(self, httpbin):
+    [USER_REDACTED].mark.parametrize(
+        'timeout', (
+            (None, 0.1),
+            Urllib3Timeout(connect=None, read=0.1)
+        ))
+    def test_read_timeout(self, httpbin, timeout):
         try:
-            requests.get(httpbin('delay/10'), timeout=(None, 0.1))
+            requests.get(httpbin('delay/10'), timeout=timeout)
             pytest.fail('The recv() request should time out.')
         except ReadTimeout:
             pass

-    def test_connect_timeout(self):
+    [USER_REDACTED].mark.parametrize(
+        'timeout', (
+            (0.1, None),
+            Urllib3Timeout(connect=0.1, read=None)
+        ))
+    def test_connect_timeout(self, timeout):
         try:
-            requests.get(TARPIT, timeout=(0.1, None))
+            requests.get(TARPIT, timeout=timeout)
             pytest.fail('The connect() request should time out.')
         except ConnectTimeout as e:
             assert isinstance(e, ConnectionError)
             assert isinstance(e, Timeout)

-    def test_total_timeout_connect(self):
+    [USER_REDACTED].mark.parametrize(
+        'timeout', (
+            (0.1, 0.1),
+            Urllib3Timeout(connect=0.1, read=0.1)
+        ))
+    def test_total_timeout_connect(self, timeout):
         try:
-            requests.get(TARPIT, timeout=(0.1, 0.1))
+            requests.get(TARPIT, timeout=timeout)
             pytest.fail('The connect() request should time out.')
         except ConnectTimeout:
             pass

Thanks for this! I've left some notes in the diff.

Cool, this is really good. I'd absolutely love it if we could come up with a test for this as well so that we can cover that line of code. Do you mind writing one?

Good idea! However, rather than copy the tests, we should use [py.test's parametrizing fixture](https://docs.pytest.org/en/latest/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions) to handle this, which will reduce the amount of duplicate code we write, and indicate very clearly to the person reading the tests that these two things are expected to behave *identically*.

Great, this looks really good! Thanks for dealing with the code review process. I'll merge this assuming all the tests pass. :sparkles: :cake: :sparkles:

Can this be moved lower down the list? A timeout object isn't a tuple, so the original `isinstance(tuple)` check won't catch it, so let's try to avoid rearranging the code too much.

Unfortunately, this change is breaking. You'll either need to remove the default timeout change, or move this PR to the 3.0 branch.

You're also welcome to split this PR into two parts, the first bit making just the non-breaking changes against the master branch and the second bit making the breaking ones against 3.0.

Hmm. Is it really breaking?
Looking at urllib3, leaving out the timeout argument will result in calling ``connectionpool.HTTPConnectionPool._get_timeout(_Default)``, which will use ``Timeout.DEFAULT_TIMEOUT`` if the ``HTTPConnectionPool`` had been initialized without a timeout. And ``Timeout.from_float(DEFAULT_TIMEOUT)`` yields the same result as if it had been called with ``None``: https://github.com/shazow/urllib3/blob/master/urllib3/util/timeout.py#L117

So the results should be the same...

Nope, they aren't the same. They both validate to the identity, but that identity is then used as the actual timeout. Note: https://github.com/shazow/urllib3/blob/master/urllib3/util/timeout.py#L33

Ah - ok. Your are right.
I removed the change for ``timeout`` is ``None``.

Ah, so the tests are failing because you've aliased this name (`Timeout`) with the name of an exception. Might be worth changing this to `from requests.packages.urllib3.util import Timeout as Urllib3Timeout` or something similar.


Raise IOError when providing an invalid path to a CA bundle or …

Use IOError whenever invalid paths are used for ssl certificates (#3926)

@@ -64,7 +64,9 @@ def send(self, request, stream=False, timeout=None, verify=True,
             data before giving up, as a float, or a :ref:`(connect timeout,
             read timeout) <timeouts>` tuple.
         :type timeout: float or tuple
-        :param verify: (optional) Whether to verify SSL certificates.
+        :param verify: (optional) Either a boolean, in which case it controls whether we verify
+            the server's TLS certificate, or a string, in which case it must be a path
+            to a CA bundle to use
         :param cert: (optional) Any user-provided SSL certificate to be trusted.
         :param proxies: (optional) The proxies dictionary to apply to the request.
         """
@@ -202,7 +204,9 @@ def cert_verify(self, conn, url, verify, cert):

         :param conn: The urllib3 connection object associated with the cert.
         :param url: The requested URL.
-        :param verify: Whether we should actually verify the certificate.
+        :param verify: Either a boolean, in which case it controls whether we verify
+            the server's TLS certificate, or a string, in which case it must be a path
+            to a CA bundle to use
         :param cert: The SSL certificate to verify.
         """
         if url.lower().startswith('https') and verify:
@@ -216,8 +220,9 @@ def cert_verify(self, conn, url, verify, cert):
             if not cert_loc:
                 cert_loc = DEFAULT_CA_BUNDLE_PATH

-            if not cert_loc:
-                raise Exception("Could not find a suitable SSL CA certificate bundle.")
+            if not cert_loc or not os.path.exists(cert_loc):
+                raise IOError("Could not find a suitable TLS CA certificate bundle, "
+                              "invalid path: {0}".format(cert_loc))

             conn.cert_reqs = 'CERT_REQUIRED'

@@ -236,6 +241,12 @@ def cert_verify(self, conn, url, verify, cert):
                 conn.key_file = cert[1]
             else:
                 conn.cert_file = cert
+            if conn.cert_file and not os.path.exists(conn.cert_file):
+                raise IOError("Could not find the TLS certificate file, "
+                              "invalid path: {0}".format(conn.cert_file))
+            if conn.key_file and not os.path.exists(conn.key_file):
+                raise IOError("Could not find the TLS key file, "
+                              "invalid path: {0}".format(conn.key_file))

     def build_response(self, req, resp):
         """Builds a :class:`Response <requests.Response>` object from a urllib3
@@ -381,7 +392,9 @@ def send(self, request, stream=False, timeout=None, verify=True, cert=None, prox
             data before giving up, as a float, or a :ref:`(connect timeout,
             read timeout) <timeouts>` tuple.
         :type timeout: float or tuple
-        :param verify: (optional) Whether to verify SSL certificates.
+        :param verify: (optional) Either a boolean, in which case it controls whether
+            we verify the server's TLS certificate, or a string, in which case it
+            must be a path to a CA bundle to use
         :param cert: (optional) Any user-provided SSL certificate to be trusted.
         :param proxies: (optional) The proxies dictionary to apply to the request.
         :rtype: requests.Response

@@ -36,7 +36,9 @@ def request(method, url, **kwargs):
     :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
     :type allow_redirects: bool
     :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
-    :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
+    :param verify: (optional) Either a boolean, in which case it controls whether we verify
+            the server's TLS certificate, or a string, in which case it must be a path
+            to a CA bundle to use. Defaults to ``True``.
     :param stream: (optional) if ``False``, the response content will be immediately downloaded.
     :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
     :return: :class:`Response <Response>` object

@@ -460,8 +460,9 @@ def request(self, method, url,
             hostname to the URL of the proxy.
         :param stream: (optional) whether to immediately download the response
             content. Defaults to ``False``.
-        :param verify: (optional) whether the SSL cert will be verified.
-            A CA_BUNDLE path can also be provided. Defaults to ``True``.
+        :param verify: (optional) Either a boolean, in which case it controls whether we verify
+            the server's TLS certificate, or a string, in which case it must be a path
+            to a CA bundle to use. Defaults to ``True``.
         :param cert: (optional) if String, path to ssl client cert file (.pem).
             If Tuple, ('cert', 'key') pair.
         :rtype: requests.Response

@@ -769,6 +769,22 @@ def test_unicode_header_name(self, httpbin):
     def test_pyopenssl_redirect(self, httpbin_secure, httpbin_ca_bundle):
         requests.get(httpbin_secure('status', '301'), verify=httpbin_ca_bundle)

+    def test_invalid_ca_certificate_path(self, httpbin_secure):
+        INVALID_PATH = '/garbage'
+        with pytest.raises(IOError) as e:
+            requests.get(httpbin_secure(), verify=INVALID_PATH)
+        assert str(e.value) == 'Could not find a suitable TLS CA certificate bundle, invalid path: {0}'.format(INVALID_PATH)
+
+    def test_invalid_ssl_certificate_files(self, httpbin_secure):
+        INVALID_PATH = '/garbage'
+        with pytest.raises(IOError) as e:
+            requests.get(httpbin_secure(), cert=INVALID_PATH)
+        assert str(e.value) == 'Could not find the TLS certificate file, invalid path: {0}'.format(INVALID_PATH)
+
+        with pytest.raises(IOError) as e:
+            requests.get(httpbin_secure(), cert=('.', INVALID_PATH))
+        assert str(e.value) == 'Could not find the TLS key file, invalid path: {0}'.format(INVALID_PATH)
+
     def test_https_warnings(self, httpbin_secure, httpbin_ca_bundle):
         """warnings are emitted with requests.get"""
         if HAS_MODERN_SSL or HAS_PYOPENSSL:

So, *in general* this is reasonable. There are some concerns about "look before you leap" here: in an ideal world we'd try to catch the error and diagnose after the fact, but that doesn't work well. So I think I'm in general ok with this, modulo a few concerns I have inline.

Thanks for addressing the review! I have a few more smaller notes.

Ok, I think I'm happy with this. Thanks so much! :sparkles:

Why did this conditional get inverted?

You can't make this change in a backwards compatible way: the current code allows for tuples longer than length two.

So, this docstring needs to be reworded. Let's have it be: `Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use`.

For new exception strings, can we use the acronym `TLS` instead of the acronym `SSL`? I don't want to change the string in the block above because people may be checking the actual string message, but these ones should use the proper name for the protocol.

For these tests, can you use the `httpbin_secure` fixture like the tests above? We want to avoid making live web requests in our tests.


Rebase #3745 and add some tests

2 years after the original #2431, and after #3745
I encountered the strange behavior of `r.iter_lines()`, found this and tried to complete it.
Rebased once more and added more tests using the breakdown of [USER_REDACTED] https://github.com/[USER_REDACTED]nethreitz/requests/pull/2431#issuecomment-72333964

@@ -776,23 +776,46 @@ def iter_lines(self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=None, delimiter=

         .. note:: This method is not reentrant safe.
         """
-
         pending = None

-        for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
+        for chunk in self.iter_content(chunk_size=chunk_size,
+                                       decode_unicode=decode_unicode):
+            # Skip any null responses: if there is pending data it is necessarily an
+            # incomplete chunk, so if we don't have more data we don't want to bother
+            # trying to get it. Unconsumed pending data will be yielded anyway in the
+            # end of the loop if the stream ends.
+            if not chunk:
+                continue

+            # Consume any pending data
             if pending is not None:
                 chunk = pending + chunk
+                pending = None

+            # Either split on a line, or split on a specified delimiter
             if delimiter:
                 lines = chunk.split(delimiter)
             else:
                 lines = chunk.splitlines()

-            if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]:
+            # Calling `.split(delimiter)` will always end with whatever text
+            # remains beyond the delimiter, or '' if the delimiter is the end
+            # of the text.  On the other hand, `.splitlines()` doesn't include
+            # a '' if the text ends in a line delimiter.
+            #
+            # For example:
+            #
+            #     'abc\ndef\n'.split('\n')  ~> ['abc', 'def', '']
+            #     'abc\ndef\n'.splitlines() ~> ['abc', 'def']
+            #
+            # So if we have a specified delimiter, we always pop the final
+            # item and prepend it to the next chunk.
+            #
+            # If we're using `splitlines()`, we only do this if the chunk
+            # ended midway through a line.
+            incomplete_line = lines[-1] and lines[-1][-1] == chunk[-1]
+            if delimiter or incomplete_line:
                 pending = lines.pop()
-            else:
-                pending = None

             for line in lines:
                 yield line

@@ -1296,6 +1296,81 @@ def test_request_and_response_are_pickleable(self, httpbin):
         assert r.request.url == pr.request.url
         assert r.request.headers == pr.request.headers

+
+    def test_response_lines(self):
+        """
+        iter_lines should be able to handle data dribbling in which delimiters
+        might not be lined up ideally.
+        """
+        mock_chunks = [
+            'This \r\n',
+            '',
+            'is\r',
+            '\n',
+            'a',
+            ' ',
+            '',
+            '',
+            'test.',
+            '\r',
+            '\n',
+            'end.',
+        ]
+        mock_data = ''.join(mock_chunks)
+
+        mock_iter_content = lambda *args, **kwargs: (e for e in mock_chunks)
+
+        r = requests.Response()
+        r._content_consumed = True
+        r.iter_content = mock_iter_content
+
+        assert list(r.iter_lines(delimiter='\r\n')) == mock_data.split('\r\n')
+
+        # Because '\n' is a single line-end, when `iter_lines()` receives
+        # the chunks containing a single '\n', it emits '' as a line -- whereas
+        # `.splitlines()` combines with the '\r' and splits on `\r\n`.
+        result = list(r.iter_lines())
+        assert result != mock_data.splitlines()
+        assert result[2] == ''
+        assert result[4] == ''
+        # If we change all the line breaks to `\r`, we should be okay.
+        mock_chunks = [chunk.replace('\n', '\r') for chunk in mock_chunks]
+        mock_data = ''.join(mock_chunks)
+        assert list(r.iter_lines()) == mock_data.splitlines()
+
+
+    [USER_REDACTED].mark.parametrize(
+        'content, expected_no_delimiter, expected_delimiter', (
+            ([''], [], []),
+            (['line\n'], ['line'], ['line\n']),
+            (['line', '\n'], ['line'], ['line\n']),
+            (['line\r\n'], ['line'], ['line', '']),
+            # Empty chunk in the end of stream, same behavior as the previous
+            (['line\r\n', ''], ['line'], ['line', '']),
+            (['line', '\r\n'], ['line'], ['line', '']),
+            (['a\r', '\nb\r'], ['a', '', 'b'], ['a', 'b\r']),
+            (['a\n', '\nb'], ['a', '', 'b'], ['a\n\nb']),
+            (['a\r\n','\rb\n'], ['a', '', 'b'], ['a', '\rb\n']),
+            (['a\nb', 'c'], ['a', 'bc'], ['a\nbc']),
+            (['a\n', '\rb', '\r\nc'], ['a', '', 'b', 'c'], ['a\n\rb', 'c']),
+            (['a\r\nb', '', 'c'], ['a', 'bc'], ['a', 'bc'])  # Empty chunk with pending data
+        ))
+    def test_response_lines_parametrized(self, content, expected_no_delimiter, expected_delimiter):
+        """
+        Test a lot of potential chunk splits to ensure consistency of
+        iter_lines(delimiter=x), as well as the legacy behavior of
+        iter_lines() without delimiter
+        https://github.com/[USER_REDACTED]nethreitz/requests/pull/2431#issuecomment-72333964
+        """
+        mock_chunks = content
+        mock_iter_content = lambda *args, **kwargs: (e for e in mock_chunks)
+
+        r = requests.Response()
+        r._content_consumed = True
+        r.iter_content = mock_iter_content
+        assert list(r.iter_lines()) == expected_no_delimiter
+        assert list(r.iter_lines(delimiter='\r\n')) == expected_delimiter
+
     def test_prepared_request_is_pickleable(self, httpbin):
         p = requests.Request('GET', httpbin('get')).prepare()

@@ -1741,11 +1816,12 @@ def test_json_param_post_should_not_override_data_param(self, httpbin):
         prep = r.prepare()
         assert 'stuff=elixr' == prep.body

-    def test_response_iter_lines(self, httpbin):
+    [USER_REDACTED].mark.parametrize('decode_unicode', (True, False))
+    def test_response_iter_lines(self, httpbin, decode_unicode):
         r = requests.get(httpbin('stream/4'), stream=True)
         assert r.status_code == 200
-
-        it = r.iter_lines()
+        r.encoding = 'utf-8'
+        it = r.iter_lines(decode_unicode=decode_unicode)
         next(it)
         assert len(list(it)) == 3

Cool, so this generally looks really good! I have one small note.

Ok, I think I'm happy with this. Thanks for pushing this over the line! I'll try to dive into what's going on with the tests.

Ok, so the tests are failing because of a bytes/string incompatibility. Want to take a look?

Cool, we seem to have resolved everything outstanding. Nicely done!

Should this check move after the pending data logic?

You're right
Fixed and added tests for this behavior :ok_hand:

I don't know if it's licit to receive empty chunks in the middle of a stream, but I guess it's possible in theory. (and it would work)

Actually I replied too fast: if we want to return the same result on
`['line\r\n']` and  `['line\r\n', '']` (which seems right),
we need to keep this check on top

Otherwise we consume `pending`  and discard it

I think it's safe to discard empty chunks early, `pending` will be carried to the next non-empty chunk

> I don't know if it's licit to receive empty chunks in the middle of a stream, but I guess it's possible in theory. (and it would work)

urllib3 won't give them to us, but people like putting things that aren't urllib3 underneath us, and they might. So we should tolerate it.

As to your note about wanting to return the same result, I think that makes sense. But we should add a comment to say that "pending is necessarily an incomplete chunk, and so if we don't have more data we don't want to bother trying to get it".

Done 👌

So, to be clear, the error in this fix was assuming that we need to tolerate `str` here. The tests were in error, not the code: `iter_content` is required to always yield `bytes`. You'll find if you go through the tests and use `bytes` everywhere the problem goes away. :grin: So we don't need this.

I'm confused: I checked in a normal usage (outside of the tests) and could reproduce the issue:
with Python2 I get `str`, and with Python3 I get `bytes` (and if I use `iter_lines(decode_unicode=True)`
I get `str` (Py3) or `unicode` (Py2)

The previous code wasn't accepting bytes (and was working only with Python 2)

This failed on Python3 before my last fix:
```
r = requests.get("https://github.com", stream=True)
list(r.iter_lines())
```
TypeError: endswith first arg must be bytes or a tuple of bytes, not int

That's why I don't think the tests were in error (a test that I didn't write was failing, `test_response_iter_lines`)

[USER_REDACTED] The issue here seems to be stemming from 46e3fdd in #3745. The conditional on master, and prior to that commit, is done by [NAME_REDACTED] of each string/bytestring. When retrieving a single index from a bytestring in Python 3 it returns an integer representation of the byte. This works in the case of `lines[-1] and lines[-1][-1] == chunk[-1]` but not `lines[-1] and lines[-1].endswith(chunk[-1])`.

I'd suggest we simply move back to using the original double indexing, rather than introducing more logic to handle the seemingly unneeded `endswith`.

Also, I'm not sure if you're going to back this commit out, but if not, `decode_unicode` doesn't seem to be used for anything here in this tests current configuration.

Thanks, I missed that... much neater. It works fine with a simple indexing in all cases.

Nice catch
I overwrote the commit, and fixed the unused `decode_unicode` parameter :ok_hand:


fix #3902 Add proposed documentation change for post data

@@ -234,6 +234,45 @@ dictionary of data will automatically be form-encoded when the request is made::
       ...
     }

+You can also pass a list of tuples to the ``data`` argument. This is particularly
+useful when the form has multiple elements that use the same key::
+
+    >>> payload = (('key1', 'value1'), ('key1', 'value2'))
+    >>> r = requests.post('http://httpbin.org/post', data=payload)
+    >>> print(r.text)
+    {
+      ...
+      "form": {
+        "key1": [
+          "value1",
+          "value2"
+        ]
+      },
+      ...
+    }
+
+Value of each payload element can be a scalar or an iterable::
+
+    >>> payload = (('key1', ('value1', 1, (1, 2))), {'key2', ('value2', 2)})
+    >>> r = requests.post('http://httpbin.org/post', data=payload)
+    >>> print(r.text)
+    {
+      ...
+      "form": {
+        "key1": [
+          "value1",
+          "1",
+          "1",
+          "2"
+        ],
+        "key2": [
+          "value2",
+          "2"
+        ]
+      },
+      ...
+    }
+
 There are many times that you want to send data that is not form-encoded. If
 you pass in a ``string`` instead of a ``dict``, that data will be posted directly.

@@ -485,7 +524,7 @@ Timeouts
 --------

 You can tell Requests to stop waiting for a response after a given number of
-seconds with the ``timeout`` parameter. Nearly all production code should use
+seconds with the ``timeout`` parameter. Nearly all production code should use
 this parameter in nearly all requests. Failure to do so can cause your program
 to hang indefinitely::

Thanks for this! I have a few small notes, but this seems pretty good to me.

This should begin "The value". Also, let's avoid the word "scalar": let's say "single element".

We try not to document that we accept non-string elements here, in part because we don't promise to support them.

Thanks. Sounds good.

In that case,  this section goes away. I can just delete this section so only tuples section above remains. Does that sound good?

Works for me. 👍

Agreed.

:sparkles: :cake: :sparkes:


Refactor Prepare Body

This PR supersedes #3338 with the branch brought up to date, as well as a few minor changes, including logic tweaks based on updates to `super_len` since the original PR.

This will forcibly prevent the transmission of headers that include both Content-Length and Transfer-Encoding, as well as simplify redundant code we use for both `prepare_content_length` and `prepare_body`. All the credit here goes to [USER_REDACTED], the original creator of this PR.

@@ -1,6 +1,10 @@
 3.0.0 (2017-xx-xx)
 ++++++++++++++++++

+- Simplified logic for determining Content-Length and Transfer-Encoding.
+  Requests will now avoid setting both headers on the same request, and
+  raise an exception if this is done manually by a user.
+
 - Remove the HTTPProxyAuth class in favor of supporting proxy auth via
   the proxies parameter.

@@ -104,9 +104,14 @@ class StreamConsumedError(RequestException, TypeError):
 class RetryError(RequestException):
     """Custom retries logic failed"""

+
 class UnrewindableBodyError(RequestException):
     """Requests encountered an error when trying to rewind a body"""

+
+class InvalidBodyError(RequestException, ValueError):
+    """An invalid request body was specified"""
+
 # Warnings

@@ -32,12 +32,14 @@
     LocationParseError, ConnectionError)
 from .exceptions import (
     HTTPError, MissingScheme, InvalidURL, ChunkedEncodingError,
-    ContentDecodingError, ConnectionError, StreamConsumedError)
+    ContentDecodingError, ConnectionError, StreamConsumedError,
+    InvalidHeader, InvalidBodyError)
 from ._internal_utils import to_native_string, unicode_is_ascii
 from .utils import (
     guess_filename, get_auth_from_url, requote_uri,
     stream_decode_response_unicode, to_key_val_list, parse_header_links,
-    iter_slices, guess_json_utf, super_len, check_header_validity)
+    iter_slices, guess_json_utf, super_len, check_header_validity,
+    is_stream)
 from .compat import (
     cookielib, urlunparse, urlsplit, urlencode, str, bytes, StringIO,
     is_py2, chardet, builtin_str, basestring)
@@ -466,17 +468,7 @@ def prepare_body(self, data, files, json=None):
             if not isinstance(body, bytes):
                 body = body.encode('utf-8')

-        is_stream = all([
-            hasattr(data, '__iter__'),
-            not isinstance(data, (basestring, list, tuple, collections.Mapping))
-        ])
-
-        try:
-            length = super_len(data)
-        except (TypeError, AttributeError, UnsupportedOperation):
-            length = None
-
-        if is_stream:
+        if is_stream(data):
             body = data

             if getattr(body, 'tell', None) is not None:
@@ -493,10 +485,6 @@ def prepare_body(self, data, files, json=None):
             if files:
                 raise NotImplementedError('Streamed bodies and files are mutually exclusive.')

-            if length:
-                self.headers['Content-Length'] = builtin_str(length)
-            else:
-                self.headers['Transfer-Encoding'] = 'chunked'
         else:
             # Multi-part file uploads.
             if files:
@@ -509,27 +497,40 @@ def prepare_body(self, data, files, json=None):
                     else:
                         content_type = 'application/x-www-form-urlencoded'

-            self.prepare_content_length(body)
-
             # Add content-type if it wasn't explicitly provided.
             if content_type and ('content-type' not in self.headers):
                 self.headers['Content-Type'] = content_type

+        self.prepare_content_length(body)
         self.body = body

     def prepare_content_length(self, body):
-        """[NAME_REDACTED] header based on request method and body"""
+        """[NAME_REDACTED] header.
+
+        If the length of the body of the request can be computed, Content-Length
+        is set using ``super_len``. If user has manually set either a
+        Transfer-Encoding or Content-Length header when it should not be set
+        (they should be mutually exclusive) an InvalidHeader
+        error will be raised.
+        """
         if body is not None:
             length = super_len(body)
+
             if length:
-                # If length exists, set it. Otherwise, we fallback
-                # to Transfer-Encoding: chunked.
                 self.headers['Content-Length'] = builtin_str(length)
+            elif is_stream(body):
+                self.headers['Transfer-Encoding'] = 'chunked'
+            else:
+                raise InvalidBodyError('Non-null body must have length or be streamable.')
         elif self.method not in ('GET', 'HEAD') and self.headers.get('Content-Length') is None:
             # [NAME_REDACTED] to 0 for methods that can have a body
             # but don't provide one. (i.e. not GET or HEAD)
             self.headers['Content-Length'] = '0'

+        if 'Transfer-Encoding' in self.headers and 'Content-Length' in self.headers:
+            raise InvalidHeader('[NAME_REDACTED]: Both Transfer-Encoding and '
+                                'Content-Length are set.')
+
     def prepare_auth(self, auth, url=''):
         """Prepares the given HTTP auth data."""

@@ -865,6 +865,7 @@ def urldefragauth(url):

     return urlunparse((scheme, netloc, path, params, query, ''))

+
 def rewind_body(prepared_request):
     """Move file pointer back to its recorded starting position
     so it can be read again on redirect.
@@ -878,3 +879,10 @@ def rewind_body(prepared_request):
                                         "body for redirect.")
     else:
         raise UnrewindableBodyError("Unable to rewind request body for redirect.")
+
+
+def is_stream(data):
+    """Given data, determines if it should be sent as a stream."""
+    is_iterable = getattr(data, '__iter__', False)
+    is_io_type = not isinstance(data, (basestring, list, tuple, collections.Mapping))
+    return is_iterable and is_io_type

@@ -25,7 +25,7 @@
 from requests.exceptions import (
     ConnectionError, ConnectTimeout, InvalidScheme, InvalidURL,
     MissingScheme, ReadTimeout, Timeout, RetryError, TooManyRedirects,
-    ProxyError, InvalidHeader, UnrewindableBodyError)
+    ProxyError, InvalidHeader, UnrewindableBodyError, InvalidBodyError)
 from requests.models import PreparedRequest
 from requests.structures import CaseInsensitiveDict
 from requests.sessions import SessionRedirectMixin
@@ -1924,6 +1924,61 @@ def test_chunked_upload_does_not_set_content_length_header(self, httpbin):
         assert 'Transfer-Encoding' in prepared_request.headers
         assert 'Content-Length' not in prepared_request.headers

+    def test_chunked_upload_with_manually_set_content_length_header_raises_error(self, httpbin):
+        """Ensure that if a user manually sets a content length header, when
+        the data is chunked, that an InvalidHeader error is raised.
+        """
+        data = (i for i in [b'a', b'b', b'c'])
+        url = httpbin('post')
+        with pytest.raises(InvalidHeader):
+            r = requests.post(url, data=data, headers={'Content-Length': 'foo'})
+
+    def test_content_length_with_manually_set_transfer_encoding_raises_error(self, httpbin):
+        """Ensure that if a user manually sets a Transfer-Encoding header when
+        data is not chunked that an InvalidHeader error is raised.
+        """
+        data = 'test data'
+        url = httpbin('post')
+        with pytest.raises(InvalidHeader):
+            r = requests.post(url, data=data, headers={'Transfer-Encoding': 'chunked'})
+
+    def test_null_body_does_not_raise_error(self, httpbin):
+        url = httpbin('post')
+        try:
+            requests.post(url, data=None)
+        except InvalidHeader:
+            pytest.fail('InvalidHeader error raised unexpectedly.')
+
+    [USER_REDACTED].mark.parametrize(
+        'body, expected', (
+            (None, ('Content-Length', '0')),
+            ('test_data', ('Content-Length', '9')),
+            (io.BytesIO(b'test_data'), ('Content-Length', '9')),
+            (StringIO.StringIO(''), ('Transfer-Encoding', 'chunked'))
+        ))
+    def test_prepare_content_length(self, httpbin, body, expected):
+        """Test prepare_content_length creates expected header."""
+        prep = requests.PreparedRequest()
+        prep.headers = {}
+        prep.method = 'POST'
+
+        # [NAME_REDACTED] is set appropriately.
+        key, value = expected
+        prep.prepare_content_length(body)
+        assert prep.headers[key] == value
+
+    def test_prepare_content_length_with_bad_body(self, httpbin):
+        """Test prepare_content_length raises exception with unsendable body."""
+        # Initialize minimum required PreparedRequest.
+        prep = requests.PreparedRequest()
+        prep.headers = {}
+        prep.method = 'POST'
+
+        with pytest.raises(InvalidBodyError) as e:
+            # Send object that isn't iterable and has no accessible content.
+            prep.prepare_content_length(object())
+            assert "Non-null body must have length or be streamable." in str(e)
+
     def test_custom_redirect_mixin(self, httpbin):
         """Tests a custom mixin to overwrite ``get_redirect_target``.

Cool, so this mostly looks good. Just a few notes. =)

Cool, this looks good. Can we get a changelog entry for it explaining the change thoroughly? This is the kind of thing that will trip people up in migration. :grin:

Then I should be ready to give you a green tick.

Looks like you committed a `.swo` file. :cry:

Cool, this looks good to me. Thanks [USER_REDACTED]! :sparkles: :cake: :sparkles:

Only minor note I have here is we've added an `InvalidHeader` exception since this PR was created which could likely be used instead of this. I didn't want to tweak the original work too much in the initial pass though. Thoughts?

Yeah, might be better to use that new exception, or possibly to make this a child exception of `InvalidHeader`.

I don't think this is ever imported, is it?

Did we need to get rid of this comment?

Nope, I think that was probably a bad merge conflict resolution.

It wasn't but  it should be added now. Thoughts on adding a test to ensure hitting this block?

I think we should do that, yeah. Ideally we'd cover all of this logic if we can.

Alright, the other tests from #3338 cover the other two cases according to pytest. We now have 100% coverage over `prepare_content_length` with the new test. This should be set after the build completes, unless you have further notes.

We probably don't need this paragraph. 😎

A little too thorough I guess 😉

Updated.


Cache proxy bypass

Added simple cache over `urllib.proxy_bypass` to remedy #2988. Holds on to entries for a minute. Once it reaches its max size, will evict the first entry that is the oldest.

Added tests over the structure, did not add a test that `should_bypass_proxies` invokes the cache.

This is my first pull request; please let me know if I've overlooked something.

Additionally, when I check out master, run `make coverage` in pipenv, the test named 'TestRequests.test_proxy_error' fails. This is still the only failing test when I use my branch. The error is `ConnectionError: ('Connection aborted.', BadStatusLine("''",))`. Is there some additional setup I need to do to run the test suite?

@@ -178,4 +178,5 @@ Patches and Suggestions
 - [NAME_REDACTED] <[EMAIL_REDACTED]> ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
 - [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])
+- [NAME_REDACTED] ([CONTRIBUTOR_REDACTED])

@@ -8,9 +8,12 @@
 """

 import collections
+import time

 from .compat import OrderedDict

+current_time = getattr(time, 'monotonic', time.time)
+

 class CaseInsensitiveDict(collections.MutableMapping):
     """A case-insensitive ``dict``-like object.
@@ -103,3 +106,89 @@ def __getitem__(self, key):

     def get(self, key, default=None):
         return self.__dict__.get(key, default)
+
+
+class TimedCacheManaged(object):
+    """
+    Wrap a function call in a timed cache
+    """
+    def __init__(self, fnc):
+        self.fnc = fnc
+        self.cache = TimedCache()
+
+    def __call__(self, *args, **kwargs):
+        key = args[0]
+        found = None
+        try:
+            found = self.cache[key]
+        except KeyError:
+            found = self.fnc(key, **kwargs)
+            self.cache[key] = found
+
+        return found
+
+
+class TimedCache(collections.MutableMapping):
+    """
+    Evicts entries after expiration_secs. If none are expired and maxlen is hit,
+    will evict the oldest cached entry
+    """
+    def __init__(self, maxlen=32, expiration_secs=60):
+        """
+        :param maxlen: most number of entries to hold on to
+        :param expiration_secs: the number of seconds to hold on
+        to entries
+        """
+        self.maxlen = maxlen
+        self.expiration_secs = expiration_secs
+        self._dict = OrderedDict()
+
+    def __repr__(self):
+        return '<TimedCache maxlen:%d len:%d expiration_secs:%d>' % \
+            (self.maxlen, len(self._dict), self.expiration_secs)
+
+    def __iter__(self):
+        return map(lambda kv: (kv[0], kv[1][1]), self._dict.items()).__iter__()
+
+    def __delitem__(self, item):
+        return self._dict.__delitem__(item)
+
+    def __getitem__(self, key):
+        """
+        Look up an item in the cache. If the item
+        has already expired, it will be invalidated and not returned
+
+        :param key: which entry to look up
+        :return: the value in the cache, or None
+        """
+        occurred, value = self._dict[key]
+        now = int(current_time())
+
+        if now - occurred > self.expiration_secs:
+            del self._dict[key]
+            raise KeyError
+        else:
+            return value
+
+    def __setitem__(self, key, value):
+        """
+        Locates the value at lookup key, if cache is full, will evict the
+        oldest entry
+
+        :param key: the key to search the cache for
+        :param value: the value to be added to the cache
+        """
+        now = int(current_time())
+
+        while len(self._dict) >= self.maxlen:
+            self._dict.popitem(last=False)
+
+        return self._dict.__setitem__(key, (now, value))
+
+    def __len__(self):
+        """:return: the length of the cache"""
+        return len(self._dict)
+
+    def clear(self):
+        """Clears the cache"""
+        return self._dict.clear()

@@ -28,7 +28,7 @@
     quote, urlparse, bytes, str, OrderedDict, unquote, getproxies,
     proxy_bypass, urlunparse, basestring, integer_types)
 from .cookies import RequestsCookieJar, cookiejar_from_dict
-from .structures import CaseInsensitiveDict
+from .structures import CaseInsensitiveDict, TimedCache, TimedCacheManaged
 from .exceptions import (
     InvalidURL, InvalidHeader, FileModeWarning, UnrewindableBodyError)

@@ -579,6 +579,16 @@ def set_environ(env_name, value):
             os.environ[env_name] = old_value

+[USER_REDACTED]
+def _proxy_bypass_cached(netloc):
+    """
+    Looks for netloc in the cache, if not found, will call proxy_bypass
+    for the netloc and store its result in the cache
+
+    :rtype: bool
+    """
+    return proxy_bypass(netloc)
+
 def should_bypass_proxies(url, no_proxy):
     """
     Returns whether we should bypass proxies or not.
@@ -626,7 +636,7 @@ def should_bypass_proxies(url, no_proxy):
     # legitimate problems.
     with set_environ('no_proxy', no_proxy_arg):
         try:
-            bypass = proxy_bypass(netloc)
+            bypass = _proxy_bypass_cached(netloc)
         except (TypeError, socket.gaierror):
             bypass = False

@@ -2,7 +2,7 @@

 import pytest

-from requests.structures import CaseInsensitiveDict, LookupDict
+from requests.structures import CaseInsensitiveDict, LookupDict, TimedCache, TimedCacheManaged

 class TestCaseInsensitiveDict:
@@ -74,3 +74,75 @@ def test_getitem(self, key, value):
     [USER_REDACTED]_item_parameters
     def test_get(self, key, value):
         assert self.lookup_dict.get(key) == value
+
+
+class TestTimedCache(object):
+    [USER_REDACTED].fixture(autouse=True)
+    def setup(self):
+        self.any_value = 'some value'
+        self.expiration_secs = 60
+        self.cache = TimedCache(expiration_secs=self.expiration_secs)
+        yield
+        self.cache.clear()
+
+    def test_get(self):
+        self.cache['a'] = self.any_value
+        assert self.cache['a'] is self.any_value
+
+    def test_repr(self):
+        repr = str(self.cache)
+        assert repr == '<TimedCache maxlen:32 len:0 expiration_secs:60>'
+
+    def test_get_expired_item(self, mocker):
+        self.cache = TimedCache(maxlen=1, expiration_secs=self.expiration_secs)
+
+        mocker.patch('requests.structures.current_time', lambda: 0)
+        self.cache['a'] = self.any_value
+        mocker.patch('requests.structures.current_time', lambda: self.expiration_secs + 1)
+        assert self.cache.get('a') is None
+
+    def test_evict_first_entry_when_full(self, mocker):
+        self.cache = TimedCache(maxlen=2, expiration_secs=2)
+        mocker.patch('requests.structures.current_time', lambda: 0)
+        self.cache['a'] = self.any_value
+        mocker.patch('requests.structures.current_time', lambda: 1)
+        self.cache['b'] = self.any_value
+        mocker.patch('requests.structures.current_time', lambda: 3)
+        self.cache['c'] = self.any_value
+        assert len(self.cache) is 2
+        with pytest.raises(KeyError, message='Expected key not found'):
+            self.cache['a']
+        assert self.cache['b'] is self.any_value
+        assert self.cache['c'] is self.any_value
+
+    def test_delete_item_removes_item(self):
+        self.cache['a'] = self.any_value
+        del self.cache['a']
+        with pytest.raises(KeyError, message='Expected key not found'):
+            self.cache['a']
+
+    def test_iterating_hides_timestamps(self):
+        self.cache['a'] = 1
+        self.cache['b'] = 2
+        expected = [('a', 1), ('b', 2)]
+        actual = [(key, val) for key, val in self.cache]
+        assert expected == actual
+
+
+class TestTimedCacheManagedDecorator(object):
+    def test_caches_repeated_calls(self, mocker):
+        mocker.patch('requests.structures.current_time', lambda: 0)
+
+        nonlocals = {'value': 0}
+
+        [USER_REDACTED]
+        def some_method(x):
+            nonlocals['value'] = nonlocals['value'] + x
+            return nonlocals['value']
+
+        first_result = some_method(1)
+        assert first_result is 1
+        second_result = some_method(1)
+        assert second_result is 1
+        third_result = some_method(2)
+        assert third_result is 3

Cool, so this is a good start! I've left some notes inline in the diff.

Cool, this is generally looking really good: substantial progress! I've written some more notes.

Awesome, really close now! A couple of minor stylistic notes and one request to expand a test slightly and we're good to go!

Sorry, I missed some extra whitespace. 😔 Let's clear all of that up and then we will be good to go. 👍

Cool, I'm happy with this! As this is a big-ish change in functionality, I'd like to give [USER_REDACTED] a chance to review before I merge, but if he doesn't get time I'll merge it Monday. Great work [USER_REDACTED]!

I'll submit these updates in a separate pull request

So, this is basically a `collections.MutableMapping`! We should make it *actually* be one, because that gives us a number of handy-dandy convenience methods.

So, generally speaking it is better to implement `get` in terms of `__getitem__` than the other way around. `__getitem__` should almost always maintain dict-like behaviour of raising `KeyError` if the entry is not present, rather than returning `None`. The core concern here is that it needs to be possible to distinguish between "key not present in dictionary" and "key present with a value of `None`".

This logic is wrong. Happily, it doesn't matter: if you make this a subclass of `MutableMapping` you don't actually need to implement this method at all, we get it for free.

So, this is a problem.

*If possible* we'd like to use `time.monotonic` instead, as `time.time` can go backwards. So I think in this module you'll need to feature detect on import to see whether `time.monotonic` is present and to prefer it if it is.

Same note about monotonic time.

You're wrapping this paragraph *very* short. Feel free to go all the way out to 89 columns!

While we're here, can you add a blank line separating the paragraph blurb from the `:param:` directives?

This complete search can probably be avoided by [NAME_REDACTED] `OrderedDict`, can't it? urllib3 includes one that works on all the Python versions we support, so you can import that and use it as the base implementation to save ourselves this iterative walk.

My suggestion is that this be reimplemented as a generic decorator, rather than written to specifically handle `proxy_bypass`. That way we get a much easier time of testing the timed cache and all the relevant code. =)

Let's avoid making this an old-style class by putting `(object)` as well.

Thanks. To clarify, you're saying if `time.monotonic` is not present, it's ok to fall back to `time.time`?

Will update.

Will update.

Yes.

Will update.

Will update to have TimedCache inherit `collections.MutableMapping`

Will update.

Will replace `self._dict` with an `OrderedDict`.

You can have a slightly wider docstring than this! :grin:

This should be `TimedCache`, and it should also provide the `expiration_secs`.

You don't need to implement this: `collections.MutableMapping` will do it for you.

This can be more cleanly expressed as:

```python
while len(self._dict) >= self.maxlen:
    self._dict.popitem(last=False)
```

This one we also technically get for free from `MutableMapping`, but in this case we can do it much faster than the generic implementation so I'm happy to have this override.

Heh, so this is a good test, but it mostly tests implementation details. For this, it's enough to just mock out the time and then write a small function that constantly returns an incrementing value, then wrap it in the decorator and use the timer to demonstrate the behaviour.

Generally speaking, avoiding mocks is a good thing wherever we can. =)

Fixed

Fixed

Fixed

Agreed. Updated.

Good to know. I've left the override intact.

Ok. I've re-made the test more along these lines. Let me know what you think.

Really minor nit, we don't need this blank line.

Same again, don't need this blank line.

This again can be flowed out to column 79 as well.

Should also check that a different argument doesn't return a cached value.

Removed

Fixed

Removed

This one.

This one

This one

This one

This one

This one

This one.

I'm curious why we don't just build a generator here.

1. `map()` is slow and `lambda` s are slow. This intentional on Guido's part.
2. Calling `__iter__` is really poor form.
3. This would be far simpler and easier to read as

```py
return ((key, value[1]) for key, value in self._dict.items())
```

Why not write this as:

```py
del self._dict[item]
```

`__delitem__` doesn't return anything so there's no need to worry about returning something here. And it would avoid calling the dunder method directly, which I'll repeat, is poor form.

This should raise a `KeyError` with the key as an argument, e.g., `KeyError(key)`.

Again, `__setitem__` doesn't return anything. This should just be

```py
self._dict[key] = (now, value)
```

There's no need to obfuscate any of this.


variable name consistency

Just need to clean up a few variable names.

@@ -87,13 +87,13 @@ def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict):

 class SessionRedirectMixin(object):
-    def get_redirect_target(self, resp):
+    def get_redirect_target(self, response):
         """Receives a Response. Returns a redirect URI or ``None``"""
-        if resp.is_redirect:
+        if response.is_redirect:
             if not is_valid_location(response):
                 raise InvalidHeader('Response contains multiple Location headers. '
                                     'Unable to perform redirect.')
-            return resp.headers['location']
+            return response.headers['location']
         return None

     def resolve_redirects(self, response, request, stream=False, timeout=None,
@@ -113,8 +113,8 @@ def resolve_redirects(self, response, request, stream=False, timeout=None,

             # Update history and keep track of redirects.
             # response.history must ignore the original request in this loop
-            hist.append(response)
-            response.history = hist[1:]
+            history.append(response)
+            response.history = history[1:]

             try:
                 response.content  # Consume socket so it can be released

One small note.

I've requested Kenneth's input here because I know he has opinions but I've frankly forgotten them on this topic.

I'm definitely happy with this adjustment.

Let's call the other instances `history`, rather than call this `hist`.